This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a QA Automation Engineer β Enterprise Data & AI based in the United States.
This role sits at the intersection of quality engineering, data validation, and modern cloud data platforms, ensuring the reliability and accuracy of enterprise-scale data pipelines. You will work within a Databricks-powered ecosystem to validate complex data transformations, ingestion flows, and business rules across large datasets.
The position plays a critical role in strengthening data quality frameworks that support analytics, reporting, and downstream business decision-making.
You will collaborate closely with data engineers and quality teams to identify issues, improve test coverage, and enhance automated validation processes.
A strong focus is placed on building scalable, reusable automation within CI/CD pipelines and ensuring consistent data integrity across environments.
This is a hands-on engineering role in a fast-paced, data-driven environment where precision and automation directly impact enterprise data trust.
You will contribute to the evolution of modern data quality practices across a cloud-based data platform.
Accountabilities:
- Execute and extend automated data validation tests within Databricks using Python, PySpark, SQL, and notebook-based frameworks.
- Validate end-to-end data pipelines, including ingestion, batch and incremental loads, transformations, joins, and business rule accuracy.
- Perform data reconciliation between source systems and target datasets to ensure completeness and consistency.
- Enhance and maintain existing data quality frameworks, including rule sets for accuracy, completeness, and reliability.
- Implement and monitor validation checks, thresholds, alerts, and exception handling mechanisms.
- Develop reusable and scalable automated test scripts aligned with enterprise data testing standards.
- Integrate automated tests into CI/CD pipelines (e.g., Azure DevOps) and ensure reliable execution across environments.
- Support testing activities across QA and staging environments, including defect triage and root cause analysis.
- Collaborate with data engineering and analytics teams to ensure data integrity for reporting and visualization tools such as Tableau.
Requirements:
- 5+ years of experience in QA automation, SDET, or data validation engineering roles.
- Strong hands-on experience with Databricks, including notebook development and data pipeline validation.
- Advanced proficiency in Python, PySpark, SQL, and data processing workflows.
- Proven experience in data reconciliation and large-scale data validation across enterprise systems.
- Experience building, extending, or maintaining data quality frameworks in complex environments.
- Familiarity with CI/CD pipelines such as Azure DevOps for test integration and execution.
- Strong analytical, debugging, and problem-solving skills with attention to detail.
- Ability to collaborate effectively with data engineers, QA teams, and cross-functional stakeholders.
- Experience with tools such as Azure Purview or Profisee MDM is a plus.
Benefits:
- Competitive compensation aligned with experience and expertise.
- Fully remote opportunity within the United States.
- Exposure to enterprise-scale data platforms and modern cloud data engineering practices.
- Opportunity to work with Databricks and advanced data quality frameworks.
- Collaborative engineering culture focused on data innovation and automation.
- Career growth in enterprise data engineering, QA automation, and AI-driven data ecosystems.
- Inclusive and flexible work environment supporting work-life balance.