ML Data QA Lead, MLO
Apple
1 day ago
On-site
Cupertino, California, United States
QA Lead
Do you believe Machine Learning and AI can change how people experience technology? We truly believe it can! We are the Machine Learning Data Ops Team, part of the Intelligent System Experience (ISE) group within Apple"s software engineering organization. We build high-quality ML datasets at scale to train the models that power AI-centric features across iPhone, iPad, Mac, Apple Watch, and AirPods. Those features include Apple Intelligence, recognizing the people you love in your Photos app, and the input experiences you rely on every day such as autocorrect, next-word prediction, and handwriting recognition. Data is the source code of these models, and its quality determines whether a feature works beautifully for everyone or only for some.\\n\\nWe are looking for a talented individual to drive data quality assurance for the ML features we support, working closely with our Data Program Managers, Data Engineers, and R\u0026D partners to ensure that the data delivered to R\u0026D meets Apple"s rigorous quality standards. This is a role for someone who is both a rigorous quality thinker and hands-on with the tooling, using AI to build new QA capabilities and extend what we already have. \\n\\nWe invite you to join us at this exciting time and positively impact multiple critical features from your first day at Apple.
The Machine Learning Data Ops QA team ensures that Research and Development teams receive complete, accurate, and consistent datasets to train the models powering continuous feature development. We support our data collection, annotation and synthesis partners with defining quality standards and verifying that data deliverables meet this high quality bar before they are consumed by R\u0026D teams. \\n\\nAs the Data Quality Lead, you own the quality of the datasets in your portfolio and the standards they are measured against. The role spans the full data request life cycle: defining what good looks like with R\u0026D before collection begins, designing checks that catch problems during collection rather than after delivery, leading the analysts who carry out review, and reporting findings to project teams, partner organizations, and vendors. You will also build and extend the team"s QA tooling, including review interfaces, analysis pipelines, and reporting, using agentic AI tools to add new capabilities and to find more efficient ways of delivering high quality data.
Owns the quality strategy and roadmap across the data request life cycle, defining the workflows and process controls that anticipate failure modes, expose edge cases, detect anomalies, and surface issues early rather than at delivery.\\nTranslates ambiguous quality expectations into explicit, documented standards and decision rules that vendors, reviewers, and partner teams can apply consistently.\\nRuns quality assurance and quality control checks across pilot and production phases, and validates trends before datasets reach R\u0026D.\\nDesigns and builds new QA tools, and extends the review interfaces, data pipelines, and reporting the team already relies on, using AI-assisted development.\\nIntroduces model-assisted checks into review workflows, iterating on prompts and measuring agreement against iterating on prompts and measuring agreement against human labels before those checks are relied on.\\nVerifies analyses, metrics, and generated artifacts independently before they are published.\\nLeads internal and external quality analysts, and presents quality findings, statistics, and recommendations to project teams, partner organizations, and vendors.\\nPartners with Collection, Annotation, and R\u0026D teams to pin down project specifications, reduce subjectivity, and ensure guidelines are unambiguous to everyone who applies them.
Bachelor"s degree, or equivalent practical experience.\\n4+ years of experience in ML data operations, data quality, or a comparable data-centric quality function.\\nWorking proficiency in Python for data manipulation and reporting.\\nHands-on experience using AI coding assistants to build working QA tools or analysis.\\nStrong written and verbal communication skills.
Experience designing labeling taxonomies or annotation guidelines and adjudicating ambiguous cases with vendors.\\nExperience leading internal or external quality analysts and designing or running human rating and evaluation programs, including rater calibration, gold sets, and ongoing quality monitoring.\\nFamiliarity with statistical quality methods, including sampling strategy, inter-rater agreement, acceptance rates, and error magnitude and confidence analysis.\\nExperience designing and iterating on prompts for quality checks assisted by large language models (LLMs) or vision language models (VLMs).\\nExperience building internal QA tooling end to end, such as a review interface, a data pipeline, or a browser-based dashboard (HTML, CSS, JavaScript).\\nExcellent attention to detail with a passion for problem solving, investigation, and root cause analysis.\\nStrong critical thinking, with the judgment to question assumptions and validate a quality signal before relying on it.\\nExcellent project management, analytical, and organizational skills, with the ability to manage several projects in parallel in a dynamic environment with shifting priorities.
The Machine Learning Data Ops QA team ensures that Research and Development teams receive complete, accurate, and consistent datasets to train the models powering continuous feature development. We support our data collection, annotation and synthesis partners with defining quality standards and verifying that data deliverables meet this high quality bar before they are consumed by R\u0026D teams. \\n\\nAs the Data Quality Lead, you own the quality of the datasets in your portfolio and the standards they are measured against. The role spans the full data request life cycle: defining what good looks like with R\u0026D before collection begins, designing checks that catch problems during collection rather than after delivery, leading the analysts who carry out review, and reporting findings to project teams, partner organizations, and vendors. You will also build and extend the team"s QA tooling, including review interfaces, analysis pipelines, and reporting, using agentic AI tools to add new capabilities and to find more efficient ways of delivering high quality data.
Owns the quality strategy and roadmap across the data request life cycle, defining the workflows and process controls that anticipate failure modes, expose edge cases, detect anomalies, and surface issues early rather than at delivery.\\nTranslates ambiguous quality expectations into explicit, documented standards and decision rules that vendors, reviewers, and partner teams can apply consistently.\\nRuns quality assurance and quality control checks across pilot and production phases, and validates trends before datasets reach R\u0026D.\\nDesigns and builds new QA tools, and extends the review interfaces, data pipelines, and reporting the team already relies on, using AI-assisted development.\\nIntroduces model-assisted checks into review workflows, iterating on prompts and measuring agreement against iterating on prompts and measuring agreement against human labels before those checks are relied on.\\nVerifies analyses, metrics, and generated artifacts independently before they are published.\\nLeads internal and external quality analysts, and presents quality findings, statistics, and recommendations to project teams, partner organizations, and vendors.\\nPartners with Collection, Annotation, and R\u0026D teams to pin down project specifications, reduce subjectivity, and ensure guidelines are unambiguous to everyone who applies them.
Bachelor"s degree, or equivalent practical experience.\\n4+ years of experience in ML data operations, data quality, or a comparable data-centric quality function.\\nWorking proficiency in Python for data manipulation and reporting.\\nHands-on experience using AI coding assistants to build working QA tools or analysis.\\nStrong written and verbal communication skills.
Experience designing labeling taxonomies or annotation guidelines and adjudicating ambiguous cases with vendors.\\nExperience leading internal or external quality analysts and designing or running human rating and evaluation programs, including rater calibration, gold sets, and ongoing quality monitoring.\\nFamiliarity with statistical quality methods, including sampling strategy, inter-rater agreement, acceptance rates, and error magnitude and confidence analysis.\\nExperience designing and iterating on prompts for quality checks assisted by large language models (LLMs) or vision language models (VLMs).\\nExperience building internal QA tooling end to end, such as a review interface, a data pipeline, or a browser-based dashboard (HTML, CSS, JavaScript).\\nExcellent attention to detail with a passion for problem solving, investigation, and root cause analysis.\\nStrong critical thinking, with the judgment to question assumptions and validate a quality signal before relying on it.\\nExcellent project management, analytical, and organizational skills, with the ability to manage several projects in parallel in a dynamic environment with shifting priorities.