AI QA Engineer

Datarails is a financial solution designed for SMBs that turns manual monthly and quarterly reporting processes into automated ones.

The solution allows organizations to move towards working with a cloud-based financial product, all while end-users continue working on their usual Excel files, as they have up until now.

The solution connects to all organizational systems, including your ERP, CRM, HRIS, as well as complex Excel files, and consolidates all of this data in one cloud-based database. Datarails automates various financial transformations, including FX conversions, eliminations, hierarchies, financial adjustments, and more.

The company’s core clientele is within the US market. Datarails is experiencing tremendous growth even during these difficult times, as organizations need to conduct rigorous financial and business analyses quickly and easily.

 This is an exciting opportunity to work on cutting-edge technologies and collaborate with a talented team of developers.

As an AI QA Engineer, you will be responsible for ensuring the quality and reliability of features built on Large Language Models (LLMs). You will evaluate and validate model outputs, assess prompt designs, identify regressions, and work closely with development teams to maintain predictable and accurate behaviour of AI-driven systems.

In this role, you will combine strong QA fundamentals with an understanding of how LLMs behave, along with deep FP&A knowledge to ensure that AI-generated insights and outputs align with financial planning and analysis requirements.

Your expertise will help detect issues, communicate root causes, and support the continuous improvement of our AI-driven financial products.

• Strong understanding of QA fundamentals and software testing principles
• Experience evaluating and validating outputs of AI / LLM systems
• Basic understanding of how Large Language Models work (prompt sensitivity, randomness, hallucinations, context, versions)
• Ability to read, review and improve prompts and test scenarios for LLM behavior
• Ability to distinguish between issues caused by prompts, models, or data
• Experience designing and executing structured test scenarios for AI features
• Ability to identify regressions and assess impact of small changes on model output
• Analytical mindset with excellent attention to detail

• Strong business and domain knowledge in FP&A (financial planning & analysis), including: budgeting & forecasting concepts, variance analysis, financial metrics & KPIs, financial modelling basics, Ability to validate correctness of financial insights generated by models
• Good communication skills and ability to work with Development, Product and Data teams
• English – Upper-Intermediate or higher

Advantages:

• Practical experience with prompt optimization or creative prompt design
• Experience using LLM evaluation / logging tools (Langfuse, W&B, custom eval dashboards)
• Familiarity with advanced LLM architecture concepts
• Experience in financial modelling