Lead AI Engineer – Quality Engineering

Company Description

KMS Technology is a strategic engineering company helping businesses turn bold ideas into high-impact solutions—faster. Founded in 2009 as a U.S.-based services company, we’ve grown into a global organization with locations in the US, Vietnam, Mexico and Poland. KMS is trusted globally for the quality of our engineering and consulting services. We bring deep expertise in product development and quality assurance, Data & AI-native engineering, and delivery excellence to every engagement. Our mission is to help customers build what’s next—accelerating innovation, crafting brilliant solutions, and creating  real-world impact. At KMS, we believe sustainable growth is built on the success of our clients and employees, and in making a lasting contribution to our communities.

More about KMS Technology:

• Website: https://kms-technology.com 
• Linkedin: https://www.linkedin.com/company/kms-technology 

Job Description

We are looking for a Lead AI Engineer to design, build, and operate intelligent agents that automate software testing end-to-end.

This is an AI engineering role applied to quality engineering — not a traditional test automation role with AI added on top. You will own agentic workflows across test-case generation, automation-script creation, execution, validation, failure analysis, self-healing, and continuous test maintenance. The role integrates AI agents with application knowledge, automation tools, APIs, databases, and CI/CD pipelines to enable reliable end-to-end testing and sets the technical direction for how agentic quality engineering is delivered across teams.

You will lead the technical direction of this capability and mentor a team of engineers (team size: TBC) as the practice scales.

Key Responsibilities

Build agentic systems

• Design and develop autonomous and multi-agent workflows for test automation.
• Build AI agents using LLMs, prompt engineering, RAG, tool calling, and MCP-based integrations.
• Develop agents for test generation, code creation, validation, debugging, failure triage, and self-healing.
• Maintain application knowledge bases (embeddings, retrieval, context strategy) and continuously improve agent decision-making.

Make them reliable

• Define agent instructions, guardrails, validation checkpoints, and human-in-the-loop review processes.
• Establish evaluation practice: golden datasets, regression evals, LLM-as-judge where appropriate, and clear accuracy/reliability metrics.
• Instrument agents with tracing and observability; monitor and improve accuracy, reliability, latency, token cost, and maintainability in production.

Integrate and ship

• Integrate agents with UI, API, database, and end-to-end testing workflows.
• Integrate agentic solutions with GitHub, Jenkins, APIs, databases, and other engineering tools.

Lead

• Set technical standards and best practices for agentic development and AI-assisted testing.
• Mentor engineers, run design reviews, and raise the team's level of AI engineering maturity.

Qualifications

Must Have

• 6+ years of software engineering experience, including proven experience leading technical delivery or mentoring engineers.
• 1.5+ years building LLM-based or agentic systems that reached production — not proof-of-concept work only.
• Strong hands-on experience with LLM integration, prompt engineering, RAG, and AI-agent development.
• Proficiency in Python or TypeScript, with solid experience building and consuming REST APIs.
• Hands-on experience with at least one agent orchestration framework (e.g. LangGraph, OpenAI Agents SDK, CrewAI, AutoGen, Semantic Kernel, or equivalent), including tool integration, structured outputs, and validation mechanisms.
• Practical experience with AI evaluation and observability — designing eval datasets, measuring agent quality, and using tracing tools (e.g. LangSmith, Langfuse, Arize Phoenix, or a custom harness).
• Experience with vector databases, embeddings, and knowledge management for retrieval.
• Solid understanding of Git, CI/CD pipelines, and modern software-development practices.
• Good English communication skills for collaborating with distributed teams and stakeholders.

Nice-to-Have

• Working knowledge of Playwright with TypeScript.
• Background in UI, API, database, or end-to-end test automation.
• Familiarity with Angular or React applications.
• Experience with MCP (Model Context Protocol) servers and clients.
• Experience optimizing LLM cost and latency (caching, model routing, token budgeting).
• Experience with Jenkins or comparable CI orchestration at scale.

Additional Information

Perks You'll Enjoy 

• Working in one of the Best Places to Work in Vietnam
• Building large-scale & global software products
• Working & growing with Passionate & Talented Team
• Diverse careers opportunities with Software Outsourcing, Software Product Development, IT Solutions & Consulting
• Attractive Salary and Benefits
• Performance appraisal once a year and performance bonus
• Onsite opportunities: short-term and long-term assignments in North American (U.S, Canada), Europe, Asia.
• Flexible working time
• Various training on hot-trend technologies, best practices and soft skills
• Premium healthcare insurance for you and your loved ones
• Company trip, big annual year-end party every year, team building, etc.
• Fitness & sport activities: football, tennis, table-tennis, badminton, yoga, swimming…
• Joining community development activities: 1% Pledge, charity every quarter, blood donation, public seminars, career orientation talks,…
• Free in-house entertainment facilities (foosball, ping pong, gym…), coffee, and snack (instant noodles, cookies, candies…)

And much more, join us and let yourself explore other fantastic things!