AI Test Manager :: Greenwood Village, CO :: ONLY LOCAL

[Hitesh Masurkar]

AI Test Manager

Job Title: AI Solutions Evaluation & Experience Manager – Contractor

Location: [Greenwood Village, CO]

Contract Duration: [6 months, with potential for extension]

 

Job Overview:
We are seeking a highly technical AI Solutions Evaluation & Experience Manager to architect the framework that determines the real-world intelligence and accuracy of our AI and analytic products and perform ongoing solution testing and monitoring. In this role, you will move beyond traditional QA to implement LLM-as-a-Judge and SLM patterns to evaluate performance at scale.

You will be responsible for managing a team that will calculate and interpret critical technical metrics, such as RAGas scores (Faithfulness, Answer Relevance), Perplexity, and Hallucination rates, to determine model accuracy. Additionally, you will analyze adoption and utilization metrics to assess the health of our product portfolio and trigger safe product retirements, when appropriate.

5+ years of experience in management and 3+ years of experience in technology or AI are required. A thorough understanding of test script development, requirements traceability, and testing execution for technical solutions is required.