HireMO

JOB SUMMARY

· The Quality Engineer, AI & Test Automation helps modernize BSWH's quality approach by moving beyond traditional manual testing toward automation-first, engineering-integrated, and AI-enabled quality practices.

· This role designs, builds, executes, and maintains automated test suites, validation utilities, evaluation assets, and quality reporting that improve release confidence and speed of delivery across digital products, enterprise applications, and AI-enabled use cases.

· The role partners closely with Product, Engineering, Architecture, Data & Analytics, AI Foundation, Ontology, Security, Privacy, Clinical, Operations, and other delivery teams to define quality expectations early and embed quality practices throughout the development lifecycle.

· This role is expected to combine strong testing discipline with technical skills, including automation, API validation, data validation, CI/CD integration, quality metrics, defect triage, and emerging AI-enabled testing practices.

· The Quality Engineer, AI & Test Automation reports to the Director of Quality Engineering and helps establish reusable practices that support safe, reliable, efficient, and scalable delivery.

ESSENTIAL FUNCTIONS OF THE ROLE

Quality Engineering & Test Strategy

· Develop test strategies, test plans, acceptance criteria, and quality approaches for software and AI-enabled capabilities in partnership with Product, Engineering, Architecture, and business stakeholders.

· Translate requirements, user stories, workflows, APIs, data dependencies, and operational expectations into clear test scenarios and quality validation plans.

· Identify quality risks early in the delivery lifecycle and recommend appropriate validation approaches, including automated, manual, exploratory, integration, regression, performance, accessibility, or AI-specific evaluation methods.

· Provide clear release-readiness input based on test evidence, defect trends, quality metrics, risk assessment, and stakeholder expectations.

Test Automation & Engineering Practices

· Design, build, maintain, and improve automated test suites for API, UI, integration, end-to-end, regression, data, and workflow validation.

· Integrate automated tests into CI/CD pipelines and delivery workflows so quality signals are available earlier and more consistently throughout the development lifecycle.

· Create reusable test data, utilities, fixtures, scripts, and automation patterns that can be used across multiple teams and products.

· Improve test reliability, maintainability, execution time, coverage, and signal-to-noise ratio by reducing brittle scripts and eliminating repetitive manual validation where practical.

AI-Enabled Testing & Evaluation

· Support testing and evaluation of AI-enabled experiences, including agent workflows, generated responses, retrieval behavior, tool/API calls, escalation paths, guardrail behavior, and human-in-the-loop patterns.

· Help define evaluation datasets, expected behaviors, failure modes, scoring rubrics, regression scenarios, and quality thresholds for AI-enabled capabilities.

· Use AI-enabled testing tools where appropriate to generate test cases, analyze logs, triage defects, identify coverage gaps, summarize results, and accelerate quality workflows.

· Partner with AI Quality, Architecture, Engineering, and Product teams to validate that AI-enabled capabilities are useful, reliable, explainable enough for the use case, and aligned to defined quality expectations.

Defect Management, Root Cause & Continuous Improvement

· Identify, document, reproduce, triage, and communicate defects with sufficient technical detail to support efficient resolution by engineering teams.

· Analyze defect patterns, escaped defects, quality trends, release issues, and production feedback to identify root causes and improvement opportunities.

· Collaborate with engineering teams to improve testability, observability, logging, error handling, and diagnostic capabilities across applications and AI-enabled workflows.

· Contribute to retrospectives, quality reviews, and process improvements that reduce rework, improve release confidence, and increase delivery velocity.

Cross-Functional Partnership & Release Readiness

· Work closely with Product Managers, Senior AI Product Managers, Software Engineers, AI Engineers, Data Product Owners, Architects, Designers, Clinical and Operational stakeholders, and vendors/partners to align on quality expectations and release readiness.

· Ensure quality criteria are embedded in requirements, design reviews, backlog refinement, development, testing, implementation, and post-release monitoring.

· Support production validation, monitoring handoffs, post-release assessment, incident follow-up, and quality reporting for assigned products and use cases.

· Communicate quality status, risks, defects, test results, and recommendations clearly to technical and non-technical stakeholders.

Quality Standards, Documentation & Enablement

· Create and maintain test documentation, automation standards, quality playbooks, release-readiness checklists, defect triage practices, and reusable testing patterns.

· Help teams adopt automation-first and AI-enabled quality practices while clarifying where manual validation remains necessary due to risk, ambiguity, or workflow complexity.

· Share knowledge with other quality, product, engineering, and business stakeholders to raise BSWH's overall quality engineering capability.

· Stay current on modern testing approaches, automation frameworks, AI evaluation practices, test data management, observability, and quality engineering methods.

KEY SUCCESS FACTORS

· Demonstrated ability to build and maintain meaningful automated tests rather than relying primarily on late-stage manual testing.

· Strong technical judgment across API testing, UI testing, integration testing, data validation, regression testing, defect triage, CI/CD integration, and release readiness.

· Ability to test AI-enabled systems where behavior may be probabilistic, workflow-dependent, or context-sensitive rather than purely deterministic.

· Ability to use AI-enabled productivity tools responsibly to improve test generation, analysis, defect triage, and quality reporting while maintaining human judgment over release quality.

· Strong collaboration with product and engineering teams to define quality expectations early and build quality into delivery practices rather than inspect quality at the end.

· Strong analytical and communication skills, including the ability to explain quality risks, evidence, tradeoffs, and release recommendations to technical and non-technical stakeholders.

· Comfort operating in a regulated or high-trust environment where safety, privacy, security, compliance, auditability, and customer trust are critical.

BENEFITS

Our competitive ...

Apply Externally

Jobs

Quality Engineer, AI & Test Automation

Summary

You Need a MO Jobs Account First

Jobs

Jobs

Quality Engineer, AI &amp; Test Automation

Summary

Do You Have a MO Jobs Account?

First, Let's Find Your MO Jobs Account

Create an Account to Apply

You Need a MO Jobs Account First

HireMO Terms of Use

1. Acceptance and Changes

2. A No-Cost Public Workforce Service

3. Eligibility, Youth, and Authority

4. Accounts and Security

5. Accuracy Attestation and Program Information

6. Job Seeker Terms

7. Employer Terms

8. Assessments and Matching

9. Artificial Intelligence Features

10. Public Records and Required Notices

11. User Content and Permissions

12. Acceptable Use

13. Intellectual Property and Government Marks

14. Other Users and Third-Party Services

15. Electronic Communications

16. Suspension, Termination, and Records

17. Disclaimers

18. Limitation of Liability and Sovereign Immunity

19. Employer Indemnification

20. Governing Law and General Terms

21. Contact, Accessibility, and Complaints

Quality Engineer, AI & Test Automation