Designs the tests, benchmarks and scoring systems that measure whether a large language model or AI product is accurate, reliable and safe enough to ship. BLS does not publish separate figures for software quality assurance analysts and testers; it counts them within qa analyst (SOC 15-1253), whose median pay is $104,300 a year (BLS OEWS, May 2025). Typical entry-level education: Bachelor's degree.
- SOC 15-1253
- Computer and Mathematical
- Last updated October 9, 2026
At a glance
LLM Evaluation Engineer in four numbers
- Median pay
- $104,300a year, or $50.14/hrBLS OEWS, May 2025Lowest 10% $61KTop 10% $167K
US median, all jobs: $50,980
- Projected growth
- +5.7%Faster than averageBLS EP, 2025–2035This career+5.7%All jobs+3.5%
Projected employment change, 2025–2035 Item Value This career +5.7% All jobs +3.5% - Openings a year
- 10,700187,600 jobs todayBLS EP, 2025–2035
- Jobs in 2025187.6K
- Projected 2035198.2K
Employment, 2025–2035 Item Value Jobs in 2025 187.6K Projected 2035 198.2K - Typical education
- Bachelor'sneeded to enterBLS EP, 2025–2035; mix: O*NET
- High school2.9%
- Certificate8.8%
- Some college8.8%
- Associate's26.5%
- Bachelor's50%
- Master's2.9%
Highest education held by software quality assurance analysts and testers Segment Value High school 2.9% Certificate 8.8% Some college 8.8% Associate's 26.5% Bachelor's 50% Master's 2.9%
Also advertised as: AI Evaluation Engineer, Model Evaluation Engineer, Evals Engineer, AI Quality Engineer, LLM Evaluation Specialist.
Pay
How much do Software Quality Assurance Analysts and Testers make?
Every llm evaluation engineer in the US, by percentile of annual wage
Median
$104,300
Middle half earn $80,310 to $133,180
- 10th
- $61,440
- 25th
- $80,310
- 75th
- $133,180
- 90th
- $167,010
All US occupations: $50,980
| Percentile | Value |
|---|---|
| 10th | $61,440 |
| 25th | $80,310 |
| Median | $104,300 |
| 75th | $133,180 |
| 90th | $167,010 |
| All US occupations | $50,980 |
Source: BLS OEWS, May 2025
- Median annual wage
- $104,300
- Median hourly wage
- $50.14/hr
- Mean annual wage
- $111,490
- People employed
- 186,740
Versus all US occupations
At $104,300, Software Quality Assurance Analysts and Testers earn 105% above the $50,980 median across all US occupations, a difference of $53,320 a year.
Versus computer and mathematical occupations
Inside the Computer and Mathematical group, where the middle occupation pays $116,580, this is 11% below the group median.
Pay range
The top 10% earn $167,010 or more while the bottom 10% earn $61,440, a gap of 2.7×. That is a typical spread for a US occupation.
Rank in the bank
This median pays more than 78% of the 1,741 occupations in the NueCareer career bank.
Cost of living
California posts the highest raw median ($128,740), but once local prices are taken into account New Mexico leads at $126,426 in national-average dollars.
Pay by state
Which states pay Software Quality Assurance Analysts and Testers the most?
State medians come from the same OEWS release as the national figures. The cost-of-living column divides each state median by that state's BEA Regional Price Parity, restating pay in national-average dollars so a high-cost state and a low-cost one can be compared directly.
| State | Median | COL-adjusted | Employed |
|---|---|---|---|
| 1California | $128,740 | $116,275 | 27,710 |
| 2Washington | $128,480 | $120,060 | 7,970 |
| 3Colorado | $123,690 | $120,027 | 5,110 |
| 4Massachusetts | $122,210 | $115,557 | 6,640 |
| 5Virginia | $121,590 | $120,262 | 10,160 |
| 6New York | $121,240 | $112,341 | 10,460 |
| 7New Jersey | $119,470 | $109,802 | 6,550 |
| 8New Mexico | $116,580 | $126,426 | 450 |
| 9Maryland | $111,100 | $105,851 | 5,880 |
| 10Oregon | $108,440 | $104,914 | 1,840 |
Job outlook
Is LLM Evaluation Engineer a growing career?
Employment change over 2025–2035, against every US occupation
Faster than average. +5.7% projected growth vs 3.5% for all occupations.
| Item | Value |
|---|---|
| Software Quality Assurance Analysts and Testers | +5.7% |
| All US occupations | +3.5% |
Source: BLS EP, 2025–2035
Faster than average
BLS projects employment of Software Quality Assurance Analysts and Testers to change 5.7% over 2025–2035, against 3.5% projected for all occupations. That is 10,600 more jobs.
Annual openings
About 10,700 openings are projected each year, 5.7% of the 187,600 jobs that existed in 2025. Most come from workers leaving the occupation, not from growth alone.
Getting in
How to become a LLM Evaluation Engineer
Most come from software engineering, data science or machine learning, usually with a bachelor's degree in computer science, statistics or a related field. Strong Python, comfort with statistics and experiment design, and familiarity with model APIs are the core skills. Some arrive from quality assurance or from a research background in linguistics or psychology measurement.
Public benchmark projects such as Stanford's HELM and open-source evaluation frameworks such as the UK AI Security Institute's Inspect are good places to learn how serious evaluations are built. A portfolio of evaluation suites or contributions to open benchmarks helps you stand out, as does experience writing clear rubrics for human graders.
- 1
Typical entry-level education
Bachelor's degree
- 2
Work experience in a related occupation
None
- 3
Typical on-the-job training
None
- 4
Preparation needed (O*NET job zone)
Medium preparation
BLS Employment Projections, 2025–2035; job zone from O*NET 31.0 Database.
What education people in the job have
O*NET incumbent survey: the highest level of education reported by people already working as software quality assurance analysts and testers.
- Bachelor's Degree50%
- Associate's Degree or other 2-year degree26.5%
- Post-Secondary Certificate8.8%
- Some College Courses8.8%
- High School Diploma or the equivalent (such as a GED)2.9%
| Item | Value |
|---|---|
| Bachelor's Degree | 50% |
| Associate's Degree or other 2-year degree | 26.5% |
| Post-Secondary Certificate | 8.8% |
| Some College Courses | 8.8% |
| High School Diploma or the equivalent (such as a GED) | 2.9% |
Source: O*NET 31.0 Database
The work
What does a LLM Evaluation Engineer do?
LLM evaluation engineers answer a hard question for AI teams: is this model actually better? They build test sets of prompts with known good answers, write scoring code, and set up automated pipelines that run every new model version through the same checks. Because language output is hard to grade, they combine exact-match tests, rubric-based human review and model-graded scoring, and they study where these methods disagree.
They also design evaluations for specific products, such as whether a customer support bot follows company policy, whether a coding assistant writes code that passes tests, or whether a model refuses harmful requests without refusing harmless ones. Their reports often decide whether a release goes ahead, so they spend time explaining results to researchers and product managers.
Day-to-day tasks
- Design test datasets that measure specific model capabilities
- Write automated scoring code and grading rubrics
- Run evaluation pipelines on new model versions
- Compare human ratings with automated and model-graded scores
- Investigate failures and report patterns to research teams
- Track evaluation results over time on dashboards
- Build safety and policy-compliance test suites for AI products
Typical tasks, researched by NueCareer from the sources below.
Working conditions and hours
LLM evaluation engineers work at AI labs, large tech companies, startups building on language models, and government AI safety institutes. The work is computer-based, often hybrid or remote, with regular hours that tighten before model launches. It combines coding, reading model transcripts and presenting findings to research and product teams.
Where the work happens
Indoors, climate-controlled
NeverEvery dayOutdoors in all weather
NeverEvery dayAlmost never works outdoors.
Physical demands
Sitting
NeverAlmost all the timeSeated for more than half the working day.
Standing
NeverAlmost all the timeBending or twisting
NeverAlmost all the time
Working with people
Contact with other people
NoneConstantFace-to-face discussions
NeverEvery dayFace-to-face discussions every day.
Dealing with the public
Not importantExtremely important
Pressure and stakes
Strict deadlines
NeverEvery dayConflict situations
NeverEvery dayCost of a mistake
Not seriousExtremely serious
Autonomy
Freedom to decide
NoneA lotA lot of freedom to make decisions without checking first.
Conditions: O*NET Work Context survey. Each item has its own scale, so each dot is read against the two answers that end its own line.
Core skills
O*NET importance, 1 to 5
- Reading Comprehension4.0
- Active Listening3.9
- Speaking3.9
- Critical Thinking3.9
- Writing3.8
- Monitoring3.6
- Active Learning3.1
- Learning Strategies2.9
| Item | Value |
|---|---|
| Reading Comprehension | 4.0 |
| Active Listening | 3.9 |
| Speaking | 3.9 |
| Critical Thinking | 3.9 |
| Writing | 3.8 |
| Monitoring | 3.6 |
| Active Learning | 3.1 |
| Learning Strategies | 2.9 |
Source: O*NET 31.0 Database
Transferable skills
O*NET importance, 1 to 5
- Programming3.6
- Complex Problem Solving3.4
- Quality Control Analysis3.4
- Systems Evaluation3.3
- Operations Analysis3.1
- Judgment and Decision Making3.1
- Systems Analysis3.1
- Coordination3.0
| Item | Value |
|---|---|
| Programming | 3.6 |
| Complex Problem Solving | 3.4 |
| Quality Control Analysis | 3.4 |
| Systems Evaluation | 3.3 |
| Operations Analysis | 3.1 |
| Judgment and Decision Making | 3.1 |
| Systems Analysis | 3.1 |
| Coordination | 3.0 |
Source: O*NET 31.0 Database
Tools and software
- Analytical or scientific software · in demand
- Application server software · in demand
- Business intelligence and data analysis software · in demand
- Cloud-based data access and sharing software · in demand
- Cloud-based management software · in demand
- Computer aided design CAD software · in demand
- Configuration management software · in demand
- Customer relationship management CRM software · in demand
- Data base management system software · in demand
- Data base reporting software · in demand
- Data base user interface and query software · in demand
- Data mining software · in demand
“In demand” marks an O*NET Employer-Based Hot Technology, software named in real job postings for this occupation.
Personality fit
What personality type suits a LLM Evaluation Engineer?
NueCareer estimate (Holland code ICR) built on O*NET's QA Analyst profile, scored 1 to 7
| Axis | LLM Evaluation Engineer |
|---|---|
| Realistic | 4 |
| Investigative | 6 |
| Artistic | 2 |
| Social | 2 |
| Enterprising | 2 |
| Conventional | 6 |
Source: O*NET 31.0 Database
What the work is built around
Day to day this is work built around investigating, analysing and solving problems with ideas and data, combined with organising, following procedures and working precisely with data.
- Investigative5.8
- Conventional5.7
- Realistic4.1
- Artistic1.7
- Enterprising1.6
- Social1.5
Closest MBTI types
NueCareer fit scores: cosine similarity between each MBTI type's interest vector and this occupation's O*NET interest profile. Not a BLS or O*NET figure; the method is documented on our methodology page.
Explore further
Careers related to LLM Evaluation Engineer
O*NET related occupations
- Computer and Mathematical
- Computer and Mathematical
- Architecture and Engineering
- Computer and Mathematical
- Computer and Mathematical
- Computer and Mathematical
More computer and mathematical careers
Median annual pay, with software quality assurance analysts and testers highlighted
- Software Developer$135,980
- Actuary$130,000
- Data Scientist$120,230
- Business Intelligence Analyst$120,230
- Database Administrator$104,620
- LLM Evaluation Engineer$104,300
- Web Designer$104,000
- Computer Programmer$100,390
- Web Developer$92,650
| Item | Value |
|---|---|
| Software Developer | $135,980 |
| Actuary | $130,000 |
| Data Scientist | $120,230 |
| Business Intelligence Analyst | $120,230 |
| Database Administrator | $104,620 |
| LLM Evaluation Engineer | $104,300 |
| Web Designer | $104,000 |
| Computer Programmer | $100,390 |
| Web Developer | $92,650 |
Source: BLS OEWS, May 2025
Lists LLM Evaluation Engineer appears on
Questions
LLM Evaluation Engineer: frequently asked questions
How much do Software Quality Assurance Analysts and Testers make?
BLS does not publish a separate wage for software quality assurance analysts and testers. It counts them within qa analyst, whose median pay is $104,300 a year (BLS OEWS, May 2025). The middle half of that occupation earn between $80,310 and $133,180, and the top 10% earn $167,010 or more.
Do you need a degree to become a LLM Evaluation Engineer?
Yes. BLS reports a bachelor's degree as the typical entry-level education for Software Quality Assurance Analysts and Testers, and O*NET rates the preparation needed as medium preparation. In O*NET's survey of people already doing the job, 50.0% hold bachelor's degree.
Is LLM Evaluation Engineer a growing career?
Yes. Employment is projected to grow 5.7% over 2025–2035, faster than the 3.5% average for all occupations (BLS EP, 2025–2035). BLS still projects about 10,700 openings a year, mostly from workers retiring or moving to other work.
What does a LLM Evaluation Engineer do?
LLM evaluation engineers answer a hard question for AI teams: is this model actually better? A typical task: design test datasets that measure specific model capabilities. Employers most often ask for Analytical or scientific software, Application server software, Business intelligence and data analysis software.
Which states pay Software Quality Assurance Analysts and Testers the most?
The highest state medians for Software Quality Assurance Analysts and Testers are California ($128,740), Washington ($128,480), Colorado ($123,690) (BLS OEWS, May 2025). Adjusted for local prices using BEA Regional Price Parities, New Mexico moves to the top at $126,426.
What personality type suits a LLM Evaluation Engineer?
O*NET scores Software Quality Assurance Analysts and Testers highest on the Investigative, Conventional and Realistic interest areas, so the work mixes investigating, analysing and solving problems with ideas and data with organising, following procedures and working precisely with data. On NueCareer's MBTI-to-interest mapping that lines up most closely with ISTJ personalities. Personality is a fit signal, not a gate. The free NueCareer quiz scores your own profile against all 1,731 careers in the bank.