How Accurate Is AI Essay Grading? We Published Our Data.
AI essay grading can save teachers time, but an important question remains: how accurately can AI grade an essay compared with a human marker?
A useful answer requires more than a demonstration or a promising grade. It requires evidence showing how AI scores compare with human scores, how often they disagree and what happens when an essay is difficult to assess.
At Progressay, our AI Grading Assistant uses grading rubrics to assess written work and provide feedback. Progressay publicly reports an 85% computer-to-human marker agreement rate in trials involving Pearson. This is an encouraging result, but understanding what the figure means requires looking at the methodology behind it.
In this article, we explain how AI grading accuracy should be measured, what our reported result tells us and why human oversight remains important.
How Accurate Is AI Essay Grading?
AI essay grading accuracy depends on the essay, the marking rubric, the AI system and the way accuracy is measured.
An AI system may award the same mark as a human examiner, a mark within one point of the examiner or a different mark while identifying similar strengths and weaknesses. These outcomes should not be treated as equivalent.
Progressay reports an 85% computer-to-human marker agreement rate from its trials. However, a percentage alone does not tell the whole story. To interpret the result properly, readers need to know the number of essays assessed, the definition of agreement, the marking criteria and how the human reference scores were established.
Our position is simple: AI grading should be evaluated against clear marking criteria and real human-marked work, not just whether its feedback sounds convincing.
What Does Progressay's 85% Marker Agreement Mean?
Progressay's AI Grading Assistant page reports 85% computer-to-human agreement in trials involving Pearson.
Agreement measures how closely AI-generated marks align with marks assigned by human markers. However, the exact interpretation depends on the agreement metric used.
For example:
-
Exact agreement: AI and the human marker award precisely the same score.
-
Agreement within one mark: The AI score differs from the human score by no more than one mark.
-
Weighted agreement: Small differences may count less than larger differences, depending on the scoring method.
These measures answer different questions. An 85% exact-agreement rate is not the same as 85% of scores falling within one mark.
To ensure transparency, Progressay should publish the precise definition used in its trials, along with the sample size and scoring conditions.
How We Evaluate AI Essay Grading
A meaningful evaluation should compare AI scores with independently assigned human marks using the same essays and marking criteria.
The following details should accompany Progressay's published trial result.
| Methodology detail | Information to report |
|---|---|
| Number of essays | [Insert actual sample size] |
| Subjects and essay types | [List subjects and question types tested] |
| Human reference marks | [Explain who marked the essays and how] |
| AI grading setup | [Describe model, rubric and relevant settings] |
| Agreement definition | [Exact agreement, within one mark or another metric] |
| Results period | [Insert trial dates] |
| Limitations | [Explain sample and subject limitations] |
These details help readers understand whether the results apply to a particular set of essays or a broader range of educational assessments.
What We Measure
A robust AI grading evaluation should examine more than one number.
1. Agreement with human markers
How frequently does the AI award the same mark as the human marker? Report exact agreement separately from agreement within a defined mark range.
2. Score differences
How far apart are AI and human scores when they disagree? A one-mark difference may have a different practical impact from a much larger difference.
3. Consistency
Does the system grade similar responses consistently when the same rubric and assessment conditions are used?
4. Feedback quality
Does the feedback identify relevant strengths, weaknesses and improvements that are supported by the student's actual response?
5. Performance across different essays
Does agreement vary by subject, question type, essay length or score band? Reporting these differences helps educators understand where additional checks may be necessary.
Where Can AI Essay Grading Disagree with Human Markers?
AI grading is not infallible. Differences can arise when an answer is ambiguous, a rubric allows room for interpretation or a response demonstrates an unusual but valid approach.
Common areas to investigate include:
-
Interpretation: The AI and human marker interpret the question or assessment criteria differently.
-
Evidence and reasoning: The AI overlooks a relevant point or gives too much weight to a less important one.
-
Mark boundaries: A response sits between two possible score bands.
-
Writing quality: The system overemphasises spelling, grammar or vocabulary instead of the criteria being assessed.
-
Unusual answers: A valid alternative argument or creative approach does not fit the expected response pattern.
These are potential sources of disagreement, not claims about the frequency of errors in Progressay's trials. A published disagreement analysis should use actual examples from the evaluated essays, with student information anonymised.
Is AI Essay Grading as Reliable as Human Marking?
There is no single answer for every AI system, subject or assessment.
A 2024 study published in Computers and Education: Artificial Intelligence compared ChatGPT scoring with human ratings across three collections of student essays. It found that agreement between AI and human scores varied across samples, even when average scores were similar. The researchers concluded that AI scoring could be useful for low-stakes formative assessment, but cautioned against treating it as a substitute for human scoring in higher-stakes assessments.
Read the research on AI and holistic essay scoring.
This distinction matters. Similar average scores do not necessarily mean an AI system gives the correct mark to every individual student. Educators should examine score-level agreement, the size of disagreements and the suitability of the system for the intended assessment.
How Progressay Uses AI to Support Essay Grading
Progressay's AI Grading Assistant allows educators to provide an essay question, configure a grading rubric and receive AI-generated grades and diagnostic feedback.
The aim is to reduce repetitive marking work while helping teachers identify areas where students may need support.
AI grading can be particularly useful for practice, formative feedback and identifying patterns across a set of responses. Teachers can use the output to guide further feedback and review, while retaining professional judgement over important assessment decisions.
Try Progressay's AI Grading Assistant
Want to explore how rubric-based AI marking could support your teaching?
Try Progressay's AI Grading Assistant
Build a rubric, grade written work and explore AI-generated feedback designed to help students understand where they can improve.
Frequently Asked Questions
How accurate is AI essay grading?
Accuracy varies by system, rubric, subject and scoring method. Progressay reports 85% computer-to-human marker agreement in trials, but the percentage should be interpreted using the trial's exact agreement definition and methodology.
Can AI grade essays as accurately as teachers?
AI can produce marks that agree with human markers, but agreement varies across assessments. Human review remains important, especially for high-stakes decisions and responses requiring nuanced judgement.
What does an 85% AI grading agreement rate mean?
It means the reported agreement rate between computer-generated and human-assigned marks was 85% under the trial's measurement method. To understand the result, readers need to know whether it represents exact agreement, agreement within a mark range or another metric.
Is AI essay grading reliable enough for exams?
AI can support practice and formative assessment, but reliability must be demonstrated for the particular exam, rubric and intended use. High-stakes grades should follow the relevant assessment and quality-assurance requirements.
How can teachers check AI grading accuracy?
Compare AI scores with independently assigned human marks, define the agreement metric in advance, analyse score differences, examine feedback quality and test performance across different essay types.
Conclusion: AI Grading Needs Evidence, Not Just Promises
AI essay grading has the potential to reduce marking workloads and provide faster feedback, but its value depends on how reliably it assesses student work.
Progressay's reported 85% marker agreement is a starting point for understanding its performance. Publishing the underlying methodology, sample details, agreement definitions and disagreement analysis would help educators assess the result in context.
The goal should not be to claim that AI never makes mistakes. It should be to measure its performance transparently, understand its limitations and use it responsibly to support better teaching and learning.
Explore Progressay's AI Grading Assistant to see how rubric-based AI grading can support your marking workflow.
