Executive Summary

Thirty-five AI models have been handed this year's Korean university entrance exam and graded on it. The CSAT-benchmark dashboard, which scores Korean and foreign models on the 2027 academic-year College Scholastic Ability Test, went up on November 20, and the person who set that exam paper is Koo Yu-gyeom, a student in the computer software engineering department at Soonchunhyang University. Not the state institute that writes the exam, and not a government agency.

Two models took the full 1,276 points there, the sum of raw scores across every subject on offer: OpenAI's GPT-6.2 Astra and Google's Gemini 4 Neon. A second individual benchmark on GitHub graded the same exam, and on that one the only model with full marks is Claude Fable 5.2. The totals differ before the models do, 1,276 against 1,226, and the difference comes from which subjects each person put inside the exam.

Sections 1 to 4 set out what the published scorecards and the reporting confirm. Section 5, which reads the situation as a problem of evaluation data, is this blog's own judgment.

Key figures

Sources: AI Times (2026-11-20) · Aju Business Daily (2026-11-20) · 27suneung-bench

The first card is the only full-subject perfect score the second exam paper awarded, and on the first exam paper a different model holds that place. The second and third cards show how far the leading Korean model has come and where it stopped. The fourth is the floor on what one sitting of this exam costs.

1,226 / 1,226

The second exam paper's only perfect score

Claude Fable 5.2, which does not hold a full-subject perfect score on the first paper

414.7 pts

K-EXAONE 2.5, the top Korean model

On the 450-point scale, more than 35 points behind the joint foreign leaders

60.4%

K-EXAONE 2.5 on questions carrying an image

The top three models answered the same questions at 100%

1,171 won

Cheapest run across every subject

GPT-6.1 Terra, one of the models inside the 450-point group

1

Two Exam Papers, Two Sets of Rules

Koo Yu-gyeom, who publishes under the handle hehee9, has graded the major AI models on the CSAT every year since 2024 and posted the results. This is the third run. This time 35 models were given the whole 2027 exam: every elective in Korean language and mathematics, English, Korean history and all 17 inquiry subjects. Out of a full-subject raw-score total of 1,276, GPT-6.2 Astra and Gemini 4 Neon took every point.

Koo is not the only person who set an exam paper this year. The GitHub user GonGe1018 published 27suneung-bench, which puts the full-subject total at 1,226 points, and there exactly one model answered all 1,226: Claude Fable 5.2. GPT-6.2 Astra, perfect on the first paper, sits second here with 1,223, followed by Claude Opus 5.5 on 1,221 and GPT-6.2 Sol on 1,220.

The same CSAT, graded by two individuals CSAT-benchmark · Koo Yu-gyeom 1,276 points across all subjects · 35 models GPT-6.2 Astra Perfect All subjects, 17 inquiry subjects included Gemini 4 Neon Perfect All subjects, 17 inquiry subjects included Claude Fable 5.2 450-point group 27suneung-bench · GonGe1018 1,226 points across all subjects Claude Fable 5.2 1,226 pts 100.00% accuracy GPT-6.2 Astra 1,223 pts 99.79% accuracy Claude Opus 5.5 / GPT-6.2 Sol 1,221 / 1,220 Why the totals are 50 points apart 1,276: all Korean and math electives + English + Korean history + 17 inquiry subjects 1,226: Korean + math + English + science I and II + social studies (no Korean history)
▲ Pebblous original diagram. It places the top full-subject results each benchmark published side by side; the scores are not directly comparable. | Sources: AI Times (2026-11-20) · 27suneung-bench

The 50-point gap between the two totals is a question of coverage. The 1,276 includes Korean history and all 17 inquiry subjects; the 1,226 covers Korean language, mathematics, English, science studies I and II and social studies, with Korean history left out. The marking rules are not the same either. 27suneung-bench states that accuracy is computed over the questions a model actually answered while raw scores apply each question's own point value, so unanswered questions drop out of the accuracy figure. Its results table does carry models with anywhere from 20 to 180 questions left blank.

Neither marking is wrong. Two different people decided how wide the exam should be and what to do with a question left blank. The trouble starts afterwards. As the sentence "AI scored full marks on the CSAT" travels into headlines and social media, the coverage and the marking rules fall away and only the ranking survives. And the ranking changes with the exam paper.

The two records of how AI did on this year's CSAT are both the work of individuals. One is run single-handedly by a university student for the third year; the other is a GitHub repository. Both published their methods and both published their results. Neither withheld anything. What is missing is the seat that reconciles the two, or decides which set of rules counts as the standard.

2

How the 35 Models Were Run and Graded

The run conditions for CSAT-benchmark were published. Models were called through their official APIs; open-weight models with no official API went through OpenRouter, and on-device models were run directly. Questions were supplied as text while graphs and tables were supplied as images. No system prompt was used, and no external tools either, meaning no search and no calculator.

That last condition fixes what kind of score this is. It is what comes out when the questions are thrown at a model with no search, no calculator and no prompt engineering. It also means the result should not be read as the answer a person would get by pasting the same question into a chatbot window.

There are two scales. One is the 1,276 that adds up the raw scores of every subject set. The other is the 450 that a real candidate faces, picking one elective and two inquiry subjects. Reporting mostly quoted the 450, and on that scale the number of models able to reach full marks rises to nine: the two joint leaders plus GPT-6.2 Sol, GPT-6.1 Terra, Gemini 4 Argon, Gemini 4 Carbon, Gemini 4 Hydrogen, Claude Fable 5.2 and Claude Opus 5.5. Change the subject combination and the perfect scorers go from two to nine.

Other things were logged besides the scores. The fastest model was GPT-6.2 Sol at about 48 minutes for the whole exam, and the cheapest was GPT-6.1 Terra at 1,171 won. Among models that are free to use or ship with open weights, the highest score was Claude Sonnet 5.5 on 442.0, and Kimi K3.1 Pro, Qwen 4 Max, DeepSeek V4.1 Flash and Claude Haiku 5.5 all cleared 420. At the other end Gemma 4 E2B landed on 206.9 and Gemma 4 E4B on 251.8, near half of full marks. That is the stretch where model size shows up directly as score.

Because one person has measured the same way for three years, the change from year to year is on the record as well. Koo said the models averaged bands two and three on Korea's nine-band grading scale at the CSAT two years ago, reached band one last year without quite managing full marks, and this year took full marks across every elective subject too. Two years ago is the year he first ran this evaluation. A result people discussed in grade bands became a perfect score within two years, and the ruler that measured the change stayed in one pair of hands the whole time.

The outlets that carried the result attached a condition of their own. ZDNet Korea wrote that this is an unofficial evaluation run by an individual developer, that models differ in whether they support image input and in their reasoning settings, and that the setup differs from the environment ordinary users meet in a web service.

3

Korean Models Fall Behind on Inquiry Subjects and Images

On the 450-point scale the Korean models line up as follows: K-EXAONE 2.5 from LG AI Research on 414.7, Upstage's Solar Mini 4.5 Preview on 397.8, the same company's Solar Pro 4 on 392.8, Krafton's Raon-1 Mini Preview on 357.2 and KT's Mi:dm K 3.0 Pro on 352.8. More than 35 points separate K-EXAONE 2.5, the domestic leader, from the joint foreign leaders.

The subject breakdown says where those 35 points went. K-EXAONE 2.5 took full marks in mathematics and in Korean history, scored 98 in Korean language and 96 in English, and then managed only 70.7 in the inquiry section. The headline subjects held. The losses came in inquiry, where the subjects are many and the passages vary.

A second divide runs along the shape of the question. In the tally reported by Aju Business Daily, accuracy on questions containing an image was 100% for GPT-6.2 Astra, Gemini 4 Neon and GPT-6.2 Sol alike. On those same questions K-EXAONE 2.5 managed 60.4% and Solar Mini 4.5 Preview 52.5%. The run condition that supplies graphs and tables as images is the one doing the work right here.

Accuracy on questions that carry an image 2027 CSAT, by model (%) GPT-6.2 Astra 100% Gemini 4 Neon 100% GPT-6.2 Sol 100% K-EXAONE 2.5 60.4% Solar Mini 4.5 Preview 52.5%
▲ Pebblous original diagram. The number of questions behind the tally was not published. | Source: Aju Business Daily (2026-11-20)

Choi Byung-ho, a research professor at Korea University's AI institute, put the gap down to a wide difference between the global technology companies and Korean firms in parameter count and in the computing infrastructure available at inference time. That explanation is worth laying over the test conditions. Support for image input varies from model to model, and the graphs and tables went in as images, so part of the score gap may come from handling an input format rather than from reasoning ability. No material has been published that separates the two shares.

4

Neither Benchmark Asked Whether Past Exams Were in the Training Data

The first thing to settle when the CSAT is used to grade AI is whether the exam is new to the model. This year's questions went public in November, so no model could have memorized them in advance. But the CSAT is an exam with decades of past papers and worked solutions sitting on the open internet. The possibility that question types, setting patterns and the passages and concepts that recur have already entered pretraining data survives the fact that this year's questions are fresh.

As far as this article could check, no published material records a procedure for testing that possibility. The AI Times report does not mention a contamination check, and neither does the 27suneung-bench repository description. The run environment and the tool policy are written down carefully. The box for how new the exam paper is to the model is the one left empty.

Put beside academic work on the previous year's exam, the difference is plain. A paper on the 2026 CSAT mathematics section posted to arXiv by ten researchers made building an environment with zero data leakage the goal of the study design itself, and said so in its title. They digitized all 46 questions, 22 common and 24 elective, within two hours of the exam's public release and fed them to the models. The evaluation environment was explained before the evaluation result. As for the scores, GPT-5 Codex was the only model to reach a perfect 100, and only with text input and Korean prompts; the paper's reasoning-intensity ladder climbs from 82.6 points to that 100, and climbing it quadrupled token usage.

Because the conditions were split that finely, the paper carries a separate result as well. Its 24 models were run across three input modalities, text only, image only and text with figures, and across Korean and English prompts, and text input consistently outperformed image input. That puts both reasoning ability and the ability to read an input format inside the distance between 60.4% and 100% seen in section 3. Telling those two apart requires putting the same questions back to a model with only the input format changed, and neither of these two exam papers was built that way.

Contamination is a problem this blog has covered more than once. We have written up cases where public benchmark questions had leaked into training data, and looked at a test whose answers could be found from the task names alone. An exam like the CSAT, with abundant public material and a fixed format, sits on the high-risk side of that. To separate this year's perfect score into ability and familiarity, the score needs a note beside it saying how contamination was checked.

It is hard to lay this gap at the feet of an individual. What went into pretraining data is known only to the company that built the model, and most companies do not publish the list. It is not the kind of information one person can verify alone. The more accurate account is that the parties in a position to demand that verification have not yet looked at this exam paper.

5

Why Pebblous Is Watching This Case

Treat a benchmark purely as an evaluation tool and the episode files away as an unofficial metric made by an individual. Seen from the data side it reads differently. A benchmark is a dataset of questions, answers and marking rules, and it is data with the power to reorder models. Which means the questions we put to any other important dataset belong here too. Who built it, how was its scope decided, where did the labels come from, and how was it verified.

These two exam papers already diverge on the first two. Different builders and different scope, therefore a different winner. This is not a field where an accredited body sets the rules, so there is no seat that rules on which side is right. The credibility problem in Korean AI evaluation has repeated inside government programs as well. This blog covered the rank movement in the second dokpamo evaluation, where changing the method shook the order. Working back through the published block scores, first and third on the total are 1.6 points apart, and lifting just the 15-point professional-user block out of a 100-point scorecard puts a different team in front. Building an evaluation system is as hard as building a model.

For the side receiving the AI report card, the companies and institutions choosing and deploying models, three things are worth checking.

  • Who set the scope of the exam this score came from. These two papers show that when the scope moves, the ranking moves with it.
  • How questions the model did not answer were counted. An accuracy figure computed with blanks excluded and one computed with blanks as wrong circulate under the same name and are not the same number.
  • Whether anyone checked that the exam paper was new to the model. With no procedure written down, the score is a mixture of ability and familiarity.

All three are problems of evaluation data rather than problems of the model. If AI-Ready Data means data a model can use as it stands, then the exam paper that judges that state deserves the same standard of care. Scrutinize the provenance and quality of training data while leaving the data used for marking unexamined, and what we are trusting is not a model's ability but somebody's exam paper. The question this year's CSAT report card leaves behind is not which model came first, but whether we have ever checked the material that produced the ranking.

Editor's Note

CSAT-benchmark's scores and run conditions were taken from the AI Times report, the image-question accuracy figures and the expert comment from Aju Business Daily, and the unofficial-evaluation caveat from ZDNet Korea. Those three reports are in Korean; their statements appear here in indirect speech rather than in quotation marks, because an English quotation would be our own translation. An English-language account of the same result is available from The Asia Business Daily. The CSAT-benchmark dashboard draws its screen with JavaScript, so its body text could not be read directly, and anything written only inside the dashboard is therefore not reflected here. The scores and marking rules of 27suneung-bench were read directly from the repository, and since no name is published for the person who posted it, only the GitHub handle is given. Because the two benchmarks cover different subjects under different marking rules, their scores are not compared directly, and the diagram in section 1 only places the two sets of results side by side. Pretraining data contamination was not verified on either side, and this article records only that the gap exists. The paper design and the per-modality result in section 4 were checked against the arXiv abstract, and the two dokpamo figures in section 5 are values this blog previously computed from published block scores and published itself. The reading of evaluation data in section 5 is this article's judgment and appears in neither benchmark's published material.

R

References

Benchmark sources

Academic

News coverage