Executive Summary

Key Numbers

The review left four figures behind: the size of the wrong answer, the money that went into it, the route by which an error can travel back to the service, and the number of times someone asked the chatbot for an answer.

Source: Kyeongin Ilbo and NewDaily Economy, carrying material released by Representative Park Jung's office.

Representative Park Jung's office put labor law questions to the Ministry of Employment and Labor's AI chatbot and matched the answers that came back against current law, one by one, as of October 23. The same questions went to a general-purpose AI. The two sets of answers were released side by side on October 27.

The passage every outlet led with is the question about an employer on the public list of wage defaulters who withholds wages again. Asked whether criminal punishment is possible in that case, the chatbot answered that it is not if the victim does not want the employer punished. The general-purpose AI, given the same question, answered that a prosecution can be brought even where the worker states no wish to see the employer punished, and that is the answer current law supports.

Withholding wages was for a long time an offense that could not be prosecuted against the victim's wishes. If the worker harmed did not want the employer punished, no charge could be brought. An exception to that principle was added, and an employer on the public list who withholds wages again falls under it. The answer the chatbot gave is the principle as it stood before the exception existed.

A general-purpose AI getting it right while a dedicated service costing 3.1 billion won gets it wrong is striking on its own. Read that one case alone, though, and it is easy to land on the conclusion that the general-purpose AI knows labor law better. Put all four cases together and a different explanation comes out.

The office counted four answers as departing from current law: the penalty provision for withholding wages, repeat non-payment by an employer on the public list, the payment standard for the insolvency substitute payment, and the standard for a habitual wage defaulter. All four fall under statutes the Ministry of Employment and Labor administers, and all four came out of a service the administering ministry runs itself.

The current standard is left blank in the bottom two rows because the material the office released and the coverage of it set the chatbot's answer against current law for the first two cases only. For the other two, what is confirmed stops at the fact that the guidance differed from current law.

Of the four, the insolvency substitute payment is the one with dates at both ends. The amended Wage Claim Guarantee Act took effect on August 20, widening coverage from the final three months of wages to the final six and raising the ceiling from 21 million won to 31.5 million won. On October 23, the date of the review, the chatbot's answer was the final three months, up to 21 million won.

Two months and three days. In that time the bill was promulgated, the effective date passed and the change ran in the press, and the one place it failed to reach was the counseling chatbot run by the ministry that administers the act. For the other three cases there is no telling when the answers started to go wrong, but this one has the length of the wrong stretch stamped in dates. In time it is half the months a worker can claim; in money it is 10.5 million won missing from the ceiling.

Three years of spending on this service are on the record: 930 million won in 2024, 200 million won in 2025 and 1.967 billion won in 2026, for a total of 3.097 billion won. The 2024 share is a government contribution from the National Information Society Agency, and this year's budget alone comes to nearly two thirds of the whole. The oldest answers came out in the year the most money went in.

Use has grown too. The 117,000 queries drawn across all of 2025 became 185,903 through September 2026. Nine months passed the whole of last year, which works out at a little over 20,000 a month.

The ministry keeps no staff of its own to maintain the service. In place of such staff, 173 certified labor attorneys from the Korea Association of Certified Public Labor Attorneys read through questions and AI answers at a rate of about 20,000 a month. That is close to the volume of use itself. They are not working from a sample; they are reading very nearly everything that comes in.

The missing piece sits elsewhere. A user who receives a wrong answer has no way to say so on the spot. Errors surface along two routes only: the attorneys' review after the fact, and an objection from outside such as this one. Twenty thousand a month is close to 700 a day. The person who meets a wrong answer first in those questions is the user, and the signal that person holds has no way into the service.

Accuracy was what the service led with at launch. Its description said hallucination had been minimized by having 173 labor attorneys clean the training data, and the attorneys' association called the result a successful model of public-private cooperation. A cost-benefit study by the Ministry of Employment and Labor put the time saved in finding labor law information at 87.5% against an ordinary search portal.

The range it covers has widened since. When it first appeared at a government expo in November 2024, it ran as an exchange over KakaoTalk and centered on subjects such as unemployment benefits and severance pay. It is now a full service at ai.moel.go.kr fielding questions across labor law, and the insolvency substitute payment and the sanctions on wage defaulters that it answered wrongly this time are inside that range.

Representative Park Jung said that a separate labor law counseling service run on this much public money should at the least show high reliability on the currency and accuracy of the statutes the ministry administers. The call that came with it was for a full review of the quality control system for verifying accuracy on a continuing basis, before the service's features and counseling range are widened any further.

What the review looks at explains how these four survived 20,000 readings a month. A reviewer is handed a question and an answer as a pair, and the insolvency substitute payment answer has nothing odd about it taken as a pair. No provision was invented and no figures were mixed. The chatbot copied the standard written in the material it reads exactly, and that material held a standard repealed in August. The fact that the material is out of date does not show in the answer.

Cleaning the data with 173 labor attorneys was in the design. Running that cleaning again every time the law changes appears not to have been. The old payment standard that stayed up for more than two months points at that empty place.

Explaining the general-purpose AI's correct answer by model performance does not hold together. A general-purpose model is unlikely to have learned Korean labor law more deeply. The difference between the two is the date of the material each had in hand when it answered. A general-purpose chatbot with web search attached can pull in a document carrying this year's standard immediately, while a dedicated chatbot that looks only inside its own corpus goes on citing the earlier provision, accurately, until that corpus is refreshed.

Dedicated tools losing this way is not a Korean story alone. A study of the leading American legal research AI tools (Magesh et al., Journal of Empirical Legal Studies, 2025) reported hallucination rates of 17 to 33 percent even in tools wired to dedicated legal databases. Attaching retrieval brings invented answers down, but it does not make them go away.

Where the remainder comes from is the point of "RAGged Edges" (arXiv:2403.01193), a study of retrieval-augmented chatbots. When the material being searched is itself out of date, the retrieval succeeds and the answer is wrong. The model found its source document properly, and the document held a standard that had been repealed. The chatbot's insolvency substitute payment answer is exactly this shape. A legal research report from Vals AI, which scored legal AI tools on practitioner tasks, also recorded items where a general-purpose chatbot with web search produced results comparable to the dedicated tools.

Nothing sits between the point where the law changed and the material the chatbot reads, and the review is attached to the answers rather than to the material.

So what this place needs is not a bigger model. It needs a path along which the fact of an amendment travels from the official gazette to the material the chatbot reads, and a device that notices when that path has been cut. Both sit outside the model.

Labor law counseling is not what Pebblous does. The structure of this case, though, takes the same shape in any organization that runs an AI over its own documents for answers. What changed is the rule, and what stayed as it was is the copy of the rule held on file.

Companies have their own things that changed yesterday. Work rules, security policies, refund terms, price lists and product specifications are amended a little every month. Stand a chatbot over those documents and a gap opens between the day a document changes and the day the chatbot knows it. The default size of that gap is not zero. Unless somebody is assigned to close it, it keeps widening.

Five things are worth checking if you run an AI inside your own organization.

• By what path does an amended rule reach the material the model reads. If it runs on somebody remembering to upload it, the gap widens by however long that person is away
• Does the answer carry the date of the document it rests on. An effective date and a version number let the reader catch the problem first
• Do amended and repealed documents actually leave the material. Add the new version only and the old one is retrieved alongside it, leaving the model to pick between the two
• Can the person who received a wrong answer say so on the spot. With no channel for reporting, a wrong answer stays until somebody outside points at it
• When a report comes in, is there a named person who goes back to the source document rather than the answer. Fix the one answer and the same question returns the same error

None of the five has anything to do with model performance. They are the items that would read the same after swapping in a better model. How much of the 3.097 billion won spent on the ministry's chatbot went to these five cannot be known from what has been released. What the four wrong answers point at, though, is this side rather than the model.

Thank you for reading this far. The comparison between the chatbot's answers and current law was checked against the material the office released and the three outlets that reported it, and we worked at not writing down comparisons the material does not make. If you run an AI over your own internal documents, it is worth measuring how many days pass on average between a rule changing and the AI knowing it. We would be glad to hear what that number turns out to be.

Pebblous Data Communication Team
October 29, 2026

News Coverage & Official Releases

• 1.Kyeongin Ilbo. (2026). "31억 들인 노동부챗봇 '헛다리 상담'… 박정, 전면점검 촉구" [31 billion won labor ministry chatbot's "misguided counseling" — Rep. Park Jung calls for full review].
• 2.Atpaju News. (2026). "31억 투입 고용노동부 'AI 노동법 챗봇', 현행 법령 오안내 다수 확인" [Multiple instances of outdated legal guidance confirmed in Labor Ministry's AI chatbot, built with 3.1 billion won].
• 3.NewDaily Economy. (2026). "31억 쏟아부은 노동부 노동법 챗봇… 현행 법령도 몰라 '헛다리'" [Labor Ministry's labor law chatbot, built with 3.1 billion won, doesn't even know current law].
• 4.Ministry of Employment and Labor. (2024). "어려운 노동법, AI가 24시간 상담해준다 — 'AI 노동법 상담' 서비스 공개" [Difficult labor law, explained by AI around the clock — "AI Labor Law Counseling" service unveiled]. Korea.kr Policy Briefing.
• 5.Maeil Shinmun. (2026). "임금채권보장법 개정 시행 — 도산대지급금 지급 범위 확대" [Amended Wage Claim Guarantee Act takes effect — insolvency substitute payment coverage expanded].

Academic Papers

• 6.Magesh, V., Surani, F., Dahl, M., Suzgun, M., Manning, C. D., & Ho, D. E. (2025). "Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools." Journal of Empirical Legal Studies, 22(2), 216–242.
• 7.Feldman, P., Foulds, J. R., & Pan, S. (2024). "RAGged Edges: The Double-Edged Sword of Retrieval-Augmented Chatbots." arXiv:2403.01193.