Executive Summary

Key Figures

Source: Kim et al., Multi-Step Tool-Calling over Korean Open Public APIs, arXiv:2609.05395 (2026-09-04, accepted to the EMNLP 2026 industry track)

The work starts by building a benchmark. Its name is KOPA-Bench. The researchers picked ten public API platforms across six domains, namely traffic, finance, education, law, politics and district administration, on three selection criteria: the domains had to be ones where public institutions genuinely open APIs, the platforms had to allow calls to be chained across them, and the Korea Open Government License had to permit derivative works. Parsing each platform's official documentation into a Model Context Protocol server yielded 2,318 tools, every one of them backed by a live endpoint. There are 145 tasks; a task takes five calls on average and up to fourteen, and 59% involve parallel execution.

The paper opens with two failure modes. One is skipping a prerequisite code lookup. The agent queries straight from a company name, but the API wants a corporate code, and one more lookup belongs in between. The other is ignoring pagination. The response arrives split across several pages and the agent answers from the first page alone. A finance-domain task in the appendix shows both at once. The question asks the agent to find the convertible bond Lightron issued in January 2026 and compute how far the previous day's closing price sits above the refixing floor price. Look up the corporate code, retrieve the convertible-bond issuance decision for the floor price of 532 won, look up the prior close of 1,680 won, and finish with the arithmetic for 215.78%. Drop any one of the four steps and no answer comes out.

Making training data for tasks like these requires knowing first which tool's output becomes which tool's input. Phase A of the pipeline the team named EDGE draws that map. Scoring all 2,318 tools pairwise is infeasible, so a dense retriever over signature embeddings narrows the candidates per tool: 15 neighbors from the same domain and 10 from other domains, where bindings are rarer. Each surviving candidate then gets a single call to Qwen3.5-122B, which returns a feasibility score and a parameter binding set, and anything below 0.3 is dropped. This is also where prior graph-based synthesis pipelines stop. As the paper's related-work appendix sets it out, Magnet, BUTTON and APIGen-MT link calls through signatures or model-proposed plans, while ToolACE adds a separate verification stage that is not aimed at live APIs. Two assumptions are common to all of them: links between calls are fixed without ever being checked against the live APIs, and each call is assumed to return a single result.

The appendix carries the prompt that told the scoring model what to judge on. It states explicitly that fields meaning the same thing should be bound even when their names differ. A mapping table shows that SIGUN_NM, REGION_NM and rgn all mean region name, and the prompt insists that a binding must not be rejected merely because the field names look different. That is an instruction to judge by meaning. Whether a semantically sound link also survives a call is a separate question.

Phase A's second half inverts the order. Rather than using the graph as drawn, the loop samples paths and executes them against the live APIs. It runs 100 iterations, sampling 500 paths in each. Thompson sampling decides which links to try, and successes and failures update each link's Beta posterior. Failures are split in two and treated differently. Structural failures, where the binding itself is wrong, are penalized heavily; environmental failures such as a transient server error are penalized lightly. Without that split, an incidental error would kill a sound link. A link whose posterior falls below the viability threshold is cut from the graph.

Starting from the same candidate set, the share of links that actually execute rose from 50.2% to 62.7%. All 12.5 points of that difference came from execution. Measured separately, the pruned links execute only 14.8% of the time. Had the pruning been random, the two sets would execute at similar rates, so the gap is itself the evidence that only genuinely broken links were removed. Among the pruned links, 70.5% never succeeded in any trial, against 27.7% of those retained.

Those figures are a cross-section taken after the loop finished, and the in-flight indicators point the same way. Over the 100 iterations, the pass rate of sampled paths rose by 31 points and the step-level call success rate by 28 points. The graph was not being reshuffled arbitrarily; it was concentrating on dependencies that execute.

The paper plots each link's prior feasibility score against its posterior mean after execution in a single scatter plot. Retained and pruned links do not separate along the axis of the model's prior score; they separate along the posterior axis. A link that scored highly still gets cut once the calls contradict it. The correction ran in both directions. Links that scored 0.6 or above frequently fell below the diagonal, and some that scored 0.3 or below rose above it. The loop cut 10,174 links in all, and not one of them was cut without being tried, since the rule requires at least two trials first. The model's judgment and the outcome of execution were different information.

A verified graph does not by itself make walking it into trajectories straightforward. A single call to a Korean public API may return zero records or several hundred thousand. In the corpus EDGE produced, 74.1% of calls returned at least two records, and 81.2% of the multi-step trajectories contain at least one junction that consumes a multi-record output rather than a single value. The median number of chained records is 27 and the maximum is 224,958. If the previous call returned 500 municipalities and the next tool takes a single municipality code, the existence of a link settles nothing about what to put in.

So the junctions are typed by count. With the fan-out budget set to five, a junction passing one value is Sequential, two to five values make it a Fan-out that issues that many parallel calls, and more than five makes it Derived. For Derived junctions an internal processing node goes in first, taking a maximum or minimum, a threshold condition or the most frequent value, to reduce the set to five or fewer before it moves on. Among the 613 single-junction sequential trajectories, Fan-out and Derived accounted for 398, or 64.9%. The majority of trajectories were ones a one-to-one template could not have produced at all.

Training the 4B model on purely sequential trajectories leaves pass@1 at 0.2327; adding the parallel and mixed ones raises it to 0.3080, and the full corpus reaches 0.3094. The spread widens on the share of tasks solved by at least one of four attempts, where the full corpus reaches 0.4690 and the parallel-plus-mixed subset 0.4000. Sequential trajectories teach the precise handoff of a value across a junction, while the parallel family teaches the comparisons and conditions that no single chain can express, and neither family substitutes for the other.

Set beside existing tool-use datasets, the difference is plain. All four datasets the paper compares against sit below 10% one-to-many chaining with a median cardinality of 1.

Source: arXiv:2609.05395, Table 11. One-to-many is the share of multi-step trajectories containing at least one junction that consumes a multi-record tool output; median and max describe the number of records consumed at such junctions.

Once the trajectories are typed and built, a large model writes the Korean question and the answer to attach to each one. From here on the quality rests on judgment rather than execution. A rule-based first stage removes non-reproducible instances, malformed structures and near-duplicates. The second stage discards questions that cannot be graded automatically, questions whose answer is already disclosed in the wording, and chains whose intermediate calls do not constrain the final answer, and it discards them only when two independent models agree. The third stage doubts the answer itself. Where prior pipelines treat the stored answer as a fixed reference, this one re-solves every task independently and replaces the stored answer when the new solution is judged better. Only open-source model outputs become training labels; proprietary models serve solely for verification. The two referees behind the second stage's agreement rule are Qwen3.5-397B-A17B and GPT-5.

The filtering has its own number. A 4B model trained on the unfiltered data reaches a pass@1 of 0.242, and on the filtered data 0.309, a difference of 6.7 points. Examples in the appendix show what got removed. Some Korean questions had picked up fragments of Chinese and Japanese. Others leaked the answer in advance, letting an intermediate call be skipped. The second kind is the more striking. The query-generation prompt tells the model never to expose the value being handed between calls, warning that revealing it lets the step be skipped, and it even requires a five-item self-check before output. The filter found leaks in the output of a model that had been given those instructions.

The resulting 1,781 tasks were used to train Qwen3.5 models at 4B and 9B with GRPO. The reward is binary, one or zero, on whether the final response and the environment state came out right. The values below are drawn from Table 1 of the paper and are averages over four rollouts per task.

Source: arXiv:2609.05395, Table 1. pass@1 is the average over four rollouts, pass@4 the share solved by at least one of the four.

The 9B model's pass@1 rose from 0.3275 to 0.4310. The 27B base model, with three times the parameters, sits at 0.4482, so the gap narrowed to 1.7 points. The paper describes this as approaching the 27B model at one-third of its parameters. The 4B model rose from 0.1758 to 0.3094, landing just below where the untuned 9B started.

The proprietary models are in the table because they show how hard the benchmark is. Claude Sonnet 4.6 scores 0.4655 and GPT-5.1 scores 0.3706. No model solves more than half of the 145 tasks on a single attempt. At pass@1 the EDGE-trained 9B is ahead of GPT-5.1. Allow four attempts, though, and the picture changes: Claude reaches 0.8207 against the trained 9B's 0.5517. The gap closed on the single attempt and widened once more chances were given.

Whether the gain came from the data or from the training method was also separated out. Supervised fine-tuning alone, on the same 1,781 tasks and the same base checkpoint, raises the 4B model's pass@1 from 0.1758 to 0.2724. That accounts for 9.7 of the 13.4 points of total improvement, which is why the authors write that most of the improvement is attributable to the corpus rather than to the objective. GRPO's own contribution shows up most clearly in the share of tasks solved by at least one of four attempts, which goes from 0.3586 to 0.4690, a rise of 11.0 points. Both methods consume the identical task set, so the difference does not come from the amount of data. It comes from how much signal each extracts per task. Supervised fine-tuning imitates one verified reference trajectory and is bounded above by its teacher, while GRPO samples several rollouts of the same problem and learns from their relative rewards.

Two tables should not be mixed when citing this gap. Appendix C re-measures the same experiment over eight independent random seeds and lands on more conservative values. There the trained 9B is at 0.4138 and the 27B base at 0.4681, a gap of 5.4 points. The authors state that both estimates support the same conclusion, and note that the 95% confidence intervals of the trained models and their base counterparts do not overlap, so the improvement itself is statistically significant. Table 1 holds the headline number and the appendix holds the robustness check.

Outside the training distribution, the direction held. On BFCL, the English-language function-calling benchmark, the 9B model rose on every axis: multi-turn from 52.25 to 58.12, single non-live calls from 82.40 to 87.65, and live single calls from 78.09 to 81.57. Even though the training data is predominantly single-turn, the largest gain landed in the multi-turn conversational setting.

Training and evaluation drew on the same pool of public APIs, which makes contamination a fair question. Zero of the 145 evaluation questions appear verbatim in the training data. The tool universes overlap by 62.2%, which the authors distinguish as the consequence of drawing on the same public API pool rather than as contamination. They then measured the platforms excluded from synthesis entirely: 10 tasks grounded in Seoul Open Data Plaza and 21 grounded in DART and KRX, 31 in total. The 4B model's pass@4 on that subset rose from 0.2903 to 0.5161, from 9 of the 31 tasks to 16, a gain of 22.6 points against the 15.9 points the same model gained over the full benchmark. A model that had memorized the call procedure of particular platforms would be unlikely to move in that direction.

At the level of links that distinction gets sharper. The tools overlap by 62.2%, yet not one of the 178 dependency-link occurrences the evaluation tasks require appears in the training data. The Jaccard similarity is 0.000. Even under the loose criterion that counts any two consecutive calls regardless of whether a dependency is present, 95.6% of occurrences are unseen in training. The benchmark and the corpus draw on the same pool of tools and connect them differently.

That is where the paper's account ends. Anyone working on the data side who wants to carry this result over should first draw the boundary precisely.

Execution replaced exactly one thing, the dependency link in the graph. Whether this link actually works against a live API is the single question the calls answered. Writing the question and the answer that attach to a trajectory, and screening out the ill-posed cases, still rest on model judgment. Execution did not take over the whole synthesis; it laid the floor the remaining judgments stand on. Without that floor, no amount of downstream filtering catches a trajectory built on a dead link, because such a trajectory looks perfectly well-formed. The judgment layer that remains is not decorative either. The 6.7 points from section 2 are what that layer is worth, and the leak examples in that same section show that writing "never leak the answer" into a prompt does not secure it.

The authors list three limitations of their own. Being grounded in live endpoints means schemas, availability and returned records change over time, so even after filtering out tasks tied to real-time data and comparing environment state by hash, exact reproduction depends on the stability of public services outside their control. Both the benchmark and the pipeline are built on Korean public-sector APIs, which leaves open how far the findings transfer to other languages, to commercial or private APIs, and to administrative systems in other countries. Nor did they run existing synthesis pipelines end to end under the same conditions. They approximated only the property those methods share, a schema-only skeleton graph, as the control, and stated that system-level comparison is outside the scope of the study.

Who produced this research? All five authors are at LG CNS, and the paper was accepted to the EMNLP 2026 industry track. This is applied industrial research from a company that wins contracts to build on-premise LLM agents. The work was supported by the Sovereign AI Foundation Model Project run by Korea's Ministry of Science and ICT, and the code and data are public. It passed peer review and carries an audit that amounts to third-party verification, so there is no reason to doubt the results, but this is not the place to read it as neutral academic work.

Pebblous reads this paper less for the public APIs than for one question. If the quality of synthetic data can be pushed from a downstream verdict into a precondition of generation, what is the cheap signal that plays that role in our own domain? The signal EDGE used was the binary outcome of a call, success or failure. Carried over to physical AI, the candidates would be running the physics engine once in a simulator, or executing a planned motion once on real hardware. This question is not one the paper asked. The authors left transfer beyond Korean public APIs to future work.

Thank you for reading this far. Every figure here was checked directly against the full text of arXiv:2609.05395, and wherever two tables give different values, the body says which table it is quoting. The code and data are published in the EDGE-KOPA repository. If you have located an execution check for synthetic data inside your own domain, we would be glad to hear which signal you used.

• 1.Kim, D., Cho, E., Kim, K., Noh, S., Lim, K. (2026). "Multi-Step Tool-Calling over Korean Open Public APIs: A Benchmark and a Data-Synthesis Recipe." arXiv:2609.05395v1, accepted to the EMNLP 2026 Industry Track. — LG CNS, the EDGE pipeline and the KOPA-Bench benchmark.
• 2.Kim, D. et al. (2026). "EDGE-KOPA code and data repository." GitHub.