Executive Summary

Editor's note. This report is about the provenance of model identity. It pairs with our report on the lineage gap in training data, the data sourcing lineage behind the MAI models, but sits one layer up. That piece asked where what went into a model came from. This one asks what the identity of the thing answering right now is. For the EU AI Act timeline and the reading of individual articles, we follow what actually switched on this August.

Four numbers measure the gap between the documentation and the artifact

The first two show where the volume of disclosure and the absence of verification came apart. The third shows what blocks an outsider who tries to measure that absence directly. The fourth shows how far a name and an artifact can drift from each other. The denominator splits into 37 and 35 depending on which items apply, and that difference flips the top rank.

Source: Abraham, S., & Bucknall, B. (2026), Silent Updates: Measuring and Closing the Post-Deployment Disclosure Gap, arXiv:2608.11803v1. Every figure is scoped to the paper's stated sample of 16 organizations (9 first-party providers and 7 hosts), and the authors label their results preliminary.

Silent Updates: Measuring and Closing the Post-Deployment Disclosure Gap, posted to arXiv on 12 August 2026, is a cs.CY paper by Sophia Abraham and Ben Bucknall, accepted to AIES-26, the conference on AI, ethics and society. The authors label their results preliminary themselves. Every figure in this report comes from that v1 text and its abstract.

The paper's starting fact is simple. A deployed foundation model is not a fixed system. A provider can change how it behaves without taking it down and without giving it a new name. The abstract names five paths. Looking at what each one touches shows why a version number fails to catch the change.

The names and order of the five paths follow the abstract of arXiv:2608.11803v1. The two right-hand columns summarize how each path shows up in practice.

What the five paths share is that none of them touches the model identifier. The string in the API call stays the same, and so does the name on the invoice. Whatever changes on the left, the name in the middle does not move.

1.1“Silently” in this paper means three things are absent at once

When the paper writes “silently” it is not reaching for a word that means sneaky. It is naming an operational condition: three absences overlapping. No public notice, no version increment, no re-evaluation. If any one of the three is present, an outside observer retains a thread to pull. Only when all three are missing does the change become, from the outside, an event that effectively never happened.

This definition matters because it narrows what is being criticized. A provider improving a model is not the problem. The problem is a deployment practice under which the fact of the improvement leaves no trace in any external record. In the paper's own framing, such changes undermine a core governance assumption: that there is an externally verifiable chain of custody connecting the model referenced in evaluation results and system cards to the model actually served to users.

1.2The measurement that the same name answers differently landed in 2023

This is not the first time the issue has been raised. In July 2023, Lingjiao Chen, Matei Zaharia and James Zou of Stanford and UC Berkeley measured a commercial API behaving differently under the same name a few months apart, in How is ChatGPT's behavior changing over time?

The most widely quoted number is accuracy on prime identification. GPT-4 scored 84% in March 2023 and 51% in June. The authors attributed part of the drop to a reduced tendency to follow chain-of-thought prompting. There is a common misreading here. On the same task, GPT-3.5 got better in June than it had been in March. The direction split by model. The paper's conclusion was not that models decay over time but that a service under the same name can change substantially in a short window. Formatting errors in code generation rose for both models, and GPT-4 grew less willing to answer sensitive questions and opinion survey items.

The sentence they put in their conclusion runs like this: the behavior of the same LLM service can change substantially in a relatively short time, which highlights the need for continuous monitoring of LLMs. The argument that measurement belongs in a standing instrument rather than a one-off benchmark was already on the table then.

Where that paper measured whether behavior changes, this one moves the axis. Instead of measuring behavioral change, it measures how providers disclose that change. The earlier work knocked on the door from outside. This one counts, from the inside of the documentation, whether the door was ever left open.

1.3Provider documentation already says the weights can change

The evidence that silent change is real rather than suspected sits inside provider documentation. Microsoft Foundry's model lifecycle page defines the preview stage this way: “Experimental. Weights, runtime, and API schema might change.” A host has written into official documentation that weights can be swapped under an unchanged name.

Reading that as dishonesty misses the point. The documentation is honest. The problem is that the customer who receives that honest notice does not also receive a way to find out when the change actually happened. The distance between a warning that says “this might change” and a record that says “this changed” is the gap the paper set out to measure.

1.4A system card that never says which snapshot was evaluated

One first-party example from the paper closes this section. The GPT-5 system card refers to its evaluation targets as gpt-5-thinking and gpt-5-main. The paper's observation is that the card does not specify which snapshot corresponds to the evaluated deployment. The evaluation was performed and the document was published. What a reader of that document cannot do is confirm that their own API call today meets the same thing that was evaluated.

This sits at a different layer from a problem we covered earlier. In the reproducibility of science done on closed models the issue was that a single provider's model is closed, so the research cannot be reproduced. This paper moves that issue off individual provider policy and onto a measurement of disclosure practice across 16 organizations. The question is not open versus closed. It is whether the documents that were opened connect to the artifact being served.

A paper that flags a problem and a paper that measures it carry different weight. This paper's contribution is not the claim that silent updates happen. It is turning the question of how much providers disclose those changes into something that can be scored. The authors call the instrument the Silent Updates Scorecard and released it as a public tool.

The scale groups 29 items into seven domains. Binary items score 0 or 1; numeric items are banded from 0 to 3 according to thresholds in the scoring guidance. The maximum applicable to a first-party provider is 37 points. Seeing how many items sit in each domain shows what the scale treats as important.

Source: arXiv:2608.11803v1. Domain labels follow the item-number prefixes used in the paper. The per-item scoring rubric lives in the paper's appendix and is not reproduced here.

The heaviest of the seven domains is safety evaluation traceability. It is not a domain that asks whether documentation exists. It asks whether that documentation is attached to the version being served. The distribution of items is itself an argument.

2.1Two providers have a denominator of 35

The denominator in the score table is not the same for every provider. The two items in domain X apply only to providers that run both an API and a chatbot. For a provider with no comparable chatbot surface they are marked not applicable, and the maximum drops accordingly. The caption on the paper's own table states that scores are normalized against the applicable maximum.

In the sample, Cohere and AI21 are those cases. Their denominator is 35 rather than 37. It looks like a minor footnote, and it reverses the ranking.

2.2What the transparency index measured and what this scale measures

The scale has a methodological ancestor: the Foundation Model Transparency Index (FMTI) from Stanford CRFM, which has scored developers against 100 indicators. Three editions produced the following.

Source: Stanford CRFM, Foundation Model Transparency Index. The December 2025 mean is 41 on the CRFM primary page; some secondary summaries report 40.

More important than why the numbers rose and fell is that the two scales ask different questions. FMTI scores whether something was disclosed, on the basis of transparency reports the developers submit. It does not independently verify the claims in those disclosures. Much of the jump in v1.1 came from information developers released during the process itself. Copyright status, data access, data labor and downstream impact stayed dark across editions.

The Silent Updates Scorecard adds a layer on top. It asks whether the disclosed documentation connects to the artifact being served now. Through the whole run from 37 up to 58 and back down to 41, that layer had never been measured.

2.3The wording of the two items that scored zero

What “0/9” means is set out in the exact wording of two items.

• M2: “Can the deployed API model be mapped to a published safety evaluation using publicly available information?”
• M6: “Is the content or usage policy explicitly linked to specific model versions?”

Neither item was satisfied by any of the nine providers in the sample. A zero on the first does not mean there are no safety evaluations. It means the evaluations exist and there is no way to confirm that they refer to the model answering now. A zero on the second means the promise that says “do not use this model for that purpose” is not attached to a specific version. When the model changes quietly, whether that promise still holds gets blurry along with it.

The core of the result is not the total but the asymmetry. On items that ask whether a document exists, most providers scored. On items that ask whether the document links to the artifact, almost nobody did. The same providers, inside the same scale, split this cleanly between the two kinds of item.

The top three items are areas providers already handle well. Eight of the nine run a dedicated changelog page, seven publish quantitative safety metrics, and six describe safety-relevant behavioral differences across versions. Going down the list, the numbers collapse. Exactly one provider named the evaluated model version precisely in its model card. That was Anthropic. M1 is a binary item, so there is no partial credit.

3.1The four failure modes the paper names

The paper sorts the shapes this asymmetry takes into four failure modes. Reading each alongside the item results makes the missing piece concrete.

Each mode pairs with an item result from the previous section. The first goes with M1 at 1/9: several providers let you pin a dated snapshot name, and one wrote the evaluated version precisely into its model card. The second has to be read against C1 at 8/9: nearly everyone maintains a changelog page, and the reliability with which that page records launches is not the reliability with which it records behavior changes. The fourth is M2 at 0/9 restated. Only the third mode carries no provider count. The two items on cross-surface consistency apply only to providers running both an API and a chatbot, and the per-provider counts for that domain are not part of what this report was able to confirm.

3.2The top rank changes depending on the measure

Per-provider totals follow. Read the table with the denominator column in view. As noted above, Cohere and AI21 have a denominator of 35 because the chatbot-surface items do not apply.

Source: arXiv:2608.11803v1. The paper's own table is sorted by raw score; the share of applicable items is computed using the normalization its caption describes. The mean of 16.4 is the simple average of raw scores (148 divided by 9), and 44.4% is the figure the paper states against each provider's applicable maximum.

By raw score OpenAI ranks first; by share of applicable items Cohere does. The two measures swap first and second place. Citing a rank without naming the measure produces a false sentence immediately. More important than either is that the top score sits in the low sixties on both measures. This table is less a report card separating good providers from bad ones than a map showing that even the best of them left a third of it blank.

3.3The name held still while the artifact changed ten times

The sharpest case in the paper is DeepSeek's deepseek-chat identifier. In the paper's framing, that identifier remained stable while the artifact served beneath it changed across ten documented releases over 19 months. The chain runs from the V2 family in May 2024 through V2.5, V3, V3.1 and into the V3.2 family.

Where that case came from matters. The paper did not uncover it covertly. It reassembled DeepSeek's own published release notes into a timeline. So the problem is not an absence of records. The records were there; they were simply not bound to the identifier callers use. Running a changelog and being able to tell from that changelog what your request meets today are two different things.

3.4Terms of service block external measurement

If providers will not open the link, measuring it yourself from outside looks like the obvious fallback. There is a threshold on that path too. Item T2 asks whether the terms of service permit benchmarking and the publication of the results. Only three of the nine did. The other six restrict it, either through an explicit benchmarking clause or through a broader clause about competing products.

The clause language the paper quotes shows the shape of the restriction. Cohere's terms contain the phrase “for any other benchmarking or competitive purposes,” and Together AI's contain “competitive analysis or benchmarking.” For OpenAI, Anthropic, Google, Mistral and xAI, the basis is not a clause naming benchmarking directly but a broader prohibition on developing competing products.

Providers do not publish the link between the documentation and the artifact, and the terms of service restrict attempts to measure that link from outside. The absence of verification is not incidental. It is closed on two sides.

In practice, few organizations take a model straight from the first-party provider. Going through a cloud platform or an inference host is common, and in that case the model's identity arrives after passing through two sets of hands. The paper evaluated seven third-party inference hosts on a separate four-item scale.

Source: arXiv:2608.11803v1. The host scale was designed separately from the 29 items used for first-party providers.

Six of the seven scored 3 out of 4, with Fireworks AI at 2. The paper notes gaps in lifecycle and deployment disclosure for Fireworks AI. Comparing these scores with the first-party 44.4% is a mistake. The item sets are different. The first-party scale mostly asks about the link between evaluation documents and deployment; the host scale asks about the transparency of the deployment implementation. A conclusion that hosts are more transparent cannot be drawn from this data.

4.1The same model name has a different lifespan on each host

That the host layer adds another coordinate is something provider documentation states directly. Anthropic's model deprecation page says that partner-operated platforms, meaning Amazon Bedrock and Google Cloud, set their own retirement schedules. A model's lifecycle status and dates can therefore differ by platform, the documentation adds.

In practice the implication is clear. An organization can agree internally that it uses a given model, and how long that model exists still depends on which host it goes through. The model name gets written into a procurement document as a single line, and that name alone does not fix the coordinates.

Identity information breaks somewhere along the path from first-party provider to user. Moving left to right, less and less is pinned down.

4.2Version pinning already exists as a product feature

There is no reason to read the host layer only as a negative. If anything, hosts are where the controls are documented most concretely. Microsoft Foundry's model lifecycle policy is the example. A generally available model gets a retirement date set programmatically 18 months out at launch, and that date can be queried through the Models API. At 12 months it enters deprecation and new customer access is cut off; after retirement at 18 months, every inference request receives a 410 Gone. The notice standard is at least 60 days for generally available models and 30 days for preview, delivered through email alongside service health notification channels.

More striking is that the option to turn auto-upgrade off is documented. Standard-series deployments are auto-upgraded on a rolling per-region basis, and a deployment's versionUpgradeOption can be set to upgrade when a new default appears, upgrade when the current version expires, or never auto-upgrade. Provisioned deployments are excluded from auto-upgrade altogether. In other words, the ability to hold a version in place is already in the product.

What auto-upgrade looks like in practice is shown by an example in the same document. When the 13 May 2024 version of gpt-4o retires on 1 October 2026, Standard SKU deployments are auto-upgraded to gpt-5.1. Leave the deployment settings alone and, as of some date, a different model answers. The setting can be changed. What rarely makes it into procurement documents is that the default, if nobody changes it, is replacement.

The asymmetry is elsewhere. Preview-tier models get no such option. The documentation states that there is no way to stay on a preview model that is retiring. Partner model lifecycles also differ: generally available models from Anthropic, DeepSeek, Fireworks and Mistral AI follow a 12-month cycle rather than the standard 18. Government clouds support one version at a time with a 30-day overlap when a new one lands. The authority to retire a model early for security or compliance reasons is spelled out as well.

4.3The documentation promises fixed behavior and gives no way to check it

The same document contains the passage where the paper's argument comes into sharpest focus. The definition of the generally available stage reads: “Production-ready. Weights and APIs are fixed. Runtime patches for security vulnerabilities don't affect outputs.”

This is a rare document, because a host is explicitly promising behavioral stability. And the means to verify that promise from outside is not supplied with it. The customer can trust the sentence and nothing more. The gap the paper set out to measure sits exactly here. The problem is not that providers are lying. It is that from outside there is no way to establish whether they are.

The same document holds one more small piece of evidence. The status shown as Deprecated in the documentation and the portal appears as Deprecating in the API, where Deprecated instead means retirement has already completed. Even lifecycle status, the simplest metadata there is, reads differently on different surfaces.

4.4Immutable snapshot pinning is not a hypothesis; it already runs

The paper highlights one counterexample directly: Replicate. In the paper's terms, it offers the strongest snapshot pinning observed in the sample. Each deployment is identified by a content hash, and past versions identified by hash remain queryable through the API. The API documentation shows that a specific version is addressed by adding a 64-character version identifier to the owner and model name, and that version can then be retrieved or run.

The sentence the authors add is this section's conclusion. The significance of the case is wider than Replicate itself. Immutable snapshot pinning is not hypothetical infrastructure; it already exists in production systems. Lineage is missing not because the technology is missing but because most providers have chosen not to expose it.

The paper does not stop at diagnosis. Its prescription is not a demand to disclose every change and re-evaluate every time, which would not survive contact with practice. Instead the authors propose a Three-Part Behavioral Trigger System, a design that varies the weight of the obligation according to the nature of the change.

Source: arXiv:2608.11803v1. The authors state explicitly that the numeric thresholds they give are illustrative, and that where to set those thresholds is a question their paper does not resolve.

The point of the design is how the weight is distributed. Requiring the same procedure for a one-line system prompt edit and for a model acquiring a capability it did not have collapses the system. The former gets a published log; the latter gets a full re-evaluation. The paper does not fix the numbers itself. The authors present numeric thresholds as illustration only. Reading the table as a regulatory draft attributes to the paper a claim it did not make.

There is a practical reason to split the weight too. Full re-evaluation is not free. Judging by commercial audit quotes, a single external red-teaming engagement runs in the thousands to tens of thousands of dollars, and a comprehensive engagement bundling several risk areas climbs into the six figures. A lightweight regression monitor running over API access, by contrast, starts in the hundreds. Quoted rates vary widely by vendor and scope, so these are safest read as orders of magnitude only. That difference in magnitude is why a design requiring full re-evaluation for every change does not hold in practice, and why attaching only a published log to lighter changes is realistic.

5.1Notice standards already differ from provider to provider

Where the trigger system aims becomes visible when current disclosure policies are laid side by side. For deprecation and retirement, providers already have codified standards. Those standards differ from each other, and every one of them carries an exception clause.

Confirmed 14 August 2026 in each provider's official policy documentation. The periods in the table are standards for deprecation and retirement notice, not for notice of behavioral change.

These are notice periods for when a model comes down. No provider has a standard for how many days it has to tell you when a model stays in place and starts behaving differently. Deprecation is codified; change is not. That is the gap the three-part trigger is aimed at.

More telling than the numbers is how they are announced in advance. Anthropic splits a model's lifecycle into four stages, active, legacy, deprecated and retired, and writes a provisional retirement date against models that are still active. The phrasing is that retirement will happen no earlier than that date. Publishing a floor rather than a fixed date lets customers plan migrations against the worst case. In the deprecation layer, coordinates are handed out in advance like this. The same provider publishes no such floor for a model's behavior changing.

Something should be granted in fairness too. Across the nine notices in Anthropic's deprecation history table, the interval from announcement to retirement works out between 60 and 189 days, with the minimum landing exactly on the published 60-day standard. Nothing in that history violates its own standard. The same document also acknowledges deprecation's side effects directly: it states that researchers lose model access mid-way through comparative studies, and it commits to preserving model weights long-term. A case where policy and enforcement match is visible inside the sample too.

5.2Regulation demands documents; it does not demand correspondence

What switched on in the EU AI Act on 2 August 2026 needs stating precisely. What took effect that day were the Article 50 transparency duties and the Article 101 enforcement powers concerning general-purpose AI models, which cover fines, technical evaluations and orders to withdraw a model from the market. The Commission had adopted implementation guidelines for Article 50 earlier, on 20 July. The ceiling on penalties is 3% of global annual turnover or 15 million euros, whichever is higher.

To clear up a common misunderstanding: the obligations on general-purpose AI models themselves have existed since 2 August 2025. Treating 2 August 2026 as the date those duties were created is inaccurate. Models placed on the market before then have until 2 August 2027 to comply. We laid out the detailed timeline and the reading of individual articles in what the EU AI Act's August deadline actually means.

Bringing that back to this report's question, the answer is this. Regulation expanded the duty to file documentation. And no provision requires anyone to verify that the filed document corresponds to the artifact actually being served. What gets placed in front of an audit is a document, and which artifact that document points to remains inside the scope of what a provider chooses to disclose.

5.3The limitations section doubles as the argument

Not hiding the paper's limitations is directly tied to this report's credibility. The scoring was performed by a single rater using public documentation and standard API access. The completed scorecard subsequently went through an independent evidence audit, but the authors state that a formal measure of inter-rater reliability is left to future work. They also label the results preliminary. The sample is 16 organizations, so stretching this into “the industry has no means of verification” says something the paper does not.

One other limitation the authors record is different in kind. Acknowledging that the scope of measurement is confined to what providers expose through public APIs and documentation, they add this: that constraint is not a flaw in the study but part of the problem the study is about. External governance mechanisms can only rely on information a provider has chosen to disclose. It is an uncommon passage, one where a statement of limits becomes the core argument itself.

Pebblous works on data in the state it is in before it enters a model. That is why this paper's findings do not read as somebody else's layer. A problem the data side has wrestled with for years is repeating itself, in the same shape, on the model side. According to the Stanford AI Index 2026, 88% of organizations use AI in at least one business function while fewer than 10% have fully scaled it in any single function. The sample and methodology differ from this paper's, so the numbers do not belong side by side, but as background for gauging how model identity turns into operational risk on terrain where adoption is broad and consolidation is shallow, they are worth holding in view.

6.1The data layer answered this with hashes and version pins

Any organization that has run a data catalog recognizes this paper's structure immediately. The fact that an entry is written in the catalog is not an answer to an audit. What the audit asks is not whether the list exists but whether you can prove the entry on it is the data now in use. The data layer answered that question with content hashes, version pins and lineage records. It is not perfect, and the practice of leaving the correspondence in a form outsiders can check has taken hold. The model layer has no such answer yet. What Replicate's 64-character version identifier shows is that the answer is not technically hard.

6.2The idea of instrumenting drift carries from data to models

DataClinic instruments dataset drift not because it expects data to go bad but to settle whether it has on the record rather than on assertion. Silent change at the model layer can only be handled the same way. A provider saying nothing changed and our own regression battery returning the same fingerprint are different classes of evidence.

In Physical AI the problem gets sharper. If the vision language model a robot's behavior policy calls changes quietly, the entire basis of the safety validation history accumulated for that robot wobbles. This is where recording the coordinates of the model that touched the data stops being a lab-notebook matter and becomes a safety-documentation matter. We covered the same gap on the training data side in the data sourcing lineage behind the MAI models, and the problem of records agents leave behind being deleted in agent memory, provenance and deletion.

6.3A three-layer checklist you can run today

There is work to do besides waiting for providers to open the link. Every item below is derived from a fact confirmed in primary documentation earlier in this report, sorted into three layers: procurement, contract, and operations.

The second item in the contract layer is the one most often skipped in practice. The number of days lands in the procurement document; the exception clause that voids that number usually goes unread. All three provider policies above contain a shortening clause for safety or compliance reasons. That is not to say the clause is unreasonable. It is to say that a migration plan built on 60 days may in fact have to execute in far less, and the plan should be built knowing that.

6.4Only organizations that keep the instrument on their side have an answer

The practical conclusion this paper leaves an organization with is defensive. For as long as providers do not open the correspondence between document and artifact, the only organizations able to treat that correspondence as a record rather than a guess are the ones that keep the instrumentation on their own side. Recording response fingerprints, retaining model identifiers bound to outputs, running a regression battery continuously: all of it works without provider cooperation. It is the same work data quality diagnostics has done for years, with the target moved from a dataset to a serving endpoint.

Editor's note. Pebblous builds tools that instrument data quality and lineage. The three-layer checklist in this report is that thinking applied to the model serving layer, and it is not a claim that our product covers that layer today. Related discussion continues in the data sourcing lineage report and the EU AI Act report.

Every paper-related figure in this report was confirmed in the primary source, arXiv:2608.11803v1. Statements about provider policy come from reading each provider's official documentation directly on 14 August 2026. Below are that primary source, the academic literature that provides prior measurement and methodological ancestry, the regulatory provisions, and adjacent Pebblous reports on the same subject.

Primary source

• 1.Abraham, S., & Bucknall, B. (2026). Silent Updates: Measuring and Closing the Post-Deployment Disclosure Gap. arXiv:2608.11803v1 [cs.CY], 2026-08-12. Accepted to AIES-26.

Prior measurement and methodology (academic)

• 2.Chen, L., Zaharia, M., & Zou, J. (2023). How is ChatGPT's behavior changing over time? arXiv:2307.09009 (v1 2023-07-18, v3 2023-10-31).
• 3.Stanford CRFM. Foundation Model Transparency Index (2023-10 v1.0 / 2024-05 v1.1 / 2025-12).
• 4.Stanford HAI. (2026). AI Index Report 2026. Contextual figures in section 6 (adoption by business function and full-scale deployment rate).

Policy and regulation

• 5.EU Artificial Intelligence Act (Regulation (EU) 2024/1689). Article 50 (transparency obligations), Article 99 (penalties), Article 101 (fines for providers of general-purpose AI models), Article 111(3) (grace period for models already placed on the market).
• 6.European Commission. (2026-07-20). Guidelines on the implementation of the Article 50 transparency obligations.

Provider documentation

• 7.OpenAI. Deprecations. Confirmed 2026-08-14.
• 8.Anthropic. Model deprecations. Confirmed 2026-08-14. The 60 to 189 day notice intervals are derived from the deprecation history table in this document.
• 9.Microsoft. Foundry Models lifecycle and support policy. Confirmed 2026-08-14.
• 10.Replicate. HTTP API reference. 64-character version identifiers and per-version retrieval endpoints. Confirmed 2026-08-14.

Adjacent Pebblous reports

• 11.Pebblous. What the EU AI Act's August deadline actually means. The basis for this report's regulatory statements.
• 12.Pebblous. The data sourcing lineage behind the MAI models. The same gap on the training data side.
• 13.Pebblous. The reproducibility of science done on closed models. The reproducibility problem seen from a single provider's perspective.
• 14.Pebblous. Agent memory, provenance and deletion. What happens when execution records disappear.