Executive Summary

Key Numbers

Four numbers carry the paper. The first two are what the method holds on to and what it costs, and the last two are the size of what stays on the phone and the result of copying the weights straight across.

Source: arXiv:2611.00650 (2026-10-31)

A language model running on a phone usually gets to know its owner through an adapter. The base model itself stays untouched, and a small bundle of weights sits on top of it, trained on personal data alone. Which tags you put on which films, how many points you give which products, all of that collects inside the bundle. Nothing leaves the device and the model still fits you, and that is the appeal of the arrangement.

The trouble is that the floor underneath gets replaced on a schedule. When a manufacturer moves the base model to its next generation, layer counts change, the vocabulary grows and token numbers shift. The Qwen upgrade the paper describes goes that way. The layer count drops from 28 to 24, and the vocabulary grows from about 150,000 entries to nearly 250,000. Three out of every four attention layers also become linear-attention layers, so the query, key and value projections the adapter used to ride on are no longer in place. An adapter is raised against particular layers and particular token numbers in one model, and against a change like that it becomes a block of numbers with nowhere to go.

Less upheaval produces the same outcome. The step from Phi-3.5-mini to Phi-4-mini keeps the layer count and the width, and all that changes is a sixfold larger vocabulary and a reorganization of the attention heads. The old adapter still cannot be used. A layer count that survives is no guarantee of compatibility.

Two paths were left. One is to swap in the generic adapter a manufacturer ships with the new model. In movie tagging that swap costs 15.4 points of accuracy on the 2B upgrade and 12.1 points on the 4B upgrade. Months of personalization disappear in a single update by that much. The other is to keep labeled user records and retrain from scratch on the new model. Accuracy survives, but a design that promised to handle personal data only on the device turns into a design that holds that data indefinitely.

Heirloom reaches not for the labels but for the old model itself, which has not been deleted yet. At the moment the base model is replaced on the device, the old model and the personal adapter above it are both still present for a short while. Standing the old model up as a teacher during that window and collecting its answers lets those answers do the work the labels used to do.

The prompts put to the teacher come from three places: unlabeled inputs still held on the device, prompts the manufacturer ships alongside the update, and prompts the new base model generates for itself. None of them carry a correct answer. The slot where the answer belongs is filled by the old personalized model's response.

The three sources are not worth the same. Asking with inputs stored on the device works best, and the paper has the manufacturer's prompts and the new model's own prompts generally trailing that by 1.5 to 3.1 points across the three upgrades. A device that keeps no record at all is cleaner, and it also leaves fewer good questions to ask the old model. Once the labels are gone, what the device still holds decides what the old model can be asked.

The form of the record depends on the task. For tasks with a fixed set of choices, such as movie tagging or product ratings, the system collects the logits of the first token of each label and keeps a probability distribution normalized across the labels alone. The student adapter trains to shrink the KL divergence between that distribution and its own. For open-ended tasks such as headline generation, the teacher produces a sentence of up to 32 tokens and the student follows that sentence with cross-entropy inside its own tokenizer.

The split into two phases comes from the phone's memory. There is no room to hold two base models at once, so all the teacher's answers are collected first, the old base model is unloaded, and only then is the new one brought up. The stored material is light. The targets for 128 movie-tagging prompts come to about 3.8KB. The diagram below follows the two phases in order.

When to stop training also has to be settled without labels. The paper compares the 2, 4, 8 and 16 epoch marks on a held-out set of prompts and takes the point where KL divergence is lowest, and it reports that fixing the count at 8 works about as well when tagging is moved using inputs stored on the device.

The evaluation covers three generation changes across two model families: Qwen3-1.7B to Qwen3.5-2B, Qwen3-4B to Qwen3.5-4B, and Phi-3.5-mini to Phi-4-mini. One refresh that updates the contents without changing the architecture went in alongside them, together with three personalization tasks. The tasks come from the LaMP benchmark: movie tagging, product rating and headline generation.

Set side by side, the movie-tagging results for the 2B upgrade show where Heirloom stands. Retraining with the labels still in hand remains the highest score, and Heirloom, which uses no labels at all, passes the level the phone had reached before the upgrade.

Movie tagging, Qwen3-1.7B to Qwen3.5-2B upgrade. Source: arXiv:2611.00650, Table 2

Across the three upgrades the distance between label retraining and Heirloom never exceeded 1.7 points. The gap shifts a little with the task. On product ratings, capped at 128 inputs, Heirloom scored 72.9% and label retraining 73.8%. Headline generation, where the answer is open-ended, shows the difference more plainly, with label retraining coming out 0.80 and 0.99 points above the 512-input Heirloom model on the two upgrades.

Similar scores do not mean the same answers. The paper counts the negative flip rate separately, the share of items the old personalized model got right that come back wrong after the upgrade. On the 2B upgrade that rate was 14% for Heirloom. Retraining on labels scored higher on accuracy and still flipped between 19% and 21%, and the new model with no adapter flipped 75%. The adapter that learned from the old model's answers held on not only to the total score but to the responses a user actually meets.

The cost was measured on real hardware. Running the 4B transfer at a fixed length on the neural processing unit of a OnePlus 15 took 38 minutes and 20% of the battery. Peak memory use was 3.6GB. The authors put this at the level of a job that can finish during an overnight charge. From the user's side, the picture is a phone handing what it learned to a new model while its owner sleeps, then waking up as though nothing had happened.

The scale of the experiments is modest. Movie tagging covers 19 users with 92 to 251 films each. Product ratings cover 100 users, and headline generation the 10 users with the longest histories. The results also come from two phones and two model families.

None of this is tidy. The old model has to be loaded as a teacher and its answers written down before the new adapter learns anything, so whether it would be simpler to rearrange the numbers in the old adapter to fit the new model's shape is a natural question. The authors walked that path and wrote down what they found.

LoRASuite is the prior work that proposed such a mapping. Because the published code does not support this pair of models, the authors implemented the paper's formulas themselves and applied them to the Phi upgrade. The remapped adapter scored 8.9%. In the same position, the new base model with no adapter at all scores 26.3%. Moving the numbers across did not raise performance; it broke the model.

A conservative version that copies only the parts whose shapes match was no help either. Of the 2,555 predictions the new base model made without an adapter, 107 changed, and accuracy rose by 0.3 points. That sits 16.8 points below the manufacturer's generic adapter.

Even when the slots survive intact the story runs the same way. The paper measured a case that keeps the architecture and the vocabulary and changes only the weights through further training, a refresh from Qwen3-4B to an instruction-tuned release in the same family. The layers and the token numbers are unchanged, so the old adapter does attach in form, and copying it dropped accuracy from 58.9% to 51.6%. Heirloom in the same position came out at 58.1%.

From here on, none of this is in the paper. What Pebblous looks at in this design is not the 38 minutes or the 1.7 points but the fact that the final storage place for personalization data has moved from a file to a set of weights.

Data management has long assumed that an original exists somewhere. Quality judgments, deletion requests and transfers to another system all run on that assumption. On-device personalization breaks it. Protecting privacy means erasing the original, and once the original is erased what remains is behavior dissolved into model weights. Heirloom goes back to the old model because that is the only route left, not because the idea is clever.

So familiar questions come back wearing new faces. Whose property is the behavior absorbed into a model? When you change phones or change manufacturers, can you carry it out with you? If you can, what do you receive, a weight file or 3.8KB of the teacher's answers? If you ask for deletion, what gets deleted? The original records are already gone, and inside the adapter your taste is mixed with the model's general knowledge.

An organization designing on-device personalization would do well to write down three things first.

• Is there a definition of what a user gets back on the way out? In a structure with no original, the portable asset is the weights or a signal that can reproduce them
• Are users told what happens to their personalization when the model is replaced? A design that resets quietly and a design that carries the learning over during the night are not the same product
• When a deletion request arrives, has anyone checked what can be erased from the adapter and how far it goes? A gap between what is technically possible and what the terms promise is where the risk sits

How behavior absorbed into a model should be handled is a thread this blog has followed before. One example is our piece on Trajectory, which asked who owns the record when a working specialist's judgment turns into a training signal. That record sat inside a company, and this one sits in the device in a pocket. The question is the same.

Thank you for reading this far. The 1.7 points, the 38 minutes and the 20% battery figure come from the paper's abstract, and the Table 2 accuracies along with 8.9%, 26.3% and 3.8KB come from the body and the tables. The authors are academics working with NEC Labs, so none of this means the design is headed for a particular phone. Section 5 carries those facts over to the work of people who handle data, and that reading is ours. If you are designing personalization on a device and weighing what to give a user back, we would like to hear your list.

Pebblous Data Communication Team
November 4, 2026

• 1.Chen, Z., Ni, J., Dong, Q., Zheng, X., Lucchesi, G. H., Sha, M., Cheng, W., & Luo, D. (2026). "Heirloom: Carrying a Phone's Personalization Across Base-Model Generations." arXiv:2611.00650, 2026-10-31.
• 2.Li, Y., Meng, F., Zhang, M., Zhu, S., Wang, S., & Xu, M. (2025). "LoRASuite: Efficient LoRA Adaptation Across Large Language Model Upgrades." arXiv:2505.13515. Cited in the body as the weight-mapping comparison. The authors state that the published code does not support this pair of models, so they implemented the paper's formulas themselves.