Would other models agree?
A new alignment seed gives another trial with the same embeddings. To speak of other model families, we must choose them before seeing the results, then report every pairing and failure.
Not yet testedAN INQUIRY INTO UNPAIRED ROSETTA 10 OCT 2026
Do the shadows agree,
or the things
themselves?
Let us ask of Unpaired Rosetta: how much of the agreement belongs to what is seen, and how much to the models through which we see it?
THE WALL AND ITS SHADOWSFIG. 01
This scene is an allegory made for the inquiry. The measured evidence is given below.
Five photographs · trial completeAll 120 assignments examined · earlier EPIC run pausedOne correct match out of five
01 / THE QUESTION
What follows from agreement?
When two shadows agree,
what have we learned?
The chosen encoders may share training data, objectives, or useful biases. If we repeat the solver but keep the encoders, we have not yet asked whether other models would agree.
Have they seen the same things?
Two datasets may bear different names and still contain the same images. The metadata shows that some COCO source images are described in Stanford Paragraph Captioning (SPC).
Metadata examinedThe thing, or merely its kind?
To find a horse is not yet to distinguish this horse, its number, its place, or what it does. Retrieval must also be tested among things of the same kind, where broad categories no longer suffice.
Further trials proposedFIVE NEW PHOTOGRAPHS · COMPLETED
Five things. Five names.
Can their distances
recover their names?
Slide, Trashcan, Tree, Basketball and Water fountain: the five human labels, each supplied with a new photograph. We asked whether the arrangement of the image features resembled the arrangement of the word features closely enough to recover their correspondence.
THE QUESTION WE ACTUALLY TESTED
Which of the 120 assignments makes the distances agree?
Original DINOv1 describes the photographs; original GloVe describes the exact human labels. Compare the ten distances between image pairs with the ten distances between word pairs. Examine every one-to-one assignment and choose the smallest mean squared difference. The matcher receives no correct image–label pairs.
We chose this test after seeing the photographs, then published its code and rules before computing their features or scores. All five images and the complete set of candidate labels take part in selection. This is a small exploratory test, with no unseen evaluation set.
The assignment it chose.
One correct match out of five.
| Photograph’s human label | Chosen label | Match |
|---|---|---|
| Slide | Basketball | No |
| Trashcan | Tree | No |
| Tree | Trashcan | No |
| Basketball | Slide | No |
| Water fountain | Water fountain | Yes |
The human assignment ranked 94th of 120 by the fixed distance rule. The selected assignment had cost 0.037819; the human assignment, 0.073473. The optimum was unique at the prescribed tolerance of 10−12, with a gap of 0.000281 to the next cost. Cost is the mean squared difference between ten pairs of distances.
For the selected assignment, those distance pairs had Pearson correlation 0.878. Four names still went to the wrong photographs. One correct match is also the average under a random assignment; this single result does not establish a general level of accuracy.
The scene came later.
The collector attests that all five photographs were taken after the model files were fixed. If that account is true, these particular captures could not have trained those files. The attestation is not independent proof of capture time; familiar objects, words and places remain possible training material.
Capture time attested by collectorLet the result keep its size.
Five photographs from one outing are five observations. The 120 assignments are alternatives, not 120 independent trials. A random assignment gets one of five right on average. We report no significance test or general accuracy estimate.
One chosen model pairDo not repair the answer afterward.
Keep the supplied words, fixed center crop and distance rule. Shared backgrounds, ambiguous words and the crop may affect the result. This calculation trains no cross-modal map and is not a run of the Rosetta algorithm.
Model choice remains untestedRead the result All 120 assignments Repeat the distance comparison Read the run record Independent numerical check Protocol, code and checks
Run on 10 October 2026. The distance objective failed to recover four of the five human matches in this set. This does not refute Rosetta or settle what another model pair would do. Code and protocol were published before inference. The two earlier proposed collections below remain separate records.
THE EARLIER 48-PHOTOGRAPH PLAN · NOT RUN
Keep the proposal distinct from the trial.
The larger collection.
A plan kept apart.
This earlier proposal asked whether two frozen models support correspondence between new photographs and descriptions when supplied with correct fitting pairs. Its 48 photographs have not been collected. The five-photo trial above follows different rules and does not complete this plan.
THE REASON FOR THAT PROPOSAL
The scene comes after the model.
Publish the exact model hashes, collection rules and analysis code before taking the photographs. If that order is honestly followed, these particular recorded events cannot have trained those already fixed files. We need not rely only on the model publisher’s account of its old training corpus.
This is a narrower defence, not a claim of complete purity. The models have seen mugs and books; ordinary phrases may recur. Markers and file hashes support the collection record but cannot independently prove capture time. The choice of models remains an open question.
- Before
Fix what will see and read.
Original DINOv1 ViT-B/16 and GloVe 6B/300d. Publish their byte hashes, image crop, text rule, sample and six fits. Recheck the files before inference. Read the fixed protocol.
- 48 photographs
Bring new scenes before them.
Eight kinds of everyday object, photographed in six rounds. Four fitting rounds use one set of physical objects; two development rounds use a different set. A person writes the captions after all photographs exist.
- After
Ask whether a known pair can meet.
Run separate image and word noun probes, a paired ridge map and three maps fitted with shuffled pairs. Keep repeated descriptions, exact ties and failures in the account. Balanced random choice is 12.5%.
A small trial, with a small claim.
Mug, bowl, plate, bottle, spoon, fork, book and shoe. Gather two physical examples of each: sixteen objects. The sixteen development photographs show eight held-out objects twice. They are not sixteen independent instances.
32 fit · 16 developmentFirst test whether the features suffice.
The correct pairs would be supplied under this proposal. If even that mapping failed, an unpaired failure would be hard to interpret. A useful paired result would give reason to design another trial; it would not establish unpaired alignment.
Paired competence fits proposedLeave room for the answer.
No tuning, added photographs or changed categories after seeing scores. No final test has been collected. Two development rounds and three shuffle references cannot bear a claim of statistical significance.
This proposed study has not runEarlier prospective protocol · 10 October 2026. Preserved collection record · The 48-photograph collection has not begun. Software checks used artificial fixtures. This plan and the paused EPIC plan below remain separate from the five-photo trial.
THE EARLIER EPIC PLAN · PAUSED
Let the claim be no larger than the trial.
The earlier plan.
Its record remains.
Can one fixed image model and one fixed word model recover object-category correspondence on later kitchen recordings, when their alignment receives no paired episodes? That was the proposed EPIC test. Its protocol and limits remain available; the five-photo trial does not resume or complete it.
WHY BEGIN HERE?
A narrower claim is easier to defend.
The training corpora predate these recorded episodes. Under the published training histories, memorizing these particular episodes cannot explain a result. This makes the case stronger against direct episode exposure. Familiar objects, ordinary phrases and the choices of the model builders remain part of the account.
The question was narrow; the proposed run was substantial. One model pair, one source of recordings and one primary measure keep the claim narrow. The frozen plan calls for 9,392 clips, 37,568 frames and 17 fits. Those alignment fits have not run.
- 2012
The images
The ImageNet-1K collection, used without class labels by original DINOv1 ViT-B/16. DINO training record · ImageNet release
- 2014 and earlier
The words
Original GloVe 6B: Wikipedia 2014 and the older Gigaword 5 collection. We average its word vectors without further language training. Original GloVe release
- 2017
The later episodes
EPIC-KITCHENS-55 recordings and their human narrations. The recording years come from the per-video metadata. Recording dates
These are corpus and recording dates, not model publication dates. The argument depends on the documented training histories and publisher-supplied dates. A short phrase may occur in both old text and a later narration; that alone is not exposure to the later episode.
Keep the test apart.
The existing sample was fixed before alignment scores: 6,639 fitting clips, 1,127 development clips and 1,626 test clips. The seven test kitchens appear in neither other partition.
Selection checkedWithhold the correspondence.
The main fit receives images from 73 videos and text from 74 different videos. Known-pair mappings, shuffled-pair mappings, random maps and separate encoder probes help us interpret success and failure.
17 fits prescribed · pausedState what a match means.
The first measure asks whether the nearest description has the same annotated noun class. Each test kitchen receives equal weight; duplicate descriptions and exact ties are accounted for. Recognizing a cup does not yet distinguish this cup or what is done with it.
No benchmark scoresWhat could follow from the result?
| What we observe | What we may conclude |
|---|---|
| Unpaired retrieval exceeds the negative controls; paired retrieval also works. | Evidence of recoverable object-category correspondence for this pair on later episodes. Shared concepts and model biases remain possible explanations for that correspondence. |
| Paired retrieval works; unpaired retrieval does not. | The frozen features support a useful mapping with pairing information. The tested unpaired procedure has not recovered it under these conditions. |
| Neither paired nor unpaired retrieval works. | The features, pooling, task or domain may be unsuitable. This outcome alone cannot identify the unpaired solver as the cause. |
Unexpected behavior between controls also needs investigation. No outcome from this one pair establishes a universal representation, understanding of actions or syntax, or independence from model choice. Changing both the models and the benchmark cannot measure how much contamination affected the original COCO/SPC result.
The run is paused. Acquisition and its supervisor were suspended on 10 October 2026 before any full EPIC alignment scores. Existing data, caches and the fixed protocol are preserved. The five-photo trial and the unrun 48-photo proposal are separate records.
The controls have limits too. Paired and shared-episode controls use all fitting clips, a larger information budget than the main disjoint fit. Annotated action windows supply preprocessing supervision. These comparisons cannot isolate contamination by themselves.
Work record · 10 October 2026. Paused-run record · Code and checks. The earlier pilot below used different models and data.
EXAMINING THE SOURCES
Two names.
Some of the same images.
We joined the SPC, Visual Genome, and COCO metadata. In the released cross-dataset setup, some descriptions and images refer to the same source.
Read the source record6,258unique COCO training images
described in SPC
described in SPC
3,335unique COCO validation images
described in SPC
described in SPC
This establishes shared sources. It does not show that the aligner was told which items were pairs. The effect on published performance is still unknown; this finding does not concern the separate disjoint-half COCO experiment.
02 / THE EARLIER PILOT
The first check held the models still.
Five conditions.
The same two models.
Keep DINOv2-B and MPNet fixed. Remove the known shared sources, then compare each removal with a random-removal control of the same size.
O / ORIGINAL
Begin with the original SPC population.
Keep the known validation and training source matches. This gives us the starting condition.
WHAT THIS COMPARISON ASKS
What do we observe before removing either kind of shared source?
19,561SPC paragraph rows retained
Starting condition0Original population: 19,561 rows
- Known validation source
- 3,337
- Known training source
- 6,261
- No known COCO match
- 9,963
- Removed
- 0
These are the full populations, before the pilot’s 4,096-row cap. Here we count paragraph rows; above we count unique COCO images. Random controls use selection seed 0.
Ask each the same questions. Every arm uses the same three 2,048-query sets and complete 40,504-item gallery. Clean, all-population, and originally exposed queries are sampled separately.
Say only what was removed. “Source-disjoint” means known metadata identities only. Missing mappings, visual near-duplicates, and pretraining overlap remain unresolved. Random deletion matches size, not semantic composition.
03 / WHAT HAS BEEN SHOWN
Recorded · 10 October 2026
The trials ran.
What did they establish?
Each of the five conditions completed three alignment seeds. The reduced pilot shows that the runs finish, the exclusions are applied, and the saved results pass consistency checks.
Each fit uses 4,096 training rows per modality. These equal caps remove the full-population size contrasts. The scores therefore do not estimate the effects of the planned full-data exclusions.
Read the checks15 / 15fits completed
92,160query records verified across 45 strata
- Fixed query and gallery identities checked
- Source-identity exclusions checked
- Saved map, rank, and configuration hashes checked
- Mean rank, median rank, and FOSCTTM recomputed from saved ranks
Verification did not recompute embedding similarities or Recall@k from scratch.
Examine all 15 reduced pilot runs
FOSCTTM below uses the same clean queries against the full gallery. Lower is better. These reduced runs check the procedure; their scores do not answer the scientific question. Every seed is shown.
| Arm | Fit seed | Status | FOSCTTM | Fit time |
|---|
THE RECORDS
Examine the account.
The protocol and evidence records are below. They describe the work as it stood on the date shown; they do not update as experiments run.
The five-photo resultEvery selected match, the human assignment’s rank and the limitsAll 120 assignmentsThe full distance search, including every error and tieThe five-photo protocolPublished after collection and before feature inferenceThe earlier 48-photo proposalPreserved prospective rules; this study has not runEarlier collection-kit hashesThe unchanged code and protocol for the 48-photo proposalPreserved EPIC protocolThe paused first case: sample, controls, measures and limitsEarlier overlap-study proposalThe separate COCO/SPC questions and comparisonsSource-identity auditCounts, metadata hashes, and source referencesProposed model-choice designThe candidates and what their selection can establishOriginal implementationThe original code at the examined commit