# Deviation Remediation Plan — Geometric Memory Reproduction (ICML 2026 #22) **Purpose:** Close the 17 deviations identified between `repro_plan.md` and the executed reproduction, bringing the reproduction from "qualitatively correct at reduced scale" to "faithful to the paper's claimed scale and mechanism." **Paper:** arXiv:2510.26745 · **Repo:** `shahriarnz14/geometric_memory` · **Reproducer:** Luis Poveda **Status of this document:** PLAN ONLY — nothing below has been executed. No experiments launched, no files modified, no jobs submitted. --- ## 0. Framing The 17 deviations fall into four buckets. The remediation effort should follow the same priority order, because the buckets differ sharply in cost and scientific value: | Bucket | Deviations | Fix cost | Scientific value of fixing | |--------|-----------|----------|----------------------------| | A. Scale-down | #1, #2, #3, #7, #11 | **High** (A100/H100 GPU-hours) | **High** — this is what separates "directional" from "reproduced" | | B. Skipped recipes / wrong model class | #4, #5, #10, #12, #13, #14, #15 | **Medium** (mostly local/tiny) | **High for #13/#14** (Claim 4 mechanism), medium for the rest | | C. Infrastructure substitutions | #16, #17 | **Low** | **None** — cosmetic; only fix if native HF delivery is required | | D. Positive deviation | #6 | Already applied | Keep; document as a required patch | **Guiding principle:** Fixing bucket B #13/#14 (Node2Vec spectral mechanism) is the single highest value-per-GPU-hour action — it is cheap (tiny models, local CPU/GPU) and it repairs the one place where the reproduction tested a *different hypothesis* than the paper (Transformer spectral bias instead of the paper's Node2Vec spectral-convergence proof). Recommend doing bucket B before the expensive bucket A. --- ## Phase 1 — Repair Claim 4 & Claim 3 mechanism (bucket B, cheap, highest value) These are the deviations where we answered a *related but different* question than the paper. All run locally on GB10 or CPU (the plan always scoped these as "tiny, no HF Jobs needed"). ### 1.1 — Fix #13: Extract genuine Node2Vec-style embeddings (Claim 4.2) - **Deviation:** Used Transformer embeddings from the Claim 1 checkpoint instead of a 2-layer Node2Vec model. - **Fix:** Run the repo's `tiny_graphs_notebooks/experiment_notebooks/tiny_nn.ipynb` to train the 2-layer Node2Vec-style model on the same tiny star graph. - **Command (as-executed notebook → script):** ```bash cd /tmp/geometric_memory jupyter nbconvert --to script --execute \ tiny_graphs_notebooks/experiment_notebooks/tiny_nn.ipynb \ --output /tmp/repro_tiny_nn ``` - **Verification:** Confirm the model has exactly 2 layers and produces per-node embeddings whose dimensionality matches the graph node count. ### 1.2 — Fix #14: Track spectral alignment *during* training (Claim 4.4, Appendix F) - **Deviation:** Only a single post-hoc snapshot was measured; the paper's claim is that alignment *increases over training*. - **Fix:** Instrument the Node2Vec training loop from 1.1 to checkpoint embeddings every N epochs, compute Procrustes similarity + top-k Laplacian-eigenvector correlation at each checkpoint, and plot the trajectory. - **Deliverable:** `claim4_alignment_over_training.png` — x-axis epochs, y-axis Procrustes similarity / max eigenvector correlation, showing a rising curve (the paper's mechanism). - **Verification (this is the actual claim):** the curve must be *monotonically increasing* toward the top eigenvectors of the negative graph Laplacian. If it does NOT rise, that is a genuine reproduction failure and must be reported as such — do not massage it. ### 1.3 — Fix #12: Node2Vec vs Transformer geometry comparison (Claim 3.4) - **Deviation:** Node2Vec comparison notebook never run. - **Fix:** Run `tiny_graphs_notebooks/experiment_notebooks/tiny_node2vec.ipynb`; extract embeddings; compute the same geometry metrics already used for the Transformer (mean norm, self-similarity, pairwise similarity, clustering) and put them side by side. - **Deliverable:** extend the existing Claim 3 comparison table to a third column (Node2Vec) alongside Unfrozen/Frozen Transformer; regenerate heatmap + UMAP for the Node2Vec embeddings. ### 1.4 — Fix #11: Add a true tiny-scale Claim 3 artifact (optional within this phase) - **Deviation:** Claim 3 used the full 21.7M-param GPT at d=10 instead of a tiny model at N≤30. - **Fix:** The `tiny_transformer.ipynb` run is cheap; run it at N≤30 as the plan specified and regenerate the heatmap/UMAP at that scale to have the exact artifact the plan called for. - **Note:** Keep the existing large-model result too — having both (tiny + larger) strengthens the claim rather than replacing it. **Phase 1 compute:** local only (GB10 or CPU). **Est. wall-clock:** 2–4 h including notebook debugging. **No HF Jobs.** --- ## Phase 2 — Complete Claim 2 as a curve, not a point (bucket B, cheap-medium) ### 2.1 — Fix #8 + #9: Accuracy-vs-l sweep and exponential-gap plot (Claim 2.2, 2.3) - **Deviation:** Only l=4 tested; the paper's whole Claim-2 argument is the *shape* of accuracy vs l (geometric flat, associative collapsing). - **Fix:** Run the frozen (associative) and unfrozen (geometric) models across **l = 2, 3, 4, 5, 6**, on the plan's small graph. This is 10 short runs (5 lengths × 2 conditions). - **Command pattern (per l, per condition):** ```bash cd /tmp/geometric_memory python train_in_weights.py \ --training_recipe mixed_full_path \ --model_family gpt --graph_type star \ --star_degree 3 --star_subtree_degree 2 \ --path_length ${L} \ --add_forward_edges --add_backward_edges \ [--freeze_token_embeddings] # include for the associative arm only --edge_memorization_epochs 2500 --path_finetuning_epochs 10000 \ --experiment_log_root ./experiment_logs --no-use_wandb ``` *(Note: honor the batch_size fix from #6 — do NOT re-inherit the 16384 default.)* - **Deliverable:** `claim2_accuracy_vs_l.png` — two curves; the associative arm should degrade sharply with l while the geometric arm stays high. This is the plot the plan (step 2.3) required and that is currently missing. ### 2.2 — Fix #7: Use the plan's actual baseline graph spec - **Deviation:** Claim 2 reused Claim 1's d=10 graph instead of the plan's `star_degree=3, subtree_degree=2` (N=22). - **Fix:** The sweep in 2.1 already adopts `--star_degree 3 --star_subtree_degree 2`, so running 2.1 resolves #7 automatically. Keep the d=10 comparison as a secondary internal-consistency check. **Phase 2 compute:** small-scale; the plan says CPU or small GPU is sufficient. Use GB10 locally. **Est. wall-clock:** 3–5 h for 10 runs. **HF Jobs only if local memory is constrained.** --- ## Phase 3 — Hardest-token contradiction (bucket B, medium) ### 3.1 — Fix #5 + #10: First-token-only training (Claim 1.4 + Claim 2.4) - **Deviation:** the `mixed_hardest_token` recipe was never run for either the geometric or associative model. - **Fix:** Run the hardest-token recipe for both conditions at the small scale used in Phase 2. ```bash python train_in_weights.py \ --training_recipe mixed_hardest_token \ --model_family gpt --graph_type star \ --star_degree 3 --star_subtree_degree 2 --path_length 4 \ --add_forward_edges --add_backward_edges \ [--freeze_token_embeddings] # associative arm --edge_memorization_epochs 2500 --path_finetuning_epochs 10000 \ --experiment_log_root ./experiment_logs --no-use_wandb ``` - **Verification (the claim):** the associative (frozen) model should *fail* the hardest-token task while the geometric (unfrozen) model succeeds — this is the paper's Section 2.3 contradiction. Report the actual first-token accuracy for both. **Phase 3 compute:** small-scale local. **Est. wall-clock:** 1–2 h. --- ## Phase 4 — Scale up Claim 1 (bucket A, expensive — the big one) This is the costliest bucket and the one that closes the headline gap (60% → target ~100%). Gate it behind Phases 1–3 so the cheap mechanism-level fixes land first. ### 4.1 — Fix #1 + #3: Full-scale Claim 1 at the paper's node count and epoch budget - **Deviation:** ran d=10 (~110 nodes) for 500+2000 epochs vs the plan's d=10000 (50k nodes) for 2500+10000 epochs. - **Decision point (needs sign-off before spending GPU-hours):** the paper reports up to 50,000 nodes. `star_degree=10000` is the plan's spec. Confirm the target scale before launching — there is a large cost difference between, say, d=1000 (intermediate, cheaper, likely enough to show accuracy climbing well above 60%) and the full d=10000. - **Recommended staged approach (de-risks the spend):** 1. **d=1000, l=6** first — one A100 job. If path accuracy climbs from 60% toward 90%+, the scaling hypothesis is confirmed cheaply. 2. Only then commit to **d=10000, l=6** and **d=10000, l=10** (the two Fig. 3 points). - **Command (full scale, HF Jobs A100-80GB):** exactly the plan's Claim 1.1 / 1.2 commands (lines 39–67 of `repro_plan.md`), with the **mandatory `batch_size=64` override** from deviation #6 added — the repo default of 16384 will silently zero out training again at these dataset sizes. - **Verification:** first-token accuracy on unseen paths should approach ~100% at l=6 and ~95%+ at l=10 (paper's Fig. 3). Log both to Trackio. ### 4.2 — Fix #2: Both path lengths (l=6 AND l=10) - Covered by launching both Fig. 3 points in 4.1 step 3. ### 4.3 — Fix #4: Mamba variant (Claim 1.2 / Fig. 14) - **Deviation:** never run; `mamba-ssm` unavailable locally due to CUDA incompatibility. - **Fix:** run on HF Jobs where the CUDA/`mamba-ssm` stack can be installed in the job container (the plan already anticipated HF Jobs as the fallback for Mamba). Same config as 4.1 with `--model_family mamba`. - **Container note:** the HF Job image must install a `mamba-ssm` build matching the job's CUDA version; pin versions explicitly to avoid the local incompatibility that blocked this originally. - **Verification:** Mamba should show similar path-memorization behavior to the Transformer (paper's Fig. 14). **Phase 4 compute:** HF Jobs, A100-80GB or H100. **Est. GPU-hours:** highly dependent on the d=1000-vs-d=10000 decision — could range from ~5 GPU-hours (d=1000 pilot only) to 50+ GPU-hours (full d=10000, both l values, plus Mamba). **This is the phase to cost-scope explicitly with the user before launch.** --- ## Phase 5 — Robustness / generality (bucket B, Section 5, medium — optional) ### 5.1 — Fix #15: The four skipped robustness checks - **Deviation:** 0 of 4 Section-5 experiments run. - **Fix (in ascending cost order):** - **5.2 self-edge ablation** — run `experiment_notebooks_self_edges/`; cheap, local. - **5.3 regularizer ablation** — run `experiment_notebooks_regularizers/`; cheap, local. - **5.1 non-star graphs** (grid, cycle, irregular) — medium; tests generality of the memorization claim beyond star graphs. - **5.4 l=8 scale-up** — medium; extends beyond the paper's reported l range. - **Priority:** these are "additional / robustness" in the original plan and were correctly deprioritized. Do them only if the user wants generality evidence; they are not required to close the core claims. **Phase 5 compute:** mostly local; non-star and l=8 may need a small GPU. **Est. wall-clock:** 2–4 h if all four. --- ## Phase 6 — Publishing delivery (bucket C, low cost — conditional) ### 6.1 — Fix #17: Native HF Space delivery - **Deviation:** HF Static Space returned 404 across 3 repos/2 accounts; final artifact served from Vercel. - **Fix options (pick per user preference):** - **(a) Accept Vercel as canonical** — it works, is verified HTTP 200, and images are backed by the HF dataset repo. Lowest effort; only cost is that delivery isn't "native HF." - **(b) Retry native HF Space once the platform recovers** — the queued commit in `/tmp/hf-space-check` (redirect page + HF-dataset image URLs) just needs `git push --force origin main` when `huggingface.co` stops returning 504. Re-run `trackio sync` afterward to restore the dashboard listing. - **(c) Investigate the Static Space 404 root cause** — if native rendering (not just redirect) is required, this needs a support ticket or a minimal-repro Static Space to isolate whether it's an account-level or platform-level build issue. Higher effort, uncertain payoff. - **Recommendation:** (b) — retry the queued push when HF is healthy; keep Vercel as the durable mirror. Only escalate to (c) if native in-Space rendering is a hard requirement. ### 6.2 — Fix #16: Poster tool - **Deviation:** built with matplotlib instead of Posterly. - **Fix:** none required — the A0 PDF deliverable exists and is correct. Only revisit if Posterly specifically is mandated; otherwise close as an acceptable substitution. ### 6.3 — Re-upload the poster PDF to the HF dataset (carry-over) - The 8th file (poster PDF) failed upload with repeated 504s. Once HF is healthy, push it via a git clone of the `luispoveda93/geometric-memory-images` dataset repo (git is more resilient than the `preupload` API for this). --- ## Dependency Graph & Recommended Order ``` Phase 1 (Claim 3/4 mechanism, local) ─┐ Phase 2 (Claim 2 curve, local) ─┼─→ cheap, do first, no gating Phase 3 (hardest-token, local) ─┘ │ Phase 4 (Claim 1 scale-up, HF Jobs) ←─────┘ gate behind a cost decision 4.1 d=1000 pilot → decision → d=10000 full 4.3 Mamba (parallel-capable once container is fixed) │ Phase 5 (robustness, optional) ──────────┤ optional, user-driven │ Phase 6 (publishing) ───────────────────── runs whenever HF recovers; independent of experiments ``` **Rationale for order:** Phases 1–3 are cheap, local, and repair the *mechanism-level* fidelity (the parts where we tested a different hypothesis than the paper). They should not wait on the expensive GPU spend. Phase 4 is the only bucket needing an explicit budget conversation. Phase 6 is orthogonal and gated only by HF platform health. --- ## Open Questions / Unresolved (require user decision before execution) 1. **Claim 1 target scale (#1):** d=1000 pilot only, or commit to the full d=10000 (50k nodes)? This is the single largest cost driver — needs an explicit budget ceiling before any A100/H100 job is launched. 2. **Mamba priority (#4):** is cross-architecture reproduction (Fig. 14) required, or is Transformer-only acceptable? Mamba adds container-setup risk and GPU cost. 3. **Section 5 robustness (#15):** in scope, or explicitly deferred? The original plan already framed these as "additional." 4. **Publishing delivery (#17):** is native HF Space rendering a hard requirement, or is Vercel (with HF dataset image backing) an acceptable canonical source? This determines whether Phase 6 is a 1-command retry or a support-ticket investigation. 5. **Compute account (#4, #4.1):** confirm which HF account / billing has A100-80GB or H100 quota — the earlier Gradio-quota failure showed the free tier has none. --- ## Verification Gate (applies to every phase before marking a deviation "closed") A deviation is only closed when: 1. The experiment ran with the plan's specified parameters (or a documented, user-approved substitute). 2. The result is logged to Trackio AND reflected in the status report's results tables. 3. For mechanism claims (#13, #14, #10, #4), the *direction* of the result is checked against the paper's prediction — and a failure to reproduce is reported honestly, not smoothed over. 4. The deviation row in `geometric_memory_repro_status.md` is updated from its current state to "closed" with the new evidence linked. --- *Plan generated 2026-07-16. Execution status as of 2026-07-16T19:18Z:* **Phase 1 (Node2Vec mechanism):** ✅ Completed. Details in status report § Phase 1. **Phase 2 (Claim 2 sweep):** ✅ Completed (negative result — 0% path accuracy across all l=2..6). Details in status report. **Phase 3 (Hardest-token):** ✅ Completed. Geometric=100%, Associative=33.33% final. Details in status report. ||**Phase 4 (Claim 1 scale-up):** ❌ **BLOCKED** — Both A100 jobs CANCELED (insufficient credits). Corrected d=100/l=6 retry prepared but cannot submit (402 Payment Required on all flavors). |- Job `6a58c104b1669a49bf077619` (staged_full_path): **CANCELED at ~4h** — edge epoch 2159/2500, accuracy regressing 42%→34%, never reached path finetuning. Model diverging at d=1000 with LR=1e-2. |- Job `6a58bda185d9643ce16d627d` (mixed_full_path): **CANCELED at ~8h** — joint epoch 2646/10000, accuracy ~45% (barely above random). Mixed recipe not converging at d=1000. |- **No duplicate submitted** — two different recipes, both canceled. |- Decision on next experiment: **BLOCKED by insufficient HF Jobs credits ($0.00 balance).** Prepared d=100/l=6 staged_full_path with LR=3e-3 but all flavors return 402 Payment Required. ||**Phase 5 (Robustness):** ✅ Completed. All 7 manifests (20260716-1311 through -1312 timestamps) consolidated and independently verified via repository evaluator `compute_topk_recovery_percent`: every artifact recomputed to **100.0%** top-k edge recovery. Verification: `/tmp/remediation_phase5/consolidated_verification.json`. Published to HF dataset and HF Space. ||**Phase 6 (Publishing):** ✅ HF platform recovered (HTTP 200). Dataset repo has all 8 files + remediation artifacts. HF Space push succeeded (`61ae92a`). Vercel remains canonical rendered report due to persistent Static Space 404 (HF build-pipeline issue, not config). Trackio sync still blocked (missing local database). ||**Remaining blocks:** ||- Phase 4 completion: **BLOCKED by insufficient HF Jobs credits ($0.00 balance).** Both d=1000/l=6 pilots canceled mid-run. Model was not converging (edge accuracy regressing from 42% to 34% with LR=1e-2). Corrected d=100/l=6 retry with LR=3e-3 prepared but cannot submit — all GPU flavors (T4-small through A100-large) return 402 Payment Required. Add $5-10 minimum credits, then submit the corrected d=100/l=6 staged_full_path job. ||- Phase 6: Trackio sync still blocked by missing local database. HF Static Space 404 persists (build-pipeline issue, not config — reproduced across 3 repos and 2 accounts). Vercel remains canonical rendered report.