You are running a new extension experiment for a research paper on
strength-dependent context-response rankings in language models.

Your job is to complete the experiment end-to-end on this A100 machine:

1. download the required public dataset and models;
2. implement and validate the intervention experiment;
3. run the full experiment;
4. analyze all results using the frozen protocol below;
5. write a complete scientific report and save all raw data/artifacts.

Do not optimize the protocol after looking at outcomes.
Do not silently substitute datasets, models, metrics, or intervention definitions.
If something is unavailable, record the blocker explicitly and continue only with
the parts that remain semantically valid.

============================================================
A. SCIENTIFIC QUESTION
============================================================

We want to test:

    Within the SAME prompt, when several candidate context spans are available,
    does the ranking of those spans remain stable as intervention strength changes?

This is different from ranking different examples across the dataset.

For each query, construct one prompt containing:

    - one answer-bearing retrieved passage;
    - five non-answer-bearing retrieved passages.

The five non-answer-bearing passages are candidate spans D_1,...,D_5.

For every candidate span D_j, measure its signed response under several
intervention strengths while keeping the prompt, passage positions, prefill,
gold answer, and model fixed.

Primary question:

    How similar is the within-prompt ranking
        D_1,...,D_5
    under lambda = -0.10 versus lambda = -2?

Secondary question:

    Across the full six-strength trajectory, do rank reversals tend to arise
    through coherent single crossings?

This experiment is intended to connect the paper's controlled context-access
measurement more directly to a memory-maintenance scenario where several
candidate memories/items are ranked within one context.

============================================================
B. PROTOCOL FREEZE
============================================================

Before running ANY model inference, create:

    protocol_v1.json
    cohort_manifest.json
    environment.txt

Commit them to git.

Once the first model score has been generated:

    - do not change the cohort;
    - do not change the primary endpoints;
    - do not change the primary metrics;
    - do not change candidate-selection rules;
    - do not change passage ordering;
    - do not change strength values.

If an implementation bug is found later, document it in CHANGELOG.md,
fix it, increment protocol version, and rerun all affected conditions.

Random seed:

    20260917

Cohort size:

    96 queries

Candidates per query:

    5 non-answer-bearing passages

Strength ladder:

    lambda in {-0.10, -0.25, -0.50, -1, -2, -4}

Primary endpoint comparison:

    lambda = -0.10 versus lambda = -2

Primary response definition:

    S_lambda(q, j) = Y_lambda(q, j) - Y_0(q)

where:

    Y_0(q)
        = mean teacher-forced log probability of the gold-answer tokens
          under natural access;

    Y_lambda(q, j)
        = the same score after adding scalar attention-logit bias lambda
          to all answer-time attention logits whose KEY positions belong
          to candidate passage D_j.

Use SIGNED S_lambda for all primary analyses.

Also compute |S_lambda| ranking as a clearly labeled SECONDARY sensitivity
analysis. Never replace the signed primary analysis with the magnitude analysis.

============================================================
C. DATA DOWNLOAD
============================================================

Use Natural Questions with DPR retrieval.

Preferred source:

    official facebookresearch/DPR resources.

Clone:

    https://github.com/facebookresearch/DPR

Use the official NQ test retrieval output resource corresponding to:

    data.retriever_results.nq.single-adv-hn.test

Inspect the downloaded format first.

If the retrieval file already contains:
    question
    answers
    ranked contexts
    title/text

use it directly.

If passage text is referenced only by ID, also download the official DPR
Wikipedia passage corpus:

    psgs_w100.tsv.gz

Do NOT rerun retrieval with a different retriever unless the official retrieval
results are genuinely unavailable.

Save original downloaded files under:

    data/raw/

Save SHA256 checksums in:

    data/raw/CHECKSUMS.txt

============================================================
D. QUERY / PASSAGE ELIGIBILITY
============================================================

For each NQ query:

1. Normalize every accepted gold-answer alias for matching:
       lowercase
       strip surrounding whitespace
       normalize repeated whitespace
       normalize Unicode
       remove surrounding punctuation for matching only

2. A retrieved passage is "answer-bearing" if its title OR body contains an
   exact normalized match of ANY gold-answer alias.

3. A passage is "non-answer-bearing" if no normalized gold-answer alias appears
   in its title or body.

4. Search only within the top 20 DPR-retrieved passages.

5. A query is structurally eligible if top-20 retrieval contains:
       >= 1 answer-bearing passage
       >= 5 non-answer-bearing passages

6. Select:
       evidence passage =
           highest-ranked answer-bearing passage;

       candidate D_1...D_5 =
           five highest-ranked non-answer-bearing passages.

Do NOT inspect model scores during this selection.

Gold teacher-forced answer:

    use the FIRST gold answer alias in the dataset's stored order.

Answer-bearing / non-answer-bearing detection should still use ALL aliases.

============================================================
E. COHORT SELECTION
============================================================

Do not manually choose queries.

For every structurally eligible query, compute:

    SHA256("multispan-v1|" + stable_query_id_or_question)

Sort eligible queries lexicographically by this hash.

Take the first 96 that also satisfy the context-length checks below.

Do not call this cohort "fresh" in the report, because overlap with previous
experiments cannot be audited on this isolated machine.

Instead call it:

    "structurally selected multi-span cohort"

Write all selected query IDs, questions, answers, retrieved passage IDs,
retrieval ranks, titles, and candidate/evidence roles to:

    cohort_manifest.json
    cohort_manifest.csv

============================================================
F. PASSAGE ORDERING INSIDE EACH PROMPT
============================================================

We want one common prompt per query.

The prompt must contain all six selected passages:

    1 answer-bearing evidence passage
    5 non-answer-bearing candidate passages

Do not always put candidates in DPR-rank order.

For each query, deterministically shuffle the six passages using:

    SHA256("multispan-order-v1|" + query_id)

Freeze this order in cohort_manifest.json before inference.

The passage order must be identical across:

    all lambda values
    all candidate interventions
    all reruns

The same semantic passage order should be used for both models.

============================================================
G. PROMPT FORMAT
============================================================

Use one fixed plain-text prompt format for both models.

Do NOT use model-specific chat templates.

Use exactly:

    You are given retrieved passages. Answer the question with a short answer.

    Passage 1:
    Title: {title_1}
    {text_1}

    Passage 2:
    Title: {title_2}
    {text_2}

    ...

    Passage 6:
    Title: {title_6}
    {text_6}

    Question: {question}
    Answer:

Then teacher-force the selected gold answer.

Do not sample or generate answers.

Store the exact rendered prompt text for every query.

============================================================
H. TARGET SPAN DEFINITION
============================================================

For a candidate passage D_j, target the passage CONTENT:

    Title: ...
    passage text

Do not include unrelated delimiter text from neighboring passages.

Use tokenizer offset mappings or another exact method to map character spans
to token indices.

For every model/query/candidate, save:

    candidate char start/end
    candidate token start/end
    number of target tokens

Sanity check:

    - candidate token spans are non-empty;
    - candidate spans do not overlap;
    - all candidate spans occur before the question/answer region;
    - target token text decodes back to the intended passage content.

============================================================
I. MODELS
============================================================

Primary models:

    Qwen/Qwen2.5-3B-Instruct

    meta-llama/Llama-3.2-3B-Instruct

Use Hugging Face Transformers and PyTorch.

Record:
    exact model repository
    exact model commit/revision SHA
    tokenizer revision
    transformers version
    torch version
    CUDA version
    GPU model
    dtype

Run in:

    torch.bfloat16

Use:

    attn_implementation="eager"

because the intervention must modify raw attention logits before softmax.

If Llama access is gated and HF_TOKEN does not have permission:

    - DO NOT replace Llama with another model;
    - run Qwen fully;
    - write the exact blocker into BLOCKERS.md;
    - leave the Llama result section explicitly incomplete.

============================================================
J. INTERVENTION SEMANTICS
============================================================

This is critical.

For prompt prefill:

    lambda = 0
    no intervention
    normal model computation

For teacher-forced ANSWER scoring only:

    at every transformer layer
    at every attention head
    for every answer-time query token

add scalar lambda to the raw attention logits for keys whose positions are
inside the selected D_j token span.

Conceptually:

    attention_logits[..., target_key_positions] += lambda

before softmax.

Do not modify:

    prompt tokens
    non-target key positions
    values
    hidden states directly
    token positions
    RoPE
    KV contents
    gold answer
    passage ordering

For each query, Y_0 is shared across all five candidate spans.

Do NOT recompute a different natural baseline for each candidate.

============================================================
K. IMPLEMENTATION STRATEGY
============================================================

Preferred implementation:

1. run the prompt prefill normally;
2. obtain the prompt KV cache;
3. teacher-force the gold answer token by token;
4. during answer decoding, patch eager attention so the chosen target-key
   positions receive the additive lambda bias;
5. compute mean gold-token log probability.

The prefill state must be identical across candidates and strengths.

For efficiency, you may reuse the same immutable prefill cache.

Be careful:
    Hugging Face DynamicCache objects may mutate during answer decoding.

If safe cache cloning is difficult:
    rerun the identical prompt prefill for each condition instead.

Correctness is more important than speed.

Do not apply intervention during prefill.

============================================================
L. REQUIRED IMPLEMENTATION VALIDATION
============================================================

Before full inference, build a 2-query smoke test and verify ALL of the following.

1. Lambda-zero equivalence

Run the patched path with lambda = 0.

It must match the completely unmodified model path to numerical tolerance.

Report:
    max absolute logit difference
    Y difference

2. Shared baseline

For one query, all five candidate conditions must share exactly the same Y_0.

3. Target-mask correctness

Print decoded target tokens for all five candidates.
Manually inspect in the report/log that each mask corresponds to the intended
passage.

4. Prefill invariance

Confirm that intervention is inactive during prompt prefill.

5. Attention direction check

For the FIRST answer token on the smoke test, record target attention mass for:

    lambda = 0
    -0.10
    -0.25
    -0.50
    -1
    -2
    -4

Target attention mass should decrease as the additive bias becomes more
negative for that fixed first-token query state.

6. Non-target logit check

Verify that the intervention tensor adds zero to all non-target key positions.

7. Deterministic rerun

Run both smoke-test queries twice.
The resulting Y and S values must reproduce to numerical precision.

Write:

    validation_report.md

Do not launch the full 96-query experiment until the validation report passes.

============================================================
M. CONTEXT LENGTH
============================================================

Require that the full prompt plus teacher-forced answer fits within 8192 tokens
for BOTH model tokenizers without truncation.

No truncation is allowed.

If a selected query exceeds 8192 tokens for either model:

    discard it structurally before any model scoring
    and continue down the precomputed hash ordering until 96 valid queries remain.

Record discarded IDs and reasons.

============================================================
N. FILE / SCRIPT STRUCTURE
============================================================

Create at least:

    setup_env.sh
    download_assets.sh
    build_multispan_cohort.py
    intervention/
        attention_bias.py
        scoring.py
        masks.py
    run_multispan.py
    analyze_multispan.py
    make_figures.py
    tests/
        test_masks.py
        test_lambda_zero.py
        test_determinism.py

Outputs:

    outputs/protocol_v1.json
    outputs/cohort_manifest.json
    outputs/cohort_manifest.csv
    outputs/raw_scores.parquet
    outputs/raw_scores.jsonl
    outputs/validation_report.md
    outputs/summary_metrics.json
    outputs/summary_metrics.csv
    outputs/report.md
    outputs/figures/
    outputs/tables/
    outputs/environment.txt
    outputs/BLOCKERS.md
    outputs/CHANGELOG.md

============================================================
O. RAW SCORE SCHEMA
============================================================

Each scored row should contain at least:

    model
    model_revision
    query_id
    question
    gold_answer
    evidence_passage_id
    candidate_passage_id
    candidate_retrieval_rank
    candidate_prompt_position
    candidate_index
    lambda
    Y0
    Y_lambda
    S_lambda
    abs_S_lambda
    prompt_token_count
    answer_token_count
    target_token_start
    target_token_end
    target_token_count
    seed
    timestamp
    protocol_version

One row per:

    model × query × candidate × lambda

Expected full row count if both models complete:

    2 × 96 × 5 × 6 = 5760 rows

plus separate natural-baseline records if useful.

============================================================
P. PRIMARY STATISTICAL ANALYSIS
============================================================

All uncertainty must use QUERY-LEVEL bootstrap.

Never bootstrap individual candidate spans or candidate pairs independently,
because the five candidates within a query are dependent.

Use 5000 bootstrap resamples of the 96 queries.

Primary endpoint:

    S_-0.10 versus S_-2

Primary metrics:

------------------------------------------------------------
P1. Pooled within-query Spearman rank agreement
------------------------------------------------------------

For each query and each endpoint:

    rank the five candidate spans by signed S_lambda.

Convert each query's five scores into within-query ranks 1...5.

Flatten all within-query ranks across the 96 queries and compute Spearman
agreement between weak and strong ranks.

Bootstrap by resampling queries.

Report:
    rho
    95% bootstrap CI

Also report the distribution of per-query Spearman rho as a secondary
description:
    median
    IQR
    mean

------------------------------------------------------------
P2. Pairwise reversal fraction
------------------------------------------------------------

For each query there are C(5,2) = 10 candidate pairs.

For pair (a,b), define:

    delta_weak   = S_-0.10(a) - S_-0.10(b)
    delta_strong = S_-2(a)    - S_-2(b)

A raw reversal occurs when:

    delta_weak * delta_strong < 0

Handle exact ties separately.

Compute the reversal fraction per query, then average across queries.

Bootstrap queries.

Report:
    mean reversal fraction
    95% CI
    exact-tie fraction

------------------------------------------------------------
P3. Top-1 agreement
------------------------------------------------------------

Using the SIGNED S_lambda ranking:

    does the same candidate rank first at -0.10 and -2?

Report:
    fraction of queries with same top-ranked candidate
    95% bootstrap CI

Also report top-2 Jaccard as secondary.

Do not reinterpret "top" as absolute influence.
It is simply the top item under the signed response definition used here.

============================================================
Q. ROBUST-REVERSAL SENSITIVITY
============================================================

Compute one secondary robust-reversal analysis using a prespecified
0.10-IQR margin.

For each model and endpoint separately, compute the IQR over ALL candidate
item scores:

    {S_lambda(q,j): q=1..96, j=1..5}

For a candidate pair, require:

    |delta_weak|   >= 0.10 * IQR_weak
    AND
    |delta_strong| >= 0.10 * IQR_strong

Among qualified pairs, compute reversal fraction.

Bootstrap at query level.

This is SECONDARY.

Always report both:
    raw reversal
    robust reversal

============================================================
R. FULL SIX-STRENGTH ANALYSIS
============================================================

Compute a 6×6 matrix of pooled within-query Spearman rank agreement across:

    -.10, -.25, -.50, -1, -2, -4

Create:

    figures/within_query_spearman_heatmap.pdf
    tables/within_query_spearman_matrix.csv

Also compute adjacent-strength agreement.

============================================================
S. CROSSING STRUCTURE
============================================================

Use the trajectory from:

    -.10 -> -.25 -> -.50 -> -1 -> -2

for candidate pairs that robustly reverse between -.10 and -2.

For each pair, define:

    Delta_q,ab(lambda) = S_lambda(q,a) - S_lambda(q,b)

Count sign changes along the ordered strength sequence.

Ignore isolated exact-zero intermediate values by removing zeros before counting
sign changes.

A "single crossing" means exactly one sign change.

Report:

    number of endpoint-reversing qualified pairs
    single-cross fraction
    multi-cross fraction
    95% query-bootstrap CI

Also run an endpoint-preserving permutation null:

    within each query and candidate pair,
    preserve endpoint values at -.10 and -2,
    independently permute the intermediate strength columns
    -.25, -.50, -1

Use 5000 permutations.

Report:
    null mean single-cross fraction
    97.5th percentile
    empirical p-value

This analysis should be described as trajectory structure, not neural mechanism.

============================================================
T. SECONDARY |S| ANALYSIS
============================================================

Repeat only the following metrics using |S_lambda|:

    pooled within-query Spearman
    pairwise reversal fraction
    top-1 agreement

Label all results:

    "magnitude-ranking sensitivity analysis"

Report them regardless of whether they strengthen or weaken the primary story.

Do not replace signed results with magnitude results.

============================================================
U. OPTIONAL SIMPLE NULLS
============================================================

For interpretability, generate a within-query random-ranking null by randomly
permuting the five strong-endpoint candidate identities within each query.

Estimate expected:

    pooled rank correlation
    reversal fraction
    top-1 agreement

Use this only as a descriptive reference.

Do not treat it as the main significance test.

============================================================
V. FIGURES
============================================================

Create publication-quality vector PDF figures.

Required:

1. within_query_spearman_heatmap.pdf

    six-strength rank-agreement matrix
    one panel per model

2. endpoint_summary.pdf

    compact comparison of:
        weak-vs-strong pooled Spearman
        pairwise reversal
        top-1 agreement

3. crossing_summary.pdf

    empirical single-cross fraction versus endpoint-preserving null

4. example_trajectories.pdf

Selection rule for illustrative queries:

    among queries containing at least one robust endpoint reversal,
    hash query IDs with:
        SHA256("trajectory-example-v1|" + query_id)
    and choose the first 4 by hash.

Do not manually choose visually attractive examples.

Plot all five candidate trajectories for each chosen query.

Clearly label these as illustrative examples.

Use restrained, colorblind-safe colors.
No high-saturation red/green defaults.

============================================================
W. REPORT
============================================================

Write:

    outputs/report.md

The report must be self-contained and include:

1. Executive summary

State:
    whether within-prompt rankings are highly stable, moderately stable, or show
    substantial disagreement;

but base that wording explicitly on the numerical results.

Do not make universal claims.

2. Data and protocol

Include:
    dataset source
    retrieval source
    cohort construction
    prompt template
    candidate definition
    model revisions
    strength ladder
    score definition

3. Validation

Include every smoke-test result from Section L.

4. Primary results

For each model, provide a table:

    Metric
    Estimate
    95% CI

including:
    pooled within-query Spearman
    raw pairwise reversal
    robust pairwise reversal
    top-1 agreement
    top-2 Jaccard

5. Six-strength structure

Include:
    full Spearman matrix
    adjacent-strength trend

6. Crossing structure

Include:
    reversing-pair count
    single-cross fraction
    null comparison

7. Magnitude sensitivity

Report |S| results separately.

8. Model comparison

Describe similarities and differences without claiming a universal model-family law.

9. Limitations

At minimum mention:
    one dataset/retrieval setup
    only five candidate spans per prompt
    answer-free candidate definition
    signed teacher-forced response
    no persistent agent-memory system
    candidate passage position remains part of each item, although position is
    fixed across intervention strengths
    overlap with prior local experiments has not yet been audited

10. Recommended manuscript interpretation

Use cautious language such as:

    "Within a shared prompt containing five candidate context spans, the
    cross-candidate ordering changes as access suppression becomes stronger."

If supported by results, also state:

    "The same-prompt extension shows that strength-dependent ranking changes are
    not limited to comparisons across independent examples."

Do NOT say:

    - agent memory deletion is solved;
    - all memory rankings are unstable;
    - weak probes are invalid;
    - stronger probes are universally better;
    - the observed crossing is a neural mechanism;
    - this is cross-dataset replication;
    - this is deployment utility.

============================================================
X. FINAL DELIVERY CHECKLIST
============================================================

Before finishing, verify that the repository contains:

    working code
    frozen protocol
    cohort manifest
    model/data provenance
    validation report
    complete raw scores
    summary CSV/JSON
    vector figures
    scientific report
    environment information
    blockers/changelog

Run:

    python -m pytest -q

and record the result.

Also save one single command or shell script:

    reproduce_all.sh

that reproduces:
    analysis + tables + figures + report

from the completed raw score file without rerunning model inference.

Do not delete intermediate raw results.

At the very end, print a concise completion summary with:

    models completed
    number of valid queries
    number of scored rows
    all validation checks PASS/FAIL
    primary endpoint metrics
    exact output paths
    unresolved blockers