# Week 10 capstone lab: an identified effect is not yet a portable claim

## Learning objectives

Estimate effects from observed source data; distinguish source weighting from target standardization; reconstruct context-stratified uncertainty and a source sharp-null test; explain overlap, stability and weight concentration; and defend a research memo with an honest access and validation appendix.

**Mode: original synthetic design reconstruction; not empirical replication.** All establishments, parameters and numerical examples are invented. Generated records carry `SYNTHETIC_TEACHING_CONSTRUCTION_NOT_PUBLISHED_REPLICATION`. Allcott (2015) motivates source identification versus out-of-site prediction; Bloom–Han–Liang (2024) supplies the personnel-study claims audit; DellaVigna–Linos (2022) supplies the evidence-selection comparison. The synthetic lottery is none of those papers' empirical designs.

## Bridge: three separate achievements

A randomized effect in a source population is distinct from a mechanism interpretation and a prediction for another organization. Better precision does not repair changed implementation, absent contexts or a recording measure that fails to track production. A one-firm study can contribute through a well-defined decision and informative comparison; a multi-firm dataset can still be selected.

Read the [README](README.md), then run:

```bash
ENV_NAME=research bash smoke.sh
```

The native default seed is **20260922**. For a separate run, use a fresh directory and this week's `--out` option:

```bash
conda run -n research python src/offline_lab.py --out /tmp/org8-week10-student-run --seed 88771
```

The runner refuses populated targets with exit 2. Keep existing work and choose a new path; use the documented staging installer when refreshing canonical generated artifacts.

## Field Core — Reproduce from observed records

### 1. Freeze source, target, and timing

Load only `data/teaching/source_sites.csv` and `target_frame.csv`; consult `teaching-schema.csv`. The source contains twelve establishments in six matched pairs. Four pairs have high implementation support and two have low support. **Implementation support means organizational resources**, not statistical common support: both contexts occur in the source. One site in each pair is treated, yielding $2^6=64$ equally likely assignments.

Potential outcomes, baselines and untreated changes are fixed before a separate assignment random stream. The initial source cohort is fully observed, measurement is stable, and source interference is absent by construction. The source composition is selected; firms were not randomly sampled into this frame.

The target frame contains twenty sites, five high-support and fifteen low-support, each with baseline target mass .05. It contains identifiers, context and weights, **no observed or potential outcomes**. Its existence does not identify a target causal effect. The source has `site_id` as primary key and two positions within each `pair_id`; one source row contains one baseline and one follow-up outcome.

### 2. Reconstruct pair and context effects

For each site compute `change = post - baseline`. For each pair subtract the control site's change from the treated site's change:

$$
\widehat d_p=(Y_{p,T,post}-Y_{p,T,pre})-(Y_{p,C,post}-Y_{p,C,pre}).
$$

This predetermined baseline adjustment is not the source of identification. The assigned comparison is randomized; it does not borrow a nonexperimental parallel-trends justification. Average the four high-context pair contrasts and the two low-context contrasts separately. The default observed context estimates are approximately **8.081225015** and **2.977534722**.

Use the predetermined context masses:

$$
\widehat\tau_S=\frac23\widehat\tau_H+\frac13\widehat\tau_L,
\qquad
\widehat\tau_{T,standardized}=\frac14\widehat\tau_H+\frac34\widehat\tau_L.
$$

Both are weighted effects for source sites under source randomization. Interpreting the second as an effect for unobserved target sites additionally requires a defined target, pretreatment effect modifiers, compatible treatment/control regimes, comparable outcomes and horizons, conditional mean-effect stability, and overlap. Balancing recorded context does not prove stability.

Compare `pair_contrasts.csv`, `stratum_estimates.csv` and `estimates.csv` under `output/reproduced/`:

| Source-data statistic | Default observed estimate | Paired/context-stratified SE | Source global-sharp-null p |
|---|---:|---:|---:|
| Source-weighted | 6.379994917 | 0.608487114 | 2/64 = .03125 |
| Target-standardized | 4.253457295 | 1.255313605 | 2/64 = .03125 |

No target outcome enters either calculation. Native checks reread the saved observed CSV and reproduce both estimates without instructor fields. A realized estimate need not equal instructor-known truth.

### 3. Diagnose weighting and uncertainty

The target-to-source density ratios are $.25/(2/3)=.375$ for high support and $.75/(1/3)=2.25$ for low support. With eight high and four low source sites, weights sum to 12 and squared weights sum to 21.375. Hence

$$
ESS=\frac{(\sum_i w_i)^2}{\sum_i w_i^2}=\frac{144}{21.375}\approx6.736842.
$$

This is weight concentration, not an independent-sample count or degrees of freedom. The experiment still has **six independent pairs**, with especially heavy target weight on only two low-support pairs.

For context masses $a_c$ and sample variance $s_c^2$ of the independent pair contrasts, use

$$
\widehat V(a)=\sum_{c\in\{H,L\}}a_c^2\frac{s_c^2}{P_c},\qquad(P_H,P_L)=(4,2).
$$

Source and target standard errors must use their respective masses. Pooling all pair differences into a single variance formula would change the stated context-stratified summary. No normal confidence interval is automatically supplied by six pairs, and ESS must not replace their count.

The variance estimator is conservative **in expectation** here because heterogeneous pair-average effects add dispersion. Its realized value need not exceed true variance; neither a single SE nor the oracle expectation proves small-sample normal coverage.

### 4. Keep the source null and instructor audit separate

Read `output/reproduced/global_sharp_null_distribution.csv`. For each statistic, enumerate all 64 legal assignments while **holding the realized source changes and predetermined weights fixed**. Count absolute statistics at least as large as observed, including ties and the actual allocation.

The null is that reform has zero effect at **every source establishment**. The target-weighted statistic still tests that source global sharp null. It does not test transport stability, the average effect in the unobserved target, or a target-only zero-effect null permitting arbitrary source effects. Exact randomization inference relies on the stipulated assignment and sharp null; it is not universally assumption-free.

Only after estimating from learner records should you open `data/instructor/`. Its potential outcomes and separate recomputed-assignment distribution verify source expectation 6 and target-standardized expectation 3.5. Under each alternative allocation this audit changes outcomes according to their known potential schedules; the Fisher calculation does not. Its exact design variances are instructor truth, not empirical SEs or information available in the target frame.

## Research Lab — Diagnose what weighting cannot repair

### 5. Separate known scenarios from observations

`output/diagnose/transport_scenarios.csv` contains three **instructor-known teaching worlds**:

| Scenario | High-support mass | High/low effects | Known effect |
|---|---:|---|---:|
| Source composition | 2/3 | 8, 2 | 6 |
| Target, stable conditional effects | 1/4 | 8, 2 | 3.5 |
| Target, changed implementation | 1/4 | 5, −1 | .5 |

The final two worlds have the same recorded target composition. Standardizing source effects still predicts 3.5 when the invented changed-implementation truth is .5. The diagnostic supplies that truth by assumption; it cannot discover it from the outcome-free target frame. Likewise, `transport_sensitivity.csv` plots assumed shifts, not an identification test or a fitted target response.

An additional, separate example places 20% of the target in a context absent from the source. Assuming its effect lies in $[-20,5]$ while the supported effect is 3.5 gives

$$.8(3.5)+.2[-20,5]=[-1.2,3.8].$$

This assumption-dependent identification range omits sampling uncertainty in 3.5. It is not a confidence interval. The main source/target frames contain both high and low contexts; this unsupported-context example adds a distinct hypothetical failure.

### 6. Audit missingness, measurement, and incidence

The worker-follow-up example is separate from the main experiment, whose source outcomes are complete. On an invented 0–100 scale, treated response .5 at respondent mean 70 yields a full mean range $[35,85]$. Control response .9 at respondent mean 60 yields $[54,64]$. Subtracting endpoints gives $[-29,31]$, despite respondent difference 10. The saved JSON and `worker_followup_bounds.csv` agree.

These are arbitrary-missingness bounds based only on the stated scale. They are neither Lee bounds, which require selection restrictions and concern a principal stratum, nor sampling confidence intervals. Do not set missing destination outcomes to zero.

| Additional original diagnostic | Interpretation |
|---|---|
| Site effects 1 and 9, initial employment 900 and 100 | Establishment mean 5 differs from initial-worker mean 1.8; choose the population before outcomes |
| Recorded gain 10 | Independently audited gain can be 10 or 0; two fields from one logging pipeline need not validate one another |
| Outputs $[80,40]$ followed by $[80]$ | Stayer mean rises 20, total output falls 40, and nobody improves |
| Pilots $[-4,-1,0,1,4,12]$ | Mean of all is 2, selected maximum is 12; a separately stipulated holdout of 2 is not a publication-bias estimator |
| Spillover model $100+4z-3e$ | Direct effect at fixed exposure is 4; all-treated versus none is 1, a different policy object |
| No-change versus bundle effect 12 | Both component vectors $(0,0,12)$ and $(-4,3,13)$ fit; the bundle does not identify the mechanism decomposition |

The separate monetary ledger has output value +12, setup 4, operations 2, pay transfer 2, worker burden 3 and amenity 1. Firm gain is $12-4-2-2=4$; worker gain is $2+1-3=0$; joint gain is 4. The pay transfer cancels in joint accounting; real costs remain. Main productivity units are not money without valuation assumptions. Leavers remain nested within the initial workforce and must not be added again as another workforce.

## Transfer: the capstone memo and constructive review

Complete the full [research memo template](research-memo-template.md), preserved from the supplied pack, and review it with the full [proposal rubric](proposal-review-rubric.md). Submit **four pages of main text plus a concise data-access and validation appendix**. The compact generated claim card is a drafting aid, not a replacement.

Choose a substantive organizational decision and two competing mechanisms with distinguishable implications. The five chapter pathways concern opportunity before application; tools versus reorganized work; learning and coordination costs; voice and unrecorded burdens; and why reforms travel differently. A distinct question is welcome if it has comparable clarity.

The memo must specify initial eligibility; observation, assignment and outcome units; treatment/control regimes and dates; an actual proposed comparison; source and target; modifiers and overlap; independent measurement; inference and missingness; worker/firm incidence; access status; validation and reproducibility; and a bounded next evidence product. Start with a decision and labor question, not a favored estimator or hypothetical perfect dataset.

The appendix includes a dated event schema, minimal records/keys and coverage, checked/pending/absent access, permissions and fallback, source-to-target comparison and checks for joins, timing, risk sets, recording revisions, response and missing destinations. Use synthetic mock records unless real access is authorized. No provider contact, restricted application, employee recruitment or data transfer is authorized by the form.

End with **the strongest claim the proposed design could support** and **a tempting claim it could not support**. A reviewer identifies the single most valuable missing comparison or measurement. Use the rubric's constructive categories rather than a numerical score, institutional prestige, significance or sample size alone. The deliverable is a defensible research plan, not a completed paper or publication guarantee.

## Methods Box: interpretation requires the right evidence

Source assignment, mechanism identification and target prediction impose different requirements. Conditional effect stability does not follow from balanced observables or similar untreated outcomes. A broader frame can help with composition while leaving treatment versions, peer exposure and measurement incomparable.

A complete pilot registry addresses selective reporting of trials, not all partnership selection or missing worker follow-up. A population-weighted effect must say whether it targets establishments or initial workers. Post-treatment headcount weighting changes the object. A source randomization test cannot validate unobserved target outcomes.

Pre-analysis plans can separate confirmatory claims, amendments and labeled exploration. Sensitivity calculations expose dependence on assumptions; they do not independently identify the missing effect. An access or data policy is not permission to upload or release confidential records. A synthetic rerun validates software and declared arithmetic, not a paper's empirical transport assumptions.

## Reading ladder and submission

The three required readings are selected Hoffman–Stanton (2025) synthesis/external-validity sections, Allcott (2015), and Bloom–Han–Liang (2024) as a claims audit. Allcott studies energy-conservation utility sites, not personnel data or random willingness to partner. Companion comparisons include DellaVigna–Linos for evidence bases and Hotz–Imbens–Mortimer for population versus treatment-component differences. The chapter provides canonical linked references.

Submit observed-source estimates and uncertainty; a precise explanation of both source sharp-null tests; one transport or missingness diagnostic; a run log and source/mode statement; and the full memo, appendix and response to a constructive criticism. State what the evidence does not yet support.

## Reproducibility and slide companion

`src/offline_lab.py` preserves the native constants, DGP and 67 assertions, including observed-CSV-only parity. `src/install-generated.py` stages into fresh storage and protects existing work. `src/smoke-checks.py` compiles, runs two actual native defaults plus seed 88771, audits saved files and checks unchanged refusal. `validation.json` mirrors native checks; the integration smoke report is separate.

The chapter and Week 10 slides use original figures from saved transport-scenario and missingness-range CSVs. Instructor scenarios are distinguished from realized estimates, and identification ranges from sampling intervals. Original-data replication is NOT RUN. The next step is the student's scoped project, not an eleventh lecture or an automatic publication/release.