# Week 9 lab: factorial reform evaluation and work design

## Learning objectives

Define a reform's components and outcome horizon; reconstruct all eight factorial contrasts and their dynamics; distinguish realized estimates from instructor-known truth; explain which sharp null the exact test evaluates; and design a reform evaluation with independent quality, worker conditions, spillovers and attrition.

**Executed mode: original synthetic design reconstruction; not empirical replication.** Every number, establishment and treatment protocol is invented. Generated results carry `ORIGINAL_SYNTHETIC_FACTORIAL_DESIGN_NOT_A_PAPER_REPLICATION`. Atkin et al. (2017) motivates implementation incentives, and Bloom et al. (2013, 2020) motivates following practices and persistence. Their empirical protocols and estimates are not reproduced. The Pakistan sequence is not a fully crossed tool-by-permanent-compensation experiment.

## Bridge: assignment, implementation, and sustained outcomes

A reform is more than a tool license. A workflow may need review capacity, competent users, implementation support and maintenance. The organizing sequence is **assignment → implementation → sustained routine → verified outcome → incidence**. A positive bundle effect neither establishes that each component helps alone nor identifies how much of the effect operates through recorded use.

Read the [README](README.md) and run from this directory:

```bash
ENV_NAME=research bash smoke.sh
```

For a separate exercise, use a fresh path:

```bash
conda run -n research python src/offline_lab.py --output /tmp/org8-week9-student-run --seed 88771
```

The numerical runner refuses a nonempty target with exit code 2. Do not erase existing work to satisfy that guard. The README explains safe staged regeneration into the documented lab.

## Field Core — Reproduce the factorial comparison

### 1. Reconstruct the initial population and timing

Read `data/teaching/establishments.csv`, `establishment_period.csv` and `teaching-schema.csv`. The roster has 12 establishments and key `unit`; the panel has 60 rows and key `(unit, period)`. Period 0 is a fixed pretreatment baseline. Periods 1–4 are common follow-up horizons, not individually chosen adoption dates.

There are three independent baseline blocks of four. Each block receives exactly one establishment in each regime: $00$ neither, $10$ tool only, $01$ redesign only, $11$ both. The arm order is always $(00,10,01,11)$. Within-block allocation is uniform over 24 permutations, independent across blocks, yielding 13,824 possible allocations.

The population and all potential outcome functions are fixed before a separate assignment random stream. Centered readiness heterogeneity affects untreated change and treatment responses, but its latent rank is absent from learner files. Baseline output is observed. The core toy assumes no cross-establishment interference and complete initial-establishment follow-up. Neither condition is guaranteed by random assignment in a real workplace.

`tool_use_fraction` depends on assignment, readiness, redesign and period. It is a post-assignment implementation measure, not a randomized mediator. `worker_conditions_index` is a separate outcome in arbitrary units, not monetized welfare or a substitute for every worker outcome. Do not replace assignment with observed use in the main causal contrast.

### 2. Compute all eight contrasts

Within each block and period, subtract each establishment's fixed period-0 output from its observed verified output. Put the resulting changes into four assigned cells. Baseline adjustment can improve precision; random assignment, not subtracting the baseline, supplies identification.

For a vector of block cell changes $(\Delta_{00},\Delta_{10},\Delta_{01},\Delta_{11})$, use the following comparisons and average them equally over the three blocks:

| Exported contrast | Block-level comparison | Final instructor-known population effect |
|---|---|---:|
| `tool_without_redesign` | $\Delta_{10}-\Delta_{00}$ | −4 |
| `tool_with_redesign` | $\Delta_{11}-\Delta_{01}$ | 9 |
| `redesign_without_tool` | $\Delta_{01}-\Delta_{00}$ | 3 |
| `redesign_with_tool` | $\Delta_{11}-\Delta_{10}$ | 16 |
| `bundle` | $\Delta_{11}-\Delta_{00}$ | 12 |
| `interaction` | $\Delta_{11}-\Delta_{10}-\Delta_{01}+\Delta_{00}$ | 13 |
| `tool_main` | Half the sum of the two tool contrasts | 2.5 |
| `redesign_main` | Half the sum of the two redesign contrasts | 9.5 |

These equal-weighted factorial marginal effects are not effects at every value of the other factor. The bundle is the sum of the two factorial marginal effects, or the sum of both standalone effects and the interaction. A trial observing only $00$ and $11$ cannot distinguish the decompositions $(-4,3,13)$ and $(5,7,0)$, both of which give bundle 12. Co-adoption does not provide the missing comparison.

The final column is known only because this world is specified. First estimate from learner files; then open `data/instructor/` to audit the truth. No potential-outcome or readiness field belongs in a learner prediction or estimation table.

### 3. Report the realized estimates and limited block uncertainty

For three block contrasts $d_b$, compute their mean and

$$
\widehat V=\frac{1}{3(3-1)}\sum_{b=1}^3(d_b-\bar d)^2,
\qquad \widehat{SE}=\sqrt{\widehat V}.
$$

Compare your calculations with `output/reproduced/final_contrasts.csv`:

| Contrast | Default-seed estimate | Block SE |
|---|---:|---:|
| Tool without redesign | −0.600000000 | 0.832666400 |
| Tool with redesign | 11.083333333 | 2.455153310 |
| Redesign without tool | 4.566666667 | 3.655741299 |
| Redesign with tool | 16.250000000 | 3.053413391 |
| Bundle | 15.650000000 | 3.386738844 |
| Interaction | 11.683333333 | 1.669663972 |
| Factorial tool marginal | 5.241666667 | 1.632057325 |
| Factorial redesign marginal | 10.408333333 | 3.262965590 |

A realized estimate need not equal its population effect. The variance estimator is conservative in expectation when true block contrasts differ. Here centered heterogeneity leaves block-average effects equal, and the oracle expected variance estimate equals the actual design variance. This equality does not make a particular three-block standard error stable or justify a normal approximation. Do not use 60 panel rows or 13,824 allocations as independent sample size.

### 4. Preserve all four follow-up periods

`output/reproduced/dynamic_estimates.csv` contains 32 rows: eight contrasts at each of four horizons. Plot at least the standalone tool, standalone redesign and bundle estimates. If you display instructor truth, use a separate, clearly labeled panel from `data/instructor/dynamic_arm_effects.csv`. Do not draw narrow normal confidence bands from three blocks.

| Period | Known tool-only effect | Known redesign-only effect | Known interaction | Known bundle effect |
|---|---:|---:|---:|---:|
| 1 | −6 | −1 | 4 | −3 |
| 2 | −5 | 1 | 8 | 4 |
| 3 | −4 | 2 | 11 | 9 |
| 4 | −4 | 3 | 13 | 12 |

These paths are invented finite-population effects. An early negative result need not reverse in another setting. State the decision horizon, setup cost, recurring costs, discount factor and terminal assumptions before turning an endline output effect into a return calculation.

## Research Lab — Diagnose identification and measurement

### 5. Name the exact null before interpreting a p value

`output/reproduced/global_sharp_null_distribution.csv` fixes each establishment's realized, baseline-adjusted final outcome and reassigns all four cells only within its original block. The two-sided p value counts absolute statistics at least as large as observed, including ties and the actual allocation.

The **global sharp null** says every establishment has the same outcome under all four regimes. Under the default seed, the bundle statistic gives $176/13{,}824\approx .0127314815$. The interaction statistic gives $3{,}072/13{,}824=2/9$. Both use the same global null. **The latter is not a test only of zero interaction while permitting nonzero main effects.** Such a weaker null does not generally impute the missing potential outcomes needed here. The lab supplies no confidence interval for average complementarity. The other six contrasts have blank p values because no exact test is computed for them.

`data/instructor/recomputed-potential-outcome-distribution.csv` performs a different calculation: it recomputes outcomes under every allocation and saves all eight estimates and their block variance estimators. `design_audit.csv` verifies design-unbiasedness and expected variance properties. Oracle variance is possible only with all four potential schedules, which empirical records do not reveal. It must not replace empirical uncertainty.

### 6. Check the outcome and resource accounting

Read `output/diagnose/original_examples.csv`, `complementarity_cells.csv`, `maintenance_path.csv` and `analytical-examples.json`.

| Original diagnostic | Calculation and implication |
|---|---|
| Four cells | $(100,96,103,112)$ gives bundle 12, interaction 13; a bundle alone does not establish complementarity |
| Draft/review bottleneck | Capacities $(10,10),(20,10),(10,16),(20,16)$ yield throughput $10,10,10,16$ |
| Implementation incidence | Material saving 12, real worker burden 4, transfer 5: owner 7, worker 1, joint gain 8 |
| Finite-horizon value | Benefits $(-3,4,9,12)$, discount .95, setup 20, zero recurring costs and terminal value yield −1.74955 |
| Maintenance | Initial stock 1 with 20% depreciation reaches .4096 at period 4; investment .2 each period maintains 1 but is costly |
| Practice retention | Keeping the first two values in $(8,1,1,0)$ retains half the count and 90% of additive value; additivity is illustrative |
| Rates and time | Output 200 with recorded hours 100→80 raises the output rate 25% without raising output; investigate real savings versus logging |
| Email | $8+30=6+32=38$ hours after reallocation; $6+30=36$ after reduced total time; neither identifies output |
| Selected stayers | $(80,100,120,140)$ to $(120,140)$ raises the mean 110→130 while total output falls 440→260, with no individual gains |
| Recording | Logged actions 100→150 with independently audited output 100→100 is a recording-only change |
| Equivalence | Estimate .1 with SE .2 gives a 90% normal interval inside prespecified ±1; SE 2 does not, although both zero tests are nonsignificant |
| Spillovers | In $b+2z+4e$, the fixed-exposure direct effect is 2, while full rollout versus none is 6 |
| Co-adoption | A common management variable can produce perfectly correlated practices and an output gap without identifying an interaction |

The equivalence example is a separate normal illustration, not the interval procedure for this three-block experiment. Complementarity depends on the outcome scale and feasible standalone regimes. A drafting-capacity gain need not raise final deliverables, and lower task time can be valuable without increasing output. Measure quality, burdens and the disposition of saved time.

## Transfer: a reform proposal and component/outcome ledger

Use the [reform-evaluation template](transfer-memo-template.md). Submit two pages plus one ledger. Choose one feasible reform and specify its initial eligible units, decision-maker, treatment version, control regime, assignment or identifying comparison, observed combinations and primary contrast. Include two competing mechanisms with distinguishable predictions, not just more controls.

Record announcement, assignment, delivery, first use, support withdrawal, maintenance and crossover dates. Include an independent quality measure, verified output, worker conditions, paid/unpaid time, setup and recurring resources. Distinguish firm gains, worker burdens, transfers and joint resources; avoid adding condition-index units to monetary values.

Draw a spillover map showing collaborators, supervisors, shared review capacity or infrastructure. State how an individual pilot differs from full rollout. Retain initial establishment and worker cohorts, and specify follow-up for exits, closure, survey nonresponse and destinations. An establishment closure can imply zero establishment output under a defined outcome; it does not imply zero income or welfare for every former worker. Missing destination outcomes must remain missing.

Name one result that would change the decision and one measurement that would separate the proposed mechanisms. Disclose actual access rather than assuming that confidential employer records or linkages exist.

## Methods Box: claims supported by the comparison

An offer, access, competence, use and a maintained routine differ. Post-assignment use does not inherit randomization, and first-stage evidence does not by itself identify mediation. Bundled consulting and sequential implementation experiments answer different questions from the fully crossed toy design.

Assignment fixes the legal inference support. Under nonrandom rollout, cluster corrections or modern staggered estimators do not validate the untreated trend, arbitrary reversals or interference. Crossover changes treatment histories. A nonsignificant pretrend or zero-effect test does not establish a counterfactual trend or equivalence.

Attrition and changes in recording can imitate production gains. Preserve the initial cohort, dated measurement definitions and an independent audit endpoint. Any selection correction or bound needs its own assumptions. Use the block SE as a limited summary and the exact test only for its stated global sharp null; neither identifies causal effects of actual use.

## Reading ladder

Read Bloom et al. (2013) for assigned assistance and implementation, Atkin et al. (2017) for sequential implementation incentives, and Brynjolfsson–Li–Raymond (2025) for staggered AI rollout within a production workflow. Use Bloom et al. (2020) for persistence and Bloom–Han–Liang (2024) for the hybrid offer, crossover and outcome interpretation. The chapter provides canonical linked references. These studies motivate the questions; no original estimates are reproduced here.

## Exercises and submission

1. Reconstruct all eight final estimates and block SEs from learner data, and explain the weights for factorial marginal effects.
2. Draw a four-period figure with realized estimates and separately labeled instructor truth. State why you did not use narrow normal bands.
3. Write the global sharp null in words. Explain why its interaction statistic is not an interaction-only test.
4. Reproduce two diagnostic calculations, including one measurement or incidence example; identify the additional data needed in an application.
5. Submit the reform ledger, eight-contrast table, dynamic figure, short inference/measurement diagnosis, reproducibility run log and two-page proposal.

## Reproducibility and slide companion

`src/offline_lab.py` preserves the supplied DGP, eight contrasts, four horizons and 82 checks. It exports explicit artifacts into a fresh directory. `src/install-generated.py` stages safely into the documented lab; `src/smoke-checks.py` executes compilation, fresh seeds, saved-file audits and unchanged-file refusal. `checks.json` and mirrored `validation.json` differ from the later integration smoke report.

The Week 9 chapter and slide deck contain the original dynamic-comparison and bottleneck/maintenance figures generated from these CSVs. Instructor truth is labeled separately from realized estimates. No original microdata or estimation code was downloaded or executed, and the guaranteed route requires no network, new dependency or private records.