# Week 3 lab: measurement is not production ## Economic question and learning objectives A dashboard change reduces recorded nonproductive time. Did productive work increase, was time reclassified, did support activity change, or did the job become more costly for workers? This lab distinguishes an identified effect on a measured outcome from an interpretation about production or welfare. By the end, you should be able to: 1. build a center-assigned worker panel and specify an equal-center-weighted estimand; 2. enumerate the actual assignment support of a teaching design and interpret a sharp-null test; 3. separate time accounting, recording changes, latent incidents and signal uncertainty; 4. explain how correlated raters and own-record inclusion affect information; 5. propose a measurement-validation study with two distinct outcome families and an explicit worker-welfare margin. The design anchor is [Bernstein and Li (2026 issue; online 2025)](https://doi.org/10.1287/mnsc.2022.02395). Read the paper's assignment and outcomes sections: it gives employees direct access to existing own and peer performance information. It does not isolate the two information components or directly manipulate worker ability. Diagnosis also uses the chapter's Kahn–Lange learning distinction, evaluator-heterogeneity discussion and multirater evidence; the toy routines are not those papers' full estimators. ## Setup and scope From the lab directory, run: ```bash ENV_NAME=research bash smoke.sh ``` This executes **Mode A: synthetic design reconstruction**, with `empirical_replication: false`. The required provenance marker is **`SYNTHETIC_TEACHING_NOT_PUBLISHED_ESTIMATES`**. The 76 workers, 1,216 rows, 480-minute budget, effects and noise processes are invented. The optional original-release **Mode B** was omitted: a supplement listing was documented, but no original data or code were downloaded or executed. This is not an empirical replication success or an empirical failure hidden by simulation. The reconstruction needs only Python's standard library. The figure uses the repository's existing Matplotlib setup. Smoke regenerates results in two fresh temporary directories and compares all generated bytes with the canonical run; it does not merely read a saved success flag. ## Reproduce: a declared synthetic assignment design ### Population, exposure and estimand Open `data/teaching/center-worker-schema.csv` and `center_worker_time.csv`. There are eleven centers with unequal baseline sizes, totaling 76 workers. Each worker has eight pre-periods and eight post-periods, with no attrition or movement in this constructed panel. Exactly four centers are assigned treatment; assignment is constant within centers and over time, while exposure begins at period zero. The primary outcome is **nonproductive minutes**, chosen before inspecting results. Productive minutes, support minutes and an independently generated quality measure are secondary diagnostics. The quality measure has separate recording noise but shares worker/center components, so “independent” does not mean statistically uncorrelated with everything else. Compute each center's mean outcome after minus before. Then average those changes across the four treated centers and subtract the average across the seven controls. Every center has equal weight. `output/reproduced/center_changes.csv` and `center_randomization_results.csv` report the construction. A worker-weighted or row-weighted comparison changes the target when center sizes differ; do not label it the same estimand. The toy DGP assigns a −12-minute direct change to nonproductive time and a +12-minute change to support, with no direct effect on productive time or audit quality. Those are known synthetic settings, not the expected exact values of noisy estimates or magnitudes from the paper. The full latent quantities are kept in `data/instructor_only/center_worker_truth.csv` and omitted from the analyst panel. ### Exact assignment enumeration The declared teaching design chooses four of eleven centers uniformly, so it has 330 allowed assignments. Under the **sharp null of zero effects for every center**, hold the observed center changes fixed and recompute the statistic for each assignment. The two-sided Fisher p-value is the fraction with absolute statistic at least as large as the observed absolute statistic, including the observed assignment. Inspect `primary-randomization-distribution.csv` and `output/transfer/synthetic-center-inference.png`. This unrestricted support belongs to the simulation. The original study describes company/union constraints on the number of employees affected; an empirical reconstruction must verify its actual support and assignment probabilities. Sharing the counts four and eleven does not establish an identical randomization design. Do not shuffle worker-period rows. Do not describe the sharp-null p-value as a general test of a zero average effect. A reference Neyman SE is saved, but eleven centers do not automatically justify a normal approximation. Secondary p-values are unadjusted; the exercise does not authorize choosing the primary outcome after looking at them or declaring equivalence from a large p-value. Tasks: name the recipient of information, assignment unit, analysis population, outcome and weight. Explain which parts are known by construction and which a real study must document. ## Diagnose ### 1. Time accounting versus production Check that productive, nonproductive and support minutes are positive and sum to 480 for every row. Their estimated contrasts also sum to zero. A reduction in nonproductive time can be absorbed by support, productive work or both. Accounting alone tells you neither which activity creates value nor whether workers are better off. Compare the separately measured quality result without imposing the time budget on its arbitrary score units. ### 2. A changed recording rule Open `output/transfer/measurement_only_counterfactual.csv`. Each of 150 synthetic tasks has an old score, a new score exactly five units higher and one unchanged blinded audit value. The underlying task and error realization are fixed within the pair. Compute the paired difference and explain why this constructed rubric shift is not a five-unit production gain. It is a measurement identity, not an observational causal estimate. An audit is another measure with its own coverage and error, not universal ground truth. ### 3. Learning about fixed versus drifting productivity `posterior-arithmetic.csv` checks a prior mean/variance of 0/1 and signal 2: noise variance 4 gives posterior mean/variance 0.4/0.8; noise variance 1 gives 1/0.5. More precise information changes the weight on the signal; it need not change productive capacity. `learning_uncertainty.csv` follows twenty updates with prior variance 1, signal-noise variance 1 and fixed signal value 1, comparing process variance 0 with 0.20. All Gaussian assumptions and parameters are hypothetical. Fixed-type uncertainty shrinks as signals accumulate. Positive process variance replenishes uncertainty before each update. Neither curve estimates an employer's actual learning speed or observes worker ability. ### 4. Several raters need not give independent information In `correlated_rater_noise.csv`, unit error variance and common pairwise correlation give mean-error variance $\rho+(1-\rho)/n$. Four raters with correlation 0.75 yield 0.8125, versus 0.25 with independent errors. Explain why a shared bias is a separate calibration problem and why greater reliability does not establish construct validity. ### 5. Detection is not incidence `detection_not_incidence.csv` assumes no false positives and multiplies latent prevalence by detection probability. The latent rate falls from 0.10 to 0.06; detection rises from 0.25 to 0.75; recorded incidence rises from 0.025 to 0.045. State the extra observation needed to distinguish better detection from more underlying misconduct in real records. ### 6. Own-record leakage `own_record_leakage.csv` enumerates all sixteen equally weighted independent four-period noise patterns with values −1 or +1. An average including the focal record has covariance 0.25 with that record's noise; the average of the other three has covariance zero. If a score also contains a persistent type, this interpretation assumes type is independent of the record errors. Serial dependence, shared shocks or endogenous assignment can invalidate the simple leave-out conclusion. Removing mechanical own-record covariance is not the same as identifying a causal supervisor effect, cross-fitting, or a general variance-component correction. ## Transfer: measurement-validation proposal Complete `transfer-memo-template.md`: two pages of design plus a one-page data dictionary. Choose one monitoring, evaluation or feedback intervention and identify the decision-maker. Define baseline eligibility and the assignment process; map who sees the information among workers, supervisors and researchers. Choose a primary measured performance margin and a distinct, independently recorded outcome family. Add a worker-welfare measure and confidentiality plan. Name one alternative explanation and a prediction that discriminates between it and your favored mechanism. Address attrition, task reassignment, interference and inference at the intervention's actual unit. State what remains unidentified even under randomization. Do not negotiate a new firm data agreement for this assignment. Classify proposed inputs as public, restricted, requiring a partnership, or not yet available. A public code/supplement page does not resolve access to original records. ## Submission Submit the schema and source/mode ledger; center changes and primary randomization result; an explanation of the time-use accounting; the five short diagnostic calculations; the study proposal/data dictionary; and a run log with command, seed and modifications. Label every synthetic table or figure. Successful arithmetic is a starting point for research judgment, not evidence about the paper's empirical magnitude or workers' welfare.