# Week 2 lab: interview sequence, risk sets, and selective performance ## Economic question and learning objectives How can the order of evaluation and the rule for filling fixed hiring slots change what an organization observes? This lab turns that question into a data-construction exercise, a controlled simulation and a recruiting-study proposal. By the end, you should be able to: 1. define candidate-event keys and construct a predecessor within the correct session; 2. distinguish known-DGP recovery from validation of a paper's identifying assumptions; 3. explain what session fixed effects and clustered uncertainty do, and what they do not solve; 4. show why changing hired-worker mean output need not change anyone's potential productivity; 5. specify a baseline risk set, a credible comparison and an outcome available for that population. The primary reading is [Radbruch and Schiprowski (2025)](https://doi.org/10.1093/restud/rdae039), which examines interview assessments in admissions and consulting hiring. These are distinct settings. Read its institutional argument about sequence and independent quality assessment before claiming that interview position or a predecessor's decision supplies causal variation. A lagged yes/no decision is not itself randomized. For diagnosis and transfer, use [Hoffman, Kahn and Li (2018), Discretion in Hiring](https://doi.org/10.1093/qje/qjx042), and [Li, Raymond and Bergman (2026), Hiring as Exploration](https://doi.org/10.1093/restud/rdaf040). Use [Lee (2009)](https://doi.org/10.1111/j.1467-937X.2009.00536.x) to state a selection assumption, not to justify applying bounds whenever headcount is stable. ## Setup and provenance From the lab directory: ```bash ENV_NAME=research bash smoke.sh ``` The mode is **synthetic design reconstruction**, saved as `synthetic_design_reconstruction`, with `empirical_replication: false`. All generated observations are artificial. The paper's original interview microdata are proprietary; this implementation did not download or inspect the archive contents. No optional empirical reproduction was attempted. Synthetic recovery is not reproduction of a published coefficient. Allow approximately two hours, followed by a two-page memo. The starter uses only the Python standard library; the figure uses installed Matplotlib and the shared course style. The command needs no internet or restricted data. `--out .` targets the lab root. ## Reproduce: a known synthetic design Open `data/teaching/candidate-event-schema.csv` and `synthetic_interview_events.csv`. The seed is 20260921. There are 96 independent sessions, each with 12 candidates. Candidate order is shuffled within each session. A candidate has a pre-interview score and a subjective rating; neither is realized job productivity or worker welfare. The teaching rule sets the own-score slope to 0.8 and the mean predecessor-score slope to −0.25. Ratings also contain a session intercept, a mean-zero session-specific predecessor-slope deviation and individual noise. These parameters are instructor choices, not empirical estimates attributed to the article. `transfer/data-notes.md` documents the distributions. The estimator excludes the first slot in each session, whose predecessor is missing. It demeans within session and estimates slopes on own and previous scores. Open `output/reproduced/synthetic_sequence_estimates.csv` and select `lag_definition = previous_score`. Check the 1,056 usable observations and 96 sessions. The table puts the known synthetic parameter next to each estimate; it also reports an i.i.d. SE, a session-clustered SE and a normal-approximation interval. The chapter's more general teaching equation also allows slot effects. This simplified DGP sets those effects to zero and the lab regression omits slot dummies. It retains session fixed effects and the own/previous score slopes; do not describe it as estimation of every term in the chapter specification. The clustered sandwich uses the declared finite-sample factor $$ \frac{G}{G-1}\frac{N-1}{N-G-2}, $$ where $G$ is the number of sessions and $N$ the usable interview count. The residual degrees of freedom subtract the session effects and two slopes. The displayed interval is estimate ±1.96 times the clustered SE. This is a many-session teaching approximation, not the article's original inference routine or exact randomization inference. The i.i.d. calculation is a comparator; it need not be numerically smaller in every realization. Tasks: state the outcome, unit, assignment pool, regressors and excluded observations. Explain why the DGP makes the scores and order suitable for this exercise. Then name evidence a real organization would need to support that interpretation. Clustering cannot supply absent variation, common support or credible assignment. ## Diagnose ### Reset the lag at session boundaries Compare `previous_score` with `wrong_global_previous_score`. The valid construction has 96 first-slot missing values and no cross-session links. The deliberately incorrect construction uses the previous CSV row: it makes 95 false links and incorrectly expands the regression sample to 1,151. Inspect `output/reproduced/lag-boundary-diagnostics.csv` and the figure `output/transfer/synthetic-lab-diagnostics.png`. A coding error remains a construction failure even when its coefficient is close to the valid estimate. Do not decide that the lag is valid by comparing significance or point estimates. The first slot's missing predecessor is not a zero-quality person. For a prospective employer dataset, specify rules for tied event times, cancellations, replacement candidates, long breaks and resumed sessions. These are new data-design questions: the synthetic file contains slots, not real timestamps, and this lab makes no claim that original hiring files contained timestamps. ### Keep analyst labels separate from synthetic truth Start with `data/teaching/synthetic_selective_labels_analyst.csv`. Only the two hires under the observed rule have job-output labels. For the four rejected candidates, output is missing. Do not fill those values with zero or use the instructor file as an extra predictor. After stating what the analyst cannot learn, inspect `data/instructor_only/synthetic_selection_truth.csv`. This contains artificial potential job outputs 10, 9, 8, 7, 6 and 5, and selection indicators under two rules. The observed rule hires candidates {0,1}; the alternative hires {0,2}. Mean output changes from 9.5 to 9, and total output from 19 to 18, while no candidate's potential output changes. `output/transfer/synthetic_fixed_slot_comparison.csv` labels these calculations **instructor truth, not estimated**. The alternative-policy mean is not identified from the analyst file. The directory split is an instructional separation, not protection for real personal data; all records are synthetic and available in the repository. Show which candidate enters and which leaves the selected set. There is one change in each direction. Stable headcount therefore does not establish one-direction selection monotonicity; this example violates that assumption. Do not run Lee-style bounds merely because both rules hire two people. Distinguish a selected-worker mean, a change in total output and an individual causal productivity effect. ### Recover the funnel identity without changing cohorts Open `data/teaching/hypothetical_nested_funnel.csv`: application → review → interview → offer → acceptance → start → retained at 90 days, with counts 100→60→20→10→8→7→5. The conditional-stage product equals 5/100 = 0.05 for this hypothetical nested cohort and completed horizon. The smoke test rejects a non-nested example, 10→12→2. Construct a second failure using denominators from different application months. Explain why starts per initial application is different from starts per accepted offer. A policy may change who reaches a later stage, so its conditional transition-rate differences are not automatically causal mechanisms. ## Transfer: a two-page recruiting study memo Use `transfer/memo-template.md`. Choose one intervention: documented test overrides, additional reviewer capacity, randomized interview ordering, referral-program introduction, or randomized marginal trial invitations. State the rule and information available when it is applied. Your memo must define the original applicant or vacancy risk set and the assignment unit; a primary outcome available for that baseline population; stage-specific denominators and horizons; missing performance among rejected applicants; a credible comparison and its main threat; dependence/interference and inference; and data access as public, restricted or requiring a partnership. End with one limited economic conclusion. For a test/discretion design, distinguish a rollout from randomized assignment of authority, and tenure from productivity. For a referral design, distinguish assignment to the program from actually being referred. For exploration, log known invitation probabilities and measure the trial outcome directly. Historical eventual-hiring labels are not universal productivity labels, and retrospective evaluation is not a prospective productivity RCT. A prediction algorithm cannot repair absent outcomes or unsupported counterfactuals. ## Submission Submit the source/mode ledger, one annotated candidate-event schema, the lag diagnostics and synthetic coefficient table with the cluster convention, the fixed-slot calculation with its truth warning, the funnel identity and a rejected counterexample, and the two-page transfer memo. Add a short run log with the command, seed and any changes. The assessment emphasizes construction, selection and identification reasoning; it does not reward presenting simulated coefficients as empirical findings.