Research memo agenda — extended secondary experiments

Post-hoc memorandum plan aligned to existing transit / CA methods

Project: PSYCH 755 — CA persona / PRCA framework
Author: Jack J. Burleson
Date: 2026-07-25

1 Purpose

After the primary persona-tier LLM pipeline and the first wave of secondary Random Forest / Welch analyses, several open questions remained at the ends of the published memos (geo → transit; CA → transit; Q27/Q28; covariate follow-ups). This agenda lists the additional research memoranda designed to answer those questions with the same methodological family: matched File A/B/C cohort, Q26 weekly+ regular_transit outcome (unless noted), stratified CV balanced Random Forests, and Welch / Mann–Whitney contrasts where appropriate.

2 Already completed (wave 1)

Memo Finding (seeded)
Transit riders → CA Regular riders lower CA (group d ≈ −0.46)
Geo → transit Lat/long AUC ≈ 0.551
CA → transit Group+IP CA AUC ≈ 0.590
Car / employment / rideshare Rideshare ≫ car ≫ CA ≈ geo ≫ employment
Q27 / Q28 traditional ML Q28 AUC ≈ 0.762; Q27 ≈ 0.589

3 Extended agenda (wave 2 — implemented)

# Memo Open question answered CLI / module Primary result (seed 42)
1 memos/demographics_predict_transit.qmd Unused Prolific demos; student status named open followup-experiments --experiments demographics AUC ≈ 0.618 (Age dominates)
2 memos/country_predicts_transit.qmd Country vs raw coordinates --experiments country AUC ≈ 0.552 ≈ geo
3 memos/q28_conditioned_on_car.qmd Does Q28 retain lift after car? --experiments nested_q28_car Q28 alone 0.665 → Q28+Q21 0.730 (n=149)
4 memos/ca_mobility_joint_predicts_transit.qmd Psych vs mobility importance (complete-case) --experiments ca_q28_car Joint AUC 0.736; CA alone weak on this subset
5 memos/country_car_predicts_transit.qmd Country × car interaction --experiments country_car Joint AUC 0.699 > either alone
6 memos/q27_intensity_among_riders.qmd Does Q27 help within regular riders? --experiments q27_among_regular Best AUC ≈ 0.549 — weak
7 memos/common_n_head_to_head.qmd AUC ranks confounded by missingness? --experiments common_n On n=139, Q28 still first (0.659)
8 memos/residual_ca_after_rideshare.qmd Third-variable / residual CA after Q28 --experiments residual_ca_q28 CA adds only +0.021 AUC over Q28
9 memos/mi_head_to_head.qmd MI restore demos/CA vs attenuate Q28? --experiments mi_head_to_head Q28 lead preserved (0.762); demos/CA not restored

4 Methods shared across wave 2

5 vLLM digital-twin memos (wave 1 — prompt v1)

Live GPU baselines are documented separately in docs/llm_vllm_memo_agenda.md:

Memo Headline
memos/vllm_v1_cross_model_comparison.qmd DeepSeek best group MAE; 3B best group bands; 8B IP collapses at transit; 70B collapsed
memos/vllm_v1_llama31_8b.qmd Llama-3.1-8B cautionary baseline
memos/vllm_v1_llama32_3b.qmd Llama-3.2-3B Instruct lower MAE than 8B
memos/vllm_v1_deepseek_r1_distill.qmd DeepSeek-R1-Distill tier-stable; best group MAE
memos/vllm_v1_llama33_70b.qmd Llama-3.3-70B mode-collapse cautionary case
memos/vllm_v2_v3_evaluation.qmd v2/v3 GPU runs: DeepSeek improves; collapse combination-specific
memos/live_llm_stereotyping_slices.qmd Real DeepSeek v2 stereotyping slices (Sex/Student/Employment)

6 Intentionally deferred (future work; need further live LLM or new data)

  • v2 / v3 canonical refresh — v2 is now GPU-evaluated on Llama-3.1-8B, Llama-3.2-3B-Instruct, and DeepSeek-R1-Distill-8B (v2_enhanced; exports/v2/), and v3 greedy 8-tier ablations on Llama-3.1, 3.2-3B-Instruct, and 3.3-70B (exports/v3/). Remaining: canonical v3_enhanced decode refresh, base-3B v2/v3, and 70B under large_model — see persona_prompt_versions.qmd
  • Completed since wave 1: live-export TreeSHAP / surrogate SHAP (memos/feature_predictive_power_ml_llm.qmd) and real stereotyping slices (memos/live_llm_stereotyping_slices.qmd)
  • Richer / MNAR-sensitive MI of car items (wave-2 MI head-to-head is implemented; see memos/mi_head_to_head.qmd)
  • City/density GIS layers beyond Qualtrics lat/long
  • Ordinal multiclass Q26 models (threshold sensitivity exists in transit-CA module)

7 Reproduction

ca-personas followup-experiments --join inner --seed 42
pytest tests/test_followup_experiments.py
quarto render