CA persona prediction framework

1 Goal

Estimate whether LLM-predicted communication apprehension (CA) from demographic / contextual personas systematically errs by group membership. Predictions are compared to ground-truth PRCA subscale scores computed from Qualtrics Likert items.

2 Pipeline

Note

Private full-cohort inputs live in ../sibling_data/ (File A + File B stacked Prolific waves; File C Qualtrics; merge on Q0). Path resolution also accepts staged cloud copies (/tmp/sibling_data or CA_SIBLING_DATA). Excerpt fixtures under data/excerpts/ are for unit tests only — Posit Connect pages never display excerpt statistics (they use full-cohort runs or committed artifacts/posit_full_cohort/).

flowchart LR
  fileA[ProlificFileA] --> stack[StackWaves]
  fileB[ProlificFileB] --> stack
  stack --> join[JoinOnParticipantId]
  fileC[QualtricsFileC] --> join
  join --> clean[CleanAnalyticSample]
  clean --> eda[RQAlignedEDA]
  clean --> score[ScoreGroundTruthPRCA]
  score --> tiers[BuildTieredPersonaPrompts]
  tiers --> llm[OllamaOrOpenRouter]
  llm --> eval[JoinPredictionsToGroundTruth]
  eval --> out[MAEByTierAndGroupBias]

flowchart LR
  fileA[ProlificFileA] --> stack[StackWaves]
  fileB[ProlificFileB] --> stack
  stack --> join[JoinOnParticipantId]
  fileC[QualtricsFileC] --> join
  join --> clean[CleanAnalyticSample]
  clean --> eda[RQAlignedEDA]
  clean --> score[ScoreGroundTruthPRCA]
  score --> tiers[BuildTieredPersonaPrompts]
  tiers --> llm[OllamaOrOpenRouter]
  llm --> eval[JoinPredictionsToGroundTruth]
  eval --> out[MAEByTierAndGroupBias]

2.1 Research-question alignment

Tier / artifact	Research question
demos	Baseline demographics for stereotyping checks (Age, Sex, Country, Student; optional ethnicity/language)
employment	RQ1 — does employment improve accuracy / change bias?
geo	Intermediate place cue (country + survey lat/long)
transit	RQ2 — does transportation-use help, and is it used sensibly?
full	RQ3 — combined signal vs redundancy (+ open-text attitudes)
outputs/eda/ca_by_group.csv	Empirical CA means by sex / student status / employment / transit for bias interpretation

2.2 Secondary RQ — regular transit vs cohort CA

Observational contrast on the matched analytic sample (no LLM):

Piece	Detail
Exposure	Q26 weekly-or-more (4-8 or 8+ days/month)
Outcomes	Ground-truth group + interpersonal PRCA (6–30)
Tests	Welch t, Mann–Whitney, Cohen’s d, bootstrap CI
CLI / notebook	ca-personas transit-ca · notebooks/secondary_rq_transit_ca.ipynb
Write-up	docs/secondary_rq_transit_ca.md
Memo	memos/transit_riders_ca.qmd
Artifacts	outputs/transit_ca/ (descriptives, comparisons, distributions, results card)

2.3 Secondary RQ — geography predicts regular transit?

Piece	Detail
Features	Qualtrics LocationLatitude, LocationLongitude
Outcome	Same regular_transit flag from Q26
Model	Balanced Random Forest + stratified CV
Baselines	Chance / majority; country-only RF
CLI / notebook	ca-personas geo-transit-rf · notebooks/secondary_rq_geo_transit_rf.ipynb
Write-up	docs/secondary_rq_geo_predicts_transit.md
Memo	memos/geo_predicts_transit.qmd
Artifacts	outputs/geo_transit_rf/

2.4 Secondary RQ — CA scores predict regular transit?

Piece	Detail
Features	Ground-truth gt_group_ca, gt_interpersonal_ca (PRCA 6–30)
Outcome	Same regular_transit flag from Q26
Model	Balanced Random Forest + stratified CV; group-only / interpersonal-only ablations
CLI / notebook	ca-personas ca-transit-rf · notebooks/secondary_rq_ca_transit_rf.ipynb
Write-up	docs/secondary_rq_ca_predicts_transit.md
Memo	memos/ca_scores_predict_transit.qmd
Artifacts	outputs/ca_transit_rf/

2.5 Secondary RQ — wave-2 extended follow-ups

Piece	Detail
Experiments	Demographics · country · nested Q28|car · CA+Q28+car · country×car · Q27-among-riders · common-N · residual CA
Outcome	Same regular_transit (except Q27-among-riders high-intensity target)
Model	Mixed / categorical balanced Random Forests (RFs) + Welch strata where noted
CLI / notebook	ca-personas followup-experiments · notebooks/secondary_rq_followup_experiments.ipynb
Agenda	docs/research_memo_agenda.md
Write-up	docs/secondary_rq_followup_experiments.md
Artifacts	outputs/followup_experiments/

2.6 Secondary focus — transit prediction with mobility held out (TF1 / TF2)

Piece	Detail
Hold-out	Q26–Q29, Q20, Q21, Q19 excluded from features and persona text
TF1	demos + employment + geo (± CA) → regular_transit
TF2	demos + employment + geo + CA → Q26 ordinal / Q27 multiclass
Model	Balanced mixed-type Random Forest + stratified CV; LLM twin prompts (tf_* tiers)
CLI	ca-personas transit-focus
Write-up	docs/secondary_rq_transit_focus.md
Memo	memos/transit_focus_regular_and_intensity.qmd
Artifacts	outputs/transit_focus/ (+ persona_prompts/)

3 Ground-truth scoring

Group items: Q1–Q6. Interpersonal items: Q13–Q18.

• Likert labels map to 1–5.
• Comfort / low-anxiety items are reverse-coded as 6 - score.
• Each subscale is the sum of six items → range 6–30.
• Bands default to low (≤13), moderate (14–19), high (≥20).

4 Persona tiers

Cumulative information revealed to the model as natural-language, second-person digital-twin narratives (AI Terrarium framing (Sun et al., 2026) — “You are a …”), not structured questionnaire checklists:

Core ladder (RQ1–RQ3):

1. demos
2. employment
3. geo (approximate survey lat/long at 1 decimal; country not repeated)
4. transit (Q26/Q28 signal-first; skip rides-per-day when frequency is Never)
5. full — all tabular fields plus Qualtrics free-response attitudes

v3 ablations (same demos→employment→geo base, focused tip):

6. v3_rideshare — Q28 only (+ Q29 if used)
7. v3_public_transit — Q26 only (+ Q27 if used)
8. v3_voice — Q18.1 / Q19 open-text only (no structured transit)

Version map (v1 baseline → v2 packaging → v3 ablations): persona_prompt_versions.qmd.
Llama-3.1 full-cohort baseline (prompt v1): llm_baseline_llama31_v1.md.
Packaging details: persona_prompt_efficiency.md.
See prompts/system_prompt.md and prompts/examples/ for the system contract and sample prompts.

5 Evaluation metrics

Against ground-truth PRCA subscales (outputs/ground_truth/):

Metric	What it measures
MAE / exact match	Precision on the continuous 6–30 score
Band accuracy	Whether predicted low/moderate/high matches the participant band
Score distance	Absolute error; normalized as \(\|pred - gt\| / 24 \in [0, 1]\)
Band distance	Ordinal steps between bands (0 same, 1 adjacent, 2 low↔︎high); normalized / 2

# Place exports in ../sibling_data/ first (never commit them)
ca-personas prepare --join inner
ca-personas score-gt --join inner
ca-personas build-personas --tiers demos employment geo transit full
ca-personas run --provider mock --join inner

6 Providers

Tip

Set CA_LLM_PROVIDER to ollama or openrouter (see .env.example). Use --provider mock for offline dry runs and unit tests. Posit Connect Cloud renders load committed full-cohort vLLM exports and artifact tables, not mock predictions.

7 vLLM batch path

For offline GPU batch inference, persona prompts are exported to the digital-twin CSV schema (caseid, prompt) and scored with python -m inference.predict_vllm. Results (generated_text) are ingested back into the same CA prediction table used by evaluate_predictions.

python -m inference.export_prompts --output-dir outputs/vllm_prompts
python -m inference.predict_vllm \
  --prompt_csv outputs/vllm_prompts/prompts.csv \
  --result_csv outputs/vllm_results/results.csv
python -m inference.ingest_results \
  --result_csv outputs/vllm_results/results.csv

References

Sun, L., Li, Z., Khwaja, O., & Rogers, T. (2026). Designing digital twins for AI opinion simulation: How prompt framing and depth drive predictive fidelity. Extended Abstracts, Computational Methods Division, 76th Annual Conference of the International Communication Association.