Memo: Does DeepSeek-R1-Distill-Llama-8B recover PRCA on prompt v1?

Research memorandum — full-cohort vLLM baseline (Terrarium 5-tier ladder)

Author

Jack J. Burleson

Published

Invalid Date

Research question: Does deepseek-ai/DeepSeek-R1-Distill-Llama-8B recover group and interpersonal PRCA on prompt-v1 Terrarium tiers better than Llama-3.1/3.2 Instruct ((Binz & Schulz, 2024)) — and is its reasoning-distill format (chain-of-thought then answer) compatible with schema-valid CA JSON at scale?

Formal write-up: docs/llm_baseline_deepseek_r1_distill_v1.md
Cross-model memo: vllm_v1_cross_model_comparison.qmd

1 Answer, Response, + Summary of Results

Full-cohort vLLM run (n = 241; 1,205 prompts; parse ****1202/1205** = 99.8%; fp8 Marlin weight-only; throughput 1.34** samples/s; wall ≈ 450 s; export psych755_vllm_deepseek_r1_distill_llama8b_full_cohort_20260726_0324). Ingest normalizes SentencePiece artifacts and prefers text after </think> before JSON parse.

Short answer: Best pooled group MAE among the three v1 runs (5.22), with the most tier-stable error profile — transit does not induce Llama-3.1’s interpersonal collapse. Interpersonal MAE (5.73) is between Llama-3.2 (5.35) and Llama-3.1 (5.82). Band accuracies sit near chance (group 33.4%, IP 35.4%). Exact match remains rare (~6%). Three prompts failed schema parse (geo/transit/full incomplete rows in the metrics table). Still above Ridge ML floor (~4.49 group MAE).

1.1 Overall vs Llama instruct baselines

Table Table 1 compares DeepSeek-R1-Distill with the Llama instruct baselines.

Table 1.. DeepSeek-R1-Distill tier-level performance.

Metric (all tiers)	DeepSeek-R1-Distill-8B	Llama-3.2–3B	Llama-3.1–8B
MAE group	5.22	5.51	5.92
MAE interpersonal	5.73	5.35	5.82
Band group	33.4%	52.7%	28.8%
Band interpersonal	35.4%	30.0%	40.2%
Parse success	99.8%	100%	100%

1.2 Metrics by tier

Table Table 2 reports MAE and band metrics by tier.

Table 2.. Tier / n GT / MAE G.

Tier	n GT	MAE G	MAE IP	Band G	Band IP	Mean err G	Mean err IP
demos	241	5.12	5.93	31.1%	32.4%	−0.55	+1.25
employment	241	5.16	5.66	31.5%	36.1%	−0.43	+1.55
geo	240	5.30	5.86	33.3%	32.1%	−1.26	+0.99
transit	240	5.15	5.80	36.3%	35.0%	−0.71	−0.03
full	240	5.40	5.42	35.0%	41.7%	−1.38	−0.52

1.3 RQ verdicts

RQ1 (employment): Negligible group change; modest IP MAE improvement (−0.27).
RQ2 (transit): Essentially flat vs demos on both subscales; no IP catastrophe.
RQ3 (full): Best IP MAE and IP band in this model’s ladder; group MAE slightly worse than demos.

Conclusion. DeepSeek-R1-Distill-Llama-8B is the most robust prompt-v1 continuous scorer for group CA in this trio and the clearest evidence that transit text need not destroy interpersonal recovery. Tradeoffs: slower throughput, rare parse failures, and weaker group-band accuracy than Llama-3.2–3B. Prefer DeepSeek when the scientific claim is about MAE / tier stability; prefer Llama-3.2 when the claim is about coarse group-band classification.

Sources: data/vllm/deepseek-distilled/deepseek_v1.md · data/vllm/deepseek_v1.csv · docs/llm_baseline_deepseek_r1_distill_v1.md · github.com/Exios66/psych755-jjb

2 What questions or uncertainties remain?

Do the three parse failures share a prompt/tier pattern? Does forcing JSON-only decoding (no visible <think>) change MAE? Will v2/v3 packaging preserve DeepSeek’s tier stability while lifting band accuracy toward Llama-3.2’s group-band performance?

3 What other analyses pair with this memo?

Cross-model synthesis: vllm_v1_cross_model_comparison.qmd. Llama baselines: vllm_v1_llama31_8b.qmd, vllm_v1_llama32_3b.qmd.

References

Binz, M., & Schulz, E. (2024). Turning large language models into cognitive models. Proceedings of the Twelfth International Conference on Learning Representations. https://doi.org/10.48550/arXiv.2306.03917

--- title: "Memo: Does DeepSeek-R1-Distill-Llama-8B recover PRCA on prompt v1?" subtitle: "Research memorandum — full-cohort vLLM baseline (Terrarium 5-tier ladder)" author: Jack J. Burleson date: 2026–07-26 bibliography: ../references.bib --- **Research question:** Does `deepseek-ai/DeepSeek-R1-Distill-Llama-8B` recover group and interpersonal PRCA on prompt-v1 Terrarium tiers better than Llama-3.1/3.2 Instruct ([@binz2024]) — and is its reasoning-distill format (chain-of-thought then answer) compatible with schema-valid CA JSON at scale? **Formal write-up:** [`docs/llm_baseline_deepseek_r1_distill_v1.md`](../docs/llm_baseline_deepseek_r1_distill_v1.md) **Cross-model memo:** [`vllm_v1_cross_model_comparison.qmd`](vllm_v1_cross_model_comparison.qmd) --- ## Answer, Response, + Summary of Results Full-cohort vLLM run (**n = 241**; 1,205 prompts; parse ****1202/1205** = **99.8%****; `fp8 Marlin` weight-only; throughput **1.34** samples/s; wall ≈ 450 s; export `psych755_vllm_deepseek_r1_distill_llama8b_full_cohort_20260726_0324`). Ingest normalizes SentencePiece artifacts and prefers text after `</think>` before JSON parse. **Short answer:** Best **pooled group MAE** among the three v1 runs (**5.22**), with the most **tier-stable** error profile — transit does **not** induce Llama-3.1’s interpersonal collapse. Interpersonal MAE (**5.73**) is between Llama-3.2 (5.35) and Llama-3.1 (5.82). Band accuracies sit near chance (group **33.4%**, IP **35.4%**). Exact match remains rare (**~6%**). Three prompts failed schema parse (geo/transit/full incomplete rows in the metrics table). Still above Ridge ML floor (~4.49 group MAE). ### Overall vs Llama instruct baselines Table @tbl-v1ds-1 compares DeepSeek-R1-Distill with the Llama instruct baselines. | Metric (all tiers) | DeepSeek-R1-Distill-8B | Llama-3.2–3B | Llama-3.1–8B | |---|---:|---:|---:| | MAE group | **5.22** | 5.51 | 5.92 | | MAE interpersonal | 5.73 | **5.35** | 5.82 | | Band group | 33.4% | **52.7%** | 28.8% | | Band interpersonal | 35.4% | 30.0% | **40.2%** | | Parse success | 99.8% | 100% | 100% | : DeepSeek-R1-Distill tier-level performance. {#tbl-v1ds-1} ### Metrics by tier Table @tbl-v1ds-2 reports MAE and band metrics by tier. | Tier | n GT | MAE G | MAE IP | Band G | Band IP | Mean err G | Mean err IP | |---|---:|---:|---:|---:|---:|---:|---:| | demos | 241 | **5.12** | 5.93 | 31.1% | 32.4% | −0.55 | +1.25 | | employment | 241 | 5.16 | 5.66 | 31.5% | 36.1% | −0.43 | +1.55 | | geo | 240 | 5.30 | 5.86 | 33.3% | 32.1% | −1.26 | +0.99 | | transit | 240 | 5.15 | 5.80 | **36.3%** | 35.0% | −0.71 | −0.03 | | full | 240 | 5.40 | **5.42** | 35.0% | **41.7%** | −1.38 | −0.52 | : Tier / n GT / MAE G. {#tbl-v1ds-2} ### RQ verdicts - **RQ1 (employment):** Negligible group change; modest IP MAE improvement (−0.27). - **RQ2 (transit):** Essentially flat vs demos on both subscales; **no IP catastrophe**. - **RQ3 (full):** Best IP MAE and IP band in this model’s ladder; group MAE slightly worse than demos. **Conclusion.** DeepSeek-R1-Distill-Llama-8B is the **most robust prompt-v1 continuous scorer for group CA** in this trio and the clearest evidence that transit text need not destroy interpersonal recovery. Tradeoffs: slower throughput, rare parse failures, and weaker group-band accuracy than Llama-3.2–3B. Prefer DeepSeek when the scientific claim is about **MAE / tier stability**; prefer Llama-3.2 when the claim is about **coarse group-band classification**. *Sources:* `data/vllm/deepseek-distilled/deepseek_v1.md` · `data/vllm/deepseek_v1.csv` · `docs/llm_baseline_deepseek_r1_distill_v1.md` · [github.com/Exios66/psych755-jjb](https://github.com/Exios66/psych755-jjb) --- ## What questions or uncertainties remain? Do the three parse failures share a prompt/tier pattern? Does forcing JSON-only decoding (no visible `<think>`) change MAE? Will v2/v3 packaging preserve DeepSeek’s tier stability while lifting band accuracy toward Llama-3.2’s group-band performance? ## What other analyses pair with this memo? Cross-model synthesis: [`vllm_v1_cross_model_comparison.qmd`](vllm_v1_cross_model_comparison.qmd). Llama baselines: [`vllm_v1_llama31_8b.qmd`](vllm_v1_llama31_8b.qmd), [`vllm_v1_llama32_3b.qmd`](vllm_v1_llama32_3b.qmd).