llmsa: Measurement System Analysis for Pipelines Built on Large Language Models

Treats a pipeline built on a large language model (LLM) as a measurement system and validates it with the tools that analytical laboratories already use: crossed precision studies with variance components (gauge repeatability and reproducibility), attribute agreement with kappa statistics, bias against a reference, Plackett-Burman screening of robustness against prompt and formatting perturbations, simulation-based planning of study size, Westgard-rule control charts for ongoing system suitability, and a reproducible validation report. The package is pipeline-agnostic: any backend that turns a prompt and a model identifier into a text response can be validated, and results produced elsewhere can be imported.

See also

Useful links:

• https://github.com/DanielMoj/llmsa

• Report bugs at https://github.com/DanielMoj/llmsa/issues

Author

Maintainer: Daniel Andreas Moj daniel.moj2@gmail.com (ORCID)