Conformal scores and the geometry of efficiency

Conformal prediction has a clean promise: if the calibration and test examples are exchangeable, a split conformal prediction set has finite-sample marginal coverage. The model used to build the score can be wrong. The likelihood can be misspecified. The geometry can be only approximate.

That guarantee I don’t want to disturb. The question here is different: if conformal validity is model-free, where can model geometry matter?

The answer is through the nonconformity score. Geometry doesn’t prove coverage — it changes efficiency. Exchangeability gives validity; score geometry affects how small the prediction sets are.

Validity comes from exchangeability

Let Z_i = (X_i, Y_i), i = 1,\ldots,n+1, be exchangeable observations, and let S: \mathcal{X} \times \mathcal{Y} \to \mathbb{R} be any measurable nonconformity score. On a calibration set, define R_i = S(X_i, Y_i). For target miscoverage \alpha, use the conformal quantile

\hat{q} = R_{\left(\lceil (n_{\mathrm{cal}}+1)(1-\alpha) \rceil\right)}.

The split conformal set is C_\alpha(x) = \{y : S(x,y) \le \hat{q}\}, and

\mathbb{P}\{Y_{n+1} \in C_\alpha(X_{n+1})\} \ge 1 - \alpha.

The key word is any — after the score has been fixed. In split conformal, the score is fitted on a separate training split and treated as fixed when calibration and test scores are compared. The score can come from a random forest, a kernel method, a neural network, a quantile model, or a density model. Conditional on the fitted score, if the calibration and test examples are exchangeable, the marginal guarantee remains.

This is why geometry isn’t a new validity proof for conformal prediction. The validity proof is already there. The role of geometry is narrower: a geometric score may rank candidate labels better, which can mean shorter intervals, smaller sets, or better conditional behavior — but it’s not what makes the method valid.

The score defines the geometry

The simplest regression score is S_{\mathrm{abs}}(x,y) = |y - \hat\mu(x)|, which creates a global residual interval

C_\alpha(x) = [\hat\mu(x) - \hat{q},\; \hat\mu(x) + \hat{q}].

This is reasonable when conditional noise is roughly symmetric and has the same scale everywhere. When the noise scale changes with x, a global threshold tends to be too wide in easy regions and too narrow in hard ones. Marginal coverage can still be right, because the errors average out — but the sets are wasteful.

A local-scale score changes the ruler:

S_{\mathrm{loc}}(x,y) = \frac{|y - \hat\mu(x)|}{\hat\sigma(x)},

giving intervals [\hat\mu(x) - \hat{q}\hat\sigma(x),\; \hat\mu(x) + \hat{q}\hat\sigma(x)]. This is a one-dimensional Mahalanobis idea — the local ruler is g(x) \approx 1/\hat\sigma^2(x). In a Gaussian conditional model, Y \mid X = x \sim N(\mu(x), \sigma^2(x)), the negative log-likelihood contains (y - \mu(x))^2 / 2\sigma^2(x), so local normalization measures residual size in the local uncertainty units of the model. Conformal calibration doesn’t care whether the Gaussian story is true. It only uses the resulting ranks.

Conformalized quantile regression uses lower and upper quantile models \hat{q}_\ell(x), \hat{q}_u(x), with score

S_{\mathrm{CQR}}(x,y) = \max\{\hat{q}_\ell(x) - y,\; y - \hat{q}_u(x)\},

giving intervals that aren’t forced to be symmetric around a mean. For multimodal data, even quantile intervals can be the wrong shape — if two separated labels are plausible and the middle isn’t, a single interval wastes volume. A density score

S_{\mathrm{dens}}(x,y) = -\log \hat{p}(y \mid x)

gives the set C_\alpha(x) = \{y : \hat{p}(y \mid x) \ge e^{-\hat{q}}\}, which can be disconnected. That’s the right geometry for multimodal conditional laws.

Geometry has to be estimated

A local metric helps only if it’s estimated stably. If \hat\sigma(x) is too small on some calibration points, then |y - \hat\mu(x)| / \hat\sigma(x) can blow up, the conformal quantile can become enormous, and the final intervals can explode.

To make this failure mode visible, I use stabilized local scales

\tilde\sigma(x) = \operatorname{clip}(\hat\sigma(x), s_{\min}, s_{\max}) + \lambda,

and a local/global blend

\tilde\sigma_\gamma(x) = (1-\gamma)\tilde\sigma_{\mathrm{local}}(x) + \gamma\tilde\sigma_{\mathrm{global}}.

Here \gamma = 0 keeps the score fully local, \gamma = 1 moves it back toward a global residual scale. Think of \gamma as a trust parameter: how much estimated local geometry should be allowed into the score?

The experiment

Synthetic regression data with mean \mu(x) = \sin(2\pi x_1) + \frac{1}{2}x_1, five settings:

Ordinary split conformal throughout: split into train, calibration, and test; fit a model on training; build one or more scores; compute conformal quantile on calibration scores; evaluate coverage and set length on test. The score changes. The calibration logic doesn’t.

Four model families: random forests, kernel ridge regression, MLPs, and GBRT-based conformalized quantile regression.

Results

Coverage stays near 90% for most methods — that’s the conformal part. Average length changes a lot. That’s the score-geometry part.

Heteroskedastic data is the clearest positive case for local geometry.

Random forests: length drops from 1.95 to 1.68. MLPs: 1.99 to 1.78. GBRT-CQR: 1.90 to 1.70. Conditional coverage diagnostics also improve.

Kernel ridge regression is the warning. In this run, even the best blended geometry is slightly longer than the global residual baseline. In more aggressive local-normalization variants, intervals can become enormous because the estimated scale is unstable. That’s not a conformal failure — coverage is still protected. It’s a score failure. Use only as much geometry as you can estimate stably.

Homoskedastic data is a useful sanity check. The simple residual score is hard to beat; for random forests, the absolute residual interval has average length about 0.856, and oracle local normalization gives essentially the same number. Local geometry has nothing useful to learn when there’s no local scale structure.

Heavy-tailed and skewed cases are mixed but informative. Blended local-scale scores give modest gains for RF and MLP in the heavy-tailed setting and often improve conditional balance. In the skewed setting, RF and MLP blended scores work well, while CQR improves conditional balance but isn’t uniformly shorter in this run. More flexible geometry isn’t automatically better — it helps only when the fitted score ranks labels well.

The multimodal case

The multimodal setting is the clearest example. Interval methods have the wrong shape: they must cover the space between modes even when the middle isn’t very plausible.

I fit a simple two-mode conditional density model

\hat{p}(y \mid x) = \hat\pi(x)\,\mathcal{N}(y;\hat\mu_+(x),\hat\sigma_+^2) + (1-\hat\pi(x))\,\mathcal{N}(y;\hat\mu_-(x),\hat\sigma_-^2),

not as a state-of-the-art density estimator, but just expressive enough to test the geometry. The density-level-set conformal method gives average length about 1.82 with coverage about 0.904.

Once the score geometry matches the conditional law, conformal sets become much more efficient.

Diagnostics should match the geometry

At first, the density method looked less locally balanced when I binned by \sigma(x). But \sigma(x) isn’t the right diagnostic for a mixture score — the score models mode structure, not local scale. Binning instead by mixture-specific quantities:

The earlier diagnostic was partly asking the wrong question. If the score models local scale, bin by local scale. If the score models mixture uncertainty, bin by mixture uncertainty.

Takeaway

Conformal calibration and score geometry do different jobs. Calibration supplies marginal validity. The score determines which candidate labels look plausible, and therefore determines the shape and efficiency of the prediction set.

Efficiency depends on whether the score ranks candidate labels according to the actual conditional structure of Y given X. A rough taxonomy: flat noise calls for a residual score, heteroskedasticity for a local scale score, varying quantiles for CQR, multimodality for a density-level-set score. The conformal layer is the safety net — it turns model-dependent scores into marginally valid prediction sets, but it doesn’t make all scores equally good.

Appendix

Why monotone score transforms didn’t matter

I tried Student-t and Huber versions of normalized residual scores. In scalar symmetric regression, these are mostly rank-equivalent to normalized absolute residuals — \log(1 + r^2/\nu) is monotone in |r|, and the Huber loss is also monotone in |r|. Since conformal prediction is rank-based, these transforms don’t change the prediction set if they use the same scale.

What density-set length means

For density-level-set conformal prediction in one-dimensional regression, I approximate |C_\alpha(x)| on a grid. Accepted grid points are those satisfying -\log \hat{p}(y \mid x) \le \hat{q}. If the density has two separated modes, the accepted grid points can form two disconnected intervals. The reported length is the total grid length of the accepted region.

A classification analogue

For a neural classifier, ordinary conformal scores use softmax probabilities, S(x,y) = 1 - \hat{p}_\theta(y \mid x), or cumulative-probability rankings. A more geometric score could use the penultimate-layer representation \phi_\theta(x), fitting class centers m_y and covariances \Sigma_y, then

S(x,y) = (\phi_\theta(x) - m_y)^\top \Sigma_y^{-1} (\phi_\theta(x) - m_y).

The network supplies the geometry. Conformal calibration supplies the validity.

What is and isn’t proved here

The coverage statements are standard split-conformal ones. The new part is empirical and conceptual: different score geometries induce different set shapes and efficiencies. A formal theorem would need to control excess set size — something like \mathbb{E}|C_\alpha(X)| - \mathbb{E}|C_\alpha^*(X)| in terms of score-ranking error, density-estimation error, or metric-estimation error. That’s not proved here.

Reproducing the benchmark

The full benchmark can be regenerated from ig_conformal.py:

raw, summary = ig.run_grid(
    settings=["homoskedastic", "heteroskedastic", "heavy_tail", "skewed", "multimodal"],
    model_families=["rf", "krr", "mlp", "cqr_gbrt"],
    n=6000,
    d=5,
    alpha=0.1,
    seeds=[0, 1, 2],
    save_csv=True,
    output_prefix="ig_conformal_results",
)

compact = ig.summarize_geometry_effect(summary)
compact.to_csv("ig_conformal_geometry_effect.csv", index=False)

multimodal_diag = ig.summarize_multimodal_density_diagnostics(summary)
multimodal_diag.to_csv("ig_conformal_multimodal_density_diagnostics.csv", index=False)

References

Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic Learning in a Random World. Springer, 2005.

Glenn Shafer and Vladimir Vovk. “A Tutorial on Conformal Prediction.” Journal of Machine Learning Research 9, 2008.

Jing Lei, Max G’Sell, Alessandro Rinaldo, Ryan J. Tibshirani, and Larry Wasserman. “Distribution-Free Predictive Inference for Regression.” Journal of the American Statistical Association 113(523), 2018.

Yaniv Romano, Evan Patterson, and Emmanuel Candès. “Conformalized Quantile Regression.” NeurIPS, 2019.

Anastasios Angelopoulos and Stephen Bates. “A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification.” arXiv:2107.07511, 2021.

Rob Hyndman. “Computing and Graphing Highest Density Regions.” The American Statistician 50(2), 1996.

Rafael Izbicki, Gilson Shimizu, and Rafael Stern. “Flexible Distribution-Free Conditional Predictive Bands using Density Estimators.” AISTATS, 2020.

Shun-ichi Amari. Information Geometry and Its Applications. Springer, 2016.

Suggested citation

@misc{miryusupov2026conformalgeometry,
  author       = {Miryusupov, Shohruh},
  title        = {Conformal Scores and the Geometry of Efficiency},
  year         = {2026},
  howpublished = {Research note},
  url = {https://www.miryusupov.com/blog/posts/geometry_conformal_scores/index.html}
}