We show that the GPT-3 model can learn how to express uncertainty about its answers in natural language without using model logits. Given a question, the model generates both an answer and a confidence level (for example, “90% confidence” or “high confidence”). These levels are mapped to appropriately adjusted probabilities. The model also remains reasonably tuned under changes in the distribution, and is sensitive to uncertainty in its own answers rather than mimicking human examples. To our knowledge, this is the first time that a model has been shown to express adjusted uncertainty about its own answer in natural language. To test the calibration, we introduce the CalibratedMath task suite. Compare the calibration of the verbalized uncertainty (“verbalized probability”) with the uncertainty extracted from the model logit. Both types of uncertainty can generalize the calibration under distribution shifts. We also provide evidence that the ability of GPT-3 to generalize its calibration relies on a pre-trained latent representation that correlates with epistemic uncertainty about its answer.
