> For the complete documentation index, see [llms.txt](https://nfa.gitbook.io/nfa-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://nfa.gitbook.io/nfa-docs/the-market-lifecycle/accuracy-and-skill.md).

# Accuracy and skill

## Accuracy and skill

Accuracy is the north star. An engine that cannot show it is accurate is just an opinion with good production values. NFA scores every forecast it can, against the market, and reports the result without flattering itself.

### Brier score

Forecasts are graded with the **Brier score**, the standard measure for probabilistic predictions. It is the squared distance between what you predicted and what happened, so a confident right call scores near zero and a confident wrong call scores near one. On a coin-flip market, guessing 50% scores 0.25. Lower is better.

### Measured against the market, like for like

Every forecast is scored two ways on the same resolved outcome: the engine's number, and the market's price at the time of the forecast. Both are computed the same way over the same outcomes, so "did NFA beat the market" is a fair, apples-to-apples comparison rather than a favourable framing.

**Skill** is the improvement over the market: how much lower the engine's Brier score was than the market's. Only genuine improvement counts toward it, so there is no credit for piling onto outcomes that were already near-certain.

{% hint style="success" %}
The track record is **forward-tested**, not back-tested. Every scored forecast was made before the market resolved, with no knowledge of the outcome. Back-tests on already-resolved markets leak the answer through research, so NFA does not count them.
{% endhint %}

### Does the debate actually earn its cost?

A structured debate is expensive, so NFA measured whether it is worth paying for, by testing the engine against itself. On each forecast a second judge ruled on the same market, from the same evidence corpus, with the debate withheld. The only difference between the two forecasts was whether the advocates had argued and rebutted.

Across 137 resolved markets:

| Forecast                                        | Mean Brier (lower is better) |
| ----------------------------------------------- | ---------------------------- |
| The full engine, with the debate                | **0.289**                    |
| The same judge and the same evidence, no debate | 0.404                        |

The advocates were worth roughly **0.11 Brier**, and the no-debate forecast was closer to the outcome on only 39 of the 137 markets. This is one ablation on one sample, and it will be re-run whenever the debate changes structurally, but it is the reason the architecture is what it is: the advocates earn their compute rather than decorate it.

### Honest by construction

* **A rating needs a sample.** A forecaster with only a handful of resolved calls is shown as unrated rather than credited with a lucky streak.
* **Exclusions are recorded, never hidden.** When a run cannot be scored fairly (for example a market that was already decided at forecast time) it is set aside with a stated reason, so the record has no silent survivorship bias.
* **No-edge families are kept out of the headline.** Markets NFA does not claim an edge on, like city temperature, are excluded from the headline accuracy rather than used to pad or drag it.

The result is a record you can trust precisely because it is not designed to look good. When NFA is wrong, the number says so.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://nfa.gitbook.io/nfa-docs/the-market-lifecycle/accuracy-and-skill.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.