# AugurScore - Evaluating Augur's probability estimates

• The purpose of this site is to explore the accuracy of the probability estimates generated by Augur. Every completed trade on Augur can be considered the market's best estimate of the probability for a given outcome. It is then natural to ask how accurate a set of probability predictions turn out to be when the events are finalized.

• Here is a simple example. Weather forecasters often say "There is a 70% chance of rain tomorrow." We would expect that given a large collection of days where the forecast is for a 70% chance of rain the following day, then approximately 70% would indeed experience rainfall on the following day. Does this expectation turn out to be the case? In other words, are weather forecasters generating well-calibrated probability estimates?

• We can use data from Augur to explore the quality of market-based probability estimates over a diverse set of events, including sporting events, political events, etc.

• In statistics and machine learning "scoring rules" are used to measure the accuracy of probability predictions. So scoring rules are similar to loss functions in machine learning. On this site we evaluate several scoring rules on Augur probability predictions. For a brief overview of scoring rules, see the Wikipedia article.

• Probabilities for outcomes change over time. For example, predictions seven days in advance are presumably less accurate than predictions one day in advance. We allow the user to specify the timepoint of interest.

• In order to estimate a probability at a given timepoint we use a weighted average of "nearby" trade prices. All probabilities (i.e. trade prices) are temporally weighted by a kernel function centered at the timepoint of interest. The user can specify the form of the kernel function and the kernel width.

• All probabilities are also weighted by trade volume (in addition to the temporal weight).

• At the moment, AugurScore only supports analysis on Yes/No markets.

• We also generate a calibration plot which directly illustrates the relationship between the predicted probabilities and the observed, empirical frequencies. A perfect probability predictor would generate a straight, diagonal line, i.e. predicted probability = observed frequency. For the calibration plot calculation we again utilize a kernel function but this time in "price/probability space."

• Technical details: We use the Augur API (augur.js) to pull the data from an Augur Node. We use Python to process the data. This website is written in Python with Flask.

• Here is the code on GitHub: github.com/mettinger/AugurScore

• TODO:
• Improve interface to allow selecting a subset of events to analyze (for example, only political events or a designated set of markets).