criterion performance measurements
overview
want to understand this report?
mkInteger/128bit/library
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.374567501071703e-7 | 5.675666289437932e-7 | 6.3107506613718e-7 |
| Standard deviation | 5.76273973395364e-8 | 1.4069771740342976e-7 | 2.5503830869271746e-7 |
Outlying measurements have severe (0.985502250117574%) effect on estimated standard deviation.
mkInteger/128bit/builtin
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.7345100502023577e-7 | 2.82032295562717e-7 | 3.1404095173448777e-7 |
| Standard deviation | 3.254173284441495e-9 | 5.374776985438979e-8 | 1.1452241861653819e-7 |
Outlying measurements have severe (0.9730722373405329%) effect on estimated standard deviation.
mkInteger/4096bit/library
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.354529232490156e-5 | 4.502722622293951e-5 | 4.902297431276999e-5 |
| Standard deviation | 3.297179293044225e-6 | 8.050106095475322e-6 | 1.480524101356485e-5 |
Outlying measurements have severe (0.9448928869608393%) effect on estimated standard deviation.
mkInteger/4096bit/builtin
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.153122029479534e-5 | 4.346904865565922e-5 | 4.70866442220212e-5 |
| Standard deviation | 5.090608240043258e-6 | 9.069493107368632e-6 | 1.528778593365153e-5 |
Outlying measurements have severe (0.9604974932829285%) effect on estimated standard deviation.
timesInteger/small/library
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.28687721714645e-8 | 5.392059762787041e-8 | 5.617751136562609e-8 |
| Standard deviation | 3.3946607500785935e-9 | 4.430622730696391e-9 | 6.657012253981569e-9 |
Outlying measurements have severe (0.8752769884022731%) effect on estimated standard deviation.
timesInteger/small/builtin
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.579730247792522e-8 | 8.88720303631296e-8 | 1.039950155543139e-7 |
| Standard deviation | 2.0388581065191845e-9 | 1.940939753411649e-8 | 4.4182268933777814e-8 |
Outlying measurements have severe (0.9796354191363611%) effect on estimated standard deviation.
timesInteger/128bit/library
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.3453473213157255e-6 | 1.4032045766936345e-6 | 1.5398209670849468e-6 |
| Standard deviation | 1.245202139879588e-7 | 2.6864746463086365e-7 | 4.6288851989054065e-7 |
Outlying measurements have severe (0.9687722261750665%) effect on estimated standard deviation.
timesInteger/128bit/builtin
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.046565679280181e-7 | 9.431414373134006e-7 | 1.0330744141942397e-6 |
| Standard deviation | 8.697905198007246e-8 | 1.917915594002612e-7 | 3.3708238778184495e-7 |
Outlying measurements have severe (0.974951421486805%) effect on estimated standard deviation.
timesInteger/4096bit/library
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.39434799738148e-5 | 8.75175952637932e-5 | 9.112315435765985e-5 |
| Standard deviation | 9.948998621666474e-6 | 1.2459793199144407e-5 | 1.370764663885951e-5 |
Outlying measurements have severe (0.9038256667031528%) effect on estimated standard deviation.
timesInteger/4096bit/builtin
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.137106772198425e-4 | 5.155222267383243e-4 | 5.223014660200438e-4 |
| Standard deviation | 2.208779265171566e-6 | 1.1140930994540808e-5 | 2.3344577839412423e-5 |
Outlying measurements have moderate (0.12412240210203208%) effect on estimated standard deviation.
quotRemInteger/small/library
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.309524636598573e-8 | 7.764019602024416e-8 | 8.772447192079604e-8 |
| Standard deviation | 1.3618887604027868e-8 | 2.4963248070053042e-8 | 4.369719465863291e-8 |
Outlying measurements have severe (0.9919215046515667%) effect on estimated standard deviation.
quotRemInteger/small/builtin
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.994307746003785e-6 | 3.0063452314627872e-6 | 3.0197189845252086e-6 |
| Standard deviation | 3.3005137536467454e-8 | 4.131523244216643e-8 | 6.002836338774338e-8 |
Outlying measurements have moderate (0.11575385341466937%) effect on estimated standard deviation.
quotRemInteger/128bit/library
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.1732433246697946e-6 | 1.1863990790949718e-6 | 1.219330345574123e-6 |
| Standard deviation | 1.0998501644224369e-8 | 6.68170931656466e-8 | 1.1439760551514615e-7 |
Outlying measurements have severe (0.7134799355600417%) effect on estimated standard deviation.
quotRemInteger/128bit/builtin
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.235533600514449e-6 | 5.400585683387701e-6 | 5.9831092946510935e-6 |
| Standard deviation | 2.3943826254105593e-7 | 8.882268031533852e-7 | 1.958888893531302e-6 |
Outlying measurements have severe (0.9520961789318251%) effect on estimated standard deviation.
quotRemInteger/4096bit/library
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.74201862577925e-5 | 7.125706157786101e-5 | 7.525146562861844e-5 |
| Standard deviation | 1.1679259800234272e-5 | 1.2367696128225185e-5 | 1.3423726250757256e-5 |
Outlying measurements have severe (0.9388453642894223%) effect on estimated standard deviation.
quotRemInteger/4096bit/builtin
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.837326662313629e-4 | 7.01351040409471e-4 | 7.825016824921499e-4 |
| Standard deviation | 1.8816446494689763e-5 | 9.71720032098532e-5 | 2.192554138138706e-4 |
Outlying measurements have severe (0.8512746483628816%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.