criterion performance measurements
overview
want to understand this report?
mkInteger/128bit/library
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.601658998992718e-7 | 5.733576419532905e-7 | 5.907375283327718e-7 |
| Standard deviation | 3.650211996853249e-8 | 5.280531265826403e-8 | 8.47334734336799e-8 |
Outlying measurements have severe (0.8769614251506723%) effect on estimated standard deviation.
mkInteger/128bit/builtin
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.5231281452665327e-7 | 4.7719826783489e-7 | 5.296352526483933e-7 |
| Standard deviation | 6.265956865686561e-8 | 1.1801442474877234e-7 | 2.1983595970102167e-7 |
Outlying measurements have severe (0.981292729876015%) effect on estimated standard deviation.
mkInteger/4096bit/library
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.524190366778291e-5 | 3.592702163577496e-5 | 3.697336600147401e-5 |
| Standard deviation | 1.7102230568404322e-6 | 3.205554353546633e-6 | 4.842870358542851e-6 |
Outlying measurements have severe (0.8106461188254519%) effect on estimated standard deviation.
mkInteger/4096bit/builtin
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.7829961392182725e-5 | 2.9888774487739707e-5 | 3.1933258442046424e-5 |
| Standard deviation | 6.503417577618895e-6 | 7.287316313797789e-6 | 8.002695375641599e-6 |
Outlying measurements have severe (0.970291281287434%) effect on estimated standard deviation.
timesInteger/small/library
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.3794855042056195e-8 | 5.5059390399082963e-8 | 5.835505775862757e-8 |
| Standard deviation | 3.763720334262661e-9 | 5.912646136726058e-9 | 1.014754730970273e-8 |
Outlying measurements have severe (0.9219102365943673%) effect on estimated standard deviation.
timesInteger/small/builtin
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.772815645941758e-8 | 4.9157009298025135e-8 | 5.122413383596607e-8 |
| Standard deviation | 3.936017187362149e-9 | 5.2619675825842886e-9 | 6.895714984378198e-9 |
Outlying measurements have severe (0.9227305839766201%) effect on estimated standard deviation.
timesInteger/128bit/library
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.6822294480712156e-6 | 1.9563614060373224e-6 | 2.2368172121296855e-6 |
| Standard deviation | 5.976752641067496e-7 | 7.327127070241201e-7 | 9.152913441043574e-7 |
Outlying measurements have severe (0.9943354213175232%) effect on estimated standard deviation.
timesInteger/128bit/builtin
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.67945126157982e-7 | 1.062935310046978e-6 | 1.2369364333631646e-6 |
| Standard deviation | 1.056054331744944e-7 | 4.326995799826509e-7 | 6.977556204249726e-7 |
Outlying measurements have severe (0.9948180221681772%) effect on estimated standard deviation.
timesInteger/4096bit/library
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.675598424018823e-5 | 9.755438895155419e-5 | 9.983017708506163e-5 |
| Standard deviation | 9.081716917392649e-7 | 4.792965275038115e-6 | 9.21511610190537e-6 |
Outlying measurements have severe (0.5102796645465975%) effect on estimated standard deviation.
timesInteger/4096bit/builtin
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.364288398252375e-5 | 6.780483397009362e-5 | 7.246083308630882e-5 |
| Standard deviation | 1.216853827691828e-5 | 1.3545347217581106e-5 | 1.5181504338038574e-5 |
Outlying measurements have severe (0.9567694493478138%) effect on estimated standard deviation.
quotRemInteger/small/library
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.899942052481606e-8 | 7.057061631140571e-8 | 7.27905321605875e-8 |
| Standard deviation | 5.057425890453651e-9 | 6.311425882227019e-9 | 8.32958041354954e-9 |
Outlying measurements have severe (0.888857592782675%) effect on estimated standard deviation.
quotRemInteger/small/builtin
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.643691721034419e-8 | 6.860817444381731e-8 | 7.2357384963036e-8 |
| Standard deviation | 7.749774351399419e-9 | 9.405544128093694e-9 | 1.3211854261377631e-8 |
Outlying measurements have severe (0.9520373390144126%) effect on estimated standard deviation.
quotRemInteger/128bit/library
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.1462377200158355e-6 | 1.1815150845594235e-6 | 1.2196326692501667e-6 |
| Standard deviation | 9.427206956297538e-8 | 1.1998327466325664e-7 | 1.5929030197442522e-7 |
Outlying measurements have severe (0.8931506980784639%) effect on estimated standard deviation.
quotRemInteger/128bit/builtin
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.095937842982912e-6 | 1.1701004597237269e-6 | 1.3356277329976227e-6 |
| Standard deviation | 2.2097469959280466e-7 | 3.732388370071895e-7 | 7.179098703004671e-7 |
Outlying measurements have severe (0.9896604510917647%) effect on estimated standard deviation.
quotRemInteger/4096bit/library
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.43376970940696e-5 | 8.751494584443541e-5 | 9.167465784609307e-5 |
| Standard deviation | 9.251195370804929e-6 | 1.3718542197374158e-5 | 2.1339259857923272e-5 |
Outlying measurements have severe (0.9211173889449344%) effect on estimated standard deviation.
quotRemInteger/4096bit/builtin
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.442248195583455e-5 | 6.793498282438347e-5 | 7.081431402781728e-5 |
| Standard deviation | 8.288400342467137e-6 | 1.1346409671223399e-5 | 1.4053744274955918e-5 |
Outlying measurements have severe (0.9320741678233181%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.