Overview

This describes the formatted results extracted from the insert benchmark output. The results are extracted by helper scripts as the insert benchmark client generates a lot of data. The helper scripts also run vmstat and iostat to collect hardware metrics that are used to compute HW efficiency metrics. The scripts generate one line of text per test step and I currently use several test steps.

The test steps are:

Below I frequently assume that writes == inserts and reads == short range queries.

Load

This is valid for l.i0, l.x and l.i1. For l.x metrics are normalized by the number of indexed rows (documents) – so IPS means indexed rows/second.

Example output, old version

ips     qps     rps     rkbps   wkbps   rpq     rkbpq   wkbpi   csps    cpups   cspq    cpupq   dbgb1   dbgb2   rss     maxop   p50     p99     tag
60000   0       0       0       9041    0.000   0.000   0.151   5990    11.2    0.100   30      0.7     1.5     0.6     0.101   62533   50146   rx56.c7b40

Example output, current version

ips     qps     rps     rmbps   wps     wmbps   rpq     rkbpq   wpi     wkbpi   csps    cpups   cspq    cpupq   dbgb1   dbgb2   rss     maxop   p50     p99     tag
103093  0       0       0.0     38.3    15.4    0.000   0.000   0.000   0.153   10495   45.5    0.102   18      1.0     2.0     1.1     0.103   101988  83911   rx56.cx6a

Note:

Legend:

Run

The legend from Load applies here but I explain where it differs. The load step is insert-only, the run step is read-write. For some of the metrics below, the denominator is qps rather than qps+ips. This approach isn’t perfect but is useful for doing comparisons with rpq, rkbpq, cpupq between different DBMS.

Legend:

Example output, old version

ips     qps     rps     rkbps   wkbps   rpq     rkbpq   wkbpi   csps    cpups   cspq    cpupq   dbgb1   dbgb2   rss     maxop   p50     p99     tag
99      3156    0       0       25      0.000   0.000   0.249   12472   5.2     3.952   264     3.5     6.1     3.3     0.001   3165    3117    rx56.c7b40

Example output, new version

ips     qps     rps     rmbps   wps     wmbps   rpq     rkbpq   wpi     wkbpi   csps    cpups   cspq    cpupq   dbgb1   dbgb2   rss     maxop   p50     p99     tag
200     5123    0       0.0     3.9     0.7     0.000   0.000   0.019   3.705   19698   26.3    3.845   205     4.8     8.5     4.7     0.010   5130    5018    rx56.cx6a

Response Time Histograms

These list the percentage of responses that are in each bucket. Each response is counted once in the bucket in which it fits. The 256us bucket has all responses that take <= 256 microseconds. The 1ms bucket has all repsonses that take between 256us and 1 millisecond, and so on. The buckets are:

256us   1ms     4ms     16ms    64ms    256ms   1s      4s      16s     gt      max     tag
0.000   0.000   82.919  15.837  1.073   0.161   0.010   0.000   0.000   0.000   0.45814 pg12.c7b40