Skip to content

Latest commit

 

History

History
91 lines (72 loc) · 3.88 KB

0128-trivial-benchmark.org

File metadata and controls

91 lines (72 loc) · 3.88 KB
Raw

trivial-benchmark

Some time ago I’ve reviewed the the-cost-of-nothing library which allowed you to check the performance of the form execution. Trivial-benchmark does a similar job but has a few pros and cons.

The main con is that you have to give it a number of iterations manually, but the pro is that the library provides a way more statistics:

POFTHEDAY> (trivial-benchmark:with-timing (1000000)
             (format nil "Symbol is: ~S" :foo))

-                SAMPLES  TOTAL      MINIMUM   MAXIMUM   MEDIAN    AVERAGE    DEVIATION  
REAL-TIME        1000000  3.78       0         0.169     0         0.000004   0.000207   
RUN-TIME         1000000  3.734      0         0.132     0         0.000004   0.000179   
USER-RUN-TIME    1000000  2.332375   0.000001  0.061505  0.000002  0.000002   0.00011    
SYSTEM-RUN-TIME  1000000  1.398129   0.000001  0.070875  0.000001  0.000001   0.000072   
PAGE-FAULTS      1000000  0          0         0         0         0          0.0        
GC-RUN-TIME      1000000  0.436      0         0.132     0         0.0        0.000168   
BYTES-CONSED     1000000  592388352  0         130976    0         592.38837  4354.098   
EVAL-CALLS       1000000  0          0         0         0         0          0.0

Another cool feature is the ability to define more custom metrics.

Here is a practical example. We’ll measure a number of SQL queries made during form execution:

;; These are owr SQL driver simulation:
POFTHEDAY> (defparameter *num-queries* 0)

POFTHEDAY> (defun execute (query)
             "A fake SQL driver"
             (declare (ignorable query))
             (incf *num-queries*))

;; The application code:
POFTHEDAY> (defun the-view ()
             (execute "SELECT some FROM data")
             (loop repeat 5
                   do (execute "SELECT some FROM other_data")))

;; Metric definition is very simple. You just provide a code
;; which returns an absolute value:
POFTHEDAY> (trivial-benchmark:define-delta-metric sql-queries
             *num-queries*)

;; Pay attention to the last line of the report:
POFTHEDAY> (trivial-benchmark:with-timing (100)
             (the-view))
-                SAMPLES  TOTAL     MINIMUM   MAXIMUM   MEDIAN    AVERAGE   DEVIATION  
REAL-TIME        100      0         0         0         0         0         0.0        
RUN-TIME         100      0         0         0         0         0         0.0        
USER-RUN-TIME    100      0.000308  0.000001  0.00012   0.000002  0.000003  0.000012   
SYSTEM-RUN-TIME  100      0.000117  0.000001  0.000002  0.000001  0.000001  0.0        
PAGE-FAULTS      100      0         0         0         0         0         0.0        
GC-RUN-TIME      100      0         0         0         0         0         0.0        
BYTES-CONSED     100      98240     0         65536     0         982.4     7258.1045  
EVAL-CALLS       100      0         0         0         0         0         0.0        
SQL-QUERIES      100      600       6         6         6         6         0.0

The trivial-benchmark is not as accurate as the-cost-of-nothing because it does not count the overhead, but overhead can be significant because trivial-benchmark uses generic functions.

Also when sampling, the trivial-benchmark executes the form only once. That is why the measurements for a very fast code will be even more inaccurate.

Another interesting feature is the ability to define benchmark suites to measure performance regression of some parts of your code. I won’t show you an example of such a suite. Just go and read nice documentation, written by @Shinmera:

https://github.com/Shinmera/trivial-benchmark#benchmark-suites