This document explains how to use SafetyNet to detect performance regressions in PDFium.
safetynet_compare.py is a script that compares the performance between two versions of pdfium. This can be used to verify if a given change has caused or will cause any positive or negative changes in performance for a set of test cases.
The supported profilers are exclusive to Linux, so for now this can only be run on Linux.
An illustrative example is below, comparing the local code version to an older version. Positive % changes mean an increase in time/instructions to run the test - a regression, while negative % changes mean a decrease in time/instructions, therefore an improvement.
$ testing/tools/safetynet_compare.py ~/test_pdfs --branch-before beef5e4 ================================================================================ % Change Time after Test case -------------------------------------------------------------------------------- -0.1980% 45,703,820,326 ~/test_pdfs/PDF Reference 1-7.pdf -0.5678% 42,038,814 ~/test_pdfs/Page 24 - PDF Reference 1-7.pdf +0.2666% 10,983,158,809 ~/test_pdfs/Rival.pdf +0.0447% 10,413,890,748 ~/test_pdfs/dynamic.pdf -7.7228% 26,161,171 ~/test_pdfs/encrypted1234.pdf -0.2763% 102,084,398 ~/test_pdfs/ghost.pdf -3.7005% 10,800,642,262 ~/test_pdfs/musician.pdf -0.2266% 45,691,618,789 ~/test_pdfs/no_metadata.pdf +1.4440% 38,442,606,162 ~/test_pdfs/test7.pdf +0.0335% 9,286,083 ~/test_pdfs/testbulletpoint.pdf ================================================================================ Test cases run: 10 Failed to measure: 0 Regressions: 0 Improvements: 2
Run the safetynet_compare.py script in testing/tools to perform a comparison. Pass one or more paths with test cases - each path can be either a .pdf file or a directory containing .pdf files. Other files in those directories are ignored.
The following comparison modes are supported:
$ testing/tools/safetynet_compare.py path/to/pdfs
$ testing/tools/safetynet_compare.py path/to/pdfs --branch-before another_branch $ testing/tools/safetynet_compare.py path/to/pdfs --branch-before 1a3c5e7
$ testing/tools/safetynet_compare.py path/to/pdfs --branch-after another_branch --branch-before yet_another_branch $ testing/tools/safetynet_compare.py path/to/pdfs --branch-after 1a3c5e7 --branch-before 0b2d4f6 $ testing/tools/safetynet_compare.py path/to/pdfs --branch-after another_branch --branch-before 0b2d4f6
$ gn args out/BuildConfig1 $ gn args out/BuildConfig2 $ testing/tools/safetynet_compare.py path/to/pdfs --build-dir out/BuildConfig2 --build-dir-before out/BuildConfig1
safetynet_compare.py takes care of checking out the appropriate branch, building it, running the test cases and comparing results.
safetynet_compare.py uses callgrind as a profiler by default. Use --profiler to specify another one. The supported ones are:
Only works on Linux. Make sure you have perf by typing in the terminal:
$ perf
This is a fast profiler, but uses sampling so it's slightly inaccurate. Expect variations of up to 1%, which is below the cutoff to consider a change significant.
Use this when running over large test sets to get good enough results.
Only works on Linux. Make sure valgrind is installed:
$ valgrind
This is a slow and accurate profiler. Expect variations of around 100 instructions. However, this takes about 50 times longer to run than perf stat.
Use this when looking for small variations (< 1%).
One advantage is that callgrind can generate callgrind.out
files (by passing --output-dir to safetynet_compare.py), which contain profiling information that can be analyzed to find the cause of a regression. KCachegrind is a good visualizer for these files.
Arguments commonly passed to safetynet_compare.py.
Most of the time these don't need to be used.
TODO: Complete with safetynet_job.py setup and usage.