SafetyNet - Performance regression detection for PDFium

This document explains how to use SafetyNet to detect performance regressions in PDFium.

Comparing performance of two versions of PDFium

safetynet_compare.py is a script that compares the performance between two versions of pdfium. This can be used to verify if a given change has caused or will cause any positive or negative changes in performance for a set of test cases.

The supported profilers are exclusive to Linux, so for now this can only be run on Linux.

An illustrative example is below, comparing the local code version to an older version. Positive % changes mean an increase in time/instructions to run the test - a regression, while negative % changes mean a decrease in time/instructions, therefore an improvement.

$ testing/tools/safetynet_compare.py ~/test_pdfs --branch-before beef5e4
================================================================================
   % Change      Time after  Test case
--------------------------------------------------------------------------------
   -0.1980%  45,703,820,326  ~/test_pdfs/PDF Reference 1-7.pdf
   -0.5678%      42,038,814  ~/test_pdfs/Page 24 - PDF Reference 1-7.pdf
   +0.2666%  10,983,158,809  ~/test_pdfs/Rival.pdf
   +0.0447%  10,413,890,748  ~/test_pdfs/dynamic.pdf
   -7.7228%      26,161,171  ~/test_pdfs/encrypted1234.pdf
   -0.2763%     102,084,398  ~/test_pdfs/ghost.pdf
   -3.7005%  10,800,642,262  ~/test_pdfs/musician.pdf
   -0.2266%  45,691,618,789  ~/test_pdfs/no_metadata.pdf
   +1.4440%  38,442,606,162  ~/test_pdfs/test7.pdf
   +0.0335%       9,286,083  ~/test_pdfs/testbulletpoint.pdf
================================================================================
Test cases run: 10
Failed to measure: 0
Regressions: 0
Improvements: 2

Usage

Run the safetynet_compare.py script in testing/tools to perform a comparison. Pass one or more paths with test cases - each path can be either a .pdf file or a directory containing .pdf files. Other files in those directories are ignored.

The following comparison modes are supported:

Compare uncommitted changes against clean branch:

$ testing/tools/safetynet_compare.py path/to/pdfs

Compare current branch with another branch or commit:

$ testing/tools/safetynet_compare.py path/to/pdfs --branch-before another_branch
$ testing/tools/safetynet_compare.py path/to/pdfs --branch-before 1a3c5e7

Compare two other branches or commits:

$ testing/tools/safetynet_compare.py path/to/pdfs --branch-after another_branch --branch-before yet_another_branch
$ testing/tools/safetynet_compare.py path/to/pdfs --branch-after 1a3c5e7 --branch-before 0b2d4f6
$ testing/tools/safetynet_compare.py path/to/pdfs --branch-after another_branch --branch-before 0b2d4f6

Compare two build flag configurations:

$ gn args out/BuildConfig1
$ gn args out/BuildConfig2
$ testing/tools/safetynet_compare.py path/to/pdfs --build-dir out/BuildConfig2 --build-dir-before out/BuildConfig1

safetynet_compare.py takes care of checking out the appropriate branch, building it, running the test cases and comparing results.

Profilers

safetynet_compare.py uses callgrind as a profiler by default. Use --profiler to specify another one. The supported ones are:

perfstat

Only works on Linux. Make sure you have perf by typing in the terminal:

$ perf

This is a fast profiler, but uses sampling so it's slightly inaccurate. Expect variations of up to 1%, which is below the cutoff to consider a change significant.

Use this when running over large test sets to get good enough results.

callgrind

Only works on Linux. Make sure valgrind is installed:

$ valgrind

This is a slow and accurate profiler. Expect variations of around 100 instructions. However, this takes about 50 times longer to run than perf stat.

Use this when looking for small variations (< 1%).

One advantage is that callgrind can generate callgrind.out files (by passing --output-dir to safetynet_compare.py), which contain profiling information that can be analyzed to find the cause of a regression. KCachegrind is a good visualizer for these files.

none

Run without any profiler, giving a performance score of 1 always. useful for running image comparisons or debugging the script.

Common Options

Arguments commonly passed to safetynet_compare.py.

--profiler: described above.
--build-dir: this specified the build config with a relative path from the pdfium src directory to the build directory. Defaults to out/Release.
--output-dir: where to place the profiling output files. These are callgrind.out.[test_case] files for callgrind, perfstat does not produce them. By default they are not written.
--case-order: sort test case results according to this metric. Can be “after”, “before”, “ratio” and “rating”. If not specified, sort by path.
--this-repo: use the repository where the script is instead of checking out a temporary one. This is faster and does not require downloads. Although it restores the state of the local repo, if the script is killed or crashes the uncommitted changes can remain stashed and you may be on another branch.

Other Options

Most of the time these don't need to be used.

--build-dir-before: if comparing different build dirs (say, to test what a flag flip does), specify the build dir for the “before” branch here and the build dir for the “after” branch with --build-dir.
--interesting-section: only the interesting section should be measured instead of all the execution of the test harness. This only works in debug, since in release the delimiters are stripped out. This does not work to compare branches that don’t have the callgrind delimiters, as it would otherwise be unfair to compare a whole run vs the interesting section of another run.
--machine-readable: output a json with the results that is easier to read by code.
--num-workers: how many workers to use to parallelize test case runs. Defaults to # of CPUs in the machine.
--threshold-significant: highlight differences that exceed this value. Defaults to 0.02.
--tmp-dir: directory in which temporary repos will be cloned and downloads will be cached, if --this-repo is not enabled. Defaults to /tmp.

Setup a nightly job

Create a separate checkout of pdfium in a new directory, for example ~/job. The safetynet_job.py script will run from this directory. This checkout needs to be git pull'ed when there are changes to the SafetyNet scripts, but otherwise it can be left alone.

Create a directory to contain the job results, for example ~/job_results. In each run, a .log file with the results will be written to this directory and a subdirectory will be created with the other artifacts.

Setup a cron job to run safetynet_job.py nightly. The example below runs it at 1:42 AM, over the corpus in two directories: ~/pdf_samples/thousand_pdfs and ~/pdf_samples/i18n

@ crontab -e
42 1 * * * bash -lc '~/job/pdfium/testing/tools/safetynet_job.py ~/job_results ~/pdf_samples/thousand_pdfs ~/pdf_samples/i18n --output-to-log >> ~/job_results/cron_nightly.log 2>&1'

The first time the job runs, it will just create a checkpoint as ~/job_results/last_revision_covered. From then on, since a checkpoint is available, each run will compare performance with the last checkpoint and update the checkpoint.

Run image comparison

Pass the --png-dir option pointing at an output directory to compare the output images from rendering the “before” and the “after” branches with pdfium_test.

$ mkdir ~/output_images
$ testing/tools/safetynet_compare.py ~/pdf_samples --branch-before before_visual_changes --branch-after after_visual_changes --png-dir ~/output_images

This will output and automatically open a ~/output_images/compare.html file showing the before/after and the diff. Hover the mouse cursor over the before/after image on the left for an easier visual comparison. The “before” image is displayed until the cursor hovers over the image, which is then replaced with the “after” image.

It is recommended to use --profiler=none with this option.