| # SafetyNet - Performance regression detection for PDFium |
| |
| [TOC] |
| |
| This document explains how to use SafetyNet to detect performance regressions |
| in PDFium. |
| |
| ## Comparing performance of two versions of PDFium |
| |
| safetynet_compare.py is a script that compares the performance between two |
| versions of pdfium. This can be used to verify if a given change has caused |
| or will cause any positive or negative changes in performance for a set of test |
| cases. |
| |
| The supported profilers are exclusive to Linux, so for now this can only be run |
| on Linux. |
| |
| An illustrative example is below, comparing the local code version to an older |
| version. Positive % changes mean an increase in time/instructions to run the |
| test - a regression, while negative % changes mean a decrease in |
| time/instructions, therefore an improvement. |
| |
| ``` |
| $ testing/tools/safetynet_compare.py ~/test_pdfs --branch-before beef5e4 |
| ================================================================================ |
| % Change Time after Test case |
| -------------------------------------------------------------------------------- |
| -0.1980% 45,703,820,326 ~/test_pdfs/PDF Reference 1-7.pdf |
| -0.5678% 42,038,814 ~/test_pdfs/Page 24 - PDF Reference 1-7.pdf |
| +0.2666% 10,983,158,809 ~/test_pdfs/Rival.pdf |
| +0.0447% 10,413,890,748 ~/test_pdfs/dynamic.pdf |
| -7.7228% 26,161,171 ~/test_pdfs/encrypted1234.pdf |
| -0.2763% 102,084,398 ~/test_pdfs/ghost.pdf |
| -3.7005% 10,800,642,262 ~/test_pdfs/musician.pdf |
| -0.2266% 45,691,618,789 ~/test_pdfs/no_metadata.pdf |
| +1.4440% 38,442,606,162 ~/test_pdfs/test7.pdf |
| +0.0335% 9,286,083 ~/test_pdfs/testbulletpoint.pdf |
| ================================================================================ |
| Test cases run: 10 |
| Failed to measure: 0 |
| Regressions: 0 |
| Improvements: 2 |
| ``` |
| |
| ### Usage |
| |
| Run the safetynet_compare.py script in testing/tools to perform a comparison. |
| Pass one or more paths with test cases - each path can be either a .pdf file or |
| a directory containing .pdf files. Other files in those directories are |
| ignored. |
| |
| The following comparison modes are supported: |
| |
| 1. Compare uncommitted changes against clean branch: |
| ```shell |
| $ testing/tools/safetynet_compare.py path/to/pdfs |
| ``` |
| |
| 2. Compare current branch with another branch or commit: |
| ```shell |
| $ testing/tools/safetynet_compare.py path/to/pdfs --branch-before another_branch |
| $ testing/tools/safetynet_compare.py path/to/pdfs --branch-before 1a3c5e7 |
| ``` |
| |
| 3. Compare two other branches or commits: |
| ```shell |
| $ testing/tools/safetynet_compare.py path/to/pdfs --branch-after another_branch --branch-before yet_another_branch |
| $ testing/tools/safetynet_compare.py path/to/pdfs --branch-after 1a3c5e7 --branch-before 0b2d4f6 |
| $ testing/tools/safetynet_compare.py path/to/pdfs --branch-after another_branch --branch-before 0b2d4f6 |
| ``` |
| |
| 4. Compare two build flag configurations: |
| ```shell |
| $ gn args out/BuildConfig1 |
| $ gn args out/BuildConfig2 |
| $ testing/tools/safetynet_compare.py path/to/pdfs --build-dir out/BuildConfig2 --build-dir-before out/BuildConfig1 |
| ``` |
| |
| safetynet_compare.py takes care of checking out the appropriate branch, building |
| it, running the test cases and comparing results. |
| |
| ### Profilers |
| |
| safetynet_compare.py uses callgrind as a profiler by default. Use --profiler |
| to specify another one. The supported ones are: |
| |
| #### perfstat |
| |
| Only works on Linux. |
| Make sure you have perf by typing in the terminal: |
| ```shell |
| $ perf |
| ``` |
| |
| This is a fast profiler, but uses sampling so it's slightly inaccurate. |
| Expect variations of up to 1%, which is below the cutoff to consider a |
| change significant. |
| |
| Use this when running over large test sets to get good enough results. |
| |
| #### callgrind |
| |
| Only works on Linux. |
| Make sure valgrind is installed: |
| ```shell |
| $ valgrind |
| ``` |
| |
| Add `ro_segment_workaround_for_valgrind=true` to `args.gn` for symbols to appear |
| correctly. |
| |
| This is a slow and accurate profiler. Expect variations of around 100 |
| instructions. However, this takes about 50 times longer to run than perf stat. |
| |
| Use this when looking for small variations (< 1%). |
| |
| One advantage is that callgrind can generate `callgrind.out` files (by passing |
| --output-dir to safetynet_compare.py), which contain profiling information that |
| can be analyzed to find the cause of a regression. KCachegrind is a good |
| visualizer for these files. |
| |
| #### none |
| |
| Run without any profiler, giving a performance score of 1 always. useful for |
| running image comparisons or debugging the script. |
| |
| ### Common Options |
| |
| Arguments commonly passed to safetynet_compare.py. |
| |
| * --profiler: described above. |
| * --build-dir: this specified the build config with a relative path from the |
| pdfium src directory to the build directory. Defaults to out/Release. |
| * --output-dir: where to place the profiling output files. These are |
| callgrind.out.[test_case] files for callgrind, perfstat does not produce them. |
| By default they are not written. |
| * --case-order: sort test case results according to this metric. Can be "after", |
| "before", "ratio" and "rating". If not specified, sort by path. |
| * --this-repo: use the repository where the script is instead of checking out a |
| temporary one. This is faster and does not require downloads. Although it |
| restores the state of the local repo, if the script is killed or crashes the |
| uncommitted changes can remain stashed and you may be on another branch. |
| |
| ### Other Options |
| |
| Most of the time these don't need to be used. |
| |
| * --build-dir-before: if comparing different build dirs (say, to test what a |
| flag flip does), specify the build dir for the “before” branch here and the |
| build dir for the “after” branch with --build-dir. |
| * --interesting-section: only the interesting section should be measured instead |
| of all the execution of the test harness. This only works in debug, since in |
| release the delimiters are stripped out. This does not work to compare branches |
| that don’t have the callgrind delimiters, as it would otherwise be unfair to |
| compare a whole run vs the interesting section of another run. |
| * --machine-readable: output a json with the results that is easier to read by |
| code. |
| * --num-workers: how many workers to use to parallelize test case runs. Defaults |
| to # of CPUs in the machine. |
| * --threshold-significant: highlight differences that exceed this value. |
| Defaults to 0.02. |
| * --tmp-dir: directory in which temporary repos will be cloned and downloads |
| will be cached, if --this-repo is not enabled. Defaults to /tmp. |
| |
| ## Setup a nightly job |
| |
| Create a separate checkout of pdfium in a new directory, for example `~/job`. |
| The safetynet_job.py script will run from this directory. This checkout needs to |
| be `git pull`'ed when there are changes to the SafetyNet scripts, but otherwise |
| it can be left alone. |
| |
| Create a directory to contain the job results, for example `~/job_results`. In |
| each run, a `.log` file with the results will be written to this directory and a |
| subdirectory will be created with the other artifacts. |
| |
| Setup a cron job to run safetynet_job.py nightly. The example below runs it at |
| 1:42 AM, over the corpus in two directories: `~/pdf_samples/thousand_pdfs` and |
| `~/pdf_samples/i18n` |
| |
| ```shell |
| @ crontab -e |
| 42 1 * * * bash -lc '~/job/pdfium/testing/tools/safetynet_job.py ~/job_results ~/pdf_samples/thousand_pdfs ~/pdf_samples/i18n --output-to-log >> ~/job_results/cron_nightly.log 2>&1' |
| ``` |
| |
| The first time the job runs, it will just create a checkpoint as |
| `~/job_results/last_revision_covered`. From then on, since a checkpoint is |
| available, each run will compare performance with the last checkpoint and update |
| the checkpoint. |
| |
| ## Run image comparison |
| |
| Pass the `--png-dir` option pointing at an output directory to compare the output |
| images from rendering the "before" and the "after" branches with pdfium_test. |
| |
| ```shell |
| $ mkdir ~/output_images |
| $ testing/tools/safetynet_compare.py ~/pdf_samples --branch-before before_visual_changes --branch-after after_visual_changes --png-dir ~/output_images |
| ``` |
| |
| This will output and automatically open a `~/output_images/compare.html` file |
| showing the before/after and the diff. Hover the mouse cursor over the |
| before/after image on the left for an easier visual comparison. The "before" |
| image is displayed until the cursor hovers over the image, which is then |
| replaced with the "after" image. |
| |
| It is recommended to use `--profiler=none` with this option. |