Firefox screenshots can now be easily captured and compared in automation

Back in 2013, during Australis[1] development, I created a tool called mozscreenshots. The purpose of this tool was to help detect UI regressions and make it easier to review how the UI looks in various configurations on each of our supported platforms. The main hindrance to its regular usage was that it required developers and designers to install it on each machine, tying it up while the images were captured.

The great news is that mozscreenshots is now running in automation on Nightlies and on-demand for try pushes, making it much easier to capture images. Comparing the captured images to a reference/base version can now be done via a web interface which means the images don’t need to be downloaded for review.

mozscreenshots has already detected two issues during its first week in automation, and I hope that this tool will improve the quality of desktop Firefox by surfacing UI issues sooner. You can read about it in the documentation on MDN, but here’s a quick start guide:

Change detection via Nightly captures

In order to detect UI regressions, screenshots of some sets run on every Nightly. These are compared to the previous day’s Nightly, using the compare_screenshots tool (which uses ImageMagick’s `compare`). For now I’m manually kicking this off each day but soon I hope to automate this and have it send an email to interested parties for investigation.

Currently, only one run of "TabsInTitlebar,Tabs,WindowSize,Toolbars,LightweightThemes" occurs on Nightlies. I plan to add runs with the DevEdition theme, DevTools, and Preferences shortly, as the code for those are already written.

Capturing on Try

You can request screenshots be captured on a Try push for UI review or comparison to a know-good base by requesting the "mochitest-browser-screenshots" test job. You can specify what you would like captured by setting the MOZSCREENSHOTS_SETS environment variable with a comma-separated list of configurations like so:
try: -b o -p linux,linux64,macosx64,win32,win64 -t none
-u mochitest-browser-screenshots[Ubuntu,10.6,10.10,Windows XP,Windows 7,Windows 8]
--setenv MOZSCREENSHOTS_SETS=TabsInTitlebar,WindowSize,LightweightThemes


Note that the job is currently Tier 3 and "excluded" on TreeHerder so you will need to toggle both of those filters to see the jobs there with the symbol: "M[Tier-3] (ss)". Unlike Nightlies, Try pushes won’t capture any images by default if MOZSCREENSHOTS_SETS isn’t specified. This avoids capturing images when developers request all mochitest runs, but don’t really want the screenshots. The capturing is implemented as a mochitest-bc test in the "screenshots" subsuite, meaning configurations can use things like BrowserTestUtils and such. See fetch_screenshots if you would like to download the captured images.

Comparing screenshots

The simplest way to compare images is via the web UI at http://screenshots.mattn.ca/compare/ (Example: http://bit.ly/1Qv4uWD ). Simply provide the project and revision of a base push with images, like the Nightly rev. from about:buildconfig, and your new revision, like a try push with some patches to review.

In the background, the images are fetched from automation via fetch_screenshots, and then compared using compare_screenshots with the output displayed on the page. The first comparison for a pair of revisions can take several minutes, as around one thousand (5 platforms x 2 revisions x 100 screenshots) images need to be downloaded and compared for the default set of screenshots. The results are cached, so subsequent comparisons for the same revision are much faster.

There are a lot of opportunities for this tool (e.g. pulse integration, notifications, simplified bug filing based on differences, etc.), and I hope to continue to improve the workflow. Please file bugs on the capturing infrastructure blocking the "mozscreenshots" meta bug (1169179) and bugs related to the web UI, comparison and fetching at https://github.com/mnoorenberghe/mozscreenshots/issues. See also the list of issues and ideas at https://public.etherpad-mozilla.org/p/mozscreenshots which haven’t been filed yet.

Thank you to everyone who has helped with getting mozscreenshots to this point, specifically Felipe Gomes, Armen Zambrano Gasparnian, Joel Maher, Brian Grinstead, Kit Cambridge, and Justin Dolske.

[1] Australis was the code name of the Firefox UI redesign that launched April 29, 2014.