Automating Design Review with Visual Regression

Posted on:

source
How to utilise visual regression testing to automate design review (UAT) within your organisation, a talk that was given during 2020 at VueJS London
https://github.com/csi-lk/visual-regression-how-to

Automating Design Review with Visual Regression

πŸ‡¦πŸ‡Ί "dartah" = data "rehpo" = repo

^ I have an Australian accent and mispronounce things

What are we trying to solve?

  • CSS Sucks [insert family guy blinds gif here]
  • Manual regression sucks
  • UAT sucks

^ As I start all of my talks, what are we trying to solve?

^ eg. The shadow is off, the color is different from the design ^

^ QA has to keep so many things in their head at once, they plain miss design changes

^ Anyone who has worked with a designer knows how annoying the 1 pixel issues are

^ This UAT process takes a long time and seems to be disjointed from the developer

            .─.                      .─.                                                      .─.
           (   )    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”    (   )   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”         β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”     (   )
           β”Œ`─'┐    β”‚  Ticket└──    β”Œ`─'┐   β”‚          β”‚   β”‚        β”‚  Merge  β”‚   QA   β”‚     β”Œ`─'┐
           β”‚Des│───>β”‚  Story   │───>β”‚Dev│──>β”‚  Branch  │──>β”‚   PR   │────────>β”‚  STG   │──┬─>β”‚QA β”‚
           β”‚ignβ”‚    β”‚          β”‚    β”‚   β”‚   β”‚          β”‚   β”‚        β”‚         β”‚  UAT   β”‚  β”‚  β”‚   β”‚
           β””β”€β”€β”€β”˜    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β””β”€β”€β”€β”˜   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜   β””β”¬β”€β”€β”€β”€β”€β”€β”€β”˜         β””β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β”‚  β””β”€β”€β”€β”˜
                                      β–²                     β”œ>Tests Pass                  β”‚   .─.
                                      β”‚                     β””>Approval                    β”‚  (   )
                                      β”‚                                                   β”‚  β”Œ`─'┐
                                      β”‚                                                   β”‚  β”‚Desβ”‚
                                      β”‚                                                   └─>β”‚ignβ”‚
                                      β”‚                     Fails Something                  β””β”€β”¬β”€β”˜
                                      β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

^ Your current workflow probably consists of design > ticket > dev > pr > merge > qa / staging / uat regression > designer finds an issue > back to dev

^ If anyone has seen my other talks there's a theme of pushing testing as close to the front of the process as possible

^ So my proposal is, let's bring this design review process forward to the PR stage with Visual Regression Testing

So what is visual regression testing?

  • Screenshot diffing
  • Gold (master) vs new (feature branch)
  • Expose unintended changes
  • Give us a review process for the visual

^ Pretty much we're diffing screenshots

^ Between master vs feature or old vs new depending on what you're doing

^ Change thing here affects thing over there (dreaded 'user reporting something broken in IE')

^ Most importantly gives us a new stream of review process for visual

                          Ξ›
                         β•± β•²
                        β•±   β•²
                       β•± E2E β•²
                      ╱───────╲
                     β•±         β•²
                    β•±           β•²
                   β•± Integration β•²    <-- We are here
                  ╱───────────────╲
                 β•±                 β•²
                β•±                   β•²
               β•±        Unit         β•²
              ╱───────────────────────╲
             β•±                         β•²
            β•±                           β•²
           β•±           Static            β•²
           ───────────────────────────────

^ The testing pyramid

^ A type of integration test

^ You're not going to cover everything with visual regression, focus on the things that matter

^ Now lets get into it

Examples

^ Pros and cons to each tool

^ Percy plugs in well and has a ton of ready made examples for different tools eg. cypress etc.

^ Happo supports all browsers and I have to call out that the founder actually was emailing with me on Sunday due to github actions issues I had and eventually did a PR to this repo showing how it works in this context which is awesome

Example - Storybook Components

^ Storybook with HTML components, could be React or Vue

^ Here's something I prepared earlier

^ Your designer can come in and edit this - request changes

^ Approve all

Example - Responsive Website

  • Eg. Jekyll, Gatsby etc.
  • Build out to HTML files
  • Breakpoints is where things go wrong
  • Example PR with review step
  • Pairs well with full page designs

^ Lots of companies have static sites now from some content source that goes through something like Jekyll or Gatsby and spits out HTML files

^ Plug: my personal website csi.lk works exactly like this

^ Point percy at the porcelain dist with your .html glob and set it up do so some diffing

^ Once again designer comes in, approves and moves on


        .─.                      .─.                                                      .─.
       (   )    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”    (   )   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”         β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”     (   )
       β”Œ`─'┐    β”‚  Ticket└──    β”Œ`─'┐   β”‚          β”‚   β”‚        β”‚  Merge  β”‚   QA   β”‚     β”Œ`─'┐
       β”‚Des│───>β”‚  Story   │───>β”‚Dev│──>β”‚  Branch  │──>β”‚   PR   │────────>β”‚  STG   β”‚  ──>β”‚QA β”‚
       β”‚ignβ”‚    β”‚          β”‚    β”‚   β”‚   β”‚          β”‚   β”‚        β”‚         β”‚  UAT   β”‚     β”‚   β”‚
       β””β”€β”€β”€β”˜    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β””β”€β”€β”€β”˜   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜   β””β”¬β”€β”€β”€β”€β”€β”€β”€β”˜         β””β”€β”€β”€β”€β”€β”€β”€β”€β”˜     β””β”€β”€β”€β”˜
                                  β–²         .─.         β”œ>Tests Pass
                                  β”‚        (   )        β””>Approval
                           Fails Something β”Œ`─'┐                 β”‚
                                  β”‚        β”‚Desβ”‚                 β”‚
                                  └────────│ignβ”‚β—€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                           β””β”€β”€β”€β”˜

^ Remember our workflow from before?

^ Shorten the loop

Pros

  • Reduce manual QA time
  • Include designers in the process
  • Reduce the feedback loop
  • Get it right once

^ No more: this is a high risk change because I edited the color of the button

^ Shorten the feedback loop, put designers in the process, create ownership

Cons

  • Extra step in a PR
  • Extra tool to manage
  • Train designers on PR process
  • Takes time to run

^ As we mentioned your designers will need github access