Visual Regression Testing with BackstopJS

- [David] Welcome to the first session

of MidCamp day three technically.

This is Visual Regression
Testing with BackstopJS.

My name is David Neeedham.

I'm a developer advocate at Pantheon.

What that means is I sort of sit between

the developers that are using
Pantheon in the real world

and the Pantheon developers
that are building it

and kind of reading new
functionality and things like that.

As part of this process,

I end up creating a lot of demo sites,

I end up creating a lot
of training curriculum,

and interacting with developers from

all sorts of different
backgrounds in WordPress

and Drupal and even other
technologies as well.

I'm based out of Champaign, Illinois

so just a couple hours south.

I've been doing Drupal
for about 11 years now.

The point of the training, session today,

is to learn a little bit about
what visual regression is,

what the problems are that
we're trying to solve with it,

and then to learn a little bit
about BackstopJS specifically

in a way that I am hoping
will be really approachable,

really easy for someone to pick up

and just try on your own site so you can

become comfortable with it
and then grow up from there.

I should also say this has originally

up to this point been a training,

either a half-day or a full-day training.

I've restructured the
slides and the format

and taken out some of the demos to

make it more of an hour
long session format

so we can go through it today.

However, my slides are available,

visual-regression.davidneedham.me.

This is something you
could actually go through,

probably not live right now,
but if you check that out,

all of the directions
for all of the commands

that I'm running are in
the slides all right there.

Visual regression testing.

The first question and the question I had

when I first heard about
visual regression testing is,

what is it?

If you take the definition of regression,

a regression is a return to a former

or less developed state.

As we're thinking about
visual regression testing,

we're trying to run a test to verify

that we're not regressing,

we're not going backwards in some sort of

functionality or design that we've built.

Typically, this comes up
in a few different ways.

One of them is just doing updates.

If you can overlook the fact that

this is a WordPress
screenshot and example,

all of these examples are relevant for

not just Drupal and WordPress,
but any website, any browser,

it should all be totally relevant.

When you're doing an update,

you don't expect anything
to be visually different.

You go in, you have a
security update, you run it.

If it works correctly,

it should look exactly the same and

work exactly the same way
before and after the update.

Here's a site, there
is an update available.

We run the updates and of course,

we want to check out the live site

so we kind of flip through it and we

take a look and familiarize
ourselves with the live site,

we make sure that everything
is working before the update,

everything looks okay.

Here's after the update under
a development environment.

We're not ready to
merge it up to live yet,

but let's just check out the site.

We click through it.

We scan.

Did anyone see the
difference between the two?

If you're doing this all the time,

you might have some tricks up your sleeve.

Maybe you put them
side-by-side and run a script

so that they'll scroll them side-by-side

and you can compare visually that way.

You should also probably be thinking about

other browsers, other
viewpoints, view ports.

This is desktop, maybe
you also care about mobile

so you wanna check that
and flip through them.

As you're seeing,

this is a problem that we as humans

are just not very good at solving.

We might be able to catch differences

that are very blatant and obvious,

but it's really difficult for
us to catch subtle differences

and that's why a tool like BackstopJS

that will highlight the differences

between two screenshots will
make our jobs a little easier.

Here is a BackstopJS report
highlighting the differences.

Just to show you here,

we have this little scrubber so you can

actually move this
scrubber back and forth.

You can see in this update,

the text on that search box changed.

It's a minor detail,

but it might be something
you wanna know about

depending on what's going on,

and it's something you
wouldn't necessarily

expect to happen in a just
mundane update that you're doing.

This is one of the things that
BackstopJS can catch for you.

It becomes even more
useful as we start again

thinking about multiple
browsers, multiple view ports,

it's able to account
for all of that for you.

Visual regression testing is able to

help you have superpowers.

You're able to leverage
computers and robots

so that you can notice these things

that are different more easily.

It should give you more time to focus on

the things that you actually care about

and really highlight the things that

you need to pay attention to.

There's a bunch of different

visual regression tools out there.

We are going to be focusing on BackstopJS,

but if you have a different
preference, you can do that.

There's also lots of third party paid

visual regression services out there

that you can pay a little bit of money to

and they'll take care of those for you.

We're just gonna be working with

running Backstop on our computer ourselves

and running tests manually at this point.

There are lots of benefits to using

a visual regression test of any kind.

First of all, you're gonna be taking these

screenshots of two versions of your site,

probably your live version
and your development version

whether it's up on Pantheon or up on

some staging site that you have,

or maybe live versus your local copy.

Maybe you wanna be testing as you're

developing a site to make sure that

you're only changing
things that you expect.

We're able to do that.

We're able to put in a reference URL,

which is the original,
the perfect live version,

versus our test URL,

which is gonna be our development
site or our local site.

We're also able to accommodate
multiple view ports.

If you know that you want
this to work on tablet

and mobile at these resolutions,
under these criteria,

you can set up your configuration file to

account for that and to generate a report

that has each of those
instances accounted for.

You also have options to
scan your entire site.

If you don't wanna have to worry about

manually adding a report or a scenario

for every single page on your site,

there are options like
Backstop Spider is a library

that will actually crawl
your site, grab all the URLs,

and then generate a report with that.

My personal feelings on that so far

in using this in the
real world has been that

it's not actually very useful to scan

your site and get every single URL because

the report that you generate takes forever

to go through and scan and do that.

Still, you might have a use
case where that's helpful.

Yes, you do get error reporting.

It's meant to be relatively
easy to write test symbols.

We'll see whether or not that's true

when we get into the code.

There are some limits to visual
regression testing though.

You might have noticed on
that screenshot earlier

when we were kind of
flipping through the gif

and we showed the report,

sometimes there are some
visual differences between

two versions of a site that are

not representative of an actual
change that you'd expect.

Maybe if you have embedded media

like a YouTube video or a Vimeo video,

sometimes they'll embed
random static on there

like loading image,

which will give you a failure
when it's not necessary.

Also sometimes, things
will just be slightly off

for some reasons that are a
little bit difficult or timing,

if you have two versions of a site,

one loads faster than the other.

There are ways to account
for a lot of these things,

but there are some
limitations that make it

a little bit more difficult to use

However, I still find
that this whole process

saves a lot of time.

It helps you identify the things that are

needing to be addressed.

Let's actually jump into using BackstopJS.

The way that this works is we
are using a headless Chrome.

We're not actually opening
up our Chrome browser

and it's not taking
screenshots that you can see.

You run the command and it
goes through and does it,

simulating it with headless
Chrome automatically.

You don't have to worry about your window

popping up or anything.

You can just use your command
line, run the command,

and step away or go do other work

and it will take care all of that for you.

You are also able to simulate
user interactions, meaning,

you can actually click on things,

you could wait for certain things to load,

you can preload cookies as well

to give people basically
a logged in instance

if you have a site that's
behind a paywall or something,

there are ways that
you can get around that

still using the command line.

You get this report that
we're gonna look at.

It's gonna pop up in our browser

and we'll use the scrubber
and all that stuff.

When you're thinking about automation,

you're also able to
accommodate a command line

interface to this with a pass-fail return.

As you're going through and
doing something like a script

or maybe you wanna automate
your updates on your site,

you could integrate a
visual regression task

that returns a pass-fail and then

does other things with the script,

the larger script that you're writing.

You do have the option
of either installing

BackstopJS on your project itself

or globally on your entire computer.

In this case, I've opted
to do it globally 'cause,

we'll see that in a minute,

but you'll have the option to then

just going and initialize Backstop on

every site that you're
working on if you'd like to,

but we'll see that in a minute.

We will also see a demo of
the standalone packaged app.

If you have a team that maybe doesn't

feel comfortable running
the command line a lot

or they don't want to have to
manually set up BackstopJS,

if you feel comfortable with Node,

you could package this up as a Node app

and you run a command and it does a

visual regression test for you all in one

sort of packaged neat bundle.

We'll also take a look at that later.

Ultimately, it comes down to
just a few simple commands,

and we'll be seeing those.

Again, everything is sort of defined

in this BackstopJS file.

It's a configuration file
and I'm just gonna explain

some of these pieces before we actually

jump in and start editing it.

We start off here at
the top with view ports.

View ports are just your dimensions of

the website that you want to capture.

We have one here for
phone and one for tablet,

maybe you also want to do desktop

or whatever other dimensions are relevant

for what you're trying to do.

Below that, we have a section
for scenarios where we can

define different pages on our website

and all the criteria for that.

We have a label which is describing

what this particular scenario is,

we have the URL which is
the URL that we're testing,

this is gonna be our development
URL or our local URL,

and then the reference URL which is the

live website that we're
testing it against.

Below that, we have a
bunch of other things.

There's ready event and
selector if we want to

wait until this thing is on the page

before we actually run the test,

we can delay the test a
certain amount of time

if we wanna account for maybe a loader

or a certain slide or something like that.

We also have options to
hide and remove selectors.

For example, if we have
that YouTube video that is

causing our test to
fail every single time,

we could either hide that or remove it

from the page completely
whenever the test is run

so it's not causing our test
to fail every single time.

Of course you can do the opposite of that

where instead of by default here,

it's selecting the entire document and

scanning and testing the entire document.

If you only wanted it to
test particular components

or if you are doing a style guide

with a component library that
you wanna be testing against,

you could add specific
selectors that it's testing

and it will generate a
report selector by selector,

which could be helpful sometimes.

Then there's all sorts of
other configurations here,

including changing the
engine and things like that

that we're just not gonna
have time to get into.

Now we're to the first demo where we are

actually going to be running
BackstopJS on a project.

I have a site.

It is a Pantheon Theater
website, pantheondemo.com,

and this is the live website.

It's a common website
that we use at Pantheon

for a few different demos that we run.

It has a slideshow, it has
a few different sections,

pretty easy, only a few different pages.

I also have a local copy.

I'm using Lando, which isn't
really relevant for this,

but it's just a local docker installation.

It's an exact copy and I'm
pretty sure it's an exact copy,

but of course I can always
run a BackstopJS test

to make sure that it's an exact copy.

We could get that going.

I'm gonna follow the directions here.

First of all, I need
to initialize Backstop.

If I jump over to my command line...

Sorry about that.

If I just do a Backstop in it,

it is going to set up the
Backstop project on this site.

It's gonna give me a Backstop.json file,

which has some pre-configured stuff in it,

it's gonna set up the
file structure in place

so it has a place to save the screenshots

that it's gonna be generating,

and just generally getting it ready to go.

When I'm ready to start configuring this

and getting it specific to my project,

I can edit the Backstop.json file.

Let's take a look at that.

It's gonna open it up in Atom,

which is my preferred code editor.

This is what you get by default.

You have phone and tablet
for your view ports.

If you wanted to add or remove those,

you can do whatever you want.

We also have an on ready
and on before script,

which isn't really useful
for me in this place.

You're just able to kind of
set up some stuff in advance.

I'm gonna leave it there just to show

that I don't have to remove anything,

but if I was optimizing this
for a real world scenario,

I'd probably remove those
'cause I'm not really using it.

Below that, I have the first scenario,

the BackstopJS homepage.

I'm gonna swap this out with
the Pantheon demo homepage.

I also have the option
here for the cookie path,

which again, I'm not
really using for this demo,

but I'll leave it in place.

We have the URL, which is
going to be my demo site,

my Lando site.

Let me put in the URL for that,

pantheonweeklydemosite.lndo.site,

and then the reference URL,

which is gonna be my live
site, pantheondemo.com.

I don't really have to
change anything else.

All those other variables are there.

I can change them if I want to,

I can look up the reference
and determine what these are,

but I'm actually just gonna
save this just as it is,

switch back to my command line,

and start with a Backstop reference.

This is gonna generate the
initial reference screenshots

for the reference site that I referenced.

That's a lot of references.

It's taking the live website,

creating screenshots for
the different view ports

for this homepage, and
then once that's done,

all I have to do is run a Backstop test

just as it says right
here to generate a report.

Now it's creating screenshots for my

test site on my local computer,

it's generating a report, comparing them.

I expected these to be the
same, but as you can see,

the reference is actually
a little bit faded.

It didn't get loaded
completely right away.

Let me just click through
the report real quick

and you can take a look.

I'll fix this issue.

We can take a look at the reference and

see the screenshot and kind
of see what it looks like.

We can jump over to the test,
see what that looks like.

We can also look at the DIFF to see

everything highlighted that's
different between the two,

and then we can actually
go to the scrubber

and we can see a comparison
of the two versions

and scrub them back and forth.

Let's actually resolve this issue.

The problem's that it's not fully loading

before the screenshot's taken

and this is in our reference image.

If we jump back to our code,

we have a few different options.

I could delay if I wanted to.

That might help just by saying
let's wait 10 milliseconds,

let's wait a full second,
something like that.

I could also go to the ready selector

and grab a class from the page and say,

when this is completely loaded,
then take the screenshot.

I'm gonna start just with the delay.

I'm gonna say, let's do 10.

It was in the reference image so I

need to redo my references.

Backstop reference.

Now let's do a Backstop test.

It still did it.

Then maybe it would make more sense

to grab one of those classes,
something on the page.

Let's see if we can find one real quick.

Let's go to our live
site, look at the code.

Let's just grab something
really generic like container.

Let's go over to the
ready selector, container,

and click save.

Let's generate some new reference images.

- [Man] Do you always have
to name them reference

or can you put any name you want?

Do you have to name it like the tag?

- [David] The process
is you Backstop in it

to generate the project
and then you just do

reference or regenerate
the reference images

and then test to compare the
live versus the reference.

I don't know if you could
create multiple versions

if you have a reference
one, reference two,

reference whatever.

Good question.

Backstop test now.

I think it still failed.

Look, this time it's on the test one.

Well that's strange.

This is the great thing about live demos.

Sometimes it doesn't
quite go how you'd expect.

That's okay.

You would massage the config file,

get it so that they equal the same thing,

and who knows the reason why
in this case why they're not.

You could also just ignore
the slideshow in this case.

If the slideshow is
consistently causing errors

when it's things that are just
it's not quite loading right,

you could just hide it or
remove it from the markup

and it wouldn't have to be
included in the test anymore.

This is one example, quickly
generating this report here.

Typically, the process is
just like I showed you.

You're modifying that config file,

you're doing reference and then test.

Once your reference is completed and it

feels like it's a good
representation like in this case,

we wouldn't have to regenerate the

reference every single time.

We could just make
whatever change we need to,

do test again, do test again
until we determine that

it's working the way we expect it to.

This brings up to the
second killer use case.

If you haven't guessed
it already, it's theming.

If you are a front-end developer

or if you make changes
on your local computer

on a regular basis and you're tweaking CSS

and things like that,

then it probably makes sense to have some

even very light version of
visual regression testing

because how often have you said,

I know exactly what this problem is.

I'm just gonna make a
quick little CSS change

and it's gonna totally fix this.

Then you realize, when we built the site,

that selector actually was on these

other pages and I forgot,
or we didn't whatever.

Whatever the scenario is,

that's a really difficult
situation to be in

because you think it's really easy

and hours later you're still debugging.

Let's fix that early and
have a test that we can run.

We can see exactly what pages are affected

and make sure that whatever
CSS change we're making

is only changing the
piece that we're expecting

and not accidentally
messing stuff up elsewhere.

It's easy.

With our specific example,

we have this Pantheon Theater website,

and let's just say our
marketing team has determined

through AB testing that if we change the

color of that buy tickets button from this

purple color to a bright red color,

it will sell more tickets,
it'll be more noticeable.

So we're like okay, no problem,
we can do that, that's easy.

Let's jump over to our CSS
and we'll make that change.

Let me go back.

I think I already have
my code editor open.

Let's go into my theme.

Let's go down to my CSS file down to this

deceptively unspecific class called CTA.

We have the background color, purple.

Let's change that to red, save.

Let me jump back and
just verify on my local

that it is working.

There we go.

It's now red.

Now what we'd do is
we'd run out test again.

Go back to our command
line, do Backstop test.

Now it's going to generate a report for us

based on those changes that we've made.

As we can see, again, if
you can ignore the top part,

which shouldn't be quite that way,

you can see pretty easily
we have the reference,

we have the test, and we
can see the difference.

We could go to the
scrubber if we wanted to

and actually compare and see it definitely

changed from one to the other,

or we could switch to
this version scrubber

and see exactly what
changed between the two.

Then if you combo this with maybe you have

a selection of pages on the
website that are high priority,

maybe you also include those as

part of your scenario as well.

You would go back to your code,

go into your Backstop.json file,

add an additional scenario here for

all of the pages that are
important to you to check,

and then you would
automatically in this report

not just get different view ports,

but you'd also get each
scenario listed out one by one.

You can go through and
see to make sure that yes,

the CTA only affected this one piece.

It didn't break things
or cause color changes

on other parts of the site as well.

- [Man] Are you saying you have to

specify each page in this scenario?

- [David] By default, yes.

There's another library
called Backstop Crawl

that will go through and crawl
every page of your website.

However, the report that
I just ran, the test,

was relatively quick because I have

two view ports and one scenario.

As soon as you start adding

three view ports and 100 scenarios,

or even three view ports and 10 scenarios,

you're having to wait longer
and longer for it to work.

You probably want rather than scanning

all 100 pages of your website,

you probably wanna pick a subset of

here's my template for an article,

here's my template for a
basic page, and a staff bio,

here are the top level navigation pages

that I actually wanna check,

and probably not scan every
single page of your site.

- [Man] Could you use the scan
all to generate a template

and then you could cherry
pick what page you want?

- [David] Sure.

Anything you want.

Again, another thing
I've seen people do is

to have a siledad that if they

built their site using components

and they know what all the
design pieces are of their site,

they could use BackstopJS to pull out

each component by class
or by ID or whatever

and then have this on this page list out

all the things that have
changed with each component,

with each CSS change you've made.

That's killer use case
number two, theming.

As you're going along and building a site,

it's easy for you to do this.

Of course, it is also easy to automate

the process a little bit more.

We're gonna be looking
at continuous integration

for the next piece.

We're not going to have time
to show any of these pieces,

but just conceptually,

let's think a little bit about how we can

let robots do the parts
of our job that are

a little bit tedious,

that we're not super great at as humans.

One great example is on Cron.

Maybe we want to automatically
run updates on our site.

We don't want to have to think
about Drupal updates anymore,

we just want them to be automated.

If everything works how we expect it to,

just push those up to the live site.

We don't want human staff
to be in the middle,

between the next Drupal gen and our site.

We can do that.

We have a script that we wrote on Pantheon

that goes through this
continuous integration process.

It's a demo script so take
it with a grain of salt,

but you can modify it, you can
do whatever you want with it.

Basically on Cron, it goes out,

checks for updates on your site.

If it detects that there are updates,

it will go back to Pantheon,

create a branch or a
multi dev environment,

a dedicated development
environment for this update,

it will apply the updates
from the safe space,

it will then do visual regression testing

and behavior testing and a
few other types of testing

to make sure everything is working

after the updates have been applied.

If everything passes without any issues,

then it will create backups on all of

your other environments,

your dev, test, and live environments,

and then deploy it up to the live site

without me having to actually
intervene in any way.

If you would prefer to have some say

in when these updates get applied or

to have a human come in and make sure

that it's going to work,

you could change this to just say

notify me on Slack when
updates are available,

show me links to the test results,

let me look over it to make
sure everything looks right,

and then you could go in and actually

click the button to deploy
it up to your live site.

This is one example of
kind of integrating,

automating the process of
using visual regression testing

so that you're more
confident in your updates

before clicking the button to apply them.

Another great example of automating

visual regression testing is on a commit.

We've so far talked and the demo I showed

was I'm running BackstopJS on my computer

and as I'm developing a site
or theming it or whatever,

I'm testing for myself to make sure that

I'm not introducing any problems.

We could also introduce
a visual regression test

on a commit or as part of a poll request

that happens automatically on GitHub.

At DrupalCon, we're gonna
be doing a training called

Automated Workflows in Drupal Eight with

GitHub, Composer and CircleCI.

The whole point of this is you
should be able to as a team,

working with a Drupal Eight
site built with Composer,

be able to collaborate with other

developers in the team through just

creating commits and doing pull requests.

If you are working on some
new piece of functionality,

you've been committing,
committing, committing,

you're like this feature is completed

and I'm ready to merge
it into the main project,

you would create a pull request,

that would pull all the commits together,

present it in front of your lead developer

or whomever is doing
the review so they can

look at all of the commits,

but if you have automated testing involved

in this point as well,

then it would actually go
in and show you the reports.

In this case, we're building
the site with Composer

and continuous integration,

spitting that over to Pantheon as

this is the completed
built version of the site,

and then hitting it with
all the automated tests,

and then putting the
results of those tests

in the pull request so the lead developer,

the person reviewing doesn't have to

manually go through line
by line to make sure that

everything's working.

They don't have to go
to the development site

to make sure everything's working.

All they have to do is
review the test results,

which could include
visual regression testing,

but also behavior
testing, clicking through,

and functionality testing,

as well as code review testing.

If you have certain
standards for code review,

with how your code should be on your site,

that's another testing
that you could introduce

and make sure it conforms
to that before the

lead developer has to actually intervene.

Again, another way that you
could automate this process

of using visual regression testing.

My whole takeaway, what
I'm trying to kind of

introduce to you all is that you can

run BackstopJS on your
computer pretty easily,

pretty lightly, pretty quickly too,

just getting it set up and run with it.

Another example is the
standalone Node app.

I've ran this as a class
a few times as I mentioned

and occasionally,

someone is unable to
get BackstopJS set up.

Occasionally it's because
of their operating system

or prerequisites or some other reason.

For whatever reason it is,

it doesn't work on their computer.

In that situation though,

the standalone app has
still worked for them.

It has let them kind of
interact with Backstop.

This was just again a simple example demo

that one of my colleagues created

to show that this is possible and it

makes the process of running Backstop

a little bit easier for people that

don't wanna have to mainly
mess with all the bits.

We have a example repo.

You can go out and grab it.

If we were in a class, I would
invite you all to do this,

but you can definitely do this

after the session today if you'd like to.

I've already got this set up.

The script that you saw
that was loading earlier

hadn't quite completed
so this may not work,

but we'll give it a try anyway.

We have this visual
regression Backstop demo.

You clone it locally, you get into it,

you create a branch so you can

make changes without
affecting anything else,

and then we do an NPM install.

Let's actually go back and try that again.

This is the part that I was trying

to do before the session started

so we wouldn't have to
wait for the NPM install,

but it had timed out which is a

little bit concerning for me.

Let me go ahead and describe what this is

supposed to be doing.

Once it gets installed,

it's going to give us an interface

for just running our site.

As a package application,

it's going to run BackstopJS on our site

and get it all up and generating a report.

It already has that configuration
file pre-configured.

All the bits are taken care of.

You just run it and it
generates the report.

Pretty easy.

Let me go ahead and skip since that's

still stuck on the one
it was stuck on before.

That's one example.

You just run it and it generates
the report and it's done.

You don't have to mess with
any configuration at all.

The next part of that is
the multi site example.

If you have multiple sites that you need

to test on a regular basis,

maybe your support team is responsible for

you could write an NPM app or a Node app

that has all of those sites preloaded

into this standalone NPM app,

and then when you run the command,

it asks you which site you'd
like to run the report on.

You go down the list, you select it,

and then it actually opens
it up and runs the report.

You're able to preload all these

configurations side-by-side,

click the button, and have the report run.

It's intended for people that wouldn't

want to have to set this
up manually themselves.

It just would be able to run standalone.

You still do have the options though

of modifying the configuration file.

If you wanted to download
this as a starting point,

make changes to it, and still
have the standalone app,

you could do that as well.

It's still not working.

No problem.

Let's keep on going.

This is running the
multiple site example one.

Hopefully it will work better for you

when you try it later.

I'll take a look after the class and

try to see what the problem might be.

Any questions?

I apologize.

It's a little bit short
'cause I'm trying to

work on the timing coming down from a

half-day class to a session.

Questions?

- [Woman] There was an
area where you're doing

continuous integration.

Are you running Backstop
inside the container?

- [David] In those scenarios,

you're running Backstop as
part of continuous integration.

We are running BackstopJS in CircleCI,

in the continuous integration process,

against in this case our Pantheon
site, our reference site.

We're creating the site
in continuous integration

and looking out at the
live site, wherever it is.

- [Woman] But is Backstop itself installed

inside the container in that case?

- [David] It's installed in the container

on continuous integration,

but it's not installed
over on the Pantheon site.

- [Woman] 'Cause where I'm going with this

is I've played with this
and I got it to work,

but when I try to get it to work

within a Doxel container,

I had this problem where I
could point it to google.com

and it would see Google and run.

Obviously I don't wanna test Google,

but I couldn't see my own Doxel site

within the same container.

It just kept saying no site.

I got really stuck.

Oh well.

- [David] I'm sorry.

I don't know much about that.

Sorry, I don't know.

- [Woman] No problem.

- [David] Yes sir.

- [Man] What about sites that use

heavy unparalax or heavy on the ajax

or the things like that,

how do you know how long to wait

or are there effects that
you can get it to say

ajax is down or resource crawl so far?

- [David] Those are the sorts of things

that you can into with
the advanced JavaScript.

At the beginning, there's all
this on before and on ready

JavaScript that you can introduce.

There are things you can do in there

that I haven't done anything with before.

That's where you'd go to accommodate

some of those situations.

Yes sir.

- [Man] Is the idea then if you change

the color of the button for example.

You know you're introducing
something it's gonna flag.

Does that mean that you introduced

a new reference in your commits if

your CI is gonna be
checking it every time?

Is that how you deal with it?

- [David] That's a good question.

I've spoken to different
people of different opinions.

Some of them don't actually commit

any of the Backstop stuff.

They don't commit the images.

In general, putting images in
a commit isn't a great idea.

This Backstop stuff, they just run locally

on their computer and that's it.

If you wanted to, package it up,

save the configuration file and

give that to other people,
that probably makes sense.

I don't know if it makes sense to

send the reference images.

Maybe.

You could.

I haven't seen that in my scenario.

- [Man] I guess I'm just
trying to get a picture

of if you're wiring it into CircleCI.

I got majority of the tests going

and I've got the NPM test going and

maybe I'm even doing some shallow
rendering in the NPM test.

How do I have it tell me by the way,

you introduced a visual
regression and you didn't mean to?

- [David] All the demo
scripts that we've set up

with the continuous integration process,

because the environment's
being destroyed and rebuilt

every time the continuous
integration process fires,

it regenerates the reference images

and it starts the test
from the very beginning.

In those cases, you know there's a change,

you know to expect that there's

gonna be an error or a problem.

Then you just have to review it.

If you're asking, is there
a way to flag something

as being this is expected,

don't give me a red flag for this,

I'm not aware of a way to do that

unless you introduce the reference images.

That could work.

I just haven't done it before.

- [Man] You change something in your code,

it fails the test, and you
have to update the test

if you're gonna use it at that level to

have it be an all type stopper CI,

or you could generate the reference

to look at somewhere between the CI

and just know that what's
in there is expected

before you test.

- [David] Yes sir, a question.

- [Man] When you have a component with

dynamic content like a
newsfeed or an event block

or something that you want to

visually regress test for regressions,

what would be a good
approach for handling that?

- [David] That's really tough.

That's one of those areas with the

visual regression limitations.

It doesn't deal with dynamic
content very well because

if you have a live site that is

constantly getting new content,

you have your local version that you're

not able to pull the database
from every single time,

you're gonna fail because
the content's different

in this dynamic block section.

In those situations, you
can ignore that section.

You're not able to accommodate

all the different things that
could happen to that section.

That's what I've done in the past is

ignore this dynamic block because

it's always gonna be failing.

- [Man] Unless you're trying to test it.

- [David] If you're trying to test it,

that's a little tricky.

- [Man] It almost seems
like you would need

to have a fixture database that you're

retrieving as your
production in that instance

to compare at the end.

It sounds like a lot of work.

- [David] With the
components testing method.

If you have all these components
or template-type pages

available on your live website that are

really just used as a style guide,

then you could certainly test against that

on your local versus the live style guide

and with some assurance that it's

gonna work the way you expect.

Question?

- [Man] Since this has
a fairly civic scope,

what are the test tools
to use to compliment this?

What do you think are
some complimentary tests?

- [David] What are some
complimentary tests?

It depends on the scenario because

with the automated section,
continuous integration section,

there's behavior testing where people

will log into the website
and make sure that

you're able to create content and

you're able to do the various
pieces of functionality,

things like that.

Is that what you're talking about?

- [Man] What tools would
have minimum overlap of this

in terms of testing?

I'm looking at the general
topography of your testing.

- [David] That's a good question.

I'm not sure actually.

With my own development process,
it's usually pretty light.

This is an improvement
over what I did before,

which was just visually look at it.

I don't have a very complicated process.

Real quick, does anyone
else have any suggestions

or ideas about where this could fit into

an overall larger testing process?

- [Man] My system's on
HP and test on code.

- [Man] I don't get the
visual, but I get the markup.

I'll often lock out here's the json

response to the paper we built around.

It pulls out a response from an actual

database to mock this so I take my mocks.

- [David] I'll be available.

I'll be at Pantheon for
the rest of the day.

Please do leave feedback, it's important,

and especially because like I said,

this is newly condensed
into session format.

There are obviously things
that I can fix about it.

Please let me know.

What are some things I could add or remove

or modify to be even better for this?

Thank you all very much.
Captions made possible by ClarityPartners.com
Chicago area Drupal consultants

(class claps)
Captions made possible by ClarityPartners.com
Chicago area Drupal consultants

The question was, will this
be on the Pantheon site?

I don't think it'll be on Pantheon,

but it'll be on MidCamp.

- [Man] No I mean the one
that you're doing in Seattle.

- [David] The one I'm doing in Seattle

is a paid training through DrupalCon

so we're not gonna have a
recording of that one available.

- [Man] Maybe after?

- [David] I'm not sure.

I don't know.

- [Man] I don't think so.

'Cause that's exactly what I wanna do.
 

Captions made possible by ClarityPartners.com. Chicago area Drupal consultants