- [David] Welcome to the first session
of MidCamp day three technically.
This is Visual Regression
Testing with BackstopJS.
My name is David Neeedham.
I'm a developer advocate at Pantheon.
What that means is I sort of sit between
the developers that are using
Pantheon in the real world
and the Pantheon developers
that are building it
and kind of reading new
functionality and things like that.
As part of this process,
I end up creating a lot of demo sites,
I end up creating a lot
of training curriculum,
and interacting with developers from
all sorts of different
backgrounds in WordPress
and Drupal and even other
technologies as well.
I'm based out of Champaign, Illinois
so just a couple hours south.
I've been doing Drupal
for about 11 years now.
The point of the training, session today,
is to learn a little bit about
what visual regression is,
what the problems are that
we're trying to solve with it,
and then to learn a little bit
about BackstopJS specifically
in a way that I am hoping
will be really approachable,
really easy for someone to pick up
and just try on your own site so you can
become comfortable with it
and then grow up from there.
I should also say this has originally
up to this point been a training,
either a half-day or a full-day training.
I've restructured the
slides and the format
and taken out some of the demos to
make it more of an hour
long session format
so we can go through it today.
However, my slides are available,
This is something you
could actually go through,
probably not live right now,
but if you check that out,
all of the directions
for all of the commands
that I'm running are in
the slides all right there.
Visual regression testing.
The first question and the question I had
when I first heard about
visual regression testing is,
what is it?
If you take the definition of regression,
a regression is a return to a former
or less developed state.
As we're thinking about
visual regression testing,
we're trying to run a test to verify
that we're not regressing,
we're not going backwards in some sort of
functionality or design that we've built.
Typically, this comes up
in a few different ways.
One of them is just doing updates.
If you can overlook the fact that
this is a WordPress
screenshot and example,
all of these examples are relevant for
not just Drupal and WordPress,
but any website, any browser,
it should all be totally relevant.
When you're doing an update,
you don't expect anything
to be visually different.
You go in, you have a
security update, you run it.
If it works correctly,
it should look exactly the same and
work exactly the same way
before and after the update.
Here's a site, there
is an update available.
We run the updates and of course,
we want to check out the live site
so we kind of flip through it and we
take a look and familiarize
ourselves with the live site,
we make sure that everything
is working before the update,
everything looks okay.
Here's after the update under
a development environment.
We're not ready to
merge it up to live yet,
but let's just check out the site.
We click through it.
Did anyone see the
difference between the two?
If you're doing this all the time,
you might have some tricks up your sleeve.
Maybe you put them
side-by-side and run a script
so that they'll scroll them side-by-side
and you can compare visually that way.
You should also probably be thinking about
other browsers, other
viewpoints, view ports.
This is desktop, maybe
you also care about mobile
so you wanna check that
and flip through them.
As you're seeing,
this is a problem that we as humans
are just not very good at solving.
We might be able to catch differences
that are very blatant and obvious,
but it's really difficult for
us to catch subtle differences
and that's why a tool like BackstopJS
that will highlight the differences
between two screenshots will
make our jobs a little easier.
Here is a BackstopJS report
highlighting the differences.
Just to show you here,
we have this little scrubber so you can
actually move this
scrubber back and forth.
You can see in this update,
the text on that search box changed.
It's a minor detail,
but it might be something
you wanna know about
depending on what's going on,
and it's something you
expect to happen in a just
mundane update that you're doing.
This is one of the things that
BackstopJS can catch for you.
It becomes even more
useful as we start again
thinking about multiple
browsers, multiple view ports,
it's able to account
for all of that for you.
Visual regression testing is able to
help you have superpowers.
You're able to leverage
computers and robots
so that you can notice these things
that are different more easily.
It should give you more time to focus on
the things that you actually care about
and really highlight the things that
you need to pay attention to.
There's a bunch of different
visual regression tools out there.
We are going to be focusing on BackstopJS,
but if you have a different
preference, you can do that.
There's also lots of third party paid
visual regression services out there
that you can pay a little bit of money to
and they'll take care of those for you.
We're just gonna be working with
running Backstop on our computer ourselves
and running tests manually at this point.
There are lots of benefits to using
a visual regression test of any kind.
First of all, you're gonna be taking these
screenshots of two versions of your site,
probably your live version
and your development version
whether it's up on Pantheon or up on
some staging site that you have,
or maybe live versus your local copy.
Maybe you wanna be testing as you're
developing a site to make sure that
you're only changing
things that you expect.
We're able to do that.
We're able to put in a reference URL,
which is the original,
the perfect live version,
versus our test URL,
which is gonna be our development
site or our local site.
We're also able to accommodate
multiple view ports.
If you know that you want
this to work on tablet
and mobile at these resolutions,
under these criteria,
you can set up your configuration file to
account for that and to generate a report
that has each of those
instances accounted for.
You also have options to
scan your entire site.
If you don't wanna have to worry about
manually adding a report or a scenario
for every single page on your site,
there are options like
Backstop Spider is a library
that will actually crawl
your site, grab all the URLs,
and then generate a report with that.
My personal feelings on that so far
in using this in the
real world has been that
it's not actually very useful to scan
your site and get every single URL because
the report that you generate takes forever
to go through and scan and do that.
Still, you might have a use
case where that's helpful.
Yes, you do get error reporting.
It's meant to be relatively
easy to write test symbols.
We'll see whether or not that's true
when we get into the code.
There are some limits to visual
regression testing though.
You might have noticed on
that screenshot earlier
when we were kind of
flipping through the gif
and we showed the report,
sometimes there are some
visual differences between
two versions of a site that are
not representative of an actual
change that you'd expect.
Maybe if you have embedded media
like a YouTube video or a Vimeo video,
sometimes they'll embed
random static on there
like loading image,
which will give you a failure
when it's not necessary.
Also sometimes, things
will just be slightly off
for some reasons that are a
little bit difficult or timing,
if you have two versions of a site,
one loads faster than the other.
There are ways to account
for a lot of these things,
but there are some
limitations that make it
a little bit more difficult to use
However, I still find
that this whole process
saves a lot of time.
It helps you identify the things that are
needing to be addressed.
Let's actually jump into using BackstopJS.
The way that this works is we
are using a headless Chrome.
We're not actually opening
up our Chrome browser
and it's not taking
screenshots that you can see.
You run the command and it
goes through and does it,
simulating it with headless
You don't have to worry about your window
popping up or anything.
You can just use your command
line, run the command,
and step away or go do other work
and it will take care all of that for you.
You are also able to simulate
user interactions, meaning,
you can actually click on things,
you could wait for certain things to load,
you can preload cookies as well
to give people basically
a logged in instance
if you have a site that's
behind a paywall or something,
there are ways that
you can get around that
still using the command line.
You get this report that
we're gonna look at.
It's gonna pop up in our browser
and we'll use the scrubber
and all that stuff.
When you're thinking about automation,
you're also able to
accommodate a command line
interface to this with a pass-fail return.
As you're going through and
doing something like a script
or maybe you wanna automate
your updates on your site,
you could integrate a
visual regression task
that returns a pass-fail and then
does other things with the script,
the larger script that you're writing.
You do have the option
of either installing
BackstopJS on your project itself
or globally on your entire computer.
In this case, I've opted
to do it globally 'cause,
we'll see that in a minute,
but you'll have the option to then
just going and initialize Backstop on
every site that you're
working on if you'd like to,
but we'll see that in a minute.
We will also see a demo of
the standalone packaged app.
If you have a team that maybe doesn't
feel comfortable running
the command line a lot
or they don't want to have to
manually set up BackstopJS,
if you feel comfortable with Node,
you could package this up as a Node app
and you run a command and it does a
visual regression test for you all in one
sort of packaged neat bundle.
We'll also take a look at that later.
Ultimately, it comes down to
just a few simple commands,
and we'll be seeing those.
Again, everything is sort of defined
in this BackstopJS file.
It's a configuration file
and I'm just gonna explain
some of these pieces before we actually
jump in and start editing it.
We start off here at
the top with view ports.
View ports are just your dimensions of
the website that you want to capture.
We have one here for
phone and one for tablet,
maybe you also want to do desktop
or whatever other dimensions are relevant
for what you're trying to do.
Below that, we have a section
for scenarios where we can
define different pages on our website
and all the criteria for that.
We have a label which is describing
what this particular scenario is,
we have the URL which is
the URL that we're testing,
this is gonna be our development
URL or our local URL,
and then the reference URL which is the
live website that we're
testing it against.
Below that, we have a
bunch of other things.
There's ready event and
selector if we want to
wait until this thing is on the page
before we actually run the test,
we can delay the test a
certain amount of time
if we wanna account for maybe a loader
or a certain slide or something like that.
We also have options to
hide and remove selectors.
For example, if we have
that YouTube video that is
causing our test to
fail every single time,
we could either hide that or remove it
from the page completely
whenever the test is run
so it's not causing our test
to fail every single time.
Of course you can do the opposite of that
where instead of by default here,
it's selecting the entire document and
scanning and testing the entire document.
If you only wanted it to
test particular components
or if you are doing a style guide
with a component library that
you wanna be testing against,
you could add specific
selectors that it's testing
and it will generate a
report selector by selector,
which could be helpful sometimes.
Then there's all sorts of
other configurations here,
including changing the
engine and things like that
that we're just not gonna
have time to get into.
Now we're to the first demo where we are
actually going to be running
BackstopJS on a project.
I have a site.
It is a Pantheon Theater
and this is the live website.
It's a common website
that we use at Pantheon
for a few different demos that we run.
It has a slideshow, it has
a few different sections,
pretty easy, only a few different pages.
I also have a local copy.
I'm using Lando, which isn't
really relevant for this,
but it's just a local docker installation.
It's an exact copy and I'm
pretty sure it's an exact copy,
but of course I can always
run a BackstopJS test
to make sure that it's an exact copy.
We could get that going.
I'm gonna follow the directions here.
First of all, I need
to initialize Backstop.
If I jump over to my command line...
Sorry about that.
If I just do a Backstop in it,
it is going to set up the
Backstop project on this site.
It's gonna give me a Backstop.json file,
which has some pre-configured stuff in it,
it's gonna set up the
file structure in place
so it has a place to save the screenshots
that it's gonna be generating,
and just generally getting it ready to go.
When I'm ready to start configuring this
and getting it specific to my project,
I can edit the Backstop.json file.
Let's take a look at that.
It's gonna open it up in Atom,
which is my preferred code editor.
This is what you get by default.
You have phone and tablet
for your view ports.
If you wanted to add or remove those,
you can do whatever you want.
We also have an on ready
and on before script,
which isn't really useful
for me in this place.
You're just able to kind of
set up some stuff in advance.
I'm gonna leave it there just to show
that I don't have to remove anything,
but if I was optimizing this
for a real world scenario,
I'd probably remove those
'cause I'm not really using it.
Below that, I have the first scenario,
the BackstopJS homepage.
I'm gonna swap this out with
the Pantheon demo homepage.
I also have the option
here for the cookie path,
which again, I'm not
really using for this demo,
but I'll leave it in place.
We have the URL, which is
going to be my demo site,
my Lando site.
Let me put in the URL for that,
and then the reference URL,
which is gonna be my live
I don't really have to
change anything else.
All those other variables are there.
I can change them if I want to,
I can look up the reference
and determine what these are,
but I'm actually just gonna
save this just as it is,
switch back to my command line,
and start with a Backstop reference.
This is gonna generate the
initial reference screenshots
for the reference site that I referenced.
That's a lot of references.
It's taking the live website,
creating screenshots for
the different view ports
for this homepage, and
then once that's done,
all I have to do is run a Backstop test
just as it says right
here to generate a report.
Now it's creating screenshots for my
test site on my local computer,
it's generating a report, comparing them.
I expected these to be the
same, but as you can see,
the reference is actually
a little bit faded.
It didn't get loaded
completely right away.
Let me just click through
the report real quick
and you can take a look.
I'll fix this issue.
We can take a look at the reference and
see the screenshot and kind
of see what it looks like.
We can jump over to the test,
see what that looks like.
We can also look at the DIFF to see
everything highlighted that's
different between the two,
and then we can actually
go to the scrubber
and we can see a comparison
of the two versions
and scrub them back and forth.
Let's actually resolve this issue.
The problem's that it's not fully loading
before the screenshot's taken
and this is in our reference image.
If we jump back to our code,
we have a few different options.
I could delay if I wanted to.
That might help just by saying
let's wait 10 milliseconds,
let's wait a full second,
something like that.
I could also go to the ready selector
and grab a class from the page and say,
when this is completely loaded,
then take the screenshot.
I'm gonna start just with the delay.
I'm gonna say, let's do 10.
It was in the reference image so I
need to redo my references.
Now let's do a Backstop test.
It still did it.
Then maybe it would make more sense
to grab one of those classes,
something on the page.
Let's see if we can find one real quick.
Let's go to our live
site, look at the code.
Let's just grab something
really generic like container.
Let's go over to the
ready selector, container,
and click save.
Let's generate some new reference images.
- [Man] Do you always have
to name them reference
or can you put any name you want?
Do you have to name it like the tag?
- [David] The process
is you Backstop in it
to generate the project
and then you just do
reference or regenerate
the reference images
and then test to compare the
live versus the reference.
I don't know if you could
create multiple versions
if you have a reference
one, reference two,
Backstop test now.
I think it still failed.
Look, this time it's on the test one.
Well that's strange.
This is the great thing about live demos.
Sometimes it doesn't
quite go how you'd expect.
You would massage the config file,
get it so that they equal the same thing,
and who knows the reason why
in this case why they're not.
You could also just ignore
the slideshow in this case.
If the slideshow is
consistently causing errors
when it's things that are just
it's not quite loading right,
you could just hide it or
remove it from the markup
and it wouldn't have to be
included in the test anymore.
This is one example, quickly
generating this report here.
Typically, the process is
just like I showed you.
You're modifying that config file,
you're doing reference and then test.
Once your reference is completed and it
feels like it's a good
representation like in this case,
we wouldn't have to regenerate the
reference every single time.
We could just make
whatever change we need to,
do test again, do test again
until we determine that
it's working the way we expect it to.
This brings up to the
second killer use case.
If you haven't guessed
it already, it's theming.
If you are a front-end developer
or if you make changes
on your local computer
on a regular basis and you're tweaking CSS
and things like that,
then it probably makes sense to have some
even very light version of
visual regression testing
because how often have you said,
I know exactly what this problem is.
I'm just gonna make a
quick little CSS change
and it's gonna totally fix this.
Then you realize, when we built the site,
that selector actually was on these
other pages and I forgot,
or we didn't whatever.
Whatever the scenario is,
that's a really difficult
situation to be in
because you think it's really easy
and hours later you're still debugging.
Let's fix that early and
have a test that we can run.
We can see exactly what pages are affected
and make sure that whatever
CSS change we're making
is only changing the
piece that we're expecting
and not accidentally
messing stuff up elsewhere.
With our specific example,
we have this Pantheon Theater website,
and let's just say our
marketing team has determined
through AB testing that if we change the
color of that buy tickets button from this
purple color to a bright red color,
it will sell more tickets,
it'll be more noticeable.
So we're like okay, no problem,
we can do that, that's easy.
Let's jump over to our CSS
and we'll make that change.
Let me go back.
I think I already have
my code editor open.
Let's go into my theme.
Let's go down to my CSS file down to this
deceptively unspecific class called CTA.
We have the background color, purple.
Let's change that to red, save.
Let me jump back and
just verify on my local
that it is working.
There we go.
It's now red.
Now what we'd do is
we'd run out test again.
Go back to our command
line, do Backstop test.
Now it's going to generate a report for us
based on those changes that we've made.
As we can see, again, if
you can ignore the top part,
which shouldn't be quite that way,
you can see pretty easily
we have the reference,
we have the test, and we
can see the difference.
We could go to the
scrubber if we wanted to
and actually compare and see it definitely
changed from one to the other,
or we could switch to
this version scrubber
and see exactly what
changed between the two.
Then if you combo this with maybe you have
a selection of pages on the
website that are high priority,
maybe you also include those as
part of your scenario as well.
You would go back to your code,
go into your Backstop.json file,
add an additional scenario here for
all of the pages that are
important to you to check,
and then you would
automatically in this report
not just get different view ports,
but you'd also get each
scenario listed out one by one.
You can go through and
see to make sure that yes,
the CTA only affected this one piece.
It didn't break things
or cause color changes
on other parts of the site as well.
- [Man] Are you saying you have to
specify each page in this scenario?
- [David] By default, yes.
There's another library
called Backstop Crawl
that will go through and crawl
every page of your website.
However, the report that
I just ran, the test,
was relatively quick because I have
two view ports and one scenario.
As soon as you start adding
three view ports and 100 scenarios,
or even three view ports and 10 scenarios,
you're having to wait longer
and longer for it to work.
You probably want rather than scanning
all 100 pages of your website,
you probably wanna pick a subset of
here's my template for an article,
here's my template for a
basic page, and a staff bio,
here are the top level navigation pages
that I actually wanna check,
and probably not scan every
single page of your site.
- [Man] Could you use the scan
all to generate a template
and then you could cherry
pick what page you want?
- [David] Sure.
Anything you want.
Again, another thing
I've seen people do is
to have a siledad that if they
built their site using components
and they know what all the
design pieces are of their site,
they could use BackstopJS to pull out
each component by class
or by ID or whatever
and then have this on this page list out
all the things that have
changed with each component,
with each CSS change you've made.
That's killer use case
number two, theming.
As you're going along and building a site,
it's easy for you to do this.
Of course, it is also easy to automate
the process a little bit more.
We're gonna be looking
at continuous integration
for the next piece.
We're not going to have time
to show any of these pieces,
but just conceptually,
let's think a little bit about how we can
let robots do the parts
of our job that are
a little bit tedious,
that we're not super great at as humans.
One great example is on Cron.
Maybe we want to automatically
run updates on our site.
We don't want to have to think
about Drupal updates anymore,
we just want them to be automated.
If everything works how we expect it to,
just push those up to the live site.
We don't want human staff
to be in the middle,
between the next Drupal gen and our site.
We can do that.
We have a script that we wrote on Pantheon
that goes through this
continuous integration process.
It's a demo script so take
it with a grain of salt,
but you can modify it, you can
do whatever you want with it.
Basically on Cron, it goes out,
checks for updates on your site.
If it detects that there are updates,
it will go back to Pantheon,
create a branch or a
multi dev environment,
a dedicated development
environment for this update,
it will apply the updates
from the safe space,
it will then do visual regression testing
and behavior testing and a
few other types of testing
to make sure everything is working
after the updates have been applied.
If everything passes without any issues,
then it will create backups on all of
your other environments,
your dev, test, and live environments,
and then deploy it up to the live site
without me having to actually
intervene in any way.
If you would prefer to have some say
in when these updates get applied or
to have a human come in and make sure
that it's going to work,
you could change this to just say
notify me on Slack when
updates are available,
show me links to the test results,
let me look over it to make
sure everything looks right,
and then you could go in and actually
click the button to deploy
it up to your live site.
This is one example of
kind of integrating,
automating the process of
using visual regression testing
so that you're more
confident in your updates
before clicking the button to apply them.
Another great example of automating
visual regression testing is on a commit.
We've so far talked and the demo I showed
was I'm running BackstopJS on my computer
and as I'm developing a site
or theming it or whatever,
I'm testing for myself to make sure that
I'm not introducing any problems.
We could also introduce
a visual regression test
on a commit or as part of a poll request
that happens automatically on GitHub.
At DrupalCon, we're gonna
be doing a training called
Automated Workflows in Drupal Eight with
GitHub, Composer and CircleCI.
The whole point of this is you
should be able to as a team,
working with a Drupal Eight
site built with Composer,
be able to collaborate with other
developers in the team through just
creating commits and doing pull requests.
If you are working on some
new piece of functionality,
you've been committing,
you're like this feature is completed
and I'm ready to merge
it into the main project,
you would create a pull request,
that would pull all the commits together,
present it in front of your lead developer
or whomever is doing
the review so they can
look at all of the commits,
but if you have automated testing involved
in this point as well,
then it would actually go
in and show you the reports.
In this case, we're building
the site with Composer
and continuous integration,
spitting that over to Pantheon as
this is the completed
built version of the site,
and then hitting it with
all the automated tests,
and then putting the
results of those tests
in the pull request so the lead developer,
the person reviewing doesn't have to
manually go through line
by line to make sure that
They don't have to go
to the development site
to make sure everything's working.
All they have to do is
review the test results,
which could include
visual regression testing,
but also behavior
testing, clicking through,
and functionality testing,
as well as code review testing.
If you have certain
standards for code review,
with how your code should be on your site,
that's another testing
that you could introduce
and make sure it conforms
to that before the
lead developer has to actually intervene.
Again, another way that you
could automate this process
of using visual regression testing.
My whole takeaway, what
I'm trying to kind of
introduce to you all is that you can
run BackstopJS on your
computer pretty easily,
pretty lightly, pretty quickly too,
just getting it set up and run with it.
Another example is the
standalone Node app.
I've ran this as a class
a few times as I mentioned
someone is unable to
get BackstopJS set up.
Occasionally it's because
of their operating system
or prerequisites or some other reason.
For whatever reason it is,
it doesn't work on their computer.
In that situation though,
the standalone app has
still worked for them.
It has let them kind of
interact with Backstop.
This was just again a simple example demo
that one of my colleagues created
to show that this is possible and it
makes the process of running Backstop
a little bit easier for people that
don't wanna have to mainly
mess with all the bits.
We have a example repo.
You can go out and grab it.
If we were in a class, I would
invite you all to do this,
but you can definitely do this
after the session today if you'd like to.
I've already got this set up.
The script that you saw
that was loading earlier
hadn't quite completed
so this may not work,
but we'll give it a try anyway.
We have this visual
regression Backstop demo.
You clone it locally, you get into it,
you create a branch so you can
make changes without
affecting anything else,
and then we do an NPM install.
Let's actually go back and try that again.
This is the part that I was trying
to do before the session started
so we wouldn't have to
wait for the NPM install,
but it had timed out which is a
little bit concerning for me.
Let me go ahead and describe what this is
supposed to be doing.
Once it gets installed,
it's going to give us an interface
for just running our site.
As a package application,
it's going to run BackstopJS on our site
and get it all up and generating a report.
It already has that configuration
All the bits are taken care of.
You just run it and it
generates the report.
Let me go ahead and skip since that's
still stuck on the one
it was stuck on before.
That's one example.
You just run it and it generates
the report and it's done.
You don't have to mess with
any configuration at all.
The next part of that is
the multi site example.
If you have multiple sites that you need
to test on a regular basis,
maybe your support team is responsible for
you could write an NPM app or a Node app
that has all of those sites preloaded
into this standalone NPM app,
and then when you run the command,
it asks you which site you'd
like to run the report on.
You go down the list, you select it,
and then it actually opens
it up and runs the report.
You're able to preload all these
click the button, and have the report run.
It's intended for people that wouldn't
want to have to set this
up manually themselves.
It just would be able to run standalone.
You still do have the options though
of modifying the configuration file.
If you wanted to download
this as a starting point,
make changes to it, and still
have the standalone app,
you could do that as well.
It's still not working.
Let's keep on going.
This is running the
multiple site example one.
Hopefully it will work better for you
when you try it later.
I'll take a look after the class and
try to see what the problem might be.
It's a little bit short
'cause I'm trying to
work on the timing coming down from a
half-day class to a session.
- [Woman] There was an
area where you're doing
Are you running Backstop
inside the container?
- [David] In those scenarios,
you're running Backstop as
part of continuous integration.
We are running BackstopJS in CircleCI,
in the continuous integration process,
against in this case our Pantheon
site, our reference site.
We're creating the site
in continuous integration
and looking out at the
live site, wherever it is.
- [Woman] But is Backstop itself installed
inside the container in that case?
- [David] It's installed in the container
on continuous integration,
but it's not installed
over on the Pantheon site.
- [Woman] 'Cause where I'm going with this
is I've played with this
and I got it to work,
but when I try to get it to work
within a Doxel container,
I had this problem where I
could point it to google.com
and it would see Google and run.
Obviously I don't wanna test Google,
but I couldn't see my own Doxel site
within the same container.
It just kept saying no site.
I got really stuck.
- [David] I'm sorry.
I don't know much about that.
Sorry, I don't know.
- [Woman] No problem.
- [David] Yes sir.
- [Man] What about sites that use
heavy unparalax or heavy on the ajax
or the things like that,
how do you know how long to wait
or are there effects that
you can get it to say
ajax is down or resource crawl so far?
- [David] Those are the sorts of things
that you can into with
At the beginning, there's all
this on before and on ready
There are things you can do in there
that I haven't done anything with before.
That's where you'd go to accommodate
some of those situations.
- [Man] Is the idea then if you change
the color of the button for example.
You know you're introducing
something it's gonna flag.
Does that mean that you introduced
a new reference in your commits if
your CI is gonna be
checking it every time?
Is that how you deal with it?
- [David] That's a good question.
I've spoken to different
people of different opinions.
Some of them don't actually commit
any of the Backstop stuff.
They don't commit the images.
In general, putting images in
a commit isn't a great idea.
This Backstop stuff, they just run locally
on their computer and that's it.
If you wanted to, package it up,
save the configuration file and
give that to other people,
that probably makes sense.
I don't know if it makes sense to
send the reference images.
I haven't seen that in my scenario.
- [Man] I guess I'm just
trying to get a picture
of if you're wiring it into CircleCI.
I got majority of the tests going
and I've got the NPM test going and
maybe I'm even doing some shallow
rendering in the NPM test.
How do I have it tell me by the way,
you introduced a visual
regression and you didn't mean to?
- [David] All the demo
scripts that we've set up
with the continuous integration process,
because the environment's
being destroyed and rebuilt
every time the continuous
integration process fires,
it regenerates the reference images
and it starts the test
from the very beginning.
In those cases, you know there's a change,
you know to expect that there's
gonna be an error or a problem.
Then you just have to review it.
If you're asking, is there
a way to flag something
as being this is expected,
don't give me a red flag for this,
I'm not aware of a way to do that
unless you introduce the reference images.
That could work.
I just haven't done it before.
- [Man] You change something in your code,
it fails the test, and you
have to update the test
if you're gonna use it at that level to
have it be an all type stopper CI,
or you could generate the reference
to look at somewhere between the CI
and just know that what's
in there is expected
before you test.
- [David] Yes sir, a question.
- [Man] When you have a component with
dynamic content like a
newsfeed or an event block
or something that you want to
visually regress test for regressions,
what would be a good
approach for handling that?
- [David] That's really tough.
That's one of those areas with the
visual regression limitations.
It doesn't deal with dynamic
content very well because
if you have a live site that is
constantly getting new content,
you have your local version that you're
not able to pull the database
from every single time,
you're gonna fail because
the content's different
in this dynamic block section.
In those situations, you
can ignore that section.
You're not able to accommodate
all the different things that
could happen to that section.
That's what I've done in the past is
ignore this dynamic block because
it's always gonna be failing.
- [Man] Unless you're trying to test it.
- [David] If you're trying to test it,
that's a little tricky.
- [Man] It almost seems
like you would need
to have a fixture database that you're
retrieving as your
production in that instance
to compare at the end.
It sounds like a lot of work.
- [David] With the
components testing method.
If you have all these components
or template-type pages
available on your live website that are
really just used as a style guide,
then you could certainly test against that
on your local versus the live style guide
and with some assurance that it's
gonna work the way you expect.
- [Man] Since this has
a fairly civic scope,
what are the test tools
to use to compliment this?
What do you think are
some complimentary tests?
- [David] What are some
It depends on the scenario because
with the automated section,
continuous integration section,
there's behavior testing where people
will log into the website
and make sure that
you're able to create content and
you're able to do the various
pieces of functionality,
things like that.
Is that what you're talking about?
- [Man] What tools would
have minimum overlap of this
in terms of testing?
I'm looking at the general
topography of your testing.
- [David] That's a good question.
I'm not sure actually.
With my own development process,
it's usually pretty light.
This is an improvement
over what I did before,
which was just visually look at it.
I don't have a very complicated process.
Real quick, does anyone
else have any suggestions
or ideas about where this could fit into
an overall larger testing process?
- [Man] My system's on
HP and test on code.
- [Man] I don't get the
visual, but I get the markup.
I'll often lock out here's the json
response to the paper we built around.
It pulls out a response from an actual
database to mock this so I take my mocks.
- [David] I'll be available.
I'll be at Pantheon for
the rest of the day.
Please do leave feedback, it's important,
and especially because like I said,
this is newly condensed
into session format.
There are obviously things
that I can fix about it.
Please let me know.
What are some things I could add or remove
or modify to be even better for this?
Thank you all very much.
Captions made possible by ClarityPartners.com
Chicago area Drupal consultants
Captions made possible by ClarityPartners.com
Chicago area Drupal consultants
The question was, will this
be on the Pantheon site?
I don't think it'll be on Pantheon,
but it'll be on MidCamp.
- [Man] No I mean the one
that you're doing in Seattle.
- [David] The one I'm doing in Seattle
is a paid training through DrupalCon
so we're not gonna have a
recording of that one available.
- [Man] Maybe after?
- [David] I'm not sure.
I don't know.
- [Man] I don't think so.
'Cause that's exactly what I wanna do.
Captions made possible by ClarityPartners.com. Chicago area Drupal consultants