Snapshot comparison

Snapshot comparison allows you to compare screenshots between previous and current versions of your application. It is useful for spotting visual errors that cannot easily be identified through automated tests alone, such as unwanted changes on styling or page element positioning.

Setting a baseline

Let's start with an example. We have an e-commerce application where one day due to a styling issue, the color of the Add to basket button was set to white. This caused the button to effectively disappear, as below:

Our automated tests were still passing, because the buttons were still there, but it was invisible to the eye of our actual users (both the button's background and text were white).

Snapshot testing enables us to set the expected visual state of the pages as the baseline, and every time a new snapshot is captured, we can compare the new states against the baseline.

This helps you to detect and ultimately prevent visual bugs, which may be invisible to your functional tests.

In the scenario above, when you capture a new snapshot of your application, you have a choice:

1. The removal of the buttons was part of your specification - if so, you set the new state as the new baseline;
2. The removal of the buttons was unexpected (regression) - if so, you report and fix the problem and run snapshot comparison again until you see what you expect.

Now that we have a better understanding of the overall flow of snapshot testing and setting baselines, next we look at how you can do this in Virtuoso, and across all your journeys.

Capturing checkpoint snapshots

You can capture the state of your entire application at any point by performing an execution using Capture checkpoint snapshots. Virtuoso will then capture snapshots for each checkpoint encountered while executing your journeys.

To capture and compare a new snapshot, go to the Project dashboard page, click on the context menu icon of one of your goals, choose Capture new snapshot (which is a shortcut for clicking on Execute advanced... and marking the option Capture checkpoint snapshots), and then click on Execute.

Wait for the execution to finish and then go to the Goal view and click on Comparison on the right-side panel:

In the right-side panel, you can see the snapshot comparisons of all the states in your application. Each comparison is assigned one of four types:

• added pages - these pages are new in Virtuoso and do not have a baseline assigned;
• changed pages - these pages have some change when compared to the baseline, this could be visual or programmatic;
• pages without changes - these pages are equivalent to the baseline;
• removed pages - these pages have a baseline assigned but no page was found for the selected snapshot of a goal.

When reviewing all of the comparisons, you can use the filters on the top bar to filter by certain comparison types. To hide/show the comparisons you have already reviewed, you can click the Show reviewed toggle.

After reviewing a page comparison, you can mark the result as expected or as unexpected by clicking the thumbs up or thumbs down icon. Marking a comparison as expected sets the page on the right-hand side as the new baseline for that checkpoint, against which captured snapshots will be compared.

You can select one or more comparisons from this page and mark all of the selected as expected/unexpected at once or you can open the detailed comparison view and mark a comparison individually.

Using "Capture checkpoint snapshots" always

Goals have a setting to define when Virtuoso should capture new checkpoint snapshots. You can change it under the Goal settings page to one of the following values: Always, Auto, or Never. Setting it to Always will make Virtuoso capture checkpoint snapshots whenever the goal is executed.

Detailed comparison view

Although the comparison list shows a preview of the states, the preview is usually not enough to help you make a decision whether the change is expected or not. You can open the detailed comparison view by clicking on any of the comparison rows shown on the page. The result will look as follows:

In this view, we can again press the thumbs up or thumbs down icons to set the right screenshot as baseline, or to reject the right screenshot. We can quickly change between states in this view. On the left side of this modal, you will find thumbnails of all the states for your checkpoints during their most recent captured snapshot. You can switch between the comparisons by clicking on these states.

The detailed comparison view contains many utilities to help identify if the new state of your application is expected. This section will provide an overview of the different tools and utilities available to you.

Side by side

By default, the detailed comparison view will show side-by-side (A|B) mode. Using the side-by-side mode, the left state shows the baseline set by you, and the right shows the current state in the most recent captured snapshot.

You can inspect the elements that changed or all the elements that Virtuoso discovered on the page by switching between All elements and Changed elements. To inspect an element mouse over the rectangle shown around the element on the screenshot.

Overlaying expected and current screenshots

When you want to accurately review the differences between two screenshots, Virtuoso gives you some advanced comparison utilities (swipe, fade, comparison, and highlight). The current screenshot is placed on top of the baseline's screenshot, and using the controls from the bottom bar you can detect fine-grain differences (e.g., font size of a paragraph might be different but the page looks the same apparently).

Swipe

The expected screenshot stops at the slider labeled Swipe on the screen and is displayed on the left side of the slider. The differences between the current screenshot and the expected one can be compared by moving the slider left and right to see which parts of the screenshot have changed.

Fade

The current screenshot is placed on top of the expected screenshot, and by default with 50% opacity. You can use the opacity slider in the bottom menu to blend between the current screenshot and the baseline's screenshot to see what changed (e.g., the color palette is slightly different now).

Comparison

With this option, Virtuoso computes the difference between the baseline and the current screenshot and shows it in the form of an image. You can change the opacity value the same as you do on the Fade tool, using the opacity slider in the bottom menu.

Grey parts of the image are unchanged and non-gray parts are different between the baseline and current screenshot. A full-gray image would mean that there are no changes.

Highlight

This is similar to the comparison mode, but differences will be highlighted in a red to a black color scale, depending on the degree of difference, with red representing the greatest difference. This mode is best for finding minute differences.

In the above example, the buttons Add to bag are highlighted in full red, as they are missing on one of the screenshots but present on the other.

Discussing comparisons collaboratively

You may want to log your findings or start conversations for clarification about certain differences. For this, you can add reviews to screenshots in the detailed comparison view:

Click on the add message icon on the top bar, above the screenshots, to make the review input appear, place it where you want on the right screenshot of the most recently captured state, and compose your review.

Each review consists of multiple messages, to add more messages to a review, click on the review number to expand the review, and send the message the same as you did for sending the first message.

Sometimes, you may want to hide all the reviews to see the screenshots without noise. You can hide all the reviews by clicking on Hide reviews.

Reviews are linked to the current snapshot of your application, and will not show on newly captured snapshots. However, they will remain on the older snapshots of your application should you need to go back to them.