Virtuoso Guide

# Frequently Asked Questions (FAQ)

# Table of contents:

# When trying to paste a checkpoint, why is the clipboard pasted into the "Checkpoint title" field when I press CTRL+V?

In order to paste a checkpoint, there needs to be no editable test step open. Having an editable test step will cause the clipboard to paste into the title of a test.

# Can I speed up the execution of my tests?

Tests with too many steps can execute slowly, especially if they have to locate elements on the screen. See how long each step takes when running against your application, and discover which checkpoints/steps take the most time. This information can help you understand how to reduce the duration of the execution.

# Why does it take too long to click an element?

Virtuoso waits for a page to stop loading before proceeding further with a step. Often a click will involve navigating to a different URL, which usually calls your backend. It is likely that your application's backend is taking too long to respond after the click. Note that this delay is something your users will be facing as well. In short, it is not Virtuoso that is taking too long to perform an action, but rather, the target application that is taking too long to respond after the action was taken. If you think you have come across an unusual waiting time that is not related to your application, please contact our support.

# My test step should be pressing keys in some input field/element but is not.

By default, the press command will execute on the whole page, or the last active element. To execute some key in a specific element, use press KEY in "identifier" (e.g., press ENTER in "Search").

# My application requires multiple windows, how do I interact with multiple windows in Virtuoso?

Virtuoso supports multiple frames and tabs using the switch command, and this can simulate using multiple windows. To open a new tab, use the navigate command (e.g., navigate to "http://www.spotqa.com" in new tab). To change to the next tab, the test command switch to next tab will open a new tab. The default tab can be accessed using switch to previous tab or switch to tab "0".

# Can I run Virtuoso tests using custom data from an external location?

There are few ways you can achieve this:

  1. Use an extension to fetch the data from your backend and make it available to your journey.
  2. Make your journey data-driven, and use the Virtuoso API to update the data (e.g., import a new CSV).
  3. Execute your journey with custom initial data using the API.

# Virtuoso is entering text in a text input field in a strange way?

When entering data, Virtuoso types rapidly. Sometimes, JavaScript on your page can change the format of text fields and this can change the order that letters are presented in the text field (e.g., in a credit card field, changing from xxxxxxxxxxxxxxxx to xxxx-xxxx-xxxx-xxxx can modify the order from that which was originally entered).

This is an issue with the JavaScript of the application, but can be worked around by splitting up the test steps using domain knowledge. For example, in the credit card example, it might be worth using 4 write test steps, one for each part, to enter the entire credit card number, ensuring that each test step appends to the input field.

# My test fails because an element was too slow to load, so it was not found by the test step. How can I delay the test step until the element has loaded?

By default, Virtuoso will not wait a long time for elements to appear. This is expected behaviour and will not change. However, using the wait command, it is possible to delay interaction with an element until it has loaded. For example, to wait 15 seconds for "your element", wait 15 for "my element" as a previous test step would ensure that the element is loaded before continuing. This implicitly waits for the element to load, such that if it loads in 2 seconds, Virtuoso will not wait 15 seconds.

# Using natural language I am unable to target the element that I want, how do I use selectors to modify which field is used?

You can add custom selectors with the add selector option when modifying a test step. When selecting an element to interact with, the selectors will be used in order of top-to-bottom. If a selector is not working, it may be worth placing a custom selector above it.

# Are parallel executions supported by Virtuoso?

Yes, Virtuoso supports parallel executions. You can run many Executions for a goal in parallel -- this will depend on the max parallelism settings (see goal environment settings, maximum parallelism of a plan, execution start times, and quotas and thresholds for more information). And you can have multiple goals from the same project executing simultaneously.

# What are differences between the element name and test step selector hint in the test step properties?

When creating a test step, Virtuoso uses the element name introduced and creates a selector Hint to find the element on the page (this selector can be seen by clicking the eye icon and opening the test step properties).

Editing the Hint selector, changes the field/element that Virtuoso Executes the test on, while element name only affects how the test step is represented. Because test steps may not have a Hint selector, Virtuoso allows you to edit the element name to customize the test step making it more readable.

For example, you can write the following test step:

click on "Sign in"

Virtuoso creates a test step with the that text and a hint selector Sign in.

Then, you notice that the button is called Sign In (capitalization change) and update the selector so that Virtuoso can find the element.

If you could not edit the element name, you would have to delete and recreate the test step or the text would be out of sync with the selectors you are using. By editing the test step element name you can now change the text of the test to present: click on "Sign In".

# I added a test step through natural language but it failed or Virtuoso interacted with the wrong field. How can I overcome this problem?

When you add a test step through Natural language, Virtuoso generates a Hint selector based on the text introduced. This text is case-sensitive so you should pay special attention to the case of the letters in the HTML. For example, if an application uses CSS to change the text to small caps variant, the text will appear to be capitalized yet in the HTML it may only have the first letter with caps.

If this still does not solve the problem, you should use another type of selector that you know that produces a unique result for the page you are in. Again, for example, a login form page can have two intractable elements with the text Sign In if you do the following steps once you reach that page:

write "[email protected]" in "Email"
write "dfg123" in "Password"
click on "Sign In"

Virtuoso will click on the first Sign In element, which can be the link at the top navigation bar to access the login form and you end with an empty form.

You can use another selector type or try guiding Virtuoso in finding the correct element by creating a more specific test step:

click on button "Sign In"

# A test step inside a checkpoint failed, why are the remaining checkpoints skipped?

Unlike journeys, the Execution of checkpoints is performed sequentially. Consider the following scenario:

Checkpoint 1: Navigate to www.google.com Latin version
Checkpoint 2: Search for "ipsum lorem"

If the first checkpoint fails because Virtuoso was not able to access the specific version of the website, it does not make sense to attempt to perform the search as the browser can be in a completely different state than what was expected.

If the checkpoints are independent, for example they do not have side effects on the browser state, you can create a new journey with the second checkpoint and have them run independently and in parallel.

# I added an invalid email to a form with a write test step. Was it not supposed to fail?

When executing write test steps, Virtuoso fills the specific form element with the text provided (see examples in Virtuoso tests → Natural language syntax). If it was able to find an element and input the text, the test step will pass independently of any form validations that may happen after the text is written.

Without this behavior, you could not assert the error messages that are expected when introducing invalid data.

# My application uses CAPTCHA or Two-factor Authentication (2FA). How can I run automated tests against my application?

Our general recommendation based on testing best practices is to ask your development team to provide a mechanism to disable verification of the CAPTCHA / 2FA. Here are some suggestions you can pass on to them:

  • Create a specific testing environment in which CAPTCHA/2FA disabled.
  • Create and use specific accounts for which CAPTCHA/2FA is disabled (e.g., disable CAPTCHA/2FA for certain test users, or to validate that the value of the submitted captcha matches a fixed value for these users).
  • Accept a special cookie: disable CAPTCHA or ignore the value of the submitted captcha when a known cookie is present and its value matches a specific value that is known by the backend (you can set cookie values in Virtuoso test steps).
  • Accept a special value: Create a special CAPTCHA or 2FA value which is always accepted. For example, see the I'd like to run automated tests with reCAPTCHA. What should I do? page on Google's suggested approach.
  • Use the user agent or an HTTP header to distinguish Virtuoso traffic to disable CAPTCHA/2FA. See the guidance on how to distinguish Virtuoso traffic on ways that you can detect Virtuoso's browser traffic.
  • Bridge: If you are able to disable CAPTCHA/2FA based on source IP address, you can use the Bridge to have Virtuoso send traffic from a machine that you can add an exemption for.

CAPTCHA on a login page

If you are using a Captcha on your login page, our recommendation is to only require a captcha submission after a number of failed login attempts (as common across the industry, for example, Google, Microsoft, etc.).

This creates a better experience for users, ensures that automated tests using the correct credentials will not face a captcha, and maintains the security safety net to avoid bruteforce attempts.

# How can I validate that my Two-factor-Authentication (2FA) mechanism is working?

If disabling 2FA for testing purposes is not an option, you can attempt to solve them during the journey, based on the type of 2FA method. For example:

  • Email based Token/Link: You can navigate to your email provider, login, and then retrieve the token/link to proceed from. WARNING: DO NOT use this against Gmail, Microsoft, Yahoo, etc. mail providers since they often have bot prevention tooling that will automatically block access to your account, thinking your account may have been compromised.
  • SMS token: You can use the natural language extension (script) feature of Virtuoso to call the API of your sms-service provider (e.g., Twilio), to retrieve the token.
  • Time-based One-time Password (TOTP): You can use the natural language extensions (scripts) feature of Virtuoso to run a Javascript implementation of an OTP generator (e.g., JS-OTP), to generate you the code on the fly. Please contact our support if you need any help setting this up.

# How can I distinguish traffic coming from Virtuoso bots on my application?

By default, all requests made using Virtuoso's default browser contain the following suffix on the user agent:

Virtuoso/If unexpected please contact [email protected]

You can look for Virtuoso in the browser user agent to identify our bots. Alternatively, you can use the custom HTTP headers capability when executing tests using the Virtuoso API.

WARNING

Cross-browser executions will not include this suffix on the user agent. Instead, they use the user agents of their respective browsers or devices.

TIP

If the user agent suffix prevents you from accessing your application, please reach out to your Account Manager for assistance.

# How do I interact with elements inside a Shadow DOM?

Use a JS path selector to target an element inside a shadow DOM. If editing a test step, add a selector of type JS path and paste your JavaScript selector or if typing a new step, just type it as part of your test step and Virtuoso will set the JS path selector for you.

For example:

see "document.querySelector('#shadowdom-open').shadowRoot.querySelector('header')"
click on "document.querySelector('#shadowdom-open').shadowRoot.querySelector('button')"

On a side note, Live Authoring takes care of this for you, for example, when fixing a failed step by just clicking on the element or when using the interaction menu to create a step.

TIP

If you are getting the error Cannot read properties of null (reading 'shadowRoot') in a test step, try adding a wait step for a few seconds before the step that errored.

# How quickly should I expect my journey to start executing?

Since Virtuoso dynamically adapt to handle workloads of any size, it can take 1 to 2 minutes to start executing your journeys (99th percentile), depending on the number of journeys you are executing and the current load on our platform. Our expectation is that for the absolute majority of journeys (95th percentile over a month period), they should typically start executing within 1 minute. As of May 2023, our data shows that for the majority of journeys (median start time over a month period), they start executing within 1 second.

We should however note that there are various other factors that can affect start times that do not correspond to the above:

  • Max parallelism limits set on your jobs: If you have set a max parallelism limit on your execution request, your plan, or your goal, then Virtuoso will ensure that the limit you have set will be respected. For example, if you set a parallelism limit of 1, then Virtuoso will only execute one journey at a time, and will wait for the previous journey to finish before starting the next one.
  • If you are using a bridge to execute your journeys, then by default, we would not execute more than 10 journeys in parallel. This is to ensure that we do not overload your bridge client. To overcome this limit, you can create more bridge clients, and use different bridge clients for different goals.
  • If you are using cross-browser executions without using your own dedicated grid, you will be subject to a fair-usage limit. If you require a higher level of concurrency, consider bringing your own dedicated grid.
  • By default, there is a limit across your organization for the maximum number of concurrent executions that can be running at any given time. This default is in place to help protect your own applications from being overwhelmed by concurrent automated executions. This limit is set to 125 by default (see Quotas and thresholds), but if you are hitting this limit and wish to increase it, contact your account manager / our customer success team. Before raising a request to increase this limit, please make sure that the infrastructure teams responsible for your applications are aware of the increase in traffic that will be generated by Virtuoso.

Note that when a parallelism limit is reached when executing a journey, Virtuoso's priority is first to ensure that the limit is respected, than ensuring that the journey is executed as quickly as possible. This means that it may take longer for your journey to start executing after it has been throttled previously. To avoid this, we recommend you to design your journeys such that they do not require a parallelism limit to be set, or in the case of a bridge, to spread your workload across various bridges.

It is important to highlight that with Live Authoring, your journeys will always start executing immediately, since your live authoring bot is your own. Therefore, the limits and timings described above do not apply to Live Authoring.

# Can I install or test Chrome extensions?

Chrome extensions are not supported or tested, and cannot be installed on Virtuoso chrome browsers, or for cross-browser testing.

# Is it possible to execute tests on hosts using only IPv6 addresses?

Currently, Virtuoso does not support testing against hosts configured exclusively with IPv6 addresses. Virtuoso's infrastructure relies on IPv4 and, as such, hosts must have an IPv4 address configured to work with our service.

# I cannot find an answer to my question, how do I contact support?

If you have any questions, doubts, concerns or suggestions, you can send us a message on Virtuoso's live chat.

Last Updated: 10/7/2024, 10:10:19 AM

← Quotas and thresholds →