How to Run and Interpret Unit Tests with Visual Studio Test Explorer

Christian Muehle

Read more posts by this author.

When you’re developing in Visual Studio, writing unit tests is key for any successful software project. But writing a unit test is only one piece of the testing workflow. You must then execute and organize the output in a logical manner. This is where Visual Studio Test Explorer comes in handy.

In this demo-driven tutorial, you will learn what Visual Studio Test Explorer is and how to leverage this handy feature to organize all of your test cases.

What is Visual Studio Test Explorer?

When you create a new project in Visual Studio, software developers typically (hopefully) also include unit tests in that project. Unit tests ensure the code runs as expected. But including tests doesn’t do much good if you don’t run and understand the results.

Visual Studio Test Explorer is your interface to unit tests. Test Explorer is a handy feature of Visual Studio that allows you to run unit tests within a project, customize how they’re run, and interpret their output.

Test Explorer allows you to manipulate the order that unit tests are run, create custom playlists to segment out which unit tests run, and run tests in various user-defined groups. Once executed, Test Explorer displays test results, how long each test ran and other messages generated during the test.

In a nutshell, Test Explorer is a feature in Visual Studio that helps you run and interpret unit test results.

Prerequisites

The remainder of this tutorial will be demo-driven. If you’d like to follow along with the demos provided, be sure to have the following:

  • Visual Studio 2019 – This tutorial will use Community edition running on Windows 10 any edition will work.

Test Explorer supports any language that targets .NET, C/C++, Python, JavaScript, and more.

Test Explorer also supports other unit-testing frameworks like xunit and MSTest.

Setting up the Example Project

Rather than ramble on about Test Explorer without any real hands-on work, let’s get your hands dirty and work with a real Visual Studio project. For the rest of this tutorial, you’re going to work with a pre-created .NET 5.0 project called TestExplorerExample.

If you have your own project, Test Explorer will work just as well but the instructions provided in the tutorial will not be exactly the same.

To get started:

1. Clone the TestExplorerExample GitHub repository using Git. This tutorial will assume you’ve cloned the Git repository to ~/TestExplorerExample.

2. Open Visual Studio 2019.

3. In Visual Studio, navigate to the ~/TestExplorerExample folder and open the TestExplorerExample.sln solution file.

The solution contains two projects; the unit tests called ImportantTests and the application called TestExplorerSample as you can see below.

Two example VS projects for the tutorial
Two example VS projects for the tutorial

4. Now, click on the Test menu at the top of the screen and click on Test Explorer.

Invoking Test Explorer in Visual Studio.
Invoking Test Explorer in Visual Studio.

Test Explorer will either dock to one side or open as a separated window.

Visual Studio Test Explorer docked to the right side of the Window
Visual Studio Test Explorer docked to the right side of the Window

Like all Visual Studio windows, you can dock the window to the sides by dragging the window to the desired position.

Understanding the Test Explorer Interface

When you first open Visual Studio Test Explorer, you’ll immediately see three main areas

1. The toolbar with various elements to control tests. You can see beaker as a button and the number 10 beside it. That’s the number of tests defined in the ImportantTests project.

2. The list of available tests defined in groups (more later). In the example below, you can see your project called ImportantTests and has 10 unit tests inside.

3. A summary section that provides an overview of your unit tests and execution status (when the time comes).

Visual Studio Test Explorer with default settings
Visual Studio Test Explorer with default settings

Important: If you don’t see any unit tests show up in Test Explorer, initiate a project rebuild.

The Toolbar

The toolbar is the most common place you’ll find yourself clicking around in. The toolbar is where you will run tests, define test groupings, and more.

You can see a description of each element of the toolbar in the following screenshot. The tutorial project contains 10 tests. You can see the total number of tests to the right of the beaker icon along with the number of passed tests (0), failed tests (0) and those that have yet to run (10).

Once your project grows to dozens, hundreds or even thousands of tests, the Change grouping option will help you focus on specific parts of your project.

Visual Studio Test Explorer toolbar
Visual Studio Test Explorer toolbar

The Tests List

Click on the ImportantTests project in the test list pane and you’ll see below that Test Explorer breaks down the project into various default groups. These groups, by default, are broken down hierarchically from the Namespace —> Class Name —> test methods.

The Visual Studio Test Explorer tests list
The Visual Studio Test Explorer tests list

You’ll also see four default columns in the tests list:

  • Test – The name of the test case.
  • Duration – Once executed, how long the test lasted.
  • Traits – Custom attributes that can be attached to unit tests .
  • Error Message – Once executed and failed, the error that occurred.
Test Explorer with values in the default column configuration
Test Explorer with values in the default column configuration

The Tests Summary

Finally, the test summary section provides information depending on the state of the selected element in the list. Below you can see an example of a failed test run which includes the outcome, total duration, and in the case of a failed test, error text.

The Visual Studio Test Explorer summary window showing a failed test
The Visual Studio Test Explorer summary window showing a failed test

Running Unit Tests

Once you have built a set of unit tests, you must execute them somehow. Visual Studio Test Explorer provides an intuitive interface to run and interpret those results. If you’re following the tutorial and have opened the TestExplorerDemo project, you’ll already have some unit tests ready to run.

Executing All Tests

Assuming you have Visual Studio Test Explorer open in the tutorial’s project, to run all tests, click the left play button on the toolbar’s left (A). Test Explorer will run all tests currently visible.

Executing a Single or Group of Tests

As you learned above, Test Explorer groups tests together by various criteria. If you’d like to run all tests within a group, right-click the group and click Run. To run a single test, perform the same action on a single method test.

Possible options on how to start a test case using the Test Explorer
Possible options on how to start a test case using the Test Explorer

For this tutorial, click on the play button to execute all tests. Once tests have been executed, Test Explorer will show you the results as shown below.

You’ll see that each test will get a new state:

  • Failed – Threw an error or failed a check
  • Passed – Finished without an error
  • Skipped – Not ran, for example because they are tagged as Ignore
  • Not run – Will run but haven’t ran yet
Test cases with different states in the Test Explorer
Test cases with different states in the Test Explorer

Now, click on a test case or group. You’ll see the summary section at the bottom will update displaying test results. The summary will contain the error message (if any) and the duration of the test.

Test summary after a single test case was selected from the test list
Test summary after a single test case was selected from the test list

Getting Custom Audio Test Notifications

You must be running Visual Studio 2019 Update 16.9 Preview 3 on Windows 10 to get custom audio test notifications at the time of this writing.

If you have many tests that take a long time to run, you’re probably not staring at the screen waiting for the results. If so, you can set up custom sounds when tests succeed or fail!

To do so, open the Windows 10 audio dialog and click on the Sounds tab. Next, scroll down the Program Events list until you see Microsoft Visual Studio. Next, click on Test Run Failed or Test Run Succeeded, select the sound from the Sound dropdown box and click OK.

Windows 10 Audio Dialog
Windows 10 Audio Dialog

Customizing Test Explorer Columns

Once you get used to the default Visual Studio Test Explorer interface, you can change up just about anything, if necessary. For example, if you’d rather see different attributes in the tests list, you can customize what’s shown.

To change up the columns in the tests list, click on the Settings gear in Test Explorer, click on Columns and select the column name you’d like to add (or remove).

Customizing the Test Explorer columns
Customizing the Test Explorer columns

Grouping Tests

By default, tests are grouped by Project, Namespace and Class. If this grouping doesn’t fit your needs, you can change the group based on other attributes. To change grouping, click on the icon in the toolbar as shown below. Test Explorer will present you with a list of options to group by.

Each group will change how Test Explorer displays the tests in the tests list.

If you decide to group the tests by Duration, Test Explorer will create a Fast, Medium and Slow group.

Test Explorer Test List
Test Explorer Test List

Finding Tests by Filtering

In this tutorial project’s test suite, you’re only working with 10 tests. In the real-world, you might have hundreds or even thousands. Rather than scrolling through tests and unfolding every group, use the built-in search console.

You can find the search console in the upper right corner of Test Explorer. The search console allows you to construct various queries to search and filter unit tests with a standard query syntax of Search term:"Search value".

For example, to find all failed tests, provide a key value of State with a value of Failed.

Filter to show all failed test cases
Filter to show all failed test cases

To find all tests that contain the string 5 in the message, use Message:"5".

Filter to show all unit tests with the string 5 in the Message.
Filter to show all unit tests with the string 5 in the Message.

And, as the last example, combine conditions together to find all failed tests that failed within a class containing the word String.

Show all failed test cases inside a class with a name containing the word “String”
Show all failed test cases inside a class with a name containing the word “String”

Understanding and Adding Custom Traits

By default, each test method has various attributes associated with it like it’s name, the class it’s in, how long its tests generate (Duration) and more. Using these attributes, you can then find and display the tests in many different ways.

Sometimes though, the default attributes aren’t enough. What if you need to assign a custom category to a test or perhaps a priority? In that case, you need to create a unit test trait.

Traits are key/value pairs or just a simple key that you can assign to a test. Traits allow you to “tag” or assign a new meaning to a particular test. In Test Explorer, once you’ve assigned a trait, you can use them to group and filter tests.

Creating a Single-Value Trait

Let’s first start off by creating a new trait in nUnit. To create a new trait for the tutorial project:

Some unit test frameworks come with pre-built traits that you can extend. NUnit, for example, comes with two classes you can use to add traits to a test method, Category and Property.

1. Open the MathTest.cs file under the ImportantTests project.

2. Copy the code below into the CalculatorTests class. The below code uses the Category attribute to define a new category value called TraitCategory.

[Test]
 [Category("Important")]
 public void TraitCategory(){}
The new test method with a Trait in the MathTest.cs file
The new test method with a Trait in the MathTest.cs file

3. Rebuild the project.

4. Once rebuilt, open the Test Explorer again and you’ll notice a new trait called TraitCategory under the CalculatorTests class.

The new test method with the Trait inside the Test Explorer
The new test method with the Trait inside the Test Explorer

Creating a Single-Value Trait Using a Custom Class

Alternatively, nUnit allows you to create a custom child class derived from the Category attribute used above by creating your own Attribute class. To do so, add the code below into the CalculatorTests class inside the MathTest.cs file and rebuild the project like before.

By default, the class name (except the word Attribute) will be used as the value of the trait.

[AttributeUsage(AttributeTargets.Method, AllowMultiple=false)]
public class ImportantAttribute : CategoryAttribute
{ }
  
[Test]
[Important]
public void TraitCustomAttribute(){}
The new test method with the custom trait class
The new test method with the custom trait class

Creating a Key/Value Trait

If you need to apply more than one value to a trait, NUnit has your back. Let’s now walk through an example of creating a trait using a key/value pair. You will create key/value traits using the NUnit Property class.

Copy the code below into the CalculatorTests class inside the MathTest.cs file, rebuild the project as before and run all tests. You’ll see below that Test Explorer now shows the new TraitKeyValue trait with the Severity [Important] key/value pair.

[Test]
[Property("Severity", "Important")]
public void TraitKeyValue(){}
The updated Test Explorer showing the new added traits
The updated Test Explorer showing the new added traits

Once you’ve created some traits, you can then use the Trait attribute when finding test cases such as Trait:"Important".

Creating a Test Playlist

To get a better overview and focus on specific parts of a project, Visual Studio allows you to create test playlists. A test playlist contains tests even from different classes and help you organize tests even further.

Test playlists are stored in single files that can be shared with anyone using the same Visual Studio project. Common use cases for test playlist include:

  • Defining smaller lists in bigger projects to focus on specific use cases
  • Creating a list specific for a module in your software
  • Creating a list that would otherwise not be possible with the filters offered by Test Explorer

To create a test playlist in this tutorial project:

  1. Right click on the ImportantTests test project and select Add to Playlist —> New Playlist from the context menu.

You can also mark multiple elements by pressing SHIFT. A new Test Explorer window will open showing only the selected test cases.

2. In the Tests: Playlist XX box, click the Save button as shown below and provide a filename to save the test playlist.

Saving the Test Playlist XX box
Saving the Test Playlist XX box

By default, test files are saved with the .playlist extension.

3. Once you’ve saved the playlist, click on the Playlist button in the toolbar and select Open Playlist File….

Open a playlist file from the Test explorer
Open a playlist file from the Test explorer

4. Find the playlist file from the file dialog and open it.

Open a Test Playlist file from your disk
Open a Test Playlist file from your disk

Test Explorer will then show the Tests: XXXX playlist dialog box where you can work with the tests included in that playlist.

The Playlist opened in Visual Studio
The Playlist opened in Visual Studio

Conclusion

The Test Explorer is more than just a simple list of tests available in a Visual Studio project. You can organize and group any size and number of tests. Now that you know how to run unit tests in Visual Studio Test Explorer (and organize them), do you think this will affect how you write unit tests in the future?

Subscribe to Stay in Touch

Never miss out on your favorite ATA posts and our latest announcements!

Looks like you're offline!