Testing API

The Testing API allows Visual Studio Code extensions to discover tests in the workspace and publish results. Users can execute tests in the Test Explorer view, from decorations, and inside commands. With these new APIs, Visual Studio Code supports richer displays of outputs and diffs than was previously possible.

Note:The Testing API is available in VS Code version 1.59 and higher.

Examples

There are two test providers maintained by the VS Code team:

• Thesample test extension,which provides tests in Markdown files.
• Theselfhost test extension,that we use for running tests in VS Code itself.

Discovering tests

Tests are provided by theTestController,which requires a globally unique ID and human-readable label to create:

constcontroller=vscode.tests.createTestController(
'helloWorldTests',
'Hello World Tests'
);

To publish tests, you addTestItems as children to the controller'sitemscollection.TestItems are the foundation of the test API in theTestIteminterface, and are a generic type that can describe a test case, suite, or tree item as it exists in code. They can, in turn, havechildrenthemselves, forming a hierarchy. For example, here's a simplified version of how the sample test extension creates tests:

parseMarkdown(content,{
onTest:(range,numberA,mathOperator,numberB,expectedValue)=>{
// If this is a top-level test, add it to its parent's children. If not,
// add it to the controller's top level items.
constcollection=parent?parent.children:controller.items;
// Create a new ID that's unique among the parent's children:
constid=[numberA,mathOperator,numberB,expectedValue].join(' ');

// Finally, create the test item:
consttest=controller.createTestItem(id,data.getLabel(),item.uri);
test.range=range;
collection.add(test);
}
//...
});

Similar to Diagnostics, it's mostly up to the extension to control when tests are discovered. A simple extension might watch the entire workspace and parse all tests in all files at activation. However, parsing everything immediately may be slow for large workspaces. Instead, you can do two things:

1. Actively discover tests for a file when it's opened in the editor, by watchingvscode.workspace.onDidOpenTextDocument.
2. Settingitem.canResolveChildren = trueand setting thecontroller.resolveHandler.TheresolveHandleris called if the user takes an action to demand tests be discovered, such as by expanding an item in the Test Explorer.

Here's how this strategy might look in an extension that parses files lazily:

// First, create the `resolveHandler`. This may initially be called with
// "undefined" to ask for all tests in the workspace to be discovered, usually
// when the user opens the Test Explorer for the first time.
controller.resolveHandler=asynctest=>{
if(!test) {
awaitdiscoverAllFilesInWorkspace();
}else{
awaitparseTestsInFileContents(test);
}
};

// When text documents are open, parse tests in them.
vscode.workspace.onDidOpenTextDocument(parseTestsInDocument);
// We could also listen to document changes to re-parse unsaved changes:
vscode.workspace.onDidChangeTextDocument(e=>parseTestsInDocument(e.document));

// In this function, we'll get the file TestItem if we've already found it,
// otherwise we'll create it with `canResolveChildren = true` to indicate it
// can be passed to the `controller.resolveHandler` to gets its children.
functiongetOrCreateFile(uri:vscode.Uri) {
constexisting=controller.items.get(uri.toString());
if(existing) {
returnexisting;
}

constfile=controller.createTestItem(uri.toString(),uri.path.split('/').pop()!,uri);
file.canResolveChildren=true;
returnfile;
}

functionparseTestsInDocument(e:vscode.TextDocument) {
if(e.uri.scheme==='file'&&e.uri.path.endsWith('.md')) {
parseTestsInFileContents(getOrCreateFile(e.uri),e.getText());
}
}

asyncfunctionparseTestsInFileContents(file:vscode.TestItem,contents?:string) {
// If a document is open, VS Code already knows its contents. If this is being
// called from the resolveHandler when a document isn't open, we'll need to
// read them from disk ourselves.
if(contents===undefined) {
constrawContent=awaitvscode.workspace.fs.readFile(file.uri);
contents=newTextDecoder().decode(rawContent);
}

// some custom logic to fill in test.children from the contents...
}

The implementation ofdiscoverAllFilesInWorkspacecan be built using VS Code' existing file watching functionality. When theresolveHandleris called, you should continue watching for changes so that the data in the Test Explorer stays up to date.

asyncfunctiondiscoverAllFilesInWorkspace() {
if(!vscode.workspace.workspaceFolders) {
return[];// handle the case of no open folders
}

returnPromise.all(
vscode.workspace.workspaceFolders.map(asyncworkspaceFolder=>{
constpattern=newvscode.RelativePattern(workspaceFolder,'**/*.md');
constwatcher=vscode.workspace.createFileSystemWatcher(pattern);

// When files are created, make sure there's a corresponding "file" node in the tree
watcher.onDidCreate(uri=>getOrCreateFile(uri));
// When files change, re-parse them. Note that you could optimize this so
// that you only re-parse children that have been resolved in the past.
watcher.onDidChange(uri=>parseTestsInFileContents(getOrCreateFile(uri)));
// And, finally, delete TestItems for removed files. This is simple, since
// we use the URI as the TestItem's ID.
watcher.onDidDelete(uri=>controller.items.delete(uri.toString()));

for(constfileofawaitvscode.workspace.findFiles(pattern)) {
getOrCreateFile(file);
}

returnwatcher;
})
);
}

TheTestIteminterface is simple and doesn't have room for custom data. If you need to associate extra information with aTestItem,you can use aWeakMap:

consttestData=newWeakMap<vscode.TestItem,MyCustomData>();

// to associate data:
constitem=controller.createTestItem(id,label);
testData.set(item,newMyCustomData());

// to get it back later:
constmyData=testData.get(item);

It's guaranteed that theTestIteminstances passed to allTestController-related methods will be the same as the ones originally created fromcreateTestItem,so you can be sure that getting the item from thetestDatamap will work.

For this example, let's just store the type of each item:

enumItemType{
File,
TestCase
}

consttestData=newWeakMap<vscode.TestItem,ItemType>();

constgetType=(testItem:vscode.TestItem)=>testData.get(testItem)!;

Running tests

Tests are executed throughTestRunProfiles. Each profile belongs to a specific executionkind:run, debug, or coverage. Most test extensions will have at most one profile in each of these groups, but more are allowed. For example, if your extension runs tests on multiple platforms, you could have one profile for each combination of platform andkind.Each profile has arunHandler,which is invoked when a run of that type is requested.

functionrunHandler(
shouldDebug:boolean,
request:vscode.TestRunRequest,
token:vscode.CancellationToken
) {
// todo
}

construnProfile=controller.createRunProfile(
'Run',
vscode.TestRunProfileKind.Run,
(request,token)=>{
runHandler(false,request,token);
}
);

constdebugProfile=controller.createRunProfile(
'Debug',
vscode.TestRunProfileKind.Debug,
(request,token)=>{
runHandler(true,request,token);
}
);

TherunHandlershould callcontroller.createTestRunat least once, passing through the original request. The request contains the tests toincludein the test run (which is omitted if the user asked to run all tests) and possibly tests toexcludefrom the run. The extension should use the resultingTestRunobject to update the state of tests involved in the run. For example:

asyncfunctionrunHandler(
shouldDebug:boolean,
request:vscode.TestRunRequest,
token:vscode.CancellationToken
) {
construn=controller.createTestRun(request);
constqueue:vscode.TestItem[]=[];

// Loop through all included tests, or all known tests, and add them to our queue
if(request.include) {
request.include.forEach(test=>queue.push(test));
}else{
controller.items.forEach(test=>queue.push(test));
}

// For every test that was queued, try to run it. Call run.passed() or run.failed().
// The `TestMessage` can contain extra information, like a failing location or
// a diff output. But here we'll just give it a textual message.
while(queue.length>0&&!token.isCancellationRequested) {
consttest=queue.pop()!;

// Skip tests the user asked to exclude
if(request.exclude?.includes(test)) {
continue;
}

switch(getType(test)) {
caseItemType.File:
// If we're running a file and don't know what it contains yet, parse it now
if(test.children.size===0) {
awaitparseTestsInFileContents(test);
}
break;
caseItemType.TestCase:
// Otherwise, just run the test case. Note that we don't need to manually
// set the state of parent tests; they'll be set automatically.
conststart=Date.now();
try{
awaitassertTestPasses(test);
run.passed(test,Date.now()-start);
}catch(e) {
run.failed(test,newvscode.TestMessage(e.message),Date.now()-start);
}
break;
}

test.children.forEach(test=>queue.push(test));
}

// Make sure to end the run after all tests have been executed:
run.end();
}

In addition to therunHandler,you can set aconfigureHandleron theTestRunProfile.If present, VS Code will have UI to allow the user to configure the test run, and call the handler when they do so. From here, you can open files, show a Quick Pick, or do whatever is appropriate for your test framework.

VS Code intentionally handles test configuration differently than debug or task configuration. These are traditionally editor or IDE-centric features, and are configured in special files in the.vscodefolder. However, tests have traditionally been executed from the command line, and most test frameworks have existing configuration strategies. Therefore, in VS Code, we avoid duplication of configuration and instead leave it up to extensions to handle.

Test Output

In addition to the messages passed toTestRun.failedorTestRun.errored,you can append generic output usingrun.appendOutput(str).This output can be displayed in a terminal using theTest: Show Outputand through various buttons in the UI, such as the terminal icon in the Test Explorer view.

Because the string is rendered in a terminal, you can use the full set ofANSI codes,including the styles available in theansi-stylesnpm package. Bear in mind that, because it is in a terminal, lines must be wrapped using CRLF (\r\n), not just LF (\n), which may be the default output from some tools.

Test Coverage

Test coverage is associated with aTestRunvia therun.addCoverage()method. Canonically this should be done byrunHandlers of profiles of theTestRunProfileKind.Coverage,but it is possible to call it during any test run. TheaddCoveragemethod takes aFileCoverageobject, which is a summary of the coverage data in that file:

asyncfunctionrunHandler(
shouldDebug:boolean,
request:vscode.TestRunRequest,
token:vscode.CancellationToken
) {
//...

forawait(constfileofreadCoverageOutput()) {
run.addCoverage(newvscode.FileCoverage(file.uri,file.statementCoverage));
}
}

TheFileCoveragecontains the overall covered and uncovered count of statements, branches, and declarations in each file. Depending on your runtime and coverage format, you might see statement coverage referred to as line coverage, or declaration coverage referred to as function or method coverage. You can add file coverage for a single URI multiple times, in which case the new information will replace the old.

Once a user opens a file with coverage or expands a file in theTest Coverageview, VS Code requests more information for that file. It does so by calling an extension-definedloadDetailedCoveragemethod on theTestRunProfilewith theTestRun,FileCoverage,and aCancellationToken.Note that the test run and file coverage instances are the same as the ones used inrun.addCoverage,which is useful for assocating data. For example, you can create a map ofFileCoverageobjects to your own data:

constcoverageData=newWeakMap<vscode.FileCoverage,MyCoverageDetails>();

profile.loadDetailedCoverage=(testRun,fileCoverage,token)=>{
returncoverageData.get(fileCoverage).load(token);
};

asyncfunctionrunHandler(
shouldDebug:boolean,
request:vscode.TestRunRequest,
token:vscode.CancellationToken
) {
//...

forawait(constfileofreadCoverageOutput()) {
constcoverage=newvscode.FileCoverage(file.uri,file.statementCoverage);
coverageData.set(coverage,file);
run.addCoverage(coverage);
}
}

Alternatively you might subclassFileCoveragewith an implementation that includes that data:

classMyFileCoverageextendsvscode.FileCoverage{
//...
}

profile.loadDetailedCoverage=async(testRun,fileCoverage,token)=>{
returnfileCoverageinstanceofMyFileCoverage?awaitfileCoverage.load():[];
};

asyncfunctionrunHandler(
shouldDebug:boolean,
request:vscode.TestRunRequest,
token:vscode.CancellationToken
) {
//...

forawait(constfileofreadCoverageOutput()) {
// 'file' is MyFileCoverage:
run.addCoverage(file);
}
}

loadDetailedCoverageis expected to return a promise to an array ofDeclarationCoverageand/orStatementCoverageobjects. Both objects include aPositionorRangeat which they can be found in the source file.DeclarationCoverageobjects contain a name of the thing being declared (such as a function or method name) and the number of times that declaration was entered or invoked. Statements include the number of times they were executed, as well as zero or more associated branches. Refer to the type definitions invscode.d.tsfor more information.

In many cases you might have persistent files lying around from your test run. It's best practice to put such coverage output in the system's temporary directory (which you can retrieve viarequire('os').tmpdir()), but you can also clean them up eagerly by listening to VS Code's cue that it no longer needs to retain the test run:

import{promisesasfs}from'fs';

asyncfunctionrunHandler(
shouldDebug:boolean,
request:vscode.TestRunRequest,
token:vscode.CancellationToken
) {
//...

run.onDidDispose(async()=>{
awaitfs.rm(coverageOutputDirectory,{recursive:true,force:true});
});
}

Test Tags

Sometime tests can only be run under certain configurations, or not at all. For these use cases, you can use Test Tags.TestRunProfiles can optionally have a tag associated with them and, if they do, only tests that have that tag can be run under the profile. Once again, if there is no eligible profile to run, debug, or gather coverage from a specific test, those options will not be shown in the UI.

// Create a new tag with an ID of "runnable"
construnnableTag=newTestTag('runnable');

// Assign it to a profile. Now this profile can only execute tests with that tag.
runProfile.tag=runnableTag;

// Add the "runnable" tag to all applicable tests.
for(consttestofgetAllRunnableTests()) {
test.tags=[...test.tags,runnableTag];
}

Users can also filter by tags in the Test Explorer UI.

Publish-only controllers

The presence of run profiles is optional. A controller is allowed to create tests, callcreateTestRunoutside of therunHandler,and update tests' states in the run without having a profile. The common use case for this are controllers who load their results from an external source, like CI or summary files.

In this case, these controllers should usually pass the optionalnameargument tocreateTestRun,andfalsefor thepersistargument. Passingfalsehere instructs VS Code not to retain the test result, like it would for runs in the editor, since these results can be reloaded from an external source externally.

constcontroller=vscode.tests.createTestController(
'myCoverageFileTests',
'Coverage File Tests'
);

vscode.commands.registerCommand('myExtension.loadTestResultFile',asyncfile=>{
constinfo=awaitreadFile(file);

// set the controller items to those read from the file:
controller.items.replace(readTestsFromInfo(info));

// create your own custom test run, then you can immediately set the state of
// items in the run and end it to publish results:
construn=controller.createTestRun(
newvscode.TestRunRequest(),
path.basename(file),
false
);
for(constresultofinfo) {
if(result.passed) {
run.passed(result.item);
}else{
run.failed(result.item,newvscode.TestMessage(result.message));
}
}
run.end();
});

Migrating from the Test Explorer UI

If you have an existing extension using the Test Explorer UI, we suggest you migrate to the native experience for additional features and efficiency. We've put together a repo with an example migration of the Test Adapter sample in itsGit history.You can view each step by selecting the commit name, starting from[1] Create a native TestController.

In summary, the general steps are:

1. Instead of retrieving and registering aTestAdapterwith the Test Explorer UI'sTestHub,callconst controller = vscode.tests.createTestController(...).

2. Rather than firingtestAdapter.testswhen you discover or rediscover tests, instead create and push tests intocontroller.items,for example by callingcontroller.items.replacewith an array of discovered tests that are created by callingvscode.test.createTestItem.Note that, as tests change, you can mutate properties on the test item and update their children, and changes will be reflected automatically in VS Code's UI.

3. To load tests initially, instead of waiting for atestAdapter.load()method call, setcontroller.resolveHandler = () => { /* discover tests */ }.See more information around how test discovery works inDiscovering Tests.

4. To run tests, you should create aRun Profilewith a handler function that callsconst run = controller.createTestRun(request).Instead of firing atestStatesevent, passTestItems to methods on therunto update their state.

Additional contribution points

Thetesting/item/contextmenu contribution pointmay be used to add menu items to Tests in the Test Explorer view. Place menu items in theinlinegroup to have them inline. All other menu item groups will be displayed in a context menu accessible using the mouse right-click.

Additionalcontext keysare available in thewhenclauses of your menu items:testId,controllerId,andtestItemHasUri.For more complexwhenscenarios, where you want actions to be optionally available for different Test Items, consider using theinconditional operator.

If you want to reveal a test in the Explorer, you can pass the test to the commandvscode mands.executeCommand('vscode.revealTestInExplorer', testItem).