Cypress screenshots are missing in Azure Pipelines

My new favorite framework for doing E2E tests in web solutions are Cypress. I usually hook up a pipeline in Azure DevOps that runs the Cypress tests after each deployment.

Its fairly easy to setup a pipeline that runs the test and publishes them to Azure DevOps. Cypress has the Mocha framework integrated as well as a JUnit reporter, which will output a JUnit test report which the Azure Pipeline Publish Test Results task can pickup.

All you have to do is to configure the reporter like this in your cypress.config.

"reporter": "junit",
"reporterOptions": {
	"mochaFile": "results/test-results-[hash].xml",
	"attachments": true
},

Unfortunately the Publish Test Results task only supports attachments for the TRX format and the NUnit3 format – which means the build in reporter for JUnit will not work.

First Attempt on a solution

The first idea was to find a Mocha reporter that supported TRX. In the end this approach did not fully work. If you are interested in how it was done you can read on. Otherwise skip to the next section to see an alternative approach.

So I found this mocha trx reporter. But looking though the code it turned out there was no support for attachments. Luckily the reporter was based on the node-trx module which happens to have support for attachments in trx.

The source for the customer reporter can be found here.

To use it simply put it in your source control and refer it in the cypress.config like this.

"reporter": "reporter/dgs-trx-reporter.js",
"reporterOptions": {
	"inputscreenshotpath": "Cypress\projects\Portal\cypress\screenshots",
	"outputscreenshotfolder": "screenshots",
        "outputpath": "results"
},

It turns out that the trx format has a semi hardcoded location for the screenshots. They need to be placed in a subfolder next to the trx file in a subfolder called outputscreenshotfolder/In/executionId.

The above parameters needs to be set as following:

inputscreenshotpath: Relative path to current working directory where Cypress stores the screenshots.
outputscreenshotfolder: Name of the folder next to the trx file where the screenshots are placed.
outputpath: The relative path where the trx file and the screenshots are saves / copied to.

I had to do a number of workarounds to get the reporter to work. It turned out that Mocha reporters does not expose the name of the screenshots. So instead I had to implement logic that matched the name of the test with the filename of the screenshots. This worked fairly well.

Finally we have to add tasks to the pipeline. I am using the classic pipelines, but you can use YAML based pipelines too. Just make sure to select VSTest as the Test result format.

So, is that it? NO – unfortunately even though the report was correctly and the attachments where found according to the logs – the screenshots where not attached. It turns out other people has reported this in general on the TRX format.

You can find the source for the customer reporter here in github.

Second and final attempt on a solution

After some consideration I come up with another option. I would keep the JUnit report and then do a post processing task which would simply loop through all failing tests and then call the Azure DevOps api to attach the screenshots.

This turns out to be a fairly simple approach. You can find the sourcecode in Github here.

All you have to do is to add your normal task to run your Cypress Test (using the JUnit reporter), Publish Test Results and then call the PowerShell script afterwards.

The PowerShell script has only a single parameter to configure in the top. You could of cause choose to pass it in instead.

# Configuration
$global:cwd = Get-Location
$global:screenshotPath = Join-Path -Path $cwd -ChildPath "Cypress\projects\Portal\cypress\screenshots"

Below the configuration is two blocks of setup of authentication. The first one which is commented in is the one that gets the access token, url etc. from Azure DevOps. In order for this to work you have to enable access to the token by checking the option Allow scripts to access the OAuth token on the job (phase).

Remember to set Continue on error in the control options on both the task for running the Cypress Tests and on the Publish Test Results task. Otherwise the PowerShell script will never be executed.

If you run into any issues with the PowerShell scripts feel free to get in touch and I will try to fix it. It seems to work well in my setup.