In this page

What is Xray?

(supported since Better Excel Exporter 5.1.0)

Xray is a complete, end-to-end test management app for Jira. It supports the entire testing and quality assurance lifecycle: test planning, test design, test execution and test reporting.

Xray integration features

  • You can export the Xray specific custom field types like Automated Test Type Select List (single choice), Cucumber Test Steps Editor, Generic Test Editor, Issue Link Count, Manual Test Steps, Manual Test Steps (Export View), Manual Test Steps Count, Pre-Condition Association with a Test, Pre-Condition Type Select List (single choice), Pre-Conditions Editor, Requirement Status, Requirement Status Custom Field, Test Association with a Pre-Condition, Test Association with a Test Plan, Test Environments Custom Field, Test Execution Association with Tests, Test Execution Defects, Test Execution Status Custom Field, Test Plan, Test Plan Association with a Test, Test Plan Root Folders Custom Field, Test Plan Status Custom Field, Test Repository Path, Test Run Status Custom Field, Test Set Status Custom Field, Test Sets Picker, Test Type Select List (single choice) and Tests to Excel.
  • You can export the test runs for detailed test reports. The following details are exported for each run: Test Execution Key, Test Run Fix Versions, Test Run Test Environments, Test Run Assignee, Test Run Executed By, Test Run Started On, Test Run Finished On, Test Run Duration, Test Run Defects, Test Run Comment, Test Run Evidences, Test Run Status, Test Step Index, Test Step, Test Step Data, Test Step Expected Result, Test Step Attachments, Test Step Defects, Test Step Comment, Test Step Evidences and Test Step Status. (supported since Xray 2.0.0)
  • Better Excel Exporter also offers two reporting templates: one focusing on tests and their relations (with test sets, test plans, test executions), while the other focusing on test runs and test results. Both include several pivot tables and pivot charts, which are super-useful as they are, but can also be used as a starting point to create custom Excel reports from Xray data.

Tutorial video

Better Excel Exporter exports all test-related information, including all Xray-specific entities and all Xray-specific custom fields, to Excel:

Export samples

Xray tests

Xray stores most of the information in its specific custom fields, all supported by Better Excel Exporter. This example is an export of Xray tests, generated from the columns selected in Issue Navigator, including the test key, summary, test steps, plus the pre-conditions, test sets and test plans associated with the given test.

jira-xray-tests-with-test-steps.xlsx

Xray test sets

Test sets are flat, ordered collections of tests, represented by their own Xray-specific issue type in Jira. This Excel export lists test sets with their contained tests and the status of those. Tip: to analyze the Test Set Status cells, convert those to multiple cells using Excel formulas.

jira-xray-test-sets.xlsx

Xray test plans

In this Excel spreadsheet, you can see the list of the test plans which collect the tests to be validated against a specific product version or within a specific sprint, their schedule, and the respective results. You can easily filter the list to any given period by using the standard Filter feature in Excel on the Begin Date and End Date columns.

jira-xray-test-plans.xlsx

Xray test executions

Test executions are assignable tasks for executing a group of tests against a specific system revision (version) in a specific environment, as a part of the testing plan. This Excel spreadsheet contains all relevant details, selected as Issue Navigator columns, including the test results and the defects (bugs) found.

jira-xray-test-executions.xlsx

Xray test runs

This example, generated from the issue-navigator-with-xray-test-runs.xlsx template, goes into the lowest level details of the individual test runs: version, environment, tester, duration, result, comment, defects, evidences for the whole test and for the individual test steps.

jira-xray-tests-with-test-runs.xlsx

Xray test report

Create reports from Xray tests using the xray-test-report.xlsx template! It calculates the test count and visualizes the test distribution per test repository path, priority, test set, test plan and pre-condition. Or, create your own Xray report by copying and customizing this Excel template.

jira-xray-test-report.xlsx

Xray test run report

The xray-test-run-report.xlsx template gives an easy-to-read overview on test executions and on the individual test runs in those. In its default configuration, it gives pivot tables and charts for test results and defects per test execution, version, environment and the tester person. Like any other template, it can be easily customized to your needs.

jira-xray-test-run-report.xlsx

Configuration

Exporting Xray custom fields

There is nothing to do. Better Excel Exporter will automatically recognize the Xray managed fields and export them accordingly.

Configuring the export format for Xray "associated with" custom fields

Xray is using several custom field types that associate an Xray-managed issue type with another, e.g. associate tests with test sets. In the Xray web interface, these values are displayed in the format "FOO-123 FOO-456" where each item is a link. (For some unknown reason, Xray does not even separate the items with commas.)

When exporting to Excel, Better Excel Exporter resolves the issue keys to summaries, and expands the values to the longer, but externally usable format "[FOO-123] My first test, [FOO-456] My second test". If you want to turn this off and return to "FOO-123, FOO-456" (with commas!), set this configuration variable to false in the top of the field-helper-tool.groovy script:

def xrayIssuesWithSummaries = true

These values are comma-separated, by default. If you want to break them to new lines within the cell, to make it easier to read at the cost of the increased cell height, it is simple. Just change the join() method's argument from ", " to "\n" in the field-helper-tool.groovy script in the following way:

case "com.xpandit.plugins.xray:precondition-test-custom-field":
case "com.xpandit.plugins.xray:test-plan-custom-field":
case "com.xpandit.plugins.xray:test-plans-associated-with-test-custom-field":
case "com.xpandit.plugins.xray:test-precondition-custom-field":
case "com.xpandit.plugins.xray:test-sets-custom-field":
case "com.xpandit.plugins.xray:test-sets-tests-custom-field":
case "com.xpandit.plugins.xray:tests-associated-with-test-plan-custom-field":
	value ? value.collect { xrayIssueKeyValueToString(it) }.join("\n") : null
	break

// ...

case "com.xpandit.plugins.xray:testexec-tests-custom-field":
	value ? value.collect { xrayIssueKeyValueToString(!it.metaClass.respondsTo(it, "a") ? it : it.a()) }.join("\n") : null
	break

Configuring the Xray REST API access

As this integration relies on the Xray REST API, you need to configure the login credentials of a valid Jira user account for the REST API calls in the templates:

  1. Go to AdministrationAdd-onsExcel Templates (under Better Excel Exporter).
  2. Open the xray-tool.groovy template for editing, and set the username and password to these configuration variables in the top part (don't remove the quotation marks around the string!):
    /* Xray REST API request parameters. */
    def restUserName = "admin"
    def restPassword = "admin"
    
  3. Save the changes. (Don't worry about storing passwords here: this file is visible only for Jira administrators, who would have super-user permissions any way.)

Notes:

  • As a good practice, this is safer to avoid usernames and passwords that contain non-English characters. Albeit our Xray script correctly encodes international usernames and passwords, their handling also depends on the configuration of the container that hosts your Jira web application (typically Tomcat). If you're having difficulties, just replace those characters in your username or password with English letters or numbers.

Finally, the following section describes a confusing situation when all REST API calls result in "Server returned HTTP response code: 403" or "401" errors. If you are not affected, you can skip this section.

Why does this happen? As its default behavior, Jira will lock your account and present a CAPTCHA on the login form after a few unsuccessful login attempts. Since the Xray integration may send several REST API requests per export, all using the same username and password, the unsuccessful login attempts limit can be reached very quickly if the password is wrong.

How to fix it? You just have to pass the CAPTHCA verification, that's it. First off, be 101% sure that the REST username and password are correct. If those are, but you are still receiving 403's, please open Jira in your browser, logout and login with the same user account which you configured for the REST calls. You will be asked for a CAPTCHA verification, but after answering that and logging in to Jira, the REST calls will also be successfully executed!

Please note that it will never happen again, unless you change the password of the corresponding Jira user account.

If you are an administrator, there is also a quicker solution: go to Administration → User management and clear the failed login attempts counter with one click. Or, you can increase the allowed login attempts or even disable this feature altogether in Jira.

Troubleshooting

I cannot export test runs from pre-2.0.0 Xray versions

Exporting the test runs with the issue-navigator-with-xray-test-runs.xlsx and xray-test-run-report.xlsx templates relies on the /rest/raven/1.0/api/test/{key}/testruns endpoint provided by the Xray REST API. This endpoint is available since Xray 2.0.0. Therefore, Better Excel Exporter exports test runs only with Xray 2.0.0 or newer versions.

Note that other information, stored in Xray-specific custom fields, can be correctly exported even with older Xray versions.

Learn more about Xray