In this page

What is Xray?

(supported since Better PDF Exporter 8.1.0)

Xray is a complete, end-to-end test management app for Jira. It supports the entire testing and quality assurance lifecycle: test planning, test design, test execution and test reporting.

Xray integration features

  • You can export the Xray specific custom field types like Automated Test Type Select List (single choice), Cucumber Test Steps Editor, Generic Test Editor, Issue Link Count, Manual Test Steps, Manual Test Steps (Export View), Manual Test Steps Count, Pre-Condition Association with a Test, Pre-Condition Type Select List (single choice), Pre-Conditions Editor, Requirement Status, Requirement Status Custom Field, Test Association with a Pre-Condition, Test Association with a Test Plan, Test Environments Custom Field, Test Execution Association with Tests, Test Execution Defects, Test Execution Status Custom Field, Test Plan, Test Plan Association with a Test, Test Plan Root Folders Custom Field, Test Plan Status Custom Field, Test Repository Path, Test Run Status Custom Field, Test Set Status Custom Field, Test Sets Picker, Test Type Select List (single choice) and Tests to PDF.
  • You can export Xray-managed information using the highly-configurable sections in the issue-fo.vm template:
    • for tests:
      • for manual tests: test repository path, test type, test steps (instructions, input/data and expected result), pre-condition list, test set list, test plan list and test run list
      • for Cucumber tests: test type, scenario type, scenario, pre-condition list, test set list, test plan list and test run list
      • for generic tests: test repository path, test type, test definition, pre-condition list, test set list, test plan list and test run list
    • for pre-conditions: pre-condition type, condition and test list
    • for test sets: overall execution status and test list
    • for test plans: begin date, end date, overall execution status, test list and test execution list
    • for test executions: test plan, test environments, begin date, end date, revision, overall execution status, test list

(Need more features? Tell us in a support ticket!)

Tutorial video

Better PDF Exporter exports all test-related information, including all Xray-specific entities and all Xray-specific custom fields, to PDF:

Export samples

Xray test (manual) with test runs

Xray stores most of the information in its specific custom fields, all supported by Better PDF Exporter. This comprehensive Xray manual test export includes the built-in issue fields, plus test type, test repository path, test steps, plus the pre-conditions, test sets and test plans associated with the test, plus the table with test runs.

jira-xray-manual-test-with-test-runs.pdf

Xray test (Cucumber) with test runs

Xray supports automated Cucumber tests to enable the behavior-driven development process (BDD). Cucumber scenarios and scenario outlines, written in the human-readable Gherkin language, are precisely exported to PDF.

jira-xray-cucumber-test-with-test-runs.pdf

Xray test (generic) with test runs

The multi-purpose generic tests in Xray define the testing instructions in free text, instead of formalized test steps. This type of test definition is precisely exported to PDF, too, preserving the rich formatting, embedded image illustrations and attachments.

jira-xray-generic-test-with-test-runs.pdf

Xray test list

This example gives a overview of the manual tests created with Xray, in a compact format optimized for reviewing, sharing or archiving test information. This PDF file is generated with just a single click from the columns selected in Issue Navigator. Note that the unlike the web interface, the last 3 columns also display the associated issues' summaries.

jira-xray-test-list.pdf

Xray test set with tests

Test sets are flat, ordered collections of tests, represented by their own Xray-specific issue type in Jira. This PDF document captures a test set with all its details, plus the tests contained by the test set with all their details, resulting in a complete self-containing snapshot of the test set information.

jira-xray-test-set-with-tests.pdf

Xray test set list

Unlike the previous example which contains all details, this PDF export focuses only on the key information of test sets. The table format works great as a high-level test set report.

jira-xray-test-sets.pdf

Xray test plan with test executions

Test plans collect the tests to be validated against a specific product version or within a specific sprint, their schedule, and the respective results. You can find a complete test plan with its details plus the related test executions with their details in this PDF document.

jira-xray-test-plan-with-executions.pdf

Xray test plan list

This PDF report summarizes the work to be done in the testing phase by listing the test plans in an easy-to-grasp format.

jira-xray-test-plans.pdf

Xray test execution with test runs

Test executions are assignable tasks for executing a group of tests against a specific system revision (version) in a specific environment, as a part of the testing plan. This file contains a test execution, the tests to executed within that test execution, and the corresponding test runs.

jira-xray-test-execution-with-test-runs.pdf

Xray test executions

This PDF document contains the most important test execution details, selected as Issue Navigator columns, including the test results and the defects (bugs) found.

jira-xray-test-executions.pdf

Xray pre-condition with tests

Xray pre-conditions define the required state of the system for the test execution to start. This PDF document contains a detailed pre-condition and all the detailed tests that require the pre-condition. In other words, tests and their linked information are archived to the same self-containing file.

jira-xray-precondition-with-tests.pdf

Xray requirement with tests

Xray tests can be linked to the requirements which they validate (and which themselves are Jira issues, too) using the "tested by" type issue links. Better PDF Exporter can follow those links and generate a Requirements Specification document that also include the corresponding tests.

jira-xray-requirement-with-tests.pdf

Configuration

Exporting Xray custom fields

There is nothing to do. Better PDF Exporter will automatically recognize the Xray managed fields and export them accordingly.

Configuring the PDF templates for Xray

The template issue-fo.vm contains the following configuration parameters in its top part, to enable these features:

## Xray
#set($exportXrayTestExecutions = true) ## set to "true" to export Xray test executions
#set($exportXrayTestRuns = true) ## set to "true" to export Xray test runs

Note that the issue-fo.vm template shows the Xray-specific sections only for the issues that are actual Xray entities. It decides whether an issue is an Xray entity by checking its issue type name (e.g. if it is a "Test Plan") and its custom fields (e.g. if it has the "Test Type" custom field). Therefore, if you rename issue types, you also need to adapt the logic around this part in the template code:

## test management data
#set($testManagementIssueTypeNames = ['Pre-Condition', 'Test', 'Test Execution', 'Test Plan', ...])
#if($testManagementIssueTypeNames.contains($issue.issueTypeObject.name))
	## ...

Configuring the export format for Xray "associated with" custom fields

Xray is using several custom field types that associate an Xray-managed issue type with another, e.g. associate tests with test sets. In the Xray web interface, these values are displayed in the format "FOO-123 FOO-456" where each item is a link. (For some unknown reason, Xray does not even separate the items with commas.)

When exporting to PDF, Better PDF Exporter resolves the issue keys to summaries, and expands the values to the longer, but externally usable format "[FOO-123] My first test (line break) [FOO-456] My second test". If you want to turn this off and return to "FOO-123, FOO-456" (with commas!), set this configuration variable to false in the top of the issue-fo.vm and issue-navigator-fo.vm templates:

#set($exportDetailedValues = true)

Note that the same variable also controls the format of some other fields, as explained in the next section.

Configuring the export format for Xray "Test Plan Status" and "Test Set Status" custom fields

When you have 4 tests and all those were executed successfully, the status overview fields will be exported in the "full" format that includes all statuses (even those with zero test counts):

PASS: 4 (100%), FAIL: 0 (0%), ABORTED: 0 (0%), EXECUTING: 0 (0%), TODO: 0 (0%)

For the easier reading, you may want to use the compact format that shows only the non-zero statuses:

PASS: 4 (100%)

You can switch to the compact format by setting this configuration variable to false in the top of the issue-fo.vm and issue-navigator-fo.vm templates:

#set($exportDetailedValues = true)

Note that the same variable also controls the format of some other fields, as explained in the previous section.

Also note that the "Test Execution Status" custom field is implemented differently in Xray (for an unknown reason), and as a consequence, it is always exported in the full format.

Configuring the export format for Xray manual test step custom fields

Xray has two separate custom fields that display the test steps of manual tests, both being supported by Better PDF Exporter. Note that their display format is slightly different in the web UI and also in the PDF documents.

This table helps to choose the one that is more suitable for your exports:

Custom field name Format in the PDF document
"Manual Test Steps" "Step", "Data" and "Expected Result" are exported to separate lines, with an empty line between steps.

Tip: choose this if the space in the paper is limited. (This format is using less paper space.)
"Manual Test Steps (Export)" Table with "Step", "Input/Data" and "Expected Results" columns.

Tip: choose this for most use cases. (This format is more human-readable.)

Exporting associated Xray test entities

In many use cases, you want to export multiple types of associated, Xray-managed issues with their full details to the same document. For example, you want to export a test set (with its details) plus the tests included in the test set (with their details) to the same document, for a comprehensive test report or test archive.

It is perfectly doable, because Better PDF Exporter allows freely mixing issues of various types within the same PDF document. The question is how to search for the input issues that will result in the expected document? In the above example, how to search for the test set and its tests with proper ordering, using the Issue Navigator?

Luckily, the JQL query language and the Xray custom JQL functions offer flexible searching for associated Xray issues. In the next sections, we will give working examples for typical use cases. While these are great to get the base idea, we strongly recommend reading the Xray queries page for all possibilities.

Exporting an Xray test entity and its associations

These examples show how to find the result merged from:

  1. a single "source" test entity identified by its issue key (e.g. "key = FOO-123")
  2. the associated issues returned by an Xray-specific JQL function which is parametrized with the source entity's key (e.g. "testPlanTests(FOO-123)" returns the tests in the FOO-123 test plan)

Our preferred ordering is exporting the source entity in the first position, then the associated issues ordered by their keys. For this, the examples use the "ORDER BY issuetype ASC, key ASC" clause, which works as expected for English issue type names. It may happen that in your language, the issue type name of the source entity and the issue type name of the associated issues are in the reverse order. If so, just use "ORDER BY issuetype DESC, key ASC".

Based on this recipe, you can easily craft your own JQL, but here are some examples with no further explanation.

Exporting a test plan and its associated tests

key = FOO-123 OR key IN testPlanTests(FOO-123) ORDER BY issuetype DESC, key ASC

Exporting a test and its associated pre-conditions

key = FOO-234 OR key IN testPreConditions(FOO-234) ORDER BY issuetype DESC, key ASC

Exporting a test and its associated test executions

key = FOO-345 OR key IN testTestExecutions(FOO-345) ORDER BY issuetype ASC, key ASC

Exporting a requirement and its associated tests

key = FOO-456 OR key IN requirementTests(FOO-456) ORDER BY issuetype ASC, key ASC

Exporting multiple Xray test entities and their associations

These examples are similar to the ones in the previous section, but in this case we have multiple source entities. For example, you want to export multiple test plans and all their associated tests.

To accomplish that, create a saved filter that returns the source entities. Then refer that twice in the final JQL query to get the merged result, totally similarly to what you did in the previous examples with the source entity's issue key.

Also in this case, our preferred ordering is the source entities first, then all the associated issues after those. For that, just use an "ORDER BY issuetype ASC, key ASC" or "ORDER BY issuetype DESC, key ASC". (Of course, if an issue is associated with more than one source, it will be exported only once.)

Exporting multiple test plans and their associated tests

filter = "My Tests Plans" OR key IN testPlanTests("My Tests Plans") ORDER BY issuetype DESC, key ASC

Configuring the Xray REST API access

As this integration relies on the Xray REST API, you need to configure the login credentials of a valid Jira user account for the REST API calls in the templates:

  1. Go to AdministrationAdd-onsPDF Templates (under Better PDF Exporter).
  2. Open the issue-fo.vm template for editing, and set the username and password to these configuration variables in the top part (don't remove the quotation marks around the string!):
    ## Jira user credentials for REST API calls
    #set($restUserName = "admin") ## Jira username for REST authentication
    #set($restPassword = "admin") ## Jira password for REST authentication
    
  3. Save the changes. (Don't worry about storing passwords here: this file is visible only for Jira administrators, who would have super-user permissions any way.)

Notes:

  • If you are using multiple REST API based integrations in the same template (ex: Gliffy plus Zephyr), then all REST API calls will use with the same user credentials.
  • As a good practice, this is safer to avoid usernames and passwords that contain non-English characters. Albeit our Xray script correctly encodes international usernames and passwords, their handling also depends on the configuration of the container that hosts your Jira web application (typically Tomcat). If you're having difficulties, just replace those characters in your username or password with English letters or numbers.

Finally, the following section describes a confusing situation when all REST API calls result in "Server returned HTTP response code: 403" or "401" errors. If you are not affected, you can skip this section.

Why does this happen? As its default behavior, Jira will lock your account and present a CAPTCHA on the login form after a few unsuccessful login attempts. Since the Xray integration may send several REST API requests per export, all using the same username and password, the unsuccessful login attempts limit can be reached very quickly if the password is wrong.

How to fix it? You just have to pass the CAPTHCA verification, that's it. First off, be 101% sure that the REST username and password are correct. If those are, but you are still receiving 403's, please open Jira in your browser, logout and login with the same user account which you configured for the REST calls. You will be asked for a CAPTCHA verification, but after answering that and logging in to Jira, the REST calls will also be successfully executed!

Please note that it will never happen again, unless you change the password of the corresponding Jira user account.

If you are an administrator, there is also a quicker solution: go to Administration → User management and clear the failed login attempts counter with one click. Or, you can increase the allowed login attempts or even disable this feature altogether in Jira.

Troubleshooting

I cannot export test runs from pre-2.0.0 Xray versions

Exporting the test runs with the issue-fo.vm template relies on the /rest/raven/1.0/api/test/{key}/testruns endpoint provided by the Xray REST API. This endpoint is available since Xray 2.0.0. Therefore, Better PDF Exporter exports test runs only with Xray 2.0.0 or newer versions.

Note that other information, stored in Xray-specific custom fields, can be correctly exported even with older Xray versions.

Learn more about Xray