DoxRunner
​
Non-Functional Requirements (NFRs)
​
As all performance testers know, performance testing comes under the umbrella of non-functional testing. Performance requirements include response times and error rates under various loads. They are an essential component of performance testing for many players in application development and upgrades, so documenting them is critical.
​
They also may vary widely from organisation to organisation, from project to project, and sometimes from test to test.
​​​​​​​
They are normally documented only in the solution document, but, if necessary, they can be documented in any test case.
The DoxRunner process raw operation adds them to the script as comments, as shown in the illustration below, which shows how they are documented in an NFR section (top), then how they are documented in a timer (middle), and how the comments are formatted in a script (bottom).
​​​​​
​
Process Raw Operation
Assuming that every Timer has an NFR column, the Process Raw operation simply adds one of two types of comment to each Timer in the script as shown at xxx and xxx in the illustration below.
​​
DoxRunner doesn't use the Category.
If the NFR in the timer table isn't present in the NFR table, then the value is used verbatim.
​
If the NFR in the timer table is present in the NFR section, then the value is used to access the Requirement from the NFR section.
​​​​​​​​
​
If the NFR in the timer table isn't present in the NFR section, then the value is used verbatim. In the illustration it's "12 secs".
​
If the NFR in the timer table is present in the NFR section, then the value is used to access the Requirement from the NFR section and that text is inserted as a comment.
​
DoxRunner doesn't use the Description.
​​​​​
​
Documentation​​
​
​​Often NFRs are written in external documents in ways that don't match how performance testing uses them.
Placing them in the solution document ensures they are all in one place for team members and the business to easily reference. They tend to be fairly stable, but since they may change from time to time or get added to, it's worthwhile reviewing this section from time to time.
​​​
​​Why would you use the solution document to document NFR rules? You don't need to, but if there are rules that are common across multiple test cases, then they may be easier to manage if placed in the solution document instead. In fact, to take an extreme view, you can document -all- NFR rules in the solution document and define -none- in the test cases, if that is easier to manage. The choice is yours.
​
If an NFR in the solution document is the same as one in a test case then the one in the test case takes precedence when processing a raw script.
​
​​If you have no intention to document NFRs, then feel free to delete this section.
​
Getting Started
​​
Assess whether documenting NFRs add value to the organisation​. If not, delete the section from the solution document and ignore this whole page.
​
Items 1 and 2 below are typically only done once, with the occasional review and tweaking if necessary.
Item 3 is where all the action takes place.
​​​​
1. When the solution document is downloaded:
-
Decide whether you want to document solution-wide NFRs;
-
If so: Leave the NFR section in place and add all NFRs that the business may require;
-
If not: Delete the NFR section manually.
-
​​
2. When the test case template is downloaded:
-
Decide whether you want to document NFRs on a test case basis;
-
If so: Update the NFR section as required;
-
If not: Delete the NFR section manually.
-
​
​3. Create the test case(s).
For each test case:
-
Add the timers;
-
Assess the NFR for each timer and see whether any in the solution document NFR section applies;
-
If so, place the NFR ID into the timer;
-
If not, either add it to the solution document NFR section or to the test case NFR section.
-
​​
​​
Solution Document
​​​
​​An NFR section is optional in the Solution Document and is semi-structured as detailed in the NFR Section paragraphs below.
​
​It must be referenced by a Bookmark that must be located immediately before the Section Title.
The Bookmark is structured as follows: V_NFRs
​If an NFR ID is the same as one in a Test Case, the one in the Test Case takes precedence in any DoxRunner operation.
​​​​​​
​​
​
Test Case
​
Normally an NFR section is not required in a Test Case - those in the Solution Document are referenced where required. However, if one or more NFR is required for a Test Case and it needs to have a different value to the corresponding one in the Solution Document, then it should be added to the Test Case.
The value in the Test Case will override any corresponding one from the Solution Document.
​
The only difference between the Test Case version and the Solution Document version is the Bookmark. It must contain the Test Case ID and is structured as follows (where CEP001 is an example Test Case ID). See also the Section Title further down this page.
P_CEP001_NFRs
​
If it is not required, the Manage Sections component of the Test Case Operations facility can be used to delete it.
​​​
​
Test Case Template
​
An NFR section is present in the Test Case Template that is downloaded from the DoxRunner site. ​
​
It should only remain if there are expected to be many test cases requiring different NFRs to those documented in the Solution Document as discussed above.​
​
It is semi-structured. The structure is similar to the versions in the Test Case and Solution Document, with the only difference being the name of the referencing bookmark, which is:
P_NFRs​​.​
​
If it is not required, it can be manually deleted.
​
​
NFR Section
​​​​​​​​​​​
NFR Description​​​​​​
​​​​​An NFR section is optional, is semi-structured, and usually appears only in the solution document. It can be located anywhere in it, so long as its bookmark is in the correct place.
However in some cases, it may optionally appear in a test case and/or the test case template, usually as a cut-down version.
​
As with most managed items, NFRs are documented within a Microsoft Word section, as shown in the example below. ​
​​​​​
​​​Each instance of an NFR section consists of:
-
a bookmark;
-
a description;
-
a table with, ideally, four columns (Category, NFR ID, Requirement, and Description);
-
Only the Type and Value columns are mandatory, but the Description column is highly recommended;
-
More columns can be added.
​​​​​
An example of the NFR Section is reproduced in the illustration below.
​​
The bookmark:​
-
Structure: Depends on which document the section is embedded in:
-
Solution document: V_Configuration
-
Test Case: P_TST110_Configuration (where TST110 is the Test Case ID)
-
Test Case Template: P_Configuration
-
​​
-
Structure: Free-format title text immediately preceded by the bookmark;
-
Style: I_Heading n or I_Appendix n, where n can be an integer from 1 to 5;
-
Length: Cannot be longer than 200 characters.
​
The body text
-
Structure: free-format text and optional:
-
Location: between the section title and the table;
-
Style: I_BodyText;
-
Length: Cannot be longer than 1,000 characters.
​​
The table:
-
Location: immediately below the body text (or section title if there is no body text):
-
A minimum of two rows:
-
A heading row and a data row;
-
If there are no NFRs, the data row can contain empty cells (or the section can be deleted);
-
-
A minimum of two columns:
-
An ID column and the Requirement column;
-
More columns can be added;
-
A Description column is recommended.
-
-
All cells in the ID and Requirement columns must conform to specific rules, as described in following paragraphs.​
​
​
​NFR Section Title​​​​
​
The section title is slightly different depending on the document in which the section resides, as discussed in the following paragraphs.
​​
​​​​
​
Solution Document Section Title
​
The example in the illustration on the right shows the section title
and its three components as it is expected in the solution document:
​​​​​​
​ The bookmark:​
-
It's mandatory;
-
Located immediately before the Title Text;
-
Indicated by the small vertical bar located immediately before the title text;
-
If you cannot see the vertical bookmark bar, it's a good idea to configure Word so that bookmarks are visible;
-
Do not move it or change it unless it's in the wrong place. DoxRunner depends on it during the process raw operation;
-
Structure: V_NFRs;
-
Length: Cannot be longer than 30 characters.
​
The title text can be manually modified to any text desired.
-
It's mandatory;
-
Located immediately after the bookmark and above the section description;
-
Structure: Free-format text;
-
Length: Cannot be longer than 200 characters;
-
Style: I_Heading n or I_Appendix n, where n can be an integer from 1 to 5.
​
The triangle following the title text is a link to the top of the test case summary section. It is useful for navigating around the Solution document. It is useful, but not critical, so it can be deleted if it annoys you.​​​
​​​​​​
​
Test Case​​​​ Section Title
​
The example in the illustration on the right shows the section title
and its three components if it were to be added to a Test Case.
​​​​​
​ The bookmark:​
-
It's mandatory;
-
Located immediately before the Title Text;
-
Indicated by the small vertical bar located immediately before the Title Text;
-
If you cannot see the vertical bookmark bar, it's a good idea to configure Word so that bookmarks are visible;
-
Do not move it or change it unless it's in the wrong place. DoxRunner depends on it during the process raw operation;
-
Structure: P_TP901_NFRs (where TP901 is the Test Case ID);
-
Length: Cannot be longer than 30 characters.
​
The title text can be manually modified to any text desired.
-
It's mandatory;
-
Located immediately after the bookmark and above the section description;
-
Structure: Free-format text;
-
Length: Cannot be longer than 200 characters;
-
Style: I_Heading n or I_Appendix n, where n can be an integer from 1 to 5.
​
The triangle following the title text is a link to the top of the test case. It is useful for navigating around the test case.
It is useful, but not critical, so it can be deleted if it annoys you.​​​
​​​​​
​
Test Case​​​​ Template Section Title
​
The example in the illustration on the right shows the section title and its
three components if it were to be added to the test case template:
​​​​​
​ The bookmark:​
-
It's mandatory;
-
Located immediately before the title text;
-
Indicated by the small vertical bar located immediately before the title text;
-
If you cannot see the vertical bookmark bar, it's a good idea to configure Word so that bookmarks are visible;
-
Do not move it or change it unless it's in the wrong place. DoxRunner depends on it during the process raw operation;
-
Structure: P_NFRs;
-
Length: Cannot be longer than 30 characters.
​
The Title Text can be manually modified to any text desired.
-
It's mandatory;
-
Located immediately after the bookmark and above the section description;
-
Structure: Free-format text;
-
Length: Cannot be longer than 200 characters;
-
Style: I_Heading n or I_Appendix n, where n can be an integer from 1 to 5.
​
The triangle following the Title Text does not link to anywhere. It is a placeholder, ready for a link when a Test Case is created.
It is useful, but not critical, so it can be deleted if it annoys you.​
​​
NFR Table
​​​
The NFR table is the heart of the NFR documentation.
​
-
It's mandatory;
-
Located: immediately below the NFR section description (or NFR title if there is no NFR section description):
-
A minimum of two rows:
-
A heading row and a data row (one data row per NFR);
-
-
Ideally four columns:
-
Category, NFR ID, Requirement, and Description;
-
More columns can be added;​
-
-
Do not change the column heading text - DoxRunner operations use them;
-
All cells must conform to specific rules, as described in the NFR table column Headings below.​
​​​​​​​​​
NFR Table Column Headings​
​​​​
​​The illustration below shows the relationship between the four columns of the NFR table and the LoadRunner script as prepared by the process raw operation.
​​​
​
Category Column
​
Refer to xxx in the illustration above.
​
This column is an attempt to group NFRs into various categories.
​
For example, a timer that includes database requests may typically be slower than simple screen interaction, so there may be several NFRs in the Database category.
It is optional, and DoxRunner does not use it.
​
​
NFR ID Column
​
Refer to xxx in the illustration above.
​​
This is the NFR's unique identifier. If it relates to a timer, then this can be used to reference it from the Response Time NFR column in a documented timer. It is meant to be short so that it fits into the NFR column of a timer. Hence the usefulness of the Category column to expand on it and offer more context.
​
It is used to reference the Requirement column which is inserted as a comment into the script near the beginning of the lr_start_transaction() cluster.
​
Represents an NFR in the Timer that does not reference an NFR in the NFR table.​ It is inserted into the script as a comment verbatim.
​
Represents the NFR with ID M in the NFR table. The Requirement text is inserted into the script as a comment.
​​​​​
Requirement Column
​
Refer to xxx in the illustration above.​
​
The text in the Requirement column is meant to be a summary of the NFR and is the text that is inserted into the script as a comment near the beginning of the start timer cluster.
​
Description Column
​
Refer to xxx in the illustration above.
​
This is simply a more expanded version of the text in the Requirement for more detailed documentation purposes. It is optional and DoxRunner does not use it.
​
​
​
​
Manage Test Case NFR Section
​​​​
​The test case operations option is used to manage an NFR section, but only within a test case.
​​
​​​​​​
​Add NFR Section
​
An NFR section is included in the Solution Document when it is downloaded. If for some you deleted it and need to re-add it, you will need to add it manually.
​
Because it is not considered necessary to be a part of a test case, it is not included in the Test Case Template when it is downloaded. If you need it to be included in any new test case, you will need to add it manually.
​
If it is not included in any new test case, but for some reason you want to add it to a specific test case, then use the Add Section function within the Test Case Operations suite. The illustration below gives some guidance.​
​ Select the Test Case
Press Test Case Operations
Select Manage Sections
The NFR option should be empty. If so, select it. It should change to Add.
​
Press Apply
Check that it created OK
Add NFRs as required.
​​​​​​​​​​​
​​
Delete NFR Section
​
If an NFR section was added to a Test Case but needs to be deleted, then use the Delete Section function within the Test Case Operations suite.
​ Select the Test Case;
Press Test Case Operations;
Select Manage Sections;
The NFR option should show
Exists. If so, select it.
It should change to Delete;
Press Apply.
Check that it deleted OK.
​​​​​​​​​​​​​