Scenario Editor

PureLoad Logo
PureLoad 5.3
April 2016
http://www.pureload.com
support@pureload.com

Documentation Index

Introduction

This chapter describes how scenarios and tasks are defined and also the various features related to scenario management in the scenario editor.

Scenarios

Scenarios of tasks is the common structure used throughout all applications in the Pure Performance suite. A task is the actual function that will be performed, i.e get a web page, insert data into a database, etc. A series of tasks are defined in a scenario. An extensive set of features is available to make tasks modular and dynamic.

A scenario in the Pure Performance suite consists of task sequences and tasks. The purpose with the scenario is to assemble all tasks as they are going to be executed in any of the run time environments. I.e PureLoad, PureAgent or PureTest.

Task sequence is basically a container that groups other task sequences and tasks. An example is a web application where a scenario shall include requests to many web pages with their specific HTML pages and graphics. All requests for a complete HTML page is in this example grouped in a task sequence:

scenario

Here we now have a Scenario (Test Login) that consists of tree task sequences, one per web-page (Start Page, Login and Logout). Each of the task sequences consist of tasks that represent one HTTP request.

Task Sequence References

Multiple task sequence can refer to another task sequence. This way a common sequence can be used multiple times from multiple scenarios (within the same PLC file).
A few restrictions applies when using task references:

To allow configuration of references sequences scenario variables may be used.

Using the Scenario Editor

Once the PureLoad Console or PureTest application has been started a window similar to the following appear. It consists of one main tab, Scenarios which have tree sub tabs at the bottom of the screen:

editor

The left pane shows the managed scenarios in tree structure. The tree can maintain several scenarios concurrently. The menu items and tool bar buttons are used to manage the content of the scenario tree. Selecting an object in the tree displays the characteristics of the object in the properties panel to the right.

The scenario tree presents objects by the value of the Name parameter. The number within parentheses indicates the number of iterations that will be executed. The Sequences/Tasks information below the tree indicates the following:

Note: There is basically no difference between a scenario and a task sequence object. Scenarios can however only be defined on the top level and may also define a distribution. Scenarios are also the object type that is executed in the runtime environment.

The Name Parameter

The Name parameter that can be specified in the property panel for scenario, task sequence and task objects need some extra attention. Name is primarily used in the scenario editor and in the result presentation. Its default value is the same as the selected task class but can be specified to anything meaningful. Name is also used when PureLoad visualizes the tasks in the the result presentation. All tasks with the same name per scenario or task sequence will be presented as one task in the result presentation.

An example is a scenario for load testing of a web site. The scenario to the left will give a result tree as shown to the right:

If we are not interested in individual results for the image URLs, we can name these tasks "image" and the results will be displayed as:


Creating a Scenario

Create a new scenario by selecting the Root node in tree and select Edit->Create Scenario or use the  button in the tool bar:


Selecting the newly created scenario in the scenario tree shows the properties for the scenario in the right area. The properties are:

Adding Task Sequences

A task sequence is a container in which tasks and other task sequences can be grouped. There is no limit on the depth of a scenario tree.

Select the object in the scenario tree into which the task sequence shall be added and select the Edit->Create Task Sequence menu option. If the currently selected object in the tree is a scenario or task sequence then the newly created task sequence will be added as a child object. If a task object is currently selected then the new task sequence will be created below the selected task object:


Selecting the newly created task sequence in the scenario tree shows the properties for the task sequence in the right area. The properties are:

Adding Task Sequence Reference

A task sequence reference is a container that is read-only and refers to another task sequence. This means that you can have groups of tasks that is used from multiple other sequences.

Select the object in the scenario tree into which the task sequence reference shall be added and select the Edit->Create Task Sequence Reference menu option.A dialog will presented with the target references that can be selected When the target sequence is selected add the Add button is choose it can look something like the following:


The newly create task sequence reference is selected. The properties are the same as for an "ordinary" task sequence, except that the path to the referred sequence is displayed.

Adding Tasks

Tasks are added either to a scenario or a task sequence. Select the Edit->Create menu choice to create a new task. If the currently selected object in the tree is a scenario or task sequence then the newly created task will be added as a child object. If a task object is currently selected then the new task will be created below the selected task object.

Note: The tool bar button  creates an object in the tree according to the current selection. If the selection is either a scenario, task sequence or task object then the new object is always a task.

The following dialog lists the Task Types to choose from. Select a task in the list and press Add or double click to add the task type.


The added task will now appear in the scenario editor:


All tasks have task parameters. These are grouped into generic task parameters and task specific parameters. All tasks have the following generic parameters:

In addition there is one task specific parameter that are included for most asks:

Specifying Task Parameter Values

All tasks are using the task parameter mechanism. Task parameters defines the input interface for the task and the actual values are specified in the scenario editor. Task parameters might be of different types and are displayed different depending on type.

The value for all task parameters can automatically be defined using task parameters generators and/or variables.

Undo/Redo

You can undo and redo up to 25 edits in the scenario tree. Edits include edits of parameters for a node, added tasks or sequences etc.
To undo or redo an edit, select Undo/Redo in the Edit menu or use

Executing and Debugging Scenarios

Scenarios of tasks are normally executed in any of the Pure Performance suite applications. This can be PureLoad for load testing, PureAgent for performance monitoring or the PureTest application itself. This section describes how to execute tasks from within the Scenario Debugger in the PureTest application.

The main purpose with the Scenario Debugger is to quickly test and verify each individual task in a scenario. This is done by single stepping through each task in a scenario. Detailed execution results and execution log is shown for each task.

The Scenario Debugger tool bar

The debugger mini tool bar is displayed by selecting the View->Show Debug Toolbar menu choice (if not already displayed) and its control buttons will appear in the Scenario Editor sub tab:


The following operations are available in the debugger tool bar:

Toggle debug/edit mode
Edit mode is the default mode and debug mode is used to run the debugger. The scenario tree can only be manipulated in edit mode.
Start debug session
Starts a debug session. The first task in the scenario will be current. This state is indicated with an arrow () in front of the current task in the tree.
Stop execution
Stop the currently executing task.
Continue/Start execution
This operation either continues or starts the execution to the next available breakpoint or to the end of the scenario.
Single step
Single step into the next task by first executing the current task and then move to the next task.
Step Over (skip)
Single step over (don't execute) the next task and move to the next task.
Toggle breakpoint
Toggles a breakpoint for the currently selected task.

Running the Debugger

Make sure debug mode is enabled (use the button in the debug tool bar).

Now press the button to start the debug session. This operation moves the execution pointer to the first task in the scenario which is indicated by the task pointer icon (). This pointer indicates that the task is going to be executed during the next Single Step (), Step Over () or Continue () operation.

The task execution result is presented to right of the scenario tree:


There are three sub tabs in the Task Execution Result view:

Now you can continue to step trough the scenario analyzing the results from each task executed.

Parameter Generators

Scenarios are often designed so that they can be parametrized and repeated many times with different data. The parameter generator mechanism makes this possible. Each task parameter value can refer to parameter generators to have the actual value generated during the test.

The following parameter generator types are available:
parameter generators
Parameter generators have five operations that they can perform:

  1. Get Next
    Gets next value.
  2. Get Previous
    Gets previous value.
  3. Get Random
    Gets a random value among possible values for the generator.
  4. Get
    Gets current value without stepping to next.
  5. Reset
    Resets generator to initial value which is also returned.

Creating a Parameter Generator

Before assigning a parameter generator to a task parameter it needs to be created and configured. This is done in the Parameter Generators sub tab.

Parameter generators are categorized as internal or external. Internal parameter generators are defined in the same scope as the currently loaded scenario and is stored in the same file. External parameter generators are stored in one or more separate file and can be shared between multiple scenarios.

Select the Internal node and use View->Create or the create button in the tool bar to create a new internal parameter generator. This will bring up a dialog with the available parameter generator types.

create parameter generator

Double click on selected parameter generators or press Add to create it. All parameter generators have a Name parameter that must be unique. It is recommended to set a name that match the purpose with the parameter generator. For example User ids, Cities, Models, etc.

To create a new external parameter file or open an existing external parameter generator file, select the Root node and use View->Create or the create button. Parameter generators can be added and removed to external nodes in the same way as for the internal node. Changes in external parameter generator files can be saved when the whole scenario is saved.
external parameter generator

Counter Parameter Generator

The counter generates numbers within specified boundaries.

counter

Start and Stop defines the range. The special constant INF is used for infinity (i.e the maximun number that can be represented). The Format parameter may be used to format the generated number.

Typical  examples are:

00000
Format the number using 6 digits, i.e 000001, 000002, 000003 etc.
##,###,###
Format the number using grouping separator, i.e. 10,001,  10,002 etc. 

File Parameter Generator

The file parameter generator read values from a file. The file can either contain one value per row or several values per row. Several values per row must be delimited with a character that is used to split the values into Elements. The file below contains 6 rows, each with 3 elements separated with a "%" character.

A road to heaven%Clark Moore%287
Somewhere in the sky%Vanessa Hearst%99
Man in the moon%Dirk Dustin%129
Java Server Pages%Hans Bergsten%199
Gagarin: a myth%Roland Petersson%19
Sun, Moon, Earth%Laura Craft%78

The following figure shows the properties for the file parameter generator. The books.txt file has been loaded and the Preview Data button has be pressed. Since the specified Field Separator matches the separator in the file will three columns appear in the grid each representing the values in the file. The name of each element can be changed by clicking on the table header and it is those names that are used to access the parameter generator elements in a task parameter.
file generator

File Path:
The location of the parameter file, i.e. the "File Path",  can be given in four different ways:

  1. Absolute path: e.g. /home/myName/myParamDir/myParamFileName.txt
  2. Relative to PLC file: $PLC/myParamDir/myParamFileName.txt, where $PLC represents the directory where the currently loaded .plc file is located
  3. Relative user home: $USERHOME/myParamDir/myParamFileName.txt, where $USERHOME represents the home directory of the current user
  4. Relative to PureLoad/ PureTest home: $PUREHOME/myParamDir/myParamFileName.txt, where $PUREHOME represents the PureLoad/PureTest home directory (this is the directory where you will find e.g. the /bin, /lib and /extensions directories)

When using a relative file path ($PLC, $USERHOME, $PUREHOME) be sure to save the .plc file before attempting to test or preview the data, or else the parameter file may not be found.

Accessing the parameters:
The following syntax is used in a task parameter to access a element in a file parameter generator:

$Books.Prices.getNext
$Books.Authors.getNext

If having several elements defined for a file parameter generator then each of these are managed separated from another. I.e doing a $Books.Prices.getNext will only forward the row pointer for that specific element and not for any other.

List Parameter Generator

The list parameter generator is handy when a small list of values shall be read. The management of the list is done in the List parameter generator options pane.

list generator

Calendar Parameter Generator

The calendar parameter generator is useful when generating timestamps values. The generator can be configured to increase parts of the timestamp between each getNext operation.

Having a Date Format set to yyyy-MM-dd HH:mm and Start Time set to 2007-10-12 13:37 and Increment set to 1 Hours will cause each use of the getNext operation to return the following values:

2007-10-12 13:37
2007-10-12 13:38
2007-10-12 13:39
etc.

The Date Format lists a few standard formats but it can be manually edited to accommodate custom patterns.

generator date

Unique Parameter Generator

The Unique parameter generator can be used to generate a random unique identifier in standard GUID/UUID format. Using getNext will give you (for example) the following values:

3cdeab54-2b6a-4fe1-bfac-b1fbe0284266
e5a01b89-61f8-45a9-b1fb-f4c8f70d38a2
53e2e70c-2c65-4b52-97af-3bab4472e2d6
etc.

Using get operation will retreive the last used value; all other operations generates a new unique identifier.

IP Address Parameter Generator

The IP Address parameter generator can be used to generate IPv4 and IPv6 IP Addresses. This is done by specifying an IP Address range. Optionally a regular expression may also be used to exclude addresses from this range.
IP Adress Parameter Generator

The above will generate IP addresses starting with 192.168.1.3 to 192.168.1.254.

IP Address Range may be specified using CIDR format or an range (from-to).

Testing a Parameter Generator

All generators can be tested using the Test button at the bottom of the pane. The following dialog is displayed for all but File generators.

generator test
The Operation buttons are used to select the operator to use when getting the value using the Test button. The Parameter field shows the variable as it shall be used in any Task Parameter.
generator file test

The main difference between the test dialog for file parameter generators is that an element (column) may be selected.

Scenario Variables

A powerful feature when testing a dynamic application is to extract information from a task, save his information to a scenario variable and use defined variables in subsequent tasks.

Variables can be used in any task parameter. The syntax of a variable is:

${variable-name}

where the value, if any, of variable-name is substituted before the task is executed. A task parameter value can mix static text with variables and also contain more than one variable. Assume that we have two variables defined itemno and itemcount, then we could for example have a URL parameter to a HTTP tasks as follows:

http://some.where.com/getItem?item=${itemno}&count=${itemcount}

Assigning Values to Variables

Variables values are set by Tasks. To assign a value to a variable a task must be used. Normally this is done by extracting part of data received from a previous task, using any of the extract tasks available. The SetVariableTask can be used to initialize a list of variables to either static or evaluated values.
assign variable values
Evaluated values can use the same syntax as used by the ScriptTask. In the above example, variable delay will be evaluated to 130 and variable s will be set to the string "mmm".

The most common use of variables is when testing a dynamic web application. Please read more about testing web applications in the Testing Web Applications guide. Using the generic extract task it is possible to extract data and assign a value to a variable from any task that returns data. This includes Telnet, FTP, Mail and JDBC tasks.

Display variables used

To find out variables used by a scenario select a scenario and choose the View->Show Variable Usage... menu choice. This will display a dialog with the variables used and the task path to variables (and parameter generators) used:

The Task column is a selectable link to the task where each variable is used.

Conditional Execution 

Task sequences can have a condition which determines if the sequence should be executed or not depending on previous errors or an execute condition expression.

Stop On Error

A scenario is normally executed task by task from beginning to end. The default behavior is to ignore errors in individual tasks and continue execution. By using the RuntimeSettingsTask the runtime settings flag "Stop On Error" can be set to false, to force the whole scenario execution to end in case of a task error.

However, sometimes it is important to execute certain tasks to perform cleanup even if an error has occurred. For this purpose, the "Stop On Error" can be overridden for a task sequence by setting the task sequence execute condition to "Always". All tasks and sub-sequences under a task sequence with condition "Always" will be executed regardless of previous errors.

Given the following scenario:


Now imagine that the CreateEntryPage results in an error. With "Stop On Error" set to false, the execution will continue after the error and cause the UpdatePage and the StoreEntryPage to fail as well, since no entry was created.

If we want only the initial error to be reported as an error this can be accomplished by setting "Stop On Error" to true. The scenario execution will then halt right after the task 'create' has been executed. In this case the user never logged out and the server will be left with a user session that must time out by itself.

In order to both report only the initial error and perform logout, the "Stop On Error" flag should be set to true and the LogoutPage task sequence should be set to "Always". This will halt the execution after the task 'create', but will be forced to resume for the LogoutPage task sequence.

Task Sequence Condition

Task sequences can be executed conditionally by setting an execute condition. An execute condition can be set to one of the following:

Expressions are written using standard Java language syntax with the addition of PureLoad variables. They must evaluate to the primitive value true or false. PureLoad variables are automatically typed. Variables holding numerical values will be typed as primitive double values. Variables holding non-numerical values will be typed as strings (java.lang.String).

For example, let us say that the PureLoad variable ${foo} has a value of "minq" and that ${bar} has a value of 17. Now the following expressions will be evaluated as:

Expression
Value
${bar} == 17
true
${foo}.equals(���minq���)
true
${foo}.length() == ${bar}
false
${bar} > 0 && ${bar} < 42
true
${foo}.startsWith("m")
true
${foo}.indexOf("q")
Error (not a boolean result)

Variable Iterations

A task or a task sequence can have variable iterations instead of a static number. This is done by entering a scenario variable reference (for example ${foo}) in the Iterations field. The actual number of iterations to execute will then be evaluated during runtime when the scenario is executed. The variable must have an integer value or an error will occur. There are two special scenario variables set by the system which contains the current task and sequence iteration value. These variables are called ${_taskiteration} and ${_seqiteration}. They start at 0 (zero) for the first iteration and are then incremented for each iteration.

Note: When variable iterations is used, it is not possible to use any Parameter Generator value inside that task or task sequence since the number of values to generate would not be known.

Replace Task Parameters

The Replace Task Parameters dialog is a powerful way to replace task parameter values for individual tasks or group of tasks. The replace functionality can either search for matching task parameter values or completely replace values.

Select the desired nodes in the scenario tree and select Edit->Replace Task Parameters. All tasks that are included in the selected nodes will be subject for replacement in the replace task parameter dialog:

Replace Task
        Parameters

One or many task parameters (of the same type) can be selected for replacement. If "Text to find" is specified, only task parameters matching the specified text will be replaced. Select the "Replace All" if all values should be replaced regardless of what they are currently set to.

Example:
By selecting both URL task parameters, the replace will operate on all URLs in the scenario:

Replace Task
        Parameters Example
Let us say that there are two tasks (one HttpGetTask and one HttpPostTask) with URL set to http://www.somewhere.com. The above replacement would replace the 'some' with a 'no' so that the resulting URL becomes http://www.nowhere.com.

Replace Scenario Parameters

The Replace Scenario Parameters dialog can be used to change distribution parameters in many scenarios. It is also possible to make a percentage increase or decrease to distribution parameters.

Select the desired nodes in the scenario tree and select Edit->Replace Scenario Parameters. All tasks that are included in the selected nodes will be subject for replacement in the replace task parameter dialog:
Replace Scenario
        Paramaters

Example:
Making  a 10% increase in the scenarios / s for all selected scenarios with distribution Ramp Up can be done by selecting both y0 and y1:
Increase
        Distribution

Distribution of Scenarios

Note: Scenario Distribution and the Function parameter for scenario objects is only applicable when using PureLoad.

Scenario distribution is controlled by a distribution function. A distribution function defines how many times per second a scenario should be executed and when the execution will start and stop. Times are given as hours : minutes : seconds. There are a number of different functions to choose from so that the total load test traffic can best match the desired traffic model. It is also possible to execute without any distribution function and instead execute as fast as possible. The available functions are described in detail below.

Distribution Policy

The combined distribution functions of all scenarios in a load test make up the total distribution function. In order to execute the load test exactly as specified by the total distribution function, both the system under test and PureLoad itself must have enough resources and be able to perform. Large systems tend to have a certain inertia and will not respond to changes immediately. Load tests should usually be performed over a long period of time (at least 15 minutes up to many hours) for the distribution to take effect.

If the system under test fails to perform fast enough to handle the traffic, the load test results will start to differ from the distribution function. In this situation, PureLoad can act in one of two ways: either increase the number of scenarios per second to try to force the exact number of scenario iterations to be executed, or to keep on following the distribution function. Which policy to use can be set in Tool Properties -> Load Execution -> Distribution Policy.

Exact Iterations

When the distribution policy Exact Iterations is used, PureLoad will try to compensate and push harder if the system under test performs slower than the load test distribution function. The goal is to achieve the exact number of scenario iterations as defined by the distribution function. This means that the load test will not finish until all scenario iterations have been executed.

Example:
A scenario is set up to execute 10 per second for 20 minutes (1200 seconds with a total of 12000 scenario iterations), but the tested system can only execute 5 scenarios per second. In this case the load test would last 40 minutes until all the 12000 scenario iterations have been executed.

Follow Distribution

For distribution policy Follow Distribution, PureLoad will follow the distribution regardless of how the system under test performs. The number of executed scenario iterations will instead be less than defined by the distribution function. Note that Follow Distribution can not be used if any of the scenarios in the load test do not have a function (set to function None) and in that case Exact Iterations will be used instead.

Example:
Consider the same situation as the previous example with 10 scenario iterations per second and a tested system which can only handle 5 scenarios per second. This time, the result would be that the load test finishes after 20 minutes and the number of executed scenario iteraions would be 6000.

Function: None

When function is set to None, there is no function and the specified number of scenario iterations will be executed as fast as possible.
Distribution None
For scenarios without distribution (function None), it is possible to specify a weight to control the ratio between all scenarios without distribution and with the same logical name. A weight of 0 (zero) indicates that the scenario will be given an average weight. The weight is usually used in combination with iterations set to infinite. If a scenario with a limited number of iterations finishes, the remaining scenarios will have their weights re-calculated.

Ratio examples:

Note: If a certain ratio between scenarios is desired, it is recommended that weights are used in combination with infinite iterations and that all weights are set explicitly (avoid using 0) so that the resulting ratio can be easily understood.

Function: Burst

The scenario instances will be executed as fast as possible starting at Start Time.

burst

Function: Linear

The scenarios will be distributed in a linear way over time. For example starting with 2 scenarios per second and ending with 5 scenarios per second. Note that it is not possible to set the total number of scenarios executed by editing the Iterations parameter. The iterations parameter is calculated by the system using the defined parameters. The calculated value is showed as "view only" in the iterations field.

linear

The example above shows:

Function: Quadratic

The specified number of scenarios will be distributed quadratically starting at Start Time.

quadratic

Function: Normal

This distribution function is used to distribute the scenarios with a Normal function.

normal

The parameters to set for this function are:

Function: Ramp Up

Ramp Up is similar to Linear, but it allows the load to start from nothing and ramp up to a certain work level. The parameter Ramp Up controls the duration of the ramp up from zero to y0 iterations per second.

ramp up

Ramp up is useful in many real-life situations where the load must increase in a controlled manner.

Scenario Distribution Summary

If you are using distributions for several scenarios it is possible to view the summary graph of all the time distributions. This is done by selecting the Root object in the scenarios Tab. The example below models the load on a system where there is a base load and an event occurs which causes a peak in the load (this could be users arriving at the office after lunch and performing some task on a server).






Copyright © 2016 PureLoad Software Group AB. All rights reserved.