Simulated Workloads

Overview - Using SimLoads

Registration and Download

The current version of simloads is freeware. On installation you must accept the SimLoads Freeware License Agreement.

If you want to download the simloads installation packages, the mandatory prerequisite is your registration. To register, please send an e-mail including your full contact information to .

On approval of your registration, you will receive an e-mail with a personalized download link.

Introduction

A simload (= simulated workload) is a compiled executable that can be registered and launched on a guest VM or a physical Windows workstation. Such a simload simulates (synthetic) user activities representing typical interactive computer work patterns. A simload launches one or multiple applications, starts logging timestamps, simulates user interactions by controlling the applications, and after a predetermined time terminates all applications under control and stops logging.

A singular simload type 1 ("Workload") is an individual test scenario for the user in focus. Each type 1 simload scenario highlights a specific graphic or multimedia format. Most type 1 simloads require a pre-installed application. Some applications open data files as part of the test scenario. Each type 1 simload can either be launched in the foreground (using the Run parameter) or in the background of a user session (using the Start/Stop parameters).

A blended simload type 2 ("Persona") is a sequence of chained or overlayed user activities, orchestrated in such a way they generate the characteristic behavior and load pattern of a predefined interactive user type or persona. Examples for such user types are Author, Knowledge Worker, Power User or Media User. Most type 2 simloads launch and control multiple applications simultaneously, with a chain of subsequent foreground applications and multiple background applications running in parallel.

A score simload type 3 runs test applications and measures predefined system metrics used to produces a number (= score) that represents the performance. Typically, each score simload is associated with a specific theme, such as graphics output performance or application load time. The individual scores collected for each theme can be used to calculate an overall score.

A runbook runs a customizable sequence of simloads over a configurable time period. Running simload runbooks requires the SL-Runbook command-line tool. As an alternative, it is also possible to use a scripting language like PowerShell or Windows CMD for chaining type 1 simload command-line interface.

Installation

Installation steps:

1. Run Install-Simloads-Base.exe with administrative privileges
2. Run Install-<SimloadName>.exe with administrative privileges

The first simload deployment step is installing Install-Simloads-Base.exe on each target system. This base package includes standard data files and supporting tools. The installation wizard includes a text input field for selecting the local path for simload files and folders. The default path is C:\SimLoads. The base package installer requires administrative privileges as it creates all necessary system-wide simload hives and values in the registry. It also creates the Windows menu folder "Simloads Base Tools" which includes icons for all simload tools, such as Simload Runner, TelemetrySplash or WinInfo.

NOTE: As an alternative to the simload base installation package, there is a simload core installation package available. On installation, the core package configures all necessary simload settings in HKLM\Software\Simloads and installs some simload tools. But the package does not include the data files which makes it very lightweight. If the core package is installed, then simloads trying to open data files from the default simload data folder will fail. This is why the simload core package can only be used in combination with "autonomous" simloads that include all required data files.

After the simload base package was installed, the next step is the installation of simload-specific packages, for example Install-SL1-Chrome.exe or Install-SL2-Author.exe. These installers check if the registry value HKLM\Software\Simloads\Folders\Simloads exists and abort if it does not. If the value exists, it holds the target folder where simload-specific files and folders are copied to during the installation. In addition, the installer registers each new simload that is included in the installation package. After a successful registration, one or multiple icons representing the new simloads are located in a Windows menu folder named "X SimLoads" where "X" represents the simload type (1 = single app, 2 = persona, 3 = score).

NOTE: A simload executable can also be launched when it is not registered. In this case, it tries to find the necessary configuration settings in the registry under HKCU\Software\Simloads. If this fails, it reads the necessary values from the side-by-side .ini file that must have the same name as the simload .exe file. This allows a copy-and-paste deployment into a user session in cases where administrative privileges for a system-wide installation are not available. Check side-by-side .ini file details in the Internals chapter below.

Launching SimLoads

Launching a simload in default mode is done by a click on the simload icon located in a Windows menu folder named "X SimLoads" where "X" represents the simload type (1 = single app, 2 = persona, 3 = score). For more sophisticated test scenarios, there is also a simload command-line interface. Running a simload with the parameter "help" from the command line always shows all available command-line parameters.

The type 1 simload command-line syntax is as follows:

• SL1-Simload.exe run <seconds> [<x> <y> [<width> <height>]]
• SL1-Simload.exe start <tempobject> [<x> <y> [<width> <height>]]
• SL1-Simload.exe stop <tempobject>
• SL1-Simload.exe info
• SL1-Simload.exe help
• SL1-Simload.exe register
• SL1-Simload.exe unregister

The parameter run launches a type 1 simload in run mode for a predefined number of seconds and at a configurable screen position (window origin coordinates and window size). A simload in run mode may include user activities. There can only be one type 1 simload in run mode up and running at any given time. If width and height are set to 0 (zero), most type 1 simloads run in full-screen mode.

The parameter start launches the selected type 1 simload in start/stop mode. This mode requires a unique tempobject parameter that is used to control the application and to terminate it with the stop parameter. Multiple simloads in start/stop mode can be running simultaneously. If width and height are set to 0 (zero), most simloads start in full-screen mode. The default behavior of a simload in start/stop mode is passive, which means that there is no user interaction.

The parameter register adds the simload name and the full executable path of the simload as a REG_SZ value to the registry key HKLM\Software\Simloads\Simloads. The parameter unregister removes the value from the registry.

The parameter info shows all relevant object properties as loaded from the registry or the side-by-side .ini file. The parameter help shows all command-line parameters.

Examples:

SL1-NotepadEdit run

Runs the SL1-NotepadEdit simload in fullscreen mode for the predefined number of seconds (typically 45 or 60 seconds)

SL1-NotepadEdit run 45 100 100 1200 800

Runs the SL1-NotepadEdit simload at x/y position 100/100 and window size 1200/800 for 45 seconds

SL1-NotepadEdit start MyNotepad 100 100 800 600

Creates a temporary object named MyNotepad and starts the SL1-NotepadEdit simload at x/y position 100/100 and window size 800/600

SL1-NotepadEdit stop MyNotepad

Stops the SL1-NotepadEdit simload and deletes the temporary object named MyNotepad

 

The type 2 and type 3 simload command-line syntax is as follows:

• SL2-Simload.exe run <seconds> [<x> <y> [<width> <height>]]
• SL2-Simload.exe help
• SL2-Simload.exe register
• SL2-Simload.exe unregister

Setting window origin coordinates [x, y] and window size values [width, height] is optional. If there is no value set or the values are [0, 0, 0, 0], the simload uses default positions for the foreground application windows. Setting x, y, width and height to values larger than 0 influences the position and size of the foreground application windows.

SimLoad Tools and Runbooks

Tools

Running simloads can be initiated through Simloads Start Menu icons. Additional mechanisms to launch simloads and collect telemetry data are provided by installing SimLoad Tools. Simply run the SimLoads Base Tools installation package Install-Simloads-Base.exe to deploy the command-line and graphical tools.

Simload Runner

SimloadRunner is a GUI tool that allows to launch registered simloads, modify performance counters used for collecting telemetry data, and read collected telemetry data files. The "Show" button in the SimloadRunner Perf Counter panel can launch TelemetrySplash to show a preview of the selected performance counters.

Collecting telemetry data with SimloadRunner is optional. When other mechanisms are used to collect performance counters, leave the built-in "Collect telemetry data" in disabled status before running a simload.

The last position of the SimloadRunner window is stored in the registry under HKCU\Software\Simloads\Runner.

Telemetry Splash

TelemetrySplash reads its configuration settings from Telemetry.ini and can be used to see the performance counter data in real-time. In most cases, you will be using TelemetrySplash to find out if you are collecting the right performance counter data prior to running simloads.

NOTE: Modify Telemetry.ini according to your test requirements. You may need to open Performance Monitor to find out the exact name of a performance counter. Using wildcards "*" is not supported.

WinInfo

WinInfo shows process ID, window handle, process name, file name, window title and other details of each application currently running in the user session. The underlying methods to retrieve this data are identical to the methods used in simloads. The displayed information can be very useful when modifying a side-by-side .ini file.

Register and Unregister Simload

Two command-line tools that require administrative privileges and allow to register or unregister simloads.

Runbooks

A runbook is a customizable sequence of chained simulated workloads (simloads) and applications used to generate a load pattern. Simloads and applications included in a Runbook can run in fixed or random order. Runbooks use .rbk files for customization. Foreground simloads and applications are launched one after another. Background simloads and applications are launched in such a way that they are running simultaneously in the background the entire time while sequences of foreground simloads and applications are launched and terminated.

NOTE: The necessary components required to launch runbooks are included in a separate installation package.

To launch the default runbook, type "SL-Runbook" and hit the Enter key. Configuration is defined in SL-Runbook.ini. To start a custom runbook, add the runbook configuration file as a parameter when launching SL-Runbook.exe.
Example: SL-Runbook.exe test.rbk

Runbook configuration file syntax

[Settings]
TotalTimeInMinutes=	Time in minutes for the runbook. If set to 0, the runbook will terminate after all foreground simloads and apps were launched once.
Random=	If set to 0, foreground simloads and apps will run as configured. If set to 1, foreground simloads and apps will run in random order.
ForegroundRuntimeInSeconds=	The time in seconds each foreground simload and app runs.
SeparatorInSeconds=	Time in seconds between two foreground simloads and apps.
[ForegroundSimloads]
1=Simload1 2=Simload2 ...	List of foreground simloads, referenced by their registered names, each specified by an individual [Simload] section. Foreground simloads and apps run in a sequence.
[BackgroundSimloads]
1=Simload1 2=Simload2 ...	List of background simloads, referenced by their registered names, each specified by an individual [Simload] section. Background simloads run simultaneously.
[ForegroundApps]
1=MyCustomApp1 2=MyCustomApp2 ...	List of foreground apps, referenced by their full installation paths, each specified by an individual [App] section. Foreground simloads and apps run in a sequence.
[BackgroundApps]
1=MyCustomApp1 2=MyCustomApp2 ...	List of background apps, referenced by their full installation paths, each specified by an individual [App] section. Background simloads run simultaneously.
[Simload1]	Configuration details of a simload
Type=Simload x=0 y=0 width=0 height=0	Type must be Simload Window xposition Window y position Window width, 0 is full screen Window height, 0 is full screen
[MyCustomApp1]	Configuration details of an application
Type=App AppName=MyCustomApp.exe AppPath=c:\temp\ AppPathType=2 DataFile=file.txt DataPath=c:\temp\ DataPathType=2 WinTitle=CustomApp AppControlType=byHandle x=400 y=200 width=1200 height=800	Type must be App Application executable name including extension Full application path None [0], Relative [1], Absolute [2], Simload [3], Internet [4], Profile [5],Public [6] Data file including extension Full data path None [0], Relative [1], Absolute [2], Simload [3], Internet [4], Profile [5], Public [6] Window title of the application with the open data file byPid, byTitle or byHandle Window x position Window y position Window width, 0 is full screen Window height, 0 is full screen

SimLoad Internals

Side-by-Side .ini Files

Each SimLoad comes with a side-by-side .ini file with the same name as the SimLoad executable. Below is the example of the SL1-NotepadEdit.ini file:

[SL1-NotepadEdit]
;sAppName=Notepad.exe
;sAppPath=c:\Windows\System32\
;fAppPathType=2
;sAppParam=
;bEventFile=1
;sDataFile=lordoftherings.txt
;sDataPath=c:\Simloads\Data\Documents\
;fDataPathType=2
;bDataRwAccessRequired=
;sWinTitle=
;sAppControlType=byHandle
;iX=0
;iY=0
;iWidth=0
;iHeight=0
;iTime=45

[Settings]
bLogVerbose=0
bDebug=0
iCountdownScreen=3

[Folders]
Data=C:\Simloads\Data\
Results=C:\Simloads\Results\
Simloads=C:\Simloads\Simloads\
Tools=C:\Simloads\Tools\

[Simloads]
SL1-NotepadEdit=C:\Simloads\Simloads\SL1-Notepad\SL1-NotepadEdit.exe

 

Please note that adding a semi-colon at the beginning of a line is interpreted as a full-line comment. This means that the leading semi-colon must be removed to activate the setting defined in a line. It's a best practice to show the hard-coded simload default settings in side-by-side .ini file comment lines.

Most settings in the side-by-side .ini file are self-explanatory, except for fAppPathType, fDataPathType, bDataRwAccessRequired and sAppControlType. Here are the short descriptions:

fAppPathType and fDataPathType are flags that define the type of the path and can have the values 0, 1, 2, 3, 4, 5 or 6.

0 (= None)

There is no application or data file

1 (= Relative)

A relative path that starts from the default data folder stored in HKLM\Software\Simloads\Folders\Data, HKCU\Software\Simloads\Folders\Data or in the side-by-side .ini file under [Folders] Data=...

2 (= Absolute)

An absolute path in the local file system

3 (= Simload Folder)

A simload-specific relative path that starts from the location of the simload .executable file that is stored in HKLM\Software\Simloads\Simloads\<SimloadName>, HKCU\Software\Simloads\Simloads\<SimloadName> or in the side-by-side .ini file under [Simloads] <SimloadName>=...

4 (= Internet)

The path is a URL located in the internet

5 (= Profile)

A relative path that starts in the MyDocuments folder of a user profile (C:\Users\<UserName>\Documents\)

6 (= Public)

A relative path that starts in the Public Documents folder (C:\Users\Public\Documents\)

When setting bDataRwAccessRequired to 1, the data file is copied to the user's MyDocuments\temp folder prior to opening it and the fDataPathType flag is changed accordingly. This gives the synthetic user full read/write access to the data file and allows individual editing.

The sAppControlType value can be set to byPID, byHandle or byTitle. This value defines how an application is controlled within the simload, for example for user interaction or application termination. In most cases, byHandle is the best choice. Setting this value to byTitle requires a valid sWinTitle value and is used when the process ID and window handle values returned at application start is not working properly. This can happen when an application creates child windows on opening documents or other data files.

Registration, Common Folders and Settings

It is recommended that each simload gets registered in the Windows registry. The registry key HKLM\Software\Simloads\Simloads contains a REG_SZ value with the full executable path for each simload. This registration mechanism allows any script or tool to launch a simload from the correct folder.

The registry key HKLM\Software\Simloads\Folders includes a REG_SZ value for each common folder. For details, see the table below.

Each simload writes into a log file named Simload_ComputerName_UserName.log, located in the Results folder (typically C:\SimLoads\Results). With default settings, only error messages are sent to the log file. By setting the bLogVerbose value in HKLM\Software\Simloads\Settings to 1, also warning and info messages go to the log file. By setting the bDebug value in HKLM\Software\Simloads\Settings to 1, also debug messages go to the log file.

NOTE: Verbose and debug mode may influence the runtime behavior of a simload. Do not use any of these modes during real test runs.

HKLM\Software\SimLoads (or HKCU\Software\SimLoads as a fallback location for test setups without administrative privileges)

Hive	Value Name	Description
Folder
	Data [SZ]	Root path to common data files (documents, HTML files, images and videos)
	Results [SZ]	Target folder that includes all results
	SimLoads [SZ]	Folder that contains the subfolders with each simload executable and .ini file
	Tools [SZ]	Folder that contains the command-line and GUI tools
Simloads
	SL0-... [SZ]	Registered Type 0 Simload = System / Test / Prototype
	SL1-... [SZ]	Registered Type 1 Simload = Single application
	SL2-... [SZ]	Registered Type 2 Simload = Multiple Apps / Persona
	SL3-... [SZ]	Registered Type 3 Simload = Score
Settings
	bLogVerbose [DWORD]	If set to 1, INFO and WARNING messages are added the log file located in the Results folder
	bDebug [DWORD]	If set to 1, DEBUG messages are added to the log file located in the Results folder
	iCountdownScreen [DWORD]	Sets the initial number of seconds in the countdown screen

 

Temporary Object Structure

Each application launched and controlled by a simload is represented by a temporary object in the user session. All temporary object properties are stored in the registry under HKCU\Software\Simloads\Temp\<ObjectName>.

TEMP CLASS STRUCTURE: The name of the object is either identical to sSimloadName or to an individual name in case of Start/Stop.

0	sSimloadName	Name of the simload stored in the temporary object
1	sStartDate	Start date, format YYYY/MM/DD
2	sStartTime	Start time, format HH:MM:SS
3	sAppName*	Application executable name, including extension
4	sAppPath*	Application path
5	fAppPathType*	Application path type: None [0], Relative [1], Absolute [2], Simload Folder [3], Internet [4], Profile [5], Public [6]
6	sAppParam*	Optional application parameters
7	sEventFilePath	Event file w/ timestamps and descriptions, consumable by SxS Player
8	bEventFile*	[1] creates an event file, [0] does not
9	sDataFile*	Data file name, including extension
10	sDataPath*	Data file path
11	fDataPathType*	Data file path type: None [0], Relative [1], Absolute [2], Simload Folder [3], Internet [4], Profile [5], Public [6]
12	bDataRwAccessRequired*	[1] copy data file to MyDocuments\temp, change fAppPathType to 2, change sDataPath
13	sWinTitle*	Title of the application window
14	sWinHandle	Handle of the application window
15	sProcessID	Application process identifier
16	sAppControlType*	byTitle, byHandle, byPID
17	iX*	X position of the application window
18	iY*	Y position of the application window
19	iWidth*	Width of the application window (0 is full screen width)
20	iHeight*	Height of the application window (0 is full screen height)
21	iTime*	Simload runtime in seconds
22	iAppLaunchTime	Application launch time in milliseconds

All structure items with an * can be defined in a simload-specific .ini file