Testerman Reference Guide¶

Logical Architectures¶

Physical Architecture¶

Testerman implements these logical view as a distributed system, at different levels:

• the SA is made of distributable entities: agents. Agents are containers that can be deployed on any physical machine providing it has an access to an agent controller (and, not to be useless, to the SUT). This controller, dubbed the TACS (Testerman Agent Controller Server), is typically co-hosted with the Testerman Server (the main test system executor front-end), and exposes the connected agents to TACS clients:

[little diagram]

All TEs are clients to the TACS, enabling them an access to all the probes hosted on the agents. The Testerman Server itself is also a TACS client, for administration and control purposes.

On the front-end side, Testerman uses a client/server model, enabling multiple users to control and executes ATSes at the same time, sharing the same Testerman installation and repository.

[another little diagram]

Test Adapter Configuration¶

TTCN-3 does not provide a standard way to assign a test system interface port with an actual test adapter implementation, implementing on-the-wire sending and receiving operations.

Testerman introduces a way to bind these ports to test adapters, i.e. to probes, and calls this relationship a “binding”. When defining a (named) Test Adapter Configuration (for now, programmatically in your ATS only), you defines the different bindings (i.e. what probe instance will be used to implement a test system interface port), including the binding configuration, i.e. the probe parameters, if any.

You may define multiple test adapter configurations and switch from one to another one depending on the test environment you’re about to run your tests against.

The test adapter configuration is supposed to be the only thing that change from one test environment to another one. It usually defines:

• real SUT IP addresses, ports
• some low level variables such as ssh login/password (likely to change from one system to another one)
• etc

Application-oriented variables may still vary from one test environment to another one, but if your testcase is carefully designed, including with correct [#PreambleandPostamble Preambles], you should be able to minimize the amount of efforts needed to run your test on another instance of your SUT.

Codec Aliasing¶

The TTCN-3 code language does not completely ignore this, though.

Basic, General Purpose SUT Adapters¶

Once introduced to TTCN-3, you may wonder “OK, that’s nice and powerful, but now, how do I connect to a tcp server, send and receive some data ?”. Protocols in use are not a (direct) concern for the standard, you are right. We’re dealing with application- and test-oriented message structures, but physical transport (such as tcp/udp/sctp or even ip) is not adressed. TTCN-3 has the concept of SUT addresses, but how do we control (perform and verify) actual connections, disconnections, stream-oriented data reception, and so on ?

Antti Hyrkkanen, from the Tampere University of Technology, defended his master thesis about a general purpose SUT Adapter for TTCN-3, bringing socket-like structures and associated functions to TTCN-3. While this approach cannot be more flexible and complete, it renders ATSes harder to write for non-programmers, forced to take into account low level details in most cases (of course, if your tests are about testing the SUT’s ability to handle tcp connections, disconnect them when expected, etc, this is fully adapted and even required).

[WARNING: feel free to correct me if I misunderstood Antti’s work]

Testerman tries to find an intermediate solution to this problem by providing a collection of transport-related probes, interfaced in userland using the same kind of templates - quite similar to Antti’s solution, but just less generic as the very low level (socket parameters) are embedded within the probe, and partially controllable through test adapter configurations.

(TODO: transport interface: to document)

Preamble and Postamble¶

Testcases may require some SUT preparation in order to be executed, typically data provisioning, configuration files settings, maybe some processes or applications restarts.

Once the test is over (independently from its verdict), the SUT needs to be restored in an “original” state so that, in particular, we can replay the testcase without any additional manipulations.

These SUT preparation and clean up phases are called “Preamble” and “Postamble” (P&P), respectively, in Testerman terminology.

Testerman provides a way to use its core features to implement an automated preamble (you may call it “automated test bed setup”, “automated prerequisites set up”, ...) at least for what Testerman can automate using its available probes and the available SUT provisioning interfaces - manual prerequisites may still be needed.

You may design campaign-level P&P, suitable for multiple ATSes (i.e. starting the Preamble at the beginning of a campaign, starting the Postamble when finished), or ATS-level P&P, where a Preamble/postable may be used for multiple testcases in a row, or testcase-level P&P, i.e. only valid for a particular testcase (in this case, they are typically embedded within the testcase definition itself).

Testerman Applications¶

The Testerman application framework is currently not available.

The idea is to provide a way to run “in the background” applications built using Testerman features to act as simulators either to help manual testing or to simulate/prototype new applications.

Basically, you can already develop such simulators in a testcase, but a testcase is not designed to run forever and not to return a verdict. A Testerman application will.

Campaigns¶

A campaign is a structured collection of ATSes that can be executed conditionally.

It is basically a black and white tree (each node has two branches: one to follow if the current node is successful, the other one in case of an error) enabling to chain ATSes (or other campaigns), executing specific ATSes or campaign depending on the execution status of the current job.

job
 job
 job
  job
job

[<branch> ]<type> <path> [groups <groups>] [with <mapping>]

# Sample campaign
ats class5/call_forward_unconditional.ats # job1
 ats class5/call_forward_busy.ats         # job2
 ats class5/call_forward_no_answer.ats    # job3
ats clip/clip_base.ats                    # job4
ats clip/clir_base.ats                    # job5

ats class5/call_forward_unconditional.ats with PX_SUT_IP=192.168.1.1,PX_SOURCE_URI=sip:campaign@somewhere.com

ats class5/call_forward_unconditional.ats with PX_SOURCE_URI=sip:campaign@somewhere.com

ats class5/call_forward_unconditional.ats with PX_SUT_IP=${PX_SUT_IP},PX_SOURCE_URI=sip:campaign@somewhere.com

ats class5/sip_test.ats with PX_SUT_IP=${PX_SIP_SUT_IP}
ats class5/ldap_provisioning_test.ats with PX_SUT_IP=${PX_LDAP_SUT_IP}

ats another_test.ats with PX_PROBE_URI=probe:_@${PX_AGENT}

Job Control¶

Some clients, for instance QTesterman, provides a user interface to control the scheduled or running testerman jobs.

Jobs are controlled sending signals to them, through the Ws interface using the sendSignal(jobId, signal) API. The job reacts differently according to its state when receiving the signal.

Job Lifecycle¶

The following diagram exposes the basic job state machine, from its birth to its multiple death possibilities:

Some state explanations:

• initializing: the job is being prepared and dependencies scanned: campaigns are parsed and missing ATSes or children are reported, leading to an error; imported modules in ATSes are checked.
• waiting: the job is now in the server’s queue, waiting for its start according to its scheduled start time.
• running: the job is now running, either executing testcases for ATSes, or ATSes for campaigns
• paused: the job has been paused. Only meaningful for an ATS job. Running timers, if any, are not frozen during the pause. As a consequence, when resuming the job, several timers may expire immediately.
• complete: the job completes its execution “successfully”, i.e. no technical errors (TTCN3-, Testerman- or Python-related errors/exceptions) occurred, and the job return code is 0 (you may alter it with the stop(retcode) statement in the control part). However, it does not mean that all testcases were OK.
• cancelling: the job is being cancelled, i.e. it waits for the pending testcase to finish, then stops.
• cancelled: the job has been cancelled, i.e. probably did not complete all its testcases (unless the cancel signal arrived during the last testcase execution). The associated log file is still valid and consistent to analyze testcases till the cancellation.
• killing: the job is being killed, i.e. stopped without waiting for a possible pending testcase to finish. This state typically lasts less than one second. Anyway, you can’t do anything more to kill the job now.
• killed: the job has been killed. The associated log file may be inconsistent, especially regarding the running testcase when killed.
• error: a problem occurred either while preparing the job (campaign parsing error, temporary files creation problems, TE syntax error, ...) or a technical error occurred preventing the ATS continuation, typically a Python exception in the control part (incorrect testcase identifier, ...). In the first case, you should have a look to the server’s logs to know what was wrong; in the second case, take a look at the log file in raw mode: the exception is probably logged. Additionally, the job’s return code could help you diagnose the problem:

Raw Documentation¶

Basically, if you need to document a test case named TC_MY_TESTCASE:

class TC_MY_TESTCASE(TestCase):
  """
  This is the test case documentation.

  This test verifies that an SNMP trap
  is actually sent when rebooting the server.
  We first listen for a trap,
  then we reboot a machine using ssh reboot.
  We then should get our trap within 30 seconds.

  Created by John Smith on 2009-05-05.
  """
  def body(self, ...):
    """
    A documentation can also apply here
    """
    ...

Documentation Tags¶

It is usually convenient to structure your test case documentation into something more formal. A test case specification, for instance, may contain a purpose, the description of the steps to perform, some prerequisites, an author, a creation date, and so on.

The documentation may be left as is, using a plain English text, or turned into something more formal and still human-readable using tags. Tags are special markers in the plain text that indicates the different parts of the documentation. If you are familiar with code documentation system such as Epydoc, Javadoc, Doxygen, there is nothing new here – the syntax is (almost) the same:

class TC_MY_TESTCASE(TestCase):
  """
  This is the test case documentation.

  @purpose: This test verifies that an SNMP trap
  is actually sent when rebooting the server.
  @steps:
  We first listen for a trap,
  then we reboot a machine using ssh reboot.

  We then should get our trap within 30 seconds.

  @author: John Smith
  @date: 2009-05-05
  """
  def body(self, ...):
    """
    A documentation can also apply here
    """
    ...

@purposes, @steps, @author, @date are used to tag different parts of the raw documentation. In this sample, however, we left what could be a test case overview untagged, as the “natural”, basic test case documentation.

Documentation plugins, for instance, can then access tag values directly to create some more formal test specification (or any other kind of documentation).

Any tags can be created at any time, anywhere in docstrings. However, they are mainly interpreted by plugins and you should match their expectations, according to the documentation strategy defined by your Testerman administrator and test managers. Testerman only provides a framework, regardless of the way you plan to use it.

Tag Format¶

A tag is defined as an identifier ([a-zA-Z0-9_-]+) following a @ character, starting a new line. Its value starts just after the following : character.

Tags are case-insensitive: for instance, @purpose and @Purpose define the same tag purpose.

Tag Value Format¶

A tag value can be written over multiple lines. Actually, the current tags value only stops when another tag is started or when the docstring ends:

@purpose: This test verifies that an SNMP trap
is actually sent when rebooting the server.

Resulting in a tag value:

This test verifies that an SNMP trap is actually sent when rebooting the server.

Notice that leading and trailing spaces are stripped, so that this is equivalent to:

@purpose:
This test verifies that an SNMP trap
is actually sent when rebooting the server.

If you need to create a value that contains carriage returns, you must either leave an empty line or starts a new line with at least one blank character (space(s), tab(s)):

@steps:
We first listen for a trap,
then we reboot a machine using ssh reboot.
We then should get our trap within 30 seconds.

Resulting in the following tag value:

We first listen for a trap, then we reboot a machine using ssh reboot. We then should get our trap within 30 seconds.

Which is not really readable. So let’s use:

@steps:
We first listen for a trap,
then we reboot a machine using ssh reboot.

We then should get our trap within 30 seconds.

Or

@steps:
We first listen for a trap, then we reboot a machine using ssh reboot.
We then should get our trap within 30 seconds.

Leading to:

We first listen for a trap, then we reboot a machine using ssh reboot.
We then should get our trap within 30 seconds.