Generate Testerman test cases with Selenium IDE ----------------------------------------------- Summary ~~~~~~~ `Selenium IDE `__ is a Firefox plugin which records user interaction with a web site. The plugin can then export the list (of user actions) to different formats. The plugin enabling Selenium IDE to export is called a formatter. The Selenium team provides bindings for several `programming languages `__. You can write a test in one of these languages and use the provided libraries to remote control a browser. Selenium IDE formatters do exactly that: convert the recorded list into source code for a specific language. The Testerman ATS Formatter let's you export your recorded interactions directly to a Testerman test script (\*.ats). The generated code will use the :doc:`autogen/ProbeSelenium` or the :doc:`autogen/ProbeSeleniumWebdriver` to control the browser. This document covers only Selenium RC but the usage of the Selenium Webdriver probe is largely equivalent. Install ~~~~~~~ The formatter is a Firefox plugin, you have to install it on top of the Selenium IDE plugin. You may find a build script in plugin/probes/seleniumrc/selenium-testerman-formatter. Run the script and install the install via Firefox: Select File -> Open File ... and select the \*.xpi file. Installing a newer versions of the formatter ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Before installing a newer version, make sure to reset the options (Selenium IDE -> Options -> Options -> Formats -> Testerman ATS Formatter -> Reset Options) in order to follow the latest changes. Please be carefully though, as this will erase all your personal configurations. How To, Best Practises and Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Please read the following instructions carefully (yes, it is long \*sigh\*). There are a lot of hidden tricks in SeleniumIDE/Testerman one has to know to efficiently generate ats files. You should be familiar with writing Testerman ats files manually (and thus you are familiar with the send/receive concept, alt statements, verdicts, ...) and know Selenium IDE a little bit (record test cases, exporting), too. You will find a lot of documentation for Selenium IDE `here `__. Understanding Selenium ^^^^^^^^^^^^^^^^^^^^^^ Selenium is a suite of different tools. We will use Selenium IDE (recording user interactions with a web interface) and Selenium RC (remote executing of "simulated" user actions on a web interface, acutally, this is done by the :doc:`autogen/ProbeSelenium`/:doc:`autogen/ProbeSeleniumWebdriver`). Selenium RC has a set of different type of commands. There are commands like click(myButton), type(myInput, blablub), getText(myElement), isEditable(myInput). Selenium IDE uses these commands but in a slightly changed way. Some commands are the same (click, open, type), but some others are not available in Selenium IDE, at least not in their orinigal form. Selenium IDE uses generated commands from the Selenium RC accessors (getText, isEditable, ... every command where you would expect a result). Instead of getXXX, one can use assertXXX, verifyXXX, storeXXX, and/or waitForXXX. These generated commandes are ultimately translated into their source command (= the command Selenium RC expects). It will note change anything for Selenium RC whether you use assertText(id=myElement, 42) or verifyText(id=myElement, 42); the actual command in both cases will be getText(id=myElement). To see which Selenium RC command will be produced, have a look at the reference tab in the lower bar of the Selenium IDE gui. When adding a command, you will see it's argument and it's base command appear. It is up to the formatter (e.g. the source code generator) to handle the different command types while still sending the base command to Selenium RC. Selenium IDE is of course not able to "record" these generated commands. It knows where you clicked, but it doesn't know your expectations concering returned results. Selenium commands are often refered to as "selenese". Using Selenium IDE generated commands ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ As an example consider you want to fill in a form and expect a certain text after clicking on the submit button. During recording, you would type your text into the form and click the button. Selenium will list something like this: :: commands target value type id=input_pw 1234 click id=login Now, to check whether your expected text appears on the following page, right-click on the element where the text should appear (or select the text, ...). The context menu will propose you a range of commands to check the elements content. After selecting one command, Selenium IDE will list something like this: :: verifyText id=header Login successful The command will become "getText(id=header)" for Selenium RC. If you had used assertText() instead, the outcoming Selenium RC commands would have been the same. The difference lies in the handling of the returned result (see below for a differentiation of assert and verify). The target argument is often a "locator", i.e. the element on the web interface you address. To get an element's locator, you could check the HTML source (Firebug!) for id or name attributes. However, it is a probably a good idea to let Selenium IDE choose the locator: "Record" a dummy click (or something) action on that element to get it's locator listed and delete the dummy action then. See http://release.seleniumhq.org/selenium-core/1.0.1/reference.html for more details Using Testerman's parameter system ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The Testerman ats formatter supports Testerman's parameter system (global variables, often called PX\_PASSWORD or PX\_HOST\_IP). There is a support for "static" and "dynamic" parameters. We call theme "static", when they are defined in the meta data block at the beginning of the generated code regardless of the content. You can setup these static parameters in the formatter's option (Selenium IDE -> Options -> Options -> Formats). Define here the parameter name, default value and type ("string" or "integer"). Every parameter listed here will always be dumped. Remember that the parameter's value can be overwritten by the campaign configuration (when launching a campaing in Testerman). The Format is "name:value:type\\n". The name has to match **/!^PX\_[\_A-Z0-9]+$/**. Examples: :: PX_DB_SERVER:string:127.0.0.1 PX_DB_PASSWORD:string:1234 PX_MAX_CONNECT:integer:10 On the other hand, "dynamic" parameters are set up within the actual selenese. After recording your actions with Selenium IDE and before formatting (exporting), you could change some command arguments in order to make the test more flexible. By replacing a "real" value with it's parameter name (and it's default value), the test can be adapted to another test environment. To define a dynamic parameter, replace the original value by ${PX\_PARA\_NAME:orginalValue}. Example: :: commands target value type id=input_pw 1234 The above command generates: .. code-block:: python sel.send(["type", "id=input_pw", "1234"]) Now consider the following :: commands target value type id=input_pw ${PX_PASSWORD:1234} ** define parameter name and default value before export ** generates: .. code-block:: python # ... sel.send(["type", "id=input_pw", str(PX_PASSWORD)]) As you can see, a new parameter is defined and during the selenium command, this parameter is used. The parameter definition is to be found in the meta data block, the value can be overwritten by external campaing config as mentioned before. Of course, you do/can not redefine the same parameter several times. Once, your PX\_XXX is defined you can use in later commands: old: :: commands target value type id=input_pw 1234 assertValue id=blub 1234 generates: .. code-block:: python sel.send(["type", "id=input_pw", "1234"]) sel.send(["getValue", "id=blub"]) alt([ [ sel.RECEIVE(template = "1234"), ... new: :: commands target value type id=input_pw ${PX_PASSWORD:1234} assertValue id=blub ${PX_PASSWORD} generates: .. code-block:: python # ... sel.send(["type", "id=input_pw", str(PX_PASSWORD)]) sel.send(["getValue", "id=blub"]) alt([ [ sel.RECEIVE(template = str(PX_PASSWORD)), ... The parameter can easily be referred to. This enables you to use static parameters as well. You do not have to define a default value because they are defined anyway. Remember that you have to adapt the selenese \*before\* exporting. Their is one disadvantage: Once you changed your values to 'PX\_SOMETHING' you won't be able to replay your test case with Selenium IDE any more. The rule of thumb is: ``${PX_XXX}`` --> replace this by a variable called ``PX_XXX`` ``${PX_XXX:1234}`` --> replace this by a variable called ``PX_XXX``, which has been set to 1234 in the meta data block ``PX_XXX`` --> take this literally (do not replace anything, used for ``storeXXX()``, see later) The replace mechanism generally applies to the last argument of a command (often called "pattern" in Selenium IDE). The px parameters can be used in conjunction with regular expressions. The value field is ``regexp:${PX_VAR_NAME:the_actual_expression}``. Fur further details see the section for regular expressions. :: commands target value assertValue id=blub regexp:${PX_PASSWORD:[a-z]?} Store Commands ^^^^^^^^^^^^^^ Seleniums provides the possiblity to store values in variables. You will find a lot of storeXXX() fonctions (storeText(), storeAlert(), storeChecked(), ...). Whenever you use a store command, the Testerman Formatter will print a send and a receive command (to the port). When using storeXXX(), the second argument of the command is the name of the variable to which the value will be assigned. There are two possibilities to store values. Either you use a "normal" variable name or you use a name like PX\_XXX (matching the abouve regexp). :: commands target value storeText link=home myvari storeText link=home PX_MYVAR The generated code will be as follows: .. code-block:: python # store (selinium): myvari = getText(link=home) sel.send(["getText", "link=home"]) sel.receive(value = 'myvari') myvari = value('myvari') log('myvari = %s' % myvari) # store (selinium): PX_MYVAR = getText(link=home) sel.send(["getText", "link=home"]) sel.receive(value = 'PX_MYVAR') PX_MYVAR = value('PX_MYVAR')) log('PX_MYVAR = %s' % str(PX_MYVAR)) To refer to the stored variables, use the **${nameOfTheVariable}** syntax as mentioned above. If you used a PX\_XXX variable you could even store a value and define a default value in the metadata block: :: commands target value storeText link=home ${PX_MYVAR:42} The generated code will be as follows: .. code-block:: python # ... # store (selinium): PX_MYVAR = getText(link=home) sel.send(["getText", "link=home"]) sel.receive(value = 'PX_MYVAR') PX_MYVAR = value('PX_MYVAR')) log('PX_MYVAR = %s' % str(PX_MYVAR)) This would enable you to use PX\_MYVAR (with it's default value) even before storing it. Of course, it's pretty hard to find a use case ;) Let's take another example. Imagine you want to change a value on the web interface, check that the settings are saved correctly and then reset the original value. The naive command list would be: :: commands target value type name=port 12345 clickAndWait id=submit_button assertValue name=port 12345 type name=port 1234 (supposing the original value was 1234) clickAndWait id=submit_button While this is pretty forward (and easily recordable), it is not very flexible. When you are not sure of the system's state before test execution, it will be hard to reset the "right" default value (in this case, the original value was 9999 or so). Use storeXXX() to copy and paste. After recording your test with Selenium IDE and before exporting to an ats file, change the following: :: commands target value storeValue name=port PX_DEFAULT_PORT type name=port ${PX_NEW_PORT:12345} clickAndWait id=submit_button assertValue name=port ${PX_NEW_PORT} type name=port ${PX_DEFAULT_PORT} clickAndWait id=submit_button Just save the value before changing it, then, during "cleanup" reuse the given name. The above example will produce the following code: .. code-block:: python # ... # store (selinium): PX_DEFAULT_PORT = getValue(name=port) sel.send(["getValue", "name=port"]) sel.receive(value = 'PX_DEFAULT_PORT') PX_DEFAULT_PORT = value('PX_DEFAULT_PORT')) log('PX_DEFAULT_PORT = %s' % str(PX_DEFAULT_PORT)) #change to new port sel.send(["type", "name=port", str(PX_NEW_PORT)]) sel.send(["click", "id=submit_button"]) sel.send(["waitForPageToLoad", "30000"]) #verify changings sel.send(["getValue", "name=port"]) alt([ [ sel.RECEIVE(template = str(PX_NEW_PORT)), lambda: self.setverdict(PASS), lambda: log('getValue(name=port) == ' + str(PX_NEW_PORT) + ' [using PX_NEW_PORT] -> Good!'), ], [ sel.RECEIVE(template = any_or_none()), lambda: self.setverdict(FAIL), lambda: log('getValue(name=port) != ' + str(PX_NEW_PORT) + ' [using PX_NEW_PORT] -> Bad!'), lambda: stop(), ], ]) #clean up sel.send(["type", "name=port", str(PX_DEFAULT_PORT)]) sel.send(["click", "id=submit_button"]) sel.send(["waitForPageToLoad", "30000"]) Regular Expression ^^^^^^^^^^^^^^^^^^ Sometime you need to check a certain value, but you know only the expected pattern. Selenium IDE has build-in support for regular expression (in fact, it says [in our case] to the formatter, that the user wants to use a regexp). The following example should answer all questions: :: commands target value assertValue name=search blub assertValue name=search regexp:^[A-Z]?$ assertValue name=search regexp:${PX_MYPATTERN} assertTextPresent regexp:[a-z].? generates: .. code-block:: python sel.send(["getValue", "name=search"]) alt([ [ sel.RECEIVE(template = "blub"), ... sel.send(["getValue", "name=search"]) alt([ [ sel.RECEIVE(template = pattern(r"^[A-Z]?$")), ... sel.send(["getValue", "name=search"]) alt([ [ sel.RECEIVE(template = pattern(str(PX_MYPATTERN))), ... sel.send(["isTextPresent", "regexp:[a-z].?"]) alt([ [ sel.RECEIVE(template = True), ... You can combine px parameter definitions with regexp: :: commands target value assertValue name=search regexp:${PX_REGEXP:[a-e]?} ** define new px parameter (with default value) and it use it as regexp ** Regular expression are a bit tricky though. In fact, Selenium RC supports regexp by itself and Testerman does so (vie Python's re.search()), too. The "problem" is that sometimes Selenium RC has to use it to determine the result of a command and sometimes Testerman can just wait for the result and then check it against the regular expression. Whenever you use a selenese generated from getXXX() (like assertText(), verifyConfirmation(), ...) the Testerman's (=Python's) regular expressions will be used. You can see them in the output (Testerman's function pattern()). On the other hand, commands like assertTextPresent() (e.g. commands where you expect a boolean response) will use Selenium RC's regexp. When dumping these commands, you will the that the literal "regexp:" is still present because it will be send to Selenium RC. There doesn't seem to be a way to unify the two regular expression systems, although it probably won't be a problem. Note: You can not use a variable stored previously via storeXXX() in an regexp if the variable is **not** a PX\_XXX parameter. By the way, Selenium IDE doesn't implement variable translation (with ${variableName}) in regular expressions at all. It's the Testerman Formatter which adds this feature only for PX\_XXX parameters. See also: http://release.seleniumhq.org/selenium-core/1.0.1/reference.html#patterns Assert vs. Verify ^^^^^^^^^^^^^^^^^ assertXXX() and verifyXXX() produce nearly the same code. In both cases, an alt statement checks whether the returned response matches the expected result. The difference is that assert() will stop the test case if the check failed, verify will not (-> assert() generates an extra "lambda: stop()" for the failing template). Most of the time, you would probably want to use assert(), but verify() can be handy for debugging purpose or small and unimportant checks. Adding comments ^^^^^^^^^^^^^^^ There are to types of comments possible. The first one is inserted via Selenium IDE -> Edit -> Insert new comment. This will generate a simply python comment. Or, use the command echo(), which will use Testerman's log() method. setverdict() ^^^^^^^^^^^^ The formatter will dump a setverdict(FAIL) during every alt step for the branch with the not-matching template (no setverdict() in storXXX(), though). A setverdict(PASS) is only printed in the very last alt branch. In fact, before dumping the code, the formatter counts all accessors (= all commands generated from isXXX() or getXXX() = every command where a result is returned). This enables him to print the setverdict(PASS) only at the very end. Best Practices ^^^^^^^^^^^^^^ If Selenium IDE is not recording your actions (although the red button is pressed), the base url is most likely wrong. Selenium IDE will only record user interaction with the site given in the base url. You can change it in the upper "address bar" in Selenium IDE gui. When using stuff like waitForCondition (e.g. functions with a timeout), you could consider to set up a high timeout and use a Testerman timer as watchdog with a lower timeout instead. If the selenium call fails (=condition didn't become true until the given timeout), the :doc:`autogen/ProbeSelenium` will throw an exception; whereas if the Testerman timer times out, the test can "fail correctly" Make sure you use [assert\|verify\|storte]Value() when checking the text of an input and not [assert\|verify\|store]Text(). Inputs do not have text, they have values. Applying getText() on an input will return an empty string. Once you did your assertions etc (e.g. your test verdict is set), reset the system to the state before the test. Wherever you want in the command list in Selenium IDE, you can use "Edit -> Insert new comment" to add a comment in the generated source coude. For example, before resetting the system to the former state you could add a comment saying "clean up" Import ats files to Selenium IDE ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ It is possible to revert the process, e.g. import a generated ats file to Selenium IDE. It does not work with every ats file, though. In fact, the generated ats files contain the list of selenese and the import mechanism exploits this list. Thus, ats without this list cannot be imported. To open an ats file with Selenium IDE, you have to change the expected format: Selenium IDE -> Options -> Format -> Testerman Anevia ATS Formatter. If this option is not available, enable it via Options -> Options -> Enable experimental features. Note that the formatter is NOT able to import ats scripts containing several test cases. ATTENTION: Make backups of your original ats file. The formatter will try to warn you if the thinks the file has been touched by hand Known Issues ~~~~~~~~~~~~ Selenese returning an array ^^^^^^^^^^^^^^^^^^^^^^^^^^^ Some commands return an array (typically commands with [assert\|verify\|waitFor]All[Buttons\|Links\|...](), like assertAllLinks()). They are handled by the formatter now in a experimental version. The generated code may not be what you expected. The formatter will warn you.