Wednesday, September 22, 2010

Test Automation, “Paper Toss” and my wife

Few days ago I download game called “Paper Toss” to my android (HTC desire), I have a tendency for addictions and this game started to be one.

scene.1283783516852

It’s very simple all you have to do is throw the paper ball to the basket. You have to take into consideration the wind strength and direction.

The problem started when my wife started to play the game. I’m very competitive and soon enough she will take the lead.

My advantage in this case is that I’m a geek. I’m using SeeTest from Experitest to test automation of Flex applications, but it do the job for Android games as well.

After ~30 minutes and few lines of code I wrote a robot that play the game for me with 0 errors.

This is how the code looks like:

image

This is how the project repository look like:

image

I played it until it score 62 (so it will not look to suspicious). Now she torture herself trying to beat me.

image

Maybe tomorrow I will let her know (and maybe I won’t…).

Friday, April 16, 2010

Games Test Automation

I was asked lately how to automate test for games. The challenge is even bigger when the game is a Facebook application.

I test-drive SeeTest from http://experitest.com to automate FarmVille.

The FarmVille is a flash application running inside Facebook.

First you have to set the application title. I set it to ‘FarmeVille on Facebook’ (that way I will be able to execute it on Firefox as well.

image

I started by capturing few scenes:

image

When you move your mouse on the captured scenes element are automatically identify (it’s cool and very easy to work with).

image

Just place your mouse on the element and right click to extract the element.

image 

Give the element a name (in our case ‘Market’).

Following are the elements that were extracted:

image

See that when you select an element it been identified in the scene tab.

Now in addition I used some interesting feature of SeeTest called overloading. You can give the same name to elements:

image

You can see I have 2 elements with the name ‘Place to seed’. When used in the script you can just used ‘Place to seed’ and the operation will be done on the first element to be found.

Now let see how the script look like:

image

Two things I like about the script:

1. It’s graphical and very simple to use.

2. You can export your script to any programming languages like: C#, Java, Python, VB-Script, Perl…

Following is how the Java export looks like:

image

And now for the execution:

Sunday, April 4, 2010

Best Practices In Web Automation with Selenium

In this post I will try to summaries my experiences with web test automation. I will refer to selenium and Java examples, although most of the issues raised in the examples are also applicable with the use of other tools as well. I will start by discussing high level strategic issues and then drill down to a more technical level.

Background

Web as GUI testing is comprised of three major elements:

  1. Object identification – find the object that the action or analysis will work on.
  2. Action – once the object was identified perform an action on it.
  3. Analysis – verify objects content and state.

Most of the complexity lies in the identification and analysis.

Business logic level automation

You’re probably aware of the pitfalls involved in the ‘record and replay’ approach. ‘Record and replay’ can be used in order to extract code snippets. It is not recommended as an automation strategy.

Why does the ‘record and replay’ fail?, it look so nice in the demo of the sales person.The reason is lack of design.

What is business logic level automation?

The idea is to separate your automation into two layers. The first layer is the building blocks infrastructure layer that uses interfaces such as GUI in order to expose the application logic. The second layer is the test layer where the building blocks layer is used to build tests.

You know that you have separated the tests well, if the tests that are written do not have any UI specific orientation.

Why is it a good practice?

  1. Organization/systems do not tend to change business logic very often. So, if you did the modeling correctly there is a chance that your tests will be valid five years from now.
  2. Most of the changes that are performed focus in a small layer of the connectivity.

Bad code example:

Window(“App”).button(“Add”).click()

Dialog(“Add user”).wait()

Dialog(“Add”).textField(“Name”).set(“guy”)

Dialog(“Add”).button(“Add”).click()

Good code example:

app.addUser(“guy”)

app.checkUserExist(“guy”)

db.checkUserExist(“guy”)

Functional testing using the user interface

There is a tendency to mix ‘user interface testing’ and ‘functional testing using the user interface’. The first is very hard to automate as it involves user experience. SeeTest from Experitest is a new tool that do just that.

Usually most of the automation focus should be ‘functional testing using the user interface’.

HTML Objects Identification Strategy

The HTML DOM is a tree of objects, you have the root tag (HTML) then you can have a child (BODY) …

In order to identify an object I have to scan the DOM tree model. There are a few ways to identify a node in the tree:

1. Tag name.

2. Tag value

3. By it relation with other nodes.

4. By attribute of the node.

Usually a combination of more than single method will be used.

In order to observe the HTML DOM I recommend Firebug ( https://addons.mozilla.org/en-US/firefox/addon/1843 )

image

After installing it (Firefox plug-in) you will see a bug icon on bottom right corner (see the red box). Pressing it will launch the plug-in for the current browser page. Then you can press the ‘Inspect’ button, when you move the mouse over the page you will see that page items becomes highlighted and the HTML section of the item can be seen.

So when looking at the Google main page the text box tag looks as follows:

<input

value=""

title="Google Search"

size="55"

name="q"

maxlength="2048"

autocomplete="off"

/>

You can see a tag called input and six attributes: value, title, size, name maxlength and autocomplete.

HTML DOM is a tree and the best way to identify object in such tree is by using XPath. “XPath is a language for finding information in an XML document” (http://www.w3schools.com/Xpath/ ).

Following are few examples to identify the input tag:

  1. //input[2] – the second input tag in the page.
  2. //input[@name=’q’] – an input tag with attribute name that equals to ‘q’.
  3. //td[input[@name='btnG']]/input[@title] – this one is little more complex. First it identifies a ‘td’ tag. The identification is done by a child tag ‘input’ that has ‘name’ attribute with ‘btnG’ value. This ‘td’ tag should have additional child with ‘titile’ attribute.

You can use selenium IDE to verify your XPath.

image

When entering the search string use the ‘Find’ button in order to identify the searched object in the HTML page. You will see a green rectangle around the found object.

When working with selenium RC (remote control) the code that uses the XPath will appear as follows:

selenium.type(“xpath=//input[@name=’q’]”, “jsystem”);

An identification strategy could appear as follows (from the highest priority and down):

  1. Use object visible name (like button name…). If does not exist or case uniqueness problem:
  2. Use unique ID tag. If does not exist:
  3. Use other unique tag. If does not exist:
  4. Zoom on possible container (like table or div) and then identify by index.
  5. Identify by index.

The guidelines used when building the identification strategy is to build a strategy that requires minimal maintenance.

Objects map

A good practice is to save all of the objects identification strings as metadata. I use the property files. For example:

map.properties file:

google.textfield= xpath=//input[@name=’q’]

code:

selenium.type(map.getProperty(“google.textfield”), “jsystem”);

State less page navigation

You should define your application home. The application home should be the starting point of every building block. A good application home can be the landing page after to login page.

Following is an example for building block implementation:

public void addUser(String username){

goHomeAndLoginIfNeeded();

navigateToPosition(Position.ADD_USER);

_addUserStrict(username);

}

Every action (addUser for example) should start with the goHomeAndLoginIfNeeded method. This method with navigate to the home URL it will check if login requires and if yes will login.

Automation sanity

The idea of ‘automation sanity’ is to create small and non-context tests that will verify all the object in the GUI map exists. The main idea is to find objects identification problems in the right context. When a functional test is executed and fails because of an identification problem the investigation can be painful and waist lot of time.

Tuesday, March 16, 2010

'Yet Another Test Automation Framework'

When coming to analyze the status of test automation tools/frameworks I see lots of confusions. Is 'Selenium is a test automation framework?' the answer is no.

There is a tendency to mix automation connectivity tools with automation framework.

The 2 main part of an automation solution are the automation framework, response for grouping of tests, parameterization, execution and view execution results, and connector responsible to manage the application under test, AutoIt can do it for MFC UI, WatiX (Watir, Watij, Watin…) do it for Internet Explorer…

Let analyze Selenium:

Selenium is automation connector for web browser, it's an excellent tool to manage browsers and analyze browser content. It comes with a module (part of the IDE) that has framework characteristics, but I don't think you can take it seriously. Selenium developing group understand it very well, so one of the most important feature in Selenium is the ability to integrate it to any programming language (Selenium RC).

Let analyze JUnit :

JUnit is a supper simple automation framework specifically for unit testing of Java. It includes mechanisms to group tests, execute them and analyze results. It doesn't come with any connector capabilities. You can use JUnit with Selenium to get a basic automation environment for web automation.

What do I expect from my automation framework?

1. Write my tests, can be some programming language or drag and drop user interface.

2. Group my tests to logical groups like: regression, sanity…

3. Execute of tests group.

4. Let me analysis the execution results

Advance features of a framework could be:

1. Deploy my test environment and execute tests remotely.

2. Give a setup abstraction layer that will enable me to execute my test on deferent setups without the need to change my tests.

3. Test parameterization - enable the user to affect the tests to some level without change to the tests.

4. Tests configuration management.

What do I expect from my connectors?

1. To manage my application under test.

2. To integrate as simple as possible to my automation framework both in aspect of programming languages and in aspect of reporting.

Examples for automation frameworks:

  1. TestShell
  2. robotframework
  3. JSystem

Examples for connectors:

  1. Selenium
  2. Watij
  3. AutoIt
  4. SeeTest

Tuesday, March 9, 2010

How to Automate YouTube Player

I was asked by one of my colleges ‘How to automate web based players like YouTube’.

Most of the player are Flash based application embedded in an HTML pages.

Found nice blog Altom blog that explore the issue:

The main issue with RIA Flex applications is that the Flash player is basically a runtime, which makes its objects invisible to its container (e.g. browser's html or javascript code), unless explicitly exposed from within

I was able to identify three ways to add automation support for Flex applications:

  1. using the Flex Automation Framework (+ExternalInterface). Adobe has developed this framework to provide a way for developing ActionScript automation-agents that facilitate the communication between the automation tools and the Flex objects. This is possible using the ExternalInterface API. I couldn't find a way of using the Automation Framework without the ExternalInterface, although it is not specified as the only option for establishing the communication. Even though the Flex SDK is open-source, the Automation Framework is not: the trial version can be used for scripts with up to 30 actions. One would have to pay 499€ (VAT not included) for the FlexBuilder Pro license in order to create larger scripts.
  2. by developing your own framework in ActionScript and make it visible through the ExternalInterface - more info about the API can be found here.
  3. IAccessible interface and MSAA. Not all the objects support the IAccessible interface by default; here you can find more info on how it can be added.

Altom blog

I’m going to try deferent approach using SeeTest from Experitest. I wrote few posts about he tool in the past but I would like to give it test drive with YouTube and test it multi browser capabilities.

The automatic test will check the play/pause capabilities, and will change movie quality.

I will execute it both on Chrome and on Internet Explorer.

Let start:

1. Set a new project, Set the application window title name to ‘YouTube’ and capture a scene:

image

image

I assume in my test that the browser is already open. So first I have to navigate to the current url.

I extract the star icon (image ). I do it by marking it on the capture scene and press the right mouse button:

image

I will use it to identify the url text field, following is the Python code as I export it from the studio.

def testUntitled(self):
    self.client.click("default", "start", 0, 1, 30, 0)
    self.client.sendText("^a")
    self.client.sendText("http://www.youtube.com/watch?v=d1_JBMrrYw8")
    self.client.sendText("{ENTER}")

You can see I clicked 30 pixels to the left of the ‘start’ element. Then sent ctrl+a to select all. Then type the url and then press Enter.

2. OK that part was easy. Now I would like to extract the play and pause buttons. To do it I just take additional scene capture when the movie is playing and extract the 2 buttons:

image

I have to set the element name to be able to use it in my scripts.

From what I understand the ‘Main color’ is used to accelerate the element identification in runtime.

image

When selecting the extracted element it can be identify in the captured scene.

3. Now let finish the script, this how the python export look like:

def testUntitled(self):
    self.client.click("default", "start", 0, 1, 30, 0)
    self.client.sendText("^a")
    self.client.sendText("http://www.youtube.com/watch?v=d1_JBMrrYw8")
    self.client.sendText("{ENTER}")
    self.client.waitForElement("default", "pause", 0, 10000)
    self.client.click("default", "pause", 0, 1)
    self.client.verifyElementFound("default", "paly")
    self.client.click("default", "paly", 0, 1)
    self.client.verifyElementFound("default", "pause")

4. Now come the part were I would like to to change the movie quality and check it changed. I extracted image element and name it ‘quality’, I change the scan algorithm to similarity with 80% identity factor:

image

I did the same to for all the quality possibilities:

image

Now I can finish the test:

def testUntitled(self):
    self.client.click("default", "start", 0, 1, 30, 0)
    self.client.sendText("^a")
    self.client.sendText("http://www.youtube.com/watch?v=d1_JBMrrYw8")
    self.client.sendText("{ENTER}")
    self.client.waitForElement("default", "pause", 0, 10000)
    self.client.click("default", "pause", 0, 1)
    self.client.verifyElementFound("default", "paly")
    self.client.click("default", "paly", 0, 1)
    self.client.verifyElementFound("default", "pause")
    self.client.click("default", "quality", 0, 0)
    self.client.click("default", "360p", 0, 1)
    self.client.click("default", "quality", 0, 0)
    self.client.verifyTextFound("default", "*360p", false)

You can see that the verification uses composition of 2 elements. It’s nice because I can extract the element once and create any composition I like (*360p, *480p and *720p).

The only part that is browser dependent in our case is the start (image) that is unique to Chrome.

To support Internet Explorer I will use overloading capability, I will extract the relevant element in IE (image ) and name is ‘start’ same as in Chrome.

Overloading is the capability to assign the same name to multiple elements. Any operation that uses the assign name will be perform on the first to found element.

So now we have a single script that can run both on Chrome and on IE. Let see how the execution and reports look like:

The main advantages are that it’s both robust and non-intrusive.

Friday, March 5, 2010

Automate Google Maps on Android

I own an IPhone and I really like it. But I don’t like Apple ‘close garden’ strategic.

So the Google launch of Android really made me happy.

Test Automation was well considered in the design of the Android. You have an automation API that is part of the operation system. But this is mainly used for unit test of applications developed by you.

What if you would like to automate application like ‘Google Maps’?

SeeTest from Experitest was already presented in my last port. I took it to a test drive in automating ‘Google Maps’. I don’t have a real device so I used the Android emulator.

The first challenge is to create a test that launch the ‘Google Maps’ application, search for a location and verify it was found in the map.

1. Start by crate new project and set the application title:
image

I remove the additional avd information in the title so it will work on any android emulator.

2. Then I capture the main desktop scene and extract the ‘Maps’ Icon. 
image

Now the default of the ‘Extract Element’ is to use the ‘strict’ scan algorithm:
image

According to the documentation it’s very fast scan algorithm but it not tolerable to any change.  So if the background changes it will not be identify. To overcome this you can change the scan algorithm to ‘Similarity’, then you can use the sensitivity bar of the algorithm:

image

3. Next I wrote a script that navigate to the application by sending {HOME} and then click on the ‘Maps’ icon, this is how the code export to Python looks like:

import unittest
from ExperitestClient import Client

class Untitled(unittest.TestCase):
    def setUp(self):
       self.host = "localhost"
        self.port = 8888
        self.client = Client()
        self.client.init(self.host, self.port)
        self.client.setProjectBaseDirectory("C:\\Users\\Ben.Miller\\workspace\\project12")
        self.client.setReporter("xml", "reports")

    def testUntitled(self):
        self.client.sendText("{HOME}")
        self.client.click("default", "Maps", 0, 1)

    def tearDown(self):
        self.client.generateReport()

if __name__ == '__main__':
    unittest.main()

From the export script you can see that the test can be execute on remote machine, in our case it’s executed on ‘localhost’.

4. The full script looks as follow:

def testFindNewYork(self):
    self.client.sendText("{HOME}")
    self.client.click("default", "maps", 0, 1)
    self.client.sendText("{F2}")
    self.client.click("default", "Search", 0, 1)
    self.client.sendText("New York")
    self.client.sendText("{ENTER}")
    self.client.sleep(3000)
    self.client.verifyTextFound("default", "New York", false)

I extracted all the ABC characters and used verifyTextFound, that way I can change the city name without any additional work. This is very important as I would like the test to be flexible as possible.

This is how the execution looks like:

5. OK we are done with our first task, Now I would like to see if I can identify all the cities in the map and get their coordination. To do that I extracted the city element, changed the algorithm to similarity and set the sensitivity to around 70%.

This is how the identification map look like:

image 

COOL!

From the site information it look like they have the ability to execute it on real device, but I didn’t try it.

Friday, February 26, 2010

Test automation for Flex application

Few days ago I ran into the 'Using the automated testing API' by Adobe. It explain how to automation Flex application using QTP, 68 pages with lots of XMLs and other technical staff.

clip_image002clip_image004clip_image006

After 4 hours of reading I'm not positive I really can operate this thing. I'm not a Flex programmer and the document includes lots of Flex internals.

I must be missing something must it be so complicated?

I googled for a while and come up with amazingly simple solution:

SeeTest from Experitest. It took me less than 10 minutes to automate my first test. Following is a description of the process:

1. This is the application I would like to automate:
 image

It call the FlexStore and it's the demo application that come with Flex.
The flow I would like to automate is very simple: select a phone, add it to the cart and verify my basket amount is correct.

2. This is the SeeTest application:
image
You start by setting a project directory and then set the title of the application under test window in our case I selected 'FlexStore' (without the Mozilla Firefox).
image

3. Next thing is to take few screenshots of the application under test and graphically extract the required elements:
image
Just mark the element you would like to extract and set the element name:
image

4. OK this is how it look with all the element extracted:
image
You can see that when I select the element you can see it identification in the captured scene.

5. Now it is time to build a basic script, you should move to the screen tab and build the script:
imageI press the play button and it work (Amazing I'm less than 5 min into the process).
Now come the interest part. I would like to analyze the cart 'Grand Total' and verify it's $140.99:
image
I can extract the all element but it's not flexible enough when the amount changes I will have to extract it again.

6. Now comes the interesting part (I was really excited when I discovered it). I can use the multi extraction feature to learn all the number digit elements:
image
image
I don't know how they do it but the extracted element is identify even when the element color or background changes.

7. Next thing we should finish the script:
image
You can see that I uses the $140.99 as an open text.

8. Following video show the execution and the report result:



9. Now you can take my script and export to any programming language you like, I used JUnit: 
image

OK my 10 minutes are up and I’m with my first script running – COOL!