'Yet Another Test Automation Framework'

When coming to analyze the status of test automation tools/frameworks I see lots of confusions. Is 'Selenium is a test automation framework?' the answer is no.

There is a tendency to mix automation connectivity tools with automation framework.

The 2 main part of an automation solution are the automation framework, response for grouping of tests, parameterization, execution and view execution results, and connector responsible to manage the application under test, AutoIt can do it for MFC UI, WatiX (Watir, Watij, Watin…) do it for Internet Explorer…

Let analyze Selenium:

Selenium is automation connector for web browser, it's an excellent tool to manage browsers and analyze browser content. It comes with a module (part of the IDE) that has framework characteristics, but I don't think you can take it seriously. Selenium developing group understand it very well, so one of the most important feature in Selenium is the ability to integrate it to any programming language (Selenium RC).

Let analyze JUnit :

JUnit is a supper simple automation framework specifically for unit testing of Java. It includes mechanisms to group tests, execute them and analyze results. It doesn't come with any connector capabilities. You can use JUnit with Selenium to get a basic automation environment for web automation.

What do I expect from my automation framework?

1. Write my tests, can be some programming language or drag and drop user interface.

2. Group my tests to logical groups like: regression, sanity…

3. Execute of tests group.

4. Let me analysis the execution results

Advance features of a framework could be:

1. Deploy my test environment and execute tests remotely.

2. Give a setup abstraction layer that will enable me to execute my test on deferent setups without the need to change my tests.

3. Test parameterization - enable the user to affect the tests to some level without change to the tests.

4. Tests configuration management.

What do I expect from my connectors?

1. To manage my application under test.

2. To integrate as simple as possible to my automation framework both in aspect of programming languages and in aspect of reporting.

Examples for automation frameworks:

1. TestShell
2. robotframework
3. JSystem

Examples for connectors:

1. Selenium
2. Watij
3. AutoIt
4. SeeTest

How to Automate YouTube Player

I was asked by one of my colleges ‘How to automate web based players like YouTube’.

Most of the player are Flash based application embedded in an HTML pages.

Found nice blog Altom blog that explore the issue:

The main issue with RIA Flex applications is that the Flash player is basically a runtime, which makes its objects invisible to its container (e.g. browser's html or javascript code), unless explicitly exposed from within … I was able to identify three ways to add automation support for Flex applications: using the Flex Automation Framework (+ExternalInterface). Adobe has developed this framework to provide a way for developing ActionScript automation-agents that facilitate the communication between the automation tools and the Flex objects. This is possible using the ExternalInterface API. I couldn't find a way of using the Automation Framework without the ExternalInterface, although it is not specified as the only option for establishing the communication. Even though the Flex SDK is open-source, the Automation Framework is not: the trial version can be used for scripts with up to 30 actions. One would have to pay 499€ (VAT not included) for the FlexBuilder Pro license in order to create larger scripts. by developing your own framework in ActionScript and make it visible through the ExternalInterface - more info about the API can be found here. IAccessible interface and MSAA. Not all the objects support the IAccessible interface by default; here you can find more info on how it can be added.

Altom blog

I’m going to try deferent approach using SeeTest from Experitest. I wrote few posts about he tool in the past but I would like to give it test drive with YouTube and test it multi browser capabilities.

The automatic test will check the play/pause capabilities, and will change movie quality.

I will execute it both on Chrome and on Internet Explorer.

Let start:

1. Set a new project, Set the application window title name to ‘YouTube’ and capture a scene:

I assume in my test that the browser is already open. So first I have to navigate to the current url.

I extract the star icon ( ). I do it by marking it on the capture scene and press the right mouse button:

I will use it to identify the url text field, following is the Python code as I export it from the studio.

def testUntitled(self):     self.client.click("default", "start", 0, 1, 30, 0)     self.client.sendText("^a")     self.client.sendText("http://www.youtube.com/watch?v=d1_JBMrrYw8")     self.client.sendText("{ENTER}")

You can see I clicked 30 pixels to the left of the ‘start’ element. Then sent ctrl+a to select all. Then type the url and then press Enter.

2. OK that part was easy. Now I would like to extract the play and pause buttons. To do it I just take additional scene capture when the movie is playing and extract the 2 buttons:

I have to set the element name to be able to use it in my scripts.

From what I understand the ‘Main color’ is used to accelerate the element identification in runtime.

When selecting the extracted element it can be identify in the captured scene.

3. Now let finish the script, this how the python export look like:

def testUntitled(self):     self.client.click("default", "start", 0, 1, 30, 0)     self.client.sendText("^a")     self.client.sendText("http://www.youtube.com/watch?v=d1_JBMrrYw8")     self.client.sendText("{ENTER}")     self.client.waitForElement("default", "pause", 0, 10000)     self.client.click("default", "pause", 0, 1)     self.client.verifyElementFound("default", "paly")     self.client.click("default", "paly", 0, 1)     self.client.verifyElementFound("default", "pause")

4. Now come the part were I would like to to change the movie quality and check it changed. I extracted element and name it ‘quality’, I change the scan algorithm to similarity with 80% identity factor:

I did the same to for all the quality possibilities:

Now I can finish the test:

def testUntitled(self):     self.client.click("default", "start", 0, 1, 30, 0)     self.client.sendText("^a")     self.client.sendText("http://www.youtube.com/watch?v=d1_JBMrrYw8")     self.client.sendText("{ENTER}")     self.client.waitForElement("default", "pause", 0, 10000)     self.client.click("default", "pause", 0, 1)     self.client.verifyElementFound("default", "paly")     self.client.click("default", "paly", 0, 1)     self.client.verifyElementFound("default", "pause")     self.client.click("default", "quality", 0, 0)     self.client.click("default", "360p", 0, 1)     self.client.click("default", "quality", 0, 0)     self.client.verifyTextFound("default", "*360p", false)

You can see that the verification uses composition of 2 elements. It’s nice because I can extract the element once and create any composition I like (*360p, *480p and *720p).

The only part that is browser dependent in our case is the start () that is unique to Chrome.

To support Internet Explorer I will use overloading capability, I will extract the relevant element in IE ( ) and name is ‘start’ same as in Chrome.

Overloading is the capability to assign the same name to multiple elements. Any operation that uses the assign name will be perform on the first to found element.

So now we have a single script that can run both on Chrome and on IE. Let see how the execution and reports look like:

The main advantages are that it’s both robust and non-intrusive.

Automate Google Maps on Android

I own an IPhone and I really like it. But I don’t like Apple ‘close garden’ strategic.

So the Google launch of Android really made me happy.

Test Automation was well considered in the design of the Android. You have an automation API that is part of the operation system. But this is mainly used for unit test of applications developed by you.

What if you would like to automate application like ‘Google Maps’?

SeeTest from Experitest was already presented in my last port. I took it to a test drive in automating ‘Google Maps’. I don’t have a real device so I used the Android emulator.

The first challenge is to create a test that launch the ‘Google Maps’ application, search for a location and verify it was found in the map.

1. Start by crate new project and set the application title:

I remove the additional avd information in the title so it will work on any android emulator.

2. Then I capture the main desktop scene and extract the ‘Maps’ Icon. 

Now the default of the ‘Extract Element’ is to use the ‘strict’ scan algorithm:

According to the documentation it’s very fast scan algorithm but it not tolerable to any change.  So if the background changes it will not be identify. To overcome this you can change the scan algorithm to ‘Similarity’, then you can use the sensitivity bar of the algorithm:

3. Next I wrote a script that navigate to the application by sending {HOME} and then click on the ‘Maps’ icon, this is how the code export to Python looks like:

import unittest from ExperitestClient import Client class Untitled(unittest.TestCase):     def setUp(self):        self.host = "localhost"         self.port = 8888         self.client = Client()         self.client.init(self.host, self.port)         self.client.setProjectBaseDirectory("C:\\Users\\Ben.Miller\\workspace\\project12")         self.client.setReporter("xml", "reports")     def testUntitled(self):         self.client.sendText("{HOME}")         self.client.click("default", "Maps", 0, 1)     def tearDown(self):         self.client.generateReport() if __name__ == '__main__':     unittest.main()

From the export script you can see that the test can be execute on remote machine, in our case it’s executed on ‘localhost’.

4. The full script looks as follow:

def testFindNewYork(self):     self.client.sendText("{HOME}")     self.client.click("default", "maps", 0, 1)     self.client.sendText("{F2}")     self.client.click("default", "Search", 0, 1)     self.client.sendText("New York")     self.client.sendText("{ENTER}")     self.client.sleep(3000)     self.client.verifyTextFound("default", "New York", false)

I extracted all the ABC characters and used verifyTextFound, that way I can change the city name without any additional work. This is very important as I would like the test to be flexible as possible.

This is how the execution looks like:

5. OK we are done with our first task, Now I would like to see if I can identify all the cities in the map and get their coordination. To do that I extracted the city element, changed the algorithm to similarity and set the sensitivity to around 70%.

This is how the identification map look like:

 

COOL!

From the site information it look like they have the ability to execute it on real device, but I didn’t try it.