Master Thesis Title: Evaluation of Software Testing Tools ( Jenkins, CruiseControl, Selenium, Cantata)

== This thesis should evaluate all the above 4 testing tools according to some evaluation criteria.  I am attaching an evaluation criteria that has been approved by my examiner. So the evaluation criteria is already designed. The evaluation criteria can be fit into the thesis report. The evaluation should strictly cope with each of the criteria and sub critiera. How to get the results for each criteria and sub-criteria for each tool is given as: 1. Install Jenkins (Continuous Integration Server) and play with it to get the required results from it and then write it down under each criteria and sub-criteria. 2. Install CruiseControl(Continuous Integration Server) and play with it to get the required results from it and then write it down under each criteria and sub criteria for CrusieControl 3. Install Selenium and do the same. 4. Install Cantata (Automated tool for unit testing of C/C++ language codes) and do the same as above.  There should be one separate chapter for Results where the each tool is compared with the other tool according to the evaluation results. After the results there should be discussion and Conclusion. and Finally the references.  Please note that this will not be possible without the installation of all the four tools and therefore no evaluation without the tool installation will be accepatiable. The examiner should see that the tools have been installed for gettting the results. Please let me know via email if I need to give further information.

Websites : Jenkins = https://jenkins-ci.org/

                    CruiseControl = http://cruisecontrol.sourceforge.net/

                    Selenium = http://www.seleniumhq.org/

                    Cantata = http://www.qa-systems.com/cantata.html

• Software Testing and Continuous Integration
• ··  60 pages, 15000 words Revised Sweden
• Harvard Revised 120 Refers
• (+46) ***********
• Jul 15, 2015 | Wednesday 06:28 am | 15 days 1 hrs ago

Table of Contents

What is Software Testing. 5

What is Unit Testing. 6

Jenkins Functionality. 6

Suitability. 7

Accurateness. 7

Interoperability. 7

• Building/testing software projects continuously. 7
• Monitoring executions of externally-run jobs. 7

Compliance. 7

1.      Easy installation. 7

2.      Easy configuration. 7

3.      Change set support 7

4.      Permanent links. 7

5.      RSS/E-mail/IM Integration. 7

6.      After-the-fact tagging. 7

7.      JUnit/TestNG test reporting. 7

8.      Distributed builds. 7

9.      File fingerprinting. 7

10.        Plugin Support 8

Security. 8

Portability. 8

Adaptability. 8

Installation. 8

Conformance. 8

Usability. 8

Understandability. 9

Learnability. 9

Ease of Use. 10

Jenkins management. 13

4.1. Plug-in management 13

Setting up a Jenkins job. 14

Prerequisites for build Android apps with Jenkins. 18

Reliability. 18

Maturity. 18

Fault Tolerance. 20

Recoverability. 20

Efficiency. 21

Testing Time. 21

Result Publishing. 22

Graphs. 22

Change Log. 24

Vendor Support. 25

Email Support 25

Phone support 25

User’s Group. 26

Training. 27

Licensing and Support. 27

Open Source/ Commercial 27

Competitors. 27

Cantata Functionality. 28

Suitability. 28

Accurateness. 28

Interoperability. 28

Compliance. 28

Security. 29

Reliability. 29

Maturity. 29

Fault Tolerance. 29

Recoverability. 30

Portability. 30

Adaptability. 30

Conformance. 30

Installation. 30

Usability. 30

Understandability. 31

Learnability. 31

• Static analysis metrics. 31
• A powerful built-in EDG C/C++ parser. 31
• Preferences for the appearance. 31
• Test and analysis generation options. 31
• Test harness. 31
• Unique testability instrumentation and wrapping. 31
• Integrated code coverage and test optimization. 31
• Results and. 31
• Automated repeatable execution through Cantata. 31

Ease of Use. 31

Using the Cantata IDE.. 31

Using the Command Line. 31

Licensing: 31

Efficiency. 31

Testing Time. 31

Result Publishing. 32

Detailed Information. 32

Vendor Support. 32

Email Support 32

Phone Support 33

User’s Group. 33

Training. 33

Licensing and Pricing. 33

Open Source/ Commercial 33

Competitors. 33

Cruisecontrol 33

Introduction. 33

Background. 34

Using the Code. 34

CruiseControl.net 35

Cruisecontrol Tray. 36

Nant / MsBuild. 37

Sandcastle. 38

Nunit 38

Principals of Unit Testing. 38

Tasks Needed to Get CruiseControl.net Working on a New Server. 39

Getting the WebDashBoard Working with IIS. 39

Difference between Cruisecontrol and Jenkis: 40

Jenkins with Selenium.. 42

What is Selenium.. 43

Executing the Test Suite. 46

Running the Test Suite. 46

History of Selenium.. 49

Why Selenium?. 50

Selenium Test Package. 50

Selenium Modes: 51

Record-Playback mode (Selenium IDE)‏: 51

Selenium Remote Control (RC) Mode. 51

Test Runner Mode. 51

Testing AJAX with Selenium.. 51

Selenium IDE – Recording and updating a script 51

Selenium IDE – Most commonly used commands. 52

Selenium Components. 53

Selenium IDE.. 53

Selenium client API. 53

Selenium Remote Control 53

Selenium WebDriver. 54

Selenium Grid. 54

The Selenium Ecosystem.. 55

Popular tools integrating with Selenium.. 55

Services running Selenium on the cloud. 55

Advantages of Selenium.. 56

Disadvantage of Selenium.. 57

Conclusion. 57

References. 58

What is Software Testing

Software testing is a conducted investigation to give stakeholders all the information about product quality or the given service under the test. Software testing will also give an objective, view that is independent for the software which willpermit the business to understand and recognize the risks which are involved with the implementation of software. The Test techniques which are included in the process for executing an application or program with the intent of finding bugs in the software (errors or other defects).

It will also involve the implementation of the component of software or component of system to estimate one or more properties of the interest. In particular, these are the properties which indicate the extent on which the system or component under test:

• convene the requirements which are guided its development and design,
• correctly and rightly responds to all types of the inputs,
• executes the functions within an acceptable time,
• is suitably usable,
• may be installed and also run in theproposed environments, and
• Achieves the common result its stakeholder’s desire.

As the amount of likely tests for still simple components of software is virtually infinite, all the testing of software uses few strategies to find tests which are feasible for the accessibleresources and time. As the result, testing of software typically (but not completely) attempts to implement an application or program with the intent of getting the bugs in software (errors or some other defects). The job of the testing is an iterative process as soon as when one of the bug is fixed; it will illuminate other, the deeper bugs, or may even generate new ones.

Software testing can provide objective, independent information about the quality of software and risk of its failure to users and/or sponsors.

Software testing can be conducted as soon as executable software (even if partially complete) exists. The overall approach to software development often determines when and how testing is conducted. For example, in a phased process, most testing occurs after system requirements have been defined and then implemented in testable programs. In contrast, under an Agile approach, requirements, programming, and testing are often done concurrently.

What is Unit Testing

Unit testing, also known as component testing refers to tests that verify the functionality of a specific section of code, usually at the function level. In an object-oriented environment, this is usually at the class level, and the minimal unit tests include the constructors and destructors.

These types of tests are usually written by developers as they work on code (white-box style), to ensure that the specific function is working as expected. One function might have multiple tests, to catch corner cases or other branches in the code. Unit testing alone cannot verify the functionality of a piece of software, but rather is used to ensure that the building blocks of the software work independently from each other.

Unit testing is a software development process that involves synchronized application of a broad spectrum of defect prevention and detection strategies in order to reduce software development risks, time, and costs. It is performed by the software developer or engineer during the construction phase of the software development lifecycle. Rather than replace traditional QA focuses, it augments it. Unit testing aims to eliminate construction errors before code is promoted to QA; this strategy is intended to increase the quality of the resulting software as well as the efficiency of the overall development and QA process.

Depending on the organization’s expectations for software development, unit testing might include static code analysis, data flow analysis, metrics analysis, peer code reviews, code coverage analysis and other software verification practices.

Here is this thesis we will study about 4 unit testing Jenkis, Selenium, Curisecontrol and Cantata.

Jenkins Functionality

Suitability

• Jenkins was originally developed as the Hudson project. So if you ever see Hudson then it will be known as Jenkins CI Tool. This is written in Java.
• Hudson’s creation started in summer of 2004 at Sun Microsystems. It was first released in java.net in Feb. 2005.Jenkins is an award-winning application that monitors executions of repeated jobs, such as building a software project or jobs run by cron.

Accurateness

Interoperability

Among those things, current Jenkins focuses on the following two jobs:

• Building/testing software projects continuously, just like CruiseControl or DamageControl. In a nutshell, Jenkins provides an easy-to-use so-called continuous integration system, making it easier for developers to integrate changes to the project, and making it easier for users to obtain a fresh build. The automated, continuous build increases the productivity.
• Monitoring executions of externally-run jobs, such as cron jobs and procmail jobs, even those that are run on a remote machine. For example, with cron, all you receive is regular e-mails that capture the output, and it is up to you to look at them diligently and notice when it broke. Jenkins keeps those outputs and makes it easy for you to notice when something is wrong.

Compliance

1. Easy installation: Just java -jar jenkins.war, or deploy it in a servlet container. No additional install, no database.
2. Easy configuration: Jenkins can be configured entirely from its friendly web GUI with extensive on-the-fly error checks and inline help. There’s no need to tweak XML manually anymore, although if you’d like to do so, you can do that, too.
3. Change set support: Jenkins can generate a list of changes made into the build from Subversion/CVS. This is also done in a fairly efficient fashion, to reduce the load on the repository.
4. Permanent links: Jenkins gives you clean readable URLs for most of its pages, including some permalinks like “latest build”/”latest successful build”, so that they can be easily linked from elsewhere.
5. RSS/E-mail/IM Integration: Monitor builds results by RSS or e-mail to get real-time notifications on failures.
6. After-the-fact tagging: Builds can be tagged long after builds are completed.
7. JUnit/TestNG test reporting: JUnit test reports can be tabulated, summarized, and displayed with history information, such as when it started breaking, etc. History trend is plotted into a graph.
8. Distributed builds: Jenkins can distribute build/test loads to multiple computers. This lets you get the most out of those idle workstations sitting beneath developers’ desks.
9. File fingerprinting: Jenkins can keep track of which build produced which jars, and which build is using which version of jars, and so on. This works even for jars that are produced outside Jenkins, and is ideal for projects to track dependency.
10. Plugin Support: Jenkins can be extended via 3rd party plugins. You can write plugins to make Jenkins support tools/processes that your team uses.

Security

Continuous Integration with automated test execution has seen broad adoption in recent years. The ideas behind Continuous Integration have changed how companies look at Build Management, Release Management, Deployment Automation, and Test Orchestration.

Portability

Adaptability

This section provides a set of best practices for Jenkins – A Continuous Integration Solution to provide executives, business managers, software developers and architects a better sense of the development progress and code quality of projects throughout the development lifecycle.

Installation

To run Jenkins, you need the Java Runtime Environment (JRE) 1.7 or later. After you download jenkins.war, you can launch it by executing java -jar jenkins.war. This is basically the same set up as the test drive, except that the output will go to console, not to a window.

Once the war file is exploded, run chmod 755 jenkins in the exploded jenkins/WEB-INF directory so that you can execute this shell script.

If you’re running on Windows you might want to run Jenkins as a service so it starts up automatically without requiring a user to log in. The easiest way to achieve this is to simply download and install Jenkins via the native Windows installer, which can be found on the Jenkins main page. Just download the .zip file, unpack it, and click through the install wizard. The wizard both installs Jenkins and sets up the Jenkins Windows service for you.

Another way to run Jenkins as a service is to first install Tomcat as a service and then deploy Jenkins to it in the usual way. Another way is to use the Java Service Wrapper. However, there may be problems using the service wrapper, because the Main class in Jenkins in the default namespace conflicts with the service wrapper main class. Deploying inside a service container (Tomcat, Jetty, etc.) is probably more straightforward, even for developers without experience with such containers.

Conformance

Alternatively, if you have a servlet container that supports Servlet 2.4/JSP 2.0 or later, such as Glassfish, Tomcat 5, JBoss, Jetty 6, etc., then you can deploy jenkins.war as you would any WAR file. See this document for more about container-specific installation instruction.

Usability

Understandability

Continuous integration is a process in which all development work is integrated at a predefined time or event and the resulting work is automatically tested and built.

Jenkins is one open source tool to perform continuous integration. The basic functionality of Jenkins is to execute a predefined list of steps based on a certain trigger.

The list of steps can for example include:

• perform a software build with Apache Maven or Gradle
• Run a shell script
• Archive the build result
• Afterwards start the integration tests

Jenkins can be extended by additional plug-ins, e.g., for building and testing Android applications or to support the Git version control system.

Learnability

Jenkins provides Debian/Ubuntu packages which install Jenkins and register Jenkins as start service. See the following URL for details.

http://pkg.jenkins-ci.org/debian/

Jenkins stores all the settings, logs and build artifacts in its home directory. The default installation directory is/var/lib/jenkins under Ubuntu.

This creates a /etc/init.d/jenkins start script which starts Jenkins automatically at boot time.

If you start it locally, you find it running under the following URL: http://localhost:8080/

For most platforms you have native packages, see the Jenkins Homepage.

Download the jenkins.war file from Jenkins Homepage.

You can also start Jenkins directly via the command line with java -jar jenkins*.war. If you start it locally, you find it running under the following URL: http://localhost:8080/

To run it in your Tomcat server, put the .WAR file into the webapps directory. If you start Tomcat, your Jenkins installation will be available under http://localhost:8080/jenkins

cd /usr/share/tomcat7

sudo mkdir .jenkins

sudo

            chown tomcat7:nogroup .jenkins

This makes the .jenkins folder writable and Jenkins can use it.

If you want to install Jenkins in a web container, you can use, for example, Tomcat or Jetty. See the Apache Tomcat Tutorial.

Ease of Use

Before using Jenkins you need to tell it where your JDK and ant installation is. Open Jenkins in your browser and click Manage Jenkins and then Configure System.

Enter the correct path to your JDK, Apache Ant and Maven and press the Save button below. Jenkins can also install these for your automatically.

It is recommended to secure Jenkins. Manage Jenkins and then Configure Global Security. Select the Enable security flag. The easiest way is to use Jenkins own user database. Create at least the user “Anonymous” with read access. Also create entries for the users you want to add in the next step.

On the login page, select Create an account to create the users you just gave access.

Go to Manage Jenkins, Manage and Assign Roles and then Assign Roles to grant the newly created user additional access rights.

Navigate to Manage Roles to define access restrictions in detail. Pattern is a regex value of the job name. The following grants unregistered users read-only access to your build jobs that start with the C-MASTER or M-MASTERprefix and only those.

Jenkins management

4.1. Plug-in management

Jenkins can be extended via additional plug-ins with more functionality. You can configure your plug-ins via theManage Jenkins → Manager Plugins link.

To install plugins in Jenkins select use the Manage Jenkins → Manager Plugins link and search for the plugin you want to install. Select it from the list and select to install it and restart Jenkins.

The following table is a summary of commonly used plug-ins.

Table 1. Jenkins plug-ins

# install vncserver

apt-get install vnc4server

# switch to jenkins user

sudo su jenkins

# connect to vncserver which creates the password

vncserver :10

Setting up a Jenkins job

The build of a project is handled via jobs in Jenkins. Select New Item from the menu

Afterwards enter a name for the job and and select Freestyle Job. Press OK to create a new Job in Jenkins.

The next page allows you to configure your job. If you for example using Git, enter the URL to the Git repository. If the repository is not public, you may also need to configure the credentials.

Specify when and how your build should be triggered. The following example polls the Git repository every 15 min and triggers a build, if something has changed in the repo.

I typically delete the workspace before a build to avoid any side-effect. In the Build section you can add a build step, e.g., a Maven build.

Press Save to finish the job definition. Press Build Now on the job page to validate the job works as expected.

After a while the job should go to green or blue (depending on your configuration), if successful. Click on the job and afterwards on Console Output to see the log file in case of an error or to validate that log looks as expected.

Prerequisites for build Android apps with Jenkins

The following descriptions for build Android applications with Jenkins assume that you have already a correct Gradle build file for your Android application.

Reliability

Maturity

Jenkins supports the automatic building and testing of Android applications.

To create a build job on Jenkins you need to have a working build setup. The Android tooling generates automatically a valid Gradle build file.

# download the Android SDK via wget

# for the correct URL see http://developer.android.com/sdk/index.html

# and look in the section "SDK Tools Only"

wget <link from the above website>

# example

# wget http://dl.google.com/android/android-sdk_r24.2-linux.tgz

# unpack it

tar zxvf filename

# Add the new directory to your patch

# assumes you exported it to /opt/

# this must be set for the Jenkins user

export ANDROID_HOME="/opt/android-sdk-linux"

export PATH="$ANDROID_HOME/tools:$ANDROID_HOME/platform-tools:$PATH"

// run this as Jenkins user

android update sdk --no-ui

Use android list targets to see what ABIs are installed. If no are listed manual install one.

You need the follwing plug-ins to build Android application with Jenkins:

• Gradle Plugin
• Git Plugin – if Git is used for the project sources
• Android Emulator Plugin – supporting starting and unlocking an Android emulator and blocks the build until the emulator has started. Required for running tests. A detailed description of this plugin can be found under the following URL: Android Emulator Plugin Jenkins page.

You configure from where the source should be cloned.

You configure the emulator which should be started. Ensure that you do not select the Show emulator windowoption, as your build server should not depend on the availability of a display server.

Add a Gradle build step to your Jenkins jobs.

Fault Tolerance

The possibility of increasing the fault tolerance of Jenkins is by running multiple instances of Jenkins in parallel and let them cooperate and share the load. In case failure occurs to one Jenkins, another Jenkins can take over. The major restriction was to not modify the source code of Jenkins. The results of the investigation are that Jenkins can be modified to achieve high availability in the form of an active/active solution. Events in Jenkins related to build jobs can be replicated, these include: creating, modifying and deleting build jobs, also replication of builds can be replicated. The data replication technique had to be done asynchronously which introduced problems such as data collisions and ping-ponging. No solutions were found that could avoid data collisions; however, they can be detected and resolved. A major issue is that other plugins in Jenkins could not be made aware of there being more than a single Jenkins. This issue and areas such as synchronization of user accounts, and the behavior of Jenkins slaves,

Recoverability

Jenkins stores all the settings, logs and build artifacts in its home directory, for example, in /var/lib/jenkinsunder the default install location of Ubuntu.

To create a backup of your Jenkins setup, just copy this directory.

The jobs directory contains the individual jobs configured in the Jenkins install. You can move a job from one Jenkins installation to another by copying the corresponding job directory. You can also copy a job directory to clone a job or rename the directory.

Click reload config button in the Jenkins web user interface to force Jenkins to reload configuration from the disk.

Jenkins supports the https://wiki.jenkins-ci.org/display/JENKINS/SCM+Sync+configuration+plugin plug-in which allows you to store every change in a Git repo.

It is also possible to manually maintain the Jenkins configuration in a Git repo.

Efficiency

Testing Time

When our needs started growing, our first thought was to add more Jenkins nodes and have multiple tests running in parallel. But there are many different environments that we want to test every commit against:

• Test the deployment of the app in a clean environment, a fresh build.
• Test the deployment of the app in a staging environment where you want to ensure backwards compatibility
• Test the web interface against all supported browsers.

All these meant that testing requirements would grow over time and a testing infrastructure based solely on Jenkins would not do the job fast enough.

Docker helps you easily create lightweight, portable, self-sufficient containers from any application. It is fast, reliable and a perfect fit for our needs. The idea, is to set up Jenkins so that every pull request to a specific branch (e.g. staging) triggers a job. Jenkins then commands our testing infrastructure to spawn different docker applications simultaneously and runs the tests in parallel.

This way, we can test any environment. We just have to describe each application the same way we would’ve done manually. On the plus side, we can use pre-made images from the docker repository to jumpstart our tests.

For example we could do this:

docker pull ubuntu

And we will end up with an ubuntu image/application. We can then run the docker container by typing:

docker run -i -t ubuntu /bin/bash

And we will be in a fresh, newly created ubuntu environment.

But we don’t want to manually run docker commands after every pull request. What we need, are Dockerfiles. Dockerfiles are sets of steps that describe an image/container. We’ll use them to build our custom docker images.

Result Publishing

While executing our tests on Jenkins many of time test fails on particular builds. Now if you want to check that when a particular test-case, test-class or a test-package has failed you may need to go to each and every build report and check for the result.

This process is very cumbersome and may take a lot of time to get the report. The said plugin solves the said issue by showing the build result history of test-class, test-class and test-package in a tabular tree format. The plugin can be used enabling the “publish junit results” feature of Jenkins.

It allows users to filter the results based on passed, failed and skipped status.

Once installed you can just click on the “Test Results Analyzer” link on the left hand side of your job, as shown in the following image:

Graphs

We all know the importance of the graphical report for our test execution as it gives us a clear picture of the execution. The said plugin also supports generation of Graphs for the test execution from 0.2.0 version onwards.

Following charts are available:

– Line Charts

– Pie Charts

– Bar Charts

Change Log

Version 0.2.1

• JENKINS-29111 – Show packages/classes/methods in sorted order
• JENKINS-27628 – Test results analyzer displays class as FAILED when a testcase is skipped
• JENKINS-29186 – Pie chart title showing a wrong build number when the chart generated for a build from line chart
• Changes done to UI look and feel for the table and charts. Thanks to “RICHARD Bruno” for the changes.

Version 0.2.0

• Changes done to generate line, bar and pie chart for the report.
• Allow user to generate a pie chart for a particular build when user clicks on a build in line chart.
• User have the options to choose from line, bar, pie or all charts

Vendor Support

Email Support

First, determine what kind of issue it is. There are three main types whose use is explained on this page:

• Bug: Something does not work as intended.
• New Feature: You want Jenkins to have new functionality.
• Improvement: Existing functionality should to be changed.

Instead of opening an issue of type Patch, consider submitting a pull request on GitHub. You’re probably not going to file a Task. Neither will be explained here. Yes, Email support is there.

Phone support

Yes support of phone is available.

• First, search for the error message or general problem description. This can help you find solutions if it’s actually e.g. a configuration problem, or point you to earlier reports in case you’re not the first one experiencing this problem. You can then watch them and vote for them in JIRA. Voting helps us determine the impact of bugs and prioritize them. If you have doubts that the problem you’re experiencing is the exact same one as you found, prefer to file a new issue (and mention the one(s) you found in the description).
• Make sure that the problem actually is with Jenkins. If your build seems to fail for no reason, try to build your project outside Jenkins, but in the same general environment (e.g. same machine, same user account, …). If Jenkins interacts with other systems, make sure that they’re working. If the connections between Jenkins master and slaves fail, make sure the network is reliable.
• Check the changelogs to see whether the issue has been fixed in newer versions of the affected component. The plugin’s wiki page also be reached directly by clicking the plugin name in Manage Jenkins » Manage Plugins » Installed.

If you’re still unable to resolve the problem and cannot find existing issue reports or related fixes in newer versions of Jenkins or the plugins, file a bug in JIRA.

Before requesting an improvement or feature

• Check whether newer versions of Jenkins or the plugin in question have been released. They may have the feature already!
• Search JIRA for existing requests for this feature. You can then watch them and vote for them. Voting helps us determine the demand for requests and prioritize them, so it’s in your interest to actually find these rather than reporting a duplicate.

If you cannot find anything, file an improvement or new feature request in JIRA.

User’s Group

You need to register an account if you haven’t already. Then, just select Create Issue on the JIRA home page.

• For Project select:
  • Jenkins for general issues with Jenkins,
  • Security Issues if you want to report a security issue privately, or
  • Infrastructure if you’re reporting a bug with the Jenkins web site, wiki, or any other services run by the Jenkins project.
• Enter a short but meaningful description of your problem as Summary.
• For Priority, see here for descriptions of the levels. This is not how urgently you want it fixed. Try to be realistic: Most bugs should be Minor or Major priority. If you found a workaround, reduce the priority, even if it’s a severe problem otherwise.
• As Component, select core if it’s a problem with Jenkins itself and not any of the plugins. Otherwise, select the name of the plugin; all of them should have a related component. You can also select multiple components e.g. if the problem is in the interaction of the components, or when you’re not sure.
• Leave the Assignee field to automatic.

Environment is mostly relevant for bug reports: What is the configuration experiencing the issue? Depending on the type of problem, the following are often relevant:

• Operating System (including version, and whether it’s 32/64 bit) of all systems involved (your client’s, the Jenkins server’s, all slave nodes’),
• All relevant JRE/JDK vendors and versions (e.g. Oracle JRE, OpenJDK, …) and the parameters set.
• Jenkins and plugin versions,
• Whether you’re running Jenkins directly or in a container like Tomcat (which one, in which version?)
• Whether Jenkins is accessed through a reverse proxy (if so, how is it configured?)
• How you installed Jenkins (Windows installer, deb/rpm, …), and how you’re launching any involved slave nodes (via SSH, web browser, command line, …)
• Your web browser (+ version). Make sure the issue occurs on supported web browsers.

The Description should be comprehensive to allow others to understand the problem or improvement request immediately. A few suggestions on what to include (again, mostly applicable to bug reports):

• Did the issue occur after an upgrade of Jenkins or any plugins? If so, what was/were the previous version(s), what is/are the current version(s)? Does downgrading resolve the issue? For plugins, the previous versions can often be seen in Plugin Manager (where it offers to downgrade to the previous version).
• Include log output and stack traces of error messages from the UI or any client programs.
• Provide a complete step-by-step description of how to reproduce the problem. We cannot see your screen, so be comprehensive. Consider switching the Jenkins UI to English by changing your web browser’s configuration to get the labels right.
• If possible, provide information on the circumstances the issue occurs with (or doesn’t), e.g. “only when starting the slave via SSH”, “only when using Tomcat as container”, etc. – this may be some work, but it’ll make reproducing and fixing the issue much easier!
• Does the issue occur with a new Jenkins installation not reusing any old configuration or data? Try to reproduce the problem with a pristine Jenkins installation with as little customization as possible.
• If the problem occurs in interaction with other systems (e.g. SCM), include their version and other relevant configuration.
• If the Jenkins UI no longer responds, get a thread dump using e.g. (on Linux) kill -3 <Jenkins PID> or jstack -l <Jenkins PID>.

After creating the issue

Once you’ve created an issue, make sure to respond to requests for additional information in a timely manner, otherwise your issue may be resolved as Incomplete.

Keep the issue updated.

Training

Yes enough training is available by the vendor on this tool.

Licensing and Support

Open Source/ Commercial

Jenkins is an open source continuous integration tool.

Competitors

Top competitors of Jenkins are:

Codeship: Codeship as a product is very targeted and provides an easy way to get simple CI up and running. It hooks into GitHub very nicely, and sports a Heroku integration as well

AccuRev: Ability to effectively track versions and changes, Ability to use 3rd party comparison tools for Diff and Merge

TeamCity:Github integration is seamless and extremely intuitive. I work in a small team of developers and our forked repositories do not count towards the number of repositories our plan supports.

AppVeyor: Once our build configuration was set up, I could just forget about it, and it just works. Aside from that, support is great–fast, and helpful.

Ansible: LIGHT WEIGHT / AGENT-LESS Ansible uses ssh and WinRM to communicate with servers, this means it is agent-less and lightweight and reduces your start-up time for new instances, and one less component to manage/update/upgrade.

Rational ClearCase: I really like that it allows dynamic and snapshot models for work. and the check ins could be reserved or unreserved, so I am not blocking anyone.

Microsoft Team Foundation Server: TFS is very easy to use and participate with. Team Members can easily pick up what they need to do. With Visual Studio, the integration with Microsoft Office and Project are great.

Bamboo:  Fortunately most of what I do is easy. It’s harder to do harder things but in most cases it is possible to work within the constraints of the system. The documentation is pretty good once you figure out where it is. Support is excellent.

MidVision

Cantata Functionality

Suitability

Cantata and Cantata++ were originally developed by IPL Information Processing Systems until 2012 when QA Systems GmbH acquired the business, and founded QA Systems Ltd in Bath (UK). In November 2012 the tool was rebranded at version 6.2 as Cantata. Until 2014 there have been 19 versions released since Cantata (for C) its release in 1992.The IDE of Cantata is based on Eclipse.

Accurateness

Cantata belongs to a category of code-driven unit testing frameworks used for dynamic testing of software. These tools bring compiled test cases to execution with the linked-in code under test.

Interoperability

The test executable will run on a computing platform, a native operating system, or a target platform. The integration of debuggers is optional. Typical users of Cantata are either developers of application software programmed in C and C++, or ofsystem software designed for embedded systems and mobile devices.

Compliance

The development and verification of functional safety software in many industries is governed by international regulatory standards. Therein requirements can be found which relate to functional Safety and how to ensure a certain Safety Integrity Level (SIL). Each of the standards below highly recommends or mandates the use of unit testing and integration testing with a code-driven testing framework:

• Aerospace (RTCA: DO-178B and DO-178C : Software Considerations in Airborne Systems and Equipment Certification)
• Automotive (ISO 26262: Road vehicles – Functional safety)
• Energy/Nuclear Power (IEC 60880 : Nuclear power plants – instrumentation and control systems important to safety – Software aspects for computer based systems performing category A functions)
• Industrial Automation (IEC 61508 : Functional Safety of Electrical/Electronic/Programmable Electronic Safety-related Systems)^[2]
• Medical Devices (IEC 62304 : Medical device software – Software lifecycle processes)^[8][9]
• Railways (EN 50128 & EN 50129:^[10] Railway applications – Communications, signalling and processing systems – Software for railway control and protection systems)

Security

Cantata may be used in each of these sectors to meet the verification requirements of the regulatory standard.

Reliability

Maturity

The Cantata tool has been independently classified and certified by the functional safety certification body SGS-TÜV GmbH, as “usable in the development of safety related software”^[11] to the highest safety integrity level in each of the above standards.The mapping of verification and validation requirements in each of the above standards for functional software testing,robustness testing and structural testing (code coverage) to the capabilities of Cantata are available from the vendor, together with tool certification kits for IS0 26262, IEC 60880, IEC 61508, IEC 62304 and EN 50128, and tool qualification kits for DO-178B and DO-178C/DO-330.

Fault Tolerance

NEBS(TM) Level 3-certified server provides affordable fault tolerance for a broad range of telecommunications applications in mobile, landline, circuit, packet, and converged networks. The powerful combination offers customers unprecedented reliability and continuous availability of mission critical applications in next generation, pre-IMS or IMS architectures.

“Stratus is world-renown for the ‘zero downtime’ of its products,” said Greg Pisano, director of market development for carrier enhanced services, Cantata Technology. “By deploying the SnowShore IP Media Server on the T30 platform, service providers can attain unprecedented reliability for their IMS and pre-IMS media services.”

Recoverability

The tool Certification Kit consists of: independent certification for the latest Cantata versions 6.0-6.2, a detailed Safety Manual, comprehensive standards briefings for the main safety standards, and automated tests demonstrating the correct tool operation on embedded targets.

Cantata has been certified as “usable in development of safety related software”, up to the highest safety integrity levels for all the main safety related standards:

• IEC 61508:2010 (general industrial),
• ISO 26262:2011 (automotive),
• EN 50128:2011 (railways),
• IEC 60880:2006 (nuclear power)
• and IEC 62304:2006 (medical devices).

Certification was performed by SGS-TÜV Saar GmbH, an independent third party certification body for functional safety, accredited by Deutsche Akkreditierungsstelle GmbH (DAkkS).

Portability

Adaptability

Cantata is an advanced tool that uses unit and integration testing to dynamically prove code safety. The Cantata Eclipse user interface provides a complete test development environment for the creation, execution, and analysis of unit and integration tests. It easily integrates with the developer desktop, compilers, and embedded target platforms.

Conformance

Cantata has been proven in all high-integrity sectors, including medical, air-traffic control, avionics, rail transport, automotive, space, defense, and nuclear energy; as well as for business-critical sectors, such as telecom, financial, and digital broadcast systems.

The Cantata Tool Qualification Kit is available free of charge, for customers wishing to qualify the tool themselves for high-integrity standards.

Installation

Thorough software testing at the developer level ensures that devices perform reliably in the field. Software testing has the power to increase your confidence, reduce your risk, and help prevent software failures after shipping. Protect your products, your brand, and the reputation of your company by completing thorough software testing.

Unit testing and low-level integration testing are the earliest tests performed during software development. This is the least expensive time to fix defects because it is before they enter the wider code stream or, far worse, are shipped with the device.

Usability

Understandability

Cantata provides the most flexible toolkit for C/C++ developers to perform unit and integration testing on host or embedded target platforms.Cantata is fully qualifiable for safety-critical use. With the most advanced integrated code coverage analysis available for C/C++ and Java.

Learnability

Cantata boasts the following features:

• Static analysis metrics provide help with planning the most efficient tests.
• A powerful built-in EDG C/C++ parser generates a detailed understanding of code, for automatic generation of test cases as thorough or light-touch as required.
• Preferences for the appearance of all views, reports, and installation details can be saved in user-specific configurations.
• Test and analysis generation options for test script layout, source language, memory, global data, and testability instrumentation can all be user-configured.
• Test harness automates testing as appropriate.
• Unique testability instrumentation and wrapping provide precision test control and verification.
• Integrated code coverage and test optimization pinpoints gaps and efficiently directs test efforts.
• Results anddiagnostics are used for rapid error-identification and to support certification.
• Automated repeatable execution through CantataMakefiles retains value of tests whenever code is built.

Ease of Use

Using the Cantata IDE: To generate Cantata Baseline tests for a set of source code files, follow the usual instructions for creating a project and generating Test Scripts and select the “Baseline” option on the first page of Test Script Wizard. The tests will be generated according to the call interface control options specified in the wizard and the Code Generation preferences for the workspace. Alternatively, test cases can be added to an existing Test Script by choosing the “Baseline test” option when adding test cases in the Test Script Manager.

Using the Command Line: For larger code bases, it may be more convenient to run Cantata Baseline test generation from the command line. Invoking this with an existing build system is as simple as prefixing the compile command with the Cantata options. Code maybe selected by source directories, files and functions, with progress information displayed in the console.

Licensing:Cantata Baseline Testing is licensed using a separate license feature (CANTATAPP_BASELINE_TEST). If this feature is not available the option to automatically generate tests from source code will be disabled in the IDE and the command line tool will report an error.

Efficiency

Testing Time

The Cantata Baseline Tests Generation Report contains summary information on:

> Hostname (the machine where the tests were generated) Note that links contained in the report require the machine where the tests were generated

> Cantata code coverage Rule Set used to define the required code coverage metrics and targets for a passing test

> When the report was generated

> Cantata version used

> The Configuration Section used to define the deployment of Cantata used

> The time taken to generate the tests

Result Publishing

Summary information is also produced for the Files and Functions from which the Cantata Baseline tests were generated:

> Fully Tested Files / Functions Counts and % of files / functions where the generated tests will exercise 100% of the coverage metric types

> Partially Tested Files / Functions Counts and % of files / functions where the generated tests will exercise less than 100% of the coverage metric types

> Untested Files / Functions Counts and % of files / functions where no tests were generated

Detailed Information: The Cantata Baseline Tests Generation Report contains the following information for Fully Tested, Partially Tested and Untested code:

> Source file

> Function

> Messages

> Time Taken to generate the test

Vendor Support

Email Support

Cantata provides maintenance and support services to its client helping them manage the product through its lifecycle. The nature of the services differs with the needs of each product segment. For discrete products, services include support through manufacturing in the form of engineering change management, technical documentation, training and managing after-sales maintenance and repairs.

For software products, the services are comprehensive including training, turn-on and monitoring, L1-Call Center, L2-Technical support and L3-Support services.

Benefits that can be realized are timely, ongoing support which is crucial to the success of any project and access to reliable technical support. We guarantee swift response to your issues through support facilities like the Global Support Center, regional support centers and even onsite support.

Phone Support

L1 – Support Services

Cantata provides you with L1-Call support / Help desk services that will provide the ability to handle product queries related to features, pricing, support, licensing, clarifications, ability to receive orders and provide status on the same, manage the spares and shipment.

L2 – Support Services

Cantata can provide L2-Technical Support Services that includes problem analysis & replication, problem resolution via work-around or escalation to L3 support or to your team for further action, etc. An offshore lab equipped with all the devices, tools and infrastructure is a key to ensuring that the L2 technical services we provide are effective.

User’s Group

As part of our product sustenance service we provide L-3 sustenance service of bug-fix, minor enhancements, maintenance testing and release management needs for your product .

Training

Yes, training which is available is sufficient.

Licensing and Pricing

Open Source/ Commercial

This Cantata is a commercial tool.Where a registered target deployment records any limitations of the Cantata capabilities imposed bythat target platform (e.g. lack of timing functions to perform timing analysis), the verificationobjectives for any test undertaking using that Cantata target deployment, must recognize suchconstraints and alternative verification strategies employed.

Competitors

The biggest competitor of Cantata tool is CrunchBase.

Cruisecontrol

Introduction

I am going to talk to you about continuous integration; some people prefer the term continuous build. I will not be going into depth about integrating entire systems but will concentrate on the integration of individual classes and assemblies that make up one distinct application.

CruiseControl.net is the application which I will be discussing in particular. At its most basic, it can be used simply to create automatic builds which can at least prove that the application will compile on a development server and not just on the developer’s own desktop.

Setting up continuous integration for a project does take some time and effort but the end result is that you can be much lazier: just let the automated process do it all for you.

• Free and open source build scheduler
• Consists of two components:
  • Build loop (runs as a service/daemon)
  • Status Dashboard (runs as web application)
  • Works with many version control systems (CVS, PVCS, Subversion, ClearCase, …)
  • Works with any build tool that produces parseable output (Ant, Maven, …)
  • Cross platform (written in Java)
  • Easily extensible:
    • Configuration easily extended by custom plugins
    • Well defined interfaces for version control systems, build tools and publishers
    • Well supported:
      • Complete reference documentation
      • Active users mailing list
      • Numerous tutorials
      • “Pragmatic Project Automation” book by Mike Clark

Background

So why would you want to implement continuous integration in the first place? Well, it eliminates many of the concerns regarding integrating code that has been created by individual developers and then has to be integrated into the final application. By building the application on a clean machine, any missing files or dependencies will be instantly recognised as these will cause the build to fail. Continuous integration need not stop at just building the software. It can also automate unit testing (e.g. Nunit), code coverage (e.g. Ncover) which indicates the amount of code that is actually being tested by unit test, documentation (e.g. Sandcastle) and standards compliance (e.g. fxCop) amongst others.

Now you can begin to see the real value of this automation. Developers can feel much more assured that the code they check into source control will build and, provided that it has adequate code coverage, it doesn’t break any other code in the process.

I must point out that this process does not test the display of the GUI, it only tests the functionality. As such, it cannot replace all manual testing but it does cover a great deal.

Using the Code

N.B. Names of servers, UNC paths and URLs have been disguised throughout this article for reasons of security. Likewise the paths in the example files are fictional. You will need to replace references to “myServer” and “MyDirectory” as appropriate to your installation.

CruiseControl.net

TeamCity by JetBrains is great for large scale projects where you need the ability to allow many teams to modify the configuration of the Build engine. However, as this is an introduction to continuous integration, I have chosen to use CruiseControl.net, it is free and easy to use.

The Web Dashboard is an ASP.NET application running on IIS. It provides you with details about all your builds including reports and stats on individual builds. See screen grab below:

Here is an example of a unit testing stats page:

Here is a build report:

Cruisecontrol Tray

CruiseControl.net comes with a client side application which is shown in the sys tray and can be viewed via a double click on the icon:

Here is the application:

OK, in this window you will see four separate builds. The bottom three builds have been created for just one application. I will discuss these three as they are the most interesting.

SSE Main Build simply builds the application in its entirety. It runs every 30 seconds looking for any new code that has been checked into source control.

SSE-NightlyBuild creates a version of the application that is the latest build which can be run on the server itself. It also performs all unit tests. This runs at 1am every day.

SSE-Documentation simply creates all the documentation from the comments.xml files created when each assembly is built. This relies on the developer adequately commenting his/her code using XML comments and predefined tags preceding any method or property.

You can choose to view as many builds as you wish and they do not have to be on the same CruiseControl.net server. To do this, select Settings by right clicking on the CCTray icon:

Now you can add a CruiseControl.net server and any builds it has that you wish to monitor.

If any or all of the builds are currently building, the icon changes colour to orange. When a build is complete, the icon is either green for successful or red when there has been an error. The same colour keys are used in the Cctray window itself.

Nant / MsBuild

In order to create these builds, you will need to write some build scripts. I have used nant Version 0.85. This is not the latest version but I encountered some problems with a later version and as this one does the job, I’ve stuck with it.

Please take a look at SSE.build (included file) You will notice that I have included building the comments.xml files for each target. Also, the .NET assembly named UnitTests is called separately. This is because the project is not included in the release build of the solution but in the debug build.

I also encountered a problem with registry keys whilst using msbuild. This is the workaround:

Edit the registry adding sdkInstallRootv2.0 to

LocalMachine\SOFTWARE\Microsoft\.NETFramework\sdkInstallRoot

and point it to the folder where the sdk exists on that computer e.g.

c:\windows\Microsoft.net\Framework\v2.0.50727\

Copy lc.exe to the bin subdirectory of the sdkInstall folder if it isn't there already

N.B. There is no need to do the above unless you encounter problems.

Sandcastle

Sandcastle is Microsoft’s tool to automate the documentation of classes by creating a *.chm file and MSDN like pages.

In order to build documentation for all the assemblies, you will need to edit the Sandcastle.config file adding additional data files elements. Please see lines 74-93 in the sandcastle.config file.

Next you need to write a nant script to call Sandcastle. This is quite complex so I have included the exampleSSE_Documentation.build.

Nunit

I am a fan of unit tests, they can be a bit of a chore to create but I consider them as necessary as the use of source control and most developers would agree that would be hard to think of life without that.

Apart from simply making sure that your code does what it says on the tin, unit testing can be very useful when you wish to test your application in isolation from any other system that it communicates with. I have used NUnit to test this application without the need of the server. It also does the tests much more swiftly as I am reading from local files containing dummy data rather than waiting for the results of SOAP calls to the server.

Principals of Unit Testing

Atomic tests – each test should test one property or method only and with one and one only expected result. A new test should be made for each method or property with additional tests for the boundaries of each.

Test cases – unit tests can be written to prove test cases. These are tests that are based on specific use cases. For example, if a use case consisted of a user logging into an application viewing some data and logging off, then the test case should follow these same steps.

I have chosen to put all my unit tests in one assembly as this makes the test automation very easy, however you might choose to create separate assemblies for each logical tier or layer of your application. This adds a bit to the automation but it does mean that your components remain discrete and developers can run the tests independently.

The nunit framework comes with its own MockObject but doesn’t support things like events and generics. You could take a look at Rhino Mocks instead. I haven’t used it but it gets good reviews.

Test driven development is where you write the unit tests first and then the code to be tested after. This focuses your mind on the exact details of regarding the purpose of each method and it can help to prevent the creation of unneeded code. It does take a bit of a change of mindset and the problem is trying to get back into it after doing some bug fixing that doesn’t require a unit test.

Maybe it is optimistic to imagine that a developer will have time to write unit tests for existing code. Visual Studio 2005 Team edition does provide a Unit test wizard which will create the basis of the unit tests, so lightening the work load. Maybe there is some other open source application out there that does something similar.

As there are already so many articles around on unit testing and NUnit in particular, I do not see the value of going into detail here. Suffice it to say that automating your tests makes sure that they do get run regularly.

You will see that Nunit is called directly from CruiseControl.net itself using the tag <nunit> as a child of <tasks> in the SSE-NightlyBuild project in the file ccnet.config.

Tasks Needed to Get CruiseControl.net Working on a New Server

This presumes that you are using a Windows server:

CruiseControl.net

html workshop

nant

nunit

msbuild

perforce P4V

Sandcastle

Getting the WebDashBoard Working with IIS

Make sure that ASP.NET is installed on server: Go to add/remove programs, add/remove Windows components, select Application Server, then click on Details. Make sure that ASP.NET is ticked.

 If you installed IIS after the .NET Framework was installed, you need to register ASP.NET with IIS. You can do this by running the aspnet_regiis.exe -i tool found under \Windows\Microsoft.NET\Framework\v2.0.50727\.

In IIS Admin, go to Web Service Extensions and make sure that ASP.NET v2.0.* is allowed and any other version prohibited.

Difference between Cruisecontrol and Jenkis:

CruiseControl is a framework for a continuous build process. It includes, but is not limited to, plugins for email notification, Ant, and various source control tools. A web interface is provided to view the details of the current and previous builds.

As a long time CruiseControl committer and someone who has never used Hudson I’m pretty biased, but my take on it is:

Hudson is much easier to get up and running (in large part from a nice web interface) and has a very active plugin development community.

CruiseControl has support from lots of 3rd party stuff and has the benefit of doing some neat tricks with the xml configuration like plugin preconfiguration and include.projects which lets you version the configuration information with the project.

If you’re only going to have a few builds I think Hudson is the clear winner. If you’re going to have lots — and don’t mind the xml — then I think CruiseControl’s xml configuration tricks become a real strength.

In short, Hudson (update: Jenkins) is likely the better choice now. First and foremost because creating and configuring jobs (“projects” in CC vocabulary) is just so much faster through Hudson’s web UI, compared to editing CruiseControl’s XML configuration file (which we used to keep in version control just to keep track of it better). The latter is not especially difficult – it simply is slower and more tedious.

CruiseControl has been great, but as noted in Dan Dyer’s aptly-named blog post, Why are you still not using Hudson?, it suffers from being first. (Um, like Britain, if you will, later into the industrial revolution, when others started overtaking it with newer technologies.)

We used CruiseControl heavily and have gradually switched over to Hudson, finally using it exclusively. And even more heavily: in the process we’ve started using the CI server for many other things than before, because setting up and managing Hudson jobs is so handy. (We now have some 40+ jobs in Hudson: the usual build & test jobs for stable and development branches; jobs related to releasing (building installers etc); jobs that run some (experimental) metrics against the codebase; ones that run (slow) UI or integration tests against a specific database version; and so on.)

From this experience I’d argue that even if you have lots of builds, including complicated ones, Hudson is a pretty safe choice because, like CC, you can use it to do anything, basically. Just configure your job to run whatever Ant or Maven targets, Unix shell scripts, or Windows .bat scripts, in the order you wish.

As for the 3rd party stuff (mentioned here by Jeffrey Fredrick) – that is a good point, but my impression is that Hudson is quickly catching up, and that there’s already a very large number of plugins availablefor it.

For me, the two things I can name that I miss about CruiseControl are:

1. Its warning emails about broken builds were more informative than those of Hudson. In most cases the root cause was evident from CC’s nicely formatted HTML mail itself, whereas with Hudson I usually need to follow the link to Hudson web UI, and click around a little to get the details.
2. The CruiseControl dashboard is better suited, out of the box, as an “information radiator” (shown on a public monitor, or projected on a wall, so that you can always quickly see the status of all projects). With Hudson’s front page, we needed some Greasemonkey tricks to get job rows all nicely green/red.

Minor disclaimer: I haven’t been following the CC project closely for the last year or so. (But from aquick look, it has not changed in any dramatic way.)

Note (2011-02-03): Hudson has been renamed/forked as Jenkins (by Hudson creator Kohsuke Kawaguchi and others). It looks as if Oracle—which controls the Hudson name—will keep “Hudson” around too, but my personal recommendation is to go with Jenkins, no matter what Oracle says.

The things I liked about Hudson:

• The upstream and downstream projects. So a commit to your data access code will eventually also trigger a build of the presentation layer.
• Easily use an existing project as the starting point of a new one – so if you are in the habit of creating development branches, then making sure these are under continuous integration is a snap.

One difference is that Hudson is the product of a single genius intellect—Kohsuke Kawaguchi. Because of that, it’s consistent, coherent, and rock solid. The downside could be some limitation on the rate of progress. However, Kohsuke is incredibly prolific, so I wouldn’t be too worried about that. And, it’s extensible, so if there’s something Kohsuke doesn’t have time for (or doesn’t want), you can probably do it yourself.

Jenkins with Selenium

• When your test cases are ready and you want to handover to client or manual testing team so that they can trigger these test cases using single click then Jenkins is the best choice.
• Using Jenkins we can create build (Build – set of Test case combined together) and we can run easily using batch file or Git or build.xml or SVN etc.
• In Jenkins  we can schedule the build periodically

Example- You want to run 100 Test case daily at 10 pm then Jenkins will take care      of this based on our pattern it will trigger your build

• Email-Notification- Jenkins provide notification mails too once build passed or failed to respective recipients (Depends of configuration)

Before starting if you have be having Eclipse , TestNG and if you are using Excel sheets or csv then these jars should be ready.

What is Selenium

As a marque tool in open source test automation space, QA decision makers always feel good if something comes to them as a freebie especially in the cost consuming test automation space. Selenium has the open source edge and moreover it supports a variety of languages that include Java, Python, PHP, C#, Ruby and even pure JavaScript. Selenium is at present the most powerful Open Source Automation tool available and it is based on java scripting to a large extent. It is more suited for the agile methodology of coding and testing.

• Javascript framework that runs in your webbrowser

• Works anywhere Javascript is supported

• Hooks for many other languages

• Java, Ruby, Python

• Can simulate a user navigating through pages and then assert for specific marks on the pages

• All you need to really know is HTML to start using it right away

Then HIT SUBMIT

Executing the Test Suite

• Selenium Core is a collection of Javascript and HTML with iFrames

• Due to security concerns Core must be deployed within the same server as the application being hosted

• The simplest way to run Pixory is to just run the Java application and let it use its own server

• Problems using Core with Pixory

• Selenium IDE is a plug-in for Firefox and thus can go around these restrictions

Running the Test Suite

• We basically want to execute the test suite using the Selenium IDE plug-in TestRunner.html

chrome://selenium-ide/content/selenium/TestRunner.html? baseURL=&test=file:///&auto=true

chrome://selenium-ide/content/selenium/TestRunner.html? baseURL=http://localhost:8081&test=file:///Users/ms333/ projects/classes/running/v_and_v/hw3/selenium/test/ TestSuite.html&auto=true

Selenium is a portable software testing framework for web applications. Selenium provides a record/playback tool for authoring tests without learning a test scripting language (Selenium IDE). It also provides a test domain-specific language (Selenese)to write tests in a number of popular programming languages, including Java, C#, Groovy, Perl,PHP, Python and Ruby. The tests can then be run against most modern web browsers. Selenium deploys on Windows,Linux, and Macintosh platforms. It is open-source software, released under the Apache 2.0 license, and can be downloaded and used without charge.

Within the software industry, once a product reaches the stable manual testing phase, every organization usually thinks of automated testing to save on the costs involved in manual testing. Since testing cost is an important factor for any project, organizations have started preferring open source test automation tools (which have reached a stage where they now rival the commercial ones) instead of investing in costly commercial testing tools. With no licensing costs, open source automation testing tools provide competitive features for automating the testing of software applications as well as Web portals.

A variety of open source automation testing tools is available for almost all types of testing such as functional, Web, UAT, regression, performance etc. Because of the extent to which these open source tools have matured, it’s time to think about them and have them in your QA automation kit. There are also various open source tools available to support the different testing types such as White Box Testing (Unit Testing, for e.g., using JUnit) and Black Box Testing (system/regression testing, for e.g., using Selenium, Sahi, Watir, TestMaker, LogiTest, TestGen4J, FitNesse etc).

The scope of this blog is to cover the basic flow for one of these open source Web testing automation tools – Selenium.

History of Selenium

Selenium was originally developed by Jason Huggins in 2004 as an internal tool at ThoughtWorks. Huggins was later joined by other programmers and testers at ThoughtWorks.,^[2] before Paul Hammant joined the team and steered the development of the second mode of operation that would later become ‘Selenium Remote Control’ (RC). The tool was open sourced that year.

In 2005 Dan Fabulich and Nelson Sproul (with help from Pat Lightbody) made an offer to accept a series of patches that would transform Selenium-RC into what it became best known for. In the same meeting, the steering of Selenium as a project would continue as a committee, with Huggins and Hammant being the ThoughtWorks representatives.

In 2007, Huggins joined Google. Together with others like Jennifer Bevan, he continued with the development and stabilization of Selenium RC. At the same time, Simon Stewart at ThoughtWorks developed a superior browser automation tool called WebDriver. In 2009, after a meeting between the developers at the Google Test Automation Conference, it was decided to merge the two projects, and call

In 2008, Philippe Hanrigou (then at ThoughtWorks) made ‘Selenium Grid’, which provides a hub allowing the running of multiple Selenium tests concurrently on any number of local or remote systems, thus minimizing test execution time. Grid offered, as open source, a similar capability to the internal/private Google cloud for Selenium RC. Pat Lightbody had already made a private cloud for ‘HostedQA’ which he went on to sell to Gomez, Inc.

The name Selenium comes from a joke made by Huggins in an email, mocking a competitor named Mercury, saying that you can cure mercury poisoning by taking selenium supplements. The others that received the email took the name and ran with it.

Why Selenium?

Selenium is probably the best option for automated testing of Websites today. It is becoming increasingly popular and it is the first choice of automation testers as well as organizations for automating the testing of Web-based applications for both the GUI as well as the functionality. Selenium can also be used as a unit testing tool for JavaScript. We at Xoriant have been working on Selenium for automating the testing of Web sites developed using AJAX for the pharmaceuticals and travel industries. The following graph shows the popularity of Selenium along with other open source automation testing tools.

Selenium Test Package

Selenium is a package of various test components which consists of the following three major tools. Each one has a specific role in aiding the development of test automation for a Web application.

1. Selenium IDE – A Firefox extension to record test cases and suites.
2. Selenium RC – Used to run tests on different browsers and systems.
3. Selenium Grid – Runs multiple instances of Selenium RC at once.
4. Qualitia and Tellurium – A wrapper for the Selenium engine.

Selenium Modes:

Bases on the components Selenium has following three modes for executing the test cases and test suites:

Record-Playback mode (Selenium IDE)‏:In this mode only Selenium IDE is used to record the test scenarios in terms of test cases in firefox. This is a great way to get started to writing tests and group them together to form the test suite. The recorded tests can be exported to many programming languages so that we can tweak them and put them in the testing framework. The test cases and test suites can be replayed back to check the verifications and validations or sent to Selenium RC or Grid for further tweaking.

Selenium Remote Control (RC) Mode: In this mode Selenium starts multiple browsers (one at a time) and then runs the recorded test-cases which are saved in your language of choice. This helps to enhance the test cases with looping and programming techniques to cover all the required test scenarios and checks.

Test Runner Mode: In this mode the test cases are recorded ad replayed in the form of HTML tables. This is just one more facility to execute the Selenium IDE as well as RC test cases. This helps to check the test results reports in better manner if not formatted already.

Testing AJAX with Selenium

Selenium IDE – Recording and updating a script

Selenium IDE is the FireFox Add-on provided by the Selenium group. This is very simple and easy to use add-on so that non-programmers can record and create the test scripts for automating web components. These automated test scripts are used as Selenium RC test cases by choosing the language code. i.e. Selenium IDE makes easier to create Selenium RC test cases. (Install Selenium IDE and Selenium RC from Selenium download page).

Selenium IDE is used for:

• Recording and updating the test cases or write them manually in table tab.
• Creating Test Suite by grouping the test cases under one group
• Exporting Test Cases/Suites the supported language and save for Selenium RC to enhance them
• Finding reference of every API and Selense commands
• Debugging test cases by toggling breakpoints through the commands

How commands generated in IDE for AJAX pages:

1. Almost all of the web pages are developed with AJAX technology since web2.0 evolution. While recording the page elements, Selenium IDE locates the page elements/objects usingXPath/DOM/CSS. XPaths are found automatically, but you may have to update the specific elements in Xpath manually by looking into the source of the page to make them unique.Firebug is the helpful firefox Add-on for this purpose.
2. Every command has maximum two arguments. The first is usually target element pointed by its ID, name, link, DOM, CSS or XPath and the second is the value to that target element if exists. The selenium IDE commands when saved in the programming language of our choice for selenium RC purpose are called as Selenese.
3. The common commands are open, click, type, assert, verify. Commands beginning with ‘assert’ or ‘verify’ are for checks and verifications. If verify command is used and it fails, it fails the test case and continue to run the remaining tests. If assert command is used and it fails, the test case is aborted and stops to run the remaining tests.
4. For synchronizing the flow of test execution Selenium has commands ending with ‘AndWait’, it tells Selenium to wait for the page to load after the action has been done. However, it fails when using AJAX calls. In this case you have to use ‘WaitFor’ commands.

Selenium IDE – Most commonly used commands

The following are the most commonly used commands in the Selenium IDE:

open: Opens a page using a URL.

click: Clicks the element/object on the page.

clickAndWait: Performs a click operation, and optionally waits for a new page to load.

verifyTitle: Verifies the expected title and continues to run if it fails.

assertTitle: Verifies an expected page title and stops the execution if it fails.

verifyTextPresent: Verifies that the expected text is present somewhere on the page.

verifyElementPresent: Verifies an expected UI element, as defined by its HTML tag.

verifyText: Verifies that the expected text and its corresponding HTML tag are present on the page.

waitForPageToLoad: Pauses execution until an expected new page loads.

waitForElementPresent:Pauses execution until an expected UI element, as defined by its HTML tag, is present on the page. Used with AJAX calls.

Selenium Components

Selenium IDE

Selenium IDE is a complete integrated development environment (IDE) for Selenium tests. It is implemented as a Firefox Add-On, and allows recording, editing, and debugging tests. It was previously known as Selenium Recorder. Selenium-IDE was originally created by Shinya Kasatani and donated to the Selenium project in 2006.

Scripts may be automatically recorded and edited manually providing autocompletion support and the ability to move commands around quickly. Scripts are recorded in Selenese, a special test scripting language for Selenium. Selenese provides commands for performing actions in a browser (click a link, select an option), and for retrieving data from the resulting pages.

Selenium client API

As an alternative to writing tests in Selenese, tests can also be written in various programming languages. These tests then communicate with Selenium by calling methods in the Selenium Client API. Selenium currently provides client APIs for Java, C#, Ruby and Python.

With Selenium 2, a new Client API was introduced (with WebDriver as its central component). However, the old API (using class Selenium) is still supported.

Selenium Remote Control

Selenium Remote Control (RC) is a server, written in Java, that accepts commands for the browser via HTTP. RC makes it possible to write automated tests for a web application in any programming language, which allows for better integration of Selenium in existing unit test frameworks. To make writing tests easier, Selenium project currently provides client drivers for PHP, Python, Ruby, .NET, Perl and Java. The Java driver can also be used with JavaScript (via the Rhinoengine). A new instance of selenium RC server is needed to launch html test case – which means that the port should be different for each parallel run.However, for Java/PHP test case only one Selenium RC instance needs to be running continuously.

Selenium Remote Control was a refactoring of Driven Selenium or Selenium B designed by Paul Hammant, credited with Jason as co-creator of Selenium. The original version directly launched a process for the browser in question, from the test language of Java, .Net, Python or Ruby. The wire protocol (called ‘Selenese’ in its day) was reimplemented in each language port. After the refactor by Dan Fabulich, and Nelson Sproul (with help from Pat Lightbody) there was an intermediate daemon process between the driving test script, and the browser. The benefits included the ability to drive remote browsers, and the reduced need to port every line of code to an increasingly growing set of languages. Selenium Remote Control completely took over from the Driven Selenium code-line in 2006. The browser pattern for ‘Driven’/’B’ and ‘RC’ was response/request, which subsequently became known as Comet.

With the release of Selenium 2, Selenium RC has been officially deprecated in favor of Selenium WebDriver.

Selenium WebDriver

Selenium WebDriver is the successor to Selenium RC. Selenium WebDriver accepts commands (sent in Selenese, or via a Client API) and sends them to a browser. This is implemented through a browser-specific browser driver, which sends commands to a browser, and retrieves results. Most browser drivers actually launch and access a browser application (such as Firefox or Internet Explorer); there is also an HtmlUnit browser driver, which simulates a browser using HtmlUnit.

Unlike in Selenium 1, where the Selenium server was necessary to run tests, Selenium WebDriver does not need a special server to execute tests. Instead, the WebDriver directly starts a browser instance and controls it. However, Selenium Grid can be used with WebDriver to execute tests on remote systems (see below).

In practice, this means that the Selenium 2.0 API has significantly fewer calls than does the Selenium 1.0 API. Where Selenium 1.0 attempted to provide a rich interface for many different browser operations, Selenium 2.0 aims to provide a basic set of building blocks from which developers can create their own Domain Specific Language. One such DSL already exists: the Watir project in the Ruby language has a rich history of good design. Watir-webdriver implements the Watir API as a wrapper for Selenium-Webdriver in Ruby. Watir-webdriver is created entirely automatically, based on the WebDriver specification and the HTML specification.

As of early 2012, Simon Stewart (inventor of WebDriver), who was then with Google and now with Facebook, and David Burns of Mozilla were negotiating with theW3C to make WebDriver an internet standard. In July 2012, the working draft was released.^[9] As such, Selenium-Webdriver (Selenium 2.0) aims to be the reference implementation of the WebDriver standard in various programming languages. Currently Selenium-WebDriver is fully implemented and supported inPython, Ruby, Java, and C#.

Selenium Grid

Selenium Grid is a server that allows tests to use web browser instances running on remote machines. With Selenium Grid, one server acts as the hub. Tests contact the hub to obtain access to browser instances. The hub has a list of servers that provide access to browser instances (WebDriver nodes), and lets tests use these instances. Selenium Grid allows running tests in parallel on multiple machines, and to manage different browser versions and browser configurations centrally (instead of in each individual test).

The ability to run tests on remote browser instances is useful to spread the load of testing across several machines, and to run tests in browsers running on different platforms or operating systems. The latter is particularly useful in cases where not all browsers to be used for testing can run on the same platform.

The Selenium Ecosystem

Popular tools integrating with Selenium

Below are several notable tools that make it easy to run Selenium tests and view their results:

• Jenkins provides plugins for integrating Selenium test results,and running Selenium Grid.
• Grails provides a plugin for easily adding Selenium tests to a Grails application.
• Apache Maven provides Selenium 2 artifacts in the central Maven repository.
• Visual Studio provides a plugin for running Selenium tests from Visual Studio programs.
• Drupal’s internal testing framework, SimpleTest, provides a Selenium integration.
• Silk Central is a test management platform that allows users to achieve real-time reporting and the ability to run Selenium scripts.
• Soatest is an API Testing tool that can use the Selenium WebDriver framework for Web UI testing, including cross-browser testing.
• Appium is a mobile test automation framework (and tool) for native, hybrid and mobile-web apps for iOS and Android. It uses JSONWireProtocol internally to interact with iOS and Android apps using Selenium’s WebDriver.
• Bstriker ALTM allows users to create, import and play Selenium scripts in multiple browsers.
• TAZ Automation directly runs Selenium IDE commands (scriptless) in a full-functioned Test Execution Manager.
• Protractor end to end testing for AngularJS

Services running Selenium on the cloud

The following are commercial services that allow users to run their Selenium tests on a large number of browsers/OS combinations, on machines running in the cloud. Most services offer the ability to capture screenshots and videos of Selenium tests, and a secure tunnel for testing web applications running on premises behind a firewall. Pricing is based on total time of tests run per month and maximum concurrent tests allowed.

Following are details for several notable Selenium services.

Advantages of Selenium

a) Selenium supports many languages such as Java, C#, ruby, python. It doesn’t support windows based application but is one of the best tools for web-based automation and supports all browsers such as IE, FF, Chrome, Safari etc.

b) Being into open source world, Selenium can integrate with just about anything to give you a robust framework such as Maven or Ant for source code compilation, Test NG to drive tests (unit or functional or integration), Jenkins or Hudson or Cruise Control to integrate into a Continuous Integration and different reports or dashboards out of Jenkins.

 c) Selenium also helps you enter bugs or issues into JIRA (Bug management tool) through Jenkins. With the help of Jenkins or Grid, testers can connect multiple nodes to run different tests in parallel.

d) Cost is the biggest advantage as it is a freeware and is integrated with other opensource tools. As mentioned above, the only cost that you need to spend is on the human resource (single or team) who knows all of the above stuff.

Disadvantage of Selenium

a) Organizations need to invest in a resource who knows coding as per standards. The resource should also be well versed in framework architecture and various components that fit in.

b) Tool support is done by way of communities and more and more people are joining together to contribute to provide solutions and support free-ware. But issues may not be addressed in the next release cycle like any other commercial vendors do routinely. So it can potentially leave Organizations themselves to find a solution on their own and contribute to community or wait for someone to give a solution.

Conclusion

At the end of the day it is as simple between buying a branded car and assembling a car on your own. Branded car has its cost & services attached whereas in assembled car, you need to take care of everything on your own. But, there is a hybrid solution where you can go to a custom workshop to get an assembled car (sales and post-sales support) and pay for their services. The same concept is now available in IT too such as you go to a service provider who has everything i.e. who has developed automation frameworks using different open source components, who has coders ready to put in muscle & tissue to the pre-existing wire-frame and deliver it to you. This concept is the new trend in town and many companies have switched to open source frameworks built by service providers thereby reducing their overall product cost.

References:

• Kaner, Cem (November 17, 2006). “Exploratory Testing” (PDF). Florida Institute of Technology, Quality Assurance Institute Worldwide Annual Software Testing Conference, Orlando, FL. Retrieved November 22, 2014.
• Software Testing by Jiantao Pan, Carnegie Mellon University
• Leitner, A., Ciupa, I., Oriol, M., Meyer, B., Fiva, A., “Contract Driven Development = Test Driven Development – Writing Test Cases”, Proceedings of ESEC/FSE’07: European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering 2007, (Dubrovnik, Croatia), September 2007
• Kaner, Cem; Falk, Jack; Nguyen, Hung Quoc (1999). Testing Computer Software, 2nd Ed. New York, et al: John Wiley and Sons, Inc. p. 480. ISBN 0-471-35846-0.
• Kolawa, Adam; Huizinga, Dorota (2007). Automated Defect Prevention: Best Practices in Software Management. Wiley-IEEE Computer Society Press. pp. 41–43. ISBN 0-470-04212-5.
• Kolawa, Adam; Huizinga, Dorota (2007). Automated Defect Prevention: Best Practices in Software Management. Wiley-IEEE Computer Society Press. p. 426.ISBN 0-470-04212-5.
• Section 1.1.2, Certified Tester Foundation Level Syllabus, International Software Testing Qualifications Board
• Principle 2, Section 1.3, Certified Tester Foundation Level Syllabus,International Software Testing Qualifications Board
• “Proceedings from the 5th International Conference on Software Testing and Validation (ICST). Software Competence Center Hagenberg. “Test Design: Lessons Learned and Practical Implications.”.
• Software errors cost U.S. economy $59.5 billion annually, NIST report
• McConnell, Steve (2004). Code Complete (2nd ed.). Microsoft Press. p. 29.ISBN 0735619670.
• Bossavit, Laurent (2013-11-20). The Leprechauns of Software Engineering–How folklore turns into fact and what to do about it. Chapter 10: leanpub. see D. Gelperin and W.C. Hetzel
• Myers, Glenford J. (1979). The Art of Software Testing. John Wiley and Sons.ISBN 0-471-04328-1.
• Company, People’s Computer (1987). “Dr. Dobb’s journal of software tools for the professional programmer”. Dr. Dobb’s journal of software tools for the professional programmer (M&T Pub) 12 (1–6): 116.
• Gelperin, D.; B. Hetzel (1988). “The Growth of Software Testing”. CACM 31 (6): 687–695. doi:10.1145/62959.62965. ISSN 0001-0782.
• Gelperin, D.; B. Hetzel (1988). “The Growth of Software Testing”.CACM 31 (6). ISSN 0001-0782.
• Gelperin, D.; B. Hetzel (1988)– From 1957–1978 there was the demonstration oriented period where debugging and testing was distinguished now – in this period it was shown, that software satisfies the requirements. ISSN 0001-0782
• Gelperin1988 — The time between 1979–1982 is announced as the destruction oriented period, where the goal was to find errors. ISSN 0001-0782.
• Introduction, Code Coverage Analysis, Steve Cornett
• Ron, Patton. Software Testing.
• Laycock, G. T. (1993). “The Theory and Practice of Specification Based Software Testing” (POSTSCRIPT). Dept of Computer Science, Sheffield University, UK. Retrieved 2008-02-13.
• Bach, James (June 1999). “Risk and Requirements-Based Testing” (PDF).Computer 32 (6): 113–114. Retrieved 2008-08-19.
• Savenkov, Roman (2008). How to Become a Software Tester. Roman Savenkov Consulting. p. 159. ISBN 978-0-615-23372-7.
• “Visual testing of software – Helsinki University of Technology” (PDF). Retrieved 2012-01-13.
• “Article on visual testing in Test Magazine”. Testmagazine.co.uk. Retrieved2012-01-13.
• Patton, Ron. Software Testing.
• “SOA Testing Tools for Black, White and Gray Box SOA Testing Techniques”. Crosschecknet.com. Retrieved 2012-12-10.
• “SWEBOK Guide – Chapter 5”. Computer.org. Retrieved 2012-01-13.
• Binder, Robert V. (1999). Testing Object-Oriented Systems: Objects, Patterns, and Tools. Addison-Wesley Professional. p. 45. ISBN 0-201-80938-9.
• Beizer, Boris (1990). Software Testing Techniques (Second ed.). New York: Van Nostrand Reinhold. pp. 21,430. ISBN 0-442-20672-0.
• Clapp, Judith A. (1995). Software Quality Control, Error Analysis, and Testing. p. 313. ISBN 0815513631.
• Mathur, Aditya P. (2008). Foundations of Software Testing. Purdue University. p. 18. ISBN 978-8131716601.
• IEEE (1990). IEEE Standard Computer Dictionary: A Compilation of IEEE Standard Computer Glossaries. New York: IEEE. ISBN 1-55937-079-3.
• Whitepaper: Operational Acceptance – an application of the ISO 29119 Software Testing standard. May 2015 Anthony Woods, Capgemini
• Paul Ammann; Jeff Offutt (2008). Introduction to Software Testing. p. 215 of 322 pages.
• Veenendaal, Erik. “Standard glossary of terms used in Software Testing”. Retrieved 4 January 2013.
• ISO/IEC/IEEE 29119-1:2013 – Software and Systems Engineering – Software Testing – Part 1 – Concepts and Definitions; Section 4.38
• “Globalization Step-by-Step: The World-Ready Approach to Testing. Microsoft Developer Network”. Msdn.microsoft.com. Retrieved 2012-01-13.
• EtestingHub-Online Free Software Testing Tutorial. “e)Testing Phase in Software Testing:”. Etestinghub.com. Retrieved 2012-01-13.
• Myers, Glenford J. (1979). The Art of Software Testing. John Wiley and Sons. pp. 145–146. ISBN 0-471-04328-1.
• Dustin, Elfriede (2002). Effective Software Testing. Addison Wesley. p. 3.ISBN 0-201-79429-2.
• Marchenko, Artem (November 16, 2007). “XP Practice: Continuous Integration”. Retrieved 2009-11-16.
• Gurses, Levent (February 19, 2007). “Agile 101: What is Continuous Integration?”. Retrieved 2009-11-16.
• Pan, Jiantao (Spring 1999). “Software Testing (18-849b Dependable Embedded Systems)”. Topics in Dependable Embedded Systems. Electrical and Computer Engineering Department, Carnegie Mellon University.
• Rodríguez, Ismael; Llana, Luis; Rabanal, Pablo (2014). “A General Testability Theory: Classes, properties, complexity, and testing reductions”. IEEE Transactions on Software Engineering 40 (9): 862–894. doi:10.1109/TSE.2014.2331690.ISSN 0098-5589.
• Rodríguez, Ismael (2009). “A General Testability Theory”. CONCUR 2009 – Concurrency Theory, 20th International Conference, CONCUR 2009, Bologna, Italy, September 1–4, 2009. Proceedings. pp. 572–586. doi:10.1007/978-3-642-04081-8_38. ISBN 978-3-642-04080-1.
• IEEE (1998). IEEE standard for software test documentation. New York: IEEE.ISBN 0-7381-1443-X.
• Kaner, Cem (2001). “NSF grant proposal to “lay a foundation for significant improvements in the quality of academic and commercial courses in software testing””
• Kaner, Cem (2003). “Measuring the Effectiveness of Software Testers”
• Black, Rex (December 2008). Advanced Software Testing- Vol. 2: Guide to the ISTQB Advanced Certification as an Advanced Test Manager. Santa Barbara: Rocky Nook Publisher. ISBN 1-933952-36-9.
• “American Society for Quality”. Asq.org. Retrieved 2012-01-13.
• “context-driven-testing.com”. context-driven-testing.com. Retrieved 2012-01-13.
• “Article on taking agile traits without the agile method”. Technicat.com. Retrieved 2012-01-13.
• “We’re all part of the story” by David Strom, July 1, 2009
• IEEE article about differences in adoption of agile trends between experienced managers vs. young students of the Project Management Institute. See alsoAgile adoption study from 2007
• Willison, John S. (April 2004). “Agile Software Development for an Agile Force”. CrossTalk (STSC) (April 2004). Archived from the original on October 29, 2005.
• “IEEE article on Exploratory vs. Non Exploratory testing” (PDF). Ieeexplore.ieee.org. Retrieved 2012-01-13.
• An example is Mark Fewster, Dorothy Graham: Software Test Automation.Addison Wesley, 1999, ISBN 0-201-33140-3.
• Tran, Eushiuan (1999). “Verification/Validation/Certification”. In Koopman, P.Topics in Dependable Embedded Systems. USA: Carnegie Mellon University. Retrieved 2008-01-13.
• “Selenium Commands – “Selenese””. Selenium Documentation. Retrieved September 9, 2011.
• “Selenium History”. Selenium HQ. Retrieved June 29, 2014.
• “The Selenium Project”. NewCircle. Retrieved June 29, 2014.
• “Relationships between different versions of Selenium”. Yi Zeng. Retrieved June 29, 2014.
• “Selenium WebDriver”. Simon Stewart, the Architecture of Open Source Applications. Retrieved June 29, 2014.
• Krill, Paul (April 6, 2011). “Open source Selenium web app test suite to support iPhone and Android”. InfoWorld. Retrieved May 9, 2012. Selenium was so named because Huggins, dissatisfied with testing tools on the market, was seeking a name that would position the product as an alternative to Mercury Interactive QuickTest Professional commercial testing software. The name, Selenium, was selected because selenium mineral supplements serve as a cure for mercury poisoning, Huggins explained.
• “Selenium History”. Selenium HQ. Retrieved March 3, 2015.
• “Selenium Recorder 0.6 Released”. Shinya Kasatani. Retrieved March 3, 2015.
• “WebDriver: W3C Working Draft 10 July 2012”. w3.org. Retrieved February 22, 2015.
• “Selenium Ecosystem”. Selenium HQ. Retrieved July 2, 2014.
• “Seleniumhq Plugin”. Jenkins Wiki. Retrieved July 2, 2014.
• “Selenium Plugin (Selenium Grid)”. Jenkins Wiki. Retrieved July 2, 2014.
• “Selenium Plugin”. Grails.org. Retrieved July 2, 2014.
• “Maven Information”. Selenium HQ. Retrieved July 2, 2014.
• “Selenium components for Coded UI Cross Browser Testing”. MSDN Visual Studio Gallery. Retrieved July 2, 2014.
• “Download & Extend – Selenium”. Drupal.org. Retrieved July 2, 2014.
• http://www.borland.com/Products/Software-Testing/Test-Management/Silk-Central. Retrieved November 26, 2014.
•  “API Testing”. Retrieved April 3, 2015.
• “The Basics Of Test Automation For Apps, Games And The Mobile Web”. SmashingMagazine. Retrieved January 21, 2015.
• “Bstriker International BV”. Retrieved March 7, 2015.
• “Aztaz Software”. Retrieved May 14, 2015.
• “Browsers & Mobile Devices for Selenium Testing”. BrowserStack. Retrieved July 2, 2014.
• “Local Testing”. BrowserStack. Retrieved July 2, 2014.
• “Browsers & Mobile Devices for Selenium Testing – Apple & iOS”. BrowserStack. Retrieved July 2, 2014.
• “Device/OS/Browser Platforms”. Sauce Labs. Retrieved July 2, 2014.
• “Sauce Connect”. Sauce Labs. Retrieved July 2, 2014.
• “Build exceptional mobile apps”. Sauce Labs. Retrieved July 2, 2014.
• “Selenium on Spoonium”. Spoonium. Retrieved July 2, 2014.
• “Device/OS/Browser Platforms”. Testdroid. Retrieved September 30, 2014.
• “Testdroid Cloud API”. Testdroid. Retrieved September 30, 2014.
• “Appium,Selenium,Testdroid”. Testdroid. Retrieved September 30, 2014.
• “List of available browsers”. Equafy. Retrieved October 11, 2014.
• “Equafy Tunnel”. Equafy. Retrieved October 11, 2014.
• “Mobile Testing”. Equafy. Retrieved October 11, 2014.
• “List of available browsers”. TestingBot. Retrieved July 2, 2014.
• “TestingBot Tunnel”. TestingBot. Retrieved July 2, 2014.
• “Mobile Testing”. TestingBot. Retrieved July 2, 2014.
• “List of available browsers”. Gridlastic. Retrieved September 17, 2014.
• Jenkins TestLink Plug-in [http://wiki.jenkins-ci.org/display/JENKINS/TestLink+Plugin]. Copyright © 2011 The Jenkins TestLink Plug-in team.
•  Jenkins-CI [http://www.jenkins-ci.org/]. Copyright © 2011 Jenkins-CI.
• TestLink [http://www.teamst.org/]. Copyright © 2011 TestLink Community.
• TestNG [http://www.testng.org/]. Copyright © 2011 TestNG.
•  JUnit [http://www.junit.org/]. Copyright © 2011 JUnit.
• Test Anything Protocol Website [http://testanything.org/]. Copyright © 2011 Test Anything Protocol Community.
•  Test Anything Protocol article in Wikipedia [http://en.wikipedia.org/wiki/ Test_Anything_Protocol]. Copyright © 2011 Wikipedia.

 YAML.org [http