Master Thesis Title: Evaluation of Software Testing Tools ( Jenkins, CruiseControl, Selenium, Cantata)

== This thesis should evaluate all the above 4 testing tools according to some evaluation criteria.  I am attaching an evaluation criteria that has been approved by my examiner. So the evaluation criteria is already designed. The evaluation criteria can be fit into the thesis report. The evaluation should strictly cope with each of the criteria and sub critiera. How to get the results for each criteria and sub-criteria for each tool is given as: 1. Install Jenkins (Continuous Integration Server) and play with it to get the required results from it and then write it down under each criteria and sub-criteria. 2. Install CruiseControl(Continuous Integration Server) and play with it to get the required results from it and then write it down under each criteria and sub criteria for CrusieControl 3. Install Selenium and do the same. 4. Install Cantata (Automated tool for unit testing of C/C++ language codes) and do the same as above.  There should be one separate chapter for Results where the each tool is compared with the other tool according to the evaluation results. After the results there should be discussion and Conclusion. and Finally the references.  Please note that this will not be possible without the installation of all the four tools and therefore no evaluation without the tool installation will be accepatiable. The examiner should see that the tools have been installed for gettting the results. Please let me know via email if I need to give further information.

Websites : Jenkins = https://jenkins-ci.org/

                    CruiseControl = http://cruisecontrol.sourceforge.net/

                    Selenium = http://www.seleniumhq.org/

                    Cantata = http://www.qa-systems.com/cantata.html

• Software Testing and Continuous Integration
• ··  60 pages, 15000 words Revised Sweden
• Harvard Revised 120 Refers
• (+46) ***********
• Jul 15, 2015 | Wednesday 06:28 am | 15 days 1 hrs ago

Table of Contents

What is Software Testing. 5

What is Unit Testing. 6

Jenkins Functionality. 6

Suitability. 7

Accurateness. 7

Interoperability. 7

• Building/testing software projects continuously. 7
• Monitoring executions of externally-run jobs. 7

Compliance. 7

1.      Easy installation. 7

2.      Easy configuration. 7

3.      Change set support 7

4.      Permanent links. 7

5.      RSS/E-mail/IM Integration. 7

6.      After-the-fact tagging. 7

7.      JUnit/TestNG test reporting. 7

8.      Distributed builds. 7

9.      File fingerprinting. 7

10.        Plugin Support 8

Security. 8

Portability. 8

Adaptability. 8

Installation. 8

Conformance. 8

Usability. 8

Understandability. 9

Learnability. 9

Ease of Use. 10

Jenkins management. 13

4.1. Plug-in management 13

Setting up a Jenkins job. 14

Prerequisites for build Android apps with Jenkins. 18

Reliability. 18

Maturity. 18

Fault Tolerance. 20

Recoverability. 20

Efficiency. 21

Testing Time. 21

Result Publishing. 22

Graphs. 22

Change Log. 24

Vendor Support. 25

Email Support 25

Phone support 25

User’s Group. 26

Training. 27

Licensing and Support. 27

Open Source/ Commercial 27

Competitors. 27

Cantata Functionality. 28

Suitability. 28

Accurateness. 28

Interoperability. 28

Compliance. 28

Security. 29

Reliability. 29

Maturity. 29

Fault Tolerance. 29

Recoverability. 30

Portability. 30

Adaptability. 30

Conformance. 30

Installation. 30

Usability. 30

Understandability. 31

Learnability. 31

• Static analysis metrics. 31
• A powerful built-in EDG C/C++ parser. 31
• Preferences for the appearance. 31
• Test and analysis generation options. 31
• Test harness. 31
• Unique testability instrumentation and wrapping. 31
• Integrated code coverage and test optimization. 31
• Results and. 31
• Automated repeatable execution through Cantata. 31

Ease of Use. 31

Using the Cantata IDE.. 31

Using the Command Line. 31

Licensing: 31

Efficiency. 31

Testing Time. 31

Result Publishing. 32

Detailed Information. 32

Vendor Support. 32

Email Support 32

Phone Support 33

User’s Group. 33

Training. 33

Licensing and Pricing. 33

Open Source/ Commercial 33

Competitors. 33

Cruisecontrol 33

Introduction. 33

Background. 34

Using the Code. 34

CruiseControl.net 35

Cruisecontrol Tray. 36

Nant / MsBuild. 37

Sandcastle. 38

Nunit 38

Principals of Unit Testing. 38

Tasks Needed to Get CruiseControl.net Working on a New Server. 39

Getting the WebDashBoard Working with IIS. 39

Difference between Cruisecontrol and Jenkis: 40

Jenkins with Selenium.. 42

What is Selenium.. 43

Executing the Test Suite. 46

Running the Test Suite. 46

History of Selenium.. 49

Why Selenium?. 50

Selenium Test Package. 50

Selenium Modes: 51

Record-Playback mode (Selenium IDE)‏: 51

Selenium Remote Control (RC) Mode. 51

Test Runner Mode. 51

Testing AJAX with Selenium.. 51

Selenium IDE – Recording and updating a script 51

Selenium IDE – Most commonly used commands. 52

Selenium Components. 53

Selenium IDE.. 53

Selenium client API. 53

Selenium Remote Control 53

Selenium WebDriver. 54

Selenium Grid. 54

The Selenium Ecosystem.. 55

Popular tools integrating with Selenium.. 55

Services running Selenium on the cloud. 55

Advantages of Selenium.. 56

Disadvantage of Selenium.. 57

Conclusion. 57

References. 58

What is Software Testing

Software testing is a conducted investigation to give stakeholders all the information about product quality or the given service under the test. Software testing will also give an objective, view that is independent for the software which willpermit the business to understand and recognize the risks which are involved with the implementation of software. The Test techniques which are included in the process for executing an application or program with the intent of finding bugs in the software (errors or other defects).

It will also involve the implementation of the component of software or component of system to estimate one or more properties of the interest. In particular, these are the properties which indicate the extent on which the system or component under test:

• convene the requirements which are guided its development and design,
• correctly and rightly responds to all types of the inputs,
• executes the functions within an acceptable time,
• is suitably usable,
• may be installed and also run in theproposed environments, and
• Achieves the common result its stakeholder’s desire.

As the amount of likely tests for still simple components of software is virtually infinite, all the testing of software uses few strategies to find tests which are feasible for the accessibleresources and time. As the result, testing of software typically (but not completely) attempts to implement an application or program with the intent of getting the bugs in software (errors or some other defects). The job of the testing is an iterative process as soon as when one of the bug is fixed; it will illuminate other, the deeper bugs, or may even generate new ones.

Software testing can provide objective, independent information about the quality of software and risk of its failure to users and/or sponsors.

Software testing can be conducted as soon as executable software (even if partially complete) exists. The overall approach to software development often determines when and how testing is conducted. For example, in a phased process, most testing occurs after system requirements have been defined and then implemented in testable programs. In contrast, under an Agile approach, requirements, programming, and testing are often done concurrently.

What is Unit Testing

Unit testing, also known as component testing refers to tests that verify the functionality of a specific section of code, usually at the function level. In an object-oriented environment, this is usually at the class level, and the minimal unit tests include the constructors and destructors.

These types of tests are usually written by developers as they work on code (white-box style), to ensure that the specific function is working as expected. One function might have multiple tests, to catch corner cases or other branches in the code. Unit testing alone cannot verify the functionality of a piece of software, but rather is used to ensure that the building blocks of the software work independently from each other.

Unit testing is a software development process that involves synchronized application of a broad spectrum of defect prevention and detection strategies in order to reduce software development risks, time, and costs. It is performed by the software developer or engineer during the construction phase of the software development lifecycle. Rather than replace traditional QA focuses, it augments it. Unit testing aims to eliminate construction errors before code is promoted to QA; this strategy is intended to increase the quality of the resulting software as well as the efficiency of the overall development and QA process.

Depending on the organization’s expectations for software development, unit testing might include static code analysis, data flow analysis, metrics analysis, peer code reviews, code coverage analysis and other software verification practices.

Here is this thesis we will study about 4 unit testing Jenkis, Selenium, Curisecontrol and Cantata.

Jenkins Functionality

Suitability

• Jenkins was originally developed as the Hudson project. So if you ever see Hudson then it will be known as Jenkins CI Tool. This is written in Java.
• Hudson’s creation started in summer of 2004 at Sun Microsystems. It was first released in java.net in Feb. 2005.Jenkins is an award-winning application that monitors executions of repeated jobs, such as building a software project or jobs run by cron.

Accurateness

Interoperability

Among those things, current Jenkins focuses on the following two jobs:

• Building/testing software projects continuously, just like CruiseControl or DamageControl. In a nutshell, Jenkins provides an easy-to-use so-called continuous integration system, making it easier for developers to integrate changes to the project, and making it easier for users to obtain a fresh build. The automated, continuous build increases the productivity.
• Monitoring executions of externally-run jobs, such as cron jobs and procmail jobs, even those that are run on a remote machine. For example, with cron, all you receive is regular e-mails that capture the output, and it is up to you to look at them diligently and notice when it broke. Jenkins keeps those outputs and makes it easy for you to notice when something is wrong.

Compliance

1. Easy installation: Just java -jar jenkins.war, or deploy it in a servlet container. No additional install, no database.
2. Easy configuration: Jenkins can be configured entirely from its friendly web GUI with extensive on-the-fly error checks and inline help. There’s no need to tweak XML manually anymore, although if you’d like to do so, you can do that, too.
3. Change set support: Jenkins can generate a list of changes made into the build from Subversion/CVS. This is also done in a fairly efficient fashion, to reduce the load on the repository.
4. Permanent links: Jenkins gives you clean readable URLs for most of its pages, including some permalinks like “latest build”/”latest successful build”, so that they can be easily linked from elsewhere.
5. RSS/E-mail/IM Integration: Monitor builds results by RSS or e-mail to get real-time notifications on failures.
6. After-the-fact tagging: Builds can be tagged long after builds are completed.
7. JUnit/TestNG test reporting: JUnit test reports can be tabulated, summarized, and displayed with history information, such as when it started breaking, etc. History trend is plotted into a graph.
8. Distributed builds: Jenkins can distribute build/test loads to multiple computers. This lets you get the most out of those idle workstations sitting beneath developers’ desks.
9. File fingerprinting: Jenkins can keep track of which build produced which jars, and which build is using which version of jars, and so on. This works even for jars that are produced outside Jenkins, and is ideal for projects to track dependency.
10. Plugin Support: Jenkins can be extended via 3rd party plugins. You can write plugins to make Jenkins support tools/processes that your team uses.

Security

Continuous Integration with automated test execution has seen broad adoption in recent years. The ideas behind Continuous Integration have changed how companies look at Build Management, Release Management, Deployment Automation, and Test Orchestration.

Portability

Adaptability

This section provides a set of best practices for Jenkins – A Continuous Integration Solution to provide executives, business managers, software developers and architects a better sense of the development progress and code quality of projects throughout the development lifecycle.

Installation

To run Jenkins, you need the Java Runtime Environment (JRE) 1.7 or later. After you download jenkins.war, you can launch it by executing java -jar jenkins.war. This is basically the same set up as the test drive, except that the output will go to console, not to a window.

Once the war file is exploded, run chmod 755 jenkins in the exploded jenkins/WEB-INF directory so that you can execute this shell script.

If you’re running on Windows you might want to run Jenkins as a service so it starts up automatically without requiring a user to log in. The easiest way to achieve this is to simply download and install Jenkins via the native Windows installer, which can be found on the Jenkins main page. Just download the .zip file, unpack it, and click through the install wizard. The wizard both installs Jenkins and sets up the Jenkins Windows service for you.

Another way to run Jenkins as a service is to first install Tomcat as a service and then deploy Jenkins to it in the usual way. Another way is to use the Java Service Wrapper. However, there may be problems using the service wrapper, because the Main class in Jenkins in the default namespace conflicts with the service wrapper main class. Deploying inside a service container (Tomcat, Jetty, etc.) is probably more straightforward, even for developers without experience with such containers.

Conformance

Alternatively, if you have a servlet container that supports Servlet 2.4/JSP 2.0 or later, such as Glassfish, Tomcat 5, JBoss, Jetty 6, etc., then you can deploy jenkins.war as you would any WAR file. See this document for more about container-specific installation instruction.

Usability

Understandability

Continuous integration is a process in which all development work is integrated at a predefined time or event and the resulting work is automatically tested and built.

Jenkins is one open source tool to perform continuous integration. The basic functionality of Jenkins is to execute a predefined list of steps based on a certain trigger.

The list of steps can for example include:

• perform a software build with Apache Maven or Gradle
• Run a shell script
• Archive the build result
• Afterwards start the integration tests

Jenkins can be extended by additional plug-ins, e.g., for building and testing Android applications or to support the Git version control system.

Learnability

Jenkins provides Debian/Ubuntu packages which install Jenkins and register Jenkins as start service. See the following URL for details.

http://pkg.jenkins-ci.org/debian/

Jenkins stores all the settings, logs and build artifacts in its home directory. The default installation directory is/var/lib/jenkins under Ubuntu.

This creates a /etc/init.d/jenkins start script which starts Jenkins automatically at boot time.

If you start it locally, you find it running under the following URL: http://localhost:8080/

For most platforms you have native packages, see the Jenkins Homepage.

Download the jenkins.war file from Jenkins Homepage.

You can also start Jenkins directly via the command line with java -jar jenkins*.war. If you start it locally, you find it running under the following URL: http://localhost:8080/

To run it in your Tomcat server, put the .WAR file into the webapps directory. If you start Tomcat, your Jenkins installation will be available under http://localhost:8080/jenkins

cd /usr/share/tomcat7

sudo mkdir .jenkins

sudo

            chown tomcat7:nogroup .jenkins

This makes the .jenkins folder writable and Jenkins can use it.

If you want to install Jenkins in a web container, you can use, for example, Tomcat or Jetty. See the Apache Tomcat Tutorial.

Ease of Use

Before using Jenkins you need to tell it where your JDK and ant installation is. Open Jenkins in your browser and click Manage Jenkins and then Configure System.

Enter the correct path to your JDK, Apache Ant and Maven and press the Save button below. Jenkins can also install these for your automatically.

It is recommended to secure Jenkins. Manage Jenkins and then Configure Global Security. Select the Enable security flag. The easiest way is to use Jenkins own user database. Create at least the user “Anonymous” with read access. Also create entries for the users you want to add in the next step.

On the login page, select Create an account to create the users you just gave access.

Go to Manage Jenkins, Manage and Assign Roles and then Assign Roles to grant the newly created user additional access rights.

Navigate to Manage Roles to define access restrictions in detail. Pattern is a regex value of the job name. The following grants unregistered users read-only access to your build jobs that start with the C-MASTER or M-MASTERprefix and only those.

Jenkins management

4.1. Plug-in management

Jenkins can be extended via additional plug-ins with more functionality. You can configure your plug-ins via theManage Jenkins → Manager Plugins link.

To install plugins in Jenkins select use the Manage Jenkins → Manager Plugins link and search for the plugin you want to install. Select it from the list and select to install it and restart Jenkins.

The following table is a summary of commonly used plug-ins.

Table 1. Jenkins plug-ins

# install vncserver

apt-get install vnc4server

# switch to jenkins user

sudo su jenkins

# connect to vncserver which creates the password

vncserver :10

Setting up a Jenkins job

The build of a project is handled via jobs in Jenkins. Select New Item from the menu

Afterwards enter a name for the job and and select Freestyle Job. Press OK to create a new Job in Jenkins.

The next page allows you to configure your job. If you for example using Git, enter the URL to the Git repository. If the repository is not public, you may also need to configure the credentials.

Specify when and how your build should be triggered. The following example polls the Git repository every 15 min and triggers a build, if something has changed in the repo.

I typically delete the workspace before a build to avoid any side-effect. In the Build section you can add a build step, e.g., a Maven build.

Press Save to finish the job definition. Press Build Now on the job page to validate the job works as expected.

After a while the job should go to green or blue (depending on your configuration), if successful. Click on the job and afterwards on Console Output to see the log file in case of an error or to validate that log looks as expected.

Prerequisites for build Android apps with Jenkins

The following descriptions for build Android applications with Jenkins assume that you have already a correct Gradle build file for your Android application.

Reliability

Maturity

Jenkins supports the automatic building and testing of Android applications.

To create a build job on Jenkins you need to have a working build setup. The Android tooling generates automatically a valid Gradle build file.

# download the Android SDK via wget

# for the correct URL see http://developer.android.com/sdk/index.html

# and look in the section "SDK Tools Only"

wget <link from the above website>

# example

# wget http://dl.google.com/android/android-sdk_r24.2-linux.tgz

# unpack it

tar zxvf filename

# Add the new directory to your patch

# assumes you exported it to /opt/

# this must be set for the Jenkins user

export ANDROID_HOME="/opt/android-sdk-linux"

export PATH="$ANDROID_HOME/tools:$ANDROID_HOME/platform-tools:$PATH"

// run this as Jenkins user

android update sdk --no-ui

Use android list targets to see what ABIs are installed. If no are listed manual install one.

You need the follwing plug-ins to build Android application with Jenkins:

• Gradle Plugin
• Git Plugin – if Git is used for the project sources
• Android Emulator Plugin – supporting starting and unlocking an Android emulator and blocks the build until the emulator has started. Required for running tests. A detailed description of this plugin can be found under the following URL: Android Emulator Plugin Jenkins page.

You configure from where the source should be cloned.

You configure the emulator which should be started. Ensure that you do not select the Show emulator windowoption, as your build server should not depend on the availability of a display server.

Add a Gradle build step to your Jenkins jobs.

Fault Tolerance

The possibility of increasing the fault tolerance of Jenkins is by running multiple instances of Jenkins in parallel and let them cooperate and share the load. In case failure occurs to one Jenkins, another Jenkins can take over. The major restriction was to not modify the source code of Jenkins. The results of the investigation are that Jenkins can be modified to achieve high availability in the form of an active/active solution. Events in Jenkins related to build jobs can be replicated, these include: creating, modifying and deleting build jobs, also replication of builds can be replicated. The data replication technique had to be done asynchronously which introduced problems such as data collisions and ping-ponging. No solutions were found that could avoid data collisions; however, they can be detected and resolved. A major issue is that other plugins in Jenkins could not be made aware of there being more than a single Jenkins. This issue and areas such as synchronization of user accounts, and the behavior of Jenkins slaves,

Recoverability

Jenkins stores all the settings, logs and build artifacts in its home directory, for example, in /var/lib/jenkinsunder the default install location of Ubuntu.

To create a backup of your Jenkins setup, just copy this directory.

The jobs directory contains the individual jobs configured in the Jenkins install. You can move a job from one Jenkins installation to another by copying the corresponding job directory. You can also copy a job directory to clone a job or rename the directory.

Click reload config button in the Jenkins web user interface to force Jenkins to reload configuration from the disk.

Jenkins supports the https://wiki.jenkins-ci.org/display/JENKINS/SCM+Sync+configuration+plugin plug-in which allows you to store every change in a Git repo.

It is also possible to manually maintain the Jenkins configuration in a Git repo.

Efficiency

Testing Time

When our needs started growing, our first thought was to add more Jenkins nodes and have multiple tests running in parallel. But there are many different environments that we want to test every commit against:

• Test the deployment of the app in a clean environment, a fresh build.
• Test the deployment of the app in a staging environment where you want to ensure backwards compatibility
• Test the web interface against all supported browsers.

All these meant that testing requirements would grow over time and a testing infrastructure based solely on Jenkins would not do the job fast enough.

Docker helps you easily create lightweight, portable, self-sufficient containers from any application. It is fast, reliable and a perfect fit for our needs. The idea, is to set up Jenkins so that every pull request to a specific branch (e.g. staging) triggers a job. Jenkins then commands our testing infrastructure to spawn different docker applications simultaneously and runs the tests in parallel.

This way, we can test any environment. We just have to describe each application the same way we would’ve done manually. On the plus side, we can use pre-made images from the docker repository to jumpstart our tests.

For example we could do this:

docker pull ubuntu

And we will end up with an ubuntu image/application. We can then run the docker container by typing:

docker run -i -t ubuntu /bin/bash

And we will be in a fresh, newly created ubuntu environment.

But we don’t want to manually run docker commands after every pull request. What we need, are Dockerfiles. Dockerfiles are sets of steps that describe an image/container. We’ll use them to build our custom docker images.

Result Publishing

While executing our tests on Jenkins many of time test fails on particular builds. Now if you want to check that when a particular test-case, test-class or a test-package has failed you may need to go to each and every build report and check for the result.

This process is very cumbersome and may take a lot of time to get the report. The said plugin solves the said issue by showing the build result history of test-class, test-class and test-package in a tabular tree format. The plugin can be used enabling the “publish junit results” feature of Jenkins.

It allows users to filter the results based on passed, failed and skipped status.

Once installed you can just click on the “Test Results Analyzer” link on the left hand side of your job, as shown in the following image: