Master Thesis Title: Evaluation of Software Testing Tools ( Jenkins, CruiseControl, Selenium, Cantata)
== This thesis should evaluate all the above 4 testing tools according to some evaluation criteria. I am attaching an evaluation criteria that has been approved by my examiner. So the evaluation criteria is already designed. The evaluation criteria can be fit into the thesis report. The evaluation should strictly cope with each of the criteria and sub critiera. How to get the results for each criteria and sub-criteria for each tool is given as: 1. Install Jenkins (Continuous Integration Server) and play with it to get the required results from it and then write it down under each criteria and sub-criteria. 2. Install CruiseControl(Continuous Integration Server) and play with it to get the required results from it and then write it down under each criteria and sub criteria for CrusieControl 3. Install Selenium and do the same. 4. Install Cantata (Automated tool for unit testing of C/C++ language codes) and do the same as above. There should be one separate chapter for Results where the each tool is compared with the other tool according to the evaluation results. After the results there should be discussion and Conclusion. and Finally the references. Please note that this will not be possible without the installation of all the four tools and therefore no evaluation without the tool installation will be accepatiable. The examiner should see that the tools have been installed for gettting the results. Please let me know via email if I need to give further information.
Websites : Jenkins = https://jenkins-ci.org/
CruiseControl = http://cruisecontrol.sourceforge.net/
Selenium = http://www.seleniumhq.org/
Cantata = http://www.qa-systems.com/cantata.html
- Software Testing and Continuous Integration
- ·· 60 pages, 15000 words Revised Sweden
- Harvard Revised 120 Refers
- (+46) ***********
- Jul 15, 2015 | Wednesday 06:28 am | 15 days 1 hrs ago
Table of Contents
What is Software Testing. 5
What is Unit Testing. 6
Jenkins Functionality. 6
- Building/testing software projects continuously. 7
- Monitoring executions of externally-run jobs. 7
1. Easy installation. 7
2. Easy configuration. 7
3. Change set support 7
4. Permanent links. 7
5. RSS/E-mail/IM Integration. 7
6. After-the-fact tagging. 7
7. JUnit/TestNG test reporting. 7
8. Distributed builds. 7
9. File fingerprinting. 7
10. Plugin Support 8
Ease of Use. 10
Jenkins management. 13
4.1. Plug-in management 13
Setting up a Jenkins job. 14
Prerequisites for build Android apps with Jenkins. 18
Fault Tolerance. 20
Testing Time. 21
Result Publishing. 22
Change Log. 24
Vendor Support. 25
Email Support 25
Phone support 25
User’s Group. 26
Licensing and Support. 27
Open Source/ Commercial 27
Cantata Functionality. 28
Fault Tolerance. 29
- Static analysis metrics. 31
- A powerful built-in EDG C/C++ parser. 31
- Preferences for the appearance. 31
- Test and analysis generation options. 31
- Test harness. 31
- Unique testability instrumentation and wrapping. 31
- Integrated code coverage and test optimization. 31
- Results and. 31
- Automated repeatable execution through Cantata. 31
Ease of Use. 31
Using the Cantata IDE.. 31
Using the Command Line. 31
Testing Time. 31
Result Publishing. 32
Detailed Information. 32
Vendor Support. 32
Email Support 32
Phone Support 33
User’s Group. 33
Licensing and Pricing. 33
Open Source/ Commercial 33
Using the Code. 34
Cruisecontrol Tray. 36
Nant / MsBuild. 37
Principals of Unit Testing. 38
Tasks Needed to Get CruiseControl.net Working on a New Server. 39
Getting the WebDashBoard Working with IIS. 39
Difference between Cruisecontrol and Jenkis: 40
Jenkins with Selenium.. 42
What is Selenium.. 43
Executing the Test Suite. 46
Running the Test Suite. 46
History of Selenium.. 49
Why Selenium?. 50
Selenium Test Package. 50
Selenium Modes: 51
Record-Playback mode (Selenium IDE): 51
Selenium Remote Control (RC) Mode. 51
Test Runner Mode. 51
Testing AJAX with Selenium.. 51
Selenium IDE – Recording and updating a script 51
Selenium IDE – Most commonly used commands. 52
Selenium Components. 53
Selenium IDE.. 53
Selenium client API. 53
Selenium Remote Control 53
Selenium WebDriver. 54
Selenium Grid. 54
The Selenium Ecosystem.. 55
Popular tools integrating with Selenium.. 55
Services running Selenium on the cloud. 55
Advantages of Selenium.. 56
Disadvantage of Selenium.. 57
What is Software Testing
Software testing is a conducted investigation to give stakeholders all the information about product quality or the given service under the test. Software testing will also give an objective, view that is independent for the software which willpermit the business to understand and recognize the risks which are involved with the implementation of software. The Test techniques which are included in the process for executing an application or program with the intent of finding bugs in the software (errors or other defects).
It will also involve the implementation of the component of software or component of system to estimate one or more properties of the interest. In particular, these are the properties which indicate the extent on which the system or component under test:
- convene the requirements which are guided its development and design,
- correctly and rightly responds to all types of the inputs,
- executes the functions within an acceptable time,
- is suitably usable,
- may be installed and also run in theproposed environments, and
- Achieves the common result its stakeholder’s desire.
As the amount of likely tests for still simple components of software is virtually infinite, all the testing of software uses few strategies to find tests which are feasible for the accessibleresources and time. As the result, testing of software typically (but not completely) attempts to implement an application or program with the intent of getting the bugs in software (errors or some other defects). The job of the testing is an iterative process as soon as when one of the bug is fixed; it will illuminate other, the deeper bugs, or may even generate new ones.
Software testing can provide objective, independent information about the quality of software and risk of its failure to users and/or sponsors.
Software testing can be conducted as soon as executable software (even if partially complete) exists. The overall approach to software development often determines when and how testing is conducted. For example, in a phased process, most testing occurs after system requirements have been defined and then implemented in testable programs. In contrast, under an Agile approach, requirements, programming, and testing are often done concurrently.
What is Unit Testing
Unit testing, also known as component testing refers to tests that verify the functionality of a specific section of code, usually at the function level. In an object-oriented environment, this is usually at the class level, and the minimal unit tests include the constructors and destructors.
These types of tests are usually written by developers as they work on code (white-box style), to ensure that the specific function is working as expected. One function might have multiple tests, to catch corner cases or other branches in the code. Unit testing alone cannot verify the functionality of a piece of software, but rather is used to ensure that the building blocks of the software work independently from each other.
Unit testing is a software development process that involves synchronized application of a broad spectrum of defect prevention and detection strategies in order to reduce software development risks, time, and costs. It is performed by the software developer or engineer during the construction phase of the software development lifecycle. Rather than replace traditional QA focuses, it augments it. Unit testing aims to eliminate construction errors before code is promoted to QA; this strategy is intended to increase the quality of the resulting software as well as the efficiency of the overall development and QA process.
Depending on the organization’s expectations for software development, unit testing might include static code analysis, data flow analysis, metrics analysis, peer code reviews, code coverage analysis and other software verification practices.
Here is this thesis we will study about 4 unit testing Jenkis, Selenium, Curisecontrol and Cantata.
- Jenkins was originally developed as the Hudson project. So if you ever see Hudson then it will be known as Jenkins CI Tool. This is written in Java.
- Hudson’s creation started in summer of 2004 at Sun Microsystems. It was first released in java.net in Feb. 2005.Jenkins is an award-winning application that monitors executions of repeated jobs, such as building a software project or jobs run by cron.
Among those things, current Jenkins focuses on the following two jobs:
- Building/testing software projects continuously, just like CruiseControl or DamageControl. In a nutshell, Jenkins provides an easy-to-use so-called continuous integration system, making it easier for developers to integrate changes to the project, and making it easier for users to obtain a fresh build. The automated, continuous build increases the productivity.
- Monitoring executions of externally-run jobs, such as cron jobs and procmail jobs, even those that are run on a remote machine. For example, with cron, all you receive is regular e-mails that capture the output, and it is up to you to look at them diligently and notice when it broke. Jenkins keeps those outputs and makes it easy for you to notice when something is wrong.
- Easy installation: Just java -jar jenkins.war, or deploy it in a servlet container. No additional install, no database.
- Easy configuration: Jenkins can be configured entirely from its friendly web GUI with extensive on-the-fly error checks and inline help. There’s no need to tweak XML manually anymore, although if you’d like to do so, you can do that, too.
- Change set support: Jenkins can generate a list of changes made into the build from Subversion/CVS. This is also done in a fairly efficient fashion, to reduce the load on the repository.
- Permanent links: Jenkins gives you clean readable URLs for most of its pages, including some permalinks like “latest build”/”latest successful build”, so that they can be easily linked from elsewhere.
- RSS/E-mail/IM Integration: Monitor builds results by RSS or e-mail to get real-time notifications on failures.
- After-the-fact tagging: Builds can be tagged long after builds are completed.
- JUnit/TestNG test reporting: JUnit test reports can be tabulated, summarized, and displayed with history information, such as when it started breaking, etc. History trend is plotted into a graph.
- Distributed builds: Jenkins can distribute build/test loads to multiple computers. This lets you get the most out of those idle workstations sitting beneath developers’ desks.
- File fingerprinting: Jenkins can keep track of which build produced which jars, and which build is using which version of jars, and so on. This works even for jars that are produced outside Jenkins, and is ideal for projects to track dependency.
- Plugin Support: Jenkins can be extended via 3rd party plugins. You can write plugins to make Jenkins support tools/processes that your team uses.
Continuous Integration with automated test execution has seen broad adoption in recent years. The ideas behind Continuous Integration have changed how companies look at Build Management, Release Management, Deployment Automation, and Test Orchestration.
This section provides a set of best practices for Jenkins – A Continuous Integration Solution to provide executives, business managers, software developers and architects a better sense of the development progress and code quality of projects throughout the development lifecycle.
To run Jenkins, you need the Java Runtime Environment (JRE) 1.7 or later. After you download jenkins.war, you can launch it by executing java -jar jenkins.war. This is basically the same set up as the test drive, except that the output will go to console, not to a window.
Once the war file is exploded, run chmod 755 jenkins in the exploded jenkins/WEB-INF directory so that you can execute this shell script.
If you’re running on Windows you might want to run Jenkins as a service so it starts up automatically without requiring a user to log in. The easiest way to achieve this is to simply download and install Jenkins via the native Windows installer, which can be found on the Jenkins main page. Just download the .zip file, unpack it, and click through the install wizard. The wizard both installs Jenkins and sets up the Jenkins Windows service for you.
Another way to run Jenkins as a service is to first install Tomcat as a service and then deploy Jenkins to it in the usual way. Another way is to use the Java Service Wrapper. However, there may be problems using the service wrapper, because the Main class in Jenkins in the default namespace conflicts with the service wrapper main class. Deploying inside a service container (Tomcat, Jetty, etc.) is probably more straightforward, even for developers without experience with such containers.
Alternatively, if you have a servlet container that supports Servlet 2.4/JSP 2.0 or later, such as Glassfish, Tomcat 5, JBoss, Jetty 6, etc., then you can deploy jenkins.war as you would any WAR file. See this document for more about container-specific installation instruction.
Continuous integration is a process in which all development work is integrated at a predefined time or event and the resulting work is automatically tested and built.
Jenkins is one open source tool to perform continuous integration. The basic functionality of Jenkins is to execute a predefined list of steps based on a certain trigger.
The list of steps can for example include:
- perform a software build with Apache Maven or Gradle
- Run a shell script
- Archive the build result
- Afterwards start the integration tests
Jenkins can be extended by additional plug-ins, e.g., for building and testing Android applications or to support the Git version control system.
Jenkins provides Debian/Ubuntu packages which install Jenkins and register Jenkins as start service. See the following URL for details.
Jenkins stores all the settings, logs and build artifacts in its home directory. The default installation directory is
/var/lib/jenkins under Ubuntu.
This creates a
/etc/init.d/jenkins start script which starts Jenkins automatically at boot time.
If you start it locally, you find it running under the following URL: http://localhost:8080/
For most platforms you have native packages, see the Jenkins Homepage.
jenkins.war file from Jenkins Homepage.
You can also start Jenkins directly via the command line with
java -jar jenkins*.war. If you start it locally, you find it running under the following URL: http://localhost:8080/
To run it in your Tomcat server, put the .WAR file into the
webapps directory. If you start Tomcat, your Jenkins installation will be available under http://localhost:8080/jenkins
sudo mkdir .jenkins
chown tomcat7:nogroup .jenkins
This makes the .jenkins folder writable and Jenkins can use it.
If you want to install Jenkins in a web container, you can use, for example, Tomcat or Jetty. See the Apache Tomcat Tutorial.
Ease of Use
Before using Jenkins you need to tell it where your JDK and ant installation is. Open Jenkins in your browser and click Manage Jenkins and then Configure System.
Enter the correct path to your JDK, Apache Ant and Maven and press the Save button below. Jenkins can also install these for your automatically.
It is recommended to secure Jenkins. Manage Jenkins and then Configure Global Security. Select the Enable security flag. The easiest way is to use Jenkins own user database. Create at least the user “Anonymous” with read access. Also create entries for the users you want to add in the next step.
On the login page, select Create an account to create the users you just gave access.
Go to Manage Jenkins, Manage and Assign Roles and then Assign Roles to grant the newly created user additional access rights.
Navigate to Manage Roles to define access restrictions in detail. Pattern is a regex value of the job name. The following grants unregistered users read-only access to your build jobs that start with the
M-MASTERprefix and only those.
4.1. Plug-in management
Jenkins can be extended via additional plug-ins with more functionality. You can configure your plug-ins via theManage Jenkins → Manager Plugins link.
To install plugins in Jenkins select use the Manage Jenkins → Manager Plugins link and search for the plugin you want to install. Select it from the list and select to install it and restart Jenkins.
The following table is a summary of commonly used plug-ins.
Table 1. Jenkins plug-ins
This plugin allows use of Git as a build SCM.
This plugin allows projects to run xvnc during a build. This allows for example to run tests which requires a display to run on a virtual display. To use this plug-in you need to connect once to your vncserver on the command line to provide a password. Use for example the following commands.
# install vncserver
apt-get install vnc4server
# switch to jenkins user
sudo su jenkins