Improving the quality of the STF tests in SAROS

worked on by: Suzana Puscasu

Problem Statement

Automated testing is one of the fastest ways to find errors in a software of high complexity. The automated tests should be conceived and implemented so that they do not take long to run, depict a well-defined test case, and are comprehensive for other QA engineers, in case changes are needed. The SAROS Test Framework (STF) enables end-to-end tests in real time. However, it was noticed that runtime of the STF test suite can be improved. Setting up the test environment is the main cause for long execution time. The test environment consists of three machines: the Jenkins, the test executing node, and two Eclipse machines that communicate via SSH and Java RMI. When it comes to the tests themselves, preparing the start point each time for each test case separately takes some extra time. For example, creating a new project takes approximately 25 ms, creating a new file 5 ms, clicking a button while creating projects and files 22 ms, establishing a connection between two users takes up to 73 s. A part of these issues will be handled by Jannis Fey in his Master’s Thesis “Saros Test Framework for the HTML GUI”. Furthermore, not setting a clear limit between the preparation of the initial state of a test case and the test itself could lead to the failure of the test without knowing where the error causing it was: in the preparation of the initial state or in the test itself.

Goal and Relevance

The goal of this thesis is to improve the quality of the STF tests in SAROS. By separating the test itself from the initial operations, the errors can be identified more precisely. Moreover,improving the execution time of the tests enables discovering the defects that led to failure of the software, much faster.

Approach

A set of tests for core functionalities will be defined and conducted before any other tests. If the tests for these specific functionalities are not successful, the other tests based on them will most probably fail as well. This could be realised with conditional testing in JUnit. For example, establishing the connection between two testers or project sharing could be some of the core functionalities. Measuring how long it takes for a test to run will give a clear picture of which parts of the tests should be modified. Time statements will be added before each critical part of the code. The measuring of the run time will be repeated more times in order to stochastically get a relevant result. Depending on the results, the general starting points will be identified and implemented according to their priority. The priority of the starting point will be a result of the runtime up to this point, the number of test cases that have this starting point, and the time invested in its implementation. Moreover, the sleep times will be adjusted to the current testing environment. Measurements will be conducted meanwhile to see if the runtime improvements are significant enough. If so, the starting point with the next priority will be implemented.