Testing Automation has become standard industry practice, saving quality assurance engineers the time and effort required for manual testing while enhancing overall code efficiency and accuracy. It also allows companies to reduce costs while speeding up product release times. These benefits can become especially apparent when developers are testing web applications that have long life cycles like those in the financial industry.
When working on these continuously expanding projects, SDETs (Software Development Engineers in Test) must implement the testing infrastructure and automated tests in such a way that the state of the system can be checked at certain designated intervals. In a sense, these engineers must travel in virtual time. This process can be quite complex and problematic. To avoid testing-related issues when working on projects with long life cycles, SDETs can follow several approaches as detailed in this article. These approaches are implemented in Ruby, but they can easily be converted to any programming language.
Approaches to Time Manipulation
Change the System Time
The first and easiest way to travel virtually forward in time is to change the system time by typing in a simple command inside your test like this: system(“sudo date -s ‘2015-11-05′”).
Positives
- The code is easy to implement and affects all the applications, along with the system itself.
- It doesn’t require any changes to the application code.
Negatives
- Because this method will affect the entire system, including all applications and environments, the web server, database, and services, you may experience any number of problems in the future. Timeouts can occur in different places, for example, and some applications, like httpsd web server, might not function properly.
- Running parallel tests on the same machine when using this method is impossible because these tests share the same environment.
- Side effects in reporting may occur. A test may be reported to run for a long time if finished in future time, or for a negative time if finished in the past.
- Socket problems can arise because of timeouts.
- Changing the system time for distributed environments is generally difficult and inefficient.
Using the Libfaketime Library
Another way you can travel virtually in time is by using libfaketime. This library mocks system time calls for particular processes, and you may choose which clocks you want to mock. For example, you may not want to mock monotonic clocks to avoid socket problems.
Positives
- You can set time calls for specific applications like database, application under test, and test script.
- You are not required to change the application code.
- This method does not affect the operation system or other applications.
Negatives
- Running parallel tests is complicated because you need to preload a library for each process separately using different configuration files. Each parallel test process should connect only to a specific web application process.
- The side effects in reporting are the same as when changing the system time.
- Using this method for distributed environments can be problematic.
- Currently, libfaketime doesn’t work properly with Selenium + Firefox. Sporadic timeout errors during Selenium command execution regularly occur.
Using Additional Parameters or HTTP Headers + Timecop
This last approach uses HTTP headers or additional parameters for requests, along with a library that allows you to set the time for the current process. In this case, you set HTTP(S) header (for example TIME_SHIFT) for all requests you make to the application, whose value will offset the current time in seconds. On the server side, you get the offset value from the header and the mock time using Timecop or another, similar library. After processing the request, you can reset the time back to the current time.
Positives
- Time is shifted only inside one process and only for one request. This allows you to run time-related tests in parallel.
- Other system applications are not affected.
- Minimal or no changes to the application are required.
- Reporting is accurate with no side-effects.
- You can use this method with AWS, Docker, or other distributed environments.
Negatives
- If database requests or relations have triggers or constraints that use current_date or current_time, requests might fail.
- You may need additional “hacks” for other applications, like sidekiq, to allow these applications to time travel together.
- You will need a mock time on the client side (browser), and/or you will have to use a proxy to set up the required headers.
- You must trigger cron jobs with specific time parameters directly from the script.
Using HTTP headers or additional parameters is the most reliable and efficient approach to time travel, primarily because you can run tests in parallel. However, it is also the most complex approach, and you may face several problems when you add features and future testing commands:
- If you have microservice architecture, you should be able to pass the TIME_SHIFT header to the request chain. You can do so by adding TIME_SHIFT headers for all requests to other services, which will modify their time to <real time> + <shift>. Using “shift” instead of “timestamp” as a TIME_SHIFT header value allows “flowing time” to be emulated. However, you must make sure the value is shifted from current, real time, and not from shifted time. Otherwise, you will have to sum shift with each subsequent cyclic request.
- Database requests can be an issue with this approach if you have SQL requests that use CURRENT_TIME, CURRENT_DATE, or another date/time related function. As a solution, I suggest avoiding these types of calls.
- When using sidekiq or another job/queue tool, you have to pay attention to shifting time. Creating a job in sidekiq from a service in which time has been shifted will place this job in a queue, and the sidekiq worker will select and process it in the order it was received. The sidekiq worker won’t be aware of the shifted time. This problem can be solved by adding a parameter to the sidekiq job. In this parameter, you can set the shift from current real time. After sidekiq selects your job, it will set the local sidekiq time to <real time> + <shift>, process the job, and then set to real time after the job is completed.
- To avoid the problem of the browser-side javascript using real time instead of future time, I recommend using client-side time mocking. This will allow you to set the time on the client side, and the proxy server will pass headers to the backend. Using Sinon.js is one possible solution for mocking time in the browser.
This diagram illustrates the schematic workflow of the entire time travel process:
Testing Automation has become an essential part of the software development process, ensuring high ROI by saving QA’s significant time and effort, decreasing the size of project development teams, increasing code accuracy and test coverage, and speeding up product release times. SDETs must be aware of the latest Testing Automation approaches available to them, especially when implementing the infrastructure and its related tests in continuously expanding projects.
The approaches I describe facilitate the testing of long-lasting, complex scenarios, but they also require SDETs to take some precautions:
- If you change or affect any object in the future, that object will also be changed in the present time. For example, in an Ecommerce application, you cannot travel into the future to force order processing, then go back in time to change the price of a product. If your application’s “real” time state must remain unchanged, you will have to simulate all future actions until you reach the current date. Similarly, you will need to “rewind” actions to the current date if traveling into the future.
- Changing system time is easy to implement, but any number of problems can arise because the entire system is affected.
- Implementing libfaketime allows you to set time calls for specific applications, but problems may arise when running parallel tests, or when making changes to distributed environments.
- Using headers to travel virtually is the most reliable method, allowing you to run tests in parallel. However, you must be careful when shifting time while using a job/queue tool that runs in a separate process like sidekiq.
