ParallelJUnit - Shifting performance testing left

Why this exist

Many organizations moving towards DevOps struggle with shifting performance testing left to include it into the CI/CD build pipelines. Many of the types of causes that induces performance or capacity problems cannot be identified in a build pipeline since the infrastructure of the CI-server rarely is production like. However, performance problems from coding mistakes can be identified.

Full fledged performance tests with a generated distributed load upon a system is seemingly trivial, but is a special skillset that takes years to get good at and decades to master. That's not feasible for developer performance testing. Instead this utility makes it possible to run your unit tests as mini-performance tests to check for concurency issues and locks - as a complimnet to the built in performance profilers of most IDEs and to APM tools for finding infrastructure problems.

I firmly believe in the laziness of well-intended developers. If I can only provide the means of easily doing performance testing with the tools they are already used to, they will.

This enables easy response time assertions over multi-thread tests.

What it is

It's a custom JUnit test runner that enables executing JUnit test methods in concurrent parallels threads by complimenting JUnit with the use of the @Test alternative annotations:

@ParallelTest
@ParallelizationTest
@LoadTest

Download

It's likely the easiest way to use this utility is to include it as a dependency to your maven pom file:

        
                    <dependency>
                        <groupId>com.github.claremontqualitymanagement</groupId>
                        <artifactId>ParallelJUnit</artifactId>
                        <version>1.0.2</version>
                    </dependency>
                

You can download the utility here:

There is also a GitHub page where you can find the code for this utility (Apache 2.0 applies).

You may also find a similar utility for .NET/C# (using MsTest as test runner) in the nuget package called TAF.UnitTestLoadTest.MsTest.

Let's go through the test types one at the time:

ParallelTest annotation

The @ParallelTest is used instead of the @Test annotation of regular JUnit. When the @ParallelTest is used the test method is executed concurrently in multiple threads. The number of concurrent threads may be set by the threadCount argument. Default threadCount for ParallelTest is 2.

Regular JUnit @Test optional arguments timeout and expected applies for @ParallelTest too.

Examples

        
            import org.junit.Assert;
            import org.junit.Test;
            import org.junit.runner.RunWith;    
            
            //Don't forget to use the adapted JUnit test runner
            @RunWith(ParallelJUnit.class)
            public class Tests {
            
               //Regular JUnit test - passed on to default JUnit runner.
               @Test 
               public void passingRegularTestShouldPass(){
        
                    ...
        
                }
               
               //Four parallel concurrent threads, passing timeout (ms)
               @ParallelTest(threadCount = 4, timeout = 200) 
               public void parallelTestShouldPass() {
        
                ... Whatever testing is supposed to be executed in parallel
        
               }
        
               //Ten parallel threads. Expecting some threads to throw exceptions (Bad test since it's fuzzy). 
               @ParallelTest(threadCount = 10, expected = TestMethodExecutionException.class)  
               public void exceptionThrowingTestMethodShouldThrowException() throws Exception {
                   if(Math.random() * 100 < 50){
                       throw new TestMethodExecutionException("Oups");
                   }
               }
               
            }
        

ParallelizationTest

A variant of ParallelTest is testing if a tested method seem to be able to handle sequential execution or truly parallel execution. In order to do this in an easy fashion the @ParallelizationTest is used. It execute the test method in a single thread first (to avoid first execution initialization problems), then clocks how log the execution of the test method takes with execution with one thread. After this the same method is executed in multiple parallel threads to see if this takes significantly longer than the execution in a single thread.

To summarize the execution sequence:

1. Executing once in one thread as a warm-up (populating caches, ready-compiling components and so forth)
2. Executing once more in a timed execution with one single thread.
3. Executing with multiple concurrent and parallel threads in a timed execution run.
4. Evaluating the success by assessing the ratio between the single thread execution duration with the multiple parallel execution duration variable (using maxExecutionDurationMultipleForMultipleThreadsExecution parameter).

For this type of test the following parameters apply:

• Number of concurrent threads when executed in parallel: multipleThreadsCount (default = 3).
• Duration ration to assess test success towards maxExecutionDurationMultipleForMultipleThreadsExecution (default = 1.5)

Example

        
               //Runs once single-threaded for warm-up, once single-threaded for benchmark, and once multi-threaded for comparison. 
               @ParallelizationTest(multipleThreadsCount = 3, maxExecutionDurationMultipleForMultipleThreadsExecution = 1.5)
               public void parallelizationTest()  {
        
                   ...
                   
               }
        

LoadTest

This annotation is for performance testing closer to LoadRunner/JMeter or equivalent tools. It enables ramp-up of load and holding a system under load for a longer period of time. The unit test method runs in concurrent parallel threads as with the other test types in this library, and the execution time for each individual method execution (for each iteration) can be assessed towards a set threshold. Using this test type the threadpool used is filled up again with a new execution when a test method execution is finished.

Examples

               //Runs this test method continuously in a thread pool of 3 concurrent threads for 1000 milliseconds 
                @LoadTest(maxThreadCount = 3, totalDurationInMilliseconds = 1000)
                public void loadTest() throws InterruptedException {
                    System.out.println("Running thread at " + new SimpleDateFormat("HH:mm:ss SS").format(new Date()));
                    Thread.sleep(200);
                }
        
                //Run this test method continuously in 10 parallel, but ramp these up evenly over 2 seconds 
                //of time until full load and halt on any test interation taking longer than 1300 milliseconds and re-use
                //the test class instances for method invocation. Halt test abruptly after 5 seconds.
                @LoadTest(
                    maxThreadCount = 10,
                    totalDurationInMilliseconds = 5000,
                    rampUpTimeInMilliseconds = 2000,
                    preEmptiveTestClassInstantiationWithTestClassObjectReUsedBetweenIterations = true,
                    haltOnError = true,
                    abruptTerminationAtTestEnd = true,
                    maxExecutionTimeIndividualIteration = 1300)
                public void loadTestWithRampUp() throws InterruptedException {
                    System.out.println("Running thread at " + new SimpleDateFormat("HH:mm:ss").format(new Date()));
                    Thread.sleep(1000);
                }
        

Options/parameters/arguments:

maxThreadCount (default 2)
rampUpTimeInMilliseconds (default 0)
preEmptiveTestClassInstantiationWithTestClassObjectReUsedBetweenIterations (default false, saves time from test class instantiation in each iteration)
totalDurationInMilliseconds (default 3000)
haltOnError (default false, makes the test halt upon errors - including execution time assertions)
maxExecutionTimeIndividualIteration (default ignored, throws an assertion error if any of the method executions takes longer than this)
abruptTerminationAtTestEnd (default true, if set to false it leaves all threads up to 30 seconds to finish);
timeout (default 30000, halts test abruptly if it takes longer than this, for compatiblity with JUnit @Test annotation)
expected (any expected exception to ignore)

Technical notes

• Each test method thread execution is executed on its own test class instance.
• Throws TestDurationCheckException if test execution takes longer than the given timeout.
• Throws TestClassInstantiationException if test class cannot be instantiated with default parameter-less constructor.
• Throws TestMethodExecutionException if any of the executed threads throws any exception (collected as inner exceptions).
• Default expected (Expected thrown exceptions from test execution) = none.

Testing notes

Testing with the same data over and over is a bit risky. In part because any cache in the system might respond after first request, giving un-normal response times, and partly because mant databases uses read-locks for requested records. Accessing the same record many times hence could create queues in the database.

Testing in a dev environment is not in any way equal of testing in a more production like environment. An SQL triggering a full table scan is way different with limited data in the database, and an environment with indexes maintained and refreshed is way quicker than any environment that's remotely neglected.

Developer performance tests can only be used to make sure the code CAN get good capacity/performance. There are numerous more obstacles due to environmental factos, infrastructural factors, data-setup (data volume and distribution), other things happening in the system at the same time (batch jobs, other requests). But it is relevant making sure the system code can deliver. That helps narrowing down any performance related issues identified - and hopefully you yourself can come out without blame.