There are a lot of factors involved in creating a successful test automation. Examples of areas that should be considered for successful test automation include:
Each of these could potentially ruin the best of intention in test automation. However, not having test automation at all often is a really safe way to make any development fail in the long run.
Happy learning, or happy confirmation that you are already skilled.
By far the most common reason to automate is to shorter the feedback loop from some code is written or any system change is made utill a confirmation that everything works as expected is received.
With manual testing this could take days or weeks, but with a structured test automation that is adapted to the context this could be down to minutes.
With shorter feedback loops it's far easier to fix problems (while the code change still is in the coders head) and it prevents a lot of badwill from annoyed end users that othervise (hopefully) will find the problems late when everybody probably are stressed over meeting deadlines anyway.
Test automation often also introduces a technical element to testing that sometimes just isn't there before. This enhances communication within the team a lot if there has been an invisible wall between dev and QA before.
With manual testing you almost always have to pick what tests to execute for each change. With a solid and fast test automation all tests can be run at all times.
The value of testing is the insights gained from the test execution. A lot of the work performed around the testing activities are only work performed to navigate into a position where structured testing can begin.
The more of these steps that are automated, the more value could be squeezed out of the testing efforts.
TBD
There are a lot of circumstances that would render test automation un-adviseable. For example when testing already is performed in an unstructured manner, or a combination of only a few releases per year and externally developed system under test.
Other things that would raise warning flags are no appointed maintenance organization for the test automation since then you cannot make sure these stakeholder get something they like.
Often test data has been something that people state as ground for choosing not to automate, but there always seem to be a way of finding usable test data, so this should not be any showstopper.
Test automation also should be avoided if the goal is to save testing money, or if the release frequency is fairly low, or the mandate over the system under test is low.
For a better understanding of the test automation situation there is a very well-proven Test Automation Assessment you may use.
Automation can often be performed on different system levels, and each have their benefit and drawback.
Not all systems have a GUI, and there are systems with no APIs. There are also a lot of organizations that don't develop the software they use themselves. This limit the scope of what is possible to test with enough ease to make it worth while.
Unit testing is really good to have. It provides a safety net for cascading errors from changes to code or code structures. Without unit testing the development will become way harder and more risk prone the longer the system has been developed.
We've all seen systems where the developers are reluctant to make nessesary changes because of the risk of breaking the code base.
This is why unit testing is essential to all development.
Often all of the tests maintained by the developers are called unit tests.
There is a notable difference though.
Sometimes developer tests include API tests that conceptually is implemented on the system test level (testing the whole system) rather than the unit testing level.
However these often target to be executed in a CI scope with a limited data set containing only constructed data.
This types of tests often are quite easy to extend to be used with the kind of test data that exist in the acceptance test environment.
Testing on the API level often is a really alternative. Most APIs are quite fast, they may be invoked in parallel, and the changes of tests when the API:s are updated are small.
Often API level tests are suitable both for data driven tests, invoking the same API:s with different data to assess the results, and for connectivity tests for integrations.
API:s also often are suitable for test data preparations by inserting data through the API:s and alter them until they are useful for a certain type of check.
There are a lot of different types of API:s but most of them share these characteristics.
One special case is Message Queueing, that some consider an API (given broker channels exist), but the sequential nature of MQ might affect test outcome.
GUI level testing often is the only way you can be certain the full business flows will work for end users.
The main challenge from GUI level test automation is that it require a lot more maintenance than most other types of test automation. This is due to that there is so much that can change in an application.
Any change to the GUI, the data structure, the test flow/sequence require a change to the automation. A test automation could be considered to be a system of its own. Looking at the test automation
relationship with the system under test it's probably safe to say it's the system with the most depencies of all to the system under test. Each test require interaction with a lot of dependency points (GUI elements).
GUI level test automation hence require an even more thought-through architecture than any other type of test to be bearable to maintain.
To ease the maintenance burden some tools make use of AI or clever algorthms. This could help to some extent, but engaging in GUI level test automation always require more effort than for example API level test automation.
Another consideration is that GUI level testing rarely tests the GUI very efficiently. Humans make use of their serendipity and awareness to identify strange behavior or looks in the GUI,
but if you create test automation to check the same amout of details you'll end up with a change-notifier that will annoy everyone.
There is also the thing with confidence. Often business experts will feel the need to test things themselves to feel confidence in the changes introduced. The business users mainly interact with the system through the GUI,
and if they always test the system regardless, they'll find any present errors.
When automating at the GUI level it's extra important to be clear on what you'll test. The tests you don't automate generally is performed by people. If they are not clear on what the test automation checks they might
end up testing the same tests that the test automation performs regardless, removing a lot of the value of the test automation. Of course it could still provide fast feedback to make sure most bugs are mitigated
before human testing - giving smooter QA process and happier human testers. This could be highly benficial when a development team struggles with miscontent and mistrusting business representatives, where the margin for
badwill is low.
Scoping tests to automation in whole areas and communicating to testers what parts of the application they don't need to test thorougly is essential here.
Given the mix of human testing and test automation the GUI based test automation is the one that most often benefit from integrations with a test management tool for realtime updates of test/requirement fullfillment
status in a human readble format.
The most common types of GUI based test automations are:
Even if modern day javascript frameworks generate a lot of the GUI code from models, it's often so heavily customized there is a risk of integration failures to backend. Even in a highly agile team, including end user stakeholders that continuously verifies that new or changed features works in the intended way there is still a risk of something breaking if the regression testing doesn't include some checks performed in the GUI.
Common pitfalls in GUI automation:
The organizational responsibility for test automation maintenance is often widely debated within an organization. Placing the responsibility on the wrong party is a sure way of having a test automation that is slowly dying.
Different organizational constructions has their pro's and con's.
Some organizations consider test automation a special skill, and use Enterprise level tools with wide technology support and special skillset needed to operate them.
| Pro's | Con's |
|---|---|
| No need to convince reluctant developers to do test automation. | Can never become proactive. Will always be slow and reactive. |
| Efficient use of tool licenses. | Easy to get tool support given vendor agreements. |
| Test automation Center of Excellence continuously evolve. | Always neglected and forgotten by the dev. teams. |
| Encourages standardized solutions wich eases handover to new staff. | Test automation maintenance secured even in maintenance phase of the system under test. |
| In organizations that get their IT systems from 3:rd parties (outsourcing, vendors) this could be their only chance testing that their business processes will continue to work with upgrades. |
It's really hard to get this type of test automation going and working in the long run. Most of their time will be spent on marketing their own services, and running around making sure they are in the communication loop.
A very common approach is to enable the tester team to use automation to ease their own work load. This is particularly common in cross-functional teams with dedicated testers.
Surprisingly often the testing team has to fight for the right to do this.
| Pro's | Con's |
|---|---|
| Testers knows of all the challenges of test environments and test data and designs the automated tests to cope with this accordingly from start. | Tester super power is to explore and deduct information about the system under test. Automating test cases may be a distraction from this. |
| It's easy to know what is not automated and needs to be tested manually. | Tester rarely are very good programmers. Code maintenance could become a struggle. |
| The tester insights in what to test and how makes it easy to redesign tests for automation if needed. | Each team will end up with their own way of working and possibly even their own toolset. This could make long term maintenance a struggle. |
| Test management tool integration become natural. | Test automation is quite efficient since it is higly adapted to the local test situation. |
| Change communication flows naturally. | Testers embrases fuzziness since it teaches them about the system - in a way that other roles don't. |
| Test automation will happen. No threshold to overcome in any way. |
In teams where the developers don't want to participate in testing activities at all this approach could be highly beneficial.
In theory the most suitable way of working with test automation would be if the developers themselves took care of it.
| Pro's | Con's |
|---|---|
| Developers are really skilled at creating code prepared for ease of maintenance. | Developers in general hate uncertainties and program the tests around anything that could break. |
| Ease of test version control. Tests will be in the same repository as the code. | The code centric view of developers makes these tests miss a lot of common problems with infrastructure and data. |
| Tests are easy to implement in CI/CD pipeline. | The developer only tests the assumptions he/she already has taken care of in coding. |
| Automation in the same tool as the code development takes several hinders away from doing automation. Very little extra skillset needed. Almost no extra tooling. | Developers has very limited understanding of the challenges of testing and find test automation above unit test level tedious in the long run. |
| Developers learn about the obstacles and challenges of testing and naturally focuses more on testability. | Hard to find priority for test implementation. The understanding of the need for system level testing is typically very low since business process responsibility includes operation and infrastructure and many more. |
| Really quick feedback. | Developers really doesn't see the point of GUI level test automation. |
One common argument while discussing test automation is developers saying something like:
"Yeah, well, we don't need help with that. We can do that ourselves.".
When countered with "Yes, I understand that, but you have always had that possibility but never taken it. What has been holding you back from doing that? Has anything changed now that will make this a priority or will you still stay at the un-used ability to do this?"
Often the core problem with developers lack of interrest in high level test automation become apparent.
Of course the examples above are simplified and driven to some kind of edge. Out there different teams and organizations are trying their best to cope with the limitations of their specific context.
Common hybrid setups include:
This will always be slow and reactive and taking care of the test automation will always be of lower priority than producing new code - making the test automation an annoying burden within a few months.
This is a way of getting traction for the center of excellence, but constant team participation will be required in the long run.
The results from the test automation are often formatted in a channel and in a format that the person that implements the tests understand and appreciate. However a lot of times the benefit of test automation will be greatly enhanced if any other stakeholders are identified and the test results propagated in a format and channel they appreciate.
A lot of time it makes sense to integrate test automation to test management tools for ease of understanding and ease of status understanding. Sometimes it is also relevant to integrate with version control systems. This gives access to the extra information about what has been changed in the application since the last time the tests were run.
One of the benefits of test automation is that it executes tests faster than any human tester would ever do. This is especially relevant when for example verifying panic patches to production. But incidents in production could occur a long time after the code has been deployed to production and the development of new versions of the system is likely to have gone a long way. If the test code is kept up to date with the most recent changes to the code base the test cannot be used to verify the panic patch unless version control of the test automation code exist.
Of course the best thing for the test automation code is to reside in the same code repository as the code of the system under test since you then get a robust version control of it automatically, but there are a lot of circumstances that could render this impossible.
For example the test automation could be developed by another team, in another programming language that would fit the repo badly, or the system code is written by a 3:rd party.
During these circumstances it is higly encouraged to perform version control of the code anyway. Maybe in git, SubVersion, ClearCase, or whatever suits you and the organization. The purpose is to be able to execute the test automation as it was in a previous state if needed.
The general advantages of version control would be a bonus:
Normally the development and test process makes it natural for when in the workflow it is natural to automate tests. Two main schools of thought exist.
Regression tests are tests that are run to repeatedly verify expected behavior. Typically it tests that nothing in the system has broken when a change somewhere is introduced.
Various methods of limiting the regression test scope for faster test execution exist, but generally the most common method is that all regression tests are run for each change.
Typically this is not a big problem since test execution duration is not that long, but sometimes it is - especially when combinatory complexity exist, like performing the same test in several browsers.
One relevant argument for this approach is that you need to explore functionality before you know what could be problematic and hence needs to be covered by regression tests.
Another argument is that humans are really good at noticing stuff outside of the test script. This is called serendipity and is something no test automation can perform.
Some emphasise that test should be written before any change to the system is performed. Not only should the tests be written. They should also be executed before any code is written. It the tests don't fail they need to be updated or no change is needed in the first place.
For some developers this seem to be a well working practice, but for many others the creative code writing process has far to much of exploring involved to be very useful. The problem for them is that so much knowledge has to be desided up front to create a test. Data formats, object fields, technology choices. Sometimes it's easier to explore a bit before settling for a workable way of solving the problem.
As an abstraction that makes you think through the expected functionality thoroughly the Gherkin syntax was developed. Through implementations like Cucumber/Specflow (among other) the expected behaviour described in Gherkin syntax can be mapped to an underlying test automation.
When automatic execution of tests written in Gherkin is in place something called Living Documentation is achieved. When a requirement is changed, the test is changed with it (or in some cases at least the test automation get notification that it needs to implement something new).
The Gherkin syntax in its basic form is based around behaviour level keywords:
GIVEN that a customer is logged in and registers a new issue
WHEN the customer pushes the Send Button
THEN he should receive a notification
AND the system should notify an administer.
Of course the Gherkin syntax descriptions can be immensely complex if they are large and detailed.
One drawback of Gherkin syntax is the times the Gherkin formatted requirement are so tightly bound to the code that they are only visible to the developers. The main goal of the Gherkin syntax is to make communication of expectations between the business and developers more easy. If only the developers can reach the requirements in this format they've only given themselves an extra layer of complexity since they could read the test code directly.
What types of tests to automate is widely debated. To some it seems really natural to automate unit level tests on method or class level. Some want to include performance related tests in the unit tests. Some want exploratory style tests like web crawlers (web crawlers are verifying no link on a web site returns an error code), or generic GUI style guide compliance tools.
These types of test activities are encouraged for automation:
Test involving batch jobs, or system integration, are generally a lot harder to get to run stable over time.
What you don't want to end up with is too many tests. Then it become brittle and an annoyance. You don't want change notifiers that gives you notice from the smallest change in the system, but rather verified functionality.
One challenge with test automation is getting hold of relevant test data that will give consistent results.
Test automation execution often consumes a lot of test data.
For example when booking flights on a test systems the chairs of a plane will eventually run out, making tests fail if no test data strategy is applied.
Many types of systems also rely on specific dates, or date intervals,
that affects the values and data in the system. In these cases the execution date is relevant for test outcome.
For example many insurance systems and financial systems calculate interrest on a basis of the amount that was on an account a specific date,
making the interrest value different in different times of the month.
Constructed data is data that is inserted into the system at runtime - as a first step of a test case.
This works fairly well in lower level testing. Inserted data can be safely created and identified without risk of collisions or dependencies to other data in a system with very limited data.
On higher testing levels there often is a lot of background data already in the system. Then constructed data need to be easily identified (often including a timestamp and test case name, and some words that make acceptance testers understand they should not interact with this data).
One problem with constructed data is that it tends to become rigid and un-dynamic, and thus giving a false sense of safety because they do not trigger any problems that production data would.
One way of dealing with this problem is making the data used in testing dynamic. For example you could select any customer matching your criteria by some kind of random function. This could eventually trigger errors from new states of data och new complexities, thus giving us new insights about the application under test and its context.
One good thing with randomization is that you can store the seed value used. Then you can generate exactly the same random data again. This is good since othervise the random nature of these kind of test data makes debugging of failing tests extra complex.
TBD
In test environments with a lot of production like data it could be beneficial to identify test data already in the system rather than to construct data of your own. Identifying test data generally is a lot faster than constructing data when the data structures are complex.
Another benefit of identifying data rather than constructing data is that you don't need to keep track of modification of the data schema. This makes the test maintenance less time consuming.
The challenge with using identified data is that it initially produces a lot of false negative test outcomes until all relevant filters for usable test data has been identified and implemeted, or management of alternatives has been covered in the test implementations.
TBD
When deciding in what order a system is built, or when a test automation is implemented, you often start with the point where data first is entered into the system. This is convenient since the rest of the functionality depend on data in the system.
The mechanisms for bringing data into the systems then often are re-used for test cases where data is altered by any means of interaction.
In many types of testing it is relevant to attempt to clean up used data afterwards.
This is for example nessesary when using data that is used in aggregated form in some kind of test.
Horrible example
Say for example you need a new customer buying stuff and the test case fails so it end up in an unknown state.
This will affect any type of report tests that uses reports over sales during the last couple of days.
A good practice would make the report test independent (creating it's own sales and checking diffs),
but sometimes this is to cumbersome and you have to make sure each test produces.
TBD
Test automation cost money in implementation, in maintenance, in delay time, and in test output analysis time.
The value of test automation is the increased trust in the system that comes from execution. The more often the tests are run could increase this trust in the ability of the system, but often this value increases a magnitude for each new production like aspect that is introduced for the tests.
To avoid the horror of root cause analysis from errors with a lot of changed things, tests should run on small changes, and tests should run in a number of different environments, each introducing new levels of complexity.
Running code test on the CI (or Dev) server, internal integration tests on the Test environment, external integration on the Staging (or Acceptance) environment not only reduces the risk of introducing corrupt data into external test systems (affecting future test results) but also makes identifying errors a lot easier.
In the ideal world we can re-use the same tests in several environments. If the cost of execution is zero it doesn't have to have a lot of added value to be beneficial. Of course some types of tests are prioritized in specific environments, but often it doesn't hurt running extra tests.
TBD
Many systems also has periodic batch jobs that alters the data during the night, or at certain times of the month/week/year. Sometimes the test sequence include running these batch jobs.
To make test execution efficient it's a benefit if these jobs can be triggered through an API (or, as fallback, on GUI level automation) or the test execution is limited to certain times per day or months, bringing down the value of many types of tests to nearly zero.
All systems of today have integrations to other systems. In fact it's more common that a system is used by another system than being used by humans. Human involvement is avoided unless nessesary due to decision making or data input that needs manual attendance.
With test automation in an environment that has integrations to other systems (rather than to mocks),
The naming convention of environments differ a lot between organizations. What differs between them however doesn't differ all that much.
| Name(s) | Data | Owner | Integrations | Security | Note |
|---|---|---|---|---|---|
| Dev/CI | Constructed, semi-chaotic, minimal (often memory database rather than persisting DB) | Dev team | Most integrations are replaced by fakes | Wide open to ease debugging | Mostly for unit testing and developer manual testing |
| Test/Ver | Constructed, structured, limited. Proper DB. | Dev team | Mix of stubs, mocks, actual integrations | Quite loose OS/network level security. Wide access to own system. | Often used in stakeholder testing/demos |
| Acceptance | Anonymized production copy | Operations | Actual integrations to a mix of test versions and production versions of other systems | Locked down to production security (network/OS/file system/firewalls), but with test accounts with a range of user rights. | Feasible for performance testing. Often used in demos. |
| Staging | Production data | Operations | Actual integrations to a mix of test versions and production versions of other systems | Locked down to production security | Meant for fail-over, and for production preparations. |
TBD - table
For your convenience the following is a checklist for test automation assessment. It provides headlines for scope so all aspects affecting test automation is known before initiatiing a test automation.
Open: TEST AUTOMATION ASSESSMENT CHECKLIST