Automation vs Manual Testing: Picking the Right One

Manual and automation testing are both used extensively for systems testing. This is the detailed testing that any application receives prior to release. Here, we look at when you should choose manual testing vs automation testing.

Introduction

Software testing has three goals: ensuring quality, checking reliability, making sure the system delivers what was required. Testing is important for any software development project. But certain industries require a higher level of testing. For instance, fintech applications must be 100% reliable and have to meet strict industry standards.

Systems testing is the most important phase of software testing. Typically, it is done with a mix of manual and automation testing. But what do those terms actually mean?

What is Manual Testing?

Manual testing is the traditional approach to software QA. There are two forms. In the first, you have a detailed test plan that you follow step by step. This lists exactly what you need to do and what the expected outcome is. Typically, these test plans (aka test cases) are stored in a test management system. They are used to check that the application is working as expected and that new features haven’t broken existing functions. This is known as regression testing. 

The other form of manual testing is more ad hoc. It involves interacting with the application to try and work out how to reproduce a bug that has been reported. As the tester goes, they make a note of each step they did. In the end, they have a set of steps to replicate the bug. Those can then be converted into a test case for future testing.

What is Test Automation

Test automation involves using a computer to run your tests for you. Some automation tools such as selenium or a good selenium alternative can be used for automation purposes. Typically, the term is used for UI testing, where the computer is interacting with your application in the way a manual tester would. However, there is also automation testing of APIs. Automation testing of UIs uses a test script to drive the UI via a web driver interface. The script selects an element in the UI and performs some action, such as clicking, or entering text. It then checks to see what happened and compares that with the expected result.

What tests can be automated

Not every test can be automated. So, how can you tell if a test is suitable or not? Here are 5 simple checks you can make:

1. Repetitive actions. Humans are bad at tests that require lots of repetitive steps. They are prone to making errors because they get bored or distracted. If possible, you should automate such tests.
2. Run frequently. Tests that need to be run really frequently should be automated. For instance, smoke tests should happen each time there is a new build. If all these tests are manual, it’ll slow your process right down.
3. Predictable outcomes. Automated tests only work when there is a clear pass/fail outcome. If your test sometimes gives a different result it is nearly impossible to automate it.
4. Business-critical. Some tests are verifying the fundamental behaviour of your application. If these fail, it means your application cannot and must not be released. These sort of tests should be automated to ensure reliable testing.
5. Complex logic. Many applications rely on complex logic. Here, you need to test every single path through the logic. Often, that requires repeating one test again and again with slightly different data. For instance, checking if an app adds the correct sales tax depending on where someone lives.

What tests can’t be automated

The following types of tests are really hard to automate.

1. Progression tests. These are new tests being developed to check new features or track down bugs. Here, ad hoc manual testing is the best way to develop the test. Once you know what is needed, you may choose to automate it.
2. Ambiguous outcomes. Tests with ambiguous outcomes have to be done manually. Ideally, the test case should be annotated to explain what to expect. Here, you can rely on the experience of the tester to determine if a test actually failed. 
3. Complex scenarios. Sometimes, a test is just too hard to script properly. A great example is insurance quotation forms. Here, the logic is complex and the form evolves depending on the answers being given to each question. That makes it nigh impossible to test with traditional scripted test automation.

How AI testing frameworks help

Nowadays, there’s a third choice for testing—AI. So, how does a framework for AI testing increase your test automation? The best way to show this is to look at the example of an insurance quote form. There are two reasons why this test is so hard to automate. Firstly, you need to add a huge number of steps to check each and every stage in the form. Secondly, every form of input affects the output. So, you end up with a test script that becomes exponentially complex. 

AI frameworks for testing solve both these problems. For starters, they allow you to record a test just by interacting with the UI as if you were using the application. This makes it much easier to create the basic test. You can then connect that test to a data source to test every different scenario. This is really easy with frameworks that offer test variables and proper test data management. The upshot is, suddenly you can automate complex test scenarios as easily as you can straightforward ones.