Web page testing – 5 BIG perks you get from automation

It is absolutely amazing what a difference automation makes. Not only for web page testing, but taking mundane tasks from people and assigning them to computers generally makes workers happier. Then, they can direct their time to more productive endeavours, right? Managers and stakeholders will be happier and more trusting of their business process. Customers are happier and safer. So what’s the catch?

Automation is not easy 🙁

Automation is not a one-size-fits-all 🙁

Automation is safer, but it is not bulletproof 🙁

Automated systems need maintenance 🙁

We’ve talked about web page testing automation in a previous article, so you might remember how Quality Assurance is crucial for reliable software delivery. But is it really worth it to test every thing, every time we release a new version of an app? It is cumbersome for testers to keep doing the same actions over and over again. Then, it is fairly impossible to cover all platforms and operating systems available out there. As it seems, testing is a perfect candidate for automation.

Software test automation is the process of using separate software to execute tests and compare actual outcomes with predicted outcomes for the system-under-test. But is it worth it? Keep reading to find out.

Perk #1. Time is money

Mundane, repetitive, thoughtless tasks are meant for computers. Consider a test suite that takes around 2 hours for a manual tester to run through. A big release comes along and they need test for regressions. Of course, they need to do this on multiple browsers and/or operating systems, so they reproduce that 2 hours-long activity a few times. That’s at least a day’s work of repetitive clicking, form inputs and waiting. Not to mention all the documenting of actual app responses, as they compare with the expected behaviour.

An automated web page testing suite runs much faster than a manual tester. In designing an equivalent automated test suite, you need to consider that:

• the same 2 hour manual test suite could take just a few minutes to run
• there is time flexibility; you can run automated tests in parallel, part of your continuous integration or continuous delivery process, only look at them if they fail etc.
• they can be reused as many times as needed without change. So if you spend 8h to develop the automated test suite, you will save that time back in short time. Maybe even a couple of releases. That is, unless…

The downside

Say your web app is changing rapidly. Or maybe the specifications are unstable. In that case, you will not get the same fantastic benefits from automated web page testing. Why? Well, you need to adjust the test suite too, every time you adjust the specs and/or the app’s behaviour. By the way, we thought about this when developing Trudon, our codeless QA automation tool. We use Artificial Intelligence to determine whether the app was changed in appearance, but not in functionality. Therefore we can continue testing, and warn about the small changes when reporting.

Furthermore, consider the skillset. Helping manual testers expand into automation can have an initial formative cost. Depending on the technology they need to learn, it can take from little overhead to learning serious programming skills.

Perk #2. Reliability

Let’s get back to our manual tester. He’s been at it for a good 4 hours now. And he’s getting into his third run of this test suite, but… things get a little boring and he starts rushing the process. It’s not that he is sloppy, or that he has any intention of slipping. It’s just that the human attention span is really not that great.

On the other hand, automated tests never falter from their set up path. Every step is performed in the exact same order it was designed to. This is a matter of self-documentation. Manual testers could stumble into a but without documenting anything, and then struggle to reproduce the bug. But with automated testing, we know the exact line where (in regards of automation step) the test failed, the data that was used at that time and the exact steps to reproduce. So, by automating, we give programmers a decent starting point for investigating a fix. And we need those clues to go from web page testing to web page fixing 🙂 

The downside

Don’t expect reliability if your app is non-deterministic. “Non-deterministic” means that an action performed with the app can have a number of different outcomes. In fact, you can hardly automate any tests on such an app, perhaps except for basic smoke tests. But more about that in a future article.

Perk 3 – A diversity of web page testing types

Not all tests are created equal. And you can always balance between writing more tests or diversifying your array of tests.

Some types of web page testing are inherently automated: unit tests and any type of non-functional test (load, stress, localisation, compatibility testing etc.). On the other hand, exploratory testing, where the QA team explores the app in search for possible issues, is by definition manual. But most test types out there support both practices.

In this case, the key difference is obviously efficiency. For integration testing, regression testing and system testing, a large chunk or the whole app need to be tested. That simply takes long for a manual tester (see perk #1). In preliminary testing methods, such as sanity testing or smoke testing, automation can do both the preliminary and the follow-up testing. This cuts time delays from formal acceptance, task assignment etc., making app delivery faster and safer.

Another interesting aspect is GUI testing. A combination of automated and manual overview is recommended. It is difficult for testers to appreciate delicate details such as 1-pixel alignment differences or subtle shade differences between colours. In fact, some test automation tools, such as Applitools, are specifically designed to compare expected and actual looks.

The downside

You simply might not need all these types of tests. The coverage target and the diversity of your tests should be cautiously considered. They also depend on your business processes and app size.

Another downside is in regards to acceptance testing. Most test automation tools are highly technical, so automated testing in the hands of a client is something rare and precious. With acceptance testing, the client checks the business flows end-to-end, but they can also delve into aspects of GUI or security testing. Easy test automation for non-technical people is in fact one of the key reasons that Trudon exists. But let’s get back to automation, in general.

Perk #4. Web page testing metrics

Automation gives you useful metrics, which supports you in having clear criteria for releasing. For example, here is how you may define your acceptance criteria for a new release:

• 100% passing unit tests
• 90% or more code coverage
• 80% or more confidence that 1000 simultaneous users will not break your app (per stress testing)

With simple manual testing, you release in the blind, without knowing which lines of code might be problematic. Even more so, without having a clear idea of performance bottlenecks or how and why they happen. Conversely, with automated web page testing, you get loads and loads of metrics you can combine and use to drive your business process.

The downside

As with the multitude of testing types, you just might not need complex metrics. For a simple web app, or one that does not handle high-risk processes, you only need a few basic numbers.

Perk #5. Headspace

If I were inclined towards philosophy, I would say automated testing it is a way of life. Because, to automate something, first it has to be in good order mentally. Web page testing is not an exception. ROI is always better for companies that put QA first. But not because automated testing is some sort of magic. It is the fruit of a coherent, pragmatic and cost-effective approach to releasing web apps. It gives testers (and the entire team) freedom to improve. Freedom to innovate and the comfort of being people, not broken records.

And, if you are expecting the downside… sorry, but there are never downsides to headspace.