Every day I hear the same question: “how can I improve my coverage with little effort?” Of course, this is a loaded question. What coverage do you mean?
There might be (at least) four ways to think about coverage:
- Code coverage
- Actions and Pages coverage
- REAL user flow coverage
- TEST coverage
In the past we defined them this way:
- Code Coverage: Did we activate every measurable part of the code? This was generally difficult to get near 100% as it would require 1000’s of scripts. In general teams achieve between 5% and 30% code coverage. To know you must instrument your code with any variety of tools which will watch to see if that section indeed gets activated during testing.
- Actions and Page Coverage: Did we get to every unique page and execute every possible action in the application? While this is time consuming to write and maintain scripts, often less than 1000 scripts may be able to achieve this. In general people achieve 40% to 80% today. While many strive for 100% but of course the more you get the more maintenance you have.
- Real User Flow Coverage: Do we execute every user story against the new build that real users actually followed in production? This has been an impossible dream (the tech did not exist), but is the ultimate test. In general people achieve a small percentage of this today (and in any case the value is not known), not knowing how people actually use the application.
- Test coverage: This is the metric QA has often used but it is a bit pointless when you dig into it. Did we test what a BA told us to? We don’t know if those user stories will ever be used…but that’s what BA’s want so we will do that. And often we say we have close to 100% test coverage. But that is essentially implementing tests which are a guess of user stories and have no relation to actions, code or anything else. So test coverage is only as good as the guess of the business analysts or product manager. Are they 20% correct? 80%? Who knows. How does it relate to everything the app can do? It doesn’t. What about what real users do? No relation. It’s just a guess, and thats not a very scientific method of measuring results.
So that brings us to modern testing which has made use of AI generated scripts now for a few years (no recording/scripting/writing – meaning fully machine generated.)
And the new question is: “How do we know we are achieving the desired coverage with AI generated scripts”
Again, we have to go back to our four definitions above to better answer this.
With Appvance IQ, we have seen the following typical results:
- Code Coverage: Did we activate every measurable part of the code? The AI Blueprint creates thousands of user stories by itself, no code or recording, following the hints and validations, to achieve close to 100% each time. Machine generated.
- Actions and Page Coverage: Did we get to every unique page and execute every possible action in the application? The AI Blueprint creates thousands of user stories itself, following the hints and validations, to achieve near 100% each time. In fact by definition, AI blueprinting must execute each and every action and get to each unique page.
- Real User Flow Coverage: Do we execute every user story against the new build that real users actually followed in production? Since the AI regression test generator recreates the actual user flows from production (with no additions to the applications code) and applies them to the new build, this is now the gold standard of testing. Will the new build work for our users the way they have been using it? The answer is yes. Typically 5000 or so regression tests are created, data driven, providing near 100% coverage of actual user flows. In fact one can setup the generation to create every user story it sees from production until it runs out of unique flows. Achieving regression coverage above 99% each time.
- Test coverage: Did we test what a BA told us to? Given the near 100% results of the categories above, this category becomes far less critical, since these user stories are simply created by the BA. However, it’s good practice to create these stories with scripts (say in AIQ Test Designer), which is 20X faster than creating them in Selenium. If for no other reason than to please the BA. Of course, there is no “black box” in any of the above tests. One can open any machine generated script to review it and even replay it. But by the time you would describe a specific user flow or even compare it to others, you would indeed have written it in Test Designer (which simply records the QA engineers actions and creates the editable script). So the combination of Test Designer scripts to satisfy your BA, with thousands of AI generated user stories, seems to be the killer combo.
What coverage do you want to achieve? Ask yourself: Code? Actions? Real regression? BA test coverage? Maybe some combination?
The technology exists today to help you achieve all of these with less effort than achieving just one using legacy scripts. If you want to achieve more, including true regression, now might be the time to consider AI script generation to augment your efforts.