Too much test automation?

The other day, a QA engineer was telling me about his legacy at his last job and mentioned that he was most proud of transitioning the teams manual tests to automated ones. Wow, great achievement I thought until he started the number game. We automated 57 tests in the first month he said and were able to run 4 times as many test cases within the first quarter. At that point, he lost me because tests are not about how many you have. Are 57 automated tests better than 10 manual ones or even 1 well written automated one? How many of you have seen test reports that show that 95% of of the 2000 tests have passed? Does that number really mean anything? What about the 100 that didn’t run or were failing? Were those critical functions?

That QA engineer would have impressed me if he would have told me that he developed an automated test suite that covered the functions most used by the customers and ones that are business critical. It doesn’t matter if that would have been 50 tests or 10 tests or even 5 tests because the number is irrelevant. The important aspect is the coverage of the tests and not the number.


When I asked the same QA engineer what was most important in the tests he wrote, he gleamed with pride as he said: 1) Clear Variable Names and 2) Lots of Comments. Hmmm.. Is that really what is really most important? What about how long it takes to run or more importantly how to avoid the pesticide paradox with the tests. For those of you that are not aware of the term pesticide paradox, to me it means that a test can only find bugs for what it was programmed or designed to find. Any variation on that will go unfound. Test automation can avoid this by built it variation on paths, data or even permissions.

Is there such a thing as too much test automation? If you read this far, I hope that you realized that the question itself is should have been phrased differently. There can be inefficiencies and redundancy in the test suite. There can also be too much automation coverage when the tests are covering “dead code” or irrelevant features. The question should have been “How do you know what your automation test suite is covering?”. When you answer this question it shouldn’t be with a number of tests that you have. Instead, it should be with what functionality is covered.

A test automation suite is a living entity that needs to be constantly updated to remove redundancy and ensure that the test are covering the most critical aspects of the system as new features are added/modified/removed. It should be our goal as test engineers to inform others who ask the as the question of “how many test cases do you have?” that the answer is not in the number of tests but instead is in the coverage and dynamics of the tests.

Read More

Automated Testing Tricks – Tickets to see Jimmy Fallon

Automated Testing Tricks is a series of posts that discusses fun things you can do with your automated testing tool that may not be directly considered automated testing.
“Hey, Hey Hey, Hey!” Would you like to go to a live taping of the Tonight Show with Jimmy Fallon? I sure did, and when my wife and I were given the opportunity to spend a long weekend in the Big Apple, we pounced on the chance to get tickets for the the show.
After doing some digging, I learned that tickets are free (Woo-Hoo!!) and distributed on a first-come, first-served basis online (Uh-oh.).
Like most people, I work long hours, coach youth sports, spend time with my family, etc. I’d have a better chance at bumping into Jimmy in a spinning class than getting tickets in a “first-come, first-served” contest.
So that got me thinking: is there some way that I could be alerted as soon as the tickets become available so that I can be one of the “first” without aimlessly clicking the Refresh button on their website over and over again?
Thankfully the answer is “Yes!”

How to get tickets

First things first. How do you get tickets for the Tonight Show online?
The Tonight Show has a reservation page where they list the dates that they’ll be filming over the next few months.
Main Page
If you click on any of the dates you’ll quickly realize that they’re all sold out. This is ultimately the result of not beating everyone to the front of the line for the “first-come, first-served” tickets.
sold out
But the good news for you is that they only release the filming dates a month or two in advance. Therefore, if you plan accordingly, you have a fair chance at getting tickets for dates that aren’t listed yet.

Use your Automated Testing Tool to alert you

There are a number of automated testing tools that you can use to test websites. My personal tool of choice is Tellurium, but you could use other tools like Selenium, HP, etc. The important thing is you’ll need to be able to schedule your test to run frequently and receive alerts when your test fails.
The first thing we need to determine is: what is it that we’re testing?
Since we’re looking for a particular date (ideally one that isn’t listed on the reservation site yet), we’ll want to create a test that checks to see whether that date appears on the page.

Creating your test

There are a few pieces to this test and we’ll walk through each one of them.
The first thing your test needs to do is navigate to the reservation page and check to see if your desired month is listed. This will most likely mean clicking the “Next” arrow to check for a future month.
Once your test finds the month that it’s looking for, you’ll want it to check for the specific date on the calendar. Thankfully the dates are links, so locating them via an automated test isn’t difficult.
If your test can click on your date, and the reservation drop-down menu appears, then your test should FAIL.
Why fail? Some testing tools will notify you whenever your test fails. In this case we WANT to be notified as soon as that drop-down menu is available. Therefore we need to trick our test into thinking that it’s failed so that it’ll notify us.

Personally, I was hoping to get tickets for June 27th. Using Tellurium, I created the following test:

For the url

If the text “May” is present

If the “Next” link is present

When the “Next” link is clicked

Once the text “June” is present

When the “27” link is clicked

If the “event_time” drop-down menu is present

Then fail this step


Then pass this step





Feel free to copy/paste that test into Tellurium and try it yourself. Simply replace “May” with the current month, “June” with your target month, and “27” with your target date.

Schedule your test to run automatically and notify you if it fails

Once you’ve created your test you need to schedule it to run automatically. Tellurium has a built in scheduler, but you could always create a cron job or use some other scheduling mechanism if you’re using another tool.
Ideally you want your test to run as frequently as possible so that you’re notified as soon as you can reserve tickets on the site. I set my frequency to every 5 minutes, but you could theoretically set it to run every minute.
Once you’ve scheduled your test, make sure that your testing tool notifies you if your test fails. Most tools have a mechanism to email you if and when critical tests fail.
And that’s it! If you set everything up correctly your test will automatically run every few minutes, checking the site to see if you can reserve tickets for your desired date.
Now you can sit back and wait for the email to arrive so that you can get your tickets before anyone else.
Have you used your automated testing tool to reserve tickets online? If so, share the details in the comments below.

Read More