Too much test automation?

The other day, a QA engineer was telling me about his legacy at his last job and mentioned that he was most proud of transitioning the teams manual tests to automated ones. Wow, great achievement I thought until he started the number game. We automated 57 tests in the first month he said and were able to run 4 times as many test cases within the first quarter. At that point, he lost me because tests are not about how many you have. Are 57 automated tests better than 10 manual ones or even 1 well written automated one? How many of you have seen test reports that show that 95% of of the 2000 tests have passed? Does that number really mean anything? What about the 100 that didn’t run or were failing? Were those critical functions?

That QA engineer would have impressed me if he would have told me that he developed an automated test suite that covered the functions most used by the customers and ones that are business critical. It doesn’t matter if that would have been 50 tests or 10 tests or even 5 tests because the number is irrelevant. The important aspect is the coverage of the tests and not the number.


When I asked the same QA engineer what was most important in the tests he wrote, he gleamed with pride as he said: 1) Clear Variable Names and 2) Lots of Comments. Hmmm.. Is that really what is really most important? What about how long it takes to run or more importantly how to avoid the pesticide paradox with the tests. For those of you that are not aware of the term pesticide paradox, to me it means that a test can only find bugs for what it was programmed or designed to find. Any variation on that will go unfound. Test automation can avoid this by built it variation on paths, data or even permissions.

Is there such a thing as too much test automation? If you read this far, I hope that you realized that the question itself is should have been phrased differently. There can be inefficiencies and redundancy in the test suite. There can also be too much automation coverage when the tests are covering “dead code” or irrelevant features. The question should have been “How do you know what your automation test suite is covering?”. When you answer this question it shouldn’t be with a number of tests that you have. Instead, it should be with what functionality is covered.

A test automation suite is a living entity that needs to be constantly updated to remove redundancy and ensure that the test are covering the most critical aspects of the system as new features are added/modified/removed. It should be our goal as test engineers to inform others who ask the as the question of “how many test cases do you have?” that the answer is not in the number of tests but instead is in the coverage and dynamics of the tests.

Read More

Testing Skills in Life

I live in Iowa. The winters are very cold and the summers are very hot. I live in a house built in 1920, so I don’t have a functional garage. Heck, I call it “The Shed.” A couple years ago, my husband got me a remote starter. I LOVE it…no ice scraping in the winter, nice cool car in the summer. However, to make that happen, I’ve figured out I need to preset my temperature controls, the window settings and the radio.


What’s that got to do with testing? The most important step before starting a testing effort is to be sure your environments and data are preset so the iteration can be as successful as possible. With my remote starter, I make sure my car is preset for the most successful morning.

Read More


I recently changed companies. With my involvement in the local QA association as well as talking with ex-coworkers, I figured this company was quite mature in QA processes. And they are very mature and done a great job of “selling” QA throughout the organization. I figured my transition would be quick and slick. I was wrong.

It’s interesting to consider as software testing professionals, we succeed by verifying the end results meet our customers’ expectations. However, to really excel, we start at same beginning as the developers and must come to the same outcome but by a completely different, but valid, route. What made me think a similarly successful testing department would have reached that feat with the same answers?

I often hear, usually from some shared FaceBook post with an inspiring sunset background, to really grow we must step out of our comfort zone. Of course what is the saying about such things? A cliché becomes a cliché because it is true. In the seven weeks I’ve been at my new company I’ve learned more about testing and the function of a testing department than I learned the last 3 years at my old company.


We need to do the same thing when it comes to our testing skills. I hate UI testing. I find it tedious. But I know because I took the time to learn UI testing. On the other hand, I love writing SQL queries which is surprising because I avoided anything like programming for most of my life. But then I took a chance and now consider ETL testing my specialty. The more experiences we encounter, the more knowledge we have to draw on to create the best testing strategies for our situation.

So where is the next place I can explore? Besides the ins and outs of my new company (and there are a LOT of ins and outs!!) I know I need to get a better understanding of automation. Any suggestions of where to start?

Read More

Automated Testing Tricks – Tickets to see Jimmy Fallon

Automated Testing Tricks is a series of posts that discusses fun things you can do with your automated testing tool that may not be directly considered automated testing.
“Hey, Hey Hey, Hey!” Would you like to go to a live taping of the Tonight Show with Jimmy Fallon? I sure did, and when my wife and I were given the opportunity to spend a long weekend in the Big Apple, we pounced on the chance to get tickets for the the show.
After doing some digging, I learned that tickets are free (Woo-Hoo!!) and distributed on a first-come, first-served basis online (Uh-oh.).
Like most people, I work long hours, coach youth sports, spend time with my family, etc. I’d have a better chance at bumping into Jimmy in a spinning class than getting tickets in a “first-come, first-served” contest.
So that got me thinking: is there some way that I could be alerted as soon as the tickets become available so that I can be one of the “first” without aimlessly clicking the Refresh button on their website over and over again?
Thankfully the answer is “Yes!”

How to get tickets

First things first. How do you get tickets for the Tonight Show online?
The Tonight Show has a reservation page where they list the dates that they’ll be filming over the next few months.
Main Page
If you click on any of the dates you’ll quickly realize that they’re all sold out. This is ultimately the result of not beating everyone to the front of the line for the “first-come, first-served” tickets.
sold out
But the good news for you is that they only release the filming dates a month or two in advance. Therefore, if you plan accordingly, you have a fair chance at getting tickets for dates that aren’t listed yet.

Use your Automated Testing Tool to alert you

There are a number of automated testing tools that you can use to test websites. My personal tool of choice is Tellurium, but you could use other tools like Selenium, HP, etc. The important thing is you’ll need to be able to schedule your test to run frequently and receive alerts when your test fails.
The first thing we need to determine is: what is it that we’re testing?
Since we’re looking for a particular date (ideally one that isn’t listed on the reservation site yet), we’ll want to create a test that checks to see whether that date appears on the page.

Creating your test

There are a few pieces to this test and we’ll walk through each one of them.
The first thing your test needs to do is navigate to the reservation page and check to see if your desired month is listed. This will most likely mean clicking the “Next” arrow to check for a future month.
Once your test finds the month that it’s looking for, you’ll want it to check for the specific date on the calendar. Thankfully the dates are links, so locating them via an automated test isn’t difficult.
If your test can click on your date, and the reservation drop-down menu appears, then your test should FAIL.
Why fail? Some testing tools will notify you whenever your test fails. In this case we WANT to be notified as soon as that drop-down menu is available. Therefore we need to trick our test into thinking that it’s failed so that it’ll notify us.

Personally, I was hoping to get tickets for June 27th. Using Tellurium, I created the following test:

For the url

If the text “May” is present

If the “Next” link is present

When the “Next” link is clicked

Once the text “June” is present

When the “27” link is clicked

If the “event_time” drop-down menu is present

Then fail this step


Then pass this step





Feel free to copy/paste that test into Tellurium and try it yourself. Simply replace “May” with the current month, “June” with your target month, and “27” with your target date.

Schedule your test to run automatically and notify you if it fails

Once you’ve created your test you need to schedule it to run automatically. Tellurium has a built in scheduler, but you could always create a cron job or use some other scheduling mechanism if you’re using another tool.
Ideally you want your test to run as frequently as possible so that you’re notified as soon as you can reserve tickets on the site. I set my frequency to every 5 minutes, but you could theoretically set it to run every minute.
Once you’ve scheduled your test, make sure that your testing tool notifies you if your test fails. Most tools have a mechanism to email you if and when critical tests fail.
And that’s it! If you set everything up correctly your test will automatically run every few minutes, checking the site to see if you can reserve tickets for your desired date.
Now you can sit back and wait for the email to arrive so that you can get your tickets before anyone else.
Have you used your automated testing tool to reserve tickets online? If so, share the details in the comments below.

Read More

What the World Cup can teach us about testing


Every 4 years the world comes together for the World Cup.  32 teams from all over the globe converge for the coveted title of the best in the beautiful game.

While watching some of the games it dawned on me that there are a lot of similarities between soccer and being a tester…

There’s a lot of pressure on you – like Brazil

Brazil is hosting the World Cup, and they’re also one of the favorites to win it.  That’s a lot of pressure on a team that, if they were playing in Europe, wouldn’t even be considered one of the top 5 teams in the tournament.


In Brazil’s first game they looked nervous.  You could feel the entire weight of the country bearing down on them, especially for the first 30 minutes.  They lacked creativity, their opponent was finding holes to exploit, and they collectively weren’t their best.

This is similar to when you take on a new testing project.  Management wants results right away, but where do you start?  You can’t test the entire application right away, just like you can’t win an entire soccer game in 5 minutes.

Brazil overcame their pressure by breaking the game down into small victories.  Step-by-step they started stringing together passes.  Then they got a boost by scoring a goal; it wasn’t a pretty goal (by Brazil standards), but it counted none-the-less.  Before you knew it their confidence had grown to the point where you could tell that they were in the driver’s seat and the game was theirs.

The same can be said about testing.  You can’t get overwhelmed by the enormity of a project.  Break it into smaller victories that will give you and management confidence moving forward.

You need to be methodical – like Germany

“Full coverage.”  Often times that’s the name of the game from management’s perspective.  They want to test every aspect of their application, leaving no area uncovered.

No team in the tournament personifies this mentality greater than the Germans.  A soccer field can be up to 120 yards long by 80 yards wide.  That’s a LOT of area to cover for the 11 players on the field.  Yet it seems that every time the opposition has the ball, Germany is right there to win it back.


In their first game, the Germans were up against Portugal and one of the best strikers in the world, Ronaldo.  Many thought that Ronaldo had a chance to score multiple goals against Germany’s aging defense.

But the Germans had a plan.  They very methodically spread themselves around the field, clogging the passing the lanes, and isolating Ronaldo so that he couldn’t receive the ball and attempt to score.

It wasn’t flashy, but they covered the field well, stuck to their plan, and ultimately ended up winning (4-0).

Same with testing.  When facing a threat, like a shortened testing period or invasive app changes, you need to come up with a plan and execute that plan methodically.  Figure out the most important areas that you need to cover and makes those a priority.

Only then will you be able to isolate your threat and increase your coverage elsewhere.

Creativity is king – like Argentina

Argentina have the blessing of playing with potentially the best player ever, Lionel Messi.  He’s minute in stature (5’7″, 150 lbs soaking wet), but he doesn’t let that stop him from being a force on the field.


Messi’s success comes from moments of brilliance, where he sees an opportunity and exploits it before anyone else realizes what’s going on.  Just when defenders think they know what’s coming, he turns the game on it’s head and does the complete opposite of expectations.

This is a vital component for any tester’s arsenal.  You can’t simply test what developers EXPECT you to test.  Sometimes you need to do the opposite and test what no one is expecting you (or your users) to do.  You never know what bugs are lurking just outside the normal workflow, and without some creativity and thinking outside the norm, you could miss potentially crippling bugs.

Be creative, and impress your peers with what you may find.

Never give up – like USA

You know what team no one wants to face in the World Cup?  USA.  You know why?  Because they NEVER give up.

USA is notorious for playing hard the entire game and always believing that they can win.  Look no further than their first game in 2014 for proof…

USA was fortunate enough to score 30 seconds into the game against a fast and punishing Ghana team.  They were riding high, until tragedy struck…

One of their best players, and their best scoring threat, suffered a hamstring injury and was carted off the field.  The immediate reaction was “We’re doomed”.

But the USA held on.

At half-time one of their starting defenders left the game with an injury, making way for a 21 year old kid to take his place.  “He’s too inexperienced.” “Ghana is going to tear him apart.”.

But once again, the USA worked as a team and held on.

Even their captain got kicked in the face and reportedly suffered a broken nose.

But you know what?  They stopped the bleeding, and despite having a hard time breathing, he and the team held on.

The crippling blow came with 10 minutes left when Ghana scored to tie the game, and you could tell that they were on the verge of scoring another.

Any other team would have conceded the tie, or even a loss, and point to too all of the adversity that they had to overcome.  But not USA.

With 3 minutes left in the game, the USA scored the game winning goal.  And it was off the head of none other than the 21 year old substitute.


Never. Give. Up.

As testers we’re constantly faced with blow after blow adversity.  “There’s not enough time to fully test features.”, “Management wants this pushed through regardless.”, “What do you mean you’re failing my code?  My code is just fine, what’s wrong with you?”.

Don’t ever allow yourself to get down from the adversity that you face.  You were hired to do a job, and you should do that job to the best of your abilities, no matter how much others may try to sway you otherwise.


Now go and prove to the world that you’re a world class tester with World Cup qualities.


Read More

Being Part of Bigger Tribe

Last month I took part in StarEast, one of the biggest yearly testing conferences in the US.


I had the opportunity to give a presentation about professional tester development, and I went with the expectation of meeting other testers and learning about the new trends and ideas shaping and challenging our professional environment today.

Much of these conferences are pretty similar.  Big rooms and long keynotes (some interesting while some less interesting), many of the biggest names in testing walking around and happily talking with anyone who wants to ask a question or share a personal experience, and obviously a number of presentations (such as mine) about testing-related stuff.

But during this particular conference I also had the fortune to connect with some cool testing peers that made me realize we are all part of a bigger testing tribe.


We all have a lot in common

During the conference I had, as I said, the chance to meet some pretty interesting testers.

It was not (only) the good times during the sponsored beers (A.K.A. cocktails) or at night, when we had fun and talked about non-testing related stuff.  It was actually during the sessions and the follow-up testing conversations, understanding that even though each of us comes from different backgrounds and even more diverse organizations, we all share a lot of the same experiences, challenges and even headaches that come from being a professional software tester.

Maybe even more than anything else we share many of the same feelings.

Feelings such as the need to constantly prove the value of our work to our non-testing peers, or the difficulty in explaining to our families what is it that we do for a living (“…wait a second, I understand that you don’t write the programs.  So what exactly is it that you do…?”), and even the always lingering question of what is it that we want to do when we grow up? :-)

There are many things that a tester can only share with fellow testers to gain some support and understanding on his/hers challenges and frustrations.


Is this some sort of cheap group therapy?

Maybe we should have weekly meetings, in a room filled with chairs facing each other in a circle, and start the session by saying:

“Hello, my name is Joel and I am a software tester…”

No, seriously, maybe we should!

I don’t mean a session resembling the stereotypical AA meeting (that is incredibly valuable and does great deeds for the people who attend them).  I mean having tester gatherings where we sit and share our thoughts and experiences, put forward our challenges, and gather strength simply by feeling that we are not alone.


Look for these meetings, they are already taking place around you!

Image courtesy of JeST
Image courtesy of JeST

In Israel, where I live, we have a testing group called JeST (the Jerusalem workshop on Software Testing), also the SoftwareTestingClub organizes sessions in many cities in Europe and America, and I am sure you can find such meetings in your city if you look for it.

Go to one of these meetings, if you don’t find one set it up yourself.  You don’t need an agenda or fancy speakers, on the contrary I believe that meetings where someone comes to talk about a specific point will miss its purpose, because everything will revolve around this person’s point of view and experience, and it will not leave room for the less formal but more personal interaction between testers.

If possible, make sure these meetings are kept informal.  Try to get coffee or even beers, and give everyone the chance to talk and share what’s on his or her mind, without all the formalism that comes from rank or years of experience.


Remember that you are not alone!

Most importantly, notice that as a tester you don’t need to feel alone!

It doesn’t matter how big or small your company or even your testing team is.  There will be times when you feel like you are working by yourself and that no one is there to assist you, to give you ideas or even to provide you with good (and sincere) feedback.

At times like this remember that you are part of a bigger testing tribe, and that we (all the members or your extended tribe) are here to help.  Many of us have overcome similar challenges and we’ll be willing to share and help.

Look for help, write in web forums and social media sites, try to talk to people.

Many times the best solution to our testing loneliness is to simply reach out blindly.  Unlike good luck in Las Vegas, many times help will come your way sooner than you expect it.

Read More