# Eight Burning Questions for Three Test Automation Leaders

We asked three of today’s top test automation leaders (Evan Edwards, Andy Knight, and Paul Merrill) about problems, strategies, and solutions they experience during testing. Read on to find out what they had to say!

But first, here’s a little bit more about them:

**Evan Edwards** is the Co-founder and former CTO of Cycle Labs who oversaw the development of Cycle, our behavior-focused test automation solution that allows all personnel to join the testing fold and helps implementation teams to deploy with confidence. He ensured that Cycle grew in response to industry needs while remaining stable, streamlined, and easy to use.

**Andy Knight** is a software engineer who specializes in building test automation systems from the ground up, which involves both software development for test code as well as the infrastructure to run it in continuous integration. He regularly speaks at testing conferences and has a well-renowned [test automation blog at automationpanda.com](https://www.automationpanda.com/).

**Paul Merrill** is Principal Software Engineer in Test and founder of Beaufort Fairmont Automated Testing Services. A frequent writer on testing and test automation in top-notch testing publications and an international speaker at testing conferences, Paul works with clients every day to accelerate testing, mitigate risks, and increase the value of testing processes with test automation. Learn more about test automation at [http://beaufortfairmont.com/webinars](http://beaufortfairmont.com/webinars).

#### 1. What’s the most common pitfall that you see testing teams stumble into, and how can they help avoid it?

**Evan Edwards:** Testing the wrong thing. You have to talk to people and communicate and make sure you are trying to test what you actually need to test.

**Andy Knight:** The most common pitfall into which I see these types of testing teams stumble is the lack of attention given to the stack that supports test automation. Most test teams focus on test cases and high-level test code without considering the framework, the packages, and the infrastructure.

**Paul Merrill:** Trying to “test everything” instead of identifying the risks they need to mitigate. Use a typical testing methodology to assess risk and identify what to test.

#### 2. What is the best way for a testing team to show ROI for test automation?

**Evan:** Focus on meeting the overall objective, which in turn should support whatever business case you’ve drawn up.

**Andy:** ROI is tough to truly measure. Here are ways I seek to measure ROI for automated tests:

- Feature priority – does the test cover important things?
- Test execution frequency – how often is the test actually run?
- Coverage uniqueness – does the test not duplicate existing coverage?
- Cost of ownership – how much time and money do we spend to keep this test running?
- Bug discovery – how soon are bugs discovered, and by what severities?

#### 3. This certainly can be project dependent, but generally speaking, what are the most important test management metrics?

**Evan:** Some combination of “coverage” and “regressions” isn’t a bad place to start.

**Andy:** Make sure tests maximize ROI. Make sure they have high, unique coverage and run frequently to provide feedback as fast as possible.

#### 4. What is your favorite method to help translate a user story into an end-to-end test case?

**Evan:** I really enjoy having an informal chat with the supervisor responsible for the job function.

**Andy:** No doubt: [Example Mapping](https://automationpanda.com/2018/02/27/bdd-example-mapping/) is the best way I’ve found to derive behaviors and tests from stories.

#### 5. How should a testing team determine how often they execute automated tests?

**Evan:** How long do your tests take to execute and how often are updated results helpful to your team members?

**Andy:** Tests should run continuously! Teams should focus on setting up CI systems to run tests quickly and reliably.

**Paul:** Depends on the methodology they use, but many teams are moving toward DevOps with Continuous Integration.

#### 6. What is the most important consideration when a team starts their test automation journey?

**Evan:** A clarified objective and really good communication. It’s important to be in agreement on the target and the plan to get there.

**Andy:** A team’s most important consideration should be their goal with testing and automation.

**Paul:** “Why are we automating?” Teams that have clear understanding for why they are moving toward automation tend to be more successful.

#### 7. Does it ever make sense to automate acceptance tests?

**Evan:** Sure. We do it all the time, but it’s naïve to assume you can skip the step where a human inspects your deliverable.

**Andy:** Acceptance tests should be automated whenever they provide ROI and make sense to run repeatedly.

**Paul:** Yes.

#### 8. What’s your favorite testing slang term, i.e. ‘Smug Report’?

**Evan:** I like Dummy Data and I really enjoy hearing “that’s functioning as designed.”

**Andy:** I like to “Gherkinize” my tests.

Thanks to Evan, Andy, and Paul for their input!

**This post was written by:**

**James Prior**

_Sales Engineer_