In this post, or perhaps truly an article, I want to explore a common approach for implementing an effective strategy for your overall agile automation development. It was Mike Cohn who established the initial model of a pyramid view towards traditional and then agile test automation.

This model is fairly widely known as the Agile Test Automation Pyramid[1].

Figure 1, Traditional Test Automation Pyramid

In the traditional view, most if not all of the effort was in developing UI-centric functional tests that explored the application via the GUI. There might be some lower-level tests and a few unit tests, but teams mostly staid at the upper tier.

Why?

For several reasons, but first and foremost, the testers were the ones primarily writing the automation, so their comfort zone was towards functional testing. It didn’t help that the majority of the automated testing tools were focused towards leveraging the functional UI as the point of entry.

Developers operated in the service and unit tiers, but typically didn’t invest much of their time in developing automated tests. There was little ‘value’ in it from their perspective, when they were being driven to deliver the features by prescribed dates.

But this strategy is flawed. It’s inherently unstable and brittle; as the application (UI) changes the automation is nearly always impacted. Therefore one problem is automation stability and ongoing maintenance costs. Another problem is that it doesn’t engage the entire team. Instead, only the testers typically develop it and they usually only comprise a minor portion of the team. Quite often they also become overloaded with time spent on testing versus automation development.

The agile test automation pyramid is a strategy that attempts to turn all of this around–literally.

Figure 2, Agile Test Automation Pyramid

Turning Things Around

The first change is taking a whole-team view. Instead of the testers being responsible for testing AND writing all of the test automation, it becomes a whole-team responsibility. The developers take most of the ownership for unit-level automation, but testers can operate here as well. The upper tier focuses on limited UI-based automation. Usually, these are longer running, core customer usage workflows that are best implemented at this level.

The testers typically operate on these, but there are very few tests within this tier. And remember that the developers can operate here as well. The two layers are met by middle-tier automation. This is often the domain of the open source ATDD/BDD tools, such as: FitNesse, Cucumber, JBehave, Robot Framework, and others.

One key thing to note is that traditional automation was often a one-tool operation, with Mercury/HP commercial tooling (Quick Test Professional or QTP) leading that space.  The agile approach is tool agnostic, but also aggregates tools that are appropriate for each layer. Therefor no “one size fits all” thinking is allowed. For example, as of this writing, these are common used tools at each tier:

1. UI tier:  Selenium, Watir, or traditional QTP
2. Middle tier:  FitNesse, Robot Framework, and Cucumber
3. Unit tier:  xUnit family variants for example JUnit or NUnit for Java and .Net respectively

The other consideration is that there are extensions to many of these. For example, both Robot Framework and Cucumber have Selenium plug-ins so that they can ‘drive’ the UI as well as the middle tier. This implies that the middle tier tooling, and automated tests for that matter, can extend or blend into the lower and upper tiers.

Advantages?

The advantages of this approach mirror the disadvantages of the traditional approaches. First and foremost, if you take a whole-team approach, you now have automation work crossing every team member—so your capacity is increased. You also get better automation designs and improved testability because developers are weighing in.

Another important benefit is maintenance. Often it becomes a team-wide responsibility to keep up with automation maintenance, both the infrastructure and the tests, as the team is developing functional user stories. I like like to make it part of the organizational and teams’ Done-Ness Criteria, for consistency and rigor. For example, a story is not “done” unless all of the automation associated with it, new and old, is working correctly (passes).

An advantage that I didn’t emphasize in the traditional case is automation coverage. One could argue that automation from the GUI down was a limiting exercise. There was always back-end functionality and business logic that was incredibly hard to “reach” via the automation. And internal infrastructure was nearly impossible to exercise / test via automation.  In a multi-tiered approach, you should be able to “reach” any behavior or functionality that the team deems to have value to automate.

Two Final Cautions

I’ve found it surprising but true that many developers don’t really understand (how) to write effective unit tests—especially when you expect those tests to be part of a cohesive whole. Partnering with testers can help immensely when designing unit tests, even if the testers can’t program in the language du jour. I’ve found that training can really be a force multiplier here. That would include a 3-4-5 day class focused on OO design skills, patterns, refactoring, and TDD. That combination can be incredibly helpful in creating a baseline of skill across your teams.

And one more final caution—don’t automate everything! The strategy or business case for 100% automation is nonsense. In fact, I’m notoriously against using any sort of “magic number” when setting goals for any sort of automation level or coverage. Rather, ask the team to look at each user story and determine the “right level” of unit, middle-tier, and UI tests in order to adequately cover it compared to its complexity and value. THEN, implement those tests as part of your Done-Ness criteria. So every story should have a different level—determined by the team.  Point being—trust your teams to determine the right balance between automation and other relevant forms of testing.

From an Agile Testing Perspective

I like to think of the agile automation triangle as THE model for automated testing strategy within the agile organization. While the execution of the triangle is a whole-team activity, the QA and Test team leaders and the testing community should spearhead the automation strategies themselves.

This involves test Architects leading the efforts to evangelize the strategy and approach and to guide their organizations towards effective tools selection. Often, having a tooling “bake-off” is an effective way to garner whole team awareness, interest, and feedback in the selection process.

A Real World Example

In Figure 3, you see an example of our implementation while I was at iContact. I literally ‘plucked’ this snapshot from a real-time status wiki page. It was where we stood in actual test counts in mid-2012.

Figure 3, iContact Automation Pyrimid circa (2012)

In our case, we leveraged the 3-tier model, but we differentiated between System-level and Integration-level tests in the middle tier.

Our view was that Integration-level tests were tests implemented in Cucumber, our tool choice in the middle tier, but they were focused more towards unit-level testing. Another way of saying that is that they were larger scale unit tests. Our System-layer tests were true middle tier tests that were focused more on functional behavior. Of course there was a “fuzzy line” between the two.

I’ve found this is often the case in real world projects, that the pyramid boundaries are not always clear-cut. In our case, we defined categories that worked for us and allowed us to report out on the execution of the various layers.

Another important point, everything below our UI level tests were intended to be run on a “check-in basis”, that is, when every developer performed a check-in, approximately 12k tests would run to provide them feedback as part of our Continuous Integration tooling and virtualization environments. So the tests needed to be developed with performance in mind.

We were constantly tweaking our tests to bring the execution time within reasonable constraints for our team. Usually are targets were between 15-30 seconds of run-time. This was based on our commitment to run as many tests as possible on finely grained code chunks so that we could get real-time feedback.

It was hard work to keep it tight, but well worth the efforts.

Wrapping Up