Over the course of my career I've had the pleasure of working with some great agile teams. I've also had some bitter disappointments working with great developers, testers and BAs who just don't get it...
Many of the teams that get it didn't actually use natural language to create executable acceptance tests, however they did have extensive suites of automated acceptance tests, usually written by the business analysts or developers but in a language and style that is not normal for the non agile developers I have encountered. So in an attempt to try to capture the difference I'm going to try to provide some useful tips and techniques to challenge those attempting to adopt acceptance test driven development within a corporate environment.
I will begin by recommending the various conference videos from GTAC. I'm not saying google are doing it perfect (I just don't know), but I am happy to believe they are probably doing lots of things right...
And most important, if we are going to go to the bother of creating executable acceptance tests, think carefully about who is accepting these. If the only person who will accept these (and I mean really accept, as in understand and even be happy to take ownership of it) is the developer, then use the most appropriate tool.
So the tips and techniques...
- Make sure the story is the right story... If you have a story that is purely technical, then it's possibly better to test these using developer tests, it's unlikely to be something the business "really" care about... If the story isn't for a paying customer but for an internal user, try to find out what benefit that internal user is going to provide for the customer and reword the story from the end user perspective.
- Don't clean up after tests... More importantly for acceptance testing is ensuring you know the state of the environment at the beginning of the test and that the test can run based on that state. Leaving the data created by the test can help immensely when issues are found. Given the amount and complexity of changes an acceptance test can inflict on an environment combined with the number of points at which an acceptance test can fail makes cleaning up extremely complex and error prone and does not provide the same level of ROI as unit tests. This has the added benefit of building a business case for better, more flexible environments and continuous delivery...
- Create unique contexts for each test... To prevent tests stepping on each other's toes if they are run in parallel, create a unique context for the test, this could be as simple as creating a user with a unique id for that test or might require creating a collection of unique items you plan to use (e.g. instruments in a trading app, pages in a cms, articles for a blog)
- Don't wait for something, make it happen... Where you come across a situation where you need to wait for something, prod the system to make it happen and use a spin loop so that in an environment where you can't prod the test still passes.
- Question everything, even the test framework... As you develop your acceptance tests, the supporting test framework and ultimately the application continually ask yourself what would happen if you replaced x with y. For a web based application, the questions you might ask could be what would happen if we wanted to make this available on an android device or iphone, does my acceptance test still hold true? Can my test framework support this easily without visiting all the fixtures? What if change the test framework I use?
- Use the english language to drive the domain model... Good acceptance tests usually make it explicit the domain model needed to support the testing, and more often than not this drives the actual domain model needed within the application.
- Use the real application code if at all possible... Rather than completely decouple your tests from the implementation, use the real implementation at the appropriate layer. This adds the benefit that changes to the implementation require no changes to the tests. To achieve this requires a suitably layered test framework to prevent these implementation changes rippling too far up resulting in broken tests. The best candidates for reuse are typically the domain models, data access components and service interfaces.
- Assume you are running everything on your own machine until you can't... Start with the assumption that everything you need is running on your local development machine, since ultimately the goal is you can actually run these tests locally to test the functionality works. Once you have a test running and passing locally, you know the functionality is working and are then in a better place to refactor the test to support different environments.
- Keep the tests isolated... Don't try to optimise tests by adding additional verifications or steps to existing stories. Create new tests. This might expose problems running the tests quickly but explore the other solutions to this rather than create huge tests that test too much. And imagine how the business will react when you say you are running 500 business tests and are getting 100% pass rate but can't test their new features because you don't have enough kit...
- Don't write the test at all... If the story doesn't have much value, or the the systems you are using are not in your control and are not test friendly then stop just short of automating it... Work out how you might automate it, the exercise will highlight the blockers and drive a better story and clearer acceptance criteria, but weight up the cost of writing, maintaining and executing the test against the value of the story and the true cost/likelihood should a defect occur in that story...
I'm sure a few of these will feel a little controversial or sit uncomfortably dependent on your experience. I'm also sure some appear on the face of it to conflict with others. For those who reach nirvana, you will end up with a suite of extremely robust acceptance tests (owned and fully understood by, the business), which a developer can run locally before committing code and which are then run again in a virtual production like cloud.
I'm about to write a few articles covering some advanced acceptance testing techniques. I don't plan to get into the nitty gritty technical details and instead want to discuss the why's... For some great material around acceptance testing I highly recommend looking at the Concordion techniques page and can't speak highly enough of Gojko Adzic and recommend you look at his blog and in particular the comments to his posts.
The question I want to ask is slightly more philosophical. Why are we really writing automated acceptance tests and who are they really for?
In an acceptance test driven environment, the acceptance tests help ensure you have solved the problem and developer tests help ensure you are solving the problem the right way. To validate you are solving the right problem we need to express the tests in a way which doesn't tie you to a particular implementation so we probably want to drive this more from a user experience and in particular the functionality we expose to the user as well as what the user can expect when using that functionality. So we are writing acceptance tests that check that the functionality we are making available to our customers is working correctly, without worrying how we will provide that functionality, but does that mean we are expecting our customer to "accept" those acceptance tests?
In agile teams you probably have a product owner and in an ideal world we would want the product owner to "own" these acceptance tests. More often than not, the product owner will happily own the story but will delegate owning the specifics (which sadly often includes testing) to a business analyst. Our goal is to get the product owner to own these tests, but with a business analyst in the way we are probably already at the stage where any tests will be implementation specific, since the business analyst is probably doing exactly that, working out how to solve the problem... In fact, business analysts probably don't want to own the tests either which leaves the team...
Let's reflect for a moment... We want the customer or product owner to own acceptance tests, but instead it usually ends up being the team that owns them, so let's explore what typically happens... The team search the web for acceptance testing techniques, they come across BDD and see there are a wealth of tools out there supporting BDD. They pick up a tool (cucumber, jbehave, etc) and all tests are now captured and represented in Pseudo english in the hope that the product owner or business analyst can start creating these tests themselves. I've yet to meet a product owner or business analysts (or indeed a developer) who uses this style of language,
a product owner walks in to a bar and says to the barman
"Given I am thirsty
and I am standing at a bar
and there is a beer mat in front of me,
When I ask the bar man for a pint of his best bitter
Then the barman will pour a pint of his best bitter
and place it on the beer mat in front of me".
Just a little bit verbose (not to mention slightly implementation specific) for expressing ordering a pint of best bitter. So my point is BDD is a technique, it is invaluable for exploring the problem domain and capturing scenarios and examples to help understand a problem, however they are not a specification in and of themselves. Using a tool too early to automate these ties you into this form of unnatural expression and eliminates a choice of how to engage with the customer later.
As a team, using the technique in discussions but then using a tool or framework (e.g. xUnit) more suited to the real owner of the executable part (developers) means you can leave the choice of customer facing tool to a more appropriate moment when they actually do want to engage and also benefit from an environment and language the developers are most comfortable with... I've written previously that even working out what you plan to test or demonstrate before working out how to implement it can add immense value as a thought exercise.
There is also another scenario which is by and far the most dangerous... Having browsed the web, we want a cross functional team so we embed a tester into the team to perform this function. The tester works closely with the business analyst and creates/owns functional tests. Most testers are new to development and don't have the skills or experience of the developers to be writing code, and worse, we are trusting these inexperienced developers with writing the most important code in the system, the tests that validate that what we are doing is correct... Inevitably we will end up with an enormous suite of functional tests that are very "script" based, not easy to understand and which add little if any value to the day to day activities of the team.
So to recap, we want to write acceptance tests to validate we are building the right thing (and that once built (or is reimplemented) it continues to work), and we want the customer (or product owner) to "own" them. If any of these are not true in your organisation then seriously ask yourself why are you doing what you are doing and put the customer's hat on and ask would you (as a customer with limited technical knowledge) ever "accept" what is being done on your "behalf"...
I've written in the past about Using English to write Acceptance Tests and the tool I choose/advocate is without doubt Concordion. Customers love seeing their stories come alive, but I've found developers can sometimes struggle to differentiate these from JUnit tests, particularly since JUnit is used to provide the execution mechanism.
I've also found that in many of the situations/organisations where I have introduced Concordion, a single story has required several tests and although the Concordion guide does present some excellent techniques to deal with this, teams new to writing acceptance tests will be uncomfortable capturing stories in this format and customers might not be happy that all their acceptance criteria are being met. I am therefore pleased to be releasing Concordion+ to the wild. At the moment it is a simple extension to Concordion which allows developers to create individual scenarios (or test cases) within a single story and also to ignore the scenarios they are actively working on. In addition, a new JUnit runner captures the state of each of these scenarios independently and reports them in the common IDEs while also allowing developers to use the JUnit @Before and @After annotations. This should simplify adoption by developers since they now have a JUnit lifecycle they understand.
I have to send a huge thank you to Nigel Charman for the concordion-extensions project which helped immensely with my development. And of course I can't dare not mention the excellent work by David Peterson on Concordion itself and particularly the facility to write extensions
I hope you enjoy using it as much as I enjoyed creating it...
This article largely follows on from some previous entries and in particular my entry on user centred test driven development.
It is often a complaint that large organisations trundle along painfully and slowly. Work can't start without following some process or other until you have sign-off. Part of this sign-off will probably involve agreement to follow certain standards and guidelines, but if these standards don't yet exist how can we start???
To challenge this and present an alternative approach, why not make the "standards" part of the delivery itself. Make it clear up front that rather than wait for the standards to be released (which would be the normal mode of attack in large organisations) you will actively work with whichever standard's body exists in the organisation to evolve just enough standards to support the actual work you are doing as you work through the backlog.
To make this work, COURAGE is imperative... Someone has to have the courage to put a stake in the ground early, recognising there is a small risk this may change. Developers should embed the standards into their automated testing as early as possible, this means that when and if a standard does change, there are tests in place which will assist developers in ensuring that all work to date is easily brought up to date...
The results of this is a design language that everyone can understand, when someone says they are writing a test which is looking for the jobs tag in the currently featured news article, everyone should know what that refers to in the wireframes, and also know how this will be identified and marked up in the implementation. This allows tests to be written before any code and even for the final "Look And Feel" to progress alongside development.
Of course, you're always free to continue in the traditional model and wait for three months until the standards body within the organisation produces a 300 page guidelines document before even starting that killer new feature that will storm the market... Or make totally random guesses, which are much more likely to be wrong, and be safe in the knowledge you have the traditional saviour of projects - Hope and Prayer!!!
Another emergn coach asked me the other day how I distinguished between an acceptance test and regression tests. For me there is a very simple rule...
- If I write the test before I write any code, it's an acceptance test.
- If I write the test after I've written the code, it's a regression test.
- If I write code to make an acceptance test pass, it is now a regression test.
Keeping it as simple as this keeps you out of trouble, I've seen so many people try to retro-fit acceptance tests after they've written code only to write a test which is based on what they've written rather than what they should have written. It's a subtle but important point that writing a test for stuff you've written (which might be wrong since you haven't got an acceptance test) means you are potentially writing a test that the system always does the wrong thing...
Following on from my previous article, I thought I would share some top tips for creating automated acceptance tests for projects using user centred design. There is nothing inherent in user centred design which prevents an agile approach being adopted and indeed agile adds huge value to user centred design by reducing the feedback loops during the "design".
The outputs from the "design" part of user centred design is usually a set of wireframes, sometimes highly graphical to be used for marketing and soliciting customer feedback. Low fidelity wireframes, as produced by balsamiq are fantastic for exploring the solution space with minimal commitment and retain that back of a napkin look and feel which prevents misinterpretation by customers and business about how much work remains to be done making them real. At some point, developers will get their hands on these wireframes and be tasked with "making them real". Of course, if the developers can help create the wireframes, then so much better and what a great way to reduce that feedback loop
So, we're a delivery team and we have the wireframes available which we want to use to start carving this up into stories (with a customer ideally), executable tests and working code. Where to start...
First we want to express what we're seeing as stories so we can assign value to the stories and prioritise based on ROI. Depending on the technology choices, these wireframes also help to perform an initial design of the application in terms of data, services, components, etc.
The stories we write can be high level stories capturing the user experience as they navigate through the site, or they can be as low level as which fields are visible within a given area of the site.
My preferred approach is to write stories and tests at the high level and as we uncover the detail capture these as smaller stories and more detailed tests. This seems to fit naturally with user centred design and allows planning and release activities to focus on the high level stories around the user experience (which is probably why they chose to do user centred design in the first place) and allow the development and build activities to focus on delivering the smaller stories.
My top tips would be...
- Write your stories and tests based on what you see in the wireframes
- e.g. When a user (who is not logged in) visits the homepage they can see the "News" in the main area. Notice that the wireframe has a single article, but this could be many articles in the future. There is also no reason to assume this is the "Latest News", "Featured News", "Breaking News" or anything else.
- Be constantly aware of the context (explicit or not) within the wireframe when writing your test
- e.g. In the latest news section, we may see a "Read More" link and assume that we can simply fnid this "Read More" link on the page. Of course, in the future, there may be more than one news article (so we want to find the "Read More" link related to a particular article) and indeed there be more sections that contain "Read More" links such as user comments (so when we implement a test around the news, we want to ensure that we are finding the correct "Read More", which means we look for it in the news section and in relation to the article(s) of interest).
- Don't assume the wireframe is what the final implementation will be so think in abstract terms
- e.g. if there is a button in the wireframe, this may be a clickable image later, or a tab panel. So word your tests not on the button, but on the capability the button provides, so instead of "...and the user can click <Help>" we might have "..and the user is able to obtain <Help>".
- Envisage alternative implementations while writing the tests and validate your test would still be valid
- e.g. If I change this from being a form to being editable inline, does the test still work?
- Try to be discrete in your tests, it is better to have lots of small, simple tests than a small number of tests that test too many things.
- when... can they see today's weather
- when... can they see today's temperature
- when... can they see tomorrow's weather
- Use "Given, When, Then... and" to think about the scenarios you need, but try expressing them in natural english.
- e.g. When someone is viewing the school holidays calendar (Given) and they enter their postcode (When), the calendar should only display holidays for schools within the catchment radius (Then) and the user should be able to remove the filter (And).
- I like to think of "Thens" as being the assertions, and with UCD this is the stuff I want to see on the screen as a result of me doing something.
- The "Ands", which are an extension to Dan North's, I like to think of as the stuff I can now do in addition and in UCD this usually relates to new functionality which has been exposed or links that have appeared.
- Refer to the Concordion Techniques page for some great examples of how to write robust tests in natural language.
- Separate site aspects (navigation, login) from functional or contextual tests
- Refactor your tests as you go to evolve a testing DSL to simplify writing future tests
- Use personas (or metaphors) to encapsulate data
- Bring your personas to life
- Write them a CV, or post their key characteristics on the team wall.
- You can validate key aspects of your persona within your test (in the "Given" section) to provide better tests by clarifying what aspects of the persona you are actually testing
- e.g. "When Joe (who is on legal aid) views the list of local lawyers, the lawyers who provide legal aid are highlighted". Notice that highlighted could mean anything, from promoting them to the top, changing colour, or adding a nice icon...
- Bring your personas to life
- Don't create too many personas
- Personas are very powerful, but when abused become difficult to manage. To test functional aspects related to simple data variations use dynamic personas (e.g. when someone, who has not yet entered their postcode, accesses their profile...)
- Think like a customer
- Ask yourself how much money you would pay for a given feature or piece of functionality
I'm sure that I'll refine this example in the future and therefore watch this space...
I read with extreme interest James Shore's blog about FIT but was dismayed that he devalues automated acceptance testing. To claim that FIT is a "natural language" is wrong, it is a developer language and this is possibly why customers don't get involved. Concordion on the other hand is natural language and I think plays much better in this arena. In addition it is much more developer friendly.
I've written previously that for me the value of test first is the thought processes surrounding it, however, where applicable converting these into automated tests, and in particular automated acceptance tests is a huge win. I would love having a customer "own" the tests, but when this isn't possible (almost always) I will try to put my "customer" hat on and think like the customer and express what I'm about to do in their language (which will be English, not FITnese, or selenese, or rspec). If the customer is happy with my specification, I can then use this directly as my test.
So for me, the lack of customer isn't the problem, but I agree with James on one point, there is a problem...
It's the people... The majority of developers I've encountered can't think like the "Customer" and instead thrive on complexity. They can't express the problem or solution correctly and write tests that become implementation specific. This means they have written a test for a specific solution, where actually there could be a multitude of solutions, even in the same technology. When they then 'implement' this solution and the customer doesn't like it, the test can't be reused and needs to be 'reworked' (I'm avoiding refactored, since the test was actually wrong, and therefore it should be fixed, not refactored). This is the problem, the test may be rewritten many times at which point the customer will be thinking, this is now the n'th time I've asked for this exact same feature and I've seen five different versions of a test for the same thing, none of which are producing what I'm asking for. If I was that customer would I want to own these "tests" which seem to be so difficult to change and can produce such a burden to tweak the implementation.
So for me, if I don't know what I'm doing, I won't do it and will ask for help from someone who does know what they're doing. I would encourage all developers to have the courage to admit when they are out of their depth with a practise and seek advice rather than struggle on developing the wrong thing which ultimately ends up having little value.
Test Driven is a loaded term and means different things to different people. I much prefer the term Test First which clearly states that the test comes before the implementation. However, for me, the value is not necessarily in creating an executable test, but in the thought processes that Test First brings out.
One of my colleagues at emergn is constantly reminding me that when presented with a solution you need to ask yourself what is the problem. This is what lead me to question TDD and it's variants. If you search the web for Test Driven Development, you'll uncover a wealth of information from many of the authors in my BlogRoll, as well as many variants on a theme. I think the wikipedia entry is a particularly good summary of Test Driven as it is currently understood by the community but for the real meat and bones you need to look at the articles from the thought leaders behind the practises.
However, when I look at this wealth of information, I'm now faced with the question... OK, these are all solutions, but what is the fundamental problem they are solving? Is it
- Code quality/design is poor
- Code is overly complex/difficult to test
- Unused functionality within the code
- Too many bugs
It is only when we understand the underlying problem fully that we can then evaluate the applicability/suitability of a particular approach. Indeed, it is this lack of a clearly defined problem which makes it impossible to determine which approach is best, since we can't define any tests up front... This is actually a little bit of a paradox, but highlights for me one of the most important points about TDD. TDD is a solution to too many problems and in certain cases is not a very good solution.
Testing should be about proving functionality, but unfortunately we too often see TDD trying to address non testing issues like design and code quality. Of course, code needs to be testable, but it is not the responsibility of the Testing to enforce this, it is the responsibility of the design, but this is (unfortunately IMHO) the missing piece in most methodologies... Indeed, the approach of writing the "Simplest Possible Thing" to pass a test is possibly my biggest bug-bear with TDD. This approach often means you can pass tests with no functionality other than hard coded return values, now that has to be the biggest process smell ever.
I'm going to stop using TDD, or Test Driven now and use instead the term I prefer which is Test First. For me, this means that before I do anything, I will determine ahead of time how I would test it. The immediate benefit (a.k.a value) I get from determining how to test something is I'm also starting to think about (dare I suggest design) things like apis/design/interaction/responsibilities. This is not the same as BDUF (big design up front), instead it is just enough thought at just the right moment to (hopefully) prevent a disaster. I can then apply some cost/benefit analysis with a pinch of risk analysis to assign a value to actually writing all the tests that TDD would have forced you to write, or just a small percentage to cover the important or more complex functionality.
Of course, in many cases I will actually create executable tests for many of those uncovered during this thought exercise, but will I go through the whole Red/Green/Refactor cycle, possibly not. For me the Red/Green/Refactor is like micro context switching. I prefer a slightly longer period of focus and therefore I may write quite a few tests at the same time before applying that context switch to go into coding mode. This of course is my personal preference and undoubtedly would be scorned upon by the dogmatic TDD zealots, but if this is what makes me more productive, and without a baseline problem statement I challenge them to prove that this is not the best way to do your testing.
Test First allows me to
- Understand the problem I'm trying to solve
- Think about how I will solve it (just enough design)
- Uncover any unknowns or risks hidden in the initial problem statement
- Produce high quality code which has just enough tests
Many of the Test Driven approaches do indeed address many of the initial problems stated earlier, but are they the only solution to these problems? Indeed, are these problems actually just symptoms of even deeper problems? If the problem is simply that your code is very tightly coupled and difficult to test then the best ROI will probably come not from TDD but from up-skilling your development team on fundamental design principles (unskilled developers was the real problem in the first place and TDD can't solve that).
- Wikipedia entry for Test Driven Development: http://en.wikipedia.org/wiki/Test-driven_development
- Introducing BDD by Dan North: http://dannorth.net/introducing-bdd (highly recommended reading)
- C2 wiki entry for Test Driven: http://c2.com/cgi/wiki?TestDriven