Test Cases vs Test Ideas - Looking for assistance

Michael Bolton (http://www.developsense.com/) commented on my blog about moving towards the concept of Test Ideas. There is still much that I need to learn about using Ideas versus Cases.  I feel though, that this maybe how originally I had the team create the maps before moving to the Given, When, Then format. I wanted to testers to think freely about scenarios, instead of having a dedicated way to complete a “test case”. We simply did not have expected results on our maps at all. Was what we were doing Test Ideas?

Our applications and the system that we test allow for multiple ways to complete a certain task, therefore it is more of an “idea”. As I continue to do research and to grow I feel there will be more clarification in my mind of the true difference between “test cases” and “test ideas”. I’m on a journey here of learning and growing, which excites me.

Please comment and share thoughts about this. Do you have a good blogs, books or articles that I should read? How do you define a test idea? Do you use ideas?


Update: Blog link from Darren:
http://www.bettertesting.co.uk/content/?p=1438 

Our Current Test Process

There have been several comments on past posts about the importance of testing changes to the system. This is done by the team currently, and is capture in reports. I thought I would simply provide some additional information for clarification.

Our process dictates that when we test a build we do the following:

1. Run Acceptance Tests.
2. Test all changes. This is done through using our bug tracking system. When a build happens, the bug is moved to a Verify state. The test team self organizes and each tester decides what to test and verifies the issues. Each change the developer does to the code is matched to either a bug in the bug tracker, or to a User Story or Task on our sprint board. This ensures we test all the changes.
3. Run through all other test cases based on Coverage Level. In an earlier post I said Priority, but in truth it is Coverage Level. That level is determined at the beginning of the sprint on how much testing we require on a map. Remember, each functional area of the product has one map. We test the maps where the changes occurred to ensure that no other areas were broken by accident.
4. Results are updated on our Dashboard.
5. A summary is sent out to all those that are interested, which include a copy of our Dashboard, a list of new bugs found, and the list of verified changes.

I hope this post gives you some clarification on what we do when we receive a build to test. A future post should also write about our overall process during a given sprint.

General Update

The past few weeks I have received a lot of feedback on my blog through comments, email and Twitter @nmacafee. Each comment, good or bad, has been very helpful. I am a tester like many of you reading this blog, who wants to grow and explore new ideas and concepts.

Thank you to everyone that has commented, as I find this to be encouraging. I feel what I have been doing to change out we test within my organization is justified, but there is still plenty of fine-tuning to do. It was almost a year ago now that I first came across mind maps, and shared this idea with the team. There were several confused faces during that first meeting, but they were willing to give it a try. 

There has been a good cultural shift within the team. Developers are now comfortable with the mind maps and are even using them for their own use. Other test organizations with the company are coming to me for guidance on how they could implement mind maps for testing with their group.  There are still many skeptics out there who do not believe that this is a real “test management system”, but I continue to push forward turning those people into believers.

Change is a good thing. We need to move past the conventional methods that we use to test simply to shake things up.  Standard test case writing still has a place and work for many. Mind maps have though opened up the door to exploring new ideas within my team. We have saved time, which I’m excited to share, has led to all on my team learning some basic automation. When we get a test case automated it is now marked on the map. There is still much to do, but I have been very encouraged with the progress we are making.

Agile Ottawa Meetup - User Stories

On Thursday night this week (November 24, 2011) I attended the Agile Ottawa Meetup Group http://www.meetup.com/Ottawa-Scrum-Users-Group/. I have attended in the past before, but did not have a very good experience unfortunately. No details will be shared, but I put that aside and thought I would go again. This session I am happy to report was well organized and I learned more about Agile.

Below are my thoughts of the evening.

The meeting was focused on writing User Stories. The presenters, who were the “Product Owners”, gave us a mockup application that they wanted us to “develop”. The group self organized into four teams, of approximately four people. We then went through and wrote User Stories for the application. I have been writing User Stories for a while now, but what I did learn was A) another template that can be used B) mind set of those involved in writing can be very different.

Templates

This one I have been using:

As a <user>

I want to <task>

In order to <goal>

The new template I learned was:

In order to <goal>

As a <user>

I want to <task>

(Side note both fit well with Given, When, Then as well :) )

The new template simply rephrases the order to help it flow better. Depending on the scenario and phrasing required, I noticed that we used the different templates. Basically, we used what sounded right.

Testing vs product owner vs developer mindset can be very different when writing a user story. The teams that were mostly developers the User Stories were very technical, while tester’s User Stories were more general in nature. This reaffirmed to me in the Power of Three was necessary for User Story writing. The team should do this. Not just the developers, not just the testers and not just the product owners. ALL should sit together to write the User Stories as this remove confusion and helps start the iteration in a good direction.

Some good points that I picked up:

•        Focus on what is necessary in a user story
•        Do not over think the user story. Some went wildly off track from what the Product Owner wanted in the product. There is nothing wrong with thinking about how the application will work in the future, but over thinking can “cloud” our tasks for our current Sprint/Iteration.
•        User story should be testable externally. With this we started to write three Acceptance Tests per User Story. This really helped us focus on writing clearer and shorter User Stories, and will help the team define Done as well.
•        The presenter suggested that each User Story should take up no more than half of your Sprint/Iteration.
•        If there is research involved in a User Story or task, make sure you time box it, or it can grow out of control.

I was happy to hear that some use diagrams to write their user stories. They didn’t really see or know that what they were doing was simply a mind map. Mind maps are just very useful and can be used for many applications. Using maps for User Stories will be something that I will like to introduce to my team going forward.

Some good (or bad) quotes from the night:

“Telling a developer how exactly it should look will take away the freedom for developers to add value.” (I am not in agreement with this. As a team, you should be working together to a common goal. Product Owners should know what is being developed, and testers should have a clear under standing before development starts (ATDD) in order to prevent confusion and test case rewrite at the end.)

“Keep User Stores big when out there, but when they come closer (in time) break them down, like the game of Asteroid.” (Again, I wonder about this. If something is not yet ready to be worked on for a Sprint, is that simply not on your backlog or as a requirement? Should you have large User Stories that cover future development, or simply statements for what they are?)

“The more self sufficient your team is, the more effective it is.” (I agree with this)

“Break down barriers between teams.” (Definitely. I was surprised how many at the meetup were working at companies were the developers, testers, UI developers and product owners worked on separate teams and even in separate locations! That does not sound Agile to me. I’m just very happy to not be in that situation currently.)

Overall I enjoyed the session and learned more. The most valuable item I got out of going though was how happy I am with my current job and the team I work with. Our team is quite mature when it comes to Agile and we are doing really well. Sometimes I think the grass is greener on the other side, so it was nice to see how other companies work (some are cutting edge, and some are still learning). I just know that I have a good balance where I currently am, and freedom to continue growing.

Priority

As our maps have grown in size, we needed to organize them in order to set priorities for testing.  Just like many standard written test cases, we gave each THEN node a priority.

Disclaimer. Priority, and using a Dashboard were based on http://www.satisfice.com/presentations/dashboard.pdf. This proved to be a valuable starting point for what we are doing.

Priority 1 – Acceptance:

This tests the basic functions of our product with simple data. We run these tests each time we get a build. As we may have several builds a week, we created one Acceptance mind map that pulled in all of the Priority 1 test cases.

Priority 2 – Acceptance+: 

Same as Priority 1 Acceptance Tests, but now with more complex data.

Priority 3 – Common:

Tests the main features of the application.

Priority 4 – Common+:

Covers all features that are not commonly used by a customer.

Priority 5 – Validation:

Tests field validation, corner cases, stress, etc

Xmind and MindManager both have five levels of priority by default, which matches this well. I am still considering reducing this to three levels and simply calling them High, Medium and Low, or Acceptance, Common and Corner. What do you think?

MindManager has good functionality that allows you to filter a map to show only the priority levels that you wish to see. This allows the testers to focus on what they are testing at a given moment.

Priority also comes into play before deciding what to test during a given sprint. During our sprint planning session the maps are discussed with the User Story. We know roughly at the end of this meeting which maps are affected by the new tasks. We also discuss as a team (development, testing, product and project management) what we should focus on during this week for testing. That helps us decide which maps should be tested and at what levels we should test the maps.

During the week, we test the maps that we should focus on, running the test cases in order of priority. Once all maps are completed with the requested priorities, testers are encouraged to test other areas. The goal before product release is to complete testing on all maps and all priorities. This information is recorded on the Dashboard as well.

For any given sprint, I can tell which maps were tested and what level was completed. An Overall Dashboard allows the team to visualize the amount of testing completed week over week. More detail about the Dashboard and the Overall Dashboard will be given in future blog posts.

Given, When, Then

Back in May when I first started blogging about mind maps for software testing I stated that the maps don’t always explicitly state the expected results. As our maps evolved, we started to add in expected results.

Having expected results stated on the map is beneficial. Before I trusted the test team to understand the functionality of performing a given action. We have also started to share our maps with teams outside of organization that take care of testing that we cannot accomplish at our location. To be clear, trust is one of the most important attributes that a manager should have with his or her employees. :)

In August I attended Agile 2011. At a session presented by Elisabeth Hendrickson she taught us a very basic concept that I am sure that some in the room already knew. To me it was a valuable lesson. For the most part, I am a self-taught tester and there are always very cool basic tidbits that I am learning.

When writing Acceptance Tests, a simple and easy to use template is:

Given <a situation>

When <an action is performed>

Then <a certain expected result occurs>

During that session, I was thinking about our maps. I realized, why couldn’t I write the maps in this format as well?

I came back from the conference, created a template map, and shared it with my team. Using Given, When, Then format has improved our maps greatly. It gives the testers focus on what they are writing, and it is easy to expand our tests. What I really like is that each When (or Action) has a one or more expected result. In the past matching steps in test cases to the expected results was sometimes confusing. Different testers have different styles. Some testers would write a test step test cases, with 4 expected results.

Here is the basic template that I came up with. This template gives the testers a common syntax of how to create and update our test maps. In several cases the testers were already writing in this format without realizing this, there were just some fine tuning needed to update them.

What you might notice is that you can easily expand the branches and continue writing tests. Another option for those writing automated tests with tools such as Cucumber mind maps in this fashion would be a great front end for it. From Cucumber syntax I do see more possibility of expanding my template.

I will write more about priority of tests, etc in upcoming posts.  Thank you to everyone for your comments and questions here and on Twitter. Feel free to email me as well.

Recording, Reporting and Storing

My post on the weekend generated several questions in regards to recording, reporting and storing test results.

On the project that I work on we have approximately 25 maps. Each map represents a functional area of the application. For example there is a map that documents the Install test cases. All of the maps are stored in a directory within a file management system. You could use SharePoint, or store the files within your source code repository system (Subversion, etc). There are numerous tools out there. The reason that we use such a system is that it gives us version control, history and the ability to “reserve” files so one person can edit the map without worrying about it being overwritten by another tester.

The folder structure could be the following:

\Master Test Maps

\Sprint 1

\Sprint 2

…

\Sprint X

If you are not testing by Sprint, you could also create folders for Builds, etc. Choose what works best for you.

When a new Sprint starts, a new folder is created for that Sprint. Copies of the test maps are made from the Master Test Maps folder to that new folder. Depending on the scope of the testing you wish to complete during a Sprint you may wish not copy all of the maps over. This is a simple process. Unmarked maps are copied over to the new folder ready for the tester to markup.

Our maps have evolved greatly and are now written in a Given, When, Then format (that will be a completely separate blog post to follow). The tester will walk through the map and will use the tools within the mapping software to say if the test case passed or failed. Xmind has a green checkmark icon (test case passes) and a red X icon (test cases fails) which can be placed on the Then statements.

Xmind is an excellent mind-mapping tool, but it does have some limitations. We are now using MindJet MindManager http://mindjet.com/. MindManager has several advantages, but one is the ability to run macros. A macro has been written that will “walk” through all the maps that were tested and will count up the number of Passed, Failed, etc.

Included also is one other file in the Sprint/Build folder, which I call the Dashboard. This file is key to recording and reporting the results. We use Excel for this, but any format would work, depending on the results that you want to record. Here is a sample template:

Test Map	Tested By	Defects Found	#Tested	#Pass	#Fail
Test Map 1	Sally	BUG123, BUG875	50	40	10
Test Map 2	Fred		20	19	1
Test Map 3	Cindy	BUG235	70	62	8

The template above is a very simplistic but will give you an idea of what you could do. The Dashboard we use has more columns to cover the level of testing Requested, Covered, Map Quality and other metrics to cover off N/A, Blocked, etc in regards to the test cases. (Future blog posts will go into Requested, Covered and Map Quality). The key to a good Dashboard is to make it as simple as possible to use and only record information that is relevant. For a long period of time we were not gathering specifics on number of test cases. I still wonder the value of this information, and go back and forth on that debate in my head often. In Agile, what I care most about is if the User Story is Done. Done to me means that it is coded, it has been tested, and there are few to no bugs remaining. To satisfy needs of the organization we now record this information, and it has proved to be beneficial on multiple levels.

When testing is done, the Dashboard is copied into an email that is sent out to the extended stakeholders.

There is a long list of other topics that I will be covering on this blog. Please keep your questions coming and I will address them.

Recap of past 10 months

It has been awhile since I have posted to my blog. I am happy to say that I have been approved to continue writing and updating this site. I hope to share with others on what we have learned about using mind maps for writing and running software test cases.

To recap, in January I started in a new organization that was using Agile. I was excited to join this team as I was curious what Agile was and how it worked. I inherited a team and their existing test cases. The test cases were written in the standard way, and the team was using Quality Center to run those tests.  They however were outdated.

Since we were using Agile, I started to think how do I make testing fit into this methodology? How do we test within a Sprint, instead of the normal waterfall approach I was so accustomed to?

I started following several Agile testers on Twitter, read several blogs and books, including Janet Gregory and Lisa Crispin’s excellent book on Agile Testing. http://lisacrispin.com/wordpress/agile-testing-book-is-now-out/

A colleague introduced me to story mapping, which she had learned about at Agile 2010. She and I used sticky notes to visualize all of the high level features of our application. It sounds simple, but this was very important for me to learn about the application. And when I saw it up on the board I realized it was not that different looking than the mind maps that I had started to read about. However, I had never seen anyone using mind maps for writing software test cases. Some people were using them for User Stories, or to brainstorm test cases, but not the actual writing of them.

The journey then began. I downloaded Xmind http://www.xmind.net/ and started to play with it. I reviewed our existing tests and noticed that several steps were often repeated in each test. After some trial and error I had converted our Smoke Tests (approximately 20 test cases) into a single map. It was visual and easy to read. I was onto something.

During a meeting with my team, I presented the test cases in the map format. There were a lot of blank faces around the room, and some confusion. Luckily after some convincing, the testers were willing to take a chance. Our tests were in need of a rewrite anyway, so they jumped on board. We worked collaboratively and rewrote the existing 1200 test cases. We reviewed the application, and went through it screen-by-screen and started developing a map for each screen.

Once done, we had about 15 maps. We printed off all the maps (each fit on a legal sized sheet), reviewed them more as a team by posting them up on our scrum room wall. We were able to cross off duplicate areas, and group tests into more functional areas.

We then presented the maps to the development team. There were some confused faces again, but the visualized presentation of the tests proved to be valuable. We could easily see missing or duplicate tests and developers could see what testing was expecting and what we were covering. The past system of storing test cases in Quality Center was a black hole for development, and they were not always aware of what we were doing.

We then started to run our manual test cases using the maps. For each sprint or build of testing, a new directory was created and copies of the maps were placed in that folder. I created a Dashboard that was used to track what maps were tested, level completed, number of passes and failures. We, as a team, had come up with and created an easy to use test case management solution.

It has been 10 months since we started this new way of testing. Our maps have improved greatly, and our test case maintenance time has been reduced significantly. The maps allow us to test new features within sprint on daily builds, which is a huge benefit. Regression testing still occurs at the end of the sprint, but with the maps we have been able to reduce the time that this takes by almost half.

The most important thing however, is that everyone enjoys using the maps. I am pleased to say that this concept has now been expanded to several other teams within our company. There have been several skeptics, but they all have been converted once they see how valuable and easy this has been. The proof of the success is that mapping, plus other improvements to our Agile processes have allowed us to deliver on time and with better quality multiple releases since January.

In future posts I will be diving into more of how we have evolved. You can read some of my past posts as well, or please feel free to contact me (email or Twitter).

Mind Map Testing: Where are the test steps?

As you can tell in previous posts, mind maps do not give exact steps for test cases. This is one potential problem with writing test cases in this method (I will list pros and cons in a future post). I have thought about it in detail, and have come to this conclusion. Step by step instructions are not needed. Yes there are some test cases where very precise information is needed but that information can be added as a Note to a node on the map.


I have several reasons why you do not need to include steps:

• When a tester gets comfortable with doing test cases they will often see the title of the test case, and simply run the test by heart. 
• If a tester follows exact steps every time the test case is run, you get into a situation where you will not find bugs in the code. Think of a minefield. If you walk the same path all the time, what are your chances with finding a new mine? The same goes with testing.
• I have also been asked "If there are no steps, how will a new tester know what to do?" This is a legitimate question. My answers to this are:
• We work in an Agile environment and team members (I am trying to get away from using Testers and Developers) should be talking and working with one another. It is a great opportunity for a new team member to ask questions about how the product works and how they should test this feature. This will give a fresh prospective to the product as well.
• If a professional person working in high tech as a Tester is unable to figure out how a test case works, does that not mean that it is too difficult for the customer to use as well?
• By not giving exact steps it allows testers freedom to explore and build in ad-hoc testing. Not every tester is going to "Create a bookmark" in the same way. To get better coverage rotate who is tests what Area.

Mind map testing may not work for everyone, but I would like to hear your thoughts and experiences. What do you think works well? What can be improved?


Some topics still to cover are:
- Pros and cons
- Setting priority
- Test case execution
- Reporting on results
- Formatting
- Test case review
- Whole team approach

Creating Test Case Maps: Where Do I Begin?

In the comments below on my first post I was asked "What do you do with multiple test cases and/or test cases with multiple steps?". The answer is that you split them up and make individual maps. But where and how do you start?


The first step is to create a Feature Map. A Feature Map highlights all of your main areas for testing at a very high level. For examples going forward I have decided to write test cases for a generic web browser so that you can easily follow along. 


For my team, I have found that group test case writing has been very successful. We met as a team and first came up with the main areas of testing. 


For a web browser your Feature Map may look something like this.

Mind Maps are a great way to brainstorm your ideas and are very visual. With some work your team can come up with a good set of main features. You will notice ideas start flowing more naturally, which will lead to more test cases and therefore better coverage. 


Next I have had success with dividing up the main nodes into individual Area Maps. In this case we would have maps for Setup, Options, Bookmarks, History, Controls and Window. Split the work among the team and have each take one Area (an Area being a group of common test cases) to start expanding and writing test cases. 


A tester may go off and create a map for Bookmarks such as shown here. 

I suggest printing this map off or display on a projector for your team to discuss. Get everyone up to talk about the test cases and find out what is missing.


There is one map created per Area. If you find a map is getting too large, split it into sub maps. Think of User Stories in Agile. Sometimes we have to split them up to make them easier to manage. The same concept applies to Area Maps.


Next post will talk about test case execution and how to report on results. Again, this is at a very high level at the moment. We will go into more details about formatting, priorities, how to include detailed information, etc at a later point.


Please let me know if you have any questions or if there is something you want me to cover. The more, the better :) I want to learn just as much as you do. 

Mind Map Testing - Part 1: Writing Very Simple Test Cases

Over the next few weeks I will be writing about Mind Map Testing. I recently moved to a new organization within the same company for a variety of reasons. One was they were using Agile, and I didn't know much about what Agile was but wanted to learn. 

I lead a small team of testers all with a background in waterfall testing. What I noticed was that development was working well in Agile, but testing was falling into mini waterfalls at the end of each sprint. Ideally testing should be done within the sprint, and the real solution is to have automation in place. With no automation (we are moving in that direction) I searched ways to help reduce the amount of overhead that testing has. One area was test case maintenance. So my first journey began.

Twitter has proved to be a valuable source of information. One of my first contacts was @darren_mcmillan who I found through a blog he contributes to called http://www.bettertesting.co.uk/content/. There is a wealth of information on that blog, and through that I came across something called a mind map. I was unfamiliar with the concept, but through exploring it I have fully embraced the idea. I downloaded Xmind and starting using it for basic brain storming. I then thought about using it for test case brainstorming, and then finally writing and running test cases in a map. I'm not the first one to think of this at all, and I take no credit for this. I just knew that I was onto something that could shift my perception of how I test.

As you know general manual test cases are often written in this format. 

Preconditions:

- software loaded

- hardware version X+

Steps:

1. Turn on device

2. Click on Button A

Expected:

Clicking on Button A shows a dialog.

The question is, what if there are multiple buttons to test? Say Button C, D and E. Testing generally copies the first test case and then changes A to C, etc. However, what happens if after Step 1, you need a new step(s) before you can click on the button? Now we are in test case maintenance hell.

Testers need to go through all the test cases and update each to include the new step. This is time consuming, and takes testers away from what they really want to do. My colleagues were skeptical about writing test cases in maps instead of Word documents, which then were imported to Quality Center. Luckily, a few testers were desperate to try something new and we worked together on the concept and it has grown.

Here is a simplistic map of how to write test cases for the above. Each Step can be considered a branch. Leafs of the branch can be the different actions.

And if you need to add a step, it is as simple as adding a node and moving items around. 

In future blog posts I will expand on this concept and show you how this will work for more in depth testing.

Your comments and feedback are very welcomed. 

Humble Beginnings

Today is the first day of my blog. I have never written a blog post before.  I have found encouragement from colleagues and people that I have met recently to "just do it" and start writing and sharing my experiences.


My goal of this blog is to share ideas and gain new knowledge from other software testers around the world.


I started today thinking of a name for this blog. I thought of Testing the Limits (taken), Mapping the Future, Building a Foundation (sounds like construction work), and many others. I was trying to play with the idea of using Agile in the title as well well, but I do not want limit myself to Agile only. Yes I am exploring Agile testing currently, but I want to be open to all types of software testing.


There is much for me to learn and I am looking forward to your opinions, your expertise and advice.


Let the journey begin....