[2022 UPDATE] An Assessment of Modern Agile Assessment Tools

Lately, I’ve been hearing a lot of buzz about agile assessment tools. You know, those tools that can tell you whether you are moving toward business agility? It makes sense that the people who are investing in agile ways of working want to have some measure of whether they are making a difference or not. Or perhaps they want to compare teams within their organization or compare their teams to a benchmark such as a global database of similar teams.

Why Measure Agility?

Different reasons exist for measuring agility. Most people want to see some progress; to know whether they are making a difference. Some of the most common reasons I have heard include:

• As a team, I want to determine if we are getting better over time?
• As a team, what areas need the most improvement?
• As a manager or leader, I’d like to establish a baseline of the current performance for one or more teams.
• As a manager or leader, I’d like to determine which teams need the most help.
• As a manager or leader, I’d like to determine if our investment in agile ways of working is paying off.
• As a manager or leader, I’d like to determine if we should continue to pursue agile ways of work.
• As a manager or leader, I’d like to determine whether to keep paying for training and coaching.

What are the popular Agile Assessment Tools today?

HomeGrown Tools

I don’t know if I have met an agile coach that hasn’t developed their own agile assessment tool. I know I did. I am embarrassed to say that over 10 years ago I developed a very crude tool that purportedly measured agile readiness (I don’t think it did). Tom Cagley was mentoring me at the time and provided me a number of pointers. He should have just told me that my baby was ugly (it was) and to give up (I finally did).

A few years later I was working with another coach supporting Global Markets at Bank of America and we (mostly him) developed a spreadsheet that Scrum Teams would use to measure 50 Scrum practices and outcomes. That tool was much better that my original one and I used versions of it over the next few years. It even provided a color coded graphical summary for multiple teams – something that is pretty common today on the modern agile assessment tools. But it was clunky and labor intensive. And the output wasn’t beautiful or intuitive.

Today I suspect there are still a lot of people using homegrown tools. The benefit of the homegrown approach is that you can tailor the assessment to your specific needs. The downside is that your specific tailoring may result in something that is not comparable to other organizations. Or you might have to go through and customize it for every organization. And home grown tools rarely have the ease of use and fancy graphical outputs that the commercial tools have.

So why would you roll your own when there are so many better options available?

What follows is a list of some of the popular agile assessment tools available today. I’ve tried to capture licensing fees, strengths and weaknesses and other factors.

The Nokia Test

One of the first agile assessment tools that I know of is the Nokia Test. If you are young, Nokia may not be a name you recognize, and it probably isn’t the first company that comes to mind today when you think about agility. But back in the mid to late 2000’s Nokia was the hottest cell phone maker and literally owned 50% of the market. I had the phone below and loved it. It didn’t shatter if you dropped it. But that is beside the point.

In 2005, Bas Vodde created the Nokia test to gauge agility across the agile teams he supported at Nokia. It was subsequently picked up by Scrum co-founder Jeff Sutherland and expanded to include Scrum practices. It remains a simple and quick test of agile and Scrum.

Here is an example of a question and the scoring (0 – 10 points) from the Nokia Test:

The great thing about the Nokia test is that it is free. The downside is that you have to do all the work yourself. You can download the questions from Jeff Sutherland’s blog here and either create a written test or make an online version that you would score by hand.

Where I think the Nokia Test could be helpful is to mix up your retrospective routine. Perhaps once a quarter or once every 6 months you could use this to bring fresh insights. By walking through the various questions as a team, you can identify where improvements are needed or possible.

[See our related post on retrospectives: A Quick Tutorial to Improve Your Retrospectives]

Comparative Agility

Mike Cohn and Kenny Rubin created the Comparative Agility assessment in 2007. As the name implies, a key feature of Comparative Agility is the comparison of your team or teams to other teams in their online database. Within five years of the launch, Comparative Agility already had over 5,500 teams in their database. As the database grew, comparisons could be made to teams of the same age, in the same industry, or developing similar solutions. Today over 14,000 organizations have used the Comparative Agility tools.

Almir Drugovic and Jorgen Hesselberg were early adopters of Comparative Agility, ironically when they were at Nokia in 2008. Almir and Jorgen purchased Comparative Agility from Cohn and Rubin in 2015 and have expanded it significantly since then.

Today, Comparative Agility is more of a continuous improvement platform. You can use the original Comparative Agile assessment or create your own unique assessment. You can also choose from one of the 20+ other surveys authored by thought leaders in various fields including Mike Cohn, Dr. Amy Edmondson, Jutta Eckstein, Dr. Rita McGrath, Esther Derby, Gil Broza and others. Here is a view of some of the currently available assessments:

Comparative Agility is a bit unique in that it doesn’t adhere to a pre-defined maturity model. There is no goal of agile nirvana that teams will work toward and arrive at. Instead, they treat maturity as a journey of continuous improvement without a set destination.

Strengths of the Comparative Agility Assessments:

• Does not require training or certification to administer
• Huge database of comparison companies and teams
• Wide range of surveys – psychological safety, organizational performance index, and even SAFe assessment
• Questions have been validated by data scientists for reliability
• Works great for helping one or just a few teams all the way up to many teams
• Sophisticated reporting based on lots of team assessment data and proprietary algorithms

Weaknesses of the Comparative Agility Assessments:

• The original CA assessment focuses solely on the development team; you have to use other assessments to address the other areas of the business
• I’ve used multiple agile assessments tools from CA on a few different clients. While not much of a weakness, the fact that they have so many options available can be a bit overwhelming.

Pricing for 5 and 20 teams:

• CA provides a free version that allows you to have 5 collectors (5 team assessments)
• The company version allows for up to 20 collectors for $299 annually – this would cover 4 assessments for 5 teams in a year, making it the most cost-competitive of the assessments I have included.

Learn More: You can check out the Comparative Agility website to learn more about the tool.

AgilityHealth

AgilityHealth is an Enterprise platform that aims to accelerate Business Agility through measurement, growth and outcome alignment. AgilityHealth was created by Sally Elatta and her team in 2014. (Disclaimer: I worked with and am a big fan of Sally and Agile Transformation.) I was trained and certified in 2014 as an AgilityHealth facilitator though to be honest, I only used it once in 2014 and haven’t spent much time on it since.

The platform was originally launched to support teams assessing and improving their TeamHealth which continues to be their global benchmark radar. I first envisioned it as a tool to support ‘retrospectives on steroids’.

It is recommended that the TeamHealth retrospective be facilitated at the end of each quarter before the next planning meeting. AgilityHealth certifies internal facilitators (AHF) who walk the team through the radar. The facilitators create an environment of psychological safety to encourage people to respond openly and honestly. Then each team member completes the assessment. Results are shown in real-time on the visual radar graph. The facilitator strives for open conversation and reflection and then helps create a clear growth plan with 2-3 items that gets logged into the platform.

Initially, AgilityHealth used Tuckman’s stages of team development like Forming, Storming, Norming, and Performing but found that teams were not as objective as they would have liked. So they moved to the following levels: Pre-Crawl, Crawl, Walk, Run, Fly.

Over time, AgilityHealth has expanded to offer a range of assessments for different agile roles, and different levels of the organization. It moved from being a tool for team measures to a platform that supports measurement and improvement. Organizations can zoom out or zoom in, look from different angles and gain a lot of insights into how things are actually working. Some examples of the different views are shown below.

Noteworthy Features of the AgilityHealth Platform:

• AgilityHealth now offers a range of assessments for different agile roles (Scrum Master, Product Owner, Product Manager, Agile Managers, Release Train Engineers and others)
• There is a Growth Portal where team members can click on any radar element (competency) to view recommendations for improvements, watch videos or contact an internally approved coach/mentor.
• AgilityHealth has launched the Outcomes & OKR Dashboard in addition to integrations with Jira and others to pull in quantitative data. This aligns with Sally’s vision of visualizing Maturity, Performance and Outcomes in the same platform.
• AgilityHealth, Business Agility Institute and Deloitte recently published the Top Predictors of Team Performance report which identifies which competencies are need to drive performance across 46000 team members.

Strengths of the AgilityHealth Platform:

• The biggest strength is that it is an online platform that provides a comprehensive view of the entire organization from teams, to programs, portfolios, value streams and the enterprise. It is a powerful tool!
• There is a 2 team free trial and a 10 team free trial if going through a partner.
• Based on various findings, the platform will recommend actions that can be pulled into growth plans and tracked to support a progression journey.
• Beautifully designed dashboards and reports that allow for aggregation and drill down.
• Ability to visualize maturity, performance (quantitative) and outcomes (OKRs) in one platform
• AgilityHealth includes complimentary products like AgileVideos, an online library of training videos.
• A large installed base of over 190k users and 18,150 teams so the platform will be around for a while

Weaknesses of AgilityHealth Platform:

• Requires a trained facilitator
• The platform includes services and functionalities that may not be applicable or would be overkill for a single team and perhaps even for a small organization.

Ballpark Pricing:

• Prices for the AgilityHealth platform have been standardized and simplified. The average cost is $2,000 per team or $200 per person (excluding facilitation fees).
• Large organizations can work directly with AgilityHealth for Enterprise pricing options.
• Regional discounts are available and AgilityHealth Partners are able to deliver a small pilot at no cost.

Learn more about AgilityHealth on the the AgilityHealth website or check out this 3 minute AgilityHealth Explainer video.

Agile Fluency

Agile Fluency ® was first published by agile veterans James Shore and Diana Larsen in 2012 and updated in 2018. They created the model based on ways that people practice agile and use it well.

It is positioned as a tool for consultants to use with client organizations. In fact, the home page emphasizes the potential fees to be generated which is a bit of a turnoff.

The tool provides feedback in terms of 5 levels of fluency: Pre-Agile, Focusing, Delivering, Strengthening and Optimizing.

Strengths of the Agile Fluency ® tool:

• Agnostic in terms of practices (Kanban or Scrum)
• Provides a fluency progression

Weaknesses of the Agile Fluency ® tool:

• Requires an experienced agile coach trained as a facilitator
• Facilitation training is relatively expensive

Ballpark Pricing:

• Getting qualified to facilitate is about $5,000
• AFAICT, as a facilitator, it looks like you can charge $5,000 to $10,000 per team per year, or whatever the market will bear.

Learn More:

You can check out the Agile Fluency ® site, download the free eBook The Agile Fluency ® Model, or watch this short video about the model which features Diana Larsen and James Shore.

Lean Agile Intelligence

I recently learned about an assessment tool from Michael McCalla and LeanAgileIntelligence.com. Lean Agile Intelligence was launched in 2017 as a continuous improvement platform that allowed agile leaders to easily build custom assessment templates to directly align to their context, change strategy, and desired outcomes. It orients to those desired outcomes (i.e., Customer Satisfaction, Time to Market, Reliability) of the transformation efforts as well as measurement of the specific practices that lead to those outcomes. McCalla created the Lean Agile Intelligence instrument based on his expertise with several enterprise-level transformations with Fortune 500 companies.

Other Assessment Tools

There are many other lesser-known agile assessment tools from a variety of companies. Some of the more recognizable company names include Thoughtworks, IBM, Eliassen Group, Atlassian, and NetObjectives. There is also the Scrum Survey which is a scrum-flavored assessment that offers free and paid functionality.

Do you have experience with one or more of these tools that you would like to share? Please leave your comments below.

Hat Tip – I’d like to thank Randall England, Vince Yabut, David Shapiro, Jorgen Hesselberg, Michael McCalla and Sally Elatta for taking the time to speak to me about the assessment tools above. Also, a belated shout-out to Ben Linders for sharing his exhaustive (exhausting?) list of self-assessments. Check them out here: Self-Assessments list from Ben Linders