The Limits of Benchmarking

 

Benchmark is a term originating in the world of land surveying but is now universally adopted across industries to mean any relevant point of reference or example against which things may be measured. In urban planning, the term is often used alongside “best practices,” referring to professional procedures recognized as being the most efficient and/or effective for a given subject. The full sentence generally goes “we need to benchmark best practices in parallel to setting the baseline,” with the baseline simply being a detailed description of the current situation. These activities are undertaken with the intent of trying to help figure out what to do, with the benchmarking, and to later determine what has been done, measuring against the baseline. Different markets indulge in benchmarking to different degrees, and in the Middle East GCC countries where I currently work, it is an absolute staple of early project development. And, while it is clearly important to understand the state of the art in any endeavor - one can save a lot of time and misery by learning from other's successes (and failures) - as with many things, the devil is in the detail, and a poorly structured benchmarking exercise can be not simply a waste of time but a counterproductive diversion.

 

In the benchmark business, as practiced in my world, the ask is basically “find me best practices from three (or five) cities that are recognized leaders in (insert activity[s] here).” The purpose is to provide the client with an understanding of how similar issues have been successfully addressed in different locations. Sensible enough. But simply identifying leading examples is not enough, they must also be widely recognized as being leaders by the client themselves, people who understandably find comfort in examples from familiar peer, near-peer, or aspirational other projects or locations. It is rarely advisable to come up with obscure “lesser” examples no matter how germane or insightful they might be.

 

There are several other obvious limitations to taking too rigid an approach to benchmarking, starting with the death of innovation. If you can’t do anything without demonstrating that someone else has done it successfully before you (or several someones), you are stuck traveling in other people’s tracks with no latitude to introduce the new ideas, means, or methods that may be needed in order to succeed in the local context. And context is key. Basing choices on incomplete snapshots of activities from lands far away is highly problematic, particularly in complex environments such as those tackled in urban development. Great ideas tend to work in a given location due to both general and specific social, economic, and administrative factors, factors that often differ in key respects from one place to the next. In my experience, a lack of good examples is rarely a principal challenge, someone, somewhere, can be found that has done a good job at most anything if you look hard enough – it is finding an outside solution fit for purpose to local conditions that is by far the greater challenge.

 

Then there is the matter of focusing only on success. I have certainly never been asked to benchmark great failures, and yet if you choose to go down the benchmarking route you should really look at what has famously worked, and what famously hasn’t worked. Avoiding pitfalls is at least as important as emulating successes, possibly more so as there are far more ways to mess something up than to make a success of it.

 

The process of selection is also problematic, with availability bias playing a large role. All people, even “experts,” go with what they know and place a higher weight of importance on instances that readily come to mind. For certain topics, there are renowned leaders, or at least usual suspects, which inevitably crop up. An example of this in my business is BRT, or bus rapid transit, a way to develop efficient high-volume mass transit using buses and foregoing the cost and infrastructure required for a full-on rail system. While hundreds of cities worldwide now have these systems, Curitiba, Brazil, an early pioneer, and innovator, is a go-to example. Whether or not they are still at the forefront of innovation is immaterial as they are the brand name, the Hermès handbag of bus systems, and no list would be complete without them. For other, more esoteric, subjects it is harder to identify relevant and well-recognized benchmarks, particularly examples a client would wish to be seen emulating. In such cases, it becomes either an internet fishing expedition, or it will require forcing a questionable fit from a more recognized location just to tick the box. I have committed pretty much all possible combinations of the above sins over the years.

 

So, what to do? This depends on how seriously and to what end the exercise is taken by the client. If it is just pro forma, a “this is what we do at the beginning of all our projects to test the resourcefulness of our consultants and maybe get some good ideas”, then everyone can just grin and bear it, continue to trot out the usual suspects, and move on as quickly as possible to the next stage. If, however, one seriously wants to learn from the exercise, then it is important to study the good and the bad, and to focus on how these ideas can be adapted to fit local conditions. At the end of the day, the only best practice that matters is one that will work for you, so beware of any benchmark analysis that doesn’t contain a serious amount of “…and this is how we would adapt and implement it here.”