Management by IT Research Report Is Inherently Flawed
In opinion / By Mike Vizard / 27 October 2016
IT leaders tend to have varying degrees of confidence in reports from IT industry analysts. Some use them to provide third-party validation for a course of action they have already decided to take. Others, however, take them for gospel that guides their every action.
Back in the day when IT decisions were made on an annual basis many of these reports provided a lot value than they do today. Now as the development of IT technologies had become more rapid many of these reports are outdated even before they are published. In fact, this rapid pace of innovation is why more purchasing decisions are being pushed to lower levels inside the organization where, for example, developers are likely to have more hands-on experience with various classes of technologies and IT platforms.
On average it takes analysts firms such as Gartner and Forrester Research well over a year to research a particular IT segment. In addition to interviewing all the vendors, there’s the painstaking process of surveying customers to get the broadest perspective possible. The trouble is that by the time all that data gets collected and correlated, any number of innovations have been brought to market.
In theory, the authors of the reports are expected to bring to bear their expertise to provide some insight into what impact these emerging technologies might have on purchasing decisions in the future. That, of course, assumes they first know about them and secondly, their crystal balls were clear enough at the time to see over a six to 18-month horizon.
Of course, if they truly possessed such prognostication powers, chances are they would in some other line of more financially rewarding work.
Whether it’s a Gartner Magic Quadrant or a Forrester Research Wave report the methodologies being employed to reach the conclusions drawn are inherently flawed. In the last 12 months the IT industry has witnessed the emergence of two forms of microservices enabled by the rise of containers such as Docker and serverless computing frameworks such as AWS Lambda.
While both these computing models are still in their infancy, most analysts are already being forced to rethink most of the assumptions that were made in any number of reports published in the last 12 months.
Of course, by the time they publish those any number of other new models are likely to have has emerged as well.
None of this suggests that analysts themselves are not smart people. They most definitely are amongst some of the best and brightest the IT industry has to offer. But the mechanisms through which those insights are being package up for consumption belong to a bygone era.
In a world where IT is business, no one should be making critical decisions based on outdated data.
That massive wave highlighted this month could turn out to be little more than a ripple. The quadrant so breathlessly touted could easily become a box an IT organization wishes they never opened in the first place.
Of course, everybody needs a little outside perspective.
But just because that perspective arrived in report consisting of hundreds of pages that cost thousands of dollars, it doesn’t necessarily mean that there is automatically a corresponding amount of value. What they do reflect is what was occurring at a moment in time.
But just as no one would want to buy a car or piece of heavy machinery using information that as outdated, the value of the information in these reports is perishable starting not from the date they were published, but rather the end date the research was concluded.
The good news is there are real-time research methodologies using crowdsourcing techniques starting to emerge in terms of how research data gets collected.
In the meantime, IT leaders would be well advised to consider both the age of the information and the rate of change in any given IT sector before putting too much stock in a report based on, for example, quadrants that wind up being more often mythical than magical because the actual report itself took months, perhaps even years, to arrive on their desk.