What is the return on your research investment?

Organizations have invested heavily in their product research capability from people to technology platforms, supplier infrastructure, processes, and practices. Despite the redundancies that swept through the market in the past 18 months, the hunger for research is as voracious as ever. But with so many moving parts how do you calculate the return on your research investment?

In this article I seek to identify the elements that constitute each side of the return on investment (ROI) equation. I take a detailed look at the direct and indirect costs as well as the benefits that should be derived from research and how their value is calculated. If you are losing control of the investment in your research capability, this article is for you.

Where has the investment in research been made?

People

The biggest investment that organizations have made is in people. From an organization maturity perspective, you need to have research capability in place before investing heavily in technology. Depending on the size of the organization that may be dedicated research resources or a more general involvement in research by the wider team.

“People who do research” (PWDR) has entered the lexicon because of the democratization of research. Generally, large organizations are now using non-researchers to do some research and hence, despite recent layoffs, the desire to do research is strong. These need to be included in any RORI calculation — if they are doing research, they are not doing whatever else it is they are being paid to do (design, dev etc.).

And make no mistake, the total number of people is growing. Back in 2017, NN/g provided a prediction about the number of people in UX roles. They suggested that by 2020 there would be 1-million people employed in UX roles and by 2050 about 100 million. A LinkedIn search for “user Experience” provides over 2-million results and search for “UX” adds a further 2.3-million (assuming no overlap).

Another example is the role of Research Operations. This is the dedicated function supporting research activities across an organization or in a team. This role has existed for only several years and yet a search on LinkedIn for people with that in the job title will deliver 13,000 results.

An organizations’ people investment can be made in full-time employees (FTE’s) or in freelancers. Freelance is huge in digital and in particular in user experience. Forbes estimated that 35% of the US workforce is freelance and that the largest group offers “skilled services” such as programming and design. Together these are a significant cost to the business.

And we can’t ignore the new roles being created such as Head of Product Insight. Not only is there data from qualitative and quantitative research, but also onsite feedback, analytics, call center, PoS and more. In a recent interview with a product leader they described the “hosepipes of data” confronting product owners. With so many data points to utilize in the drive for high performing products & services, a dedicated resource is becoming a must have.

Technology

Pre-pandemic, organizations needed to invest in hardware if they wanted to do research. The researcher’s “kitbag” contained high-powered laptops with dedicated graphics cards, picture in picture software, webcams, desk mics and various other paraphernalia required when doing research in person. Post-pandemic, 99% of research is carried out remotely using organization-wide tools like Zoom, Google-Meet or Teams. And also, specialist tools like Lookback and increasingly platforms like Usertesting.com.

In addition, there is an increasingly long menu of technologies being used for research. Dedicated tools for specific methodologies such as Optimalsort for card sorting or SurveyMonkey for quant research are just two examples. Think also of the evolution of Figma, Jira and the research capabilities being built into these previously dedicated design tools.

There are also research repositories, which have been gaining growing momentum. These are configurable cloud storage systems, many with built in research tools. Once an organization commits to these platforms their entire research history is locked into an ongoing subscription cost.

The payment model for these different tools varies considerably. There are some that charge on a project-by-project basis and others that offer project, monthly or annual subscriptions. In all cases the prices increase as additional services are bolted on. Services such as participant recruitment panels, multiple user profiles and team accounts, additional storage, data policy management and more.

Where tools are dedicated to the research team it is relatively easy to calculate all the costs associated with the research activities. Where they are shared, such as with Figma, or the use of SharePoint as a repository, or even Zoom for running interviews, it can be far harder to understand the cost allocation. You also need to be on top of license and subscription management which if not done in a timely fashion, can generate a cost burden even if no research is taking place.

Service suppliers

Before all the cloud software subscriptions costs, the biggest external costs for research were research facilities (Labs) and participant recruitment. Approximately 20% of a research projects’ cost could be invested in these two elements alone.

With remote research now dominant, participant recruitment is the main third-party cost outside of tech. Recruitment providers charge project management fees ranging from $500 to $1,500 per project plus a recruitment fee per participant from $75 to $300. On top of this we have incentive fees which again range from $50 to $300 and higher.

If you are doing international research, there will also be costs associated with translation of documents and simultaneous interpreters. If you don’t have researchers with the right language capabilities, you may also be paying for research agency support.

Soft costs

Even more difficult to measure are the indirect costs that are created simply as a result of doing research. Involvement in deciding what research to do or when to do it, can involve stakeholders not directly linked to the research capability. Doing research when you don’t need to and not doing when you should be, are also costs that are hard to measure.

There are also activities such as feeding back insight, which again pulls in senior stakeholders who invest time to learn about the findings from research. There may also be extended sessions, such as workshops used to make decisions based on the insight gathered. Even watching research is an investment of time that is a soft cost for those not directly involved.

Although research is far more agile than it used to be, there can be an impact to the progress of design or development. While evaluative research can be built into the process more easily generative methodologies may well involve non-research stakeholders. If critical product decisions are to be taken off the back of research findings there may well need to be a pause while they are generated.

And speaking of critical product decisions, with the introduction of democratization, there is not only the cost of the person conducting the research, but also the potential cost of using inaccurate data. For example, a person untrained in how to avoid the use leading questions, may generate insight that delivers false positives. There is inevitably a pay-off between funding professional researchers and throttling the volume of insight generated and driving volume through democratization potentially at the expense of quality.

Where should the investment in research deliver returns?

There are two sides to the ROI equation: the investment and the return. So far, we have identified how to track down the full costs associated with the research investment. And for every investment we require a return so what are we looking for from product research?

There has been quite a bit published about the ROI from usability testing (evaluative research). Expected returns range from 10:1 to 100:1 but hard and fast calculations are hard to come by. Most of the customers I speak to who “get” research, tell me they intuitively know it is both saving and making them money, they just haven’t collected the evidence to prove it.

For user/ux/product research overall I have never seen a calculation in my 20+ years in the industry and I suspect that is because it is in the “too-hard” box. Nevertheless, I’d still like to explore where we should be looking for returns so that we can attempt to balance our equation.

More revenue

Ultimately, our investment in product research should deliver high-performing products and services. If these are directly revenue generating, such as cloud software or services, it is easier to capture the total revenue generated over the lifetime of the product or service. It becomes harder when products and services evolve over time so that there is no clearly ring-fenced timescale either for the investment side or the period over which to calculate returns.

There are also products and services that facilitate transactions, such as in banking, e-Commerce, gaming, entertainment etc., that indirectly generate revenue. With these, we can look to conversion rates, growth in customer value, lifetime customer value, average revenue per user (ARPU), and various other financial success measures.

I think one of the key challenges here is accountability. The period over which these revenue generation measures are captured for ROI is so long that the people involved change before the data is captured. Long timescales don’t really fit with the agile, speedy nature of digital development — even if your transformation project is in its 5th year!

Reduced costs

Many organizations have embarked on significant digital transformations or transitions. And with these, they are often motivated by reducing costs. Self-service applications have a direct financial benefit on organizations attempting to reduce operational costs in call-centers, store real-estate and more. Given the size and scale of these projects they generally have a very well-defined business case that clearly identifies the expected investment and return.

But not all cost-savings are so clearly targeted. Some can be made as a result of operational benefits gained indirectly from doing research. Many product owners struggle to prioritize their product roadmap or to know with clarity what product features their team should be working on. Reducing lost time and investment from developing the wrong things is an indirect cost saving generated by doing product research.

I suspect this is another reason why calculating the ROI from research is complicated and inaccurate. There are many indirect costs that you have to be pretty determined to find. Is it really worth the effort when an organization has accepted the idea that being more customer centered will be beneficial and that doing research is part of that?

Summary

The challenges involved in calculating the ROI from product research are not trivial. I suspect that is why we end up with believers and non-believers because those who haven’t seen it or tried it don’t have any tangible arguments for why they should do it. However, as an industry I don’t think that is good enough and we have to find a way to more closely link the action of research with the outcome in product performance — whatever that may be and however that is measured.

In the meantime, our focus should be on how efficient and effective our research capability is. We must strive to be in control of the myriad moving parts, costs and activities that comprise the function. And ensure that we know when to and when not to invest in product research.