Rapid Rolling Research
Why rolling research is gaining in importance, how it can benefit your organization, and key considerations when planning to start a program
What is rapid rolling research?
To gain an understanding of what rapid rolling research is, it is helpful to first consider how traditional UX research projects are organized. How they work in general, is that a research requirement is identified, be that generative or evaluative, research questions are defined, a hypothesis is established, research assets are created (i.e. prototypes), participant recruitment is organized and then research takes place. There is a defined need and research is designed and executed to meet that need.
With rapid rolling research, there is a program constantly running, perhaps weekly or bi-weekly, where research sessions are booked in advance without necessarily knowing what the subject will be or precisely which stakeholder they will be for. There is an assumption that demand will be there, and that the shape of the program will be able to meet the needs of stakeholders broadly, rather than precisely.
It’s a lot like eating a meal at Yo Sushi. There is a menu of sorts but in any case, the conveyor belt is constantly moving with the knowledge that someone is going to consume what is on offer.
I define rapid rolling research as follows:
Definition: Rapid rolling research is a system to deliver primarily evaluative research using a consistent approach to a pre-agreed, fast-moving and ongoing schedule.
The UX research industry has a habit of slightly altering something and renaming it, and in case you are wondering if that is what’s happened here, I’d like to contrast other programmatic approaches and consider the similarities and differences. The following table compares rapid rolling research with rolling research, rapid research, continuous research, agile research and iterative research.
I have not included “ongoing research” which is sometimes confused with continuous research. Whereas the others are all qualitative in nature, ongoing research is more about quantitative data collection through surveys (NPS) and analytics. That doesn’t mean it isn’t valid, it is, and you should be doing it, and unless you were cryogenically frozen in 1999 and have just been woken, you almost certainly will be.
As you can see in the table, there is some cross-over between the approaches. My advice would be to not get too bogged down in the names. They will inevitably evolve further over time and where we land is likely to be driven by which research SW platform provider spends the most money on advertising! New research approaches that drive technology adoption are a great form of marketing, but Rapid Rolling Research is more than a marketer’s invention.
The reason I use the term “Rapid Rolling Research” is that to me, it best describes what is happening. It isn’t just rapid, and it isn’t just rolling, it’s both or why do it?
Why is rolling research growing in importance?
Researchers will suggest that the reason for doing rolling research is that it allows for a continuous flow of insights into the product development process. And certainly, in terms of outputs that is true. However, that is not the sole reason why organisations adopt the approach, and in most cases, it is not the driving force at all.
As I illustrated in the table above, the focus of Rapid Rolling Research programs tends to be efficiency and scale. Those are not terms typically associated with UX research. For sure the emergence of Research Operations (ReOps) has raised the efficiency stakes, but this is still nascent. It is indicative of organizations trying to get a handle on their capability.
I posit that the rise of Rapid Rolling Research is linked to the rise in total cost of ownership of research platforms. As research democratization has emerged, organizations are investing increasing amounts in research platforms and in the resources that use them: whether directly, in the form of researchers, or indirectly with “people who do research” (PWDR).
It has long been the case that the more UX research an organization does, the more it wants to do. Now that the barriers to doing research have been reduced through technology, cultural shift away from labs thanks to the pandemic, and far wider involvement of the team members via PWDR/democratization, the demand for research is exploding. We have now reached a point where organizations are spending millions of dollars on research, and they want to take back control of that investment.
How can Rapid Rolling Research benefit your organization?
The Pareto Principle (aka the 80:20 rule) states that for many outcomes, roughly 80% of the consequences come from 20% of the causes. When applied to UX research in product development I think it is a fair approximation that 20% of our time is spent in discovery research and 80% on evaluation. Whereas 80% of the value of research comes from discovery and 20% from the evaluative stage.
A program of Rapid Rolling Research enables an organization to get control of the 80% spend and make it more efficient. It establishes a home for the high-volume requests for usability testing. And it delivers the research needed in the most predictable, consistent, effective and efficient way possible.
If your organization has built a dedicated UX Research team, you might be keen to ensure that team is investing its time in the most valuable areas. Constant demand for evaluative research is hugely distracting, and the implementation of a Rapid Rolling Research program can free this team, or parts of it, to focus on discovery.
The program economics are also a major benefit. If you invest in a Rapid Rolling Research program, the amount you invest will deliver a defined outcome. Over the course of a year, you will know that if you invest [say] $2 million dollars you will deliver 200 studies. You can then focus on measuring the impact of research and ultimately generating an accurate return on investment figure.
They key considerations when starting a program
To finish off, I thought I’d share some of what we have learned by delivering these programs over the past 20 years. Pre-pandemic these would have been very lab-centric, whereas now they tend to be remote. However, many of the learnings apply to both.
It is a team sport
Building a Rapid Rolling Research program does not result in the creation of a research factory where stakeholders throw their requirements over the wall and get back insights a few days later. UX Research is always a team sport, but with Rapid Rolling Research it is fundamental to the programs’ success. There are two main reasons for this as follows:
1. Speed: because it moves so fast, the researcher needs the stakeholder to engage. If the study cadence is weekly and a new requirement each week, the researcher may not know anything about it until the briefing meeting. The stakeholder needs to work in collaboration to ensure the researcher gets the support they need to deliver the insight required.
2. Cadence: my colleague Jason calls studies perishable in Rapid Rolling Research programs, and he is spot on. If the stakeholder isn’t ready they may lose their study slot. It is therefore really importance to educate the stakeholders about the nature of the program, how it works, and their responsibilities as part of the team.
In well run programs the researchers and stakeholders form great partnerships both during the week or two of the study and also outside of it. That is because it is almost inevitable that the stakeholder will be using the program multiple times across the course of a year.
Before you even begin planning the program, start engaging with your stakeholders. Make sure they understand what the program is, what it can deliver for them and their role in its success.
Researcher fatigue / burnout
Whilst Rapid Rolling Research relies heavily on software to make it possible, it is important to understand, it is ultimately a service being run by a human. Research is mentally taxing, and your team will become fatigued. There are a few things we recommend:
· Recruit for the right behaviours: it is not just about hiring great researchers, you should try and identified people with the right behaviours that will relish the opportunity to be constantly researching.
· Create a team approach: if you can afford it, have the teamwork in pairs, one moderator, one note taker. That allows built in down-time and will reduce the researcher fatigue.
· Build in breaks: if you have the ability to rotate researchers in and out of the team that will help keep them fresh. It may also help the to develop valuable skills and capabilities they bring back to the Rolling Research Program.
Quality
With the rise of research democratization, the execution has been put in the hands of a wider range of people than ever before. Not all of these have research training, and if you are going to build a Rapid Rolling Research program, you need to equip your team to deliver quality insights.
Given the high cadence of these programs, there can be a lot of pressure on the moderator. This is exacerbated by the lack of emersion time in the subject and that they may be learning about the research topic for the first time within a day or two of needing to run moderation. When your researcher gets stressed, you need them to rely on their training and avoid leading the interview participant, biasing answers, asking closed questions.
In my opinion, this type of program is not the place you “blood” your junior researchers or ask your designers or product owners to step in. If you can’t resource it properly, it probably isn’t the right solution to the efficiency and scale problem you are trying to solve.
ReOps capability is crucial
Assuming you have good researchers in place and have educated your stakeholders to fit in with the requirements of the Rapid Rolling Research program, the only other areas that can catch you out all live in ReOps.
Rapid Rolling Research is like a furnace that needs to be constantly fed with participants. They must be recruited, have their personally identifiable information managed, have consent and NDAs administered, be given the correct access links to join the research and must be paid the stipend when they have completed it. If you are not set up to recruit participants at scale, you won’t succeed.
Don’t think that because you can recruit for traditional research projects the same processes and approach will work here. They won’t. Invest time in building the capability to support the program before you start as part of the planning stage. That could even involve building your own opt-in customer panel to support the program.
Be prepared to say no!
According to Elton John, sorry seems to be the hardest word. In Rapid Rolling Research the hardest word is “NO”. The no in question is “no this research cannot be done using the Rapid Rolling Research program”.
I already mentioned that Rapid Rolling Research programs work best with evaluative research. Usability testing and the like are ideal methodologies for high-cadence research. This is not least because the participant profiles tend to be generic rather than specific.
Stakeholders, that maybe new to UX research, or have less experience with it, can’t be expected to know what will and won’t work. They may bring requirements that are not a good fit for the research, such as discovery or generative requirements. It is important that your researchers are empowered to say no to requests where the research cannot be delivered reliably.
The researcher doesn’t have to deliver a hard no, and the opportunity can be used to develop the relationship with the stakeholder. Ideally, you will have a parallel research program for more strategic / generative type research requests that you can quickly switch them to. This will keep the stakeholder engaged and ensure they get the insight they need. If the program develops a reputation for delivering weak findings it will be very damaging.
Demand will outstrip supply
Finally, if the program is running well, be prepared for it to grow. There will be more demand than you originally anticipated, and you should have a plan in mind for scaling.
Conclusion
Rapid Rolling Research is a system for enabling organizations to scale their evaluative research capability. It is a high-cadence approach that will deliver a predetermined number of studies at an identifiable cost-point. The approach is not to be confused with continuous, agile and iterative research.
To be successful we recommend that you engage with everyone that will be involved in the program whether researcher, ReOps or stakeholder. Setting appropriate expectations, establishing systems and processes and managing participants effectively are all vital considerations. So too is ensuring you have the right researcher skills in place and pay attention to their levels of fatigue.
Rapid Rolling Research is for organizations that are growing and where demand for research is high. They offer a great way to deliver efficiency and scale. They are a natural evolution from democratization and will enhance an organization’s ability to deliver evaluative research at scale.
