Ideas for innovation—how do you tell the good ones from the bad ones?

Some business leaders, employees, entrepreneurs, and organisations cope with constant change by crowdsourcing ideas from stakeholders. But which ideas are best for business and how should managers select them? A recent study by associate professor Dr Dirk Deichmann, assistant professor Dr Inga Hoever and former researcher Violetta Rodopoulou, all from Rotterdam School of Management, Erasmus University (RSM), measured the ‘efficiency’ of ideas in a database of crowdsourced suggestions from customers of a high street retail business with a strong online brand. The researchers’ aim was to find a formula or tool for evaluating ideas for their benefits in comparison to the cost of developing and implementing them.

The RSM researchers used suggestions from Starbucks’ idea crowdsourcing platform to test their theory, and created a toolkit for filtering ideas – by submitting them to an efficiency test as a first step.

Ideas and comments on the ‘My Starbucks Idea’ platform were publicly visible until recently. The research was conducted while this was still the case; the researchers selected 315 ideas from 2016 focusing on 10 categories; atmosphere and locations, building community, merchandise and music, new technology, ordering payment and pick-up, other experience ideas, social responsibility, and the Starbucks card.

The researchers used data envelopment analysis (DEA) to generate an ‘efficiency score’ representing the relative capacity of each idea to convert inputs to outputs. For this, they determined appropriate input and output criteria which were relevant in and suited the context of the ‘My Starbucks Idea’ platform. In the end, they developed five input criteria which reflect what would be needed to implement ideas, and five output criteria about outcomes that ideas might produce. The set of ideas was presented to participants in a survey. Respondents assessed the inputs of one set of ideas, and the outputs of another set on a scale of 1 (very negative) to 5 (very positive). The researchers then checked their respondents’ judgements and compared those with the judgements that came from Starbucks’ management.

The outcome? The alignment was not perfect:

  • Allowing customers to send a gift drink to friend via text message – the evaluation of the idea using the researchers’ coding formula showed this idea scored well for boosting customer satisfaction and loyalty, and improving the brand image, providing that the technological infrastructure exists. Staff training costs would be minimal so this is a highly efficient idea according to the researcher’s hypothesis. But apparently Starbucks rejected it.
  • Vary the food menu to time to incorporate more seasonal and regional items – the researcher’s formula scored this idea relatively highly for customer satisfaction, brand image, and brand loyalty, but showed that it would need plenty of resources to put it into action. The researchers’ assessment was that it would be an inefficient idea – but it got a thumbs-up from Starbucks.
  • A mobile Starbucks coffee shop – like an ice cream van or truck calling at workplaces like call centres or where night shift workers are employed. The researchers’ assessment of this idea using their toolkit showed it would demand more resources to get it off the ground compared to other ideas; spending on such resources would outstrip any potential benefits to customer satisfaction and loyalty. The team’s research classed it as an inefficient idea; Starbucks also rejected it.

Practical and consistent evaluation

The researchers don’t know why Starbucks’ management decided to implement ideas that the toolkit had shown to be inefficient. However, the researchers still suggest that having some quantitative method for measuring the value of crowdsourced ideas for innovating businesses might give a more practical and consistent way of evaluating them – almost like an investment filter.

Initially they also looked into visitors’ votes about these ideas on the ‘My Starbucks Idea’ platform. Interestingly, Starbucks’ judgement didn’t differ significantly from that of external visitors to the site. Generally, the ideas launched and implemented by Starbucks also received a large number of votes by users and thus had a high idea score on the ‘My Starbucks Idea’ platform, but some of the implemented ideas had a surprisingly low efficiency score.

But don’t trust the votes

Previous research shows that while votes may be a cost-effective and fast way to screen and select ideas, they are not a good indication of the quality of an idea. The researchers found that the instinct for reciprocity among contestants makes vote tallies a better indicator of the personal popularity of an inventor than of an idea’s quality and innovativeness. Managers should be wary of using votes as a tool to select ideas, they warned.

While the RSM researchers’ method is slightly more complicated than counting votes, it offers a more nuanced picture of the investment that would be necessary to implement an idea, and an evaluation of the benefits it offers. It allows companies to consider ideas relative to each other.

An efficiency score makes it easier to prioritise and if necessary make a more in-depth evaluation of those ideas that scored above a threshold for efficiency. It’s flexible too, because the input and output dimensions can be adapted as a company sees fit. Ideally these dimensions would then be communicated to potential inventors because it indicates the kinds of ideas that the company considers useful.

Easier to avoid the bad ideas

"Our method might not make it easier to pick great ideas, but it should make it easier to avoid picking bad ones,” said Deichmann. “It helps managers to identify the good ideas, and helps to prioritise them. Companies can adapt the input and output criteria to make it flexible. And it’s easy to show the results of the assessments to potential investors.”

Deichmann also thinks the method is a good way of encouraging people who chip in with their ideas not to give up if their ideas are rejected – because it produces a structure that makes it easier to give feedback.

This research has been published as an article in the Harvard Business Review.

For more information about RSM or this article, please contact Irene Bosman, Media Officer for RSM, on +31 10 408 2028 or by email at

Rotterdam School of Management, Erasmus University (RSM) is one of Europe’s top 10 business schools. RSM provides ground-breaking research and education furthering excellence in all aspects of management and is based in the international port city of Rotterdam – a vital nexus of business, logistics and trade. RSM’s primary focus is on developing business leaders with international careers who can become a force for positive change by carrying their innovative mindset into a sustainable future. Our first-class range of bachelor, master, MBA, PhD and executive programmes encourage them to become to become critical, creative, caring and collaborative thinkers and doers. Study information and activities for future students, executives and alumni are also organised from the RSM office in Chengdu, China.

Image by Oberaichwald from Pixabay


Share this article: