The last year has seen a number of companies try to tap into the “wisdom of the crowds” to brainstorm product and service improvement ideas (let’s call it ‘crowdstorming’). Perhaps the most famous of these is Dell Ideastorm, built on Salesforce’s crowdsourcing platform. Starbucks’ MyStarbucksIdea is built on the same platform, with a custom skin.
While these platform works okay as a general idea collection mechanism, it doesn’t work particularly well as a brainstorm. For one, the same ideas are always at the top — unlike the real Digg, where there’s a notion of recency mixed in with popularity, the same ideas have been on the front page of IdeaStorm for the last 4 months. And we found consistently negative comments, and a lack of building off each other’s ideas (constructive ideation, as we called it). So we set out to try to build a better crowdstorming system…but first:
A quick background on brainstorming
While we all have a pretty good idea of what “brainstorming” means (typical image: one person standing at a whiteboard, other group sitting around throwing out ‘wild ideas’), it’s worthwhile to see what’s been said about it in the literature. The term was coined by Alex Osborn in the 1950’s, and the debate has raged since then on whether brainstorming works. In 1957, Taylor, Berry & Block wrote Does Group Participation when Using Brainstorming Facilitate or Inhibit Creative Thinking?. It turns out that, in terms of pure idea generation, “nominal” brainstorming groups (people split up and asked to come up with a bunch of ideas, with those ideas then pooled together) beat out group brainstormers, and this result was replicated multiple times in the following decades. Of course, measuring number of ideas generated doesn’t measure idea quality, or the experience of each of the brainstormers, but served as the metric for most studies. Most papers attributed group brainstorming’s shortcomings to “process loss” — the inefficiencies and other stumbling blocks that come when humans come together to interact, including shyness, folks talking over each other, and more.
The Computer-Supported Cooperative Work field tackled this problem by introducing electronic brainstorming tools — by reducing process loss, these electronic brainstormers were able to match nominal brainstorming groups. And finally, the idea of “brainwriting” emerged in the ’90s — the concept being that each participant would write down ideas, and the next participant would use these ideas as a starting point, a sort of hybrid between nominal and group brainstorming. This last technique was found to outperform nominal, group, and electronic brainstorming.
In terms of what to do in a brainstorm, we can turn towards IDEO, who formalized the much-cited rules for brainstorming:
- Defer judgment
- Encourage wild ideas
- One conversation at a time
- Be visual
- Build off other’s ideas
- Go for quantity
- Stay on topic
What does this mean for crowdstorming?
It seemed to us that ‘brainwriting’ would be a great concept to bring to the crowdstorming space. Our hypothesis was that current Digg-like systems encouraged Digg-like behavior — contributing your own 2 cents without much concern about building off other ideas. What if we could build a system that felt more like a real brainstorm, with the concept of brainwriting thrown in? We chose three design principles:
- Use a familiar metaphor (we chose Post-Its) to bring folks into a brainstorming mind-set
- Provide a varying set of ideas that visitors are presented with, to encourage seeing the ‘long tail’ of ideas
- Make constructive ideation the primary interaction in the system.
We built Ideas2Ideas, a Web-based brainstorming system:
The system uses a Radial Graph visualization to show ideas, and was built with a Javascript frontend (using the very cool Javascript Infovis Toolkit), and Django on the server.
Evaluation
We wanted to test if this system actually did encourage constructive ideation, when compared to a Digg-like system such as Dell’s. We used Mechanical Turk to recruit almost 100 participants, and ran them through either condition (between-subjects test). To build the control condition, we used Pligg, an open-source Digg clone, and replaced all mentions of “news stories” for “ideas”.
The results were pretty dramatic:
| Idea Type |
Dell-style system |
Ideas2Ideas |
| Constructive ideas |
25 |
43 |
| New, not constructive ideas |
52 |
35 |
Results from user study; number of each idea type by condition.
Using a X2 test of independence, we found that that Ideas2Ideas was significantly (p < .001) more likely to encourage constructive ideas.
So what?
Of course, getting people to build off each other’s ideas is only one step to a successful brainstorm. As Mozilla and other organizations move to brainstorms with the crowd, however, it’s one design goal to keep in mind: how can we make crowdstorming be constructive, rather than “2 cents”-based like so much other internet discourse? We think a system like Ideas2Ideas is a start. This work will be shown as a poster at UIST 2008, so come by and chat if you’re there.
Note: this project was done with YanYan Wang
