crowdlog

Storyboarding — a technique that grew from comic books to the film industry, where it’s used to prototype scenes and story flows — is a useful tool in design, as it allows designers and other stakeholders to understand the full context around an idea, software, or service. Rather than just a simple sequence of screen-shots or wireframes, storyboards often include the context of use —- where is the user, what is she doing, how did she get there? They’re also great for non-desktop based interactions, like ubiquitous computing and mobile interactions.

We were interested in two questions around storyboarding and online masses of users:

1. Can masses of users quickly understand and respond to storyboards?
2. Can participants from a mass create compelling storyboards?

We quickly put together two variations on a simple idea —- how could you use an iPhone in a museum — and mocked up a 3-panel storyboard. The results were the following:

We asked 30 people on Mechanical Turk, for $0.10, to tell us which idea they preferred, and why. We got the 30 answers back within 19 hours (from 2pm of one today to 9am the next). Here are some of my favorite responses:

Both storyboards present viable points. While it would be prudent to first show the options in the museum, it may not be necessary to give a user map to the exhibit-many users do not need a tiny GPS system when they are wanting to be able to explore somewhere (and not have to worry about directions for once). I would therefore recommend the first storyboard since it gives the user more freedom in the museum and they can look at the information available at their own will. It’s simple and straightforward

Or another:

I like storyboard 2. Museums can be overwhelming at times. The map would allow a user to easily find and view exhibits they are interested in. Students who are visiting for a school assignment and have limited time to partake in all the exhibits would definitely find that option helpful.

Out of 30 people, 29 spoke to the ideas in the storyboard, which is the desired effect; one response focused on the art, instead:

I prefer the second storyboard. There’s a little bit more continuity within the graphics. The second and third panel both depict the iPhone and the 1st and 4th panel show the viewer relating to the environment. In the first storyboard, the box in the frame represents something entirely different each time: first the museum, then a painting, then an iPhone representing a painting. I also think that the feature described in the second storyboard is more immediately appealing—the phone helps orient you in an unfamiliar environment. The feature in the first storyboard is arguably a little more annoying—the phone will continue to solicit your attention as you’re looking at artworks.

Next, we wanted to know whether participants on Mechanical Turk could generate these storyboards; we asked them to put together a 3-to-5 panel storyboard in PowerPoint illustrating an idea they had that related to using location awareness on a mobile phone. We set up a task at $0.50 each. Response rates were far, far slower. Our first submission came within 40 minutes, but the next one took 26 hours; none others came.

Was the money not enough for this task? We changed it to $1.00 and received 6 responses in 24 hours, followed by a final two within 5 days. Still not the rapid response we had for the previous assignment, but reasonable.

Now, back to the research question: were the storyboards compelling and understandable? Some missed the point entirely, having one static composition for the 3 panel, only changing the labels below. A few really grasped the idea, though, and came up with some surprisingly effective results:

Overall, this initial exploration into crowds storyboarding taught us that the storyboards work well as a quick way to communicate a design idea with a crowd and get rapid feedback, but due to either the insufficient existing storyboard tools (in this case, PowerPoint) and the substantial time investment needed to make a reasonably complete storyboard make it less suited to gathering information from a crowd.

Dean Eckles has a very thoughtful post on his blog about how to manage throughput and prioritization of Human Intelligence Tasks (HITs) on Mechanical Turk. It’s something I’ve been thinking a lot about, and so I thought I’d elaborate on some of his points.

Dean breaks down the factors that affect latency and throughput on mTurk:

1. HIT and sub-tasks duration
2. Available workers
3. Appeal of your HITs
4. Reliability

In conversation with Brendan O’Connor from Dolores Labs a few months back, we tried to come up with a simple way of thinking about what affects the likelihood of participation. Building from these thoughts, I think these are the factors:

1. Appeal — How fun the task is
2. Effort — How much effort is involved
3. Reward — How much the payment is
4. Duration — How long the task takes
5. Visibility — How likely your task is to be seen

These are similar to Dean’s, which shows they echo the experience of multiple ‘requesters’ on mTurk. One addition I’d like to suggest is to think of the relationship between factors: a particularly fun task will get good response even if the payment is low, and vice-versa; task length and payment also are related (I’d like to measure this by scaling payment with time and measuring throughput); particularly difficult tasks are often ignored unless they have higher rewards; and, finally, none of this is any good if your task doesn’t get selected by anyone.

This last issue is one I hadn’t understood until I got engaged with the “Turker Nation” community, asking them why our tasks were getting fewer responses than I’d like. The answer was that experienced ‘turkers’ look for three things when choosing a Human Intelligence Task (HIT):

1. A Requester they know and trust (Amazon, one of the better-known transcription services, etc)
2. A HIT with a lot of assignments that they can perform in a row, so they can get in a ‘groove’
3. A recently created HIT

As independent researchers without the brand-name appeal of 1 and often without the tasks that provide for 2, the recency becomes an important factor. Distributing your assignments over a period of time by adding them to the system as interest in your task dies off has been an effective tactic for managing recency; updating your HIT (with any new detail or assignments) will push you to the top of the recency list.

Labmate Björn Hartmann has lately also been getting interested in using Mechanical Turk for some rapid data collection and story-sharing; he has an awesome summary of some of the stories collected so far in “Let’s talk about cats”, presented in toonlet form.

A fun experiment Emily and I ran this week had a simple question: what kind of interesting, unique, or surprising stories could we gather on Mechanical Turk regarding use/mis-use/re-use of popular sites like Google and Facebook? We were inspired by Eric von Hippel and Don Norman’s recent articles (von Hippel; Norman) on end- and lead-user innovation. Their claim is that inspiration for design and innovation can come from observing and collecting the ways your users are adapting products for use in their own lives.

We paid 1¢ each to participants. At first, we asked them:

What’s a strange, funny, or weird way in which you’ve used Google, Wikipedia, eBay, or Facebook?

Our first 100 answers back were of limited success; here’s what we found:

1. Too many people repeating obvious uses — “I use it to search for things” for Google, “I buy things on it” for eBay
2. Not enough stories, mostly one-liners
3. A few nuggets within the pile
4. eBay answers were almost worthless; there wasn’t much variety in what you could do on eBay — the platform is a well-scoped service

We revised our prompt and design and also increased the size of the response text box, to encourage longer answers. Our new prompt was:

Please tell us a story about how you used Google, Wikipedia, Facebook, or Twitter in a surprising way in which most other people don’t

We varied the wording a bit, but this was the gist of our prompt. Our responses this time were far better; paragraphs rather than one-liners, stories with depth and surprise. We collected 229 responses ($2.29 plus commission to Amazon), about 150 of which made it in to the final list (after we got rid of offensive, incoherent, and non-interesting answers). We grouped and visualized the ideas by site; here are the thumbnails, click for full size (about ~1MB-1.5MB each):

Google



Wikipedia



Facebook



Twitter (only 11 responses)

Also check out the 164kb PDF with all four together.

Some of the most interesting responses

On Facebook or other Social Network:

It was actually through Facebook that I found a sister that I never knew. My parents had given her up for adoption when she was born because they were not yet married and no means of caring for her. I searched and searched for her, but it wasn’t until I discovered Facebook that I discovered her:)

I have had a renewed communication with my daughter using Facebook. She is 26 and i am 50, so this is really wonderful! Every couple days, we chat through Facebook, and we can see significant details of each other’s lives. This is very helpful, since we live 2000 miles apart.

I’m a young faculty member at a university. I use facebook to keep up with my friends and colleagues — and to get important information out to my classes! When there is an announcement that people need to see quickly, I’ll write a facebook note and tag all of the members of the class concerned. That way, everyone who needs to know gets notified as soon as they check facebook — usually much faster than email!!

I made a fake myspace page and pretended to be someone else so I can spy on my boyfriend and to see if my boyfriend would flirt with my fake identity

On Google:

We use this service to follow our daughter’s sports in local newpapers and then can forward the info directly on to colleges that are interested in recruiting her for their sports teams

I google my name and the serp is empty; it is very humbling.

When enteing names in the Google seach engine My children and I entered my husbands name after already entering several others and receiving nothing interesting. Upon entering the name several entries were received back with the name. Upon investigating these entries we discovered my sister inlaws mom. My husband had never met this sister and it seems that she was looking for his dad who had disappeared some time ago. Upon further investigation of these hits we discovered some medical problems that ran in the family that explained fully some odd behaviors my husband had been having that could also effect our children. Without the google seach engine This info would never have been found.

I googled my mother. I was 31 years old when I found my Mom through a service I found on Google. I’ve still not met her in person, but I’ve spoken to her several times on the phone. She has two other children but even though I wasn’t raised arond her I am the one who is the most like her.

I had tried numerous times over the years to find my best friend from primary school but she was not listed on Friends Reunited and none of our classmates knew what had happened to her after university either. She was from Sri Lanka with a long surname that I could not remember in its entirety. One day I decided to Google just her first name, even though I felt that there were probably hundreds of Sri Lanks called “Pireeni”. Miraculously, I hit on her immediately. She is now a well know poet in San Francisco, we have remade contact, are in touch regularly and I hope to go to San Francisco to meet up with her in the future.

I have used Google to determine whether one of my college students’ papers is partly or wholly plagiarized. We don’t use turnitin at my campus, so I must rely on Google. It’s amazing that students don’t realize that I have access to the very same Internet that they do! They copy and paste entire pages or articles into their papers and then pretend that they wrote them on their own. And this is AFTER I have told them not to do this because they cannot outsmart me!

Years ago I used google as a sort of divination thing. I needed something to do cause I was bored so I typed in “Jessica likes to” and it gave me results for things I could possibly do. I read Jessica likes to dance, so I ended up going dancing that night.

On Wikipedia:

I was in the Navy and the computers on the ship would allow .org sites to be visited anytime, while .com sites were limited to save bandwidth. I learned ALOT in those 6 months surfing wikipedia and it is still one of my favorite sites to use.

Yes. I bet a friend that there was a permanent space station docked on the moon. I thenk changed the wikipedia article and added the false statement about a space station being docked on the moon. i then showed him the article and he believed me.

Wikipedia is how we resolve pretty much any dispute we have at the bar, using mobile internet.

One day i spent 4 hours on Wikipedia trying to see if someone/anyone in any article had mentioned my name. I checked my school, community, city, hometown and just about anything and everything associated with me. Bad luck.

I like to play a game I call ‘6 degrees of Wikipedia’. I click on the random article button, and decide some topic I want to find from this starting point. For example: Mr. T, or Trigonometry, or Star Wars. From the random article I try to click through the links towards my goal in as few clicks as possible. It’s a fun and challenging game that kills time and can have surprising results. It’s interesting (to me, anyhow) how things link together!

I have used Wikipedia to ask questions that I would normally not ask family or friends. Sometimes it is something that I probably should have known already by my age. People are usually nice on Wikipedia and I can remain anonymous.

Finally, on twitter:

I found out my sister-in-law was using Twitter. I was curious what she was writing about and although she mentioned just about every minor detail of her boring life on the site, she made no mention whatsoever of attending an evening recently with my husband and I. I figured we were low on her list, but there’s the proof.

I live-blogged via Twitter during the birth of my son! It gave family and friends moment-by-moment updates at the labor and delivery without having to deal with multiple telephone calls.

letting my friends know when a inpromptu concert happened. People were there in 15 minutes.

Several of our friends who live far away use Twitter, and my partner and I tweeted our wedding vows to one another so our friends who were unable to attend could still share our experience virtually.

Overall, there was a remarkable breadth and depth in responses; I’m unaware of any other needfinding/data-collection method where you could get such a diversity of thought for less than $3. Of course, there are trade-offs: the stories may be made up; it is much harder to follow up with each person; the stories often lack context and the demographic information needed to group and tally these needs; and finally, self-report will miss other surprising behaviors from participants that would require a trained observer to verify. But we found the results encouraging and are looking forward to following up with more.

As part of our research on what user activity we might want to understand better through crowd-based approaches, we started to think about what people actually do on the Web. This list is by no means exhaustive; any suggestions for additions or modifications would be welcome. I think it’s a good starting point for thinking about how we can improve each of these activities or measure them:

• Classify
  • Rate (an image, a video, an article, a song)
  • Tag (an image, a product, a video)
• Contribute
  • Post (an image, an article, a classified ad, a video, a song)
  • Write (a Wikipedia article, a blog post, an e-mail)
  • Comment (on an article, on an image, on a video)
  • Discuss (an idea, a position, an article)
  • Draw or Paint
  • Edit (a Wikipedia article)
  • Sell (a used book, your home-made cookies)
• Communicate
  • Talk (to friends, through Skype, to tech support)
  • Share/Recommend (a link to friends)
  • Email (a friend, someone you admire, a contact)
  • Connect (with people with shared interests, with potential employers)
• Consume
  • Click (on a link)
  • Listen (to a podcast, to a song)
  • Watch (a news video, a video podcast, a TV show or movie)
  • Read (almost anything)
  • Buy (a book, a ticket, almost anything)
  • Search (for information, for people)

Which of these will be most important in the next five years? Which will become irrelevant? Are there ones you think aren’t even on this list yet, but will be?