Credit: CS147 group Martin Alonso, Dennis Paiz-Ramirez, Kate Swanson, and Minjeong Kim.
]]>Joel did most of the dev for this one, and I helped out with polish & UI. Check it out the app store, it’s 99¢ and will hopefully help you (or a friend) out on your next date.
]]>Symbolic Systems (or SymSys) is an interdisciplinary major at Stanford that has graduated 22 classes of students, dating back to 1987. It was founded in the ’80s by several professors who saw a gap in Stanford’s offerings as relates to Cognitive Science (and cognition more broadly). Since it’s interdisciplinary and highly “customizable”, SymSys means different things to different people, but I think Ivan Sag, current Director of the program, sums it up best: SymSys focuses on how people think (Cognitive Science, Neuroscience), and how machines can either help us think (Human-Computer Interaction), or help us understand how we think (AI, computational models). SymSys is unique to Stanford.
I first found out about SymSys through a group on Orkut, Google’s social networking service that no one in the States but everyone at home (Brazil) does; the description was interesting enough to get me to go to the weekly SymSys Forum (Thursday @ 4:15pm in 380-380C) when I started at Stanford. Though some of the talks were over my head, in general I was attracted to the breadth of topics and how interdisciplinarity informed all research in the program. I’ve been a part of the program ever since, through being an undergraduate student, an academic advising fellow, and now one of 8 Master’s students.
In general, there are some strong “pro”s: The SymSys program is relatively small (~30 people per class year), and also very tight knit. From the collection of polaroid pictures right outside the program office, to the quarterly barbeques, you’ll get to know basically everyone in the program, and often get to meet alumni that have gone on to do terrific things (like Marissa Mayer @ Google, Scott Forstall @ Apple, Reid Hoffman @ LinkedIn, and many others). I’ve had countless hours-long conversations with SymSys students & alumni. SymSys has a more liberal arts focus than a “techie” major like Computer Science (which appealed to me coming from a smaller high school), and allows for a sort of breadth in thinking that’s valuable as an undergraduate.
Of course, there are some “con”s as well: it doesn’t have the name recognition of Computer Science or other more well-known degrees (though you can spin this to your advantage in job interviews if you have a well-thought-out reason for choosing SymSys), it’s not classified as a Science & Technology degree (important if you’re a foreign student on an F-1 hoping to get a STEM extension to your Optional Practical Training), and, if some care isn’t taken, can lead to a program with too much breadth and not enough technical skills.
My main interest (and concentration for both undergrad and Master’s) is Human-Computer Interaction — so studying HCI through Computer Science would have made sense as well. At the time, Computer Science didn’t have specific “tracks” like they do now, and the CS core curriculum was much broader, with classes like Compilers and Operating Systems, both of which I appreciate but had little interest in learning much more than the basics. Instead, SymSys had upper-level courses in logic, cognitive science, and experimental design. These were a better fit for what I was interested in.
This section is called SymSys & Computer Science (rather than “vs.”) for a reason: regardless of which route you choose to pursue an HCI education at Stanford, the curricula for both programs are available online, and make a very good source for course recommendations to round out your SymSys or CS classes.
When it came time to choose a Master’s program, there was one additional factor I took into consideration: the SymSys MS program is very research-heavy, and requires a thesis that contributes significant knowledge within your chosen concentration. While it’s possible to do extra work and write a thesis as part of the CS MS program, it’s far less of a priority (and is not as baked into the structure) as the SymSys MS. There are monthly SymSys MS research meetings where students get feedback from the program professors and other students. Also, it’s a much smaller program.
My advisor (Scott Klemmer) encouraged me to supplement the Symbolic Systems curriculum with CS courses that rounded out my ability to think like a computer scientist and actually implement the concepts covered in the HCI courses. While I took classes like ArtStudio60 (one of the better classes I’ve ever taken), at Prof. Klemmer’s advice I also took courses in Object-Oriented Programming, Programming Languages, and Computer Graphics.
With the new CS curriculum, started this year, the picture has changed a little. The new HCI track looks as follows:
Requirements:
a) CS147, CS247 (HCI Foundations)
b) Any one of: CS148, CS248, CS376, CS377, CS378 (Advanced HCI)
c) Any one of: CS108, CS140, CS221, CS223B, CS229, CS249A (Buttressing CS)
d) Any one of: Psych55, Psych252, MS&E184, ME101, ME115 (Designing for People)
Track electives
– Courses in categories (b), (c), and (d) above, as well as: ArtStudi60, Comm269, CME340, CS447, CS448B, Ling180, EE118, MS&E216A, Psych205, Psych221
*requires approval of undergraduate advisor.
Which isn’t too vastly different from the HCI concentration in Symbolic Systems. It’s great that you can get a strong HCI education through either program now; I think the choice comes down to:
Undergraduates
If you’d like to approach HCI from a technical standpoint and gain strong fundamentals in CS disciplines that extend beyond what might be immediately applicable to HCI, the new CS program with an HCI track is a great choice. On the other hand, if you’re interested in HCI from a cognitive, psychology, or social psych standpoint, the SymSys program provides an excellent perspective on how our mind works and how it relates to designing user interfaces. It’s worth coming to some of the SymSys barbeques and other events to get to know the program a bit more.
Graduates
If you’re considering an eventual PhD (or just want to do more research before graduating), SymSys is great way of learning more about research and undertaking your own project, though as mentioned above, it’s important to combine the HCI courses that you’re required to take with some deeper technical expertise courses. If you’re interested in getting a deeper technical knowledge of CS and focusing on how to apply it in an HCI context, the CS MS is a great choice.
Feel free to shoot me an email, or comment below, if you have more questions about these programs or Stanford in general.
Update: thanks to Todd Davies, our Associate Director in SymSys, for helping to clarify some points above.
]]>
Heat City from Paul Doersch (and Mike Lindquist and Chris McCarthy) on Vimeo.
A lot of the students also posted to YouTube.
]]>
Credit: CS147 group Martin Alonso, Dennis Paiz-Ramirez, Kate Swanson, and Minjeong Kim.
]]>
Here are the results, run through Wordle. Please contact me if you want the dataset (~400 answers). Click to see larger versions:
Mostly just a fun experiment, and works much better if you follow semi-famous people whose pages have been tagged on Delicious. It does do a decent job of capturing interests though; here’s the cloud for Twitter CEO Jack Dorsey (@jack):
If you want to check it out, here’s the source (Python). Shoot me an email (mkrieger [at] hci.stanford.edu) if you’d like me to generate a cloud for you.
If you make any improvements, please let me know! Thanks to Brynn Evans for getting me to brush it up and blog about it.
]]>While these platform works okay as a general idea collection mechanism, it doesn’t work particularly well as a brainstorm. For one, the same ideas are always at the top — unlike the real Digg, where there’s a notion of recency mixed in with popularity, the same ideas have been on the front page of IdeaStorm for the last 4 months. And we found consistently negative comments, and a lack of building off each other’s ideas (constructive ideation, as we called it). So we set out to try to build a better crowdstorming system…but first:
While we all have a pretty good idea of what “brainstorming” means (typical image: one person standing at a whiteboard, other group sitting around throwing out ‘wild ideas’), it’s worthwhile to see what’s been said about it in the literature. The term was coined by Alex Osborn in the 1950’s, and the debate has raged since then on whether brainstorming works. In 1957, Taylor, Berry & Block wrote Does Group Participation when Using Brainstorming Facilitate or Inhibit Creative Thinking?. It turns out that, in terms of pure idea generation, “nominal” brainstorming groups (people split up and asked to come up with a bunch of ideas, with those ideas then pooled together) beat out group brainstormers, and this result was replicated multiple times in the following decades. Of course, measuring number of ideas generated doesn’t measure idea quality, or the experience of each of the brainstormers, but served as the metric for most studies. Most papers attributed group brainstorming’s shortcomings to “process loss” — the inefficiencies and other stumbling blocks that come when humans come together to interact, including shyness, folks talking over each other, and more.
The Computer-Supported Cooperative Work field tackled this problem by introducing electronic brainstorming tools — by reducing process loss, these electronic brainstormers were able to match nominal brainstorming groups. And finally, the idea of “brainwriting” emerged in the ’90s — the concept being that each participant would write down ideas, and the next participant would use these ideas as a starting point, a sort of hybrid between nominal and group brainstorming. This last technique was found to outperform nominal, group, and electronic brainstorming.
In terms of what to do in a brainstorm, we can turn towards IDEO, who formalized the much-cited rules for brainstorming:
It seemed to us that ‘brainwriting’ would be a great concept to bring to the crowdstorming space. Our hypothesis was that current Digg-like systems encouraged Digg-like behavior — contributing your own 2 cents without much concern about building off other ideas. What if we could build a system that felt more like a real brainstorm, with the concept of brainwriting thrown in? We chose three design principles:
We built Ideas2Ideas, a Web-based brainstorming system:
The system uses a Radial Graph visualization to show ideas, and was built with a Javascript frontend (using the very cool Javascript Infovis Toolkit), and Django on the server.
We wanted to test if this system actually did encourage constructive ideation, when compared to a Digg-like system such as Dell’s. We used Mechanical Turk to recruit almost 100 participants, and ran them through either condition (between-subjects test). To build the control condition, we used Pligg, an open-source Digg clone, and replaced all mentions of “news stories” for “ideas”.
The results were pretty dramatic:
| Idea Type | Dell-style system | Ideas2Ideas |
|---|---|---|
| Constructive ideas | 25 | 43 |
| New, not constructive ideas | 52 | 35 |
Results from user study; number of each idea type by condition.
Using a X2 test of independence, we found that that Ideas2Ideas was significantly (p < .001) more likely to encourage constructive ideas.
Of course, getting people to build off each other’s ideas is only one step to a successful brainstorm. As Mozilla and other organizations move to brainstorms with the crowd, however, it’s one design goal to keep in mind: how can we make crowdstorming be constructive, rather than “2 cents”-based like so much other internet discourse? We think a system like Ideas2Ideas is a start. This work will be shown as a poster at UIST 2008, so come by and chat if you’re there.
Note: this project was done with YanYan Wang
]]>You don’t have to be a software engineer to get involved, and you don’t have to program. Everyone is welcome to participate. We’re particularly interested in engaging with designers who have not typically been involved with open source projects. And we’re biasing towards broad participation, not finished implementations.
What I like about this call is its emphasis on prototyping and sketching/mockups as communication media, and as intermediate representations. These tools and artifacts fit the open-source development model well, and have been under-used by a community that until a couple of years ago was posting UI mockups using ASCII art in Bugzilla — though recent blog posts documenting design process from the Mozilla Labs and UI folks have been a welcome turn, and Aza Raskin’s blog is a constant source of inspiration.
The Concept Series kicks things off with three video prototypes (one from Adaptive Path), and provide a message forum for the community to jump in and start talking. This last part is what concerns me; what I believe strongly is that online tools have a strong effect on both quality and quantity of participation. Dell’s IdeaStorm, for example, is designed (both visually and conceptually) as a Digg-clone, and so encourages very Digg-like behaviors, which work for a popular links site, but less so for an ideation/brainstorming site.
If a community of thinkers, sketchers, prototypers, and (in one word) designers is to flourish around this Concept Series idea, a site geared towards providing feedback on ideas, running small user tests on prototypes, branching and merging mockups and prototypes, and otherwise letting the design process flourish would be welcome. Maybe that would be a strong early submission to this Series: how can we rethink design collaboration on the Web, given an open-source mentality and a topic as daunting (yet exciting) as “The future of the Web”?
]]>Which of these will be most important in the next five years? Which will become irrelevant? Are there ones you think aren’t even on this list yet, but will be?
]]>We paid 1¢ each to participants. At first, we asked them:
What’s a strange, funny, or weird way in which you’ve used Google, Wikipedia, eBay, or Facebook?
Our first 100 answers back were of limited success; here’s what we found:
We revised our prompt and design and also increased the size of the response text box, to encourage longer answers. Our new prompt was:
Please tell us a story about how you used Google, Wikipedia, Facebook, or Twitter in a surprising way in which most other people don’t
We varied the wording a bit, but this was the gist of our prompt. Our responses this time were far better; paragraphs rather than one-liners, stories with depth and surprise. We collected 229 responses ($2.29 plus commission to Amazon), about 150 of which made it in to the final list (after we got rid of offensive, incoherent, and non-interesting answers). We grouped and visualized the ideas by site; here are the thumbnails, click for full size (about ~1MB-1.5MB each):
Google
Wikipedia
Facebook
Twitter (only 11 responses)
Also check out the 164kb PDF with all four together.
On Facebook or other Social Network:
It was actually through Facebook that I found a sister that I never knew. My parents had given her up for adoption when she was born because they were not yet married and no means of caring for her. I searched and searched for her, but it wasn’t until I discovered Facebook that I discovered her:)
I have had a renewed communication with my daughter using Facebook. She is 26 and i am 50, so this is really wonderful! Every couple days, we chat through Facebook, and we can see significant details of each other’s lives. This is very helpful, since we live 2000 miles apart.
I’m a young faculty member at a university. I use facebook to keep up with my friends and colleagues — and to get important information out to my classes! When there is an announcement that people need to see quickly, I’ll write a facebook note and tag all of the members of the class concerned. That way, everyone who needs to know gets notified as soon as they check facebook — usually much faster than email!!
I made a fake myspace page and pretended to be someone else so I can spy on my boyfriend and to see if my boyfriend would flirt with my fake identity
On Google:
We use this service to follow our daughter’s sports in local newpapers and then can forward the info directly on to colleges that are interested in recruiting her for their sports teams
I google my name and the serp is empty; it is very humbling.
When enteing names in the Google seach engine My children and I entered my husbands name after already entering several others and receiving nothing interesting. Upon entering the name several entries were received back with the name. Upon investigating these entries we discovered my sister inlaws mom. My husband had never met this sister and it seems that she was looking for his dad who had disappeared some time ago. Upon further investigation of these hits we discovered some medical problems that ran in the family that explained fully some odd behaviors my husband had been having that could also effect our children. Without the google seach engine This info would never have been found.
I googled my mother. I was 31 years old when I found my Mom through a service I found on Google. I’ve still not met her in person, but I’ve spoken to her several times on the phone. She has two other children but even though I wasn’t raised arond her I am the one who is the most like her.
I had tried numerous times over the years to find my best friend from primary school but she was not listed on Friends Reunited and none of our classmates knew what had happened to her after university either. She was from Sri Lanka with a long surname that I could not remember in its entirety. One day I decided to Google just her first name, even though I felt that there were probably hundreds of Sri Lanks called “Pireeni”. Miraculously, I hit on her immediately. She is now a well know poet in San Francisco, we have remade contact, are in touch regularly and I hope to go to San Francisco to meet up with her in the future.
I have used Google to determine whether one of my college students’ papers is partly or wholly plagiarized. We don’t use turnitin at my campus, so I must rely on Google. It’s amazing that students don’t realize that I have access to the very same Internet that they do! They copy and paste entire pages or articles into their papers and then pretend that they wrote them on their own. And this is AFTER I have told them not to do this because they cannot outsmart me!
Years ago I used google as a sort of divination thing. I needed something to do cause I was bored so I typed in “Jessica likes to” and it gave me results for things I could possibly do. I read Jessica likes to dance, so I ended up going dancing that night.
On Wikipedia:
I was in the Navy and the computers on the ship would allow .org sites to be visited anytime, while .com sites were limited to save bandwidth. I learned ALOT in those 6 months surfing wikipedia and it is still one of my favorite sites to use.
Yes. I bet a friend that there was a permanent space station docked on the moon. I thenk changed the wikipedia article and added the false statement about a space station being docked on the moon. i then showed him the article and he believed me.
Wikipedia is how we resolve pretty much any dispute we have at the bar, using mobile internet.
One day i spent 4 hours on Wikipedia trying to see if someone/anyone in any article had mentioned my name. I checked my school, community, city, hometown and just about anything and everything associated with me. Bad luck.
I like to play a game I call ‘6 degrees of Wikipedia’. I click on the random article button, and decide some topic I want to find from this starting point. For example: Mr. T, or Trigonometry, or Star Wars. From the random article I try to click through the links towards my goal in as few clicks as possible. It’s a fun and challenging game that kills time and can have surprising results. It’s interesting (to me, anyhow) how things link together!
I have used Wikipedia to ask questions that I would normally not ask family or friends. Sometimes it is something that I probably should have known already by my age. People are usually nice on Wikipedia and I can remain anonymous.
Finally, on twitter:
I found out my sister-in-law was using Twitter. I was curious what she was writing about and although she mentioned just about every minor detail of her boring life on the site, she made no mention whatsoever of attending an evening recently with my husband and I. I figured we were low on her list, but there’s the proof.
I live-blogged via Twitter during the birth of my son! It gave family and friends moment-by-moment updates at the labor and delivery without having to deal with multiple telephone calls.
letting my friends know when a inpromptu concert happened. People were there in 15 minutes.
Several of our friends who live far away use Twitter, and my partner and I tweeted our wedding vows to one another so our friends who were unable to attend could still share our experience virtually.
Overall, there was a remarkable breadth and depth in responses; I’m unaware of any other needfinding/data-collection method where you could get such a diversity of thought for less than $3. Of course, there are trade-offs: the stories may be made up; it is much harder to follow up with each person; the stories often lack context and the demographic information needed to group and tally these needs; and finally, self-report will miss other surprising behaviors from participants that would require a trained observer to verify. But we found the results encouraging and are looking forward to following up with more.
]]>Dean breaks down the factors that affect latency and throughput on mTurk:
- HIT and sub-tasks duration
- Available workers
- Appeal of your HITs
- Reliability
In conversation with Brendan O’Connor from Dolores Labs a few months back, we tried to come up with a simple way of thinking about what affects the likelihood of participation. Building from these thoughts, I think these are the factors:
These are similar to Dean’s, which shows they echo the experience of multiple ‘requesters’ on mTurk. One addition I’d like to suggest is to think of the relationship between factors: a particularly fun task will get good response even if the payment is low, and vice-versa; task length and payment also are related (I’d like to measure this by scaling payment with time and measuring throughput); particularly difficult tasks are often ignored unless they have higher rewards; and, finally, none of this is any good if your task doesn’t get selected by anyone.
This last issue is one I hadn’t understood until I got engaged with the “Turker Nation” community, asking them why our tasks were getting fewer responses than I’d like. The answer was that experienced ‘turkers’ look for three things when choosing a Human Intelligence Task (HIT):
As independent researchers without the brand-name appeal of 1 and often without the tasks that provide for 2, the recency becomes an important factor. Distributing your assignments over a period of time by adding them to the system as interest in your task dies off has been an effective tactic for managing recency; updating your HIT (with any new detail or assignments) will push you to the top of the recency list.
]]>We were interested in two questions around storyboarding and online masses of users:
We quickly put together two variations on a simple idea —- how could you use an iPhone in a museum — and mocked up a 3-panel storyboard. The results were the following:
We asked 30 people on Mechanical Turk, for $0.10, to tell us which idea they preferred, and why. We got the 30 answers back within 19 hours (from 2pm of one today to 9am the next). Here are some of my favorite responses:
Both storyboards present viable points. While it would be prudent to first show the options in the museum, it may not be necessary to give a user map to the exhibit-many users do not need a tiny GPS system when they are wanting to be able to explore somewhere (and not have to worry about directions for once). I would therefore recommend the first storyboard since it gives the user more freedom in the museum and they can look at the information available at their own will. It’s simple and straightforward
Or another:
I like storyboard 2. Museums can be overwhelming at times. The map would allow a user to easily find and view exhibits they are interested in. Students who are visiting for a school assignment and have limited time to partake in all the exhibits would definitely find that option helpful.
Out of 30 people, 29 spoke to the ideas in the storyboard, which is the desired effect; one response focused on the art, instead:
I prefer the second storyboard. There’s a little bit more continuity within the graphics. The second and third panel both depict the iPhone and the 1st and 4th panel show the viewer relating to the environment. In the first storyboard, the box in the frame represents something entirely different each time: first the museum, then a painting, then an iPhone representing a painting. I also think that the feature described in the second storyboard is more immediately appealing—the phone helps orient you in an unfamiliar environment. The feature in the first storyboard is arguably a little more annoying—the phone will continue to solicit your attention as you’re looking at artworks.
Next, we wanted to know whether participants on Mechanical Turk could generate these storyboards; we asked them to put together a 3-to-5 panel storyboard in PowerPoint illustrating an idea they had that related to using location awareness on a mobile phone. We set up a task at $0.50 each. Response rates were far, far slower. Our first submission came within 40 minutes, but the next one took 26 hours; none others came.
Was the money not enough for this task? We changed it to $1.00 and received 6 responses in 24 hours, followed by a final two within 5 days. Still not the rapid response we had for the previous assignment, but reasonable.
Now, back to the research question: were the storyboards compelling and understandable? Some missed the point entirely, having one static composition for the 3 panel, only changing the labels below. A few really grasped the idea, though, and came up with some surprisingly effective results:
Overall, this initial exploration into crowds storyboarding taught us that the storyboards work well as a quick way to communicate a design idea with a crowd and get rapid feedback, but due to either the insufficient existing storyboard tools (in this case, PowerPoint) and the substantial time investment needed to make a reasonably complete storyboard make it less suited to gathering information from a crowd.
]]>
