(10) Cascade: Crowdsourcing Taxonomy Creation

[CHI '13]
Cascade: Crowdsourcing Taxonomy Creation

Lydia B. Chilton, University of Washington
Greg Little, oDesk Research
Darren Edge, Microsoft Research Asia
Daniel S. Weld, University of Washington
James A. Landay, University of Washington

Taxonomies are a useful and ubiquitous way of organizing information. However, creating organizational hierarchies is difficult because the process requires a global understanding of the objects to be categorized. Usually one is created by an individual or a small group of people working together for hours or even days. Unfortunately, this centralized approach does not work well for the large, quickly-changing datasets found on the web. Cascade is an automated workflow that creates a taxonomy from the collective efforts of crowd workers who spend as little as 20 seconds each. We evaluate Cascade and show that on three datasets its quality is 80-90% of that of experts. The cost of Cascade is competitive with expert information architects,
despite taking six times more human labor. Fortunately, this labor can be parallelized such that Cascade will run in as fast as five minutes instead of hours or days.

(9) Webzeitgeist: Design Mining the Web

[CHI '13]
Webzeitgeist: Design Mining the Web

Ranjitha Kumar, Stanford University
Arvind Satyanarayan, Stanford University
Cesar Torres, Stanford University
Maxine Lim, Stanford University
Salman Ahmad, Massachusetts Institute of Technology
Scott R. Klemmer, Stanford University
Jerry O. Talton, Intel Corporation

Advances in data mining and knowledge discovery have transformed the way Web sites are designed. However, while visual presentation is an intrinsic part of the Web, traditional data mining techniques ignore render-time page structures and their attributes. This paper introduces design mining for the Web: using knowledge discovery techniques to understand design demographics, automate design curation, and support data-driven design tools. This idea is manifest in webzeitgeist, a platform for large-scale design mining comprising arepository of over 100,000 Web pages and 100 million design elements. This paper describes the principles driving design mining, the implementation of the WEBZEITGEIST architecture, and the new class of data-driven design applications it enables.

(8) Catalyst: Triggering Collective Action with Thresholds

[CSCW '14]
Catalyst: Triggering Collective Action with Thresholds

Justin Cheng, Stanford HCI Group, Computer Science Department Stanford University
Michael S. Bernstein, Stanford HCI Group, Computer Science Department Stanford University

The web is a catalyst for drawing people together around shared goals, but many groups never reach critical mass. It can thus be risky to commit time or effort to a goal: participants show up only to discover that nobody else did, and organizers devote significant effort to causes that never get off the ground. Crowdfunding has lessened some of this risk by only calling in donations when an effort reaches a collective monetary goal. However, it leaves unsolved the harder problem of mobilizing effort, time and participation. We generalize the concept into activation thresholds, commitments that are conditioned on others' participation. With activation thresholds, supporters only need to show up for an event if enough other people commit as well. Catalyst is a platform that introduces activation thresholds for on-demand events. For more complex coordination needs, Catalyst also provides thresholds based on time or role (e.g., a bake sale requiring commitments for bakers, decorators, and sellers). In a multi-month field deployment, Catalyst helped users organize events including food bank volunteering, on-demand study groups, and mass participation events like a human chess game. Our results suggest that activation thresholds can indeed catalyze a large class of new collective efforts. 

(7) A Colorful Approach to Text Processing by Example

[UIST '13]
A Colorful Approach to Text Processing by Example

Kuat Yessenov   Massachusetts Institute of Technology, Cambridge, USA
Shubham Tulsiani IIT, Kanpur, India
Aditya Menon         University of California, San Diego, San Diego, USA
Robert C. Miller Massachusetts Institute of Technology, Cambridge, USA
Sumit Gulwani Microsoft, Redmond, USA
Butler Lampson Microsoft Research, Cambridge, USA
Adam Kalai         Microsoft Research, Cambridge, USA

Text processing, tedious and error-prone even for programmers, remains one of the most alluring targets of Programming by Example. An examination of real-world text processing tasks found on help forums reveals that many such tasks, beyond simple string manipulation, involve latent hierarchical structures.

We present STEPS, a programming system for processing structured and semi-structured text by example. STEPS users create and manipulate hierarchical structure by example. In a between-subject user study on fourteen computer scientists, STEPS compares favorably to traditional programming.

(6) Shepherding the Crowd Yields Better Work

[CSCW '12]
Shepherding the Crowd Yields Better Work

Steven Dow Carnegie Mellon University, Pittsburgh, Pennsylvania, USA
Anand Kulkarni University of California, Berkeley, Berkeley, California, USA
Scott Klemmer Stanford University, Stanford, California, USA
Björn Hartmann University of California, Berkeley, Berkeley, California, USA

Micro-task platforms provide massively parallel, on-demand labor. However, it can be difficult to reliably achieve high-quality work because online workers may behave irresponsibly, misunderstand the task, or lack necessary skills. This paper investigates whether timely, task-specific feedback helps crowd workers learn, persevere, and produce better results. We investigate this question through Shepherd, a feedback system for crowdsourced work. In a between-subjects study with three conditions, crowd workers wrote consumer reviews for six products they own. Participants in the None condition received no immediate feedback, consistent with most current crowdsourcing practices. Participants in the Self-assessment condition judged their own work. Participants in the External assessment condition received expert feedback. Self-assessment alone yielded better overall work than the None condition and helped workers improve over time. External assessment also yielded these benefits. Participants who received external assessment also revised their work more. We conclude by discussing interaction and infrastructure approaches for integrating real-time assessment into online work.

(5) Crowdsourcing Step-by-Step Information Extraction to Enhance Existing How-to Videos

[CHI '14]
Crowdsourcing Step-by-Step Information Extraction to Enhance Existing How-to Videos

Juho Kim Massachusetts Institute of Technology, Cambridge, MA, USA
Phu Tran Nguyen Massachusetts Institute of Technology, Cambridge, MA, USA
Sarah Weir Massachusetts Institute of Technology, Cambridge, MA, USA
Philip J. Guo University of Rochester & Massachusetts Institute of Technology, Rochester, NY, USA
Robert C. Miller Massachusetts Institute of Technology, Cambridge, MA, USA
Krzysztof Z. Gajos Harvard University, Cambridge, MA, USA

Millions of learners today use how-to videos to master new skills in a variety of domains. But browsing such videos is often tedious and inefficient because video player interfaces are not optimized for the unique step-by-step structure of such videos. This research aims to improve the learning experience of watching existing how-to videos with step-by-step annotations.
We first performed a formative study to verify that annotations are actually useful to learners. For this study, we created ToolScape, an interactive video player that displays step descriptions and intermediate result thumbnails in the video timeline. Learners in our study performed better and gained more self-efficacy using ToolScape versus a traditional video player.
To add the necessary step annotations to existing how-to videos at scale, we introduce a novel crowdsourcing work- flow. It extracts step-by-step structure from an existing video, including step times, descriptions, and before and after images. We introduce the Find-Verify-Expand design pattern for temporal and visual annotation, which applies clustering, text processing, and visual analysis algorithms to merge crowd output. The workflow does not rely on domain-specific customization, works on top of existing videos, and recruits untrained crowd workers. We evaluated the workflow with Mechanical Turk, using 75 cooking, makeup, and Photoshop videos on YouTube. Results show that our workflow can extract steps with a quality comparable to that of trained annotators across all domains with 77% precision and 81% recall.

(4) Cobi: A Community-Informed Conference Scheduling Tool

[UIST '13]
Cobi: A Community-Informed Conference Scheduling Tool

Juho Kim                             Massachusetts Institute of Technology, Cambridge, MA, USA
Haoqi Zhang                     Northwestern University, Evanston, IL, USA
Paul André                     Carnegie Mellon University, Pittsburgh, PA, USA
Lydia B. Chilton             University of Washington, Seattle, WA, USA
Wendy Mackay             INRIA, Orsay, France
Michel Beaudouin-Lafon   Université Paris-Sud, Orsay, France
Robert C. Miller             Massachusetts Institute of Technology, Cambridge, MA, USA
Steven P. Dow                     Carnegie Mellon University, Pittsburgh, PA, USA

Effectively planning a large multi-track conference requires an understanding of the preferences and constraints of organizers, authors, and attendees. Traditionally, the onus of scheduling the program falls on a few dedicated organizers. Resolving conflicts becomes difficult due to the size and complexity of the schedule and the lack of insight into community members' needs and desires. Cobi presents an alternative approach to conference scheduling that engages the entire community in the planning process. Cobi comprises (a) communitysourcing applications that collect preferences, constraints, and affinity data from community members, and (b) a visual scheduling interface that combines communitysourced data and constraint-solving to enable organizers to make informed improvements to the schedule. This paper describes Cobi's scheduling tool and reports on a live deployment for planning CHI 2013, where organizers considered input from 645 authors and resolved 168 scheduling conflicts. Results show the value of integrating community input with an intelligent user interface to solve cmplex planning tasks.

(3) Community clustering: Leveraging an academic crowd to form coherent conference sessions

[HCOMP '13]
Community clustering: Leveraging an academic crowd to form coherent conference sessions

Paul André, Haoqi Zhang, Juho Kim, Lydia Chilton, Steven P. Dow, Robert C. Miller

Creating sessions of related papers for a large conference is a complex and time-consuming task. Traditionally, a few conference organizers group papers into sessions manually. Organizers often fail to capture the affinities between papers beyond created sessions, making incoherent sessions difficult to fix and alternative groupings hard to discover. This paper proposes committeesourcing and authorsourcing approaches to session creation (a specific instance of clustering and constraint satisfaction) that tap into the expertise and interest of committee members and authors for identifying paper affinities. During the planning of ACM CHI’13, a large conference on human-computer interaction, we recruited committee members to group papers using two online distributed clustering methods. To refine these paper affinities—and to evaluate the committeesourcing methods against existing manual and automated approaches—we recruited authors to identify papers that fit well in a session with their own. Results show that authors found papers grouped by the distributed clustering methods to be as relevant as, or more relevant than, papers suggested through the existing in-person meeting. Results also demonstrate that communitysourced results capture affinities beyond sessions and provide flexibility during scheduling.

(2) Frenzy: Collaborative Data Organization for Creating Conference Sessions

[CHI '14]
Frenzy: Collaborative Data Organization for Creating Conference Sessions

Lydia Chilton, University of Washington Seattle, WA
Juho Kim, MIT CSAIL Cambridge, MA
Paul André, HCI Institute, CMU Pittsburgh, PA
Felicia Cordeiro, University of Washington Seattle, WA
James Landay, University of Washington Seattle, WA
Daniel S. Weld,  University of Washington Seattle, WA
Steven P. Dow, HCI Institute, CMU Pittsburgh, PA
Robert C. Miller, MIT CSAIL Cambridge, MA
Haoqi Zhang, MIT CSAIL Cambridge, MA
                     Northwestern University Evanston, IL

Organizing conference sessions around themes improves the experience for attendees. However, the session creation process can be difficult and time-consuming due to the amount of expertise and effort required to consider alternative paper groupings. We present a collaborative web application called Frenzy to draw on the efforts and knowledge of an entire program committee. Frenzy comprises (a) inter-faces to support large numbers of experts working collectively to create sessions, and (b) a two-stage process that decomposes the session-creation problem into meta-data elicitation and global constraint satisfaction. Meta-data elicitation involves a large group of experts working simultaneously, while global constraint satisfaction involves a smaller group that uses the meta-data to form sessions. We evaluated Frenzy with 48 people during a deployment at the CSCW 2014 program committee meeting. The session making process was much faster than the traditional process, taking 88 minutes instead of a full day. We found that meta-data elicitation was useful for session creation. Moreover, the sessions created by Frenzy were the basis of the CSCW 2014 schedule.

(1) Pair Research: Matching People for Collaboration, Learning, and Productivity

[CSCW '14]
Pair Research: Matching People for Collaboration, Learning, and Productivity

Robert C. Miller Massachusetts Institute of Technology, Cambridge, MA, USA
Haoqi Zhang Northwestern University, Evanston, IL, USA
Eric Gilbert Georgia Institute of Technology, Atlanta, GA, USA
Elizabeth Gerber Northwestern University, Evanston, IL, USA

To increase productivity, informal learning, and collabora- tions within and across research groups, we have been ex- perimenting with a new kind of interaction that we call pair research, in which members are paired up weekly to work together on each other’s projects. In this paper, we present a system for making pairings and present results from two deployments. Results show that members used pair research in a wide variety of ways including pair programming, user testing, brainstorming, and data collection and analysis. Pair research helped members get things done and share their ex- pertise with others.

Stable Marriage and Roommate Problems with Individual-Based Stability

[AAMAS '13]
Stable Marriage and Roommate Problems with Individual-Based Stability

Haris Aziz NICTA and University of New South Wales, Sydney, Australia

Research regarding the stable marriage and roommate problem has a long and distinguished history in mathematics, computer science and economics. Stability in this context is predominantly core stability or one of its variants in which each deviation is by a group of players. We consider stability concepts such as Nash stability and individual stability in which the deviation is by a single player. Such stability concepts are suitable especially when trust for the other party is limited, complex coordination is not feasible, or when only unmatched agents can be approached. Furthermore, weaker stability notions such as individual stability may in principle circumvent the negative existence and computational complexity results in matching theory. We characterize the computational complexity of checking the existence and computing individual-based stable matchings for the marriage and roommate settings. Some of our key computational results also carry over to different classes of hedonic games and network formation games for which individual-based stability has already been of much interest.


full paper:
http://dl.acm.org/citation.cfm?id=2484968

Eliciting High Quality Feedback from Crowdsourced Tree Networks Using Continuous Scoring Rules

[AAMAS '13]
Eliciting High Quality Feedback from Crowdsourced Tree Networks Using Continuous Scoring Rules

Ratul Ray Indian Institute of Science, Bangalore, India
Rohith D. Vallam Indian Institute of Science, Bangalore, India
Y. Narahari Indian Institute of Science, Bangalore, India

Eliciting accurate information on any object (perhaps a new product or service or person) using the wisdom of a crowd of individuals utilizing web-based platforms such as social networks is an important and interesting problem. Peer-prediction method is one of the known efforts in this direction but is limited to a single level of participating nodes. We non-trivially generalize the peer-prediction mechanism to the setting of a tree network of participating nodes that would get formed when the query about the object originates at a root node and propagates to nodes in a social network through forwarding. The feedback provided by the participating nodes must be aggregated hierarchically to generate a high quality answer at the root level. In the proposed tree-based peer-prediction mechanism, we use proper scoring rules for continuous distributions and prove that honest reporting is a Nash Equilibrium when prior probabilities are common knowledge in the tree and the observations made by the sibling nodes are stochastically relevant. To compute payments, we explore the logarithmic, quadratic, and spherical scoring rules using techniques from complex analysis. Through detailed simulations, we obtain several insights including the relationship between the budget of the mechanism designer and the quality of answer generated at the root node.

Online Implicit Agent Modeling

[AAMAS '13]
Online Implicit Agent Modelling

Nolan Bard University of Alberta, Edmonton, AB, Canada
Michael Johanson University of Alberta, Edmonton, AB, Canada
Neil Burch University of Alberta, Edmonton, AB, Canada
Michael Bowling University of Alberta, Edmonton, AB, Canada

The traditional view of agent modelling is to infer the explicit parameters of another agent's strategy (i.e., their probability of taking each action in each situation). Unfortunately, in complex domains with high dimensional strategy spaces, modelling every parameter often requires a prohibitive number of observations. Furthermore, given a model of such a strategy, computing a response strategy that is robust to modelling error may be impractical to compute online. Instead, we propose an implicit modelling framework where agents aim to estimate the utility of a fixed portfolio of pre-computed strategies. Using the domain of heads-up limit Texas hold'em poker, this work describes an end-to-end approach for building an implicit modelling agent. We compute robust response strategies, show how to select strategies for the portfolio, and apply existing variance reduction and online learning techniques to dynamically adapt the agent's strategy to its opponent. We validate the approach by showing that our implicit modelling agent would have won the heads-up limit opponent exploitation event in the 2011 Annual Computer Poker Competition.

A Parameterized Family of Equilibrium Profiles for Three-Player Kuhn Poker

[AAMAS '13]
A Parameterized Family of Equilibrium Profiles for Three-Player Kuhn Poker

Duane Szafron, University of Alberta, Edmonton, AB, Canada
Richard Gibson, University of Alberta, Edmonton, AB, Canada
Nathan Sturtevant, University of Denver, Denver, CO, USA

This paper presents a parameterized family of equilibrium strategy profiles for three-player Kuhn poker. This family illustrates an important feature of three-player equilibrium profiles that is not present in two-player equilibrium profiles - the ability of one player to transfer utility to a second player at the expense of the third player, while playing a strategy in the profile family. This family of strategy profiles was derived analytically and the proof that the members of this family are equilibrium profiles is an analytic one. In addition, the problem of selecting a robust strategy from an equilibrium profile is discussed.


full paper:
http://web.cs.du.edu/~sturtevant/papers/3pkuhn.pdf