How Many People Found the Same Event? Effort and Collaboration in Teams

Five messages about a disaster need not describe five separate events. Several people may have reported the same damaged bridge, while another message gives it the wrong location. A team turning these messages into a map must do more than produce additional records. It must connect fragments, reconcile conflicting descriptions, and preserve a usable account. Increasing the number of people working together may therefore have a different effect from collecting the output of people who worked independently.

A 2016 study by Andrew Mao, Winter Mason, Siddharth Suri, and Duncan J. Watts examined that difference in an online crisis-mapping experiment. Two tendencies appeared together: individual effort, as measured in the study, was lower in larger teams, while collaboration increased. Some large teams produced a better combined result than a comparison constructed by merging independent workers' outputs. This is not a recommendation to enlarge every team. Understanding the finding requires examining the task and what counted as success. [1]

A fixed collection of messages, no assigned organization

The experiment took place in five one-hour online sessions in 2014. A total of 258 unique participants were randomly assigned to forty-seven teams with nominal sizes ranging from one to thirty-two. They were recruited through Mechanical Turk. Each team received the same 1,567 messages drawn from material concerning Typhoon Pablo in the Philippines in 2012. The experiment was a retrospective task using previously collected information, not an active disaster response in 2014. [1]

Participants had to select relevant messages, create event records, classify damage, establish locations, and check or correct existing reports. The CrowdMapper interface included a shared message stream, editable event table, map, chat rooms, and documents. Participants received interface training but were not assigned specialized roles or instructed how to organize themselves. The experiment consequently examined collaboration emerging within that environment, rather than testing a fully specified management procedure. [1]

Payment combined individual participation and collective performance. A performance-based team hourly wage was scaled by each person's active time. Participants knew that activity would be monitored, so the arrangement supplied incentives to contribute as well as to produce a good team outcome. Finding lower individual effort under these conditions complicates the idea that reduced effort occurs only when contributions are invisible. It does not, however, directly reveal anyone's internal motivation. [1]

Counting events rather than messages

To evaluate the output, the researchers combined events identified by all teams, removed duplicates, and checked which records could be verified. Their reference map contained forty-nine distinct events associated with 245 messages. Those different counts matter. Many messages could refer to one incident, so counting messages alone could reward repeated reports of the same damage. The unit being evaluated was the event. [1]

The researchers did not describe this reference as complete ground truth. Real events missed by every experimental team could also be absent from the reference. Performance was therefore measured against the verified set constructed for the experiment, not against a demonstrably exhaustive inventory of the disaster. The boundary identifies what a good score means and what it leaves unknown. [1]

Precision measured the fraction of a team's reports that matched the reference. Recall measured how much of the reference set the team identified. The F1 score combined precision and recall using their harmonic mean. Reporting many events could improve coverage while introducing mistakes that lowered precision. Restricting output to a few unambiguous events could avoid errors while leaving much of the reference undiscovered. Neither quantity alone captured the intended balance. [1]

Active time was not the same as measured effort

A nominal team size did not mean every member worked for the entire hour. Some participants arrived late or became inactive. The researchers therefore used active person-hours for comparisons. Two teams both labeled as having thirty-two members could have different amounts of labor available. A list of assigned names and a record of active working time were not equivalent measures of size. [1]

Effort was assessed with another measure. The researchers estimated average completion times for different action types, then weighted each worker's action counts by those average times. This was a behavioral index for comparing different activities, not a direct measurement of mental fatigue, motivation, or energy expenditure. Inferring a person's character from that index would go beyond what the experiment established. [1]

By this measure, average individual effort in the largest groups was approximately thirty percent lower than for independent workers. Allocation also shifted: less effort went into classification, while relatively more went into filtering messages, which could involve simpler actions. The interesting extension of social loafing here was not limited to doing nothing. Someone could remain active while moving toward less demanding subtasks. The pattern does not demonstrate that every participant deliberately intended to exploit coworkers. [1]

Building a comparison from independent outputs

Collaboration was not measured merely by counting chat messages. The study used an effort-weighted measure of how many people contributed to an event record. Shared contributions increased in larger teams. This was not inevitable just because more people were present: a large team could have worked on entirely separate records, leaving little collaboration on individual events. The measure addressed shared work on the output itself. [1]

To examine its value, the researchers constructed synthetic teams from the outputs of eighteen people who had worked independently. These were not new experimental groups. They were combinations of existing independent work, evaluated after merging the associated reports. Precision, recall, and F1 were recalculated for those combinations. Small synthetic teams generally compared favorably, while large real teams achieved higher F1 in the relevant large-team comparison. The independent pool was limited to eighteen, so the procedure did not create a separately tested thirty-two-person independent comparison at every size. [1]

The synthetic teams maintained high recall, but their precision declined as adding independent outputs also added incorrect report variants. It would therefore be misleading to describe real teams as superior because they found more reference events on every measure. Their stronger combined performance involved the precision–recall balance. Reconciling contradictory reports and correcting shared records offered plausible explanations. Those mechanisms were not each independently randomized and established as separate causal effects. [1]

This distinction makes the comparison useful without making it decisive about every aspect of collaboration. Merging independent outputs can preserve correct information from many workers, but it can preserve their errors too. A shared workspace gives people an opportunity to challenge and revise conflicting versions before the final record is evaluated. The experiment showed that the collective product could benefit despite a reduction in the measured effort of its individual contributors. It did not show that all coordination is beneficial or costless.

No universal team size emerges

The study did not identify an ideal headcount for every organization. Participants were inexperienced, the task lasted one hour, and organizational guidance was deliberately limited. The negative quadratic coefficient used to represent diminishing returns in the performance model was not significant. A fitted curve should not be converted into a precisely established optimum or a law about unlimited team growth. [1]

The more concrete conclusion concerns how collective output is produced. In a task containing repeated and conflicting information, a team can generate a more useful shared record even while individual action-based effort declines. Explaining that outcome calls for examining task allocation, error correction, and the comparison being made, rather than attaching moral labels to people. The final map represents more than a count of how many workers found the same event independently.

Sources

[1] Mao, A., Mason, W., Suri, S., and Watts, D. J. (2016). An Experimental Study of Team Size and Performance on a Complex Task. PLOS ONE, 11(4), e0153048. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0153048 . DOI: 10.1371/journal.pone.0153048. Original research under a Creative Commons Attribution license; this article is an independently written explanation and assessment.