When Open Access is the norm, how do scientists work together online? By Jon Udell
The Web was invented to enable scientists to collaborate.
In 2000 the Los Alamos National Laboratory commissioned me to write a progress report on web-based collaboration between scientists, Internet Groupware for Scientific Collaboration.
Blogs, social media, and Open Access publishing of scientific literature, data, and software were trends then visible on the horizon but not yet central to the experiences of most working scientists.
For this PLOS-commissioned survey, which I view as a bookend to my 2000 report, I’ve investigated how researchers are collaborating to transform scientific communication in an Open Access environment. I’ve found that adoption remains incomplete, but progress has been dramatic and encouraging. New modes of collaboration, unforeseen in 2000, are emerging in a variety of scientific subcultures.
From interviews I’ve conducted with researchers and software developers who are modeling aspects of modern online collaboration, I’ve highlighted the most useful and reproducible practices.
Necessity breeds invention
For Bryan Jones, a vision researcher at the University of Utah, a key driver for online collaboration has been the steep decline in funding for basic research. “Researchers used to be able to get together a couple of times a year at Woods Hole,” Jones says, “share data, and talk about science.” Now, more than ever, the serendipitous conversations enabled by such gatherings must find other ways to happen. He’s been surprised by Twitter’s efficacy:
“My first reaction was, 140 characters, what’s the point? But I had completely underestimated the tool. It’s like an amazing radio that you can tune to just what want to pay attention to. Really, it’s been life-changing, and has led to scientific collaborations that I wouldn’t have expected otherwise. But you have to learn how to use it right.”
What does “use it right” mean? Jones’ Twitter feed models a number of best practices. He delivers a steady (but not overwhelming) stream of links to, and thoughtful reflections on, the kinds of science he practices. He follows and interacts with researchers, in fields near or distant, who do the same. And he thinks carefully about how to strike a balance between his personal interest in photography and his scientific interests.
Jones has also, for many years, run Webvision, an evolving textbook on the organization of the retina and the visual system. When he inherited the site from its founder in 2000, he added a blog component, and he uses the blog in a highly strategic way to enhance the Google rank of the people and publications associated with Webvision.
Because the funding crisis has stretched resources so thin, he says, “the people reviewing your grant proposal may not be subject matter experts in your area, so they go to Google.” When one of Jones’ colleagues began doing pioneering work on TRP (transient receptor potential) channels, he was an unknown newcomer to that field. “We did a single blog post,” says Jones, “and almost immediately a Google search for ‘TRP channels’ and ‘retina’ found it as the first result.” It has since slipped to third as more recent published papers have come into the field.
Scientists who are not engaged with social media question whether such benefits are replicable. There are no guarantees. It’s a long game. And successful outcomes depend upon collective buy-in. If your peers aren’t active in social media, that limits what you can achieve there. It’s true that being a first mover in a given field can confer a huge advantage, because attention follows a power law distribution that favors early adopters. But in small and well-defined communities of interest, there’s a steady state in which online interactions usefully augment those that occur offline. The effect is cumulative, so participation in social media is best regarded as a long-term investment. If you start early, behave judiciously, and contribute steadily over time, your investment of time and effort will likely be repaid, and perhaps many times over.
From Polymath to MathOverflow
In 2009 Michael Nielsen, author of Reinventing Discovery, wrote a blog post entitled Doing science online in which he dissected a post on Terence Tao’s blog, writing:
“The post is filled to the brim with clever perspective, insightful observations, ideas, and so on. It’s like having a chat with a top-notch mathematician, who has thought deeply about the Navier-Stokes problem, and who is willingly sharing their best thinking with you…Following the post, there are 89 comments. Many of the comments are from well-known professional mathematicians, people like Greg Kuperberg, Nets Katz, and Gil Kalai. They bat the ideas in Tao’s post backwards and forwards, throwing in new insights and ideas of their own. It spawned posts on other mathematical blogs, where the conversation continued.”
The technical enabler for this mathematical discourse was the LaTeX support that Mike Adams had added to WordPress.com in 2007. In January 2009, things took off in a big way. Tim Gowers asked: Is massively collaborative mathematics possible? Soon thereafter the Polymath experiment was launched. And in March, Nielsen described the first successful outcome:
“Over the past seven weeks, mathematician Tim Gowers has been running a remarkable experiment in how mathematics is done, a project he dubbed the Polymath1 project. Using principles similar to those employed in open source programming projects, he used blogs and a wiki to organize an open mathematical collaboration attempting to find a new proof of an important mathematical theorem known as the density Hales-Jewett (DHJ) theorem.”
Success had been uncertain. But not only did proof for this hard problem emerge, it exceeded expectations, proving the full theorem rather than a special case that was the original goal. For Nielsen this was an example of how online collaboration can “restructure expert attention,” focusing the best minds in the world on a challenge that requires their collective attention.
Polymath is ongoing. The first effort, in 2009, was called Polymath1. The most recent, Polymath9, happened in January 2014. But it’s not the only bold experiment in restructuring the attention of expert mathematicians. In September of 2009, the creators of Stack Overflow, a question-and-answer site for programmers, made the site’s engine available to other disciplines. The following month a trio of Berkeley graduate students and postdocs — Anton Geraschenko, David Zureick-Brown, and Scott Morrison — launched MathOverflow, a question-and-answer site for professional mathematicians.
Setting expectations for productive online discourse
Prior to launch they seeded it with examples of the kind of discourse they wanted the site to support. When they turned it on, success was almost immediate. MathOverflow serves a very well-defined community. Those inclined to participate found out about it very quickly, via blogs and word-of-mouth, and established a level of engagement that continues to this day.
Morrison attributes that overnight success to several factors. First, unlike Polymath which tackles open-ended research problems, MathOverflow focuses on “questions that admit definitive answers.” The founders initially thought that Polymath and MathOverflow would prove complementary. Definitive questions that arose in Polymath discussion might be answered on MathOverflow. Things didn’t turn out that way.
“Plenty of the answers on MathOverflow qualify as original math,” Morrison says, but the two modes of collaboration proceed in parallel, meeting different needs, with no apparent need to cross-fertilize. Another key success factor was the founders’ strong leadership in the early going. They set high expectations for the quality and tone of discourse. “We made it clear that you wouldn’t just ask an unmotivated question,” Morrison says. “You’d put a lot of thought into your question, you’d explain why you were asking, and you’d specify what would constitute a good answer.” Discourse was expected to adhere to the norms of professional math. “We had to say, look, you wouldn’t behave that way in a graduate seminar, and you won’t behave that way here.”
Today the site mostly runs itself. There are still moderators who “take out the trash” when needed, but the need rarely arises. That’s partly a reflection the StackExchange’s “wonderful support for community moderation,” Morrison says. But it’s also a validation of how the founders began with a clear statement of values and expectations.
One of those expectations was that the game-like aspects of the site should not be taken seriously. As in many other systems, users rate one another on the site, and those ratings produce numerical reputation scores. “It’s unfortunate that the MathOverflow software calls its number reputation,” says Morrison, “we push back pretty hard on that.”
“Nobody should put a MathOverflow number on a CV, “he says, “or base any professional evaluation on such a number.”
Restructuring expert attention
MathOverflow is clearly restructuring expert attention in the way Michael Nielsen envisioned. Math is hugely diverse, Morrison points out, and divided into many narrow sub-disciplines. Researchers often run into problem that require help from others.
“Traditionally you hoped to be in a department with other mathematicians you could walk down the hall and chat with,” says Morrison. “MathOverflow gives us a way to not need to find that person down the hall, but rather to quickly find that person anywhere in the world.”
So far this approach has worked best, he says, in math and closely related fields — theoretical physics and computer science — characterized by many small clusters of expertise. That’s an important reminder that there isn’t a one-size-fits-all architecture of participation. Effective online collaboration, in science or elsewhere, may require platforms that align with discipline-specific cultural norms and social structures.
Grant Miller, a former exoplanetary scientist, is now community manager for Zooniverse, the family of citizen-science projects that grew out of Galaxy Zoo, a crowdsourced census of galaxies. As liaison between scientists and volunteers he has his finger on the pulse of a powerful trend in online scientific collaboration. The raw ingredients are an embarrassment of riches. There’s more data than scientists can analyze, and there are a lot of Internet-connected amateurs who might want to help. Stirring those ingredients together in the right way can lead to notable discoveries like the discovery of Green Pea galaxies and of the ionization cloud known as Hanny’s Voorwerp. But what’s the right mix?
Web tools that distribute images to volunteers, and support annotation and decision-tree analysis, are necessary but not sufficient. Because analysis does not (yet) yield to algorithmic automation, Miller says, success hinges on effective collaboration between scientists and volunteers. For Galaxy Zoo and later Zooniverse projects, discussion forums have been the key enabler. It’s another example of Michael Nielsen’s notion of restructuring expert attention.
Hanny’s Voorwerp is so called because Hanny van Arkle, a Dutch schoolteacher, saw an anomaly in an image and asked: “What is that object?” (Voorwerp is Dutch for object.) Likewise in the case of Green Pea galaxies. “In the forum, people were saying ‘These things aren’t like anything you’ve trained us to recognize’,” Miller says, “and they brought them to the attention of the scientists.”
Skip the games, focus on the science
Other recipes for collaboration have been less successful. The Old Weather project invites volunteers to transcribe handwritten marine logs from the 19th century. The historical data about temperature, barometric pressure, and weather conditions have (among other uses) supported a number of reconstructions of the climate during that era. The site uses a game mechanism to motivate contributors who begin as cadets and can aspire to become captains.
But the gamification strategy hasn’t worked well, Miller says, and won’t be part of a forthcoming rebuild of the site.”We realized gamification wasn’t the right approach,” Miller says. “The real motivation is, as it should be, engagement with science and with scientists.” That engagement happens in the forum, and also on a monthly status call involving the scientific team and a few dozen of the most committed volunteers.
How do you assure the quality of citizen-contributed data? Consensus is the first line of defense. When contributors are an abundant resource, you can repeat a given observation arbitrarily many times and merge the results. To check the validity of that approach, Galaxy Zoo projects benchmark themselves against gold standard data sets. Snapshot Serengeti, for example, uses “camera traps” to capture images of animals in the Serengeti that volunteers then identify. When experts classified a 4000-image sample, their results correlated very highly with those of the volunteers.
How broadly useful is crowdsourced consensus? For Bryan Jones, it wasn’t an option. His lab, the Marc Lab for Connectomics, is mapping the retinal connectome. The data are massive. A 250-nanometer-wide plug of a retina yields 18-60 terabytes of image data. Crowdsourcing does play a role in connectomics.
The Seung Lab EyeWire project uses that approach. For their purposes, though, the Marc lab found the error rate unacceptable, so instead uses a team of trained (and paid) student annotators. “It’s taken a couple of years,” Jones says, “but we’ve now annotated 60% of that 18 terabyte dataset.”
A platform for experimentation
Given that scientific data is not only big but also wildly diverse, we shouldn’t expect that there’s one right way to crowdsource its analysis. Such methods should, themselves, be subject to experimentation. That’s why the latest development at Zooniverse is so exciting. In the same way that StackExchange was made available as a platform that enabled MathOverflow to emerge, Zooniverse will soon release a platform that enables users to create their own citizen-science projects.
“We get about 50 proposals a year to build sites for new projects,” Miller says, “and we can only do about five or ten of those ourselves.” Capabilities that the Zooniverse platform will generalize include marking regions in images, performing decision-tree workflow, and running consensus algorithms for both of those interaction patterns. There’s no way to know, a priori, whether a data set in any given discipline, for any given purpose, will yield to crowdsourced analysis, and it’s expensive to find out. If the Zooniverse platform can slash the cost of experimenting with these methods, it could prove to be a huge accelerator.
Among the benefits of expanding the breadth of citizen science: more opportunities for young people to explore fields they might wish to enter. When Hannah Hutchins was 12, in 2007, she began classifying galaxies on Galaxy Zoo. Today she studies astrophysics at the Open University. The opportunity to meet and collaborate with working scientists is a form of apprenticeship that’s valuable in both positive and negative ways. For Hannah Hutchins it was the beginning of a scientific career. Another young person might not find astrophysics to her liking, but by figuring that out early, might instead redirect to another field before investing time, effort, and money following a wrong path.
Open source software, open data, and reproducibility
Modern science generates vast and growing amounts of data and is increasingly computational. So it was inevitable that the collaborative culture of software development would influence scientific data management and computation. That culture has evolved most notably on GitHub, the home of millions of open source software projects. At the core of GitHub is Git, the distributed version control system created to manage the development of the Linux kernel. GitHub wraps a friendly web interface around Git and, as importantly, codifies best practices for collaborative development of sets of versioned digital artifacts that typically include source code but may also include data, images, or documentation.
Reproducibility isn’t an optional, nice-to-have feature of software development. Software is fragile and ever-changing. Software packages must be reliably reusable. Programmers have no choice but to closely track every revision, discuss changes in a disciplined way, hold one another accountable for those changes, maintain a transparent record of project history, and automate everything that can be automated.
Science can learn from these best practices, and that transfer of knowledge is underway. In The What, Why, and How of Born-Open Data, for example, the cognitive psychologist Jeff Rouder opens with this confession:
“I was committed to open data. I made this commitment boldly in my National Science Foundation data-management plans. My data were supposed to be archived at my institution’s curated, open repository. Yet, sadly, this did not often happen.”
Lessons from software development
His solution will be familiar to every programmer, if not yet to every scientist: a script that automatically uploads data files to GitHub on a nightly basis. It’s a simple strategy that succeeds by establishing a new default condition. When a data set is born open, no action is required to make it so and no error or omission will prevent that.
To implement such a strategy a scientist needs to know first that it’s possible and then how it’s doable. The Software Carpentry project has been meeting both needs since 1998. This organization, now operating as the Software Carpentry Foundation, offers workshops on the basic tools and methods of software development. The workshops are designed for scientists who find themselves writing or customizing software but lack the necessary training. They’re supported by online lessons hosted on GitHub and developed collaboratively by hundreds of contributors. One measure of the demand for the workshops: a paper on Best Practices for Scientific Computing, written by a team of Software Carpentry instructors, was PLOS Biology’s most-read article in 2014.
Last year Software Carpentry spun out a sister project, Data Carpentry, which focuses specifically on methods and tools for managing open scientific data.
And while its online lessons are also being managed in a GitHub repository, project leader Tracy Teal, a microbial ecologist and bioinformatician, cautions that GitHub is less than ideal for the purpose. “The capacity to share, track versions and do diffs makes it very appealing for collaborative document and lesson development,” she says, “but it currently has too many technical barriers for people without programming experience. The activation energy to use it is just too high.”
Maybe GitHub will continue to evolve in ways that lower those barriers. Or maybe open source alternatives, like GitLab, will enable that evolution. Although it hosts millions of open source projects, GitHub itself isn’t an open source project. GitLab, though, operates under a dual license and offers an open source community edition. One way or another, the model of collaboration that GitHub has popularized is vital to the progress of science, and seems likely to prevail.
Using the right tools
GitHub’s high activation threshold isn’t the only obstacle in Data Carpentry’s way. The best tools for scientific data management don’t lie on the path of least resistance either. The familiar and easy choice, Excel, is poorly suited to open, collaborative, and reproducible science.
“We don’t shame people for using Excel,” says Karthik Ram, a fellow at the Berkeley Institute for Data Science, “and we help people make best use of it, but we also try to bring them platforms with better support for automation.” One such platform is R, an open source tool for statistical computing and graphics. Its advantages include a more modern and powerful scripting language than Excel’s, and a rich (and rapidly growing) set of libraries — collected at ROpenSci — for accessing and processing a wide variety of scientific data sets.
Another emerging platform is the IPython Notebook which combines interactive computation, data, equations, plots, and rich media on just the sort of universal canvas envisioned in Internet Groupware for Scientific Collaboration. Benjamin Laken is a geophysicist who recently published a refutation of a paper entitled Influence of cosmic-ray variability on the monsoon rainfall and temperature. His published response was accompanied by an IPython Notebook that can be downloaded from GitHub and used interactively or viewed directly. Making the notebook available in these ways, Laken wrote in his paper, enables others to “easily check, repeat, and alter the analysis.”
Echoing WordPress.com’s support for LaTeX, blogs can also display IPython Notebooks. Jake Vanderplas, a research director at the University of Washington’s eScience Institute, shows how in blog posts like Kernel Density Estimation in Python.
This is a great way to think out loud, to tell evolving stories with code, data, and charts, and to converse with peers.
Appealing to enlightened self-interest
Willingness to explore these new modes of scientific discourse, and to do so in reproducible ways, remains unevenly distributed. For Karthik Ram, a quantitative ecologist, it’s been an uphill slog. Recently he was invited to present to a federal institute in Spain. It turned out that only one senior researcher and his team was interested in what Ram had to say about open collaboration.
“Everyone else looked at me like I was an alien trying to sell them snake oil,” he says. “So I don’t try to sell open collaboration and reproducibility for the greater good of science.” Instead he appeals to enlightened self-interest: “These approaches enable you to do more science, do it better, and do it faster.”
Appeals to the greater good are, however, resonating with some psychologists. Jeff Rouder’s self-critical manifesto about born-open data is one example. The Center for Open Science’s Reproducibility Project: Psychology is another. Project coordinators Johanna Cohoon and Mallory Kidwell are supporting 50 teams working to reproduce published results. Their protocols and data are stored in the Open Science Framework(OSF), a simplified GitHub-like repository that tracks versions of documents and data. Replication studies hosted in the OSF comprise the entire May 2014 issue of Social Psychology.
A model from social science
As a newly-hired professor at Virginia Commonwealth University, sociologist Tressie McMillan Cottom uses her blog to develop ideas that originate, typically, in social media interactions.
“I monitor social media for trends relevant to my research interests,” she says, referring to her focus on phenomena that are observable in social media, such as inequality and stratification in higher education. Thus it follows that she “relies heavily on social network ties as a filter.”
Cottom can use tools like Facebook and Twitter to discover and connect with research subjects, an advantage that is, admittedly, particular to her field. But the kinds of interactions that flow from social media through her blog, and then into wider scientific and popular online venues, are reproducible by scientists working in other disciplines – with a few key factors determining their viability. These include perceived risk of competition, rank, personality, and level of proficiency with blogging and social media.
Rank, in Cotton’s experience, tends to play out in a bimodal way. Junior researchers, whose personal lives are often deeply intertwined with social media, are less likely to project their professional identities into that sphere than senior researchers. Why? She speculates that the two groups evaluate risks and benefits differently. Junior researchers who are generationally more comfortable with personal interaction in social media worry about damage to nascent professional reputations. For senior researchers it’s the reverse. Their academic brands are solid, they want to promote them, and they don’t see social media as a professional risk. But they are generationally less comfortable using social media for personal interaction. Appearances, however, can be deceiving.
“The junior researchers actually are often there [on social media], it’s just that you don’t always know who they are.” And they’re not the only ones who may downplay their professional identities. Cottom recalls a pivotal moment in her career. “I’d been talking for months to somebody who I didn’t realize worked for Inside Higher Ed,” she recalls. “I wrote something, she cross-posted it, and it completely changed my audience.”
Collapsing the prestige hierarchy
When professional identity is muted in this way, another unexpected outcome can occur. “Social media collapses the prestige hierarchy,” Cottom says. “I’ve been surprised by some of the very senior people who have put themselves into my network, people who otherwise would never be talking with me, not in a million years.”
In Cottom’s experience, Social media can also usefully cross disciplines.
“I can’t think of any part of my professional life that hasn’t been touched by that dynamic,” she says. “When I was looking at inequality, my approach was very disciplinary, and still is. But my project deepened by talking to, first of all, economists, but also to a philosopher whose way of thinking about the public good led me to a different approach in my research methods.”
In 2000, the year that the first installment of this report was published, the founders of PLOS circulated an open letter that began like so:
“We support the establishment of an online public library that would provide the full contents of the published record of research and scholarly discourse in medicine and the life sciences in a freely accessible, fully searchable, interlinked form.”
Today, in 2015, open access to scientific literature, while not yet universal, is firmly established. Such access is a necessary but not sufficient basis for the open scientific collaboration that the Internet can and must enable. Progress has been encouraging on a number of fronts, but cultural and technical barriers remain. Here’s hoping that the 2030 installment of this report will describe a fuller realization of the Internet’s potential to accelerate and sustain science.
Jon Udell is an author, information architect, software developer, and new media innovator. His 1999 book, Practical Internet Groupware, helped lay the foundation for what we now call social software. Formerly a software developer at Lotus and Microsoft, and BYTE Magazine’s executive editor, Udell is now product manager for Hypothesis, maker of open web annotation software, and a contributing editor at InfoWorld. On Twitter: @judell