Ben Williamson
Innovation labs have become a key focus for research emerging from the Code Acts in Education project. These labs are organizations that are becoming increasingly influential in the ways that problems in the public sector, such as in education, are defined and solutions identified, and many of them focus in particular on the role of highly coded technologies such as data analytics in both the identification of problems and their solutions. In this sense, innovation labs are key social institutions in the circulation of the ideas that make computer code, software, and the digital data they facilitate, into technological participants in the public sector–as my own research on labs’ role in the formation of the computing curriculum for schools in England illustrates.
On 9th July 2015 I was invited to participate in LabWorks 2015: Global Labs Gathering, an international conference of labs organized and hosted by Nesta in London. One of the notable refrains of the event was around ‘theory.’ Derek Miller from The Policy Lab in Boston, for example, asked delegates to ‘get serious about theory’ rather than to rely on a kind of data agnosticism that assumes evidence about public innovation can provide impartial and objective proof of ‘what works.’ What are the theoretical explanations required to help labs make sense of the data?, he asked. Much of the conference was taken up with claims about data as a form of ‘proof,’ and with ideas such as using ‘data science for social good,’ ‘discovery from big data,’ and data as a measure of ‘how the world works.’
My own brief and modest contribution was to ask how academic research and lab practice might work in concert, and I should have been clearer about the need, as I see it, for greater theoretical elaboration of labs’ data work, especially given the strong framing throughout LabWorks around labs as research-based organizations. Labs are only just becoming the focus for academic research–the subjects of research as well as organizations that conduct research–as a new literature review by Piret Tonurist demonstrates. In reflecting on the conference (partly by tracing key ideas through the #LabWorks hashtag on Twitter), I want to suggest some possible routes towards a research agenda and potential theoretical lines of thinking for doing research on, in and with labs—and to suggest the need for labs to engage with critical debates about evidence, lab practices, methods and imagination that have become central in social scientific studies of science, technology and innovation.
Giving and taking evidence
One of the dominant mantras in the labs field is about ‘what works’ in public sector innovation. Nesta, for example, has itself been involved in establishing the national network of ‘What Works Centres’ to collect evidence on ‘what works’ in innovation across sectors, primarily through systematic trials, founded the ‘Alliance for Useful Evidence’ and designed a ‘Standards of Evidence Framework’—a common language for talking about data and evaluation. Nesta has produced a series of articles and reports detailing the importance of ‘experimental’ methods in the practices of government. A recent Nesta piece for The Guardian suggested that:
there are times when government must experiment on us in the search for knowledge and better policy. … We have to experiment on a small scale to have a better understanding of how things work before rolling out policies across the UK. This is just as relevant to social policy, as it is to science and medicine.
Nesta itself has detailed many of its own projects in public and social policy innovation to ‘make government more experimental.’ These approaches are paradigmatic of the enthusiasm among many labs for systematic trials, data and measures in the production of definitive evidence.
But such approaches are not uncontroversial. As the sociologist Will Davies has argued, new evidence practices such as RCTs and the evidence centres represent a shift in political thinking:
RCTs operate according to induction. The facts are meant to speak for themselves; the data and the theory are kept neatly and self-consciously separate from each other. … This is supplemented epistemologically by the rise of Big Data…. The very character of Big Data is that it is collected with no particular purpose or theory in mind; it arises as a side-effect of other transactions and activities. It is, supposedly, ‘theory neutral’, like RCTs.
In this context, Davies suggests, ‘the state becomes a theory-less, inductivist, RCT-ing, data-analytical state, accumulating more and more data to find out “what works.”’ The lab methods of tests and trials, supplemented by data-based metrics and measurement, are premised on the big data epistemology that pattern recognition methods and techniques can reveal meaningful connections, associations, relationships, effects and correlations about human behaviours without the need for prior hypotheses, theoretical frameworks or further experimentation. The human element that goes into any methodological inquiry is erased by such claims, and replaced by the assumption, as geographer Rob Kitchin puts it in The Data Revolution, that ‘through the application of agnostic data analytics the data can speak for themselves free of human bias or framing, and that any patterns and relationships within big data are inherently meaningful and truthful’.
The apparent theory-neutrality and data-agnosticism of labs is one feature requiring much greater scrutiny and theorization. Through ethnographic work with Policy Lab UK in the Cabinet Office, for example, Lucy Kimbell has mobilized the theoretical work of the American pragmatist CS Peirce to theorize the different kinds of evidence that labs work with, drawing attention to the differences between deductive, inductive and abductive evidence practices. Another potential line of inquiry around labs’ use of evidence might be around the extent to which they make data. ‘Data’ itself is derived etymologically from the Latin dare, meaning ‘to give.’ When we use the word data, however, we are usually referring to those elements that are ‘taken’ (capere) or selected, not those units that have been given by nature to the scientist. Yet for some enthusiast advocates of data, and particularly digital data, it appears as though those data are indeed naturally given representations of reality. It would be a useful starting place for research on labs to detail the specific assumptions about data they work with and the kind of evidence they produce—is this evidence taken as a partial selection from all that could have possibly been given, or is the assumption that reality is giving evidence that labs are merely capturing in the form of raw and unmediated data?
Lab practices
In his LabWorks presentation, and drawing on an earlier article for Nesta, Charles Leadbeater claimed that ‘labs are places where people test theories.’ This is partly true. But it neglects the extent to which ‘laboratory life’ is always shaped by a range of social, personal, technical, political and economic circumstances. Theories get tested when funding arrangements are in place. Theories get tested when the right social networks of expertise form around them. Theories get tested through particular technical devices, which are themselves produced by particular organizations with devices to sell. Moreover, the work that gets performed in labs always takes place within a particular scientific style of thinking—a more or less coherent way of making arguments, constructing explanations, and building conceptual models within a particular scientific community, or a ‘thought collective’ as Lugwig Fleck, the philosopher of science, termed it:
a community of persons mutually exchanging ideas or maintaining intellectual interaction. Members of that collective not only adopt certain ways of perceiving and thinking, but they also continually transform it—and this transformation does occur not so much “in their heads” as in their interpersonal space. … When a thought style, developed and employed by a collective, becomes sufficiently sophisticated, the collective breaks into a small esoteric circle—a group of specialists which “are in the know”—and a wide exoteric circle for all those members, who are under the influence of the style, but do not play an active role in its formation.
At LabWorks itself, it seemed clear that there was a small esoteric circle in this emerging thought collective–or, perhaps better, a collective thought network, as this social network diagram from LabWorks illustrates–around which a wider exoteric circle was invited to gather.
These points make it clear that any research undertaken to understand innovation labs would need to get inside the ‘laboratory life’ of such spaces. As I have written before, when Bruno Latour and Steve Woolgar produced their classic ethnographic account of the work of scientists in Laboratory Life, their conclusion was that scientific laboratories are deeply complex places where negotiations, arguments, disagreements and compromises are constantly hammered out as scientists seek to construct ‘scientific facts,’ or models of how the world works through their distinctive lab practices. They drew attention to the need to ‘follow the actors’ that inhabit labs: to follow the scientists in their everyday laboratory practices and the thought collectives to which they belong, but also to follow the nonhuman actors such as the pieces of paper that govern how and when the work gets done; the political and institutional funding incentives that dictate the resources available for it; the technical devices that shape the ways in which phenomena are observed and recorded; and the written papers that communicate those findings and circulate beyond the lab to make ‘scientific facts’ known and accepted. Indeed, Charles Leadbeater suggested that innovation labs should get in the habit of both sharing their data in data archives and peer reviewing each others’ findings–both lab practices very particular to the laboratory life of scientific thought collectives.
Innovation labs are places where new kinds of social facts are now being created and circulated according to the styles of thinking of particular thought collectives. Through new kinds of evidence practices and data analytics, labs claim, they are generating new insights into contemporary social and public problems. A particular style of thinking percolates through labs, one that could be discerned at LabWorks in the appeal to digital data, claims about ‘what works,’ a desire for citizen-centricity, and in the call to engage in design thinking and other design-for-policy methods. Fruitful work could be undertaken by engaging ethnographically in the laboratory life of labs, tracing and unpacking the style of thinking that governs their work and then working backwards to track how such a thought style has been convened from complex genealogical lines of thinking. This would consider, for example, how ideas about design, digital R&D and data analytics have been translated into kind of style of thinking that informs labs’ production of methodological guidance for policy professionals. Science and technology studies could provide the theoretical and empirical resources for ‘following the actors’–both human and nonhuman (e.g. following the #psilabs hashtag in Twitter)–in such research in labs.
Socializing methods
As their production of methodological toolkits, handbooks and guidance suggests, a key part of the laboratory life of innovation labs is the methods they use to identify public and social problems for intervention and rectification. As with the social and technical apparatus of the laboratory more generally, the specification of research methods ultimately involves methodological decisions about the design of instruments, the selection of samples, decisions made about analysis, and the interpretations brought to bear on the data. Moreover, methods are underpinned by particular views of the reality to be examined. So, for example, many data science methods are based on the assumption that social reality can be understood through its data; data are viewed as ‘statistical facts,’ and the more data that are available are therefore seen as producing a richer and more detailed picture of that reality. Other, more ethnographic methods, in contrast, tend to see social reality in terms of complex social, cultural and embodied experiences that can only be traced through the ‘little data’ of up-close observation in the field. The tensions between the two approaches were dramatized in part during a LabWorks panel debate between David Halpern of the Behavioural Insights Team and Christian Bason of the Danish Design Centre. This was ultimately a methodological debate that represented the extent to which methods are embedded in particular epistemological assumptions about how we can see and know the social world.
Indeed, methods themselves have ‘social lives’: they are designed in particular social settings, by specific actors and their sponsors, to surface particular kinds of data; they are underpinned by particular assumptions, commitments and aspirations; they generate data that are collected in ways that make them available to be interpreted according to specific theoretical frameworks of understanding; and they are predicated on existing views and theories of how social reality works. In other words, methods are both socially produced and socially productive: socially produced in that methods do not provide an impartial ‘view from nowhere,’ but are embedded in distinctive disciplinary approaches and assumptions; and socially productive in that methods are consequential to how particular aspects of social reality are known, and to how that reality might therefore be acted upon in order to improve, enhance or modify it. The notion of ‘socializing methods’ registers this double process of breathing methods to life through social means, and of then mobilizing methods to make sense of the social world and wrap new social norms around it. This raises questions about how methods are used to collect, store and transmit numerical, textual, aural or visual signals; how they work in relation to standard social science techniques; how they relate to social and political institutions; and how they enable particular social ‘realities’ to be made known and thus made amenable to being acted upon in the name of improvement.
The commitment of public and social innovation labs to emerging methods requires critical alertness to the social life of the methods now increasingly being mobilized to make sense of the problems that government faces and to which policymakers are seeking solutions. By hybridizing methods of digital R&D, data science approaches, design-oriented methodologies such as user ethnography and user-centred design, with tests, trials and experiments, labs have become expert methodologists of the social, with the methods for making the social world known and the techniques for rectifying its problems. A research agenda around innovation labs might seek to engage critically with the social life of their methods—the socially-specific epistemological underpinnings, the technical histories, and the institutional origins of methods, but also the socially productive capacity of their methods to shape modes of perception and inform decision-making.
Future imaginaries
Many LabWorks presentations drew specific attention to labs’ involvement in imagining and designing alternative futures. Christian Bason, for example, talked about the role of design-for-policy methods in ‘giving form to desirable futures,’ while Charles Leadbeater urged labs to become ‘clients of the future system’ working from a shared vision of the future to devise strategies in the present that might act to materialize it.
In approaching these claims social scientifically, the kinds of desirable futures articulated by labs might be seen as ‘sociotechnical imaginaries’ as defined by the sociologist of science and technology Sheila Jasanoff: collectively held, institutionally stabilized, and publicly performed visions of desirable futures, that are animated by shared understandings of forms of social life and social order and made attainable through the design of technological projects. By tracing something of the laboratory life of labs, it may be possible to examine how their key methods and the messages derived from them inscribe particular futures. Taking this approach could help to provide a kind of ‘history of the present’ of labs—a sense of how different lines of thinking have gradually coalesced and stabilized in the current work of labs—as well as a sense of the ‘history of the future’ that labs project—how the sociotechnical imaginaries of desirable futures projected by labs have been formed in specific historical contexts. Understanding the imagination of innovation labs may help broader audiences to appreciate the visions of desirable societies that they are trying to create.
One comment on Twitter after my talk questioned the idea of academics doing research on labs with the claim that it would be better to do action research with people. I’m not trying to dispute the usefulness of labs doing action research, or citizen-centred ethnographies, and so on. Rather, my sense is that labs might benefit as well from engaging with the critical debates about evidence, lab practices, methods and imaginaries that have animated academic research in fields such as sociology, geography and science, technology and innovation studies for the last couple of decades. There are critical theoretical resources, and empirical approaches, available in these fields to enable research to be conducted on labs, but also to help make sense of ethnographic research happening in labs, and maybe to support collaborative research by academics working with labs. It would be a missed opportunity if academic research and lab-based research became mutually antagonistic. Getting serious about theory, by considering theoretical ways of approaching the evidence, practices, methods and imagination of labs, would be a way forward.
code acts in education 