10 definitions of datafication (in education)

Ben Williamson


What is datafication? And how does it affect education? These questions were put to me ahead of conference discussion panel recently. While writing a few notes, it quickly became apparent I needed some categories to sort out my thinking. In simple terms, datafication can be said to refer to ways of seeing, understanding and engaging with the world through digital data. This definition draws attention to how data makes things visible, knowable, and explainable, and thus amenable to some form of action or intervention. However, to be a bit more specific, there are at least ten ways of defining datafication.

1 Historically
Datafication as we know it today has a long history, going back at least as far as the industrial revolution and efforts  then to capture statistical knowledge of the state, society and its population, and then to use that knowledge to come up with better institutions and practices of management and intervention. David Beer offers a really good historical view of the historical evolution of ‘metric power.

In terms of education, Michel Foucault of course articulated how children could be counted in terms of their development, knowledge, behaviour, progress, worth, cleanliness, age, social class and character, in order that they could then be ranked, supervised and disciplined more effectively. He called schools and classrooms ‘learning machines’. Within education policy, Martin Lawn and others have charted the historical rise of data in education systems. These authors have shown, for example, how the nineteenth century Great Expositions became carefully stage-managed presentations of different states’ educational performance rates, so allowing different national systems to be compared for their effectiveness in producing the labour required for social and economic progress. These early historical developments in the datafication of education have slowly given rise to the ‘global race’ that we still see in education policy today, driven by comparative analysis of performance in large scale assessments (LSAs).

Although there are clear continuities from the past to the present, the current version of datafication through ‘big data’ also represents a bit of a rupture with the past. The assessment data that dominates LSAs is sampled, collected at long temporal intervals, and slow to collect. New digital datafication technologies such as ‘learning analytics,’ by contrast, harvest data in real-time as students complete tasks, enable high-speed automated analysis and feedback or adaptivity, and can capture data from all participants rather than a sample. They also allow individuals to be compared against each other and with aggregated norms calculated in massive datasets, rather than the broad-brush comparison of national systems enabled by LSAs.

2 Technically
In technical terms, datafication is a process of transforming diverse processes, qualities, actions and phenomena into forms that are machine-readable by digital technologies. Datafication allows things, relationships, events, processes to be examined for patterns and insights, often today using technical processes such as data analytics and machine learning which rely on complex algorithms to join up and make sense out of thousands or millions of individual data points. The technical language of datafication can get quite bewildering, proliferating to include technical concepts and methods which are even being modelled to some degree on human processes–so-called ‘cognitive’ computing, deep ‘learning’, and ‘neural’ networks.

Thinking educationally, it’s intriguing that much of the language associated with digital datafication refers to learning, training and neural processes of cognition. Datafication relies to a significant technical degree on ‘learning machines’. Algorithms have to be ‘taught’, using ‘training sets’ of past data to determine how to act when put ‘into the wild’ to process live and less structured data. This can be done through ‘supervised learning’, which sounds rather like direct instruction, or through ‘unsupervised learning,’ which is more like autodidactic learning through experience. DeepMind’s AlphaGo Zero–a highly advanced AI program for unsupervised learning–for example, learns purely from its own experience and from a ‘self-reinforcement learning algorithm’ that rewards it for every ‘success’ it experiences. BF Skinner’s famous behaviourist ‘teaching machines’ have been encoded in algorithmic form.

Also, in the technical sense datafication relies on the material infrastructure of hardware, software, servers, cables, connectors, microprocessors—all of the ‘stuff of bits’, as Paul Dourish has argued, that has to be assembled together in order to generate data. The materialities of datafication significantly shape how data are generated and how they can be put to use.

3 Epistemologically
Thinking epistemologically about datafication concerns what we can know from data. For some, datafication rests on the assumption that the patterns and relationships contained within datasets inherently produce meaningful, objective and insightful knowledge about complex phenomena. As Rob Kitchin has shown, this empiricist epistemology assumes that through the ‘application of agnostic data analytics the data can speak for themselves free of human bias or framing.’

For critics, however, this empiricist epistemology is flawed because all data are always framed and sampled; data are not simply natural and essential elements that are abstracted from the world in neutral and objective ways to be accepted at face value. As Nathan Jurgenson has put it, data do not provide a ‘view from nowhere‘ because factors such as algorithms, databases, and venture capital pre-format data and so shape what may be seen or known. Data don’t tell the unbiased ‘truth’ because the data points captured and analysed are always affected by the choices of the original designers.  Making sense of data is also always framed–data are examined through a particular lens that influences how they are interpreted. Jose van Dijck has described an epistemological ‘data-ist’ trust in the numbers provided through datafication.

Epistemology in this sense extends to include a methodological definition of datafication. Datafication is a process of employing certain data scientific methods to produce, analyse and circulate data. These methods have their own social origins–or ‘social lives’ as John Law claims–and derive from and reproduce the particular epistemological assumptions of the expert groups that created them. Datafication, in other words, is epistemological and methodological.

4 Ontologically
Datafication also raises ontological questions about what data really are. One view is that data are simply ‘out there’ waiting for collection, as supposed in the term ‘raw data’ and in the view that ‘data speak for themselves’. The other, more common in contemporary social science, is that data are inseparable from the software and knowledge employed to produce them. Data do not simply represent the reality of the world independent from human thought but are constructions about the world. These insights into the ontology of data are often associated with sociological theories of science, technology, statistics and economics–Sheila Jasanoff pulls these strands together in a recent article on ‘data assemblages’.

Moreover, however, data have consequences and shape individual actions, experiences, decisions and choices. In that sense, they shape and change reality; they have ontological consequences and partake in making up reality. So, ontologically, datafication is a product of the social world and of specific practices, but it also acts upon the world and on other practices, changing them in various ways. For example, Marion Fourcade has  shown how the statistical practices of economists, of ‘ever-finer precision in measurement and mathematics … have constructed a wholly separate and artificial reality,’ a ‘make believe substitution’ that is entirely made out of historical and disciplinary conventions, ‘nothing more’. Yet, as Fourcade adds, if you change the statistical convention, ‘the picture of economic reality changes too’, sometimes with dramatic real-world results. As such, datafication is ontological because it has the potential to produce or perform different versions of reality–what actor-network theorists call ‘ontological politics’.

5 Socially
Datafication is accomplished by social actors, organizations, institutions and practices. So today we have data scientists, data analysts, algorithm designers, analytics engineers and so on all bringing their expertise to the examination of data of all kinds. These people or experts are housed in businesses, governments, philanthropies, social media firms, financial institutions, which have their own objectives, business plans, projects and so on, which frame how and why digital data are captured and processed. In this sense, datafication can be defined socially because it is always socially situated in specific settings and framed by socially-located viewpoints.

In education, we have ‘education data scientists’ and learning analytics practitioners, engineers and vendors of personalized learning platforms, even entrepreneurs of artificial intelligence in education, all now bringing their own particular forms of expertise to the examination and understanding of learning processes, teaching practices, schools, universities and educational systems. They are supported by funding streams from venture capital firms, philanthropic donations from wealthy technology entrepreneurs, impact investment programs which all direct financial resources to the datafication of education. Putting it super-simply, datafication exists because people and institutions of society make it so.

Moreover, datafication needs to be defined socially because much data is captured from the social world—people, institutions, behaviours and the full range of societal phenomena are the stuff of data. As Geoffrey Bowker has memorably put it, ‘if you are not data, you do not exist’! People are data; societies are the data. Even more consequentially, these social data can be used to reshape social behaviours. Bowker adds that as data about people are stored in thousands of virtual locations, reworked and processed by algorithms, their ‘possibilities for action are being shaped’.

6 Politically
The new actors undertaking datafication are invested with a certain form of data power. Expert authority, as William Davies argues, increasingly resides with those who can work with complex data systems to generate analyses, and then narrate the results to the public, the media and policymakers. This is why governments are increasingly interested in capturing the digital traces and datastreams of citizens’ activities. By knowing much more about what people do, how they behave, how they respond to events or to policies, it becomes possible to generate predictions and forecasts about best possible courses of action, and then to intervene to either pre-empt how people behave or prompt them to behave in a certain way. For example, there’s a whole ‘Data for Policy’ movement and new funding streams for ‘GovTech’ applications in the UK to realize the potential of ‘Government by Algorithm’. Evelyn Ruppert and colleagues have termed this ‘data politics’ and note that power over data no longer only belongs to bureaucracies of state, but to a constellation of new actors in different sectoral positions.

Something of an arms race is underway by those organizations that want to attain data power in education. Education businesses like Pearson are putting large financial, material and human resources into technologies of datafication, and are seeking both to make it commercially profitable and also attractive to policymakers as a source of intelligence into learning processes. Dorothea Anagnostopoulos and colleagues have written about the ‘informatic power’ possessed by the organizations and technologies involved in processing test-based data. But some of that power is now being assumed by those actors, organizations and analytics technologies that process digital learning data and turn it into actionable intelligence and adaptive, personalized prescriptions for pedagogic intervention.

7 Culturally
This definition draws attention to datafication as a cultural phenomenon and as a concept that has attained a privileged position in the view of the public, businesses, governments and the media. Increasingly, it seems, data and algorithms are invested with promises of objectivity and impartiality, at a time when human experts are not necessarily to be trusted because they’re too clouded by subjective opinion, bias and partiality. An article in the Silicon Valley ed-tech magazine EdSurge effectively represented how the objectivity of data has been culturally adopted and accepted in some parts of the education sector. It claimed teachers are unable to recognize how well students are engaging with their own learning because the teachers are too subjectively biased. This speaks to a cultural narrative which frames datafication in terms of mechanical objectivity, certainty, impartiality.

But the cultural acceptance or otherwise of datafication is of course context-specific. In some European countries such as Germany the cultural narrative of datafication and algorithms is more contested, and perhaps legally and politically inflected. It would be interesting to tease out how datafication in general and datafication of education in particular becomes culturally embedded or not in different geographical, political and social locations. So for example, datafication in education may appear to be a largely Anglophone phenomenon. Recently, however, a new report on ‘Learning Analytics for the Global South’ appeared which considered ‘how the collection, analysis, and use of data about learners and their contexts have the potential to broaden access to quality education and improve the efficiency of educational processes and systems in developing countries around the world’. Datafication of education is becoming culturally sensitive.

8 Imaginatively
Datafication is the subject of breathless utopian fantasies of real-time responsive smart cities, global Internet of Things, human-machine symbiosis, algorithmic certainty, hyperpersonalized services, driverless cars and so on—a world plastered with a new shiny surface of machine-readable data, which acts as a fuel for an automated, responsive, personalized environment which constantly moulds itself around us.

Education is affected by the same fantasies and utopian imaginaries. At last year’s British Science Festival, Sir Anthony Seldon, Master of Wellington College and VC of the University of Buckingham, presented a picture of a robotized future of schools, with ‘extraordinarily inspirational’ machines completely personalizing the education journey, ‘adaptive machines that adapt to the individual,’ that ‘listen to the voices of the learners, read their faces and study them in the way gifted teachers study their students,’ know what ‘excites’ learners and can ‘light up the brain’ through ‘intellectual excitement.’

Datafication is in this sense the subject of imagination–but imaginary visions can sometimes catalyse real-world applications, with powerful visionaries gathering coalitions of support to make reality conform with their utopian ideals. SIlicon Valley entrepreneurs have animated their visions of data-driven education through the capture or donation of funding and engineering teams, for example.

9 Dystopically
In contrast to the utopian imagination, datafication is also a great source of anxiety. Concerns circulate about gross privacy invasion, panoptic dataveillance, data bias against ethnic groups, the manipulation of behaviours through persuasive design, viral spread of computational propaganda powered by data-driven profiling and targeting, information war, data breaches, hacking and cyberterrorism, and that datafication reduces people to their data points—as if we are our data, perfectly knowable through our digital traces.

In education, the children’s writer Michael Rosen recently posted a tweet along these lines, writing that: ‘First they said they needed data about the children to find out what they’re learning. Then they said they needed data about the children to make sure they are learning. Then the children only learnt what could be turned into data. Then the children became data.’

Recently a lot of commentary has emerged about the social and emotional anxiety experienced by students in both schools and universities. Many of these psychological frailties are at least partly blamed on social media and other technologies that harvest up data from young people and then target and manipulate them for commercial profit. Richard Freed calls it the ‘tech industry’s psychological war on kids’. These kinds of stories are now part of a kind of cultural narrative about the dystopian, nightmarish effects of datafication on children, which is happening both in their own time and in an increasingly data-driven education.

10 Legally & ethically
Finally, there are legal, ethical and regulatory mechanisms shaping datafication. Europe is much more privacy-focused than the US, for example, as the incoming EU General Data Protection Regulation shows. So how datafication plays out—what datafication is—is itself shaped by law, ethics and politics.

In the US, for example, specific federal acts such as COPPA and FERPA exist to protect children’s privacy, and organizations like the Internet Keep Safe Coalition enforce them. Other organizations such as the Future of Privacy Forum exist to produce ‘policy guidance and scholarship about finding the balance between protecting student privacy and allowing for the important use of data and technology in education’. The US also has the 2015 Every Student Succeeds Act (ESSA), which has made it possible for states and schools to apply for additional funding for personalized learning technologies. So there’s a new federal act in place which performs the double task of stimulating market growth in adaptive personalized learning software and incentivizing schools to invest in such technologies in the absence (or at least shortage) of public funding for state schooling.

Of course, the ethical issues of datafication of education are considerable and fairly well rehearsed. An interesting one is the ethics of data quality–a topic discussed by Neil Selwyn at the recent Learning Analytics and Knowledge (LAK) conference. There are significant potential consequences of poor data in learning analytics platforms. In other spaces, such as healthcare and military drones, the consequences of poor data quality can lead to disastrous, even fatal, effects. Poor quality datafication of education may not be quite so drastic, but it has the potential to significantly disrupt students’ education by leading to mismeasurement of their progress, misdiagnosis of their problems, or by diverting them on to the ‘wrong’ personalized pathways.

I’m sure datafication could be cut in different ways. But hopefully these categories capture some of its complexity.

Image by Kevin Steinhardt
Posted in Uncategorized | Tagged , , , , , | 2 Comments

Globalizing education standards with ISO 21001

Ben Williamson

Compass Vivek Raj large

Standards are everywhere but largely invisible. They define the production of things, from the size of a brick or the dimensions of a credit card to the programming languages used to code software, and act as rules for specific processes and practices. The creation of a new global standard for management processes in education may not at first seem terribly significant. This post interrogates ISO 21001, a standard for ‘Educational Organizations Management Systems’ due for release by the International Organization for Standardization (ISO) later in 2018. While few will have heard of it, it could define priorities, products, processes and practices in the education sector at global scale for years to come.

Standard definitions
A lively academic literature has grown around standards over the last two decades. In the influential Sorting Things Out Geoffrey Bowker and Susan Star defined standards as ‘any agreed-upon rules for the production of (textual or material) objects’ which are ‘deployed in making things work together.’ Standards, they added, persist over time and different places, are often enforced, difficult to change, and define certain ways of organizing things and actions.

That may sound a little abstract. In the tangible world, standards define almost everything. There are standards for the dimensions of kitchen goods and furniture, standard measures, standard fonts and paper sizes, standard economic models, standards for food products, standard business practices, standard forms to fill in, standard formats for cataloguing and indexing, governmental standards, standard classifications of illness and healthiness, standards for ensuring software can operate on computer hardware and that data are interoperable across systems, and much more.

People are standardized too. Standard measures of personality or citizenship, standards of dress and behaviour, standards for credit-scoring and social media profiling, and standards that define social class, socio-economic status, gender, nationality and ethnicity all affect people’s everyday lives. Standard linguistic definitions help us make sense of ourselves and the world we inhabit.

Standards are, as Lawrence Busch memorably called them, ‘recipes by which we create realities,’ and they are ‘about the ways we order ourselves, other people, things, processes, numbers, even language itself.’ And as Susan Star and Martha Lampland have noted, they are not innocent or neutral, but ‘codify, embody or prescribe ethics and values, often with great consequences for individuals (consider standardized testing in schools for example).’

In short, standards may seem invisible, but they matter—they are consequential to how the world is organized, how people and their behaviour are regulated, and how processes and objects are defined and measured. Those who control standards therefore hold great power to coordinate and organize social, economic, cultural, ethical and political life. Standards constitute societies.

Standardizing education
The field of education is awash with standards. Although standards have always played a significant part in education—in standard class sizes, teacher standards, standards of achievements, standard codes of behaviour, standard school uniforms and so on—today the most common references are to standardized curricula, learning standards, and standardized tests. Rules specifying consistent approaches to defining the content, objectives, and levels of achievement expected from students of a given course underpin them.

Dorothea Anagnostopoulos and coauthors have argued that quantitative scores from standardized tests based on defined learning standards and courses of study in standardized curricula have become the major measure of student, teacher and school performance, and thus of what ‘counts’ as a ‘good’ student, teacher or school. Standardized tests, standardized curricula and learning standards play significant roles in the complex of technology, people and policies that make up ‘data infrastructures of test-based accountability.’

With the expansion of international large-scale assessments such as the OECD’s PISA, researchers such as Sam Sellar and Bob Lingard have drawn attention to how these ‘products’ are now standardizing how education is approached in countries around the globe. Globalized data infrastructures based on standardized assessment products exert powerful effects on national education systems, schools, staff and students alike. Data infrastructure may even reshape education policy to become more globally standardized, supported by a global education industry of products, technologies and services. One such service provider is the International Standardization Organization, currently preparing a standard for educational management processes and practices.

Global standards
The ISO is a global organization based in Geneva and made up of over 160 national standards organizations working in partnership. It has published 22,047 International Standards, described as ‘documents that provide requirements, specifications, guidelines or characteristics that can be used consistently to ensure that materials, products, processes and services are fit for their purpose.’

The ISO states that ‘standards touch almost every aspect of our lives,’ and its mission is therefore ‘successfully sharing the best ideas and methods’ through its standards. In this sense, although it describes itself as a ‘neutral environment,’ the ISO is profoundly normative, seeking to promote its standards to achieve specific objectives as agreed by expert participants in its technical committees. It sees itself as a ‘global platform for creating consensus.’ A promotional ISO video explains how its standards can help promote growth, open markets, make trade fairer, tackle global challenges, maintain connectivity, keep us entertained, enhance productivity, promote innovation, and keep us safe and healthy. Keller Easterling, in a chapter of Extrastatecraft, defines the ISO’s standards as a ‘global operating system’ that ‘formats and calibrates’ many aspects of the contemporary world.

ISO 21001
Later in 2018, the ISO intends to release a new international standard for education. It has the potential to become part of the global operating system for the education sector.

ISO 21001 ‘Educational Organizations Management Systems’ has been in development for nearly 5 years, having been first approved as a proposal in early 2014. Development of the standard has been undertaken by Technical Committee 288, which itself consists of 140 expert members from 44 participating countries, plus 14 ‘observer’ countries, led by the Korean Agency for Technology and Standards. A map of participation shows involvement of the UK and much of mainland Europe, Australia, Canada, and many South American and Asian countries.

Keller Easterling claims that the ISO is hard to examine because of its opaque technical language, labyrinthine website, and inaccessibility of its standards documentation. Because the ISO’s standards are only available if you purchase them, it is hard to tell exactly what requirements, guidance or characteristics they promote. This is true of ISO 21001, not helped by the fact that it is at present still in ‘final draft’ form. Although  notionally open for consultation, even access to the draft requires the user to purchase it.

As Easterling notes, while the ISO ‘strives for universal impact’ it operates as a ‘secretive institution with no truly public dimension—no appeal to a citizen who is not also a consumer.’ But a published ‘Briefing Note’ and publicly available PowerPoint slides on the planned implementation and implications of ISO 21001 can help us get closer.

Standardizing educational organizations
The standard applies to management systems used by educational organizations, by which it means ‘what the organization does to manage its processes or activities in order that its products or services meet the organization’s objectives.’ These include ‘satisfying learners’ requirements’ and ‘meeting educational objectives’ as well as ‘balancing requirements from other stakeholders’ and ‘complying to regulations.’ In slightly different terms, ISO 21001 ‘concerns how an organization goes about its work,’ and ‘provides a common management tool for organizations providing educational products and services capable of meeting learner and other customer requirements and needs.’

So ISO 21001 is primarily a standard of management process for administering large and complicated educational organizations. Notably, the ISO claims the standard is making available a ‘comprehensive set’ of ‘successful practices’ that is for ‘applicable to all educational organizations that provide, share and facilitate the construction of knowledge to learners through teaching, training or research, regardless of type, size and product and service provided.’ It is not confined to either the schools or HE sectors, but ‘applies to the management system of any organization which utilizes a curriculum to provide, share and transfer knowledge.’ In other words, it appears to take a highly standardizing approach to educational organizations, treating them as universally comparable and manageable through the same best practices.

Standard benefits
Additionally, ‘Although learners and educational organizations worldwide are the main beneficiaries from this new management system standard, all stakeholders (everyone) will benefit from the output of standardized management systems in educational organization.’ The explanatory PowerPoint details that while learners have requirements of educational organizations, so too do the labour market and government, which, like learners, expect ‘satisfaction’ from a well-managed service.

The benefits listed in the briefing note are: a) better alignment of educational mission, vision, objectives and action plans, b) inclusive and equitable quality education for all, c) promotion of self-learning and lifelong learning opportunities, d) more personalized learning and effective response to special educational needs, e) consistent processes and evaluation tools to demonstrate and increase effectiveness and efficiency, f) increased credibility of the educational organization, g) recognized means to enable organizations to demonstrate commitment to education management practices in the most effective manner, h) a model for improvement, i) harmonization of national standards within an international framework, j) widened participation of interested parties, and k) stimulation of excellence and innovation.

Some of these need unpacking, because they are full of assumptions about the ways educational organizations should operate.

Standard language of learning
The first thing to note is the language of learning it operationalizes. The slides and briefing booklet refer to ‘self-learning,’ ‘personalized learning’ and related terms. The available PowerPoint notes that the ‘first principle’ of the standard is that ‘Educational Organizations should actively engage learners in their own learning’ and that ‘teaching is defined as working with learners to assist and support them with learning.’

Along with references to learners’ ‘construction of knowledge’ these sound laudable principles for those with a slightly ‘progressive’ view of education, but they do clearly presuppose that such approaches to education are widely agreed-upon. Given that educational research and practice remains philosophically divided between those who (broadly speaking) take a progressive ‘student-centred’ view and those who emphasize more liberal values regarding the transmission of powerful knowledge through subject-based curricula, the ISO’s standard-setting of student-centredness in educational organizations is controversial.

Even more controversial, perhaps, is that the ISO’s endorsement of ‘personalized learning’ resonates strongly with a particular (contested) view of education that has become closely associated with the entry of big data and artificial intelligence into education. In the slides promoting the standard, one of the stated aims is standardizing processes for the ‘identification and traceability of learners throughout the organization.’ Making students trackable through their data traces is therefore being put forward as a requirement of an effective, ‘personalized’ educational organization.

Standard harmonization
The standard’s benefits also include harmonization of national standards with international frameworks. This kind of harmonization has of course been the long-term goal of other international organizations such as the OECD, which has used evidence from its international large-scale assessments to compare national systems and derive ‘policy-relevant’ findings that might shape national-level decision-making, intervention and reform.

In this sense, the new ISO standard is supposed to participate in the contested practice of ‘commensuration’ whereby different qualities of educational systems and institutions are rendered equivalent through quantitative methods. As Steven Lewis points out, it ‘involves comparing how different national schooling systems are positioned within a commensurate global space of measurement’ and, coupled with the production of ‘examples’ of ‘best practice,’ influences how schooling is understood and practised. ISO 21001 is a commensuration instrument for easing the harmonization of school measurement at national and international scales.

Management standards
Another aspect of the standard is its references to the language of effectiveness, efficiency, evaluation, and improvement. As a management standard, it’s inevitable perhaps that it would appear managerialist. Indeed, ISO 21001 is in fact an adaptation for the education sector of its existing ISO 9001 quality management standard.

Easterling has documented how the ISO 9000 family of standards have become the ISO’s most popular products, thereby imposing uniform management and quality assurance processes on organizations worldwide. They are based on existing management theories pertaining to ‘the process of production, the procedures and practices of a company’ and their ‘social architecture,’ and have catalysed the development of a global consultancy industry regarding standards compliance. Compliance with ISO 9000 standards involves an organization evaluating itself in terms of its objectives, such as customer satisfaction, and often, Easterling argues, lead to ‘obsessive data gathering and metrics  … to quantify or prove that deliberate objectives have been met.’

Indeed, the ISO 21001 PowerPoint itself says it contains over 50 references to documented information and records that must be maintained or retained, and ‘an entire informative annexe gives examples of measures and tools which can be used in collecting and managing information.’ Compliance with the standard will involve mandatory internal auditing, review of programmes and annual reviews of the organization’s management systems ‘to address deficiencies.’

Data standards
As a further requirement, institutions must communicate publicly ‘learners’ performance data’ and ‘learner and other feedback, including satisfaction surveys and complaints.’ As these examples indicate, compliance with the standards will require data infrastructure to enable collection, analysis and presentation to the public of the required information. Moreover, making such measures public is likely to configure organizational behaviours. Geoffrey Bowker has argued that ‘It is not only the bits and bytes that get hustled into standard form in order for the technical infrastructure to work. People’s discursive and work practices get hustled into standard form as well. Working infrastructures standardize both people and machines.’

As an adaptation of ISO 9000 standards, then, ISO 21001 makes data gathering on customer satisfaction into an essential requirement of educational organizations. Of course, there is nothing especially unique about this. In the UK, a new Higher Education infrastructure for student data collection is already being built that requires universities comply with ISO 9001 quality management standards. As with ISO 21001 prioritizing students’ requirements and satisfaction, the UK data infrastructure starts from the principle of putting ‘students at the centre of the system.’ It will enable data to flow from HE institutions for centralized analysis to produce competitor rankings on many metrics, and is supposed to make HE data more usable for customers (students) as well as for policymakers and institutions themselves.

In this sense, compliance with ISO 9001 or its ISO 21001 outgrowth will reshape people through standardizing their practices. It will reshape academic staff as providers of ‘satisfying’ services, and students as consumers with rights to feed back and complain.

Market standards
What should not be overlooked here is how that data infrastructure is being developed to participate in greater marketization of HE in the UK. Janja Komljenovic and Susan Robertson have persuasively shown how HE markets are being ‘made’ through ‘ideological and political changes in the governance of higher education to make it a more globally competitive producer of knowledge, and a services sector.’ They also, importantly, point out that these macro-objectives of marketization are only possible through micro-processes such as the ‘development and deployment of policies, technologies, instruments and other “formatting devices”.’

Although Komljenovic and Robertson do not mention standards specifically, it is possible to view ISO 21001 as a kind of ‘formatting device’ to ensure that educational organizations fulfil metric requirements of data collection and reporting. As Komljenovic and Robertson note:

New technologies are constantly being invested in and deployed ‘to create efficiencies’ in methods of working, as a means to gather more and more information on the institution and its processes, to monitor staff and to measure a wide range of forms of satisfaction—most particularly student satisfaction. … At the same time, these technologies are also part of governing tools as its results influence university structures, policies and actions. Besides student surveys, there are also national and international rankings, benchmarks, indicators and so on.

The main achievement of ISO 21001, arguably, will be to create a new global standard for educational organizations to be measured and monitored as a market, according to metrics formulated from a strange hybrid of management theory and ‘student-centred’ educational discourse. Educational organizations will be ‘formatted’ by the standard to produce the data required to evidence how well they are achieving their objectives. Those objectives themselves, however, appear to be pre-formatted by ISO assumptions about what counts as ‘good’ teaching and learning.

A global educational operating system
Time will tell what difference ISO 21001 makes to educational organizations, national systems, or even global education policy. In many ways, it represents little that’s new since standardization, quantification and marketization are already well entrenched in many education systems around the world. But it does raise a few headline points:

  • To a significant extent, the ISO is joining a policy network of international organizations that have increasingly participated in shaping global education policy over the last two decades—among them the OECD, UNESCO, the World Bank, and the World Economic Forum
  • ISO 21001 is a potential key instrument for the expansion of global education policy and a global education industry, a standards-based means for identifying policy problems and a specification tool for providers of policy solutions
  • ISO 21001 is intended as part of a global operating system for educational organizations, a kind of ‘formatting device’ to ensure institutions follow uniform management processes which require them to evidence their achievements according to ‘quality’ measures such as students’ and other stakeholders’/customers’ service ‘satisfaction’
  • The standard constitutes the basic set of rules for educational infrastructures–it defines how educational organizations should be organized, managed, monitored, recorded, compared, and acts as a coordinating device to align all actors in the system around the same shared standard
  • It is supposed to stimulate and further enable greater harmonization and commensuration between national standards and international frameworks, leading to national education systems being reconfigured around standards derived in secret by the ‘consensus’ of ISO expert members
  • The ISO endorsement of personalized self-learning and student-centredness enshrines a disputed philosophy of education in standardized form as a ‘principle’ of education systems which  organizations would then be expected to adopt as an objective, and against which they would monitor themselves, and be monitored
  • The standard configures educational organizations as service providers and students as customers whose aggregated levels of consumer satisfaction may be read as economic indicators of an institution’s market competitiveness and positioning, and data on which must be made publicly available
  • The ISO is further entrenching the language and practices of management theory in education systems at global scale, reshaping the organizational practices and social architecture of the education sector around managerialist demands of performance measurement, audit, evaluation, accountability and marketization
  • Compliance with the standard, as with ISO 9000 standards, will bring more consultancies and companies into the education sector able to offer third-party certification and compliance services, further displacing governance of education from the public sector to the private sector

Even if these are not novel or unique, it remains concerning that a secretive international organization is now seeking to stamp a global standard on such practices, effectively creating a uniform international template by which to manage and judge educational organizations and all who inhabit them.

Although it is not yet clear how ISO 21001 will be enacted, the fact that 44 member countries are participating in development of the standard suggests it is likely to have wide international impact in years to come. It could become a ‘recipe for reality’ in education that will shape organizations and practices to conform to its rules and prescriptions. It could have the effect of further standardizing schools, universities and even education ministries. It could configure people into standard form, with practices and behaviours shaped as much as possible to ‘deliver’ standardized services and standardized learning outcomes. As such, it is an important example of why invisible standards need to be brought into public visibility.

Image credit: Vivek Raj

Posted in Uncategorized | Tagged , , , , | Leave a comment

PISA for personality testing – the OECD and the psychometric science of social-emotional skills

Ben Williamson

OECD SEL survey

The Organization for Economic Cooperation and Development (OECD) has published details of its new Study on Social and Emotional Skills (SSES). While the OECD has been administering international large-scale assessments on ‘cognitive skills’ and ‘competencies’ with both children and adults for many years, the new SSES survey represents a significant shift in focus to ‘non-cognitive’ aspects of learning and skills. While details of the science behind its cognitive skills and competencies tests are relatively well known, it is now becoming clear that the OECD’s social-emotional skills programme will emphasize the psychometric science of ‘personality’ measurement.

As part of ongoing research on social-emotional learning and skills (SELS) policies, practices and technologies, this (lengthy) post summarizes some of the key aspects of SSES, detailing its policy context, the ways it will generate and use student data, its conceptual basis in psychometrics, and the ways the OECD frames it as an objective ‘policy-relevant’ science programme with positive social and economic outcomes for participating countries.

In recent years, the OECD’s PISA (Programme for International Student Assessment) and PIAAC (Programme for the International Assessment of Adult Competencies) tests have been the subject of extensive debate and research. New tests, such as the PISA-based Test for Schools to help schools compare themselves to international standards, as well as the expansion of its tests to include factors like problem-solving and well-being, have become available as the OECD has gradually extended its logic of measurement and comparison into policymaking systems globally.

The OECD first began signalling its interest in measuring and assessing social and emotional skills in 2014. That year, it published Fostering and Measuring Skills: Improving cognitive and non-cognitive skills to promote lifetime success, followed in 2015 by its report Skills for Social Progress: The power of social and emotional skills. In 2017 the OECD published Personality Matters: Relevance and assessment of personality characteristics, an extensive review of the scientific literature on personality theory and the measurement of personality factors. Although Personality Matters was developed as part of the PIAAC survey of adult skills, it has been deployed as the scientific rationale for the Study of Social and Emotional Skills announced in its 2017 ‘brochure’ Social and Emotional Skills: Well-being, connectedness and success and the accompanying SSES website.

Before going into some of the detail of SSES, the OECD’s focus in this area needs to be seen in a larger policy context. Over the last five years, as I’ve documented elsewhere, social-emotional learning and skills (SELS) have become a significant education policy priority and a key focus for education technology development and investment. Organisations including the global education business Pearson and the Nudge Unit have produced research summaries and guidance on developing SELS. The core idea behind many social-emotional learning and skills approaches is that the ‘non-cognitive’ aspects of learning are fundamentally linked to academic progress and to a range of social and economic outcomes, such as productivity, labour market behaviours and overall well-being.

Moreover, many advocates maintain, SELS are malleable and can be improved through direct teaching intervention. Improving SE skills is, therefore, seen as an important prerequisite for raising attainment, achieving social and economic progress, and improving individuals’ success, and an attractive prospect for policymakers seeking new ways to boost student achievement and employability.

Major lobbying groups based in the US have produced scientific justifications for focusing on SE learning and skills. The Collaborative for Academic, Social, and Emotional Learning (CASEL) has produced its own meta-analyses on social-emotional learning research and evidence. Similarly, the National Commission on Social, Emotional, and Academic Development (NCSEAD) at the Aspen Institute has published a research consensus drawing from evidence in brain science, medicine, economics, psychology, and education research. It claims to demonstrate that ‘the success of young people in school and beyond is inextricably linked to healthy social and emotional development, such as the ability to pay attention, understand and manage emotions, and work effectively in a team.’

Although CASEL and NCSEAD appear to have identified a consensus about what constitutes social-emotional learning and skills, the terminology remains confusing. Terms used for SELS including ‘character,’ ‘growth mindset,’ ‘grit,’ ‘resilience,’ and other ‘non-cognitive’ or ‘non-academic’ ‘personal qualities’ are often used interchangeably and gain traction with different academic, practitioner and policymaking communities. ‘Character’ has become the policy focus for the Department for Education in the UK following the 2014 publication of a cross-party Character and Resilience Manifesto,  while ‘grit’ has been favoured by the US Department of Education, as in its 2013 report Promoting Grit, Tenacity, and Perseverance—Critical Factors for Success in the 21st Century. Emerging education policies in the European Union appear to emphasize ‘soft skills’ as a category that encompasses SELS.

The OECD itself has adopted ‘social and emotional skills,’ or ‘socio-emotional skills,’ in its own publications and projects. This choice is not just a minor issue of nomenclature. It also references how the OECD has established itself as an authoritative global organization focused specifically on cross-cutting, learnable skills and competencies with international, cross-cultural applicability and measurability rather than on country-specific subject achievement or locally-grounded policy agendas.

Social-emotional datafication
The SSES programme was launched in 2017 with a timetable toward delivery of the first results in 2020. According to a published project schedule, during 2018 the OECD is developing the instruments and test items, before field testing in participating countries later in the year. The first formal round of the survey will take place in 2019, with final results released to the public late in 2020. The OECD also plans to administer SSES repeatedly in order to generate longitudinal data.

The expected outputs from the project include:

  • a set of validated international instruments to measure social and emotional skills of school-aged children
  • a dataset with information on the level and spread of social and emotional skills of children at ages 10 and 15, obtained from multiple sources, and accompanied with a wide scope of background and contextual variables
  • an improved understanding amongst policy-makers, education leaders, teachers, parents and other stakeholders on the critical role of social and emotional skills and the types of policies and practices that support the development of these skills
  • an improved understanding of whole child development, specifically as it relates to the development of social and emotional skills of children and youth

Although it is anticipated that only 10-12 countries and cities will take part in the first study, a huge quantity of data will be collected from the participants to deliver these outputs. The student survey of 10 and 15 year-olds itself consists of two assessments.

The first is a direct assessment. This is to be administered as a self-report questionnaire, which will be completed online as a computer-based survey. Students will respond to questions that are designed to assess behaviours considered indicative of selected SE skills. Indirect assessment will add to the dataset, including reports from parents and teachers based on similar questions on the typical behaviours of individual participating students.

In addition to the core assessment instruments will be contextual questionnaires for completion by children, their parents, teachers and principals. The contextual questionnaire for students will gather data on demographics, family culture, subjective health and well-being, academic expectations, and perception of their own SE skills.

Parents will provide information about their children’s SE skills, family background, child’s performance, home learning environment, parent-child-relationship, parental styles, learning activities, and parents’ own attitudes and opinion. As this list indicates, SSES is not just focused on school factors involved in developing SELS, but on distinctive family factors too, including learning activities undertaken out of school.

Schools themselves will provide contextual information from teachers in the form of reports on students’ SE skills, teachers’ own backgrounds, school characteristics, teaching practices, and teachers’ values and expectations about SE skills. Principals will add data on school background, school management, principles and rules, school climate, and the role of SE skills in curriculum and school agenda, as well as further administrative data for calculating other behavioural correlates and outcomes.

The data production expectations on schools, students and their families are, as the list demonstrates, extensive and extend well beyond the normal jurisdiction of the education sector into the extraction of information about homes, family relationships and parenting practices.

A further OECD document suggested it was also considering ‘exploring ways to link social and emotional skill measurement of the proposed study with other OECD measurement instruments such as those used in PISA and PIAAC, as well as with local measurement instruments such as standardised achievement tests.’

The direct assessment will be delivered online using a centralized software platform for assessment of children’s SE skills. Notably, the OECD claims it will use log file data obtained during the test as additional indicators of SE skills.

Log file information collected during computer-based international assessments has been described by Bryan Maddox as ‘process data’ collected about such things as  response times and key strokes, which can be studied with ‘micro-analytic precision’ in the analysis of larger-scale assessment data. These log file data are increasingly used in assessment software platforms as an extension of the test and can be conceptualized as ‘the mechanisms that underlie what people do, think, or feel, when interacting with, and responding to, the item or task.’

It’s not entirely clear how SSES will use log file data, but other projects have sought to correlate process metadata such as keystroke and response times to SELS. For example, the winner of CASEL’s 2017 design challenge on technologies to assess social-emotional learning was designed to capture the metadata generated as students took a computer-based test. It claimed its ‘measure quantifies how often students respond extremely quickly over the course of a test, which is strongly correlated with scores from measures of social-emotional learning constructs like self-regulation and self-management.’

This project exemplifies a form of stealth assessment whereby students are being assessed on criteria they know nothing about, and which rely on micro-analytics of their gestures across interfaces and keyboards. It appears likely that SSES, too, will involve correlating such process metadata with the OECD’s own SELS constructs to produce stealth assessments for quantifying student skills.

As the range of its data collection activities demonstrate, the OECD has designed SSES to include not only direct survey assessments, but extensive contextual information, school-level data on students behaviours and outcomes, and log file information that can be analysed as digital signals of SE skills. Importantly, though, these data all rely on specific conceptualizations of socio-emotional skills that the OECD has invested significant institutional effort in researching and defining.

Personality measurement
Behind the OECD SSES survey lies a set of psychological knowledge about the measurable qualities and characteristics of socio-emotional skills. The SSES brochure gives an overview of how the OECD defines SE skills. It claims socio-emotional skills constructs can be classified into five broad domains, which it refers to as a well-known framework called the ‘Big Five model’: emotional regulation (emotional stability); engaging with others (extroversion); collaboration (agreeableness); task performance (conscientiousness); open-mindedness (openness). The SSES survey itself will be administered to assess 19 skills which fit into each of these five categories.

The brochure notes the ‘five-factor structure of personality characteristics’ has been extensively researched and empirically validated in multiple studies, leading to ‘widespread acceptance of the model.’ It further adds that there is ‘extensive evidence that the Big Five domains and sub-domains can be generalised across cultures and nations’ and is suitable for describing socio-emotional skills in both children and adults.

OECD SELS categories

A much fuller account of the Big Five model is provided in Personality Matters, which the SSES brochure references directly. Written by an OECD policy analyst with academic experience in educational psychology, international social policy and cross-cultural survey methodology, Personality Matters is an extensive review of psychological and psychometric research on the conceptualization and measurement of human personality.

The document reviews research on a variety of potential measures of social-emotional learning and skills, including ‘grit,’ ‘character skills’ and other socio-emotional competencies (though it makes no reference to ‘growth mindset,’ a current popular psycho-policy concept). The review favours the five factor model of personality consisting of openness, conscientiousness, extroversion, agreeableness and neuroticism (OCEAN). It acknowledges that psychologists have developed tests and assessments such as the Big Five Inventory (BFI), the Neuroticism-Extraversion-Openness Personality Inventory (NEO-PI) the International Personality Item Pool (IPIP) and Trait Descriptive Adjectives (TDA) to measure these personality factors.

It is of course clear that the OECD’s SSES categories map exactly on to the five factor personality categories, with ‘emotional stability’ standing in for ‘neuroticism.’ The model was approved at an OECD meeting in 2015, following a presentation by Oliver John, a psychologist at the Berkeley University Personality Lab and original author of the Big Five Inventory personality test. Likewise, the SSES survey has been designed to assess 19 skills associated with its Big Five model, in ways which emulate the structure of many of the personality tests cited in its review. As this indicates, the way the OECD has formulated social and emotional skills is a direct translation of the OCEAN categories used by psychologists for personality testing. In this sense, SSES appears to represent a therapeutic shift in OECD focus, with its target being the development of emotionally stable individuals who can cope with intellectual challenge and real-world problems.

Crucially, the review also notes strong correlations between high scores in the Big Five and other outcomes such as academic achievement, job performance, and standardized test scores. Notably, it emphasizes the ‘policy relevance’ of the insight that many personality characteristics—or socio-emotional skills as the SSES describes them—are malleable and can therefore become a ‘potential target for policy intervention.’  Arguably, policy interventions that would be relevant to remedying identified personality problems would be forms of therapeutic provision, such as remedial classes in socio-emotional skills, which schools would then be responsible for delivering. ‘Therapeutic education,’ as Kathryn Ecclestone and Denis Hayes characterized it a decade ago, includes activities and underlying assumptions ‘paving the way for coaching “appropriate” emotions’ and ‘replace education with the social engineering of emotionally literate citizens who are also coached to experience emotional wellbeing.’

The combination of the human sciences with public policymaking has a long history, enabling governments to act upon the capacities of those subjects they govern. The promise of scientific objectivity regarding human behaviour and emotion is very attractive to policymakers wishing to manage or social-engineer the ‘appropriate’ behaviours of populations more effectively. However, as Sheila Jasanoff has argued, the objectivity of ‘policy-relevant knowledge’ is always achieved through hard work, argument, and the strategic deployment of persuasive, authoritative claims. The ‘objectivity-making’ practices of psychologists of personality underpin the recent policy shift to SELS embodied by the OECD, and the forms of therapeutic education that are likely to proliferate as schools realize they are to be measured and assessed according to SELS categories.

The OECD has made a science itself out of crafting objective policy-relevant knowledge from larger, contested bodies of scientific evidence about human competencies and personalities. With SSES, it either dismisses or omits the ‘grit,’ ‘growth mindset’ and ‘character skills’ literature—which in the Personality Matters report it suggests imply ‘moral connotations that many researchers and policy advisers would like to avoid’—and instead translates the concepts and practices of psychometric personality testing into policy-relevant approaches to measuring and assessing the social and emotional worlds of children.

Socio-emotional indicators and socio-economic outcomes
Beyond the presumed scientific objectivity of personality testing, interest in SELS among government departments and policymakers is also due at least in part to the economic arguments made by its advocates.

In the US, SELS are a lucrative investment opportunity under the banner of ‘impact investing.’ These ‘pay for success’ schemes allow investment banks and wealthy philanthropies to invest in educational services and programs and then collect public money with additional interest as profits if they meet agreed outcomes metrics.  The metrics for calculating the social benefit and monetary value of SELS schemes have  been published as a cost-benefit analysis with the title The economic value of social and emotional learning.

Beyond direct profitability of SELS programs for investors, however, the OECD makes a strong argument to governments that its assessment of socio-emotional skills can produce indicators of socio-economic outcomes. As such, it makes the case that government investment in SELS through departments of education will generate a substantial return in the shape of productive human capital. This is an argument the OECD has refined through years of PISA and PIAAC testing and analysis.

The Nobel laureate of economics James Heckman has advised the OECD on its social-emotional learning program through co-authoring its 2014 report on measuring non-cognitive skills. The report claimed that some programmes to support non-cognitive skills development ‘have annual rates of return that are comparable to those from investments in the stock market.’ Based on  extensive economics analysis twinned with developmental psychology and the neuroscience of ‘human capability formation,’ Heckman has influentially argued for over a decade that non-cognitive social-emotional skills and ‘personality factors are also powerfully predictive of socioeconomic success and are as powerful as cognitive abilities in producing many adult outcomes.’ Making ‘personality investments’ in young people, he claims, leads to high returns in labour market outcomes.

It’s notable that the organization contracted to lead SSES is the Center for Human Resource Research (CHRR) at The Ohio State University. The CHRR’s mission is to provide ‘substantive analyses of economic, social, and psychological aspects of individual labor market behavior to examining the impact of government programs and policies.’ According to the CHRR, the SSES project will identify ‘those social and emotional skills that are cross-cultural, malleable, measurable, and that contribute to the success and well-being of both the youth and their society.’ The assessment of SELS is therefore to be undertaken through the logic of human resource management and the analysis of labour market behaviours.

As the OECD itself phrases it, the purpose of SSES is to ‘provide participating cities and countries with robust and reliable information on the social and emotional skills of their students,’ and also to ‘have policy relevance’ by identifying ‘the policies, practices and other conditions that help or hinder the development of these critical skills.’ As a ‘policy-relevant project,’ the OECD claims, ‘study findings can also be used by policy makers to devise better policy instruments geared towards promoting these types of skills in students.’ Its brochure gives examples of ‘critical life outcomes’ that correlate with socio-emotional skills, including school achievement, college graduation, job performance, health and well-being, behavioural problems, and citizenship participation. These, it claims, can be improved because social and emotional skills are learnable and personality is malleable.

Psycho-economic policymaking & personality modification
This is necessarily a very partial overview of some of the key features of SSES. However, it does raise a few headline points:

  • SSES extends international-large scale assessment beyond cognitive skills to the measurement of personality and social-emotional skills as a way of predicting future economic and labour market outcomes
  • SSES will deliver a direct assessment instrument modelled on psychological personality tests
  • SSES enacts a psychological five-factor model of personality traits for the assessment of students, adopting a psychometric realist assumption that personality test data capture the whole range of cross-cultural human behaviour and emotions in discrete quantifiable categories
  • SSES extends the reach of datafication of education beyond school walls into the surveillance of home contexts and family life, treating them as a ‘home learning environment’ to be assessed on how it enables or impedes students’ development of valuable socio-emotional skills
  • SSES normalizes computer-based assessment in schools, with students required to produce direct survey data while also being measured through indirect assessments provided by teachers, parents and leaders
  • SSES produces increasingly fine-grained, detailed data on students’ behaviours and activities at school and at home that can be used for targeted intervention based on analyses performed at a distance by an international contractor
  • SSES involves linking data across different datasets, with direct assessment data, indirect assessments, school admninistrative data, and process metadata generated during assessment as multiple sources for both large-scale macro-analysis and fine-grained micro-analytics–with potential for linking data from other OECD assessments such as PISA
  • SSES uses digital signals such as response times and keystrokes, captured as process metadata in software log files, as sources for stealth assessment based on assumptions about their correlation with specific social-emotional skills
  • SSES promotes a therapeutic role for education systems and schools, by identifying ‘success’ factors in SELS provision and encouraging policymakers to develop targeted intervention where such success factors are not evident
  • SSES treats students’ personalities as malleable, and social-emotional skills as learnable, seeking to produce policy-relevant psychometric knowledge for policymakers to design interventions to target student personalities
  • SSES exemplifies how policy-relevant knowledge is produced by networks of influential international organizations, connected discursively and organizationally to think tanks, government departments and outsourced contractors
  • SSES represents a psycho-economic hybridization of psychological and psychometric concepts and personality measurement practices with economic logics relating to the management of labour market behaviours and human resources

There is likely to be additional concern that the OECD will use SSES to conduct large-scale international comparison of children’s social-emotional learning and skills. At present the first stage study appears too limited for that, with only an estimated 10-12 participating cities and countries.

However, over time SSES could experience function creep. PISA testing has itself evolved considerably and gradually been taken up in more and more countries over different iterations of the test. The new PISA-based Test for Schools was produced in response to demand from schools. Organizations like CASEL are already lobbying hard for social-emotional learning to be used as an accountability measure in US education—and has produced a State-Scan Scorecard to assess each of the 50 states on SEL goals and standards. Even if the OECD resists ranking and comparing countries by SELS, national governments and the media are likely to interpret the data comparatively anyway.

If these developments are taken as indicators, it is possible that over time the OECD may generate international comparisons, accountability metrics and league tables of education systems based on intimate assessments of students’ personalities.

Image credits: OECD
Posted in Uncategorized | Tagged , , , , , , , | 4 Comments

Mapping the data infrastructure of market reform in higher education

Ben Williamson

Cables_Thomas Williams

A new regulator for Higher Education in England came into legal existence on 1st January 2018. Announced as part of the 2017 Higher Education and Research Act (HERA), the Office for Students is already controversial before it formally begins operations in April. The appointment to its board of Toby Young, the free schools champion and journalist, appalled critics who vocally called for his sacking over previous misogynistic comments in the press and on social media. Despite Conservative Party ministers including Jo Johnson, Boris Johnson and Michael Gove defending his selection, Young resigned within 10 days.

The Toby Young storm, however, has distracted attention from one of the most significant aspects of the HE reforms the Office for Students will preside over. That is the escalation and acceleration in the collection, analysis and use of student data, and the building of a new HE data infrastructure to enact that task. Under the Office for Students, student data is to become a significant source for regulating the HE sector, as universities are put under increasing pressures of market reform, metrics and competition by HERA.

As with all data infrastructure, mapping the HE data infrastructure is a complex task. In this initial attempt to document it (part of a forthcoming paper), I am following Rob Kitchin’s call for case studies that trace out the ‘sociotechnical arrangements’ of people, organizations, policies, discourses and technologies involved in the development, evolution, influence, dead-ends and failures of data infrastructures. It is necessarily a very partial account of a much larger project to follow the development, rollout and upkeep of a new data infrastructure in UK HE, and to chart how big data, learning analytics and adaptive learning technologies are being positioned as part of this program to deliver a reformed ‘smart’ sector for the future.

Metrics, markets and HE reform
As will be familiar to many working within UK HE, the Higher Education and Research Act (HERA) came into effect in 2017. It is the result of governmental reforms to the sector that have been underway since the beginning of the decade, as detailed in government papers produced by the Department for Business, Innovation and Skills (BIS)—notably 2011’s Students at the Heart of the System and 2016’s Success as a Knowledge Economy. These reforms, as others have documented and debated, have unleashed a ‘metric tide’ of performance measurement across the sector in an effort to create a marketized system of mass higher education.

The creation of the Office for Students (OfS) as a new regulator for HE in England has now consolidated the metricization and marketization of the sector as demanded by HERA.

Described by BIS as a ‘non-departmental public body’ operating ‘at arm’s length from Government,’ the OfS is intended as an ‘explicitly pro-competition and pro-student choice’ organization, as well as a ‘consumer focused market regulator’ much like Ofcom in the media sector and Ofwat in water services. Its chair is Sir Michael Barber, formerly the ‘deliverology’ champion under Tony Blair’s government, a partner at management consultancy McKinsey, and more recently the chief education adviser at the global education business Pearson. At Pearson Barber oversaw its attempted transformation into a ‘digital-first’ company focused on digital data analytics and adaptive learning technologies for both the schools and universities sectors.

Barber was also the co-author of a 2013 report on the future of HE in the UK with the IPPR think tank, which argued:

With a massive diversification in the range of providers, methods and technologies delivering tertiary education worldwide, the assumptions underlying the traditional relationship between universities, students and local and national economies are increasingly under great pressure – a revolution is coming.

WonkHE named Barber the most powerful person in HE in 2017, noting his ‘legendary fondness for metrics,’ and the OfS began recruiting high-profile positions in data analytics ‘for those who understand the latest thinking in data science practice and how data can support policy development.’

The combination of BIS, HERA and Barber’s OfS represent the most visible aspects of current HE reform. As Neil Selwyn argues, ‘neoliberal logics’ of competition, performance measurement, quality management, marketization, commercialization and privatization have been growing in HE around the globe for many years, and are part of an increasingly powerful ideal of ‘digital universities’ which use data and metrics for monitoring performance and planning.  Underlying the reforms, however, is a less acknowledged project to upgrade and rebuild the data infrastructure that will allow the necessary information to be collected, analysed and put to use. The business of marketization in HE, as Janja Komljenovic and Susan Robertson have argued, involves ‘not just people, but technologies such as software, algorithms, computers, procedures and so on, in a rich collage of people, technology and programmes … that align the work of the university with the logics of capitalist markets.’

Data Futures
The enactment of HERA and the OfS will depend on massive quantities of data from universities, and also the utilization of new digital technologies to collect and process those data. In the last couple of years, as a recent Westminster event illustrates, government has begun to take more and more interest in ‘the role of big data and learning analytics for universities, including targeted marketing of prospective students, improving retention and personalising learning experience for individuals.’

Similarly, a collaboration between the Higher Education Commission and the think tank Policy Connect has produced From Bricks to Clicks: The potential of data and analytics in Higher Education to focus on the use of ‘fluid data’ that is ‘generated through the increasingly digital way a student interacts with their university.’ It highlights the potential of learning analytics to ‘improve the student experience at university, by allowing the institution to provide targeted and personalised support and assistance to each student.’ The government’s major 2017 Industrial Strategy also committed investment in big data, AI and machine learning within digital courses.

These future aspirations are beginning to be realized through the ‘Data Futures’ program being undertaken by the Higher Education Statistics Agency (HESA). Designated the official statistics and data body for HE since 1993, HESA compiles huge quantities of data about students, staff and institutions, departments, courses and finances, as well as performance indicators used to evaluate and compare providers. It maintains the data infrastructure for HE recording and reporting first established in its current form in 1994.

Data Futures is HESA’s flagship data infrastructure upgrade program, which it initiated as part of its corporate strategy in 2016, in response to government demands, and plans to operationalize fully by 2020. Funded with £7.5million from the HE funding councils, Data Futures is intended to enhance HE data quality, reduce duplication, and make data more useful and useable by members of the public, policymakers, providers, and the media. Its first priority is upgrading the systems for student data collection and analysis, in order to satisfy government demands that prospective students, as potential consumers of HE, can receive the best possible information about courses and providers, while current students might be able to monitor their own progress and rate the value-for-money provision they receive from their chosen courses.

The data infrastructure model being developed by HESA was first proposed through a series of reports by the global consulting firms Deloitte and KPMG, as part of the Higher Education Data and Information Improvement Programme (HEDIIP) hosted by HESA. In 2013 Deloitte produced ‘a proposal for a coherent set of arrangements for the collection, sharing and dissemination of data for the higher education data and information landscape.’ KPMG followed it with a ‘blueprint’ for a ‘New Data Landscape’, envisioned as ‘a data and information landscape for Higher Education in the UK that has effective governance and leadership, promotes data standards, rationalises data flows and maximises the value of technology and enables improved data capability.’

In the KPMG blueprint, used to establish Data Futures, HESA is positioned as a central ‘data warehouse’ for all HE data collection and access. Rather than once-a-year reporting, under Data Futures all HE providers will be required to conduct ‘in-year’ reporting. This will speed up the flow of data between institutions and HESA, and enable HESA to produce analyses and make them available to the public, media and policymakers more swiftly.

Writing in the Times Higher Education, HESA’s chief executive Paul Clark has described how the environment in which HE institutions operate is becoming increasingly data-intensive, with policymakers, students, funders and regulators all seeking information for their own purposes and needs:

Good data allow students to make informed choices, allow policymakers and regulators to make better decisions, promote public trust and confidence in the system, enable institutions to be competitive and provide a lever to incentivise or penalise behaviour in the absence of public funding.

At the same time as these ‘trends are being driven by developments in higher education policy,’ added Clark in another piece, ‘changes in the worlds of data, digital service delivery, and technology’ are taking place as big data technologies and practices are embedded across sectors and industries.

In this context, Data Futures instantiates, perhaps, a tentative first move toward the use of ‘live data’ and ‘real-time metrics’ which could be used for continuous performance measurement and comparison of the competitive HE market of providers. Although a real-time data model was suggested in the KPMG blueprint, Data Futures is not going quite that far, just yet. As Clark noted in his THE article,

Further developments can build out from this – providing enhanced analytical tools for users and providers, opening up larger stores of data for analysis and innovation, linking datasets across government departments and policy areas to improve decision-making and reducing the transactional costs associated with data flows around the sector.

While its first goal is to implement in-year reporting, HESA is clearly positioning itself to introduce advanced digital data methods and large linked datasets drawn from across government departments into some aspects of HE reporting and decision-making.

Data platforms and dashboards
Although Data Futures appears rather mundane as an effort to streamline and improve student data collection and analysis, it is an immense organizational and technical undertaking. At its core is the construction of a new ‘data platform’ for data collection, and new ‘data dashboards’ and visualization technologies to analyse the data and communicate results.

With regard to the data platform, HESA released a specification document in 2016 for potential suppliers. The specification reveals the platform would include a vast number of interconnected technical components, including three ‘user interfaces’: a data collection portal, an analytics portal and a governance portal. Behind these interface portals would be a range of ‘services,’ all underpinned by ‘human and machine readable specifications,’ a ‘logical model’ and ‘physical data model,’ a ‘unique student identifier lookup service,’ and a ‘reporting engine.’ The data platform would also include cloud storage, encryption, secure file transfer services, metadata, code, rules, data files, metrics, and specifications in terms of quality, reports, and data delivery, and more.

The appointed supplier to deliver the specification, announced in 2017, is Civica, which ‘provides a wide range of software, digital solutions and technology-based outsourcing’ for ‘organisations to improve and automate the provision of efficient, high quality services, and to transform the way they work in response to a rapidly changing and increasingly digitalised environment.’ In its announcement of the contract, HESA said Civica would deliver an ‘improved data model and extended capabilities [which] will offer users of HESA data a regular flow of accessible information through an enhanced user interface and visualisation tools.’

Data dashboard development to communicate findings from the data platform is being undertaken through a collaboration between HESA itself and Jisc (Joint Information Services Committee) as part of their ‘business intelligence shared service.’ The Analytics Labs collaboration provides an agile data processing environment using ‘advanced education data analytics’ in order ‘to rapidly produce analyses, visualisations and dashboards for a wide variety of stakeholders to aid with decision making.’ It emphasizes ‘cutting-edge data manipulation and analysis,’ access to current and historic data for time series analysis of the sector, and competitor benchmarking, using the Heidi Plus software platform provided by the commercial supplier Tableau Server.

Notably, early in 2018 HESA signed an agreement with both The Guardian and The Times newspapers to use Heidi Plus to produce interactive HE dashboards of rankings and measures based on their league tables. This, claimed HESA, would ‘enable universities to accurately and rapidly compare and analyse competitor information at provider and subject level, changes in rank year on year,’ and ‘the highest climbers and the biggest “fallers.”’ It also noted that the dissemination and presentation of league table data help shape public opinion about different providers.

The data platform being built by Civica, twinned with dashboards produced using Heidi Plus, are therefore interfaces to the data infrastructure being built by HESA. Together, these software platforms enable student data to be collected, analysed, visualized and circulated to the public, the press, providers themselves, and policymakers and politicians—and in so doing, to shape opinion and influence decision-making. As such, these systems of measurement and visualization are intimately tied to political ambitions to subject HE to increased marketization and competition.

Measurement and visualization are never simply neutral representations. As David Beer has argued, ‘systems of measurement become powerful’ as ‘mechanisms of competition,’ especially when the results of those measurements are disseminated and circulated to circumscribe future possibilities. Moreover, Jamie Bartlett and Nathaniel Tkacz have described how data dashboards ‘bring about a new “ambience of performance”, whereby members of staff or the public become more attuned to how whatever is measured is performing.’ Metrics and their graphical representation shape how people and institutions think, behave, compare themselves, and act to change themselves based on those representations.

The metrics operating inside the Data Futures platform and dashboards, in this sense, are mechanisms of competition and performance measurement within the HE sector, acting to make marketization part of the ambient conditions of Higher Education. They  extend governmental ambitions around increased metricization and marketization through the computer interface and into the eyes, hands and decisions of university administrators and leaders. As HESA’s Andy Youell has written, the work to build a new HE information landscape is not so much about systems but about ‘changing behaviours’ and ‘changing attitudes to the value of data within institutions.’

A market of smart, connected universities
The utopian dream behind current efforts to reform HE, of which Data Futures provides the material infrastructure, is that marketization and competition will drive up innovation within universities. An innovative HE sector is at the forefront of the 2017 Industrial Strategy. The ‘winners’ will be those institutions able to provide measurable evidence of innovation and quality in research, teaching, and impact on society. Student data, in addition to ratings of research excellence, teaching excellence and knowledge exchange, are to become the measure of market competitiveness.

But Data Futures also signals a more long-term utopian vision of the future of the sector. In a 2016 article, HESA’s chief executive Paul Clark wrote in visionary terms of a future HE in which learning analytics are part of teaching intervention, research performance metrics are pervasive, policymakers routinely base decisions on sector data, and ‘universities are gathering and benchmarking more and more data to ensure they are operating efficiently, and competing effectively with their peers’:

In ten years’ time, it’s possible to envisage a digital HE sector, with data-driven universities operating within a smart, connected environment. In this vision, universities would routinely use data drawn from many sources and devices to design and deliver their services, allocate resources, and monitor their performance. Policy-makers would similarly pool data from across government and the public sector to design interventions, monitor progress, and gain a far better and more granular understanding of how policy should be designed and delivered in order to achieve their aims. And users of the system would be able to access critical real-time data and information on their own progress, the resources available to them, and what they can do to maximise their chances of success.

Although Data Futures alone won’t deliver this vision when it rolls out nationwide in 2020, it is building the infrastructure necessary for new data-driven universities of the imagined future.

Late in 2017, in fact, HESA ran the Data Matters conference for HE data practitioners and leaders, where it not only showcased Data Futures developments but also hosted speakers and panels on the latest developments in big data, learning analytics, data dashboards and visualization. Data Futures is establishing the necessary infrastructure to support the application of these technologies as part of a utopian vision of smart, connected universities driven by market demands and real-time metrics. It is also worth noting that there are similar ambitions to create a national student data network in the US, through the campaigning of the Postsecondary Data Collaborative and Gates Foundation funding.

Much more remains to be done to unpack and understand these new infrastructure projects and the implications of accelerating and expanding student data collection and analysis. This post simply represents a first attempt to map some aspects of the new data infrastructure that will orchestrate aspects of UK HE in years to come. It’s an infrastructure made up of policies, people, organizations and future imaginings as much as technologies, and an ideal illustration of a dynamic sociotechnical network in which politics, practices and technologies interpenetrate one another. It also highlights how governance of HE is extended across software companies, think tanks and consultancies, as well as government departments, public bodies and arms-length agencies, and of how software can act as a relay of government strategy into university planning.

As part of the regulatory apparatus for a reformed HE, controversial appointees to the board of the OfS have understandably generated concern. But they should not distract attention from current ongoing infrastructural work to establish a new digital architecture for Higher Education that universities, staff and students will inhabit for decades.

Image by Thomas Williams
Posted in Uncategorized | Tagged , , , , | Leave a comment

The Nudge Unit, data science and experimental education

Ben Williamson

light graph victor

The UK government’s Behavioural Insights Team has announced it has been experimenting with data science methods in school inspections. In partnership with the Office for Standards in Education (Ofsted), it has designed machine learning algorithms to predict and rate school performance.

Originally established as part of the Cabinet Office in 2010, the Behavioural Insights Team—or ‘Nudge Unit’ as it informally known—became a ‘social purpose company’ in 2014, jointly owned by the UK Government, the innovation charity Nesta, and its employees. Its staff have academic grounding in economics, psychology or a background in government policymaking, and it has expanded its office from London to Manchester, New York, Singapore and Sydney. It has always been closely associated with the ‘what works’ model of policy, employing randomized control trials (RCTs) to test out policy interventions. The BIT has also established a revenue-generating arm that uses ‘behavioural insights to design and build scalable products and services that have social impact.’ One of its advisory panel is Richard Thaler, the behavioural economist recently awarded a Nobel economics prize for his work on the application of behavioural science and ‘nudge’ techniques in public policy.

The BIT’s Data Science Team published details of its experiments in a new report in December 2017. The team’s defined aims are to ‘make use of publicly available data, web scraped data, and textual data, to produce better predictive models to help government’; ‘to test the implications of these models using RCTs’; and ‘to begin developing tools that would allow us to put the implications of our data into the hands of policymakers and practitioners.’

The report, Using Data Science in Policy, detailed a number of projects the Data Science Team had undertaken to apply behavioural insights to diverse areas of public policy:

over the past year we have been working to conduct rapid exemplar projects in the use of data science, in a way that produces actionable intelligence or insight that can be used not simply as a tool for understanding the world, or for monitoring performance, but also to suggest practical interventions that can be put into place by governments.

The experiments were in policy areas including health, social care and education.

School-evaluating algorithms
In its education project with Ofsted, the BIT described how it used ‘publically available datasets to predict which institutions are most likely to fail and thereby target their inspections accordingly. We showed that this data, married to machine learning techniques such as gradient boosted decision trees, can significantly outperform both random and systematic inspection targeting. … We are excited to be working with Ofsted to put the insights from this work into action.’

In order to apply data science and machine learning to school inspection, the BIT compiled publicly available data from the year before an inspection happened. These data, its report said, included workforce data, UK census and deprivation data from the local area, school type, financial data (sources of finance and spending), performance data (Key Stage 2 for primary schools and Key Stages 4 and 5 for secondary schools) and Ofsted Parent View answers to survey questions. Parent View is Ofsted’s online tool to allow parents to record their own views on their child’s school. These data are then considered in Ofsted inspections.

According to a report of the Ofsted experiment in Wired magazine, its ‘school-evaluating algorithm pulls together data from a large number of sources to decide whether a school is potentially performing inadequately.’ By matching statistical data to the Parent View data, which includes textual information that can be analysed for sentiment, BIT claims it can predict which schools are not performing well and are likely to fail an inspection. The system ‘can help to identify more schools that are inadequate, when compared to random inspections’ and may even be used to automate decisions made by Ofsted in the future.

So far, the Nudge Unit’s trial with Ofsted has not been used to inform any real-world decisions, although the two organizations plan to extend their partnership in 2018, and are considering the use of further datasets, including data that are not open to the public.

An important aspect of the experiment with Ofsted is that the BIT doesn’t want schools to know how the algorithm works, as the project’s director told Wired. ‘The process is a little bit of a black box—that’s sort of the point of it,’ he said. In other words, schools are to be kept in the dark about the school-evaluating algorithm so that they don’t have the opportunity to ‘game’ their data in advance, which would result in skewing the predictive model.

It’s not the first time the Nudge Unit has been involved in education in the UK. Earlier in 2017 it was reported that the Department for Education was recruiting a permanent behavioural insights manager and an adviser. The aim was to change the culture of the department with psychology specialists applying behavioural science in strategic policymaking processes, and to commission research, trials and interventions drawing on behavioural insights to ‘improve our education and children’s services’.

The Nudge Unit’s experiment with Ofsted, and the DfE’s recruitment of behavioural scientists, exemplify the increasing role of behavioural science agencies to produce policy-relevant science in public education in recent years, as Alice Bradbury, Ian McGimpsey and Diego Santori have previously documented. This raises a number of issues.

Data labs
The first issue is how public education is increasingly being influenced by arms-length agencies. As a co-owned entity of the Cabinet Office and Nesta, the Nudge Unit is strictly independent of government but now acting as an outsourced contractor within public policy. It was reported in Wired that if Ofsted rolls out the school evaluation algorithm developed by BIT, local authorities would be required to pay between £10,000-£100,000 to implement.

Nesta itself, the co-owner of BIT, is an often-overlooked organization in UK education. As a ‘policy innovation lab’ it has successfully campaigned for ‘coding’ to be included in the English National Curriculum and for data science to be applied in the analysis of public services. A core hub in a global network of policy labs, Nesta and similar organizations worldwide are seeking to innovate in public policy, often using technological innovations as models for government reforms.

As the US GovLab has reported, the application of data science in public policy by ‘data labs’ can help create a ‘smarter state.’ Indeed, Nesta and the Cabinet Office have previously collaborated to develop ideas about a ‘new operating system for government,’ using  data science, predictive analytics, artificial intelligence, sensors, autonomous machines, and platforms to redefine the role of government.

As such, organizations such as Nesta and the Nudge Unit, which perceive data science as a new model for enacting government, are now seeking to locate data science methods within the institutions and processes of educational policymaking and school evaluation. The Ofsted project is part of their wider ambitions around digital governance using data science to drive policymaking. They are seeking to attach arms-length ‘data labs’ to centres of public policy, bringing new forms of technical and statistical expertise—as well as economic, behavioural and psychological science—into policy processes, including education. This exemplifies what I have elsewhere described as ‘digital education governance’—the use of digital data to make education visible for inspection and intervention.

Inspecting algorithms
Second, as part of this shift, the Nudge Unit is seeking to transform the way school inspections are performed. Rather than inspection through embodied expertise, school evaluation is now to be enacted predictively, before the inspector arrives. Jenny Ozga has previously written of how digitally recorded data increasingly surrounds the inspection process. The Nudge Unit is seeking to pre-empt the inspection process through the application of machine learning algorithms which have been trained to spot patterns and make predictions from pulling together a wide range of multimodal data sources about schools and their contexts.

These deliberately ‘black-boxed’ and opaque systems, which schools would be unable to understand, could be significant actors in practices of school accountability. If, as anticipated, some of Ofsted’s tasks are automated by the Nudge Unit’s intervention, then it may be unclear how certain decisions have been made in relation to a school’s overall evaluation. Although the BIT claims it doesn’t wish to replace the professional inspector, it is clear that school inspection will become a more distributed task involving both human and nonhuman decisionmaking and judgment, with data science methods perceived as more objective and impartial means for producing evidence than professional observation. In this sense, it is entirely consistent with behavioural science claims that human decision-making is less rational and evidence-based–and more emotionally-charged, cognitively-biased and subjective–than is commonly assumed.

At a time when there is increasing political, public and legal concern about machine learning opacity and its lack of ‘explainability’ or transparency, it seems ethically questionable to create systems that are deliberately black boxed, not least as their algorithms may well contain biases and potential for statistical discrimination. The cognitive bias of the school inspector is to be combated with systems that may have their own encoded biases. If a school is predicted to be inadequate by the algorithm, its stakeholders will expect and need to know what factors and calculations produced that evaluation.

It is notable too that BIT claims ‘missing data’ is predictive of a failed inspection, presumably the consequence of human error in the data-inputting process, and that it is seeking other non-public data sources to improve its predictive models. It remains unclear how deeply BIT intends to scrape schools for data, or which additional data would be included in their calculations, raising methodological questions about reliability and commensurability of their analyses.

Behavioural government
The third issue relates to the application of behavioural science within education. Mark Whitehead and coauthors describe how ‘behavioural government’ has proliferated across public policy in many countries in recent years—especially the UK and US—through the application of ‘nudge’ strategies. Nudging involves the design of ‘choice architectures’ that can shape and condition choices, decisions and behaviours, and is deeply informed by behavioural and psychological sciences. The Nudge Unit exemplifies behavioural government.

In its project with Ofsted, the BIT is seeking to use data science as a way of constructing choice architectures for inspectors. The results of the data analyses can identify particular areas for concern, as predicted by the algorithm, that may then be targeted by the inspectors, thus creating a more efficient and cost-effective machinery of inspection. The BIT is, in effect, nudging Ofsted to make strategically informed choices about how to conduct inspection. This, claims BIT, would reduce the number of inspections required and free up Ofsted staff to work on improvement interventions with schools. (Though it might also lead to Ofsted staff reduction and cost-savings.) In these ways, the Ofsted inspector is being reimagined as a nudge operative, intervening in schools by offering them targeted improvement frameworks. At the same time, it is seeking to supplement subjective human judgment, with all the flaws that behavioural science claims comes with it, with algorithmic objectivity.

The Nudge Unit also makes extensive use of psychological insight. Perhaps the most obvious use of psychological data in the Nudge Unit’s project with Ofsted is the sentiment analysis it is performing on Parent View data, with aggregation of parents’ subjective feelings into patterns that can be used as objective indicators to supplement the statistical inspection.

More innovative sources of psychological data, however, could be used by the Nudge Unit to undertake algorithmic school inspections in the future.

Behavioural sciences have already amalgamated with data science in relation to the policy area of ‘social-emotional learning.’ The logic of the social-emotional learning movement is that ‘non-academic’ qualities are strongly linked to academic outcomes; students need to be trained to be socially and emotionally resilient if they are to succeed in school. There are close ties between many of the major voices in the social-emotional learning movement and the behavioural sciences, and educational technologies have become central to efforts to monitor and nudge students’ non-academic learning.

As I’ve documented elsewhere, a range of technical innovations to support social-emotional learning has been proposed and developed, such as behaviour monitoring apps and wearable biometric monitors, that might be able to detect indicators of student emotions such as engagement, attention, frustration and anxiety. Data from these devices could then be fed back to the teacher, who would be able to prompt the student in ways that might generate a more positive response.

Real-time student data, it is possible to speculate, could well become part of the school inspection process under a Nudge Unit style of data scientific experimentation. Student sentiment data, tracked to progress and attainment, would then become a measure, defined by behavioural economists, to be used for purposes of school accountability

Real-time psychological data, as well as more mundane user data scraped from the web or captured by mobile smartphones, argue Mark Whitehead and colleagues, now appear to present rich opportunities for behavioural scientists to both record and nudge behaviours and emotions.  They cite an article from Forbes claiming ‘the proliferation of connected devices—smartphones, wearables, thermostats, autos—combined with powerful and integrated software spells a golden age of behavioral science. Data will no longer reflect who we are—it will help determine it.’

Behavioural government is, then, informed by the testing culture of the tech sector, which constantly experiments on its users to see how they respond to small changes in design. As such, ‘behavioural design’ is the application of behavioural science to technological environments to influence and determine user behaviours. With increased behavioural science influence in education, twinned with massive escalation in data-processing ed-tech applications, the culture of testing and behavioural design could significantly impact on policy, schools and professional practitioners in years to come.

The education experiment
As with its work in other sectors, the Nudge Unit’s involvement with Ofsted and the Department for Education is bringing the methodological logic of data-driven experimentation and behavioural design into education. Increasingly, automated algorithms are being trusted to perform tasks previously undertaken by embodied professionals. Their opacity makes the decisionmaking these systems perform difficult to scrutinize. Ofsted has long been a source of concern for schools, of course. It is hard to see how transforming the inspector into an algorithm which is better at identifying more inadequate schools will reduce teachers’ worries about performance measurement. Data and the culture of performativity in education have a long history.

More generally, the application of data science to education policy is indicative of how the education sector itself is becoming the subject of increasing levels of experimentation with data science methods. The Department for Education is currently seeking to reintroduce baseline testing into pre-school settings. A previous trial of early years baseline testing in 2015 collapsed amid concerns over the methodology of the original contractor. In the tender for the second version of baseline assessment, however, the DfE has more carefully specified the testing methods it expects the contracted assessment company to use. In an experimental education sector, schools, professionals and students look increasingly like laboratory specimens, repeatedly subjected to tests and trials, inspections and interventions, as part of the pursuit of identifying ‘what works’ in education policy.

Image by Victor
Posted in Uncategorized | Tagged , , , , , | 4 Comments

Big Data in Education book launch

Ben Williamson

I was asked to prepare a few things to say to launch my book Big Data in Education: The digital future of learning, policy and practice in the Faculty of Social Sciences at the University of Stirling. Below are my notes.

Big data book cover

Seven years ago, like many other well-off parents in San Francisco, Max Ventilla was scoping out local schools. What he saw appalled him. State education, he concluded, was dangerously broken; a new model of school was required.

So he got a load of ‘progressive’ education literature about the state of American education, child-centred learning, school accountability, education technology and school design. He quit his job at Google, where he ran projects using big data to profile its millions of users, to set up his own new school.

AltSchool, as he called it, would be a ‘lab school’ combining child-centred progressivism with big data methods to deliver ‘personalized education’.

Max called venture capitalists he knew in Silicon Valley. 33 million dollars later, he hired teachers, managers—and a team of data analysts and software engineers to work on a ‘new operating system for education.’

Mark Zuckerberg of Facebook, among other investors, gave Max another 100 million for more AltSchools.

The tech and business press went wild. The Financial Times called AltSchool an example of ‘Silicon Valley’s classrooms of the future.’

Then Max revealed what his engineers were up to.

They’d built a software platform that could crunch data about almost everything students did. Student work could be uploaded to the system. Teachers’ responses would be logged. This all fed into a ‘Progress’ app—a ‘data dashboard’ displaying the progress students were making in academic learning and social-emotional development.

A ‘Playlist’ app was developed to recommend personalized tasks for students based on analysis of their past performance and predictions of their likely future progress.

Then AltSchool revealed it had cameras everywhere, tracking every movement and gesture of each student to assess engagement and attention.

Critics started to call it a ‘surveillance school’—using students as ‘guinea pigs’ for experimental data analytics. But Max and his investors, wanted it to scale-up across state education, to make more schools look like AltSchools.

Max had figured out a business model to satisfy investors. The AltSchool software platform would be offered for sale to all schools, starting in 2019. Meanwhile, last month Max shut down two of his lab schools, with three more to close in spring.

With the experimental beta-testing over, now Max and donors such as Mark Zuckerberg want to install the laboratory in every school.

AltSchool is prototypical of big data in education, and highlights a number of themes explored in the book.

So this book is about how educational data are produced and for what purposes, and about the technologies and companies that generate and process it.

And it’s about fantasy. A ‘big data imaginary’ of education is not just hype dreamt up in Silicon Valley, but a normative vision of education for the future shared by many. It has a seductive new data discourse of ‘personalization,’ ‘adaptive learning,’ ‘student playlists,’ ‘learning analytics,’ ‘computer-adaptive testing,’ ‘data-enriched assessment,’ and even ‘artificial intelligence tutors.’

It’s about ‘evidence-based’ education policy—that data analytics can provide real-time diagnostics and feedback at state, school, class and student levels—and commercial lobbying, venture capital and new forms of corporate philanthropy too, with ed-tech trying to capture public education for profit while attracting policymakers to their persuasive ideas.

It’s about science, with psychological, cognitive and neuro-scientists becoming expert in the experimental uses of student data.

And it’s about challenges to education research. Education research usually deals with human learning within social institutions, but now nonhuman ‘learning machines’ that can learn from and feedback to their human companions are starting to inhabit learning spaces as well. Some social science education researchers feel under threat from ‘education data science’ too.

Finally, the book is about power and the everyday ‘public pedagogies’ that teach lessons to millions globally, not just in educational institutions. Social media’s trending algorithms and filters direct attention to current events, politics, culture, and more, based on calculations of what you might like, what you’ve done, who you know. Tastes are being shaped, opinions and sentiments tweaked, and political views targeted and entrenched by political bots and computational propaganda. The power of big data in education extends beyond school to these public pedagogies of mis-education too.

Posted in Uncategorized | Tagged | Leave a comment

Learning machines

Ben Williamson

TerryKimura_facial recognition

When educators talk about theories of learning they are normally referring to psychological conceptions of human cognition and thinking. Current trends in machine learning, data analytics, deep learning, and artificial intelligence, however, complicate human-centred psychological accounts about learning. Today’s most influential theories of learning are those that apply to how computers ‘learn’ from ‘experience,’ how algorithms are ‘trained’ on selections of data, and how engineers ‘teach’ their machines to ‘behave’ through specific ‘instructions.’

It is important for education research to engage with how some of its central concerns—learning, training, experience, behaviour, curriculum selection, teaching, instruction and pedagogy—are being reworked and applied within the tech sector. In some ways, we might say that engineers, data scientists, programmers and algorithm designers are becoming today’s most powerful teachers, since they are enabling machines to learn to do things that are radically changing our everyday lives.

Can the field of social scientific educational research yet account for how its core concerns have escaped the classroom and entered the programming lab—and, recursively, how technical ‘learning machines’ are re-entering classrooms and other digitized learning environments?

Non-human machine learning processes, and their effects in the world, ought to be the object of scrutiny if the field of education research is to have a voice with which to intervene in the data revolution. While educational research from different disciplinary perspectives has long fought over the ways that ‘learning’ is conceptualized and understood as a human process, we also need to understand better the nonhuman learning that occurs in machines. This is especially important as machines that have been designed to learn are performing a kind of ‘public pedagogy’ role in contemporary societies, and are also being pushed in commercial and political efforts to reform education systems at large scale.

Algorithmic autodidacts
One of the big tech stories of recent months concerns DeepMind, the Google-owned AI company pioneering next-generation machine learning and deep learning techniques. Machine learning is often divided into two categories. ‘Supervised learning’ involves algorithms being ‘trained’ on a selected dataset in order to spot patterns in other data later encountered ‘in the wild.’ ‘Unsupervised learning,’ by contrast, refers to systems that can learn ‘from scratch’ through immersion in data.

In 2016 DeepMind demonstrated AlphaGo, a Go-playing AI system that learned in a supervised way from a training dataset of thousands of games played by professionals and accomplished amateurs. Its improved 2017 version, AlphaGo Zero, however, is able to learn without any human supervision or assistance other than being taught the rules of the game. It simply plays the game millions of times over at rapid speed to work out winning strategies.

In essence, AlphaGo Zero is a self-teaching autodidactic algorithmic system.

‘It’s more powerful than previous approaches because by not using human data, or human expertise in any fashion, we’ve removed the constraints of human knowledge and it is able to create knowledge itself,’ said AlphaGo’s lead researcher in The Guardian.

At the core of AlphaGo Zero is a training technique that will sound familiar to any education researchers who have encountered the psychological learning theory of ‘behaviourism’—the theory that learning is an observable change in behaviours that can be influenced and conditioned through reinforcements and rewards.

Alongside neural network architecture, a cutting-edge ‘self-play reinforcement learning algorithm’ is AlphaGo Zero’s primary technical innovation, It is ‘trained solely by self-play reinforcement learning, starting from random play, without any supervision or use of human data,’ as its science team described it in Nature. Its ‘reinforcement learning systems are trained from their own experience, in principle allowing them to exceed human capabilities, and to operate in domains where human expertise is lacking.’ As the reinforcement algorithm processes its own experiences in the game, it is ‘rewarded’ and ‘reinforced’ by the wins it achieves, in order ‘to train to superhuman level.’

Beyond being a superhuman learning machine in itself, however, AlphaGo may also be used ‘to help teach human AlphaGo players about additional, “alien” moves and stratagems that they can study to improve their own play,’ according to DeepMind’s CEO and co-founder Demis Hassabis. During testing, AlphaGo Zero was able not just to recover past human knowledge about Go, but also to produce new knowledge based on a constant process of self-play reinforcement.

The implication, in other words, is that powerful learning algorithms could be put to the task of training better humans, or even of outperforming humans to solve real-world problems.

The computing company IBM, which has also piled huge effort and resources into ‘cognitive computing’ in the shape of IBM Watson, has applied similar claims in relation to the optimization of human cognition. Its own cognitive systems, it claims, are based on neuroscientific insights into the structure and functioning of the human brain—as Jessica Pykett, Selena Nemorin and I have documented.

‘It’s true that cognitive systems are machines that are inspired by the human brain,’ IBM’s senior vice-president of research and solutions has argued in a recent paper. ‘But it’s also true that these machines will inspire the human brain, increase our capacity for reason and rewire the ways in which we learn.’

DeepMind and IBM Watson are both based on scientific theories of learning—psychological behaviourism and cognitive neuroscience—which are being utilized to create ‘superhuman’ algorithmic systems of learning and knowledge creation. They translate the underlying theories of behaviourist psychology and cognitive neuroscience into code and algorithms which can be trained, reinforced and rewarded, and even become auodidactic self-reinforcing machines that can exceed human expertise.

For educators and researchers of education this should raise pressing questions. In particular, it challenges us to rethink how well we are able to comprehend processes normally considered part of our domain as they are now being refigured computationally. What does it mean to talk about theories of learning when the learning in question takes place in neural network algorithms?

‘Machine behaviourism’ of the kind developed at DeepMind may be one of today’s most significant theories of learning. But because the processes it explains occur in computers rather than in humans, education research has little to say about it or its implications.

Developments in machine learning, autodidactic algorithms and self-reinforcement processes might enlarge the scope for educational studies. Cognitive science and neuroscience already embrace computational methods to understand learning processes—in ways which sometimes appear to reduce the human mind to algorithmic processes and the brain to software. IBM’s engineers for cognitive computing in education, for example, believe their technical developments will inspire new understandings of human cognition.

A social scientific approach to these computational theories of learning will be essential, as we seek to understand better how a population of nonhuman systems is being trained to learn from experience and thereby learning to interact with human learning processes. In this sense, the models of learning that are encoded in machine learning systems may have significant social consequences. They need to be examined as closely as previous sociological studies have examined the expertise of the ‘psy-sciences’ in contemporary expressions of authority and management over human beings.

Public hypernudge pedagogy
The social implications of machine learning can be approached in two ways requiring further educational examination. The first relates to how behavioural psychology has become a source of inspiration for social media platform designers, and how social media platforms are taking on a distinctively pedagogic role.

Most modern social media platforms are based on insights from behaviour change science, or related variants of behavioural economics. They make use of extensive data about users to produce recommendations and prompts which might shape users’ subsequent experiences. Machine learning processes are utilized to mine user data for patterns of behaviours, preferences and sentiments, compare those data and results with vast databases of other users’ activities, and then filter, recommend or suggest what the user sees or experiences on the platform.

Machine learning-based data analytics processes have, of course, become controversial following  news about psychological profiling and microtargeting via social media during elections—otherwise described as ‘public opinion manipulation’ and ‘computational propaganda.’ The field of education needs to be involved in this debate because the machine learning conducted on social media performs the role of a kind of ‘public pedagogy’—that is, the lessons taught outside of formal educational institutions by popular culture, informal institutions, public spaces, dominant cultural discourses, and both the traditional and social media.

The public pedagogies of social media are significant not just because they are led by machine learning, though. They are also deeply informed by psychology, and specifically by behavioural psychology. The behavioural psy-sciences are today deeply involved in defining the nature of human behaviours through their disciplinary explanations, and in informing strategic commercial and governmental aspirations.

In Neuroliberalism, Mark Whitehead and coauthors suggest that big data software is being regarded as spelling a ‘golden age’ for behavioural science, since data will be used not just to reflect the user’s behaviour but to determine it as well. At the core of the social media and behavioural science connection are the psychological ideas that people’s attention can be ‘hooked’ through simple psychological tricks, and then that their subsequent behaviours and persistent habits can be ‘triggered’ through ‘persuasive computing’ and ‘behavioural design.’

Silicon Valley’s social media designers know how to shape behaviour through technical design since, according to Jacob Weisberg, ‘the disciplines that prepare you for such a career are software architecture, applied psychology, and behavioral economics—using what we know about human vulnerabilities in order to engineer compulsion.’ Weisberg highlights how many of Silicon Valley’s engineers are graduates of the Persuasive Computing Lab at Stanford University, which uses ‘methods from experimental psychology to demonstrate that computers can change people’s thoughts and behaviors in predictable ways.’

Behaviourist rewards—or reinforcement learning—is important in the field of persuasive computing since it compels people to keep coming back to the platform. In so doing, they generate more data about themselves, their preferences and behaviours, which can then be processed to make the platform experience more rewarding. These techniques are, in turn, interesting to behaviour change scientists and policymakers because they offer ways of triggering certain behaviours or ‘nudging’ people to make decisions within the ‘choice architecture’ offered by the environment.

Karen Yeung describes the application of psychological data about people to predict, target and change their emotions and behaviours as ‘hypernudging.’ Hypernudging techniques make use of both persuasive computing techniques of hooking users and of behavioural change science insights into how to trigger particular actions and responses.

‘These techniques are being used to shape the informational choice context in which individual decision-making occurs,’ argues Yeung, ‘with the aim of channelling attention and decision-making in directions preferred by the “choice architect”.’

Through the design of psychological nudging strategies, digital media organizations are beginning to play a powerful role in shaping and governing behaviours and sentiments.

Some Silicon Valley engineers have begun to worry about the negative psychological and neurological consequences of social media’s ‘psychological tricks’ on people’s attention and cognition. Silicon Valley has become a ‘global behaviour-modification empire,’ claims Jaron Lanier. Likewise, AI critics are concerned that increasingly sophisticated algorithms will nudge and cajole people to act in ways which have been deemed most appropriate—or optimally rewarding—by their underlying algorithms, with significant potential social implications.

Underpinning all of this is a particular behaviourist view of learning which holds that people’s behaviours can be manipulated and conditioned through the design of digital architectures. Audrey Watters has suggested that behaviourism is already re-emerging in the field of ed-tech, through apps and platforms that emphasize ‘continuous automatic reinforcement’ of ‘correct behaviours’ as defined by software engineers. In both the public pedagogies of social media and the pedagogies of the tech-enhanced classroom, a digital re-boot of behaviourist learning theory is being put into practice.

Behavioural nudging through algorithmic machine learning is now becoming integral to the public hypernudge pedagogies of social media. It is part of the instructional architecture of the digital environment that people inhabit in their everyday lives, constantly seeking to hook, trigger and nudge people towards particular persistent routines and to condition ‘correct’ behavioural habits that have been defined by platform designers as preferable in some way. Educational research should engage closely with the public hypernudge pedagogies that occur when the behavioural sciences combine with the behaviourism of algorithmic machine learning, and look more closely at the underlying behavioural science theories of learning on which they are based and the behaviours they are designed to condition.

Big Dewey
The second major set of implications of machine learning relates to the uptake of data-driven technologies within education specifically. Although the concept of ‘personalized learning’ has many different faces, its dominant contemporary framing is through the logic of big data analytics. Personalized learning has become a powerful idea for the ed-tech sector, which is increasingly influential in envisioning large-scale educational reform through its adaptive platforms.

Personalized learning platforms usually consist of some combination of data-mining, learning analytics, and adaptive software. Student data are collected by such systems, then compared with an ideal model of student performance, in order to generate predictions of likely future progress and outcomes, or adapt responsively to meet individual students’ needs as deemed appropriate by the analysis.

In short, personalized learning depends on autodidactic machine learning algorithms being put to work to mine, extract and process student data in an automated fashion.

The discourse surrounding personalized learning frames it as a new mode of ‘progressive’ education, with conscious echoes of John Dewey’s student-centred pedagogies and associated models of project-based, experiential and inquiry-based learning. Dewey’s work has proven to be one of the most influential and durable philosophical theories in education, often used in conjunction with more overtly psychological accounts of the role that experience plays in learning.

With its combination of big data analytics and machine learning with progressivism, we could call the learning theory behind personalization ‘Big Dewey.’

Mark Zuckerberg’s philanthropic Chan-Zuckerberg Initiative is typical of the application of Big Dewey to education. CZI aims ‘to support the development and broad adoption of powerful personalized learning solutions. … Many philanthropic organizations give away money, but the Chan Zuckerberg Initiative is uniquely positioned to design, build and scale software systems … to help teachers bring personalized learning tools into hundreds of schools.’

To test out this model of learning in practice, new startup ‘lab schools’ have been established by Silicon Valley entrepreneurs. Many act as experimental beta-testing sites for personalized learning platforms–using students as guinea pigs–that might then be sold to other schools. As Benjamin Doxtdator has documented, these new lab school models of ‘hyperpersonalization’ utilize digital data technologies to ‘extract’ the ‘mental work’ of students from the learning environment in order to tune and optimize their platforms prior to marketing to other institutions.

Larry Cuban, however, has detailed the variety of ways that personalized learning has been taken up in schools in Silicon Valley, and himself sees strong traces of progressivism in their practices.

However, Cuban also notes that many employ methods more similar to the kind of ‘administrative progressivism’ associated with the psychologist EL Thorndike than Dewey. Thorndike was interested in identifying the ‘laws of learning’ through statistical analysis, which might then be used to inform the design of interventions to improve ‘human resources.’ Measurement of learning could thereby contribute to the optimization of ‘industrial management’ techniques both within the school and the workplace. Administrative progressivism was concerned with measurement, standardization and scientific management of schools rather than the student-centred pedagogies of Dewey.

‘What exists now is a re-emergence of the efficiency-minded “administrative progressives” from a century ago,’ argues Cuban, ‘who now, as entrepreneurs and practical reformers want public schools to be more market-like where supply and demand reign, and more realistic in preparing students for a competitive job market.’

With machine learning as its basis, personalization is a twenty-first century algorithmic spin on administrative progressivism. The ‘laws of learning’ are becoming visible to those organizations with the technical capacity to mine and analyse student data, who can then use this knowledge to derive new theoretical explanations of learning processes and produce personalized learning software solutions. As an emerging form of algorithmic progressivism, personalization combines the appeal of Dewey with the scientific promise of big data and autodidactic machine learning.

Ultimately, with the Big Dewey model, the logics of machine learning are being applied to the personalization of the learning experiences to be had by human learners. With this new model of education being supported with massive financial power and influence by Bill Gates, Mark Zuckerberg, and other edtech entrepreneurs, philanthropists and investors, Big Dewey is being forwarded as the philosophy and learning theory for the technological reform of education.

Machine learning escapes the lab
The machine behaviourism of autodidactic algorithm systems, public hypernudge pedagogies and personalized learning have become three of the most significant educational developments of recent years. All are challenging to educational research in related ways.

Machine behaviourism requires educational researchers to move their focus on to the kinds of reinforcement learning that occurs in automated nonhuman systems, and on how computational systems are being taught and trained by programmers, algorithm designers and engineers to learn from experience in an increasingly autodidactic way.

It’s not a sufficient response to claim that companies like DeepMind, IBM and so on take a reductionist view of what learning is—DeepMind’s Nature paper reveals an incredibly sophisticated learning model as pertains to neural networks software, while IBM has built its cognitive systems on the basis of established neuroscience knowledge about the human brain.

These systems can learn, but are not the same forms of learning known to most education researchers. As technical innovation proceeds, more and more learning is going to be happening inside computers. Just as educators hope to cultivate young minds to become lifelong independent learners, the tech sector is super-powering learning processes to create increasingly automated nonhuman machine learning agents to share the world with humans. What’s to say that educational researchers should not seek to develop their expertise in understanding nonhuman machine learning?

Theories of nonhuman learning are also becoming increasingly influential since machine learning processes underpin both the public hypernudge pedagogies of social media and personalized learning platforms I’ve outlined. The new behaviourist public hypernudge pedagogies, inspired both by behavioural science and behaviour design, are occurring at great scale among different publics, often according to political and commercial objectives, yet education research is oddly silent in this area.

While much has been written about big data and personalization, we’ve also still to fully explore how the tech sector philosophy of Big Dewey might affect and influence schools, teachers and students as adaptive learning platforms escape from the beta-testing lab and begin to colonize state education. Future studies of personalized learning could examine the forms of autodidactic machine learning occurring in the computer as well as the educational effects and outcomes produced in the classroom.

Image by Terry Kimura
Posted in Uncategorized | Tagged , , , , , | 1 Comment