The Office for Students as the data scientist of the higher education sector

Ben Williamson

Data play a huge role in British higher education. The new regulator for the sector, the Office for Students, will escalate data collection and use in HE in years to come. Improving student information and data to ‘help students make informed decisions’ is one of its four key strategic priorities, but it has also raised concerns about its use of student data to increase competition and market pressure. The Labour Party has recently tried to block it in a bid to prevent the further entrenchment of market-oriented higher education policy in the system. However, there remains a need to focus in close detail on how the OfS will use data in its remit as a market regulator.

The cover of the recently published OfS regulatory framework for higher education in England gives some indication of how the new regulator sees itself as a data-centred site of sectoral expertise. It features a scientist peering into a microscope with apparent satisfaction about what she sees. The scientist, naturally, is the OfS, performing experiments and observing the results; the microscope is the technical and methodological apparatus that allows it to see the sector; and (out of shot) is the university, flattened on to glass for inspection–made legible as data to be zoomed in on, scrolled across, examined and compared with other samples from the sector.

The idea of the OfS as a scientist of the sector–or more specifically, as a data scientist of the sector–is intriguing. It smacks of assumptions of scientific rigor, objectivity, and innovation. This form of metric realism, which assumes data tell the truth, is the central epistemology of trends in datafication. The reality of ‘laboratory life’ inside the OfS, like all labs, is doubtless more fraught with disagreement, negotiation and compromise, as STS studies of science practices might note. Nonetheless, the OfS regulatory framework document is a key inscription device that, for the time being at least, gives us the best clues of its planned data activities over the coming years.

As part of ongoing work into the data infrastructure of higher education, I’ve spent some time with the regulatory document, trying to figure out how student data are likely to be used in future years (on uses of research data see the Real-time REF Review project). The OfS is just one of many actors involved in a project to upgrade the core infrastructure for student data collection–a decade-long project that’s been going on 7 years already and is due for national rollout in 2019/2020.

In these notes, I lay out some of the key things the OfS says about ‘data’ in the document. There are 87 uses of the word data in it, so through light-touch discourse analysis I’ve attempted to categorize the various ways the OfS approaches data. I’ve deliberately kept a lot of quotations intact with the addition of a few annotations.

Data as strategy
The first point is that ‘The OfS will develop a data strategy in 2018’ and ‘The information and data the OfS requires to fulfil its functions will be wide-ranging’ (20). This is both mundane and not. The fact that it is developing a data strategy at all–and a wide-ranging one at that–is indicative of how the OfS will make data into a central aspect of HE regulation. As Andy Youell of HESA (Higher Education Statistics Agency) has written, the framework represents a shift from ‘data informed to data led’ regulation with data analysts playing an increasingly influential role in HE policy.

The chair of the OfS, of course, is Sir Michael Barber, a long-standing advocate of metrics and performance delivery models in different aspects of government. His most recent role was as education adviser to the global educational company Pearson, where he oversaw its organizational pivot toward big data, predictive learning analytics and adaptive learning. Under his leadership, the OfS too approaches data as a core strategy for fulfilling its mandate.

Regulatory data
The OfS primary remit is to regulate HE, and it is positioning data as a core component to that work:

The use of information, including data and qualitative intelligence, will underpin how the OfS undertakes its regulatory functions. The OfS will take an information-led and proportionate approach to monitoring individual providers, ensuring that students can access reliable information to inform their decisions. (19)

Key terms here include ‘monitoring’, which confirms concerns that the OfS will possess powers of data-led performance measurement. As well as monitoring individual institutions, the OfS will ‘Monitor the sector as a whole, to understand trends and emerging risks at a sector level and work with the sector to address them’ (20). However:

This regulatory framework does not … set out numerical performance targets, or lists of detailed requirements for providers to meet. Instead it sets out the approach that the OfS will take as it makes judgements about individual providers on the basis of data and contextual evidence. (15)

From ‘monitoring’, then, comes ‘judgement’ from data and other evidence. The OfS comes across as a suspicious actor of evidence-based policy.

Improvement data
Another key use of data by the OfS is to ‘Target, evaluate and improve access and participation, and equality and diversity activities’ (20). As such, monitoring and judgement become the basis for targeted improvement plans, with HE institutions specifically singled out if underperformance is detected from the data in specific areas. As is well-known, the OfS will also ‘Operate the TEF’ (20) and take the outcomes of the 2018 statutory TEF review ‘into account as it considers the future scope and shape of the TEF’ (24). As such, it will be the main data-led judge of teaching quality and improvement in the sector.

Student choice data
As the Office for Students, driven by the political rhetoric of ‘putting students as the heart of the system’, a key ambition is to put students themselves in touch with sector data. This includes efforts to ‘improve the quality of information available to students’ (25). Two key quotes from the framework stand out:

Prospective students will be equipped with the means, underpinned by innovative and meaningful datasets and high quality information, to enable them to make informed choices about the courses that are right for them. (10)

[OfS will] ensure students can access reliable and appropriate information to inform their decisions about whether to study for a higher education qualification and, if so, identify which provider and course is most likely to meet their needs and aspirations. (20)

Here the OfS mirrors recently-announced plans by Universities Minister Sam Gyimah to support software developers to develop student-facing apps for price-comparison of university courses. It’s a controversial idea, announced as part of the renewed Teaching Excellence Framework (TEF), which requires the use of Longitudinal Educational Outcomes (LEO) datasets linking courses to earnings. It’s also controversial because it makes student choice conform with the MoneySupermarket model of product comparison based on value-for-money calculations, and further solidifies the idea of students as consumers and courses as products in a marketplace of comparable choices.

Data alignment
Alongside choice comes constraint. The OfS will ‘publish student outcomes and current and future employer needs as a way of informing student choice’ (17). This short sentence appears to carry two main messages: first, that access to outcomes data from institutions will help shape the choices of prospective students; and second that those choices should also be made through reference to ’employer needs’.

Indeed, the OfS is actively seeking to align HE outcomes to industry requirements, and will ‘Work with employers and with regional and national industry representatives to ensure that student choices are aligned with current and future needs for higher level skills’ (20). This is a very instrumentalized view of HE as part of the employment pipeline for high-skills jobs. Of course, students can choose to ignore this information. But presenting HE data in this way may itself shape the choice environment for students, with certain choices made more attractive than others.

Nudge data
Talking of choice environments, the OfS is ‘taking the latest thinking on behavioural science into account, to consider how best to present this data in a consistent and helpful way to ensure that students have access to an authoritative source of information about higher education’ (25).

Clearly, the idea of intervening in students’ choice through subtle behavioural means is not an accident; the OfS is actively engaging with the psychology of choicemaking to shape and nudge how students decide on their courses. In this sense, the OfS is seeking to instantiate the experimental methods of behavioural public policy in HE, using data to prompt or even persuade students to make ‘desirable’ choices.

But it may over time extend to logic of behavioural science to sectoral nudging at scale. According to one commentary on the regulatory framework, ‘the OfS should be encouraged to further consider behavioural theory and its various insights, such as those contained in “nudge” theory, and thus design interventions that incentivise compliance from the outset.’

Government data
Though it is notionally an arms-length agency–geographically, it’s located in the south west, along with all the other HE agencies–the OfS appears to enjoy a remarkably close and mutually reinforcing relationship with government. Not only did it emerge from BIS (now BEIS), but it will also use its expertise in HE data to:

Support the Department for Education, given its overall responsibility for the policy and funding framework in which the sector operates, and other public bodies such as UKRI in the delivery of their prescribed functions.

In contrast the role of a ‘broker‘ between government and the sector performed by HEFCE, the OfS appears to have a much more hand-holding relationship with government–despite being at arms-length, a government minister has the power to give it directions and demand advice or reports–while simultaneously strong-arming the sector into compliance.

Designated data body
In my longer project, I have focused on the work of HESA as a central agency for delivery of the new student data infrastructure for HE. HESA is part of the family of ‘official statistics’ agencies in the UK, and in 2017 applied for the position of ‘designated data body’ (DDB) to work with the OfS, a position conferred on it early in 2018 by central government ‘on the recommendation of the OfS’ (19). As such, the OfS will ‘Work with, and have oversight of, the designated data body (DDB) to coordinate, collect and disseminate information’ (17).

The DDB will collect, make available, and publish appropriate information on behalf of the OfS, and the OfS will be responsible for holding the DDB to account for the performance of those functions. (19)

As this makes clear, HESA is now subordinate to the OfS, acting on its behalf and held to account for its own performance in the statistical delivery of the data required by the OfS. As such, the work of HESA has shifted from statistical reporting to a much more politicized position, ‘play[ing] a key role in supporting and enhancing the competitive strength of the sector.’

Indicator data
Indicators are the principal power source in the OfS machinery. Through indicators, the OfS will ultimately receive regular signals of institutional performance which can then be used to assess risk or to identify need for intervention:

All providers will be monitored using lead indicators, reportable events and other intelligence…. These will be used to identify early, and close to real-time warnings that a provider risks not meeting each of its ongoing conditions of registration. (18)

The OfS will identify a small number of lead indicators that will provide signals of change in a provider’s circumstances or performance. Such change may signal that the OfS needs to consider whether the provider is at increased risk of a breach of one or more it its ongoing conditions of registration. These indicators will be based on regular flows of reliable data and information from providers and additional data sources. (49)

The mention of ‘close to real-time warnings’ is especially important, as it signals a significant acceleration in the temporality of HE data reporting, analysis and action. Under the OfS, universities are to be monitored for performance fluctuations and changes that, like economic spikes and dips, may be presented as informational flows on data dashboards to affect prompt and timely decision-making.

Longitudinal data
In addition to ‘close to real-time’ data, the OfS is seeking to expand and improve the use of longitudinal datasets and analyses:

The OfS will draw on the longitudinal education outcomes (LEO) dataset as an important source of information about graduate outcomes. Its further development will be a priority for the OfS, taking into account both its limitations and its significant potential. (25)

LEO consists of experimental statistics on employment and earnings of higher education graduates using matched data from different government departments, which has controversially been used to suggest that students can choose courses based on future earnings potential. It is also a significant methodological accomplishment, linking datasets about education, personal characteristics, employment and income, and benefits gathered from the departments of education, work and pensions, HESA and HMRC.

Comparative data
The data will also be used comparatively to assess different institutions against each other:

It is anticipated that this data will be largely quantitative and generated as a result of a provider’s existing management functions … allowing for greater consistency, comparability and objectivity when looking across a range of providers. (50-51)

Data-led comparison and benchmarking is of course at the heart of rows over HE marketization, as universities are incited to compete for prospective students and income. It drives institutions to showcase themselves as competitive, high-performing organizations, and is visible in all kinds of HE rankings such as UniStats and Complete University Guide tables.

Anticipatory data
Furthermore, the data used by the OfS will not be merely historical, real-time and comparative–it will be anticipatory too. The OfS will undertake ‘horizon scanning to understand and evaluate the health of the sector’ (17) and will use indicator data ‘to anticipate future events’ (161). In this sense, the OfS is simply mirroring the increasing use of predictive analytics in HE, with institutions in the UK already using data to forecast student progress or identify students at-risk of drop-out or non-completion. The use of predictive data practices by the OfS, however, will be applied to institutions and the sector as a whole–to predict, for example, providers at-risk of underperformance or financial difficulty.

Data burden
All this data collection and analysis activity sounds like it will be a heavy burden on institutions, and the OfS admits:

The implementation of the OfS’s data strategy may initially increase regulatory burden, but the long term aim is to use data to reduce regulatory burden. Such data requirements are not therefore intended as a regulatory burden on providers but to provide the information that allows the OfS to be an effective and proportionate regulator.

Perhaps, however, the heaviest burden will be the threat of punitive action based on constant OfS investigation of institutional data.

Data auditing & investigation
Regimes of audit and inspection are of course familiar across many sectors, and the OfS will make ‘data audits’ a part of the HE landscape:

The OfS will assess, as part of its routine monitoring activities, the quality, reliability and timeliness of information supplied by a provider including through scheduled or ad hoc data audit activity. If the OfS has reason to believe that information received is not reliable, it may choose to investigate the matter. (131)

It may even, in certain cases, ‘require information to be re-audited by a specified auditor, where the OfS has reasonable concern that the audit opinion does not provide the necessary assurance’ (56). It therefore appears that the OfS will demand new forms of meta-auditing of existing audit data.

Targeted action
Finally, the OfS proposes to use data as the basis for taking targeted action on institutions and the sector:

The OfS may also take targeted action if it needs to establish the facts before reaching a judgement about whether there is, or is likely to be, a breach of one or more ongoing conditions of registration.

May require the provider to take particular co-operative action by a specified deadline – these actions may include access to, information (including data), records or people, to enable the OfS to investigate any concerns effectively and efficiently. (56)

All in all, the OfS will instantiate a new regime of data in HE, emphasizing an empiricist faith in the ‘truth-telling’ capacities of digitally generated information. It is positioning itself as a source of data scientific expertise in the sector, treating universities as samples to be observed, students as specimens to be nudged to make choices based on data, and the sector as a whole as a laboratory for its experiment in data-led regulation.

Image: Office for Students