PISA for personality testing – the OECD and the psychometric science of social-emotional skills

Ben Williamson

OECD SEL survey

The Organization for Economic Cooperation and Development (OECD) has published details of its new Study on Social and Emotional Skills (SSES). While the OECD has been administering international large-scale assessments on ‘cognitive skills’ and ‘competencies’ with both children and adults for many years, the new SSES survey represents a significant shift in focus to ‘non-cognitive’ aspects of learning and skills. While details of the science behind its cognitive skills and competencies tests are relatively well known, it is now becoming clear that the OECD’s social-emotional skills programme will emphasize the psychometric science of ‘personality’ measurement.

As part of ongoing research on social-emotional learning and skills (SELS) policies, practices and technologies, this (lengthy) post summarizes some of the key aspects of SSES, detailing its policy context, the ways it will generate and use student data, its conceptual basis in psychometrics, and the ways the OECD frames it as an objective ‘policy-relevant’ science programme with positive social and economic outcomes for participating countries.

From PISA to SSES
In recent years, the OECD’s PISA (Programme for International Student Assessment) and PIAAC (Programme for the International Assessment of Adult Competencies) tests have been the subject of extensive debate and research. New tests, such as the PISA-based Test for Schools to help schools compare themselves to international standards, as well as the expansion of its tests to include factors like problem-solving and well-being, have become available as the OECD has gradually extended its logic of measurement and comparison into policymaking systems globally.

The OECD first began signalling its interest in measuring and assessing social and emotional skills in 2014. That year, it published Fostering and Measuring Skills: Improving cognitive and non-cognitive skills to promote lifetime success, followed in 2015 by its report Skills for Social Progress: The power of social and emotional skills. In 2017 the OECD published Personality Matters: Relevance and assessment of personality characteristics, an extensive review of the scientific literature on personality theory and the measurement of personality factors. Although Personality Matters was developed as part of the PIAAC survey of adult skills, it has been deployed as the scientific rationale for the Study of Social and Emotional Skills announced in its 2017 ‘brochure’ Social and Emotional Skills: Well-being, connectedness and success and the accompanying SSES website.

Before going into some of the detail of SSES, the OECD’s focus in this area needs to be seen in a larger policy context. Over the last five years, as I’ve documented elsewhere, social-emotional learning and skills (SELS) have become a significant education policy priority and a key focus for education technology development and investment. Organisations including the global education business Pearson and the Nudge Unit have produced research summaries and guidance on developing SELS. The core idea behind many social-emotional learning and skills approaches is that the ‘non-cognitive’ aspects of learning are fundamentally linked to academic progress and to a range of social and economic outcomes, such as productivity, labour market behaviours and overall well-being.

Moreover, many advocates maintain, SELS are malleable and can be improved through direct teaching intervention. Improving SE skills is, therefore, seen as an important prerequisite for raising attainment, achieving social and economic progress, and improving individuals’ success, and an attractive prospect for policymakers seeking new ways to boost student achievement and employability.

Major lobbying groups based in the US have produced scientific justifications for focusing on SE learning and skills. The Collaborative for Academic, Social, and Emotional Learning (CASEL) has produced its own meta-analyses on social-emotional learning research and evidence. Similarly, the National Commission on Social, Emotional, and Academic Development (NCSEAD) at the Aspen Institute has published a research consensus drawing from evidence in brain science, medicine, economics, psychology, and education research. It claims to demonstrate that ‘the success of young people in school and beyond is inextricably linked to healthy social and emotional development, such as the ability to pay attention, understand and manage emotions, and work effectively in a team.’

Although CASEL and NCSEAD appear to have identified a consensus about what constitutes social-emotional learning and skills, the terminology remains confusing. Terms used for SELS including ‘character,’ ‘growth mindset,’ ‘grit,’ ‘resilience,’ and other ‘non-cognitive’ or ‘non-academic’ ‘personal qualities’ are often used interchangeably and gain traction with different academic, practitioner and policymaking communities. ‘Character’ has become the policy focus for the Department for Education in the UK following the 2014 publication of a cross-party Character and Resilience Manifesto,  while ‘grit’ has been favoured by the US Department of Education, as in its 2013 report Promoting Grit, Tenacity, and Perseverance—Critical Factors for Success in the 21st Century. Emerging education policies in the European Union appear to emphasize ‘soft skills’ as a category that encompasses SELS.

The OECD itself has adopted ‘social and emotional skills,’ or ‘socio-emotional skills,’ in its own publications and projects. This choice is not just a minor issue of nomenclature. It also references how the OECD has established itself as an authoritative global organization focused specifically on cross-cutting, learnable skills and competencies with international, cross-cultural applicability and measurability rather than on country-specific subject achievement or locally-grounded policy agendas.

Social-emotional datafication
The SSES programme was launched in 2017 with a timetable toward delivery of the first results in 2020. According to a published project schedule, during 2018 the OECD is developing the instruments and test items, before field testing in participating countries later in the year. The first formal round of the survey will take place in 2019, with final results released to the public late in 2020. The OECD also plans to administer SSES repeatedly in order to generate longitudinal data.

The expected outputs from the project include:

  • a set of validated international instruments to measure social and emotional skills of school-aged children
  • a dataset with information on the level and spread of social and emotional skills of children at ages 10 and 15, obtained from multiple sources, and accompanied with a wide scope of background and contextual variables
  • an improved understanding amongst policy-makers, education leaders, teachers, parents and other stakeholders on the critical role of social and emotional skills and the types of policies and practices that support the development of these skills
  • an improved understanding of whole child development, specifically as it relates to the development of social and emotional skills of children and youth

Although it is anticipated that only 10-12 countries and cities will take part in the first study, a huge quantity of data will be collected from the participants to deliver these outputs. The student survey of 10 and 15 year-olds itself consists of two assessments.

The first is a direct assessment. This is to be administered as a self-report questionnaire, which will be completed online as a computer-based survey. Students will respond to questions that are designed to assess behaviours considered indicative of selected SE skills. Indirect assessment will add to the dataset, including reports from parents and teachers based on similar questions on the typical behaviours of individual participating students.

In addition to the core assessment instruments will be contextual questionnaires for completion by children, their parents, teachers and principals. The contextual questionnaire for students will gather data on demographics, family culture, subjective health and well-being, academic expectations, and perception of their own SE skills.

Parents will provide information about their children’s SE skills, family background, child’s performance, home learning environment, parent-child-relationship, parental styles, learning activities, and parents’ own attitudes and opinion. As this list indicates, SSES is not just focused on school factors involved in developing SELS, but on distinctive family factors too, including learning activities undertaken out of school.

Schools themselves will provide contextual information from teachers in the form of reports on students’ SE skills, teachers’ own backgrounds, school characteristics, teaching practices, and teachers’ values and expectations about SE skills. Principals will add data on school background, school management, principles and rules, school climate, and the role of SE skills in curriculum and school agenda, as well as further administrative data for calculating other behavioural correlates and outcomes.

The data production expectations on schools, students and their families are, as the list demonstrates, extensive and extend well beyond the normal jurisdiction of the education sector into the extraction of information about homes, family relationships and parenting practices.

A further OECD document suggested it was also considering ‘exploring ways to link social and emotional skill measurement of the proposed study with other OECD measurement instruments such as those used in PISA and PIAAC, as well as with local measurement instruments such as standardised achievement tests.’

The direct assessment will be delivered online using a centralized software platform for assessment of children’s SE skills. Notably, the OECD claims it will use log file data obtained during the test as additional indicators of SE skills.

Log file information collected during computer-based international assessments has been described by Bryan Maddox as ‘process data’ collected about such things as  response times and key strokes, which can be studied with ‘micro-analytic precision’ in the analysis of larger-scale assessment data. These log file data are increasingly used in assessment software platforms as an extension of the test and can be conceptualized as ‘the mechanisms that underlie what people do, think, or feel, when interacting with, and responding to, the item or task.’

It’s not entirely clear how SSES will use log file data, but other projects have sought to correlate process metadata such as keystroke and response times to SELS. For example, the winner of CASEL’s 2017 design challenge on technologies to assess social-emotional learning was designed to capture the metadata generated as students took a computer-based test. It claimed its ‘measure quantifies how often students respond extremely quickly over the course of a test, which is strongly correlated with scores from measures of social-emotional learning constructs like self-regulation and self-management.’

This project exemplifies a form of stealth assessment whereby students are being assessed on criteria they know nothing about, and which rely on micro-analytics of their gestures across interfaces and keyboards. It appears likely that SSES, too, will involve correlating such process metadata with the OECD’s own SELS constructs to produce stealth assessments for quantifying student skills.

As the range of its data collection activities demonstrate, the OECD has designed SSES to include not only direct survey assessments, but extensive contextual information, school-level data on students behaviours and outcomes, and log file information that can be analysed as digital signals of SE skills. Importantly, though, these data all rely on specific conceptualizations of socio-emotional skills that the OECD has invested significant institutional effort in researching and defining.

Personality measurement
Behind the OECD SSES survey lies a set of psychological knowledge about the measurable qualities and characteristics of socio-emotional skills. The SSES brochure gives an overview of how the OECD defines SE skills. It claims socio-emotional skills constructs can be classified into five broad domains, which it refers to as a well-known framework called the ‘Big Five model’: emotional regulation (emotional stability); engaging with others (extroversion); collaboration (agreeableness); task performance (conscientiousness); open-mindedness (openness). The SSES survey itself will be administered to assess 19 skills which fit into each of these five categories.

The brochure notes the ‘five-factor structure of personality characteristics’ has been extensively researched and empirically validated in multiple studies, leading to ‘widespread acceptance of the model.’ It further adds that there is ‘extensive evidence that the Big Five domains and sub-domains can be generalised across cultures and nations’ and is suitable for describing socio-emotional skills in both children and adults.

OECD SELS categories

A much fuller account of the Big Five model is provided in Personality Matters, which the SSES brochure references directly. Written by an OECD policy analyst with academic experience in educational psychology, international social policy and cross-cultural survey methodology, Personality Matters is an extensive review of psychological and psychometric research on the conceptualization and measurement of human personality.

The document reviews research on a variety of potential measures of social-emotional learning and skills, including ‘grit,’ ‘character skills’ and other socio-emotional competencies (though it makes no reference to ‘growth mindset,’ a current popular psycho-policy concept). The review favours the five factor model of personality consisting of openness, conscientiousness, extroversion, agreeableness and neuroticism (OCEAN). It acknowledges that psychologists have developed tests and assessments such as the Big Five Inventory (BFI), the Neuroticism-Extraversion-Openness Personality Inventory (NEO-PI) the International Personality Item Pool (IPIP) and Trait Descriptive Adjectives (TDA) to measure these personality factors.

It is of course clear that the OECD’s SSES categories map exactly on to the five factor personality categories, with ‘emotional stability’ standing in for ‘neuroticism.’ The model was approved at an OECD meeting in 2015, following a presentation by Oliver John, a psychologist at the Berkeley University Personality Lab and original author of the Big Five Inventory personality test. Likewise, the SSES survey has been designed to assess 19 skills associated with its Big Five model, in ways which emulate the structure of many of the personality tests cited in its review. As this indicates, the way the OECD has formulated social and emotional skills is a direct translation of the OCEAN categories used by psychologists for personality testing. In this sense, SSES appears to represent a therapeutic shift in OECD focus, with its target being the development of emotionally stable individuals who can cope with intellectual challenge and real-world problems.

Crucially, the review also notes strong correlations between high scores in the Big Five and other outcomes such as academic achievement, job performance, and standardized test scores. Notably, it emphasizes the ‘policy relevance’ of the insight that many personality characteristics—or socio-emotional skills as the SSES describes them—are malleable and can therefore become a ‘potential target for policy intervention.’  Arguably, policy interventions that would be relevant to remedying identified personality problems would be forms of therapeutic provision, such as remedial classes in socio-emotional skills, which schools would then be responsible for delivering. ‘Therapeutic education,’ as Kathryn Ecclestone and Denis Hayes characterized it a decade ago, includes activities and underlying assumptions ‘paving the way for coaching “appropriate” emotions’ and ‘replace education with the social engineering of emotionally literate citizens who are also coached to experience emotional wellbeing.’

The combination of the human sciences with public policymaking has a long history, enabling governments to act upon the capacities of those subjects they govern. The promise of scientific objectivity regarding human behaviour and emotion is very attractive to policymakers wishing to manage or social-engineer the ‘appropriate’ behaviours of populations more effectively. However, as Sheila Jasanoff has argued, the objectivity of ‘policy-relevant knowledge’ is always achieved through hard work, argument, and the strategic deployment of persuasive, authoritative claims. The ‘objectivity-making’ practices of psychologists of personality underpin the recent policy shift to SELS embodied by the OECD, and the forms of therapeutic education that are likely to proliferate as schools realize they are to be measured and assessed according to SELS categories.

The OECD has made a science itself out of crafting objective policy-relevant knowledge from larger, contested bodies of scientific evidence about human competencies and personalities. With SSES, it either dismisses or omits the ‘grit,’ ‘growth mindset’ and ‘character skills’ literature—which in the Personality Matters report it suggests imply ‘moral connotations that many researchers and policy advisers would like to avoid’—and instead translates the concepts and practices of psychometric personality testing into policy-relevant approaches to measuring and assessing the social and emotional worlds of children.

Socio-emotional indicators and socio-economic outcomes
Beyond the presumed scientific objectivity of personality testing, interest in SELS among government departments and policymakers is also due at least in part to the economic arguments made by its advocates.

In the US, SELS are a lucrative investment opportunity under the banner of ‘impact investing.’ These ‘pay for success’ schemes allow investment banks and wealthy philanthropies to invest in educational services and programs and then collect public money with additional interest as profits if they meet agreed outcomes metrics.  The metrics for calculating the social benefit and monetary value of SELS schemes have  been published as a cost-benefit analysis with the title The economic value of social and emotional learning.

Beyond direct profitability of SELS programs for investors, however, the OECD makes a strong argument to governments that its assessment of socio-emotional skills can produce indicators of socio-economic outcomes. As such, it makes the case that government investment in SELS through departments of education will generate a substantial return in the shape of productive human capital. This is an argument the OECD has refined through years of PISA and PIAAC testing and analysis.

The Nobel laureate of economics James Heckman has advised the OECD on its social-emotional learning program through co-authoring its 2014 report on measuring non-cognitive skills. The report claimed that some programmes to support non-cognitive skills development ‘have annual rates of return that are comparable to those from investments in the stock market.’ Based on  extensive economics analysis twinned with developmental psychology and the neuroscience of ‘human capability formation,’ Heckman has influentially argued for over a decade that non-cognitive social-emotional skills and ‘personality factors are also powerfully predictive of socioeconomic success and are as powerful as cognitive abilities in producing many adult outcomes.’ Making ‘personality investments’ in young people, he claims, leads to high returns in labour market outcomes.

It’s notable that the organization contracted to lead SSES is the Center for Human Resource Research (CHRR) at The Ohio State University. The CHRR’s mission is to provide ‘substantive analyses of economic, social, and psychological aspects of individual labor market behavior to examining the impact of government programs and policies.’ According to the CHRR, the SSES project will identify ‘those social and emotional skills that are cross-cultural, malleable, measurable, and that contribute to the success and well-being of both the youth and their society.’ The assessment of SELS is therefore to be undertaken through the logic of human resource management and the analysis of labour market behaviours.

As the OECD itself phrases it, the purpose of SSES is to ‘provide participating cities and countries with robust and reliable information on the social and emotional skills of their students,’ and also to ‘have policy relevance’ by identifying ‘the policies, practices and other conditions that help or hinder the development of these critical skills.’ As a ‘policy-relevant project,’ the OECD claims, ‘study findings can also be used by policy makers to devise better policy instruments geared towards promoting these types of skills in students.’ Its brochure gives examples of ‘critical life outcomes’ that correlate with socio-emotional skills, including school achievement, college graduation, job performance, health and well-being, behavioural problems, and citizenship participation. These, it claims, can be improved because social and emotional skills are learnable and personality is malleable.

Psycho-economic policymaking & personality modification
This is necessarily a very partial overview of some of the key features of SSES. However, it does raise a few headline points:

  • SSES extends international-large scale assessment beyond cognitive skills to the measurement of personality and social-emotional skills as a way of predicting future economic and labour market outcomes
  • SSES will deliver a direct assessment instrument modelled on psychological personality tests
  • SSES enacts a psychological five-factor model of personality traits for the assessment of students, adopting a psychometric realist assumption that personality test data capture the whole range of cross-cultural human behaviour and emotions in discrete quantifiable categories
  • SSES extends the reach of datafication of education beyond school walls into the surveillance of home contexts and family life, treating them as a ‘home learning environment’ to be assessed on how it enables or impedes students’ development of valuable socio-emotional skills
  • SSES normalizes computer-based assessment in schools, with students required to produce direct survey data while also being measured through indirect assessments provided by teachers, parents and leaders
  • SSES produces increasingly fine-grained, detailed data on students’ behaviours and activities at school and at home that can be used for targeted intervention based on analyses performed at a distance by an international contractor
  • SSES involves linking data across different datasets, with direct assessment data, indirect assessments, school admninistrative data, and process metadata generated during assessment as multiple sources for both large-scale macro-analysis and fine-grained micro-analytics–with potential for linking data from other OECD assessments such as PISA
  • SSES uses digital signals such as response times and keystrokes, captured as process metadata in software log files, as sources for stealth assessment based on assumptions about their correlation with specific social-emotional skills
  • SSES promotes a therapeutic role for education systems and schools, by identifying ‘success’ factors in SELS provision and encouraging policymakers to develop targeted intervention where such success factors are not evident
  • SSES treats students’ personalities as malleable, and social-emotional skills as learnable, seeking to produce policy-relevant psychometric knowledge for policymakers to design interventions to target student personalities
  • SSES exemplifies how policy-relevant knowledge is produced by networks of influential international organizations, connected discursively and organizationally to think tanks, government departments and outsourced contractors
  • SSES represents a psycho-economic hybridization of psychological and psychometric concepts and personality measurement practices with economic logics relating to the management of labour market behaviours and human resources

There is likely to be additional concern that the OECD will use SSES to conduct large-scale international comparison of children’s social-emotional learning and skills. At present the first stage study appears too limited for that, with only an estimated 10-12 participating cities and countries.

However, over time SSES could experience function creep. PISA testing has itself evolved considerably and gradually been taken up in more and more countries over different iterations of the test. The new PISA-based Test for Schools was produced in response to demand from schools. Organizations like CASEL are already lobbying hard for social-emotional learning to be used as an accountability measure in US education—and has produced a State-Scan Scorecard to assess each of the 50 states on SEL goals and standards. Even if the OECD resists ranking and comparing countries by SELS, national governments and the media are likely to interpret the data comparatively anyway.

If these developments are taken as indicators, it is possible that over time the OECD may generate international comparisons, accountability metrics and league tables of education systems based on intimate assessments of students’ personalities.

Image credits: OECD
Advertisements
This entry was posted in Uncategorized and tagged , , , , , , , . Bookmark the permalink.

4 Responses to PISA for personality testing – the OECD and the psychometric science of social-emotional skills

  1. Pingback: Personal development: OECD’s social and emotional skills. Is it science? | onderwijs_2032 science check

  2. Pingback: The Imitation Game – Informative Flights

  3. Pingback: Stealth Assessment: What do Facebook Data Mining, OECD Testing, and ‘Well-Being’ Assessment have in Common? | Educhatter

  4. Pingback: Intimate data analytics in education | Doug Belshaw's Thought Shrapnel

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s