The importance of data: developing education policies is impossible without evidence

“It is a capital mistake to theorise before one has data,” said Sherlock Holmes, the famous fictional detective in A Study in Scarlet.

It would equally be a mistake to develop education policies, earmark resources and implement programmes without having data and evidence to help guide the process.

“We need data to build evidence that leads to better evidence-informed decision-making” said Hugues Moussy, Head of Systems Performance and Assessment Unit at the ETF.

Data is also essential to provide an overview of the state of development - be it progress or regression – of education policies at a national as well as comparative level.

The Torino Process

Such data gathering is at the heart of the Torino Process. Introduced in 2010 by the ETF, it is an evidence-based policy analysis of human capital development and vocational education and training (VET) systems in ETF partner countries.

Carried out every two years, it provides an overview of the state of development of VET systems. It uses a participatory approach whereby partner countries provide information “about the process and an overview of changes from preceding rounds; describe the key deliverables and the reporting framework for their preparation; present the implementation plan and timeline; and provide orientation concerning the choice and use of statistics in support of national reporting”.

For the VET sector, such data is useful in a myriad of ways, and in ways not always commonly considered, such as investment. “Data is not just about shaping policy dialogue and advice, but where you want to invest in the system, be it in people, innovation, training or the like,” he said.

Moussy gave an example of how this works. “If, say, only 16% of youth are trained in VET or X% of the adult population has access to work-based learning programmes, and a government wants those figures to reach 25%, quantitative data can tell you how much you want to invest and the level of investment needed to reach that goal.”

Such data also informs a government or decision-makers with whom they need to talk to: “Perhaps that is within different ministries, or to attract more people into work-based learning programmes you need to talk with employers. It is about having a broad picture to give an indication of where and in what direction investment should go,” said Moussy.

An evolving process

A core strength of the Torino Process over the past 12 years is that it has evolved as the number of VET stakeholders has expanded – at the national and international level – and to reflect shifting developments and expectations in the VET segment.

This has involved revising the process for the sixth review which began in 2022, with data gathering currently underway.

Many dimensions

The Torino Process itself involves the collecting of quantitative data “to give a sense of what the system achieves, the outcomes and the overall performance of the system through different dimensions,” he said.

The Process involves eight dimensions that are subdivided into thirty. For each dimension, different quantitative databases are used, such as Eurostat, UNESCO, and the OECD’s Programme for International Student Assessment (PISA), with 130 item responses from the database then aggregated by the ETF.

“For one particular dimension we tap into different quantitative databases that already exist and aggregate them together. For example, regarding innovation, we have a subtitle: ‘Systemic innovation in promoting participation and graduation,’ so we use three different items from different databases,” explained Moussy.

The ETF utilises such data “to provide policy-advice based on the outcomes of the quantitative data, but also to provide a broad picture on where investment should be directed. They are supposed to be linked, but often are not,” he said.

 

The national level

With the data extracted from international databases, the ETF’s partner countries may not use or participate in providing data to such organisations, complicating the data gathering process

International databases are utilised as there is a clear methodology and quality assurance, allowing for comparisons to be drawn.

When the data is missing, comparisons cannot be made, so the ETF develops questions that can be answered by partner countries to have a better picture of what is happening in the development of VET.

“To replace the data we ask a particular question to a country, asking them to give the source and why they responded this way, and a score from worst to best,” he said.

The data gathered at a strictly national level is used at a secondary level of the evaluation by the ETF for country policy reviews.

Overall, the Torino Process has “definitely helped countries to have a clearer view of their weakness, strengths and shortcomings,” said Moussy. “It has also helped clarify how they want to monitor their system.