The ETF’s work on data gathering, analysis and usage
For over two decades the ETF has provided support to partner countries, from Eastern Europe to the Mediterranean and Central Asia, to develop national data collection on education, skills development and employability. Building data intelligence by working in partnership with other organi...
For over two decades the ETF has provided support to partner countries, from Eastern Europe to the Mediterranean and Central Asia, to develop national data collection on education, skills development and employability. Building data intelligence by working in partnership with other organisations at national and international levels, the ETF uses the findings to inform a broad range of stakeholders about progress – and regression – in key areas, as well as to monitor and inform education and skills development policies.
The importance of data
It is no secret that policymakers usually prefer statistical data for decision-making which is often viewed as being more reliable, but this is not necessarily the case. What is important is the rigour and reliability of the data, the scope and context of the exercise, and that it gets communicated and used, if appropriate.
The importance of data cannot be overstated. Good quality data is essential in the development of education policies, the earmarking of resources and in assessing the implementation of programmes. As Hugues Moussy, Head of Systems Performance and Assessment Unit at the ETF, pointed out in a blog article on the importance of data:
“Data is not just about shaping policy dialogue and advice, but where you want to invest in the system, be it in people, innovation, training or the like.”
How data is used
The ETF works in partnership to produce data and expand data use with other EU institutions and agencies, international intergovernmental organisations, regional organisations such as the Education Reform Initiative of South Eastern Europe (ERI SEE), bilateral donor organisations, and the private sector.
Data collected and analysed at a national and international level by the ETF and partner organisations, such as sister EU agency Cedefop, has been crucial in providing a comprehensive overview of education and skills needs. Over the past decade, data has been used to better understand the modernisation process of vocational education and training (VET) systems, and the implementation of human capital development and lifelong learning policies.
The ETF’s data collection, processing and analysis has enabled the monitoring and the evaluation of policies and reforms, both within and across countries. The ETF is involved in various monitoring exercises, as requested by partner countries and the European Commission. This includes the monitoring of policy and system performance within the Torino Process, the monitoring of VET in EU candidate countries under the Osnabrück Declaration, and the introduction and monitoring of the Youth Guarantee in the Western Balkans region.
The ETF’s Key Indicators on Education, Skills and Employment (KIESE) report holds a bounty of statistics on the main findings and results from the 2022 data collection. This has enabled a review of developments in the field of education, skills and employment in the partner countries. A statistical snapshot of the indictors also allows the ETF partner countries and the seven EU candidate countries to reference themselves with the EU.
The report also discusses the results from more recent ETF work on youth transition and skills mismatch. For instance, in partner countries, one-third of all employed adults and up to 40% of young people are overqualified for the jobs they hold, noted Mircea Badescu, ETF Human Capital Development Statistician, at a May event, Consolidation of Evidence on Active Labour Market Policies and Transition to Work.
Other areas in which the ETF contributes to improving data intelligence in partner countries for better policymaking in education and training include graduate tracer studies and regular surveys on issues such as the professional development of teachers and trainers.
The ETF also undertakes targeted country-specific data collection such as the ETF monitoring and collation of data on the impact of the Russian invasion of Ukraine on schools, education and labour markets. See the latest update.
What are the challenges?
The ETF draws on data from international databases, where available, as there is a clear methodology and quality assurance, which allows for comparisons to be drawn. There are gaps on certain elements of countries’ education and training systems in international databases, especially in developing and transition countries as in many of the ETF’s partner countries.
The ETF’s Torino Process launched 13 years ago is in its sixth review. The Process involves collecting quantitative data in three areas of system performance: access, participation, and opportunities for lifelong learning; quality of lifelong learning outcomes; and system organisation. The monitoring framework has eight dimensions (which are subdivided into 30 outcomes). For each dimension, different quantitative databases are used, such as Eurostat, UNESCO, and the OECD’s Programme for International Student Assessment (PISA), with 130 item responses from the database then aggregated by the ETF.
A challenge is that data may be missing at international level, making it harder to draw comparisons. The Torino Process exercise attempts to redress this issue through the gathering of qualitative and contextual country data drawing upon its long experience working with key stakeholders at national level.
As the KIESE report noted,
“Data availability varies greatly among partner countries and remains the most significant challenge. Although the coverage of labour market statistics is satisfactory, the availability of other indicators, in particular for adult training, remains limited. Only very few countries provide information on educational outcomes such as graduate employability and early leavers.”
To address such shortcomings, there is a need for more cooperation at the EU and international levels, and the development of skills intelligence at national level. The ETF has joined forces with partners such as Cedefop, the European Foundation for the Improvement of Living and Working aConditions (Eurofound), the ILO and the OECD to consolidate its role in monitoring, the sharing of data and findings, and the conducting of surveys, such as the European Skills and Jobs Survey, which is being done jointly with Cedefop, and the Living, Working and Covid-19 survey done jointly with Eurofound.
The ETF draws upon nationally generated evidence produced by central statistical offices, social partners, civil society, think tanks, universities and other research organisations for its country and thematic knowledge based on various topics, such as qualifications, quality assurance, work-based learning, platform work, digitalisation, career guidance, skills needs.
Responding to policymakers on ‘what works’ for investing in VET for instance, often has no straightforward answer but the ETF works to elaborate sound options based on quality data using both quantitative and qualitative methods complemented with experience-based information and expert knowledge. Indeed, the latter is one of the key contributors to evidence outlined in the European Commission’s publication Support Mechanisms for Evidence-based Policy-Making in Education.
Looking beyond big data
Data is a force that matters in shaping decision-making and developing policies. Data has gained in significance in the field of education and t...
Data is a force that matters in shaping decision-making and developing policies. Data has gained in significance in the field of education and training, particularly in areas that used to be neglected in terms of investment and international cooperation, such as vocational education and training (VET) and adult learning programmes.
As American statistician W. Edwards Deming famously said: “Without data, you’re just another person with an opinion”.
Data is important, said Mihaylo Milovanovitch, ETF Senior Human Capital Development Expert and Coordinator for System Change and Lifelong Learning, as at the simplest level, it allows for informed decisions.
“Data injects knowledge into discourse. In policy-making, evidence is just one aspect taken into consideration when decisions are made, as there are other considerations – feasibility, politics, and so on – but evidence allows for a choice. If you don’t have data, you don’t have the choice to make informed decisions, or at least know what you’re ignoring as a trade-off,” he said.
A second reason concerning data’s importance is that it allows for the creation of a basis for building a consensus around an idea or policy, which may also involve countering a bad idea. Thirdly, data can help build trust.
“It is a good way to promote accountability and to make beneficiaries trust that what is going on is the right thing to do,” he said.
But for data to be truly useful, it must be rooted in the context of what is trying to be achieved.
“If you don’t invest in facilitating or revealing the context, you will never know what the data really means, be it the data analysis or for those affected by the data. It is extremely important to talk about data and invest the same effort in capturing the context as in generating data,” he added.
The limits to Big Data
Over the past decade, there has been a flurry of discussion around data gathering, including the upsides and pitfalls of how it is used. This has been spurred on by the explosive growth of Big Data through increased computing power, advanced software tools and data storage (such as through the cloud).
“There is a renaissance of reflection after years of Big Data usage – What is missing? What are the factors influencing it, and the willingness of practitioners to use the data they have? There is a realisation that it is not just about having the data, or one type of data, but the context and conditions of how data is generated, and how it is supposed to be used. How data was used to support policies wasn’t such an issue before,” Milovanovitch explained.
At the educational level, the limitations of Big Data are shaping how and what data should be gathered.
“Big Data covers a whole system or whole populations, but this kind of data has limits to what you can do with it to improve the fate of individual students, or how a school is doing. Big Data allows for a bird’s eye view and for big presentations, but, in reality, it is not enough to guide reforms. It does not tell you what to change specifically, like this or that to have a certain result,” he said.
A more specific focus on what data is needed to guide decision-making has been essential for developing segments of education that have been neglected until recently, like VET. Since the introduction of the Torino Process in 2010, evidence-based analysis of VET policies in ETF partner countries has been developed to provide a snapshot of the state of development of VET systems every few years.
There have been challenges to implementing policies, with a common complaint from countries about the lack of data and evidence. Milovanovitch said the ETF’s message is that countries can get evidence if they have the right approach to data gathering and are flexible.
“In most of what we are trying to monitor, it is possible to have the evidence. Countries need to find the right way to generate it,” he said.
This involves being open to using different types of evidence that mix the qualitative and quantitative and carve out a space for people concerned by the evidence to have a say.
A sound methodology is equally crucial but it “should not shy away from combining bits and pieces of what is needed. Because life is so colourful, and policies are messy, you may need to combine things that might not seem combinable at first glance,” he said.
This takes on added importance with a ‘moving target’ like monitoring priorities in education.
“We have this situation where life gallops ahead, policies try to respond to developments, and people try to develop evidence, usually as the last step, which is a challenge, as this really should happen at the beginning, to know what needs to be looked at,” he said.
A variety of sources
At the pragmatic level, the ETF is working to include data from a wide diversity of sources to provide both the micro and the macro view.
“Both information streams – national and international data – need to come together for a single message. This is a first step to acknowledge that information coming from national sources is as important as international data,” Milovanovitch said.
Yet, while gathering data is an apparent challenge, countries also have more data than they often realise, as information is not always properly coordinated or collated in one place to be analysed.
There can be unexpected positive outcomes from the requirement to share information under the Torino Process, which needs strong communication between central and regional governments. In Ukraine (prior to the conflict), Tunisia and Kazakhstan the 'process of generating evidence turned out to be an instrument for facilitating dialogue between the centre and regions, and horizontally, such as with stakeholders. Educators appreciated this and used the collection of information to talk in ways they didn’t do before.
The ETF is working to support countries’ efforts in collecting and analysing evidence, and is actively allocating resources to ensure that the data generated is reliable.
“We care about involving our partner countries as if we don’t do that, and they don’t work with us, we have a problem,” he added.
Evidence-based active labour market policies essential across EU's neighbouring regions
The ETF frequently brings together representatives from its partner countries for knowledge-sharing regarding national policies, current priorit...
The ETF frequently brings together representatives from its partner countries for knowledge-sharing regarding national policies, current priorities and challenges, in the forms of workshops and panel discussions, one of which earlier in the year focused on the Consolidation of Evidence on Active Labour Market Policies and Transition to Work, which also addressed skills mismatches, the effect of digitalisation, and youth employment initiatives in EU neighbouring countries.
Pilvi Torsti, who had recently taken up the position of ETF Director opened the event held in May 2023 by drawing upon upon her previous professional experience as State Secretary of the Finnish Ministry of Economy and Employment noting:
"Responding to the social, economic and democratic challenges we face evidence-based policymaking in education and training within a lifelong learning is more important than ever to ensure wellbeing, prosperity and a self-determined future for all."
Research insights on evidence in the field of active labour market policies and transition to work in ETF partner countries presented by Cristina Mereuta, ETF Senior Human Capital Development Expert and Ben Kriechel, Senior Researcher at Economix Research & Consulting showed areas requiring improvement
"The frequency of statistical production, analysis, and reporting, as well as the shortage of staff allocated for these tasks needs to be addressed," said Mereuta, adding that the reliance on external projects is often a contributor to fragmented datasets.
"Labour ministries and public employment services in developing and transition countries should strive to develop comprehensive monitoring frameworks and collaborate with donors to align various projects targeting jobseekers under the same reporting umbrella," concluded Mereuta.
Thus, Representatives from Serbia, Bosnia and Herzegovina, Georgia, Kazakhstan, Kyrgyzstan, Jordan and Palestine, reflected on different policies and measures introduced towards evidence collection for active labour market policies given common challenges from a regional-geographical perspective in countries of Central Asia, the Eastern Partnership region, South Eastern Europe and the Southern and Eastern Mediterranean notwithstanding the diversity in terms of data availability, indicator definitions, and statistical practices related to active labour market policies.
Outlining the results of a study on skills mismatches in ETF partner countries and the preliminary results of the most recent European Skills and Jobs Survey led by the ETF's sister agency, Cedefop, in EU member states, Mircea Badescu, ETF Human Capital Development Statistician, explained that one-third of all employed adults, and up to 40% of young people are overqualified for the jobs they hold.
"When it comes to digitalisation and upskilling, most top-end workers feel both a threat to job security and a need for continued learning. However, worryingly, the findings also showed that the majority of unskilled workers surveyed appear unaware of the potential for change or any need to improve their digital skills," said Badescu.
Ratka Babic, Team Leader, Employment and Social Affairs Platform, Regional Cooperation Council, presented the use of data to advance the Youth Guarantee in the Western Balkans (Albania, Bosnia and Herzegovina, Kosovo, Montenegro, North Macedonia, and Serbia) which borrows from the EU's programme that aims to ensure that all young people under the age of 30 receive a good quality offer of employment, continued education, apprenticeship or traineeship within four months after completing their education.
Youth unemployment rate has reached the double of that for the overall population. One in four young individuals gets categorised into the NEET group (not in employment, education, or training). Some have abandoned their home countries, further fuelling the region’s “notorious brain drain” among “highly skilled and educated young people”. Others may work in the informal sector.
“They're not reached by the public employment services,” she said. “They're not actively looking for employment. They’re nowhere to be found.”
Data from youth unemployment studies in the Western Balkans has supported the development of a series of recommendations for the region developed in partnership with the ETF.
Returning to the words of ETF Director Torsti, experience shared from throughout the EU's neighbouring regions confirmed that developing skills intelligence with a good quality evidence base is one of the critical components to ensure that young people especially are reached.
Reflecting on the role of data in the ETF's work: Interview with Manuela Prina
The use of data in the ETF’s work has increased exponentially in recent years, with the evidence generated a core part of monitoring and assessm...
The use of data in the ETF’s work has increased exponentially in recent years, with the evidence generated a core part of monitoring and assessment, and for informing decision-makers. The rise in data use has come at the same time as the roll-out of Big Data and, more recently, generative artificial intelligence.
To reflect on the use of data and new technologies, the ETF held internal Data Days for its experts to get a better grasp of what is possible, and what the future of data use might be. Manuela Prina, Head of Skills Identification and Development Unit at the ETF, gives us an insight on what was discussed at the Data Days and the importance of data.
A fixation on data
The importance of data may seem evident to many, particularly when it comes to the development of evidence-based decision-making and how that impacts stakeholders, beneficiaries and the system at large. But to some, said Prina, the ETF appears “obsessed with data.”
"While experience and creativity are important components of our work, they are not enough in themselves. Our work has to be grounded in data, because it substantiates the advice we give and offers options for decision-making. This is a core value of our professionalism and credibility,” she said.
The internal gathering reflected on what the ETF does, how data is managed, and the approach taken. It was also an opportunity to look at innovations and define certain priorities.
“The idea was to be more familiar with the enormous quantity of information and data, and to talk about data in a larger sense. We are also in a transition period in terms of documentation management systems and databases. There was a need to take stock of the variety of data we have available, and to be more effective and efficient in making use of data in different services, without having to reinvent the wheel every time,” said Prina.
Data Days had three objectives. The first was to reflect on data at large, while the second objective was a deep dive into certain data work to enable ETF experts to better understand how such data can be exploited.
“This comes more naturally to some experts, who are more creative about data and its use for different objectives, from training to advice, to comparing experience, while for some others this process is more laborious. We wanted to share how some experts exploit and squeeze data to have the best outcomes, and to foster peer learning,” she said.
The third objective concerned generative artificial intelligence (AI), which can produce various types of content, including text and synthetic data in response to prompts. The public deployment of large language models (LLMs) like AI chatbot ChatGPT, over the past year has caused much excitement as well as consternation around what the technology can be used for.
“We pictured the future, in particular how we treat in-house knowledge and data creation, the management and use of data, and had a brainstorming on AI,” said Prina.
To enable participants to see what ChatGPT is capable of, a digital engineer programmed the tool to work with ETF products and data to generate output.
Participants reflected on how AI could speed up certain processes. For instance, in the future generative AI could lead to a shift in the ETF’s role as a producer of data to becoming a prompter and verifier. There is also the possibility of providing greater access to ETF data to external partners, who could then prompt their own questions.
AI and its future uses
Integrating AI into the ETF’s processes would however require the assessment of opportunities and the risks to be managed. “It is clear from the work we did during the Data Days that we are not there yet, in terms of using AI to create databases or to use AI to accelerate certain analytical work, but that doesn’t mean it will not happen in the future,” said Prina.
Regarding analytical work, AI could be used to assess certain areas, such as what skills are required in a certain sector.
“We could take 100 reports on say the energy sector in a certain country or area, and prompt AI to find answers from the reports. This might generate the acceleration of an output to six or eight months instead of two years. That could really have value when requests from stakeholders at a policy level are multiple, and we need to accelerate our work,” she said.
The uses of Big Data was a further area of reflection. Prina said that before the Covid pandemic, the ETF had started to assess online job offers in several countries – including Morocco, Tunisia, Egypt, Georgia and Ukraine – to reveal dynamics in the levels of skills required and educational levels needed in the labour market. “Until recently this looked almost futuristic,” she said.
In Ukraine for instance, online job vacancies “gives us quite a good overview about how the labour market is moving. You can see the moments of increased bombings or attacks, when there is a drop in the labour market, and when there are moments of calm, when there is an up-tick. It is quite revealing,” she said.
Prina noted that while such data needs to be considered in relation to the informal job market, big data can be useful in analysis.
“Going forward, this could be an integral part of how we monitor a labour market.
“How the ETF evolves in tandem with the data evolution is an exciting and challenging topic that we are fully embracing,” concluded Prina.
Spotlight on ETF podcasts: 3 episodes you can’t miss!
In January 2021 we launched Skills Factory, our very first series of podcasts. We dreamed of engaging in lively conversations with a variety of ...
In January 2021 we launched Skills Factory, our very first series of podcasts. We dreamed of engaging in lively conversations with a variety of educational and labour market experts, to bring new ideas and narratives to light. Two years and a half and 30 episodes later, we looked back at these conversations and compiled for you the three most popular ones…
Skills for tomorrow: What can you do to be ready for the post-pandemic world of work?
A discussion on the skills that will be needed in the future world of work and how people can prepare themselves for the constantly changing labour market. We also touched upon the new trends in recruitment processes and how the unemployed people can increase their chances of getting a job.
The emergence of the Covid-19 pandemic led to an unprecedented level of disruption in the world of education. For more than 1.2 billion students globally, face-to-face classes stopped. At least 463 millions of school learners could not benefit from digital and online learning. Being inclusive and ensuring access to education for all remains one of the biggest challenges for education and educators worldwide.
Education on the brink: Why a global vision is crucial
In this conversation we asked ourselves: What does it mean to have a global vision of education? Could it be a solution for fit-for-purpose and more inclusive education systems? What are the challenges and opportunities facing education and training worldwide, including the impact of artificial intelligence? ETF director, Pilvi Torsti, highlighted the global crisis facing education today and the issues that educators and learners need to address urgently.
What is the impact of digital platforms on work and skills? What are the challenges and benefits of platform-based employment? Is there a gender issue? Are platform workers at risk of economic inequality? This 30th episode of our Skills Factory podcast series highlights the growing influence of digital platforms (such as Uber, Airbnb, Deliveroo) on the labour market and the implications for individuals, businesses, and society. The conversation also emphasises the need for proactive measures to address the labour market evolving dynamics.