Home page model-en.doc

LEONARDO CESOS (Roma) - FGB (Roma) - Università Cattolica (Milano) - HBS (Düsseldorf) - SFS (Dortmund) Nene College (Northampton) - CIREM (Barcelona) - Pantheon Sorbonne (Paris) - Tempo (Paris) Commissione Europea DG XXII - Direzione B.1 Programma LEONARDO da Vinci - Strand III 2,a. - 1995 Contract n. I/95/1/233/III.2.a/FPC dell’1/12/95 Identification 3908 Firenze - Novembre 1997 A Model for the Evaluation of Continuing Training Quality Appraisal and Cost-Benefit Analysis  in Continuing Vocational Training Projects Strategy Planing Delivery Output

GLOSS   Introduction The Requisites of the System Creating an Evaluation Culture Criteria for Selecting The model

The quality of vocational training takes on special relevance in the EU debate. According to the European Commission’s White Paper ‘Growth, Competitiveness, Employment’, vocational training is defined as "... the catalyst of technological change...": this implicitly defines the quality requirements of training projects in terms of their success in achieving (i) the reduction of unemployment, (ii) the enhancement of business competitiveness, and (iii) the renewal of ‘traditional’ markets and creation of new ones. For these reasons the quality of training depends on changing expectations on training itself, which at the present stage seem to involve factors such as its contribution to 1) labour quality; 2) performance of business organisations; and 3) enhancement of the employment opportunities.

In this framework, a lively debate on vocational training appraisal and Evaluation, often seen as "... a yardstick of training quality ...", has taken place over the last years. In this perspective, there is a sound link between data collection and measurement, monitoring, Evaluation and certification, training needs analysis and assessment: these are all closely connected elements of the overall quality assessment of vocational training projects.

With particular emphasis on ‘continuing vocational training’, the partnership between nine leading European Research Institutes has developed a common strategy in order to provide common concepts, tools and procedures to improve the quality of vocational training processes and practices. Such strategy is centred on three research projects in the areas of (i) quality appraisal, (ii) vocational training certification, and (iii) training needs assessment. The partnership has presented all three of these projects to the European Commission, in the framework of the Leonardo da Vinci Programme .

The Model which is presented here below is the main product of the first project. After a period of validation carried out at each partner national level, the final version has been drawn up. A secondary product is a multilingual Glossary for use in the field of vocational training.

 

The aim of devising a Model to evaluate quality in continuing vocational training that can be applied to all European contexts is undoubtedly ambitious. In order to be really operational, and thus to meet the goal of being a truly European tool, the Model has been be characterised in the following way:

1. it is simple in its operation, and its purpose is easy to understand by training personnel and agencies throughout the European Union (EU);

2. it has a relatively unique meaning; that is, its basic concepts, tools and procedures are in principle simple, universal and well defined;

3. it is flexible and widely applicable to a number of different situations, sectors and organisational contexts as well as geographical and cultural areas;

4. once applied in one situation, it should be easy to reproduce in other situations;

5. for the above-mentioned reasons, the system is minimal, in the sense that the part strictly defined and universal should be limited to a core. At the same time, in order to cope with local and specific needs, is easy to attach a number of expansions and specific tools and procedures to this basic Model;

6. it has been designed to "capture" the quality of the training projects, with reference to their different agents and stakeholders, as well as to the various dimensions, moments, functions, etc., involved in their life-cycle;

7. finally, the Model focuses on both processes and outcomes.

The above list of characteristics has had consequences on the methodological options underlying the construction of the Model. The requirements were very stringent, so that only a very narrow range of choices was actually available.

The most viable solution was to develop a complete set of checklists that could pose the main quality questions and base the Model on a limited set of tools and procedures (selected by the user), designed to meet the requirements with the minimum necessary information. Such choice is quite common in the literature of training Evaluation models but, nonetheless, it brings with it considerable problems as to how to combine and aggregate the different measures provided by the various tools into one complex and very broad quality Evaluation.

The aggregation problems arising from the choice of basing the Evaluation procedure on the use of indicators can be overcome in two different ways.

The first is to proceed in the direction of formalising the Evaluation procedure, by making use of statistical techniques specifically devised to solve aggregation and ranking problems. This group of techniques mainly refers to multi-criterion analysis, a tool that is scientifically well established, but full of implications on the way in which Evaluation activities should be organised. The main implication is that their use is open only to high-skilled, specialised personnel, so that Evaluation becomes a separate function in the operation of the training agencies, performed by high wage specialists, living in a world of their own. It is, in other words, a "technocratic choice", not suitable for small firms and, above all, for a great bulk of today training activities, that increasingly require a high degree of collaboration between trainers, training organisations, trainees and firms in order to be effective.

The second alternative to overcome the aggregation problem is to leave the information synthesis to the judgement of the people taking part in the training process (‘self-assessment’). This choice has the advantage of drawing on the expertise of people actually involved in the project. It has the disadvantage of being open to the risks of arbitrary discretion, and therefore of weakening the uniqueness of the procedure and the transferability of the results. In order to minimise such risks, many things can be done, most of which refer to quality assessment and certification, in the direction pioneered by ISO 9000 or EQA standards. But, even before this, in order to minimise the discretion risk, it is necessary for the training agency to build an Evaluation process that relies on a sound and widespread "Evaluation culture", based on the practice of self-assessment. This can be built by complying to a standard procedure made up of complete checks and controls aimed at quality improvement.

 

In the Model, the Evaluation process tends to coincide with the training process itself, as it becomes a fundamental part of all the activities and skills involved in preparing, developing and delivering the training project. Furthermore, the person responsible for the self-assessment is no longer a specialist who lives and works separately from the training activities. On the contrary, the Evaluation skills become common practice, an "invisible asset" of the training provider (which itself becomes a "learning organisation").

Taking this second option, the Evaluation procedure becomes a process, shaped on the characteristics and needs of the training agency and related to all stage of the training project life-cycle, more than a formalised tool to measure the quality of a specific aspect of the training action. In this sense, an evaluating process can develop and make use of many Evaluation tools, each of them relevant to the assessment of the quality from a particular perspective (dimension, stage, function). Each tool must be connected to the others and to the definition of the overall quality of the entire training project, with the aim of developing a common "Evaluation culture", shared by all the stakeholders involved in the event itself, and continuously enhanced through collaboration.

 

 

According to the Evaluation philosophy expressed above, the composition of the checklists, that provides the information for the Evaluation, were managed in order to meet the following criteria:

1. to be suitable for implementation by all of the stakeholders (with relevant minor adjustments) by taking into account the points of view of the provider, the end-user, the trainee and eventually other stakeholders (national and European funding bodies, others);

2. to cover the main areas of the training project life-cycle, that is: (i) its stages (preparation, design, delivery, follow-up), (ii) its functions (marketing, training development, project management), (iii) its dimensions (financial, organisational, didactic, economic), etc.;

3. to capture the fundamental elements of training quality, such as the relation of the provider to the end-user and the quality of its training supply, the efficiency of the single training projects, and their effectiveness, both internally (for the provider) and externally (for the trainee as well as for the client).

 

The Model is an articulated set of ‘light and flexible’ checklists aimed at evaluating the overall quality of vocational training projects with special reference to continuing vocational training. In this respect, the Model is targeted at being used at every stage of the training project cycle: (i) in the ex-ante stage, (ii) during the monitoring activities of the project, (iii)  the final Evaluation and (iv) for the ex-post stage.

The basic postulate for the design of the Model is that training is a process and Evaluation is an inherent component of the training project. The project manager himself must be an evaluator and thus there is the actual need, a priori, to define reference standards and parameters in order to weigh up the quality of the training provided and its improvements.

 

The reference user of the proposed Model is the provider. Nevertheless the Model may be used by any of the stakeholders involved in the vocational training project, either the companies or public administrations ‘end-users’ of the implemented activities, or the trainees, or entities financing the project. In this case questions can be adapted to specific perspective needs of the specific user.

The object of the Evaluation activity is the single training project. In any case, however, the Model analyses even the productive processes of the Training Agency in order to contextualise training activity with relevant indications about the body that is providing the training services.

The Model can be organised into a table which integrates, in a cascade connection, project activity, logical stages and relevant stakeholders who may be interested in some way in appraising and evaluating the project: