arrowrightarrowleft1973-77 NDPCAL Evaluation

Evaluation was an important part of NDPCAL's approach. It set up two idependent evaluations as well as building in evaluation through its regular project reviews and assessments. The two independent evaluations were an educational evaluation carried out by the University of East Anglia and a financial evaluation by Peat, Marwick, Mitchell and Co.

quoteNDPCAL had an "unusually prominent commitment to evaluation procedures" (MacDonald and Kemmis, 1976).Project evaluation

The National Development Programme for Computer Aided Learning placed great emphasis on evaluation and made it a contractual requirement for every one of its thirty projects.

As Hooper (1975) makes clear, evaluation was important to the national programme for two very good reasons. First because it was, for its time, at £2M a very costly development programme happening at a time of severe public sector constaints. Second, the approach taken by the programme was to develop projects through different phases of feasability, design and development. The Programme Directorate used evaluation to step their funding so that project progress was dependent on positive outcomes from the four-monthly reviews. It was also a contractural obligation for each project to have its own internal evaluation linked to Programme Committee appraisals.

Alongside this system the programme appointed two independent evaluators as an additional resource.

Educational Evaluation

In terms of its two major aims of institutional assimilation and generalisation from its experience the National Programme may be judged broadly successful. (MacDonald, 1977, p.188)

This was carried out by Barry MacDonald and a team from the University of East Anglia. Barry MacDonald was initially contracted as a consultant but after six months a £94,303 contract for a three year evaluation project was awarded to him and a team from UEA who called themselves UNCAL ( 'Understanding Computer Assisted Learning) as a conscious play on the TV series "The Men from UNCLE", hoping to remove the tension of evaluators acting as spies from the centre (MacDonald and Kemmis,1975). As MacDonald and Kemmis (1975) explained UNCAL was in part a 'primary' evaluation, collecting and presenting its own data, and in part a 'meta' evaluation, assessing the evaluation efforts and reports of those within the National Programme who have evaluation responsibilities.

MacDonald (1977, p182) reports some findings about CAL in general:

  1. It is the versatility of the computer as an aid that assures its educational future
  2. CAL, like most innovation, provides an add-on experience at an add-on cost
  3. Much of the learning seen within NDPCAL fell into the category of higher-order learning
  4. CAL is a demanding medium for learning - virtually guaranteeing the students engagement
  5. Some forms of CAL enforce a strict role of learner on the student - this may need to be complemented by other forms
  6. CAL offers the student uninhibitated learning opportunities within a 'privacy of risk'
  7. Learning may be inhibited by interface problems - where the student needs to put extra effort into keyboard skills, learning new computer protocols
  8. Current CAL still requires more adaption of the student to the machine
  9. Students like working on CAL but are drustrated by technical problems
  10. CAL is change-oriented not efficiency-oriented
  11. CAL supports teacher development since it encourages a team approach
  12. At present CAL development requires access to high level computer expertise.

 

Financial Evaluation

The financial evaluation was carried out by John Fielden of the management consultants, Peat Marwick, Mitchell & Co. As Fielden explains (1977, p.190) the decision to consider financial implications was down to Richard Hooper whose rationale for a financial evaluation was mainly based on the fact that CAL was a high cost innovation and that most evaluations, particularly in the USA, had avoided considering cost-implications.

Fielden reported (1977, p.195) some tentative but interesting conclusions froim his study:

  1. CAL will always be an extra cost
  2. There are no realizable cash savings or benefits from CAL
  3. Claims that CAL will 'save' academic staff time are oversimplified and unjustifiable
  4. Only in very few cases will students sit at CAL terminals for than 20 hours each per annum
  5. Precise statements of the cost of CAL are not possible owing to the large number of significant variable factors in the cost calculations
  6. The costs differs by orders of magnitude according to the level at which they are drawn - national, institutional, departmental
  7. The marginal cost to a department of taking on CAL is usually low
  8. The evaluation calculated 'national or total cost per student terminal hour' in the range £4-£15
  9. By comparison the cost of conventional means is in the range £0.60-£2.50 per student hour
  10. Realistic targets for the use of terminals are in range 500-750 hours per year
  11. The time taken to develop science packages varies between 200 and 400 hours
  12. Program exchange schemes can achieve significant economies
  13. Inter-institutional development has been a success leading to substantial savings
  14. Large scale applications of CAL require full time staff and regular computer time.

 

There were clearly some tensions both between the projects and the independent evaluators.

References

Hooper R., 1975, Two years On, National Development Programme in Computer Aided Learning, Report of the Director, London: CET

Hooper R., 1977, An Introduction to the National Development Programme in Computer Assisted Learning, British Journal of Educational Technology, 8-3 p165-175.

Macdonald, B and Kemmis, S (1976) Macro-project and Meta-evaluation – the UNCAL Experience. Research Intelligence, 2. pp. 36-39 (online version)

MacDonald B., 1977, The Educational Evaluation of NDPCAL, British Journal of Educational Technology, 8-3 p176-189.

Fielden J., 1977, The Financial Evaluation of NDPCAL, British Journal of Educational Technology, 8-3 p190-200.