From the onset, my assumptions were proven wrong. As Rob van Son stated in his opening post:

However, the most remarkable thing about any ICT4E assessments to decide on the introduction of ICT in education would be their uniqueness in history. One reason such assessments are so scarce is that there are few (if any) historical examples of assessments of any kind done before the introduction of an educational reform. Even less examples where the outcomes of the assessments really mattered in decision making.

This opinion was backed up in the comments with the general consensus similar to John LeBaron’s comment that while assessemnts themselves are not bad, but we don’t have the right assessment tools and their outcomes would be misused anyway.

Mary Hooker’s response was more in line with my thoughts – she opened with:

Yes we do need to assess ICT4E initiatives more particularly when we are working in environments with scarce resources as in the developing world where investment in ICT can constitute what Unwin (2004) describes as a ‘wasteful tragedy’ if it is not managed and utilized properly.

But then surprisingly, quickly agreed with many of the issues raised by Rob van Son – ICT4E assessments don’t capture outcomes very well, and are often disregarded because the entrenched practices of conventional schooling has strict rules that constrain and retard transformational curricular innovation, especially ICT integration.

Yet, Clayton Wright brought up a great point:

From a western perspective, the benefits of technology are obvious. But in the developing world, perhaps only the benefits of mobile phones, ATMs, and radio seem obvious. When given a choice, local officials would probably spend funds on providing clean drinking water, toilet facilities, medicine, and seeds for crops rather than spend it on ICTs. Local educators would want more teachers not computers. Thus, I believe that evaluations are necessary to demonstrate to the local officials and national policy makers that ICTs are worth the investment. They need to know what local problem(s) ICTs can address or opportunities that are possible.

How can we assure them that technology should be on an equal footing with other educational investments then? Can we really take Rob van Son’s title, “Stop Wasting Children with ICT4E Assessments” to heart? Is he right when he says:

There is little hope that ICT4E assessment as it is organized today will uncover anything that will actually influence educational practices. History is plainly against us. A full blown assessment about “Is [fill in ICT4E solution] a cost-effective improvement” will take years to complete, cost serious amounts of money, and will be irrelevant when published. The worst effect will be the delay which will deprive yet again several cohorts of school children in poor countries of adequate education.

In addition, if all teaching methods that are based on “transmitting knowledge” instead of “recreating knowledge” (aka Constructivism) fail to change the naive preconceptions of students, how does ICT then impart a better knowledge creation system than an increased focus on better teachers and more involved instruction? Rob van Son responded with a reasonable argument:

I think the old Oxford model of a lector and a few students is best. What ICT can do is help in situations where there is a lack of adequate teachers. But such a support cannot be “proven effective” using standard evaluation procedures based on current grading practices. On the other hand, evidence based educational methods are still not generally accepted. So I suggest to try to find a common ground where both the “establishment” and “evidence based research” can meet to improve education for disadvantaged schools. This is much better than waiting for an (elusive) consensus on evidence based educational reform.

In her second post, Mary Hooker agrees with Rob – there are inadequacies of evaluating the use of ICT and its potential for transformational innovation in education systems that are intent on simply harnessing it for maintaining the status quo. It doesn’t help that ICT is also multifaceted, with effects and impact beyond the expected, event by experts.

Worse, ICT is also very fast moving. With the quick change in technology (netbooks, mobile phones, Internet access) over the last few years, long-term studies of ICT impact can be obsolete before they are even finished. To this Mary Hooker responds that the focus should be not the devices then. ICT does not exist in isolation but within a larger socio-cultural context of the school and education system.

Which to me, brings us to classic Change Management – the systematic process of creating organizational change.

In that context, should ICT4E assessments ignore the technology – it changes to fast and is too hard measure its impact – and focus on assessing the ability of educational systems to best adopt ICT usage to their goals? A shift from “device X helps Y amount in reaching Z learning outcome” into “school system A has B ability to incorporate C amount of technological change within its current resource constraints” and then leave the school systems themselves to experiment with different technology?

It seems that it does not matter so much in what you excel, only that you excel. Not coincidentally, this was also the main driving force behind the Chinese imperial examinations in the previous post.

In academic circles, there is a lot more interest in really measuring performance. But in these studies, very specific questions are asked in relation to bounded problems. Nothing like, “Are computers useful?”, because such a question is unanswerable in principle. There are well researched evaluation methods, see the companion post by Mary Hooker. A lot has been learned how children respond to formal education, and how they will learn (better). So the question is, why are these methods not used?

To get a glimpse of an answer, it is illuminating to listen (literally) to David Hestenes who has done ground braking research on the understanding of basic physics in students. His talk “Naïve beliefs about physics and education” is available on-line as a presentation and audio talk.

Simplified to the bare basics, almost everyone seems to believe that education is about the transmission of a substance, called knowledge, to the memories of the students. The schools function as a retail outlet of knowledge. Evaluation of education centres around determining how much of this substance ends up inside the heads of the students. Research has shown that students can indeed reproduce a lot of the factoids sprinkled in the teaching and textbooks when tested. However, when tested in ways that require real understanding of the basic concepts, a majority of students fail completely (see examples in the slides).

His research brought David Hestenes to the conclusion that all teaching methods that are based on “transmitting knowledge” instead of “recreating knowledge” (aka Constructivism) fail to change the naive preconceptions of students.

Summarizing the above in combination with the previous post, there is little hope that ICT4E assessment as it is organized today will uncover anything that will actually influence educational practices. History is plainly against us. A full blown assessment about “Is [fill in ICT4E solution] a cost-effective improvement” will take years to complete, cost serious amounts of money, and will be irrelevant when published. The worst effect will be the delay which will deprive yet again several cohorts of school children in poor countries of adequate education.

I think we must take a common sense approach instead. While waiting for the dissemination of the results of scientific research into the general population we should start with trying to find common ground with teachers, politicians, and parents. That is, seek for approaches that will allow all involved to reach a consensus on that will benefit children now.

Computers alone have benefits

Computers are useful in disseminating information, eg, electronic books, libraries, wikipedia, and as communication devices, eg, email, IM, video. They are great for writing and calculations and can greatly improve collaborative efforts. They are also great at stimulating children to read and write, eg, email, stories, their own blog, to explore their society and the world, and to get exposed and experienced in new languages. And they require and exercise skills that will be valuable in the workplace later.

There might be problems in languages with very little digital content for children. However, experience taught us that the existence of a large “market” of on-line computers is quickly followed by content. Even if the content has to be free (as on the Internet).

Computers in education have more benefits

One really unique benefit of computers is that it is possible to set up applications that allow students to practice skills that could before only be practiced in the presence of a teacher. Teachers can then spend more time on children who need personal attention.

This has been used in language learning for decades. Speaking practice, and reading and writing assignments, have been automated before. Students listen to recordings and record their own speech. Just listening to recordings of your own voice helps you correcting mistakes. There are currently even (limited) applications where a computer application can actually help students correct their writing and pronunciation.

As an example, I will plug here a Free Software (GPL) project to learn Mandarin tone distinction in which I participated: SpeakGoodChinese.

Enough with Assessments – Implement Already

I conclude that schools that lack resources like books and libraries, and over all, the required number of qualified teachers, will greatly benefit from implementing sensible ICT solutions that substitute for these shortages and improve teachers’ effectiveness.

All parties involved seem to agree that ICT solutions could ameliorate a least some of the problems in resource poor schools. But this approach also implies that ICT solutions should not be restricted to the classroom. Practice, studying, and reading should be done at home, collaboration and communication is done everywhere. What use is it to have school books and email on a computer when the student has no access to the computer?

There is obviously one caveat: ICT can only be useful if the (real) total cost of ownership can be made bearable.

I’ve heard this refrain repeated often since then, and not just by those promoting technology in schools. Its a equal thought from those that feel geek lust is clouding our judgment and we should focus on teachers, not technology. Its also promoted by those that point out changes to educational methodologies have often happened by force of will, not empirical results.

Now, Nicholas Negroponte is putting forth the idea that one computer per child is like electricity – such an accepted benefit for society that we’ve moved on from discussing its impact to just looking for the right models to fund it.

While we may have differencing opinions on OLPC or its benefits, the basic questioning of ICT4E evaluations is compelling. Starting with the simple question of “Do we need assessments?” we can branch into related questions that examine the basic assumptions we hold dear, like:

1. Are ICT4E assessments effective in measuring outcomes?
2. Do we even have the tools to tell if they are effective?
3. What tools are those?
4. Are we really using these assessment tools correctly?
5. And regardless of the outcomes, should we really wait for long-term results, or should we implement ICT4E deployments now, as the case is compelling enough already?

For November, the Educational Technology Debate will focus on assessments of ICT initiatives in education – how we can both validate them and use them correctly to improve ICT4E overall. For discussants we’ll be joined by the following experts:

• Mary Hooker
  Mary Hooker is an education specialist with over 30 years experience working in the educational sector in Ireland and Africa. Since 2007 Mary has been working with the Global eSchools and Communities Initiative. Mary is currently engaged in studies for a Doctorate in Education with Queen’s University Belfast, Northern Ireland.
• Rob van Son
  Rob van Son was a subject in early Computer Supported Education experiment in the 1980′s, and since worked on everything from small 8088 PCs and the first Mac to modern multi-core file and web servers. Rob is a linguistics expert with a focus on integrating information in spoken communication for Universiteit van Amsterdam. Rob has a PhD in linguistics.

Please join us for what we all expect to be a lively and informative conversation exploring assessment validity and tools for ICT4E. Your input can start right now in the comments below, and Mary and Rob will post their opening remarks beginning Monday, November 9.

Don’t miss a moment of the action!

Subscribe now and get the latest articles from Educational Technology Debate sent directly to your inbox.

.

To bridge this gap, the Education Technology Debate invites thought leaders and opinion makers that directly influence the confluence of technology and education to focus on constructive conversations and open discussions across groups as much as within them.

Become an Educational Technology Debate Discussant

If you’d like to join in building the Educational Technology Debate conversation, please contact us with your ideas, comments, and suggestions on ways to make ETD more informative and engaging.

Previous Educational Technology Debate Discussants

in Assessing ICT4E Evaluations:

Mary Hooker
Mary Hooker is an education specialist with over 30 years experience working in the educational sector in Ireland and Africa. Since 2007 Mary has been working with the Global eSchools and Communities Initiative. Mary is currently engaged in studies for a Doctorate in Education with Queen’s University Belfast, Northern Ireland.

Rob van Son
Rob van Son participated in early Computer Supported Education experiment in the 1980′s, and since worked on everything from small 8088 PCs and the first Mac to modern multi-core file and web servers. Rob is a linguistics expert with a focus on integrating information in spoken communication for Universiteit van Amsterdam. Rob has a PhD in linguistics.

in ICT4E Sustainability:

James BonTempo
James BonTempo is the Learning Technology Advisor for Jhpiego, an international non-profit health organization affiliated with Johns Hopkins University. He is responsible for strategic planning for the integration of Information and Communication Technology (ICT) into pre-service education and in-service training programs. He also leads efforts to design, develop, implement and evaluate ICT initiatives in both arenas.

Atanu Dey
Atanu Dey works as the chief economist at NetCore, a technology firm in Mumbai. His area of interest are the use of technology in education, economic growth of India, and the development of rural populations. He worked in product marketing for several years at Hewlett Packard in California, before receiving his PhD in economics from UC Berkeley. He developed a model called “RISC — Rural Infrastructure & Services Commons” while a Reuters Digital Vision Fellow at Stanford.

in How Can ICT in Education Excite Girls and Boys?:

Alexa Joyce
Alexa Joyce is a specialist in education technology with European Schoolnet. She has consulted for UNESCO Bangkok Asia-Pacific Bureau for Education, UNESCO International Institute of Educational Planning and for the OECD Centre for Educational Research and Innovation. She has a Masters in Biological Sciences from the University of Oxford and an MBA from Solvay Business School, Brussels.

Brooke Partridge
Brook Partridge is CEO and founder of Vital Wave Consulting, which she created to further emerging markets as a new discipline in business management. Previously, she was the Business Director of the Emerging Market Solutions Organization at HP where she lead HP’s first technology solutions for developing economies. She lectured in the Department of Spanish and Portuguese at Stanford University and holds a Master’s of Pacific International Affairs from UC San Diego.

Karen Coppock
Karen Coppock, PhD is Vice President of Vital Wave Consulting with over a decade of experience in strategic business planning for emerging markets. Previously, Dr. Coppock served as the Director of Industry Collaboration for the Reuters Digital Vision Program at Stanford University, and also held positions with Telcordia Technologies, Williams Communications, INTELSAT, Pacific Bell, AT&T and Harvard’s Center for International Development (Information Technology Group), Santa Clara University’s Global Social Benefit Incubator and the US Peace Corps. She received her Doctoral and Master’s degrees in international business from the Fletcher School, Tufts University.

in Creating Electronic Educational Content:

Richard Rowe
Richard Rowe is the Chair and CEO of the Open Learning Exchange, a network of nation-based NGO’s committed to achieving Quality Universal Basic Education by 2015 . Dr. Rowe has served as Director of Test Development and Research for the West African Examinations Council, Associate Dean of Harvard’s Graduate School of Education and a member of the World Economic Forum’s Global Agenda Council on Technology and Education.

Angus Scrimgeour
Angus Scrimgeour is the President of the International Association for Digital Publications, a program to provide university students and academic staff in developing countries with affordable access to e-books, and support for the identification, development, and effective use of open access e-learning resources. Mr. Scrimgeour is also a former Vice President of the World Bank Group a member of the Knowledge and Learning Council.

in Individual and Communal Computer Usage:

Walter Bender
Walter Bender currently heads Sugar Labs, focusing on the award-winning Sugar Learning Platform (download it now). Previously he was president for software and content development at One Laptop per Child, and is on leave from MIT, where he was executive director of the MIT Media Laboratory.

Mark Beckford
Mark Beckford is currently Vice President of Global Business Development at NComputing, Inc, whose virtualization software and hardware allows multiple users to work off a single computer. Previously, he led diverse global teams at Intel to extend its market leadership and promote growth in new and emerging markets.

in Mobile Phones and Computers:

Dr. Robert B. Kozma
Dr. Kozma has directed or co-directed more than 25 projects that have examined the impact of ICT on teaching and learning and developed advanced computer environments for education.

Michael Trucano:
Mike Trucano is the World Bank’s Senior ICT and Education Policy Specialist, providing support to World Bank education projects with ICT-related ‘components’, and is involved in a variety of research activities.

in Educational Vision and ICT in Education:

Tim Kelly
Dr. Tim Kelly is the Lead ICT Policy Specialist at infoDev, at the World Bank in Washington DC, where he has responsibility for access for all and for mainstreaming ICTs for development. He was previously Head of the Strategy and Policy Unit (SPU) at the International Telecommunication Union (ITU).

Wayan Vota
Wayan Vota is a is a technology expert focused on appropriate information and communication technologies (ICT) for rural and underserved areas of the developing world. He is currently the Senior Director of the Inveneo Certified ICT Partner Program, publisher of OLPC News, and hosts the Technology Salon.