We recognize the inadequacies of evaluating the use of a tool and its potential for transformational innovation in education systems that are intent on simply harnessing it for maintaining the status quo. As Rob (on the other side) observed ‘any real assessment of educational reform requires a new reflection on what skills and knowledge the children are supposed to acquire at school’.

And so in my response I would like to revisit the question presented by Wayan and reflect a little more on its parameters. I would also like to draw on commentary from both sides of the discussion (quite a lot of comments on your side Rob) to tease out some of the issues.

Do we need to assess ICT4E initiatives?

Rob notes that most reforms have historically been imposed without scientific support but rather on political prejudices. However the sense of fatigue with the failure of education reform syndrome is perhaps changing as we migrate into a 21st century information age. And as we do so, we are witnessing a growing discrepancy between school and the ‘outside world’ – where information, knowledge, innovation and creativity are replacing the traditional sectors of commerce and industry – and where new technologies are changing the way we interact, communicate, socialize and network.

It is a world where mobile connectivity is becoming commonplace and where digital literacy is a critical tool for social interaction, knowledge exchange and construction. If schooling fails to transform itself, it may be transformed albeit haphazardly by the technological transformation outside its gate – and perhaps in a way that may be detrimental to learning.

There is also the challenge of digital divides, both between societies and within societies – with access denied to the poorest and most marginalized. ICT is seen as bridging such major divides. There is thus a renewed sense of urgency, despite the fatigue, for systemic ICT investment and reform to provide all learners with skills they will need for meaningful participation in the economic, social and culture life of new knowledge-based economies and societies.

In such scenarios of massive large-scale investment and reform, assessments are needed to hold systems accountable. Assessments can also provide policymakers with the gateway they need to direct systemic change. As Clayton observes in his comments ‘evaluations are necessary to demonstrate to the local officials and national policy makers that ICTs are worth the investment’. They can help them to identify factors to best influence ICT impact (changes in curriculum, pedagogy, assessment, teacher training) and well as the barriers to ICT use, such as lack of skilled support and adequate infrastructure.

If we assess, how do we do it?

Juan in his commentary describes his skepticism as to the relevance of some of the ICT evaluations he has come across over the years – in particular studies on proprietary software where the emphasis is more on evaluating technology than learning. He notes the lack of comparison with alternative activities and for cost effectiveness. John also comments that effective assessments have not been designed. On cost effectiveness John finds shocking a US study illustrating a lack of empirical research in an area where billions of dollars have been invested. What is particularly ‘unsettling’ is the notion that politicians don’t seem to care.

Yet I wonder John if it is a question that politicians don’t care or that there is a sense of exasperation with the lack of defined mechanisms for informing decision making for such a massive scale of investment and change? Scheuermann, Kikis and Villalba (2009) discuss the lack of clear information in most studies about the multifaced effects and impact of ICT on the learner and learning. It is a situation that is ‘especially unsatisfying for policy-making stakeholders that aim at defining evidence-based strategies and regulatory measures for effective ICT implementation and efficient use of resources’ (ibid. p. 1).

There are calls for more widely accepted indicators and methodological approaches to assess inputs, utilization and outcome/ impact of ICT integration initiatives in order to address this gap (Trucano, 2003; Blanskat et al. 2006; cited in ibid.). Yet there still remain limitations in these approaches on measuring the impact of ICT use – as they often represent a snapshot – a one time, one level approach. Ian comments on the ‘imperfection’ of the data collection in such evaluations more often conducted to appease funder insistence for seeing ‘educational’ results. He also draws attention to the difficulty in attributing the said results to the ICT intervention.

A more powerful approach is the use of indicators within development models of ICT integration in education – to study the progressive phases through which teachers and students adopt and use ICT. Morel’s Matrix is an instrument that can be used for evaluating the degree to which ICTs have been integrated in an educational system through four distinct successive phases: a) emerging, b) applying, c) integrating, and d) transforming.

In GeSCI we have developed an ICT-Education matrix to assist our partners in focusing on what teachers and learners actually do when they use ICTs in schools and institutions through each of the four successive phases (Figure 1). Such models when used to guide Planning, Monitoring & Evaluation (PME) in combination with the indicators approach can offer clearer outcomes on what the integration of ICTs in education should look like at each development stage.

Figure 1: GeSCI ICT- Education Matrix

I like Mark Beckford’s observation in that the key to assessment is to keep it ‘simple but useful’. We hope in developing PME tools as the ICT-Education matrix for our partners that we can do just that.

Related Link

Scheuermann, F., Kikis, K. & Villala, E. 2009. A framework for Understanding and Evaluating the Impact of Information and Communication Technology in Education Available online & accessed 23 November 2009

Reading the resulting commentary, I’d like to declare success. I feel we have found a new model, that is an child of these two parents, mixing genes of both to create a new, better ICT4E model where multiple platforms plus a single learning environment equals more educational beneficiaries.

Multiple Platforms

From the beginning, this discussion recognized that different communities allocate their limited resources differently. Some will have the resources for high saturation of computing tools, while others will not. In fact a single community may have multiple computing models within its own educational system, based on age, maturity, and progress of its students. Mark Beckford gave us a great example:

In Macedonia, NComputing deployed over 100,000 virtual desktops which made Macedonia the country with the greatest density of computers to students. But Macedonia also issued a tender to deploy a smaller quantity of netbooks. They cannot afford mobility for all students, and yet even at 1:1 desktop computing they see the advantages of mobility.

So educators need not feel that its a either-or decision. Communities can have both personal and shared computing environments in the same school. And as Alex Van de Sande points out, its not the technology that matters, but the way educators use it:

The most important is that in either case, the experience must be saturated, shared and free. The shared PC lab experience, where there are many peers around you who can quickly teach you is invaluable. But all that is nullified by models with restrict hours and usage rules. The 1:1 laptops are great on the fact that the freedom from “this is how you are supposed to use this” rules make you experiment more. But doing it alone may lead to the laptops being used for more private entertainment – like gaming.

In that context, a mixed environment may be the best choice. One where students use computer labs in the school setting, where usage can be monitored and directed, and on a more personal basis when outside the school.

Single Learning Environment

With all these platforms, there quickly becomes the need to maintain a homogeneous learning environment. One familiar look and feel that follows the child as they access different platforms during the day and their education. Walter Bender is working on such an environment with Sugar on a Stick.

This USB memory stick-based educational software platform is based on the principles of cognitive and social constructivism, and contains its own operating system (Fedora 11) so it can be run from just the memory device itself – no hard drive or specific operating system needed.

Caroline gives us her thoughts on the advantages of such an approach:

Sugar on a Stick should make mobility cheaper. If kids take their sticks with them they can use them on clusters of computers in day care centers, community centers and at home if the parent has a computer. Thus by using computers in different places in their environment they can get quite a bit more hours of computing time per week and their desktop and all their work is mobile. I wonder if we can run numbers on that type of solution, and maybe instead of running them per machine, run the numbers to compare $ per hour the child uses a computer.

And Walter Bender confirms that the Sugar on a Stick approach can be complimentary to current and new platform investments:

It is great that there are many different such platforms being developed: a diversity of hardware configurations is necessary to meet the demands of schools, budgets, and cultures. But one can remain agnostic about hardware platforms and configurations, while providing a great learning experience, better utilizing the installed base of computers while tapping the potential to engage every child in critical thinking, arming them with the complementary tools of science and the arts.

More Beneficiaries

So with a single learning environment on multiple platforms, let’s start talking about the real numbers of beneficiaries. Either in school or at home, let’s move away from the assumption that only the child assigned to the computer is using it. At any given point in time, children are usually in groups, learning from each other. In fact, it seems children learn best when learning with others. Alexa Joyce notes that:

Sugata Mitra’s research suggests that groups of 3-4 children per computer can be more fruitful than 1:1. In groups of such a size, children readily exchange ideas and knowledge about the topic they are investigating, as well as the computer itself.

Let’s not stop at children. When they are home, they are not necessarily alone. Siblings, parents, and others are nearby and they too hear the call of a glowing screen as Walter Bender tells us:

A study done by Claudia Urrea in Costa Rica found that the majority of parents use the computer at home for their own learning – a further leveraging of the investment. Other programs, where it is infeasible to let the children travel between school and home with a computer, have instituted “technology goes home” programs – a subsidy to parents to purchase new or used equipment to have in the home. The goals of such programs have been to bridge learning from school into the home and to engage parents and siblings in the school community and in their own learning.

This new usage model, where a single learning environment over multiple technology platforms, is used by more than just students, may change the way in which we think about costs, which is one of the largest barriers to adoption, just after plain inertia & fear of change.

Costs are often calculated on a per-student basis. Yet, with siblings and parents as co-learners with their children, education leaders may change their mindset around platform costs. Instead, divide platform costs by student + 1 parent & 1 sibling. Yet also reduce costs, as there is only one software system to maintain.

And so I say we have a whole new ICT4E model with multiple platforms, a single learning environment, that empowers more beneficiaries to learn at a lower cost. A success, eh?
.

“OLPC’s key development in our view is the software – who cares about the hardware as long as it gets the job done inexpensively?”

Nonetheless, it is still worthwhile discussing the advantages (and disadvantages) of various hardware approaches to deploying learning software such as the Sugar Learning Platform. I am ignorant of the current total-cost-of-ownership of the various options available; I leave this complex calculation to Mark Beckford and assume that he will take into consideration not just the lifetime cost of the hardware, but also training, administration, maintenance and repair, energy and disposal costs.

Other important factors, not often considered, include where the cost is born and to whom the investment brings benefit. For example, can support be delegated to enterprising high-school students (or some other local community resource) or must it be outsourced (out of district or even out of the country)? All other factors being equal, it matters where you make the investment.

What I will discuss here is “reach” in terms of who is learning, what is learned, and how it is learned. If we set our sights higher than using the computer to access information and learn “21st century computer skills”, the choice of model needs to be taken into consideration.

1:1 Computing in the USA

Of course, the principle advantage of a saturated model of computing is that the computer is available to every child at all times. In a middle school in Massachusetts, the children use their laptops to write in every class, including music, dance, and gym.

The children learn computer skills, but they also learn to use computation to enhance other curricula goals, through demonstrations, projects, and critiques. Learners develop craft, engage and persist, envision, express, observe, reflect, stretch and explore, and understand. Computation is used as a critical-thinking tool in the context of open-ended exploration and discovery, going beyond the use of the computer as a tool of instruction.

It is often the stated goal of one-to-one computer programs that laptops go home with the children. This is desired in that not only does child ownership result in less breakage – statistics from the Maine laptop program bear this out – but also, the computer, which otherwise was only available at school is now available to the entire family.

A study done by Claudia Urrea in Costa Rica found that the majority of parents use the computer at home for their own learning – a further leveraging of the investment. Other programs, where it is infeasible to let the children travel between school and home with a computer, have instituted “technology goes home” programs – a subsidy to parents to purchase new or used equipment to have in the home. The goals of such programs have been to bridge learning from school into the home and to engage parents and siblings in the school community and in their own learning.

Mobility and form factor also play a role. In a trip to rural Thailand, I got to see first hand how netbooks were being used to change what is learned and how it is learned. Two examples really resonated with me. In one example, the children organized a orchestra in which they combined computer music with the traditional instruments of their village. They sat on the floor with their parents, performing together.

Here, the form factor of a netbook computer was an enabling element to their physically being able to interweave the new and the old into a community and musical synthesis. In a second example, the children took their netbooks into the forest to capture the sights and sounds of their habitat. In some schools, students attach sensors to their netbooks so as to capture data such as temperature “in vivo”. Back in the classroom, they process these raw data as part of their science curriculum.

Mobility has also played a role in extra curricular activities. For example, an elementary-school student established an after-school photography business in a community for Afghani refugees in Pakistan, using a netbook.

But the most dramatic example is one that Benjamin M. Schwartz recently brought to my attention. Ben is the author of the Distance Activity, which measures the distance between two OLPC XO-1 computers through a measurement of sound-wave prorogation. It is incredibly engaging and has been used in a wide variety of situations, including gym class, where it has be used to introduce measurement and calculation to activities usually not considered academic.

Acoustic Tape Measure

Ben recently blogged about Stephanie Selvick’s experience with the use of Distance in a school in Senegal. A previously skeptical teacher enthusiastically began to brainstorm about connections to traditional curricula goals, such as “assignments for his kids to measure the distance around their homes or rooms” and to “figure out area from those numbers.”

As was reported from Senegal, “The goal of making teachers the experts for each other felt underway.” Opening the door to teachers to engage in learning “constructively” is of tremendous value.

Opening their eyes to the fact that they have ideas to contribute and that the software and hardware are able to be shaped to meet their needs is hard to put a price on, but also should be a factor in estimating the cost of deployment. While such “awakenings” could take place in a school computer lab, they are much more likely to occur when computation is always at hand, as part of everyday life.

I have largely avoided discussing the saturation vs. shared-use dichotomy. The same software, e.g. Sugar, can be used in both configurations. I echo Dukker in being supportive of whatever means we can deploy to get great software into the hands of children, inexpensively. However, we should not lose sight of the benefits afforded by the form factor and mobility of the netbook computer.

“21st century computer skills” seem to be about the acquisition of some specific knowledge – necessary but not sufficient. Learning is about the acquisition of a new “outlook” – what we are capable of doing with that knowledge .

Yet this seemingly natural order has a whole other life when we look at information and communication technologies in educational systems. Especially when we focus on computers. Educators and technologists promote either full saturation, a 1:1 model where each student has a computer, usually a laptop, or a shared-use model where computers, often desktops, are deployed in school labs.

Between these two models there is the initial debate around the different computing platforms and their different benefits, which highlights a more subtle difference in pedagogy, and through that, the basic foundations of what “school” means to a society.

For July, the Educational Technology Debate we will examine the two models, 1:1 and computer labs, and their respective benefits. Our goal will be to understand which benefits are key, and look for a way in which we can fuse these key benefits into a model that can be deployed in the many educational environments of the developing world.

Our two respected discussants on this topic will be:

• Walter Bender
  Walter Bender currently heads Sugar Labs, focusing on the award-winning Sugar Learning Platform (download it now). Previously he was president for software and content development at One Laptop per Child, and is on leave from MIT, where he was executive director of the MIT Media Laboratory.
• Mark Beckford
  Mark Beckford is currently Vice President of Global Business Development at NComputing, Inc, whose virtualization software and hardware allows multiple users to work off a single computer. Previously, he led diverse global teams at Intel to extend its market leadership and promote growth in new and emerging markets.

Please join us for what we all expect to be a lively and informative conversation – your input can start right now in the comments below, and Walter and Mark will post their opening remarks beginning Monday, July 6.

To bridge this gap, the Education Technology Debate invites thought leaders and opinion makers that directly influence the confluence of technology and education to focus on constructive conversations and open discussions across groups as much as within them.

Become an Educational Technology Debate Discussant

If you’d like to join in building the Educational Technology Debate conversation, please contact us with your ideas, comments, and suggestions on ways to make ETD more informative and engaging.

Previous Educational Technology Debate Discussants

in Assessing ICT4E Evaluations:

Mary Hooker
Mary Hooker is an education specialist with over 30 years experience working in the educational sector in Ireland and Africa. Since 2007 Mary has been working with the Global eSchools and Communities Initiative. Mary is currently engaged in studies for a Doctorate in Education with Queen’s University Belfast, Northern Ireland.

Rob van Son
Rob van Son participated in early Computer Supported Education experiment in the 1980′s, and since worked on everything from small 8088 PCs and the first Mac to modern multi-core file and web servers. Rob is a linguistics expert with a focus on integrating information in spoken communication for Universiteit van Amsterdam. Rob has a PhD in linguistics.

in ICT4E Sustainability:

James BonTempo
James BonTempo is the Learning Technology Advisor for Jhpiego, an international non-profit health organization affiliated with Johns Hopkins University. He is responsible for strategic planning for the integration of Information and Communication Technology (ICT) into pre-service education and in-service training programs. He also leads efforts to design, develop, implement and evaluate ICT initiatives in both arenas.

Atanu Dey
Atanu Dey works as the chief economist at NetCore, a technology firm in Mumbai. His area of interest are the use of technology in education, economic growth of India, and the development of rural populations. He worked in product marketing for several years at Hewlett Packard in California, before receiving his PhD in economics from UC Berkeley. He developed a model called “RISC — Rural Infrastructure & Services Commons” while a Reuters Digital Vision Fellow at Stanford.

in How Can ICT in Education Excite Girls and Boys?:

Alexa Joyce
Alexa Joyce is a specialist in education technology with European Schoolnet. She has consulted for UNESCO Bangkok Asia-Pacific Bureau for Education, UNESCO International Institute of Educational Planning and for the OECD Centre for Educational Research and Innovation. She has a Masters in Biological Sciences from the University of Oxford and an MBA from Solvay Business School, Brussels.

Brooke Partridge
Brook Partridge is CEO and founder of Vital Wave Consulting, which she created to further emerging markets as a new discipline in business management. Previously, she was the Business Director of the Emerging Market Solutions Organization at HP where she lead HP’s first technology solutions for developing economies. She lectured in the Department of Spanish and Portuguese at Stanford University and holds a Master’s of Pacific International Affairs from UC San Diego.

Karen Coppock
Karen Coppock, PhD is Vice President of Vital Wave Consulting with over a decade of experience in strategic business planning for emerging markets. Previously, Dr. Coppock served as the Director of Industry Collaboration for the Reuters Digital Vision Program at Stanford University, and also held positions with Telcordia Technologies, Williams Communications, INTELSAT, Pacific Bell, AT&T and Harvard’s Center for International Development (Information Technology Group), Santa Clara University’s Global Social Benefit Incubator and the US Peace Corps. She received her Doctoral and Master’s degrees in international business from the Fletcher School, Tufts University.

in Creating Electronic Educational Content:

Richard Rowe
Richard Rowe is the Chair and CEO of the Open Learning Exchange, a network of nation-based NGO’s committed to achieving Quality Universal Basic Education by 2015 . Dr. Rowe has served as Director of Test Development and Research for the West African Examinations Council, Associate Dean of Harvard’s Graduate School of Education and a member of the World Economic Forum’s Global Agenda Council on Technology and Education.

Angus Scrimgeour
Angus Scrimgeour is the President of the International Association for Digital Publications, a program to provide university students and academic staff in developing countries with affordable access to e-books, and support for the identification, development, and effective use of open access e-learning resources. Mr. Scrimgeour is also a former Vice President of the World Bank Group a member of the Knowledge and Learning Council.

in Individual and Communal Computer Usage:

Walter Bender
Walter Bender currently heads Sugar Labs, focusing on the award-winning Sugar Learning Platform (download it now). Previously he was president for software and content development at One Laptop per Child, and is on leave from MIT, where he was executive director of the MIT Media Laboratory.

Mark Beckford
Mark Beckford is currently Vice President of Global Business Development at NComputing, Inc, whose virtualization software and hardware allows multiple users to work off a single computer. Previously, he led diverse global teams at Intel to extend its market leadership and promote growth in new and emerging markets.

in Mobile Phones and Computers:

Dr. Robert B. Kozma
Dr. Kozma has directed or co-directed more than 25 projects that have examined the impact of ICT on teaching and learning and developed advanced computer environments for education.

Michael Trucano:
Mike Trucano is the World Bank’s Senior ICT and Education Policy Specialist, providing support to World Bank education projects with ICT-related ‘components’, and is involved in a variety of research activities.

in Educational Vision and ICT in Education:

Tim Kelly
Dr. Tim Kelly is the Lead ICT Policy Specialist at infoDev, at the World Bank in Washington DC, where he has responsibility for access for all and for mainstreaming ICTs for development. He was previously Head of the Strategy and Policy Unit (SPU) at the International Telecommunication Union (ITU).

Wayan Vota
Wayan Vota is a is a technology expert focused on appropriate information and communication technologies (ICT) for rural and underserved areas of the developing world. He is currently the Senior Director of the Inveneo Certified ICT Partner Program, publisher of OLPC News, and hosts the Technology Salon.