The following system-wide approach to deploying ICT is excerpted from Deploying ICTs in Schools: A framework for identifying and assessing technology options, their benefits, feasibility and total cost of ownership by GeSCI
.
A new approach

Once a country – or a school district or even an individual school – has decided to invest in ICTs, it must choose how to go about it. Choosing a technology platform is like making any other major investment, such as buying a home or a car. You decide what you would like, work out what it takes to supply that, and see whether you can afford it. If you can, you go ahead, if not, you adjust your plan.

The framework and corresponding system-wide approach is based on 3 key considerations that arise directly out of some of the major problems facing the deployment of ICTs in schools today:

• Focus on educational objectives ICTs are a tool and not an end in themselves. What tools one chooses to use for any given task depends on the task and anticipated outcomes and not the capabilities of the tool. In the same regard, choosing and deploying ICTs for education must stem from the desired educational objective and outcome.
• Target system-wide approach Purchasing and installing the ICT platform in schools is not the end of the story but rather a part of an integrated (wide) system that requires that a plan be developed in advance, ICTs purchased and installed, training conducted, provisions for user support, technical support and maintenance made and continuous assessment and evaluation conducted to ensure that educational objectives are being met. While GeSCI advocates the use of a system-wide approach designed to extract full impact from deploying ICTs in schools, this document focuses on the benefits, feasibility and costs associated with the deployment of ICT platform. It does explore in some detail and assesses the types and options of education content, initial and ongoing user training and support of teachers, technical support and maintenance. It does not however discuss management, monitoring and support in any detail. The Total Cost of Ownership (TCO) tools however capture all the components of the system-wide approach. It is important to recognize that there are relationships between the various components and the educational objectives as shown in Figure 4. These relationships have an impact on making choices of ICT platforms and are explored in detail in the next few chapters.
• Consider benefits, feasibility and long term costs Benefits and feasibility of both the technology selected and the overall approach to deployment should be considered along with the long term costs of introducing ICTs in schools. It is dangerous to focus on the immediate or initial costs such as those for buying and installing computers in a school lab without considering the long term recurring costs, which are usually much higher than the initial or immediate costs. The feasibility of any given ICT is important to determine whether that particular ICT is applicable in a given context irrespective of the inherent benefits. Feasibility is usually influenced by local conditions. For example, the lack of wired telecommunications infrastructure at a remote village may mean that the only connectivity options are satellite or none at all. Or, cultural considerations such as teachers’ lack of readiness to use technology in the classroom may mean a deployment of technology in teacher offices only.

The Strategy

Drawing from the considerations above, a strategy to select and deploy ICTs in schools has 5 key steps:

1. Define the educational objectives: what are you trying to achieve with the technology?
2. Design suitable “e-school model(s)” that best achieves these objectives: who uses/ will use the ICTs, where do they use it, how many devices are deployed, what basic functionalities should it have and what content and applications accompany the devices? Note that a school may implement more than one e-school model to achieve its objectives.
3. Pick the specific technology platform (what hardware, software, connectivity and services to buy?) along with the necessary content and applications, user training and support and maintenance and technical support that suit this model and the educational objectives to be achieved.
4. Work out how much this technology will cost, not merely to buy in the first place, but throughout the life of the project. In addition to the initial purchase of the equipment and other costs such as telecommunications and modifications to physical school infrastructure, this TCO should include all the accompanying components of the system-wide approach: content and applications, user training and support and maintenance and technical support.
5. Compare this TCO to the budget. If it is within the budget, you can move forward to design a strategy around the chosen technology platform. If it is too expensive, you must go back and review the earlier choices, starting off with your selected technology platform and then the e-school model. Finally, if the cost is still too high, you must go all the way back to your educational objectives, and make compromises until the TCO falls to an acceptable level.

Remember that every stage of this process will be shaped by local conditions and constraints, which could influence or limit the choices at any point of the approach. As you will have undoubtedly have noted, the approach to choosing an e-school technology strategy is a complicated one and does not necessarily have a “right” answer.

In fact, the reader may well have come across numerous other frameworks that strive to achieve the same goal. Therefore the approach described above is not meant to be a definitive one, but merely one that we have found to be useful in guiding our thinking.

We will now consider each of these steps in more detail in the succeeding chapters of Deploying ICTs in Schools. Chapter 2 considers each of the 5 steps in the strategy in more detail. Chapter 3 provides a detailed assessment of the possible e-schools models and technology options and Chapter 4 presents the electronic tools that accompany the framework and discusses how they can be used in the selection of technology options.

Excerpted from Deploying ICTs in Schools: A framework for identifying and assessing technology options, their benefits, feasibility and total cost of ownership by Global e-Schools and Communities Initiative

Don’t miss a moment of the action!

Subscribe now and get the latest articles from Educational Technology Debate sent directly to your inbox.

This is the concluding statements from all six discussants to the question: Are most investments in technology for schools wasted?

Dr. Tim Kelly at Live Debate India

.
Dr. Tim Kelly:

I will invite the speakers to present their closing arguments, beginning with this time the “against the motion” side, to conclude their comments and also to respond to some of the issues that were raised in the discussion. So if you would like to conclude your comments in exactly 3 minutes and we will begin with Wayan Vota. (download the podcast)

.
Wayan Vota:

Again I invite you to vote against the motion and to vote for technology in schools. Yes there are issues, and yes we have bad tests. The tests don’t test us on creative thinking but tests on regurgitation. Yes teacher training and teacher motivation is horribly lacking. I have actually heard some teachers saying:

“You want me to teach computer that is a new subject…pay me more. You want me to learn computers I have to go somewhere to learn this…pay me for that. Or I am just going to stay home and do whatever I really want to do.”

As we all know the buyers here, the government, is often not the users or the beneficiary and depending on your opinion, may or may not even have a clue about the technology much less anything else. But the issues are human ware, they are not software. They are not hardware. The issues are political, “I have something sexy, people vote for me.” The issues are administrative “oh multiple choice tests that is so much easier to grade”.

The issues are also cultural. We are in love with technology, right? No parent here says, “Excuse me, I don’t want my child to grow up to be the next president of Infosys.” Is there any parent here who wants to say, “I don’t want my child to have computer ever?”

Technology again is just tool like any tool and so we should invest in the tool. We should make it better. We should invest in content. Real content that makes children excited to go to school, teachers excited to teach, and parents excited to read and understand and interact with the child about their lesson that day. Let’s change testing so instead of it being multiple choice and about regurgitation it is about thought and about extrapolation and about real learning.

And let’s change the whole way that we think about ICT. Let’s realize that we should not be focusing on ICT, it should be education and ICT is the surrounding area that happens to go with it. When we talk about business we don’t talk about “oh they use a lot of technology to make their product” or “wow I really like this because there is so much technology that went into making this suit” no you say “I look good” and that is what matters to me.

So we should really shift the focus from talking about ICT, and talk about learning, and ICT is merely a tool to achieve that. Again I am calling you to vote against the motion and for ICT because we want invest in the tools that will make our future bright. Thank you.

.
Sam Carlson:

Thank you for your attention and participation in this afternoon excellent debate. I learnt a lot just by listening to each one of your comments and I also learnt from my opponent’s points of view as well. That said I would urge you to vote for the motion and the reason is this:

Once again I say that the question put forward today is whether or not we should invest in education technology in the future and why we should do that. The question is right now is, most investments in education technology wasted?

That is a present tense question. It is not looking into the future. It is looking at what is happening right now, and I would argue that right now most education technology investment is being wasted.

I would say the big difference between the private sector, where we have measurable returns in terms of bottom lines and profits; you see we don’t have the same kind of measurable returns in the same way in the school system. In the school system, the measurable returns are marks, are grades, and are board exam results. They are not problem solving skills, information reasoning skills, and team work skills. All the things that employers value and it will be important for a country’s social and economic growth down the line.

That is not how a school measures its returns now, and unless schools and society changes the way it measures the returns to education in terms of the investments, then education technology investment will continue to be wasted. Because they are generating learning skills which are not measured or valued by the current education system today, and so teachers will not pay attention to anything other than how they are being measured in the system. What gets measured gets done.

If it is not being measured by the board exam, if it is not being captured by your learning score in math or science, it is not going to be valued by the teachers, and it is not going to be valued by the parents who wants to know is my child going to get into that engineering school or not. If they are not, then I don’t want this noise of education technology.

So there is a far larger ecosystem change within the education sector that has to happen for education technology investments to generate the returns that we would like to see, that we see in the labor market and regular work force. So I will end on that point with the final, I guess, hope to the future.

I believe in education technology’s power to transform pedagogy within the education system. But until that transformed pedagogy towards high order thinking skills and information reasoning skills, until that is valued by the system, then the investment will continue to be wasted and I would urge you in that case to vote for the motion. Thank you.

.
Robert Lattimore

Yes I think Sam has some additional good points there. No doubt the risk is trying to strive for the absolute maximum level of excellence. The most optimized level seems so out of reach that you almost don’t know where to start. But it always starts with that single step. That is where you learn, as we have already said a lot, from wasting. But you learn from that and that is where opportunities are.

I have a partner here in my firm who grew up in a farm very near here. This is one of my most relevant close personal stories learning from him. Being educated as he stays outdoors, growing up on a farm, you know sleeping outside, watching the crops, and all those concepts I could barely get my head around. Yet he didn’t get much education and yet he got good education.

He continued to advance and work through higher education and got hired by the firm, continued to work hard, and he has been a good partnership for my firm. Now that is like one of the rag to riches kind of stories but that happened in your city. And you know what, he consulted technology. Now he has gone from barely inaccessible to technology to advising leading companies in India in technology. Now something is working. He didn’t go to private schools. He is not a private school educated guy.

So speaking about waste, well there may be some waste, but something is working.

Yesterday I completed an e-learning course. I like the idea about learning as the way of life. I think that is a good thing that all students should aspire to, so I would like to point out that. So I had access to learning material in United States that my Indian colleagues don’t have access to because I have a US laptop. It just has to be configured with the service learning library which is not available here, but I can assure you it was developed here. That is a fascinating thing.

It was developed here – we didn’t develop in United States. It was developed here and accessed by the US professionals, but not by Indians here, but as a model it is clearly available. I traveled 8000 miles here. India has an opportunity to lead the product.

Continue to stay in course with ICT and investments – you have the opportunity to lead a lot other lesson learners and I think Sam has made a good point about that. You should learn from your mistakes. There has been probably some wastes, but in the long run it has made a lot of progress and I would continue to say vote against this motion because it is worth it. Thank you.

.
Benjamin Vergel De Dios

He who wants to economize will have to organize. Both the parties in this room are in the same boat. We want more ICT investments in education. We want public and private sector to make this a reality in many of our schools.

But do you know what discourages investors to invest in education? How about ICT in education investment? Well, ICT in education projects fail when people don’t see the benefits promised to them.

There is hope and we can do something to improve the current situation. We can raise the awareness of people at all levels, raise awareness of policy makers on how to develop good policies, raise the awareness of education leaders to make better managed schools, raise awareness of principals so that they can better guide their teachers, and raise awareness of the teachers so that they can teach better and utilize technologies provided in schools.

Capacity building is our best investment right now. Secondly, we must improve planning and coordination. We must develop ICT in education master plans to guide governments and implementers. We must clarify the roles of institutions to better improve implementation and coordination and lastly we must continue learning from experience. We learn from our mistakes and hopefully not to repeat the same mistakes over and over again. Thank you.

.
Ashish Garag

I want to respond to Shabnam’s question about lack of research and successful methods of monitoring and evaluation of ICT in education projects. Sam, we have worked with him, and he has developed the outcome mapping – this was in 2005 – and the logical framework analysis and both of these are great tools for mapping ICT in education projects.

The former focusing on assessing the changes that programs bring about in behavior and the actions of stakeholder. So in GeSCI even for our personal work plans we now use the framework and I think it is a great tool to use.

Those are the kind of things and to conclude and to use my 3 minutes even though Dr. Kelly gave the other side the last word, I think we will have the last laugh.

We have seen plenty of libraries in this country and across the world. I haven’t yet met a student who has read up all the books in library so that he could justify optimum, effective, and efficient use of the library. I don’t know why ICTs are being put on the block to kind of justify their presence there and it is just a tool and the child uses it for whatever he likes.

So he uses for gaming if that is what he thinks it is going to help in. He uses it for collaboration. He uses it for communication and we have seen the immense success of the social network and these are all really tools of learning. So I don’t think there is any problem in trying to say whether ICTs are really kind of doing that or not.

For the last two years with no success, I have been talking about two very important things that we should do and I talk completely from an Indian perspective because of the work that we have done with NHRD and that comes to Satish your point, an absolutely fantastic point, about we do not know where we are going. That is the simple truth of the entire thing.

We really need to have an exercise in national vision. We need to develop a national vision as far as education is concerned and we need to do this through a multi-stake holder across the education. We need to get the stakeholders on board. We need to get the teachers from the schools in rural India and we need to get students and parents.

We need to create a national vision for the country as far as education is concerned. ICT will only be one of the components of the various things. This national vision is actually important because it will improve implementation, as I have been hearing in the afternoon that implementation is a big problem. It is will improve implementation because it will set standards for accountability and responsibility – those are very important.

In order to aid the implementation of the national vision, I think first and foremost we need to have standards and benchmarks. This is perhaps the only country where we work without the any benchmarks. We are happy putting computers in schools but we do not have standards and benchmarks for them. That does not take away the work that ICT does and that doesn’t take away from ICT as such, but all it does is create a sense about a lot of investments that is going into ICTs.

So to go back to my first point about if we need to assess what is really required, is to see ICTs as a part of a larger ecosystem and that really is what matters. With that I wouldn’t really urge you because I think your presence here really means that you are voting for us. Thank you very much.

.
Atanu Dey

I think my side has won the moment that proposition was put up on the board for the simple reason that it is not a question whether ICT is useful or can be useful and so on. It is very simple that it has been reached so many times – most of the investment is wasted right now and we have to do something about it.

Mr. Rao there said that wouldn’t somebody think about the children. I think that is the main reason we are saying that it is wasted. We have to start thinking about the children. We have to say that this system is wrong. If the system is broken then there is no point in trying to continue putting ICT on it because ICT is not the problem over here. If you try to fix it by putting ICT in there it is like trying to put lipstick on a pig, pardon that expression.

The system is old because it was invented 300 years ago when the technology and the requirement of the society was different. The old present system is outdated and we have to think about not a system where the teacher is the center of the exercise. Teaching is not that important but the learning is as lifelong learners that we need. We need people who can think. We need people who can ask questions. We need people who know how to answer questions. They have to think very deeply about how to ask questions and the answer is out there you just need to find them.

So from a teacher centric system, we have to move to a learning centric system. We have to learn how to learn because that is what life is all about. We have to learn collaboratively and all that. All this thing about teacher training is I think beside the point. So there is no point in running very hard and in case you are not going the right direction so we have to ask ourselves exactly what do we want to do? What is this education system all about?

It isn’t that the current education system is meeting the needs of the society today. It is not even meeting today’s needs and what we have to do is create an education system which will meet the needs of the people who would be in the workforce 10 to 20 years from now. The world is changing so fast that we cannot ever imagine what it is going to be when these kids grow up.

So we have to have a flexible system and for that we need the help of everything that we can use including information and communication technologies and unless we recognize that there is something wrong going on right now we will be stuck on that problem like wouldn’t somebody think of children and therefore whatever we are spending right now is inadequate. We will just put more money in that.

So I would urge all of you to think deeply about it. I know that you have all finished voting and this doesn’t matter anymore but you are out there doing stuff and intervening in the world out there please be very careful because you still have to think about the children. Thank you.

.
Dr. Tim Kelly

Before I announce the winners of the vote, I would like to thank all of our panelists for being excellent sport and giving very convincing arguments.

So at start we had 35% for the motion, 60% against the motion, and 5% undecided. The new votes are 45% for the motion, 50% against the motion, and 5% still undecided. So congratulations to the winners, those FOR the motion, and commiseration to the losers, those AGAINST the motion.

This we hope will be one of the series of live debate that we will be holding in the next year or so. So there will be plenty of other opportunities to join in. With that, I conclude the debate. Thank you.

.

Don’t miss a moment of the action!

Subscribe now and get the latest articles from Educational Technology Debate sent directly to your inbox.

This is the opening remarks and initial response of Ashish Garg, Asian Regional Coordinator for Global E-Schools and Community Initiative to the question: Are most investments in technology for schools wasted?

.
Ashish Garg: (download the podcast)

Thank you Dr. Kelly and thank you Atanu for trying to make this debate interesting. Even though, I don’t see any reason for us to be here debating about the affrication of using ICTs in schools and education. Nevertheless let me start by quoting not Shakespeare but Ban Ki-moon from recent times.

“Information and communication technology have a central role to play in the quest for development, dignity, and peace. The international consensus on this point is clear. We saw it at the Millennium Summit in 2000 and at the 2005 World Summit and we saw it at the two phases of the World Summit of Information Society.”

Already a substantial number of examples have demonstrated that ICT based systems and servers have the power to improve the quality of life not just for people in the cities but also more importantly for the marginalized and the poor. In the years that have followed, we have seen global spent on ICTs increase consistently the number of internet subscribers have multiplied across the globe depicting the hunger for knowledge, communication, and collaboration. To be debating the efficacies of using ICTs in schools in 2010 in this phase of tremendous progress across the world to me is nothing short of incongruity.

I am afraid I have to fall back on clichéd argument that has now been used a zillion times to support my motion of the day which is investment in ICTs in school is not a waste. I think first and foremost what is required is we need to set expectations right. It is far too easy to take the myopic view of the role and impact of ICTs in the society. ICTs do not exist in isolation and therefore they cannot be measured in isolation to all other elements that impact education. They exist within an educational framework that is part of a larger societal ecosystem.

Jyrki Pulkinnen, CEO of GeSCI, writes:

“I think it is very important to recognize that basically ICT applications are standardized work processes and therefore always social by nature”

and as Shahid Akhtar writes:

“the main challenge across the region is less the matter of access and distribution of technology per say. It is more a matter of creating the enabling environment and capacity building approach.”

I agree that computers may continue to sit in their boxes but the point is that there needs to be a development of an ecosystem and for that it is very important to understand what is the way to assess the investment that is made in ICTs. Wayan ran on this topic on Education Technology Debate, Asessing ICT Evaluations, and there were varied responses to that starting from priorities in developing countries versus investment in ICTs, lack of appropriate tools to measure the impact of ICT versus are ICTs for e-assessments actually effective or not and so on.

To mention Tim Kelly who first started talking about ICTs for E-assessments will help avoid wasteful tragedies and so on. I would really urge you to read that blog for some really insightful articulations on the use of ICTs in schools. As Dr. Kelly even said right the cost of a computer is equivalent to providing a class with a couple of books each but providing the computer is linked to the internet the students and their teachers will then have access to the boundless library of the worldwide web which is constantly updated and which contains a hugely diverse range of views and experiences.

By contrast, the textbook inevitably provides a pre-digested view of the world and one that is out of date the day it is printed. It also brings us to another very important element which is the lack or the presence of political wind and remember long time ago we all lamented the fact that there is no political way to push this wonderful technology across in schools but today we are actually moving from our focus just simply hardware.

Thanks to research and awareness building in not just countries like India but also in Bangladesh and Nepal, countries are more open on spending on teacher training plans, reforming school curriculum, and providing new assessment tool for technology in every class. Clearly I think the risk does not lie in failing to adopt the technology-enabled strategies which are inevitable. You could see the lucrative use of mobile even though I do not really stand for the use of mobile for basic literacy and things like that but these are inevitable and they are lucrative as you can see them across.

The risk is rather tonight doing a poor job of adopting these strategies and then the final point that I want to make is to Atanu’s point where he says that the number schools and the number of people that we have to educate in India and how do we provide technology for all of them? I think brick and mortar possibilities are really going to be difficult.

The recent RTE now says 290 million more students will start attending schools. I think if we are planning on putting up more schools then we might have to stop losing roles because probably we have to put schools everywhere but contrast that with the statistics or the data that IGNOU printed out sometime back, 24% of India’s school going population, higher education population, uses ODL, open distance learning methods, for education. So therefore I think, and Dr. Kelly is showing me the time already, there is really a lot of scopes for ICTs provided we understand a few basic mechanisms of how to make this work.

The last point that I want to make is there really is a phased evolution process on how we use the technology in our countries. Jyrki Pulkinnen talks about the society where the society actually stops the question of relevance of ICT in education. Thank you.

.
Dr Kelly: How do you think we should be doing evaluation of ICT in education, to make the investment worthwhile?

Actually, the government is going to be beaten any which ways. If they did not spend the money on the ICT then there would be a brigade that would rise up saying the government is not doing anything to put technology in the schools and today the government is doing then the response is that because the government is doing it there is no responsibility and there is no accountability.

I agree that yes ICTs help a lot in various things and there is no debate on that and Sam brought forward a very important point about a tail that wags the dog and yes definitely it is the ecosystem but my point is that for 62 years of India’s independence nobody really decided to question the response of the ecosystem, the development of the educational ecosystem, or the readiness of the educational ecosystem. So what has lead especially in India and this part of the world? What has lead to this question?

It is the coming of the ICT that brings up these questions that what is it about that needs to be done? The reformative or the transformative reforms that need to come in so that new technologies can be adopted. The last point that I want to make is that it is not about just raising a point about what has been the worth of that particular investment. It is not like your log book which says credit and debit and in the end of the day both the sides have to be equalized. It is education.

It is a social change. It is social reform so there is a gestation period. So sufficient gestation period has to be given in order for ICT to prove their point but beyond that let me just talk about the PISA results, the program for international student assessment. Likewise there are several such impact studies monitoring and evaluation studies which have particularly shown how ICTs have helped move scores forward and in one of the blogs Dr. Kelly writes that available evidence of benefits of ICTs in schools is sometimes mixed and hard to interpret.

In the same way we say in the latest survey of 2006 shows the fastest gain in reading standards in any country observant with in the Republic of Korea where students have increased their reading standards by 31 points and not coincidentally he continues to write Korea also scored top in the ITUs digital opportunity index, DOI index, in 2006 which is the most respected measure of an economy’s ICT performance.

So if you kind of correlate the two I think there is a lot to be drawn from there and I think finally that it is these evaluations that are necessary to demonstrate to the local officers and to the national policy makers that ICTs are worth the investment. They need to know what local problems ICT can address or opportunities that are possible.

.
Dr. Kelly: Do you want to challenge the other side or shake the arguments?

I would really like to go back to what I put forward some time back why we are asking ourselves these questions today? How come we are questioning the ecosystem and its ability to deliver which we didn’t do? I actually want to go back to the time when I was probably in class 10 and then multiple choice questions were starting to get introduced and before that everybody thought it was great to write 5 answers of 20 marks each for a paper of 100 marks and you had to memorize as much as you could.

The reason why these multiple choice tests started coming up was because first computers had these multiple choice standardized tests fed into them and then the government thought that it would be great. So I am just trying to bring the fact forward that technology has been influencing change. Technology has been influencing innovation and it is just not possible for a country like India to wait until the mindset of the government has been changed and we are in a state of readiness to accept not just the hardware but be transformed ecosystem processes or the human ware.

So I don’t think that is going to work and it is a process of evolution but we see a whole lot of new indigenous dynamism that is coming up and it is only a matter of recording them and I am sure a lot of organizations that work at the grass roots level already just-so stories but they are there and you cannot take it away from the process of evolution.

We recognize the inadequacies of evaluating the use of a tool and its potential for transformational innovation in education systems that are intent on simply harnessing it for maintaining the status quo. As Rob (on the other side) observed ‘any real assessment of educational reform requires a new reflection on what skills and knowledge the children are supposed to acquire at school’.

And so in my response I would like to revisit the question presented by Wayan and reflect a little more on its parameters. I would also like to draw on commentary from both sides of the discussion (quite a lot of comments on your side Rob) to tease out some of the issues.

Do we need to assess ICT4E initiatives?

Rob notes that most reforms have historically been imposed without scientific support but rather on political prejudices. However the sense of fatigue with the failure of education reform syndrome is perhaps changing as we migrate into a 21st century information age. And as we do so, we are witnessing a growing discrepancy between school and the ‘outside world’ – where information, knowledge, innovation and creativity are replacing the traditional sectors of commerce and industry – and where new technologies are changing the way we interact, communicate, socialize and network.

It is a world where mobile connectivity is becoming commonplace and where digital literacy is a critical tool for social interaction, knowledge exchange and construction. If schooling fails to transform itself, it may be transformed albeit haphazardly by the technological transformation outside its gate – and perhaps in a way that may be detrimental to learning.

There is also the challenge of digital divides, both between societies and within societies – with access denied to the poorest and most marginalized. ICT is seen as bridging such major divides. There is thus a renewed sense of urgency, despite the fatigue, for systemic ICT investment and reform to provide all learners with skills they will need for meaningful participation in the economic, social and culture life of new knowledge-based economies and societies.

In such scenarios of massive large-scale investment and reform, assessments are needed to hold systems accountable. Assessments can also provide policymakers with the gateway they need to direct systemic change. As Clayton observes in his comments ‘evaluations are necessary to demonstrate to the local officials and national policy makers that ICTs are worth the investment’. They can help them to identify factors to best influence ICT impact (changes in curriculum, pedagogy, assessment, teacher training) and well as the barriers to ICT use, such as lack of skilled support and adequate infrastructure.

If we assess, how do we do it?

Juan in his commentary describes his skepticism as to the relevance of some of the ICT evaluations he has come across over the years – in particular studies on proprietary software where the emphasis is more on evaluating technology than learning. He notes the lack of comparison with alternative activities and for cost effectiveness. John also comments that effective assessments have not been designed. On cost effectiveness John finds shocking a US study illustrating a lack of empirical research in an area where billions of dollars have been invested. What is particularly ‘unsettling’ is the notion that politicians don’t seem to care.

Yet I wonder John if it is a question that politicians don’t care or that there is a sense of exasperation with the lack of defined mechanisms for informing decision making for such a massive scale of investment and change? Scheuermann, Kikis and Villalba (2009) discuss the lack of clear information in most studies about the multifaced effects and impact of ICT on the learner and learning. It is a situation that is ‘especially unsatisfying for policy-making stakeholders that aim at defining evidence-based strategies and regulatory measures for effective ICT implementation and efficient use of resources’ (ibid. p. 1).

There are calls for more widely accepted indicators and methodological approaches to assess inputs, utilization and outcome/ impact of ICT integration initiatives in order to address this gap (Trucano, 2003; Blanskat et al. 2006; cited in ibid.). Yet there still remain limitations in these approaches on measuring the impact of ICT use – as they often represent a snapshot – a one time, one level approach. Ian comments on the ‘imperfection’ of the data collection in such evaluations more often conducted to appease funder insistence for seeing ‘educational’ results. He also draws attention to the difficulty in attributing the said results to the ICT intervention.

A more powerful approach is the use of indicators within development models of ICT integration in education – to study the progressive phases through which teachers and students adopt and use ICT. Morel’s Matrix is an instrument that can be used for evaluating the degree to which ICTs have been integrated in an educational system through four distinct successive phases: a) emerging, b) applying, c) integrating, and d) transforming.

In GeSCI we have developed an ICT-Education matrix to assist our partners in focusing on what teachers and learners actually do when they use ICTs in schools and institutions through each of the four successive phases (Figure 1). Such models when used to guide Planning, Monitoring & Evaluation (PME) in combination with the indicators approach can offer clearer outcomes on what the integration of ICTs in education should look like at each development stage.

Figure 1: GeSCI ICT- Education Matrix

I like Mark Beckford’s observation in that the key to assessment is to keep it ‘simple but useful’. We hope in developing PME tools as the ICT-Education matrix for our partners that we can do just that.

Related Link

Scheuermann, F., Kikis, K. & Villala, E. 2009. A framework for Understanding and Evaluating the Impact of Information and Communication Technology in Education Available online & accessed 23 November 2009

ICTs can be powerful, essential tools for learning: understanding, interpreting and communicating about the real world OR they can be black holes into which we pour our money, intelligence and time, getting very little in return.
Dr. Patti Swarts, Education Specialist (GeSCI)

1. Do we really need to assess ICT4E initiatives?

In a word – yes. Yes we do need to assess ICT4E initiatives more particularly when we are working in environments with scarce resources as in the developing world where investment in ICT can constitute what Unwin (2004) describes as a ‘wasteful tragedy’ if it is not managed and utilized properly.

There is no doubt that ICT in education presents countries with great opportunities. I suppose this would support the perceived notion presented in this forum discussion that the benefits of technology are obvious – on a par with the obvious benefits of electricity. However the use of any technology whether ICT or otherwise also bring challenges – which need to be addressed if the benefits are to be realized.

Some of the ICT challenges in Education relate to, cost, sustainability, optimising usage and making teaching and learning meaningful for students, and relevant for the development of the country. Many policy makers, planners, managers, and practitioners still lack experience, knowledge, and judgment capabilities in the systems, methods, and media involved in what is still an emerging field of ICT in Education.

This lack of experience is particularly evident when dealing with the integration of the newer ICT technologies such as 1:1 saturation models. In my organization, the Global e-Schools and Community Initiative (GeSCI), we believe that assessment is essential for addressing such challenges.

2. Are ICT4E assessments effective in measuring outcomes?

I find that the belief presented in this discussion that ICT evaluations are necessarily flawed – because we do not have the correct tools to assess ICT impact in conventional systems – has some resonance in the literature. In a GeSCI commissioned meta-review research on ICT in Education, Le Baron and Mc Donough (2009) discuss the “grammar of schooling” (Arbelaiz & Gorospe 2009 cited in ibid.) – as in the entrenched practices of conventional schooling where ‘rules constrain transformational curricular innovation especially ICT integration’.

In this context Angeli and Valanides (2009 cited in ibid.) support a view that the transformational impact of ICT would require ‘a clear, commonly understood epistemological framework’ in order for ‘teachers to understand ICT’s transformational potential or for educational decision makers to assess whether or not high standards are being met’.

Despite the lack of such a common framework for understanding and assessment, I would agree with Clayton’s comment in this forum that even if we don’t have a perfect tool to evaluate ICT impact as yet, we should still employ what we have and in the process refine our tools.

3. Do we even have the tools to tell if they are effective? What tools are those?

There are several methods for monitoring and evaluating ICT in Education projects. Two of the most common approaches cited by Wagner et al. (2005) are Outcome Mapping (OM) and Logical Framework Analysis (LFA) – the former focusing on assessing the changes that programmes bring about in the behaviour and actions of stakeholders (e.g. officials, teachers, parents) – the latter focusing on assessing results in terms of products and services (e.g. curricula, teacher training, educational software) of a programme.

4. Are we really using these assessment tools correctly?

There is an emerging view that M&E traditional frameworks are not enough. There is a need to include components of applied research for ‘proof of concept’ – to provide more rigorous, field-tested knowledge about a) what works and why, b) how the initiative could contribute to educational development priorities (access, quality, capacity and relevance), c) the enabling conditions and barriers for taking the initiative to scale, and d) what might be the result of large scale application ( Batchelor & Norrish n.d.).

In GeSCI we take the process further – we believe that for M&E/research on ICT in Education to have any impact, the results must inform and shape policies and programmes and be adopted in practice. It is the interaction between these three dimensions that we believe is necessary to maximize the potential of ICT use for transformational impact in education systems.

Figure 1. Stakeholder facilitation in GeSCI research framework

5. And regardless of the outcomes, should we really wait for long-term results, or should we implement ICT4E deployments now, as the case is compelling enough already?

In a word – maybe. The case is not compelling. However governments made nervous by the digital / knowledge divide between developed and developing worlds are implementing ICT4E deployments anyway. What we should do is focus our efforts to pursue a deeper and broader research and evaluation agenda.

In my view such an agenda should shift from the current focus of much contemporary ICT –Education evaluation and research on quantitiatively measured learning outcomes (long term results and potentially problematic in terms of cause/effect attribution) – towards a focus on examining the qualitative potentially ‘distruptive’ force of technology such as saturation models to shake the “grammar of schooling” in transofmrative ways.

What do others think? Is the case compelling? Should we assess? What should be the focus of our assessment?

References

• Batchelor, S. and Norrish. P. n.d. Framework for the Assessment of ICT Pilot Projects . Available from infoDev & accessed 10 November 2009
• Le Baron, J. and Mc Donough, E. 2009. Research Report for GeSCI Meta-Review of ICT in Education – Phases One and Two. Available from GeSCI & acccessed 10 November 2009
• Unwin, T. 2004. Towards a framework for the use of ICT in Teacher Training in Africa. Available from infoDev & accessed 11 November 2009
• Wagner, D.A., Day, B., James, T., Kozma, R.B., Miller, J., and Unwin, T. 2005. Monitoring and Evaluation of ICT in Education Projects: A Handbook for Developing Countries. Available from infoDev & accessed 10 November 2009

.

I’ve heard this refrain repeated often since then, and not just by those promoting technology in schools. Its a equal thought from those that feel geek lust is clouding our judgment and we should focus on teachers, not technology. Its also promoted by those that point out changes to educational methodologies have often happened by force of will, not empirical results.

Now, Nicholas Negroponte is putting forth the idea that one computer per child is like electricity – such an accepted benefit for society that we’ve moved on from discussing its impact to just looking for the right models to fund it.

While we may have differencing opinions on OLPC or its benefits, the basic questioning of ICT4E evaluations is compelling. Starting with the simple question of “Do we need assessments?” we can branch into related questions that examine the basic assumptions we hold dear, like:

1. Are ICT4E assessments effective in measuring outcomes?
2. Do we even have the tools to tell if they are effective?
3. What tools are those?
4. Are we really using these assessment tools correctly?
5. And regardless of the outcomes, should we really wait for long-term results, or should we implement ICT4E deployments now, as the case is compelling enough already?

For November, the Educational Technology Debate will focus on assessments of ICT initiatives in education – how we can both validate them and use them correctly to improve ICT4E overall. For discussants we’ll be joined by the following experts:

• Mary Hooker
  Mary Hooker is an education specialist with over 30 years experience working in the educational sector in Ireland and Africa. Since 2007 Mary has been working with the Global eSchools and Communities Initiative. Mary is currently engaged in studies for a Doctorate in Education with Queen’s University Belfast, Northern Ireland.
• Rob van Son
  Rob van Son was a subject in early Computer Supported Education experiment in the 1980′s, and since worked on everything from small 8088 PCs and the first Mac to modern multi-core file and web servers. Rob is a linguistics expert with a focus on integrating information in spoken communication for Universiteit van Amsterdam. Rob has a PhD in linguistics.

Please join us for what we all expect to be a lively and informative conversation exploring assessment validity and tools for ICT4E. Your input can start right now in the comments below, and Mary and Rob will post their opening remarks beginning Monday, November 9.

Don’t miss a moment of the action!

Subscribe now and get the latest articles from Educational Technology Debate sent directly to your inbox.

.

To bridge this gap, the Education Technology Debate invites thought leaders and opinion makers that directly influence the confluence of technology and education to focus on constructive conversations and open discussions across groups as much as within them.

Become an Educational Technology Debate Discussant

If you’d like to join in building the Educational Technology Debate conversation, please contact us with your ideas, comments, and suggestions on ways to make ETD more informative and engaging.

Previous Educational Technology Debate Discussants

in Assessing ICT4E Evaluations:

Mary Hooker
Mary Hooker is an education specialist with over 30 years experience working in the educational sector in Ireland and Africa. Since 2007 Mary has been working with the Global eSchools and Communities Initiative. Mary is currently engaged in studies for a Doctorate in Education with Queen’s University Belfast, Northern Ireland.

Rob van Son
Rob van Son participated in early Computer Supported Education experiment in the 1980′s, and since worked on everything from small 8088 PCs and the first Mac to modern multi-core file and web servers. Rob is a linguistics expert with a focus on integrating information in spoken communication for Universiteit van Amsterdam. Rob has a PhD in linguistics.

in ICT4E Sustainability:

James BonTempo
James BonTempo is the Learning Technology Advisor for Jhpiego, an international non-profit health organization affiliated with Johns Hopkins University. He is responsible for strategic planning for the integration of Information and Communication Technology (ICT) into pre-service education and in-service training programs. He also leads efforts to design, develop, implement and evaluate ICT initiatives in both arenas.

Atanu Dey
Atanu Dey works as the chief economist at NetCore, a technology firm in Mumbai. His area of interest are the use of technology in education, economic growth of India, and the development of rural populations. He worked in product marketing for several years at Hewlett Packard in California, before receiving his PhD in economics from UC Berkeley. He developed a model called “RISC — Rural Infrastructure & Services Commons” while a Reuters Digital Vision Fellow at Stanford.

in How Can ICT in Education Excite Girls and Boys?:

Alexa Joyce
Alexa Joyce is a specialist in education technology with European Schoolnet. She has consulted for UNESCO Bangkok Asia-Pacific Bureau for Education, UNESCO International Institute of Educational Planning and for the OECD Centre for Educational Research and Innovation. She has a Masters in Biological Sciences from the University of Oxford and an MBA from Solvay Business School, Brussels.

Brooke Partridge
Brook Partridge is CEO and founder of Vital Wave Consulting, which she created to further emerging markets as a new discipline in business management. Previously, she was the Business Director of the Emerging Market Solutions Organization at HP where she lead HP’s first technology solutions for developing economies. She lectured in the Department of Spanish and Portuguese at Stanford University and holds a Master’s of Pacific International Affairs from UC San Diego.

Karen Coppock
Karen Coppock, PhD is Vice President of Vital Wave Consulting with over a decade of experience in strategic business planning for emerging markets. Previously, Dr. Coppock served as the Director of Industry Collaboration for the Reuters Digital Vision Program at Stanford University, and also held positions with Telcordia Technologies, Williams Communications, INTELSAT, Pacific Bell, AT&T and Harvard’s Center for International Development (Information Technology Group), Santa Clara University’s Global Social Benefit Incubator and the US Peace Corps. She received her Doctoral and Master’s degrees in international business from the Fletcher School, Tufts University.

in Creating Electronic Educational Content:

Richard Rowe
Richard Rowe is the Chair and CEO of the Open Learning Exchange, a network of nation-based NGO’s committed to achieving Quality Universal Basic Education by 2015 . Dr. Rowe has served as Director of Test Development and Research for the West African Examinations Council, Associate Dean of Harvard’s Graduate School of Education and a member of the World Economic Forum’s Global Agenda Council on Technology and Education.

Angus Scrimgeour
Angus Scrimgeour is the President of the International Association for Digital Publications, a program to provide university students and academic staff in developing countries with affordable access to e-books, and support for the identification, development, and effective use of open access e-learning resources. Mr. Scrimgeour is also a former Vice President of the World Bank Group a member of the Knowledge and Learning Council.

in Individual and Communal Computer Usage:

Walter Bender
Walter Bender currently heads Sugar Labs, focusing on the award-winning Sugar Learning Platform (download it now). Previously he was president for software and content development at One Laptop per Child, and is on leave from MIT, where he was executive director of the MIT Media Laboratory.

Mark Beckford
Mark Beckford is currently Vice President of Global Business Development at NComputing, Inc, whose virtualization software and hardware allows multiple users to work off a single computer. Previously, he led diverse global teams at Intel to extend its market leadership and promote growth in new and emerging markets.

in Mobile Phones and Computers:

Dr. Robert B. Kozma
Dr. Kozma has directed or co-directed more than 25 projects that have examined the impact of ICT on teaching and learning and developed advanced computer environments for education.

Michael Trucano:
Mike Trucano is the World Bank’s Senior ICT and Education Policy Specialist, providing support to World Bank education projects with ICT-related ‘components’, and is involved in a variety of research activities.

in Educational Vision and ICT in Education:

Tim Kelly
Dr. Tim Kelly is the Lead ICT Policy Specialist at infoDev, at the World Bank in Washington DC, where he has responsibility for access for all and for mainstreaming ICTs for development. He was previously Head of the Strategy and Policy Unit (SPU) at the International Telecommunication Union (ITU).

Wayan Vota
Wayan Vota is a is a technology expert focused on appropriate information and communication technologies (ICT) for rural and underserved areas of the developing world. He is currently the Senior Director of the Inveneo Certified ICT Partner Program, publisher of OLPC News, and hosts the Technology Salon.