EGRA testing in Ethiopia

Experts agree that measuring reading progress early offers the benefits of informing remediation, taking a snapshot in time or showing progress over time of children’s reading abilities and informing stakeholders and policy makers about what programs or methods work.

Frequent diagnostic testing at national or classroom levels can serve to establish benchmarks; and monitoring progress against these benchmarks can be a key factor in motivating schools, teachers, students, and families (Davidson, Korda, & Collins, 2011).

The Education for All Fast Track Initiative recently set two indicators related to reading skills:

1. Proportion of students who, after two years of schooling, demonstrate sufficient reading fluency and comprehension to “read to learn”
2. Proportion of students who are able to read with comprehension, according to their countries’ curricular goals, by the end of primary school

These indicators are considered an effective measure of a school system’s overall health as well as a specific diagnosis of reading performance that can inform policy and implementation of curriculum and teacher training, among other things. According to Gove and Wetterberg (2011),

“The Early Grade Reading Assessment (EGRA) is one tool used to measure students’ progress toward learning to read. It is a test that is administered orally, one student at a time. In about 15 minutes, it examines a student’s ability to perform fundamental prereading and reading skills” (p. 2).

Over the past five years, we at RTI International, various donors, and experts in the field of early reading have worked to “develop, pilot, and implement EGRA in more than 50 countries and 70 languages” (p. 2). Assessments like EGRA help teachers focus on results, by describing what children know or do not know, and where instruction must focus in order to change that. For example, in Egypt, the first Arabic EGRA survey showed very clearly that children who knew letter sounds performed better on reading a short passage than children who only knew letter names; yet 50% of children tested could not identify a single letter sound. These findings signaled that a fundamental shift in instructional methods was required, and after schools adopted a phonics-based approach using letter sounds, performance increased nearly 200% over baseline one year later (Cvelich, 2011).

That said, to measure for results, teachers and their supervisors must find the tools accessible and easy to use to inform their own instruction. It also helps if the results underpin communication with parents and communities, as well as national politicians. (Crouch, 2011). Too often, results from national standardized tests remain at the national level, with teachers rarely getting feedback on performance, much less feedback that is more specific than classroom averages. Furthermore, it can sometimes be months, if not years, before the results of large national assessments are made available, at which time it is too late to change instructional practices – at least for that set of children.

How can ICT play a role?

Systematic use of mobile devices to assess early literacy and numeracy, especially in developing countries, remains limited to date. Reasons include:

• Initial procurement cost of the devices and the necessity for specific training in their use;
• Lack of robust cost-benefit analyses to inform sustainability of this type of approach; and
• Limitations in local capacity to develop or manipulate the necessary data collection software.

As we state elsewhere (Pouezevara & Strigel, 2011), there are several ways in which information and communication technologies (ICT) may be applied to the assessment process to make implementation and use of the results more accessible:

• Creating or tailoring tests
• Training data collectors
• Collecting actual field data
• Manipulating and managing the data to extract and present the most significant findings.

Among these, the greatest added value is in using electronic devices for data collection and rapid analysis in place of paper-based assessments.

• Electronic devices can reduce the amount of paper needed, as well as the associated costs. Expenses dispensed with include the actual purchase of paper, clipboards, pencils, timers and so on, as well as the labor involved in the lengthy processes of checking student sheets for copy quality, stapling individual packets, counting instruments out by team and school in advance of data collection in the field, and distributing the packets. Paper-related costs such as printing, supplies, data entry, and data cleaning can make up 5%–15% of the entire budget of an EGRA implementation, according to an RTI internal review.
• Collecting data digitally means that it can move directly from a device into a database for analysis. This has several benefits in terms of efficiency: less time for data entry, lower data-entry costs, and less time to report out results. Quicker access can encourage stakeholders to do such assessments even when they need data rapidly to make important decisions based on results.
• Electronic means have the potential to reduce the number of points for human error in moving from paper to database to analysis software. As with most sophisticated survey software, programmers can build in checks or stops to help assessors recognize data-entry errors immediately, at the time of administration.
• Electronic media can be less physically challenging than dealing with paper-related administration: “An electronic solution may also reduce measurement errors arising from problems in handling the timers and other testing materials. Difficulties include forgetting to start the timer, setting the wrong amount of time on the timer, or leaving student prompt sheets with the student when they should have been taken away” (Pouezevara & Strigel, 2011, p. 188).

What solutions are available?

In theory, there are many potential ways to transform paper assessments into an electronic equivalent, but a custom solution is required because of differences between oral reading assessments like EGRA and other standard surveys. For example, data have to be entered at the child’s pace on the subtasks, not that of the assessor. Therefore, survey data collection applications on the market for phones, PDAs, or portable computers typically are not appropriate.

After investigating a wide range of potential hardware and software platforms, we developed Tangerine™, a digital assessment interface for touch-screen tablet computers running the Android operating system (see photographs). It can be used for the standard EGRA approach, or customized for other types of surveys such as early math diagnostics or school information surveys.

Other organizations are also exploring a variety of solutions. Prodigy Systems, an organization that has partnered with RTI in Yemen, successfully developed iProSurveyor for use with Arabic assessments on the iPad. Its first large-scale implementation in Yemen in early 2011 confirmed many of the benefits of the digital approach.

• The database output was easily readable by any data analysis program, avoiding time-consuming manual data transcription and recoding before statistical analysis.
• Administration errors, such as forgetting to start the timer or enter a response, were minimized through built-in error control.
• Significantly fewer materials had to be transported in challenging terrain and an environment unfavorable to printed materials.
• No issues arose linked to poor printing quality or stapling.
• Total administration time was quicker relative to paper assessment (comparison conducted over one assessment administrator).

Cost-Benefit Analysis

At RTI we recently conducted a preliminary cost-benefit analysis using approximate costs from recent EGRA implementations in four different African countries. The analysis aimed to identify the point of cost recovery at which the digital approach would actually yield cost savings. We modeled not one, but three data collection rounds for each country, because it is common to repeat assessments  - e.g., for program baseline, midterm, and post-intervention evaluation, or annual monitoring of student outcomes.

In our cost calculation for the digital approach, we assumed hardware costs of USD300/enumerator plus a 10% contingency for spares and accessories, such as a wireless access point for field-based data back-up for the first data collection (e.g., baseline). For the cost of a second digital data collection, we assumed re-use of the tablets from the first data collection, but factored in a 15% contingency just in case replacements are needed.

To calculate the cost of a second paper-based data collection we multiplied the paper-related costs by two, as the same costs for printing, data entry, and data cleaning would incur again. We followed the same process for adding a third data collection to the calculation (assuming baseline, mid-term, and post-intervention assessments).

As shown in Exhibit 1, for most small-sample data collections or one-time assessments, the cost of the hardware may not be offset by the eliminated paper-related costs. The return on investment in repeated implementations, however, is clear in terms of cumulative costs.

Exhibit 1: Cost of EGRA implementation, paper vs. electronic, for three administrations

In addition to making large national assessments more efficient, the same devices can be adapted for use as classroom-based continuous assessment tools, or as data entry interfaces for situations that still require paper-based tests. With such devices in their hands, teachers or school supervisors can do regular mastery checks more frequently, and capture the results at student and classroom levels.

The resulting data set is a rich one, and if it is supported by built-in computer-based analytics, it can be analyzed in multiple ways to indicate not only whether the methods in place are improving reading ability, but also what areas of the curriculum need more attention, and which children or groups of children are falling behind. For example, detailed item analysis at the classroom or individual level might show a recurring problem with vowel sounds, or decoding. This subsequently provides clear instructional recommendations to focus on.

Limitations and pitfalls

However, electronic administration is not necessarily a cure-all:

Obviously, using electronic data collection at either national or classroom levels does not solve all the limitations of print-based testing; indeed, doing so might introduce new challenges. For example, although a digital solution would eliminate the risk of environmental damage to paper forms during difficult transport situations, it might pose a great risk that all assessment data could be lost at once through loss, damage, or theft of a single device, if proper backup procedures were not in place. Likewise, handling of the new device might prove to be more challenging than handling the timer and all associated materials. […] Thus, strong electronic quality control and supportive supervision during data collection would be crucial. (Pouezevara & Strigel, 2011, p. 188)

Furthermore, the EGRA approach is intended to be a simple solution that can be adopted by countries with minimum technical assistance. An electronic solution should be flexible enough that it does not create dependency of users on software programmers or hardware technicians to change test items and configuration as needed.

In terms of costs, clearly, initial investment costs for specialized hardware may be prohibitive in some situations, but our preliminary cost-benefit analysis indicated that over time the investment will pay off if used for multiple large-scale implementations. Additionally, implementers can leverage the initial investment by choosing tools that can be used for other purposes when not in use for assessment—for example, by loading tablet computers with other instructional materials, training resources, or literacy materials.

We can also foresee assessment software being linked not only to automatically generated analysis of results, but also to suggested instructional resources tailored to those results and a record of day-to-day time on task. It is also possible, using the same technologies that power Tangerine™, to adapt the assessment methodology to more common and less expensive handheld devices, such as mobile phones. These smaller devices might be particularly useful for the most rapid types of literacy assessments, such as Pratham’s yearly literacy and numeracy surveys, which involve fewer subtasks than EGRA and fewer items per test.

Another potential pitfall related to making national or continuous assessments more readily accessible is that they could be used for excessive assessment, and focus on “teaching to the test” at the expense of other higher order or student-centered activities. Too much focus on averages or aggregated results can draw attention away from the achievement of specific subgroups. Additionally, care must be taken that classroom-level results are not misused by aggregating small samples and reporting them up to the national level or attempting to generalize them.

This is a rapidly evolving field, with new technologies arriving on the market almost daily, and prices falling significantly, so it is expected that it will become increasingly feasible to implement electronic methods for literacy assessments in developing countries. Meanwhile, we are piloting various solutions and collaborating with other institutions that have similar goals. Further interest and ideas from the international development community are welcome.

References

Crouch, L. (2011). Motivating early grade instruction and learning: Institutional issues. Ch. 7 in A. Gove & A. Wetterberg, The Early Grade Reading Assessment: Applications and interventions to improve basic literacy (pp. 227–250). Research Triangle Park, NC: RTI Press. Available from http://www.rti.org/pubs/bk-0007-1109-wetterberg.pdf

Cvelich, P. (2011, September/October). Egypt shakes up the classroom. Frontlines. Washington, DC: United States Agency for International Development (USAID). Available from http://www.usaid.gov/press/frontlines/fl_sep11/FL_sep11_EDU_EGYPT.html

Davidson, M., Korda, M., & White Collins, O. (2011). Teachers’ use of EGRA for continuous assessment: The case of EGRA Plus: Liberia. Ch. 4 in A. Gove & A. Wetterberg, The Early Grade Reading Assessment: Applications and interventions to improve basic literacy (pp. 113–138). Research Triangle Park, NC: RTI Press. Available from http://www.rti.org/pubs/bk-0007-1109-wetterberg.pdf

Gove, A., & Wetterberg, A. (2011). The Early Grade Reading Assessment: An introduction. Ch. 1 in A. Gove & A. Wetterberg, The Early Grade Reading Assessment: Applications and interventions to improve basic literacy (pp. 1–38). Research Triangle Park, NC: RTI Press. Available from http://www.rti.org/pubs/bk-0007-1109-wetterberg.pdf

Pouezevara, S., & Strigel, C. (2011). Using information and communication technologies to support EGRA. Ch. 6 in A. Gove & A. Wetterberg, The Early Grade Reading Assessment: Applications and interventions to improve basic literacy (pp. 183–226). Research Triangle Park, NC: RTI Press. Available from http://www.rti.org/pubs/bk-0007-1109-wetterberg.pdf

Reading the resulting commentary, I’d like to declare success. I feel we have found a new model, that is an child of these two parents, mixing genes of both to create a new, better ICT4E model where multiple platforms plus a single learning environment equals more educational beneficiaries.

Multiple Platforms

From the beginning, this discussion recognized that different communities allocate their limited resources differently. Some will have the resources for high saturation of computing tools, while others will not. In fact a single community may have multiple computing models within its own educational system, based on age, maturity, and progress of its students. Mark Beckford gave us a great example:

In Macedonia, NComputing deployed over 100,000 virtual desktops which made Macedonia the country with the greatest density of computers to students. But Macedonia also issued a tender to deploy a smaller quantity of netbooks. They cannot afford mobility for all students, and yet even at 1:1 desktop computing they see the advantages of mobility.

So educators need not feel that its a either-or decision. Communities can have both personal and shared computing environments in the same school. And as Alex Van de Sande points out, its not the technology that matters, but the way educators use it:

The most important is that in either case, the experience must be saturated, shared and free. The shared PC lab experience, where there are many peers around you who can quickly teach you is invaluable. But all that is nullified by models with restrict hours and usage rules. The 1:1 laptops are great on the fact that the freedom from “this is how you are supposed to use this” rules make you experiment more. But doing it alone may lead to the laptops being used for more private entertainment – like gaming.

In that context, a mixed environment may be the best choice. One where students use computer labs in the school setting, where usage can be monitored and directed, and on a more personal basis when outside the school.

Single Learning Environment

With all these platforms, there quickly becomes the need to maintain a homogeneous learning environment. One familiar look and feel that follows the child as they access different platforms during the day and their education. Walter Bender is working on such an environment with Sugar on a Stick.

This USB memory stick-based educational software platform is based on the principles of cognitive and social constructivism, and contains its own operating system (Fedora 11) so it can be run from just the memory device itself – no hard drive or specific operating system needed.

Caroline gives us her thoughts on the advantages of such an approach:

Sugar on a Stick should make mobility cheaper. If kids take their sticks with them they can use them on clusters of computers in day care centers, community centers and at home if the parent has a computer. Thus by using computers in different places in their environment they can get quite a bit more hours of computing time per week and their desktop and all their work is mobile. I wonder if we can run numbers on that type of solution, and maybe instead of running them per machine, run the numbers to compare $ per hour the child uses a computer.

And Walter Bender confirms that the Sugar on a Stick approach can be complimentary to current and new platform investments:

It is great that there are many different such platforms being developed: a diversity of hardware configurations is necessary to meet the demands of schools, budgets, and cultures. But one can remain agnostic about hardware platforms and configurations, while providing a great learning experience, better utilizing the installed base of computers while tapping the potential to engage every child in critical thinking, arming them with the complementary tools of science and the arts.

More Beneficiaries

So with a single learning environment on multiple platforms, let’s start talking about the real numbers of beneficiaries. Either in school or at home, let’s move away from the assumption that only the child assigned to the computer is using it. At any given point in time, children are usually in groups, learning from each other. In fact, it seems children learn best when learning with others. Alexa Joyce notes that:

Sugata Mitra’s research suggests that groups of 3-4 children per computer can be more fruitful than 1:1. In groups of such a size, children readily exchange ideas and knowledge about the topic they are investigating, as well as the computer itself.

Let’s not stop at children. When they are home, they are not necessarily alone. Siblings, parents, and others are nearby and they too hear the call of a glowing screen as Walter Bender tells us:

A study done by Claudia Urrea in Costa Rica found that the majority of parents use the computer at home for their own learning – a further leveraging of the investment. Other programs, where it is infeasible to let the children travel between school and home with a computer, have instituted “technology goes home” programs – a subsidy to parents to purchase new or used equipment to have in the home. The goals of such programs have been to bridge learning from school into the home and to engage parents and siblings in the school community and in their own learning.

This new usage model, where a single learning environment over multiple technology platforms, is used by more than just students, may change the way in which we think about costs, which is one of the largest barriers to adoption, just after plain inertia & fear of change.

Costs are often calculated on a per-student basis. Yet, with siblings and parents as co-learners with their children, education leaders may change their mindset around platform costs. Instead, divide platform costs by student + 1 parent & 1 sibling. Yet also reduce costs, as there is only one software system to maintain.

And so I say we have a whole new ICT4E model with multiple platforms, a single learning environment, that empowers more beneficiaries to learn at a lower cost. A success, eh?
.

Less Top-Down Approaches

In this post, I frame the discussion somewhat differently. I assert that different communities are going to allocate their limited resources differently – not exactly a stretch. Economics, infrastructure, inertia, and pedagogy all play a role. Typically, there is a inhomogeneous collection of old and new, mobile and desktop, network-enabled and stand-alone machines available in a school, at home, and in the community at large.

This situation might change over time as in-bulk purchases for “top-down”, government-sponsored deployments of one-to-one laptop programs or terminal-server solutions become more common place, but such deployments remain the exception, not the rule. One size doesn’t fit all.

Maine's laptop learners

Even in places where such programs are being put into place on a large scale, sustaining the deployment is often a local burden. (The Maine Learning Technology Initiative has evolved along these lines – local townships are being asked to fund the “refresh” of the program, which is resulting in more diversity of both equipment and configurations across the state.)

Further, the way in which these resources are used is quite varied from place to place and program to program. Again, making reference to the Maine program, the choice of whether or not the laptops go home with the children is a decision made at the school or even the classroom level. In the case of computer labs, the schedule of access also varies – from daily use across all classes to occasional, specialized use.

Empowering a Bottom-Up Approach

It has be argued that teachers are able to incorporate computers into their day-to-day teaching only when they themselves are comfortable with the technology and cognizant of its promise. How can we help teachers and learners experiment and explore, regardless of the configuration or setting? How can we support a teacher with computers in the classroom but – as is most often the case – no administrative access to those computers and little support from the central information technology (IT) department? How can we support a school that has a computer lab, but again with little customized support from central IT?

At Sugar Labs, we are trying to address the diverse needs mandated by heterogeneous computer environments while trying to support “bottom-up” grassroots adoption by teachers, parents, and informal learning communities. Regardless of the constraints imposed by a school-district’s IT, we want to maximize learning opportunities and provide a consistent framework for teachers and students.

Taking advantage of the Fedora LiveUSB Creator, it is possible to store everything you need to run the Sugar Learning Platform on a single USB memory stick (minimum size of one GB). “Sugar on a Stick” gives children access to a personal Sugar environment on any computer with just a USB memory stick.

Sugar on a Stick on Classmate

It is the Sugar Learning Platform packaged onto a memory stick that can be plugged into almost any computer and run without affecting its “host”. It bypasses the software on the hard drive. In fact, Sugar on a Stick will work even if the host computer does not have a hard drive!

With Sugar on a Stick, the learning experience is the same on any computer: the operating system, the Sugar software, and the child’s work are stored on the stick, ensuring a consistent learning experience in school, in the classroom or the lab, and after-school, in the library, the museum, at home, or at grandmother’s house.

The initial targets of Sugar on a Stick are early-adopter teachers with “geek” parental support; but the model can be readily adopted more widely across a school district. There are a number of advantages to the Sugar on a Stick approach:

1. It reduces costs with flexible hardware choices by allowing institutions to continue using their existing investment in hardware while reducing support costs and user frustration.
2. It enables low-cost options when purchasing new computers.
3. It also makes it easy to accept donated older machines; it increases the life of older computers, reducing disposal costs and enabling the reuse of existing resources.
4. It provides a coherent and consistent computing experience even during times of fluctuating technology funding and changes in hardware choices.
5. It allows communities to take advantage of the increasing household computer ownership, while still providing a consistent, comparable computing environment.
6. It gives learners access to the projects and creations and explorations they have previously done regardless of where they did them.
7. It provides off-line access to applications and content: not every learner has access to broadband or the Internet in the classroom or at home.

Platform Agnostic Yet Education Focused

Live USB distribution need not be restricted to the Sugar Learning Platform. For example, there is a beta version of “Squeak on a Stick” being developed by Bert Freudenberg that would enable access to the Etoys environment in much the same way as Sugar on a Stick allows access to Sugar.

Also, harking back to last month’s Educational Technology Debate on the potential of mobile devices for learning, essentially the same “bits” that go on a LiveUSB image also run in a virtual machine. We are exploring the use of a Sugar VM on a mobile phone (of course, this would require a relatively high-end phone) that would provide many of the same advantages outlined above.

Our goal at Sugar Labs is to put an emphasis on learning through doing and debugging: more engaged learners are able to tackle authentic problems. Sugar on a Stick combines powerful tools with a simple and flexible medium of distribution. All of the necessary tools for guide discovery are on the stick. It is also possible to include training and curricula materials targeting specific audiences on the stick. Sugar on a Stick allows one to experience learning software with almost no effort and no risk.

The Live USB approach to distribution of learning tools to a large extent by passes the theme of this debate. The Sugar on a Stick approach allows us to emphasizes access to a learning process over any specific technology or platform.

It is great that there are many different such platforms being developed: a diversity of hardware configurations is necessary to meet the demands of schools, budgets, and cultures. But one can remain agnostic about hardware platforms and configurations, while providing a great learning experience, better utilizing the installed base of computers while tapping the potential to engage every child in critical thinking, arming them with the complementary tools of science and the arts.

“It’s an education project”, after all.