2011: The Manaiakalani Project Evaluation

What is the impact of the Manaiakalani One-to-One Device Project on literacy teaching and learning?

The Evaluation Report for 2011 was prepared by Colleen Gleeson.  It can be downloaded from the bottom of this page.



Executive Summary


Background

Seven schools in the Tamaki Cluster are involved in the e-Learning Initiative called “Manaiakalani” which links key learning from schooling improvement with developing practices from the national e-Learning Action Plan (Enabling the 21st Century Learner). It aims to raise student achievement outcomes in Literacy and student engagement. The key objective is to empower students with an evidence based belief that their personal voice is valuable, powerful and can be heard globally. It provides opportunities to become confident and effective digital citizens.

Manaiakalani began with 11 lead teachers in 2008 and increased to 15 lead teachers in 2009 and 2010 with a focus on students from Y1 to Y10 publishing their work on blogs to provide them with an authentic audience. Over 2011 one-to-one devices (Netbooks) were introduced in a staggered roll out to students from Y5 to Y13 in 18 selected classrooms. For most of the first three terms this involved 465 students in 18 classrooms, and by the end of the year there were 140 more students in 21 classrooms. Wireless networks were installed throughout all schools so that students had access to their work anywhere around the school.

The operating system, developed specifically for the purpose of teaching and learning in the Manaiakalani cluster is a Linux system with open source software in tandem with Google Apps for Education. Students used these to present their work and used files which were shared with the teacher. The Teacher Dashboard enabled teachers to view and track the history of student work. 

From 2008 to 2010 the Manaiakalani Project teachers used and developed an innovative instructional design, the Literacy Cycle, with three major components – Learn, Create, Share. This was explicitly taught to, and used as a learning framework by, students to present a digital learning object and to reflect on their work. With the introduction of the one-to-one devices the Literacy Cycle was developed to be used for all subject areas.

During 2011, two facilitators provided support and guidance to teachers. One facilitator worked mainly in the secondary college and with Principals and Lead Teachers, the other worked in all other primary and intermediate one-to-one device classrooms. Each teacher had a facilitator in their class for at least one hour per week. Principals met at least twice per term. Lead teachers met twice per term. Teachers in one-to-one device classrooms met twice per term in  discussion forums.

Technical assistance increased from 1 full time shared position in Term 1 to an additional part time position in Term 3. In addition, student technicians were trained.

The 2011 evaluation

The 2011 evaluation was designed to answer two major questions: 

(1) How does The Manaiakalani One-to-One Device Project contribute to the teaching of literacy in one-to-one device classrooms? 
(2) How does The Manaiakalani One-to-One Device  Project impact on student learning of literacy and engagement in one-to-one device  classrooms? 

Only the 465 students in 18 classrooms that effectively had a full year of implementation were part of the evaluation. The evaluation used achievement measures, video records and in situ observations of classrooms; one to one interviews with teachers and students; teacher surveys, and artifacts from class and individual blogs. 


Infrastructure

Survey data from 2010 indicated that some students had become frustrated by the limited access to multi media technology in their classrooms. In 2011 most schools had adequate equipment and by term 3 observations indicated that 13 teachers were using devices to project images as an aid to more effective teaching and learning. However, the classrooms had not been purpose built for one-to-one devices (eg for secure storage, appropriate desk tops, power sockets) which meant that adaptations needed to be made in some schools. Very few instances were observed of students handling the devices incorrectly. Observations showed a number of issues with timing of technical assistance and full implementation of the technical assistance system; repeated observations indicated that at any one time a total of 8-10 students were unable to use devices at time. In each area of infrastructure teachers had developed innovative ways of incorporating and using the devices to achieve valued aims. 


Teaching and learning

The introduction of one-to-one devices meant teachers needed to develop new pedagogical approaches and routines, which took up to two terms to establish. Observations suggested further adaptations were needed in the secondary school to facilitate consistency of use as students moved across subject lessons. 

A major change in pedagogy was the development of class Google Sites and online writing products including blogs. The Google Sites began as an online ‘taskboard’ for students to refer to and have developed into a planning document shared, and sometimes co-constructed, with students. It provides the learning intentions, success criteria, links to online reading and relevant websites, exemplars, instructions and tasks, follow-up activities and reflection questions. The design and use of Sites are being continually developed by teachers. Usage depended on teaching students how and where to file documents and to monitor that filing, otherwise work went uncompleted or got lost. By Term 3, 10 teachers used a class or subject Site and they commented that it cut down their instruction time and made the students more responsible and independent. The observations showed that students in these classrooms were less reliant on the teacher and independently (or with peer support) completed work and were able to move on to relevant activities that extended their thinking. Teachers not using the Teacher Dashboard function found it difficult to keep track of student work. 

The blog was the original “hook” to provide an authentic audience for student writing.  It began in 2008 with each teacher having a class blog and all students contributed to this blog. In 2009 one teacher implemented individual student blogs as a trial. In 2010 more teachers allowed students to have an individual blog and by the end of that year 142 students across the cluster had individual blogs. In 2011 each student in a one-to-one device classroom had their own individual blog. On average, across the cluster, individual students posted approximately once per fortnight per student compared to the average rate of posting in previous years of once per fortnight per class blog – this is a 400% increase in posts per fortnight. The majority of posts were school related work presentations, mainly text and photos with some movies and animations. 

There was some informal evidence for the tool having a generalized effect. During the Term 3 observations secondary teachers and students commented that deadlines were being achieved more easily because students were able to connect anywhere, anytime. These students have use of their devices during breaks and were observed in the grounds of the school working on their assignments. Other teachers commented that more work was being completed by students since the devices were introduced.

The literacy cycle was another pedagogical development. Term 2 observations indicated the literacy cycle, multi-media and Google applications and the one-to-one devices were employed effectively in a writing or social studies lesson but most reading sessions tended to be activity based (eg. Games, worksheet-type templates). However, the majority of classrooms in Term 3 were developing ways to use the ‘Learn, Create, Share’ framework. 

The overall classroom pattern observed was the teacher took small groups for micro-teaching while other students worked either individually, in pairs or small groups. Term 3 observations indicated 13 teachers adopted this pattern and in 13 classrooms teachers were judged as providing a learning focused environment by providing clear learning intentions, and the students knew these requirements. 6 teachers did whole class teaching and also moved around the room working with individuals.

A risk was indentified in both primary and secondary schools of effective integration of new frameworks with core foci, such as meeting NCEA requirements, or for maintaining intensity of elements of the reading programme such as guided reading. These may suffer if too much time is spent in the “Creating” phase. It appears that the teachers with an already developed strong pedagogical understanding and practice integrated the one-to-one devices into their teaching and learning programmes much easier than teachers in earlier stages of developing knowledge and practices.


Student Achievement 

A mixture of assessment tools were used within years and across schools. Both e asTTle (reading) and V4 asTTle (reading) were used and e asTTle writing. There was limited commonality across the school in the use of cluster agreed tests derived from the tools. In writing Common protocols and marking were used in 4 schools and moderation occurred in 3 schools. These inconsistencies mean that the data base is limited and reliable judgments of cluster wide outcomes similarly are limited. 

1. Reading: The evidence from e asTTle reading in Y4-Y8 (n=409) indicates accelerated gains (rates of gain higher than nationally expected rates) occurred in 4 out of 5 year levels (up to three times expected rates). The patterns were largely the same when broken down for gender and for ethnicity. Despite this only the small Y4 cohort (n=17) was at or above national expectations by Term 4, all other cohorts were greater than 20 aRs below nationally expected levels (scores within 20 aRs of the expected level may be considered ‘at’). The data from one school using  asTTle V4 at two time points indicate a different picture. The Y5 ( n=11) and Y6 (n=14) cohorts started above nationally expected levels and made smaller than expected rates of progress and were at national levels at the end of the eyar. In the Y10 secondary school cohort (n=61 from 2 ‘extension classes’) 61% of the students achieved at or above the expected mean curriculum level (5B) for Year 10 students. Given the use of two diffenrt tools it is not known what progress this represents.

2. Writing: The evidence from e asTTle writing from 4 schools in Y4-Y8 (n= 409) indicates at all year levels there was marked acceleration in rates of progress (up to six time expected rates) but no cohorts reached nationally expected levels. One school measured writing using asTTle writing V4 over the year in Y5 and Y6. Both Year 5 and 6 made accelerated progress ( four times more than the expected gain) and ended the year above nationally expected levels. The patterns were largely the same when broken down for gender and for ethnicity. 

3. Other achievement: The Key Competencies identified in the NZ curriculum were evident in the students’ attitudes and they way they worked with the devices. There was evidence of self management, independence, problem solving and critical thinking. Spot observations showed in 2 of the classrooms groups of students were outside filming for projects. Observation of these groups showed the self management and responsibility of students to complete tasks well, unsupervised by a teacher. Students in 5 classes were observed using or taking photographs independently. The annual Film Festival is a forum for schools to share the learning that has occurred during the year. The quantity and quality of the presentations was judged to be significantly higher in 2011. 


Student engagement

The introduction of one-to-one devices was associated with marked changes in the engagement of students. For example, by the end of the year in 15  classrooms the majority of students were on task from between 80-100% of the time. Student (n=415) surveys completed in Term 2 indicated students found their learning easier, had two or three strategies for solving difficulties with the Netbooks, mostly identified Netbooks as aiding their writing and mostly preferred one to one communication with the teacher for help with learning (39% identified a preference for via Google doc, or web site contact).  Teachers all commented that the quantity of work had increased since the introduction of the devices


Digital citizenship

The access students have on the internet is limited by the firewalls set up by each school, however there is always the possibility that students will access sites that are not suitable, illegally download music or movies, misuse the email, or social networking programmes or be a victim of cyber bullying. Most of the schools dealt with minor issues related to internet use throughout the year and have dealt with them well. 

It was found that students mostly used sources appropriately according to established protocols (eg use of legal images). They knew to put the text from an internet or print source into their own words but most were not sure how to do this properly. Schools have discussed issues around the misuse of the devices and have put some procedures in place. It is increasingly obvious that the students need to be explicitly taught the Information Literacy skills because of the amount of information they now have at their fingertips.


Use outside the school

The aim of the Tamaki learning Network (TLN) was to give the whole Tamaki area wireless capability so that students could access the school network from home (if they live in the designated area). The development of this network and a trial group of 40 students from two schools, living within a small section of the community began at the end of Term 3. The wireless technicians stabilised this small area in 2011, before rolling the network out across the area.

Students at Tamaki College took their netbook home from the outset and some primary school classes across the cluster began sending devices home in Term 3. However most students did not have internet access unless they had a home connection. If students could not access the internet they used ‘Gedit’ and uploaded the work when they got to school the next day. Once students began taking devices home, teachers commented that they were contacted by some students after hours and sometimes late into the night. The students wanted clarification on a point of learning or teacher comment on their work. The contacts were always learning related and while this was encouraging and no teachers complained about this contact, they were concerned that students were working until quite late into the night.

A number of recommendations are made in each area for improved implementation and evaluation in 2012.
Ċ
David Clarke,
5 Mar 2014, 14:28