Evidence of the impact of technology on learning is a hot topic and, due to the seeming intractability of the issues that surround the debate, is likely to continue to be for quite some time. I’m not going to waste your limited patience with explanations of why notions of ‘truth’ and ‘proof’ in the domain of educational efficacy are problematic. If your measuring instrument is a sundial (designed to roughly approximate the time during daylight), don’t be surprised if time still passes even when it’s cloudy. The invisibility of this progress is more likely to be a limitation of the measuring instrument than it is to be that you’ve managed to pause the Universe.
Excellent – that’s the overblown and barely-coherent metaphor out of the way for yet another post. Now we can get down to business.
At the tail-end of the last academic year, I kicked off a project with five of the schools for whom I work to try and identify if the 1-to-1 provision they were making for students was having an impact. You can read about the mixed methods we’re employing to try and triangulate qualitative progress data and quantitative attitudinal trends here.
In this post I am going to share the first half of the pre-test question set that the children responded to and pick out any of the interesting things I have noticed. You may spot things I miss, for which I’ll be grateful. This set of questions will be revisited with the same population post-treatment to see if anything has changed – this is due to take place in the summer term. The questions themselves were deliberately written without reference to technology – they’re an attempt to sneak up on its impact via a tangential path. Many of the questions build upon what we learn from meta-analyses of studies by John Hattie and the Education Endowment Foundation. The underlying hypothesis goes something like ‘These processes have been shown to impact on learning, does technology appear to enhance, accelerate or improve them?’
1092 pupils from 5 schools in Years 7, 8 and 9 responded to the survey. The questions were all statements which the respondents were asked to ‘Strongly Disagree’, ‘Disagree’, ‘Agree’ or ‘Strongly Agree’ with – a 4-point Likert scale.
- ‘I think I am hardworking’
One’s view of oneself has a disproportionately powerful impact on performance. The children gave surprisingly positive responses here – very little to pick between the 3 year groups, with 98.2% of Year 7s giving a positive response. This dipped to 93.5% in Y8 before showing a small recovery by Y9 (94%). Perhaps more telling was the gap between genders, where the male respondents were less likely to characterise themselves as hardworking – there’s a gap of almost 5%.
- Research question: will the introduction of mobile technology have an impact on self-perception of work ethic?
- Hypothesis: Positive impact on treatment group, related to task authenticity/ engagement. This will certainly need unpicking in follow-up interviews
- ‘I find it easy to learn new things’
A question aimed at the pupils’ confidence in themselves as effective learners. Responses followed a similar pattern to the question above, with Y7 the most positive (88.1%), a big dip in Y8 and recovery in Y9. Boys, again, were 4.3% less positive about this aspect of their learning than girls.
- Research question: will the introduction of mobile technology have an impact on self-perception of the ease of learning new skills/ knowledge?
- Hypothesis: Positive impact on treatment group, possibly short-lived due to exposure to new tools/ methods.
- ‘My handwriting is pretty good’
This was an interesting one. Year 7 positive responses totalled 79.5%, slipping a couple of % in Y8 and falling off a cliff in Y9 (67.2%). Boys again gave a more negative assessment here – only 61.3% Agreed or Strongly Agreed with this statement (girls: 82%) and 7.5% Strongly Disagreed, a high percentage in the context of other questions.
- Research question: will the introduction of mobile technology have an impact on self-perception of the quality of handwriting?
- Hypothesis: Negative impact on treatment group, as less time is spent writing on paper. It is also possible that the decline in the importance that handwriting has on positive reinforcement (as children receive more praise for digital work) will perversely result in an increase in this measure.
- ‘I can spell most words that I need to without a problem’
More interesting numbers which confounded my prejudices: confidence steadily rose with age (Y7 82.3% positive responses, Y9 85.3%, despite the normative phrasing of the question) and boys were almost 4% more positive than girls in their ability to spell – the only question in which the gender gap was in boys’ favour.
- Research question: will the introduction of mobile technology have an impact on self-perception of one’s ability to spell without issue?
- Hypothesis: Positive impact on treatment group, possibly obscured by the technology bearing more of the spelling workload (auto-correct, etc). Consider including spelling test data to triangulate.
- ‘I can produce detailed written work of a good standard’
Confidence fell as respondents aged – most noticeably in Y9. Perhaps this is related to the quantity of and quality of the written outcomes expected by this stage of children’s schooling? Male respondents were noticeably less affirmative in this respect (though still largely positive – 87.4% compared to the girls’ 95.5%)
- Research question: will the introduction of mobile technology have an impact on self-perception of the quality of one’s written output?
- Hypothesis: Positive impact on treatment group, due to the scaffolding effect of digital composition and pre-writing tasks carried out with technology.
- The feedback my teachers give me helps me to make progress in my learning
The first question about an external process and its impact on their learning. Cynicism – or perhaps acuity – about the impact of marking/ verbal feedback increased with age, no one will be shocked to discover (Y7 91.2% Agree/ Strongly Agree, Y9 84.9%) and boys were around 4% less positive than the girls surveyed. There was a dramatic drop-off in ‘Strongly Agree’ responses between Y7 and Y8 – from 42.3% to 23.3%. 4.4% of boys Strongly Disagreed with this statement.
- Research question: will the introduction of mobile technology have an impact on the perception of the usefulness of teacher feedback for learning?
- Hypothesis: Positive impact on treatment group, due to the increased pace of the feedback process and the richer tools available to teachers (video, voice, comments).
- The feedback my teachers give me is regular and happens soon after I’ve done the work
This one tries to get at the notion that for feedback to be at its most impactful, it needs to happen in a timely fashion (e.g. while the pupil has fresh memories of the work concerned). The Year 7 optimism effect is again apparent – the fresh-faced 86.5% positive responses crashes to 67.9% by Year 8. Boys’ relative negativity (8% more negative responses than girls) is interesting, and perhaps reveals inter-school differences, as some of the schools involved are single-sex.
- Research question: will the introduction of mobile technology have an impact on the perception of the rapidity and regularity of teacher feedback?
- Hypothesis: Positive impact on treatment group, due to the increased pace of a digital feedback process and its ease leading to more feedback given compared to the same time spent on a paper process.
- I always know what homework I have and can organise my time so that it’s not late
I expected noticeable disparities here between genders and a gradual decline in confidence with age. I was not to be proved wrong by this initial data, though as with all the responses so far, the picture was a broadly positive one, with 77.4% of Y9s Agreeing/ Strongly Agreeing with the statement.
- Research question: will the introduction of mobile technology have an impact on self-perception of organisation/ time-planning skills?
- Hypothesis: Positive impact on treatment group, due to the digital organisation/ homework tools available to them and to their teachers
- Homework is as useful in helping me to make progress as my lessons are
I wasn’t anticipating that many children would wax lyrical about the positive impact of homework on their lives. Y7s begin by toe-ing the party line (82.1% are positive about this) but Y8 and 9 responses are down in the high-50s and 13-14% of respondents Strongly Disagreeing. Boys lag 10.5% behind girls overall, with 63.5% Agreeing/ Strongly Agreeing. To me, this strongly suggests that homework is not felt by those on the receiving end to be having the intended impact.
- Research question: will the introduction of mobile technology have an impact on the perception of the efficacy of homework in making learning progress?
- Hypothesis: Positive impact on treatment group, due to the increased authenticity of homework tasks and the richer tools available to work with others and produce an output.
10. I can concentrate on my work without getting distracted too often
Reduced attention span is often identified as a negative around the use of technology by children. This data revealed the by-now-familiar pattern of decline in positivity through the Years – 82.8% to 76% to 65.7% and seems to confirm the perception that boys struggle more than girls in this respect (70.1% Agree/ Strongly Agree as opposed to 79.8). Of course, this could just be an echo of everything these young people are told by adults about their generation…
- Research question: will the introduction of mobile technology have an impact on the self-perception of ones ability to concentrate on school work?
- Hypothesis: Negative impact on treatment group, due to the increased availability of communications channels and leisure activities on a personal device. It will be a very interesting one to analyse and triangulate with interview data.
So what have we learned?
- ‘School effect’ seems particularly strong, with Y7 responses being dramatically more positive than those of children who have been at the schools for more than a year. Maturity and other factors related to child development could also be at play here. Any analysis of the post- survey will have to consider this confounding factor.
- It may be hard to separate out what children actually think about themselves/ their learning from what they have picked up from adults – e.g. their responses may to some degree reflect what others think about them.
- Gender differences are visible in all the responses, almost always in the disfavour of boys. Boys’ perceptions of their learning behaviours/ attitudes seem lower than girls’.
In the 2nd post in this series I’ll look at the final 10 questions that were asked.