Dear OFSTED, now about this Data Dashboard malarky …

ID-10088180Hello OFSTED,

Perhaps you can help me with a knotty little problem I am having this morning? I am trying to look at the comparative results of some local primary schools, over time. First of all, I wanted to see how they have been doing over the last ten years or so. Now for the period 2004-2010, this has been comparatively straightforward, as I can look up the English, Maths, and Science Key Stage 2 SATS results in each case, where they exist. If I ferret about on the BBC News website, I can easily find simple and easily readable data to tell me what sorts of pupils attend these schools derived from OFSTED’s own data. It’s then a simple matter to plot this onto a graph so I can map trends over time. (That’s not to say that I think SATS have ever been anything other than a blunt instrument in terms of assessing learning, but for the purposes of statistical comparisons, we have a fairly straightforward methodology there). Now it’s the period 2011-2014 that has started causing me all the problems. We appear to have had what sociologists may provocatively call a ‘rupture’, as though the data’s entrails have emerged in a disorderly fashion. I am sitting here looking at the Data Dashboard, and I am seeing the national picture, but not the regional one, which is the first step in the data being decontextualised for me. I can get the regional data, but I have to know where to dig for it. The Data Dashboard tells me that the school is ‘compared to the national picture’ on the basis of Grammar/Punctuation/Spelling, Reading, Writing and Mathematics. My eye is then drawn to the series of little boxes labelled ‘quintiles’. Now these are particularly baffling, as these quintiles are based on what is called ‘similar schools’. If I click on the list of schools associated with one of these primaries, in each case there is a massive list of institutions that vary significantly according to characteristics like these, all of which may impact on school processes and outcomes, especially when they are combined:

  • School size (the larger the school, the more valid the sample)
  • Number of pupils eligible for Free School Meals/Pupil Premium/Ever 6 (social deprivation is linked to pupil under-attainment in certain circumstances, making it difficult to measure the exact impact of an individual school, especially when deprivation is linked to pupil mobility, as it often is)
  • Pupil mobility (unless we know how long a pupil has been in a school, we cannot assess the impact of the school – we may be assessing the impact of the previous school or even an education system in a different country altogether)
  • English as an Additional Language (once again, unless we have an idea about children’s prior level of English, as well as how long they have been in a particular school, we cannot usefully determine the impact a particular school has had on their reading, writing, spelling, punctuation and grammar in English).
  • Prior attainment of pupils prior to joining the school (see my points about deprivation and pupil mobility, above)
  • Number of pupils with special education needs (in the present system, this is often defined in terms of chronology, i.e. development being behind peers, so we need a nuanced method to establish school impact once again, otherwise smaller schools with many children who have developmental delays will look as though they are underperforming compared to larger schools with those with few children who happen to be developmentally delayed).

Having established that a different definition of ‘context’ is being used to determine ‘similar schools’, I looked up OFSTED’s official documentation in order to establish exactly how these similarities were calculated. I looked here:

School data dashboard guidance http://dashboard.ofsted.gov.uk/sdd_guidance.pdf

and I also looked at the technical guidance: http://dashboard.ofsted.gov.uk/news.php

However I have some questions about the way you are calculating these ‘similarities’.

1. When you say ‘average’, do you mean the mode (most common outcome), median (the mid point of the pupil results range) or the arithmetical mean (adding up all the results and dividing them by the number of results)? These may tell us starkly different things about the way the Reading, Writing, Maths and Spelling/Punctuation/Grammar tests are formulated, and what type of results pupils in particular schools attain. To be honest, OFSTED, I am not even sure if you take your calculations down to pupil level or whether you just take the arithmetical mean for the whole cohort, and compare it to a crude arithmetical mean for the whole country (which would be fairly meaningless statistically and educationally, so I hope that’s not your approach). Which brings me to my next point.

2. Do you remove outliers from your calculations? Clearly the results of smaller schools are likely to vary more annually, and pupil mobility and the development of local housing will be significant factors here. If you can’t remove isolated results at the extremities, you are not really getting a true picture of the impact of a school’s teaching on a cohort.

3. Finally, is it true that what you are doing here is taking past performance based on some sort of average, and then projecting it forwards on the assumption that this is a stable measure (sometimes called a ‘predict and control’ model)? And then linking up the ‘averages’ (however these are calculated, see questions 1 and 2 above) to create these groups you term ‘similar schools’? This is certainly what you seem to be saying in your guidance. If so, that makes me a bit worried.

If you look in my book ‘Teachers Under Siege’ , on pages 77-78, you will see why blindly modelling forwards like this is a bad idea. I give the example of UK birth rates between 1951 and 2001. If we plotted these on a graph, we would see a particular trend over time during this period, which is down. However if we went back in time to 1963/64, we would see an ongoing and quite dramatic increase that we might assume would continue indefinitely, possibly eventually resulting in 1 million births a year. With the benefit of hindsight we know that the birth rate actually started to fall as dramatically as it initially rose, resulting in falling school rolls and school closures later on.  (Another example people often use as to why the blind modelling on limited variables fails is the oil crisis of the 1970s. Many oil companies assumed continues exponential growth and ordered new tankers and plant accordingly. Shell, on the other hand, asked itself the question ,”What do we do if it doesn’t continue to grow?” and positioned themselves more intelligently within the market. They got to eat all the proverbial pies while other companies were left with oil tankers and plant they couldn’t use). Now clearly school and pupil attainment are a different kettle of fish. First of all, it is very difficult to quantify the impact of schooling precisely, particularly amongst 11 year olds of varied backgrounds. This is why I said earlier that SATS were something of a blunt instrument. We are not counting the numbers of births or barrels of oil here. Also drops in birth rate are not an indication of failure amongst the childbearing population, just as politically-driven drops in oil production and distribution in 1973 did not automatically mean that oil executives had failed. However even if we take test results at face value, the modelling you are using still looks odd. Why would it be helpful to group schools together on the basis of their ‘average’ results without taking into account any other variables? If you really think this is worthwhile, OFSTED, then you need to make your methodology and justification a lot clearer than they are here, I would suggest. Otherwise it is difficult for us to have confidence in your processes and outcomes.

Now OFSTED, I want you to feel free to comment below on this. Many of us are genuinely perplexed by the Data Dashboard and would welcome clarification.

With best wishes,

Dr Leaton Gray

[Image Courtesy of Stuart Miles, Free Digital Photos]

Excellent or going wrong?

sandraleatongray:

You make some valid and important points, particularly about the extra-curricular aspects of a good school-based education.

Originally posted on Mr Brown says:

I have been looking at two posts on the characteristics of an excellent school or a school that is going wrong. It is not a definitive checklist for good or bad practice but it is pretty hard to argue with the characteristic placements

I did think there were a couple of notable exceptions. For example nothing was made of intervention. A good programme of targeted intervention can promote rapid progress in certain students. One pupil I teach would be dallying through the curriculum missing lessons and falling behind but because of the excellent intervention sessions set up. However some intervention sessions can just rob Peter to pay Paul. Pupils need catch-up reading during maths lessons which causes them to require numeracy catch-up later in the term so he is removed from PE (his favourite subject). Pupil gets annoyed because he misses PE, his self-esteem is lowered because people keep…

View original 132 more words

When education is going wrong

Bentham's Panopticon - prisoners can be watched from the watch tower at any time of the day or night, thanks for permanent illumination.

The flip side to my previous post about what excellent education looks like, is what poor education looks like. Again, this is all open to debate, but my list is aimed at making a start. Even some schools generally regarded as successful may want to think about their professional practice if they see any of these negative characteristics on their own turf.

  • It is hard for others to work out exactly what is going on in a teacher’s classroom.
  • There are high rates of casual and long term teacher absence, and high use of supply teachers and trainee teachers.
  • There is a blame culture within the school, with teachers excusing poor attainment on the grounds of children’s social disadvantage, management shortcomings, or children’s resistance to schooling.
  • The same children tend to answer questions and participate in lessons, while others remain quiet and sometimes disengaged.
  • Children in the top ability groups are regularly used to coach other children.
  • Children in the bottom ability groups are confined to separate tables and effectively taught by (usually unqualified) Teaching Assistants.
  • Teachers review children’s progress only in response to Government policy.
  • Teachers insist on high levels of pupil conformity in terms of humour, interests, and attitudes towards society.
  • Staff morale is low and teachers do not socialize with each other outside school.
  • Teachers and children feel they need to behave in a physically rigid, controlled manner in school. for example always looking forwards and attentive.
  • Teachers are not sure what children know, and where there are gaps.
  • Access to books and educational resources is controlled and rationed.
  • Children and teachers spend a lot of time discussing discipline.
  • The methods used when teaching do not fit the task, for example using direct instruction at the wrong time, or too much unfocused small group discussion that is off task.
  • Lessons drift away at the end, without any summary.
  • Learning is focused on achieving Government targets.
  • Higher status is theoretically given to mathematics, the sciences and English, but children are not taught by teachers properly trained in these subject areas.
  • In school, the emphasis is on getting through the curriculum rather than developing knowledge and expertise, and developing an intellectual life.
  • Children and parents are reluctant to come into school, speak to teachers, and support school events.
  • When parents come into school, they are spoken to as pupils and required to sit on small chairs.
  • Discipline is obvious, sometimes loud and varies in its application.
  • Former pupils do not remain involved with the school.
Posted in General. 1 Comment »

What does excellent education look like?

handsIn a school governors’ meeting recently, I was speaking my regular motherhood and apple pie piece about the need for excellent education, and about nothing being too good for the kids in our care, and quite rightly one of the other governors asked me the $64,000 question. that I had repeatedly sidestepped in previous meetings.

“Sandy, can you tell us exactly what excellent education is? And how will we know it when we see it?”

That put me on the spot. I am aware very many people have tried to define excellent education, and that there is great variation in the priorities different people have in seeking to ensure it happens, which makes describing it rather daunting. However here I have decided to lay out what I think the process looks like when it is happening, and what prospective parents and teachers might want to look for if they visit a school. Feel free to comment if you want to debate it; I am open to persuasion and argument.

  • Teaching is well organized and teachers have well-established, consistent routines easily understood by other teachers, children and parents.
  • Teacher absence rates are low, and there is little use of supply teachers.
  • Teaching is personalized and properly differentiated. Teachers are aware of what their pupils know, and don’t know.
  • All children are routinely encouraged to answer questions and participate in lessons.
  • Children in the top ability groups are given enrichment tasks and further study opportunities when they have finished their work, rather than being used to coach other children.
  • Children in the bottom ability groups have plenty of contact with the most experienced teachers in the school, and are not confined to separate tables and effectively taught by Teaching Assistants.
  • Teachers review children’s progress frequently, and communicate this to children, teachers and parents.
  • Teachers understand the context of their children’s lives outside school.
  • People associated with the school like each other and are happy working together.
  • Teachers and children feel they can express their own personalities at school.
  • Children have no gaps in their knowledge. If a child misses something at school because of illness or other absence, the teacher advises the parents and helps the child fill the gap.
  • Children have access to good books and educational resources, and willingly take advantage of what is on offer.
  • Children and teachers spend a lot of time discussing teaching, learning and knowledge, to mutual advantage.
  • Learning involves a mix of methods, appropriate to particular tasks. These can include direct instruction, small group discussion and collaboration, self-study, and plenary sessions.
  • Lessons are summarized at the end, usually through group discussion.
  • Learning is linked with the local area and the outside world.
  • Equal status is given to mathematics, the sciences, the arts and the humanities and children are taught by teachers properly trained in these subject areas.
  • Children have the possibility to extend their personal knowledge and interests through independent or guided study.
  • Children and parents enjoy coming into the school, speaking to teachers, and supporting school events.
  • Discipline is quiet and consistent.
  • Former pupils are happy to return to the school and support it.

NAHT Education without soundbites

NAHT Education without soundbites. Here journalist Susan Young writes about the panel debate on 17th June that we organised, on the subject of “Whose Schools Are They Anyway?”

Part III – Assessing aptitude

teacherSo far in my series of three linked series of three posts we have addressed how to tell if someone has general ability, and how to tell if they are learning anything at all. This post will comment on how schools try to work out if a pupil has potential in an area of specialism directly related to what the school offers. Once again, like assessing intelligence, this is a highly controversial area. We found this during the period 1944-1979 when the 11+ examination was sat by most children in Britain. They were tested in English, Maths and Verbal Reasoning, but whether they got a coveted place at grammar school often depended on gender (there were more places for boys in many areas, and they needed a lower mark to access grammar schools), and social class (middle class children outnumbered working class ones in grammar schools).

One of the problems with all aptitude tests is judging whether they demonstrate aptitude or in fact only test achievement. A good example might be music tests – you are considerably more likely to be able to differentiate between minute musical tones if you have been learning to do so on the violin for five years.Indeed, it is possible to pay for a course that will help you prepare for such a test:

http://www.hofnote.co.uk/courses.asp?id=15

In terms of aptitude tests, there is also perceived to be a bias in favour of the middle classes, as they have access to a broader range of extra curricular activities to support the development of ability in various specialised areas, such as learning modern foreign languages. For that reason, maintained specialist schools are now only entitled to select 10% of their secondary pupils via selection bands, and this is overseen by the Schools Adjudicator.

For interest, there are some language aptitude tests from Oxford University here:

Modern Languages Admissions Tests.pdf (256 kb)

The moral of the story is that, while aptitude tests can sometimes show us talent where it might not otherwise be spotted, just as often it shows us the background of the people being tested. The secret is knowing the difference.

 

 

RIP Rob Moore, an open letter from one of your former doctoral students

RobDear Rob,

We were an unlikely couple, you and I. Invariably I would be sitting there in some country pub or other in smart trousers and pearls. with you opposite in your rumpled outfit straight from Sociology central casting. Between us would be placed carefully on the table one pint of bitter (for you) and a half of IPA (for me),  and some scrappy pieces of academic writing I had created in an attempt to make progress towards my elusive dissertation. We would be in a pub because then I knew I would have you captive for two or three hours at least, during which time I would be asking you all sorts of simplistic questions about sociology while patiently, you would break down the entire discipline so I could apply it to my rather mad cap doctoral project. Now that’s what I call teaching.

It didn’t stop there. A rite of passage at most universities is to end up doing a lot of teaching, but having nowhere to do it, so you allowed me to share your office in the attics of Homerton College and use it for supervising undergraduates, and research. I would sit in there, surrounded by prints of historic Norwich, and your Commitments poster, pulling your books off the shelf and generally hearing your calm sociological voice in my head.  I do now when I am writing this. If you were still here, you would probably take me out for chips somewhere, and you would tell me that if I just calmed down a bit, and listened, you had a plan. And then you would relate the plan, and it would be immensely logical and brilliant, and the writing would just come. Because that’s what you did.

You were the one who first made me read Basil Bernstein’s work – Basil had been your own PhD supervisor. You told me to get out of the library, and to grab pens and paper and spread out his ideas all over my living room floor. You then introduced me to the great man himself. On this occasion, naturally all logical thought escaped me. We ended up with a less than earth shattering conversation about which of Basil’s book chapters to read next, in relation to my study of teachers.  But you didn’t mind, because all you were really interested in was knowledge, and you always treated your students as intellectual peers joining you on the journey.

But your legacy goes well beyond me, and all the beers, and all the reading. You leave behind a body of work on the Sociology of Education so precisely conceived and formulated that you helped transform thinking in the field, from the highly technical work of Bernstein to the Social Realism that pervades your writing.  This has filtered down to many of your postgraduate students, and in turn we have now become research supervisors, bringing on another generation of educational sociologists. So in Sociology terms, you live on.

RIP Rob, and I hope the beer is good up there.

Sandy x

 

 

 

Follow

Get every new post delivered to your Inbox.

Join 1,066 other followers

%d bloggers like this: