Ben Goldacre. The Guardian, Saturday 27 November 2010
Wrong isn’t enough: we need interestingly wrong, and this week that came in some research from Stonewall, an organisation for whom I generally have great respect, which was reported in the Guardian. Stonewall have conducted a survey, and their press release says it shows “the average coming out age has fallen by over 20 years”.
People may well be coming out earlier than before – intuitively that seems plausible – but Stonewall’s survey is flawed by design, and contains some interesting statistical traps.
They gathered data from 1,536 people who were already out, asking them what age they came out at, through social networking sites. Among the over-60s, the average age they had come out was 37; those in their 30s had come out at an average age of 21; in the group aged 18 to 24, the average age for coming out was 17.
Why is the age coming down? Here’s one reason. Obviously there are no out gay people in the 18 to 24 group who came out any time later than age 24, so the average age at which people came out in the 18 to 24 group cannot possibly be greater than the average age of that group, and certainly it will be lower than, say, 37, the average age at which people in their 60s came out.
For the same reason, it’s very likely indeed that the average age of coming out will increase as the average age of each age group rises: in fact, if we assumed (in formal terms we could call this a “model”) that at any time, all the people who are out have simply always come out at a uniform rate between the age of 10 and their current age, you would get almost exactly the same figures (you’d get 15, 23, and 35, instead of 17, 21, and 37). This is almost certainly why “the average coming out age has fallen by over 20 years”: in fact you could say that Stonewall’s survey has found that on average, as people get older, they get older.
But there is also an interesting problem around whether, with the data they collected, Stonewall could ever have created a meaningful answer to the question “have people started coming out earlier?” It’s a difficult analysis to design, because in each age band, there is no information on gay people who are not yet out, but may come out later, and also it’s hard to compare each age band with the others.
You could try to fix this by restricting all the data to only include people who came out under 24, and then measure mean age of coming out for each age group (18-24, 30s, 60+) in this subgroup alone. That would give you some kind of answer for this very narrow age band, but it makes some very dubious statistical assumptions. And if we allowed ourselves that move, we’d then be working with an extremely small set of data: there were only 33 respondents aged over 60 in total.
Even then, the discussion of this poll also assumes that the age at which people know their sexuality has remained unchanged. Some believe that everyone’s sexuality is fixed and known from birth – I may be walking into a minefield here – but if the age at which people recognise their own sexuality is changing, then a more relevant figure by which to measure discomfort at coming out might be the delay, rather than the absolute age.
I thought I’d already covered all the ways that a survey could get things wrong, but this one brought something new. Maybe we should accept that all research of this kind is only produced as a hook for a news story about a political issue, and isn’t ever supposed to be taken seriously. And in any case, my hunch is that a well-constructed study would probably confirm Stonewall’s original hypothesis. But it’s still fun to dig.
No comments:
Post a Comment