225

Policy-based evidence making 

“All of our descriptive statements move within an often invisible network of value-categories, and indeed without such categories we would have nothing to say to each other at all. It is not just as though we have something called factual knowledge which may then be distorted by particular interests and judgements, although this is certainly possible; it is also that without particular interests we would have no knowledge at all, because we would not see the point of bothering to get to know anything.” - Terry Eagleton 

The ideal of evidence-based policy is a hard taskmaster. Seldom does evidence emerge that is so strong, so universal, that it is capable of “bypass[ing] politics with truth”. 

Policy-based evidence, ideally, implies a sequence. Evidence first, then policy. But is this always – ever – really the case? What if we reversed things? What is policy-based evidence making? Is it always a ‘bad thing’? 

My colleague and I first encountered the phrase ‘policy-based evidence making’ in the course of a research interview we were conducting as a part of a study investigating aspects of the regulation of animal research in the UK today. As a part of this, we were in the relationship between scientific evidence about the capacity of animals to feel pain and suffer, and the protected status different animals are awarded (or not) in the associated legislation. The phrase was used by an ex-regulator of the use of animals in science who we’d asked to reflect on her experiences of interpreting and enforcing the law in this area.

She used the phrase ‘policy-based evidence making’ critically in her reply, implying that the expediency of administration often outran the evidence base that was supposed to underpin it, in particular, with respect to the sentience of certain invertebrates and juvenile forms of common vertebrate model organisms. For example, should crustaceans be encompassed by the regulation? And if not, why are cephalopods? And what about juvenile zebrafish larvae?  

 On these and other cases, the scientific jury is to some extent still deliberating, though the law in many cases has to take a view, despite the inevitable uncertainty. 

But let’s step back. 

What is the function of evidence in politics? Common answers surely include: 

a) To legitimate policy decisions that have already been made, or which simply have to be made. 

b) To produce ‘sound’ policy decisions. 

‘Sound’ policy decisions though are in fact routinely made in the absence of expert or scientific evidence. What makes a policy ‘sound’ is contextual and relational. Importantly, this includes winning political struggles such that any policy can be made at all, as well as ensuring that the policy is administratively workable given inevitable opposition.  

Indeed, a ‘sound’ policy is an entirely distinct thing from reliable evidence, though there may be occasions in which the latter contributes to the former. 

 So, there is no a priori reason to either abhor a), or idol worship (scientific) ‘evidence’ as the only good basis for b).  

Furthermore, what fool-proof method is there to help us decide what evidence is going to be a reliable basis for policy anyway? Reliability of evidence cannot be deduced from method alone. And in either case, evidence may be a part of what is used to quell dissent in the agora.  

Scandalous and old fashioned relativism!  

 Not so fast. No one is being thrown into the populist maw here. The intuition that there is something awry going on here seems to come from the idea that policy-based evidence making and a) are synonymous and that, what is really going in the hurly burly of science-politics is the wilful disregard of certain knowledge in favour of expedience, and that this must lead to the negation of b), ‘sound’ policy. 

 Indeed, because expert or scientific evidence is so routinely thought of as the guarantor of ‘sound’ policy, any thought of post hoc evidence making is likely to smack of duplicity. 

 The idea puts one in mind of a wonderful passage about the use of expert advice in politics from the hit satire The Thick of It: 

• Yeah, well, my expert would totally oppose that [says one political operator] 

• Who is your expert? [says another] 

• No idea, but I can get one by this afternoon. You have spoken to the wrong expert. You've got to ask the right expert. You've got to know what an expert's going to advise before he advises. [the first replies] 

Jokes aside, what’s disturbing is really the idea that false evidence is being fabricated to ‘fit’ an existing policy decision and that in some cases this may lead to adverse outcomes for some sections of society.  

This is a possibility that can never be dismissed out of hand. Maybe a distinction is useful? Policy-based false evidence making versus plain old policy-based evidence making. The former bad, though not necessarily from a purely policy point of view, the latter potentially OK, depending on what one thinks about the original policy. (‘Reliable’ evidence alone obviously does not mean that all policies formulated will be beneficial to all sections of society).   

 How ‘good’ and ‘bad’ evidence – rather than policy – is adjudicated in general in the absence of the secure guide rail of infallible method is, however, a different (and certainly vexed) topic.   

The simple point here is that policy decisions can generate scientific or research activity which generates evidence – but this is no reason to get ants in one’s pants per se. Moreover, not all evidence made on a basis of policy will turn out to have any direct bearing on the policy in question at all. 

As noted above, the world of animal welfare science, and especially the field of animal sentience studies, furnishes numerous examples. (The same may be true of all forms of ‘mandated’ science?).  

It’s easy to see why in this case: Beliefs about animal sentience are inseparable from beliefs about animal consciousness, and, given that animals cannot self-report on their experiences, these are sure to be difficult matters to ever be certain about. Indeed, for this reason, our  sceptical regulator was not being dismissive. She appreciated the fundamental quandary. Society has legitimate expectations about the humane treatment of sentient animals, as well as reasons for wanting to use them in experiments, and so some decision about where the line between sentient and non-sentient animals should be drawn is for all intents and purposes unavoidable.  

Indeed, without lines, however arbitrary, the law itself would become administratively unworkable, and this would not suit anyone. 

And arbitrary does not mean indefensible. Research utilising using zebrafish embryos and larvae is only regulated from the point at which these animals become capable of independent feeding. Independent feeding is considered a proxy for the likelihood that the neural circuitry which may be capable of generating the experience of pain in these animals (if it exists), would be sufficiently advanced to in fact do so at this point in development.  

Talk about an absence of certainty.  

In any event, this point was determined – not without some debate – to be at the age of 5 days’ post-fertilisation under standard husbandry conditions. 5 days’ post-fertilisation (dpf) thus stands in as a proxy for independent feeding, itself a proxy of possible sentience. 

But the scientific evidence that is now cited about independent feeding really commencing at this time doesn’t appear to have existed when the policy was first introduced in the UK. Rather, it was only produced when the policy was standardised and extended across Europe.  

The policy, however, had proven administratively useful from the get-go for numerous reasons: it was pragmatic and provided a reasonable and broadly acceptable way of defining the point at which these animals became legal animals –  legally assumed to be sentient and capable of experiencing pain – and it kept different parties (scientists, animal technicians, regulators and so forth) on side.  

Recognising this does not invalidate the claim that independent feeding (probably) really does begin at around 5 days post-fertilization. (This said, it’s true that this does rather depend on a prior agreement about the meaning of ‘independent feeding’. Does it mean reliant on exogenous food sources for survival? Or does it mean simply ingesting exogenous food? In what state is the gut and anus required to be? All these questions have been debated.)  

Policy can drive research through the demand for evidence. The existence of the rule contributed to stimulating further research. For example, even as people challenge the appropriateness of the ‘5 day rule’ by showing, for example, that zebrafish can survive on their endogenous food reserves for longer than 5 days, a body of scientific evidence about the welfare physiology and dietary requirements of these animals develops, which is welcome on various fronts.  

The ‘5 day rule’ is however also a prompt to scientists who wish to make more direct inferences about the capacity for pain experience in fish of this tender age and younger, rather than rely upon the insecure if expedient proxy of independent feeding. So, it’s possible that ‘the line’ may eventually produce its own downfall, but also that, even if robust evidence is produced that very young fish are sentient and capable of experiencing pain, the policy will remain in place because it is ‘sound’ for other reasons.  

Debates about evidence in politics will perhaps always be functionally useful because they perpetuate the belief that all parties subscribe to a second-order agreement that all disputes will ultimately be settled rationally according to some potent if mysterious deliberative rulebook. Often taking the form of kicking the can down the road, this remains a useful source of social cohesion. Perhaps it is the loss of this facility that underpins the fear of policy-based evidence making, rather than concerns about either epistemological debasement or the production of ‘bad’ policy?  

I don’t know.  

But I do think that acknowledging these dynamics – in particular, that policy is the basis for evidence as much as evidence is the basis of policy, and both can be world-making with hard-to-predict consequences – more openly may actually be disarming and generative in our ‘post-truth’ age.  

In any event, I think doing so is unlikely to cause either the house of science or of politics to collapse. 

Reuben Message