In Part I of this blogpost, I outlined the rationale for evidence-based policymaking. Part II focuses on the different approaches that could be used and their relative merits.
In order to determine the effects of a program, the analyst could adopt the following strategies:
i) Analyse a number of individuals before and after the ‘treatment’ (i.e. participation in the program). However, this approach will not help the analyst disentangle the effects of the treatment from other contemporaneous factors.
ii) Observe two identical individuals (one who receives government assistance and one who doesn’t) and observe them over time. However, it is impossible to have two identical individuals. If there are systematic differences between the two, then it is difficult to disentangle the effects of the policy from differences in the individuals.
iii) Observe both treatment and control groups before and after the participation in the program. But this doesn’t enable the researcher to separate the effects of the program from other contemporaneous factors.
iv) Observe the same individual in two different states of the world at the same time (which is impossible).
v) Take advantage of a quasi-natural experiment where one group is subjected to an exogenous effect and an otherwise identical group is not. By ‘exogenous’, we mean the affected people had no choice in whether or not they are part of the treatment group.
vi) Randomly allocate individuals to either a treatment or a control group and compare them over time.
The disagreements between the ‘experimental’ and ‘non-experimental’ camps – often referred to as the ‘randomistas’ and the ‘regressionistas’ respectively – are quite marked.
The experimental approach in social science mimics the approach adopted in the natural sciences by randomly assigning individuals to a ‘treatment’. By effectively flipping a coin to determine whether an individual is in the treatment or control group, it is possible to overcome the problems associated with unobserved differences (e.g. differences in how motivated the individuals are) in the ‘treatment’ and ‘control’ groups.
The non-experimental approach uses econometrics to try and deal with the fact that people choose to participate in the program (i.e. participation is not random). Selection into the program on factors that we cannot observe directly (which may be correlated with variables of interest) is indeed a major problem. But with the advent of more (and cheaper) data, econometricians can deal with this. However, self-selection is often done on unobservable characteristics which are extremely difficult to capture.
What limitations there are with regard to the ability of the experimental approach in social science? One obvious pitfall in the social sciences is the lack of a ‘placebo’: in medical trials, two groups are given pills, but one is given a pill which turns out to have no active ingredient. This approach simply can’t be imitated in the social sciences: it is impossible to fool the members of the control group into thinking they might be receiving a treatment when they aren’t.
Experiments are also potentially subject to ethical concerns. The counter to the proposition that it is unethical to simply toss a coin to determine who receives the ‘treatment’ is that the reason the randomised trial is taking place is to determine whether the policy works: it is only unethical to conduct the trial if we already know that the policy works. If you don’t know whether a specific policy works, it is unethical i) to do nothing; or ii) not to conduct an experiment. However, there is residual concern that experiments have been used in areas where we do know whether the policy works.
Another concern with experiments is that they are typically conducted in environments with unique characteristics which may not be representative of all possible environments. Therefore, the results observed in one setting might not be generalizable to all contexts. Problems of this nature arise in non-experimental analysis too. But experiments tend to get criticised for this shortcoming more than other methodological approaches simply because experiments have solved most of the other methodological issues!
Notwithstanding these concerns, there is much more that could be done to improve the quality of the evidence base used in designing public policy. For example, we should renew efforts to improve access to unit-record data, think carefully ex ante about how to design and implement program evaluation, continue to build capability within the Government with regard to undertaking evaluation, and continue to promote the adoption of best-practice evaluation methodologies.
Taking these steps is crucial if we are to continue to live in a prosperous nation. Moreover, it will help ensure that we spend taxpayers’ money wisely and prudently, which is an important part of the covenant between government and the people.