Why is causal inference so complicated? Why does the relationship between, say, the number on police forces on the one hand, and crime rates on the other hand, tell us nothing about the causality between the two? It has to do with two major problems related to observational stud-ies (studies in which the independent variable is not manipulated by the researcher), namely reverse causality and omitted variable bias (confounding).

Reverse causality implies that by looking at the relationship between two variables (say, X and Y), one is not able to discern what the direction of the causal channel is; is it X-->Y or X<--Y or maybe both? Coming back to the example above, if a researcher observes a positive rela-tionship between the size of police force and the rate of crime, would she conclude that police causes crime? The answer is of course no, as it may also be the case that more crime leads to more police deployment.

The omitted variable bias, on the other hand, stems from the fact that the observed relationship between X and Y might be induced by a third variable Z, which causes both X and Y. Suppose you found a negative relationship between the number of police forces and crime rate. This relationship may be driven by a third variable, such as affluence approximated by the average income of residents living in a given area. Wealthy people are less likely to commit (street) crimes, but more affluent areas can also afford more police forces. If that holds true and one does not control for affluence, the relationship between the size of police force and crime rate is confounded.

Designing causal research

So how to design your research if the goal is to make a causal statement? The gold standard is to perform an experiment, whereby an explanatory variable of interest is assigned randomly to subjects under investigation. The power of experiments, and randomization in particular, lies in the fact that it makes confounders irrelevant.

Incentivized experiments typically involve a monetary compensation. They come in two main forms.

• Field experiments tend to have a higher external validity, as they are performed in more generalizable setups. The downside is that the environment is often less control-lable, which may impair the internal validity of the experiment.

• Lab experiments score well in terms on internal validity as in these setups researchers typically have a high control over experimental treatments. Yet, the type of respond-ents participating in lab experiments (usually students) precludes making generaliza-tion to other groups in the population.

Survey experiments, often population-based, are compatible with research investigating how audiences perceive law or, for instance, how audiences would hypothetically react to the intro-duction or breach of legal rules.

Performing experiments, however, is not always feasible. Consider again our example of re-searching the effect of police on crime. To randomly assign numbers of police officers to var-ious municipalities would be complicated for various reasons. In these cases, one could use experiments that have occurred by chance.

These so-called natural experiments are setups whereby random assignment was not led by a researcher but was triggered by natural forces (such as a natural disaster) or by someone’s arbitrary decision (such as drawing a discretional border). Studies that employ natural experi-ment designs provide some of the most robust results in research on the effects of police on crime. Natural experiments are predominantly used by economists and political scientists, but are getting increasing attention from empirical legal researchers.

In cases where neither lab, nor field, nor natural experiments are applicable, there is still a plethora of other options.

If researchers have access to a large number of observations, they can perform multiple re-gression analyses with the goal to control for as many confounders as possible. If, however, the number of observations is small, then comparative case studies methods, such as most similar system design, are more suitable. Although regression analysis and comparative case studies differ in terms of how the analysis is performed, the general idea of controlling for confounders is the same. Note that these designs score lower in terms of reaching causal infer-ences, since it is not possible to control for all confounders because they cannot be a priori identified or measured.