You Can't A/B test in Reality

As a programmer, I get to run usability tests. So when a user comes to my website, I can offer 50% of the traffic one version and 50% of the traffic another version, and see which works better. This is called A/B testing or split testing.

Ah if only we could do this in real life. Would crime really go down if we got rid of (legal) guns in D.C.? While I believe this to be the case, ultimately we can't know unless we do. And even when we do, there are other factors - perhaps a contract for a new stadium created many new jobs, lowering crime rates. The point is, we can not perform perfect tests. So what do we do?

When discussing issues, its important to agree upon metrics. While this sounds obvious, its tricky because metrics suffer the A/B testing problem. Looking again at gun control, one person might feel looking at two different areas covering the same time span is a good metric, while another person might feel that looking at the same area over different time spans is a good metric. Amazingly, before the issue of gun control even comes up, the pre-debate about which metrics are best often turns out to be a difficult talk. The time is not wasted however, because in actuality you are learning the deep (usually unspoken) beliefs about what drives someone's final opinion.