Prior Expectations and Bayes' Theorem

I am the most naive and gullible person I know.  I mean, I'm the reason they have those announcements in airports telling you not to accept any bags from strangers.  That's an extreme case, but there are more realistic applications of my gullibility, too.  Coin flipping, for example.

Coin flipping is my go-to method of settling disputes.  But that method is only fair if the coin is fair, meaning it's just as likely to return Heads as it is to return Tails.  That's not always the case, and this blog post is going to tell you why.

First, I'll start with a simple description of Bayesian Statistics.  As far as I can tell, this is exactly like the statistics we've been doing, but instead of letting the data dominate every step in the fitting process, we have things called "priors."

Priors are just expectations you have about what the parameters you're trying to find will be.

Let's say, for example, that someone hands you a bowl of ice cream, but doesn't tell you what it is.  You have a set of expectations -- or priors -- about what that ice cream will taste and feel like based on your experience (in science, that experience can be your own or it can be knowledge gleaned from already-done studies).  You expect that ice cream to be sweet and cold, right?  Well, plot twist! It's bacon ice cream.

Now that you've taken that bite and recovered from the initial shock, will you expect the next bite to be sweet? No, you know it's going to taste like bacon.  This is because you've adapted your expectations -- or modified your priors -- to match the data you observed when you took that bite.

In math language, this process looks like this:

$p(a|D) = \frac{\pi (A)\emph{L}(D|a)}{E}$

where \(\pi (A)\) is your prior expectation for parameter A (the flavor of the ice cream) and \(\emph{L}(D|A)\) is the likelihood of observing your Data D if A were true (what's the likelihood that you're eating ice cream if it tastes like bacon).  E is just a scaling factor used to represent the evidence you gather as you conduct more research. 

We wanted to test this out to gain a better understanding. We didn't have any ice cream readily available (and I wouldn't really want to eat bacon flavored ice cream, anyway), so we flipped coins instead.  In this case, the question we were trying to answer is: Is this coin fair?

We used an Oregon quarter from 2005 (with the custom state design, not the standard eagle) to conduct this test.  We flipped it 20 times and recorded how many times it returned Heads. 

We started with a prior of .25, meaning the coin should return Heads about once every four flips.  This was actually cheating a little, for the sake of learning.  We actually expected the coin to return Heads about half the time, but we wanted to see how the prior can be overtaken by the observed data. 

I don't know if it was a self-fulfilling prophecy type thing or if the coin was actually unfair (I'm leaning slightly toward the latter), but the coin returned Heads 5 out of 20 times. We plugged those numbers into our equations for likelihood and then plugged those numbers into the probability equation above, and voila! 

Everything converges around .25, which tells us that the probability of that 2005 Oregon quarter returning Heads really is 1 in 4.  I guess I know now which coin I'm using if I ever want to settle a dispute with someone.  

But, we weren't done.  Our data just happened to serendipitously match our priors.  That's great for science, I guess, but it's not all that great for learning.  So I re-did the analysis with the exact same numbers, but with a different prior, this time telling my computer that I expect the coin to be fair.  This is what I got:

The lines all appear to have the same amplitude because they've all been normalizes to have an amp of 1.  In reality, the lines corresponding to higher Ns would have smaller amplitudes because they represent smaller likelihoods.

Here, you can see that the prior is 0.5 and the first line is close to that (the green line corresponds to the point in the trial where we had flipped the coin 5 times and it had returned Heads twice).  But the others are far away from being fair.  They are closer to .25, which is the true probability of the coin returning Heads.  

I really liked this exercise.  It provided me with empirical evidence that there's a flaw in the way I view the world.  If something as basic as a quarter can lie so egregiously to me, I should really try to dial back that gullibility.