Frequentist vs. Empirical Bayes

Text: Wei-Ting Sun

I wrote my Bachelor's thesis last year, titled "Frequentist vs. Empirical Bayes: A comparison of procedures for high - dimensional two-sample statistical inference, in particular for microRNA data" — you can read it at https://essay.utwente.nl/92207/.

I've been invited by Ideaal! to share with you what I've done. Let's get started!

The problem

MicroRNAs are short strands of RNA that affect gene expression, and nearly 2000 human microRNAs have been identified. They can be found in bodily fluids at relatively consistent levels for healthy people, even across individuals. At the same time, research has shown that their levels can correlate with the presence of diseases, such as cancer. Because bodily fluids can be extracted easily, microRNAs have the potential to be used to non-invasively diagnose cancer.

However, to be able to do that, researchers still need to figure out which microRNAs correlate with which types of cancer. Luckily, this question can be tackled with a statistical approach. I was given a dataset containing two groups of subjects — people with lung cancer and healthy people — with data on their microRNA levels for many microRNAs. How can we then classify the set of microRNAs into two types: type 0, those with effectively the same distribution of microRNA levels for the two groups, and type 1, those with significantly different distributions of levels between the groups (which can potentially be used to test for lung cancer)? This kind of problem is called an unpaired two-sample statistical inference problem, and there are different statistical methods used to solve this problem. This is where my bachelor's thesis comes in — in my thesis, I compared two such statistical methods: the frequentist and empirical Bayes approaches.

Frequentist vs. Bayesian statistics

Before we compare the two methods, we need to first understand what the words "frequentist" and "Bayes" mean. Frequentist and Bayesian statistics are two approaches to statistics, and they differ in some fundamental ways.

Frequentist statistics is based on the following core ideas:

• The "probability" of a random event is the limit of the relative frequency of the random event happening as the number of samples goes to infinity.

• All model parameters (e.g. the μ and σ2 of a normal distribution) have fixed values, even if they are unknown to us.

A direct consequence of these ideas is that in frequentist statistics, it does not make sense to talk about the probability of a parameter being a certain value — to quantify our knowledge of parameter values, we use concepts such as p-values, hypothesis testing, confidence intervals, etc. If you've taken a statistics course, this should be familiar to you — the statistics taught in the bachelor's program (i.e. Mathematical Statistics in Module 5) all fall within frequentist statistics.

On the other hand, Bayesian statistics uses a different conception of the concept of "probability". Its core ideas are:

• The "probability" of an event is the degree of subjective belief of the event happening.

• Probabilities can be updated when new information is available using Bayes' rule: the probability of event A can be updated, if we know that event B happened, using the formula

In contrast with frequentist statistics, there is nothing stopping us from talking about the probability of a model parameter being a certain value — for example, if we have a normally distributed variable X ~ ��(μ,σ2) with unknown parameters and we know that its mean should be close to 0, it makes perfect sense to express that as

Or Even

In fact, the distribution of the model parameters in Bayesian statistics serves the same purpose as p-values and confidence intervals in frequentist statistics.

Empirical Bayes is a term describing methods that use the Bayesian statistics framework, but for some model parameters, instead of describing its value using probability distributions, it uses an empirical estimate for the value. This simplifies the problem, at the cost of being non-exact.

As you can probably see, these two approaches to statistics are quite different, and so a natural question to ask would be "How do their results differ in practice?" That is exactly what I investigated in my bachelor's thesis.

Procedure

I won't describe how the two methods work in detail here — it's not that important, and if you're interested, you can always read my thesis. What's important is that the output of both methods is a scalar value P between 0 and 1 for each microRNA that quantifies how confident the method is that the microRNA has effectively the same distribution between groups — close to 1 if the distributions are effectively the same, and close to 0 if they are significantly different — and we classify microRNAs as type 1 (different distributions) if and only if P ≤ α for some constant α. However, this classification has different interpretations between the two methods. In the frequentist method, the value P is what is called the Benjamini–Hochberg adjusted p-value — the normal pvalue is calculated, then adjusted for the fact that many microRNAs are investigated together, using the Benjamini–Hochberg procedure — you will learn about this if you take the elective course Simultaneous Statistical Inference. Using this method, the classification satisfies where 0/0 is interpreted as being 0, and in probabilistic terms, it is roughly equivalent to made to resemble the actual microRNA dataset. The simulated datasets differ by 4 parameters: the number of subjects (patients) in each group m, the number of microRNAs n, the maximum amount of deviation between the microRNA levels of the two groups c, and the number of microRNAs with a big deviation between groups a. That resulted in a total of 272 simulated datasets. I then applied both procedures to each dataset, with the classification level at α = 0.05, and compared the results.

Results

With the empirical Bayes method, on the other hand, the scalar P has a more straightforward interpretation, namely the probability that the microRNA has is of type 0, given its microRNA data. This means that when classifying the microRNA using this method, it holds that

The first thing to note is that the outputs of the two methods do not correlate strongly with each other. Figure

Notice that, despite the two methods stemming from different conceptions of probability and employing different statistical techniques, their classification false positive rate should be around the same, and so we should expect the two methods to classify the microRNAs in similar ways.

To compare the two methods, I applied them on simulated datasets

1 shows the histogram of the correlation coefficient between the values of P between the two methods for all 272 datasets, and while the correlation is positive for most datasets, for a large portion of the datasets this is close to 0.

In Figure 2, the output values P of the two methods are plotted against each other for one dataset — the dots are all over the place. Hence we conclude that while in theory the two methods should produce similar results, in practice they do not, though this is likely largely due to differences in the details I chose to implement in the two methods, and not due to the conceptual differences between frequentist and Bayesian statistics.

Now on to the results of the analyses. To illustrate and compare the performance of the two methods, for each dataset I plot the "type 1 classification rate" (the probability of classifying a microRNA as type 1) as a function of the actual difference between the microRNA levels of the two groups underlying the simulated data. Figure 3 shows how this plot changes when varying m (the number of patients per group) and n (the number of microRNAs), and Figure 4 varies c (the maximum deviation between the microRNA levels of the two groups) and a (the number of microRNAs with a big deviation between groups).

From the plots, we observe the following:

• For small m, the frequentist method tends to classify microRNAs as type 0, while the empirical Bayes method tends to classify them as type 1.

• As n increases, the frequentist method tends to classify microRNAs as type 0, while n has little effect on the results of the empirical Bayes method.

• The parameter c has little effect on results of the frequentist method, while the empirical Bayes method tends to classify microRNAs as type 1 as c decreases.

• The parameter a has little effect on either procedure.

In general, the results show that the empirical Bayes procedure performs better than the frequentist procedure at classifying microRNA with large deviations between groups when the sample size m is small. However, the empirical Bayes procedure is less consistent in its classification, and the frequentist procedure overclassifies variables with a small deviation between groups as being of type 0.

At the end, I revisited the real microRNA dataset and applied the two methods on it. The frequentist method classified all microRNAs as type 0 (effectively no difference) at α = 0.05, and at α = 0.5, it classified three microRNAs (hsa. miR.204.3p, hsa.miR.424.3p, and hsa.miR.509.3p) as type 1. The empirical Bayes method, on the other hand, classifies only hsa. miR.10a.5p as type 1 at α = 0.05, while for α = 0.5, it classified most (1067) microRNA as being of type 1, which obviously cannot be right.

Figure

Type 1 classification rate curves (solid = frequentist, dashed = empirical Bayes) with varying number of subjects per group m and number of variables n. For all plots, c = 4, a = 64, and α = 0.05.

Figure 4: Type 1 classification rate curves (solid = frequentist, dashed = empirical Bayes) with varying c and a. For all plots, the number of subjects per group m = 80, the number of variables n = 500, and α = 0.05.

I thus concluded that these four microRNAs are candidates for further research into lung cancer biomarkers.

Some final thoughts

While this Bachelor's thesis topic was initially advertised as having multiple comparisons — the thing with the Benjamini–Hochberg procedure — and the challenges of high-dimensional data be the focus, my advisors allowed me to pick really any topic related to the microRNA problem, and being a big fan of Bayesian statistics, I decided

Staff Says

to go for a frequentist vs. Bayesian statistics focus for my thesis.

When I was writing my thesis, I really struggled with time management, so I ended up, during the 3 weeks before it was due, scrambling every day to finish getting results, thinking about the analysis, and writing all at the same time. After I was done, though, I felt somewhat disappointed, as unlike more pure math theses where you prove some theorem(s), mine didn't contain any strong results — it was at best "oh, this particular method works a bit better in this specific

Text: Maria Christina van der Weele

Back to Twente

My name is Maria Christina van der Weele and in September 2022 I started working as a lecturer at the Department of Applied Mathematics of the University of Twente. I was asked to write this column since I am a new staff member, and I thought it might be a nice way of introducing myself to all of you.

I was born in Enschede nearly 30 years ago, to a Dutch father and a Greek mother. I spent 11 happy years in Twente until I moved to Greece together with my family. More specifically, we moved to Patras, which is a city on the Peloponnese on a narrow strip of land between the sea and the mountains. This is where I went to secondary school and later also to university. During my undergraduate studies I focused mostly on Pure Mathematics and I participated in several Mathematical Olympiads.

For my MSc I was admitted to the University of Cambridge, where I was fortunate enough to be award- ed a scholarship. The master’s programme has a duration of one year (effectively condensed in less than 9 months) and is called “Part III of the Mathematical Tripos”, after the fact that in the past candidates took the exams seated on threelegged stools or tripods. That year was one of the most beautiful and challenging of my life. On the one hand, studying in Cambridge was filled with Harry Potter-like scenes and being pervaded by the scientific history of the place, with giants like Newton, Darwin, Maxwell, and Hawking, to name a few; the latter was still very active at the Department of Applied Mathematics and Theoretical Physics (DAMTP) when I was a student there. On the other hand, Part III is an incredibly intense and tough master’s programme, which hardly left me any time to do other things than studying, studying and studying. case than that other particular method… maybe", and I wasn't very satisfied. But because of this, I hope that for my master's thesis, I can get things done with better time management, and tackle a topic where I can obtain more substantial results!

After my graduation, I stayed on in Cambridge and did my PhD there under the inspiring supervision of Prof. Athanassios Fokas. It was at this time that I developed an interest for the field of Partial Differential Equations and, more specifically, nonlinear integrable systems.

These systems are highly non-trivial, being nonlinear and dispersive, yet it turns out that they have special properties which allow them to be solved analytically. Among the solutions they admit, probably the most celebrated ones are the socalled solitons, which are strongly stable wave solutions that play a major role in shallow water flow, optical fibres, and other fields.

And now, after having been away for 18 years, I am back in Twente! This is really what you call the Circle of Life.

During my years in Cambridge, I found out that I very much enjoy teaching in an academic environment. Hence, I am very happy with this lecturer position at the UT. My colleagues are all exceptionally nice and I enjoy the international vibe. So far, I have been teaching mainly to students from other departments, but I am very much looking forward to meeting also the maths students of the UT. Hope to see you soon!

Text: Eva van Laar

Last Christmas I was lucky enough to be invited to spend my Christmas break in Milan thanks to the kind invitation of Lavinia. So along with her two cats (Storm and Wolf), I was driven all the way to northern Italy. During the two weeks I stayed with Lavinia’s family, I learned a lot about Italian food culture and visited many iconic sites in the city.

The largest culture shock for me was the times people eat food. Italians tend to eat lunch in the middle of the afternoon from 13-15 o'clock in the evening and as a result, dinner is also much later around 8-9 in the evening. This was a difficult adjustment for me but I could not complain because every meal I had was amazing. I had been told that real Italian pizza is much different than the American pizza I was used to back in the states but I did not realize the extent of that difference until I experienced Italian pizza for

The Pen

Text: Daan Velthuis

Whenever I turn on the TV, I am confronted with people more successful than me. Whether it be businesspeople or football players, every time someone who has done something special enters the frame, I hear my grandpa’s words echoing in my head: “Why haven’t you done that?”. I always had the same excuse ready: “They are 10 years older than me, I still have time.”. However, this December this excuse was no longer valid when I visited my first E-sports LAN.

As some people might know, I am a huge fan of the game “Rocket League”. It is a very simple game to understand. It is football with a car; the first time. The pizza dough is lighter and as a result, you are able to eat much more of it. There are also more creative combinations of toppings. For instance, in one restaurant I had a pizza topped with thinly sliced smoked ham, basil, walnuts, walnut sauce, and cheese. These pizzas are so packed with toppings that it makes sense that many use a knife and fork to eat them.

Another cultural difference was the coffee culture. As someone who doesn’t drink coffee, I cannot comment much on this but I have never seen people drink coffee so often, it was offered after lunch and after dinner. But as a hot chocolate connoisseur, I can say that the thick and rich hot chocolate of the Italians is far superior to most hot chocolates I’ve had. If you ever get the opportunity it is highly recommended.

The city of Milan is also full of interesting sites. On my trip, I went to the Duomo, a unique beautiful church made of multicolored marble. This church took nearly 6 centuries to complete and sits in the center of Milan. We climbed many stairs to reach the top of the church and got an amazing view of the central plaza below. It is also interesting to note that Milan is the home of Michelangelo’s The Last Supper which I wasn’t aware of until I visited. We also visited Castello Sforzesco which is a castle made in the 15th century and now houses many interesting museums including a museum of armory and the Rondanini Pieta Museum which houses an unfinished work of Michelangelo. Overall I would highly recommend visiting Milan if you are able to. The city is full of history, culture, and amazing food. you control a flying rocket powered car and you need to drive into the ball to hit it into the goal. For this tournament, sixteen teams from all over the world were flown out to compete in Rotterdam, Ahoy. Usually, I watched these kind of LANs at home, but with one being only 2,5 hours away, I decided to go. On Saturday the matches started and now you will see what I was talking about in the introduction. Almost 10.000 people were in the venue and a couple of 100.000 people were watching at home how the first players entered the stage. It were just a bunch of kids. Yes, the best players in the world are almost all under the age of seventeen. If I joined, I would be the second to oldest.

To further show my point, one of the highlights of the weekend was when former World Champion and Dutchie Jos “Violentpanda” van Meurs (shown in the picture) proposed to his girlfriends. Violentpanda has retired as a player at age 23, because you are basically worthless at that point in this game.

This extremely fun event, did make my excuse to my grandpa invalid so now whenever we are watching TV together and he will say to me: “Why is that not you who accomplished that?”, I will just respond: “I am not that talented, grandpa.”.