15.5: Testing Statistical Significance

• Robotics is an experimental discipline. This means that algorithms and systems you develop need to be validated by real hardware experiments. Doing an experiment to validate your hypothesis is at the core of the scientific method and doing it right is a discipline on its own. The key is to show that your results are not simply a result of chance. In practice, this is impossible to show. Instead, it is possible to express the likelihood that your results have not been obtained by chance. This is known as the statistical significance level. How to calculate the statistical significance level depends on the problem you are studying. This section will introduce three common problems in robotics:

1. testing whether data is indeed distributed according to a specific distribution
2. testing whether two sets of data are generated from different distributions
3. testing whether true-false experiments are a sequence of luck or not

15.5.1. Null Hypothesis on Distributions

The Null Hypothesis is a term from the statistical significance literature and formally captures your main claim. A statistical test can either reject the Null Hypothesis or fail to reject it. It can never be proven as there will always be a non-zero probability that all your experiments are just a lucky coincidence. The statistical significance level of a Null Hypothesis is known as the p-value.

An import class of Null Hypothesis are on the distribution of data. Consider the following example from Lab 1 (message passing in ROS). Students were asked to experimentally study the time it takes to pass a message from one process to another:

• Histogram of the time it takes to send a ROS message from one process to another based on 10 trials.

We observe three peaks in this Histogram. What can we say about message passing times? For example

• H0: Message passing times follow a Gaussian distribution.
• H0: Message passing times follow a bi-modal distribution.
• H0: Message passing times follow a log-normal distribution.

The first Null Hypothesis implies that messages take sometimes a little longer and sometimes a little shorter, but have an average and a variance. The second Null Hypothesis implies that usually messages take some low average time, but occasionally are delayed due to the influence of some other process, for example operating system duties. You can now test each of these hypotheses by calculating the parameters of the distribution to expect and calculate the joint probability that each of your measurements are actually drawn from this distribution. You will find, that all of the above hypotheses are almost equally likely. Together, none of your tests will reject your hypothesis. You therefore will need more data:

• Histogram of message passing times in ROS based on a 1000 trials.

You can now again calculate parameters for each distribution you suspect. For example, you can calculate the mean and variance of this data and plot the resulting Gaussian distribution. In this example, the Gaussian distribution will have a mean slightly offset to the right of the peak. You can also fit the data to a log-normal distribution. You can now calculate the likelihood for the data actually be drawn from either of the two distributions. You will see that the joint probability (the product of all likelihoods) for all data points is actually much higher than that for any Gaussian distribution or any bimodal distribution that you are able to fit.

Formally, this can be done by following Pearsons χ² -Test (read Chi-Squared Test). This test calculates a value that will approximate a χ² -distribution from all samples and the likelihood of that sample based on the expected distribution. Plugging the resulting value into the χ² -distribution leads to the statistical significance level (or p-value).

The value of the test-statistic is calculated as follows:

\[\chi ^{2}=\sum_{i=1}^{n}\frac{(O_{i}-E_{i})^{2}}{E_{i}}\]

where

• χ² = Pearson’s cumulative test statistic, which asymptotically approaches a chi-squared distribution.
• O_i = an observed frequency in the data histogram
• E_i = an expected (theoretical) frequency, asserted by the null hypothesis, i.e., the distribution you think the data should follow
• n = the number of samples.

This example also illustrates how statistical tests can be used to determine if you have enough data. If you don’t, you will get very poor p-values. In practice, it is up to you what likelihood you determine to be significant. Standard significance levels are 10%, 5% and 1%. If you are unsatisfied with your p-values you can collect more data and check, whether your p-value improves.

15.5.2. Testing Whether Two Distributions are Independent

Testing whether the data of two experiments are independent is probably the most common statistical test. For example, you might run 10 experiments using algorithm 1 and 10 experiments using algorithm 2. It is up to you to show that the resulting distributions are indeed statistically significantly different. In other words, you need to show that the differences between the algorithm indeed lead to a systematic improvement, and that it was not purely luck that one set of experiments turned out “better” than another.

If you have good reasons to believe that your data is normal distributed, there exist a series of simple tests. For example, to test whether two sets of data are distributed with Gaussian distributions that have the same mean, can be done using Student’s t-test. A generalization of Student’s t-test to 3 or more groups is ANOVA. These tests have to be done with care as most distributions in robotics are not normal distributed. Examples where Gaussian distributions are commonly assumed are sensor noise on distance measurements such as obtained by infrared or odometry.

If data is not Gaussian distributed, there exist a series of numerical tests to test the likelihood that two distributions are independent. For example, you could test the message passing time with and without running some computationally expensive image processing routines. You can then test whether the additional computation affects message passing time. If it does, both distributions need to be significantly different. Just using Student’s t-test does not work as the distributions are not Gaussian!

Instead, testing whether two sets of data have the same mean, needs to be done numerically. A common test is MannWilcoxon’s Ranked Sum test. An implementation of this test is part of most mathematical calculation programs such as Matlab or Mathematica. An algorithm to calculate this test statistic and the corresponding p-values is available on the Wikipedia page above. An extension of the Mann-Wilcoxon’s Ranked Sum test for 3 or more groups is the Kruskal-Wallis one-way analysis of variance test.

15.5.3. Statistical Significance of True-False Tests

There exists a class of experiments that do not lead to distributions, but result in simple true-false outcomes. For example, a question one might ask is “does the robot correctly understand a spoken command”. This class of experiments is captured by the Lady tasting tea example. Here, a lady claims that she can identify the brewing method of a cup of tea: tea prepared by first adding milk and tea prepared by later adding milk. Unfortunately, it is easy to cheat as the likelihood of guessing right is 50%. Testing the hypothesis that the lady can indeed differentiate the two brewing methods therefore requires to conduct a series of experiments to reduce the likelihood of winning by guesswork. In order to do this, one needs to calculate the number of total permutations (or, possible outcomes over the entire series of experiments). For example, one could present the lady 8 cups of tea, 4 brewed one way and four the other. One can now enumerate all possible outcomes of this experiment, ranging from all cups guessed correctly to all cups guessed wrong. There are a total of 70 possible outcomes (see the example provided here). Guessing all cups correctly has now a likelihood of 1/70 or 1.4%. The likelihood to make a single mistake (16 possible outcomes in this example) is around 23%.

15.5.4. Summary

Statistical significance test allow you to express the likelihood that your experiment is not just the result of chance. There exist different tests for different underlying distributions. Therefore, your first task is to convincingly argue what the underlying distribution of your data is. Formally testing how your data is distributed can be achieved using the Chi-Square Test. In order to test whether two sets of data are coming from two different distributions can then be achieved using Student’s t-test (if the distribution is Gaussian) or using the Mann-Wilcoxon Ranked Sum test if the probability distribution is non-parametric.