7  Chapter 7 - Hypothesis Testing

The idea of hypothesis testing is to provide strong evidence - using facts - about a specific belief that is usually supported by a theory or common sense. This belief is usually the belief of the person conducting the hypothesis testing. This belief is called the Alternative Hypothesis.

The person who wants to show evidence about his/her belief is supposed to be very humble so the only way to be convincing is by using data and a rigorous statistical method.

Let’s imagine 2 individuals, Juanito and Diablito. Juanito wants to convince Diablito about a belief. Diablito is very, very skeptical and intolerant. Diablito is also an expert in Mathematics and Statistics! Then, Juanito needs a very robust statistical method to convince and persuade Diablito. Juanito also needs be very humble so Diablito does not get angry.

Then, Juanito decides to start assuming that his belief is NOT TRUE, so Diablito will be receptive to continue listening Juanito. Juanito decides to collect real data about his belief and decide to define 2 hypotheses:

• H0: The Null Hypothesis. This hypothesis states the opposite of Juanito’s belief, but he starts accepting that this hypothesis is TRUE.

• Ha: The Alternative Hypothesis. This is what Juanito beliefs, but he starts accepting that this is NOT TRUE.

Diablito is an expert in Statistics, so he knows the Central Limit Theorem very well! However, Juanito is humble, but he also knows the CLT very well.

Then Juanito does the following to try to convince Diablito:

• Juanito collects a random sample related to his belief. Imagine that his belief is about the mean of a variable X; he believes that the mean of X is greater than zero.

• He calculates the mean and standard deviation of the sample.

Since he collected a random sample, then he and Diablito know that, thanks to the CLT:

1. The mean of this sample will behave VERY SIMILAR to a normal distribution,

2. The standard deviation of this sample mean is much less than the standard deviation of the individuals of the sample and this can be calculated by dividing the individual standard deviation by the square root of sample size.

Since the sample mean behaves like a normally distributed variable, the interval [Sample Mean − 2 Standard Errors, Sample Mean + 2 Standard Errors] will contain the true population mean about 95% of the time, if we were to repeat the sampling many times. This is called a 95% confidence interval.

Now here is Juanito’s argument. He starts by granting Diablito that the true mean is zero. If that were so, then a sample mean landing more than 2 standard errors away from zero would be a rare event — it would happen less than 5% of the time. So if Juanito’s actual data shows a sample mean that far out, he can turn to Diablito and say: “Either something unusual just happened, or your assumption that the mean is zero is wrong.”

That is the whole logic of hypothesis testing, and it is a proof by contradiction under uncertainty. Notice what it does not do: it never computes the probability that the belief is true. It computes how surprising the data would be if the belief were false.

Juanito, being smart, computes a single number that measures how far his sample mean sits from the hypothetical mean of zero, expressed in standard errors. This standardized distance is called z or t. If |t| \geq 2, the data would be surprising under Diablito’s assumption, and Juanito has strong evidence against it.

Although the CLT says the sample mean is approximately normal, in practice we use the t-Student distribution instead. Why? Because to standardize the sample mean we need to divide by its standard error — and we do not know the true population standard deviation \sigma, so we must estimate it from the same sample. That extra estimation adds uncertainty, and the t distribution accounts for it by having slightly fatter tails than the normal.

The t distribution is essentially indistinguishable from the normal once the sample size exceeds about 30, but for small samples it is noticeably wider — which correctly makes it harder to reject the null hypothesis when you have little data. The t-Student distribution was published in 1908 by William Sealy Gosset, a chemist at the Guinness brewery in Dublin, who worked with small samples of barley and hops. Guinness did not allow employees to publish, so he wrote under the pseudonym “Student”.

Then, for hypothesis testing we will use the t-Student distribution instead of the z normal distribution.

The hypothesis testing to compare the mean of a variable with a value or with another mean is called one-sample t-test.

We can use the same method of hypothesis testing for other types of random variables. For example, hypothesis testing is automatically performed to model coefficients when we run any linear regression model, which is very popular in machine learning.

For now, I will illustrate the method of hypothesis testing for the simple case of testing whether the mean of a variable is different to a specific value - the one-sample t-test.

7.1 One-sample t-test

Imagine that I believe that investing in the S&P 500 index is a good long-term investment. The S&P500 index is a portfolio composed of the largest 500 US public firms. If I believe this, at least I have to show statistical evidence that the mean of monthly return of the S&500 is greater than zero, so when I compound this monthly return to long periods, the long-term investment will be positive and attractive.

The steps for doing a hypothesis test are the following:

1. Identify the variable of study for the hypothesis test. In this case, it can be the monthly average return of the S&P500: \bar{X}.

2. State the null and the alternative hypothesis using the variable of study. The alternative hypothesis is my belief, and the null hypothesis is its opposite. I begin by assuming the null hypothesis is true, and the purpose of the test is to look for evidence strong enough to REJECT it.

We have to distinguish between the sample variable and the population variable. In this case, the sample variable is the sample mean that I will calculate after collecting historical monthly returns of the S&P500. The population mean will be the TRUE variable of the population, which in this case can be considered as the whole future of the monthly returns of the S&P500. Then:

\mu = Population Mean

\bar{X} = Sample Mean

In this example, the hypothesis can be stated as:

H0 (Null hypothesis): \mu=0 (Initially considered as 100% TRUE)

Ha (Alternative hypothesis): \mu>0 (Initially considered as FALSE)

3. Collect data that measures the variable of study. It is recommended collecting a representative sample from the population of the variable. In the case of cross-sectional variables, it is recommended to pick a representative random sample. For time-series variables - such as the historical monthly returns of the S&P500- we need to make some assumptions.

In this example, the variable of study is actually a time-series variable, so the individuals that can be collected are actually historical monthly returns. Here I need to do an assumption that my horizon for my investment is about 3 years, and I assume that the average return of the S&P500 will remain very similar than the average monthly return for the last 3-years. This is a very straight-forward assumption, that many people would not agree, but for the sake of this example, I will accept my assumption.

4. Calculate the t-Statistic of the test. The t-Statistic is a measure of how far away the sample variable - in this case, the sample mean - is away from the hypothetical value of the population variable - in this case, it is zero (the value of variable stated in the Null hypothesis). We can call the t-Statistic as a standardized distance of the sample variable from its hypothetical true value. This standardized distance is measured in # of Standard Deviations of the variable of study - in this case, the sample mean of the S%P monthly returns.

Then, I can define the t-Statistic as the following:

t\text{-Statistic}=\frac{\bar{X}-0}{SE(\bar{X})}

Where SE(\bar{X}) is the standard error of the sample mean — the standard deviation of the variable of study, which here is the average monthly return of the S&P 500.

The numerator is the raw distance of the sample statistic from the hypothesized value (zero in this case). Dividing that distance by the standard error expresses it in “number of standard errors”, which is a scale-free measure we can compare against a known probability distribution.

In statistics, the standard deviation of a sample statistic is called its standard error, to distinguish it from the standard deviation of the individual observations.

So I need to calculate two things: 1) the sample statistic — the sample mean of the historical monthly returns of the S&P 500, and 2) its standard error, which by the CLT is SD/\sqrt{N}.

5. Calculate the p-value of the test.

ImportantWhat the p-value actually is — and is not

The p-value is:

the probability of observing a sample statistic at least as extreme as the one we got, assuming the null hypothesis is true.

In symbols, p = P(\text{data this extreme} \mid H_0 \text{ true}).

It is not the probability that the null hypothesis is true, and it is not the probability that we are wrong if we reject it. Those would be P(H_0 \mid \text{data}) — the conditional probability the other way around, which the test does not compute and cannot compute without knowing how plausible H_0 was to begin with.

The distinction matters, and confusing the two is the single most common error in applied statistics. An analogy: the probability that a person is over 2 metres tall given that they play professional basketball is high; the probability that a person plays professional basketball given that they are over 2 metres tall is low. Same two facts, opposite conditioning, wildly different answers.

So a p-value of 0.03 means: “if the true mean really were zero, data this extreme would appear only 3% of the time.” That is a statement about the data under an assumption, not a statement about the assumption.

Similarly, “1 - p” is not the probability that your hypothesis is true. Reporting it as such overstates what the test has established. What you can properly say is that the result is statistically significant at the corresponding level.

How do we calculate the p-value?

6. Make the conclusion of the test. We fix a significance level \alpha before looking at the data — conventionally 0.05 — and then:

• If the p-value is less than or equal to \alpha = 0.05, we reject the null hypothesis and say the result is statistically significant at the 5% level.

• If the p-value is greater than 0.05, we fail to reject the null hypothesis. Note the careful phrasing: failing to reject is not the same as proving the null is true. Absence of evidence is not evidence of absence — a small sample can fail to detect a real effect.

In practice we must apply judgement. The 0.05 threshold is a convention inherited from Ronald Fisher, not a law of nature, and it is worth reporting the actual p-value rather than only whether it crossed the line.

Note that we can reach the same conclusion from the t-Statistic directly, since t-Statistics and p-values map onto one another given the degrees of freedom.

A general rule of thumb: if |t| \geq 2 then the two-tailed p-value is below 0.05, so we can reject the null hypothesis when the absolute value of the t-Statistic is 2 or more.

7.2 Example of a one-sample t-test

Imagine that you believe that investing in the US financial market is a good long-term investment. You have been learning about the financial performance of the largest US companies for the last 3 years, and also the leadership of US tech firms -such as Microsoft, Google, Meta, Nvidia- in the Artificial Intelligence innovations and applications.

Step 1 - Identify the variable of study

In the world of investment, there is a quick way to invest in the whole US market if you just buy a market index. In this case, you can buy the S&P500 index, which is a virtual portfolio composed of the largest 500 US companies. Then, you define that the variable of study will be the average monthly return of the S&P500 index

Step 2 - State the hypotheses

You decide to first show that the average monthly return of the S&P 500 is positive. Afterwards you plan a comparative analysis with alternative investments, but for now the single claim is enough. So you run a one-sample t-test to convince your father, who has said he will support you if you put your money into a good long-term investment.

Then, the stated hypotheses are:

In this example, the hypothesis can be stated as:

H0 (Null hypothesis): \mu=0 (Initially considered as 100% TRUE)

Ha (Alternative hypothesis): \mu>0 (Initially considered as FALSE)

Where:

\mu = Population Mean, which in this case is the overall monthly average return of the S&P500 index

\bar{X} = Sample Mean, which in this case is the actual monthly average return of the S&P500 index for the last 3 years

Step 3 - Collect data to measure the variable of study.

In this case, we can collect real data of past monthly quotations (like prices) for the S&P500 from Yahoo-Finance:

We download historical monthly data for the S&P500 market index. In this case, the variable of study is the average monthly return, so I need do a transformation from monthly quotations (prices) to monthly returns.

In most statistical analysis, when dealing with percentage changes, it is strongly recommended to use logarithmic percentage changes — what we called continuously compounded (cc) returns in Chapter 4 — because they are additive over time and closer to normally distributed.

Here is the code to collect the S&500 monthly quotations and transforming them to cc returns:

import yfinance as yf
import numpy as np
import pandas as pd
import matplotlib
import matplotlib.pyplot as plt

# Download stock data
data = yf.download('^GSPC', start='2020-01-01', end='2025-07-31', interval='1mo', auto_adjust=True)
adjprices = data['Close']  # Get adjusted close prices

# Calculate continuously compounded returns as the difference of the log prices:

ccr = np.log(adjprices) - np.log(adjprices.shift(1))
ccr = ccr.dropna()

Step 4 - Calculate the t-Statistic of the test.

# H0: mean(ccr$GSPC) = 0
# Ha: mean(ccr$GSPC) <> 0

# Standard error
se_GSPC = np.std(ccr,ddof=1) / np.sqrt(len(ccr))
print(f"Standard error S&P 500 = {se_GSPC}")

# t-value
mean_ccr = np.mean(ccr)
t_GSPC = (mean_ccr - 0) / se_GSPC
print(f"t-value S&P 500 = {t_GSPC}")

Standard error S&P 500 = 0.006349935414602851
t-value S&P 500 = 1.6122530653992084

The t-Statistic already allows a preliminary conclusion before we compute the p-value. Apply the rule of thumb: if |t| \geq 2 we have evidence at roughly the 5% level; if |t| < 2 we do not. Note that this test is one-sided (we only care about the mean being greater than zero), so the relevant critical value is about 1.65 rather than 2 — we will see why in the final section of this chapter.

Remember also that the t-Statistic is scale-free: it does not tell us whether the mean return is large, only whether it is distinguishable from zero given the noise in the data. Those are different questions, and we return to the distinction at the end of the chapter.

Step 5 - Calculate the p-value of the test

We can calculate the p-value of the test.

Recalling the definition:

The p-value is the probability of obtaining a result at least as extreme as the observed one, assuming the null hypothesis is true.

The smaller the p-value, the more the data conflicts with the null hypothesis. The threshold below which we call the result significant is the significance level \alpha, which we choose before running the test.

To compute a p-value we need two things:

• the t-Statistic, and

• the degrees of freedom, which here equal the number of observations minus 1.

There is a direct relationship between the t-Statistic and the p-value, once the degrees of freedom are fixed.

There is no simple closed-form formula for the p-value, because it is the area under the tail of the t-Student probability density function — that is, a definite integral of a function with no elementary antiderivative. It must be evaluated numerically.

Fortunately every statistical package computes it instantly. Here is how in Python:

from scipy import stats as st

# One-sided t-test
ttest_GSPC = st.ttest_1samp(ccr, 0, alternative='greater')

# Showing the t-Statistics and the p-value:
ttest_GSPC

TtestResult(statistic=array([1.61225307]), pvalue=array([0.05587612]), df=array([65]))

Actually this function automatically calculates the t-Statistic and the p-value.

See that I got the same value for the t-Statistic than the value I calculated above.

The p-value that is calculated here is the 1-tailed p-value since I specified the parameter alternative = ‘greater’.

Actually there are 2 types of p-values:

• 1-tailed p-value

• 2-tailed p-value

What are these p-values? In this example, since my belief is that the monthly return will be positive (1-tailed), then I can use the 1-tailed p-value. The 2-tailed p-value is always equal to the double of the 1-tailed p-value, so it is a more conservative p-value. When we use the 2-tailed p-value? we can use the 2-tailed p-value when our belief is not quite clear, but we believe that the value of the variable is DIFFERENT to a specific value. We can also use 2-tailed p-value to be more conservative with our analysis. In this case, I will keep using the 1-tailed p-value.

Step 6 - Make a conclusion for the test

Let’s extract the p-value and state the conclusion explicitly:

t_stat  = float(np.ravel(ttest_GSPC.statistic)[0])
p_value = float(np.ravel(ttest_GSPC.pvalue)[0])

print(f"t-Statistic      = {t_stat:.4f}")
print(f"one-tailed p-value = {p_value:.4f}")
print(f"mean monthly cc return = {100*float(np.ravel(mean_ccr)[0]):.3f}%")

if p_value <= 0.05:
    print("\nConclusion: we REJECT H0 at the 5% significance level.")
elif p_value <= 0.10:
    print("\nConclusion: the result is MARGINALLY significant (5% < p <= 10%).")
else:
    print("\nConclusion: we FAIL TO REJECT H0 at the 10% significance level.")

t-Statistic      = 1.6123
one-tailed p-value = 0.0559
mean monthly cc return = 1.024%

Conclusion: the result is MARGINALLY significant (5% < p <= 10%).

Now, how should we read this? Suppose the p-value comes out a little above 0.05. Strictly, we do not have evidence at the conventional 5% level to reject the null. Does that mean investing in the S&P 500 will not give positive returns over time? No — and this is an important lesson.

Important“Not significant” does not mean “no effect”

Three things to keep separate:

1. Failing to reject H_0 is not accepting H_0. With about 66 monthly observations and the enormous month-to-month volatility of equity returns, the test simply may not have the statistical power to detect an average monthly return that is genuinely positive but small. A larger sample would narrow the standard error, and the same mean return could then become significant.

2. The 0.05 threshold is a convention. A p-value of 0.049 and one of 0.051 constitute almost identical evidence. Treating them as categorically different is an artifact of the convention, not of the data. Report the actual p-value and let the reader judge.

3. Statistical significance is not economic significance. A mean monthly return of 0.9% is highly meaningful economically — compounded over 20 years it more than triples the investment — regardless of whether this particular sample size can distinguish it from zero at a chosen threshold. Conversely, with a big enough sample, a completely trivial effect can become “highly significant”. Always report the size of the effect alongside its p-value.

In statistics, when p \leq 0.05 we say the result is statistically significant; when 0.05 < p \leq 0.10 we say it is marginally significant.

A useful complement to the p-value is the confidence interval, which reports the effect size and its uncertainty together:

from scipy import stats as st

n  = len(ccr)
se = float(np.ravel(se_GSPC)[0])
m  = float(np.ravel(mean_ccr)[0])
tcrit = st.t.ppf(0.975, df=n-1)

print(f"Sample size: {n} months")
print(f"95% CI for the mean monthly cc return: "
      f"[{100*(m - tcrit*se):.3f}%, {100*(m + tcrit*se):.3f}%]")

Sample size: 66 months
95% CI for the mean monthly cc return: [-0.244%, 2.292%]

The confidence interval says much more than “significant or not”: it shows both the plausible magnitude of the mean return and how precisely the data pins it down. Many statisticians argue confidence intervals should be reported in preference to p-values for exactly this reason.

Next I illustrate another type of hypothesis test — the two-sample t-test — with a practical example.

7.3 Hypothesis testing - two-sample t-test

Now we will do a hypothesis testing to compare the means of two groups. This test is usually named two-sample t-test.

In the case of the two-sample t-test we try to check whether the mean of a group is greater than the mean of another group.

Imagine we have two random variables X and Y and we take a random sample of each variable to check whether the mean of X is greater than the mean of Y.

We start writing the null and alternative hypothesis as follows:

H0:\mu_{x}=\mu_{y}

Ha:\mu_{x}\neq\mu_{y}

We do simple algebra to leave a number in the right-hand side of the equality, and a random variable in the left-hand side of the equation. Then, we re-write these hypotheses as:

H0:(\mu_{x}-\mu_{y})=0

Ha:(\mu_{x}-\mu_{y})\neq0

The Greek letter \mu is used to represent the population mean of a variable.

To test this hypothesis we take a random sample of X and Y and calculate their means.

Then, in this case, the variable of study is the difference of 2 means! Then, we can name the variable of study as diff:

diff = (\mu_{x}-\mu_{y})

Since we use sample means instead of population means, we can re-define this difference as:

diff = (\bar{X}-\bar{Y})

The steps for all hypothesis tests are basically the same. What changes is the calculation of the standard deviation of the variable of study, which is usually names standard error.

For the case of one-sample t-test, the standard error was calculated as \frac{SD}{\sqrt{N}}, where SD is the individual sample standard deviation of the variable, and N is the sample size.

In the case of two-sample t-test, the standard error SE can be calculated using different formulas depending on the assumptions of the test. In this workshop, we will assume that the population variances of both groups are NOT EQUAL, and the sample size of both groups is the same (N). For these assumptions, the formula is the following:

SD(diff)=SE=\sqrt{\frac{Var(X)+Var(Y)}{N}}

But, where does this formula come from?

We can easily derive this formula by applying basic probability rules to the variance of a difference of 2 means. Let’s do so.

The variances of each group of X and Y might be different, so we can estimate the variance of the DIFFERENCE as:

Var(\bar{X}-\bar{Y})=Var(\bar{X})+Var(\bar{Y})

This is true if only if \bar{X} and \bar{Y} are independent. We will assume that both random variables are not dependent of each other. This might not apply for certain real-world problems, but we will assume that for simplicity. If there is dependence I need to add another term that is equal to 2 times the covariance between both variables.

Why is the variance of a difference the sum of the variances? This sounds counter-intuitive but it is correct. The intuition: variance measures uncertainty, and subtracting one uncertain quantity from another does not cancel uncertainty — it compounds it. If both \bar{X} and \bar{Y} can wander, then their difference can wander even further, because \bar{X} might be high at the same moment \bar{Y} happens to be low. Formally, the minus sign gets squared: VAR(-\bar{Y}) = (-1)^2 VAR(\bar{Y}) = VAR(\bar{Y}).

As we learned in the CLT, the variance of the mean of a random variable is reduced according to its sample size: Var(\bar{X)}=\frac{Var(X)}{N}. Then:

Var(\bar{X}-\bar{Y})=\frac{Var(X)}{N}+\frac{Var(Y)}{N}

Factorizing:

Var(\bar{X}-\bar{Y})=\frac{1}{N}\left[Var(X)+Var(Y)\right]

Taking the square root gives the standard error of the difference:

SE(\bar{X}-\bar{Y})=\sqrt{\frac{1}{N}\left[Var(X)+Var(Y)\right]}

The rest of the procedure is identical to the one-sample t-test — we simply use this formula as the denominator of the t-Statistic. This is the general pattern worth remembering: every t-test has the same structure,

t = \frac{\text{estimate} - \text{hypothesized value}}{\text{standard error of the estimate}}

and what changes from one test to another is only how the standard error is computed. The same structure reappears in Chapter 10 for regression coefficients.

The t-Statistic for the two-sample t-test is therefore:

t=\frac{(\bar{X}-\bar{Y})-0}{\sqrt{\frac{Var(X)+Var(Y)}{N}}}

Remember that the value of t is the # of standard deviations of the variable of study (in this case, the difference of the 2 means) that the empirical difference we got from the data is away from the hypothetical value, zero.

The rule of thumb we have used is that if |t|>2 we have have statistical evidence at least at the 95% confidence level to reject the null hypothesis (or to support our alternative hypothesis).

7.4 Example — is AMD’s mean return higher than Oracle’s?

Let’s run a t-test to check whether the mean monthly cc return of AMD is significantly different from the mean monthly return of Oracle (ORCL), using data from January 2020 onward.

# Getting price data. I indicate getting adjusting prices as close prices:
sprices=yf.download(tickers='AMD ORCL', start='2020-01-01', end='2025-01-31',interval='1mo', auto_adjust=True)
# I select the Close columns for both stocks:
sprices=sprices['Close']

[                       0%                       ]
[*********************100%***********************]  2 of 2 completed

The sprices data frame has two columns: the adjusted close price for AMD and for ORCL:

sprices.head(5)

Ticker	AMD	ORCL
Date
2020-01-01	47.000000	47.829319
2020-02-01	45.480000	45.102734
2020-03-01	45.480000	44.072285
2020-04-01	52.389999	48.532993
2020-05-01	53.799999	49.265987

Now we calculate monthly continuously compounded returns for both stocks:

# Calculating cc returns as the difference of the log price and the log price of the previous month:
sr = np.log(sprices) - np.log(sprices.shift(1))
# we can also calculate cc returns using the diff function:
# sr = np.log(sprices).diff(1)

# Deleting the first month with NAs:
sr=sr.dropna()

sr.head()

Ticker	AMD	ORCL
Date
2020-02-01	-0.032875	-0.058696
2020-03-01	0.000000	-0.023112
2020-04-01	0.141443	0.096413
2020-05-01	0.026558	0.014990
2020-06-01	-0.022367	0.027514

We can calculate the monthly mean return for both stocks:

amd_mean = sr['AMD'].mean()
orcl_mean = sr['ORCL'].mean()
print(f"AMD mean cc % return= {100*amd_mean}%")
print(f"Oracle mean cc % return= {100*orcl_mean}%")

AMD mean cc % return= 1.5050190594531336%
Oracle mean cc % return= 2.08578469990237%

The two sample means differ. But a difference between two sample means is guaranteed — two random samples will essentially never produce identical averages, even when the underlying populations are identical. The question is whether this particular difference is larger than what sampling noise alone would routinely produce. That is what the two-sample t-test answers.

We state the hypothesis and calculate the t-Statistic:

# Stating the hypotheses: 
# H0: (mean(rAMD) - mean(rORACLE)) = 0
# Ha: (mean(rAMD) - mean(rORACLE)) <> 0

# Calculating the standard error of the difference of the means:
# Getting the number of non-missing observations for the sample:
N = sr['AMD'].count()
# Getting the variances of both columns: 
amd_var = sr['AMD'].var()
orcl_var = sr['ORCL'].var()
# Now we get the standard error for the mean difference:
sediff = np.sqrt((1/N) * (amd_var + orcl_var ) )

# Calculating the t-Statistic:
t = (sr['AMD'].mean() - sr['ORCL'].mean()) / sediff
print(f"t-Statistic = {t}")

t-Statistic = -0.26621334869741947

Fortunately we can use a Python function to easily calculate the t-value along with its p-value and 95% confidence interval:

# I do the 2-way sample t-test using the ttest_ind function from stats:
st.ttest_ind(sr['AMD'],sr['ORCL'],equal_var=False)
# With this function we avoid calculating all steps of the hypothesis test!

TtestResult(statistic=np.float64(-0.2662133486974195), pvalue=np.float64(0.7906626265663412), df=np.float64(93.25293187086828))

We get the same t-Statistic we computed by hand, plus the p-value. Let’s state the conclusion explicitly rather than by inspection, so that it stays correct as the data is updated:

res = st.ttest_ind(sr['AMD'], sr['ORCL'], equal_var=False)
print(f"Difference in mean monthly cc return = "
      f"{100*(sr['AMD'].mean() - sr['ORCL'].mean()):.3f} percentage points")
print(f"t-Statistic = {res.statistic:.4f}")
print(f"two-tailed p-value = {res.pvalue:.4f}")

if res.pvalue <= 0.05:
    print("\nWe REJECT H0: the two mean returns differ significantly at the 5% level.")
else:
    print("\nWe FAIL TO REJECT H0: the observed difference in mean returns is "
          "not distinguishable from sampling noise at the 5% level.")

Difference in mean monthly cc return = -0.581 percentage points
t-Statistic = -0.2662
two-tailed p-value = 0.7907

We FAIL TO REJECT H0: the observed difference in mean returns is not distinguishable from sampling noise at the 5% level.

Interpret the output by applying the rule of thumb consistently: a |t| above roughly 2 corresponds to a two-tailed p-value below 0.05 and lets us reject the null; a |t| below 2 does not.

If the test fails to reject, the correct statement is: “although one stock’s average return is higher in this sample, we do not have sufficient evidence to conclude that its true mean return is higher.” Monthly stock returns are extremely volatile relative to their means, so it takes a very large difference — or a very long sample — before a difference in average returns becomes statistically detectable. This is not a shortcoming of the test; it is an accurate reflection of how noisy financial data is, and it is precisely why claims about one stock “outperforming” another over short windows should be treated sceptically.

ttest_ind gives us the essentials, but for reporting we usually want a fuller picture: the descriptive statistics of each group, the difference, the degrees of freedom, both one-sided p-values, and a measure of effect size. Let’s write a small function that assembles all of it. It uses only numpy, pandas and scipy, so there is nothing extra to install.

import numpy as np
import pandas as pd
from scipy import stats as st


def ttest_table(x, y, equal_variances=False, alpha=0.05):
    """Two-sample t-test with descriptive statistics and effect sizes.

    equal_variances=False -> Welch / Satterthwaite test (the safe default)
    equal_variances=True  -> classical pooled-variance (Student) test
    """
    x, y = pd.Series(x).dropna(), pd.Series(y).dropna()
    n1, n2 = len(x), len(y)
    m1, m2 = x.mean(), y.mean()
    s1, s2 = x.std(ddof=1), y.std(ddof=1)

    # --- descriptive table, one row per group plus the combined sample ---
    def row(name, v):
        n, m, s = len(v), v.mean(), v.std(ddof=1)
        se = s / np.sqrt(n)
        tc = st.t.ppf(1 - alpha/2, n - 1)
        return [name, n, m, s, se, m - tc*se, m + tc*se]

    lo = f"{int((1-alpha)*100)}% Conf."
    desc = pd.DataFrame(
        [row(x.name or "group1", x),
         row(y.name or "group2", y),
         row("combined", pd.concat([x, y], ignore_index=True))],
        columns=["Variable", "N", "Mean", "SD", "SE", lo, "Interval"])

    # --- the test itself ---
    diff = m1 - m2
    # pooled SD is used for the effect sizes in both cases:
    sp = np.sqrt(((n1-1)*s1**2 + (n2-1)*s2**2) / (n1 + n2 - 2))

    if equal_variances:                      # Student: pooled variance
        df = n1 + n2 - 2
        se = sp * np.sqrt(1/n1 + 1/n2)
        title = "Independent t-test"
    else:                                    # Welch: Satterthwaite d.f.
        se = np.sqrt(s1**2/n1 + s2**2/n2)
        df = (s1**2/n1 + s2**2/n2)**2 / ((s1**2/n1)**2/(n1-1) +
                                         (s2**2/n2)**2/(n2-1))
        title = "Satterthwaite t-test"

    t = diff / se
    d = diff / sp                            # Cohen's d

    results = pd.DataFrame({
        title: [f"Difference ({x.name} - {y.name}) = ",
                "Degrees of freedom = ", "t = ",
                "Two side test p value = ",
                "Difference < 0 p value = ",
                "Difference > 0 p value = ",
                "Cohen's d = ", "Hedge's g = ",
                "Glass's delta1 = ", "Point-Biserial r = "],
        "results": [diff, df, t,
                    2 * st.t.sf(abs(t), df),
                    st.t.cdf(t, df),
                    st.t.sf(t, df),
                    d,
                    d * (1 - 3/(4*(n1 + n2) - 9)),   # Hedge's g
                    diff / s1,                        # Glass's delta
                    t / np.sqrt(t**2 + df)]           # point-biserial r
    })
    results["results"] = results["results"].round(4)
    return desc, results

Now we can apply it:

desc, results = ttest_table(sr['AMD'], sr['ORCL'], equal_variances=False)
desc.round(4)

	Variable	N	Mean	SD	SE	95% Conf.	Interval
0	AMD	60	0.0151	0.1471	0.0190	-0.0229	0.0530
1	ORCL	60	0.0209	0.0832	0.0107	-0.0006	0.0424
2	combined	120	0.0180	0.1190	0.0109	-0.0036	0.0395

results

	Satterthwaite t-test	results
0	Difference (AMD - ORCL) =	-0.0058
1	Degrees of freedom =	93.2529
2	t =	-0.2662
3	Two side test p value =	0.7907
4	Difference < 0 p value =	0.3953
5	Difference > 0 p value =	0.6047
6	Cohen's d =	-0.0486
7	Hedge's g =	-0.0483
8	Glass's delta1 =	-0.0395
9	Point-Biserial r =	-0.0276

The t-statistic and p-value match what ttest_ind reported — but now we can also see the mean, standard deviation and confidence interval of each stock separately, the size of the difference, and four measures of effect size.

NoteEffect size: how big is the difference?

The p-value tells us whether a difference is distinguishable from noise. It says nothing about whether the difference is large. Effect sizes fill that gap, and reporting one alongside every p-value is good practice.

Cohen’s d is the most widely used. It expresses the difference between the two means in units of the pooled standard deviation:

d = \frac{\bar{X} - \bar{Y}}{s_{pooled}}

So d = 0.5 means the two group means are half a standard deviation apart. Conventional (and loose) benchmarks are 0.2 for a small effect, 0.5 for medium and 0.8 for large.

Notice the structural similarity to the t-statistic: both divide the same difference by a measure of spread. But the t-statistic divides by the standard error, which shrinks as the sample grows, while Cohen’s d divides by the standard deviation, which does not. That is exactly why a big enough sample makes any difference “significant” while leaving its effect size unchanged — and why the two numbers answer genuinely different questions.

Hedge’s g is Cohen’s d with a small-sample correction. Glass’s delta uses only the first group’s standard deviation, which is preferable when one group is a control whose variance you trust more. Point-biserial r re-expresses the same result as a correlation between group membership and the outcome, so it can be squared to give a proportion of variance explained, as in Chapter 8.

7.5 Confidence level, Type I Error and p-value

The significance level \alpha of a test is the error rate we are willing to tolerate. If we adopt \alpha = 0.05 (a 95% confidence level) and the null hypothesis is in fact true, then we will wrongly reject it 5% of the time. This mistake — rejecting a true null hypothesis, a “false positive” — is called a Type I Error.

There is a second kind of mistake. Failing to reject a null hypothesis that is actually false — a “false negative” — is called a Type II Error, with probability \beta. The power of a test is 1-\beta: the probability of detecting an effect that really exists.

The two types of error in hypothesis testing

	H_0 is true	H_0 is false
We reject H_0	Type I Error (prob. \alpha)	Correct decision (power = 1-\beta)
We do not reject H_0	Correct decision	Type II Error (prob. \beta)

The two errors trade off against each other. Lowering \alpha from 0.05 to 0.01 makes false positives rarer, but makes it harder to detect real effects, so \beta rises. The only way to reduce both simultaneously is to increase the sample size.

NoteAn analogy that keeps the two straight

Think of a criminal trial, where H_0 is “the defendant is innocent”. A Type I error convicts an innocent person; a Type II error acquits a guilty one. Legal systems set \alpha deliberately low — “beyond reasonable doubt” — because they judge the first error more costly. In business analytics, the relative cost depends entirely on context: for a medical screening test, missing a real disease (Type II) is usually worse than a false alarm (Type I).

The p-value is the smallest significance level at which the observed result would lead us to reject the null. Equivalently, it is the probability of data at least this extreme given that H_0 is true. It is a property of the data, whereas \alpha is a threshold we choose in advance.

For each t-Statistic there is a corresponding p-value. The relationship is shown in the following figure of the t-Student PDF:

Illustrating t-Statistics vs pvalue

For a 95% confidence level and 2-tailed pvalue, the critical t value is close to 2 (is not exactly 2); it can change according to N, the # of observation of the sample.

When the sample size N>30, the t-Student distribution approximates the Z normal distribution. In the above figure we can see that when N>30 and t=2, the approximates pvalues are: 1-tailed pvalue = 2.5%, and 2-tailed pvalues=5%.

Then, what is 1-tailed and 2-tailed pvalues? The 2-tailed pvalue will always be twice the value of the 1-tailed pvalue since the t-Student distribution is symmetric.

We always want to have a very small pvalue in order to reject H0. Then, the 1-tailed pvalue seems to be the one to use. However, the 2-tailed pvalue is a more conservative value (the diablito will feel ok with this value). Most of the statistical software and computer languages report 2-tailed pvalue.

Then, which pvalue is the right to use? It depends on the context. When there is a theory that supports the alternative hypothesis, we can use the 1-tailed pvalue. For now, we can be conservative and use the 2-tailed pvalue for our t-tests.

The relationship between the significance level and the confidence level of a test is:

\alpha = 1 - \text{Confidence Level}

For a one-tailed test at the 95% confidence level, the critical t-value is smaller than 2 — approximately 1.65. This is because all 5% of the rejection region sits in a single tail instead of being split between two:

Illustrating one-tailed t-values and p-values

WarningChoose the tail before you see the data

Because a one-tailed test has a lower critical value, it is easier to obtain a significant result. That makes it tempting to look at the data first, notice which direction the effect went, and then declare a one-tailed test in that direction. This is not legitimate — it silently doubles the true Type I error rate. The direction of a one-tailed test must be justified by theory before the data is examined.

The same principle applies to a broader problem known as p-hacking: running many tests and reporting only the ones that came out significant. If you test 20 independent hypotheses at \alpha = 0.05 and none of them is true, you should expect one “significant” result purely by chance. This is why pre-registering hypotheses, reporting all the tests you ran, and replicating findings matter so much.

7.6 Summary

The logic of hypothesis testing, in five lines:

1. State a null hypothesis H_0 (the sceptic’s position) and an alternative H_a (your belief).
2. Choose a significance level \alpha before looking at the data.
3. Compute the test statistic: t = \dfrac{\text{estimate} - \text{hypothesized value}}{\text{standard error}}.
4. Compute the p-value: the probability of data this extreme if H_0 were true.
5. If p \leq \alpha, reject H_0; otherwise, fail to reject it.

And the three things most often gotten wrong:

• The p-value is not the probability that the hypothesis is true.
• Failing to reject H_0 is not proof that H_0 is true.
• Statistical significance is not the same as practical importance — always report the effect size, ideally as a confidence interval.

In the next chapter we move from comparing means to measuring the relationship between two variables, which is the bridge to regression modeling.