4  Chapter 4 - Basic Data treatments

Before running any statistical or machine learning model, we need data that represents the phenomenon we want to model. We usually work with samples drawn from populations, measured on several variables. Raw data is almost never ready for modeling: variables arrive on wildly different scales, with missing values, with extreme observations, and with categories that a model cannot read. Data treatment is the process of fixing this — so that models train efficiently and are not biased by scale or encoding artifacts.

This chapter is the practical companion to Chapter 3. There we learned to diagnose a variable (is it skewed? does it have extreme values?); here we learn what to do about the diagnosis.

ImportantThe golden rule of data treatment

Every transformation in this chapter is estimated from data — a mean, a standard deviation, a median, a percentile. When you plan to validate a model on held-out data, those quantities must be computed on the training set only and then applied to the test set. Computing them on the full dataset lets information from the test set leak into training, and the model will look better than it is. This is the single most common methodological error in applied machine learning.

The most common data treatments and transformations for numerical variables are:

The most common data treatments for categorical variables are:

Of these transformations, I now explain in more detail the ones most used in statistical modeling.

4.1 Normalization and Scaling

Scaling matters because many methods are not scale-invariant. Distance-based methods (k-nearest neighbours, k-means, PCA) will be dominated by whichever variable happens to be measured in the largest units; regularized regressions penalize coefficients on a common scale, so a variable measured in pesos and one measured in millions of pesos are penalized very differently. Ordinary least squares regression, by contrast, is scale-invariant in its fit — rescaling a predictor just rescales its coefficient — so there scaling is about interpretation rather than correctness.

4.1.1 Standardization as a normalization method

For a variable x, we normalize by computing its z-score: the number of standard deviations each value lies away from the mean.

z_i = \frac{x_i - \bar{x}}{\sigma_{x}}

Where \bar{x} is the arithmetic mean of x and \sigma_{x} its standard deviation.

The resulting variable z always has a mean of exactly zero and a standard deviation of exactly one. Note that standardizing changes the units of the variable but not the shape of its distribution: a skewed variable is still skewed after standardization. This is a point students often get wrong — standardization is not a cure for non-normality.

Standardized variables are also directly comparable: a z of 2.5 means “unusually high relative to its own distribution” whether the variable is a stock return or a salary.

4.1.2 Min-Max scaling

x'= \frac{x-x_{min}}{x_{max}-x_{min}}

x' is a rescaled version of x whose values lie between 0 and 1 inclusive.

Min-max scaling is useful when a bounded range is required (for example, as input to a neural network activation function). Its weakness is obvious once you look at the formula: both the minimum and the maximum are the most extreme observations in the sample, so a single outlier compresses everything else into a tiny fraction of the [0, 1] interval. Use it only on variables with a genuine, known range.

4.1.3 Robust scaling

Robust scaling is analogous to standardization, but it replaces the mean with the median as the measure of center and the standard deviation with the interquartile range as the measure of dispersion:

IQR = Q_{75} - Q_{25}

where Q_{75} is the 75th percentile and Q_{25} the 25th percentile. The robust scaled variable is:

x' = \frac{x-\text{median}(x)}{IQR}

Why use percentiles?

  • The median and the IQR are robust statistics: they are far less sensitive to extreme values than the mean and the standard deviation, for exactly the reasons developed in Chapter 3.

  • Outliers therefore do not distort the scaling. Under standardization, a single extreme value inflates \sigma and squashes all the ordinary observations toward zero; under robust scaling it does not.

  • The transformed data has a median of zero and an IQR of one.

Robust scaling is the natural default for the skewed business variables — assets, salaries, sales, market capitalization — that we met in Chapter 3.

4.2 Imputation

4.2.1 Imputation for missing values

Depending on the context, when variables have missing values we can fill in a numeric value to avoid losing observations while maintaining the main patterns of the data. The most common imputation values for numeric variables are the mean, the median, or an interpolation from a regression on other variables.

Before imputing, however, ask why the value is missing. The answer determines whether imputation is safe:

Missingness mechanism Meaning Is simple imputation safe?
Missing completely at random (MCAR) Missingness is unrelated to anything Yes
Missing at random (MAR) Missingness depends on observed variables Yes, if the model conditions on those variables
Missing not at random (MNAR) Missingness depends on the unobserved value itself No — imputation will bias the results

The MNAR case is common in business data and is worth an example. If firms with poor results are more likely to omit a disclosure, imputing the mean for those firms systematically overstates the performance of exactly the firms that are doing worst. In such cases the fact that a value is missing is itself information, and a good practice is to add a binary indicator column flagging which observations were imputed.

Also note that imputing the mean for many observations artificially shrinks the variance of the variable and weakens its correlation with everything else — so heavy imputation biases regression coefficients toward zero.

4.2.2 Imputation for outliers and extreme values

4.2.2.1 Winsorization

Winsorization flattens extreme observations of a variable. It can be applied to high values, low values, or both. For high values we choose a percentile above which we consider values to be outliers, and every value above that percentile is replaced with the value at that percentile. For low values we do the same at the bottom of the distribution. A typical choice is to winsorize at the 1st and 99th percentiles.

Winsorization is common for the independent variables of multiple regression models, where a handful of extreme values can otherwise dominate the estimated coefficients.

Note the difference between winsorizing and deleting: winsorization keeps the observation and keeps its rank, it only limits how far the value can pull the estimates. Deleting outliers throws away information and, if the extreme values are real, biases the sample.

4.2.2.2 Clipping

Clipping is similar to winsorization, but the limits are fixed values chosen in advance rather than percentiles of the sample. Clipping is the right choice when the bounds come from domain knowledge — a market share cannot exceed 100%, an age cannot be negative — because those limits are true regardless of what this particular sample happens to contain.

4.3 Mathematical transformations:

4.3.1 Logarithmic transformation

Applying the natural log to a numeric variable reduces skewness and stabilizes variance. It is probably the single most useful mathematical transformation in applied statistics and, for financial and economic data, it is almost the default.

Why does it work? Because the logarithm compresses large values much more than small ones. The distance from 1 to 10 and the distance from 100 to 1,000 are both exactly \ln(10) \approx 2.30 in log units. In a right-skewed variable, that long upper tail — the handful of enormous firms, the few very rich countries — gets pulled in, while the crowded lower range gets spread out. The result is a distribution far closer to symmetric, which is exactly what most statistical methods assume.

There are three further reasons the log transformation earns its place:

  1. It turns multiplicative relationships into additive ones: \ln(a \times b) = \ln(a) + \ln(b). Since regression models are additive, this lets us model processes that are naturally multiplicative — and growth is multiplicative.

  2. It gives coefficients an elasticity interpretation. In a regression of \ln(y) on \ln(x), the slope is the percentage change in y per 1% change in x. In a regression of \ln(y) on x, the coefficient is approximately the percentage change in y per one-unit change in x.

  3. It stabilizes variance. Business variables typically vary proportionally to their level: a firm with $10 billion in sales varies by hundreds of millions, a corner shop by hundreds. Taking logs converts that proportional variation into roughly constant variation — which is precisely the homoskedasticity assumption of the regression models in Chapter 9.

Two cautions. The log is undefined for zero and negative values, so it cannot be applied directly to returns, profits, or any variable that can be non-positive. (A common workaround is \ln(1+x), which is well defined down to x > -1.) And the mean of the logs is not the log of the mean — back-transforming a prediction with e^{\hat{y}} gives you a median, not a mean.

What is a natural logarithm?

The natural logarithm of a number is the exponent that the number e (=2.71…) needs to be raised to get another number. For example, let’s name x=natural logarithm of a stock price p. Then:

e^x = p

The way to get the value of x that satisfies this equality is actually getting the natural log of p:

x = log_e(p)

Then, we have to remember that the natural logarithm is actually an exponent that you need to raise the number e to get a result or a specific number.

The natural log is the logarithm of base e (=2.71…). The number e is an irrational number (it cannot be expressed as a division of 2 natural numbers), and it is also called the Euler constant. Leonard Euler (1707-1783) took the idea of the logarithm from the great mathematician Jacob Bernoulli, and discovered very astonishing features of the e number. Euler is considered the most productive mathematician of all times. Some historians believe that Jacob Bernoulli discovered the number e around 1690 when he was playing with calculations to know how an amount of money grows over time with an interest rate.

How e is related to the grow of any amount over time? It is mainly related with the concept of compounding.

Next I give an example of the effect of compounding when calculating percentage growth rates

4.3.1.1 The effect of compounding in calculating percentage growth rates

Here is a simple example:

If I invest $100.00 today (t=0) with an annual interest rate of 50%, then the end balance of my investment at the end of the first year will be:

I_1=100*(1+0.50)=150

If the interest rate is 100%, then I would get:

I_1=100*(1+1)=200

Then, the general formula to get the final amount of my investment at the beginning of year 2, for any interest rate R can be:

I_1=I_0*(1+R)

The (1+R) is the growth factor of my investment.

In Finance, the investment amount is called principal. If the interests are calculated (compounded) each month instead of each year, then I would end up with a higher amount at the end of the year.

Monthly compounding means that a monthly interest rate is applied to the amount to get the interest of the month, and then the interest of the month is added to the investment (principal). Then, at the beginning of month 2 the principal will be higher than the initial investment. At the end of month 2 the interest will be calculated using the updated principal amount. Putting in simple math terms, the final balance of an investment at the end of month 1 when doing monthly compounding will be:

I_1=I_0*\left(1+\frac{R}{12}\right)

We can do the same for month 2:

I_2=I_1*\left(1+\frac{R}{12}\right)^{1}

We can plug the calculation for I_1 in this formula to express I_2 in terms of the initial investment:

I_2=I_0*\left(1+\frac{R}{12}\right)\left(1+\frac{R}{12}\right)

We group the growth factor using an exponent:

I_2=I_0*\left(1+\frac{R}{12}\right)^{2}

We can now see the pattern for the end balance after 12 months of monthly compounding. The monthly interest rate is the annual rate R divided by 12. With an annual rate of 100% and monthly compounding (N=12), the end value of the investment is:

I_{12}=100*\left(1+\frac{1}{12}\right)^{1*12}=100*(2.613..)

In this case, the growth factor is (1+1/12)^{12}, which is equal to 2.613.

Instead of compounding each month, if the compounding is every moment, then we are calculating a continuously compounded rate.

If we do a continuously compounding for the previous example, then the growth factor for one year becomes the astonishing Euler constant e:

Let’s do an example for a compounding of each second (1 year has 31,536,000 seconds). The investment at the end of the year 1 (or month 12) will be:

I_{12}=100*\left(1+\frac{1}{31536000}\right)^{1*31536000}=100*(2.718282..)\cong100*e^1

Now we see that e^1 is the GROWTH FACTOR after 1 year if we do the compounding of the interests every moment!

We can generalize to any other annual interest rate R, so that e^R is the growth factor for an annual nominal rate R when the interest is compounded every moment.

When compounding every instant, we use small r instead of R for the interest rate. Then, the growth factor will be: e^r

Then we can do a relationship between this growth rate and an effective equivalent rate:

\left(1+EffectiveRate\right)=e^{r}

If we apply the natural logarithm to both sides of the equation:

ln\left(1+EffectiveRate\right)=ln\left(e^r\right)

Since the natural logarithm function is the inverse of the exponential function, then:

ln\left(1+EffectiveRate\right)=r

In the previous example with a nominal rate of 100%, when doing a continuously compounding, then the effective rate will be:

\left(1+EffectiveRate\right)=e^{r}=2.7182

EffectiveRate=e^{r}-1

Doing the calculation of the effective rate for this example:

EffectiveRate=e^{1}-1 = 2.7182.. - 1 = 1.7182 = 171.82\%

Then, when compounding every moment, starting with a nominal rate of 100% annual interest rate, the actual effective annual rate would be 171.82%!

4.3.2 First difference of the log - continuously compounded growth rate

One way to calculate cc growth rates is by subtracting the log of the current value of the variable (at t) minus the log of the previous value (at t-1):

Let’s assume we want to know the growth rate of a stock price of a company. In this case, the growth rate of the price is called return.

r_{t}=log(price_{t})-log(price_{t-1})

This is also called as the difference of the log of the value (price in this example).

We can also calculate cc returns as the log of the current adjusted price (at t) divided by the previous adjusted price (at t-1):

r_{t}=log\left(\frac{price_{t}}{price_{t-1}}\right)

Continuously compounded returns are conventionally written with a lowercase r, while simple returns use an uppercase R. This notation is used throughout the rest of the book.

Why do we prefer cc returns for statistical work? Three reasons:

  1. They are additive over time. The cc return over two periods is the sum of the two one-period cc returns, because \ln(P_2/P_0) = \ln(P_2/P_1) + \ln(P_1/P_0). Simple returns must be compounded multiplicatively. Additivity is what allows us to model returns with linear models.

  2. They are symmetric. A price that goes from 100 to 50 and back to 100 gives simple returns of −50% and +100%, whose average is a misleading +25%. The cc returns are −0.693 and +0.693, which correctly average to zero.

  3. They are closer to normally distributed, which matters for the inference procedures of Chapters 6 and 7.

For small changes the two measures are almost identical — r \approx R when R is near zero — so for daily data the distinction rarely changes a conclusion. For monthly or annual data it can matter a great deal.

4.4 Illustrating the transformations

Let’s apply everything in this chapter to a single skewed variable so the effects are visible side by side. We simulate firm sales from a lognormal distribution, which produces exactly the right-skewed shape we saw in Chapter 3, and then add a few missing values and one data-entry error.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

rng = np.random.default_rng(2026)

# Simulated annual sales of 500 firms (millions of pesos), right-skewed:
sales = pd.Series(rng.lognormal(mean=4.0, sigma=1.2, size=500), name="sales")

# Inject 20 missing values and one data-entry error (an extra factor of 100):
sales.iloc[rng.choice(500, 20, replace=False)] = np.nan
sales.iloc[7] = sales.max() * 100

def describe(x, label):
    x = pd.Series(x).dropna()
    q1, q3 = x.quantile(.25), x.quantile(.75)
    return pd.Series({"n": len(x), "mean": x.mean(), "median": x.median(),
                      "SD": x.std(ddof=1), "IQR": q3 - q1,
                      "skewness": x.skew()}, name=label)

pd.DataFrame([describe(sales, "raw sales")]).round(2)
n mean median SD IQR skewness
raw sales 480.0 627.18 59.0 10927.81 109.2 21.89

The skewness coefficient is enormous and the mean sits far above the median — the Chapter 3 diagnosis of a badly skewed variable with at least one extreme observation. Now we apply each treatment in turn:

s = sales.copy()

# 1. Impute missing values with the MEDIAN (robust to the skew) and flag them:
was_missing = s.isna().astype(int)
s = s.fillna(s.median())

# 2. Winsorize at the 1st and 99th percentiles:
lo, hi = s.quantile(0.01), s.quantile(0.99)
s_wins = s.clip(lower=lo, upper=hi)

# 3. The three scalings, applied to the winsorized variable:
z_score  = (s_wins - s_wins.mean()) / s_wins.std(ddof=1)
min_max  = (s_wins - s_wins.min()) / (s_wins.max() - s_wins.min())
robust   = (s_wins - s_wins.median()) / (s_wins.quantile(.75) - s_wins.quantile(.25))

# 4. The logarithmic transformation, applied to the ORIGINAL (unwinsorized) values:
log_s    = np.log(s)

pd.DataFrame([
    describe(s,        "after imputation"),
    describe(s_wins,   "after winsorizing"),
    describe(z_score,  "standardized (z)"),
    describe(min_max,  "min-max scaled"),
    describe(robust,   "robust scaled"),
    describe(log_s,    "log transformed"),
]).round(3)
n mean median SD IQR skewness
after imputation 500.0 604.450 58.995 10707.156 104.898 22.345
after winsorizing 500.0 122.826 58.995 200.604 104.898 3.948
standardized (z) 500.0 0.000 -0.318 1.000 0.523 3.948
min-max scaled 500.0 0.089 0.042 0.150 0.079 3.948
robust scaled 500.0 0.609 0.000 1.912 1.000 3.948
log transformed 500.0 4.067 4.077 1.291 1.660 0.520

Read the skewness column carefully, because it makes the chapter’s central point in one number. Standardization and min-max scaling leave skewness completely unchanged — they only shift and rescale. Winsorizing reduces it substantially by capping the tail. Only the logarithm changes the shape of the distribution, bringing skewness close to zero. Scaling and reshaping are different operations, and choosing the wrong one is a common mistake.

Notice also that after standardization the SD is exactly 1, after min-max scaling the range is exactly [0, 1], and after robust scaling the IQR is exactly 1 — each transformation delivers precisely what its formula promises.

fig, ax = plt.subplots(1, 3, figsize=(11, 3.4))

ax[0].hist(sales.dropna(), bins=40, edgecolor="white")
ax[0].set_title("Raw sales (with the error)")

ax[1].hist(s_wins, bins=40, edgecolor="white")
ax[1].set_title("After winsorizing at 1% / 99%")

ax[2].hist(log_s, bins=40, edgecolor="white")
ax[2].set_title("After log transformation")

for a in ax:
    a.set_ylabel("Number of firms")
plt.tight_layout()
plt.show()
Figure 4.1: The same variable before and after transformation. The log is the only transformation that changes the shape of the distribution.

The third panel is the payoff. The log-transformed variable is close to symmetric and bell-shaped — which is no accident, since we generated the data from a lognormal distribution, defined as a variable whose logarithm is normal. Many real business variables behave this way for a good reason: they result from a long sequence of multiplicative growth shocks, and the Central Limit Theorem of Chapter 6, applied to the sum of the logs, produces exactly this shape.