Encoding Categorical Variables: A Deep Dive into Goal Encoding | by Juan Jose Munoz | Feb, 2024
Information is available in completely different shapes and varieties. A type of shapes and varieties is called categorical knowledge.
This poses an issue as a result of most Machine Studying algorithms use solely numerical knowledge as enter. Nevertheless, categorical knowledge is normally not a problem to take care of, due to easy, well-defined capabilities that remodel them into numerical values. You probably have taken any knowledge science course, you can be conversant in the one scorching encoding technique for categorical options. This technique is nice when your options have restricted classes. Nevertheless, you’ll run into some points when coping with excessive cardinal options (options with many classes)
Right here is how you need to use goal encoding to remodel Categorical options into numerical values.
Early in any knowledge science course, you might be launched to 1 scorching encoding as a key technique to take care of categorical values, and rightfully so, as this technique works rather well on low cardinal options (options with restricted classes).
In a nutshell, One scorching encoding transforms every class right into a binary vector, the place the corresponding class is marked as ‘True’ or ‘1’, and all different classes are marked with ‘False’ or ‘0’.
import pandas as pd# Pattern categorical knowledge
knowledge = {'Class': ['Red', 'Green', 'Blue', 'Red', 'Green']}
# Create a DataFrame
df = pd.DataFrame(knowledge)
# Carry out one-hot encoding
one_hot_encoded = pd.get_dummies(df['Category'])
# Show the end result
print(one_hot_encoded)
Whereas this works nice for options with restricted classes (Lower than 10–20 classes), because the variety of classes will increase, the one-hot encoded vectors turn out to be longer and sparser, doubtlessly resulting in elevated reminiscence utilization and computational complexity, let’s have a look at an instance.
The beneath code makes use of Amazon Worker Entry knowledge, made publicity out there in kaggle: https://www.kaggle.com/datasets/lucamassaron/amazon-employee-access-challenge
The info comprises eight categorical function columns indicating traits of the required useful resource, function, and workgroup of the worker at Amazon.
knowledge.information()
# Show the variety of distinctive values in every column
unique_values_per_column = knowledge.nunique()print("Variety of distinctive values in every column:")
print(unique_values_per_column)
Utilizing one scorching encoding could possibly be difficult in a dataset like this because of the excessive variety of distinct classes for every function.
#Preliminary knowledge reminiscence utilization
memory_usage = knowledge.memory_usage(deep=True)
total_memory_usage = memory_usage.sum()
print(f"nTotal reminiscence utilization of the DataFrame: {total_memory_usage / (1024 ** 2):.2f} MB")
#one-hot encoding categorical options
data_encoded = pd.get_dummies(knowledge,
columns=knowledge.select_dtypes(embody='object').columns,
drop_first=True)data_encoded.form
# Reminiscence utilization for the one-hot encoded dataset
memory_usage = data_encoded.memory_usage(deep=True)
total_memory_usage = memory_usage.sum()
print(f"nTotal reminiscence utilization of the DataFrame: {total_memory_usage / (1024 ** 2):.2f} MB")
As you may see, one-hot encoding just isn’t a viable answer to take care of excessive cardinal categorical options, because it considerably will increase the dimensions of the dataset.
In instances with excessive cardinal options, goal encoding is a greater possibility.
Goal encoding transforms a categorical function right into a numeric function with out including any further columns, avoiding turning the dataset into a bigger and sparser dataset.
Goal encoding works by changing every class of a categorical function into its corresponding anticipated worth. The method to calculating the anticipated worth will depend upon the worth you are attempting to foretell.
For Regression issues, the anticipated worth is solely the common worth for that class.
For Classification issues, the anticipated worth is the conditional likelihood on condition that class.
In each instances, we will get the outcomes by merely utilizing the ‘group_by’ operate in pandas.
#Instance of how you can calculate the anticipated worth for Goal encoding of a Binary final result
expected_values = knowledge.groupby('ROLE_TITLE')['ACTION'].value_counts(normalize=True).unstack()
expected_values
The ensuing desk signifies the likelihood of every “ACTION” final result by distinctive “ROLE_TITLE” id. All that’s left to do is exchange the “ROLE_TITLE” id with the values from the likelihood of “ACTION” being 1 within the authentic dataset. (i.e as an alternative of class 117879 the dataset will present 0.889331)
Whereas this can provide us an instinct of how goal encoding works, utilizing this straightforward technique runs the chance of overfitting. Particularly for uncommon classes, as in these instances, goal encoding will primarily present the goal worth to the mannequin. Additionally, the above technique can solely take care of seen classes, so in case your check knowledge has a brand new class, it received’t have the ability to deal with it.
To keep away from these errors, you must make the goal encoding transformer extra sturdy.
To make goal encoding extra sturdy, you may create a customized transformer class and combine it with scikit-learn in order that it may be utilized in any mannequin pipeline.
NOTE: The beneath code is taken from the ebook “The Kaggle Guide” and could be present in Kaggle: https://www.kaggle.com/code/lucamassaron/meta-features-and-target-encoding
import numpy as np
import pandas as pdfrom sklearn.base import BaseEstimator, TransformerMixin
class TargetEncode(BaseEstimator, TransformerMixin):
def __init__(self, classes='auto', ok=1, f=1,
noise_level=0, random_state=None):
if sort(classes)==str and classes!='auto':
self.classes = [categories]
else:
self.classes = classes
self.ok = ok
self.f = f
self.noise_level = noise_level
self.encodings = dict()
self.prior = None
self.random_state = random_state
def add_noise(self, sequence, noise_level):
return sequence * (1 + noise_level *
np.random.randn(len(sequence)))
def match(self, X, y=None):
if sort(self.classes)=='auto':
self.classes = np.the place(X.dtypes == sort(object()))[0]
temp = X.loc[:, self.categories].copy()
temp['target'] = y
self.prior = np.imply(y)
for variable in self.classes:
avg = (temp.groupby(by=variable)['target']
.agg(['mean', 'count']))
# Compute smoothing
smoothing = (1 / (1 + np.exp(-(avg['count'] - self.ok) /
self.f)))
# The larger the depend the much less full_avg is accounted
self.encodings[variable] = dict(self.prior * (1 -
smoothing) + avg['mean'] * smoothing)
return self
def remodel(self, X):
Xt = X.copy()
for variable in self.classes:
Xt[variable].exchange(self.encodings[variable],
inplace=True)
unknown_value = {worth:self.prior for worth in
X[variable].distinctive()
if worth not in
self.encodings[variable].keys()}
if len(unknown_value) > 0:
Xt[variable].exchange(unknown_value, inplace=True)
Xt[variable] = Xt[variable].astype(float)
if self.noise_level > 0:
if self.random_state just isn't None:
np.random.seed(self.random_state)
Xt[variable] = self.add_noise(Xt[variable],
self.noise_level)
return Xt
def fit_transform(self, X, y=None):
self.match(X, y)
return self.remodel(X)
It’d look daunting at first, however let’s break down every a part of the code to grasp how you can create a strong Goal encoder.
Class Definition
class TargetEncode(BaseEstimator, TransformerMixin):
This primary step ensures that you need to use this transformer class in scikit-learn pipelines for knowledge preprocessing, function engineering, and machine studying workflows. It achieves this by inheriting the scikit-learn lessons BaseEstimator and TransformerMixin.
Inheritance permits the TargetEncode class to reuse or override strategies and attributes outlined within the base lessons, on this case, BaseEstimator and TransformerMixin
BaseEstimator is a base class for all scikit-learn estimators. Estimators are objects in scikit-learn with a “match” technique for coaching on knowledge and a “predict” technique for making predictions.
TransformerMixin is a mixin class for transformers in scikit-learn, it gives extra strategies corresponding to “fit_transform”, which mixes becoming and reworking in a single step.
Inheriting from BaseEstimator & TransformerMixin, permits TargetEncode to implement these strategies, making it appropriate with the scikit-learn API.
Defining the constructor
def __init__(self, classes='auto', ok=1, f=1,
noise_level=0, random_state=None):
if sort(classes)==str and classes!='auto':
self.classes = [categories]
else:
self.classes = classes
self.ok = ok
self.f = f
self.noise_level = noise_level
self.encodings = dict()
self.prior = None
self.random_state = random_state
This second step defines the constructor for the “TargetEncode” class and initializes the occasion variables with default or user-specified values.
The “classes” parameter determines which columns within the enter knowledge must be thought of as categorical variables for goal encoding. It’s Set by default to ‘auto’ to routinely determine categorical columns throughout the becoming course of.
The parameters ok, f, and noise_level management the smoothing impact throughout goal encoding and the extent of noise added throughout transformation.
Including noise
This subsequent step is essential to keep away from overfitting.
def add_noise(self, sequence, noise_level):
return sequence * (1 + noise_level *
np.random.randn(len(sequence)))
The “add_noise” technique provides random noise to introduce variability and forestall overfitting throughout the transformation section.
“np.random.randn(len(sequence))” generates an array of random numbers from an ordinary regular distribution (imply = 0, customary deviation = 1).
Multiplying this array by “noise_level” scales the random noise based mostly on the desired noise degree.”
This step contributes to the robustness and generalization capabilities of the goal encoding course of.
Becoming the Goal encoder
This a part of the code trains the goal encoder on the supplied knowledge by calculating the goal encodings for categorical columns and storing them for later use throughout transformation.
def match(self, X, y=None):
if sort(self.classes)=='auto':
self.classes = np.the place(X.dtypes == sort(object()))[0]temp = X.loc[:, self.categories].copy()
temp['target'] = y
self.prior = np.imply(y)
for variable in self.classes:
avg = (temp.groupby(by=variable)['target']
.agg(['mean', 'count']))
# Compute smoothing
smoothing = (1 / (1 + np.exp(-(avg['count'] - self.ok) /
self.f)))
# The larger the depend the much less full_avg is accounted
self.encodings[variable] = dict(self.prior * (1 -
smoothing) + avg['mean'] * smoothing)
The smoothing time period helps stop overfitting, particularly when coping with classes with small samples.
The tactic follows the scikit-learn conference for match strategies in transformers.
It begins by checking and figuring out the specific columns and creating a short lived DataFrame, containing solely the chosen categorical columns from the enter X and the goal variable y.
The prior imply of the goal variable is calculated and saved within the prior attribute. This represents the general imply of the goal variable throughout all the dataset.
Then, it calculates the imply and depend of the goal variable for every class utilizing the group-by technique, as seen beforehand.
There’s an extra smoothing step to forestall overfitting on classes with small numbers of samples. Smoothing is calculated based mostly on the variety of samples in every class. The bigger the depend, the much less the smoothing impact.
The calculated encodings for every class within the present variable are saved within the encodings dictionary. This dictionary will likely be used later throughout the transformation section.
Remodeling the info
This a part of the code replaces the unique categorical values with their corresponding target-encoded values saved in self.encodings.
def remodel(self, X):
Xt = X.copy()
for variable in self.classes:
Xt[variable].exchange(self.encodings[variable],
inplace=True)
unknown_value = {worth:self.prior for worth in
X[variable].distinctive()
if worth not in
self.encodings[variable].keys()}
if len(unknown_value) > 0:
Xt[variable].exchange(unknown_value, inplace=True)
Xt[variable] = Xt[variable].astype(float)
if self.noise_level > 0:
if self.random_state just isn't None:
np.random.seed(self.random_state)
Xt[variable] = self.add_noise(Xt[variable],
self.noise_level)
return Xt
This step has an extra robustness verify to make sure the goal encoder can deal with new or unseen classes. For these new or unknown classes, it replaces them with the imply of the goal variable saved within the prior_mean variable.
In case you want extra robustness in opposition to overfitting, you may arrange a noise_level higher than 0 so as to add random noise to the encoded values.
The fit_transform technique combines the performance of becoming and reworking the info by first becoming the transformer to the coaching knowledge after which reworking it based mostly on the calculated encodings.
Now that you just perceive how the code works, let’s see it in motion.
#Instantiate TargetEncode class
te = TargetEncode(classes='ROLE_TITLE')
te.match(knowledge, knowledge['ACTION'])
te.remodel(knowledge[['ROLE_TITLE']])
The Goal encoder changed every “ROLE_TITLE” id with the likelihood of every class. Now, let’s do the identical for all options and verify the reminiscence utilization after utilizing Goal Encoding.
y = knowledge['ACTION']
options = knowledge.drop('ACTION',axis=1)te = TargetEncode(classes=options.columns)
te.match(options,y)
te_data = te.remodel(options)
te_data.head()
memory_usage = te_data.memory_usage(deep=True)
total_memory_usage = memory_usage.sum()
print(f"nTotal reminiscence utilization of the DataFrame: {total_memory_usage / (1024 ** 2):.2f} MB")
Goal encoding efficiently remodeled the specific knowledge into numerical with out creating further columns or growing reminiscence utilization.
Thus far we now have created our personal goal encoder class, nonetheless you don’t have to do that anymore.
In scikit-learn model 1.3 launch, someplace round June 2023, they launched the Goal Encoder class to their API. Right here is how you need to use goal encoding with Scikit Be taught
from sklearn.preprocessing import TargetEncoder#Splitting the info
y = knowledge['ACTION']
options = knowledge.drop('ACTION',axis=1)
#Specify the goal sort
te = TargetEncoder(clean="auto",target_type='binary')
X_trans = te.fit_transform(options, y)
#Making a Dataframe
features_encoded = pd.DataFrame(X_trans, columns = options.columns)
Word that we’re getting barely completely different outcomes from the guide Goal encoder class due to the graceful parameter and randomness on the noise degree.
As you see, sklearn makes it straightforward to run goal encoding transformations. Nevertheless, you will need to perceive how the transformation works beneath the hood first to grasp and clarify the output.
Whereas Goal encoding is a robust encoding technique, it’s essential to think about the particular necessities and traits of your dataset and select the encoding technique that most accurately fits your wants and the necessities of the machine studying algorithm you intend to make use of.
[1] Banachewicz, Ok. & Massaron, L. (2022). The Kaggle Guide: Information Evaluation and Machine Studying for Aggressive Information Science. Packt>
[2] Massaron, L. (2022, January). Amazon Worker Entry Problem. Retrieved February 1, 2024, from https://www.kaggle.com/datasets/lucamassaron/amazon-employee-access-challenge
[3] Massaron, L. Meta-features and goal encoding. Retrieved February 1, 2024, from https://www.kaggle.com/luca-massaron/meta-features-and-target-encoding
[4] Scikit-learn.sklearn.preprocessing.TargetEncoder. In scikit-learn: Machine studying in Python (Model 1.3). Retrieved February 1, 2024, from https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.TargetEncoder.html
