Sturdy One-Scorching Encoding. Manufacturing grade one-hot encoding… | by Hans Christian Ekne | Apr, 2024

The best way we construct conventional machine studying fashions is to first prepare the fashions on a “coaching dataset” — usually a dataset of historic values — after which later we generate predictions on a brand new dataset, the “inference dataset.” If the columns of the coaching dataset and the inference dataset don’t match, your machine studying algorithm will normally fail. That is primarily because of both lacking or new issue ranges within the inference dataset.

The primary downside: Lacking components

For the next examples, assume that you simply used the dataset above to coach your machine studying mannequin. You one-hot encoded the dataset into dummy variables, and your absolutely reworked coaching information seems like under:

Now, let’s introduce the inference dataset, that is what you’ll use for making predictions. Let’s say it’s given like under:

# Creating the inference_data DataFrame in Python
inference_data = pd.DataFrame({
'numerical_1': [11, 12, 13, 14, 15, 16, 17, 18],
'color_1_': ['black', 'blue', 'black', 'green', 
'green', 'black', 'black', 'blue'],
'color_2_': ['orange', 'orange', 'black', 'orange', 
'black', 'orange', 'orange', 'orange']
})

Utilizing a naive one-hot encoding technique like we used above (pd.get_dummies)

# Changing categorical columns in inference_data to 
# Dummy variables with integers
inference_data_dummies = pd.get_dummies(inference_data, 
columns=['color_1_', 'color_2_']).astype(int)

This might remodel your inference dataset in the identical manner, and also you acquire the dataset under:

Do you discover the issues? The primary downside is that the inference dataset is lacking the columns:

missing_colmns =['color_1__red', 'color_2__pink', 
'color_2__blue', 'color_2__purple']

In the event you ran this in a mannequin skilled with the “coaching dataset” it could normally crash.

The second downside: New components

The opposite downside that may happen with one-hot encoding is that if your inference dataset consists of new and unseen components. Think about once more the identical datasets as above. In the event you study intently, you see that the inference dataset now has a brand new column: color_2__orange.

That is the other downside as beforehand, and our inference dataset comprises new columns which our coaching dataset didn’t have. That is really a typical prevalence and might occur if certainly one of your issue variables had adjustments. For instance, if the colors above signify colors of a automotive, and a automotive producer out of the blue began making orange vehicles, then this information won’t be out there within the coaching information, however might nonetheless present up within the inference information. On this case you want a strong manner of coping with the problem.

One might argue, effectively why don’t you record all of the columns within the reworked coaching dataset as columns that may be wanted on your inference dataset? The issue right here is that you simply usually don’t know what issue ranges are within the coaching information upfront.

For instance, new ranges might be launched repeatedly, which might make it troublesome to keep up. On prime of that comes the method of then matching your inference dataset with the coaching information, so that you would wish to test all precise reworked column names that went into the coaching algorithm, after which match them with the reworked inference dataset. If any columns had been lacking you would wish to insert new columns with 0 values and in case you had further columns, just like the color_2__orange columns above, these would should be deleted. It is a relatively cumbersome manner of fixing the problem, and fortunately there are higher choices out there.

The answer to this downside is relatively simple, nonetheless lots of the packages and libraries that try and streamline the method of making prediction fashions fail to implement it effectively. The important thing lies in having a operate or class that’s first fitted on the coaching information, after which use that very same occasion of the operate or class to remodel each the coaching dataset and the inference dataset. Under we discover how that is finished utilizing each Python and R.

In Python

Python is arguably one the most effective programming language to make use of for machine studying, largely because of its intensive community of builders and mature bundle libraries, and its ease of use, which promotes fast growth.

Relating to the problems associated to one-hot encoding we described above, they are often mitigated through the use of the broadly out there and examined scikit-learn library, and extra particularly the sklearn.preprocessing.OneHotEncoder class. So, let’s see how we will use that on our coaching and inference datasets to create a strong one-hot encoding.

from sklearn.preprocessing import OneHotEncoder
# Initialize the encoder
enc = OneHotEncoder(handle_unknown='ignore')
# Outline columns to remodel
trans_columns = ['color_1_', 'color_2_']
# Match and remodel the information
enc_data = enc.fit_transform(training_data[trans_columns])
# Get function names
feature_names = enc.get_feature_names_out(trans_columns)
# Convert to DataFrame
enc_df = pd.DataFrame(enc_data.toarray(), 
columns=feature_names)
# Concatenate with the numerical information
final_df = pd.concat([training_data[['numerical_1']], 
enc_df], axis=1)

This produces a last DataFrameof reworked values as proven under:

If we break down the code above, we see that step one is to initialize the an occasion of the encoder class. We use the choice handle_unknown='ignore' in order that we keep away from points with unknow values for the columns once we use the encoder to remodel on our inference dataset.

After that, we mix a match and remodel motion into one step with the fit_transform technique. And at last, we create a brand new information body from the encoded information and concatenate it with the remainder of the unique dataset.