Unlock ML insights utilizing the Amazon SageMaker Function Retailer Function Processor

Amazon SageMaker Feature Store offers an end-to-end answer to automate function engineering for machine studying (ML). For a lot of ML use circumstances, uncooked knowledge like log recordsdata, sensor readings, or transaction data must be remodeled into significant options which might be optimized for mannequin coaching.

Function high quality is vital to make sure a extremely correct ML mannequin. Reworking uncooked knowledge into options utilizing aggregation, encoding, normalization, and different operations is commonly wanted and might require vital effort. Engineers should manually write customized knowledge preprocessing and aggregation logic in Python or Spark for every use case.

This undifferentiated heavy lifting is cumbersome, repetitive, and error-prone. The SageMaker Feature Store Feature Processor reduces this burden by robotically reworking uncooked knowledge into aggregated options appropriate for batch coaching ML fashions. It lets engineers present easy knowledge transformation features, then handles working them at scale on Spark and managing the underlying infrastructure. This permits knowledge scientists and knowledge engineers to give attention to the function engineering logic moderately than implementation particulars.

On this submit, we show how a automotive gross sales firm can use the Function Processor to remodel uncooked gross sales transaction knowledge into options in three steps:

1. Native runs of information transformations.
2. Distant runs at scale utilizing Spark.
3. Operationalization through pipelines.

We present how SageMaker Function Retailer ingests the uncooked knowledge, runs function transformations remotely utilizing Spark, and hundreds the ensuing aggregated options right into a feature group. These engineered options are can then be used to coach ML fashions.

For this use case, we see how SageMaker Function Retailer helps convert the uncooked automotive gross sales knowledge into structured options. These options are subsequently used to realize insights like:

• Common and most worth of crimson convertibles from 2010
• Fashions with finest mileage vs. worth
• Gross sales tendencies of recent vs. used automobiles over time
• Variations in common MSRP throughout areas

We additionally see how SageMaker Function Retailer pipelines preserve the options up to date as new knowledge is available in, enabling the corporate to repeatedly achieve insights over time.

Answer overview

We work with the dataset car_data.csv, which incorporates specs comparable to mannequin, yr, standing, mileage, worth, and MSRP for used and new automobiles offered by the corporate. The next screenshot exhibits an instance of the dataset.

The answer pocket book feature_processor.ipynb incorporates the next principal steps, which we clarify on this submit:

1. Create two function teams: one known as car-data for uncooked automotive gross sales data and one other known as car-data-aggregated for aggregated automotive gross sales data.
2. Use the @feature_processor decorator to load knowledge into the car-data function group from Amazon Simple Storage Service (Amazon S3).
3. Run the @feature_processor code remotely as a Spark utility to mixture the info.
4. Operationalize the function processor through SageMaker pipelines and schedule runs.
5. Discover the function processing pipelines and lineage in Amazon SageMaker Studio.
6. Use aggregated options to coach an ML mannequin.

Conditions

To comply with this tutorial, you want the next:

For this submit, we consult with the next notebook, which demonstrates how you can get began with Function Processor utilizing the SageMaker Python SDK.

Create function teams

To create the function teams, full the next steps:

1. Create a function group definition for car-data as follows:

   # Function Group - Automobile Gross sales CAR_SALES_FG_NAME = "car-data"
   CAR_SALES_FG_ARN = f"arn:aws:sagemaker:{area}:{aws_account_id}:feature-group/{CAR_SALES_FG_NAME}"
   CAR_SALES_FG_ROLE_ARN = offline_store_role
   CAR_SALES_FG_OFFLINE_STORE_S3_URI = f"s3://{s3_bucket}/{s3_offline_store_prefix}"
   CAR_SALES_FG_FEATURE_DEFINITIONS = [
       FeatureDefinition(feature_name="id", feature_type=FeatureTypeEnum.STRING),
       FeatureDefinition(feature_name="model", feature_type=FeatureTypeEnum.STRING),
       FeatureDefinition(feature_name="model_year", feature_type=FeatureTypeEnum.STRING),
       FeatureDefinition(feature_name="status", feature_type=FeatureTypeEnum.STRING),
       FeatureDefinition(feature_name="mileage", feature_type=FeatureTypeEnum.STRING),
       FeatureDefinition(feature_name="price", feature_type=FeatureTypeEnum.STRING),
       FeatureDefinition(feature_name="msrp", feature_type=FeatureTypeEnum.STRING),
       FeatureDefinition(feature_name="ingest_time", feature_type=FeatureTypeEnum.FRACTIONAL),
   ]

The options correspond to every column within the car_data.csv dataset (Mannequin, Yr, Standing, Mileage, Worth, and MSRP).

2. Add the file identifier id and occasion time ingest_time to the function group:

CAR_SALES_FG_RECORD_IDENTIFIER_NAME = "id"
CAR_SALES_FG_EVENT_TIME_FEATURE_NAME = "ingest_time"

3. Create a function group definition for car-data-aggregated as follows:

# Function Group - Aggregated Automobile SalesAGG_CAR_SALES_FG_NAME = "car-data-aggregated"
AGG_CAR_SALES_FG_ARN = (
    f"arn:aws:sagemaker:{area}:{aws_account_id}:feature-group/{AGG_CAR_SALES_FG_NAME}"
)
AGG_CAR_SALES_FG_ROLE_ARN = offline_store_role
AGG_CAR_SALES_FG_OFFLINE_STORE_S3_URI = f"s3://{s3_bucket}/{s3_offline_store_prefix}"
AGG_CAR_SALES_FG_FEATURE_DEFINITIONS = [
    FeatureDefinition(feature_name="model_year_status", feature_type=FeatureTypeEnum.STRING),
    FeatureDefinition(feature_name="avg_mileage", feature_type=FeatureTypeEnum.STRING),
    FeatureDefinition(feature_name="max_mileage", feature_type=FeatureTypeEnum.STRING),
    FeatureDefinition(feature_name="avg_price", feature_type=FeatureTypeEnum.STRING),
    FeatureDefinition(feature_name="max_price", feature_type=FeatureTypeEnum.STRING),
    FeatureDefinition(feature_name="avg_msrp", feature_type=FeatureTypeEnum.STRING),
    FeatureDefinition(feature_name="max_msrp", feature_type=FeatureTypeEnum.STRING),
    FeatureDefinition(feature_name="ingest_time", feature_type=FeatureTypeEnum.FRACTIONAL),
]

For the aggregated function group, the options are mannequin yr standing, common mileage, max mileage, common worth, max worth, common MSRP, max MSRP, and ingest time. We add the file identifier model_year_status and occasion time ingest_time to this function group.

4. Now, create the car-data function group:

# Create Function Group - Automobile sale data.
car_sales_fg = FeatureGroup(
    identify=CAR_SALES_FG_NAME,
    feature_definitions=CAR_SALES_FG_FEATURE_DEFINITIONS,
    sagemaker_session=sagemaker_session,
)

create_car_sales_fg_resp = car_sales_fg.create(
        record_identifier_name=CAR_SALES_FG_RECORD_IDENTIFIER_NAME,
        event_time_feature_name=CAR_SALES_FG_EVENT_TIME_FEATURE_NAME,
        s3_uri=CAR_SALES_FG_OFFLINE_STORE_S3_URI,
        enable_online_store=True,
        role_arn=CAR_SALES_FG_ROLE_ARN,
    )

5. Create the car-data-aggregated function group:

# Create Function Group - Aggregated automotive gross sales data.
agg_car_sales_fg = FeatureGroup(
    identify=AGG_CAR_SALES_FG_NAME,
    feature_definitions=AGG_CAR_SALES_FG_FEATURE_DEFINITIONS,
    sagemaker_session=sagemaker_session,
)

create_agg_car_sales_fg_resp = agg_car_sales_fg.create(
        record_identifier_name=AGG_CAR_SALES_FG_RECORD_IDENTIFIER_NAME,  
        event_time_feature_name=AGG_CAR_SALES_FG_EVENT_TIME_FEATURE_NAME,
        s3_uri=AGG_CAR_SALES_FG_OFFLINE_STORE_S3_URI,
        enable_online_store=True,
        role_arn=AGG_CAR_SALES_FG_ROLE_ARN,
    )

You’ll be able to navigate to the SageMaker Function Retailer possibility beneath Knowledge on the SageMaker Studio Residence menu to see the function teams.

Use the @feature_processor decorator to load knowledge

On this part, we domestically remodel the uncooked enter knowledge (car_data.csv) from Amazon S3 into the car-data function group utilizing the Function Retailer Function Processor. This preliminary native run permits us to develop and iterate earlier than working remotely, and may very well be executed on a pattern of the info if desired for sooner iteration.

With the @feature_processor decorator, your transformation perform runs in a Spark runtime surroundings the place the enter arguments supplied to your perform and its return worth are Spark DataFrames.

1. Set up the Feature Processor SDK from the SageMaker Python SDK and its extras utilizing the next command:

pip set up sagemaker[feature-processor]

The variety of enter parameters in your transformation perform should match the variety of inputs configured within the @feature_processor decorator. On this case, the @feature_processor decorator has car-data.csv as enter and the car-data function group as output, indicating this can be a batch operation with the target_store as OfflineStore:

from sagemaker.feature_store.feature_processor import (
    feature_processor,
    FeatureGroupDataSource,
    CSVDataSource,
)

@feature_processor(
    inputs=[CSVDataSource(RAW_CAR_SALES_S3_URI)],
    output=CAR_SALES_FG_ARN,
    target_stores=["OfflineStore"],
)

2. Outline the remodel() perform to remodel the info. This perform performs the next actions:
   • Convert column names to lowercase.
   • Add the occasion time to the ingest_time column.
   • Take away punctuation and change lacking values with NA.

def remodel(raw_s3_data_as_df):
    """Load knowledge from S3, carry out primary function engineering, retailer it in a Function Group"""
    from pyspark.sql.features import regexp_replace
    from pyspark.sql.features import lit
    import time

    transformed_df = (
        raw_s3_data_as_df.withColumn("Worth", regexp_replace("Worth", "$", ""))
        # Rename Columns
        .withColumnRenamed("Id", "id")
        .withColumnRenamed("Mannequin", "mannequin")
        .withColumnRenamed("Yr", "model_year")
        .withColumnRenamed("Standing", "standing")
        .withColumnRenamed("Mileage", "mileage")
        .withColumnRenamed("Worth", "worth")
        .withColumnRenamed("MSRP", "msrp")
        # Add Occasion Time
        .withColumn("ingest_time", lit(int(time.time())))
        # Take away punctuation and fluff; change with NA
        .withColumn("mileage", regexp_replace("mileage", "(,)|(mi.)", ""))
        .withColumn("mileage", regexp_replace("mileage", "Not accessible", "NA"))
        .withColumn("worth", regexp_replace("worth", ",", ""))
        .withColumn("msrp", regexp_replace("msrp", "(^MSRPs$)|(,)", ""))
        .withColumn("msrp", regexp_replace("msrp", "Not specified", "NA"))
        .withColumn("msrp", regexp_replace("msrp", "$d+[a-zA-Zs]+", "NA"))
        .withColumn("mannequin", regexp_replace("mannequin", "^dddds", ""))
    )

3. Name the remodel() perform to retailer the info within the car-data function group:

# Execute the FeatureProcessor
remodel()

The output exhibits that the info is ingested efficiently into the car-data function group.

The output of the transform_df.present() perform is as follows:

INFO:sagemaker:Ingesting remodeled knowledge to arn:aws:sagemaker:us-west-2:416578662734:feature-group/car-data with target_stores: ['OfflineStore']

+---+--------------------+----------+------+-------+--------+-----+-----------+
| id|               mannequin|model_year|standing|mileage|   worth| msrp|ingest_time|
+---+--------------------+----------+------+-------+--------+-----+-----------+
|  0|    Acura TLX A-Spec|      2022|   New|     NA|49445.00|49445| 1686627154|
|  1|    Acura RDX A-Spec|      2023|   New|     NA|50895.00|   NA| 1686627154|
|  2|    Acura TLX Sort S|      2023|   New|     NA|57745.00|   NA| 1686627154|
|  3|    Acura TLX Sort S|      2023|   New|     NA|57545.00|   NA| 1686627154|
|  4|Acura MDX Sport H...|      2019|  Used| 32675 |40990.00|   NA| 1686627154|
|  5|    Acura TLX A-Spec|      2023|   New|     NA|50195.00|50195| 1686627154|
|  6|    Acura TLX A-Spec|      2023|   New|     NA|50195.00|50195| 1686627154|
|  7|    Acura TLX Sort S|      2023|   New|     NA|57745.00|   NA| 1686627154|
|  8|    Acura TLX A-Spec|      2023|   New|     NA|47995.00|   NA| 1686627154|
|  9|    Acura TLX A-Spec|      2022|   New|     NA|49545.00|   NA| 1686627154|
| 10|Acura Integra w/A...|      2023|   New|     NA|36895.00|36895| 1686627154|
| 11|    Acura TLX A-Spec|      2023|   New|     NA|48395.00|48395| 1686627154|
| 12|Acura MDX Sort S ...|      2023|   New|     NA|75590.00|   NA| 1686627154|
| 13|Acura RDX A-Spec ...|      2023|   New|     NA|55345.00|   NA| 1686627154|
| 14|    Acura TLX A-Spec|      2023|   New|     NA|50195.00|50195| 1686627154|
| 15|Acura RDX A-Spec ...|      2023|   New|     NA|55045.00|   NA| 1686627154|
| 16|    Acura TLX Sort S|      2023|   New|     NA|56445.00|   NA| 1686627154|
| 17|    Acura TLX A-Spec|      2023|   New|     NA|47495.00|47495| 1686627154|
| 18|   Acura TLX Advance|      2023|   New|     NA|52245.00|52245| 1686627154|
| 19|    Acura TLX A-Spec|      2023|   New|     NA|50595.00|50595| 1686627154|
+---+--------------------+----------+------+-------+--------+-----+-----------+
solely exhibiting high 20 rows

We now have efficiently remodeled the enter knowledge and ingested it within the car-data function group.

Run the @feature_processor code remotely

On this part, we show working the function processing code remotely as a Spark utility utilizing the @distant decorator described earlier. We run the function processing remotely utilizing Spark to scale to massive datasets. Spark offers distributed processing on clusters to deal with knowledge that’s too large for a single machine. The @distant decorator runs the native Python code as a single or multi-node SageMaker coaching job.

1. Use the @distant decorator together with the @feature_processor decorator as follows:

@distant(spark_config=SparkConfig(), instance_type = "ml.m5.xlarge", ...)
@feature_processor(inputs=[FeatureGroupDataSource(CAR_SALES_FG_ARN)],
                   output=AGG_CAR_SALES_FG_ARN, target_stores=["OfflineStore"], enable_ingestion=False )

The spark_config parameter signifies that is run as a Spark utility. The SparkConfig occasion configures the Spark configuration and dependencies.

2. Outline the mixture() perform to mixture the info utilizing PySpark SQL and user-defined features (UDFs). This perform performs the next actions:
   • Concatenate mannequin, yr, and standing to create model_year_status.
   • Take the typical of worth to create avg_price.
   • Take the max worth of worth to create max_price.
   • Take the typical of mileage to create avg_mileage.
   • Take the max worth of mileage to create max_mileage.
   • Take the typical of msrp to create avg_msrp.
   • Take the max worth of msrp to create max_msrp.
   • Group by model_year_status.

def mixture(source_feature_group, spark):
    """
    Mixture the info utilizing a SQL question and UDF.
    """
    import time
    from pyspark.sql.sorts import StringType
    from pyspark.sql.features import udf

    @udf(returnType=StringType())
    def custom_concat(*cols, delimeter: str = ""):
        return delimeter.be a part of(cols)

    spark.udf.register("custom_concat", custom_concat)

    # Execute SQL string.
    source_feature_group.createOrReplaceTempView("car_data")
    aggregated_car_data = spark.sql(
        f"""
        SELECT
            custom_concat(mannequin, "_", model_year, "_", standing) as model_year_status,
            AVG(worth) as avg_price,
            MAX(worth) as max_price,
            AVG(mileage) as avg_mileage,
            MAX(mileage) as max_mileage,
            AVG(msrp) as avg_msrp,
            MAX(msrp) as max_msrp,
            "{int(time.time())}" as ingest_time
        FROM car_data
        GROUP BY model_year_status
        """
    )

    aggregated_car_data.present()

    return aggregated_car_data

3. Run the mixture() perform, which creates a SageMaker coaching job to run the Spark utility:

# Execute the combination perform
mixture()

In consequence, SageMaker creates a coaching job to the Spark utility outlined earlier. It can create a Spark runtime surroundings utilizing the sagemaker-spark-processing picture.

We use SageMaker Coaching jobs right here to run our Spark function processing utility. With SageMaker Coaching, you’ll be able to scale back startup occasions to 1 minute or much less through the use of heat pooling, which is unavailable in SageMaker Processing. This makes SageMaker Coaching higher optimized for brief batch jobs like function processing the place startup time is necessary.

4. To view the small print, on the SageMaker console, select Coaching jobs beneath Coaching within the navigation pane, then select the job with the identify aggregate-<timestamp>.

The output of the mixture() perform generates telemetry code. Contained in the output, you will notice the aggregated knowledge as follows:

+--------------------+------------------+---------+------------------+-----------+--------+--------+-----------+
|   model_year_status|         avg_price|max_price|       avg_mileage|max_mileage|avg_msrp|max_msrp|ingest_time|
+--------------------+------------------+---------+------------------+-----------+--------+--------+-----------+
|Acura CL 3.0_1997...|            7950.0|  7950.00|          100934.0|    100934 |    null|      NA| 1686634807|
|Acura CL 3.2 Sort...|            6795.0|  7591.00|          118692.5|    135760 |    null|      NA| 1686634807|
|Acura CL 3_1998_Used|            9899.0|  9899.00|           63000.0|     63000 |    null|      NA| 1686634807|
|Acura ILX 2.0L Te...|         14014.125| 18995.00|         95534.875|     89103 |    null|      NA| 1686634807|
|Acura ILX 2.0L Te...|           15008.2| 16998.00|           94935.0|     88449 |    null|      NA| 1686634807|
|Acura ILX 2.0L Te...|           16394.6| 19985.00|           97719.4|     80000 |    null|      NA| 1686634807|
|Acura ILX 2.0L w/...|14567.181818181818| 16999.00| 96624.72727272728|     98919 |    null|      NA| 1686634807|
|Acura ILX 2.0L w/...|           16673.4| 18995.00|           84848.6|     96637 |    null|      NA| 1686634807|
|Acura ILX 2.0L w/...|12580.333333333334| 14546.00|100207.33333333333|     95782 |    null|      NA| 1686634807|
|Acura ILX 2.0L_20...|         14565.375| 17590.00|         92941.125|     81842 |    null|      NA| 1686634807|
|Acura ILX 2.0L_20...|           14877.9|  9995.00|           99739.5|     89252 |    null|      NA| 1686634807|
|Acura ILX 2.0L_20...|           15659.5| 15660.00|           82136.0|     89942 |    null|      NA| 1686634807|
|Acura ILX 2.0L_20...|17121.785714285714| 20990.00| 78278.14285714286|     96067 |    null|      NA| 1686634807|
|Acura ILX 2.4L (A...|           17846.0| 21995.00|          101558.0|     85974 |    null|      NA| 1686634807|
|Acura ILX 2.4L Pr...|           16327.0| 16995.00|           85238.0|     95356 |    null|      NA| 1686634807|
|Acura ILX 2.4L w/...|           12846.0| 12846.00|           75209.0|     75209 |    null|      NA| 1686634807|
|Acura ILX 2.4L_20...|           18998.0| 18998.00|           51002.0|     51002 |    null|      NA| 1686634807|
|Acura ILX 2.4L_20...|17908.615384615383| 19316.00| 74325.38461538461|     89116 |    null|      NA| 1686634807|
|Acura ILX 4DR SDN...|           18995.0| 18995.00|           37017.0|     37017 |    null|      NA| 1686634807|
|Acura ILX 8-SPD_2...|           24995.0| 24995.00|           22334.0|     22334 |    null|      NA| 1686634807|
+--------------------+------------------+---------+------------------+-----------+--------+--------+-----------+
solely exhibiting high 20 rows

When the coaching job is full, it’s best to see following output:

06-13 05:40 smspark-submit INFO     spark submit was profitable. main node exiting.
Coaching seconds: 153
Billable seconds: 153

Operationalize the function processor through SageMaker pipelines

On this part, we show how you can operationalize the function processor by selling it to a SageMaker pipeline and scheduling runs.

1. First, add the transformation_code.py file containing the function processing logic to Amazon S3:

car_data_s3_uri = s3_path_join("s3://", sagemaker_session.default_bucket(),
                               'transformation_code', 'car-data-ingestion.py')
S3Uploader.add(local_path="car-data-ingestion.py", desired_s3_uri=car_data_s3_uri)
print(car_data_s3_uri)

2. Subsequent, create a Function Processor pipeline car_data_pipeline utilizing the .to_pipeline() perform:

car_data_pipeline_name = f"{CAR_SALES_FG_NAME}-ingestion-pipeline"
car_data_pipeline_arn = fp.to_pipeline(pipeline_name=car_data_pipeline_name,
                                      step=remodel,
                                      transformation_code=TransformationCode(s3_uri=car_data_s3_uri) )
print(f"Created SageMaker Pipeline: {car_data_pipeline_arn}.")

3. To run the pipeline, use the next code:

car_data_pipeline_execution_arn = fp.execute(pipeline_name=car_data_pipeline_name)
print(f"Began an execution with execution arn: {car_data_pipeline_execution_arn}")

4. Equally, you’ll be able to create a pipeline for aggregated options known as car_data_aggregated_pipeline and begin a run.
5. Schedule the car_data_aggregated_pipeline to run each 24 hours:

fp.schedule(pipeline_name=car_data_aggregated_pipeline_name,
           schedule_expression="price(24 hours)", state="ENABLED")
print(f"Created a schedule.")

Within the output part, you will notice the ARN of pipeline and the pipeline execution function, and the schedule particulars:

{'pipeline_arn': 'arn:aws:sagemaker:us-west-2:416578662734:pipeline/car-data-aggregated-ingestion-pipeline',
 'pipeline_execution_role_arn': 'arn:aws:iam::416578662734:function/service-role/AmazonSageMaker-ExecutionRole-20230612T120731',
 'schedule_arn': 'arn:aws:scheduler:us-west-2:416578662734:schedule/default/car-data-aggregated-ingestion-pipeline',
 'schedule_expression': 'price(24 hours)',
 'schedule_state': 'ENABLED',
 'schedule_start_date': '2023-06-13T06:05:17Z',
 'schedule_role': 'arn:aws:iam::416578662734:function/service-role/AmazonSageMaker-ExecutionRole-20230612T120731'}

6. To get all of the Function Processor pipelines on this account, use the list_pipelines() perform on the Function Processor:

The output might be as follows:

[{'pipeline_name': 'car-data-aggregated-ingestion-pipeline'},
 {'pipeline_name': 'car-data-ingestion-pipeline'}]

We now have efficiently created SageMaker Function Processor pipelines.

Discover function processing pipelines and ML lineage

In SageMaker Studio, full the next steps:

1. On the SageMaker Studio console, on the Residence menu, select Pipelines.

It’s best to see two pipelines created: car-data-ingestion-pipeline and car-data-aggregated-ingestion-pipeline.

2. Select the car-data-ingestion-pipeline.

It exhibits the run particulars on the Executions tab.

3. To view the function group populated by the pipeline, select Function Retailer beneath Knowledge and select car-data.

You will note the 2 function teams we created within the earlier steps.

4. Select the car-data function group.

You will note the options particulars on the Options tab.

View pipeline runs

To view the pipeline runs, full the next steps:

1. On the Pipeline Executions tab, choose car-data-ingestion-pipeline.

It will present all of the runs.

2. Select one of many hyperlinks to see the small print of the run.

3. To view lineage, select Lineage.

The total lineage for car-data exhibits the enter knowledge supply car_data.csv and upstream entities. The lineage for car-data-aggregated exhibits the enter car-data function group.

4. Select Load options after which select Question upstream lineage on car-data and car-data-ingestion-pipeline to see all of the upstream entities.

The total lineage for car-data function group ought to appear to be the next screenshot.

Equally, the lineage for the car-aggregated-data function group ought to appear to be the next screenshot.

SageMaker Studio offers a single surroundings to trace scheduled pipelines, view runs, discover lineage, and look at the function processing code.

The aggregated options comparable to common worth, max worth, common mileage, and extra within the car-data-aggregated function group present perception into the character of the info. You can too use these options as a dataset to coach a mannequin to foretell automotive costs, or for different operations. Nevertheless, coaching the mannequin is out of scope for this submit, which focuses on demonstrating the SageMaker Function Retailer capabilities for function engineering.

Clear up

Don’t overlook to scrub up the assets created as a part of this submit to keep away from incurring ongoing costs.

1. Disable the scheduled pipeline through the fp.schedule() technique with the state parameter as Disabled:

# Disable the scheduled pipeline
fp.schedule(
pipeline_name=car_data_aggregated_pipeline_name,
schedule_expression="price(24 hours)",
state="DISABLED",
)

2. Delete each function teams:

# Delete function teams
car_sales_fg.delete()
agg_car_sales_fg.delete()

The info residing within the S3 bucket and offline function retailer can incur prices, so it’s best to delete them to keep away from any costs.

3. Delete the S3 objects.
4. Delete the records from the function retailer.

Conclusion

On this submit, we demonstrated how a automotive gross sales firm used SageMaker Function Retailer Function Processor to realize invaluable insights from their uncooked gross sales knowledge by:

• Ingesting and remodeling batch knowledge at scale utilizing Spark
• Operationalizing function engineering workflows through SageMaker pipelines
• Offering lineage monitoring and a single surroundings to observe pipelines and discover options
• Getting ready aggregated options optimized for coaching ML fashions

By following these steps, the corporate was in a position to remodel beforehand unusable knowledge into structured options that might then be used to coach a mannequin to foretell automotive costs. SageMaker Function Retailer enabled them to give attention to function engineering moderately than the underlying infrastructure.

We hope this submit helps you unlock invaluable ML insights from your personal knowledge utilizing SageMaker Function Retailer Function Processor!

For extra data on this, consult with Feature Processing and the SageMaker instance on Amazon SageMaker Feature Store: Feature Processor Introduction.

In regards to the Authors

Dhaval Shah is a Senior Options Architect at AWS, specializing in Machine Studying. With a powerful give attention to digital native companies, he empowers prospects to leverage AWS and drive their enterprise progress. As an ML fanatic, Dhaval is pushed by his ardour for creating impactful options that deliver constructive change. In his leisure time, he indulges in his love for journey and cherishes high quality moments along with his household.

Ninad Joshi is a Senior Options Architect at AWS, serving to world AWS prospects design safe, scalable, and value efficient options in cloud to resolve their complicated real-world enterprise challenges. His work in Machine Studying (ML) covers a variety of AI/ML use circumstances, with a main give attention to Finish-to-Finish ML, Pure Language Processing, and Laptop Imaginative and prescient. Previous to becoming a member of AWS, Ninad labored as a software program developer for 12+ years. Exterior of his skilled endeavors, Ninad enjoys enjoying chess and exploring totally different gambits.