Organizing a Machine Studying Monorepo with Pants | by Michał Oleszak | Aug, 2023

MLOps

Streamline your ML workflow administration

Have you ever ever copy-pasted chunks of utility code between tasks, leading to a number of variations of the identical code dwelling in several repositories? Or, maybe, you needed to make pull requests to tens of tasks after the identify of the GCP bucket during which you retailer your knowledge was up to date?

Conditions described above come up method too typically in ML groups, and their penalties fluctuate from a single developer’s annoyance to the group’s incapacity to ship their code as wanted. Fortunately, there’s a treatment.

Let’s dive into the world of monorepos, an structure extensively adopted in main tech firms like Google, and the way they will improve your ML workflows. A monorepo gives a plethora of benefits which, regardless of some drawbacks, make it a compelling selection for managing advanced machine studying ecosystems.

We’ll briefly debate monorepos’ deserves and demerits, study why it’s a superb structure selection for machine studying groups, and peek into how Large Tech is utilizing it. Lastly, we’ll see methods to harness the facility of the Pants construct system to arrange your machine studying monorepo into a sturdy CI/CD construct system.

Strap in as we embark on this journey to streamline your ML mission administration.

This text was first revealed on the neptune.ai blog.

A monorepo (brief for monolithic repository) is a software program growth technique the place code for a lot of tasks is saved in the identical repository. The concept could be as broad as all of the corporate code written in quite a lot of programming languages saved collectively (did anyone say Google?) or as slim as a few Python tasks developed by a small group thrown right into a single repository.

On this weblog put up, we concentrate on repositories storing machine studying code.