Automate Video Chaptering with LLMs and TF-IDF | by Yann-Aël Le Borgne | Sep, 2024
The important thing steps within the workflow lie in structuring the transcript in paragraphs (step 2) earlier than grouping the paragraphs into chapters from which a desk of contents is derived (step 4). Observe that these two steps might depend on totally different LLMs: A quick and low-cost LLM comparable to LLama 3 8B for the straightforward process of textual content enhancing and paragraph identification, and a extra refined LLM comparable to GPT-4o-mini for the technology of the desk of contents. In between, TF-IDF is used so as to add again timestamp info to the structured paragraphs.
The remainder of the publish describes every step in additional element.
Try the accompanying Github repository and Colab notebook to discover by yourself!
Allow us to use for instance the first lecture of the course ‘MIT 6.S191: Introduction to Deep Studying’ (IntroToDeepLearning.com) by Alexander Amini and Ava Amini (licensed under the MIT License).
Observe that chapters are already supplied within the video description.
This gives us with a baseline to qualitatively evaluate our chaptering later on this publish.
YouTube transcript API
For YouTube movies, an routinely generated transcript is normally made out there by YouTube. A handy solution to retrieve that transcript is by calling the get_transcript methodology of the Python youtube_transcript_api library. The tactic takes the YouTube video_id library as argument:
# https://www.youtube.com/watch?v=ErnWZxJovaM
video_id = "ErnWZxJovaM" # MIT Introduction to Deep Studying - 2024# Retrieve transcript with the youtube_transcript_api library
from youtube_transcript_api import YouTubeTranscriptApi
transcript = YouTubeTranscriptApi.get_transcript(video_id, languages=["en"])
This returns the transcript as an inventory of textual content and timestamp key-value pairs:
[{'text': '[Music]', 'begin': 1.17},
{'textual content': 'good afternoon everybody and welcome to', 'begin': 10.28},
{'textual content': 'MIT sus1 191 my title is Alexander amini', 'begin': 12.88},
{'textual content': "and I will be one among your instructors for", 'begin': 16.84},
...]
The transcript is nonetheless poorly formatted: it lacks punctuation and comprises typos (‘MIT sus1 191’ as a substitute of ‘MIT 6.S191′, or ‘amini’ as a substitute of ‘Amini’).
Speech-to-text with Whisper
Alternatively, a speech-to-text library can be utilized to deduce the transcript from a video or audio file. We advocate utilizing faster-whisper, which is a quick implementation of the state-of-the-art open-source whisper mannequin.
The fashions come in several measurement. Probably the most correct is the ‘large-v3’, which is ready to transcribe about quarter-hour of audio per minute on a T4 GPU (out there free of charge on Google Colab).
from faster_whisper import WhisperModel# Load Whisper mannequin
whisper_model = WhisperModel("large-v3",
machine="cuda" if torch.cuda.is_available() else "cpu",
compute_type="float16",
)
# Name the Whisper transcribe operate on the audio file
initial_prompt = "Use punctuation, like this."
segments, transcript_info = whisper_model.transcribe(audio_file, initial_prompt=initial_prompt, language="en")
The results of the transcription is supplied as segments which may be simply transformed in an inventory of textual content and timestamps as with the youtube_transcript_api library.
Tip: Whisper might typically not include the punctuation. The initial_prompt argument can be utilized to nudge the mannequin so as to add punctuation by offering a small sentence containing punctuation.
Beneath is an excerpt of the transcription of the our video instance with whisper large-v3:
[{'start': 0.0, 'text': ' Good afternoon, everyone, and welcome to MIT Success 191.'},
{'start': 15.28, 'text': " My name is Alexander Amini, and I'll be one of your instructors for the course this year"},
{'start': 19.32, 'duration': 2.08, 'text': ' along with Ava.'}
...]
Observe that in comparison with the YouTube transcription, the punctuation is added. Some transcription errors nonetheless nonetheless stay (‘MIT Success 191’ as a substitute of ‘MIT 6.S191′).
As soon as a transcript is accessible, the second stage consists in enhancing and structuring the transcript in paragraphs.
Transcript enhancing refers to modifications made to enhance readability. This entails, for instance, including punctuation whether it is lacking, correcting grammatical errors, eradicating verbal tics, and many others.
The structuring in paragraphs additionally improves readability, and additionnally serves as a preprocessing step for figuring out chapters in stage 4, since chapters will probably be shaped by grouping paragraphs collectively.
Paragraph enhancing and structuring may be carried out in a single operation, utilizing an LLM. We illustrated beneath the anticipated results of this stage:
This process doesn’t require a really refined LLM because it principally consists in reformulating content material. On the time of writing this text, respectable outcomes could possibly be obtained with for instance GPT-4o-mini or Llama 3 8B, and the next system immediate:
You’re a useful assistant.
Your process is to enhance the consumer enter’s readability: add punctuation if wanted and take away verbal tics, and construction the textual content in paragraphs separated with ‘nn’.
Preserve the wording as trustworthy as potential to the unique textual content.
Put your reply inside <reply></reply> tags.
We depend on OpenAI compatible chat completion API for LLM calling, with messages having the roles of both ‘system’, ‘consumer’ or ‘assistant’. The code beneath illustrates the instantiation of an LLM consumer with Groq, utilizing LLama 3 8B:
# Hook up with Groq with a Groq API key
llm_client = Groq(api_key=api_key)
mannequin = "llama-8b-8192"# Extract textual content from transcript
transcript_text = ' '.be part of([s['text'] for s in transcript])
# Name LLM
response = consumer.chat.completions.create(
messages=[
{
"role": "system",
"content": system_prompt
},
{
"role": "user",
"content": transcript_text
}
],
mannequin=mannequin,
temperature=0,
seed=42
)
Given a bit of uncooked ‘transcript_text’ as enter, this returns an edited piece of textual content inside <reply> tags:
response_content=response.decisions[0].message.content materialprint(response_content)
"""
<reply>
Good afternoon, everybody, and welcome to MIT 6.S191. My title is Alexander Amini, and I will be one among your instructors for the course this yr, together with Ava. We're actually excited to welcome you to this unbelievable course.
This can be a fast-paced and intense one-week course that we're about to undergo collectively. We'll be overlaying the foundations of a quickly altering subject, and a subject that has been revolutionizing many areas of science, arithmetic, physics, and extra.
Over the previous decade, AI and deep studying have been quickly advancing and fixing issues that we did not assume had been solvable in our lifetimes. In the present day, AI is fixing issues past human efficiency, and annually, this lecture is getting more durable and more durable to show as a result of it is presupposed to cowl the foundations of the sector.
</reply>
"""
Allow us to then extract the edited textual content from the <reply> tags, divide it into paragraphs, and construction the outcomes as a JSON dictionary consisting of paragraph numbers and items of textual content:
import re
sample = re.compile(r'<reply>(.*?)</reply>', re.DOTALL)
response_content_edited = sample.findall(response_content)
paragraphs = response_content_edited.strip().break up('nn')
paragraphs_dict = [{'paragraph_number': i, 'paragraph_text': paragraph} for i, paragraph in enumerate(paragraphs)print(paragraph_dict)
[{'paragraph_number': 0,
'paragraph_text': "Good afternoon, everyone, and welcome to MIT 6.S191. My name is Alexander Amini, and I'll be one of your instructors for the course this year, along with Ava. We're really excited to welcome you to this incredible course."},
{'paragraph_number': 1,
'paragraph_text': "This is a fast-paced and intense one-week course that we're about to go through together. We'll be covering the foundations of a rapidly changing field, and a field that has been revolutionizing many areas of science, mathematics, physics, and more."},
{'paragraph_number': 2,
'paragraph_text': "Over the past decade, AI and deep learning have been rapidly advancing and solving problems that we didn't think were solvable in our lifetimes. Today, AI is solving problems beyond human performance, and each year, this lecture is getting harder and harder to teach because it's supposed to cover the foundations of the field."}]
Observe that the enter shouldn’t be too lengthy because the LLM will in any other case ‘overlook’ a part of the textual content. For lengthy inputs, the transcript should be break up in chunks to enhance reliability. We observed that GPT-4o-mini handles nicely as much as 5000 characters, whereas Llama 3 8B can solely deal with as much as 1500 characters. The pocket book gives the operate transcript_to_paragraphs which takes care of splitting the transcript in chunks.
The transcript is now structured as an inventory of edited paragraphs, however the timestamps have been misplaced within the course of.
The third stage consists in including again timestamps, by inferring which section within the uncooked transcript is the closest to every paragraph.
We rely for this process on the TF-IDF metric. TF-IDF stands for time period frequency–inverse doc frequency and is a similarity measure for evaluating two items of textual content. The measure works by computing the variety of comparable phrases, giving extra weight to phrases which seem much less incessantly.
As a preprocessing step, we modify the transcript segments and paragraph beginnings in order that they comprise the identical variety of phrases. The textual content items needs to be lengthy sufficient in order that paragraph beginnings may be efficiently matched to a novel transcript section. We discover that utilizing 50 phrases works nicely in apply.
num_words = 50transcript_num_words = transform_text_segments(transcript, num_words=num_words)
paragraphs_start_text = [{"start": p['paragraph_number'], "textual content": p['paragraph_text']} for p in paragraphs]
paragraphs_num_words = transform_text_segments(paragraphs_start_text, num_words=num_words)
We then depend on the sklearn library and its TfidfVectorizer and cosine_similarity operate to run TF-IDF and compute similarities between every paragraph starting and transcript section. beneath is an instance of code for locating the very best match index within the transcript segments for the primary paragraph.
from sklearn.feature_extraction.textual content import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity# Paragraph for which to search out the timestamp
paragraph_i = 0
# Create a TF-IDF vectorizer
vectorizer = TfidfVectorizer().fit_transform(transcript_num_words + paragraphs_num_words)
# Get the TF-IDF vectors for the transcript and the excerpt
vectors = vectorizer.toarray()
# Extract the TF-IDF vector for the paragraph
paragraph_vector = vectors[len(transcript_num_words) + paragraph_i]
# Calculate the cosine similarity between the paragraph vector and every transcript chunk
similarities = cosine_similarity(vectors[:len(transcript_num_words)], paragraph_vector.reshape(1, -1))
# Discover the index of essentially the most comparable chunk
best_match_index = int(np.argmax(similarities))
We wrapped the method in a add_timestamps_to_paragraphs operate, which provides timestamps to paragraphs, along with the matched section index and textual content:
paragraphs = add_timestamps_to_paragraphs(transcript, paragraphs, num_words=50)#Instance of output for the primary paragraph:
print(paragraphs[0])
{'paragraph_number': 0,
'paragraph_text': "Good afternoon, everybody, and welcome to MIT 6.S191. My title is Alexander Amini, and I will be one among your instructors for the course this yr, together with Ava. We're actually excited to welcome you to this unbelievable course.",
'matched_index': 1,
'matched_text': 'good afternoon everybody and welcome to',
'start_time': 10}
Within the instance above, the primary paragraph (numbered 0) is discovered to match the transcript section no 1 that begins at time 10 (in seconds).
The desk of content material is then discovered by grouping consecutive paragraphs into chapters and figuring out significant chapter titles. The duty is generally carried out by an LLM, which is instructed to remodel an enter consisting in an inventory of JSON paragraphs into an output consisting in an inventory of JSON chapter titles with the beginning paragraph numbers:
system_prompt_paragraphs_to_toc = """You're a useful assistant.
You're given a transcript of a course in JSON format as an inventory of paragraphs, every containing 'paragraph_number' and 'paragraph_text' keys.
Your process is to group consecutive paragraphs in chapters for the course and determine significant chapter titles.
Listed below are the steps to observe:
1. Learn the transcript fastidiously to know its normal construction and the primary subjects coated.
2. Search for clues {that a} new chapter is about to begin. This could possibly be a change of matter, a change of time or setting, the introduction of latest themes or subjects, or the speaker's express point out of a brand new half.
3. For every chapter, maintain observe of the paragraph quantity that begins the chapter and determine a significant chapter title.
4. Chapters ought to ideally be equally spaced all through the transcript, and focus on a selected matter.
Format your end in JSON, with an inventory dictionaries for chapters, with 'start_paragraph_number':integer and 'title':string as key:worth.
Instance:
{"chapters":
[{"start_paragraph_number": 0, "title": "Introduction"},
{"start_paragraph_number": 10, "title": "Chapter 1"}
]
}
"""
An necessary ingredient is to particularly ask for a JSON output, which will increase the possibilities to get a accurately formatted JSON output that may later be loaded again in Python.
GPT-4o-mini is used for this process, as it’s more cost effective than OpenAI’s GPT-4o and usually gives good outcomes. The directions are supplied by the ‘system’ function, and paragraphs are supplied in JSON format by the ‘consumer’ function.
# Hook up with OpenAI with an OpenAI API key
llm_client_get_toc = OpenAI(api_key=api_key)
model_get_toc = "gpt-4o-mini-2024-07-18"# Dump JSON paragraphs as textual content
paragraphs_number_text = [{'paragraph_number': p['paragraph_number'], 'paragraph_text': p['paragraph_text']} for p in paragraphs]
paragraphs_json_dump = json.dumps(paragraphs_number_text)
# Name LLM
response = client_get_toc.chat.completions.create(
messages=[
{
"role": "system",
"content": system_prompt_paragraphs_to_toc
},
{
"role": "user",
"content": paragraphs_json_dump
}
],
mannequin=model_get_toc,
temperature=0,
seed=42
)
Et voilà! The decision returns the record of chapter titles along with the beginning paragraph quantity in JSON format:
print(response){
"chapters": [
{
"start_paragraph_number": 0,
"title": "Introduction to the Course"
},
{
"start_paragraph_number": 17,
"title": "Foundations of Intelligence and Deep Learning"
},
{
"start_paragraph_number": 24,
"title": "Course Structure and Expectations"
}
....
]
}
As in step 2, the LLM might wrestle with lengthy inputs and dismiss a part of the enter. The answer consists once more in splitting the enter into chunks, which is applied within the pocket book with the paragraphs_to_toc operate and the chunk_size parameter.
This final stage combines the paragraphs and the desk of content material to create a structured JSON file with chapters, an instance of which is supplied within the accompanying Github repository.
We illustrate beneath the ensuing chaptering (proper), in comparison with the baseline chaptering that was out there from the YouTube description (left):
