CV VideoPlayer — As soon as and For All | by Daniel Tomer | Dec, 2024

A Python video participant bundle made for pc imaginative and prescient analysis

When growing pc imaginative and prescient algorithms, the journey from idea to working implementation usually entails numerous iterations of watching, analyzing, and debugging video frames. As I dove deeper into pc imaginative and prescient tasks, I discovered myself repeatedly writing the identical boilerplate code for video visualization and debugging.

In some unspecified time in the future, I made a decision sufficient was sufficient, so I created CV VideoPlayer, a Python-based open-source video participant bundle, particularly designed for pc imaginative and prescient practitioners that can clear up this downside as soon as and for all.

For those who’ve ever developed an algorithm for video evaluation, you’ve most likely written some model of the next code that can assist you visualize and debug it:

import cv2
cap = cv2.VideoCapture(<video_path>)
ret = True
whereas ret:
ret, body = cap.learn()
algo_output = some_video_analsys_algorithm(body)
frame_to_display = visualizer(body, algo_output)
cv2.imshow(frame_to_display)
cv2.waitKey()

However in virtually all tasks I’ve labored on this code was not often sufficient. Because the venture went on I discovered myself including increasingly performance to assist me perceive what was occurring.

For instance:

• Navigation by the video backwards and forwards body by body.
• The flexibility to report the output to a file.
• Supporting sources aside from a easy video file (body folder, stream, distant storage, and many others.)

However the factor that aggravated me essentially the most was the shortage of interactivity. Utilizing this type of code, The visualization is created earlier than rendering and can’t change as soon as displayed. And, whereas that is okay for easy algorithms, for the extra advanced ones, there’s simply means an excessive amount of data wanted for every body. And with out the power to determine, on the fly, what you need to show, you end up operating the identical video time and again, every time with totally different visualization parameters.

This course of was tedious and exhausting.

CV VideoPlayer was born from the necessity for a easy customizable answer for interactively rendering movies and frames. It permits any variety of overlays, sidebars, or another body edits, every of which could be simply switched on and off by the consumer throughout run time. let’s see an instance of how that is completed:

Set up

We begin by putting in the bundle utilizing pip set up cvvideoplayer

Taking part in vanilla video

We are able to then import the video participant and run an unedited video with the next code:

from cvvideoplayer import create_video_player
VIDEO_OR_FRAME_FOLDER_PATH = "<add native path right here>"
video_player = create_video_player(video_source=VIDEO_OR_FRAME_FOLDER_PATH)
video_player.run()

It will open the video participant and help you play it with the spacebar or utilizing the arrows, it’ll additionally add some default built-in frame-edit-callbacks which we’ll elaborate on within the following part.

So as to add custom-built visualization to the video we are able to use the frame_edit_callbacks argument of the create_video_player constructor operate like so:

from cvvideoplayer import VideoPlayer
VIDEO_OR_FRAME_FOLDER_PATH = "<add native path right here>"
video_player = create_video_player(
video_source=VIDEO_OR_FRAME_FOLDER_PATH,
frame_edit_callbacks=[
FitFrameToScreen(),
FrameInfoOverlay(),
KeyMapOverlay(),
]
)
video_player.run()

When unspecified, the default record will likely be precisely the one within the instance above.

Constructed-in callbacks

There are a bunch of built-in callbacks to make use of corresponding to:

• FitFrameToScreen — Routinely resizes the body to suit the display screen measurement.
• FrameInfoOverlay — Prints the body quantity and authentic body decision on the highest left nook.
• KeyMapOverlay — Routinely detects and prints all out there keyboard shortcuts (Additionally these added by the consumer).
• DetectionCsvPlotter — Plots Bounding containers laid out in a CSV with the next Header: frame_id, label, x1, y1, width, top, rating
• FrameNormlizer — Permits the consumer to regulate the dynamic vary of the picture.
• HistogramEqulizer — self-explanatory

And extra are added with every model.

Making a {custom} callback

Right here is the place the usefulness of the bundle shines. So as to add your individual {custom} visualization you create a brand new class that inherits BaseFrameEditCallback and implements the edit_frame methodology, for instance:

class MyCallback(BaseFrameEditCallback):
def __init__(
self,
enable_by_default: bool = True,
enable_disable_key: Elective[str] = None,
additional_keyboard_shortcuts: Elective[List[KeyFunction]] = None
**any_other_needed_params
):
tremendous().__init__(
enable_by_default,
enable_disable_key,
additional_keyboard_shortcuts
)
def edit_frame(
self,
video_player: "VideoPlayer",
body: np.ndarray,
frame_num: int,
original_frame: np.ndarray,
) -> np.ndarray:
"""
This operate receives the displayed body and will return it
after it has been altered in any means fascinating by the consumer
Args:
video_player: an occasion fo VideoPlayer
body (): the body to be edited and displayed
frame_num ():
original_frame () the body earlier than any alterations
Returns: the edited body
"""
body = add_any_visalizations(body)
return body

Moreover, you’ll be able to add setup and teardown strategies by overriding these strategies within the dad or mum class:

class MyCallback(BaseFrameEditCallback):
... 
def setup(self, video_player: "VideoPlayer", body) -> None:
"""
Optionally configure extra parameters in keeping with the
first incoming body
"""
def teardown(self) -> None:
"""
Optionally outline how the callback ought to shut when the
video participant is closed
"""

For every callback, CV Video Participant means that you can add {custom} keyboard shortcuts that may change the visualization it does at run time.

Probably the most primary shortcut is enabling/disabling the callback and is created utilizing the enable_disable_key parameter like so:

my_callback = MyCallback(
enable_disable_key="ctrl+a"
)

The string handed right here could be any mixture of modifiers (ctrl, alt, and shift) with a letter or quantity for instance: “crtl+alt+s”, “g”, “shift+v”, “crtl+1” and so forth.

So as to add shortcuts that change the visualization itself, you’ll be able to override theadditional_keyboard_shortcuts property which returns an inventory of the dataclassKeyFunction .

from cvvideoplayer import KeyFunction
class MyCallback(BaseFrameEditCallback):
...
@property
def additional_keyboard_shortcuts(self) -> Listing[KeyFunction]:
[
KeyFunction(
key="alt+r",
function=self.a_function_to_modify_the_visualiztion,
description="what this does"
)
]

A KeyFunction is constructed utilizing three arguments:

• The key argument — Identical as for enable_disable_key , The string handed right here could be any mixture of modifiers (ctrl, alt, and shift) with a letter or quantity for instance: “crtl+alt+s”, “g”, “shift+v”, “crtl+1”
• The description argument — That is utilized by the KeyMapOverlay callback to print all of the out there shortcuts on the display screen.
• The operate argument — Needs to be a operate that accepts no arguments.

In lots of circumstances, the KeyFunction will obtain a operate that toggles some boolean attribute of the callback, which can change one thing that the edit_framemethodology does. So one thing like:

from cvvideoplayer import KeyFunction
class MyCallback(BaseFrameEditCallback):
...
@property
def additional_keyboard_shortcuts(self) -> Listing[KeyFunction]:
[
KeyFunction(
key="alt+r",
function=self.a_function_to_modify_the_visualiztion,
description="what this does"
)
]
def a_function_to_modify_the_visualiztion():
self._draw_something = bool(1 - self._draw_somthing)

Many occasions, I discovered myself wanting to check two totally different visualizations facet by facet. For instance, evaluating two detectors or an algorithm’s output with the unique body with out modifications, and so forth.

To try this I added double_frame_mode which could be turned on by:

video_player = create_video_player(
...
double_frame_mode=True
)

The video in the beginning of this weblog is an instance of what this mode seems to be like.

On this mode, you should use “ctrl+1” and “ctrl+2″ to determine which body visualization you need to management with the keyboard.

By default, each frames may have the identical callbacks out there however if you would like totally different callbacks for the proper body you should use the right_frame_callback argument to offer the proper body a special set of callbacks (the left body may have those handed to the frame_edit_callback argument):

video_player = create_video_player(
...
double_frame_mode=True
right_frame_callbacks = [callback1, callback2, ...]
)

I Hope this instrument is useful for all of you. When you’ve got any concepts on the way to enhance it, please let me know within the points tab on the venture’s GitHub page, and don’t overlook to depart a star when you’re at it 🙂 …