Salesforce AI Introduces FOFPred: A Language-Pushed Future Optical Movement Prediction Framework that Allows Improved Robotic Management and Video Era

Salesforce AI analysis workforce current FOFPred, a language pushed future optical movement prediction framework that connects massive imaginative and prescient language fashions with diffusion transformers for dense movement forecasting in management and video technology settings. FOFPred takes a number of photographs and a pure language instruction akin to ‘transferring the bottle from proper to left’ and predicts 4 future optical movement frames that describe how each pixel is anticipated to maneuver over time.

Future optical movement as a movement illustration

Optical movement is the obvious per pixel displacement between two frames. FOFPred focuses on future optical movement, which implies predicting dense displacement fields for future frames given solely present observations and textual content, with out entry to future photographs at inference.

Future optical movement is a compact movement solely illustration. It removes static look and retains solely pixel degree movement, so it’s effectively suited as an intermediate state for robotic management insurance policies and as a conditioning sign for video diffusion fashions. In comparison with predicting future RGB frames, it reduces the complexity of the output distribution and avoids modeling textures and excessive frequency particulars that aren’t required for movement planning.

To plug into present latent diffusion infrastructure, the analysis workforce encode optical movement as RGB photographs. They map movement magnitude and path from polar type into HSV channels, then convert to RGB. The scaling of every channel is tuned in order that consecutive movement frames are visually easy and resemble animated graphics. A typical Flux.1 variational autoencoder then encodes and decodes these movement photographs.

Unified VLM Diffusion spine

FOFPred makes use of a unified structure that mixes a frozen imaginative and prescient language mannequin, a frozen VAE and a trainable diffusion transformer. The pipeline is:

• Qwen2.5-VL is used because the imaginative and prescient language encoder to collectively encode the caption and visible inputs.
• Flux.1 VAE encodes the enter photographs and the coaching optical movement targets into latent tensors.
• An OmniGen fashion diffusion transformer, DiT, takes projected visible and textual options as conditional inputs and generates latent future movement sequences.

Solely the DiT and small MLP projectors are educated. The Qwen2.5-VL and Flux.1 weights keep frozen, which lets the mannequin reuse picture enhancing pretraining and multimodal reasoning capability from prior work. Temporal modeling is added by extending the RoPE positional encoding and a spotlight blocks from two dimensional spatial positions to full spatio-temporal positions throughout enter and output body sequences. This provides full spatio-temporal consideration with out including further parameters, so the DiT can reuse OmniGen picture pretraining straight.

Coaching on noisy internet movies with relative optical movement

The core mannequin is educated on internet scale human exercise movies with paired captions. The analysis workforce makes use of the One thing One thing V2 dataset and the EgoDex selfish manipulation dataset to acquire round 500,000 video caption pairs.

Coaching makes use of an finish to finish movement matching goal in latent area. Future optical movement sequences are first computed offline, then encoded by the VAE and used as targets in a movement matching diffusion loss for the DiT. Throughout coaching the strategy additionally applies classifier free steering on each textual content and visible circumstances and masks some frames and viewpoints to enhance robustness.

A crucial contribution is the relative optical movement calculation used to construct clear coaching targets from noisy selfish movies. For every body pair the strategy:

1. Computes dense optical movement with an off the shelf estimator.
2. Estimates digital camera movement by way of homography utilizing deep options.
3. Makes use of projective geometry to subtract digital camera movement and acquire object centric relative movement vectors.
4. Filters body pairs by deciding on these the place the highest okay % movement magnitudes exceed a threshold, which focuses coaching on segments with significant movement.

These steps are run offline at decrease decision for effectivity, then recomputed at unique decision for the ultimate targets. The ablation examine reveals that static body targets or uncooked movement with out digital camera movement elimination hurt downstream efficiency, whereas disentangled relative movement targets give the most effective outcomes.

Language pushed robotic manipulation

The primary downstream use case is robotic management. FOFPred is finetuned on robotic video caption knowledge to foretell future optical movement from each fastened and wrist mounted cameras. On prime of FOFPred, the analysis workforce connect a diffusion coverage community that takes predicted movement, textual content and robotic state, and outputs steady actions. This setup follows prior diffusion coverage work however makes use of future optical movement as a substitute of predicted RGB frames because the core illustration.

On the CALVIN ABCD benchmark, which evaluates lengthy horizon zero shot chains of 5 language specified manipulation duties, FOFPred reaches a median chain size of 4.48. VPP reaches 4.33 and DreamVLA reaches 4.44 beneath the identical protocol. FOFPred additionally attains a Job 5 success fee of 78.7 %, which is the most effective amongst reported strategies. In a low knowledge setting with 10 % of CALVIN demonstrations, FOFPred nonetheless reaches 3.43 common size, increased than the three.25 of VPP.

On RoboTwin 2.0, a twin arm manipulation benchmark with 5 duties that require each arms, FOFPred attains a median success fee of 68.6 %. The VPP baseline reaches 61.8 % beneath equivalent coaching settings. FOFPred improves success on each process within the subset.

Movement conscious textual content to video technology

The second downstream process is movement management in textual content to video technology. The analysis workforce construct a two stage pipeline by connecting FOFPred with the Go together with the Movement video diffusion mannequin. FOFPred takes an preliminary body and a language description of movement, predicts a sequence of future movement frames, and interpolates them right into a dense movement area. Go together with the Movement then makes use of this movement area and the preliminary body to synthesize the ultimate video, imposing the described movement sample.

On the movement heavy One thing One thing V2 benchmark, the FOFPred together with Go together with the Movement pipeline improves over the CogVideoX baseline beneath equivalent circumstances. The tactic reaches SSIM 68.4, PSNR 22.26, LPIPS 28.5, FVD 75.39, KVD 11.38, and movement constancy 0.662, that are constantly higher than CogVideoX. Importantly, FOFPred solely makes use of language and a single body at inference, whereas a number of controllable video baselines require hand or object masks or trajectories as further inputs.

Key Take aways

1. FOFPred reframes movement prediction as language pushed future optical movement, predicting 4 dense optical movement frames from a number of present photographs and a textual content instruction, which supplies a compact movement solely illustration for downstream duties.
2. The mannequin makes use of a unified VLM Diffusion spine, with Qwen2.5-VL as a frozen imaginative and prescient language encoder, Flux.1-VAE as a frozen latent encoder for photographs and movement, and an OmniGen fashion DiT as the one educated element with spatio temporal RoPE based mostly consideration.
3. Coaching depends on massive scale internet and selfish video from One thing One thing-V2 and EgoDex, and builds relative optical movement targets by estimating ego-motion by way of homography, subtracting digital camera movement and filtering for prime movement segments, which considerably improves downstream efficiency.
4. In robotic manipulation, FOFPred acts as a movement spine for a diffusion coverage head and achieves cutting-edge or higher outcomes on CALVIN ABCD and RoboTwin 2.0, together with 4.48 common process chain size on CALVIN and 68.6 % common success on RoboTwin, outperforming VPP and DreamVLA variants.
5. For textual content to video technology, connecting FOFPred to Go together with the Movement yields higher SSv2 metrics than CogVideoX, with increased SSIM and PSNR, decrease FVD and KVD, and improved movement constancy, whereas requiring solely language and a single body at inference, making FOFPred a reusable movement controller for each robotics and video synthesis pipelines.

Try the Paper, Model and Repo. Additionally, be happy to observe us on Twitter and don’t overlook to hitch our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Michal Sutter is an information science skilled with a Grasp of Science in Information Science from the College of Padova. With a strong basis in statistical evaluation, machine studying, and knowledge engineering, Michal excels at remodeling advanced datasets into actionable insights.