Skip to content

feat: ffmpeg GPU-accelerated encoding path for video acquisition #6

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tabedzki wants to merge 1 commit into
base: feature/video-acquisition
Choose a base branch
from
Open

feat: ffmpeg GPU-accelerated encoding path for video acquisition #6

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 24 additions & 1 deletion extras/RigParameters.m.example
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,11 +47,34 @@ classdef RigParameters
% -----------------------------------------------------------------------
hasCamera = false % master on/off switch
video_parent_path = 'D:\Videos' % root directory for video files
video_ext = '.mj2' % output file extension
video_acquisition_rate = 30 % camera frame rate (fps)
video_exposure_time_in_microseconds = 5000 % exposure time; comment out to use camera default
video_gain = 0 % camera analog gain

% --- Encoder: VideoWriter (default) ---
% No extra dependencies. Produces Motion JPEG 2000 (.mj2) files.
useFFmpeg = false
video_ext = '.mj2'

% --- Encoder: ffmpeg (optional, requires ffmpeg on system PATH) ---
% Set useFFmpeg = true and choose ONE of the two encoder configurations
% below. Falls back to VideoWriter automatically if ffmpeg is unavailable.
%
% NVENC — NVIDIA GPU (Windows, recommended for rigs with NVIDIA cards):
% ffmpegEncoder = 'h264_nvenc'
% ffmpegPreset = 'fast' % 'fast' | 'medium' | 'slow' | 'lossless'
% ffmpegBitrate = '5M' % target bitrate; higher = better quality / larger file
% ffmpegCRF = [] % leave empty — NVENC uses bitrate control, not CRF
%
% libx264 — CPU fallback (any machine with ffmpeg, no GPU required):
% ffmpegEncoder = 'libx264'
% ffmpegPreset = 'fast' % 'ultrafast' | 'fast' | 'medium' | 'slow'
% ffmpegCRF = 23 % 0-51, lower = better quality; 18 ≈ visually lossless
% ffmpegBitrate = [] % leave empty — libx264 uses CRF, not bitrate
%
% Power-user escape hatch (appended verbatim to the ffmpeg command):
% ffmpegExtraArgs = ''

end

methods (Static, Access = protected)
Expand Down
26 changes: 25 additions & 1 deletion sensors/camera/checkCameraRigParameters.m
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,11 @@ function checkCameraRigParameters()
%
% Throws an error listing any missing properties required by
% startVideoAcquisition / configureSingleCamera.
%
% When RigParameters.useFFmpeg is true, also checks that ffmpegEncoder is
% set and that ffmpeg is available on the system PATH (warns and does not
% error if ffmpeg is missing — configureSingleCamera will fall back to
% VideoWriter automatically).

required = {'video_parent_path', 'video_ext', 'video_acquisition_rate', 'video_gain'};

Expand All @@ -15,8 +20,27 @@ function checkCameraRigParameters()

if ~isempty(missing)
error('PureVirmen:missingRigParameters', ...
'Missing required RigParameters for video acquisition: %s\n', ...
'Missing required RigParameters for video acquisition: %s', ...
strjoin(missing, ', '));
end

% ffmpeg-specific checks
if isprop(RigParameters, 'useFFmpeg') && RigParameters.useFFmpeg

if ~isprop(RigParameters, 'ffmpegEncoder') || isempty(RigParameters.ffmpegEncoder)
error('PureVirmen:missingRigParameters', ...
['RigParameters.ffmpegEncoder must be set when useFFmpeg = true.\n' ...
'Use ''h264_nvenc'' (NVIDIA GPU) or ''libx264'' (CPU).']);
end

[status, ~] = system('ffmpeg -version');
if status ~= 0
warning('PureVirmen:noFFmpeg', ...
['ffmpeg not found on system PATH. ' ...
'Recording will fall back to VideoWriter (Motion JPEG 2000). ' ...
'To use ffmpeg, install it and ensure it is on the PATH.']);
end

end

end
257 changes: 243 additions & 14 deletions sensors/camera/configureSingleCamera.m
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,24 +1,49 @@
function v = configureSingleCamera(RigParameters, video_filename)
%CONFIGURESINGLECAMERA Configure a single FLIR camera for disk-logged recording.
%CONFIGURESINGLECAMERA Configure a FLIR camera for disk-logged recording.
%
% v = configureSingleCamera(RigParameters, video_filename)
%
% Configures a FLIR camera via the GenTL adapter (Image Acquisition Toolbox
% + FLIR Spinnaker GenTL support) and sets up a VideoWriter disk logger
% (Motion JPEG 2000). The returned videoinput object v is ready to start;
% call start(v) to begin acquisition.
% Two recording paths are supported, selected by RigParameters.useFFmpeg:
%
% VideoWriter path (default, useFFmpeg = false):
% Configures a FLIR camera via the GenTL adapter and sets up a VideoWriter
% (Motion JPEG 2000) disk logger. Returns a videoinput object.
% Requires: Image Acquisition Toolbox + FLIR Spinnaker GenTL support.
%
% ffmpeg path (useFFmpeg = true):
% Configures the FLIR camera via GenTL and pipes raw frames to an ffmpeg
% subprocess for GPU-accelerated (NVENC) or CPU (libx264) H.264 encoding.
% Returns a struct instead of a videoinput object; pass it directly to
% startVideoAcquisition / stopVideoAcquisition which handle both types.
% Requires: ffmpeg on system PATH.
% See RigParameters.m.example for encoder configuration.
%
% Required RigParameters properties:
% video_acquisition_rate - frame rate in fps
% video_gain - analog gain
%
% Optional RigParameters properties:
% video_exposure_time_in_microseconds - if present and nonzero, sets
% camera exposure; otherwise the camera default is used.
%
% Requires: Image Acquisition Toolbox with FLIR Spinnaker GenTL support.
% Install the support package via MATLAB Add-On Explorer by searching for
% "FLIR Spinnaker".
% video_exposure_time_in_microseconds - exposure; omit to use camera default
% useFFmpeg - logical, default false
% ffmpegEncoder - 'h264_nvenc' (NVIDIA GPU) or 'libx264' (CPU)
% ffmpegPreset - encoder preset string
% ffmpegBitrate - target bitrate string for NVENC (e.g. '5M')
% ffmpegCRF - quality value for libx264 (0-51)
% ffmpegExtraArgs - raw string appended to ffmpeg command

useFFmpeg = isprop(RigParameters, 'useFFmpeg') && RigParameters.useFFmpeg;

if useFFmpeg
v = configureWithFFmpeg(RigParameters, video_filename);
else
v = configureWithVideoWriter(RigParameters, video_filename);
end

end

% -------------------------------------------------------------------------

function v = configureWithVideoWriter(RigParameters, video_filename)

imaqreset;
v = videoinput('gentl', 1, 'Mono8');
Expand All @@ -37,11 +62,215 @@
v.FramesPerTrigger = Inf;
v.TriggerRepeat = Inf;

logfile = VideoWriter(video_filename, 'Motion JPEG 2000');
logfile.MJ2BitDepth = 8;
logfile.FrameRate = RigParameters.video_acquisition_rate;
logfile = VideoWriter(video_filename, 'Motion JPEG 2000');
logfile.MJ2BitDepth = 8;
logfile.FrameRate = RigParameters.video_acquisition_rate;

v.LoggingMode = 'disk';
v.DiskLogger = logfile;

end

% -------------------------------------------------------------------------

function v = configureWithFFmpeg(RigParameters, video_filename)
% Check ffmpeg availability
[status, ~] = system('ffmpeg -version');
if status ~= 0
warning('PureVirmen:noFFmpeg', ...
['ffmpeg not found on PATH. ' ...
'Falling back to VideoWriter (Motion JPEG 2000).']);
v = configureWithVideoWriter(RigParameters, video_filename);
return;
end

% Configure the FLIR camera. Frames are NOT written to disk by IMAQ; they are
% buffered in memory and a FramesAcquiredFcn callback (wired up in
% startVideoAcquisition) drains them and streams raw bytes to ffmpeg's stdin.
imaqreset;
cam = videoinput('gentl', 1, 'Mono8');
src = getselectedsource(cam);

src.AcquisitionFrameRateEnable = 'true';
src.AcquisitionFrameRate = RigParameters.video_acquisition_rate;

if isprop(RigParameters, 'video_exposure_time_in_microseconds') && ...
RigParameters.video_exposure_time_in_microseconds
src.ExposureTime = RigParameters.video_exposure_time_in_microseconds;
end

src.Gain = RigParameters.video_gain;

cam.FramesPerTrigger = Inf;
cam.TriggerRepeat = Inf;
% Buffer frames in memory for the callback to drain (no IMAQ disk logger on
% the ffmpeg path). 'memory' is the IMAQ default but we set it explicitly so
% a recycled/preconfigured object can't leave a stale DiskLogger attached.
cam.LoggingMode = 'memory';

% Build ffmpeg command
vidInfo = cam.VideoResolution; % [width height]
width = vidInfo(1);
height = vidInfo(2);
fps = RigParameters.video_acquisition_rate;
encoder = 'h264_nvenc';
if isprop(RigParameters, 'ffmpegEncoder')
encoder = RigParameters.ffmpegEncoder;
end

encoderArgs = buildEncoderArgs(RigParameters, encoder);

extraArgs = '';
if isprop(RigParameters, 'ffmpegExtraArgs') && ~isempty(RigParameters.ffmpegExtraArgs)
extraArgs = RigParameters.ffmpegExtraArgs;
end

% Replace .mj2 extension with .mp4 if caller passed the wrong extension
[fdir, fname, ~] = fileparts(video_filename);
mp4_filename = fullfile(fdir, [fname '.mp4']);

% Camera delivers single-channel Mono8 (-pix_fmt gray on input). Force a
% widely-compatible output pixel format so the .mp4 plays in standard
% players / H.264 decoders that don't accept grayscale streams.
cmdParts = { 'ffmpeg', '-y', ...
'-f', 'rawvideo', '-pix_fmt', 'gray', ...
'-s', sprintf('%dx%d', width, height), '-r', num2str(fps), ...
'-i', 'pipe:0', ...
'-c:v', encoder };
cmdParts = [cmdParts, splitArgs(encoderArgs), splitArgs(extraArgs), ...
{'-pix_fmt', 'yuv420p', mp4_filename}];

% Launch ffmpeg via Java ProcessBuilder so we get a real handle on its stdin.
% redirectErrorStream keeps stderr from filling the OS pipe buffer (which
% would otherwise deadlock ffmpeg); a drain thread empties it continuously.
try
pb = java.lang.ProcessBuilder(cmdParts);
pb.redirectErrorStream(true);
proc = pb.start();
catch err
delete(cam);
error('PureVirmen:ffmpegLaunchFailed', ...
'Failed to launch ffmpeg.\n Command: %s\n Error: %s', ...
strjoin(cmdParts, ' '), err.message);
end

stdin = proc.getOutputStream(); % we write raw frames here

% Background-drain ffmpeg's merged stdout/stderr so its output buffer never
% fills and stalls encoding. A MATLAB timer polling the reader is portable
% (no custom Java class needed); it stops itself once ffmpeg closes the stream.
logReader = java.io.BufferedReader( ...
java.io.InputStreamReader(proc.getInputStream()));
drainTimer = timer( ...
'ExecutionMode', 'fixedSpacing', ...
'Period', 0.5, ...
'BusyMode', 'drop', ...
'TimerFcn', @(t, ~) drainReader(t, logReader), ...
'Name', 'ffmpegLogDrain');
start(drainTimer);

% Package everything the start/stop functions need. The frame-streaming
% callback is wired up in startVideoAcquisition once acquisition begins.
v = struct();
v.type = 'ffmpeg';
v.cam = cam;
v.proc = proc;
v.stdin = stdin;
v.drainTimer = drainTimer;
v.filename = mp4_filename;
v.width = width;
v.height = height;
v.cmd = strjoin(cmdParts, ' ');

end

% -------------------------------------------------------------------------

function drainReader(t, reader)
% Read and discard all currently-available lines from ffmpeg's output. When
% the stream is closed (ffmpeg exited), readLine returns [] and we stop.
try
while reader.ready()
line = reader.readLine();
if isempty(line)
stop(t);
return;
end
end
catch
stop(t); % stream closed underneath us; nothing more to drain
end
end

% -------------------------------------------------------------------------

function parts = splitArgs(argStr)
% Split a whitespace-separated argument string into a cell array, dropping
% empties. ProcessBuilder needs each token as a separate element (no shell
% to do word-splitting for us).
if isempty(argStr)
parts = {};
return;
end
parts = strsplit(strtrim(argStr));
parts = parts(~cellfun(@isempty, parts));
end

% -------------------------------------------------------------------------

function drainProcessOutput(proc)
% Continuously read and discard ffmpeg's (merged) stdout/stderr on a Java
% thread so its output buffer never fills and stalls encoding. The thread
% exits on its own when ffmpeg closes the stream at shutdown.
reader = java.io.BufferedReader( ...
java.io.InputStreamReader(proc.getInputStream()));
runnable = java.lang.Runnable.empty; %#ok<NASGU>
t = java.lang.Thread(DrainRunnable(reader)); %#ok<NASGU>
% DrainRunnable is unavailable as a Java class in base MATLAB, so fall back
% to a timer that polls the reader if the helper class is absent.
% (Kept simple: use a MATLAB timer to drain.)
end

% -------------------------------------------------------------------------

function args = buildEncoderArgs(RigParameters, encoder)
% Build encoder-specific ffmpeg argument string.
%
% NVENC (h264_nvenc / hevc_nvenc):
% Uses ffmpegBitrate and ffmpegPreset.
% CRF is not supported by NVENC — use bitrate control instead.
%
% libx264 / libx265 (CPU):
% Uses ffmpegCRF and ffmpegPreset.
% Bitrate is not set — CRF gives constant-quality output.

isNvenc = contains(encoder, 'nvenc');
isX264 = contains(encoder, 'libx264') || contains(encoder, 'libx265');

preset = 'fast';
if isprop(RigParameters, 'ffmpegPreset') && ~isempty(RigParameters.ffmpegPreset)
preset = RigParameters.ffmpegPreset;
end

if isNvenc
% NVENC: bitrate-controlled
bitrate = '5M';
if isprop(RigParameters, 'ffmpegBitrate') && ~isempty(RigParameters.ffmpegBitrate)
bitrate = RigParameters.ffmpegBitrate;
end
args = sprintf('-preset %s -b:v %s', preset, bitrate);

elseif isX264
% libx264/libx265: CRF quality-controlled
crf = 23;
if isprop(RigParameters, 'ffmpegCRF') && ~isempty(RigParameters.ffmpegCRF)
crf = RigParameters.ffmpegCRF;
end
args = sprintf('-preset %s -crf %d', preset, crf);

else
% Unknown encoder — pass no extra args; rely on ffmpegExtraArgs if needed
args = '';
end

end
42 changes: 42 additions & 0 deletions sensors/camera/pushFramesToFFmpeg.m
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
function pushFramesToFFmpeg(cam, stdin, width, height) %#ok<INUSD>
%PUSHFRAMESTOFFMPEG Drain buffered camera frames to an ffmpeg stdin stream.
%
% pushFramesToFFmpeg(cam, stdin, width, height)
%
% Used as the FramesAcquiredFcn callback for the ffmpeg recording path
% (see startVideoAcquisition) and called once more at shutdown to flush
% the final buffered frames (see stopVideoAcquisition).
%
% Inputs:
% cam - videoinput object currently acquiring Mono8 frames
% stdin - Java OutputStream connected to ffmpeg's stdin
% width, height - frame dimensions (unused; kept for call-site clarity)
%
% getdata returns frames as height x width x 1 x N (uint8, column-major).
% ffmpeg's rawvideo 'gray' format expects each frame in row-major order, so
% we transpose rows<->cols before serialising. Frames are written one at a
% time to keep memory flat regardless of how many were buffered.

try
nAvail = cam.FramesAvailable;
if nAvail < 1
return;
end
frames = getdata(cam, nAvail); % H x W x 1 x nAvail, uint8
frames = squeeze(frames); % H x W x nAvail (or H x W if nAvail==1)
for k = 1:size(frames, 3)
% Transpose so column-major (:) serialisation yields row-major bytes,
% which is the layout ffmpeg's rawvideo 'gray' input expects.
frameRowMajor = frames(:, :, k)';
stdin.write(typecast(frameRowMajor(:), 'int8'));
end
stdin.flush();
catch err
% A broken pipe means ffmpeg died; surface it so the rig operator knows
% the recording stopped, but don't crash the experiment loop.
warning('PureVirmen:ffmpegWriteFailed', ...
'Failed writing frame to ffmpeg (recording may be incomplete): %s', ...
err.message);
end

end
Loading