fluxcoil wiki

What's here?

My attempts to grab HDMI with an addon board for the Raspi worked, but did not give me audio. For this, I ordered this device here, usb id 534d:2109. Works nicely on Linux for up to 720p@60Hz 1080p@30Hz. Latency induced by the grabber is ~130ms, also audio is grabbed. Stereo sound initially required a patch, it's now in the upstream Linux kernel. Michael Lynch has built a tinypilot, a KVM solution from this usb-device and a Raspi.

A summary of the HDMI grabbers I looked at is here.

USB2/1080p@30Hz CamLink, the device I used in this article
- Buying: @amazon for 700円
Potential alternatives:
- USB2/1080p@30Hz Wiistar USB Video Capture, @aliexpress
- USB2/1080p@30Hz CHD201, @soundhouse jp
- USB3/1080p@60Hz Kebidumei, was available on aliexpress, driver was merged into the Linux 5.8 tree
- USB3/1080p@60Hz: Marcan42 had written about this on X/Twitter, @aliexpress, fake 60Hz?
- 1080p@60Hz ezcap311
- 1080p@60Hz CamLink, @amazon
- For pure encoding, raspi alternative: the NVidia Jetson Nano 2GB ($60) has some impressive specs, encoding 4kp@30 live, details
- H2C-RPI-B01 addon board for Raspi/Nvidia Jetson Nano

Configuration

These are the steps to find out which video device is used, and to verify some basic settings.

dnf -y install v4l-utils ffmpeg

# video0 is my thinkpad internal camera
[chris@電脳 ~]$ v4l2-ctl --list-devices
UVC Camera (534d:2109): USB Vid (usb-0000:00:14.0-2):  <-- the usb grabber
        /dev/video2
        /dev/video3

Integrated Camera: Integrated C (usb-0000:00:14.0-7):  <-- internal Thinkpad cam
        /dev/video0
        /dev/video1

# The hdmi grabber provides video2:
[chris@電脳 ~]$ lsusb
[..]
Bus 001 Device 007: ID 534d:2109 MACROSILICON 
[..]
[chris@電脳 ~]$ v4l2-ctl --all -d /dev/video2
Driver Info:
        Driver name      : uvcvideo
        Card type        : UVC Camera (534d:2109): USB Vid
        Bus info         : usb-0000:00:14.0-2
        Driver version   : 5.6.19
        Capabilities     : 0x84a00001
                Video Capture
                Metadata Capture
                Streaming
                Extended Pix Format
                Device Capabilities
        Device Caps      : 0x04200001
                Video Capture
                Streaming
                Extended Pix Format
Priority: 2
Video input : 0 (Camera 1: ok)
Format Video Capture:
        Width/Height      : 1280/720
        Pixel Format      : 'YUYV' (YUYV 4:2:2)
        Field             : None
        Bytes per Line    : 2560
        Size Image        : 1843200
        Colorspace        : sRGB
        Transfer Function : Default (maps to sRGB)
        YCbCr/HSV Encoding: Default (maps to ITU-R 601)
        Quantization      : Default (maps to Limited Range)
        Flags             : 
[..]

# Let's list the supported resolutions:
[chris@電脳 ~]$ ffmpeg -f v4l2 -list_formats all -i /dev/video2
ffmpeg version 4.2.3 Copyright (c) 2000-2020 the FFmpeg developers
[..]
[video4linux2,v4l2 @ 0x558496514600] Compressed:       mjpeg :          Motion-JPEG : 1920x1080 1600x1200 1360x768 1280x1024 1280x960 1280x720 1024x768 800x600 720x576 720x480 640x480
[video4linux2,v4l2 @ 0x558496514600] Raw       :     yuyv422 :           YUYV 4:2:2 : 1920x1080 1600x1200 1360x768 1280x1024 1280x960 1280x720 1024x768 800x600 720x576 720x480 640x480
/dev/video2: Immediate exit requested
[chris@電脳 ~]$

Simple network streaming, no audio

# Server: the process to be started on the Raspi
#   exchange the ip with your clients IP, who should receive the stream
ffmpeg -f v4l2 -input_format yuyv422 -s 1280x720 -r 60 -i /dev/video2 \
  -tune zerolatency -vcodec mpeg2video -f mpegts udp://192.168.0.2:4242

# Note: also do not forget to deal with the firewall on your client,
# it needs to allow incoming packets!

# Client process, exchange with the IP of the system sending the stream:
ffplay -an -sn -i -fflags nobuffer udp://192.168.0.3:4242?listen

Audio test

[chris@電脳 ~]$ arecord -l
**** List of CAPTURE Hardware Devices ****
card 0: PCH [HDA Intel PCH], device 0: ALC257 Analog [ALC257 Analog]
  Subdevices: 0/1
  Subdevice #0: subdevice #0
card 1: U0x534d0x2109 [USB Device 0x534d:0x2109], device 0: USB Audio [USB Audio]
  Subdevices: 0/1
  Subdevice #0: subdevice #0
[chris@電脳 ~]$ 

==> The second device is from our HDMI grabber, good!

[chris@電脳 ~]$ arecord -d 25 -f cd test1.wav

==> This is recording audio from HDMI!

### Setup pure audio streaming for a start, no video.
# server:
[chris@電脳 ~]$ ffmpeg -f alsa -ar 44100 \
  -i hw:1 -acodec mp3 -f mpegts udp://192.168.0.2:4242
# client:
[chris@電脳 ~]$ ffplay -acodec mp3 -i -fflags nobuffer udp://192.168.0.2:4242?listen

==> This grabs audio from the HDMI stream, makes it audible.
    I had to add my user chris to group "audio", to access
    /dev/snd/* devices

Best low latency local streaming, including audio

This streams not over the network, just displays on the system where the usb adapter is plugged in. Optimized for low latency.

### Where is our HDMI grabber soundcard?  Here card 1, device 0:
$ arecord -l
**** List of CAPTURE Hardware Devices ****
card 0: PCH [HDA Intel PCH], device 0: ALC257 Analog [ALC257 Analog]
  Subdevices: 0/1
  Subdevice #0: subdevice #0
card 1: U0x534d0x2109 [USB Device 0x534d:0x2109], device 0: USB Audio [USB Audio]
  Subdevices: 0/1
  Subdevice #0: subdevice #0

# Quickest local streaming is for me with mplayer:
# - here reading from /dev/video2, 720p
# - alsa audio card 1, device 0
 
$ mplayer tv:// -vo gl_nosw -tv \
driver=v4l2:device=/dev/video2:width=1280:height=720:fps=60:outfmt=mjpeg:\
alsa:adevice=hw.1,0:forceaudio:immediatemode=0:audiorate=44100

Latency testing, local

### following gives me 220ms-350ms
### Sends data uncompressed, 15MB/sec on usb
$ ffplay /dev/video2

### following gives me 100-120ms
### Sends data mjpeg compressed, 4MB/sec go over usb
$ xawtv /dev/video2

### following gives me 124ms latency in average (95ms-164ms)
### Sends data mjpeg compressed, 4MB/sec go over usb
$ mplayer tv:// -vo gl_nosw -tv \
  driver=v4l2:device=/dev/video2:width=1280:height=720:fps=60:outfmt=mjpeg
  
### following gives me 400ms latency in average (321ms-474ms)
### Sends data uncompressed, 18.4MB/sec on usb -> maxed out
$ mplayer tv:// -vo gl_nosw -tv \
  driver=v4l2:device=/dev/video2:width=1280:height=720:fps=60:outfmt=yuy2

Streaming via internet

This uses owncast. Following components:

a system on the internet, where we will run Owncast. Should not be behind a NAT, so incoming packets make it to the system. Only requirement: ffmpeg 4.1.5 or later. I use a KVM system running Debian 10, which comes with ffmpeg 4.1.6.
a system where the usb-grabber is connected to, Fedora 33 for me. ffmpeg needs to be installed.
the HDMI source

### on our server system
# https://owncast.online/docs/quickstart/
mkdir /opt/soft/owncast-0.0.2-linux-64bit
cd /opt/soft/owncast-0.0.2-linux-64bit
wget https://github.com/owncast/owncast/releases/download/v0.0.2/owncast-0.0.2-linux-64bit.zip
unzip owncast-0.0.2-linux-64bit.zip
vi config.yaml
# customize at least the streamingKey
./owncast

### on the system where the usb-grabber is connected to
# this assumes /dev/video2 as video device, and that
# hw:1,0 is the alsa device from the grabber
ffmpeg -f alsa -ac 2 -i hw:1,0 -thread_queue_size 64 \
  -f v4l2 -framerate 60 -video_size 1280x720 -input_format yuyv422 -i /dev/video2 \
  -c:v libx264 -preset veryfast -b:v 1984k -maxrate 1984k -bufsize 3968k \
  -vf "format=yuv420p" -g 60 -c:a aac -b:a 128k -ar 44100 \
  -f flv rtmp://<ip-of-your-server>/live/<your-streaming-key>

### on clients
On clients start a browser, and access <server-ip>:8080 .

Latency testing, network

Latency testing with this setup: USB-HDMI-grabber → Raspi4 → Gbit → Fedora desktop. For testing, I send my desktop via HDMI, capture this with the USB-HDMI-grabber, display it eventually on the desktop and measure latency.

latency [1]	command on server	command on client	HDMI latency [2]
0.1ms (GBit)	ssh -X / mplayer / xforwarding [3]	-	~400ms
0.1ms (GBit)	ffmpeg [4]	ffplay [5]	~700ms
0.1ms (GBit)	VLC: cvlc [6]	ffplay [7]	~330ms
0.1ms (GBit)	ustreamer [8]	firefox [9]	~120ms

[1] round trip time over network, between server with usb-grabber connected,
    and the client
[2] Timespan between drawing something in HDMI input, vs. it shown on client
[3] mplayer tv:// -tv driver=v4l2:device=/dev/video1:width=1280:height=720:fps=30:outfmt=yuy2
[4] ffmpeg -f v4l2 -input_format yuyv422 -s 1280x720 -r 60 -i /dev/video2 \
      -tune zerolatency -vcodec mpeg2video -f mpegts udp://192.168.0.2:4242
[5] ffplay -an -sn -i -fflags nobuffer udp://192.168.0.3:4242?listen
[6] cvlc -vvv v4l2:// :v4l2-dev=/dev/video1 :v4l2-width=1280  \
      :v4l2-height=720 :v4l2-fps 25 --sout '#standard{access=http,mux=raw,dst=:8081}'
[7] ffplay -f rawvideo -pixel_format yuyv422 -video_size 1280x720 \
      -framerate 25 http://192.168.0.3:8081
[8] ustreamer --device=/dev/video2 -r 1280x720 --format=jpeg \
      --workers=4 --persistent --drop-same-frames=30 --host=0.0.0.0 --port=8080
[9] firefox http://<ip-of-raspi>:8080/stream

Hints

Building ustreamer

# on Fedora/x86_64
sudo dnf -y install libevent-devel libbsd-devel \
  libjpeg-turbo-devel
git clone --depth=1 https://github.com/pikvm/ustreamer
cd ustreamer/
make

fluxcoil wiki

Sidebar

Table of Contents

What's here?

Configuration

Simple network streaming, no audio

Audio test

Best low latency local streaming, including audio

Latency testing, local

Streaming via internet

Latency testing, network

Hints

Building ustreamer

fluxcoil wiki

User Tools

Site Tools

Sidebar

Table of Contents

What's here?

Configuration

Simple network streaming, no audio

Audio test

Best low latency local streaming, including audio

Latency testing, local

Streaming via internet

Latency testing, network

Hints

Building ustreamer

Page Tools