fluxcoil wiki

What's here?

My attempts to grab HDMI with an addon board for the Raspi worked, but did not give me audio. For this, I ordered this device here, usb id 534d:2109. Works nicely on Linux for up to 720p@60Hz 1080p@30Hz. Latency induced by the grabber is ~130ms, also audio is grabbed. Stereo sound initially required a patch (now upstreamed): link alsa-devel. Michael Lynch has built a tinypilot, a KVM solution from this usb-device and a Raspi.

A summary of the HDMI grabbers I looked at is here.

USB2/1080p@30Hz CamLink, the device I used in this article
- I learned about the device from this tweet, Engadget Japan featured the device
- Sources: link now dead - was @aliexpress for 1190円, @amazon for 700円
Potential alternatives:
- USB2/1080p@30Hz Wiistar USB Video Capture, @aliexpress
- USB2/1080p@30Hz CHD201, @soundhouse jp
- USB3/1080p@60Hz Kebidumei link, driver merged into the Linux 5.8 tree
- USB3/1080p@60Hz: alternative @aliexpress, fake 60Hz?
- 1080p@60Hz ezcap311
- 1080p@60Hz CamLink, @amazon, engadget jp on the device
- For pure encoding, raspi alternative: the NVidia Jetson Nano 2GB ($60) has some impressive specs, encoding 4kp@30 live, details
- H2C-RPI-B01 addon board for Raspi/Nvidia Jetson Nano

Configuration

These are the steps to find out which video device is used, and to verify some basic settings.

dnf -y install v4l-utils ffmpeg

# video0 is my thinkpad internal camera
[chris@電脳 ~]$ v4l2-ctl --list-devices
UVC Camera (534d:2109): USB Vid (usb-0000:00:14.0-2):  <-- the usb grabber
        /dev/video2
        /dev/video3

Integrated Camera: Integrated C (usb-0000:00:14.0-7):  <-- internal Thinkpad cam
        /dev/video0
        /dev/video1

# The hdmi grabber provides video2:
[chris@電脳 ~]$ lsusb
[..]
Bus 001 Device 007: ID 534d:2109 MACROSILICON 
[..]
[chris@電脳 ~]$ v4l2-ctl --all -d /dev/video2
Driver Info:
        Driver name      : uvcvideo
        Card type        : UVC Camera (534d:2109): USB Vid
        Bus info         : usb-0000:00:14.0-2
        Driver version   : 5.6.19
        Capabilities     : 0x84a00001
                Video Capture
                Metadata Capture
                Streaming
                Extended Pix Format
                Device Capabilities
        Device Caps      : 0x04200001
                Video Capture
                Streaming
                Extended Pix Format
Priority: 2
Video input : 0 (Camera 1: ok)
Format Video Capture:
        Width/Height      : 1280/720
        Pixel Format      : 'YUYV' (YUYV 4:2:2)
        Field             : None
        Bytes per Line    : 2560
        Size Image        : 1843200
        Colorspace        : sRGB
        Transfer Function : Default (maps to sRGB)
        YCbCr/HSV Encoding: Default (maps to ITU-R 601)
        Quantization      : Default (maps to Limited Range)
        Flags             : 
[..]

# Let's list the supported resolutions:
[chris@電脳 ~]$ ffmpeg -f v4l2 -list_formats all -i /dev/video2
ffmpeg version 4.2.3 Copyright (c) 2000-2020 the FFmpeg developers
[..]
[video4linux2,v4l2 @ 0x558496514600] Compressed:       mjpeg :          Motion-JPEG : 1920x1080 1600x1200 1360x768 1280x1024 1280x960 1280x720 1024x768 800x600 720x576 720x480 640x480
[video4linux2,v4l2 @ 0x558496514600] Raw       :     yuyv422 :           YUYV 4:2:2 : 1920x1080 1600x1200 1360x768 1280x1024 1280x960 1280x720 1024x768 800x600 720x576 720x480 640x480
/dev/video2: Immediate exit requested
[chris@電脳 ~]$

Simple network streaming, no audio

# Server: the process to be started on the Raspi
#   exchange the ip with your clients IP, who should receive the stream
ffmpeg -f v4l2 -input_format yuyv422 -s 1280x720 -r 60 -i /dev/video2 \
  -tune zerolatency -vcodec mpeg2video -f mpegts udp://192.168.0.2:4242

# Note: also do not forget to deal with the firewall on your client,
# it needs to allow incoming packets!

# Client process, exchange with the IP of the system sending the stream:
ffplay -an -sn -i -fflags nobuffer udp://192.168.0.3:4242?listen

Audio test

[chris@電脳 ~]$ arecord -l
**** List of CAPTURE Hardware Devices ****
card 0: PCH [HDA Intel PCH], device 0: ALC257 Analog [ALC257 Analog]
  Subdevices: 0/1
  Subdevice #0: subdevice #0
card 1: U0x534d0x2109 [USB Device 0x534d:0x2109], device 0: USB Audio [USB Audio]
  Subdevices: 0/1
  Subdevice #0: subdevice #0
[chris@電脳 ~]$ 

==> The second device is from our HDMI grabber, good!

[chris@電脳 ~]$ arecord -d 25 -f cd test1.wav

==> This is recording audio from HDMI!

### Setup pure audio streaming for a start, no video.
# server:
[chris@電脳 ~]$ ffmpeg -f alsa -ar 44100 \
  -i hw:1 -acodec mp3 -f mpegts udp://192.168.0.2:4242
# client:
[chris@電脳 ~]$ ffplay -acodec mp3 -i -fflags nobuffer udp://192.168.0.2:4242?listen

==> This grabs audio from the HDMI stream, makes it audible.
    I had to add my user chris to group "audio", to access
    /dev/snd/* devices

Best low latency local streaming, including audio

This streams not over the network, just displays on the system where the usb adapter is plugged in. Optimized for low latency.

### Where is our HDMI grabber soundcard?  Here card 1, device 0:
$ arecord -l
**** List of CAPTURE Hardware Devices ****
card 0: PCH [HDA Intel PCH], device 0: ALC257 Analog [ALC257 Analog]
  Subdevices: 0/1
  Subdevice #0: subdevice #0
card 1: U0x534d0x2109 [USB Device 0x534d:0x2109], device 0: USB Audio [USB Audio]
  Subdevices: 0/1
  Subdevice #0: subdevice #0

# Quickest local streaming is for me with mplayer:
# - here reading from /dev/video0, 720p
# - alsa audio card 1, device 0
 
$ mplayer tv:// -vo gl_nosw -tv \
driver=v4l2:device=/dev/video2:width=1280:height=720:fps=60:outfmt=mjpeg:\
alsa:adevice=hw.1,0:forceaudio:immediatemode=0:audiorate=44100

Latency testing, local

### following gives me 220ms-350ms
### Sends data uncompressed, 15MB/sec on usb
$ ffplay /dev/video2

### following gives me 100-120ms
### Sends data mjpeg compressed, 4MB/sec go over usb
$ xawtv /dev/video2

### following gives me 124ms latency in average (95ms-164ms)
### Sends data mjpeg compressed, 4MB/sec go over usb
$ mplayer tv:// -vo gl_nosw -tv \
  driver=v4l2:device=/dev/video2:width=1280:height=720:fps=60:outfmt=mjpeg
  
### following gives me 400ms latency in average (321ms-474ms)
### Sends data uncompressed, 18.4MB/sec on usb -> maxed out
$ mplayer tv:// -vo gl_nosw -tv \
  driver=v4l2:device=/dev/video2:width=1280:height=720:fps=60:outfmt=yuy2

Streaming via internet

This uses owncast. Following components:

a system on the internet, where we will run Owncast. Should not be behind a NAT, so incoming packets make it to the system. Only requirement: ffmpeg 4.1.5 or later. I use a KVM system running Debian 10, which comes with ffmpeg 4.1.6.
a system where the usb-grabber is connected to, Fedora 33 for me. ffmpeg needs to be installed.
the HDMI source

### on our server system
# https://owncast.online/docs/quickstart/
mkdir /opt/soft/owncast-0.0.2-linux-64bit
cd /opt/soft/owncast-0.0.2-linux-64bit
wget https://github.com/owncast/owncast/releases/download/v0.0.2/owncast-0.0.2-linux-64bit.zip
unzip owncast-0.0.2-linux-64bit.zip
vi config.yaml
# customize at least the streamingKey
./owncast

### on the system where the usb-grabber is connected to
# this assumes /dev/video2 as video device, and that
# hw:1,0 is the alsa device from the grabber
ffmpeg -f alsa -ac 2 -i hw:1,0 -thread_queue_size 64 \
  -f v4l2 -framerate 60 -video_size 1280x720 -input_format yuyv422 -i /dev/video2 \
  -c:v libx264 -preset veryfast -b:v 1984k -maxrate 1984k -bufsize 3968k \
  -vf "format=yuv420p" -g 60 -c:a aac -b:a 128k -ar 44100 \
  -f flv rtmp://<ip-of-your-server>/live/<your-streaming-key>

### on clients
On clients start a browser, and access <server-ip>:8080 .

Latency testing, network

Latency testing with this setup: USB-HDMI-grabber → Raspi4 → Gbit → Fedora desktop. For testing, I send my desktop via HDMI, capture this with the USB-HDMI-grabber, display it eventually on the desktop and measure latency.

latency [1]	command on server	command on client	HDMI latency [2]
0.1ms (GBit)	ssh -X / mplayer / xforwarding [3]	-	~400ms
0.1ms (GBit)	ffmpeg [4]	ffplay [5]	~700ms
0.1ms (GBit)	VLC: cvlc [6]	ffplay [7]	~330ms
0.1ms (GBit)	ustreamer [8]	firefox [9]	~120ms

[1] round trip time over network, between server with usb-grabber connected,
    and the client
[2] Timespan between drawing something in HDMI input, vs. it shown on client
[3] mplayer tv:// -tv driver=v4l2:device=/dev/video1:width=1280:height=720:fps=30:outfmt=yuy2
[4] ffmpeg -f v4l2 -input_format yuyv422 -s 1280x720 -r 60 -i /dev/video2 \
      -tune zerolatency -vcodec mpeg2video -f mpegts udp://192.168.0.2:4242
[5] ffplay -an -sn -i -fflags nobuffer udp://192.168.0.3:4242?listen
[6] cvlc -vvv v4l2:// :v4l2-dev=/dev/video1 :v4l2-width=1280  \
      :v4l2-height=720 :v4l2-fps 25 --sout '#standard{access=http,mux=raw,dst=:8081}'
[7] ffplay -f rawvideo -pixel_format yuyv422 -video_size 1280x720 \
      -framerate 25 http://192.168.0.3:8081
[8] ustreamer --device=/dev/video2 -r 1280x720 --format=jpeg \
      --workers=4 --persistent --drop-same-frames=30 --host=0.0.0.0 --port=8080
[9] firefox http://<ip-of-raspi>:8080/stream

Hints

Building ustreamer

# on Fedora/x86_64
sudo dnf -y install libevent-devel libbsd-devel
git clone --depth=1 https://github.com/pikvm/ustreamer
cd ustreamer/
make

Table of Contents