Table of Contents
What's here?
My attempts to grab HDMI with an addon board for the Raspi worked, but did not give me audio. For this, I ordered this device here, usb id 534d:2109. Works nicely on Linux for up to 720p@60Hz 1080p@30Hz. Latency induced by the grabber is ~130ms, also audio is grabbed. Stereo sound initially required a patch (now upstreamed): link alsa-devel. Michael Lynch has built a tinypilot, a KVM solution from this usb-device and a Raspi.
A summary of the HDMI grabbers I looked at is here.
- USB2/1080p@30Hz CamLink, the device I used in this article
- I learned about the device from this tweet, Engadget Japan featured the device
- Sources: link now dead - was @aliexpress for 1190円, @amazon for 700円
- Potential alternatives:
- USB2/1080p@30Hz Wiistar USB Video Capture, @aliexpress
- USB2/1080p@30Hz CHD201, @soundhouse jp
- USB3/1080p@60Hz Kebidumei link, driver merged into the Linux 5.8 tree
- USB3/1080p@60Hz: alternative @aliexpress, fake 60Hz?
- 1080p@60Hz ezcap311
- 1080p@60Hz CamLink, @amazon, engadget jp on the device
- For pure encoding, raspi alternative: the NVidia Jetson Nano 2GB ($60) has some impressive specs, encoding 4kp@30 live, details
- H2C-RPI-B01 addon board for Raspi/Nvidia Jetson Nano
Configuration
These are the steps to find out which video device is used, and to verify some basic settings.
dnf -y install v4l-utils ffmpeg # video0 is my thinkpad internal camera [chris@電脳 ~]$ v4l2-ctl --list-devices UVC Camera (534d:2109): USB Vid (usb-0000:00:14.0-2): <-- the usb grabber /dev/video2 /dev/video3 Integrated Camera: Integrated C (usb-0000:00:14.0-7): <-- internal Thinkpad cam /dev/video0 /dev/video1 # The hdmi grabber provides video2: [chris@電脳 ~]$ lsusb [..] Bus 001 Device 007: ID 534d:2109 MACROSILICON [..] [chris@電脳 ~]$ v4l2-ctl --all -d /dev/video2 Driver Info: Driver name : uvcvideo Card type : UVC Camera (534d:2109): USB Vid Bus info : usb-0000:00:14.0-2 Driver version : 5.6.19 Capabilities : 0x84a00001 Video Capture Metadata Capture Streaming Extended Pix Format Device Capabilities Device Caps : 0x04200001 Video Capture Streaming Extended Pix Format Priority: 2 Video input : 0 (Camera 1: ok) Format Video Capture: Width/Height : 1280/720 Pixel Format : 'YUYV' (YUYV 4:2:2) Field : None Bytes per Line : 2560 Size Image : 1843200 Colorspace : sRGB Transfer Function : Default (maps to sRGB) YCbCr/HSV Encoding: Default (maps to ITU-R 601) Quantization : Default (maps to Limited Range) Flags : [..] # Let's list the supported resolutions: [chris@電脳 ~]$ ffmpeg -f v4l2 -list_formats all -i /dev/video2 ffmpeg version 4.2.3 Copyright (c) 2000-2020 the FFmpeg developers [..] [video4linux2,v4l2 @ 0x558496514600] Compressed: mjpeg : Motion-JPEG : 1920x1080 1600x1200 1360x768 1280x1024 1280x960 1280x720 1024x768 800x600 720x576 720x480 640x480 [video4linux2,v4l2 @ 0x558496514600] Raw : yuyv422 : YUYV 4:2:2 : 1920x1080 1600x1200 1360x768 1280x1024 1280x960 1280x720 1024x768 800x600 720x576 720x480 640x480 /dev/video2: Immediate exit requested [chris@電脳 ~]$
Simple network streaming, no audio
# Server: the process to be started on the Raspi # exchange the ip with your clients IP, who should receive the stream ffmpeg -f v4l2 -input_format yuyv422 -s 1280x720 -r 60 -i /dev/video2 \ -tune zerolatency -vcodec mpeg2video -f mpegts udp://192.168.0.2:4242 # Note: also do not forget to deal with the firewall on your client, # it needs to allow incoming packets! # Client process, exchange with the IP of the system sending the stream: ffplay -an -sn -i -fflags nobuffer udp://192.168.0.3:4242?listen
Audio test
[chris@電脳 ~]$ arecord -l **** List of CAPTURE Hardware Devices **** card 0: PCH [HDA Intel PCH], device 0: ALC257 Analog [ALC257 Analog] Subdevices: 0/1 Subdevice #0: subdevice #0 card 1: U0x534d0x2109 [USB Device 0x534d:0x2109], device 0: USB Audio [USB Audio] Subdevices: 0/1 Subdevice #0: subdevice #0 [chris@電脳 ~]$ ==> The second device is from our HDMI grabber, good! [chris@電脳 ~]$ arecord -d 25 -f cd test1.wav ==> This is recording audio from HDMI! ### Setup pure audio streaming for a start, no video. # server: [chris@電脳 ~]$ ffmpeg -f alsa -ar 44100 \ -i hw:1 -acodec mp3 -f mpegts udp://192.168.0.2:4242 # client: [chris@電脳 ~]$ ffplay -acodec mp3 -i -fflags nobuffer udp://192.168.0.2:4242?listen ==> This grabs audio from the HDMI stream, makes it audible. I had to add my user chris to group "audio", to access /dev/snd/* devices
Best low latency local streaming, including audio
This streams not over the network, just displays on the system where the usb adapter is plugged in. Optimized for low latency.
### Where is our HDMI grabber soundcard? Here card 1, device 0: $ arecord -l **** List of CAPTURE Hardware Devices **** card 0: PCH [HDA Intel PCH], device 0: ALC257 Analog [ALC257 Analog] Subdevices: 0/1 Subdevice #0: subdevice #0 card 1: U0x534d0x2109 [USB Device 0x534d:0x2109], device 0: USB Audio [USB Audio] Subdevices: 0/1 Subdevice #0: subdevice #0 # Quickest local streaming is for me with mplayer: # - here reading from /dev/video0, 720p # - alsa audio card 1, device 0 $ mplayer tv:// -vo gl_nosw -tv \ driver=v4l2:device=/dev/video2:width=1280:height=720:fps=60:outfmt=mjpeg:\ alsa:adevice=hw.1,0:forceaudio:immediatemode=0:audiorate=44100
Latency testing, local
### following gives me 220ms-350ms ### Sends data uncompressed, 15MB/sec on usb $ ffplay /dev/video2 ### following gives me 100-120ms ### Sends data mjpeg compressed, 4MB/sec go over usb $ xawtv /dev/video2 ### following gives me 124ms latency in average (95ms-164ms) ### Sends data mjpeg compressed, 4MB/sec go over usb $ mplayer tv:// -vo gl_nosw -tv \ driver=v4l2:device=/dev/video2:width=1280:height=720:fps=60:outfmt=mjpeg ### following gives me 400ms latency in average (321ms-474ms) ### Sends data uncompressed, 18.4MB/sec on usb -> maxed out $ mplayer tv:// -vo gl_nosw -tv \ driver=v4l2:device=/dev/video2:width=1280:height=720:fps=60:outfmt=yuy2
Streaming via internet
This uses owncast. Following components:
- a system on the internet, where we will run Owncast. Should not be behind a NAT, so incoming packets make it to the system. Only requirement: ffmpeg 4.1.5 or later. I use a KVM system running Debian 10, which comes with ffmpeg 4.1.6.
- a system where the usb-grabber is connected to, Fedora 33 for me. ffmpeg needs to be installed.
- the HDMI source
### on our server system # https://owncast.online/docs/quickstart/ mkdir /opt/soft/owncast-0.0.2-linux-64bit cd /opt/soft/owncast-0.0.2-linux-64bit wget https://github.com/owncast/owncast/releases/download/v0.0.2/owncast-0.0.2-linux-64bit.zip unzip owncast-0.0.2-linux-64bit.zip vi config.yaml # customize at least the streamingKey ./owncast ### on the system where the usb-grabber is connected to # this assumes /dev/video2 as video device, and that # hw:1,0 is the alsa device from the grabber ffmpeg -f alsa -ac 2 -i hw:1,0 -thread_queue_size 64 \ -f v4l2 -framerate 60 -video_size 1280x720 -input_format yuyv422 -i /dev/video2 \ -c:v libx264 -preset veryfast -b:v 1984k -maxrate 1984k -bufsize 3968k \ -vf "format=yuv420p" -g 60 -c:a aac -b:a 128k -ar 44100 \ -f flv rtmp://<ip-of-your-server>/live/<your-streaming-key> ### on clients On clients start a browser, and access <server-ip>:8080 .
Latency testing, network
Latency testing with this setup: USB-HDMI-grabber → Raspi4 → Gbit → Fedora desktop. For testing, I send my desktop via HDMI, capture this with the USB-HDMI-grabber, display it eventually on the desktop and measure latency.
latency [1] | command on server | command on client | HDMI latency [2] |
---|---|---|---|
0.1ms (GBit) | ssh -X / mplayer / xforwarding [3] | - | ~400ms |
0.1ms (GBit) | ffmpeg [4] | ffplay [5] | ~700ms |
0.1ms (GBit) | VLC: cvlc [6] | ffplay [7] | ~330ms |
0.1ms (GBit) | ustreamer [8] | firefox [9] | ~120ms |
[1] round trip time over network, between server with usb-grabber connected, and the client [2] Timespan between drawing something in HDMI input, vs. it shown on client [3] mplayer tv:// -tv driver=v4l2:device=/dev/video1:width=1280:height=720:fps=30:outfmt=yuy2 [4] ffmpeg -f v4l2 -input_format yuyv422 -s 1280x720 -r 60 -i /dev/video2 \ -tune zerolatency -vcodec mpeg2video -f mpegts udp://192.168.0.2:4242 [5] ffplay -an -sn -i -fflags nobuffer udp://192.168.0.3:4242?listen [6] cvlc -vvv v4l2:// :v4l2-dev=/dev/video1 :v4l2-width=1280 \ :v4l2-height=720 :v4l2-fps 25 --sout '#standard{access=http,mux=raw,dst=:8081}' [7] ffplay -f rawvideo -pixel_format yuyv422 -video_size 1280x720 \ -framerate 25 http://192.168.0.3:8081 [8] ustreamer --device=/dev/video2 -r 1280x720 --format=jpeg \ --workers=4 --persistent --drop-same-frames=30 --host=0.0.0.0 --port=8080 [9] firefox http://<ip-of-raspi>:8080/stream
Hints
Building ustreamer
# on Fedora/x86_64 sudo dnf -y install libevent-devel libbsd-devel git clone --depth=1 https://github.com/pikvm/ustreamer cd ustreamer/ make