===== What's here? =====
My attempts to grab HDMI with [[|an addon board for the Raspi]] worked, but did not give me audio. For this, I ordered this device here, usb id 534d:2109. Works nicely on Linux for up to 720p@60Hz 1080p@30Hz. Latency induced by the grabber is ~130ms, also audio is grabbed. Stereo sound initially required a patch, it's now in the upstream Linux kernel. Michael Lynch has built a [[https://mtlynch.io/tinypilot/|tinypilot, a KVM solution]] from this usb-device and a Raspi.

**A summary of the HDMI grabbers I looked at is [[https://blog.fluxcoil.net/posts/2020/11/linux-hdmi-grabber-1/|here]].**

{{ https://fluxcoil.net/files/tmp/raspi-grabber/20201107_043709411_usb_grabberb.jpg?300x|usb-grabber}}

  * USB2/1080p@30Hz CamLink, the device I used in this article
    * Buying: [[https://www.amazon.co.jp/dp/B088T5M7GX/|@amazon for 700円]]
  * Potential alternatives:
    * USB2/1080p@30Hz Wiistar USB Video Capture, [[https://de.aliexpress.com/item/4000315619046.html|@aliexpress]]
    * USB2/1080p@30Hz CHD201, [[https://www.soundhouse.co.jp/products/detail/item/280483/|@soundhouse jp]]
    * USB3/1080p@60Hz Kebidumei, was available on aliexpress, driver was merged into the Linux 5.8 tree
    * USB3/1080p@60Hz: Marcan42 had written about this on X/Twitter, [[https://www.aliexpress.com/item/4001044872081.html|@aliexpress]], fake 60Hz?
    * 1080p@60Hz ezcap311
    * 1080p@60Hz CamLink, [[https://www.amazon.co.jp/dp/B08DTY1VG8/|@amazon]]
    * For pure encoding, raspi alternative: the NVidia Jetson Nano 2GB ($60) has some impressive specs, encoding 4kp@30 live, [[https://gigazine.net/news/20201006-nvidia-jetson-nano-2gb/|details]]
    * [[hardwarerelated/raspberry_pi_4_tc358743|H2C-RPI-B01]] addon board for Raspi/Nvidia Jetson Nano
===== Configuration =====
These are the steps to find out which video device is used, and to verify some basic settings.
<code>
dnf -y install v4l-utils ffmpeg

# video0 is my thinkpad internal camera
[chris@電脳 ~]$ v4l2-ctl --list-devices
UVC Camera (534d:2109): USB Vid (usb-0000:00:14.0-2):  <-- the usb grabber
        /dev/video2
        /dev/video3

Integrated Camera: Integrated C (usb-0000:00:14.0-7):  <-- internal Thinkpad cam
        /dev/video0
        /dev/video1

# The hdmi grabber provides video2:
[chris@電脳 ~]$ lsusb
[..]
Bus 001 Device 007: ID 534d:2109 MACROSILICON 
[..]
[chris@電脳 ~]$ v4l2-ctl --all -d /dev/video2
Driver Info:
        Driver name      : uvcvideo
        Card type        : UVC Camera (534d:2109): USB Vid
        Bus info         : usb-0000:00:14.0-2
        Driver version   : 5.6.19
        Capabilities     : 0x84a00001
                Video Capture
                Metadata Capture
                Streaming
                Extended Pix Format
                Device Capabilities
        Device Caps      : 0x04200001
                Video Capture
                Streaming
                Extended Pix Format
Priority: 2
Video input : 0 (Camera 1: ok)
Format Video Capture:
        Width/Height      : 1280/720
        Pixel Format      : 'YUYV' (YUYV 4:2:2)
        Field             : None
        Bytes per Line    : 2560
        Size Image        : 1843200
        Colorspace        : sRGB
        Transfer Function : Default (maps to sRGB)
        YCbCr/HSV Encoding: Default (maps to ITU-R 601)
        Quantization      : Default (maps to Limited Range)
        Flags             : 
[..]

# Let's list the supported resolutions:
[chris@電脳 ~]$ ffmpeg -f v4l2 -list_formats all -i /dev/video2
ffmpeg version 4.2.3 Copyright (c) 2000-2020 the FFmpeg developers
[..]
[video4linux2,v4l2 @ 0x558496514600] Compressed:       mjpeg :          Motion-JPEG : 1920x1080 1600x1200 1360x768 1280x1024 1280x960 1280x720 1024x768 800x600 720x576 720x480 640x480
[video4linux2,v4l2 @ 0x558496514600] Raw       :     yuyv422 :           YUYV 4:2:2 : 1920x1080 1600x1200 1360x768 1280x1024 1280x960 1280x720 1024x768 800x600 720x576 720x480 640x480
/dev/video2: Immediate exit requested
[chris@電脳 ~]$ 
</code>

===== Simple network streaming, no audio =====
<code>
# Server: the process to be started on the Raspi
#   exchange the ip with your clients IP, who should receive the stream
ffmpeg -f v4l2 -input_format yuyv422 -s 1280x720 -r 60 -i /dev/video2 \
  -tune zerolatency -vcodec mpeg2video -f mpegts udp://192.168.0.2:4242

# Note: also do not forget to deal with the firewall on your client,
# it needs to allow incoming packets!

# Client process, exchange with the IP of the system sending the stream:
ffplay -an -sn -i -fflags nobuffer udp://192.168.0.3:4242?listen
</code>

===== Audio test =====
  * https://wiki.archlinux.org/index.php/Advanced_Linux_Sound_Architecture
  * https://sysplay.in/blog/linux/2019/06/playing-with-alsa-loopback-devices/
<code>
[chris@電脳 ~]$ arecord -l
**** List of CAPTURE Hardware Devices ****
card 0: PCH [HDA Intel PCH], device 0: ALC257 Analog [ALC257 Analog]
  Subdevices: 0/1
  Subdevice #0: subdevice #0
card 1: U0x534d0x2109 [USB Device 0x534d:0x2109], device 0: USB Audio [USB Audio]
  Subdevices: 0/1
  Subdevice #0: subdevice #0
[chris@電脳 ~]$ 

==> The second device is from our HDMI grabber, good!

[chris@電脳 ~]$ arecord -d 25 -f cd test1.wav

==> This is recording audio from HDMI!

### Setup pure audio streaming for a start, no video.
# server:
[chris@電脳 ~]$ ffmpeg -f alsa -ar 44100 \
  -i hw:1 -acodec mp3 -f mpegts udp://192.168.0.2:4242
# client:
[chris@電脳 ~]$ ffplay -acodec mp3 -i -fflags nobuffer udp://192.168.0.2:4242?listen

==> This grabs audio from the HDMI stream, makes it audible.
    I had to add my user chris to group "audio", to access
    /dev/snd/* devices
</code>

===== Best low latency local streaming, including audio =====
This streams not over the network, just displays on the system where the usb adapter is plugged in. Optimized for low latency.
<code>
### Where is our HDMI grabber soundcard?  Here card 1, device 0:
$ arecord -l
**** List of CAPTURE Hardware Devices ****
card 0: PCH [HDA Intel PCH], device 0: ALC257 Analog [ALC257 Analog]
  Subdevices: 0/1
  Subdevice #0: subdevice #0
card 1: U0x534d0x2109 [USB Device 0x534d:0x2109], device 0: USB Audio [USB Audio]
  Subdevices: 0/1
  Subdevice #0: subdevice #0

# Quickest local streaming is for me with mplayer:
# - here reading from /dev/video2, 720p
# - alsa audio card 1, device 0
 
$ mplayer tv:// -vo gl_nosw -tv \
driver=v4l2:device=/dev/video2:width=1280:height=720:fps=60:outfmt=mjpeg:\
alsa:adevice=hw.1,0:forceaudio:immediatemode=0:audiorate=44100
</code>

===== Latency testing, local =====
<code>
### following gives me 220ms-350ms
### Sends data uncompressed, 15MB/sec on usb
$ ffplay /dev/video2

### following gives me 100-120ms
### Sends data mjpeg compressed, 4MB/sec go over usb
$ xawtv /dev/video2

### following gives me 124ms latency in average (95ms-164ms)
### Sends data mjpeg compressed, 4MB/sec go over usb
$ mplayer tv:// -vo gl_nosw -tv \
  driver=v4l2:device=/dev/video2:width=1280:height=720:fps=60:outfmt=mjpeg
  
### following gives me 400ms latency in average (321ms-474ms)
### Sends data uncompressed, 18.4MB/sec on usb -> maxed out
$ mplayer tv:// -vo gl_nosw -tv \
  driver=v4l2:device=/dev/video2:width=1280:height=720:fps=60:outfmt=yuy2
</code>

===== Streaming via internet =====
This uses [[https://github.com/owncast/owncast|owncast]]. Following components:
  * a system on the internet, where we will run Owncast. Should not be behind a NAT, so incoming packets make it to the system. Only requirement: ffmpeg 4.1.5 or later. I use a KVM system running Debian 10, which comes with ffmpeg 4.1.6.
  * a system where the usb-grabber is connected to, Fedora 33 for me.  ffmpeg needs to be installed.
  * the HDMI source
<code>
### on our server system
# https://owncast.online/docs/quickstart/
mkdir /opt/soft/owncast-0.0.2-linux-64bit
cd /opt/soft/owncast-0.0.2-linux-64bit
wget https://github.com/owncast/owncast/releases/download/v0.0.2/owncast-0.0.2-linux-64bit.zip
unzip owncast-0.0.2-linux-64bit.zip
vi config.yaml
# customize at least the streamingKey
./owncast

### on the system where the usb-grabber is connected to
# this assumes /dev/video2 as video device, and that
# hw:1,0 is the alsa device from the grabber
ffmpeg -f alsa -ac 2 -i hw:1,0 -thread_queue_size 64 \
  -f v4l2 -framerate 60 -video_size 1280x720 -input_format yuyv422 -i /dev/video2 \
  -c:v libx264 -preset veryfast -b:v 1984k -maxrate 1984k -bufsize 3968k \
  -vf "format=yuv420p" -g 60 -c:a aac -b:a 128k -ar 44100 \
  -f flv rtmp://<ip-of-your-server>/live/<your-streaming-key>

### on clients
On clients start a browser, and access <server-ip>:8080 .
</code>

===== Latency testing, network =====
Latency testing with this setup: USB-HDMI-grabber -> Raspi4 -> Gbit -> Fedora desktop. For testing, I send my desktop via HDMI, capture this with the USB-HDMI-grabber, display it eventually on the desktop and measure latency.

^ latency [1]  ^ command on server                  ^ command on client ^ HDMI latency [2] ^
| 0.1ms (GBit) | ssh -X / mplayer / xforwarding [3] | -                 |   ~400ms         |
| 0.1ms (GBit) | ffmpeg [4]                         | ffplay [5]        |   ~700ms         |
| 0.1ms (GBit) | VLC: cvlc [6]                      | ffplay [7]        |   ~330ms         |
| 0.1ms (GBit) | ustreamer [8]                      | firefox [9]       |   ~120ms         |

<code>
[1] round trip time over network, between server with usb-grabber connected,
    and the client
[2] Timespan between drawing something in HDMI input, vs. it shown on client
[3] mplayer tv:// -tv driver=v4l2:device=/dev/video1:width=1280:height=720:fps=30:outfmt=yuy2
[4] ffmpeg -f v4l2 -input_format yuyv422 -s 1280x720 -r 60 -i /dev/video2 \
      -tune zerolatency -vcodec mpeg2video -f mpegts udp://192.168.0.2:4242
[5] ffplay -an -sn -i -fflags nobuffer udp://192.168.0.3:4242?listen
[6] cvlc -vvv v4l2:// :v4l2-dev=/dev/video1 :v4l2-width=1280  \
      :v4l2-height=720 :v4l2-fps 25 --sout '#standard{access=http,mux=raw,dst=:8081}'
[7] ffplay -f rawvideo -pixel_format yuyv422 -video_size 1280x720 \
      -framerate 25 http://192.168.0.3:8081
[8] ustreamer --device=/dev/video2 -r 1280x720 --format=jpeg \
      --workers=4 --persistent --drop-same-frames=30 --host=0.0.0.0 --port=8080
[9] firefox http://<ip-of-raspi>:8080/stream
</code>

===== Hints =====
== Building ustreamer ==
<code>
# on Fedora/x86_64
sudo dnf -y install libevent-devel libbsd-devel \
  libjpeg-turbo-devel
git clone --depth=1 https://github.com/pikvm/ustreamer
cd ustreamer/
make
</code>