Jellyfin: hardware video decoding (jellyfin.hardwareDecoding)

Jellyfin Desktop hard-sets mpv's hwdec=auto-copy, whose probe list leaves
out V4L2 M2M, and the Flatpak can't see the Frame's V4L2 decoder. An
LD_PRELOAD shim (preloaded straight from the store, only that path exposed
read-only) rewrites hwdec to v4l2m2m-copy,auto-copy; devices=all makes the
decoder visible. Overrides via nix-flatpak, or a home-manager-owned override
file without it.

README.md

@@ -3,7 +3,8 @@
 [Home Manager](https://github.com/nix-community/home-manager) modules for the
 Valve Steam Frame (SteamOS, `aarch64-linux`, standalone home-manager). They
 work around quirks of the Frame's two graphical sessions (portal, keyboard
-layout, clipboard, Firefox) and extend Steam's and SteamVR's UIs at runtime
+layout, clipboard, Firefox), enable hardware video decoding in Jellyfin, and
+extend Steam's and SteamVR's UIs at runtime
 (VR keyboard, "+" menu, dashboard windows, Steam close button, window
 curvature, window controls). Everything is declarative and reverts by
 activating an older generation.
@@ -30,7 +31,8 @@ on by default; everything else is opt-in.
   [SteamVR debugger](#steamvr-debugger-steamvrdebuggerenable),
   [hidden apps](#hidden-apps-launchermenuhiddenapps),
   [clipboard sync](#clipboard-sync-clipboardsyncenable),
-  [Firefox](#firefox-firefox)
+  [Firefox](#firefox-firefox),
+  [Jellyfin hardware decoding](#jellyfin-hardware-decoding-jellyfinhardwaredecoding)
 - [Rollback](#rollback)
 
 ## Install
@@ -156,6 +158,7 @@ a commented version of these two files ([`template/`](template)):
       frameControls.enable = true;
     };
     firefox.enable = true;
+    jellyfin.hardwareDecoding.enable = true;
   };
 
   # Example: an app that must use the single wallet on the outer bus.
@@ -176,7 +179,7 @@ home-manager switch --flake .#steamos
 ```
 
 Individual modules:
-`homeManagerModules.{session,portal,keyboard-layout,steam-keyboard-patch,vr-keyboard,hidden-apps,steam-ui-patches,launcher-menu,steamvr-debugger,dashboard-windows,steam-close-button,window-curvature,frame-controls,clipboard-sync,firefox}`;
+`homeManagerModules.{session,portal,keyboard-layout,steam-keyboard-patch,vr-keyboard,hidden-apps,steam-ui-patches,launcher-menu,steamvr-debugger,dashboard-windows,steam-close-button,window-curvature,frame-controls,clipboard-sync,firefox,jellyfin}`;
 `default` imports all.
 
 **Steam Developer Mode** (a Steam setting, not managed here) makes the "+"
@@ -256,6 +259,8 @@ without it.
 | `steamFrame.firefox.enable` | bool | `false` | Launcher for the Flathub Firefox Flatpak with the fixes below. |
 | `steamFrame.firefox.vrFullscreenFix` | bool | `true` | Link a `user.js` with `full-screen-api.ignore-widgets` into profiles. |
 | `steamFrame.firefox.desktopProfile` | null or str | `"desktop"` | Separate profile for the nested desktop; `null`: none. |
+| `steamFrame.jellyfin.hardwareDecoding.enable` | bool | `false` | Hardware video decoding in the Jellyfin Desktop Flatpak, see [Jellyfin](#jellyfin-hardware-decoding-jellyfinhardwaredecoding). |
+| `steamFrame.jellyfin.hardwareDecoding.hwdec` | str | `"v4l2m2m-copy,auto-copy"` | mpv `hwdec` used instead of Jellyfin's automatic one. |
 
 Renamed options still work under their old names, with a warning:
 
@@ -810,6 +815,60 @@ associations keep working.
   second instance stops at the locked profile; in the nested desktop the
   launcher uses this separate profile.
 
+### Jellyfin hardware decoding (`jellyfin.hardwareDecoding.*`)
+
+For the Flathub [Jellyfin Desktop](https://github.com/jellyfin/jellyfin-desktop)
+Flatpak (`org.jellyfin.JellyfinDesktop`), which plays video with libmpv.
+
+**Problem:** video is decoded in software (1080p H.264: ~40 % CPU). The
+Frame's hardware decoder is a V4L2 memory-to-memory device (`qcom-iris`,
+`/dev/video*`), which
+- the Flatpak can't open: its `devices=dri` covers only the GPU, and Flatpak
+  has nothing between that and `devices=all`;
+- mpv never tries: Jellyfin hard-sets `hwdec=auto-copy`, and mpv's `auto`
+  probing leaves out V4L2 M2M on purpose (its quality varies by SoC).
+  Jellyfin has no way to pass mpv options.
+
+**Fix:** a Flatpak override with `devices=all`, and an `LD_PRELOAD` shim that
+rewrites an `hwdec` value starting with `auto` (set through libmpv's
+`mpv_set_*` functions) to `$SFN_MPV_HWDEC` (`hwdec`, set in the override);
+explicit values such as `no` stay. mpv tries the listed decoders in order and
+falls back to software decoding per stream. With the default, 1080p H.264
+plays through `v4l2m2m-copy` at ~15-20 % CPU. The shim (a few libmpv
+wrappers, only libc) is preloaded from the Nix store: the override exposes
+its store path (only that one, read-only) to the sandbox. It would work for
+any libmpv app that sets `hwdec=auto*`, but only Jellyfin Desktop is set up
+here.
+
+The override goes through nix-flatpak's `services.flatpak.overrides` if
+[nix-flatpak](https://github.com/gmodena/nix-flatpak) is imported and
+enabled (merged with your own overrides there). Without it, home-manager
+owns `~/.local/share/flatpak/overrides/org.jellyfin.JellyfinDesktop` (a link
+to the store): overrides of your own for this app belong in Nix then;
+`flatpak override --user org.jellyfin.JellyfinDesktop ...` changes are
+replaced on the next switch.
+
+Changes take effect at the next start of Jellyfin. Log
+(`flatpak run org.jellyfin.JellyfinDesktop` in a terminal):
+`mpv-hwdec-shim: hwdec "auto-copy" -> "v4l2m2m-copy,auto-copy"`, then mpv's
+`Using hardware decoding (v4l2m2m-copy)`.
+
+Installing the Flatpak is up to you, e.g.
+`flatpak install --user flathub org.jellyfin.JellyfinDesktop`, or with
+nix-flatpak:
+
+```nix
+services.flatpak.packages = [ "org.jellyfin.JellyfinDesktop" ];
+```
+
+**Caveats:**
+- `devices=all` gives the app all of `/dev` (cameras, input devices, ...),
+  not just the decoder.
+- V4L2 M2M decoding quality varies with drivers and codecs. Tested: 8-bit
+  H.264; 10-bit HEVC is untested. mpv falls back to software only when the
+  decoder fails; for streams that decode with artifacts, disable the option
+  (or set `hwdec = "auto-copy"`, Jellyfin's own value).
+
 ## Rollback
 
 `home-manager generations` lists previous generations; run

flake.nix

@@ -32,6 +32,7 @@
         imports = [ (import ./modules/clipboard-sync.nix { inherit clipboard-sync-src; }) ];
       };
       firefox = ./modules/firefox.nix;
+      jellyfin = ./modules/jellyfin.nix;
     };
   in {
     homeManagerModules = modules // {
@@ -39,8 +40,10 @@
     };
 
     # Tests of the VR keyboard (text model, corrector, swipe decoder on the
-    # default German + English dictionary): nix flake check
+    # default German + English dictionary) and of the Jellyfin mpv shim:
+    # nix flake check
     checks = nixpkgs.lib.genAttrs [ "aarch64-linux" "x86_64-linux" ] (system: {
+      jellyfin = import ./modules/jellyfin/check.nix { pkgs = nixpkgs.legacyPackages.${system}; };
       vr-keyboard = (import ./modules/vr-keyboard/build.nix {
         pkgs = nixpkgs.legacyPackages.${system};
         dictionary = {

modules/jellyfin.nix

@@ -0,0 +1,74 @@
+# Hardware video decoding for the Jellyfin Desktop Flatpak
+# (org.jellyfin.JellyfinDesktop; installing it is up to the user).
+# The Frame's decoder is a V4L2 M2M device (qcom-iris, /dev/video*): the
+# Flatpak's devices=dri doesn't expose it (nothing finer than devices=all
+# exists), and Jellyfin hard-sets mpv's hwdec=auto-copy, whose probe list
+# leaves out V4L2 M2M, without a way to pass mpv options. So:
+# - an LD_PRELOAD shim (jellyfin/mpv-hwdec-shim.c) rewrites an "auto*" hwdec
+#   to $SFN_MPV_HWDEC. Preloaded from the store: the override exposes just
+#   its store path (read-only) to the sandbox.
+# - override: devices=all, that filesystem, LD_PRELOAD, SFN_MPV_HWDEC.
+#   Through nix-flatpak's services.flatpak.overrides if it is imported and
+#   enabled (merged with the user's own overrides there); else home-manager
+#   owns the app's override file.
+{ config, options, lib, pkgs, ... }:
+let
+  cfg = config.steamFrame.jellyfin.hardwareDecoding;
+  app = "org.jellyfin.JellyfinDesktop";
+
+  shim = pkgs.callPackage ./jellyfin/shim.nix { };
+
+  override = {
+    Context = { devices = [ "all" ]; filesystems = [ "${shim}:ro" ]; };
+    Environment = { LD_PRELOAD = "${shim}/lib/mpv-hwdec-shim.so"; SFN_MPV_HWDEC = cfg.hwdec; };
+  };
+
+  useNixFlatpak = options ? services.flatpak.overrides && config.services.flatpak.enable;
+
+  # Flatpak's keyfile format (lists as "a;b;").
+  overrideFile = pkgs.writeText "${app}-override" (lib.concatStringsSep "\n" (lib.mapAttrsToList
+    (section: entries: "[${section}]\n" + lib.concatStrings (lib.mapAttrsToList (key: value:
+      "${key}=${if lib.isList value then lib.concatMapStrings (v: "${v};") value else value}\n") entries))
+    override));
+in {
+  options.steamFrame.jellyfin.hardwareDecoding = {
+    enable = lib.mkEnableOption ''
+      hardware video decoding (the Frame's V4L2 decoder) in the Jellyfin
+      Desktop Flatpak: device access for the sandbox (devices=all) and an
+      LD_PRELOAD shim that makes mpv try hwdec's value below. Without
+      nix-flatpak, home-manager owns the app's Flatpak override file. Takes
+      effect at the next start of Jellyfin'';
+    hwdec = lib.mkOption {
+      type = lib.types.strMatching "[^[:space:]]+";
+      default = "v4l2m2m-copy,auto-copy";
+      example = "v4l2m2m-copy";
+      description = ''
+        mpv hwdec value used instead of Jellyfin's automatic one ("auto*");
+        tried in order, software decoding if none handles the stream.
+        Set as SFN_MPV_HWDEC in the Flatpak's environment.
+      '';
+    };
+  };
+
+  config = lib.mkMerge [
+    {
+      # Remove the shim copy of earlier versions (copied into the app's data
+      # dir, marked in $XDG_STATE_HOME/steam-frame-nix).
+      home.activation.steamFrameJellyfinHwdec = lib.hm.dag.entryAfter [ "writeBoundary" ] ''
+        marker="''${XDG_STATE_HOME:-$HOME/.local/state}/steam-frame-nix/jellyfin-hwdec-shim"
+        if [ -e "$marker" ]; then
+          run rm -f "$HOME/.var/app/${app}/mpv-hwdec-shim.so" "$marker"
+          run rmdir --ignore-fail-on-non-empty "''${marker%/*}"
+        fi
+      '';
+    }
+    (lib.mkIf (cfg.enable && !useNixFlatpak) {
+      # force: `flatpak override --user` replaces the link with a file.
+      xdg.dataFile."flatpak/overrides/${app}" = { source = overrideFile; force = true; };
+    })
+    # Only if nix-flatpak is imported: the option doesn't exist otherwise.
+    (lib.optionalAttrs (options ? services.flatpak.overrides) {
+      services.flatpak.overrides = lib.mkIf (cfg.enable && useNixFlatpak) { ${app} = override; };
+    })
+  ];
+}

modules/jellyfin/check.nix

@@ -0,0 +1,70 @@
+# Checks of the Jellyfin hardware decoding pieces: the mpv hwdec shim's ELF
+# (libc only, no rpath, GLIBC <= 2.34) and rewriting (against a fake libmpv
+# that prints what it receives).
+{ pkgs }:
+let
+  shim = pkgs.callPackage ./shim.nix { };
+in
+pkgs.runCommandCC "jellyfin-check" { nativeBuildInputs = [ pkgs.binutils ]; } ''
+  so=${shim}/lib/mpv-hwdec-shim.so
+  fail() { echo "FAIL: $*" >&2; exit 1; }
+
+  needed=$(readelf -dW $so | sed -n 's/.*(NEEDED).*\[\(.*\)\]/\1/p')
+  for lib in $needed; do
+    case $lib in libc.so.6|ld-linux*) ;; *) fail "needs $lib" ;; esac
+  done
+  ! readelf -dW $so | grep -Eq '\((RPATH|RUNPATH)\)' || fail "has an rpath"
+  glibc=$(readelf -VW $so | grep -o 'GLIBC_[0-9.]*' | sort -uV | tail -1)
+  [ "$(printf '%s\n' "$glibc" GLIBC_2.34 | sort -V | tail -1)" = GLIBC_2.34 ] ||
+    fail "needs $glibc"
+  echo "ELF ok: NEEDED" $needed", newest $glibc"
+
+  cat > mpv.c <<'C'
+  #include <stdio.h>
+  #include <stdint.h>
+  typedef struct { union { char *string; } u; int format; } mpv_node;
+  static void show(const char *n, int f, void *d) {
+    printf("%s=%s\n", n, f == 1 ? *(char **)d : ((mpv_node *)d)->u.string);
+  }
+  int mpv_set_property(void *c, const char *n, int f, void *d) { show(n, f, d); return 0; }
+  int mpv_set_option(void *c, const char *n, int f, void *d) { show(n, f, d); return 0; }
+  int mpv_set_property_async(void *c, uint64_t u, const char *n, int f, void *d) { show(n, f, d); return 0; }
+  int mpv_set_property_string(void *c, const char *n, const char *d) { printf("%s=%s\n", n, d); return 0; }
+  int mpv_set_option_string(void *c, const char *n, const char *d) { printf("%s=%s\n", n, d); return 0; }
+  C
+  cat > app.c <<'C'
+  #include <stdint.h>
+  typedef struct { union { char *string; } u; int format; } mpv_node;
+  int mpv_set_property(void *, const char *, int, void *);
+  int mpv_set_option(void *, const char *, int, void *);
+  int mpv_set_property_async(void *, uint64_t, const char *, int, void *);
+  int mpv_set_property_string(void *, const char *, const char *);
+  int mpv_set_option_string(void *, const char *, const char *);
+  int main(void) {
+    char *s = "auto-copy", *no = "no";
+    mpv_node node = { { "auto" }, 1 };
+    mpv_set_property_string(0, "hwdec", "auto-copy");
+    mpv_set_option_string(0, "hwdec", "auto");
+    mpv_set_property(0, "hwdec", 1, &s);
+    mpv_set_option(0, "hwdec", 6, &node);
+    mpv_set_property_async(0, 0, "hwdec", 1, &s);
+    mpv_set_property(0, "hwdec", 1, &no);
+    mpv_set_property_string(0, "vo", "auto");
+    return 0;
+  }
+  C
+  $CC -shared -fPIC -o libmpv.so mpv.c
+  $CC -o app app.c -L. -lmpv -Wl,-rpath,$PWD
+
+  h=v4l2m2m-copy,auto-copy
+  expect="hwdec=$h hwdec=$h hwdec=$h hwdec=$h hwdec=$h hwdec=no vo=auto"
+  got=$(LD_PRELOAD=$so ./app 2> err)
+  [ "$(echo $got)" = "$expect" ] || fail "got: $(echo $got)"
+  [ "$(grep -c mpv-hwdec-shim: err)" = 1 ] || fail "logged: $(cat err)"
+  got=$(SFN_MPV_HWDEC=vaapi LD_PRELOAD=$so ./app 2> /dev/null)
+  [ "$(echo $got)" = "$(echo "$expect" | sed "s/$h/vaapi/g")" ] || fail "SFN_MPV_HWDEC: $(echo $got)"
+  # Processes without libmpv (preloaded all over the sandbox) are unaffected.
+  [ "$(LD_PRELOAD=$so ${pkgs.coreutils}/bin/echo ok)" = ok ] || fail "without libmpv"
+  echo "rewrite ok"
+  touch $out
+''

modules/jellyfin/mpv-hwdec-shim.c

@@ -0,0 +1,96 @@
+// LD_PRELOAD shim for libmpv apps that set hwdec to an "auto*" value, such as
+// Jellyfin Desktop (hwdec=auto-copy). mpv's auto probe list leaves out V4L2
+// M2M decoders, so the Steam Frame's hardware decoder (qcom-iris) is never
+// tried. An "auto*" hwdec is rewritten to $SFN_MPV_HWDEC (default below); an
+// explicit value such as "no" is left alone. mpv falls back to software
+// decoding per stream if the decoder can't handle it.
+//
+// The preload applies to every process in the Flatpak sandbox; the wrappers
+// only run when a process calls libmpv, and need nothing but libc.
+#define _GNU_SOURCE
+#include <dlfcn.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#define DEFAULT_HWDEC "v4l2m2m-copy,auto-copy"
+
+// From mpv/client.h (stable ABI).
+typedef struct mpv_handle mpv_handle;
+enum { MPV_FORMAT_STRING = 1, MPV_FORMAT_NODE = 6 };
+enum { MPV_ERROR_GENERIC = -20 };
+typedef struct mpv_node {
+  union { char *string; int flag; int64_t int64; double double_; void *list; void *ba; } u;
+  int format;
+} mpv_node;
+
+// The replacement for an automatic hwdec value, or NULL to keep `value`.
+static const char *hwdec_for(const char *name, const char *value) {
+  if (!name || !value || strcmp(name, "hwdec") != 0 || strncmp(value, "auto", 4) != 0) return NULL;
+  const char *v = getenv("SFN_MPV_HWDEC");
+  if (!v || !*v) v = DEFAULT_HWDEC;
+  static int logged;
+  if (!__atomic_exchange_n(&logged, 1, __ATOMIC_RELAXED))
+    fprintf(stderr, "mpv-hwdec-shim: hwdec \"%s\" -> \"%s\"\n", value, v);
+  return v;
+}
+
+// `data` of an mpv_set_* call in `format`, with a rewritten hwdec in *node or
+// *str if applicable.
+static void *rewrite(const char *name, int format, void *data, mpv_node *node, const char **str) {
+  const char *v;
+  if (!data) return data;
+  if (format == MPV_FORMAT_STRING && (v = hwdec_for(name, *(char **)data))) {
+    *str = v;
+    return str;
+  }
+  if (format == MPV_FORMAT_NODE && ((mpv_node *)data)->format == MPV_FORMAT_STRING &&
+      (v = hwdec_for(name, ((mpv_node *)data)->u.string))) {
+    node->format = MPV_FORMAT_STRING;
+    node->u.string = (char *)v;
+    return node;
+  }
+  return data;
+}
+
+// The real function (libmpv's), looked up once; returns MPV_ERROR_GENERIC if
+// it can't be found (no libmpv in the process: then nothing calls us anyway).
+#define NEXT(fn)                                                     \
+  static __typeof__(fn) *next;                                       \
+  __typeof__(fn) *real = __atomic_load_n(&next, __ATOMIC_RELAXED);   \
+  if (!real) {                                                       \
+    real = (__typeof__(fn) *)dlsym(RTLD_NEXT, #fn);                  \
+    if (!real) return MPV_ERROR_GENERIC;                             \
+    __atomic_store_n(&next, real, __ATOMIC_RELAXED);                 \
+  }
+
+int mpv_set_property(mpv_handle *ctx, const char *name, int format, void *data) {
+  NEXT(mpv_set_property);
+  mpv_node n; const char *s;
+  return real(ctx, name, format, rewrite(name, format, data, &n, &s));
+}
+
+int mpv_set_option(mpv_handle *ctx, const char *name, int format, void *data) {
+  NEXT(mpv_set_option);
+  mpv_node n; const char *s;
+  return real(ctx, name, format, rewrite(name, format, data, &n, &s));
+}
+
+int mpv_set_property_async(mpv_handle *ctx, uint64_t ud, const char *name, int format, void *data) {
+  NEXT(mpv_set_property_async);
+  mpv_node n; const char *s;
+  return real(ctx, ud, name, format, rewrite(name, format, data, &n, &s));
+}
+
+int mpv_set_property_string(mpv_handle *ctx, const char *name, const char *data) {
+  NEXT(mpv_set_property_string);
+  const char *v = hwdec_for(name, data);
+  return real(ctx, name, v ? v : data);
+}
+
+int mpv_set_option_string(mpv_handle *ctx, const char *name, const char *data) {
+  NEXT(mpv_set_option_string);
+  const char *v = hwdec_for(name, data);
+  return real(ctx, name, v ? v : data);
+}

modules/jellyfin/shim.nix

@@ -0,0 +1,9 @@
+# The mpv hwdec shim (mpv-hwdec-shim.c) as lib/mpv-hwdec-shim.so. It runs
+# against the Flatpak runtime's glibc, so it links only libc (no rpath) and
+# must not need symbols newer than GLIBC_2.34 (dlsym's version; see check.nix).
+{ runCommandCC, patchelf }:
+runCommandCC "mpv-hwdec-shim" { nativeBuildInputs = [ patchelf ]; } ''
+  mkdir -p $out/lib
+  $CC -shared -fPIC -O2 -Wall -Wextra -Werror -o $out/lib/mpv-hwdec-shim.so ${./mpv-hwdec-shim.c}
+  patchelf --remove-rpath $out/lib/mpv-hwdec-shim.so
+''

template/home.nix

@@ -54,5 +54,8 @@
   #     frameControls.enable = true;
   #   };
   #   firefox.enable = true;             # launcher for the Firefox Flatpak
+  #   # Hardware video decoding in the Jellyfin Desktop Flatpak (install
+  #   # org.jellyfin.JellyfinDesktop yourself); gives it devices=all:
+  #   jellyfin.hardwareDecoding.enable = true;
   # };
 }