Fixing Xdg-desktop-portal-wlr: Missing Screen Capture Selector

So, you're diving into the world of Wayland compositors like Sway on NixOS, and you're eager to capture your screen with applications like OBS Studio. You hit that "+" button in OBS, ready to add a PipeWire screen capture source, and then... nothing. The expected dmenu selector, which is crucial for choosing what to capture, simply doesn't appear. This is a common frustration, and it often boils down to a missing dependency in your xdg-desktop-portal-wlr setup. Let's break down why this happens and, more importantly, how to fix it, ensuring your screen recording and streaming dreams on NixOS become a reality.

Understanding the xdg-desktop-portal-wlr Dependency

The xdg-desktop-portal framework is the modern way for applications to interact with system-level features like screen capturing, file choosers, and more, especially in Wayland environments. xdg-desktop-portal-wlr is the specific implementation designed for Wayland compositors that follow the wlroots protocol, which includes popular choices like Sway and Hyprland. When an application, like OBS Studio, needs to capture your screen, it asks the xdg-desktop-portal service to facilitate this. This service, in turn, relies on a separate program to present you with a visual choice of which window or screen area you want to capture. The xdg-desktop-portal-wlr code specifically looks for one of a few established menu programs: wmenu, wofi, rofi, or bemenu. If none of these are available in your system's PATH when xdg-desktop-portal-wlr runs, it simply can't present the selector, leaving you with a non-functional screen capture source. This dependency is hardcoded into the portal's logic, as seen in the source code snippet: https://github.com/emersion/xdg-desktop-portal-wlr/blob/main/src/screencast/chooser.c#L223. Debugging this issue involves running the xdg-desktop-portal-wlr service with debug flags enabled, which will often reveal messages about the missing selector program. Once you identify the missing piece, the solution involves ensuring one of the compatible menu programs is installed and accessible to the portal service. This might involve adding it directly to your NixOS configuration or using an overlay to modify the xdg-desktop-portal-wlr package itself to include the necessary dependency and ensure it's correctly wrapped in your PATH.

Reproducing the Missing Selector Bug

To truly understand and fix the problem, you need to be able to reproduce it reliably. The scenario is quite straightforward if you're using a wlroots-based Wayland compositor like Sway, which is a popular choice in the NixOS ecosystem. First, ensure that xdg-desktop-portal-wlr is correctly configured in your Sway setup. A typical configuration in your configuration.nix might look something like this:

xdg = {
  autostart.enable = true;
  portal = {
    config.common.default = "gtk"; # Or your preferred backend
    enable = true;
    wlr.enable = true;
    extraPortals = with pkgs; [ xdg-desktop-portal-gtk ];
  };
};

This configuration enables the xdg-desktop-portal service and specifically activates the wlr backend, ensuring it integrates with your Wayland compositor. It also includes xdg-desktop-portal-gtk as an extra portal, which is common for broader application compatibility. Once this is set up and applied (sudo nixos-rebuild switch), you'll need to install and launch OBS Studio. If you don't already have it, you can add it to your environment.systemPackages in your configuration.nix or install it via nix-env -iA nixos.obs-studio. With OBS Studio running, the crucial step is to add a new source. Click the "+" button in the "Sources" panel and select "PipeWire Window Capture" (or a similar screen capture option). When prompted to "Open selector," clicking this button should, in theory, bring up a menu listing your available windows or screens. However, if the xdg-desktop-portal-wlr is missing one of its required menu dependencies (like wmenu, wofi, rofi, or bemenu), this button will do absolutely nothing. No menu appears, no selection can be made, and consequently, you cannot capture your screen. This lack of response is the key indicator of the bug. If you observe this behavior, you've successfully reproduced the issue, and you're ready to proceed with the fix. The absence of any visual feedback after clicking "open selector" is the definitive sign that the portal service cannot find the necessary tool to present the selection menu.

The Solution: Ensuring Menu Dependencies are Met

Now that we've identified the root cause – the missing menu program dependency – let's talk about the fix. The goal is to make sure that xdg-desktop-portal-wlr can find one of the supported menu applications. There are a couple of ways to approach this within the NixOS environment. The most direct method, as demonstrated in the user's provided overlay, is to modify the xdg-desktop-portal-wlr package itself. This involves using Nix's powerful overlay system to augment the package's build process.

Here's how the overlay works:

1. xdg-desktop-portal-wlr = prev.xdg-desktop-portal-wlr.overrideAttrs (...): This line targets the existing xdg-desktop-portal-wlr package in nixpkgs and allows us to override its attributes, essentially customizing how it's built.
2. nativeBuildInputs = oldAttrs.nativeBuildInputs ++ [ prev.makeWrapper ];: This adds makeWrapper to the build-time dependencies. makeWrapper is a Nix utility used to modify executable scripts or binaries, often to prepend or append paths to their environment variables, like PATH.
3. buildInputs = oldAttrs.buildInputs ++ [ prev.wmenu ];: This is the crucial part where we add the actual menu program dependency. In this example, wmenu is chosen. You could substitute prev.wofi, prev.rofi, or prev.bemenu here if you prefer one of the other supported options. By including it in buildInputs, we ensure wmenu is available during the build process.
4. postInstall = '' ... wrapProgram $out/libexec/xdg-desktop-portal-wlr --prefix PATH : ${lib.makeBinPath [ prev.wmenu ]}'';: This postInstall phase runs after the package has been built and installed. It uses makeWrapper to wrap the main xdg-desktop-portal-wlr executable. The --prefix PATH : ${lib.makeBinPath [ prev.wmenu ]} part is key: it modifies the PATH environment variable for the xdg-desktop-portal-wlr executable so that it can find wmenu (or whichever program you've included) when it runs. lib.makeBinPath is a Nix helper function that creates a colon-separated path string from a list of packages.

To implement this overlay, you would add it to your NixOS configuration, typically within the nixpkgs.overlays section:

nixpkgs.overlays = [
  (self: super: {
    xdg-desktop-portal-wlr = super.xdg-desktop-portal-wlr.overrideAttrs (oldAttrs: {
      nativeBuildInputs = oldAttrs.nativeBuildInputs ++ [ super.makeWrapper ];
      buildInputs = oldAttrs.buildInputs ++ [ super.wmenu ]; # Or wofi, rofi, bemenu
      postInstall = ''
        ${oldAttrs.postInstall or """}
        wrapProgram $out/libexec/xdg-desktop-portal-wlr \
          --prefix PATH : ${self.lib.makeBinPath [ super.wmenu ]}
      '';
    });
  })
];

After applying this configuration (sudo nixos-rebuild switch), the xdg-desktop-portal-wlr service will have the necessary menu program integrated into its execution environment. When OBS Studio requests the screen capture selector, xdg-desktop-portal-wlr will now be able to find and launch wmenu (or your chosen alternative), presenting you with the selection menu as expected. This ensures seamless screen capturing functionality within your Wayland session.

Alternative: System-Wide Installation

While the overlay method is robust and directly addresses the package's dependency, there's a simpler alternative for users who might not need or want to create custom overlays: simply install one of the supported menu programs system-wide. If you ensure that wmenu, wofi, rofi, or bemenu is available in your system's general PATH, xdg-desktop-portal-wlr will be able to find it without needing modification. This is often the easiest approach if you're already using one of these tools for other purposes or are happy to install one just for this functionality.

To do this, you would add your chosen menu program to your environment.systemPackages list in your configuration.nix file. For example, if you decide to use wofi, your configuration would include:

environment.systemPackages = with pkgs;
  [ 
    # ... other packages
    wofi
    # Or rofi, wmenu, bemenu
  ];

After running sudo nixos-rebuild switch, the wofi executable will be available in the system's default PATH. Since xdg-desktop-portal-wlr searches the PATH for compatible menu programs, it will detect wofi and use it when the screen capture selector is invoked. This method is less intrusive than an overlay, as it doesn't modify the xdg-desktop-portal-wlr package itself but rather ensures the necessary external dependency is present in the environment where the portal runs. This approach is particularly user-friendly if you're new to NixOS or prefer to keep your package management straightforward. Both methods achieve the same outcome: enabling the PipeWire screen capture selector to appear correctly when needed, thereby fixing the bug and restoring full screen-casting functionality. The choice between an overlay and a system-wide package installation often comes down to personal preference and the specific customization needs of your NixOS setup.

Conclusion: Enjoying Seamless Screen Captures

Dealing with missing dependencies can sometimes feel like a maze, especially in a declarative system like NixOS. However, understanding how xdg-desktop-portal-wlr selects its screen capture menu program is key to resolving this common issue. Whether you opt for the precise control offered by a Nix overlay to bundle the dependency directly with the portal package or choose the simpler route of installing a compatible menu program like wofi or rofi system-wide, the result is the same: a fully functional screen capture selector. This means you can now confidently use OBS Studio or other applications to record your screen, share your workspace, or stream your content without interruption. NixOS's power lies in its configurability, and fixing issues like this often involves leveraging that very flexibility. By ensuring that xdg-desktop-portal-wlr has access to wmenu, wofi, rofi, or bemenu, you've successfully overcome the hurdle and unlocked the full potential of screen sharing on your Wayland desktop.

For more information on Wayland, screen sharing, and NixOS configurations, you can explore these excellent resources:

• The Arch Wiki on PipeWire: A comprehensive guide to PipeWire, its features, and configuration.
• The Sway Wiki: Detailed information about the Sway window manager and its integration with Wayland.
• NixOS Wiki: The community-driven wiki for NixOS, offering guides, troubleshooting tips, and advanced configurations.