Permission on Every Message: Notes From Writing a Remote Desktop in Rust

Most remote desktop tools treat permission as a checkbox. You click "allow" once at the start, and after that the other side can do whatever the protocol lets it do.

free-remote is my attempt at something stricter: share a screen and let someone use the mouse and keyboard, but only with permission a person explicitly gave, and only as much as they gave.

The permission model

This is the part that matters, so it comes first.

• Nothing is granted by default. A host with no configuration denies every request unless a person answers the prompt.
• Permissions are per capability. view-screen, mouse and keyboard are separate, so you can let someone watch without letting them touch.
• Every inbound message is checked against the live grant, not just the handshake. Revoking takes effect on the very next event, including events already in the queue.
• Held keys are released when control is revoked, so a viewer that disconnects mid-shortcut can't leave Ctrl stuck down.

There's a test that proves the last point against a real X server: with only view granted, an injected pointer move never reaches the display.

Saying no is the default

When a connection arrives, the host operator sees who is asking and what for, and picks: allow as asked, view only, view plus mouse, or deny.

Pressing enter denies. Not answering denies. An answer it doesn't recognise asks again rather than guessing. If there's no terminal to ask on, the host is closed by default, not open.

The name the other side sends is untrusted text, so control characters, ANSI escapes and bidi overrides are stripped before it's displayed. A hostile peer can't redraw the prompt to fake an approval.

During a session the operator can press v to drop to view-only immediately, or d to disconnect.

Encryption and identity

Every session runs over a Noise XX handshake (Noise_XX_25519_ChaChaPoly_BLAKE2s), which gives mutual authentication and forward secrecy. Both ends see a short fingerprint for the other. Compare them on both screens before approving, or pin the host on the viewer side:

free-remote-viewer 192.168.1.10:7777 --expect-fingerprint <64-hex-from-host>

The unattended password is stored as an Argon2 hash and is never accepted on the command line, so it stays out of ps and shell history.

Making it fast enough

Input was never the problem. Events are tiny and the host sends them ahead of video frames, so they arrive in about one round trip.

Video was the problem. Now each frame is compared with the last one in 128x128 tiles, and only the tiles that changed are encoded, so a still screen costs nothing. Before tiling, an idle desktop cost 31 Mbit/s with 47 ms between frames. After, it was 5.9 Mbit/s and 17.6 ms.

Two more rules keep latency honest under load:

• Frames are dropped, not queued, when the network falls behind, so delay can't build up. A dropped tile update is rolled back and re-sent, so the viewer never keeps a stale patch.
• Capture backs off to 4 fps once the screen has been still for a second. That took idle CPU from about 38% of a core to 6%. Remote input restores the full rate immediately.

How the code is split

fr-proto       wire types, the permission model, message codec
fr-transport   Noise XX handshake, encrypted framing, fingerprints
fr-capture     screen capture per OS, tile diffing, JPEG encoding
fr-input       permission-gated input injection
fr-host        the host binary: approval, streaming, enforcement
fr-viewer      the viewer binary: egui window, input forwarding

It builds on Linux, Windows and macOS with a stable Rust toolchain and nothing else: no system libraries and no C compiler.

Honest limits

There's no clipboard sync, file transfer or audio yet, no NAT traversal, and one controller at a time. Wayland hosting isn't supported. Video uses JPEG tiles rather than a real video codec, which is simple but expensive for full-screen video.

Most importantly, this is not audited software. The cryptography is a standard construction from a well-regarded library, but the way I put it together hasn't had an outside review. Treat it that way.

Source: hoysengleang/free-remoter.