diff --git a/.dockerignore b/.dockerignore new file mode 100644 index 0000000..86f871c --- /dev/null +++ b/.dockerignore @@ -0,0 +1,10 @@ +node_modules +dist +dist-web +dist-proxy +dist-electron +release +.git +docs +*.log +.env diff --git a/.gitignore b/.gitignore index 95533f8..f853b22 100644 --- a/.gitignore +++ b/.gitignore @@ -5,3 +5,4 @@ dist-proxy/ dist-electron/ release/ *.log +.env diff --git a/CHANGELOG.md b/CHANGELOG.md index 543a0a8..a52f0ca 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,7 @@ All notable changes to mumh5. Versions follow the `version` in `package.json`. ### Added +- Screen sharing between mumh5 users in a channel: a dialog to choose the screen or window (with preview), the sound and the quality; sound from one program or the whole system on Linux through PipeWire. Viewers watch in the voice tiles, enlarge a stream like in a meeting and set its volume. Streams go directly between clients; a STUN server can be set in Settings, Voice. - Browser version: `npm run build:web` builds the web app and a small self-hosted proxy that bridges browsers to Mumble servers on an allowlist. Identities are kept in the browser, voice goes through the TCP tunnel. See "Browser version" in the README. - Link previews in chat: title, description and image for web links (up to two per message). By default they are fetched by your f0ckm upload host, so the linked sites never see your IP address. Can be switched to "fetched by this computer" or off in Settings, Chat and files. - Voice tiles on small windows: when the window is too narrow for the member list, the people in your voice channel appear as tiles above the chat, light up while they talk, and keep mute and deafen at hand. The tiles can be collapsed. diff --git a/CLAUDE.md b/CLAUDE.md index 9759383..4f79a1a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -26,6 +26,7 @@ npm run build:web # browser build to dist-web/, proxy bundle to dist-proxy/ npm run dev:web # browser build with hot reload: starts the proxy (any server allowed) and Vite npm run proxy # run the proxy from source (MUMH5_SERVERS=host:port required) npm run test:e2e:web # drives the browser build through the proxy (needs npm run build:web first) +npm run test:e2e:share # screen sharing between two instances of the built app npm run proto # regenerate src/core/mumble-pb.js and mumble-udp-pb.js from proto/ ``` @@ -42,10 +43,11 @@ A reused test server keeps registrations and channels from earlier runs; tests m ## Architecture - `electron/` (Node, main process): window, TLS sockets to Mumble servers (`tls-transport.ts`), encrypted UDP voice (`udp-voice.ts`, `ocb2.ts`, tested against Mumble's OCB2 vectors), identities and PKCS#12 (`identity.ts`, `identity-store.ts`), certificate parsing (`certs.ts`), tray (`tray.ts`). The renderer only gets the narrow `window.mumh5Native` API from `preload.ts` (context isolation, sandbox). -- `server/` (Node): the web proxy. `proxy.ts` serves `dist-web`, bridges WebSocket connections to Mumble over TLS (reusing `electron/tls-transport.ts`) and has stateless identity endpoints. It stores nothing; the browser keeps identities in localStorage and sends one with each connect. +- `server/` (Node): the web proxy. `proxy.ts` serves `dist-web`, bridges WebSocket connections to Mumble over TLS (reusing `electron/tls-transport.ts`) and has stateless identity endpoints. It stores nothing; the browser keeps identities in localStorage and sends one with each connect. It also answers STUN on UDP for screen sharing between browser users. - `src/core/` (browser-safe TypeScript, also runs in Node for tests): framing and codec (`proto.ts`), the Mumble client state machine (`client.ts`), voice packet formats (`voice-packet.ts`). No DOM, no Electron, no Node imports here. - `src/lib/native.ts`: `desktop` is the Electron preload API or null; `native` is what both platforms provide (identities, certificates), backed by `web.svelte.ts` in the browser build (`isWeb`, vite `--mode web`). Desktop-only features check `desktop`. - `src/lib/`: app state. `session.svelte.ts` has one `Session` per server plus the `sessions` manager; `session` is a Proxy to the active one. `audio/voice.svelte.ts` is the voice engine (WebCodecs Opus, capture and playback AudioWorklets). `html.ts` sanitizes incoming HTML and serializes outgoing rich text. +- Screen sharing: `src/core/share-signal.ts` (signals as Mumble plugin data, id `mumh5.share`, compressed and chunked) and `src/lib/share.svelte.ts` (one WebRTC connection per viewer, candidates inside the description, no trickle). The start dialog is `ShareDialog.svelte`, the tiles and the large view are `VoiceStage.svelte`. Desktop source listing is `share:sources` / `share:pick` in `electron/main.ts`; Linux sound is `electron/pipewire.ts` (a virtual microphone fed with pw-link, never with this app's own playback). - `src/ui/`: Svelte components. `App.svelte` owns layout and global dialogs (`ui.svelte.ts` store). Voice runs on one server at a time (where you last joined a channel); background servers are auto self-deafened and restored when voice returns. @@ -62,6 +64,7 @@ Voice runs on one server at a time (where you last joined a channel); background - The CSP allows media only from `self`, `blob:` and http(s). `data:` audio is blocked silently: play user files through object URLs. Large user files go to IndexedDB (`src/lib/blobstore.ts`), not localStorage. - `window.prompt` does not exist in Electron; use `ui.prompt`. +- `desktopCapturer.getSources` can return an empty list on its first calls under X11 and takes seconds on the bare Xvfb test display; main retries and the dialog has Refresh. With `--use-fake-device-for-media-stream` the captured picture is Chromium's test pattern, not the screen. On Wayland the call itself opens the system picker, so it is never repeated. - A drop handler must read derived state before clearing the drag item it derives from. ## Svelte reactivity pitfalls (all hit before) @@ -76,6 +79,7 @@ Voice runs on one server at a time (where you last joined a channel); background - Murmur silently drops own UserState, TextMessage, ChannelState, ACL and Version past a leaky bucket (burst 5, 1/s). The client paces these; don't bypass `send()`. - 1.5 servers use the protobuf UDP voice format (type byte 0 + MumbleUDP.Audio) with clients announcing 1.5; older ones the legacy format. The UDPTunnel TCP body is the raw voice packet, not a protobuf message. Sequence numbers count 10 ms frames. - Long comments and descriptions arrive as a hash only; a new hash invalidates the old text; fetch with RequestBlob. +- Murmur drops PluginDataTransmission over 1000 bytes and rate limits it; keep packets at 900 bytes and few. - Renaming while connected is not possible; mumh5 reconnects with the new name. - Murmur never sends SuperUser (user id 0) PermissionQuery answers; treat SuperUser as allowed everything. - In ACLs, Write overrides denies. Grant test rights to one user (`$` group), not `@all`. diff --git a/Dockerfile b/Dockerfile new file mode 100644 index 0000000..7da8a72 --- /dev/null +++ b/Dockerfile @@ -0,0 +1,18 @@ +# The browser version: web app and proxy in one image. +# docker compose up -d --build +FROM node:22-slim AS build +WORKDIR /src +COPY package.json package-lock.json ./ +# No install scripts: the Electron download they would trigger is only needed for the desktop app +RUN npm ci --ignore-scripts +COPY . . +RUN npm run build:web + +FROM node:22-slim +WORKDIR /app +COPY --from=build /src/dist-web ./dist-web +COPY --from=build /src/dist-proxy ./dist-proxy +USER node +ENV MUMH5_PORT=8080 +EXPOSE 8080/tcp 3478/udp +CMD ["node", "dist-proxy/proxy.mjs"] diff --git a/README.md b/README.md index 0124713..2886218 100644 --- a/README.md +++ b/README.md @@ -76,6 +76,15 @@ mumh5 keeps the foundation and replaces the experience. ![Profile with a formatted description](docs/screenshots/profile.png) +### Screen sharing + +- Share a screen or a window with the mumh5 users in your channel. A dialog lets you choose what to show, with a live preview, which sound goes with it and the quality, before anything is sent +- Sound: on Linux one program or everything except mumh5 itself (through PipeWire, so viewers do not hear the voice chat twice); on Windows the whole system; in a browser what the browser offers +- People in the channel see an indicator next to your name, on your tile and in your profile, and click to watch. Your tile shows the picture; a click makes it large with everyone else in a strip below, like a meeting. Viewers set the stream's volume +- No server setup and no extra account: the setup messages travel through the Mumble server, the stream goes directly between the two clients (WebRTC), up to 8 viewers. Regular Mumble clients do not see streams +- Direct connections mean sharer and viewer see each other's IP address; mumh5 says so before the first use. Across the internet both sides need a STUN server. The browser version uses the one built into its proxy; in the desktop app you enter one in Settings, Voice (the proxy's address works there too), and none is contacted unless you do +- Tested between two desktop instances on one machine with a test picture. Sound capture, real screens, connections across the internet, Windows and the browser build are untested + ### Chat - Messages grouped by author, direct messages, unread badges - File sharing: drag and drop, paste, or the attach button, with upload progress @@ -159,6 +168,15 @@ MUMH5_SERVERS="mumble.example.com=My server" node dist-proxy/proxy.mjs Then open `http://127.0.0.1:8080`. For development, `npm run dev:web` starts the proxy and a hot-reloading page together, with any server allowed. To deploy, copy `dist-web/` and `dist-proxy/` next to each other on the server (Node 22 or newer, no `node_modules` needed) and put a reverse proxy with HTTPS in front that forwards WebSocket upgrades. Browsers only allow the microphone on HTTPS pages (or on localhost). +With Docker, the same thing is one command; settings go in a `.env` file next to `docker-compose.yml`: + +```bash +echo 'MUMH5_SERVERS=mumble.example.com=My server' > .env +docker compose up -d --build +``` + +The container uses the host's network: the proxy listens on `127.0.0.1:8080` for your reverse proxy and on UDP 3478 for STUN, and a Mumble server on the same machine is reachable as `localhost`. + | Variable | Default | Meaning | | --- | --- | --- | | `MUMH5_SERVERS` | none | Mumble servers people may connect to: `host[:port][=Label]`, comma-separated. Required unless `MUMH5_ALLOW_ANY=1` | @@ -167,6 +185,7 @@ Then open `http://127.0.0.1:8080`. For development, `npm run dev:web` starts the | `MUMH5_ORIGINS` | same host | Origins allowed to use the API, comma-separated, when the page is hosted elsewhere | | `MUMH5_TRUST_PROXY` | off | Take client addresses from `X-Forwarded-For` (set this behind a reverse proxy) | | `MUMH5_SEND_PROXY` | off | Announce each visitor's address to the server with the PROXY protocol (see below). Breaks connections to a plain Mumble server | +| `MUMH5_STUN_PORT`, `MUMH5_STUN_BIND` | `3478`, all addresses | UDP port of the built-in STUN responder that lets browser users find a direct route for screen sharing. Open this UDP port in the firewall; it does not go through nginx. `0` turns it off | | `MUMH5_STATIC` | `../dist-web` | Folder with the web build | | `MUMH5_MAX_CONNECTIONS`, `MUMH5_MAX_PER_ADDRESS` | `200`, `8` | Connection limits, in total and per client address | @@ -254,7 +273,7 @@ Without an upload host, mumh5 still sends images, scaled to fit the server's lim - System-wide push to talk - Whisper and shout - Rich chat between mumh5 users: replies, reactions, edits, typing indicators -- Screen sharing between mumh5 users in a channel, with sound (one application or the whole system, also on Linux through PipeWire). Anyone can start a stream; no extra key, your Mumble certificate is your identity. Streams go directly between clients with WebRTC, set up through the Mumble server, with an optional self-hosted relay for many viewers +- Screen sharing: several streams at once, and an optional self-hosted relay for many viewers - Release builds for all platforms --- diff --git a/docker-compose.yml b/docker-compose.yml new file mode 100644 index 0000000..461363c --- /dev/null +++ b/docker-compose.yml @@ -0,0 +1,16 @@ +# The mumh5 web proxy. Settings come from a .env file next to this one, for example: +# MUMH5_SERVERS=mumble.example.com=My server +# MUMH5_TRUST_PROXY=1 +services: + mumh5: + build: . + restart: unless-stopped + # Host networking, so STUN sees each visitor's real address (through Docker's port + # forwarding it would often see Docker's own) and a Mumble server on this machine is + # reachable as localhost. The proxy listens on 127.0.0.1:8080 for nginx and on UDP 3478. + network_mode: host + env_file: + - path: .env + required: false + environment: + MUMH5_BIND: ${MUMH5_BIND:-127.0.0.1} diff --git a/electron/main.ts b/electron/main.ts index f3aba65..2c5465e 100644 --- a/electron/main.ts +++ b/electron/main.ts @@ -1,4 +1,4 @@ -import { app, BrowserWindow, dialog, ipcMain, net, shell, session, systemPreferences, type WebContents } from 'electron'; +import { app, BrowserWindow, desktopCapturer, dialog, ipcMain, net, shell, session, systemPreferences, type WebContents } from 'electron'; import { promises as fs } from 'node:fs'; import os from 'node:os'; import path from 'node:path'; @@ -9,9 +9,15 @@ import { fetchLinkPreview } from './link-preview.ts'; import { fetchPublicListWith, pingServer } from './publist.ts'; import { openTls } from './tls-transport.ts'; import { udpChannel } from './udp-voice.ts'; +import { ShareAudio, pipewireAvailable, VIRTUAL_MIC_LABEL, type AudioTarget } from './pipewire.ts'; const devUrl = process.env.VITE_DEV_SERVER_URL; +// Screen sharing connects mumh5 users directly. Chromium would hide this computer's addresses +// behind names that only resolve on the local network, so two people on IPv6 could not reach +// each other without a helper server. Only this app's own page runs here. +app.commandLine.appendSwitch('disable-features', 'WebRtcHideLocalIpsWithMdns'); + let identityStore: IdentityStore | null = null; const identities = () => (identityStore ??= new IdentityStore(app.getPath('userData'))); @@ -120,6 +126,49 @@ ipcMain.handle('publist:ping', (_e, host: string, port: number) => pingServer(St ipcMain.handle('preview:fetch', (_e, url: string) => fetchLinkPreview(String(url).slice(0, 2048))); +// ─── Screen sharing ─────────────────────────────────────────────────────────── +// The renderer lists the sources, the user picks one, and the next getDisplayMedia call gets it. +// On Wayland, listing already opens the system's picker and returns the one chosen source; +// `chosen` tells the renderer not to ask a second time. +let screenSources: Electron.DesktopCapturerSource[] = []; +let pickedSource: string | null = null; +const onWayland = () => process.platform === 'linux' && app.commandLine.getSwitchValue('ozone-platform') !== 'x11' && + (process.env.XDG_SESSION_TYPE === 'wayland' || !!process.env.WAYLAND_DISPLAY); + +// Sound: PipeWire on Linux (one program or everything but this app), system loopback on Windows +const shareAudio = new ShareAudio(() => app.getAppMetrics().map(m => m.pid)); +ipcMain.handle('share:info', async () => ({ + portal: onWayland(), + audio: process.platform === 'win32' ? 'loopback' : (await pipewireAvailable()) ? 'pipewire' : 'none', + micLabel: VIRTUAL_MIC_LABEL +})); +ipcMain.handle('share:audioApps', () => shareAudio.apps().catch(() => [])); +ipcMain.handle('share:audioStart', (_e, target: AudioTarget) => + shareAudio.start(target?.kind === 'app' ? { kind: 'app', name: String(target.name) } : { kind: 'all' })); +ipcMain.handle('share:audioStop', () => shareAudio.stop()); +// The virtual microphone outlives the process otherwise +app.on('will-quit', () => { shareAudio.stop(); }); + +// Names first (fast and dependable), thumbnails in a second call: grabbing the pictures can be +// slow and, under X11, can come back with nothing. +ipcMain.handle('share:sources', async (_e, thumbnails: boolean) => { + const wayland = onWayland(); + const size = thumbnails ? { width: 320, height: 180 } : { width: 0, height: 0 }; + const list = () => desktopCapturer.getSources({ types: ['screen', 'window'], thumbnailSize: size }); + let found = await list(); + // Not on Wayland, where empty means the user cancelled the system's picker and asking again would reopen it + for (let i = 0; i < 3 && !found.length && !wayland; i++) { + await new Promise(r => setTimeout(r, 250)); + found = await list(); + } + if (found.length || !thumbnails) screenSources = found; + return { + chosen: wayland && found.length === 1, + sources: found.map(s => ({ id: s.id, name: s.name, thumbnail: s.thumbnail.isEmpty() ? '' : s.thumbnail.toDataURL() })) + }; +}); +ipcMain.on('share:pick', (_e, id: string) => { pickedSource = String(id); }); + ipcMain.on('tray:update', (_e, state: tray.TrayState) => tray.update(state)); ipcMain.handle('platform:info', () => ({ os: process.platform, osVersion: os.release() })); @@ -195,6 +244,17 @@ app.whenReady().then(async () => { cb({ requestHeaders: details.requestHeaders }); }); + session.defaultSession.setDisplayMediaRequestHandler((request, callback) => { + const source = screenSources.find(s => s.id === pickedSource); + pickedSource = null; + // Without a picked source the request is refused + try { + if (!source) callback({}); + else if (request.audioRequested && process.platform === 'win32') callback({ video: source, audio: 'loopback' }); + else callback({ video: source }); + } catch { /* refused */ } + }); + // Microphone, camera and screen capture for voice and video; nothing else session.defaultSession.setPermissionRequestHandler((_wc, permission, cb) => { cb(['media', 'display-capture', 'clipboard-sanitized-write', 'notifications'].includes(permission)); diff --git a/electron/pipewire.ts b/electron/pipewire.ts new file mode 100644 index 0000000..6e5bffc --- /dev/null +++ b/electron/pipewire.ts @@ -0,0 +1,158 @@ +// Sound for screen sharing on Linux. Chromium cannot capture what other programs play, so a +// virtual microphone is created in PipeWire and the chosen programs' output is linked into it; +// the renderer then records that microphone like any other. Works through PipeWire's command +// line tools (pw-cli, pw-dump, pw-link, pw-mon), no native module. +import { execFile, spawn, type ChildProcess } from 'node:child_process'; + +export const VIRTUAL_MIC = 'mumh5-share-audio'; +export const VIRTUAL_MIC_LABEL = 'mumh5 screen share audio'; + +// What to capture: everything except this app's own sound, or one program +export type AudioTarget = { kind: 'all' } | { kind: 'app'; name: string }; + +interface PwObject { id: number; type: string; info?: { direction?: string; props?: Record } } + +function run(cmd: string, args: string[], maxBuffer = 64 * 1024 * 1024): Promise { + return new Promise((resolve, reject) => { + execFile(cmd, args, { maxBuffer, timeout: 5000 }, (err, stdout) => err ? reject(err) : resolve(stdout)); + }); +} + +const dump = async (): Promise => JSON.parse(await run('pw-dump', [])); + +// A playing program: its name and process come from the stream itself or from the client that owns it +function describe(node: PwObject, byId: Map): { name: string; pid: number | null } { + const p = node.info?.props ?? {}; + const client = byId.get(p['client.id'])?.info?.props ?? {}; + return { + name: String(p['application.name'] ?? client['application.name'] ?? p['node.name'] ?? ''), + pid: Number(p['application.process.id'] ?? client['application.process.id']) || null + }; +} + +const outputs = (objects: PwObject[]) => + objects.filter(o => o.type === 'PipeWire:Interface:Node' && o.info?.props?.['media.class'] === 'Stream/Output/Audio'); + +// Programs currently playing sound, without this app +export function audioApps(objects: PwObject[], ownPids: number[]): string[] { + const byId = new Map(objects.map(o => [o.id, o])); + const names = new Set(); + for (const node of outputs(objects)) { + const d = describe(node, byId); + if (d.name && !(d.pid && ownPids.includes(d.pid))) names.add(d.name); + } + return [...names].sort((a, b) => a.localeCompare(b)); +} + +// Which output ports to connect to which inputs of the virtual microphone, and which existing +// links to remove. This app's own playback is never linked: the people watching are in the same +// voice channel and would hear themselves back. +export function planLinks(objects: PwObject[], target: AudioTarget, ownPids: number[]): { add: [number, number][]; remove: number[] } { + const byId = new Map(objects.map(o => [o.id, o])); + const mic = objects.find(o => o.type === 'PipeWire:Interface:Node' && o.info?.props?.['node.name'] === VIRTUAL_MIC); + if (!mic) return { add: [], remove: [] }; + const ports = objects.filter(o => o.type === 'PipeWire:Interface:Port'); + const micIn = ports.filter(p => p.info?.props?.['node.id'] === mic.id && p.info?.direction === 'input'); + const input = (channel: string) => micIn.find(p => p.info?.props?.['audio.channel'] === channel)?.id; + const left = input('FL'), right = input('FR'); + if (left == null || right == null) return { add: [], remove: [] }; + + const wanted = new Set(); + for (const node of outputs(objects)) { + const d = describe(node, byId); + if (d.pid && ownPids.includes(d.pid)) continue; + if (target.kind === 'app' && d.name !== target.name) continue; + for (const port of ports) { + if (port.info?.props?.['node.id'] !== node.id || port.info?.direction !== 'output') continue; + const channel = port.info.props['audio.channel']; + // Mono programs go to both sides; surround beyond the front pair is left out + if (channel === 'FL' || channel === 'MONO') wanted.add(`${port.id}:${left}`); + if (channel === 'FR' || channel === 'MONO') wanted.add(`${port.id}:${right}`); + } + } + const remove: number[] = []; + for (const link of objects.filter(o => o.type === 'PipeWire:Interface:Link')) { + const info = link.info as any; + if (info?.['input-node-id'] !== mic.id) continue; + const key = `${info['output-port-id']}:${info['input-port-id']}`; + if (wanted.has(key)) wanted.delete(key); + else remove.push(link.id); + } + return { add: [...wanted].map(k => k.split(':').map(Number) as [number, number]), remove }; +} + +// Whether PipeWire and its tools are there +export async function pipewireAvailable(): Promise { + if (process.platform !== 'linux') return false; + try { await dump(); await run('pw-link', ['--version']); await run('pw-cli', ['--version']); return true; } catch { return false; } +} + +export class ShareAudio { + private target: AudioTarget | null = null; + private monitor: ChildProcess | null = null; + private debounce: ReturnType | undefined; + private busy: Promise = Promise.resolve(); + private ownPids: () => number[]; + + constructor(ownPids: () => number[]) { this.ownPids = ownPids; } + + async apps(): Promise { + return audioApps(await dump(), this.ownPids()); + } + + // Creates the microphone, links the target and keeps the links right while programs come and go + async start(target: AudioTarget): Promise { + this.target = target; + if (!(await dump()).some(o => o.info?.props?.['node.name'] === VIRTUAL_MIC)) { + await run('pw-cli', ['create-node', 'adapter', + `{ factory.name=support.null-audio-sink node.name=${VIRTUAL_MIC} node.description="${VIRTUAL_MIC_LABEL}" media.class=Audio/Source/Virtual object.linger=1 audio.position=[FL,FR] audio.rate=48000 audio.channels=2 }`]); + // Give PipeWire a moment to publish the node and its ports + for (let i = 0; i < 20; i++) { + const objects = await dump(); + const mic = objects.find(o => o.info?.props?.['node.name'] === VIRTUAL_MIC); + if (mic && objects.some(o => o.type === 'PipeWire:Interface:Port' && o.info?.props?.['node.id'] === mic.id)) break; + await new Promise(r => setTimeout(r, 100)); + } + } + await this.relink(); + if (!this.monitor) { + const proc = spawn('pw-mon', ['--color=never'], { stdio: ['ignore', 'pipe', 'ignore'] }); + this.monitor = proc; + proc.stdout!.on('data', (data: Buffer) => { + const text = data.toString(); + if (!text.includes('added') && !text.includes('removed')) return; + // Starting a program fires many events at once + clearTimeout(this.debounce); + this.debounce = setTimeout(() => this.relink().catch(() => {}), 400); + }); + proc.on('error', () => { if (this.monitor === proc) this.monitor = null; }); + proc.on('exit', () => { if (this.monitor === proc) this.monitor = null; }); + } + } + + // One at a time, so two graph changes cannot interleave their link commands + private relink(): Promise { + const next = this.busy.then(async () => { + if (!this.target) return; + const { add, remove } = planLinks(await dump(), this.target, this.ownPids()); + for (const id of remove) await run('pw-link', ['-d', String(id)]).catch(() => {}); + // "File exists" when a link is already there is fine + for (const [out, input] of add) await run('pw-link', [String(out), String(input)]).catch(() => {}); + }); + this.busy = next.catch(() => {}); + return next; + } + + async stop(): Promise { + this.target = null; + clearTimeout(this.debounce); + this.monitor?.kill(); + this.monitor = null; + await this.busy; + try { + for (const o of await dump()) { + if (o.type === 'PipeWire:Interface:Node' && o.info?.props?.['node.name'] === VIRTUAL_MIC) await run('pw-cli', ['destroy', String(o.id)]).catch(() => {}); + } + } catch { /* PipeWire gone */ } + } +} diff --git a/electron/preload.ts b/electron/preload.ts index 704d66c..2f1dbc7 100644 --- a/electron/preload.ts +++ b/electron/preload.ts @@ -21,6 +21,12 @@ contextBridge.exposeInMainWorld('mumh5Native', { onContextMenu: (fn: (params: unknown) => void) => { ipcRenderer.on('context-menu', (_e, p) => fn(p)); }, editAction: (action: string, arg?: unknown) => ipcRenderer.send('edit:action', action, arg), onTrayAction: (fn: (action: string) => void) => { ipcRenderer.on('tray:action', (_e, a: string) => fn(a)); }, + screenSources: (thumbnails: boolean) => ipcRenderer.invoke('share:sources', !!thumbnails), + pickScreenSource: (id: string) => ipcRenderer.send('share:pick', id), + shareInfo: () => ipcRenderer.invoke('share:info'), + shareAudioApps: () => ipcRenderer.invoke('share:audioApps'), + shareAudioStart: (target: unknown) => ipcRenderer.invoke('share:audioStart', target), + shareAudioStop: () => ipcRenderer.invoke('share:audioStop'), describeCerts: (ders: Uint8Array[]) => ipcRenderer.invoke('certs:describe', ders), identities: { list: () => ipcRenderer.invoke('identities:list'), diff --git a/package.json b/package.json index 0504d72..78d7e2b 100644 --- a/package.json +++ b/package.json @@ -20,6 +20,7 @@ "dist:mac": "npm run build && electron-builder --mac", "test:e2e": "node test/e2e/app.e2e.ts", "test:e2e:web": "node test/e2e/web.e2e.ts", + "test:e2e:share": "node test/e2e/share.e2e.ts", "proto": "pbjs -t static-module -w es6 --keep-case --no-delimited --no-service --no-comments --force-number proto/Mumble.proto -o src/core/mumble-pb.js && pbjs -t static-module -w es6 --keep-case --no-delimited --no-service --no-comments --force-number proto/MumbleUDP.proto -o src/core/mumble-udp-pb.js", "screenshots": "node test/e2e/screenshots.ts" }, diff --git a/scripts/dev-web.mjs b/scripts/dev-web.mjs index a1f599b..268abd5 100644 --- a/scripts/dev-web.mjs +++ b/scripts/dev-web.mjs @@ -4,7 +4,7 @@ import { createServer } from 'vite'; // The browser build for development: the web proxy on 127.0.0.1:8080 and the Vite dev server, // which forwards /api to it (vite.config.ts). Without MUMH5_SERVERS the proxy allows any // server, private addresses included; it only listens on this machine. -const env = { ...process.env, MUMH5_PORT: '8080', MUMH5_BIND: '127.0.0.1' }; +const env = { MUMH5_STUN_BIND: '127.0.0.1', ...process.env, MUMH5_PORT: '8080', MUMH5_BIND: '127.0.0.1' }; if (!env.MUMH5_SERVERS && !env.MUMH5_ALLOW_ANY) Object.assign(env, { MUMH5_ALLOW_ANY: '1', MUMH5_ALLOW_PRIVATE: '1' }); const proxy = spawn(process.execPath, ['server/main.ts'], { stdio: 'inherit', env }); diff --git a/server/main.ts b/server/main.ts index ddde361..d0ea103 100644 --- a/server/main.ts +++ b/server/main.ts @@ -8,10 +8,11 @@ const config = configFromEnv(process.env); config.staticDir ??= [path.resolve(import.meta.dirname, '../dist-web')].find(d => existsSync(path.join(d, 'index.html'))) ?? null; try { - const { port } = await startProxy(config); + const { port, stunPort } = await startProxy(config); console.log(`mumh5 proxy listening on http://${config.bind}:${port}`); console.log(config.allowAny ? `Allowed servers: any${config.allowPrivate ? ', private addresses included' : ' public address'}` : `Allowed servers: ${config.servers.map(s => `${s.host}:${s.port}`).join(', ')}`); if (config.sendProxy) console.log('Announcing client addresses with the PROXY protocol; the allowed servers must expect it'); + console.log(stunPort ? `STUN for screen sharing on UDP port ${stunPort} (must be reachable from the internet)` : 'STUN is off; screen sharing between browser users will only work on the same network'); console.log(config.staticDir ? `Serving the web app from ${config.staticDir}` : 'No web build found (npm run build:web); serving the API only'); } catch (e) { console.error((e as Error).message); diff --git a/server/proxy.ts b/server/proxy.ts index 1e872ae..b820411 100644 --- a/server/proxy.ts +++ b/server/proxy.ts @@ -4,6 +4,7 @@ import http from 'node:http'; import dns from 'node:dns'; import net from 'node:net'; +import dgram from 'node:dgram'; import path from 'node:path'; import { promises as fs } from 'node:fs'; import { WebSocketServer, type WebSocket } from 'ws'; @@ -29,13 +30,16 @@ export interface ProxyConfig { // Only for servers behind something that understands it (go-mmproxy); plain Mumble does not. sendProxy: boolean; staticDir: string | null; + // UDP port of the built-in STUN responder for screen sharing between browser users; null turns it off + stunPort: number | null; + stunBind: string; maxConnections: number; maxPerAddress: number; } export const defaults: ProxyConfig = { port: 8080, bind: '127.0.0.1', servers: [], allowAny: false, allowPrivate: false, origins: [], - trustProxy: false, sendProxy: false, staticDir: null, maxConnections: 200, maxPerAddress: 8 + trustProxy: false, sendProxy: false, staticDir: null, stunPort: null, stunBind: '::', maxConnections: 200, maxPerAddress: 8 }; // "host", "host:port", "[v6]:port", each optionally followed by "=Label" @@ -64,6 +68,8 @@ export function configFromEnv(env: NodeJS.ProcessEnv): ProxyConfig { trustProxy: on(env.MUMH5_TRUST_PROXY), sendProxy: on(env.MUMH5_SEND_PROXY), staticDir: env.MUMH5_STATIC ?? null, + stunPort: Number(env.MUMH5_STUN_PORT ?? 3478) || null, + stunBind: env.MUMH5_STUN_BIND ?? defaults.stunBind, maxConnections: Number(env.MUMH5_MAX_CONNECTIONS ?? defaults.maxConnections), maxPerAddress: Number(env.MUMH5_MAX_PER_ADDRESS ?? defaults.maxPerAddress) }; @@ -93,6 +99,39 @@ const publicLookup: net.LookupFunction = (hostname, options, callback) => { }); }; +// Answer to a STUN binding request (RFC 5389): tells the sender the address its packet came +// from, which is how two browsers behind routers find a direct route for screen sharing. +// Returns null for anything that is not a binding request. The answer is about as small as +// the request, so the port is of no use for amplifying traffic. +export function stunResponse(msg: Uint8Array, address: string, port: number): Uint8Array | null { + const COOKIE = 0x2112a442; + const view = new DataView(msg.buffer, msg.byteOffset, msg.byteLength); + if (msg.length < 20 || view.getUint16(0) !== 0x0001 || view.getUint32(4) !== COOKIE) return null; + if (view.getUint16(2) !== msg.length - 20) return null; + const v4 = address.replace(/^::ffff:(\d+\.\d+\.\d+\.\d+)$/i, '$1'); + let bytes: number[]; + if (net.isIPv4(v4)) bytes = v4.split('.').map(Number); + else if (net.isIPv6(address)) { + // Expand "::" and write the eight groups out as bytes + const [head, tail = ''] = address.split('%')[0].split('::'); + const h = head ? head.split(':') : [], t = tail ? tail.split(':') : []; + const groups = address.includes('::') ? [...h, ...new Array(8 - h.length - t.length).fill('0'), ...t] : h; + bytes = groups.flatMap(g => { const n = parseInt(g, 16); return [n >> 8, n & 255]; }); + } else return null; + const out = new Uint8Array(20 + 8 + bytes.length); + const o = new DataView(out.buffer); + o.setUint16(0, 0x0101); // binding success + o.setUint16(2, 8 + bytes.length); + out.set(msg.subarray(4, 20), 4); // cookie and transaction id + o.setUint16(20, 0x0020); // XOR-MAPPED-ADDRESS + o.setUint16(22, 4 + bytes.length); + out[25] = bytes.length === 4 ? 1 : 2; + o.setUint16(26, port ^ (COOKIE >>> 16)); + // The address is masked with the cookie, and for IPv6 with the transaction id after it + for (let i = 0; i < bytes.length; i++) out[28 + i] = bytes[i] ^ msg[4 + i]; + return out; +} + const TYPES: Record = { '.html': 'text/html; charset=utf-8', '.js': 'text/javascript; charset=utf-8', '.css': 'text/css; charset=utf-8', '.json': 'application/json', '.png': 'image/png', '.svg': 'image/svg+xml', '.ico': 'image/x-icon', '.jpg': 'image/jpeg', @@ -110,12 +149,13 @@ class HttpError extends Error { constructor(status: number, message: string) { super(message); this.status = status; } } -export async function startProxy(config: ProxyConfig): Promise<{ port: number; close(): Promise }> { +export async function startProxy(config: ProxyConfig): Promise<{ port: number; stunPort: number | null; close(): Promise }> { if (!config.allowAny && !config.servers.length) { throw new Error('No servers allowed. Set MUMH5_SERVERS=host[:port][=Label],... or MUMH5_ALLOW_ANY=1.'); } const staticDir = config.staticDir ? path.resolve(config.staticDir) : null; const perAddress = new Map(); + let stunPort: number | null = null; // Identity requests per address in the current minute; key generation is the costly part const identityUse = new Map(); const sweep = setInterval(() => identityUse.clear(), 60000); @@ -154,7 +194,7 @@ export async function startProxy(config: ProxyConfig): Promise<{ port: number; c async function api(req: http.IncomingMessage, route: string): Promise { if (route === 'config' && req.method === 'GET') { - return { servers: config.servers, any: config.allowAny }; + return { servers: config.servers, any: config.allowAny, stun: stunPort }; } if (req.method !== 'POST') throw new HttpError(404, 'Not found'); if (!originOk(req)) throw new HttpError(403, 'Origin not allowed'); @@ -305,10 +345,27 @@ export async function startProxy(config: ProxyConfig): Promise<{ port: number; c server.once('error', reject); server.listen(config.port, config.bind, resolve); }); + + // STUN on UDP. Failing to open it (port taken, no permission) only costs screen sharing its helper. + let stun: dgram.Socket | null = null; + if (config.stunPort != null) { + const socket = dgram.createSocket(net.isIPv4(config.stunBind) ? 'udp4' : 'udp6'); + socket.on('message', (msg, from) => { + const answer = stunResponse(msg, from.address, from.port); + if (answer) socket.send(answer, from.port, from.address); + }); + await new Promise(resolve => { + socket.once('error', () => { socket.close(); resolve(); }); + socket.bind(config.stunPort!, config.stunBind, () => { socket.removeAllListeners('error'); socket.on('error', () => {}); stun = socket; stunPort = socket.address().port; resolve(); }); + }); + } + return { port: (server.address() as net.AddressInfo).port, + stunPort, close: () => new Promise(resolve => { clearInterval(sweep); + (stun as dgram.Socket | null)?.close(); for (const ws of wss.clients) ws.terminate(); wss.close(); server.close(() => resolve()); diff --git a/src/App.svelte b/src/App.svelte index 82cc44f..eef5bc1 100644 --- a/src/App.svelte +++ b/src/App.svelte @@ -15,6 +15,8 @@ import ChannelDialog from './ui/ChannelDialog.svelte'; import PublicServers from './ui/PublicServers.svelte'; import ServerInfoDialog from './ui/ServerInfoDialog.svelte'; + import ShareDialog from './ui/ShareDialog.svelte'; + import { share } from './lib/share.svelte.ts'; import ResizeHandle from './ui/ResizeHandle.svelte'; import { identities } from './lib/identities.svelte.ts'; import { session } from './lib/session.svelte.ts'; @@ -129,7 +131,8 @@ {#if ui.channelDialog && session.status === 'connected'} {#key ui.channelDialog} (ui.channelDialog = null)} />{/key} {/if} -{#if ui.prompt} (ui.prompt = null)} />{/if} +{#if share.dialog} (share.dialog = null)} />{/if} +{#if ui.prompt} { const p = ui.prompt; ui.prompt = null; p?.oncancel?.(); }} />{/if} diff --git a/src/ui/Chat.svelte b/src/ui/Chat.svelte index 4fc7532..60b2346 100644 --- a/src/ui/Chat.svelte +++ b/src/ui/Chat.svelte @@ -27,7 +27,7 @@ - {#if showStage && session.status === 'connected'}{/if} + {#if session.status === 'connected'}{/if} {#if session.status === 'idle' && session.disconnectInfo} {@const d = session.disconnectInfo} diff --git a/src/ui/RightPanel.svelte b/src/ui/RightPanel.svelte index 61431a1..f8de5b0 100644 --- a/src/ui/RightPanel.svelte +++ b/src/ui/RightPanel.svelte @@ -13,6 +13,8 @@ import { voice } from '../lib/audio/voice.svelte.ts'; import { renderIncoming } from '../lib/html.ts'; import RichEditor from './RichEditor.svelte'; + import { share } from '../lib/share.svelte.ts'; + import { sessions } from '../lib/session.svelte.ts'; let { onnavigate, onclose }: { onnavigate: () => void; onclose: () => void } = $props(); @@ -124,6 +126,16 @@ + {#if session.sharing[user.session]} + + {/if} + {#if isSelf}
{/if} @@ -158,6 +170,9 @@ diff --git a/src/ui/Sidebar.svelte b/src/ui/Sidebar.svelte index 6ccfb94..dbef956 100644 --- a/src/ui/Sidebar.svelte +++ b/src/ui/Sidebar.svelte @@ -9,6 +9,7 @@ import { PERM } from '../core/client.ts'; import ResizeHandle from './ResizeHandle.svelte'; import { voice } from '../lib/audio/voice.svelte.ts'; + import { share } from '../lib/share.svelte.ts'; // resizable: false in the phone drawer, which keeps a fixed width // handleEdge: which side the resize handle sits on (left when the list is on the right, classic layout) @@ -50,6 +51,14 @@ ui.panelOpen = true; } + // Sharing runs on one server at a time; the button stops it from anywhere + const sharingHere = $derived(!!share.stream); + const canShare = $derived(session.status === 'connected' && share.supported(sessions.active)); + function toggleShare() { + if (share.stream) share.stop(); + else share.start(sessions.active); + } + // One button for the whole tree: open everything, then close everything let allOpen = $state(false); function toggleAll() { @@ -159,6 +168,10 @@ onclick={() => session.setSelfDeaf(!deafened)} oncontextmenu={e => openQuick(e, 'output')}> + {/if} + +{#if visible && self} +
+ {#if mine} + You are sharing your screen{mine.getAudioTracks().length ? ' with sound' : ''}{share.viewers ? `, ${share.viewers} watching` : ''} + + {/if}
+ + {#if watching?.stream}{/if} + + {#if focused && focus != null} +
+
+ + {focus === self.session ? 'Your screen' : focusUser ? store.displayName(focusUser) : 'Screen share'} + {#if focus !== self.session && hasSound} + + store.saveSettings()} + aria-label="Stream volume" title="Stream volume" /> + {/if} + + {#if focus !== self.session}{/if} + +
+ + +
+ {/if} + {#if !collapsed} -
    +
      {#each people as u (u.session)} -
    • -
      • {/each}
    + {#if watching?.state === 'failed'} +

    Could not connect to the stream. Across the internet both of you need a STUN server set in Settings, Voice (the person sharing has to restart the stream after setting it).

    + {/if} {/if}
{/if} @@ -89,4 +195,28 @@ .tile.self .who { color: var(--text); font-weight: 600; } .badge { position: absolute; top: 4px; right: 4px; display: inline-flex; color: var(--text-faint); } .badge.admin { color: var(--danger); } + .badge.live { color: var(--speaking); } + + /* Streams */ + .bar { flex: none; display: flex; align-items: center; gap: 8px; padding: 6px 12px; font-size: 13px; background: var(--bg-1); border-bottom: 1px solid var(--line); } + .bar span { flex: 1; min-width: 0; } + .bar.bad { color: var(--danger); } + .btn.small, .bar .btn { padding: 4px 10px; font-size: 12px; flex: none; } + .status { font-size: 12px; color: var(--speaking); white-space: nowrap; overflow: hidden; text-overflow: ellipsis; min-width: 0; margin-right: 4px; } + .tiles li.wide { grid-column: span 2; } + .tile.video { padding: 0 0 6px; overflow: hidden; } + .tile.video video { width: 100%; aspect-ratio: 16 / 9; object-fit: contain; background: #000; display: block; } + .tile.focused { border-color: var(--accent); box-shadow: inset 0 0 0 1px var(--accent); } + .note { font-size: 11px; color: var(--text-faint); } + .failed { margin: 0; padding: 0 12px 10px; font-size: 12px; color: var(--danger); } + /* Meeting view: the chosen picture large, everyone else in one row below */ + .stage.meeting { display: flex; flex-direction: column; max-height: 72%; min-height: 220px; } + .spot { flex: 1; min-height: 0; display: flex; flex-direction: column; background: #000; } + .spot-head { flex: none; display: flex; align-items: center; gap: 8px; padding: 2px 6px 2px 12px; background: var(--bg-0); color: var(--text-dim); font-size: 13px; } + .spot-who { flex: 1; min-width: 0; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; color: var(--text); font-weight: 600; } + .spot video { flex: 1; min-height: 0; width: 100%; object-fit: contain; background: #000; display: block; } + .vol { width: 110px; flex: none; padding: 0; } + .tiles.strip { flex: none; display: flex; overflow-x: auto; overflow-y: hidden; padding-top: 8px; max-height: none; } + .tiles.strip li { flex: 0 0 96px; } + .tiles.strip li.wide { flex-basis: 150px; } diff --git a/test/e2e/app.e2e.ts b/test/e2e/app.e2e.ts index 06e8866..0889fa3 100644 --- a/test/e2e/app.e2e.ts +++ b/test/e2e/app.e2e.ts @@ -886,6 +886,8 @@ try { // Custom icon from the tile's context menu: square-cropped and shown instead of initials const iconFile = path.join(userData, 'icon.png'); writeFileSync(iconFile, testPng(300, 200)); + // With hints off, the title of whatever the pointer rests on is set aside; move off the tile first + await page.mouse.move(700, 400); await page.locator('.rail .tile[title^="Second"]').click({ button: 'right' }); const chooser = page.waitForEvent('filechooser'); await page.getByRole('menuitem', { name: 'Change icon...' }).click(); diff --git a/test/e2e/share.e2e.ts b/test/e2e/share.e2e.ts new file mode 100644 index 0000000..3956cee --- /dev/null +++ b/test/e2e/share.e2e.ts @@ -0,0 +1,145 @@ +// Screen sharing between two instances of the built app on a real Mumble server. +// npm run build && MUMBLE_TEST_HOST=localhost:64739 npm run test:e2e:share +import { _electron as electron, type Page } from 'playwright-core'; +import electronPath from 'electron'; +import assert from 'node:assert/strict'; +import { mkdtempSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import path from 'node:path'; + +const target = process.env.MUMBLE_TEST_HOST; +if (!target) { + console.log('MUMBLE_TEST_HOST not set, skipping'); + process.exit(0); +} +const [host, port] = target.split(':'); +const root = path.resolve(import.meta.dirname, '../..'); +const { ELECTRON_RUN_AS_NODE, ...env } = process.env; +const tag = Date.now() % 100000; + +async function launch() { + const app = await electron.launch({ + executablePath: electronPath as unknown as string, + args: [root, `--user-data-dir=${mkdtempSync(path.join(tmpdir(), 'mumh5-e2e-'))}`, '--ozone-platform=x11', + '--use-fake-device-for-media-stream', '--use-fake-ui-for-media-stream'], + env: env as Record + }); + const page = await app.firstWindow(); + page.on('console', m => { if (m.type() === 'error') console.log('[renderer]', m.text()); }); + await page.setViewportSize({ width: 1280, height: 800 }); + return { app, page }; +} + +// Identity wizard, then connect; the username comes from the identity +async function join(page: Page, name: string) { + await page.getByRole('dialog', { name: 'Set up your identity' }).waitFor(); + await page.getByRole('button', { name: /Create a new identity/ }).click(); + await page.getByLabel('Name', { exact: true }).fill(name); + await page.getByRole('button', { name: 'Create', exact: true }).click(); + await page.getByRole('button', { name: 'Skip for now' }).click(); + await page.getByRole('button', { name: 'Done' }).click(); + await page.getByTitle('Add a server').click(); + await page.getByLabel('Address').fill(host); + await page.getByLabel('Port').fill(port); + await page.getByRole('button', { name: 'Save and connect' }).click(); + await page.getByText(/Connected/).first().waitFor(); +} + +const alice = await launch(); +const bob = await launch(); +const aliceName = `share-a-${tag}`, bobName = `share-b-${tag}`; +try { + await join(alice.page, aliceName); + + // Alice starts sharing: a note about IP addresses once, then the source picker + const shareBtn = alice.page.locator('.me').getByRole('button', { name: 'Share your screen' }); + await shareBtn.click(); + await alice.page.getByText(/You see each other's IP address/).waitFor(); + await alice.page.getByRole('button', { name: 'Continue' }).click(); + // The dialog: pick a source, see the preview, then start + const picker = alice.page.getByRole('dialog', { name: 'Share your screen' }); + const startBtn = picker.getByRole('button', { name: 'Start sharing' }); + const pickAndStart = async () => { + assert.ok(await startBtn.isDisabled(), 'nothing can start before a source is chosen'); + // Listing is slow on the bare test display and can come back empty the first time + for (let i = 0; i < 6 && !(await picker.locator('.sources button').count()); i++) { + await Promise.race([picker.locator('.sources button').first().waitFor({ timeout: 40000 }), picker.getByText('Nothing found').waitFor({ timeout: 40000 })]).catch(() => {}); + if (await picker.getByText('Nothing found').count()) await picker.getByRole('button', { name: 'Refresh' }).click(); + } + await picker.locator('.sources button').first().click(); + await alice.page.waitForFunction(() => (document.querySelector('.preview video') as HTMLVideoElement | null)?.videoWidth! > 0, null, { timeout: 15000 }); + await startBtn.click(); + }; + await picker.getByLabel('Quality').selectOption('low'); + await pickAndStart(); + await alice.page.getByText('You are sharing your screen').waitFor(); + await alice.page.locator('.sidebar .row.user.self').getByTitle('You are sharing your screen').waitFor(); + // The sharer's own tile shows what is being sent; clicking it makes it large + await alice.page.waitForFunction(() => (document.querySelector('.stage .tile.self video') as HTMLVideoElement | null)?.videoWidth! > 0, null, { timeout: 15000 }); + await alice.page.locator('.stage .tile.self').click(); + const ownView = alice.page.getByRole('region', { name: 'Screen share' }); + await ownView.getByText('Your screen').waitFor(); + await ownView.getByRole('button', { name: 'Back to tiles' }).click(); + await ownView.waitFor({ state: 'detached' }); + console.log('ok: sharing started from the dialog, own tile shows the stream'); + + // Bob connects afterwards and still learns about the stream + await join(bob.page, bobName); + // Next to the name in the channel list, and on the tile above the chat + const watch = bob.page.locator('.sidebar').getByRole('button', { name: `Watch the screen of ${aliceName}`, exact: true }); + await watch.waitFor(); + const tile = bob.page.locator('.stage .tile', { hasText: aliceName }); + await tile.waitFor(); + console.log('ok: a late joiner sees who is sharing'); + + await tile.click(); + await bob.page.getByRole('button', { name: 'Continue' }).click(); + const view = bob.page.getByRole('region', { name: 'Screen share' }); + await view.getByText(aliceName).waitFor(); + await bob.page.waitForFunction(() => (document.querySelector('.spot video') as HTMLVideoElement | null)?.videoWidth! > 0, null, { timeout: 20000 }); + const size = await bob.page.evaluate(() => { const v = document.querySelector('.spot video') as HTMLVideoElement; return [v.videoWidth, v.videoHeight]; }); + await alice.page.getByText('You are sharing your screen, 1 watching').waitFor(); + console.log(`ok: bob receives the picture (${size[0]}x${size[1]}), alice sees one viewer`); + + // Like a meeting: back to tiles keeps the picture in the tile, a click makes it large again + await view.getByRole('button', { name: 'Back to tiles' }).click(); + await view.waitFor({ state: 'detached' }); + await bob.page.waitForFunction(() => (document.querySelector('.stage .tile.video video') as HTMLVideoElement | null)?.videoWidth! > 0); + await tile.click(); + await view.waitFor(); + console.log('ok: stream in the tile, large on click'); + + // Leaving and coming back + await view.getByRole('button', { name: 'Stop watching' }).click(); + await view.waitFor({ state: 'detached' }); + await alice.page.getByText('You are sharing your screen', { exact: true }).waitFor(); + // The sharer's profile says so and offers to watch + await bob.page.locator('.sidebar .row.user', { hasText: aliceName }).click(); + const profile = bob.page.locator('.panel .sharing'); + await profile.getByText('Sharing their screen').waitFor(); + await profile.getByRole('button', { name: 'Watch' }).click(); + await profile.getByRole('button', { name: 'Stop watching' }).waitFor(); + await bob.page.waitForFunction(() => (document.querySelector('.spot video') as HTMLVideoElement | null)?.videoWidth! > 0, null, { timeout: 20000 }); + console.log('ok: stop watching, watch again from the profile'); + + // Alice stops: the view and the indicator go away + await alice.page.locator('.me').getByRole('button', { name: 'Stop sharing your screen' }).click(); + await view.waitFor({ state: 'detached' }); + await watch.waitFor({ state: 'detached' }); + await profile.waitFor({ state: 'detached' }); + assert.equal(await alice.page.getByText('You are sharing your screen').count(), 0); + console.log('ok: stopping ends the stream for viewers'); + + // A sharer who disconnects takes the stream with them + await shareBtn.click(); + await pickAndStart(); + await watch.waitFor(); + await alice.page.getByTitle('Disconnect').click(); + await watch.waitFor({ state: 'detached' }); + console.log('ok: disconnecting ends the stream'); + + console.log('SHARE E2E PASSED'); +} finally { + await alice.app.close().catch(() => {}); + await bob.app.close().catch(() => {}); +} diff --git a/test/e2e/web-shell.cjs b/test/e2e/web-shell.cjs index 9fdf083..eb30fdd 100644 --- a/test/e2e/web-shell.cjs +++ b/test/e2e/web-shell.cjs @@ -1,5 +1,5 @@ // A bare browser window for the web E2E: no preload, so the page runs as it would in a browser. -const { app, BrowserWindow, session } = require('electron'); +const { app, BrowserWindow, desktopCapturer, session } = require('electron'); const fs = require('node:fs'); const path = require('node:path'); @@ -11,6 +11,12 @@ app.whenReady().then(() => { item.setSavePath(file); item.once('done', (_ev, state) => { if (state === 'completed') fs.writeFileSync(file + '.done', ''); }); }); + // Stands in for the browser's own "choose what to share" dialog: always the first screen + session.defaultSession.setDisplayMediaRequestHandler(async (_request, callback) => { + let sources = []; + for (let i = 0; i < 10 && !sources.length; i++) sources = await desktopCapturer.getSources({ types: ['screen'] }); + try { callback(sources.length ? { video: sources[0] } : {}); } catch { /* refused */ } + }); const win = new BrowserWindow({ width: 1280, height: 800, webPreferences: { contextIsolation: true, sandbox: true, nodeIntegration: false } }); win.loadURL(process.env.MUMH5_WEB_URL); }); diff --git a/test/e2e/web.e2e.ts b/test/e2e/web.e2e.ts index 17bbad5..5f34787 100644 --- a/test/e2e/web.e2e.ts +++ b/test/e2e/web.e2e.ts @@ -42,7 +42,7 @@ function within(p: Promise, label: string, ms = 10000): Promise { const proxyPort = 18000 + Math.floor(Math.random() * 1000); const { ELECTRON_RUN_AS_NODE, ...env } = process.env; const proxy = spawn(process.execPath, [path.join(root, 'dist-proxy/proxy.mjs')], { - env: { ...env, MUMH5_PORT: String(proxyPort), MUMH5_SERVERS: `${target}=Test Server` }, stdio: ['ignore', 'pipe', 'inherit'] + env: { ...env, MUMH5_PORT: String(proxyPort), MUMH5_SERVERS: `${target}=Test Server`, MUMH5_STUN_PORT: String(proxyPort + 1000), MUMH5_STUN_BIND: '127.0.0.1' }, stdio: ['ignore', 'pipe', 'inherit'] }); await within(new Promise((res, rej) => { proxy.stdout.on('data', d => { if (String(d).includes('listening')) res(); }); @@ -50,12 +50,14 @@ await within(new Promise((res, rej) => { }), 'proxy start'); const downloads = mkdtempSync(path.join(tmpdir(), 'mumh5-e2e-dl-')); -const app = await electron.launch({ +const browser = () => electron.launch({ executablePath: electronPath as unknown as string, args: [path.join(root, 'test/e2e/web-shell.cjs'), `--user-data-dir=${mkdtempSync(path.join(tmpdir(), 'mumh5-e2e-'))}`, '--ozone-platform=x11', '--use-fake-device-for-media-stream', '--use-fake-ui-for-media-stream'], env: { ...env, MUMH5_WEB_URL: `http://127.0.0.1:${proxyPort}/`, MUMH5_DOWNLOADS: downloads } as Record }); +const app = await browser(); +let second: Awaited> | null = null; const bob = await headless('bob'); const name = `webalice${Date.now() % 100000}`; try { @@ -160,9 +162,38 @@ try { assert.equal(hash(), before, 'same certificate after a reload'); console.log('ok: identity kept across a reload'); + // Screen sharing between two browsers: the proxy's own STUN is used without any setting + assert.equal((await (await fetch(`http://127.0.0.1:${proxyPort}/api/config`)).json()).stun, proxyPort + 1000); + second = await browser(); + const page2 = await second.firstWindow(); + page2.on('console', m => { if (m.type() === 'error') console.log('[page2]', m.text()); }); + await page2.setViewportSize({ width: 1280, height: 800 }); + await page2.getByRole('button', { name: /Create a new identity/ }).click(); + await page2.getByLabel('Name', { exact: true }).fill(`webcarol${Date.now() % 100000}`); + await page2.getByRole('button', { name: 'Create', exact: true }).click(); + await page2.getByRole('button', { name: 'Skip for now' }).click(); + await page2.getByRole('button', { name: 'Done' }).click(); + await page2.getByTitle('Add a server').click(); + await page2.getByRole('button', { name: 'Save and connect' }).click(); + await page2.getByText(/Connected/).first().waitFor(); + + await page.locator('.me').getByRole('button', { name: 'Share your screen' }).click(); + await page.getByRole('button', { name: 'Continue' }).click(); + const dialog = page.getByRole('dialog', { name: 'Share your screen' }); + await dialog.getByRole('button', { name: 'Choose a screen or window...' }).click(); + await page.waitForFunction(() => (document.querySelector('.preview video') as HTMLVideoElement | null)?.videoWidth! > 0, null, { timeout: 60000 }); + await dialog.getByRole('button', { name: 'Start sharing' }).click(); + await page.getByText('You are sharing your screen').waitFor(); + await page2.locator('.stage .tile', { hasText: name }).click(); + await page2.getByRole('button', { name: 'Continue' }).click(); + await page2.waitForFunction(() => (document.querySelector('.spot video') as HTMLVideoElement | null)?.videoWidth! > 0, null, { timeout: 30000 }); + await page.getByText('You are sharing your screen, 1 watching').waitFor(); + console.log('ok: screen sharing between two browsers'); + console.log('WEB E2E PASSED'); } finally { bob.disconnect(); await app.close().catch(() => {}); + await second?.close().catch(() => {}); proxy.kill(); } diff --git a/test/pipewire.test.ts b/test/pipewire.test.ts new file mode 100644 index 0000000..c83ed2a --- /dev/null +++ b/test/pipewire.test.ts @@ -0,0 +1,47 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { audioApps, planLinks, VIRTUAL_MIC } from '../electron/pipewire.ts'; + +// A cut-down pw-dump: the virtual microphone (10), this app (20, process 500), a browser whose +// stream has no name of its own (30, named by its client), a mono game (40) and a music player (50) +const node = (id: number, props: Record) => ({ id, type: 'PipeWire:Interface:Node', info: { props } }); +const client = (id: number, name: string, pid: number) => ({ id, type: 'PipeWire:Interface:Client', info: { props: { 'application.name': name, 'application.process.id': pid } } }); +const port = (id: number, nodeId: number, direction: string, channel: string) => ({ id, type: 'PipeWire:Interface:Port', info: { direction, props: { 'node.id': nodeId, 'audio.channel': channel } } }); +const link = (id: number, outPort: number, inPort: number, inNode: number) => + ({ id, type: 'PipeWire:Interface:Link', info: { 'output-port-id': outPort, 'input-port-id': inPort, 'input-node-id': inNode } as any }); + +const graph = [ + node(10, { 'node.name': VIRTUAL_MIC, 'media.class': 'Audio/Source/Virtual' }), port(11, 10, 'input', 'FL'), port(12, 10, 'input', 'FR'), + port(13, 10, 'output', 'FL'), port(14, 10, 'output', 'FR'), + client(2, 'mumh5', 500), node(20, { 'media.class': 'Stream/Output/Audio', 'node.name': 'mumh5', 'client.id': 2 }), port(21, 20, 'output', 'FL'), port(22, 20, 'output', 'FR'), + client(3, 'Firefox', 600), node(30, { 'media.class': 'Stream/Output/Audio', 'node.name': 'Firefox', 'client.id': 3 }), port(31, 30, 'output', 'FL'), port(32, 30, 'output', 'FR'), + node(40, { 'media.class': 'Stream/Output/Audio', 'application.name': 'Game', 'application.process.id': 700 }), port(41, 40, 'output', 'MONO'), + node(50, { 'media.class': 'Stream/Output/Audio', 'application.name': 'Spotify' }), port(51, 50, 'output', 'FL'), port(52, 50, 'output', 'FR'), port(53, 50, 'output', 'LFE'), + node(60, { 'media.class': 'Audio/Sink', 'node.name': 'speakers' }) +]; + +test('lists programs that play sound, without this app', () => { + assert.deepEqual(audioApps(graph, [500]), ['Firefox', 'Game', 'Spotify']); +}); + +test('whole system links every program but this app', () => { + const { add, remove } = planLinks(graph, { kind: 'all' }, [500]); + assert.deepEqual(add.sort(), [[31, 11], [32, 12], [41, 11], [41, 12], [51, 11], [52, 12]].sort()); + assert.deepEqual(remove, []); +}); + +test('one program links only that program', () => { + assert.deepEqual(planLinks(graph, { kind: 'app', name: 'Firefox' }, [500]).add, [[31, 11], [32, 12]]); +}); + +test('existing links are kept, stale ones removed', () => { + const linked = [...graph, link(90, 31, 11, 10), link(91, 51, 11, 10), link(92, 21, 60, 60)]; + const { add, remove } = planLinks(linked, { kind: 'app', name: 'Firefox' }, [500]); + assert.deepEqual(add, [[32, 12]]); + // Spotify's link goes; the link to the speakers is none of our business + assert.deepEqual(remove, [91]); +}); + +test('nothing without the virtual microphone', () => { + assert.deepEqual(planLinks(graph.slice(5), { kind: 'all' }, []), { add: [], remove: [] }); +}); diff --git a/test/proxy.test.ts b/test/proxy.test.ts index 68f1ec5..33eabef 100644 --- a/test/proxy.test.ts +++ b/test/proxy.test.ts @@ -2,8 +2,10 @@ // real Mumble server and are skipped unless MUMBLE_TEST_HOST is set (see server.test.ts). import { test } from 'node:test'; import net from 'node:net'; +import dgram from 'node:dgram'; import assert from 'node:assert/strict'; import { proxyLine } from '../electron/tls-transport.ts'; +import { stunResponse } from '../server/proxy.ts'; import { startProxy, defaults, parseServers, isPrivateAddress, type ProxyConfig } from '../server/proxy.ts'; import { WebSocket as WsClient } from 'ws'; import { WebSocketTransport } from '../src/core/ws-transport.ts'; @@ -40,7 +42,7 @@ test('refuses to start without allowed servers', async () => { test('config lists the allowed servers', async () => { await withProxy({ servers: parseServers('voice.example.org=Friends') }, async base => { - assert.deepEqual(await (await fetch(`${base}/api/config`)).json(), { servers: [{ host: 'voice.example.org', port: 64738, label: 'Friends' }], any: false }); + assert.deepEqual(await (await fetch(`${base}/api/config`)).json(), { servers: [{ host: 'voice.example.org', port: 64738, label: 'Friends' }], any: false, stun: null }); }); }); @@ -64,6 +66,47 @@ test('identity create, describe, export and import round trip', async () => { }); }); +// A binding request: type, length 0, magic cookie, 12 byte transaction id +const stunRequest = () => Uint8Array.from([0, 1, 0, 0, 0x21, 0x12, 0xa4, 0x42, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]); + +test('STUN answers carry the sender address, masked as the protocol asks', () => { + const v4 = stunResponse(stunRequest(), '203.0.113.7', 54321)!; + assert.deepEqual([...v4.subarray(0, 4)], [1, 1, 0, 12]); + assert.deepEqual([...v4.subarray(4, 20)], [...stunRequest().subarray(4, 20)]); + assert.deepEqual([...v4.subarray(20, 26)], [0, 0x20, 0, 8, 0, 1]); + assert.equal(((v4[26] << 8) | v4[27]) ^ 0x2112, 54321); + assert.deepEqual([...v4.subarray(28)].map((b, i) => b ^ [0x21, 0x12, 0xa4, 0x42][i]), [203, 0, 113, 7]); + // An IPv4 sender seen through an IPv6 socket is still IPv4 + assert.deepEqual([...stunResponse(stunRequest(), '::ffff:203.0.113.7', 54321)!], [...v4]); + const v6 = stunResponse(stunRequest(), '2001:db8::7', 1)!; + assert.equal(v6[25], 2); + const mask = stunRequest().subarray(4, 20); + assert.deepEqual([...v6.subarray(28)].map((b, i) => b ^ mask[i]), [0x20, 0x01, 0x0d, 0xb8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 7]); + // Not a binding request + assert.equal(stunResponse(new Uint8Array(20), '203.0.113.7', 1), null); + assert.equal(stunResponse(stunRequest().subarray(0, 12), '203.0.113.7', 1), null); +}); + +test('the proxy answers STUN over UDP and announces the port', async () => { + const proxy = await startProxy({ ...defaults, port: 0, origins: ['*'], servers: parseServers('voice.example.org'), stunPort: 0, stunBind: '127.0.0.1' }); + try { + assert.ok(proxy.stunPort); + assert.equal((await (await fetch(`http://127.0.0.1:${proxy.port}/api/config`)).json()).stun, proxy.stunPort); + const client = dgram.createSocket('udp4'); + const answer = await new Promise((resolve, reject) => { + client.once('message', resolve); + client.once('error', reject); + client.send(stunRequest(), proxy.stunPort!, '127.0.0.1'); + setTimeout(() => reject(new Error('no STUN answer')), 3000); + }); + assert.equal(((answer[26] << 8) | answer[27]) ^ 0x2112, client.address().port); + assert.deepEqual([...answer.subarray(28)].map((b, i) => b ^ [0x21, 0x12, 0xa4, 0x42][i]), [127, 0, 0, 1]); + client.close(); + } finally { + await proxy.close(); + } +}); + test('requests from other origins are refused', async () => { await withProxy({ servers: parseServers('voice.example.org'), origins: [] }, async base => { const cross = await post(base, 'identity/create', { name: 'x' }, { Origin: 'https://evil.example' }); @@ -181,7 +224,8 @@ test('two clients talk through the proxy', { skip: !target }, async () => { const a = await connect(`proxy-a-${Date.now() % 100000}`); const b = await connect(`proxy-b-${Date.now() % 100000}`); const got = new Promise(resolve => b.on('text', m => resolve(m.html))); - a.sendText({ channels: [0] }, 'hello through the proxy'); + // A direct message, so other test files listening in the root channel are not disturbed + a.sendText({ users: [b.session!] }, 'hello through the proxy'); assert.equal(await got, 'hello through the proxy'); // No UDP in a browser: voice stays on the TCP tunnel assert.equal(a.udpOk, false); diff --git a/test/share-signal.test.ts b/test/share-signal.test.ts new file mode 100644 index 0000000..c5bcfae --- /dev/null +++ b/test/share-signal.test.ts @@ -0,0 +1,42 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { SHARE, encodeShare, ShareAssembler } from '../src/core/share-signal.ts'; + +test('bare signals are one byte', async () => { + const packets = await encodeShare(SHARE.on); + assert.deepEqual(packets, [new Uint8Array([SHARE.on])]); + assert.deepEqual(await new ShareAssembler().push(1, packets[0]), { type: SHARE.on }); +}); + +test('session descriptions survive chunking, in any order', async () => { + // Random-looking text compresses badly, forcing several chunks + const sdp = Array.from({ length: 400 }, (_, i) => `a=candidate:${(i * 2654435761 >>> 0).toString(36)} 1 udp ${i} 2001:db8::${i} typ host`).join('\r\n'); + const packets = await encodeShare(SHARE.offer, sdp); + assert.ok(packets.length > 1, 'needs more than one packet'); + // Murmur drops plugin data over 1000 bytes + assert.ok(packets.every(p => p.length <= 1000)); + const assembler = new ShareAssembler(); + let result = null; + for (const p of [...packets].reverse()) result = await assembler.push(7, p); + assert.deepEqual(result, { type: SHARE.offer, text: sdp }); +}); + +test('chunks from different senders do not mix', async () => { + const a = await encodeShare(SHARE.answer, Array.from({ length: 400 }, (_, i) => (i * 40503 * 2654435761 >>> 0).toString(36)).join(' ')); + assert.ok(a.length > 1); + const assembler = new ShareAssembler(); + assert.equal(await assembler.push(1, a[0]), null); + for (const p of a.slice(1)) assert.equal(await assembler.push(2, p), null); + let result = null; + for (const p of a.slice(1)) result = await assembler.push(1, p); + assert.equal(result?.type, SHARE.answer); +}); + +test('garbage is ignored', async () => { + const assembler = new ShareAssembler(); + assert.equal(await assembler.push(1, new Uint8Array([99])), null); + assert.equal(await assembler.push(1, new Uint8Array([SHARE.offer, 1])), null); + assert.equal(await assembler.push(1, new Uint8Array([SHARE.offer, 1, 5, 2, 1, 2, 3])), null); + // A complete transfer that is not valid compressed data + assert.equal(await assembler.push(1, new Uint8Array([SHARE.offer, 1, 0, 1, 255, 255, 255, 255])), null); +});