Camera rounds: several people's cameras at once in the voice tiles #14

Merged
kibi merged 1 commits from screen-sharing into main 2026-10-01 22:16:17 +00:00
13 changed files with 252 additions and 115 deletions
Showing only changes of commit 1e4bda4e74 - Show all commits
+1 -1
View File
@@ -6,7 +6,7 @@ All notable changes to mumh5. Versions follow the `version` in `package.json`.
### Added
- Camera sharing, as a second source in the screen sharing dialog.
- Camera sharing, as a second source in the screen sharing dialog, and camera rounds: several people's cameras at once in the voice tiles.
- A relay for streams in the web proxy, used automatically by the browser version and, with its address entered in Settings, by the desktop app, for networks and browsers that allow no direct connection.
- Screen sharing between mumh5 users in a channel: a dialog to choose the screen or window (with preview), the sound and the quality; sound from one program or the whole system on Linux through PipeWire. Viewers watch in the voice tiles, enlarge a stream like in a meeting and set its volume. Streams go directly between clients; a STUN server can be set in Settings, Voice.
- Browser version: `npm run build:web` builds the web app and a small self-hosted proxy that bridges browsers to Mumble servers on an allowlist. Identities are kept in the browser, voice goes through the TCP tunnel. See "Browser version" in the README.
+1 -1
View File
@@ -47,7 +47,7 @@ A reused test server keeps registrations and channels from earlier runs; tests m
- `src/core/` (browser-safe TypeScript, also runs in Node for tests): framing and codec (`proto.ts`), the Mumble client state machine (`client.ts`), voice packet formats (`voice-packet.ts`). No DOM, no Electron, no Node imports here.
- `src/lib/native.ts`: `desktop` is the Electron preload API or null; `native` is what both platforms provide (identities, certificates), backed by `web.svelte.ts` in the browser build (`isWeb`, vite `--mode web`). Desktop-only features check `desktop`.
- `src/lib/`: app state. `session.svelte.ts` has one `Session` per server plus the `sessions` manager; `session` is a Proxy to the active one. `audio/voice.svelte.ts` is the voice engine (WebCodecs Opus, capture and playback AudioWorklets). `html.ts` sanitizes incoming HTML and serializes outgoing rich text.
- Screen sharing: `src/core/share-signal.ts` (signals as Mumble plugin data, id `mumh5.share`, compressed and chunked) and `src/lib/share.svelte.ts` (one WebRTC connection per viewer, candidates inside the description, no trickle). The start dialog is `ShareDialog.svelte`, the tiles and the large view are `VoiceStage.svelte`. Desktop source listing is `share:sources` / `share:pick` in `electron/main.ts`; Linux sound is `electron/pipewire.ts` (a virtual microphone fed with pw-link, never with this app's own playback).
- Screen sharing: `src/core/share-signal.ts` (signals as Mumble plugin data, id `mumh5.share`, compressed and chunked) and `src/lib/share.svelte.ts` (one WebRTC connection per viewer and stream, candidates inside the description, no trickle; a viewer holds one screen and any number of cameras in `share.watches`). The start dialog is `ShareDialog.svelte`, the tiles and the large view are `VoiceStage.svelte`. Desktop source listing is `share:sources` / `share:pick` in `electron/main.ts`; Linux sound is `electron/pipewire.ts` (a virtual microphone fed with pw-link, never with this app's own playback).
- `src/ui/`: Svelte components. `App.svelte` owns layout and global dialogs (`ui.svelte.ts` store).
Voice runs on one server at a time (where you last joined a channel); background servers are auto self-deafened and restored when voice returns.
+1 -1
View File
@@ -14,5 +14,5 @@ COPY --from=build /src/dist-web ./dist-web
COPY --from=build /src/dist-proxy ./dist-proxy
USER node
ENV MUMH5_PORT=8080
EXPOSE 8080/tcp 3478/udp 3478/tcp 49160-49359/udp
EXPOSE 8080/tcp 3478/udp 3478/tcp 49160-49659/udp
CMD ["node", "dist-proxy/proxy.mjs"]
+5 -3
View File
@@ -78,6 +78,7 @@ mumh5 keeps the foundation and replaces the experience.
### Screen sharing
- Camera rounds: everyone who turns their camera on appears in the tiles at once, like a video meeting; one click joins or leaves the round, and a click on a tile enlarges that camera
- Share a screen, a window or a camera with the mumh5 users in your channel. A dialog lets you choose what to show, with a live preview, which sound goes with it and the quality, before anything is sent
- Sound: on Linux one program or everything except mumh5 itself (through PipeWire, so viewers do not hear the voice chat twice); on Windows the whole system; in a browser what the browser offers
- People in the channel see an indicator next to your name, on your tile and in your profile, and click to watch. The tiles above the chat fill a stage you can drag taller or shorter. A stream opens large, with everyone as a strip of tiles below, like a meeting, and can move to a window of its own; in the stacked layout it is just the picture, as wide as the chat. Sounds announce streams and viewers. Viewers set the stream's volume
@@ -177,7 +178,7 @@ docker compose up -d --build
The container uses the host's network: the proxy listens on `127.0.0.1:8080` for your reverse proxy, and a Mumble server on the same machine is reachable as `localhost`.
For screen sharing, open these in the firewall (and forward them on a router in front of the server): port 3478 for UDP and TCP, and UDP 49160-49359. They carry STUN and the relay, which the browser version uses without any setting. Relayed streams pass through your server and use its bandwidth, a few Mbit/s per viewer; anyone who knows the site's address can get credentials for it (the desktop app asks from outside the site), limited by connection quotas per address; there is no bandwidth cap yet.
For screen sharing, open these in the firewall (and forward them on a router in front of the server): port 3478 for UDP and TCP, and UDP 49160-49659. They carry STUN and the relay, which the browser version uses without any setting. Relayed streams pass through your server and use its bandwidth, a few Mbit/s per viewer; anyone who knows the site's address can get credentials for it (the desktop app asks from outside the site), limited by connection quotas per address; there is no bandwidth cap yet.
| Variable | Default | Meaning |
| --- | --- | --- |
@@ -189,7 +190,8 @@ For screen sharing, open these in the firewall (and forward them on a router in
| `MUMH5_SEND_PROXY` | off | Announce each visitor's address to the server with the PROXY protocol (see below). Breaks connections to a plain Mumble server |
| `MUMH5_STUN_PORT`, `MUMH5_STUN_BIND` | `3478`, all addresses | Port for screen sharing between browser users: STUN over UDP, and the relay over UDP and TCP. Browsers reach it directly, not through nginx. `0` turns both off |
| `MUMH5_TURN` | on | The relay (TURN) for people who cannot connect directly: mobile networks, strict company networks, browsers that forbid direct UDP such as Vanadium. `0` leaves only STUN |
| `MUMH5_TURN_PORTS` | `49160-49359` | UDP ports the relayed streams use |
| `MUMH5_TURN_PORTS` | `49160-49659` | UDP ports the relayed streams use, one per relayed route |
| `MUMH5_TURN_MAX`, `MUMH5_TURN_PER_ADDRESS` | `500`, `64` | Relayed routes in total and per client address. A camera round uses several per person |
| `MUMH5_TURN_IP` | found automatically | The server's public address, announced for relayed streams. Set it when the server sits behind a 1:1 NAT, as on many cloud hosts |
| `MUMH5_STATIC` | `../dist-web` | Folder with the web build |
| `MUMH5_MAX_CONNECTIONS`, `MUMH5_MAX_PER_ADDRESS` | `200`, `8` | Connection limits, in total and per client address |
@@ -278,7 +280,7 @@ Without an upload host, mumh5 still sends images, scaled to fit the server's lim
- System-wide push to talk
- Whisper and shout
- Rich chat between mumh5 users: replies, reactions, edits, typing indicators
- Screen sharing: several streams at once, and an optional self-hosted relay for many viewers
- Screen sharing: a media server for large rounds (today every sender uploads once per viewer, up to 8)
- Release builds for all platforms
---
+3 -1
View File
@@ -8,7 +8,7 @@ services:
# Host networking, so STUN sees each visitor's real address (through Docker's port
# forwarding it would often see Docker's own) and a Mumble server on this machine is
# reachable as localhost. The proxy listens on 127.0.0.1:8080 for nginx, on port 3478 (UDP and
# TCP) for STUN and the relay, and relays streams on UDP 49160-49359.
# TCP) for STUN and the relay, and relays streams on UDP 49160-49659.
network_mode: host
# Passed on from .env (or the shell); empty means the proxy's default. Written out one by
# one because older docker-compose versions cannot mark an env_file as optional.
@@ -26,5 +26,7 @@ services:
MUMH5_TURN: ${MUMH5_TURN:-}
MUMH5_TURN_IP: ${MUMH5_TURN_IP:-}
MUMH5_TURN_PORTS: ${MUMH5_TURN_PORTS:-}
MUMH5_TURN_MAX: ${MUMH5_TURN_MAX:-}
MUMH5_TURN_PER_ADDRESS: ${MUMH5_TURN_PER_ADDRESS:-}
MUMH5_MAX_CONNECTIONS: ${MUMH5_MAX_CONNECTIONS:-}
MUMH5_MAX_PER_ADDRESS: ${MUMH5_MAX_PER_ADDRESS:-}
+8 -2
View File
@@ -42,13 +42,17 @@ export interface ProxyConfig {
turnIp: string | null;
turnMinPort: number;
turnMaxPort: number;
// Relay routes in total and per client address. A browser opens one per stream and network
// interface, and a camera round has a stream per pair of people, so these run high.
turnMax: number;
turnPerAddress: number;
maxConnections: number;
maxPerAddress: number;
}
export const defaults: ProxyConfig = {
port: 8080, bind: '127.0.0.1', servers: [], allowAny: false, allowPrivate: false, origins: [],
trustProxy: false, sendProxy: false, staticDir: null, stunPort: null, stunBind: '::', turn: true, turnIp: null, turnMinPort: 49160, turnMaxPort: 49359, maxConnections: 200, maxPerAddress: 8
trustProxy: false, sendProxy: false, staticDir: null, stunPort: null, stunBind: '::', turn: true, turnIp: null, turnMinPort: 49160, turnMaxPort: 49659, turnMax: 500, turnPerAddress: 64, maxConnections: 200, maxPerAddress: 8
};
// "host", "host:port", "[v6]:port", each optionally followed by "=Label"
@@ -85,6 +89,8 @@ export function configFromEnv(source: NodeJS.ProcessEnv): ProxyConfig {
turnIp: env.MUMH5_TURN_IP ?? null,
turnMinPort: Number((env.MUMH5_TURN_PORTS ?? '').split('-')[0]) || defaults.turnMinPort,
turnMaxPort: Number((env.MUMH5_TURN_PORTS ?? '').split('-')[1]) || defaults.turnMaxPort,
turnMax: Number(env.MUMH5_TURN_MAX ?? defaults.turnMax),
turnPerAddress: Number(env.MUMH5_TURN_PER_ADDRESS ?? defaults.turnPerAddress),
maxConnections: Number(env.MUMH5_MAX_CONNECTIONS ?? defaults.maxConnections),
maxPerAddress: Number(env.MUMH5_MAX_PER_ADDRESS ?? defaults.maxPerAddress)
};
@@ -341,7 +347,7 @@ export async function startProxy(config: ProxyConfig): Promise<{ port: number; s
try {
turn = await startTurn({
port: config.stunPort, bind: config.stunBind, relay: config.turn, publicIp: config.turnIp,
minPort: config.turnMinPort, maxPort: config.turnMaxPort, maxAllocations: config.maxConnections, maxPerAddress: config.maxPerAddress,
minPort: config.turnMinPort, maxPort: config.turnMaxPort, maxAllocations: config.turnMax, maxPerAddress: config.turnPerAddress,
peerAllowed: ip => config.allowPrivate || !isPrivateAddress(ip)
}, turnSecret);
stunPort = turn.port;
+88 -40
View File
@@ -13,6 +13,10 @@ import { sounds } from './audio/sounds.svelte.ts';
// Every viewer is a separate upload from the sharer
const MAX_VIEWERS = 8;
// Cameras watched at once; each is a download and a decoder
const MAX_CAMERAS = 12;
// A face needs far less than a screen full of text
const CAMERA_BITRATE = 1_200_000;
export const QUALITY = {
low: { label: 'Low (1 Mbit/s, 30 fps)', bitrate: 1_000_000, fps: 30 },
medium: { label: 'Medium (4 Mbit/s, 30 fps)', bitrate: 4_000_000, fps: 30 },
@@ -29,6 +33,7 @@ export type ShareKind = 'screen' | 'camera';
export interface Watching {
host: Session;
session: number;
kind: ShareKind;
stream: MediaStream | null;
state: 'connecting' | 'live' | 'failed';
}
@@ -39,15 +44,19 @@ class ScreenShare {
stream = $state.raw<MediaStream | null>(null);
kind = $state<ShareKind>('screen');
viewers = $state(0);
watching = $state.raw<Watching | null>(null);
// Streams we receive: one screen at a time, and any number of cameras next to it
watches = $state.raw<Watching[]>([]);
// In the camera round: everyone's camera in the channel is watched as it comes on
cams = $state(false);
private camHost: Session | null = null;
error = $state('');
// The server the start dialog is open for
dialog = $state.raw<Session | null>(null);
private usesVirtualMic = false;
private peers = new Map<number, RTCPeerConnection>();
private watchPc: RTCPeerConnection | null = null;
private watchTimer: ReturnType<typeof setTimeout> | undefined;
private watchPcs = new Map<number, RTCPeerConnection>();
private watchTimers = new Map<number, ReturnType<typeof setTimeout>>();
private assemblers = new WeakMap<Session, ShareAssembler>();
// Viewers whose connection came up, so leaving is only announced for those
private live = new Set<number>();
@@ -139,6 +148,8 @@ class ScreenShare {
this.send(s, this.others(s), SHARE.on, kind);
s.setSharing(s.client.session!, kind);
sounds.play('shareStarted');
// Turning the camera on joins the round
if (kind === 'camera') this.setCams(s, true);
}
private release(): void {
@@ -174,33 +185,63 @@ class ScreenShare {
this.viewers = this.peers.size;
}
// The screen being watched, if any (there is at most one)
get watching(): Watching | null {
return this.watches.find(w => w.kind === 'screen') ?? null;
}
watchOf(session: number): Watching | undefined {
return this.watches.find(w => w.session === session);
}
async watch(s: Session, session: number): Promise<void> {
if (!this.supported(s) || !s.sharing[session] || !(await this.notice())) return;
this.unwatch();
this.watching = { host: s, session, stream: null, state: 'connecting' };
const kind = s.sharing[session];
if (!this.supported(s) || !kind || this.watchOf(session) || !(await this.notice())) return;
// Streams from another server, and another screen, make way
for (const w of this.watches) if (w.host !== s || (kind === 'screen' && w.kind === 'screen')) this.unwatch(w.session);
if (kind === 'camera' && this.watches.filter(w => w.kind === 'camera').length >= MAX_CAMERAS) return;
if (!s.sharing[session] || this.watchOf(session)) return;
this.watches = [...this.watches, { host: s, session, kind, stream: null, state: 'connecting' }];
this.send(s, [session], SHARE.watch);
this.watchTimer = setTimeout(() => this.setWatchState('failed'), 20000);
this.watchTimers.set(session, setTimeout(() => this.setWatchState(session, 'failed'), 20000));
}
unwatch(): void {
const w = this.watching;
if (!w) return;
// One stream, or all of them
unwatch(session?: number): void {
for (const w of this.watches) {
if (session != null && w.session !== session) continue;
if (w.host.client) this.send(w.host, [w.session], SHARE.leave);
this.endWatch();
this.endWatch(w.session);
}
}
private endWatch(): void {
clearTimeout(this.watchTimer);
this.watchPc?.close();
this.watchPc = null;
this.watching = null;
private endWatch(session: number): void {
clearTimeout(this.watchTimers.get(session));
this.watchTimers.delete(session);
this.watchPcs.get(session)?.close();
this.watchPcs.delete(session);
this.watches = this.watches.filter(w => w.session !== session);
}
private setWatchState(state: Watching['state'], stream?: MediaStream): void {
const w = this.watching;
if (!w) return;
if (state !== 'connecting') clearTimeout(this.watchTimer);
this.watching = { ...w, state, stream: stream ?? w.stream };
private setWatchState(session: number, state: Watching['state'], stream?: MediaStream): void {
if (!this.watchOf(session)) return;
if (state !== 'connecting') { clearTimeout(this.watchTimers.get(session)); this.watchTimers.delete(session); }
this.watches = this.watches.map(w => w.session === session ? { ...w, state, stream: stream ?? w.stream } : w);
}
// The camera round: watch every camera in the channel, now and as they come on
async setCams(s: Session, on: boolean): Promise<void> {
if (!on) {
this.cams = false;
this.camHost = null;
for (const w of this.watches) if (w.kind === 'camera') this.unwatch(w.session);
return;
}
if (!this.supported(s) || !(await this.notice())) return;
this.cams = true;
this.camHost = s;
for (const [session, kind] of Object.entries(s.sharing)) {
if (kind === 'camera' && Number(session) !== s.client?.session) this.watch(s, Number(session));
}
}
// Candidates travel inside the description, so wait for them instead of sending each one
@@ -237,7 +278,8 @@ class ScreenShare {
const params = sender.getParameters();
// Keep the picture sharp and let the frame rate drop when the connection is tight
params.degradationPreference = this.kind === 'camera' ? 'maintain-framerate' : 'maintain-resolution';
for (const enc of params.encodings ?? []) enc.maxBitrate = QUALITY[store.settings.shareQuality].bitrate;
const wanted = QUALITY[store.settings.shareQuality].bitrate;
for (const enc of params.encodings ?? []) enc.maxBitrate = this.kind === 'camera' ? Math.min(wanted, CAMERA_BITRATE) : wanted;
await sender.setParameters(params).catch(() => {});
}
await pc.setLocalDescription({ type: 'offer', sdp: stereo((await pc.createOffer()).sdp ?? '') });
@@ -246,36 +288,40 @@ class ScreenShare {
}
private async answerTo(s: Session, sharer: number, sdp: string): Promise<void> {
const w = this.watching;
if (!w || w.host !== s || w.session !== sharer) return;
const mine = () => { const w = this.watchOf(sharer); return !!w && w.host === s; };
if (!mine()) return;
const config = await this.rtcConfig();
if (this.watching?.host !== s || this.watching.session !== sharer) return;
this.watchPc?.close();
if (!mine()) return;
this.watchPcs.get(sharer)?.close();
const pc = new RTCPeerConnection(config);
this.watchPc = pc;
this.watchPcs.set(sharer, pc);
const current = () => this.watchPcs.get(sharer) === pc;
// The track is announced with the description, before any route exists: the stream only
// counts as live once the connection is up, otherwise the viewer stares at a black picture
pc.addEventListener('track', e => {
if (this.watchPc === pc) this.setWatchState(pc.connectionState === 'connected' ? 'live' : 'connecting', e.streams[0]);
if (current()) this.setWatchState(sharer, pc.connectionState === 'connected' ? 'live' : 'connecting', e.streams[0]);
});
pc.addEventListener('connectionstatechange', () => {
if (this.watchPc !== pc) return;
if (pc.connectionState === 'connected') this.setWatchState('live');
else if (pc.connectionState === 'failed') this.setWatchState('failed');
if (!current()) return;
if (pc.connectionState === 'connected') this.setWatchState(sharer, 'live');
else if (pc.connectionState === 'failed') this.setWatchState(sharer, 'failed');
});
try {
await pc.setRemoteDescription({ type: 'offer', sdp });
await pc.setLocalDescription({ type: 'answer', sdp: stereo((await pc.createAnswer()).sdp ?? '') });
await this.gathered(pc);
if (this.watchPc === pc && pc.localDescription) await this.send(s, [sharer], SHARE.answer, pc.localDescription.sdp);
if (current() && pc.localDescription) await this.send(s, [sharer], SHARE.answer, pc.localDescription.sdp);
} catch {
if (this.watchPc === pc) this.setWatchState('failed');
if (current()) this.setWatchState(sharer, 'failed');
}
}
// Raw WebRTC statistics for the overlay: of the stream we watch, or one per viewer of ours
async reports(own: boolean): Promise<{ viewer: number | null; report: RTCStatsReport }[]> {
if (!own) return this.watchPc ? [{ viewer: null, report: await this.watchPc.getStats() }] : [];
// Raw WebRTC statistics for the overlay: of a stream we watch, or one per viewer of ours
async reports(own: boolean, session?: number): Promise<{ viewer: number | null; report: RTCStatsReport }[]> {
if (!own) {
const pc = session != null ? this.watchPcs.get(session) : undefined;
return pc ? [{ viewer: null, report: await pc.getStats() }] : [];
}
return Promise.all([...this.peers].map(async ([viewer, pc]) => ({ viewer, report: await pc.getStats() })));
}
@@ -292,11 +338,12 @@ class ScreenShare {
case SHARE.on:
if (!s.sharing[sender] && Date.now() - (this.enteredAt.get(s) ?? 0) > 3000) sounds.play('shareStarted');
s.setSharing(sender, msg.text === 'camera' ? 'camera' : 'screen');
if (msg.text === 'camera' && this.cams && this.camHost === s) this.watch(s, sender);
break;
case SHARE.off:
if (s.sharing[sender]) sounds.play('shareStopped');
s.setSharing(sender, false);
if (this.watching?.host === s && this.watching.session === sender) this.endWatch();
if (this.watchOf(sender)?.host === s) this.endWatch(sender);
break;
case SHARE.who: if (this.host === s) this.send(s, [sender], SHARE.on, this.kind); break;
case SHARE.watch: if (this.host === s) this.offerTo(s, sender).catch(() => this.dropPeer(sender)); break;
@@ -314,7 +361,7 @@ class ScreenShare {
const self = s.client!.session!;
this.enteredAt.set(s, Date.now());
s.clearSharing();
if (this.watching?.host === s) this.unwatch();
for (const w of this.watches) if (w.host === s) this.unwatch(w.session);
if (this.host === s) {
this.dropPeers();
s.setSharing(self, this.kind);
@@ -334,13 +381,14 @@ class ScreenShare {
if (s.sharing[user.session]) sounds.play('shareStopped');
s.setSharing(user.session, false);
if (this.host === s) this.dropPeer(user.session);
if (this.watching?.host === s && this.watching.session === user.session) this.endWatch();
if (this.watchOf(user.session)?.host === s) this.endWatch(user.session);
}
closed(s: Session): void {
if (this.host === s) this.release();
if (this.dialog === s) this.dialog = null;
if (this.watching?.host === s) this.endWatch();
for (const w of this.watches) if (w.host === s) this.endWatch(w.session);
if (this.camHost === s) { this.cams = false; this.camHost = null; }
}
}
+1 -1
View File
@@ -16,7 +16,7 @@
// its minimum; the picture gets the rest. Unset until dragged, then the picture decides.
const LOG_MIN = 124;
const here = (s: { server?: { id: string } | null } | null | undefined) => !!s && s.server?.id === session.server?.id;
const streamAbove = $derived(videoOnly && session.status === 'connected' && ((!!share.stream && here(share.host)) || (!!share.watching && here(share.watching.host))));
const streamAbove = $derived(videoOnly && session.status === 'connected' && ((!!share.stream && here(share.host)) || share.watches.some(w => here(w.host))));
let logHeight = $state(Number((() => { try { return localStorage.getItem('mumh5.stackedChatHeight'); } catch { return null; } })()) || 0);
const logFixed = $derived(streamAbove && logHeight >= LOG_MIN);
+1 -1
View File
@@ -131,7 +131,7 @@
<Icon name={session.sharing[user.session]} size={16} />
<span>{isSelf ? `You are sharing your ${session.sharing[user.session]}` : `Sharing their ${session.sharing[user.session]}`}</span>
{#if isSelf}<button class="btn" onclick={() => share.stop()}>Stop sharing</button>
{:else if share.watching?.session === user.session}<button class="btn" onclick={() => share.unwatch()}>Stop watching</button>
{:else if share.watchOf(user.session)}<button class="btn" onclick={() => share.unwatch(user.session)}>Stop watching</button>
{:else}<button class="btn primary" onclick={() => share.watch(sessions.active, user.session)}>Watch</button>{/if}
</div>
{/if}
+3 -2
View File
@@ -6,7 +6,8 @@
// "Stats for nerds" over the stream: codecs, picture, bitrate and the network path, read
// from the WebRTC connection once a second. own: our stream (one block per viewer).
let { own, onclose }: { own: boolean; onclose: () => void } = $props();
// session: whose stream, when it is one we watch
let { own, session: watched = null, onclose }: { own: boolean; session?: number | null; onclose: () => void } = $props();
type Row = [string, string];
let blocks = $state<{ title: string; rows: Row[] }[]>([]);
@@ -83,7 +84,7 @@
}
async function read() {
const reports = await share.reports(own);
const reports = await share.reports(own, watched ?? undefined);
blocks = reports.map(r => {
const user = r.viewer != null ? session.user(r.viewer) : undefined;
return { title: r.viewer == null ? '' : `To ${user ? store.displayName(user) : `#${r.viewer}`}`, rows: describe(r.report, String(r.viewer)) };
+101 -55
View File
@@ -13,8 +13,9 @@
import ShareStats from './ShareStats.svelte';
// People in your voice channel as tiles. Shown on windows too narrow for the member list
// (always), and on any window while someone in the channel shares a screen. A stream is shown
// large above the tiles, which become a strip; the tiles themselves never carry the picture.
// (always), and on any window while someone in the channel shares something. A screen is shown
// large above the tiles, which become a strip; cameras play in the tiles themselves, as many
// as are on, and a click enlarges one.
// videoOnly: just the picture, as wide as the chat (stacked layout, where the channel list
// above already shows who is here)
let { always = false, videoOnly = false }: { always?: boolean; videoOnly?: boolean } = $props();
@@ -24,20 +25,28 @@
const inChannel = $derived(self ? session.usersIn(self.channelId) : []);
const here = (s: { server?: { id: string } | null } | null | undefined) => !!s && s.server?.id === session.server?.id;
const watching = $derived(share.watching && here(share.watching.host) ? share.watching : null);
const watches = $derived(share.watches.filter(w => here(w.host)));
// The screen being watched; cameras are watched next to it
const watching = $derived(watches.find(w => w.kind === 'screen') ?? null);
const watchOf = (s: number) => watches.find(w => w.session === s);
const mine = $derived(share.stream && here(share.host) ? share.stream : null);
const anySharing = $derived(inChannel.some(u => session.sharing[u.session]));
const visible = $derived(!!self && (videoOnly ? !!watching || !!mine : always || anySharing || !!watching || !!mine));
const anyCamera = $derived(inChannel.some(u => session.sharing[u.session] === 'camera' && u.session !== self?.session));
const visible = $derived(!!self && (videoOnly ? watches.length > 0 || !!mine : always || anySharing || watches.length > 0 || !!mine));
// The picture a tile can show: our own stream, or the one we are watching
// The picture we have of someone: our own stream, or one we are watching
function streamOf(s: number): MediaStream | null {
if (s === self?.session) return mine;
return watching?.session === s ? watching.stream : null;
return watchOf(s)?.stream ?? null;
}
// A camera plays in the person's tile; a screen never does
const camOf = (s: number) => session.sharing[s] === 'camera' ? streamOf(s) : null;
// People with a stream we can show come first, then others who share
const rank = (s: number) => streamOf(s) ? 0 : session.sharing[s] ? 1 : 2;
const people = $derived([...inChannel].sort((a, b) => rank(a.session) - rank(b.session)));
// Who gets a tile: everyone, or in the stacked layout only the people whose camera we see
const shown = $derived(videoOnly ? people.filter(u => camOf(u.session)) : people);
// Tile on the large stage
let focus = $state<number | null>(null);
@@ -59,12 +68,17 @@
let hadMine = false;
$effect(() => {
const has = !!mine;
if (has && !hadMine && focus == null && self) focus = self.session;
if (has && !hadMine && focus == null && self && share.kind === 'screen') focus = self.session;
else if (!has && focus === self?.session) focus = null;
hadMine = has;
});
// Without tiles there is nothing to choose: the watched stream, else our own
$effect(() => { if (videoOnly) focus = watching ? watching.session : mine ? self?.session ?? null : null; });
let lastSolo: number | null = null;
$effect(() => {
if (!videoOnly) return;
const solo = watching ? watching.session : mine && share.kind === 'screen' ? self?.session ?? null : null;
if (solo !== lastSolo) { focus = solo; lastSolo = solo; }
});
// The stage is as tall as the user drags it; the tiles grow and shrink to fill it. The
// meeting view (one picture large) remembers its own height.
@@ -79,7 +93,7 @@
let gridW = $state(0), gridH = $state(0);
const GAP = 8, PAD = 8;
const cols = $derived.by(() => {
const n = people.length;
const n = shown.length;
const w = gridW - PAD * 2, h = gridH - PAD * 2;
if (n < 2 || w <= 0 || h <= 0) return 1;
let best = 1, bestSize = 0;
@@ -93,9 +107,9 @@
});
// In the meeting view everyone sits in one row under the picture
const rows = $derived.by(() => {
if (focused) return [people];
const out: (typeof people)[] = [];
for (let i = 0; i < people.length; i += cols) out.push(people.slice(i, i + cols));
if (focused) return [shown];
const out: (typeof shown)[] = [];
for (let i = 0; i < shown.length; i += cols) out.push(shown.slice(i, i + cols));
return out;
});
const avatar = $derived.by(() => {
@@ -126,14 +140,17 @@
return { update(next: MediaStream | null) { if (node.srcObject !== next) attach(next); } };
}
// Sound comes from one hidden player, so it plays once however many pictures are shown
let player = $state<HTMLAudioElement>();
const hasSound = $derived(!!watching?.stream?.getAudioTracks().length);
// Sound comes from hidden players, one per stream that has any, so it plays once however
// many pictures are shown. They all follow the stream volume.
const withSound = $derived(watches.filter(w => w.stream?.getAudioTracks().length));
const hasSound = $derived(focus != null && withSound.some(w => w.session === focus));
function loud(node: HTMLMediaElement) {
$effect(() => {
if (!player) return;
player.volume = store.settings.shareVolume;
player.muted = store.settings.shareVolume === 0;
node.volume = store.settings.shareVolume;
node.muted = store.settings.shareVolume === 0;
});
}
const focusWatch = $derived(focus != null ? watchOf(focus) : undefined);
function tileClick(session_: number) {
if (streamOf(session_)) focus = focus === session_ ? null : session_;
@@ -236,12 +253,12 @@
<button class="icon-btn" title="Open in a separate window" aria-label="Open in a separate window" onclick={popOut}><Icon name="popout" size={16} /></button>
<button class="icon-btn" title="Full screen" aria-label="Full screen" onclick={fullscreen}><Icon name="expand" size={16} /></button>
{/if}
{#if focus !== self.session}<button class="btn small" onclick={() => share.unwatch()}>Stop watching</button>
{#if focus !== self.session}<button class="btn small" onclick={() => { if (focus != null) share.unwatch(focus); }}>Stop watching</button>
{:else if videoOnly}
<span class="status" role="status" title="People watching your {share.kind}">{share.viewers} watching</span>
<button class="btn small" onclick={() => share.stop()}>Stop sharing</button>
{/if}
{#if !videoOnly}<button class="icon-btn" title="Back to tiles" aria-label="Back to tiles" onclick={() => (focus = null)}><Icon name="x" size={16} /></button>{/if}
{#if !videoOnly || shown.length}<button class="icon-btn" title="Back to tiles" aria-label="Back to tiles" onclick={() => (focus = null)}><Icon name="x" size={16} /></button>{/if}
</div>
<!-- svelte-ignore a11y_media_has_caption -->
{#if popped}
@@ -250,29 +267,74 @@
<div class="pic" role="presentation" oncontextmenu={streamMenu}
ontouchstart={touchStart} ontouchend={touchEnd} ontouchmove={touchEnd} ontouchcancel={touchEnd}>
<video bind:this={spotVideo} use:media={focused} autoplay playsinline muted ondblclick={fullscreen}></video>
{#if focus !== self.session && watching && watching.state !== 'live'}
<p class="pending" class:bad={watching.state === 'failed'}>{watching.state === 'failed'
{#if focus !== self.session && focusWatch && focusWatch.state !== 'live'}
<p class="pending" class:bad={focusWatch.state === 'failed'}>{focusWatch.state === 'failed'
? 'Could not connect to the stream. Across the internet both of you need a STUN server; networks that block direct connections need a relay. Both are in Settings, Voice.'
: 'Connecting to the stream...'}</p>
{/if}
{#if showStats}<ShareStats own={focus === self.session} onclose={() => (showStats = false)} />{/if}
{#if showStats}<ShareStats own={focus === self.session} session={focus} onclose={() => (showStats = false)} />{/if}
</div>
{/if}
</div>
{/if}
{/snippet}
{#snippet tiles()}
<div class="tiles" class:strip={!!focused} role="group" aria-label="People in this channel" bind:clientWidth={gridW} bind:clientHeight={gridH}>
{#each rows as row, r (r)}
<div class="row" role="presentation">
{#each row as u (u.session)}
{@const stream = streamOf(u.session)}
{@const sharing = session.sharing[u.session]}
{@const isSelf = u.session === self?.session}
{@const cam = camOf(u.session)}
{@const w = watchOf(u.session)}
<button class="tile" class:talking={talking(u.session)} class:self={isSelf} class:focused={focus === u.session}
onclick={() => tileClick(u.session)}
oncontextmenu={e => menu.show(e, userMenu(u))}
title={stream ? (focus === u.session ? 'Hide the stream' : 'Show the stream') : sharing && !isSelf ? `Sharing their ${sharing}. Click to watch` : store.displayName(u)}
aria-label={stream ? `${focus === u.session ? 'Hide' : 'Show'} the ${sharing || 'stream'} of ${store.displayName(u)}` : sharing && !isSelf ? `Watch the ${sharing} of ${store.displayName(u)}` : store.displayName(u)}>
{#if cam}
<!-- svelte-ignore a11y_media_has_caption -->
<video class:mirror={isSelf} use:media={cam} autoplay playsinline muted></video>
{:else}
<Avatar name={u.name} size={avatar} speaking={talking(u.session)} />
{/if}
<span class="who">{store.displayName(u)}</span>
{#if w && w.state !== 'live'}
<span class="note">{w.state === 'connecting' ? 'Connecting...' : 'Could not connect'}</span>
{/if}
{#if sharing}<span class="badge live"><Icon name={sharing} size={13} /></span>
{:else if u.deaf || u.selfDeaf}
<span class="badge" class:admin={u.deaf}><Icon name="headphones-off" size={13} /></span>
{:else if u.mute || u.selfMute || u.suppress}
<span class="badge" class:admin={u.mute || u.suppress}><Icon name="mic-off" size={13} /></span>
{/if}
</button>
{/each}
</div>
{/each}
</div>
{/snippet}
{#if visible && self && videoOnly}
<section class="stage solo" aria-label="Voice channel">
{#if watching?.stream}<audio bind:this={player} use:media={watching.stream} autoplay></audio>{/if}
{#each withSound as w (w.session)}<audio use:media={w.stream} use:loud autoplay></audio>{/each}
{#if focused}
{@render spot()}
{:else if watching}
<p class="wait" class:bad={watching.state === 'failed'}>
<span>{watching.state === 'failed' ? 'Could not connect to the stream. Across the internet both of you need a STUN server set in Settings, Voice.' : 'Connecting to the stream...'}</span>
<button class="btn small" onclick={() => share.unwatch()}>{watching.state === 'failed' ? 'Close' : 'Cancel'}</button>
<button class="btn small" onclick={() => share.unwatch(watching.session)}>{watching.state === 'failed' ? 'Close' : 'Cancel'}</button>
</p>
{/if}
<!-- Cameras: a grid of just the pictures -->
{#if shown.length && !focused}
<div class="camgrid" style:height="{tilesHeight}px">
<ResizeHandle bind:width={tilesHeight} edge="bottom" min={110} max={maxHeight} initial={210} storageKey="mumh5.stageHeight" label="Resize the tiles" />
{@render tiles()}
</div>
{/if}
</section>
{:else if visible && self}
<section class="stage" class:meeting={!!focused} class:collapsed aria-label="Voice channel"
@@ -295,6 +357,12 @@
</span>
{/if}
</button>
{#if anyCamera || share.cams}
<button class="icon-btn" class:live={share.cams} aria-pressed={share.cams} title={share.cams ? 'Stop watching the cameras' : 'Watch everyone\'s camera'}
aria-label={share.cams ? 'Stop watching the cameras' : 'Watch all cameras'} onclick={() => share.setCams(sessions.active, !share.cams)}>
<Icon name="camera" size={18} />
</button>
{/if}
{#if mine}
<span class="status" role="status" title="People watching your {share.kind}"><Icon name={share.kind} size={14} />{share.viewers} watching</span>
<button class="btn small" onclick={() => share.stop()}>Stop sharing</button>
@@ -309,40 +377,13 @@
</button>
</header>
{#if watching?.stream}<audio bind:this={player} use:media={watching.stream} autoplay></audio>{/if}
{#each withSound as w (w.session)}<audio use:media={w.stream} use:loud autoplay></audio>{/each}
{#if focused && focus != null && !collapsed}{@render spot()}{/if}
{#if !collapsed}
<div class="tiles" class:strip={!!focused} role="group" aria-label="People in this channel" bind:clientWidth={gridW} bind:clientHeight={gridH}>
{#each rows as row, r (r)}
<div class="row" role="presentation">
{#each row as u (u.session)}
{@const stream = streamOf(u.session)}
{@const sharing = session.sharing[u.session]}
{@const isSelf = u.session === self.session}
<button class="tile" class:talking={talking(u.session)} class:self={isSelf} class:focused={focus === u.session}
onclick={() => tileClick(u.session)}
oncontextmenu={e => menu.show(e, userMenu(u))}
title={stream ? (focus === u.session ? 'Hide the stream' : 'Show the stream') : sharing && !isSelf ? `Sharing their ${sharing}. Click to watch` : store.displayName(u)}
aria-label={stream ? `${focus === u.session ? 'Hide' : 'Show'} the ${sharing || 'stream'} of ${store.displayName(u)}` : sharing && !isSelf ? `Watch the ${sharing} of ${store.displayName(u)}` : store.displayName(u)}>
<Avatar name={u.name} size={avatar} speaking={talking(u.session)} />
<span class="who">{store.displayName(u)}</span>
{#if watching?.session === u.session && watching.state !== 'live'}
<span class="note">{watching.state === 'connecting' ? 'Connecting...' : 'Could not connect'}</span>
{/if}
{#if sharing}<span class="badge live"><Icon name={sharing} size={13} /></span>
{:else if u.deaf || u.selfDeaf}
<span class="badge" class:admin={u.deaf}><Icon name="headphones-off" size={13} /></span>
{:else if u.mute || u.selfMute || u.suppress}
<span class="badge" class:admin={u.mute || u.suppress}><Icon name="mic-off" size={13} /></span>
{/if}
</button>
{/each}
</div>
{/each}
</div>
{#if watching?.state === 'failed'}
{@render tiles()}
{#if watches.some(w => w.state === 'failed')}
<p class="failed">Could not connect to the stream. Across the internet both of you need a STUN server, chosen in Settings, Voice (the person sharing has to restart the stream afterwards).</p>
{/if}
{/if}
@@ -368,6 +409,11 @@
.tile:hover { background: var(--bg-hover); }
.tile.talking { border-color: var(--speaking); box-shadow: 0 0 0 1px var(--speaking), 0 0 18px -4px var(--speaking); }
.tile.focused { border-color: var(--accent); }
/* A camera fills its tile; your own is mirrored, the way a mirror shows you */
.tile video { position: absolute; inset: 0; width: 100%; height: 100%; object-fit: cover; background: #000; }
.tile video.mirror { transform: scaleX(-1); }
.icon-btn.live, .icon-btn.live:hover { color: var(--speaking); }
.camgrid { position: relative; display: flex; flex-direction: column; flex: none; }
/* Name tag over the lower left corner */
.who { position: absolute; left: 4px; bottom: 4px; max-width: calc(100% - 8px); padding: 1px 6px; font-size: 12px; color: #fff; background: rgb(0 0 0 / 0.6);
white-space: nowrap; overflow: hidden; text-overflow: ellipsis; }
+34 -2
View File
@@ -268,15 +268,47 @@ try {
assert.equal(await picker.getByLabel('Sound').count(), 0, 'no extra sound with a camera');
await startBtn.click();
await alice.page.locator('.sidebar .row.user.self').getByTitle('You are sharing your camera').waitFor();
// Cameras play in the tiles. Bob joins the camera round with one click and sees Alice's
const cam = bob.page.locator('.sidebar').getByRole('button', { name: `Watch the camera of ${aliceName}`, exact: true });
await cam.waitFor();
await cam.click();
const liveCams = (p: Page) => p.evaluate(() => [...document.querySelectorAll('.stage .tile video')].filter(v => (v as HTMLVideoElement).videoWidth > 0).length);
const camsShown = (p: Page, n: number) => p.waitForFunction(
want => [...document.querySelectorAll('.stage .tile video')].filter(v => (v as HTMLVideoElement).videoWidth > 0).length === want, n, { timeout: 30000 });
await bob.page.locator('.stage').getByRole('button', { name: 'Watch all cameras' }).click();
await camsShown(bob.page, 1);
assert.equal(await bob.page.getByRole('region', { name: 'Screen share' }).count(), 0, 'a camera does not take over the stage by itself');
// Bob turns his camera on too: both now see both, each one their own mirrored
await bob.page.locator('.me').getByRole('button', { name: 'Share your screen or camera' }).click();
const bobDialog = bob.page.getByRole('dialog', { name: 'Share your screen' });
await bobDialog.getByRole('tab', { name: 'Camera' }).click();
await bob.page.waitForFunction(() => (document.querySelector('.preview video') as HTMLVideoElement | null)?.videoWidth! > 0, null, { timeout: 15000 });
await bobDialog.getByRole('button', { name: 'Start sharing' }).click();
await camsShown(bob.page, 2);
await camsShown(alice.page, 2);
assert.equal(await bob.page.locator('.stage .tile.self video.mirror').count(), 1, 'own camera is mirrored');
assert.equal(await bob.page.locator('.stage .tile:not(.self) video.mirror').count(), 0);
await alice.page.locator('.stage .status', { hasText: '1 watching' }).waitFor();
await bob.page.locator('.stage .status', { hasText: '1 watching' }).waitFor();
if (process.env.SHOTS_DIR) await bob.page.screenshot({ path: `${process.env.SHOTS_DIR}/cams.png` });
// A click enlarges one camera; the statistics show it came through the relay
await bob.page.locator('.stage .tile', { hasText: aliceName }).click();
await bob.page.waitForFunction(() => (document.querySelector('.spot video') as HTMLVideoElement | null)?.videoWidth! > 0, null, { timeout: 20000 });
const camStats = await nerd(bob.page, [/^relayed \(TURN\), reached over TCP/, /^0 direct, 0 through STUN, [1-9]\d* relayed/]);
await bob.page.getByRole('region', { name: 'Screen share' }).getByRole('button', { name: 'Back to tiles' }).click();
// Leaving the round keeps Bob's own camera on and drops the others
await bob.page.locator('.stage').getByRole('button', { name: 'Stop watching the cameras' }).click();
await camsShown(bob.page, 1);
await alice.page.locator('.stage .status', { hasText: '0 watching' }).waitFor();
await bob.page.locator('.me').getByRole('button', { name: 'Stop sharing your camera' }).click();
await camsShown(alice.page, 1);
await alice.page.locator('.me').getByRole('button', { name: 'Stop sharing your camera' }).click();
await cam.waitFor({ state: 'detached' });
await bob.page.evaluate(() => localStorage.removeItem('mumh5.iceDebug'));
console.log(`ok: camera shared, watched through the relay set in the desktop app (${camStats.match(/Picture \| [^|]+/)?.[0]})`);
console.log(`ok: cameras of several people at once, in the tiles, through the relay (${camStats.match(/Picture \| [^|]+/)?.[0]})`);
void liveCams;
// A sharer who disconnects takes the stream with them
await shareBtn.click();
+2 -2
View File
@@ -220,9 +220,9 @@ try {
await page.waitForFunction(() => (document.querySelector('.preview video') as HTMLVideoElement | null)?.videoWidth! > 0, null, { timeout: 30000 });
await dialog.getByRole('button', { name: 'Start sharing' }).click();
await page2.locator('.sidebar').getByRole('button', { name: `Watch the camera of ${name}`, exact: true }).click();
await page2.waitForFunction(() => (document.querySelector('.spot video') as HTMLVideoElement | null)?.videoWidth! > 0, null, { timeout: 30000 });
await page2.waitForFunction(() => (document.querySelector('.stage .tile video') as HTMLVideoElement | null)?.videoWidth! > 0, null, { timeout: 30000 });
await page.locator('.stage .status', { hasText: '1 watching' }).waitFor();
console.log('ok: camera shared between two browsers, through the relay');
console.log('ok: camera shared between two browsers, shown in the tile, through the relay');
console.log('WEB E2E PASSED');
} finally {