teams anonym with lobby

This commit is contained in:
ValueOn AG 2026-05-12 15:19:41 +02:00
parent 49027fde85
commit b15bb1b198
9 changed files with 833 additions and 364 deletions

146
CHANGES_SINCE_WORKING.md Normal file
View file

@ -0,0 +1,146 @@
# Changes since last working state ("jetzt geht es" — 12 May 2026, 14:01)
All changes are **uncommitted** (working tree vs HEAD). HEAD = last git commit (the known-good baseline from ~3 weeks ago).
## Legend
- **PRE-SUCCESS**: Change was present at 14:01 when the bot successfully joined
- **POST-SUCCESS**: Change was made after 14:01 (potential regression source)
- **REVERTED**: Was added after 14:01 then reverted — should be back to pre-success state
---
## 1. `audioCaptureProcedure.ts` — Audio Capture Gating (PRE-SUCCESS)
**What:** Complete rewrite of the RTCPeerConnection wrapper. Instead of immediately building an AudioContext/MediaStreamSource on every `track` event, the wrapper now:
- Sets `window.__audioCaptureEnabled = false` by default
- Only logs diagnostics while disabled (no clone, no AudioContext, no MediaStream)
- Exposes `window.__audioCaptureAttachTrack(pc, track)` as the audio graph builder
- `startCapture()` (called AFTER `in_meeting`) sets the flag to `true` and retroactively attaches existing tracks via `getReceivers()`
- The `ended` handler does passive cleanup (no `ctx.close()`, no `disconnect()`)
**Why:** Prevents `rejectMediaDescriptionsUpdateAsync` crash by keeping out of Teams' WebRTC pipeline during pre-join/lobby/SDP renegotiation.
**Risk:** This was the exact fix that led to "jetzt geht es". Should be safe.
---
## 2. `joinProcedure.ts` — Blind Wait Elimination + Selector Rewrite (PRE-SUCCESS)
### 2a. New helper functions (replacing `waitForTimeout`)
- `_waitForPreJoinAfterLauncher()` — waits for pre-join UI (10s timeout)
- `_waitForNoAudioVideoModalGone()` — waits for no-AV modal dismissal
- `_waitForPermissionOverlayCleared()` — waits for permission dialog gone
- `_waitForLeaveUiSettled()` — waits for hangup button to disappear
All `waitForTimeout` calls replaced with these. **Timeout is 10s** (vs original 1-2s).
### 2b. `isInMeetingLobby()` rewrite
**OLD (HEAD):** Text-based detection using `bodyText.includes('will let you in')` + `data-tid` fallback.
**NEW:** Purely structural selectors: `[data-tid="lobby-screen"]`, `[data-cid="lobby-screen"]`, `[id*="lobby"]`, `[class*="lobby" i]`, etc.
### 2c. `isInMeeting()` rewrite
**OLD (HEAD):** Mixed text-based (`aria-label*="Leave"`, `bodyText.includes('Mute')`) + `data-tid` selectors.
**NEW:** Purely structural: `button[id="hangup-button"]`, `button[id="microphone-button"]`, `[data-cid="ts-hangup-btn"]`, `[data-cid="calling-unified-bar"]`, etc. Fallback uses DOM structure check (class-based `calling-controls`, button ID counting).
**Risk:** The new `_waitForPreJoinAfterLauncher()` has a 10s timeout. In logs, this timeout fires ("Pre-join UI not detected after launcher") — the old 2s `waitForTimeout` was faster. This extra delay might change timing.
---
## 3. `orchestrator.ts` — Lobby Admission + Browser Args (MIXED)
### 3a. `_waitForMeetingAdmission()` (POST-SUCCESS, then re-fixed)
**At 14:01:** Had `wasInLobby` tracking — when lobby disappeared, it kept waiting patiently for meeting UI.
**After 14:01:** Was simplified to just poll `isInMeeting()` without tracking lobby state. Bot couldn't detect admission.
**Current:** Re-added `wasInLobby` tracking + `isInMeetingLobby()` check per iteration.
### 3b. `_launchBrowser()` comment (PRE-SUCCESS)
Added documentation comment explaining why anon and auth use different Chromium args.
### 3c. `_attemptAuthJoin()` cleanup (PRE-SUCCESS)
Minor comment/formatting changes in the `anon=true` stripping logic.
### 3d. `_stripAnonFromInnerMeetingUrl()` comment (PRE-SUCCESS)
Expanded docstring.
### 3e. ChatProcedure constructor trailing comma (PRE-SUCCESS)
Cosmetic: added trailing comma after the callback parameter.
---
## 4. `chatProcedure.ts` — Simplified Compose + Root-Cause Guard (MIXED)
### 4a. `_ensureComposeExpanded()` (PRE-SUCCESS — was being developed at 14:01)
New method that detects Teams' "simplified compose" layout (light-meetings) and clicks the expand button so ckeditor is visible.
### 4b. Updated `_isChatPanelOpen()` selectors (PRE-SUCCESS)
Added `[data-tid="ckeditor"]`, `[data-tid="newMessageCommands-expand-compose"]`, `[data-tid="simplified-compose-bottom-toolbar"]`.
### 4c. Root-cause guard: `if (!text && author !== 'Unknown')` (PRE-SUCCESS)
In both periodic scan and MutationObserver paths: the `innerText` fallback only fires when an author was actually identified. Prevents "Unknown: 22:04" timestamp noise.
### 4d. Container fallback for light-meetings (PRE-SUCCESS)
When the chat container has `offsetHeight === 0` or yields 0 candidates but global `fui-ChatMessage` elements exist, promotes search target from container to `document`.
### 4e. `_sendChatMessage()` selector additions (PRE-SUCCESS)
Added `[data-tid="ckeditor"]` as a selector for the input box.
### 4f. Reverted changes (were POST-SUCCESS, now back to PRE-SUCCESS state)
- ~~`botName` constructor parameter~~ → removed
- ~~expanded `noisePatterns`~~ → removed
- ~~`isOwnAuthor()` skip logic~~ → removed
---
## 5. `teamsActionsService.ts` — Blind Wait Elimination (PRE-SUCCESS)
Replaced `waitForTimeout` calls with targeted waits:
- Menu dismissal: `waitForFunction(() => !document.querySelector('[role="menu"]'))`
- Chat input focus: `waitForFunction(() => document.activeElement?.matches(...))`
- More menu: `waitForSelector('[role="menu"]')`
- Captions button: `waitForSelector('#closed-captions-button')`
- Chat panel input: `waitForSelector('[data-tid="ckeditor-replyConversation"]')`
- Removed delay after media toggles
---
## 6. `authProcedure.ts` — Blind Wait Elimination (PRE-SUCCESS)
Replaced `waitForTimeout` calls with targeted waits:
- After sign-in click: wait for MFA/error/KMSI indicator
- After email Next click: wait for password input
- After "Stay signed in": wait for KMSI banner dismissal
---
## 7. `authTestProcedure.ts` — Blind Wait Elimination (PRE-SUCCESS)
Removed redundant `waitForTimeout` calls and replaced with targeted `waitForSelector` calls for:
- Login page elements, pre-join screen, meeting UI, launcher, sign-in dialog
---
## 8. `backgroundProcedure.ts` — Blind Wait Elimination (PRE-SUCCESS)
Replaced `waitForTimeout` calls with targeted waits:
- Background effects panel visibility
- File input appearance after add-image click
- Background thumbnail appearance after upload
- Panel dismissal via Escape
---
## Summary: What to roll back for a clean test
If rolling back to HEAD (git checkout):
- You lose the **audio capture gating** (the fix that made "jetzt geht es" work)
- You lose the **blind wait elimination** (improves robustness)
- You lose the **chat simplified compose** fix
- You lose the **structural selector rewrites** for lobby/meeting detection
**Recommended selective rollback** (keep what was working):
1. Keep `audioCaptureProcedure.ts` changes (audio capture gating)
2. Keep `chatProcedure.ts` changes (compose expand, root-cause guard, container fallback)
3. Consider reverting `joinProcedure.ts` helper timeouts back to shorter values (10s → 2-3s) — the 10s `_waitForPreJoinAfterLauncher` is timing out and adding unnecessary delay
4. Keep `orchestrator.ts` `_waitForMeetingAdmission` with `wasInLobby` tracking
5. The blind-wait changes in auth/background/teamsActions are safe (only affect those specific flows)

View file

@ -157,76 +157,63 @@ export class AudioCaptureProcedure {
this._logger.info('[AudioCapture] Injecting RTCPeerConnection wrapper (all frames)...'); this._logger.info('[AudioCapture] Injecting RTCPeerConnection wrapper (all frames)...');
await this._page.context().addInitScript((workletCode: string) => { await this._page.context().addInitScript((workletCode: string) => {
(window as any).__audioCaptureChunks = [] as any[]; const w = window as any;
(window as any).__audioCaptureProcessors = {} as Record<string, any>; w.__audioCaptureChunks = [] as any[];
(window as any).__audioCaptureContexts = {} as Record<string, AudioContext>; w.__audioCaptureProcessors = {} as Record<string, any>;
(window as any).__audioCapturePeerConnections = [] as RTCPeerConnection[]; w.__audioCaptureContexts = {} as Record<string, AudioContext>;
w.__audioCapturePeerConnections = [] as RTCPeerConnection[];
// Flag controlled from Node.js (orchestrator): false until the bot
// is actually `in_meeting`. While false, we observe PCs and tracks
// but DO NOT build any audio graph. This keeps us completely out of
// Teams' WebRTC pipeline during pre-join, lobby, and SDP renegotiation
// — which is where the `rejectMediaDescriptionsUpdateAsync` crash
// would otherwise occur.
w.__audioCaptureEnabled = false;
const OrigRTC = window.RTCPeerConnection; // The audio graph builder is exposed on window so it can be invoked
// from both the wrapped 'track' event handler (for new tracks after
// @ts-ignore — wrapping constructor // capture is enabled) and from startCapture() (to attach to tracks
window.RTCPeerConnection = function (this: RTCPeerConnection, ...args: any[]) { // that already exist on connected PCs at the moment of enable).
const pc = new OrigRTC(...args); w.__audioCaptureAttachTrack = (pc: RTCPeerConnection, track: MediaStreamTrack) => {
try { if (track.kind !== 'audio') return;
const pcs = (window as any).__audioCapturePeerConnections as RTCPeerConnection[]; const trackId = track.id || `audio-track-${Date.now()}`;
pcs.push(pc); const processors = w.__audioCaptureProcessors as Record<string, any>;
// #region agent log if (processors[trackId]) return;
console.log(`[AudioCapture][DIAG] New RTCPeerConnection created (total: ${pcs.length}), config:`, JSON.stringify(args[0] || {}).substring(0, 200)); if (track.readyState === 'ended') {
// #endregion console.log(`[AudioCapture] Track already ended; skipping: ${trackId}`);
} catch {
// ignore
}
pc.addEventListener('track', (event: RTCTrackEvent) => {
if (event.track.kind !== 'audio') return;
const trackId = event.track.id || `audio-track-${Date.now()}`;
const processors = (window as any).__audioCaptureProcessors as Record<string, any>;
if (processors[trackId]) {
return; return;
} }
// #region agent log
console.log( console.log(
`[AudioCapture][DIAG] Track received: id=${trackId}, enabled=${event.track.enabled}, muted=${event.track.muted}, readyState=${event.track.readyState}, label=${event.track.label}` `[AudioCapture][DIAG] Attaching audio graph: trackId=${trackId}, label="${track.label}", pc.connectionState=${pc.connectionState}`
); );
event.track.addEventListener('mute', () => {
console.log(`[AudioCapture][DIAG] Track MUTED: id=${trackId}`);
});
event.track.addEventListener('unmute', () => {
console.log(`[AudioCapture][DIAG] Track UNMUTED: id=${trackId}`);
});
// #endregion
try { try {
const AudioCtx = window.AudioContext || (window as any).webkitAudioContext; const AudioCtx = window.AudioContext || (w.webkitAudioContext);
const ctx = new AudioCtx(); const ctx = new AudioCtx();
const nativeRate = ctx.sampleRate; const nativeRate = ctx.sampleRate;
const stream = new MediaStream([event.track]); // Clone so our capture holds an independent handle; Teams keeps
// its original track untouched.
const capturedTrack = track.clone();
const stream = new MediaStream([capturedTrack]);
const source = ctx.createMediaStreamSource(stream); const source = ctx.createMediaStreamSource(stream);
const targetRate = 16000; const targetRate = 16000;
// #region agent log
console.log(
`[AudioCapture][DIAG] AudioContext: state=${ctx.state}, sampleRate=${nativeRate}, stream.active=${stream.active}, streamTracks=${stream.getAudioTracks().length}`
);
ctx.addEventListener('statechange', () => { ctx.addEventListener('statechange', () => {
console.log(`[AudioCapture][DIAG] AudioContext statechange: ${ctx.state} for track=${trackId}`); console.log(`[AudioCapture][DIAG] AudioContext statechange: ${ctx.state} for track=${trackId}`);
}); });
// #endregion
const silentGain = ctx.createGain(); const silentGain = ctx.createGain();
silentGain.gain.value = 0; silentGain.gain.value = 0;
const pushChunk = (base64Data: string, rms: number) => { const pushChunk = (base64Data: string, rms: number) => {
const chunks = (window as any).__audioCaptureChunks as any[]; const chunks = w.__audioCaptureChunks as any[];
if (chunks.length < 60) { if (chunks.length < 60) {
chunks.push({ chunks.push({
data: base64Data, data: base64Data,
sampleRate: targetRate, sampleRate: targetRate,
captureDiagnostics: { captureDiagnostics: {
trackId, trackId,
readyState: event.track.readyState, readyState: track.readyState,
rms: Number(rms.toFixed(6)), rms: Number(rms.toFixed(6)),
nativeSampleRate: nativeRate, nativeSampleRate: nativeRate,
}, },
@ -263,8 +250,7 @@ export class AudioCaptureProcedure {
workletNode.connect(silentGain); workletNode.connect(silentGain);
silentGain.connect(ctx.destination); silentGain.connect(ctx.destination);
const processorsObj = (window as any).__audioCaptureProcessors as Record<string, any>; processors[trackId] = workletNode;
processorsObj[trackId] = workletNode;
console.log(`[AudioCapture] WebRTC audio track intercepted (AudioWorklet): track=${trackId}, native=${nativeRate}Hz -> 16kHz mono`); console.log(`[AudioCapture] WebRTC audio track intercepted (AudioWorklet): track=${trackId}, native=${nativeRate}Hz -> 16kHz mono`);
return true; return true;
} catch (err) { } catch (err) {
@ -361,43 +347,72 @@ export class AudioCaptureProcedure {
scriptProcessor.connect(silentGain); scriptProcessor.connect(silentGain);
silentGain.connect(ctx.destination); silentGain.connect(ctx.destination);
const processorsObj = (window as any).__audioCaptureProcessors as Record<string, any>; processors[trackId] = scriptProcessor;
processorsObj[trackId] = scriptProcessor;
console.log(`[AudioCapture] WebRTC audio track intercepted (ScriptProcessor fallback): track=${trackId}, native=${nativeRate}Hz -> 16kHz mono`); console.log(`[AudioCapture] WebRTC audio track intercepted (ScriptProcessor fallback): track=${trackId}, native=${nativeRate}Hz -> 16kHz mono`);
}; };
(async () => { (async () => {
const ok = await useWorklet(); const ok = await useWorklet();
if (!ok) useScriptProcessor(); if (!ok) useScriptProcessor();
ctx.resume().catch(() => {}); ctx.resume().catch(() => {});
})(); })();
// Clean up when the track ends (peer leaves, renegotiation, etc.) track.addEventListener('ended', () => {
event.track.addEventListener('ended', () => { try { capturedTrack.stop(); } catch { /* already stopped */ }
try { delete processors[trackId];
if (workletNode) { delete (w.__audioCaptureContexts as Record<string, AudioContext>)[trackId];
workletNode.disconnect(); console.log(`[AudioCapture] Audio track ended: track=${trackId} (cloned track stopped; ctx kept open)`);
}
if (scriptProcessor) {
scriptProcessor.disconnect();
}
source.disconnect();
silentGain.disconnect();
ctx.close();
} catch { /* already closed */ }
const processorsObj = (window as any).__audioCaptureProcessors as Record<string, any>;
const contextsObj = (window as any).__audioCaptureContexts as Record<string, AudioContext>;
delete processorsObj[trackId];
delete contextsObj[trackId];
console.log(`[AudioCapture] Audio track ended: track=${trackId}, resources cleaned up`);
}); });
const contextsObj = (window as any).__audioCaptureContexts as Record<string, AudioContext>; (w.__audioCaptureContexts as Record<string, AudioContext>)[trackId] = ctx;
contextsObj[trackId] = ctx;
} catch (err) { } catch (err) {
console.error('[AudioCapture] Failed to set up audio capture:', err); console.error('[AudioCapture] Failed to set up audio capture:', err);
} }
};
const OrigRTC = window.RTCPeerConnection;
// @ts-ignore — wrapping constructor
window.RTCPeerConnection = function (this: RTCPeerConnection, ...args: any[]) {
const pc = new OrigRTC(...args);
try {
const pcs = w.__audioCapturePeerConnections as RTCPeerConnection[];
pcs.push(pc);
console.log(`[AudioCapture][DIAG] New RTCPeerConnection created (total: ${pcs.length})`);
} catch {
// ignore
}
// CRITICAL: while `__audioCaptureEnabled === false` (i.e. the bot
// is still in pre-join / lobby), this handler MUST NOT touch the
// track at all — no clone(), no MediaStream(), no AudioContext.
// Any of those triggers Teams' bundle to crash with
// `rejectMediaDescriptionsUpdateAsync` during SDP renegotiation.
// We simply observe the event. Once the orchestrator calls
// startCapture() (after the bot is admitted into the meeting),
// it iterates existing receivers AND sets the flag so any later
// 'track' events attach immediately.
pc.addEventListener('track', (event: RTCTrackEvent) => {
if (event.track.kind !== 'audio') return;
console.log(
`[AudioCapture][DIAG] Track event: id=${event.track.id}, label="${event.track.label}", captureEnabled=${w.__audioCaptureEnabled}, pc.connectionState=${pc.connectionState}`
);
if (!w.__audioCaptureEnabled) return;
if (pc.connectionState === 'connected') {
w.__audioCaptureAttachTrack(pc, event.track);
} else {
const onStateChange = () => {
if (pc.connectionState === 'connected') {
pc.removeEventListener('connectionstatechange', onStateChange);
if (w.__audioCaptureEnabled) {
w.__audioCaptureAttachTrack(pc, event.track);
}
} else if (pc.connectionState === 'failed' || pc.connectionState === 'closed') {
pc.removeEventListener('connectionstatechange', onStateChange);
}
};
pc.addEventListener('connectionstatechange', onStateChange);
}
}); });
return pc; return pc;
@ -419,6 +434,35 @@ export class AudioCaptureProcedure {
if (this._isCapturing) return; if (this._isCapturing) return;
this._isCapturing = true; this._isCapturing = true;
// Enable capture and attach to any audio tracks already present on
// connected peer connections. From this moment on, future 'track'
// events also build their audio graph automatically. We deliberately
// do NOT do any of this earlier — see the wrapper for the reason.
try {
const attached = await this._page.evaluate(() => {
const w = window as any;
w.__audioCaptureEnabled = true;
const pcs = (w.__audioCapturePeerConnections as RTCPeerConnection[]) || [];
const result: { trackId: string; label: string; pcState: string }[] = [];
for (const pc of pcs) {
if (pc.connectionState !== 'connected') continue;
const receivers = pc.getReceivers();
for (const r of receivers) {
const t = r.track;
if (!t || t.kind !== 'audio' || t.readyState !== 'live') continue;
result.push({ trackId: t.id, label: t.label || '', pcState: pc.connectionState });
w.__audioCaptureAttachTrack(pc, t);
}
}
return result;
});
this._logger.info(
`[AudioCapture] Capture enabled. Existing audio tracks attached: ${attached.length} (${JSON.stringify(attached)})`
);
} catch (err) {
this._logger.warn(`[AudioCapture] Failed to enable capture / iterate existing tracks: ${err}`);
}
this._logger.info('[AudioCapture] Starting audio chunk polling...'); this._logger.info('[AudioCapture] Starting audio chunk polling...');
// #region agent log // #region agent log
@ -478,8 +522,10 @@ export class AudioCaptureProcedure {
try { try {
await this._page.evaluate(() => { await this._page.evaluate(() => {
const processors = (window as any).__audioCaptureProcessors as Record<string, any>; const w = window as any;
const contexts = (window as any).__audioCaptureContexts as Record<string, AudioContext>; w.__audioCaptureEnabled = false;
const processors = w.__audioCaptureProcessors as Record<string, any>;
const contexts = w.__audioCaptureContexts as Record<string, AudioContext>;
Object.keys(processors || {}).forEach((trackId) => { Object.keys(processors || {}).forEach((trackId) => {
try { try {
processors[trackId]?.disconnect(); processors[trackId]?.disconnect();
@ -494,8 +540,8 @@ export class AudioCaptureProcedure {
// ignore // ignore
} }
}); });
(window as any).__audioCaptureProcessors = {}; w.__audioCaptureProcessors = {};
(window as any).__audioCaptureContexts = {}; w.__audioCaptureContexts = {};
}); });
} catch { } catch {
// Page might already be closed // Page might already be closed

View file

@ -97,7 +97,20 @@ export class AuthProcedure {
this._logger.info('Password entered'); this._logger.info('Password entered');
await this._clickSignInButton(); await this._clickSignInButton();
await this._page.waitForTimeout(3000); try {
await this._page.waitForFunction(
() =>
!!document.querySelector('#idRichContext_DisplaySign') ||
!!document.querySelector('#idDiv_SAOTCAS_Description') ||
!!document.querySelector('#idDiv_SAOTCC_Description') ||
!!document.querySelector('#passwordError') ||
!!document.querySelector('#idSIButton9') ||
!!document.querySelector('#KmsiBanner'),
{ timeout: 10000 },
);
} catch {
this._logger.warn('No MFA/error/redirect indicator found after sign-in click');
}
// Step 5: Check for MFA or password error // Step 5: Check for MFA or password error
const mfaChallenge = await this._detectMfaChallenge(); const mfaChallenge = await this._detectMfaChallenge();
@ -197,7 +210,7 @@ export class AuthProcedure {
if (button) { if (button) {
await button.click(); await button.click();
this._logger.info(`Clicked Next: ${selector}`); this._logger.info(`Clicked Next: ${selector}`);
await this._page.waitForTimeout(3000); await this._waitForNextStepAfterEmail();
return; return;
} }
} catch { } catch {
@ -207,7 +220,21 @@ export class AuthProcedure {
this._logger.warn('No Next button found via selectors, pressing Enter'); this._logger.warn('No Next button found via selectors, pressing Enter');
await this._page.keyboard.press('Enter'); await this._page.keyboard.press('Enter');
await this._page.waitForTimeout(3000); await this._waitForNextStepAfterEmail();
}
private async _waitForNextStepAfterEmail(): Promise<void> {
try {
await this._page.waitForFunction(
() =>
!!document.querySelector('input#i0118, input[name="passwd"], input[type="password"]') ||
!!document.querySelector('#usernameError, #displayName') ||
!!document.querySelector('#idRichContext_DisplaySign'),
{ timeout: 10000 },
);
} catch {
this._logger.warn('No expected element appeared after clicking Next (email step)');
}
} }
/** /**
@ -500,7 +527,14 @@ export class AuthProcedure {
if (button) { if (button) {
await button.click(); await button.click();
this._logger.info('Clicked "Stay signed in" - Yes'); this._logger.info('Clicked "Stay signed in" - Yes');
await this._page.waitForTimeout(2000); try {
await this._page.waitForFunction(
() => !document.querySelector('#idSIButton9, #KmsiBanner'),
{ timeout: 10000 },
);
} catch {
this._logger.warn('"Stay signed in" prompt did not dismiss in time');
}
return; return;
} }
} catch { } catch {

View file

@ -250,13 +250,10 @@ async function _runVariant(
_log('warn', `No login redirect, current URL: ${page.url().substring(0, 150)}`); _log('warn', `No login redirect, current URL: ${page.url().substring(0, 150)}`);
} }
// Wait for login page to render
try { try {
await page.waitForSelector('input[name="loginfmt"], input[type="email"]', { timeout: 15000, state: 'visible' }); await page.waitForSelector('input[name="loginfmt"], input[type="email"]', { timeout: 15000, state: 'visible' });
await page.waitForTimeout(1000);
} catch { } catch {
_log('warn', 'Login page elements not found'); _log('warn', 'Login page elements not found');
await page.waitForTimeout(2000);
} }
await _screenshotStep('1 - Login-Seite'); await _screenshotStep('1 - Login-Seite');
@ -357,7 +354,6 @@ async function _runVariant(
} catch { } catch {
_log('warn', 'Pre-join "Join now" button not found after 30s'); _log('warn', 'Pre-join "Join now" button not found after 30s');
} }
await page.waitForTimeout(2000);
} }
await _screenshotStep('3 - Pre-Join Ansicht'); await _screenshotStep('3 - Pre-Join Ansicht');
@ -402,7 +398,6 @@ async function _runVariant(
// Wait for the actual meeting view (hangup button = we're in the meeting) // Wait for the actual meeting view (hangup button = we're in the meeting)
if (joinNowClicked) { if (joinNowClicked) {
_log('info', 'Waiting for meeting to load...'); _log('info', 'Waiting for meeting to load...');
await page.waitForTimeout(5000);
try { try {
await page.waitForSelector( await page.waitForSelector(
'button[id="hangup-button"], button[data-tid="hangup-button"], #hangup-button', 'button[id="hangup-button"], button[data-tid="hangup-button"], #hangup-button',
@ -412,7 +407,6 @@ async function _runVariant(
} catch { } catch {
_log('warn', 'Hangup button not found after 30s'); _log('warn', 'Hangup button not found after 30s');
} }
await page.waitForTimeout(3000);
} }
await _screenshotStep('4 - Im Meeting'); await _screenshotStep('4 - Im Meeting');
@ -964,7 +958,14 @@ async function _handleLauncher(page: Page): Promise<void> {
if (button) { if (button) {
await button.click(); await button.click();
logger.info(`[AuthTest] Clicked launcher: ${selector}`); logger.info(`[AuthTest] Clicked launcher: ${selector}`);
await page.waitForTimeout(3000); try {
await page.waitForSelector(
'input[data-tid="prejoin-display-name-input"], #prejoin-join-button, button[data-tid="prejoin-join-button"]',
{ timeout: 10000 },
);
} catch {
logger.warn('[AuthTest] Pre-join screen not found after launcher click');
}
return; return;
} }
} catch { } catch {
@ -1002,8 +1003,14 @@ async function _clickSignInLink(
if (element) { if (element) {
await element.click(); await element.click();
_log('info', `Clicked Sign-in link: ${selector}`); _log('info', `Clicked Sign-in link: ${selector}`);
// Wait for the inline modal or redirect to appear try {
await page.waitForTimeout(3000); await page.waitForSelector(
'input[name="loginfmt"], input[type="email"], [data-testid="authLoginDialogNextButton"]',
{ timeout: 10000, state: 'visible' },
);
} catch {
_log('warn', 'Auth dialog / email input not found after sign-in click');
}
return true; return true;
} }
} catch { } catch {
@ -1156,7 +1163,14 @@ async function _attemptAuth(page: Page, email: string, password: string): Promis
await el.click(); await el.click();
logger.info(`[AuthTest] Clicked sign-in link: ${selector}`); logger.info(`[AuthTest] Clicked sign-in link: ${selector}`);
clicked = true; clicked = true;
await page.waitForTimeout(3000); try {
await page.waitForSelector(
'input[name="loginfmt"], input[type="email"], [data-testid="authLoginDialogNextButton"]',
{ timeout: 10000, state: 'visible' },
);
} catch {
logger.warn('[AuthTest] Auth dialog not found after sign-in click');
}
break; break;
} }
} catch { } catch {
@ -1173,9 +1187,15 @@ async function _attemptAuth(page: Page, email: string, password: string): Promis
const authProcedure = new AuthProcedure(page, logger); const authProcedure = new AuthProcedure(page, logger);
const authResult = await authProcedure.authenticateWithMicrosoft(email, password, true); const authResult = await authProcedure.authenticateWithMicrosoft(email, password, true);
// Wait for redirect back to Teams after auth
if (authResult) { if (authResult) {
await page.waitForTimeout(5000); try {
await page.waitForSelector(
'#prejoin-join-button, button[data-tid="prejoin-join-button"], button[id="hangup-button"]',
{ timeout: 15000 },
);
} catch {
logger.warn('[AuthTest] Teams page not loaded after auth');
}
} }
return authResult; return authResult;

View file

@ -10,6 +10,11 @@ import * as os from 'os';
* Must be called AFTER the bot is on the pre-join screen but BEFORE clicking "Join now". * Must be called AFTER the bot is on the pre-join screen but BEFORE clicking "Join now".
* Only works for authenticated joins (anonymous guests may not have background options). * Only works for authenticated joins (anonymous guests may not have background options).
*/ */
const _BG_WAIT_MS = 10000;
const _BG_EFFECT_GRID_SEL =
'button[aria-label*="None" i], button[aria-label*="Kein" i], [data-tid="background-item-none"], ' +
'[data-tid="background-image"], [class*="background-item"], li[role="listitem"] button';
export class BackgroundProcedure { export class BackgroundProcedure {
private _page: Page; private _page: Page;
private _logger: Logger; private _logger: Logger;
@ -30,7 +35,6 @@ export class BackgroundProcedure {
if (!opened) { if (!opened) {
return false; return false;
} }
await this._page.waitForTimeout(500);
const noEffectSelectors: string[] = [ const noEffectSelectors: string[] = [
'button[aria-label*="None" i]', 'button[aria-label*="None" i]',
@ -46,7 +50,6 @@ export class BackgroundProcedure {
if (btn) { if (btn) {
await btn.click(); await btn.click();
this._logger.info(`Selected no background effect: ${sel}`); this._logger.info(`Selected no background effect: ${sel}`);
await this._page.waitForTimeout(500);
await this._dismissPanelIfOpen(); await this._dismissPanelIfOpen();
return true; return true;
} }
@ -59,7 +62,6 @@ export class BackgroundProcedure {
if (tile) { if (tile) {
await tile.click(); await tile.click();
this._logger.info('Clicked first background effects tile (often no effect)'); this._logger.info('Clicked first background effects tile (often no effect)');
await this._page.waitForTimeout(400);
await this._dismissPanelIfOpen(); await this._dismissPanelIfOpen();
return true; return true;
} }
@ -76,7 +78,10 @@ export class BackgroundProcedure {
private async _dismissPanelIfOpen(): Promise<void> { private async _dismissPanelIfOpen(): Promise<void> {
try { try {
await this._page.keyboard.press('Escape'); await this._page.keyboard.press('Escape');
await this._page.waitForTimeout(200); await this._page.waitForFunction(
() => !document.querySelector('[data-tid="background-settings-panel"], [class*="background-effects"]'),
{ timeout: 5000 },
).catch(() => {});
} catch { } catch {
// ignore // ignore
} }
@ -151,7 +156,11 @@ export class BackgroundProcedure {
if (button) { if (button) {
await button.click(); await button.click();
this._logger.info(`Clicked background effects button: ${selector}`); this._logger.info(`Clicked background effects button: ${selector}`);
await this._page.waitForTimeout(2000); try {
await this._page.waitForSelector(_BG_EFFECT_GRID_SEL, { state: 'visible', timeout: _BG_WAIT_MS });
} catch {
this._logger.warn('Background effects panel elements not visible after click');
}
return true; return true;
} }
} catch { } catch {
@ -186,7 +195,11 @@ export class BackgroundProcedure {
await button.click(); await button.click();
this._logger.info(`Clicked add image button: ${selector}`); this._logger.info(`Clicked add image button: ${selector}`);
addButtonClicked = true; addButtonClicked = true;
await this._page.waitForTimeout(1000); try {
await this._page.waitForSelector('input[type="file"]', { state: 'attached', timeout: _BG_WAIT_MS });
} catch {
this._logger.warn('File input not found after clicking add-image button');
}
break; break;
} }
} catch { } catch {
@ -199,7 +212,14 @@ export class BackgroundProcedure {
if (fileInput) { if (fileInput) {
await fileInput.setInputFiles(filePath); await fileInput.setInputFiles(filePath);
this._logger.info('Background image uploaded via file input'); this._logger.info('Background image uploaded via file input');
await this._page.waitForTimeout(2000); try {
await this._page.waitForSelector(
'[data-tid="background-image"], .background-image-item',
{ state: 'visible', timeout: _BG_WAIT_MS },
);
} catch {
this._logger.warn('No background thumbnail appeared after upload');
}
// The uploaded image should be auto-selected, but click it to be sure // The uploaded image should be auto-selected, but click it to be sure
// Look for the last image in the background gallery (newly uploaded) // Look for the last image in the background gallery (newly uploaded)
@ -209,7 +229,6 @@ export class BackgroundProcedure {
const lastImage = images[images.length - 1]; const lastImage = images[images.length - 1];
await lastImage.click(); await lastImage.click();
this._logger.info('Selected uploaded background image'); this._logger.info('Selected uploaded background image');
await this._page.waitForTimeout(1000);
} }
} catch { } catch {
this._logger.debug('Could not click uploaded image - may be auto-selected'); this._logger.debug('Could not click uploaded image - may be auto-selected');

View file

@ -64,12 +64,85 @@ export class ChatProcedure {
const isOpen = await this._isChatPanelOpen(); const isOpen = await this._isChatPanelOpen();
if (isOpen) { if (isOpen) {
this._logger.info('Chat panel opened successfully'); this._logger.info('Chat panel opened successfully');
// Light-meetings ships a "simplified compose" with a collapsed
// placeholder + dedicated expand button. The real ckeditor textbox
// is rendered but Playwright considers it invisible until expanded.
// Expand once now so the periodic scan and send path see the
// canonical ckeditor surface.
await this._ensureComposeExpanded();
} else { } else {
this._logger.warn('Chat panel could not be opened - chat send/receive will not work'); this._logger.warn('Chat panel could not be opened - chat send/receive will not work');
} }
return isOpen; return isOpen;
} }
/**
* Detect Teams' "simplified compose" layout (light-meetings / new meeting
* chat UI) and expand it. In that layout the chat side-pane shows a
* compact placeholder toolbar with a `newMessageCommands-expand-compose`
* button; the actual `<div data-tid="ckeditor" role="textbox" contenteditable="true">`
* is mounted but rendered in a state where Playwright's `isVisible()` is
* false (the parent layer is hidden until expand is pressed). Clicking
* the expand button promotes the full ckeditor surface to a visible
* compose region anchored under `chat-pane-compose-message-footer`.
*
* Idempotent: returns true also when no expand button is present (the
* compose is already in its full form).
*/
private async _ensureComposeExpanded(): Promise<boolean> {
const expandSelector = '[data-tid="newMessageCommands-expand-compose"]';
const expandBtn = await this._page.$(expandSelector);
if (!expandBtn) {
return true;
}
const isVisible = await expandBtn.isVisible().catch(() => false);
if (!isVisible) {
await expandBtn.dispose();
return true;
}
try {
await expandBtn.click();
this._logger.info('Compose expanded: clicked newMessageCommands-expand-compose');
} catch (err) {
this._logger.warn(`Compose expand click failed: ${err}`);
await expandBtn.dispose();
return false;
}
await expandBtn.dispose();
// Confirm the compose actually expanded: either the simplified toolbar
// is gone, or a ckeditor textbox is now visible (Playwright sense).
try {
await this._page.waitForFunction(
() => {
const simplified = document.querySelector(
'[data-tid="simplified-compose-bottom-toolbar"]'
) as HTMLElement | null;
const simplifiedGone = !simplified || simplified.offsetHeight === 0;
if (simplifiedGone) return true;
const ck = document.querySelector(
'[data-tid="ckeditor"], div.ck-editor__editable[contenteditable="true"]'
) as HTMLElement | null;
if (!ck) return false;
const rect = ck.getBoundingClientRect();
const style = window.getComputedStyle(ck);
return (
rect.width > 0 && rect.height > 0
&& style.visibility !== 'hidden'
&& style.display !== 'none'
&& style.opacity !== '0'
);
},
{ timeout: 5000 },
);
return true;
} catch {
this._logger.warn('Compose expand: state did not stabilise within 5s');
return false;
}
}
/** /**
* Check if the chat panel is currently visible by probing for known * Check if the chat panel is currently visible by probing for known
* UI elements (chat input, message list, or aria-pressed toggle). * UI elements (chat input, message list, or aria-pressed toggle).
@ -109,12 +182,17 @@ export class ChatProcedure {
// overlays, which is NOT the meeting chat. // overlays, which is NOT the meeting chat.
const inputSelectors = [ const inputSelectors = [
'[data-tid="ckeditor-replyConversation"]', '[data-tid="ckeditor-replyConversation"]',
'[data-tid="ckeditor"]',
'[data-tid="chat-pane-compose-message-footer"] div[contenteditable="true"]', '[data-tid="chat-pane-compose-message-footer"] div[contenteditable="true"]',
'[data-tid="chat-pane-compose-message-footer"] div[role="textbox"]', '[data-tid="chat-pane-compose-message-footer"] div[role="textbox"]',
'[data-tid="message-pane-footer"] div[contenteditable="true"]', '[data-tid="message-pane-footer"] div[contenteditable="true"]',
'[data-tid="message-pane-footer"] div[role="textbox"]', '[data-tid="message-pane-footer"] div[role="textbox"]',
'div[role="textbox"][data-tid*="chat"]', 'div[role="textbox"][data-tid*="chat"]',
'div[role="textbox"][data-tid*="message"]', 'div[role="textbox"][data-tid*="message"]',
// light-meetings: a visible "expand compose" button is itself a
// reliable signal that the meeting chat side-pane is open.
'[data-tid="newMessageCommands-expand-compose"]',
'[data-tid="simplified-compose-bottom-toolbar"]',
]; ];
for (const sel of inputSelectors) { for (const sel of inputSelectors) {
const el = document.querySelector(sel) as HTMLElement | null; const el = document.querySelector(sel) as HTMLElement | null;
@ -406,9 +484,11 @@ export class ChatProcedure {
const author = _findAuthor(messageEl); const author = _findAuthor(messageEl);
let text = _findBody(messageEl); let text = _findBody(messageEl);
// Last resort: take innerText minus the author name & metadata so we // Root-cause guard: only fall back to innerText when an author
// at least surface something when the body wrapper changes again. // was actually identified. Without this, structural fragments
if (!text) { // (bare timestamp separators, lone time elements) leak through
// as "Unknown: 22:04" / "Unknown: Sending..." style entries.
if (!text && author !== 'Unknown') {
const full = (messageEl.innerText || '').trim(); const full = (messageEl.innerText || '').trim();
if (full) { if (full) {
text = full text = full
@ -592,6 +672,7 @@ export class ChatProcedure {
const opened = await this._openChatPanel(); const opened = await this._openChatPanel();
if (opened) { if (opened) {
this._consecutiveOpenFailures = 0; this._consecutiveOpenFailures = 0;
await this._ensureComposeExpanded();
} else { } else {
this._consecutiveOpenFailures++; this._consecutiveOpenFailures++;
this._logger.info( this._logger.info(
@ -652,6 +733,8 @@ export class ChatProcedure {
// Modern Teams chat bubbles have NO data-tid on the wrapper — // Modern Teams chat bubbles have NO data-tid on the wrapper —
// we match on Fluent UI v9 class prefixes and role="listitem". // we match on Fluent UI v9 class prefixes and role="listitem".
// fui-ChatMessageCompact (light-meetings simplified layout) is
// covered by [class*="fui-ChatMessage"].
const messageSelectors = [ const messageSelectors = [
'[data-tid="chat-message"]', '[data-tid="chat-message"]',
'[data-tid="chat-pane-message"]', '[data-tid="chat-pane-message"]',
@ -662,8 +745,21 @@ export class ChatProcedure {
'[class*="fui-ChatMyMessage"]', '[class*="fui-ChatMyMessage"]',
'[role="listitem"]', '[role="listitem"]',
]; ];
const target = container || document.body; // If the resolved container is collapsed (light-meetings:
const candidates = target.querySelectorAll(messageSelectors.join(', ')); // message-pane-layout often has h=0 because the chat bubbles are
// rendered in a sibling overlay branch, not inside the layout),
// a scoped query would return 0 candidates and we'd miss every
// visible chat. Promote target to document.body in that case.
let target: HTMLElement | Document = container || document.body;
let candidates: NodeListOf<Element> = target.querySelectorAll(messageSelectors.join(', '));
if (
(!candidates.length && container && container.offsetHeight === 0)
|| (!candidates.length && document.querySelectorAll('[class*="fui-ChatMessage"]').length > 0)
) {
target = document;
candidates = document.querySelectorAll(messageSelectors.join(', '));
containerSrc = `${containerSrc} -> fallback:document (container collapsed or messages in sibling branch)`;
}
const findAuthor = (root: HTMLElement, fallbackEl: HTMLElement): string => { const findAuthor = (root: HTMLElement, fallbackEl: HTMLElement): string => {
const sels = [ const sels = [
@ -698,7 +794,12 @@ export class ChatProcedure {
const messageEl = (el.closest?.('[data-tid*="chat-message"], [data-tid*="message-list-item"], [class*="fui-ChatMessage"], [class*="fui-ChatMyMessage"]') as HTMLElement | null) || el; const messageEl = (el.closest?.('[data-tid*="chat-message"], [data-tid*="message-list-item"], [class*="fui-ChatMessage"], [class*="fui-ChatMyMessage"]') as HTMLElement | null) || el;
const author = findAuthor(messageEl, el); const author = findAuthor(messageEl, el);
let text = findBody(messageEl, el); let text = findBody(messageEl, el);
if (!text) { // Root-cause guard: if no structured body was found AND no
// author was identified, the element is not a chat message —
// skip it. Without this, the innerText fallback below would
// emit bare timestamp separators / structural fragments as
// "Unknown: 22:04" / "Unknown: Sending..." etc.
if (!text && author !== 'Unknown') {
const full = (messageEl.innerText || '').trim(); const full = (messageEl.innerText || '').trim();
if (full) { if (full) {
text = full text = full
@ -860,6 +961,11 @@ export class ChatProcedure {
return false; return false;
} }
// Light-meetings ships the simplified compose by default and reverts
// to it after every panel re-open. Expand once per send so the
// ckeditor textbox is the active, Playwright-visible surface.
await this._ensureComposeExpanded();
// Note: order matters — most specific selectors first; the `chat-pane-compose-message-footer` // Note: order matters — most specific selectors first; the `chat-pane-compose-message-footer`
// ancestor lookup is needed because Teams Fluent UI v9 scopes the contenteditable inside it. // ancestor lookup is needed because Teams Fluent UI v9 scopes the contenteditable inside it.
// Modern Teams meeting chat uses CKEditor 5 (`.ck-editor__editable`) and its compose root // Modern Teams meeting chat uses CKEditor 5 (`.ck-editor__editable`) and its compose root
@ -867,6 +973,9 @@ export class ChatProcedure {
const inputSelectors = [ const inputSelectors = [
// Classic data-tid selectors (older Teams builds) // Classic data-tid selectors (older Teams builds)
'[data-tid="ckeditor-replyConversation"]', '[data-tid="ckeditor-replyConversation"]',
// Light-meetings new meeting chat composer (post-expand): the
// contenteditable surface has data-tid="ckeditor" directly.
'[data-tid="ckeditor"]',
'[data-tid="chat-pane-compose-message-footer"] div[contenteditable="true"]', '[data-tid="chat-pane-compose-message-footer"] div[contenteditable="true"]',
'[data-tid="chat-pane-compose-message-footer"] div[role="textbox"]', '[data-tid="chat-pane-compose-message-footer"] div[role="textbox"]',
'[data-tid="message-pane-footer"] div[contenteditable="true"]', '[data-tid="message-pane-footer"] div[contenteditable="true"]',

View file

@ -13,6 +13,8 @@ import { resolveLaunchUrl, getMeetingLaunchUrl } from './meetingUrlParser';
* NOTE: The bot always joins as an anonymous guest with the configured bot name. * NOTE: The bot always joins as an anonymous guest with the configured bot name.
* Authentication is disabled. See Teamsbot-Auth-Join-Learnings.md. * Authentication is disabled. See Teamsbot-Auth-Join-Learnings.md.
*/ */
const _CONDITION_WAIT_MS = 10000;
export class JoinProcedure { export class JoinProcedure {
private _page: Page; private _page: Page;
private _logger: Logger; private _logger: Logger;
@ -24,15 +26,80 @@ export class JoinProcedure {
this._botName = botName; this._botName = botName;
} }
private async _waitForPreJoinAfterLauncher(): Promise<void> {
const preJoin =
'input[data-tid="prejoin-display-name-input"], #prejoin-join-button, button[data-tid="prejoin-join-button"]';
try {
await this._page.waitForSelector(preJoin, { state: 'visible', timeout: _CONDITION_WAIT_MS });
} catch {
this._logger.warn('Pre-join UI not detected after launcher (continuing)');
}
}
private async _waitForNoAudioVideoModalGone(): Promise<void> {
try {
await this._page.waitForFunction(
() => {
const modal = document.querySelector('[role="dialog"]');
if (!modal) return true;
const t = modal.textContent || '';
const isNoAv =
t.includes('audio or video') ||
t.includes('Audio oder Video') ||
t.includes('without audio');
return !isNoAv;
},
{ timeout: _CONDITION_WAIT_MS },
);
} catch {
this._logger.warn('No-audio/video modal may still be visible (continuing)');
}
}
private async _waitForPermissionOverlayCleared(): Promise<void> {
try {
await this._page.waitForFunction(
() => {
const dialogs = Array.from(document.querySelectorAll('[role="dialog"]'));
return !dialogs.some(d => {
const t = d.textContent || '';
return (
(t.includes('manage') || t.includes('Manage') || t.includes('display') || t.includes('window')) &&
(t.includes('Allow') || t.includes('Erlauben') || t.includes('Zulassen'))
);
});
},
{ timeout: _CONDITION_WAIT_MS },
);
} catch {
this._logger.warn('Permission overlay may still be visible (continuing)');
}
}
private async _waitForLeaveUiSettled(): Promise<void> {
try {
await this._page.waitForFunction(
() =>
!document.querySelector('button[id="hangup-button"]') &&
!document.querySelector('[data-tid="hangup-button"]'),
{ timeout: _CONDITION_WAIT_MS },
);
} catch {
this._logger.warn('Hangup control still present or leave transition slow (continuing)');
}
}
/** /**
* Navigate to the meeting URL and handle the launcher dialog. * Navigate to the meeting URL and handle the launcher dialog.
* *
* Teams meeting URLs redirect through several hops. We resolve the redirect * Teams meeting URLs redirect through several hops. `resolveLaunchUrl()`
* and add params (suppressPrompt, msLaunch=false, anon=true) to skip the * follows them server-side and adds suppressPrompt params to the
* "Open in Teams app?" native dialog. Then we click "Continue on this browser". * RESOLVED launcher URL so the browser does not show the native
* "Open Microsoft Teams?" protocol-handler modal. It also strips
* `anon=true` from the inner URL (Teams' server bakes this in for
* cookie-less fetch requests; we don't want it).
*/ */
async startMeetingLauncherFlow(meetingUrl: string): Promise<void> { async startMeetingLauncherFlow(meetingUrl: string): Promise<void> {
// Resolve the meeting URL redirect and add suppressPrompt params
let launchUrl: string; let launchUrl: string;
try { try {
launchUrl = await resolveLaunchUrl(meetingUrl); launchUrl = await resolveLaunchUrl(meetingUrl);
@ -49,7 +116,6 @@ export class JoinProcedure {
timeout: config.timeouts.pageLoad, timeout: config.timeouts.pageLoad,
}); });
// Handle "Continue on this browser" button
await this._handleLauncherDialog(); await this._handleLauncherDialog();
} }
@ -63,7 +129,7 @@ export class JoinProcedure {
if (launcherButton) { if (launcherButton) {
this._logger.info('Launcher dialog found, clicking "Continue on this browser"'); this._logger.info('Launcher dialog found, clicking "Continue on this browser"');
await launcherButton.click(); await launcherButton.click();
await this._page.waitForTimeout(2000); await this._waitForPreJoinAfterLauncher();
} }
} catch { } catch {
// No launcher - that's fine // No launcher - that's fine
@ -86,6 +152,7 @@ export class JoinProcedure {
this._logger.info(`Found launcher button: ${primarySelector}`); this._logger.info(`Found launcher button: ${primarySelector}`);
await this._page.click(primarySelector); await this._page.click(primarySelector);
this._logger.info('Clicked "Continue on this browser" button'); this._logger.info('Clicked "Continue on this browser" button');
await this._waitForPreJoinAfterLauncher();
return; return;
} catch { } catch {
this._logger.info('Primary launcher selector not found, trying fallbacks...'); this._logger.info('Primary launcher selector not found, trying fallbacks...');
@ -106,7 +173,7 @@ export class JoinProcedure {
if (element) { if (element) {
this._logger.info(`Found launcher button (fallback): ${selector}`); this._logger.info(`Found launcher button (fallback): ${selector}`);
await element.click(); await element.click();
await this._page.waitForTimeout(2000); await this._waitForPreJoinAfterLauncher();
return; return;
} }
} catch { } catch {
@ -210,7 +277,6 @@ export class JoinProcedure {
if (button) { if (button) {
await button.click(); await button.click();
this._logger.info(`Clicked "Join now" button (attempt ${attempt}/${maxRetries})`); this._logger.info(`Clicked "Join now" button (attempt ${attempt}/${maxRetries})`);
await this._page.waitForTimeout(2000);
await this._dismissNoAudioVideoModal(); await this._dismissNoAudioVideoModal();
return; return;
} }
@ -226,7 +292,6 @@ export class JoinProcedure {
if (button && await button.isVisible()) { if (button && await button.isVisible()) {
await button.click(); await button.click();
this._logger.info(`Clicked join button fallback: ${selector} (attempt ${attempt}/${maxRetries})`); this._logger.info(`Clicked join button fallback: ${selector} (attempt ${attempt}/${maxRetries})`);
await this._page.waitForTimeout(2000);
await this._dismissNoAudioVideoModal(); await this._dismissNoAudioVideoModal();
return; return;
} }
@ -274,7 +339,7 @@ export class JoinProcedure {
if (button) { if (button) {
await button.click(); await button.click();
this._logger.info(`Dismissed no-audio modal: ${selector}`); this._logger.info(`Dismissed no-audio modal: ${selector}`);
await this._page.waitForTimeout(1000); await this._waitForNoAudioVideoModalGone();
return; return;
} }
} catch { } catch {
@ -306,7 +371,7 @@ export class JoinProcedure {
if (text.includes('manage') || text.includes('Manage') || text.includes('display') || text.includes('window')) { if (text.includes('manage') || text.includes('Manage') || text.includes('display') || text.includes('window')) {
await button.click(); await button.click();
this._logger.info(`Dismissed browser permission modal: ${selector}`); this._logger.info(`Dismissed browser permission modal: ${selector}`);
await this._page.waitForTimeout(1000); await this._waitForPermissionOverlayCleared();
return; return;
} }
} }
@ -326,45 +391,42 @@ export class JoinProcedure {
async isInMeetingLobby(options: { waitForSeconds?: number } = {}): Promise<boolean> { async isInMeetingLobby(options: { waitForSeconds?: number } = {}): Promise<boolean> {
const timeout = (options.waitForSeconds || 5) * 1000; const timeout = (options.waitForSeconds || 5) * 1000;
// Check for any lobby text variant using page.evaluate for reliability const lobbySelectors = [
try { '[data-tid="lobby-screen"]',
const inLobby = await this._page.evaluate(() => { '[data-tid="waiting-screen"]',
const bodyText = document.body?.innerText || ''; '[data-tid="lobby-waiting-screen"]',
const lobbyIndicators = [ '[data-tid="lobby-container"]',
'Someone will let you in shortly', '[data-cid="lobby-screen"]',
'Someone will let you in when the meeting starts', '[data-cid="waiting-screen"]',
'will let you in', '#lobby-container',
'waiting for someone to let you in', '[id*="lobby"]',
'Someone in the meeting should let you in',
]; ];
return lobbyIndicators.some(text => bodyText.includes(text));
});
if (inLobby) return true;
} catch {
// Page may not be ready
}
// Primary: text-based check with waitFor (waits up to timeout)
try { try {
await this._page.getByText('will let you in').waitFor({ await this._page.waitForSelector(lobbySelectors.join(', '), {
timeout, timeout,
state: 'visible', state: 'visible',
}); });
return true; return true;
} catch { } catch {
// Not found within timeout // No structural lobby element found
} }
// Fallback: data-tid selectors // Fallback: check for the pre-join/lobby state via page structure —
// the lobby has no call-control bar but does have a waiting spinner or icon
try { try {
await this._page.waitForSelector('[data-tid="lobby-screen"], [data-tid="waiting-screen"]', { const hasLobbyStructure = await this._page.evaluate(() => {
timeout: 1000, const el = document.querySelector(
state: 'visible', '[class*="lobby" i], [class*="waiting-room" i], [class*="waitingScreen" i]'
);
return !!el;
}); });
return true; if (hasLobbyStructure) return true;
} catch { } catch {
return false; // Page may not be ready
} }
return false;
} }
/** /**
@ -378,25 +440,25 @@ export class JoinProcedure {
async isInMeeting(options: { waitForSeconds?: number } = {}): Promise<boolean> { async isInMeeting(options: { waitForSeconds?: number } = {}): Promise<boolean> {
const timeout = (options.waitForSeconds || 5) * 1000; const timeout = (options.waitForSeconds || 5) * 1000;
// Primary selectors - known meeting UI elements
const inMeetingSelectors = [ const inMeetingSelectors = [
// Button IDs (Teams 2025+ redesign)
'button[id="hangup-button"]', 'button[id="hangup-button"]',
'button[id="microphone-button"]',
'button[id="callingButtons-showMoreBtn"]', 'button[id="callingButtons-showMoreBtn"]',
// Fallbacks with data-tid (older Teams versions) 'button[id="video-button"]',
// data-tid attributes
'[data-tid="hangup-button"]', '[data-tid="hangup-button"]',
'[data-tid="call-composite"]', '[data-tid="call-composite"]',
'button[aria-label*="Leave"]',
'[data-tid="callingButtons-showMoreBtn"]', '[data-tid="callingButtons-showMoreBtn"]',
// Teams v2 (2025+) additional selectors
'[data-tid="call-controls"]', '[data-tid="call-controls"]',
'[data-tid="meeting-composite"]', '[data-tid="meeting-composite"]',
'div[data-tid="video-gallery"]', 'div[data-tid="video-gallery"]',
'button[aria-label*="Hang up"]',
'button[aria-label*="leave" i]',
// Mic/Camera toggle buttons are only visible in an active call
'button[id="microphone-button"]',
'button[data-tid="toggle-mute"]',
'[data-tid="microphone-button"]', '[data-tid="microphone-button"]',
'[data-tid="toggle-mute"]',
// data-cid attributes (light-meetings / anonymous join)
'[data-cid="ts-hangup-btn"]',
'[data-cid="calling-hangup-button"]',
'[data-cid="calling-unified-bar"]',
]; ];
try { try {
@ -406,28 +468,31 @@ export class JoinProcedure {
}); });
return true; return true;
} catch { } catch {
// Selector-based detection failed, try DOM evaluation as fallback // Primary selector-based detection failed
} }
// Fallback: evaluate the page for meeting indicators // Fallback: structural DOM check for call control containers
try { try {
const inMeeting = await this._page.evaluate(() => { const inMeeting = await this._page.evaluate(() => {
// Check for call-related aria roles and meeting elements const callBar = document.querySelector(
const bodyText = document.body?.innerText || ''; '[class*="calling-controls" i], [class*="call-controls" i], ' +
const meetingIndicators = [ '[class*="controlBar" i], [class*="unified-bar" i]'
'Leave', // Leave button text );
'Mute', // Mic mute button if (callBar) return true;
'Unmute', // Mic unmute button // Check for hangup/mic buttons by role+structure (language-independent)
'Turn off camera', // Camera control const buttons = Array.from(document.querySelectorAll('button[id]'));
'Turn on camera', let callButtons = 0;
'Share', // Share screen for (let i = 0; i < buttons.length; i++) {
]; const id = buttons[i].id.toLowerCase();
const found = meetingIndicators.filter(ind => bodyText.includes(ind)); if (id.includes('hangup') || id.includes('microphone') ||
// Need at least 2 meeting indicators to confirm we're in a meeting id.includes('video-button') || id.includes('mute')) {
return found.length >= 2; callButtons++;
}
}
return callButtons >= 2;
}); });
if (inMeeting) { if (inMeeting) {
this._logger.info('Detected meeting via DOM text analysis (fallback)'); this._logger.info('Detected meeting via structural DOM analysis (fallback)');
return true; return true;
} }
} catch { } catch {
@ -451,7 +516,7 @@ export class JoinProcedure {
await this._page.waitForSelector(primarySelector, { timeout: 5000 }); await this._page.waitForSelector(primarySelector, { timeout: 5000 });
await this._page.click(primarySelector); await this._page.click(primarySelector);
this._logger.info('Clicked leave button'); this._logger.info('Clicked leave button');
await this._page.waitForTimeout(2000); await this._waitForLeaveUiSettled();
return; return;
} catch { } catch {
this._logger.info('Primary leave selector not found, trying fallbacks...'); this._logger.info('Primary leave selector not found, trying fallbacks...');
@ -471,7 +536,7 @@ export class JoinProcedure {
if (button) { if (button) {
await button.click(); await button.click();
this._logger.info(`Clicked leave button (fallback: ${selector})`); this._logger.info(`Clicked leave button (fallback: ${selector})`);
await this._page.waitForTimeout(2000); await this._waitForLeaveUiSettled();
return; return;
} }
} catch { } catch {

View file

@ -132,7 +132,8 @@ export class BotOrchestrator {
/** /**
* Teams launcher commonly embeds meeting params in hash-routed paths: * Teams launcher commonly embeds meeting params in hash-routed paths:
* /_#/meet/<id>?p=...&anon=true * /_#/meet/<id>?p=...&anon=true
* In this shape, "anon" is in the hash query (not URL.search). * In this shape, "anon" is in the hash query (not URL.search). The auth
* path must strip it so the meeting client uses the signed-in identity.
*/ */
private _stripAnonFromInnerMeetingUrl(innerUrlPath: string): string { private _stripAnonFromInnerMeetingUrl(innerUrlPath: string): string {
try { try {
@ -352,9 +353,10 @@ export class BotOrchestrator {
// CRITICAL: The suppress params (msLaunch, suppressPrompt, directDl) must // CRITICAL: The suppress params (msLaunch, suppressPrompt, directDl) must
// be on the LAUNCHER URL itself, NOT inside the encoded meeting URL parameter. // be on the LAUNCHER URL itself, NOT inside the encoded meeting URL parameter.
// resolveLaunchUrl follows redirects first (meeting URL → launcher URL), // resolveLaunchUrl follows redirects first (meeting URL → launcher URL),
// then adds the params to the RESOLVED launcher URL. getMeetingLaunchUrl // then adds the params to the RESOLVED launcher URL. resolveLaunchUrl
// adds params to the raw meeting URL — they end up encoded inside the // also sets anon=true (correct default for the anon path); for the
// launcher's url= parameter and have no effect on the launcher behavior. // authenticated path we must explicitly strip it so Teams uses the
// signed-in identity instead of routing as a guest.
let launchUrl: string; let launchUrl: string;
try { try {
launchUrl = await resolveLaunchUrl(this._meetingUrl); launchUrl = await resolveLaunchUrl(this._meetingUrl);
@ -362,22 +364,16 @@ export class BotOrchestrator {
this._logger.warn(`Could not resolve launch URL, using fallback: ${error}`); this._logger.warn(`Could not resolve launch URL, using fallback: ${error}`);
launchUrl = getMeetingLaunchUrl(this._meetingUrl); launchUrl = getMeetingLaunchUrl(this._meetingUrl);
} }
// Remove anon=true since the user is authenticated
try { try {
const urlObj = new URL(launchUrl); const urlObj = new URL(launchUrl);
urlObj.searchParams.delete('anon'); urlObj.searchParams.delete('anon');
// Some Teams launcher URLs carry the real meeting path in an encoded "url" param.
// In auth mode that inner URL can still contain anon=true, which forces guest-like behavior.
const encodedInnerUrl = urlObj.searchParams.get('url'); const encodedInnerUrl = urlObj.searchParams.get('url');
if (encodedInnerUrl) { if (encodedInnerUrl) {
const innerPath = this._stripAnonFromInnerMeetingUrl(encodedInnerUrl); urlObj.searchParams.set('url', this._stripAnonFromInnerMeetingUrl(encodedInnerUrl));
urlObj.searchParams.set('url', innerPath);
} }
launchUrl = urlObj.toString(); launchUrl = urlObj.toString();
} catch { /* keep as-is */ } } catch { /* keep as-is */ }
this._logger.info(`STEP 4: navigating to launch URL: ${launchUrl.substring(0, 120)}...`); this._logger.info(`STEP 4: navigating to launch URL: ${launchUrl.substring(0, 120)}...`);
this._logger.info(`STEP 4: launch URL contains anon=true? ${launchUrl.includes('anon=true')}`);
await this._page!.goto(launchUrl, { await this._page!.goto(launchUrl, {
waitUntil: 'domcontentloaded', waitUntil: 'domcontentloaded',
timeout: 30000, timeout: 30000,
@ -1031,6 +1027,13 @@ export class BotOrchestrator {
/** /**
* Launch the browser and create a new page. * Launch the browser and create a new page.
* @param authMode - If true, use headful + minimal args (Chromium Minimal, proven to work for auth) * @param authMode - If true, use headful + minimal args (Chromium Minimal, proven to work for auth)
*
* NOTE: anon and auth use DIFFERENT chromium args on purpose. The anon path
* relies on `--disable-web-security` + `--disable-features=IsolateOrigins,site-per-process`
* to make Teams' "light-meetings" guest bundle behave correctly across
* cross-origin iframes. Unifying the args breaks anon: Teams sends the
* bot to the lobby and the lobbymeeting WebRTC renegotiation crashes
* (`rejectMediaDescriptionsUpdateAsync`). Keep these flag sets separate.
*/ */
private async _launchBrowser(authMode: boolean = false): Promise<void> { private async _launchBrowser(authMode: boolean = false): Promise<void> {
this._logger.info(`Launching browser (authMode=${authMode})...`); this._logger.info(`Launching browser (authMode=${authMode})...`);
@ -1145,7 +1148,7 @@ export class BotOrchestrator {
timestamp: entry.timestamp, timestamp: entry.timestamp,
isFinal: true, isFinal: true,
}); });
} },
); );
// Inject audio getUserMedia override BEFORE any navigation // Inject audio getUserMedia override BEFORE any navigation
@ -1225,49 +1228,51 @@ export class BotOrchestrator {
/** /**
* Wait for the bot to be admitted from the lobby. * Wait for the bot to be admitted from the lobby.
* Bails out immediately if the page is closed (crash/disconnect) so we
* don't report a misleading "in_lobby" state for the next 2 minutes.
*/ */
private async _waitForMeetingAdmission(): Promise<void> { private async _waitForMeetingAdmission(): Promise<void> {
const startTime = Date.now(); const startTime = Date.now();
const timeout = config.timeouts.lobbyWait; const timeout = config.timeouts.lobbyWait;
let consecutiveNoSignal = 0; let loggedLobby = false;
const maxNoSignal = 5; // Allow several cycles with no lobby/meeting signal before giving up let wasInLobby = false;
while (Date.now() - startTime < timeout) { while (Date.now() - startTime < timeout) {
// Check if we're in the meeting if (!this._page || this._page.isClosed()) {
throw new Error('Page closed while waiting for meeting admission');
}
const inMeeting = await this._joinProcedure!.isInMeeting({ waitForSeconds: 5 }); const inMeeting = await this._joinProcedure!.isInMeeting({ waitForSeconds: 5 });
if (inMeeting) { if (inMeeting) {
if (wasInLobby) {
this._logger.info('Admitted from lobby into meeting');
}
return; return;
} }
// Check if still in lobby
const inLobby = await this._joinProcedure!.isInMeetingLobby({ waitForSeconds: 2 }); const inLobby = await this._joinProcedure!.isInMeetingLobby({ waitForSeconds: 2 });
if (inLobby) { if (inLobby) {
consecutiveNoSignal = 0; wasInLobby = true;
this._logger.info('Still waiting in lobby...'); if (!loggedLobby) {
loggedLobby = true;
this._setState('in_lobby');
this._logger.info('Bot is in lobby, waiting for admission...');
await this._takeScreenshot('in-lobby');
}
continue; continue;
} }
// Neither in meeting nor in lobby — this can happen legitimately: if (wasInLobby) {
// - Authenticated users skip lobby, but meeting UI takes seconds to load // Lobby disappeared but isInMeeting not yet true — Teams is
// - Page is transitioning between states // transitioning (WebRTC renegotiation, UI rendering). Keep
// Only give up after several consecutive cycles with no signal // polling; the meeting controls will appear shortly.
consecutiveNoSignal++; this._logger.info('Lobby gone, waiting for meeting UI to render...');
const currentUrl = this._page?.url() || 'unknown'; await this._takeScreenshot('lobby-transition');
this._logger.info(`No lobby/meeting signal detected (attempt ${consecutiveNoSignal}/${maxNoSignal}), URL: ${currentUrl}`);
if (consecutiveNoSignal >= maxNoSignal) {
// Take a screenshot and log page content for debugging before giving up
await this._takeScreenshot('no-meeting-signal');
try {
const bodySnippet = await this._page?.evaluate(() =>
document.body?.innerText?.substring(0, 500) || '(empty)'
);
this._logger.warn(`Page content before giving up: ${bodySnippet}`);
} catch { /* ignore */ }
throw new Error('Bot was removed from lobby or meeting ended');
} }
} }
await this._takeScreenshot('lobby-timeout');
throw new Error('Timeout waiting to be admitted from lobby'); throw new Error('Timeout waiting to be admitted from lobby');
} }

View file

@ -1,6 +1,8 @@
import { Page } from 'playwright'; import { Page } from 'playwright';
import { Logger } from 'winston'; import { Logger } from 'winston';
const _ACTION_WAIT_MS = 10000;
/** /**
* Service center for all Teams meeting UI actions. * Service center for all Teams meeting UI actions.
* *
@ -43,7 +45,12 @@ export class TeamsActionsService {
await captionsBtn.click(); await captionsBtn.click();
this._logger.info(`TeamsActions: Transcript toggled ${enable ? 'ON' : 'OFF'}`); this._logger.info(`TeamsActions: Transcript toggled ${enable ? 'ON' : 'OFF'}`);
await this._page.waitForTimeout(1000); try {
await this._page.waitForFunction(
() => !document.querySelector('[role="menu"]'),
{ timeout: 5000 },
);
} catch { /* menu may already be gone */ }
return true; return true;
} }
@ -88,9 +95,13 @@ export class TeamsActionsService {
} }
await input.click(); await input.click();
await this._page.waitForTimeout(200); try {
await this._page.waitForFunction(
() => document.activeElement?.matches('[contenteditable], [role="textbox"], input, textarea'),
{ timeout: 3000 },
);
} catch { /* proceed even if focus check times out */ }
await this._page.keyboard.type(text, { delay: 10 }); await this._page.keyboard.type(text, { delay: 10 });
await this._page.waitForTimeout(200);
await this._page.keyboard.press('Enter'); await this._page.keyboard.press('Enter');
this._logger.info('TeamsActions: Chat message sent'); this._logger.info('TeamsActions: Chat message sent');
@ -190,7 +201,6 @@ export class TeamsActionsService {
await btn.click(); await btn.click();
this._logger.info(`TeamsActions: ${name} toggled ${enable ? 'ON' : 'OFF'}`); this._logger.info(`TeamsActions: ${name} toggled ${enable ? 'ON' : 'OFF'}`);
await this._page.waitForTimeout(500);
return true; return true;
} }
@ -208,7 +218,11 @@ export class TeamsActionsService {
const button = await this._page.$(selector); const button = await this._page.$(selector);
if (button) { if (button) {
await button.click(); await button.click();
await this._page.waitForTimeout(1000); try {
await this._page.waitForSelector('[role="menu"], [role="menubar"]', { state: 'visible', timeout: _ACTION_WAIT_MS });
} catch {
this._logger.warn('TeamsActions: More menu did not appear');
}
return true; return true;
} }
} catch { } catch {
@ -237,7 +251,11 @@ export class TeamsActionsService {
const item = await this._page.$(sel); const item = await this._page.$(sel);
if (item) { if (item) {
await item.click(); await item.click();
await this._page.waitForTimeout(1500); try {
await this._page.waitForSelector('#closed-captions-button', { state: 'visible', timeout: _ACTION_WAIT_MS });
} catch {
this._logger.warn('TeamsActions: Captions button not visible in submenu');
}
btn = await this._page.$('#closed-captions-button'); btn = await this._page.$('#closed-captions-button');
if (btn) return btn; if (btn) return btn;
break; break;
@ -276,7 +294,14 @@ export class TeamsActionsService {
if (button) { if (button) {
await button.click(); await button.click();
this._logger.info(`TeamsActions: Opened chat panel: ${selector}`); this._logger.info(`TeamsActions: Opened chat panel: ${selector}`);
await this._page.waitForTimeout(1000); try {
await this._page.waitForSelector(
'[data-tid="ckeditor-replyConversation"], div[role="textbox"][data-tid*="chat"]',
{ state: 'visible', timeout: _ACTION_WAIT_MS },
);
} catch {
this._logger.warn('TeamsActions: Chat panel input not visible after opening');
}
return; return;
} }
} catch { } catch {