Issue with Chrome Tab Audio Capture : can't maintain both playback and transcription

[guillaume olivieri]

Hi all !

I'm developing a Chrome extension that captures tab audio for real-time transcription using Deepgram API.

My issue is that I can either:
- Get the transcription working but lose the tab audio playback
- Or maintain audio playback but lose the transcription

Here's a minimal reproducible example: 

manifest.json
{
  "name": "Audio Capture Test",
  "description": "Records tab audio with transcription.",
  "version": "1",
  "manifest_version": 3,
  "minimum_chrome_version": "116",
 
  "action": {
    "default_icon": "icon.png"
  },

  "background": {
    "service_worker": "service-worker.js",
    "type": "module"
  },

  "permissions": [
    "activeTab",
    "scripting",
    "tabs",
    "tabCapture"
  ]
}

service-worker.js

chrome.action.onClicked.addListener(async (tab) => { try { const streamId = await chrome.tabCapture.getMediaStreamId({ targetTabId: tab.id, consumerTabId: tab.id }); await chrome.scripting.executeScript({ target: { tabId: tab.id }, files: ['content.js'] }); await chrome.tabs.sendMessage(tab.id, { type: 'start-recording', streamId: streamId }); } catch (error) { console.error('Error:', error); } });

content.js

let audioContext = null; let tabStream = null; chrome.runtime.onMessage.addListener(async (message, sender, sendResponse) => { if (message.type === 'start-recording') { try { audioContext = new AudioContext({ sampleRate: 48000, latencyHint: 'interactive' }); // Get tab audio stream tabStream = await navigator.mediaDevices.getUserMedia({ audio: { mandatory: { chromeMediaSource: 'tab', chromeMediaSourceId: message.streamId } }, video: false }); const tabSource = audioContext.createMediaStreamSource(tabStream); const tabProcessor = audioContext.createScriptProcessor(4096, 1, 1); tabSource.connect(tabProcessor); tabProcessor.connect(audioContext.destination); // This setup works for transcription but mutes the tab audio // How can we maintain the original audio playback? } catch (error) { console.error('Error:', error); } } return true; });

I've tried several approaches:

1. Using Audio() element with the stream:

const audioElement = new Audio(); audioElement.srcObject = tabStream; audioElement.play();

2. Creating dual audio paths with GainNode:

const gainNode = audioContext.createGain(); gainNode.gain.value = 1.0; tabSource.connect(gainNode); gainNode.connect(audioContext.destination);

3. Cloning the stream:

const playbackStream = new MediaStream([tabStream.getAudioTracks()[0].clone()]); const audioElement = new Audio(); audioElement.srcObject = playbackStream; audioElement.play();

but each attempt either breaks the transcription or doesn't restore the audio.

How can I maintain both the original tab audio playback AND capture the audio for transcription?

Technical details:

• Chrome version: Latest
• Manifest V3
• Using chrome.tabCapture.getMediaStreamId() to get the stream

Any help would be greatly appreciated!

[Oliver Dunk]

Hi Guillaume,

It sounds like you've been on quite the adventure, so great job sticking at this!

My understanding of the audio APIs is a little limited, however it sounds like you might be running into this: https://issues.chromium.org/issues/40885587

My experience is that the bug does not occur if you record from outside of a content script, for example within an offscreen document or popup. Could you try one of those two approaches to see if it helps?

When I was last looking into this I wrote up some notes here: https://developer.chrome.com/docs/extensions/how-to/web-platform/screen-capture

Thanks,

Oliver Dunk | DevRel, Chrome Extensions | https://developer.chrome.com/ | London, GB

--
You received this message because you are subscribed to the Google Groups "Chromium Extensions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-extens...@chromium.org.
To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/b3205368-5cde-43c6-a4e1-47c64b9e8e0an%40chromium.org.

[guillaume olivieri]

Hi Oliver,

Thank you for your response and suggestion !

very happy to say that I've successfully solved the issue with a working implementation!

You were right - this is indeed related to the Chromium bug you linked. After experimenting with different approaches, I found a reliable solution that keeps the tab audio playing AND allows transcription via Deepgram.

Here's how I solved it:

1. I used the recommended architecture:
   • Service worker to coordinate everything
   • Offscreen document to handle tab audio capture
   • Content script for UI and microphone recording
2. The critical part that fixed the audio muting issue:
   
   // In the offscreen document
   const tabSource = audioContext.createMediaStreamSource(tabStream);
   
   // This is the key line that prevents Chrome from muting the audio
   tabSource.connect(audioContext.destination);
   
   // For transcription

1. 
   tabProcessor = audioContext.createScriptProcessor(4096, 1, 1);
   tabSource.connect(tabProcessor);
   tabProcessor.connect(audioContext.destination);
   

1. By connecting the audio source directly to the destination in the offscreen document, the tab audio continues to play normally while still allowing the processor to access the audio data for transcription.
   

   The most important part was indeed connecting the audio source directly to the destination, which kept the audio playing, while also sending it through the processor for transcription.

   I've now implemented this solution in my extension, and it's working perfectly - users can hear the tab audio while the real-time transcription happens simultaneously. I've also added multi-language support and microphone recording/transcription alongside the tab audio.

   Thank you again !
   
    Guillaume

[Oliver Dunk]

Hi Guillaume,

That's great to hear! Thank you so much for taking the time to write this up - I'm sure someone will find it in the future and it will be helpful.

Best of luck with the rest of the extension :)

Oliver Dunk | DevRel, Chrome Extensions | https://developer.chrome.com/ | London, GB