Skip to content

Add Realtime Voice Support like OpenAI/Grok/Anthropic/Google Realtime support #189

@JamesFieldist

Description

@JamesFieldist

Please expand this to do full voice to voice.

And the UI should support, just like OpenAI and Grok, the ability to switch the chat from voice to text and text to voice right on the spot.

So what this really needs to do is:

  1. Have full realtime audio support with noise cancelling on each platform. (i.e. sort of like the realtime_audio library but expaned to all platforms and fixed)
  2. Have visualization while you're talking.
  3. Have playback audio support with visualization. (using flutter_sound or whatever, or your own)
  4. Have transcription from server of STT. (this is just adding a message to the screen)
  5. Have the transcript of what the AI is saying be injectable from your provider. (this is just adding the message to the screen)
  6. Make sure that you can interrupt properly. (i.e. I should be able to talk over the AI and you should notice that I did, and kill the AI and start streaming me nicely back to the server)

You could go further and:

  1. Add the ability to provide the SST in this mode with voice so that you can just send the text to the backend (See [feat] Allow replacing the entire speech-to-text recording and transcription process #141 ). But this could be hacked in with your own provider that takes the streaming audio from the input and does this with whatever package you wanted.)
  2. Add TTS streaming text from the backend to the front end that then TTSed that into voice.

By doing this you could use any text chat AI, and you'd get realtime, with most of the realtime work done on the front end.

But these "further" items are optional. We really need support for realtime voice to voice.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions