​

Establishing a Connection

To initiate a conversation, establish a websocket connection to our talk URL, including the agentId as a path parameter:

wss://api.play.ai/v1/talk/<your_agent_id>

For example, assuming Agent-XP5tVPa8GDWym6j is the ID of an agent you have created via our Web UI or through our Create Agent endpoint, the WebSocket URL should look like:

const myWs = new WebSocket('wss://api.play.ai/v1/talk/Agent-XP5tVPa8GDWym6j');

​

Initial Setup Message

Before you can start sending and receiving audio data, you must first send a setup message to authenticate and configure your session.

The only required field in the setup message is the apiKey. This assumes you are comfortable with the default values for audio input and audio output formats. In this scenario, your first setup message could be as simple as:

{ "type": "setup", "apiKey": "yourKey" }

Code example:

const myWs = new WebSocket('wss://api.play.ai/v1/talk/Agent-XP5tVPa8GDWym6j');
myWs.onopen = () => {
  console.log('connected!');
  myWs.send(JSON.stringify({ type: 'setup', apiKey: 'yourApiKey' }));
};

​

Setup Options

The setup message configures important details of the session, including the format/encoding of the audio that you intend to send us and the format that you expect to receive.

// mulaw 16KHz as input
{ "type": "setup", "apiKey": "...", "inputEncoding": "mulaw", "inputSampleRate": 16000 }
// 24Khz mp3 output
{ "type": "setup", "apiKey": "...", "outputFormat": "mp3", "outputSampleRate": 24000 }
// mulaw 8KHz in and out
{ "type": "setup", "apiKey": "...", "inputEncoding": "mulaw", "inputSampleRate": 8000, "outputFormat": "mulaw", "outputSampleRate": 8000 }

The following fields are available for configuration:

​

audioIn: Sending Audio Input

After the setup, you can send audio input in the form of an audioIn message.

The audio must be sent as a base64 encoded string in the data field. The message format is:

{ "type": "audioIn", "data": "<base64Data>" }

​

Sample Code for Sending Audio

Assuming myWs is a WebSocket connected to our /v1/talk endpoint, the sample code below would send audio directly from the browser:

const stream = await navigator.mediaDevices.getUserMedia({
  audio: {
    channelCount: 1,
    echoCancellation: true,
    autoGainControl: true,
    noiseSuppression: true,
  },
});
const mediaRecorder = new MediaRecorder(stream);
mediaRecorder.ondataavailable = async (event) => {
  const base64Data = await blobToBase64(event.data);

  // Relevant:
  myWs.send(JSON.stringify({ type: 'audioIn', data: base64Data }));
};

async function blobToBase64(blob) {
  const reader = new FileReader();
  reader.readAsDataURL(blob);
  return new Promise((resolve) => {
    reader.onloadend = () => resolve(reader.result.split(',')[1]);
  });
}

​

audioStream: Receiving Audio Output

Audio output from the server will be received in an audioStream message. The message format is:

{ "type": "audioStream", "data": "<base64Data>" }

​

Sample Code for Receiving Audio

myWs.on('message', (message) => {
  const event = JSON.parse(message);
  if (event.type === 'audioStream') {
    // deserialize event.data from a base64 string to binary
    // enqueue/play the binary data at your player
    return;
  }
});

​

Voice Activity Detection: voiceActivityStart and voiceActivityEnd

During the conversation, you will receive voiceActivityStart and voiceActivityEnd messages indicating the detection of speech activity in the audio input. These messages help in understanding when the user starts and stops speaking.

When our service detects that the user started to speak, it will emit a voiceActivityStart event. Such a message will have the format:

{ "type": "voiceActivityStart" }

It is up to you to decide how to react to this event. We highly recommend you stop playing whatever audio is being played, since the voiceActivityStart generally indicates the user wanted to interrupt the agent.

Similarly, when our service detects that the user stopped speaking, it emits a voiceActivityEnd event:

{ "type": "voiceActivityEnd" }

​

newAudioStream: Handling New Audio Streams

A newAudioStream message indicates the start the audio of a new response. It is recommended to clear your player buffer and start playing the new stream content upon receiving this message. This message contains no additional fields.

​

Error Handling

Errors from the server are sent as error message type, a numeric code and a message in the following format:

{ "type": "error", "code": <errorCode>, "message": "<errorMessage>" }

The table below provides a quick reference to the various error codes and their corresponding messages for the Agent Websocket API.

This documentation covers the essential aspects of interacting with the PlayAI Websocket API for agent conversations. Ensure that your implementation handles the specified message types and follows the outlined protocols for a seamless integration.