π Features
- ποΈ Two-way voice conversations with AI agents
- π Voice Activity Detection (VAD) for natural conversations
- π§ Custom actions that allow agents to trigger code in your app
- π± Cross-platform - works on iOS, Android, and Web
- π Audio session management for handling interruptions and device changes
- π Real-time transcripts of both user and agent speech
- π¦ Rich state management with ValueNotifiers for UI integration
Installation
Add the package to your pubspec.yaml
:
dependencies:
agents:
git:
url: https://github.com/playht/agents-client-sdk-flutter.git
ref: main
Then save, or run:
- Add the following to your
Info.plist
:
<key>NSMicrophoneUsageDescription</key>
<string>We need access to your microphone to enable voice conversations with the AI agent.</string>
- Add the following to your
Podfile
, since we depend on permission_handler
to manage permissions and audio_session
to manage audio sessions.
post_install do |installer|
installer.pods_project.targets.each do |target|
target.build_configurations.each do |config|
config.build_settings['GCC_PREPROCESSOR_DEFINITIONS'] ||= [
'$(inherited)',
# audio_session settings
'AUDIO_SESSION_MICROPHONE=0',
# For microphone access
'PERMISSION_MICROPHONE=1'
end
end
end
- Due to an issue of the Onnx Runtime getting stripped by XCode when archived, you need to follow these steps in XCode for the voice activity detector (VAD) to work on iOS builds:
- Under βTargetsβ, choose βRunnerβ (or your projectβs name)
- Go to βBuild Settingsβ tab
- Filter for βDeploymentβ
- Set βStripped Linked Productβ to βNoβ
- Set βStrip Styleβ to βNon-Global-Symbolsβ
Android
- Add the following permissions to your
AndroidManifest.xml
:
<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-permission android:name="android.permission.INTERNET" />
<uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS" />
<uses-permission android:name="android.permission.BLUETOOTH" />
<uses-permission android:name="android.permission.BLUETOOTH_CONNECT" />
- Add the following to
android/gradle.properties
(unless theyβre already there):
android.useAndroidX=true
android.enableJetifier=true
- Add the following settings to
android/app/build.gradle
:
android {
compileSdkVersion 34
...
}
For VAD to work on web platforms, please following the instructions here.
Getting Started
1. Create an Agent on PlayAI
Follow the instructions here to create an agent on PlayAI.
2. Implement the Agent in Your Flutter App
final agent = Agent(
// Replace with your agent ID from PlayAI
agentId: 'your-agent-id-here',
// Customize your agent's behavior
prompt: 'You are a helpful assistant who speaks in a friendly, casual tone.',
// Define actions the agent can take in your app
actions: [
AgentAction(
name: 'show_weather',
triggerInstructions: 'Trigger this when the user asks about weather.',
argumentSchema: {
'city': AgentActionParameter(
type: 'string',
description: 'The city to show weather for',
),
},
callback: (data) async {
final city = data['city'] as String;
// In a real app, you would fetch weather data here
return 'Weather data fetched for $city!';
},
),
],
// Configure callbacks to respond to agent events
callbackConfig: AgentCallbackConfig(
// Get user speech transcript
onUserTranscript: (text) {
setState(() => _messages.add(ChatMessage(text, isUser: true)));
},
// Get agent speech transcript
onAgentTranscript: (text) {
setState(() => _messages.add(ChatMessage(text, isUser: false)));
},
// Handle any errors
onError: (error, isFatal) {
ScaffoldMessenger.of(context).showSnackBar(
SnackBar(content: Text('Error: $error')),
);
},
),
);
3. Connect the Agent to Start a Conversation
4. Mute and Unmute the User during a Conversation
await agent.muteUser();
await agent.unmuteUser();
5. Disconnect the Agent
await agent.disconnect();
Key Features
Monitor the Agentβs State
AgentState
: The agent can be in one of four states:
idle
: Not connected to a conversation
connecting
: In the process of establishing a connection
connected
: Connected and ready to converse
disconnecting
: In the process of ending a conversation
Agent
also exposes ValueListenable
s which you can listen to for changes in the agentβs state.
ValueListenableBuilder<AgentState>(
valueListenable: agent.isUserSpeakingNotifier,
builder: (context, isUserSpeaking, _) => Text('User is speaking: $isUserSpeaking'),
)
- Pass callbacks as
AgentCallbackConfig
to the Agent
constructor to handle events from the agent.
final config = AgentCallbackConfig(
onUserTranscript: (text) => print('User just said: $text'),
onAgentTranscript: (text) => print('Agent just said: $text'),
)
final agent = Agent(
// ...
callbackConfig: config,
);
Agent Actions
One of the most exciting features of the PlayAI Agents SDK is the ability to define custom actions that allow the agent to interact with your app.
AgentAction(
name: 'open_settings',
triggerInstructions: 'Trigger this when the user asks to open settings',
argumentSchema: {
'section': AgentActionParameter(
type: 'string',
description: 'The settings section to open',
),
},
callback: (data) async {
final section = data['section'] as String;
// Navigate to settings section in your app
return 'Opened $section settings';
},
)
Developer Messages
Send contextual information to the agent during a conversation to inform it of changes in your app.
// When user navigates to a new screen
void _onNavigate(String routeName) {
agent.sendDeveloperMessage(
'User navigated to $routeName screen. You can now discuss the content on this page.',
);
}
// When relevant data changes
void _onCartUpdated(List<Product> products) {
agent.sendDeveloperMessage(
'User\'s cart has been updated, now containing: ${products.map((p) => p.name).join(", ")}.',
);
}
Error Handling
The package uses a robust error handling system with specific exception types:
try {
await agent.connect();
} on MicrophonePermissionDenied {
// Handle microphone permission issues
} on WebSocketConnectionError catch (e) {
// Handle connection issues
} on ServerError catch (e) {
// Handle server-side errors
if (e.isFatal) {
// Handle fatal errors
}
} on AgentException catch (e) {
// Handle all other agent exceptions
print('Error code: ${e.code}, Message: ${e.readableMessage}');
}
Lifecycle Management
Donβt forget to dispose of the agent when itβs no longer needed to free up resources.
@override
void dispose() {
// Clean up resources
agent.dispose();
super.dispose();
}
UI Integration Examples
ValueListenableBuilder<bool>(
valueListenable: agent.isMutedNotifier,
builder: (context, isMuted, _) => IconButton(
icon: Icon(isMuted ? Icons.mic_off : Icons.mic),
onPressed: () => isMuted ? agent.unmuteUser() : agent.muteUser(),
tooltip: isMuted ? 'Unmute' : 'Mute',
),
)
Speaking Indicator
ValueListenableBuilder<bool>(
valueListenable: agent.isAgentSpeakingNotifier,
builder: (context, isSpeaking, _) => AnimatedContainer(
duration: const Duration(milliseconds: 300),
width: 40,
height: 40,
decoration: BoxDecoration(
shape: BoxShape.circle,
color: isSpeaking ? Colors.blue : Colors.grey.shade300,
),
child: Center(
child: Icon(
Icons.record_voice_over,
size: 24,
color: Colors.white,
),
),
),
)
Tips for Effective Usage
- Prompt Engineering: Craft clear, specific prompts to guide agent behavior
- Action Design: Design actions with clear trigger instructions and parameter descriptions
- Context Management: Use
sendDeveloperMessage
to keep the agent updated on app state
- Error Handling: Implement comprehensive error handling for a smooth user experience
- UI Feedback: Use the provided
ValueListenable
s to give clear feedback on conversation state
Acknowledgments
- Voice Activity Detection powered by vad
- Audio session management by audio_session