Execute invokeModelWithBidirectionalStream returned timed out

[zhu-xiaowei]

Describe the bug

When Execute the api invokeModelWithBidirectionalStream :

let input = InvokeModelWithBidirectionalStreamInput(
    body: inputStream,
    modelId: modelId
)

let output = try await bedrockClient.invokeModelWithBidirectionalStream(input: input)

Xcode returned the timed out error:

Task <7F051A7E-617F-47C0-8E79-B5DFA63FC6CB>.<1> finished with error [-1001] Error Domain=NSURLErrorDomain Code=-1001 "The request timed out." UserInfo={_kCFStreamErrorCodeKey=-2102, NSUnderlyingError=0x30340de60 {Error Domain=kCFErrorDomainCFNetwork Code=-1001 "(null)" UserInfo={_kCFStreamErrorCodeKey=-2102, _kCFStreamErrorDomainKey=4}}, _NSURLErrorFailingURLSessionTaskErrorKey=LocalDataTask <7F051A7E-617F-47C0-8E79-B5DFA63FC6CB>.<1>, _NSURLErrorRelatedURLSessionTaskErrorKey=(
    "LocalDataTask <7F051A7E-617F-47C0-8E79-B5DFA63FC6CB>.<1>"
), NSLocalizedDescription=The request timed out., NSErrorFailingURLStringKey=https://bedrock-runtime.us-east-1.amazonaws.com/model/amazon.nova-sonic-v1%3A0/invoke-with-bidirectional-stream, NSErrorFailingURLKey=https://bedrock-runtime.us-east-1.amazonaws.com/model/amazon.nova-sonic-v1%3A0/invoke-with-bidirectional-stream, _kCFStreamErrorDomainKey=4}


2025-04-14T11:06:19+0800 error URLSessionHTTPClient : [ClientRuntime] URLRequest(POST https://bedrock-runtime.us-east-1.amazonaws.com/model/amazon.nova-sonic-v1%3A0/invoke-with-bidirectional-stream) failed with error: Error Domain=NSURLErrorDomain Code=-1001 "The request timed out." UserInfo={_kCFStreamErrorCodeKey=-2102, NSUnderlyingError=0x30340de60 {Error Domain=kCFErrorDomainCFNetwork Code=-1001 "(null)" UserInfo={_kCFStreamErrorCodeKey=-2102, _kCFStreamErrorDomainKey=4}}, _NSURLErrorFailingURLSessionTaskErrorKey=LocalDataTask <7F051A7E-617F-47C0-8E79-B5DFA63FC6CB>.<1>, _NSURLErrorRelatedURLSessionTaskErrorKey=(
    "LocalDataTask <7F051A7E-617F-47C0-8E79-B5DFA63FC6CB>.<1>"
), NSLocalizedDescription=The request timed out., NSErrorFailingURLStringKey=https://bedrock-runtime.us-east-1.amazonaws.com/model/amazon.nova-sonic-v1%3A0/invoke-with-bidirectional-stream, NSErrorFailingURLKey=https://bedrock-runtime.us-east-1.amazonaws.com/model/amazon.nova-sonic-v1%3A0/invoke-with-bidirectional-stream, _kCFStreamErrorDomainKey=4}

The Python demo for invokeModelWithBidirectionalStream runs normally on my local machine, so there is no issue with the computer's network connection.

Expected Behavior

After calling invokeModelWithBidirectionalStream, output data is successfully returned.

Current Behavior

Returned the timed out error above.

Reproduction Steps

See describe the bug

Possible Solution

No response

Additional Information/Context

No response

AWS SWIFT SDK version used

1.2.58

Compiler and Version used

Xcode 16.2, Swift version 6.0.3, swift-tools-version: 6.0

Operating System and version

macOS 15.4

[sichanyoo]

Hi @zhu-xiaowei, could I get the full code snippet relevant to your use of the API to better determine what the cause of the issue may be?

[zhu-xiaowei]

Hi @sichanyoo following is the full code snippet:

//
//  NovaSonicService.swift
//
//  Created on 2025/4/10.
//

import Foundation
import AWSBedrockRuntime
import AWSSDKIdentity

enum NovaSonicError: Error {
    case invalidCredentials
    case streamInitializationFailed(String)
    case streamError(String)
}

class NovaSonicService {
    // AWS Configuration
    private var client: BedrockRuntimeClient?
    private var region: String
    private var accessKey: String
    private var secretKey: String
    private var modelId: String = "amazon.nova-sonic-v1:0"
    
    // Stream state
    private var isActive: Bool = false
    private var promptName: String
    private var contentName: String
    private var audioContentName: String
    
    // Stream handling
    private var inputContinuation: AsyncThrowingStream<BedrockRuntimeClientTypes.InvokeModelWithBidirectionalStreamInput, Error>.Continuation?
    
    // Callbacks
    var onTranscriptReceived: ((String, String) -> Void)? // (role, text)
    var onStateChanged: ((String, String?) -> Void)? // (state, error?)
    var onAudioReceived: ((Data) -> Void)?
    var onError: ((Error) -> Void)?
    
    init(region: String, accessKey: String, secretKey: String) {
        self.region = region
        self.accessKey = accessKey
        self.secretKey = secretKey
        self.promptName = UUID().uuidString
        self.contentName = UUID().uuidString
        self.audioContentName = UUID().uuidString
    }
    
    // MARK: - Client Initialization
    
    func initializeClient() throws {
        guard !accessKey.isEmpty && !secretKey.isEmpty else {
            throw NovaSonicError.invalidCredentials
        }
        
        // Create AWS credentials
        let credentials = AWSCredentialIdentity(
            accessKey: accessKey,
            secret: secretKey
        )
        let identityResolver = try StaticAWSCredentialIdentityResolver(credentials)
        
        // Create client configuration
        let clientConfig = try BedrockRuntimeClient.BedrockRuntimeClientConfiguration(
            region: region
        )
        clientConfig.awsCredentialIdentityResolver = identityResolver
        
        // Initialize the client
        client = BedrockRuntimeClient(config: clientConfig)
    }
    
    // MARK: - Stream Management
    
    // Create input stream
    private func createInputStream() -> AsyncThrowingStream<BedrockRuntimeClientTypes.InvokeModelWithBidirectionalStreamInput, Error> {
        return AsyncThrowingStream { continuation in
            self.inputContinuation = continuation
        }
    }
    
    
    // MARK: - Session Management
    
    func startSession() async throws {
        if client == nil {
            try initializeClient()
        }
        guard let bedrockClient = client else {
            throw NovaSonicError.streamInitializationFailed("Failed to initialize Bedrock client")
        }
        
        do {
            // Create input stream
            let inputStream = createInputStream()
            
            // Create input object
            let input = InvokeModelWithBidirectionalStreamInput(
                body: inputStream,
                modelId: modelId
            )
            
            // Call API
            print("Call API")
            let output = try await bedrockClient.invokeModelWithBidirectionalStream(input: input)
            
            // Save output stream
            print("Save output stream")
            guard let outputStream = output.body else {
                throw NovaSonicError.streamInitializationFailed("No output stream returned")
            }
            
            isActive = true
        } catch {
            isActive = false
            onError?(NovaSonicError.streamInitializationFailed("Failed to start session: \(error.localizedDescription)"))
            throw NovaSonicError.streamInitializationFailed("Failed to start session: \(error.localizedDescription)")
        }
    }
}

I use the same accessKey, secretKey and region us-east-1 to invoke the model through the nova sonic Python sample, and it works.

Where is the problem, or could you provide example code for calling the invokeModelWithBidirectionalStream API, similar to the Python version, to demonstrate that this function can be used? I can't find any Swift version example code for this API in the documentation.

[sichanyoo]

Hi @zhu-xiaowei, the issue is most likely occurring because no event is being sent via the input stream, and so the connection just hangs there for some time until server closes the connection.

Although it's in PR stage (not yet merged), you can take a look at the integration test in this PR: #1917 for an example usage of the invokeModelWithBidirectionalStream API.

[zhu-xiaowei]

@sichanyoo Thank you very much. The integration tests in this PR looks very detailed. I will debug carefully according to this to find the specific cause.

And we are planning to provide an end-to-end Nova Sonic usage example in the aws-samples/swift-chat cross-platform application, which will better showcase Nova Sonic model capabilities while providing best practices for integrating the SDK code and UI interactions.

[zhu-xiaowei]

Hi @sichanyoo, After testing, the test case runs correctly and can parse the user's sent text content, returned text content and audio information. Here's the code snippet:

...
case .chunk(let payload):
// Process response data
if let responseData = payload.bytes {
    handleResponseData(responseData)
}
...

private func handleResponseData(_ data: Data) {
    // Parse response data
    guard let responseString = String(data: data, encoding: .utf8) else {
        onError?(NovaSonicError.invalidResponse("Failed to decode response data"))
        return
    }
    
    guard let jsonData = responseString.data(using: .utf8),
          let json = try? JSONSerialization.jsonObject(with: jsonData) as? [String: Any],
          let event = json["event"] as? [String: Any] else {
        onError?(NovaSonicError.invalidResponse("Invalid JSON response"))
        return
    }
    
    // Log event type for debugging
    let eventType = event.keys.first ?? "unknown"
    
    // Handle text output event
    if let textOutput = event["textOutput"] as? [String: Any],
       let content = textOutput["content"] as? String {
        if let role = textOutput["role"] as? String {
            print("💬 \(role): \(content)")
            onTranscriptReceived?(role, content)
        } else {
            // Default to ASSISTANT if role is not specified
            print("💬 ASSISTANT: \(content)")
            onTranscriptReceived?("ASSISTANT", content)
        }
    }
    
    // Handle audio output
    if let audioOutput = event["audioOutput"] as? [String: Any],
       let content = audioOutput["content"] as? String {
        
        guard let audioData = Data(base64Encoded: content) else {
            return
        }
        
        // Check if audio data is valid (non-zero)
        if audioData.count > 0 {
            // Send to audio manager
            onAudioReceived?(audioData)
        } else {
            print("⚠️ Received empty audio data")
        }
    }
}

I would like to ask whether the audioData parsing is correct by Data(base64Encoded: content), and what are the default values for FORMAT and SAMPLE_RATE of audioData, And how should I play this streaming audio content? Is there sample code for best practices?

[jbelkins]

@zhu-xiaowei Yes, you should be able to create a chunk of raw audio data as you have described.

For more information on configuring Bedrock audio streaming, see the references we have cited above, plus this doc which describes the Amazon Nova model, including events which may be sent to the server to configure the service:
https://docs.aws.amazon.com/nova/latest/userguide/speech.html

[jbelkins]

How to play audio is more of a platform-dependent matter. Assuming you are developing on an Apple platform, I would look closely at AVAudioEngine (https://developer.apple.com/documentation/avfaudio/avaudioengine) which provides high-level yet powerful access to device audio hardware.

[zhu-xiaowei]

@jbelkins Thank you very much for providing such useful information. Over the past month, I have implemented end-to-end smooth voice conversation capabilities for Nova Sonic based on AVAudioEngine using AWS Swift SDK on iPhone and Mac, and open source it in aws-samples/swift-chat version 2.3.0.

Thank you also to @sichanyoo for providing the test case that allowed me to quickly understand how to use the invokeModelWithBidirectionalStream API.

Finally based on the AWS Swift SDK, we have implemented the following Features for Speech to Speech

• Built-in English Speaking Coach for words and sentences, as well as storytelling scenarios. You can also add Custom System Prompts for voice chatting in different scenarios.
• Support Barge In by default, Also you can disable in system prompt.
• Support Selecting Voice ID in the settings page, including American/British English, and options for male and female voices.
• Support Echo Cancellation, You can talk directly to the device without wearing headphones.
• Support Voice Waveform to display volume level.

The following are some examples.

General Talk

voice_chat.mov

Learn Sentences

learn_sentences.mov

Telling Story on Mac (With barge in feature)

story_mac.mov

Since it involves audio processing, the entire implementation is indeed not simple. Finally, I hope this aws-samples/swift-chat project can increase customers' interest in Nova Sonic and provide some Code References for developers who use this SDK.