In-depth analysis of streaming output
I have been calling the openai key before, but I just do streaming calls according to the document. I only know that the streaming is different from the API, but I have not studied its implementation principle in a systematic way.
Let’s take the official streaming output of openai as the entry point.
Overview
Streaming Output is a pattern in HTTP responses, where the server can send some content to the client immediately when generating some content without waiting for the entire response content to be generated. This method is often used in real-time interaction, high-latency operations, or long-term tasks, such as OpenAI's GPT model to generate streaming conversations.
package main
import (
"bufio"
"bytes"
"encoding/json"
"fmt"
"net/http"
"strings"
"time"
)
// Define the necessary data structuretype Message struct {
Role string `json:"role"`
Content string `json:"content"`
}
type RequestBody struct {
Model string `json:"model"`
Messages []Message `json:"messages"`
Temperature float64 `json:"temperature"`
Stream bool `json:"stream"`
}
type Choice struct {
Delta struct {
Content string `json:"content"`
} `json:"delta"`
}
type ResponseBody struct {
Choices []Choice `json:"choices"`
}
const (
apiURL = "/v1/chat/completions" // Replace with the actual API address authToken = "your-auth-token" // Replace with the actual token model = "gpt-3.5-turbo"
temperature = 0.7
)
func StreamHandler(w , r *) {
// Get input from query parameters content := ().Get("content")
if content == "" {
(w, "Missing 'content' parameter", )
return
}
// Construct the request body message := Message{
Role: "user",
Content: content,
}
requestBody := RequestBody{
Model: model,
Messages: []Message{message},
Temperature: temperature,
Stream: true,
}
jsonData, err := (requestBody)
if err != nil {
(w, "Failed to marshal request body", )
return
}
// Create HTTP request req, err := ("POST", apiURL, (jsonData))
if err != nil {
(w, "Failed to create request", )
return
}
("Content-Type", "application/json")
("Authorization", "Bearer "+authToken)
// Set up HTTP client client := &{Timeout: * 50}
resp, err := (req)
if err != nil {
(w, "Failed to get response", )
return
}
defer ()
// Set the response header and turn on the streaming output ().Set("Content-Type", "text/event-stream; charset=utf-8")
().Set("Cache-Control", "no-cache")
().Set("Connection", "keep-alive")
// Make sure ResponseWriter supports Flusher flusher, ok := w.()
if !ok {
(w, "Streaming unsupported", )
return
}
// Handle streaming response scanner := ()
for () {
line := ()
// Process lines starting with "data: " if (line, "data: ") {
line = (line, "data: ")
}
if line == "[DONE]" {
break
}
if line == "" {
continue
}
// parse the response content var chunk ResponseBody
if err := ([]byte(line), &chunk); err != nil {
continue
}
// Send response data to the client step by step for _, choice := range {
content :=
_, err := ([]byte(content))
if err != nil {
(w, "Failed to write response", )
return
}
() // Refresh the buffer }
}
if err := (); err != nil {
(w, "Scanner error", )
return
}
}
func main() {
("/stream", StreamHandler)
("Server started at :8080")
(":8080", nil)
}Core process
After receiving user input, it is sent to the target API as a content parameter.
Turn on streaming output mode and set Stream: true.
Use to send content received from the remote interface to the client step by step.
Key points
1. Streaming response head settings
().Set("Content-Type", "text/event-stream; charset=utf-8")
().Set("Cache-Control", "no-cache")
().Set("Connection", "keep-alive")
Real-time output: Call () after outputting the content to ensure that the data is sent in real time.
After starting the service, access URLs similar to the following through your browser:
http://localhost:8080/stream?content=Hello%20world
The client will gradually receive content, similar to real-time printing of the command line.
1. Streaming response in HTTP protocol
Streaming output uses the characteristics of the HTTP protocol, without closing the connection, and gradually sending data to the client. A typical streaming response will be set as follows:
Content-Type: text/event-stream means that this is an Event Stream, which is used to continuously send data fragments to the client.
Cache-Control: no-cache prevents responses from being cached to ensure that the client receives real-time content.
Connection: keep-alive Keep the connection active and supports multiple data transfers.
2. How streaming output works
The client initiates a request, and the server starts responding after receiving the request.
The server does not generate complete response content at once, but sends some of the generated data segment by segment.
The client processes the data immediately after receiving it without waiting for the complete response to end.
After the data transmission is complete, the server can choose to close the connection or keep the connection to send subsequent data.
Common application scenarios for streaming output
Live chat: When the chat model is generated word by word/sentence, data can be transmitted in real time.
Log monitoring: Push the server's real-time logs line by line to the front end.
Streaming file transfer: such as large file or video streaming.
Real-time progress update: such as task progress bar update.
This is the end of this article about using Golang to implement streaming output. For more related Golang streaming output content, please search for my previous articles or continue browsing the related articles below. I hope everyone will support me in the future!