introduction
In today's highly interconnected world, the performance and responsiveness of web applications become critical. Most web applications need to communicate with multiple external services, for exampleDatabase, API, third-party serviceswait. Concurrently sending HTTP requests is key to improving application performance and maintaining responsiveness. As an efficient programming language, Golang provides multiple ways to concurrently send HTTP requests. This article will dive into the best techniques and practices for concurrently sending HTTP requests in Golang.
Basic ways to use Goroutines
When it comes to implementing concurrency in Golang, the most straightforward way is to use routine. These are concurrent building blocks in Go, providing a simple and powerful way to execute functions concurrently.
Getting started with Goroutine
To start a routine, just add the `` keyword before the function call. This starts the function as a routine, allowing the main program to continue running independently. It's like starting a task and moving on without waiting for it to complete.
For example, consider the scenario where HTTP requests are sent. Usually, you will call a function similar tosendRequest(), and your program will wait for the function to complete. Using routine, you can do this at the same time:
package main
import (
"fmt"
"net/http"
)
func main() {
urls := []string{"", ""}
for _, url := range urls {
go func(url string) {
resp, err := (url)
if err != nil {
("Error:", err)
return
}
defer ()
// Process response content or other logic }(url)
}
// Wait for all requests to complete // ...
}
This loop starts a new routine for each URL, greatly reducing the time it takes for the program to send all requests. By creating multiple goroutines, each goroutine sends an HTTP request, we can implement multiple requests concurrently.
Methods for concurrent HTTP requests
Since there is no limitgoroutineQuantity, if we put all tasks into concurrencyGoroutineIn-time execution, although it is relatively efficient. But when it is uncontrollablegoroutineDuring the crazy creation, the server system resource utilization rate soared and went down until the process was automaticallykillOtherwise, no other services will be provided.
The above case is that we do not control itgoroutineThe number limit leads to downtime, so as long as we control itgoroutineQuantity can avoid this problem!
WaitGroup
You can use . The WaitGroup type provided by the sync package can help us wait for the completion of a set of concurrent operations.
package main
import (
"fmt"
"net/http"
"sync"
)
func main() {
urls := []string{"", ""}
var wg
for _, url := range urls {
(1)
go func(url string) {
defer ()
resp, err := (url)
if err != nil {
("Error:", err)
return
}
defer ()
// Process response content or other logic }(url)
}
() // Wait for all requests to complete // ...
}
By calling the Wait() method, we can ensure that all goroutines are executed before continuing to perform subsequent operations.
Channels
We can use Channels to control the number of concurrent requests. By creating a capacity-limited channel, effective control over the number of concurrent requests can be achieved
package main
import (
"fmt"
"net/http"
)
func main() {
urls := []string{"", ""}
concurrency := 5
sem := make(chan bool, concurrency)
for _, url := range urls {
sem <- true // Send signals to control the number of concurrent go func(url string) {
defer func() { <-sem }() // Release signal
resp, err := (url)
if err != nil {
("Error:", err)
return
}
defer ()
// Process response content or other logic }(url)
}
// Wait for all requests to complete // ...
}
By setting the capacity of the channel to the number of concurrents, we can ensure that the number of requests sent simultaneously does not exceed the set limit. This method is very effective for controlling the number of concurrent requests.
Worker Pools
Worker Pools is a common concurrency pattern that effectively controls the number of HTTP requests sent concurrently. By creating a fixed set of worker goroutines and assigning request tasks to them to process, we can control the number of concurrent requests, alleviating the pressure of resource competition and overloading.
package main
import (
"fmt"
"net/http"
"sync"
)
type Worker struct {
ID int
Request chan string
Responses chan string
}
func NewWorker(id int) *Worker {
return &Worker{
ID: id,
Request: make(chan string),
Responses: make(chan string),
}
}
func (w *Worker) Start(wg *) {
go func() {
defer ()
for url := range {
resp, err := (url)
if err != nil {
<- ("Error: %s", err)
continue
}
defer ()
// Process response content or other logic <- ("Worker %d: %s", , "Response")
}
}()
}
func main() {
urls := []string{"", ""}
concurrency := 5
var wg
(concurrency)
workers := make([]*Worker, concurrency)
for i := 0; i < concurrency; i++ {
workers[i] = NewWorker(i)
workers[i].Start(&wg)
}
go func() {
for _, url := range urls {
for _, worker := range workers {
<- url
}
}
for _, worker := range workers {
close()
}
}()
for _, worker := range workers {
for response := range {
(response)
}
}
()
}
By using Worker Pools, we can control the number of concurrent requests, and the number of concurrent requests does not exceed the number of workers in Worker Pools.
Please check it out for more in-depth studyExplore the advanced features of Go [processing 1 minute million requests]
Semaphore limits for use Goroutines
The sync/semaphore package provides a clean and efficient way to limit the number of routines running concurrently. This method is especially useful when you want to manage resource allocation more systematically.
package main
import (
"context"
"fmt"
"/x/sync/semaphore"
"net/http"
)
func main() {
// Create a requester and load the configuration requester :=
// Define the list of URLs to be processed urls := []string{"", "", ""}
maxConcurrency := int64(2) // Set the maximum number of concurrent requests
// Create a weighted semaphore sem := (maxConcurrency)
ctx := ()
// traverse the URL list for _, url := range urls {
// Get semaphore weight before starting goroutine if err := (ctx, 1); err != nil {
("Cannot get semaphore: %v\n", err)
continue
}
go func(url string) {
defer (1) // Release semaphore weights when finished
// Use the requester to get the response corresponding to the URL res, err := (url)
if err != nil {
("Request failed: %v\n", err)
return
}
defer ()
("%s: %d\n", url, )
}(url)
}
// Wait for all goroutines to release their semaphore weights if err := (ctx, maxConcurrency); err != nil {
("Cannot get semaphore while waiting: %v\n", err)
}
}
So, what's the best way
After exploring various ways to handle concurrent HTTP requests in Go, the question arises: What is the best way to do it? As is often the case in software engineering, the answer depends on the specific requirements and constraints of the application. Let's consider the key factors for determining the most appropriate approach:
Assess your needs
- Request Scale: If you are dealing with a large number of requests, a work pool or semaphore-based approach can better control resource usage.
- Error handling: If powerful error handling is critical, using a channel or semaphore package can provide more structured error management.
- Rate limit: For applications that need to comply with rate limits, using a channel or semaphore packet limiting routine may be effective.
- Complexity and maintainability: Consider the complexity of each approach. While channels provide more control, they also add complexity. Semaphore packages, on the other hand, provide a more direct solution.
Error handling
Due to the nature of concurrent execution in Go, error handling in routines is a tricky topic. Since routine runs independently, managing and propagating errors can be challenging, but it is critical to building robust applications. Here are some strategies to effectively handle errors in concurrent Go programs:
Concentrated error channel
A common approach is to use a centralized error channel through which all routines can send errors. The main routine can then listen to the channel and take appropriate actions.
func worker(errChan chan<- error) {
// Execute tasks if err := doTask(); err != nil {
errChan <- err // Send any errors to the error channel }
}
func main() {
errChan := make(chan error, 1) // Buffer channel for storing errors go worker(errChan)
if err := <-errChan; err != nil {
// Handle errors ("Error occurred: %v", err)
}
}
Or you can listen to errChan in a different routine.
func worker(errChan chan<- error, job Job) {
// Execute tasks if err := doTask(job); err != nil {
errChan <- err // Send any errors to the error channel }
}
func listenErrors(done chan struct{}, errChan <-chan error) {
for {
select {
case err := <-errChan:
// Handle errors case <-done:
return
}
}
}
func main() {
errChan := make(chan error, 1000) // The channel that stores the error done := make(chan struct{}) // The channel used to notify goroutine to stop go listenErrors(done, errChan)
for _, job := range jobs {
go worker(errChan, job)
}
// Wait for all goroutines to complete (the specific method needs to be implemented according to the actual situation of the code) done <- struct{}{} // Notify goroutine to stop listening error}
Error Group
/x/sync/errgroupPackages provide a convenient way to group multiple routines and handle any errors they generate.Make sure that any subsequent operations will be cancelled once any routine error occurs.
import "/x/sync/errgroup"
func main() {
g, ctx := (())
urls := []string{"", ""}
for _, url := range urls {
// Start a goroutine for each URL (func() error {
// Replace with the actual HTTP request logic _, err := fetchURL(ctx, url)
return err
})
}
// Wait for all requests to complete if err := (); err != nil {
("Error occurred: %v", err)
}
}
This approach simplifies error handling, especially when dealing with a large number of routines.
The above is the detailed explanation of the concurrent processing http of Go advanced features. For more information about Go concurrent processing http, please follow my other related articles!