go http requests in loop

Question

I've recently started with GoLang and tried following..

package main

import (
    "fmt"
    "strings"
    "net/http"
    "io/ioutil"
)

func main() {

    url := "https://uri.api.dev"

    payload := strings.NewReader("param1=example&version=2")

    req, _ := http.NewRequest("POST", url, payload)

    req.Header.Add("content-type", "application/x-www-form-urlencoded")

    for i := 1; i <= 10; i++ {

        res, _ := http.DefaultClient.Do(req)

        body, _ := ioutil.ReadAll(res.Body)

        fmt.Println(string(body))

    }

    defer res.Body.Close()

}

It throws 'undefined: res' error when i try to run this. I want to make 10 http requests to an API uri and res.Body.Close() at last to keep connection persistent for speed. How can we access res variable outside of its scope and make this code work. Please help, Thanks

you don't want to access it outside of its scope, otherwise you can only properly handle the last response. Also, don't ignore errors, and don't try to reuse the http.Request. — Mr_Pink
– Mr_Pink, Commented Oct 30, 2017 at 18:48
I've tried this approach to make speed/response faster. Would you be able to post an example to understand this better please? Thanks @JimB — seoppc
– seoppc, Commented Oct 30, 2017 at 18:50
Things can appear very fast when they error out immediately and you ignore the errors. The proper pattern for handling an http response is right at the top of the documentation page: golang.org/pkg/net/http — Mr_Pink
– Mr_Pink, Commented Oct 30, 2017 at 18:55
You do it the same way you make one request; make the http.Request with a new payload, Do() the request, handle the response. — Mr_Pink
– Mr_Pink, Commented Oct 30, 2017 at 19:12
That's not an issue if you're handling the response in the loop. — Mr_Pink
– Mr_Pink, Commented Oct 30, 2017 at 19:15

Nate · Accepted Answer · 2017-10-30 20:07:18Z

6

Go has block scope so if you create a variable in an inner scope you cannot access it in the outer scope. You would need to declare res outside your loop to access it after the loop is complete.

var res *http.Response

for i := 1; i <= 10; i++ {

    res, _ = http.DefaultClient.Do(req)

    body, _ := ioutil.ReadAll(res.Body)

    fmt.Println(string(body))

}

//This is wrong
defer res.Body.Close()

As noted above, defer res.Body.Close() above is wrong too because you should close each new body you read.

Also as noted, you should definitely be using and checking err on all your calls.

One brief and naive way I might change this:

for i := 1; i <= 10; i++ {
    url := "https://uri.api.dev"
    payload := strings.NewReader("param1=example&version=2")
    req, err := http.NewRequest("POST", url, payload)
    if err != nil {
        //Specific error handling would depend on scenario
        fmt.Printf("%v\n", err)
        return
    }
    req.Header.Add("content-type", "application/x-www-form-urlencoded")

    res, err := http.DefaultClient.Do(req)
    if err != nil {
        //Specific error handling would depend on scenario
        fmt.Printf("%v\n", err)
        return
    }

    body, err := ioutil.ReadAll(res.Body)
        if err != nil {
        //Specific error handling would depend on scenario
        fmt.Printf("%v\n", err)
        return
    }

    fmt.Println(string(body))
    res.Body.Close()
}

Something to think about here. You are wanting to make 10 calls to an API in serial. A natural extension to that is to do the test in parallel or at least concurrently using the 'go' keyword. For that you can't share variables across your loop iterations. Something like:

for i := 1; i <= 10; i++ {
    //Where testApi is the whole process encapsulated for one iteration
    go testApi()
}

Finally, don't prematurely optimize. Variables with restricted scope will prevent bugs and maintenance headaches long term.

Hope this helps.

Ninja Edit:

Do not make loops start at a non-zero number without a good reason, this is probably the most common way to think about your loop below. The 1 - 10 version feels very VB/VB.Net.

for i := 0; i < 10; i++ {
    //Where testApi is the whole process encapsulated for one iteration
    go testApi()
}

edited Oct 30, 2017 at 20:07

answered Oct 30, 2017 at 19:54

Nate

5,4078 gold badges46 silver badges52 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

seoppc Over a year ago

Thanks a lot for great explanation. How can we connection persistent using this approach as calling res.Body.Close() inside loop would close the connection each time.

Nate Over a year ago

Here's some info from the docs. It appears that closing the body is critical to doing exactly what you want, keep the TCP connection open. The call to Close() is intended for the Reader, not the TCP connection. "If the returned error is nil, the Response will contain a non-nil Body which the user is expected to close. If the Body is not closed, the Client's underlying RoundTripper (typically Transport) may not be able to re-use a persistent TCP connection to the server for a subsequent "keep-alive" request."

seoppc Over a year ago

Thanks @Nate for great answer, you helped a lot. :)

seoppc Over a year ago

I've already implemented goroutines for concurrency, I also want to implement parallelism, Can you please recommend me a better way for that, could not find anything related online. Thanks once again.

Nate Over a year ago

Take a look at golang.org/pkg/runtime/#GOMAXPROCS. Basically you set the number of processors you want to use and the runtime spreads your goroutines across them. With modern Go, if you have more than one CPU or core, it should adjust GOMAXPROCS accordingly.

Nate · Accepted Answer · 2017-10-31 16:07:10Z

0

You are trying to close the var res that is inside the for loop.

package main

import (
    "fmt"
    "io/ioutil"
    "net/http"
    "strings"
)

func main() {

    url := "https://uri.api.dev"

    payload := strings.NewReader("param1=example&version=2")

    req, _ := http.NewRequest("POST", url, payload)

    req.Header.Add("content-type", "application/x-www-form-urlencoded")

    for i := 1; i <= 10; i++ {
        // just close it here
        res, _ := http.DefaultClient.Do(req)
        defer res.Body.Close()

        body, _ := ioutil.ReadAll(res.Body)

        fmt.Println(string(body))
    }
}

ps.: you could run those in go routines and sync the response for better performance

Update by "Nate" to avoid confusion. The defer call above is not recommended. Checkout this code at https://play.golang.org/p/6mQ_3-4aHM or below:

package main

import "fmt"

func main() {

    for i := 0; i < 10; i++ {
        defer fmt.Printf("iter %d\n", i)
    }
    fmt.Println("end of main")

}

and the output:

end of main
iter 9
iter 8
iter 7
iter 6
iter 5
iter 4
iter 3
iter 2
iter 1
iter 0

edited Oct 31, 2017 at 16:07

Nate

5,4078 gold badges46 silver badges52 bronze badges

answered Oct 30, 2017 at 19:53

Rafael Quintela

2,2382 gold badges15 silver badges14 bronze badges

2 Comments

Mr_Pink Over a year ago

Calling defer in a loop is usually a bug. Here it's preventing the re-use of the http connections between calls, and since you're ignoring the error, it will panic on the second round.

Nate Over a year ago

Good catch JimB. I didn't realize that was the case so I edited the answer to prevent confusion.

Collectives™ on Stack Overflow

go http requests in loop

2 Answers 2

5 Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related