Golang url.Parse always return Invalid Control Character URL

Question

I'm trying to scrape some site using proxy that i get from free-proxy-list.net and apply it in my local http request using Golang, but when i parse the proxy using url.Parse() always return Invalid Control Character URL

func getProxy() *url.URL {
    proxyUrl := "https://www.proxy-list.download/api/v1/get?type=http&country=US"
    client := &http.Client{}
    req, err := http.NewRequest("GET", proxyUrl, nil)
    resp, err := client.Do(req)
    if err != nil {
        fmt.Println("Error proxy ", err)
    }
    defer resp.Body.Close()
    body, err := ioutil.ReadAll(resp.Body)
    if err != nil {
        fmt.Println("Error response body", err)
    }
    details := string(body)
    temp := strings.Split(details, "\n")
    fmt.Println("http://" + temp[rand.Intn(30)])
    checkProxy, err := url.Parse("http://" + temp[rand.Intn(10)])
    if err != nil {
        fmt.Println("Bad proxy URL", err)
    }

    return checkProxy
}

@Volker, but when i try it manually example "104.129.11.35:3128" it's work but when i try to get the value from strings.split, it's always return Invalid Control Character URL — Rizal Yogi Pratama
– Rizal Yogi Pratama, Commented May 2, 2019 at 3:39

Steffen Ullrich · Accepted Answer · 2019-05-02 04:17:32Z

7

proxyUrl := "https://www.proxy-list.download/api/v1/get?type=http&country=US"

The content of this URL are lines in the format ip:port\r\n´, i.e. the line delimiter is\r\n`, (DOS/Windows style).

temp := strings.Split(details, "\n")

This splits the content by \n, i.e. the UNIX style line delimiter. This leaves the \r from the DOS line delimiter in the string, resulting in ip:port\r.

... always return Invalid Control Character URL

It is the remaining \r in the line it is complaining about.

answered May 2, 2019 at 4:17

Steffen Ullrich

125k11 gold badges155 silver badges194 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

tushortz Feb 6 at 0:32

I had to use strings.Trim("mytext", "\n\r")

svetha.cvl · Accepted Answer · 2019-05-02 04:34:01Z

3

There are trailing spaces and carriage returns in the proxy list. Try

fmt.Println("http://" + strings.TrimSpace(temp[rand.Intn(30)]))
checkProxy, err := url.Parse("http://" + strings.TrimSpace(temp[rand.Intn(10)]))

This works.

answered May 2, 2019 at 4:34

svetha.cvl

8444 silver badges8 bronze badges

Collectives™ on Stack Overflow

Golang url.Parse always return Invalid Control Character URL

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related