parameters for async GET requests with HTTP.jl

Question

I want to set up a function that sends async GET requests to a list of URLs. For an example, I want to open the google home page with the search bar containing a letter of the alphabet.

using HTTP

endpoint = "https://www.google.com/?q="
letters = vcat('a':'z')

function query_sync()    
    for i in letters
        HTTP.request("GET", "$endpoint/?letter=$i")
        println(i)
    end
end

@time query_sync()
# 3.537998 seconds (7.75 k allocations: 1.795 MiB)

function query_async()    
    @sync for i in letters
        @async begin
            HTTP.request("GET", "$endpoint/?letter=$i")
            println(i)
        end
    end
end

@time query_async()
# 0.399345 seconds (8.95 k allocations: 1.993 MiB, 1.17% compilation time)

So far so good. But if I have rate limits or want to set the number of concurrent connections, what options do I have?

I know that there are connection pool options such as connection_limit and pipeline_limit but it just makes me feel that I've implemented a hackey solution to an async call instead of using the package properly. Will those connection pool options take into account the @async macro, or is it part of the module's larger server functions, unrelated to my use case?

Some considerations I've had if the Connection Pool options aren't for my use case: using channels to limit the availability of URLs to the function, therefore limiting the concurrency. And good old fashion rate limiting by checking calls per unit of time and throttling.

Przemyslaw Szufel · Accepted Answer · 2022-01-30 21:16:23Z

3

Use asynmap with the ntasks parameter:

asyncmap(letters; ntasks=8) do i
    HTTP.request("GET", "$endpoint/?letter=$i")
    println(i)
end

asyncmap "...uses multiple concurrent tasks to map f over a collection (or multiple equal length collections). For multiple collection arguments, f is applied elementwise. ntasks specifies the number of tasks to run concurrently. Depending on the length of the collections, if ntasks is unspecified, up to 100 tasks will be used for concurrent mapping..."

answered Jan 30, 2022 at 21:16

Przemyslaw Szufel

42.5k3 gold badges42 silver badges73 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

parameters for async GET requests with HTTP.jl

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related