From here on I will show the approach I took while developing the io module for the pypeln library. The main idea here will be to also use a semaphore, but it will be usedt to limit the creation of the tasks instead of requests, this avoids having to continuously monitor the running tasks. We will encapsulate all the main logic in a class called TaskPool we will define below

Much like a Queue , this TaskPool class has a put method that creates tasks from coroutines, but uses a semaphore before actually calling asyncio.ensure_future to trigger the task. However, instead of using the semaphore via async with , we are calling the methods acquire and release manually because we don’t know when the task will end. The trick to get this working efficiently is adding a callback to the task’s on_done event so that it releases the semaphore for us, this is highly efficient because we don’t have to constantly scan tasks checking if they are done as in Andy’s approach.

TaskPool is used as an async context manager so that when the context is being exited, we await on the remaining tasks using asyncio.gather . This class can be imported form pypeln for whoever wants to make use of it; using it to our advantage we can solve the problem quite easily

Basically we just have to pass the limit variable to the constructor of the TaskPool and then inside the main loop await on the put method, passing it the fetch coroutine. Same as in Andy’s approach we are also able to make an unbounded/unlimited amount of requests if we so desired thanks to the efficient resource management.

You can also use the pypeln.asyncio_task.each function to simplify code a bit

Here we have created a generator called urls iterates to make things clearer. each iterates over urls concurrently and runs the fetch coroutine on each element (works much like the map function from the standard library except it doesn’t return any values). We set the workers parameter to limit (under the hood this creates a TaskPool which receives the parameter), and use the on_start and on_done callbacks to handle the aiohttp.ClientSession which is passed as a parameter to fetch .

For completeness check out this more idiomatic (2020) version of the previous which is more suitable if you are already running inside an async context.