Write a CSV file asynchronously in Python

Question

I am writing a CSV file with the following function:

import csv
import os
import aiofiles


async def write_extract_file(output_filename: str, csv_list: list):
    """
    Write the extracted content into the file
    """
    try:
        async with aiofiles.open(output_filename, "w+") as csv_file:
            writer = csv.DictWriter(csv_file, fieldnames=columns.keys())
            writer.writeheader()
            writer.writerows(csv_list)
    except FileNotFoundError:
        print("Output file not present", output_filename)
        print("Current dir: ", os.getcwd())
        raise FileNotFoundError

However, as there is no await allowed over writerows method, there are no rows being written into the CSV file.
How to resolve this issue? Is there any workaround available?
Thank you.
Entire code can be found here.

note that using async io for local files tends to be slower than synchronous io. you're probably off just using a sync (i.e. non-async) method and wrapping in loop.run_in_executor so that the async code can interact with it nicely — Sam Mason
– Sam Mason, Commented Aug 9, 2020 at 9:02

alex_noname · Accepted Answer · 2020-08-10 09:02:44Z

8

In my opinion it’s better not to try to use the aiofiles with the csv module and run the synchronous code using loop.run_in_executor and wait it asynchronously like below:

def write_extract_file(output_filename: str, csv_list: list):
    """
    Write the extracted content into the file
    """
    try:
        with open(output_filename, "w+") as csv_file:
            writer = csv.DictWriter(csv_file, fieldnames=columns.keys())
            writer.writeheader()
            writer.writerows(csv_list)
    except FileNotFoundError:
        print("Output file not present", output_filename)
        print("Current dir: ", os.getcwd())
        raise FileNotFoundError


async def main():
    loop = asyncio.get_running_loop()
    await loop.run_in_executor(None, write_extract_file, 'test.csv', csv_list)

edited Aug 10, 2020 at 9:02

answered Aug 9, 2020 at 19:32

alex_noname

33.2k6 gold badges95 silver badges110 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Sam Mason Over a year ago

note that this runs write_extract_file synchronously. you need to use await loop.run_in_executor(None, write_extract_file, 'test.csv', csv_list) to actually run it asynchronously

royce3 Over a year ago

The problem with this approach is if you need to perform async operations in the middle of writing the csv

Learner Jan 8 at 2:51

@royce3 exactly! If I liverage one coroutine for collecting data and another for converting data into a pandas dataframe after some preprocessing and then write to csv, then it will cause problems.

Fomalhaut · Accepted Answer · 2021-12-09 10:25:33Z

8

You can use aiocsv. Here is a quick example of writing a row to a CSV file asynchronously:

import asyncio
import aiofiles
from aiocsv import AsyncWriter

async def main():
    async with aiofiles.open('your-path.csv', 'w') as f:
        writer = AsyncWriter(f)
        await writer.writerow(['name', 'age'])
        await writer.writerow(['John', 25])

asyncio.run(main())

For more examples follow: https://pypi.org/project/aiocsv/

edited Dec 9, 2021 at 10:25

answered Dec 9, 2021 at 10:18

Fomalhaut

9,99111 gold badges59 silver badges111 bronze badges

Comments

CrookedCloud · Accepted Answer · 2020-08-13 14:07:34Z

-1

You can use aiofiles, you just gotta convert the dict to a row :)

import aiofiles

async def write_extract_file(
    output_filename: str, csv_list: list
):

    cols = columns.keys()

    async with aiofiles.open(output_filename, mode='w+') as f_out:

        await f_out.write(','.join(cols)+'\n')

        for data in csv_list:

            line = []

            for c in cols:
                line.append(str(data[c]) if c in data else '')

            line = ','.join(line) + '\n'
    
            await f_out.write(line)

answered Aug 13, 2020 at 14:07

CrookedCloud

1221 silver badge4 bronze badges

1 Comment

royce3 Over a year ago

This code doesn't handle a lot of the logic that the csv module is performing, like quoting columns that have commas in them for example.

Collectives™ on Stack Overflow

Write a CSV file asynchronously in Python

3 Answers 3

3 Comments

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

3 Comments

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related