10

On my Windows box, I usually did this in python 2 to write a csv file:

import csv
f = open("out.csv","wb")
cr = csv.writer(f,delimiter=';')
cr.writerow(["a","b","c"])
f.close()

Now that python 3 forbids writing text files as binary, that piece of code does not work anymore. That works:

import csv
f = open("out.csv","w",newline='')
cr = csv.writer(f,delimiter=';')
cr.writerow(["a","b","c"])
f.close()

Problem is: newline parameter is unknown to Python 2.

Of course, omitting the newline results in a csv file with too many \r chars, so not acceptable.

I'm currently performing a backwards compatible process to progressively migrate from python 2 to python 3.5 There are a lot of those statements in all my modules.

My solution was embedding the code in a custom module, and the custom module returns file handler + writer object. A python version check is done inside the module, which allows any module using my module to work whatever python version without too much hacking.

Is there a better way?

Improve this question
1
  • 1
    I can't help but wonder what @Raymond Hettinger — Python core developer and creator of the csv module — would suggest... Commented Oct 6, 2016 at 12:46

2 Answers 2

Reset to default
14

On Windows, I found a python 2 & 3 compliant way of doing it changing csv lineterminator option (which defaults to "\r\n" which makes one \r too many when file is open in text mode in Windows)

import csv

with open("out.csv","w") as f:
    cr = csv.writer(f,delimiter=";",lineterminator="\n")
    cr.writerow(["a","b","c"])
    cr.writerow(["d","e","f"])
    cr.writerow(["a","b","c"])
    cr.writerow(["d","e","f"])

Whatever the python version, that will create a csv file without the infamous "blank lines".

The only drawback is that on Linux, this method would produce \r-free files, which is maybe not the standard (although files still opens properly in excel, no blank lines and still several lines :))

the problem persists on 3.6.2 (Just checked myself like I should have some time ago)

An alternative is to use a dictionary as arguments:

write_args = {"mode":"wb"} if bytes is str else {"mode":"w","newline":""}

(comparing bytes to str is one of the many ways to tell python 2 from python 3, in python 3 types are different, and it's very related to our current problem BTW).

Now we can pass those arguments with args unpacking:

with open("out.csv",**write_args) as f:
    cr = csv.writer(f,delimiter=";")
Improve this answer
Sign up to request clarification or add additional context in comments.

13 Comments

Interesting, because the lines produced are still terminated by "\r\n" in the output file (on Windows in both Python 2 & 3).
of course, since when you write \n in text mode on windows it writes \r\n. BUT default line terminator is \r\n for csv (which is probably an issue) so writing this on windows does \r\r\n (file does not check if there's already a \r!). Yes you did well in pinging Raymond Hettinger because I'm sure the lineterminator thing would need an update in csv python 3 module. My vision: changing it to \n only for windows platform would fix everything.
I just tried it (on Windows) in both Python 2 & 3, with and without a lineterminator="\n", and there was never a \r\r\n produced — so I'm not sure I understand the issue to which you refer.
@martineau: what did you witness? without lineterminator, in both python 2 and 3 it should produce corrupt files (with 1 blank line after every data line).
In all cases each line was terminated with \r\n. I also changed the first writerow to cr.writerow(["a","b\nx","c"]) and it changed the embedded newline too (and put quotes around the string: i.e. a;"b\r\nx";c was written to the file).
|
4

For both reading and writing csv files, I've found no better way either — however I would encapsulate into a separate function as shown below. The advantage being that the logic is all in one place instead of duplicated if it's needed more than once.

import csv
import sys

def open_csv(filename, mode='r'):
    """Open a csv file in proper mode depending on Python verion."""
    return(open(filename, mode=mode+'b') if sys.version_info[0] == 2 else
           open(filename, mode=mode, newline=''))

with open_csv('out.csv', 'w') as f:
    writer = csv.writer(f, delimiter=';')
    writer.writerow([1, 2, 3])
    writer.writerow(['a', 'b', 'c'])

The open_csv() utility could be simplified slightly by using the technique shown in @Jean-François Fabre's Dec 8, 2020 update to his answer to detect what version of Python is being used:

def open_csv(filename, mode='r'):
    """Open a csv file in proper mode depending on Python verion."""
    return(open(filename, mode=mode+'b') if bytes is str else
           open(filename, mode=mode, newline=''))
Improve this answer

3 Comments

good, but I found an even better way. I updated my own answer.
@Jean-FrançoisFabre: I like how your Dec 8, 2020 update checks the version and something like it could be used in my answer instead of sys.version to determine what to do. FWIW, I also think the args unpacking used in that same update is also very clever.
thanks for the positive feedback! feel free to edit your answer. I don't mind.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.