Can't open csv file in python without opening it in excel

Question

I have a .csv file generated by a program. When I try to open it with the following code the output makes no sense, even though I have tried the same code with not program generated csv and it works fine.

g = 'datos/1.81/IR20211103_2275.csv'
f = open(g, "r", newline = "")
f = f.readlines()
print(f)

The output of the code looks like this

['ÿþA\x00l\x00l\x00 \x00t\x00e\x00m\x00p\x00e\x00r\x00a\x00t\x00u\x00r\x00e\x00s\x00 \x00i\x00n\x00 \x00°\x00F\x00.\x00\r',
 '\x00\n',
 '\x00\r',
 '\x00\n',
 '\x00D\x00:\x00\\\x00O\x00n\x00e\x00D\x00r\x00i\x00v\x00e\x00\\\x00M\x00A\x00E\x00S\x00T\x00R\x00I\x00A\x00 \x00I\x00M\x00E\x00C\x00\\\x00T\x00e\x00s\x00i\x00s\x00\\\x00d\x00a\x00t\x00o\x00s\x00\\\x001\x00.\x008\x001\x00\\\x00I\x00R\x002\x000\x002\x001\x001\x001\x000\x003\x00_\x002\x002\x007\x005\x00.\x00i\x00s\x002\x00\r',

However, when I first open the file with excel and save it as a .csv (replacing the original with the .csv from excel), the output is as expected, like this:

['All temperatures in °F.\r\n',
 '\r\n',
 'D:\\OneDrive\\MAESTRIA IMEC\\Tesis\\datos\\1.81\\IR20211103_2275.is2\r\n',
 '\r\n',
 '",1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,"\r\n',

I have also tried csv.reader() and doesn't work either.

Does anyone know what's going on and how can I solve it? How can I open my .csv without opening and saving it from excel first? The source program is SmartView from Fluke which reads thermal image file .is2 and converts it into a .csv file

Thank you very much

Hi Try opening in notepad/text editor and see whether the characters are displayed properly before saving using excel. — Allen Johnson
– Allen Johnson, Commented Nov 9, 2021 at 20:02
open(r"datos/1.81/IR20211103_2275.csv", encoding="utf16", newline="") It's UTF-16 encoding, so you need to specify it, @DanielGomez. Also I'd recommend you to use with working with files, check Reading and Writing Files. — Olvin Roght
– Olvin Roght, Commented Nov 9, 2021 at 20:13
Hello @AllenJohnson. When I open it in notepad the characters are displayed properly without the \x00 bite — jRortma
– jRortma, Commented Nov 9, 2021 at 20:13
Ok @DanielGomez, Check Olvin's comment, That is the solution. — Allen Johnson
– Allen Johnson, Commented Nov 9, 2021 at 20:29

Olvin Roght · Accepted Answer · 2021-11-09 21:25:19Z

1

Your file is encoded with UTF-16 (Little Endian byte order). You can specify file encoding using encoding argument of open() function (list of standard encodings and their names you can find here).

Also I'd recommend to not use .readlines() as it will keep trailing newline chars. You can read all file content into as string (using .read()) and apply str.splitlines() to ... split string into a list of lines. Alternatively you can also consume file line by line and call str.rstrip() to cut trailing newline chars.

Final code:

filename = "datos/1.81/IR20211103_2275.csv"
with open(filename, encoding="utf16") as f:
    lines = f.read().splitlines()
    # OR
    lines = [line.rstrip() for line in f]

answered Nov 9, 2021 at 21:25

Olvin Roght

7,8432 gold badges19 silver badges39 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Mustafa Fatih Şen · Accepted Answer · 2021-11-10 21:03:11Z

0

g = 'datos/1.81/IR20211103_2275.csv'

f = open(g, "r", newline = "",encoding="utf-16")

f = f.readlines()

print(f)

try this one it may help

edited Nov 10, 2021 at 21:03

answered Nov 9, 2021 at 20:13

Mustafa Fatih Şen

745 bronze badges

1 Comment

Olvin Roght Over a year ago

It won't help as it is not UTF-8 but UTF-16.

Collectives™ on Stack Overflow

Can't open csv file in python without opening it in excel

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related