Unable to write values from list to csv python

Question

I'm trying to write some data from my InfluxDB database to a csv. While the influxdb-cli does allow me to export data to csv, I am not allowed to run the cli on the server myself. I'm using the influx-db python package to query the database.

Now, here's the code that I'm using:

import csv
from influxdb import InfluxDBClient

USER='root'
PASSWORD='root'
DBNAME='sensordata'
HOST='localhost'
PORT=8086

client = InfluxDBClient(HOST,PORT,USER,PASSWORD,DBNAME)
query="select * from home where time > '2017-11-15 14:55:00' AND time <= '2017-11-15 14:55:05' tz('Asia/Kolkata')"
result = client.query(query,epoch='ns')
exported_data = list(result.get_points())
with open("output.csv", "a", newline='') as fp:
    writer = csv.writer(fp, dialect='excel')
    for line in exported_data:
        print(line)
        writer.writerow(line)

The problem is that when I print the lines, I get both key and values like so:

{'time': 1510737900336919297, 'value1': 18.84, 'value10': 19.83, 'value2': 18.56, 'value3': 12.61, 'value4': 17.57, 'value5': 16.6, 'value6': 16.81, 'value7': 12.84, 'value8': 11.54, 'value9': 14.26}
{'time': 1510737901370333995, 'value1': 11.32, 'value10': 12.98, 'value2': 12.34, 'value3': 12.22, 'value4': 11.08, 'value5': 12.07, 'value6': 17.62, 'value7': 14.68, 'value8': 16.87, 'value9': 11.4}
{'time': 1510737902403461281, 'value1': 12.37, 'value10': 16.18, 'value2': 18.83, 'value3': 14.59, 'value4': 11.79, 'value5': 18.52, 'value6': 11.25, 'value7': 17.28, 'value8': 10.54, 'value9': 19.1}
{'time': 1510737903436997966, 'value1': 13, 'value10': 12.04, 'value2': 10.02, 'value3': 14.28, 'value4': 14.51, 'value5': 17.3, 'value6': 16.14, 'value7': 15.04, 'value8': 13.16, 'value9': 10.47}
{'time': 1510737904470366806, 'value1': 16.2, 'value10': 10.83, 'value2': 12.64, 'value3': 13.51, 'value4': 13.74, 'value5': 11.52, 'value6': 13.42, 'value7': 13.14, 'value8': 16.6, 'value9': 11.24}

But, the csv file contains just the keys like so:

time,value1,value10,value2,value3,value4,value5,value6,value7,value8,value9
time,value1,value10,value2,value3,value4,value5,value6,value7,value8,value9
time,value1,value10,value2,value3,value4,value5,value6,value7,value8,value9
time,value1,value10,value2,value3,value4,value5,value6,value7,value8,value9
time,value1,value10,value2,value3,value4,value5,value6,value7,value8,value9

What am I doing wrong here?

Also, is it possible for me to have the csv formatted like:

time_heading, value1_heading, value2_heading ....
time_value, value1_value, value2_value ....
time_value, value1_value, value2_value ....
time_value, value1_value, value2_value ....
.
.
.

I did try out a few solutions that I googled but none of them helped. Does anyone know what's going wrong?

Edit

I also wanted to "natural_sort" the headers in since it makes sense to have it in that format. To do that I referred to this answer on SO. But, to do that, I needed to ignore the first item which is "time" and sort the "value" headers only which I did using this

One last thing that I needed is to calculate the offset of the text since some of the measurements have "val" as the field name and some have "value". The offset is 3 and 5 respectively which I will use in the natural sorting snippet. To do that, I referred to this. I finally ended up having the data look like

time,value1,value2,value3,....value10 and not time,value1,value10,value2,value3,...value9 which is what I was looking for.

Here's what the final code looks like:

import csv
from influxdb import InfluxDBClient

USER = 'root'
PASSWORD = 'root'
DB_NAME = 'sensordata'
HOST = 'localhost'
PORT = 8086

client = InfluxDBClient(HOST, PORT, USER, PASSWORD, DB_NAME)
query = "select * from home where time > '2017-11-15 14:55:00' AND time <= '2017-11-15 14:55:05' tz('Asia/Kolkata')"
result = client.query(query, epoch='ns')
exported_data = list(result.get_points())
header_list = list(exported_data[0].keys())

with open("output.csv", "w", newline='') as fp:
    writer = csv.writer(fp, dialect='excel')
    # print(header_list[1:])
    value_header = header_list[1]
    offset = sum(c.isalpha() for c in value_header)
    # print(offset)
    header_list[1:] = sorted(header_list[1:], key=lambda x: int(x[offset:]))
    # print(header_list)
    writer.writerow(header_list)
    for line in exported_data:
        # print(line)
        writer.writerow([line[kn] for kn in header_list])

Thanks to @be_good_do_good for the answer

be_good_do_good · Accepted Answer · 2017-11-20 17:16:48Z

1

You need to do it in below way. This way has a problem of random order of writing data in the csv:

with open("output.csv", "a", newline='') as fp:
    writer = csv.writer(fp, dialect='excel')
    writer.writerow(exported_data[0].keys())
    for line in exported_data:
        print(line)
        writer.writerow(line.values())

You did everything right, but when you pass a dictionary, keys are taken as list, so you need to specify what needs to be written to csv - line.values()

If you are interested in the order of the keys, then do it below way:

with open("output.csv", "a", newline='') as fp:
    writer = csv.writer(fp, dialect='excel')
    header_list = ['time,value1,value10,value2,value3,value4,value5,value6,value7,value8,value9']
    writer.writerow(header_list)
    for line in exported_data:
        print(line)
        writer.writerow([line[kn] for kn in header_list])

edited Nov 20, 2017 at 17:16

answered Nov 20, 2017 at 7:35

be_good_do_good

4,4693 gold badges31 silver badges42 bronze badges

Sign up to request clarification or add additional context in comments.

13 Comments

timgeb Over a year ago

line.values() will be in arbitrary order and therefore produce garbage results.

be_good_do_good Over a year ago

But it will be the same arbitrary as keys are, isn't it?

timgeb Over a year ago

The keys have no order. line.values() may give you a different ordering with respect to the keys which are represented in every iteration of the for loop.

be_good_do_good Over a year ago

i double checked it. line.values are printed as same order of line.keys()

be_good_do_good Over a year ago

Changed the answer in such a way that headers now match the values

|

timgeb · Accepted Answer · 2017-11-20 07:14:44Z

1

Since you don't just have iterables of values, but dictionaries of key:value pairs, you should use a csv.DictWriter instead of a csv.writer.

answered Nov 20, 2017 at 7:14

timgeb

79.2k20 gold badges129 silver badges150 bronze badges

2 Comments

karanrajpal14 Over a year ago

As you can see, I'm using a list here: exported_data = list(result.get_points()) Will a DictWriter help in this case? I'm assuming it won't because of how it's named but I'll give it a whirl either way.

timgeb Over a year ago

@karanRajpal the point is that your list contains dicts. That's exactly what the DictWriter was made for. Each dict in your list is treated as a row by the DictWriter.

Sagun Shrestha · Accepted Answer · 2017-11-20 07:36:08Z

0

Use csv.DictWriter() for dictionaries and csv.writer() for lists.

with open('output.csv','a') as file:
    writer = csv.DictWriter(file, exported_data[0].keys())
    writer.writeheader()
    writer.writerows(exported_data)

answered Nov 20, 2017 at 7:36

Sagun Shrestha

1,19812 silver badges24 bronze badges

1 Comment

karanrajpal14 Over a year ago

I get this error

File "/usr/lib/python3.6/csv.py", line 143, in writeheader     header = dict(zip(self.fieldnames, self.fieldnames)) TypeError: zip argument #1 must support iteration

From what I could gather with a little bit of searching, a dict writer would only be useful if my "exported_data" is a dictionary, correct? In this case, I'm using a list as you can see here: exported_data = list(result.get_points()) What would you suggest me to do here?

Collectives™ on Stack Overflow

Unable to write values from list to csv python

3 Answers 3

13 Comments

2 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

13 Comments

2 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related