Writing Sensor Data in Single Row in Python

Question

I am trying to write data coming in from 2 sensor nodes to a CSV file using Python. Communication is done via Xbee Series 1 in AT mode, with the Xbee End Devices at the 2 sensor nodes passing data to an XBee coordinator connected to my computer. These data would then have to be written to a CSV file. The sensors are connected to an Arduino.

The current problem I am facing, is that sensor data coming in from the 2 nodes are written in this way in the CSV file:

However, I would like the data format to be: i.e. have data from 2 sensor nodes being written into a single row in an instance.

My Python code for writing the CSV files below is:

import serial
import time
import csv

# Arduino
arduino = serial.Serial('COM11', 9600, timeout=1)

time.sleep(3) # wait for Arduino to initialize

while True:
    for datastring in arduino:
        datastring = datastring.strip()  # removes whitespaces and newline char

        if datastring:
            datasplit = datastring.split(',')  # Splits line of txt -> array of strings composed of EA individual sensor data
            field1 = datasplit[0]
            field2 = datasplit[1]
            field3 = datasplit[2]

            with open('testing1.csv', 'ab') as csvfile: # 'ab' to remove newline char after each print
                field1 = field1.replace('\n',"")
                sensor_fields = [field1, field2, field3, time.strftime("%H%M%S")]
                writer = csv.writer(csvfile)
                writer.writerow(sensor_fields)

May I ask where did I go wrong in my code? Thank you all so much! :)

Where in this code are you trying to combine two sensor nodes? — Stephen Rauch
– Stephen Rauch ♦, Commented Jan 15, 2018 at 3:07
also you're constantly opening and closing the file in every loop, highly inefficient — gold_cy
– gold_cy, Commented Jan 15, 2018 at 3:10
@StephenRauch I have serial data coming in from two sensor nodes, to be fed into a computer. As the data is sent in packets by the Xbee modules, and I get 3 sets of data per node, I have split the data into 3 sets as seen in the Python code above: datasplit[0] to datasplit[2]. Thank you for your kind help! :) — galaxy_twirl
– galaxy_twirl, Commented Jan 15, 2018 at 3:15
@aws_apprentice Oh dear. I see May I ask if there is a better way to implement this? (: Thank you for your kind help! :) — galaxy_twirl
– galaxy_twirl, Commented Jan 15, 2018 at 3:16
datastring is still one event from arduino. And it appears to be three columns, but you claim to have more columns than that. Where are these other columns supposed to come from? — Stephen Rauch
– Stephen Rauch ♦, Commented Jan 15, 2018 at 3:19

Mike Robins · Accepted Answer · 2018-01-15 09:28:14Z

1

I get the impression that your difficulty comes from the fact that there are two alternating messages from the serial port. You need to synchronize the CSV write to be when you have complete data, storing the partial data from the first message in the mean time.

import serial
import time
import csv

arduino = serial.Serial('COM11', 9600, timeout=1)

time.sleep(3)

with open('testing1.csv', 'ab') as csvfile:
    writer = csv.writer(csvfile)

    data = []
    while True:

        for datastring in arduino:

            # one should really put some error checking and reporting on the input data
            # as it is coming from an uncontrolled external source
            fields = datastring.strip().split(',')
            fields[0] = fields[0].replace('\n', '')

            # We need to store first set of fields until the second set has been read.
            # first set is tagged '665b'
            # The list data is being used as storage and to indicate that the first line has been read
            if fields[0] == '665b':
                data = fields

            # check if we have complete data from both sensors and write out if so.
            # and ignore if alternating data is out of sync (i.e. first time round)
            elif len(data) == 3 and fields[0] != '665b': 
                writer.writerow(data + fields + [time.strftime("%H%M%S")])
                data = []

I would strongly recommend that you check the format of the data that you read and report errors.

answered Jan 15, 2018 at 9:28

Mike Robins

1,78310 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

galaxy_twirl Over a year ago

Thank you very much for your kind help! ^^ I will try your code later as I was cautioned of formatting my data in the format: 665b, 0b, 1b (for example) as it would make it machine learning unfriendly. I have changed the data instead, to: b665, 0, 1 and I will now find the start character of every data sent by my Xbee. As I have 2 sensor nodes - one at the table, and another at my bed, I have the following: b665, 0, 1 t255, 1, 0 (for example) t -> table; b -> bed I will think of how to adapt your code into my situation. :) Thank you once again! ^^

Mike Robins Over a year ago

I wonder if you need the while True: loop, i would expect that the for d in arduino loop would keep going until the serial link is lost.

galaxy_twirl Over a year ago

I see~ Thank you for your feedback. I have re-written my entire code and I will post it up in my original post later, as an edit. My code now works, but it still has a bit of a problem which I would like to consult the community. May I ask if this will be confusing? Thank you so much once again for your kind help. :)

Collectives™ on Stack Overflow

Writing Sensor Data in Single Row in Python

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related