Create numpy array with fromiter

Question

I am trying to make numpy arrays from data stored in sqlite3 and have an issue with the strings. From what I understand, fromiter is the most efficient way to accomplish this task. After getting the data with cursor.fetchall(), the string is correct. However after going into the numpy array it gets lost.

import sqlite3
import numpy as np

connection = sqlite3.connect('fermControlData.db')

with connection:
    cursor = connection.cursor()
    sql = "SELECT Time, bTemp, aTemp, heaterStatus, gasMonitor, Location FROM dataLog WHERE beerName= 'Red Sled' ORDER BY Time"
    cursor.execute(sql)
    data = cursor.fetchall()
    print(data[0], type(data[0][5]))
    dataLog = np.fromiter(data, dtype=('f, f, f, i, i8, S'))
    print(dataLog[0])

And this is the output-

(1480890498.4052606, 65.53, 66.42, 0, 0.0, 'Warm1') <class 'str'>
(1480890496.0, 65.52999877929688, 66.41999816894531, 0, 0, b'')

As you can see the string 'Warm1' is getting converted to b' '.

Do you know why that happens and how to fix it so that the string is imported?

Many thanks!

when using fromiter, you have to give max size of strings. could you try dtype=('f, f, f, i, i8, <U5') ? BTW i8 seems strange since corresponding data is a float. — Jean-François Fabre
– Jean-François Fabre ♦, Commented Jan 5, 2017 at 21:50
b' ' is how Py3 displays bytestrings, the S dtype. Try U5 for the native unicode. — hpaulj
– hpaulj, Commented Jan 5, 2017 at 22:42

user7138814 · Accepted Answer · 2017-01-05 21:53:47Z

1

Strings in numpy arrays are fixed lenght and in the case of fromiter you have to specify that length in advance. E.g. with dtype=('f, f, f, i, i8, S5').

answered Jan 5, 2017 at 21:53

user7138814

2,05112 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Create numpy array with fromiter

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related