Trouble loading numpy array where they show pickle data error in python

Question

I can successfully save & load small arrays using Numpy. Now I am saving the below array using np.save('array.npy')

[0, 100, 0, 5, 10, 15, 20, 25, 30, 25, 20, 15, 10, 5, 0]

when I try to load using np.load('array.npy'), it shows the below error:

    raise ValueError("Cannot load file containing pickled data "
ValueError: Cannot load file containing pickled data when allow_pickle=False

If I try to solve it by adding allow_pickle=True then it shows the below error:

    raise IOError(
OSError: Failed to interpret file 'array.npy' as a pickle

Its really a difficult situation. Please advise! :(

The code I am referring to is below:

def recv():
    import socket
    import time
    import numpy as np

    TCP_IP = "0.0.0.0"
    BUFFER_SIZE = 20  # Normally 1024, but we want fast response

    # receiving CAN frame payload
    TCP_PORT = 5003
    s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
    s.bind((TCP_IP,TCP_PORT))
    s.listen(1)
    conn,addr = s.accept()
    while 1:
        data1 = conn.recv(BUFFER_SIZE)
        if not data1: break
        datalist = list(data1)
        print("CAN payload: %s" % datalist)
        conn.send(data1)  # echo
    conn.close()
    time.sleep(2)
    # ------------------------------------------------------------
    # assembling CAN frame
    from can import Message
    can_msg = Message(is_extended_id=bool(datalist[0]),arbitration_id=datalist[1],data=datalist[2:])

    # printing all received payloads
    print("CAN frame: ",can_msg)
    print("Vehicle speed: ",datalist[2:])

    # Saving all received payloads
    np.save('array.npy',datalist[2:])  # save


def EPS_process():
    # EPS process for Right turn, high speed
    import numpy as np

    print("Starting EPS process")
    speed_array = np.load('array.npy')  # load

Please post the non-working code. When running your example, it works fine, even without allow_pickle=True — herveRilos
– herveRilos, Commented Dec 31, 2020 at 12:04

Farhan Kabir · Accepted Answer · 2021-01-02 13:29:39Z

2

Okay, I want to share what resolves the situation. So basically @hpaulj was suggesting that for multiple processes the read might start before the save is finished. So I added a 3-sec delay before speed_array = np.load('array.npy')

This gives enough time for the NumPy array to save before I can read again. Otherwise, it will read the previously saved array and hence the pickle error.

answered Jan 2, 2021 at 13:29

Farhan Kabir

1692 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

hpaulj · Accepted Answer · 2020-12-31 16:42:14Z

1

That array shouldn't give you any problems:

In [1]: arr = np.array([0, 100, 0, 5, 10, 15, 20, 25, 30, 25, 20, 15, 10, 5, 0])
In [2]: arr
Out[2]: 
array([  0, 100,   0,   5,  10,  15,  20,  25,  30,  25,  20,  15,  10,
         5,   0])
In [3]: np.save('test.npy',arr)
In [4]: np.load('test.npy')
Out[4]: 
array([  0, 100,   0,   5,  10,  15,  20,  25,  30,  25,  20,  15,  10,
         5,   0])

answered Dec 31, 2020 at 16:42

hpaulj

233k14 gold badges260 silver badges392 bronze badges

4 Comments

Farhan Kabir Over a year ago

It is a socket server. I have edited to add the code here. Please check

hpaulj Over a year ago

That potentially makes a big difference. You have to, in some sense, verify that the file that's being read is the same as the one that's saved. For example with multiple processes the read might start before the save is finished.

Farhan Kabir Over a year ago

actually, I am calling these def functions separately at separate times, so it should get ample time to save.

Farhan Kabir Over a year ago

You were right. I added 3 sec time delay, now it is working fine :D

Collectives™ on Stack Overflow

Trouble loading numpy array where they show pickle data error in python

2 Answers 2

Comments

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related