Creating a live plot of CSV data with Matplotlib

Question

I am trying to use Matplotlib to visualize some measurements. The measurements typically last about 24hrs and will include ~30k lines of data in the csv. I've been struggling mostly with getting my plots to actually animate. I can execute the code and it will display a snapshot up to the current point in time but nothing else. When I try to autoscale it, nothing plots and it just defaults to a view of -.6 to +.6 on both axes. Am I supposed to call plt.draw() for this to work? This is what I have so far:

import numpy as np
import datetime as dt
import matplotlib
import matplotlib.pyplot as plt
from matplotlib import animation

FMT = '%Y-%m-%d %H:%M:%S.%f'
data = np.genfromtxt('sampleData.csv', delimiter=',', skip_header=5,
                 names=['x', 'y', 'z', 'w'], dtype=['object', 'int8', 'int8', 'float'])                  

mytime = [dt.datetime.strptime(i.decode('ascii'), FMT) for i in data['x']]
thickness = data['w']

fig = plt.figure()
axes = fig.add_subplot(111)
line, = axes.plot([], [], '.')
plt.show(block=False)

def init():
    line.set_data([],[]) 
    return line,

fig.canvas.draw()

def animate(i):
    xdata = mytime[:i]
    ydata = thickness[:i]
    line.set_data(xdata, ydata)
    plt.draw()
    axes.relim()
    axes.autoscale(True,'both',True) 
    axes.autoscale_view(True,True,True)
    return line,

anim = animation.FuncAnimation(fig, animate, init_func=init,  
                           interval=0, blit=True)

plt.show()

This is a sample line of data from the CSV:

2013-09-25 14:51:15.329091,1,0,439.80,,,,0,0,

Do you mean there's still data incoming and the plot doesn't update? Or that you wanted a moving animation of the existing data? — cphlewis
– cphlewis, Commented Aug 10, 2015 at 19:01
There is data coming in until I stop it. I believe I have gotten it to work but its plotting soo slow. I'm measuring at approx 60Hz and now that im looking at this im not sure matplotlib can keep up. Any advice? — MBlankfield
– MBlankfield, Commented Aug 10, 2015 at 19:51
I wouldn't expect to manage that speed if I was passing all the data through a CSV file, for one thing. (Note: you didn't actually answer my question, and if you've improved the code, you haven't updated the text in the question. Not helpful.) — cphlewis
– cphlewis, Commented Aug 10, 2015 at 20:33
It's really not clear what you are trying to achieve here, so I'm going to try rephrasing @cphlewis' question. You say you're using a CSV file. That implies an existing, static data store. Then you use animation and talk about incoming data over a period of time, which implies an input data stream. So - can you explain what's happening - is there another process constantly appending to / replacing the .csv and you want to update the plot as the new data comes in? Or do you want to animate the data in the csv as if it is incoming in "real time"? Or something else? — J Richard Snape
– J Richard Snape, Commented Aug 11, 2015 at 9:27
Sorry guys, I see the confusion. The .csv is continuously being written by my measurement program. Its not changing previous data, just adding it line by line. I'm attempting to display a realtime visual of that ever growing data. The problem I'm seeing is that the visual slows down tremendously after the first 15 or 20 seconds to the point where it basically stalls out. — MBlankfield
– MBlankfield, Commented Aug 11, 2015 at 11:16

Joe Kington · Accepted Answer · 2015-08-11 13:37:01Z

12

You have two issues. One is because you're effectively drawing things twice, the second is purely human psychology (the plot appears to slow over time because you're adding one point to 10000 vs adding one point to 10 or 100).

Let's discuss the double-draw first:

The FuncAnimation will draw things for you, but by telling it to use blitting, it's only updating the inside of the axes and not the ticks, etc. Therefore, you need to manually call draw, but the animation will be calling draw_artist, as well.

You should be able to get at least a 2x speedup by removing blit=True and plt.draw()

Furthermore, by setting interval=0, you're forcing it to draw constantly, which will effectively force things to lock up. Set the interval to something more reasonable, e.g. 25 (the interval is in milliseconds. "25" is 40 fps.).

For example, this is very smooth for me:

import numpy as np
import matplotlib.pyplot as plt
from matplotlib import animation

y = np.random.normal(0, 1, 10000).cumsum(axis=0)
x = np.arange(y.size)

fig, ax = plt.subplots()
line, = ax.plot([], [], '.')
ax.margins(0.05)

def init():
    line.set_data(x[:2],y[:2])
    return line,

def animate(i):
    i = min(i, x.size)
    xdata = x[:i]
    ydata = y[:i]
    line.set_data(xdata, ydata)
    ax.relim()
    ax.autoscale()
    return line,

anim = animation.FuncAnimation(fig, animate, init_func=init, interval=25)

plt.show()

I've also added ax.margins(0.05) to avoid the axes limits snapping to the next nearest "even" number and giving a "jerky" appearance.

However, because you're progressively plotting more and more data, the rate of change will appear to slow, simply because less of the data appears to be changing over time. Adding one point at the end of 10000 is hardly noticeable, but adding one point at the end of 10 is very noticable.

Therefore, the plot looks much more "exciting" at the beginning than at the end even though it's updating at the same rate.

This has nothing whatsoever to do with matplotlib, and is a consequence of the way you're choosing to animate your data.

To get around that, you might consider moving a "sliding window" through your data and plotting a constant number of points at a time. As an example:

import numpy as np
import matplotlib.pyplot as plt
from matplotlib import animation

y = np.random.normal(0, 1, 1000000).cumsum(axis=0)
x = np.arange(y.size) + 1

fig, ax = plt.subplots()
line, = ax.plot([], [], 'k-')
ax.margins(0.05)

def init():
    line.set_data(x[:2],y[:2])
    return line,

def animate(i):
    win = 300
    imin = min(max(0, i - win), x.size - win)
    xdata = x[imin:i]
    ydata = y[imin:i]
    line.set_data(xdata, ydata)
    ax.relim()
    ax.autoscale()
    return line,

anim = animation.FuncAnimation(fig, animate, init_func=init, interval=25)

plt.show()

edited Aug 11, 2015 at 13:37

answered Aug 11, 2015 at 13:29

Joe Kington

287k73 gold badges621 silver badges474 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

MBlankfield Over a year ago

Thank you Joe! For my specific implementation, it's more beneficial to see the entire plot's shape but I will keep your second example in my back pocket for other projects I have in mind. I also appreciate you cleaning up my code, I'm somewhat new to this. The changes you made helped me realize that I'm going about this visualization all wrong. When I modified your code and ran it, I noticed that it only displayed the data in the .csv up to the point when I executed my code. The slowdown I was seeing was it running out of data, my code just took far longer to get there.

Joe Kington Over a year ago

@MBlankfield - Glad to help! I left out the re-reading the csv file portion, so as you mentioned, it reads it once. To get "live" updates from a csv file efficiently, you'll need a slightly more complex solution. You might consider storing things in a database or HDF at that point, as it will make querying the recently-written points considerably easier/faster.

cphlewis Over a year ago

Alternately, you might be able to read directly from the instrument, so that your code is responsible both for plotting and for writing the log file.

Jaffer Wilson · Accepted Answer · 2018-02-20 11:56:33Z

-1

I have below columns as an operation of pandas filter column :

    Object      A_START_TIME   B_START_TIME  B_STOP_TIME       A_STOP_TIME
0     17          27158762       27158803      27178575          27259032 

1     18          27259557       27259605      27263225          27323232

2     19          27323266       27323299      27332959          27341878

I am looking at plotting something like explained below :

for 17 : A_START_TIME----[B_START_TIME========B_STOP_TIME]-------A_STOP_TIME

For 18 : A_START_TIME----[B_START_TIME========B_STOP_TIME]-------A_STOP_TIME

edited Feb 20, 2018 at 11:56

Jaffer Wilson

7,30313 gold badges77 silver badges161 bronze badges

answered Feb 20, 2018 at 11:20

vish

11 bronze badge

Collectives™ on Stack Overflow

Creating a live plot of CSV data with Matplotlib

2 Answers 2

3 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related