Pandas type error trying to plot

Question

I'm trying to create a basic scatter plot based on a Pandas dataframe. But when I call the scatter routine I get an error "TypeError: invalid type promotion". Sample code to reproduce the problem is shown below:

t1 = pd.to_datetime('2015-11-01 00:00:00')
t2 = pd.to_datetime('2015-11-02 00:00:00')

Time = pd.Series([t1, t2])
r = pd.Series([-1, 1])

df = pd.DataFrame({'Time': Time, 'Value': r})
print(df)

print(type(df.Time))
print(type(df.Time[0]))

fig = plt.figure(figsize=(x_size,y_size))
ax = fig.add_subplot(111)
ax.scatter(df.Time, y=df.Value, marker='o')

The resulting output is

        Time  Value
0 2015-11-01     -1
1 2015-11-02      1
<class 'pandas.core.series.Series'>
<class 'pandas.tslib.Timestamp'>

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-285-f4ed0443bf4d> in <module>()
     15 fig = plt.figure(figsize=(x_size,y_size))
     16 ax = fig.add_subplot(111)
---> 17 ax.scatter(df.Time, y=df.Value, marker='o')

C:\Anaconda3\lib\site-packages\matplotlib\axes\_axes.py in scatter(self, x,    y, s, c, marker, cmap, norm, vmin, vmax, alpha, linewidths, verts, **kwargs)
   3635             edgecolors = 'face'
   3636 
-> 3637         offsets = np.dstack((x, y))
   3638 
   3639         collection = mcoll.PathCollection(

C:\Anaconda3\lib\site-packages\numpy\lib\shape_base.py in dstack(tup)
    365 
    366     """
--> 367     return _nx.concatenate([atleast_3d(_m) for _m in tup], 2)
    368 
    369 def _replace_zero_by_x_arrays(sub_arys):

TypeError: invalid type promotion

Searching around I've found a similar post Pandas Series TypeError and ValueError when using datetime which suggests that the error is caused by having multiple data types in the series. But that does not appear to be the issue in my example, as evidenced by the type information I'm printing.

Note that if I stop using pandas datetime objects and make the 'Time' a float instead this works fine, e.g.

t1 = 1.1 #
t2 = 1.2

Time = pd.Series([t1, t2])
r = pd.Series([-1, 1])

df = pd.DataFrame({'Time': Time, 'Value': r})
print(df)

print(type(df.Time))
print(type(df.Time[0]))

fig = plt.figure(figsize=(x_size,y_size))
ax = fig.add_subplot(111)
ax.scatter(df.Time, y=df.Value, marker='o')

with output

   Time  Value
0   1.1     -1
1   1.2      1
<class 'pandas.core.series.Series'>
<class 'numpy.float64'>

and the graph looks just fine. I'm at a loss as to why the use of a datetime is causing the invalid type promotion error? I'm using Python 3.4.3 and pandas 0.16.2.

Tom Johnson · Accepted Answer · 2015-11-13 18:25:45Z

10

Thanks @martinvseticka. I think your assessment is correct based on the numpy code you pointed me to. I was able to simplify your tweaks a bit more (and added a third sample point) to get

t1 = pd.to_datetime('2015-11-01 00:00:00')
t2 = pd.to_datetime('2015-11-02 00:00:00')
t3 = pd.to_datetime('2015-11-03 00:00:00')

Time = pd.Series([t1, t2, t3])
r = pd.Series([-1, 1, 0.5])

df = pd.DataFrame({'Time': Time, 'Value': r})

fig = plt.figure(figsize=(x_size,y_size))
ax = fig.add_subplot(111)
ax.plot_date(x=df.Time, y=df.Value, marker='o')

The key seems to be calling 'plot_date' rather than 'plot'. This seems to inform mapplotlib to not try to concatenate the arrays.

answered Nov 13, 2015 at 18:25

Tom Johnson

2,2243 gold badges20 silver badges38 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Jeff · Accepted Answer · 2017-08-15 06:11:09Z

8

There is another way, that we should drop uses Series. Just use list for time.

t1 = pd.to_datetime('2015-11-01 00:00:00')
t2 = pd.to_datetime('2015-11-02 00:00:00')

Time = pd.Series([t1, t2])
r = pd.Series([-1, 1])

df = pd.DataFrame({'Time': Time, 'Value': r})
print(df)

print(type(df.Time))
print(type(df.Time[0]))
x_size = 800
y_size = 600
fig = plt.figure(figsize=(x_size,y_size))
ax = fig.add_subplot(111)
ax.scatter(list(df.Time.values), list(df.Value.values), marker='o')

answered Aug 15, 2017 at 6:11

Jeff

1171 silver badge4 bronze badges

2 Comments

Tom Johnson Over a year ago

your approach works nicely too. I edited it to fix the figure size and removed the explicit calls to list (not necessary).

Pier Over a year ago

This worked nicely for me as well, many thanks. But are lists OK to use? So that Matplotlib doesn't try to modify them?

Community · Accepted Answer · 2017-05-23 12:33:33Z

5

Is this what you are looking for?

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt 
import matplotlib.dates as dates

t1 = pd.to_datetime('2015-11-01 00:00:00')
t2 = pd.to_datetime('2015-11-02 00:00:00')

idx = pd.Series([t1, t2])
s = pd.Series([-1, 1], index=idx)

fig, ax = plt.subplots()
ax.plot_date(idx, s, 'v-')
plt.tight_layout()
plt.show()

I'm new to Python so hopefully I'm not wrong. Basically, I tried to adapt your example according to https://stackoverflow.com/a/13674286/99256.

The problem with your script is that numpy tries to concatenate df.Time and df.Value series and it can't find a suitable type for the new array because one array is numeric and the second one is composed of Timestamp instances.

edited May 23, 2017 at 12:33

CommunityBot

11 silver badge

answered Nov 13, 2015 at 13:26

MartyIX

28.8k33 gold badges143 silver badges218 bronze badges

2 Comments

MartyIX Over a year ago

Yeah, well, I was looking extensively into the issue and this is really all I know even though it is not a solution per se.

Tom Johnson Over a year ago

Thanks @martinvseticka. I was able to update your suggestion a bit (see below). This is certainly non-intuitive behavior - I would have thought the X and Y arrays could be different data types.

N.C. van Gilse · Accepted Answer · 2017-05-29 09:22:08Z

5

scatter plots have some properties that cannot be simulated in plot or plot_date (as the ability to plot markers with varying size).

Converting the Time series of type:pandas.tslib.Timestamp to a list of type:datetime.datetime before plotting the scatter did the trick for me:

times = [d.to_pydatetime() for d in df.Time]]
ax.scatter(times, y=df.Value, marker='o')

answered May 29, 2017 at 9:22

N.C. van Gilse

1312 silver badges3 bronze badges

Comments

Edward Weinert · Accepted Answer · 2018-02-14 21:55:05Z

2

I've changed the type of datetime column to string in fly:

plt.scatter(df['Date'].astype('str'), df['Category'], s=df['count'])

and the scatter plot works. Regards

answered Feb 14, 2018 at 21:55

Edward Weinert

5531 gold badge5 silver badges9 bronze badges

Comments

Gingerbread · Accepted Answer · 2017-07-14 17:43:45Z

1

You can also do something like this:

    import matplotlib.pyplot as plt
    import numpy as np
    import pandas as pd
    import datetime

    df = pd.DataFrame({"Time":["2015-11-01 00:00:00", "2015-11-02 00:00:00"], "value":[ 1, -1]})
    df['Time'] = pd.to_datetime(df['Time'])
    fig, ax = plt.subplots()
    ax.scatter(np.arange(len(df['Time'])), df['value'], marker='o')
    ax.xaxis.set_ticks(np.arange(len(df['Time'])))
    ax.xaxis.set_ticklabels(df['Time'], rotation=90)
    plt.xlabel("Time")
    plt.ylabel("value")

    plt.show()

answered Jul 14, 2017 at 17:43

Gingerbread

2,1328 gold badges27 silver badges39 bronze badges

Comments

hardik jamnal · Accepted Answer · 2019-08-09 10:54:56Z

0

All the above answers are amazing. But, in my case, the error is fixed by updating the libraries. That you can do by Conda terminal with the command, conda update --all

answered Aug 9, 2019 at 10:54

hardik jamnal

1

Collectives™ on Stack Overflow

Pandas type error trying to plot

7 Answers 7

Comments

2 Comments

2 Comments

Comments

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

7 Answers 7

Comments

2 Comments

2 Comments

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related