Concatenating pandas DataFrames with datetime columns

Question

I'm trying to concatenate two DataFrames (called firstData and lastData) created from SQL queries. However, when I try to concatenate them, I get an error that says TypeError: can't compare datetime.datetime to int. Judging by this message, I'm thinking the issue is the concatenation with the AUTHORIZED field. Dummy data with what the data frame looks like is below. Any thoughts on how to get around this?

print firstData
  ORDER_ID    EMAIL             IP       AUTHORIZED  
0   1234567   [email protected]   x.x.x.x  2008-11-15 19:16:07   
1   8911234   [email protected]   x.x.x.x  2008-11-15 17:59:55   
2   4567833   [email protected]   x.x.x.x  2008-11-15 17:50:20   

     HOUR  DAY_OF_WEEK SHIPMENT_TYPE  ZIPS_MATCH     MERCH_SUBCAT     
0    19    7           Standard       1              Her Accessory      
1    17    7           Standard       1              Her Accessory      
2    17    7           Standard       1              Kid toy/clothes

print lastData   
  ORDER_ID  EMAIL              IP     AUTHORIZED 
3   1234567 [email protected]  x.x.x.x  2008-11-15 17:43:13 
4   8901234 [email protected]  x.x.x.x  2008-11-15 13:18:01    
5   4567890 [email protected]  x.x.x.x  2008-11-15 09:29:10   

   HOUR DAY_OF_WEEK  SHIPMENT_TYPE      ZIPS_MATCH    MERCH_SUBCAT
3  17    7           Standard           1             Bag   
4  13    7           Standard           1             Paperweight/boxes     
5  09    7           Standard           0             Candle   


concat(firstData, lastData)
   TypeError: can't compare datetime.datetime to int

Phillip Cloud · Accepted Answer · 2013-10-07 16:16:22Z

You need to pass a list, dict, tuple, or generator of DataFrame or Series objects to concat. What you're doing now will throw a TypeError because lastData is being passed as the axis argument (and there are numerous lines like axis == 0, which will trigger a TypeError). Do it like this:

In [31]: a
Out[31]:
   0        1                2        3           4         5
0  0  1234567  [email protected]  x.x.x.x  2008-11-15  19:16:07
1  1  8911234  [email protected]  x.x.x.x  2008-11-15  17:59:55
2  2  4567833  [email protected]  x.x.x.x  2008-11-15  17:50:20

In [32]: b
Out[32]:
   0        1                2        3           4         5
0  3  1234567  [email protected]  x.x.x.x  2008-11-15  17:43:13
1  4  8901234  [email protected]  x.x.x.x  2008-11-15  13:18:01
2  5  4567890  [email protected]  x.x.x.x  2008-11-15  09:29:10

In [33]: concat([a, b])
Out[33]:
   0        1                2        3           4         5
0  0  1234567  [email protected]  x.x.x.x  2008-11-15  19:16:07
1  1  8911234  [email protected]  x.x.x.x  2008-11-15  17:59:55
2  2  4567833  [email protected]  x.x.x.x  2008-11-15  17:50:20
0  3  1234567  [email protected]  x.x.x.x  2008-11-15  17:43:13
1  4  8901234  [email protected]  x.x.x.x  2008-11-15  13:18:01
2  5  4567890  [email protected]  x.x.x.x  2008-11-15  09:29:10

In [34]: concat([a, b], axis=1)
Out[34]:
   0        1                2        3           4         5  0        1  \
0  0  1234567  [email protected]  x.x.x.x  2008-11-15  19:16:07  3  1234567
1  1  8911234  [email protected]  x.x.x.x  2008-11-15  17:59:55  4  8901234
2  2  4567833  [email protected]  x.x.x.x  2008-11-15  17:50:20  5  4567890

                 2        3           4         5
0  [email protected]  x.x.x.x  2008-11-15  17:43:13
1  [email protected]  x.x.x.x  2008-11-15  13:18:01
2  [email protected]  x.x.x.x  2008-11-15  09:29:10

This now raises an AssertionError, since commit 36142334 (which is in the v0.12.0 release) so you won't get this cryptic error anymore.

Collectives™ on Stack Overflow

Concatenating pandas DataFrames with datetime columns

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related