Python Pandas replicate rows in dataframe

Question

If the dataframe looks like:

Store,Dept,Date,Weekly_Sales,IsHoliday
1,1,2010-02-05,24924.5,FALSE
1,1,2010-02-12,46039.49,TRUE
1,1,2010-02-19,41595.55,FALSE
1,1,2010-02-26,19403.54,FALSE
1,1,2010-03-05,21827.9,FALSE
1,1,2010-03-12,21043.39,FALSE
1,1,2010-03-19,22136.64,FALSE
1,1,2010-03-26,26229.21,FALSE
1,1,2010-04-02,57258.43,FALSE

And I wanna duplicate rows with IsHoliday equal to TRUE, I can do:

is_hol = df['IsHoliday'] == True
df_try = df[is_hol]
df=df.append(df_try*10)

But is there a better way to do this as I need to duplicate holiday rows 5 times, and I have to append 5 times if using the above way.

stackoverflow.com/questions/50788508/replicating-rows-in-pandas/… — U13-Forward
– U13-Forward, Commented Jan 1, 2021 at 9:55

Karl D. · Accepted Answer · 2014-06-04 07:08:56Z

125

You can put df_try inside a list and then do what you have in mind:

>>> df.append([df_try]*5,ignore_index=True)

    Store  Dept       Date  Weekly_Sales IsHoliday
0       1     1 2010-02-05      24924.50     False
1       1     1 2010-02-12      46039.49      True
2       1     1 2010-02-19      41595.55     False
3       1     1 2010-02-26      19403.54     False
4       1     1 2010-03-05      21827.90     False
5       1     1 2010-03-12      21043.39     False
6       1     1 2010-03-19      22136.64     False
7       1     1 2010-03-26      26229.21     False
8       1     1 2010-04-02      57258.43     False
9       1     1 2010-02-12      46039.49      True
10      1     1 2010-02-12      46039.49      True
11      1     1 2010-02-12      46039.49      True
12      1     1 2010-02-12      46039.49      True
13      1     1 2010-02-12      46039.49      True

edited Jun 4, 2014 at 7:08

answered Jun 4, 2014 at 5:47

Karl D.

13.8k5 gold badges59 silver badges41 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

penelope Oct 31 at 14:55

'DataFrame' object has no attribute 'append' pandas.pydata.org/pandas-docs/version/1.4/whatsnew/… Deprecated since 2022

Surya Chhetri · Accepted Answer · 2016-12-29 22:58:08Z

54

Other way is using concat() function:

import pandas as pd

In [603]: df = pd.DataFrame({'col1':list("abc"),'col2':range(3)},index = range(3))

In [604]: df
Out[604]: 
  col1  col2
0    a     0
1    b     1
2    c     2

In [605]: pd.concat([df]*3, ignore_index=True) # Ignores the index
Out[605]: 
  col1  col2
0    a     0
1    b     1
2    c     2
3    a     0
4    b     1
5    c     2
6    a     0
7    b     1
8    c     2

In [606]: pd.concat([df]*3)
Out[606]: 
  col1  col2
0    a     0
1    b     1
2    c     2
0    a     0
1    b     1
2    c     2
0    a     0
1    b     1
2    c     2

answered Dec 29, 2016 at 22:58

Surya Chhetri

11.7k4 gold badges61 silver badges39 bronze badges

3 Comments

gsamaras Over a year ago

This should be preferred over accepted answer, not that append() got deprecated.

piogor Over a year ago

How does this answer the question of replicating specific rows that match some expression?

data princess Over a year ago

This is much slower than @grofte's answer (for me)

snooze_bear · Accepted Answer · 2018-05-23 14:23:40Z

30

This is an old question, but since it still comes up at the top of my results in Google, here's another way.

import pandas as pd
import numpy as np

df = pd.DataFrame({'col1':list("abc"),'col2':range(3)},index = range(3))

Say you want to replicate the rows where col1="b".

reps = [3 if val=="b" else 1 for val in df.col1]
df.loc[np.repeat(df.index.values, reps)]

You could replace the 3 if val=="b" else 1 in the list interpretation with another function that could return 3 if val=="b" or 4 if val=="c" and so on, so it's pretty flexible.

answered May 23, 2018 at 14:23

snooze_bear

6097 silver badges15 bronze badges

4 Comments

RufusVS Over a year ago

This will also replicate the index values as well, correct?

grofte Over a year ago

I don't know if I'm doing it wrong but this is very slow for me.

kawingkelvin Over a year ago

This more elegant way is also pretty fast for me compared with using .append(..). My use case is just to replicate a dataframe with a single row 1000x of times.

Marco Over a year ago

Not scalable: try doing for val in df.col1 on 100 million line dataframe.. :(

grofte · Accepted Answer · 2019-07-25 11:59:53Z

10

Appending and concatenating is usually slow in Pandas so I recommend just making a new list of the rows and turning that into a dataframe (unless appending a single row or concatenating a few dataframes).

import pandas as pd

df = pd.DataFrame([
[1,1,'2010-02-05',24924.5,False],
[1,1,'2010-02-12',46039.49,True],
[1,1,'2010-02-19',41595.55,False],
[1,1,'2010-02-26',19403.54,False],
[1,1,'2010-03-05',21827.9,False],
[1,1,'2010-03-12',21043.39,False],
[1,1,'2010-03-19',22136.64,False],
[1,1,'2010-03-26',26229.21,False],
[1,1,'2010-04-02',57258.43,False]
], columns=['Store','Dept','Date','Weekly_Sales','IsHoliday'])

temp_df = []
for row in df.itertuples(index=False):
    if row.IsHoliday:
        temp_df.extend([list(row)]*5)
    else:
        temp_df.append(list(row))

df = pd.DataFrame(temp_df, columns=df.columns)

edited Jul 25, 2019 at 11:59

answered Jul 25, 2019 at 11:30

grofte

2,1991 gold badge17 silver badges18 bronze badges

2 Comments

grofte Over a year ago

For the record, it is probably faster to do an outer join but the code gets quite convoluted for a small gain.

data princess Over a year ago

This approach is significantly faster than the others, for me anyway

Mykola Zotko · Accepted Answer · 2021-06-23 06:34:35Z

6

You can do it in one line:

df.append([df[df['IsHoliday'] == True]] * 5, ignore_index=True)

or

df.append([df[df['IsHoliday']]] * 5, ignore_index=True)

answered Jun 23, 2021 at 6:34

Mykola Zotko

18.2k6 gold badges88 silver badges90 bronze badges

Comments

buddemat · Accepted Answer · 2021-10-03 14:24:20Z

5

Another alternative to append() is to first replace the values of a column by a list of entries and then explode() (either using ignore_index=True or not, depending on what you want):

df['IsHoliday'] = df['IsHoliday'].apply(lambda x: 5*[x] if (x == True) else x)

df.explode('IsHoliday', ignore_index=True)

The nice thing about this one is that you can already use the list in the apply() call to build copies of rows with modified values in a column, in case you wanted to do that later anyways...

edited Oct 3, 2021 at 14:24

answered Oct 1, 2021 at 12:22

buddemat

5,33016 gold badges37 silver badges61 bronze badges

1 Comment

Kauê de Moraes Vestena Over a year ago

This is the best of all answers!!

Văn Dũng Lê · Accepted Answer · 2024-04-17 11:53:14Z

Here is a another way of doing it using numpy.tile()

df = pd.DataFrame([
[1,1,'2010-02-05',24924.5,False],
[1,1,'2010-02-12',46039.49,True],
[1,1,'2010-02-19',41595.55,False],
[1,1,'2010-02-26',19403.54,False],
[1,1,'2010-03-05',21827.9,False],
[1,1,'2010-03-12',21043.39,False],
[1,1,'2010-03-19',22136.64,False],
[1,1,'2010-03-26',26229.21,False],
[1,1,'2010-04-02',57258.43,False]
], columns=['Store','Dept','Date','Weekly_Sales','IsHoliday'])
df= df.loc[df['IsHoliday']==True]


%%timeit
pd.concat([df]*5,axis=0)

801 µs ± 29.4 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

%%timeit
pd.DataFrame(np.tile(df.to_numpy(), (5,1)), columns = df.columns)

261 µs ± 16.3 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

Collectives™ on Stack Overflow

Python Pandas replicate rows in dataframe

7 Answers 7

1 Comment

3 Comments

4 Comments

2 Comments

Comments

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

7 Answers 7

1 Comment

3 Comments

4 Comments

2 Comments

Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related