Remove duplicate data in a text file with php

Question

I am in need to open a text file (file.txt) which contains data in the following format

ai
bt
bt
gh
ai
gh
lo
ki
ki
lo

ultimately I want to remove all the duplicate lines so only one of each data remains. So the result would look like this

ai
bt
gh
lo
ki

any help with this would be awesome

$unique = array_unique(explode("\n", file_get_contents('file.txt'))); — nkamm
– nkamm, Commented Nov 14, 2012 at 18:01
@nkamm please do not use comments to post solutions. Please delete your above comment. — mickmackusa
– mickmackusa ♦, Commented Jan 8, 2018 at 12:40

wassup · Accepted Answer · 2012-11-14 18:16:55Z

26

This should do the trick:

$lines = file('file.txt');
$lines = array_unique($lines);

file() reads the file and puts every line in an array.

array_unique() removes duplicate elements from the array.

Also, to put everything back into the file:

file_put_contents('file.txt', implode($lines));

edited Nov 14, 2012 at 18:16

answered Nov 14, 2012 at 18:07

wassup

2,5412 gold badges23 silver badges31 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Cesar Bielich Over a year ago

that did the trick, but it left a blank line between each one. Is there a way to remove the blank lines?

wassup Over a year ago

Oh, sorry. I forgot that file() doesn't remove newline characters. Answer updated. :)

Arush Kamboj Over a year ago

I have a 100MB text file and long rows for which this code is not working. Any alternatives?

Cesar Bielich Over a year ago

@ArushKamboj Maybe checkout stackoverflow.com/questions/4822165/…

David Müller · Accepted Answer · 2012-11-14 18:00:17Z

2

Take the php function file() to read the file. You get an array of lines from your file. After that, take array_unique to kick out the duplicates.

In the end, you will have something like

$lines = array_unique(file("your_file.txt"));

answered Nov 14, 2012 at 18:00

David Müller

5,3612 gold badges32 silver badges34 bronze badges

Comments

Paul Lo · Accepted Answer · 2015-03-21 03:51:45Z

0

This might work:

$txt = implode('\n',array_unique(explode('\n', $txt)));

edited Mar 21, 2015 at 3:51

Paul Lo

6,1466 gold badges33 silver badges37 bronze badges

answered Mar 20, 2015 at 16:38

taoufiqaitali

4905 silver badges11 bronze badges

3 Comments

mickmackusa Over a year ago

Please add some explanation to this code-only answer.

SuN Over a year ago

@mickmackusa: $txt contains the data and explode() puts every line in an array (using \n delimiter) so we can de-duplicate with array_unique(). Everything is then merged back together using implode() and the same delimiter.

mickmackusa Over a year ago

Try this instead: edit

Sam Martini · Accepted Answer · 2018-01-08 09:07:34Z

0

$lines = file_get_contents('file.txt');
$lines = explode('\n', $lines);
$lines = array_unique($lines);
$lines = implode('\n', $lines);
file_put_contents('file.txt', $lines);

answered Jan 8, 2018 at 9:07

Sam Martini

1

1 Comment

mickmackusa Over a year ago

Your answer is in the Review Queue for being low quality and may be deleted. Please add some explanation with your code-only answer so that future readers can fully benefit. Never post code-only answers on SO.

Collectives™ on Stack Overflow

Remove duplicate data in a text file with php

4 Answers 4

4 Comments

Comments

3 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

4 Comments

Comments

3 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related