Remove duplicates from CSV file using PHP

Question

First of all I load PHPExcel.php

Secondly, I am using this code:

    $location = '/path/file.csv';



    $inputFileType = 'CSV';
    $objReader = PHPExcel_IOFactory::createReader($inputFileType);
    $objPHPExcel = $objReader->load($location);

                $worksheet = $objPHPExcel->getActiveSheet();
                $list = array();
                foreach ($worksheet->getRowIterator() as $row) 
                {
                    $rowIndex = $row->getRowIndex();
                    $cellValue = $worksheet->getCell('A'.$rowIndex)->getValue();
                    array_push($list, $cellValue);       
                }
                $count = count($list);
                for ($rowIndex = $count; $rowIndex != 1; $rowIndex--) 
                {
                    $cellValue = $worksheet->getCell('A'.$rowIndex)->getValue();
                    for ($i = $rowIndex - 2; $i != 0; $i--) 
{
                        if ($list[$i] == $cellValue) 
                        {
                            $worksheet->removeRow($rowIndex);
                            $objWriter = PHPExcel_IOFactory::createWriter($objPHPExcel, 'CSV');
                            $objWriter->save($location);
                            break;  
                        }
                    }
                }

So, I am trying to remove the rows when there are duplicate values in the first column. The code does not work. When I first run it in putty, I have to wait for ages. I interrupt the process and then I run it again. Then it runs, but in my csv file I have wrong results (duplicates are 300 but I am getting -600 rows).

it is seems there is a problem in logic, not the technical issue related to phpexcel lib or csv read write process, can you pass csv? if dont mind. i can play with the code using csv. — Amit Shah
– Amit Shah, Commented Dec 23, 2015 at 11:30
My 2 cents: If you have a database that you already use, why not import and do further processing there? A data-base is much better suited for data-processing than mangling them from CSV to CSV... — Piskvor left the building
– Piskvor left the building, Commented Dec 23, 2015 at 13:59

ThatMSG · Accepted Answer · 2015-12-23 13:56:08Z

2

In order to read a CSV file you dont have to use PHPExcel. Instead you can use a native php code like this one:

<?php
// Array which will hold all analyzed lines
$uniqueEntries = array();
$dublicatedEntries = array();
$delimiter = ',';
$file = 'test.csv';

//Open the file
if (($handle = fopen($file, "r")) !== false) {
    // read each line into an array
    while (($data = fgetcsv($handle, 8192, $delimiter)) !== false) {
        // build a "line" from the parsed data
        $line = join($delimiter, $data);

        //If the line content has ben discovered before - save to duplicated and skip the rest..
        if (isset($uniqueEntries[$line])){
            dublicatedEntries[] = $line;
            continue;
        }

        // save the line
        $uniqueEntries[$line] = true;
    }
    fclose($handle);
}

// build the new content-data
$contents = '';
foreach ($uniqueEntries as $line => $bool) $contents .= $line . "\r\n";

// save it to a new file
file_put_contents("test_unique.csv", $contents);
?>

This code is untested but should work. This will give you a .csv file with all unique entries.

answered Dec 23, 2015 at 13:56

ThatMSG

1,5061 gold badge16 silver badges28 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Datacrawler Over a year ago

My code works finally. Probably there was a problem with PUTTY. I tested it many times and in different computers and it works fine. It is not that fast but it is normal because I have 993 rows in my first file. Your solution is really interesting.

Cristal Over a year ago

It does not work for it. It copies the same content and move to the new file.

Collectives™ on Stack Overflow

Remove duplicates from CSV file using PHP

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related