Delete specific row and corresponding file in python

Question

I want to delete 90% of rows which have "steering" value qual to 0. And there is a corresponding image file for all three f them, center, left and right. I want to delete them too. The csv file is like this:

I have written the following code to at least get the files which have steering value 0. All I need is the code to randomly get 90% of those files and delete them.

with open('data/driving_log.csv') as csvfile:
    reader = csv.reader(csvfile)
    for i,line in enumerate(reader):
        lines.append(line)
        index.append(i)

lines = np.delete(lines,(0), axis = 0)
for i, line in enumerate(lines):
    #print(type(line[3].astype(np.float)))
    line_no.append(line[3].astype(np.float32))
    #print(line_no[i])
    if line_no[i]==0.0:
          # this gets the first column of the row.
        for j in range(3):
            source_path = line[j]
            filename = source_path.split('/')[-1]
            print(filename)
        count += 1

Have you searched for how to generate random numbers & delete files in python? Generate random numbers. Deleting files. — Andrew Lamarra
– Andrew Lamarra, Commented Apr 24, 2017 at 16:47
Yes, I need to import random and use os.remove() to remove the file. But, I'm getting confused in two places, removing a row form the csv file and randomly removing 90% of the files which have steering value equal to 0. — Sarvagya Gupta
– Sarvagya Gupta, Commented Apr 24, 2017 at 17:01
Could you post a portion of your CSV file in text format? Preferably with commas as the delimiter. — Andrew Lamarra
– Andrew Lamarra, Commented Apr 24, 2017 at 18:22
IMG/center_2016_12_01_13_30_48_287.jpg, IMG/left_2016_12_01_13_30_48_287.jpg, IMG/right_2016_12_01_13_30_48_287.jpg, 0, 0, 0, 22.14829. — Sarvagya Gupta
– Sarvagya Gupta, Commented Apr 24, 2017 at 19:56

Andrew Lamarra · Accepted Answer · 2017-04-24 19:56:54Z

3

I think this will do what you want:

import csv
from random import randint
from os import remove

# Create a 2D list from which we can work with
lines = []
with open('data/driving_log.csv', newline='') as csvfile:
    reader = csv.reader(csvfile)
    for line in reader:
        lines.append(line)

# Find 10% of total lines (to keep), not including header row
numToKeep = round(sum(1 for i in lines if i[3] == '0') * 0.1)

# Save 10% of lines to a new 2D list
toKeep = []
for i in range(numToKeep):
    while True:
        index = randint(1, len(lines)-1)
        # Make sure we haven't already selected the same line
        if lines[index] not in toKeep and lines[index][3] == '0':
            toKeep.append(lines[index])
            break

# Deleting all files of the selected 90% of rows
for i, line in enumerate(lines):
    if i == 0:  # Omit the header row
        continue
    if lines[i][3] != '0':  # Keep rows that don't have a steering value of 0
        toKeep.append(lines[i])
    if line not in toKeep:
        print("Deleting: {}".format(line))
        for i in range(3):
            remove(line[i])

with open('data/driving_log.csv', 'w', newline='') as csvfile:
    writer = csv.writer(csvfile)
    writer.writerows([lines[0]])  # Put the header back in
    writer.writerows(toKeep)

I realize this isn't the most elegant solution. I'm not familiar with numpy and don't have time to learn it right now, but this should work.

edited Apr 24, 2017 at 19:56

answered Apr 24, 2017 at 19:32

Andrew Lamarra

5374 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Sarvagya Gupta Over a year ago

Hey, thanks for that. But I don't see the condition where the steering angle needs to be 0. Plus, I also want to delete the corresponding files. like I want to delete center_2016_12_01_13_30_48_287.jpg in the IMG folder. Can you help me with these two things?

Andrew Lamarra Over a year ago

This line if lines[index] not in toKeep and lines[index][3] == '0': states that if we haven't already added the line to our list of lines to keep AND if the steering column is equal to 0, then...

Andrew Lamarra Over a year ago

These lines: for i in range(3): AND remove(line[i]) will delete each of the 3 files for that particular row.

Collectives™ on Stack Overflow

Delete specific row and corresponding file in python

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related