Python delete line/lines from file without modifying existing contents

Question

I have to delete string or list of strings based on user input from a file. I referred the below link and things are working fine.

Deleting a specific line in a file (python)

However, the above approach reads the existing file contents in memory and if the line to delete is not found writes it back in the same file. This approach is not suitable if we dealing with files with huge amount of confidential data.

All i wish to know is, Is there a better way to do the same thing.

  valid_List=["10.1.2.3","10.2.3.4","10.2.4.5","10.2.3.7"]
  filename="abc.txt"
  for i in valid_List:
    f = open(filename,"r")
    lines = f.readlines()
    f.close()
    f = open(filename,"w")
    for line in lines:
      if line!=i+" "+ "ok"+"\n":
        #print("Writing ip not to be deleted")
        f.write(line)
      else:
        print(i," Deleted")
        user_response.append(i+" Deleted")
        logger.info('Response returned to user%s',user_response)
    f.close()

I don't think most file system support that.

YOU
– YOU

2016-05-19 08:00:18 +00:00
Commented May 19, 2016 at 8:00 — YOU
– YOU, Commented May 19, 2016 at 8:00
Why does it matter that the data is confidential?

flodin
– flodin

2016-05-19 08:00:25 +00:00
Commented May 19, 2016 at 8:00 — flodin
– flodin, Commented May 19, 2016 at 8:00

Community · Accepted Answer · 2017-05-23 10:28:35Z

4

You can read and write to two different files and do the operation elementwise.

Afterwards you replace the inputfile with the outputfile

import shutil

valid_List = ["10.1.2.3", "10.2.3.4", "10.2.4.5", "10.2.3.7"]
filename = "abc.txt"
outfile = "outfile.txt"

with open(filename, "r") as f:
    with open(outfile, "w") as o:
        for line in f:
            if all([line != "%s ok\n" % i for i in valid_List]):
                o.write(line)
            else:
                print("%s Deleted" % line.strip())

shutil.move(outfile, filename)

Caveat This uses the a fixed filename for output, which might cause collisions when you run the program multiple times in parallel. If you use this atomic save recipe you can simplify the code to

valid_List = ["10.1.2.3", "10.2.3.4", "10.2.4.5", "10.2.3.7"]
filename = "abc.txt"

with atomic_open(filename, "w") as o:
    with open(filename, "r") as f:
        for line in f:
            if all([line != "%s ok\n" % i for i in valid_List]):
                o.write(line)
            else:
                print("%s Deleted" % line.strip())

This will automatically choose a temporary file (collision-free) for you and replace the input file with the output file on completion.

Also you will notice that I have replaced your outer loop (opening files once for each entry in valid_list) with an all() statement. This saves you a lot of overhead, too.

edited May 23, 2017 at 10:28

CommunityBot

11 silver badge

answered May 19, 2016 at 8:06

Nils Werner

37.2k7 gold badges85 silver badges108 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Iram Khan Over a year ago

i have imported shutil, however it throws an error 'atomic_open' not defined. Also i want to read and write in same file

Nils Werner Over a year ago

Have you had a look at the recipe I linked to in my answer? You need that too

Abdul Fatir · Accepted Answer · 2016-05-19 10:35:55Z

1

You're opening and closing the huge file multiple times, once for each element in valid_List. You should instead open the file just once and check if any line of file matches with your valid_List.

Try like this (the code is not tested but it should work):

valid_List=["10.1.2.3","10.2.3.4","10.2.4.5","10.2.3.7"]
filename="abc.txt"

f = open(filename,"r")
lines = f.readlines()
f.close()

f = open(filename,"w")
for line in lines:
    flag = True
    deleted = ''
    for i in valid_List:
        if line == i+" "+ "ok"+"\n":
            flag = False
            deleted = i
            break
    if flag:
        #print("Writing ip not to be deleted")
        f.write(line)
    else:
        print(deleted," Deleted")
f.close()

EDIT
Added check for not-found IPs.

valid_List=["10.1.2.3","10.2.3.4","10.2.4.5","10.2.3.7"]
filename="abc.txt"

if_found = [False for v in valid_List]

f = open(filename,"r")
lines = f.readlines()
f.close()

f = open(filename,"w")
for line in lines:
    flag = True
    deleted = ''
    for _,i in enumerate(valid_List):
        if line == i+" "+ "ok"+"\n":
            flag = False
            if_found[_] = True
            deleted = i
            break
    if flag:
        #print("Writing ip not to be deleted")
        f.write(line)
    else:
        print(deleted," Deleted")
f.close()

for _,i in enumerate(if_found):
    if not i:
        print(valid_List[_]," Not Found")

edited May 19, 2016 at 10:35

answered May 19, 2016 at 8:06

Abdul Fatir

6,4075 gold badges33 silver badges61 bronze badges

9 Comments

Iram Khan Over a year ago

sorry buddyy...mistakes from my side.. thanks a lot ih helped me.. However i have a doubt..can u tell me how i can show if the ip entered to delete is "not there in file" or "is invalid" ? tried ways bt its not happening.

Iram Khan Over a year ago

i have upvoted you, but i do not have enough reputation score. Once i get it, you see the change :)

Abdul Fatir Over a year ago

If this worked, you can mark this as the answer by clicking on the tick mark. For checking if the IP is in file just keep a variable associated with each IP entered and change its value once the IP is found. Finally check if the variable did not change state, which will mean the IP is not in the file.

Abdul Fatir Over a year ago

For checking valid IP, you can validate using regex. Please let me know if you don't know what regex is or you need code.

Iram Khan Over a year ago

i have a list of invalid ip where i have checked invalid ip using Ipy library... so now i am just looping my invalid loop after the loop for line in lines is ended just like this ---> for i in invalid_List: user_response.append(i+" Invalid") logger.info('Response returned to user%s',user_response)...

|

Hani Yassine · Accepted Answer · 2018-06-22 15:23:32Z

1

i created this script basically you put bunch of lines strings in a list if any of them was found it get deleted and it works on batch so it open multiple files you input the number of files obviously it only for personal use not users because it doesnt have input check and the files need to be in the same dir as the script:

n=int(input('enter the number of files:'))
for i in range (1,n):
    f = open(f"{i}.txt","r")
    lines = f.readlines()
    f.close()
    f = open(f"{i}.txt","w")
    strings_to_remove=['Edited at','test']
    for line in lines:
        if line.strip() not in strings_to_remove:
            f.write(line)
    f.close()

answered Jun 22, 2018 at 15:23

Hani Yassine

315 bronze badges

Collectives™ on Stack Overflow

Python delete line/lines from file without modifying existing contents

3 Answers 3

2 Comments

9 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

9 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related