python unicode replace MemoryError

Question

i want replace unicode character to a file with python

this is my code :

with codecs.open('/etc/bluetooth/main.conf', "r", "utf8") as fi:
            mainconf=fi.read()
        forrep = ''.decode('utf8')
        for line in mainconf.splitlines():
            if('Name = ' in line):
                forrep = line.split('=')[1]
        print 'name',type(name)
        print 'mainconf',type(mainconf)
        print 'forrep',type(forrep)
        mainconf = mainconf.replace(forrep, name)
        #mainconf = mainconf.replace(forrep.decode('utf8'),' '+name)
        with codecs.open('/etc/bluetooth/main.conf','w',"utf8") as fi:
            fi.write(mainconf)

but python always get me error MemoryError...

this out :

name <type 'unicode'>
mainconf <type 'unicode'>
forrep <type 'unicode'>
Traceback (most recent call last):
  File "WORK/Bluetooth/Bluetooth.py", line 359, in <module>
    if __name__ == '__main__':main()
  File "WORK/Bluetooth/Bluetooth.py", line 336, in main
    BLMan.SetAllHCIName(common.cfg.get('BLUETOOTH', 'HCI_DEVICE_NAME'))
  File "WORK/Bluetooth/Bluetooth.py", line 194, in SetAllHCIName
    mainconf = mainconf.replace(forrep, name)
MemoryError

open file read and write mode at first line and write to this. — VelikiiNehochuha
– VelikiiNehochuha, Commented Apr 17, 2015 at 16:35
As a side note: What is the forrep = ''.decode('utf8') for? All that's going to do is create the empty unicode string u'', which (a) you could just write as a literal, and (b) you don't use anywhere. I'm a bit worried that you're expecting that decode to somehow be remembered and automatically re-applied any time you store anything to the name forrep later or something, which it won't be. — abarnert
– abarnert, Commented Apr 17, 2015 at 16:44
Also, I suspect you want if line.startswith('Name = '): or if line.lstrip().startswth('Name ='), so that if the substring 'Name =' happens to appear in a value somewhere, it won't get matched, only if the key is Name. — abarnert
– abarnert, Commented Apr 17, 2015 at 16:45
@abarnert main.conf is small file , about 30 lines ... i don't understand ... please correct my code and write here ... tnx — abbas-h
– abbas-h, Commented Apr 17, 2015 at 17:12

Padraic Cunningham · Accepted Answer · 2015-04-17 19:12:16Z

2

Iterate over the file object, you are storing the whole file content in memory using mainconf=fi.read() :

with codecs.open('/etc/bluetooth/main.conf', "r", "utf8") as fi:
    for line in fi:

You store all the lines with read then you store a list of all the lines using splitlines so you are storing all the file content twice and as @abarnet pointed out in a comment you then try to store a third copy with mainconf = mainconf.replace(forrep, name).

Iterating over the file object will give you a line at a time, if you need to store the lines after replacing do so each time through the loop so at most you will only have one copy of the file content in memory.

I have no idea what name is but writing to a tempfile will be the most efficient way to do what you want:

from tempfile import NamedTemporaryFile

with open('/etc/bluetooth/main.conf') as fi, NamedTemporaryFile(dir=".", delete=False) as out:
    for line in fi:
        if line.startswith("Name ="):
            a, b = line.split("=",1)
            out.write("{} = {}".format(a, name.encode("utf-8")))
        else:
            out.write(line)

move(out.name, '/etc/bluetooth/main.conf')

edited Apr 17, 2015 at 19:12

answered Apr 17, 2015 at 16:31

Padraic Cunningham

181k30 gold badges264 silver badges327 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

abarnert Over a year ago

This is the right idea, but notice that he wants to overwrite the file at the end. So, he needs a bit more. Probably opening a temporary file, writing line by line as he reads line by line, and then moving the temporary file over the original.

Padraic Cunningham Over a year ago

@abarnert, True a NamedTempFile should work or maybe use fileinput.input. The OP is actually storing the contents twice which does not help either

abarnert Over a year ago

Yea, NamedTempfile is perfect, except it's a bit of a pain to wrap it in the codecs module. (If this is Python 2.6+, he could use io instead of codecs, though…)

Padraic Cunningham Over a year ago

@abarnert, I am not sure that codecs is even necessary. I will wait until the OP replies to some comments before deciding what the best approach is.

abarnert Over a year ago

Actually, he's not just creating two copies of the content, he's creating another copy by doing mainconf = mainconf.replace(forrep, name) instead of just doing the replace on the line that needs it.

|

Collectives™ on Stack Overflow

python unicode replace MemoryError

1 Answer 1

7 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Your Answer

Sign up or log in

Post as a guest

Related