Python Write dynamically huge files avoiding 100% CPU Usage

Question

I am parsing a huge CSV approx 2 GB files with the help of this great stuff. Now have to generate dynamic files for each column in a new file where column name as file name. So I written this code to write the dynamic files:

def write_CSV_dynamically(self, header, reader):
  """
  :header - CSVs first row in string format
  :reader - CSVs all other rows in list format  
  """

  try:
    headerlist =header.split(',') #-- string headers 
    zipof = lambda x, y: zip(x.split(','), y.split(','))
    filename = "{}.csv".format(self.dtstamp)
    filename = "{}_"+filename
    filesdct = {filename.format(k.strip()):open(filename.format(k.strip()), 'a')\
    for k in headerlist}
    for row in reader:
      for key, data in zipof(header, row):
        filesdct[filename.format(key.strip())].write( str(data) +"\n" )
    for _, v in filesdct.iteritems():
      v.close()
  except Exception, e:
    print e

Now its taking around 50 secs to write these huge files using 100% CPU.As there are other heavy things running on my server. I want to block my program to use only 10% to 20% of the CPU and write these files. No matter if it takes 10-15 mins. How can I optimize my code, so that it should limit 10-20% CPU usage.

@AndreyTaptunov - Cent os for production and Ubuntu for development. We can ignore development server though! — Laxmikant
– Laxmikant, Commented Feb 22, 2016 at 7:20

Andrey Taptunov · Accepted Answer · 2016-02-22 08:48:06Z

1

There is number of ways to achieve this:

Nice the process - plain and simple.
cpulimit - just pass your script and cpu usage as parameters:

cpulimit -P /path/to/your/script -l 20
Python's resource package to set limits from the script. Bear in mind it works with absolute CPU time.

edited Feb 22, 2016 at 8:48

answered Feb 22, 2016 at 7:28

Andrey Taptunov

9,5655 gold badges35 silver badges44 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Laxmikant Over a year ago

Seems interesting, Let me go through it.

Collectives™ on Stack Overflow

Python Write dynamically huge files avoiding 100% CPU Usage

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related