Python: Memory allocation error

Question

I am reading rows from a sqlite db, looping over them and then using the variables from each row as arguments for a function which generates plots, something like the following pseudo code

conn=sqlite3.connect(db)
c=conn.cursor()
myrows=c.execute("select * from values WHERE var=1")
for burst in myrows:
    met=burst[1]
    make_plot(met)
c.close()

After a certain amount of time this gives either one of the following errors:

1) 86392 items requested but only 0 read Segmentation fault

2) unable to alloc 3072000 bytes Aborted

3) Segmentation fault

The make_plot() has to read in a large file (~8 mB) and do some operations on the data, and there are ~500 elements in myrows. From the errors I think that I am running out of memory(?) but I haven't managed to find anything that would help me track down/diagnose this problem. Any ideas on how I would do this?

yosukesabai · Accepted Answer · 2011-11-03 14:37:07Z

1

can't you say something like below, if all you care is this second field?

myrows=c.execute("select values.bust from values WHERE var=1")

also, i am wondering you really want to create plot for each line... dont you want to plot all the data in one plot?

answered Nov 3, 2011 at 14:37

yosukesabai

6,2644 gold badges35 silver badges42 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Ger Over a year ago

Hi, Each entry in the database that I am selecting has its own data & its own plot that has to be made. The code I posted is a simplified version which just contains the steps (i.e. I have removed most of the variables). Cheers

yosukesabai Over a year ago

so this make_plot() opens a file of ~8MB for every call and process that combining with a record in myrows? Do you know error is coming from my_plot() or from this loop across records in your database? Is there any chance that there is some memory leak in there or illegal access of memory?

Ger Over a year ago

It turned out that the version of the pyfits module that I was using (2.4) had a memory leak, I upgraded to 3.0 and it improved the problem, i.e. it ran for longer before using all my ram. It turns out that I was also missing a plt.close(fig) at the end of my plotting function - popped that it and it all worked fine. Cheers

yosukesabai Over a year ago

@user1027686: it is always great to hear that things start workin!

Nicola Musatti · Accepted Answer · 2011-11-03 14:42:14Z

0

Try and see if by changing your for statement to:

for burst in c.execute("select * from values WHERE var=1"):
    # your code here

rows aren't retrieved one at a time rather than in a single batch.

answered Nov 3, 2011 at 14:42

Nicola Musatti

18.4k3 gold badges49 silver badges56 bronze badges

Collectives™ on Stack Overflow

Python: Memory allocation error

2 Answers 2

4 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related