Python watchdog for another Python process technique?

Question

I have a realtime data grabber that runs indefinitely, grabbing data over HTTP and putting it in a MySQL database every few seconds.

In my program, I have a while True loop that spawns workers (functions that download the data and save it) whenever the last spawned time is greater than X seconds:

while True:
    if _last_updated - datetime.now() > timedelta(seconds=5):
        green_pool.spawn_n(worker) # yes I'm using Eventlet!
        _last_updated = datetime.now()

What would be the best way to ensure that this module always does work, never freezes and is never down? Should I be checking the green pool size? I was thinking about writing a watchdog for it in Python, would you recommend doing so? If so, what things should I keep in mind?

Best

Rob Boyle · Accepted Answer · 2012-06-06 22:01:09Z

1

It might be overkill, but I would look at using supervisord. It's a process for controlling other processes (somewhat like init.d). It will allow you to start/stop/restart your control script containing the while True: loop. It will also auto-restart the control script if it stops working.

As you mention, you should keep tabs on the pool size and the success/failure of worker spawning within your control script. But to ensure that the control script is always running, supervisord fits the bill.

answered Jun 6, 2012 at 22:01

Rob Boyle

4303 silver badges9 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

user1094786 Over a year ago

thank you for your comment. this makes me think harder about making sure that not only is the process running, but that everything makes sense within the process. what would be a good approach to checking pool size / managing high pool sizes?

Rob Boyle Over a year ago

Glad it was helpful. As for maintaining your worker pool, that's going to be very specific to your implementation. I'd try it out with varying #'s of workers and see what performs the best.

Collectives™ on Stack Overflow

Python watchdog for another Python process technique?

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related