0

I have 2 Python scripts on a Raspberry Pi, 1 runs in a background shell emptying a queue, the other runs in the foreground adding user inputs to the queue. I've build a version where the queue is stored as a SQLite database and in order to make this work each script has to connect to the database before each operation and disconnect afterwards which avoids locking conflict but slows the process down significantly. This overhead means I am actually able to hammer the inputs fast enough to confuse the script and make it miss/ignore some inputs. Is a single SQLite datbase the fastest method for storing my shared queue or is there some faster alternative (possibly one that uses RAM instead of writing to disk) that both scripts can access quicker?

(Note: I'm open to drastic suggestions like switching language to NodeJS)

I didn't post the code because it's more a question of appropriate technology for the job but if you want to see my current repo it's at https://github.com/martinjoiner/bookfetch-scanner-python

Improve this question
5
  • Why do you need to close and reopen the collection? SQLite don't support concurrent solutions? A less radical solution may be to change for a DB that does. MongoDB for example. Commented Oct 19, 2016 at 14:58
  • 1
    There are a lot of messaging solutions that would seem to be better for IPC messaging (e.g., ZeroMQ, or a messaging server like redis or rabbitmq, etc) Commented Oct 19, 2016 at 15:00
  • why use an SQLite database in the first place? Are you allowing queries or something? Commented Oct 19, 2016 at 15:03
  • @Ev.Kounis Traditionally I am a LAMP stack web developer. When I am building 2 things that share data the first thing I think of is a database. Why? What non-database technologies exist in the Linux environment that I could leverage? Commented Oct 19, 2016 at 15:09
  • 2
    Point taken. How about not checking them instantly in one-by-one, but collecting them (appending the codes to a list in RAM or smthg) and doing it collectively when the "hammering" stops. Like if 30 sec passed from the last hammering: INSERT... in pseudocode? Commented Oct 19, 2016 at 15:19
Related questions
4
python: how to share an sqlite connection among threads with queues?
11
SQLite3 and Multiprocessing
1
how to run 2 different processes at the same time?
Load 7 more related questions Show fewer related questions

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.