0

I have a monte carlo simulation (essentially a simulation of game involving random numbers that I run millions of times) that I've coded in Node.js. The simulation runs a batch of 100,000 games and then inserts the data from that batch into a local MySQL database. After all batches have run a number of queries are run against the database.

I recently decided to try and rewrite the simulator for multithreading. I now have a main thread that assigns batches one at a time to 7 worker threads. They send their results back to the main thread and this thread inserts them into the database. This is working fine, but I've noticed that it doesn't seem to cut down the total time taken very much at all. I'm assuming this is because the I/O operation takes so much longer than the simulation itself.

I'm wondering if I should have each thread insert it's own results into the database, but I'm not sure if this will speed things up since presumably the MySQL database can only handle one query at a time anyway (and the database server is on the same machine)?

Would this speed things up or not? And is there anything I can do to speed up the INSERTs, perhaps on the server side?

Improve this question
4
  • mysql does support concurrent queries. To answer your question: test it and you will see. Commented Jun 29, 2023 at 14:16
  • 1
    Also look into batch insertion if you're adding lots of records at the same time / same thread. This can be coupled with the concurrent workers to basically dump the data into your DB. If you can avoid SQL's need for sequential operations (e.g. an auto-increment primary key), you can potentially get even better results. Commented Jun 29, 2023 at 14:35
  • @Shadow Is that strictly true? There's no concurrent handling of read vs. read/write queries in different transactions? Commented Jul 3, 2023 at 0:35
  • @Brad I wrote that mysql does support concurrent queries. Commented Jul 3, 2023 at 8:43
Related questions
2191
How to decide when to use Node.js?
2249
How can I update Node.js and npm to their latest versions?
1458
How can I output MySQL query results in CSV format?
Load 7 more related questions Show fewer related questions

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.