I have a project on AWS which will eventually have over 100 client databases each estimated in the 1~2GB size. I need to do daily backups of each database via mysqldump not the standard AWS RDS daily snapshots.
In bash I loop through a list and dump each db one after the other which is likely to take a few hours. My question is
Is it adviseable to do a bash fork and dump the DBs in parallel?
Assuming you're having a file client_db.list that contains a list of databases name, all databases belong to one RDS instance live-db.example.internal
> cat client_db.list
client01_dbname
client02_dbname
client03_dbname
...
...
client_template
The solution is using GNU parallel apt install parallel for executing mysqldump each database in parallel
parallel -j2 -a client_db.list mysqldump -uroot -pXXXXXXXXX -hlive-db.example.internal \
--single-transaction \
--skip-lock-tables {} '>' ./backup/{.}.sql
-j2 means executing mysqldump for 2 databases at the same time, based on number of vCPU of RDS instance
The output will be:
ls -lh
total 3.4G
-rw-r--r-- 1 jenkins jenkins 26M Jun 4 10:56 client01_dbname.sql
-rw-r--r-- 1 jenkins jenkins 26M Jun 4 10:54 client02_dbname.sql
-rw-r--r-- 1 jenkins jenkins 26M Jun 4 10:54 client03_dbname.sql
...
...
-rw-r--r-- 1 jenkins jenkins 26M Jun 4 10:54 client_template.sql
