JavaScript running in a web browser cannot directly make the raw network connections required to connect to MySQL.
so I am not sure if it is a really good idea.
It's not a bad idea! Applications interact with databases all the time, so it's logical to want to do so.
You can integrate a React-based JavaScript application with any number of backend database layers, but you'll need to employ an additional web service to do so.
You can use JavaScript in the browser (your React application) to make and receive HTTP requests via AJAX to this additional service.
Typical implementations might use Node.js, but you could feasibly use any language/framework of your choice.
This is a solved problem, too. Make use of existing, powerful libraries such as Axios or jQuery.ajax and refer to the React docs as needed.
Potential duplicate of this question, this one or this one.
The basic answer is: you'll need an API of some sort.
Here are some other, non-StackOverflow resources on the subject that may help you take the next step.
https://www.quora.com/How-can-I-connect-the-create-react-app-with-a-Node-SQL-backend
https://dev.to/jerodimusprime/building-dynamic-react-apps-with-database-data-3apo
https://dev.to/kmaryam27/step-by-step-react-nodejs-and-mysql-simple-full-stack-application-2018-part-4-2bhg
https://github.com/stmoreau/React-Express-MySQL
https://medium.com/@carloscuba014/building-a-react-app-that-connects-to-mysql-via-nodejs-using-docker-a8acbb0e9788