0

I've just finished a basic PHP file, that lets indie game developers / application developers store user data, handle user logins, self-deleting variables etc. It all revolves around storage.

I've made systems like this before, but always hit the max_user_connections issue - which I personally can't currently change, as I use a friends hosting - and often free hosting providers limit the max_user_connections anyway. This time, I've made the system fully text file based (each of them holding JSON structures).

The system works fine currently, as it's being tested by only me and another 4/5 users per second. The PHP script basically opens a text file (based upon query arguments), uses json_decode to convert the contents into the relevant PHP structures, then alters and writes back to the file. Again, this works fine at the moment, as there are few users using the system - but I believe if two users attempted to alter a single file at the same time, the person who writes to it last will overwrite the data that the previous user wrote to it.

Using SQL databases always seemed to handle queries quite slowly - even basic queries. Should I try to implement some form of server-side caching system, or possibly file write stacking system? Or should I just attempt to bump up the max_user_connections, and make it fully SQL based?

Are there limits to the number of users that can READ text files per second?

I know game / application / web developers must create optimized PHP storage solutions all the time, but what are the best practices in dealing with traffic?

It seems most hosting companies set the max_user_connections to a fairly low number to begin with - is there any way to alter this within the PHP file?

Here's the current PHP file, if you wish to view it: https://www.dropbox.com/s/rr5ua4175w3rhw0/storage.php

And here's a forum topic showing the queries: http://gmc.yoyogames.com/index.php?showtopic=623357

I did plan to release the PHP file, so developers could host it on their own site, but I would like to make it work as well as possible, before doing this.

Many thanks for any help provided.

  • Dan.
Improve this question
5
  • let's see... TCP socket handle limits, system filehandle limits, memory limits, bandwidth limits, webserver connection limits, number of people on the planet limits, etc... Commented Jul 8, 2014 at 14:12
  • There exist solutions to all these problems. For limited connections, you should use connection pooling and/or reduce the number of required database accesses using caching. Using a "real" server instead of limited shared hosting is also a solution. For overwriting files, there exist a number of different locking strategies. There's is no "best", figure out the right set of tradeoffs for your situation. Commented Jul 8, 2014 at 14:14
  • What's the question in the end? SQL queries are slow - but you use file based approach which uses up I/O and you're not even using exclusive locks (so yes, stuff will get overwritten). Yes, there is always a limit to how many connections / open files there can be, it's controlled by the operating system (and can be altered). Commented Jul 8, 2014 at 14:14
  • Yes, sorry that became a bit jumbled. There are so many questions to ask, and I guess there are no set 'how-to's - What's the best way to deal with high levels of traffic, using a file based system? Commented Jul 8, 2014 at 14:20
  • If you compare PHP's file-system based session management vs. database session management you will see that file system based access is a lot slower on about 99% of all systems. Depending on the type of queries you perform (select vs. update/insert/delete) you might use a MySQL cluster (best when high read volume expected) or implement a write cache. If you are not restricted to MySQL, Apache Cassandra could be a good alternative for you. Commented Jul 8, 2014 at 14:43

2 Answers 2

Reset to default
1

I strongly suggest you not re-invent the wheel. There are many options available for persistent storage. If you don't want to use SQL consider trying out any of the popular "NoSQL" options like MongoDB, Redis, CouchDB, etc. Many smart people have spent many hours solving the problems you are mentioning already, and they are hard at work improving and supporting their software.

Scaling a MySQL database service is outside the scope of this answer, but if you want to throttle up what your database service can handle you need to move out of a shared hosting environment in any case.

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

Ah, yes - I'm bad for wanting to do everything myself, rather than using what's available. I will look into these non SQL based solutions you've mentioned. Thank you!
1

"but I believe if two users attempted to alter a single file at the same time, the person who writes to it last will overwrite the data that the previous user wrote to it." - that is for sure. It even throws an error if the 2nd tries to save while the first has it open.

"Are there limits to the number of users that can READ text files per second?" - no, but it is pointless to open a file, just for read multiple times. That file needs to be cached in a content management network.

"I know game / application / web developers must create optimized PHP storage solutions all the time, but what are the best practices in dealing with traffic?" - usually a new database will do a better job than files, starting from the fact that the most often selects are stored in the RAM, the most often .txt files are not. As @oliakaoil read about the DB difference and see what you need.

Improve this answer

6 Comments

I built in an error catching system in case there was an error writing to any of the files - it normally results in the PHP file returning '0'. Now that I know it will return an error, I will build in a retry X times loop.
If I set up multiple users on one SQL database (say 10 - 20, rather than the 1 user I currently use), then select a user at random, would it let more people access it, or does it count how many connections there are open to the server?
Usually, people use caching for the database and data manipulation on the client side. Your goal is to send back to the user all the required information so that they do as less queries as possible. On top of that, most of the online application run the processing in batches. Look at the facebook games, everything happens 1 or 2 seconds later. You can configure the number of connections a DB can have opened concurrently. You can also go for a data replication mode. This allows you to have multiple DBs to read from and one or 2 to write into.
Hmm - I think I will try implementing the retry system for the moment - Most of the queries read, and only really two write queries will become a problem when dealing with multiple users. System variables - which are essentially global variables, and timeout_ variables, which are basically arrays. Both need re-making, upon revision. I think using one file for each is a bit silly - I need instead to use one directory for each, and possibly sub-directories within the timeout variables (as they are basically arrays). Later, I will look at data replication, only if max users viewing a text file
... becomes an issue. I could even check which file had been written to last, when attempting to read duplicate variables, then simply delete the older one (if possible), and only read from the newer one.
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.