9

Background: I am a complete beginner when it comes to servers, but I know my way around programming in Python.

I am trying to setup a simple server using the basic Python 2.7 modules (SimpleHTTPServer, CGIHTTPServer, etc). This server needs to load a global, read-only variable with several GB of data from a file when it starts; then, when each user accesses the page, the server uses the big data to generate some output which is then given to the user.

For the sake of example, let's suppose I have a 4 GB file names.txt which contains all possible proper nouns of English:

Jack
John
Allison
Richard
...

Let's suppose that my goal is to read the whole list of names into memory, and then choose 1 name at random from this big list of proper nouns. I am currently able to use Python's native CGIHTTPServer module to accomplish this. To start, I just run the CGIHTTPServer module directly, by executing from a terminal:

python -m CGIHTTPServer

Then, someone accesses www.example-server.net:8000/foo.py and they are given one of these names at random. I have the following code in foo.py:

#!/usr/bin/env python

import random

name_list = list()
FILE = open('names.txt','r')
for line in FILE:
    name = line[:-1]
    name_list.append(name)

FILE.close()
name_to_return = random.choice(name_list)

print "Content-type: text/html"
print
print "<title>Here is your name</title>"
print "<p>" + name_to_return + "</p>"

This does what I want; however, it is extremely inefficient, because every access forces the server to re-read a 4 GB file.

How can I make this into an efficient process, where the variable name_list is created as global immediately when the server starts, and each access only reads from that variable?

Improve this question

4 Answers 4

Reset to default
7

Just for future reference, if anyone ever faces the same problem: I ended up sub-classing CGIHTTPServer's request handler and implementing a new do_POST() function. If you had a working CGI script without global variables, something like this should get you started:

import CGIHTTPServer
import random
import sys
import cgi

class MyRequestHandler(CGIHTTPServer.CGIHTTPRequestHandler):
    global super_important_list
    super_important_list = range(10)
    random.shuffle(super_important_list)

    def do_POST(s):    
        """Respond to a POST request."""
        form = cgi.FieldStorage(fp=s.rfile,headers=s.headers,environ={'REQUEST_METHOD':'POST','CONTENT_TYPE':s.headers['Content-Type'],})
        s.wfile.write("<html><head><title>Title goes here.</title></head>")
        s.wfile.write("<body><p>This is a test.</p>")
        s.wfile.write("<p>You accessed path: %s</p>" % s.path)
        s.wfile.write("<p>Also, super_important_list is:</p>")
        s.wfile.write(str(super_important_list))
        s.wfile.write("<p>Furthermore, you POSTed the following info: ")
        for item in form.keys():
            s.wfile.write("<p>Item: " + item)
            s.wfile.write("<p>Value: " + form[item].value)
        s.wfile.write("</body></html>")

if __name__ == '__main__':
    server_address = ('', 8000)
    httpd = CGIHTTPServer.BaseHTTPServer.HTTPServer(server_address, MyRequestHandler)
    try:
        httpd.serve_forever()
    except KeyboardInterrupt:
        sys.exit()

Whenever someone fills out your form and performs a POST, the variable form will be a dictionary-like object with key-value pairs which may differ for each user of your site, but the global variable super_important_list will be the same for every user.

Thanks to everyone who answered my question, especially Mike Steder, who pointed me in the right direction!

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

4

CGI works by spawning a process to handle each request. You need to run a server process that stays in memory handles HTTP requests.

You could use a modified BaseHTTPServer, just define your own Handler class. You'd load the dataset once in your code and then the do_GET method of your handler would just pick one randomly.

Personally, I'd look into something like CherryPy as a simple solution that is IMO a lot nicer than BaseHTTPServer. There are tons of options other than CherryPy like bottle, flask, twisted, django, etc. Of course if you need this server to be behind some other webserver you'll need to look into setting up a reverse proxy or running CherryPy as a WSGI app.

Improve this answer

4 Comments

I have actually dwindled into sub-classing BaseHTTPServer. Am I correct in assuming that I have to re-define ALL the methods of BaseHTTPServer (i.e., do_GET, do_POST, etc)? This is why I assumed that something better already existed. Regarding CherryPy, can you point me to a "for dummies" tutorial? I've looked into their page but even their documentation, which others describe as "excellent", is too hard to understand for me.
@HerrKaputt: each of those methods correspond to an HTTP method you may want to support. For your use case I think you only need to support do_GET.
Thanks a lot, Mike! While your answer isn't exactly what I'm looking for, it actually prompted me to dig further. I omitted from my question that I need to pass parameters through POST methods. What I need, then, is to subclass SimpleHTTPServer and create my own do_POST() function. I'll draw inspiration from the one in CGIHTTPServer. Hopefully I won't need to ask further questions!
@HerrKaputt: Happy to help! Don't hesitate to ask further questions, that's what we're all here for.
2

You may want to store the values of the names in a db and store the names according to the letter that they start with. Then you can do a random for a letter between a and z and from there randomize again to get a random name from your random beginning letter.

Improve this answer

4 Comments

Thanks for the answer. Databases are in my list of things to learn, but it seems like complete overkill just for this need.
Okay I would then possibly try to generate a random number and only read that line of the file. This way you don't have to loop through every line.
That would work for this simple example. However, it would not work for the application which I have in mind, which truly requires reading the whole file into memory. Obviously, that misunderstanding is not your fault. I'll edit the original question to reflect that.
Ah ok. Well I am sorry I couldn't be of more help.
2

Build a prefix tree (a.k.a. trie) once and generate a random walk whenever you receive a query.

That should be pretty efficient.

Improve this answer

3 Comments

It is efficient. But it still doesn't answer my question: how do I setup this as a server which builds a read-only global variably which is shared by all users?
@HerrKaputt Then I obviously was mislead by the complexity of your example. You rather want a "hello world" example for setting up a basic http server?
Sort of. A basic HTTP server is something I can make in Python; however, it will not be able to share variables among different users. On the other hand, I am able to create global variables in a Python script, but not among different users, because (as Mike said) CGI creates independent processes for different users. I don't know what I need to do to combine these two things.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.