6

I think I don't completely understand the deployment process. Here is what I know:

  • when we need to do hot deployment -- meaning that we need to change the code that is live -- we can do it by reloading the modules, but
  • imp.reload is a bad idea, and we should restart the application instead of reloading the changed modules
  • ideally the running code should be a clone of your code repository, and any time you need to deploy, you just pull the changes

Now, let's say I have multiple instances of wsgi app running behind a reverse proxy like nginx (on ports like 8011, 8012, ...). And, let's also assume that I get 5 requests per second.

Now in this case, how should I update my code in all the running instances of the application.

  • If I stop all the instances, then update all of them, then restart them all -- I will certainly lose some requests
  • If I update each instance one by one -- then the instances will be in inconsistent states (some will be running old code, and some new) until all of them are updated. Now if a request hits an updated instance, and then a subsequent (and related) request hits an older instance (yet to be updated) -- then I will get wrong results.

Can somebody explain thoroughly how busy applications like this are hot-deployed?

Improve this question
1
  • Great question. Looking forward to the answers! Commented Jun 19, 2012 at 13:11

2 Answers 2

Reset to default
2

For deployment across several hot instances that are behind a load balancer like nginx I like to do rolling deployments with a tool like Fabric.

  1. Fabric connects you to Server 1
  2. Shut down the web-server
  3. Deploy changes, either by using your VCS or transferring tarball with the new application
  4. Start up the web-server
  5. GOTO1 and connect to the next server.

That way you're never offline, and it's seamless as nginx knows when a webserver is taken down when it tries to round-robin to it and will move onto the next one instead, and as soon as the node/instance is back up it will be back into production usage.

EDIT:

You can use the ip_hash module in nginx to ensure all requests from one IP Address goes to the same server for the length of the session

This directive causes requests to be distributed between upstreams based on the IP-address of the client. The key for the hash is the class-C network address of the client. This method guarantees that the client request will always be transferred to the same server. But if this server is considered inoperative, then the request of this client will be transferred to another server. This gives a high probability clients will always connect to the same server.

What this means to you, is that once your web-server is updated and a client has connected to the new instance, all connections for that session will continue to be forwarded to the same server.

This does leave you in the situation of

  1. Client connects to site, gets served from Server 1
  2. Server 1 is updated before client finishes whatever they're doing
  3. Client potentially left in a state of limbo?

This scenario begs the question, are you removing things from your API/Site which could potentially leave the client in a state of limbo ? If all you're doing is for example updating UI elements or adding pages etc but not changing any back-end APIs you should not have any problems. If you are removing API functions, you might end up with issues.

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

BUT, as I outline above, this leaves your application servers in inconsistent state. What if a request hits the updated server, and then a related request hits the one yet to be updated? How do you manage that span of time while your servers are being updated?
Edited my answer to include information about ip_hash for persistent connections to a server for the session.
0

Couldn't you take half your servers offline (say by pulling them out of the load balancing pool) and then update those. Then bring them back online while simultaneously pulling down the other half. Then update those and bring them back online.

This will ensure that you stay online while also ensuring that you never have the old and new versions of your application online at the same time. Yes, this will mean that your site would run at half its capacity during the time. But that might be ok?

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.