1

This is an old app, running since about two years on Heroku. Now suddenly, when I deploy (standard git push), it breaks the Python interpreter, both on regular and one-off dynos. Here's what it looks like:

$ heroku run python
Running `python` attached to terminal... up, run.8338
python: error while loading shared libraries: libpython2.7.so.1.0: cannot open shared object file: No such file or directory

Further pushes, heroku restart, scaling the dynos to zero and back up, that all doesn't fix it.

The only changes contained in this deployment which I could imagine being related to this problem are: gevent upgraded from 0.13.6 to 1.0.1 and the introduction of a runtime.txt (before there was none, resulting in 2.7.4, now there's one for 2.7.6).

I rolled all of this back to no effect, however. In fact, I went back about 30 commits (the deployment contained maybe five) and pushed that and the app is still broken.

Rolling back with heroku rollback is the only way I've found to take the app back into a usable state. But of course that doesn't help me going forward.

What might cause this? Can I somehow rebuild my whole app environment from scratch?

EDIT 1: I opened a shell in a one-off dyno and I can see the libpython2.7.so.1.0 file there:

/ $ ls -la /app/.heroku/python/lib/libpython2.7.so.1.0
-r-x------ 1 u49295 49295 5694572 2014-06-03 23:39 /app/.heroku/python/lib/libpython2.7.so.1.0

Of course I don't know if that's where it's supposed to be.

Improve this question

3 Answers 3

Reset to default
2

Somehow certain apps did not upgrade properly. A temporary fix to locate the correct python library:

heroku config:set LD_LIBRARY_PATH=/app/.heroku/python/lib
Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

Strangely, it seems like I can immediately afterwards heroku config:unset LD_LIBRARY_PATH and it will keep working. (I guess this config flag actually just sets the environment variable and then doesn't remove it when you unset.)
Before my deployment (after a release:rollback), the LD_LIBRARY_PATH environment variable is set to /app/.heroku/vendor/lib (even though /app/.heroku/vendor does not exist), but the Python interpreter still works. After the deployment, it's set to the same but no longer works. So I don't quite understand what's happening here. But at any rate, the fix works.
Yeah I don't fully understand want is happening under the hood either :S. Some of my apps have this non-existent path, but works fine, but one of my apps broke. It is probably some heroku magic under it all.
1

Kenneth Reitz from the Python team here.

So, we're rolling out security updates to the version of Python that we have installed on our base system. Customers shouldn't be affected by this in any way, because their apps use their own version of Python, and because we set local runtime-specific (such as LD_LIBRARY_PATH) in a .profile.d scripts, outside of user-set configuration.

However, we allow power users to override these environment variables with $ heroku config. That's basically what the application was doing — although, not knowingly. This was an accidental side-effect of a much, much older Heroku. In older days, we couldn't have any runtime-specific configuration without being a part of user configuration. This is why your application had an LD_LIBRARY_PATH config set, and this is what caused this bug.

Because of this, I've disabled the overridability of LD_LIBRARY_PATH for Python apps, and all should be well moving forward.

Thanks for being a part of the gradual rollout process, and thanks for helping us get to the bottom of this regression. I'm very sorry for the inconvenience.

Improve this answer

Comments

0

Are you using the default Python buildpack? Heroku is in the process of updating the Stack image, and especially if you're not using a current buildpack, there may be incompatibilities.

To see if you're using a default buildpack, run

$ heroku config | grep BUILDPACK_URL

Please contact Heroku support if you think this might be the cause.

Improve this answer

1 Comment

Good idea, but no, I'm not using a custom buildpack, I'm using the default auto-detected Python buildpack. Should have mentioned this right away. Where did you read about them updating the Stack image? Maybe that's related to my issue.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.