17

I am attempting to upload a small file to gcloud using a simple python program

client = storage.Client(project=GCLOUD_PROJECT)
bucket = client.get_bucket(GCLOUD_BUCKET)
blob = bucket.blob(GCLOUD_FILE_ON_CLOUD)
blob.upload_from_filename(GCLOUD_FILE_LOCAL)

It had been working until recently and something changed. Now, whenever I upload a file greater than 5MB I get the below error. Files less than or equal to 5MB goes through. The size isn't large enough to implement resumable upload, is it?

Traceback (most recent call last):
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/gcloud_upload.py", line 40, in <module>
    blob.upload_from_filename(GCLOUD_FILE_LOCAL)
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/gcloud/storage/blob.py", line 597, in upload_from_filename
    encryption_key=encryption_key, client=client)
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/gcloud/storage/blob.py", line 543, in upload_from_file
    http_response = upload.stream_file(use_chunks=True)
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/gcloud/streaming/transfer.py", line 1086, in stream_file
    response = send_func(self.stream.tell())
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/gcloud/streaming/transfer.py", line 1215, in _send_chunk
    return self._send_media_request(request, end)
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/gcloud/streaming/transfer.py", line 1125, in _send_media_request
    self.bytes_http, request, retries=self.num_retries)
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/gcloud/streaming/http_wrapper.py", line 423, in make_api_request
    check_response_func=check_response_func)
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/gcloud/streaming/http_wrapper.py", line 371, in _make_api_request_no_retry
    redirections=redirections, connection_type=connection_type)
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/oauth2client/transport.py", line 175, in new_request
    redirections, connection_type)
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/oauth2client/transport.py", line 282, in request
    connection_type=connection_type)
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/httplib2/__init__.py", line 1986, in request
    cachekey,
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/httplib2/__init__.py", line 1685, in _request
    content,
httplib2.RedirectMissingLocation: Redirected but the response is missing a Location: header.

When I debug I see the below. 

{
 'content-type': 'text/plain; charset=utf-8', 
 'range': 'bytes=0-1048575', 
 'content-length': '0', 
 'date': 'Sun, 19 Jan 2020 23:52:13 GMT', 
 'server': 'UploadServer', 
 'alt-svc': 'quic=":443"; ma=2592000; v="46,43",h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000', 
 'status': '308'}

My pip list for reference. OS is MAC OSx. I tested on Linux as well. Same issue.

adal                     1.2.2    
bson                     0.5.8    
cachetools               3.1.1    
certifi                  2019.9.11
cffi                     1.13.1   
chardet                  3.0.4    
cryptography             2.8      
dnspython                1.16.0   
gcloud                   0.18.3   
gitdb2                   2.0.6    
GitPython                3.0.5    
google-auth              1.6.3    
googleapis-common-protos 1.51.0   
httplib2                 0.16.0   
idna                     2.7      
itsdangerous             1.1.0    
MarkupSafe               1.1.1    
oauth2client             4.1.3    
oauthlib                 3.1.0    
pip                      19.0.3   
protobuf                 3.11.2   
pyasn1                   0.4.7    
pyasn1-modules           0.2.7    
pycparser                2.19     
PyJWT                    1.7.1    
python-dateutil          2.8.0    
requests                 2.19.1   
requests-oauthlib        1.2.0    
rsa                      4.0      
setuptools               40.8.0   
six                      1.12.0   
smmap2                   2.0.5    
urllib3                  1.23     
websocket-client         0.56.0   
Werkzeug                 0.16.0

This was working last week. Has something changed recently?

Improve this question
4
  • Same issue but with the python google cloud SDK. Started happening after I had to reinstall python, but as part of that process I had to update httplib2 to 0.16.0 which I believe is causing the issue. Unfortunately I don't have a fix yet either. Commented Jan 20, 2020 at 15:09
  • 1
    Filed to track: github.com/httplib2/httplib2/issues/156 Commented Jan 21, 2020 at 18:00
  • Looks like it was caused by this change: github.com/httplib2/httplib2/commit/… Commented Jan 21, 2020 at 18:45
  • I suggest that you follow this github issue for more information. Commented Jan 22, 2020 at 16:14

5 Answers 5

Reset to default
20

I solved this problem with:

pip install httplib2==0.15.0

pip install google-api-python-client==1.6

Edit: faster on load is: pip install httplib2==0.15.0 pip install google-api-python-client==1.7.11

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

I'm from future to say that this error is still present at version of 2.95.0 of the google-api-python-client. You don't need to downgrade de google-api-python-client version, just downgrade httplib2 to 0.15.0 and it works. But it's weird that nobody on Google solved this almost 4 years later.
18

Solutions

gcloud package is deprecated twice and is not compatible with httplib2>=0.16. Proper solution is to use google-cloud-* packages family.

google-api-python-client>=1.7.12 is using redirect_codes API, please upgrade, it just works.

httplib2 v0.17.0 is just released with ability to modify set of response codes treated as redirects. This is the best option if you can modify code that creates Http object:

http = httplib2.Http()
http.redirect_codes = http.redirect_codes - {308}

Iff that's not possible, edit your requirements.txt to pin down httplib2<0.16.0

Long story

google cloud storage server uses HTTP 308 for special resumable uploads feature which somewhat resembles "retry same method to same location", but not quite.

Above is (probably) rationale for PyPI package google-resumable-media which is used by more recent incarnations of gcloud related packages and handles 200 and 308 in similar manner, unlike generic HTTP client should.

History context:

Sorry for bad news. As HTTP enthusiast, I'm biased towards 308 support. Please reach if you have better idea how to handle this situation more gracefully.

Improve this answer

5 Comments

Looks like google-api-python-client is not compatible with httplib2>=0.16.0 and is not deprecated. @temoto do you know anything about this?
@Shcheklein yes that one may have a better way forward solution, subscribe to this github.com/googleapis/google-api-python-client/issues/803
@Shcheklein ping google-api-python-client has just merged fix.
This should be marked as an answer. The solution is still valid with: Python 2.7, google-api-python-client==1.12.8, google-auth-httplib2==0.0.3, httplib2==0.18.1, oauth2client==4.1.3
Awesome in-depth explanation!
16

Downgrade your httplib2 version to 0.15.0. Worked for me on the python google-cloud-sdk.

Improve this answer

1 Comment

It worked for me! It is not very reassuring concerning GCP...
1

I had this error on Dataflow (masquerading also as a BrokenPipe). Downgrading google-api-python-client to the versions >=1.7.8,<1.7.12 fixed it, because 1.7.12 introduced a dependency on httplib2 0.17.0 which was somehow incompatible.

Improve this answer

Comments

0

I can't believe it. this problem was listed over 3 years ago and it's still a problem. I've been playing whack-a-version between the following python packages to figure this thing out because they still haven't figured it out on the package side.

Here is my situation - I have a process that upload certain somewhat larger files (archive files created from gmail/chat/etc) to a google drive location.

The error (httplib2.RedirectMissingLocation: Redirected but the response is missing a Location: header.) happened for me when my process tried to upload larger files.

I can't upgrade any of the following packages:

google-auth-httplib2
google-api-python-client
httplib2
google-auth

This is a process that I inherited from programmers who have now moved on to greener pastures, so I think I'll have to rewrite this process to try and use all google-cloud-* packages (like @temoto suggested).

between now and last week (when this process worked), it looks like a newer version of google-auth-httplib2 was made available (v. 0.1.1), my process used it, and that error showed up. I downgraded that and now it looks like it is working.

For now, this is the package versions that work for me:

google-auth-httplib2==0.1.0
google-api-python-client==1.7.8
httplib2==0.15.0
google-auth==1.12.0
Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.