Using python to push data into HDFS

Question

I'm using the following python code, run through PuTTy, to call an API for some weather forecast data:

r = requests.get('http://datapoint.metoffice.gov.uk/public/data/val/wxfcs/all/xml/350509?res=3hourly&key=abc-123-mnb-0987')

Please can anyone advise on the correct syntax to pipe this data straight through into HDFS (i'm running the HDF 2.4 sandbox on a VM)?

Possibly duplicate stackoverflow.com/questions/32099664/…

BruceWayne
– BruceWayne

2017-01-07 01:24:28 +00:00
Commented Jan 7, 2017 at 1:24 — BruceWayne
– BruceWayne, Commented Jan 7, 2017 at 1:24

Ani Menon · Accepted Answer · 2017-01-07 05:10:48Z

0

Something like this:

r = requests.get('http://datapoint.metoffice.gov.uk/public/data/val/wxfcs/all/xml/350509?res=3hourly&key=abc-123-mnb-0987') 
os.system('echo "%s" | hdfs dfs -copyFromLocal - /path/in/hdfs' %(r.text))

edited Jan 7, 2017 at 5:10

answered Jan 7, 2017 at 4:42

Ani Menon

28.4k17 gold badges111 silver badges128 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Using python to push data into HDFS

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related