Save JSON to HDFS using python

Question

I have a python script which currently accesses an API which returns JSON. It then takes the JSON string and saves it off as a file on the local file system, where i then move it into HDFS manually. I would like to change this so my python script is saving directly to HDFS instead of hitting the local file system first. I am currently trying to save the file using HDFS and DFS command but I don't think the copy command is the correct way to do this because it isn't a file but rather a JSON string when I am trying to save it.

Current Code

import urllib2
import json
import os

f = urllib2.urlopen('RESTful_API_URL.json')
json_string = json.loads(f.read().decode('utf-8'))
with open('\home\user\filename.json', 'w') as outfile:
    json.dump(json_string,outfile)

New Code

f = urllib2.urlopen('RESTful_API_URL.json')
json_string = json.loads(f.read().decode('utf-8'))
os.environ['json_string'] = json.dump(json_string)
os.system('hdfs dfs -cp -f $json_string hdfs/user/test')

Community · Accepted Answer · 2017-05-23 11:48:28Z

7

I think the problem is the same with this thread Stream data into hdfs directly without copying.

Firstly, this command can redirect stdin to hdfs file,

hadoop fs -put - /path/to/file/in/hdfs.txt

Then, you can do this in python,

os.system('echo "%s" | hadoop fs -put - /path/to/file/in/hdfs.txt' %(json.dump(json_string)))

edited May 23, 2017 at 11:48

CommunityBot

11 silver badge

answered Aug 19, 2015 at 17:58

eval

1,26914 silver badges22 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

SRIRAM RAMACHANDRAN Over a year ago

the file is created but no contents have written in to the file in HDFS... what would be the issue?

G.v. Sridhar Over a year ago

If you use echo "%s" , your json file will be stored without double quotes

Mohammad Rijwan Over a year ago

I used the same technique, the JSON string is not copied in HDFS? any reason.

Rahul Over a year ago

@eval , when i use the os.system('echo "%s" | hadoop fs -put - /path/to/file/in/hdfs.txt' %(json.dump(json_string))) iam getting prefix u and ' ' char before. {u'manager':u'robert',u'name':u'kim'} ,how i can remove u and ' ' for every key ,value in my json

mattinbits · Accepted Answer · 2015-08-19 16:02:24Z

1

Take a look at the HDFS put command at http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/FileSystemShell.html#put

You can put to HDFS from the command line using standard in with syntax like the following (-put - means read from stdin).

hadoop fs -put - hdfs://nn.example.com/hadoop/hadoopfile

If you can start this command as a sub process within your python code, you should be able to pipe your json string to the sub process.

answered Aug 19, 2015 at 16:02

mattinbits

10.4k1 gold badge28 silver badges35 bronze badges

Comments

Mohammad Rijwan · Accepted Answer · 2020-06-23 12:14:59Z

-1

It helped in my case:

import os
import requests

r = requests.get(url = url,headers=headers)
json_string = r.json()
os.system('echo "%s" | hadoop fs -put - /<your_hdfs_path>/json_name.json' %(json_string))

answered Jun 23, 2020 at 12:14

Mohammad Rijwan

3574 silver badges20 bronze badges

Collectives™ on Stack Overflow

Save JSON to HDFS using python

3 Answers 3

4 Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

4 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related