Walk HDFS using Python

Question

Is it possible to walk HDFS using Python? If it's possible, how can I do it? Thanks!

Haven't tried anything yet, but I'm looking at code.google.com/p/libpyhdfs . Basically I want to walk the HDFS and add partitions to a HIVE table. — darcyy
– darcyy, Commented Dec 4, 2012 at 14:46
You will frequently get a more positive response if you make it appear you have applied a little bit of thought or effort to the topic before posting, even if it is just to link to the documentation of the libraries you have been looking at. — Marcin
– Marcin, Commented Dec 4, 2012 at 15:02

quasiben · Accepted Answer · 2016-03-30 02:50:40Z

1

hdfs3 which is based on libhdfs3 supports this

from hdfs3 import HDFileSystem
hdfs = HDFileSystem()
hdfs.walk('/path/to/directory')

answered Mar 30, 2016 at 2:50

quasiben

1,4641 gold badge11 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Faruk Sahin · Accepted Answer · 2012-12-04 14:45:18Z

0

There are some libraries that you can take a look:

answered Dec 4, 2012 at 14:45

Faruk Sahin

8,8465 gold badges31 silver badges34 bronze badges

2 Comments

darcyy Over a year ago

In terms of walking file structure, which one would you recommend to use?

Faruk Sahin Over a year ago

Well, I haven't used any of them really. But, you have the listing directory method on both of them, so you can use both. pydoop seems to be better documented though.

Collectives™ on Stack Overflow

Walk HDFS using Python

2 Answers 2

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related