Writing file to HDFS using Java

Question

I'm trying to write a file to HDFS, the file get created but it is empty on the cluster, however when I run the code locally it works like a charm.

here's my code :

FSDataOutputStream recOutputWriter = null;
FileSystem fs = null;
try {
    //OutputWriter = new FileWriter(outputFileName,true);
    Configuration configuration = new Configuration();
    fs = FileSystem.get(configuration);
    Path testOutFile = new Path(outputFileName);
    recOutputWriter = fs.create(testOutFile);

    //outputWriter = new FileWriter(outputFileName,true);
} catch (IOException e) {
    e.printStackTrace();
}
recOutputWriter.writeBytes("======================================\n");
recOutputWriter.writeBytes("OK\n");
recOutputWriter.writeBytes("======================================\n");

if (recOutputWriter != null) {
    recOutputWriter.close();
}
fs.close();

did I miss something ?

flush data ? hadoop.apache.org/docs/r2.6.1/api/org/apache/hadoop/fs/… — Pierre
– Pierre, Commented Jun 29, 2016 at 11:05
@Pierre thanks for your reply, even when I flush the data, the file still blank! — Jay
– Jay, Commented Jun 29, 2016 at 11:10
What is outputFileName's value? and did you try fs and testOutFile instantiation as this(with hdfs URI) stackoverflow.com/questions/16000840/… — Ronak Patel
– Ronak Patel, Commented Jun 29, 2016 at 13:31
@BigDataLearner , outputFileName can be a path like /files/out.txt OR out.txt — Jay
– Jay, Commented Jun 29, 2016 at 14:52
@BigDataLearner I've tried the AddResources but didn't succeed — Jay
– Jay, Commented Jun 29, 2016 at 16:29

Jay · Accepted Answer · 2016-06-30 06:35:12Z

1

In order to write data to a file after creating it on the cluster I had to add :

System.setProperty("HADOOP_USER_NAME", "vagrant");

Refrence - Writing files to Hadoop HDFS using Scala

answered Jun 30, 2016 at 6:35

Jay

7571 gold badge12 silver badges38 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

usamazf Over a year ago

Hi can I bother you with some noob questions? I didn't wan to open a new thread for something like this. But how do you actually write to HDFS? Is there some tutorial or blog I can follow? I absolutely beginner and have managed to create a cluster after 36 hours of banging my head at the screen. Thanks!

Collectives™ on Stack Overflow

Writing file to HDFS using Java

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related