Append data to file in hadoop using java api

Question

I have created a file with the results of a sequence of map-reduce jobs. The program i' ve made iteratively outputs some results. I want to append those data in the result file using the java api. I have tried fs.append but it doesn't work. For the time being I am using built-in libraries of java (eclipse 4.2.2) and when I'm ok with the debugin I'll make it a jar and throw it in a cluster.

First of all, is "append" accepted in hdfs? And if yes can anyone tell me how it's done? Thnx in advance.

The code I am using to do this job is the following:

try{
    Path pt = new Path("/home/results.txt");
    FileSystem fs = FileSystem.get(new Configuration());
    BufferedWriter br = new BufferedWriter(new OutputStreamWriter(fs.append(pt)));
    String line = "something";
    br.write(line);
    br.close();
} catch (Exception e) {
    System.out.println("File not found");
}

what version of Hadoop are you using? Also add the code you're trying to use — Arnon Rotem-Gal-Oz
– Arnon Rotem-Gal-Oz, Commented May 31, 2014 at 19:40
Your code will be a lot more readable if you edit your question and add it in there. — nelsonda
– nelsonda, Commented Jun 2, 2014 at 15:15
Kindly check my answer here that worked for me: stackoverflow.com/a/43824906/6129092 — Lovish chaudhary
– Lovish chaudhary, Commented May 29, 2017 at 7:45
Kindly check my answer here : stackoverflow.com/a/43824906/6129092 — Lovish chaudhary
– Lovish chaudhary, Commented May 29, 2017 at 7:47

Rishi Dwivedi · Accepted Answer · 2014-06-05 04:54:00Z

2

Early versions of HDFS had no support for an append operation. Once a file was closed, it was immutable and could only be changed by writing a new copy with a different filename.

see more information here

if you using old version this is work for me ......

 BufferedReader bfr=new BufferedReader(new InputStreamReader(hdfs.open(path)));     //open file first
            String str = null;
            BufferedWriter br=new BufferedWriter(new OutputStreamWriter(hdfs.create(path,true))); 
            while ((str = bfr.readLine())!= null)
            {
                br.write(str); // write file content
                br.newLine();
               System.out.println("   ->>>>>  "+str);
             }
            br.write("Hello     ");  // append into file
            br.newLine();
            br.close(); // close it

edited Jun 5, 2014 at 4:54

answered Jun 4, 2014 at 13:04

Rishi Dwivedi

9385 silver badges19 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

kostas Over a year ago

Thanks! I've seen that. What I did to fix my problem was to delete and re-write the file along with anything I want to append everytime. That's ok for now but I'm still interested in using fs.append. Thanks anyway!

Collectives™ on Stack Overflow

Append data to file in hadoop using java api

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related