Import problems with hdfs java api

Question

I'm trying to run the example found here: http://thysmichels.com/2012/01/31/java-based-hdfs-api-tutorial/

But when I go to compile the java program I get errors saying the packages don't exist eg.

error: package org.apache.hadoop.conf does not exist
import org.apache.hadoop.conf.Configuration;

Hadoop 1.0.4 is installed and works fine. Every tutorial I've looked at for dealing with hdfs just starts with a program like in the link I provided earlier and they do not talk about any special prereqs I need. So I'm wondering what do I need to do to make this compile? I'm assuming I need to edit my classpath to point to the appropriate packages but I do not know where those are located.

Also I'm running Ubuntu 12.04, Hadoop 1.0.4 on a single node cluster following the instructions here: http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/

How are you compiling the program? (Ant, Maven, IDE, command line?). You'll get this error if the hadoop-core jar isn't on the classpath — Chris White
– Chris White, Commented Dec 3, 2012 at 1:49
And can you post the exact command line you are using to compile the code (edit your question rather than psot it as a comment) — Chris White
– Chris White, Commented Dec 3, 2012 at 11:37
I'm just gonna put it as a comment since I don't think you'll be notified if I just do an edit: javac HDFSExample.java — user1443362
– user1443362, Commented Dec 4, 2012 at 0:13

Chris White · Accepted Answer · 2012-12-04 00:45:01Z

1

I'd suggest you brush up on some basic java compilation basics.

You need to do more than just a javac HDFSExample.java - in that you need to include some dependency jars on the classpath. Something more like javac -cp hadoop-core-1.0.4.jar HDFSExample.java

Personally, i'd recommend looking into using a build tool (such as Maven, Ant) or an IDE as this will make things far less painful when you start to organize your code into packages and depend on multiple external libraries.

EDIT: For example, maven configuration is as simple as (ok i'm not including some of the other boiler plate pom decalarations..):

<dependencies>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-core</artifactId>
        <version>1.0.4</version>
        <scope>provided</scope>
    </dependency>
<dependency>

Then to compile into a jar:

#> mvn jar

edited Dec 4, 2012 at 0:45

answered Dec 4, 2012 at 0:21

Chris White

30.1k4 gold badges75 silver badges96 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

user1443362 Over a year ago

Well actually I just changed my classpath to include the hadoop core file instead of doing it in the commandline. The reason I asked the question was so you would tell me what file actually held those specific files for the import not because I don't know how to link to external dependencies. Now the problem is that when it runs I get an error saying: "Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/logging/LogFactory". Again I am assuming this is a class path issue so if you happen to know what jar contains the LogFactory it would be appreciated.

Chris White Over a year ago

commons-logging, but seriously look into maven - you just specify hadoop-core dependency and it will figure out the other things that hadoop depends on

Collectives™ on Stack Overflow

Import problems with hdfs java api

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related