java.lang.OutOfMemoryError: Java heap space with hive

Question

I used hadoop hive 0.9.0 and 1.1.2 and netbeans, but I got this error and I can not solve this problem please help me code :

public class Hive_test {

private static String driverName = "org.apache.hadoop.hive.jdbc.HiveDriver";

   @SuppressWarnings("CallToThreadDumpStack")
public static void main(String[] args) throws SQLException {
    try {
        Class.forName(driverName);
    } catch (ClassNotFoundException e){
        e.printStackTrace();
        System.exit(1);
    }
            System.out.println("commencer la connexion");
    Connection con = DriverManager.getConnection("jdbc:hive://localhost:10000/default",""," ");
    Statement stmt = con.createStatement();
    ResultSet res = stmt.executeQuery("select * from STATE");
    while (res.next()){
        System.out.println(String.valueOf(res.getInt(1)) + "\t" + res.getString(2));
                    System.out.println("sql terminer");
    }
}

Error below;

error :
commencer la connexion
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:353)
    at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215)
    at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
    at org.apache.hadoop.hive.service.ThriftHive$Client.recv_execute(ThriftHive.java:116)
    at org.apache.hadoop.hive.service.ThriftHive$Client.execute(ThriftHive.java:103)
    at org.apache.hadoop.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:192)
    at org.apache.hadoop.hive.jdbc.HiveStatement.execute(HiveStatement.java:132)
    at org.apache.hadoop.hive.jdbc.HiveConnection.configureConnection(HiveConnection.java:132)
    at org.apache.hadoop.hive.jdbc.HiveConnection.<init>(HiveConnection.java:122)
    at org.apache.hadoop.hive.jdbc.HiveDriver.connect(HiveDriver.java:106)
    at java.sql.DriverManager.getConnection(DriverManager.java:571)
    at java.sql.DriverManager.getConnection(DriverManager.java:215)
    at hive.Hive_test.main(Hive_test.java:22)

Did you try with more memory ? with -Xmx ?

Jayan
– Jayan

2015-04-16 11:21:04 +00:00
Commented Apr 16, 2015 at 11:21 — Jayan
– Jayan, Commented Apr 16, 2015 at 11:21
@Jayan no I do not know how, because I work netbeans

kawther
– kawther

2015-04-16 11:24:18 +00:00
Commented Apr 16, 2015 at 11:24 — kawther
– kawther, Commented Apr 16, 2015 at 11:24

Modus Tollens · Accepted Answer · 2015-04-16 11:40:03Z

24

You can set the container heapsize in Hive and resolve this error:

Most tools that operate on top of the Hadoop MapReduce framework provide ways to tune these Hadoop level settings for its jobs. There are multiple ways to do this in Hive. Three of these are shown here:

1) Pass it directly via the Hive command line:

hive -hiveconf mapreduce.map.memory.mb=4096 -hiveconf mapreduce.reduce.memory.mb=5120 -e "select count(*) from test_table;"

2) Set the ENV variable before invoking Hive:

export HIVE_OPTS="-hiveconf mapreduce.map.memory.mb=4096 -hiveconf mapreduce.reduce.memory.mb=5120"

3) Use the "set" command within the hive CLI.

hive> set mapreduce.map.memory.mb=4096;
hive> set mapreduce.reduce.memory.mb=5120;
hive> select count(*) from test_table;

edited Apr 16, 2015 at 11:40

Modus Tollens

5,1343 gold badges41 silver badges47 bronze badges

answered Apr 16, 2015 at 11:23

Ranjeet

6346 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

kawther Over a year ago

how i can set the container heapsize in Hive, i use netbeans please help me

Reihan_amn Over a year ago

this setting won't help fot java GC overhead limit error, am I correcT?

EasternXiang · Accepted Answer · 2019-04-19 08:44:20Z

7

Well, in my case, I also need to set memory in java.opts

set mapreduce.map.memory.mb=4096;
set mapreduce.map.java.opts=-Xmx3686m;
set mapreduce.reduce.memory.mb=4096;
set mapreduce.reduce.java.opts=-Xmx3686m;

answered Apr 19, 2019 at 8:44

EasternXiang

5996 silver badges7 bronze badges

Comments

user2720864 · Accepted Answer · 2019-11-20 10:34:12Z

5

For me the below solution works.
Before starting the hive CLI use export HADOOP_CLIENT_OPTS=" -Xmx8192m" and then launch the cli

answered Nov 20, 2019 at 10:34

user2720864

8,2115 gold badges51 silver badges61 bronze badges

Collectives™ on Stack Overflow

java.lang.OutOfMemoryError: Java heap space with hive

3 Answers 3

2 Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related