3

I'm having the following issue with Hbase.

I have a script which starts the HBase shell and inserts many rows into a table with a single column. I have tried inserting 10,000 rows but after about 1,700 I get the dreaded "java.lang.OutOfMemoryError: unable to create new native thread" error. I have tried changing the Java heap size from 1000mb default to 1800mb, but this doesn't allow me to insert any more than the 1700 or so rows.

However, I've noticed that I can insert 1000 rows, exit the shell, restart the shell, insert 1000 more into the same table, exit again, and so on and so forth. I don't really understand enough about the JVM to figure out why it's allowing me to do this in several sessions, but not allowing me to batch insert in the same session.

Can someone please explain to me what is going on here, and what I might do about it?

EDIT:

I am now using 64-bit machine, red hat linux 5, with Java 1.6. I'm giving HBase a heapsize of 20gb (I have ~32 gigs memory total). For stack size, I'm giving 8mb. The default on 64-bit is 2mb I believe; with 2mb I got this same error, and increasing it to 8mb did not help at all (I was only able to insert the same amount of rows regardless of stack size, ~1700).

I have read that decreasing the heap size could make this error go away but that did not help either. Below are the jvm options that I'm setting (everything is default except for stack size).

HBASE_OPTS="$HBASE_OPTS -ea -Xss8M -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode"
Improve this question
1
  • Can you share what JVM args you are setting? Commented Jan 24, 2011 at 21:54

3 Answers 3

Reset to default
4

I encountered this error yesterday. What was happening in my case is that I was creating a lot of instances of HTable which created way too many threads when I was using the put on a record. (I was using a mapper and creating it inside the map function)

I'd check to see if your connection to HBase is being created a lot (inside a loop or a map function. If that is happening, then moving it to instantiate fewer connections to HBase (I used HTable) may solve the problem. It solved mine.

HTH

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

This has happened when I connected to HBase using a Java client as well as when i pass a script to the HBase shell. The shell script is just a list of put commands, one right after the other. In both cases (using the Java client and using the shell script), I get the error after the same number of rows.
To continue my earlier comment, I made this change in the Java client and it did work...I was able to insert A LOT more rows this time. However, I am confused why I am still not able to insert more than 1600 rows through the shell, or what causes this error in the first place. Do you have any idea?
In JAVA, it's a matter of allocating memory to each thread in the NVM and running out of space to allocate another thread. I'm not great at the internals of JVMs and such, so I may be glossing over some details. With the shell I have less idea - but it might be doing the same type of thing, and trying to create a new thread for each put? shrug I haven't played with it through that, so I'm not sure.
0

I encountered this error when I was using a HTablePool instance to get my HTableInterface instances, but after the utilization I forget to call the close() method on it.

Improve this answer

Comments

0

I also encountered the same issue and as explained by kosii above, the root cause was not closing the HTableInterface instance which I got from the HTablePool after utilization.

HTableInterface table = tablePool.getTable(tableName);
// Do the work
....
....
table.close()
Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.