java.lang.OutOfMemoryError: Java heap space Hadoop Ubuntu

Question

I'm using Hadoop in my application, but just before the program exits I get this error java.lang.OutOfMemoryError: Java heap space I already modified the mapred-site.xml and added this property to it

<property> <name>mapred.child.java.opts</name> <value>-Xmx4096m</value> </property>

but still the exception appears. I used this command in terminal: java -XX:+PrintFlagsFinal -version | grep -iE 'HeapSize|PermSize|ThreadStackSize' and this was the result:

uintx AdaptivePermSizeWeight                    = 20              {product}           
     intx CompilerThreadStackSize                   = 0               {pd product}        
    uintx ErgoHeapSizeLimit                         = 0               {product}           
    uintx HeapSizePerGCThread                       = 87241520        {product}           
    uintx InitialHeapSize                          := 1054841728      {product}           
    uintx LargePageHeapSizeThreshold                = 134217728       {product}           
    uintx MaxHeapSize                              := 16877879296     {product}           
    uintx MaxPermSize                               = 174063616       {pd product}        
    uintx PermSize                                  = 21757952        {pd product}        
     intx ThreadStackSize                           = 1024            {pd product}        
     intx VMThreadStackSize                         = 1024            {pd product}        
java version "1.6.0_31"
OpenJDK Runtime Environment (IcedTea6 1.13.3) (6b31-1.13.3-1ubuntu1~0.12.04.2)
OpenJDK 64-Bit Server VM (build 23.25-b01, mixed mode)

If anyone could please advise how to fix this issue please.

The max heap size appears to be nearly 16 GiB, which is well over the 4GiB limit you set. Perhaps it has something to do with that? — Nathan Tuggy
– Nathan Tuggy, Commented Feb 19, 2015 at 1:15
so you think I should increase the heap size of hadoop to be 16Gb? — Tak
– Tak, Commented Feb 19, 2015 at 1:20
Possibly, but it might be better to track down why it's using so much. — Nathan Tuggy
– Nathan Tuggy, Commented Feb 19, 2015 at 1:22
Generally when you experience a outofmemory exception you have a leak and it is not your computer it is all about your code — Ya Wang
– Ya Wang, Commented Feb 19, 2015 at 1:25

Tom Lima · Accepted Answer · 2015-02-19 01:37:46Z

1

Your problem is a memory leak.

You should consider reviewing your code to see what's causing that resource leak. Usually, it's caused by instances where the GC is not able to remove the data from your memory.

answered Feb 19, 2015 at 1:37

Tom Lima

508 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Tak Over a year ago

but I think my code is working fine as I'm expecting it to use lots of memory due to large amount of processing,... and my computer has 32GB memory

Collectives™ on Stack Overflow

java.lang.OutOfMemoryError: Java heap space Hadoop Ubuntu

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related