How to resolve java.lang.outOfMemoryError in ThreadPoolExecutor program?

Question

I am running a ThreadPoolExecutor program which creates 1000 url and send to executor service.

Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method)
    at java.lang.Thread.start(Thread.java:673)
    at java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:681)
    at java.util.concurrent.ThreadPoolExecutor.addIfUnderCorePoolSize(ThreadPoolExecutor.java:706)
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:650)
    at MyProgramName.main(MyProgramName.java:175)
Aug 16, 2014 8:46:20 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->MyURLURLURLURLRURL Connection reset
Aug 16, 2014 8:46:20 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->MyURLURLURLURLRURL Connection reset
Aug 16, 2014 8:46:20 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->MyURLURLURLURLRURL
Aug 16, 2014 8:46:20 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->MyURLURLURLURLRURL
2014-08-16 20:46:21.341 java[5100:d07] java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method)
    at java.lang.Thread.start(Thread.java:673)
    at sun.awt.AWTAutoShutdown.activateBlockerThread(AWTAutoShutdown.java:312)
    at sun.awt.AWTAutoShutdown.setToolkitBusy(AWTAutoShutdown.java:232)
    at sun.awt.AWTAutoShutdown.notifyToolkitThreadBusy(AWTAutoShutdown.java:118)
2014-08-16 20:46:21.342 java[5100:d07] (
    0   CoreFoundation                      0x00007fff9127825c __exceptionPreprocess + 172
    1   libobjc.A.dylib                     0x00007fff9031ce75 objc_exception_throw + 43
    2   CoreFoundation                      0x00007fff91277e09 -[NSException raise] + 9
    3   JavaNativeFoundation                0x000000010c754c3f JNFCallStaticVoidMethod + 213
    4   libawt.jnilib                       0x0000000115fb8151 setBusy + 53
    5   CoreFoundation                      0x00007fff911a8d67 __CFRUNLOOP_IS_CALLING_OUT_TO_AN_OBSERVER_CALLBACK_FUNCTION__ + 23
    6   CoreFoundation                      0x00007fff911a8cd7 __CFRunLoopDoObservers + 391
    7   CoreFoundation                      0x00007fff9119a608 __CFRunLoopRun + 1368
    8   CoreFoundation                      0x00007fff91199e75 CFRunLoopRunSpecific + 309
    9   HIToolbox                           0x00007fff8fdd9a0d RunCurrentEventLoopInMode + 226
    10  HIToolbox                           0x00007fff8fdd97b7 ReceiveNextEventCommon + 479
    11  HIToolbox                           0x00007fff8fdd95bc _BlockUntilNextEventMatchingListInModeWithFilter + 65
    12  AppKit                              0x00007fff8ac5d24e _DPSNextEvent + 1434
    13  AppKit                              0x00007fff8ac5c89b -[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:] + 122
    14  libawt.jnilib                       0x0000000115fb987f -[NSApplicationAWT nextEventMatchingMask:untilDate:inMode:dequeue:] + 124
    15  AppKit                              0x00007fff8ac5099c -[NSApplication run] + 553
    16  libawt.jnilib                       0x0000000115fb7f30 +[AWTStarter startAWT:] + 1495
    17  libawt.jnilib                       0x0000000115fb78aa -[CPerformer perform] + 93
    18  Foundation                          0x00007fff8c64313e __NSThreadPerformPerform + 229
    19  CoreFoundation                      0x00007fff911a95b1 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 17
    20  CoreFoundation                      0x00007fff9119ac62 __CFRunLoopDoSources0 + 242
    21  CoreFoundation                      0x00007fff9119a3ef __CFRunLoopRun + 831
    22  CoreFoundation                      0x00007fff91199e75 CFRunLoopRunSpecific + 309
    23  java                                0x000000010be86843 java + 18499
    24  java                                0x000000010be8629a java + 17050
    25  java                                0x000000010be83a98 java + 6808
    26  ???                                 0x0000000000000005 0x0 + 5
)
Aug 16, 2014 8:46:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->MyURLURLURLURLRURL Connection reset
Aug 16, 2014 8:46:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->MyURLURLURLURLRURL
Aug 16, 2014 8:46:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->MyURLURLURLURLRURL Connection reset
Aug 16, 2014 8:46:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->MyURLURLURLURLRURL
Aug 16, 2014 8:46:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->MyURLURLURLURLRURL Connection reset
Aug 16, 2014 8:46:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->MyURLURLURLURLRURL
Aug 16, 2014 8:46:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->MyURLURLURLURLRURL Connection reset
Aug 16, 2014 8:46:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->MyURLURLURLURLRURL Connection reset
Aug 16, 2014 8:46:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->MyURLURLURLURLRURL

I already made some research in internet and stackoverflow. Some answers say I need to reduce stack size. Some say I need to increase the ulimit using ulimit -a. I am totally confused. I am confused with those answers since they are not specific to ThreadExecutor. Please help me

I run the code in Mac OS, i7 Quad core processor, 16 gb RAM, 1 Tb HD

Code of my program is here in this question Why does the following executor service java Thread program doesn't shut down?

How much memory does your system have, and how much memory did you allocate to the program (-Xms and -Xmx)? — Jim Garrison
– Jim Garrison, Commented Aug 17, 2014 at 4:11
My system has 1Tb hard disk. May I know how to find the memory allocated for my program. I don't know how to find that. Please hel pme. — sofs1
– sofs1, Commented Aug 17, 2014 at 4:12

Stephen C · Accepted Answer · 2014-08-17 04:13:52Z

6

The real problem is that you are trying to create too many threads using too much stack memory.

You could reduce the default size for the thread stacks (as suggested), but that isn't a complete solution. (When you decide to run your program on more URLs, you will run into the same problem again. And eventually you will hit "the wall" where you can't reduce the stack size any further; e.g. because you are getting StackOverflowError exceptions.)

A better solution is to NOT create so many threads. You've configured your program to use an infinite thread pool. That is frankly ridiculous. Change this:

ThreadPoolExecutor executor = 
      new ThreadPoolExecutor(Integer.MAX_VALUE, Integer.MAX_VALUE, 
                             20, TimeUnit.MILLISECONDS, blockingQueue);

to something more realistic:

ThreadPoolExecutor executor = 
      new ThreadPoolExecutor(100, 100, 
                             20, TimeUnit.MILLISECONDS, blockingQueue);

Beyond a certain point, adding extra threads won't make an application go any faster. Indeed, it will often start to make the application go slower, and sometimes catastrophically slower.

answered Aug 17, 2014 at 4:13

Stephen C

723k95 gold badges849 silver badges1.3k bronze badges

Sign up to request clarification or add additional context in comments.

16 Comments

sofs1 Over a year ago

Wow. That is working like a charm. May I know why changing this works. I also read this in ThreadPoolExecutor documentation, but it was ambiguous and I thought having the poolsize as Integer.MAX_VALUE would make things easier. But I get this error after few minutes Exception in thread "pool-1-thread-990" java.lang.OutOfMemoryError: Java heap space at java.io.BufferedReader.<init>(BufferedReader.java:80) at java.io.BufferedReader.<init>(BufferedReader.java:91) at MyProgram$MyRunnable.run(MyProgram.java:232) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.jav

sofs1 Over a year ago

Also if I terminate the program when I get "pool-1-thread-990" java.lang.OutOfMemoryError: Java heap space at java.io.BufferedReader.<init> will my program won't process the urls in pipeline or it will still be on hold?

Stephen C Over a year ago

"What it the optimal number" - There is no general answer to that. Make it a tuning parameter ... and tune it.

sofs1 Over a year ago

I guess answering here would help several other users.

Stack Exchange Broke The Law Over a year ago

@user3705478 you should try to ask one question per question.

|

Flexo - Save the data dump · Accepted Answer · 2015-05-06 21:27:13Z

You face the “java.lang.OutOfMemoryError: Unable to create new native thread” whenever JVM is asking a new thread from the OS. Whenever the underlying OS cannot allocate a new native thread, this OutOfMemoryError will be thrown. The exact limit for native threads is very platform-dependent thus we recommend to find your limits by running a small test to find out your platform-specific limit to threads. But, in general, the situation causing java.lang.OutOfMemoryError: Unable to create new native thread goes through the following phases:

A new Java thread is requested by an application running inside the JVM
JVM native code proxies the request to create a new native thread to the OS
OS tries to create a new native thread which requires memory to be allocated to the thread
The OS will refuse native memory allocation either because the 32-bit Java process size has depleted its memory address space – e.g. (2-4) GB process size limit has been hit – or the virtual memory of the OS has been fully depleted
The java.lang.OutOfMemoryError: Unable to create new native thread error is thrown

In your case, the best way to overcome the issue is definitely follow the suggested path of limiting the thread creation.

Collectives™ on Stack Overflow

How to resolve java.lang.outOfMemoryError in ThreadPoolExecutor program?

2 Answers 2

16 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

16 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related