Infinispan: java.lang.OutOfMemoryError: Java heap space

Question

I'm running a JBoss 7 ha cluster with 4 nodes across 2 servers. Since a few weeks, the following exception is thrown sporadically, sometimes 2 or 3 times day, then one 1 oder 2 times a week. It looks like that the chance that the exceptions is thrown, is higher when there a more users active.

 21:38:24,337 ERROR [org.infinispan.remoting.ReplicationQueue] (Scheduled-replicationQueue-thread-0) [, , ] ISPN000072: Failed replicating 3 elements in replication queueorg.infinispan.CacheException: java.lang.OutOfMemoryError: Java heap space
      at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:166) [infinispan-core-5.1.0.BETA5.jar:]
      at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:181) [infinispan-core-5.1.0.BETA5.jar:]
      at org.infinispan.remoting.ReplicationQueueImpl.flush(ReplicationQueueImpl.java:159) [infinispan-core-5.1.0.BETA5.jar:]
      at org.infinispan.remoting.ReplicationQueueImpl$1.run(ReplicationQueueImpl.java:112) [infinispan-core-5.1.0.BETA5.jar:]
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) [:1.6.0_29]
      at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) [:1.6.0_29]
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) [:1.6.0_29]
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) [:1.6.0_29]
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) [:1.6.0_29]
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) [:1.6.0_29]
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_29]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_29]
      at java.lang.Thread.run(Thread.java:662) [:1.6.0_29] Caused by: java.lang.OutOfMemoryError: Java heap space
      at org.infinispan.io.ExposedByteArrayOutputStream.write(ExposedByteArrayOutputStream.java:104) [infinispan-api-5.1.0.BETA5.jar:]
      at java.io.FilterOutputStream.write(FilterOutputStream.java:60) [:1.6.0_29]
      at java.io.FilterOutputStream.write(FilterOutputStream.java:108) [:1.6.0_29]
      at org.jboss.marshalling.SimpleDataOutput.write(SimpleDataOutput.java:83) [jboss-marshalling-1.3.4.GA.jar:]
      at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:113) [infinispan-core-5.1.0.BETA5.jar:]
      at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:66) [infinispan-core-5.1.0.BETA5.jar:]
      at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:389) [infinispan-core-5.1.0.BETA5.jar:]
      at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145)
      at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62) [jboss-marshalling-1.3.4.GA.jar:]
      at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119) [jboss-marshalling-1.3.4.GA.jar:]
      at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectToObjectStream(AbstractJBossMarshaller.java:59) [infinispan-commons-5.1.0.BETA5.jar:]
      at org.infinispan.marshall.VersionAwareMarshaller.objectToBuffer(VersionAwareMarshaller.java:91) [infinispan-core-5.1.0.BETA5.jar:]
      at org.infinispan.marshall.AbstractMarshaller.objectToBuffer(AbstractMarshaller.java:57) [infinispan-commons-5.1.0.BETA5.jar:]
      at org.infinispan.marshall.AbstractDelegatingMarshaller.objectToBuffer(AbstractDelegatingMarshaller.java:84) [infinispan-commons-5.1.0.BETA5.jar:]
      at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectToBuffer(MarshallerAdapter.java:45) [infinispan-core-5.1.0.BETA5.jar:]
      at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$ReplicationTask.marshallCall(CommandAwareRpcDispatcher.java:211) [infinispan-core-5.1.0.BETA5.jar:]
      at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$ReplicationTask.call(CommandAwareRpcDispatcher.java:232) [infinispan-core-5.1.0.BETA5.jar:]
      at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:110) [infinispan-core-5.1.0.BETA5.jar:]
      at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:411) [infinispan-core-5.1.0.BETA5.jar:]
      at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:153) [infinispan-core-5.1.0.BETA5.jar:]
      ... 12 more

I've already tried to increase the JVM heap and permgen size in host.xml up to 8/16 GB without success.

 <jvms>
      <jvm name="default">
           <heap size="16g" max-size="16g"/>
           <permgen size="8g" max-size="8g"/>
      </jvm>
 </jvms>

Any ideas? Or is it possible to increase the Infispan heap size?

Tips only: try FindBugs. For instance to find an inner class being serialized (taking with it the outer class, and recreating that as null). Profile it locally, heap. Maybe JavaMelody. JSF has some defaults that per user can come costly. — Joop Eggen
– Joop Eggen, Commented Jun 19, 2013 at 10:07
Give JBoss EAP 6.1.0 a go! It uses a more recent JBoss AS 7 version underneath, which probably has your issue solved... jboss.org/products/eap.html — Galder Zamarreño
– Galder Zamarreño, Commented Jun 21, 2013 at 16:49

Aldjinn · Accepted Answer · 2013-10-23 05:36:33Z

1

This topic is already a few months old, but I was able to resolve this problem myself.

The issue was related to a specific application that stores data inside the user's session. Under some special and very rare circumstances, the amount of data was still too much and the time for serialization and de-serialization increased. For example, a normal user session had 10 custom attributes added, the sessions causing the problem had more than 10000 attributes added.

The solution (in my case!) was not to store this attributes inside the user's session anymore but to add them to a distributed ehcache.

This tool was a great help for discovering the problem: http://messadmin.sourceforge.net/

answered Oct 23, 2013 at 5:36

Aldjinn

7457 silver badges15 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Artiom Gourevitch · Accepted Answer · 2013-06-19 13:01:16Z

0

There is a memory leak (most likely) or a huge invocation happening.

Stacktrace in case of memory leak may be not related to the memory leak. It shows just what is a straw that broke the camel's back.

Adding JVM arg of "-XX:+HeapDumpOnOutOfMemoryError" to get a heap dump may help to find the real cause.

Also Perm Gen may be harmlessly decreased from 8gb to few hundreds of MBs, it is not related to out of memory error that the system experiences.

answered Jun 19, 2013 at 13:01

Artiom Gourevitch

7961 gold badge11 silver badges22 bronze badges

3 Comments

Aldjinn Over a year ago

OK, I will try this. But when the exception is thrown, the JBoss processes are still up and running and there is no need to restart them. Only a few minutes before the exception is thrown, users will be affeced by slow performance and some connection timeouts.

Artiom Gourevitch Over a year ago

I wouldn't suggest running in production environment with this kind of error happening.

Aldjinn Over a year ago

Just shutting down the production environment without fixing the issue isn't an option. ;)

Collectives™ on Stack Overflow

Infinispan: java.lang.OutOfMemoryError: Java heap space

2 Answers 2

Comments

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related