2

We have a replication setup with a primary, secondary and arbiter node running mongodb community server v3.4.16.

We are using jasperserver to generate 100+ reports from mongodb database.

Recently we have started facing connection reset problems while connecting to mongodb server due to which reports are failing randomly.

The application (jasperserver) and mongodb are in same network and there' no firewall in between them.

Can someone provide any directions to further investigate the root cause so that it can be fixed?

2018-12-07 03:46:00,631 ERROR MongoDbConnectionFactory,quartzScheduler_Worker-34:85 - Cannot validate object; Caused by: Cannot test MongoDB connection; Caused by: Exception receiving message2018-12-07 03:46:00,631 ERROR MongoDbConnectionFactory,quartzScheduler_Worker-34:85 - Cannot validate object; Caused by: Cannot test MongoDB connection; Caused by: Exception receiving messagenet.sf.jasperreports.engine.JRException: Cannot test MongoDB connection; Caused by: Exception receiving message at com.jaspersoft.mongodb.connection.MongoDbConnection.test(MongoDbConnection.java:193) at com.jaspersoft.mongodb.connection.MongoDbConnectionFactory.validateObject(MongoDbConnectionFactory.java:83) at com.jaspersoft.mongodb.connection.MongoDbConnectionFactory.validateObject(MongoDbConnectionFactory.java:29) at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1165) at com.jaspersoft.mongodb.connection.MongoDbConnectionManager.borrowConnection(MongoDbConnectionManager.java:66) at com.jaspersoft.mongodb.jasperserver.MongoDbDataSourceService45.createConnection(MongoDbDataSourceService45.java:90) at com.jaspersoft.mongodb.jasperserver.MongoDbDataSourceService45.setReportParameterValues(MongoDbDataSourceService45.java:75) at com.jaspersoft.jasperserver.api.engine.jasperreports.service.impl.EngineServiceImpl.fillReport(EngineServiceImpl.java:1806) at com.jaspersoft.jasperserver.api.engine.jasperreports.service.impl.EngineServiceImpl$ReportFill.runWithDataSource(EngineServiceImpl.java:1143) at com.jaspersoft.jasperserver.api.engine.jasperreports.service.impl.EngineServiceImpl$ReportFill.runReport(EngineServiceImpl.java:1072) at com.jaspersoft.jasperserver.api.engine.jasperreports.service.impl.EngineServiceImpl$ReportRunnable.run(EngineServiceImpl.java:967) at com.jaspersoft.jasperserver.api.engine.jasperreports.service.impl.EngineServiceImpl$SynchronousExecutor.execute(EngineServiceImpl.java:945) at com.jaspersoft.jasperserver.api.engine.jasperreports.service.impl.EngineServiceImpl.fillReport(EngineServiceImpl.java:651) at com.jaspersoft.jasperserver.api.engine.jasperreports.service.impl.EngineServiceImpl.executeReportUnitRequest(EngineServiceImpl.java:2068) at com.jaspersoft.jasperserver.api.engine.jasperreports.domain.impl.ReportUnitRequest.execute(ReportUnitRequest.java:67) at com.jaspersoft.jasperserver.api.engine.jasperreports.service.impl.EngineServiceImpl.execute(EngineServiceImpl.java:529) at com.jaspersoft.jasperserver.api.engine.scheduling.quartz.ReportExecutionJob.runReport(ReportExecutionJob.java:836) at com.jaspersoft.jasperserver.api.engine.scheduling.quartz.ReportExecutionJob.executeReport(ReportExecutionJob.java:803) at com.jaspersoft.jasperserver.api.engine.scheduling.quartz.ReportExecutionJob.executeReport(ReportExecutionJob.java:704) at com.jaspersoft.jasperserver.api.engine.scheduling.quartz.ReportExecutionJob.executeAndSendReport(ReportExecutionJob.java:499) at com.jaspersoft.jasperserver.api.engine.scheduling.quartz.ReportExecutionJob.execute(ReportExecutionJob.java:248) at org.quartz.core.JobRunShell.run(JobRunShell.java:213) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)Caused by: com.mongodb.MongoSocketReadException: Exception receiving message at com.mongodb.connection.InternalStreamConnection.translateReadException(InternalStreamConnection.java:480) at com.mongodb.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:225) at com.mongodb.connection.UsageTrackingInternalConnection.receiveMessage(UsageTrackingInternalConnection.java:102) at com.mongodb.connection.DefaultConnectionPool$PooledConnection.receiveMessage(DefaultConnectionPool.java:435) at com.mongodb.connection.CommandProtocol.execute(CommandProtocol.java:112) at com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:159) at com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:286) at com.mongodb.connection.DefaultServerConnection.command(DefaultServerConnection.java:173) at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:215) at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:186) at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:178) at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:91) at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:84) at com.mongodb.operation.CommandReadOperation.execute(CommandReadOperation.java:55) at com.mongodb.Mongo.execute(Mongo.java:772) at com.mongodb.Mongo$2.execute(Mongo.java:759) at com.mongodb.DB.executeCommand(DB.java:653) at com.mongodb.DB.command(DB.java:423) at com.mongodb.DB.command(DB.java:439) at com.mongodb.DB.command(DB.java:381) at com.jaspersoft.mongodb.connection.MongoDbConnection.test(MongoDbConnection.java:188) ... 22 moreCaused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:210) at java.net.SocketInputStream.read(SocketInputStream.java:141) at com.mongodb.connection.SocketStream.read(SocketStream.java:85) at com.mongodb.connection.InternalStreamConnection.receiveResponseBuffers(InternalStreamConnection.java:491) at com.mongodb.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:221) ... 41 more

Improve this question
3
  • Do you have MongoDB logs for the time - it could be a change in the replicaset (election of a new primary) that caused the connection to be interrupted. Commented Dec 10, 2018 at 11:13
  • @Ross: We investigated that but there was no such event at that time. We tried switching manually primary and secondary nodes but problem still persists. Its appearing randomly. We have also tried setting net.ipv4.tcp_keepalive_time to 300 from default 7200. Then we have also tried tuning connectionTimeout and socket timeouts in dsn with 1 hour from default 0,0 but there was no logs in application side with connection timeout issue. Commented Dec 11, 2018 at 8:36
  • 1
    59 The javadoc for SocketException states that it is: Thrown to indicate that there is an error in the underlying protocol such as a TCP error In your case it seems that the connection has been closed by the server end of the connection. To find out why I'd suggest increasing your log verbosity in MongoD Commented Dec 12, 2018 at 9:50

1 Answer 1

Reset to default
1

I had a similar problem using the native java mongodb driver. This was the problem - net.ipv4.tcp_keepalive_time = 7200 which meant that the MONGO Server would sever the connection post 7200 of life & maxConnectionIdleTime was not set(infinite) in this case. Post the termination the connection would still remain in the connection pool but was a stale one and would create this exception.

I would recommend you to - -> set socketKeepAlive to true (which is by default true in the latest versions of the native driver) -> set maxConnectionIdleTime < net.ipv4.tcp_keepalive_time in this way your JVM would reap the connection and not the other way around.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.