I am getting java.lang.OutOfMemoryError: unable to create new native thread Threads exception while trying to run my go cd pipeline. When I ran top -H command I seen Threads: 6350 total, 1 running, 6345 sleeping, 3 stopped, 1 zombie! This is clearly waaaaaaaay to many especially I see other build severs only have Threads: 359 total, 1 running, 358 sleeping, 0 stopped, 0 zombie
I seen the following posts but I'm not sure they apply as they are code implementation and asking how to kill threads respectively:
java.lang.OutOfMemoryError: unable to create new native thread -XX:OnOutOfMemoryError doesn't work for java.lang.OutOfMemoryError: unable to create new native thread
My question is there a way to clean up all the sleeping threads? There are roughly about 20 times the amount?
Build logs:
22:37:48.789 [go] Job Started: 2020-08-25 22:37:48 UTC
22:37:48.789 [go] Start to prepare PATGUI-build/132/Execute_Script/1/build_job on edipgo-po-b4p.sys.comcast.net [/app/go-agent1]
22:37:48.794 [go] Start to update materials.
22:37:48.794 [go] Start updating files at revision 1c9e6a31946f62af1285305f338178d2a8fd13f2 from [email protected]:TPP-Provisioning/PortOutGui.git
22:37:48.801 Cloning into '/app/go-agent1/pipelines/PATGUI-build'...
22:38:00.435 [GIT] Fetching changes
22:38:01.640 [GIT] Performing git gc
22:38:01.743 [GIT] Reset working directory pipelines/PATGUI-build
22:38:01.743 [GIT] Cleaning all unversioned files in working copy
22:38:01.953 [GIT] Cleaning submodule configurations in .git/config
22:38:02.056 [GIT] Updating working copy to revision 1c9e6a31946f62af1285305f338178d2a8fd13f2
22:38:02.413 HEAD is now at 1c9e6a3 correcting object creation methods
22:38:02.517 [GIT] Removing modified files in submodules
22:38:02.716 [GIT] Updating git sub-modules
22:38:12.506 [GIT] Cleaning unversioned files and sub-modules
22:38:12.506 [GIT] Git sub-module status
22:38:12.583 51c608012d468edce285f7e60084b5dfa8e87a7a trunk (PAT-9.10@79-997-g51c6080)
22:38:12.584 [GIT] Cleaning all unversioned files in working copy
22:38:12.901 [go] Done.
22:38:12.902 [go] setting environment variable 'GO_SERVER_URL' to value 'https://edipgo-po-b1p.sys.comcast.net:8154/go'
22:38:12.902 [go] setting environment variable 'GO_TRIGGER_USER' to value 'cdvprov'
22:38:12.902 [go] setting environment variable 'GO_PIPELINE_NAME' to value 'PATGUI-build'
22:38:12.902 [go] setting environment variable 'GO_PIPELINE_COUNTER' to value '132'
22:38:12.902 [go] setting environment variable 'GO_PIPELINE_LABEL' to value '132'
22:38:12.902 [go] setting environment variable 'GO_STAGE_NAME' to value 'Execute_Script'
22:38:12.902 [go] setting environment variable 'GO_STAGE_COUNTER' to value '1'
22:38:12.902 [go] setting environment variable 'GO_JOB_NAME' to value 'build_job'
22:38:12.902 [go] setting environment variable 'GO_REVISION_SOURCE' to value '1c9e6a31946f62af1285305f338178d2a8fd13f2'
22:38:12.902 [go] setting environment variable 'GO_TO_REVISION_SOURCE' to value '1c9e6a31946f62af1285305f338178d2a8fd13f2'
22:38:12.902 [go] setting environment variable 'GO_FROM_REVISION_SOURCE' to value '1c9e6a31946f62af1285305f338178d2a8fd13f2'
22:38:12.910 [go] Start to build PATGUI-build/132/Execute_Script/1/build_job on edipgo-po-b4p.sys.comcast.net [/app/go-agent1]
22:38:12.911 [go] Current job status: passed.
22:38:12.911 [go] Start to execute task: Plugin with ID: script-executor.
22:38:12.952 [script-executor] OS detected: 'Linux'. Is Windows? false
22:38:12.963 [script-executor] Script written into '/app/go-agent1/pipelines/PATGUI-build/2f0ed637-c699-472a-8abd-d843898af4b9.sh'.
22:38:12.968 +++ date +%m
22:38:12.969 ++ BUILD_MONTH=08
22:38:12.970 +++ date +%m
22:38:12.971 ++ echo BUILD_MONTH=08
22:38:12.972 +++ cat pom.xml
22:38:12.972 +++ grep '<version>'
22:38:12.972 +++ head -1
22:38:12.973 +++ cut -d '>' -f2
22:38:12.973 +++ cut -d '<' -f1
22:38:12.975 ++ APP_VERSION=20.01
22:38:12.976 +++ cat pom.xml
22:38:12.976 +++ grep '<sprintNumber>'
22:38:12.976 +++ head -1
22:38:12.977 +++ cut -d '>' -f2
22:38:12.977 +++ cut -d '<' -f1
22:38:12.979 ++ SPRINT_NUMBER=01
22:38:12.979 ++ PATCH_NUMBER=132
22:38:12.979 ++ echo APP_VERSION=20.01
22:38:12.979 ++ echo PATCH_NUMBER=20.01.01.132
22:38:12.979 ++ /app/platform/apache-maven-3.0.3/bin/mvn -Dmaven.test.skip=true -DbuildNumber=132 clean install
22:38:13.208 Error occurred during initialization of VM
22:38:13.208 java.lang.OutOfMemoryError: unable to create new native thread
22:38:13.212 [script-executor] Script completed with exit code: 1.
22:38:13.226 [go] Current job status: failed.
22:38:13.229 [go] Start to create properties PATGUI-build/132/Execute_Script/1/build_job on edipgo-po-b4p.sys.comcast.net [/app/go-agent1]
22:38:13.229 [go] Start to upload PATGUI-build/132/Execute_Script/1/build_job on edipgo-po-b4p.sys.comcast.net [/app/go-agent1]
22:38:13.232 [go] The rule [PortOutApp/target/*ear] cannot match any resource under [pipelines/PATGUI-build]
22:38:13.246 [go] Uploading artifacts from /app/go-agent1/pipelines/PATGUI-build/build-vars to target/
22:38:13.266 [go] Uploading finished. Failed to upload [PortOutApp/target/*ear]
22:38:13.281 [go] Job completed PATGUI-build/132/Execute_Script/1/build_job on exmaple-po-b4p.sys.comcast.net [/app/go-agent1]
