Repository navigation
CitGM jobs don't end on AIX #900
Description
I took test-osuosl-aix61-ppc64_be-2 offline in Jenkins (so I guess that's "offline" rather than offline) as it was failing repeatedly with:
Started by upstream project "node-test-commit-aix" build number 8918
originally caused by:
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on test-osuosl-aix61-ppc64_be-2 (aix61-ppc64) in workspace /home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64
Cloning the remote Git repository
Cloning repository https://github.com/nodejs/node.git
> git init /home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64 # timeout=30
ERROR: Error cloning remote repo 'origin'
hudson.plugins.git.GitException: Could not init /home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$5.execute(CliGitAPIImpl.java:717)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:511)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
at hudson.remoting.UserRequest.perform(UserRequest.java:152)
at hudson.remoting.UserRequest.perform(UserRequest.java:50)
at hudson.remoting.Request$2.run(Request.java:332)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1157)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:627)
at hudson.remoting.Engine$1$1.run(Engine.java:85)
at java.lang.Thread.run(Thread.java:809)
at ......remote call to Channel to /140.211.9.100(Native Method)
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1545)
at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
at hudson.remoting.Channel.call(Channel.java:830)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
at sun.reflect.GeneratedMethodAccessor539.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
at com.sun.proxy.$Proxy79.execute(Unknown Source)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1075)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1115)
at hudson.scm.SCM.checkout(SCM.java:496)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1281)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:604)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
at hudson.model.Run.execute(Run.java:1728)
at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:405)
Caused by: hudson.plugins.git.GitException: Error performing command: git init /home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1931)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1892)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1888)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1533)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$5.execute(CliGitAPIImpl.java:715)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:511)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
at hudson.remoting.UserRequest.perform(UserRequest.java:152)
at hudson.remoting.UserRequest.perform(UserRequest.java:50)
at hudson.remoting.Request$2.run(Request.java:332)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1157)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:627)
at hudson.remoting.Engine$1$1.run(Engine.java:85)
at java.lang.Thread.run(Thread.java:809)
Caused by: java.io.IOException: Cannot run program "git" (in directory "/home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64"): error=11, Resource temporarily unavailable
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1059)
at hudson.Proc$LocalProc.<init>(Proc.java:245)
at hudson.Proc$LocalProc.<init>(Proc.java:214)
at hudson.Launcher$LocalLauncher.launch(Launcher.java:846)
at hudson.Launcher$ProcStarter.start(Launcher.java:384)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1920)
... 16 more
Caused by: java.io.IOException: error=11, Resource temporarily unavailable
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:212)
at java.lang.ProcessImpl.start(ProcessImpl.java:164)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1040)
... 21 more
ERROR: Error cloning remote repo 'origin'
Run condition [Always] enabling perform for step [[]]
TAP Reports Processing: START
Looking for TAP results report in workspace using pattern: *.tap
Did not find any matching files. Setting build result to FAILURE.
Checking ^not ok
Jenkins Text Finder: File set '*.tap' is empty
Sending e-mails to: [email protected] [email protected] [email protected]
Notifying upstream projects of job completion
Finished: FAILUREping @mhdawson
Activity
gibfahn commentedon Oct 2, 2017
seems to have fixed it. We should look at adding cleanup to the citgm jobs.
gibfahn commentedon Oct 2, 2017
@Trott btw it's worth cc'ing @nodejs/platform-aix
gibfahn commentedon Oct 2, 2017
First build that ran on it seems to be going okay, so closing.
https://ci.nodejs.org/job/node-test-commit-aix/nodes=aix61-ppc64/8923/console
refack commentedon Oct 2, 2017
BTW: @gibfahn did you clean the ramdrive? I think the jobs go zombie when the ramdrive if full.
gibfahn commentedon Oct 2, 2017
@refack it was only 10% full.
refack commentedon Oct 2, 2017
Puff goes my hypothesis 🤔
refack commentedon Oct 2, 2017
So how about adding
ps -ef | grep citgm | awk '{print $2}' | xargs killunconditionally to the end of the job?gibfahn commentedon Oct 2, 2017
SGTM
refack commentedon Oct 23, 2017
@Trott and I just found 88 unterminated node processes on
test-osuosl-aix61-ppc64_be-1I'm reopening, since we need to either kill the processes on cigtm job end, or take the AIXes out of the citgm roster.
[-]test-osuosl-aix61-ppc64_be-2 failing to build, temporarily taken offline[/-][+]CitGM jobs don't end on AIX[/+]mhdawson commentedon Oct 24, 2017
@gdams I know you did a lot of the work to get AIX into citgm, do you think you could add the kill at citgm job end ?
maclover7 commentedon May 24, 2018
These jobs are able to finish now