This seems very similar to a recently closed #57143. We have spent quite a bit of time troubleshooting it though not able to find the root case. We had nginx and replace that by elb and both cases leads to same problem below. Please let me know what additional logs/info are needed and will provide. Any help/support/guidance is appreciated.
Env/Setup
Agent: OSX Mojave 10.14.5 Jenkins: Linux (Jenkins LTS 2.176.3) Java on agent: OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_212-b03) OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.212-b03, mixed mode)
Java on master: java.runtime.name OpenJDK Runtime Environment java.runtime.version 1.8.0_222-b10 java.version 1.8.0_222 java.vm.info mixed mode java.vm.name OpenJDK 64-Bit Server VM
netcat from slave towards master. $ nc -z jenkins.infrastructure.XXXXX.care 50000
Connection to jenkins.infrastructure.XXXXX.care port 50000 [tcp/*] succeeded!
ELB
kubectl -n infrastructure get svc infrastructure-nginx-ingress-internal-controller LoadBalancer 100.66.210.45 internal-ab39ee254472111e9aa5b0a4f0f93d47-951468028.us-east-1.elb.amazonaws.com 80:30664/TCP,443:30937/TCP,50000:32123/TCP 166d
Logs Aug 31, 2019 7:39:48 PM hudson.remoting.jnlp.Main createEngine
INFO: Setting up agent: ICL-Slave
Aug 31, 2019 7:39:48 PM hudson.remoting.jnlp.Main$CuiListener
INFO: Jenkins agent is running in headless mode.
Aug 31, 2019 7:39:48 PM hudson.remoting.Engine startEngine
INFO: Using Remoting version: 3.29
Aug 31, 2019 7:39:48 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using /Users/vmccbuild/jenkins/remoting as a remoting work directory
Aug 31, 2019 7:39:48 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among
Aug 31, 2019 7:39:48 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
Aug 31, 2019 7:39:48 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Agent discovery successful
Agent address: jenkins.infrastructure.xxxxx.care
Agent port: 50000
Identity: 6b:c4:c2:2b:ed:b7:da:bc:72:58:c8:83:dd:d0:67:46
Aug 31, 2019 7:39:48 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Aug 31, 2019 7:39:48 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to jenkins.infrastructure.xxxxx.care:50000
Aug 31, 2019 7:39:49 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP4-connect
Aug 31, 2019 7:39:59 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Protocol JNLP4-connect encountered an unexpected exception
java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Connection closed before acknowledgement sent
at org.jenkinsci.remoting.util.SettableFuture.get(SettableFuture.java:223)
at hudson.remoting.Engine.innerRun(Engine.java:614)
at hudson.remoting.Engine.run(Engine.java:474)
Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Connection closed before acknowledgement sent
at org.jenkinsci.remoting.protocol.impl.AckFilterLayer.onRecvClosed(AckFilterLayer.java:280)
at org.jenkinsci.remoting.protocol.FilterLayer.abort(FilterLayer.java:164)
at org.jenkinsci.remoting.protocol.impl.AckFilterLayer.access$000(AckFilterLayer.java:43)
at org.jenkinsci.remoting.protocol.impl.AckFilterLayer$1.run(AckFilterLayer.java:176)
at org.jenkinsci.remoting.protocol.IOHub$DelayedRunnable.run(IOHub.java:964)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
at java.lang.Thread.run(Thread.java:748)
Aug 31, 2019 7:39:59 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to jenkins.infrastructure.xxxxx.care:50000
Aug 31, 2019 7:39:59 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Server reports protocol JNLP4-plaintext not supported, skipping
Aug 31, 2019 7:39:59 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Server reports protocol JNLP3-connect not supported, skipping
Aug 31, 2019 7:39:59 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Server reports protocol JNLP2-connect not supported, skipping
Aug 31, 2019 7:39:59 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Server reports protocol JNLP-connect not supported, skipping
Aug 31, 2019 7:39:59 PM hudson.remoting.jnlp.Main$CuiListener error
SEVERE: The server rejected the connection: None of the protocols were accepted
java.lang.Exception: The server rejected the connection: None of the protocols were accepted
at hudson.remoting.Engine.onConnectionRejected(Engine.java:682)
at hudson.remoting.Engine.innerRun(Engine.java:639)
at hudson.remoting.Engine.run(Engine.java:474)
3 Answers
Solution to this problem: tunnel through the ELB as to get slave/master connected. How to do this?
Go to Jenkins Master → Nodes → → Advanced → Tunnel: ernal-aef67c013ca2d11e9b6ab02485d01370-111334679.us-west-1.elb.amazonaws.com:50000 (Above takes for granted that security group/inbound has also been properly configured)
This assumes that you have already configured agent port static to 50000 - how? Jenkins -> global security -> Agents -> TCP port for inbound agents -> Fixed: 50000
FYI in case it helps any – I just had to go through a big diag effort on my site with a bunch of windows JNLP agents that stopped working sometime in past few days with the same above failures/errors.
After a long set of rabbit trails - what I found in my case – the problem wound up being reverse DNS resolution stopped working (unrelated local problem – "It's always DNS", right?) - but that inability to reverse resolve the connecting client (windows box) was causing the JNLP server side to drop the connection.
Just wanted to add this here in case it helps anyone else with future similar issues if it happens to be related.
I have Jenkins behind a NGINX (using https) and the problem was that the Java agent uses TCP to communicate and not HTTPs. I tried to configure NGINX to do an SSL Pass-Thru but it didn't work for me.
Finally I end-up creating a Network Load Balancer (AWS) and using the dns name in the Advanced configuration of the Jenkins Node so that it uses this LB to go to the Jenkins server using TCP and not HTTP as the main protocol.
