ss/config file not working but ssh works

I am trying to login to final_host from localhost via an intermediate hop, i.e:

localhost -> hop -> final_host

This works:

localhost:~$ ssh -t user@hop "ssh user@final_host"

This also works:

localhost:~$ ssh user@hop
hop:~$ ssh user@final_host
finalhost:~$

But this doesn't:

localhost:~$ ssh -J user@hop_ip "ssh user@final_host_ip" -vvv
OpenSSH_7.6p1, LibreSSL 2.6.2
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 48: Applying options for *
debug1: Setting implicit ProxyCommand from ProxyJump: ssh -l user -vvv -W '[%h]:%p' hop_ip
debug1: Executing proxy command: exec ssh -l user -vvv -W '[final_host_ip]:22' hop_ip
debug1: key_load_public: No such file or directory
debug1: identity file /Users/user/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/user/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/user/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/user/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/user/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/user/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/user/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/user/.ssh/id_ed25519-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.6
debug1: permanently_drop_suid: 501
OpenSSH_7.6p1, LibreSSL 2.6.2
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 48: Applying options for *
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to hop_ip port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /Users/user/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/user/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/user/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/user/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/user/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/user/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/user/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/user/.ssh/id_ed25519-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.6
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.4
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.4 pat OpenSSH* compat 0x04000000
debug3: fd 5 is O_NONBLOCK
debug1: Authenticating to hop_ip:22 as 'user'
debug3: hostkeys_foreach: reading file "/Users/user/.ssh/known_hosts"
debug3: record_hostkey: found key type ED25519 in file /Users/user/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from hop_ip
debug3: order_hostkeyalgs: prefer hostkeyalgs: ,ssh-ed25519
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ,ssh-ed25519,,,,,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: ,aes128-ctr,aes192-ctr,aes256-ctr,,
debug2: ciphers stoc: ,aes128-ctr,aes192-ctr,aes256-ctr,,
debug2: MACs ctos: ,,,,,,,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: ,,,,,,,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,,zlib
debug2: compression stoc: none,,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: ,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ssh-ed25519
debug2: ciphers ctos: ,aes128-ctr,aes192-ctr,aes256-ctr,,
debug2: ciphers stoc: ,aes128-ctr,aes192-ctr,aes256-ctr,,
debug2: MACs ctos: ,,,,,,,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: ,,,,,,,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,
debug2: compression stoc: none,
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm:
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: MAC: <implicit> compression: none
debug1: kex: client->server cipher: MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ssh-ed25519 SHA256:c18UMgj7nokTZJHMGnbsOgxDHlIZc2r184efDHtoTLE
debug3: hostkeys_foreach: reading file "/Users/user/.ssh/known_hosts"
debug3: record_hostkey: found key type ED25519 in file /Users/user/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from hop_ip
debug1: Host 'hop_ip' is known and matches the ED25519 host key.
debug1: Found key in /Users/user/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug2: key: server_key_folder/.ssh/id_rsa (0x7f953cc00d70), agent
debug2: key: /Users/user/.ssh/id_rsa (0x0)
debug2: key: /Users/user/.ssh/id_dsa (0x0)
debug2: key: /Users/user/.ssh/id_ecdsa (0x0)
debug2: key: /Users/user/.ssh/id_ed25519 (0x0)
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:IhOfM2s2i/vFgY/Mj962CoNez631HDIMDRjxFvDhOEI server_key_folder/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: pkalg rsa-sha2-512 blen 279
debug2: input_userauth_pk_ok: fp SHA256:IhOfM2s2i/vFgY/Mj963CoNez631HDIMDRvxFvDhOEI
debug3: sign_and_send_pubkey: RSA SHA256:IhOfM2s2i/vFgY/Mj963CoNez631HDIMDRvxFvDhOEI
debug3: send packet: type 50
debug3: receive packet: type 52
debug1: Authentication succeeded (publickey).
Authenticated to hop_ip ([hop_ip]:22).
debug3: ssh_init_stdio_forwarding: final_host_ip:22
debug1: channel_connect_stdio_fwd final_host_ip:22
debug1: channel 0: new [stdio-forward]
debug2: fd 7 setting O_NONBLOCK
debug2: fd 8 setting O_NONBLOCK
debug1: getpeername failed: Bad file descriptor
debug3: send packet: type 90
debug2: fd 5 setting TCP_NODELAY
debug3: ssh_packet_set_tos: set IP_TOS 0x10
debug1: Requesting
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: network
debug3: receive packet: type 80
debug1: client_input_global_request: rtype want_reply 0
debug3: receive packet: type 91
debug2: channel_input_open_confirmation: channel 0: callback start
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 2097152 rmax 32768
debug1: ssh_exchange_identification: \033]1337;PushKeyLabels=fish_%self\033]1337;SetKeyLabel=F2=GitSSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.4
debug1: ssh_exchange_identification:
debug1: ssh_exchange_identification: \024n��3�\030d\206\206�\235A6
debug1: ssh_exchange_identification: ,aes128-ctr,aes192-ctr,aes256-ctr,,
debug1: ssh_exchange_identification: sh.com,,,,,,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 0 rtype reply 1
debug3: send packet: type 100
debug3: receive packet: type 96
debug2: channel 0: rcvd eof
debug2: channel 0: output open -> drain
debug2: channel 0: obuf empty
debug2: channel 0: close_write
debug2: channel 0: output drain -> closed
debug1: channel 0: FORCE input drain
debug2: channel 0: ibuf empty
debug2: channel 0: send eof
debug3: send packet: type 96
debug2: channel 0: input drain -> closed
debug3: receive packet: type 97
debug2: channel 0: rcvd close
debug3: channel 0: will not send data after close
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug1: stdio forwarding: done
ssh_exchange_identification: Connection closed by remote host

It gets stuck for a while before closing the connection. Using a config file with ProxyJump, ProxyCommand -W or ProxyCommand + nc also doesn't work and gets stuck at the same point. Additionally this stopped working at some point, I was able to use ProxyCommand in the past and it suddenly stopped working one day without apparent config change from my side. What could be going on here and what can I do to debug further?

EDIT: I should point out that other users of the same system are able to login without issue via config file so the problem is most likely local.

1 Answer

Authentication to final_host succeeds, but you get this error: getpeername failed: Bad file descriptor.

The ssh daemon on the final_host and hop should connect file descriptors for you to use (stdin, stdout, stderr) on your localhost. This fails.

The function mentioned should point you in the right direction. You are using an IP address that might not have a reverse dns entry, nor a line in /etc/hosts.