Sometimes when IP addresses change or servers get moved around the known_hosts file in the .ssh hidden directory gets updated with two keys for the same IP. In that case, when you try to ssh to the sever again, you get a message about an offending key:

Warning: the ECDSA host key for 'server.com' differs from the key for the IP address '118.82.143.119'
Offending key for IP in /home/debian/.ssh/known_hosts:30
Matching host key in /home/debian/.ssh/known_hosts:38
Are you sure you want to continue connecting (yes/no)?

The offending key can be removed easily using sed:

sed -i 30d /home/debian/.ssh/known_hosts

which deletes line 30 from the known_hosts file.

For rsa:

ssh-keygen -t rsa

For dsa:

ssh-keygen -t dsa

As pointed out by Josh Enders, ssh on OSX performs a series of calls to close triggered by:

closefrom(STDERR_FILENO + 1);

This can be checked with dtruss:

sh-3.2# dtruss /usr/bin/ssh
# ...
stat64("/usr/lib/system/libcorecrypto.dylib\0", 0x7FFF570D4828, 0x1)		 = 0 0
getpid(0x7FFF925948F8, 0x7FFF570D48B8, 0x1)		 = 79341 0
close(0x7DE)		 = -1 Err#9
close(0x7DF)		 = -1 Err#9
close(0x7E0)		 = -1 Err#9
close(0x7E1)		 = -1 Err#9
close(0x7E2)		 = -1 Err#9
close(0x7E3)		 = -1 Err#9
close(0x7E4)		 = -1 Err#9
close(0x7E5)		 = -1 Err#9
close(0x7E6)		 = -1 Err#9
close(0x7E7)		 = -1 Err#9
close(0x7E8)		 = -1 Err#9
close(0x7E9)		 = -1 Err#9
close(0x7EA)		 = -1 Err#9
close(0x7EB)		 = -1 Err#9
close(0x7EC)		 = -1 Err#9
close(0x7ED)		 = -1 Err#9
close(0x7EE)		 = -1 Err#9
close(0x7EF)		 = -1 Err#9
close(0x7F0)		 = -1 Err#9
close(0x7F1)		 = -1 Err#9
 

From the ssh.c source-file, this function is called in main at the following code-segment:

	__progname = ssh_get_progname(av[0]);
 
#ifndef HAVE_SETPROCTITLE
	/* Prepare for later setproctitle emulation */
	/* Save argv so it isn't clobbered by setproctitle() emulation */
	saved_av = xcalloc(ac + 1, sizeof(*saved_av));
	for (i = 0; i < ac; i++)
		saved_av[i] = xstrdup(av[i]);
	saved_av[i] = NULL;
	compat_init_setproctitle(ac, av);
	av = saved_av;
#endif
 
	/*
	 * Discard other fds that are hanging around. These can cause problem
	 * with backgrounded ssh processes started by ControlPersist.
	 */
	closefrom(STDERR_FILENO + 1);
 
	/*
	 * Save the original real uid.  It will be needed later (uid-swapping
	 * may clobber the real uid).
	 */
	original_real_uid = getuid();
	original_effective_uid = geteuid();

The closeform function can be nop-ed out using a disassembler:

000000010000d792 BF03000000                      mov        edi, 0x3
; nop closeform()
000000010000d797 E8D7510200                      call       sub_100032973 
000000010000d79c E8DD8E0400                      call       imp___stubs__getuid

resulting in:

000000010000d792 BF03000000                      mov        edi, 0x3
000000010000d797 90                              nop        
000000010000d798 90                              nop        
000000010000d799 90                              nop        
000000010000d79a 90                              nop        
000000010000d79b 90                              nop        
000000010000d79c E8DD8E0400                      call       imp___stubs__getuid

which avoids the close spam when ssh starts:

sh-3.2# dtruss ./ssh
# ...
open("/dev/null\0", 0x2, 0x7FFF551ECBC0)		 = 3 0
close(0x3)		 = 0 0
getuid(0x3, 0x2, 0x7FFF551ECBC0)		 = 0 0
geteuid(0x3, 0x2, 0x7FFF551ECBC0)		 = 0 0
seteuid(0x0, 0x2, 0x7FFF551ECBC0)		 = 0 0
shm_open(0x7FFF8475DCCA, 0x0, 0x0)		 = 3 0
mmap(0x0, 0x1000, 0x1, 0x1, 0x3, 0x0)		 = 0x10AAE6000 0
close_nocancel(0x3)		 = 0 0
open_nocancel("/etc/.mdns_debug\0", 0x0, 0x0)		 = -1 Err#2
issetugid(0x7FFF88B67E15, 0x0, 0x0)		 = 0 0
issetugid(0x7FFF88B67E15, 0x0, 0x0)		 = 0 0

Or, on one line:

sudo dd if=<(printf '\x90\x90\x90\x90\x90') of=$(which ssh) count=5 bs=1 seek=59287 conv=notrunc

Note that this modification will most likely break ControlMaster and should not be used unless you are experiencing heavy slowdowns.

The following settings will make sshd respond faster. They will:

• disable DNS lookups
• turn off compression
• restrict the ciphers and hashes to faster variants while still being backward-compatible.

Edit /etc/ssh/sshd_config and append the lines:

# Server configuration
UseDNS no
compression no
ciphers arcfour,blowfish-cbc
macs hmac-md5-96

On OSX, the client configuration should be changed in order to include arcfour and blowfish-cbs as cipher and hmac-md5-96 as mac. In order to do that, open /etc/ssh/ssh_config and change the Host * directive:

Host *
        SendEnv LANG LC_*
        compression no
        Ciphers arcfour,blowfish-cbc,aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc
        MACs hmac-md5-96,hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160

Edit /etc/ssh/sshd_config and specify:

PermitRootLogin no
AllowUsers merlin

where merlin is an username.

The following command:

ssh -f jack@home.internal -L 2000:home.internal:25 -N

creates an SSH tunnel between the local machine that the command is run on and home.internal by forwarding the local 2000 port to the remote 25 port and without executing any command (-N).

The command can be broken down into the following options:

• -f tells ssh to background as soon as the connection to home.internal is established
• jack@home.internal is the user-name and remote machine to log-in to
• -L 2000:home.internal:25 is the actual forwarding part that will forward the local 2000 port to the remote 25 port on home.internal
• -N tells ssh to not execute an initial command

Sometimes one has to copy files that either contain special characters or the path to the file contains special characters. In such cases, you will find yourself experiencing either the dreaded error:

scp: ambiguous target

in case scp did not like the target path, or the dreaded error:

No such file or directory

In order to avoid that, both the source path and the target path must be escaped just that the target path must be double-escaped.

For example, suppose we wanted to send the file from the local machine:

/Users/bravo/My Documents/phile.txt

to johnny@192.168.0.1 with the path:

/Backups/Phile Pholders/

We then have several methods of doing this:

• scp /Users/bravo/My\ Documents/phile.txt johnny@192.168.0.1:"'/Backups/Phile Pholders/'"

• scp /Users/bravo/My\ Documents/phile.txt johnny@192.168.0.1:'"/Backups/Phile Pholders/"'

• scp /Users/bravo/My\ Documents/phile.txt johnny@192.168.0.1:'/Backups/Phile\ Pholders/'

• scp /Users/bravo/My\ Documents/phile.txt johnny@192.168.0.1:"/Backups/Phile\ Pholders/"

• scp /Users/bravo/My\ Documents/phile.txt johnny@192.168.0.1:/Backups/Phile\\\ Pholders/

To restrict users or groups to use SFTP only without being able to log-in, first create a designated SFTP-only group:

groupadd sftponly

and adding users to the sftponly group that should only have SFTP access, for instance, for user john:

usermod -g users -G users,sftponly john

and then modifying /etc/ssh/sshd_config:

Match group sftponly
	chrootDirectory /var/www
	ForceCommand internal-sftp
	AllowTcpForwarding no
	X11Forwarding no

where /var/www is a path to jail the users in the sftponly group to.

Finally, set the shell to /usr/bin/nologin for all users in the sftponly group:

usermod -s /usr/bin/nologin john

Some older hardware requires the diffie-hellman-group1-sha1 key exchange in order to establish a successful SSH connection. Enabling the key exchange algorithm can be performed inline without having to change settings permanently:

ssh -oKexAlgorithms=+diffie-hellman-group1-sha1 192.168.0.2

alternatively, edit /etc/ssh/ssh_config and adjust the parameters:

Host 192.168.0.2
    KexAlgorithms +diffie-hellman-group1-sha1

Any of the following symptoms are to be detected?

• SSH connections drop,
• issuing commands with large output over SSH makes the SSH session hang

They may all boil down to a non-uniform MTU setting across the network. This problem typically shows up with IoT devices (including the Raspberry Pi) that have a trimmed down networking stack (either wireless or low end network devices) that do not allow setting the MTU past 9000.

In case changing the MTU for the entire network is undesirable, then make sure that the settings:

net.ipv4.ip_no_pmtu_disc = 1
net.ipv4.tcp_mtu_probing = 2

are set with sysctl on the device and on the gateway endpoint. Similarly, ICMP and IGMP should not be blocked across the network because the packets are crucial to negotiating packet transfers.