Date created: Tuesday, June 10, 2014 2:16:09 PM. Last modified: Monday, October 14, 2024 2:34:43 PM

'ssh', 'sshfs', 'scp' & 'rsync' - Notes

Host Key Checking

The following options might be acceptable in a lab environment for testing but not acceptable in production:

-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no

Instead, when using scripts for the first time in production, setting StrictHostKeyChecking to "accept-new" should help:

-o StrictHostKeyChecking=accept-new

 

SSH SOCKS Proxy/Dynamic Port Forwarding

Local SOCKS proxy over SSH using -D:

-C Request compression of all data
-2 Enforce SSH protocol version 2
-q Qiuet mode
-T Disable pseudo-tty allocation
-n Prevents reading from stdin
-N Do not execute a remote command
-D[bind_addr:]port Specifies a local “dynamic” application-level port forwarding

ssh -C2qTnN -D 8080 ssh_user@ssh_server

By default the -D bind address is "any interface" on the local machine, but won't accept connection from any interface other than 127.0.0.1 with "GatewayPorts yes" in the local client /etc/ssh/ssh_config file. This can also be specified on the CLI as "ssh -o GatewayPorts=yes -D 8080 ssh_user@ssh_server".

By default, SSH will use the SOCKS5 protocol, which forwards TCP and UDP traffic.

The above can be tested with CURL:

curl -x socks5://127.0.0.1:12345 https://ifconfig.me/

 

SSH Local Port Forwarding

SSH can forward a port on a local IP to a port on a remote IP, through an SSH server ("localaddress" defaults to 127.0.0.1/localhost). The SSH server needs "AllowTcpForwarding yes" in the /etc/ssh/sshd_config file, but this is normally in by default:

-L [localaddress:]localport:serveraddress:serverport

In this example, there is no connectivity from the local machine to the remote machine example.com but SSH is allowed to a server, which has access to the example.com server. SSH can redirect a local port to a port on a remote machine through an SSH server:

# Note: sudo is required to interact with elevated port numnbers (lower than or equal to 1024):
sudo ssh -L 80:example.com:80 ssh_user@ssh_server

-f forks the ssh process into the background
-n prevents reading from STDIN
-N do not run remote commands. Used when only forwarding ports
-T disables TTY allocation

sudo ssh -fnNT -L 127.0.0.1:80:example.com:80 ssh_user@ssh_server

If only port 22/SSH access is allowed to a remote server, and access is needed to another port on that SSH server  e.g., a WebApp on port 8080, specifying "localhost" as the target server_address in the -L option redirects between the SSH client and SSH server only, not onward to a nother server like in the above example:

ssh -fnNT -L 8080:localhost:8080 ssh_user@ssh_server
# Telnet to local port 8080 will connect you to remotehost:8080
telnet 127.0.0.1 8080

The local address of the -L flag defaults to the 127.0.0.1/localhost (loopback) interface, a specific interface IP can be specified or it can bet set to any inferface using an asterisk (any interface other than 127.0.0.1 on the local/client system requires "GatewayPorts yes" in the ssh config file /etc/ssh/ssh_config):

ssh -nNT -L *:8080:example.com:80 ssh_user@ssh_server

 

Combining Local and Dynamic Port Forwarding

Example: SOCKS proxy tunneled through two hosts, by redirecting localhost port 8080 to server 5.5.5.5 on port 8080, connect to 5.5.5.5 and redirect local port 8080 there to 6.6.6.6.

-L [localaddress:]localport:serveraddress:serverport Specifies that the given port on the local (client) host is to be forwarded to the given server and port on the remote server

ssh -nL 8080:localhost:8080 username@machine1 "ssh -C2qnN -D 8080 username@machine2"

The above allows for an interactive login by keeping the local SSH process in the foreground. If key-based authentication is enabled then SSH can be sent to the background (note: machine 5.5.5.5 needs key authentication to 6.6.6.6, not just from the local machine to 5.5.5.5, in order to keep authentication non-interactive):

-f Requests ssh to go to background just before command execution

# When finished with the proxy you will need kill the background SSH process on the local machine and on 5.5.5.5:
ssh -f -L 8080:localhost:8080 username@5.5.5.5 "ssh -f -N -D 8080 username@6.6.6.6"

SOCKS proxy tunneled through two hosts again, but this time broken into two seperate commands:

# 5.5.5.5 runs SSH on non-standard port, and no -f keeps SSH in the front.
# When finished with the proxy you will need to kill the background SSH process on 5.5.5.5 only, because no use of -f on the local machine
ssh -L 8080:localhost:8080 -p 2222 username@5.5.5.5 "ssh -f -N -D 8080 username@6.6.6.6"

 

Remote/Revers Port Forwarding

Remote port forwarding can be used to forward a port on the remote SSH server to the local SSH client mahine.

For example, to allow SSH to a machine behind NAT using a machine outside of the NAT (with a public IP) by forwarding a port on the remote public SSH server to port 22 on the local SSH client behind the NAT;

# On the machine behind the NAT
ssh -R 5555:localhost:22 ssh_user@ssh_server

In the above example, port 5555 on the SSH server with a public IP needs to be reachable e.g, allowed through it's firewall. Also "GatewayPorts yes" must be in the SSH server's /etc/ssh/sshd_config config file, which is non-default. If that isn't possible, one can SSH to that public SSH server from a remote location then connect to port 5555 locally to connect back to the machine behind the NAT:

ssh localhost -p 5555

To expose a local web server through the ssh-server on port 8080:

ssh -R 8080:localhost:80 ssh-server

To expose a local web server through the ssh-server on port 8080, but limit connection to port 8080 on the SSH server to the source IP 192.0.2.1:

ssh -R 192.0.2.1:8080:localhost:80 ssh-server

The local SSH server can forward the traffic onward, in this example traffic entering the SSH server on port 8080 is sent over SSH to the local SSH client machine, which then forwards the traffic on to server example.com on port 80. The traffic reaching example.com on port 80 appears to come from the local SSH client:

ssh -R 8080:example.org:80 ssh-server

 

SSH IP VPN / Tunnel

SSH Server:

Enabled SSH tunnels on the server in the SSHD config file and set up a TUN interface with a static route to the client's LAN subnet:

sudo bash -c "echo 'PermitTunnel yes' >> /etc/ssh/sshd_config" && sudo /etc/init.d/sshd restart

sudo modprobe tun
lsmod | grep tun
sudo bash -c "echo 'tun' >> /etc/modules"

sudo ip tuntap add tun50 mode tun
sudo ip link set dev tun50 mtu 1390 # This is the max ping without fragmentation: ping -vvv -M do -c 1 -s 1390 192.168.254.1
sudo ip addr add 192.168.254.0/31 dev tun50
sudo ip link set up dev tun50
sudo ip route add 10.0.0.0/24 via 192.168.254.1

# To make this permenant, add the following to the interface file:
sudo bash -c 'echo "auto tun50
iface tun50 inet static
	mtu 1390
	address 192.168.254.0
	netmask 255.255.255.254
	# The pre-up command is required to create the interface
	pre-up ip tuntap add tun50 mode tun
        # Remote LAN
        up route add -net 10.0.0.0 netmask 255.255.255.0 gw 192.168.254.1
        down route del -net 10.0.0.0 netmask 255.255.255.0 gw 192.168.254.1
        # Delete the tun interface otherwise ifup will fail next time
        post-down ip tuntap del tun50 mode tun" >> /etc/network/interfaces'

SSH Client:

Set up a TUN interface with static route to the server's LAN subnet:

sudo modprobe tun
lsmod | grep tun
sudo bash -c "echo 'tun' >> /etc/modules"

sudo ip tuntap add tun50 mode tun
sudo ip link set dev tun50 mtu 1390 # This is the max ping without fragmentation: ping -vvv -M do -c 1 -s 1390 192.168.254.0
sudo ip addr add 192.168.254.1/31 dev tun50
sudo ip link set up dev tun50
sudo ip route add 10.0.1.0/24 via 192.168.254.0

sudo bash -c 'echo "auto tun50
iface eth0 inet static
    mtu 1390
    address 192.168.254.1
    netmask 255.255.255.254
    pre-up ip tuntap add tun50 mode tun
    up route add -net 10.0.1.0 netmask 255.255.255.0 gw 192.168.254.0
    down route del -net 10.0.1.0 netmask 255.255.255.0 gw 192.168.254.0
    post-down ip tuntap del tun50 mode tun" >> /etc/network/interfaces'

Set up an SSH IP tunnel between the two tun50 interfaces (from the client), the user-account on the server needs sudo permissions:

ssh -i /home/myuser/.ssh/id_rsa -w 50:50 user-account@server

 

SSH Config

Host entries in the SSH config file are evaluated from most specific to least specific, and in alphabetical order. Regardless of the actual order of the following Host entries in the SSH config file, these entries will always be evaluated in the order listed below. The configuration options are merged from most specific to least meaning and conflicting entries from less specific Host sections are ignored. This means that connecting to 192.168.0.1 will uses public key based authentication despite "PubkeyAuthentication no" under the final "Host *" section, even though that final generic section is evaluated last, after the 192.168.0.1 specific section. The 192.168.0.1 host will inherit the ServerAliveInterval value from the "Host *" section though because nothing is defined under the 192.168.0.1 specific section.

Host 192.168.0.1
    PubkeyAuthentication yes
    IdentityFile ~/.ssh/id_rsa
    Port 22
    AddKeysToAgent Yes

Host fd::1:0:0:0:1
    PubkeyAuthentication yes
    IdentityFile ~/.ssh/id_rsa

Host home_router
    HostName fd::1:0:0:0:1
    PubkeyAuthentication yes
    IdentityFile ~/.ssh/id_rsa

Match Host aaa*
    User customer_username
    Ciphers +aes256-cbc

Match Host zzz*
    ProxyJump jumpuser@jumphost
    User customer_username

Host *
    ServerAliveInterval 60
    ServerAliveCountMax 5
    PubkeyAuthentication no
    IdentitiesOnly yes

 

SSH Cipher, MAC and Kex Options

Force cipher, MAC and key exchange from client:

ssh -c chacha20-poly1305@openssh.com -m hmac-sha2-512-etm@openssh.com -oKexAlgorithms=+curve25519-sha256@libssh.org user@host

Example for legacy Cisco equipment:

ssh -oKexAlgorithms=+diffie-hellman-group1-sha1 test@192.168.0.1

Or in the SSH config file:

Host remoteserver
    HostKeyAlgorithms=+ssh-dss
    Ciphers +aes256-ctr
    Ciphers +aes256-cbc

 

Jump Hosts / Bastion Hosts

Basic jump host config in ~/.ssh/config:

Host remoteserver-hostname
    HostName ip.addr.of.remoteserver
    Port 5678
    ProxyJump jump-user@jump.host.ip.addr:1234
    User remoteserver-user

Or directly on the CLI with -J:

ssh -J jumpuser@jumphost1.example.org end_host_user@end_host.example.org
ssh -J user1@jumphost1.example.org:22 user@remotehost
ssh -J jumpuser@jumphost1,jumpuser@jumphost2, remoteuser@remotehost

Use a wildcard match on hostname so that all devices with a similar name use the same jump host:

Match Host sr1*
    ProxyJump jump-user@jump.host.ip.addr
    User my.user.name

SSH config can't contain multiple HostName entries for a single host e.g., to have a fallback IP/hostname. One can use multiple Match statements for the same host with an "exec" statement though, which can return false and fall through to the next Match statement. Below jumpbox2.example.org is only connected to when jumpbox1 .example.org isn't reachable (netcat fails to connect within 1 second):

Match Host jumpbox exec "nc -G 1 -z jumpbox1.example.org %p"
    HostName jumpbox1.example.org
    User jump.user
Match Host jumpbox exec "nc -G 1 -z jumpbox2.example.org %p"
    HostName jumpbox2.example.org
    User jump.user

 

SSH Keepalives

Set a keep alive interval in the client and/or server SSH config file to prevent TCP session idle timeouts and stateful firewall idle NAT timeouts. SSH uses TCPKeepAlives by default but they are not sent over the encrypted SSH tunnel.

~/.ssh/config /etc/ssh/ssh_config:

Host *
    ServerAliveInterval 60
    ServerAliveCountMax 4

/etc/ssh/sshd_config:

Host *
    ClientAliveInterval 60
    ClientAliveCountMax 4
man sshd_config
     ClientAliveCountMax
             Sets the number of client alive messages which may be sent without sshd(8) receiving any messages back from the client.
             If this threshold is reached while client alive messages are being sent, sshd will disconnect the client, terminating the
             session.  It is important to note that the use of client alive messages is very different from TCPKeepAlive.  The client
             alive messages are sent through the encrypted channel and therefore will not be spoofable.  The TCP keepalive option
             enabled by TCPKeepAlive is spoofable.  The client alive mechanism is valuable when the client or server depend on knowing
             when a connection has become unresponsive.
     ClientAliveInterval
             Sets a timeout interval in seconds after which if no data has been received from the client, sshd(8) will send a message
             through the encrypted channel to request a response from the client.  The default is 0, indicating that these messages
             will not be sent to the client.

man ssh_config
     ServerAliveCountMax
             Sets the number of server alive messages (see below) which may be sent without ssh(1) receiving any messages back from the
             server.  If this threshold is reached while server alive messages are being sent, ssh will disconnect from the server,
             terminating the session.  It is important to note that the use of server alive messages is very different from
             TCPKeepAlive (below).  The server alive messages are sent through the encrypted channel and therefore will not be spoofa-
             ble.  The TCP keepalive option enabled by TCPKeepAlive is spoofable.  The server alive mechanism is valuable when the
             client or server depend on knowing when a connection has become unresponsive.

             The default value is 3.  If, for example, ServerAliveInterval (see below) is set to 15 and ServerAliveCountMax is left at
             the default, if the server becomes unresponsive, ssh will disconnect after approximately 45 seconds.
     ServerAliveInterval
             Sets a timeout interval in seconds after which if no data has been received from the server, ssh(1) will send a message
             through the encrypted channel to request a response from the server.  The default is 0, indicating that these messages
             will not be sent to the server.

 

SCP

Copy using a jump host:

scp -J jump.username@jumphost network.username@remote.device:~/dhcpd.pcap ./

 

SSHFS

Mount remote filesystem;

# via smb
sudo smbmount //172.16.0.5/bensley /media/smbserver/ -o username=bensley
# via sshfs
sudo sshfs -o idmap=user -o allow_other bensley@172.16.0.5:/path/to/bensley /media/smbserver/

# Enable auto-remount if SSH session drops
sshfs -o rw -o reconnect -o idmap=user -o allow_other bensley@172.16.0.5:/path/to/bensley /media/smbserver/

# Via sshfs with non-standard ssh port
sudo sshfs -o idmap=user -o allow_other -o ssh_command="ssh -p 65535" username@172.16.0.5:/path/to/bensley /media/smbserver

# SSHFS to a box through an intermediate box (jump host), both on non-standard ports (jump box on 1234, end host on 5678)
sshfs -o idmap=user -o allow_other -o ssh_command="ssh -J jumpuser@jumpbox.ip.addr.here:1234 -p 5678" sshfs-user@sshfs.server.ip.addr:/a/path/ /local/mount/point

# Force IPv4
# Rease password from stdin
sshfs -o ssh_command='ssh -4' -o password_stdin -o rw -o reconnect -o idmap=user -o allow_other user@nas:/ /sshfs <<< 'password_here'

 

RSYNC

rsync options:

# Common rsync options
# a = archive mode; equals -rlptgoD (no -H,-A,-X)
# c = skip based on checksum, not mod-time & size
# D = same as --devices --specials
# g = preserve group
# h = output numbers in a human-readable format
# i = output a change-summary for all updates
# l = copy symlinks as symlinks
# o = preserve owner (super-user only)
# p = preserve permissions
# P = same as --partial and --progress (don't delete partial transfers, show progress of current transfer)
# r = recurse
# t = preserves times
# v = increase verbosity
# z = use compression
# --compress-level=9 = set the compression level to 9 (max)
# --devices = preserve device files (super-user only)
# --delete-during = delete files as we go (don't scan through before hand or after, faster)
# --force = force the deletion of non-empty directorys
# --ignore-errors = ignore I/O errors when deleting files
# --specials = preserve special files
# --stats print out transfer stats at the end

# Example when copying from a remote rsync folder name "pair1" defined in the rsyncd config, which helps to hide the full path on the remote side:

rsync -rzhitP --delete-during --ignore-errors --force --stats --compress-level=9 user@172.16.0.15::pair1 /backup/pair1

Resume rsync failed transfer over non-standard port (with a bandwidth limit):

rsync -rlptgoDivc --progress --stats --partial --bwlimit=300 -e "ssh -p 1234" user@10.0.0.1:~/path/*.txt /media/storage/backup

To exclude a directory use a relative path to the source directory. E.g. to exclude Firefox history stored in /Users/MyUser/Library/Caches/Firefox/Profiles/jn6pe0j8.default/cache2/entries/ remove either /Users/MyUser/ or ~/ from the exclude option:

rsync -vrhit --partial --progress --stats --delete --exclude=.DS_Store --exclude=Library/Caches/Firefox/Profiles/jn6pe0j8.default/cache2/entries/ /Users/MyUser/ /Volumes/NAS/

Only copy jpg pictures. Excluding everything with --exclude '*' and including *.jpg and *.jpg doesn't copy anything. The exclude supersedes the includes. Using --include '*/' to include all directories with the '*.jpg' and '*.JPG' matches:

rsync -rzhitP --partial --progress --stats --include '*/' --include '*.jpg' --include '*.JPG' --exclude "*" ./source/ ./destination/

When excluding file or directory patterns, use --delete-excluded to remove any ocurrances of them that already exist in the destination. In this example any temporary files matching "._*" are not copied, and any that already exist at the destination are deleted. --exclude 'foo' excludes files that match that pattern. --exclude=.git excludes directories with that name.

/usr/bin/rsync -vrhit --partial --progress --stats --delete -e "ssh -p 5555" --exclude '._*' --exclude '^.*' --exclude '.~*' --exclude '.DS_Store' --exclude 'desktop.ini' --exclude=.Trash-1000 --exclude=__pycache__ --exclude=.git --delete-excluded /path/to/local/source/ user@10.0.01:~/backups/

 


^ Top

Previous page: 'rtorrent' - Notes
Next page: 'tcpdump', 'editcap', & 'ngrep' - Notes