Automatic SCP using expect and bash

Quick one for those of you who work with SCP (Secure Copy) over Linux. You’ve probably wanted to write a bash script that copies files between servers where you can’t set up keyless login. Here’s how.

The solution uses the expect command. You can write entire scripts with expect by declaring #!/usr/local/bin/expect as the first line of your bash script. It’s powerful on its own, but not when the rest of your code is plain bash. (For reference, have a look at this article explaining the use cases and options for expect.)

Say you’ve written a bash script that runs a set of commands on a remote server, dumps the output into a file, and you want to pull that file back via scp.

(A note on the password obfuscation below: base64-encoding the password isn’t great security, but it’s better than having it in plain text. A casual observer won’t know what to do with the echo output.)

#!/bin/bash
#Setup a few static variables

HOST_IP="1.1.1.1"
REMOTE_IP="2.2.2.2"
SCP_PASSWORD_E="eW91IGFyZSBjdXJpb3VzLCBhcmVuJ3QgeW91PyA6LSkK"
SCP_PASSWORD=`echo "$SCP_PASSWORD_E" | base64 -di`

#Remote execution of a command that collects data and saves it to a file - you can do whatever you want here
ssh user@$REMOTE_IP 'for i in `find /var/lib/mysql/database/ -type f | egrep -vi ".trn|.frm|.trg"  | sort`; do e=`md5sum "$i"`; echo "$e" >> /tmp/my.file; done'

#And now transfer the file over
expect -c "
    set timeout 1
    spawn scp user@$REMOTE_IP:/tmp/my.file user@$HOST_IP:/home/.
    expect yes/no { send yes\r ; exp_continue }
    expect password: { send $SCP_PASSWORD\r }
    expect 100%
    sleep 1
    exit
"
if [ -f "/home/my.file" ]; then
echo "Success"
fi

#Carry on with the code ...

expect acts as a “human being”, answering prompts and entering the password. Once the file transfer hits 100%, it exits after a one-second pause (just to be safe).

I use this trick in a script at work that pulls database, user, and file system information from various servers, then compares them for integrity checks. It runs from cron automatically and fires off emails with the results. Without expect, none of that would run unattended.