Automatic SCP using expect and bash

This post is 4 years old. (Or older!) Code samples may not work, screenshots may be missing and links could be broken. Although some of the content may be relevant please take it with a pinch of salt.

I am currently enjoying the vacation season, and I apologise for not posting that regularly. Here's a quick post for those of you who work with SCP - Secure Copy - over Linux. I'm sure you wanted to write a bash script that copies something over servers where you can't setup keyless login for whatever reason. I have a solution for you, please keep on reading.

The solution is very simple and it's utilising the expect command. You can write whole scripts using expect by making the following declaration in the first line of you bash script: #!/usr/local/bin/expect and it can be really useful but not when the rest of your code is in bash. (For informational purposes have a look at this great article explaining the use cases and options for expect.)

Let's assume that you wrote a bash script that executes a set of commands on a remote server, you put all that information into a file and you'd like to receive that file - i.e. scp it off from the remote server.

(A quick note on the password encryption used here: obfuscating the server password in the way I've done below is not the best, however it's still better than just displaying it in plain text - a regular user may not know how to interpret the result of the echo statement.)

#!/bin/bash
#Setup a few static variables

HOST_IP="1.1.1.1"
REMOTE_IP="2.2.2.2"
SCP_PASSWORD_E="eW91IGFyZSBjdXJpb3VzLCBhcmVuJ3QgeW91PyA6LSkK"
SCP_PASSWORD=`echo "$SCP_PASSWORD_E" | base64 -di`

#Remote execution of a command that collects data and saves it to a file - you can do whatever you want here
ssh user@$REMOTE_IP 'for i in `find /var/lib/mysql/database/ -type f | egrep -vi ".trn|.frm|.trg"  | sort`; do e=`md5sum "$i"`; echo "$e" >> /tmp/my.file; done'

#And now transfer the file over
expect -c "
    set timeout 1
    spawn scp user@$REMOTE_IP:/tmp/my.file user@$HOST_IP:/home/.
    expect yes/no { send yes\r ; exp_continue }
    expect password: { send $SCP_PASSWORD\r }
    expect 100%
    sleep 1
    exit
"
if [ -f "/home/my.file" ]; then
echo "Success"
fi

#Carry on with the code ...

And there you have it - expect will act as a 'human being' and answer the prompted questions as well as entering the password, and once the file transfer is complete (100%) it will exit after sleeping for 1 second (just to be on the safe side).

I utilise this 'trick' in a script at work that collects database, user and file system information from various servers and compares them for integrity purposes - it runs from cron automatically and sends out emails with the result of the integrity check. Without the expect portion I wouldn't be able to run this automatically and without human intervention.