2.5 Using remote resources
Teaching: 15 min || Exercises: 5 min
Overview
If you are a self-learner (rather than attending one of our workshops), you will need an account on a remote server to follow this section. You will need to adjust the commands shown on this page to match your username and hostname on that server.
2.5.1 Background
Let’s take a closer look at what happens when we use the shell on a desktop or laptop computer. The first step is to log in so that the operating system knows who we are and what we’re allowed to do. We do this by typing our username and password; the operating system checks those values against its records, and if they match, runs a shell for us.
As we type commands, the 1’s and 0’s that represent the characters we’re typing are sent from the keyboard to the shell. The shell displays those characters on the screen to represent what we type, and then, if what we typed was a command, the shell executes it and displays its output (if any).
What if we want to run some commands on another machine, such as the Noguchi server in the Prince Alwaleed building that manages our database of field results? To do this, we have to first log in to that machine. We call this a remote login.
In order for us to be able to login, the remote computer must be running a remote login server and we will run a client program that can talk to that server. The client program passes our login credentials to the remote login server and, if we are allowed to login, that server then runs a shell for us on the remote computer.
Once our local client is connected to the remote server, everything we type into the client is passed on, by the server, to the shell running on the remote computer. That remote shell runs those commands on our behalf, just as a local shell would, then sends back output, via the server, to our client, for our computer to display.
This phenomenon works just like accessing your email. In fact, if you have used an email address to send mails then you have unknowingly used a remote server. Have you wondered where all your email messages are sitting? Your guess is as good as mine!
2.5.2 The ssh protocol
The Secure SHell (SSH) is a protocol which allows us to send secure encrypted information across an unsecured network, like the internet. The underlying protocol supports a number of commands we can use to move information of different types in different ways. The simplest and most straightforward is the ssh command which facilitates a remote login session connecting our local user and shell to any remote user we have permission to access.
ssh sshuser@remote-machineThe first argument specifies the location of the remote machine (by IP address or a URL) as well as the user we want to connect to seperated by an @ sign. For the purpose of this course we’ve set up a container on our cluster for you to connect to the address remote-machine is actually a special kind of URL that only your computer will understand. In real life this would normally be an address on the internet which can be access from anywhere or at least an institutional local area network.
When you connect to a computer for the first time you should see a warning like the one above. This signifies that the computer is trying to prove it’s identity by sending a fingerprint which relates to a key that only it knows. Depending on the security of the server you are connecting to they might distribute the fingerprint ahead of time for you to compare and advise you to double check it in case it changes at a later log on. In our case it is safe to type yes .
sshuser@192.168.1.59's password: ********Now you are prompted for a password. Enter the appropriate password and hit enter.
sshuser@remote_machine:~$You should now have a prompt very similar to the one you started with but with a new username and computer hostname. Take a look around with the ls command and you should see that your new session has its own completely independent filesystem. Unfortunately None of the files are particularly relevant to us. Let’s change that, but first we need to go back to our original computer’s shell. Use the key combination Ctrl + D (Ctrl+D) on an empty command prompt to log out.
2.5.3 Moving files
ssh has a simple file copying counterpart called secure copy scp which uses all the same methods for authentication and encryption but focuses on moving files between computers in a similar manner to the cp command we learnt about before.
Making sure we’re in the 02_unix_intro directory let’s copy the bacteria.txt file to the remote machine
cd ~/Desktop/workshop_files_Bact_Genomics_2023/02_unix_intro
scp bacteria.txt sshuser@remote-machine:/home/userYou will be prompted to provide your login credentials while attempting to copy to/from a remote source.
The format of the command should be quite familiar when comparing to the cp command for local copying. The last two arguments specify the source and the destination of the copy respectively. The difference comes in that any remote locations involved in the copy must be preceded by the username@IP syntax used in the ssh command previously. The first half tells scp how to access the computer and the second half tells it where in the filesystem to operate, these two segments are separated by a : .
Now it looks like we’ve copied the file, but we should check.
Establishing a whole ssh session just to run one command might be a bit cumbersome. Instead we can tell ssh all the commands it needs to run at the same time we connect by adding an extra argument to the end. ssh will automatically disconnect after it completes the full command string.
Success!
2.5.4 Managing multiple remote files
Sometimes we need to manage a large number of files across two locations, often maintaining a specific directory structure. scp can handle this with it’s -r flag. Just like cp this puts the command in recursive mode allowing it to copy entire directories of files
Did you realise the format of the above command was different from the previous scp command? Here, we copied files from the remote server unto our local machine. Check in your current directory if you will find the files you just copied.
However scp isn’t always the best tool to use for managing this kind of operation.
When you run scp it copies the entirety of every single file you specify. For an initial copy this is probably what you want, but if you change only a few files and want to synchronize the server copy to keep up with your changes it wouldn’t make sense to copy the entire directory structure again.
For this scenario rsync can be an excellent tool.
2.5.5 rsync
First lets add some new files to our 02_unix_intro directory using the touch command. This command does nothing but create an empty file or update the timestamp of an existing file.
touch newfile1 newfile2 newfile3Now we have everything set up we can issue the rsync command to sync our two directories.
rsync -avz 02_unix_intro sshuser@remote-machine:/home/user/workshop_files_Bact_Genomics_2023/02_unix_introThe -v flag/option stands for verbose and outputs a lot of status information during the transfer. This is helpful when running the command interactively but should generally be removed when writing scripts.
The -z flag/option means that the data will be compressed in transit. This is good practice depending on the speed of your connection (although in our case it is a little unnecessary).
Whilst we’ve used rsync in mostly the same way as we did scp it has many more customisation options as we can observe on the man page
man rsync-a is useful for preserving filesystem metadata regarding the files being copied. This is particularly relevant for backups or situations where you intend to sync a copy back over your original at a later point.
--exclude and --include can be used to more granularly control which files are copied.
Finally --delete is very useful when you want to maintain an exact copy of the source including the deletion of files only present in the destination. Let’s tidy up our 02_unix_intro directory with this option.
2.5.6 SshFs
Sshfs is another way of using the same ssh protocol to share files in a slightly different way. This software allows us to connect the file system of one machine with the file system of another using a “mount point”. Let’s start by making a directory in /home/ubuntu/Desktop/workshop_files_Bact_Genomics_2023/02_unix_intro to act as this mount point. Convention tells us to call it mnt.
mkdir /home/ubuntu/Desktop/workshop_files_Bact_Genomics_2023/02_unix_intro/mntNow we can run the sshfs command
sshfs sshuser@remote-machine:/home/user /home/ubuntu/Desktop/workshop_files_Bact_Genomics_2023/02_unix_intro/mnt/It looks fairly similar to the previous copying commands. The first argument is a remote source, the second argument is a local destination. The difference is that now whenever we interact with our local mount point it will be as if we were interacting with the remote filesystem starting at the directory we specified /home/sshuser.
The files shown are not on the local machine at all but we can still access and edit them. Much like the concept of a shared network drive in Windows.
This approach is particularly useful when you need to use a program which isn’t available on the remote server to edit or visualize files stored remotely. Keep in mind however, that every action taken on these files is being encrypted and sent over the network. Using this approach for computationally intense tasks could substantially slow down the time to completion.
It’s also worth noting that this isn’t the only way to mount remote directories in linux. Protocols such as nfs and samba are actually more common and may be more appropriate for a given use case. Sshfs has the advantage of running over ssh so it require very little set up on the remote computer.
2.5.7 wget - Accessing web resources
wget is in a different category of command compared to the others in this section. It is mainly for accessing resources that can be downloaded over http(s) and doesn’t have a mechanism for uploading/pushing files.
Whilst this tool can be customised to do a wide range of tasks, at its simplest it can be used to download datasets for processing at the start of a processing pipeline.
The data for this course can be downloaded as follows
For large files or sets of files there are also a few useful flags/options
-bPlaces the task in the background and writes all output into a log file-ccan be used to resume a download that has failed mid-way-ican take a file with a list of URLs for downloading
Where tools like wget shine in particular is in using URLs generated by web-based REST APIs like the one offered by Ensembl .
We can use bash to combine strings and form a valid query URL that meets our requirements.
wget https://rest.ensembl.org/genetree/member/symbol/homo_sapiens/BRCA2?prune_taxon=9526;content-type=text/x-orthoxml%2Bxml;prune_species=cowAlthough using these resources is very good especially being able to include them in scripts, currently, there are GUI applications that can enable you communicate just as good with remote servers. We will look at the vscode in our next lesson.
2.5.8 Credit
Information on this page has been adapted and modified from the following source(s):
https://github.com/cambiotraining/UnixIntro
https://github.com/cambiotraining/unix-shell