File descriptors re-visited

We looked at input-output redirection earlier in this course. Remember we had:

file descriptor 0 stdin
file descriptor 1 stdout
file descriptor 2 stderr

We were restricted to only these 3 file descriptors (FD)?

No, any process can have up to 9 file descriptors, we have only discussed 3 thus far. By default though every terminal that is created, is created with the above three file descriptors.

Firstly let us establish which terminal we are currently logged on to:

tty
            

The output may be one of those described below:

/dev/pts/1	# a pseudo-terminal if you're using X11
            

or

/dev/ttyx	# where x is a number between 1 and 6 (usually) 
		# if you're on a console
            

Now run:

lsof -a -p $$ -d0,1,2
            

This shows a list of open files for this PID (remember $$ was the current PID).

Read the man pages for lsof if you need more information about this command.

If you run the above command, since all terminals are opened with the above three file descriptors you should see our three file descriptors. All three of them should be pointing to the same place, my terminal.

The output generated by these commands is shown below (of course you will see slightly different output to mine):

$ps
PID TTY          TIME CMD
1585 pts/1    00:00:00 bash

$echo $$
1585

$tty
/dev/pts/1

$lsof -a -p $$ -d0,1,2
COMMAND  PID   USER   FD   TYPE DEVICE SIZE NODE NAME
bash    1585 hamish    0u   CHR  136,1         3 /dev/pts/1
bash    1585 hamish    1u   CHR  136,1         3 /dev/pts/1
bash    1585 hamish    2u   CHR  136,1         3 /dev/pts/1	
$
            

The need for extra file descriptors is based upon the need to be able to redirect output or input on a semi-permanent basis. We need to have a way of creating additional file descriptors. Say for example we wanted all our scripts to log output to particular log file then we would have the following (or something similar) in a script:

#!/bin/bash
LOGFILE=/var/log/script.log
cmd1 >$LOGFILE
cmd2 >$LOGFILE
            

This is not a very appealing solution.

Another way of achieving this is by creating a new file descriptor or alternatively assign our existing stdout file descriptor to a logfile (the latter option is illustrated below).

Re-assigning an existing file descriptor using the exec command:

  1     #!/bin/bash     LOGFILE=/var/log/script.log     exec 1>$LOGFILE   5 cmd1     cmd2    

You will notice that line 3 redirects stdout to $LOGFILE, so that lines 4 and 5 need not redirect their output explicitly.

Now every command that we run after that ensures that its output is directed to LOGFILE, which is used as the new standard output.

Try this on your command line as follows:

exec 1>script.log
            

Remember you have to have write permissions to be able to write to a system file such as /var/log, so here we are just writing the log file in our current directory.

We've now redirected any output from the console (or terminal) to script.log. Well that's fair enough, but how to test it? On the command line, type:

ls
            

What happens? You DON'T get the listing you were expecting! Type:

pwd
            

and it doesn't show you the working directory either. The command seems to complete, but nothing seems to be happening - or at least we can't see if anything is happening. What's actually happening is that the output of these commands is going to our script.log file as we set it up to do.

Try a:

lsof -a -p $$ -d0,1,2
            

Again the output is sent to script.log. Well, surely we can just cat the log file:

cat script.log 
            

What happens? Well the same thing that happens when you type pwd, ls or lsof - nothing (or you may even get an error). The question is how to get back your stdout? Well the answer is YOU CAN'T!

You see, before re-assiging stdout, you didn't save your initial standard output file descriptor. So in some ways - you've actually lost your stdout. The only way to get your standard output back is to kill the shell using:

exit
            

or press Ctrl-D to exit your shell. This will then reset stdout, but it will also kill the shell. That's pretty extreme and a tad useless!

What we want is a better way of doing this, so instead of just redirecting my stdout, I'm going to save my stdout file descriptor to a new file descriptor.

Look at the following:

exec 3>&1	# create a new FD, 3, and point it to the 
			# same place FD 1 is pointed
			
exec 1>script.log	# Now, redirect FD 1 to point to the 
			# log file.
			
cmd1			# Execute commands, their stdout going 
			# to script.log
			
cmd2			# Execute commands, their stdout going 
			# to script.log
			
exec 1>&3	# Reset FD 1 to point to the same
			# place as FD 3

cat script.log		# Aaah, that's better.
lsof -a -p $$ -d0,1,2,3	# check that we now have 4 FD associated
			# with this PID
            

You will notice that we now have four file descriptors (0,1,2 and 3), which are all pointing to the same node name.

With exec, we are able to create up to 9 new file descriptors, but we should save our existing file descriptors if we wish to return them to their previous state afterwards.

Let's try to reassign FD 3 to the file riaan.log

exec 3>riaan.log
lsof -a -p $$ -d0,1,2,3

COMMAND  PID  USER   FD   TYPE DEVICE SIZE  NODE NAME
bash    3443 riaan    0u   CHR 136,35         37 /dev/pts/35
bash    3443 riaan    1u   CHR 136,35         37 /dev/pts/35
bash    3443 riaan    2u   CHR 136,35         37 /dev/pts/35
bash    3443 riaan    3u   REG    3,1    0 86956 /home/riaan/ShellScripts/riaan.log
            

Now you should see something different because the node name has been updated to point to riaan.log for file descriptor 3.

Remember, that this redirection of file descriptors is only valid for this shell, not for child processes.[21]

We are able to create up to 9 file descriptors per process and we are able to save our existing file descriptors in order that we can restore them later. We can close a file descriptor with:

exec 3>&-
            

To check that file descriptor 3 has in fact closed, run:

lsof -a -p $$ -d0,1,2,3
            

and you will only see file descriptors 0,1 and 2.

Manipulating the file descriptors can be used to great effect in our scripts, because instead of having to redirect every command to a log file, we can now just redirect stdout:

#!/bin/bash
LOGFILE=~/script.log
exec 3>&1				#save FD1
exec 1>$LOGFILE			#stdout going to $LOGFILE
ls -alh /usr/share			#do a command
pwd					# and another command
who am i				# at least now I know ;-)
echo "Finished" >&3		# This now goes to stdout
echo "Now I'm writing to the log file again"
exec 1>&3				#Reset FD1
exec 3>&-				#Close FD3
            

This will then echo "Finished" to the console, because we've saved stdout file descriptor in file descriptor 3.

Redirecting the input would work in a similar fashion:

exec 4<&amp;0
exec <restaurants.txt
while read rating type place tel 
do 
	echo $type,$rating,$place,$tel
done
            

That would then take all our input from the file restaurants.txt.

Exercises

  1. Modify your eatout script in such a manner that any errors produced by the script will be redirected to a file called eatout.err in your home directory.

  2. Allow the user to select from the menu in eatout.sh, but ensure that their keystrokes are recorded in a file called eatout.log

  3. Write a script that should take two arguments, an input file (-i infile) and an output file (-o outfile). Using file descriptor redirection, the script should convert all data from the input file (infile) to uppercase and write the uppercased file to the output file (outfile). Ensure that your script does all necessary error checking, that it cannot be 'broken out of', killed, etc. and that all user options are adequately checked to ensure they conform to that required. Ensure that exit status' are supplied if errors are detected. An example of the command line is given below:

    upcase.sh -i bazaar.txt -o BAZAAR.TXT
                            

This is a good time to put together all these things you have learned en-route. It is always a good idea to complete the script with comments on what it is doing, to give a usage message to the user if they use a -h or -help option, and to make the script almost self explanatory. Don't be sloppy because you will regret it when the script needs to be maintained.



[21] you can check that this is the case by starting another bash, and running the lsof command for this new process. Exiting from this bash will return you to your original file descriptors