Linux - Pipes, Redirection, and REGEX

标签：

Part 1: Command Line Pipes

Previous chapters discussed how to use individual commands to perform actions on the operating system, including how to create/move/delete files and move around the system. Typically, when a command has output or generates an error, the output is displayed to the screen; however, this does not have to be the case.

The pipe (|) character can be used to send the output of one command to another. Instead of being printed to the screen, the output of one command becomes input for the next command. This can be a powerful tool, especially when looking for specific data; piping is often used to refine the results of an initial command.

The head and tail commands will be used in many examples below to illustrate the use of pipes. These commands can be used to display only the first few or last few lines of a file (or, when used with a pipe, the output of a previous command).

By default the head and tail commands will display ten lines. For example, the following command will display the first ten lines of the /etc/sysctl.conffile:

In the next example, the last ten lines of the file will be displayed:

The pipe character will allow users to utilize these commands not only on files, but on the output of other commands. This can be useful when listing a large directory, for example the /etc directory:

If you look at the output of the previous command, you will note that first filename is fstab. But there are other files listed "above" that can only be viewed if the user uses the scroll bar. What if you just wanted to list the first few files of the /etc directory?

Instead of displaying the full output of the above command, piping it to the headcommand will display only the first ten lines:

The full output of the ls command is passed to the head command by the shell instead of being printed to the screen. The head command takes this output (from ls) as "input data" and the output of head is then printed to the screen.

Multiple pipes can be used consecutively to link multiple commands together. If three commands are piped together, the first command‘s output is passed to the second command. The output of the second command is then passed to the third command. The output of the third command would then be printed to the screen.

It is important to carefully choose the order in which commands are piped, as the third command will only see input from the output of the second. The examples below illustrate this using the nl command. In the first example, the nlcommand is used to number the lines of the output of a previous command:

In the next example, note that the ls command is executed first and its output is sent to the nl command, numbering all of the lines from the output of the lscommand. Then the tail command is executed, displaying the last five lines from the output of the nl command:

Compare the output above with the next example:

Notice how the line numbers are different. Why is this?

In the second example, the output of the ls command is first sent to the tailcommand which "grabs" only the last five lines of the output. Then the tailcommand sends those five lines to the nl command, which numbers them 1-5.

Pipes can be powerful, but it is important to consider how commands are piped to ensure that the desired output is displayed.

Part 2: I/O Redirection

Input/Output (I/O) redirection allows for command line information to be passed to different streams. Before discussing redirection, it is important to understand standard streams.

1. STDIN 

Standard input, or STDIN, is information entered normally by the user via the keyboard. When a command prompts the shell for data, the shell provides the user with the ability to type commands that, in turn, are sent to the command as STDIN.

2. STDOUT

Standard output, or STDOUT, is the normal output of commands. When a command functions correctly (without errors) the output it produces is called STDOUT. By default, STDOUT is displayed in the terminal window (screen) where the command is executing.

3. STDERR

Standard error, or STDERR, are error messages generated by commands. By default, STDERR is displayed in the terminal window (screen) where the command is executing.

I/O redirection allows the user to redirect STDIN so data comes from a file and STDOUT/STDERR so output goes to a file. Redirection is achieved by using the arrow characters: ( < ) and ( > ).

4. Redirecting STDOUT

STDOUT can be directed to files. To begin, observe the output of the following command which will display to the screen:

Using the > character the output can be redirected to a file:

This command displays no output, because STDOUT was sent to the fileexample.txt instead of the screen. You can see the new file with the output of the ls command. The newly-created file contains the output of the echocommand when the file is viewed with the cat command.

It is important to realize that the single arrow will overwrite any contents of an existing file:

The original contents of the file are gone, replaced with the output of the newecho command.

It is also possible to preserve the contents of an existing file by appending to it. Use "double arrow" ( >> ) to append to a file instead of overwriting it:

Instead of being overwritten, the output of the most recent echo command is added to the bottom of the file.

5. Redirecting STDERR

STDERR can be redirected in a similar fashion to STDOUT. STDOUT is also known as stream (or channel) #1. STDERR is assigned stream #2.

When using arrows to redirect, stream #1 is assumed unless another stream is specified. Thus, stream #2 must be specified when redirecting STDERR.

To demonstrate redirecting STDERR, first observe the following command which will produce an error because the specified directory does not exist:

Note that there is nothing in the example above that implies that the output is STDERR. The output is clearly an error message, but how could you tell that it is being sent to STDERR? One easy way to determine this is to redirect STDOUT:

In the example above, STDOUT was redirected to the output.txt file. So, the output that is displayed can‘t be STDOUT because it would have been placed in the output.txt file. Because all command output goes either to STDOUT or STDERR, the output displayed above must be STDERR.

The STDERR output of a command can be sent to a file:

In the command above, the 2> indicates that all error messages should be sent to the file error.txt.

6. Redirecting Multiple Streams

It is possible to direct both the STDOUT and STDERR of a command at the same time. The following command will produce both STDOUT and STDERR because one of the specified directories exists and the other does not:

If only the STDOUT is sent to a file, STDERR will still be printed to the screen:

If only the STDERR is sent to a file, STDOUT will still be printed to the screen:

Both STDOUT and STDERR can be sent to a file by using &>, a character set that means "both 1> and 2>":

Note that when you use &>, the output appears in the file with all of the STDERR messages at the top and all of the STDOUT messages below all STDERR messages:

If you don‘t want STDERR and STDOUT to both go to the same file, they can be redirected to different files by using both > and 2>. For example:

The order the streams are specified in does not matter.

7. Redirecting STDIN

The concept of redirecting STDIN is a difficult one because it is more difficult to understand why you would want to redirect STDIN. With STDOUT and STDERR, the answer to why is fairly easy: because sometimes you want to store the output into a file for future use.

Most Linux users end up redirecting STDOUT routinely, STDERR on occasion and STDIN...well, very rarely. There are very few commands that require you to redirect STDIN because with most commands if you want to read data from a file into a command, you can just specify the filename as an argument to the command. The command will then look into the file.

For some commands, if you don‘t specify a filename as an argument, they will revert to using STDIN to get data. For example, consider the following catcommand:

In the example above, the cat command wasn‘t provided a filename as an argument. So, it asked for the data to display on the screen from STDIN. The user typed "hello" and then the cat command displayed "hello" on the screen. Perhaps this is useful for lonely people, but not really a good use of the catcommand.

However, perhaps if the output of the cat command were redirected to a file, then this method could be used either to add to an existing file or to place text into a new file:

While the previous example demonstrates another advantage of redirecting STDOUT, it doesn‘t address why or how STDIN can be directed. To understand this, first consider a new command called tr. This command will take a set of characters and translate them into another set of characters.

For example, suppose you wanted to capitalize a line of text. You could use thetr command as follows:

The tr command took the STDIN from the keyboard ("watch how this works") and converted all lower case letters before sending STDOUT to the screen ("WATCH HOW THIS WORKS").

It would seem that a better use of the tr command would be to perform translation on a file, not keyboard input. However, the tr command does not support filename arguments:

You can, however, tell the shell to get STDIN from a file instead of from the keyboard by using the < character:

This is fairly rare because most commands do accept filenames as arguments. But, for those that do not, this method could be used to have the shell read from the file instead of relying on the command to have this ability.

One last note: In most cases you probably want to take the resulting output and place it back into another file:

Part 3: Searching for Files Using the Find Command

find [starting directory] [search option] [ search criteria] [result option]

1. Search by File Name

To search for a file by name, use the -name option to the find command:

Note that two files were found: /etc/hosts and /etc/avahi/hosts. The rest of the output was STDERR messages because the user who ran the command didn‘t have the permission to access certain subdirectories.

Recall that you can redirect STDERR to a file so you don‘t need to see these error messages on the screen:

While the output is easier to read, there really is no purpose to storing the error messages in the error.txt file. The developers of Linux realized that it would be good to have a "junk file" to send unnecessary data; any file that you send to the /dev/null file is discarded:

Linux - Pipes, Redirection, and REGEX

标签：

原文地址：http://www.cnblogs.com/elewei/p/4820261.html