{"id":185,"date":"2020-08-15T20:14:44","date_gmt":"2020-08-16T01:14:44","guid":{"rendered":"https:\/\/www.brezeale.com\/?p=185"},"modified":"2024-04-14T09:35:27","modified_gmt":"2024-04-14T14:35:27","slug":"185","status":"publish","type":"post","link":"https:\/\/www.brezeale.com\/?p=185","title":{"rendered":"Text Processing in Linux"},"content":{"rendered":"\r\n

I originally created and posted this November 22, 2004.<\/p>\r\n

Here are some examples of using the utilities found on Unix (available on some other platforms also) for manipulating the text in files. awk and perl both allow writing full programs, but I primarily use both as short one-liner programs which allows them to be piped to\/from other Unix programs. Each of these programs has capabilities that make it better than the others in some situations which I have attempted to demonstrate below. I don’t claim any of these to be original to me; references are at the bottom of the page.<\/p>\r\n

I have collected this information over the course of several years, during which time I have used Sun Solaris and various flavors of Linux. Note that the versions of these tools included with Solaris don’t entirely match the GNU versions, so some of what you see below may need to be tinkered with to make work.<\/p>\r\n

The philosophy of Unix utilities is to develop a tool that is very good at doing a specific thing. The output of a tool can be sent to another tool via the pipe (i.e., the | <\/span>character) as shown in several examples below. So, one program’s output becomes the next program’s input.<\/p>\r\n

awk<\/a>\u00a0\u00a0cat<\/a>\u00a0\u00a0csplit<\/a>\u00a0\u00a0cut<\/a>\u00a0\u00a0find<\/a>\u00a0\u00a0fmt<\/a>\u00a0\u00a0fold<\/a>\u00a0\u00a0grep<\/a>\u00a0\u00a0head<\/a>\u00a0\u00a0join<\/a>\u00a0\u00a0nl<\/a>\u00a0\u00a0paste<\/a>\u00a0\u00a0perl<\/a>\u00a0\u00a0sdiff<\/a>\u00a0\u00a0sed<\/a>\u00a0\u00a0sort<\/a>\u00a0\u00a0split<\/a>\u00a0\u00a0tail<\/a>\u00a0\u00a0uniq<\/a>\u00a0\u00a0wc<\/a>

Examples<\/a> References<\/a><\/p>\r\n

<\/a>sed, awk, and perl<\/h3>\r\n

<\/a>awk<\/b> \u2014 good for working with files that contain information in columns.<\/p>\r\n

    \r\n
  1. \r\n
      \r\n
    1. Display only the first three columns of the file SOMEFILE<\/span>, using tabs to separate the results:<\/li>\r\n<\/ol>\r\n<\/li>\r\n<\/ol>\r\n
      \r\n\r\n\r\n\r\n
      awk ‘{print $1 “\\t\\t” $2 “\\t” $3}’ SOMEFILE<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/blockquote>\r\n
        \r\n
      1. \r\n
          \r\n
        1. Display the first and fifth columns of the password file with a tab between them<\/li>\r\n<\/ol>\r\n<\/li>\r\n<\/ol>\r\n
          \r\n\r\n\r\n\r\n
          awk -F: ‘{print $1 “\\t” $5}’ \/etc\/passwd<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/blockquote>\r\n

          -F: <\/span>changes the column delimiter from spaces (the default) to a colon (:)<\/p>\r\n

            \r\n
          1. \r\n
              \r\n
            1. Display the second column of the file using double colons as the field separator<\/li>\r\n<\/ol>\r\n<\/li>\r\n<\/ol>\r\n
              \r\n\r\n\r\n\r\n
              awk -v ‘FS=::’ ‘{print $2}’ ratings.dat<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/blockquote>\r\n
                \r\n
              1. \r\n
                  \r\n
                1. replace first column as “ORACLE” in SOMEFILE<\/span><\/li>\r\n<\/ol>\r\n<\/li>\r\n<\/ol>\r\n
                  \r\n\r\n\r\n\r\n
                  awk ‘{$1 = “ORACLE”; print }’ SOMEFILE<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/blockquote>\r\n
                    \r\n
                  1. \r\n
                      \r\n
                    1. print the last field of every input line:<\/li>\r\n<\/ol>\r\n<\/li>\r\n<\/ol>\r\n
                      \r\n\r\n\r\n\r\n
                      awk ‘{ print $NF }’ SOMEFILE<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/blockquote>\r\n
                        \r\n
                      1. \r\n
                          \r\n
                        1. print the first 50 characters of each line. if a line has fewer than 50 characters, then the line is padded with spaces.<\/li>\r\n<\/ol>\r\n<\/li>\r\n<\/ol>\r\n
                          \r\n\r\n\r\n\r\n
                          awk ‘{ printf(“%-50.50s\\n”, $0) }’ SOMEFILE<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/blockquote>\r\n
                            \r\n
                          1. \r\n
                              \r\n
                            1. sum the values in column 1<\/li>\r\n<\/ol>\r\n<\/li>\r\n<\/ol>\r\n
                              \r\n\r\n\r\n\r\n
                              awk ‘BEGIN{total=0;} {total += $1;} END{print “total is “, total}’ SOMEFILE<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/blockquote>\r\n
                                \r\n
                              1. \r\n
                                  \r\n
                                1. sum the values in columns 1, 2 and 4 in order to calculate precision and recall<\/li>\r\n<\/ol>\r\n<\/li>\r\n<\/ol>\r\n
                                  \r\n\r\n\r\n\r\n
                                  awk -F ‘,’ ‘BEGIN{TP=0; FP=0; FN=0} {TP += $1; FP += $2; FN += $4} END{print “precision is “, TP\/(FP+TP); print “recall is “, TP\/(FN+TP)}’ prec-recall-2states.txt<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/blockquote>\r\n
                                    \r\n
                                  1. \r\n
                                      \r\n
                                    1. sum each row<\/li>\r\n<\/ol>\r\n<\/li>\r\n<\/ol>\r\n
                                      \r\n\r\n\r\n\r\n
                                      awk ‘{sum=0; for(i=1; i<=NF; i++){sum+=$i}; print sum}’ SOMEFILE<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/blockquote>\r\n

                                       <\/p>\r\n

                                      <\/a>sed<\/b> \u2014 from the man page:<\/p>\r\n

                                      \r\n

                                      Sed is a stream editor. A stream editor is used to perform basic text transformations on an input stream (a file or input from a pipeline). While in some ways similar to an editor which permits scripted edits (such as ed), sed works by making only one pass over the input(s), and is consequently more efficient. But it is sed\u2019s ability to filter text in a pipeline which particularly distinguishes it from other types of editors.<\/p>\r\n<\/blockquote>\r\n

                                        \r\n
                                      1. \r\n
                                          \r\n
                                        1. Double space infile<\/span> and send the output to outfile<\/span><\/li>\r\n<\/ol>\r\n<\/li>\r\n<\/ol>\r\n
                                          \r\n\r\n\r\n\r\n
                                          sed G < infile > outfile<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n

                                          I use the input\/output notation shown above. It is appropriate in many, if not all, cases to leave out the less than sign, e.g., sed G infile > outfile<\/span><\/p>\r\n<\/blockquote>\r\n

                                            \r\n
                                          1. \r\n
                                              \r\n
                                            1. Double space a file which already has blank lines in it. Output file should contain no more than one blank line between lines of text.<\/li>\r\n<\/ol>\r\n<\/li>\r\n<\/ol>\r\n
                                              \r\n\r\n\r\n\r\n
                                              sed ‘\/^$\/d;G’ < infile > outfile<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/blockquote>\r\n
                                                \r\n
                                              1. \r\n
                                                  \r\n
                                                1. Triple space a file<\/li>\r\n<\/ol>\r\n<\/li>\r\n<\/ol>\r\n
                                                  \r\n\r\n\r\n\r\n
                                                  sed ‘G;G’ < infile > outfile<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/blockquote>\r\n
                                                    \r\n
                                                  1. \r\n
                                                      \r\n
                                                    1. Undo double-spacing (assumes even-numbered lines are always blank)<\/li>\r\n<\/ol>\r\n<\/li>\r\n<\/ol>\r\n
                                                      \r\n\r\n\r\n\r\n
                                                      sed ‘n;d’ < infile > outfile<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/blockquote>\r\n
                                                        \r\n
                                                      1. \r\n
                                                          \r\n
                                                        1. Insert a blank line above every line which matches regex <\/span>(“regex” represents a regular expression<\/a>)<\/li>\r\n<\/ol>\r\n<\/li>\r\n<\/ol>\r\n
                                                          \r\n\r\n\r\n\r\n
                                                          sed ‘\/regex\/{x;p;x;}’ < infile > outfile<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/blockquote>\r\n
                                                            \r\n
                                                          1. \r\n
                                                              \r\n
                                                            1. Print the line immediately before regex<\/span>, but not the line containing regex<\/span><\/li>\r\n<\/ol>\r\n<\/li>\r\n<\/ol>\r\n
                                                              \r\n\r\n\r\n\r\n
                                                              sed -n ‘\/regexp\/{g;1!p;};h’ < infile > outfile<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/blockquote>\r\n
                                                                \r\n
                                                              1. \r\n
                                                                  \r\n
                                                                1. Print the line immediately after regex<\/span>, but not the line containing regex<\/span><\/li>\r\n<\/ol>\r\n<\/li>\r\n<\/ol>\r\n
                                                                  \r\n\r\n\r\n\r\n
                                                                  sed -n ‘\/regexp\/{n;p;}’ < infile > outfile<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/blockquote>\r\n
                                                                    \r\n
                                                                  1. \r\n
                                                                      \r\n
                                                                    1. Insert a blank line below every line which matches regex<\/span><\/li>\r\n<\/ol>\r\n<\/li>\r\n<\/ol>\r\n
                                                                      \r\n\r\n\r\n\r\n
                                                                      sed ‘\/regex\/G’ < infile > outfile<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/blockquote>\r\n
                                                                        \r\n
                                                                      1. \r\n
                                                                          \r\n
                                                                        1. Insert a blank line above and below every line which matches regex <\/span><\/li>\r\n<\/ol>\r\n<\/li>\r\n<\/ol>\r\n
                                                                          \r\n\r\n\r\n\r\n
                                                                          sed ‘\/regex\/{x;p;x;G;}’ < infile > outfile<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/blockquote>\r\n
                                                                            \r\n
                                                                          1. \r\n
                                                                              \r\n
                                                                            1. Convert DOS newlines (CR\/LF) to Unix format<\/li>\r\n<\/ol>\r\n<\/li>\r\n<\/ol>\r\n
                                                                              \r\n\r\n\r\n\r\n
                                                                              sed ‘s\/^M$\/\/’ < infile > outfile <\/span># in bash\/tcsh, to get ^M<\/span> press Ctrl-V then Ctrl-M<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/blockquote>\r\n
                                                                                \r\n
                                                                              1. \r\n
                                                                                  \r\n
                                                                                1. Print only those lines matching the regular expression\u2014similar to grep<\/li>\r\n<\/ol>\r\n<\/li>\r\n<\/ol>\r\n
                                                                                  \r\n\r\n\r\n\r\n
                                                                                  \r\n

                                                                                  sed -n ‘\/some_word\/p’ infile
                                                                                  sed ‘\/some_word\/!d’<\/p>\r\n<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/blockquote>\r\n

                                                                                    \r\n
                                                                                  1. \r\n
                                                                                      \r\n
                                                                                    1. Print those lines that do not match the regular expression\u2014similar to grep -v<\/li>\r\n<\/ol>\r\n<\/li>\r\n<\/ol>\r\n
                                                                                      \r\n\r\n\r\n\r\n
                                                                                      \r\n

                                                                                      sed -n ‘\/regexp\/!p’
                                                                                      sed ‘\/regexp\/d’<\/p>\r\n<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/blockquote>\r\n

                                                                                        \r\n
                                                                                      1. \r\n
                                                                                          \r\n
                                                                                        1. Skip the first two lines (start at line 3) and then alternate between printing 5 lines and skipping 3 for the entire file<\/li>\r\n<\/ol>\r\n<\/li>\r\n<\/ol>\r\n
                                                                                          \r\n\r\n\r\n\r\n
                                                                                          sed -n ‘3,${p;n;p;n;p;n;p;n;p;n;n;n;}’ < infile > outfile<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n
                                                                                          Notice that there are five p’s in the sequence, representing the five lines to print. The three lines to skip between each set of lines to print are represented by the n;n;n;<\/span> at the end of the sequence.<\/blockquote>\r\n
                                                                                            \r\n
                                                                                          1. \r\n
                                                                                              \r\n
                                                                                            1. Delete trailing whitespace (spaces, tabs) from end of each line<\/li>\r\n<\/ol>\r\n<\/li>\r\n<\/ol>\r\n
                                                                                              \r\n\r\n\r\n\r\n
                                                                                              sed ‘s\/[ \\t]*$\/\/’ < infile > outfile<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/blockquote>\r\n
                                                                                                \r\n
                                                                                              1. \r\n
                                                                                                  \r\n
                                                                                                1. Substitute (find and replace) foo <\/span>with bar<\/span> on each line<\/li>\r\n<\/ol>\r\n<\/li>\r\n<\/ol>\r\n
                                                                                                  \r\n\r\n\r\n\r\n
                                                                                                  sed ‘s\/foo\/bar\/’ < infile > outfile <\/span># replaces only 1st instance in a line
                                                                                                  sed ‘s\/foo\/bar\/4’ < infile > outfile <\/span> # replaces only 4th instance in a line
                                                                                                  sed ‘s\/foo\/bar\/g’ < infile > outfile <\/span> # replaces ALL instances in a line<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/blockquote>\r\n
                                                                                                    \r\n
                                                                                                  1. \r\n
                                                                                                      \r\n
                                                                                                    1. Replace each occurrence of the hexadecimal character 92 with an apostrophe:<\/li>\r\n<\/ol>\r\n<\/li>\r\n<\/ol>\r\n
                                                                                                      \r\n\r\n\r\n\r\n
                                                                                                      sed s\/\\x92\/’\/g” < old_file.txt > new_file.txt<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/blockquote>\r\n
                                                                                                        \r\n
                                                                                                      1. \r\n
                                                                                                          \r\n
                                                                                                        1. Print section of file between two regular expressions (inclusive)<\/li>\r\n<\/ol>\r\n<\/li>\r\n<\/ol>\r\n
                                                                                                          \r\n\r\n\r\n\r\n
                                                                                                          sed -n ‘\/regex1\/,\/regex1\/p’ < old_file.txt > new_file.txt<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/blockquote>\r\n
                                                                                                            \r\n
                                                                                                          1. \r\n
                                                                                                              \r\n
                                                                                                            1. Combine the line containing REGEX<\/span> with the line that follows it<\/li>\r\n<\/ol>\r\n<\/li>\r\n<\/ol>\r\n
                                                                                                              \r\n\r\n\r\n\r\n
                                                                                                              sed -e ‘N’ -e ‘s\/REGEX\\n\/REGEX\/’ < old_file.txt > new_file.txt<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/blockquote>\r\n

                                                                                                               <\/p>\r\n

                                                                                                              <\/a>perl<\/b> \u2014 can do anything sed and awk can do, but not always as easily as shown in the examples above.<\/p>\r\n

                                                                                                                \r\n
                                                                                                              1. \r\n
                                                                                                                  \r\n
                                                                                                                1. replace OLDSTRING with NEWSTRING in the file(s) in FILELIST [e.g., file1 file2<\/span> or *.txt<\/span>]<\/li>\r\n<\/ol>\r\n<\/li>\r\n<\/ol>\r\n
                                                                                                                  \r\n\r\n\r\n\r\n
                                                                                                                  perl -pi.bak -e ‘s\/OLDSTRING\/NEWSTRING\/g’ FILELIST<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/blockquote>\r\n

                                                                                                                  The options used are:<\/p>\r\n

                                                                                                                    \r\n
                                                                                                                  1. \r\n
                                                                                                                      \r\n
                                                                                                                    1. \r\n