Page 1

Awk Introduction Tutorial – 7 Awk Print Examples by Sasikala on January 6, 2010

This is the first article on the new awk tutorial series. We‘ll be posting several articles on awk in the upcoming weeks that will cover all features of awk with practical examples. In this article, let us review the fundamental awk working methodology along with 7 practical awk print examples. Note: Make sure you review our earlier Sed Tutorial Series.

Awk Introduction and Printing Operations Awk is a programming language which allows easy manipulation of structured data and the generation of formatted reports. Awk stands for the names of its authors ―Aho, Weinberger, and Kernighan‖ The Awk is mostly used for pattern scanning and processing. It searches one or more files to see if they contain lines that matches with the specified patterns and then perform associated actions. Some of the key features of Awk are:    

Awk views a text file as records and fields. Like common programming language, Awk has variables, conditionals and loops Awk has arithmetic and string operators. Awk can generate formatted reports

Awk reads from a file or from its standard input, and outputs to its standard output. Awk does not get along with non-text files. Syntax:


awk '/search pattern1/ {Actions} /search pattern2/ {Actions}' file

In the above awk syntax:     

search pattern is a regular expression. Actions – statement(s) to be performed. several patterns and actions are possible in Awk. file – Input file. Single quotes around program is to avoid shell not to interpret any of its special characters.

Awk Working Methodology 1. Awk reads the input files one line at a time. 2. For each line, it matches with given pattern in the given order, if matches performs the corresponding action. 3. If no pattern matches, no action will be performed. 4. In the above syntax, either search pattern or action are optional, But not both. 5. If the search pattern is not given, then Awk performs the given actions for each line of the input. 6. If the action is not given, print all that lines that matches with the given patterns which is the default action. 7. Empty braces with out any action does nothing. It wont perform default printing operation. 8. Each statement in Actions should be delimited by semicolon. Let us create employee.txt file which has the following content, which will be used in the examples mentioned below. $cat 100 200 300 400 500

employee.txt Thomas Manager Jason Developer Sanjay Sysadmin Nisha Manager Randy DBA

Sales Technology Technology Marketing Technology

$5,000 $5,500 $7,000 $9,500 $6,000

Awk Example 1. Default behavior of Awk By default Awk prints every line from the file. $ awk '{print;}' employee.txt 100 Thomas Manager Sales 200 Jason Developer Technology 300 Sanjay Sysadmin Technology 400 Nisha Manager Marketing 500 Randy DBA Technology

$5,000 $5,500 $7,000 $9,500 $6,000


In the above example pattern is not given. So the actions are applicable to all the lines. Action print with out any argument prints the whole line by default. So it prints all the lines of the file with out fail. Actions has to be enclosed with in the braces.

Awk Example 2. Print the lines which matches with the pattern. $ awk '/Thomas/ > /Nisha/' employee.txt 100 Thomas Manager Sales 400 Nisha Manager Marketing

$5,000 $9,500

In the above example it prints all the line which matches with the ‗Thomas‘ or ‗Nisha‘. It has two patterns. Awk accepts any number of patterns, but each set (patterns and its corresponding actions) has to be separated by newline.

Awk Example 3. Print only specific field. Awk has number of built in variables. For each record i.e line, it splits the record delimited by whitespace character by default and stores it in the $n variables. If the line has 4 words, it will be stored in $1, $2, $3 and $4. $0 represents whole line. NF is a built in variable which represents total number of fields in a record. $ awk '{print $2,$5;}' employee.txt Thomas $5,000 Jason $5,500 Sanjay $7,000 Nisha $9,500 Randy $6,000 $ awk '{print $2,$NF;}' employee.txt Thomas $5,000 Jason $5,500 Sanjay $7,000 Nisha $9,500 Randy $6,000

In the above example $2 and $5 represents Name and Salary respectively. We can get the Salary using $NF also, where $NF represents last field. In the print statement ‗,‘ is a concatenator.

Awk Example 4. Initialization and Final Action Awk has two important patterns which are specified by the keyword called BEGIN and END. Syntax: BEGIN { Actions} {ACTION} # Action for everyline in a file END { Actions } # is for comments in Awk


Actions specified in the BEGIN section will be executed before starts reading the lines from the input. END actions will be performed after completing the reading and processing the lines from the input. $ awk 'BEGIN {print "Name\tDesignation\tDepartment\tSalary";} > {print $2,"\t",$3,"\t",$4,"\t",$NF;} > END{print "Report Generated\n--------------"; > }' employee.txt Name Designation Department Salary Thomas Manager Sales $5,000 Jason Developer Technology $5,500 Sanjay Sysadmin Technology $7,000 Nisha Manager Marketing $9,500 Randy DBA Technology $6,000 Report Generated --------------

In the above example, it prints headline and last file for the reports.

Awk Example 5. Find the employees who has employee id greater than 200 $ awk '$1 >200' employee.txt 300 Sanjay Sysadmin Technology 400 Nisha Manager Marketing 500 Randy DBA Technology

$7,000 $9,500 $6,000

In the above example, first field ($1) is employee id. So if $1 is greater than 200, then just do the default print action to print the whole line.

Awk Example 6. Print the list of employees in Technology department Now department name is available as a fourth field, so need to check if $4 matches with the string ―Technology‖, if yes print the line. $ awk '$4 ~/Technology/' employee.txt 200 Jason Developer Technology $5,500 300 Sanjay Sysadmin Technology $7,000 500 Randy DBA Technology $6,000

Operator ~ is for comparing with the regular expressions. If it matches the default action i.e print whole line will be performed.

Awk Example 7. Print number of employees in Technology department The below example, checks if the department is Technology, if it is yes, in the Action, just increment the count variable, which was initialized with zero in the BEGIN section. $ awk 'BEGIN { count=0;} $4 ~ /Technology/ { count++; }


END { print "Number of employees in Technology Dept =",count;}' employee.txt Number of employees in Tehcnology Dept = 3

Then at the end of the process, just print the value of count which gives you the number of employees in Technology department.


Unix Sed Tutorial:

Printing File Lines using Address and Patterns by Sasikala on September 14, 2009

Let us review how to print file lines using address and patterns in this first part of sed tutorial. We‘ll be posting several awesome sed tutorials with examples in the upcoming weeks.

Unix Sed Introduction     

sed is a ―non-interactive‖ stream-oriented editor. Since its an ―non-interactive‖ it can be used to automate editing if desired. The name sed is an abbreviation for stream editor, and the utility derives many of its commands from the ed line-editor (ed was the first UNIX text editor). This allows you to edit multiple files, or to perform common editing operations without ever having to open vi or emacs. sed reads from a file or from its standard input, and outputs to its standard output. sed has two buffers which are called pattern buffer and hold buffer. Both are initially empty.

Unix Sed Working methodology This is called as one execution cycle. Cycle continues till end of file/input is reached.


1. 2. 3. 4. 5.

Read a entire line from stdin/file. Removes any trailing newline. Places the line, in its pattern buffer. Modify the pattern buffer according to the supplied commands. Print the pattern buffer to stdout.

Printing Operation in Sed Linux Sed command allows you to print only specific lines based on the line number or pattern matches. “p” is a command for printing the data from the pattern buffer. To suppress automatic printing of pattern space use -n command with sed. sed -n option will not print anything, unless an explicit request to print is found. Syntax: # sed -n 'ADDRESS'p filename # sed -n '/PATTERN/p' filename

Let us first create thegeekstuff.txt file that will be used in all the examples mentioned below. # cat thegeekstuff.txt 1. Linux - Sysadmin, Scripting etc. 2. Databases - Oracle, mySQL etc. 3. Hardware 4. Security (Firewall, Network, Online Security etc) 5. Storage 6. Cool gadgets and websites 7. Productivity (Too many technologies to explore, not much time available) 8. Website Design 9. Software Development 10.Windows- Sysadmin, reboot etc.

5 Sed ADDRESS Format Examples Sed Address Format 1: NUMBER This will match only Nth line in the input. # sed -n „N‟p filename For example, 3p prints third line of input file thegeekstuff.txt as shown below. # sed -n '3'p thegeekstuff.txt 3. Hardware

Sed Address Format 2: NUMBER1~NUMBER2


M~N with ―p‖ command prints every Nth line starting from line M. # sed -n „M~N‟p filename For example, 3~2p prints every 2nd line starting from 3rd line as shown below. # sed -n '3~2'p thegeekstuff.txt 3. Hardware 5. Storage 7. Productivity (Too many technologies to explore, not much time available) 9. Software Development

Sed Address Format 3: START,END M,N with ―p‖ command prints Mth line to Nth line. # sed -n „M,N‟p filename For example, 4,8p prints from 4th line to 8th line from input file thegeekstuff.txt # sed -n '4,8'p thegeekstuff.txt 4. Security (Firewall, Network, Online Security etc) 5. Storage 6. Cool gadgets and websites 7. Productivity (Too many technologies to explore, not much time available) 8. Website Design

Sed Address Format 4: „$‟ Last Line $ with ―p‖ command matches only the last line from the input. # sed -n „$‟p filename For example, $p prints only the last line as shown below. # sed -n '$'p thegeekstuff.txt 10.Windows- Sysadmin, reboot etc.

Sed Address Format 5: NUMBER,$ N,$ with ―p‖ command prints from Nth line to end of file. # sed -n „N,$p‟ filename For example 4,$p prints from 4th line to end of file. # sed -n '4,$p' thegeekstuff.txt 4. Security (Firewall, Network, Online Security etc) 5. Storage


6. Cool gadgets and websites 7. Productivity (Too many technologies to explore, not much time available) 8. Website Design 9. Software Development 10.Windows- Sysadmin, reboot etc.

6 Sed PATTERN Format Examples Sed Pattern Format 1: PATTERN PATTERN could be unix regular expression. The below command prints only the line which matches the given pattern. # sed -n /PATTERN/p filename For example, following prints the line only which matches the pattern ―Sysadmin‖. # sed -n /Sysadmin/p thegeekstuff.txt 1. Linux - Sysadmin, Scripting etc. 10.Windows- Sysadmin, reboot etc.

Sed Pattern Format 2: /PATTERN/,ADDRESS # sed -n „/PATTERN/,Np‟ filename For example, following prints lines which matches the pattern to Nth line, from input. 3rd line matches the pattern ―Hardware‖, so it prints from 3rd line to 6th line. # sed -n '/Hardware/,6p' thegeekstuff.txt 3. Hardware 4. Security (Firewall, Network, Online Security etc) 5. Storage 6. Cool gadgets and websites

Sed Pattern Format 3: ADDRESS,/PATTERN/ It prints from the Nth line of the input, to the line which matches the pattern. If the pattern doesnt match, it prints upto end of the input. # sed -n „N,/PATTERN/p‟ filename For example, 4th line matches the pattern ―Security‖, so it prints from 3rd line to 4th line. # sed -n '3,/Security/p' thegeekstuff.txt 3. Hardware 4. Security (Firewall, Network, Online Security etc)


Sed Pattern Format 4: /PATTERN/,$ It prints from the line matches the given pattern to end of file. # sed -n „/PATTERN/,$p‟ filename # sed -n '/Website/,$p' thegeekstuff.txt 8. Website Design 9. Software Development 10.Windows- Sysadmin, reboot etc.

Sed Pattern Format 5: /PATTERN/,+N It prints the lines which matches the pattern and next N lines following the matched line. # sed -n „/PATTERN/,+Np‟ filename For example, following prints the 5th line which matches the pattern /Storage/ and next two lines following /Storage/. # sed -n '/Storage/,+2p' thegeekstuff.txt 5. Storage 6. Cool gadgets and websites 7. Productivity (Too many technologies to explore, not much time available)

Sed Pattern Format 6: /PATTERN/,/PATTERN/ Prints the section of file between two regular expression (including the matched line ). # sed -n „/P1/,/P2/p‟ filename For example, 5th line matches ―Storage‖ and 8th line matches ―Design‖, so it prints 5th to 8th. # sed -n '/Storage/,/Design/p' thegeekstuff.txt 5. Storage 6. Cool gadgets and websites 7. Productivity (Too many technologies to explore, not much time available) 8. Website Design

Advanced Sed Substitution Examples by Sasikala on October 26, 2009


This article is part of the on-going Unix Sed Tips and Tricks series. In our previous sed articles we learned — sed printing, sed deletion, sed substitute , sed file write, and sed multiple commands. In this article, let us review some interesting workarounds with the ―s‖ substitute command in sed with several practical examples.

I. Sed Substitution Delimiter As we discussed in our previous post, we can use the different delimiters such as @ % | ; : in sed substitute command. Let us first create path.txt file that will be used in all the examples mentioned below. $ cat path.txt /usr/kbos/bin:/usr/local/bin:/usr/jbin:/usr/bin:/usr/sas/bin /usr/local/sbin:/sbin:/bin/:/usr/sbin:/usr/bin:/opt/omni/bin: /opt/omni/lbin:/opt/omni/sbin:/root/bin

Example 1 – sed @ delimiter: Substitute /opt/omni/lbin to /opt/tools/bin When you substitute a path name which has ‗/‘, you can use @ as a delimiter instead of ‗/‘. In the sed example below, in the last line of the input file, /opt/omni/lbin was changed to /opt/tools/bin. $ sed 's@/opt/omni/lbin@/opt/tools/bin@g' path.txt /usr/kbos/bin:/usr/local/bin:/usr/jbin/:/usr/bin:/usr/sas/bin /usr/local/sbin:/sbin:/bin/:/usr/sbin:/usr/bin:/opt/omni/bin: /opt/tools/bin:/opt/omni/sbin:/root/bin

Example 2 – sed / delimiter: Substitute /opt/omni/lbin to /opt/tools/bin When you should use ‗/‘ in path name related substitution, you have to escape ‗/‘ in the substitution data as shown below. In this sed example, the delimiter ‗/‘ was escaped in the REGEXP and REPLACEMENT part.


$ sed 's/\/opt\/omni\/lbin/\/opt\/tools\/bin/g' path.txt /usr/kbos/bin:/usr/local/bin:/usr/jbin/:/usr/bin:/usr/sas/bin /usr/local/sbin:/sbin:/bin/:/usr/sbin:/usr/bin:/opt/omni/bin: /opt/tools/bin:/opt/omni/sbin:/root/bin

II. Sed „&‟ Get Matched String The precise part of an input line on which the Regular Expression matches is represented by &, which can then be used in the replacement part.

Example 1 – sed & Usage: Substitute /usr/bin/ to /usr/bin/local $ sed 's@/usr/bin@&/local@g' path.txt /usr/kbos/bin:/usr/local/bin:/usr/jbin/:/usr/bin/local:/usr/sas/bin /usr/local/sbin:/sbin:/bin/:/usr/sbin:/usr/bin/local:/opt/omni/bin: /opt/omni/lbin:/opt/omni/sbin:/root/bin

In the above example ‗&‘ in the replacement part will replace with /usr/bin which is matched pattern and add it with /local. So in the output all the occurrance of /usr/bin will be replaced with /usr/bin/local

Example 2 – sed & Usage: Match the whole line & replaces whatever matches with the given REGEXP. $ sed 's@^.*$@<<<&>>>@g' path.txt <<</usr/kbos/bin:/usr/local/bin:/usr/jbin/:/usr/bin:/usr/sas/bin>>> <<</usr/local/sbin:/sbin:/bin/:/usr/sbin:/usr/bin:/opt/omni/bin:>>> <<</opt/omni/lbin:/opt/omni/sbin:/root/bin>>>

In the above example regexp has ―^.*$‖ which matches the whole line. Replacement part <<<&>>> writes the whole line with <<< and >>> in the beginning and end of the line respectively.

III. Grouping and Back-references in Sed Grouping can be used in sed like normal regular expression. A group is opened with ―\(‖ and closed with ―\)‖.Grouping can be used in combination with back-referencing. Back-reference is the re-use of a part of a Regular Expression selected by grouping. Backreferences in sed can be used in both a Regular Expression and in the replacement part of the substitute command.

Example 1: Get only the first path in each line $ sed 's/\(\/[^:]*\).*/\1/g' path.txt /usr/kbos/bin /usr/local/sbin


/opt/omni/lbin

In the above example, \(\/[^:]*\) matches the path available before first : comes. \1 replaces the first matched group.

Example 2: Multigrouping In the file path.txt change the order of field in the last line of the file. $ sed '$s@\([^:]*\):\([^:]*\):\([^:]*\)@\3:\2:\1@g' path.txt /usr/kbos/bin:/usr/local/bin:/usr/jbin:/usr/bin:/usr/sas/bin /usr/local/sbin:/sbin:/bin:/usr/sbin:/usr/bin:/opt/omni/bin: /root/bin:/opt/omni/sbin:/opt/omni/lbin

In the above command $ specifies substitution to happen only for the last line.Output shows that the order of the path values in the last line has been reversed.

Example 3: Get the list of usernames in /etc/passwd file This sed example displays only the first field from the /etc/passwd file. $sed 's/\([^:]*\).*/\1/' /etc/passwd root bin daemon adm lp sync shutdown

Example 4: Parenthesize first character of each word This sed example prints the first character of every word in paranthesis. $ echo "Welcome To The Geek Stuff" | sed 's/\(\b[A-Z]\)/\(\1\)/g' (W)elcome (T)o (T)he (G)eek (S)tuff

Example 5: Commify the simple number. Let us create file called numbers which has list of numbers. The below sed command example is used to commify the numbers till thousands. $ cat 1234 12121 3434 123

numbers

$sed 's/\(^\|[^0-9.]\)\([0-9]\+\)\([0-9]\{3\}\)/\1\2,\3/g' numbers 1,234


12,121 3,434 123


AWK  

Command Line

Advertisement
Read more
Read more
Similar to
Popular now
Just for you