summaryrefslogtreecommitdiffhomepage
path: root/awk.html.markdown
diff options
context:
space:
mode:
Diffstat (limited to 'awk.html.markdown')
-rw-r--r--awk.html.markdown359
1 files changed, 359 insertions, 0 deletions
diff --git a/awk.html.markdown b/awk.html.markdown
new file mode 100644
index 00000000..90f88b1a
--- /dev/null
+++ b/awk.html.markdown
@@ -0,0 +1,359 @@
+---
+language: awk
+filename: learnawk.awk
+contributors:
+ - ["Marshall Mason", "http://github.com/marshallmason"]
+lang: en
+---
+
+AWK is a standard tool on every POSIX-compliant UNIX system. It's like a
+stripped-down Perl, perfect for text-processing tasks and other scripting
+needs. It has a C-like syntax, but without semicolons, manual memory
+management, or static typing. It excels at text processing. You can call to it
+from a shell script, or you can use it as a stand-alone scripting language.
+
+Why use AWK instead of Perl? Mostly because AWK is part of UNIX. You can always
+count on it, whereas Perl's future is in question. AWK is also easier to read
+than Perl. For simple text-processing scripts, particularly ones that read
+files line by line and split on delimiters, AWK is probably the right tool for
+the job.
+
+```awk
+#!/usr/bin/awk -f
+
+# Comments are like this
+
+# AWK programs consist of a collection of patterns and actions. The most
+# important pattern is called BEGIN. Actions go into brace blocks.
+BEGIN {
+
+ # BEGIN will run at the beginning of the program. It's where you put all
+ # the preliminary set-up code, before you process any text files. If you
+ # have no text files, then think of BEGIN as the main entry point.
+
+ # Variables are global. Just set them or use them, no need to declare..
+ count = 0
+
+ # Operators just like in C and friends
+ a = count + 1
+ b = count - 1
+ c = count * 1
+ d = count / 1
+ e = count % 1 # modulus
+ f = count ^ 1 # exponentiation
+
+ a += 1
+ b -= 1
+ c *= 1
+ d /= 1
+ e %= 1
+ f ^= 1
+
+ # Incrementing and decrementing by one
+ a++
+ b--
+
+ # As a prefix operator, it returns the incremented value
+ ++a
+ --b
+
+ # Notice, also, no punctuation such as semicolons to terminate statements
+
+ # Control statements
+ if (count == 0)
+ print "Starting with count of 0"
+ else
+ print "Huh?"
+
+ # Or you could use the ternary operator
+ print (count == 0) ? "Starting with count of 0" : "Huh?"
+
+ # Blocks consisting of multiple lines use braces
+ while (a < 10) {
+ print "String concatenation is done" " with a series" " of"
+ " space-separated strings"
+ print a
+
+ a++
+ }
+
+ for (i = 0; i < 10; i++)
+ print "Good ol' for loop"
+
+ # As for comparisons, they're the standards:
+ a < b # Less than
+ a <= b # Less than or equal
+ a != b # Not equal
+ a == b # Equal
+ a > b # Greater than
+ a >= b # Greater than or equal
+
+ # Logical operators as well
+ a && b # AND
+ a || b # OR
+
+ # In addition, there's the super useful regular expression match
+ if ("foo" ~ "^fo+$")
+ print "Fooey!"
+ if ("boo" !~ "^fo+$")
+ print "Boo!"
+
+ # Arrays
+ arr[0] = "foo"
+ arr[1] = "bar"
+ # Unfortunately, there is no other way to initialize an array. Ya just
+ # gotta chug through every value line by line like that.
+
+ # You also have associative arrays
+ assoc["foo"] = "bar"
+ assoc["bar"] = "baz"
+
+ # And multi-dimensional arrays, with some limitations I won't mention here
+ multidim[0,0] = "foo"
+ multidim[0,1] = "bar"
+ multidim[1,0] = "baz"
+ multidim[1,1] = "boo"
+
+ # You can test for array membership
+ if ("foo" in assoc)
+ print "Fooey!"
+
+ # You can also use the 'in' operator to traverse the keys of an array
+ for (key in assoc)
+ print assoc[key]
+
+ # The command line is in a special array called ARGV
+ for (argnum in ARGV)
+ print ARGV[argnum]
+
+ # You can remove elements of an array
+ # This is particularly useful to prevent AWK from assuming the arguments
+ # are files for it to process
+ delete ARGV[1]
+
+ # The number of command line arguments is in a variable called ARGC
+ print ARGC
+
+ # AWK has several built-in functions. They fall into three categories. I'll
+ # demonstrate each of them in their own functions, defined later.
+
+ return_value = arithmetic_functions(a, b, c)
+ string_functions()
+ io_functions()
+}
+
+# Here's how you define a function
+function arithmetic_functions(a, b, c, localvar) {
+
+ # Probably the most annoying part of AWK is that there are no local
+ # variables. Everything is global. For short scripts, this is fine, even
+ # useful, but for longer scripts, this can be a problem.
+
+ # There is a work-around (ahem, hack). Function arguments are local to the
+ # function, and AWK allows you to define more function arguments than it
+ # needs. So just stick local variable in the function declaration, like I
+ # did above. As a convention, stick in some extra whitespace to distinguish
+ # between actual function parameters and local variables. In this example,
+ # a, b, and c are actual parameters, while d is merely a local variable.
+
+ # Now, to demonstrate the arithmetic functions
+
+ # Most AWK implementations have some standard trig functions
+ localvar = sin(a)
+ localvar = cos(a)
+ localvar = atan2(a, b) # arc tangent of b / a
+
+ # And logarithmic stuff
+ localvar = exp(a)
+ localvar = log(a)
+
+ # Square root
+ localvar = sqrt(a)
+
+ # Truncate floating point to integer
+ localvar = int(5.34) # localvar => 5
+
+ # Random numbers
+ srand() # Supply a seed as an argument. By default, it uses the time of day
+ localvar = rand() # Random number between 0 and 1.
+
+ # Here's how to return a value
+ return localvar
+}
+
+function string_functions( localvar, arr) {
+
+ # AWK, being a string-processing language, has several string-related
+ # functions, many of which rely heavily on regular expressions.
+
+ # Search and replace, first instance (sub) or all instances (gsub)
+ # Both return number of matches replaced
+ localvar = "fooooobar"
+ sub("fo+", "Meet me at the ", localvar) # localvar => "Meet me at the bar"
+ gsub("e+", ".", localvar) # localvar => "m..t m. at th. bar"
+
+ # Search for a string that matches a regular expression
+ # index() does the same thing, but doesn't allow a regular expression
+ match(localvar, "t") # => 4, since the 't' is the fourth character
+
+ # Split on a delimiter
+ split("foo-bar-baz", arr, "-") # a => ["foo", "bar", "baz"]
+
+ # Other useful stuff
+ sprintf("%s %d %d %d", "Testing", 1, 2, 3) # => "Testing 1 2 3"
+ substr("foobar", 2, 3) # => "oob"
+ substr("foobar", 4) # => "bar"
+ length("foo") # => 3
+ tolower("FOO") # => "foo"
+ toupper("foo") # => "FOO"
+}
+
+function io_functions( localvar) {
+
+ # You've already seen print
+ print "Hello world"
+
+ # There's also printf
+ printf("%s %d %d %d\n", "Testing", 1, 2, 3)
+
+ # AWK doesn't have file handles, per se. It will automatically open a file
+ # handle for you when you use something that needs one. The string you used
+ # for this can be treated as a file handle, for purposes of I/O. This makes
+ # it feel sort of like shell scripting:
+
+ print "foobar" >"/tmp/foobar.txt"
+
+ # Now the string "/tmp/foobar.txt" is a file handle. You can close it:
+ close("/tmp/foobar.txt")
+
+ # Here's how you run something in the shell
+ system("echo foobar") # => prints foobar
+
+ # Reads a line from standard input and stores in localvar
+ getline localvar
+
+ # Reads a line from a pipe
+ "echo foobar" | getline localvar # localvar => "foobar"
+ close("echo foobar")
+
+ # Reads a line from a file and stores in localvar
+ getline localvar <"/tmp/foobar.txt"
+ close("/tmp/foobar.txt")
+}
+
+# As I said at the beginning, AWK programs consist of a collection of patterns
+# and actions. You've already seen the all-important BEGIN pattern. Other
+# patterns are used only if you're processing lines from files or standard
+# input.
+#
+# When you pass arguments to AWK, they are treated as file names to process.
+# It will process them all, in order. Think of it like an implicit for loop,
+# iterating over the lines in these files. these patterns and actions are like
+# switch statements inside the loop.
+
+/^fo+bar$/ {
+
+ # This action will execute for every line that matches the regular
+ # expression, /^fo+bar$/, and will be skipped for any line that fails to
+ # match it. Let's just print the line:
+
+ print
+
+ # Whoa, no argument! That's because print has a default argument: $0.
+ # $0 is the name of the current line being processed. It is created
+ # automatically for you.
+
+ # You can probably guess there are other $ variables. Every line is
+ # implicitely split before every action is called, much like the shell
+ # does. And, like the shell, each field can be access with a dollar sign
+
+ # This will print the second and fourth fields in the line
+ print $2, $4
+
+ # AWK automatically defines many other variables to help you inspect and
+ # process each line. The most important one is NF
+
+ # Prints the number of fields on this line
+ print NF
+
+ # Print the last field on this line
+ print $NF
+}
+
+# Every pattern is actually a true/false test. The regular expression in the
+# last pattern is also a true/false test, but part of it was hidden. If you
+# don't give it a string to test, it will assume $0, the line that it's
+# currently processing. Thus, the complete version of it is this:
+
+$0 ~ /^fo+bar$/ {
+ print "Equivalent to the last pattern"
+}
+
+a > 0 {
+ # This will execute once for each line, as long as a is positive
+}
+
+# You get the idea. Processing text files, reading in a line at a time, and
+# doing something with it, particularly splitting on a delimiter, is so common
+# in UNIX that AWK is a scripting language that does all of it for you, without
+# you needing to ask. All you have to do is write the patterns and actions
+# based on what you expect of the input, and what you want to do with it.
+
+# Here's a quick example of a simple script, the sort of thing AWK is perfect
+# for. It will read a name from standard input and then will print the average
+# age of everyone with that first name. Let's say you supply as an argument the
+# name of a this data file:
+#
+# Bob Jones 32
+# Jane Doe 22
+# Steve Stevens 83
+# Bob Smith 29
+# Bob Barker 72
+#
+# Here's the script:
+
+BEGIN {
+
+ # First, ask the user for the name
+ print "What name would you like the average age for?"
+
+ # Get a line from standard input, not from files on the command line
+ getline name <"/dev/stdin"
+}
+
+# Now, match every line whose first field is the given name
+$1 == name {
+
+ # Inside here, we have access to a number of useful variables, already
+ # pre-loaded for us:
+ # $0 is the entire line
+ # $3 is the third field, the age, which is what we're interested in here
+ # NF is the number of fields, which should be 3
+ # NR is the number of records (lines) seen so far
+ # FILENAME is the name of the file being processed
+ # FS is the field separator being used, which is " " here
+ # ...etc. There are plenty more, documented in the man page.
+
+ # Keep track of a running total and how many lines matched
+ sum += $3
+ nlines++
+}
+
+# Another special pattern is called END. It will run after processing all the
+# text files. Unlike BEGIN, it will only run if you've given it input to
+# process. It will run after all the files have been read and processed
+# according to the rules and actions you've provided. The purpose of it is
+# usually to output some kind of final report, or do something with the
+# aggregate of the data you've accumulated over the course of the script.
+
+END {
+ if (nlines)
+ print "The average age for " name " is " sum / nlines
+}
+
+```
+Further Reading:
+
+* [Awk tutorial](http://www.grymoire.com/Unix/Awk.html)
+* [Awk man page](https://linux.die.net/man/1/awk)
+* [The GNU Awk User's Guide](https://www.gnu.org/software/gawk/manual/gawk.html) GNU Awk is found on most Linux systems.