Python Clone of Unix wc Program using Click Framework
By
Introduction
In this article I demonstrate the power of the Python based Click package for building beautiful Command Line Interface (CLI) programs by creating a clone of the popular wc Unix program. The wc program is a simple program used to count lines, words, and bytes present in files and serves as a perfect example to demonstrate the basics of the Click CLI framework. For those interested in learning more about the awesome Click package please consider taking my course Building Python CLI Apps with Click.
Prefer Video? Watch on Youtube
Understanding the Requirements
The first place to start in any software project is with a clear understanding of the requirements of what is to be accomplished. In this example it is pretty clear since there is a working example complete with well articulated documentation.
Pulling up the wc man page like so.
man wcOn my macOS Monterey gives the following docs.
WC(1) General Commands Manual WC(1)
NAME
wc – word, line, character, and byte count
SYNOPSIS
wc [-clmw] [file ...]
DESCRIPTION
The wc utility displays the number of lines, words, and bytes contained in
each input file, or standard input (if no file is specified) to the
standard output. A line is defined as a string of characters delimited by
a ⟨newline⟩ character. Characters beyond the final ⟨newline⟩ character
will not be included in the line count.
A word is defined as a string of characters delimited by white space
characters. White space characters are the set of characters for which the
iswspace(3) function returns true. If more than one input file is
specified, a line of cumulative counts for all the files is displayed on a
separate line after the output for the last file.
The following options are available:
-c The number of bytes in each input file is written to the standard
output. This will cancel out any prior usage of the -m option.
-l The number of lines in each input file is written to the standard
output.
-m The number of characters in each input file is written to the
standard output. If the current locale does not support multibyte
characters, this is equivalent to the -c option. This will cancel
out any prior usage of the -c option.
-w The number of words in each input file is written to the standard
output.
When an option is specified, wc only reports the information requested by
that option. The order of output always takes the form of line, word,
byte, and file name. The default action is equivalent to specifying the
-c, -l and -w options.
If no files are specified, the standard input is used and no file name is
displayed. The prompt will accept input until receiving EOF, or [^D] in
most environments.
ENVIRONMENT
The LANG, LC_ALL and LC_CTYPE environment variables affect the execution of
wc as described in environ(7).
EXIT STATUS
The wc utility exits 0 on success, and >0 if an error occurs.
EXAMPLES
Count the number of characters, words and lines in each of the files
report1 and report2 as well as the totals for both:
wc -mlw report1 report2An example would be useful here so, I'll create the following two demo files.
First file is named martin-fowler.txt and is a quote from Martin Fowler
Any fool can write code that a computer can understand.
Good programmers write code that humans can understand.Second file is named linus-torvalds.txt and is quote by Linus Torvalds
Any program is only as good as it is useful. 😊Then run them through wc.
wc martin-fowler.txt linus-torvalds.txtOutput.
2 18 112 martin-fowler.txt
1 10 45 linus-torvalds.txt
3 28 157 totalSetting Up the pywc Click Project
For this simple project I'll need just two Python files. One will be the pywc.py which will contain the source code for the pywc clone of the wc program. The second file is a setup.py file which will use setuptools to create an installable and executable programs.
I start by filling out the setup.py file as shown below.
# setup.py
from setuptools import setup
setup(
name='pywc',
version='1.0.0',
py_modules=['pywc'],
python_requires=">=3.6",
install_requires=['Click>=8.0.0'],
entry_points={
'console_scripts': [
'pywc=pywc:main'
]
}
)The above setup(...) function establishes the name as pywc for the installable package, specifies that it requires at least Python 3.6 (because I'll use f-strings) plus the Click library of at least version 8.0.0 and the pywc command will map to a main(...) function in a module named pywc.py
Next I scaffold out the pywc.py module with a main() function that provides a minimal function doc-string. The main() function is decorated with the @click.command() decorator which tells click to treat this function as a command.
# pywc.py
import click
@click.command()
def main():
"""Python clone of Unix wc program."""
pass
if __name__ == '__main__':
main()Next I create a Python virtual environment with the built in venv module then activate it.
python3 -m venv venv
source venv/bin/active # use .\venv\Scripts\activate.bat if on windowsAt this point my current working directory looks as follows.
.
├── venv/
├── linus-torvalds.txt
├── martin-fowler.txt
├── pywc.py
└── setup.pyNow with the Python virtual environment active I can install the pywc package in editable mode.
pip isntall -e .At this point I should be able the pull up the help page of the base, unimplemented, pywc command as follows.
pywc --helpOutput.
Usage: pywc [OPTIONS]
Python clone of Unix wc program.
Options:
--help Show this message and exit.This concludes the basic setup of the project. I can now progress on to implementing the functionality.
Adding an Argument for File Handing
A good starting point is to add a command argument for specifying either a file or standard input for the program to consume. The Click library provides a decorator named argument for this purpose.
# pywc.py
import click
class DataRow:
def __init__(self, name):
self.name = name
self.lines = 0
self.chars = 0
self.words = 0
self.bytes = 0
@click.command()
@click.argument('inputs', type=click.File('r'), nargs=-1)
def main(inputs):
"""Python clone of Unix wc program."""
for file in inputs:
row = DataRow(file.name)
for line in file:
click.echo(f"{row:>18}-> {line}", nl=False)
if __name__ == '__main__':
main()This first update uses the argument decorator of the Click library to specify that the command requires an argument of type click.File. This file based argument maps to the main(...) function's argument named inputs and represents a colletion of input files which is specified by using the nargs keyword parameter set to -1 but, I could have also restricted this to a hard value like 1,2 3, ect ... I also provided a temporary implementation of simply iterating over the contents of each input file and use the click.echo(...) function to display each line of input in the source files.
If I run the pywc program on my two demo files like so.
pywc martin-fowler.txt linus-torvalds.txtI get the following output.
martin-fowler.txt-> Any fool can write code that a computer can understand.
martin-fowler.txt-> Good programmers write code that humans can understand.
linus-torvalds.txt-> Any program is only as good as it is useful.Adding Option Flags
Next I need to add options flags for counting bytes (-c), lines (-l), characters (-m), and words (-w). To accomplish this I'll add a option(...) decorator from the Click library above the main method for each option along with a help keyword parameter for documenting them. Since these are boolean flags I use the is_flag=True keyword argument on the option decorator.
# pywc.py
import click
class DataRow:
def __init__(self, name):
self.name = name
self.lines = 0
self.chars = 0
self.words = 0
self.bytes = 0
def output(self,
show_lines=False,
show_chars=False,
show_words=False,
show_bytes=False
):
num_counts = sum([show_lines, show_chars, show_words, show_bytes])
fmts = ['{:>8}'] * num_counts
fmts.append('{:<20}')
fmt = ' '.join(fmts)
fields = []
if show_lines:
fields.append(self.lines)
if show_words:
fields.append(self.words)
if show_bytes:
fields.append(self.bytes)
if show_chars:
fields.append(self.chars)
fields.append(self.name)
return fmt.format(*fields)
@click.command()
@click.argument('inputs', type=click.File('r'), nargs=-1)
@click.option('-c', help='Count bytes in each input file.', is_flag=True)
@click.option('-l', help='Count lines in each input file.', is_flag=True)
@click.option('-m', help='Count characters in each input file.', is_flag=True)
@click.option('-w', help='Count words in each input file.', is_flag=True)
def main(inputs, c, l, m , w):
"""Python clone of Unix wc program."""
show_default = not any([c, l, m , w])
total_row = DataRow('total')
for file in inputs:
row = DataRow(file.name)
for line in file:
row.bytes += len(line.encode('utf-8'))
row.lines += 1
row.chars += len(line)
row.words += len(line.strip().split())
total_row.bytes += row.bytes
total_row.lines += row.lines
total_row.chars += row.chars
total_row.words += row.words
click.echo(row.output(
show_lines=l or show_default,
show_bytes=c or show_default,
show_words=w or show_default,
show_chars=m
))
if len(inputs) > 1:
click.echo(total_row.output(
show_lines=l or show_default,
show_bytes=c or show_default,
show_words=w or show_default,
show_chars=m
))
if __name__ == '__main__':
main()I've also added the implementation that counts the metrics for each input file and formats the output for printing to standard out via the click.echo(...) function.
For completeness I should check the output from the wc and pywc.
First I run the two files through the wc program counting the lines and words.
wc -wl linus-torvalds.txt martin-fowler.txtOutput.
1 11 linus-torvalds.txt
2 18 martin-fowler.txt
3 29 totalNext I run them through the pywc clone.
pywc -wl linus-torvalds.txt martin-fowler.txtOutput.
1 11 linus-torvalds.txt
2 18 martin-fowler.txt
3 29 totalMission complete!
Conclusion
In this short blog post I was able to make a Python based clone of the unix wc program with a suprisingly small amount of code which is made possible through the use of the Click framework for CLI development.
As always, I thank you for reading and please feel free to ask questions or critique in the comments section below.