How To Use File and Standard Input CLI Arguments with Python Click Package

By Adam McQuistan in Python  12/07/2021 Comment

Introduction

In this short "How To" article I demonstrate implementing a Python based Command Line Interface (CLI) application using the Click package to consume text input from both files and standard input streams. In demonstrating how to approach this challenge I create a word frequency CLI program named wf which counts the frequency of words in a given input source.

For those interested in learning more about the awesome Click package please consider taking my course Building Python CLI Apps with Click.

Prefer Video? Watch on Youtube

Project Setup

I start off in an empty directory named click-wf-demo.

mkdir click-wf-demo
cd click-wf-demo

Then inside this directory I create a sample text file named martin-fowler.txt to count word frequencies which is simply a quote from famous software engineer Martin Fowler

Any fool can write code that a computer can understand.
Good programmers write code that humans can understand.

Next I create a setup.py file which will contain an implementation of the setup(...) function from the setuptools module. This allows me to generate a installable package with a console script named wf and specify that the Click package is a dependency. Its worth calling out that the entry_points argument maps a wf CLI program to the main(...) function inside a wf.py Python module.

# setup.py
from setuptools import setup

setup(
  name='wf',
  version='1.0.0',
  py_modules=['wf'],
  python_requires=">=3.6",
  install_requires=['Click'],
  entry_points={
    'console_scripts': [
      'wf=wf:main'
    ]
  }
)

Next I create a Python module (aka Python file) named wf.py and scaffold out the following basic program structure stubbed out.

# wf.py

import click


@click.command()
def main():
    """Counts word frequencies in given file
    or standard input."""
    pass



if __name__ == '__main__':
    main()

And as a final piece of setup I create a Python virtual environment, activate it, then install this wf package as an editable package using pip as seen below.

python3 -m venv venv
source venv/bin/active # or .\venv\Scripts\activate.bat on windows CMD
pip install -e .

I can verify the wf CLI program was installed like so.

wf --help

Output.

Usage: wf [OPTIONS]

  Counts word frequencies in given file or standard input.

Options:
  --help  Show this message and exit.

Adding File Argument to wf Program

The Click framework provides a decorator named argument(...) that can be typed to the special click.File type to intelligently handle file arguments in Click commands. Below in an updated version of the wf.py source code which adds a file argument which gets opened in read mode. I've also added the implementation for counting and displaying the word frequencies in the input file.

# wf.py

import string
import click


@click.command()
@click.argument('file', type=click.File('r'))
def main(file):
    """Counts word frequencies in given file
    or standard input."""
    words = {}
    for line in file:
        for word in line.strip().split():
            word = word.lower()\
                    .translate(str.maketrans('', '', string.punctuation))
            words[word] = words.get(word, 0) + 1

    words_sorted = sorted(words.items(),
                          key=lambda item: item[1],
                          reverse=True)

    for word, cnt in words_sorted:
        click.echo(f"{word:^12} {cnt:<4}")


if __name__ == '__main__':
    main()

Testing this with my demo martin-fowler.txt file.

wf martin-fowler.txt

Gives this output.

    can      3   
   write     2   
    code     2   
    that     2   
 understand  2   
    any      1   
    fool     1   
     a       1   
  computer   1   
    good     1   
programmers  1   
   humans    1 

Adding Standard Input Handling to wf Program

A common function of many CLI programs that work with files as arguments are to also work with text that has been redirected in from standard input sources. This could be fed in from a file or just a string of text. As it stands right now my wf program doesn't possess this capability.

For example if I redirect text in from the same martin-fowler.txt sample file I get the following error.

wf < martin-fowler.txt

Or if I pipe the contents into the wf command.

cat martin-fowler.txt | wf

I get the following error.

Usage: wf [OPTIONS] FILE
Try 'wf --help' for help.

Error: Missing argument 'FILE'.

In both cases the error message actually is a rather helpful clue to what is wrong and how I might update the program to accept standard input. The solution is to add a default source to the Click argument() decorator which is what I've done below.

# wf.py

import string
import sys

import click


@click.command()
@click.argument('file', type=click.File('r'), default=sys.stdin)
def main(file):
    """Counts word frequencies in given file
    or standard input."""
    words = {}
    for line in file:
        for word in line.strip().split():
            word = word.lower()\
                    .translate(str.maketrans('', '', string.punctuation))
            words[word] = words.get(word, 0) + 1

    words_sorted = sorted(words.items(),
                          key=lambda item: item[1],
                          reverse=True)

    for word, cnt in words_sorted:
        click.echo(f"{word:^12} {cnt:<4}")


if __name__ == '__main__':
    main()

Now lets recheck the program.

wf < martin-fowler.txt

And for completeness I cat the file and pipe it to the wf command like so.

cat martin-fowler.txt | wf

Now in both cases I get the expected output.

    can      3   
   write     2   
    code     2   
    that     2   
 understand  2   
    any      1   
    fool     1   
     a       1   
  computer   1   
    good     1   
programmers  1   
   humans    1

Conclusion

In this How To article I used an example CLI program which utilizes the Python based Click library to consume text input data from a file or standard input then count and display word frequencies.