Welcome to cliapp’s documentation!

cliapp is a Python framework for Unix-like command line programs, which typically have the following characteristics:

  • non-interactive
  • the programs read input files named on the command line, or the standard input
  • each line of input is processed individually
  • output is to the standard output
  • there are various options to modify how the program works
  • certain options are common to all: --help, --version

Programs like the above are often used as filters in a pipeline. The scaffoling to set up a command line parser, open each input file, read each line of input, etc, is the same in each program. Only the logic of what to do with each line differs.

cliapp is not restricted to line-based filters, but is a more general framework. It provides ways for its users to override most behavior. For example:

  • you can treat command line arguments as URLs, or record identfiers in a database, or whatever you like
  • you can read input files in whatever chunks you like, or not at all, rather than forcing a line-based paradigm

Despite all the flexibility, writing simple line-based filters remains very straightforward. The point is to get the framework to do all the usual things, and avoid repeating code across users of the framework.

Example

class ExampleApp(cliapp.Application):

    def add_settings(self):
        self.settings.string_list(['pattern', 'e'],
                                  'search for regular expression PATTERN',
                                  metavar='REGEXP')

    # We override process_inputs to be able to do something after the last
    # input line.
    def process_inputs(self, args):
        self.matches = 0
        cliapp.Application.process_inputs(self, args)
        self.output.write('There were %s matches.\\n' % self.matches)

    def process_input_line(self, name, line):
        for pattern in self.settings['pattern']:
            if pattern in line:
                self.output.write('%s:%s: %s' % (name, self.lineno, line))
                self.matches += 1
                logging.debug('Match: %s line %d' % (name, self.lineno))


if __name__ == '__main__':
    ExampleApp().run()

Walkthrough

Every application should be a class that subclasses cliapp.Application. The subclass should provide specific methods. Read the documentation for the cliapp.Application class to see all methods, but a rough summary is here:

  • the settings attribute is the cliapp.Settings instance used by the application
  • override add_settings to add new settings for the application
  • override process_* methods to override various stages in how arguments and input files are processed
  • override process_args to decide how each argument is processed; by default, this called process_inputs or handles subcommands
  • process_inputs calls process_input (note singular) for each argument, or on - to process standard input if no files are named on the command line
  • process_input calls open_input to open each file, then calls process_input_line for each input line
  • process_input_line does nothing, by default

This cascade of overrideable methods is started by the run method, which also sets up logging, loads configuration files, parses the command line, and handles reporting of exceptions. It can also run the rest of the code under the Python profiler, if the appropriate environment variable is set.

Logging

Logging support: by default, no log file is written, it must be requested explicitly by the user. The default log level is info.

Subcommands

Sometimes a command line tool needs to support subcommands. For example, version control tools often do this: git commit, git clone, etc. To do this with cliapp, you need to add methods with names like cmd_commit and cmd_clone:

class VersionControlTool(cliapp.Application):

    def cmd_commit(self, args):
        '''commit command description'''
        pass
    def cmd_clone(self, args):
        '''clone command description'''
        pass

If any such methods exist, cliapp automatically supports subcommands. The name of the method, without the cmd_ prefix, forms the name of the subcommand. Any underscores in the method name get converted to dashes in the command line. Case is preserved.

Subcommands may also be added using the add_subcommand method.

All options are global, not specific to the subcommand. All non-option arguments are passed to the method in its only argument.

Subcommands are implemented by the process_args method. If you override that method, you need to support subcommands yourself (perhaps by calling the cliapp implementation).

Manual pages

cliapp provides a way to fill in a manual page template, in troff format, with information about all options. This allows you to write the rest of the manual page without having to remember to update all options. This is a compromise between ease-of-development and manual page quality.

A high quality manual page probably needs to be written from scratch. For example, the description of each option in a manual page should usually be longer than what is suitable for --help output. However, it is tedious to write option descriptions many times.

To use this, use the --generate-manpage=TEMPLATE option, where TEMPLATE is the name of the template file. See example.1 in the cliapp source tree for an example.

Profiling support

If sys.argv[0] is foo, and the environment variable FOO_PROFILE is set, then the execution of the application (the run method) is profiled, using cProfile, and the profile written to the file named in the environment variable.

Reference manual

cliapp.runcmd(argv, *args, **kwargs)

Run external command or pipeline.

Example: ``runcmd([‘grep’, ‘foo’], [‘wc’, ‘-l’],
feed_stdin=’foo

bar ‘)``

Return the standard output of the command.

Raise cliapp.AppException if external command returns non-zero exit code. *args and **kwargs are passed onto subprocess.Popen.

class cliapp.Application(progname=None, version='0.0.0', description=None, epilog=None)

A framework for Unix-like command line programs.

The user should subclass this base class for each application. The subclass does not need code for the mundane, boilerplate parts that are the same in every utility, and can concentrate on the interesting part that is unique to it.

To start the application, call the run method.

The progname argument sets tne name of the program, which is used for various purposes, such as determining the name of the configuration file.

Similarly, version sets the version number of the program.

description and epilog are included in the output of --help. They are formatted to fit the screen. Unlike the default behavior of optparse, empty lines separate paragraphs.

add_default_subcommands()
add_settings()

Add application specific settings.

add_subcommand(name, func, arg_synopsis=None)

Add a subcommand.

Normally, subcommands are defined by add cmd_foo methods to the application class. However, sometimes it is more convenient to have them elsewhere (e.g., in plugins). This method allows doing that.

The callback function must accept a list of command line non-option arguments.

app_directory()

Return the directory where the application class is defined.

Plugins are searched relative to this directory, in the subdirectory specified by self.plugin_subdir.

cleanup()

Clean up after process_args.

This method is called just after process_args. By default it does nothing, but subclasses may override it with a suitable implementation. This is easier than overriding process_args itself.

disable_plugins()
dump_memory_profile(msg)

Log memory profiling information.

Get the memory profiling method from the dump-memory-profile setting, and log the results at DEBUG level. msg is a message the caller provides to identify at what point the profiling happens.

enable_plugins()

Load plugins.

help(args)

Print help.

log_config()
open_input(name, mode='r')

Open an input file for reading.

The default behaviour is to open a file named on the local filesystem. A subclass might override this behavior for URLs, for example.

The optional mode argument speficies the mode in which the file gets opened. It should allow reading. Some files should perhaps be opened in binary mode (‘rb’) instead of the default text mode.

parse_args(args, configs_only=False)

Parse the command line.

Return list of non-option arguments.

process_args(args)

Process command line non-option arguments.

The default is to call process_inputs with the argument list, or to invoke the requested subcommand, if subcommands have been defined.

process_input(name, stdin=<open file '<stdin>', mode 'r' at 0x7ffc5ed22150>)

Process a particular input file.

The stdin argument is meant for unit test only.

process_input_line(filename, line)

Process one line of the input file.

Applications that are line-oriented can redefine only this method in a subclass, and should not need to care about the other methods.

process_inputs(args)

Process all arguments as input filenames.

The default implementation calls process_input for each input filename. If no filenames were given, then process_input is called with - as the argument name. This implements the usual Unix command line practice of reading from stdin if no inputs are named.

The attributes fileno, global_lineno, and lineno are set, and count files and lines. The global line number is the line number as if all input files were one.

run(args=None, stderr=<open file '<stderr>', mode 'w' at 0x7ffc5ed22270>, sysargv=['/usr/bin/sphinx-build', '-b', 'html', '-d', '_build/doctrees', '.', '_build/html'], log=<function critical at 0x1f77d70>)

Run the application.

runcmd(*args, **kwargs)
runcmd_unchecked(*args, **kwargs)
setup()

Prepare for process_args.

This method is called just before enabling plugins. By default it does nothing, but subclasses may override it with a suitable implementation. This is easier than overriding process_args itself.

setup_logging()

Set up logging.

setup_plugin_manager()

Create a plugin manager.

exception cliapp.AppException(msg)

Base class for application specific exceptions.

Any exceptions that are subclasses of this one get printed as nice errors to the user. Any other exceptions cause a Python stack trace to be written to stderr.

class cliapp.HookManager

Manage the set of hooks the application defines.

add_callback(name, callback)

Add a callback to a named hook.

call(name, *args, **kwargs)

Call callbacks for a named hook, using given arguments.

new(name, hook)

Add a new hook to the manager.

If a hook with that name already exists, nothing happens.

remove_callback(name, callback_id)

Remove a specific callback from a named hook.

class cliapp.Settings(progname, version, usage=None, description=None, epilog=None)

Settings for a cliapp application.

You probably don’t need to create a settings object yourself, since cliapp.Application does it for you.

Settings are read from configuration files, and parsed from the command line. Every setting has a type, name, and help text, and may have a default value as well.

For example:

settings.boolean(['verbose', 'v'], 'show what is going on')

This would create a new setting, verbose, with a shorter alias v. On the command line, the options --verbose and -v would work equally well. There can be any number of aliases.

The help text is shown if the user uses --help or --generate-manpage. You can use the metavar keyword argument to set the name shown in the generated option lists; the default name is whatever optparse decides (i.e., name of option).

Use load_configs to read configuration files, and parse_args to parse command line arguments.

The current value of a setting can be accessed by indexing the settings class:

settings['verbose']

The list of configuration files for the appliation is stored in config_files. Add or remove from the list if you wish. The files need to exist: those that don’t are silently ignored.

as_cp()

Return a ConfigParser instance with current values of settings.

boolean(names, help, default=False, **kwargs)

Add a setting with a boolean value.

build_parser(configs_only=False, arg_synopsis=None, cmd_synopsis=None)

Build OptionParser for parsing command line.

bytesize(names, help, default=0, **kwargs)

Add a setting with a size in bytes.

The user can use suffixes for kilo/mega/giga/tera/kibi/mibi/gibi/tibi.

choice(names, possibilities, help, **kwargs)

Add a setting which chooses from list of acceptable values.

An example would be an option to set debugging level to be one of a set of accepted names: debug, info, warning, etc.

The default value is the first possibility.

dump_config(output)
integer(names, help, default=0, **kwargs)

Add an integer setting.

keys()

Return canonical settings names.

load_configs(open=<built-in function open>)

Load all config files in self.config_files.

Silently ignore files that do not exist.

parse_args(args, parser=None, suppress_errors=False, configs_only=False, arg_synopsis=None, cmd_synopsis=None)

Parse the command line.

Return list of non-option arguments. args would usually be sys.argv[1:].

require(name)

Raise exception if setting has not been set.

Option must have a value, and a default value is OK.

set_from_raw_string(name, raw_string)

Set value of a setting from a raw, unparsed string value.

string(names, help, default='', **kwargs)

Add a setting with a string value.

string_list(names, help, default=None, **kwargs)

Add a setting which have multiple string values.

An example would be an option that can be given multiple times on the command line, e.g., “–exclude=foo –exclude=bar”.

class cliapp.Plugin

Base class for plugins.

A plugin MUST NOT have any side effects when it is instantiated. This is necessary so that it can be safely loaded by unit tests, and so that a user interface can allow the user to disable it, even if it is installed, with no ill effects. Any side effects that would normally happen should occur in the enable() method, and be undone by the disable() method. These methods must be callable any number of times.

The subclass MAY define the following attributes:

  • name
  • description
  • version
  • required_application_version

name is the user-visible identifier for the plugin. It defaults to the plugin’s classname.

description is the user-visible description of the plugin. It may be arbitrarily long, and can use pango markup language. Defaults to the empty string.

version is the plugin version. Defaults to ‘0.0.0’. It MUST be a sequence of integers separated by periods. If several plugins with the same name are found, the newest version is used. Versions are compared integer by integer, starting with the first one, and a missing integer treated as a zero. If two plugins have the same version, either might be used.

required_application_version gives the version of the minimal application version the plugin is written for. The first integer must match exactly: if the application is version 2.3.4, the plugin’s required_application_version must be at least 2 and at most 2.3.4 to be loaded. Defaults to 0.

description
disable()

Disable the plugin.

disable_wrapper()

Corresponds to enable_wrapper, but for disabling a plugin.

enable()

Enable the plugin.

enable_wrapper()

Enable plugin.

The plugin manager will call this method, which then calls the enable method. Plugins should implement the enable method. The wrapper method is there to allow an application to provide an extended base class that does some application specific magic when plugins are enabled or disabled.

name
required_application_version
setup()

Setup plugin.

This is called at plugin load time. It should not yet enable the plugin (the enable method does that), but it might do things like add itself into a hook that adds command line arguments to the application.

version
class cliapp.FilterHook

A hook which filters data through callbacks.

Every hook of this type accepts a piece of data as its first argument Each callback gets the return value of the previous one as its argument. The caller gets the value of the final callback.

Other arguments (with or without keywords) are passed as-is to each callback.

call_callbacks(data, *args, **kwargs)
cliapp.runcmd_unchecked(argv, *argvs, **kwargs)

Run external command or pipeline.

Return the exit code, and contents of standard output and error of the command.

See also runcmd.

class cliapp.PluginManager

Manage plugins.

This class finds and loads plugins, and keeps a list of them that can be accessed in various ways.

The locations are set via the locations attribute, which is a list.

When a plugin is loaded, an instance of its class is created. This instance is initialized using normal and keyword arguments specified in the plugin manager attributes plugin_arguments and plugin_keyword_arguments.

The version of the application using the plugin manager is set via the application_version attribute. This defaults to ‘0.0.0’.

compatible_version(required_application_version)

Check that the plugin is version-compatible with the application.

This checks the plugin’s required_application_version against the declared application version and returns True if they are compatible, and False if not.

disable_plugins(plugins=None)

Disable all or selected plugins.

enable_plugins(plugins=None)

Enable all or selected plugins.

find_plugin_files()

Find files that may contain plugins.

This finds all files named *_plugin.py in all locations. The returned list is sorted.

is_older(version1, version2)

Is version1 older than version2?

load_plugin_file(pathname)

Return plugin classes in a plugin file.

load_plugins()

Load plugins from all plugin files.

parse_version(version)

Parse a string represenation of a version into list of ints.

plugin_files
plugins
suffix = '_plugin.py'
class cliapp.Hook

A hook.

add_callback(callback)

Add a callback to this hook.

Return an identifier that can be used to remove this callback.

call_callbacks(*args, **kwargs)

Call all callbacks with the given arguments.

remove_callback(callback_id)

Remove a specific callback.

Indices and tables

Table Of Contents

This Page