Datakit is a pluggable command-line tool for managing the life cycle of data projects.
The Associated Press Data Team uses Datakit to auto-generate project skeletons, archive and share data on Amazon S3, and other routine tasks.
Datakit is a thin wrapper around the Cliff command-line framework and is intended for use with a growing ecosystem of plugins.
Feel free to use our plugins on Github, or fork and modify them to suit your needs.
If you're comfortable programming in Python, you can create your own plugins (see :ref:`creating-plugins`).
Datakit is distributed as a command-line tool. The recommended way to install
it is with uv, which installs the datakit command into its own isolated
environment:
$ uv tool install datakit-core
Plugins must live in the same environment as datakit so it can discover
them. Install them alongside core with --with:
$ uv tool install datakit-core --with datakit-project
To add a plugin to an existing install, re-run the command with the plugins you want (uv will update the tool in place):
$ uv tool install datakit-core --with datakit-project --with datakit-data
If you don't have uv, see its installation docs.
After installing one or more plugins, Datakit can be used to invoke the commands provided by those plugins.
To see which commands plugins provide, try the --help flag:
$ datakit --help
Install datakit-project alongside core:
$ uv tool install datakit-core --with datakit-project
The plugin provides a project create command. You need to specify a Cookiecutter template to use this command, for example the AP's R template:
That's the basic recipe for working with plugins: install, explore, and invoke! [1]
Many plugins need configuration to work — API tokens, bucket names, default
paths, and the like. Datakit ships a generic config command family that
reads each installed plugin's declared configuration schema, so you use the
same commands no matter which plugin you're setting up. Configuration is stored
per plugin under ~/.datakit/plugins/<plugin>/config.json.
List every plugin's configuration and whether each value is set:
$ datakit config list
datakit config status is an alias for list.
Interactively fill in any unset values (secrets are entered hidden). Pass a plugin slug to limit it to one plugin:
$ datakit config init $ datakit config init datakit-github
Set a single value. Omit the value to be prompted for it (hidden, if the field is a secret):
$ datakit config set datakit-github github_api_key $ datakit config set datakit-project default_template gh:associatedpress/cookiecutter-r-project
Verify that configured values actually work — plugins can attach validators that check a token authenticates or a bucket is reachable. A non-zero exit status signals a failure, so this is handy in scripts:
$ datakit config verify $ datakit config verify datakit-github
This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.
| [1] | Plugins may also provide more robust docs, so don't forget to check those out when available. |