npm install idl[![build status][build-png]][build]
[![Coverage Status][cover-png]][cover]
[![Davis Dependency status][dep-png]][dep]
[![NPM][npm-png]][npm]
A CLI for managing thrift IDL files.
idl provides a "package manager" for thrift interfaces.
It comes with two CLI commands, idl-daemon and idl.
- The source of truth for a service (the Thrift IDL file) should live
with the service (i.e. the code repository).
You want the thrift definition to be checked into the repository
- Every service should keep a copy of the thrift definition for any
service that it wants to talk too. This definition should be static.
You do not want it to change at run-time, since that can result in
mismatched interfaces that cause runtime exceptions and kill a
process.
- There should only be one version of the world. Your company runs at
a single version of each service in production; all the Thrift IDL
files representing the current version of every service in production
should ultimately be treated as a single versioned collection. This
versioned collection is the registry of all service definitions.
- The one versioned collection should be live. Developers should not
manually publish new versions; instead we should just take master
containing the most recent Thrift IDL file for every service as the
source of truth.
- When service interface definitions are fetched and copied to a
a project that consumes those services, they should be organized
such that cross-service include/import statements are possible (i.e.
the directory structure should support relative filepath import/include
statements).
The CLI is currently broken down into two commands:
- idl - CLI tool meant for end-users, but the publish command
can also be run a continuous integration job when interface changes
land in production (push-based publishing)
- idl-daemon - daemonized process that can be configured to
poll all service repositories for interface changes (pull-based
publishing).
As a developer I want to be able to talk to other services;
To do this I need to find their Thrift interface definitions.
The idl CLI solves this need.
The idl CLI tool has the following sub-commands:
- idl list - List all available services published to the
registry.
- idl fetch - Fetch a particular service's idl
and copy it to your project in a standard location.
- idl update - Update all previously fetched services to the most
recent versions. Note: You cannot pick and choose which services to
update. This is intentional.
- idl publish - Publish the Thrift IDL file for your service
to the idl registry repository. This command should set up to be
automatically executed when a change to the service IDL lands on
master or when that change on master is deployed to production.
All commands follow the unix standard of being silent if successful. If
you would like more information about what is happening, run the CLI
with the --verbose flag.
#### idl init
This command will create a starter IDL file at the correct filepath in
a new service. You should use it if you've got a new service and want
to quickly scaffold a simple thrift IDL file at the path expected
by the idl publish command.
Example:
$ idl init
$
#### idl list
This command will list all available services published to the registry.
Example:
$ idl list
- github.com:/foo/bar 2015-09-11T23:07:57.610Z
- github.com:/foo/baz 2015-09-11T23:07:58.159Z
- github.com:/qux/quux 2015-09-11T23:07:58.716Z
$
#### idl fetch
This command will fetch a particular service's idl file and copy it
to your project in a standard location.
Once you fetch your first service we also write the ./idl/meta.json
meta file that contains the version of the registry as well as the time
it was last changed.
Note: Fetching a new service will result in an implicit update of any
services that have been fetched. For example, if you fetched servicefoo a month ago and then fetch service bar, idl will
first update service foo to the most current version before fetchingbar.
This command will also update the ./idl/meta.json file in your
project.
Example:
$ idl fetch github.com/foo/bar
$
#### idl update
This command will update all previously fetched services to the most
recent versions. Note: You cannot pick and choose which services to update. This is intentional.
Since the thrift definitions define the interfaces of the services
in production, there is only one version for all files. When you
update anything, you update everything to the current version.
This command will also update the ./idl/meta.json file in your
project.
Example.
$ idl update
$
#### idl publish
This command will publish the Thrift IDL file for your service to the
idl registry repository. This command should set up to be
automatically executed when a change to the service IDL lands onmaster or when that change on master is deployed to production.
Example:
$ idl publish
$
#### Command line flags
The idl CLI takes the following command line flags. The
first flag, --repository, is mandatory until this tool has .rc file
support.
- --repository= - The idl command needs to
know the git URL of the registry to be able to run any of the
commands above. e.g. --repository=git@github.com:foo/registry
- --cacheDir= - This is the path to the cache
directory that idl should use. The default value is
~/.idl/
- --cwd= - The path to the current
working directory in which to execute idl
- --verbose - This tool follows the unix philosophy of being
silent on success. Use this flag if you want to see output of
what it is doing.
- --config= - The path to a JSON config
file that specifies any of the above properties.
#### Configuration
Internally idl uses [dominictarr/rc][dominictarr/rc] for
options configuration. This module helps with parsing configuration
from arguments specified at the command line, from environment
variables and from .rc files. It will probe the following locations:
Given your application name (appname), rc will look in all the obvious places for configuration.
* command line arguments (parsed by [substack/minimist][substack/minimist])
* environment variables prefixed with IDL_
* or use "\_\_" to indicate nested properties
_(e.g. IDL_REGISTRY => github.com/some-org/some repo)_
* keys will automatically be camelCased to match the possible options
above.
* if you passed an option --config file then from that file
* a local .idlrc or the first found looking in ./ ../ ../../ ../../../ etc.
* $HOME/.idlrc
* $HOME/.idl/config
* $HOME/.config/idl
* $HOME/.config/idl/config
* /etc/idlrc
* /etc/idl/config
All services and clients will have a ./idl/ directory at the root of
the repo. All thrift IDL files are contained in this directory.
Service authors need to understand how this directory is organized and
should only every edit/modify the thrift IDL files for the service in
question. Client authors should never have to edit/modify files in this
directory and should only use the files contained therein as references
for the interfaces they are consuming.
The idl directory is organized so that every thrift IDL file (for the
service being authored or the services being consumed) is located at a
sub-path that mirrors the git remote origin URL of a service.
When you execute git remote -v in your service's repository, you will
see output similar to one of the following:
$ git remote -v
origin git@github.com:uber/foo-service.git (fetch)
origin git@github.com:uber/foo-service.git (push)
or
$ git remote -v
origin ssh://git@github.com/uber/foo-service.git (fetch)
origin ssh://git@github.com/uber/foo-service.git (push)
The idl directory for your service mirrors these two addresses.
Assuming the output above, the idl path for the service
being authoring will be ./idl/github.com/uber/foo-service/.
This directory will contain the thrift IDL files that will be
published to your idl registry repo when idl publish
is executed. The IDL files in this particular sub-directory are to be
manually managed by service authors.
All other directories are contain service defitions for services being
consumed and should not be edited/modified and should only be
consulted as a reference when looking up a type definition for a
service or a function definition for a service being consumed.
For service repositories, where the service is also a client of
other services, the unmanaged definitions for that service and
managed definitions for the services being consumed will live
side by side in sibling directories.
The reason for mixing both managed and unmanaged directories together
is to support relative filepath includes in thrift files.
For example, if the service git@github.com:foo/bar.git references
type definitions from the services git@github.com:foo/baz.git and
git@github.com:qux/quux.git, then you might see the following
directory and file structure:
$ tree
.
└── idl
├── github.com
│ ├── foo
│ │ ├── bar
│ │ │ └── bar.thrift
│ │ └── baz
│ │ └── baz.thrift
│ └── qux
│ └── quux
│ └── quux.thrift
└── meta.json
7 directories, 4 files
$
... and the ./idl/github.com/foo/bar/bar.thrift would
contain the following includes in its header section:
include "../baz/baz.thrift"
include "../../qux/quux/quux.thrift"
The complexity of how this directory is organized is a necessary
evil to support relative file includes. If, in the future, theinclude directive supports richer semantics, it may be possible
to simplify this directory, but for now it is what is is.
The idl-daemon will fetch all the remotes and place
their thrift files in the upstream repository. You can useidl fetch to fetch from the upstream repository.
The idl-daemon is a command that should be run with
cron.
To set up the idl repository you can run theidl-daemon. Run idl-daemon --config-file={path}
and it will populate the idl remote repository.
The config file contains the following fields
``json`
{
"upstream": "git+ssh://git@github.com/my-company/idl-registry",
"repositoryDirectory": "/var/lib/my-company/idl/repo",
"cacheLocation": "/var/lib/my-company/idl/cache",
"remotes": [{
"repository": "git+ssh://git@github.com/my-company/user-service",
"branch": "master"
}, {
"repository": "git+ssh://git@github.com/my-company/product-service",
"branch": "master"
}]
}
This project is not done yet:
- [ ] Implicit update whenever idl fetch is run.idl validate
- [ ] Implement so that service authors can locallyidl init
validate the thrift IDL files for their service before publishing.
- [ ] Implement to automatically create a boilerplateidl config get
thrift IDL file using the git URL of the remote origin.
- [ ] Implement to get a idlidl config set
configuration property.
- [ ] Implement to set aremotes
idl configuration property.
- [ ] Implement fetching from into upstream.main
- [ ] Support file in config to indicate service entry point.branch
- [ ] Support in config.publish
- [ ] Disable command using a regex saved in the .rc file.
npm install idl --global
npm test`
- andrewdeandrade (Andrew de Andrade)
- Raynos (Jake Verbaten)
- kriskowal (Kris Kowal)
- prashantv (Prashant Varanasi)
[build-png]: https://secure.travis-ci.org/uber-node/idl.png
[build]: https://travis-ci.org/uber-node/idl
[cover-png]: https://coveralls.io/repos/uber-node/idl/badge.png
[cover]: https://coveralls.io/r/uber-node/idl
[dep-png]: https://david-dm.org/uber-node/idl.png
[dep]: https://david-dm.org/uber-node/idl
[npm-png]: https://nodei.co/npm/idl.png?stars&downloads
[npm]: https://nodei.co/npm/idl
[dominictarr/rc]: https://github.com/dominictarr/rc
[substack/minimist]: https://github.com/substack/minimist