Juttle


Juttle is an analytics system for developers that simplifies and empowers data
driven application development. At the core of Juttle is a dataflow
language that lets you query, transform, enrich,
and analyze live and historical data from many different backends, and then send
output to files, data stores, alerting systems, or streaming
visualizations.

This repository contains the core Juttle compiler, the JavaScript runtime, a set
of basic adapters to connect to files or http sources, and a command line
interface with text-based and tabular views. As such it is most useful for
learning the language, doing simple exploration of data, or powering periodic
transformations or running periodic or continuous programs that generate alerts to an external system.

For a more complete package demonstrating the full spectrum of Juttle's
capabilities, the Juttle Engine
project embeds the juttle core in a REST API based execution
service along with a viewer
application and a full set of
supported adapters. This assembly can be used to run Juttle
programs with rich charts and dynamic input controls in development or
production.

Visit the project website to get an overview of the project and see how it can fit into your stack. To learn more about the language, see the documentation site to read about why juttle exists,
get an overview of the language, learn about the
dataflow features and how to program in juttle, see the list of supported visualizations, step through the tutorial, and more. For information about the Juttle development project, see the wiki.

Installation

Juttle requires node.js version 4.2 or later.

To use Juttle as a command-line tool, the simplest approach is to install the juttle package globally:

``
$ npm install -g juttle
`

To use the full Juttle Engine assembly, run:

`
$ npm install -g juttle-engine
`

In both cases you should now have a juttle executable in your path which you can use as follows:

`
$ juttle -e "emit -limit 2 | put message='hello'"
`

This produces:

`
┌────────────────────────────────────┬───────────┐
│ time │ message │
├────────────────────────────────────┼───────────┤
│ 2015-12-18T21:04:52.322Z │ hello │
├────────────────────────────────────┼───────────┤
│ 2015-12-18T21:04:53.322Z │ hello │
└────────────────────────────────────┴───────────┘
`

For detailed usage, see the command line reference for more information about how to configure and use the Juttle CLI.
See the Juttle Engine README for command line options and configuration instructions.

Examples

Here are some more examples of what you can do with Juttle.

Note that most of these examples require the use of external systems using adapters and refer to the visualizations embedded in Juttle Engine, so they are meant to be illustrative and not necessarily functional out of the box.

For runnable end-to-end examples of juttle usage, see the Juttle Engine examples, or step through the tutorial.

$3

Hello world in Juttle:

`juttle
emit -every :1 second: -limit 10 | put message='hello world' | view table
`

$3

This example prompts a user to input a time range to query, pulls a timeseries
metric of counts of user signups from graphite, searches for 100 logs from
Elasticsearch in which the app field is 'login' and the string 'error' occurs,
and then plots the metric along with overlaid events on the same timechart along
with a table showing the errors themselves.

`javascript
input time_period: duration -label 'Time period to query' -default :5 minutes:;

read graphite -last time_period name~'app.login.*.signup.count'
| view timechart -title 'User Signups' -id 'signup_chart';

read elastic -last time_period app='login' 'errors'
| head 100
| (
view table -title 'Errors';
view events -on 'signup_chart'
)
`

$3

This example taps into the stream of real-time twitter events searching for 'apple' and printing them to a table. If more than 10 posts occur in a five second window, it posts a message to a slack webhook.

`juttle
read twitter -from :now: -to :end: 'apple'
| (
view table -title 'Tweets about apple';

reduce -every :5 seconds: value=count()
| filter value > 10
| put message='apple is trending'
| write http -maxLength 1 -url 'https://hooks.slack.com/services/ABCDEF12345/BB8739872984/BADF00DFEEDDAB'
)
`

Ecosystem

Here's how the juttle module fits into the overall Juttle Ecosystem:

![Juttle Ecosystem - Juttle](./docs/juttle_ecosystem.md)

Adapters

Juttle includes support for a few basic adapters
out of the box to interact with files and some external systems. In addition,
through the external adapter API, Juttle can be easily extended to interact with
other storage systems or services.

#### Builtin

These adapters can be used in Juttle programs directly without special
configuration.

* file
* http
* http_server
* stdio
* stochastic

#### External

This is a list of the currently supported external adapters.

All are included as part of a Juttle Engine installation. If you've installed
the standalone juttle CLI, you will need to separately install them using npm
and make sure to install them in the same location as juttle itself.

* Elasticsearch
* Graphite
* InfluxDB
* SQLite
* PostgreSQL
* MySQL
* Twitter
* Gmail
* OpenTSDB
* AWS (Amazon Web Services)
* Amazon CloudWatch

Connections to external adapters are configured in the "adapters" section of the runtime configuration. See the CLI reference for specific instructions.

Contributing

Contributions are welcome! Please file an issue or open a pull request.

To check code style and run unit tests:
`
npm test
`

Both are run automatically by Travis.

When developing you may run into failures during linting where eslint complains
about your coding style and an easy way to fix those files is to simply run
eslint --fix test or eslint --fix lib` from the root directory of the project.
After eslint fixes things you should proceed to check that those changes are
reasonable as auto-fixing may not produce the nicest of looking code.