Hubot Luigi Script

Interact with luigi central scheduler, to get job status, worker status, etc.

Installation

Add the package hubot-luigi entry to the packages.json file
(you may need to create this file).

"dependencies": {
"hubot-luigi": "1.0.x"
}

Run the following command to make sure the module is installed.

#npm install hubot-luigi

To enable the script, add the hubot-luigi entry to the external-scripts.json
file (you may need to create this file).

["hubots-luigi"]

Configuration

* HUBOT_LUIGI_ENDPOINT - specify luigi scheduler api endpoint, like 'http://localhost:8082/api/'

Usage Examples

$3

user1> hubot luigi stats
hubot> 8 jobs running, 1315 jobs pending, 76 jobs failed, 5737 jobs disabled

$3

user1> hubot luigi show pending
hubot> Wed Sep 02 2015 13:27:36 GMT-0700 (PDT) Task(param1=value) p=50
Wed Sep 02 2015 13:27:36 GMT-0700 (PDT) WrapperTask(run=all_tasks) p=50
Wed Sep 02 2015 13:27:36 GMT-0700 (PDT) HadoopHourlyJob(jar=process_logs.jar, time=2015-08-30T00) p=100
Wed Sep 02 2015 13:27:36 GMT-0700 (PDT) ScpTask(file=filename.csv, target=server) p=10

Show can be run with any task status, including pending, running, disabled, failed. If more than 20
tasks are to be shown, it will truncate the list and state how many more there are.

$3

user1> hubot luigi search Hadoop
hubot> DONE Sun Aug 30 2015 00:12:21 GMT-0700 (PDT) HadoopHourlyJob(jar=process_logs.jar, time=2015-08-30T01) p=100
DONE Sun Aug 30 2015 00:12:21 GMT-0700 (PDT) HadoopHourlyJob(jar=process_logs.jar, time=2015-08-30T00) p=100
DONE Sun Aug 30 2015 00:12:21 GMT-0700 (PDT) HadoopHourlyJob(jar=process_logs.jar, time=2015-08-30T02) p=100
DONE Sun Aug 30 2015 01:12:18 GMT-0700 (PDT) HadoopHourlyJob(jar=analytics_reports.jar, time=2015-08-30T00) p=15

Search matches the query against task ids and is case sensitive. Results are truncated as in show.

$3

user1> hubot luigi resources
hubot> impala: 3/4
hadoop_xl: 2/2
mysql_access: 0/5

This will only show resources that are defined in luigi.cfg and not ones that are created in response
to task scheduling.

$3

user1> hubot luigi refreshresources
hubot> impala: 3/2
hadoop_xl: 2/3
mysql_access: 0/4

This causes the scheduler to dynamically reload the resource constraints from luigi.cfg. The current
resource usage is then displayed, which can violate the new constraints until existing jobs have had
a chance to finish.

$3

user1> hubot luigi workers
hubot> 781608491 AllTasksFrontFill(days=1, end_time=2015-09-02T13:30) [10] 1 running, 637 pending
377433553 AllTasksFrontFill(days=1, end_time=2015-09-02T13:00) [10] 6 running, 486 pending
042179238 AllTasksFrontFill(days=1, end_time=2015-09-01T16:00) [10] 0 running, 46 pending
208362896 AllTasksFrontFill(days=5, end_time=2015-09-01T03:00) [10] 2 running, 496 pending

$3

user1> hubot luigi worker 208362896
hubot> 208362896 AllTasksFrontFill(days=5, end_time=2015-09-01T03:00) [10] 3 running, 496 pending, 478 unique pending

running tasks: NumberedTask(n=7), NumberedTask(n=18), NumberedTask(n=3)