Hull Connector Template

> TL;DR: start with Five seconds overview.

This is a scaffolding tool for creating and maintaining Hull connectors. It comes with base directory and file structure, default configuration and documentation files templates.

The complete boilerplate template is in template directory.
During generation it's being run through lodash template function to replace <%= var %> with values from configuration and copied to a selected working path.

Installation

Install it as a global node package:

npm i -g hull-connector-template

Now you can use the hull-connector-template command to generate or update connector base code.

Creating new connector

hull-connector-template [working-path]

You can execute this command over new directory which will be created, or run it inside an empty, already existing directory.

When run script will ask you couple of questions interactively to fill in templates with data.

Updating existing connnector

When you have an existing connector already generated by hull-connector-template or not, but you want to make it compatible with the template see below two options how to peform the difference comparison:

Simple update

Running hull-connector-template over an existing connector directory will pick configuration values from package.json and manifest.json files, so you won't have to answer configuration questions again, unless you want to change any value.

It will also detect changes and conflicts over all files and will allow you to decide if overwrite file or keep old one.
This method works for files which are not modified on connector level, such as .eslintrc, .babelrc etc. since it allows to update only whole file.

Advanded update

For files which are usually updated on the connector level - such as package.json, manifest.json or JS source code files etc. - simple method of overwriting/not overwriting whole file would not work.
In this case using dedicated GUI diffing tool is the best option to decide line-by-line what to update.

To get the filesystem path to the hull-connector-template directory which should be used for this diff execute following command:

hull-connector-template --path

This will return something like this:

/usr/local/lib/node_modules/hull-connector-template/template

Then use it to run your diffing tool and compare template with actual connector code to peform more complex update:

difftool /usr/local/lib/node_modules/hull-connector-template/template ./connector-working-path

Connector development

This section describes how the further connector development process looks like in details. The boilerplate code comes with README.md file which contains high level overview of this section and links here for full information.

This document also include some information about Hull's official development process which should allow 3rd party developers to contribute to our open-source projects.

$3

Step #1: generate new connector and install deps
``bash
$ hull-connector-template new-connector


answer couple of questions from the generator

and enter the newly generated connector directory

$ cd new-connector

and install all dependencies

$ yarn
`

This will install both production and development dependencies of your base connector. The post install script will also build the front-end and back-end application with babeljs. But you don't need to worry about that. This is a pre-deployment script.

To learn more about the basic structure of the code, please go to Repository structure.

Step #2: now start the application
` bash


start the application


$ yarn start:dev
`
Your back-end server from server directory will be listening by default on 8082 port and it will be serving your front-end application from src directory. It's using babel-watch and hot reload component to not make you restart the whole thing everytime you do a change.

To learn more about the development scripts, please go to Developing.

Step #3: code and verify!
`bash


open the files in your favorite editor

and start coding

$ edit server/server.js

then perform whole test script to check

if you didn't break anything

$ yarn test

or if you want to be more specific,

run only linting

$ yarn test:lint

or unit tests only

$ yarn test:unit
`

To learn more about the testing scripts, please go to Testing/Debugging.

Steph #4: deploy or contribute

If you want to deploy your connector on your own you can run it on production like this:

`bash
yarn start
`

or if you want to contribute to one of Hull's official connectors please fork our repository and issue a PR when work is done.

Now you are good to jump into details below. Good luck!

$3

This is a reference of how the common boilerplate code structure looks like:

`text
root/

# development tools configuration
.circleci/ - Configuration for CircleCI, it runs all tests on each build and enforce test coverage
.babelrc - the server side only babeljs configuration
src/.babelrc - the client side only babeljs configuration
.eslintrc - the server side eslintrc configuration
src/.eslintrc. - the client side eslint configuration which extends the one at root
.prettierrc - global prettier configuration, applied for the whole code base
.jest.config.js - test runnner configuration file
webpack.config.js -


# Project definitions and docs
manifest.json - The main file telling Hull how this connector works
package.json - npm project defition
CHANGELOG.md - each release changes are descibed here
README.md - technical overview of the connector

# Code base
assets/ - Images, Logos and User Guide (readme.md)
readme.md - The User Guide which is served in the hull dashboard and linked from technical README.md
docs/ - Images and other assets for User Guide

flow-typed/ - Flow type definitions

server/ - Server-side code for the connector
actions/ - Route handlers for express application
lib/ - Business logic of the connector
sync-agent.js

src/ - Front-end application which will be served by the backend application

test/
integration/ - Integration tests
fixtures/ - Fixtures for notifications, payloads, etc.
helper/ - Mocking helpers for tests
scenarios/ - Expectations and inputs for various test scenarios
unit/ - Unit tests
`

$3

To successfully build the sources on your machine, make sure that you have the correct version of node along with one package manager installed. See engines in package.json for details.

$3

Execute yarn run test or npm run test in the repository root to run all tests.

If you want to run the connector on your local machine, execute yarn run start:dev or npm run start:dev which will start a new node server.
Make sure to set the proper environment variables when running the code locally.

#### Running and writing tests

There are two sets of tests, unit tests and integration tests. Please use unit tests for all features testing. The purpose of integration tests is just end-to-end validation of functionality on sample applications.

Integration tests for the SyncAgent are organized in scenarios. Please see the Test Scenarios Guide for a detailed description of the scenarios.

#### Integration-testing for logs and connector responses
Mockr is a testing addition to make it easy to simulate calls and settings and write assertions on the connector's responses to Hull

This boilerplate comes with mocha/chai/sinon/nock already setup for server tests. It also includes test/integration/support/mockr package which sets up some mocks and minihull, which is a stripped down version of hull that's able to send messages to connectors and offer expectations on what the connector should send to the Firehose.

##### Here's how:

`js
const { expect } = require("chai");
const mockr = require("./support/mockr");

// Your server's entry point, with the same format as the one hull-connector bundles.
// Options will be passed to it.
const server = require("../../server/server");

describe("Test Group", () => {
// Start the mocks. they will run beforeEach and afterEach cleanups for you,
// Start a development server
const mocks = mockr({
server
beforeEach,
afterEach,
port: 8000,
segments: [{ id: "1", name: "A" }], // Segments that should exist on the server
});

it("should behave properly", done => {
const myNock = mocks
.nock("https://api.myremote.test.com")
.get("/test")
.query({ foo: "bar" })
.reply(200, [{ email: "foo@foo.bar", id: "foobar" }]);

// Optional, if you want to stub more things that your connector
// will access during it's flow.
mocks.minihull.stubApp("/api/v1/search/user_reports").respond({
pagination: { total: 0 },
aggregations: {
without_email: { doc_count: 0 },
by_source: { buckets: [] },
},
});

// Send a user:update call to the connector.
mocks.minihull.userUpdate(
{
// Connector Settings
connector: {
id: "123456789012345678901234",
private_settings: {
api_key: "123",
handle_accounts: true,
prospect_enabled: true,
prospect_segments: ["1"],
prospect_filter_titles: ["foo"],
prospect_limit_count: 2,
},
},
// Message payload
messages: [
{
user: { id: "abc", "traits/clearbit/source": "reveal" },
account: { id: "ACCOUNTID", domain: "domain.com" },
segments: [{ id: "1" }],
},
],
},
// This is what the Firehose receives.
({ batch, logs }) => {
const [first, second, third, fourth] = batch;
expect(batch.length).to.equal(4);
expect(logs[1].message).to.equal("outgoing.user.start");
myNock.done();
done();
}
);
});
});
`

$3

We follow the Git Flow model, semver for versioning and we maintain CHANGELOG.md for each release of the production build.

Once you have the prerequisites installed, execute yarn run build or npm run build` in the repository root to build the project locally.