Simple data pipeline framework for GCP's BigQuery
npm install mashr[TODO: Diagram TBD]
Jacob Coker-Dukowitz Software Engineer San
Francisco, CA
Linus Phan Software Engineer Los Angeles, CA
Mat Sachs Software Engineer Portland, OR
* GCP (Google Cloud Platform) account
* GCP project, service account email, and json keyfile
* GCP Cloud SDK
* Node.js >= 8
* NPM
Mashr requires that users have a project with a service account on GCP and
set up the Cloud SDK CLI on their local machine. If you have not already done
so, please Download the Cloud
SDK and use the
"console" to create a
project
and a service
account
with a role of "owner". Mashr will use the project id, service account email,
and service account json keyfile to to interact with GCP services.
```
npm install -g mashr
-------------------------------------------------------------------------------
[TODO: paragraph for the init process]
Starting with a mashr_config.yml file and the terminal command mashr deploy,
Mashr creates a Google Compute Engine (GCE) instance, Google Cloud Storage
(GCS) staging and archive buckets, BigQuery dataset and table, and a Google
Cloud Function (GCF) to automate the porting of data between the GCS buckets
and BigQuery table.
The GCE instance hosts a docker container with the
Embulk data loader running on it. A cron job
runs an embulk job every 10 minutes. The embulk job pulls data from an external
source, like Salesforce or a postgres database, and puts it into a GCS staging
bucket. When data is loaded to the GCS staging bucket, the GCF is triggered.
The GCF moves the data to a coldline storage bucket for archiving and failover,
and then loads the data into the appropriate BigQuery table.
* Make sure you have a Google Cloud Platform (GCP) account
* Create a new project in your Google Cloud account
* After creating a new project, you will need to enable the Cloud Functions
API from the web console.
- Go to the main menu and choose "APIs & Services"
- Click the "+ Enable APIs and Services" button at the top of the page
- Search for "Cloud Function" (no 's')
- Click "Cloud Functions API"
- Click "enable" to enable the API
* Download and install the gcloud CLI
SDK from Google.
``
mashr init [--template
Initializes your current working directory with a template mashr_config.yml
file necessary for running mashr deploy. Optionally include the --templatehttp
flag and name of the template. Template names include , psql,random.
-------------------------------------------------------------------------------
``
mashr deploy
Deploys the integration: adds it to the list of mashr integrations and creates
related GCP resources including staging and archive GCS buckets, a cloud
function and a GCE instance.
Creates a 'function' folder that stores the code the cloud function uses in
this integration. You can edit and redeploy the cloud function with gcloud.
A mashr_config.yml file in the user's working directory is required. Runmashr init to see a template file you can use.
-------------------------------------------------------------------------------
``
mashr list
Lists all integrations that the user has deployed.
-------------------------------------------------------------------------------
``
mashr destroy
Destroys the integration: removes it from the list of mashr integrations and
destroys related GCP resources including the staging and archive GCS buckets,
the cloud function and the GCE instance.
-------------------------------------------------------------------------------
```
mashr help
Documentation of commands.
-------------------------------------------------------------------------------
[ TODO ]
[ TODO ]
-------------------------------------------------------------------------------
Consider colocating as many as your services as possible. For example, it's
required that your GCS (Google Cloud Storage) and GBQ (Google Big Query) be
located in the same regions. See the Locations Considerations
document for more information
-------------------------------------------------------------------------------
Dockerfile > build > image > start a container from the image