Extract, Transform, Load Kusama/Polkadot history into a PostgreSQL DB
npm install @substrate/node-watcherThe tasks specific to Nomidot are in src/tasks
The script itself lives in nodeWatcher.ts, and anyone can write their own tasks suited to their needs as long as they conform to the Task interface. It is agnostic of what server or db one decides to use.
docker-compose up -dyarn startminikube startkubectl cluster-infokubectl create -f path/to/secrets.yamlkubectl create -f path/to/namespace.yamlkubectl config set-context --current --namespacekubectl config view --minify | grep namespace:kubectl create -f path/to/nodewatcher_deployment.yamlkubectl create -f path/to/service.yamlkubectl describe deployments nodewatcher PRISMA_ENDPOINT in nomidotwatcher-deployment.yaml to External IP from step 6.gcloud compute firewall-rules create allow-nodewatcher-nodeport --allow=tcp:31000docker tag node_watcher_main eu.gcr.io/kusama-etl/node_watcher:v1.0docker push eu.gcr.io/kusama-etl/node_watcher:v1.0kubectl describe pods ${POD_NAME}kubectl set image ${current_image} ${new_image}kubectl rollout status deployments/${new_image}##### Cloudsql Proxy
- Make sure Cloudsql Proxy is installed: curl -o cloud_sql_proxy https://dl.google.com/cloudsql/cloud_sql_proxy.darwin.amd64
- Make it executable: chmod +x cloud_sql_proxy
- Locate the credentials.json file containing credentials to the service account.
- Start the proxy: ./cloud_sql_proxy -instances=test-installations-222013:europe-west1:kusama-etl=tcp:0.0.0.0:5432 -credential_file=./key.json
- It should say something like:
```
Listening on 127.0.0.1:5432 for test-installations-222013:europe-west1:kusama-etl
2020/01/30 17:05:37 Ready for new connections
##### PgAdmin
- Get pgadmin: https://hub.docker.com/r/dpage/pgadmin4/
- Run it: docker run -p 80:80 -e PGADMIN_DEFAULT_EMAIL="name@parity.io" -e PGADMIN_DEFAULT_PASSWORD="**" -d dpage/pgadmin4localhost:80
- Navigate to in a browserSSL mode
- Login with the above credentials
- Click "New Server"
- In "Connection" tab, fill in the DB_NAME (e.g. nomidot-node-watcher), DB_USER (e.g. admin), and DB_PASS, as appropriate.
- Make sure is set to Disabled. n.b. "Even though the sslmode parameter is set to disabled, the proxy does provide an encrypted connection."
- Click Connect.
##### Port Forwarding
It is also possible to forward our localhost:4467 to our cluster port :4466 with, port forwarding. This will allow us to go to open the GraphQl interface from our local machine.
1. kubectl get pods --namespace nomidotwatcher-stagingkubectl port-forward --namespace nomidotwatcher-staging
2.
Then just go to localhost:4467 in a browser and that will forward your requests to the cluster's 4466 port (where the deployed Prisma server lives).
You can also deploy a new prisma schema by setting the PRISMA_ENDPOINT environment variable to be localhost:4467, and running prisma deploy, e.g. PRISMA_ENDPOINT="http://127.0.0.1:4467" START_FROM="66647" yarn prisma deploy
##### Updating Kubernetes Job
1. Make sure you're logged into gcloud with an account that has access to Container Registry. You can also push to Docker Hub but for the sake of having everything in one place, we'll keep it all in GCR.
!gcloud auth list
2. Run gcloud auth configure-dockergcloud container images list-tags eu.gcr.io/test-installations-222013/nomidot_watcher
3. Check the last build image version with docker build -t nomidot_watcher:v1.2.15 .
4. Making sure you're on the appropriate branch, build a new image. docker tag nomidot_watcher:v1.2.15 eu.gcr.io/test-installations-222013/nomidot_watcher:v1.2.15
!docker build...
5. Tag that image in the appropraite format for GCR e.g docker push eu.gcr.io/test-installations-222013/nomidot_watcher:v1.2.15
!docker tag
6. Push that image with kubectl create -f /path/to/config.yaml
!docker push...
7. Copy the SHA256 hash
!copy sha...
8. Paste it into the job config
!paste sha...
9. Create the job kubectl get jobs
!k8 create job
10. Check it's running , kubectl logs -l job-name={jobname} -n nodewatcher-stagingkubectl delete jobs old-job-name`
!k8 check job
11. Eventually delete the outdated job too