A tool to generate yaml file use in-block profile, to make gitops more comfortable
npm install @holytiny/yprofileA tool to generate yaml file use 'in-block' profile.
- Motivation
- Profile
- Directly
- Quick Start
- Install
- Basic Usage
- Details
- Profile Usage
- Profiles
- Path
- Selector
- Array Selector
- Add
- Replace
- Remove
- File Generate
- Logic
- Why
- Directly Modify
- Add
- Remove
- Replace
- NOTICE
Sometimes, we would find that we need to modify the yaml file slightly under different conditions or in the different stages of the project. For example, we might use a yaml file to control the ci/cd system, for develop:
``yaml`
images:
backend:
image: john/devbackend
backend-debugger:
image: john/debugger
deployments:
- name: backend
helm:
componentChart: true
values:
containers:
- image: john/devbackend
- image: john/debugger
For staging:
`yaml`
images:
backend:
image: john/stagingbackend
backend-debugger:
image: john/debugger
deployments:
- name: backend
helm:
componentChart: true
values:
containers:
- image: john/stagingbackend
- image: john/debugger
And for production:
`yaml`
images:
backend:
image: john/prodbackend
deployments:
- name: backend
helm:
componentChart: true
values:
containers:
- image: john/devbackend
- image: john/cache
Maintaining such yaml files is tendious and error prone. An elegant solution is to generate the yaml files from a template file, and save the modification actions in git repo. A devops tool devspace provides a method to support such solutions, however, saddly it can only be used to its own config file.
Motivated by devspace, yprofile provide nearly the same profile operations to any yaml files with profiles section, and use this template to generate templates as profile describing. These functions are supported by yprofile as generate command.
`sh-session`
yprofile generate
We will find that sometimes we have to modify the yaml file directly in ci script, especially in gitops processes. For example, if we use helm as the manifest, during the ci process we need to modify the helm chart, at least the appVersion property, in another git repo for cd tools to deploy our containers to dock compose or k8s.
So it may be helpful the the yaml file can be directly modified from the command line. These functions are supported by yprofile as operate command.
`sh-session`
yprofile add
yprofile remove
yprofile replace
`sh-session`
npm install @holytiny/yprofile -D
`sh-session
USAGE
$ yprofile [COMMAND]
COMMANDS
add add a property and value pair in a yaml, mainly used for gitops ci
generate generate yaml file from yaml file template with in-block profiles, for manual use
help display help for yprofile
remove remove a property and its value from a yaml, mainly used for gitops ci
replace replace a property and value pair in a yaml, mainly used for gitops ci
`
If we have a template yaml file named test.template.yaml:
`yaml`
images:
backend:
image: john/devbackend
backend-debugger:
image: john/debugger
deployments:
- name: backend
helm:
componentChart: true
values:
containers:
- image: john/devbackend
- image: john/debugger
profiles:
- name: staging
patches:
- op: replace
path: images.backend.image
value: john/stagingbackend
- op: remove
path: deployments.name=backend.helm.values.containers[1]
- name: production
patches:
- op: replace
path: images.backend.image
value: john/prodbackend
- op: remove
path: deployments.name=backend.helm.values.containers[1]
- op: add
path: deployments.name=backend.helm.values.containers
value:
image: john/cache
After we run the command:
`sh-session`
npx yprofile generate test.template.yaml staging --output=test.yaml
We would get a yaml file named test.yaml:
`yaml`
images:
backend:
image: john/stagingbackend
backend-debugger:
image: john/debugger
deployments:
- name: backend
helm:
componentChart: true
values:
containers:
- image: john/devbackend
If a yaml file named Chart.yaml:
`yaml`
apiVersion: v2
name: first-chart
description: A Helm chart for Kubernetes
type: application
version: 0.1.0
appVersion: 1.16.0
After we run the command:
`
yprofile replace Chart.yaml appVersion 1.16.1
The Chart.yaml will become:
`yaml`
apiVersion: v2
name: first-chart
description: A Helm chart for Kubernetes
type: application
version: 0.1.0
appVersion: 1.16.1
This would be very useful in gitops ci script.
Simply speaking, yprofile uses generate or abbreviation form gen command to generate the yaml file from a template which has profiles section.
`sh-session
USAGE
$ yprofile generate INPUT_FILE PROFILE
ARGUMENTS
INPUT_FILE input yaml file with profiles in block
PROFILE the profile used to generate the yaml file
OPTIONS
-f, --force generate the yaml file to output file regardless whether a file has already existed
-h, --help show CLI help
-o, --output=output the output file path. Default to the same path as input file and suffix with .out
ALIASES
$ yprofile gen
`
Simply speaking, profiles section are groups of operations. Profiles contains several profile, a profile should contains tow sections, name and patches.
The name of a profile represents the stage, env or process of the operations in it.The patches contains several patch in it. The patch is the real operation.
A patch contains op, path and an operational value. The op is the action to yaml template content, currently, remove, replace and add ops are supported.
`yaml`
profiles:
- name: staging
patches:
- op: replace
path: images.backend.image
value: john/stagingbackend
- op: remove
path: deployments.name=backend.helm.values.containers[1]
The path specifies which property or element of the template yaml to be modifed or added. The path contains selector which is seperated by the . dot sign. For example:
`yaml`
path: deployments.name=backend.helm.values.containers[1]
The path above contains 5 selector:
- deployments
- name=backend
- helm
- values
- containers[1]
The selector is used to trace down the yaml properties and specify the element to be operated.
There are tow type of selectors, map selector and array selector. The map selector is straight forward, it uses . sign to trace down the yaml element. For convenient and compatible with the devspace syntex, the array selector supports two styles: [] sign and = sign, which are equivalent in the result.
- deployments: map selector
- name=backend: array selector uses = sign
- helm: map selector
- values: map selector
- containers[1]: array selector uses [] sign
`yaml`
images:
backend:
image: john/devbackend
backend-debugger:
image: john/debugger
deployments:
- name: backend
helm:
componentChart: true
values:
containers:
- image: john/devbackend
- image: john/debugger
To select a deployment named backend in the template above, these tow selectors are the same:
- deployments[0]
- deployments.name=backend
To select a container, which image is john/debugger, the two selectors below are the same:
- deployments.name=backend.helm.values.containers.image=john/debugger
- deployments[0].helm.values.containers[1]
The add op can only add the value to the element DOES NOT EXIST, if the element has already existed, plase use replace.
If a yaml template named add-test.yaml:
`yaml`
images:
backend:
image: john/devbackend
backend-debugger:
image: john/debugger
deployments:
- name: backend
helm:
componentChart: true
values:
containers:
- image: john/devbackend
- image: john/debugger
profiles:
- name: production
patches:
- op: add
path: images.frontend
value:
image: john/frontend
- op: add
path: deployments.name=backend.helm.values.containers
value:
image: john/cache
- op: add
path: deployments[0].helm.values.containers
value:
image: john/frontend
Run yprofile:
`sh-session`
npx yprofile gen add-test.yaml production --output=add-test-res.yaml
The output file of add-test-res.yaml should be:
`yaml`
images:
backend:
image: john/devbackend
backend-debugger:
image: john/debugger
frontend:
image: john/frontend
deployments:
- name: backend
helm:
componentChart: true
values:
containers:
- image: john/devbackend
- image: john/debugger
- image: john/cache
- image: john/frontend
If a yaml template named replace-test.yaml:
`yaml`
images:
backend:
image: john/devbackend
backend-debugger:
image: john/debugger
deployments:
- name: backend
helm:
componentChart: true
values:
containers:
- image: john/devbackend
- image: john/devfrontend
- image: john/debugger
profiles:
- name: staging
patches:
- op: replace
path: images.backend.image
value: john/stagingbackend
- op: replace
path: deployments[0].helm.values.containers[0]
value:
image: john/backend
- op: replace
path: deployments.name=backend.helm.values.containers.image=john/devfrontend
value:
image: john/frontend
- op: replace
path: deployments.name=backend.helm.values.containers[2]
value:
image: john/deploy
Run yprofile:
`sh-session`
npx yprofile gen replace-test.yaml staging --output=replace-test-res.yaml
The output file of replace-test-res.yaml should be:
`yaml`
images:
backend:
image: john/stagingbackend
backend-debugger:
image: john/debugger
deployments:
- name: backend
helm:
componentChart: true
values:
containers:
- image: john/backend
- image: john/frontend
- image: john/deploy
If a yaml template named remove-test.yaml:
`yaml`
images:
backend:
image: john/devbackend
backend-debugger:
image: john/debugger
deployments:
- name: backend
helm:
componentChart: true
values:
containers:
- image: john/devbackend
- image: john/debugger
- image: john/cache
- image: john/frontend
profiles:
- name: staging
patches:
- op: remove
path: images.backend-debugger
- op: remove
path: deployments.name=backend.helm.values.containers[3]
- op: remove
path: deployments.name=backend.helm.values.containers.image=john/cache
Run yprofile:
`sh-session`
npx yprofile gen replace-test.yaml staging --output=replace-test-res.yaml
The output file of replace-test-res.yaml should be:
`yaml`
images:
backend:
image: john/stagingbackend
backend-debugger:
image: john/debugger
deployments:
- name: backend
helm:
componentChart: true
values:
containers:
- image: john/backend
- image: john/frontend
- image: john/deploy
If there is no file in the output path, the output file is always generated.
If there is already a file in the output path, yprofile will try to read the content of that file, if the content of that file is the same as the generated content, output process will be skipped.
If the content of that file is different from the content generated, if the -f or --force flag is not set, yprofile will issue an error and exit. The content of that file will not be changed.
If the -f or --force flag is set, yprofile will backup that file to a name associated with the current date time, and rewrite that file.
This feature is for file debug.
Suppose that you generate a file named test.yaml using yprofile from the yaml template file test.template.yaml, and the test.yaml doesn't work, you can modify the content of test.yaml directly to test the function. After every things being done, you can use -f flag to regenerate the test.yaml file and you would have tow file, the test.yaml file which is buggy, and the backup file may named test-2020-10-11-13:45:18.yaml which is correct, then you can compare the two file to verify that the content of the template is correct.
We can run yprofile to modify the yaml file directly rather than using profiles section.
The concepts used in such directly modify command is the same as profile methods.
`sh-session
add a property and value pair in a yaml, mainly used for gitops ci
USAGE
$ yprofile add INPUT_FILE PATH VALUE
ARGUMENTS
INPUT_FILE the file path of the yaml, such as /home/john/workspace/ci.yaml
PATH the path of the value to be added, such as ingress.enable
VALUE the value to be added, such as true or false
OPTIONS
-h, --help show CLI help
`
If we have a yaml file named add.yaml:
`yaml`
images:
backend:
image: john/devbackend
backend-debugger:
image: john/debugger
deployments:
- name: backend
helm:
componentChart: true
values:
containers:
- image: john/devbackend
- image: john/debugger
Run yprofile add:
`sh-session
npx yprofile add add.yaml version 1.0.0
npx yprofile add add.yaml images.frontend image: john/frontend
npx yprofile add add.yaml deployments.name=backend.helm.values.containers image: john/cache
npx yprofile add add.yaml deployments[0].helm.values.containers image: john/frontend
`
The add.yaml should become:
`yaml`
images:
backend:
image: john/devbackend
backend-debugger:
image: john/debugger
frontend:
image: john/frontend
deployments:
- name: backend
helm:
componentChart: true
values:
containers:
- image: john/devbackend
- image: john/debugger
- image: john/cache
- image: john/frontend
version: 1.0.0
`sh-session
remove a property and its value from a yaml, mainly used for gitops ci
USAGE
$ yprofile remove INPUT_FILE PATH
ARGUMENTS
INPUT_FILE the file path of the yaml, such as /home/john/workspace/ci.yaml
PATH the path of the value to be removed, such as ingress.enable
OPTIONS
-h, --help show CLI help
`
If we have a yaml file named remove.yaml:
`yaml`
images:
backend:
image: john/devbackend
backend-debugger:
image: john/debugger
deployments:
- name: backend
helm:
componentChart: true
values:
containers:
- image: john/devbackend
- image: john/debugger
- image: john/cache
- image: john/frontend
Run yprofile remove:
`sh-session
npx yprofile remove remove.yaml images.backend-debugger
npx yprofile remove remove.yaml deployments.name=backend.
npx yprofile remove remove.yaml deployments.name=backend.helm.values.containers.image=john/cache
`
The remove.yaml should become:
`yaml`
images:
backend:
image: john/devbackend
deployments:
- name: backend
helm:
componentChart: true
values:
containers:
- image: john/devbackend
- image: john/debugger
`sh-session
replace a property and value pair in a yaml, mainly used for gitops ci
USAGE
$ yprofile replace INPUT_FILE PATH VALUE
ARGUMENTS
INPUT_FILE the file path of the yaml, such as /home/john/workspace/ci.yaml
PATH the path of the value to be replaced, such as ingress.enable
VALUE the value to be replaced, such as true or false
OPTIONS
-h, --help show CLI help
`
If we have a file named replace.yaml:
`yaml`
images:
backend:
image: john/devbackend
backend-debugger:
image: john/debugger
deployments:
- name: backend
helm:
componentChart: true
values:
containers:
- image: john/devbackend
- image: john/devfrontend
- image: john/debugger
If we run yprofile replace:
`sh-session
npx yprofile replace replace.yaml images.backend.image john/stagingbackend
npx yprofile replace replace.yaml deployments[0].helm.values.containers[0] image: john/backend
npx yprofile replace replace.yaml deployments.name=backend.helm.values.containers.image=john/devfrontend image: john/frontend
npx yprofile replace replace.yaml deployments.name=backend.helm.values.containers[2] image: john/deploy
`
The replace.yaml should become:
`yaml`
images:
backend:
image: john/stagingbackend
backend-debugger:
image: john/debugger
deployments:
- name: backend
helm:
componentChart: true
values:
containers:
- image: john/backend
- image: john/frontend
- image: john/deploy
- Try to always use the = sign to select the arry
- Only use [] sign at the end of the path, if you want to use [] sign selector to select array element
- Try hard to avoid create patch rely on the results of other patches
- The content of output yaml file is sorted
Because the content of the yaml template would change frequently and the index of the element in arry would change...
For the sake of efficiency and speed, yprofile DO NOT reparse the output yaml after eache patch. The sequence of the yprofile is something like:
`
read the template -> read the profile -> do all patches to the template -> output
So the conent of the output DO NOT CHANGE actually after each patch before it is write to the output file.
This may conflict with your intuition，so please don't generate the file rely on the logic between each patch!!!
If you really need such functions, please consider using directly modify methods.
To sematicly compare two yaml files, the yaml file is sorted immediatly after it is read, so the output of the yaml file is also sorted.
So the sequence of the output yaml file sections may be different from the input file.