fish-pepper - Spicing¹ up the ocean

fish-pepper is a multi-dimensional docker build generator . It
allows you to create many similar Docker builds with the help of
templates and building blocks. It allows for
compositions of building blocks in addition to the usual Docker
inheritance from base images.

For example consider a Java base image: Some users
might require Java 7, some want Java 8. For running Microservices a
JRE might be sufficient. In other use cases you need a full JDK. These
four variants are all quite similar with respect to documentation,
Dockerfiles and support files like startup scripts. Copy-and-paste
might work but is not a good solution considering the image evolution
over time or when introducing even more parameters.

With fish-pepper you can use flexible templates which are filled
with variations of the base image (like 'version' :
[ 'java7', 'java8'], 'type': ['jdk', 'jre']) and which will create
multiple, similar Dockerfile builds. The example below
dives into this in more details.

The generated build files can also be used directly to create the
images with fish-pepper against a running Docker daemon or they can
be used as the content for automated Docker Hub builds when checked in
into Github.

$3

fish-pepper can be installed as any other node.js application
by calling npm:

``
npm -g install fish-pepper
`

It is recommended to install fish-pepper globally so that it is easily
accessible from your build directories.

If you want to install the most current version, call npm from the
project directory:

`
cd fish-pepper
npm -g install
`

If you are hitting Error: libcurl-gnutls.so.4: cannot open shared object file: No such file or directory, see issue #13.

$3

``
Usage: fish-pepper [OPTION]

Multidimensional Docker Build Generator

-i, --image=ARG+ Images to create (e.g. "tomcat")
-p, --param=ARG Params to use for the build. Must be a comma separated list
-a, --all Process all parameters images
-c, --connect Docker URL (default: $DOCKER_HOST)
-d, --dir=ARG Directory holding the image definitions
-n, --nocache Don't cache when building images
-e, --experimental Include images which are marked as experimental
-h, --help display this help

The argument is interpreted as the command to perform. The following commands are supported:

make -- Create Docker build files from templates (default)
build -- Build Docker images from generated build files. Implies 'make'

The configuration is taken from the file "fish-pepper.json" or "fish-pepper.yml" from the
current directory or from the directory provided with the option '-d'. Alternatively the
first parent directory containing one of the configuration files is used.

Examples:

# Find a 'fish-pepper.yml' in this or a parent directory and use
# the images found there to create multiple Docker build directories.
fish-pepper

# Create all image families found in "example" directory
fish-pepper -d example

# Create only the image family "java" in "example" and build the images, too
fish-pepper -d example -i java build
``

#### Examples

* Make all docker builds and create the images with the Docker daemon
running at $DOCKER_HOST:

fish-pepper build

* Create the image family "java" which can be found in the directory
example:

fish-pepper -d example -i java

* Create only the image java-openjdk8-jre:

fish-pepper -d example -i java -p openjdk8,jre

* Asuming your current working directory is in
examples/java/images/openjdk7/jdk and you call fish-pepper
without arguments, only the Docker build for the image "java" with
the parameter values "opendjk8" for the parameter "version", and
"jre" for the parameter "type" is used ("version" and "type" are the
parameter types as defined in fish-pepper.yml). This corresponds
to

fish-pepper -d example -i java -p openjdk7,jdk

* Assume the same directory as above, but when you call fish-pepper
-a then all images with all parameters are build nevertheless.

$3

$3

There are two kinds of configuration files:

* A global configuration fish-pepper.yml global, image independent
configuration, like a default Docker user name or the maintainer to
use within the Dockerfile.
* A per image family configuration images.yml which declares the
parameterization of the images and image specific configuration.

The configuration can be given either in YAML
syntax (with the file extensions .yml or .yaml) or in plain JSON
(with extension .json).

Configuration within a fish-pepper section section in those files
influence the behaviour of the image generation and the property names
have a special meaning. All other configurations are mostly relevant
for the templating.

#### fish-pepper.yml

The top-level configuration file is typically quite slim:

`yaml


Variable influencing the behaviour of fish-pepper are given in an extra object 'fish-pepper'

fish-pepper:
# Registry for building the name when building images with '-b'. Can be omitted
# in which case no registry is used
registry: "docker.io"
# A user which is used as default when no image stem is given
repoUser: "fabric8"

# Custom global variables useful in templates
maintainer: "rhuss@redhat.com"
`

As mentioned above, the section fish-pepper has a special
meaning. The following keys are used

* registry - A default registry to use when creating images with
fish-pepper -b
* repoUser - A default repo user to use when creating images in
build mode

These two parameters are used for calculating the base image name when
doing a Docker build in build-mode. See below for how
the name is calculated.

Any other key can be used by any image family. E.g. this is also
perfect for defining a maintainer or global
labels.

#### images.yml

For each image family found in a sub directory below the root
directory where fish-pepper.yml is stored a dedicated configuration
file images.yml is used. As the global config file, fish-pepper:
blocks defined configuration values with a special meaning.

* params defines the parameterization of the image family. See below.
* name the name stem to use when building images with -b. If not
given, the name is calculated as described in an extra
section.

Parameters are a central concept of fish-pepper. They are used to
fan-out a image family into multiple image builds. A parameter has a
type (like 'version') and one or more possible values. For each value,
a Docker build directory is created from the templates. The can be
multiple parameter types, each with a dedicated set of possible
values. The generated Docker build directories' hierarchy reflects the
parameterization space: On the first directory level sub-directories
are named like the values of the first parameter type, on the second
level the directories are named after the values of the second
parametere type and so on. See section How it works
for an example layout. Note, that using multiple parameters can easily
result in a multitude of Docker builds. So the set of possible
parameters as well as their values should be chosen carefully.

fish-pepper will iterate over all parameter values and used
dedicated, parameter value specific configurations for creating the
template's context. This configuration is stored in a special
config: object which looks like:

`yaml
fish-pepper:
params:
- type1
- type2
config:
type1:
value1.1:
.....
value1.2:
.....
type2:
value2.1:
....
value2.2:
....
value2.3:
....
`

So, for each parameter type there's a set of config values. In the
abstract example above, 6 (2 * 3) Docker build directories would be
created, one for each permutation of parameter values. The .....
represent the specific configuration for this parameter values. The
configuration of all select parameter values for a specific
combination are merged into one single configuration which is used to
fill in the template.

An example:

`yaml
config:
version:
openjdk7:
fish-pepper:
version: "1.7"
tags:
- "7u79"
java: "java:7u79"
fullVersion: "OpenJDK 1.7.0_79 (7u79-2.5.5-1~deb8u)"
openjdk8:
fish-pepper:
version: "1.8"
tags:
- "8u45"
java: "java:8u45"
fullVersion: "OpenJDK 1.8.0_45 (1.8.0_45-internal-b14)"
type:
jre:
extension: "-jre"
jdk:
extension: "-jdk"
`

Here are two types with two values each, resulting in four Docker
builds. For the build with version=openjdk7 and type=jre the
template gets a template context which holds this information:

`yaml
config:
version:
java: "java:7u79"
fullVersion: "OpenJDK 1.7.0_79 (7u79-2.5.5-1~deb8u)"
type:
extension: "-jre"
param:
version: "openjdk7"
type: "jre"
`

As you can see, the parameter values are included, too. In the example
above you can also see, that each parameter value's configuration can
also contain a fish-pepper: section. As for the top-level
fish-pepper: the properties specified here influence the behaviour
of the build files generation.

The template context is described in detail in
Template context.

* experimental if set, this parameter value is considered to be
experimental and the value will only be used when the command line
option --experimental (or -e) is given.
* version
A paramer value specific version number which is used to generate
the image name when building images with -b. See
Image naming for details.
* tags
A list of tags which will be added to the created images in build
mode.

$3

Fish pepper templates are
DoT.js templates. It is a
fast template library which allows for the full expressiveness of
JavaScript. Its a bit similar to JSP or PHP. The template syntax is
described in detail [here] (section "Usage").

The most important directives are

* {{= ... }} will evaluate the JavaScript within the parentheses and
evaluate it as string which then is inserted literally into the
text.
* {{ ... }} will add the JavaScript code (which can be partially
complete only) to the generated JavaScript rendering
function. E.g.

{{ images.forEach(function image) { }}
* {{= image.name }}
{{ } }}

will iterate over images (which needs to be initialized
beforehand) and create a bullet list of the image names.
* {{~ array :value:index}} can be used as shortcut for iteration
over arrays. So, the example above can be written more elegantly
with

{{~ images :image:index}}
* {{= image.name }}
{{~}}

* With {{? if-condition} ... {{?? else-if-conition}} ... {??} (else)
... {{?}} conditions can be build up easily:

{{? images.length > 1 }}
More than one image
{{?? images.length == 1 }}
Exactly one image
{{??}}
No image
{{?}}

#### Template context

All fish-pepper templates have access to the fish-pepper context
object. This accessible as variable fp from within the templates.

The fp context has the following properties:

* param is a map holding the current parameter values. As described
in Configuration template are evaluated for every
parameter values tuple. In each iteration the param property holds
a map with the current parameter values. For the example above e.g
when the current parameter values are version == "openjdk7" and
type == "jdk" then fp.param is

{
version: "openjdk7"
type: "jdk"
}

* config is an object which holds the configuration for the
selected parameter values for the current template
evaluation. E.g. assuming the example configuration given
above, then when the template for the parameter
values version == 'openjdk8' and type == 'jre' is used,
then fp.config.version.java evaluated to java:8u45 and
fp.config.type.extension to -jre. The general scheme is
fp.config.parameter type which references to the currently
active parameter's configuration.
* All other properties defined in fish-pepper.yml and images.yml
are directly accesible as properties from fp, so you can easily
define image global and global global properties. Properties with
the same name in images.yml take precedence over the properties in
fish-pepper.yml.
* block() is a function to use blocks

Examples of the context usage can be found in the
templates used in the Java fish-pepper demo
included in this repository.

$3

One of the major features of fish-peppers are reusable
blocks. These are reusable components which can be parameterized
like any other template. A block itself can consist of two different
kinds:

* template snippets which will be inserted as a template fragment
where referenced from within a template
* files which are copied over into the Docker build direct.

These blocks can be defined locally or referenced remotely and a
referenced by a unique ame. It is easy to share blocks across multiple
image deinitions. The following two sections explain how to use blocks
and how to create blocks.

#### Block usage

Defined Blocks can be referenced from within templates with a function
on the template context.

`javascript
{{= fp.block('version-info') }}
`

will refer to a block named "version-info". This block is processed as
a template which receives the same context as the calling
template. The processed content is the insert in place where the
method is called.

Sub-snippets can be declared with an optional second argument:

`javascript
{{= fp.block('version-info','java') }}
`

An (optional) third argument specifies additional processing
instructions and additional arguments for the blocks as an JavaScript
object:

`javascript
{{= fp.block('version-info','java',{ "no-files": true, "copy-dir" :
"/usr/local/sti" }) }}
`

Processing instructions all start with fp-. The followin
instructions are support:

* fp-no-files : Don't copy over any files into the build directory

#### Block definitions

Blocks are stored in dedicated blocks/ directories. These will be
looked up in multiple locations:

* Top-level blocks/ directory where you global fish-pepper.yml resides. The blocks
defined here are available across all defined images.
* blocks/ directory on the image level where images.yml is located.
* The location referenced in the blocks: sections in
fish-pepper.yml.

There are two kind of blocks.

##### Simple blocks

Simple blocks are files within the blocks directory. They can have an
arbitrary file extension which should match the content. The name
before the extension defines the block name. E.g. a file
version-info.md in on of the blocks/ directories or in one of the
locations referenced in the configuration will defined a block named
"version-info" (and is probably written in markdown). This block can
easily be referenced from within a template with {{=
fp.block('version-info') }}. The text itself is a template, too and
is processed before inserted.

The block itself can reference the fp context object as described in
Templates. In addition is access to extra information
which is available only for this block. This information is available
as an object via the property fp.blockContext and has the following
properties:

* name : Name of the block
* subname : Sub-snippet name (which is empty for simple blocks)
* opts : Extra option given a third argument to the block call

##### Extended blocks

Extended blocks consist of multiple files which are stored within a
directory in the blocks location. The name of the directory is also
the block name. Any file within this directory defines a
sub-snippet. The base filename of the sub snippets are the name of
the sub-snippets, the extension can be anything. This directory can
also contain a directory fp-files which holds files which should be
copied over into a Docker build directory. This directory can hold
other directories, which are deeply copied.

For example consider the following setup:

`

blocks/
|
+-- run-sh/
|
+-- run-commands.dck
+-- readme.md
+-- fp-files
|
+-- run.sh

`

This defines a block named run-sh with the template snippets
run-commands.dck and readme.md. The former holds the ADD command
to put into the Dockerfile via {{=
fp.block('run-sh','commands.dck')}}. This will also copy over all
files in fp-files directory, in this case run.sh. Alls files
copied are also processed as templates. The readme.md contains the usage
instructions which can be included in the README template with {{=
fp.block('run-sh','readme.md',{ 'fp-no-files' : true }) }}. The third
argument to this call indicates that no files should be copied in
this case.

#### Remote Block definitions

Blocks can be also defined in a Git repository which must be
accessible with https. These external references are defined in the
main fish-pepper.yml configuration file in a dedicated blocks
section.

For example

`yml
blocks:
- type: "git"
url: "https://github.com/fabric8io/run-java-sh.git"
path: "fish-pepper"
`

The blocks sections contains a list of external references. This
external reference has a type (currently only git is supported), an
access URL (https is mandatory for now). Optionally a path pointing
in this Git Repo is provided. This directory is then used as a blocks
directory as described above.

If type is omitted, the type is extracted from the url (i.e. if
it ends with .git its of type "git"). If instead of an object a string
is provided as block, this string is interpreted as URL. If no path is given,
the defaul path fish-pepper is assumed. The example above hence can
be written also as

`yml
blocks:
- "https://github.com/fabric8io/run-java-sh.git"
`

By default master is checked out, but this can be influenced either
with a tag or branch property in which case the specific
tag or branch is used.

$3

For each parameter configuration default can be configured. Assume the
following part of an images.yml:

`yaml


....

config:
version:
default:
downloadUrl: "http://download.eclipse.org/jetty/${JETTY_VERSION}/dist/jetty-distribution-${JETTY_VERSION}.tar.gz"
from:
jre8: "fabric8/java-centos-openjdk8-jre"
jdk7: "fabric8/java-centos-openjdk7-jdk"
version: "1.0.0"
9:
version: "9.3.2.v20150730"
8:
version: "8.1.17.v20150415"
7:
version: "7.6.17.v20150415"

...

`

When iterating over the versions fp.config.version will also hold
the properties downloadUrl and from which come from the default
section if not overriden by a specfic version. The advantage is the
you can avoid duplication of common parameter, the only drawback is
that you can't have a parameter value of default.

$3

For more complex variations of Dockerfile which would lead into
complicated Templates with a lof of conditionals it is possible to
provide to use alternative templates based on parameter values.

This is best explained with an example: The project
fabric8/base-images use
fish pepper to generate a collection of base images, also for Jetty
down to version 4. However the download process of the Jetty archives
changes significantly when Jetty moved from Mortbay to Eclipse. So
base images provides two different Dockerfile templates: One for
Jetty 7 to 9
and one for
Jetty 4 to 6

The relevant part in images.yml looks then like

`yaml


...


config:
version:
9:
version: "9.3.2.v20150730"
8:
version: "8.1.17.v20150415"
7:
version: "7.6.17.v20150415"
6:
version: "6.1.18"
fish-pepper:
mappings:
__Dockerfile-456: "Dockerfile"
# version 4 & 5 are similar
`

For Jetty version 6 there is a special section
fish-pepper.mappings. This section contains an object which maps
source files to its destination in the Docker build directory. In this
example the template __Dockerfile-456 is copied over Dockerfile
after it has been processed as a template. That way it is quite easy
to create alternativs for certain template files.

$3

When using the build mode with -b, image names are calculated from
various ways. The base name is taken from a fish-pepper.name when
given in images.yml. If not the base is calculated as
registry/userRepo/image, where registry and userRepo can be
globally defined in fish-pepper.yml and image is the diretory name
where images.yml is stored. registry and userRepo can be both
left out.

From this base name the full name is calculated by appending the
concrete parameter values with dashs (-). For example, when building
an image for the param values version=openjdk7 and type=jre then
image name will be java-openjdk7-jre, assuming that java is the
base name as described above.

The image's tag is also calculated: Each param parameter value can have a
version parameter in its fish-pepper section. For all parameter
values this version are concatenaed with - to form an overall
version number. When there is a top-level fish-pepper.buildVersion
in the images file, then this will be appended, too. If no version
is found at all latest is used.

In addition to this major tag more tags can be provided by the
parameter values' configuration. Each tag creates an additional tag.

Sounds complicated ? Hopefully an example sheds some light on
this. Consider the following images.yml for a java image family:

`yaml
# Two dimensional build: 1st dimension == version (7 or 8),
# 2nd dimension == type (jdk or jre)
fish-pepper:
params:
- "version"
- "type"
# Name stem to use for the images to create with -b (param value will be appended with -)
name: "jolokia/fish-pepper-java"
# Internal build number. Should be increased for each change
build: 2

# .......

# Parameter specific configuration. For each type there is entry which holds all
# possible values. This values are used to create the directory hierarchy and
# also used within the image names.
config:
version:
openjdk7:
# The version is used in the tag
fish-pepper:
version: "1.7"
# Additional tags to add
tags:
- "7u79"
# ....
openjdk8:
fish-pepper:
version: "1.8"
tags:
- "8u45"
- "latest"
# ....
type:
jre:
# ....
jdk:
# ....
`

When using fish-pepper build with this image config you will get the
following images (user and registry omitted):

* java-openjdk7-jdk:1.7-jdk-2
* java-openjdk7-jre:1.7-jre-2
* java-openjdk8-jdk:1.8-jdk-2
* java-openjdk8-jre:1.8-jre-2

and the additional tagged images:

* java-openjdk7-jdk:7u79
* java-openjdk7-jre:7u79
* java-openjdk8-jdk:8u45, java-openjdk8-jdk:latest
* java-openjdk8-jre:8u45, java-openjdk8-jre:latest

It is recommended to use and count up fish-pepper.buildVersion` for
any change in you build files.

$3

A full featured example showing most of fish-pepper's possibilities
can be found in the example directory which holds one configuration
for building a agent-bond
enabled Java image. The build it quite similar and we will build for
OpenJDK 7 and 8 with a JDK and JRE, respectively.

All the configuration files are documented, so please have a look a
them to see how the tepmplating works.

-----

1: fish pepper is an
ancient chili pepper variety
coming from Baltimore and are famous for the ornamental
qualities. They are not too hot and are ideal to spice up everything, even Docker builds :)