> Practical notes, debugging and fixes for a modular/dynamic CircleCI pipeline > It’s focused on my CircleCI implementation, the real problems i found, the > exact fixes i applied, and the impact those fixes had
npm install parallaxturtlelibcircle> Practical notes, debugging and fixes for a modular/dynamic CircleCI pipeline
> It’s focused on my CircleCI implementation, the real problems i found, the
> exact fixes i applied, and the impact those fixes had
---
- Overview
- Goals
- What I implemented
- Impact of the fixes
- Problems encountered (summary)
- Deep root-cause analysis
- Fixes (what I changed, why)
- Example config snippets
- Debug checklist / commands
- Best practices & recommendations
- Result of the fixes
---
This project demonstrates a robust and scalable CI/CD pipeline built with
CircleCI. It utilizes a modular approach to configuration, dynamic pipeline
generation, and intelligent caching to optimize build times and enhance security
the core of this setup is circleci/path-filtering orb to detect file changes,
map and pass parameters and generate pipeline and then run only the relevant
jobs . During implementation I hit several practical problems (workspace
ordering, missing commands in BusyBox, tag-trigger filtering and requires
logic, and parameter propagation). This README explains what went wrong, how I
fixed it, and why those fixes matter.
---
- Modularize Configuration: Break down the monolithic config.yml into
smaller, manageable files and folders for different jobs and workflows.
- Dynamically packing modular files: Use a preprocessor script
(preprocessor.sh) to dynamically pack modular code into a single
configuration file in the correct order.
- Conditional Execution: Control pipeline and workflow execution based on
specific conditions like pipeline.parameters with when rules, branch pushes,
Git tags, and API triggers from GitHub Actions.
- Optimize Performance: Implement effective caching strategies for
dependencies (e.g., node_modules) and utilize lightweight Docker images to
speed up pipeline execution.
- Efficient Artifact Management: Use persist_to_workspace and
attach_workspace to share artifacts and dependencies between jobs, avoiding
redundant work.
- Secure Credential Management: Store and access sensitive information and
tokens via CircleCI's environment variables and contexts.
- Inter-Pipeline Communication: Pass parameters between parent and child
pipelines to enable complex, multi-stage workflows.
---
- Preprocessor Script (preprocessor.sh): A custom shell script that reads
a list of modular YAML files, concatenates them in a specified order, and
outputs a final config_continue.yml. This bypasses CircleCI's alphabetical
loading order and ensures dependencies are correctly defined. this script has
all the workflows and parameters in it
- Dynamic Config Generation: A config.yml (setup: true) that runs a small
generation job and then uses the path-filtering orb to decide whether to
continue with .circleci/config_continue.yml .
- Parameter Passing: The mapped parameters on relevant file change detected
by matching the regex of related file, is used to trigger a child pipeline
from a parent pipeline, and to control jobs in the child pipeline.
- Conditional Logic:
- Pipeline Generation: The path-filtering/filter orb is used to
conditionally generate a child pipeline only when specific files are
changed.
- Workflow Execution: Workflows are conditionally executed using filters
for branch names, tag patterns (only: /^v\d+\.\d+\.\d+/), and
when: << pipeline.parameters.* >> rules
- Workspace Management:
- After the preprocessor, the generated config.yml and any other required
files are persisted to the workspace using persist_to_workspace.
- Dependent jobs in case of path-filtering/filter workflow, workspace_path
to access these files, ensuring they have the correct configuration and
artifacts.
- Caching Strategy:
- Cache: save_cache and restore_cache are used to store and retrieve
dependencies like node_modules. A cache key based on the yarn.lock file
ensures the cache is only updated when dependencies change. This is ideal
for independent jobs.
- Workspaces: persist_to_workspace is used to share artifacts and
dependencies between jobs within the same workflow, especially when a job
depends on the output of a previous job (e.g., yarn install).
---
- The implemented changes result in a highly flexible, performant, and secure
CI/CD pipeline.
- Build times are significantly reduced due to effective caching and dependency
management.
- The modular configuration is easy to maintain and scale.
- By controlling pipeline execution, we avoid unnecessary builds, saving credits
and providing a clear, auditable workflow.
- Net effect: fewer wasted runs, clearer modular structure, and reliable tag
& branch behavior , clearer separation of concerns, faster pipelines (smaller
images + caching), and true dynamic configuration flexibility for real-world
projects.
---
- CircleCI's default alphabetical config file loading prevents modular files
from being loaded in the correct dependency order.
- generate-config workflow requires a tag filter on the job if the dependent
workflow path-filtering/filter has one, otherwise the workflow won't be
generated.
- Getting caching strategies right between save_cache/restore_cache and
persist_to_workspace to optimize for both independent and dependent jobs.
- preprocessor.sh generated config not visible to path-filtering/filter —
file missing or empty.
- attach_workspace / checkout ordering caused
Directory not empty and not a git repository errors.
- Conditionally running workflows on multiple conditions (e.g., tags and API
triggers) was complex and required careful use of regex
---
- Workspace & Ordering
- Each workflow job runs in its own isolated container.
- Files must be explicitly shared via workspaces.
- If the generated continuation config (config_continued.yml) is not
persisted in the generator job and attached in the consumer job, the
path-filtering orb cannot find it.
- Fix → Always persist_to_workspace in the generator and attach_workspace
in the next job (or run the generator as a pre-step).
- Checkout vs Attach Workspace
- checkout must come before attach_workspace in jobs that need both
repo code + workspace.
- Otherwise, Git throws errors (directory not empty) or double-checkouts
occur.
- Note: Some orb jobs automatically run checkout, so ordering in pre-steps
is critical.
- Filters Evaluation
- Filters (branch, tags) are resolved at compile-time, not at runtime.
- If the generate-config job does not include the same tag filters as its
dependent jobs, those dependent jobs are silently excluded on tag runs.
- Fix → Ensure generate-config has matching filters for branches/tags as
its dependent path-filtering/filter job.
- File Concatenation Order
- CircleCI’s config pack logic doesn’t infer semantic ordering.
- By default, it merges files in alphabetical order.
- Fix → Explicitly control file concatenation order (e.g., via the
preprocessor script).
---
- Job: generate-config:
- checkout
- sh .circleci/preprocessor.sh (creates .circleci/config_continue.yml)
- persist_to_workspace: root: . paths: - .circleci/config_continue.yml
- Consumer job (path-filtering/filter or pre-steps) must attach to workspace
using workspace_path before using the file.
Why: persist + attach guarantees the dynamically generated file exists when the
filter runs.
- Add input to path-filtering/filter:
``yaml`
path-filtering/filter:
checkout: true
workspace_path: .
Why: avoids Directory not empty and not a git repository and prevents failure
when attaching workspace overlays files onto working dir.
Note: if checktout and attach_workspace is used as a prestep will cause a double
checkout in logs because some orb jobs internally checkout — harmless but
expected.
- Replace cimg/base:stable with one of:busybox:latest
- for parent pipeline as it has no complex code in itnode-18:alpine
- for child pipeline to run node scripts
Why: base images are bloated and takes high time to load, lightweight images
load faster
- Add tag filters to the generate-config job so it runs for release tags:
``yaml`
jobs:
generate-config:
filters:
tags:
only: /^v\d+\.\d+\.\d+-circleci\.\d+$/
Why: If generate-config is excluded on tags, jobs requiring it will be
excluded too.
- Created a clear distinction: \_ persist_to_workspace: For jobs that have
a direct dependency on a preceding job within the same workflow (e.g.,
install-dependencies -\> test). save_cache: For jobs that arenode_modules
largely independent but need to restore a common set of dependencies, like
for a build and a test job running in separate, parallel
workflows.
- Used a combination of CircleCI's built-in branch/tag filters and
when: < create precise conditional logic.
---
`yaml`
jobs:
generate-config:
docker:
- image: cimg/base:stable
steps:
- checkout
- run: chmod +x .circleci/preprocessor.sh
- run: .circleci/preprocessor.sh # writes .circleci/config_continue.yml
- run: cat .circleci/config_continue.yml
- persist_to_workspace:
root: .
paths:
- .circleci/config_continue.yml
`yaml`
workflows:
path-filtering-setup:
jobs:
- generate-config:
filters:
tags:
only: /^v\d+\.\d+\.\d+-circleci\.\d+$/ # ensure tag behavior
- path-filtering/filter:
requires: [generate-config] # or omit requires
checkout: true
workspace_path: .
base-revision: circleci
config-path: .circleci/config_continue.yml
mapping:
(.\.(js|json|yml|lock|sh)$)|(\..rc$) run-build-and-release true
filters:
tags:
only: /^v\d+\.\d+\.\d+-circleci\.\d+$/ # ensure tag behavior
---
`bash
#!/usr/bin/env bash
set -eo pipefail
---
Debug checklist & useful commands
- After
job, verify output:
`sh
cat .circleci/config_continue.yml
`- If
fails to load the config, check workspace:
`sh
ls -la .
`- Validate generated config:
sh
circleci config validate /tmp/generated-config.yml
`- Check GitHub webhook deliveries if pipeline never triggered.
- If script errors show
, ensure shebang is bash.---
Best practices & recommendations
- Filters: If a dependent job has a filter, the job that generates its
configuration must also have the same filter. Otherwise, the workflow won't be
generated.
- Caching: Use
persist_to_workspace for dependent jobs in a sequence. Use
save_cache and restore_cache for independent jobs to avoid redundant
installations.
- Lock Dependencies: Always use yarn install --frozen-lockfile or npm ci
to ensure strict adherence to your lock file, preventing inconsistent builds.
- Docker Images: Use lightweight Docker images (e.g., versions, but
never busybox) to reduce build times.
- Shell Scripts: Always make shell scripts executable
(chmod +x script.sh).
- Security: Store sensitive data in CircleCI contexts or environment
variables. Access information like branch names and tags using built-in
environment variables ($CIRCLE_BRANCH, $CIRCLE_TAG, $NPM_TOKEN).
- Persist and access generated files with persist_to_workspace and
attach_workspace.
- Always run checkout before attach_workspace in steps/pre-steps.
- Be explicit about tag filters for generation jobs if you expect tag-triggered
runs.---
Result of the fixes
- Fixed missing tag runs and ensured
generate-config runs for release tags by
adding tag filters (or removing improper requires).
- Guaranteed .circleci/config_continue.yml is produced and available at
runtime — path-filtering/filter sees and uses it correctly.
- Resolved workspace/checkout race conditions — eliminated
Directory not empty and not a git repository errors.
- Switched base image — eliminated missing git/ssh/bash problems.
- pipeline.parameters (mapping booleans) are correctly used to gate when:`---
This README documents the practical issues I hit while building a real
modular/dynamic CircleCI pipeline and the applied engineering fixes. The fixes
are small but crucial (workspace ordering, explicit parameters, correct images,
and filter/require discipline) — together they turn the dynamic configuration
pattern from fragile into reliable.
---