fawkes

This project is used to centralize indexability logic for our Search
microservices and Sitemap Generator. The main purpose is to completely remove
discrepancies between pages indexed by Search and page URLs stored in Sitemap.

Link to documentation: https://studyportals.atlassian.net/wiki/x/BwAdvw

Base structure

- organisations-seo (used by Organisation Search)
- scholarships-seo (used by Scholarship Search)
- sitemap-generator-seo (used by Sitemap Generator)
- src
- common
- organisations
- scholarships
- sitemap-generator

Automatic Documentation System

Overview

This system automatically generates and maintains documentation for Fawkes
indexing rules across multiple projects. It creates a hierarchical structure of
Confluence pages that document all indexing policies and rules, ensuring that
the documentation is always up-to-date with the actual codebase.

Link to documentation:
Fawkes Indexing Rules Documentation

Features

- Dynamic Documentation: Automatically extracts indexing rules and policies
directly from the codebase
- Multi-Page Structure: Creates a main index page with separate pages for
each project
- Hierarchical Organization: Structures content as Projects → Policies →
Rules
- Automatic Page Creation: Intelligently creates or updates pages as needed
- Cross-Page Navigation: Provides intuitive navigation between all pages

Architecture

The system follows SOLID design principles to ensure maintainability and
extensibility:

- Single Responsibility: Each class has one specific responsibility
- Open/Closed: Easily extendable without modifying existing code
- Liskov Substitution: Interchangeable components through well-defined
interfaces
- Interface Segregation: Clean, focused interfaces
- Dependency Inversion: High-level modules depend on abstractions

$3

1. Content Builder: Extracts project, policy, and rule data from the Fawkes
system
2. Policy Fetchers: Project-specific components that fetch and format policy
data
3. Document Builder: Generates structured HTML content for Confluence pages
4. Confluence Updater: Handles creation and updating of Confluence pages
5. Token Retriever: Securely retrieves API token for Confluence access

How It Works

1. The system starts by collecting all project data using the ContentBuilder
2. For each project, it extracts policies and rules using project-specific
fetchers
3. The DocumentBuilder generates the content for the main index page and each
project page
4. The ConfluenceUpdater creates or updates pages in Confluence as needed
5. Each project page links back to the main index, and the main index links to
all project pages

Setup and Configuration

$3

The system requires the following environment variables:

``
CONFLUENCE_EMAIL=your-email@example.com
CONFLUENCE_API_TOKEN=your-api-token
CONFLUENCE_BASE_URL=https://your-instance.atlassian.net
CONFLUENCE_SPACE_KEY=YOUR_SPACE_KEY
CONFLUENCE_MAIN_PAGE_TITLE=Fawkes Indexing Rules Documentation
`

> Important Note on Credentials: Currently, the system uses personal
> credentials (email and API token) for Confluence access. This is a temporary
> solution and should be replaced with a team or service account in the future
> to avoid dependency on a specific individual's account.

$3

The system runs as an AWS Lambda function and is automatically triggered as part
of the CI/CD pipeline. This ensures documentation is always updated whenever new
code is deployed:

`yaml

buildspec.yml (excerpt)

post_build:
commands:
- |
if [ -z "$PR_ID" ]; then
LAMBDA_NAME="FawkesConfluenceUpdater-$STAGE"
echo "Invoking Lambda function $LAMBDA_NAME"
echo '{"source":"cicd-pipeline","timestamp":"'$(date -u +"%Y-%m-%dT%H:%M:%SZ")'"}' > payload.json
aws lambda invoke --function-name $LAMBDA_NAME --invocation-type Event --payload fileb://payload.json response.json || echo "Lambda invocation failed, but continuing"
fi
`

Trigger Flow:

1. When code is pushed to the main branch, the CI/CD pipeline is triggered
2. After successful deployment of a new Fawkes version, the post-build phase
executes
3. The Lambda function is invoked with a simple payload containing the source
and timestamp
4. The Lambda function authenticates with Confluence and executes the
documentation generation process
5. This only happens for production deployments (not PR builds)

This automated approach ensures documentation stays in sync with the actual
codebase without manual intervention.

Scalability and Extension

$3

To add documentation for a new project:

1. Create a new class that implements the IPolicyFetcher interface
2. Implement the getProjectName() and fetchPolicies() methods
3. Register the new fetcher in the ContentBuilder constructor

Example:

`typescript
export class NewProjectPolicyFetcher implements IPolicyFetcher {
getProjectName(): string {
return 'New Project Name';
}

fetchPolicies(): IPolicy[] {
// Implement policy fetching logic
return [...policies];
}
}

// Then update ContentBuilder:
constructor(fetchers?: IPolicyFetcher[]) {
this.policyFetchers = fetchers || [
// Existing fetchers
new NewProjectPolicyFetcher()
];
}
`

$3

The DocumentBuilder class contains methods for generating page content. To
customize the format:

1. Modify the buildMainDocument() method for changes to the main index page
2. Modify the buildProjectDocument() method for changes to project pages
3. Modify the buildPolicySection() and buildRulesTable()` methods for changes
to policy and rule sections

Conclusion

This documentation system ensures that Fawkes indexing rules are always
well-documented and accessible to the team. By automatically generating
documentation directly from the codebase, it eliminates the risk of
documentation becoming outdated or inconsistent with the actual implementation.