---

[![Build Status][build-badge]][build]
[![version][version-badge]][package]
[![MIT License][license-badge]][license]

The problem

Deploying your app to multiple regions along with your data is a great way to
make your app really fast, but there are two issues:

1. Read replica instances can only read from the database, they cannot write to
it.
2. There's an edge case where the user could write to the primary instance and
then read from a replica instance before replication is finished.

The first problem is as simple as making sure you use a special fly-replay
response so Fly can pass the request to the primary instance:

!a visualization of the user making a request which is sent to a read replica and replayed to the primary instance

But the second problem is a little harder. Here's how we visualize that:

!continuing the previous visualization with the edge case that the read replica responds to a get request before the replication has finished

This solution

This module comes with several utilities to help you work around these issues.
Specifically, it allows you an easy way to add a special cookie to the client
that identifies the client's "transaction number" which is then used by read
replicas to compare to their local transaction number and force the client to
wait until replication has finished if necessary (with a timeout).

Here's how we visualize that:

!a visualization that shows the primary server sending a transaction number to the client and then the subsequent get request is sent to the replica which waits for replication to finish before responding

The even better proxy solution

At the time of this writing, LiteFS just released experimental support for a
proxy server that will handle much of this stuff for you. You simply configure
the proxy server in your litefs.yml and then you don't need to bother with the
tx number cookie or ensuring primary on non-get requests at all. The litefs-js
module is still useful for one-off situations where you're making mutations in
GET requests for example, or if you need to know more about the running
instances of your application, but for most of the use cases, you can get away
with using the proxy.
Learn more about using the proxy from this PR.

Installation

This module is distributed via [npm][npm] which is bundled with [node][node] and
should be installed as one of your project's dependencies:

``
npm install --save litefs-js
`

Unless you plan on using lower-level utilities, you'll need to set two
environment variables on your server:

- LITEFS_DIR - the directory where the .primary file is stored. This should
be what you set your fuse.dir config to in the litefs.yml config.
- DATABASE_FILENAME - the filename of your sqlite database. This is used to
determine the location of the -pos file which LiteFS uses to track the
transaction number.
- INTERNAL_PORT - the port set in the fly.toml (can be different from PORT
if you're using the litefs proxy). This is useful for the
getInternalInstanceDomain utility.

Usage

The best way to use this is with the latest version of LiteFS which supports the
proxy server:

`yml


litefs.yml

...

proxy:
# matches the internal_port in fly.toml
addr: ':${INTERNAL_PORT}'
target: 'localhost:${PORT}'
db: '${DATABASE_FILENAME}'

...

`

From there all you really need litefs-js for is when you run your database
migrations. For example:

`js
// start.js
const fs = require('fs')
const { spawn } = require('child_process')
const os = require('os')
const path = require('path')
const { getInstanceInfo } = require('litefs-js')

async function go() {
const { currentInstance, currentIsPrimary, primaryInstance } =
await getInstanceInfo()

if (currentIsPrimary) {
console.log(
Instance (${currentInstance}) in ${process.env.FLY_REGION} is primary. Deploying migrations.,
)
await exec('npx prisma migrate deploy')
} else {
console.log(
Instance (${currentInstance}) in ${process.env.FLY_REGION} is not primary (the primary instance is ${primaryInstance}). Skipping migrations.,
)
}

console.log('Starting app...')
await exec('node ./build')
}
go()

async function exec(command) {
const child = spawn(command, { shell: true, stdio: 'inherit' })
await new Promise((res, rej) => {
child.on('exit', code => {
if (code === 0) {
res()
} else {
rej()
}
})
})
}
`

The only other thing you need to worry about is if you ever perform a mutation
as a part of a GET request. The proxy will handle forwarding non-GET requests
for you automatically, but if you need to write to a non-primary in a GET,
you'll need to use some of the utilities to ensure the server replays the
request to the primary.

$3

Integrating this with your existing server requires integration in two places:

1. Setting the transaction number cookie on the client after mutations have
finished
2. Waiting for replication to finish before responding to requests

Low-level utilities are exposed, but higher level utilities are also available
for express and remix.

Additionally, any routes that trigger database mutations will need to ensure
they are running on the primary instance, which is where ensurePrimary comes
in handy.

$3

`ts
import express from 'express'
import {
getSetTxNumberMiddleware,
getTransactionalConsistencyMiddleware,
getEnsurePrimaryMiddleware,
} from 'litefs-js/express'

const app = express()
// this should appear before any middleware that mutates the database
app.use(getEnsurePrimaryMiddleware())

// this should appear before any middleware that retrieves something from the database
app.use(getTransactionalConsistencyMiddleware())

// ... other middleware that might mutate the database here
app.use(getSetTxNumberMiddleware())

// ... middleware that send the response here
`

The tricky bit here is that often your middleware that mutates the database is
also responsible for sending the responses, so you may need to use a lower-level
utility like setTxCookie to set the cookie after mutations.

$3

Until we have proper middleware support in Remix, you'll have to use the express
or other lower-level utilities. You cannot currently use this module with the
built-in Remix server because there's no way to force the server to wait before
calling your loaders. Normally, you just need to use
getTransactionalConsistencyMiddleware in express, and then you can use
appendTxNumberCookie as shown below.

Of course, instead of using express with
getTransactionalConsistencyMiddleware, you could use
await handleTransactionalConsistency(request) to the top of every loader if
you like:

`tsx
// app/root.tsx (and app/routes/*.tsx... and every other loader in your app)
export function loader({ request }: DataFunctionArgs) {
await handleTransactionalConsistency(request)
// ... your loader code here
}
`

The same thing applies to getEnsurePrimaryMiddleware as well. If you need or
like, you can use await ensurePrimary() in every action call or any
loaders that mutate the database (of which, there should be few because you
should avoid mutations in loaders).

We're umm... really looking forward to Remix middleware...

The appendTxNumberCookie utility should be used in the entry.server.ts file
in both the default export (normally people call this handleDocumentRequest
or handleRequest) and the handleDataRequest export.

`tsx
// app/entry.server.ts
import { appendTxNumberCookie } from 'litefs-js/remix'

export default async function handleRequest(
request: Request,
responseStatusCode: number,
responseHeaders: Headers,
remixContext: EntryContext,
) {
// Most of the time, all mutations are finished by now, but just make sure
// you're finished with all mutations before this line:
await appendTxNumberCookie(request, responseHeaders)
// send the response
}

export async function handleDataRequest(
response: Response,
{ request }: Parameters[1],
) {
// Most of the time, all mutations are finished by now, but just make sure
// you're finished with all mutations before this line:
await appendTxNumberCookie(request, response.headers)
return response
}
`

$3

There are several other lower-level utilities that you can use. They allow for
more customization and are documented via jsdoc. Utilities you may find helpful:

- ensurePrimary - Use this to ensure that the server that's handling the
request is the primary server. This is useful if you know you need to do a
mutation for that request.
- getInstanceInfo - get the currentInstance and primaryInstance hostnames
from the filesystem.
- waitForUpToDateTxNumber - wait for the local transaction number to match the
one you give it
- getTxNumber - read the transaction number from the filesystem.
- getTxSetCookieHeader - get the Set-Cookie header value for the transaction
number
- checkCookieForTransactionalConsistency - the logic used to check the
transaction number cookie for consistency and wait for replication if
necessary.
- getAllInstances - get all the instances of your app currently running
- getInternalInstanceDomain - get the internal domain for the current instance
so you can communicate between instances of your app (ensure you've set the
INTERNAL_PORT environment variable to what appears in your fly.toml).

How it works

This module uses the special .primary directory in your Fuse filesystem to
determine the primary
(litefs primary docs), and the -pos
file to determine the transaction number
(litefs transaction number docs).

When necessary, replay requests are made by responding with a 409 status code
and a fly-replay` header
(docs on dynamic request routing).

Inspiration

This was built to make it much easier for people to take advantage of
distributed SQLite with LiteFS on Fly.io. The bulk of the logic was extracted
from
kentcdodds/kentcdodds.com.

LICENSE

MIT

[npm]: https://www.npmjs.com
[node]: https://nodejs.org
[build-badge]: https://img.shields.io/github/actions/workflow/status/fly-apps/litefs-js/validate.yml?logo=github&style=flat-square
[build]: https://github.com/fly-apps/litefs-js/actions?query=workflow%3Avalidate
[version-badge]: https://img.shields.io/npm/v/litefs-js.svg?style=flat-square
[package]: https://www.npmjs.com/package/litefs-js
[license-badge]: https://img.shields.io/npm/l/litefs-js.svg?style=flat-square
[license]: https://github.com/fly-apps/litefs-js/blob/main/LICENSE