WatermelonDB Firestore sync
npm install melon-fireMelonFire implements the WatermelonDB sync
protocol via
RNFirebase's React Native integration
with Firestore. By integrating MelonFire, you can easily back up your
WatermelonDB database with your Firestore instance in the cloud without writing
any code yourself.
``
npm install melon-fire
or
`
yarn add melon-fire
`Typescript
import firestore from "@react-native-firebase/firestore";
import { Database } from "@nozbe/watermelondb";
import { syncMelonFire } from "melon-fire";
async function onSyncButtonPushOrRandomTimer(
db: Database,
userId: string,
) {
const syncDocRef = firestore()
.collection("users")
.doc(userId);
return await syncMelonFire(db, syncDocRef);
}
`
, which was needed because microbundle doesn't support having both a default export and a named export. So you'll now need to import { syncMelonFire } from "melon-fire" instead.$3
-
userId doesn't have to literally be a user id. It can be any valid Firestore
doc ID that you want associated with the backup/sync in Firestore. Similarly,
your collection doesn't have to be "users" — it can be anything you want.
- You pass a DocumentReference to MelonFire, which will then write several of
its own fields into the document (currently: melonLatestRevision,
melonLatestDate, and melonBatchTokens). It'll also create
a collection under syncDocRef for each table in your database, as well as a
collection called melonDeletes where it stores deletions. If you don't
mind these fields and collections living in a shared doc with other things
(e.g. if you want MelonFire's data to be kept in your user's profile doc),
it's fine; MelonFire won't overwrite your other fields/collections.
- syncMelonFire will throw if there are sync errors. Note that, in accordance
with WatermelonDB's guidance, syncMelonFire will actually catch the first
error it receives and retry sync one time on its own. This, for instance,
automatically resolves the most common sync issue, which is that another
writer has updated the cloud records since you last pulled changes. But in
cases where the first retry fails, syncMelonFire will throw.
- Depending on the size of your DB and your network connection, syncMelonFire
could take a while (especially if, say, you're pulling or pushing a ton of
changes). You don't necessarily have to await if you're sure
your app can move on and do other things while sync is processing in the
background, but that might be a bad idea for most apps because you don't want
to be modifying the database even while sync is running (since your new
changes might conflict with what it pulls down from the cloud). But this is
ultimately up to your judgment — you know your app best.
- Please consider MelonFire's limitations before adopting it.How It Works
The WatermelonDB sync
protocol is actually
non-trivial to implement, especially if you'd like to sync your DB into
Firestore. For example:
1. Timestamps are suggested by the WatermelonDB docs, but they're subtly and
annoyingly tricky. Have you considered leap seconds?
Spring-forward/fall-back? Millisecond-level jitter/inconsistency on servers?
If you get timestamps wrong, your DB backup gets corrupted.
2. Firestore has a batch write / transaction
limit
of 500 writes. How will you guarantee atomic write operations when backing
up, if you have more than 500 edits since the last write? Existing solutions
ignore this, but MelonFire makes it possible to keep a consistent backup in
Firestore despite this limitation.
3. WatermelonDB requires you to track the difference between creates and
updates, as well as to maintain lazy deletion records. This makes the
"obvious" implementation of simply calling
firestore().blah.set(),
firestore().blah.update(), and firestore().blah.delete() incorrect.
4. Reads can be inconsistent if you're not careful to shield against concurrency
between readers and writers. MelonFire ensures consistent reads without
succumbing to Firestore's transaction limits.$3
MelonFire is a client-side library that relies on RNFirebase's Firestore
integration. This means you won't need to push or maintain cloud functions, but
can instead just integrate MelonFire into your app for easy DB sync or backup
into Firestore. MelonFire overcomes the challenges listed above by:
1. using atomically-incremented change counters instead of timestamps. This
guarantees ordering even if the server changes its mind about what time it
is. This also makes sync independent of client time.
2. using Firestore transactions when you have less than 500 changes, for maximum
efficiency. But when your changeset is larger than that, MelonFire writes
your changes into a side doc and a set of collections (one per table); these
are then atomically integrated into the main backup only when all writes are
successful, guaranteeing that half-completed large batches never corrupt the
database.
3. tracking changesets distinctly so that creates remain creates, updates remain
updates, and deletes are persisted lazily in order for future updates to be
correct regardless of the number of revisions pulled.
4. implementing an append-only, version-tracked set of records in Firestore that
shield readers against concurrent writers.
Though MelonFire's architecture allows it to maintain consistency despite
Firestore's 500-write limit, note that a consequence is that large batch writes
are stored in Firestore without merging with past or future records. At the
extreme, if your app only ever pushes huge batches (e.g. you modify the same 10k
records every day, and then want to sync with the cloud daily), you'll be
storing a copy of those records for each sync. Whereas if you only ever pushed
less than 500 records at a time, MelonFire would never have more than one doc in
Firestore for any one row in your database. If someone has a clever (and
functionally correct) alternative to this implementation, please let me
know.
Limitations
-
schemaVersion and migrations` from WatermelonDB's sync protocol aren't