markdown-graph-content-layer-database
npm install @braindb/remark-dataview> [!WARNING]
> Work in progress. Expect breaking changes.
> Dataview is a live index and query engine over your personal knowledge base. You can add metadata to your notes and query them with the Dataview Query Language to list, filter, sort or group your data. Dataview keeps your queries always up to date and makes data aggregation a breeze.
>
> -- Obsidian Dataview
The project is inspired by Obsidian DataView, but there are some differences.
First, I decided to use real SQL instead of the custom query language by DataView (DQL).
Pros:
- It is easy to learn due to the large community and extensive documentation.
- It is easy to implement since the project already uses a relational database (SQLite).
Cons:
- It exposes the data structure, so any changes to the model will break downstream projects.
- It locks the project into using a relational database, though I had considered experimenting with a graph database.
Real SQL would at least allow the data to be displayed as a table (since all SQL clients output results in tables, even if there's only one cell). Other templates could include:
- A list, if there's only one column.
- A nested list.
- Etc.
The template can be customized through a meta string (the "fence meta"):
```md`dataview `
SELECT a, b, c FROM nodes;
Additionally, I can add a custom function to format output. For example, dv_link(text, url) would output a Markdown link: text.
Remark plugin will find all code blocks (or "fence blocks") with the language "dataview," treat the content as a query, execute the query using BrainDB, transform the result into MDAST, and replace the original code block. This way, the result can be post-processed by other Remark/Rehype plugins.
To implement custom functions, I would use an SQL parser to remove custom functions from the SQL before executing it. These functions would then be used to format the resulting data before transforming it into MDAST.
`mermaid`
flowchart LR
s[SQL parser] --> t[data query] --> e[fetch data] --> tm[transform into MDAST]
s --> t1[formatting config] --> tm
That's it.
As an improvement, a VSCode extension could be implemented to highlight syntax and provide autocompletion. I believe this should be possible with langium-sql.
Related: VSCode Markdown Fenced Code Block Grammar Injection Example
Perhaps the table template could be combined with sortable tables?
For example, one could select data from the links table, convert it to DOT format, and output it as a code block, which would then be processed by @beoe/rehype-graphviz.
Alternatively, one could use @beoe/rehype-gnuplot to generate plots based on the data.
This is another bonus of this architecture — it is modular and can be integrated with other solutions.
``md`dataview list root_class=column-list`
SELECT upper(substr(frontmatter ->> '$.title', 1, 1)), dv_link()
FROM documents
WHERE frontmatter ->> '$.draft' IS NULL OR frontmatter ->> '$.draft' = false
ORDER BY frontmatter ->> '$.title'
LIMIT 2;
``md`dataview list root_class=column-list`
SELECT date(updated_at / 1000, 'unixepoch'), dv_link()
FROM documents
WHERE frontmatter ->> '$.draft' IS NULL OR frontmatter ->> '$.draft' = false
ORDER BY updated_at DESC
LIMIT 2;
``md`dataview list`
SELECT dv_link(), dv_task()
FROM tasks JOIN documents ON documents.path = tasks.from
WHERE frontmatter ->> '$.draft' IS NULL OR frontmatter ->> '$.draft' = false
ORDER BY updated_at DESC, path, tasks.start
LIMIT 2;
``md`dataview list root_class=column-list`
SELECT tags.value as tag, dv_link()
FROM documents, json_each(frontmatter, '$.tags') tags
WHERE frontmatter ->> '$.draft' IS NULL OR frontmatter ->> '$.draft' = false
ORDER BY tag
LIMIT 2;
`javascript
import unified from "unified";
import markdown from "remark-parse";
import remarkDataview from "@braindb/remark-dataview";
// bdb is an instnce of BrainDB
let processor = unified()
.use(markdown, { gfm: true })
.use(remarkDataview, { bdb });
``