DTL

WebSite | Repository |
Wiki | Bug Reports |
Live Help (discord) | By Jay Kuri

DTL, short for Data Transformation Language, is a versatile tool ideal for
managing and transforming structured data, especially from sources like
NoSQL databases and APIs. It seamlessly integrates into your JavaScript
projects, whether you're working in a Node.js environment or directly in the
browser. Additionally, DTL offers powerful command-line tools that excel in
handling various formats, including JSON, YAML, CSV, and plain text. These
tools are perfect for processing and exploring existing data sets.

Whether you need something as simple as JSON templating, more complex tasks
like format conversion, or intricate data restructuring, DTL stands out as
an invaluable resource for developers dealing with any form of structured
data manipulation.

If your work involves manipulating structured data, and what development
doesn't, DTL offers an efficient and easy to understand toolset to help you.

What DTL looks like

DTL looks like JSON and has a very familiar syntax:

``json
{
"out": {
"name": "(: &( $first_name ' ' $last_name ) :)",
"order_total": "(: $order_subtotal + $tax_amt :)",
"origin": "Import from legacy database",
"address": {
"street": "(: $address1 :)",
"unit": "(: $address2 :)",
"city": "(: $addr_city :)",
"state": "(: $addr_state :)",
"postal": "(: $addr_zip :)"
}
}
}
`

We'll explore this example in detail a bit later.

Installing DTL

DTL can be installed within your project, or globally for command-line use.

$3

For global access to DTL's command-line tools, install DTL globally using npm or yarn:

- Using npm
`bash
npm install -g dtl-js
`

- Using yarn
`bash
yarn global add dtl-js
`

$3

To install DTL as part of your project, navigate to your project directory and run:

- Using npm
`bash
npm install dtl-js
`

- Using yarn
`bash
yarn add dtl-js
`

$3

After installation, you can verify the installation of DTL:

- For global installations, check the version to ensure it's installed correctly:
`bash
dtl --version
`

- For local installations, you can check your package.json to see if dtl-js is listed under dependencies.




What DTL is for...

DTL, is built for transforming and manipulating data structures. At its most
basic level, DTL can act as a templating tool for JSON or YAML, enabling
straightforward data reformatting and restructuring. However, **DTL's
capabilities extend far beyond simple templating**. DTL is ideal for complex
tasks such as data **extraction, conversion, and preparation for analysis or
reporting**. Its simple syntax and robust functions facilitate the clear and
concise expression of complex data transformations, making DTL an
extremely useful tool for a wide range of data processing needs.

Basic Structure

In DTL, transforms are generally defined within a JSON object, which
forms the transform library. Each key in this JSON object is a distinct
transform, functioning like a template or function for how input data should
be outputted. The out key is special as it's the default transform used by
DTL.apply(), akin to the main function in other languages.

In most cases, the value of out will be an object that looks like your
desired output structure, guiding the transformation of input data to this
format. This library can have multiple transforms, each capable of
referencing and calling one another. When a transform is executed through
DTL.apply(), the provided data is represented as $. within the
transform.

Syntax Essentials

$3

Every DTL expression must be enclosed in happy tags: (: :). Happy tags
tell DTL to do something. Anything not in Happy tags is passed through
untouched. Happy tags are called that because they resemble happy emoji.

$3

Reference data using dot notation, with the root represented as $.
For nested data access, use dot notation, e.g., $user.address.street
Data can be referenced with or without a leading ., in other words,
$.user and $user are equivalent.

And here's a bit of good news: in DTL, trying to access a property of a
non-existent or non-object value won't send you into the dreaded JavaScript
spiral of Cannot read properties of undefined. In DTL, it just sensibly
returns undefined. No drama, no crash – just the peace of mind knowing DTL
has got your back in these scenarios.

$3

In DTL expressions, commas are treated like whitespace and are not necessary.

$3

Use built-in helper functions for complex operations, e.g.,
math.round($number). To call another transform in the library, use
($input_data -> transform_name). In this example, $input_data would be
sent to transform_name and the result would be the new data.

Operations look like this: $data op $otherdata, e.g., $number1 + $number2.

$3

Use parentheses to control the order that things should be done, e.g.,
($number1 + $number2) * $number3. DTL understands mathematical order of
operations, but there's no PEMDAS for data types, so parenthesis will help
make your intentions clearer.

$3

For conditionals, use ?(: $condition $truevalue $falsevalue). You can use
this to send data to one or another transform depending on some condition:
?(!empty($address) ($. -> process_address) ($. -> no_address))

In this example, ?(!empty($address) ($. -> process_address) ($. -> no_address)),
DTL checks if $address is not empty. If it's not, the data
is processed with the process_address transform; if $address is empty,
the no_address transform is used instead. This showcases DTL's ability to
elegantly handle conditional data routing based on the presence or absence
of data.

$3

A mix of static (literal values) and dynamic (DTL expressions) is common
in transforms. Only dynamic expressions use happy tags.

$3

You can loop or iterate over lists of data. In all helpers that iterate,
like map or grep, use $item, $index, and $all for referencing.
$item is the current item. $index is the index of the current item and
$all is the entire list or object.

$3

Transforms only have access to what was given to them (from DTL.apply() or
another transform in the library) and have no way to access information
other than that. This is a good thing, it means your transform will always
do exactly what you want. Within a transform, $. refers to the input data
of that transform.

$3

In DTL, mathematical operations are carried out using Arbitrary Precision
Mathematics. This approach is a game-changer, especially when you compare it
to JavaScript, where 0.1 + 0.2 equals 0.30000000000000004 (seriously, try it
in your web or Node.js console). In DTL, it simply equals 0.3, just as you
would logically expect. This means that when you're doing calculations in
DTL, you're getting the exactly correct answer every time. DTL's precision
makes it ideal for scenarios where mathematical accuracy isn't just a
preference, it's a necessity.

$3

In DTL, the concept of empty is key. Empty in DTL means the data has no
meaningful value, e.g. undefined, null, an empty string, array, or object.
The empty() function returns true if a value is 'empty'

If you've ever wondered why [] evaluates to true or been bitten by
if(formfield) evaluating to false if someone entered 0, we feel your pain.
In most cases, you really want to know if a variable has a meaningful
value. empty() is here to help.

Complementing this, the fne() (First Non-Empty) function scans a list of
values and returns the first one that is not considered empty, streamlining
the selection of valid data from multiple sources. It looks like this:
fne($first_place_to_look $second_place_to_look defaultvalue)




Example in JSON

Here's an example of DTL:

`json
{
"out": {
"name": "(: &( $first_name ' ' $last_name ) :)",
"order_total": "(: $order_subtotal + $tax_amt :)",
"origin": "Import from legacy database",
"address": {
"street": "(: $address1 :)",
"unit": "(: $address2 :)",
"city": "(: $addr_city :)",
"state": "(: $addr_state :)",
"postal": "(: $addr_zip :)"
}
}
}
`

In this JSON transformation example:

- The name field uses the &() helper to concatenate $first_name and
$last_name with a space in between.
- The order_total field does some simple math, adding $order_subtotal
and $tax_amt together.
- origin is a static string, "Import from legacy database".
- The address field is an object containing multiple expressions. This shows
how your transforms mirror your desired output format.

`

$3

Using DTL is as straightforward as it gets:

1. Create Your Transform: Defining how you want your output data to
look. This step involves crafting the template or transformation logic in
a JSON object, specifying how the input data should be processed and
structured.

2. Call DTL.apply(): Once your transform is ready, the next step is to
apply it to your data. This is done using the DTL.apply() method, which
takes your input data and the transform, performs the transformation as defined,
and returns the result.

And that's it — there is no step 3. DTL's design is centered around this
simplicity, making data transformations easy and accessible.

Below is a practical example demonstrating these two steps in a Node.js script:

$3

Now that we've covered the basics of DTL, let's see it in action with a
simple Node.js script. This example demonstrates how to use DTL to
transform some sample data and output the result.

`javascript
// Import the DTL module
const DTL = require('dtl-js');

// Example input data
const inputData = {
first_name: "John",
last_name: "Doe",
order_subtotal: 100,
tax_amt: 20,
address1: "123 Main St",
address2: "Apt 4",
addr_city: "Springfield",
addr_state: "IL",
addr_zip: "62704"
};

// Example DTL transform
const transformLibrary = {
"out": {
"name": "(: &( $first_name ' ' $last_name ) :)",
"order_total": "(: $order_subtotal + $tax_amt :)",
"origin": "Import from legacy database",
"address": "(: $. -> formatAddress :)"
},
"formatAddress": {
"street": "(: $address1 :)",
"unit": "(: $address2 :)",
"city": "(: $addr_city :)",
"state": "(: $addr_state :)",
"postal": "(: $addr_zip :)"
}
};

// Applying the transform to the input data
const transformedData = DTL.apply(inputData, transformLibrary);

console.log("Transformed Data:", JSON.stringify(transformedData, undefined, 4));
`

This script demonstrates how to import DTL, define a transform library,
apply it to some sample data, and output the transformed result. It's a
practical example of how DTL can be utilized in a typical Node.js
environment to manipulate and transform data structures.

As you can see, integrating DTL into your project is super simple. With its
simple syntax and powerful capabilities, DTL is all set to make your job
easier. Now, go use it to streamline your code, make your workflows more
efficient or just do some awesome stuff!

Full Editor support

Working with DTL is made significantly easier thanks to comprehensive editor
support. Full syntax highlighting in the editor bundles not only clarifies
DTL syntax but also enhances your coding efficiency. With snippets for all
built-in functions, these bundles facilitate a more intuitive and error-free
coding experience. We encourage you to download the appropriate bundle for
your editor of choice.

Look for DTL in the VSCode extension marketplace, or if you use Vim, Sublime
Text or other Textmate compatible editors, you can download the language
support files from the
Releases page of the DTL Repository
Streamline your coding with these powerful features for an improved,
error-free DTL coding experience.

Common Problems That DTL Solves

DTL is adept at addressing a wide range of frequently encountered data
transformation challenges. Here are some common problems where DTL can be
particularly beneficial:

1. Creating Data Templates: Effortlessly generate structured data from
templates. DTL excels in scenarios like auto-generating JSON configurations
or preparing data payloads for APIs, making these tasks quick and error-free.

2. Simple Data Transformations: Whether it's renaming keys in JSON objects,
restructuring data (such as flattening nested structures), or performing
basic aggregations and calculations, DTL simplifies these common tasks.

3. Format Conversion: DTL streamlines the conversion between various data
formats. Transforming data from JSON to CSV, XML to JSON, or YAML to JSON
becomes a hassle-free process.

4. Data Cleaning and Normalization: Clean or normalize data from diverse
sources to fit specific standards. Tasks like removing unwanted fields,
standardizing date formats, or converting data types are made straightforward
with DTL.

5. Data Validation: Ensure that incoming data conforms to specified
criteria or formats. DTL's data validation capabilities are particularly
useful for data from external sources or user inputs.

6. Extracting Specific Data from Complex Structures: Extracting necessary
information from complex or nested data structures, such as certain fields
from deeply nested JSON objects, is made efficient and simple.

7. Automating Repetitive Data Tasks: DTL helps automate repetitive data
manipulation tasks, saving time and minimizing the risk of errors in routine
operations.

Using DTL, you can address these common data transformation challenges
more efficiently, allowing you to focus on the more intricate aspects of your
projects.

TL;DR

!demonstration of DTL REPL toolDTL Repl demo

MOAR Usage

To open the dtlr data exploration environment:

`
$ dtlr filename.json
`

To do bulk data manipulation on the command line:

See available command line options:

`
$ dtl -h
`

(Movie examples below use sample movie data from Awesome JSON Datasets )

`


Convert movies.json to yaml

$ dtl -o movies.yaml movies.json

How many movies are listed?

$ dtl -e 'length($.)' movies.json

Get all the movies where Bill Murray was in the cast

$ dtl -e 'grep($. "(: member($item.cast Bill Murray) :)")' movies.json

Extract the name field in the JSON file in uppercase.

$ dtl -e 'uc($name)' data.json

Apply the transform expression specified in the transform.dtl file to the CSV input file.

$ dtl -f transform.dtl input.csv

Process input.json, output CSV data in output.csv including all fields found in input.json.

$ dtl -o output.csv input.json

Process input.json, output CSV data on stdout using the fields first_name, last_name and primary_email


dtl -O csv -C first_name,last_name,primary_email input.json
`

Is DTL safe?

Unlike regular code, the output of DTL can only include the information
provided to the DTL call, so DTL transforms are much safer to use than
the code that would be required to produce the same output. They're also
a heck of a lot easier to read... AND since they are self-contained and
don't refer to your code, they are safe and easy to share.

DTL can be used within javascript code (node.js and browser, and even inside
MongoDB) or it can be used on the command line with the DTL cli tools.

Why should I care?

DTL is interesting for several reasons:

* Clarity - DTL is purpose-built for data transformation and only data
transformation. It is not intended to be a general-purpose programming
language and is therefore simple to learn and free of unnecessary
components.

* Portable - DTL transforms are self-contained and transferrable between
applications. Since they can be stored as JSON, they can even be
kept in your database.

* Security - DTL transforms only have access to the data
that was provided as input. DTL transforms have no other variable or
system access, so they are much safer to use than custom code.

* Stateless - DTL transforms have no access to previous state, only
to the data provided and therefore avoid bugs related to bleed
over or inadvertant modification, one of the most common sources of
bugs.

* Provable - It is trivial to create a DTL transform to verify the
output of another. This obviously allows for simple test-creation.
What may not be obvious is that these verification transforms can
be used to check data at run-time.

* Non-linear - DTL transforms define how to arrive at the desired
data. They do not define a sequence of steps. This means that
each expression is independent and not subject to bugs due to
issues that occurred in other expressions.

* Stable - DTL has been in use in production since 2013 and has
been its own separate project since 2015. It is being used in
many production applications, handling many millions of
transformations every day.

* DTL is a language with an implementation. The DTL npm module is only
one implementation of the DTL language. The DTL module contains
hundreds of tests that verify the language is behaving properly.
This allows DTL to be implemented and verified in any programming
language.

Where did DTL come from?

Truth be told, DTL was not originally intended to be it's own thing.
DTL began as an expression language inside a meta-programming engine built
by Jay Kuri (me) during my work at Ionzero, a
company I founded. One of the first applications of this engine was a system
built to handle linking other systems together. I created the language out
of the need for a way to define how to map data from one system to another
without resorting to hard-coded custom code.

I also realized during the course of this work that DTL could be used
for far more than I ever had originally envisioned. As a result of this
realization, over time, I refined the DTL language and eventually
extracted it into a self-contained module that could be used in any
system and proceeded to do so.

I decided to release DTL as open source in the hopes that others would
find it as useful and as powerful as I have.

The DTL command line tools

As mentioned earlier in this document, if you have installed the DTL
package globally with npm or yarn, you will have two command line tools for
working with DTL. The dtl cli tool works on bulk data

If you want to just take DTL for a spin without coding you can use the
dtlr tool. The dtlr cli tool is an interactive REPL (Read
Execute Print Loop) tool you can use to test out expressions and get
help.

The dtl cli tool works on bulk data and is designed to process CSV, yaml
and JSON, as well as JSONLines data. It can produce CSV, yaml, JSON and
JSONLines data as well, regardless of whether the input data was the
same type. You can learn more about how to use it by using the
dtl -h command. Note that by default it sends its output to stdout.
If you'd rather have the output go into a file, use the -o filename.json
option.

Feedback and where to get help

We are always looking for constructive feedback. If you have ideas on
how we might improve DTL, please reach out. If you are looking for
help on how to use DTL, we also want to hear from you.

For help learning DTL, the dtlr tool has help built in by
using the .help` command. You can also You can visit the
docs or look at the
DTL Expression Syntax.
You can also view all the helper function docs here.

If you want to see examples of DTL or try it out, head to the DTL
website and click the 'Try DTL' button to explore
DTL right in your browser.

If you want to see more examples, you can take a look at the Test
Suite where you can
find an example of just about anything DTL can do.

If you have ChatGPT Plus, you can talk to the DTL
Helper to get help with
DTL.

If you want something a bit more real-time and real-person, you can talk with us on the
DTL discord.

And, if you encounter a bug, please don't hesitate to file an
Issue.