Transporter 0.3.0 Released - Transporter Streamlined

TL;DR: Transporter 0.3.0 is out and it has a whole new way of working which will streamline its use on the mission to make moving data a joy, not a chore.

Transporter 0.3.0 has been tagged and released. This is the most tangible overhaul to how you work with the open-source Transporter to date, making it a more productive and elegant tool for moving your data between data sources, be they files, databases or, now, message queues.

In the process of releasing version 0.2.x of Transporter, it became clear that we really wanted to refine the process of how you configured the Transporter. We'd expected that addressing this would have taken longer, but Compose engineering has worked at a tremendous rate to bring us this new version.

What's new

Here's a brief summary of the changes:

There's a new JavaScript-based DSL which handles everything

The transporter.yaml file is gone

file is gone Defining sources, sinks and transformers has been unified

Namespaces have been simplified and matching is now just a regular expression

New Native Transformers offer fast, JavaScript-free standard functions

A new JavaScript engine, goja, for evaluating JavaScript

A RabbitMQ adaptor joins the existing adaptors

MongoDB's adaptor can now filter collections

The new Native Transformers are implemented in Go and allow messages to have common operations performed on them without the overhead of the JavaScript engine.

omit() - removes top-level fields from a message

- removes top-level fields from a message pick() - selects only particular top-level fields from a message

- selects only particular top-level fields from a message rename() - renames top-level fields

- renames top-level fields skip() - skips messages based on a test of a top-level field against a value

- skips messages based on a test of a top-level field against a value pretty() - pretty prints the message to the log for debugging

There's also "native" transformers to call up JavaScript:

goja() - uses the Goja JavaScript engine

- uses the Goja JavaScript engine otto() - uses the Otto JavaScript engine

Both take a filename as a parameter and use the contained JavaScript to process the message through a transformation function. We will have a detailed look at the new Native Transformers in a new article.

Getting going

You can download ready-built Transporter 0.3.0 binaries from the releases page or you can build your own with the available source.

We have also swept the Transporter issues and if you want to get involved we now have a great list of issues to get started on. We do Transporter development in the open and you'll also be able to track design and development through the Transporter issues, pull requests, commits and other activity.

Migrating to 0.3.0

If you are an existing Transporter user, be prepared for a lot of breaking changes for the better in 0.3.0. The first thing you may notice is that there are fewer commands. We've dropped the eval and list commands to simplify working with Transporter.

To show how the configuration has changed, if we look back to moving Transporter data from database to disk, when we ran transporter init mongodb file , we got a transporter.yaml file and a pipeline.js file.

If we run transporter init mongodb file with 0.3.0, we get just a pipeline.js file. The first stanzas of that file duplicate the intention of the old yaml file, but by creating JavaScript objects:

var source = mongodb({ "uri": "${MONGODB_URI}" // "timeout": "30s", // "tail": false, // "ssl": false, // "cacerts": ["/path/to/cert.pem"], // "wc": 1, // "fsync": false, // "bulk": false, // "collection_filters": "{}" }) var sink = file({ "uri": "stdout://" })

Now, each adapter is defined with var variablename=adaptorname(options) where the options are in the form of a JavaScript object. To get an example of the options for an adaptor, transporter about adaptorname will print out what options are available formatted as a JavaScript object and ready to cut and paste into your code. In our old tutorial, we set our MONGODB_URI environment variable and set the ssl option to true, and set the file to write to a file. With the new configuration that looks like this:

var source = mongodb({ "uri": "${MONGODB_URI}", "ssl": true }) var sink = file({ "uri": "file://dump.json" })

The init function also generated the pipeline script which looked like this:

Source({name:"source", namespace:"test./.*/"}).save({name:"sink", namespace:"test./.*/"})

This needed modifying to select the database to read from, even though it was in the MONGODB_URI, and there were redundant elements in the namespace for the sink.

Now, with 0.3.0, the pipeline looks like this:

t.Source("source", source, "/.*/").Save("sink", sink, "/.*/")

The t. is the "Transporter" variable; all pipelines are rooted with that now. The Source and Save functions take an (optional) name, adapter variable, and (optional) namespace. Notice also, that Save is upper-case now. The namespaces are simplified to just a regular expression. In the Source, the namespace is used to select the collections, in the Save, the namespace selects the messages that flow into the sink.

We don't need to edit this at all; the simplified namespaces mean these namespaces match everything. We can just do:

transporter run

And our Transporter will run the pipeline.js file. That's another change; if no pipeline is specified, the Transporter looks for pipeline.js as a default.

As an example, if we wanted to add the pretty Native Transformer, that would look like this"

t.Source("source", source, "/.*/").Transform(pretty({}).Save("sink", sink, "/.*/")

There's lots more to cover in Transporter 0.3.0 and we'll be doing that in the Wiki and here in Compose Articles. Look out for it and come and join us on Github.com/compose/transporter where the development happens.