Example Usage of nodesh and npx-scripts

A common problem I’ve faced in my career, is being able to share and maintain bash scripts. Many times these scripts are write-once, as it is much easier to iterate on a script, than it is to architect one. The Unix shell is a powerful construct, and the system tools at your disposal are amazing. Each tool has umpteen number of flags and configuration options, and in conjunction with line oriented text are extremely powerful.

Something as seemingly simple as

Using curl to read lines with URLs out of a wikipedia page

could be hundreds of lines of code in other languages. The Unix philosophy states: “Do one thing and do it well.”. This pattern serves us well in that the curl command is only concerned with making http requests, and the grep command is focused on filtering streams of data.

Unfortunately, this power is often coupled with tools that produce hard to maintain scripts. My general alternative in the past has been to use a language like python, as the “Batteries Included” ideology makes portability a breeze. As I spend more time in the node ecosystem, I would prefer to use the libraries and language features that I’m accustomed to.

To that end, my goal is to benefit from the Unix concepts, and integrate them with the node ecosystem. This would produce scripts that are easier to maintain as well as able to utilize all the available npm packages. Additionally, the goal is to leverage this functionality in a type-safe manner providing a high quality authoring experience.

npx as a Script Runner

In addition to interacting with the node ecosystem, a driving goal here is to rely upon npx to provide a truly portable script execution environment. This does imply the machine is connected to the internet, or has all of the dependencies preloaded.

Simple example of npx as a shebang command

Note, the construct relies on using a shebang with support for multiple arguments. This is achieved using the -S flag with the /usr/bin/env command. Usually a shebang command only supports a single argument, but to run npx with a module, we need support for more.

Relying upon npx as the shebang command, allows for some pretty interesting solutions. The two projects I’ve created, rely upon on this construct.

Nodesh

Nodesh is a library with the purpose of providing a shell-like programming experience within node. The library is written in typescript, and so the typing support is fantastic. It is async-generator oriented, meaning it streams iteratively, similar to how shell commands and piping work. The goal here is simplicity of usage as well as authoring. To that end, and due to the nature of execution as stand alone scripts, the library itself exposes itself in the global scope and on the prototypes of existing objects.

upper.js — Converts stream to upper case letters

Above is a simple example of a script, with a global variable $stdin which represents process.stdin as an async generator that produces lines of text. A more complex example could look like:

remote-names.js — Fetches urls out of stdin, and returns their resultant HTML pages as a series of tokesn

This script will extract all urls from process.stdin, fetch them, and extract all the proper words from the resultant output.

Usage of remote-names.js, sort and counting unique tokens in output

The library is broken into the following main categories:

Core — Standard operators (map, flatMap, filter, ..)

— Standard operators (map, flatMap, filter, ..) File — Directory searching and file reading

— Directory searching and file reading Transform — Stream Transformations (reduce, collect, sort, unique, …)

— Stream Transformations (reduce, collect, sort, unique, …) Text — Text oriented operations (columns, replace, match, trim, …)

— Text oriented operations (columns, replace, match, trim, …) Limit — Stream flow operations (First, Last, Skip, Repeat)

— Stream flow operations (First, Last, Skip, Repeat) Exec — Support for command execution (execute)

— Support for command execution (execute) Export — Output support (console, stdout, write to file)

— Output support (console, stdout, write to file) Advanced — More esoteric behavior (parallel support)

npx-scripts

npx-scripts is a VSCode plugin aimed at authoring of scripts with npx shebangs. The primary features it provides are

Type acquisition of scripts with npx shebangs

Support for launching scripts (with performance enhancement)

Snippets for authoring npx shebangs

Type Acquisition

During the editing process, the plugin will attempt to assist with acquiring types. If a file with an npx shebang is opened, or an npx shebang is added to a file, the plugin will check the referenced npm module. If the module contains types, the user will be prompted to add a type reference to the script.

Acquiring typings for @arcsine/nodesh

If a script is opened that already has the typing information in it, the plugin will alert the user to the invalid typings, and offer to fix them.

Loading of a script with invalid typings, user is prompted to reinstall to fix

Launching Scripts

Given the goal of stand alone scripts, the scripts produced should be easily executable. Whether manually, or via the npm debugger, the script execution should be straightforward.

The plugin also provides the ability to execute the script with the referenced module, which will stream the output to the editor. One of the primary benefits here, is that npx will not be invoked, and so installation will be skipped. This provides a speed boost across multiple executions.

Additionally, there is a command to Run Script With Input, which will prompt for a file to use as stdin for the script execution.

Running of a @arcsine/nodesh script within npx-scripts plugin

Snippets

Simply supports inserting a new shebang of the appropriate format. There is also support for inserting a nodesh shebang, as it should be a common use case.

Using the npx-shebang snippet

Wrap Up

To follow up, the goal of producing truly portable and useful scripts is challenging. Each of the projects mentioned aim to solve the problem in some part. Each can be used independently, but also work well together. No solution will ever be perfect, but the hope is that these projects can provide a meaningful boost in productivity.