How It Went

Part 1: The Remote Shell Daemon

Initially I was not fully aware of the difference between full-blown interactive shell emulation and just sending bash commands and receiving the results on the front-end, so I started with looking for an easy-to-integrate front-end solution that I could attach to the existing Visual Editor (the panel). A quick google search got me to Xterm.js, which looks like the de-facto front-end tool for that. I checked out some of the projects using Xterm.js and quickly realized that what I needed was complete terminal emulation, so I basically had to have a process on each instance doing that job for me as well, ideally working nicely with Xterm.js on its own too.

The first candidate for that was WeTTy, simply because it seemed like I could just run it on my existing Node process in the container. However the API doc (or the README) were really not helping on how to do attach a WeTTy server on an existing Express server, and further-more a simple test proved that it couldn’t be done seamlessly without some minimum level of configuration (you can check it out on this repl.it piece). As a super-lazy developer, this caused me to simple drop WeTTy as a candidate here, with the prospect of not having a separate process for the remote-shell besides the Node process.

I found other options, most prominently GoTTy and ttyd. Both seemed really similar and capable of getting the job done, and both also provide the Shell-Client as well (so no need to wrestle with Xterm.js on client-side). However not only ttyd has instructions on how to install on linux systems, it has a proper Dockerfile alongside one specifically for alpine images, which is the base image for CONNECT-platform containers as well. Hence, it was the obvious winner.

Part 2: Security Layer

So to have a web-based remote-shell, I would be running a ttyd service alongside each instance of CONNECT platform. ttyd would listen on some port and offer the whatever I was looking for. However, the security concerns mentioned above would not be satisfied if I was to just run the ttyd within each container and expose it to the internet. To recap, these are the security criteria I wanted to meet:

The remote-shell should have identical access-level to the Node VM, i.e. something can be done by the shell if and only if it can be done by the NodeVM

Access to the remote-shell is identical to access to the panel, i.e. someone can access the remote-shell if and only if they can access the panel.

The first criteria could easily be solved by ensuring that the NodeVM spawns the ttyd process as a child process:

const { exec } = require(‘child_process’); const run = () => {

let proc = exec('ttyd bash');

process.on('exit', () => proc.kill());

}

the extra-line here is to ensure that when the Node process is dead (for whatever reason), so would be the ttyd process.

I could have alternatively just ensured that the process for the shell is executed by a user (or user group) with identical access level. However, that would mean that anyone making subsequent changes to the access level of the Node process should also be careful to update the access level of the remote-shell, which would increase the chance of error and hence unnoticed security flaws.

As for the second constraint, I decided to seal-off the ttyd server within the container and only give the Node VM access to it. Since the panel has a secure line to the Node VM, the Node VM could then authenticate and authorize requests to access the shell originating from the panel, similar to how it would authenticate and authorize other requests from the panel, and if authorized, proxy the requests to the ttyd process. As with the first constraint, I could keep the two services completely separate and ensure identical authorization mechanism on a higher-level, however again that would mean that any subsequent changes to auth process of one of the services would need to follow corresponding changes to auth process of the other, again, increasing possibility of error.

Part 3: Proxying ttyd

So the security design meant that there should be an authentication mechanism known only to ttyd and the Node process in the first place. ttyd supports basic HTTP authentication, so it would suffice to create a random user-name and password on each execution and feed it to the process:

const run = (credentials) => {

let proc = exec(`ttyd -c ${credentials} bash`);

process.on('exit', () => proc.kill();

}

and in the main guy supposed to run this whole charade:

let credentials = randomToken() + ':' + randomToken();

run(credentials);

Now I needed to track what are the requests that the ttyd client makes to its server, which of those are authenticated. For these requests then I would need to:

Authenticate the incoming request through Panel’s auth process, Modify the proxied request to add the basic authentication HTTP header according to the generated credentials, Proxy the request

As it turned out upon further inspection, the requests in need of authentication are the first original request to get the shell, and another request for a file named /auth_token.js , which seems to be used to establish and authenticate a WebSocket with the ttyd server.

As a lazy developer, this meant that I would need a simple proxy tool working nice with Express also properly capable of handling WebSockets. As it turned out with another quick googling, http-proxy-middleware is just that. So the proxying code would look something like this:

const proxy = require('http-proxy-middleware'); const SHELL_URL = '<local address for ttyd>';

// this will be the url that the shell will be accessible on

// publicly, e.g. if the shell is to be accessed via:

//

// then TTY_PATH would be '/shell'

//

const TTY_PATH = '<some url sub-path>'; //// this will be the url that the shell will be accessible on// publicly, e.g. if the shell is to be accessed via:// https://whatever.connect-platform.com/shell // then TTY_PATH would be '/shell'//const TTY_PATH = ' '; const doproxy = credentials => {

return proxy([TTY_PATH, '/auth_token.js'],

{

target: SHELL_URL, // so we send everything to ttyd,

ws: true, // so we proxy WebSockets,

changeOrigin: true, // so that ttyd is not aware of proxying

pathRewrite: path => {

//

// well, ttyd expects requests to reach it on '/', so

// we should properly trim them.

//

if (path.startsWith(TTY_PATH))

return path.substr(TTY_PATH.length);

else return path;

},

onProxyReq: (proxyReq, req) => {

if (auth(req)) { // check if the request is authenticated,

//

// well we need to set 'Authorization'

// header on requests, with credentials

// in base64 format.

//

proxyReq.setHeader(

'Authorization',

'Basic ' + new Buffer(credentials).toString('base64')

);

}

}

}

);

}

With the main code being modified to look like this:

const main = app => {

let credentials = randomToken() + ':' + randomToken();

run(credentials);

app.use(doproxy(credentials));

}

As for authenticating the requests, since the request to the root path of the shell ( TTY_PATH ) originate from panel itself, and since panel’s authorization is done with verifying the proper access token, I could just inject the token to the URL on panel’s side, and read and verify it here. In other words, the initial request to access the shell would look like this:

https://whatever.connect-platform.com/<TTY_PATH>?token=<the_token>

so it could be easily checked like this:

const auth = req => {

if (req.originalUrl.startsWith(TTY_PATH))

return req.query.token && verify(req.query.token);

return false;

}

Note that the verify() function here can be any form of token verification. On CONNECT-platform, we use JWT (which also has a pretty nice Node package).

However, the other request to /auth_token.js is made by the ttyd client itself (which I suspect is possible to modify, but again, lazy developer here). However, the referrer on that request must still be the original request to TTY_PATH , which means the token must reside in the referrer and can be verified similarly. Which means the auth() function would turn into this:

const { URL } = require('url'); const auth = req => {

if (req.originalUrl.startsWith(TTY_PATH)) {

return req.query.token && verify(req.query.token);

}

else if (req.originalUrl == '/auth_token.js') { const referrer = new URL(req.get('Referer'));

return referrer.searchParams.has('token') &&

verify(referrer.searchParams.get('token')); } return false;

}

Part 4: Configuring OpenResty

So now we have a ttyd running on the container, providing the Shell-Client as well, while also being fully concealed by Node VM and secured (or at least, exactly as much as the Node VM). The task seemed to be mostly done. I made the whole remote-shell (and its access path) configurable, in case that ttyd is not even installed in the environment that the platform is running in (because remember, the core of the platform is a NPM package that can be run anywhere). I also added a bit more tracing of the ttyd process to the JS code, so that if it would detect errors in executing the ttyd process, it would not share the link to the remote-shell with the client.

Subsequently, I added the proper piece of code to the boilerplate project to read configuration of the remote-shell from the environment variables, and added the proper environment variables to the Dockerfile. This is also where the alpine Dockerfile of ttyd came in REAL handy, as I just added the minimal content to the Dockerfile of the platform itself.

Now everything seemed nicely working, so I published the new version of the NPM package, pushed to the boilerplate repo’s master and re-built and pushed the latest version of the Docker image. I asked my colleague to update the Docker image used on the PaaS, and sat back thinking the task was done.

Obviously though, it wasn’t. As it turned out, It seemed like the WebSocket could not conduct a proper handshake. As it turned out, the handshake process is initially a simple HTTP(S) request bearing an Upgrade and a Connection header with values indicating that this request should be “Upgraded” to an open-tunnel for the WebSocket. Apparently, NGINX (and by extension, OpenResty) does not set these headers by default, so you should ask it explicitly to do so, simply by adding the following to its config:

proxy_http_version 1.1;

proxy_set_header Upgrade $http_upgrade;

proxy_set_header Connection "Upgrade";

The http version is just to ensure that the proper version of the protocol is used for establishing a WebSocket connection.