Sorry SUCKAH, this website works way best in Chrome.

It uses web ANIMATIONS you damn fool.

Those ain't supported in your FOOL browser. ![vb](img/vb.gif) ![lights](img/lights.gif) LNUG talk ![microphone](img/microphone.gif) May 27th 1998 - ~~James Pike~~ Mr Friend # Build systems for the web and node.js / io.js Current javascript state of the art compared to sigh, a functional reactive declarative build automation system. ![waving](img/uk-waving.gif) ![geocities](img/geocities.gif) ![evil-cameron](img/cam.png) ![suit](img/suit.png) [email coolguy82@hotmail.com](http://chilon.net) Press ⇨ to go forward, ⇦ to go back, f for fullscreen.

Wikipedia: "In computing, a pipeline is a set of data processing elements connected in series, where the output of one element is the input of the next one." * How to build code: * Task runner * Asset pipeline * Good for anything that reacts to code changes * Transpile es6/coffee/typescript/scss. * Run tests after everything is built. * Create production version of code (minified etc.) * Talks to your browser via livereload. * Incrementally rebuild. * Reactive development automation system

# History * "All your base are belong to us" - 1998 * "Don't tase me bro" - 2007 * node 0.0.1 - May 2009. * CoffeeScript 0.1 - December 2009. * npm 0.0.1 - January 2010 (node at 0.1). * jake 0.1.5 - August 2010. * npm 1.0.1 - April 2011 (node at 0.4) * grunt 0.1.0 - January 2012. (& "Grumpy Grandma") * gulp 0.0.1 - July 2013, 1.0.0 - September (& "Grumpy Cat"). * traceur 0.0.10 - January 2014. * broccoli (first beta) - Feb 2014.

# Before grunt * Gnu make: Makefile (gruntfile.js, gulpfile.js etc.) * jake * Shell scripts * Custom build scripts written in javascript * Other stuff

Makefile syntax ❯ make make ❯ touch touch ❯ make make : @echo feeding lovely sad cat feed --animal $^ : Nobody INTERFERES with Mr T or Mr T's BINS SUCKAH cat $^ > $@ : @mplayer inspirational-montage.mkv

Hey SUCKAH. Mr T 'aint WEARING no Red HAT SUCKAH. And don't you COME in to MY GARAGE at NIGHT and KICK over MY BINS. ## Jake * Jake could do pretty much everything make could. * Much more powerful/flexible task scheduling than grunt. * Never caught on, maybe due to lack of plugin system?

❯ grunt build ## npm ```javascript // package.json "scripts": { "lint": "jshint src/*.js", "mocha": "mocha test/*.spec.js", "build": "npm run lint && npm run mocha", } ``` ❯ npm run build ## Grunt ```javascript // gruntfile.js module.exports = function(grunt) { grunt.initConfig({ jshint: { files: ['src/*.js'], options: JSON.parse(require('fs').readFileSync('./.jshintrc')) }, mochaTest: { files: ['test/*.spec.js'] }, }) grunt.loadNpmTasks('grunt-contrib-jshint') grunt.loadNpmTasks('grunt-mocha-test') grunt.registerTask('build', [ 'jshint', 'mochaTest' ]) } ```## npm ```javascript // package.json "scripts": { "lint": "jshint src/*.js", "mocha": "mocha test/*.spec.js", "build": "npm run lint && npm run mocha", } ``` Scroll here ⇩⇩⇩

## Can schedule tasks to run after current task ```javascript // i cannot ride the catbus like this!!! grunt.registerTask('ride-catbus', function() { var done = this.async() grunt.task.run(['wait-at-bus-stop', 'give-totoro-umbrella']) console.log('Where are you totoro?') setTimeout(function() { console.log('Totoro... Did I do something wrong???') done() // after this function exits the tasks are run }, 10000) }) ``` Not before or during current task. ~~That would be wrong.~~ ## A task can schedule other tasks in sequence ```javascript // i just need to write another task to ride the catbus grunt.registerTask('get-on-catbus', function() { // ... }) grunt.registerTask('ride-catbus', [ 'wait-at-bus-stop', 'give-totoro-umbrella', 'other-stuff-happens', 'get-on-catbus' ]) ```

* No release for 11 months. * Is a task runner rather than a build system. * It was just used as one. * Because nobody seemed to like jake. * And there was nothing else. * Pipeline = plugins taking turns to compile code in temp directories. * yeoman angular fullstack generator "best practice" gruntfile, 667 lines long:

# Gulp * gulp was released 1.5yrs after grunt in July 2013 * pipelines are node streams * plugins are (mostly) stream transforms, 1 source file in 1 destination file out. * The streams are exposed directly to build files.

```javascript // gulpfile.js var gulp = require('gulp') var traceur = require('gulp-traceur') var sourcemaps = require('gulp-sourcemaps') var concat = require('gulp-concat') var merge = require('merge-stream') var paths = { es6: ['app/**/*.js'], raw: ['raw/*.js'] } gulp.task('client-scripts', function() { merge( gulp.src(paths.es6) .pipe(sourcemaps.init()) .pipe(traceur()), gulp.src(paths.raw) .pipe(sourcemaps.init()) ) .pipe(concat('output.js')) .pipe(sourcemaps.write()) .pipe(gulp.dest('dist/assets')) }) gulp.task('watch', function() { gulp.watch(paths.es6.concat(paths.raw), ['client-scripts']) }) gulp.task('default', ['client-scripts', 'watch']) ```

* ... that naive gulpfile is not good: * No incremental rebuilds. * Invalid source map. * Need to restart gulp on error. * Concatenating source maps doesn't always work. * Streams transforms are built around transforming files 1:1 * n:n/1:n/n:1 transforms more difficult. * Node stream API not so good for this task: * Node streams stop accepting incoming data on first error. * gulp-plumber plugin needed to monkey patch it.

* Need to use three plugins to get incremental rebuilds. * gulp-cached : only pass through changed files. * There is also gulp-changed / gulp-newer. * gulp-remember: to allow cache to work with many:1 plugins. * gulp-order: To maintain file order.

```javascript // gulpfile.js var gulp = require('gulp') var traceur = require('gulp-traceur') var sourcemaps = require('gulp-sourcemaps') var concat = require('gulp-concat') var merge = require('merge-stream') // these are needed for incremental rebuilds var cache = require('gulp-cached') var remember = require('gulp-remember') var order = require('gulp-order') // this is needed to fix bad stream behaviour var plumber = require('gulp-plumber') var paths = { es6: ['app/**/*.js'], raw: ['raw/*.js'] } var allPaths = paths.es6.concat(paths.raw) gulp.task('client-scripts', function() { merge( gulp.src(paths.es6) .pipe(sourcemaps.init()) .pipe(plumber()) .pipe(cache('traceurCode')) .pipe(traceur()), gulp.src(paths.raw) .pipe(sourcemaps.init()) ) .pipe(remember('traceurCode')) .pipe(order(allPaths)) .pipe(concat('output.js')) .pipe(sourcemaps.write()) .pipe(gulp.dest('dist/assets')) }) gulp.task('watch', function() { gulp.watch(allPaths, ['client-scripts']) }) gulp.task('default', ['client-scripts', 'watch']) ```

Mr T recommends you EAT your BROCCOLI SUCKAH.

Mr T also RECOMMENDS you stay AWAY

from Mr T's BINS and Mr T's VAN. # Broccoli * First release May 2013 * Currently in beta * Plugins take one or more trees (filesystem directories). * Output one or more trees (containing built assets). * "The filesystem is the API"

# That example from earlier in broccoli ```javascript var funnel = require('broccoli-funnel') var mergeTrees = require('broccoli-merge-trees') var es6Transpiler = require('broccoli-babel-transpiler') var concat = require('broccoli-concat') var es6Tree = funnel('app', { destDir: 'es6', include: [ /\.js$/ ] }) var es6TranspiledTree = esTranspiler(es6Tree) var rawJsTree = funnel('raw', { destDir: 'raw', include: [ /\.js$/ ] }) var allJsTree = mergeTrees([ es6TranspiledTree, rawJsTree ]) var concatTree = concat(allJsTree, { // ensure raw files concatted after es6 files inputFiles: [ 'es6/*', 'raw/*' ], outputFile: 'dist/assets/output.js' }) module.exports = concatTree ```

# But... * Source maps rarely working. * Not working in previous example. * Many plugins lack transitive application. * Use of filesystem questionable: * Plugins must clean-up. * Infrastructure code more complicated, temp files etc. * Filesystem is slow. * "No parallelism" * "Only a 40% speed gain." * Possible but "broccoli's primitives + helper code encourage deterministic sequential code patterns." * Brocfile.js files can get verbose.

# jspm * Based on EcmaScript 6 module loader. * Gives you everything browserify + bower does and more. * `jspm install bower:lodash` * `jspm install github:bluebird` * `jspm install npm:totoro-love-helper` * Can load AMD/CommonJS and EcmaScript 6 modules. * Can load/transpile dependencies at **run-time**. * ... or can prepare self-contained bundles.

* Have/Get to use ES6 modules + ES6 transpiler in your code: ```javascript import _ from 'lodash' import { promisify } from 'bluebird' ``` * In development mode modules will be transpiled many times: * Each time browser reloads. * During tests (once per browser + reload). * Modules are transpiled one by one: * Delaying page load. * High CPU usage. * Janking main JavaScript thread. * "Bundle" builds slow, not incrementally rebuilt. * Can use jspm + sigh together for this.

# Other build systems * metalsmith * gobble * brunch ## Those without changes in 6+ months. * plumber * cha * grunt

# Problems with most things * You make a change while the tests are running and: * Change is ignored. * Waits until tests have finished then runs them again. * Production source maps. * Need special handling for watching files.

# sigh * Functional Reactive Programming * Like lodash of streams (flatten, reduce etc.) * `n:n` / `1:n` / `1:1` operations: stream payload is array of events. * Can switch between sequential/parallel easily. * Does not use fs except when absolutely necessary. * Provides lovely APIs for plugin writers * Working with processes. * Working with source maps (application, concatenation). * Source maps from minified/transpiled files back to original sources. * Very neat Assetfile.js syntax. * Plugin authors have powerful FRP stream API. * Stream API hidden from users by pipeline compiler.

# That example from earlier with sigh ```javascript // sigh.js var glob, babel, concat // plugins injected into globals module.exports = function(pipelines) { pipeline.build = [ glob('app/**/*.js'), babel(), glob('raw/*.js'), concat('dist/assets/output.js') ] } ``` # More parallelism ```javascript // sigh.js var merge, glob, babel, concat module.exports = function(pipelines) { pipeline.build = [ merge( [ glob('app/**/*.js'), babel() ], glob('raw/*.js') ), concat('dist/assets/output.js') ] } ``` Run once ❯ sigh Watch files, incremental rebuilds ❯ sigh -w

# A more complicated example ```javascript // sigh.js var glob, babel, sass, write, livereload, process module.exports = function(pipelines) { pipelines.client = [ merge( [ glob({ basePath: 'src' }, '*.js', 'app.scss'), sass(), babel(), write({ clobber: '!(jspm_packages)' }, 'build'), ], glob('index.html', 'config.js') ), livereload() ] pipelines.server = [ glob('server.js'), process('node server.js') ] } ``` # With more parallelism and environments ```javascript var glob, babel, sass, merge, write, livereload, process var env, concat, uglify module.exports = function(pipelines) { var globOpts = { basePath: 'src' } pipelines.client = [ merge( [ merge( [ glob(globOpts, 'app.scss'), sass() ], [ glob(glopOpts, '*.js'), babel(), env( [ concat('combined.js'), uglify() ], [ 'production', 'staging' ] ) ] ), write({ clobber: '!(jspm_packages)' }, 'build'), ], glob('index.html', 'config.js') ), livereload() ] pipelines.server = [ glob('server.js'), process('node server.js') ] } ``` Run all pipelines: ❯ sigh Same as: ❯ sigh client server Production build: ❯ sigh -e production

# Connecting pipelines together ```javascript var glob, pipeline, babel, debounce, write, mocha module.exports = function(pipelines) { pipelines['build-sources'] = [ glob({ basePath: 'src' }, '*.js', 'plugin/*.js'), babel({ modules: 'common' }), write('lib') ] pipelines['build-tests'] = [ glob({ basePath: 'src/test' }, '*.js', 'plugin/*.js'), babel({ modules: 'common' }), write('lib/test') ] pipelines.alias.build = ['build-sources', 'build-tests'] pipelines['run-tests'] = [ // pipeline does not force pipelines to run, if user // runs 'sigh build-tests run-tests' then // 'build-source' will not be run pipeline('build-sources', 'build-tests'), debounce(700), // { activate: true } forces mocha pipeline to // run even if it was not selected pipeline({ activate: true }, 'mocha') ] // explicit pipelines only run if selected or activated. pipelines.explicit.mocha = [ mocha({ files: 'lib/test/**/*.spec.js' }) ] } ``` Incremental rebuilds without tests: ❯ sigh -w build Incremental rebuilds with tests: ❯ sigh -w Only run tests: ❯ sigh mocha

# sigh uses multiple processes for extra speed. * Tests: keeps two processes cached and waiting. * Tests killed when new value received while running. * Second process already waiting to avoid startup delay. * Safe-parallelism: * Array of events split up and distributed to processes. * Collects results from all processes. * Forwards all events in one payload, just as if one process did it. * Compiles ES6 files faster than the native transpiler. # Structure * Make bigger pipelines by connecting pipelines together. * Supports gulp plugins.

❯ sigh -p plugin-name

? What is the name of this plugin? plugin name

? Which github username/organisation should own this plugin? username

? What's your github author? Your Name <name@email.net>

? Which of the following apply to your plugin? (Press <space> to select)

❯◉ Maps input files to output files 1:1

◯ Spreads work over multiple CPUs

? Which features would you like to use? (Press <space> to select)

❯◉ CircleCI integration

◯ TravisCI integration

? Which dependencies do you need? (Press <space> to select)

❯◉ bluebird

◉ lodash

◉ sigh-core Github organisation/author and CI remembered for next time. # sigh helps scaffold plugins (with yeoman)Github organisation/author and CI remembered for next time. Then fill in the remaining code and publish ❯ cd plugin-name

❯ sigh -w

# edit code, sigh will recompile and run tests as you do

❯ npm publish

