Creating a tutorial from Git repo

What if author of a programming book could maintain his content and related code in the same place? In this entry I'll share with you my experimental idea for code-heavy F# tutorials.

This post is part of the F# Advent Calendar 2016 initiative - make sure to go check out rest of posts as well. I'd like to thank Sergey Tihon for organising F# Advent. Previous edition of the event gave birth to this blog, encouraging me to write my first post ever.

Motivation

More than a year ago, as part of my master thesis I've written Suave Music Store tutorial. To my delight, this GitBook turned out to be a pretty good starting point for developers wanting to learn Suave for web development. As I'm not probably the best technical writer, I was wondering what could have made this resource that successful. I arrived at a conclusion, that what was special about format of this tutorial when compared to other tutorials available on web was the close connection of source code to content, which guided step-by-step, change-by-change through the process of creating a web app.

What did this mean in practice? Let's have a look at following example from tutorial:

In addition to thorough explanation of every single change, I included (at the end of each main section) a hyperlink to git tag pointing to GitHub commit with state of the codebase at the exact point of tutorial. Thanks to this, the reader could have a look at original sources in case he got stuck during his experiments.

To maintain the tutorial I decided to keep track of two separate repositories: first for the source code of the app, and second for the GitBook tutorial content. I could have probably done the same with a single git repository, however then I'd go for separate branches for source and content so that I could easily make distinction between source and content related changes. Also one of my main goals was to enable the reader go checkout the source repository and browse through the commit history, so I had to keep the history as clean as possible, with no junk commits in between.

As seen in above screenshot, git commit history for the source repository was linear, with few changes between commits so that the reader could easily check it out (or git checkout it ಠ⌣ಠ).

Keeping sync between those two repos wasn't very convenient. While copying pieces of code between source and content or fixing typos were rather kind of usual activities, one of the hardest issue to deal with was amending the source code. To ensure my initial goal of allowing reader to browse through each commit, I had to overwrite remote git history, which meant the (in)famous git push --force , as well as recreating proper git tags, so that the hyperlinks pointed to right commit.

I initially based the tutorial on Suave 0.28.1, which back then was the latest version. When Suave 1.0 got released, I wanted to update the content accordingly. I even got help from the OSS community to do so, but then I got back to the problem of sync with source code git history. What I ended up was a new git branch which I rebased in interactive mode from the root commit of source code repository. This required quite a lot effort, but I managed to retain my original goal.

Now that Suave 2.0 is close to its stable release, I'd like to start works on and publish new version of the tutorial. To help me deal with my authoring problems, I figured out a rather unusual idea, which I'm going to describe in this entry.

Note: The idea is still a work in progress, and I'm not even sure where it goes. It might happen that I decide at certain point that the solution isn't as awesome as I thought it could be and drop it completely. Anyways I thought it can be worth sharing with public. Feedback can really do a great job, so feel free to leave your comments.

The idea

The idea is based on following assumptions:

unit of change in sources, a git commit maps to a single section (or nested subsection) of tutorial,

maps to a single section (or nested subsection) of tutorial, every commit contains only those changes that are described in corresponding (sub)section,

content of the section is stored in commit message , formatted in markdown,

, formatted in markdown, each commit (section) starts with a header, and the nesting level is determined by count of # (which corresponds to markdown header).

Example: "Hello World from Suave" commit

Example: "Hello world from Suave" section

From the above example we can see how "Hello world from Suave" section is a one-to-one mapping of a git commit to tutorial content.

In above we can see a one-line log from the repository showing headers of the sections:

"Introduction" is 1st level section

"Hello world from Suave" is 1st level section

"WebPart" is 1st level section

"Basic routing" is 1st level section "Url parameters" is 2nd level sub section "Query parameters" is 2nd level sub section



... and so on ...

To prototype this idea, I've extracted couple of sections from the tutorial and tried to recreate them on a separate branch, with earlier assumptions in mind.

A preview of tutorial created in this manner can be viewed here. There are a couple of sections there from the original tutorial (still based on Suave v1.0). I've played around the bits and tried to utilize the fact that I have a fully-fledged Git repository underneath the tutorial.

Literal Programming

Presented idea might seem related to the concept of Literal Programming. In F# world, Literal Programming is often associated with FSharp.Formatting library maintained by Tomas Petricek. Indeed, the library can be embraced for providing already well-known syntax highlighting which is better and looks nicer than the standard offered by GitBook website:

Standard GitBook syntax highlighting for F#

FSharp.Formatting syntax highlighting for F#

FSharp.Formatting also includes a capability to display tooltips for F# code snippets, however I haven't yet managed to make them work properly in the preview.

What's more, code snippets do not have to be manually kept in sync between tutorial content and the source repositories. Instead, they can be just marked with a placeholder in a git commit:

And later expanded with corresponding lines:

Also, basing on the fact that each section corresponds to a git commit, a link to this specific GitHub commit can be generated at the end of each section. Under the link, there is also a list of files modified within this commit:

Link to specific commit on GitHub

The commit shown on GitHub

Implementation

Now I'd like to share how the GitBook content is actually generated. For prototyping the idea I've written a FAKE script, which I run from the command line with a few parameters, the main parameter being path to the prepared repository. Eventually, if the prototype turns out successful, the script could be rearranged to a standalone executable to be run from the directory of target repository.

Moving now to the code itself:

1: 2: 3: 4: let commits = Git . CommandHelper . getGitResult repo ( "log --reverse --pretty=%H " + branch )

To fetch list of commits' hashes, I used GitHelper from the FAKE library and pointed to a chosen branch.

1: 2: 3: 4: 5: 6: 7: let msg = Git . CommandHelper . getGitResult repo ( "log --format=%B -n 1 " + commit ) |> Seq . toList let firstLine = msg |> Seq . item 0 let level = max 0 ( firstLine . LastIndexOf ( "#" )) let title = firstLine . Substring ( level + 1 ) . Trim ()

Then for each commit, I extracted the title (first line of message), as well as rest of the message.

1: git config core.commentChar ";"

By default Git uses # character for marking a comment line within a git commit, so I've changed it to semicolon, so that I could use # for markdown headers.

1: 2: 3: 4: msg |> insertSnippets commit |> insertGithubCommit commit |> insertGitDiff commit

The rest of commit message gets enriched with a couple of helper functions.

1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19: 20: 21: 22: 23: let fileContentsAt commit file = Git . CommandHelper . getGitResult repo ( sprintf "show %s:%s" commit file ) let insertSnippet ( commit : string ) ( line : string ) = if line . StartsWith "==> " then let contents , snipId = match line with | Regex "^==> ([\w\.]+):(\d+)-(\d+)$" [ file ; Int32 lstart ; Int32 lend ] -> fileContentsAt commit file |> Seq . skip ( lstart - 1 ) |> Seq . take ( lend - lstart + 1 ) |> Seq . toList , line . Substring ( "==> " . Length ) | Regex "^==> ([\w\.]+)$" [ file ] -> fileContentsAt commit file |> Seq . toList , line . Substring ( "==> " . Length ) | _ -> failwithf "invalid format '%s'" line // do magic with FSharp.Formatting and return formatted string else line let insertSnippets commit code = List . map ( insertSnippet commit ) code

To insert code snippets, I replaced all occurrences of lines starting with ==> to the corresponding code in given file.

1: 2: 3: 4: 5: 6: 7: 8: 9: let insertGithubCommit commit code = sprintf "GitHub commit: [ %s ](https://github.com/ %s / %s /commit/ %s )" commit githubAccount githubRepo commit |> List . singleton |> List . append [ "" ; "---" ; "" ] |> List . append code

Given the commit hash was given, it was very easy to add a link to GitHub with that commit.

1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19: 20: 21: 22: 23: 24: 25: 26: 27: 28: 29: let numStat line = let split = function | Regex "^(\w)\s+(.*)$" [ status ; name ] -> status , name | _ -> failwithf "split cannot parse: %s" line let human = function | "A" -> "added" | "D" -> "deleted" | "M" -> "modified" | x -> failwithf "human: %s " x let ( stat , name ) = split line sprintf "* %s ( %s )" name ( human stat ) let insertGitDiff commit code = let filesChanged = Git . CommandHelper . getGitResult repo ( sprintf "diff %s^..%s --name-status" commit commit ) |> Seq . toList if filesChanged = List . empty then code else filesChanged |> List . map numStat |> List . append [ "" ; "Files changed:" ; "" ] |> List . append code

And finally insert a list of modified source files within a single commit.

The code is available on GitHub. For lack of a better name I've named it experimental-gitbook.

Downsides

While the whole idea looks good, there are a few issues which make less promising:

Any amends, be it source code or tutorial content require rebasing Git history and rewriting it on remote (which is not recommended in general),

First bullet in turn means that there would be no easy way to contribute to the repository, as any new commit would imply a new section,

There's no support for multi language book (commit message is currently only in English), which is really a shame taking into account fact that the tutorial has already been translated to Japanese

Summary

In this entry I've shared with you my pre-release idea for maintaining GitBook based tutorials. Plans for nearest future are to polish the script and then use it to prepare tutorial for Suave v2.0. However, as stated above, I'm not really sure where this approach leads me to. What I'm hoping is that the idea could help me maintain the tutorial, as well as creating awesome tutorials for other F# libraries / components.

Not all the details have been shared, and I apologize for a bit of speeding (I bet you can point out a lot of quirks in above text), but I really wanted to deliver this post on time for F# Advent :-)

Till next time!

val commits : obj



Full name: creatingatutorialfromgitrepo.commits

val msg : obj list



Full name: creatingatutorialfromgitrepo.msg

module Seq



from Microsoft.FSharp.Collections

val toList : source:seq<'T> -> 'T list



Full name: Microsoft.FSharp.Collections.Seq.toList

val firstLine : obj



Full name: creatingatutorialfromgitrepo.firstLine

val item : index:int -> source:seq<'T> -> 'T



Full name: Microsoft.FSharp.Collections.Seq.item

val level : int



Full name: creatingatutorialfromgitrepo.level

val max : e1:'T -> e2:'T -> 'T (requires comparison)



Full name: Microsoft.FSharp.Core.Operators.max

val title : obj



Full name: creatingatutorialfromgitrepo.title

val fileContentsAt : commit:'a -> file:'b -> 'c



Full name: creatingatutorialfromgitrepo.fileContentsAt

val commit : 'a

val file : 'b

val sprintf : format:Printf.StringFormat<'T> -> 'T



Full name: Microsoft.FSharp.Core.ExtraTopLevelOperators.sprintf

val insertSnippet : commit:string -> line:string -> string



Full name: creatingatutorialfromgitrepo.insertSnippet

val commit : string

Multiple items

val string : value:'T -> string



Full name: Microsoft.FSharp.Core.Operators.string



--------------------

type string = System.String



Full name: Microsoft.FSharp.Core.string

val line : string

System.String.StartsWith(value: char) : bool

System.String.StartsWith(value: string) : bool

System.String.StartsWith(value: string, comparisonType: System.StringComparison) : bool

System.String.StartsWith(value: string, ignoreCase: bool, culture: System.Globalization.CultureInfo) : bool

val contents : obj

val snipId : obj

val skip : count:int -> source:seq<'T> -> seq<'T>



Full name: Microsoft.FSharp.Collections.Seq.skip

val take : count:int -> source:seq<'T> -> seq<'T>



Full name: Microsoft.FSharp.Collections.Seq.take

System.String.Substring(startIndex: int) : string

System.String.Substring(startIndex: int, length: int) : string

val failwithf : format:Printf.StringFormat<'T,'Result> -> 'T



Full name: Microsoft.FSharp.Core.ExtraTopLevelOperators.failwithf

val insertSnippets : commit:string -> code:string list -> string list



Full name: creatingatutorialfromgitrepo.insertSnippets

val code : string list

Multiple items

module List



from Microsoft.FSharp.Collections



--------------------

type List<'T> =

| ( [] )

| ( :: ) of Head: 'T * Tail: 'T list

interface IReadOnlyCollection<'T>

interface IEnumerable

interface IEnumerable<'T>

member GetSlice : startIndex:int option * endIndex:int option -> 'T list

member Head : 'T

member IsEmpty : bool

member Item : index:int -> 'T with get

member Length : int

member Tail : 'T list

static member Cons : head:'T * tail:'T list -> 'T list

static member Empty : 'T list



Full name: Microsoft.FSharp.Collections.List<_>

val map : mapping:('T -> 'U) -> list:'T list -> 'U list



Full name: Microsoft.FSharp.Collections.List.map

val insertGithubCommit : commit:string -> code:string list -> string list



Full name: creatingatutorialfromgitrepo.insertGithubCommit

val singleton : value:'T -> 'T list



Full name: Microsoft.FSharp.Collections.List.singleton

val append : list1:'T list -> list2:'T list -> 'T list



Full name: Microsoft.FSharp.Collections.List.append

val numStat : line:'a -> string



Full name: creatingatutorialfromgitrepo.numStat

val line : 'a

val split : ('a -> string * string)

val human : (string -> string)

val x : string

val stat : string

val name : string

val insertGitDiff : commit:'a -> code:string list -> string list



Full name: creatingatutorialfromgitrepo.insertGitDiff

val filesChanged : obj list

val empty<'T> : 'T list



Full name: Microsoft.FSharp.Collections.List.empty

Please enable JavaScript to view the comments powered by Disqus.