Creating web sites with Suave How to contribute to F# Snippets

The core of many web sites and web APIs is very simple. Given an HTTP request, produce a HTTP response. In F#, we can represent this as a function with type Request -> Response . To make our server scalable, we should make the function asynchronous to avoid unnecessary blocking of threads. In F#, this can be captured as Request -> Async<Response> . Sounds pretty simple, right? So why are there so many evil frameworks that make simple web programming difficult?

Fortunately, there is a nice F# library called Suave.io that is based exactly on the above idea:

Suave is a simple web development F# library providing a lightweight web server and a set of combinators to manipulate route flow and task composition.

I recently decided to start a new version of the F# Snippets web site and I wanted to keep the implementation functional, simple, cross-platform and easy to contrbute to. I wrote a first prototype of the implementation using Suave and already received a few contributions via pull requests! In this blog post, I'll share a few interesting aspects of the implementation and I'll give you some good pointers where you can learn more about Suave. There is no excuse for not contributing to F# Snippets v2 after reading this blog post!

Getting started with Suave

I recently did a couple of talks about Suave at user groups and conferences and many of them have been recorded. There are also a couple of nice examples online and some good documentation on the official web site. So if you want to learn more about Suave, here are some links for you:

Introducing F# Snippets v2

As already mentioned, I started using Suave for the new version of the F# Snippets web site. The web site is basically a pastebin for F# code snippets. The nice thing is that it uses F# Formatting for formatting the code snippets and generating tool tips. I never released the source code for the old version, because it was simoply too ugly. The new version fixes this!

The source code is on GitHub - to run it locally, you'll need to download sample data as discussed in the README.

The prototype runs on Azure - this is automatically deployed from the master branch in the GitHub project and it runs as Azure Website.

branch in the GitHub project and it runs as Azure Website. And here is a list of remaining issues before it can replace the old version - the project is quite simple, so this is a great place where you can contribute!

The previous version of F# Snippets stored all data in an SQL database. When creating the new one, I was wondering what is the best option given the size of the web site. It turns out that the meta-data about all the snippets is small enough to fit in memory (about 1MB in JSON format) and so the new version is a lot simpler.

It keeps the meta-data in memory. The formatted snippets are stored in local file system (when testing things locally) or in Azure blob storage (when running on Azure) - though you can also use Azure storage during development. When the meta-data change, it is also saved to a JSON file in the blob storage (so that it can be reloaded if the application is shut down).

You can find more details in the project architecture section of the project README document.

Interesting Suave snippets

There is a number of things that make Suave really nice to use. As you can have a look at the materials above to learn everything about it, I want to give you just a few examples based on my experience with F# Snippets.

The first nice thing about Suave is that it is a library rather than a framework. This means that you are in control of starting and running the server. This makes it easy to deploy it to Azure, Heroku or anywhere else. In F# Snippets, we have one entry-point in the app.fsx file. This composes the server from individual components.

Composing server from web parts

The following code snippet shows how the server is composed. As you can see, we have functionality for showing the home page, displaying snippets, inserting new snippets, listing snippets and the RSS feed:

1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19: 20: 21: 22: 23: 24: 25: 26: 27: 28: 29: 30: let app = choose [ // When accessing '/' we display the homepage path "/" > > = Home . showHome // Display snippet (latest, specific version and raw source) pathWithId "/%s" ( fun id -> Snippet . showSnippet id Latest ) pathWithId "/raw/%s" ( fun id -> Snippet . showRawSnippet id Latest ) pathScan "/ %s / %d " ( fun ( id , r ) -> Snippet . showSnippet id ( Revision r )) pathScan "/raw/ %s / %d " ( fun ( id , r ) -> Snippet . showRawSnippet id ( Revision r )) // Insert page, with simple REST API to check snippet for errors path "/pages/insert" > > = Insert . insertSnippet path "/pages/insert/check" > > = Insert . checkSnippet // Listing of snippets by author and by tag path "/authors/" > > = Author . showAll pathScan "/authors/ %s " Author . showSnippets path "/tags/" > > = Tag . showAll pathScan "/tags/ %s " Tag . showSnippets // Display RSS feed (allowing number of different path formats) ( path "/rss/" <|> path "/rss" <|> path "/pages/Rss" <|> path "/pages/Rss/" ) > > = Rss . getRss // Otherwise, try to process the request as a static file // (this handles all the CSS and JS files as well as images) browseStaticFiles ]

The choose combinator takes a list of web parts and composes them. A Suave web part is essentially one of those functions from the introduction - web parts can handle requests and produce response. Here, we are building a single web part that goes through the web parts in the list and uses the first one that can handle an incoming request. The path combinator is used to restrict what requests a web part handles - so for example path "/" >>= Home.showHome means that we should display the home page if the request is for the path / . A very nice function is pathScan - it takes an F# format string and builds a web part that recognizes requests to URL with the specified pattern. We can, for example, say pathScan "/raw/%s/%d" to detect URLs such as /raw/cJ/5 .

Displaying snippets with DotLiquid

The Suave library does not force you to use any specific templating engine and I actually used Suave for some time with just string concatenation or str.Replace . But if you want to use some templating library, it is really easy to add support for it. To see just how easy, look at my pull request adding support for DotLiquid. We're using DotLiquid in F# snippets, so here is how the code looks.

The code sample below shows how we handle request to display a snippet. We get the snippet ID, get information about it from the meta-data and read the file from storage. If everything succeeds, we create a record FormattedSnippet and pass it to the template loaded from snippet.html :

1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19: 20: 21: 22: 23: 24: 25: type FormattedSnippet = { Html : string Details : Data . Snippet Revision : int } let showSnippet id r = let id' = demangleId id let snippetOpt = publicSnippets |> Seq . tryFind ( fun s -> s . ID = id' ) match snippetOpt with | Some snippetInfo -> match Data . loadSnippet id r with | Some snippet -> { Html = snippet Details = Data . snippets |> Seq . find ( fun s -> s . ID = demangleId id ) Revision = match r with | Latest -> snippetInfo . Versions - 1 | Revision r -> r } |> DotLiquid . page < FormattedSnippet > "snippet.html" | None -> invalidSnippetId id | None -> invalidSnippetId id

You can find the full template on GitHub. The value of the record is exposed as model and we can access its properties in the template. For example, the heading is generated by <h1>{{ model.Details.Title }}</h1> .

Checking F# code during insertion

The new F# Snippets web site reports all the errors in your F# code on the fly when you are inserting the snippet. Go to the insert snippet page, type some invalid F#, wait a second and you should see the compiler errors and warnings!

The implementation of this uses a simple JavaScript with timer and it calls the /insert/check API end-point implemented by the server. This then returns a simple JSON with a list of the errors and warning.

This is another elegant piece of F# code that uses Suave composable web parts and the JSON type provider from F# Data to generate the JSON response. Check out the following snippet:

1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19: 20: 21: 22: 23: 24: open FSharp . Data type Errors = JsonProvider < """ [ {"location":[1,1,10,10], "error":true, "message":"sth"} ]""" > let noCache = setHeader "Cache-Control" "no-cache, no-store, must-revalidate" > > = setHeader "Pragma" "no-cache" > > = setHeader "Expires" "0" > > = setMimeType "application/json" let checkSnippet ctx = async { use sr = new StreamReader ( new MemoryStream ( ctx . request . rawForm )) let request = sr . ReadToEnd () let doc = Literate . ParseScriptString ( request , "/temp/Snippet.fsx" , formatAgent ) let json = JsonValue . Array [| for SourceError (( l1 , c1 ),( l2 , c2 ), kind , msg ) in doc . Errors -> Errors . Root ( [| l1 ; c1 ; l2 ; c2 |], ( kind = ErrorKind . Error ), msg ) . JsonValue |] return! ctx |> ( noCache > > = Successful . OK ( json . ToString ()) ) }

There are a few nice things worth mentioning:

The example shows an interesting use of the JSON type provider. We give it a sample JSON (list with one error), but we're not using it to read data but instead to generate response. As you can ee on line 20, we can then use the provided type Errors.Root to easily build a JSON value representing the error or warning.

We need to disable all caching in the HTTP response. To do this, we use composition of web parts. We define noCache which sets all the different HTTP headers required for this (lines 6-9) and then we use it when producing the result on line 24.

The compositional nature of Suave means that you can really easily define reusable components and structure your code in the way that works for you. For F# Snippets, I wanted to make the project easy to contribute to, and so there is a fairly large number of small independent files implementing the different components.

Summary

This blog post had two purposes. First, I wanted to share some of the resources that you might find useful if you want to learn about web development with F# using Suave. There are many more information available on the internet, including blog post from Scott Hanselman and a cool series by Claus Sørensen, so my list is just scratching the surface!

My second secret goal was to convince you to contribute to the new F# Snippets project. Writing the prototype was a lot of fun and I think you'd have fun contributing too. There is also a very large number of features that people asked about (commenting, search, clustering, suggesting tags, etc.), so I think anyone will find something interesting. To start with, there are a few high-priority issues that need to be resolved before we can replace the old version.

namespace System

namespace System.IO

namespace Suave

module Web



from Suave

module Http



from Suave

module Files



from Suave.Http

module Applicatives



from Suave.Http

module Writers



from Suave.Http

val app : Suave.Types.WebPart



Full name: Fssnip-suave.app

val choose : options:Suave.Types.WebPart list -> Suave.Types.WebPart



Full name: Suave.Http.choose

val path : s:string -> Suave.Types.WebPart



Full name: Suave.Http.Applicatives.path

val id : x:'T -> 'T



Full name: Microsoft.FSharp.Core.Operators.id

val pathScan : pf:PrintfFormat<'a,'b,'c,'d,'t> -> h:('t -> Suave.Types.WebPart) -> Suave.Types.WebPart



Full name: Suave.Http.Applicatives.pathScan

val id : string

val r : int

type FormattedSnippet =

{Html: string;

Details: obj;

Revision: int;}



Full name: Fssnip-suave.FormattedSnippet

FormattedSnippet.Html: string

Multiple items

val string : value:'T -> string



Full name: Microsoft.FSharp.Core.Operators.string



--------------------

type string = System.String



Full name: Microsoft.FSharp.Core.string

FormattedSnippet.Details: obj

namespace Microsoft.FSharp.Data

FormattedSnippet.Revision: int

Multiple items

val int : value:'T -> int (requires member op_Explicit)



Full name: Microsoft.FSharp.Core.Operators.int



--------------------

type int = int32



Full name: Microsoft.FSharp.Core.int



--------------------

type int<'Measure> = int



Full name: Microsoft.FSharp.Core.int<_>

val showSnippet : id:'a -> r:'b -> 'c



Full name: Fssnip-suave.showSnippet

val id : 'a

val r : 'b

val id' : obj

val snippetOpt : obj option

module Seq



from Microsoft.FSharp.Collections

val tryFind : predicate:('T -> bool) -> source:seq<'T> -> 'T option



Full name: Microsoft.FSharp.Collections.Seq.tryFind

val s : obj

union case Option.Some: Value: 'T -> Option<'T>

val snippetInfo : obj

val snippet : string

val find : predicate:('T -> bool) -> source:seq<'T> -> 'T



Full name: Microsoft.FSharp.Collections.Seq.find

val Latest : 'b

union case Option.None: Option<'T>

Multiple items

namespace FSharp



--------------------

namespace Microsoft.FSharp

Multiple items

namespace FSharp.Data



--------------------

namespace Microsoft.FSharp.Data

type Errors = JsonProvider<...>



Full name: Fssnip-suave.Errors

type JsonProvider



Full name: FSharp.Data.JsonProvider





<summary>Typed representation of a JSON document</summary>

<param name='Sample'>Location of a JSON sample file or a string containing a sample JSON document</param>

<param name='SampleList'>If true, sample should be a list of individual samples for the inference.</param>

<param name='Culture'>The culture used for parsing numbers and dates.</param>

<param name='ResolutionFolder'>A directory that is used when resolving relative file references (at design time and in hosted execution)</param>

val noCache : (Suave.Types.HttpContext -> Async<Suave.Types.HttpContext option>)



Full name: Fssnip-suave.noCache

val setHeader : key:string -> value:string -> Suave.Types.WebPart



Full name: Suave.Http.Writers.setHeader

val setMimeType : mimeType:string -> Suave.Types.WebPart



Full name: Suave.Http.Writers.setMimeType

val checkSnippet : ctx:Suave.Types.HttpContext -> Async<Suave.Types.HttpContext option>



Full name: Fssnip-suave.checkSnippet

val ctx : Suave.Types.HttpContext

val async : AsyncBuilder



Full name: Microsoft.FSharp.Core.ExtraTopLevelOperators.async

val sr : StreamReader

Multiple items

type StreamReader =

inherit TextReader

new : stream:Stream -> StreamReader + 9 overloads

member BaseStream : Stream

member Close : unit -> unit

member CurrentEncoding : Encoding

member DiscardBufferedData : unit -> unit

member EndOfStream : bool

member Peek : unit -> int

member Read : unit -> int + 1 overload

member ReadLine : unit -> string

member ReadToEnd : unit -> string

...



Full name: System.IO.StreamReader



--------------------

StreamReader(stream: Stream) : unit

StreamReader(path: string) : unit

StreamReader(stream: Stream, detectEncodingFromByteOrderMarks: bool) : unit

StreamReader(stream: Stream, encoding: System.Text.Encoding) : unit

StreamReader(path: string, detectEncodingFromByteOrderMarks: bool) : unit

StreamReader(path: string, encoding: System.Text.Encoding) : unit

StreamReader(stream: Stream, encoding: System.Text.Encoding, detectEncodingFromByteOrderMarks: bool) : unit

StreamReader(path: string, encoding: System.Text.Encoding, detectEncodingFromByteOrderMarks: bool) : unit

StreamReader(stream: Stream, encoding: System.Text.Encoding, detectEncodingFromByteOrderMarks: bool, bufferSize: int) : unit

StreamReader(path: string, encoding: System.Text.Encoding, detectEncodingFromByteOrderMarks: bool, bufferSize: int) : unit

Multiple items

type MemoryStream =

inherit Stream

new : unit -> MemoryStream + 6 overloads

member CanRead : bool

member CanSeek : bool

member CanWrite : bool

member Capacity : int with get, set

member Flush : unit -> unit

member GetBuffer : unit -> byte[]

member Length : int64

member Position : int64 with get, set

member Read : buffer:byte[] * offset:int * count:int -> int

...



Full name: System.IO.MemoryStream



--------------------

MemoryStream() : unit

MemoryStream(capacity: int) : unit

MemoryStream(buffer: byte []) : unit

MemoryStream(buffer: byte [], writable: bool) : unit

MemoryStream(buffer: byte [], index: int, count: int) : unit

MemoryStream(buffer: byte [], index: int, count: int, writable: bool) : unit

MemoryStream(buffer: byte [], index: int, count: int, writable: bool, publiclyVisible: bool) : unit

Suave.Types.HttpContext.request: Suave.Types.HttpRequest

Suave.Types.HttpRequest.rawForm: byte []

val request : string

StreamReader.ReadToEnd() : string

val doc : obj

val json : JsonValue

type JsonValue =

| String of string

| Number of decimal

| Float of float

| Record of properties: (string * JsonValue) []

| Array of elements: JsonValue []

| Boolean of bool

| Null

member Request : uri:string * ?httpMethod:string * ?headers:seq<string * string> -> HttpResponse

member RequestAsync : uri:string * ?httpMethod:string * ?headers:seq<string * string> -> Async<HttpResponse>

override ToString : unit -> string

member ToString : saveOptions:JsonSaveOptions -> string

member WriteTo : w:TextWriter * saveOptions:JsonSaveOptions -> unit

static member AsyncLoad : uri:string * ?cultureInfo:CultureInfo -> Async<JsonValue>

static member private JsonStringEncodeTo : w:TextWriter -> value:string -> unit

static member Load : uri:string * ?cultureInfo:CultureInfo -> JsonValue

static member Load : reader:TextReader * ?cultureInfo:CultureInfo -> JsonValue

static member Load : stream:Stream * ?cultureInfo:CultureInfo -> JsonValue

static member Parse : text:string * ?cultureInfo:CultureInfo -> JsonValue

static member ParseMultiple : text:string * ?cultureInfo:CultureInfo -> seq<JsonValue>

static member ParseSample : text:string * ?cultureInfo:CultureInfo -> JsonValue



Full name: FSharp.Data.JsonValue

union case JsonValue.Array: elements: JsonValue [] -> JsonValue

module Successful



from Suave.Http

val OK : a:string -> Suave.Types.WebPart



Full name: Suave.Http.Successful.OK

override JsonValue.ToString : unit -> string

member JsonValue.ToString : saveOptions:JsonSaveOptions -> string