This blog covers a zio-based fetch design for scala.js, react web applications. It heavily uses the zio “environment” feature to help motivate a point of view on the modules pattern in effect systems. At the end, I summarize some thoughts on ergonomics and complexity.

This blog assumes a basic knowledge of zio and react. scala.js is just like scala so you mostly already know scala.js.

This blog is a follow-on to previous blogs: https://appddeevvmeanderings.blogspot.com/2019/10/scalajs-react-fetcher-hook_48.html and https://appddeevvmeanderings.blogspot.com/2019/11/scalajs-and-react-suspense.html.

I have been keeping an eye on and culling ideas from typscript/javascript fetching libraries that address query management concerns. The formulation below incorporates some of those concerns but does not try to comprehensively duplicate well established javascript libraries. I only care about building UIs because I have to create experimental UIs to as part of ML algorithm and solution development.

I am going to cheat a bit–here’s a quick reminder on basic zio types:

ZIO[R,E,A] is the general effect type. R is the environment. E is the error type. A is the value produced.

is the general effect type. R is the environment. E is the error type. A is the value produced. RIO[R,A] is a ZIO with E = Throwable.

is a with E = Throwable. IO[E,A] assumes Any as the environment.

assumes Any as the environment. UIO[A] assumes Any as the environment and Nothing as the Error type–it cannot fail.

assumes Any as the environment and Nothing as the Error type–it cannot fail. A E=Unit might loook strange, but that means an error can occur with a value of () meaning an error occurred but there are no details on the error and it should be interpreted in the context of A being produced.

Things we need to do

In the browser, we need to:

Run asynchronous queries

Link dependent queries together so if one query is re-run, another is run in sequence or parallel.

Support react rendering including the new Suspense capabilities.

Make it easy to run various processing strategies such as caching, retry and refreshing.

If we want an ergonomic and performant API, we need to track the state of running and store the values from completed effects. We will also need to cache query parameters, if any, so that the query engine can re-run a query when required. The query parameters must be updated independent of the effect itself.

As a reminder, a Reader monad allows you to access an environment to create a value. In scala, this is often expressed as A => B but for our purposes B will be an effect. So, we have something like A => ZIO[Env, Throwable, B] or equivalently A => RIO[Env,B] . You can even introduce the idea of a Kleisli. None of these category theory labels are important to know though but in reality that’s all we are doing in this blog.

Top-Level Design

Lets focus on the query part first. A query is usually something like:

// fetch(...) returns an effect,a js.Promise or a zio.Task, etc. def query(id: Int) = fetch(id).flatMap(...process...) // if query returns an effect, still need to "run" it rts.unsafeRunAsync(query(10))(exit => ...)

where we need to provide some query parameters, such as id , to a function that calls the remote system.

If we are working with a Reader monad, we might suspect that we could get the id from the “environment”–the query args are part of an enviromnent.

There are really two environments.

The environment needed to provide the engine capabilities such as formulating a query.

The enviromnent specific to a query containing the query parameters. The query parameter environment can be thought of a subtype of the overall environment.

It is common to give a query a “key” so we can access the results independently of the details of running an effect that produces the results. Lets assume that a query key is a string–something human readable or a hash. The key may be dependent on the query parameters so we can cannot always a-priori know the key without knowing the input parameters. Let’s define:

type AppEnv = Env trait QueryArgs [ P ] { def qargs : P } type QueryAppEnv [ P ] = AppEnv with QueryArgs [ P ] type QueryKey = String type QueryEffect [ T ] = RIO [ AppEnv , T ] type Query [ T ] = ( QueryKey , QueryEffect [ T ] ) type QueryDescriptor [ P , T ] = RIO [ QueryAppEnv [ P ] , Query [ T ] ]

A QueryDescriptor contains all of the logic we need to create an effect that when run, returns our data. We assume that the query parameters will be read from an environment. Hence. we do not need QueryDescriptor to be a function. A QueryDescriptor is a value. Since its a value, in order to allow query parameters to be dynamic, we had to use a “value” that when “run” knows how to access query parameters and create a data-fetching effect. Sometimes, when programming with values, we need this extra level of indirection.

It seems like a rather complicated formulation, but in order to convert the simple function from the start of this section to a “value” that can be run any time, you must provide a way to “bundle” the query parameter. Since we are using zio, we need to formulate a zio environment to hold the query parameters. Creating the enviroment is the “bundling” action.

The query effect itself must be run and the results cached until they are used. Results are uniquely identifed by their key. If the key changes, say the id value for a customer in a view changes and the key is based on the id , then the result should be stored under a key that incorporates the id , e.g., key="/person/1 . Data could change in the remote repository so we may need to run the query again and re-cache the latest results.

There are alot of type aliases in the above list but let’s see how me might create a QueryDescriptor:

val qkey = "neverchanging" val myquery : QueryDescriptor [ Unit , String ] = for { env <- ZIO . environment [ QueryAppEnv [ Unit ] ] qe = RIO . access [ AppEnv ] ( _ => "return value" ) } yield ( qkey , effect )

This query has a constant key and always returns a string. There are no query parameters. It does not access a remote system since the return value is a strict string “return value.”

With zio, type inference is good, so you do not need to declare the type of myquery explicitly. You may notice that qe = ... is not qe <- ... . In a for-comprehension, if I had used <- I would have peeled away the effect and qe would be the result value. Since I need to return an effect the = ensures that a the effect is not peeled away. In the case of this for-comprehension, I only peel out the environment so the for is a bit overkill but still convenient. Instead of for { ... } I could have also used ZIO.access[QueryAppEnv[Unit]]{ env => ... } and achieved the same thing.

We could shrink the code a bit more:

val myquery = for { env <- ZIO . environment [ QueryAppEnv [ Unit ] ] qe = RIO . access [ AppEnv ] ( _ => "return value" ) } yield ( "neverchanging" , effect )

If I had parameters, say a string id parameter, I could:

val myquery = for { env <- ZIO . environment [ QueryAppEnv [ String ] ] qk = s "neverchanging/${env.quargs}" qe = RIO . access [ AppEnv ] ( _ => s "return value: ${env.qargs}" ) } yield ( qk , qe )

Typically, qe = RIO.access... would really be qe = myFetchBasedClient(...) where myFetchBasedClient returns a zio effect based on the browser’s fetch library. If you don’t want to hard code “fetch” you would create a RemoteSystem service that exposes a client member that can create a “get” effect:

val myquery = for { env <- ZIO . environment [ QueryAppEnv [ String ] ] qe = env . client . get ( env . qargs ) } yield ( "neverchanging" , effect )

You have lots of choices on how to create your services and you can make the environment customized to your needs. You also need to choose the complexity of your services. I could add a cache service and within the myquery definition manage the cache directly. Or, cache management could be built into the “client.” Or, it could be built into a react hook. You can compose your services and your query in a wide variety of ways.

Since myquery is just a value, and if you knew the client and the environment, you could also define:

def get ( key : String , qstring : String => String ) = for { env <- ZIO . environment [ QueryAppEnv [ String ] ] qe = env . client . get ( qstring ( env . qargs ) ) } yield ( key , effect )

Then call val people = makeQuery("people_fetch", _ => "/people") to create a query in your component or anywhere in your code since it is a value and you provide the parameters to the effect when you want to run it.

In this blog, I’ll assume we want to type out the details each time versus hiding behind simplified functions like the last get definition. Just recognize that once the infrastructure is worked out, the API is fairly simple.

Browser is Single Threaded

Since a browser is single threaded, you do not need to worry about concurrent access to shared data structures like you would on the JVM. The toplevel environment you might make, AppEnv , can be created and accessed anywhere in your program as a global variable with no need to provide concurrent access mechanisms.

Literally, you can define

object Toplevel { val env = new Env { . . . } }

You can use this env anywhere in your program. If you have services in your environment that return effects such as a js.Promise, you will need to eventually wrap the computation in an effect. While some services, such as a cache, can be written mostly effect free in the browser, other services may explicitly use effects.

You may also want to scope caches and other services to specific parts of your application UI tree. You may decide, for example, that a specific type of cache should only be used for a part of your tree.

Frameworks and libraries like relay, useSWR and react-query assume a single, global cache created as a javascript module. They are providing module level shared data structures. In most cases that works fine, but with a Reader monad formulation, we have the ability to use either a global instance or a scoped set of services. With a services/Reader formulation, you can seemlessly swap in different implementations or allow users to swap in theirs based on application requirements. A Reader monad gives you benefits similar to those found in dependency injection (DI) libraries.

The net is that while we will use a global zio “runtime service” for running a query effect, we will formulate all of the queries to exclusively use the local environment provided by the Reader monad. This helps make the functions more pure so that we know what data they are using inside–that’s an important FP concept. Like many js libraries, we could skip using the Reader’s environment and just rely on “module” scope or global variables. We will not code it that way as we like to have local reasoning so other than using a global variable store the environment and provide it to the react tree, we won’t use any other global variables.

Environment & Service Definition

If we look at other js libraries for query management, there is a cache for query results, a service for storing the query, a system for updating parameters to the query and some other services for setting timeouts and retry. We can organize those capabilities using only a few services, including a service to formulate the HTTP requests. We would normally organize these services using scala objects, which are like js modules, regardless of zio.

We might want to use dependency injection libraries but that might be overkill for a web appliction or undesirable. Defining an environment as a set of services is how we might normally structure an application. It just so happens, that zio also uses an “environment” as the input parameter for its “Reader” monad formulation. We can use a single environment for both uses. A typical list of services is below:

RemoteSystem: System for creating HTTP calls using the browser’s fetch library. We could also create a separate AuthService or have auth be part of RemoteSystem directly.

RuntimeSystem: System for running effects. In this case zio. Zio has a Runtime “service” that we will use more or less directly. We are not trying to hide zio from the application so we expose it directly.

“service” that we will use more or less directly. We are not trying to hide zio from the application so we expose it directly. Clock: Standard zio Clock service.

CacheSystem: Cache results and other content.

DataManagement: Using the CacheSystem, Clock, and potentially other services, allow caching of HTTP results and provide other services such as specifying retries and recovery from errors.

QueryArgs: Service to provide query args to a query descriptor. This service is created and added to the standard environment per query.

ConfigSystem: Statically compiled configuration information.

Environment

To keep it simple, we will have a base environment and an environment that can have query parameters. We to easily derive a query parameter environment from the base environment. We could choose to hide many of the services from the base environment, but in order to keep it simple, the query enviromnent is a subtype of the base enviromnent.

Some services may have “live” members because they can be created independent of any effect or independently of other dependencies. You see Live services in many of the services that zio provides, for example, zio.clock.Clock.Live . You will probably have Live services in the services you define for the same reason.

sealed abstract trait BaseEnv extends CacheSystem with ConfigSystem with RemoteSystem with DataManagement with zio . clock . Clock case class Env ( cache : CacheSystem . Service , remote : RemoteSystem . Service , rts : RuntimeSystem . Service ) extends BaseEnv with ConfigySystem . Live with zio . clock . Clock . Live with DataManagement . Live

Then our query enviromnent is just:

trait QueryArgs [ T ] { def qargs : T } object QueryArgs { def qargs [ T ] = ZIO . access [ QueryArgs [ T ] ] ( _ . qargs ) object LiveUnit extends QueryArgs [ Unit ] { val qargs = ( ) } } case class QueryEnv [ P ] { cache : CacheSystem . Service , remote : RemoteSystem . Service , rts : RuntimeSystem . Service , qargs : QueryArgs [ P ] } extends BaseEnv with ConfigSystem . Live with zio . clock . Clock . Live { def from [ P ] ( env : BaseEnv , p : P ) = { . . . copy relevant members . . . } }

Remote System

Let’s define the RemoteSystem that understands how to fetch data. We need a “Client” that understands HTTP and browser fetch, but also provides standard response management and error recovery. Personally, I have found that if your emphasis is on error management and recovery in an chaotic environment like the web/browser, the use of a particular HTTP client library is less important than the choice of the effect system. We will not show all of the concrete client implementation since it is too long for a blog.

trait RemoteSystem [ E ] { def remote : RemoteSystem . Service [ E ] } object RemoteSystem { trait Service [ E ] { val client : ZioClient [ E ] val builder : ZioClient [ E ] #Builder } def serviceFromClient [ E ] ( c : ZioClient [ E ] , b : ZioClient [ E ] #Builder ) = new Service [ E ] { val client = c val builder = b } def client [ E ] = ZIO . access [ RemoteSystem [ E ] ] ( _ . remote . client ) def builder [ E ] = ZIO . access [ RemoteSystem [ E ] ] ( _ . remote . builder ) }

I separated out the actual client and builder instances as the client really acts as a module that incorporates some dependent types and the builder is where the actual HTTP “verb” commands are to build a “request.” We can define a simple zio based client that uses the browser fetch method quite easily. Anything dependent on the client’s E parameter goes into the client class.

trait ResponseAs [ T ] { self => def apply ( r : Response ) : zio . Task [ T ] def andThen [ U ] ( next : ResponseAs [ U ] ) = ResponseAs . instance [ U ] ( r => self ( r ) * > next ( r ) ) def map [ U ] ( f : T => U ) = ResponseAs . instance [ U ] ( self ( _ ) . map ( f ) ) def flatMap [ U ] ( f : T => zio . Task [ U ] ) = ResponseAs . instance [ U ] ( self ( _ ) . flatMap ( f ) ) } object ResponseAs { def instance [ T ] ( f : Response => zio . Task [ T ] ) = new ResponseAs [ T ] { def apply ( r : Response ) = f ( r ) } } package client { val convertJSError : PartialFunction [ Throwable , Throwable ] = _ match { case scala . util . control . NonFatal ( t : js . JavaScriptException ) => val x = t . exception . asInstanceOf [ js . Error ] TransportFailure ( s "JS exception: ${x.toString}" , Option ( x ) ) . initCause ( t ) } def process ( filter : Option [ String ] ) : String = { filter . filterNot ( _ . isEmpty ) . map ( "?q=" + js . URIUtils . encodeURIComponent ( _ ) ) . getOrElse ( "" ) } def jsPromiseToZIO [ T ] ( p : js . Thenable [ T ] ) = Task . effectAsync [ T ] { cb => p . `then` [ Unit ] ( { ( t : T ) => cb ( Task . succeed ( t ) ) } , js . defined { ( e : scala . Any ) => cb ( Task . fail ( wrapJavaScriptException ( e ) ) ) } ) } def asJSON [ T < : js . Any ] = new ResponseAs [ T ] { def apply ( r : Response ) = ( r . json ( ) pipe jsPromiseToZIO ) . map ( _ . asInstanceOf [ T ] ) } def mapJSErrorToTransportFailureM [ T ] : PartialFunction [ Throwable , Task [ T ] ] = _ match { case scala . util . control . NonFatal ( t : js . JavaScriptException ) => val x = t . exception . asInstanceOf [ js . Error ] Task . fail ( client . TransportFailure ( s "JS error ${x.name}" , Option ( x ) ) . initCause ( t ) ) case scala . util . control . NonFatal ( x : Throwable ) => Task . fail ( client . TransportFailure ( s "${x.getMessage}" , None ) . initCause ( x ) ) } } trait ZioClient [ E ] { protected def extractError ( r : Response ) : Task [ Option [ E ] ] case class UnexpectedStatus ( status : Status , detail : Option [ E ] = None ) extends ClientException ( s "Unexpected status: $status" ) val checkOk = ResponseAs . instance [ Response ] { r => if ( r . ok || r . status == 304 ) Task . succeed ( r ) else extractError ( r ) . flatMap ( detailopt => Task . fail ( UnexpectedStatus ( Status . lookup ( r . status ) , detailopt ) ) ) . mapError ( UnexpectedStatus ( Status . lookup ( r . status ) ) . initCause ( _ ) ) } class Builder ( base : String ) { def getHeaders ( headers : Seq [ ( String , String ) ] = Nil ) = new Headers ( js . Array ( js . Array ( "Accept" , "application/json" ) , ++ headers . map ( p => js . Array ( p . _1 , p . _2 ) ) ) def mkUrl ( v : String ) = if ( v . startsWith ( "http" ) ) v else s "${base}$v" def get [ T < : js . Any ] ( url : String , convert : ResponseAs [ T ] , options : RequestInit = RequestInit ( ) ) = { Task . effectSuspendTotal ( ( Fetch . fetch ( mkUrl ( url ) , options ) pipe jsPromiseToZIO [ Response ] ) . flatMap ( ( checkOk andThen convert ) ( _ ) ) . catchSome ( mapJSErrorToTransportFailureM ) ) } } }

The client was parameterized with the error object type since adaptability to the request format is handled in the get but the error object returned from a REST server is often common across all requests. E is not important to the zio “module/environment” conversation as the E represents the error from the server assuming the effect completes correctly so the zio error channel (zio’s E ) can remain a Throwable for convenience. We did not show all the definitions associated with the Client trait or the customization for a specific E type. You could define E as:

trait HTTPError extends js . Object { val message : String val code : Int }

I have not shown alot of details but you can build out a HTTP client like above quite quickly and I typically do that for most web apps. Probably the most important thing to note is that some web services return non-200 status to indicate the result of some “verbs” such as DELETE or POST. Typically, checkOk should be left to each caller instead of buried in the client so that error handling from the remote call can be distinguished from error handing due to communications failure. That’s where ResponseAs combinators (not shown) come in handy. Often HTTP libraries make too many assumptions about response status codes and their semantics making the use of those libraries much more confusing than it should be. A future blog will incorporate the client’s E into the zio E to show how that works.

CacheSystem

We showed the cache system in a previous blog, https://appddeevvmeanderings.blogspot.com/2019/10/scalajs-react-fetcher-hook_48.html and we repeat it below a bit more succinctly. The cache system is a general cache and is not focused only on caching results from remote fetches.

sealed trait Dependency [ + T ] extends Product with Serializable case class Available [ + T ] ( data : T , inflight : Boolean , cache : Boolean ) extends Dependency [ T ] case class Error ( t : Throwable ) extends Dependency [ Nothing ] case object InProgress extends Dependency [ Nothing ] case object NotRequested extends Dependency [ Nothing ] trait CacheSystem { def cache : CacheSystem . Service } object CacheSystem { trait Service { def clear ( key : String ) : Unit def put ( key : String , item : scala . Any ) : Unit def get [ A ] ( key : String ) : Option [ A ] def reset ( ) : Unit def update [ A ] ( key : String , f : A => A ) : Option [ A ] def updateOrPut [ A ] ( key : String , orElse : A ) ( f : A => A ) : Unit } case class DefaultCache ( cache : LRUCache ) extends Service { def clear ( key : String ) = cache . clear ( ) def put ( key : String , item : scala . Any ) = cache . set ( key , item ) def get [ A ] ( key : String ) = cache . get ( key ) . asInstanceOf [ js . UndefOr [ A ] ] . toOption def reset ( ) = cache . clear ( ) def update [ A ] ( key : String , f : A => A ) = get [ A ] ( key ) . fold ( Option . empty [ A ] ) { a => val n = f ( a ) ; put ( key , n ) ; Option ( n ) } def updateOrPut [ A ] ( key : String , orElse : A ) ( f : A => A ) = get [ A ] ( key ) . fold ( put ( key , orElse ) ) ( _ => update ( key , f ) ) } def clear ( key : String ) = ZIO . access [ CacheSystem ] ( _ . cache . clear ( key ) ) def put ( key : String , item : scala . Any ) = ZIO . access [ CacheSystem ] ( _ . cache . put ( key , item ) ) def get [ A ] ( key : String ) = ZIO . access [ CacheSystem ] ( _ . cache . get [ A ] ( key ) ) def reset ( ) = ZIO . access [ CacheSystem ] ( _ . cache . reset ( ) ) def update [ A ] ( key : String , f : A => A ) = ZIO . access [ CacheSystem ] ( _ . cache . update [ A ] ( key , f ) ) def updateOrPut [ A ] ( key : String , orElse : A ) ( f : A => A ) = ZIO . access [ CacheSystem ] ( _ . cache . updateOrPut [ A ] ( key , orElse ) ( f ) ) def make ( max_items : Int = 100 , max_age : Int = 1000 * 60 * 30 ) = new CacheSystem { val cache = DefaultCache ( new LRUCache ( new LRUOptions { max = max_items ; maxAge = max_age } ) ) } lazy val Live = make ( ) } trait LRUOptions extends js . Object { var max : js . UndefOr [ Int ] = js . undefined var maxAge : js . UndefOr [ Int ] = js . undefined } @js . native @JSImport ( "lru-cache" , JSImport . Default ) class LRUCache ( options : LRUOptions ) extends js . Object { def set ( key : String , item : scala . Any ) : Unit = js . native def get [ T ] ( key : String ) : js . UndefOr [ T ] = js . native def peek [ T ] ( key : String ) : js . UndefOr [ T ] = js . native def del ( key : String ) : Unit = js . native def clear ( ) : Unit = js . native def has ( key : String ) : Boolean = js . native def keys ( ) : js . Array [ String ] = js . native def values ( ) : js . Array [ scala . Any ] = js . native def length : Int = js . native def itemCount : Int = js . native def prune ( ) : Unit = js . native }

Data Management

The data management services provides a “layer” in the overall cache system for caching results from “requests”. You may notice that we did not define the dependency on the CacheSystm using a def parameter in DataManagement trait directly. Instead, we express the CacheSystem dependency as a dependency on the environment inside the service layer– DataManagement with CacheSystem . You may also notice that the CacheSystem dependency is concretely consumed in the actual “Live” implementation where we use the Requests object (shown below) that returns an effect that has a CacheSystem dependency. The way these dependencies are created is specific to my implementation but it is not hard to believe that the data management layer with a method called cache probably needs a cache sub-system to run correctly and another implementation may us something much simpler, e.g., a cache useful for testing.

trait DataManagement { def dm : DataManagement . Service } object DataManagement { trait Service { def cache [ T ] ( key : String , policy : CachePolicy ) ( effect : => Task [ T ] ) : ZIO [ DataManagement with CacheSystem , Throwable , Dependency [ T ] ] def cacheWithLog [ T ] ( key : String , effect : => Task [ T ] , policy : CachePolicy ) ( log : Dependency [ T ] => Task [ Unit ] ) : ZIO [ DataManagement with CacheSystem , Throwable , Dependency [ T ] ] } def cache [ T ] ( key : String , policy : CachePolicy ) ( effect : => Task [ T ] ) = ZIO . accessM [ DataManagement with CacheSystem ] ( _ . dm . cache ( key , policy ) ( effect ) ) def cacheWithLog [ T ] ( key : String , effect : => Task [ T ] , policy : CachePolicy ) ( log : Dependency [ T ] => Task [ Unit ] ) = ZIO . accessM [ DataManagement with CacheSystem ] ( _ . dm . cacheWithLog ( key , effect , policy ) ( log ) ) trait Live extends DataManagement { val dm = new Service { def cache [ T ] ( key : String , policy : CachePolicy ) ( effect : => Task [ T ] ) = Requests . withCache [ T ] ( effect , key , _ => Task . unit , policy ) def cacheWithLog [ T ] ( key : String , effect : => Task [ T ] , policy : CachePolicy ) ( log : Dependency [ T ] => Task [ Unit ] ) = Requests . withCache [ T ] ( effect , key , log , policy ) } } object Live extends Live }

The code for caching was shown in a previous blog. It’s still quite messy becaue we want to support UI features to improve the customer experience and I coded it in only a few minutes. We place the heavy lifting in the Requests object but it could have gone anywhere including inline in the DataManagement sub-system above.

case class AsyncData [ T ] ( current : Dependency [ T ] = NotRequested , last : Option [ Available [ T ] ] = None ) object Requests { def withCache [ T ] ( effect : Task [ T ] , key : String , log : Dependency [ T ] => Task [ Unit ] , policy : CachePolicy = CachePolicy . CacheFirst , ) : ZIO [ CacheSystem , Throwable , Dependency [ T ] ] = ZIO . accessM { r => type ADT = AsyncData [ T ] import CacheSystem . _ def networkFetch ( skipLog : Boolean ) = { val updateit = ( if ( skipLog ) Task . unit else log ( InProgress ) ) * > updateOrPut [ ADT ] ( key , AsyncData [ T ] ( InProgress ) ) ( v => v . copy ( current = InProgress ) ) ( updateit . provide ( r ) * > effect ) . foldM ( e => Task . succeed ( Error ( e ) ) , d => { val s = Available [ T ] ( d , false , false ) updateOrPut [ ADT ] ( key , AsyncData [ T ] ( s , Option ( s ) ) ) ( v => v . copy ( last = Option ( s ) ) ) . provide ( r ) * > Task . succeed ( s ) } ) } val adata_opt = r . cache . get [ ADT ] ( key ) val last_opt = adata_opt . flatMap ( _ . last ) val current_opt = adata_opt . map ( _ . current ) for { result <- ( last_opt , current_opt , policy ) match { case ( _ , _ , CachePolicy . NetworkOnly ) => networkFetch ( false ) case ( _ , Some ( x @Available ( s , i , c ) ) , CachePolicy . CacheFirst ) => if ( ! i && c ) Task . succeed ( x ) else Task . succeed ( x . copy ( inflight = false , cache = true ) ) case ( Some ( last ) , Some ( InProgress ) , CachePolicy . CacheFirst ) => Task ( last . copy ( inflight = true , cache = true ) ) case ( Some ( last ) , _ , CachePolicy . CacheFirst ) => Task ( last . copy ( inflight = false , cache = true ) ) case ( _ , Some ( x @Available ( s , i , c ) ) , CachePolicy . CacheAndNetwork ) => networkFetch ( true ) * > Task . succeed ( x ) case ( Some ( last ) , Some ( InProgress ) , CachePolicy . CacheAndNetwork ) => if ( last . inflight && last . cache ) networkFetch ( true ) * > Task . succeed ( last ) else networkFetch ( true ) * > Task . succeed ( last . copy ( inflight = true , cache = true ) ) case ( Some ( last ) , _ , CachePolicy . CacheAndNetwork ) => if ( last . cache ) networkFetch ( true ) * > Task . succeed ( last . copy ( inflight = false ) ) else networkFetch ( true ) * > Task . succeed ( last ) case _ => networkFetch ( false ) } _ <- log ( result ) * > updateOrPut [ ADT ] ( key , AsyncData [ T ] ( result ) ) ( v => v . copy ( current = result ) ) } yield result } }

Config System

The config system merely provides some web app config values. In a web app, these are often injected via a bundler such as webpack so that you can switch out config parameters based on the build type, production versus development. You might think that the RemoteSystem should be dependent on the ConfigSystem to obtain a base URL. That’s not wrong to do, but it does cause more coupling than necessary. At environment creation time, the RemoteSystm needs a base URL, but it does not need it during use. Hence, it is better to reduce coupling between services.

trait ConfigSystem { def config : Config . Service } object ConfigSystem { trait Service { val build : BuildConstants } trait Live extends Config { val config = new Service { val build = BuildConstants } } object Live extends Live }

Runtime System

This is a bit of a goofy service.

We provide the zio runtime system for convenience, as a service. Since we are using the enviromnent both as a global variable to access the zio runtime system as well as an enviroment for zio effects, we can define it like:

trait RuntimeSystem { def runner : RuntimeSystem . Service } object RuntimeSystem { trait Service { val runtime : zio . Runtime [ ZEnv ] def run [ T ] ( effect : => Task [ T ] ) ( handler : Exit [ Throwable , T ] => Unit ) : Unit def run_ [ T ] ( effect : => Task [ T ] ) : Unit } def live ( r : zio . Runtime [ ZEnv ] ) = new RuntimeSystem { val runner = new Service { val runtime = r def run [ T ] ( effect : => Task [ T ] ) ( handler : Exit [ Throwable , T ] => Unit ) : Unit = runtime . unsafeRunAsync ( effect ) ( handler ) def run_ [ T ] ( effect : => Task [ T ] ) : Unit = runtime . unsafeRunAsync_ ( effect ) } } }

We could create the zio runtime in a package. Again, we could use this directly as is from the package, but we will stuff it into the environment as well. We have chosen to promote two “run” methods at the service level in order to make running effects more convenient.

package object toplevel { private val rts = new zio . DefaultRuntime { } }

and access it via runtime.runner.run(...) outside zio.

We can make the runtime environment available to any component using a react hook:

object toplevel { val ZioEnvContext = react . context . create [ AppEnv ] ( null ) def useZioEnvironment ( ) = { react . React . useContext ( ZioEnvContext ) } }

and then wrap our entire application once:

val zenv = Env ( . . . , . . . , toplevel . rts ) reactdom . createRoot ( "container" ) match { case Left ( e ) => println ( "Did not find container." ) case Right ( render ) = > render ( ZioEnvContext . Provider ( zenv ) ( Application ( . . . ) ) ) }

Use It

We can now use zio to create queries. Creating the end queries is quite simple. Here is a query that fetches the “people” home view in the web app. Let’s define some supporting structures:

trait APIArrayResponse [ T ] extends js . Object { val value : js . Array [ T ] } trait Item extends js . Object { val id : String val name : String }

Now define the query itself:

import RemoteSystem . _ val fetchItems = for { env <- ZIO . environment [ QueryAppEnv [ String ] ] qk = "/people/home" + process ( env . qargs ) b <- builder [ HTTPError ] qe = b . get ( qk , asJSON [ api . APIArrayResponse [ Item ] ] , RequestInit ( headers = b . getHeaders ( ) ) ) . map ( _ . value ) } yield ( qk , qe )

Again, notice that we used qe = ... to define the last value because our QueryDescriptor is an effect that produces a tuple whose ._2 value is an effect. If we wanted to call other services in this query definition, we could inside the for comprehension. For example, could explicitly log the request or fetch two pieces of data.

To run a QueryDescriptor in the react component, we can define a react hook that handles the plumbing for us. Other than the zenv used for running the effect, the effect is composed only using services from the environment.

def useZio [ P < : js . Any , T ] ( queryd : QueryDescriptor [ P , T ] , args : P , autorun : Boolean = false , cachePolicy : CachePolicy = CachePolicy . CacheFirst , ) : ( Dependency [ T ] , ( ) => Unit ) = { import DataManagement . _ val mounted = React . useRef [ Boolean ] ( false ) React . useEffectMounting ( ( ) => mounted . current = true ) val zenv = useZioEnvironment ( ) val ( state , setState ) = React . useStateStrictDirect [ Dependency [ T ] ] ( NotRequested ) val run = React . useCallback [ Unit ] ( mounted . current , args . asJsAny ) { ( ) => val qenv = QueryEnv . withArgs [ P ] ( args , zenv ) val effect = for { kandf <- queryd key = kandf . _1 query = kandf . _2 data <- cacheWithLog [ T ] ( key , query . provide ( qenv ) , cachePolicy ) { d => if ( ! mounted . current ) Task . succeed ( ( ) ) else Task ( setState ( d ) ) } } yield data zenv . rts . run ( effect . provide ( qenv ) ) { case Failure ( e ) => setState ( Error ( e . squash ) ) case _ => } } React . useEffectMounting { ( ) => mounted . current = true if ( autorun ) run ( ) ( ) => mounted . current = false } ( state , run ) }

This is a basic hook that tracks the mount status so it does not perform calls on an unmounted component. We have minimized reliance on “global variables” using library support in zio and react. When the effect runs, the result is “set” into the state which forces a react component re-render.

To add this to a react function component:

object MyComponent { val fetchItems = . . . trait Props extends js . Object { val id : String } def apply ( props : Props ) = sfc ( props ) val sfc = SFC1 [ Props ] { props => val ( fstate , dofetch ) = useZio [ String , js . Array [ Item ] ] ( fetchItems , props . id , autorun = true ) } }

In the real world, we would actually initiate the fetch for the item in the react router or anyhere prior to rendering this component. Fetching earlier means that the results may be available more quickly creating a better user experience.

To initiate the fetch outside the component, we need a fetch function that is much like the hook. The code below shows why we want to be able to use the general “environment” anywhere in the program and not just in a hook or zio effect

object toplevel { def runAndCache [ P , T ] ( env : Env , queryd : QueryDescriptor [ P , T ] , args : P ) = { val qenv = QueryEnv . withArgs [ P ] ( args , env ) val effect = for { kandf <- queryd key = kandf . _1 query = kandf . _2 data <- cache [ T ] ( key , query . provide ( qenv ) , CachePolicy . CacheFirst ) } yield data env . rts . run_ ( effect . provide ( qenv ) ) } }

Here’s where we eagerly start the fetch prior to rendering MyComponent when the user clicks on something that would navigate to a view that shows MyComponent.

BigComponent ( new BigComponent . Props { onClick = id => { toplevel . runAndCache ( toplevel . env , MyComponent . fetchItems , id ) history . push ( "/MyComponent" ) } } )

We could use react suspense and other techniques to enhance the UI responsiveness, but that’s another blog :-)

What about retry and all that?

We have a couple of choices around features such as retry for failed effects. zio supports retry directly in the API. However, that may or may not be the right answer depending on how our effects are created. We may need to add security signatures to our effects that are only good for 5 minutes so its possible that we need to compose retry in the client layer or in the react useZio hook where we compose our effect with caching, we could add retry there across the entire application.

If there are no showstoppers to doing so, we could just add retry to each affect that we want retry on. For example,

import zio . Schedule . { exponential , elapsed } val fetchItems = for { env <- ZIO . environment [ QueryAppEnv [ String ] ] qk = "/people/home" + process ( env . qargs ) b <- builder [ HTTPError ] qe = b . get ( qk , asJSON [ api . APIArrayResponse [ Item ] ] , RequestInit ( headers = b . getHeaders ( ) ) ) . map ( _ . value ) . retry ( exponential ( 10 . milliseconds ) && elapsed . whileOutput ( _ < 30 . seconds ) ) } yield ( qk , qe )

That’s alot of typing, so its best to embed this in useZio or the client implementation directly. We could even provide a service that provides a retry “policy” as a value (the Schedule parameter to retry is value) but that’s over-engineering in most cases unless different environments reflect accessing different databases which have different performance characteristics.

As another example, we may not want to use the cache we defined as a service but the browser’s localStorage that persists between sessions. For example, we may want to cache a user image to display in the UI’s “user profile” page. We want to first check local storage then fetch it remotely using, for example, the Microsoft Graph API.

Below is the code that accesses browser storage for a data URI. We do not make browser storage a service although we could and abstract out cross-session storage APIs. Since we are always in the browser, that feels like over-engineering unless there is another compelling reason to make it part of the environment. Hence, we can use the below value:

val imageKey = "msgraph.me.image" val meImageURLFromLocalStorage = ZIO . fromOption ( Option ( dom . window . localStorage . getItem ( imageKey ) ) ) . flatMap ( durl => blobhelpers . blob ( durl ) pipe jsPromiseToZIO ) . map ( blob => Option ( URL . createObjectURL ( blob ) ) ) . orElse ( Task . fail ( new NoSuchElementException ( s "Empty option" ) ) )

We access the browser storage. ZIO.fromOption since getItem could return null if the key is missing. fromOption returns a IO[Unit, A] which indicates that the only error value that can be generated is () if there is no value in the Option. The blobhelpers help convert values back and forth using browser APIs but we do not show those details here. Rest assured, browser APIs are horrible and confusing.

We need to use .orElse to convert the error type from Unit to Throwable so we can combine it later with another effect that does a remote fetch with an error type of Throwable. You often run into the need to convert error types in ZIO. orElse is one way, refineOrDie or even flatMap can help you with error type conversion.

To fetch the image from Microsoft Graph we can compose an effect using a for-comprehension and our “client.” But we need an access token first. We could do something like:

class MSGraph ( accessToken : String ) { val token = Seq ( "Authorization" - > s "Bearer $accessToken" ) val fetchMeImageURL = for { env <- ZIO . environment [ RemoteSystem [ HTTPError ] ] b <- builder [ HTTPError ] blob <- b . get ( "/me/photos/48x48/$value" , asBlob , RequestInit ( headers = b . getHeaders ( token ) ) ) # use token data_url <- blobhelpers . dataURL ( blob ) pipe jsPromiseToZIO _ = data_url . foreach ( url => dom . window . localStorage . setItem ( MSGraph . imageKey , url ) ) } yield data_url . map ( _ => URL . createObjectURL ( blob ) ) }

But that’s a bit yucky because we should really just make accessing tokens a service in our environment so we can change how tokens are retrieved as our application evolves–common when building a SaaS application. Of course, in normal procedural code, the design above is not too horrible, but there is really no reason to formulate it as a class.

trait TokenSystem { def tokens : TokenSystem . Service } object TokenSystem { trait Service { def accessToken : Task [ String ] } def accessToken = ZIO . accessM [ TokenSystem ] ( _ . tokens . accessToken ) trait Live extends TokenSystem { val tokens = new Service { val accessToken = Auth . acquireAccessTokenMaybeInteractive . map ( _ . accessToken ) } } object Live extends Live }

Here we access a token, which is a String wrapped in an effect, using a singleton Auth object which we don’t define in this blog. Auth calls the MS Graph javascript API to obtain an access token assuming the user is already logged in. The MSAL identity management library from Microsoft caches the access token so it typically returns quickly if a valid access token is available. Keeping it as a service is a good idea because the token could also come from another call. For example, if you change your deployment model to Azure with AppServices, you need to fetch the token from fetch("/.auth/me") .

We need to change our Env definition:

trait BaseEnv . . . . with TokenSystem

Now we can change our effect definition to access the token from the environment. This is quite manual and we would typically define some functions that automatically insert an access token for us somewhere in our infrastructure:

object MSGraph { import RemoteSystem . _ import TokenSystem . _ val endpoint = "https://graph.microsoft.com/v1.0" def mk_auth_header ( t : String ) = Seq ( "Authorization" - > ( "Bearer " + t ) ) val fetchMeImageURL = for { env <- ZIO . environment [ RemoteSystem [ HTTPError ] with TokenSystem ] b <- builder [ HTTPError ] t <- accessToken auth_header = mk_auth_header ( t ) blob <- b . get ( "/me/photos/48x48/$value" , asBlob , RequestInit ( headers = b . getHeaders ( auth_header ) ) ) data_url <- blobhelpers . dataURL ( blob ) pipe jsPromiseToZIO _ = data_url . foreach ( url => dom . window . localStorage . setItem ( MSGraph . imageKey , url ) ) } yield data_url . map ( _ => URL . createObjectURL ( blob ) ) }

We can define a hook that just runs effects and does not cache. This is a just a cut-down version of the useZio hook which caches the results by default. For illustrative purposes, this hook just runs an effect and returns the result.

object hooks { def useZioSimple [ R > : AppEnv , T ] ( effect : RIO [ R , T ] , autorunOnMount : Boolean = false , ) : ( Dependency [ T ] , ( ) => Unit ) = { import DataManagement . _ val mounted = React . useRef [ Boolean ] ( false ) val zenv = useZioEnvironment ( ) val ( state , setState ) = React . useStateStrictDirect [ Dependency [ T ] ] ( NotRequested ) val run = React . useCallback [ Dependency [ T ] => Unit , Unit ] ( mounted . current ) { ( ) => zenv . rts . run ( Task ( setState ( InProgress ) ) * > effect . provide ( zenv ) ) { case Success ( value ) => if ( mounted . current ) setState ( Available ( value , false , false ) ) case Failure ( cause ) => if ( mounted . current ) setState ( Error ( cause . squash ) ) } } React . useEffectMounting { ( ) => mounted . current = true if ( autorunOnMount ) run ( ) ( ) => mounted . current = false } ( state , run ) } }

That defines some basic, reusable infrastructure. Back to obtaining the user image…

Neither “fetch” methods values have their types explicitly written but if we were to right them down. they would be:

val fetchMeImageURL : ZIO [ RemoteSystem [ HTTPError ] , Throwable , Option [ String ] ] = . . . val meImageURLFromLocalStorage : ZIO [ Any , Throwable , Option [ String ] ] =

We have 2 different environments but fortunately, RemoteSystem is a subtypeof Any . We can now create the real method using:

val mePhoto = meImageURLFromLocalStorage orElse fetchMeImageURL

It’s type is RIO[RemoteSystem[HTTPError],Option[String]] which shows that it retains the Throwable error type. I typically do not write-out the type unless there is a problem or need to document the type in source code. It’s probably best to explicitly list the return type.

To use all of this we just need to use the hook:

object ImageComponent { trait Props extends js . Object { } def apply ( props : Props ) = sfc ( props ) val sfc = SFC1 [ Props ] { props => val ( state , fetch ) = hooks . useZioSimple ( MSGraph . mePhoto , autorun = true ) val url match { case Available ( urlopt , _ , _ ) => urlopt case _ => None } Image ( src = url . orUndefined ) } }

Stick Everything into the Environment?

In the design above, we placed all the parts we needed into the environment. That’s probably overkill for a web application but it does focus the design on the dependencies needed much better than having a few other global methods floating around. You need to choose how much to place into the environment. You might put more into the environment if you are going to use the environment not just for zio and effects but for use in other parts of the appliction, e.g., for some singletons. Many javascript web apps use ES6 modules to control the scope of their “global variables” and this often leads to the inability to customize those packages.

If you need customization, say due to rapidly changing requirements, then using the environment as shown above makes alot more sense. However, even for a small, relatively static application, I can easily look at the environment definition and know exactly what is being used in the application. The application feels like a statically typed dependency injection design–which may be good or bad depending on your experience with DI.

Overall, using services with clean separation is a good idea and if those services interact with each other, the DI/module style approach does help make it manageable. But, you need to learn what should go into the environment and what can stay out of it. The environment should not be used as a catch all everything. I like the UI-tree scoped Context feature in react as it provides a way to easily control what environment a component uses. Together, the ability to define modules and easily scope them to where I need them seems like a win.

I also like using simple module patterns versus complex structures like Free or other things. Personally, I prefer using features that are well supported in the programming language and Scala is not Haskell or Lisp, so using features that the language supports well means I’m not fighting the compiler and ecosystem. I’m personally not against a little sub-typing when sub-typing helps you orgarnize our code. Excessive sub-typing is probably bad though but the module pattern in zio is pretty thin so I did not find it to be a problem.

It’s also easy to see where Scala 3 will provide some benefits and we could pull out some services from the zio environment and use the builtin reader monad in Scala 3 for some aspects of the program. I found Scala 3’s “reader monad” support helpful my recent machine learning program. It’s great to have good choices.

There’s more to do around query management but that’s another blog :-).

That’s it!