I decided to write my first blog post on ZIO/ZIO Streams. Hope it is time well spend with the extra time we have got at home during the Coronavirus lockdown :). And hopefully you find it useful to get you started with ZIO Streams.

I have been following ZIO for quite a while now and have found the library fascinating to have One Monad to Rule Them All in Scala functional programming world. Recently I have been playing with ZIO Streams specifically. Comparing to Akka streams, ZIO Streams is much simpler, more flexible and more expressive.

I am going to write a simple batch program using ZIO Streams, to show you how simple it is to use ZIO Streams to write a processing pipeline in just few lines of code. Let’s begin!

We have production log messages coming out from somewhere like a stream. We would like to process the log line by line when the message comes. Because stream is reactive and lazy, program writing in stream fashion is not loading all data into memory at once, so the program is more efficient and scallable. For simplicity reason we will just create the stream from a log file for our program here . The log file content looks like this:

Id_1 2017–10–28 00:04:43 INFO c.m.CamelApp: Running with Spring Boot

Id_3 2017–10–28 00:04:43 WARN c.m.CamelApp: No active profile set, falling back to default profiles: default

Id_4 2017–10–28 00:04:56 INFO c.m.amelBean: Process Data File: C:\data\full\Residential_201709301056.txt

Id_5 2017–10–28 00:06:37 ERROR c.m.s.RepoServiceImpl: null: 1048 Portage Rd

Id_6 2017–10–28 00:06:37 WARN c.m.s.RepoServiceImpl: Total properties in file: 25338, New: 25337, Existing: 0

Id_7 2017–10–28 00:34:34 INFO c.m.s.RepoServiceImpl: Active input: 25344

Id_8 2017–10–28 00:35:54 DEBUG c.m.s.RepoServiceImpl: Done parsing file: 871.xml

These are the requirements:

process the log line by line using stream

filter out messages that is not ERROR or WARN

split the stream into 2 streams: one for ERROR, another one for WARN messages

process the ERROR message according to certain time schedule eg. Every 2 seconds

process the WARN messages in batch. Eg group 10 messages into one batch

ZStream is the main construct in ZIO Streams. First we are going to create ZStream from file input stream:

ZStream.fromInputStream(fileInputStream)

What you get is a Chunk[T] of stream. You can think Chunk[T] as immutable version of Array[T] but more efficient.

.chunks .aggregate(ZSink.utf8DecodeChunk) .aggregate(ZSink.splitLines) .mapConcatChunk(identity)

Then we convert the stream of Chunk[String] into a stream of String and each element is one line message from the log file.

ZIO Streams has a .tap function which you can use to peak the messages inside the stream. It is a great tool for debugging your stream app.

.tap(data => putStrLn(s"> $data"))

Then we filter out the messages that are not ERROR nor WARN

.filter(isErrorWarning)

The next line of code splits current stream in to 2 streams based on the isError predicate returns true or false. Number 4 is the buffer size, so the two sub streams can run at different speed up to the buffer size:

.partition(isError, 4) val errorStream = leftStream .mapM(processError(_)) .schedule(Schedule.fixed(2.second))

Each ERROR message in left stream gets processed every 2 seconds using ZIO Schedule. ZIO Schedule provides powerful combinators allows you compose rich schedule logic like this:

val thisSchedule = Schedule.spaced(2.second) && Schedule.recurs(5)

For WARN messages in the right stream, we are going to group them into 6 messages at a time and process them in batch:

val warningStream = rightStream .aggregate(ZSink.collectAllN[String](10)) .mapM(processWarning(_))

You can think ZSink is the message consumer in the context of message producer/consumer if it is easier for you to understand. (And think ZStream as the message producer). ZSink is often used for aggregation function in ZIO Stream.

At the last, we merge 2 streams into one and collect all elements:

errorStream.merge(warningStream).runCollect

This is the complete code:

import java.nio.file.{Files, Paths}



import zio._

import zio.console._

import zio.duration._

import zio.stream._



object LogStreamApp extends App {



def isErrorWarning(data: String) = {

data.contains("ERROR") || data.contains("WARN")

}



def isError(data: String): Boolean = {

data.contains("ERROR")

}



def processError(data: String) = {

putStrLn(s"process error message: ${data}") *>

Task.succeed()

}



def processWarning(list: List[String]) = {

putStrLn(s"process warning messages in batch: ${list.length} => $list") *>

Task.succeed()

}



def run(args: List[String]): ZIO[ZEnv, Nothing, Int] = {

val is = Files.newInputStream(Paths.get(ClassLoader.getSystemResource("prod_log.txt").toURI()))



val theJob = (for {

streams <- ZStream

.fromInputStream(is)

.chunks

.aggregate(ZSink.utf8DecodeChunk)

.aggregate(ZSink.splitLines)

.mapConcatChunk(identity)

.tap(data => putStrLn(s"> $data"))

.filter(isErrorWarning)

.partition(isError, 4)

} yield streams).use {

case (leftStream, rightStream) => {

val errorStream = leftStream

.mapM(processError(_))

.schedule(Schedule.fixed(2.second))



val warningStream = rightStream

.aggregate(ZSink.collectAllN[String](10))

.mapM(processWarning(_))



errorStream.merge(warningStream).runCollect

}

}



theJob.fold(_ => 1, _ => 0)

}

}

You can find the code at github. Happy coding!

Thanks John De Goes for creating ZIO and for promoting pure functional programming in Scala world! Thanks Itamar Ravid for his amazing talk on ZIO Streams!