View source on Github

Review and Status

Types

data ConduitResult input m output = Producing ( Conduit input m output) [output] | Finished ( Maybe input) [output] data Conduit input m output = Conduit { conduitPush :: input -> ResourceT m ( ConduitResult input m output) , conduitClose :: ResourceT m [output] }

When producing (the equivalent of processing for a sink), we can return output. This is because a conduit will product a new stream of output instead of producing a single output value at the end of processing.

A sink always returns a single output value, while a conduit returns 0 or more outputs (a list). To understand why, consider conduits such as concatMap (produces multiple outputs for one input) and filter (returns 0 or 1 output for each input).

(produces multiple outputs for one input) and (returns 0 or 1 output for each input). We have no special constructor like SinkNoData . That's because we provide no Monad instance for conduits. We'll see later how you can still use a familiar Monadic approach to creating conduits.

Simple conduits

import Prelude hiding (map, concatMap) import Data.Conduit -- A simple conduit that just passes on the data as-is. passThrough :: Monad m => Conduit input m input passThrough = Conduit { conduitPush = \input -> return $ Producing passThrough [input] , conduitClose = return [] } -- map values in a stream map :: Monad m => (input -> output) -> Conduit input m output map f = Conduit { conduitPush = \input -> return $ Producing (map f) [f input] , conduitClose = return [] } -- map and concatenate concatMap :: Monad m => (input -> [output]) -> Conduit input m output concatMap f = Conduit { conduitPush = \input -> return $ Producing (concatMap f) $ f input , conduitClose = return [] }

Stateful conduits

import Prelude hiding (reverse) import qualified Data.List import Data.Conduit import Control.Monad.Trans.Resource -- Reverse the elements in the stream. Note that this has the same downside as -- the standard reverse function: you have to read the entire stream into -- memory before producing any output. reverse :: Resource m => Conduit input m input reverse = mkConduit [] where mkConduit state = Conduit (push state) (close state) push state input = return $ Producing (mkConduit $ input : state) [] close state = return state -- Same thing with sort: it will pull everything into memory sort :: ( Ord input, Resource m) => Conduit input m input sort = mkConduit [] where mkConduit state = Conduit (push state) (close state) push state input = return $ Producing (mkConduit $ input : state) [] close state = return $ Data.List. sort state

import Prelude hiding (reverse) import qualified Data.List import Data.Conduit -- Reverse the elements in the stream. Note that this has the same downside as -- the standard reverse function: you have to read the entire stream into -- memory before producing any output. reverse :: Resource m => Conduit input m input reverse = conduitState [] push close where push state input = return $ StateProducing (input : state) [] close state = return state -- Same thing with sort: it will pull everything into memory sort :: ( Ord input, Resource m) => Conduit input m input sort = conduitState [] push close where push state input = return $ StateProducing (input : state) [] close state = return $ Data.List. sort state

Using conduits

-- Left fusion: source + conduit = source ($=) :: ( Resource m, IsSource src) => src m a -> Conduit a m b -> Source m b -- Right fusion: conduit + sink = sink (=$) :: Resource m => Conduit a m b -> Sink b m c -> Sink a m c -- Middle fusion: conduit + conduit = conduit (=$=) :: Resource m => Conduit a m b -> Conduit b m c -> Conduit a m c

useConduits = do runResourceT $ CL. sourceList [ 1 .. 10 ] $= reverse $= CL. map show $$ CL. consume -- equivalent to runResourceT $ CL. sourceList [ 1 .. 10 ] $$ reverse =$ CL. map show =$ CL. consume -- and equivalent to runResourceT $ CL. sourceList [ 1 .. 10 ] $$ (reverse =$= CL. map show) =$ CL. consume

If you have a stream of numbers, and you want to apply a conduit (e.g., map show ) to only some of the stream that will be passed to a specific sink, you'll want to use the right fusion operator.

) to only some of the stream that will be passed to a specific sink, you'll want to use the right fusion operator. If you're reading a file, and want to parse the entire file as textual data, you'll want to use left fusion to convert the entire stream.

If you want to create reusable conduits that combine together individual, smaller conduits, you'll use middle fusion.

Data loss

main = do let list = [ 1 .. 10 ] transformed = map show list (begin, end) = splitAt 5 transformed untransformed = map read end mapM_ putStrLn begin print $ sum untransformed

main = do let list = [ 1 .. 10 ] (begin, end) = splitAt 5 list transformed = map show begin mapM_ putStrLn transformed print $ sum end

deviousTransform = concatMap go where go 1 = [show 1 ] go 2 = [show 2 , "two" ] go 3 = replicate 5 "three" go x = [show x]

deviousTransform 1 = [show 1 ] deviousTransform 2 = [show 2 , "two" ] deviousTransform 3 = replicate 5 "three" deviousTransform x = [show x] transform5 :: [ Int ] -> ([ String ], [ Int ]) transform5 list = go [] list where go output (x:xs) | newLen >= 5 = (take 5 output', xs) | otherwise = go output' xs where output' = output ++ deviousTransform x newLen = length output' -- Degenerate case: not enough input to make 5 outputs go output [] = (output, []) main = do let list = [ 1 .. 10 ] (begin, end) = transform5 list mapM_ putStrLn begin print $ sum end

1 2 two three three 49

take 5

SequencedSink

sum5Raw :: Resource m => Conduit Int m Int sum5Raw = conduitState ( 0 , 0 ) push close where push (total, count) input | newCount == 5 = return $ StateProducing ( 0 , 0 ) [newTotal] | otherwise = return $ StateProducing (newTotal, newCount) [] where newTotal = total + input newCount = count + 1 close (total, count) | count == 0 = return [] | otherwise = return [total]

sum5 :: Resource m => Conduit Int m Int sum5 = sequenceSink () $ \() -> do nextSum <- CL. isolate 5 =$ CL. fold (+) 0 return $ Emit () [nextSum]

sum5Pass :: Resource m => Conduit Int m Int sum5Pass = sequenceSink 0 $ \count -> do if count == 8 then return $ StartConduit $ CL. map (* 2 ) else do nextSum <- CL. isolate 5 =$ CL. fold (+) 0 return $ Emit (count + 1 ) [nextSum]

Summary