Introduction

This is the second article in a series about continuations, forking, and monad transformers. Previous article. Today we will solve the “withLifted” riddle we posed at the very end of the last article.

Motivation Recapitulating the conclusion from the last article. Given: -- capture any monad stacks with a base of `ContT () IO`. class MonadIO m => MonadContIO m where liftContIO :: ContT () IO a -> m a ()m a -- insert the trivial instances forkCont2With :: MonadContIO m => ( IO () -> IO ()) -> m Bool ()()) = liftContIO $ ContT $ \c -> forker (c False ) >> c True forkCont2With forkerliftContIO\cforker (c moveContWith :: MonadContIO m => ( IO () -> IO ()) -> m () ()())m () = liftContIO $ ContT $ \c -> forker (c ()) moveContWith forkerliftContIO\cforker (c ()) moveToNewThread :: MonadContIO m => m () m () = moveContWith (void . forkIO) moveToNewThreadmoveContWith (voidforkIO) we can fork and “move”/“fork” any monadic operations who have a base of ContT () IO , allowing us for example to write this: = do main $ flip (evalStateT ( 0 :: Int )) $ do evalContT(evalStateT ()) + 1 ) modify ( moveToNewThread s <- get -- this and below happens in non-main thread get $ print s liftIO .. And we noted that in general, the “host” that runs the remaining effectful/stateful computation (in the above example forkIO ) requires no knowledge of the specific monad at work - the interface of the host is a plain IO () . With this somewhat surprising result, we started to wonder: How powerful exactly is ContT () IO , and can it solve other similar problems in the area of “lifting and unlifting” of transformer-stacks? The first thing we will consider in this area are “with” functions. Let us phrase the problem in haskell: -- mock-version of a resource acquire/release computation with a -- continuation-based interface. Consider this rather similar real-world -- function straight from the `base` library: -- Foreign.Marshal.Utils.with :: Storable a => a -> (Ptr a -> IO b) -> IO b mockWith :: String -> (() -> IO a) -> IO a (()a) = do mockWith s c putStrLn $ "acquire " ++ s x <- c () c () putStrLn $ "release " ++ s pure x main :: IO () () = flip evalStateT ( 0 :: Int ) $ do mainevalStateT ( $ mockWith "myResource" $ \() -> do liftIOmockWith\() x <- get -- again: d'oh! get .. ..

withLifted

Desired Semantics

Sometimes the type signature already is half of the solution, but in this case I will go ahead and directly propose:

withLifted :: MonadContIO m => ( forall r . (a -> IO r) -> IO r) -> (a -> m b) -> m b (ar)r)(am b)m b

If this has an implementation, then our motivating example could be written as:

main :: IO () () = evalContT $ flip evalStateT ( 0 :: Int ) $ do mainevalContTevalStateT ( "myResource" ) $ \() -> do withLifted (mockWith\() x <- get -- works, at least by types get .. ..

Testcases:

Proper ordering of acquire/release/what comes after: = evalContT $ do test1evalContT "myResource" ) $ \() -> liftIO $ print "inside" withLifted (mockWith\()liftIO $ print "outside" liftIO should print “acquire myResource, inside, release myResource, outside” Effects to pass through properly: = evalContT $ flip evalStateT ( 0 :: Int ) $ do test2evalContTevalStateT ( "myResource" ) $ \() -> modify ( + 1 ) withLifted (mockWith\()modify ( >>= liftIO . print getliftIO should print “acquire myResource; release myResource; 1” Nesting poses no problem: = evalContT $ flip evalStateT ( 0 :: Int ) $ do test3evalContTevalStateT ( "a" ) $ \() -> withLifted (mockWith\() "b" ) $ \() -> withLifted (mockWith\() + 1 ) modify ( >>= liftIO . print getliftIO should print “acquire a; acquire b; release b; release a; 1”

Towards a Solution

We will approach this in a few steps, because these steps help building intuition regarding expressiveness of ContT () IO in general. But feel free to step to the next section for the completed solution.

We will start with the blank template:

= do -- m withLifted with c -- with :: forall r . (a -> IO r) -> IO r -- c :: a -> m b _ -- :: m b

The first thing to note about this is that c is in m -monadic context, so any approaches where we have dropped down to IO will not work. For example, we will not be able to use c a in the right part of ContT $ \k -> . This directly mandates that our solution looks like this:

= do -- m withLifted with c a <- _ -- :: m a (yay for hole-driven development) b <- c a c a _ -- :: m () pure b

Focus on the first hole. We must get an a out, but our only “a-provider” is with :: forall r . (a -> IO r) -> IO r . So we need to drop to IO somehow anyway. So we are probably fine doing just that:

= do -- m withLifted with c a <- liftContIO $ ContT $ \k -> liftContIO\k _ with k -- ^ :: (forall r . (a -> IO r) -> IO r) -> (a -> IO ()) -> IO () b <- c a c a _ pure b

This directly leads to

attempt number 1 = do -- m withLifted with c a <- ContT $ \k -> with k \kwith k b <- c a c a pure b which type-checks. But does not work. Consider that k captures the entire remaining computation, including anything “below”. Meaning that = evalContT $ do test1evalContT "myResource" ) $ \() -> liftIO $ print "inside" withLifted (mockWith\()liftIO $ print "outside" liftIO would print “acquire myResource; inside; outside; release myResource”. Back to the drawing-board!

There is one escape-hatch we have not considered so far: The “second application” from the last article - we can “move” to a different thread. And we can move back. And nothing stops us from forking the “host” in beforehand as a first step. And from aligning things so that lifetime of this host thread aligns with that of resource allocated via with . Start with slightly modifying the relevant bits from the previous article:

contSingleProcessor :: MVar (a -> IO ()) -> a -> IO () (a())() = takeMVar mvar >>= \c -> c a contSingleProcessor mvar atakeMVar mvar\cc a moveToSingleProcessor :: MVar (a -> IO ()) -> ContT () IO a (a())() = ContT $ \c -> putMVar mvar c moveToSingleProcessor mvar\cputMVar mvar c

We use MVar s because only a single continuation will need to be processed.

= do -- m withLifted with c <- liftIO $ newEmptyMVar mvarliftIOnewEmptyMVar $ void $ forkIO $ with $ \a -> contSingleProcessor mvar a liftIOvoidforkIOwith\acontSingleProcessor mvar a a <- moveToSingleProcessor mvar moveToSingleProcessor mvar b <- c a c a -- remaining piece of this puzzle _somehowReturnFromChannel pure b

Taking into account that the with currently (again) captures all that comes “below”, but the remaining hole should escape that, it is clear that we need some sort of interaction with code from above the forkIO . We can just write this:

= do -- m withLifted with c <- liftContIO $ ContT $ _ -- (m () -> IO ()) -> IO () escapeliftContIO <- liftIO $ newEmptyMVar mvarliftIOnewEmptyMVar $ void $ forkIO $ with $ \a -> contSingleProcessor mvar a liftIOvoidforkIOwith\acontSingleProcessor mvar a a <- moveToSingleProcessor mvar moveToSingleProcessor mvar b <- c a c a escape pure b

So we pass something down, but we need to pass something up (the continuation at the point of invoking escape ). So, another MVar . We will focus on the operation that produces escape and define it separately:

createReturnPoint0 :: m (m ()) m (m ()) = liftContIO $ ContT $ \k -> do createReturnPoint0liftContIO\k <- newEmptyMVar mvarnewEmptyMVar k $ liftContIO $ ContT $ putMVar mvar liftContIOputMVar mvar -- ^ pass down an "escape" `m ()` that places the -- continuation into an MVar without executing it. <- takeMVar mvar k2takeMVar mvar -- run that MVar, but only after `k` has completed k2 () -- (by reaching "escape")

We will make one minor adjustment that makes both implementation and usage a bit shorter:

createReturnPoint :: m (a -> m a) m (am a) = ContT $ \k -> do createReturnPoint\k <- newEmptyMVar mvarnewEmptyMVar k $ \a -> liftContIO $ ContT $ \k2 -> putMVar mvar (k2 a) \aliftContIO\k2putMVar mvar (k2 a) $ takeMVar mvar jointakeMVar mvar

Which gives us:

attempt number 2 (almost!) = do withLifted with c <- createReturnPoint moveToOriginalContextcreateReturnPoint <- liftIO $ newEmptyMVar mvarliftIOnewEmptyMVar $ void $ forkIO $ with $ contSingleProcessor mvar liftIOvoidforkIOwithcontSingleProcessor mvar a <- moveToSingleProcessor moveToSingleProcessor b <- c a c a moveToOriginalContext b

This comes close, but suffers from race-conditions. To explain, we visualize the nested resource acquisition example:

main thread ║ resource "a" thread ║ evalContT resource "b" thread ║ │ created ║ │ ║ acquire a ║ └───── move via MVar ────┐ ║ takeMVar ┊ │ ║ start running continuation (blocked on takeMVar) │ ║ created ┊ │ ║ ║ acquire b ┊ └───── move via MVar ──────────┐ ║ takeMVar ┊ ┊ │ ║ ┊ (blocked on takeMVar) │ ║ inner part ┊ ┊ │ ║ ┊ ┌──────────────────────────────┘ ║ move back ┊ ┌────────────────────────┘ ║ directly move back ║ release b ┊ │ ║ release a x ║ │ any "outside" stuff x ║ │

Note that the ordering of release a and release b is not in any way guaranteed. (Nor that of “outside stuff” versus the releases.) We could start throwing some more MVar s at this - but we can also notice that we do not actually want any concurrency: The important stuff happens in createReturnPoint , while forkIO is superfluous:

Solution to the Riddle

createReturnPoint :: MonadContIO m => m (a -> m a) m (am a) = ContT $ \k -> do createReturnPoint\k <- newEmptyMVar mvarnewEmptyMVar k $ \a -> liftContIO $ ContT $ \k2 -> putMVar mvar (k2 a) \aliftContIO\k2putMVar mvar (k2 a) $ takeMVar mvar jointakeMVar mvar withLifted :: MonadContIO m => ( forall r . (a -> IO r) -> IO r) -> (a -> m b) -> m b (ar)r)(am b)m b = do withLifted withC f <- createReturnPoint escapecreateReturnPoint a <- liftContIO $ ContT withC liftContIOwithC b <- f a f a escape b

The magic really lies within createReturnPoint : It runs the continuation, but passes it a “continuation-continuation” (consider the type a -> m a , where m is has a base that is ContT () IO ). And afterwards, it grabs the continuation that the continuation-continuation has passed back upwards via MVar , and executes that. Crucially, this passing of the continuation back upwards escapes the scope of the with that we invoke in withLifted , resulting in exactly the desired ordering of IO (and non- IO ) effects.

Properties and Interactions

Our implementation passes the three tests we wrote above. That gives us three properties already:

withLifted retains the intended ordering of acquire/inner/release/outer effects. “inner” effects carry to the “outside”. nesting works - withLifted interacts well with withLifted .

and we may mention that withLifted does not fork itself in any fashion (no uses of forkIO )

Further, not too surprisingly: withLifted does not interact nicely with forkCont2 or plain moveTo -type functions. Consider $ flip evalStateT ( 0 :: Int ) $ do evalContTevalStateT ( "myResource" ) $ \() -> forkCont2 withLifted (mockWith\()forkCont2 .. or $ flip evalStateT ( 0 :: Int ) $ do evalContTevalStateT ( "myResource" ) $ \() -> moveToNewThread >> k withLifted (mockWith\()moveToNewThread .. In the first case, there will be two threads writing to an MVar , but we only take (and continue) once. In the second case, “release” will fire as soon as moveToNewThread is reached, meaning that k happens concurrently/after the “release” part of the with . However this is not new - using forkIO in plain IO “with”-functions has the same behaviour, and invites use-after-free problems or similar. This really is not a downside of our implementation, but inherent property of “with” functions. In the same vein we have to assume that the user does only use actions that call the continuation exactly once - there is high risk for deadlocks and race-conditions otherwise.

createReturnPoint and resetContIO

Our usage of createReturnPoint suggest that there is one more notion in this area, because we can write:

resetContIO :: MonadContIO m => m a -> m a m am a = do resetContIO m <- createFinalPoint escapecreateFinalPoint r <- m escape r -- or the unreadable one-liner: -- resetContIO = (createFinalPoint >>=) . (>>=) -- (also this permits a one-liner for `withLifted`:) -- withLifted withC f = resetContIO $ liftContIO (ContT withC) >>= f

Compare that to the resetT as defined in the transformers package:

resetT :: Monad m => ContT r m r -> ContT r' m r r m rr' m r = lift . evalContT resetTliftevalContT

So while they behave similar, there are two central differences:

resetT is polymorphic over the monad “below” in the monad stack but assumes that ContT is on top, while resetContIO is polymorphic over the “above” transformers but assumes that ContT () IO is at the bottom. They behave the same in that they “return to the original host” which means that “host-final” operations are run (the “release” in withLifted ). They behave different in that resetContIO blocks the original host, while resetT does not (see example below):

Interactions

withLifted interacts well with resetContIO , which is not surprising given that we implement the former using the latter and nesting already works.

In the interaction with moveTo type functions, resetT and resetContIO differ:

= evalContT $ do ~ test1 = do test1evalContTtest1 $ do void $ forkIO $ evalContT k resetTvoidforkIOevalContT k moveToNewThread evalContT o k -- must be `m ()` due to types o -- of resetT and moveToNewThread = evalContT $ do ~ test2 = do test2evalContTtest2 x <- resetContIO $ do mvar <- newEmptyMVar resetContIOmvarnewEmptyMVar $ forkIO $ moveToNewThread voidforkIO >>= putMVar mvar k evalContT kputMVar mvar >>= evalContT . o o x takeMVar mvarevalContT

createReturnPoint on its own however can easily create deadlocks, for example the following code deadlocks:

do <- createReturnPoint escape1createReturnPoint <- createReturnPoint escape2createReturnPoint escape1 escape2

On the other hand, creating a deadlock is not possible with resetContIO .

A Short Look at shiftT and callCC

Given the resetT / resetContIO pair, and the fact that MVar s require the call-continuation-exactly-once constraint, we will now consider interactions of shiftT and callCC with the new constructs. For reference:

callCC :: ((a -> ContT r m b) -> ContT r m a) -> ContT r m a ((ar m b)r m a)r m a = ContT $ \ c -> runContT (f (\ x -> ContT $ \ _ -> c x)) c callCC f\ crunContT (f (\ x\ _c x)) c shiftT :: ( Monad m) => ((a -> m r) -> ContT r m r) -> ContT r m a m)((am r)r m r)r m a = ContT (evalContT . f) shiftT f(evalContTf)

Bad news - deadlocks:

= evalContT $ resetContIO $ shiftT $ \_ -> pure () dead1evalContTresetContIOshiftT\_() = evalContT $ callCC $ \c -> resetContIO $ c () dead2evalContTcallCC\cresetContIOc ()

shiftT gives the user full control over what to do with the continuation, and of course this can breaks things for resetContIO (user ignores continuation - nobody fills MVar - deadlock)

gives the user full control over what to do with the continuation, and of course this can breaks things for (user ignores continuation - nobody fills - deadlock) callCC provide an “escape” type mechanism that also allows the user to effectively drop some continuation, which clashes with resetContIO (abort before reaching the putMVar step - deadlock)

While the interaction with forkCont2 or moveToNewThread appears to be relatively harmless (no deadlocks, both remain in the same “host”), they do not bring much to the table either. Neither shiftT nor callCC can easily be lifted to our MonadContIO m => m level. Together with the potential for deadlocks this seems to be enough reason to stay away from these two functions in the context of ContT () IO .

Outlook

In the next article we will approach throw and try for ContT () IO . This time however I cannot pose a concise riddle, because the solution will be a good bit more complex - we will even have to switch to a somewhat more powerful base monad because ContT () IO is not good enough.