Question: Simple example of a functional process with state mutation (and error handling)

[OlliMartin]

Hi!

INB4: I barely understand what I am talking about. Please forgive me if something is very stupid.

I am trying to learn about functional programming currently and already implemented some logic with Either and a state delegate.
This morning I migrated to version 5 and most of my (somehow working) code broke, giving me a strong feeling that what I did was completely wrong.

After some reading I saw that version 5 will drop support for everything async and read a bit more.

I tried to follow some of the examples that are available in the repository, being especially interested in the ones carrying state. This one I found TESTING.cs and I tried to adjust it (simplified) to my needs.

In general I usually come across the following requirements:

• A process should be executed where multiple steps can fail
• Subprocesses can but are not required to run async
• In case of an error functional subprocesses should be skipped, unless specifically recovered from the error

I believe that chaining (?) Either is a good approach for this, however I again and again run into the problem regarding state.

In the past I would use one single class (record) that contains the entire state and if a (sub)process would need to change the state, it would create a new instance of the state with a certain property now populated. So that would make the process look roughly like:

await StartProcess()
  .Bind(ctx => SubProcess1(ctx)
  .Bind(ctx => SubProcess2(ctx);

Where the context has multiple (init-only) properties that are nullable, so they can be populated by the individual sub-processes.

That's stupid. First of all these nullabe properties force me to write null-checks for something that cannot happen, making the code unreadable and prevents me from covering it sensefully with tests.
One thing that came to my mind is to have the individial subprocesses return a new (sub)type when they are called, where the property doesn't need to be nullable; Take these classes as an example what I want to achieve
(Big Asteriks: I want to achieve given that this is the right approach in general - If this doesn't make any sense.. please tell me)

class State
{
    public State(string initialState) { InitialState = initialState; }
    
    public string InitialState { get; init; }
    
    public class First : State {
        
        public First(State state, string additionalProp) : base(state.InitialState)
        {
            AdditionalProp = additionalProp;
        }
        
        public string AdditionalProp { get; init; }
    }

    public class Second : First
    {
        public Second(First first) : base(first, first.AdditionalProp) { }
    }
}

Based on the linked example from the tests I know how to modify a state, but it didnt click so far how to achieve a chaining of different states. I reckon it is some kind of Transformer (?) I'm missing. When I naively try to change the type of the state, there is no possible SelectMany (?)

OptionT<StateT<State, IO>, string> m0 = from w in Pure(123)
    from q in m1
    from x in StateT.get<IO, State>()
    from _ in StateT.put<IO, State.First>(new(x, "my-transformed-state-prop"))
    from _2 in StateT.get<IO, State.First>()
    from y in m2
    select $"{q + w + y}";

Is this going into the right direction? How would a process that step-by-step fills a state be handled correctly?

I would be very thankful for any help and I sincerly apologize if there is an example for this exact case already and I didnt find it.

Edit: I'm reading through all of the amazing blog posts currently.. But I have to admit I'm more a "learning-by-doing" guy

[louthy]

@OlliMartin There's quite a bit to unpack here, so I'll try and cover what I can.

If you're just learning FP then be a little wary of going all-in with monad-transformers (OptionT, StateT, etc.) - they're quite advanced topics... very powerful, but can be a bit of a head-scratcher to start with. My Higher Kinds series will help, but if you get stuck, know that there's a simpler route to functional programming without going quite all the way in like this. The code ends up less elegant, but is cognitively easier to start...

1. Pass state as arguments:

static X foo(..., YourState state) =>
   ...

2. Return updated state:

static (YourState State, X Value) foo(..., YourState state) =>
   ...

This is what the State monad does automatically, which is why they're so powerful, but sometimes this more explicit approach can aid the understanding more.

If you're happy to go all-in, then ...

This morning I migrated to version 5 and most of my (somehow working) code broke

v5 is a once in a decade refresh of the library. The State monad (which wasn't particularly useful before) has been refactored significantly. Apologies if your code broke, but I doubt many people were using State before. State is most useful when combined with other monadic types (which is what the StateT monad-transformer is about).

After some reading I saw that version 5 will drop support for everything async and read a bit more.

The idea is to remove the duplication of *Async variants of every method, function, and type. Not to drop support for async in its entirety. To do async operations now you lift them into the IO monad using IO.liftAsync(...). The IO monad is the IO and side-effects type. It's there to contain all the messy stuff of talking with the outside world.

I tried to follow some of the examples that are available in the repository, being especially interested in the ones carrying state. This one I found TESTING.cs and I tried to adjust it (simplified) to my needs.

I actually forgot about that. That was just some test-code I used whilst prototyping v5 ideas. I'm not sure that will be super useful. I'll probably delete that at some point...

OptionT<StateT<State, IO>, string>

One critical aspect of stacking monad-transformers is that order matters. A monad-transformer stack must be 'unstacked' when run. Usually, that would mean calling the Run method several times (in your example it would be once for OptionT, once for StateT, and once for IO).

Each layer of that unstacking creates a context for the inner layers. So, if you want state for all of your expressions, then you should have the StateT monad on the outside. If you want an optional result, then you should have the optional-monad on the inside. IO is always the inner-most monad (if it is needed at all).

So, instead of:

OptionT<StateT<State, IO>, A>

Use:

StateT<State, OptionT<IO>, A>

The other thing to note is that carrying types like StateT<State, OptionT<IO>, A> around your code-base is quite cumbersome. It's also effort to change later if you want to add some other effect to your monad-transformer stack. So, it's good practice to wrap all of that up into a new type and give it a name related to the domain it covers (Db<A>. App<A>, ...)

As luck would have it, there is a sample in the Samples folder that uses StateT<State, OptionT<IO>, A> to create a Game<A> monad. It is a really good example of how you can do all of the things you're asking:

• CardGame/Game.cs - this is the entry point to the game. This is where the players are entered, the deck is shuffled, and the game is played; and finally the winners announced. This should show you how there's almost no visible parameters or manual state or failure-management going on, it is all done automatically by the Game monad.
• CardGame/Game.Monad - this folder contains the code that wraps StateT<State, OptionT<IO>, A> into a new type called Game<A>.
  • Game.cs - This is the main type. It'a a record with a single StateT<State, OptionT<IO>, A> property.
  • Game.Monad.cs - This implements the traits: Monad<Game>, SemigroupK<Game>, and Stateful<Game, GameState>. This implementation enables this the Game<A> type to be used in generic functions that expect those traits.
  • Game.Module.cs - This has a lot of the core functions that are the basic behaviours of the Game monad. Lots of the state-management is here (like addPlayer or currentHighScore).

Before I write anything else I think it'd be a good thing for you to dig into that sample. It takes the ideas as far as they can go and is as close to Haskell as you're likely to get in C#. The entire application is pure (apart from Program.cs where the game is launched).

[OlliMartin]

This is awesome, thank you for your immediate response (and apologies for not seeing the discussion tab.. facepalm).
I had already looked at the CardGame project, but was quite frankly overwhelmed by all the different one-line methods with approximately 20 generics.
Having read through most of the "Higher kinds" posts, I think I slowly start figuring out the purpose here (the traits forcing game-specific implementations ?)

While I will finish the blog posts and then go through the card game again, this time hopefully less clueless.. Maybe I already ask this:

As written above, usually I want to build some kind of functional flow, carrying a (changing) state, in the cardgame the GameState type seems to remain the same all the time.
Maybe this is me still not fully grasping it, but how would I manage a functional flow that progressively fills properties?
Say the first step in a flow chooses one object (by some kind of logic) out of 10 (from the state?) and later in the execution a subprocess depends on that value?

[louthy]

This is awesome, thank you for your immediate response

No problem and thank you for the donation!

As written above, usually I want to build some kind of functional flow, carrying a (changing) state, in the cardgame the GameState type seems to remain the same all the time.

The reason for creating these wrapper types (like Game<A>) is to hide the non-changing elements and expose only the parametric aspects. So, you could instead write:

public record Game<S, A>(StateT<S, OptionT<IO>, A> runGame) : K<Game, A>
{
    // ...
}

Or, more usefully, a constrained parametric state:

public record Game<S, A>(StateT<S, OptionT<IO>, A> runGame) : K<Game, A>
   where S : IGameState
{
    // ...
}

So, if you want parametric state this approach still hides the monad-transformer stack and the specifics of the implementation. Let's say that if at a later date you wanted to carry some configuration with your Game, you could change the implementation to:

public record Game<S, A>(ReaderT<Config, StateT<S, OptionT<IO>>, A> runGame) : K<Game, A>
{
    // ...
}

And the only code you'd need to change in the rest of the code-base is within the Game related types. Anything that consumes Game<A> wouldn't need changing. It's refactor friendly.

Other common approaches are to make the innermost monad parametric:

public record Game<M, A>(StateT<GameState, OptionT<M>, A> runGame) : K<Game, A>
    where M : Monad<M>
{
    // ...
}

Then you can lift any monad you like into the bespoke type:

Game<IO, A>
Game<Eff, A>
Game<Identity A>

Or, whatever you like.

When it comes to state, it really depends on the use-case. If the bespoke monad you create is effectively a state-machine, then having a fixed and hidden state-type can work well. Then the monad can manage the rules of the transformation through the 'Module' functions (see Game.Module).

If you want a completely general purpose state carrying monad then make it parametric. In practice most people don't want that and instead want to build well-defined state management into the bespoke monad module functions. A bit like how you'd do it in a class. The main difference between developing a class and a monad is that the monad is a cross-cutting concern in your application, it's not a 'thing you call', it's a regional rules system for your code.

[OlliMartin]

Thanks again!
I will mess around with transformers and see if it helps me wrap my head around it.
This stacking of transformers sounds very intriguing to me and I feel like things like retry or timeout handling are more or less free with them.
Searching through the codebase there is no RetryT or TimeoutT, which clearly shows that I didnt understand it yet. :D
Presumably I am thinking to much from a Polly point of view, where you wrap the execution into a policy.
(Please do not get this as a question.. I will dissect and play with all I learned now and try to figure out why these are not there on my own.)

Thanks Paul!

[louthy]

Searching through the codebase there is no RetryT or TimeoutT, which clearly shows that I didnt understand it yet. :D

The way to think about it is this:

• What needs retrying?
  • Things that can fail.
• What fails?
  • Things that talk to the outside world (reading a file, sending a message over a network, etc.)
• Other things fail too, right? Like parsing some text into an integer?
  • Yeah, but will retrying that help?
    • No.

So, the only thing that needs retrying are exceptional events. Not expected failures, like a failure to parse some text. Exceptional events are where the outside world creates some form of randomness that we can't handle.

And so, that's why retrying is attached to the IO<A> monad. Because that's where these side-effects belong.

IO has a special place in the monad-transformer stacks. Because IO (if it is used) is expected to be the inner-monad monad of a transformer-stack, there are a few MonadIO trait functions that (when implemented) make IO functionality 'just work' in the parent type.

• MonadIO.LiftIO - will take any IO<A> monad and inject it into your bespoke monad
• MonadIO.ToIO - will take any K<M, A> monad and will extract the IO monad from within it (to K<M, IO<A>>)
• MonadIO.MapIO - will take any K<M, A> monad and will extract the IO monad from within it so it can be mapped.

MapIO means you can get something like a ReaderT<Env, IO, A> and run Retry on it like so:

ReaderT<Env, IO, A> ma = ...;
ma = ma.MapIO(io => io.Retry(...)).As();

So, all of the IO functionality is available if MapIO is implemented.

MapIO has a default implementation:

public static virtual K<M, B> MapIO<A, B>(K<M, A> ma, Func<IO<A>, IO<B>> f) =>
    M.ToIO(ma).Bind(io => M.LiftIO(f(io)));

So, all IO functionality is available automatically in the containing types, if the containing types implement ToIO and LiftIO.

Much of the core IO functionality has been written to extend K<M, A> (i.e. any monad), it just expects there to be ToIO and LiftIO support. For example, the RetryIO extension is implemented like this:

public static K<M, A> RetryIO<M, A>(this K<M, A> ma) 
    where M : Monad<M> =>
    ma.MapIO(io => io.Retry());

So, RetryIO will work for any monad (as long as it overrides and implements LiftIO and ToIO).

It's the same for TimeoutIO, ForkIO, BracketIO, RepeatIO, FoldIO, ...

All that's happening is that the transformer-stack is being unpacked to inject the behaviour (Timeout, Retry, etc.) and then being repacked so that it's got its bespoke wrapper with the extra functionality.

It may well seem like sorcery I'm sure!

[OlliMartin]

Me again - I think I'm making some progress.. I've got the Game example "mapped" to something like a flow, but there's a few unclear points.

    private static Flow<Transformer> FindMatchingTransformer =>
        (
            from tuple                in gets(state => (state.SearchCriteria, state.Transformers))
            from foundTransformer     in lift(tuple.Transformers.Find(t => t.Handles(tuple.SearchCriteria)))
            select foundTransformer
        ).As();

This is a piece of the game code where the player is looked up in the cardgame (little bit adjusted to my needs).

The Find() returns an Option, but the foundTransformer type is Transformer, not Option.
Is the assumption that this is the OptionT doing its magic correct? I find it a bit confusing since it doesnt show up in any type hierarchy at play here.

However, when I run the thing, the types are as expected:

Expression<Func<FlowState, Option<(Transformer Value, FlowState State)>>> expression = (initialState) 
    => Flow.OverallFlow.executeFlow.Run(initialState).Run().Run(null);

var func = expression.Compile();

Option<(Transformer Value, FlowState State)> result = func(inputState1);
Console.WriteLine(result);

I shamelessly copied over the lift function from the cardgame:

    public static Flow<A> lift<A>(Option<A> ma) => 
        new (StateT<FlowState>.lift(OptionT<IO>.lift(ma)));

So I guess this is where the (for me) magic happens - even though I cannot pinpoint some kind of pattern match (to Some/None) when I F12 into the library.

EitherT

For what I would like to achieve, having a generic definition of a functional flow, I need some kind of Error propagation (not necessarily handling - just bubble it up and conditionally stop processing). I understand that this is where Fin and Either come into play.
Either is what I (mis)used with moderate success in version 4 and I like how I can provide my own Error type instead of having it baked in, so that is what I tried.

Following the cardgame I tried to setup my monad stack as a first thing, which I think should look like this:

public record FlowEither<A>(StateT<FlowState, EitherT<Error, IO, FlowOutcome>, A> executeFlow) : K<FlowEither, A>

I want my state on the outside and the IO monad is always the inner-most, so I was pretty confident this is correct. But no luck, the compiler complains that EitherT<Err, IO, Outcome> must be convertible to Monad<EitherT<Err, IO, Outcome>.
I am not sure if this is again me doing something terribly wrong, potentially (very unlikely) missing functionality in the library, or if I am expected to create my own specific Monad class that extends EitherT.
I tried the same with FinT which has the same issue; While ReaderT, WriterT (and OptionT) work.

Is it expected to provide an own wrapper type to make this work? I am somewhat confident that I am not missing an implicit cast somewhere, as the compiler complains directly about the EitherT not being a Monad. Hopefully this is not another instance of the C# compiler giving horrible error messages when it comes to more advanced generics.

Cheers!
Olli

[louthy]

The Find() returns an Option, but the foundTransformer type is Transformer, not Option.
Is the assumption that this is the OptionT doing its magic correct? I find it a bit confusing since it doesnt show up in any type hierarchy at play here.

Yes, if you're implementing the same lift as in the CardGame sample, then it will take an Option and make it into the wrapped monad. But it does that by lifting the Option into OptionT, then lifting OptionT into StateT, and then finally constructing a Game<A> from the StateT.

The left-hand-side of the from ... in ... is the unpacked value of the packed right-hand-side. So, you can think of in as the operator the runs the rules of the monad. If you have a monad that's made from a monad-transformer stack, then in will run all of the rules of the stack (state-management, optionality, side-effects).

even though I cannot pinpoint some kind of pattern match (to Some/None) when I F12 into the library.

From earlier: "The Find() returns an Option". That's where the Some/None is coming from.

the compiler complains that EitherT<Err, IO, Outcome> must be convertible to Monad<EitherT<Err, IO, Outcome>

You've embedded the full EitherT<L, M, R> type into the transformer-stack, you need to use its trait-type, which is the same just without the bound-value type R: EitherT<L, M>...

public record FlowEither<A>(StateT<FlowState, EitherT<Error, IO>, A> executeFlow) : K<FlowEither, A>

The A type-variable is the bound-value type. There should only be one per stack. Not sure what FlowOutcome is, but it can't work in this context.

By the way, you might want to consider not using EitherT at all if you're going to use Error as the base-type for all of your raised errors. The IO<A> monad, although an exception throwing type, does use the currency of Error:

• IO.fail(Error) - allows you to raise any Error type in an expression that supports IO
• @catch, .Catch, .IfFail, .MapFail, and .BiMap - all support Error, so you can can catch, process, transform, and resolve errors of type Error.
• For None you can use IO.Empty which is IO.fail(Errors.None)

The benefit of using IO for this is that you can remove OptionT and EitherT from the stack completely (for a small performance improvement and simplification of the monad).

Obviously, if you're learning this stuff for fun, then it may make sense to keep EitherT in the stack also.