Process managers

[alexeyzimarev]

I get lots of questions about process managers. Personally, I am not a big fan because:

• Processes are indeed everywhere
• Most processes are natural and not orchestrated
• Orchestration adds complexity

But, for the sake of it, I can try implementing it by adding some syntactic sugar to the existing building blocs:

• Aggregate -> Process
• AggregateState -> ProcessState
• ApplicationService -> ProcessManager

The manager needs to be called by a subscription. Possible by multiple subscriptions, but that can be done later. It also needs a producer to send commands. Here comes the question if I need to wire an existing app service that processes commands coming via HTTP also to consume command messages. It's possible even today, but maybe I can make it easier.

With a proper registration extensions, it should provide an easy to use API to wire all the things together.

[diegosasw]

I have a few questions about this.

Aren't process managers defined as "the opposite to an aggregate" as in

• an aggregate receives commands and emits domain events
• a process manager receives domain events and emits commands

or are they something else?

But, for the sake of it, I can try implementing it by adding some syntactic sugar to the existing building blocs

If Eventuous supports them as syntactic sugar, would its state be also event sourced? Because process managers don't usually have the technical limitation to be short lived, right?

Here comes the question if I need to wire an existing app service that processes commands coming via HTTP also to consume command messages.

To me, if there is a process manager is because there is no need for http commands, as the transaction is initiated automatically without the need of a UI. In other words, I see process managers as a replacement to UI and http communication.

---

For example, in a Restaurant domain where there are orders, waiters and each waiter wants to have a list of things that could do next each time it goes to the kitchen, this could be modelled with the following 3 aggregates.

Waiter producing events like

• AvailableSet
• UnavailableSet

and Order producing events like

• OrderPrepared
• OrderPicked (with waiter Id)
• OrderDelivered

and WaiterTodoQueue, as a queue of tasks associated to a specific waiter, producing events like

• OrderSuggestionAdded
• OrderSuggestionRemoved

Customers create orders and someone prepares them. Waiters pick prepared orders.
A waiter goes to a read model (fed by WaiterTodoQueue) and picks one of the suggestions. When picked, this suggestion disappears from other waiters' todo queues. When a waiter delivers it, the order is delivered.

If I want a "process manager" called Dispatcher in charge of keeping waiters busy by suggesting to them orders to be picked, I would want this dispatcher to have state + behaviour (probably business logic).

So the Dispatcher would keep track of available waiters and orders prepared in order to dispatch/send commands to *WaiterTodoQueue, this Dispatcher is indeed something that receives events and emits commands (towards WaiterTodoQueue aggregate).

The problem is that the Dispatcher state, if event sourced, could grow rapidly and become slower unless it uses snapshots or similar.

Most processes are natural and not orchestrated
Does that mean that the Dispatcher in the example above could be modelled differently without using orchestration?

TL;DR: What's the definition of process manager? Is its state event sourced? Can things always be modelled without introducing orchestration (something that receives events and emits commands)?

[alexeyzimarev]

What you describe as "aggregate" is basically a decider, which can be done using aggregates. Eventuous command services are quote close to that pattern.

Process managers, in general, receive messages and send messages. Whether the PM reacts to commands is discussable, but a process is usually started by a command anyway.

What I was looking in this issue initially is event-sourced state. As we have the state store now, the state object can be used in functional services, as well as in PMs. So, that's covered.

My assumption is that atm the only missing thing is multi-subscription support. Probably it's not even needed for the first iteration. But, definitely, it is needed later as I'd expect PMs to react on integration events, which might be emitted to different topics/streams. The question is more about the API rather than what's behind it. Doing AddSubscription(...).AddProcessManager<T>() and using the same instance of T across subscriptions is the easiest, and it will definitely work. I was thinking into a PM-first registration API like AddProcess<T>().SubscribeTo<T1, T2>().SubscribeTo<T3, T4>(), but it's much more work.

[gius]

There is one thing I don't get - let's assume that I need to conduct two aggregate types - A and B. Then I create a new instance of ProcessManager/Aggregate that handles the process, let's call it PM. How should I property identify instance of PM based on events from A and B that only contain AId, or BId respectively? Do I need to create a projection that allows finding PMId based on AId or BId?

In other words, how should I identify a particular PMId for handling an event coming from a particular A (thus containing just AId).

[bartelink]

As with anything else, you need to figure out some way to correlate the two sides
in some cases there's a parent-child relationship, and you don't need multiple instances - for those cases maybe the child id will sufffice

In the more general case, if you consider the process a workflow, you might mint a fresh identity for the instance

One benefit of this is that each instance of the process has an independent set of events and/or state and you don't get long streams. Of course that can potentially also be a problem if you have a need to coordinate the multiple running instances.

The you can make anything enlisted in it aware of same; either

• by feeding the id of the flow into each side out of band/naturally
• or when kicking off the process, you seed it with the ids of the participants, and make the first step to make those aware of their enlistment

(Will be interested to see/hear if Eventuous has any specific affordances wrt this, but for me process managers are just programming)

[gius]

You are right, especially with the parent-child relationship, you don't need another entity. My concern is exactly about the second case you mentioned - creating a new process manager instance with ids of its participants. What is the best way to notify the manager back when an event from any of its participants arrive.

The only idea I have is to create a projection that provides links from participantIds back to managerId and event subscription for all participant events that 'gateways' them to the respective owning manager. This should work (provided that there is not partitioning for events and thus participant or manager cannot get behind?) but just feels like a lot of boilerplate code for quite a typical case.

Feeding managerId to all participants feels a little bit like leaking abstraction, but might be appropriate in many cases, though.

Thank you for these ideas and brainstorming ;-)

[bartelink]

I would tend to have a reactor that sees all the events.
It should be handling the events in bulk across streams and forwarding them en masse
The question is how, based on solely the event, it determines the PM to route to either

• the event carries enough context to let you route it
• or you need to go back to the source to determine the PM to feed it to
  Once you know where to go, submitting an event to the PM is just a transaction against the workflow aggregate

(IIRC in there is a that sample in https://github.com/jet/dotnet-templates that does that, but the general principle applies regardless)

[alexeyzimarev]

My idea was more or less the same, in terms of implementation. Some questions remain unanswered though:

1. Should it be one subscription or multiple (transports? streams?). Because some processes happen across context boundaries and might be arriving using non-event-store-based transport (broker).
2. Having one subscription (or group of subscriptions, see (1)) per process manager type can very costly. Should PMs be maintained as a group? Adding a PM to process history is not a common scenario, so grouping should work.

Reactors are quite trivial, but (2) still applies. I think grouping is the way to deal with that, and, again, it's already supported:

eventuous/samples/postgres/Bookings/Registrations.cs

Lines 58 to 63 in 58b05ff

	services.AddSubscription<RabbitMqSubscription, RabbitMqSubscriptionOptions>(
	"PaymentIntegration",
	builder => builder
	.Configure(x => x.Exchange = PaymentsIntegrationHandler.Stream)
	.AddEventHandler<PaymentsIntegrationHandler>()
	);

eventuous/samples/postgres/Bookings/Integration/Payments.cs

Lines 9 to 30 in 58b05ff

	public class PaymentsIntegrationHandler : EventHandler {
	public const string Stream = "PaymentsIntegration";
	readonly ICommandService<BookingState> _applicationService;
	public PaymentsIntegrationHandler(ICommandService<BookingState> applicationService) {
	_applicationService = applicationService;
	On<BookingPaymentRecorded>(async ctx => await HandlePayment(ctx.Message, ctx.CancellationToken));
	}
	Task HandlePayment(BookingPaymentRecorded evt, CancellationToken cancellationToken)
	=> _applicationService.Handle(
	new RecordPayment(
	evt.BookingId,
	evt.Amount,
	evt.Currency,
	evt.PaymentId,
	""
	),
	cancellationToken
	);
	}

If no multi-transport support is required, combining a subscription with one or more command services would work, and it's easy to support with the existing model. Adding some shortcuts for the wiring is the only thing needed.

[alexeyzimarev]

It made me think that building blocks are already there. Eventuous didn't have functional services when I opened this issue, but now it does, and it's perfect to build a process manager as a functional service as it doesn't use aggregates but has state.