Valkey triggers proposal

[ranshid]

This is a proposal to introduce Valkey Triggers.
Valkey triggers are persistent procedures which are executed in response to some keyspace event.
Today Valkey keyspace Notifications offer the ability to publish events to dedicated pub/sub channels.
Valkey triggers add the ability to place extending logic on such keyspace events and to localize actions on the server side reducing the need to monitor and react to events on the application side.
We propose to extend the existing Valkey Functions with the ability to define trigger function call on specific keyspace event.
As being integrated in the Valkey Functions infrastructure, Triggers are persisted as part of the user data.

[daniel-house]

I think they should be reset at the same time that function-statistics are reset.

[daniel-house]

In this context I take the user to be a developer.

[ranshid]

yes

[daniel-house]

Why isn't the log sufficient?

Based on how errors are counted, it seems that a triggered function might attempt to report an error that should be delivered to the client. This suggests that the message will not normally be delivered. What would be delivered?

I have not worked with LUA or EVAL in the past. It seems likely that the answers to some of my questions would be obvious if I had. Please point me at any reading you think would help.

[ranshid]

Since the trigger execution is hidden from the user we cannot just return it as an error.

For the case when the trigger is issued based on internal events (eg evictions) there is no request context to issue the error to.
For case were the event was issued as part of the complex command flow (eg script or transaction),
we have 2 options to consider trigger execution:

1. without breaking the action atomicity (in which case the trigger will be executed AFTER the command was executed) in which case the response to the command was already sent or written to the COB, so not much point or ability to return it to the client.
2. we can execute the trigger as soon as the event is reported, practically breaking the atomicity and in this case I agree there is a problem we need to consider how to handle trigger failure, since it potentially already changed the data.

[stockholmux]

First, let me say on on the record of conceptually likely stored procedures and triggers. I also find Lua to be fun and useful. I'm not sold on the combination described here though.

Second, While I personally like Lua, that being the only engine limits it for a lot of users.

Third, the sections about atomicity, recursive, and ACLs in this doc reveal how tricky it is to get this to work in the context of Valkey. The complexity in each of these makes this tricky to navigate for users in balancing security, stability, and data guarantees: someone attempting to use this is very likely to get it wrong and unwittingly sacrifice something they didn't intend.

Might the goals of triggers be accomplished by some other means: maybe a persistent keyspace notification mechanism? Perhaps hook interface to another runtime that would prove to be more flexible? There are some good inspirations for these in other NoSQL databases that might work here.

[stockholmux]

So, I'm a little confused here. It appears that you're using something different from the keyspace/keyevent notification format? Why not the same?

[ranshid]

The event description is identical to the channel name used to report the keyspace events.

[stockholmux]

How does this work with [WATCH]MUTLI/EXEC? Does it interrupt a transaction?

[ranshid]

I was hoping to discuss this section. IMO we should NOT break atomicity with triggers, although this was the way I implemented the PoC. allowing triggers during scripts and transactions can indeed lead to much confusion but alternatively, queuing them and allow them to be executed after the transaction can be problematic since the trigger context might be lagging behind the data. I agree this part is the most problematic.

[stockholmux]

This should default to least privilege rather than greatest.

[stockholmux]

All or nothing? What if someone wanted to disable some triggers?

[ranshid]

We can extend this in the future if we want. the point of this ability is that the developer/admin wants to access the data and have no interference of triggers while doing so.

[stockholmux]

why are triggers limited only to keyspace events? Could you also trigger on an event through some arbitrary channel/pattern?

[ranshid]

SO I thought of the keyspace events since they are already defined and known to users, so I think it would make it easier for users to adopt. I am open to other suggestions.

[stockholmux]

Yes please. The use of images is also inaccessible.

[ranshid]

.

[ranshid]

First, let me say on on the record of conceptually likely stored procedures and triggers. I also find Lua to be fun and useful. I'm not sold on the combination described here though.

thank you @stockholmux. Can you tell me what in the combination blocks you?

Second, While I personally like Lua, that being the only engine limits it for a lot of users.

While I might agree, LUA is still a very efficient engine. Also I find this out of scope of this discussion. in case we add more engines the support for the triggers will still be there.

Third, the sections about atomicity, recursive, and ACLs in this doc reveal how tricky it is to get this to work in the context of Valkey. The complexity in each of these makes this tricky to navigate for users in balancing security, stability, and data guarantees: someone attempting to use this is very likely to get it wrong and unwittingly sacrifice something they didn't intend.

I agree about atomicity. this is a challenge but lets be honest - it is not kids who are playing with Valkey - it is usually developers that can be thought to consider the actions they are placing. the execution will still be serialized and the developer can just as well shoot himself in the lag while communicating several clients to the same Valkey server.

Might the goals of triggers be accomplished by some other means: maybe a persistent keyspace notification mechanism? Perhaps hook interface to another runtime that would prove to be more flexible? There are some good inspirations for these in other NoSQL databases that might work here.

Sure It can. We can introduce timers which would help in many of the described cases.
However I think there will always be a usecase where someone wants to impact the data in a new way and will not have the feature in place.

[ranshid]

@stockholmux @daniel-house I would like to thank you for spending the time reviewing this!
Sorry I was only able to address some of your comments today.