vault: delay the usage of renewed auth tokens #488
Conversation
aead
force-pushed
the
vault-replication-lag
branch
from
f491ea2 to
549a251
Compare
| const Retry = 3 // Retry token renewal N times before re-authenticating. | ||
| const ( | ||
| Retry = 3 // Retry token renewal N times before re-authenticating. | ||
| Delay = 30 * time.Second // Wait a certain amount of time before using a new token to account for Vault replication lag |
There was a problem hiding this comment.
should this be configurable? is 30sec enough?
There was a problem hiding this comment.
Delay is the time KES waits before it uses a new token received from one of the Vault nodes.
So one (actually N/2+1) Vault nodes consider this token as valid. However, due to replication lag some of the other (N/2-1) nodes may haven't received the token and cannot validate it. Hence, KES waits for 30s to allow other Vault nodes to catch up.
There is no "correct" value since Vault replication lag can be arbitrarily large. There could be a network partition between Vault nodes such that they "never" sync. However, 30s should be sufficient for all practical purposes. If you have higher replication lag, other issues arise most probably.
This commit adds a delay that has to ellapse before a new Vault auth. token is used. A dist. Vault cluster may experience some replication lag. Hence, some cluster nodes already have the renewed token while others don't. If KES makes a request to a Vault node that does not have the renewed token, then the Vault node will reject the request. By waiting a certain amount of time (e.g. 30s) KES allows the Vault cluster to sync a renewed token to all peer nodes. KES cannot know when the token will be replicated but 30s seems a reasonable value for most practical purposes. Signed-off-by: Andreas Auernhammer <[email protected]>
aead
force-pushed
the
vault-replication-lag
branch
from
549a251 to
db37961
Compare
This commit adds a delay that has to ellapse before a new Vault auth. token is used. A dist. Vault cluster may experience some replication lag. Hence, some cluster nodes already have the renewed token while others don't. If KES makes a request to a Vault node that does not have the renewed token, then the Vault node will reject the request.
By waiting a certain amount of time (e.g. 30s) KES allows the Vault cluster to sync a renewed token to all peer nodes. KES cannot know when the token will be replicated but 30s seems a reasonable value for most practical purposes.