Skip to content

Potential Policy workflow for Synchronizer leader election #2543

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
jvanderhoof wants to merge 1 commit into
base: master
Choose a base branch
from
+110 −0
Draft

Potential Policy workflow for Synchronizer leader election #2543

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added design/synchronizer/activity-diagram.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
18 changes: 18 additions & 0 deletions design/synchronizer/activity-diagram.puml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
@startuml
start
repeat
if (am leader?) then (yes)
:update `leader-expiration` to future time (1);
else (no)
if (`leader-expiration` time has passed (2)?) then (yes)
:delete leader host (3);
:create leader host (4);
if (host create includes API key?) then (yes)
:update `leader-expiration` to future time (5);
else (no)
endif
else (no)
endif
endif
repeat while (sleep)
@enduml
92 changes: 92 additions & 0 deletions design/synchronizer/auto-failover.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
# Synchronizer Auto-Failover

The following is an overview of the workflow for a Vault Sychronizer to gain leadership:

![](./activity-diagram.png)

Given the following Policy:
```yml
- !policy
id: conjur
body:
- !policy
id: vault-synchronizer
body:
- !variable leader-expiration

# The Group all Synchronizer hosts need to be added to.

@GrzegorzK-CA GrzegorzK-CA Apr 20, 2022

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How can I do that?
I see a few use cases here

  1. First installation
  • Synchronizer installer creates this policy
  • Synchronizer installer creates another policy for safes, lobs etc altogether with Synchronizer host
  • Synchronizer installer needs to update this policy - with host how?
  1. Installation of second instance of Synchronizer
  • Synchronizer installer updates another policy for safes, lobs etc altogether with new Synchronizer host
  • Synchronizer installer needs to update this policy - with host how?
  1. A few Synchronizer installers already exists - manual update needed
  • This policy is created manually
  • How to add already existing hosts here
  1. Synchronizer uninstall
  • How to remove host from this group

@jvanderhoof jvanderhoof Apr 21, 2022

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@GrzegorzK-CA, I put together a working example in the Conjur Intro project: https://github.com/conjurdemos/conjur-intro/blob/vault-synchronizer/bin/dap#L428.

The only thing missing here is setting the JSON document with safe names and last updated dates. Bash does not like the double quotes of JSON. Because the JSON is passed through so many functions, I was not able to get it escaped correctly. I updated the variable via the UI to make sure it worked as expected. You should have no problem doing this with the .NET SDK.

- !group members

- !permit
role: !group members
privileges: [ read, execute, update ]
resources:
- !variable leader-expiration

- !policy
id: leader
owner: !group members

- !policy
id: vault-lines-of-business
owner: !group members
body:

@GrzegorzK-CA GrzegorzK-CA Apr 20, 2022

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess this "vault-lines-of-business" policy should be created without body, and Synchronizer should create variables for each LOBs during synchonization intervals.
It would be great to have examples for that as well

@jvanderhoof jvanderhoof Apr 21, 2022

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great catch. I updated this in the Conjur Intro example above.

- !variable
id: lob-1
mime_type: application/json
- !variable
id: lob-2
mime_type: application/json
```

The following are the steps (using the API) listed above:

1. Update `leader-expiration` to future time:

```sh
POST /secrets/{account}/variable/conjur%2Fvault-synchronizer%2Fleader-expiration

@GrzegorzK-CA GrzegorzK-CA Apr 20, 2022

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"secrets"?
I thought it would be something like "conjur/vault-synchronizer/...."
but I guess this might be my lack of knowledge here

how about creating/updating variables for LOBs - will that also be under "secrets"?

@jvanderhoof jvanderhoof Apr 21, 2022

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this case secrets is the resource type. conjur%2Fvault-synchronizer%2Fleader-expiration is the url escaped variable ID.


Time.now() + <time in seconds> // Save as Unix epoch time
```

1. `leader-expiration` time has passed:

```sh
GET /secrets/{account}/variable/conjur%2Fvault-synchronizer%2Fleader-expiration
```

Then compare returned Unix epoch time time to current Unix epoch time.

1. Delete leader host:

@GrzegorzK-CA GrzegorzK-CA Apr 20, 2022

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens in the very first flow:

  • no leader
  • trying to delete vault-synchronizer-leader if that doesn't exist?

Should I check if that variable is set altogether with expiration date or can I just try to delete it and then try to create it.

@jvanderhoof jvanderhoof Apr 21, 2022

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Delete on a resource that's not present results in a no-op, so it should be fine to always attempt the delete before trying to claim "leader" status.


```sh
PUT /policies/{account}/policy/conjur%2Fvault-synchronizer%2Fleader

# Policy to delete the current leader
- !delete
record: !host current-leader
```

1. Create leader host:

```sh
POST /policies/{account}/policy/conjur%2Fvault-synchronizer%2Fleader

# Policy to delete the current leader
- !host
id: current-leader
annotations:
host: "<this synchronizer's host name>"
```

If the above command returns a JSON object with an API, this Synchronizer is the new leader. If not, another Synchronizer won.

The annotation provides an indicator of the "current leader".

1. Update `leader-expiration` to future time:

```sh
POST /secrets/{account}/variable/conjur%2Fvault-synchronizer%2Fleader-expiration

Time.now() + <time in seconds> // Save as Unix epoch time
```