Ensure update-agent waits for all volumes to be detached before rebooting

[invidian]

Original issue: coreos/container-linux-update-operator#191

Perhaps waiting for kubectl get volumeattachments to get empty with the right selector would be sufficient?

[Jasper-Ben]

Scenario: Running a ceph cluster using the rook operator. During drain, the volumes are detatched, however it might take some time to propagate the the kernel unmount. I have not looked into details, but according to @martin31821 this is caused by the ceph kernel client doing some foo during unmount, thus trying to change this from userspace is not possible. #62 introduces a quick workaround by just adding some wait time after draining the node.

[martin31821]

Maybe we can solve this by introducing a possibility to run one or more kubernetes jobs prior to rebooting, which could be used e.g. to change DNS records, wait a certain amount of time or run host commands prior to rebooting.

[invidian]

Not ideal, but I guess we could test against that on Lokomotive, as we have there a pipeline testing FLUO and Rook together. CC @surajssd

[invidian]

Note: existing capabilities for running hooks runs before node is drained, which indeed can make it impossible right now to deploy a custom hook which could ensure it. Perhaps this could be addressed.

[invidian]

As part of #37, I'm analyzing how FLUO works in details, as there is no documentation or tests and what comes to my mind is, that perhaps hooks model could be extended, so it's possible to run a workflow between each significant action taken, which would be:

• before draining the node (currently supported)
• before rebooting, but after draining the node (currently missing, requested by this issue)
• after rebooting, but before uncordoning the node (currently supported)
• after uncordoning? (currently missing/acts as a regular DaemonSet?)

However, existing state tracking model is overly complex and right now I don't feel comfortable adding another step to it. Perhaps we try to simplify it first, then extend with extra hook.

[Nuckal777]

We are affected by this is well. Some seconds of sleep after draining, like in #62 would help mitigating.

[invidian]

Just realized I think I hit this issue on my cluster as well 😄