Repository navigation
Add RuleEvaluationSummary to NodeReadinessRule status and CRD - #502
AnuragThePathak wants to merge 1 commit into
Conversation
👷 Deploy request for node-readiness-controller pending review.Visit the deploys page to approve it
|
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: AnuragThePathak The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Hi @AnuragThePathak. Thanks for your PR. I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with Tip We noticed you've done this a few times! Consider joining the org to skip this step and gain Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
f38441a to
52f1211
Compare
52f1211 to
d48285c
Compare
|
/kind feature |
Description
Currently the NodeReadinessRule reports the impact of the rule across the cluster through the nodeEvaluations field. For people leveraging prometheus, there are metrics to provide that information numerically. However, there's no numerical representation of that for kubectl users, or someone using tools like Lens. While working on Headlamp also we had this exact requirement and we ended up iterating through the nodeEvaluations array to calculate this information per rule.
This PR addresses that gap, by introducing an evaluationSummary field in NodeReadinessRule status. The RuleEvaluationSummary struct contains 3 integer fields -
Targeted = Satisfied + Unsatisfied
Also added Kubebuilder printcolumns to display Targeted and Satisfied (priority=1) in
kubectl get nrr -o wide.This paves a clear pathway for deprecation of nodeEvaluations field in combination with NRE. NRE provides fine-tuned Node level evaluation results, while RuleEvaluationSummary provides rule specific evaluation results statistics. FailedNodes is yet another unaddressed area still.
Trade-offs considered
Originally proposed idea in #484, explored around the idea of using nodeEvaluation to calculate the summary on the fly inside
processAllNodesForRuleandprocessNodeAgainstAllRules. After that there was another evolution to populate the satisfied and unsatisfied stats from spec directly and using nodeFailure for the failure count. Both these methods had a serious drawback of triggering O(n) loop for every node event and O(n) status write risk for n node events.This experience led us to look for references in upstream projects -
Why we went with Debounced Workqueue Coalescing pattern: Our constraints were 5k nodes at max and a humanly manageable number of rules, and it seemed like per node event, it would be highly optimal to map nodes to applicable rules and queue the rules combined with coalescing rules in the queue over a 1-2 second window.
O(n) node iteration for evaluation is still there but when you limit the frequency and consider the other constraints, it no longer remains the bottleneck rather the etcd API write to update the rule status. This was convincing enough to not to opt for the Kueue style pattern.
How we adopted it here:
It was identified that rule reconciler should be in charge of writing to rules. As part of that, the rule reconciler was configured to have a secondary watch on the node resources with appropriate predicates to filter out the noise, then perform node to rule mapping and queue the rules for reconciliation but with TypedDelayingInterface.AddAfter which puts a handbrake on the standard rate at which reconciliation happens and allows duplicate queue items to merge. For now we have resorted to a window of 2 seconds, considering the fact the evaluation summary is meant for human inspection purpose mainly.
Also because we had to accommodate the change within the existing reconcile method for Rule Reconciler, we used check on
Generation != ObservedGenerationto detect whether it is triggered by pre-existing Rule GenerationChangedPredicate event or the new Node events. In case we find the spec has changed (Generation != ObservedGeneration) we follow the existing flow, accompanied by RuleEvaluationSummary computation which ends up in a single status write anyway for the whole updated status. If it's not a Rule GenerationChangedPredicate event, then we compute the RuleEvaluationSummary only and write the status to etcd and complete the reconciliation process. In case of merger of a node event driven reconciliation request and rule GenerationChangedPredicate driven request, the first path is taken and it's intentional only.Also it may be worth noting that, we avoid API status write for the summary only path when there's no changes in the counts of any field.
There is also a new method named nodeSatisfiesRule been added with this PR, which actually may be useful for avoiding duplication of same logic in multiple places.
Tests that could have been written as pure unit functions and benefit heavily from repetition of same setup are added as go-style unit tests only in separate files, keeping in mind the increasing size of nodereadinessrule_controller_test.go file.
The original issue, requests for an error count too. However during active development it was observed that errors appear purely from the network failure, RBAC or other kind of infra platform failures while applying/removing taints etc and not anywhere in the process of evaluation of nodes against the rules. That forced us to rely on nodeFailures of the status field. However, technically speaking nodeFailures in the worst case scenario of cluster wide failure can also lead to high memory usage pushing beyond 1.5MiB limit, and also the fact that it can trigger a rule write on each node event, makes it not so suitable field to retain in the long run just like nodeEvaluation. There were some other considerations around possible successor. However the final decision was to exclude error or failure count from initial RuleEvaluationSummary, because first of all the main purpose and name justifies that (because error is actually not associated with evaluation process directly) and to considering the fact that removing an existing field is harder than adding a new one (to not introduce tech debt now by using nodeFailure + to keep choices open for v1alpha2).
Related Issue
Type of Change
/kind feature
/kind api-change
Testing
Tested with KWOK running the local build.
Checklist
make testpassesmake lintpassesDoes this PR introduce a user-facing change?
Generative AI Usage Disclosure
How they were used:
AI was used to fetch references from upstream projects and other kind of information combined with reference quotation for me to review and understand. Also leveraged it for idea exploration including design options, and bootstrapping of code wherever applicable. Also used for test environment setup with Kwok and more.