Skip to content

Add RuleEvaluationSummary to NodeReadinessRule status and CRD - #502

Open
AnuragThePathak wants to merge 1 commit into
kubernetes-sigs:mainfrom
AnuragThePathak:feat/evaluation-summary
Open

AnuragThePathak wants to merge 1 commit into
kubernetes-sigs:mainfrom
AnuragThePathak:feat/evaluation-summary

Conversation

@AnuragThePathak

@AnuragThePathak AnuragThePathak commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Description

Currently the NodeReadinessRule reports the impact of the rule across the cluster through the nodeEvaluations field. For people leveraging prometheus, there are metrics to provide that information numerically. However, there's no numerical representation of that for kubectl users, or someone using tools like Lens. While working on Headlamp also we had this exact requirement and we ended up iterating through the nodeEvaluations array to calculate this information per rule.

This PR addresses that gap, by introducing an evaluationSummary field in NodeReadinessRule status. The RuleEvaluationSummary struct contains 3 integer fields -

  • Targeted: Total number of nodes in cluster the Rule's NodeSelector matches.
  • Satisfied: Number of nodes that satisfy the rule as per the ConditionPolicy and ConditionRequirements (bootstrap-completed scenario is also considered as satisfied).
  • Unsatisfied: Number of nodes that do not satisfy the rule (opposite of satisfied)

Targeted = Satisfied + Unsatisfied

Also added Kubebuilder printcolumns to display Targeted and Satisfied (priority=1) in kubectl get nrr -o wide.

This paves a clear pathway for deprecation of nodeEvaluations field in combination with NRE. NRE provides fine-tuned Node level evaluation results, while RuleEvaluationSummary provides rule specific evaluation results statistics. FailedNodes is yet another unaddressed area still.

Trade-offs considered

Originally proposed idea in #484, explored around the idea of using nodeEvaluation to calculate the summary on the fly inside processAllNodesForRule and processNodeAgainstAllRules. After that there was another evolution to populate the satisfied and unsatisfied stats from spec directly and using nodeFailure for the failure count. Both these methods had a serious drawback of triggering O(n) loop for every node event and O(n) status write risk for n node events.

This experience led us to look for references in upstream projects -

  • Kueue style In-Memory Scorecard: This model operates on the idea of O(1) writes in-memory against an event using the concept of a map. Only the value against the impacted key/keys are updated directly instead of having to iterate through the whole workloads (in Kueue). Of course, for writing the update to the CRD status requires handling through separate channel/event setup because if you go to write to Etcd after performing the computation in O(1) in-memory, you will end up with net negative result.
  • Debounced Workqueue Coalescing from CAPI and EndpointSlice: This model accepts the cost of having to iterate through the workload or child resource in O(n) time per evaluation. Instead it aims to minimize the number of evaluation loops by Coalescing multiple events against a parent resource. The actual implementation of EndpointSlice controller involves a few more steps as they have slightly trickier constraints but this core pattern is consistent.

Why we went with Debounced Workqueue Coalescing pattern: Our constraints were 5k nodes at max and a humanly manageable number of rules, and it seemed like per node event, it would be highly optimal to map nodes to applicable rules and queue the rules combined with coalescing rules in the queue over a 1-2 second window.
O(n) node iteration for evaluation is still there but when you limit the frequency and consider the other constraints, it no longer remains the bottleneck rather the etcd API write to update the rule status. This was convincing enough to not to opt for the Kueue style pattern.

How we adopted it here:

It was identified that rule reconciler should be in charge of writing to rules. As part of that, the rule reconciler was configured to have a secondary watch on the node resources with appropriate predicates to filter out the noise, then perform node to rule mapping and queue the rules for reconciliation but with TypedDelayingInterface.AddAfter which puts a handbrake on the standard rate at which reconciliation happens and allows duplicate queue items to merge. For now we have resorted to a window of 2 seconds, considering the fact the evaluation summary is meant for human inspection purpose mainly.

Also because we had to accommodate the change within the existing reconcile method for Rule Reconciler, we used check on Generation != ObservedGeneration to detect whether it is triggered by pre-existing Rule GenerationChangedPredicate event or the new Node events. In case we find the spec has changed (Generation != ObservedGeneration) we follow the existing flow, accompanied by RuleEvaluationSummary computation which ends up in a single status write anyway for the whole updated status. If it's not a Rule GenerationChangedPredicate event, then we compute the RuleEvaluationSummary only and write the status to etcd and complete the reconciliation process. In case of merger of a node event driven reconciliation request and rule GenerationChangedPredicate driven request, the first path is taken and it's intentional only.

Also it may be worth noting that, we avoid API status write for the summary only path when there's no changes in the counts of any field.

There is also a new method named nodeSatisfiesRule been added with this PR, which actually may be useful for avoiding duplication of same logic in multiple places.

Tests that could have been written as pure unit functions and benefit heavily from repetition of same setup are added as go-style unit tests only in separate files, keeping in mind the increasing size of nodereadinessrule_controller_test.go file.

The original issue, requests for an error count too. However during active development it was observed that errors appear purely from the network failure, RBAC or other kind of infra platform failures while applying/removing taints etc and not anywhere in the process of evaluation of nodes against the rules. That forced us to rely on nodeFailures of the status field. However, technically speaking nodeFailures in the worst case scenario of cluster wide failure can also lead to high memory usage pushing beyond 1.5MiB limit, and also the fact that it can trigger a rule write on each node event, makes it not so suitable field to retain in the long run just like nodeEvaluation. There were some other considerations around possible successor. However the final decision was to exclude error or failure count from initial RuleEvaluationSummary, because first of all the main purpose and name justifies that (because error is actually not associated with evaluation process directly) and to considering the fact that removing an existing field is harder than adding a new one (to not introduce tech debt now by using nodeFailure + to keep choices open for v1alpha2).

Related Issue

Type of Change

/kind feature
/kind api-change

Testing

Tested with KWOK running the local build.

Screenshot 2026-11-14 at 2 54 31 AM Screenshot 2026-11-14 at 2 56 30 AM

Checklist

  • make test passes
  • make lint passes

Does this PR introduce a user-facing change?

Add RuleEvaluationSummary to NodeReadinessStatus, presenting Targeted, Satisfied, and Unsatisfied node count against a Rule.

Generative AI Usage Disclosure

  • No AI tools were used
  • AI tools were used (complete below)

How they were used:

AI was used to fetch references from upstream projects and other kind of information combined with reference quotation for me to review and understand. Also leveraged it for idea exploration including design options, and bootstrapping of code wherever applicable. Also used for test environment setup with Kwok and more.

@kubernetes-prow kubernetes-prow Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 11, 2026
@netlify

netlify Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

👷 Deploy request for node-readiness-controller pending review.

Visit the deploys page to approve it

Name Link
🔨 Latest commit d48285c

@kubernetes-prow

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: AnuragThePathak
Once this PR has been reviewed and has the lgtm label, please assign tallclair for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubernetes-prow kubernetes-prow Bot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Nov 11, 2026
@kubernetes-prow

Copy link
Copy Markdown

Hi @AnuragThePathak. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Tip

We noticed you've done this a few times! Consider joining the org to skip this step and gain /lgtm and other bot rights. We recommend asking approvers on your previous PRs to sponsor you.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@kubernetes-prow kubernetes-prow Bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Nov 11, 2026
@AnuragThePathak AnuragThePathak changed the title feat: add evaluation summary to NodeReadinessRule status and CRD Add RuleEvaluationSummary to NodeReadinessRule status and CRD Nov 12, 2026
@AnuragThePathak
AnuragThePathak marked this pull request as ready for review November 13, 2026 21:29
@kubernetes-prow kubernetes-prow Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 13, 2026
@AnuragThePathak

Copy link
Copy Markdown
Contributor Author

cc @ajaysundark @Karthik-K-N

@AnuragThePathak

Copy link
Copy Markdown
Contributor Author

/kind feature
/kind api-change

@kubernetes-prow kubernetes-prow Bot added kind/feature Categorizes issue or PR as related to a new feature. kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API labels Nov 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API kind/feature Categorizes issue or PR as related to a new feature. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant