Expose http timeout setting in Kubernetes package for kube state metrics by MichaelKatsoulis · Pull Request #19391 · elastic/integrations

MichaelKatsoulis · 2026-06-05T09:03:38Z

Proposed commit message

In large kubernetes cluster, fetching metrics from kube state metrics can exceed the default http timeout of 10s.
This error leads to leader nodes failures that are unrecoverable.
With this PR we expose the timeout option for KSM datasets that controls the http tiemout.
This option is availble in beats and standalone agent.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

Screenshots

github-actions · 2026-06-05T09:04:54Z

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

gizas · 2026-06-05T09:24:47Z

+          - name: timeout
+            type: text
+            title: Timeout
+            description: HTTP request timeout. Raise on large clusters if scrapes exceed 10s and you see `context deadline exceeded` errors.


Valid time units are ns, us, ms, s, m, h.

This is what we have in other timeouts in the integration

This points to this https://github.com/elastic/beats/blob/5f41561f849f0c899682cb5909f1a96dac43b59b/metricbeat/mb/mb.go#L349 beats which validates there.
In case of s is missing it treats this as seconds.

gizas · 2026-06-05T09:25:27Z

+            multi: false
+            required: false
+            show_user: false
+            default: 10s


qq: why in the screenshot shows 30s?
Did u change the value? just to verify that we dont miss sth

elastic-vault-github-plugin-prod · 2026-06-05T09:47:26Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

github-actions · 2026-06-05T11:28:28Z

TL;DR

Buildkite failed in Dev scripts unit tests before reaching your Kubernetes changes. The failing test relies on package names that do not exist on the PR base backport branch, so the failure is not caused by the files changed in this PR.

Remediation

Update dev/scripts/test_get_release_commit.sh to avoid hardcoded real-repo package fixtures (prometheus, zscaler_zpa, security_detection_engine) for branch-agnostic CI; use only the existing dummy-repo fixture section (setup_dummy_repo) for deterministic tests.
Re-run the failing step after that test adjustment.

Investigation details

Root Cause

dev/scripts/test_get_release_commit.sh currently starts with hardcoded real-package assertions:

dev/scripts/test_get_release_commit.sh:70
dev/scripts/test_get_release_commit.sh:74
dev/scripts/test_get_release_commit.sh:78

Those call dev/scripts/get_release_commit.sh, which exits with:

dev/scripts/get_release_commit.sh:52 (Error: package '${PACKAGE_NAME}' not found.)

In this Buildkite run (backport branch context), those package manifests are absent, so the test fails deterministically before validating PR-specific logic.

Evidence

Build: https://buildkite.com/elastic/integrations/builds/44199
Job/step: :bash: Dev scripts unit tests
Key log excerpt:
- Error: package 'prometheus' not found.
- Command: .buildkite/scripts/run_dev_scripts_tests.sh

Verification

Reproduced the same failure mode from the test/script paths above (missing package fixture causes immediate non-zero exit from get_release_commit.sh).

Follow-up

Consider making these script tests fully fixture-driven (dummy repo only) to keep them stable across backport branches with different package inventories.

Note

🔒 Integrity filter blocked 2 items

The following items were blocked because they don't meet the GitHub integrity level.

#19391 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
Expose http timeout setting in Kubernetes package for kube state metrics #19391 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

…s-ksm

elastic-vault-github-plugin-prod · 2026-06-05T11:59:36Z

✅ All changelog entries have the correct PR link.

elasticmachine · 2026-06-05T12:31:36Z

💚 Build Succeeded

Buildkite Build
Commit: 09212d0

History

💔 Build #44199 failed b91540e
💔 Build #44189 failed 381e2cc
💔 Build #44188 failed c012084

elastic-vault-github-plugin-prod · 2026-06-05T13:01:31Z

Package kubernetes - 1.83.1 containing this change is available at https://epr.elastic.co/package/kubernetes/1.83.1/

…ics (#19662) In large Kubernetes clusters, fetching metrics from kube state metrics can exceed the default HTTP timeout of 10s. This error leads to leader node failures that are unrecoverable. This PR exposes the `timeout` option for KSM datasets, controlling the HTTP timeout. This option is available in Beats and standalone agent. This is a forward-port of #19391 (merged into `backport-kubernetes-1.83`) to `main`.

elastic-vault-github-plugin-prod · 2026-06-19T11:03:43Z

Package kubernetes - 1.85.2 containing this change is available at https://epr.elastic.co/package/kubernetes/1.85.2/

Expose http timeout setting in Kubernetes package for kube state metrics

c012084

MichaelKatsoulis requested a review from a team as a code owner June 5, 2026 09:03

MichaelKatsoulis requested a review from gizas June 5, 2026 09:05

Update changelog.yml

381e2cc

This comment has been minimized.

Sign in to view

gizas reviewed Jun 5, 2026

View reviewed changes

gizas approved these changes Jun 5, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

This was referenced Jun 5, 2026

[dev/scripts] Make get_release_commit tests agnostic of real packages #19393

Merged

[dev/scripts] Make get_release_commit tests agnostic of real packages #19394

Merged

Add valid values for tiemout

b91540e

Merge branch 'backport-kubernetes-1.83' into expose-timeout-kubernete…

09212d0

…s-ksm

andrewkroh added Integration:kubernetes Kubernetes Team:obs-ds-hosted-services Observability Hosted Services team [elastic/obs-ds-hosted-services] labels Jun 5, 2026

MichaelKatsoulis merged commit d62851b into elastic:backport-kubernetes-1.83 Jun 5, 2026
9 checks passed

github-actions Bot mentioned this pull request Jun 9, 2026

[bug-hunter] get_release_commit.sh returns oldest matching release commit when version repeats #19460

Open

zmoog mentioned this pull request Jun 19, 2026

Expose http timeout setting in Kubernetes package for kube state metrics #19662

Merged

5 tasks

Uh oh!

Conversation

MichaelKatsoulis commented Jun 5, 2026

Proposed commit message

Checklist

Screenshots

Uh oh!

github-actions Bot commented Jun 5, 2026

✅ Elastic Docs Style Checker (Vale)

Uh oh!

This comment has been minimized.

gizas Jun 5, 2026

Choose a reason for hiding this comment

Uh oh!

MichaelKatsoulis Jun 5, 2026

Choose a reason for hiding this comment

Uh oh!

gizas Jun 5, 2026

Choose a reason for hiding this comment

Uh oh!

MichaelKatsoulis Jun 5, 2026

Choose a reason for hiding this comment

Uh oh!

elastic-vault-github-plugin-prod Bot commented Jun 5, 2026

🚀 Benchmarks report

Uh oh!

This comment has been minimized.

github-actions Bot commented Jun 5, 2026

TL;DR

Remediation

Root Cause

Evidence

Verification

Follow-up

Uh oh!

elastic-vault-github-plugin-prod Bot commented Jun 5, 2026

Uh oh!

elasticmachine commented Jun 5, 2026

💚 Build Succeeded

History

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod Bot commented Jun 5, 2026

Uh oh!

elastic-vault-github-plugin-prod Bot commented Jun 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants