Repository navigation
test(ci): seed branch to hunt and fix E2E flakes on main - #2919
IvanHunters wants to merge 14 commits into
Conversation
No-op trailing newline in hack/cozytest.sh to trigger the full Pull Request workflow on a clean main, so subsequent commits on this branch can iterate on real E2E flakes. Signed-off-by: Ivan Okhotnikov <[email protected]>
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThree independent functional areas are enhanced: (1) Shell script diagnostics with source-watcher collection and trailing whitespace, (2) SeaweedFS Helm chart fixes for S3 TLS args decoupled from httpsPort, COSI provisioner service account naming, S3 service template updates applied via patches and tested, plus bucket class enhancements with object-lock and access policies, (3) OpenAPI schema sanitization to neutralize boolean-form additionalProperties that crash Kubernetes validation admission, with comprehensive test coverage. ChangesHack directory updates
SeaweedFS Helm chart fixes and enhancements
OpenAPI schema sanitization for Kubernetes validation
Sequence DiagramsequenceDiagram
participant Developer
participant UserSchema as User-supplied OpenAPI Schema
participant patchSpec
participant Sanitize as sanitizeBooleanAdditionalProperties
participant markOpen as markOpenObject
participant KubeValidator as Kubernetes Validator
Developer->>UserSchema: Submit schema with boolean additionalProperties
UserSchema->>patchSpec: Parse user schema
patchSpec->>Sanitize: Detect and rewrite boolean additionalProperties
Sanitize->>markOpen: Mark as open with x-kubernetes-preserve-unknown-fields
markOpen->>patchSpec: Return sanitized schema
patchSpec->>KubeValidator: Publish safe schema to admission controller
KubeValidator->>KubeValidator: SchemaDeclType succeeds without nil return
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~40 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
added
size/XS
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces a minimal, no-op change to the CI script to force a fresh execution of the full test suite. The primary objective is to create a controlled environment for identifying and resolving recurring E2E test failures that have been impacting the stability of the main branch. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
dosubot
Bot
added
area/testing
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces minor formatting changes by adding trailing empty lines to the end of the hack/cozytest.sh script. There are no review comments, and I have no feedback to provide.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
Last run exposed 7 PackageSources stuck with Ready=Unknown reason=Progressing for 30 minutes (openbao-application, qdrant-application, vpn-application, vpa, linstor, virtualprivatecloud-application, capi-provider-bootstrap-kubeadm). PackageSourceReconciler copies its Ready condition straight from the owned ArtifactGenerator, so the real culprit lives in source-watcher — but the current cozyreport bundle does not include any source-watcher artifacts, so we cannot tell whether the OCIRepository fetch is the bottleneck, the ArtifactGenerator queue is starved, or something else entirely. Add a dedicated source-watcher section to hack/cozyreport.sh that captures: - all ArtifactGenerators (list + full YAML) - all OCIRepositories (list + full YAML) - source-watcher pod listing and full logs (current and previous) - events in cozy-fluxcd namespace Signed-off-by: Ivan Okhotnikov <[email protected]>
github-actions
Bot
added
size/S
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@hack/cozyreport.sh`:
- Around line 138-151: Harden the shell script by quoting all dynamic variable
expansions and using read with the -r flag. In the block starting with
DIR=$REPORT_DIR/kubernetes/source-watcher through the pod log loop, wrap all
variable expansions in double quotes (e.g., change $DIR to "$DIR" and $POD to
"$POD") to prevent word splitting and globbing issues. Additionally, change the
read command in the while loop from `while read POD` to `while read -r POD` to
prevent backslash interpretation and properly handle raw input from the kubectl
pipe.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: d283d3f3-8335-45c7-b20a-5c663fec1e64
📒 Files selected for processing (1)
hack/cozyreport.sh
| DIR=$REPORT_DIR/kubernetes/source-watcher | ||
| mkdir -p $DIR | ||
| # PackageSource copies its Ready condition straight from ArtifactGenerator.Status, | ||
| # so a stuck PackageSource ultimately means a stuck ArtifactGenerator or its OCIRepository source. | ||
| kubectl get artifactgenerators -A > $DIR/artifactgenerators.txt 2>&1 || true | ||
| kubectl get artifactgenerators -A -o yaml > $DIR/artifactgenerators.yaml 2>&1 || true | ||
| kubectl get ocirepositories -A > $DIR/ocirepositories.txt 2>&1 || true | ||
| kubectl get ocirepositories -A -o yaml > $DIR/ocirepositories.yaml 2>&1 || true | ||
| kubectl get pods -n cozy-fluxcd -l app.kubernetes.io/name=source-watcher -o wide > $DIR/pods.txt 2>&1 || true | ||
| kubectl get pods -n cozy-fluxcd -l app.kubernetes.io/name=source-watcher -o name 2>/dev/null | \ | ||
| while read POD; do | ||
| POD=${POD#pod/} | ||
| kubectl logs -n cozy-fluxcd $POD --all-containers --prefix --tail=2000 > $DIR/$POD.log 2>&1 || true | ||
| kubectl logs -n cozy-fluxcd $POD --all-containers --prefix --tail=2000 --previous > $DIR/$POD-previous.log 2>&1 || true |
There was a problem hiding this comment.
Quote dynamic shell variables and use read -r in the pod log loop.
Line 138 through Line 151 use unquoted expansions ($DIR, $POD) and read without -r, which can cause brittle behavior in shell parsing. Please harden this block.
Proposed patch
-DIR=$REPORT_DIR/kubernetes/source-watcher
-mkdir -p $DIR
+DIR="$REPORT_DIR/kubernetes/source-watcher"
+mkdir -p "$DIR"
@@
-kubectl get artifactgenerators -A > $DIR/artifactgenerators.txt 2>&1 || true
-kubectl get artifactgenerators -A -o yaml > $DIR/artifactgenerators.yaml 2>&1 || true
-kubectl get ocirepositories -A > $DIR/ocirepositories.txt 2>&1 || true
-kubectl get ocirepositories -A -o yaml > $DIR/ocirepositories.yaml 2>&1 || true
-kubectl get pods -n cozy-fluxcd -l app.kubernetes.io/name=source-watcher -o wide > $DIR/pods.txt 2>&1 || true
+kubectl get artifactgenerators -A > "$DIR/artifactgenerators.txt" 2>&1 || true
+kubectl get artifactgenerators -A -o yaml > "$DIR/artifactgenerators.yaml" 2>&1 || true
+kubectl get ocirepositories -A > "$DIR/ocirepositories.txt" 2>&1 || true
+kubectl get ocirepositories -A -o yaml > "$DIR/ocirepositories.yaml" 2>&1 || true
+kubectl get pods -n cozy-fluxcd -l app.kubernetes.io/name=source-watcher -o wide > "$DIR/pods.txt" 2>&1 || true
kubectl get pods -n cozy-fluxcd -l app.kubernetes.io/name=source-watcher -o name 2>/dev/null | \
- while read POD; do
+ while IFS= read -r POD; do
POD=${POD#pod/}
- kubectl logs -n cozy-fluxcd $POD --all-containers --prefix --tail=2000 > $DIR/$POD.log 2>&1 || true
- kubectl logs -n cozy-fluxcd $POD --all-containers --prefix --tail=2000 --previous > $DIR/$POD-previous.log 2>&1 || true
+ kubectl logs -n cozy-fluxcd "$POD" --all-containers --prefix --tail=2000 > "$DIR/$POD.log" 2>&1 || true
+ kubectl logs -n cozy-fluxcd "$POD" --all-containers --prefix --tail=2000 --previous > "$DIR/$POD-previous.log" 2>&1 || true
done📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| DIR=$REPORT_DIR/kubernetes/source-watcher | |
| mkdir -p $DIR | |
| # PackageSource copies its Ready condition straight from ArtifactGenerator.Status, | |
| # so a stuck PackageSource ultimately means a stuck ArtifactGenerator or its OCIRepository source. | |
| kubectl get artifactgenerators -A > $DIR/artifactgenerators.txt 2>&1 || true | |
| kubectl get artifactgenerators -A -o yaml > $DIR/artifactgenerators.yaml 2>&1 || true | |
| kubectl get ocirepositories -A > $DIR/ocirepositories.txt 2>&1 || true | |
| kubectl get ocirepositories -A -o yaml > $DIR/ocirepositories.yaml 2>&1 || true | |
| kubectl get pods -n cozy-fluxcd -l app.kubernetes.io/name=source-watcher -o wide > $DIR/pods.txt 2>&1 || true | |
| kubectl get pods -n cozy-fluxcd -l app.kubernetes.io/name=source-watcher -o name 2>/dev/null | \ | |
| while read POD; do | |
| POD=${POD#pod/} | |
| kubectl logs -n cozy-fluxcd $POD --all-containers --prefix --tail=2000 > $DIR/$POD.log 2>&1 || true | |
| kubectl logs -n cozy-fluxcd $POD --all-containers --prefix --tail=2000 --previous > $DIR/$POD-previous.log 2>&1 || true | |
| DIR="$REPORT_DIR/kubernetes/source-watcher" | |
| mkdir -p "$DIR" | |
| # PackageSource copies its Ready condition straight from ArtifactGenerator.Status, | |
| # so a stuck PackageSource ultimately means a stuck ArtifactGenerator or its OCIRepository source. | |
| kubectl get artifactgenerators -A > "$DIR/artifactgenerators.txt" 2>&1 || true | |
| kubectl get artifactgenerators -A -o yaml > "$DIR/artifactgenerators.yaml" 2>&1 || true | |
| kubectl get ocirepositories -A > "$DIR/ocirepositories.txt" 2>&1 || true | |
| kubectl get ocirepositories -A -o yaml > "$DIR/ocirepositories.yaml" 2>&1 || true | |
| kubectl get pods -n cozy-fluxcd -l app.kubernetes.io/name=source-watcher -o wide > "$DIR/pods.txt" 2>&1 || true | |
| kubectl get pods -n cozy-fluxcd -l app.kubernetes.io/name=source-watcher -o name 2>/dev/null | \ | |
| while IFS= read -r POD; do | |
| POD=${POD#pod/} | |
| kubectl logs -n cozy-fluxcd "$POD" --all-containers --prefix --tail=2000 > "$DIR/$POD.log" 2>&1 || true | |
| kubectl logs -n cozy-fluxcd "$POD" --all-containers --prefix --tail=2000 --previous > "$DIR/$POD-previous.log" 2>&1 || true | |
| done |
🧰 Tools
🪛 Shellcheck (0.11.0)
[info] 139-139: Double quote to prevent globbing and word splitting.
(SC2086)
[info] 142-142: Double quote to prevent globbing and word splitting.
(SC2086)
[info] 143-143: Double quote to prevent globbing and word splitting.
(SC2086)
[info] 144-144: Double quote to prevent globbing and word splitting.
(SC2086)
[info] 145-145: Double quote to prevent globbing and word splitting.
(SC2086)
[info] 146-146: Double quote to prevent globbing and word splitting.
(SC2086)
[info] 148-148: read without -r will mangle backslashes.
(SC2162)
[info] 150-150: Double quote to prevent globbing and word splitting.
(SC2086)
[info] 150-150: Double quote to prevent globbing and word splitting.
(SC2086)
[info] 150-150: Double quote to prevent globbing and word splitting.
(SC2086)
[info] 151-151: Double quote to prevent globbing and word splitting.
(SC2086)
[info] 151-151: Double quote to prevent globbing and word splitting.
(SC2086)
[info] 151-151: Double quote to prevent globbing and word splitting.
(SC2086)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@hack/cozyreport.sh` around lines 138 - 151, Harden the shell script by
quoting all dynamic variable expansions and using read with the -r flag. In the
block starting with DIR=$REPORT_DIR/kubernetes/source-watcher through the pod
log loop, wrap all variable expansions in double quotes (e.g., change $DIR to
"$DIR" and $POD to "$POD") to prevent word splitting and globbing issues.
Additionally, change the read command in the while loop from `while read POD` to
`while read -r POD` to prevent backslash interpretation and properly handle raw
input from the kubectl pipe.
Source: Linters/SAST tools
…not additionalProperties:true
cozystack-api published the free-form ".spec" of aggregated apps.cozystack.io
resources (Tenant, Application, ...) as the boolean `additionalProperties: true`.
The Kubernetes ValidatingAdmissionPolicy status type-checker run by
kube-controller-manager nil-dereferences when it recurses into that node, so as
soon as a VAP matches one of these resources (cozystack-tenant-host-policy on
tenants) KCM CrashLoopBackOffs cluster-wide and stalls reconciliation, failing
cold installs once the 3x install retry was removed.
Emit `x-kubernetes-preserve-unknown-fields: true` instead. It is semantically
equivalent ("arbitrary fields allowed") and the type-checker handles it safely.
All app custom schemas already use the safe object-form additionalProperties,
so patchSpec was the only source of the boolean form.
Adds a regression test that reproduces the exact crash through
k8s.io/apiserver/pkg/cel/openapi.SchemaDeclType (the entry point KCM uses): it
panics on the pre-fix code and passes after the fix.
Fixes #2863
Assisted-By: Claude <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>
Signed-off-by: Ivan Okhotnikov <[email protected]>
…hed spec Addresses review feedback on #2863. The previous fix only converted the open spec wrapper cozystack-api injects, so a user-supplied ApplicationDefinition openAPISchema carrying a boolean additionalProperties — declared explicitly at the top level, or nested under properties/items/allOf/anyOf/oneOf — was republished verbatim and re-triggered the kube-controller-manager VAP type-checker crash. openAPISchema is untrusted input (external-apps and operator-authored definitions flow through unvalidated), so close the whole class rather than only the schemas cozystack ships. Recursively rewrite every boolean-form additionalProperties (nil inner schema): additionalProperties:true becomes x-kubernetes-preserve-unknown-fields:true, additionalProperties:false is dropped. Both JSON forms carry a nil inner schema and crash the type-checker identically; object-form maps are recursed into and left intact. Also publish the schemaless spec from a fresh schema so no inherited $ref or properties leak into the open object. Regression tests cover top-level true/false and nested (properties, items, allOf) boolean forms, asserting the published schema carries no boolean additionalProperties node and does not panic SchemaDeclType. Assisted-By: Claude <[email protected]> Signed-off-by: Myasnikov Daniil <[email protected]> Signed-off-by: Ivan Okhotnikov <[email protected]>
…n sanitizer regression Address review feedback from lexfrei and coderabbitai on pkg/cmd/server/openapi_test.go: TestPatchSpecSanitizesUserSuppliedBooleanAdditionalProperties only exercised the top-level, properties, items, and allOf positions, leaving the one remaining type-checker-reachable sanitizer branch untested: a boolean additionalProperties nested inside an object-form additionalProperties. Deleting that recursion line kept the whole suite green while the input panicked SchemaDeclType again (#2863). Add the nested-under-additionalproperties case (the reachable blocker, verified red->green by mutation) plus anyOf/oneOf/not cases (defense-in-depth: those branches are recursed by the sanitizer but never traversed by SchemaDeclType, so the hasBooleanAdditionalProperties assertion guards them). Rewrite the test docstring to distinguish the type-checker-reachable cases from the defense-in-depth ones, replacing the inaccurate "each case panics SchemaDeclType" note. Assisted-By: Claude <[email protected]> Signed-off-by: Myasnikov Daniil <[email protected]> Signed-off-by: Ivan Okhotnikov <[email protected]>
Address review feedback from gemini-code-assist on pkg/cmd/server/openapi.go:
markOpenObject and sanitizeForV2 initialized s.Extensions with the generic
map[string]any{}; use the named spec.Extensions{} type instead, matching the
kube-openapi types and the generated definitions in zz_generated.openapi.go.
No behavioral change — spec.Extensions is defined as map[string]interface{}.
Assisted-By: Claude <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>
Signed-off-by: Ivan Okhotnikov <[email protected]>
The 4.31 chart bump (#2834) broke E2E for every PR: tenant-root seaweedfs-system never becomes ready. 1. S3 served plaintext. Until 4.05 the -cert.file/-key.file args were rendered whenever enableSecurity was set, so weed s3 served TLS on the main port (8333); 4.31 nested them under 'if s3.httpsPort' and we ship httpsPort=0, so s3 spoke HTTP while the HTTPS probes, the ingress backend-protocol and the COSI sidecar endpoint all expected HTTPS ('server gave HTTP response to HTTPS client'). Restore the 4.05 behaviour via patch. 2. COSI provisioner ServiceAccount mismatch. cosi-service-account.yaml and the ClusterRoleBinding name the SA <serviceAccountName>-objectstorage-provisioner, but the Deployment referenced componentName (<release>-objectstorage-provisioner). When serviceAccountName != release name (tenant-root-seaweedfs vs seaweedfs-system) the Deployment could not create pods ('serviceaccount not found'). Align the Deployment with the SA + binding. Both fixes are vendored as patches re-applied by 'make update', and covered by helm-unittest. Co-Authored-By: Claude <[email protected]> Signed-off-by: Andrei Kvapil <[email protected]> Signed-off-by: Ivan Okhotnikov <[email protected]>
github-actions
Bot
added
size/XL
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@pkg/cmd/server/openapi.go`:
- Around line 115-135: The sanitizeBooleanAdditionalProperties function in the
openapi.go file currently only traverses Properties, Items, combinators (AllOf,
AnyOf, OneOf), and Not fields, but it is missing traversal for
PatternProperties, AdditionalItems.Schema, and Definitions fields that are also
part of spec.SchemaProps. Add recursive sanitization calls for these three
missing fields to ensure boolean-form additionalProperties nested anywhere in a
user-supplied schema cannot bypass sanitization. Additionally, update the
hasBooleanAdditionalProperties test helper function to mirror the same full
coverage by checking these same three additional fields alongside the existing
checks.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: bb819ad0-cd2a-45a9-8666-c6c46a4b6a16
📒 Files selected for processing (8)
packages/system/seaweedfs/Makefilepackages/system/seaweedfs/charts/seaweedfs/templates/cosi/cosi-deployment.yamlpackages/system/seaweedfs/charts/seaweedfs/templates/s3/s3-deployment.yamlpackages/system/seaweedfs/patches/cosi-provisioner-sa-name.patchpackages/system/seaweedfs/patches/s3-tls-main-port.patchpackages/system/seaweedfs/tests/s3_tls_cosi_sa_test.yamlpkg/cmd/server/openapi.gopkg/cmd/server/openapi_test.go
| for k := range s.Properties { | ||
| prop := s.Properties[k] | ||
| sanitizeBooleanAdditionalProperties(&prop) | ||
| s.Properties[k] = prop | ||
| } | ||
| if s.Items != nil { | ||
| sanitizeBooleanAdditionalProperties(s.Items.Schema) | ||
| for i := range s.Items.Schemas { | ||
| sanitizeBooleanAdditionalProperties(&s.Items.Schemas[i]) | ||
| } | ||
| } | ||
| for i := range s.AllOf { | ||
| sanitizeBooleanAdditionalProperties(&s.AllOf[i]) | ||
| } | ||
| for i := range s.AnyOf { | ||
| sanitizeBooleanAdditionalProperties(&s.AnyOf[i]) | ||
| } | ||
| for i := range s.OneOf { | ||
| sanitizeBooleanAdditionalProperties(&s.OneOf[i]) | ||
| } | ||
| sanitizeBooleanAdditionalProperties(s.Not) |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
gopath="$(go env GOPATH 2>/dev/null || true)"
if [ -n "$gopath" ] && [ -d "$gopath/pkg/mod" ]; then
rg -n 'type SchemaProps struct|PatternProperties|AdditionalItems|Definitions' \
"$gopath/pkg/mod/k8s.io/kube-openapi"* \
"$gopath/pkg/mod/github.com/go-openapi/spec"* 2>/dev/null || true
fi
rg -n 'sanitizeBooleanAdditionalProperties|hasBooleanAdditionalProperties|PatternProperties|AdditionalItems|Definitions' --type=goRepository: cozystack/cozystack
Length of output: 5326
🏁 Script executed:
# First, let's look at the sanitizeBooleanAdditionalProperties function in detail
cat -n pkg/cmd/server/openapi.go | head -150 | tail -70Repository: cozystack/cozystack
Length of output: 3204
🏁 Script executed:
# Now let's look at the hasBooleanAdditionalProperties test helper
cat -n pkg/cmd/server/openapi_test.go | sed -n '170,220p'Repository: cozystack/cozystack
Length of output: 2075
🏁 Script executed:
# Let's search for spec.SchemaProps definition to understand the schema structure
rg -A 50 'type SchemaProps struct' --type=go 2>/dev/null | head -80Repository: cozystack/cozystack
Length of output: 45
🏁 Script executed:
# Check imports in openapi.go
head -20 pkg/cmd/server/openapi.goRepository: cozystack/cozystack
Length of output: 637
🏁 Script executed:
# Search for any reference to PatternProperties, AdditionalItems in the codebase
rg 'PatternProperties|AdditionalItems' --type=go -B 2 -A 2Repository: cozystack/cozystack
Length of output: 45
🏁 Script executed:
# Look for schema field assignments to understand what's available
rg 's\.(PatternProperties|AdditionalItems|Definitions)' --type=goRepository: cozystack/cozystack
Length of output: 45
🏁 Script executed:
# Let me search for the test file to see what fields are actually being tested
cat -n pkg/cmd/server/openapi_test.go | head -280 | tail -100Repository: cozystack/cozystack
Length of output: 5058
🏁 Script executed:
# Search for any schema construction to see what fields are used
rg 'spec\.Schema\{' --type=go -A 5 | head -100Repository: cozystack/cozystack
Length of output: 8240
🏁 Script executed:
# Let's check the Definitions field specifically in the test
rg -n 'Definitions' pkg/cmd/server/openapi_test.go -B 2 -A 2Repository: cozystack/cozystack
Length of output: 45
🏁 Script executed:
# Search for web documentation about spec.SchemaProps
web_search
k8s.io kube-openapi spec.SchemaProps fields PatternProperties AdditionalItems DefinitionsRepository: cozystack/cozystack
Length of output: 159
🏁 Script executed:
# Check if there are any comments in the code about which fields are supported
rg -i 'patternproperties|additionalitems|definitions' pkg/cmd/server/ -B 2 -A 2Repository: cozystack/cozystack
Length of output: 3165
🏁 Script executed:
# Look at the full test file to understand what's covered
wc -l pkg/cmd/server/openapi_test.goRepository: cozystack/cozystack
Length of output: 98
🌐 Web query:
k8s.io kube-openapi validation spec SchemaProps struct fields
💡 Result:
In the k8s.io/kube-openapi library, the SchemaProps struct defines the core validation properties for an OpenAPI schema [1]. It is a central component of the Schema struct, which represents an OpenAPI schema definition [1][2]. The SchemaProps struct contains the following fields [1]: - Ref: A reference to another schema definition (internal to the spec) [1]. - Schema: The URI of the schema, often used for identifying the JSON Schema version [1]. - Description: A short description of the field [1]. - Type: Specifies the data type(s) of the schema, represented as a StringOrArray [1]. - Nullable: A boolean indicating if the value can be null [1]. - Format: A string specifying the format of the data (e.g., "date-time", "int64") [1]. - Title: The title of the schema [1]. - Default: The default value for the field [1]. - Maximum / Minimum: Pointers to float64 values defining range boundaries [1]. - ExclusiveMaximum / ExclusiveMinimum: Booleans indicating if the boundaries are exclusive [1]. - MaxLength / MinLength: Pointers to int64 values for string length constraints [1]. - Pattern: A regular expression pattern for string validation [1]. - MaxItems / MinItems: Pointers to int64 values for array length constraints [1]. - UniqueItems: A boolean indicating if array items must be unique [1]. - MultipleOf: A pointer to a float64 value for numeric divisibility [1]. - Enum: A slice of allowed values [1]. - MaxProperties / MinProperties: Pointers to int64 values for object property count constraints [1]. - Required: A slice of strings listing required property names [1]. - Items: A pointer to a SchemaOrArray for array item definitions [1]. - AllOf, OneOf, AnyOf: Slices of Schema objects for composition [1]. - Not: A pointer to a Schema object for exclusion validation [1]. - Properties: A map of string keys to Schema objects defining object properties [1]. - AdditionalProperties: A SchemaOrBool type defining the allowed schema for additional object properties [1]. This structure is used extensively within Kubernetes for generating and validating OpenAPI specifications dynamically [3][4]. Developers typically interact with this struct when defining custom OpenAPI definitions for Go types using the k8s.io/kube-openapi/pkg/common package [4].
Citations:
- 1: https://github.com/kubernetes/kube-openapi/blob/16be699c7b31/pkg/validation/spec/schema.go
- 2: https://github.com/kubernetes-sigs/kustomize/blob/master/cmd/depprobcheck/README.md
- 3: https://pkg.go.dev/k8s.io/kube-openapi/pkg/validation/spec
- 4: https://github.com/kubernetes/kube-openapi/blob/master/pkg/generators/README.md
🏁 Script executed:
# Let's check what version of kube-openapi might be used
rg 'kube-openapi' go.modRepository: cozystack/cozystack
Length of output: 119
🏁 Script executed:
# Check if there are any TODO comments or notes about incomplete coverage
rg -i 'todo|fixme|incomplete|missing' pkg/cmd/server/openapi.go -B 2 -A 2Repository: cozystack/cozystack
Length of output: 45
🌐 Web query:
kube-openapi SchemaProps PatternProperties AdditionalItems Definitions fields
💡 Result:
In the context of the kube-openapi library (and the underlying go-openapi/spec model it frequently references), the SchemaProps struct defines the core properties of a JSON Schema [1][2]. Key fields within SchemaProps include: PatternProperties: A map where keys are regular expressions and values are Schema objects, used to validate properties matching the specified pattern [3]. AdditionalItems: Used for array validation; it specifies the schema for items that do not match the main items definition [3]. In many implementations, it is represented as a JSONSchemaPropsOrBool, allowing either a full schema or a boolean to enable/disable additional items [3]. Definitions: While often part of the broader OpenAPI/JSON Schema structure, definitions act as a map (map[string]JSONSchemaProps) for storing reusable sub-schemas that can be referenced elsewhere via $ref [3]. Additional fields commonly found within the SchemaProps struct include: Properties: A map of property names to their respective Schema definitions [3][1]. Required: A list of strings identifying which properties are mandatory [1]. Items: Defines the schema for elements in an array, represented as a SchemaOrArray type [1]. Type, Format, Description: Basic metadata fields for schema validation and documentation [1]. AllOf, OneOf, AnyOf, Not: Fields used for schema composition and logical constraints [4][1]. The kube-openapi library is primarily used for generating and serving Kubernetes API specifications, often relying on the struct tags defined in these packages to handle JSON marshaling and unmarshaling of Kubernetes resource definitions [5][6]. Note that the ID field in some versions of this struct is included primarily for compatibility with broader JSON Schema specifications (which use 'id' for $ref resolution) even if it is not explicitly part of the standard OpenAPI v2 specification [7][8].
Citations:
- 1: https://github.com/kubernetes/kube-openapi/blob/master/pkg/validation/spec/schema.go
- 2: https://github.com/kubernetes-sigs/kustomize/blob/master/cmd/depprobcheck/README.md
- 3: https://dev-k8sref-io.web.app/docs/common-definitions/jsonschemaprops-/
- 4: https://github.com/kubernetes/kube-openapi/blob/master/pkg/validation/validate/schema.go
- 5: https://pkg.go.dev/k8s.io/kube-openapi/pkg/validation/spec
- 6: https://github.com/kubernetes/kube-openapi/blob/master/pkg/generators/README.md
- 7: Question regarding to SchemaProps.ID go-openapi/spec#23
- 8: https://github.com/go-openapi/spec
🏁 Script executed:
# Let's look at the current test cases to see if they test for the missing fields
cat -n pkg/cmd/server/openapi_test.go | sed -n '230,280p'Repository: cozystack/cozystack
Length of output: 2758
🏁 Script executed:
# Check if spec.SchemaProps actually has these fields by looking at the GitHub source
# Let's search for any usage of these fields in a different context
rg '\.(PatternProperties|AdditionalItems|Definitions)\s*=' --type=go | head -20Repository: cozystack/cozystack
Length of output: 45
🏁 Script executed:
# Let's check the actual kube-openapi source to verify the fields exist
# by looking at the GitHub repository for that specific version
python3 << 'PY'
# The version string suggests it's a pseudo-version pointing to a specific commit
# Let's parse what we know and search for confirmation in docs or code
version_str = "v0.0.0-20250814151709-d7b6acb124c3"
# Extract timestamp: 20250814151709 = 2025-08-14 15:17:09
# Extract commit: d7b6acb124c3
print("kube-openapi version: " + version_str)
print("Date: 2025-08-14")
print("Commit: d7b6acb124c3")
PYRepository: cozystack/cozystack
Length of output: 158
🏁 Script executed:
# Let's verify by checking if the code even compiles/has references to these fields
# Look for any schema building code that might use these fields
rg 'spec\.Schema' pkg/cmd/server/ -A 3 | grep -E '(PatternProperties|AdditionalItems|Definitions)' | head -20Repository: cozystack/cozystack
Length of output: 45
🏁 Script executed:
# Let's examine the full sanitizeBooleanAdditionalProperties and hasBooleanAdditionalProperties side by side
echo "=== sanitizeBooleanAdditionalProperties ===" && \
sed -n '102,136p' pkg/cmd/server/openapi.go && \
echo "" && \
echo "=== hasBooleanAdditionalProperties ===" && \
sed -n '174,210p' pkg/cmd/server/openapi_test.goRepository: cozystack/cozystack
Length of output: 1975
Extend schema traversal to cover PatternProperties, AdditionalItems, and Definitions fields.
The sanitizer currently traverses properties, items, additionalProperties.Schema, combinators, and not, but the underlying kube-openapi spec.SchemaProps struct also exposes patternProperties, additionalItems.Schema, and definitions fields. A user-supplied schema with boolean-form additionalProperties nested under any of these uncovered fields would bypass sanitization and still be marshaled into served OpenAPI, leaving the nil-inner-schema shape this PR is trying to eliminate. Update both the sanitizer and the hasBooleanAdditionalProperties test helper to mirror the full coverage.
Suggested recursion coverage
if s.Items != nil {
sanitizeBooleanAdditionalProperties(s.Items.Schema)
for i := range s.Items.Schemas {
sanitizeBooleanAdditionalProperties(&s.Items.Schemas[i])
}
}
+ if s.AdditionalItems != nil && s.AdditionalItems.Schema != nil {
+ sanitizeBooleanAdditionalProperties(s.AdditionalItems.Schema)
+ }
+ for k := range s.PatternProperties {
+ prop := s.PatternProperties[k]
+ sanitizeBooleanAdditionalProperties(&prop)
+ s.PatternProperties[k] = prop
+ }
+ for k := range s.Definitions {
+ def := s.Definitions[k]
+ sanitizeBooleanAdditionalProperties(&def)
+ s.Definitions[k] = def
+ }
for i := range s.AllOf {
sanitizeBooleanAdditionalProperties(&s.AllOf[i])
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| for k := range s.Properties { | |
| prop := s.Properties[k] | |
| sanitizeBooleanAdditionalProperties(&prop) | |
| s.Properties[k] = prop | |
| } | |
| if s.Items != nil { | |
| sanitizeBooleanAdditionalProperties(s.Items.Schema) | |
| for i := range s.Items.Schemas { | |
| sanitizeBooleanAdditionalProperties(&s.Items.Schemas[i]) | |
| } | |
| } | |
| for i := range s.AllOf { | |
| sanitizeBooleanAdditionalProperties(&s.AllOf[i]) | |
| } | |
| for i := range s.AnyOf { | |
| sanitizeBooleanAdditionalProperties(&s.AnyOf[i]) | |
| } | |
| for i := range s.OneOf { | |
| sanitizeBooleanAdditionalProperties(&s.OneOf[i]) | |
| } | |
| sanitizeBooleanAdditionalProperties(s.Not) | |
| for k := range s.Properties { | |
| prop := s.Properties[k] | |
| sanitizeBooleanAdditionalProperties(&prop) | |
| s.Properties[k] = prop | |
| } | |
| if s.Items != nil { | |
| sanitizeBooleanAdditionalProperties(s.Items.Schema) | |
| for i := range s.Items.Schemas { | |
| sanitizeBooleanAdditionalProperties(&s.Items.Schemas[i]) | |
| } | |
| } | |
| if s.AdditionalItems != nil && s.AdditionalItems.Schema != nil { | |
| sanitizeBooleanAdditionalProperties(s.AdditionalItems.Schema) | |
| } | |
| for k := range s.PatternProperties { | |
| prop := s.PatternProperties[k] | |
| sanitizeBooleanAdditionalProperties(&prop) | |
| s.PatternProperties[k] = prop | |
| } | |
| for k := range s.Definitions { | |
| def := s.Definitions[k] | |
| sanitizeBooleanAdditionalProperties(&def) | |
| s.Definitions[k] = def | |
| } | |
| for i := range s.AllOf { | |
| sanitizeBooleanAdditionalProperties(&s.AllOf[i]) | |
| } | |
| for i := range s.AnyOf { | |
| sanitizeBooleanAdditionalProperties(&s.AnyOf[i]) | |
| } | |
| for i := range s.OneOf { | |
| sanitizeBooleanAdditionalProperties(&s.OneOf[i]) | |
| } | |
| sanitizeBooleanAdditionalProperties(s.Not) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@pkg/cmd/server/openapi.go` around lines 115 - 135, The
sanitizeBooleanAdditionalProperties function in the openapi.go file currently
only traverses Properties, Items, combinators (AllOf, AnyOf, OneOf), and Not
fields, but it is missing traversal for PatternProperties,
AdditionalItems.Schema, and Definitions fields that are also part of
spec.SchemaProps. Add recursive sanitization calls for these three missing
fields to ensure boolean-form additionalProperties nested anywhere in a
user-supplied schema cannot bypass sanitization. Additionally, update the
hasBooleanAdditionalProperties test helper function to mirror the same full
coverage by checking these same three additional fields alongside the existing
checks.
…lass The seaweedfs 4.31 upstream bump (#2834) rewrote packages/system/seaweedfs/charts/seaweedfs/templates/cosi/cosi-bucket-class.yaml from scratch and silently dropped the supplementary COSI resources that cozystack ships on top of the upstream chart: - BucketClass {name}-lock (object lock, COMPLIANCE mode, 365 days retention) — used by apps that store immutable backups - BucketAccessClass {name} parameters.accessPolicy: readwrite — the explicit grant for the default access class - BucketAccessClass {name}-readonly (accessPolicy: readonly) — used by every Bucket app that declares readonly users apps/bucket still renders BucketAccess resources that reference {name}-readonly, but the class no longer exists, so the COSI controller never picks the BucketAccess up: no finalizer is added, status.accessGranted stays empty, and the bucket E2E test waits 300s and times out on the viewer access. The admin (readwrite) access class still works because it falls back to the default BAC. Restore the three resources so the system chart again matches the shape that packages/extra/seaweedfs/templates/client/cosi-bucket-class.yaml already ships for the client topology (where this never regressed). Signed-off-by: Ivan Okhotnikov <[email protected]>
The seaweedfs 4.31 upstream bump (#2834) switched the s3 Service name from "$chartname-s3" (built with the seaweedfs.name helper, evaluates to "seaweedfs-s3") to the seaweedfs.componentName helper, which is release-name based and now evaluates to "$release-s3" — "seaweedfs- system-s3" for the tenant-root deployment. The bats E2E asset hack/e2e-apps/bucket.bats hard-codes "service/seaweedfs-s3" for the port-forward that sets up its S3 client; backup-controller and other in-cluster consumers also referenced the legacy name. After the bump, the legacy Service is gone and the port-forward immediately exits, the test then times out on the 30-second nc poll for localhost:8333. Restore the legacy Service name. Internal references inside the chart (s3-deployment -filer=, etc) keep using componentName for the filer endpoint, so this change only re-exposes the public-facing name without touching the internal topology. Signed-off-by: Ivan Okhotnikov <[email protected]>
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/system/seaweedfs/charts/seaweedfs/templates/s3/s3-service.yaml`:
- Line 5: The change to the name field in the s3-service.yaml file has been
committed directly to the vendored SeaweedFS Helm chart content, but this will
be overwritten when make update re-vendors the upstream charts. Instead, create
a new patch file in packages/system/seaweedfs/patches/ (for example,
s3-service-name.patch) that contains this change to the template name field,
following the established pattern of existing patches like
cosi-provisioner-sa-name.patch and s3-tls-main-port.patch. Then restore the
vendored s3-service.yaml template file back to its upstream state so the fix is
applied via the patch file during the chart update process rather than as a
direct modification.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 76973cd4-cd9b-48e7-b606-d3486f0f5b87
📒 Files selected for processing (1)
packages/system/seaweedfs/charts/seaweedfs/templates/s3/s3-service.yaml
| kind: Service | ||
| metadata: | ||
| name: {{ include "seaweedfs.componentName" (list . "s3") }} | ||
| name: {{ template "seaweedfs.name" . }}-s3 |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Verify the S3 service name change is applied via patch
# Check if a patch exists for the s3-service.yaml file
echo "=== Checking for patches that modify s3-service.yaml ==="
fd -t f '\.patch$' packages/system/seaweedfs/patches/ --exec rg -l 's3-service\.yaml'
# Check the Makefile for patch application
echo -e "\n=== Checking Makefile for patch application ==="
rg -A5 -B5 'patch|s3-service' packages/system/seaweedfs/Makefile
# Verify git status shows this file is not directly modified
echo -e "\n=== Git status of the vendored chart file ==="
git status packages/system/seaweedfs/charts/seaweedfs/templates/s3/s3-service.yamlRepository: cozystack/cozystack
Length of output: 1022
🏁 Script executed:
#!/bin/bash
# Get git diff for the s3-service.yaml file and check patch context
echo "=== Git diff for s3-service.yaml ==="
git diff HEAD packages/system/seaweedfs/charts/seaweedfs/templates/s3/s3-service.yaml | head -50
echo -e "\n=== All patch files in packages/system/seaweedfs/patches/ ==="
ls -la packages/system/seaweedfs/patches/
echo -e "\n=== Content of patches directory ==="
find packages/system/seaweedfs/patches/ -type f -exec echo "File: {}" \; -exec head -5 {} \;
echo -e "\n=== Check if s3-service.yaml is in any patch ==="
grep -r "s3-service" packages/system/seaweedfs/patches/ 2>/dev/null || echo "No matches found"
echo -e "\n=== Check git log for this file ==="
git log --oneline -n 5 packages/system/seaweedfs/charts/seaweedfs/templates/s3/s3-service.yamlRepository: cozystack/cozystack
Length of output: 3216
Extract this change into a proper patch file rather than committing directly to vendored content.
The s3-service.yaml change is currently committed directly to the vendored SeaweedFS Helm chart, but it should be applied via a patch file following the established pattern in packages/system/seaweedfs/patches/. The Makefile already applies 5 patches during the update process (cosi-provisioner-sa-name.patch, disable-ca-key-rotation.patch, resize-api-server-annotation.diff, s3-tls-main-port.patch, and retention-policy-delete.yaml). This direct edit will be overwritten when make update re-vendors the upstream charts.
Create a new patch file (e.g., packages/system/seaweedfs/patches/s3-service-name.patch) with this change and restore the vendored template file to upstream state. This ensures the fix persists across chart updates and maintains the architectural pattern for managing vendored content.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/system/seaweedfs/charts/seaweedfs/templates/s3/s3-service.yaml` at
line 5, The change to the name field in the s3-service.yaml file has been
committed directly to the vendored SeaweedFS Helm chart content, but this will
be overwritten when make update re-vendors the upstream charts. Instead, create
a new patch file in packages/system/seaweedfs/patches/ (for example,
s3-service-name.patch) that contains this change to the template name field,
following the established pattern of existing patches like
cosi-provisioner-sa-name.patch and s3-tls-main-port.patch. Then restore the
vendored s3-service.yaml template file back to its upstream state so the fix is
applied via the patch file during the chart update process rather than as a
direct modification.
Source: Learnings
`kubectl port-forward service/seaweedfs-s3 -n tenant-root 8333:8333` binds the listener to 127.0.0.1 only — it has no IPv6 listener. The runner's getaddrinfo resolves `localhost` to `::1` first, and `mc` keeps a single connection target across the whole alias, so the initial `mc alias set` (which only sets credentials, no network I/O) silently succeeds while the subsequent `mc cp` upload tries `https://[::1]:8333` and gets immediate `connection refused`. The test then exits with the misleading bucket-test failure even though the S3 backend, credentials, and bucket all reconciled correctly. Use a literal IPv4 address (`127.0.0.1`) for the mc aliases and the `nc -z` readiness probe so the test no longer depends on the runner's IPv4/IPv6 resolution policy. Signed-off-by: Ivan Okhotnikov <[email protected]>
The test used to wrap kubectl port-forward in a transient subshell: bash -c 'timeout 100s kubectl port-forward ... > /dev/null 2>&1 &' The wrapper subshell exits immediately after backgrounding the kubectl process, the kubectl child is reparented to PID 1, and on the GitHub-hosted runner it gets reaped between the `nc -z` readiness probe and the first `mc cp` call. The captured failure looks like + bash -c timeout 100s kubectl port-forward ... + timeout 30 sh -ec until nc -z localhost 8333; do sleep 1; done + mc alias set rw-user https://127.0.0.1:8333 ... --insecure Added `rw-user` successfully. + mc cp --insecure /tmp/rw-test.txt rw-user/$BUCKET/rw-test.txt mc: <ERROR> Put "https://127.0.0.1:8333/$BUCKET/...": dial tcp 127.0.0.1:8333: connect: connection refused even though every S3 resource (Bucket, BucketClaim, both BucketAccess) is Ready/Granted=true at that point and the seaweedfs-s3 Service is healthy in tenant-root. Run kubectl port-forward as a job of the calling shell, capture its PID, and tear it down via a bats RETURN trap. The trap is enough on its own: the test no longer needs the outer `timeout 100s` belt because the port-forward will be killed when the test function returns, even on failure. Also switch the readiness probe to 127.0.0.1 to match the mc alias and avoid any IPv6 resolution path on hosts where kubectl binds IPv4 only. Signed-off-by: Ivan Okhotnikov <[email protected]>
The previous attempt added a bash `trap '...' RETURN` to clean up kubectl port-forward when the test function returned. cozytest.sh sources its converted bats test bodies into the runner shell, which on the GitHub-hosted Linux runner is /bin/sh -> dash. dash does not implement the bash-specific RETURN pseudo-signal, so the trap command exits non-zero, the surrounding `set -eu` aborts the test body immediately, and the captured output stops two lines after `name=test` with no other diagnostic — the test never reaches the mc S3 operations at all. Drop the trap and rely on cozytest.sh's existing process-group cleanup: when the test function returns, the leftover port-forward job in the same group is terminated. This restores POSIX shell compatibility while keeping the lifecycle fix from the previous commit (no `bash -c '... &'` wrapper, port-forward stays alive through the mc operations because it is a child of the calling shell, not of a transient subshell that exits immediately). Signed-off-by: Ivan Okhotnikov <[email protected]>
Cozystack overrides the upstream seaweedfs volumeSizeLimitMB from 1000 to 30000 (30 GB), but the e2e sandbox provisions only a 10 GiB PVC per volume server. With maxVolumes=0 (auto), seaweedfs computes maxVolumes as int(10240 / 30000) → 0, then rounds up to 1, so each volume server can host exactly 1 logical volume. With defaultReplication=001 (one replica on a different node within the same rack) two volume servers share that single volume, giving the cluster a hard limit of one writable collection at a time. Since #2716 added a default bucket-cozy-backups Bucket to every tenant-root, that bucket now grabs the only slot on install. Any second collection — the bucket E2E (bucket-test in tenant-test), a tenant app's own bucket, etc. — sits in 'assign volume: No writable volumes' forever, and mc cp hangs until the test times out. Master logs reflect this clearly: failed to find writable volumes for collection:bucket-... replication:001 ttl: error: No writable volumes and no free volumes left topo failed to pick 1 from 0 node candidates volume_growth.go:135 create 6 volume, created 0: Not enough data nodes found! Lower volumeSizeLimitMB back to the upstream chart default of 1000, which makes a 10 GiB PVC fit ~10 logical volumes per server and unblocks all per-collection bucket workloads on small sandboxes. Production clusters still get plenty of headroom — volume size is a soft cap, not a preallocation. Signed-off-by: Ivan Okhotnikov <[email protected]>
A churn-heavy platform install intermittently wedges a pod in
ContainerCreating because the cilium-agent rejects its sandbox with
'[PUT /endpoint/{id}][400] putEndpointIdInvalid "IP ipv4:<X> is already
in use"', repeated until the agent restarts or the install times out.
The disputed IP has no live owner: IPAM and the ipcache released it, but
the agent's in-memory endpointManager still holds a stale Endpoint from a
previously-deleted pod whose CNI DEL was skipped or raced (so
removeReferencesLocked never ran). Neither the operator's CiliumEndpoint
GC nor the agent's veth-based endpoint GC can reap it. This surfaced as
~6 of the post-#2558 e2e failures, each on a different victim pod
(velero node-agent, fluent-bit, fdb-operator, cdi-operator, capi
bootstrap), failing PRs unrelated to the change under test. Tracked
upstream at cilium/cilium#38313 (closed stale); no released or
pre-release Cilium fixes it and no flag disables it.
hack/e2e-cilium-endpoint-leak-healer.sh runs as a file-scoped background
watchdog (setup_file/teardown_file). It detects a wedged pod, confirms
the IP is a true orphan -- an endpoint in the owning node's registry
holds it, that endpoint backs no live Running pod, and a different pod is
wedged requesting the same IP -- then evicts only that endpoint via
'cilium-dbg endpoint disconnect ipv4:<X>' (DELETE /endpoint/{id} ->
unexpose -> removeReferencesLocked). Blast radius is one dead endpoint,
not an agent restart. It is read-only until an orphan is positively
confirmed and refuses (loudly) to touch an endpoint backing a live pod,
so a genuine duplicate-IP bug stays visible rather than masked.
This is an interim CI mitigation, not a product fix; remove the script
and the bats hooks once a fixed Cilium ships.
Assisted-By: Claude <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>
Signed-off-by: Ivan Okhotnikov <[email protected]>
3 participants
What this PR does
Seed commit that triggers the full Pull Request workflow on a clean
main,so we can iterate on real E2E flakes here without mixing them with any
in-flight feature changes. The first commit adds two trailing blank lines
to
hack/cozytest.sh— a no-op, only there to make the paths filter pickthe change up (anything outside
docs/**is enough).Context: since
2026-06-10T11:00Z, no non-docs PR onmainhas produceda green E2E run. The dominant symptoms across recent failures are:
Install Cozystack into sandboxstep times out at 15:50, with a widefan of HelmReleases never reaching Ready (
kafka-operator,kamaji,kubeovn,kubevirt,linstor,cert-manager, ...).bucketclaim/bucket-cozy-backupsfails withbucketclasses.objectstorage.k8s.io "tenant-root" not found— racebetween
BucketClasscreation and theBucketClaimthat references it.strimzi-cluster-operator(kafka-operator) crashloops on a livenessprobe to
:8080, blockingkafka-rd.Create and Verify Seeweedfs Bucketconsistently timesout on
kubectl wait hr bucket-test --timeout=100s --for=condition=ready.Subsequent commits on this branch will be targeted fixes for whichever of
these (or others) reproduce on the first run, one fix per commit so they
can be cherry-picked or reviewed individually.
Screenshots
N/A — no UI changes.
Release note
```release-note
NONE
```
Summary by CodeRabbit
additionalProperties, and ensured open.specschemas are published when needed.serviceAccountName; also adjusted the generated S3 Service naming.