Repository navigation
Readiness check supports configuring minimum number of ready instances #1375
Description
Feature Description
- Allow specify a number of least necessary connected cloud-sql replica for unhealthy condition.
Alternatives Considered
- Create new cloud-sql-proxy for each cloud-sql (read) replica and make special script for checking ready cloud-sql-proxies.
- it need to up a lot of other cloud-sql-proxy instances.
- it mess up readiness probe definitions.
Additional Context
- If you are using multiple cloud-sql replicas, you can use
query retrystrategies in your application code.-
- try to query for 1st instance in the (randomized) list.
-
- if it doesn't work, retry the query in 2nd instance in the list. (continue to last of the list)
-
- when correct response has come, you can use the values.
-
- Such query strategy is making our application robust for downing some replicas. (it will avoid a read replica will be a single point of failure)
- But, current cloud-sql-proxy's readiness check will be unhealthy if single node was down.
- In Kubernetes, this behavior results the Pod's down if single cloud-sql replica has down the pod cannot handle all requests regardless our application can response it (by using another replica.)
- This means, the retry strategy is not working with cloud-sql-proxy and a single read replica will be a single point of failure.
- If there is this feature, we can use this for avoiding such system down.
Activity
enocom commentedon Sep 5, 2022
Thanks for the feature request, @rugamaga. This is something we considered when writing the initial readiness check. So to briefly summarize, are you suggesting the Cloud SQL Proxy support a configurable readiness check? In other words, maybe we allow callers to specify how many instances can fail before the readiness check fails?
rugamaga commentedon Sep 5, 2022
@enocom Yes. If you can specify how many instances can fail before the readiness check fails, it would be useful for making it robust.
enocom commentedon Sep 6, 2022
How about something like:
—readiness-check-threshold=n
where n is the number of instances which can fail before the check fails?
rugamaga commentedon Sep 6, 2022
That's nice. but in this context, I think we want to check
minimum alived instances.so better option is
--min-ready-instances=nwhere n is ready instances. for example, when n =2, cloud-sql-proxy will respondreadyif over than two instances are alived.enocom commentedon Sep 6, 2022
OK, got it. I think that's a pretty simple fix. I'll pull this into my queue.
[-]Response ready when connected with over than specified number replica instances[/-][+]Readiness check supports configuring number of ready instances[/+][-]Readiness check supports configuring number of ready instances[/-][+]Readiness check supports configuring minimum number of ready instances[/+]feat: add support for min ready instances
feat: add support for min ready instances
enocom commentedon Oct 21, 2022
I decided to add support for a
min-readyquery param instead of adding another CLI flag. See #1496 for details.feat: add support for min ready instances
feat: add support for min ready instances (#1496)