Repository navigation
Exceptions can block forever when minConnections is set to 1 [JIRA: CLIENTS-501] #523
Description
Activity
- changed the title
[-]Exceptions can block forever when minConnections is set to 1[/-][+]Exceptions can block forever when minConnections is set to 1 [JIRA: CLIENTS-501][/+]on Jun 30, 2015 FYI I can still see this with riak-client 2.0.2.
Hi @hankipanky,
This appears to be a connection/thread contention issue / race condition when you are doing UpdateValue commands in combination with having only a few connections available. Basically we're doing a no-no and trying to run a blocking operation on the Netty worker thread.
- Netty runs our Update Operation, and calls our response listener from the Netty worker thread.
We get our Fetch half the Update operation, update the object, and try to store it back (Note that we're still on the Netty Worker Thread). - We were able to get a permit, and try to get a connection, but in the process we block our Netty Worker Thread: https://github.com/basho/riak-java-client/blob/develop/src/main/java/com/basho/riak/client/core/RiakNode.java#L671
- Netty notices that we did that and throws a BlockingOperationException.
Since it's on the Netty thread, the exception will never show up on the thread where you're waiting for your UpdateOperation to complete on.
Do you use "block on max connections" at all? What's your maxconnections setting set to?
Thanks,
Alex- Netty runs our Update Operation, and calls our response listener from the Netty worker thread.
Hi Alex,
thanks for your response.
Do you use "block on max connections" at all? What's your maxconnections setting set to?
Initially I observed the issue sporadically, when connecting with default values:
RiakNode riakNode = new RiakNode.Builder() .withConnectionTimeout(5000) .withRemoteAddress(Config.getInstance().getRiakAddress()) .withRemotePort(Config.getInstance().getRiakPort()) .build();I noticed lowering maxConnections produces the error faster, which is what I am using it for. Is there a recommendation to ever set maxConnections?
(FYI, I'm not sure if this is relevant: Currently I've only ever run this code against a single-node Riak-Cluster running on the same development box. Eventually it will run against multi-node clusters.)
I don't need
blockOnMaxConnections(true). My application is expecting operations to fail at any moment, which is why I am so confused. CallingRiakClient.execute(), I expect either a response or an exception. From the javadoc:Execute a RiakCommand synchronously.
Calling this method causes the client to execute the provided RiakCommand synchronously. It will block until the operation completes then either return the response on success or throw an exception on failure.But the observation is that my thread waits forever for the operation to complete.
Since it's on the Netty thread, the exception will never show up on the thread where you're waiting for your UpdateOperation to complete on.
Is this an issue in the riak-java-client?
Am I doing something wrong?Thanks, Henning
Hi Henning,
Sorry for the delay.
Is this an issue in the riak-java-client?
Yes.
Am I doing something wrong?
Probably not.
I noticed lowering maxConnections produces the error faster, which is what I am using it for. Is there a recommendation to ever set maxConnections?
You should set maxConnections if you need to tightly control the number of connections to your Riak Cluster (network reasons), or if your clients are overloading the cluster with too many connections.
SideNote - With the overload scenario Riak would refuse connections once it's limit is reached, or it would start giving out overload errors. This would be symptom of too many long-running operations, or an underprovisioned cluster. We're not seeing this here though 😃.
But the observation is that my thread waits forever for the operation to complete.
Yes, this is what the bug is. An UpdateValue Command is really 2 commands put in one: a fetch, and then a store. We receive the fetch response in a piece of code that runs on Netty's threads, and then try to open a connection for the store in that same thread.
If we are out of connections at that very moment, the thread will block, which causes Netty to throw a BlockingOperationException. This exception blocks the Netty thread, and never gets back to the user thread, so the future never completes.I'm still looking into the best way to fix this, but for now you could try increasing your min connections more, or avoiding UpdateValue commands.
Thanks,
Alex@hankipanky Do you still have the code to replicate this behavior? Looking to get a fix in for this soon and having trouble making Netty do bad things :)
Hi Alex,sorry, am on vacation at the moment.On 10.11.2015, at 18:42, Alex Moore [email protected] wrote:@hankipanky Do you still have the code to replicate this behavior? Looking to get a fix in for this soon and having trouble making Netty do bad things :)—Reply to this email directly or view it on GitHub.
Hi Alex,Sorry am on vacation at the moment, i'll check the code when I get back. Sorry for the double posting. Cheers, Hank Sent from my BlackBerry 10 smartphone. From: Alex MooreSent: Dienstag, 10. November 2015 21:42To: basho/riak-java-clientReply To: basho/riak-java-clientCc: Henning VerbeekSubject: Re: [riak-java-client] Exceptions can block forever when minConnections is set to 1 [JIRA: CLIENTS-501] (#523)@hankipanky Do you still have the code to replicate this behavior? Looking to get a fix in for this soon and having trouble making Netty do bad things :)—Reply to this email directly or view it on GitHub.
Hi guys,
We are seeing that same behavior in our application. Is there any update on this issue?
I don't have code to reproduce the issue in isolation yet, but I would gladly try to help if you have time to solve this.
Thanks,
Sebastian.
As reported by @hankipanky on the riak-users mailing list.