Repository navigation
Overload leads to BlockingOperationException [JIRA: CLIENTS-925] #638
Description
Activity
- changed the title
[-]Overload leads to BlockingOperationException[/-][+]Overload leads to BlockingOperationException [JIRA: CLIENTS-925][/+]on Jul 15, 2016 As I'm loadtesting my application, trying to find a sweetspot, I notice that limitting
maxConnectionsalone doesn't work. Sometimes I get BlockingOperationExceptions, and there are only about 40 connections established. Sometimes way more than 100 connections are established, and the loadtest goes through fine.My current conclusion is: I'd have to set maxConnections (what feels like) really really low to be guaranteed safe, but I'd probably be sacrificing performance, right?
@hankipanky How many nodes do you have the client configured to talk to in your tests?
In our architecture, we place one instance of our (Riak-CS-like) application colocated with / in front of each Riak node, and loadbalance over them with HAproxy.
For testing purposes, I use a single-node cluster on a (relatively meager) VM on my development machine.
We have since done a number of things to reduce likelihood of an overload (reduce size of chunks that are stored in riak; pre-start around 30 connections to Riak; reduce HTTP threadpool size of the application, to queue requests better; etc).
The (probably naive) point of this ticket is though: through bad configuration or silly loadtesting, I'll probably still find a way to get Riak into overload. I'd prefer if that results in a
RiakException(just like in #523), which I can react to.Ok, so it seems like we're not catching a few exceptions where we should be. Since the UpdateValue's Store operation runs on the Netty worker thread, we can get into trouble if it decides it's "deadlocked", when actually we're waiting on either a channel permit, or a channel.
The two areas/ blocking operations that I can see us getting hung up on are:
I believe if I catch
BlockingOperationExceptionin those two blocks and send it down the normal error path, it should fix this. If it fails it would then go through the retrier code path, and eventually spit out aRiakException.@hankipanky I released a fix for the "BlockingOperationException" issue with 2.0.7, let me know if you still see the issue with overload. If so I'll have to write more complicated tests.
Thanks so much. I was following the progress and looked at the changeset - looks great. I'll report back after testing. you guys rock!
Riak Java Client: 2.0.6
Java version: 1.8.0_74
Riak version: 2.1.2
Operating system: Linux, Debian 7
When updating objects in Riak under load, the riak-java-client occasionally throws a io.netty.util.concurrent.BlockingOperationException. Since #523 is fixed, this happens only in - what I assume is - an overload-scenario.
The exception:
Then shortly thereafter:
The result is a hanging thread, never returning from the call to
RiakClient.execute(). It would be better, if in such a case, the client threw a RiakException (as javadoc says).Riak's error.log shows nothing about this.
In the meantime, how do I protect against overloading Riak? Setting
RiakNode.Builder().withMaxConnections()to something like 60 or below (on my single-node dev system here) seems to work: Attempts to open more connections lead to acom.basho.riak.client.core.NoNodesAvailableException. What is considered overload?