Bugs with load balancer [JIRA: CLIENTS-657]

@A1

Connects to basho/riak-clients#1

When customers are using a load balancer, the load balancer can add and remove IP’s dynamically. When an IP is removed, the Riak client fails in two subtle ways.

If the previous connect was created less than DNSTTL timespan ago, the Java client gets another connection. If it is more then a full process of doing a DNS check and getting a connection is followed. If an IP has been dynamically changed in the “less than” timespan, issues arise.
If a domain name is being used, the Java client gets only the IP addresses for it on RiakNode:start. If the load balancer changes IP address, the InetSocketAddress is never updated as the Java client never does another DNS lookup for the domain name case, meaning we never know if the IP address changes.

One customer implemented a fix in their system by changing the doGetConnection() method in the following way:

InetSocketAddress socketAddress = new InetSocketAddress(remoteAddress, port);

bootstrap.handler(new RiakChannelInitializer(this))

.remoteAddress(socketAddress);

Before this call:

ChannelFuture f = bootstrap.connect();

Additionally, the fix included adding the java opt -Dsun.net.inetaddr.ttl=10.

This fix has been tested by @A1 although, of course, other fixes may be possible.

We are currently using Riak in AWS and have been told repeatedly by Basho support to use ELB's in front of our Riak cluster as a best practice (to ease maintenance, to only send traffic to healthy nodes, etc...). Especially in AWS's environment, this defect guarantee's periodic outages of our services - requiring them to be restarted. We raised this same issue with Riak 1.4.x last year and it appears to still not be fixed. When will this be tested and fixed?

Fix is in #573, but the PR needs #594 merged in for some failing tests.

_[posted via JIRA by Alexander Moore]_

1. Problem

If you used the Java Client in a cloud environment where IP addresses for DNS entries could change often, the Java client would not expire the old DNS lookup to get the newest IP address.

2. Solution

Force a refresh any time a connection is made outside of the cluster/node start() method. (Lookup is done once at start).
https://github.com/basho/riak-java-client/pull/573/files#diff-28c20364743c5fc78c7c41c7c8b89e88R293

Also added a log warning if they have their DNS cache setting set to "infinity", which would also cause this issue: https://github.com/basho/riak-java-client/pull/573/files#diff-28c20364743c5fc78c7c41c7c8b89e88R925

3. Test

Setup a "good" mock Netty Bootstrap / ChannelFuture / Channel with a RiakNode
Start RiakNode, should do address lookup Import trifork's PBC Client, Map/Reduce filter support #1.
Fetch/return 1 good connection.
Force the mock to return false for channel.isOpen() when we get the next connection, which should force a second lookup.
Get next connection.
Verify that there were two address lookups.

https://github.com/basho/riak-java-client/pull/573/files#diff-4361a4b26a0905c4782f996ce653ccaaR416

_[posted via JIRA by Alexander Moore]_

Merge pull request #573 from basho/fixes/am/572

Can we please have an update on when this will be released? I see it marked as "Milestone 2.0.3"... But, since you already released 2.0.4, when should we expect this to be released?

Navigation Menu

Repository navigation

Bugs with load balancer [JIRA: CLIENTS-657] #572

1. Problem

2. Solution

3. Test

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Bugs with load balancer [JIRA: CLIENTS-657] #572

Description

Activity

allen-servedio commented on Feb 14, 2016

Basho-JIRA commented on Mar 1, 2016

Basho-JIRA commented on Mar 2, 2016

1. Problem

2. Solution

3. Test

allen-servedio commented on Mar 11, 2016

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions