This repository was archived by the owner on Aug 13, 2026. It is now read-only.
Repository navigation
This repository was archived by the owner on Aug 13, 2026. It is now read-only.
Webapp classloader leaks problem with RiakNode.Builder or RiakClient [JIRA: CLIENTS-962] #660
Description
I'm using Riak java client (2.0.4) for our jax-rs services and I've encountered permgen OOM leak when reloading services.
I've created an example, just using this Listener (and uncomment cluster/client code) for any webapps, and seeing permgen space increase when reloading services (at least 2 times).
public class ContextListener implements ServletContextListener {
public static RiakCluster cluster = null;
public static RiakClient client = null;
@Override
public void contextInitialized(ServletContextEvent sce) {
try {
// RiakNode.Builder have perm gen leak when initialize any instance
RiakNode.Builder builder = null;
// builder = new RiakNode.Builder();
List<String> addresses = new LinkedList<String>();
addresses.addAll(Arrays.asList("anyip:port".split(",")));
// or memleak when call buildNodes also
List<RiakNode> nodes = null;
// nodes = RiakNode.Builder.buildNodes(builder, addresses);
// cluster = new RiakCluster.Builder(nodes).build();
// cluster.start();
// client = RiakClient.newClient();
} catch (Exception e) {
}
}
@Override
public void contextDestroyed(ServletContextEvent sce) {
if (cluster != null)
cluster.shutdown();
if (client !=null)
client.shutdown();
}
}
Thanks,
Cong Nguyen
Activity
[-]Webapp classloader leaks problem with RiakNode.Builder or RiakClient[/-][+]Webapp classloader leaks problem with RiakNode.Builder or RiakClient [JIRA: CLIENTS-962][/+]alexmoore commentedon Aug 19, 2016
Hi Cong,
What happens if you run
await()on theFuture<Boolean>thatcluster.shutdown()andclient.shutdown()return?alexmoore commentedon Aug 19, 2016
@monday0rsunday Also, do you have a minimum reproducing java project you could share to help speed up bug hunting?
monday0rsunday commentedon Aug 21, 2016
@alexmoore see https://github.com/monday0rsunday/riak_perm
alexmoore commentedon Sep 14, 2016
So after running the refresh a few times, I see some extra WebAppClassLoaders sitting on the Heap:
If we dig down and "Merge Shortest Paths to GC Roots", it shows something on the Jetty Scanner TimerThread, but no big red flags.
Do you know if you can reproduce this with a console app, or another webapp container?
alexmoore commentedon Sep 19, 2016
@monday0rsunday What version of Java are you running?
alexmoore commentedon Sep 20, 2016
@monday0rsunday Aha! I think I found our culprit. It looks like the object that gets stuck in memory is one
io.netty.util.internal.IternalThreadLocalMapobject, which isThreadLocal. It looks like this is a known issue that the Netty team has no control over. I'll see if there's a workaround...monday0rsunday commentedon Sep 20, 2016
@alexmoore I'm not sure about version exactly, but there are two major versions I tested: 7 and 8.
alexmoore commentedon Sep 20, 2016
@monday0rsunday One workaround is to add the following to your
contextDestroyed(..)method after you callclient.shutdown.get();:This seems to allow the most PermGen/Metaspace space to be GC'd (Java 7 running here):
I'll need to do some more investigation to see if there's any more permanent fixes/workarounds for this Netty ThreadLocal issue, or if there's anything else not getting collected.
monday0rsunday commentedon Sep 20, 2016
@alexmoore Thank you, I'll try it and feed back to you.
alexmoore commentedon Sep 20, 2016
@monday0rsunday So looking into it more today, if we:
After the refresh + GC, there are no additional instances of any
io.nettyorcom.basho.riakclasses in dump 2, which means that everything in those two libs are getting GC'd over the refresh.There are some new additional objects in dump 2 that I can't find homes for, but I think they are related to the Jetty reload:
Digging further into the big
java.lang.ref.Finalizerdifference, this seems related to the Jar file reloading, same with thebyte[],String, etc differences. Those objects seem to get GC'd later on after everything is reloaded or finalized.Please try adding those two lines to your web service
contextDestroyed(...)method and let me know if you see any more errant behavior.monday0rsunday commentedon Sep 20, 2016
@alexmoore yes, after adding two lines of code, I haven't seen any leaked WebAppClassLoader. Although there're some other leaked classes, but they won't be more serious than WebAppClassLoader, so the problem can be considered to be solved.