Skip to content

Bug in netty 4.1.43 prevents from using Linux native Epoll transport #6330

Closed
#6424
Closed
Bug in netty 4.1.43 prevents from using Linux native Epoll transport#6330
#6424
Labels
release/2.5.1type/bugThe PR fixed a bug or issue reported a bug

Description

@dzmitryk
dzmitryk
opened on Feb 15, 2020

Describe the bug
netty 4.1.43 has a bug preventing it from using Linux native Epoll transport - netty/netty#9758

This results in pulsar brokers failing over to NioEventLoopGroup even when running on Linux.

The bug seems to be fixed in netty releases 4.1.44+

To Reproduce

  1. Run pulsar docker container:
docker run -it apachepulsar/pulsar:latest bin/pulsar standalone
  1. Observe the logs. Following errors appears on startup
23:56:05.428 [main] WARN  org.apache.distributedlog.impl.BKNamespaceDriver - Could not use Netty Epoll event loop for bookie server:
java.lang.NoClassDefFoundError: Could not initialize class io.netty.channel.epoll.EpollEventLoop
        at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:142) ~[io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:35) ~[io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84) ~[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58) ~[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47) ~[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:104) ~[io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:91) ~[io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:68) ~[io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final]
        at org.apache.distributedlog.impl.BKNamespaceDriver.getDefaultEventLoopGroup(BKNamespaceDriver.java:257) ~[org.apache.distributedlog-distributedlog-core-4.10.0.jar:4.10.0]
        at org.apache.distributedlog.impl.BKNamespaceDriver.initializeBookKeeperClients(BKNamespaceDriver.java:268) ~[org.apache.distributedlog-distributedlog-core-4.10.0.jar:4.10.0]
        at org.apache.distributedlog.impl.BKNamespaceDriver.initialize(BKNamespaceDriver.java:206) ~[org.apache.distributedlog-distributedlog-core-4.10.0.jar:4.10.0]
        at org.apache.distributedlog.api.namespace.NamespaceBuilder.build(NamespaceBuilder.java:239) ~[org.apache.distributedlog-distributedlog-core-4.10.0.jar:4.10.0]
        at org.apache.bookkeeper.stream.server.service.DLNamespaceProviderService.doStart(DLNamespaceProviderService.java:120) ~[org.apache.bookkeeper-stream-storage-server-4.10.0.jar:4.10.0]
        at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:83) ~[org.apache.bookkeeper-bookkeeper-common-4.10.0.jar:4.10.0]
        at org.apache.bookkeeper.common.component.LifecycleComponentStack.lambda$start$2(LifecycleComponentStack.java:113) ~[org.apache.bookkeeper-bookkeeper-common-4.10.0.jar:4.10.0]
        at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:407) [com.google.guava-guava-25.1-jre.jar:?]
        at org.apache.bookkeeper.common.component.LifecycleComponentStack.start(LifecycleComponentStack.java:113) [org.apache.bookkeeper-bookkeeper-common-4.10.0.jar:4.10.0]
        at org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent.doStart(StreamStorageLifecycleComponent.java:51) [org.apache.bookkeeper-stream-storage-server-4.10.0.jar:4.10.0]
        at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:83) [org.apache.bookkeeper-bookkeeper-common-4.10.0.jar:4.10.0]
        at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.runStreamStorage(LocalBookkeeperEnsemble.java:323) [org.apache.pulsar-pulsar-zookeeper-utils-2.5.0.jar:2.5.0]
        at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.startStandalone(LocalBookkeeperEnsemble.java:405) [org.apache.pulsar-pulsar-zookeeper-utils-2.5.0.jar:2.5.0]
        at org.apache.pulsar.PulsarStandalone.start(PulsarStandalone.java:260) [org.apache.pulsar-pulsar-broker-2.5.0.jar:2.5.0]
        at org.apache.pulsar.PulsarStandaloneStarter.main(PulsarStandaloneStarter.java:119) [org.apache.pulsar-pulsar-broker-2.5.0.jar:2.5.0]

With DEBUG logging enabled following error appears as well:

22:31:13.747 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - Unable to load the library 'netty_transport_native_epoll_x86_64', trying other loading mechanism.
java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll_x86_64 in java.library.path
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1860) ~[?:1.8.0_232]
        at java.lang.Runtime.loadLibrary0(Runtime.java:870) ~[?:1.8.0_232]
        at java.lang.System.loadLibrary(System.java:1122) ~[?:1.8.0_232]
        at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38) ~[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_232]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_232]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_232]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_232]
        at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:369) ~[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_232]
        at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:361) ~[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:339) [io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136) [io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:228) [io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.epoll.Native.<clinit>(Native.java:57) [io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.epoll.Epoll.<clinit>(Epoll.java:39) [io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.epoll.EpollEventLoop.<clinit>(EpollEventLoop.java:52) [io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:142) [io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:35) [io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84) [io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58) [io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47) [io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:104) [io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:91) [io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:68) [io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final]
        at org.apache.bookkeeper.util.EventLoopUtil.getEventLoopGroup(EventLoopUtil.java:59) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
        at org.apache.bookkeeper.util.EventLoopUtil.getClientEventLoopGroup(EventLoopUtil.java:43) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
        at org.apache.bookkeeper.client.BookKeeper.<init>(BookKeeper.java:458) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
        at org.apache.bookkeeper.client.BookKeeper$Builder.build(BookKeeper.java:296) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
        at org.apache.pulsar.broker.BookKeeperClientFactoryImpl.create(BookKeeperClientFactoryImpl.java:70) [org.apache.pulsar-pulsar-broker-2.5.0.jar:2.5.0]
        at org.apache.pulsar.broker.ManagedLedgerClientFactory.<init>(ManagedLedgerClientFactory.java:61) [org.apache.pulsar-pulsar-broker-2.5.0.jar:2.5.0]
        at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:394) [org.apache.pulsar-pulsar-broker-2.5.0.jar:2.5.0]
        at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:264) [org.apache.pulsar-pulsar-broker-2.5.0.jar:2.5.0]
        at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:329) [org.apache.pulsar-pulsar-broker-2.5.0.jar:2.5.0]

Expected behavior
Netty EpollEventLoopGroup is initialized correctly.

Activity

  1. dzmitryk
    added
    type/bugThe PR fixed a bug or issue reported a bug
    on Feb 15, 2020
  2. sijie
    added
    triage/week-8
    release/2.5.1
    on Feb 17, 2020

  3. jiazhai commented on Feb 25, 2020

    @jiazhai
    jiazhai
    on Feb 25, 2020
    Member

    @dzmitryk Thanks for the issue, and root cause. Would you like to provide a PR to bump the netty version?

  4. xuesongxs commented on Feb 25, 2020

    @xuesongxs
    xuesongxs
    on Feb 25, 2020
    Contributor

    start broker with function, show errors.

    16:11:01.231 [main] WARN org.apache.distributedlog.impl.BKNamespaceDriver - Could not use Netty Epoll event loop for bookie server:
    java.lang.NoClassDefFoundError: Could not initialize class io.netty.channel.epoll.EpollEventLoop
    at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:142) ~[io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:35) ~[io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:84) ~[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:58) ~[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:47) ~[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.MultithreadEventLoopGroup.(MultithreadEventLoopGroup.java:59) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:104) ~[io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:91) ~[io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:68) ~[io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final]
    at org.apache.distributedlog.impl.BKNamespaceDriver.getDefaultEventLoopGroup(BKNamespaceDriver.java:257) [org.apache.distributedlog-distributedlog-core-4.10.0.jar:4.10.0]
    at org.apache.distributedlog.impl.BKNamespaceDriver.initializeBookKeeperClients(BKNamespaceDriver.java:268) [org.apache.distributedlog-distributedlog-core-4.10.0.jar:4.10.0]
    at org.apache.distributedlog.impl.BKNamespaceDriver.initialize(BKNamespaceDriver.java:206) [org.apache.distributedlog-distributedlog-core-4.10.0.jar:4.10.0]
    at org.apache.distributedlog.api.namespace.NamespaceBuilder.build(NamespaceBuilder.java:239) [org.apache.distributedlog-distributedlog-core-4.10.0.jar:4.10.0]
    at org.apache.pulsar.functions.worker.WorkerService.start(WorkerService.java:105) [org.apache.pulsar-pulsar-functions-worker-2.5.0.jar:2.5.0]
    at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1108) [org.apache.pulsar-pulsar-broker-2.5.0.jar:2.5.0]
    at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:505) [org.apache.pulsar-pulsar-broker-2.5.0.jar:2.5.0]
    at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:264) [org.apache.pulsar-pulsar-broker-2.5.0.jar:2.5.0]
    at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:329) [org.apache.pulsar-pulsar-broker-2.5.0.jar:2.5.0]

  5. dzmitryk commented on Feb 26, 2020

    @dzmitryk
    dzmitryk
    on Feb 26, 2020
    MemberAuthor

    @jiazhai I've opened a PR to bump the netty version - #6424, but I am not sure how to go about testing it. Do you think the existing integration tests cover it?

  6. jiazhai commented on Feb 27, 2020

    @jiazhai
    jiazhai
    on Feb 27, 2020
    Member

    @dzmitryk, @xuesong172500. Thanks a lot for the help.
    @dzmitryk . it should be fine for this bump version, current tests should be enough.

  7. jiazhai
    closed this as completedin #6424on Feb 27, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    release/2.5.1type/bugThe PR fixed a bug or issue reported a bug

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions