Skip to content

[Github Action] java.lang.NullPointerException and Docker disk space issue when running tests #6207

Closed
Closed
[Github Action] java.lang.NullPointerException and Docker disk space issue when running tests#6207
Labels
type/featureThe PR added a new feature or issue requested a new featuretype/flaky-tests

Description

@devinbost
devinbost
opened on Feb 4, 2020

We're getting this test failure from the Github Action Standalone test:

[ERROR] Failures: 
[ERROR] org.apache.pulsar.tests.integration.standalone.SmokeTest.pulsar-standalone-suite(org.apache.pulsar.tests.integration.standalone.SmokeTest)
[ERROR]   Run 1: SmokeTest>PulsarStandaloneTestSuite.setUpCluster:31->PulsarStandaloneTestBase.startCluster:62 Docker environment should have more than 2GB free disk space

Here's the stacktrace of the issue:

01:31:45.766 [main:org.testcontainers.utility.RegistryAuthLocator@135] WARN  org.testcontainers.utility.RegistryAuthLocator - Failure when attempting to lookup auth config (dockerImageName: quay.io/testcontainers/ryuk:0.2.3, configFile: /home/runner/.docker/config.json. Falling back to docker-java default behaviour. Exception message: /home/runner/.docker/config.json (No such file or directory)
01:31:48.589 [main:org.testcontainers.utility.RegistryAuthLocator@135] WARN  org.testcontainers.utility.RegistryAuthLocator - Failure when attempting to lookup auth config (dockerImageName: quay.io/testcontainers/ryuk:0.2.3, configFile: /home/runner/.docker/config.json. Falling back to docker-java default behaviour. Exception message: /home/runner/.docker/config.json (No such file or directory)
        ℹ︎ Checking the system...
        ✔ Docker version should be at least 1.6.0
        ✘ Docker environment should have more than 2GB free disk space
[Utils] [ERROR] [Error] java.lang.NullPointerException
	at org.apache.pulsar.tests.integration.topologies.PulsarStandaloneTestBase.serviceUrlAndTopics(PulsarStandaloneTestBase.java:45)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:74)
	at org.testng.internal.MethodInvocationHelper.invokeMethodNoCheckedException(MethodInvocationHelper.java:45)
	at org.testng.internal.MethodInvocationHelper.invokeDataProvider(MethodInvocationHelper.java:131)
	at org.testng.internal.Parameters.handleParameters(Parameters.java:706)
	at org.testng.internal.ParameterHandler.handleParameters(ParameterHandler.java:49)
	at org.testng.internal.ParameterHandler.createParameters(ParameterHandler.java:37)
	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:924)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
	at org.testng.TestRunner.privateRun(TestRunner.java:648)
	at org.testng.TestRunner.run(TestRunner.java:505)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:450)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:415)
	at org.testng.SuiteRunner.run(SuiteRunner.java:364)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1208)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1137)
	at org.testng.TestNG.runSuites(TestNG.java:1049)
	at org.testng.TestNG.run(TestNG.java:1017)
	at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:283)
	at org.apache.maven.surefire.testng.TestNGXmlTestSuite.execute(TestNGXmlTestSuite.java:75)
	at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:120)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)

~~~~~~~~~ SKIPPED -- [TestClass name=class org.apache.pulsar.tests.integration.standalone.SmokeTest].testPublishAndConsume([])-------[ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 2, Time elapsed: 4.947 s <<< FAILURE! - in TestSuite
[ERROR] pulsar-standalone-suite(org.apache.pulsar.tests.integration.standalone.SmokeTest)  Time elapsed: 4.882 s  <<< FAILURE!
java.lang.AssertionError: Docker environment should have more than 2GB free disk space
	at org.rnorth.visibleassertions.VisibleAssertions.fail(VisibleAssertions.java:437)
	at org.rnorth.visibleassertions.VisibleAssertions.assertTrue(VisibleAssertions.java:129)
	at org.testcontainers.DockerClientFactory.checkDiskSpace(DockerClientFactory.java:185)
	at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:134)
	at org.testcontainers.containers.GenericContainer.<init>(GenericContainer.java:138)
	at org.apache.pulsar.tests.integration.containers.ChaosContainer.<init>(ChaosContainer.java:46)
	at org.apache.pulsar.tests.integration.containers.PulsarContainer.<init>(PulsarContainer.java:90)
	at org.apache.pulsar.tests.integration.containers.StandaloneContainer.<init>(StandaloneContainer.java:43)
	at org.apache.pulsar.tests.integration.topologies.PulsarStandaloneTestBase.startCluster(PulsarStandaloneTestBase.java:62)
	at org.apache.pulsar.tests.integration.suites.PulsarStandaloneTestSuite.setUpCluster(PulsarStandaloneTestSuite.java:31)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
	at org.testng.internal.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:59)
	at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:458)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:222)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:142)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:401)
	at org.testng.SuiteRunner.run(SuiteRunner.java:364)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1208)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1137)
	at org.testng.TestNG.runSuites(TestNG.java:1049)
	at org.testng.TestNG.run(TestNG.java:1017)
	at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:283)
	at org.apache.maven.surefire.testng.TestNGXmlTestSuite.execute(TestNGXmlTestSuite.java:75)
	at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:120)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)

[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR] org.apache.pulsar.tests.integration.standalone.SmokeTest.pulsar-standalone-suite(org.apache.pulsar.tests.integration.standalone.SmokeTest)
[ERROR]   Run 1: SmokeTest>PulsarStandaloneTestSuite.setUpCluster:31->PulsarStandaloneTestBase.startCluster:62 Docker environment should have more than 2GB free disk space

We're also getting the same issue with these tests:

CI - Integration - Tiered JCloud / tiered-jcloud
CI - Integration - Tiered FileSystem / tiered-filesystem
CI - Integration - Messaging
CI - Integration - Schema
CI - Integration - Sql
CI - Integration - Standalone

The stack traces appear almost identical.

This issue seems similar to this resolved issue: #5977

I have also attached the log archives from the first three failing tests. (I didn't see any more value in adding more.)

logs_11737.zip
logs_11738.zip
logs_11742.zip

Activity

  1. devinbost
    added
    type/featureThe PR added a new feature or issue requested a new feature
    on Feb 4, 2020

  2. devinbost commented on Feb 4, 2020

    @devinbost
    devinbost
    on Feb 4, 2020
    ContributorAuthor

    I found this in .github/workflows/ci-integration-backwards-compatibility.yaml

          - name: clean docker container
        if: steps.docs.outputs.changed_only == 'no'
        run: docker system prune -f

    It seems like this block might not be getting run because otherwise, it's very odd that there wouldn't be enough disk space available.

    Perhaps steps.docs.outputs.changed_only is not evaluating to no?
    Where is this value set?

    @tuteng @sijie Anyone know?

  3. devinbost
    changed the title [-][Github Action] Clean up space before running standalone tests[/-] [+][Github Action] java.lang.NullPointerException when running tests[/+] on Feb 4, 2020
  4. devinbost
    changed the title [-][Github Action] java.lang.NullPointerException when running tests[/-] [+][Github Action] java.lang.NullPointerException and Docker disk space issue when running tests[/+] on Feb 4, 2020

  5. sijie commented on Feb 4, 2020

    @sijie
    sijie
    on Feb 4, 2020
    Member

    if: steps.docs.outputs.changed_only == 'no' this block is used for skipping tests if a pull request is a documentation-only change. it is evaluated by this action: https://github.com/apache/pulsar/blob/master/.github/workflows/ci-integration-backwards-compatibility.yaml#L40

    All the tests in that workflow are using same condition. So either all the checks run or they don't run.

  6. jiazhai
    added
    triage/week-6
    on Feb 4, 2020
  7. devinbost
    added 5 commits that reference this issue on Feb 4, 2020

    Attempt to fix apache#6207 and add more debugging information by prun…

    e292cba

    Fixed typo in docker commands for getting debug info. apache#6207.

    79afaee

    Added awaitility to two pom files.

    db9c80b

    Added awaitility to two pom files.

    2be4cc9

    Added awaitility to two pom files.

    2393d08
  8. jiazhai
    added
    type/flaky-tests
    on Feb 7, 2020

  9. jiazhai commented on Feb 7, 2020

    @jiazhai
    jiazhai
    on Feb 7, 2020
    Member

    Thanks a lot @devinbost for helping on the flaky tests.

  10. devinbost
    added 4 commits that reference this issue on Feb 11, 2020

    Added awaitility to two pom files.

    f04bac9

    Added awaitility to two pom files.

    d5a02ea

    Added awaitility to two pom files.

    914bd6c

    Added awaitility to two pom files.

    fc80336

  11. devinbost commented on Feb 19, 2020

    @devinbost
    devinbost
    on Feb 19, 2020
    ContributorAuthor

    I believe this issue was fixed by #6098
    @tuteng Can you please confirm?

  12. tuteng commented on Feb 19, 2020

    @tuteng
    tuteng
    on Feb 19, 2020
    Member

    Yes, I think #6279 and #6098 for have been fixed this issue. @devinbost We can consider closing this issue now.

  13. sijie
    closed this as completedon Feb 19, 2020
  14. devinbost
    added 2 commits that reference this issue on Feb 22, 2020

    Added awaitility to two pom files.

    6904aa5

    Added awaitility to two pom files.

    e768d50
  15. devinbost
    added a commit that references this issue on Mar 13, 2020

    Combination of many commits.

    81e23eb
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    type/featureThe PR added a new feature or issue requested a new featuretype/flaky-tests

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions