Skip to content

Logging system tests are hanging on CircleCI #3237

Closed
Closed
Logging system tests are hanging on CircleCI#3237
Assignees
tseavertheacodesdhermes
Labels
api: loggingIssues related to the Cloud Logging API.flakypriority: p1Important issue which blocks shipping the next release. Will be fixed prior to next release.testingtype: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.type: cleanupAn internal cleanup or hygiene concern.

Description

@dhermes
dhermes
opened on Mar 29, 2017

Four builds all hung:

I have run these tests 8+ times locally since the original failure (and using the same Cloud Console project) and they run without issue.

@waprin Any ideas?

Activity

  1. dhermes
    added
    api: loggingIssues related to the Cloud Logging API.
    flaky
    testing
    on Mar 29, 2017

  2. dhermes commented on Mar 29, 2017

    @dhermes
    dhermes
    on Mar 29, 2017
    ContributorAuthor

    Note this is "blocking" the release of error_reporting == 0.23.3 /cc @DizzeePascall

  3. dhermes commented on Mar 29, 2017

    @dhermes
    dhermes
    on Mar 29, 2017
    ContributorAuthor

    Running py.test tests/system.py -vv (under the logging directory) the (deterministic) order of test cases is:

    01 tests/system.py::TestLogging::test_create_metric PASSED
02 tests/system.py::TestLogging::test_create_sink_bigquery_dataset PASSED
03 tests/system.py::TestLogging::test_create_sink_pubsub_topic PASSED
04 tests/system.py::TestLogging::test_create_sink_storage_bucket PASSED
05 tests/system.py::TestLogging::test_list_metrics PASSED
06 tests/system.py::TestLogging::test_list_sinks PASSED
07 tests/system.py::TestLogging::test_log_handler_async PASSED
08 tests/system.py::TestLogging::test_log_handler_sync PASSED
09 tests/system.py::TestLogging::test_log_root_handler PASSED
10 tests/system.py::TestLogging::test_log_struct PASSED
11 tests/system.py::TestLogging::test_log_struct_w_metadata PASSED
12 tests/system.py::TestLogging::test_log_text PASSED
13 tests/system.py::TestLogging::test_log_text_w_metadata PASSED
14 tests/system.py::TestLogging::test_log_text_with_timestamp PASSED
15 tests/system.py::TestLogging::test_reload_metric PASSED
16 tests/system.py::TestLogging::test_reload_sink PASSED
17 tests/system.py::TestLogging::test_update_metric PASSED
18 tests/system.py::TestLogging::test_update_sink PASSED

    (with numbers added by me).

  4. waprin commented on Mar 29, 2017

    @waprin
    waprin
    on Mar 29, 2017
    Contributor

    I'm a bit confused as to under what circumstances these tests are failing? Is it with those error reporting changes or is it just sporadically failing everywhere (sorry Circle interface isn't obviously answering this question to me).

    test_log_root_handler(self) is the most complex test and worth commenting out to see if it fixes anything.

  5. dhermes commented on Mar 29, 2017

    @dhermes
    dhermes
    on Mar 29, 2017
    ContributorAuthor

    The logging environment is completely isolated from the error_reporting changes, so they are unrelated. (Though logging is a dependency of error_reporting, so we want to make sure logging is good before doing a downstream release.)

    Every CI run since the first failure has failed, i.e. it doesn't seem sporadic. To replicate we'd want to use the same env. vars. locally?

  6. dhermes commented on Mar 30, 2017

    @dhermes
    dhermes
    on Mar 30, 2017
    ContributorAuthor

    @waprin We've now got a fourth instance.

  7. lukesneeringer commented on Mar 30, 2017

    @lukesneeringer
    lukesneeringer
    on Mar 30, 2017
    Contributor

    Adding an issue-3237 branch with -vvv so that we can see it more easily. Nevermind, @dhermes beat me to it.

  8. dhermes commented on Mar 30, 2017

    @dhermes
    dhermes
    on Mar 30, 2017
    ContributorAuthor

    55452314

  9. lukesneeringer commented on Mar 30, 2017

    @lukesneeringer
    lukesneeringer
    on Mar 30, 2017
    Contributor

    Closing, but merging #3244 to help us watch.

  10. lukesneeringer
    closed this as completedon Mar 30, 2017

  11. dhermes commented on May 9, 2017

    @dhermes
    dhermes
    on May 9, 2017
    ContributorAuthor

    This issue has come back

    • Build 1780 (test_log_handler_async in 2.7)
    • Build 1774 (test_log_text_with_timestamp in 3.6, 2.7 was green)
    • Build 1753 (test_log_struct_w_metadata in 3.6, 2.7 was green)
    • Build 1806 (test_log_handler_sync in 2.7)
    • Build 1833 (test_log_text_with_timestamp in 3.6, 2.7 was green)
    • Build 1859 (test_log_text_with_timestamp in 3.6, 2.7 was green)
    • Build 1861 (test_log_handler_async in 2.7)
    • Build 1862 (test_log_handler_async in 2.7)
    • Build 1869 (test_log_handler_async in 2.7)

    The previous failures were:

    • Build 1413 (test_log_struct in 2.7)
    • Build 1415 (test_log_text_with_timestamp in 2.7)
    • Build 1418 (test_log_text in 2.7)
    • Build 1426 (test_log_handler_async in 2.7)

    Third round of failures:

    • Build 2072 (test_log_text_w_metadata in 2.7)
    • Build 2079 (test_log_handler_async in 2.7)
    • Build 2091 (test_log_text in 3.6)
    • Build 2093 (test_log_root_handler in 2.7, test_log_text_with_resource in 3.6)
    • Build 2095 (test_log_handler_async in 2.7, test_log_struct_w_metadata in 3.6)
    • Build 2156 (test_log_text_with_resource in 3.6)
    • Build 2161 (test_log_handler_sync in 2.7)
  12. dhermes
    reopened this on May 9, 2017

  13. dhermes commented on May 11, 2017

    @dhermes
    dhermes
    on May 11, 2017
    ContributorAuthor

    @waprin @jonparrott This is somewhat concerning. Do you guys have any ideas what could be causing this?

  14. 28 remaining items

  15. chemelnucfin
    assigned
    chemelnucfin
    ,
    tseaver
    ,
    theacodes
    ,
    dhermes
    and
    waprin
    on Jan 15, 2018
  16. theacodes
    added
    type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.
    priority: p1Important issue which blocks shipping the next release. Will be fixed prior to next release.
    on Jan 19, 2018

  17. theacodes commented on Jan 19, 2018

    @theacodes
    theacodes
    on Jan 19, 2018
    Contributor

    This is now outright failures. We should get this fixed.

  18. theacodes commented on Jan 19, 2018

    @theacodes
    theacodes
    on Jan 19, 2018
    Contributor

    Whoever takes this on should wait until after #4759 is merged.

  19. theacodes commented on Jan 19, 2018

    @theacodes
    theacodes
    on Jan 19, 2018
    Contributor

    @chemelnucfin can I task you with taking care of the low-hanging fruit here? (there's some tests that refer to old bigquery names, so should be trivial to fix).

    In terms of hanging, see if you can still make it happen after #4759 & I'll see what I can do to get it fixed.

  20. chemelnucfin commented on Jan 19, 2018

    @chemelnucfin
    chemelnucfin
    on Jan 19, 2018
    Contributor

    I saw the merge, working on the bigquery ones.

  21. chemelnucfin commented on Jan 22, 2018

    @chemelnucfin
    chemelnucfin
    on Jan 22, 2018
    Contributor

    Possibly fixed by #4768. Will reopen if that's not the case.

  22. chemelnucfin
    closed this as completedon Jan 22, 2018
  23. theacodes
    unassigned
    waprin
    and
    chemelnucfin
    on Sep 28, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

api: loggingIssues related to the Cloud Logging API.flakypriority: p1Important issue which blocks shipping the next release. Will be fixed prior to next release.testingtype: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.type: cleanupAn internal cleanup or hygiene concern.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions