Linux tool_integration_tests_4_4 flakes on cache_test

(part of #97036)

Hit one flake today (02/09/2022):

cache_test.dart

02:05 +20 ~20 -1: test/integration.shard/cache_test.dart: Cache.lock should log a message to stderr when lock is not acquired [E]                                                                      
  Expected: 'Waiting for another flutter command to release the startup lock...\n'
              ''
    Actual: ''
     Which: is different. Both strings start the same, but the actual value is missing the following trailing characters: Waiting fo ...
  
  package:test_api                             expect
  test/integration.shard/cache_test.dart 78:7  main.<fn>.<fn>

@zanderso

/cc @zanderso

This test is inherently racy due to this setup:

https://github.com/flutter/flutter/blame/02cfbc320ea9c26ddde9ed5bab5cf01997ea7f4d/packages/flutter_tools/test/integration.shard/cache_test.dart#L63

It will fail when the parent process wins the race.

I'm assuming this is happening because it took longer than 500ms to lock the file, which is pretty long for a Unix machine.

The docs for Process say:

Returns a Future<Process> that completes with a Process instance when the process has been successfully
started.

So, either the docs are wrong, or it's taking longer than 500ms to lock a file, or I'm wrong about why the race is happening.

Both the child script and the parent have timed delays and the child delay is twice as long as the parent delay, so as long as the child is launched and locks the file within the parent's 500ms file locking delay, this shouldn't incorrectly fail.

I don't see a good way to test this other than with an external process that locks the same file: it's not possible to hold separate locks to the same file within the same process.

I suppose I could write a tight loop that checks the locking status of the file, and only tries to lock the cache once it's locked. I'll pursue that.

You can always expect VMs/bots to introduce delays of arbitrary duration at the worst possible time...

Another flake: https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20tool_integration_tests_4_4/3450/overview

@keyonghan

Marking P1 to match the priority of other flakes. Though if this one is only happening once every two weeks, that is probably below the 2% threshold. Maybe @keyonghan can supply that data.

@keyonghan

Marking P1 to match the priority of other flakes. Though if this one is only happening once every two weeks, that is probably below the 2% threshold. Maybe @keyonghan can supply that data.

We hit the above flake (https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20tool_integration_tests_4_4/3450/overview) since the bug was filed (02/09), namely 1 over the past 200 commits.

Since this is below the 2% threshold, I'll drop this back down to P3.

I have a fix for this, in #99423

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

Navigation Menu

Repository navigation

Linux tool_integration_tests_4_4 flakes on `cache_test` #98141

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Linux tool_integration_tests_4_4 flakes on cache_test #98141

Description

Activity

keyonghan commented on Feb 9, 2022

zanderso commented on Feb 9, 2022

gspencergoog commented on Feb 14, 2022

zanderso commented on Feb 14, 2022

keyonghan commented on Mar 2, 2022

zanderso commented on Mar 2, 2022

keyonghan commented on Mar 2, 2022

zanderso commented on Mar 2, 2022

gspencergoog commented on Mar 2, 2022

github-actions commented on Mar 24, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Linux tool_integration_tests_4_4 flakes on `cache_test` #98141