analyzer_benchmark should run on only one configuration #124065

Activity

1. zanderso

   added

   team-infraOwned by Infrastructure teamOwned by Infrastructure team

   on Apr 3, 2023
2. keyonghan commented on Apr 3, 2023

   

   keyonghan

   on Apr 3, 2023

   Contributor

   More actions

   This benchmark is bimodal because some of the Linux hosts have 8 cores and others have 4 cores. The bimodal nature of the benchmark means that SkiaPerf generates spurious reports and/or masks real regressions. Instead the benchmark should be pinned to a single configuration.

   Yeah, we have two types of devicelab bots: 4 cores and 8 cores. Which do you want to enforce?

   As a separate issue, I think this benchmark does not need a phone attached, but I'm not 100% sure about that.

   Validated it works on a vm bot, but the benchmark is much slower:

       "flutter_repo_batch": 172.325934,
       "flutter_repo_batch_minimum": 168.90769,
       "flutter_repo_batch_maximum": 174.241438,
       "flutter_repo_watch": 159.38031566666666,
       "flutter_repo_watch_minimum": 154.742325,
       "flutter_repo_watch_maximum": 167.512652,
       "mega_gallery_batch": 19.769443999999996,
       "mega_gallery_batch_minimum": 19.116,
       "mega_gallery_batch_maximum": 21.044355,
       "mega_gallery_watch": 19.275951000000003,
       "mega_gallery_watch_minimum": 19.206895,
       "mega_gallery_watch_maximum": 19.345438
   

   https://luci-milo.appspot.com/raw/build/logs.chromium.org/flutter/led/keyonghan_google.com/bd8e22d53a5c66633fa991eae3552f5d71c82c6e0c6cfcdadc73160cb5f8687c/+/build.proto

   Do we not care about the number, but just the trend? If not, then we can switch it to the vm if a new trend series sounds okay.
3. zanderso commented on Apr 4, 2023

   

   zanderso

   on Apr 4, 2023

   MemberAuthor

   More actions

   Is it possible to specify a bigger VM instance so that the slowdown isn't so big? Either way, let's try moving to a VM and get a sense for whether that's a problem.
4. keyonghan commented on Apr 4, 2023

   

   keyonghan

   on Apr 4, 2023

   Contributor

   More actions

   A 32-core VM gives relatively lower metrics, though not that much:

       "flutter_repo_batch": 157.02144266666667,
       "flutter_repo_batch_minimum": 155.004229,
       "flutter_repo_batch_maximum": 158.313348,
       "flutter_repo_watch": 155.359091,
       "flutter_repo_watch_minimum": 153.770459,
       "flutter_repo_watch_maximum": 156.545673,
       "mega_gallery_batch": 19.723341,
       "mega_gallery_batch_minimum": 18.779556,
       "mega_gallery_batch_maximum": 21.449513,
       "mega_gallery_watch": 19.044307333333336,
       "mega_gallery_watch_minimum": 18.900754,
       "mega_gallery_watch_maximum": 19.163864
   

   #124147 to move it to a 8-core fleet as bringup: true.
5. keyonghan

   self-assigned this

   on Apr 7, 2023
6. auto-submit

   closed this as completedin #124394on Apr 7, 2023
7. github-actions commented on Apr 21, 2023

   

   github-actionsbot

   on Apr 21, 2023

   More actions

   This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.
8. github-actions

   locked as resolved and limited conversation to collaborators on Apr 21, 2023
9. keyonghan

   moved this to Done in [DEPRECATED] Infra Ticket Queueon May 1, 2023
10. keyonghan

    added this to [DEPRECATED] Infra Ticket Queueon May 1, 2023
11. keyonghan

    moved this from Verified to To be archived in [DEPRECATED] Infra Ticket Queueon Jan 31, 2024
12. keyonghan

    moved this from To be archived to Verified in [DEPRECATED] Infra Ticket Queueon Jan 31, 2024