Repository navigation
System.Diagnostics.Tests.PerformanceCounterTests failed in CI #60933
Description
Frequency:
| Day | Run | Failures | Notes |
|---|---|---|---|
| 4/1-7/22 | |||
| 6/30 | PR #68648 | 10 failures | Win11 - Mono |
| 6/28 | PR #71397 | 10 failures | Win10 - x86 |
| 6/15 | Official run | 10 failures | Win11 |
| 6/13 | PR #70403 | 10 failures | Win10 - x86 |
| 5/7 | PR #68993 | 10 failures | Win11 |
| 4/6 | PR #67557 | 10 failures | Win11 |
| 4/2 | Official run | 10 failures | Win10 |
let failedTests = (methodName : string, includePR : bool, messageSubstr: string, includePassedOnRerun : bool) {
cluster('engsrvprod.kusto.windows.net').database('engineeringdata').AzureDevOpsTests
| where TestName contains methodName
| where includePassedOnRerun or (Outcome == 'Failed')
| extend startOfTestName = indexof_regex(TestName, @"[^.]+$")
| extend Method = substring(TestName, startOfTestName)
| extend Type = substring(TestName, 0, startOfTestName - 1)
| project-away startOfTestName
| where Message contains messageSubstr
| distinct JobId, WorkItemId, Message, StackTrace, Method, Type, Arguments, Outcome
| join kind=inner (cluster('engsrvprod.kusto.windows.net').database('engineeringdata').Jobs
| where ((Branch == 'refs/heads/main') or (Branch == 'refs/heads/master') or (includePR and (Source startswith "pr/")))
| where Type startswith "test/functional/cli/"
and not(Properties contains "runtime-staging")
| summarize arg_max(Finished, Properties, Type, Branch, Source, Started, QueueName) by JobId
| project-rename JobType = Type) on JobId
| extend PropertiesJson = parse_json(Properties)
| extend OS = replace_regex(tostring(PropertiesJson.operatingSystem), @'\((.*)\).*|([^\(].*)', @'\1\2')
| extend Runtime = iif(PropertiesJson.runtimeFlavor == "mono", "Mono", iif(PropertiesJson.DefinitionName contains "coreclr", "CoreCLR", ""))
| extend TargetBranch = extractjson("$.['System.PullRequest.TargetBranch']", Properties)
| extend Architecture = PropertiesJson.architecture
| extend Scenario = iif(isempty(PropertiesJson.scenario), "--", PropertiesJson.scenario)
| project-away PropertiesJson
};
failedTests('System.Diagnostics.Tests.InstanceDataTests', true, 'Category', true);Test System.Diagnostics.Tests.PerformanceCounterTests.PerformanceCounter_BeginInitEndInit_ProcessorCounter has failed. Check the Test tab or this console log: https://helix.dot.net/api/2019-06-17/jobs/7fb481c8-b204-4809-9ac6-1f7db90f5e5d/workitems/System.Diagnostics.PerformanceCounter.Tests/console
Console log: 'System.Diagnostics.PerformanceCounter.Tests' from job 7fb481c8-b204-4809-9ac6-1f7db90f5e5d workitem a0e2d365-2975-4dd7-b36e-cc56d3df3997 (windows.7.amd64.open.rt) executed on machine a002OCP
C:\h\w\AA5A0982\w\ADD20987\e>taskkill.exe /f /im corerun.exe
ERROR: The process "corerun.exe" not found.
C:\h\w\AA5A0982\w\ADD20987\e>call RunTests.cmd --runtime-path C:\h\w\AA5A0982\p
----- start Wed 10/27/2021 3:46:34.80 =============== To repro directly: =====================================================
pushd C:\h\w\AA5A0982\w\ADD20987\e\
"C:\h\w\AA5A0982\p\dotnet.exe" exec --runtimeconfig System.Diagnostics.PerformanceCounter.Tests.runtimeconfig.json --depsfile System.Diagnostics.PerformanceCounter.Tests.deps.json xunit.console.dll System.Diagnostics.PerformanceCounter.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing
popd
===========================================================================================================
C:\h\w\AA5A0982\w\ADD20987\e>"C:\h\w\AA5A0982\p\dotnet.exe" exec --runtimeconfig System.Diagnostics.PerformanceCounter.Tests.runtimeconfig.json --depsfile System.Diagnostics.PerformanceCounter.Tests.deps.json xunit.console.dll System.Diagnostics.PerformanceCounter.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing
Discovering: System.Diagnostics.PerformanceCounter.Tests (method display = ClassAndMethod, method display options = None)
Discovered: System.Diagnostics.PerformanceCounter.Tests (found 91 test cases)
Starting: System.Diagnostics.PerformanceCounter.Tests (parallel test collections = on, max threads = 2)
System.Diagnostics.Tests.PerformanceCounterTests.PerformanceCounter_BeginInitEndInit_ProcessorCounter [FAIL]
System.InvalidOperationException : Category does not exist.
Stack Trace:
/_/src/libraries/System.Diagnostics.PerformanceCounter/src/System/Diagnostics/PerformanceCounterLib.cs(373,0): at System.Diagnostics.PerformanceCounterLib.CounterExists(String machine, String category, String counter)
/_/src/libraries/System.Diagnostics.PerformanceCounter/src/System/Diagnostics/PerformanceCounter.cs(479,0): at System.Diagnostics.PerformanceCounter.InitializeImpl()
/_/src/libraries/System.Diagnostics.PerformanceCounter/src/System/Diagnostics/PerformanceCounter.cs(453,0): at System.Diagnostics.PerformanceCounter.Initialize()
/_/src/libraries/System.Diagnostics.PerformanceCounter/src/System/Diagnostics/PerformanceCounter.cs(78,0): at System.Diagnostics.PerformanceCounter..ctor(String categoryName, String counterName, String instanceName, String machineName)
/_/src/libraries/System.Diagnostics.PerformanceCounter/tests/PerformanceCounterTests.cs(192,0): at System.Diagnostics.Tests.PerformanceCounterTests.PerformanceCounter_BeginInitEndInit_ProcessorCounter()
Info: Successfully installed performance counters in C:\h\w\AA5A0982\t\dxbpy5oa.jfn\provider.man System.Diagnostics.Tests.PerformanceDataTests.PerformanceCounter_PerformanceData [SKIP]
Condition(s) not met: "IsRunnableEnvironment"
Info: Successfully uninstalled the performance counters from the counter definition XML file C:\h\w\AA5A0982\t\dxbpy5oa.jfn\provider.man. Finished: System.Diagnostics.PerformanceCounter.Tests
=== TEST EXECUTION SUMMARY ===
System.Diagnostics.PerformanceCounter.Tests Total: 94, Errors: 0, Failed: 1, Skipped: 1, Time: 66.084s
----- end Wed 10/27/2021 3:47:42.10 ----- exit code 1 ----------------------------------------------------------
2021-10-27 03:47:42,569: INFO: 2516: run(48): main: Beginning reading of test results.
2021-10-27 03:47:42,569: INFO: 2516: __init__(48): get_log_files: Searching 'C:\h\w\AA5A0982\w\ADD20987\e\..' for log files
2021-10-27 03:47:42,569: INFO: 2516: __init__(66): construct_log_list: Generated log list: console.0927879d.log:
https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-60900-merge-7fb481c8b20448099a/System.Diagnostics.PerformanceCounter.Tests/console.0927879d.log?sv=2019-07-07&se=2021-11-16T03%3A32%3A07Z&sr=c&sp=rl&sig=w7TmCMmLjLzI7c%2Bfh8x2jkOV9DWDX64m%2BgXE4z5eMCg%3D
2021-10-27 03:47:42,569: INFO: 2516: __init__(90): read_results: Searching 'C:\h\w\AA5A0982\w\ADD20987\e' for test results files
2021-10-27 03:47:42,569: INFO: 2516: __init__(96): read_results: Found results file C:\h\w\AA5A0982\w\ADD20987\e\testResults.xml with format xunit
2021-10-27 03:47:42,569: INFO: 2516: __init__(90): read_results: Searching 'C:\h\w\AA5A0982\w\ADD20987\uploads' for test results files
2021-10-27 03:47:42,569: INFO: 2516: packing_test_reporter(30): report_results: Packing 94 test reports to 'C:\h\w\AA5A0982\w\ADD20987\e\__test_report.json'
2021-10-27 03:47:42,569: INFO: 2516: packing_test_reporter(33): report_results: Packed 31823 bytes
Found log 'C:\h\w\AA5A0982\w\ADD20987\e\..\console.0927879d.log'
Uri 'https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-60900-merge-7fb481c8b20448099a/System.Diagnostics.PerformanceCounter.Tests/console.0927879d.log?sv=2019-07-07&se=2021-11-16T03%3A32%3A07Z&sr=c&sp=rl&sig=w7TmCMmLjLzI7c%2Bfh8x2jkOV9DWDX64m%2BgXE4z5eMCg%3D'
Did not find dumps, skipping dump docs generation.
| Build | Kind | Start Time |
|---|---|---|
| 1583845 | Rolling | 2022-01-02 |
Activity
[-]PerformanceCounter_BeginInitEndInit_ProcessorCounter failed in CI[/-][+]System.Diagnostics.Tests.PerformanceCounterTests failed in CI[/+]krwq commentedon Nov 2, 2021
I'm reusing this issue since I've seen similar issue in different random tests across PerformanceCounters.
List of tests I've seen failing similarly in the past 14 days:
krwq commentedon Nov 2, 2021
I'm unclear if Mono issue: #53261 is the same or not so leaving that one for now. The above are Windows only (happened on Win 10, Win 11, Win 7) all on amd64 (although unclear if it's specific to this arch)
krwq commentedon Nov 2, 2021
Also related: #60403
danmoseley commentedon Nov 2, 2021
Noting from offline discussion -- for some reason, we are getting an incomplete read from the registry key (which I think is dynamically populated?). Recommendation is that if a category isn’t present, we should sleep for a little while, call CloseSharedResources(), and try again. We would do this in tests only -- unless something has changed in Windows, it doesn't seem to be affecting real-world use unless we hear otherwise. If it does affect customers, we could potentially do it inside CounterExists(..) but could not "fix" GetCategories() since we have no way to tell whether it's returning everything.
sbomer commentedon Feb 1, 2022
Hit again in ci for net7.0-windows-Release-x86-CoreCLR_release-Windows.7.Amd64.Open: build, log.
Disable some PerformanceCounter tests on more platforms (#64633)
danmoseley commentedon May 7, 2022
More, where retries aren't enough and we should try this "CloseSharedResources()" ...
radical commentedon May 10, 2022
Hit again on #68921 . Build, and log:
karelz commentedon Jun 20, 2022
Happened again in 1825417 (6/14 PM) - 10 test failures:
karelz commentedon Jul 22, 2022
Frequency updated in previous post. Given it is not super-common lately, removing blocking-clean-ci label.