Repository navigation
ActiveIssue not skipped on CI for iOS/tvOS devices in Release Configuration #59488
Description
System.Collections.Generic.Tests.ComparerTests.NullableComparisons<Int32> is not being skipped on CI as evidenced by this build result even though there is an ActiveIssue attribute
Oddly enough, running the suite locally on iOSSimulator and on iOS (local iPad), the test is skipped.
Activity
mdh1418 commentedon Sep 22, 2021
Looks like the test started failing since 9/17? https://dev.azure.com/dnceng/public/_test/analytics?definitionId=924&contextType=build scrolled down and clicked on System.Collections.Generic.Tests.ComparerTests.NullableComparisons
Not sure why it would be considered a pass before then if it is supposed to be skipped.
mdh1418 commentedon Sep 22, 2021
This is the first instance of the test failing (ActiveIssue not being skipped)
Which seems to point at #57326, and it has been flakey (in skipping this ActiveIssue attribute)
mdh1418 commentedon Sep 23, 2021
Filtered on iOS arm64 and tvOS arm64 test runs. Unfortunately, for iOS and tvOS there will not be a passed test because the test should be skipped if the attribute is handled properly, but judging by the counts, this behaviour is likely flakey.
mdh1418 commentedon Sep 24, 2021
Looks like this is a Release configuration specific problem. on iOS arm64 release, tests are failing, and this ActiveIssue is being skipped, the reason why only this test was caught was because we are only running a small subset of test suites at the moment for devices. When adding an [ActiveIssue("", TestPlatforms.iOS)] to a test in System.Buffers, the test is still ran when in iOS arm64 Release.
[-]ActiveIssue not skipped on CI for iOS/tvOS devices[/-][+]ActiveIssue not skipped on CI for iOS/tvOS devices in Release Configuration[/+]safern commentedon Sep 24, 2021
We don't do much special logic for ActiveIssue, we just check if we are running on that platform:
https://github.com/dotnet/arcade/blob/e43b85e9b4e9ca62b89e407acdf0e56a2cbfd81a/src/Microsoft.DotNet.XUnitExtensions/src/DiscovererHelpers.cs#L28
https://github.com/dotnet/arcade/blob/e43b85e9b4e9ca62b89e407acdf0e56a2cbfd81a/src/Microsoft.DotNet.XUnitExtensions/src/DiscovererHelpers.cs#L29
So maybe something else is happening in
RuntimeInformation.IsOSPlatform?cc: @steveisok
steveisok commentedon Sep 24, 2021
IsOSPlatformlooks good. We're trimming in our test runs, so we're trying to see what things are like w/o it.steveisok commentedon Sep 24, 2021
No better... An untrimmed app build in release fails the same way. Passes on debug.
steveisok commentedon Sep 26, 2021
I found why and we should be able to work around it.
https://github.com/dotnet/arcade/blob/9ab314e39dc623d53c3c7a63a6f63772b01ff6b3/src/Microsoft.DotNet.XUnitExtensions/src/DiscovererHelpers.cs#L17
This returns false because in release mode,
Environment.Version.Majorreturns 0 andRuntimeInformation.FrameworkDescriptionreturns ".NET". It's not clear why the version returns 0.steveisok commentedon Sep 27, 2021
We'll need an arcade bump in order to take dotnet/arcade#7965
That should fix the problem.
safern commentedon Sep 27, 2021
Looks like a bug when running on iOS/tvOS?
steveisok commentedon Sep 27, 2021
On the version, for sure. I don't know why yet. I've ruled out the linker on my end. The only other thing that mucks w/ the assemblies is
ILStrip. I have yet to run w/ that task disabled. That's the next thing I'm going to try.steveisok commentedon Sep 27, 2021
If I disable the
ILStriptask, this does not happen.@akoeplinger Can you please check if you experience the same thing?
steveisok commentedon Jul 25, 2022
Confirmed this is no longer an issue.