Repository navigation
[Wasm] Tracking active test failures #38422
Description
We began running tests for wasm on CI, however not all test suites currently pass. This issue tracks what's failing:
OOB = out of box library
Suites that contain 1 or more failures
- Common.Tests ([wasm] Make Common.Tests pass #38667)
- OOB: Microsoft.Extensions.Configuration.FileExtensions.Tests ([Wasm]Adjust AppContext.BaseDirectory and enable Microsoft.Extensions tests #38721)
- OOB: Microsoft.Extensions.Configuration.Ini.Tests ([Wasm]Adjust AppContext.BaseDirectory and enable Microsoft.Extensions tests #38721)
- OOB: Microsoft.Extensions.Configuration.Json.Tests ([Wasm]Adjust AppContext.BaseDirectory and enable Microsoft.Extensions tests #38721)
- OOB: Microsoft.Extensions.Configuration.UserSecrets.Tests ([Wasm]Adjust AppContext.BaseDirectory and enable Microsoft.Extensions tests #38721)
- OOB: Microsoft.Extensions.Configuration.Xml.Tests ([Wasm]Adjust AppContext.BaseDirectory and enable Microsoft.Extensions tests #38721)
- OOB: Microsoft.Extensions.Configuration.Functional.Tests ([Wasm]Adjust AppContext.BaseDirectory and enable Microsoft.Extensions tests #38721)
- OOB: Microsoft.Extensions.DependencyModel.Tests ([Wasm]Adjust AppContext.BaseDirectory and enable Microsoft.Extensions tests #38721)
- OOB: Microsoft.Extensions.FileProviders.Physical.Tests ([Wasm]Adjust AppContext.BaseDirectory and enable Microsoft.Extensions tests #38721)
- OOB: Microsoft.Extensions.Hosting.Unit.Tests ([Wasm]Adjust AppContext.BaseDirectory and enable Microsoft.Extensions tests #38721)
- Microsoft.VisualBasic.Core.Tests (WASM: Fix System.IO.FileSystem.DriveInfo and Microsoft.VisualBasic.Core tests #39276)
- OOB: System.CodeDom.Tests ([wasm] Enable System.CodeDom.Tests test suite #39626 )
- System.Collections.NonGeneric.Tests (WASM enable System.Collections.NonGeneric.Tests #39020)
- System.Collections.Specialized.Tests (WASM enable System.Collections.Specialized.Tests #39012)
- System.Collections.Tests (WASM Enable System.Collections tests #38976)
- System.ComponentModel.Annotations.Tests (WASM: Disable tests in System.ComponentModel.Annotations that rely on globalization #39228)
- System.ComponentModel.Primitives.Tests ([wasm] Enable the library test suites that have no failures at the moment #39427)
- System.ComponentModel.TypeConverter.Tests ([wasm] Enable System.ComponentModel.TypeConverter test suite #39710)
- System.Data.Common.Tests ([Wasm] Enable System.Data.Common tests #39463)
- System.Diagnostics.DiagnosticSource.Switches.Tests (requires runtimeconfig.template.json support - [wasm] Mark DiagnosticSource.Switches.Tests with an active issue #39952)
- System.Diagnostics.FileVersionInfo.Tests (WASM: Disable tests for System.Diagnostics.Process and .FileVersionInfo #39088)
- System.Diagnostics.Process.Tests (WASM: Disable tests for System.Diagnostics.Process and .FileVersionInfo #39088)
- System.Diagnostics.StackTrace.Tests (WASM: Disable tests on System.Diagnostics.StackTrace that fail #39226)
- System.Diagnostics.TextWriterTraceListener.Tests (WASM: Fix System.Diagnostics.TextWriterTraceListener tests #39186)
- System.Diagnostics.TraceSource.Tests (WASM: Fix System.Diagnostics.TraceSource tests #39224)
- System.Diagnostics.Tracing.Tests (WASM: Fix System.Diagnostics.Tracing #39271)
- OOB: System.Drawing.Common.Tests ([wasm] Modify System.Drawing.Common to throw PNSE on Browser #38999)
- System.Formats.Asn1.Tests (WASM: Fix System.Formats.Asn1 tests #39104)
- System.Globalization Invariant.Tests (WASM: Enable Invariant.Tests #39814)
- System.Globalization.CalendarsWithConfigSwitch.Tests (requires runtimeconfig.template.json support)
- System.Globalization.Calendars.Tests (ICU integration and asset loading overhaul #37971)
- System.Globalization.Extensions.Tests (ICU integration and asset loading overhaul #37971)
- System.Globalization.Tests (ICU integration and asset loading overhaul #37971)
- System.IO.Compression.Brotli.Tests ([Wasm] Skip System.IO.Compression.Brotli tests b/c the library is PNSE #38941)
- System.IO.Compression.ZipFile.Tests ([Wasm] Enable System.IO.Compression.ZipFile tests #39092)
- System.IO.Compression.Tests (WASM: Disable thread related tests on System.IO.Compression #39231)
- System.IO.FileSystem.DriveInfo.Tests (WASM: Fix System.IO.FileSystem.DriveInfo and Microsoft.VisualBasic.Core tests #39276)
- System.IO.FileSystem.Tests ([browser][file system] Tests System.IO.FileSystem #39768)
- System.IO.IsolatedStorage.Tests ([Wasm] Modify System.IO.IsolatedStorage to throw PNSE #38898)
- System.IO.MemoryMappedFiles.Tests (WASM: Fix System.IO.MemoryMappedFiles tests #39355)
- OOB: System.IO.Packaging.Tests ([wasm] Enable the library test suites that have no failures at the moment #39427)
- System.IO.Pipes.Tests (WASM: Add PlatformNotSupportedException for System.IO.Pipes #39089)
- System.Linq.Expressions.Tests ([wasm] Enable System.Linq.Expressions.Tests test suite #39772)
- System.Memory.Tests (WASM: Enable System.Memory tests #39319)
- System.Net.Http.Functional.Tests @akoeplinger
- System.Net.Http.Unit.Tests ([wasm] Enable System.Net.Http.Unit test suite #39181)
- System.Net.Mail.Unit.Tests (WebAssembly: Add PNSE for System.Net.Mail #38207)
- System.Net.NameResolution.Functional.Tests (WASM: Add PNSE for System.Net.NameResolution and disable tests #39219)
- System.Net.NameResolution.Pal.Tests (WASM: Add PNSE for System.Net.NameResolution and disable tests #39219)
- System.Net.Ping.Functional.Tests (WASM: Add PlatformNotSupportedException for System.Net.Ping #39091)
- System.Net.Primitives.Functional.Tests (WASM: Fix System.Net.Primitives and tests #39748)
- System.Net.Primitives.Pal.Tests (WASM: Fix System.Net.Primitives and tests #39748)
- System.Net.Security.Tests ([wasm] Modify System.Net.Security to throw PNSE #39358)
- System.Net.Sockets.Tests (Add PlatformNotSupportedException for Browser version of System.Net.Sockets #39346)
- System.Net.WebSockets.Client.Tests @akoeplinger
- System.ObjectModel.Tests (WASM: enable System.ObjectModel.Tests #39849)
- System.Private.Uri.Functional.Tests
- System.Xml.XmlSerializer.ReflectionOnly.Tests ([wasm] Enable several library test suites #39767)
- System.Xml.XmlSerializer.Tests ([wasm] Enable several library test suites #39767)
- System.Xml.Xsl.XslCompiledTransformApi.Tests ([wasm] Enable the library test suites that have no failures at the moment #39427)
- System.Xml.Xsl.XslTransformApi.Tests ([wasm] Enable the library test suites that have no failures at the moment #39427)
- System.Reflection.Metadata.Tests (WASM: Enable System.Reflection.* tests #39696)
- OOB: System.Reflection.MetadataLoadContext.Tests (WASM: Enable System.Reflection.MetadataLoadContext tests #39651)
- System.Reflection.CoreCLR.Tests (WASM: Enable System.Reflection.* tests #39696)
- System.Reflection.Tests (WASM: Enable System.Reflection.* tests #39696)
- OOB: System.Resources.Extensions.Tests (will be fixed by the ICU PR: ICU integration and asset loading overhaul #37971) + [wasm] Enable System.Resources.Extensions.Tests and System.Resources.ResourceManager.Tests test suites #39513
- System.Resources.Reader.Tests ([wasm] Enable the library test suites that have no failures at the moment #39427)
- System.Resources.ResourceManager.Tests (will be fixed by the ICU PR: ICU integration and asset loading overhaul #37971) + [wasm] Enable System.Resources.Extensions.Tests and System.Resources.ResourceManager.Tests test suites #39513
- OOB: System.Runtime.Caching.Tests ([Wasm] Disable tests for HostFileChangeMonitor #38386)
- System.Runtime.Extensions.Tests ([wasm] Addressing System.Runtime.Extensions failures #38996)
- System.Runtime.InteropServices.RuntimeInformation.Tests ([wasm] Addressing System.Runtime.InteropServices.RuntimeInformation.Tests failures #39073)
- System.Runtime.InteropServices.Tests ([wasm] Address System.Runtime.InteropServices failures on Browser #39188)
- System.Runtime.Loader.DefaultContext.Tests ([wasm] Address System.Runtime.Loader.DefaultContext failure #39204)
- System.Runtime.Loader.Tests ([wasm] Address System.Runtime.Loader failures #39306)
- System.Runtime.Serialization.Formatters.Tests (WASM: Enable System.Runtime.Serialization.Formatters tests #38948)
- System.Runtime.Serialization.Json.ReflectionOnly.Tests ([wasm] Enable System.Runtime.Serialization.Json test suite #39193)
- System.Runtime.Serialization.Json.Tests ([wasm] Enable System.Runtime.Serialization.Json test suite #39193)
- System.Runtime.Serialization.Primitives.Tests ([wasm] Enable System.Runtime.Serialization.Primitives test suite #39264)
- System.Runtime.Serialization.Xml.ReflectionOnly.Tests ([wasm] Enable System.Runtime.Serialization.Xml and System.Runtime.Serialization.Xml.ReflectionOnly test suites #39262)
- System.Runtime.Serialization.Xml.Tests ([wasm] Enable System.Runtime.Serialization.Xml and System.Runtime.Serialization.Xml.ReflectionOnly test suites #39262)
- System.Runtime.Tests (WASM: enable System.Runtime.Tests #39344)
- OOB: System.Security.Permissions.Tests ([wasm] Enable several library test suites #39767 )
- System.Text.RegularExpressions.Tests ([wasm] Address System.Text.RegularExpressions.Tests Failures #39470)
- System.Threading.Thread.Tests ([wasm] Enable System.Threading.Thread.Tests test suite #39826)
- System.Transactions.Local.Tests ([wasm] Enable the library test suites that have no failures at the moment #39427)
Suites that do not produce results
These tests most likely contain >= 1 thread related failures and need to be skipped
- OOB: System.Configuration.ConfigurationManager.Tests ([wasm] Skip System.Configuration.ConfigurationManager test suite #38749)
- System.Net.Http.Json.Functional.Tests (Rewrite System.Net.Http.Json functional tests to use a custom HttpMessageHandler #38733)
- System.Net.Mail.Functional.Tests (WebAssembly: Add PNSE for System.Net.Mail #38207)
- System.Net.NetworkInformation.Functional.Tests ([wasm] Modify System.Net.NetworkInformation to throw PNSE #38928)
- System.Net.Requests.Tests ([wasm] Add assembly level skip for System.Net.Requests on Browser #38674)
- System.Net.WebClient.Tests ([wasm] Skip System.Net.WebClient test suite #38857)
- System.Net.WebSockets.Tests ([wasm] Enable System.Net.WebSockets tests; mark System.Net.WebSocketsTests.WebSocketTests with an active issue #38808)
- OOB: System.Net.WebSockets.WebSocketProtocol.Tests ([wasm] Mark System.Net.WebSockets.Tests.WebSocketProtocolCreateTests class with an active issue #38854)
- System.Text.Json.Tests (Rewrite System.Text.Json stream tests to be async friendly and enable on WASM #38663)
- System.Threading.Tasks.Tests ([wasm] Disable problematic threading tests in System.Threading.Tasks.Tests #38355) Needs another look after async Task
- System.Threading.Channels.Tests ([wasm] Enable System.Threading.Channels.Tests #38849)
- System.Threading.Tasks.Dataflow.Tests (WASM: Enable System.Threading.Tasks.Dataflow tests #38723)
- System.Threading.Tasks.Extensions.Tests ([wasm] Enable System.Threading.Tasks.Extensions tests #38815)
Flaky suites
- System.Linq.Parallel.Tests, passes but had to be disabled again to crashes: [Wasm] Disable System.Linq.Parallel test suite #39305
Activity
Dotnet-GitSync-Bot commentedon Jun 26, 2020
I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.
safern commentedon Jul 8, 2020
Synced up with @steveisok and @tarekgh and it makes sense to keep them disabled from CI until ICU is turned on and then we can adjust/skip tests based on the ICU globalization story for WASM. If we want to get more test coverage when running on Invariant mode, we can add those tests to
System.Globalization/tests/Invariant/Invariant.Tests.csproj[wasm] Enable System.CodeDom.Tests test suite (#39626)
[wasm] Enable System.CodeDom.Tests test suite (dotnet#39626)
akoeplinger commentedon Sep 3, 2020
There are still three test suites we need to enable and also take a look at the flaky System.Linq.Parallel.Tests, keeping it open and in 6.0
lewing commentedon Jul 31, 2021
@steveisok what is the state of this now?
steveisok commentedon Jul 31, 2021
Once you get runtimeconfig.json support, the calendar test should light up. Mostly good it looks like. Probably can close this and deal with the few remaining in another issue.
lewing commentedon Jul 31, 2021
yeah I think this is safe to close not that we are tracking the remaining work elsewhere