NullReferenceException inside ConcurrentDictionary.GrowTable

@jkotas

Pipeline: https://dev.azure.com/dnceng/public/_build/results?buildId=1764865&view=results
Job: Installer Build and Test coreclr Linux_arm Release

/root/runtime/artifacts/bin/Crossgen2Tasks/Debug/net7.0/Microsoft.NET.CrossGen.targets(463,5): error : Error: Object reference not set to an instance of an object. [/root/runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.Composite.sfxproj]
/root/runtime/artifacts/bin/Crossgen2Tasks/Debug/net7.0/Microsoft.NET.CrossGen.targets(463,5): error : System.NullReferenceException: Object reference not set to an instance of an object. [/root/runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.Composite.sfxproj]
/root/runtime/artifacts/bin/Crossgen2Tasks/Debug/net7.0/Microsoft.NET.CrossGen.targets(463,5): error :    at System.Collections.Concurrent.ConcurrentDictionary`2.GrowTable(Tables tables) [/root/runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.Composite.sfxproj]
/root/runtime/artifacts/bin/Crossgen2Tasks/Debug/net7.0/Microsoft.NET.CrossGen.targets(463,5): error :    at System.Collections.Concurrent.ConcurrentDictionary`2.TryAddInternal(TKey key, Nullable`1 nullableHashcode, TValue value, Boolean updateIfExists, Boolean acquireLock, TValue& resultingValue) [/root/runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.Composite.sfxproj]
/root/runtime/artifacts/bin/Crossgen2Tasks/Debug/net7.0/Microsoft.NET.CrossGen.targets(463,5): error :    at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory) [/root/runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.Composite.sfxproj]
/root/runtime/artifacts/bin/Crossgen2Tasks/Debug/net7.0/Microsoft.NET.CrossGen.targets(463,5): error :    at ILCompiler.DependencyAnalysis.NodeCache`2.GetOrAdd(TKey key) [/root/runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.Composite.sfxproj]
/root/runtime/artifacts/bin/Crossgen2Tasks/Debug/net7.0/Microsoft.NET.CrossGen.targets(463,5): error :    at ILCompiler.DependencyAnalysis.NodeFactory.CompiledMethodNode(MethodDesc method) [/root/runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.Composite.sfxproj]
/root/runtime/artifacts/bin/Crossgen2Tasks/Debug/net7.0/Microsoft.NET.CrossGen.targets(463,5): error :    at ILCompiler.Compilation.RootingServiceProvider.AddCompilationRoot(MethodDesc method, Boolean rootMinimalDependencies, String reason) [/root/runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.Composite.sfxproj]
/root/runtime/artifacts/bin/Crossgen2Tasks/Debug/net7.0/Microsoft.NET.CrossGen.targets(463,5): error :    at ILCompiler.ReadyToRunRootProvider.RootMethods(TypeDesc type, String reason, IRootingServiceProvider rootProvider) [/root/runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.Composite.sfxproj]
/root/runtime/artifacts/bin/Crossgen2Tasks/Debug/net7.0/Microsoft.NET.CrossGen.targets(463,5): error :    at ILCompiler.ReadyToRunRootProvider.AddCompilationRoots(IRootingServiceProvider rootProvider) [/root/runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.Composite.sfxproj]
/root/runtime/artifacts/bin/Crossgen2Tasks/Debug/net7.0/Microsoft.NET.CrossGen.targets(463,5): error :    at ILCompiler.Compilation..ctor(DependencyAnalyzerBase`1 dependencyGraph, NodeFactory nodeFactory, IEnumerable`1 compilationRoots, ILProvider ilProvider, DevirtualizationManager devirtualizationManager, IEnumerable`1 modulesBeingInstrumented, Logger logger, InstructionSetSupport instructionSetSupport) [/root/runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.Composite.sfxproj]
/root/runtime/artifacts/bin/Crossgen2Tasks/Debug/net7.0/Microsoft.NET.CrossGen.targets(463,5): error :    at ILCompiler.ReadyToRunCodegenCompilation..ctor(DependencyAnalyzerBase`1 dependencyGraph, NodeFactory nodeFactory, IEnumerable`1 roots, ILProvider ilProvider, Logger logger, DevirtualizationManager devirtualizationManager, IEnumerable`1 inputFiles, String compositeRootPath, InstructionSetSupport instructionSetSupport, Boolean resilient, Boolean generateMapFile, Boolean generateMapCsvFile, Boolean generatePdbFile, Func`2 printReproInstructions, String pdbPath, Boolean generatePerfMapFile, String perfMapPath, Int32 perfMapFormatVersion, Boolean generateProfileFile, Int32 parallelism, ProfileDataManager profileData, ReadyToRunMethodLayoutAlgorithm methodLayoutAlgorithm, ReadyToRunFileLayoutAlgorithm fileLayoutAlgorithm, Int32 customPESectionAlignment, Boolean verifyTypeAndFieldLayout) [/root/runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.Composite.sfxproj]
/root/runtime/artifacts/bin/Crossgen2Tasks/Debug/net7.0/Microsoft.NET.CrossGen.targets(463,5): error :    at ILCompiler.ReadyToRunCodegenCompilationBuilder.ToCompilation() [/root/runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.Composite.sfxproj]
/root/runtime/artifacts/bin/Crossgen2Tasks/Debug/net7.0/Microsoft.NET.CrossGen.targets(463,5): error :    at ILCompiler.Program.RunSingleCompilation(Dictionary`2 inFilePaths, InstructionSetSupport instructionSetSupport, String compositeRootPath, Dictionary`2 unrootedInputFilePaths, HashSet`1 versionBubbleModulesHash, CompilerTypeSystemContext typeSystemContext) [/root/runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.Composite.sfxproj]
/root/runtime/artifacts/bin/Crossgen2Tasks/Debug/net7.0/Microsoft.NET.CrossGen.targets(463,5): error :    at ILCompiler.Program.Run(String[] args) [/root/runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.Composite.sfxproj]
/root/runtime/artifacts/bin/Crossgen2Tasks/Debug/net7.0/Microsoft.NET.CrossGen.targets(463,5): error :    at ILCompiler.Program.Main(String[] args) [/root/runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.Composite.sfxproj]
/root/runtime/artifacts/bin/Crossgen2Tasks/Debug/net7.0/Microsoft.NET.CrossGen.targets(351,5): error NETSDK1096: Optimizing assemblies for performance failed. You can either exclude the failing assemblies from being optimized, or set the PublishReadyToRun property to false. [/root/runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.Composite.sfxproj]

@jkotas do you think this could also be a symptom of #68443?

It can be a symptom of #68443. It is hard to tell with any level of confidence without a crash dump.

Another similar-looking-but-not-quite example in #69068:

    Generating native code
    Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
       at System.Collections.Concurrent.ConcurrentDictionary`2.TryGetValueInternal(TKey key, Int32 hashcode, TValue& value)
       at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
       at ILCompiler.DependencyAnalysis.NodeFactory.NodeCache`2.GetOrAdd(TKey key) in /_/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/NodeFactory.cs:line 156
       at ILCompiler.DependencyAnalysis.NodeFactory.MethodMetadata(MethodDesc method) in /_/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/NodeFactory.cs:line 998
       at ILCompiler.UsageBasedMetadataManager.GeneratedTypesAndCodeMetadataPolicy.GeneratesMetadata(MethodDesc methodDef) in /_/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/UsageBasedMetadataManager.cs:line 873
       at ILCompiler.Metadata.Transform`1.HandleQualifiedMethod(MethodDesc method) in /_/src/coreclr/tools/aot/ILCompiler.MetadataTransform/ILCompiler/Metadata/Transform.Method.cs:line 36
       at ILCompiler.MetadataManager.CreateStackTraceRecord(MetadataTransform transform, MethodDesc method) in /_/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/MetadataManager.cs:line 573

Pipeline run: https://dev.azure.com/dnceng/public/_build/results?buildId=1769350&view=results
Job name: Build windows x64 debug NativeAOT

Failed again in: runtime-coreclr outerloop 20220526.2

Failed test:

R2R-CG2 windows x64 Checked no_tiered_compilation @ Windows.10.Amd64.Open

- JIT\\opt\\virtualstubdispatch\\bigvtbl\\bigvtbl_cs_do\\bigvtbl_cs_do.cmd

Error message:

Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
at Internal.NativeFormat.VertexArray.VertexTree.Save(NativeWriter writer) in //src/coreclr/tools/Common/Internal/NativeFormat/NativeFormatWriter.cs:line 1806
at Internal.NativeFormat.NativeWriter.Save(Stream stream) in //src/coreclr/tools/Common/Internal/NativeFormat/NativeFormatWriter.cs:line 334
at ILCompiler.DependencyAnalysis.ReadyToRun.DebugInfoTableNode.GetData(NodeFactory factory, Boolean relocsOnly) in //src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/DependencyAnalysis/ReadyToRun/DebugInfoTableNode.cs:line 123
at ILCompiler.DependencyAnalysis.ReadyToRunObjectWriter.EmitPortableExecutable() in //src/coreclr/tools/aot/ILCompiler.ReadyToRun/CodeGen/ReadyToRunObjectWriter.cs:line 256
at ILCompiler.ReadyToRunCodegenCompilation.Compile(String outputFile) in //src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/ReadyToRunCodegenCompilation.cs:line 347
at ILCompiler.Program.RunSingleCompilation(Dictionary2 inFilePaths, InstructionSetSupport instructionSetSupport, String compositeRootPath, Dictionary2 unrootedInputFilePaths, HashSet`1 versionBubbleModulesHash, CompilerTypeSystemContext typeSystemContext) in //src/coreclr/tools/aot/crossgen2/Program.cs:line 769
at ILCompiler.Program.Run(String[] args) in //src/coreclr/tools/aot/crossgen2/Program.cs:line 545
at ILCompiler.Program.Main(String[] args) in //src/coreclr/tools/aot/crossgen2/Program.cs:line 1046

Return code:      1
Raw output file:      C:\h\w\B9CC0A01\w\C1A90AAD\uploads\Reports\JIT.opt\virtualstubdispatch\bigvtbl\bigvtbl_cs_do\bigvtbl_cs_do.output.txt
Raw output:
BEGIN EXECUTION
bigvtbl_cs_do.dll
1 file(s) copied.
20:36:39.30
Response file: C:\h\w\B9CC0A01\w\C1A90AAD\e\JIT\opt\virtualstubdispatch\bigvtbl\bigvtbl_cs_do\bigvtbl_cs_do.dll.rsp
C:\h\w\B9CC0A01\w\C1A90AAD\e\JIT\opt\virtualstubdispatch\bigvtbl\bigvtbl_cs_do\IL-CG2\bigvtbl_cs_do.dll
-o:C:\h\w\B9CC0A01\w\C1A90AAD\e\JIT\opt\virtualstubdispatch\bigvtbl\bigvtbl_cs_do\bigvtbl_cs_do.dll
--targetarch:x64
--verify-type-and-field-layout
--method-layout:random
-r:C:\h\w\B9CC0A01\p\System..dll
-r:C:\h\w\B9CC0A01\p\Microsoft..dll
-r:C:\h\w\B9CC0A01\p\mscorlib.dll
-r:C:\h\w\B9CC0A01\p\netstandard.dll
-O
" "dotnet" "C:\h\w\B9CC0A01\p\crossgen2\crossgen2.dll" @"C:\h\w\B9CC0A01\w\C1A90AAD\e\JIT\opt\virtualstubdispatch\bigvtbl\bigvtbl_cs_do\bigvtbl_cs_do.dll.rsp"   -r:C:\h\w\B9CC0A01\w\C1A90AAD\e\JIT\opt\virtualstubdispatch\bigvtbl\bigvtbl_cs_do\IL-CG2*.dll"
Emitting R2R PE file: C:\h\w\B9CC0A01\w\C1A90AAD\e\JIT\opt\virtualstubdispatch\bigvtbl\bigvtbl_cs_do\bigvtbl_cs_do.dll
20:37:01.30
Crossgen2 failed with exitcode - -1073741819
Test Harness Exitcode is : 1
To run the test:

set CORE_ROOT=C:\h\w\B9CC0A01\p
C:\h\w\B9CC0A01\w\C1A90AAD\e\JIT\opt\virtualstubdispatch\bigvtbl\bigvtbl_cs_do\bigvtbl_cs_do.cmd
Expected: True
Actual:   False


Stack trace
   at JIT_opt._virtualstubdispatch_bigvtbl_bigvtbl_cs_do_bigvtbl_cs_do_._virtualstubdispatch_bigvtbl_bigvtbl_cs_do_bigvtbl_cs_do_cmd()

Failed in #70251

2022-06-14T06:19:40.6750905Z     Reflection -> D:\a\_work\1\s\artifacts\tests\coreclr\windows.x64.Debug\nativeaot\SmokeTests\Reflection\Reflection\Reflection.dll
2022-06-14T06:19:48.1234901Z        Creating library D:\a\_work\1\s\artifacts\tests\coreclr\windows.x64.Debug\nativeaot\SmokeTests\Exceptions\Exceptions\native\Exceptions.lib and object D:\a\_work\1\s\artifacts\tests\coreclr\windows.x64.Debug\nativeaot\SmokeTests\Exceptions\Exceptions\native\Exceptions.exp
2022-06-14T06:19:48.2128877Z     Optimizing assemblies for size may change the behavior of the app. Be sure to test after publishing. See: https://aka.ms/dotnet-illink
2022-06-14T06:19:48.2982092Z     Generating native code
2022-06-14T06:19:49.0561844Z     Dataflow -> D:\a\_work\1\s\artifacts\tests\coreclr\windows.x64.Debug\nativeaot\SmokeTests\Dataflow\Dataflow\Dataflow.dll
2022-06-14T06:20:44.7213835Z     Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
2022-06-14T06:20:44.7215826Z        at System.Collections.Concurrent.ConcurrentDictionary`2.TryGetValueInternal(TKey key, Int32 hashcode, TValue& value)
2022-06-14T06:20:46.3259241Z        at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
2022-06-14T06:20:46.5183248Z        at ILCompiler.DependencyAnalysis.NodeFactory.NodeCache`2.GetOrAdd(TKey key) in /_/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/NodeFactory.cs:line 156
2022-06-14T06:20:46.7880895Z        at ILCompiler.DependencyAnalysis.NodeFactory.MethodMetadata(MethodDesc method) in /_/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/NodeFactory.cs:line 1009
2022-06-14T06:20:48.7366778Z        at ILCompiler.UsageBasedMetadataManager.GeneratedTypesAndCodeMetadataPolicy.GeneratesMetadata(MethodDesc methodDef) in /_/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/UsageBasedMetadataManager.cs:line 884
2022-06-14T06:20:49.7672564Z        at ILCompiler.Metadata.Transform`1.HandleQualifiedMethod(MethodDesc method) in /_/src/coreclr/tools/aot/ILCompiler.MetadataTransform/ILCompiler/Metadata/Transform.Method.cs:line 36
2022-06-14T06:20:49.7965896Z        at ILCompiler.MetadataManager.CreateStackTraceRecord(MetadataTransform transform, MethodDesc method) in /_/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/MetadataManager.cs:line 594
2022-06-14T06:20:49.8022336Z        at ILCompiler.GeneratingMetadataManager.ComputeMetadata[TPolicy](TPolicy policy, NodeFactory factory, Byte[]& metadataBlob, List`1& typeMappings, List`1& methodMappings, List`1& fieldMappings, List`1& stackTraceMapping) in /_/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/GeneratingMetadataManager.cs:line 88
2022-06-14T06:20:49.8063628Z        at ILCompiler.UsageBasedMetadataManager.ComputeMetadata(NodeFactory factory, Byte[]& metadataBlob, List`1& typeMappings, List`1& methodMappings, List`1& fieldMappings, List`1& stackTraceMapping) in /_/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/UsageBasedMetadataManager.cs:line 199
2022-06-14T06:20:49.8094842Z        at ILCompiler.MetadataManager.EnsureMetadataGenerated(NodeFactory factory) in /_/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/MetadataManager.cs:line 574
2022-06-14T06:20:49.8120382Z        at ILCompiler.MetadataManager.GetMetadataBlob(NodeFactory factory) in /_/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/MetadataManager.cs:line 651
2022-06-14T06:20:49.8153898Z        at ILCompiler.DependencyAnalysis.MetadataNode.GetData(NodeFactory factory, Boolean relocsOnly) in /_/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/MetadataNode.cs:line 45
2022-06-14T06:20:49.8178075Z        at ILCompiler.DependencyAnalysis.ObjectWriter.EmitObject(String objectFilePath, IReadOnlyCollection`1 nodes, NodeFactory factory, ObjectWritingOptions options, IObjectDumper dumper, Logger logger) in /_/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ObjectWriter.cs:line 1018
2022-06-14T06:20:49.8195831Z        at ILCompiler.RyuJitCompilation.CompileInternal(String outputFile, ObjectDumper dumper) in /_/src/coreclr/tools/aot/ILCompiler.RyuJit/Compiler/RyuJitCompilation.cs:line 103
2022-06-14T06:20:49.8212995Z        at ILCompiler.Compilation.ILCompiler.ICompilation.Compile(String outputFile, ObjectDumper dumper) in /_/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/Compilation.cs:line 526
2022-06-14T06:20:49.8274259Z        at ILCompiler.Program.Run(String[] args) in /_/src/coreclr/tools/aot/ILCompiler/Program.cs:line 893
2022-06-14T06:20:49.8360953Z        at ILCompiler.Program.Main(String[] args) in /_/src/coreclr/tools/aot/ILCompiler/Program.cs:line 1076

I have a more or less reproducible crash attached. Is this related?

I'm on .NET 6.0, VS 2022.

Object reference not set to an instance of an object.

at System.Collections.Concurrent.ConcurrentDictionary`2.TryGetValueInternal(TKey key, Int32 hashcode, TValue& value)
in /_/src/libraries/System.Collections.Concurrent/src/System/Collections/Concurrent/ConcurrentDictionary.cs:line 498

This is the line. The visible IP is in two lines below, probably because it is optimized.

                else
                {
498:                for (Node? n = Volatile.Read(ref tables.GetBucket(hashcode)); n != null; n = n._next)
                    {
                        if (hashcode == n._hashcode && _defaultComparer.Equals(n._key, key))
                        {
                            value = n._value;
                            return true;
                        }
                    }
                }

Seems to happen at large item counts (Count = 94644926).
I verified it is not related to exhaustion of physical RAM.

I don't see which value exactly is null.

n
Unable to read memory

tables
Cannot obtain value of the local variable or argument because it is not available at this instruction pointer, possibly because it has been optimized away.

_tables
{System.Collections.Concurrent.ConcurrentDictionary<X, Y>.Tables}
    _buckets: {System.Collections.Concurrent.ConcurrentDictionary<X, Y>.Node[587202557]}
    _countPerLock: {int[1536]}
    _fastModBucketsMultiplier: 31414618097
    _locks: {object[1536]}

_tables.GetBucket(hashcode)
{System.Collections.Concurrent.ConcurrentDictionary<X, Y>.Node}
    _hashcode: 50331649
    _key: {xxx}
    _next: {System.Collections.Concurrent.ConcurrentDictionary<X, Y>.Node}
    _value: {yyy}

As far as I can tell this occurs even when no second thread is currently holding a lock.

Update

The linked list had a length of 42 before next turned into Unable to read memory.
I could reproduce this scenario precisely.

When I replaced the hash function with a better one, the problem disappeared.

I'm wondering whether this might be related to #66079 - in this issue we're observing unbounded growth of certain dictionaries due to the generic explosion, at one point I observed 32-bit integer wraparound that might theoretically explain the nullref. I continue working on the fix for 66079 with the hope that this may go away once it's available.

@nerai

I have a more or less reproducible crash attached.

@nerai I do not see the repro attached. Could you please share it?

@nerai

@nerai I do not see the repro attached. Could you please share it?

Sorry, I worded that poorly. The crash was reproducible, but it was part of a large in-prod project. It took hours/days to manifest during runtime, so it's not easy to create a separate project that I could share here.

@nerai

Hi @nerai, would you be able to share a few dumps of the failure so we could investigate further? Thanks

No repro since June 18th?

Ok to close then and re-evaluate if it shows up again.

Navigation Menu

Repository navigation

NullReferenceException inside ConcurrentDictionary.GrowTable #69192

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

NullReferenceException inside ConcurrentDictionary.GrowTable #69192

Description

Activity

jkotas commented on May 11, 2022

jakobbotsch commented on May 13, 2022

VincentBu commented on May 27, 2022

AaronRobinsonMSFT commented on Jun 14, 2022

nerai commented on Jun 17, 2022

trylek commented on Jun 20, 2022

AntonLapounov commented on Aug 12, 2022

nerai commented on Aug 12, 2022

mangod9 commented on Aug 24, 2022

AntonLapounov commented on Sep 6, 2022

mangod9 commented on Sep 6, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions