The case of the not so ConcurrentDictionary

03/16/2018

2 minutes to read

In this article

I was looking at our DevOps dashboards and saw some really weird patterns:

So I pinged my colleague who owns this service and he noticed it was actual very predictable:

Like clockwork, once a minute - he went further and got a PerfView which showed high contention on a newly added ConcurrentDictionary:

He then asked me to take a look since that ConcurrentDictionary was added on my suggestion to work around another issue (which I will blog about one day). Having had that problem before, I figured we either had a hot spot or a hashing function problem - so I got a dump of the process to see which (I could have saved some time and looked at the source...but as they say, there's nothing like a good dump).

0:000> !do 0000015d19e676f0 Name: System.Collections.Concurrent.ConcurrentDictionary`2 MethodTable: 00007ff8e18f9728 EEClass: 00007ff8e18c5de0 Size: 64(0x40) bytes File: D:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll Fields: MT Field Offset Type VT Attr Value Name 00007ff8e187d930 4001830 8 ....Byte, mscorlib]] 0 instance 0000015e3a819078 m_tables 00007ff93b3c4300 4001831 10 ...Canon, mscorlib]] 0 instance 0000000000000000 m_comparer 00007ff93b3b1f28 4001832 30 System.Boolean 1 instance 1 m_growLockArray 00007ff93b3a9288 4001833 20 System.Int32 1 instance 0 m_keyRehashCount 00007ff93b3a9288 4001834 24 System.Int32 1 instance 256 m_budget 0000000000000000 4001835 18 SZARRAY 0 instance 0000000000000000 m_serializationArray 00007ff93b3a9288 4001836 28 System.Int32 1 instance 0 m_serializationConcurrencyLevel 00007ff93b3a9288 4001837 2c System.Int32 1 instance 0 m_serializationCapacity 00007ff93b3b1f28 400183b 10 System.Boolean 1 static <no information> 0:000> !do 0000015e3a819078 Name: System.Collections.Concurrent.ConcurrentDictionary`2+Tables MethodTable: 00007ff8e18fafd0 EEClass: 00007ff8e18c6ab0 Size: 48(0x30) bytes File: D:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll Fields: MT Field Offset Type VT Attr Value Name 0000000000000000 400341d 8 SZARRAY 0 instance 0000015e3a816ca8 m_buckets 00007ff93b3a6fc0 400341e 10 System.Object[] 0 instance 0000015e3a816290 m_locks 00007ff93b3a9220 400341f 18 System.Int32[] 0 instance 0000015e3a817e28 m_countPerLock 00007ff93b3c4300 4003420 20...Canon, mscorlib]] 0 instance 0000015d19e677a0 m_comparer 0:000> !DumpArray 0000015e3a816ca8 Name: System.Collections.Concurrent.ConcurrentDictionary`2+Node MethodTable: 00007ff8e18fae50 EEClass: 00007ff93ad6aa00 Size: 4480(0x1180) bytes Array: Rank 1, Number of elements 557, Type CLASS Element Methodtable: 00007ff8e18fad88 [0] null [1] null <...> [428] null [429] 0000015d46de8280 [430] null <...> [556] null 0:000> !do 0000015d46de8280 Name: System.Collections.Concurrent.ConcurrentDictionary`2+Node MethodTable: 00007ff8e18fad88 EEClass: 00007ff8e18c6990 Size: 40(0x28) bytes File: D:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll Fields: MT Field Offset Type VT Attr Value Name 00007ff93b3abf10 4003421 8 System.__Canon 0 instance 0000015d46de8240 m_key 00007ff93b3a8940 4003422 1c System.Byte 1 instance 0 m_value 00007ff8e187d800 4003423 10 ....Byte, mscorlib]] 0 instance 0000015e47c672e8 m_next 00007ff93b3a9288 4003424 18 System.Int32 1 instance 37103870 m_hashcode 0:000> !do 0000015e47c672e8 Name: System.Collections.Concurrent.ConcurrentDictionary`2+Node MethodTable: 00007ff8e18fad88 EEClass: 00007ff8e18c6990 Size: 40(0x28) bytes File: D:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll Fields: MT Field Offset Type VT Attr Value Name 00007ff93b3abf10 4003421 8 System.__Canon 0 instance 0000015e47c672a8 m_key 00007ff93b3a8940 4003422 1c System.Byte 1 instance 0 m_value 00007ff8e187d800 4003423 10 ....Byte, mscorlib]] 0 instance 0000015e4781f3f8 m_next 00007ff93b3a9288 4003424 18 System.Int32 1 instance 37103870 m_hashcode

Sure enough, there's only one bucket occupied, and all of the items in that bucket have the same hash code, so our ConcurrentDictionary is really a giant linked list, with a giant lock on top...

Our ConcurrentDictionary's keys are System.EventHandler which is really a delegate - which default HashCode implementation is...the hash code of the underlying type which means all of our delegates have the same hashcode, hence the same bucket....DOH!