Race Condition in WebAppTypeFinder leads to TypeInitializationException in System.Linq.Dynamic.Core (Multi-instance environments)

[JoPetra]

In high-concurrency environments or multi-instance deployments (e.g., Azure App Service with scale-out enabled), a race condition occurs within WebAppTypeFinder when multiple threads attempt to resolve assemblies simultaneously during the first few requests of an AppDomain.

This specifically impacts the initialization of System.Linq.Dynamic.Core, which is used by BaseNopValidator to set database validation rules. When the static constructor of System.Linq.Dynamic.Core.Parser.PredefinedTypesHelper calls back into nopCommerce to find assemblies, the non-thread-safe collection in WebAppTypeFinder becomes corrupted, resulting in a System.InvalidOperationException.

nopCommerce version: 4.93
Hosting: Azure App Service (Multiple Instances)
Framework: .NET 9

Steps to reproduce:

Deploy nopCommerce to an environment with multiple instances or high initial concurrent traffic.

Access the Admin area (e.g., All Settings or Customer Create) immediately after an AppDomain restart.

Observe intermittent Autofac.Core.DependencyResolutionException related to SettingValidator or CustomerValidator.

Expected behavior:
WebAppTypeFinder should handle concurrent assembly lookups safely without corrupting internal collections.

Actual behavior (Stack Trace Snippet):

Autofac.Core.DependencyResolutionException: An exception was thrown while activating Nop.Web.Areas.Admin.Validators.Customers.CustomerValidator.
---> Autofac.Core.DependencyResolutionException: An exception was thrown while invoking the constructor 'Void .ctor(Nop.Core.Domain.Customers.CustomerSettings, Nop.Services.Customers.ICustomerService, Nop.Services.Localization.ILocalizationService, Nop.Services.Directory.IStateProvinceService)' on type 'CustomerValidator'.
---> System.TypeInitializationException: The type initializer for 'System.Linq.Dynamic.Core.Parser.KeywordsHelper' threw an exception.
---> System.TypeInitializationException: The type initializer for 'System.Linq.Dynamic.Core.Parser.PredefinedTypesHelper' threw an exception.
---> System.IO.FileLoadException: Could not load file or assembly 'Microsoft.EntityFrameworkCore.DynamicLinq, Version=1.6.7.0, Culture=neutral, PublicKeyToken=974e7e1b462f3693'. An operation is not legal in the current state. (0x80131509)
File name: 'Microsoft.EntityFrameworkCore.DynamicLinq, Version=1.6.7.0, Culture=neutral, PublicKeyToken=974e7e1b462f3693'
---> System.InvalidOperationException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.
   at System.Collections.Generic.Dictionary2.FindValue(TKey key)    at System.Collections.Generic.Dictionary2.TryGetValue(TKey key, TValue& value)
   at Nop.Core.Infrastructure.WebAppTypeFinder.GetAssemblyByName(String assemblyFullName) in D:\AiGlean\AiGlean-Commerce\src\Libraries\Nop.Core\Infrastructure\WebAppTypeFinder.cs:line 272
   at System.Runtime.Loader.AssemblyLoadContext.InvokeResolveEvent(ResolveEventHandler eventHandler, RuntimeAssembly assembly, String name)
   at System.Reflection.RuntimeAssembly.g____PInvoke|49_0(NativeAssemblyNameParts* __pAssemblyNameParts_native, ObjectHandleOnStack __requestingAssembly_native, StackCrawlMarkHandle __stackMark_native, Int32 __throwOnFileNotFound_native, ObjectHandleOnStack __assemblyLoadContext_native, ObjectHandleOnStack __retAssembly_native)
   at System.Reflection.RuntimeAssembly.g____PInvoke|49_0(NativeAssemblyNameParts* __pAssemblyNameParts_native, ObjectHandleOnStack __requestingAssembly_native, StackCrawlMarkHandle __stackMark_native, Int32 __throwOnFileNotFound_native, ObjectHandleOnStack __assemblyLoadContext_native, ObjectHandleOnStack __retAssembly_native)
   at System.Reflection.RuntimeAssembly.InternalLoad(AssemblyName assemblyName, StackCrawlMark& stackMark, AssemblyLoadContext assemblyLoadContext, RuntimeAssembly requestingAssembly, Boolean throwOnFileNotFound)
   at System.Reflection.TypeNameResolver.ResolveAssembly(AssemblyName assemblyName)
   at System.Reflection.TypeNameResolver.GetType(String escapedTypeName, ReadOnlySpan1 nestedTypeNames, TypeName parsedName)    at System.Reflection.TypeNameResolver.Resolve(TypeName typeName)    at System.Reflection.TypeNameResolver.GetType(String typeName, Func2 assemblyResolver, Func4 typeResolver, Assembly requestingAssembly, Boolean throwOnError, Boolean ignoreCase, Boolean extensibleParser)    at System.Type.GetType(String typeName)    at System.Linq.Dynamic.Core.Parser.PredefinedTypesHelper..cctor()    --- End of inner exception stack trace ---    at System.Linq.Dynamic.Core.Parser.KeywordsHelper..cctor()    --- End of inner exception stack trace ---    at System.Linq.Dynamic.Core.Parser.KeywordsHelper..ctor(ParsingConfig config)    at System.Linq.Dynamic.Core.Parser.ExpressionParser..ctor(ParameterExpression[] parameters, String expression, Object[] values, ParsingConfig parsingConfig, Boolean usedForOrderBy)    at System.Linq.Dynamic.Core.Parser.ExpressionParser..ctor(ParameterExpression[] parameters, String expression, Object[] values, ParsingConfig parsingConfig)    at System.Linq.Dynamic.Core.DynamicExpressionParser.ParseLambda(Type delegateType, ParsingConfig parsingConfig, Boolean createParameterCtor, ParameterExpression[] parameters, Type resultType, String expression, Object[] values)    at System.Linq.Dynamic.Core.DynamicExpressionParser.ParseLambda(ParsingConfig parsingConfig, Boolean createParameterCtor, ParameterExpression[] parameters, Type resultType, String expression, Object[] values)    at System.Linq.Dynamic.Core.DynamicExpressionParser.ParseLambda[T,TResult](ParsingConfig parsingConfig, Boolean createParameterCtor, String expression, Object[] values)    at Nop.Web.Framework.Validators.BaseNopValidator1.<>c.b__1_3(NopEntityFieldDescriptor field) in D:\AiGlean\AiGlean-Commerce\src\Presentation\Nop.Web.Framework\Validators\BaseNopValidator.cs:line 52
   at System.Linq.Enumerable.ArrayWhereSelectIterator2.ToList(ReadOnlySpan1 source, Func2 predicate, Func2 selector)
   at System.Linq.Enumerable.ListWhereSelectIterator2.ToList()    at System.Linq.Enumerable.ToList[TSource](IEnumerable1 source)
   at Nop.Web.Framework.Validators.BaseNopValidator1.SetStringPropertiesMaxLength(NopEntityDescriptor entityDescriptor, String[] filterPropertyNames)    at Nop.Web.Framework.Validators.BaseNopValidator1.SetDatabaseValidationRules[TEntity](String[] filterStringPropertyNames) in D:\AiGlean\AiGlean-Commerce\src\Presentation\Nop.Web.Framework\Validators\BaseNopValidator.cs:line 28
   at Nop.Web.Areas.Admin.Validators.Customers.CustomerValidator..ctor(CustomerSettings customerSettings, ICustomerService customerService, ILocalizationService localizationService, IStateProvinceService stateProvinceService) in D:\AiGlean\AiGlean-Commerce\src\Presentation\Nop.Web\Areas\Admin\Validators\Customers\CustomerValidator.cs:line 119
   at lambda_method700(Closure, Object[])
   at Autofac.Core.Activators.Reflection.BoundConstructor.Instantiate()
   --- End of inner exception stack trace ---
   at Autofac.Core.Activators.Reflection.BoundConstructor.Instantiate()
   at Autofac.Core.Activators.Reflection.ReflectionActivator.<>c__DisplayClass14_0.b__0(ResolveRequestContext context, Action1 next)    at Autofac.Core.Resolving.Middleware.DisposalTrackingMiddleware.Execute(ResolveRequestContext context, Action1 next)
   at Autofac.Core.Resolving.Middleware.ActivatorErrorHandlingMiddleware.Execute(ResolveRequestContext context, Action1 next)    --- End of inner exception stack trace ---    at Autofac.Core.Resolving.Middleware.ActivatorErrorHandlingMiddleware.Execute(ResolveRequestContext context, Action1 next)
   at Autofac.Core.Lifetime.LifetimeScope.CreateSharedInstance(Guid id, Func1 creator)    at Autofac.Core.Lifetime.LifetimeScope.CreateSharedInstance(Guid primaryId, Nullable1 qualifyingId, Func1 creator)    at Autofac.Core.Resolving.Middleware.SharingMiddleware.Execute(ResolveRequestContext context, Action1 next)
   at Autofac.Core.Resolving.Middleware.CircularDependencyDetectorMiddleware.Execute(ResolveRequestContext context, Action1 next)    at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, ResolveRequest& request)    at Autofac.Core.Resolving.ResolveOperation.ExecuteOperation(ResolveRequest& request)    at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable1 parameters, Object& instance)
   at Autofac.ResolutionExtensions.ResolveOptionalService(IComponentContext context, Service service, IEnumerable1 parameters)    at FluentValidation.AspNetCore.ValidatorDescriptorCache.GetCachedDescriptor(ClientValidatorProviderContext context, IHttpContextAccessor httpContextAccessor) in /_/src/FluentValidation.AspNetCore/ValidatorDescriptorCache.cs:line 59    at FluentValidation.AspNetCore.FluentValidationClientModelValidatorProvider.CreateValidators(ClientValidatorProviderContext context) in /_/src/FluentValidation.AspNetCore/FluentValidationClientModelValidatorProvider.cs:line 66    at Microsoft.AspNetCore.Mvc.ModelBinding.Validation.CompositeClientModelValidatorProvider.CreateValidators(ClientValidatorProviderContext context)    at Microsoft.AspNetCore.Mvc.ModelBinding.Validation.ClientValidatorCache.GetValidators(ModelMetadata metadata, IClientModelValidatorProvider validatorProvider)    at Microsoft.AspNetCore.Mvc.ViewFeatures.DefaultValidationHtmlAttributeProvider.AddValidationAttributes(ViewContext viewContext, ModelExplorer modelExplorer, IDictionary2 attributes)
   at Microsoft.AspNetCore.Mvc.ViewFeatures.DefaultHtmlGenerator.GenerateInput(ViewContext viewContext, InputType inputType, ModelExplorer modelExplorer, String expression, Object value, Boolean useViewData, Boolean isChecked, Boolean setId, Boolean isExplicitValue, String format, IDictionary2 htmlAttributes)    at Microsoft.AspNetCore.Mvc.TagHelpers.InputTagHelper.GenerateHidden(ModelExplorer modelExplorer, IDictionary2 htmlAttributes)
   at Microsoft.AspNetCore.Mvc.TagHelpers.InputTagHelper.Process(TagHelperContext context, TagHelperOutput output)
   at Microsoft.AspNetCore.Razor.TagHelpers.TagHelper.ProcessAsync(TagHelperContext context, TagHelperOutput output)
   at Microsoft.AspNetCore.Razor.Runtime.TagHelpers.TagHelperRunner.RunAsync(TagHelperExecutionContext executionContext)
   at AspNetCoreGeneratedDocument.Areas_Admin_Views_Customer__CreateOrUpdate.ExecuteAsync()
   at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderPageCoreAsync(IRazorPage page, ViewContext context)
   at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderPageAsync(IRazorPage page, ViewContext context, Boolean invokeViewStarts)
   at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderAsync(ViewContext context)
   at Microsoft.AspNetCore.Mvc.ViewFeatures.HtmlHelper.RenderPartialCoreAsync(String partialViewName, Object model, ViewDataDictionary viewData, TextWriter writer)
   at Microsoft.AspNetCore.Mvc.ViewFeatures.HtmlHelper.PartialAsync(String partialViewName, Object model, ViewDataDictionary viewData)
   at AspNetCoreGeneratedDocument.Areas_Admin_Views_Customer_Create.b__13_0()
   at Microsoft.AspNetCore.Razor.Runtime.TagHelpers.TagHelperExecutionContext.GetChildContentAsync(Boolean useCachedResult, HtmlEncoder encoder)
   at Microsoft.AspNetCore.Mvc.TagHelpers.RenderAtEndOfFormTagHelper.ProcessAsync(TagHelperContext context, TagHelperOutput output)
   at Microsoft.AspNetCore.Razor.Runtime.TagHelpers.TagHelperRunner.g__Awaited|0_0(Task task, TagHelperExecutionContext executionContext, Int32 i, Int32 count)
   at AspNetCoreGeneratedDocument.Areas_Admin_Views_Customer_Create.ExecuteAsync() in D:\AiGlean\AiGlean-Commerce\src\Presentation\Nop.Web\Areas\Admin\Views\Customer\Create.cshtml:line 7
   at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderPageCoreAsync(IRazorPage page, ViewContext context)
   at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderPageAsync(IRazorPage page, ViewContext context, Boolean invokeViewStarts)
   at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderAsync(ViewContext context)
   at Microsoft.AspNetCore.Mvc.ViewFeatures.ViewExecutor.ExecuteAsync(ViewContext viewContext, String contentType, Nullable1 statusCode)    at Microsoft.AspNetCore.Mvc.ViewFeatures.ViewExecutor.ExecuteAsync(ViewContext viewContext, String contentType, Nullable1 statusCode)
   at Microsoft.AspNetCore.Mvc.ViewFeatures.ViewExecutor.ExecuteAsync(ActionContext actionContext, IView view, ViewDataDictionary viewData, ITempDataDictionary tempData, String contentType, Nullable`1 statusCode)
   at Microsoft.AspNetCore.Mvc.ViewFeatures.ViewResultExecutor.ExecuteAsync(ActionContext context, ViewResult result)
   at Microsoft.AspNetCore.Mvc.ViewResult.ExecuteResultAsync(ActionContext context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Logged|22_0(ResourceInvoker invoker, IActionResult result)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|30_0[TFilter,TFilterAsync](ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResultExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.ResultNext[TFilter,TFilterAsync](State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeResultFilters()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Logged|17_1(ResourceInvoker invoker)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Logged|17_1(ResourceInvoker invoker)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.g__AwaitRequestTask|7_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Nop.Services.Authentication.AuthenticationMiddleware.InvokeAsync(HttpContext context) in D:\AiGlean\AiGlean-Commerce\src\Libraries\Nop.Services\Authentication\AuthenticationMiddleware.cs:line 86
   at WebMarkupMin.AspNetCoreLatest.WebMarkupMinMiddleware.InvokeCore(HttpContext context, Boolean useMinification, Boolean useCompression)
   at WebMarkupMin.AspNetCoreLatest.WebMarkupMinMiddleware.InvokeCore(HttpContext context, Boolean useMinification, Boolean useCompression)
   at Microsoft.AspNetCore.Localization.RequestLocalizationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
   at Nop.Services.Installation.InstallUrlMiddleware.InvokeAsync(HttpContext context, IWebHelper webHelper) in D:\AiGlean\AiGlean-Commerce\src\Libraries\Nop.Services\Installation\InstallUrlMiddleware.cs:line 50
   at Nop.Services.Common.KeepAliveMiddleware.InvokeAsync(HttpContext context, IWebHelper webHelper) in D:\AiGlean\AiGlean-Commerce\src\Libraries\Nop.Services\Common\KeepAliveMiddleware.cs:line 47
   at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.InvokeCore(HttpContext context)
   at AiGlean.Nop.Plugin.SaaSTenantDB.Middlewares.TaskInitializationMiddleware.InvokeAsync(HttpContext context)
   at AiGlean.Nop.Plugin.SaaSTenantDB.Middlewares.PluginsInstallMiddleware.InvokeAsync(HttpContext context)
   at AiGlean.Nop.Plugin.SaaSTenantDB.Middlewares.TenantAvailabilityMiddleware.InvokeAsync(HttpContext context)
   at AiGlean.Nop.Plugin.SaaSTenantDB.Middlewares.DatabaseInitializationMiddleware.InvokeAsync(HttpContext context)
   at AiGlean.Nop.Plugin.SaaSTenantDB.Middlewares.TenantResolutionMiddleware.InvokeAsync(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.StatusCodePagesMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.StatusCodePagesMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)

[JoPetra]

@skoshelev have you managed to look at the issue?