In this article I will explain how you can remove up to 90% of all log lines from the Application Insights log, but still keep every log line in the file log. All of this without loosing any important information.

Sitecore uses a Log4Net Appender to inject log lines into Application Insights. If you take a look at the App_Config/Sitecore/Azure/Sitecore.Cloud.ApplicationInsights.config file that is deployed to your CM/CD servers, you will find the appender at the top, and it could look like this:

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:env="http://www.sitecore.net/xmlconfig/env/" xmlns:role="http://www.sitecore.net/xmlconfig/role/"> <sitecore> <log4net> <root> <appender-ref ref="ApplicationInsightsAppender" patch:after="appender-ref[@ref='LogFileAppender']" /> </root> <appender name="ApplicationInsightsAppender" type="Sitecore.Cloud.ApplicationInsights.Logging.LevelTraceAppender, Sitecore.Cloud.ApplicationInsights" patch:after="appender[@name='LogFileAppender']"> <threshold value="INFO" /> <layout type="log4net.Layout.PatternLayout"> <conversionPattern value="%4t %d{ABSOLUTE} %-5p %m%n" /> </layout> </appender> </log4net> </sitecore> </configuration>

Sitecore does log a lot of information that probably should have been marked as debug information. This is fine when logging to a log file. But with Application Insights there are issues with logging too much information:

You have to pay for what you log. Although Azure is priced reasonable, it is not free. Any excessive logging can make your budget explode.

You might run into sampling issues (“Data received from your application is being sampled to reduce the volume of telemetry data retained; only sampled documents will be returned. The sampling may be applied by the Application Insights SDK or on ingestion by Application Insights.“). This can be disabled, but sampling usually are there for a reason, as Application Insights cost money the more lines you log.

When I analyze my log files, there are several lines that have never helped me:

ManagedPoolThread #6 00:02:46 INFO Job started: Job started: Index_Update_IndexName=sitecore_Master_index ManagedPoolThread #9 00:04:48 INFO Scheduling.DatabaseAgent started. Database: master ManagedPoolThread #13 00:04:48 INFO Cache created: 'ExperienceAnalytics.DimensionItems' (max size: 2MB, running total: 9090MB)

The “Job started” alone can take up to 90% of the log lines in a solution!

To remove these log lines from the Application Insights log, we can use the log4net.Filter.StringMatchFilter. This filter can add or remove lines containing a string.

Filters are added to the appender attribute in the config file like this:

<log4net> <root> <appender-ref ref="ApplicationInsightsAppender" patch:after="appender-ref[@ref='LogFileAppender']" /> </root> <appender name="ApplicationInsightsAppender" type="Sitecore.Cloud.ApplicationInsights.Logging.LevelTraceAppender, Sitecore.Cloud.ApplicationInsights" patch:after="appender[@name='LogFileAppender']"> <threshold value="INFO" /> <layout type="log4net.Layout.PatternLayout"> <conversionPattern value="%4t %d{ABSOLUTE} %-5p %m%n" /> </layout> <filter type="log4net.Filter.StringMatchFilter"> <stringToMatch value="Job started: Index_Update_IndexName=" /> <acceptOnMatch value="false" /> </filter> ... ... </appender> </log4net>

The “stringToMatch” is the string that Log4Net looks for, and the acceptOnMatch=false ensure that the log line is not appended to the log.

I have found the following candidates for removal from the Application Insights log:

Job started: Index_Update_IndexName=

Job ended: Index_Update_IndexName=

Job started: UpdateIndexDocuments

Job ended: UpdateIndexDocuments

Job started: Sitecore.

Job ended: Sitecore.

Request is redirected to no layout page

HttpModule is being initialized

File is being deleted by cleanup task

Cache created:

Health.

There are probably more depending on your Sitecore installation. You should use the old but excellent Sitecore Log Analyzer to analyse the file logs and find the repetitive log lines that you never use, and filter them from the Application Insights log. Remember that what does not help me, might help you. But removing noise really does give a better overview, and your client’s wallet will be happy too :).

MORE TO READ: