In a previous post we saw the Microsoft requirements for the exclusions that must be added to file system AV on Exchange servers. In a recent CritSit, basically an uber urgent support request where the customer is down or as good as down, I also got to examine some of the other causes for file system AV not being correctly configured for Exchange.
In the aforementioned post the majority of the issues were caused by the lack of exclusions to a scheduled scan task. The issue below was not related to that but how different processes are identified and what exclusions get applied to the various processes.
Please note that this post is not intended to slight the AV vendor’s product in any way whatsoever. The product was performing as designed, it was how the customer’s AV team had configured the product which was the issue. The underlying intent for this and the other post is to raise field awareness of the types of issues that we see, and to facilitate better and more focussed discussions with the various AV and security teams that we work with on a daily basis.
Process Definitions
The file system AV product in question has the option to categorise processes into different risk levels. By default this feature is not enabled, and the customer must explicitly enable it. The different process levels that you may see are Default, Low Risk and High Risk. The below is a brief description:
-
Default - This is the default option that is enabled. All processes thus fall into the same level. Since all processes are treated as equal then the same set of exclusions apply to all processes on the server.
-
Low Risk – Not enabled by default. When this is enabled High Risk is also enabled. Processes that are defined by the AV administrator adding them to this level will get the exclusions defined at the low risk level.
-
High Risk – Enabled when Low Risk is enabled. Processes that are defined by the AV administrator adding them to this level will get the exclusions defined at the High risk level.
The key concept to note is that the level a process is defined at will dictate which set of exclusions will apply. For example a process like Trojan.exe can be defined at the high risk level. This means that the exclusions applied to what Trojan.exe touches will be the exclusions defined at the high risk level. Typically by default there will be minimal exclusions at the high risk level.
What happened to cause the issue to get me onsite in a hurry?
It’s All Gone A Bit Pete Tong
(Subject should read Pete Tong MBE, and refers to cockney rhyming slang “It’s gone wrong”.)
The customer’s Exchange team correctly identified that file system AV exclusions were required as part of the design. The required exclusions were passed to the customer’s AV team. Consider this the WHAT of this story. The exclusions are WHAT is required. HOW they get implemented varies depending upon the file system AV product the customer has implemented. AV products each have their own best practices and implementation requirements, for details on this you must consult with your AV team and their vendor. Microsoft cannot provide guidance on HOW a 3rd party vendor’s product be configured to achieve the required results.
In this case, the customer started off by defining all of the required exclusions in the default process section. As noted above this will apply to all process on the system uniformly. What happened next was a bit baffling. For some reason, that was not well understood, they then enabled the low risk process section (and by extension this also enabled high risk). All of the Exchange processes were then added to the low risk section. Job done, no?
<Borat> Not so much </Borat>
Since the Exchange processes were now defined as a low risk process, they picked up the exclusions that were defined at the low risk level. In the paragraph above note that there was no mention of the exclusions being copied over from the default process section, and that was the crux of the issue. The Exchange content was now being scanned by file system AV since it was not excluded at the same level as the defined process. In this case every read and write to the database was intercepted by file system AV. The performance on the system was terrible, CPU consumption was through the roof and since the business was so unsatisfied with Exchange performance I won a free trip to go and fix it.....
Learning Points
Again, the AV product was working as designed. Absolutely no issues were identified with it apart from the configuration the customer had applied. After I noted that not all of the required exclusions were present, I requested the customer’s AV team, the AV vendor and the Exchange team get on a conference call to thrash this out. I have to applaud the level of support we got from the AV support person on the call, she was fantastic! In the space of 60 minutes she clearly and precisely identified the configuration issues, stated what needs to be corrected and then provided multiple other items the customer should address.
What can we take away from this?
-
As Exchange people we cannot just throw the exclusion list over the wall and expect that it is correctly implemented. We need to follow up to ensure that what we see looks correct.
-
Do not be shy to ask the vendor for assistance or as part of the design process to ensure that their product is being correctly implemented. You are paying for support – why not use it?
-
Look at the logs on disk to see what has been detected and repaired. The AV product may log to clear text files on disk, that you are able to view. This is useful when if you are locked out of the local AV console, and these text logs can be your friend. They will show you things like what the AV product is doing, the number of exclusions and how long certain scan operations take. This allows the Exchange admin to keep the AV team honest.
-
Mount Points continue to be an issue as certain AV vendors need to exclude the mount point path such as D:Mounts and also the disk GUIDVolume ID. While wildcards can make this easier to implement, Exchange admins must impress the requirement to investigate this area with their AV colleagues. If the AV team do not know you are using mount points then they will not be aware they have to be excluded!
-
Certain file system AV products allow you to use environment variables such as %Program Files% – however this is no use when you install a product like Exchange to the D: drive…..
-
Add Exchange servers and the FSW server to the correct AV group as part of the OS build prior to installing Exchange.
-
If scanned by file system AV, Microsoft cannot guarantee the integrity of the database.
Please also refer to the previous post for the other learning items also presented there.
Cheers,
Rhoderick
