Microsoft Exchange Mailbox Disk Space

I had an interesting issue where one of my Exchange 2010 Mailbox servers was throwing alerts that it was running low on disk space.

Like many people running exchange I run a report to check free space on the database volumes.  Interestingly it was only the mailbox server that was being backed up.  The environment is a 4 node exchange mailbox dag with 3 copies of the database.  Nodes 1 to 3 have the active copies and backup copies of the databases striped across them so 12 databases per server. Node 4 has a copy of all 24 databases and and resided on separate storage to provide redundancy.  To ensure backup of the mailbox data without impacting the production systems only node 4 was backed up using Veeam with AAIP turned on.  Once the backup of node 4 was complete it would truncate the logs across all other nodes without having to worry about DAG fail-over or impact to production data.  Anyone that has used Veeam to backup large exchange mailbox servers knows that exchange 2010 has a 20 second timeout where the VSS freeze command is given to the exchange vss writers.

Now that I know where the extra disk space is going I attempted to remove using vssadmin and received the following error.

When attempting to remove the shadows with vssadmin you recieve the following error.

Error: Snapshots were found, but they were outside of your allowed context.  Try removing them with the backup application which created them.

When Veeam issues the command to create the persistent snapshot if something interrupts the backup process where the Veeam agent fails or the backup server has issues it will leave the vss snapshot and not cleanup after itself.  This is what was using my extra disk space.  Normally once Veeam has the vss snapshot and backs up the VM with that snapshot on it successfully it will then cleanup the snapshot.

Once I had identified that there were orphaned VSS snapshots on the disks with the database volumes and that I could not remove them with VSS commands or powershell I found that by setting the the VSS storage space to 4095 it will clean up the old snapshots.  After verifying that the disk usage was back to normal I set it back to the defaults, don’t forget to set it back to no limit or your next backup will fail.

I have not worked with other backup vendors for Exchange but I am assuming anything that will try to create a VSS snapshot of the database volume can run into this.