Exchange 2013: When your Load Balancer marks your Client Access Server as offline.. and all services are up and running

Just recently, I was asked to help with an issue in an Exchange 2013 Organization. The problem was that the KEMP Load Balancer, that balances requests for multiple Exchange protocols, between several Exchange 2013 servers, was marking one of the CAS servers as being offline.

The server was not being marked as offline for all services, it was just for one, Outlook Web Access (OWA).

Before I go into server details, let me give you a visual representation of what the above means. In my case we had a KEMP Load Balancer, but this can apply to any load balancer.


What you can see above is the KEMP admin portal. On the left hand side if you go to Virtual Services and click “View/Modify Services”, you’ll see a table (that I conveniently don’t show, just so I don’t have to cover all information in it anyway). In that table you will have, per server, one or more “Real Servers”, and when that real server health cannot be verified, it will show in red.

What that means from an operational perspective, is that no request from that protocol (https/smtp/sip etc) will be sent to that server, meaning you can be in a situation of single point of failure and you won’t know it.

Now lets have a look at how those real servers health is checked.


If you click to modify the virtual service, in the “Real Servers” section you will see the check parameters. Basically what KEMP does is to get an https request to https://server/owa/healthcheck.htm

If that request fails, you should see a page not found and the server is marked as being offline. The below is the result you should expect from a healthy server.


In the URL above, you should see the server name. Below the 200 OK you should see the server FQDN.

Now that I explained, very high level, how the process works, lets drill down to the problem that I had.

Before detailing the problem that I had, I just want to point out that there are many reasons for a server to be marked as offline by a load balancer, and that you should check the obvious ones first, such as if all services are running, if Exchange is coming back as healthy when using tools such as the Get-ServerHealthย or Exchange cmdlet, System Center Operations Manager (SCOM) if you have it, etc.

In my case there were no obvious reasons, plus I found it strange that only the https protocol was having issues, so this is what I did:

1 – I browsed to the https website that KEMP uses and noticed that I was getting page not found, which explains KEMP marking the server as not available

2- I then used another very handy Exchange PowerShell cmdlet, the Get-ServerComponent, to check which components were in an Active or Inactive state

See below an example of a good output for the server components, with all of the relevant ones being active.


But what I was getting was not what you see above. I was seeing the “OwaProxy” component as inactive, and that was a problem.

3- So what I did next was to look at the Exchange 2013 Health Mailboxes, searching for inconsistencies

And I found them…


When I ran a Get-Mailbox -Monitoring, to list all Health mailboxes, I found that some were corrupt, due to being associated with missing databases or missing mailbox servers.

This Exchange Organization has been downsized for the past year and during that cleanup those mailboxes were forgotten. If you have the same issue, don’t worry. Although it’s important to keep those mailboxes in a good state, you can easily recreate them, and that leads me to my next step.

4- Recreate the health mailboxes

So instead of me doing a lot of copy past into my blog post, check this Exchange 2013/2016 Monitor mailboxes article, from the product group blog, that not only has a lot of information about those mailboxes, but also has a perfect step by step at the bottom, on how exactly you recreate them.

And that’s what I did, recreate them, re-tested the URL https://server/owa/healthcheck.htm, which now worked perfectly and my load balancer was no longer marking my CAS server as unavailable in my https virtual service.

Hopefully this article can help you resolve your issue!



What does it really mean to be in the last year of support for Microsoft Exchange 2010?

The Microsoft Exchange product team, sent out a reminder on January 14th 2019, regarding the end of support for Exchange 2010. It will happen in 1 year! Don’t think one year is a long time, read the below to have my perspective of what this means to Microsoft but more importantly what this means to you, and start planning and executing the best roadmap for you. Check here what the Microsoft recommended roadmap is.

Let me start by outlining the simple reasons (in my perspective, of course) of why Exchange 2010 is going out of support:

  • Age and Versions – Believe it or not, Exchange 2010 RTM was launched in November 2009, almost 10 years ago. To add to that, we had 3 new versions being launched since: 2013, 2016 and 2019.
  • Office 365 and Hybrid compatibility – Exchange online servers get upgraded and the same compatibility standards apply. This means that it makes perfect sense that, as Exchange Online versions evolve, On Premises supported versions tend to be raised as well. Today, if you look at the Hybrid Deployment Prerequisites official information, you’ll see that Exchange 2010 based hybrid deployment is still supported, but I am sure that this won’t last long, and being in a Hybrid deployment will require at least one Exchange 2013, very soon. Logic says that once you start the On Premises upgrade path, by installing higher versions, you should be ready as well to move services to the higher versions and think about decommissioning the lower ones.

Moving on, what exactly does this announcement means, from the Microsoft perspective?

  • No more support, bug or security fixes or time zone updates – Basically Microsoft here tells you that you’re on your own, should you choose to keep Exchange 2010 in your Exchange Organization.
  • Move to Office 365, move to Office 365 (that’s actually not a typo or duplicate, it’s to show how determined Microsoft is to have you move to Exchange online) or upgrade your Exchange Servers On Premises to a newer version (Exchange 2013+) and decommission all Exchange 2010 servers from your Organization – It’s no secret that for Microsoft, the best option for most organizations, is to move to Exchange Online. Making a product with 10 years and 3 newer versions end of support, is not a consequence of that, but they do take the opportunity to call out those companies, that still have Exchange 2010 or older versions, and tell them to move to the cloud. I do agree that this will be the best option for most of the organizations out there, just not the best option for all, and the reasons are so vast that I could write an dedicated and extremely long blog post about it. In summary probably 95%+ of the organizations, are a perfect fit for Exchange Online.
  • If you need help, use the Microsoft FastTrack (did I mentioned that they highly recommend that you go to Exchange Online? ๐Ÿ™‚ ), or search for a partner here – This part of the article, the “What if I need help?” section, is actually interesting because they don’t even mention anything directly for if you need help to upgrade your On Premises servers. They basically say that, if you need help, the FastTrack can help you or you can hire a partner, again very Office 365 focused (and I can’t stress this enough: I agree that moving to Office 365 is the best move for most organizations). You can however use the partner search link to, if you choose to, hire one of many great partners out there that will upgrade your Exchange On Premises, if they feel that you’re not the best fit for online, or give you a migration path to Exchange Online and Office 365, better than the FastTrack can provide, if they identify that is the case, by using a third party tool.

And finally, what is probably the most important part, what exactly does this mean to you and your Exchange Organization?

  • You don’t have to upgrade, or take any action for that matter, but you should – Your current Exchange 2010 servers will not stop working, and believe me I’ve recently seen Exchange 2003 or older versions still sending and receiving emails, but you should start planning now to either upgrade and/or decommission your Exchange 2010 servers. Do not go into an unsupported scenario, with a service as critical as email should be for your organization.ย 
  • It doesn’t matter if your Exchange 2010 servers aren’t hosting mailboxes and/or processing Client Access Servicesย  – This can affect you because you have servers with mailboxes and active CAS services, but that’s not the only scenario. If you have some old Exchange 2010 Servers, as part of an Exchange 2013 or Exchange 2016 organization, you just recently had one reason to remove them – Exchange 2019 – and now consider this as one more. Don’t keep non supported servers in your Organization. If it’s because of that printer or app using the server to relay email, or because of an old backup or email signature software that lets you stuck to 2010, upgrade that as well or get rid of it.
  • It’s not only about if you’re planning to go to Office 365, it’s also if you’re already there and under a Hybrid – If you’re planning to go to Office 365, getting rid of the Exchange 2010 in the process is a good idea, but what if you’re already there and under a Hybrid deployment? If that’s your case, plan and execute. The ideal scenario is always towards having Exchange 2019 (the latest available version) Hybrid servers, and that won’t happen before you remove those 2010. If your Hybrid is a 2010 based hybrid than the urgency is even bigger: Don’t wait!

Hopefully all of the above makes sense. If not, please reach out to me. It’s time to move on from Exchange 2010. Time to upgrade or migrate, whatever suits best your needs.

Azure Data Box Disks just went from preview to general availability and became available in more regions

Yesterday, Microsoft announced the general availability for Azure Data Box Disks.

For those who don’t know what this is, Azure Data Box Disks are basically a fast (SSD disk based), reliable and secure solution to do offline data transfer to Azure.

It’s been a while since Microsoft announced the preview program, and that was available only for the EU and US regions. General availability is for EU, US, Australia and Canada. As Microsoft promised, the service is expanding to more Azure data centers worldwide.

When compared to the Azure Import/Export service, using the Azure Data Box Disks is, in theory, a simpler process, since Microsoft will provide the disks and handle all the logistics.

We’ll have to wait and see where Microsoft will drive this service towards, since the expectation of some customers is to see it handle other things, besides just simple data transfer, such as initial seeding for Azure Backups.

Office 365: Not sure if your vanity domain is being used by another Office 365 tenant? Don’t worry.

I remember not long ago, the pain of trying to find out in which Office 365 tenant your vanity domain was validated, when you bumped into the error stating that the domain was in use, while adding it to your current tenant.

This was maybe because I work with multiple tenants and I recycle my tenants quite often, for testing purposes, but I’ve also seen it with others while trying to assist them in their migration projects.

Fortunately Microsoft now is very clear in the error message you’ll see, when trying to add a domain to a tenant, that is in use. This is what you will see now:


As you can see above, it will tell you exactly in which tenant the domain is, just so you can login to it and remove it.

Now is there a catch with this? Of course!! Microsoft won’t give you such privileged information, until you enter a valid DNS record for the domain validation. You’ll see something similar to this, if the domain validation is not done properly:


So remember to add the DNS record first and click “Verify”, Microsoft will either add the domain or explain exactly why they can’t, which I am sure it was for a long time one of the main asks of Office 365 admins and consultants.

Azure Tip: Use PowerShell to check all blob spaced used in a Storage Account

Just recently, I had the need to be able to know the exact volume of all blob container data, within a specific Azure Storage Account.

This was part of a migration project, which in this case meant that I needed to report that data amount multiple times per day. Data was constantly being copied to and deleted from that Storage account, and the same applies to Blob containers being created, filled with data and deleted afterwards. So my only constant was the Storage Account, and I needed to know, every 2 hours, what was the volume of blob container data in that account.

After a quick of research I found this outstanding Microsoft article on how to leverage the Azure PowerShell module (yes, PowerShell to save the day again!!) to calculate the size of a Blob Storage Container.

The only limitation with the script in the article above was that it’s calculating the size of a single blob container, and I needed the combined size of all blob containers in my Storage Account.

So I had to adapt that script to my scenario, and I turned it into the following script:

# Connect to Azure

# Static Values for Resource Group and Storage Account Names
$resourceGroup = "ChangeToYourResourceGroupName"
$storageAccountName = "changetoyourstorageaccountname"

# Get a reference to the storage account and the context
$storageAccount = Get-AzureRmStorageAccount `
-ResourceGroupName $resourceGroup `
-Name $storageAccountName
$ctx = $storageAccount.Context

# Get All Blob Containers
$AllContainers = Get-AzureStorageContainer -Context $ctx
$AllContainersCount = $AllContainers.Count
Write-Host "We found '$($AllContainersCount)' containers. Processing size for each one"

# Zero counters
$TotalLength = 0
$TotalContainers = 0

# Loop to go over each container and calculate size
Foreach ($Container in $AllContainers){
$TotalContainers = $TotalContainers + 1
Write-Host "Processing Container '$($TotalContainers)'/'$($AllContainersCount)'"
$listOfBLobs = Get-AzureStorageBlob -Container $Container.Name -Context $ctx

# zero out our total
$length = 0

# this loops through the list of blobs and retrieves the length for each blob and adds it to the total
$listOfBlobs | ForEach-Object {$length = $length + $_.Length}
$TotalLength = $TotalLength + $length
# end container loop

#Convert length to GB
$TotalLengthGB = $TotalLength /1024 /1024 /1024

# Result output
Write-Host "Total Length = " $TotallengthGB "GB"


The script above will provide you an output into the console of the total volume, in GB, that you have on a specific storage account.

To execute the script, follow the steps below:

  • Copy the entire code above to a notepad
  • Change the values of line 2 and 3, to the correct names of your Azure Resource group and your Azure Storage Account
  • Save the file as .ps1
  • Open a PowerShell window and execute the “script.ps1” file you just saved (see screenshot below)
  • Authenticate with your Azure username and password, when prompted


Execute the script as shown above.


When prompted, authenticate.


And this is how the end result should look like.

Before I end this blog post I’d just like to point out that this script was written in a very simplistic way, and to address an urgent need that I had. With a couple more hours of work, you can make this script even easier to use and add all sorts of different features to it, such as:

  • Error handling
  • remove the hard coded values and list for selection all available storage accounts and resource groups
  • change the output format (i.e to CSV) and list sizes per blob container
  • allow you to select between multiple Azure subscriptions under the same account

The above are just some ideas on how to improve the script. I haven’t done it because I had no need for it, but by all means please let me know if you want/need an improved version. This one works just fine, if all you want is the total volume of blob data in a specific storage account.

Happy New Year!!!

Azure Identity training, anyone?

With the New Year starting, I am looking at a training plan for 2019.

I don’t think that training is the only thing that makes you improve your skills, and at least in my opinion you should add to that as much real live consulting experience as you can, as well as make your training as much “hands on” as possible. Don’t just read or watch videos, build a lab and execute everything that you’re learning.

This blog post is to share with you what seems to be an excellent training resource in Azure Identity: Microsoft Azure Identity training in the platform.

With this training, as they state in their website, you’ll learn the following:

  • How to create and manage Azure Active Directory (AD) directories.
  • How to implement applications in Azure AD.
  • How to extend on-premises AD to Azure.
  • How to configure multi-factor authentication.

The prerequisites for this training are:

  • General understanding of cloud computing models.
  • General understanding of virtualization, networking and Active Directory.
  • Basic proficiency in PowerShell and command line interface scripting.

The above basically means that you should have some experience with Azure, virtualization and of course as this training is focused in Microsoft Identity Management, that means you need to clearly understand how Active Directory works. Finally, as in everything Azure related, PowerShell knowledge is a must! ๐Ÿ™‚

All training that I did with has been great. The training is free, but if you choose to get a certificate at the end, you can pay 99USD, knowing you’d be helping the only nonprofit and open source learning platform.

Most of my posts are about real life scenarios, tips and tricks, etc, so I am sure I will be blogging a lot about Azure Identity in the near future.