Yet another Exchange legacy Public Folders replication issue: Old content not replicating to new PF databases

If you´re reading this post, chances are you´re stuck with a public folder replication issue. I´ve seen all sorts of issues with the replication of legacy public folders, but none without solution, so hopefully your solution is here 🙂

My scenario:

  • Exchange 2010 Service Pack 2 in 3 servers (2 existing and 1 new)
  • CAS/HT/Mailbox roles on all servers
  • Each server has a public folder mailbox
  • One server in Asia, one in America and one in the EMEA region

My problem:

The new public folder database was not getting all the existing public folder replicas/content.

Instead of going direct to my issue, let me guide you through all the steps you should take to make sure you understand why your replicas are not being pushed to the new PF database:

Have you added all the replicas (or the ones you need) to your new public folder database?

AddReplicaToPFRecursive.ps1 -Server <ExistingServer> -ServerToAdd <NewServer> -TopPublicFolder “\”

AddReplicaToPFRecursive.ps1 -Server <ExistingServer> -ServerToAdd <NewServer> -TopPublicFolder “\NON_IPM_SUBTREE”

The script above is on the Exchange scripts folder, and can be ran from an Exchange Management Shell. Look here for more details on the script.

You can check which replicas a public folder has, by running:

GetPublicFolder -Identity ‘\’ –Recurse | fl Name, Replicas

Get-PublicFolder -Server <NewServer> -Recurse

Is the above done and your public folder database is still not up to date?

If you do have the replicas pushed to the new server, and the content is still not being replicated, then please also check if the mail flow between the source and destination server is working.

In my case of course it was working, because as stated on the title of this post, only the old content of the public folders was not being replicated. New content was fine, which means that issues like mail flow or invalid replicas were ruled out.

An example:

UserA was connecting to a database on the new server that was using the new public folder database. Via OWA UserA opened the Public Folders, and created a new item. The item was replicated to all the Public Folder databases on all servers. UserA can only see a very limited number of items on the Public folders, all recent and created after the creation of the new PF Database.

So lets take a closer look at the statistics of a specific folder:

pferror1

The Public Folder “US” had zero items in the new server and 1087 on the old one. An item created now would replicate to both the new and the old server, but nothing was happening to old content.

Now before I continue, it´s mandatory for me to recommend you read an excellent Public Folder Troubleshooting article from the Ehlo Blog.

The next step is to raise the event log level on both the new and the old server. The logs you need to set to expert are under the “MSExchangeIS > 9001 Public” and are named “Replication Incoming Messages” and “Replication Outgoing Messages”.

pferror2

Now let’s force an update on a public folder by running on the new server Exchange Management Shell:

Update-PublicFolder -identity “\US” -server <OldServer>

And on the new server event viewer you can see the 0x20 Status Request event (see the troubleshooting guide provided above for more info on event codes).

pferror3

On the old server you can see the status request reply:

pferror4

But what you never see is an event like the one below, with status 0x4 that indicates that content is being replicated.

pferror5

So what is the solution? Update all the content using the ExFolders tool.

Click here to download it, and for full instructions on how to run it.

Once you have the Exfolders opened and connected to the old server, select the folders you want to update (in my case I selected the route because I wanted to update all) and click on “Modify All Items”.

pferror6

IMPORTANT NOTE: This will cause a replication storm, which means that all public folders on all databases will be updated and replicated. If you have a large public folder infrastructure you might want to consider doing this procedure during off work hours.

You will then see the folders being updated.

pferror7

And once that is done, wait for the replication to take place and run the public folder statistics cmdlet again.

pferror8

All up to date!

Hope that the above was helpful..

Advertisements

Exchange Active Sync on-boarding to Office 365 – The seamless experience is finally here

For those who are thinking on moving to Office 365, and in what that might mean from a user experience perspective, the release of the Exchange 2013 Cumulative Update 8 (CU8) and Exchange 2010 SP3 Rollup Update 9 (RU9) brings a long expected feature – seamless experience on the on-boarding process of ActiveSync users, to Office 365.

You can read the full details on the article Exchange ActiveSync on-boarding to Office 365, published on the Exchange Team Ehlo Blog.

Instead of duplicating all the information that you can read on the official blog article, i will just give you my thoughts on how this works and highlight the key points.

When a user is moved to Office 365, under a Hybrid deployment, that mailbox on premises is converted to a remote mailbox, and a “RemoteRoutingAddress” of the type User@tenant.mail.onmicrosoft.com is configured for the remote mailbox.

That remote routing address should be configured as a domain name on an existing Organization Relationship “On premises to O365”.

Before Exchange 2013 CU8 and Exchange 2010 SP3 RU9, the experience was only seamless for users via Outlook or OWA. When those users, moved to Office 365, tried to connect to the Client Access server on premises, a mailbox wasn’t found.. so what happened next?

“The Client Access server triggers a query to find the “TargetOWAURL” property present on the organization relationship object for the Office 365 tenant. The “RemoteRoutingAddress” property, present on the remote mailbox, is used to find the correct organization relationship.”

That “TargetOWAURL” is then used by Outlook (automatically reconfigured the profile) or OWA (presents the new URL to the user) to redirect the user to the Office 365 mailbox.

After Exchange 2013 CU8 and Exchange 2010 SP3 RU9, that process will also work when the user is connecting via Exchange ActiveSync, making the experience seamless as well for all the ActiveSync users.

Of course for all of you that, like me, spend countless hours explaining to customers that recreating the exchange partnership on all of their user’s phones, was the only option, and helping on the creation of user guides, this new feature is excellent news.

From my personal perspective it makes perfect sense that, with the Client Access services already using the Organization Relationships and the TargetOWAURL, to redirect Outlook and OWA clients, the capability to redirect ActiveSync clients is now also an available feature.

Of course there are some limitations, such as the device EAS version not supporting HTTP 451 redirects or cross-forest migrations.

I highly recommend that you read the official article, for all the details on this new feature.

Well done for the Microsoft Exchange Product Group! 🙂

Public Folder migration request error creating the Public Folder Hierarchy – “Property Expression [property name] isn’t valid”

To migrate your legacy public folders to Exchange 2013, you should follow all the steps described on the official Microsoft article.

Step 2 of the article mentioned above, helps you prevent errors related to the public folder name, such as having a “\”. But this blog post is related with something which is not covered by the article, and that can make your public folder migration request throw an error: Invalid alias on mail enabled public folders.

So as described on step 5 of the article, you start your migration request by running the following cmdlet:

1-done

 

New-PublicFolderMigrationRequest -SourceDatabase (Get-PublicFolderDatabase -Server <Source server name>) -CSVData (Get-Content <Folder to mailbox map path> -Encoding Byte) -BadItemLimit 200 -acceptlargedataloss -largeitemlimit 200

Note: On the above cmdlet i used both the baditemlimit and largeitemlimit set to 200, because i knew that my public folders had a significant number of both bad and large items. If that is not your case keep the bad and the large item limits to a minimum if at all specified.

Once the public folder migration starts you can run the following cmdlet to see the progress. If you have invalid alias on the mail enabled public folders, the migration request will fail at 10%, when creating the Public Folder Hierarchy:

3.1-Done

 

 

To get the details of the error you need to run the same cmdlet, but with the “|fl” at the end, as shown below:

2-done

 

Get-PublicFolderMigrationRequest |Get-PublicFolderMigrationRequestStatistics |fl

On the details you can see the following:

3.2-done

On the screenshot above you can see that the mail enabled public folder “Accountancy Properties” has an invalid alias, and that is because you cannot have spaces (and other characters as shown above) on the alias. So the next step would be to fix all mail enabled public folders with invalid aliases.

Go to your Exchange Management Shell and run a cmdlet that will list all the mail enabled public folders with spaces. Of course some other mail enabled public folders can have other problems, but in my case it was only the spaces. If you have other problems besides spaces you can adapt the “where-object” filtering from the cmdlet below, to those characters. Also to get an output of all the mail enabled public folders with issues, you can run the following cmdled and see which public folders throw a warning:

Get-MailPublicFolder

All the mail enabled public folders will throw a warning on the output, but of course we need to have a list of only the ones with problems to resolve the issue quicker. So to have a list with all the ones with spaces, and export it into a csv file, run the following cmdlet:

5-done

 

 

 

 

Get-MailPublicFolder | Where-Object {$_.Alias -like "* *"} | Select-Object alias, identity |export-csv [CSV file path and name]

Note: On the cmdlet above, if you’re problem is not limited to spaces on the alias, you can change the where-object filtering to try and find other invalid characters.

Once that is done you will get a csv file with all the mail enabled public folders with invalid aliases, as shown below:

6.1-done

 

 

Having that information you can now fix all the mail enabled public folders. You have two ways of doing it:

Option 1:

You can open your Exchange Management Console, go to tools and then open the Public folder Management Console. Expand the Public Folder tree and go to the properties of each public folder with problems. On the “Exchange General” tab change the alias and apply, as shown below.

7-done

 

 

 

 

Option 2:

You can use the Exchange Management Shell, and run the following cmdlet:

8-done

 

 

 

 

Get-MailPublicFolder [Public Folder Name] |Set-MailPublicFolder -Alias [PublicFolderAlias]

Once you set the valid alias you problem is solved for that specific public folder. You might be thinking “how can i automate this for the dozens or hundreds of public folders i have with issues?”. Well the answer is you should use that csv file you exported with all the public folders, insert a column with valid aliases, build a script that reads from the csv file and run it from the Exchange Management Shell to have all your folders fixed in one go. In my case i had a low number of folders with issues, so i haven’t built the script. I am planning to build 2 scripts, one to export public folders with issues and another one to use that exported file and fix those issues, but at the moment i don’t have it. Feel free to ping me an e-mail or comment if you’re interested on the script and i can find some time to build it. It should be a fairly simple script. Or feel free to have a go on doing that.

But back to the issue. Once you have all your mail public folders with valid aliases, i recommend that you remove the current public folder migration request and create a new one. To do that run the sequence of cmdlets shown below:

9-done

 

 

 

 

 

To see the current failed request:
Get-PublicFolderMigrationRequest

To delete the current failed request:
Get-PublicFolderMigrationRequest |Remove-PublicFolderMigrationRequest
To create a new request:
New-PublicFolderMigrationRequest -SourceDatabase (Get-PublicFolderDatabase -Server <Source server name>) -CSVData (Get-Content <Folder to mailbox map path> -Encoding Byte) -BadItemLimit 200 -acceptlargedataloss -largeitemlimit 200

Finally after the new request is created, you can see that now the migration is ongoing and went past the Public Folder Hierarchy creation without issues.

10-done

 

 

Problem solved and happy migration! 🙂

 

 

 

 

 

 

Public Folders migration to 2013 gets “StalledDueToMailboxLock” Status

Are you trying to migrate your public folders from legacy Exchange versions to Exchange 2013? And are you getting stuck on a “StalledDueToMailboxLock” error when you try to resume the migration? Well then continue reading. I will try to keep this post short and simple, just like the solution to your problem.

First things first.. the migration process.. To successfully complete the migration, and if you don’t have experience on doing it, you should follow all the steps from the link below:

https://technet.microsoft.com/en-us/library/jj150486(v=exchg.150).aspx

So in summary, you download the scripts, estimate the number of public folder mailboxes you will have, name and create those mailboxes and start the migration process, that will suspend at 95% with all the data migrated except the delta.

Once that is done you then lock the public folders on the legacy version (step 6 on the link above), by running the following cmdlet:

Set-OrganizationConfig -PublicFoldersLockedForMigration:$true

And what is the purpose of this cmdlet? Simple.. to block the access from the users to the legacy public folders, so that you can migrate the delta, unlock the access on 2013 (Step 8), and have all users using the new modern public folders.

So what can go wrong and when can you see the “StalledDueToMailboxLock” Status?

After you run the Set-OrganizationConfig -PublicFoldersLockedForMigration:$true cmdlet, the changes need to apply on the legacy public folders, before you resume your migration by running the Resume-PublicFolderMigrationRequest -Identity \PublicFolderMigration cmdlet.

If the changes are not applied yet, than your public folder migration request will resume and a minute or two later it will get into a “StalledDueToMailboxLock” status.

When you get into that status there are two things you need to do:

First try and access the public folders, from an Outlook client. Can you still access them? Well that means they are not locked and the changes didn’t apply yet.

So you can either wait for the changes to apply, and keep checking the access to the public folders until it’s blocked, or alternatively you can force the changes to apply, by restarting the Information Store service on the legacy server(s), where the public folder database being migrated is. Once you restart the Information Store service, retry accessing the public folders from an Outlook client. The access should now be blocked. Then you can suspend and resume your public folder migration request, to get it going quicker, by running:

Suspend-PublicFolderMigrationRequest -Identity \PublicFolderMigration
Resume-PublicFolderMigrationRequest -Identity \PublicFolderMigration

The request should now complete without issues, and you can continue following the steps on the link i provided earlier on this post, to get your public folders running on 2013.

Any questions about the entire process let me know.

Exchange 2013: Receive connector authentication error

I recently had an issue with an application relaying e-mail via an Exchange 2013 Client Access Server. The error i was getting on the logs was:
“The account ‘Domain\AccountName’ provided valid credentials, but it does not have submit permissions on SMTP Receive connector ‘Receive Connector Name’; failing authentication”
The application was configured to authenticate with a valid username and password, on the front-end transport service of the Client Access server, which was listening on port 25.
So why was this happening? The answer is simple, the receive connector used was not allowing authenticated relay of emails.
The Client Access Server in question had several receive connectors, and how do we know which one is that specific server/application using? Well it will use the more specific receive connector, meaning that if your application server IP is 10.1.1.1 and that IP is specified on the “RemoteIPRanges” attribute of the receive connector, than that is the receive connector being used, and it’s there that you need to look and see what authentication options is the receive connector advertising.
To check the Remote IP Ranges of a receive connector you can use the Exchange Admin Centre, and go to the Scoping tab on the receive connector properties.
If the IP address of your application server is not specified on any receive connector, chances are it will use the default receive connector to try and relay (or any other that accepts all IP ranges), or if your default receive connector is not allowing relay from any IP (it shouldn’t so if it is you should change it) the relay would be denied and you’re looking at a different error than the one i am blogging about today.
But back to the authentication problem. I checked which receive connector was being used, and went to check the authentication options. I verified that “Exchange Users” and not selected. Problem found! Selected Exchange users, tried again, and job done! See below my receive connector security options, for guidance:
Post4-ReceiveConnectorAuthOptions
In my case this was good enough to sort the issue. Allowing authenticated users to submit on that specific receive connector. But if you want to allow just a specific user, or make sure that a specific user can submit on a specific receive connector, you can also run the following Exchange Management Shell cmdlet:
Get-ReceiveConnector “<ConnectorName>” | Add-ADPermission -User “UserName” -ExtendedRights ms-Exch-SMTP-Submit
For more information on the cmdlet above go to:
I hope the above was helpful! As always, any questions or queries let me know.

Lync Server 2013 Cookbook – Available this January

Recently i was invited to write a chapter for a Microsoft Lync 2013 Cookbook. My contribution to the book was around the integration between Microsoft Lync and Microsoft Exchange. It was an amazing experience to participate as a co-author on this book, and the final result was excellent. The book is amazing and highly recommended both for consultants or administrators that work with Microsoft Lync and Microsoft Exchange on a daily basis. You can find it here:

https://www.packtpub.com/networking-and-servers/lync-server-2013-cookbook

It was a pleasure to work with the highly skilled Lync experts Fabrizio Volpe, Alessio Giombini, Lasse Nordvik Wedø and all the technical reviewers. It’s an experience that hopefully i will repeat soon.

Enjoy!

Exchange 2010 DAG – Error when adding database copies [Event MSExchangeRepl ID:2059]

Recently i was adding new Exchange 2010 SP3 servers into an existing Database Availability Group, and adding database copies on those servers, to all the existing databases.

To add the database copy i ran the cmdlet:

Add-MailboxDatabaseCopy -Identity <DatabaseName> -MailboxServer <NewDagMember> -ActivationPreference 2

The copy was added and started seeding, but when it finished, it immediately went into a failed state, as shown below:

Post2-DatabaseFailed

I went to the NewDagMember and the log file directory was empty, meaning that no log was seeded during the process of adding the copy. Then i went to the event viewer of the NewDagMember and found two relevant events.

Event ID 2059 – MSExchangeRepl: The required log file <file number> for <Database\Server> is missing on the active copy. If you removed the log file, please replace it. If the log file is lost, the database copy will need to be reseeded using Update-MailboxDatabaseCopy.

Post2-FileMissing

Event ID 117 – ExchangeStoreDB: At <Date Time> the copy of <Database Name> on this server experienced an error that requires it to be reseeded. For more details about this failure, consult the event log on the server for other storage and ExchangeStoreDB events. The passive copy has been suspended.

Post2-DatabaseReseeded

 

 

So in summary event ID 117 was telling me that the local copy experienced an error and had to be reseeded and event ID 2059 was telling me the cause of that error. A log was missing on the source.

To better understand why was that log missing i went to investigate the source, and found out that:

  • the database on the source never had other copies up until i added that one
  • the database on the source mounted and functional with no problems being reported from the users
  • the database was being backed up with full and incremental backups. The last backup on the database was an incremental one
  • the database on the source had thousands of log files waiting to be truncated by the next full backup

At this point i was sure on how to resolve this: I needed to truncate all those database logs. The incremental and probably also the full backups were not truncating the logs as expected and some logs were missing, so if i was able to truncate them asap it was problem solved for me.

So how do you truncate the logs on the database? You have several options:

  • A full backup (if the backup software is well configured and 100% compatible with Exchange 2010)
  • Circular logging
  • the eseutil tool to identify and remove unnecessary logs

Before telling you what option i made i would like to recommend that you read this fine article:

http://support.microsoft.com/kb/240145/en-gb

So what i did was, i got a small maintenance window and enabled circular logging on the databases. If you choose to do that be aware that: You need to dismount the database twice (hence the maintenance window), the database cannot have any copies when you enable the circular logging and you need to monitor the logs folder of that database as circular logging might take some time to truncate the logs.

To use the same procedure i did, do the following steps:

  1. make sure you remove the existing copy (the one that is failed and suspended) of the database, from the new DAG server
  2. Enable circular logging on the database
  3. dismount the database
  4. mount the database
  5. wait for circular logging to truncate all the logs
  6. Disable circular logging on the database
  7. dismount and mount the database for the change to apply

Please see the following for instructions on how to enable circular logging on Exchange 2010:

http://technet.microsoft.com/en-gb/library/dd297937(v=exchg.141).aspx

Once i did the procedure described above, i re-ran the Add-MailboxDatabaseCopy cmdlet and my problem was solved.

Hope that was helpful. Enjoy

Exchange HCW Error – Exchange OAuth authentication couldn’t find any accepted domains

Recently I’ve bumped into a strange issue, when setting up an Exchange Hybrid Scenario on a customer. The customer has a pure Exchange 2013 on premises, with no legacy versions of Exchange, and when you run the Exchange Hybrid configuration wizard it will try and configure OAuth authentication between the Exchange On Premises and Exchange Online. The problem I had was, when trying to configure the OAuth authentication I was getting the following error:

“Exchange OAuth authentication couldn’t find any accepted domains in your on premises organization. Verify that you’ve configured at least one on-premises accepted domain.”.

HCW-Error-01

So what was the actual real impact of the error above? The answer is: Free/busy information between Exchange online users and Exchange on-premises users, on both directions, was not working? Why? Well i am going to try and keep the explanation simple and focus more on the solution. Free/busy was broken because it relies on IntraOrganizationConnectors, and the IntraOrganizationConnectors rely on OAuth authentication.

The first thing that i did was to check if my accepted domains on-premises were ok. I ran on the on-premises Exchange Management Shell:

get-accepteddomain |fl

the output showed all my on premises domains, as expected, and the *.onmicrosoft.com domains created by the Hybrid Wizard. So all good here.

The next step was to check the IntraorganizationConnectors. I ran both on the on-premises and online Exchange Management Shell:

get-intraorganizationconnector |fl

(More Info: http://technet.microsoft.com/en-us/library/dn551175%28v=exchg.150%29.aspx)

The main purpose here was to make sure that the IntraOrganizationConnector was there and enabled. I verified that i had both connectors, one on-premises and one online. Then i disabled the connector to force the Free/Busy information to be handled by the OrganizationRelationship. I ran on both Exchange Shells:

Set-IntraOrganizationConnector <Connector Name> -Enabled $false

(More Info: http://technet.microsoft.com/en-us/library/dn551177(v=exchg.150).aspx)

You need to wait up to one hour to test the Free/busy from an online to an on-premises user. From one on-premises to one online should be almost instant, depending of course on Active Directory replication.

So did disabling the IntraOrganizationConnectors fixed the Free/Busy issue? For me it did, which means that i had the issue identified.

When i ran the Hybrid Configuration Wizard, the OAuth authentication was only partially configured, and therefore the IntraorgConnectors were not working. Disabling them or removing them sorted the issue, but that is not the ideal solution, as the goal is to use them correctly.

And what was the solution? To manually configure the OAuth authentication between my Exchange on-premises and Exchange Online. To do that follow all the steps from the link below, and you should get your problem sorted, as i got mine. Any questions or comments let me know.

http://technet.microsoft.com/en-gb/library/dn594521(v=exchg.150).aspx

I hope that the post above was helpful!