Tag Archives: Exchange 2010

Event ID 5003 Generated When Starting Exchange 2010 Information Store Service (Even With Correct Server Time)

I ran across an interesting issue with an Exchange 2010 Mailbox Server (SP1 RU6) recently where the Information Store service would not start on a DAG member after a reboot. Every time I tried, even after additional reboots, I got this error message:

Source: MSExchangeIS
Event ID: 5003
Description: Unable to initialize the Information Store service because the clocks on the client and server are skewed. This may be caused by a time change either in the client or the server, and may require a reboot of that computer. Verify that your domain is properly configured and is currently online.

I checked the clock and compared it to the domain controllers in the site as well as the other DAG members, they all matched within a second of each other (technically you should not receive an error like this unless the difference is greater than five minutes).

As we run an NTP server on our domain, all of the clocks are automatically synchronized everywhere so it was a bit puzzling whywe woudl receive such an error.

Finally, I caught on to what was going on. When the server was booting, it was initially setting its clock according to the BIOS (which in this case was provided by the VMWare ESX host) before synchronizing with the domain. After consulting with the engineering team that manages the ESX farm here, the host this particular Exchange server was on did not have time synchronization configured, and it had fallen out of sync with the rest of the network by seven minutes.

Moments after booting up, the clock on the Exchange server would be reset to the domain time, but by then the Information Store service had already tried, and failed, to start. It appears there is some sort of “feature” that then prevents it from starting even after the time is corrected.

The fix was simply to fix the time on the ESX host (as well as configure NTP on that host) so that the time would be correct on boot up. After rebooting the Exchange server again, the Information Store service started up without any problems.

How to Enable Opportunistic TLS on an Exchange 2010 Edge Transport Server

Microsoft has an excellent guide available for configuring Mutual TLS with a trusted partner. However, the same cannot be said for enabling Opportunistic TLS. That’s probably because Opporunistic TLS is generally automatically enabled set up as part of a routine configuration of an Edge Transport server. However, that’s may not have happened in your case and if so, the instructions here will get it enabled for you.

To get an idea if you are currently offering Opportunistic TLS on a mail server, you can use the tools at http://www.checktls.com or do the following:

  1. Run “telnet ServerNameToTest 25” from a command line
  2. Type in “EHLO LOCALHOST”

If “250-STARTTLS” is listed in the response, Opportunistic TLS is offered.

If not, then there are four things that must be done:

  1. Request a signed SSL certificate containing the appropriate Subject Name or Subject Alternative Names
  2. Import the certificate into Exchange
  3. Assign the certificate for use with SMTP
  4. Enable TLS on the Receive Connector

First, you will need to create a certificate request for use with your Exchange server. If you’re unfamiliar with how to do this and you are using Windows Server 2008, you can follow my instructions here:

How to Create a Certificate Request in Windows Server 2008 R2 for Use with Threat Management Gateway 2010

Once you’ve created the certificate request, you can either self-sign it if you a have an internal certificate authority or you can get it signed by a trusted third-party Certificate Authority such as GeoTrust (my preference). Personally, I’m in favor of using a third-party CA for any Internet-facing services… strictly speaking though this is not required for TLS.

While you can create the certificate signing request (CSR) through the Certificates snap-in, you CANNOT use it to actually import the signed certificate from the certificate signing authority. In order to do it, you’ll need to use the Import-ExchangeCertificate cmdlet, which should end up looking something like this:

Import-ExchangeCertificate -FileData ([Byte[]]$(Get-Content -Path c:\certificate.crt -Encoding byte -ReadCount 0)) -PrivateKeyExportable $true -Password:(Get-Credential).password | Enable-ExchangeCertificate -Services SMTP

The “-Services SMTP” tag on the end allows Exchange to use this certificate with SMTP, and more specifically with TLS. Now that the first three steps are done you just have to enable TLS on the Receive connector. To do that, just run the following from the Exchange Management Shell:

Set-ReceiveConnector “ReceiveConnectorName” –DomainSecureEnabled $true –AuthMechanism TLS

Note that while we are enabling Domain Security, it actually wouldn’t be used unless you were setting up mutual TLS with a specific domain. Enabling it now saves you a step later on.

That’s it, at this point if you use http://www.checktls.com to check your configuration. If your MX records don’t yet match up with your Edge Transport, you can go to “Tests – Custom/Private” to test a specific IP.

Would you like to know more? Here are some excellent resources to get a better understanding of TLS in Exchange 2010.

Securing Transport Servers

Understanding TLS Certificates

Using Domain Security: Configuring Mutual TLS

Selection of Inbound STARTTLS Certificates


Outlook 2007/2010 on Windows XP Cannot Connect to Availability Service with Exchange 2007/2010

I ran in to an interesting issue recently with some users reporting that they could not see Free/Busy information. Upon further discovery, I found out that they couldn’t use the Out-of-Office Assistant either. The immediately tipped me off that we had some sort of issue Availability/AutoDiscover.

We narrowed it down to our users using Windows XP and Outlook 2007. Our Windows 7 Users with Outlook 2010 and our Windows XP users with Outlook 2003 were unaffected.

This coincided with our deployment of some additional Client Access Servers in a new site. It seemed like it had to be related, but I couldn’t immediately think of how.

All of our Exchange 2010 AutoDiscover tests were successful, and the Best Practices Analyzer didn’t reveal anything telling. The Application Log was clear as well.

So, if our Exchange 2010 infrastructure was testing out fine, and there really wasn’t anything apparently wrong with the client PCs either, then what could be the problem?

The answer came when I tried to browse directly to the /owa virtual directory on one of the new client access servers via HTTPS. I got “Page cannot be displayed” and nothing else, even though it worked fine from Windows 7.

It turns out there is a “gotcha” when creating CSR’s with Windows Server 2008 R2.
When the original SSL certificate request was created for the new Client Access Servers, it was created using Windows Server 2008 R2 using the non-default “Legacy Key” custom request. This causes the private key to be stored in Microsoft’s Legacy Cryptographic API framework.
However, Internet Information Services, under Windows Server 2008 R2, will try to process the request using its brand new Cryptographic Next Generation (CNG) framework… which works with the Legacy Key, but has some limitations. Specifically, it will only support two AES cipher suites:


…and those two AES cipher’s are not supported by Windows XP by default.
Therefore, an SSL certificate that is the product of a Legacy Key CSR will result in an SSL failure between Windows XP and Server 2008 R2.
To fix it, you need to generate a new CSR using the default “CNG Key” (which supports numerous ciphers under Server 2008 R2) instead of “Legacy Key” and issue the certificate with your favorite signing authority. Then just apply it to your Client Access Servers and you should be good to go. Don’t forget to assign the IIS services to the new certificate using the Exchange Management Console/Shell.
Be careful though, not everything in your Exchange 2010 technology stack is compatible with a CNG-based certificate request. I know that because I ran into issues when I chose the default selection “CNG Key” for a certificate on Threat Management Gateway 2010 and found that it was not compatible.

So, having already been through troubleshooting that during the deployment of TMG, “Legacy Key” seemed like the safe (and logical) choice when generating CSR’s for use with our Client Access Servers.

Had I not done that though, I wouldn’t have had this problem, and I wouldn’t have written this article… which means you would still be looking for a solution.

How to Bulk Modify “Safe Senders” List in Outlook with Exchange Management Shell

Adding an email address or domain to the Safe Senders list in Outlook for all of your user mailboxes can be handy for a number of reasons. For me, we wanted to add the email address of an externally generated newsletter from a trusted source to everyone’s “Safe Sender” list so that images within the newsletter were automatically downloaded without requiring the user to click the yellow alert bar.

It’s also important if you want to maintain some control over Safelist Aggregation, which had a number of improvements in Exchange 2010.

In order to modify a user’s Safe Senders list, we’ll use the Set-MailboxJunkEmailConfiguration cmdlet .

Microsoft gives us some examples, but they’re pretty limited in their functionality if you want to modify multiple mailboxes.

It’s actually much easier to use a hash table with the ‘Add’ keyword to modify the Trusted SendersAndDomains property. This allows us to perform the operation with just a single line and without writing a more complicated PowerShell script.

get-Mailbox "ALIAS" | Set-MailboxJunkEmailConfiguration -TrustedSendersAndDomains @{Add='address@domain1.com','domain2.com'}

It also allows us to perform this action on multiple mailboxes at once filtered by Distribution Group

get-DistributionGroupMember -identity "ALIAS" | Set-MailboxJunkEmailConfiguration -TrustedSendersAndDomains @{Add='address@domain1.com','domain2.com'}

…by Custom Attribute

get-Mailbox -ResultSize 5000 | Where {$_.CustomAttribute1 -eq "VALUE"} | Set-MailboxJunkEmailConfiguration -TrustedSendersAndDomains @{Add='address@domain1.com','domain2.com'}

…or just for Everyone

get-Mailbox | Set-MailboxJunkEmailConfiguration -TrustedSendersAndDomains @{Add='address@domain1.com','domain2.com'}

If you want to check what is in the Safe Senders list for any particular user, just use the get-MailboxJunkEmailConfiguration command:

get-MailboxJunkEmailConfiguration -identity "ALIAS"

The lesson here? Never limit yourself to Microsoft’s stock examples as there are likely even more flexible ways available.

UPDATE: While running this command against all user mailboxes in our organization, I received this error on some accounts:

 The Junk E-Mail configuration couldn't be set. The user needs to sign in to Outlook Web App before they can modify their Safe Senders and Recipients or Blocked Senders lists.
    + CategoryInfo          : NotSpecified: (0:Int32) [Set-MailboxJunkEmailConfiguration], DataSourceOperationException
    + FullyQualifiedErrorId : 44A9456E,Microsoft.Exchange.Management.StoreTasks.SetMailboxJunkEmailConfiguration

The error seems to suggest that a user must sign in to Outlook Web App before their Junk Email configuration can be configured. That is somewhat misleading though, in that the only requirement is that the user has signed in to Outlook (I’ve tested this with Outlook 2003 & 2010). As it turned out I received the message for new hires that had not yet started, so it was nothing to really worry about.

Attempting to Remove Additional Mailboxes from Outlook 2010 Fails

I had an issue escalated to me today where a mailbox that had been properly removed from the “Open These Additional Mailboxes” box in the Advanced properties of an Outlook profile did not disappear from Outlook as expected.

We attempted to remove the relevant keys that were still hanging out in:

HKCU\Software\Microsoft\Windows NT\CurrentVersion\Windows Messaging Subsystem\Profile

…and that did not correct the issue, even after a full reboot of the PC.

We then blew away the Outlook profile, permanently deleted everything in   C:\Users\%username%\AppData\Local\Microsoft\Outlook, and recreated the profile. That didn’t work either, so at this point I knew it had to be something on the Exchange server.

I removed the “Full Access” permission using the Exchange Management Console, and then closed and reopened Outlook. Sure enough, the additional mailboxes that were opened were now gone.

Now I’m curious 🙂 So, I go into the Exchange Management Console and add myself to the Full Access list for a resource mailbox I had never connected to or had the occasion to access. I opened up Outlook, and there it was. Funny enough, if I looked in the Advanced properties of my Outlook profile, it did not appear in the list of additional mailboxes to be opened.

Even more curious, I did some digging and found that this is in fact a “feature” of Exchange Server 2010 SP1 when using Outlook 2007 or Outlook 2010:

…which you can see for yourself at http://technet.microsoft.com/en-us/library/bb676551.aspx

I had not yet run in to this “feature,” but found it interesting enough to write about.

Windows Server Backup stuck on “Running Consistency Check for Application Exchange” for Exchange 2010

If you use Windows Backup to back up your Exchange 2010 databases, you may find yourself in a situation where the backup appears hung at “Running consistency check for application Exchange” for hours and hours. You should also see Event ID 3156 in the Application log on the server.

The most likely cause of this is that there are an excessive amount of log files present (like thousands and thousands), which can happen with a large amount of activity or if you haven’t performed a successful backup in a while.

This can really pose a problem if you find yourself rapidly running out of hard disk space, and are concerned the backup will not complete and flush the logs before you do run out.

There are two easy fixes. The first (and safest) thing to do is to simply let it complete. However, depending on the number of log files that can take a really long time… hours and even days.

The second is a little more drastic, but desperate times call for desperate measures. First, stop the active backup. Then, simply go to the properties of each database, and enable Circular Logging. With Exchange 2010 SP1, you do not need to dismount the mailbox store and then mount it again for this to take effect. It should take less than an hour, but ultimately you will see the log files evaporate into the ether. Once that happens, you can disable circular logging and then run your backup again. This time, you should see it pass by the “Running consistency check for application Exchange” step much more quickly.

If that doesn’t resolve it, you might have a problem with the VSS writers. If you run the following command from an elevated command prompt you can check that all of them are in “Stable” state.

vssadmin list writers

If it looks like you have a problem, you can increase your logging level by using the following command in the Exchange Management Shell.

Set-EventLogLevel "MSExchangeIS\9002 System\Exchange Writer" -Level Expert

For good measure, I would also suggest restarting the Microsoft Exchange Server Extension for Windows Server Backup service.

The enhanced logging should point you in the right direction if there are deeper issues. If you do have deeper issues, I would suggest a full server reboot first. before taking any further troubleshooting steps. If you have a chronic issue that keeps coming back though, I would highly recommend troubleshooting it to resolution rather than just rebooting every time.

When finished, don’t forget to return it to the default logging level:

Set-EventLogLevel "MSExchangeIS\9002 System\Exchange Writer" -Level Lowest

Hope this helps!

Common Exchange 2010 Shell Commands for Recipient Administrators

I made this list of Common Exchange 2010 commands for the recipient administrators on our help desk, and it’s come in quite handy to train new people or serve as a quick reference for commands that aren’t used that often.

Migrate an Existing Mailbox from Legacy Version to Exchange 2010
This initiates a new, immediate move request for a mailbox. The BadItemLimit simply allows for 100 corrupt items to be permanently removed before the move request will fail, and the AcceptLargeDataLoss is just a gut check that you really mean it can delete up to 100 corrupt items.

New-MoveRequest -Identity "MailboxAlias" -BadItemLimit 100 –AcceptLargeDataLoss

Give Full Access Permissions to a Mailbox
If someone needs full access to another user’s mailbox or a resource mailbox, this will do the trick.

add-MailboxPermission -identity "MailboxAlias" -User UserID -AccessRights FullAccess

Apply Exchange 2010 Properties to Mailbox
A safe procedure, this ensures all properties are applied to a mailbox that need to be. If you have a mailbox displaying as a “Legacy Mailbox,” even though it is on an Exchange 2010 database, this command will set it straight.

Set-Mailbox –identity “MailboxAlias” –ApplyMandatoryProperties

Apply Default Exchange RBAC permissions to User
If the mailbox creation process goes really wrong or a user is having difficulties using features of OWA or Outlook, the problem might be that the user has not had a Role Assignment Policy applied (generally done by default).

get-mailbox “MailboxAlias” | set-Mailbox -RoleAssignmentPolicy "Default Role Assignment Policy"

Export Maibox to PST
No need to use Outlook anymore to export a mailbox to PST, this lets you do it right from Exchange.

New-MailboxExportRequest -Mailbox “MailboxAlias” -FilePath \\\$\.pst

To Check Status of Export Request
Self explanatory, sometimes it’s nice to know the export request is still proceeding

Get-MailboxExportRequest \MailboxExport | fl

Convert Exchange 2010 User Mailbox to Room Mailbox
If you migrated a User Mailbox from Exchange 2003 that was really a room, this command will convert it into an actual “Room Type” mailbox.

set-Mailbox –identity “MailboxAlias” –Type Room

Allow Everyone to See Calendar Details for Room Mailbox
Handy if you want to allow all users to see the calendar details for Room Mailbox bookings.

Set-MailboxFolderPermission MailboxAlias:\Calendar -User Default -AccessRights Reviewer

Enable Resource Booking Attendant
After creating a room mailbox, you may want to enable the Booking Attendant, this allows you to do that quickly from the EMC.

Set-CalendarProcessing "MailboxAlias" -AutomateProcessing AutoAccept

Give Permissions to User to “Send As” Distribution Group
Kind of self explanatory

add-ADPermission -identity "GroupDisplayName" -user "UserNameThatGetsSendAsRights" -AccessRights ExtendedRight -ExtendedRights "send as"

Give Permissions to User to “Send on Behalf Of” Distribution Group
Kind of self-explanatory; The BypassSecurityGroupManagerCheck flag allows a Recipient Administrator to perform this action without also being an owner of the group. Note, for the BypassSecurityGroupManagerCheck flag to work, the administrator must have the “Role Management” role assigned either directly or indirectly.

set-DistributionGroup "GroupDisplayName" -GrantSendOnBehalfTo "UserName" –BypassSecurityGroupManagerCheck

Upgrade an Existing Distribution Group from Legacy Version to Exchange 2010
If you need to just do a one-shot upgrade of a distribution group from legacy exchange to Exchange 2010, this will do the trick. I have another article if you want to do all distribution groups at once.

set-DistributionGroup –identity “DistributionGroupName”

How to Migrate Receive Connector’s from Exchange 2003 to Exchange 2007/2010 (Multiple IP Address Restrictions)

Setting up a new receive connector in Exchange 2007/2010 is pretty straightforward stuff and would hardly be worth writing a whole article about.

However, in a migration scenario where you need to migrate a specific receive connector from Exchange 2003 to multiple Exchange 2007/2010 servers and it has LOTS of IPs listed under Relay Restrictions the process can be a little more daunting.

Further, if you want to allow that list of IPs to relay to external domains without restriction, there is an additional step you must take under Exchange 2007/2010. Simply allowing those IPs to relay is insufficient.

Get the Massive List of IPs from Exchange 2003
The first thing you need to do is get the list of IPs under Relay Restrictions. If you have just a few IPs, this is no big deal and you’ll probably just write them down. If you’re in a situation like I was this past week though, you may have dozens and dozens or even hundreds of IPs, and writing them down is simply out of the question.

Fortunately, Microsoft makes it semi-easy with the SMTP Internet Protocol Restriction and Accept/Deny List Configuration (IPSec.vbs) script, which you can download here:

How to use the IPsec.vbs program to export an SMTP relay list from a computer that is running Exchange Server 2003

Once you download and extract the contents to your C: drive, just run the following to export a list of Relay Restrictions.

cscript IPsec.vbs –s ExchangeServer –o e –r relay –d DomainController > C:\exipsecurity\exportconnectionlist.txt

NOTE: You must TYPE this command in the command window from the c:\ExIPSecurity directory. DO NOT COPY AND PASTE… it won’t work if you do that. No, it makes no sense, it’s just one of the mysteries of the Microsoft ‘verse. Now that you have your list of IPs, it’s time to create your receive connector. Create a New Receive Connector on an Exchange 2007/2010 Server As I said before, creating a new receive connector in Exchange 2007/2010 is relatively straightforward, and you can find full details from Microsoft here: New-ReceiveConnector http://technet.microsoft.com/en-us/library/bb125139.aspxI’ve taken the liberty of writing out the command below though to do what we want. This is the command we’ll run on the first Exchange Hub Transport server (technically you could run it against all of your hub transports, but I’ve also included another command to copy the Remote IP Ranges to another server).  Note that you must list out all of the IPs you wish to add to the RemoteIPRanges separated by a comma:

New-ReceiveConnector "ReceiveConnectorName" -Server Exchange2010Server -Bindings -AuthMechanism None -PermissionGroups AnonymousUsers –RemoteIPRanges IPAddress1,IPAddress2,IPAddress3

Once you have the connector created on your first Hub Transport server, you can run this command to create an identical connector on the next Exchange Hub Transport server. Again, you could use the previous command. For the sake of argument though let’s say you’re adding a Hub Transport at a much later date and don’t have the original command or list of IPs handy. This command will create a new connector, and copy the Remote IP Ranges:

New-ReceiveConnector "ReceiveConnectorName" -Server Exchange2010Server -Bindings -AuthMechanism None -PermissionGroups AnonymousUsers -RemoteIPRanges ( Get-ReceiveConnector "Exchange2010Server\ReceiveConnectorName" ).RemoteIPRanges

Now that you have your connectors on all of your Hub Transport servers, it’s time to allow them to relay to external recipients.

Assign Permissions to Receive Connector to Relay to External Recipients

In Exchange 2010, it’s not enough to allow a server to relay by listing its IP, you must further elevate permissions to give it this capability.

To do so, just use the following command on each of your Hub Transport servers with the relevant Receive Connector installed, replacing the bold parts with the appropriate names for your environment.

Get-ReceiveConnector "Exchange2010Server\RecieveConnectorName" | Add-ADPermission -User "NT AUTHORITY\ANONYMOUS LOGON" -ExtendedRights "ms-Exch-SMTP-Accept-Any-Recipient"

At this point, you’ll be able to change DNS records or otherwise redirect the relaying servers/devices to your Exchange 2010 environment, and all will work just as well as it did with Exchange 2003.

Microsoft Exchange Team re-releases Update Rollup 4 for Exchange Server 2010 SP1 (with a Quirk)

On Wednesday, July 27th, the Microsoft Exchange Team released Update Rollup 4 v2, the replacement for Update Rollup 4 released on June 20th that was pulled for various issues.

While patching my own servers, I ran in to a bit of an issue trying to get it to install. I kept getting error 1603 (see below for full error).

Upon further review it was stating that I had installed an interim patch for Exchange 2010. The error log also stated I needed to remove that interim patch before I could install Update Rollup 4, and that I needed to run the update as an administrator.

One problem, I hadn’t installed any interim patches and the account I was using has god rights on the domain.

It turns out the solution was simple, but undocumented. I simply ran the update from the command line with elevated administrative permissions (i.e. I right-clicked on Command Prompt and selected Run as Administrator), and it installed without incident.

Also, if you want to dramatically speed up the amount of time it takes to install Hotfixes in Exchange 2010, remember to do the following:

  1. Open Internet Explorer
  2. Go to Tools – Internet Options
  3. Click on the Advanced tab
  4. Scroll down to Security
  5. Uncheck “Check for publisher’s certificate revocation”
  6. Hit Apply, then OK and install the Hotfix according the procedure above.

You can read all about the patch here:


…and read the more candid release on the Microsoft Exchange Team blog here:


Here’s the full error I was getting:

MSI (c) (34:D8) [08:18:24:200]: Product: Microsoft Exchange Server – Update ‘Update Rollup 4-v2 for Exchange Server 2010 Service Pack 1 (KB2579150) 14.1.323.6’ could not be installed. Error code 1603. Additional information is available in the log file [redacted]. MSI (c) (34:D8) [08:18:24:200]: Windows Installer installed an update. Product Name: Microsoft Exchange Server. Product Version: Product Language: 1033. Manufacturer: Microsoft Corporation. Update Name: Update Rollup 4-v2 for Exchange Server 2010 Service Pack 1 (KB2579150) 14.1.323.6. Installation success or error status: 1603. MSI (c) (34:D8) [08:18:24:216]: Product: Microsoft Exchange Server — Configuration failed. MSI (c) (34:D8) [08:18:24:216]: Windows Installer reconfigured the product. Product Name: Microsoft Exchange Server. Product Version: Product Language: 1033. Manufacturer: Microsoft Corporation. Reconfiguration success or error status: 1603.

How to Upgrade the Distribution Group version from Exchange 2003 to Exchange 2010 for All Objects

In my Exchange environment, I have thousands and thousands of distribution groups, all of which needed to be upgraded to the latest Exchange version so that they could be easily managed in the Exchange Management Console by our help desk.

Not wanting them to have to manually upgrade each one as they came across them, I looked for a way to upgrade all of them all at once. It’s actually quite simple, just run the command below… it simply retrieves each Distribution Group and then runs the “Set” command against it. It doesn’t actually make any modifications to the object though, it just upgrades it to the latest version.

get-DistributionGroup | set-DistributionGroup