Another MIM Password Reset Issue

Recently ran into ANOTHER issue with MIM SSPR that I haven’t been able to find documented anywhere – and with the same customer as last time no less.

The issue is with the “base64EncodePasswordFields” configuration parameter for the password reset and registration portals introduced in FIM 4.1.3441 (https://support.microsoft.com/en-us/help/2832389/a-hotfix-rollup-package-build-4-1-3441-0-is-available-for-forefront-id). The parameter was added to support the use of characters in passwords that break ASP.

This is (was) all well and good.  MIM introduced the ability to unlock an account in addition to (or instead of) resetting the password. It appears though, that when “base64EncodePasswordFields” is enabled, it breaks the account unlock functionality (at least on MIM 4.4.1642.0).  This doesn’t appear to be address in the issues resolved by 4.4.1749 either.

The error in the event log looks like this:

Log Name: Forefront Identity Manager
Source: Microsoft.CredentialManagement.ResetPortal
Date: 1/24/2018 10:59:51 AM
Event ID: 3
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: mim.domain.com
Description:
Microsoft.IdentityManagement.CredentialManagement.Portal: System.Web.HttpUnhandledException: ScriptManager_AsyncPostBackError ---> System.FormatException: Invalid length for a Base-64 char array.
at System.Convert.FromBase64String(String s)
at Microsoft.IdentityManagement.CredentialManagement.Portal.Controls.Base64EncodedTextBoxControl.get_Text()
at Microsoft.IdentityManagement.CredentialManagement.Portal.Reset.ValidateNewPassword(Object source, ServerValidateEventArgs args)
at System.Web.UI.WebControls.CustomValidator.OnServerValidate(String value)
at System.Web.UI.WebControls.BaseValidator.Validate()
at System.Web.UI.Page.Validate()
at System.Web.UI.WebControls.Button.RaisePostBackEvent(String eventArgument)
at System.Web.UI.Page.RaisePostBackEvent(IPostBackEventHandler sourceControl, String eventArgument)
at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
--- End of inner exception stack trace ---
at Microsoft.IdentityManagement.CredentialManagement.Portal.Site.ScriptManager_AsyncPostBackError(Object sender, AsyncPostBackErrorEventArgs eventArgs)
at System.Web.UI.ScriptManager.OnAsyncPostBackError(AsyncPostBackErrorEventArgs e)
at System.Web.UI.PageRequestManager.OnPageError(Object sender, EventArgs e)
at System.Web.UI.TemplateControl.OnError(EventArgs e)
at System.Web.UI.Page.HandleError(Exception e)
at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
at System.Web.UI.Page.ProcessRequest(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
at System.Web.UI.Page.ProcessRequest()
at System.Web.UI.Page.ProcessRequest(HttpContext context)
at ASP.default_aspx.ProcessRequest(HttpContext context)
at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

With the exception “Invalid length for a Base-64 char array”, I can only guess that is trying to encode or decode the null values for the new password (as those fields disappear when “Account Unlock” is selected). So right now, it appears as though you have to choose between the two.

Hope it helps somebody.

MIM Password Reset Issue

Recently ran into an issue with MIM SSPR I felt was worth sharing as I haven’t been able to find it documented anywhere.  This was found on 4.4.1642.0, but I suspect it’s been there for a while.  The issue presented itself as a particular user unable to reset their password through SSPR.  Other users were able to just fine, so SSPR configuration wasn’t the issue.

The user gets the standard “An error has occurred. Please try again, and if the problem persists, contact your help desk or system administrator. (Error 3000)” as shown below.

Tracking it through from the SSPR Portal to the MIM Sync server, the following error message was discovered on the Sync server, so it was getting there at least.  Couldn’t find anything in any logs beyond this though. Nothing on the DCs either.

Log Name: Application
Source: FIMSynchronizationService
Date: 1/8/2018 11:46:45 AM
Event ID: 6328
Task Category: Server
Level: Warning
Keywords: Classic
User: N/A
Computer: MIMSync.domain.com
Description:
The server encountered an error while attempting to perform a set/change password operation.
 
"BAIL: MMS(2516): ..\dnutils.cpp(1341): 0x800700b7 (Cannot create a file when that file already exists.)
BAIL: MMS(2516): ..\dnutils.cpp(1341): 0x800700b7 (Cannot create a file when that file already exists.)
BAIL: MMS(2516): ..\dnutils.cpp(1341): 0x800700b7 (Cannot create a file when that file already exists.)
ERR_: MMS(2516): D:\bt\52580\sources\dev\Sync\ma\shared\inc\MAUtils.h(58): Failed getting registry value 'ADMADoNormalization', 0x2
BAIL: MMS(2516): D:\bt\52580\sources\dev\Sync\ma\shared\inc\MAUtils.h(59): 0x80070002 (The system cannot find the file specified.): Win32 API failure: 2
BAIL: MMS(2516): D:\bt\52580\sources\dev\Sync\ma\shared\inc\MAUtils.h(114): 0x80070002 (The system cannot find the file specified.)
ERR_: MMS(2516): D:\bt\52580\sources\dev\Sync\ma\shared\inc\MAUtils.h(58): Failed getting registry value 'ADMARecursiveUserDelete', 0x2
BAIL: MMS(2516): D:\bt\52580\sources\dev\Sync\ma\shared\inc\MAUtils.h(59): 0x80070002 (The system cannot find the file specified.): Win32 API failure: 2
BAIL: MMS(2516): D:\bt\52580\sources\dev\Sync\ma\shared\inc\MAUtils.h(114): 0x80070002 (The system cannot find the file specified.)
ERR_: MMS(2516): D:\bt\52580\sources\dev\Sync\ma\shared\inc\MAUtils.h(58): Failed getting registry value 'ADMARecursiveComputerDelete', 0x2
BAIL: MMS(2516): D:\bt\52580\sources\dev\Sync\ma\shared\inc\MAUtils.h(59): 0x80070002 (The system cannot find the file specified.): Win32 API failure: 2
BAIL: MMS(2516): D:\bt\52580\sources\dev\Sync\ma\shared\inc\MAUtils.h(114): 0x80070002 (The system cannot find the file specified.)
BAIL: MMS(2516): admaexport.cpp(2905): 0x80005000 (unable to get error text)
BAIL: MMS(2516): admaexport.cpp(3292): 0x80005000 (unable to get error text)
ERR_: MMS(2516): ..\ma.cpp(8531): ExportPasswordSet failed with 0x80005000
Forefront Identity Manager 4.4.1642.0"

I’ll spare you the rest of the things I checked and get to the point.  The issue was the user was in an OU with a forward slash in the name (“Something Like/This”).  Moved the user up one level and out of that OU – SSPR works great (even without any sync cycle run).  Move them back to the OU, the problem returns.

So as it turns out, there were more users affected (anyone in an OU with a forward slash), they just hadn’t had to reset their passwords apparently.

EDIT: I just wanted to add that because this issue occurs when calling MIISPasswordSet, this is also likely an issue when using MIM Password Sync as it is the same mechanism to call to AD.

Issues with MIM 2016 SP1

I recently upgraded a MIM deployment to SP1 using the new in-place upgrade, rather than with the previously released SP1 version that required an uninstall and reinstall. While the upgrade wizards themselves ran without issue, there has been some fallout that is probably worth sharing. Here is what I ran into:

  1. Pop ups in the portal (opening a user, sync rule, etc.) just say “Loading…”, but never actually load (at least with IE).
  2. Customized branding done in the “%Program Files%\Common Files\microsoft shared\Web Server Extensions\14\TEMPLATE\LAYOUTS\1033\STYLES\fim.css” file was lost.

Regarding item #1, I tried clearing browser cache, adding the site URL to and removing from Compatibility View, installing updates (on the machine with the browser and the portal servers), and rebooting the client and servers. No change. Last was a complete reset of IE, which required another reboot. That did the trick. I’m not sure what setting in IE was causing the issue, but at least it is fixed.

Item #2 was resolved by temporarily reverting back to the pre-SP1 snapshot of one of the MIM Portal servers, getting the file, and then reinstalling SP1.  Now I need to re-incorporate the customizations inside the CSS.  Fun times.  But least we took the appropriate backups first.  ALWAYS TAKE A BACKUP FIRST!!

The Tedious Azure AD Connect “InvalidSoftMatch”

When doing a deployment of AADConnect, you will inevitably run into the “InvalidSoftMatch” error on exports to Azure AD. In AAD, there are a number of places where an email address or user principal name could be duplicated and causing issues. This is especially the case in larger and multi-forest deployments. To help identify the offending objects in AAD, I’ve put together the following script.

The script connects to Azure AD and caches all users, contacts, and groups, and then it checks for a supplied UPN or SMTP address value (user@domain.com) against those objects and the appropriate attributes. You can supply multiple values without re-querying AAD each time. Sadly, it won’t fix them for you, but it will make your job easier. Remember, because it caches the AAD data, if you make a correction in AAD it will not be reflected in the script results unless you re-run it.

# Find AAD Sync Error Users
# v1.3
# Keith Crosby
# 11.07.2016
#
# Version history
#
#	v1.0	2016-02-03	Initial build
#	v1.1	2016-10-05	Now stops on errors, added check for PS Modules, checks contacts,
#				added looping to avoid re-querying AAD each time, removed AD query
#	v1.2	2016-11-07	Removed "Run As Admin" requirement; added 'No Match Found' text.
#	v1.3    2016-11-17	Fixed search issue with proxy addresses, added groups, changed quit mechanism.
 
#set error preference
$ErrorActionPreference = "Stop"
 
#check for MSOnline Module
if (Get-Module -ListAvailable -Name MSOnline) {
    Import-Module MSOnline
	$MSOversion = (Get-Module MSOnline).Version
	Write-Host "Using Azure AD PowerShell v$($MSOversion.ToString())"
} else {
    Write-Error "Azure AD PowerShell module not installed."
}
 
#Log in to Azure AD
$sp = Get-MSOLServicePrincipal -ErrorAction SilentlyContinue
if (!$sp) {
	Write-Host "Waiting for Azure AD Authentication..."
    Connect-MSOLService
}
 
Write-Host "Querying users..."
$users = Get-MsolUser -All
Write-Host "User Count: $($users.Count)"
Write-Host "Querying deleted users..."
$delUsers = Get-MsolUser -All -ReturnDeletedUsers
Write-Host "Deleted User Count: $($delUsers.Count)"
Write-Host "Querying contacts..."
$contacts = Get-MsolContact -All
Write-Host "Contact Count: $($contacts.Count)"
Write-Host "Querying groups..."
$groups = Get-MsolGroup -All
Write-Host "Group Count: $($groups.Count)"
 
do {
 
	$SearchTerm = Read-Host "`nEnter the SMTP Address or UPN to search for (Enter 'q' to quit)"
	Write-Host 
	if ($SearchTerm.ToLower() -ne "q" -and $SearchTerm) {
		$matchFound = $false
		$users | foreach-object {
            $upn = $_.UserPrincipalName
            $disp = $_.DisplayName
			if ($_.UserPrincipalName -eq $SearchTerm) {
				Write-Host "Matched to AAD User '$disp' $upn on UserPrincipalName" -ForegroundColor Yellow
				$matchFound = $true
			}
			if ($_.SignInName -eq $SearchTerm) {
				Write-Host "Matched to AAD User '$disp' $upn on SignInName" -ForegroundColor Yellow
				$matchFound = $true
			}
			if ($_.AlternateEmailAddress -eq $SearchTerm) {
				Write-Host "Matched to AAD User '$disp' $upn on AlternateEmailAddress" -ForegroundColor Yellow
				$matchFound = $true
			}
			if ($_.ProxyAddresses) {
				$_.ProxyAddresses | foreach-object {
					if ($_.ToLower() -eq "smtp:$SearchTerm".ToLower()) {
						Write-Host "Matched to AAD User '$disp' $upn on ProxyAddress" -ForegroundColor Yellow
						$matchFound = $true
					}
				}
			}
		}
 
		$delUsers | foreach-object {
            $upn = $_.UserPrincipalName
            $disp = $_.DisplayName
			if ($_.UserPrincipalName -eq $SearchTerm) {
				Write-Host "Matched to deleted AAD User '$disp' $upn on UserPrincipalName" -ForegroundColor Yellow
				$matchFound = $true
			}
			if ($_.SignInName -eq $SearchTerm) {
				Write-Host "Matched to deleted AAD User '$disp' $upn on SignInName" -ForegroundColor Yellow
				$matchFound = $true
			}
			if ($_.AlternateEmailAddress -eq $SearchTerm) {
				Write-Host "Matched to deleted AAD User '$disp' $upn on AlternateEmailAddress" -ForegroundColor Yellow
				$matchFound = $true
			}
			if ($_.ProxyAddresses) {
				$_.ProxyAddresses | foreach-object {
					if ($_.ToLower() -eq "smtp:$SearchTerm".ToLower()) {
						Write-Host "Matched to deleted AAD User '$disp' $upn on ProxyAddress" -ForegroundColor Yellow
						$matchFound = $true
					}
				}
			}
		}
 
		$contacts | foreach-object {
            $disp = $_.DisplayName
			if ($_.EmailAddress -eq $SearchTerm) {
				Write-Host "Matched to AAD Contact '$disp' on EmailAddress" -ForegroundColor Yellow
				$matchFound = $true
			}
			if ($_.ProxyAddresses) {
				$_.ProxyAddresses | foreach-object {
					if ($_.ToLower() -eq "smtp:$SearchTerm".ToLower()) {
						Write-Host "Matched to AAD Contact '$disp' on ProxyAddress" -ForegroundColor Yellow
						$matchFound = $true
					}
				}
			}
		}
 
		$groups | foreach-object {
            $disp = $_.DisplayName
			if ($_.EmailAddress -eq $SearchTerm) {
				Write-Host "Matched to AAD Group '$disp' on EmailAddress" -ForegroundColor Yellow
				$matchFound = $true
			}
			if ($_.ProxyAddresses) {
				$_.ProxyAddresses | foreach-object {
					if ($_.ToLower() -eq "smtp:$SearchTerm".ToLower()) {
						Write-Host "Matched to AAD Group '$disp' on ProxyAddress" -ForegroundColor Yellow
						$matchFound = $true
					}
				}
			}
		}
 
	    If (!$matchFound) {
		    Write-Host "No match found!" -ForegroundColor Yellow
	    }
 
	}
 
 
 
}
while ($SearchTerm.ToLower() -ne "q")

Notes on MIM with SQL Always On Availability Groups

Recently got a chance to set up MIM 2016 with SQL Always On Availability Groups. Release notes for the MIM 4.4.1459 hotfix says it is supported, but there doesn’t seem to be a lot of documentation on the “how to” side, so here are my quick notes.  This is not meant to be an step by step guide, just a summary of the key items.

  • Requires MIM build 4.4.1459.0 (first hotfix after SP1)
  • This was all done on Windows Server 2016 and SQL Server 2016 SP1 (just FYI)
  • Requires a Windows Failover Cluster to be created (I didn’t do that part, so no notes there)
  • I also did not do the two SQL installs (primary/secondary)
  • Ensure that the SQL Service is configured to run as a domain account
    • I ran both nodes under the same account without issue
  • Install both MIM DBs normally to the primary node
  • Change FIMSynchronizationService DB from SIMPLE recovery to FULL (required for AoA)
  • If SQL 2016 Standard
    • Only have one DB per AG, so you’ll need two
    • Each AG will need an IP and a dedicated TCP Port
    • You can failover each DB independently this way though
  • If SQL 2016 Enterprise
    • Many DBs per AG, so you can use one
    • Only 1 IP & TCP Port needed
    • All DBs failover together (if in the same AG)
  • The Computer Object for the Windows Failover Cluster must be granted the rights in AD to “create computer objects”.
    • I granted it Full Control on computer objects in the domain, probably overkill.
    • This is to allow SQL to create the computer objects for the Availability Group Listener
  • Create the AG, add the database(s)
    • You can create the AG Listener during the AG creation or later, doesn’t matter
    • When you create the AG Listener, it will create a computer account in AD with that name and register your chosen IP in DNS (you can do DHCP IP, but why would you?)
    • I am using Synchronous Commit, we’ll see how performance is later
    • Also using automatic failover
  • Verify Replication
  • Failover both DBs to secondary
    • Connect SQL Mgmt Studio to the secondary
    • Add SQL Logins for MIM Sync, MIM Service, and MIM Service MA service accounts, you don’t have to give it them permissions; basically the DB permissions are replicated but the SQL logins are not.
  • Ensure Windows Firewall on both SQL nodes allows inbound traffic on your custom ports
  • For MIM Sync
    • Edit Registry Key for DB
      • Find HKLM\System\CurrentControlSet\Services\FIMSynchronizationService\Parameters
      • Change “Server” to be your new AG DNS name and port, separated with a comma:
        • mynewag.domain.com,5001
    • Restart MIM Sync and test connectivity
  • For MIM Service
    • MIM Service did not seem to like the hostname,port syntax
    • Create a SQL Alias using SQL Server Configuration Manager (sqlconfg.exe)
    • Edit Registry Key for DB
      • Find HKLM\System\CurrentControlSet\Services\FIMService
      • Change “DatabaseServer” to your new Alias name
    • Restart MIM Service and test connectivity
  • Test failover and failback, ensuring MIM components have connectivity at all times
  • The final task is to script out the 9 FIM SQL Agent Jobs and re-create them on the secondary node
    • Right click the Job; CREATE TO -> Query Editor
    • Right click in SQL code, select Connection -> Change Connection, select secondary SQL node
    • Execute script (repeat for each one)
    • NOTE: These jobs will fail on the secondary node as it does not have DB access. If you don’t want errors in the logs about it, you can disable them, but you’ll have to figure out how to handle this in a true failover scenario

I’m Back…

After a nice hiccup and outage – I managed to delete the WordPress database in Azure (oops) – the site is back up.  Lost some content, but some was still out there on the old Blogger site.

Moving On Up!

This blog has been moved to a shiny new WordPress site running in Microsoft Azure!  Actually, it’s still a default template, but I’ll try to dress it up a bit.

The new URL is http://www.crosbysite.com.

I know the blog’s been idle for a while, but I’m hoping change that in the near future.

FIM 2010 R2 High CPU with Mmsscrpt.exe

I recently had issue in which mmsscrpt.exe would peg one CPU and hang indefinitely during an sync to the FIM Service MA.  My environment is a single Windows Server 2008 R2 box.  My run sequences consisted of delta imports and delta sync to my two data MAs (SQL and AD) and to the FIM service.  These were followed by an export and a delta import delta sync to the same MAs.  All was fine until the final delta import and delta sync ran.  When it did mmsscrpt.exe jumped to consume one thread and the runs effectively stopped.

Also, a run of full imports and full syncs on all MAs did not show the problem, it was only when the exports were introduced.

In an effort to fix it, I upgraded my FIM 2010 R2 from RTM (4.1.2273.0) to Hotfix Rollup 1 (4.1.2548.0) – no change.

I found a similar issue on the FIM Forum, but it was caused by a .Net Framework bug.  I already had all of the current updates, so those hotfixes were not relevant to me.  However, it was mentioned in there that turning off Exchange 2010 provisioning caused the issue to go away.  The same was true for me, which was the light bulb moment – PowerShell.

In my environment I had deployed the Windows Management Framework 3.0 (PowerShell 3.0, WMI, & WinRM) so that I could use Server 2012’s Server Manager with the older operating systems.  I figured PowerShell 3.0 was breaking something with the remote PowerShell call done when provisioning Exchange, so I went to remove WMF 3.0.  In doing so, I noticed that I apparently had deployed the WMF 3.0 BETA.  I removed it, rebooted, and the issue was gone.

Finally, after realizing it was a beta, I download the current installer for WMF 3.0 (RTM) and installed it.  After another reboot, the issue did not return.

So, the bottom line is that the WMF 3.0 Beta is incompatible with FIM 2010 R2 (not tested with FIM 2010, but I’d be willing to bet it causes issues there too).

TMG 2010: Outbound FTP Pain

Another TMG blog post… 🙂

Was working with a client to replace an ISA 2004 server with a TMG 2010 server.  Both were configured as the clients only firewall, and clients were configured to be both SecureNAT and Web Proxy clients.

The issue was with outbound FTP traffic (internal users access external FTP sites).  When configured as SecureNAT (no proxy configuration in IE) FTP worked fine.  When the client was configured as a Web Proxy client (proxy configured to “Automatically Detect Settings” or proxy server hard set to the IP/name of TMG), FTP would time out and fail to connect to various FTP sites.

The clients are configured to do passive FTP.  As it turns out, when a SecureNAT client uses FTP, TMG connects to the external site with passive FTP.  And when a Web Proxy client uses FTP, TMG connects to the external site with active FTP, which often fails.

The solution is to use a little documented setting in TMG to force the use of passive FTP for Web Proxy clients.  So little documented that all the links refer to ISA 2006.  To resolve, set the DWORD value NonPassiveFTPTransfer to 0 in the registry on the TMG server, which sets the mode to Passive. The default value is 1, indicating that Active mode is used.  The value will likely need to be created and it goes here:

HKEY_LOCAL_MACHINE/SYSTEM/CurrentControlSet/Services/W3Proxy/Parameters

It is also likely that you will need to create the Parameters key.

Make the change and restart the Microsoft Firewall service.

This particular issue is actually documented here and here, but refers to ISA 2006/2004/2000 and is obscure enough that you probably won’t find it unless you know exactly the right keywords to search for.

On a related note, here is the single best article I have seen on working with FTP on ISA and TMG:

http://microsoftguru.com.au/2010/08/27/troubleshooting-outbound-ftp-access-in-isa-tmg-server/

TMG 2010 and Exchange 2010 Resource Forest: Fun with NTLM and Outlook Anywhere

I recently wrapped up a large TMG deployment in support of a new Exchange 2010 resource forest and there were a lot of lessons learned (read: issues that needed to be overcome), so I figured I would try to capture the main ones for the blogosphere.

Part 3 of 3 – Fun with NTLM and Outlook Anywhere

This article assumes a fairly decent knowledge of both TMG and Exchange. It is not meant to be a detailed step-by-step configuration guide. All steps should be tested prior to production rollout.

Before I get into the issue in detail, a little background on the environment. A new Exchange resource forest was built to host Exchange for two separate forests/domains where the user accounts lived.  Everything in the resource forest was built on Windows Server 2008 R2.  TMG is in the same forest and domain as Exchange and Kerberos Constrained Delegation (KCD) is configured. TMG must be in the same domain as whatever is being published in order to use KCD. With KCD configured, our testing from a Windows 7 PC showed that Outlook Anywhere was working perfectly and not prompting for credentials when opening Outlook.

In another round of testing (from an XP PC in a different domain), the user was prompted for authentication. After reviewing all TMG settings and watching TMG logs, it did not appear to be a TMG issue. To test, we forced the client to go direct to a CAS server by editing the host file. They were still prompted for authentication. We tried fetching all windows and office updates, no luck. Since my Windows 7 test PC in the first domain was working perfectly, we decided to try a Windows 7 PC joined to the second domain. The Windows 7 PC in the second domain worked perfectly directly to the CAS (no prompts) and worked perfectly to TMG. So TMG is off the hook here.

The issue, as it turns out, is that Server 2008 R2 is only taking NTLMv2 authentication by default, but the default setting on Windows XP is to only allow LM and NTLM authentication, and never NTLMv2.  The authentication methods are controlled by the LmCompatibilityLevel registry key, found at HKLMSYSTEMCurrentControlSetControlLsa.

Rather than dumbing down the Server 2008 R2 CAS servers, the client changed the LmCompatibilityLevel on the XP workstations from the default value of 0 to the new value of 2 through Group Policy.  The default value of 3 was left alone on the CAS servers.  No more authentication prompts!

 

Part 1 of 3: TMG 2010 and Exchange 2010 Resource Forest: Redirection to Legacy Exchange 2003

Part 2 of 3: TMG 2010 and Exchange 2010 Resource Forest: OWA Login Issues (Account is Disabled??)