Hybrid Search: Inbound Query fails with 401

Last year I came across an issue where Inbound query was failing consistently.

Issue applies to: April 2014 Cumulative Update or a later cumulative update forr SharePoint 2013 on-premises farm.

When doing an Inbound Search ( i.e You search from SPO and expect to get results from SPO  and Onprem) you see the query fails with below error message:

1¾System.Net.WebException: The remote server returned an error: (401) Unauthorized. at System.Net.HttpWebRequest.GetResponse() at Microsoft.SharePoint.Client.SPWebRequestExecutor.Execute() at Microsoft.SharePoint.Client.ClientContext.GetFormDigestInfoPrivate() at Microsoft.SharePoint.Client.ClientContext.EnsureFormDigest() at Microsoft.SharePoint.Client.ClientContext.ExecuteQuery() at Microsoft.Office.Server.Search.RemoteSharepoint.RemoteSharepointEvaluator.RemoteSharepointProducer.RetrieveDataFromRemoteServer(Object unused) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) at Microsoft.Office.Server.Search.RemoteSharepoint.RemoteSharepointEvaluator.RemoteSharepointProducer.ProcessRecordCore(IRecord record)

To investigate further I took a look at the Onprem and SPO side of logs: If you are interested in the stack trace look below or can skip to solution section :)

Logs from OnPrem:

08/07/2014 13:39:07.48        w3wp.exe     0x08A4        SharePoint Portal Server        User Profiles        ae0sx        Unexpected        Error trying to search in the UPA. The exception message is 'System.ArgumentException: Exception of type 'System.ArgumentException' was thrown.  Parameter name: value     at Microsoft.SharePoint.Administration.Claims.SPIdentityProviders.GetIdentityProviderType(String value)     at Microsoft.Office.Server.Security.UserProfileIdentityClaimMapper.SearchUsingNameIdOrThrow(UserProfileManager upManager, String nameId, String nameIdIssuer)     at Microsoft.Office.Server.Security.UserProfileIdentityClaimMapper.GetSingleUserProfileFromClaimsList(UserProfileManager upManager, IEnumerable`1 identityClaims)'        a53bac9c-90c4-0075-b18e-d8c02dfc8e41

08/07/2014 13:39:07.48        w3wp.exe    0x08A4        SharePoint Portal Server        User Profiles        ae0su        High        The set of claims could not be mapped to a single user identity. Exception Exception of type 'System.ArgumentException' was thrown.  Parameter name: value has occured.        a53bac9c-90c4-0075-b18e-d8c02dfc8e41

08/07/2014 13:39:07.48        w3wp.exe    0x08A4        SharePoint Foundation        Claims Authentication        ae0tc        High        The registered mappered failed to resolve to one identity claim. Exception: System.InvalidOperationException: Exception of type 'System.ArgumentException' was thrown.  Parameter name: value ---> System.ArgumentException: Exception of type 'System.ArgumentException' was thrown.  Parameter name: value     at Microsoft.SharePoint.Administration.Claims.SPIdentityProviders.GetIdentityProviderType(String value)     at Microsoft.Office.Server.Security.UserProfileIdentityClaimMapper.SearchUsingNameIdOrThrow(UserProfileManager upManager, String nameId, String nameIdIssuer)     at Microsoft.Office.Server.Security.UserProfileIdentityClaimMapper.GetSingleUserProfileFromClaimsList(UserProfileManager upManager, IEnumerable`1 identityClaims)     --- End of inner exception stack trace ---     at Microsoft.Office.Server.Security.UserProfileIdentityClaimMapper.GetSingleUserProfileFromClaimsList(UserProfileManager upManager, IEnumerable`1 identityClaims)     at Microsoft.Office.Server.Security.UserProfileIdentityClaimMapper.<>c__DisplayClass2.<GetMappedIdentityClaim>b__0()     at Microsoft.SharePoint.SPSecurity.<>c__DisplayClass5.<RunWithElevatedPrivileges>b__3()     at Microsoft.SharePoint.Utilities.SecurityContext.RunAsProcess(CodeToRunElevated secureCode)     at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(WaitCallback secureCode, Object param)     at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(CodeToRunElevated secureCode)     at Microsoft.Office.Server.Security.UserProfileIdentityClaimMapper.GetMappedIdentityClaim(Uri context, IEnumerable`1 identityClaims)     at Microsoft.SharePoint.IdentityModel.SPIdentityClaimMapperOperations.GetClaimFromExternalMapper(Uri contextUri, List`1 claims)        a53bac9c-90c4-0075-b18e-d8c02dfc8e41

08/07/2014 13:39:07.48        w3wp.exe       0x08A4        SharePoint Foundation        Claims Authentication        af3zp        Unexpected        STS Call Claims Saml: Problem getting output claims identity. Exception: 'System.InvalidOperationException: Exception of type 'System.ArgumentException' was thrown.  Parameter name: value ---> System.ArgumentException: Exception of type 'System.ArgumentException' was thrown.  Parameter name: value     at Microsoft.SharePoint.Administration.Claims.SPIdentityProviders.GetIdentityProviderType(String value)     at Microsoft.Office.Server.Security.UserProfileIdentityClaimMapper.SearchUsingNameIdOrThrow(UserProfileManager upManager, String nameId, String nameIdIssuer)     at Microsoft.Office.Server.Security.UserProfileIdentityClaimMapper.GetSingleUserProfileFromClaimsList(UserProfileManager upManager, IEnumerable`1 identityClaims)     --- End of inner exception stack trace ---     at Microsoft.Office.Server.Security.UserProfileIdentityClaimMapper.GetSingleUserProfileFromClaimsList(UserProfileManager upManager, IEnumerable`1 identityClaims)     at Microsoft.Office.Server.Security.UserProfileIdentityClaimMapper.<>c__DisplayClass2.<GetMappedIdentityClaim>b__0()     at Microsoft.SharePoint.SPSecurity.<>c__DisplayClass5.<RunWithElevatedPrivileges>b__3()     at Microsoft.SharePoint.Utilities.SecurityContext.RunAsProcess(CodeToRunElevated secureCode)     at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(WaitCallback secureCode, Object param)     at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(CodeToRunElevated secureCode)     at Microsoft.Office.Server.Security.UserProfileIdentityClaimMapper.GetMappedIdentityClaim(Uri context, IEnumerable`1 identityClaims)     at Microsoft.SharePoint.IdentityModel.SPIdentityClaimMapperOperations.GetClaimFromExternalMapper(Uri contextUri, List`1 claims)     at Microsoft.SharePoint.IdentityModel.SPIdentityClaimMapperOperations.ResolveUserIdentityClaim(Uri contextUri, ClaimCollection inputClaims)     at Microsoft.SharePoint.IdentityModel.SPIdentityClaimMapperOperations.GetIdentityClaim(Uri contextUri, ClaimCollection inputClaims, SPCallingIdentityType callerType)     at Microsoft.SharePoint.IdentityModel.SPSecurityTokenService.GetLogonIdentityClaim(SPRequestInfo requestInfo, IClaimsIdentity inputIdentity, IClaimsIdentity outputIdentity, SPCallingIdentityType callerType)     at Microsoft.SharePoint.IdentityModel.SPSecurityTokenService.EnsureSharePointClaims(SPRequestInfo requestInfo, IClaimsIdentity outputIdentity, SPCallingIdentityType callerType)     at Microsoft.SharePoint.IdentityModel.SPSecurityTokenService.AugmentOutputIdentityForRequest(SPRequestInfo requestInfo, IClaimsIdentity outputIdentity)     at Microsoft.SharePoint.IdentityModel.SPSecurityTokenService.GetOutputClaimsIdentity(IClaimsPrincipal principal, RequestSecurityToken request, Scope scope)'.        a53bac9c-90c4-0075-b18e-d8c02dfc8e41

Logs from SPO: ( Currently only MS can pull these logs) 

RemoteSharepointProducerSystem.Net.WebException: The remote server returned an error: (401) Unauthorized.

at System.Net.HttpWebRequest.GetResponse()

at Microsoft.SharePoint.Client.SPWebRequestExecutor.Execute()

at Microsoft.SharePoint.Client.ClientContext.GetFormDigestInfoPrivate()

at Microsoft.SharePoint.Client.ClientContext.EnsureFormDigest()

at Microsoft.SharePoint.Client.ClientContext.ExecuteQuery()

at Microsoft.Office.Server.Search.RemoteSharepoint.RemoteSharepointEvaluator.RemoteSharepointProducer.RetrieveDataFromRemoteServer(Object unused)

at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)

at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)

at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)

at Microsoft.Office.Server.Search.RemoteSharepoint.RemoteSharepointEvaluator.RemoteSharepointProducer.ProcessRecordCore(IRecord record)

 Solution:

$config = Get-SPSecurityTokenServiceConfig

$config.AuthenticationPipelineClaimMappingRules.AddIdentityProviderNameMappingRule("OrgId Rule", [Microsoft.SharePoint.Administration.Claims.SPIdentityProviderTypes]::Forms, "membership", "urn:federation:microsoftonline")

$config.Update()

Do an IISRESET after this.

Note: Steps need to be executed only on any one SP onprem server.