I recently ran into an odd problem at a customer where the local computer accounts of a computer joined to a domain started showing the SID periodically. For those not in the know, this is BAD.
WARNING: I did not find a solution to my problem, but this should server as a documentation of what I tried, what did not work, and possible resolutions that were unachievable in the time frame. In fact, all of this could be nothing to do with the problem at all… just saying…
I first notices that even when I added “Domain Users” to a security group in TFS my “Assigned To” list was empty of all users except those specifically added. It looks like when I name a user my local credentials are used and it is successful, but when I add a group it fails.
Figure: Holy service accounts, batman
So I checked the event log and found an error in the TFS Job Service:
1Log Name: Application
2Source: TFS Services
3Date: 2/3/2012 10:01:27 AM
4Event ID: 3071
5Task Category: None
6Level: Warning
7Keywords: Classic
8User: N/A
9Computer: TFS01.company.com
10Description:
11TF53010: The following error has occurred in a Team Foundation component or extension:
12Date (UTC): 2/3/2012 6:01:27 PM
13Machine: TFS01
14Application Domain: TfsJobAgent.exe
15Assembly: Microsoft.TeamFoundation.Framework.Server, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a; v2.0.50727
16Service Host:
17Process Details:
18 Process Name: TFSJobAgent
19 Process Id: 2128
20 Thread Id: 4028
21 Account name: companytfs_service
22
23Detailed Message: TF200035: One or more errors occurred when Team Foundation Server attempted to synchronize the following identity: Domain Admins. Number of errors that occurred: 1.
24++++++++++++++++++++++
25Sync error for identity: Domain Admins
26The server does not support the requested critical extension.
27
28 at System.DirectoryServices.SearchResultCollection.ResultsEnumerator.MoveNext()
29 at System.DirectoryServices.DirectorySearcher.FindOne()
30 at Microsoft.TeamFoundation.Framework.Server.WindowsProvider.GetMembersDirect(Boolean getProperties, DirectoryEntry groupEntry, String groupDomainName, Dictionary`2 members, IIdentitySyncHelper syncHelper, SyncErrors syncErrors, TeamFoundationRequestContext requestContext)
31 at Microsoft.TeamFoundation.Framework.Server.WindowsProvider.SyncADIdentity(TeamFoundationIdentity identity, Boolean includeMembers, TeamFoundationRequestContext requestContext, SyncErrors syncErrors)
32 at Microsoft.TeamFoundation.Framework.Server.WindowsProvider.SyncIdentity(IdentityDescriptor descriptor, Boolean includeMembership, String providerInfo, TeamFoundationRequestContext requestContext, SyncErrors syncErrors)
33 at Microsoft.TeamFoundation.Framework.Server.IdentitySynchronizer.SyncOneGroupMembership(TeamFoundationRequestContext requestContext, TeamFoundationIdentity groupToSync, IdentityComponent myComponent)
34
35Event Xml:
36<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
37 <System>
38 <Provider Name="TFS Services" />
39 <EventID Qualifiers="0">3071</EventID>
40 <Level>3</Level>
41 <Task>0</Task>
42 <Keywords>0x80000000000000</Keywords>
43 <TimeCreated SystemTime="2012-02-03T18:01:27.000000000Z" />
44 <EventRecordID>16464</EventRecordID>
45 <Channel>Application</Channel>
46 <Computer>TFS01.company.com</Computer>
47 <Security />
48 </System>
49 <EventData>
50 <Data>TF53010: The following error has occurred in a Team Foundation component or extension:
51Date (UTC): 2/3/2012 6:01:27 PM
52Machine: TFS01
53Application Domain: TfsJobAgent.exe
54Assembly: Microsoft.TeamFoundation.Framework.Server, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a; v2.0.50727
55Service Host:
56Process Details:
57 Process Name: TFSJobAgent
58 Process Id: 2128
59 Thread Id: 4028
60 Account name: companytfs_service
61
62Detailed Message: TF200035: One or more errors occurred when Team Foundation Server attempted to synchronize the following identity: Domain Admins. Number of errors that occurred: 1.
63++++++++++++++++++++++
64Sync error for identity: Domain Admins
65The server does not support the requested critical extension.
66
67 at System.DirectoryServices.SearchResultCollection.ResultsEnumerator.MoveNext()
68 at System.DirectoryServices.DirectorySearcher.FindOne()
69 at Microsoft.TeamFoundation.Framework.Server.WindowsProvider.GetMembersDirect(Boolean getProperties, DirectoryEntry groupEntry, String groupDomainName, Dictionary`2 members, IIdentitySyncHelper syncHelper, SyncErrors syncErrors, TeamFoundationRequestContext requestContext)
70 at Microsoft.TeamFoundation.Framework.Server.WindowsProvider.SyncADIdentity(TeamFoundationIdentity identity, Boolean includeMembers, TeamFoundationRequestContext requestContext, SyncErrors syncErrors)
71 at Microsoft.TeamFoundation.Framework.Server.WindowsProvider.SyncIdentity(IdentityDescriptor descriptor, Boolean includeMembership, String providerInfo, TeamFoundationRequestContext requestContext, SyncErrors syncErrors)
72 at Microsoft.TeamFoundation.Framework.Server.IdentitySynchronizer.SyncOneGroupMembership(TeamFoundationRequestContext requestContext, TeamFoundationIdentity groupToSync, IdentityComponent myComponent)
73</Data>
74 </EventData>
75</Event>
Figure: TF200035: One or more errors occurred when Team Foundation Server attempted to synchronize the following identity
Lets take a look….
So I went spelunking and found some interesting things. First was that SQL, running under Network Service was not able to contact the domain properly to query information.
1Log Name: Security
2Source: Microsoft-Windows-Security-Auditing
3Date: 2/3/2012 9:55:38 AM
4Event ID: 4625
5Task Category: Logon
6Level: Information
7Keywords: Audit Failure
8User: N/A
9Computer: TFS01.companydomain.com
10Description:
11An account failed to log on.
12
13Subject:
14 Security ID: S-1-5-20
15 Account Name: TFS01$
16 Account Domain: COMPANYDOMAIN
17 Logon ID: 0x3e4
18
19Logon Type: 3
20
21Account For Which Logon Failed:
22 Security ID: S-1-0-0
23 Account Name:
24 Account Domain:
25
26Failure Information:
27 Failure Reason: An Error occured during Logon.
28 Status: 0xc000040a
29 Sub Status: 0x0
30
31Process Information:
32 Caller Process ID: 0x51c
33 Caller Process Name: C:Program FilesMicrosoft SQL ServerMSSQL10_50.MSSQLSERVERMSSQLBinnsqlservr.exe
34
35Network Information:
36 Workstation Name: TFS01
37 Source Network Address: -
38 Source Port: -
39
40Detailed Authentication Information:
41 Logon Process: Authz
42 Authentication Package: Kerberos
43 Transited Services: -
44 Package Name (NTLM only): -
45 Key Length: 0
46
47This event is generated when a logon request fails. It is generated on the computer where access was attempted.
48
49The Subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.
50
51The Logon Type field indicates the kind of logon that was requested. The most common types are 2 (interactive) and 3 (network).
52
53The Process Information fields indicate which account and process on the system requested the logon.
54
55The Network Information fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.
56
57The authentication information fields provide detailed information about this specific logon request.
58 - Transited services indicate which intermediate services have participated in this logon request.
59 - Package name indicates which sub-protocol was used among the NTLM protocols.
60 - Key length indicates the length of the generated session key. This will be 0 if no session key was requested.
61Event Xml:
62<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
63 <System>
64 <Provider Name="Microsoft-Windows-Security-Auditing" Guid="{54849625-5478-4994-A5BA-3E3B0328C30D}" />
65 <EventID>4625</EventID>
66 <Version>0</Version>
67 <Level>0</Level>
68 <Task>12544</Task>
69 <Opcode>0</Opcode>
70 <Keywords>0x8010000000000000</Keywords>
71 <TimeCreated SystemTime="2012-02-03T17:55:38.958188500Z" />
72 <EventRecordID>590839</EventRecordID>
73 <Correlation />
74 <Execution ProcessID="532" ThreadID="5544" />
75 <Channel>Security</Channel>
76 <Computer>TFS01.companydomain.com</Computer>
77 <Security />
78 </System>
79 <EventData>
80 <Data Name="SubjectUserSid">S-1-5-20</Data>
81 <Data Name="SubjectUserName">TFS01$</Data>
82 <Data Name="SubjectDomainName">COMPANYDOMAIN</Data>
83 <Data Name="SubjectLogonId">0x3e4</Data>
84 <Data Name="TargetUserSid">S-1-0-0</Data>
85 <Data Name="TargetUserName">
86 </Data>
87 <Data Name="TargetDomainName">
88 </Data>
89 <Data Name="Status">0xc000040a</Data>
90 <Data Name="FailureReason">%%2304</Data>
91 <Data Name="SubStatus">0x0</Data>
92 <Data Name="LogonType">3</Data>
93 <Data Name="LogonProcessName">Authz </Data>
94 <Data Name="AuthenticationPackageName">Kerberos</Data>
95 <Data Name="WorkstationName">TFS01</Data>
96 <Data Name="TransmittedServices">-</Data>
97 <Data Name="LmPackageName">-</Data>
98 <Data Name="KeyLength">0</Data>
99 <Data Name="ProcessId">0x51c</Data>
100 <Data Name="ProcessName">C:Program FilesMicrosoft SQL ServerMSSQL10_50.MSSQLSERVERMSSQLBinnsqlservr.exe</Data>
101 <Data Name="IpAddress">-</Data>
102 <Data Name="IpPort">-</Data>
103 </EventData>
104</Event>
Figure: This implies that the Machine account does not have permission to AD
Now, when you join a computer to a domain it actually gets an account on the domain and a password just like a user. The account is always the computer name appended with a $ which looks like “tfs01$”. The password for that account is set every 30 days by request of the client, and there can rarely be any problems.
Figure: Domain Controller error in Event Log
But in this case it looks for it looks as if the machine account has become out of sync, so lets try resetting the computer account .
1C:Userstfs_service>netdom reset "tfs01" /domain:companydomain
2The secure channel from TFS01 to the domain COMPANYDOMAIN has
3been reset. The connection is with the
4machine DC01.COMPANYDOMAIN.COM.
5
6The command completed successfully.
Figure: Using netdom to reset the secure channel
This should update AD with a new password set by the client, so lets test it out:
1C:Userstfs_service>Nltest /SC_Verify:companydomain /SERVER:tfs01
2Flags: b0 HAS_IP HAS_TIMESERV
3Trusted DC Name dc01.companydomain.com
4Trusted DC Connection Status Status = 0 0x0 NERR_Success
5Trust Verification Status = 0 0x0 NERR_Success
6
7The command completed successfully
Figure: Using nltest to check the secure channel
After resetting the account I rebooted the server and checked again for the error messages, and dam but they were still there. My next approach was to reset the computer account in Active Directory Users and Computers. After you do this you will need to re-join the computer to the domain as you will get an error on login that states “ The trust relationship between this workstation and the primary domain failed ” for which it it worth investigating. In this case however it is a problem of our own making with the reset, and can be fixed by following:
Even this is not you done. When you do a remove, re-join certain services do not like it and you will have to do the following:
While this should solve most problems with authentication, it did not solve this one. You may also see a Event ID 5722 is logged on your domain controller which should also be fixed, but is a result of the things that I have tried so far.
So where next?
Well, one thing to make sure of is that the TFS 2010 service account has permission to read from the domain , but it unusual to have this problem as this is the default for accounts in AD. If you have a more locked down configuration it may be something you need to look at.
One of my colleagues (Rennie) thought that if the Active Directory Domain Services have not been maintained properly (this is a very small company with no real AD skills in-house) then they may have lost one or more of their domain roles. Specifically the PDC Emulation role.
Figure: It looks like the PDC Emulation role is OK
This was definitely worth a check, and while it was OK, I did notice something that confused and then shocked me. You now those moments when you find out something that just does not compute, and you just stare at it!
This is a Windows 2000 Domain Controller!
Now that I know that a whole host more potential issues rear their ugly heads.
So, now to check that Windows 2000 Service Pack 4 is installed but who knows which hotfix level if any, and what about bugs that were only fixed in later versions of the OS!
Pha! (throws up hands in disgust) Is Windows 2000 domains even supported in TFS?
Team Foundation Server is supported in the following Active Directory modes and functional levels:
- Windows 2000 Active Directory in native mode.
- Windows Server 2003 Active Directory in Windows 2000 native mode.
- Windows Server 2003 Active Directory in Windows Server 2003 functional level.
- Windows Server 2003 R2 in Windows Server 2003 R2 Active Directory forest functional level.
- Trusts and Forests Considerations for Team Foundation Server , MSDN
What do you know, it is supported (somewhat)
Windows 2000 domains are only supported in Native mode and will not work in mixed mode (does anyone still have Windows NT4 in prodution?) so lets take a look.
Figure: Will I am a monkeys uncle, it is!
While this may be true, I still hold some reservation for the notion that a Domain upgrade may be necessary! To be honest, if it is a miss configuration of the domain somewhere in the mists of time, then a domain replacement may be a better option and creating a brand new “Windows 2008 R2 Domain” would have at least he benefit of modern defaults.
While on the Domain Controller I also noticed Event ID 1789 in the event log that got me looking into another article on Error 1789 when you use the LookupAccountName function on a computer that is running Windows Server 2008 R2 that while technically is included in the last service pack was not working of me. I could not even apply the Hotfix as it game me the “this update is not valid for your computer” message that is no help. There are manual steps for Local Groups not resolving domain group name which stopped that error from being listed, which is one more step closer to a stable working system.
Figure: These manual steps are needed on old domains
After all of those steps and spelunking I only have one error message left. The one that started it all… the TF200035… and I can’t seam to get it to talk to Active Directory.
One thing you may want to try is using TfsSecurity.exe to check wither the accounts are in sync. This can be done easily and there are two things I want to check. First the TFS_Service account:
1C:Program FilesMicrosoft Team Foundation Server 2010Tools>TfsSecurity /server
2:http://tfs01:8080/tfs /imx companydomaintfs_service
3TFSSecurity - Team Foundation Server Security Tool
4Copyright (c) Microsoft Corporation. All rights reserved.
5The target Team Foundation Server is http://tfs01:8080/tfs.
6Resolving identity "companydomaintfs_service"...
7
8SID: S-1-5-21-448539723-789336058-1957994488-1766
9
10DN: CN=TFS_Service,OU=Resource,DC=companydomain,DC=com
11
12Identity type: Windows user
13 Logon name: COMPANYDOMAINtfs_service
14 Display name: TFS_Service
15
16Member of 18 group(s):
17e [A] [NwcSandbox]Project Collection Valid Users
18a [A] [CUSTOMER1]Project Administrators
19a [A] [CUSTOMER1-0-2-1]Project Administrators
20s [A] [NwcSandbox]Project Collection Service Accounts
21e [A] [COMPANY]Project Collection Valid Users
22 [A] [TEAM FOUNDATION]SharePoint Web Application Services
23s [A] [TEAM FOUNDATION]Team Foundation Service Accounts
24e [A] [DefaultCollection]Project Collection Valid Users
25a [A] [TEAM FOUNDATION]Team Foundation Administrators
26s [A] [DefaultCollection]Project Collection Service Accounts
27e [A] [TEAM FOUNDATION]Team Foundation Valid Users
28a [A] [DefaultCollection]Project Collection Administrators
29a [A] [COMPANY]Project Collection Administrators
30a [A] [TfsAdmin]Project Administrators
31s [A] [COMPANY]Project Collection Service Accounts
32a [A] [CUSTOMER2]Project Administrators
33 [G] BUILTINAdministrators
34a [A] [NwcSandbox]Project Collection Administrators
35
36Done.
Figure: TfsSecurity /server :http://tfs01:8080/tfs /imx companydomaintfs_service
And second is the machine account:
1C:Program FilesMicrosoft Team Foundation Server 2010Tools>TfsSecurity /server
2:http://tfs01:8080/tfs /imx compnaydomaintfs01$
3TFSSecurity - Team Foundation Server Security Tool
4Copyright (c) Microsoft Corporation. All rights reserved.
5The target Team Foundation Server is http://tfs01:8080/tfs.
6Resolving identity "companydomaintfs01$"...
7
8SID: S-1-5-21-448539723-789336058-1957994488-1761
9
10DN: CN=TFS01,CN=Computers,DC=companydomain,DC=com
11
12Identity type: Windows user
13 Logon name: COMPANYDOMAINTFS01$
14 Display name: TFS01$
15
16Member of 12 group(s):
17e [A] [NwcSandbox]Project Collection Valid Users
18s [A] [NwcSandbox]Project Collection Service Accounts
19e [A] [COMPANY]Project Collection Valid Users
20s [A] [TEAM FOUNDATION]Team Foundation Service Accounts
21e [A] [DefaultCollection]Project Collection Valid Users
22a [A] [TEAM FOUNDATION]Team Foundation Administrators
23s [A] [DefaultCollection]Project Collection Service Accounts
24e [A] [TEAM FOUNDATION]Team Foundation Valid Users
25a [A] [DefaultCollection]Project Collection Administrators
26a [A] [COMPANY]Project Collection Administrators
27s [A] [COMPANY]Project Collection Service Accounts
28a [A] [NwcSandbox]Project Collection Administrators
29
30Done.
Figure: TfsSecurity /server :http://tfs01:8080/tfs /imx compnaydomaintfs01$
Dag-namit, but I was hoping for some sort of help here! Everything looks just fine except for not being able ot query AD. This is looking more and more like a… “someone ticked a box 7 years ago in AD and no one remembers where or why problem”.
For those that have encountered them before, there are almost impossible to debug. This is why small companies tend to use the out-of-the-box config and bug companies buy auditing software.
The very last thing that I can check is that the accounts that have been added to TFS are in deed syncing, even if the groups are not:
Figure: making sure the identities in TFS are up to date
Well, that was my last option and I have no further insights. I am sure that I will need to return to this in the future, but for now I have implemented a workaround for the customer. they can use the system, but without AD groups.
While this sucks…sometimes an internal network configuration beats you…. Humph….
As I have no real solution for this problem that I chased around I have to come up with a workaround that will provide the customer with at least the ability to use TFS. So I created the following local groups at the Server level:
I added them with permission onto the individual team projects and added user accounts directly into these TFS Groups. This lets us secure some work item types, states and have users listed in the drop downs.
Symptom treated….
Solving the problem looks to be something that the customer is unwilling ot pay me to do, but I left them with some advice:
In saying that this may not even be the problem!
Additional Useful Links:
Have fun….
No related videos found.
If you've made it this far, it's worth connecting with our principal consultant and coach, Martin Hinshelwood, for a 30-minute 'ask me anything' call.
We partner with businesses across diverse industries, including finance, insurance, healthcare, pharmaceuticals, technology, engineering, transportation, hospitality, entertainment, legal, government, and military sectors.
Healthgrades
Flowmaster (a Mentor Graphics Company)
Trayport
Xceptor - Process and Data Automation
CR2
Hubtel Ghana
Philips
Milliman
YearUp.org
Higher Education Statistics Agency
Teleplan
Boxit Document Solutions
Capita Secure Information Solutions Ltd
NIT A/S
Microsoft
Genus Breeding Ltd
Graham & Brown
Ericson
Washington Department of Transport
Royal Air Force
Ghana Police Service
Washington Department of Enterprise Services
Department of Work and Pensions (UK)
Nottingham County Council
Xceptor - Process and Data Automation
CR2
Genus Breeding Ltd
Workday
Ericson
DFDS
