Posts RSS Comments RSS 253 Posts and 411 Comments till now

Using the LDAP stats control with Powershell

At TEC this year I did a session with Darren the "GPO Guy" regarding using S.DS.Protocols to get back query stats from Active Directory. I mentioned my stats control code several times and I promised I would post the code so here it is:

What: The stats control is a LDAP control that you can pass that will tell the server to return its internal stats on a query.

Why: The stats control is a great way to see what the Domain Controller does with your filter. Like what indexes it hits, how many entries it had to visit, how much time the DC spent, and entries visited. It is very useful in creating the most efficient LDAP Query possible.

How: I Used System.DirectoryServices.Protocols.DirectoryControl to pass the LDAP control to the Server and I used System.DirectoryServices.Protocols.BERConverter along with the protocol spec here: LDAP_SERVER_GET_STATS_OID: 1.2.840.113556.1.4.970 to decode the Byte Array that was returned.

Here is what is Returned: For 2000
- threadCount: Number of threads that were processing LDAP requests on the DC at the time the search operation was performed.
- coreTime: The time in milliseconds which core logic in the DC spent processing the request.
- callTime: The overall time in milliseconds that the DC spent processing the request.
- searchSubOperations: The number of individual operations which the DC performed in processing the request.

For 2003/2008
- threadCount: Number of threads that were processing LDAP requests on the DC at the time the search operation was performed.
- callTime: The overall time in milliseconds that the DC spent processing the request
- entriesReturned: The number of objects returned in the search result.
- entriesVisited: The number of objects that the DC considered for inclusion in the search result.
- filter: String which represents the optimized form of the search filter used by the DC to perform a search. This very well may be different than the filter that was passed.
- index: String which indicates which database indexes were used by the DC to perform the search.

For 2008 Only
- pagesReferenced: The number of database pages referenced by the DC in processing the search.
- pagesRead: The number of database pages read from disk.
- pagesPreread: The number of database pages preread from disk by the DC in processing the search.
- pagesDirtied: The number of clean database pages modified by the DC in processing the search.
- pagesRedirtied: The number of previously modified database pages that were modified by the DC in processing the search.
- logRecordCount: The number of database log records generated by the DC in processing the search.
- logRecordBytes: The size in bytes of database log records generated by the DC in processing the search.

Note:
- Must have SE_DEBUG_PRIVILEGE
- I did NOT implement SO_EXTENDED_FMT flag.
- I did NOT test 2000.
- The functions that decodes Byte Array actually return objects, but for this test I just outputed the test to mimic ADFind.exe
- Special thanks to Robin Caron, joe Richards, and Dmitri Gavrilov for help with the decoding.
- Here is GREAT Doc on the Controls (and everything else AD) [MS-ADTS]: Active Directory Technical Specification

Code:
  1. Param(
  2. $filter = "(objectclass=*)",
  3. $base,
  4. $Server,
  5. [int]$pageSize = 1000,
  6. [string[]]$props = @("1.1"),
  7. [switch]$StatsOnly,
  8. [switch]$Verbose
  9. )
  10. function CreateStatsObject2008{
  11. Param($StatsArray)
  12. $DecodedArray = [System.DirectoryServices.Protocols.BerConverter]::Decode("{iiiiiiiiiaiaiiiiiiiiiiiiii}",$StatsArray) # Win2008
  13. $myStatsObject = New-Object System.Object
  14. $myStatsObject | Add-Member -Name "ThreadCount" -Value $DecodedArray[1] -MemberType "NoteProperty"
  15. $myStatsObject | Add-Member -Name "CallTime" -Value $DecodedArray[3] -MemberType "NoteProperty"
  16. $myStatsObject | Add-Member -Name "EntriesReturned" -Value $DecodedArray[5] -MemberType "NoteProperty"
  17. $myStatsObject | Add-Member -Name "EntriesVisited" -Value $DecodedArray[7] -MemberType "NoteProperty"
  18. $myStatsObject | Add-Member -Name "Filter" -Value $DecodedArray[9] -MemberType "NoteProperty"
  19. $myStatsObject | Add-Member -Name "Index" -Value $DecodedArray[11] -MemberType "NoteProperty"
  20. $myStatsObject | Add-Member -Name "PagesReferenced" -Value $DecodedArray[13] -MemberType "NoteProperty"
  21. $myStatsObject | Add-Member -Name "PagesRead" -Value $DecodedArray[15] -MemberType "NoteProperty"
  22. $myStatsObject | Add-Member -Name "PagesPreread" -Value $DecodedArray[17] -MemberType "NoteProperty"
  23. $myStatsObject | Add-Member -Name "PagesDirtied" -Value $DecodedArray[19] -MemberType "NoteProperty"
  24. $myStatsObject | Add-Member -Name "PagesRedirtied" -Value $DecodedArray[21] -MemberType "NoteProperty"
  25. $myStatsObject | Add-Member -Name "LogRecordCount" -Value $DecodedArray[23] -MemberType "NoteProperty"
  26. $myStatsObject | Add-Member -Name "LogRecordBytes" -Value $DecodedArray[25] -MemberType "NoteProperty"
  27. $myStatsObject
  28. }
  29. function CreateStatsObject2003{
  30. Param($StatsArray)
  31. $DecodedArray = [System.DirectoryServices.Protocols.BerConverter]::Decode("{iiiiiiiiiaia}",$StatsArray) # Win2003
  32. $myStatsObject = New-Object System.Object
  33. $myStatsObject | Add-Member -Name "ThreadCount" -Value $DecodedArray[1] -MemberType "NoteProperty"
  34. $myStatsObject | Add-Member -Name "CallTime" -Value $DecodedArray[3] -MemberType "NoteProperty"
  35. $myStatsObject | Add-Member -Name "EntriesReturned" -Value $DecodedArray[5] -MemberType "NoteProperty"
  36. $myStatsObject | Add-Member -Name "EntriesVisited" -Value $DecodedArray[7] -MemberType "NoteProperty"
  37. $myStatsObject | Add-Member -Name "Filter" -Value $DecodedArray[9] -MemberType "NoteProperty"
  38. $myStatsObject | Add-Member -Name "Index" -Value $DecodedArray[11] -MemberType "NoteProperty"
  39. $myStatsObject
  40. }
  41. function CreateStatsObject2000{
  42. Param($StatsArray)
  43. $DecodedArray = [System.DirectoryServices.Protocols.BerConverter]::Decode("{iiiiiiii}",$StatsArray) # Win2000
  44. $myStatsObject = New-Object System.Object
  45. $myStatsObject | Add-Member -Name "ThreadCount" -Value $DecodedArray[1] -MemberType "NoteProperty"
  46. $myStatsObject | Add-Member -Name "CoreTime" -Value $DecodedArray[3] -MemberType "NoteProperty"
  47. $myStatsObject | Add-Member -Name "CallTime" -Value $DecodedArray[5] -MemberType "NoteProperty"
  48. $myStatsObject | Add-Member -Name "searchSubOperations" -Value $DecodedArray[7] -MemberType "NoteProperty"
  49. $myStatsObject
  50. }
  51.  
  52. if($Verbose){$VerbosePreference\3 = "Continue"}
  53.  
  54. Write-Verbose " - Loading System.DirectoryServices.Protocols"
  55. [VOID][System.Reflection.Assembly]::LoadWithPartialName("System.DirectoryServices.Protocols")
  56.  
  57. [int]$pageCount = 0
  58. [int]$objcount = 0
  59.  
  60. if(!$Server)
  61. {
  62. $rootDSE = [ADSI]"LDAP://rootDSE"
  63. $Server = $rootDSE.dnsHostName
  64. if(!$base){$base = $rootDSE.defaultNamingContext}
  65. switch ($rootDSE.domainControllerFunctionality)
  66. {
  67. 0 {$expression = 'CreateStatsObject2000 $stats'}
  68. 2 {$expression = 'CreateStatsObject2003 $stats'}
  69. 3 {$expression = 'CreateStatsObject2008 $stats'}
  70. }
  71. }
  72.  
  73. Write-Verbose " - Using Server: [$Server]"
  74. Write-Verbose " - Using Base: [$base]"
  75. Write-Verbose " - Using Filter: [$filter]"
  76. Write-Verbose " - Page Size: [$PageSize]"
  77. Write-Verbose " - Returning: [$props]"
  78. Write-Verbose " - CSV: [$csv]"
  79. Write-Verbose " - NoHeaders: [$noHeader]"
  80. Write-Verbose " - Count: [$Count]"
  81. Write-Verbose " - StatsOnly: [$StatsOnly]"
  82. Write-Verbose " - Expression: [$expression]"
  83.  
  84. Write-Verbose " - Creating LDAP connection Object"
  85. $connection = New-Object System.DirectoryServices.Protocols.LdapConnection($Server)
  86. $Subtree = [System.DirectoryServices.Protocols.SearchScope]"Subtree"
  87.  
  88. Write-Verbose " + Creating SearchRequest Object"
  89. $SearchRequest = New-Object System.DirectoryServices.Protocols.SearchRequest($base,$filter,$Subtree,$props)
  90.  
  91. Write-Verbose " - Creating System.DirectoryServices.Protocols.PageResultRequestControl Object"
  92. $PagedRequest = New-Object System.DirectoryServices.Protocols.PageResultRequestControl($pageSize)
  93.  
  94. Write-Verbose " - Creating System.DirectoryServices.Protocols.SearchOptionsControl Object"
  95. $SearchOptions = New-Object System.DirectoryServices.Protocols.SearchOptionsControl([System.DirectoryServices.Protocols.SearchOption]::DomainScope)
  96.  
  97. Write-Verbose " - Creating System.DirectoryServices.Protocols.DirectoryControl Control for OID: [1.2.840.113556.1.4.970]"
  98. $oid = "1.2.840.113556.1.4.970"
  99. $StatsControl = New-Object System.DirectoryServices.Protocols.DirectoryControl($oid,$null,$false,$true)
  100.  
  101. Write-Verbose " - Adding Controls"
  102. [void]$SearchRequest.Controls.add($pagedRequest)
  103. [void]$SearchRequest.Controls.Add($searchOptions)
  104. [void]$SearchRequest.Controls.Add($StatsControl)
  105.  
  106. $start = Get-Date
  107. while ($True)
  108. {
  109. # Increment the pageCount by 1
  110. $pageCount++
  111.  
  112. # Cast the directory response into a SearchResponse object
  113. Write-Verbose " - Cast the directory response into a SearchResponse object"
  114. $searchResponse = $connection.SendRequest($searchRequest)
  115.  
  116. # Display the retrieved page number and the number of directory entries in the retrieved page
  117. Write-Verbose (" - Page:{0} Contains {1} response entries" -f $pageCount,$searchResponse.entries.count)
  118.  
  119.  
  120. Write-Verbose " - Returning Stats for Page:$PageCount"
  121. $stats = $searchResponse.Controls[0].GetValue()
  122. $ResultStats = invoke-Expression $expression
  123. if($pageCount -eq 1)
  124. {
  125. $StatsFilter = $ResultStats.Filter
  126. $StatsIndex = $ResultStats.Index
  127. Write-Verbose " + Setting Filter to [$StatsFilter]"
  128. Write-Verbose " + Setting Index to [$StatsIndex]"
  129. }
  130.  
  131. # If Cookie Length is 0, there are no more pages to request"
  132. if ($searchResponse.Controls[1].Cookie.Length -eq 0)
  133. {
  134. if($count){$objcount}
  135. "`nStatistics"
  136. "================================="
  137. "Elapsed Time: {0} (ms)" -f ((Get-Date).Subtract($start).TotalMilliseconds)
  138. "Returned {0} entries of {1} visited - ({2})`n" -f $ResultStats.EntriesReturned,$ResultStats.EntriesVisited,($ResultStats.EntriesReturned/$ResultStats.EntriesVisited).ToString('p')
  139. "Used Filter:"
  140. "- {0}`n" -f $StatsFilter
  141. "Used Indices:"
  142. "- {0}`n" -f $StatsIndex
  143. break
  144. }
  145.  
  146. # Set the cookie of the pageRequest equal to the cookie of the pageResponse to request the next
  147. # page of data in the send request and cast the directory control into a PageResultResponseControl object
  148. Write-Verbose " - Setting Cookie on SearchResponse to the PageReQuest"
  149. $pagedRequest.Cookie = $searchResponse.Controls[1].Cookie
  150. }

joe’s Response to my perf testing.

I really appreciate him taking the time to post. He didn’t have to. It can be found HERE

A couple of things here:

1) I don’t want to pretend to be a software developer. I have not the first clue when it comes to writing apps. I am just now learning how data types affect performance. It is quite a leap from scripter to developer. My point here is that adfind is an app that has grown over time and has what is called “feature creep.” Most of the features were not intended and joe has just squeezed them in. I have no doubt if joe wanted to he could re-write adfind in a way that would absolutely blow my little feature lacking script away.

2) My point in this exercise is that when it comes to AD and Perf is concern there is a potential option out there if you do not mind the extra work.

3) I respect joe not only for his developer skills but as a person with a ton of knowledge about AD and a willingness to share that knowledge. joe does not advertise on his site and he gives all his tools away for free. He doesnt ask for anything in return.

4) My testing is not over. IMO for this to be a real success I need to be able to achieve similar performance returning objects… I mean that is the whole point right?

5) I will be working with Darren
(GPOGuy) from SDMSoftware to produce a CMDLet version of my script with a ton more features so watch his site.

Working with LDAP Stats Control in Powershell

What: The stats control is a LDAP control that you can pass that will tell the server to return its internal stats on a query.

Why: The stats control is a great way to see what the Domain Controller does with your filter. Like what indexes it hits, how many entries it had to visit, how much time the DC spent, and entries visited. It is very useful in creating the most efficient LDAP Query possible.

How: I Used System.DirectoryServices.Protocols.DirectoryControl to pass the LDAP control to the Server and I used System.DirectoryServices.Protocols.BERConverter along with the protocol spec here: LDAP_SERVER_GET_STATS_OID: 1.2.840.113556.1.4.970 to decode the Byte Array that was returned.

Here is what is Returned:
For 2000
threadCount: Number of threads that were processing LDAP requests on the DC at the time the search operation was performed.
coreTime: The time in milliseconds which core logic in the DC spent processing the request.
callTime: The overall time in milliseconds that the DC spent processing the request.
searchSubOperations: The number of individual operations which the DC performed in processing the request.

For 2003/2008
threadCount: Number of threads that were processing LDAP requests on the DC at the time the search operation was performed.
callTime: The overall time in milliseconds that the DC spent processing the request
entriesReturned: The number of objects returned in the search result.
entriesVisited: The number of objects that the DC considered for inclusion in the search result.
filter: String which represents the optimized form of the search filter used by the DC to perform a search. This very well may be different than the filter that was passed.
index: String which indicates which database indexes were used by the DC to perform the search.

For 2008 Only
pagesReferenced: The number of database pages referenced by the DC in processing the search.
pagesRead: The number of database pages read from disk.
pagesPreread: The number of database pages preread from disk by the DC in processing the search.
pagesDirtied: The number of clean database pages modified by the DC in processing the search.
pagesRedirtied: The number of previously modified database pages that were modified by the DC in processing the search.
logRecordCount: The number of database log records generated by the DC in processing the search.
logRecordBytes: The size in bytes of database log records generated by the DC in processing the search.

Note:
– Must have SE_DEBUG_PRIVILEGE
– I did NOT implement SO_EXTENDED_FMT flag.
– I did NOT test 2000.
– The functions that decodes Byte Array actually return objects, but for this test I just outputed the test to mimic ADFind.exe
– Special thanks to Robin Caron, joe Richards, and Dmitri Gavrilov for help with the decoding.
– Here is GREAT Doc on the Controls (and everything else AD) [MS-ADTS]: Active Directory Technical Specification

Code:

Param(
        $filter = "(objectclass=*)",
        $base,
        $Server,
        [int]$pageSize = 1000,
        [string[]]$props = @("1.1"),
        [switch]$StatsOnly,
        [switch]$Verbose
    )
function CreateStatsObject2008{
    Param($StatsArray)
    $DecodedArray = [System.DirectoryServices.Protocols.BerConverter]::Decode("{iiiiiiiiiaiaiiiiiiiiiiiiii}",$StatsArray) # Win2008
    $myStatsObject = New-Object System.Object
    $myStatsObject | Add-Member -Name "ThreadCount"     -Value $DecodedArray[1]  -MemberType "NoteProperty"
    $myStatsObject | Add-Member -Name "CallTime"        -Value $DecodedArray[3]  -MemberType "NoteProperty"
    $myStatsObject | Add-Member -Name "EntriesReturned" -Value $DecodedArray[5]  -MemberType "NoteProperty"
    $myStatsObject | Add-Member -Name "EntriesVisited"  -Value $DecodedArray[7]  -MemberType "NoteProperty"
    $myStatsObject | Add-Member -Name "Filter"          -Value $DecodedArray[9]  -MemberType "NoteProperty"
    $myStatsObject | Add-Member -Name "Index"           -Value $DecodedArray[11] -MemberType "NoteProperty"
    $myStatsObject | Add-Member -Name "PagesReferenced" -Value $DecodedArray[13] -MemberType "NoteProperty"
    $myStatsObject | Add-Member -Name "PagesRead"       -Value $DecodedArray[15] -MemberType "NoteProperty"
    $myStatsObject | Add-Member -Name "PagesPreread"    -Value $DecodedArray[17] -MemberType "NoteProperty"
    $myStatsObject | Add-Member -Name "PagesDirtied"    -Value $DecodedArray[19] -MemberType "NoteProperty"
    $myStatsObject | Add-Member -Name "PagesRedirtied"  -Value $DecodedArray[21] -MemberType "NoteProperty"
    $myStatsObject | Add-Member -Name "LogRecordCount"  -Value $DecodedArray[23] -MemberType "NoteProperty"
    $myStatsObject | Add-Member -Name "LogRecordBytes"  -Value $DecodedArray[25] -MemberType "NoteProperty"
    $myStatsObject
}
function CreateStatsObject2003{
    Param($StatsArray)
    $DecodedArray = [System.DirectoryServices.Protocols.BerConverter]::Decode("{iiiiiiiiiaia}",$StatsArray) # Win2003
    $myStatsObject = New-Object System.Object
    $myStatsObject | Add-Member -Name "ThreadCount"     -Value $DecodedArray[1]  -MemberType "NoteProperty"
    $myStatsObject | Add-Member -Name "CallTime"        -Value $DecodedArray[3]  -MemberType "NoteProperty"
    $myStatsObject | Add-Member -Name "EntriesReturned" -Value $DecodedArray[5]  -MemberType "NoteProperty"
    $myStatsObject | Add-Member -Name "EntriesVisited"  -Value $DecodedArray[7]  -MemberType "NoteProperty"
    $myStatsObject | Add-Member -Name "Filter"          -Value $DecodedArray[9]  -MemberType "NoteProperty"
    $myStatsObject | Add-Member -Name "Index"           -Value $DecodedArray[11] -MemberType "NoteProperty"
    $myStatsObject
}
function CreateStatsObject2000{
    Param($StatsArray)
    $DecodedArray = [System.DirectoryServices.Protocols.BerConverter]::Decode("{iiiiiiii}",$StatsArray) # Win2000
    $myStatsObject = New-Object System.Object
    $myStatsObject | Add-Member -Name "ThreadCount"          -Value $DecodedArray[1]  -MemberType "NoteProperty"
    $myStatsObject | Add-Member -Name "CoreTime"             -Value $DecodedArray[3]  -MemberType "NoteProperty"
    $myStatsObject | Add-Member -Name "CallTime"             -Value $DecodedArray[5]  -MemberType "NoteProperty"
    $myStatsObject | Add-Member -Name "searchSubOperations"  -Value $DecodedArray[7]  -MemberType "NoteProperty"
    $myStatsObject
}

if($Verbose){$VerbosePreference = "Continue"}

Write-Verbose " – Loading System.DirectoryServices.Protocols"
[VOID][System.Reflection.Assembly]::LoadWithPartialName("System.DirectoryServices.Protocols")
   
[int]$pageCount = 0
[int]$objcount = 0

if(!$Server)
{
    $rootDSE = [ADSI]"LDAP://rootDSE"
    $Server = $rootDSE.dnsHostName
    if(!$base){$base = $rootDSE.defaultNamingContext}
    switch ($rootDSE.domainControllerFunctionality)
    {
        0 {$expression = ‘CreateStatsObject2000 $stats’}
        2 {$expression = ‘CreateStatsObject2003 $stats’}
        3 {$expression = ‘CreateStatsObject2008 $stats’}
    }
}

Write-Verbose " – Using Server:  [$Server]"
Write-Verbose " – Using Base:    [$base]"
Write-Verbose " – Using Filter:  [$filter]"
Write-Verbose " – Page Size:     [$PageSize]"
Write-Verbose " – Returning:     [$props]"
Write-Verbose " – CSV:           [$csv]"
Write-Verbose " – NoHeaders:     [$noHeader]"
Write-Verbose " – Count:         [$Count]"
Write-Verbose " – StatsOnly:     [$StatsOnly]"
Write-Verbose " – Expression:    [$expression]"

Write-Verbose " – Creating LDAP connection Object"
$connection = New-Object System.DirectoryServices.Protocols.LdapConnection($Server)  
$Subtree = [System.DirectoryServices.Protocols.SearchScope]"Subtree"

Write-Verbose " + Creating SearchRequest Object"
$SearchRequest = New-Object System.DirectoryServices.Protocols.SearchRequest($base,$filter,$Subtree,$props)

Write-Verbose "   – Creating System.DirectoryServices.Protocols.PageResultRequestControl Object"
$PagedRequest  = New-Object System.DirectoryServices.Protocols.PageResultRequestControl($pageSize)

Write-Verbose "   – Creating System.DirectoryServices.Protocols.SearchOptionsControl Object"
$SearchOptions = New-Object System.DirectoryServices.Protocols.SearchOptionsControl([System.DirectoryServices.Protocols.SearchOption]::DomainScope)

Write-Verbose "   – Creating System.DirectoryServices.Protocols.DirectoryControl Control for OID: [1.2.840.113556.1.4.970]"
$oid = "1.2.840.113556.1.4.970"
$StatsControl = New-Object System.DirectoryServices.Protocols.DirectoryControl($oid,$null,$false,$true)

Write-Verbose "   – Adding Controls"
[void]$SearchRequest.Controls.add($pagedRequest)
[void]$SearchRequest.Controls.Add($searchOptions)
[void]$SearchRequest.Controls.Add($StatsControl)

$start = Get-Date
while ($True)
{
    # Increment the pageCount by 1
    $pageCount++

    # Cast the directory response into a SearchResponse object
    Write-Verbose " – Cast the directory response into a SearchResponse object"
    $searchResponse = $connection.SendRequest($searchRequest)

    # Display the retrieved page number and the number of directory entries in the retrieved page
    Write-Verbose (" – Page:{0} Contains {1} response entries" -f $pageCount,$searchResponse.entries.count)

    Write-Verbose " – Returning Stats for Page:$PageCount"
    $stats = $searchResponse.Controls[0].GetValue()
    $ResultStats = invoke-Expression $expression
    if($pageCount -eq 1)
    {
        $StatsFilter = $ResultStats.Filter
        $StatsIndex = $ResultStats.Index
        Write-Verbose "   + Setting Filter to [$StatsFilter]"
        Write-Verbose "   + Setting Index  to [$StatsIndex]"
    }
   
    # If Cookie Length is 0, there are no more pages to request"
    if ($searchResponse.Controls[1].Cookie.Length -eq 0)
    {
        if($count){$objcount}
        "`nStatistics"
        "================================="
        "Elapsed Time: {0} (ms)" -f ((Get-Date).Subtract($start).TotalMilliseconds)
        "Returned {0} entries of {1} visited – ({2})`n" -f $ResultStats.EntriesReturned,$ResultStats.EntriesVisited,($ResultStats.EntriesReturned/$ResultStats.EntriesVisited).ToString(‘p’)
        "Used Filter:"
        "- {0}`n" -f $StatsFilter
        "Used Indices:"
        "- {0}`n" -f $StatsIndex
        break
    }

    # Set the cookie of the pageRequest equal to the cookie of the pageResponse to request the next
    # page of data in the send request and cast the directory control into a PageResultResponseControl object
    Write-Verbose " – Setting Cookie on SearchResponse to the PageReQuest"
    $pagedRequest.Cookie = $searchResponse.Controls[1].Cookie
}

DSP vs ADFind Chart

In an earlier blog entry Phase 3 of S.DS.P “CSV Output” (as much as 21 sec faster!) I posted some stats that proved that you could achieve similar performance using a Powershell Script and System.DirectoryServices.Protocols as you could with Raw LDAP API from a C/C++ app.

I had several people suggest that I should have a consolidated view of the results. So here it is. I took the Average from each test and put into and Excel spreadsheet and then plotted on a chart.

Here is the screenshot (lower is better)
DSP vs ADFind Chart

Phase 3 of S.DS.P “CSV Output” (as much as 21 sec faster!)

In the next phase I wanted to actually get some data back and display it. Because I just wanted to test performance I kept the test as close as possible. ADFind has the built in ability to output a csv file so I mimicked his output. I also added two properties to return name and sAMAccountName. For those interested, my next step will be to output objects instead of text (really the whole point.)

My expectation for this test was that ADFind would pull ahead due to the processing of each record, but I was pleasantly surprised when the script was actually faster in all environments except 700k Environment. I will let joe explain why his is slower.

Testing
– I ran each 10x in a Row (as joe Suggested) in its own environement
– ADFind I tested using CMD.exe and used ptime.exe for time measurements.
– Get-DSPObject I used Measure-Command and outputed only TotalSeconds

UPDATED!: I had several people suggest that I should have a consolidated view of the results. So here it is. I took the Average from each test and put into and Excel spreadsheet and then plotted on a chart.

Here is the screenshot (lower is better)
DSP vs ADFind Chart

400k objects in a Prod Quality VM running Win2008 RTM 64bit (local)
Average: ADFind = 90.45
Average: DSP = 68.79
Winner: DSP 21.66 secs faster

Get-DSPObject -prop “name”,”sAMAccountName” -csv
Execution time: 70.455s
Execution time: 69.551s
Execution time: 68.466s
Execution time: 68.085s
Execution time: 68.611s
Execution time: 68.179s
Execution time: 68.837s
Execution time: 68.670s
Execution time: 69.105s
Execution time: 67.994s
adfind -b “your dn here” -f “(objectclass=user)” name samaccountname -csv
Execution time: 100.538 s
Execution time: 109.511 s
Execution time: 92.499 s
Execution time: 96.063 s
Execution time: 91.601 s
Execution time: 80.693 s
Execution time: 81.551 s
Execution time: 81.044 s
Execution time: 90.945 s
Execution time: 80.143 s

400k objects in a Prod Quality VM running Win2008 RTM 64bit (remote)
Average: ADFind = 77.18
Average: DSP = 64.23
Winner: DSP 13 secs faster

Get-DSPObject -prop “name”,”sAMAccountName” -csv
Execution time: 67.065 s
Execution time: 63.871 s
Execution time: 63.330 s
Execution time: 63.027 s
Execution time: 62.630 s
Execution time: 64.692 s
Execution time: 64.451 s
Execution time: 64.450 s
Execution time: 64.594 s
Execution time: 64.219 s
adfind -b “your dn here” -f “(objectclass=user)” name samaccountname -csv
Execution time: 77.280 s
Execution time: 77.616 s
Execution time: 77.304 s
Execution time: 76.899 s
Execution time: 77.426 s
Execution time: 76.773 s
Execution time: 76.699 s
Execution time: 77.445 s
Execution time: 77.463 s
Execution time: 76.912 s

700k objects on a Physical machine Win2k3 x86
Average: ADFind = 101.383
Average: DSP = 109.600
Winner: ADFind 8.21 secs faster

Get-DSPObject -prop “name”,”sAMAccountName” -csv
Execution time: 111.088s
Execution time: 109.962s
Execution time: 110.112s
Execution time: 110.832s
Execution time: 110.167s
Execution time: 109.853s
Execution time: 109.757s
Execution time: 110.387s
Execution time: 109.070s
Execution time: 109.065s
adfind -b “your dn here” -f “(objectclass=user)” name samaccountname -csv
Execution time: 101.571 s
Execution time: 101.313 s
Execution time: 101.386 s
Execution time: 101.593 s
Execution time: 101.539 s
Execution time: 100.917 s
Execution time: 101.020 s
Execution time: 101.286 s
Execution time: 101.544 s
Execution time: 101.667 s

300k Objects on Physical Win2k3 x86 (Remote)
Average: ADFind = 49.90
Average: DSP = 44.44
Winner: DSP 5 secs Faster

Get-DSPObject -prop “name”,”sAMAccountName” -csv
Execution time: 44.782 s
Execution time: 44.316 s
Execution time: 44.473 s
Execution time: 44.322 s
Execution time: 44.380 s
Execution time: 44.440 s
Execution time: 44.394 s
Execution time: 44.544 s
Execution time: 44.428 s
Execution time: 44.247 s
adfind -b “your dn here” -f “(objectclass=user)” name samaccountname -csv
Execution time: 51.983 s
Execution time: 51.937 s
Execution time: 51.810 s
Execution time: 49.142 s
Execution time: 48.913 s
Execution time: 49.073 s
Execution time: 48.765 s
Execution time: 49.125 s
Execution time: 49.110 s
Execution time: 49.105 s

200k Objects on my Server at Home (Win2k8 x64)
Average: ADFind = 35.50
Average: DSP = 31.74
Winner: DSP 4 secs Faster

Get-DSPObject -prop “name”,”sAMAccountName” -csv
Execution time: 31.887 s
Execution time: 31.888 s
Execution time: 31.044 s
Execution time: 30.746 s
Execution time: 31.549 s
Execution time: 31.601 s
Execution time: 31.168 s
Execution time: 31.528 s
Execution time: 31.481 s
Execution time: 31.269 s
adfind -b “your dn here” -f “(objectclass=user)” name samaccountname -csv
Execution time: 37.305 s
Execution time: 34.815 s
Execution time: 34.275 s
Execution time: 34.193 s
Execution time: 39.116 s
Execution time: 34.634 s
Execution time: 39.265 s
Execution time: 33.686 s
Execution time: 33.793 s
Execution time: 33.917 s

Here is the script I used for the DSP Tests

function Get-DSPObject {
    Param(
            $filter = "(objectclass=user)",
            $base = ([ADSI]"").distinguishedName,
            $Server,
            [int]$pageSize = 1000,
            [string[]]$props = @("1.1"),
            [switch]$noHeader,
            [switch]$csv,
            [switch]$count
        )
       
    [VOID][System.Reflection.Assembly]::LoadWithPartialName("System.DirectoryServices.Protocols")
       
    [int]$pageCount = 0
    [int]$objcount = 0
   
    if(!$server){$server = ([ADSI]"").distinguishedName -replace  ",","." -replace "dc=","" }
   
    $connection = New-Object System.DirectoryServices.Protocols.LdapConnection($Server)  
    $subtree = [System.DirectoryServices.Protocols.SearchScope]"Subtree"
   
    $searchRequest = New-Object System.DirectoryServices.Protocols.SearchRequest($base,$filter,$subtree,$props)  
    $pagedRequest = New-Object System.DirectoryServices.Protocols.PageResultRequestControl($pageSize)
    $searchOptions = New-Object System.DirectoryServices.Protocols.SearchOptionsControl([System.DirectoryServices.Protocols.SearchOption]::DomainScope)
    $searchRequest.Controls.add($pagedRequest) | out-null
    $searchRequest.Controls.Add($searchOptions) | out-null
   
    # Output Prep
    if($props -notcontains "1.1")
    {
        $MyProps = @()
        foreach($prop in $props){$MyProps += $prop.ToLower()}
        if($csv)
        {
            if(!$noHeader)
            {
                $header = "distinguishedName"
                foreach($prop in $props){$header += ",$prop"}
                $header
            }
        }
        else
        {
            $MyUserObj = New-Object System.Object
            $MyUserObj | Add-Member -name distinguishedName -MemberType "NoteProperty" -value $null
            foreach($prop in $props){$MyUserObj | Add-Member -name $prop -MemberType "NoteProperty" -value $null}
        }
    }
   
    # Process Pages
    while ($True)
    {
        # Increment the pageCount by 1
        $pageCount++
   
        # Cast the directory response into a SearchResponse object
        $searchResponse = $connection.SendRequest($searchRequest)
   
        # Display the retrieved page number and the number of directory entries in the retrieved page
        # "Page:{0} Contains {1} response entries" -f $pageCount,$searchResponse.entries.count
        $objcount += ($searchResponse.entries).count
       
        # Display the entries within this page
        if(!$count)
        {
            if($props -notcontains "1.1")
            {
                foreach($entry in $searchResponse.Entries)
                {
                    if($csv)
                    {
                        $results = "`"{0}`"" -f $entry.distinguishedName
                        foreach($prop in $MyProps)
                        {
                            $results += ",`"{0}`"" -f ($entry.Attributes[$prop][0])
                        }
                        $results
                    }
                    else
                    {
                        $MyUserObj.distinguishedName = $entry.distinguishedName
                        foreach($prop in $MyProps)
                        {
                            $MyUserObj."$prop" = $null
                            $MyUserObj."$prop" = $entry.Attributes[$prop][0]
                        }
                        $MyUserObj
                    }
                }
            }
            else{$searchResponse.Entries | select distinguishedName}
        }
       
        # if this is true, there are no more pages to request
        if ($searchResponse.Controls[0].Cookie.Length -eq 0){if($count){$objcount};break}
   
        # Set the cookie of the pageRequest equal to the cookie of the pageResponse to request the next
        # page of data in the send request and cast the directory control into a PageResultResponseControl object
        $pagedRequest.Cookie = $searchResponse.Controls[0].Cookie
    }
}

foreach($i in (1..10))
{
    $time = Measure-Command { Get-DSPObject -prop "name","sAMAccountName" -csv }
    "Execution time: {0:n3} s" -f $time.TotalSeconds
}

Further Testing with S.DS.P (~1 sec slower)

I have to say I am in awe. I never expected to get so close to ADFind.exe in performance, but if you look below you will notice how close s.ds.p really got.

Before I show the results I want to respond to some of joe’s comments on his blog entry here: PowerShell + S.DS.Protocols Versus AdFind…
btw… I STRONGLY recommend reading this entry… actually subscribing to his blog. While he does spout off random quotes… he provides some incredibly useful info from time to time.

note: References to joe as joe (case intentional) is not ment as disrespect, but simply the way joe refers to himself. If you have the pleasure of getting to know him… you will understand this.

[1] I understand fully that joe really wants .NET and Powershell to succeed, but finds it unlikely.

[2] My testing (although possibly not perfect) was much more precise and I assume his testing had to be skewed by something. Its just not possible. I tried this on 4 completely different systems. Different OS’s, x64 and x86, Hyper-v and ESX, and different domain sizes. By the end of the testing.. I ran approximately 600 test (although I only documented 240.)

[3] While I can appreciate his lack of faith in measure-command. It was the only way I could remove human error and maintain an unbiased count for each test. Measure-command’s overhead is maintained completely outside of the measured expression.

[4] MOST IMPORTANTLY: Perhaps the reason joe questions my purpose of doing these tests is because I did not make it clear. My purpose in these test was to determine if it is feasible to reach the same performance of adfind.exe in Powershell. Why? adfind.exe works great, why not just that. Simply put, if I can achieve close to the same performance, I’m able to get the speed while maintaining the benefit of dealing with objects. While this concept may escape people, once you grasp it… it is key to easy and fast scripts or commands. Nothing more, nothing less. While I will not argue I like a good debate, that is not my soul reason for being.

To be clear, this is only the first step. I now need to process some properties and see if s.ds.p/powershell can compete in that area. As I was with the count I am skeptical about achieving performance even close to adfind, but perhaps that will surprise me as the search did.

[5] My count was NOT off 😛 The logic of using userclass=* vs userclass=user was.

For those inclined to do so… Here is the COMPLETE output of the results without the average
Prod700kResults
Results Prod 400k VM
Small 200k Domain on Local Hardware
Small 200k Domain on VM

This first test was done in a LARGE environment with 700k+ userclass objects.

Prod 700k

This test was done on a production quality VM with approx 400k users.

Prod 400k

This test was on my Home Win2k8 DC. I only have about 200k userclass objects at home, but it is a beefy box.

Local 200k

This is on my laptop that has about 200k userclass objects as well.

VM 200k

Here are the scripts I ran for the testing

This is the actual worker code

[System.Reflection.Assembly]::LoadWithPartialName("System.DirectoryServices.Protocols") | Out-Null

$domainDN = ([ADSI]"").distinguishedName
$domain = $domainDN -replace  ",","." -replace "dc=",""

[int]$pageCount = 0
[int]$pageSize = 1000
[int]$count = 0

$connection = New-Object System.DirectoryServices.Protocols.LdapConnection($domain)  
$subtree = [System.DirectoryServices.Protocols.SearchScope]"Subtree"
$filter = "(objectclass=*)"

$searchRequest = New-Object System.DirectoryServices.Protocols.SearchRequest($DomainDN,$filter,$subtree,"1.1")  
$pagedRequest = New-Object System.DirectoryServices.Protocols.PageResultRequestControl($pageSize)
$searchRequest.Controls.add($pagedRequest) | out-null
$searchOptions = New-Object System.DirectoryServices.Protocols.SearchOptionsControl([System.DirectoryServices.Protocols.SearchOption]::DomainScope)
$searchRequest.Controls.Add($searchOptions) | out-null
 
while ($True)
{
    # Increment the pageCount by 1
    $pageCount++

    # Cast the directory response into a SearchResponse object
    $searchResponse = $connection.SendRequest($searchRequest)

    # Display the retrieved page number and the number of directory entries in the retrieved page
    # "Page:{0} Contains {1} response entries" -f $pageCount,$searchResponse.entries.count
    $count += ($searchResponse.entries).count
   
    # Display the entries within this page
    $searchResponse.Entries | select distinguishedName
    # if this is true, there are no more pages to request
    if ($searchResponse.Controls[0].Cookie.Length -eq 0){write-Output $count;break}

    # Set the cookie of the pageRequest equal to the cookie of the pageResponse to request the next
    # page of data in the send request and cast the directory control into a PageResultResponseControl object
    $pagedRequest.Cookie = $searchResponse.Controls[0].Cookie
}

Script I used to automate the 60 test per Environment.

Param($count=30)

$dn = ([ADSI]"").distinguishedName

function TestOne {
    Write-Host "  + Test ${i}.1"
    Write-Host "    + Running ADFind Test"
    $joeTime      = Measure-Command { D:\Scripts\adfind -b "$dn" -c -f "(objectclass=user)" 2>&1 | out-Null }
    Write-Host "      – $($joetime.TotalSeconds)"
    Write-Host "    + Running DSP Test"
    $DSPTime11    = Measure-command { D:\Scripts\Test-DSProtocalsSP.ps1 }
    Write-Host "      – $($DSPTime11.TotalSeconds)"
    $myresults  = "" | select @{n="ADFind"           ;e={$joeTime.TotalSeconds}},
                              @{n="DSP Using 1.1"    ;e={$DSPTime11.TotalSeconds}}
    $myresults
}
function TestTwo {
    Write-Host "  + Test ${i}.2"
    Write-Host "    + Running DSP Test"
    $DSPTime11    = Measure-command { D:\Scripts\Test-DSProtocalsSP.ps1 }
    Write-Host "      – $($DSPTime11.TotalSeconds)"
    Write-Host "    + Running ADFind Test"
    $joeTime      = Measure-Command { D:\Scripts\adfind -b "$dn" -c -f "(objectclass=user)" 2>&1 | out-Null }
    Write-Host "      – $($joetime.TotalSeconds)"
    $myresults  = "" | select @{n="ADFind"           ;e={$joeTime.TotalSeconds}},
                              @{n="DSP Using 1.1"    ;e={$DSPTime11.TotalSeconds}}
    $myresults
}

Write-Host
for($i = 0 ; $i -le $count ; $i++)
{
    Write-Host "+ Test $i"
    TestOne
    TestTwo
}
Write-Host

My First Venture into S.DS.P and Powershell

There has be much debate and agony on the the slowness of System.DirectoryServices.DirectorySearcher Class. This has lead me down the path of playing with System.DirectoryServices.Protocols (aka s.ds.p.)

By far the best tool out there right now is ADFind by joe Richards. So I used this for my target (although I didnt expect to get close.)

Here is the Guide I used in my Journey:
Introduction to System.DirectoryServices.Protocols (S.DS.P)

I started by writing the script you will find below and tested in a domain with well over 700k users and one with 200K. I did the test three different times in each Domain changing the order so cache hits wouldn’t be an issue. I also use objectclass on all three so index wouldn’t be a factor. I should NOTE that my script ONLY returns the count ATM. I am going to add properties next.

As you will see by the test results below… I got pretty darn close to adfind.exe in regards to perf. I was quite impressed with S.DS.P. Now to be fair, this was just counting objects. I am sure adfind.exe will start sneaking ahead abit further when we start processing properties and such. Can’t wait to see! Understand these test were just to see what tests I should focus on. I will posting another entry on more extensive count test with just DSP Using 1.1 and Adfind.

Test 1 700K Users done Remote

DirectorySearcher : 125.9123257
ADFind : 46.3763349
DSP Using DN : 69.5628776
DSP Using 1.1 : 49.4458161

Test 2 700K Users done Remote

DirectorySearcher : 125.0230257
ADFind : 46.4486472
DSP Using DN : 68.9255288
DSP Using 1.1 : 49.0780736

Test 3 700K Users done Remote

DirectorySearcher : 125.0230257
ADFind : 47.9162918
DSP Using DN : 79.6885386
DSP Using 1.1 : 54.152966

Test 1 200K done Local

DirectorySearcher : 121.1063569
ADFind : 55.2775406
DSP Using DN : 67.897922
DSP Using 1.1 : 28.5441615

Test 2 200K done Local

DirectorySearcher : 80.0894455
ADFind : 23.558696
DSP Using DN : 54.3111576
DSP Using 1.1 : 42.3485998

Test 3 200K done Local

DirectorySearcher : 99.1125363
ADFind : 80.1497852
DSP Using DN : 64.3716824
DSP Using 1.1 : 64.1940421

Summary: adfind.exe was faster (by bout 4sec on Avg.) remotely and larger domain, but protocals was faster (by bout 8 sec on Avg.) local on smaller domain.
Remote

ADFind: 46.92
DSP Using 1.1: 50.89
DSP Using DN: 72.73
DirectorySearcher: 125.32

Local

DSP Using 1.1: 45.03
ADFind: 52.99
DSP Using DN: 62.19
DirectorySearcher: 100.10

Here the script I ran for the test and how I measured the commands. I am going to play with passing the stats control and see what the server says later.

$SearcherExpression = @’
$searcher = new-object System.DirectoryServices.DirectorySearcher([ADSI]"","(objectclass=user)",@("distinguishedName"))
$searcher.pagesize = 1000
$searcher.findall()
‘@

Write-Host "Test 1"
Write-Host ("-"*40)
$myresults1 = "" | select @{n="DirectorySearcher";e={(Measure-command {invoke-expression $SearcherExpression}).TotalSeconds}},
                         @{n="ADFind";e={(Measure-Command { .\adfind -b "dc=corp,dc=lab" -c -f "(objectclass=user)" }).TotalSeconds}},
                         @{n="DSP Using DN";e={(Measure-command { .\Test-DSProtocals.ps1 }).TotalSeconds}},
                         @{n="DSP Using 1.1";e={(Measure-command { .\Test-DSProtocalsSP.ps1 }).TotalSeconds}}
$myresults1 | fl

Write-Host "Test 2"
Write-Host ("-"*40)
$myresults2 = "" | select @{n="ADFind";e={(Measure-Command { .\adfind -b "dc=corp,dc=lab" -c -f "(objectclass=user)" }).TotalSeconds}},
                         @{n="DSP Using 1.1";e={(Measure-command { .\Test-DSProtocalsSP.ps1 }).TotalSeconds}},
                         @{n="DSP Using DN";e={(Measure-command { .\Test-DSProtocals.ps1 }).TotalSeconds}},
                         @{n="DirectorySearcher";e={(Measure-command {invoke-expression $SearcherExpression}).TotalSeconds}}

$myresults2 | fl

Write-Host "Test 3"
Write-Host ("-"*40)
$myresults3 = "" | select @{n="DSP Using DN";e={(Measure-command { .\Test-DSProtocals.ps1 }).TotalSeconds}},
                         @{n="DSP Using 1.1";e={(Measure-command { .\Test-DSProtocalsSP.ps1 }).TotalSeconds}},
                         @{n="DirectorySearcher";e={(Measure-command {invoke-expression $SearcherExpression}).TotalSeconds}},
                         @{n="ADFind";e={(Measure-Command { .\adfind -b "dc=corp,dc=lab" -c -f "(objectclass=user)" }).TotalSeconds}}
$myresults3 | fl

$myresults1,$myresults2,$myresults3

Here is what the output of that Script looks like

S.DS.P : MyTest.ps1 Output

Here is the System.DirectoryServices.Protocols Code

[System.Reflection.assembly]::LoadWithPartialName("system.directoryservices.protocols") | Out-Null
$domain = ([ADSI]"").distinguishedName -replace  ",","." -replace "dc=",""
$DomainDN = "DC=" + $Domain -replace "\.",",DC="
[int]$pageCount = 0
[int]$pageSize = 1000
[int]$count = 0
$connection = New-Object System.DirectoryServices.Protocols.LdapConnection($domain)  
$subtree = [System.DirectoryServices.Protocols.SearchScope]"Subtree"
$filter = "(objectclass=user)"
$searchRequest = New-Object System.DirectoryServices.Protocols.SearchRequest($DomainDN,$filter,$subtree,@("1.1"))  
$pagedRequest = New-Object System.DirectoryServices.Protocols.PageResultRequestControl($pageSize)
$searchRequest.Controls.add($pagedRequest) | out-null
$searchOptions = new-object System.DirectoryServices.Protocols.SearchOptionsControl([System.DirectoryServices.Protocols.SearchOption]::DomainScope)
$searchRequest.Controls.Add($searchOptions) | out-null

while ($true)
{
    ## increment the pageCount by 1
    $pageCount++
    ## cast the directory response into a
    ## SearchResponse object
    $searchResponse = $connection.SendRequest($searchRequest)
    ## verify support for this advanced search operation
    if (($searchResponse.Controls.Length -lt 1) -or
        !($searchResponse.Controls[0] -is [System.DirectoryServices.Protocols.PageResultResponseControl]))
    {
        Write-Host "The server cannot page the result set"
        return;
    }
    ## cast the diretory control into
    ## a PageResultResponseControl object.
    $pageResponse = $searchResponse.Controls[0]
    ## display the retrieved page number and the number of
    ## directory entries in the retrieved page                    
    #"Page:{0} Contains {1} response entries" -f $pageCount,$searchResponse.entries.count
    $count += $searchResponse.entries.count
    ## display the entries within this page
    ## foreach($entry in $searchResponse.entries){$entry.DistinguishedName}
    ## if this is true, there
    ## are no more pages to request
    if ($pageResponse.Cookie.Length -eq 0){write-Host $count;break}
    ## set the cookie of the pageRequest equal to the cookie
    ## of the pageResponse to request the next page of data
    ## in the send request
    $pagedRequest.Cookie = $pageResponse.Cookie
}