Posts RSS Comments RSS 253 Posts and 411 Comments till now

Sync Folder Script.

I initially wrote this for a guy on the NG, but I decided to use at work to sync some application folders I used to store tools and scripts and such. Please Try it out and tell me what you think.

Basic Flow
– Checks for Folders that Source has the Destination Does NOT and Creates.
– Checks for Folders that Destination has the Source Does NOT and Creates.
– Checks for files that Source has the Destination Does NOT and Copies.
– If the file is found in both, the MD5 of both files are checked… Last Write Wins
– Checks for Folders that Destination has the Source Does NOT
– It does NOT Delete anything yet… I am considering using a mirror switch

Param($Source,$Destination)
function Get-FileMD5 {
   Param([string]$file)
   $mode = [System.IO.FileMode]("open")
   $access = [System.IO.FileAccess]("Read")
   $md5 = New-Object System.Security.Cryptography.MD5CryptoServiceProvider
   $fs = New-Object System.IO.FileStream($file,$mode,$access)
   $Hash = $md5.ComputeHash($fs)
   $fs.Close()
   [string]$Hash = $Hash
   Return $Hash
}
function Copy-LatestFile{
    Param($File1,$File2,[switch]$whatif)
    $File1Date = get-Item $File1 | foreach-Object{$_.LastWriteTimeUTC}
    $File2Date = get-Item $File2 | foreach-Object{$_.LastWriteTimeUTC}
    if($File1Date -gt $File2Date)
    {
        Write-Host "$File1 is Newer… Copying…"
        if($whatif){Copy-Item -path $File1 -dest $File2 -force -whatif}
        else{Copy-Item -path $File1 -dest $File2 -force}
    }
    else
    {
        Write-Host "$File2 is Newer… Copying…"
        if($whatif){Copy-Item -path $File2 -dest $File1 -force -whatif}
        else{Copy-Item -path $File2 -dest $File1 -force}
    }
    Write-Host
}

if(!(test-Path $Destination))
{
    New-Item $Destination -type Directory -force | out-Null
}

# Getting Files/Folders from Source and Destination
$SrcEntries = Get-ChildItem $Source -Recurse
$DesEntries = Get-ChildItem $Destination -Recurse

# Parsing the folders and Files from Collections
$Srcfolders = $SrcEntries | Where-Object{$_.PSIsContainer}
$SrcFiles = $SrcEntries | Where-Object{!$_.PSIsContainer}
$Desfolders = $DesEntries | Where-Object{$_.PSIsContainer}
$DesFiles = $DesEntries | Where-Object{!$_.PSIsContainer}

# Checking for Folders that are in Source, but not in Destination
foreach($folder in $Srcfolders)
{
    $SrcFolderPath = $source -replace "\\","\\" -replace "\:","\:"
    $DesFolder = $folder.Fullname -replace $SrcFolderPath,$Destination
    if($DesFolder -ne ""){
        if(!(test-path $DesFolder))
        {
            Write-Host "Folder $DesFolder Missing. Creating it!"
            new-Item $DesFolder -type Directory | out-Null
        }
    }
}

# Checking for Folders that are in Destinatino, but not in Source
foreach($folder in $Desfolders)
{
    $DesFilePath = $Destination -replace "\\","\\" -replace "\:","\:"
    $SrcFolder = $folder.Fullname -replace $DesFilePath,$Source
    if($srcFolder -ne "")
    {
        if(!(test-path $SrcFolder))
        {
            Write-Host "Folder $SrcFolder Missing. Creating it!"
            new-Item $SrcFolder -type Directory | out-Null
        }
    }
}

# Checking for Files that are in the Source, but not in Destination
foreach($entry in $SrcFiles)
{
    $SrcFullname = $entry.fullname
    $SrcName = $entry.Name
    $SrcFilePath = $Source -replace "\\","\\" -replace "\:","\:"
    $DesFile = $SrcFullname -replace $SrcFilePath,$Destination
    if(test-Path $Desfile)
    {
        $SrcMD5 = Get-FileMD5 $SrcFullname
        $DesMD5 = Get-FileMD5 $DesFile
        If($srcMD5 -ne $desMD5)
        {
            Write-Host "The Files MD5’s are Different… Checking Write Dates"
            Write-Host $SrcMD5
            Write-Host $DesMD5
            Copy-LatestFile $SrcFullname $DesFile
        }
    }
    else
    {
        Write-Host "$Desfile Missing… Copying from $SrcFullname"
        copy-Item -path $SrcFullName -dest $DesFile -force
    }
}

# Checking for Files that are in the Destinatino, but not in Source
foreach($entry in $DesFiles)
{
    $DesFullname = $entry.fullname
    $DesName = $entry.Name
    $DesFilePath = $Destination -replace "\\","\\" -replace "\:","\:"
    $SrcFile = $DesFullname -replace $DesFilePath,$Source
    if($SrcFile -ne "")
    {
        if(!(test-Path $SrcFile))
        {
            Write-Host "$SrcFile Missing… Copying from $DesFullname"
            copy-Item -path $DesFullname -dest $SrcFile -force
        }
    }
}

Write-Host

13 Responses to “Sync Folder Script.”

  1. on 06 Sep 2007 at 2:45 amGreg M

    Why not just use Robocopy with the appropriate parameters?

  2. on 06 Sep 2007 at 9:56 amBrandon

    Perfectly Valid Comment and one I asked myself in the beginning and I did it for three reasons.
    1) To see how hard it would be
    and More importantly. I learn a lot writing this script.
    2) I found value in the ability to customize the functionality at will.
    3) To show the power of Powershell.

    One of my main goals of Powershell is negate the need to EVER use external exe.

  3. on 06 Sep 2007 at 11:45 amGreg M

    You know, as soon as I wrote that I knew the answer. Valid pursuits all and a great effort. Thanks for keeping us up on the TMTOWTDI aspects of Posh.

    Cheers

  4. on 30 Nov 2007 at 4:28 pmSloot

    through experimentation, I have determined that -whatif can be passed to a cmdlet as follows
    Copy-Item -path $File1 -dest $File2 -force -whatif:$whatif

    the following syntax also works

    Copy-Item -path $File1 -dest $File2 -force -whatif $whatif

    but I don’t like to read that as much.

    Using this technique removes the need for your if ($whatif) blocks.

  5. on 30 Nov 2007 at 4:43 pmBrandon

    Great catch!

    I have used that before, but when I post on the blog I try to be super clear.

    Perhaps I will do a blog entry on this 🙂

  6. on 12 Jan 2010 at 12:28 pmJeffrey

    I’m running into two problems. The first is that the thing seems to be fairly CPU intensive for large numbers of files (over 20k). That isn’t as big of a deal.

    The other problem is that the script seems to have a problem when the filename includes either “[” or “]” It seems to think those files haven’t been copied even if they already have been. Thanks for your post!

  7. on 14 Jan 2010 at 6:33 amtshell

    Sorry about the CPU thing… it calculates the MD5 of every file so it is pretty heavy weight. As for the special character problem… I will have to take a look at that. Thanks for pointing it out.

  8. on 02 Mar 2010 at 10:54 amlifer

    I notice that the script doesn’t like file names with the ‘[‘ and ‘]’ characters. Neither do I though.

    I haven’t really looked yet but I assume it might be an issue with copy-item. The script output shows detecting a file with these characters as missing, but it will never get copied.

  9. on 02 Mar 2010 at 10:54 amlifer

    I notice that the script doesn’t like file names with the ‘[‘ and ‘]’ characters. Neither do I though.

    I haven’t really looked yet but I assume it might be an issue with copy-item. The script output shows detecting a file with these characters as missing from source\dest, but it will never get copied.

  10. on 16 Nov 2012 at 3:52 amdrmiru

    Hi Brandon

    Thx for sharing, very useful, as I’m still waiting on Appsense to extend their crappy folder mirror native action.

    I’ve extended the script by the parameter $excludefiletypes which allows exclusion for a certain file type eg.

    MirrorFolder.ps1 -Source C:\test1 -Destination C:\temp\ -excludefiletypes lnk
    Unfortunately the -exclude parameter of copy-item does not allow multiple strings.

    ###############################3
    Param($Source,$Destination,$excludefiletypes)
    function Get-FileMD5 {
    Param([string]$file)
    $mode = [System.IO.FileMode](“open”)
    $access = [System.IO.FileAccess](“Read”)
    $md5 = New-Object System.Security.Cryptography.MD5CryptoServiceProvider
    $fs = New-Object System.IO.FileStream($file,$mode,$access)
    $Hash = $md5.ComputeHash($fs)
    $fs.Close()
    [string]$Hash = $Hash
    Return $Hash
    }
    function Copy-LatestFile{
    Param($File1,$File2,[switch]$whatif)
    $File1Date = get-Item $File1 | foreach-Object{$_.LastWriteTimeUTC}
    $File2Date = get-Item $File2 | foreach-Object{$_.LastWriteTimeUTC}
    if($File1Date -gt $File2Date)
    {
    Write-Host “$File1 is Newer… Copying…”
    if($whatif){Copy-Item -path $File1 -dest $File2 -force -whatif}
    else{Copy-Item -path $File1 -dest $File2 -force}
    }
    else
    {
    Write-Host “$File2 is Newer… Copying…”
    if($whatif){Copy-Item -path $File2 -dest $File1 -force -whatif}
    else{Copy-Item -path $File2 -dest $File1 -force}
    }
    Write-Host
    }

    if(!(test-Path $Destination))
    {
    New-Item $Destination -type Directory -force | out-Null
    }

    # Getting Files/Folders from Source and Destination
    $SrcEntries = Get-ChildItem $Source -Recurse -Exclude *.$excludefiletypes
    $DesEntries = Get-ChildItem $Destination -Recurse *.$excludefiletypes

    # Parsing the folders and Files from Collections
    $Srcfolders = $SrcEntries | Where-Object{$_.PSIsContainer}
    $SrcFiles = $SrcEntries | Where-Object{!$_.PSIsContainer}
    $Desfolders = $DesEntries | Where-Object{$_.PSIsContainer}
    $DesFiles = $DesEntries | Where-Object{!$_.PSIsContainer}

    # Checking for Folders that are in Source, but not in Destination
    foreach($folder in $Srcfolders)
    {
    $SrcFolderPath = $source -replace “\\”,”\\” -replace “\:”,”\:”
    $DesFolder = $folder.Fullname -replace $SrcFolderPath,$Destination
    if($DesFolder -ne “”){
    if(!(test-path $DesFolder))
    {
    Write-Host “Folder $DesFolder Missing. Creating it!”
    new-Item $DesFolder -type Directory | out-Null
    }
    }
    }

    # Checking for Folders that are in Destinatino, but not in Source
    foreach($folder in $Desfolders)
    {
    $DesFilePath = $Destination -replace “\\”,”\\” -replace “\:”,”\:”
    $SrcFolder = $folder.Fullname -replace $DesFilePath,$Source
    if($srcFolder -ne “”)
    {
    if(!(test-path $SrcFolder))
    {
    Write-Host “Folder $SrcFolder Missing. Creating it!”
    new-Item $SrcFolder -type Directory | out-Null
    }
    }
    }

    # Checking for Files that are in the Source, but not in Destination
    foreach($entry in $SrcFiles)
    {
    $SrcFullname = $entry.fullname
    $SrcName = $entry.Name
    $SrcFilePath = $Source -replace “\\”,”\\” -replace “\:”,”\:”
    $DesFile = $SrcFullname -replace $SrcFilePath,$Destination
    if(test-Path $Desfile)
    {
    $SrcMD5 = Get-FileMD5 $SrcFullname
    $DesMD5 = Get-FileMD5 $DesFile
    If($srcMD5 -ne $desMD5)
    {
    Write-Host “The Files MD5′s are Different… Checking Write Dates”
    Write-Host $SrcMD5
    Write-Host $DesMD5
    Copy-LatestFile $SrcFullname $DesFile
    }
    }
    else
    {
    Write-Host “$Desfile Missing… Copying from $SrcFullname”
    copy-Item -path $SrcFullName -dest $DesFile -force
    }
    }

    # Checking for Files that are in the Destinatino, but not in Source
    foreach($entry in $DesFiles)
    {
    $DesFullname = $entry.fullname
    $DesName = $entry.Name
    $DesFilePath = $Destination -replace “\\”,”\\” -replace “\:”,”\:”
    $SrcFile = $DesFullname -replace $DesFilePath,$Source
    if($SrcFile -ne “”)
    {
    if(!(test-Path $SrcFile))
    {
    Write-Host “$SrcFile Missing… Copying from $DesFullname”
    copy-Item -path $DesFullname -dest $SrcFile -force
    }
    }
    }

  11. on 20 May 2013 at 4:58 pmcurtmcgirt

    six years later, anything you’d do to update/improve this script? I was just starting out writing one of my own, considering the following:

    1. surface RT does not have a skydrive sync app for offline viewing of files on the desktop.
    2. surface RT is capable of mapping a drive to skydrive using a specially formatted URL.
    3. surface RT does have powershell.

    on my windows 7 and full windows 8 machines, I have redirected “My Documents” and IE Favorites folders to skydrive, using the skydrive desktop app. using this script I could theoretically do pretty much the same thing with Surface RT, if I ran it as a scheduled task every X minutes and maybe we could speed it up a little bit, and/or add some logic to exclude certain subfolders or file extensions. what if it only collected files with a timestamp newer than the last time it ran?

  12. on 22 Aug 2013 at 11:23 amIngmar Verheij

    Hi Brandon,

    Thanks for this simple, yet effective, script to synchronize the content of two folders.
    I’ve updated your script with an include filter, in case someone is interested:

    Replace
    Param($Source,$Destination)
    with

    Param(
    [Parameter(Mandatory=$True)]
    [string]$Source,

    [Parameter(Mandatory=$True)]
    [string]$Destination,

    [string]$Filter = “”
    )

    and
    $SrcEntries = Get-ChildItem $Source -Recurse
    $DesEntries = Get-ChildItem $Destination -Recurse
    with
    $SrcEntries = Get-ChildItem $Source -Filter $Filter -Recurse
    $DesEntries = Get-ChildItem $Destination -Filter $Filter -Recurse

    Cheers,

    Ingmar Verheij
    http://www.ingmarverheij.com/

  13. on 01 Jul 2014 at 9:02 amJivey

    Remove these lines
    $DesFilePath = $Destination -replace “\\”,”\\” -replace “\:”,”\:”
    $SrcFilePath = $Source -replace “\\”,”\\” -replace “\:”,”\:”
    $DesFilePath = $Destination -replace “\\”,”\\” -replace “\:”,”\:”
    $SrcFolderPath = $source -replace “\\”,”\\” -replace “\:”,”\:”

    Replace
    $DesFolder = $folder.Fullname -replace $SrcFolderPath,$Destination with
    $DesFolder = $folder.Fullname -Replace ([regex]::Escape(“$Source”),”$Destination”)

    $SrcFolder = $folder.Fullname -replace $DesFilePath,$Source with
    $SrcFolder = $folder.Fullname -Replace ([regex]::Escape(“$Destination”),”$Source”)

    $SrcFile = $DesFullname -replace $DesFilePath,$Source with
    $SrcFile = $DesFullname -Replace ([regex]::Escape(“$Destination”),”$Source”)

    $DesFile = $SrcFullname -replace $SrcFilePath,$Destination with
    $DesFile = $SrcFullname -Replace ([regex]::Escape(“$Source”),”$Destination”)

Trackback this post | Feed on Comments to this post

Leave a Reply

You must be logged in to post a comment.