How to check duplicate multiple file using powershell?

Question

I want to check duplicate file.If the condition of the file like this, it means duplicate. The same name but different extension.

AAA18WWQ6BT602.PRO
AAA18WWQ6BT602.XML

I can figure out this case with my script. But I have problem if I have this more than 1 .XML file like this

AAA18WWQ6BT602.PRO
AAA18WWQ6BT602.XML
AAA18WWQ6BT601.XML
AAA18WWQ6BT604.XML

This case, it will not detect that file AAA18WWQ6BT602.PRO and AAA18WWQ6BT602.XML duplicated. Anyone can help me please. Thanks

$duplicate = @()
    @(Get-ChildItem "$Flag_Path\*.xml") | ForEach-Object { $duplicate += $_.basename }

    if(Test-Path -Path "$Flag_Path\*$duplicate*" -Exclude *.xml)
    {
        Get-ChildItem -Path "$Flag_Path\*$duplicate*" -Include *.xml | Out-File $Flag_Path\Flag_Duplicate
        Write-Host "Flag duplicated, continue for Error_Monitoring" 
        pause
        Error_Monitoring
    }
    else{
    Write-Host "Flag does not duplicate, continue the process"
}

Theo · Accepted Answer · 2019-10-31 11:30:31Z

3

The -Include parameter only works if the path on Get-ChildItem ends in \* OR if the -Recurse switch is used.

The following should do what you want:

$flagFolder = 'D:\*'
$dupeReport = 'D:\Flag_Duplicate.txt'
$duplicates = Get-ChildItem -Path $flagFolder -File -Include '*.xml', '*.pro' | 
              Group-Object -Property BaseName | Where-Object { $_.Count -gt 1 }

if ($duplicates) {
    # output the duplicate XML to Flag_Duplicate.txt
    $duplicates.Group | Where-Object {$_.Extension -eq '.xml' } | ForEach-Object {
        $_.FullName | Out-File -FilePath $dupeReport -Append
    }
    # do the rest of your code
    Write-Host "Flag duplicated, continue for Error_Monitoring" 
    Error_Monitoring
}
else {
    Write-Host "Flag does not duplicate, continue the process"
}

answered Oct 31, 2019 at 11:30

Theo

61.9k8 gold badges28 silver badges48 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

tukan · Accepted Answer · 2019-11-01 08:39:09Z

1

Your script does not iterate correctly. You need to have an iteration to check. The Test-Path logic looks mixed up to me. I tried to keep as much of your code as possible.

This script checks for a any xml basename filename against any suffix duplicate (not only pro):

$Flag_Path = "C:\dir_to_be_checked"
$xmlFilesArray = @()
$allFilesExceptXml = @() # all files excluding xml files

# Get all the xml files
Get-ChildItem -Path $Flag_Path -Include "*.xml" | ForEach-Object { $xmlFilesArray += $_.basename }
# Get all files from the directory the xml files
Get-ChildItem -Path $Flag_Path -Exclude "*.xml" | ForEach-Object { $allFilesExceptXml += $_.basename }

# Iterate over list of files names without suffix
ForEach ($xmlFile in $xmlFilesArray) { 
    ForEach ($fileToCheck in $allFilesExceptXml) {
        If ($xmlFile -eq $fileToCheck) {
            # logging the duplicate file (specifying utf8 or the output would be UTF-16)
            Write-Output "$Flag_Path\$xmlFile.xml" | Out-File -Append -Encoding utf8 $Flag_Path\Flag_Duplicate
            Write-Host "Flag duplicated, continue with duplicate search" 
            # pause
            Write-Host "Press any key to continue ..."
            $x = $host.UI.RawUI.ReadKey("NoEcho,IncludeKeyDown")
            Error_Monitoring
        } Else {
            Write-Host "Flag is not duplicated.  Continue with the search."
        }
    }
}

edited Nov 1, 2019 at 8:39

answered Oct 31, 2019 at 11:49

tukan

17.5k1 gold badge25 silver badges51 bronze badges

2 Comments

tukan Over a year ago

@SBR you are welcome. I have noticed now that it could have been done more efficiently. The Get-ChildItem could have been executed only once loading all files into the array. Next you would filter in/out the xml files. That would save you one filesystem iteration which tends to be slow.

Cheries Over a year ago

Thank you @tukan. If you don't mind, can help me to give idea of my other question. Thanks a lot. stackoverflow.com/q/58690113/11076819

Collectives™ on Stack Overflow

How to check duplicate multiple file using powershell?

2 Answers 2

Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related