There's a few errors here.
$validUsers = Import-Csv 'C:\Users\Bang\Desktop\Bulk.csv' | Select-Object -Expand Number
Here you're expanding the Number column. That means $validUsers will be an array of strings, not an array of objects with a property named Number. Compare Get-ChildItem C:\Windows | Select-Object -Property Name -First 5 to Get-ChildItem C:\Windows | Select-Object -ExpandProperty Name -First 5.
$Users = Get-ADGroupMember 'Test' | %{Get-ADUser $_.SamAccountName | select UserPrincipalName}
Here, you didn't specify -ExpandProperty. That means the objects assigned to $Users will be an array of objects with a property named UserPrincipalName.
foreach ($Member in $Users)
{
if ($Member -match $validUsers.Number){
Write-Output $Member}
}
Here you're doing all kinds of stuff wrong. First, $Member is an element of $Users, which means it has the property UserPrincipalName. Second, $validUsers doesn't have any properties at all. It's just an array of unnamed strings. Third, -match is for matching a string with a regular expression. If you want to know if an element is in an array, you need to use the -in operator.
Try this:
$GroupName = 'Test'
$validUsers = Import-Csv 'C:\Users\Bang\Desktop\Bulk.csv' |
Select-Object -ExpandProperty Number -Unique
$CurrentUsers = Get-ADGroupMember $GroupName |
Get-ADUser |
Select-Object -ExpandProperty UserPrincipalName -Unique
# Add valid users not already in the group
$validUsers | Where-Object { $_ -notin $CurrentUsers } |
ForEach-Object { Get-AdUser -Filter "UserPrincipalName -eq '$_'" } |
Add-ADPrincipalGroupMembership -MemberOf $GroupName -WhatIf
# Remove invalid users currently in the group
$CurrentUsers | Where-Object { $_ -notin $validUsers } |
ForEach-Object { Get-AdUser -Filter "UserPrincipalName -eq '$_'" } |
Remove-ADPrincipalGroupMembership -MemberOf $GroupName -WhatIf
The -WhatIf parameters on Add-ADPrincipalGroupMembership and Remove-ADPrincipalGroupMembership will cause those commands to just output what they're going to do instead of actually taking any actions. To actually run the script and update your group, you'll want to remove those parameters.
Edit: Unfortunately, most AD commands don't accept the user principal name as pipeline input. They wants distinguished name, SAM account name, SID, or GUID. The ForEach-Object is added in there to do the lookup again, which is a bit unfortunate.
This script could definitely be improved, but it should work for you now, based on what I think you're trying to do.