Validate and Deploy only the changed artifacts with Synapse workspace deployment@2 task

Your idea that using the 'git diff' command to filter the changed files should be okay.

You can try to configure your pipeline with the following steps:

1. Add a script/PowerShell/Bash task before the Synapse workspace deployment@2 task to do the things below.

   • Call the REST API "Builds - Get Build Changes" to get all the commits associated with current pipeline run.
   • Use the 'git diff' command to list all the changed files from the commits.
   • Copy the changed files into another directory.

2. On the Synapse workspace deployment@2 task, you can set the configurations below.

   • Select validateDeploy as the value of operation (Operation Type) option.
   • Pass the path of the directory as the value of the ArtifactsFolder (Artifacts root folder) option.
   • Ensure the option DeleteArtifactsNotInTemplate (Delete Artifacts Not In Template) is false.

See below sample as reference.

variables:
  armConnection: myArmConnection
  resourceGroup: myResourceGroup
  synapseWorkspace: myWorkspace

steps:
- task: PowerShell@2
  displayName: 'Pick changed files'
  env:
    SYSTEM_ACCESSTOKEN: $(System.AccessToken)
  inputs:
    pwsh: true
    targetType: inline
    script: |
      Write-Host "Getting modified files..."
      $prefix_url = "$(System.CollectionUri)"
      $project = "$(System.TeamProject)"
      $buildId = $(Build.BuildId)
      $url = "${prefix_url}${project}/_apis/build/builds/${buildId}/changes?includeSourceChange=true&api-version=7.0"
      $headers = @{Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"}
      $response=Invoke-RestMethod -Uri $url -Headers $headers
      $changesCount = $response.count

      $modifiedFiles= git diff HEAD HEAD~$changesCount --name-only

      Write-Host "Copying modified files..."
      $srcDir = "$(Build.SourcesDirectory)"
      $destDir = "$(Build.ArtifactStagingDirectory)"

      foreach ($file in $modifiedFiles)
      {
        Write-Host "Copying file: $srcFile ---> $destFile"
        $srcFile = "$srcDir/$file"
        $destFile = "$destDir/$file"

        $parentDir = Split-Path -Parent $destFile
        if (-not (Test-Path $parentDir))
        {
          New-Item -Path $parentDir -ItemType Directory
        }

        Copy-Item "$srcFile" -Destination "$destFile"
      }

- task: Synapse workspace deployment@2
  inputs:
    operation: 'validateDeploy'
    ArtifactsFolder: '$(Build.ArtifactStagingDirectory)'
    azureSubscription: $(armConnection)
    ResourceGroupName: $(resourceGroup)
    TargetWorkspaceName: $(synapseWorkspace)
    DeleteArtifactsNotInTemplate: false