node-ssh user doesn't have access to ssh keys on server

Attempting to run the following for a deploy script in node

I'm using node-ssh to ssh onto a server and run a git pull, the user I'm using has a deploy key setup for this repo (it works if I run these commands manually on the server)

import { NodeSSH } from 'node-ssh'

await ssh.connect({
    host: process.env.SSH_HOST,
    username: process.env.SSH_USERNAME,
    password: process.env.SERVER_PASSWORD,
})

await ssh.execCommand(`git pull`, {
    onStdout: (out) => console.log(out.toString('utf8')),
    onStderr: (out) => {
      console.error(out.toString('utf8'))
      throw out.toString()
    },
    cwd: '/path-to-my-repo',
})

but the above errors with the following

Error [ERR_UNHANDLED_ERROR]: Unhandled error. ('[email protected]: Permission denied (publickey).\r\n')

The deploy key does have a passphrase, but ssh-agent has already been launched with that identity added. If I run git pull as this user on the server, I don't get prompted for the password.

Is there some configuration for node-ssh I'm missing that will give it access to this key?