Passport-Azure-AD not validating access_token from MSAL Angular authentication

I have an Angular 10 app that I am using the MSAL-Angular package on to authenticate with Azure AD. It successfully authenticates and retrieves the access_token and id_token using the Authorization flow.

I also have an Express API backend that I am attaching the access_token to (using the Interceptor from Msal-Angular) onto the header of the request. I want to validate this token to verify that the Angular app has authenticated before allowing access to the api. To do this I am trying to use passport-azure-ad with the BearerStrategy. I believe I have it all setup correctly but it keeps telling me that the token can't be validated.

I figured out that if I use the id_token instead of access_token it can be verified; through the passport strategy and using jwt.io.

So the real question here is can I do something different in my config to validate the access_token that gets passed using Msal-Angular or should I just write my own Interceptor that sends the id_token instead of the access_token?

API Passport config code - All I am trying to get done here is to get the passport validation to pass then will fill out the callback function.

const options: IBearerStrategyOptionWithRequest = {
  identityMetadata:
    'https://login.microsoftonline.com/{tenet}.onmicrosoft.com/v2.0/.well-known/openid-configuration',
  clientID: '{my_client_id}',
  validateIssuer: true,
  loggingLevel: 'info',
  loggingNoPII: false,
  passReqToCallback: true,
};

const bearerStrategy = new BearerStrategy(options,
  function (req, token, done) {
    logger.debug(req);
    logger.debug(token);

    return done(null, { name: 'user name' }, token);
  });

passport.use(bearerStrategy);

Securing Route

function (req, res, next) {passport.authenticate('oauth-bearer', { session: false }),  listTasks);

Error given:

{
  "name":"AzureAD: Bearer Strategy",
  "hostname":"my_host",
  "pid":36344,
  "level":30,
  "msg":"authentication failed due to: invalid signature",
  "time":"2021-03-02T20:50:02.140Z",
  "v":0
}