Since you're using @angular/material, you should bind to the color @Input of their component rather than just adding a css-class of your own.
That way you do not have to worry about css-priority and your code will not be fragile to changes in the @angular/material implementation of component color.
<button mat-icon-button
[color]="buttonColor"
(click)="buttonColor = 'warn'">
<mat-icon>remove_circle</mat-icon>
</button>
and in your component
buttonColor: ThemePalette = 'primary';
If your warn-color is not red, then consider changing your theme or making a new theme.