Is there a better/simpler way?
Not generically. How you handle API errors depends on the calling conventions of the API and how you will incorporate the results into your application. Some APIs return Unauthorized when the Bearer token has expired, signaling you to use your refresh token to get a new access token. In some endpoints Unauthorized might just mean the user doesn't have the appropriate role or permissions.
The answer is "yes, you need to handle unauthorized responses" and you'll need to do it in a way that is appropriate for the API you are calling in the context of the place you are integrating it into your app. Sometimes that means you can use a wrapper which will perform some generic action on unauthorized or other errors, but the only one that can make that determination is the dev that is familiar enough with both while integrating with the API and writing the access code for their app.
Some of that advice changes if you are trying to handle transient errors, but you specifically asked about unauthorized responses.