Rendering PDF via Angular $http instead of XMLHttpRequest

So I have spent weeks researching how to get a pdf from the server via ajax and then show it to the client. The reason I need to do this is a user can't just go to the pdf directly on the server, because they won't provide a token to pass security, but with javascript I can. Using the following code I am able to get what I am expecting:

 var xhr = new XMLHttpRequest();
 xhr.open('GET', url, true);
 xhr.responseType = 'arraybuffer';
 xhr.setRequestHeader("token", token.value);
 xhr.onload = function (e) {
    if (this.status == 200) {
       var file = new Blob([this.response], { type: 'application/pdf' });
       var fileURL = URL.createObjectURL(file);
       window.open(fileURL, '_blank');
    }
 };
 xhr.send();

However when I try and do this exact thing in angular like this (I'm doing half of this stuff just so the headers match in both):

var config = {
    url: url,
    method: 'GET',
    responseType: 'arraybuffer',
    headers: {
        'Accept':'*/*',
        "token": token.value
    }
}

$http(config)
    .success(function (response) {
        var file = new Blob([response], { type: 'application/pdf' });
        var fileURL = URL.createObjectURL(file);
        window.open(fileURL, '_blank')
    });

Here is what the headers look like in both:

GET http://example.com HTTP/1.1
Host: example.com
Connection: keep-alive
Accept: */*
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36
token: test
Referer: http://example.com
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8

The only difference is the xmlhttprequest version has Authorization: Negotiate header, but I can render both of them in fiddler from the responses from the server. This tells me that 'response' from $http isn't the same as this.response in xmlhttprequest. Can someone please tell me what is different and how I can fix it?

I thought for a long time that it was a charset problem and I have even base64 the bytes before sending it over the wire and then used atob() to switch it back, but that didn't seem to make it any better. It seems the $http version takes the response and converts the bytes to utf8 even if they aren't...my bytes are I think are windows-1252/latin1/ios-8895-1

The reason I say this is because I saw this in a hex editor:

btye 85 (on server) switched to two bytes of C2 85 (after $http code) http://www.fileformat.info/info/unicode/char/85/charset_support.htm

btye c5 (on server) switched to two bytes of c3 85 (after $http code) http://www.fileformat.info/info/unicode/char/c5/charset_support.htm

I have even tried to encode/decode if various combinations with no avail. I really think the pdf is getting corrupted some how when it's going through angular, but I don't know how.

UPDATE 1

On the server side the code looks like this:

public class LabelController : Controller
{
  //using Microsoft.AspNet.Mvc
  [HttpGet, Route("api/labels/box")]
  public IActionResult BoxLabels(Guid requestId)
  {
    var data = this.getPDF("http://foobar.com/example.pdf");
    Response.Headers.Add("Content-Length", data.Length.ToString());
    Response.Headers.Add("Content-Disposition", "inline; filename=example.pdf");
    return new FileContentResult(data, "application/pdf");
  }

  //using restsharp
  private bytes[] getPDF(url)
  {
    var client = new RestClient
    {
      BaseUrl = new Uri(url),
      Authenticator = new NtlmAuthenticator()
    };

    var request = new RestRequest
    {
      Method = Method.POST,
      Credentials = CredentialCache.DefaultNetworkCredentials,
      RequestFormat = DataFormat.Xml
    };

    var response = client.Execute(request);
    return response.RawBytes;
  }
}

Here is sort of what it looks like: http://plnkr.co/edit/R3FiesPk7Os22SMsTTfE without the "No 'Access-Control-Allow-Origin'" error

UPDATE 2

Here is what the network traffic tells me:

Here is what the pdf looks like when using $http: