Node.js request.js HPE_INVALID_HEADER_TOKEN

Question

I got desperate about one problem and I need some help... I'm using node.js to crawl a list of websites, some of them gives me this error, for example: http://www.fz-juelich.de/portal/DE/Home/home_node.html, Parse Error, HPE_INVALID_HEADER_TOKEN

request.get({
    url: uri,
    timeout: timeout,
    headers: {
        referer: domain
    }
}, (error, response, body) => {
    if (error)
        console.log(error);
    console.log(body);
});

though, curl -i --raw http://www.fz-juelich.de/portal/DE/Home/home_node.html works just perfect

HTTP/1.1 404 Not Found
Server: Apache-Coyote/1.1
Cache-Control: no-cache
JSESSIONID=E594677A6CCA13BE0338E1D00A729C34; Path=/cae:
Content-Type: text/html;charset=utf-8
Content-Language: de
Set-Cookie: JSESSIONID=E594677A6CCA13BE0338E1D00A729C34; Path=/
Content-Length: 19677

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd" >

Also I'm able to see this website in my chrome browser

Any ideas in which side should I dig to get rid of this errors?

No ideas?... I start thinking about using some 3rd party C++/C extension — let4be
– let4be, Commented Oct 15, 2014 at 17:25

Yehr59 · Accepted Answer · 2020-07-16 15:22:50Z

1

I use quotes in properties and that resolve for me :

request.post(url,{
    headers: {
      'Authorization': 'Basic onEnAGrosEncodedBase64',
      'Content-Type': 'application/x-www-form-urlencoded'
    },
    form: {
      'grant_type': 'client_credentials'
    }
 })

I hope that can help someone ;)

answered Jul 16, 2020 at 15:22

Yehr59

232 silver badges8 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

let4be · Accepted Answer · 2015-09-12 12:19:50Z

0

I the end of this journey I no longer use node.js for crawling and parsing

Go lang crawler fits much better here, more flixibility in http library and easier to write really concurrent stuff

answered Sep 12, 2015 at 12:19

let4be

1,08112 silver badges32 bronze badges

2 Comments

Mike Over a year ago

Yes, Node is very picky about HTTP Headers - I had the identical issue mentioned above. The source website was sending a Link header with HTML inside which was crashing Node. To get around it I wrote a separate script which cURLed the data I needed then called that from my Node scripts

Marc Over a year ago

I also struggled with this problem but insisted on using Node. Fortunately I found a lib node-libcurl which does a fantastic job and is also backed by Insomnia (free alternative to Postman) so it will be around for a while :)

Collectives™ on Stack Overflow

Node.js request.js HPE_INVALID_HEADER_TOKEN

2 Answers 2

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related