1

I'm working on an application that fetches repository information for a given GitHub username.

The GitHub API has a limit of 100 items per request; so, if a given account has, say 110 repos, I'd need to make a second call to grab the remaining ten (you may see where this is going). Essentially, I want my code to be able to be able to determine how many pages of repos are available for a given account, and make one or as many calls as it needs in order to get all the data.

I've written the code that determines whether to make one or multiple calls. It's in the block that makes more than one call that I'm running into a problem.

I'm working in Node.js, and the GitHub API client for Node provides some really handy methods to achieve what I'm trying to achieve.

github.repos.getForUser({
      username: 'nodejs',
      per_page: 100
    }).then(result => {
      if(!result.meta.link) {
          // if there's NO link header, it means
          // that there are NO pages to traverse.
          // that is: it means we've retrieved all
          // available repositories.
          console.log('no link header');
          console.log('we have retrieved all available repositories');
      } else {
          // if there IS a link header, we need to
          // capture the number of the last page; this
          // will allow us tp determine how many pages
          // we need to traverse in order to capture 
          // all available repositories
          console.log('link header found');
          var link = result.meta;
          var num_of_pages = github.hasLastPage(link).match(/&page=(\d{1,})/)[1];
          console.log(`At a rate of 100 repos per_page, there are ${num_of_pages} pages to traverse`);
          var repos = result.data,
              hasNextPage = github.hasNextPage(link);
          while(hasNextPage !== undefined) {
            github.getNextPage(link, function(err, res) {
              console.log(repos.length, res.data.length);
              repos = repos.concat(res.data);
              hasNextPage = github.hasNextPage(res.meta);
              console.log(repos.length, res.data.length, hasNextPage);
            });
          } //end while

        } //end else
    });

github.hasNextPage(link) takes in the content of the Link header (or the response object) returns a string, which may look something like this

https://api.github.com/user/9950313/repos?per_page=100&access_token={MY_TOKEN}&page=2

It will basically extract this url from the Link header, which github.getNextPage(link, [optional header], callback) uses to fetch the data from the next page, if there is a next page. If there's not, github.hasNextPage() will return undefined.

When I run the code inside the while loop without the while loop, it works. But when I run it inside the while loop, the program hangs. I've tried a few test to check whether the while loop is triggering or not; it is. It seems instead that for some reason, github.getNextPage(link, [optional header], callback) is not being triggered inside the while loop. I've been looking at this for a few hours now and can't seem to spot the problem.

Can anyone help please?

UPDATE I tried letting the code run to see what happened and after some time, I got this

<--- Last few GCs --->

[9094:0x4044a30]   111879 ms: Mark-sweep 1402.2 (1490.1) -> 1402.2 (1462.1) MB, 953.1 / 3.3 ms  (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 953 ms) last resort GC in old space requested
[9094:0x4044a30]   112921 ms: Mark-sweep 1402.2 (1462.1) -> 1402.2 (1462.1) MB, 1042.2 / 4.0 ms  last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x22e243225ee1 <JSObject>
    1: new WritableState [_stream_writable.js:~41] [pc=0x2445e09697e8](this=0x299f6bcfe731 <WritableState map = 0x9510adbf1e9>,options=0x299f6bcb8bd9 <Object map = 0x330625c96891>,stream=0x299f6bcb8b01 <TLSSocket map = 0x330625c97079>)
    3: Writable [_stream_writable.js:~194] [pc=0x2445e0974360](this=0x299f6bcb8b01 <TLSSocket map = 0x330625c97079>,options=0x299f6bcb8bd9 <Object map = 0x330625c...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [node]
 2: 0x121a2cc [node]
 3: v8::Utils::ReportOOMFailure(char const*, bool) [node]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]
 5: v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [node]
 6: v8::internal::Runtime_AllocateInTargetSpace(int, v8::internal::Object**, v8::internal::Isolate*) [node]
 7: 0x2445e078463d
Aborted (core dumped)

This seems to suggest that the loop is in fact running, but that the code inside of it isn't... what could be causing this?

UPDATE

Someone pointed out that the link variable wasn't getting updated in the loop... I corrected this so that it would update, but it still doesn't work...! It seems that the code inside getNextPage's callback isn't getting run at all.

while (hasNextPage !== undefined) {
        github.getNextPage(link, function(err, res) {
          console.log(repos.length, res.data.length);
          repos = repos.concat(res.data);
          link = res.meta; hasNextPage = github.hasNextPage(res.meta);
          console.log(repos.length, res.data.length, hasNextPage, link);
        });
      } //end while

But if I run it like this, it works

github.getNextPage(link, function(err, res) {
      console.log(repos.length, res.data.length);
      repos = repos.concat(res.data);
      link = res.meta; hasNextPage = github.hasNextPage(res.meta);
      console.log(repos.length, res.data.length, hasNextPage, link);
    });
9
  • What is the value / type of hasNextPage right before the loop is called? Commented Feb 13, 2018 at 0:02
  • https://api.github.com/user/9950313/repos?per_page=100&access_token={MY_TOKEN}&page=2 Commented Feb 13, 2018 at 0:10
  • 1
    It seems like variable “link” is not getting updated in the while loop. Commented Feb 13, 2018 at 0:35
  • 1
    Working with async API calls should not be handled using a while loop, as it will not wait for a callback. I recommend using async.each() for making async call in a loop. Give it a try if possible. Commented Feb 13, 2018 at 0:45
  • 1
    It is used to execute async calls inside a loop. Loop will not iterare untill you trigger a callback. Please look at this example.. stackoverflow.com/questions/46798670/… Commented Feb 13, 2018 at 0:55

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.