Get integer number using regex in javascript

Maybe this will do it for you:

(?<!\d\.)\b(\d+)\\frac{([^{}]+(?:{(?:[^{}]+)}|))}{([^{}]+(?:{(?:[^{}]+)}|))}

It captures an integer expression before \fraq, unless it's preceded by a digit and a full stop.

(?<!\d\.) This ensures the number isn't preceded by a digit followed by a full stop, i.e.
          the integer part of a floating.
\b        Must be at the start of a number (to make sure we don't get a match with the end
          of a multi digit number).
(\d+)     Captures the integer number
\\frac    Matches the string "\fraq"

The rest is the same as you original expression.

See it here at regex101.

Edit

Since there obviously are people out there, however unbelievable, that still haven't moved to a real browser ;) - the answer has to change to:

It depends of the syntax of latex, whether you can do it or not.
(And since I don't know that, I shouldn't have said anything in the first place ;)

The problem is that, without look behinds, you can't do it without matching characters outside the expression your interested in. In your regexr-example you clearly show that you want to be able to match expression not only at the beginning of strings, but also in the middle of the. Thus we need to be able to tell that the part before our match isn't the integer part of a decimal number. Now, doing this with matching isn't a problem. E.g.

(?:^|[^\d.])(\d+)\\frac{...

like Wiktor suggested in comments, will match if the expression is on the start of the line (^), or is preceded by something that isn't a decimal point or a digit ([^\d.]). That should do it. (Here at regex101.)

Well, as pointed out earlier, it depends on the syntax of latex. If two expressions can be directly adjacent, without any operators or stuff between them, you can't (as far as I can tell) do it with JS regex. Consider 1\fraq{0}{1}2\fraq{2}{3} (which I have no idea if it's syntactically correct). The first expression 1\fraq{0}{1} is a piece of a cake. But after that has been matched, we'd need to match a character before the second expression to verify the it doesn't start with a decimal number, but since the first expression already ate the characters, we can't. Because the test (?:^|[^\d.]) to verify that our expression doesn't start with a decimal number, would match one of the characters that actually belongs to our expression (the 2 in 2\fraq{2}{3}), thus making the match fail, because the remaining part doesn't start with the digit needed to satisfy the rest of the regex (\d+)\\frac{....

If, however, an expression always starts the string tested, or is preceded by and operator, or such, then it should be possible using

(?:^|[^\d.])(\d+)\\frac{([^{}]+(?:{(?:[^{}]+)})?)}{([^{}]+(?:{(?:[^{}]+)})?)}

Here at regex101.

(Sorry for my rambling)