Lexer support for non-ascii literals

[lhaussknecht]

I'm consuming a JSON service wich returns umlauts (ä,ö,ü) in some filed names. Binding them leads to a lexer Error.

    Error: Lexer Error: Unexpected next character  at columns 4-4 [ä] in expression [Einsätze].
     at Error (<anonymous>)
     at throwError (http://code.angularjs.org/angular-1.0.1.js:5646:11)
     at lex (http://code.angularjs.org/angular-1.0.1.js:5610:9)
     at parser (http://code.angularjs.org/angular-1.0.1.js:5803:16)
     at http://code.angularjs.org/angular-1.0.1.js:6387:29
     at http://code.angularjs.org/angular-1.0.1.js:4767:27
     at addTextInterpolateDirective (http://code.angularjs.org/angular-1.0.1.js:4399:27)
     at collectDirectives (http://code.angularjs.org/angular-1.0.1.js:3901:11)
     at compileNodes (http://code.angularjs.org/angular-1.0.1.js:3791:21)
     at compileNodes (http://code.angularjs.org/angular-1.0.1.js:3799:14) 

The lexer currently only supports:

    function isIdent(ch) {
          return 'a' <= ch && ch <= 'z' ||
              'A' <= ch && ch <= 'Z' ||
              '_' == ch || ch == '$';
      }

It would be great if we could expand this function to accept a wider range of input. Maybe unicode.

I'll try to come up with a pull request.

[IgorMinar]

The non-asci characters are in templates or in data that you are binding the template with?

[lhaussknecht]

Both, I receive a JSON object and need to bind it in my template.

[dipold]

Please consider to include also the character '@' given the fact that serializers (like Gson) include class information when the object has polymorphism.
e.g:

 {"id": 1,"actor": {"@class": "person","id": 4,"name": "Jim"}}
 {"id": 2,"actor": {"@class": "organization","id": 5,"name": "Google Inc."}}

[gurdiga]

Here is the conversation around a similar issue in JShint: jshint/jshint#716. At least it’ll give a hint on the complexity and the approach used there.

There is also another simpler approach at the jsLi project: https://github.com/mazko/jsli/blob/master/src/isUnicodeLetter.js. While it’s not as robust as the JSHint’s approach (it doesn’t recognize characters with 5 hex digit codes) it may be the 20% of code that covers 80% of the needs. ;)

[lhaussknecht]

Great thanks, I came up with something simple like this to fix my issue:

129 <= ch.charCodeAt(0) && ch.charCodeAt(0) <= 496; //Unicode 1.0 Latin

This range includes all Latin Extended-A and Latin Extended-B characters defined in Unicode 1.0.

[kaililleby]

any new on this ? the isIdent function still only accepts asci chars..

[jonakerstrom]

BUMP! This is still an issue !

[jensaug]

Had to patch the isIdent property/function in angular.js file to get around this. In AngularJS 1.2.8 I did

  isIdent: function(ch) {
    return ('a' <= ch && ch <= 'z' ||
            'A' <= ch && ch <= 'Z' ||
            'À' <= ch && ch <= 'ÿ' || /* Extended characters */
            '_' === ch || ch === '$');
  },

No way to up-vote this issue in GitHub?

[jensaug]

I created a Pull Request for this issue - it got slightly more advanced that above solution... Pls check it out and give comments.

It doesn't solve everything - since isIdent(ch) is a sandboxing function not everything can be allowed, but this PR at least support latin letters with grave, accute, circumflex, tilde, diaeresis, ring above, cedilla. Including Nordic languages.

Perhaps someone with better insights can work out a better compromise between using valid JSON and how to "prevent exploits/ against the expression language"