Lex匹配unicode字符

想用lex&yacc写一个json的解析, 而json的string类型是包含unicode的, 词法解析工具Lex是不直接支持unicode字符匹配的, 那如果要想匹配unicode字符应该怎么办呢, 在stack overflow上看到一个很好的解答: http://stackoverflow.com/questions/9611682/flexlexer-support-for-unicode.

基本思想就是unicode字符写一个匹配模式,

ASC     [\x00-\x7f]
ASCN    [\x00-\t\v-\x7f]
U       [\x80-\xbf]
U2      [\xc2-\xdf]
U3      [\xe0-\xef]
U4      [\xf0-\xf4]

UANY    {ASC}|{U2}{U}|{U3}{U}{U}|{U4}{U}{U}{U}
UANYN   {ASCN}|{U2}{U}|{U3}{U}{U}|{U4}{U}{U}{U} 
UONLY   {U2}{U}|{U3}{U}{U}|{U4}{U}{U}{U}

上面匹配模式的意义如下:

UANY: 匹配unicode和ascii字符
UANYN: 与UANY类似, 只是不匹配换行符
UONLY: 只匹配unicode字符, 不匹配ascii字符

DISCLAIMER: Note that the scanner’s rules use a function called
utf8_dup_from to convert the yytext to wide character strings
containing Unicode codepoints. That function is robust; it detects
problems like overlong sequences and invalid bytes and properly
handles them. I.e. this program is not relying on these lex rules to
do the validation and conversion, just to do the basic lexical
recognition. These rules will recognize an overlong form (like an
ASCII code encoded using several bytes) as valid syntax, but the
conversion function will treat them properly. In any case, I don’t
expect UTF-8 related security issues in the program source code, since
you have to trust source code to be running it anyway (but data
handled by the program may not be trusted!) If you’re writing a
scanner for untrusted UTF-8 data, take care!