Request: make parser locale independent

[Kyros]

I noticed that one of the latest bugfixes introduced locale-dependency in the parser. By that i mean that the parser does not work when used from an environment that uses "locales" (in particular the LC_NUMERIC locale) that is not the default "POSIX" / US locale.

This was what i discovered:

I am trying to parse a JSON that looks like this:

Code: Select all

{
    "mantHistoryInfo": [
        {
            "code": "AFC-1094-C0",
            "maxRate": 0.0,
            "minRate": 0.0,
            "phase": 0,
            "timeSlot": "2024-01-01T00:00",
            "value": 0.0
        },
        {
            "code": "AFC-1094-C0",
            "maxRate": 30.0,
            "minRate": 0.0,
            "phase": 0,
            "timeSlot": "2024-01-04T18:00",
            "value": 0.000459
        }
	]
}

At the first while parsing the second value "0.000459" YAJL was crashing. The first thing i did was to look if there were bugfixes because i was using a version of YAJL that was quite old. I found exactly what i was looking for: https://www.scottklement.com/forums/vie ... mber#p1614.

So i downloaded the latest version of YAJL and tried again. The parser was not crashing anymore but it was not working as intended because the YAJL_GET_NUMBER function was returning the value 4.000 instead of 0.000459.
I immediatly thought that this was strange, it seemed like the bugfix was not working but the user Jeron in the topic that i linked above said that for him the bugfix had resolved the issue.
So what i did was look in the source code and tried to figure out what was going on.

This is the piece of code that was causing problems:

Code: Select all

       if YAJL_IS_NUMBER(node);
          init_gconv();
          str = yajl_iconv_toLocal( gConv: nv.number.R );
          mem.yajl_iconv_string += 1;
          monitor;
            retval = %dec(%str(str):30:9);
          on-error 105;
            // Remove the RNX0105 (error converting numeric)
            QMHRCVPM( RCVM0100
                    : %size(RCVM0100)
                    : 'RCVM0100'
                    : '*'
                    : 0
                    : '*ESCAPE'
                    : *blanks
                    : 0
                    : '*REMOVE'
                    : x'0000000800000000');
            eval(h) retval = atof(str);
          endmon;
          yajl_iconv_string_free(str);
          mem.yajl_iconv_string -= 1;
       else;
          retval = 0;
       endif;
       return retval;

After a bit of debugging i discovered that the "atof" function was not working as expected and after a bit of research i found this: https://www.ibm.com/support/pages/examp ... es-locales

This explained everything. I am italian and my IBM i has italian locales. The "atof" function is locale-dependent, meaning that if it's used in an "italian" environment it must receive the string "0,000459" to work correctly.
This even explained why the other user had no more complaints, he is probably using a US locale where the decimal separator is the "dot".

To fix my bug i followed the IBM guide on how to "force" a specific locale when doing stuff that is locale-sensitive.

Even if my program is working now, i still think that this can be considered a "bug" of the parser. That is because the JSON grammar does not allow for ambiguity, the notation for numbers is such that only the "dot" che be used as a decimal separator. For this reason when the parser calls the atof function it can safely assume that the "locale" in which the number is written is the POSIX / US locale, and for this reason it should enforce that locale just how the IBM guide explains to do.

Otherwise the parser does not work "out of the box" for people that use different locales, but only for those who use the point as the decimal separator.

[Scott Klement]

That's a good find!

Please download the current one from https://www.scottklement.com/yajl/ and see if it fixes the problem for you.

Thanks!

[Kyros]

I compiled the latest version and commented out the part where i forced the locale but it seems that the problem persists.

Here is a portion of my code:

Code: Select all

       //  oldLocalePtr = setlocale (LC_ALL : *NULL);
       //  oldLocaleSave = %str(oldLocalePtr);
       //  setlocale (LC_ALL: LOCALETYPE_POSIX);

         // Input: RESPATH
         docNode = yajl_stmf_load_tree(resPath:errMsg);
         if errMsg <> ' ';
           LEAVESR;
         ENDIF;
         list = YAJL_object_find(docNode: 'mantHistoryInfo');
         j = 0;
         dow YAJL_ARRAY_LOOP( list: j: dev_values );

           element = YAJL_OBJECT_FIND(dev_values:'timeSlot');
           dev_tim = yajl_get_string(element);

           isoString = %replace('-':dev_tim:11:1);
           isoString = %trim(%replace('.':isoString:14:1)) + '.00.000000';


           element = YAJL_OBJECT_FIND(dev_values:'value');
           dev_val = yajl_get_number(element);                                       

I am not using YAJL-INTO but the "standard" version of the parser because i need to compile YAJL (and all of my programs) for V6R1M0.
Could it be that the fix only works if i use YAJL-INTO? I tried having a quick look at what changed in the source code and saw some changes in YAJLINTO but no changes in YAJLR4, no idea if this is the reason though.

Thanks for the support!

[Scott Klement]

The changes I made aren't related to DATA-INTO... I didn't even think about DATA-INTO. They are only related to the yajl_get_number and yajl_get_bignumber subprocedures.

You said there weren't any changes to YAJLR4? That shouldn't be the case. I added a new (internal) subprocedure called getDouble() that does the setlocale, etc, logic. Then changed YAJL_get_number and YAJL_get_bignumber() to call it... So there are definitely changes.

I tried re-downloading the YAJL.zip from my website and restoring it, and yes, those changes are still there.

Here's what bugs me, though... you say you're making it work on V6R1. Does that mean you downloaded the "IBM i 6.1" version of YAJL from my site? That version is not supported, basically, it was frozen back when 6.1 was discontinued... IBM stopped supporting it in 2016. I don't have access to a machine that old to test anything on, and didn't want to having to limit my code to what was available in 2008 (when 6.1 was released, 16 years ago!)

So I kept that old one around for people who are very far behind... but I can't support it, I have no way to test if code will work that far back. You'll have to make those changes yourself.

[Kyros]

Hi Scott, so here is how i download and compile YAJL:

i download the first package of the table, the one named "YAJL for IBM i". I don't use the V6R1M0 version because i already noticed that it was far behind the other version.
What i then do is follow the readme section named "TO INSTALL ALL SOURCES AND BUILD FROM SOURCE" but before i compile and launch BUILDYAJL i make 2 changes:

• Change the value of &tgtrls to V6R1M0

• Force the value of &VERSION to V6R1M0.

I force the value of &VERSION because i don't want to compile anything that does not work with &tgtrls V6R1M0.

Just as i am writing this i re-did all the steps in the README and i can confirm that i see no changes in YAJL/QRPGLESRC/YAJLR4 (there is no getDouble() function) so could it be that the source members were not updated?

[Scott Klement]

After your first statement that YAJLR4 didn't change, I downloaded the file from my site and restored and check the source member, and indeed, it had the new code in it.

I've now done it again. So TWICE I've re-downloaded from my site and restored it, and indeed, the updated source is there.

You're saying that it's not... something must be going wrong. Either you're getting a cached version when you download it, so are getting an old one, or somewhere else in your download/restore process, something is wrong.

[Kyros]

I spent the entire afternoon trying to figure out what i was doing wrong, but i did not arrive to a conclusion

I was obviusly doing something wrong because at some point after many attempts i managed to get the new source on my IBM i...
After that i edited YAJLR4 to comment out the function YAJL_GET_TIMESTAMP because it was using %scanrpl that unfortunately we cannot use since we have to compile to V6R1M0 (since i don't need the function and i was in a hurry to test the latest version i just commented it out).

After that I tested your fix and it works fine, sorry for wasting your time and thank your for the fix!

[Scott Klement]

I understand. I sympathize with you, but I can't support V6R1.

If you'd like to contribute what you have working on 6.1 back to the project, I can update the download that's made for IBM i 6.1 customers to the newer code that you've got working.

[Kyros]

I don't think there is much to contribute, the changes i made are extremely small:

YAJL/QCLSRC/BUILDYAJL.CLLE:

• Change the value of &tgtrls from V7R1M0 to V6R1M0

• Change the value of &VERSION from being dynamic to V6R1M0 (this makes it so that the "modern" part is not even compiled)

YAJL/QRPGLESRC/YAJLR4
YAJL/QSRVSRC/YAJLR4:

• I have just commented out the function YAJL_GET_TIMESTAMP and getISOTimestamp

After these changes everything that i need compiles and works just fine.

If you prefer i can create a SAVF with these changes so that you can use it to update the downlaod for the 6.1 version, or if there is a public repository i can make a PR with the proposed changes.

[Scott Klement]

This is perfect, I'll go ahead and make those few changes and put it online for the 6.1 version.

Right now there is no public repository for my fork of YAJL -- but, that is on my to-do list.