YAJL_OBJECT_FIND with no list to find

Discussions relating to the ScottKlement.com port of the open source YAJL JSON Reader/Generator. This includes the YAJL tool as well as the YAJLR4, YAJLGEN, YAJLINTO and YAJLDTAGEN add-ons from ScottKlement.com. http://www.scottklement.com/yajl/
Post Reply
mgallipeau
Posts: 5
Joined: Fri Mar 25, 2022 8:54 pm

YAJL_OBJECT_FIND with no list to find

Post by mgallipeau »

I'm completely new to YAJL and completely new to this forum, but I'm trying to use the YAJL tree and am having some issues. I initially tried the DATA-INTO, but discovered that I can't because the incoming data isn't in a consistent format.

Here is a snippet of the JSON text:

Code: Select all

[
    {
        "id": "502a-sdf4sf-b5af-08da00b3d4",
        "key": "000039",
        "name": "ITEM NUMBER",
        "extensionDataByField": {
            "batch": "B0000025",
            "quantity": "100",
            "customer": "CUSTOMER NAME",
            "status": "In Transit",
            "statustime": null
        },
        "extensionData": {
            "status": "In Transit",
            "batch": "B0000025",
            "quantity": "100",
            "customer": "CUSTOMER NAME",
            "laststatuschange": "Thu Feb 10 2022"
        },
        "ownerId": null,
        "ownerKey": null,
        "ownerName": null,
        "ownerExtensionDataByField": null,
        "ownerExtensionData": null,
        "tagId": "tag0sdr-es24d-468b-597c-08da0b4a0dec",
        "tagKey": "A00000000000000000000039"
    }
]
Looking at the example in the documentation, I don't have a "list" identifier or anything to label the element, and I think I just don't know how to point to the node without it. Right before the loop, the "list" variable is coming back as null because there isn't a "list" in the JSON document. How can I point to the right place without it?

Code: Select all

    dcl-ds list_t   qualified template;

      id      varchar(50) inz('');
      key     varchar(6);
      name    varchar(25);

      dcl-ds extensionDataByField;
        batch           char(8);
        quantity        int(5);
        customer        varchar(40);
        status          varchar(25);
        statusTime      varchar(50);
      end-ds;

      dcl-ds extensionData;
        glstatus              varchar(25);                         
        glbatch               char(8);                          
        glquantity            int(5);                           
        glcustomer            varchar(40);                        
        gllaststatuschange    varchar(50);
      end-ds;

      ownerId                     varchar(50);
      ownerKey                    varchar(50);
      ownerName                   varchar(50);
      ownerExtensionDataByField   varchar(50);
      ownerExtensionData          varchar(50);
      tagId                       varchar(50);
      tagKey                      char(24);

    end-ds;


    dcl-ds result qualified;
      list     likeds(list_t) dim(999);
    end-ds;

    docNode = yajl_stmf_load_tree(FILE_LOCATION : errMsg);
    if errMsg <> '';
      // handle error
    endif;

    list = yajl_object_find(docNode : 'list');  <-- issue here

    i = 0;
    dow yajl_array_loop(list : i : node);

      j = 0;
      dow yajl_object_loop(node : j : key : val);

        select;
          when key = 'id';
            result.list(i).id = yajl_get_string(val);
          when key = 'key';
            result.list(i).key = yajl_get_string(val);
          when key = 'name';
            result.list(i).name = yajl_get_string(val);
        endsl;
      enddo;

    enddo;                            
jonboy49
Posts: 200
Joined: Wed Jul 28, 2021 8:18 pm

Re: YAJL_OBJECT_FIND with no list to find

Post by jonboy49 »

Basically your code, as far as it goes, is just using the wrong node for the object loop.

You have this:

Code: Select all

docNode = yajl_stmf_load_tree(FILE_LOCATION : errMsg);
...
    list = yajl_object_find(docNode : 'list');  <-- issue here

    i = 0;
    dow yajl_array_loop(list : i : node);
I changed it to this and it works fine.

Code: Select all

// list = yajl_object_find(docNode : 'list');  <===  Deleted this line

i = 0;

dow yajl_array_loop( docNode : i : node);  <=== Changed this to use docNode.
There's a bunch of other stuff you'll need to do to drill down into the nested levels but this was your basic problem.

Just out of interest why won't DATA-INTO work for you? There's nothing in this sample it would handle easily and it can deal with missing elements just fine so what is the problem? It should be much simpler code than using YAJL directly is going to require.
mgallipeau
Posts: 5
Joined: Fri Mar 25, 2022 8:54 pm

Re: YAJL_OBJECT_FIND with no list to find

Post by mgallipeau »

YES! Thank you! That is so obvious once you point that out. That did the trick.

I couldn't get the DATA-INTO to work because they company sending the JSON document couldn't guarantee that the order of the items would come through consistently. What I copied into the initial post was just a snippet of the full JSON data. In some cases, the batch field would come first in that extensionData area, and in others the status would come first. And then they did have cases where a field in there didn't come through at all, so it always gave me an error when I tried to use it.

When I asked the company if they could make it consistent, they said they couldn't and that normal JSON doesn't require it. So that's when I shifted and went for the YAJL method.

Thanks again for your help. Much appreciated!
Scott Klement
Site Admin
Posts: 636
Joined: Sun Jul 04, 2021 5:12 am

Re: YAJL_OBJECT_FIND with no list to find

Post by Scott Klement »

The order of the fields doesn't matter to DATA-INTO.

If a field can be missing, though, you do have to write code that tells it that it can be missing. But, it definitely would work.
mgallipeau
Posts: 5
Joined: Fri Mar 25, 2022 8:54 pm

Re: YAJL_OBJECT_FIND with no list to find

Post by mgallipeau »

I guess I don't really understand then. The documentation seemed to say the data structure had to exactly match and that was the trickiest part of the DATA-INTO. It also seemed to suggest the tree version, though it was trickier to learn and used more memory, was much easier to use, so I'm not sure why everyone seems to try and steer me to the DATA-INTO.

I have both missing fields and data in the JSON structure I'm receiving, and it's regularly not in the same order. I kept getting an error when I tried to use the full JSON file rather than just a uniform snippet of it. The documentation I found seemed to suggest it was because they all had to match exactly to the data structure, so I assumed that's what my issue was and tried a different way.
jonboy49
Posts: 200
Joined: Wed Jul 28, 2021 8:18 pm

Re: YAJL_OBJECT_FIND with no list to find

Post by jonboy49 »

Figured since I had the OPs code set up I'd show the modifications to make it work for DATA-INTO even if the elements are in a different sequence as Scott noted. I also added a couple of count fields to allow for testing of omitted elements and added the PSDS count so I could simplify the array definition. Tested it by basically doubling up the supplied json and messing with it some. This should give you (and future readers of this thread - which is the real point of this) a good idea of how you can make it work.

There are JSON docs that DATA-INTO can't handle well but I have not seen that many so far.

Code: Select all

**free
Ctl-opt  DftActGrp(*No)  BndDir( 'YAJL/YAJL' );

Dcl-Ds progStatus psds;
   resultCount  Int(20)  POS(372);  // Number of result rows loaded
End-Ds;

dcl-ds result  qualified Dim(999);  // No need for the template - just load into this direct

   id      varchar(50) inz('');
   name    varchar(25);

   num_tagKey         int(5);  // << Cater for tagKey being optional
   tagKey                   char(24);  // << Changed position
   dcl-ds extensionData;  // Had to change the field names here because they didn't match
      status              varchar(25);
      batch               char(8);
      quantity            int(5);
      customer            varchar(40);
      laststatuschange    varchar(50);
   end-ds;

   key     varchar(6);  // << changed position

   // Change position of extensionDataByField and allow to be omitted
   num_extensionDataByField  int(5);
   dcl-ds extensionDataByField;
      batch           char(8);
      quantity        int(5);
      customer        varchar(40);
      status          varchar(25);
      statusTime      varchar(50);
   end-ds;

   ownerId                     varchar(50);
   ownerKey                    varchar(50);
   ownerName                   varchar(50);
   ownerExtensionDataByField   varchar(50);
   ownerExtensionData          varchar(50);
   tagId                       varchar(50);

end-ds;

dcl-s jsonFile  varchar(40) inz('/Home/Paris/jsonstuff/playjson.json');

data-into result  %Data( jsonFile : 'case=convert doc=file countprefix=num_')
                  %Parser('YAJL/YAJLINTO');

Dsply (%char(resultCount) + ' rows loaded');

*InLr = *On;
mgallipeau
Posts: 5
Joined: Fri Mar 25, 2022 8:54 pm

Re: YAJL_OBJECT_FIND with no list to find

Post by mgallipeau »

Thank you for that. With your example, I was able to get it to work that way as well.

Can you explain exactly what the num_ fields all do? How does having that before the inner data structure allow the program to function when one of the fields within that inner data structure are not there? Like you have one before the tagKey, presumably to allow that element to be optional. I get that. But then I would have expected you to have a num_ variable for the extensionDataByField.statusTime if that statusTime value was optional rather than having the num_ variable for the entire extensionDataByField.

Also, that resultCount variable is great. How would I have known that was available before you used it in your example?

I really appreciate you helping to make sense of some of this for me.
jonboy49
Posts: 200
Joined: Wed Jul 28, 2021 8:18 pm

Re: YAJL_OBJECT_FIND with no list to find

Post by jonboy49 »

mgallipeau wrote: Sat Mar 26, 2022 7:53 pm Can you explain exactly what the num_ fields all do? How does having that before the inner data structure allow the program to function when one of the fields within that inner data structure are not there? Like you have one before the tagKey, presumably to allow that element to be optional. I get that. But then I would have expected you to have a num_ variable for the extensionDataByField.statusTime if that statusTime value was optional rather than having the num_ variable for the entire extensionDataByField.
I just put arbitrary ones in there as an example. Figured showing you to handle individual fields as well as whole structures would be useful for you.

As to the num_ fields in general. If you place a field in the DS then RPG expects there to be data for it. If there is no data is is considered a "missing" error. The same thing applies to arrays. If you say Dim(99) and there are only 4 then 95 are "missing". However, RPG will allow fields to be "missing" as long as it has a way to tell you about it. And that is what the num_ fields do (they can be named anything you like by the way - just change what is specified to countprefix). The num_ fields can be anywhere in the structure as long as they are at the same hierarchical level as the structure/field they are counting.
mgallipeau wrote: Sat Mar 26, 2022 7:53 pm Also, that resultCount variable is great. How would I have known that was available before you used it in your example?
Originally back in the early days of XML-INTO that was the only way to tell how many array elements had been loaded. RPG will always populate it if the top level target (result in my case) is an array. You would have known if you'd read the manual <grin>. Alternatively if you had read any of my articles on XML-INTO and DATA-INO which you can find here for DATA0INTO and JSON generally: https://authory.com/JonParisAndSusanGan ... ction=JSON or here for XML (many of the DATA-INTO ones refer back to these): https://authory.com/JonParisAndSusanGan ... Processing
mgallipeau
Posts: 5
Joined: Fri Mar 25, 2022 8:54 pm

Re: YAJL_OBJECT_FIND with no list to find

Post by mgallipeau »

Thanks again, Jon. Greatly appreciate your expertise. I'll definitely be checking out your page in the future. It looks like you've got some great stuff.

Is there really a manual or is that a joke? I thought you were serious, but now I'm guessing that was meant to be funny. This is what I had been using: https://www.scottklement.com/presentati ... %20RPG.pdf. If there really is a manual, that would be amazing! :lol:
jonboy49
Posts: 200
Joined: Wed Jul 28, 2021 8:18 pm

Re: YAJL_OBJECT_FIND with no list to find

Post by jonboy49 »

Scott's presentation along with my articles should give you most of what you need. But yes there is a manual - the RPG manual! Both DATA-INTO and DATA-GEN along with the basic options are described there. https://www.ibm.com/docs/en/i/7.4?topic ... o-variable

If you ever wanted to write your own parser (as Scott did for YAJLINTO and YAJLDTAGEN) then they were described in the Open Access documentation. https://www.ibm.com/docs/en/i/7.4?topic ... ss-edition

So the <grin> was not intended as a joke about there being a manual but rather a wink at the fact that RPGers tend not to read it!
Post Reply