YAJLINTO handling varying number of array records

[Scudge]

Hi,

I'm new to YAJLINTO and working with JSON so please go easy on me

Doesn't help that my first attempt is with a (I think) complex JSON!

Below is a an example of the JSON response I am trying to parse. It's returned from an API call to retrieve a set of exchange rates on a given date, and the only data I am interested in is the 10 elements shown in red within the NAVEXRATELIST.Results array.

{
"d" : {
"results": [{
"__MetaData": {
"id": "https:sakjdskaljdklaskljsanndksadklsajdsjadjsadjsa=dsadshjkadsa=dsahkj",
"uri" : "hjkhkjhkjhkjhkjhjhjhjhkjhjkhhkjhjk",
"type": "exchange-rates.ExchRateList"},
"field1": "",
"InputDate": "",
"DateType": "",
"RateType": "",
"CurrFrom": "",
"CurrTo": "",
"ShowProtocol": false,
"NAVRETURN": {
"results": []
},
"NAVEXCRATELIST" : {
"Results" : [
{
"__MetaData": {
"id": "https:sakjdskaljdklaskljsanndksadklsajdsjadjsadjsa=dsadshjkadsa=dsahkj",
"uri" : "hjkhkjhkjhkjhkjhjhjhjhkjhjkhhkjhjk",
"type": "exchange-rates.ExchRateList"},
"RateType": "001A",
"FromCurr": "AAA",
"ToCurrncy": "BBB",
"ValidFrom": "/Date(1682812800000)/",
"ExchRate": "24.94164",
"FromFactor": "100",
"ToFactor": "1",
"ExchRateV": "0.00000",
"FromFactorV": "0",
"ToFactorV": "0"
},
{
"__MetaData": {
"id": "https:sakjdskaljdklaskljsanndksadklsajdsjadjsadjsa=dsadshjkadsa=dsahkj",
"uri" : "hjkhkjhkjhkjhkjhjhjhjhkjhjkhhkjhjk",
"type": "exchange-rates.ExchRateList"},
"RateType": "001A",
"FromCurr": "CCC",
"ToCurrncy": "DDD",
"ValidFrom": "/Date(1682812800000)/",
"ExchRate": "24.94164",
"FromFactor": "100",
"ToFactor": "1",
"ExchRateV": "0.00000",
"FromFactorV": "0",
"ToFactorV": "0"
},
{
"__MetaData": {
"id": "https:sakjdskaljdklaskljsanndksadklsajdsjadjsadjsa=dsadshjkadsa=dsahkj",
"uri" : "hjkhkjhkjhkjhkjhjhjhjhkjhjkhhkjhjk",
"type": "exchange-rates.ExchRateList"},
"RateType": "001A",
"FromCurr": "EEE",
"ToCurrncy": "FFF",
"ValidFrom": "/Date(1682812800000)/",
"ExchRate": "24.94164",
"FromFactor": "100",
"ToFactor": "1",
"ExchRateV": "0.00000",
"FromFactorV": "0",
"ToFactorV": "0"
}
]
}
}
]
}

It took me longer than I care to admit to get my data structures defined to handle this, but it is now working and the data is being parsed.

However, my issue is with the array size.

Here is my code:

Code: Select all

// Data structure to hold the 10 exchange rate items  (I'm just throwing everything into VarChar for now)

   Dcl-ds ExRateItem Qualified;            
     RateType VarChar(4);                  
     FromCurr VarChar(3);                  
     ToCurrncy VarChar(3);                 
     ValidFrom VarChar (25);               
     ExchRate  VarChar (13);               
     FromFactor  VarChar (3);              
     ToFactor    VarChar (3);              
     ExchRateV   VarChar (13);             
     FromFactorV VarChar (13);             
     ToFactorV   VarChar (3);              
   End-Ds;                                 
 
//Data Structure for the List of Exchange Rates, i.e. the array I am interested in
                                          
   Dcl-Ds ExRateList Qualified;            
     results LikeDS(ExRateItem) DIM(450);  
   End-Ds;          

//Data Structure for the NAVEXCRATELIST object
 
   Dcl-Ds ListSet Qualified;              
    NAVEXCRATELIST LikeDS(ExRateList);    
   End-Ds;                                
 
//Data Structure for the 'Main Result' object 
                                         
   Dcl-Ds MainResult Qualified;           
     results LikeDS(ListSet) DIM(1);      //(there will only ever be 1 "main" result)
   End-Ds;                    

//Final Data Structure for the root object itself           
                                          
   Dcl-Ds Root Qualified;                 
     d LikeDS(MainResult);                
   End-ds; 

// Parse the JSON.  allowextra=yes means I don't need to define data structures for everything. 

     data-into Root                                                 
        %data(%trim(ifsfile)                                        
        : 'doc=file case=any allowextra=yes allowmissing=yes')      
           %parser('YAJLINTO') ;         

// Quick loop just to count how many sets of exchange rates I loaded                           
                                                                    
  itemCount = 0;                                                    
   for i = 1 to %elem(Root.D.results(1).NAVEXCRATELIST.results);    
    if Root.D.results(1).NAVEXCRATELIST.results(i).FromCurr <> ' '; 
    itemCount +=1 ;                                                 
   endif;                                                           
   endfor;                                                                                                                

My sample JSON above just has 3 sets of Exchange Rates, but the real JSON has approximately 350 - BUT THIS IS VARIABLE (can be + or - a few specific currencies depending on certain conditions).

Because my test data just has 3 sets of rates the parse failed as it 'expects' 450, and likewise trying to parse a real file (357 sets) failed for the same reason.

As soon as I added "allowmissing=yes" the parse was fine.

The problem is I have no idea how many sets of data I actually loaded. As soon as my count loop (i) exceeds the number of items it actually found my program fails with "Length of varying length variable is out of range", so using this test data it correctly loops 3 times and itemCount = 3, but on the 4th loop it fails because there is no 4th+ array element data. Likewise using a real file, it loops 357 times but then fails on loop 358.

I have seen some comments about using countprefix_ but I'm struggling to figure out exactly where to code that. I did try it in a few places but the count was nowhere near what it should be (e.g. when parsing the real file with 357 items the count came back at 64?), and furthermore the array always ended up empty anyway.

Of course I will be doing more on each iteration of the loop, not just counting (I will be reformatting each data item (if required) and writing to an externally described physical file, I've just put this quick count loop in place to get the basic program structure in place.

Hope this makes sense.

Many thanks.

[jonboy49]

Look into the countprefix option. That is what you need to use to get a count of the actual number. Can’t type code on my phone or I’d try to give you an example.

If you really only want that data you should also look at using the path option to direct the parse to the bit though want.

[Scudge]

Thanks for the reply.

I have tried using the countprefix option and no matter which data structure I add the num_ variable to, I always get the answer of 64 and I have no idea why.

Below is where I have added num_ExRateList into the top level Root object, but as I say it doesn't matter which data structure I add it to, I always get 64.

Code: Select all

 Dcl-ds ExRateItem Qualified; 
   RateType VarChar(4);       
   FromCurr VarChar(3);       
   ToCurrncy VarChar(3);      
   ValidFrom VarChar (25);    
   ExchRate  VarChar (13);    
   FromFactor  VarChar (3);   
   ToFactor    VarChar (3);   
   ExchRateV   VarChar (13);  
   FromFactorV VarChar (13);  
   ToFactorV   VarChar (3);   
 End-Ds;       

Dcl-Ds ExRateList Qualified;           
  results LikeDS(ExRateItem) DIM(5);   
End-Ds;                                
                                       
Dcl-Ds ListSet Qualified;              
 NAVEXCRATELIST LikeDS(ExRateList);    
End-Ds;                                
                                       
Dcl-Ds MainResult Qualified;           
  results LikeDS(ListSet) DIM(1);      
End-Ds;                                
                                       
Dcl-Ds Root Qualified;                 
 num_ExRateList INT(3);                
  d LikeDS(MainResult);                
End-ds;                

 data-into Root                                               
    %data(%trim(ifsfile)                                      
    : 'doc=file case=any allowextra=yes allowmissing=yes ' +  
      'countprefix=num_')                                     
       %parser('YAJLINTO') ;                                                   
                

Also, from what I have read on other posts, using the countprefix means I can remove the allowmissing=yes option, but if I do that I get a parse failure (DATA-INTO operation does not match the RPG variable) which suggests maybe I'm not using countprefix correctly?

I have also tried to simplify completely using the path option as you suggested. I can get the document to parse, but if I inspect the array variables they are all empty so nothing has actually parsed in.

Below is the simplified code using the path option. I read in the Nerds Guide that I had to name the document, hence the use of document_name.

I do like this simplified code as it's stripped right back to a single Data Structure, although I will still need to add the countprefix to this so I know how many items actually got parsed into the array.

Code: Select all

 Dcl-ds Results Qualified DIM(5);
   RateType VarChar(4);          
   FromCurr VarChar(3);          
   ToCurrncy VarChar(3);         
   ValidFrom VarChar (25);       
   ExchRate  VarChar (13);       
   FromFactor  VarChar (3);      
   ToFactor    VarChar (3);      
   ExchRateV   VarChar (13);     
   FromFactorV VarChar (13);     
   ToFactorV   VarChar (3);      
 End-Ds;              
 
   data-into Results                               
     %data(%trim(ifsfile)                         
     : 'doc=file case=any allowextra=yes ' +      
       'allowmissing=yes ' +                      
       'path=GHORates/d/results/NAVEXCRATELIST')  
        %parser('YAJLINTO'                        
         :'{ "document_name": "GHORates" }');                

Regards

[Scudge]

I have implemented a temporary work around on this one (in case it helps anyone else in the future).

The program/API will only get called once per day so +nanoseconds extra run time is irrelevant.

I have reverted back to the code from my original post, but just before executing the data-into command I have added the below code to FORCE the full 450 dimension array to be initialised, specifically setting RateType = 'STOP' (this will never be a valid value in the JSON response).

Code: Select all

For i=1 to %elem(Root.D.results(1).NAVEXCRATELIST.results);    
  Root.D.results(1).NAVEXCRATELIST.results(i).RateType =  'STOP';
Endfor;                                                        

When data-into parses the data it correctly 'replaces' the 'STOP' value with the actual value from the JSON for each item it finds in the array.

Now when I execute the below code after the parse it works perfectly fine with no array size/length errors.

Code: Select all

itemCount = 0;                                                    
for i = 1 to %elem(Root.D.results(1).NAVEXCRATELIST.results);     
  if Root.D.results(1).NAVEXCRATELIST.results(i).RateType <> 'STOP';
     // Do some more stuff   
      itemCount +=1 ;                                                
  else;                                                             
      leave;                                                         
  endif;                                                            
endfor;                                                                                                                     

So, I may not have mastered the countprefix and/or the path options, but I now have the basic code structure working exactly as needed.

[Scott Klement]

Hello,

First of all, please initialize your data structures. RPG has a "bad old feature" where it initializes everything in a data structure to *BLANKS unless you have the INZ keyword. Blanks in EBCDIC are hex x'40' or decimal 64. That is why you are getting a result of 64 in your INT(3) fields. Also, I would recommend using INT(10) as that is the fastest performing. (Indeed, I thought it was required for countprefix, but I may be wrong about that.)

Code: Select all

                                       
Dcl-Ds Root Qualified inz;               <-- code INZ on EVERY data structure
  d LikeDS(MainResult) inz(*likeds);     <-- code INZ(*LIKEDS) on every LIKEDS of a data structure.
End-ds;                

Second, you must code the num_field where you need it. The following code MAKES NO SENSE AT ALL:

Code: Select all

Dcl-Ds Root Qualified;                 
 num_ExRateList INT(3);                
  d LikeDS(MainResult);                
End-ds;                

You can't count the number of ExRateList in the Root data structure BECAUSE IT DOESN'T HAVE A FIELD NAMED EXRATELIST. So RPG will leave it untouched, and since it is an INT(3) that is not initialized, it will be 64.

You could code num_d, but there's always only 1 of these, so why do that? The part that is variable is the results DS, so that is where you should be using the num_ field

Code: Select all

Dcl-Ds ExRateList Qualified inz;        
  num_results int(10);   
  results LikeDS(ExRateItem) DIM(450) inz(*likeds);   
End-Ds;                                

The following code works perfectly for me:

Code: Select all

**free

 dcl-s ifsFile varchar(200) inz('/home/sklement/scudge.json');
 dcl-s i int(10);

 Dcl-ds ExRateItem Qualified inz; 
   RateType VarChar(4);       
   FromCurr VarChar(3);       
   ToCurrncy VarChar(3);      
   ValidFrom VarChar (25);    
   ExchRate  VarChar (13);    
   FromFactor  VarChar (3);   
   ToFactor    VarChar (3);   
   ExchRateV   VarChar (13);  
   FromFactorV VarChar (13);  
   ToFactorV   VarChar (3);   
 End-Ds;       

Dcl-Ds ExRateList Qualified inz;        
  num_results int(10);   
  results LikeDS(ExRateItem) DIM(450) inz(*likeds);   
End-Ds;                                
                                       
Dcl-Ds ListSet Qualified inz;              
 NAVEXCRATELIST LikeDS(ExRateList) inz(*likeds);    
End-Ds;                                
                                       
Dcl-Ds MainResult Qualified inz;            
  results LikeDS(ListSet) DIM(1) inz(*likeds);      
End-Ds;                                
                                       
Dcl-Ds Root Qualified inz;                 
  d LikeDS(MainResult) inz(*likeds);                
End-ds;                

 data-into Root                                               
    %data(%trim(ifsfile)                                      
    : 'doc=file case=any allowextra=yes allowmissing=yes ' +  
      'countprefix=num_')                                     
       %parser('YAJLINTO') ;                                                   
                
for i = 1 to root.d.results(1).NAVEXCRATELIST.num_results;

  snd-msg *info 'FromCurr = ' + 
    root.d.results(1).NAVEXCRATELIST.results(i).FromCurr;

  snd-msg *info 'ToCurrncy = ' + 
    root.d.results(1).NAVEXCRATELIST.results(i).ToCurrncy;

endfor;

*inlr = *on;

When I run this, I get:

Code: Select all

FromCurr = AAA     
ToCurrncy = BBB    
FromCurr = CCC     
ToCurrncy = DDD    
FromCurr = EEE     
ToCurrncy = FFF    

[Scott Klement]

I have implemented a temporary work around on this one (in case it helps anyone else in the future).

The program/API will only get called once per day so +nanoseconds extra run time is irrelevant.

I have reverted back to the code from my original post, but just before executing the data-into command I have added the below code to FORCE the full 450 dimension array to be initialised, specifically setting RateType = 'STOP' (this will never be a valid value in the JSON response).

When IBM first created XML-INTO (the forebear of DATA-INTO) they didn't have countprefix, so I would use a solution like the one you've provided. There are situations where you can't use this approach, but perhaps for your document it is fine.

That said, it definitely won't work when using DATA-GEN to generate documents. So unless you only want to work with reading documents, and this is the only document you'll ever read... I would take the time to get countprefix working. It's not hard, and I've already posted an example above where it works fine with your document.

[Scudge]

Thanks for your reply Scott.

I have made the changes as per your example and it now works perfectly (my temporary fix has also been removed).

I have also been able to remove the 'allowmissing=yes" option which is another bonus.

I must have added the num_field to every data structure, starting with ExRateList for the exact reason you stated, but I just kept getting "64" so kept moving it up the 'tree' until I got right up to the root where I still got 64, but you have explained why.

As soon as I used INZ and INZ(*LIKEDS) on every data structure as you suggested it worked fine - a more elegant solution to my temporary fix.

Thanks again

[jonboy49]

Glad to hear you have it working. For the path option the following works. Note the use of the element count from the psds. RPG populates this when you are parsing into an array as there is no opportunity to add a countprefix field.

Code: Select all

dcl-ds pgmStat psds;
   elements int(20) pos(372);
end-ds;

Dcl-ds ExRateItem Dim(999) Qualified;            
   RateType VarChar(4);                  
   FromCurr VarChar(3);                  
   ToCurrncy VarChar(3);                 
   ValidFrom VarChar (25);               
   ExchRate  VarChar (13);               
   FromFactor  VarChar (3);              
   ToFactor    VarChar (3);              
   ExchRateV   VarChar (13);             
   FromFactorV VarChar (13);             
   ToFactorV   VarChar (3);              
End-Ds;                                 
 

data-into ExRateItem                                                 
        %data('/home/PARIS/JSONStuff/bigjson.json'                                        
        : 'doc=file case=convert allowextra=yes +
        path=root/d/results/NAVEXCRATELIST/Results')      
           %parser('YAJLINTO':'{ "document_name": "root" }') ;   

Dsply ('Found ' + %Char(elements) + ' elements');

[Scudge]

Thanks for the reply, I will have a play using the path option following your example (I was close!).

Sorry, just one more question if I may, and I appreciate this is more of an RPGLE query than YAJLINTO.

My JSON response has a data item called __Metadata which luckily I don't need so have ignored.

However, if I did need this data item I am unable to add it to the Data Structure as _ is invalid as a field name prefix, let alone double _

The ALIAS keyword can't be used here either.

How would you get around that one?

Thanks