DATA-INTO with YAJLINTO

Discussions related to HTTPAPI (An HTTP Client Package for RPG programming.) http://www.scottklement.com/httpapi/
davidb
Posts: 12
Joined: Tue May 31, 2022 3:48 pm

DATA-INTO with YAJLINTO

Post by davidb »

I'm receiving JSON on a webservice call, that is written to a temporary file in the IFS. That file is then fed into an RPG data structure using DATA-INTO with YAJLINTO.

I believe the incoming data is causing a problem for the parser were it has an empty array named "pastFireOverview" (see below).

The RPG data structure and DATA-INTO command are very simple, defined as :

D List ds qualified
d riskScore 2a varying
d trackingid 32a varying

DATA-INTO List %Data( %trimr(Tempfile)
: 'doc=file case=any allowextra=yes')
%PARSER('YAJLINTO');

When the DATA-INTO command executes, no error is encountered but, whereas the riskScore variable is populated, the trackingId is not. Note that trackingId comes after the empty array variable in the incoming data. In fact if I change the RPG data structure to include other names found in the incoming data, any corresponding entity, that is located after the empty array, fails to map to the corresponding variable in the data structure. If I remove the empty "pastFireOverview" array from the incoming JSON, all data maps correctly.

Has anyone experienced this and if so, have you found a solution? I think it's unlikely that the webservice provider would be open to changing the content.

{"riskScore":66,"riskDescription":"Urban","riskLevel":1,"nearestHighRiskLevel":"746","nearestVeryHighRiskLevel":"5614","preburnRiskScore":-1,"preburnRiskDescription":"NA","preburnDistanceHigh":"NA","preburnDistanceVeryHigh":"NA","landUseDensityClass":"High Density Residential","landUseDensityLevel":2,"distanceWildlandFeet":1816,"avgDayHighWind":"2","recentlyBurned":"No","numFiresRecentYears":0,"pastFireOverview":[],"fuel":"The wildfire fuel on your property is considered low and does not pose a significant threat for wildfire","drought":"The drought conditions over the past 5 years are contributing to the wildfire threat at a high level at your location","wind":"The wind conditions in the areas that encompass your property over the last 5 years are contributing to the wildfire threat at a low level.","slope":"The slope associated with your property is not a factor in the wildfire threat.","aspect":"The aspect associated with your property is a factor in the wildfire threat.","distanceHighVeryHighFuels":"Wildfire fuels are in relatively close proximity to your location. As the distance to higher hazard wildfire fuel decreases, the threat of embers increases.","distanceWildland":"The distance from your property to surrounding large open areas that are not developed poses moderate wildfire threat. ","firebreakCategory":"The density of the structures in the area surrounding your property is high and is not a factor in the wildfire threat to this property. ","trackingId":"c71024af17b85203ab79dc94bdb81756"}
jonboy49
Posts: 220
Joined: Wed Jul 28, 2021 8:18 pm

Re: DATA-INTO with YAJLINTO

Post by jonboy49 »

I just tested your basic scenario and encountered no problem. The trackingid populated as expected. This is my code:

Code: Select all

**free
Dcl-ds List qualified;
   riskScore varchar(2);
   trackingid varchar(32);
end-ds;
 
dcl-s testJson varchar(200)
   inz('{"riskScore":66,"pastFireOverview":[],-
   "trackingId":"c71024af17b85203ab79dc94bdb81756"}');
 
DATA-INTO List %Data( testJson : 'case=any allowextra=yes')
                 %parser('YAJL/YAJLINTO');
Try this and see if it works. If it does then the problem must be elsewhere in the document and not the empty array.

If it does not work I suggest checking that you have the latest version of YAJLINTO.

If that still doesn't help then set the DATA-INTO debug environment variable and see if that helps.
To set it use ADDENVVAR ENVVAR(QIBM_RPG_DATA_INTO_TRACE_PARSER) VALUE(*STDOUT)
- Details here: https://www.ibm.com/docs/en/i/7.3?topic ... nto-parser
davidb
Posts: 12
Joined: Tue May 31, 2022 3:48 pm

Re: DATA-INTO with YAJLINTO

Post by davidb »

Very odd, I downloaded the latest YAJL from Scott's repo, loaded the YAJL library using the yajllib72 version (we're on V7R3M0).

I set the relevant parts of the program to use the test data that you identified. Recompiled the rpg with the new yajl library top of my libl...

trackingId is still not populating into the variable.

However the parser log shows it correctly..

Data CCSID 13488
Converting data to UTF-8
Allocating YAJL stream parser
Parsing JSON data (yajl_parse)
No name provided for struct, assuming name of RPG variable
StartStruct
ReportName: 'riskScore'
ReportValue: '66'
ReportName: 'pastFireOverview'
StartArray
EndArray
ReportName: 'trackingId'
ReportValue: 'c71024af17b85203ab79dc94bdb81756'
EndStruct
YAJL parser status 0 after 95 bytes
YAJL parser final status: 0
---------------- Finish --------------
Press ENTER to end terminal session.
jonboy49
Posts: 220
Joined: Wed Jul 28, 2021 8:18 pm

Re: DATA-INTO with YAJLINTO

Post by jonboy49 »

Hmmmm very strange.

Did you actually try running my code as-is ? That would at least rule out the possibility of a missing PTF or something,
davidb
Posts: 12
Joined: Tue May 31, 2022 3:48 pm

Re: DATA-INTO with YAJLINTO

Post by davidb »

Yes I copied your code into the program and executed that.

However, the plot thickens.. this is running on our dev iSeries, which is at V7R3M0.

I copied all the code and source to our production iSeries, which is also at V7R3M0. recompiled the code there and ran it, and the trackingId populates there.

I cannot see any difference between environments as far as installed programs. 'go licpgm' installed programs appear to be identical.

PTF's may be different but there are to many to compare & contrast. I'm betting on something in there being the culprit. I think we've proved that the code is ok. Since we're upgrading i the near future, I'll take it as a temporary glitch.

Thanks so much for our help !
jonboy49
Posts: 220
Joined: Wed Jul 28, 2021 8:18 pm

Re: DATA-INTO with YAJLINTO

Post by jonboy49 »

Glad to hear you at least know that the empty array is not an issue.

There have been one or two PTFs related to RPG's DATA-INTO so I'm guessing that one of them is missing on one system.

Good luck.
Scott Klement
Site Admin
Posts: 732
Joined: Sun Jul 04, 2021 5:12 am

Re: DATA-INTO with YAJLINTO

Post by Scott Klement »

I guessing there's something else going on in your program that you haven't posted here.

I copy/pasted your code exactly as you posted it, and it works fine,.

Code: Select all

.....D*ame+++++++++++ETDsFrom+++To/L+++IDc.Keywords+++++++++++++++++++++++++
     D List            ds                  qualified
     d   riskScore                    2a   varying
     d   trackingid                  32a   varying

     D tempfile        c                   'davidb.json'      

       DATA-INTO List %Data( %trimr(Tempfile)
                          : 'doc=file case=any allowextra=yes')
                      %PARSER('YAJLINTO');

       *inlr = *on;                      
I copy/pasted the JSON you provided into a file named 'davidb.json', and ran it, and got the following in debug:

Code: Select all

 LIST.RISKSCORE = '66'                               
 LIST.TRACKINGID = 'c71024af17b85203ab79dc94bdb81756'
I haven't seen any bugs in either YAJL or DATA-INTO that would affect functionality this simple/basic. I think you've got something coded wrong in your program that you haven't posted here.

Please write a simple program (like the one I provided above) that simply recreates the problem without any dependencies on anything else (aside from the RPG compiler and YAJL). And post that code here (make sure you put [code] before and [/code] after your code sample so we don't have to guess how it was originally formatted.

I'm guessing you can't provide code that fails, because I'm pretty sure the problem is in another part of your program.
davidb
Posts: 12
Joined: Tue May 31, 2022 3:48 pm

Re: DATA-INTO with YAJLINTO

Post by davidb »

Apologies for the long delay, I was out...

So I have cut down my program to run the appropriate parts to get what we're looking at here...

The JSON file is being imported from another separate call to an API and it contains more data than just the riskScore and trackingId, however I when I cut the file contents down to just those two fields, DATA-INTO populates my List correctly, i.e. both RiskScore and trackingId are populated.

However, if I re-insert what I believe to be the offending item (pastFireOverview) in various places in the JSON file, this has a predictable effect if we assume that this field is breaking the process.

Code/:

dcl-ds List qualified;
riskScore varchar(2);
trackingId varchar(32);
end-ds;

Tempfile = '/tmp/Tempfile.JSON';

DATA-INTO List %Data( %trimr(Tempfile)
: 'doc=file case=any allowextra=yes ')
%PARSER('YAJLINTO');

/Code

I tried changing the json file as follows:

Tempfile.JSON contents:

Both of these populate riskScore and trackingId:
{"riskScore":68,"trackingId":"1fa10d3893e46e5bc549554094e80a34"}
{"riskScore":68,"trackingId":"1fa10d3893e46e5bc549554094e80a34","pastFireOverview":[]}

This only populates riskScore:
{"riskScore":68,"pastFireOverview":[],"trackingId":"1fa10d3893e46e5bc549554094e80a34"}

This populates neither riskScore nor trackingId:
{"pastFireOverview":[],"riskScore":68,"trackingId":"1fa10d3893e46e5bc549554094e80a34"}
Scott Klement
Site Admin
Posts: 732
Joined: Sun Jul 04, 2021 5:12 am

Re: DATA-INTO with YAJLINTO

Post by Scott Klement »

This is a bug in RPG that was fixed sometime in mid-2020. Please update your PTFs.
davidb
Posts: 12
Joined: Tue May 31, 2022 3:48 pm

Re: DATA-INTO with YAJLINTO

Post by davidb »

Also, this short test program only has what it needs to execute the same code:

This generates exactly the same results with the JSON test data laid out on my previous post.

Code/

H DFTACTGRP(*NO) ACTGRP(*NEW) BNDDIR('YAJL')

/include yajl_h

D TST_XXXXXX PR ExtPgm('TST_XXXXXX')
D Msg 256A
D TST_XXXXXX PI
D Msg 256A


D Tempfile s 50A


/free

dcl-ds List qualified;
riskScore varchar(2);
trackingId varchar(32);
end-ds;

Tempfile = '/tmp/Tempfile.JSON';

DATA-INTO List %Data( %trimr(Tempfile)
: 'doc=file case=any allowextra=yes ')
%PARSER('YAJLINTO');

return;
/end-free

/Code
Post Reply