YAJLINTO with object array throwing reason code 9 error

[mike_faust]

I have an application that accepts documents that contain JSON arrays, which are in turn loaded into data structure arrays. The application works fine as long as the JSON array only contains a single element but when it has more than one element DATA-INTO using YAJLINTO throws "The information supplied by the parser for the DATA-INTO operation was in error. Reason code 9."

Sample JSON Doc;

Code: Select all

{"documents":
	 {"first_structure": 
	 	[{"el1": "A"}, 
	 	{"el2": "B"}
	 	] 
	 }
} 

Sample Data Structure;

Code: Select all

dcl-ds document qualified;
     dcl-ds first_structure dim(10);
     	el_1 char(1);
     	el_2 char(1);
     end-ds;
 end-ds;
 

DATA-INTO statement;

Code: Select all

DATA-INTO documents.first_structure                  
      %DATA(%trim(jsonString):        
       'path=response/documents/first_structure ' +  
       'case=any allowmissing=yes allowextra=yes') 
        %PARSER('YAJLINTO':                        
                    '{ '  +                        
                  '"value_true": "true",' +        
                  '"value_false": "false",' +      
                  '"value_null": "",'  +           
                  '"skip_nulls": true, ' +         
                '"document_name": "response"' +    
                  '}');                            

[jonboy49]

That is one very weird json doc.

Your elx names were wrong - you had added an _ so I don't know how any of it could have worked but ...

The DS needs to look like this:

Code: Select all

dcl-ds documents qualified;
   num_first_structure int(5);
   dcl-ds first_structure dim(10);
      num_el1 int(5); 
      el1 char(1);
      num_el2 int(5);      
      el2 char(1);
   end-ds;
end-ds;

Reason is that in the json the el1 entry and the el2 are separated by a comma - in other words they are individual elements of first_structure. So the first_structure array has to allow for both fields and I have used num_ with countprefix=num_ to allow them to be optional.

I have a horrible feeling that you may be dealing with a json doc where the element names (e.g. el1) are arbitrary. In other words the entry after el2 could be "janice" or anything else. If this is the case then DATA-INTO is not going to work for you and you will have to use the raw YAJL APIs.

[mike_faust]

Thanks for the response. The JSON doc I used along with the RPG Data Strucure are both just examples so the format may be wrong, however the actual doc (which I can't easily share because it contains proprietary information and PII) are properly formatted JSON. What has me confused is I only see the issue when we receive a message with multiple occurrences in the array, even though all of them are defined as JSON arrays on both sides.

[mike_faust]

One quick update on this. I realized that the issue only occurs if the last element in the first array occurrence contains a null value. I was using the "skip_nulls" option to avoid issues with null values but removing that and replacing it with "value_null": "" to replace the null value with an empty string seems to work.

[jonboy49]

Glad you got it sorted.

If you can share an anonymized sample of the json and the DS you used it will help others in the future.

[mike_faust]

Absolutely. Here's a sample of the JSON that will cause the issue.

Sample JSON Doc

Code: Select all

{
    "documents": {
        "test_doc1": [
            {
                "element1": "value1",
                "element2": "value2"
            }
        ],
        "test_doc2": [
            {
                "element1": "value3",
                "element2": null
            },
            {
                "element1": "value4",
                "element2": "value5"
            }
        ]
    }
}

Note that the last element in the first array occurrence for test_doc2 is null.

RPG Code to call YAJLINTO that will throw the error:

Code: Select all

       ctl-opt dftactgrp(*no);

       readTheJson();
       *inlr = *on;

       dcl-proc readTheJson;

         dcl-ds jsonDoc qualified;
           dcl-ds DOCUMENTS;
             num_TEST_DOC1 int(10) inz(0);
             dcl-ds TEST_DOC1 dim(10);
               ELEMENT1 varchar(6) inz('');
               ELEMENT2 varchar(6) inz('');
             end-ds;
             num_TEST_DOC2 int(10) inz(0);
             dcl-ds TEST_DOC2 dim(10);
               ELEMENT1 varchar(6) inz('');
               ELEMENT2 varchar(6) inz('');
             end-ds;
           end-ds;
         end-ds;

         dcl-s ifsPathName varchar(5000);

         ifsPathName = '/tmp/test.json';

         data-into jsonDoc %DATA( ifsPathname
                                : 'doc=file case=convert countprefix=num_')
                         %PARSER( 'YAJLINTO'
                                : '{ "document_name": "jsonDoc", +
                                     "skip_nulls": true, +
                                     "number_prefix": "YAJL_" }');

       end-proc;

Note that this code was generated by the YAJLGEN command but I added the skil_nulls option. The fix is to replace /// "skip_nulls": true with "value_null": "" but this may not work for everyone.

[jonboy49]

I don't understand how this fixed anything. skip_nulls true or false makes no difference with the sample you supplied. It only really has any impact when a compound entry is null itself. Something like this:

Code: Select all

{
   "documents": {
       "test_doc1": null
       ...

Without the value_null it still works - just puts the value *null in the element.

[mike_faust]

In my case I need the null to not get loaded into the database as *NULL and I'm dealing with an extremely complex JSON doc so I need it to not load the nulls at all. You're correct though the real issue appears to be with the skip_nulls option itself when the last element in a structure that is part of an array with more than one entry.

[LUCAG]

I had a problem like your in the past, my solution was to fill the array of all the values coming from the json, also the null, and then "clear" (simply setting up to the default allowed for the data type) the nulls before inserting the values in the db table.

[LloydB]

Im experiencing the same issue with using "skip_null". I can concur about the pattern found above. The solution provided here, replacing "skip_nulls": true with "value_null": "", will not work for me. My JSON structure has many nulls with mixed data types (string or numeric). So replacing nulls with one data type will not work in my situation.

Wed love to utilize the "skip_null", but this issue is making us to have to code work arounds. Can this be looked into and repaired?

To reiterate the bug patter: If the JSON structure has multiple objects in an array, and the first object's last element is null, then DATA-INTO crashes. Here is a simple example to demonstrate the issue. I have repeated this pattern on simple structures, like the code here,.......all the way up to very complex JSON structures.

Code: Select all

**FREE

dcl-s t_json varchar(10000) template;

dcl-ds t_structure qualified template;
  first_name char(25);
  last_name char(25);
end-ds;

dcl-ds t_people qualified template;
  num_people int(10);
  people likeds(t_structure) dim(10);
end-ds;

dcl-s json like(t_json);
dcl-ds parsed likeds(t_people);

//******************************************************************************
// This will fail: 1st Array's last element is null.
//******************************************************************************
json='{"people": [ +
       { +
        "first_name":"George", +
        "last_name":null +
       }, +
       { +
        "first_name":"John", +
        "last_name":"Adams" +
       } +
      ]}';

//******************************************************************************
// This will work: 2nd Array's last element is null.
//******************************************************************************
//json='{"people": [ +
//       { +
//        "first_name":"George", +
//        "last_name":"Washington" +
//       }, +
//       { +
//        "first_name":"John", +
//        "last_name":null +
//       } +
//      ]}';

data-into parsed
       %data(json:'case=any allowextra=yes allowmissing=yes countprefix=num_')
       %parser('YAJL202409/YAJLINTO': '{"skip_nulls": true}');

*inlr = '1';