Combining JSON

Discussions relating to the ScottKlement.com port of the open source YAJL JSON Reader/Generator. This includes the YAJL tool as well as the YAJLR4, YAJLGEN, YAJLINTO and YAJLDTAGEN add-ons from ScottKlement.com. http://www.scottklement.com/yajl/
Post Reply
eddiesmith
Posts: 3
Joined: Fri Jun 10, 2022 2:14 pm

Combining JSON

Post by eddiesmith »

I've used SK's YAJL for quite a few years - absolutely cool tool. But as I got more familiar with it our company has expanded what we can do with it. Anyway we have quite a few Web Services running using this tool but I started to think about trying to reuse the JSON between RPG programs as opposed to just a response to APACHE. What I'm interested in doing is calling a program from a main program, the sub program returns the JSON (currently in a buffer) and add the JSON to he JSON document in the main program. Seems a relatively easy thing to do - I used yajl_buf_load_tree to create a Node and then yajl_genFromNode to add it to the main programs JSON document.
What I noticed was that I had to make sure that the code that did the yajl_buf_load_tree was carried out before the yajl_genOpen or else it would fail. I think the issue is that once yajl_genOpen has been run the the generator handle is gGenerator whereas if yajl_genOpen hasn't be run the yajl_buf_load_tree uses an arbitrary handler to create the node, then the node can be added to the document.

So this works - getAddress is just a subprogram that returns an address and flood score as JSON in the buffer

Code: Select all

                                                                  
getAddress(p_buffer :p_bufferlen);                                
addressNode = yajl_buf_load_tree(p_buffer :p_bufferlen :errMsg);  
                                                                  
yajl_genOpen(jsonPretty());                                       
yajl_beginObj();                                                  
yajl_addChar('type': 'person');                                   
yajl_addChar('name': 'Eddie');                                    
rc = yajl_genFromNode('address'                                   
          :YAJL_object_find(addressNode: 'address'));             
yajl_tree_free(addressNode);                                      
yajl_endObj();                                                    
                                                                  
if (not isError);                                                 
  httpStatus = 200;                                               
else;                                                             
  httpStatus = 400;                                               
endif;                                                            
                                                                  
yajl_writeStdout(httpStatus:errorDescription);                    

But if the getAddress is called after yajl_genOpen it fails, so this code fails

Code: Select all

yajl_genOpen(jsonPretty());                                      
yajl_beginObj();                                                 
yajl_addChar('type': 'person');                                  
yajl_addChar('name': 'Eddie');                                   
                                                                 
getAddress(p_buffer :p_bufferlen);                               
addressNode = yajl_buf_load_tree(p_buffer :p_bufferlen :errMsg); 
                                                                 
rc = yajl_genFromNode('address'                                  
          :YAJL_object_find(addressNode: 'address'));            
yajl_tree_free(addressNode);                                     
yajl_endObj();                                                   
                                                                 
if (not isError);                                                
  httpStatus = 200;                                              
else;                                                            
  httpStatus = 400;                                              
endif;                                                           
In reality I can code it like version 1 most of the time. But the use case that I'm trying to solve is getting a list of customers. So in this instance the main programs JSON would build an array of customers by call the sub program for each customer. The sub progam is already built and it accepts a customer ID and returns a full customer JSON. But I only want the address portion of the JSON, I don't want anything else from the address - ignore flood scores etc. So that kind of rules out using yajl_addPreformattedPtr because I need to traverse the JSON to find the correct subnode of just the address.

If it could be made to work in some fashion the advantage is that if any changes were made in the sub program then it bubbles up to the main program without major rework or recompiles. So for example in Ireland we have eventually added our version of the ZIP code called the Eircode. So in this instance we would only have to add the Eircode key :value pair into the Address node in the subprogram, it should then appear in the main program's JSON.

I'm probably not explaining myself too well - the main idea of mine is to try to modularise the JSON programs but be able to pass the JSON data between programs so they can be combined into other JSON documents.

Anyway if this is impossible I don't mind at all. And if Scott reads this - again thank you so much for a brilliant tool.
Eddie
Scott Klement
Site Admin
Posts: 635
Joined: Sun Jul 04, 2021 5:12 am

Re: Combining JSON

Post by Scott Klement »

eddiesmith wrote: Fri Jun 10, 2022 3:00 pm What I noticed was that I had to make sure that the code that did the yajl_buf_load_tree was carried out before the yajl_genOpen or else it would fail. I think the issue is that once yajl_genOpen has been run the the generator handle is gGenerator whereas if yajl_genOpen hasn't be run the yajl_buf_load_tree uses an arbitrary handler to create the node, then the node can be added to the document.
yajl_buf_load_tree doesn't use the generator at all. The handle that it uses is not related in any way to the gGenerator handle used by yajl_genOpen and the other generator functions.
eddiesmith wrote: Fri Jun 10, 2022 3:00 pm So this works - getAddress is just a subprogram that returns an address and flood score as JSON in the buffer
I have no idea what that means. Are you referring to a memory address? What on earth is a flood score? What does the memory address point to? What does this have to do with YAJL or your question?




But if the getAddress is called after yajl_genOpen it fails, so this code fails
[/quote]

Above, you said that yajl_buf_load_tree couldn't run after yajl_genOpen (which doesn't make any sense) now you're saying that getAddress cannot. Which one did you mean? If it is getAddress, I'll need to know a lot more about it than just "it returns an address and a flood score". I can't tell you why it malfunctions when I know nothing about it.
eddiesmith wrote: Fri Jun 10, 2022 3:00 pm

Code: Select all

yajl_genOpen(jsonPretty());                                      
yajl_beginObj();                                                 
yajl_addChar('type': 'person');                                  
yajl_addChar('name': 'Eddie');                                   
                                                                 
getAddress(p_buffer :p_bufferlen);                               
addressNode = yajl_buf_load_tree(p_buffer :p_bufferlen :errMsg); 
So, again, yajl_genOpen, and the other routines to generate JSON do not use anything related to yajl_buf_load_tree. I can't see why this wouldn't work -- but then, I haven't a clue what getAddress does. I think it's weird that you have a pointer to the buffer length, though. Or, at least, in my naming convention p_bufferLen would be a pointer to the buffer length. (The "p_" means "pointer to" vs just "bufferLen" being the buffer length.)

yajl_buf_load_tree is not expecting a pointer in the 2nd parameter.

When this fails, what is in the "errMsg" variable? The point of that variable is to tell you why things are failing. But, you seem to be ignoring it.

If the problem is that p_buffer and p_bufferlen aren't set properly, that would imply that getAddress is malfunctioning -- and again, I am not familiar with that, it is not a part of yajl.
eddiesmith wrote: Fri Jun 10, 2022 3:00 pm In reality I can code it like version 1 most of the time. But the use case that I'm trying to solve is getting a list of customers. So in this instance the main programs JSON would build an array of customers by call the sub program for each customer. The sub progam is already built and it accepts a customer ID and returns a full customer JSON. But I only want the address portion of the JSON, I don't want anything else from the address - ignore flood scores etc. So that kind of rules out using yajl_addPreformattedPtr because I need to traverse the JSON to find the correct subnode of just the address.
Again, I have no idea what you are saying here.
eddiesmith wrote: Fri Jun 10, 2022 3:00 pm If it could be made to work in some fashion the advantage is that if any changes were made in the sub program then it bubbles up to the main program without major rework or recompiles. So for example in Ireland we have eventually added our version of the ZIP code called the Eircode. So in this instance we would only have to add the Eircode key :value pair into the Address node in the subprogram, it should then appear in the main program's JSON.
Seems to me that all of the code is in the YAJLR4 and YAJL service programs, so it doesn't matter where you call it from. Unless they are in different activation groups? But then, you haven't actually told us what the problem is that you're trying to solve. All you've told us is that it has to do with the sequence in which you call various routines. But, you haven't us what problem you're encountering or what errors you're receiving, so it's hard to guess why the sequence or the "subprograms" would make any difference.
eddiesmith
Posts: 3
Joined: Fri Jun 10, 2022 2:14 pm

Re: Combining JSON

Post by eddiesmith »

Apologies Scott about not explaining things correctly. I'll give the full set of code as an example which should help. Just to give an example of what I'm trying to achieve in this example. I have a a subprogram that returns a customers address and also flood score (how likely their property is going to be flooded) - the Json looks like this

Code: Select all

{
    "address": {
        "line1": "1 Main Street",
        "line2": "Village x",
        "line3": "Ireland"
    },
    "floodScores": {
        "river": "Low",
        "sea": "High"
    }
}
If the main program i have the customer name etc and I want to integrate the 'address' portion of the JSON from the subprogram into it's output. So the JSON on the main program would look like

Code: Select all

{
    "customers": [
        {
            "number": 1,
            "name": "John",
            "surname": "Doe",
            "address": {
                "line1": "1 Main Street",
                "line2": "Village x",
                "line3": "Ireland"
            }
        }
    ]
}


The Subprogram code has two customers in it (please forgive the hard coding - it is just for testing etc) and looks like this

Code: Select all

       ctl-opt thread(*serialize)
               option(*srcstmt: *nodebugio: *noshowcpy)
               ccsid(*char:*jobrun)
               decedit('0.')
               bnddir('YAJL');

      /copy yajl_h

       // entry parameters
       dcl-pi entry extpgm('TSJSON02');
         customerNumber int(10:0);
         p_buffer pointer;
         bufferlen int(10:0);
       end-pi;

       // --------------------------------------------------
       // standard web/json error data structure
       // --------------------------------------------------
       dcl-ds err qualified;
         bytesProv int(10:0) inz(0);
         bytesAvail int(10:0) inz(0);
       end-ds;

      // --------------------------------------------------
      // standard web/json fields
      // --------------------------------------------------

       // work fields
       dcl-s i int(10) inz;
       dcl-s w_ccsid int(10:0) inz(0);
       dcl-s w_json_buffer pointer;
       dcl-s w_json_buffer_length uns(10:0);

       // Code-----------------------------------------------------------

       yajl_genOpen(*OFF);
       yajl_beginObj();
       if customerNumber = 1;
         yajl_beginObj('address');
           yajl_addChar('line1': '1 Main Street');
           yajl_addChar('line2': 'Village x');
           yajl_addChar('line3': 'Ireland');
         yajl_endObj();
         yajl_beginObj('floodScores');
           yajl_addChar('river': 'Low');
           yajl_addChar('sea': 'High');
         yajl_endObj();
       else;
         yajl_beginObj('address');
           yajl_addChar('line1': '2 Big Street');
           yajl_addChar('line2': 'Village Y');
           yajl_addChar('line3': 'Ireland');
         yajl_endObj();
         yajl_beginObj('floodScores');
           yajl_addChar('river': 'Medium');
           yajl_addChar('sea': 'Low');
         yajl_endObj();
       endif;
       yajl_endObj();

       // copy the json buffer to the response buffer
       yajl_getBuf(w_json_buffer: w_json_buffer_length);
       p_buffer = %alloc(w_json_buffer_length);
       yajl_copyBuf(w_ccsid: p_buffer: w_json_buffer_length:
                    bufferlen);

       yajl_genClose();

       *inlr = *on;
       return;

The main program works fine if I call the subprogram and create a node for the customerAddress before I try to integrate it into the main JSON. Here is my code

Code: Select all

       ctl-opt thread(*serialize)
               option(*srcstmt: *nodebugio: *noshowcpy)
               ccsid(*char:*jobrun)
               decedit('0.')
               bnddir('YAJL');

      /copy yajl_h

       // entry parameters
       dcl-pi tsjson_entry extpgm('TSJSON01');
       end-pi;

       // get customers address details as JSON
       dcl-pr getCustomerAddress extpgm('TSJSON02');
         customerNumber int(10:0);
         p_buffer pointer;
         bufferlen int(10:0);
       end-pr;

       // --------------------------------------------------
       // standard web/json error data structure
       // --------------------------------------------------
       dcl-ds err qualified;
         bytesProv int(10:0) inz(0);
         bytesAvail int(10:0) inz(0);
       end-ds;

       // --------------------------------------------------
       // standard web/json fields
       // --------------------------------------------------
       dcl-s httpStatus packed(3:0);

       // work fields
       dcl-s p_buffer pointer;
       dcl-s bufferlen int(10:0);

       dcl-s customerNumber int(10) inz;
       dcl-s addressNode like(yajl_val);

       dcl-s errmsg varchar(500);
       dcl-s stmf varchar(5000) inz;
       dcl-s rc int(10) inz;

       // Code-----------------------------------------------------------
       customerNumber = 1;
       getCustomerAddress(customerNumber :p_buffer :bufferlen);
       addressNode = yajl_buf_load_tree(p_buffer :bufferlen :errMsg);

       yajl_genOpen(*on);

       yajl_beginObj();
       yajl_beginArray('customers');
       yajl_beginObj();
       yajl_addNum('number': %char(customerNumber));
       yajl_addChar('name': 'John');
       yajl_addChar('surname': 'Doe');
       rc = yajl_genFromNode('address'
                 :YAJL_object_find(addressNode: 'address'));
       yajl_endObj();
       yajl_endArray();
       yajl_endObj();

       yajl_tree_free(addressNode);

       httpStatus = 200;
       //yajl_writeStdout(httpStatus:errorDescription);
       stmf = '/dev/eddie.json';
       rc = yajl_saveBuf(stmf: errmsg);
       yajl_genClose();

       *inlr = *on;
       return;
But to simulate that the json_array has multiple customers, I moved the subprogram call further down the code as if there are 2 customers. The code fails at the 2nd yajl_beginObj on line 54. The revised main program is now called tsjson03 and the code is here

Code: Select all

       ctl-opt thread(*serialize)
               option(*srcstmt: *nodebugio: *noshowcpy)
               ccsid(*char:*jobrun)
               decedit('0.')
               bnddir('YAJL');

      /copy yajl_h

       // entry parameters
       dcl-pi tsjson_entry extpgm('TSJSON03');
       end-pi;

       // get customers address details as JSON
       dcl-pr getCustomerAddress extpgm('TSJSON02');
         customerNumber int(10:0);
         p_buffer pointer;
         bufferlen int(10:0);
       end-pr;

       // --------------------------------------------------
       // standard web/json error data structure
       // --------------------------------------------------
       dcl-ds err qualified;
         bytesProv int(10:0) inz(0);
         bytesAvail int(10:0) inz(0);
       end-ds;

       // --------------------------------------------------
       // standard web/json fields
       // --------------------------------------------------
       dcl-s httpStatus packed(3:0);

       // work fields
       dcl-s p_buffer pointer;
       dcl-s bufferlen int(10:0);

       dcl-s customerNumber int(10) inz;
       dcl-s addressNode like(yajl_val);

       dcl-s errmsg varchar(500);
       dcl-s stmf varchar(5000) inz;
       dcl-s rc int(10) inz;

       // Code-----------------------------------------------------------

       yajl_genOpen(*on);

       yajl_beginObj();
       yajl_beginArray('customers');

       customerNumber = 1;
       getCustomerAddress(customerNumber :p_buffer :bufferlen);
       addressNode = yajl_buf_load_tree(p_buffer :bufferlen :errMsg);
       yajl_beginObj();
       yajl_addNum('number': %char(customerNumber));
       yajl_addChar('name': 'John');
       yajl_addChar('surname': 'Doe');
       rc = yajl_genFromNode('address'
                 :YAJL_object_find(addressNode: 'address'));
       yajl_endObj();
       yajl_tree_free(addressNode);

       customerNumber = 2;
       getCustomerAddress(customerNumber :p_buffer :bufferlen);
       addressNode = yajl_buf_load_tree(p_buffer :bufferlen :errMsg);
       yajl_beginObj();
       yajl_addNum('number': %char(customerNumber));
       yajl_addChar('name': 'John');
       yajl_addChar('surname': 'Doe');
       rc = yajl_genFromNode('address'
                 :YAJL_object_find(addressNode: 'address'));
       yajl_endObj();
       yajl_tree_free(addressNode);

       yajl_endArray();
       yajl_endObj();

       httpStatus = 200;
       //yajl_writeStdout(httpStatus:errorDescription);
       stmf = '/dev/eddie.json';
       rc = yajl_saveBuf(stmf: errmsg);
       yajl_genClose();

       *inlr = *on;
       return;
Maybe this isn't the best way of coding this - you may have a much better technique. I hope this explains it a bit better. Eddie
Scott Klement
Site Admin
Posts: 635
Joined: Sun Jul 04, 2021 5:12 am

Re: Combining JSON

Post by Scott Klement »

I must admit, I don't like this design.
  • Each program depends on each other, so even though they are in separate programs, they act like one big routine because the subprogram expects the main program to be at a particular point in the document, and expects it to do things like deallocate the dynamic memory that it allocated.
  • You are assuming that the length of the buffer returned by yajl_getBuf will be the same length as the buffer returned by yajl_copyBuf. Since it is converting between character sets, I would not make that assumption.
  • You are forcing the data to EBCDIC, which is one thing I try to avoid as much as possible.
  • You are intermixing business logic (how to get customer addresses, how to determine flood scores, etc) together with "view" logic (i.e. how data is presented). For example, if you ever needed to support YAML or XML, you'd have to rewrite it, since the business logic code is mixed in with the JSON code, so it's only capable of working with JSON.
My preference would be to use data structures and/or arrays to pass data between the different programs/subprograms. Have the subprograms implement all of the business logic to populate the various arrays, and then just have one routine (possibly a separate subprogram) whose job is to format it in JSON format. That way, when you have a requirement to output it in a different format (XML, YAML or whatever other format comes along in the future), you only need to replace that one routine.
Scott Klement
Site Admin
Posts: 635
Joined: Sun Jul 04, 2021 5:12 am

Re: Combining JSON

Post by Scott Klement »

Having said that... if you decide that you still want to do it this way, I think your key mistake is thinking of each program as being completely separate.

i.e. you are thinking that the subprogram can call yajl_genOpen(), and that will generate a completely separate document from the main program also calling yajl_genOpen(). That'd make sense if these were opcodes rather than routines that you are calling -- but, they're not. They are subprocedures in the YAJLR4 service program. The generator doesn't actually exist in the main program or the subprogram, it exists in YAJLR4. So when you do yajl_genOpen() in the main program, you are starting the generator in YAJLR4, then when you call yajl_genOpen() again in the subprogram, you are once again telling it to start the generator in YAJLR4 -- which will wipe out what you have already done in the main program. When the subprogram calls yajl_genClose(), it will be closing the generator, so when you come back to the main program and it tries to add more data to the JSON document, the generator is already closed, and it will crash.

You can't think of these programs as being separate, because the logic isn't in your programs, it's in YAJLR4, and you are sharing the same copy of YAJLR4.

If you really wanted to do what you are attemping, though, you could simply NOT call yajl_genOpen() in the subprogram. Since the main program has already started up the generator, there's no need to start it again. When you call yajl_beginObj(), yajl_addChar(), etc, it will simply add the data to the existing JSON document that you've already begun in the main program. There's no need to call yajl_getBuf, yajl_copyBuf, yajl_genClose in the subprogram, it has added the data to the document itself, so it's already there -- no need to copy it to a string and then add that string into the document in the main program, it's already in the document -- Remember: the document isn't actually in your main program, it's in YAJLR4.

That eliminates the messy pointer logic, the multiple opens/closes of the generator, etc. As I said in my previous message, I don't really think this is a smart way to divide/modularize your workload. But, if you must do it this way, it's actually much easier to just continue the existing document than it would be to do it the way you are attempting.
eddiesmith
Posts: 3
Joined: Fri Jun 10, 2022 2:14 pm

Re: Combining JSON

Post by eddiesmith »

Scott,

Thank you very very much. Your explanation changed my way of thinking how to solve the problem, once you suggested that each sub program didn't need to have the yajl_genOpen it was like a light has been switched. I've changed the 'real' business code to use that technique and it is so cool.

Again thanks very much

Eddie Smith
Post Reply