What's the best way to log JSON request and response?

Discussions relating to the ScottKlement.com port of the open source YAJL JSON Reader/Generator. This includes the YAJL tool as well as the YAJLR4, YAJLGEN, YAJLINTO and YAJLDTAGEN add-ons from ScottKlement.com. http://www.scottklement.com/yajl/
Post Reply
fauzyd2
Posts: 3
Joined: Sun Jan 14, 2024 9:11 am

What's the best way to log JSON request and response?

Post by fauzyd2 »

Hello everyone,

i used this code below to get string json request and response:

Code: Select all

D docNode         S                   Like(yajl_val)
D requestBody     S          16100A
D responseBody    S          16100A
 ...
docNode = yajl_stdin_load_tree(*on:error);
requestBody = yajl_StringifyStr(docNode);

yajl_genOpen(*Off);
  yajl_beginObj();
    ...
  yajl_endObj();
  ResponseBody = yajl_copyBufStr();
  yajl_writeStdout(stsCode:error);
yajl_genClose();
I found that these 2 procedures, yajl_StringifyStr and yajl_copyBufStr made the response time quite slow. For yajl_copyBufStr i have no idea, but for yajl_StringifyStr I try to use another job to handle this in batch, send docNode data to dataq, receive dataq with other job, and process log in that job. Code bellow:

Code: Select all

D docNode         S                   Like(yajl_val)
D dataDocnode     S          16100A   Based(docNode)
D requestBody     S          16100A
 ...
docNode = yajl_stdin_load_tree(*on:error);
SendDTAQ(dataqName:dataqLib:dataqLength:dataDocnode);
Other job to receive:

Code: Select all

D docNode         S                   Like(yajl_val)
D dataDocnode     S          16100A
D requestBody     S          16100A
 ...
ReceiveDTAQ(dataqName:dataqLib:dataqLength:dataDocnode:waitTime);

docNode = %Addr(dataDocnode);
requestBody = yajl_stringifyStr(docNode);
But this job got error on yajl_stringifyStr:

Code: Select all

Pointer not set for location referenced.                        
Unmonitored exception at statement 5223 of procedure GENFROMNODE
Any advice about this? Or how can I use these 2 procedures in separate jobs, but still be able to get the json request and response data that I want? Or what's the best way to log JSON request and response with quicker response time?

Thanks,
Fauzi
Scott Klement
Site Admin
Posts: 658
Joined: Sun Jul 04, 2021 5:12 am

Re: What's the best way to log JSON request and response?

Post by Scott Klement »

fauzyd2 wrote: Sun Jan 14, 2024 10:06 am Hello everyone,

i used this code below to get string json request and response:

Code: Select all

D docNode         S                   Like(yajl_val)
D requestBody     S          16100A
D responseBody    S          16100A
 ...
docNode = yajl_stdin_load_tree(*on:error);
requestBody = yajl_StringifyStr(docNode);
Do you understand what you're doing? You are taking the JSON document and asking YAJL to parse it. Once you've parsed it, you are telling it to generate a JSON document from the parsed data. This seems like a lot of work for the computer considering that you already had it in JSON format, that's how you parsed it in the first place.

It's sort of like saying "see this picture? I want you to cut it up into a 100,000 piece jigsaw puzzle. Very good! Now I want you to reassemble the puzzle because I need the picture." Err... but you already have the original picture, there's no need to do all that work...

The yajl_stdin_load_tree subprocedure has a parameter that lets you write the data to a file. That would be MUCH more efficient.

If you don't want it written to a file, you could read STDIN yourself, then you'd have it in a variable. Then you could call yajl_buf_load_tree to parse the JSON.
fauzyd2 wrote: Sun Jan 14, 2024 10:06 am I found that these 2 procedures, yajl_StringifyStr and yajl_copyBufStr made the response time quite slow. For yajl_copyBufStr i have no idea, but for yajl_StringifyStr I try to use another job to handle this in batch, send docNode data to dataq, receive dataq with other job, and process log in that job.
docNode is just a pointer to data that's located inside the YAJL service program. It won't be valid in another job, you'll just get a pointer error.

If you wanted to offload it to another job, you'd need to write the JSON data to a file, and pass the filename to the other job, then parse the JSON from that file the other job. I do not recommend putting JSON data directly in a data queue, JSON is not meant to be stored in EBCDIC, and data queues are not meant for Unicode data.

But I don't see the value of doing this. Since the data is already in memory as I explained earlier, this is just unnecessary work.

yajl_writeStdout() should be as efficient as it can reasonably be. You can use yajl_saveBuf to copy it to the IFS.

yajl_copyBuf is likely to be more efficient than yajl_copyBufStr (at the expense of being more complicated to use.) This assumes that you need the data in a variable and need that variable to be translated to EBCDIC for some purpose. If you don't need it translated, you could use yajl_getBuf to get a pointer to the internal buffer (which should require almost no work for the computer, so should be very fast.) This would give you a pointer to the data in UTF-8 format if that's useful to you.

Which routines you use really depend on what you need the data for, and you haven't explained that in technical terms. All you've told us is that you want to log it... it's not clear why or where it's being logged to. I would suggest storing the data in a stream file. If you must put it into a database, it should go into a CLOB (*not* a standard CHAR field.)
fauzyd2
Posts: 3
Joined: Sun Jan 14, 2024 9:11 am

Re: What's the best way to log JSON request and response?

Post by fauzyd2 »

Sorry for not clear before. I made a server using ibm http server, redirect client request with cgi to rpgle program. This server provide API for client to consume like Inquiry Account Info, etc. Code bellow is overview:

Code: Select all

FDBALLZ    IF   E           K Disk
FJSLOG     O    E           K Disk
*
 /include yajl_h
*
D timeStamp       DS            20    Qualified                  
D  yymd                          8  0 Overlay(timeStamp:1)       
D  hmsu                          8  0 Overlay(timeStamp:9)       
D  hmsu3                         9  0 Overlay(timeStamp:9)       
                                                                 
D dateIn          S              8  0                            
D timeIn          S              8  0                          

D docNode         S                   Like(yajl_val)
D node            S                   Like(yajl_val)

D requestBody     S          16100A
D responseBody    S          16100A

D benefActNo      S             34A
D benefAccName    S            100A
D benefAccSts     S             16A
D benefAccTyp     S              1A

/FREE
...
timeStamp = %Char(%TimeStamp():*ISO0);
dateIn   = timeStamp.yymd;            
timeIn   = timeStamp.hmsu;

docNode = yajl_stdin_load_tree(*on:error);
requestBody = yajl_StringifyStr(docNode);

node = yajl_object_find(docNode:'beneficiaryAccountNo');
benefActNo = yajl_get_string(node);
DMACCT =  benefActNo;
Chain DMACCT DBALLZ;
If (%Found);
  benefAccName = DMSNAM;
  benefAccTyp = DMAPCD;
  benefAccSts = DXTDSC;
EndIf;

yajl_genOpen(*Off);
  yajl_beginObj();
    yajl_addChar('beneficiaryAccountNo'     :%Trim(benefActNo));  
    yajl_addChar('beneficiaryAccountName'   :%Trim(benefAccName));
    yajl_addChar('beneficiaryAccountType'   :%Trim(benefAccTyp));
    yajl_addChar('beneficiaryAccountStatus' :%Trim(benefAccSts)); 
  yajl_endObj();
  ResponseBody = yajl_copyBufStr();
  yajl_writeStdout(stsCode:error);
yajl_genClose();

Clear RJSLOG;
  VJDATE = dateIn;
  VJTMIN = timeIn;
  VJRQST = requestBody;                 
  VJRPST = responseBody;                
  timeStamp = %Char(%TimeStamp():*ISO0);
  VJTMOU = timeStamp.hmsu;
Write RJSLOG;     
...
/END-FREE
I write this log to database, result like this:
Image

Image

So I want this process and write logs on database to be more efficient. Hopefully this explanation is clearer.

Thanks in advance.
Scott Klement
Site Admin
Posts: 658
Joined: Sun Jul 04, 2021 5:12 am

Re: What's the best way to log JSON request and response?

Post by Scott Klement »

And, again... I don't recommend doing that.

But, I feel like I've already explained how you can make your code more efficient.
fauzyd2
Posts: 3
Joined: Sun Jan 14, 2024 9:11 am

Re: What's the best way to log JSON request and response?

Post by fauzyd2 »

Thanks for your advice, Scott!

Yes, now my code is much faster than before :D
Post Reply