DATA INTO - Memory Leak Issues
Posted: Wed Jan 12, 2022 3:51 pm
hi Scott,
I have a very simple program that will parse a JSON document using DATA-INTO and YAJLINTO. We receive the JSON request from other systems via MQ. We have a never ending job that will be reading the MQ. This job will run for months in the system and processes about about million requests per day . What we are seeing is that when the temp storage of the job is increasing every day.
Here in the sample program, I am running the job for 10,00 cycles in a loop. You will see that the memory of the job keeps on increasing. At the end of the job the job would have used about 27 MB of temp storage. In production, we are running with much larger json payload and over a million cycles per day. We are seeing the temp storage in GB.
I tried debugging it but was not able to point out the issue. Can you please help us identify the issue or if there is a way to clean up the memory after using YAJLINTO explicitly using a subprocedure.
Edit1: Few more data points.
1. We are using the TERASPACE version.
2. In the PEX report when TERASPACE = (*YES)
yajl_internal_free() function calls, there never appear to be any corresponding calls to _C_TS_free().
3. In the PEX report when TERASPACE = (*NO)
We see the standard C library free() functions in the QC2UTIL1 *SRVPGM from the QC2ALLOC.do_free_default__FPv() procedure.
Thanks in Advance !
I have a very simple program that will parse a JSON document using DATA-INTO and YAJLINTO. We receive the JSON request from other systems via MQ. We have a never ending job that will be reading the MQ. This job will run for months in the system and processes about about million requests per day . What we are seeing is that when the temp storage of the job is increasing every day.
Here in the sample program, I am running the job for 10,00 cycles in a loop. You will see that the memory of the job keeps on increasing. At the end of the job the job would have used about 27 MB of temp storage. In production, we are running with much larger json payload and over a million cycles per day. We are seeing the temp storage in GB.
I tried debugging it but was not able to point out the issue. Can you please help us identify the issue or if there is a way to clean up the memory after using YAJLINTO explicitly using a subprocedure.
Edit1: Few more data points.
1. We are using the TERASPACE version.
2. In the PEX report when TERASPACE = (*YES)
yajl_internal_free() function calls, there never appear to be any corresponding calls to _C_TS_free().
3. In the PEX report when TERASPACE = (*NO)
We see the standard C library free() functions in the QC2UTIL1 *SRVPGM from the QC2ALLOC.do_free_default__FPv() procedure.
Thanks in Advance !
Code: Select all
**free
ctl-opt DFTACTGRP(*NO) ACTGRP('SGP') BNDDIR('YAJL') DECEDIT('0.');
/define YAJL_C_PROTOTYPES
/include YAJL_H
/include IFSIO_H
/include ICONV_H
dcl-s json_string char(20000);
dcl-s errMsg varchar(500);
dcl-s i int(5) ;
dcl-s num_sample int(5);
dcl-ds sample qualified dim(005);
id char(30);
type char(30);
name char(30);
ppu char(9);
dcl-ds batters;
dcl-ds batter dim(100);
id char(30);
type char(30);
end-ds;
end-ds;
dcl-ds topping dim(100);
id char(30);
type char(30);
end-ds;
end-ds;
for i = 1 to 10000;
json_string = ' +
[{"id":"0001","type":"donut","name":"Cake","ppu":"0.55","batters": +
{"batter":[{"id":"1001","type":"Regular"},{"id":"1002","type": +
"Chocolate"},{"id":"1003","type":"Blueberry"},{"id":"1004","type": +
"DevilsFood"}]},"topping":[{"id":"5001","type":"None"}, +
{"id":"5002","type":"Glazed"},{"id":"5005","type":"Sugar"}, +
{"id":"5007","type":"PowderedSugar"},{"id":"5006","type" +
:"ChocolatewithSprinkles"},{"id":"5003","type":"Chocolate"},{"id": +
"5004","type":"Maple"}]},{"id":"0002","type":"donut","name":"Raised", +
"ppu":"0.55","batters":{"batter":[{"id":"1001","type":"Regular"}]}, +
"topping":[{"id":"5001","type":"None"},{"id":"5002","type":"Glazed"}, +
{"id":"5005","type":"Sugar"},{"id":"5003","type":"Chocolate"}, +
{"id":"5004","type":"Maple"}]},{"id":"0003","type":"donut","name": +
"OldFashioned","ppu":"0.55","batters":{"batter":[{"id":"1001","type": +
"Regular"},{"id":"1002","type":"Chocolate"}]},"topping":[{"id":"5001", +
"type":"None"},{"id":"5002","type":"Glazed"},{"id":"5003","type": +
"Chocolate"},{"id":"5004","type":"Maple"}]}]';
data-into sample %DATA(json_string
:'doc=string case=any allowmissing=yes +
allowextra=yes trim=none countprefix=num_')
%PARSER('YAJLINTO');
endfor;
*inlr = *on;