Page 1 of 1
Huge XML file processing from IFS (IBMi)
Posted: Tue Jul 25, 2023 3:21 pm
by pshankmd
Hi All,
Would like to get your thoughts on the situation below -
Need to process a huge XML (around 20MB size currently and could grow) from IFS folder using ILE RPG.
One option is to use RPG's built-in XML-INTO. Understand that there is a size limitation to use this function. Would this be a better option still?
Another option is to use the SQL capabilities. Came to know that it also has some limits. Not sure what are the limitations and how to get around these?
Any assistance on this would be appreciated.
Thanks!
Re: Huge XML file processing from IFS (IBMi)
Posted: Tue Jul 25, 2023 3:31 pm
by Scott Klement
I don't know what the SQL limits are (I prefer not to use SQL for this, I don't like how it works.)
For XML-INTO, you can use the %HANDLER BIF so that the entire XML document doesn't need to fit into a single data structure. That usually solves size limits...
If not, XML-SAX is more work to code, but allows you to work with more data and have a smaller memory footprint.
Finally, you could consider using Expat. Not many people have done that since XML-INTO and XML-SAX were added to the language, but if neither XML-INTO nor XML-SAX works for you, perhaps Expat would.
Re: Huge XML file processing from IFS (IBMi)
Posted: Tue Jul 25, 2023 7:07 pm
by pshankmd
Thanks for your inputs Scott.
Will try XML-INTO with %HANDLER first. Are there any samples to refer for this?
Re: Huge XML file processing from IFS (IBMi)
Posted: Tue Jul 25, 2023 8:48 pm
by Scott Klement
I don't know -- I basically never use XML anymore (I used it a lot 15 years ago... but...) Please use Google to see if you can find examples.
Re: Huge XML file processing from IFS (IBMi)
Posted: Tue Jul 25, 2023 9:37 pm
by pshankmd
Sure Scott, will check it, thanks!
Re: Huge XML file processing from IFS (IBMi)
Posted: Thu Jul 27, 2023 1:54 pm
by pshankmd
Trying to use XML-INTO command to get data from XML file from IFS and got below error, due to usage of CDATA.
XML Element: <PriceListID><![CDATA[1]]></PriceListID>
DS subfield declaration: PriceListID varchar(10);
Error Message: The XML document does not match the RPG variable; reason code 5.
Reason code 5. The XML document contains extra XML attributes or elements that do not match subfields.
Kindly suggest how to parse <![CDATA[...]], to get value 1
Re: Huge XML file processing from IFS (IBMi)
Posted: Thu Jul 27, 2023 10:30 pm
by Scott Klement
I don't understand -- why would that CDATA cause the error you've listed? That doesn't make sense.
Re: Huge XML file processing from IFS (IBMi)
Posted: Thu Jul 27, 2023 10:53 pm
by Scott Klement
I tried it -- the CDATA works just fine.
Code: Select all
**free
dcl-c myXml '<test><PriceListID><![CDATA[1]]></PriceListID></test>';
dcl-ds test;
PriceListId varchar(10);
end-ds;
xml-into test %xml(myXml: 'case=convert path=test');
dsply PriceListId;
*inlr = *on;
The problem is somewhere else, but since you have only told us about the CDATA and nothing else, we cannot possibly tell you what's causing the actual problem.
Re: Huge XML file processing from IFS (IBMi)
Posted: Fri Jul 28, 2023 9:05 pm
by pshankmd
Thank you Scott.
Looks like the issue was with DS declaration. Corrected it and now working fine with array of 29K elements.
Will change it to using %Handler and check it.