Need to return Large XML and JSON data over 16 meg

[gbaldcgi]

First please excuse my ignorance, I am new to this process.

I have a client that will be sending in large XML and JSON data (passed as a pointer) to my RPG program (up to 30 meg long). I need to send the data to the server and then the server will return a response that can be up to 40-50 meg long. These are details on insurance policies that are very detailed with 100's of vehicles and coverages all at one time. Using a pointer I can send the request using the following.

I do have the control option set in the program to use Teraspace: ctl-opt ALLOC(*TERASPACE);

Code: Select all

rc = http_url_post_xml(           
          %trim(url)              
         :POLEXTPTR               
         :POLEXTLEN               
         :%paddr(StartProc)       
         :%paddr(EndProc)         
         :*NULL                   
         :180                     
         :HTTP_USERAGENT          
         :'text/xml;charset=UTF-8'
         : %trim(soapaction));    

My issue is when the data is returned in the ENDPROC procedure, I am only getting the first 32767 bytes of the Response string tag in the XML response. I need to get the full response that could be as large as 50meg back and put it in a pointer that is allocated at the needed size for the full return string.

Code: Select all

 P EndProc         B                                             
 D EndProc         PI                                            
 D   userdata                      *   value                     
 D   depth                       10I 0 value                     
 D   name                      1024A   varying const             
 D   path                     24576A   varying const             
 D   value                         A   LEN(16000000)             
 D   attrs                         *   dim(32767)                
 D                                     const options(*varsize)   
     /free                                                       
                                                                 
       If (Name = 'ResponseValue');                              
          DSPLY %LEN(VALUE);                                     

in this case I see that the field value has a length of 16000000 but I still only see the first 32767 bytes of data in the field VALUE.

Does anyone have a solution to my problem? Any code examples of how to return more than the 32767 bytes back would be a big help for me.


Thanks in advance,

George

[Scott Klement]

I do have the control option set in the program to use Teraspace: ctl-opt ALLOC(*TERASPACE);

This changes how the ALLOC opcode in your program (not in HTTPAPI) works. It is not relevant to what you are doing with HTTPAPI.

My issue is when the data is returned in the ENDPROC procedure, I am only getting the first 32767 bytes of the Response string tag in the XML response. I need to get the full response that could be as large as 50meg back and put it in a pointer that is allocated at the needed size for the full return string.

Correct, the 'EndProc' procedure receives 32k of data in response.

Changing the size of the parameter in your program won't change that. All of the code that receives and processes the XML and gathers the response to send back to your program is still coded with that 32k limit. Just changing it in your program won't solve this problem, you'd have to change it throughout HTTPAPI.

HTTPAPI does have a method of handling elements larger than 32k, it is the http_XmlReturnPtr() functionality.

However:

1. Remember that the data returned here is for a single XML element, it is NOT the limit of the entire document.
2. Remember that this stuff was designed for fixed-format RPG in the early 2000s. Today we have built-in XML support in RPG. Have you considered using that instead?
3. That said, using the http_XmlReturnPtr functionality should work fine... you just have to be willing to work with pointers.

[gbaldcgi]

Scott,

Thanks for your response. The http_XmlReturnPtr(*ON) did work great and I can just pull data as needed from the pointer.

I have 1 additional question.

Is there a size limit that can be returned using this and the Value_T DS? I am on version V7R3M0 and I seen comment from you to another person referring to the http_XmlReturnPtr(*ON) and that there was a limit of 16meg. I will have some data as large as 30-40 meg being returned from the server sometimes.

URL to what I am refering to above: https://www.scottklement.com/archives/f ... 00110.html


Thanks again for your help,

George

[Scott Klement]

I believe the limit is about 2 gb currently.