Need to return Large XML and JSON data over 16 meg

[gbaldcgi]

First please excuse my ignorance, I am new to this process.

I have a client that will be sending in large XML and JSON data (passed as a pointer) to my RPG program (up to 30 meg long). I need to send the data to the server and then the server will return a response that can be up to 40-50 meg long. These are details on insurance policies that are very detailed with 100's of vehicles and coverages all at one time. Using a pointer I can send the request using the following.

I do have the control option set in the program to use Teraspace: ctl-opt ALLOC(*TERASPACE);

Code: Select all

rc = http_url_post_xml(           
          %trim(url)              
         :POLEXTPTR               
         :POLEXTLEN               
         :%paddr(StartProc)       
         :%paddr(EndProc)         
         :*NULL                   
         :180                     
         :HTTP_USERAGENT          
         :'text/xml;charset=UTF-8'
         : %trim(soapaction));    

My issue is when the data is returned in the ENDPROC procedure, I am only getting the first 32767 bytes of the Response string tag in the XML response. I need to get the full response that could be as large as 50meg back and put it in a pointer that is allocated at the needed size for the full return string.

Code: Select all

 P EndProc         B                                             
 D EndProc         PI                                            
 D   userdata                      *   value                     
 D   depth                       10I 0 value                     
 D   name                      1024A   varying const             
 D   path                     24576A   varying const             
 D   value                         A   LEN(16000000)             
 D   attrs                         *   dim(32767)                
 D                                     const options(*varsize)   
     /free                                                       
                                                                 
       If (Name = 'ResponseValue');                              
          DSPLY %LEN(VALUE);                                     

in this case I see that the field value has a length of 16000000 but I still only see the first 32767 bytes of data in the field VALUE.

Does anyone have a solution to my problem? Any code examples of how to return more than the 32767 bytes back would be a big help for me.


Thanks in advance,

George

[Scott Klement]

I do have the control option set in the program to use Teraspace: ctl-opt ALLOC(*TERASPACE);

This changes how the ALLOC opcode in your program (not in HTTPAPI) works. It is not relevant to what you are doing with HTTPAPI.

My issue is when the data is returned in the ENDPROC procedure, I am only getting the first 32767 bytes of the Response string tag in the XML response. I need to get the full response that could be as large as 50meg back and put it in a pointer that is allocated at the needed size for the full return string.

Correct, the 'EndProc' procedure receives 32k of data in response.

Changing the size of the parameter in your program won't change that. All of the code that receives and processes the XML and gathers the response to send back to your program is still coded with that 32k limit. Just changing it in your program won't solve this problem, you'd have to change it throughout HTTPAPI.

HTTPAPI does have a method of handling elements larger than 32k, it is the http_XmlReturnPtr() functionality.

However:

1. Remember that the data returned here is for a single XML element, it is NOT the limit of the entire document.
2. Remember that this stuff was designed for fixed-format RPG in the early 2000s. Today we have built-in XML support in RPG. Have you considered using that instead?
3. That said, using the http_XmlReturnPtr functionality should work fine... you just have to be willing to work with pointers.

[gbaldcgi]

Scott,

Thanks for your response. The http_XmlReturnPtr(*ON) did work great and I can just pull data as needed from the pointer.

I have 1 additional question.

Is there a size limit that can be returned using this and the Value_T DS? I am on version V7R3M0 and I seen comment from you to another person referring to the http_XmlReturnPtr(*ON) and that there was a limit of 16meg. I will have some data as large as 30-40 meg being returned from the server sometimes.

URL to what I am refering to above: https://www.scottklement.com/archives/f ... 00110.html


Thanks again for your help,

George

[Scott Klement]

I believe the limit is about 2 gb currently.

[gbaldcgi]

Hello Scott,

I was able to use the above for XML and now they are asking me to do the same with large JSON strings (up to 30 meg long). The JSON string will be passed in as a pointer to my program. All I need to do is send the request (as a pointer) with a TOKEN (as a header) then return the full response (also up to 30 meg) into a pointer and then pass the pointer back to the calling program in which we will parse out the data as needed. I am guessing I can use the http_url_post_raw to do this but not sure what is best to use?

I notice the http_url_post_raw uses a ENDPROC that returns only 65535 bytes. Is there a way to get it to return the full response into a pointer?

Here is my ENDPROC definition and process: Is this correct way to do it?

P EndProc B
D EndProc PI 10i 0
D fd 10i 0 value
D retdata 65535a options(*varsize)
D RetLen 10i 0 value


/free

// Make sure we don't overflow the string:
retlen = (nextpos + datalen) - 1;
if retlen > %size(retdata);
datalen=datalen-(retlen-%size(retdata));
endif;

// If there is nothing to write, return THAT...
if datalen < 1;
return 0;
endif;

// Here we add any data sent to the end of our 'retdata' string:
%subst(retdata: nextpos) =
%subst(data:1:datalen);
nextpos = nextpos + datalen;

Any recommendations and examples you have on how to handle this would be greatly appreciated. If there is a better way to receive the full response back from the server into a pointer directly, please let me know.

Thanks in advance for your help.

George

[Scott Klement]

Sorry, I don't understand. Why is this subprocedure called "endproc"? What is it the end of?

http_url_post_raw() is something I don't recommend using today. It is meant for doing all the logic to handle the data in your program (i.e. it is the raw response.) Instead, please consider using http_req().

If you MUST process the raw response, the receive procedure (I don't know why it'd be called 'endproc') is called repeatedly, each time providing one chunk of the data. The length can not be predicted, but it will never exceed 32k. The purpose of the procedure is to save the data as it arrives over the network, So you will get whatever has been received from the network, then you will be called again with whatever has been received from the network, etc. It is your job to accumulate it. There is no size limit because the parameter does not represent the document in it's entirety.

By contrast, http_req() will retrieve everything in one call. It returns the entire document all at once, sparing you the responsibility to accumulate or translate it.

http_req() was introduced more than 10 years ago. Please use it.