SOAP call response parsing

[sbehera]

Hello, i am using HTTPAPI and making SOAP call, getting XML response back but not able to parse properly, please help.

Full response from server attached below but i am interested to capture error messages only from XML tag <DatErrMsg>" and seperate values of MsgCIgd, Dsc and DtlDsc. Running through debug, able to see variable act value is 3, that means it is able to recognize 3 occurrences of name <DatErrMsg> but the path it is reading is "/SOAP-ENV:Envelope/SOAP-ENV:Body/tns:shipmentValidateAndImageArchiveResp/MSG/DatTrErr " instead reading "<ErrMsgDtl MsgCIgd="200001" MsgClg="IPESB" Dsc="Shipment details were validated successfully" DtlDsc="Shipment details were validated successfully"></ErrMsgDtl>" and value returning is blank.

Variable values:
Act = 3
A_NAME(1) = DatErrMsg
A_NAME(2) = DatErrMsg
A_NAME(3) = DatErrMsg
A_VAL(1) =
A_VAL(2) =
A_VAL(3) =
A_PATH(1) = /SOAP-ENV:Envelope/SOAP-ENV:Body/tns:shipmentValidateAndImageArchiveResp/MSG/DatTrErr
A_PATH(2) = /SOAP-ENV:Envelope/SOAP-ENV:Body/tns:shipmentValidateAndImageArchiveResp/MSG/DatTrErr
A_PATH(3) = /SOAP-ENV:Envelope/SOAP-ENV:Body/tns:shipmentValidateAndImageArchiveResp/MSG/DatTrErr



Giving only part of the code which is relevant to diagnosis the issue:
rc = http_url_post_xml(%Trim(I1_Url)
: %addr(Soap) + 2
: %len(Soap)
: *NULL
: %paddr(IncomingData)
: *NULL
: HTTP_TIMEOUT
: *omit
: 'application/soap+xml; ' +
'charset=UTF-8; action="DocumentManagem' +
'ent_WSConsumer_WS_DocumentManagement_Bind' +
'er_ShipmentValidateAndImageArchive"');

P E

P IncomingData B
D IncomingData PI
D UserData * value
D depth 10I 0 value
D name 1024A varying const
D path 24576A varying const
D value 65535A varying const
D attrs * dim(32767)
D const options(*varsize)
/free

select;
when name = 'DatErrMsg';
act = act + 1;
A_Val(act) = %Trim(value);
A_Name(act) = %Trim(name);
A_Path(act) = %Trim(path);
Other;
endsl;
/end-free
P E

Debug Log: Respose back:
recvdoc(): Receiving 3023 bytes.
<?xml version="1.0" encoding="UTF-8"?>
<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<SOAP-ENV:Header xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelop ... P-ENV:Body>
<tns:shipmentValidateAndImageArchiveResp xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:tns="DocumentManagement/WSConsumer/WS/DocumentManagement">
<MSG>
<Hdr Id="XXXXXXX" Ver="1.038" Dtm="2022-01-07T20:53:32">
<Sndr AppCd="ESB" AppNm="ESB"></Sndr>
<Rcp AppCd="99951" AppNm="Test_Shipping">
<GId Id="1" IdTp="RQMSGID"></GId>
</Rcp>
</Hdr>
<Bd>
<Shp Id="XXXXXXXXX">
<ShpTr CstValCurCd="USD" CstVal="50.25" MgNProdCd="P" OrgFcCd="PHX" OrgSrvaCd="PHX">
<SCDtl CRlTyCd="SP" CtryCd="US"></SCDtl>
<SCDtl CRlTyCd="RV" CtryCd="NO"></SCDtl>
</ShpTr>
<ShpInDoc>
<SDoc DocId="1" DocTyCd="CIN">
<Img ImgID="1" GenDtm="2022-01-06T15:10:00" SrcAppNm="Test_Shipping" ImgEncdTy="PDF" ImgMimeTy="application/pdf"></Img>
</SDoc>
</ShpInDoc>
<ShpInDoc>
<SDoc DocId="2" DocTyCd="HWB">
<Img ImgID="2" GenDtm="2022-01-06T15:10:00" SrcAppNm="Test_Shipping" ImgEncdTy="PDF" ImgMimeTy="application/pdf"></Img>
</SDoc>
</ShpInDoc>
</Shp>
</Bd>
<DatTrErr>
<OrgHdr CorrId="1"></OrgHdr>
<Rp CmptNm="DocumentManagement.ShipmentValidateAndImageArchiveProvider"></Rp>
<ErrMsgGI Ty="CF" Pri="3" Sev="10"></ErrMsgGI>
<DatErrMsg>
<ErrMsgDtl MsgCIgd="200001" MsgClg="IPESB" Dsc="Shipment details were validated successfully" DtlDsc="Shipment details were validated successfully"></ErrMsgDtl>
</DatErrMsg>
</DatTrErr>
<DatTrErr>
<OrgH dr CorrId="1"></OrgHdr>
<Rp CmptNm="DocumentManagement.ShipmentValidateAndImageArchiveProvider"></Rp>
<ErrMsgGI Ty="CF" Pri="3" Sev="10"></ErrMsgGI>
<DatErrMsg>
<ErrMsgDtl MsgCIgd="200006" MsgClg="IPESB" Dsc="Document image archive request was processed to image archive successfully" DtlDsc="Image: AM1/Test_Shipping.US.P.1109313192_INV_PHX_PHX_YR8_20220106_151010.PDF is successfully archived"></ErrMsgDtl>
</DatErrMsg>
</DatTrErr>
<DatTrErr>
<OrgHdr CorrId="1"></OrgHdr>
<Rp CmptNm="DocumentManagement.ShipmentValidateAndImageArchiveProvider"></Rp>
<ErrMsgGI Ty="CF" Pri="2" Sev="10"></ErrMsgGI>
<DatErrMsg>
<ErrMsgDtl MsgCIgd="100000" MsgClg="IPESB" Dsc="Service operation execution finished successfully" DtlDsc="Service operation execution finished successfully"></ErrMsgDtl>
</DatErrMsg>
</DatTrErr>
</MSG>
</tns:shipmentValidateAndImageArchiveResp>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>

http_close(): entered

[Scott Klement]

Hello,

The MsgCldd, Dsc and DtlDsc values you are trying to retrieve appear to attributes of the ErrMsgDtl tag

Code: Select all

<ErrMsgDtl MsgCIgd="200001" MsgClg="IPESB" Dsc="Shipment details were validated successfully" DtlDsc="Shipment details were validated successfully">

I think if you tried to get them from attributes of the ErrMsgDtl tag rather than the path of the DatErrMsg tag you'd have much more success.

[sbehera]

You are obsoletely right, did not think/know what is use of attrs parameter. Now added call to http_nextXmlAttr to read attributes and getting the messages as desired. You rock, Scott... thank you very much for help...

Before your response i did write/changed to call http_url_post instead of http_url_post_xml and had the response write to stream file with 4th parameter ifs file name. And was writing code to directly read/parse the file. For this i was using your IFSEBOOK routine Read(fd:%addr(Input):%size(Input)) which reads whole file in one read to variable Input. Then was having issue how to break down into line by line. The response XML i thought has carriage return after each tag and was using %scan to look for (D CRLF C CONST(X'0d25')) but scan was not able to detect CRLF character in the Input string. May be if you can give some light how to break down whole response document into tag by tag might help for future understanding/use?

[Scott Klement]

May be if you can give some light how to break down whole response document into tag by tag might help for future understanding/use?

I would suggest using the RPG XML-INTO opcode unless you have a good reason not to. Most beginners will find this easier to use than the (much older) XML routines in HTTPAPI. That's why I didn't include XML parsing support in the newer HTTP request routines such as http_string() or http_req()... most people today will just retrieve the XML (or more commonly today, people will use JSON) into a string variable. Then they can pass that variable to XML-INTO (or DATA-INTO, etc.)

You could also consider using SQL's XML or JSON routines instead. Some people prefer those. (I do not like them -- but, everyone is different.)

But, if you do want to use HTTPAPI's XML routines, the way they work is:

1. They loop through all of the XML tags in the document, and call your subprocedures. Sometimes this is hard for people to understand, but, you do not code the loop or even see it in your code -- it is inside of the EXPAT tool that HTTPAPI uses. It loops through every single XML tag from the start of the document to the end.
2. It calls the "start" subprocedure for each opening XML tag. For example, if you have <myTag attr1="x" attr2="y">(other stuff)</myTag> it will call the start subprocedure as soon as it has read <myTag attr1="x" attr2="y"> from the document. It will pass the tag name, the attributes, and the "path" (which is a list of other XML tags that myTag is inside of, if any.)
3. It then continues proceding the next tags that it finds, etc. So the "(other stuff)" in the above example.
4. When it gets to an end tag, such as </myTag> it calls the "end" subprocedure. That's what you've coded -- you called it "incoming" in your example. Once it again it passes the attributes (which it remembered from when it handled the start tag) and it also passes any non-tag data that was in between the starting/ending tag in a parameter that I call "value"
5. It keeps looping through all of the tags, calling the "start" subprocedure and "end" subprocedure repeatedly, once for each start/end tag, respectively, until it has processed the entire document.

So to write a subprocedure similar to your "incoming" one, you have to think in terms of being in the middle of the loop and being called many times, and then you have to figure out how to determine what data you will need for the various tags you want to handle, and how you should store it.

[sbehera]

I am trying to use http_string to get returned value to a variable then seperate desired tags using xml-into
(not sure difference between xml-into and data-into, if you can tell difference and which is appropriate for
this case):

Question:
1. How to use xml-into/data-into to parse specific field value? In this case i need to read/seperate every thing after tag metadata, which is the value
back from server, any vaue before that is what i have passed in my call.
2. When http_string used, how to detect when there is http call error?
(when rc = http_url_post_xml is used, rc value <> 1 means some kind of error)
3. http_error() returns value "HTTP/1.1 201 Created" but the server i am calling they claim there is no error
4. Why responses back is broken into 2 chunks?

Code:
result = http_string( 'POST': url: postdata: 'application/json');
xml-into tagvalue %xml(result: DO NOT KNOW WHAT TO PASS, i have several values to read from response including base64 encoded label field);
scmsg = http_error();

Using debug, do see variable "result" has every thing returned from server (even though in log it shows response
was received in 2 chunks, first chunk 3599 and second chunk 309 - not sure why it is in chunks?)

Response part of HTTP Log:
X-Runtime: 0.288755
Vary: Origin
Strict-Transport-Security: max-age=15724800; includeSubDomains


SetError() #13: HTTP/1.1 201 Created
recvresp(): end with 201
recvdoc parms: chunked 0
header_load_cookies() entered
recvchunk(): entered
get_chunk_size(): entered
e0f

chunk size = 3599
get_chunk_size returned 3599
calling comm_blockread
{"reference":"ab0d64f1-84","request_id":null,"order_number":null,"weight":2.0,"pricing_weight":2.0,"pricing_cubic_ft":null,"weight_unit":"oz",
"dimensions":null,"dimensions_unit":null,"value":0.0,"postmark_date":"2022-01-20T15:49:58.094-08:00","status":"created","postage_amount":3.37,
"fees_amount":0.0,"discounted_amount":0.0,"total_amount":3.37,"estimated_delivery_days":2.0,"from_address":{"first_name":"fn","middle_name":null,
"last_name":"ln","company_name":"XYZ","line1":"Ads1","line2":null,"line3":null,"city":"cityname","state_province":"ST","postal_code":"ZIP",
"phone_number":"12345","email":"emailid","sms":"12345","iso_country_code":"US"},"to_address":{"first_name":"tfn","middle_name":null,"last_name":
"tln","company_name":"Product Returns Warehouse","line1":"ral1","line2":null,"line3":null,"city":"tocity","state_province":"st","postal_code":
"123","phone_number":"12345","email":"temid","sms":null,"iso_country_code":"US"},"metadata":[],"usps":{"mail_class":"FirstClass","fees":
[{"name":"USPS Tracking","fee":0.0}],"pricing":"Commercial Base","zone":2,"country_group":null,"canada_zone":null,"tracking_numbers":
["9200100000000000717609"]},"refund_detail":null,"manifest_detail":null,"return_detail":null,"user":{"billing_type":"meter","account_type":"epostage",
"account_balance":875.02},"base64_labels":["XlhBCl5M
comm_blockread returned 3599


get_chunk_size(): entered
135

chunk size = 309
get_chunk_size returned 309
calling comm_blockread
jksMTEzMl5YWg=="],"carrier":"usps","hold_for_pickup_post_office_address":null,"adjustments":[],"is_return":false,"shipping_status":null,"promotion":null,
"customs_form":null}
comm_blockread returned 309


get_chunk_size(): entered
0

chunk size = 0
get_chunk_size returned 0
SetError() #13: HTTP/1.1 201 Created
http_close(): entered

[Scott Klement]

I am trying to use http_string to get returned value to a variable then seperate desired tags using xml-into
(not sure difference between xml-into and data-into, if you can tell difference and which is appropriate for
this case):

XML-INTO is an RPG opcode released many years ago for reading XML documents in RPG. (At the same time, they also introduced XML-SAX, a different approach to XML parsing.)

DATA-INTO was released much more recently because RPG programmers were asking IBM for a way of reading other documents (aside from XML.) DATA-INTO must work together with a 3rd-party tool called a "parser" that understands the format of the document you are working with. If you find (or write) a parser for XML, then DATA-INTO would work the same as XML-INTO. But, if you find a parser for JSON, DATA-INTO will work with JSON, if you find one for YAML, it will read YAML, etc. If you don't have a parser, DATA-INTO can do nothing. XML-INTO only works with XML, and requires no additional parser.

Think of DATA-INTO's parser the way you would think of a printer driver. All of your PC programs can work with any printer -- it doesn't matter how that printer works -- as long as you have a driver for it. The driver understands how the printer works, and interfaces Windows (or macOs, Linux, whatever you use) with that specific printer. Without any driver, you can't print at all.

The same is true for a DATA-INTO parser... The parser interfaces with a particular document type. Without a parser, you can't use DATA-INTO at all.

If it helps, you can think of XML-INTO as the same as DATA-INTO, except it has a built-in XML parser, so there's no need to install a separate parser.

I'm probably over-explaining.

Question:
1. How to use xml-into/data-into to parse specific field value? In this case i need to read/seperate every thing after tag metadata, which is the value
back from server, any vaue before that is what i have passed in my call.

You can use the path= option to go directly to a specific field if you don't need to read the entire document.

2. When http_string used, how to detect when there is http call error?
(when rc = http_url_post_xml is used, rc value <> 1 means some kind of error)

http_string() will return an exception to your program if there is an error. For example:

Code: Select all

monitor;
   result = http_string(...parameters here...);
on-error;
   // something went wrong.
   message = http_error();
end-mon;

Alternately, you can use http_req() instead of http_string() which returns a 'return code' (rc) similar to http_url_post_xml

Code: Select all

3. http_error() returns value "HTTP/1.1 201 Created" but the server i am calling they claim there is no error

Correct. Any number in the range of 200-299 indicates success. It is not only the number 1.

Code: Select all

4. Why responses back is broken into 2 chunks?

You mean that you see "chunked" transfer-coding in the HTTP diagnostic log (produced by http_debug)?? That is one of the ways that the HTTP network protocol sends data. It can send it all at once ("content-length" coding) or it can stream the data, i.e. send it as it is generated by the program... in the latter case the HTTP server doesn't know the total length of the data when it starts sending it, so it sends as much as it has, together with a length, then sends the next part, together with a length, until it is finished, and that is called "chunked" transfer coding.

However, you should not need to worry about how that all works, HTTPAPI takes care of it putting it all together into one string for you. (Unless you are using one of the "raw" methods, but I strongly discourage you from doing that.)

Code:
result = http_string( 'POST': url: postdata: 'application/json');
xml-into tagvalue %xml(result: DO NOT KNOW WHAT TO PASS, i have several values to read from response including base64 encoded label field);
scmsg = http_error();

If you search the Internet for XML-INTO you will find many resources that will teach you how to use XML-INTO.

{"reference":"ab0d64f1-84","request_id":null,"order_number":null,"weight":2.0,"pricing_weight":2.0,"pricing_cubic_ft":null,"weight_unit":"oz",
"dimensions":null,"dimensions_unit":null,"value":0.0,"postmark_date":"2022-01-20T15:49:58.094-08:00","status":"created","postage_amount":3.37,
"fees_amount":0.0,"discounted_amount":0.0,"total_amount":3.37,"estimated_delivery_days":2.0,"from_address":{"first_name":"fn","middle_name":null,
"last_name":"ln","company_name":"XYZ","line1":"Ads1","line2":null,"line3":null,"city":"cityname","state_province":"ST","postal_code":"ZIP",
"phone_number":"12345","email":"emailid","sms":"12345","iso_country_code":"US"},"to_address":{"first_name":"tfn","middle_name":null,"last_name":
"tln","company_name":"Product Returns Warehouse","line1":"ral1","line2":null,"line3":null,"city":"tocity","state_province":"st","postal_code":
"123","phone_number":"12345","email":"temid","sms":null,"iso_country_code":"US"},"metadata":[],"usps":{"mail_class":"FirstClass","fees":
[{"name":"USPS Tracking","fee":0.0}],"pricing":"Commercial Base","zone":2,"country_group":null,"canada_zone":null,"tracking_numbers":
["9200100000000000717609"]},"refund_detail":null,"manifest_detail":null,"return_detail":null,"user":{"billing_type":"meter","account_type":"epostage",
"account_balance":875.02},"base64_labels":["XlhBCl5M
comm_blockread returned 3599

That's not XML!!! Your data is a JSON document, and isn't XML or SOAP. Why are you asking me how to use XML-INTO if you don't have an XML document?!?!

For a JSON document like this, I would recommend using DATA-INTO instead of XML-INTO. I have an open source JSON parser for DATA-INTO called YAJL (you will see discussions about it in other areas of this forum). See https://www.scottklement.com/yajl/ and the presentations about JSON under https://www.scottklement.com/presentations/