Google API json response is unable to be pulled into rpgle data structure using DATA-INTO (Status Code 00356)

[JBdev]

I graduated from college about 2 months ago and landed a role as a junior rpg developer, and I'm having some issues with the project I've been assigned. I'm sorry if I forget to add any relevant information to my post - I've never worked with anything this complex or made a forum post like this before.

I'm using http_string() to submit a POST request to a Google address validation API, but I'm having trouble parsing the json response into the data structure I've defined. The structure definition corresponds to Google's documentation for the json response body (https://developers.google.com/maps/docu ... ateAddress), but parsing the json fails every time with reason code 5: "The document for the data-into operation does not match the rpg variable". Currently, I've tried to avoid having any data type incompatibilities between the json and the DS by just making every field a CHAR type, but I've also tried versions where I used ind for boolean, zoned for double, int for integer, and so on.

I've included the debug log, along with the data structure and the snippet where I'm trying to parse the json response below. I'm using the 21 Jan 2022 version of YAJL, and version 1.49 of HTTPAPI. I sincerely appreciate any help the community is able to offer.

HTTPAPI Debug Log:
HTTPAPI Ver 1.49 released 2024-04-16
NTLM Ver 1.4.0 released 2014-12-22
OS/400 Ver V7R5M0

http_setauth(): entered
New iconv() objects set, PostRem=1208. PostLoc=0. ProtRem=819. ProtLoc=0
http_persist_open(): entered
http_long_ParseURL(): entered
DNS resolver retrans: 2
DNS resolver retry : 2
DNS resolver options: x'00000136'
DNS default domain: ISSICONSULTING.LOCAL
DNS server found: 192.158.1.235
Nagle's algorithm (TCP_NODELAY) disabled.
SNI hostname set to: addressvalidation.googleapis.com
-------------------------------------------------------------------------------------
Dump of server-side certificate information:
-------------------------------------------------------------------------------------
Cert Validation Code = 6000
-----BEGIN CERTIFICATE-----
[certificate]
-----END CERTIFICATE-----
Serial Number: 00:A6:AC:EA:B2:5D:AF:FF:81:10:5A:8E:CB:10:2C:A5:D4
Common Name: upload.video.google.com
Issuer CN: WR2
Issuer Country: US
Issuer Org: Google Trust Services
Version: 3
not before: 20240613093233
Unknown Field: 09:32:33 13-06-2024
not after: 20240905093232
Unknown Field: 09:32:32 05-09-2024
pub key alg: 1.2.840.10045.2.1
signature algorithm: 1.2.840.113549.1.1.11
Unknown Field: 0342000460CDA692794B40AD7F7E81F464DB258639CCC64E6182CF5F9F4E9110A4731D1743ED8286D13123E0924DDE49E2957B2F9438DC85B1DD29A5F2A9A07A47890A00
Unknown Field: 256
Unknown Field: F97FB483DA38D1C7F07BC28BDFC14B51
Unknown Field: 1.2.840.113549.2.5
Unknown Field: C712523ABDE07320ADA85FDF12DBC6DEAF63886B
Unknown Field: D52E9DB3DAFA7EB63FFBFE44DAB909DBFA4097D54154694C8B0874C98FC55159
Unknown Field: 1
Unknown Field: bg-call-donation-dev.goog
Unknown Field: bg-call-donation-canary.goog
Unknown Field: bg-call-donation-alpha.goog
Unknown Field: bg-call-donation.goog
Unknown Field: uploads.stage.gdata.youtube.com
Unknown Field: *.upload.youtube.com
Unknown Field: upload.youtube.com
Unknown Field: *.upload.google.com
Unknown Field: upload.google.com
Unknown Field: *.youtube-3rd-party.com
Unknown Field: *.photos.google.com
Unknown Field: *.googleapis.com
Unknown Field: *.gdata.youtube.com
Unknown Field: *.drive.google.com
Unknown Field: *.docs.google.com
Unknown Field: *.clients.google.com
Unknown Field: upload.video.google.com
Unknown Field: 0
Unknown Field: 1.3.6.1.5.5.7.3.1
Unknown Field: 2.23.140.1.2.1
Unknown Field: http://c.pki.goog/wr2/75r4ZyA3vA0.crl
Unknown Field: http://i.pki.goog/wr2.crt
Unknown Field: http://o.pki.goog/wr2

Protocol Used: TLS Version 1.3
http_persist_req(POST) entered.
http_long_ParseURL(): entered
http_long_ParseURL(): entered
do_oper(POST): entered
There are 0 cookies in the cache
POST /v1:validateAddress?key=[api-key] HTTP/1.1
Host: addressvalidation.googleapis.com
User-Agent: http-api/1.48
Content-Type: application/json
Content-Length: 147
Authorization: Basic YXBpa2V5OkFJemFTeUR5ZmhtVGRxWUhRbkZLaUdIbjNBMEtNRlp5Y0pLZWl6Zw==


senddoc(): entered
{"address": {"addressLines": "DUNK WESRERSER 23423, D, , CITYJP, AS 5555555554, UK"}, "previousResponseId": " "}
recvresp(): entered
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Vary: X-Origin
Vary: Referer
Date: Wed, 03 Jul 2024 18:33:28 GMT
Server: scaffolding on HTTPServer2
Cache-Control: private
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Accept-Ranges: none
Vary: Origin,Accept-Encoding
Transfer-Encoding: chunked


SetError() #13: HTTP/1.1 200 OK
recvresp(): end with 200
recvdoc parms: chunked 0
header_load_cookies() entered
recvchunk(): entered
get_chunk_size(): entered
bc8

chunk size = 3016
get_chunk_size returned 3016
calling comm_blockread

Code: Select all

{
  "result": {
    "verdict": {
      "inputGranularity": "SUB_PREMISE",
      "validationGranularity": "OTHER",
      "geocodeGranularity": "OTHER",
      "hasUnconfirmedComponents": true
    },
    "address": {
      "formattedAddress": "WESRERSER 23423, D, AS, DUNK, UK",
      "postalAddress": {
        "regionCode": "GB",
        "languageCode": "en-GB",
        "locality": "DUNK",
        "addressLines": [
          "WESRERSER 23423, D",
          "AS"
        ]
      },
      "addressComponents": [
        {
          "componentName": {
            "text": "DUNK",
            "languageCode": "en"
          },
          "componentType": "postal_town",
          "confirmationLevel": "UNCONFIRMED_BUT_PLAUSIBLE",
          "unexpected": true
        },
        {
          "componentName": {
            "text": "WESRERSER 23423",
            "languageCode": "en"
          },
          "componentType": "subpremise",
          "confirmationLevel": "UNCONFIRMED_BUT_PLAUSIBLE"
        },
        {
          "componentName": {
            "text": "D",
            "languageCode": "en"
          },
          "componentType": "premise",
          "confirmationLevel": "UNCONFIRMED_BUT_PLAUSIBLE"
        },
        {
          "componentName": {
            "text": "CITYJP",
            "languageCode": "en"
          },
          "componentType": "postal_town",
          "confirmationLevel": "UNCONFIRMED_BUT_PLAUSIBLE",
          "unexpected": true
        },
        {
          "componentName": {
            "text": "AS",
            "languageCode": "en"
          },
          "componentType": "premise",
          "confirmationLevel": "UNCONFIRMED_BUT_PLAUSIBLE"
        },
        {
          "componentName": {
            "text": "5555555554",
            "languageCode": "en"
          },
          "componentType": "postal_town",
          "confirmationLevel": "UNCONFIRMED_BUT_PLAUSIBLE",
          "unexpected": true
        },
        {
          "componentName": {
            "text": "UK",
            "languageCode": "en"
          },
          "componentType": "country",
          "confirmationLevel": "CONFIRMED"
        }
      ],
      "missingComponentTypes": [
        "postal_code"
      ],
      "unconfirmedComponentTypes": [
        "postal_town",
        "subpremise",
        "premise",
        "postal_town",
        "premise",
        "postal_town"
      ]
    },
    "geocode": {
      "location": {
        "latitude": 55.378051,
        "longitude": -3.435973
      },
      "plusCode": {
        "globalCode": "9C7R9HH7+6J"
      },
      "bounds": {
        "low": {
          "latitude": 34.5614,
          "longitude": -8.8988999
        },
        "high": {
          "latitude": 60.9157,
          "longitude": 33.9165549
        }
      },
      "featureSizeMeters": 20000,
      "placeId": "ChIJqZHHQhE7WgIReiWIMkOg-MQ",
      "placeTypes": [
        "country",
        "political"
      ]
    }
  },
  "responseId": "fa3da8c4-a829-4aea-bec3-860ae659bc84"
}

comm_blockread returned 3016


get_chunk_size(): entered
0

chunk size = 0
get_chunk_size returned 0
http_close(): entered

The DS I'm trying to pull the json response into:

Code: Select all

dcl-ds jsonResponse qualified inz;
        dcl-ds result;
            dcl-ds verdict;
                inputGranularity char(20);
                validationGranularity char(20);
                geocodeGranularity char(20);
                addressComplete char(5);
                hasUnconfirmedComponents char(5);
                hasInferredComponents char(5);
                hasReplacedComponents char(5);
            end-Ds;
            dcl-ds address;
                formattedAddress char(256);
                dcl-ds postalAddress;
                    revision char(5);
                    regionCode char(2);
                    languageCode char(2);
                    postalCode char(10);
                    administrativeArea char(2);
                    locality char(50);
                    sublocality char(50);
                    addressLines char(100) dim(999);
                    recipients char(100) dim(999);
                    organization char(100);
                end-Ds;
                dcl-ds addressComponents dim(999);
                    dcl-ds componentName;
                        text char(50);
                        languageCode char(2);
                    end-Ds;
                    componentType char(30);
                    confirmationLevel char(30);
                    inferred char(5);
                    spellCorrected char(5);
                    replaced char(5);
                    unexpected char(5);
                end-Ds;
                missingComponentTypes char(30) dim(999);
                unconfirmedComponentTypes char(50) dim(999);
                unresolvedTokens char(50) dim(999);
            end-Ds;
            dcl-ds geocode;
                dcl-ds location;
                    latitude char(15);
                    longitude char(15);
                end-Ds;
                dcl-ds plusCode;
                    globalCode char(50);
                    compoundCode char(50);
                end-Ds;
                dcl-ds bounds;
                    dcl-ds low;
                        latitude char(15);
                        longitude char(15);
                    end-Ds;
                    dcl-ds high;
                        latitude char(15);
                        longitude char(15);
                    end-Ds;
                end-Ds;
                featureSizeMeters char(15);
                placeId char(50);
                placeTypes char(25) dim(999);
            end-Ds;
            dcl-ds metadata;
                business char(5);
                poBox char(5);
                residential char(5);
            end-Ds;
            dcl-ds uspsData;
                dcl-ds standardizedAddress;
                    firstAddressLine char(50);
                    firm char(50);
                    secondAddressLine char(50);
                    urbanization char(50);
                    cityStateZipAddressLine char(50);
                    city char(50);
                    state char(2);
                    zipCode char(5);
                    zipCodeExtension char(4);
                end-Ds;
                deliveryPointCode char(2);
                deliveryPointCheckDigit char(1);
                dpvConfirmation char(1);
                dpvFootnote char(40);
                dpvCmra char(1);
                dpvVacant char(1);
                dpvNoStat char(1);
                dpvNoStatReason int(10);
                dpvDrop char(1);
                dpvThrowback char(1);
                dpvNonDeliveryDays char(1);
                dpvNonDeliveryDaysValues int(10);
                dpvNoSecureLocation char(1);
                dpvPbsa char(1);
                dpvDoorNotAccessible char(1);
                dpvEnhancedDeliveryCode char(1);
                carrierRoute char(4);
                carrierRouteIndicator char(1);
                ewsNoMatch char(5);
                postOfficeCity char(50);
                postOfficeState char(2);
                fipsCountyCode char(3);
                county char(50);
                elotNumber char(10);
                elotFlag char(1);
                lacsLinkReturnCode char(20);
                lacsLinkIndicator char(20);
                poBoxOnlyPostalCode char(20);
                suitelinkFootnote char(20);
                pmbDesignator char(20);
                pmbNumber char(20);
                addressRecordType char(1);
                defaultAddress char(5);
                errorMessage char(100);
                cassProcessed char(5);
            end-Ds;
        end-Ds;
        responseId char(100);
    end-Ds;

The line where I attempt to parse the json into the DS:

Code: Select all

// Call the HTTP service
            response = http_string('POST': url : request : 'application/json');

        // Parse the JSON response
            DATA-INTO jsonResponse %DATA(response) %PARSER('YAJLINTO');

[jonboy49]

First of all congratulations on joining the club!

OK - your two basic problems which are fairly simple to handle. You may have others but ...

First, DATA-INTO needs to match elements by name and hierarchy. And "by name" includes the case in which the name is written. Now you may think that by naming the variable the same as the json (e.g. inputGranularity) you are indeed matching the case - but you are not. RPG is a case-insensitive language and therefore the names ABC, Abc, and abc all refer to the _same_ variable. So, RPG under the covers _always_ upper-cases all names. In other words it is trying to compare "inputGranularity" with "INPUTGRANULARITY" and that will fail. The cure is to add the option 'case=any' to the %DATA BIF. In fact, since you are dealing with json, you may want 'case=convert' as this will deal with json names that include characters that are illegal in RPG names. For example, "My Name" is a valid json name - but you cannot have a space in an RPG name. 'case=convert' deals with this.

Second, DATA-INTO by default expects every element in the DS to have a corresponding element in the json. But in your example, there are three elements of verdict DS that have no data in the json. addressComplete being one of them.

There are two major ways of handling this. The first is to use the option allowmissing=yes - the danger with this being that there is no granularity and therefore the minute you use this option there could be no matching data at all and RPG would be happy!

To gain more granular control you would have to add a count field to each optional element in the DS and use the option countprefix=xxx (where xxx is any string of characters you want to use - I use num_ or count_ most of the time).

So - if I use the countprefix option then just to accommodate addressComplete, I would change the DS to look like this:

Code: Select all

dcl-ds jsonResponse qualified inz;
        dcl-ds result;
            dcl-ds verdict;
                inputGranularity char(20);
                validationGranularity char(20);
                geocodeGranularity char(20);
                count_addressComplete int(5);  <<< I use int(5) but can be any numeric
                addressComplete char(5);

And modify the DATA-INTO like so:

Code: Select all

DATA-INTO jsonResponse %DATA(response : 'case=convert countprefix=count_' ) %PARSER('YAJLINTO');

I've written quite a lot about DATA-INTO that explains all this stuff - you can find them here: https://authory.com/JonParisAndSusanGan ... ction=JSON

Hope this helps.

[Scott Klement]

I agree with that error message.

1. This is not a problem with HTTPAPI -- you can determine that by putting together a quick program that reads the JSON directly from disk rather than downloading it. You'll get the same error, so that eliminates HTTPAPI from being the cause.
2. By default, DATA-INTO expects all of the key names (field names) to be in all-lowercase. This is because the default option in DATA-INTO is case=lower. I recommend changing it to case=convert.
3. There are keys (fields) in your DS that do not exist in the JSON. For example, your DS has "addressComplete", "hasInferredComponents", and "hasReplacedComponents" in it that do not exist in the JSON you posted. (There are probably many others.) You need to write code to tell RPG that these are optional, or at least tell it to ignore things that are missing from the document. I would also recommend allowing the document to contain extra data that's not in your DS -- because some day Google may add some new features, and you don't want it to break your code.

For example:

Code: Select all

DATA-INTO jsonResponse %DATA( response
                            : 'case=convert +
                               allowmissing=yes +
                               allowextra=yes') 
                           %PARSER('YAJLINTO');

"allowmissing" basically says that ANY/ALL fields in the JSON are optional -- which might not be what you want.

The other alternative is to do this:

Code: Select all

DATA-INTO jsonResponse %DATA( response
                            : 'case=convert +
                               countprefix=num_ +
                               allowextra=yes') 
                           %PARSER('YAJLINTO');

Then add a "num_XXXX" field for anything that is optional. For example, since the addressComplete is (apparently) optional, you would do:

Code: Select all

     . . .
      addressComplete char(5);
      num_addressComplete int(10);
     . . .

DATA-INTO will set the "num_addressComplete" field to 0 if addressComplete wasn't in the document, and to 1 if it was. This allows you to say that that particular field is optional, and to check if it was provided. Whereas allowmissing says everything is optional, and it's difficult to tell if it was provided or not. (Though, in this specific example, you could blank out the whole DS before calling DATA-INTO, then check if the field is blank afterwards to see if it is passed. That doesn't work in all cases, but it'd work in this one.)

[JBdev]

Thank you both!

Using the optional parms as recommended made everything work as expected.

Code: Select all

DATA-INTO JsonResponse %DATA(response
                                        : 'case=convert +
                                        allowmissing=yes +
                                        allowextra=yes')
                                        %PARSER('YAJLINTO');

Now I'm just reading through the copy files for HTTPAPIR4 trying to figure out how errors like HTTP/1.1 400 Bad Request are sent to the job log so I can figure out how to capture them in my program.

I appreciate your help!

UPDATE/EDIT:
I browsed the forum here and ended up finding the solution to my problem. I was using http_string() to process my request, which wasn't capturing any return codes outside the 200 to 299 range. In order to capture responses for 400/500 return codes I had to use http_req() instead as detailed in this post: https://www.scottklement.com/forums/vie ... error#p561

[Scott Klement]

Now I'm just reading through the copy files for HTTPAPIR4 trying to figure out how errors like HTTP/1.1 400 Bad Request are sent to the job log so I can figure out how to capture them in my program.

Code: Select all

monitor;
  // call http_string here
on-error;
  error_msg = http_error(errorCode: httpStatus);
end-mon;

if httpStatus = 400;
  // do special logic that is only done for 400 errors
endif;

Not sure why you'd want it in the job log -- but if you do, you could add it via the snd-msg opcode.