YAJLINTO parsing error

Discussions relating to the ScottKlement.com port of the open source YAJL JSON Reader/Generator. This includes the YAJL tool as well as the YAJLR4, YAJLGEN, YAJLINTO and YAJLDTAGEN add-ons from ScottKlement.com. http://www.scottklement.com/yajl/
Post Reply
erica.griffitt
Posts: 2
Joined: Wed Sep 14, 2022 2:08 pm

YAJLINTO parsing error

Post by erica.griffitt »

I am using HTTP_POST_VERBOSE to retrieve product data from an external web service. The data returned contains a trademark symbol that is causing an error when parsing the json response data using YAJLINTO.
1002: lexical error: invalid character inside string.█
The value is "Name": "Gildan® Heavy Cotton™ T-Shirt", but that is not what displays in the debug view. There is a "?" where the trademark should be.

I am almost certain that the actual issue is the encoding of the response data, but I have yet to find a CCSID value that results in data that can be parsed correctly. I would rather not scan and replace characters in the response data if I can avoid it.
Dcl-s wsResponseCLOB SQLTYPE(CLOB:500000);
I have tried specifying CCSID 1252, 1208, 819, and some other random values I found online. The CLOB variable shows the unrecognized character. That field gets moved to a varchar field that is returned to the requestor for parsing. I have tried manipulating the character set in the calling program as well without any success.

Is there a way around this? What am I missing?
Scott Klement
Site Admin
Posts: 636
Joined: Sun Jul 04, 2021 5:12 am

Re: YAJLINTO parsing error

Post by Scott Klement »

You will need to use Unicode fields to prevent it from converting the data into EBCDIC, since the ® symbol doesn't exist in EBCDIC.

In RPG, you'd have to define the variable as UCS2 (or type 'C' in fixed format.) You'll also need to review your SQL statement to make sure it uses Unicode data throughout.
erica.griffitt
Posts: 2
Joined: Wed Sep 14, 2022 2:08 pm

Re: YAJLINTO parsing error

Post by erica.griffitt »

The registered symbol is working fine. I can't find any method that support the trademark ™
The other issue is that we like to use a single standard service program for all of our http post calls. This helps us train other developers, who then don't have to be familiar with the actual http_post functions. So, I have a single clob defined for the response data and we have dozens of services up an running. This is the first one where we have encountered an issue. I don't need the trademark symbol, and I don't want to create a custom version of the process for this one instance. I was hoping there would be an easier way.
Scott Klement
Site Admin
Posts: 636
Joined: Sun Jul 04, 2021 5:12 am

Re: YAJLINTO parsing error

Post by Scott Klement »

It still sounds like the same issue -- the ™ symbol doesn't exist, so it's unable to translate it properly. Since the character doesn't exist in EBCDIC, when it receives it, it translates it to a certain character (that essentially means "nonexistent character" and is used for all characters that don't exist in EBCDIC.) When you call YAJL, it's getting translated back to Unicode, but now it's the unicode character for "doesn't exist", which isn't valid in a JSON document.

If you can write your service program to handle it properly, that will solve the problem. In fact, having it all in a service program should make this easier, but from the tone of your message I'm guessing that isn't the case?!
Post Reply