Using CCSID 1208 - accent characters

[glenhart]

Hi,

I am totally stuck. I know there have been discussions on this, but in most cases, it relates to importing the json from a web service.

I am importing the JSON file from the IFS using yajl_stmf_load_tree(IFSFILE : errMsg); This file holds an accent character in the data portion.

When I run e1Sfcd = yajl_get_string(node); the results do not correctly translate the accent character. It becomes an unprintable character.

I have where some have used http_setOption ( 'network-ccsid' : '1208' ); Which I tried to set as the first step I m my program, but that did not resolve the issue.

Any help at all.. I am desperate for a solution.

Many thanks
Glen.

[Scott Klement]

http_setOption is for setting options in HTTPAPI - it has nothing whatsoever to do with YAJL.

You will need to troubleshoot what's happening.

• Use the WRKLNK command to see what the CCSID on your JSON file is.
• Use DSPF and F10=Display Hex to see the hex value of the accented character.
• Look on a table and verify that it is the correct hex value for that character under that particular CCSID.
• Debug YAJLR4, the yajl_stmf_load_tree routine should be reading the data so that it will translate to UTF-8 (CCSID 1208). Look at what it has read in the debugger, view it in hex... verify that it has the correct hex value for the accented character in UTF-8.
• The output from yajl_get_string() is in the job's CCSID. Look at your job CCSID, and the hex value of the character as output from yajl_get_string and see if it is correct for your job ccsid.

If all of this is correct, then the problem isn't YAJL.

If it's failing at some point, then what point? What is the hex value reported? What CCSID is it supposed to be? What is the correct hex value for that CCSID?

[glenhart]

Thank you so much for providing me with direction.. I will go through the checklist and later post my findings for others that may have the same issue.
Dealing with CCSID is new for me.

[glenhart]

Hi,

The issue was that the file CCSID was 437.
For it to work with a French accent I needed it to be UTF-8 (CCSID 1208)

To achieve this I ran the following command on the file before I imported:
CPY OBJ('/home/tfr_t/sfc/in/product_202325140925.json') TOOBJ('/home/tfr_t/sfc/in/product_202325140925.json') TOCCSID(1208) REPLACE(*YES)

Then doing a YAML import from the IFS worked correct..
Thanks for your guidance.

[mschutte369]

I realize this is an old post but I am having a similar issue but on an http_req. I thought after reading this that I found the issue but sadly it did not solve my problem. Our problem is similar with accented characters in a person's name.

For example, ñ

When we make the http_req, we get a bad request. The API host is telling us that it should be encoded properly.

We are using yajl and from what I am reading the yajl procedures are converting everything to UTF-8. But when I was reviewing code, I noticed that they used 0 as the CCSID parameter on the

Code: Select all

yajl_copyBuf(0:x_stream:10000:retsize); 

So I figured it was using my job's CCSID of 37.

I attempted to change to 1208. But that was a complete failure.

Any help would be appreciated. I could start a new post too if necessary.

Code: Select all

yajl_copyBuf(0:x_stream:10000:retsize);       
                                                              
monitor;                                      
clear xmlret;                                 
RQ = http_req('POST'                          
                 : URL                        
                 : *OMIT                      
                 : xmlret                     
                 : *OMIT                      
                 : %trim(xstream)             
                 : 'application/json');       

NOTE: if I replace the ntilde, with just an "n". the call goes through without issue.

[Scott Klement]

When we make the http_req, we get a bad request. The API host is telling us that it should be encoded properly.

As far as I can tell, you are generating the JSON, not the remote host. Why would you ask them if their document is encoded properly?

We are using yajl and from what I am reading the yajl procedures are converting everything to UTF-8. But when I was reviewing code, I noticed that they used 0 as the CCSID parameter on the

Code: Select all

yajl_copyBuf(0:x_stream:10000:retsize); 

This is a line of code inside of YAJL? Not in my copy of YAJL.

The JSON format requires UTF-8... so YAJL "thinks" in UTF-8. It merely converts your EBCDIC to UTF-8 so that it can work with it. The yajl_copyBuf() code you cite is converting from UTF-8 to whatever the job's CCSID is. I dont recommend doing that, unless you really need the data in EBCDIC for some reason, because where UTF-8 supports 300000 different characters, EBCDIC typically only supports around 200. That's a huge difference, and creates a big chance for characters to get screwed up. Keep it in UTF-8 if at all possible.

So I figured it was using my job's CCSID of 37.

I attempted to change to 1208. But that was a complete failure.

"A complete failure" doesn't tell us what happened or how we can help with this. Why do you want to convert something that is already UTF-8 to UTF-8?

Any help would be appreciated. I could start a new post too if necessary.

That would be the best practice.

Code: Select all

yajl_copyBuf(0:x_stream:10000:retsize);       
                                                              
monitor;                                      
clear xmlret;                                 
RQ = http_req('POST'                          
                 : URL                        
                 : *OMIT                      
                 : xmlret                     
                 : *OMIT                      
                 : %trim(xstream)             
                 : 'application/json');       

Perhaps the easiest solution here is to save the JSON to disk with yajl_saveBuf() and then send that file with http_req() instead of sending a string. Then you can delete the file if you don't need it.

[mschutte369]

Hi Scott,

Thank you for taking the time to reply, and sorry for the delay. I kept checking back, but something seemed to be going on with the site. I really appreciate your patience.

I left out some details initially, thinking they wouldn’t matter, but maybe they will help now. I’ll admit, while HTTP isn’t exactly new territory for me, CCSID handling definitely is. Normally I just get things working by relying heavily on documentation (especially yours!), and I haven’t really had to dig into CCSID intricacies — until now.

In this project, we asked a vendor to call an external API (parcel shipper) by sending JSON data (so the code in question isn’t mine). I was reviewing their implementation and pointing out a few things based on my understanding — mostly just thinking out loud. My boss even reminded me that I don’t always have to be the hero.

Here’s the situation:

* We are building a JSON string using YAJL, then assigning it to the HTTP request variable.
* The API owner claims they can accept special characters like `ñ`, but when such characters are present, we get a "Bad Request" error with no further details.
* Their only suggestion was “I think it needs to be escaped,” but they didn’t sound very confident, so I’m not sure they truly understand the issue.

Regarding the code:

Code: Select all

yajl_copyBuf(0:x_stream:10000:retsize);

You’re correct — this line isn’t from YAJL itself, but rather in the wrapper program calling the YAJL procedures to build the JSON.

Initially, we used `0` for the CCSID, which I now understand means “convert to the job’s CCSID.” Since our job is running under CCSID 37 (EBCDIC), I suspected that might be why the n-tilde (`ñ`) wasn’t making it through properly. To test this, I changed the CCSID to 1208 (UTF-8), which did bypass the “Bad Request” error, but the API response came back as an HTML error page — not the expected ZPL.

Here’s a subset of what the request body looked like when viewed in the log:

Code: Select all

#‚ÊÁÉÍÁËȬÑÀ‚š‚‘‘“˜“

I recognize that this is probably due to CCSID mismatch between the stream (UTF-8) and the log viewer (819), so that’s understandable.

The response we received was a 500 error with an HTML body like this:

Code: Select all

<title>We're sorry, but something went wrong (500)</title>
<p>If you are the application owner check the logs for more information.</p>

So while changing the CCSID did result in the request being accepted, it seems like the JSON may still have issues (possibly invalid formatting or quoting), or the API itself can’t handle some aspect of the content.

I definitely plan to try your suggestion of writing the JSON to a file and sending that instead — that might help isolate the issue or at least confirm how the data is being constructed and encoded.

Thanks again for your help and your patience while I wrap my head around all of this.

[mschutte369]

Scott, thank you for the help. By using saveBuf and using the stream file, I was able to successfully get a good result. the n-tilde did not return a bad request error, however, the label returned showed it as what looks to be a yen. ¥

I think I can live with that, as long as it is not stopping our ops. I will request the API owner to review and see what they have in their side though.