Issue with Character Encoding in HTTPAPI on IBM iSeries (V7R2)

[Jaume]

Hi Scott,
We are experiencing an issue with character encoding when using Scott Klement's HTTPAPI library on an IBM iSeries system running V7R2. We have implemented HTTPAPI to make RESTful API calls to external endpoints, and while the same program runs correctly on an IBM iSeries in Spain, we are encountering unexpected character encoding issues when running it on a different IBM iSeries system in Germany.
Issue Description:
When sending a JSON payload from the system in Germany, an unexpected character "Â" (UTF-8: C3 82) appears before the opening square bracket ([).
Example JSON output from Spain (correct):
[{"FiscalId": "37281892L","FiscalName": "JAIME BIGAS"}]
Example JSON output from Germany (incorrect):
Â[{"FiscalId": "37281892L","FiscalName": "JAIME BIGAS"}]
This suggests that the conversion from EBCDIC to UTF-8 is not being handled correctly for certain characters, possibly the opening square bracket [ (EBCDIC 37: AD).
Steps Taken to Resolve the Issue:
1. Checked HTTPAPI Encoding Options:
o We set the following options:
o http_setOption('local-ccsid': '37'); // EBCDIC 37
o http_setOption('network-ccsid': '1208'); // UTF-8
o http_setOption('send-encoding', 'utf-8');
These settings did not resolve the issue.
2. Changed Job CCSID:
o We tried changing the CCSID of the job using CHGJOB CCSID(273), and later CHGJOB CCSID(37), but the issue persisted.
3. Verified Compilation CCSID:
o We confirmed that the program is compiled with CCSID 37.
4. Analyzed Hexadecimal Output:
o We compared the hex values of the output from Spain and Germany.
o In Germany, the unexpected character  (C3 82) appears before the first bracket ([).
Request for Support:
• Could this issue be related to a different system configuration between the two IBM iSeries environments?
• Is there an additional setting in HTTPAPI or system-level CCSID configuration that we should check?
• Could the default encoding of the job or PASE environment be interfering with HTTPAPI's encoding conversion?
Any guidance on resolving this encoding issue would be greatly appreciated.

Many thanks for your help
Jaume Bigas
GESISA

[Scott Klement]

When sending a JSON payload from the system in Germany, an unexpected character "Â" (UTF-8: C3 82) appears before the opening square bracket ([).

What is the data in hex? Please provide the byte values both before and after translation.

This suggests that the conversion from EBCDIC to UTF-8 is not being handled correctly for certain characters, possibly the opening square bracket [ (EBCDIC 37: AD).

I'm not sure that I agree with that. Translation issues don't usually cause an extra byte to appear in the data. The only exception that I can think of is if you treat input data that is in a multi-byte encoding like UTF-8 as if it is not multibyte like ASCII. But that doesn't at all appear to be what you are doing.

Steps Taken to Resolve the Issue:
1. Checked HTTPAPI Encoding Options:
o We set the following options:
o http_setOption('local-ccsid': '37'); // EBCDIC 37
o http_setOption('network-ccsid': '1208'); // UTF-8
o http_setOption('send-encoding', 'utf-8');

Hmmm.. I don't agree with these settings. (1) 37 not normal for Germany. Why are you using 37? (2) there is no such option as 'send-encoding', so that line of code makes no sense and is doing nothing.

That said, this doesn't explain how an extra character would be appearing.

Request for Support:
• Could this issue be related to a different system configuration between the two IBM iSeries environments?
• Is there an additional setting in HTTPAPI or system-level CCSID configuration that we should check?
• Could the default encoding of the job or PASE environment be interfering with HTTPAPI's encoding conversion?

My approach to solving problems is to troubleshoot the problem and discover what is wrong, then come up with a fix that addresses that problem. So to help you further, I need a way to troubleshoot it. (You've told me what the bad output is, but nothing about what causes it.)

To help you further, I need to know how to reproduce the problem so that I can troubleshoot it. Please provide a piece of code that will make the same thing happen on my machine.

A debug/trace log might also help us spot the issue. You can create that by adding a line of code like this to the top of your program:

Code: Select all

http_debug(*on: '/tmp/debug_log.txt');

Then download the debug_log.txt file and attach it to message here.