YAJL lexical error: invalid character inside string

[Dharshani]

Hi,

Could you please let me know how to remove invalid json character from the response json.
I have tried with %scan and %replace build in functions to remove '\"' , '\n '. But I found more than that characters. Please consider that i'm using YAJL library to read the json.
How to determine which characters could be invalid?

Please find the following example key values.
"remarks": "Founder and leader of Ansar Eddine (QDe.135). Member of the Tuareg Ifogas\n tribe. Linked to the Organization of Al-Qaida in the Islamic Maghreb (QDe.014) and\n Mouvement pour l’Unification et le Jihad en Afrique de l’Ouest (MUJAO) (QDe.134).\n Name of father is Ag Bobacer Arhali, name of mother is Rhiachatou Wallet Sidi.\n Review pursuant to Security Council resolution 2253 (2015) was concluded on 21 Feb.\n 2019. Review pursuant to Security Council resolution 2610 (2021) was concluded on 8 November 2022. INTERPOL-UN Security Council Special Notice web link:https://www.interpol.int/en/How-we-work ... ndividuals",

"action": "Asset freeze|Arms embargo|Travel Ban - Founder and leader of Ansar Eddine (QDe.135). Member of the Tuareg Ifogas tribe. Linked to the Organization of Al-Qaida in the Islamic Maghreb (QDe.014) and Mouvement pour l’Unification et le Jihad en Afrique de l’Ouest (MUJAO) (QDe.134). Name of father is Ag Bobacer Arhali, name of mother is Rhiachatou Wallet Sidi. Review pursuant to Security Council resolution 2253 (2015) was concluded on 21 Feb. 2019. INTERPOL-UN Security Council Special Notice web link: https://www.interpol.int/en/notice/search/un/5278332",

"action": "Asset freeze|Arms embargo|Travel Ban - In approximately 2005, ran a \"basic training\" camp for Al-Qaida (QDe.004) in Pakistan. Review pursuant to Security Council resolution 2253 (2015) was concluded on 21 Feb. 2019. INTERPOL-UN Security Council Special Notice web link: https://www.interpol.int/en/notice/search/un/1928966",

Thank you

[jonboy49]

Well they are not "invalid". They appear to be escape characters.

In JSON any character that has special meaning must be escaped if it is part of the data. For example this:

Code: Select all

In json special characters, 
such as " must be escaped

Would have to be represented as:

Code: Select all

In json special characters, \nsuch as \" must be escaped

This web site gives a list of the json escape sequences and their meanings https://www.lambdatest.com/free-online- ... son-escape and provides a tool to allow you to generate escaped json strings.

As to translating these "invalid" characters, it depends on what you want to do with the data. For example /n represents a line break as my example shows. If you wanted to store that as HTML (for example) you might choose to translate that to a <br> tag. As I say - it all depends on what you plan to do with the data.

[Scott Klement]

I don't see any problems with the parts of the document that you posted.

Post your actual document (not just random snippets) into a website like jsonlint.com and see what the problems are instead of guessing.

[Dharshani]

Hi Scott,

Thanks for your response. By the way the json is valid.

Code: Select all

documentNode = yajl_string_load_tree(response: errorMessage);   

response is a char type variable.

When I used char, error returned as 'documentNode = yajl_string_load_tree(response: errorMessage); ' & when I used varchar, error returned as 'parse error: trailing garbage '.

I tried with %scan, %replace to remove json escapes. But I'm looking for a best solution to remove or convert json escape from response.
Data 1

Code: Select all

{
  "error" : false,
  "searchTerm" : "IYAD AG GHALI NANA",
  "sourcesUsed" : [ {
    "source" : "DPL"
  }, {
    "source" : "NONSDN"
  }, {
    "source" : "UN"
  }, {
    "source" : "OFSI"
  }, {
    "source" : "SDN"
  } ],
  "matches" : [ {
    "score" : 90,
    "source" : "SDN",
    "firstName" : "Iyad ag",
    "lastName" : "GHALI",
    "fullName" : "Iyad ag GHALI",
    "dob" : "1954",
    "addresses" : [ ],
    "citizenship" : [ ],
    "phoneNumbers" : [ ],
    "uid" : 0,
    "sdnType" : "Individual",
    "programs" : [ "SDGT" ],
    "additionalSanctions" : [ ],
    "passports" : [ ],
    "driversLicenses" : [ ],
    "eccn" : [ ],
    "ids" : [ ],
    "akas" : [ {
      "score" : 0,
      "uid" : 23670,
      "category" : "strong",
      "lastName" : "GHALY",
      "firstName" : "Iyad ag"
    } ]
  }, {
    "score" : 90,
    "source" : "UN",
    "fullName" : "IYAD AG GHALI ",
    "addresses" : [ {
      "country" : "Mali"
    } ],
    "citizenship" : [ "Mali" ],
    "phoneNumbers" : [ ],
    "uid" : 0,
    "sdnType" : "Individual",
    "remarks" : "Founder and leader of Ansar Eddine (QDe.135). Member of the Tuareg Ifogas\n                tribe. Linked to the Organization of Al-Qaida in the Islamic Maghreb (QDe.014) and\n                Mouvement pour l’Unification et le Jihad en Afrique de l’Ouest (MUJAO) (QDe.134).\n                Name of father is Ag Bobacer Arhali, name of mother is Rhiachatou Wallet Sidi.\n                Review pursuant to Security Council resolution 2253 (2015) was concluded on 21 Feb.\n                2019. Review pursuant to Security Council resolution 2610 (2021) was concluded on 8 November 2022. INTERPOL-UN Security Council Special Notice web link:https://www.interpol.int/en/How-we-work/Notices/View-UN-Notices-Individuals",
    "programs" : [ "UN List Al-Qaida" ],
    "additionalSanctions" : [ ],
    "passports" : [ ],
    "driversLicenses" : [ ],
    "eccn" : [ ],
    "ids" : [ ],
    "akas" : [ {
      "score" : 0,
      "category" : "Good",
      "firstName" : "sidi mohamed arhali"
    } ]
  }, {
    "score" : 90,
    "source" : "OFSI",
    "firstName" : "Iyad",
    "lastName" : "AG GHALI",
    "fullName" : "Iyad AG GHALI",
    "dob" : "1958",
    "addresses" : [ {
      "country" : "Mali"
    } ],
    "citizenship" : [ ],
    "phoneNumbers" : [ ],
    "uid" : 0,
    "sdnType" : "Individual",
    "programs" : [ ],
    "additionalSanctions" : [ ],
    "passports" : [ {
      "passport" : "A1037434",
      "passportCountry" : "Mali number, issued on 10 Aug. 2001 (expired 31 Dec. 2014)"
    } ],
    "driversLicenses" : [ ],
    "action" : "Asset freeze|Arms embargo|Travel Ban - Founder and leader of Ansar Eddine (QDe.135). Member of the Tuareg Ifogas tribe. Linked to the Organization of Al-Qaida in the Islamic Maghreb (QDe.014) and Mouvement pour l’Unification et le Jihad en Afrique de l’Ouest (MUJAO) (QDe.134). Name of father is Ag Bobacer Arhali, name of mother is Rhiachatou Wallet Sidi. Review pursuant to Security Council resolution 2253 (2015) was concluded on 21 Feb. 2019. INTERPOL-UN Security Council Special Notice web link: https://www.interpol.int/en/notice/search/un/5278332",
    "eccn" : [ ],
    "ids" : [ ],
    "akas" : [ {
      "score" : 0,
      "category" : "Alias",
      "lastName" : "Arhali",
      "firstName" : "Sidi"
    } ]
  } ]
}

Data 2

Code: Select all

{
"error": false,
"searchTerm": "SAID JAN ABD AL-SALAM",
"sourcesUsed": [
{
"source": "DPL"
},
{
"source": "NONSDN"
},
{
"source": "UN"
},
{
"source": "OFSI"
},
{
"source": "SDN"
}
],
"matches": [
{
"score": 100,
"source": "SDN",
"firstName": "Said Jan",
"lastName": "'ABD AL-SALAM",
"fullName": "Said Jan 'ABD AL-SALAM",
"dob": "05 Feb 1981",
"addresses": [],
"citizenship": [],
"phoneNumbers": [],
"uid": 0,
"sdnType": "Individual",
"remarks": "Passport OR801168 and Kuwaiti National ID No. 281020505755 issued under the name Said Jan 'Abd al-Salam; Passport 4117921 issued under the name Dilawar Khan Zain Khan",
"programs": [
"SDGT"
],
"additionalSanctions": [],
"passports": [
{
"passport": "4117921",
"passportCountry": "Pakistan"
},
{
"passport": "OR801168",
"passportCountry": "Afghanistan"
}
],
"driversLicenses": [],
"eccn": [],
"akas": [
{
"score": 0,
"uid": 13475,
"category": "strong",
"lastName": "'ABD-AL-SALAM",
"firstName": "Sa'id Jan"
},
{
"score": 0,
"uid": 13476,
"category": "strong",
"lastName": "'ABDALLAH",
"firstName": "Qazi"
},
{
"score": 0,
"uid": 13477,
"category": "strong",
"lastName": "ABDULLAH",
"firstName": "Qazi"
},
{
"score": 0,
"uid": 13478,
"category": "strong",
"lastName": "WALID",
"firstName": "Ibrahim"
},
{
"score": 0,
"uid": 13479,
"category": "strong",
"lastName": "ZAIN KHAN",
"firstName": "Dilawar Khan"
},
{
"score": 0,
"uid": 13480,
"category": "strong",
"lastName": "SA'ID JAN",
"firstName": "Qasi"
},
{
"score": 0,
"uid": 13481,
"category": "strong",
"lastName": "JHAN",
"firstName": "Said"
},
{
"score": 0,
"uid": 13482,
"category": "strong",
"lastName": "KHAN",
"firstName": "Farhan"
},
{
"score": 0,
"uid": 13483,
"category": "strong",
"lastName": "CAIRO",
"firstName": "Aziz"
},
{
"score": 0,
"uid": 13484,
"category": "weak",
"lastName": "NANGIALI"
}
],
"ids": [
{
"type": "National ID No.",
"id": "281020505755"
}
]
},
{
"score": 100,
"source": "UN",
"fullName": "SAID JAN ‘ABD AL-SALAM ",
"dob": "1981-02-05",
"addresses": [],
"citizenship": [
"Afghanistan"
],
"phoneNumbers": [],
"uid": 0,
"sdnType": "Individual",
"remarks": "In approximately 2005, ran a \"basic training\" camp for Al-Qaida (QDe.004) in Pakistan.  Review pursuant to Security Council resolution 2253 (2015) was concluded on 21 Feb. 2019. Review pursuant to Security Council resolution 2610 (2021) was concluded on 8 November 2022. INTERPOL-UN Security Council Special Notice web link:https://www.interpol.int/en/How-we-work/Notices/View-UN-Notices-Individuals",
"programs": [
"UN List Al-Qaida"
],
"additionalSanctions": [],
"passports": [],
"driversLicenses": [],
"eccn": [],
"akas": [
{
"score": 0,
"category": "Good",
"firstName": "said jan abd al salam"
},
{
"score": 0,
"category": "Good",
"firstName": "dilawar khan zain khan"
},
{
"score": 0,
"category": "Low",
"firstName": "qazi abdallah"
},
{
"score": 0,
"category": "Low",
"firstName": "qazi abdullah"
},
{
"score": 0,
"category": "Low",
"firstName": "ibrahim walid"
},
{
"score": 0,
"category": "Low",
"firstName": "qasi said jan"
},
{
"score": 0,
"category": "Low",
"firstName": "said jhan"
},
{
"score": 0,
"category": "Low",
"firstName": "farhan khan"
},
{
"score": 0,
"category": "Low",
"firstName": "aziz cairo"
},
{
"score": 0,
"category": "Low",
"firstName": "nangiali"
}
],
"ids": []
},
{
"score": 100,
"source": "OFSI",
"firstName": "Said Jan",
"lastName": "'ABD AL-SALAM",
"fullName": "Said Jan 'ABD AL-SALAM",
"dob": "01/01/1972",
"addresses": [],
"citizenship": [],
"phoneNumbers": [],
"uid": 0,
"sdnType": "Individual",
"programs": [],
"additionalSanctions": [],
"passports": [
{
"passport": "OR801168",
"passportCountry": "Afghanistan passport number, expired 27/02/2011, under name Said Jan 'Abd al-Salam"
},
{
"passport": "4117921",
"passportCountry": "Pakistan passport number, issued date: 09/09/2008, expired 09/09/2013, issued under name Dilawar Khan Zain Khan"
}
],
"driversLicenses": [],
"action": "Asset freeze|Arms embargo|Travel Ban - In approximately 2005, ran a \"basic training\" camp for Al-Qaida (QDe.004) in Pakistan. Review pursuant to Security Council resolution 2253 (2015) was concluded on 21 Feb. 2019. INTERPOL-UN Security Council Special Notice web link: https://www.interpol.int/en/notice/search/un/1928966",
"eccn": [],
"akas": [
{
"score": 0,
"category": "Alias",
"lastName": "Abdallah",
"firstName": "Qazi"
},
{
"score": 0,
"category": "Alias",
"lastName": "Abd-al-Salam",
"firstName": "Sa'id"
},
{
"score": 0,
"category": "Alias",
"lastName": "Abdullah",
"firstName": "Qazi"
},
{
"score": 0,
"category": "Alias",
"lastName": "Cairo",
"firstName": "Aziz"
},
{
"score": 0,
"category": "Alias",
"lastName": "Jan",
"firstName": "Qasi"
},
{
"score": 0,
"category": "Alias",
"lastName": "Jhan",
"firstName": "Said"
},
{
"score": 0,
"category": "Alias",
"lastName": "Khan",
"firstName": "Dilawar"
},
{
"score": 0,
"category": "Alias",
"lastName": "Khan",
"firstName": "Farhan"
},
{
"score": 0,
"category": "Alias",
"lastName": "Nangiali"
},
{
"score": 0,
"category": "Alias",
"lastName": "Walid",
"firstName": "Ibrahim"
}
],
"ids": []
}
]
}

[Scott Klement]

Code: Select all

documentNode = yajl_string_load_tree(response: errorMessage);   

response is a char type variable.

When I used char, error returned as 'documentNode = yajl_string_load_tree(response: errorMessage); ' & when I used varchar, error returned as 'parse error: trailing garbage '.

I suspect you have that backwards. When you use CHAR you get "trailing garbage" because CHAR is fixed-length and therefore will be longer than the actual json document, so you'll have a bunch of extra stuff at the end. These will probably be blanks -- but, nevertheless, it is extra stuff at the end of the document that shouldn't be there.

yajl_string_load_tree expects data in a VARCHAR (VARYING) format. But since the parameter is CONST, RPG will automatically convert it as needed (albeit, at a performance cost.) Just pass it as varchar and save the hassle.

I tried with %scan, %replace to remove json escapes. But I'm looking for a best solution to remove or convert json escape from response.

Removing the escape sequences will make your JSON invalid, and impossible for YAJL to process. Can you explain why you want to remove them?