Problem with import stream file json

Discussions relating to the ScottKlement.com port of the open source YAJL JSON Reader/Generator. This includes the YAJL tool as well as the YAJLR4, YAJLGEN, YAJLINTO and YAJLDTAGEN add-ons from ScottKlement.com. http://www.scottklement.com/yajl/
Post Reply
Simon.Ishakov
Posts: 1
Joined: Tue Jul 12, 2022 11:00 am

Problem with import stream file json

Post by Simon.Ishakov »

Hii to all,
I have a problem with YAJLGen command in as400 environment (OP 7.2).
The result of generator is the source rpg program with it I want to parse my json files
The problem is the json file has two different kinds of data for the same property - bulkarray - it can be or "null" or "structure".
When the generator creates the source program it base on the first appearance of that property. When it start parse process and see the second type it's fail.

Somebody can help me with this issue

Error message:

Code: Select all

Message ID . . . . . . :   RNX0356                                            
Date sent  . . . . . . :   12/07/22      Time sent  . . . . . . :   14:13:15  
                                                                              
Message . . . . :   The document for the DATA-INTO operation does not match   
  the RPG variable; reason code 5.                                            
                                                                              
Cause . . . . . :   While parsing a document for the DATA-INTO operation, the 
  parser found that the document does not correspond to RPG variable "jsondoc"
  and the options do not allow for this. The reason code is 5. The exact      
  subfield for which the error was detected is                                
  "jsondoc.instrumentdatas(1).data(2)". The options are "doc=file            
  case=convert countprefix=num_". The document name is /new 1.json; *N
  indicates that the document is not an external file. The parser is          
  'YAJLINTO'. *N indicates that the parser is a procedure pointer.            
Recovery  . . . :   Contact the person responsible for program maintenance to 
  determine the cause of the problem.                                         

Source json file:

Code: Select all

{
	"responseId": "646463461346-686863536536",
	"headers": {
		"actions": null,
		"actions_dateSpecified": false,
		"daterange": {
			"period": {
				"start": "2022-07-01T00:00:00",
				"end": "2022-07-10T00:00:00"
			},
			"duration": null
		},
		"dateformatSpecified": false,
		"programflag": 0,
		"programflagSpecified": true,
		"rundate": "20220710",
		"time": null,
		"secidSpecified": false,
		"snSpecified": false,
		"usernumberSpecified": false,
		"versionSpecified": false,
		"wsSpecified": false,
		"yellowkeySpecified": false
	},
	"timestarted": "2022-07-10T15:53:03+03:00",
	"timestartedSpecified": true,
	"instrumentDatas": [
		{
			"code": "0",
			"instrument": {
				"id": "SWM US",
				"yellowkey": 6,
				"yellowkeySpecified": true,
				"typeSpecified": false,
				"overrides": null
			},
			"macro": null,
			"standardFields": {
				"companyId": 179489,
				"companyIdSpecified": true,
				"securityId": 0,
				"securityIdSpecified": true,
				"actionId": 229880930,
				"actionIdSpecified": true,
				"mnemonic": "ACQUIS",
				"flag": "U",
				"companyName": "Mativ Inc",
				"secIdType": "N.A.",
				"secId": "N.A.",
				"currency": "N.A.",
				"marketSectorDesSpecified": false,
				"bbUnique": "N.A.",
				"announceDate": "03/28/2022",
				"effectiveDate": "07/06/2022",
				"amendDate": "07/06/2022",
				"bbGlobal": "N.A.",
				"bbGlobalCompany": "BBG001FH7915",
				"bbSecNumDes": "N.A.",
				"feedSource": "N.A."
			},
			"data": [
				{
					"bulkarray": null,
					"field": "CP_SH_FRACTIONAL",
					"value": "0",
					"isArraySpecified": false,
					"rowsSpecified": false
				},
				{
					"bulkarray": [
						{
							"data": [
								{
									"value": "Target Legal Adviser(s)",
									"type": 0,
									"typeSpecified": true
								},
								{
									"value": "Bryan Cave Leigh, Cravath Swaine",
									"type": 0,
									"typeSpecified": true
								},
								{
									"value": "100.000000",
									"type": 12,
									"typeSpecified": true
								}
							],
							"columns": 3,
							"columnsSpecified": true
						},
						{
							"data": [
								{
									"value": "Target Financial Adviser(s)",
									"type": 0,
									"typeSpecified": true
								},
								{
									"value": "Perella Weinberg",
									"type": 0,
									"typeSpecified": true
								},
								{
									"value": "100.000000",
									"type": 12,
									"typeSpecified": true
								}
							],
							"columns": 3,
							"columnsSpecified": true
						}
					],
					"field": "CP_ADVISORS",
					"value": null,
					"isArray": true,
					"isArraySpecified": true,
					"rows": 5,
					"rowsSpecified": true
				},
				{
					"bulkarray": null,
					"field": "CP_ACQ_TKR",
					"value": "MATV US",
					"isArraySpecified": false,
					"rowsSpecified": false
				}
			]
		},
		{
			"code": "0",
			"instrument": {
				"id": "SWM US",
				"yellowkey": 6,
				"yellowkeySpecified": true,
				"typeSpecified": false,
				"overrides": null
			},
			"macro": null,
			"standardFields": {
				"companyId": 179489,
				"companyIdSpecified": true,
				"securityId": 0,
				"securityIdSpecified": true,
				"actionId": 230803894,
				"actionIdSpecified": true,
				"mnemonic": "CHG_NAME",
				"flag": "U",
				"companyName": "Mativ Inc",
				"secIdType": "N.A.",
				"secId": "N.A.",
				"currency": "N.A.",
				"marketSectorDesSpecified": false,
				"bbUnique": "N.A.",
				"announceDate": "06/21/2022",
				"effectiveDate": "07/06/2022",
				"amendDate": "07/06/2022",
				"bbGlobal": "N.A.",
				"bbGlobalCompany": "BBG001FH7915",
				"bbSecNumDes": "N.A.",
				"feedSource": "N.A."
			},
			"data": [
				{
					"bulkarray": null,
					"field": "CP_OLD_NAME",
					"value": "Schweitzer-Mauduit Int Inc",
					"isArraySpecified": false,
					"rowsSpecified": false
				},
				{
					"bulkarray": null,
					"field": "CP_NEW_NAME",
					"value": "Mativ Inc",
					"isArraySpecified": false,
					"rowsSpecified": false
				},
				{
					"bulkarray": null,
					"field": "CP_NOTES",
					"value": "Full Old Name: Schweitzer-Mauduit International Inc",
					"isArraySpecified": false,
					"rowsSpecified": false
				},
				{
					"bulkarray": null,
					"field": "CP_ACTION_STATUS",
					"value": "R",
					"isArraySpecified": false,
					"rowsSpecified": false
				},
				{
					"bulkarray": null,
					"field": "CP_INFO_SOURCE",
					"value": "10",
					"isArraySpecified": false,
					"rowsSpecified": false
				}
			]
		}
	],
	"timefinished": "2022-07-10T15:53:04+03:00",
	"timefinishedSpecified": true,
	"statusCode": {
		"code": 0,
		"description": "Success"
	},
	"requestId": "2b5af636-e7d3-40ed-9de2-9749d81cdd2d"
}
jonboy49
Posts: 93
Joined: Wed Jul 28, 2021 8:18 pm

Re: Problem with import stream file json

Post by jonboy49 »

There is currently not a good answer for this problem.

DATA-INTO has no way of handling this. It could handle it if there were no entries by using the count field - but when the entry is null it can't handle it.

Off the top of my head I can't see a simple way for YAJLINTO y=to handle it other than perhaps to simply ignore a null when associated with a compound element but that would undoubtedly cause other problems. We probably need some kind of fix from RPG but that would take a while.

Obviously you could parse the JSON using a different method - raw YAJL or perhaps SQL? - the only way I can think of to have it work with DATA-INTO would be to scan the raw json for an instance of null at the bulk array level and strip out the whole entry. That should work.
Scott Klement
Site Admin
Posts: 305
Joined: Sun Jul 04, 2021 5:12 am

Re: Problem with import stream file json

Post by Scott Klement »

jonboy49 wrote: Fri Jul 15, 2022 11:55 pm Off the top of my head I can't see a simple way for YAJLINTO y=to handle it other than perhaps to simply ignore a null when associated with a compound element but that would undoubtedly cause other problems. We probably need some kind of fix from RPG but that would take a while.
YAJLINTO doesn't know that it's a compound element. The DATA-INTO opcode does not tell the parser what type of RPG element it is reading into, so for YAJLINTO to know, it'd need some form of "mapping instruction file" to understand what might be expected -- imho, something like this makes it less elegant.

It could have an option that tells it to always ignore nulls (treat them as if the element was never passed.) Do you think that's a good idea?
jonboy49
Posts: 93
Joined: Wed Jul 28, 2021 8:18 pm

Re: Problem with import stream file json

Post by jonboy49 »

Scott Klement wrote: Sat Jul 16, 2022 2:00 pm <snip>
YAJLINTO doesn't know that it's a compound element. The DATA-INTO opcode does not tell the parser what type of RPG element it is reading into, so for YAJLINTO to know, it'd need some form of "mapping instruction file" to understand what might be expected -- imho, something like this makes it less elegant.

It could have an option that tells it to always ignore nulls (treat them as if the element was never passed.) Do you think that's a good idea?
I had been formulating ideas for an IDEA submission and this was one the one I had decided to write up. So yes - I think it is probably as good a solution as we could get. RPG knows that it is dealing with a compound element and therefore should be able to differentiate between null as a value and null as in "nothing to see here".

The more situations like this I encounter the less I like JSON!
Scott Klement
Site Admin
Posts: 305
Joined: Sun Jul 04, 2021 5:12 am

Re: Problem with import stream file json

Post by Scott Klement »

Okay I added a new %PARSER option called "skip_nulls". If any element in the JSON document is set to null, it will skip it entirely (as if it isn't in the document.)

You will need to use options like countprefix or allowmissing/allowextra, since RPG will think the element doesnt exist in the JSON at all... but that's about the best I can do.

For example:

Code: Select all

         data-into jsonDoc %DATA( ifsPathname
                                : 'doc=file case=convert countprefix=num_ +
                                   allowmissing=yes allowextra=yes')
                         %PARSER( 'YAJLINTO'
                                : '{ "document_name": "jsonDoc", +
                                     "number_prefix": "YAJL_", +
                                     "skip_nulls": true +
                                   }');
To try it you'll need to download the July 16 2022 (or newer if you're reading this in the future) version of YAJL from here:
https://www.scottklement.com/yajl/
jonboy49
Posts: 93
Joined: Wed Jul 28, 2021 8:18 pm

Re: Problem with import stream file json

Post by jonboy49 »

Nice Scott - I think I'll still put an IDEA forward to have DATA-INTO offer an option for compound elements but this will certainly help a lot of people who have encountered this issue.

Thanks.
Post Reply