Hi everyone,
I’ve been working with HTTPAPI on IBM i for some time now and recently ran into a situation where I need to handle large JSON responses from an API (in the order of several megabytes). While HTTPAPI can fetch and parse the data fine for smaller payloads, I’m concerned about memory usage and performance when the response is large.
Has anyone developed a strategy for:
Streaming or chunking the response instead of loading everything at once?
Efficiently parsing large JSON with HTTPAPI + YAJL?
Any tips on avoiding out of memory or slow performance?
Any insights, examples, or links to useful code snippets would be much appreciated. Thanks in advance!
Best Practices for Handling Large JSON Responses with HTTPAPI
-
Scott Klement
- Site Admin
- Posts: 964
- Joined: Sun Jul 04, 2021 5:12 am
Re: Best Practices for Handling Large JSON Responses with HTTPAPI
Are you running into problems? if so, can you describe them?
HTTPAPI doesn't actually know how to process JSON. (you may be confusing it with YAJL?)
When it reads a response from an HTTP server and saves the result to disk, it only loads 32k into memory at once, then writes it and reads the next 32k. This is a tiny amount of memory, i can't imagine it causing problems.
If you are asking it to read the contents into memory/variable instead of to disk, well, then of course it will all be in memory. But the IBM i operating system handles memory storage very very effectively, so thsi shouldn't be a problem unless you are running many simultaneous jobs. (In which case you would simply need to use a save to disk strategy instead.)
What problem are you having?
HTTPAPI doesn't actually know how to process JSON. (you may be confusing it with YAJL?)
When it reads a response from an HTTP server and saves the result to disk, it only loads 32k into memory at once, then writes it and reads the next 32k. This is a tiny amount of memory, i can't imagine it causing problems.
If you are asking it to read the contents into memory/variable instead of to disk, well, then of course it will all be in memory. But the IBM i operating system handles memory storage very very effectively, so thsi shouldn't be a problem unless you are running many simultaneous jobs. (In which case you would simply need to use a save to disk strategy instead.)
What problem are you having?