Convert Json objects Collection to Array

Question

I have a json file which contains a collection of numerous JSON objects. A sample format is given below:

{"ID": 123,"Name": "TEST-1","val11": {},"url": "test1.com","val12": []}
{"ID": 456,"Name": "TEST-2","val21": {},"url": "test2.com","val22": []}
{"ID": 789,"Name": "TEST-3","val31": {},"url": "test3.com","val32": []}

As you see, it is not an array ([ ] and commas missing). I need to convert this into a valid JSON array. The code that I tried is:

%dw 2.0
output application/json
var PayloadSplit = payload splitBy('\n')
var PayloadArray = (PayloadSplit map (value, index) -> read(value, 'application/json'))
---
PayloadArray

This works fine for a small sized payload. However, if I try to perform this on the entire file (size about 320 MB with ~20k JSON objects), it fails with a java.lang.OutOfMemoryError: Java heap space error. Is there a way to overcome this? Or can I split the main file into multiple files and then try this (in a ForEach Loop perhaps?). Please advise

Edit1 - Attaching the mule flow below:

    <flow name="convert-object-to-array-test1Flow" doc:id="0645e9bd-7f77-4b1e-93d0-dedd9d154ef7" >
        <http:listener doc:name="Listener" doc:id="551cd3b6-e4c8-4b7a-aff3-305effbe8a8b" config-ref="HTTP_Listener_config" path="/file"/>
        <file:read doc:name="Read" doc:id="21a310c1-5887-4bc0-83b9-b8968e145f0d" path="C:\Desktop\NDJsonSample.json" outputMimeType="application/ndjson" />
        <ee:transform doc:name="Transform Message" doc:id="95235c56-2f5a-4f39-ba96-8be7c4e501b5" >
            <ee:message >
                <ee:set-payload ><![CDATA[%dw 2.0
output application/json
---
payload]]></ee:set-payload>
            </ee:message>
        </ee:transform>
        <logger level="INFO" doc:name="Logger" doc:id="935530dd-17fd-41c9-8fe0-1561ba3de703" />
    </flow>

machaval · Accepted Answer · 2020-07-02 18:40:37Z

7

DW already have support for this format. It is called ndjson. Please visit the documentation. You just need to set application/ndjson to the payload.

answered Jul 2, 2020 at 18:40

machaval

5,06916 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Triumph Spitfire Over a year ago

Thanks for your answer. Please refer my flow in the question. However, the output that I am getting is not in an expected format. "{\"ID\": 123,\"Name\": \"TEST-1\",\"val11\": {},\"url\": \"test1.com\",\"val12\": []}\r\n{\"ID\": 456,\"Name\": \"TEST-2\",\"val21\": {},\"url\": \"test2.com\",\"val22\": []}\r\n{\"ID\": 789,\"Name\": \"TEST-3\",\"val31\": {},\"url\": \"test3.com\",\"val32\": []}"

Triumph Spitfire Over a year ago

It worked after I changed the mime type to "application/x-ndjson" Thank you!

Collectives™ on Stack Overflow

Convert Json objects Collection to Array

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related