Is it possible to serialize to NDJSON (Newline Delimited JSON) using Json.NET? The Elasticsearch API uses NDJSON for bulk operations, and I can find nothing suggesting that this format is supported by any .NET libraries.
This answer provides guidance for deserializing NDJSON, and it was noted that one could serialize each row independently and join with newline, but I would not necessarily call that supported.
As Json.NET does not currently have a built-in method to serialize a collection to NDJSON, the simplest answer would be to write to a single TextWriter using a separate JsonTextWriter for each line, setting CloseOutput = false for each:
using Newtonsoft.Json;
using System.Collections.Generic;
using System.Text;
using System.IO;
public static partial class JsonExtensions
{
public static void ToNewlineDelimitedJson<T>(Stream stream, IEnumerable<T> items)
{
// Let caller dispose the underlying stream
using (var textWriter = new StreamWriter(stream, new UTF8Encoding(false, true), 1024, true))
{
ToNewlineDelimitedJson(textWriter, items);
}
}
public static void ToNewlineDelimitedJson<T>(TextWriter textWriter, IEnumerable<T> items)
{
var serializer = JsonSerializer.CreateDefault();
foreach (var item in items)
{
// Formatting.None is the default; I set it here for clarity.
using (var writer = new JsonTextWriter(textWriter) { Formatting = Formatting.None, CloseOutput = false })
{
serializer.Serialize(writer, item);
}
// https://web.archive.org/web/20180513150745/http://specs.okfnlabs.org/ndjson/
// Each JSON text MUST conform to the [RFC7159] standard and MUST be written to the stream followed by the newline character \n (0x0A).
// The newline charater MAY be preceeded by a carriage return \r (0x0D). The JSON texts MUST NOT contain newlines or carriage returns.
textWriter.Write("\n");
}
}
}
Sample fiddle.
Since the individual NDJSON lines are likely to be short but the number of lines might be large, this answer suggests a streaming solution to avoid the necessity of allocating a single string larger than 85kb. As explained in Newtonsoft Json.NET Performance Tips, such large strings end up on the large object heap and may subsequently degrade application performance.
JsonConvert.SerializeObject() internally creates both a StringWriter and a JsonTextWriter; see here for details. Since the individual JSON lines are likely to be short but the number of lines might be large, I suggested a streaming solution to avoid allocating a single string larger than 85kb as recommended here.
JsonTextWriter to force the JsonTextWriter to call Flush(). This is what @dbc was explaining in his comment reply about large strings. As an alternative to recreating the JsonTextWriter every time, you can move its using block outside of the foreach and explicitly call Flush() at the end of each iteration.
You could try this:
string ndJson = JsonConvert.SerializeObject(value, Formatting.Indented);
but now I see that you are not just wanting the serialized object to be pretty printed. If the object you are serializing is some kind of collection or enumeration, could you not just do this yourself by serializing each element?
StringBuilder sb = new StringBuilder();
foreach (var element in collection)
{
sb.AppendLine(JsonConvert.SerializeObject(element, Formatting.None));
}
// use the NDJSON output
Console.WriteLine(sb.ToString());