Table of Contents

Asynchronous sequence streaming

Building on the incremental async reader, Nerdbank.Json can stream a sequence of values as an IAsyncEnumerable<T>, reading only enough of the input to produce each element on demand.

[GenerateShape]
public partial record LogEntry(long Timestamp, string Message);

[GenerateShape]
public partial record LogFile(string Source, LogEntry[] Entries);

public static class AsyncSequenceSample
{
    // Stream the elements of a large top-level JSON array without buffering the whole document.
    public static async Task ProcessArrayAsync(Stream stream, JsonSerializer serializer)
    {
        await foreach (LogEntry? entry in serializer.DeserializeEnumerableAsync<LogEntry>(stream, new() { Format = JsonSequenceFormat.JsonArray }))
        {
            System.Console.WriteLine(entry!.Message);
        }
    }

    // Stream newline-delimited JSON (JSON Lines): one value per line.
    public static async Task ProcessNdjsonAsync(PipeReader reader, JsonSerializer serializer, CancellationToken cancellationToken)
    {
        await foreach (LogEntry? entry in serializer
            .DeserializeEnumerableAsync<LogEntry>(reader, new() { Format = JsonSequenceFormat.NDJson })
            .WithCancellation(cancellationToken))
        {
            System.Console.WriteLine(entry!.Timestamp);
        }
    }

    // Stream an array selected by a typed path inside an object envelope. The preamble ("source") and the
    // trailing envelope are consumed incrementally; only one entry is materialized at a time.
    public static async Task ProcessEnvelopeAsync(Stream stream, JsonSerializer serializer)
    {
        await foreach (LogEntry? entry in serializer.DeserializePathEnumerableAsync<LogFile, LogEntry>(stream, new JsonSerializer.StreamingEnumerationOptions<LogFile, LogEntry>(f => f.Entries)))
        {
            System.Console.WriteLine(entry!.Message);
        }
    }

    // Serialize an asynchronous source as a top-level JSON array with backpressure.
    public static async Task WriteArrayAsync(Stream stream, JsonSerializer serializer)
    {
        await serializer.SerializeEnumerableAsync<LogEntry>(stream, ProduceAsync(), new() { Format = JsonSequenceFormat.JsonArray });

        static async IAsyncEnumerable<LogEntry> ProduceAsync()
        {
            for (int i = 0; i < 100; i++)
            {
                await Task.Yield();
                yield return new LogEntry(i, $"entry {i}");
            }
        }
    }
}

Framing modes

DeserializeEnumerableAsync accepts JsonSerializer.SequenceOptions to select framing through its Format property:

  • Newline-delimited JSON (NDJSON / JSON Lines) is the default, NDJson. It requires a line terminator between values; two values on one line are rejected with a JsonSerializationException whose cause is a FormatException.
  • Top-level JSON array, JsonArray, streams each element of a [ ... ] document. A top-level null yields an empty sequence. The dedicated JSON-specific DeserializeArrayAsync API remains available.

For a sequence inside an envelope, use DeserializePathEnumerableAsync with JsonSerializer.StreamingEnumerationOptions<TRoot, TElement> containing a member-access expression such as e => e.Items. The preamble is parsed and skipped incrementally; full enumeration drains and validates the remainder of the envelope and rejects trailing data. Expression paths can traverse intermediate union or custom-converter representations. Raw JsonPath overloads are also available for wire-name navigation through plain objects and arrays.

Typed paths throw JsonPathNotFoundException when the path is absent or resolves to null, unless EmptySequenceForUndiscoverablePath is enabled. Raw paths use MissingPathBehavior to control missing paths and yield an empty sequence for a selected null. A selected non-array value throws.

Streaming semantics and ownership

  • No read-ahead beyond framing. An element is produced as soon as its bytes arrive; the enumerator does not read the following separator, closing bracket, or next element until the consumer asks for more. Memory is bounded by the largest single element, never the whole sequence or document.
  • Cancellation is honored via EnumeratorCancellationAttribute, so WithCancellation on the enumerator flows through.
  • Early disposal — breaking out of the loop disposes the enumerator, which releases buffered bytes but does not read to the end or validate trailing data.
  • Stream and pipe ownership is explicit. Caller-supplied streams remain open. Enumerable and dedicated array APIs leave caller-supplied pipes open, as do raw-path overloads. Typed path enumeration completes its input pipe by default; set LeaveOpen to retain ownership. Internally created pipe adapters are completed when enumeration ends.
  • Reference preservation is not supported for sequence streaming (a clear NotSupportedException is thrown), because reference metadata wraps every value.

Serializing a sequence

SerializeEnumerableAsync writes an IAsyncEnumerable<T> using the same JsonSerializer.SequenceOptions framing choices, defaulting to NDJSON. NDJSON output writes a newline after each value and does not indent values across multiple lines. The dedicated SerializeArrayAsync API writes a top-level JSON array. Both APIs flush periodically so a long or unbounded source streams with bounded memory and backpressure.

Edit this page