GitHub - CAPCOM-TD-OSS/REDox: High-performance, token-based structured data engine for .NET. A core component of REX, the technology behind CAPCOM's next-generation game engine.

RE:Dox

1. What is RE:Dox?

RE:Dox is a high-performance, token-based structured data engine for .NET, developed as one of the core components of 'REX' Technology for CAPCOM's next-generation game engine.

RE:Dox is not just a JSON serializer. It parses structured data into a compact, fixed-size token DOM / IR, then uses that structural index for reading, editing, format conversion, serialization, deserialization, and automatic parallel deserialization.

JSON / JSON5 / CBOR / MessagePack / TOML / XML / HTML / CSV / INI / DOX
        ↓
Compact token DOM / IR
        ↓
Reader / Writer / Serializer / Deserializer
        ↓
.NET objects, JSON, CBOR, MessagePack, TOML, XML, HTML, DOX, ...

The same token representation acts as both a compact parsed document model and a common structural layer shared by serializers, converters, and supported formats.

2. Why should I care?

RE:Dox combines three goals that usually conflict:

• the parse speed of a tape DOM (System.Text.Json.JsonDocument),
• the editability of a node DOM (System.Text.Json.Nodes.JsonNode),
• the flexibility of Newtonsoft.Json (position-independent $type, $id / $ref, out-of-order constructor binding).

What you get:

• High-performance serialization and deserialization — on the benchmark datasets below, up to ~1.8x faster sequential deserialization, up to ~2.8x faster automatic parallel deserialization, and up to ~1.6x faster serialization than System.Text.Json.
• Lower allocation on tested workloads — for example, canada.json deserializes with about 2.56 MB allocated by RE:Dox versus about 8.53 MB by System.Text.Json. JSON, JSON5, CBOR, MessagePack, INI, and DOX are integrated with the core model; TOML, XML, HTML, and CSV are currently preview components.
• One converter model across formats — DataConverter<T> targets the format-agnostic DataReader / DataWriter abstractions instead of a specific wire format.
• Mutable token DOM — edit objects and arrays without replacing the document with a heavyweight managed object tree.
• Trivia-preserving JSON5 editing — comments and other preserved trivia can survive document edits and re-encoding.
• Compatibility layers for System.Text.Json, Newtonsoft.Json, and DataContractJsonSerializer.
• Apache-2.0 licensed.

3. 30-second example

Install the core package:

dotnet add package CAPCOM.REDox

Serialize, deserialize, parse, and edit through the same API surface:

using REDox.Json;

var player = new Player
{
    Name = "Leon",
    Level = 42,
    Items = ["Handgun", "Green Herb"]
};

// Serialize / deserialize
var json = JsonSerializer.Serialize(player);
var restored = JsonSerializer.Deserialize<Player>(json);

// Parse into a token DOM and edit it in place
using var doc = JsonDocument.Parse(json);

var root = doc.RootElement.AsObject();
root["Name"] = "Claire";            // replace
root.Add("Hp", 100);                // add
root.Remove("Level");               // remove

var items = root["Items"].AsArray();
items.Add("First Aid Spray");       // append
items.Insert(0, "Knife");           // insert
items.RemoveAt(1);                  // remove by index

var edited = doc.RootElement.ToJsonString();

public sealed class Player
{
    public string? Name { get; set; }
    public int Level { get; set; }
    public string[] Items { get; set; } = [];
}

Environment: BenchmarkDotNet, .NET 10 (X64 RyuJIT x86-64-v3), AMD Ryzen Threadripper PRO 5975WX, Windows 11.

Summary: speedup vs System.Text.Json JsonSerializer

These ratios describe the benchmark datasets shown below; they are not universal performance guarantees.

Deserialize: canada.json (numeric-heavy, ~2.2 MB)

Deserialize: Mean / Allocated

Serialize: Mean / Allocated

Reproduce with:

dotnet run -c Release --project benchmarks/REDox.Json.Benchmarks

Benchmark results depend on data shape, target type, runtime, CPU, and serializer options. Always benchmark with your own workload.

5. Architecture

A 64-bit dual-mode token at the core

Every value is described by a fixed-size 64-bit token. An extension bit selects how the payload is interpreted:

extension bit = 0  →  payload interpretation is delegated to the document/format layer
                      (source-backed, format-specific payload; many formats, one token shape)

extension bit = 1  →  payload interpretation is fixed by RE:Dox, and control tokens enable
                      format-agnostic editing (insert / remove / replace)

This allows one token structure to represent both a source-backed view of format data and a RE:Dox-owned editable DOM. Where the document/format layer can retain the original payload, unchanged values can continue to reuse source slices instead of being eagerly materialized.

A token can represent null, boolean, integer, floating point, string, binary, timestamp, big number, array, map/object, trivia (comment/whitespace), or format-specific extension data. It may store inline data, a source offset/length, a container count, link information, or an extension id.

Asymmetric read/write design

Reading and writing have different information requirements, so RE:Dox uses different paths for each direction:

Serialize    : DataWriter, one pass, written directly to the output buffer
               (information is known → no look-ahead, no intermediate DOM)

Deserialize  : DataReader over the token DOM, random access by token id
               (information is unknown → look-ahead, out-of-order, context lookup)

On the write path the value is already known, so RE:Dox writes straight to the buffer using bulk (WriteValues) and fused-property (WriteProperty*) helpers — no intermediate object tree is built.

On the read path the structure is unknown, so RE:Dox first builds a compact token DOM. Because the structure is then fully addressable, RE:Dox can:

• pre-size arrays and collections;
• know each element's token id before materialization;
• resolve $type regardless of where it appears in the object;
• collect constructor arguments out of order and bind them once (records and primary constructors);
• resolve $id / $ref cycles using the whole-document context;
• choose sequential or parallel deserialization automatically;
• decode strings and numbers only when requested;
• reuse raw source slices for unchanged values.

Lazy decoding and on-demand materialization

Tokens can store offsets and lengths into the original source buffer. Strings, numbers, timestamps, and binary values are decoded only when requested. Elements are lightweight handles over a document and token id, while object and array views are materialized only when needed.

Mutable tape DOM

A tape DOM is normally read-only because its logical structure is represented by a flat token sequence. RE:Dox adds indirection at container views: DArray, DObject, and DMap keep value slots that can be re-linked, a free list reuses emptied token slots, and extension/control tokens carry edit state.

This preserves the cache-friendly characteristics of a sequential token representation while providing node-like operations such as insert, remove, and replace without rebuilding the entire document for each edit.

Unified converters

Converters (DataConverter<T>) are written against the format-agnostic DataReader / DataWriter abstractions rather than a specific format implementation.

Conceptually:

                 DataConverter<T>
                        │
               DataReader / Writer
                /       |        \
             JSON      CBOR    MessagePack   ...

This allows converter logic to be shared by formats that participate in the common reader/writer model. Instead of duplicating equivalent conversion logic for each type-format pair, RE:Dox can reuse one converter implementation across those formats — also providing a foundation for future AOT / source-generator support.

6. Formats/packages

NuGet package IDs use the CAPCOM.REDox.* prefix. C# namespaces use REDox.*.

Preview packages use NuGet pre-release versions such as 0.1.0-preview.1; the package name itself does not include -preview. Preview components may change API behavior before stable release.

Install public packages:

dotnet add package CAPCOM.REDox
dotnet add package CAPCOM.REDox.Cbor
dotnet add package CAPCOM.REDox.MessagePack
dotnet add package CAPCOM.REDox.Dynamic
dotnet add package CAPCOM.REDox.Serialization.DataContractJson
dotnet add package CAPCOM.REDox.Ini

Install preview packages with --prerelease:

dotnet add package CAPCOM.REDox.Serialization.SystemTextJson --prerelease
dotnet add package CAPCOM.REDox.Serialization.NewtonsoftJson --prerelease
dotnet add package CAPCOM.REDox.Toml --prerelease
dotnet add package CAPCOM.REDox.Xml --prerelease
dotnet add package CAPCOM.REDox.Html --prerelease
dotnet add package CAPCOM.REDox.Csv --prerelease

CBOR

using REDox.Cbor;

byte[] cbor = CborSerializer.Serialize(new Player
{
    Name = "Jill",
    Level = 30,
    Items = ["Lockpick"]
});

var player = CborSerializer.Deserialize<Player>(cbor);

// Parse into the token DOM and re-encode
using var doc = CborDocument.Parse(cbor);
byte[] encoded = CborDocument.Encode(doc.RootElement);

MessagePack

using REDox.MessagePack;

byte[] msgpack = MessagePackSerializer.Serialize(new Player
{
    Name = "Chris",
    Level = 35,
    Items = ["Knife", "First Aid Spray"]
});

var player = MessagePackSerializer.Deserialize<Player>(msgpack);

using var doc = MessagePackDocument.Parse(msgpack);
byte[] encoded = MessagePackDocument.Encode(doc.RootElement);

Dynamic

using REDox;
using REDox.Dynamic;
using REDox.Json;

using var doc = JsonDocument.Parse("""{"name":"Leon","level":40,"stats":{"alive":true}}""");

var player = doc.RootElement.AsDynamic()!;
var name = (string)player.name;
var level = (int)player.level;
var alive = (bool)player["stats"]["alive"];

// Build a new object dynamically
var obj = new DObject().AsDynamic();
obj.Name = "Ada";
obj.Items = new[] { "Hookshot" };

string json = JsonSerializer.Serialize(obj); // {"Name":"Ada","Items":["Hookshot"]}

Cross-format conversion

Formats that share the common token IR can be converted through the document model:

using REDox.Cbor;
using REDox.Json;

using var doc = JsonDocument.Parse(json);
byte[] cbor = CborDocument.Encode(doc.RootElement);

JSON5  → token IR → JSON
JSON   → token IR → CBOR
CBOR   → token IR → MessagePack
TOML   → token IR → JSON
XML    → token IR → JSON-like structural representation

Formats with different native data models may use a structural projection rather than a lossless semantic round-trip.

7. Advanced features

Automatic parallel deserialization

The deserializer inspects each array's element count and token range before materializing objects. When an array is large enough, element token ids are collected and each element is deserialized into its corresponding index in parallel. Small arrays stay sequential to avoid parallel scheduling overhead.

using REDox;
using REDox.Json;
using REDox.Serialization;

var settings = new DoxSerializerSettings
{
    ParallelOptions = new ParallelDeserializeOptions
    {
        ParallelDeserializeEnabled = true,
        MinimumNumberOfElements = 1024,
        MinimumNumberOfTokens = 4096
    }
};

var players = JsonSerializer.Deserialize<Player[]>(json, settings);

JSON5 with comment and trivia preservation

using REDox.Json;

var json5 = """
{
  // Player name
  name: 'Leon',

  // Inventory
  items: [
    'Handgun',
    'Green Herb',
  ],
}
""";

using var doc = Json5Document.Parse(json5, options: new Json5DocumentOptions
{
    PreserveTrivia = true,
    EnableValueValidation = true
});

var root = doc.RootElement.AsObject();
root["name"] = "Claire";

var output = Json5Document.EncodeToString(doc.RootElement, new Json5WriteOptions
{
    PreserveTrivia = true
});

Async streaming of NDJSON / JSON sequences

JsonSequence reads a stream of JSON values (NDJSON / JSON Lines, or a large top-level array) asynchronously, one value at a time, without loading the whole input into memory. For NDJSON, enable UseNewlineDelimitedFormat in JsonDocumentOptions.

using REDox.Json;

var options = new JsonDocumentOptions { UseNewlineDelimitedFormat = true };

await using var stream = File.OpenRead("players.ndjson");

// Deserialize each line into a .NET object
await foreach (var player in JsonSequence.DeserializeAsync<Player>(stream, options: options, cancellationToken: ct))
{
    Console.WriteLine(player?.Name);
}

To work with the token DOM directly, use ParseAsync. Without UseNewlineDelimitedFormat, a top-level JSON array is streamed element by element. The yielded DElement is backed by a reused document and is valid only until the next iteration; call Clone() or convert it with To<T>() if it must be retained.

using REDox;
using REDox.Json;

await using var stream = File.OpenRead("events.json"); // [ { "level": "info", ... }, { "level": "error", ... }, ... ]

var errors = new List<DElement>();

await foreach (var element in JsonSequence.ParseAsync(stream, cancellationToken: ct))
{
    if (element.TryGetProperty("level", out var level) && level.GetString() == "error")
    {
        errors.Add(element.Clone());
    }
}

DOX format and deep clone

DOX is a binary layout designed to mirror the in-memory token array closely. Constructing a document from DOX requires little structural reconstruction, which also makes DOX suitable for serialization-based deep clone: the result is an independent, editable copy rather than a shared read-only view.

using REDox;

byte[] dox = DoxSerializer.Serialize(player);
var clone = DoxSerializer.Deserialize<Player>(dox);

Compatibility layers

Compatibility packages allow existing serializer options to be adapted to RE:Dox.

System.Text.Json

using REDox.Serialization.SystemTextJson;

var stjSettings = new SystemTextJsonSerializerSettings(new System.Text.Json.JsonSerializerOptions
{
    PropertyNamingPolicy = System.Text.Json.JsonNamingPolicy.CamelCase
});

Newtonsoft.Json

using Newtonsoft.Json;
using REDox.Serialization.NewtonsoftJson;

var nsjSettings = new NewtonsoftJsonSerializerSettings(new JsonSerializerSettings
{
    NullValueHandling = NullValueHandling.Ignore
});

DataContractJsonSerializer

using REDox.Serialization.DataContractJson;

var dcjSettings = new DataContractJsonSerializerSettings(
    new System.Runtime.Serialization.Json.DataContractJsonSerializerSettings());

Requirements

• .NET 10 or later.

How to build

1. Clone with submodules

Test and benchmark data (simdjson-data, JSONTestSuite, json5-tests, toml-test) live under external/ as git submodules.

git clone --recurse-submodules https://github.com/CAPCOM-TD-OSS/REDox.git
cd REDox

If you already cloned without submodules:

git submodule update --init --recursive

2. Build

dotnet build REDox.slnx -c Release

3. Run tests

dotnet test REDox.slnx -c Release

4. Run benchmarks

Benchmarks use BenchmarkDotNet and must be run in Release configuration.

dotnet run -c Release --project benchmarks/REDox.Json.Benchmarks

Third-party components

The RE:Dox library packages do not bundle third-party code. The following third-party projects are referenced by tests and benchmarks. Each is distributed under its own license.

Test

License

RE:Dox is released under the Apache License 2.0.

Status

RE:Dox is under active development. APIs, package boundaries, and preview format support may evolve before the first stable public release.

RE:Dox is developed as part of 'REX' Technology for CAPCOM's next-generation game engine.