Skip to content

json.mqh: fix O(n^2) JSON serialization (46s -> 0.5s for a month of M1 candles) - #312

Open
biohazardxxx wants to merge 1 commit into
vdemydiuk:devfrom
biohazardxxx:fix/mql-json-serialization-perf
Open

json.mqh: fix O(n^2) JSON serialization (46s -> 0.5s for a month of M1 candles)#312
biohazardxxx wants to merge 1 commit into
vdemydiuk:devfrom
biohazardxxx:fix/mql-json-serialization-perf

Conversation

@biohazardxxx

Copy link
Copy Markdown

Summary

JSONObject::toString() and JSONArray::toString() in json.mqh build their output by repeated string concatenation — every element copies the entire accumulated string, making serialization O(n²) in the output size. For large responses this dominates end-to-end latency: serializing ~30k MqlRates (~6 MB of JSON, i.e. one month of M1 candles via CopyRates) took ~46 seconds inside the expert.

This PR makes serialization linear:

  • New JsonBuffer class: writes characters into a pre-grown ushort array (in-place StringToShortArray, amortized-doubling ArrayResize) and converts to a string once at the end.
  • New virtual void serialize(JsonBuffer &buf) on JSONValue; JSONObject/JSONArray override it to serialize recursively into the single buffer. Scalar types keep their existing toString() (used by the base serialize).
  • JSONObject::toString() / JSONArray::toString() keep their signatures and now delegate to serialize — no call-site changes anywhere.
  • Applied identically to mq5/json.mqh and mq4/json.mqh (same defect in both).

Why an array buffer instead of string appends

Measured on MT5 build 5830: MQL string appends cannot be made linear. +=/StringAdd re-allocate the string to its exact new size and discard any capacity set with StringReserve, so an "efficient append" still copies the whole string per call — a StringReserve+StringAdd variant of this fix measured 20× slower than the original code (the re-reserve after every append doubles the copying). The ushort[]-backed buffer is the reliable in-place path.

Measured results (XAUUSD M1, live MT5 terminal, build 5830)

bars before after
1,000 116 ms 74 ms
8,000 3,012 ms 225 ms
16,000 15,668 ms 268 ms
32,000 50,968 ms 694 ms
Aug-2021 month range (30,281) 46,224 ms 548 ms (~84×)

Before: ms/bar grows 0.12 → 1.59 (quadratic). After: flat ~0.02 ms/bar (linear).

Testing

  • Both MtApi5.mq5 and MtApi.mq4 compile with Result: 0 errors, 0 warnings; recompiled .ex5/.ex4 included (built against this branch's json.mqh).
  • Runtime-verified against a live MT5 terminal: the scaling measurements above, plus functional checks on the same session (CalendarCountries, Print, PositionsTotal) confirming response JSON still parses correctly on the .NET side.
  • MQL4 change is source-identical; compile-verified only.

🤖 Generated with Claude Code

JSONObject/JSONArray::toString() built output by repeated string
concatenation, copying the entire accumulated string per element -
O(n^2) in the output size. Serializing 30k MqlRates (~6 MB JSON) took
~46 s; a month of M1 candles was effectively unusable.

String appends cannot be made linear in MQL: += and StringAdd
re-allocate to exact size and discard reserved capacity (measured:
StringReserve + StringAdd was 20x SLOWER due to a full realloc+copy per
append). Instead, a JsonBuffer class writes characters into a pre-grown
ushort array (in-place StringToShortArray, amortized-doubling
ArrayResize) and converts to a string once at the end.

Measured (XAUUSD M1, live terminal): 32,000 bars 50,968 ms -> 694 ms;
the Aug-2021 month range (30,281 bars) 46,224 ms -> 548 ms (~84x).
Scaling is now flat ~0.02 ms/bar.

Applied to both mq5/json.mqh and mq4/json.mqh (same defect).

Co-Authored-By: Claude Fable 5 <[email protected]>
hosseinkhojany pushed a commit to hosseinkhojany/mtapi that referenced this pull request Jul 22, 2026
…ation fix

# Conflicts:
#	mq4/MtApi.ex4
#	mq5/MtApi5.ex5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant