json.mqh: fix O(n^2) JSON serialization (46s -> 0.5s for a month of M1 candles) - #312
Open
biohazardxxx wants to merge 1 commit into
Open
json.mqh: fix O(n^2) JSON serialization (46s -> 0.5s for a month of M1 candles)#312biohazardxxx wants to merge 1 commit into
biohazardxxx wants to merge 1 commit into
Conversation
JSONObject/JSONArray::toString() built output by repeated string concatenation, copying the entire accumulated string per element - O(n^2) in the output size. Serializing 30k MqlRates (~6 MB JSON) took ~46 s; a month of M1 candles was effectively unusable. String appends cannot be made linear in MQL: += and StringAdd re-allocate to exact size and discard reserved capacity (measured: StringReserve + StringAdd was 20x SLOWER due to a full realloc+copy per append). Instead, a JsonBuffer class writes characters into a pre-grown ushort array (in-place StringToShortArray, amortized-doubling ArrayResize) and converts to a string once at the end. Measured (XAUUSD M1, live terminal): 32,000 bars 50,968 ms -> 694 ms; the Aug-2021 month range (30,281 bars) 46,224 ms -> 548 ms (~84x). Scaling is now flat ~0.02 ms/bar. Applied to both mq5/json.mqh and mq4/json.mqh (same defect). Co-Authored-By: Claude Fable 5 <[email protected]>
hosseinkhojany
pushed a commit
to hosseinkhojany/mtapi
that referenced
this pull request
Jul 22, 2026
…ation fix # Conflicts: # mq4/MtApi.ex4 # mq5/MtApi5.ex5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
JSONObject::toString()andJSONArray::toString()injson.mqhbuild their output by repeated string concatenation — every element copies the entire accumulated string, making serialization O(n²) in the output size. For large responses this dominates end-to-end latency: serializing ~30kMqlRates(~6 MB of JSON, i.e. one month of M1 candles viaCopyRates) took ~46 seconds inside the expert.This PR makes serialization linear:
JsonBufferclass: writes characters into a pre-grownushortarray (in-placeStringToShortArray, amortized-doublingArrayResize) and converts to a string once at the end.virtual void serialize(JsonBuffer &buf)onJSONValue;JSONObject/JSONArrayoverride it to serialize recursively into the single buffer. Scalar types keep their existingtoString()(used by the baseserialize).JSONObject::toString()/JSONArray::toString()keep their signatures and now delegate toserialize— no call-site changes anywhere.mq5/json.mqhandmq4/json.mqh(same defect in both).Why an array buffer instead of string appends
Measured on MT5 build 5830: MQL string appends cannot be made linear.
+=/StringAddre-allocate the string to its exact new size and discard any capacity set withStringReserve, so an "efficient append" still copies the whole string per call — aStringReserve+StringAddvariant of this fix measured 20× slower than the original code (the re-reserve after every append doubles the copying). Theushort[]-backed buffer is the reliable in-place path.Measured results (XAUUSD M1, live MT5 terminal, build 5830)
Before: ms/bar grows 0.12 → 1.59 (quadratic). After: flat ~0.02 ms/bar (linear).
Testing
MtApi5.mq5andMtApi.mq4compile withResult: 0 errors, 0 warnings; recompiled.ex5/.ex4included (built against this branch'sjson.mqh).CalendarCountries,Print,PositionsTotal) confirming response JSON still parses correctly on the .NET side.🤖 Generated with Claude Code