Skip to content

fix(cache): Include mutable layer in cost calculation#22

Merged
darkcoderrises merged 1 commit intopredictable-labs:mainfrom
xqqp:predict_fix_cache_cost
Oct 22, 2025
Merged

fix(cache): Include mutable layer in cost calculation#22
darkcoderrises merged 1 commit intopredictable-labs:mainfrom
xqqp:predict_fix_cache_cost

Conversation

@xqqp
Copy link
Copy Markdown

@xqqp xqqp commented Oct 21, 2025

Included the size of the mutable layer in the cost calculation. Also included some more struct variables in the size calculation. This brings the configured size much closer to the actual size. With this change the occupied memory of Dgraph increases linearly, when changing the cache size. Before it exploded with cache-size higher than 4096.
Increased the default cache size to 4096 MiB, which should be similar a cache size of 1024 without this change.
Below are results from experiments with the cache size.

size-mb=1024

Dgraph process occupies up to 3.7 GiB of memory

heap:

  344.49MB 30.34% 30.34%   407.26MB 35.87%  github.com/hypermodeinc/dgraph/v25/posting.(*List).calculateUids
  130.37MB 11.48% 41.83%   130.37MB 11.48%  github.com/dgraph-io/ristretto/v2.newCmRow
   83.20MB  7.33% 49.15%    83.20MB  7.33%  github.com/dgraph-io/badger/v4/skl.newArena
   69.50MB  6.12% 55.28%    69.50MB  6.12%  github.com/hypermodeinc/dgraph/v25/posting.(*List).Uids.func1
   65.16MB  5.74% 61.02%    65.16MB  5.74%  github.com/dgraph-io/ristretto/v2/z.(*Bloom).Size
   53.27MB  4.69% 65.71%    53.27MB  4.69%  github.com/hypermodeinc/dgraph/v25/posting.(*List).calculateUids.func1

size-mb=2048

Dgraph process occupies up to 5.4 GiB of memory

heap:

  637.07MB 33.68% 33.68%   699.33MB 36.97%  github.com/hypermodeinc/dgraph/v25/posting.(*List).calculateUids
  260.63MB 13.78% 47.46%   260.63MB 13.78%  github.com/dgraph-io/ristretto/v2.newCmRow
  130.04MB  6.88% 54.34%   130.04MB  6.88%  github.com/dgraph-io/ristretto/v2/z.(*Bloom).Size
   99.01MB  5.23% 59.57%    99.01MB  5.23%  github.com/hypermodeinc/dgraph/v25/posting.(*List).Uids.func1
   89.89MB  4.75% 64.32%    89.89MB  4.75%  github.com/dgraph-io/ristretto/v2.(*lockedMap[go.shape.*uint8]).Set
   83.20MB  4.40% 68.72%    83.20MB  4.40%  github.com/dgraph-io/badger/v4/skl.newArena

size-mb=4096

Dgraph process occupies up to 8.4 GiB of memory

heap:

 1343.44MB 41.61% 41.61%  1404.71MB 43.51%  github.com/hypermodeinc/dgraph/v25/posting.(*List).calculateUids
  520.15MB 16.11% 57.72%   520.15MB 16.11%  github.com/dgraph-io/ristretto/v2.newCmRow
     260MB  8.05% 65.77%      260MB  8.05%  github.com/dgraph-io/ristretto/v2/z.(*Bloom).Size
  136.70MB  4.23% 70.01%   136.70MB  4.23%  github.com/dgraph-io/ristretto/v2.(*lockedMap[go.shape.*uint8]).Set
  106.51MB  3.30% 73.31%   106.51MB  3.30%  reflect.New
   83.20MB  2.58% 75.88%    83.20MB  2.58%  github.com/dgraph-io/badger/v4/skl.newArena

size-mb:8192

Dgraph process occupies up to 13.3 GiB of memory

heap:

 2539.78MB 45.10% 45.10%  2601.10MB 46.19%  github.com/hypermodeinc/dgraph/v25/posting.(*List).calculateUids
 1040.01MB 18.47% 63.56%  1040.01MB 18.47%  github.com/dgraph-io/ristretto/v2.newCmRow
     520MB  9.23% 72.80%      520MB  9.23%  github.com/dgraph-io/ristretto/v2/z.(*Bloom).Size
  223.52MB  3.97% 76.77%   223.52MB  3.97%  reflect.New
  157.97MB  2.81% 79.57%   157.97MB  2.81%  github.com/dgraph-io/ristretto/v2.(*lockedMap[go.shape.*uint8]).Set
  136.01MB  2.42% 81.99%   259.03MB  4.60%  github.com/hypermodeinc/dgraph/v25/posting.copyList

@xqqp xqqp force-pushed the predict_fix_cache_cost branch from ee214b7 to 1bbdc74 Compare October 21, 2025 20:12
Comment thread posting/size.go
@darkcoderrises
Copy link
Copy Markdown
Collaborator

Minor change asked for. Thanks a lot for the fix.

@xqqp xqqp force-pushed the predict_fix_cache_cost branch from 1bbdc74 to 97204c2 Compare October 22, 2025 10:15
@darkcoderrises darkcoderrises merged commit 89c22e7 into predictable-labs:main Oct 22, 2025
5 of 9 checks passed
@xqqp xqqp deleted the predict_fix_cache_cost branch December 1, 2025 20:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants