Skip to content

Recover from out-of-memory, thrown exceptions, and aborts in plugins and firmware#601

Open
danngreen wants to merge 39 commits into
mainfrom
oom-recovery-2
Open

Recover from out-of-memory, thrown exceptions, and aborts in plugins and firmware#601
danngreen wants to merge 39 commits into
mainfrom
oom-recovery-2

Conversation

@danngreen

@danngreen danngreen commented Jul 22, 2026

Copy link
Copy Markdown
Member

This is a large refactor to the dynamic memory system that prevents the MM from crashing if a plugin tries to allocate more memory than is available, or otherwise hits abort(). If a MM does not have enough memory available to load a module, a notification will be shown to the user and the module will not be added. For modules in plugins with SDK 2.2 or earlier, memory may be leaked. For plugins SDK v2.3 or later, exception handling is enabled, so the plugin's stack can be unwound and the destructors will be called (hopefully cleaning up any memory used).

Also, if a plugin calls abort() (due to an uncaught exception or a terminate from some other error, the MM will intercept this and try to recover gracefully.

It also catches internal OOM and other exceptions thrown, and handles them reasonably.

Changes:

Exceptions:

  • exceptions are enabled for firmware
  • SDK bumped up to v2.3, which enabled exceptions for plugins
  • exception index (exidx) "bridge" between plugins and the MM host, so the unwinder can traverse the plugin<->host boundary. We keep a registry of each plugin's exidx tables, and inour exception unwinder if the faulting address is in the plugin's code then we pass the plugin's exception table to the gnu unwinder.

Heap

  • Plugins get their own heap arena, separate from firmware's heap. This helps prevent a crashing plugin module from corrupting firmware
  • The plugin arena and the main heap are dynamically sized, based on need. Each one can grow in 16MB+ increments as needed.
  • free()/free_r() is --wraped with our own version that checks which arena/heap the address is in. If it's in the plugin arena, then it calls the plugin arena free(). Otherwise it calls the main newlib free()
  • We re-direct plugin calls to malloc/realloc/etc to our own wrappers that which use an TLSF allocator for the plugin arena

Error/exception handling

  • If an exception is thrown:

    • when creating a module (allocating memory and calling its constructor)
    • loading a plugin (calling its init()->addModel() functions, instantiating a Module and ModuleWidget)
    • running an AsyncThread
    • loading the plugin executable code, assets and metadata
      ...then we report it to the user via a Notification and unwind the stack (hopefully if the module uses RAII or std containers then memory will get cleaned up)
  • If a module calls abort() because

    • it failed to allocate memory but had exceptions disabled
    • it terminated via C-style functions
    • it threw an exception that it didn't catch
    • any other reason it hit terminate->abort()
      ...then we use the C-style setjmp/longjmp to try to clean up (unload what we can, abort the process).
  • If an exception is thrown in the audio thread, we still will crash (no recovery)

danngreen added 30 commits July 8, 2026 23:02
- newlib wrapper functions for free/free_r/realloc/realloc_r that check ptr and route
…a instead

TLSF cannot easily be forced to align to > sizeof(void*), so instead we
just force alignment in plugin arena
Catches OOM in plugin's init() and also in loading the metadata.
OOM in allocating space for the assets and/or the .so code are caught elsewhere
They are just printed to console, so a fatal error could cause a crash.
Because the preview buffer is shared between pages
-AbortRescue holds the jmp_buf, activates it when a context calls "arm()"
-One AbortRescue can be active per context (previous one is swapped back in when one is destructed)
-Disabled if malloc or tslf arena lock is held (hard crash if exception throws inside malloc/tslf alloc)
-
If the module crashes/oom in its constructor, then we'll catch the abort()
@danngreen

Copy link
Copy Markdown
Member Author

This bumps the firmware size over 9.5MB, so it requires #602 before it can be run from NOR flash (running from SD card or JTAG debugger is fine).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant