Scene render order

[ffreyer]

Description

This is a new attempt at #4150. If this gets merged it will also include changes from #4724 and changes for WGLMakie.

Fixes #5292
Related: #4904, #4650

Problems/Differences

Each backend handles scenes a little bit different. Using the example that is also added as a test (same as #4724):

Calling display(scene) after creating all scenes and plots will cause the full scene tree to be added. This includes scenes without plots. Scenes render in a depth first order (i.e. with a render(scene); render.(scene.children) recursion). Only WGLMakie treats plots under a cleared scene as covered.
The same save(filename, scene) command is creating different image sizes for me here... Is this a bug on breaking? master? Expected?

CairoMakie	GLMakie	WGLMakie (expected?)

Calling display(scene) immediately after creating the root scene, i.e. adding scenes and plots interactively causes backends to diverge more.
CairoMakie behaves the same as before, as it evaluates the scene tree whenever it creates an image. There is no interactive adding of scenes here.
GLMakie and WGLMakie do not interactively add scenes. Instead they add them once a plot is added to them. This means some scenes maybe missing from the render, and some may be in the wrong order.

CairoMakie	GLMakie	WGLMakie

Emptying a scene does not (interactively) remove child scenes from the backend screen. This currently happens when scenes are GC'd, which never happens if the scene is hold on to as a variable in the global scope. The test has variables for every scene, so calling empty!(scene) after displaying will leave behind scenes. (This again doesn't matter to CairoMakie as it reevaluates the scene tree on draw.) Emptying the first example yields:

CairoMakie	GLMakie	WGLMakie

Related issues: #5292

After this pr

Early display (second example)

CairoMakie	GLMakie	WGLMakie

After empty (third)

CairoMakie	GLMakie	WGLMakie

Goals and other things to consider

Rules:

1. The order of scenes in the scene tree directly corresponds to the draw order of scenes
   • this follows the order scenes are visited in with depth first search (first being the back, last the front)
   • scenes should have a z-index that controls where they are inserted in a parent scene (this is not translate!(scene, ...))
2. A scene with clear = true erases everything that is drawn behind it
3. A child scene is (expected to be) restricted to the viewport of the parent scene

With these rules we can naturally define groups of scenes that start with a cleared scene (back) and stop just before another cleared scene would be added (front). These groups fully define what is drawn on top of the cleared scene (if not occluded by another clear). This means that we don't need plot sorting (draw order) or depth buffering (rendering) beyond these groups.

I'm still unsure if we should restrict this further and make draw order and/or depth per scene. Per scene depth would require the render pipeline to partially run per scene, which seems unnecessarily difficult to me. Per scene draw order would simplify backend code (don't need to find and manage groups) but may occasionally complicate front end code. (If a back scene draws a scatter/text/lines plot and the front scene draws something else behind that, the back scene plot would need to be moved to the front scene (or a scene in front of it) to correctly blend its AA. Currently you can translate to fix this.)

Menu

Menu is a rather complicated example since it has a dropdown scene that should draw over other scenes regardless of when it is created. Currently this sort of works via translate!(scene, 0, 0, high_z), but that can cause z-fighting issues, e.g. #5292. With working clear (2), a scene that is added after the menu can also erase the dropdown content.

To fix this we need a way to keep the menu dropdown on top. This is why I want to add a z-index to scenes. Instead of having the dropdown scene as a child of blockscene, it would need to be a child of blockscene.parent with a high z-index so it remains in front of other scenes in the parent. Usually the parent is the figure, meaning that the dropdown would draw on top of all other blocks. But we could also add floating windows/subfigures in the future to which the Menu would naturally be constrained. And if we don't want this constraint (for Menu or another block) we could seek the root scene and directly add to that instead.

TODO

• GLMakie: re-group scenes when clear updates simplify to not GLScene instead of scene groups
• CairoMakie (render order/scene groups)
• WGLMakie (insert scenes)
• CairoMakie/General: Fix issues with scene-local sorting resulting in different render orders (either fix cases in Makie or group plots of transparent scenes in CairoMakie)
• WGLMakie: fix scene deletion
• maybe add zindex to plots to separate draw order from depth/position
• document (zindex, scene render order, plot render order, clear if it's not documented, update old docs)
• after feat(text)!: Pluggable text layout via Makie.layout_text #5717, if I add actual depth based plot sorting to CairoMakie, simplify transformation in text specs to :nothing (or directly include model transformations there)

Changes

Makie:

• added zindex to scene
• moved Menu's dropdown scene to blockscene.parent
• moved legendelement plot to Box scene

CairoMakie:

• creates back-to-front sorted list of scenes
• within that lists, collects scene groups that start with a cleared scene (back) and contains all un-cleared scenes up to the next cleared scene (in back to front order)
• sorts plots and renders like before within that group

WGLMakie:

• connected insert_scene! to Scene(parent, ....) creation in Makie
• fixed delete! for scenes

GLMakie:

• added GLScene as a wrapper around Scene and the associated renderobjects
• added RenderContext as a manager object for scenes and renderobjects (replacing renderlist, screens, scene2screen in Screen)
  • contains GLScenes in back-to-front order for rendering
• rendering:
  • clearing now happens as part of the render pipeline, in back to front order (renderloop still externally does a global clear for color, objectid, depth and stencil buffers)
  • plot sorting now happens per GLScene
  • rendering now happens per GLScene, front to back, with a stencil buffer successively excluding regions cleared by a scene whose plots have finished drawing
• added copy_to_screen(screen, framebuffer) function for easier compat with other window libraries

Type of change

Delete options that do not apply:

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist

• Added an entry in CHANGELOG.md (for new features and breaking changes)
• Added or changed relevant sections in the documentation
• Added unit tests for new algorithms, conversion methods, etc.
• Added reference image tests for new plotting functions, recipes, visual options, etc.

[ffreyer]

/benchmark 20

[asinghvi17]

There is definitely a use to allow scenes to merge with each other, in the case where you are translating a scene in order to apply that effect to all plots within it. I think we had a robot arm example which did something like that.

[ffreyer]

The scene merging thing only affects z order and not as much as I thought. Iirc, I originally had each scene complete a full render (with postprocessor), and then stacked the resulting images on top of each other. That would make z ordering per scene and could break stuff where a covering scene translates plots to be under the covered scenes content.

The code I have here/locally doesn't do that. It still renders all plots together, which means OpenGL's depth testing applies and you still get ordering across scenes from that. What changes is only when the render calls are made (across scenes), which may mess with the AA of scatter, text and lines. (I.e. a scatter markers AA may blend with a scatter marker that should be on top rather than the background)

WGLMakie has been rendering scenes independently already.

CairoMakie is tricky again, because it entirely relies on plot sorting for z order. So it either needs grouping or we need to adjust the few cases where it causes issues (see test failures)

[ffreyer]

/benchmark 20

[ffreyer]

/benchmark 20

[MakieBot]

Benchmark Results

SHA: 57ac2571608f439cbb07573d8c0d947984836b96 · 20 samples

Warning

These results are subject to substantial noise because GitHub's CI runs on shared machines that are not ideally suited for benchmarking.

[ffreyer]

/benchmark 20

[MakieBot]

Benchmark Results

SHA: cc7c164f7be766aa426488e4605d19738a53753a · 20 samples

Warning

These results are subject to substantial noise because GitHub's CI runs on shared machines that are not ideally suited for benchmarking.

[ffreyer]

I changed Axis a bit to rely on clear instead of a background poly if the backgroundcolor is opaque. This makes it unnecessary to translate inset scene but causes grid lines to disappear under transparent plots. Probably because of per-scene plot rendering

[ffreyer]

Before I forget what's on my mind about this pr:

First, I'm not sure about scene.children always reflecting scene render order. On one hand it wold be nice if scenes always render parent first, then child 1, then child 2, etc. But with zindices (which are required for e.g. Menu) scenes can be inserted before other scenes, which means scene.children[4] can change as more children are inserted. Though the same is true regardless when deleting scenes (which empty!(child) does, apparently), so maybe it's not a big deal?

Second I'm still all over the place when it comes to plot draw/render order (and depth order). CairoMakie doesn't have depth testing, so draw order must be back to front. W/GLMakie are more nuanced. Fully opaque plots would optimally render front to back (or do front to back depth prepass). Or it might be better to group them by plot type and scene to reuse uniforms and reduce shader program switches. Transparent plots (including natively anti-aliased plots like scatter) should render later, in back to front order.

Isolating scenes with respect to draw order (as I currently have) is also questionable. Only clear = true scenes currently clear the depth buffer, so depth order can (and usually does) involve plots from multiple scenes. Not having draw order act across scenes can thus create artifacts from not stacking plots correctly (i.e. transparent plots blending with the background rather than another plot introduced in a later scene).
I think the solution here might be to add a secondary clear_depth field for scenes. When false the scene merges with its predecessor and all grouped plots are sorted together, when true it clears depth and starts a new group to be sorted. (I have no idea how to do the grouping with WGLMakie btw)

There is also the problem that draw order is currently based on model translations, which is completely wrong in 3D. A better approach would be calculating the real depth from the plots bounding box. For plots where the bounding box spans a range of depth values (i.e. 3D) the layering can still be wrong with this. So I think there should also be an overwrite, but I'm not sure how that would integrate with defaults (which have to be calculated down the line, i.e. they're not super easy to view) and backend specific ordering (e.g. how do we combine this with opaque-transparent ordering?)

Third, I'm also not 100% convinced we should require child scenes to be within their parent scenes viewport. It's already mentioned in the architecture docs on master and would make some coverage analysis easier, but it's not actually required/used anywhere yet, afaik. Though the opposite is not required or used either. Enforcing it seems hard to me, since we'd have to compare parent and child viewports whenever they update if we want to be thorough. That might introduce more synchronization headaches.

[ffreyer]

To track some discussions I had with Simon:

1. Do immediately sort scene.children by zindex
2. I think we agreed that backends should be given freedom with respect to plot render order, but that plots should also be able to influence that order. Instead of a numeric zindex it might make more sense to have something like plot.after = other_plot_or_plots because it's unclear how a backend should react to/interact with a numeric zindex. I don't think we reached much of a conclusion on scene.clear_depth
3. I don't think we reached much of a conclusion on child.viewport in parent.viewport either