Skip to content

Commit

Permalink
Moved each design into it's on file for clarity;
Browse files Browse the repository at this point in the history 
Expanded more on the custom widget design & API, finished a rough prototype;
Made it more clear what the final API exposed to the user would be in each design case
  • Loading branch information
@bungoboingo
bungoboingo committed May 5, 2023
1 parent f350571 commit 1aadf4c
Show file tree
Hide file tree
Showing 5 changed files with 432 additions and 152 deletions.
214 changes: 62 additions & 152 deletions text/23-custom-pipelines.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -55,124 +55,55 @@ hops? Could we do this with a minimal unsafe abstraction layer? Should we even d
![](silvia.jpeg)

As you can see, this is a somewhat complex topic with a lot of tradeoffs between implementation strategies. Of
course, I would love if there was a better idea floating around out there that I haven't thought of! That being
said, **here are some of the concepts that you will need to understand before you can understand what this
RFC aims to address.**
course, I would love if there was a better idea floating around out there that I haven't thought of! That being said,
here are a few different designs & their concepts that would need to be introduced to Iced. I've broken them into
their own markdown files for an easier time reading!

🖼 **Custom Pipelines**
1) [Custom Primitive Pointer Design](designs/23-01-pointer.md)
2) [Custom Shader Widget Design](designs/23-02-widget.md)
3) [Multi-Backend Design](designs/23-03-multi-backend.md)

This is essentially just a regular ol' wgpu pipeline implementation, except one that isn't already integrated into
Iced! This can be as simple or complex as you want it to be. For example, in a prototype that I made to render a
simple triangle, this was as simple as this struct:

```rust
pub struct CustomPipeline {
pipeline: wgpu::RenderPipeline,
vertices: wgpu::Buffer,
}
```

In Iced, `Primitive`s are mapped internally to the appropriate pipeline, though have no direct relationship to each
other (for instance, a `Pipeline` doesn't have a primitive type `P`). Each is chosen manually for what is
appropriate. There is currently also no abstraction for what a `Pipeline` actually is; by their nature they are all
somewhat unique from each other, with minor underlying similarities (for example, every render pipeline must at some
point allocate some data to a `wgpu::Buffer` & submit a `draw` command).
All of these designs must be flagged under `wgpu`, unless we wanted to do some kind of fallback for tiny-skia which
I don't think is viable. What would we fall back to for the software renderer if a user tries to render a 3D object,
which tiny-skia does not support? Blue screen? :P

Overall, I'm the most happy with design #3 and think that it offers the most flexibility for advanced users to
truly render anything they want.

💠 **Custom Primitives**

What, exactly, pray tell, are we rendering? Ultimately this is some chunk of data that gets used by a custom
pipeline. This could take the form of data that's passed directly to the existing `Primitive` enum (like
current `Primitive`s are), or something as simple as a single pointer.

One implementation might mean that a custom primitive could be defined within the existing `Primitive` enum as just a
pointer to some pipeline state that implements certain methods required for rendering.

```rust
pub enum Primitive {
//...,
Custom {
id: u64, // a pipeline reference ID
pipeline_init: fn(device: &wgpu::Device, format: wgpu::TextureFormat) -> Box<dyn Renderable + 'static>,
// where "Renderable" defines a set of methods necessary for drawing
}
}
```

Another implementation might define a custom primitive as a generic type that is unique to a `Renderer` or `Backend`.
## 🎯 Implementation strategy

```rust
pub trait Backend<Primitive> {
//...
}
```
### 🙌 #1: Custom Primitive Pointer Design

🧅 **Layers**
Behind the scenes, this would require very little changes to Iced!

In Iced, layering currently happens at the presentation level. Primitives are submitted to the `Renderer` in a
queue-like fashion, and are then grouped before being drawn back to front. This allows primitives that are meant to be
rendered together to be transformed together, somewhat like a scene. For example, when clipping primitives let's
say in a `Canvas`, this will create a new layer. Note that every layer added to the layer stack will incur
additional performance costs with pipeline switching & extra draw commands submitted to the GPU!
A `Primitive::Custom` variant would need to be added to the existing `iced_graphics::Primitive` enum, in order to
have a way to pass a pipeline pointer to the `Renderer` and indicate its proper order in the primitive stack.

When considering a layering approach for custom primitives, we must think about how they are processed. Should a
custom primitive be included in the existing `iced_wgpu::Layer` with some sort of ID matching?
We would also need to add a new field to an `iced_wgpu::Layer`, something along the lines of:

```rust
pub struct Layer<'a> {
//...
// some kind of reference to what can be rendered in this layer that
// we can match to a pipeline
custom: Vec<PipelineId>,
//...
custom: Vec<PipelineId>,
}
```

Or perhaps we should enforce that all custom pipelines must group its supported primitives within its own layer?
This needs some considering, but could be implemented further down the line as an optimization.

## 🎯 Implementation strategy

I've gone through a few ~~hundred~~ dozen implementation strategies. I'll share a few here:

### 🤾 Just throw a pointer at the `Renderer`

You can view a small, very, *very* rough and unrefined prototype of this strategy [here](https://github.com/bungoboingo/iced/tree/custom-shader/pipeline-marker/examples/custom_shader/src).

This example has a pointer-based approach, where the "state" of a pipeline & its associated primitive data is just
stored as a heap allocated pointer within the existing `iced_wgpu::Backend`. Within a custom widget's `draw`,
primitives are drawn like this:

```rust
//...
renderer.draw_primitive(Primitive::Custom {
bounds,
pipeline: CustomPipeline {
id: self.id,
init: State::init,
},
})
```

Where `State::init` is a fn pointer with type signature:
```rust
pub init: fn(device: &wgpu::Device, format: wgpu::TextureFormat,) -> Box<dyn Renderable>
```
To indicate which pipelines are grouped within this layer. Or perhaps we could require that all custom pipelines are
on separate layers, though that has performance implications.

`Renderable` refers to a trait which allows a custom pipeline to call `prepare()` (for preparing data for
rendering, similar to how we are doing it in every other pipeline we support in our existing `iced_wgpu::Backend`, e.
g. allocating & resizing wgpu buffers, writing uniform values, etc.).
We would also need a way to cache & perform lookups for trait objects which implement `Renderable`; in my prototype
I've simply used a `HashMap<PipelineId, Box<dyn Renderable>` inside of the `iced_wgpu::Backend`.

`Primitive::Custom` is then processed for inclusion in an `iced_wgpu::Layer`, where (if not already initialized)
it's initialization (`init` above) is performed & added to a lookup map in the `iced_wgpu::Backend`. Then, come
render time, if there are any `custom_primitives` within the `iced_wgpu::Layer`, we simply do a lookup for its
pipeline pointer & call `prepare()` and `render()` as needed.
Then, when rendering during frame presentation, we simply perform a lookup for the `PipelineId`s contains within the
`Layer`, and perform their `prepare()` and `render()` methods. Done!

✅ **Pros of this strategy:**
✅ **Pros of this design:**

- Simple to integrate into existing Iced infrastructure without major refactors.
- Performance is acceptable

❌ **Cons of this strategy:**
❌ **Cons of this design:**

- Not flexible
- Even with preparing this very simple example I found myself needing to adjust the `Renderable` trait to give me
Expand All @@ -185,30 +116,19 @@ pipeline pointer & call `prepare()` and `render()` as needed.
Overall I'm pretty unhappy with this implementation strategy, and feel as though it's too narrow for creating a truly
flexible & modular system of adding custom shaders & pipelines to Iced.

### 🎨 Custom Shader Widget
### 🎨 #2: Custom Shader Widget

Similar to how we currently have `Canvas` in Iced, this strategy would involve creating a custom widget which is
dependent on `wgpu` that has its own `Program` where a user can define how to render their own custom primitive.
The internals for this custom shader widget are very similar to the previous strategy; the main difference is that
internally, *we* would create the custom primitive which holds the pointer to the pipeline data, not the user. The
other difference is that the `Program` trait is merged with the `Renderable` trait, and that we create the widget
implementation for the user, no custom widget required.

A custom shader widget might have a method that is defined like:
Other concepts must be added, like the concept of `Time` in a render pass. In my prototype, I've implemented it at
the `wgpu::Backend` level, but in its final form we would need to shift it up to the `Compositor` level, I believe.
It's exposed to the user as a simple `Duration`, which is calculated from the difference between when the
`iced_wgpu::Backend` is initialized up until that frame.

```rust
pub trait Program {
type State: Default + 'static;

fn render(
&self,
state: &Self::State,
device: &wgpu::Device,
encoder: &mut wgpu::CommandEncoder,
//...
);
```

Or something similar, which, when implemented, would allow a user to define how to render their custom `State`. I
found that with this strategy, a `Primtive::Custom` wrapper of some kind was still needed, which ended up being
pretty similar to the previous strategy and just replacing `iced_graphics::Renderable` with
`iced_graphics::custom::Program`, so I did not finish a fully flushed out prototype.
There may be other information needed in the `Program` trait which is discovered as the implementation evolves.

✅ **Pros:**

Expand All @@ -217,64 +137,52 @@ pretty similar to the previous strategy and just replacing `iced_graphics::Rende

And, like the previous strategy:
- Simple to integrate into existing Iced infrastructure without major refactors.
- Performance is acceptable, but worse than previous strategy
- Performance is acceptable

❌ **Cons:**
- Same cons as the previous strategy; very little flexibility, users must shoehorn their pipeline code to fit into
this very specific trait `Program` provided by Iced.
- When I was prototyping this out, I found it nearly impossible to do this implementation without doing some kind of
reflection with `Program::State` in addition to the required dynamic dispatching. This could possibly not be a
real con as there might be a different, more performant way to do it!
this very specific trait `Program` provided by Iced.

### 🔠 Multiple Backend Support for Compositors

I have no prototype to speak of this with strategy; it will involve a good amount of restructuring, possibly some
codegen for performance reasons, and some intermediate data structures added to the compositor. That being said, I
believe this is more along the lines of a "correct" solution for integrating custom shaders & pipelines into Iced as
it allows the most flexibility & feels the least hacky.

This strategy involves adding support for multiple `Backend`s per `Compositor`. See the diagram below for a rough
outline of how it would work:

![](diagram.png)

Every `Backend` (probably should be renamed to something more appropriate, like `Pipelines` or `PipelineManager` or
something for clarity) would be responsible for its own primitive type that it can support. In the case of
`iced_wgpu::Backend`, this would be the `iced_graphics::Primitive` enum. The command encoder would be passed down
into every backend for recording before being submitted to the GPU.

This would require a few new concepts added to the wgpu `Compositor`:
Internally, this design is the most complex and requires the most changes to Iced, but I don't think it's so wildly
complex that it would be hard to maintain! This design would require a few new concepts added to the wgpu `Compositor`:

💠 **Primitive Queue**

There must be a backend-aware queue which keeps track of the actual ordering of how primitives should be
rendered across all backends. I believe this could be implemented fairly easily either by having each `Backend` keep
track of its own queue and having some data structure delegate at the appropriate moment with some form of marker
indicating that we need to start rendering on another `Backend`. Some kind of order-tracking data structure is
There must be a backend-aware queue which keeps track of the actual ordering of how primitives should be
rendered across all backends. I believe this could be implemented fairly easily either by having each `Backend` keep
track of its own queue and having some data structure delegate at the appropriate moment with some form of marker
indicating that we need to start rendering on another `Backend`. Some kind of order-tracking data structure is
essential for ensuring proper rendering order when there are multiple backends.

Widgets would request that their custom primitives be added to this queue when calling `renderer.draw_primitive()`.

👨‍💼 **Backend "Manager"**

This would essentially be responsible for initializing all the backends (lazily, perhaps!) & delegating the proper
primitives to the multiple `Backend`s for rendering. This would be initialized with the `Compositor` on application
This would essentially be responsible for initializing all the backends (lazily, perhaps!) & delegating the proper
primitives to the multiple `Backend`s for rendering. This would be initialized with the `Compositor` on application
start.

👨‍💻 **Declarative backends!() macro**

This would be initialized in `Application::run()` as a parameter, or could be exposed somewhere else potentially
(perhaps as an associated type of `Application`?). I haven't super thoroughly thought it through, but my initial
idea is to have it return a `backend::Manager` from its `TokenStream` which would be moved into the `Compositor`.

✅ **Pros:**
- Flexible, users can do whatever they want with their own custom `Backend`.
- Modular & additive; users can create a custom `Backend` library with their own primitives that it supports that
can be initialized with the `Compositor`.
- For users wanting to use a custom primitive from another library, or one they made, they would use it very
similarly to how you use currently supported `Primitive`s in Iced, which would feel intuitive.
- For users wanting to use a custom primitive from another library, or one they made, they would use it exactly how
you use currently supported `Primitive`s in Iced, which would feel intuitive.

❌ **Cons:**
- Doing this strategy performantly without creating a bunch of trait objects might be challenging! At least just
from thinking about it for a few days I've not come up with that many ideas other than using a hefty amount of
codegen via generics or declarative macros.
- Would involve a hefty amount of codegen to do performantly
- This would be quite a heavy refactor for the `iced_wgpu::Compositor`!
- This would (possibly?) preclude custom primitives being grouped together with other backend's primitives in
its own `Layer` for transformations, scaling, etc. which might be undesirable.
- This design would preclude custom primitives being clipped together with other backend's primitives in
its own `Layer` for transformations, scaling, etc. which might be undesirable. There might be a way to implement
this within the `backend::Manager`, however!

### 🤔 Other Ideas

Expand Down Expand Up @@ -370,4 +278,6 @@ and use seamlessly as part of Iced's widget tree is the ultimate form of customi

### If you made it to the end, congratulations! 🥳

Now, let's discuss!
Let's go forward and render some scuffed ice(d) cubes!

![](scuffed_iced_cubes.mov)
Loading

0 comments on commit 1aadf4c

Please sign in to comment.