[WIP] Moving closer to Dask

[caspervdw]

I am trying to get some ideas written down to move this project closer to dask. The current state of the document is a comparison of relevant parts of dask and dask-geomodeling.

[daanvaningen]

Very interesting! I don't feel like I have a good feel for the implications of implementing this but I think you pointed out some good reasons to move closer to Dask.
I think your concern is also mine: what is the effect of these optimizations and parallel performance when our chunks are small and data request sizes limited by our web-application.

[caspervdw]

Very interesting! I don't feel like I have a good feel for the implications of implementing this but I think you pointed out some good reasons to move closer to Dask.

I find this very hard to wrap my head around too. I am not sure if it is worth the effort, or if we should wait for the dask improvements to be released. In the meantime, we could focus on the infrastructure part rather than rewriting dask-geomodeling.

I think your concern is also mine: what is the effect of these optimizations and parallel performance when our chunks are small and data request sizes limited by our web-application.

For web-application, we effectively do not parallelize across multiple chunks. This is not necessary as the chunks are so small. We may parallelize operations if the graph allows it (e.g. reading the same chunk from multiple sources concurrently instead of after each other).
I am worried that the graph optimization itself (the "overhead") will be large.