Make DuckPGQ loadable in DuckDB WASM

[Dtenwolde]

What happens?

Currently the DuckPGQ extension is not easily loadable in DuckDB WASM. For instance, go to https://duckdb.org/docs/api/wasm/overview.html and type:

install duckpgq from community; 
load duckpgq; 

And you'll receive the following error:

IO Error: Extension "https://community-extensions.duckdb.org/v1.1.3/wasm_eh/duckpgq.duckdb_extension.wasm" could not be loaded: Could not load dynamic lib: duckpgq
Error: bad export type for '_ZTIN6duckdb23CreatePropertyGraphInfoE': undefined

To Reproduce

Go to https://duckdb.org/docs/api/wasm/overview.html and type:

install duckpgq from community; 
load duckpgq; 

And you'll receive the following error:

IO Error: Extension "https://community-extensions.duckdb.org/v1.1.3/wasm_eh/duckpgq.duckdb_extension.wasm" could not be loaded: Could not load dynamic lib: duckpgq
Error: bad export type for '_ZTIN6duckdb23CreatePropertyGraphInfoE': undefined

OS:

macOs 13 - Apple M1 Pro

DuckDB Version:

v1.1.3

DuckDB Client:

WASM

Full Name:

Daniel ten Wolde

Affiliation:

CWI

How did you load the extension?

Community extension version

Did you include all relevant data sets for reproducing the issue?

Yes

Did you include all code required to reproduce the issue?

• Yes, I have

Did you include all relevant configuration (e.g., CPU architecture, Python version, Linux distribution) to reproduce the issue?

• Yes, I have

[Dtenwolde]

Mentioned here: #166

[carlopi]

Thanks, this is at the moment expected, problem is that DuckPGQ is somehow different than other extensions in the sense that it bundles DuckDB (and it's parser).

I think this might require wrapping duckdb code in a duckdb_impl namespace, so that it do not collides with duckdb functions (that might or might not be equivalent) that are provided on static linking by duckdb-wasm.

Anther possible way out is enabling DuckDB to be statically linked in, and do the remapping at the Wasm layer, but I am not sure that's viable.

DuckPGQ can work free-standing in the browser, but I would consider more proper to work in having this supported on top of regular duckdb-wasm.

Thanks for opening the tracking issue. Unsure what's the path forward, but can work, and would be super cool to have it working.

[ThijsDreef]

I was wondering if this incompatibility is due to the fact that extensions which are built out of tree are missing the wasm flag to be compiled as a side module.

In the main duckdb repository the SIDEMODULE=1 flag for emscripten is set in the top level cmakelists.txt. But I cannot spot this flag in any of the following repositories

• https://github.com/duckdb/extension-template
• https://github.com/duckdb/extension-ci-tools (Does use SIDEMODULE=2)
• https://github.com/cwida/duckpgq-extension

My concern would then be one of the following

• Module is compiled without any SIDEMODULE flags (dlopen functions incorrectly).
• Module is compiled with SIDEMODULE=2 (some code is considered dead which is actually used)

[carlopi]

Hi @ThijsDreef: I think duckpgq-extensions gets SIDE_MODULE=1 from its DuckDB submodule at https://github.com/search?q=repo%3Acwida%2Fduckdb-pgq%20SIDE_MODULE&type=code

I am not sure I see easy solution, problem here is that there are 2 DuckDB codebases at play: 1 provided by DuckDB-Wasm (as MAIN_MODULE) and 1 provided by duckdb-pgq, but those are different.

Something that might be tried, and might even work even if not super solid could be changing to SIDE_MODULE=2 (in the submodule) and linking the duckdb-own submodule in duckdb-pgq, mimicking what's proved to work for native extensions.

[ThijsDreef]

So I think i understand the situation now.

The extension uses a partially modified version of duckdb and the bad export type error is thrown because it requires a function from the modified version of duckdb (which is not linked into the extension).

So would a solution where we link in the required functions into the extension be a solution?

I did try to link in the required functionality by hand (which succeeds for the object file) but is still an import in the generated wasm file.
I did get the duckdb fork to compile to wasm by using the duckdb-wasm repository which then does export the required import.

[peterboncz]

Hi Thijs: Duckdb extensions in C++ link statically with duckdb and all duckdb symbols that they use are included in the extension binary (The reason this was done was because of dlopen()/dlsym() problems in windows) This means that duckdb symbols are replicated, namely they are part of duckdb, but a copy of them also may exist in dynamically loaded extensions It is one of the reasons there must be an absolute version compatibility between ducks and the extension module. Duckpgq is a special case It axploits the fact that symbols are linked twice to its advantage. Specifically, the SQL parser and binder that the duckpgq uses are indeed a modified (fork) of duckdb. Namely, a parser that is extended with the PGQ syntax. We make sure that fork is merged with the precise version of duckdb we are building for, such that all duckdb classes and methods have exactly the same layout and signature. The modified binding phase constructs a LogicalPlan that is completely compatible, and that is passed back to the duckdb binary, which executes the plan. So yes, the modified duckdb needs to be linked. What is more, it should take precedence with repespect to the extension code. Whenever the extension code calls duckdb methods (like, yyparse), th methods of the modified duckdb should be called. Would this possible in the wondrous world of emscriptem? I know I am asking for rather impossible things. Peter

…

On 8 Jan 2025, at 20:17, ThijsDreef ***@***.***> wrote: So I think i understand the situation now. The extension uses a partially modified version of duckdb and the bad export type error is thrown because it requires a function from the modified version of duckdb (which is not linked into the extension). So would a solution where we link in the required functions into the extension be a solution? I did try to link in the required functionality by hand (which succeeds for the object file) but is still an import in the generated wasm file. I did get the duckdb fork to compile to wasm by using the duckdb-wasm repository which then does export the required import. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>