Implementation details - WASM + PostgresQL

[dberardo-com]

I was wondering some implementation related details:

1. is this duckdb extension automatically available also as wasm package? i.e. can it be installed on frontend .wasm duckdb already?
2. how can this extension be efficiently integrated with postgresql?

point number 1 might be already clear enough, while for point 2 i mean that duckdb is not yet used to store data but rather to import it and then work on it (ETL). however duckpgq is thought to be working on potentially frequently changing graph data. So the question is, assuming that the underlying data structure is store and updated in a postgresql database, how can this update be efficiently syncrhonized with duckdb / duckpgq "on-the-fly" ?

my quess is to use postgresql scan, and periodically call the clear cache method: https://duckdb.org/docs/extensions/postgres.html#schema-cache but is there any better way to achieve this ?

and also: i am assuming here that duckpgq can already work with tables that are loaded in duckdb using the pgsql attach command

[Dtenwolde]

Hi there,

1. I think currently DuckPGQ doesn't quite work with WASM like other DuckDB extensions. Perhaps @carlopi (DuckDB Labs Engineer working on WASM support among other things) could help me provide a better answer here :) Either way, I created an issue related to WASM support here: Make DuckPGQ loadable in DuckDB WASM #170
2. I've never tried to use the DuckPGQ extension with PostgreSQL, but I am aware of an issue with using ATTACH to another database which I am tracking here Support property graphs on attached databases #160. That might be a blocker for using the extension with PostgreSQL, but feel free to give it a try.

Also, are you aware of https://github.com/duckdb/pg_duckdb which might be interesting for you. Again, I haven't tested if DuckPGQ works with this extension (this is all quite new) but there's a chance it works.

[Dtenwolde]

I'll bump up the priority of #160 and hopefully get a PR up by the end of this week, but again, there are no guarantees :)

[dberardo-com]

sure, i have also got a lot "on my desk" before end of the year, so i might be running some tests not earlier than 2 weeks from now

[Dtenwolde]

Hey there, I just merged #171 which should allow property graphs to be created on attached databases. I tested this with SQLite using the sqlite_scanner extension which worked, so I assume PostgreSQL works as well.

It will be part of the next update to the community extension, which will likely be next week :)

I'll keep you posted!

[dberardo-com]

Sounds awesome, I will try that out on postgres too. Also have you noticed perhaps whether the attached tables are automatically updated when the underlying SQLite tables are updated? Or does it require a call to the decaching function each time?

[Dtenwolde]

I just tested on PostgreSQL and there it works as well :)

The attached tables are automatically updated whenever you update the data in SQLite or PostgreSQL. You only need to attach and create a property graph once.