Drift database access hanging specifically when being served from S3

[justro]

I am running into an issue that seems to be specifically occurring when I am serving a flutter app from S3. I didn't want to file a bug yet since I don't have a minimal repro and am unsure if this is a drift problem or (maybe more likely) has to do with how I have my S3 deployment configured. I will be super grateful for any advice/suggestions anyone has.

I have a flutter app that I build for iOS, Android, and web. It uses drift as a local store. I have:

environment:
  sdk: ">=2.19.0 <3.0.0"
  flutter: 3.3.10
.......
drift: ^2.10.0
sqlite3: ^2.1.0
sqlite3_flutter_libs: ^0.5.15
......
(dev deps)
drift_dev: ^2.10.0

My web folder has drift_worker.dart 2.10 and sqlite3.wasm 2.1 pulled from the repos linked in the web instructions. (I currently am using the sqlite3.debug.wasm build actually, but I don't know how to get any information from it, not sure if I need to set some flag to get logs or what?)

When I load my app, it initializes the DB, and eventually will perform some requests against it. On native platforms, this works without issue.

On web, if I run locally it works, whether I build in debug or release mode.

When I run on web, specifically when it's being served from an aws cloudfront/s3 deployment, it seems to hang on initialization. Running from a local web build, built release or otherwise, db initialization seems to proceed as expected. The hang happens whichever browser I've tried (from mac chrome/safari or windows edge/firefox), and whether I'm in normal or private mode.

For example, using the code here, on a working (local) run, I can see log lines dropped from the initializeDatabase function, but when I'm loading from a cloudfront/s3 distribution, I do not see the log lines happen.

Basically, any async call, including the database open, seem to never return when served from cf/s3, but work fine from others.

I had seen similar behavior when using older versions of drift, but it seemed to happen really infrequently, and when I used an incognito window or cleared local data, it worked fine. Having upgraded to 2.10 recently, I run into the error 100% of the time when deployed from s3.

It sounds like maybe it's a header configuration issue. I'm not seeing the CORS headers I expect coming from cf/s3, so I will check into that next.

[justro]

Sorry, another note, FWIW whether it hangs or not, the browsers I'm using seem to fallback to sharedIndexedDB:

│ 💡 Using WasmStorageImplementation.sharedIndexedDb due to missing browser features: {MissingBrowserFeature.dedicatedWorkersInSharedWorkers, MissingBrowserFeature.sharedArrayBuffers}

Happens whether it works or not, so maybe not relevant, but thought perhaps the specific storage impl might be worth knowing.

[justro]

After further troubleshooting, it seems like I was wrong, it does depend on my build environment. The broken build is done in AWS codepipeline, but if I copy my local sources into the s3 bucket for the deployment, problem seems to be resolved.

[simolus3]

(I currently am using the sqlite3.debug.wasm build actually, but I don't know how to get any information from it

It should print them to the console.

Using WasmStorageImplementation.sharedIndexedDb

If you're not enabling CORS headers, this is actually expected.

Do you see logs about workers not being loaded or something similar that would point towards a possible cause?

[simolus3]

It's not exactly clear to me where it's going wrong either. I think there might be a deadlock happening because you're re-using the IndexedDB database name? You're using goal_setting for the old and the new database.

You could try also adding debug logs to DriftServerController.serve in drift/lib/src/web/wasm_setup/shared.dart - in particular before and after the await switch to see if it maybe gets stuck there.

[justro]

After about 6 days of troubleshooting, I think I finally figured out what's going on here. By default, when an aws cloudfront deployment serves a wasm file out of s3, it serves it as content-type: application/octet-stream. I added a behavior to the deployment to serve it as application/wasm, and it immediately started working. I did not know about chrome://inspect for getting logs out of shared workers, so I was not seeing the log lines I had in the shared worker...

It turns out, DriftServerController.serve gets stuck at this line:

    final sqlite3 = await WasmSqlite3.loadFromUrl(message.sqlite3WasmUri);

when the content type is not application/wasm.

I had been using an older version of drift (2.5.0) and associated worker/sqlite3.wasm builds, and hadn't run into this problem with those older, experimental versions. I'm not sure if I would consider this a bug or what, although some kind of error message or something if the content-type is going to be an issue would be great.

This is a really great library and I deeply appreciate the time and effort that must have gone into this...especially since I've gotten a better idea of how much there is to it. Please let me know if you'd like me to file an issue or provide any more details, document it, etc.

Thanks a ton!

[simolus3]

Good catch! And I completely agree that this is a bug in drift. The worker should catch the wasm file not loading and properly return this error to the main document. These kinds of mistakes are easy to make and drift should therefore be very explicit about what went wrong (instead of just doing nothing).

You can open an issue for this if you want to track this bug, but I'll also keep it in mind from this discussion and I'll hopefully fix it soon.

[justro]

Thanks so much! I won't bother with a bug unless it's helpful to you...if so, lmk, otherwise, thanks again!

[simolus3]

After some further investigation, I think this is happening because the error is getting reported asynchronously, and we only try to load the wasm file when the database is first used (e.g. the first statement to it is run).

That's how drift generally works. I've tried moving this to an earlier place but that would require changing the worker protocol which would be pretty painful to do (since drift needs to support older worker versions as well). In my testing, I was able to reliably catch the error by creating an instance of the app database class and calling await db.executor.ensureOpen(db). This will open the database and run all migrations, throwing if that fails. Otherwise, you'd get the error at the first statement, which might not be handled correctly.