[Enhancement]: Improve storage of BOSS objects

[binaryoverload]

Checked Existing

• I have checked the repository for duplicate issues.

What enhancement would you like to see?

Currently, BOSS has several ways that it stores/serves files:

• NPDI: Serves encrypted data for games (Local Directory: cdn/content/encrypted/${titleHash}/${fileHash})
• NPDL: Serves files for games (S3: ${appID}/${taskID}/${file.hash})
• NPFL: Serves files for games (MongoDB in the "files" collection)
• NPPL: Serves policy lists for consoles (Statically defined in code)
• NPTS: Serves task sheets for games (Local Directory: cdn/tasksheet/${id}/${hash})
• SPR: Receives/Sends SPR data for consoles (MongoDB in the "cecdatas" collection - files stored at a 33% penalty in base 64)

Ideally, the storage for many of these items should be stored in one place and not locally on disk.

This issue is raised with the new infrastructure in mind, specifically looking at the 2 main topics:

• Local Directory - Storing files within the container
  • On a larger scale storing files in directories (and subsequently volumes) gets more difficult to manage
  • Storing these files in a local directory also means it's hard to automate them for things like Splatoon rotations
• Large volume DB storage
  • SPR stores a large amount of data in MongoDB (~200GB per month) which presents a logistical issue with having a DB being multiple 100s of GB

I would like to propose the following:

• Store all SPR data in S3 - using object storage is the perfect use case for this large amount of data, and will also save space by not having to use Base64
• Store the local "cdn/" files also in S3 - they will be more easily accessible by things like automation and work around the issues with local files in containers
  • There is a fair amount of the CDN folder which is committed and looks to be hardcoded data that won't change
  • I think it's good to keep those committed and served from the container (volume wouldn't be needed if it's not dynamic).
  • But, I don't know what the best way to separate the items would be since they are served from the same API routes/folders as the dynamic (i.e Splatoon) data.

In the future, I think the files for NPFL and NPPL could also be moved, but that is out scope of this issue imo.

Any other details to share? (OPTIONAL)

No response

[binaryoverload]

But, I don't know what the best way to separate the items would be since they are served from the same API routes/folders as the dynamic (i.e Splatoon) data.

One thing we could do is store the Splatoon data in the Database - then we could use the DB as a "fallback" for certain whitelisted keys (I.e. The data for bb6tOEckvgZ50ciH wasn't found in the CDN folder, look for an entry in the database)

This would allow us to do automation, and get around the volume issues with local files - but not fully send the data off to S3

[ashquarky]

NPTS: Serves task sheets for games (Local Directory: cdn/tasksheet/${id}/${hash})

Worth noting that these can and almost certainly should be generated by the server, they're just descriptions of the.. I think NPDI data? They're static for ~30days at a time so if we cache it after generation that's fine

It's been a frequent source of errors in the past where the tasksheets were improperly updated by paw and had e.g. the wrong file length

[jonbarrow]

This lines up with what we were already planning to do 👍

The end goal was always to move the underlying files entirely to S3 and generate the lists (NPFL/NPTS) on the fly with some caching (I intended to use both memoization and Cloudflare for this, so that non-Cloudflare users still get some caching and we can do some per-list caching more easily)

The local storage option shouldn't be removed entirely however, as part of our efforts to not completely vendor-lock

[jonbarrow]

For some additional context on what these files/services are:

• NPPL - "Nintendo ? Policy List". This file tells the console what types of system services should be enabled and how they should function (like how often they should reach out to the servers for updates). These are values we have historically needed to tweak to get things working, and not everything is 100% known, and contents of the policy list also can differ officially (for example, when using NN this only contains the entry for SPR when connected to a Nintendo Zone) which might be something we also need/want to do in the future, so this shouldn't be a static file imo (right now it's using hard-coded values, but the file itself is generated on the fly so we have the ground work done to hook up whatever dynamic settings we need to it)
• NPFL - "Nintendo ? File List". Only used by the 3DS. This is the precursor to NPTS. These files list the names of files available for download in NPDL, along with some properties of the files, using a custom plaintext format (rather than something standard like xml). Like the policy list, we will likely need to update the contents of these files dynamically so imo they also should not be static files but rather should remain generated on the fly
• NPDL - "Nintendo ? Download". Only used by the 3DS. Used to download files listed in NPFL. This serves the actual BOSS contents, so these files would be stored in S3
• NPTS - "Nintendo ? Task Sheet". Used by the Wii U and I think some 3DS games. This is the replacement for NPFL. NPTS serves a list of available files for a given BOSS task, now using xml. Like NPFL it also lists the name of the file and has some properties about each file, but is more flexible and easier to work with. With NPFL the "file name" was purely for download purposes, the stored file would not have this name, and the format used in NPFL limited the amount of properties associated with the file. With NPTS the file name is what the stored file is also named, each file can be assigned a "type", they can be configured to notify the user when downloaded, the task as a whole can be toggled on and off, and rather than only having a file name and using NPDL the task sheet also provides the full URL to download from. NPTS lacks an update timestamp but instead has a "Data ID" much like DataStore, which increments every time the file changes so the console knows when there's new data. Same reasoning as NPFL and NPPL, NPTS should stay as generated on the fly imo
• NPDI - "Nintendo ? Data Item/ID". Used by NPTS to serve BOSS contents. This is, actually, mostly just a convention. Neither console REQUIRES this NPDI service as far as we know, since NPTS provides the full download URL unlike with NPFL/NPDI. By convention the download URLs provided by NPTS look like https://npdi.cdn.nintendo.net/p01/data/1/07E3nY6lAwlwrQRo/39015/fc64b04bbe8e7ff26f9584d0a94ff8e8, where 07E3nY6lAwlwrQRo is the BOSS application ID, 39015 is the files "Data ID", and fc64b04bbe8e7ff26f9584d0a94ff8e8 is the MD5 hash of the encrypted contents. As far as I know the console will accept anything for the download URL (so long as it meets to SSL/TLS requirements and is valid BOSS data), but we implement this the way Nintendo did just for consistency reasons, we MIGHT be able to get away with just slapping an S3 download URL in NPTS to begin with? But this should probably remain, regardless, for non-S3 users
• SPR - "StreetPass Relay". StreetPass but online. Normally StreetPass only works when you walk by another 3DS user. When this happens, both consoles automatically exchange ("pass") data between each other in the background and you can see the Mii's of the users you've exchanged with. StreetPass Relay utilized Nintendo Zone access points. Nintendo would partner with certain companies like McDonalds, GameStop, Home Depot, etc. and have their public wifi networks act as "Nintendo Zones". When connected to a Nintendo Zone, you could download stuff like previews/demos of games and other content. When connected to a Nintendo Zone your 3DS would also upload it's own StreetPass data to the server, linked to that Nintendo Zone, and then download the data of other users who had previously connected to the same Nintendo Zone, effectively allowing for online StreetPasses without the requirement of being near the other users. We implement this the same way though globally. We do not require the user to be connected to a Nintendo Zone, and instead have NPPL just always report that the user is connected to a Nintendo Zone so that the download/upload happens regardless (this is why SPR is so heavy). We also don't restrict content to specific Nintendo Zones (because we've done away with the concept entirely) which results in global SPR. Contents should absolutely be moved to S3