Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Decentralized storage (IPFS/IPNS or similar) as DDO metadata storage (including media files) #1

Open
3 tasks
MBadea17 opened this issue Jan 22, 2024 · 0 comments · Fixed by #11 · May be fixed by #18
Open
3 tasks

Decentralized storage (IPFS/IPNS or similar) as DDO metadata storage (including media files) #1

MBadea17 opened this issue Jan 22, 2024 · 0 comments · Fixed by #11 · May be fixed by #18
Assignees
Labels
Milestone

Comments

@MBadea17
Copy link

MBadea17 commented Jan 22, 2024

Feature # 66.
Later edit by @alexcos20 :

Status-quo:
Whenever an asset is published, the coresponding DDO is published at the smartcontract level (using setMetaData or updateMetaData - where an event is emited which contains the DDO). DDO can be unecrypted or encrypted (see flags option). DDO (which is a JSON Object) has a lot of fields (see https://docs.oceanprotocol.com/developers/ddo-specification)

Request: add additional types of storages (beside on-chain), so publisher can have more control on the DDO

How to:

  • Define how the DDO should be published:
    Proposal: DDO should be a JSON object, with only ONE key, called "remote". Value of that field is a match of supported storages (see https://docs.oceanprotocol.com/developers/storage#static-urls, etc). If object has multiple keys or 'remote' key is missing, then is should be not considered as remote DDO and handled like status-quo (this way we ensure backwards compatibility)
    Pro: no changes required to ocean.js
    Cons: none? (TBD)

Example :

{
   "remote":  {
        "type": "url",
        "url": "https://url.com/ddo_x.json",
        "method": "GET",
    }
}

Encryption:
We should add two extra fields on the Storage spec (for all types of storages):
- encryptedBy: public node address who encrypted the file
- encryptMethod: file is encrypted by encryptedBy, using method ("ecies" or "aes")

So, a fully encrypted DDO stored on IPFS will look like:

{
   "remote":  {
        "type": "ipfs",
        "hash": "12334343",
        "encryptedBy": "16Uiu2HAmUWwsSj39eAfi3GG9U2niNKi3FVxh3eTwyRxbs8cwCq72"
        "encryptMethod": "ecies"
    }
}

The indexing flow should be:

  • get data from metadataCreated/Updated event
  • if flags & 2 (encrypted on chain) , call endpoint decrypt on that remote node and index if it returns a DDO, then exit
    It's not encrypted, so we move forward:
  • convert JSON string to object
  • check no of keys, should == 1
  • check "remote" key exists
  • if no_of_keys==1 and 'remote' in object:
    • if encryptedBy is present:
      • if encryptedBy === our_node_id, decrypt the file (using storage classes) and index it
      • else, call endpoint decrypt on that remote node, index if it returns a DDO, then exit
    • if not, fetch file using storage classes and index it
  • else:
    - index JSON.decoded DDO

The decrypt endpoint flow should be:

  • get data from metadataCreated/Updated event
  • if flags==0 (unencrypted on chain): convert JSON string to object
  • else decrypt from chain
    Now, that we have the raw DDO:
    • check no of keys, should == 1
    • check "remote" key exists
    • if no_of_keys==1 and 'remote' in object:
      • if encryptedBy is present:
        • if encryptedBy === our_node_id, decrypt the file (using storage classes) and index it
        • else, fail with error, because we cannot decrypt
      • else, read the ddo using storage classes and return it
    • if not, return the DDO

Tasks:

  • extend Storage classes: add function decrypt(stream,method=['ecies' or 'aes'], returns stream
  • update ocean-node to parse remote ddo when indexing (simple fetch of remote ddo, using already existing storage clases)
  • update ocean-node to parse remote ddo when decrypting
@alexcos20 alexcos20 added this to the Q1 milestone Jan 23, 2024
@MBadea17 MBadea17 added the OPF label Jan 25, 2024
@georgevlada georgevlada self-assigned this Feb 6, 2024
@MarianButucel MarianButucel linked a pull request Jun 11, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment