Upgrade validate-refs to work with new AOSS registry

[jordanpadams]

💡 Description

Blocked by NASA-PDS/registry#350

[jordanpadams]

moving this to future sprint since this is blocked by NASA-PDS/registry#350

[jordanpadams]

Per separate email discussions on NASA-PDS/registry#350, this may be unblocked. Validate will need to be updated to support multiple indexes.

Is there a way to perform this operation in the OpenSearch Java API in order to get the known registry indices?

GET /_cat/indices?format=json&pretty=true

If not, we will need a config file that is included in the package that we can pull from GitHub whenever validate-refs executed.

[al-niessner]

@jordanpadams

As per separate email: comma, semi-colon, or both?

Also, I think NASA-PDS/registry#350 was about comma, semi-colon, or a virtual index that resolved to all of our indices. I do not remember it being about read permissions although that could have distracted us.

[jordanpadams]

@al-niessner NASA-PDS/registry#350 was pertaining to enabling this for other users. that being said, an alias index seems to make a lot of sense for something like this.

[tloubrieu-jpl]

@jordanpadams @al-niessner , the registry-api is using a configuration containing the list of prefixes of the existing indexes (e.g. en,geo,atm,...).

If we had a virtual index aggregating them all I guess we would use it as well but this is not blocking.

[al-niessner]

@jordanpadams @tloubrieu-jpl

Sigh - around the loop one more time. Yes, that is correct Java SDK2 uses list of indices. However, validate does not because it uses the connection XML file instead of "knowning" the right answer a priori as registry does. In the connection XML file, do you want to use comma, semi-colon, or both for the list? If a separated list in a string attribute is unappealing, then do you want to do a virtual node that does it for us (the "knowing" a priori for everyone that keeps everyone in sync rather than registry knowing a different answer than validate)?

[jordanpadams]

@al-niessner this information will not be supplied by the user in their connection XML file because that file is a re-use of the same config from harvest/registr-mgr, which is index-specific. So we will need to maintain this list of available indices in validate similar to registry-api. I would recommend we keep this in a separate config file so we can eventually code up a means of automatically updating that file by grabbing the latest from GitHub or elsewhere. For now, let's just hard code it in a config file.

[al-niessner]

@jordanpadams

Okay, but comma, semi-colon, or both?

[jordanpadams]

@al-niessner It is something we will maintain, so whatever you prefer.