fix for replacing cluster id with new supervisor id

[divyenpatel]

What this PR does / why we need it:

When TKG-HA feature was introduced in the vCenter, we changed cluster-id to supervisor-id, as with TKGS-HA we have support for stretching supervisor to multiple clusters.

We have a code in the full sync to update volume metadata with older cluster-id with new supervisor-id. Unfortunately we used QueryAllVolume API to fetch metadata, but QueryAllVolume API does not return metadata.

We need to fix this by calling QueryVolumeAsync API with list of VolumeIDs with older cluster id to fetch metdata and then issue update volume metadata with new supervisor cluster id.

Without this fix, we see CreateVolume API calls happening for volumes created using older cluster-id before upgrade.
Every full sync is issuing unnecessary create volume calls with new supervisor, as volumes are not updated with new supervisor-id. and all these CreateVolume calls are failing with CnsAlreadyRegisteredFault.

Testing done:
Created Volume with new Supervisor-ID.
From Backend changed Supervisor-ID to Older Cluster-ID which is Cluster-Moref
Applied this change.
Observed two UpdateVolumemetadata calls which replaced cluster id with new supervisor-id

Special notes for your reviewer:

Release note:

fix for replacing cluster id with new supervisor id

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: divyenpatel

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [divyenpatel]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

@divyenpatel: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-vsphere-csi-driver-unit-test	777df9b	link	true	/test pull-vsphere-csi-driver-unit-test

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.