Add support for Portworx and future storage modules #37
Conversation
There was a problem hiding this comment.
Per our conversation, we'll want to add lvm2 to https://github.com/equinix/terraform-metal-anthos-on-baremetal/blob/master/templates/pre_reqs.sh#L11 and https://github.com/equinix/terraform-metal-anthos-on-baremetal/blob/master/templates/pre_reqs.sh#L31 (I don't know the package name for Fedora-based systems, nor if the package is preinstalled or must be added).
|
Hey folks. I'm opinionated on how we should do this. I think we should have a variable that is labeled something like Then we should be able to write different CSI modules that we apply based on the CSI they choose. I would like to see a CSI directory and then then a Provider directory and then a TF file and whatever else is needed in there to execute on. I'm not sure how to optionally include a module in TF like this at the moment. But I think that would be the most flexible. |
|
I completely agree with keeping storage providers abstract in this project. While these changes are being developed (and the PR is in draft) the changes are taking an opinionated stance. We'll definitely want to see more generic definitions before this can be merged. |
Signed-off-by: Bikash Roy Choudhury <[email protected]>
|
@c0dyhi11 There are some changes coming through in this PR that may be generally applicable for Portworx, Rook, longhorn, or other storage providers. We could accept all of those changes until there is a conflict between storage providers or overhead introduced in the dependencies. Longhorn, for example, requires |
Signed-off-by: Bikash Roy Choudhury <[email protected]>
Signed-off-by: Marques Johansson <[email protected]>
templates/device_name.sh
Outdated
| if ! lsblk -f -b -n -oNAME,SIZE,FSTYPE -i /dev/$disk | egrep "xfs|ext3|ext4|btrfs|sr0" >/dev/null; then | ||
| echo -en "$disk $size" | ||
| fi | ||
| done | sort -n -k2 | head -n1 | cut -f1 -d" " |
There was a problem hiding this comment.
This should be sort -n -k2 -r, so the largest disk is first (or change head to tail).
There was a problem hiding this comment.
I recall conversations about using the smallest/fastest disk for the KV DB, I don't see anything here to support that.
If we do copy this function to get the smallest disk, and then pvcreate/vgcreate/lvcreate that disk, we would need to be cautious about cases where there are only 2 disks and the smallest and largest free disk are the same disk.
|
We'll want some README.md notes about how to enable the use of Portworx in this project. We will also need to know what the installed solution looks like to the user (in terms of what StorageClasses are available, how to interact with the UI). Should the user customize the StorageClasses (manually or does Portworx do this automatically) so that nvme storage can be chosen over hdd storage ( We should also point out which plans will work with the Portworx installation, for example, the 2-disk c3.small plan may not work with the Portworx installation that this module provides. And we'll also need to explain what happens when the trial period ends, what are the user's options, what happens to existing volumes and snapshots, and what capabilities are lost. Does this effectively become the OSS version? |
|
The README should also include notes about how to interact with the Portworx UI within the cluster. |
Signed-off-by: Bikash Roy Choudhury <[email protected]>
Signed-off-by: Bikash Roy Choudhury <[email protected]>
Signed-off-by: Marques Johansson <[email protected]>
|
I'm attempting to modularize this PR here: I haven't fully tested this yet because I ran into a limitation that I'm looking into other ways to accomplish this or workaround the problem. Update: a simple count / count.index worked. |
Signed-off-by: Marques Johansson <[email protected]>
Signed-off-by: Marques Johansson <[email protected]>
Signed-off-by: Marques Johansson <[email protected]>
|
@bikashrc25 Can you pull the latest version of this branch and give it a try? You'll need to add the following to @c0dyhi11 this is ready for a review. |
README.md
Outdated
| ```hcl | ||
| storage_module = "portworx" | ||
| storage_options = { | ||
| portworx_version = "2.6" |
There was a problem hiding this comment.
| portworx_version = "2.6" | |
| version = "2.6" |
displague
changed the title
modules/portworx/README.md
Outdated
|
|
||
| Login to any one of the Anthos cluster nodes and run `pxctl status` to check the portworx state or run `kubectl get pods -lapp=portworx -n kube-system` to check if the portworx pods are running. Portworx logs can be viewed by running: `kubectl logs -lapp=portworx -n kube-system --all-containers`. | ||
|
|
||
| By default, Portworx 2.6 is installed in the Anthos Cluster. The version of Portworx can be changed using the `portworx_version` variable. |
There was a problem hiding this comment.
Let's add a link to https://docs.portworx.com/reference/knowledge-base/px-licensing/#trial-license because I think it is helpful for users to know in advance what will happen when the trial ends.
We can also introduce a variable for the portworx_license so the user can active portworx immediately upon deployment (or by setting the variable and running terraform apply later).
Signed-off-by: Marques Johansson <[email protected]>
|
@displague I created a new cluster this morning after I did a Here are some of the inconsistencies:
I am including some of the snippets. This is worker node 1 where it could not create the pwx_vg and you can clearly see the warning message. This worker node 2 where there is no pwx_vg for the KVDB. Finally this is worker node 3. This node creates the pwx_vg on the larger capacity drive. Any thoughts regarding these inconsistencies. |
|
@displague Thanks for fixing the terraform script to create pwx_vg on the worker nodes. However, the script should pick the lowest disk size for creating the pwx_vg for KVDB. I am attaching the output of the "lsblk" from all the 3 worker nodes. You would notice that worker node 2 and 3 are using a 480GB drive instead of 24GB. Worker node1: Worker node 2: |
The Portworx licensing step can fail if applied too soon. The documentation is updated to reflect that, providing remediations.
|
@bikashrc25 I created #45 to track the issue you mentioned about the disk choices. In recent commits I documented that the Portworx license options should only be used after the Portworx install is ready. If used too early the failure can be fixed with another @c0dyhi11 We followed the pattern of offering generic storage options, I think this is ready to go so we can build upon this pattern. Awaiting your review before a merge. |
deploy_anthos_cluster is run on the first control plane node, but it triggers ssh provisioning from that node to the workers outside of what Terraform is doing. Signed-off-by: Marques Johansson <[email protected]>
Addresses #24 by adding support for Portworx. Portworx is software define storage layer that provisions persistent storage and data mangement capabilities for stateful applications. Portworx is using GKE/Anthos cluster on Equinix Metal and automating the installation of the complete stack.
Included is a script that detects the best disk on each worker node to act as the Portworx KVDB drive. This script was needed because the disk device assignments on each worker node were not consistently assigned.