Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deploy.sh will have the option to choose deployment type #1049

Open
wants to merge 9 commits into
base: e2e/databricks/parking-sensors-V1
Choose a base branch
from
3 changes: 3 additions & 0 deletions e2e_samples/parking_sensors/.devcontainer/.envtemplate
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,9 @@ AZDO_ORGANIZATION_URL=
# OPTIONAL. Environment variables for deployment script.
# Uncomment to set values.

# Define if Dev(example: #ENV_DEPLOY=1) or Dev and Stage (example: #ENV_DEPLOY=2) or or Dev, Stage and Prod(example: #ENV_DEPLOY=3) will be deployed.
# ENV_DEPLOY=

# Azure location to deploy resources. Default: westus
# AZURE_LOCATION=

Expand Down
21 changes: 17 additions & 4 deletions e2e_samples/parking_sensors/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -277,6 +277,17 @@ Set up the environment variables as specified, fork the GitHub repository, and l

Optionally, set the following environment variables:

- **ENV_DEPLOY**- Specifies the number of environments to deploy.

- If the variable is set as shown in the optional configuration below, the prompt will be skipped.
- If not configured, a prompt will display the following options.

- The options are:

- **Option 1**: Deploy only to the development environment (Dev). Optional configuration for environment variable set #ENV_DEPLOY=1.
- **Option 2**: Deploy to both development (Dev) and staging (Stage) environments. Optional configuration for environment variable #ENV_DEPLOY=2.
- **Option 3** : Deploy to development (Dev), staging (Stage), and production (Prod) environments. Optional configuration for environment variable #ENV_DEPLOY=3

- **AZURE_LOCATION** - Azure location to deploy resources. *Default*: `westus`.
- **DEPLOYMENT_ID** - string appended to all resource names. This is to ensure uniqueness of azure resource names. *Default*: random five character string.
- **AZDO_PIPELINES_BRANCH_NAME** - git branch where Azure DevOps pipelines definitions are retrieved from. *Default*: main.
Expand All @@ -298,7 +309,11 @@ Set up the environment variables as specified, fork the GitHub repository, and l
2. **Deploy Azure resources**
- `cd` into the `e2e_samples/parking_sensors` folder of the repo.
- Run `./deploy.sh`.
- The login process for deployment is interactive. When you run the script **deploy.sh**, a browser window will be open, prompting you to log in to Azure. If there is an open session from a previous deployment, it may log you out and request you to log in again.
- The login process for deployment is interactive. When you run the script **deploy.sh**, a browser window will be open, prompting you to log in to Azure. If there is an open session from a previous deployment, it may log you out and request you to log in again- .
LiliamLeme marked this conversation as resolved.
Show resolved Hide resolved
- During deployment, you will be presented with three options if the optional environment variable #ENV_DEPLOY for deployment is not set. Answer the prompt by typing 1,2 or 3 from the keyboard according to the options shown. Keep in mind that deployments are standalone and cannot be performed incrementally:
- **Option 1**: Deploy only to the development environment (Dev). Note that this option does not deploy release pipelines, as a minimum of two environments is required for pipeline releases.
- **Option 2**: Deploy to both development (Dev) and staging (Stage) environments.
- **Option 3** (Or Press Enter): Deploy to development (Dev), staging (Stage), and production (Prod) environments.
- This may take around **~30mins or more** to run end to end. So grab yourself a cup of coffee... ☕ But before you do so keep the following in mind:
- You might encounter deployment issues if the script attempts to create a Key Vault that conflicts with a previously soft-deleted Key Vault. In such cases, the deployment script may prompt you to confirm the purge of the previously deleted Key Vault.
- There are 3 points in time where you will need to authenticate to the databricks workspace, before the script continues to run. You will find the following message for the deployment of the dev, stage and production environments. Click the link highlighted in green, consent to authenticate to the databricks workspace and when the workspace opens successfully, return to the deployment windows and press Enter to continue: ![image](docs/images/databricks_ws.png)
Expand Down Expand Up @@ -434,9 +449,7 @@ The following lists some limitations of the solution and associated deployment s
- **Workaround**: Deployment add sensitive configuration as "secrets" in Variable Groups with the downside of duplicated information. If you wish, you may manually link a second Variable Group to KeyVault to pull out the secrets. KeyVault secret names should line up with required variables in the Azure DevOps pipelines. See [here](https://docs.microsoft.com/en-us/azure/devops/pipelines/library/variable-groups?view=azure-devops&tabs=yaml#link-secrets-from-an-azure-key-vault) for more information.
- Azure DevOps Service Connection Removal: If you encounter an error like: *"Cannot delete this service connection while federated credentials for app <app-id> exist in Entra tenant <tenant-id>. Please make sure federated credentials have been removed prior to deleting the service connection."* This issue occurs when you try to delete a Service Connection in the Azure DevOps (AzDo) portal, but the Service Connection has federated credentials that need to be manually removed from the Azure Portal.
- **Workaround - Manually Deleting Federated Credentials:**
Navigate to the Azure portal and locate your app registration under App Registrations. In the left navigation pane, select Certificates & Secrets and then the Federated Credentials
tab. Delete the federated credential from this section. Once the credential is deleted, you can proceed to delete the app registration in the Azure Portal and the Azure Service
Connection in the AzDo portal.
Navigate to the Azure portal and find your app registration under App Registrations. In the left-hand navigation menu, choose Certificates & Secrets, then navigate to the Federated Credentials tab. Remove the federated credential from this section. After deleting the credential, proceed to delete the app registration in the Azure portal and the Azure Service Connection in the Azure DevOps (AzDo) portal.
- Azure DevOps Environment and Approval Gates can only be managed via the UI, cannot be managed programmatically and was not incorporated in the automated deployment of the solution.
- **Workaround**: Approval Gates can be easily configured manually. See [here](https://docs.microsoft.com/en-us/azure/devops/pipelines/process/environments?view=azure-devops#approvals) for more information.
- ADF publishing through the CI/CD pipeline using the npm task still throws and error in the logs due to the missing publish_config.json file but the pipeline completes successfully.
Expand Down
28 changes: 15 additions & 13 deletions e2e_samples/parking_sensors/deploy.sh
Original file line number Diff line number Diff line change
Expand Up @@ -24,27 +24,29 @@ set -o nounset
. ./scripts/verify_prerequisites.sh



project=mdwdops # CONSTANT - this is prefixes to all resources of the Parking Sensor sample
# CONSTANT - this is prefixes to all resources of the Parking Sensor sample
project=mdwdops
github_repo_url="https://github.com/$GITHUB_REPO"

#Ask the user the following options:
bsherwin marked this conversation as resolved.
Show resolved Hide resolved
####
## 1) Only Dev
LiliamLeme marked this conversation as resolved.
Show resolved Hide resolved
## 2) Dev and Stage
## 3) All
####

###################
# DEPLOY ALL FOR EACH ENVIRONMENT
if [ -z "$ENV_DEPLOY" ]; then
read -r -p "Do you wish to deploy:"$'\n'" 1) Dev Environment Only?"$'\n'" 2) Dev and Stage Environments?"$'\n'" 3) Dev, Stage and Prod (Or Press Enter)?"$'\n'" Choose 1, 2 or 3: " ENV_DEPLOY
log "Option Selected: $ENV_DEPLOY" "info"
fi

for env_name in dev stg prod; do # dev stg prod
PROJECT=$project \
DEPLOYMENT_ID=$DEPLOYMENT_ID \
ENV_NAME=$env_name \
AZURE_LOCATION=$AZURE_LOCATION \
AZURE_SUBSCRIPTION_ID=$AZURE_SUBSCRIPTION_ID \
AZURESQL_SERVER_PASSWORD=$AZURESQL_SERVER_PASSWORD \
bash -c "./scripts/deploy_infrastructure.sh" # includes AzDevOps Azure Service Connections and Variable Groups
done
# Call the deploy function
deploy_infrastructure_environment "$ENV_DEPLOY" "$project"


###################
# Deploy AzDevOps Pipelines
###################

# Create AzDo Github Service Connection -- required only once for the entire deployment
PROJECT=$project \
Expand Down
55 changes: 53 additions & 2 deletions e2e_samples/parking_sensors/scripts/common.sh
Original file line number Diff line number Diff line change
Expand Up @@ -134,13 +134,19 @@ wait_for_process() {
log "Giving the portal $seconds seconds to process the information..."
sleep "$seconds"
}

cleanup_federated_credentials() {
##Function used in the Clean_up.sh and deploy_azdo_service_connections_azure.sh scripts
local sc_id=$1
local spnAppObjId=$(az devops service-endpoint show --id "$sc_id" --org "$AZDO_ORGANIZATION_URL" -p "$AZDO_PROJECT" --query "data.appObjectId" -o tsv)
# if the Service connection does not have an associated Service Principal,
# then it means it won't have associated federated credentials
if [ -z "$spnAppObjId" ]; then
log "Service Principal Object ID not found for Service Connection ID: $sc_id. Skipping federated credential cleanup."
return
fi

local spnCredlist=$(az ad app federated-credential list --id "$spnAppObjId" --query "[].id" -o json)
log "Found existing federated credentials. Deleting..."
log "Attempting to delete federated credentials."

# Sometimes the Azure Portal needs a little bit more time to process the information.
if [ -z "$spnCredlist" ]; then
Expand Down Expand Up @@ -169,3 +175,48 @@ cleanup_federated_credentials() {
log "Completed federated credential cleanup for the Service Principal: $spnAppObjId"
}

deploy_infrastructure_environment() {
##function to allow user deploy enviromnents
## 1) Only Dev
## 2) Dev and Stage
## 3) Dev, Stage and Prod
##Default is option 3.
ENV_DEPLOY=${1:-3}
project=${2:-mdwdops}
case $ENV_DEPLOY in
1)
log "Deploying Dev Environment only..."
env_names="dev"
;;
2)
log "Deploying Dev and Stage Environments..."
env_names="dev stg"
;;
3)
log "Full Deploy: Dev, Stage and Prod Environments..."
env_names="dev stg prod"
;;
*)
log "Invalid choice. Exiting..." "warning"
exit
;;
esac

# Loop through the environments and deploy
for env_name in $env_names; do
echo "Currently deploying to the environment: $env_name"
export PROJECT=$project
export DEPLOYMENT_ID=$DEPLOYMENT_ID
export ENV_NAME=$env_name
export AZURE_LOCATION=$AZURE_LOCATION
export AZURE_SUBSCRIPTION_ID=$AZURE_SUBSCRIPTION_ID
export AZURESQL_SERVER_PASSWORD=$AZURESQL_SERVER_PASSWORD
bash -c "./scripts/deploy_infrastructure.sh" || {
echo "Deployment failed for $env_name"
exit 1
}
export ENV_DEPLOY=$ENV_DEPLOY

done

}
22 changes: 13 additions & 9 deletions e2e_samples/parking_sensors/scripts/deploy_azdo_pipelines.sh
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ set -o nounset
# AZDO_PIPELINES_BRANCH_NAME
# DEV_DATAFACTORY_NAME

source ./scripts/common.sh
. ./scripts/init_environment.sh

# Retrieve Github Service Connection Id
github_sc_name="${PROJECT}-github"
Expand All @@ -57,12 +57,16 @@ create_azdo_pipeline "ci-qa-sql" "This pipeline builds the sql dacpac"

create_azdo_pipeline "ci-artifacts" "This pipeline publishes build artifacts"

###################
# Release Pipelines
cd_release_pipeline_id=$(create_azdo_pipeline "cd-release" "This pipeline releases across environments")


az pipelines variable create \
--name devAdfName \
--pipeline-id "$cd_release_pipeline_id" \
--value "$DEV_DATAFACTORY_NAME" \
-o none
###################
# Release Pipelines - only if it has at least 2 environments
if [ "$ENV_DEPLOY" -eq 2 ] || [ "$ENV_DEPLOY" -eq 3 ]; then
log " Release Pipeline are been created - option selected: $ENV_DEPLOY"
cd_release_pipeline_id=$(create_azdo_pipeline "cd-release" "This pipeline releases across environments")
az pipelines variable create \
--name devAdfName \
--pipeline-id "$cd_release_pipeline_id" \
--value "$DEV_DATAFACTORY_NAME" \
-o none
fi
7 changes: 6 additions & 1 deletion e2e_samples/parking_sensors/scripts/init_environment.sh
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,6 @@ az config set core.login_experience_v2=on
az account set -s $AZURE_SUBSCRIPTION_ID -o none

az devops configure --defaults organization=$AZDO_ORGANIZATION_URL project=$AZDO_PROJECT -o none

# check required variables are specified.

if [ -z "$GITHUB_REPO" ]
Expand Down Expand Up @@ -94,4 +93,10 @@ then
# set purge protection variable to true if the env variable has not been set
export ENABLE_KEYVAULT_PURGE_PROTECTION=${ENABLE_KEYVAULT_PURGE_PROTECTION:-true}
log "No ENABLE_KEYVAULT_PURGE specified. Defaulting to $ENABLE_KEYVAULT_PURGE_PROTECTION" "info"
fi

ENV_DEPLOY=${ENV_DEPLOY:-}
if [ -z "$ENV_DEPLOY" ]; then
export ENV_DEPLOY
log "No Deployment option was specified in the configuration" "info"
fi
Loading