diff --git a/README.md b/README.md index 1c66db11..7b1ee53b 100644 --- a/README.md +++ b/README.md @@ -46,11 +46,11 @@ The folder `examples` contains the following Terraform implementation examples : | Azure | ~~adb-external-hive-metastore~~ **REMOVED** | This example was removed in February 2026. External Hive metastore has been superseded by [Unity Catalog](https://docs.databricks.com/en/data-governance/unity-catalog/index.html). Use [adb-unity-catalog-basic-demo](examples/adb-unity-catalog-basic-demo/) instead. | | Azure | [adb-kafka](examples/adb-kafka/) | ADB - single node kafka template | | Azure | [adb-private-links](examples/adb-private-links/) | Azure Databricks Private Links | -| Azure | [adb-squid-proxy](examples/adb-squid-proxy/) | ADB clusters with HTTP proxy | +| Azure | ~~adb-squid-proxy~~ **REMOVED** | This example was removed in August 2026. | | Azure | [adb-teradata](examples/adb-teradata/) | ADB with single VM Teradata integration | -| Azure | [adb-uc](examples/adb-uc/) | ADB Unity Catalog Process | +| Azure | ~~adb-uc~~ **REMOVED** | This example was removed in August 2026. Use [adb-unity-catalog-basic-demo](examples/adb-unity-catalog-basic-demo/) instead. | | Azure | [adb-unity-catalog-basic-demo](examples/adb-unity-catalog-basic-demo/) | ADB Unity Catalog end-to-end demo including UC metastore setup, Users/groups sync from AAD to databricks account, UC Catalog, External locations, Schemas, & Access Grants | -| Azure | [adb-overwatch](examples/adb-overwatch/) | Overwatch multi-workspace deployment on Azure | +| Azure | ~~adb-overwatch~~ **REMOVED** | This example and its supporting `adb-overwatch-*` modules were removed in August 2026. | | AWS | [aws-workspace-basic](examples/aws-workspace-basic/) | Provisioning AWS Databricks E2 | | AWS | [aws-workspace-with-firewall](examples/aws-workspace-with-firewall/) | Provisioning AWS Databricks E2 with an AWS Firewall | | AWS | [aws-exfiltration-protection](examples/aws-exfiltration-protection/) | An implementation of [Data Exfiltration Protection on AWS](https://www.databricks.com/blog/2021/02/02/data-exfiltration-protection-with-databricks-on-aws.html) | @@ -77,11 +77,6 @@ The folder `modules` contains the following Terraform modules : | Azure | [adb-with-private-link-standard](modules/adb-with-private-link-standard/) | Provisioning Databricks on Azure with Private Link - Standard deployment | | Azure | [adb-exfiltration-protection](modules/adb-exfiltration-protection/) | A sample implementation of [Data Exfiltration Protection](https://www.databricks.com/blog/2020/03/27/data-exfiltration-protection-with-azure-databricks.html) | | Azure | [adb-with-private-links-exfiltration-protection](modules/adb-with-private-links-exfiltration-protection/) | Provisioning Databricks on Azure with Private Link and [Data Exfiltration Protection](https://www.databricks.com/blog/2020/03/27/data-exfiltration-protection-with-azure-databricks.html) | -| Azure | [adb-overwatch-regional-config](modules/adb-overwatch-regional-config/) | Overwatch regional configuration on Azure | -| Azure | [adb-overwatch-mws-config](modules/adb-overwatch-mws-config/) | Overwatch multi-workspace deployment on Azure | -| Azure | [adb-overwatch-main-ws](modules/adb-overwatch-main-ws/) | Main Overwatch workspace deployment | -| Azure | [adb-overwatch-ws-to-monitor](modules/adb-overwatch-ws-to-monitor/) | Overwatch deployment on the Azure workspace to monitor | -| Azure | [adb-overwatch-analysis](modules/adb-overwatch-analysis/) | Overwatch analysis notebooks deployment on Azure | | AWS | [aws-workspace-basic](modules/aws-workspace-basic/) | Provisioning AWS Databricks E2 | | AWS | [aws-databricks-base-infra](modules/aws-databricks-base-infra/) | Provisioning AWS Infrastructure to be used for the deployment of a Databricks E2 workspace | | AWS | [aws-databricks-unity-catalog](modules/aws-databricks-unity-catalog/) | Provisioning the AWS Infrastructure and setting up the metastore for Databricks Unity Catalog | diff --git a/examples/adb-overwatch/Makefile b/examples/adb-overwatch/Makefile deleted file mode 100644 index 653039d8..00000000 --- a/examples/adb-overwatch/Makefile +++ /dev/null @@ -1,7 +0,0 @@ -.PHONY: docs test_docs - -docs: - terraform-docs -c ../../.terraform-docs.yml . - -test_docs: - terraform-docs -c ../../.terraform-docs.yml --output-check . diff --git a/examples/adb-overwatch/README.md b/examples/adb-overwatch/README.md deleted file mode 100644 index e3459207..00000000 --- a/examples/adb-overwatch/README.md +++ /dev/null @@ -1,104 +0,0 @@ -# Deploying Overwatch on Azure Databricks - -This example contains Terraform code used to deploy Overwatch using the following modules : -- [adb-overwatch-regional-config](../../modules/adb-overwatch-regional-config) -- [adb-overwatch-mws-config](../../modules/adb-overwatch-mws-config) -- [adb-overwatch-main-ws](../../modules/adb-overwatch-main-ws) -- [adb-overwatch-ws-to-monitor](../../modules/adb-overwatch-ws-to-monitor) -- [adb-overwatch-analysis](../../modules/adb-overwatch-analysis) - - -## Example content - -This code uses the [multi-workspace deployment of Overwatch](https://databrickslabs.github.io/overwatch/deployoverwatch/cloudinfra/azure/#reference-architecturehttps://databrickslabs.github.io/overwatch/deployoverwatch/cloudinfra/azure/#reference-architecture). Overwatch runs in a dedicated, or existing, Azure Databricks workspace, and monitors the specified workspaces in the config file [overwatch_deployment_config.csv](./overwatch_deployment_config.csv). This configuration file is generated automatically by the module [adb-overwatch-ws-to-monitor](../../modules/adb-overwatch-ws-to-monitor). - - ![Overwatch_Arch_Azure](https://user-images.githubusercontent.com/103026825/230571464-5892c5c7-82c2-4808-9003-61b501b75f69.png?raw=true) - -The deployment is structured as followed : -* Use an existing **Resource group** -* Deploy **Eventhubs** topic per workspace, that could be in the same **Eventhubs** namespace -* Deploy **Storage Accounts**, one for the cluster logs and one for Overwatch database output -* Deploy the dedicated **Azure Databricks** workspace, or use an existing one for Overwatch, with some Databricks quick-start notebooks to analyse the results -* Deploy **Azure Key Vault** to store the secrets -* Configure **Role Assignments** and **mounts** to attribute the necessary permissions -* Configure **Diagnostic Logs** on the Databricks workspaces to monitor - -> **Note** -> As Terraform requires providers and modules to be declared statically before deploying the resources, we are using in this example a [bash script](./dynamic_providers_modules_generation.sh) -> that generates the provider configurations for N workspaces along with the modules references. - -## How to use - -1. Configure the workspaces that will be observed by Overwatch in [workspaces_to_monitor.json](./workspaces_to_monitor.json) -2. Make the script [dynamic_providers_modules_generation.sh](./dynamic_providers_modules_generation.sh) executable : `chmod +x dynamic_providers_modules_generation.sh` -3. Update the `terraform.tfvars` file with your environment values -4. Run the script [dynamic_providers_modules_generation.sh](./dynamic_providers_modules_generation.sh) : `./dynamic_providers_modules_generation.sh`. This will dynamically generate `providers_ws_to_monitor.tf` and `main_ws_to_monitor.tf` files with the right terraform setup for all the workspaces defined in [workspaces_to_monitor.json](./workspaces_to_monitor.json) -5. Run `terraform init` to initialize terraform and get provider ready -6. Run `terraform plan` to check the resources that are affected -7. Run `terraform apply` to create the resources - - -## Requirements - -No requirements. - -## Providers - -| Name | Version | -|------|---------| -| [random](#provider\_random) | n/a | - -## Modules - -| Name | Source | Version | -|------|--------|---------| -| [adb-overwatch-analysis](#module\_adb-overwatch-analysis) | ../../modules/adb-overwatch-analysis | n/a | -| [adb-overwatch-main-ws](#module\_adb-overwatch-main-ws) | ../../modules/adb-overwatch-main-ws | n/a | -| [adb-overwatch-mws-config](#module\_adb-overwatch-mws-config) | ../../modules/adb-overwatch-mws-config | n/a | -| [adb-overwatch-regional-config](#module\_adb-overwatch-regional-config) | ../../modules/adb-overwatch-regional-config | n/a | - -## Resources - -| Name | Type | -|------|------| -| [random_string.strapp](https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/string) | resource | - -## Inputs - -| Name | Description | Type | Default | Required | -|------|-------------|------|---------|:--------:| -| [ehn\_name](#input\_ehn\_name) | Eventhubs namespace name | `string` | n/a | yes | -| [key\_vault\_prefix](#input\_key\_vault\_prefix) | AKV prefix | `string` | n/a | yes | -| [logs\_sa\_name](#input\_logs\_sa\_name) | Logs storage account name | `string` | n/a | yes | -| [overwatch\_spn\_app\_id](#input\_overwatch\_spn\_app\_id) | Azure SPN application ID | `string` | n/a | yes | -| [overwatch\_spn\_secret](#input\_overwatch\_spn\_secret) | Azure SPN secret | `string` | n/a | yes | -| [overwatch\_ws\_name](#input\_overwatch\_ws\_name) | Overwatch Databricks workspace name | `string` | n/a | yes | -| [ow\_sa\_name](#input\_ow\_sa\_name) | Overwatch ETL storage account name | `string` | n/a | yes | -| [rg\_name](#input\_rg\_name) | Resource group name | `string` | n/a | yes | -| [subscription\_id](#input\_subscription\_id) | Azure subscription ID | `string` | n/a | yes | -| [tenant\_id](#input\_tenant\_id) | Azure tenant ID | `string` | n/a | yes | -| [active](#input\_active) | Whether or not the workspace should be validated / deployed | `bool` | `true` | no | -| [api\_waiting\_time](#input\_api\_waiting\_time) | API Tunable - Overwatch makes async api calls in parallel, api\_waiting\_time signifies the max wait time in case of no response received from the api call | `string` | `""` | no | -| [auditlog\_prefix\_source\_path](#input\_auditlog\_prefix\_source\_path) | Location of auditlog (AWS/GCP Only) | `string` | `""` | no | -| [automated\_dbu\_price](#input\_automated\_dbu\_price) | Contract price for automated DBUs | `number` | `0.3` | no | -| [databricks\_secret\_scope\_name](#input\_databricks\_secret\_scope\_name) | Databricks secret scope name (backed by Azure Key-Vault) | `string` | `"overwatch-akv"` | no | -| [enable\_unsafe\_SSL](#input\_enable\_unsafe\_SSL) | API Tunable - Enables unsafe SSL | `string` | `""` | no | -| [error\_batch\_size](#input\_error\_batch\_size) | API Tunable - Indicates the size of the error writer buffer containing API call errors | `string` | `""` | no | -| [excluded\_scopes](#input\_excluded\_scopes) | Scopes that should not be excluded from the pipelines | `string` | `""` | no | -| [interactive\_dbu\_price](#input\_interactive\_dbu\_price) | Contract price for interactive DBUs | `number` | `0.55` | no | -| [jobs\_light\_dbu\_price](#input\_jobs\_light\_dbu\_price) | Contract price for interactive DBUs | `number` | `0.1` | no | -| [max\_days](#input\_max\_days) | This is the max incremental days that will be loaded. Usually only relevant for historical loading and rebuilds | `number` | `30` | no | -| [proxy\_host](#input\_proxy\_host) | Proxy url for the workspace | `string` | `""` | no | -| [proxy\_password\_key](#input\_proxy\_password\_key) | Key which contains proxy password | `string` | `""` | no | -| [proxy\_password\_scope](#input\_proxy\_password\_scope) | Scope which contains the proxy password key | `string` | `""` | no | -| [proxy\_port](#input\_proxy\_port) | Proxy port for the workspace | `string` | `""` | no | -| [proxy\_user\_name](#input\_proxy\_user\_name) | Proxy user name for the workspace | `string` | `""` | no | -| [sql\_compute\_dbu\_price](#input\_sql\_compute\_dbu\_price) | Contract price for DBSQL DBUs | `number` | `0.22` | no | -| [success\_batch\_size](#input\_success\_batch\_size) | API Tunable - Indicates the size of the buffer on filling of which the result will be written to a temp location. This is used to tune performance in certain circumstance | `string` | `""` | no | -| [thread\_pool\_size](#input\_thread\_pool\_size) | API Tunable - Max number of API calls Overwatch is allowed to make in parallel | `string` | `""` | no | -| [use\_existing\_overwatch\_ws](#input\_use\_existing\_overwatch\_ws) | Overwatch ETL storage prefix, which represents a mount point to the ETL storage account | `string` | `false` | no | - -## Outputs - -No outputs. - \ No newline at end of file diff --git a/examples/adb-overwatch/dynamic_providers_modules_generation.sh b/examples/adb-overwatch/dynamic_providers_modules_generation.sh deleted file mode 100755 index 0c9fe7a3..00000000 --- a/examples/adb-overwatch/dynamic_providers_modules_generation.sh +++ /dev/null @@ -1,94 +0,0 @@ -#!/bin/bash - -# Load JSON file into variable -json=$(cat workspaces_to_monitor.json) - -# set the name of the module's source -module_source="../../modules/adb-overwatch-ws-to-monitor" -modules_list=(module.adb-overwatch-mws-config) - -# Loop through JSON objects -for row in $(echo "${json}" | jq -r '.[] | @base64'); do - # Decode JSON object - _jq() { - echo "${row}" | base64 --decode | jq -r "${1}" - } - - # Generate Terraform Databricks provider configuration - workspace_name=$(_jq '.workspace_name') - host=$(_jq '.host') - - echo -e "provider \"databricks\" { - alias = \"$workspace_name\" - host = \"$host\" -}\n" >> providers_ws_to_monitor.tf - - # Set the name of the module dynamically - module_name="adb-overwatch-monitor-$workspace_name" - - # Set the name of the provider and the Databricks value you want to use - provider_name="databricks" - databricks_provider_value="databricks.$workspace_name" - - # Generate the JSON block - json_block=$(cat <> main_ws_to_monitor.tf - - element="module.adb-overwatch-monitor-$workspace_name" - modules_list+=("$element") - -done - -echo "workspace_name,workspace_id,workspace_url,api_url,cloud,primordial_date,etl_storage_prefix,etl_database_name,consumer_database_name,secret_scope,secret_key_dbpat,auditlogprefix_source_path,eh_name,eh_scope_key,interactive_dbu_price,automated_dbu_price,sql_compute_dbu_price,jobs_light_dbu_price,max_days,excluded_scopes,active,proxy_host,proxy_port,proxy_user_name,proxy_password_scope,proxy_password_key,success_batch_size,error_batch_size,enable_unsafe_SSL,thread_pool_size,api_waiting_time" > overwatch_deployment_config.csv - -dependencies="["$(IFS=, ; echo "${modules_list[*]}")"]" - -echo -e "resource \"databricks_dbfs_file\" \"overwatch_deployment_config\" { - provider = databricks.adb-ow-main-ws - - source = \"\${path.module}/overwatch_deployment_config.csv\" - path = \"/mnt/\${module.adb-overwatch-mws-config.databricks_mount_db_name}/config/overwatch_deployment_config.csv\" - depends_on = $dependencies -}" >> main_ws_to_monitor.tf \ No newline at end of file diff --git a/examples/adb-overwatch/main.tf b/examples/adb-overwatch/main.tf deleted file mode 100644 index 4f01acc8..00000000 --- a/examples/adb-overwatch/main.tf +++ /dev/null @@ -1,62 +0,0 @@ -resource "random_string" "strapp" { - length = 5 - lower = true - upper = false - special = false -} - - -module "adb-overwatch-regional-config" { - source = "../../modules/adb-overwatch-regional-config" - - random_string = random_string.strapp.result - rg_name = var.rg_name - overwatch_spn_app_id = var.overwatch_spn_app_id - ehn_name = var.ehn_name - logs_sa_name = var.logs_sa_name - key_vault_prefix = var.key_vault_prefix - overwatch_spn_secret = var.overwatch_spn_secret -} - - -module "adb-overwatch-main-ws" { - source = "../../modules/adb-overwatch-main-ws" - - subscription_id = var.subscription_id - rg_name = var.rg_name - use_existing_ws = var.use_existing_overwatch_ws - overwatch_ws_name = var.overwatch_ws_name -} - - -module "adb-overwatch-mws-config" { - source = "../../modules/adb-overwatch-mws-config" - providers = { - databricks = databricks.adb-ow-main-ws - } - - tenant_id = var.tenant_id - rg_name = var.rg_name - overwatch_spn_app_id = var.overwatch_spn_app_id - overwatch_ws_name = var.overwatch_ws_name - akv_name = module.adb-overwatch-regional-config.akv_name - databricks_secret_scope_name = var.databricks_secret_scope_name - latest_dbr_lts = module.adb-overwatch-main-ws.latest_lts - random_string = random_string.strapp.result - ow_sa_name = var.ow_sa_name - - depends_on = [module.adb-overwatch-main-ws, module.adb-overwatch-regional-config] -} - - -module "adb-overwatch-analysis" { - source = "../../modules/adb-overwatch-analysis" - providers = { - databricks = databricks.adb-ow-main-ws - } - - rg_name = var.rg_name - overwatch_ws_name = var.overwatch_ws_name - - depends_on = [module.adb-overwatch-main-ws] -} \ No newline at end of file diff --git a/examples/adb-overwatch/overwatch_deployment_config.csv b/examples/adb-overwatch/overwatch_deployment_config.csv deleted file mode 100644 index 7d51a5c0..00000000 --- a/examples/adb-overwatch/overwatch_deployment_config.csv +++ /dev/null @@ -1 +0,0 @@ -workspace_name,workspace_id,workspace_url,api_url,cloud,primordial_date,etl_storage_prefix,etl_database_name,consumer_database_name,secret_scope,secret_key_dbpat,auditlogprefix_source_path,eh_name,eh_scope_key,interactive_dbu_price,automated_dbu_price,sql_compute_dbu_price,jobs_light_dbu_price,max_days,excluded_scopes,active,proxy_host,proxy_port,proxy_user_name,proxy_password_scope,proxy_password_key,success_batch_size,error_batch_size,enable_unsafe_SSL,thread_pool_size,api_waiting_time \ No newline at end of file diff --git a/examples/adb-overwatch/providers.tf b/examples/adb-overwatch/providers.tf deleted file mode 100644 index aa991818..00000000 --- a/examples/adb-overwatch/providers.tf +++ /dev/null @@ -1,21 +0,0 @@ -terraform { - required_providers { - azurerm = { - source = "hashicorp/azurerm" - } - - databricks = { - source = "databricks/databricks" - } - } -} - -provider "azurerm" { - features {} - subscription_id = var.subscription_id -} - -provider "databricks" { - alias = "adb-ow-main-ws" - host = module.adb-overwatch-main-ws.adb_ow_main_ws_url -} \ No newline at end of file diff --git a/examples/adb-overwatch/terraform.tfvars b/examples/adb-overwatch/terraform.tfvars deleted file mode 100644 index 7810d0bb..00000000 --- a/examples/adb-overwatch/terraform.tfvars +++ /dev/null @@ -1,11 +0,0 @@ -subscription_id = "" -tenant_id = "" -overwatch_spn_app_id = "" -overwatch_spn_secret = "" -ehn_name = "" -logs_sa_name = "" -ow_sa_name = "" -key_vault_prefix = "" -rg_name = "" -overwatch_ws_name = "" -use_existing_overwatch_ws = false \ No newline at end of file diff --git a/examples/adb-overwatch/variables.tf b/examples/adb-overwatch/variables.tf deleted file mode 100644 index 4bcf1458..00000000 --- a/examples/adb-overwatch/variables.tf +++ /dev/null @@ -1,170 +0,0 @@ -variable "subscription_id" { - type = string - description = "Azure subscription ID" -} - -variable "rg_name" { - type = string - description = "Resource group name" -} - -variable "ehn_name" { - type = string - description = "Eventhubs namespace name" -} - -variable "tenant_id" { - type = string - description = "Azure tenant ID" -} - -variable "overwatch_spn_app_id" { - type = string - description = "Azure SPN application ID" -} - -variable "overwatch_spn_secret" { - type = string - description = "Azure SPN secret" -} - -variable "logs_sa_name" { - type = string - description = "Logs storage account name" -} - -variable "ow_sa_name" { - type = string - description = "Overwatch ETL storage account name" -} - -variable "key_vault_prefix" { - type = string - description = "AKV prefix" -} - -variable "overwatch_ws_name" { - type = string - description = "Overwatch Databricks workspace name" -} - -variable "databricks_secret_scope_name" { - type = string - description = "Databricks secret scope name (backed by Azure Key-Vault)" - default = "overwatch-akv" -} - -variable "use_existing_overwatch_ws" { - type = string - description = "Overwatch ETL storage prefix, which represents a mount point to the ETL storage account" - default = false -} - -variable "interactive_dbu_price" { - type = number - description = "Contract price for interactive DBUs" - default = 0.55 -} - -variable "automated_dbu_price" { - type = number - description = "Contract price for automated DBUs" - default = 0.3 -} - -variable "sql_compute_dbu_price" { - type = number - description = "Contract price for DBSQL DBUs" - default = 0.22 -} - -variable "jobs_light_dbu_price" { - type = number - description = "Contract price for interactive DBUs" - default = 0.1 -} - -variable "max_days" { - type = number - description = "This is the max incremental days that will be loaded. Usually only relevant for historical loading and rebuilds" - default = 30 -} - -variable "excluded_scopes" { - type = string - description = "Scopes that should not be excluded from the pipelines" - default = "" -} - -variable "active" { - type = bool - description = "Whether or not the workspace should be validated / deployed" - default = true -} - -variable "proxy_host" { - type = string - description = "Proxy url for the workspace" - default = "" -} - -variable "proxy_port" { - type = string - description = "Proxy port for the workspace" - default = "" - -} - -variable "proxy_user_name" { - type = string - description = "Proxy user name for the workspace" - default = "" -} - -variable "proxy_password_scope" { - type = string - description = "Scope which contains the proxy password key" - default = "" -} - -variable "proxy_password_key" { - type = string - description = "Key which contains proxy password" - default = "" -} - -variable "success_batch_size" { - type = string - description = "API Tunable - Indicates the size of the buffer on filling of which the result will be written to a temp location. This is used to tune performance in certain circumstance" - default = "" -} - -variable "error_batch_size" { - type = string - description = "API Tunable - Indicates the size of the error writer buffer containing API call errors" - default = "" -} - -variable "enable_unsafe_SSL" { - type = string - description = "API Tunable - Enables unsafe SSL" - default = "" -} - -variable "thread_pool_size" { - type = string - description = "API Tunable - Max number of API calls Overwatch is allowed to make in parallel" - default = "" -} - -variable "api_waiting_time" { - type = string - description = "API Tunable - Overwatch makes async api calls in parallel, api_waiting_time signifies the max wait time in case of no response received from the api call" - default = "" -} - -variable "auditlog_prefix_source_path" { - type = string - description = "Location of auditlog (AWS/GCP Only)" - default = "" -} \ No newline at end of file diff --git a/examples/adb-overwatch/workspaces_to_monitor.json b/examples/adb-overwatch/workspaces_to_monitor.json deleted file mode 100644 index 68e08f1a..00000000 --- a/examples/adb-overwatch/workspaces_to_monitor.json +++ /dev/null @@ -1,10 +0,0 @@ -[ - { - "workspace_name": "ws-1", - "host": "https://adb-8861999999206043.3.azuredatabricks.net" - }, - { - "workspace_name": "ws-2", - "host": "https://adb-6396842068490886.6.azuredatabricks.net" - } -] \ No newline at end of file diff --git a/examples/adb-squid-proxy/Makefile b/examples/adb-squid-proxy/Makefile deleted file mode 100644 index 653039d8..00000000 --- a/examples/adb-squid-proxy/Makefile +++ /dev/null @@ -1,7 +0,0 @@ -.PHONY: docs test_docs - -docs: - terraform-docs -c ../../.terraform-docs.yml . - -test_docs: - terraform-docs -c ../../.terraform-docs.yml --output-check . diff --git a/examples/adb-squid-proxy/README.md b/examples/adb-squid-proxy/README.md deleted file mode 100644 index cd5d3b53..00000000 --- a/examples/adb-squid-proxy/README.md +++ /dev/null @@ -1,99 +0,0 @@ -## Objective -1. Use packer to create azure image that functions as a squid proxy. -2. Use terraform to create vm / vmss, as squid proxy instance(s). -3. Deploy necessary networking infra. -4. Deploy Azure Databricks Workspace with all outbound traffic going through squid proxy, as such, we can achieve granular ACL control for outbound destinations. - -## Credits - -Credits to andrew.weaver@databricks.com for creating the original instructions to set up squid proxy and shu.wu@databricks.com for efforts in testing and debugging init scripts for databricks cluster proxy setup. - -## Overall Architecture: -![alt text](https://raw.githubusercontent.com/databricks/terraform-databricks-examples/main/examples/adb-squid-proxy/images/adb-squid-proxy.png?raw=true) - -Narratives: Databricks workspace 1 is deployed into a VNet, which is peered to another VNet hosting a single Squid proxy server, every databricks spark cluster will be configured using init script to direct traffic to this Squid server. We control ACL in squid.conf, such that we can allow/deny traffic to certain outbound destinations. - -Does this apply to azure services as well (that goes through azure backbone network) - -## Execution Steps: -### Step 1: - -This step creates an empty resource group for hosting custom-built squid image and a local file of config variables in `/packer/os`. - -Redirect to `/packer/tf_coldstart`, run: - 1. `terraform init` - 2. `terraform apply` - -### Step 2: - -This step you will use packer to build squid image. Packer will read the auto-generated `*.auto.pkrvars.hcl` file and build the image. - -Redirect to `/packer/os`, run: - 1. `packer build .` - -### Step 3: - -This step creates all the other infra for this project, specified in `/main`. - -Redirect to `/main`, run: - 1. `terraform init` - 2. `terraform apply` - -Now in folder of `/main`, you can find the auto-generated private key for ssh, to ssh into the provisioned vm, run: -`ssh -i ./ssh_private.pem azureuser@52.230.84.169`, change to the public ip of the squid vm accordingly. Check the nsg rules of the squid vm, we have inbound rule 300 allowing any source for ssh, this is for testing purpose only! You do not need ssh on squid vm for production setup. Once you ssh into squid vm, vi /etc/squid/squid.conf and you will find similar content like: - -![alt text](https://raw.githubusercontent.com/databricks/terraform-databricks-examples/main/examples/adb-squid-proxy/images/squid-conf.png?raw=true) - -The content was auto inserted by packer in step 2. - -### Step 4: - -Open your databricks workspace, you will find a notebook been created in `Shared/` folder, this is the notebook to create the cluster init script. -Create a small vanilla cluster to run this notebook. - -### Step 5: -Spin up another cluster using the init script generated in step 4, spark traffic will be routed to the squid proxy. - -![alt text](https://raw.githubusercontent.com/databricks/terraform-databricks-examples/main/examples/adb-squid-proxy/images/set-proxy.png?raw=true) - -### Step 6: - -Now all your clusters that spins up using this init script, will route spark/non-spark traffic to the squid proxy and ACL rules in squid.conf will be applied. Example effects are shown below: - -![alt text](https://raw.githubusercontent.com/databricks/terraform-databricks-examples/main/examples/adb-squid-proxy/images/http-proxy.png?raw=true) - -Traffic to storage accounts will also be allowed / blocked by the proxy. These rules are to be set in `/packer/scripts/setproxy.sh` script. - -## Conclusion - -We used a single instance of squid proxy server to control granular outbound traffic from Databricks clusters. You can use cluster proxy to enforce the init script, such that all clusters will abide to the init script config and go through the proxy. - -## Future Work - -To expand to VMSS from current 1 vm setup, with load balancer. For now this project achieves the purpose of granular outbound traffic control, without using a firewall. - - -## Requirements - -No requirements. - -## Providers - -No providers. - -## Modules - -No modules. - -## Resources - -No resources. - -## Inputs - -No inputs. - -## Outputs - -No outputs. - diff --git a/examples/adb-squid-proxy/images/adb-squid-proxy.png b/examples/adb-squid-proxy/images/adb-squid-proxy.png deleted file mode 100644 index 16ab5d62..00000000 Binary files a/examples/adb-squid-proxy/images/adb-squid-proxy.png and /dev/null differ diff --git a/examples/adb-squid-proxy/images/http-proxy.png b/examples/adb-squid-proxy/images/http-proxy.png deleted file mode 100644 index d27d8081..00000000 Binary files a/examples/adb-squid-proxy/images/http-proxy.png and /dev/null differ diff --git a/examples/adb-squid-proxy/images/set-proxy.png b/examples/adb-squid-proxy/images/set-proxy.png deleted file mode 100644 index 5c78b59c..00000000 Binary files a/examples/adb-squid-proxy/images/set-proxy.png and /dev/null differ diff --git a/examples/adb-squid-proxy/images/squid-conf.png b/examples/adb-squid-proxy/images/squid-conf.png deleted file mode 100644 index 93dce03c..00000000 Binary files a/examples/adb-squid-proxy/images/squid-conf.png and /dev/null differ diff --git a/examples/adb-squid-proxy/main/.terraform.lock.hcl b/examples/adb-squid-proxy/main/.terraform.lock.hcl deleted file mode 100644 index 12e0f418..00000000 --- a/examples/adb-squid-proxy/main/.terraform.lock.hcl +++ /dev/null @@ -1,129 +0,0 @@ -# This file is maintained automatically by "terraform init". -# Manual edits may be lost in future updates. - -provider "registry.terraform.io/databrickslabs/databricks" { - version = "0.3.10" - constraints = "0.3.10" - hashes = [ - "h1:u/RRXN0G5gkb3A41b3lEj/OytyVmqecmINQzhvOHeHs=", - "zh:002ecdbf4bdffc36cefe9299aaaf3c209ab2093a61f522347fe3a34ef9def5a3", - "zh:1bf683cb6efdfae289bed84d071af6c77f8497d723ee6254aba8722e4574a87d", - "zh:24247a8ae1855cd26470c2ecb362d503f293e4ade2a8b743abb34b49c9c4dae4", - "zh:39d09541981ef278c813b958a91fe191d361c73aedbdb533d6b86aeccce45e45", - "zh:3fada6aa9d86a3a3415e3f6a6475fd7f6f53d564c5614f7c5e82f9d7104a192f", - "zh:58fb40c5be7539f7eed86458915ea920a5fa6ccfb424678804cad9e461e83dea", - "zh:bcdaef7d181890b1b953ac510b53a71c2084b3bf221dee1f40afdb13b12a5a63", - "zh:bf61fa45d0b813dce7d155617dfdd49f3640bcd44a8e9bc93740833a84f2734c", - "zh:d1f0e66e9c05dfed041dc53c5d005f8132b69975bbab79336cdc9e04ba0ce7c8", - ] -} - -provider "registry.terraform.io/hashicorp/azurerm" { - version = "2.83.0" - constraints = "2.83.0" - hashes = [ - "h1:azYL1nLRrCeulAxVMBWClng0Q84zwcACzVLhyAcHtaY=", - "zh:238dea5f98d8c6e43172d41efb53d60133766c5e94f9ace7df49ab635b85ad04", - "zh:408e8ceda6f621db27f1392f6c676eb33e4c95e4aa75a2345b0451d0e788b6b9", - "zh:4291a57ccddd81991fe0fc092050ff67b85991a8b1dc4534a7d4e87b11f80a47", - "zh:498624df29576f267512e7d5c994319c455a585b28b8ba34611492bae2f2f36e", - "zh:69a11f895730d3f6c88537ad44feb2ca6c8a23b16d28ac4600279b74d3de5b68", - "zh:7e0119214fb775de696680538c50fa2fa605c0fd6ce2ea8cdf5edae65990ea87", - "zh:ac9023c966e16dee974ed0bfc583827d9c0e3db13291e8504e8ddd55589f376b", - "zh:da8faf1869b84cd8d51d6e7f63d77b1f250a6e6ee006b45314aa45628f4497b8", - "zh:dff619d8dcd5d0896104c518c40df8a4c44496d59ff2268a8e28d187c21c216d", - "zh:e4da8cde2158c1a6224aed77bf722cbdfe2429e2f504bfbf1eb1ab5572c51d2c", - "zh:f987c55b147f814023e32452d6c6aafb2ffa64391f13f4a8137c4f2cbde01f6a", - ] -} - -provider "registry.terraform.io/hashicorp/external" { - version = "2.1.1" - hashes = [ - "h1:5YnKC77SbQ7ayCcjpWkDF8Hv6wP3ZUFNDcGKsHol5yI=", - "zh:2726dcef652c01c718b6400ac22e598e7d296baa91ab5ea6d85bd94d1db229d9", - "zh:3923daa6deb847f88f0b1bf1481066f9d61f49557108945be7e2960cb1b61d0d", - "zh:3d8e894d60b362cb7fbdf22889d69f3c96a3c98ed2fec492dca28f52e07254e7", - "zh:43f1501519e2ee0dfa35c827cd2b24b4940a11d0348e77b20ad2d77bb22d9b11", - "zh:52e75f6c880ca5aa1175f288396101b625eb227ca8b89f31ff9be30e4280cfba", - "zh:6b0f7951cc9f4a7b06f399505d590fed637d07eacbae7781a7fab81dc6e26f99", - "zh:8d7103552cadda26c01230722184c0a7489515cfa8d1fabfc1aa95aecb5490f6", - "zh:962df74629cfefe8119b2dc23abc34bebd4bd0da506062732edca8dc4427c130", - "zh:b66e41a41140f5e1b960345fc91670dea12a9e1d23ea9dc2ad456cf9224e199d", - "zh:c4df3d6ea9dc6d8803f00400f4e075a43fa6ae9dac160b46c9de4b8714428f3d", - "zh:cab80dca018c63619fa7a9f71be329bb43f174a2b07f65ba608d59146a487a35", - ] -} - -provider "registry.terraform.io/hashicorp/local" { - version = "2.1.0" - hashes = [ - "h1:KfieWtVyGWwplSoLIB5usKAUnrIkDQBkWaR5TI+4WYg=", - "zh:0f1ec65101fa35050978d483d6e8916664b7556800348456ff3d09454ac1eae2", - "zh:36e42ac19f5d68467aacf07e6adcf83c7486f2e5b5f4339e9671f68525fc87ab", - "zh:6db9db2a1819e77b1642ec3b5e95042b202aee8151a0256d289f2e141bf3ceb3", - "zh:719dfd97bb9ddce99f7d741260b8ece2682b363735c764cac83303f02386075a", - "zh:7598bb86e0378fd97eaa04638c1a4c75f960f62f69d3662e6d80ffa5a89847fe", - "zh:ad0a188b52517fec9eca393f1e2c9daea362b33ae2eb38a857b6b09949a727c1", - "zh:c46846c8df66a13fee6eff7dc5d528a7f868ae0dcf92d79deaac73cc297ed20c", - "zh:dc1a20a2eec12095d04bf6da5321f535351a594a636912361db20eb2a707ccc4", - "zh:e57ab4771a9d999401f6badd8b018558357d3cbdf3d33cc0c4f83e818ca8e94b", - "zh:ebdcde208072b4b0f8d305ebf2bfdc62c926e0717599dcf8ec2fd8c5845031c3", - "zh:ef34c52b68933bedd0868a13ccfd59ff1c820f299760b3c02e008dc95e2ece91", - ] -} - -provider "registry.terraform.io/hashicorp/null" { - version = "3.1.0" - hashes = [ - "h1:xhbHC6in3nQryvTQBWKxebi3inG5OCgHgc4fRxL0ymc=", - "zh:02a1675fd8de126a00460942aaae242e65ca3380b5bb192e8773ef3da9073fd2", - "zh:53e30545ff8926a8e30ad30648991ca8b93b6fa496272cd23b26763c8ee84515", - "zh:5f9200bf708913621d0f6514179d89700e9aa3097c77dac730e8ba6e5901d521", - "zh:9ebf4d9704faba06b3ec7242c773c0fbfe12d62db7d00356d4f55385fc69bfb2", - "zh:a6576c81adc70326e4e1c999c04ad9ca37113a6e925aefab4765e5a5198efa7e", - "zh:a8a42d13346347aff6c63a37cda9b2c6aa5cc384a55b2fe6d6adfa390e609c53", - "zh:c797744d08a5307d50210e0454f91ca4d1c7621c68740441cf4579390452321d", - "zh:cecb6a304046df34c11229f20a80b24b1603960b794d68361a67c5efe58e62b8", - "zh:e1371aa1e502000d9974cfaff5be4cfa02f47b17400005a16f14d2ef30dc2a70", - "zh:fc39cc1fe71234a0b0369d5c5c7f876c71b956d23d7d6f518289737a001ba69b", - "zh:fea4227271ebf7d9e2b61b89ce2328c7262acd9fd190e1fd6d15a591abfa848e", - ] -} - -provider "registry.terraform.io/hashicorp/random" { - version = "3.1.0" - hashes = [ - "h1:rKYu5ZUbXwrLG1w81k7H3nce/Ys6yAxXhWcbtk36HjY=", - "zh:2bbb3339f0643b5daa07480ef4397bd23a79963cc364cdfbb4e86354cb7725bc", - "zh:3cd456047805bf639fbf2c761b1848880ea703a054f76db51852008b11008626", - "zh:4f251b0eda5bb5e3dc26ea4400dba200018213654b69b4a5f96abee815b4f5ff", - "zh:7011332745ea061e517fe1319bd6c75054a314155cb2c1199a5b01fe1889a7e2", - "zh:738ed82858317ccc246691c8b85995bc125ac3b4143043219bd0437adc56c992", - "zh:7dbe52fac7bb21227acd7529b487511c91f4107db9cc4414f50d04ffc3cab427", - "zh:a3a9251fb15f93e4cfc1789800fc2d7414bbc18944ad4c5c98f466e6477c42bc", - "zh:a543ec1a3a8c20635cf374110bd2f87c07374cf2c50617eee2c669b3ceeeaa9f", - "zh:d9ab41d556a48bd7059f0810cf020500635bfc696c9fc3adab5ea8915c1d886b", - "zh:d9e13427a7d011dbd654e591b0337e6074eef8c3b9bb11b2e39eaaf257044fd7", - "zh:f7605bd1437752114baf601bdf6931debe6dc6bfe3006eb7e9bb9080931dca8a", - ] -} - -provider "registry.terraform.io/hashicorp/tls" { - version = "3.1.0" - constraints = ">= 3.1.0" - hashes = [ - "h1:XTU9f6sGMZHOT8r/+LWCz2BZOPH127FBTPjMMEAAu1U=", - "zh:3d46616b41fea215566f4a957b6d3a1aa43f1f75c26776d72a98bdba79439db6", - "zh:623a203817a6dafa86f1b4141b645159e07ec418c82fe40acd4d2a27543cbaa2", - "zh:668217e78b210a6572e7b0ecb4134a6781cc4d738f4f5d09eb756085b082592e", - "zh:95354df03710691773c8f50a32e31fca25f124b7f3d6078265fdf3c4e1384dca", - "zh:9f97ab190380430d57392303e3f36f4f7835c74ea83276baa98d6b9a997c3698", - "zh:a16f0bab665f8d933e95ca055b9c8d5707f1a0dd8c8ecca6c13091f40dc1e99d", - "zh:be274d5008c24dc0d6540c19e22dbb31ee6bfdd0b2cddd4d97f3cd8a8d657841", - "zh:d5faa9dce0a5fc9d26b2463cea5be35f8586ab75030e7fa4d4920cd73ee26989", - "zh:e9b672210b7fb410780e7b429975adcc76dd557738ecc7c890ea18942eb321a5", - "zh:eb1f8368573d2370605d6dbf60f9aaa5b64e55741d96b5fb026dbfe91de67c0d", - "zh:fc1e12b713837b85daf6c3bb703d7795eaf1c5177aebae1afcf811dd7009f4b0", - ] -} diff --git a/examples/adb-squid-proxy/main/artifacts/proxy_setup.scala b/examples/adb-squid-proxy/main/artifacts/proxy_setup.scala deleted file mode 100644 index 9fd4bfbf..00000000 --- a/examples/adb-squid-proxy/main/artifacts/proxy_setup.scala +++ /dev/null @@ -1,48 +0,0 @@ -// Databricks notebook source -val proxy = "http://10.178.0.4:3128" // set this to your actual proxy -val proxy_host = "10.178.0.4" -val proxy_port = "3128" -val no_proxy = "127.0.0.1,.local,169.254.169.254,s3.amazonaws.com,s3.us-east-1.amazonaws.com" // make sure to update no proxy as needed (e.g. for S3 region or any other internal domains) -val java_no_proxy = "localhost|127.*|[::1]|169.254.169.254|s3.amazonaws.com|*.s3.amazonaws.com|s3.us-east-1.amazonaws.com|*.s3.us-east-1.amazonaws.com|10.179.0.0/20" // replace 10.* with cluster IP range!!!!!! - -dbutils.fs.put("/databricks/init/setproxy.sh", s"""#!/bin/bash -echo "export http_proxy=$proxy" >> /databricks/spark/conf/spark-env.sh -echo "export https_proxy=$proxy" >> /databricks/spark/conf/spark-env.sh -echo "export no_proxy=$no_proxy" >> /databricks/spark/conf/spark-env.sh -echo "export HTTP_PROXY=$proxy" >> /databricks/spark/conf/spark-env.sh -echo "export HTTPS_PROXY=$proxy" >> /databricks/spark/conf/spark-env.sh -echo "export NO_PROXY=$no_proxy" >> /databricks/spark/conf/spark-env.sh -echo "export _JAVA_OPTIONS=\\"-Dhttps.proxyHost=${proxy_host} -Dhttps.proxyPort=${proxy_port} -Dhttp.proxyHost=${proxy_host} -Dhttp.proxyPort=${proxy_port} -Dhttp.nonProxyHosts=${java_no_proxy}\\"" >> /databricks/spark/conf/spark-env.sh - -echo "http_proxy=$proxy" >> /etc/environment -echo "https_proxy=$proxy" >> /etc/environment -echo "no_proxy=$no_proxy" >> /etc/environment -echo "HTTP_PROXY=$proxy" >> /etc/environment -echo "HTTPS_PROXY=$proxy" >> /etc/environment -echo "NO_PROXY=$no_proxy" >> /etc/environment - -cat >> /etc/R/Renviron << EOF -http_proxy=$proxy -https_proxy=$proxy -no_proxy=$no_proxy -EOF -""", true) - -// COMMAND ---------- - -// MAGIC %sh -// MAGIC telnet 10.178.0.4 3128 - -// COMMAND ---------- - -// MAGIC %sh -// MAGIC wget www.google.com - -// COMMAND ---------- - -// MAGIC %sh -// MAGIC wget www.facebook.com - -// COMMAND ---------- - - diff --git a/examples/adb-squid-proxy/main/locals.tf b/examples/adb-squid-proxy/main/locals.tf deleted file mode 100644 index 922ee6f4..00000000 --- a/examples/adb-squid-proxy/main/locals.tf +++ /dev/null @@ -1,12 +0,0 @@ -locals { - prefix = join("-", [var.workspace_prefix, "${random_string.naming.result}"]) - location = var.rglocation - squidcidr = var.vnetcidr - dbcidr = var.dbvnetcidr - // tags that are propagated down to all resources - tags = { - Environment = "Testing" - Owner = lookup(data.external.me.result, "name") - Epoch = random_string.naming.result - } -} diff --git a/examples/adb-squid-proxy/main/main.tf b/examples/adb-squid-proxy/main/main.tf deleted file mode 100644 index 7951c3d5..00000000 --- a/examples/adb-squid-proxy/main/main.tf +++ /dev/null @@ -1,28 +0,0 @@ -provider "azurerm" { - features {} -} - -provider "random" { -} - -provider "local" { -} - -resource "random_string" "naming" { - special = false - upper = false - length = 6 -} - -data "azurerm_client_config" "current" { -} - -data "external" "me" { - program = ["az", "account", "show", "--query", "user"] -} - -resource "azurerm_resource_group" "this" { - name = "${local.prefix}-rg" - location = local.location - tags = local.tags -} diff --git a/examples/adb-squid-proxy/main/outputs.tf b/examples/adb-squid-proxy/main/outputs.tf deleted file mode 100644 index 57297c08..00000000 --- a/examples/adb-squid-proxy/main/outputs.tf +++ /dev/null @@ -1,24 +0,0 @@ -output "arm_client_id" { - value = data.azurerm_client_config.current.client_id -} - -output "arm_subscription_id" { - value = data.azurerm_client_config.current.subscription_id -} - -output "arm_tenant_id" { - value = data.azurerm_client_config.current.tenant_id -} - -output "azure_region" { - value = local.location -} - -output "resource_group" { - value = azurerm_resource_group.this.name -} - -output "tls_private_key" { - value = tls_private_key.squid_ssh.private_key_pem - sensitive = true -} diff --git a/examples/adb-squid-proxy/main/squid.tf b/examples/adb-squid-proxy/main/squid.tf deleted file mode 100644 index 08b31c15..00000000 --- a/examples/adb-squid-proxy/main/squid.tf +++ /dev/null @@ -1,80 +0,0 @@ -resource "azurerm_network_interface" "squid-nic" { - name = "squid-nic" - location = azurerm_resource_group.this.location - resource_group_name = azurerm_resource_group.this.name - - ip_configuration { - name = "internal" - subnet_id = azurerm_subnet.squid-public-subnet.id - private_ip_address_allocation = "Dynamic" - public_ip_address_id = azurerm_public_ip.squid-nic-pubip.id - } -} - -resource "tls_private_key" "squid_ssh" { - algorithm = "RSA" - rsa_bits = 4096 -} - -resource "local_file" "private_key" { - content = tls_private_key.squid_ssh.private_key_pem - filename = "ssh_private.pem" - file_permission = "0600" -} - -resource "azurerm_public_ip" "squid-nic-pubip" { - name = "squid-nic-pubip" - resource_group_name = azurerm_resource_group.this.name - location = azurerm_resource_group.this.location - allocation_method = "Static" -} - - -# Packer creates the custom image - use this to create VM -data "azurerm_image" "customimage" { - name = var.managed_image_name - resource_group_name = var.managed_image_resource_group_name -} - -resource "azurerm_linux_virtual_machine" "example" { - name = "squid-vm" - resource_group_name = azurerm_resource_group.this.name - location = azurerm_resource_group.this.location - size = "Standard_F2" - admin_username = "azureuser" - - network_interface_ids = [ - azurerm_network_interface.squid-nic.id, - ] - - admin_ssh_key { - username = "azureuser" - public_key = tls_private_key.squid_ssh.public_key_openssh // using generated ssh key - # public_key = file("/home/azureuser/.ssh/authorized_keys") //using existing ssh key - } - - os_disk { - caching = "ReadWrite" - storage_account_type = "Standard_LRS" - } - - # use custom image to build vm - source_image_id = data.azurerm_image.customimage.id -} - -/* -resource "null_resource" "test_null" { - triggers = { - always_run = "${timestamp()}" - } - provisioner "local-exec" { - command = <<-EOT - terraform output -raw tls_private_key > ssh_private.pem - chmod 400 ssh_private.pem - EOT - } - depends_on = [ - tls_private_key.squid_ssh, - ] -} - */ diff --git a/examples/adb-squid-proxy/main/storage.tf b/examples/adb-squid-proxy/main/storage.tf deleted file mode 100644 index 94d04e1f..00000000 --- a/examples/adb-squid-proxy/main/storage.tf +++ /dev/null @@ -1,13 +0,0 @@ -resource "random_id" "storage_account" { - byte_length = 8 -} - -resource "azurerm_storage_account" "testsa" { - name = lower(random_id.storage_account.hex) - resource_group_name = azurerm_resource_group.this.name - is_hns_enabled = true - location = "southeastasia" - account_tier = "Standard" - account_replication_type = "LRS" - tags = local.tags -} diff --git a/examples/adb-squid-proxy/main/terraform.tfvars b/examples/adb-squid-proxy/main/terraform.tfvars deleted file mode 100644 index e69de29b..00000000 diff --git a/examples/adb-squid-proxy/main/variables.tf b/examples/adb-squid-proxy/main/variables.tf deleted file mode 100644 index 610d6149..00000000 --- a/examples/adb-squid-proxy/main/variables.tf +++ /dev/null @@ -1,29 +0,0 @@ -variable "vnetcidr" { - type = string - default = "10.178.0.0/20" -} - -variable "dbvnetcidr" { - type = string - default = "10.179.0.0/20" -} -variable "rglocation" { - type = string - default = "southeastasia" -} - -variable "workspace_prefix" { - type = string - default = "adb" -} - -# the 2 vars below must be the same to packer config -variable "managed_image_name" { - type = string - default = "coldstart-5fhmdn-image" -} - -variable "managed_image_resource_group_name" { - type = string - default = "coldstart-5fhmdn-rg" -} diff --git a/examples/adb-squid-proxy/main/versions.tf b/examples/adb-squid-proxy/main/versions.tf deleted file mode 100644 index 362309fc..00000000 --- a/examples/adb-squid-proxy/main/versions.tf +++ /dev/null @@ -1,19 +0,0 @@ -# versions.tf -terraform { - required_providers { - databricks = { - source = "databricks/databricks" - version = "0.3.10" - } - - azurerm = { - source = "hashicorp/azurerm" - version = "=2.83.0" - } - - tls = { - source = "hashicorp/tls" - version = ">= 3.1" - } - } -} diff --git a/examples/adb-squid-proxy/main/vnet.tf b/examples/adb-squid-proxy/main/vnet.tf deleted file mode 100644 index e586831f..00000000 --- a/examples/adb-squid-proxy/main/vnet.tf +++ /dev/null @@ -1,175 +0,0 @@ -resource "azurerm_virtual_network" "dbvnet" { - name = "${local.prefix}-vnet" - location = azurerm_resource_group.this.location - resource_group_name = azurerm_resource_group.this.name - address_space = [local.dbcidr] - tags = local.tags -} - -resource "azurerm_network_security_group" "dbnsg" { - name = "${local.prefix}-nsg" - location = azurerm_resource_group.this.location - resource_group_name = azurerm_resource_group.this.name - tags = local.tags -} - -resource "azurerm_subnet" "public" { - name = "${local.prefix}-public" - resource_group_name = azurerm_resource_group.this.name - virtual_network_name = azurerm_virtual_network.dbvnet.name - address_prefixes = [cidrsubnet(local.dbcidr, 3, 0)] - - delegation { - name = "databricks" - service_delegation { - name = "Microsoft.Databricks/workspaces" - actions = [ - "Microsoft.Network/virtualNetworks/subnets/join/action", - "Microsoft.Network/virtualNetworks/subnets/prepareNetworkPolicies/action", - "Microsoft.Network/virtualNetworks/subnets/unprepareNetworkPolicies/action"] - } - } -} - -resource "azurerm_subnet_network_security_group_association" "public" { - subnet_id = azurerm_subnet.public.id - network_security_group_id = azurerm_network_security_group.dbnsg.id -} - -resource "azurerm_subnet" "private" { - name = "${local.prefix}-private" - resource_group_name = azurerm_resource_group.this.name - virtual_network_name = azurerm_virtual_network.dbvnet.name - address_prefixes = [cidrsubnet(local.dbcidr, 3, 1)] - - delegation { - name = "databricks" - service_delegation { - name = "Microsoft.Databricks/workspaces" - actions = [ - "Microsoft.Network/virtualNetworks/subnets/join/action", - "Microsoft.Network/virtualNetworks/subnets/prepareNetworkPolicies/action", - "Microsoft.Network/virtualNetworks/subnets/unprepareNetworkPolicies/action"] - } - } -} - -resource "azurerm_subnet_network_security_group_association" "private" { - subnet_id = azurerm_subnet.private.id - network_security_group_id = azurerm_network_security_group.dbnsg.id -} - - -resource "azurerm_virtual_network" "squidvnet" { - name = "${local.prefix}-squid-vnet" - location = azurerm_resource_group.this.location - resource_group_name = azurerm_resource_group.this.name - address_space = [local.squidcidr] - tags = local.tags -} - -resource "azurerm_subnet" "squid-public-subnet" { - name = "${local.prefix}-squid-public" - resource_group_name = azurerm_resource_group.this.name - virtual_network_name = azurerm_virtual_network.squidvnet.name - address_prefixes = [cidrsubnet(local.squidcidr, 3, 0)] -} - - -# peering -resource "azurerm_virtual_network_peering" "squid2db" { - name = "squid2db" - resource_group_name = azurerm_resource_group.this.name - virtual_network_name = azurerm_virtual_network.squidvnet.name - remote_virtual_network_id = azurerm_virtual_network.dbvnet.id -} - -resource "azurerm_virtual_network_peering" "db2squid" { - name = "db2squid" - resource_group_name = azurerm_resource_group.this.name - virtual_network_name = azurerm_virtual_network.dbvnet.name - remote_virtual_network_id = azurerm_virtual_network.squidvnet.id -} - -resource "azurerm_network_security_group" "squidnsg" { - name = "${local.prefix}-nsg" - location = azurerm_resource_group.this.location - resource_group_name = azurerm_resource_group.this.name - tags = local.tags -} - -resource "azurerm_network_security_rule" "ssh" { - name = "ssh_squid" - priority = 300 - direction = "Inbound" - access = "Allow" - protocol = "Tcp" - source_port_range = "*" - destination_port_range = "22" - source_address_prefix = "*" //temporary rule for testing, allow any ip to connect; you can change to your client ip - destination_address_prefix = "*" - resource_group_name = azurerm_resource_group.this.name - network_security_group_name = azurerm_network_security_group.squidnsg.name -} - -resource "azurerm_network_security_rule" "http_squid" { - name = "http_squid" - priority = 301 - direction = "Inbound" - access = "Allow" - protocol = "Tcp" - source_port_range = "*" - destination_port_range = "3128" - source_address_prefix = azurerm_virtual_network.dbvnet.address_space.0 //from db clusters - destination_address_prefix = "*" - resource_group_name = azurerm_resource_group.this.name - network_security_group_name = azurerm_network_security_group.squidnsg.name -} - -resource "azurerm_network_security_rule" "https_squid" { - name = "https_squid" - priority = 302 - direction = "Inbound" - access = "Allow" - protocol = "Tcp" - source_port_range = "*" - destination_port_range = "3130" - source_address_prefix = azurerm_virtual_network.dbvnet.address_space.0 //from db clusters - destination_address_prefix = "*" - resource_group_name = azurerm_resource_group.this.name - network_security_group_name = azurerm_network_security_group.squidnsg.name -} - - -resource "azurerm_network_security_rule" "http_out_squid" { - name = "http_out_squid" - priority = 303 - direction = "Outbound" - access = "Allow" - protocol = "Tcp" - source_port_range = "*" - destination_port_range = "80" - source_address_prefix = "0.0.0.0/0" - destination_address_prefix = "*" - resource_group_name = azurerm_resource_group.this.name - network_security_group_name = azurerm_network_security_group.squidnsg.name -} - -resource "azurerm_network_security_rule" "https_out_squid" { - name = "https_out_squid" - priority = 304 - direction = "Outbound" - access = "Allow" - protocol = "Tcp" - source_port_range = "*" - destination_port_range = "443" - source_address_prefix = "0.0.0.0/0" - destination_address_prefix = "*" - resource_group_name = azurerm_resource_group.this.name - network_security_group_name = azurerm_network_security_group.squidnsg.name -} - -resource "azurerm_network_interface_security_group_association" "nsg_nic_assoc" { - network_interface_id = azurerm_network_interface.squid-nic.id - network_security_group_id = azurerm_network_security_group.squidnsg.id -} diff --git a/examples/adb-squid-proxy/main/workspace.tf b/examples/adb-squid-proxy/main/workspace.tf deleted file mode 100644 index 0006bad7..00000000 --- a/examples/adb-squid-proxy/main/workspace.tf +++ /dev/null @@ -1,43 +0,0 @@ -provider "databricks" { - host = azurerm_databricks_workspace.this.workspace_url -} - -resource "azurerm_databricks_workspace" "this" { - name = "${local.prefix}-workspace" - resource_group_name = azurerm_resource_group.this.name - location = azurerm_resource_group.this.location - sku = "premium" - tags = local.tags - - custom_parameters { - no_public_ip = true - virtual_network_id = azurerm_virtual_network.dbvnet.id - private_subnet_name = azurerm_subnet.private.name - public_subnet_name = azurerm_subnet.public.name - public_subnet_network_security_group_association_id = azurerm_subnet_network_security_group_association.public.id - private_subnet_network_security_group_association_id = azurerm_subnet_network_security_group_association.private.id - } - # We need this, otherwise destroy doesn't cleanup things correctly - depends_on = [ - azurerm_subnet_network_security_group_association.public, - azurerm_subnet_network_security_group_association.private, - azurerm_linux_virtual_machine.example // make sure workspace is after squid ready and configured - ] -} - -# create tf managed notebook for convenience, user just need to run the notebook to create the cluster init script -resource "databricks_notebook" "cluster_setup_notebook" { - source = "${path.module}/artifacts/proxy_setup.scala" - path = "/Shared/setup_proxy" -} - -output "databricks_azure_workspace_resource_id" { - // The ID of the Databricks Workspace in the Azure management plane. - value = azurerm_databricks_workspace.this.id -} - -output "workspace_url" { - // The workspace URL which is of the format 'adb-{workspaceId}.{random}.azuredatabricks.net' - // this is not named as DATABRICKS_HOST, because it affect authentication - value = "https://${azurerm_databricks_workspace.this.workspace_url}/" -} diff --git a/examples/adb-squid-proxy/modules/storage/main.tf b/examples/adb-squid-proxy/modules/storage/main.tf deleted file mode 100644 index cc8725c8..00000000 --- a/examples/adb-squid-proxy/modules/storage/main.tf +++ /dev/null @@ -1,8 +0,0 @@ -resource "azurerm_storage_account" "storage" { - name = var.storagename - resource_group_name = var.resource_group_name - location = var.locationtest - account_tier = "Standard" - account_replication_type = "LRS" - is_hns_enabled = true -} diff --git a/examples/adb-squid-proxy/modules/storage/outputs.tf b/examples/adb-squid-proxy/modules/storage/outputs.tf deleted file mode 100644 index ad5f259c..00000000 --- a/examples/adb-squid-proxy/modules/storage/outputs.tf +++ /dev/null @@ -1,3 +0,0 @@ -output "dns" { - value = "na" -} diff --git a/examples/adb-squid-proxy/modules/storage/variables.tf b/examples/adb-squid-proxy/modules/storage/variables.tf deleted file mode 100644 index b2a5f365..00000000 --- a/examples/adb-squid-proxy/modules/storage/variables.tf +++ /dev/null @@ -1,13 +0,0 @@ -variable "storagename" { - type = string - default = "testname" -} - -variable "resource_group_name" { - description = "Azure resource group name" -} - - -variable "locationtest" { - description = "region of storage account" -} diff --git a/examples/adb-squid-proxy/modules/vm/main.tf b/examples/adb-squid-proxy/modules/vm/main.tf deleted file mode 100644 index 73acc28e..00000000 --- a/examples/adb-squid-proxy/modules/vm/main.tf +++ /dev/null @@ -1,221 +0,0 @@ -terraform { - - required_version = ">=0.12" - - required_providers { - azurerm = { - source = "hashicorp/azurerm" - version = "~>2.0" - } - } -} - -provider "azurerm" { - features {} -} - -resource "azurerm_resource_group" "vmss" { - name = var.resource_group_name - location = var.location - tags = var.tags -} - -resource "random_string" "fqdn" { - length = 6 - special = false - upper = false - number = false -} - -resource "azurerm_virtual_network" "vmss" { - name = "vmss-vnet" - address_space = ["10.0.0.0/16"] - location = var.location - resource_group_name = azurerm_resource_group.vmss.name - tags = var.tags -} - -resource "azurerm_subnet" "vmss" { - name = "vmss-subnet" - resource_group_name = azurerm_resource_group.vmss.name - virtual_network_name = azurerm_virtual_network.vmss.name - address_prefixes = ["10.0.2.0/24"] -} - -resource "azurerm_public_ip" "vmss" { - name = "vmss-public-ip" - location = var.location - resource_group_name = azurerm_resource_group.vmss.name - allocation_method = "Static" - domain_name_label = random_string.fqdn.result - tags = var.tags -} - -resource "azurerm_lb" "vmss" { - name = "vmss-lb" - location = var.location - resource_group_name = azurerm_resource_group.vmss.name - - frontend_ip_configuration { - name = "PublicIPAddress" - public_ip_address_id = azurerm_public_ip.vmss.id - } - - tags = var.tags -} - -resource "azurerm_lb_backend_address_pool" "bpepool" { - loadbalancer_id = azurerm_lb.vmss.id - name = "BackEndAddressPool" -} - -resource "azurerm_lb_probe" "vmss" { - resource_group_name = azurerm_resource_group.vmss.name - loadbalancer_id = azurerm_lb.vmss.id - name = "ssh-running-probe" - port = var.application_port -} - -resource "azurerm_lb_rule" "lbnatrule" { - resource_group_name = azurerm_resource_group.vmss.name - loadbalancer_id = azurerm_lb.vmss.id - name = "http" - protocol = "Tcp" - frontend_port = var.application_port - backend_port = var.application_port - backend_address_pool_id = azurerm_lb_backend_address_pool.bpepool.id - frontend_ip_configuration_name = "PublicIPAddress" - probe_id = azurerm_lb_probe.vmss.id -} - -data "azurerm_resource_group" "image" { - name = var.packer_resource_group_name -} - -data "azurerm_image" "image" { - name = var.packer_image_name - resource_group_name = data.azurerm_resource_group.image.name -} - -resource "azurerm_virtual_machine_scale_set" "vmss" { - name = "vmscaleset" - location = var.location - resource_group_name = azurerm_resource_group.vmss.name - upgrade_policy_mode = "Manual" - - sku { - name = "Standard_DS1_v2" - tier = "Standard" - capacity = 2 - } - - storage_profile_image_reference { - id = data.azurerm_image.image.id - } - - storage_profile_os_disk { - name = "" - caching = "ReadWrite" - create_option = "FromImage" - managed_disk_type = "Standard_LRS" - } - - storage_profile_data_disk { - lun = 0 - caching = "ReadWrite" - create_option = "Empty" - disk_size_gb = 10 - } - - os_profile { - computer_name_prefix = "vmlab" - admin_username = var.admin_user - admin_password = var.admin_password - } - - os_profile_linux_config { - disable_password_authentication = true - - ssh_keys { - path = "/home/azureuser/.ssh/authorized_keys" - key_data = file("~/.ssh/id_rsa.pub") - } - } - - network_profile { - name = "terraformnetworkprofile" - primary = true - - ip_configuration { - name = "IPConfiguration" - subnet_id = azurerm_subnet.vmss.id - load_balancer_backend_address_pool_ids = [azurerm_lb_backend_address_pool.bpepool.id] - primary = true - } - } - - tags = var.tags -} - -resource "azurerm_public_ip" "jumpbox" { - name = "jumpbox-public-ip" - location = var.location - resource_group_name = azurerm_resource_group.vmss.name - allocation_method = "Static" - domain_name_label = "${random_string.fqdn.result}-ssh" - tags = var.tags -} - -resource "azurerm_network_interface" "jumpbox" { - name = "jumpbox-nic" - location = var.location - resource_group_name = azurerm_resource_group.vmss.name - - ip_configuration { - name = "IPConfiguration" - subnet_id = azurerm_subnet.vmss.id - private_ip_address_allocation = "dynamic" - public_ip_address_id = azurerm_public_ip.jumpbox.id - } - - tags = var.tags -} - -resource "azurerm_virtual_machine" "jumpbox" { - name = "jumpbox" - location = var.location - resource_group_name = azurerm_resource_group.vmss.name - network_interface_ids = [azurerm_network_interface.jumpbox.id] - vm_size = "Standard_DS1_v2" - - storage_image_reference { - publisher = "Canonical" - offer = "UbuntuServer" - sku = "16.04-LTS" - version = "latest" - } - - storage_os_disk { - name = "jumpbox-osdisk" - caching = "ReadWrite" - create_option = "FromImage" - managed_disk_type = "Standard_LRS" - } - - os_profile { - computer_name = "jumpbox" - admin_username = var.admin_user - admin_password = var.admin_password - } - - os_profile_linux_config { - disable_password_authentication = true - - ssh_keys { - path = "/home/azureuser/.ssh/authorized_keys" - key_data = file("~/.ssh/id_rsa.pub") - } - } - - tags = var.tags -} \ No newline at end of file diff --git a/examples/adb-squid-proxy/modules/vm/outputs.tf b/examples/adb-squid-proxy/modules/vm/outputs.tf deleted file mode 100644 index e69de29b..00000000 diff --git a/examples/adb-squid-proxy/modules/vm/variables.tf b/examples/adb-squid-proxy/modules/vm/variables.tf deleted file mode 100644 index b12ef2c9..00000000 --- a/examples/adb-squid-proxy/modules/vm/variables.tf +++ /dev/null @@ -1,13 +0,0 @@ -variable "resource_group_name" { - description = "Azure resource group name" -} - -variable "vnetcidr" { - type = string - default = "10.178.0.0/20" -} - -variable "loc" { - type = string - default = "southeastasia" -} \ No newline at end of file diff --git a/examples/adb-squid-proxy/packer/os/ubuntu.pkr.hcl b/examples/adb-squid-proxy/packer/os/ubuntu.pkr.hcl deleted file mode 100644 index 4a89f77a..00000000 --- a/examples/adb-squid-proxy/packer/os/ubuntu.pkr.hcl +++ /dev/null @@ -1,28 +0,0 @@ -source "azure-arm" "my-example" { - subscription_id = "${var.subscription_id}" - - # Managed Image, saving generated image to rg - managed_image_resource_group_name = var.managed_img_rg_name - managed_image_name = var.managed_img_name - - os_type = "Linux" - image_publisher = "Canonical" - image_offer = "UbuntuServer" - image_sku = "18.04-LTS" - - location = "Southeast Asia" - vm_size = "Standard_DS2_v2" - - azure_tags = { - dept = "hwangtest" - } -} - -build { - sources = ["sources.azure-arm.my-example"] - - provisioner "shell" { - execute_command = "chmod +x {{ .Path }}; {{ .Vars }} sudo {{ .Path }}" - script = "../scripts/setup.sh" - } -} \ No newline at end of file diff --git a/examples/adb-squid-proxy/packer/os/variables.auto.pkrvars.hcl b/examples/adb-squid-proxy/packer/os/variables.auto.pkrvars.hcl deleted file mode 100755 index 2a5d1a03..00000000 --- a/examples/adb-squid-proxy/packer/os/variables.auto.pkrvars.hcl +++ /dev/null @@ -1,3 +0,0 @@ -managed_img_name="coldstart-5fhmdn-image" -managed_img_rg_name="coldstart-5fhmdn-rg" -subscription_id="3f2e4d32-8e8d-46d6-82bc-5bb8d962328b" diff --git a/examples/adb-squid-proxy/packer/os/variables.pkr.hcl b/examples/adb-squid-proxy/packer/os/variables.pkr.hcl deleted file mode 100644 index a7dcc911..00000000 --- a/examples/adb-squid-proxy/packer/os/variables.pkr.hcl +++ /dev/null @@ -1,14 +0,0 @@ -# using interaction az login authentication, only need to specify subscription id -variable "subscription_id" { - type = string -} - -variable "managed_img_rg_name" { - type = string - default = "unique_rg_name" -} - -variable "managed_img_name" { - type = string - default = "unique_managed_img_name" -} \ No newline at end of file diff --git a/examples/adb-squid-proxy/packer/scripts/setup.sh b/examples/adb-squid-proxy/packer/scripts/setup.sh deleted file mode 100644 index 9525caa0..00000000 --- a/examples/adb-squid-proxy/packer/scripts/setup.sh +++ /dev/null @@ -1,65 +0,0 @@ -#!/bin/bash - -if [ $(whoami) != root ]; then - echo "ERROR: You need to run the script as user root or add sudo before command." - exit 1 -fi - -# install squid -apt-get -y update -apt-get -y upgrade - -# purge squid if it was installed -apt-get -y purge squid - -# install squid, this will create /etc/squid/squid.conf -apt-get -y install squid - -# create squid.conf.bk backup, if not already created -FILE=/etc/squid/squid.conf.bk - -if [ -f "$FILE" ]; then - echo "squid.conf backup exists, no change applied" -else - cp /etc/squid/squid.conf /etc/squid/squid.conf.bk - echo "backup created: squid.conf.bk" -fi - -cat >/etc/squid/proxy-allow-list.acl </etc/squid/proxy-block-list.acl <squid.conf <>squid.conf -# copy and overwrite original squid conf -cp -f squid.conf /etc/squid/ - -chmod 644 /etc/squid/squid.conf -chmod 644 /var/log/squid/cache.log - -iptables -t nat -A PREROUTING -p tcp --dport 443 -j REDIRECT --to-port 3130 -# start squid -systemctl enable squid -systemctl restart squid diff --git a/examples/adb-squid-proxy/packer/tf_coldstart/.terraform.lock.hcl b/examples/adb-squid-proxy/packer/tf_coldstart/.terraform.lock.hcl deleted file mode 100644 index f298fcdb..00000000 --- a/examples/adb-squid-proxy/packer/tf_coldstart/.terraform.lock.hcl +++ /dev/null @@ -1,92 +0,0 @@ -# This file is maintained automatically by "terraform init". -# Manual edits may be lost in future updates. - -provider "registry.terraform.io/hashicorp/azurerm" { - version = "2.90.0" - hashes = [ - "h1:j2SkWgabdECJo8vGy5Q/cTLF3yVaWR1kRaO/Nopb2VY=", - "zh:11e484585bc324e56ece3d75bd1d8371fc1d56501fb7f15542be0185eaf2c05e", - "zh:189f11bba0665a45405d12b66ac789de87582d23863f0599016bcc28f399a4e3", - "zh:1e4103ab153f16b614c98898d4f4d46c953687a35a0381e05b4a2958b27165b8", - "zh:1f8bb4631f8206736e1ff560b59453d9cc6d7dd0598072feaba10b6833cda1c4", - "zh:424acd64ee42cf3cc78919dc32b07a8dc4e7e9cc103eca37cce2f256a48f5e6d", - "zh:4ce7c5469e1fbed3066bb05c3c4ad8a194b76a4d01e2bbd7093537aa3eafeb0a", - "zh:6b4b592f1f3684a4df9fad146b4cbe125fffbadc2239f7011c0390199094676f", - "zh:8525bb868cd3e117fb1278e54c899a22ab6533c134779a64c224e651983fc253", - "zh:9873ee69ee94059af70ee544e0b27b68064dcc21e4d2912aa7186f46acca47c1", - "zh:af305aa28b532c78a1c51cf707a02bd1993ff1b8e7b5d656db8043c3eb484eab", - "zh:f96ecf6c3f208ad50af0959042e5fe1730b2bf26582430a61aac5b2a970917c7", - ] -} - -provider "registry.terraform.io/hashicorp/external" { - version = "2.1.1" - hashes = [ - "h1:5YnKC77SbQ7ayCcjpWkDF8Hv6wP3ZUFNDcGKsHol5yI=", - "zh:2726dcef652c01c718b6400ac22e598e7d296baa91ab5ea6d85bd94d1db229d9", - "zh:3923daa6deb847f88f0b1bf1481066f9d61f49557108945be7e2960cb1b61d0d", - "zh:3d8e894d60b362cb7fbdf22889d69f3c96a3c98ed2fec492dca28f52e07254e7", - "zh:43f1501519e2ee0dfa35c827cd2b24b4940a11d0348e77b20ad2d77bb22d9b11", - "zh:52e75f6c880ca5aa1175f288396101b625eb227ca8b89f31ff9be30e4280cfba", - "zh:6b0f7951cc9f4a7b06f399505d590fed637d07eacbae7781a7fab81dc6e26f99", - "zh:8d7103552cadda26c01230722184c0a7489515cfa8d1fabfc1aa95aecb5490f6", - "zh:962df74629cfefe8119b2dc23abc34bebd4bd0da506062732edca8dc4427c130", - "zh:b66e41a41140f5e1b960345fc91670dea12a9e1d23ea9dc2ad456cf9224e199d", - "zh:c4df3d6ea9dc6d8803f00400f4e075a43fa6ae9dac160b46c9de4b8714428f3d", - "zh:cab80dca018c63619fa7a9f71be329bb43f174a2b07f65ba608d59146a487a35", - ] -} - -provider "registry.terraform.io/hashicorp/local" { - version = "2.1.0" - hashes = [ - "h1:KfieWtVyGWwplSoLIB5usKAUnrIkDQBkWaR5TI+4WYg=", - "zh:0f1ec65101fa35050978d483d6e8916664b7556800348456ff3d09454ac1eae2", - "zh:36e42ac19f5d68467aacf07e6adcf83c7486f2e5b5f4339e9671f68525fc87ab", - "zh:6db9db2a1819e77b1642ec3b5e95042b202aee8151a0256d289f2e141bf3ceb3", - "zh:719dfd97bb9ddce99f7d741260b8ece2682b363735c764cac83303f02386075a", - "zh:7598bb86e0378fd97eaa04638c1a4c75f960f62f69d3662e6d80ffa5a89847fe", - "zh:ad0a188b52517fec9eca393f1e2c9daea362b33ae2eb38a857b6b09949a727c1", - "zh:c46846c8df66a13fee6eff7dc5d528a7f868ae0dcf92d79deaac73cc297ed20c", - "zh:dc1a20a2eec12095d04bf6da5321f535351a594a636912361db20eb2a707ccc4", - "zh:e57ab4771a9d999401f6badd8b018558357d3cbdf3d33cc0c4f83e818ca8e94b", - "zh:ebdcde208072b4b0f8d305ebf2bfdc62c926e0717599dcf8ec2fd8c5845031c3", - "zh:ef34c52b68933bedd0868a13ccfd59ff1c820f299760b3c02e008dc95e2ece91", - ] -} - -provider "registry.terraform.io/hashicorp/null" { - version = "3.1.0" - hashes = [ - "h1:xhbHC6in3nQryvTQBWKxebi3inG5OCgHgc4fRxL0ymc=", - "zh:02a1675fd8de126a00460942aaae242e65ca3380b5bb192e8773ef3da9073fd2", - "zh:53e30545ff8926a8e30ad30648991ca8b93b6fa496272cd23b26763c8ee84515", - "zh:5f9200bf708913621d0f6514179d89700e9aa3097c77dac730e8ba6e5901d521", - "zh:9ebf4d9704faba06b3ec7242c773c0fbfe12d62db7d00356d4f55385fc69bfb2", - "zh:a6576c81adc70326e4e1c999c04ad9ca37113a6e925aefab4765e5a5198efa7e", - "zh:a8a42d13346347aff6c63a37cda9b2c6aa5cc384a55b2fe6d6adfa390e609c53", - "zh:c797744d08a5307d50210e0454f91ca4d1c7621c68740441cf4579390452321d", - "zh:cecb6a304046df34c11229f20a80b24b1603960b794d68361a67c5efe58e62b8", - "zh:e1371aa1e502000d9974cfaff5be4cfa02f47b17400005a16f14d2ef30dc2a70", - "zh:fc39cc1fe71234a0b0369d5c5c7f876c71b956d23d7d6f518289737a001ba69b", - "zh:fea4227271ebf7d9e2b61b89ce2328c7262acd9fd190e1fd6d15a591abfa848e", - ] -} - -provider "registry.terraform.io/hashicorp/random" { - version = "3.1.0" - hashes = [ - "h1:rKYu5ZUbXwrLG1w81k7H3nce/Ys6yAxXhWcbtk36HjY=", - "zh:2bbb3339f0643b5daa07480ef4397bd23a79963cc364cdfbb4e86354cb7725bc", - "zh:3cd456047805bf639fbf2c761b1848880ea703a054f76db51852008b11008626", - "zh:4f251b0eda5bb5e3dc26ea4400dba200018213654b69b4a5f96abee815b4f5ff", - "zh:7011332745ea061e517fe1319bd6c75054a314155cb2c1199a5b01fe1889a7e2", - "zh:738ed82858317ccc246691c8b85995bc125ac3b4143043219bd0437adc56c992", - "zh:7dbe52fac7bb21227acd7529b487511c91f4107db9cc4414f50d04ffc3cab427", - "zh:a3a9251fb15f93e4cfc1789800fc2d7414bbc18944ad4c5c98f466e6477c42bc", - "zh:a543ec1a3a8c20635cf374110bd2f87c07374cf2c50617eee2c669b3ceeeaa9f", - "zh:d9ab41d556a48bd7059f0810cf020500635bfc696c9fc3adab5ea8915c1d886b", - "zh:d9e13427a7d011dbd654e591b0337e6074eef8c3b9bb11b2e39eaaf257044fd7", - "zh:f7605bd1437752114baf601bdf6931debe6dc6bfe3006eb7e9bb9080931dca8a", - ] -} diff --git a/examples/adb-squid-proxy/packer/tf_coldstart/main.tf b/examples/adb-squid-proxy/packer/tf_coldstart/main.tf deleted file mode 100644 index d3b3a912..00000000 --- a/examples/adb-squid-proxy/packer/tf_coldstart/main.tf +++ /dev/null @@ -1,43 +0,0 @@ -provider "azurerm" { - features {} -} - -provider "random" { -} - -provider "null" { -} - -provider "local" { -} - -resource "random_string" "naming" { - special = false - upper = false - length = 6 -} - -data "azurerm_client_config" "current" { -} - -data "external" "me" { - program = ["az", "account", "show", "--query", "user"] -} - -resource "azurerm_resource_group" "coldstart_image_rg" { - name = "${local.prefix}-rg" - location = local.location - tags = local.tags -} - -resource "local_file" "packer_config" { - content = <<-EOT - managed_img_name="${local.prefix}-image" - managed_img_rg_name="${azurerm_resource_group.coldstart_image_rg.name}" - subscription_id="${data.azurerm_client_config.current.subscription_id}" - EOT - filename = "../os/variables.auto.pkrvars.hcl" - depends_on = [ - azurerm_resource_group.coldstart_image_rg, - ] -} diff --git a/examples/adb-squid-proxy/packer/tf_coldstart/outputs.tf b/examples/adb-squid-proxy/packer/tf_coldstart/outputs.tf deleted file mode 100644 index 709d9567..00000000 --- a/examples/adb-squid-proxy/packer/tf_coldstart/outputs.tf +++ /dev/null @@ -1,19 +0,0 @@ -output "arm_client_id" { - value = data.azurerm_client_config.current.client_id -} - -output "arm_subscription_id" { - value = data.azurerm_client_config.current.subscription_id -} - -output "arm_tenant_id" { - value = data.azurerm_client_config.current.tenant_id -} - -output "azure_region" { - value = local.location -} - -output "resource_group" { - value = azurerm_resource_group.coldstart_image_rg.name -} diff --git a/examples/adb-squid-proxy/packer/tf_coldstart/variables.tf b/examples/adb-squid-proxy/packer/tf_coldstart/variables.tf deleted file mode 100644 index c2e61a69..00000000 --- a/examples/adb-squid-proxy/packer/tf_coldstart/variables.tf +++ /dev/null @@ -1,31 +0,0 @@ -variable "rglocation" { - type = string - default = "southeastasia" -} - -variable "workspace_prefix" { - type = string - default = "coldstart" -} - -# the 2 vars below must be the same to packer config -variable "managed_image_name_prefix" { - type = string - default = "squid_img" -} - -variable "managed_image_resource_group_name_prefix" { - type = string - default = "squid" -} - -locals { - prefix = join("-", [var.workspace_prefix, "${random_string.naming.result}"]) - location = var.rglocation - // tags that are propagated down to all resources - tags = { - Environment = "Testing" - Owner = lookup(data.external.me.result, "name") - Epoch = random_string.naming.result - } -} diff --git a/examples/adb-uc/Makefile b/examples/adb-uc/Makefile deleted file mode 100644 index 653039d8..00000000 --- a/examples/adb-uc/Makefile +++ /dev/null @@ -1,7 +0,0 @@ -.PHONY: docs test_docs - -docs: - terraform-docs -c ../../.terraform-docs.yml . - -test_docs: - terraform-docs -c ../../.terraform-docs.yml --output-check . diff --git a/examples/adb-uc/README.md b/examples/adb-uc/README.md deleted file mode 100644 index a5d9bd00..00000000 --- a/examples/adb-uc/README.md +++ /dev/null @@ -1,38 +0,0 @@ -## Azure Databricks Unity Catalog Examples - -This folder shows how to automate ADB UC resources deployment using Service Principal authentication. In stage 1 we create an AAD Global Admin SPN, in stage 2 we make this SPN the first Databricks account admin, and in stage 3 we use the long lasting SPN to deploy UC resources. - -The reason of having a 2-fold structure is that we can just have the first AAD Global Admin SPN temporarily deployed, at the moment of you have created the second Databricks account admin, you can safely remove the Global Admin role. Thus to minimize the priviledge footprint. - -Go through this tutorial by sequence of stage 1 - 2 - 3 and modify the logic according to your existing scripts. - -The flowchart below shows how to get the first Databricks account admin, deploy UC resources using SPN, while minimizing the exposure of AAD Global Admin role: - -![alt text](https://raw.githubusercontent.com/databricks/terraform-databricks-examples/main/examples/adb-uc/images/uc-adb-spn.png?raw=true) - - - -## Requirements - -No requirements. - -## Providers - -No providers. - -## Modules - -No modules. - -## Resources - -No resources. - -## Inputs - -No inputs. - -## Outputs - -No outputs. - diff --git a/examples/adb-uc/images/spn-tf.png b/examples/adb-uc/images/spn-tf.png deleted file mode 100644 index 66f4a832..00000000 Binary files a/examples/adb-uc/images/spn-tf.png and /dev/null differ diff --git a/examples/adb-uc/images/uc-adb-spn.png b/examples/adb-uc/images/uc-adb-spn.png deleted file mode 100644 index 662d3694..00000000 Binary files a/examples/adb-uc/images/uc-adb-spn.png and /dev/null differ diff --git a/examples/adb-uc/stage_1_spawn_global_admin_spn/.terraform.lock.hcl b/examples/adb-uc/stage_1_spawn_global_admin_spn/.terraform.lock.hcl deleted file mode 100644 index 03dabeab..00000000 --- a/examples/adb-uc/stage_1_spawn_global_admin_spn/.terraform.lock.hcl +++ /dev/null @@ -1,41 +0,0 @@ -# This file is maintained automatically by "terraform init". -# Manual edits may be lost in future updates. - -provider "registry.terraform.io/hashicorp/azuread" { - version = "2.30.0" - constraints = "2.30.0" - hashes = [ - "h1:WnSPiREAFwnBUKREokMdHQ8Cjs47MzvS9pG8VS1ktec=", - "zh:1c3e89cf19118fc07d7b04257251fc9897e722c16e0a0df7b07fcd261f8c12e7", - "zh:2e62c193030e04ebb10cc0526119cf69824bf2d7e4ea5a2f45bd5d5fb7221d36", - "zh:2f3c7a35257332d68b778cefc5201a5f044e4914dd03794a4da662ddfe756483", - "zh:35d0d3a1b58fdb8b8c4462d6b7e7016042da43ea9cc734ce897f52a73407d9b0", - "zh:47ede0cd0206ec953d40bf4a80aa6e59af64e26cbbd877614ac424533dbb693b", - "zh:48c190307d4d42ea67c9b8cc544025024753f46cef6ea64db84735e7055a72da", - "zh:6fff9b2c6a962252a70a15b400147789ab369b35a781e9d21cce3804b04d29af", - "zh:7646980cf3438bff29c91ffedb74458febbb00a996638751fbd204ab1c628c9b", - "zh:77aa2fa7ca6d5446afa71d4ff83cb87b70a2f3b72110fc442c339e8e710b2928", - "zh:e20b2b2c37175b89dd0db058a096544d448032e28e3b56e2db368343533a9684", - "zh:eab175b1dfe9865ad9404dccb6d5542899f8c435095aa7c679314b811c717ce7", - "zh:efc862bd78c55d2ff089729e2a34c1831ab4b0644fc11b36ee4ebed00a4797ba", - ] -} - -provider "registry.terraform.io/hashicorp/azurerm" { - version = "3.32.0" - hashes = [ - "h1:/ziRWmSDLF+zqfYDBiO4lCBJ20gIwPjDErIR8whhYSg=", - "zh:3ee1992144e6bf9801c44df0ed1e10413fa83ad605e3ce751cb342dd46904c41", - "zh:4f083079909f929b76c0cb2819b107803ecbf26c761832aaa1e7b4a667025665", - "zh:52ad565c4bd37c2b4f0bba78639277ef98caaebf2c4c00c67a2659561079c21c", - "zh:5ecf7a8470e066cc27b837a8fbc9a02629bb85797007475539983496bcccbc53", - "zh:6348154495cd838862b27a9bc0a2714e8f76cd2919df55fce8da0f64ce240ab1", - "zh:8325c4f5f65e30bba2537c7df702c80ae29999fba6194c258b075b3cbde5a709", - "zh:8b4d33aa76474a9fac9a6859e759c03ffeadb787abf7a9ba5a05b4ca3914c008", - "zh:95ccd31450909582ebcf01548ee20df658049783530d79adcb53a601bb163597", - "zh:c104f977b96c6402276c82a8d9d6fee14381511e832e9c3593e589e5ee4e708c", - "zh:e12372a41a981c24323a467f6c54b0a17e26c85a0fb569e4b733b2a76c9ba6b6", - "zh:e80bf9b674914f91ed00984758288b7266ba5772fad728cd1b4cd2f776851ed8", - "zh:f569b65999264a9416862bca5cd2a6177d94ccb0424f3a4ef424428912b9cb3c", - ] -} diff --git a/examples/adb-uc/stage_1_spawn_global_admin_spn/outputs.tf b/examples/adb-uc/stage_1_spawn_global_admin_spn/outputs.tf deleted file mode 100644 index e551dc17..00000000 --- a/examples/adb-uc/stage_1_spawn_global_admin_spn/outputs.tf +++ /dev/null @@ -1,3 +0,0 @@ -output "aad_global_admin_spn_object_id" { - value = azuread_service_principal.example.object_id -} diff --git a/examples/adb-uc/stage_1_spawn_global_admin_spn/providers.tf b/examples/adb-uc/stage_1_spawn_global_admin_spn/providers.tf deleted file mode 100644 index 52d2b08a..00000000 --- a/examples/adb-uc/stage_1_spawn_global_admin_spn/providers.tf +++ /dev/null @@ -1,18 +0,0 @@ -terraform { - required_providers { - azurerm = { - source = "hashicorp/azurerm" - } - azuread = { - source = "hashicorp/azuread" - version = "~> 2.30.0" - } - } -} - -provider "azuread" { -} - -provider "azurerm" { - features {} -} diff --git a/examples/adb-uc/stage_1_spawn_global_admin_spn/readme.md b/examples/adb-uc/stage_1_spawn_global_admin_spn/readme.md deleted file mode 100644 index d9897d09..00000000 --- a/examples/adb-uc/stage_1_spawn_global_admin_spn/readme.md +++ /dev/null @@ -1,25 +0,0 @@ -### Stage 1 Generate a Global Admin SPN - -In Stage 1, we use `az login` authentication with a **user principal** to `azuread` and `azurerm` providers (you can also log in as a service principal) to create an AAD Global Admin SPN. The user principal / servicipal you use to authenticate with Stage 1 must be able to grant AAD Global Admin role to other users/SPNs. - -Stage 1 deployment steps: - -On your local machine, run the following commands to log in with your user principal interactively: -```bash -az login -``` - -or az login with an existing SPN that's able to grant other SPNs AAD Global Admin role, you can log in as a service principal like below: -```bash -az login --service-principal -u -p --tenant -``` - -In this stage 1 folder, run -```bash -terraform init & terraform apply -``` -This deploys a new SPN and grant it as AAD Global Admin. You should see under AAD - Roles - Global Admin, with the new assignment to SPN like below: - -![alt text](https://raw.githubusercontent.com/databricks/terraform-databricks-examples/main/examples/adb-uc/images/spn-tf.png?raw=true) - -This completes Stage 1. \ No newline at end of file diff --git a/examples/adb-uc/stage_1_spawn_global_admin_spn/spn.tf b/examples/adb-uc/stage_1_spawn_global_admin_spn/spn.tf deleted file mode 100644 index 9ae2cb72..00000000 --- a/examples/adb-uc/stage_1_spawn_global_admin_spn/spn.tf +++ /dev/null @@ -1,17 +0,0 @@ -data "azuread_client_config" "current" {} - -resource "azuread_application" "example" { - display_name = "spn-tf-1" - owners = [data.azuread_client_config.current.object_id] -} - -resource "azuread_service_principal" "example" { - application_id = azuread_application.example.application_id - app_role_assignment_required = false - owners = [data.azuread_client_config.current.object_id] -} - -resource "azuread_directory_role_assignment" "global_administrator" { - role_id = "62e90394-69f5-4237-9190-012177145e10" # Global Administrator templateID - principal_object_id = azuread_service_principal.example.object_id -} diff --git a/examples/adb-uc/stage_2_getting_first_second_account_admin/.terraform.lock.hcl b/examples/adb-uc/stage_2_getting_first_second_account_admin/.terraform.lock.hcl deleted file mode 100644 index aa4bbd6a..00000000 --- a/examples/adb-uc/stage_2_getting_first_second_account_admin/.terraform.lock.hcl +++ /dev/null @@ -1,58 +0,0 @@ -# This file is maintained automatically by "terraform init". -# Manual edits may be lost in future updates. - -provider "registry.terraform.io/databricks/databricks" { - version = "1.6.5" - hashes = [ - "h1:QS9NBwTx/hE+NKvZyPeheIBlL17y+gOXbtFqkyWkaC0=", - "zh:28ecc2d3f822c8453871af975b91bd6fbfafeeb7d2f6a6e1ffb88c5bad3b8d65", - "zh:5f2377443f5e7934fab7ae18edbc0eedf422c12b6f8be99408b73e1104308694", - "zh:62f6fc8a4e6ecf5412f2923b3349a0c197f27d5d475210f77a1306824e6641a3", - "zh:6db69d9ea3b170bb457132426d60aa9b2cc01a098137300a969339c0ce11c792", - "zh:7e101671d31e45485a7cb0b52dc537e6648dfb9290de6662f4a0e5c22393a53d", - "zh:856a0d4d74af7765894c3f3ece6185f97a5a2ee290946ad32ddfbb3011b2f7cd", - "zh:86b8ccde61c664a0d092152fccc0a95c8dcbb5b6b03b544bcd8c4db084067edc", - "zh:910c7241576b93e3b36ccaa434bbbe155bb9f409453aa3cce8855f94ff490dd7", - "zh:d16ab0aee6e9df81ef0cbddd830654ace7a705d6f0239a895e9fec946a38e37a", - "zh:dddc2189b481eeef0c1969a4be09a586d46f729b1e8bce7d2d0320a530cd4259", - ] -} - -provider "registry.terraform.io/hashicorp/azuread" { - version = "2.30.0" - constraints = "~> 2.30.0" - hashes = [ - "h1:WnSPiREAFwnBUKREokMdHQ8Cjs47MzvS9pG8VS1ktec=", - "zh:1c3e89cf19118fc07d7b04257251fc9897e722c16e0a0df7b07fcd261f8c12e7", - "zh:2e62c193030e04ebb10cc0526119cf69824bf2d7e4ea5a2f45bd5d5fb7221d36", - "zh:2f3c7a35257332d68b778cefc5201a5f044e4914dd03794a4da662ddfe756483", - "zh:35d0d3a1b58fdb8b8c4462d6b7e7016042da43ea9cc734ce897f52a73407d9b0", - "zh:47ede0cd0206ec953d40bf4a80aa6e59af64e26cbbd877614ac424533dbb693b", - "zh:48c190307d4d42ea67c9b8cc544025024753f46cef6ea64db84735e7055a72da", - "zh:6fff9b2c6a962252a70a15b400147789ab369b35a781e9d21cce3804b04d29af", - "zh:7646980cf3438bff29c91ffedb74458febbb00a996638751fbd204ab1c628c9b", - "zh:77aa2fa7ca6d5446afa71d4ff83cb87b70a2f3b72110fc442c339e8e710b2928", - "zh:e20b2b2c37175b89dd0db058a096544d448032e28e3b56e2db368343533a9684", - "zh:eab175b1dfe9865ad9404dccb6d5542899f8c435095aa7c679314b811c717ce7", - "zh:efc862bd78c55d2ff089729e2a34c1831ab4b0644fc11b36ee4ebed00a4797ba", - ] -} - -provider "registry.terraform.io/hashicorp/azurerm" { - version = "3.33.0" - hashes = [ - "h1:UEAlSpnLUdpXP3hIyZRhTG3iYb++FJejLRT2AddA8SU=", - "zh:136d9c642746d8d84e62ecd8ab0c7dc015eac504c1f068e06fad438ae222d934", - "zh:266e64b8e32a94ddcc20954ebad1d8ff3921d318addf576e981b1390e5d5ba79", - "zh:3bd84a1e5b3bbe34a5870f271d6a5bf9b35a4c924db32b450a1fb53bc910c37a", - "zh:3c6604041472bb4691b502877cf9d886ed9f973fbadf11389ec9499fdc66045e", - "zh:680c00a73c8054c36a58115a44d02d1ebb675c2ad3afaaab2d74a01f978f16ce", - "zh:6dab47ef64f90e43b75ed240a974c4119f5268be4433f3c1c3e97559e7ef2f38", - "zh:9f73f19fdc340c443693dc03f1a145c6bd0ee5fd425eab7473d06abbe39b99d7", - "zh:9ff008b6737e880f191b4be6dfcef95ff019969dd787c44a58c2d7d6aaf6623b", - "zh:be297f1515e9ac63886e3e092a0bcd10aa8aa2b69c2b0995ce4e069176b07a95", - "zh:f569b65999264a9416862bca5cd2a6177d94ccb0424f3a4ef424428912b9cb3c", - "zh:fb29a566e7698cfae477f3efa3bba38526ec8343355763178c6e9c96e51399f3", - "zh:fbc3b625733ce5f0970fa8d9743f6db51064c168d6be5fc7a5e3d1a54af28bb7", - ] -} diff --git a/examples/adb-uc/stage_2_getting_first_second_account_admin/main.tf b/examples/adb-uc/stage_2_getting_first_second_account_admin/main.tf deleted file mode 100644 index 4b9b729c..00000000 --- a/examples/adb-uc/stage_2_getting_first_second_account_admin/main.tf +++ /dev/null @@ -1,15 +0,0 @@ -variable "long_lasting_spn_id" { -} - -resource "databricks_group" "this" { - provider = databricks.azure_account - display_name = "test_tf_sp" -} - -resource "databricks_service_principal_role" "sp_account_admin" { - service_principal_id = var.long_lasting_spn_id // this is the long living SPN client id - role = "account_admin" - depends_on = [ - databricks_group.this // force this to run after the group is created - ] -} diff --git a/examples/adb-uc/stage_2_getting_first_second_account_admin/providers.tf b/examples/adb-uc/stage_2_getting_first_second_account_admin/providers.tf deleted file mode 100644 index a47fded1..00000000 --- a/examples/adb-uc/stage_2_getting_first_second_account_admin/providers.tf +++ /dev/null @@ -1,29 +0,0 @@ -terraform { - required_providers { - azurerm = { - source = "hashicorp/azurerm" - } - databricks = { - source = "databricks/databricks" - } - azuread = { - source = "hashicorp/azuread" - version = "~> 2.30.0" - } - } -} - -provider "azurerm" { - features {} -} - -provider "azuread" { - # Configuration options -} - -provider "databricks" { // TF account level endpoint - alias = "azure_account" - host = "https://accounts.azuredatabricks.net" - account_id = "f3b0d159-720f-4d2e-bdc4-18104f13f419" // Databricks will provide - auth_type = "azure-cli" // az login with SPN -} diff --git a/examples/adb-uc/stage_2_getting_first_second_account_admin/readme.md b/examples/adb-uc/stage_2_getting_first_second_account_admin/readme.md deleted file mode 100644 index b0029eac..00000000 --- a/examples/adb-uc/stage_2_getting_first_second_account_admin/readme.md +++ /dev/null @@ -1,32 +0,0 @@ -### Stage 2 Make AAD Global Admin SPN to be the first Databricks account admin - -In Stage 2, we az login using an AAD Global Admin SPN, and use this identity to hit the Databricks Provider Account Level endpoint to make it as the first account admin. - -Once this SPN becomes the first Databricks Account Admin, then this SPN can make other SPNs as account admins, then you can remove the AAD Global Admin role from this initial SPN. - -This sample script requires a variable `long_lasting_spn_id`, you should have prepared this SPN and supply its id to this variable. - -Stage 3 deployment steps: - -az login as AAD Global Admin SPN: -`az login --service-principal -u -p --tenant ` - -Supply value to `long_lasting_spn_id` variable. - -Change `account_id` attribute inside the databricks provider: -``` -provider "databricks" { // account level endpoint - alias = "azure_account" - host = "https://accounts.azuredatabricks.net" - account_id = "f3b0d159-720f-4d2e-bdc4-18104f13f419" // Databricks will provide - auth_type = "azure-cli" // az login with SPN -} -``` - -In this stage 2 folder, run: - -```bash -terraform init & terraform apply -``` - -This completes Stage 2, now you should have a long lasting SPN with Databricks Account Admin role and it can make other identities as account admins. You can now remove the first SPN's AAD Global Admin role. \ No newline at end of file diff --git a/examples/adb-uc/stage_3_spn_deploys_uc/.terraform.lock.hcl b/examples/adb-uc/stage_3_spn_deploys_uc/.terraform.lock.hcl deleted file mode 100644 index c4ebea23..00000000 --- a/examples/adb-uc/stage_3_spn_deploys_uc/.terraform.lock.hcl +++ /dev/null @@ -1,58 +0,0 @@ -# This file is maintained automatically by "terraform init". -# Manual edits may be lost in future updates. - -provider "registry.terraform.io/databricks/databricks" { - version = "1.6.5" - hashes = [ - "h1:QS9NBwTx/hE+NKvZyPeheIBlL17y+gOXbtFqkyWkaC0=", - "zh:28ecc2d3f822c8453871af975b91bd6fbfafeeb7d2f6a6e1ffb88c5bad3b8d65", - "zh:5f2377443f5e7934fab7ae18edbc0eedf422c12b6f8be99408b73e1104308694", - "zh:62f6fc8a4e6ecf5412f2923b3349a0c197f27d5d475210f77a1306824e6641a3", - "zh:6db69d9ea3b170bb457132426d60aa9b2cc01a098137300a969339c0ce11c792", - "zh:7e101671d31e45485a7cb0b52dc537e6648dfb9290de6662f4a0e5c22393a53d", - "zh:856a0d4d74af7765894c3f3ece6185f97a5a2ee290946ad32ddfbb3011b2f7cd", - "zh:86b8ccde61c664a0d092152fccc0a95c8dcbb5b6b03b544bcd8c4db084067edc", - "zh:910c7241576b93e3b36ccaa434bbbe155bb9f409453aa3cce8855f94ff490dd7", - "zh:d16ab0aee6e9df81ef0cbddd830654ace7a705d6f0239a895e9fec946a38e37a", - "zh:dddc2189b481eeef0c1969a4be09a586d46f729b1e8bce7d2d0320a530cd4259", - ] -} - -provider "registry.terraform.io/hashicorp/azuread" { - version = "2.30.0" - constraints = "2.30.0" - hashes = [ - "h1:WnSPiREAFwnBUKREokMdHQ8Cjs47MzvS9pG8VS1ktec=", - "zh:1c3e89cf19118fc07d7b04257251fc9897e722c16e0a0df7b07fcd261f8c12e7", - "zh:2e62c193030e04ebb10cc0526119cf69824bf2d7e4ea5a2f45bd5d5fb7221d36", - "zh:2f3c7a35257332d68b778cefc5201a5f044e4914dd03794a4da662ddfe756483", - "zh:35d0d3a1b58fdb8b8c4462d6b7e7016042da43ea9cc734ce897f52a73407d9b0", - "zh:47ede0cd0206ec953d40bf4a80aa6e59af64e26cbbd877614ac424533dbb693b", - "zh:48c190307d4d42ea67c9b8cc544025024753f46cef6ea64db84735e7055a72da", - "zh:6fff9b2c6a962252a70a15b400147789ab369b35a781e9d21cce3804b04d29af", - "zh:7646980cf3438bff29c91ffedb74458febbb00a996638751fbd204ab1c628c9b", - "zh:77aa2fa7ca6d5446afa71d4ff83cb87b70a2f3b72110fc442c339e8e710b2928", - "zh:e20b2b2c37175b89dd0db058a096544d448032e28e3b56e2db368343533a9684", - "zh:eab175b1dfe9865ad9404dccb6d5542899f8c435095aa7c679314b811c717ce7", - "zh:efc862bd78c55d2ff089729e2a34c1831ab4b0644fc11b36ee4ebed00a4797ba", - ] -} - -provider "registry.terraform.io/hashicorp/azurerm" { - version = "3.32.0" - hashes = [ - "h1:/ziRWmSDLF+zqfYDBiO4lCBJ20gIwPjDErIR8whhYSg=", - "zh:3ee1992144e6bf9801c44df0ed1e10413fa83ad605e3ce751cb342dd46904c41", - "zh:4f083079909f929b76c0cb2819b107803ecbf26c761832aaa1e7b4a667025665", - "zh:52ad565c4bd37c2b4f0bba78639277ef98caaebf2c4c00c67a2659561079c21c", - "zh:5ecf7a8470e066cc27b837a8fbc9a02629bb85797007475539983496bcccbc53", - "zh:6348154495cd838862b27a9bc0a2714e8f76cd2919df55fce8da0f64ce240ab1", - "zh:8325c4f5f65e30bba2537c7df702c80ae29999fba6194c258b075b3cbde5a709", - "zh:8b4d33aa76474a9fac9a6859e759c03ffeadb787abf7a9ba5a05b4ca3914c008", - "zh:95ccd31450909582ebcf01548ee20df658049783530d79adcb53a601bb163597", - "zh:c104f977b96c6402276c82a8d9d6fee14381511e832e9c3593e589e5ee4e708c", - "zh:e12372a41a981c24323a467f6c54b0a17e26c85a0fb569e4b733b2a76c9ba6b6", - "zh:e80bf9b674914f91ed00984758288b7266ba5772fad728cd1b4cd2f776851ed8", - "zh:f569b65999264a9416862bca5cd2a6177d94ccb0424f3a4ef424428912b9cb3c", - ] -} diff --git a/examples/adb-uc/stage_3_spn_deploys_uc/data.tf b/examples/adb-uc/stage_3_spn_deploys_uc/data.tf deleted file mode 100644 index b52a645a..00000000 --- a/examples/adb-uc/stage_3_spn_deploys_uc/data.tf +++ /dev/null @@ -1,11 +0,0 @@ -data "azurerm_resource_group" "this" { - name = local.resource_group -} - -data "azurerm_client_config" "current" { -} - -data "azurerm_databricks_workspace" "this" { - name = local.databricks_workspace_name - resource_group_name = local.resource_group -} diff --git a/examples/adb-uc/stage_3_spn_deploys_uc/metastore.tf b/examples/adb-uc/stage_3_spn_deploys_uc/metastore.tf deleted file mode 100644 index c880e1ce..00000000 --- a/examples/adb-uc/stage_3_spn_deploys_uc/metastore.tf +++ /dev/null @@ -1,28 +0,0 @@ -resource "databricks_metastore" "this" { - name = "primary" - storage_root = format("abfss://%s@%s.dfs.core.windows.net/", - azurerm_storage_container.unity_catalog.name, - azurerm_storage_account.unity_catalog.name) - force_destroy = true -} - -resource "databricks_metastore_data_access" "first" { - metastore_id = databricks_metastore.this.id - name = "the-keys" - azure_managed_identity { - access_connector_id = azurerm_databricks_access_connector.unity.id - } - - is_default = true -} - -resource "databricks_metastore_assignment" "this" { - workspace_id = local.databricks_workspace_id - metastore_id = databricks_metastore.this.id -} - -resource "databricks_default_namespace_setting" "this" { - namespace { - value = "main" - } -} \ No newline at end of file diff --git a/examples/adb-uc/stage_3_spn_deploys_uc/providers.tf b/examples/adb-uc/stage_3_spn_deploys_uc/providers.tf deleted file mode 100644 index a8f85cac..00000000 --- a/examples/adb-uc/stage_3_spn_deploys_uc/providers.tf +++ /dev/null @@ -1,50 +0,0 @@ -terraform { - required_providers { - azurerm = { - source = "hashicorp/azurerm" - } - databricks = { - source = "databricks/databricks" - } - azuread = { - source = "hashicorp/azuread" - version = "2.30.0" - } - } -} - -provider "azuread" { - # Configuration options -} - -provider "azurerm" { - features {} -} - -/* -provider "databricks" { - alias = "azure_account" - host = "https://accounts.azuredatabricks.net" - account_id = "34f1bb02-ac1d-4c04-b051-58e094fa668c" // Databricks can provide - auth_type = "azure-cli" -} -*/ -provider "databricks" { - alias = "azure_account" - host = "https://accounts.azuredatabricks.net" -} - -provider "databricks" { - host = local.databricks_workspace_host -} - -locals { - resource_regex = "(?i)subscriptions/(.+)/resourceGroups/(.+)/providers/Microsoft.Databricks/workspaces/(.+)" - subscription_id = regex(local.resource_regex, var.databricks_resource_id)[0] - resource_group = regex(local.resource_regex, var.databricks_resource_id)[1] - databricks_workspace_name = regex(local.resource_regex, var.databricks_resource_id)[2] - tenant_id = data.azurerm_client_config.current.tenant_id - databricks_workspace_host = data.azurerm_databricks_workspace.this.workspace_url - databricks_workspace_id = data.azurerm_databricks_workspace.this.workspace_id - prefix = replace(replace(lower(data.azurerm_resource_group.this.name), "rg", ""), "-", "") -} diff --git a/examples/adb-uc/stage_3_spn_deploys_uc/readme.md b/examples/adb-uc/stage_3_spn_deploys_uc/readme.md deleted file mode 100644 index 56b21a7d..00000000 --- a/examples/adb-uc/stage_3_spn_deploys_uc/readme.md +++ /dev/null @@ -1,34 +0,0 @@ -### Stage 3 SPN deploys UC resources - -In this stage, we use a SPN that has been granted Databricks Account Admin role to deploy UC resouces. This SPN does not require AAD Global Admin role. - -Stage 3 deployment steps: - -az login as SPN: - -```bash -az login --service-principal -u -p --tenant -``` - -Change the `account_id` attribute in the Databricks provider, you should already have the account id info from stage 2, if not you can get it from Azure Databricks Account Console, or reach out to your Databricks contacts: - -``` -provider "databricks" { - alias = "azure_account" - host = "https://accounts.azuredatabricks.net" - account_id = "34f1bb02-ac1d-4c04-b051-58e094fa668c" // Databricks can provide - auth_type = "azure-cli" -} -``` - -Supply your Databricks workspace Resource ID to the variable `databricks_resource_id`. - -Alternatively, we interactively log in as Account Admin using `azure login`. - -In this stage 3 folder, run - -```bash -terraform init & terraform apply -``` - -This completes Stage 3 and you should now have UC resources deployed, including UC metastore. \ No newline at end of file diff --git a/examples/adb-uc/stage_3_spn_deploys_uc/storage.tf b/examples/adb-uc/stage_3_spn_deploys_uc/storage.tf deleted file mode 100644 index b99f1ace..00000000 --- a/examples/adb-uc/stage_3_spn_deploys_uc/storage.tf +++ /dev/null @@ -1,38 +0,0 @@ -resource "azurerm_databricks_access_connector" "unity" { - name = "${local.prefix}-databricks-mi" - resource_group_name = data.azurerm_resource_group.this.name - location = data.azurerm_resource_group.this.location - identity { - type = "SystemAssigned" - } -} - -resource "azurerm_storage_account" "unity_catalog" { - name = "${local.prefix}ucmetastore" - resource_group_name = data.azurerm_resource_group.this.name - location = data.azurerm_resource_group.this.location - tags = data.azurerm_resource_group.this.tags - account_tier = "Standard" - account_replication_type = "ZRS" - is_hns_enabled = true - - network_rules { - default_action = "Deny" - bypass = ["None"] - private_link_access { - endpoint_resource_id = azurerm_databricks_access_connector.unity.id - } - } -} - -resource "azurerm_storage_container" "unity_catalog" { - name = "${local.prefix}metastorecontainer" - storage_account_name = azurerm_storage_account.unity_catalog.name - container_access_type = "private" -} - -resource "azurerm_role_assignment" "example" { - scope = azurerm_storage_account.unity_catalog.id - role_definition_name = "Storage Blob Data Contributor" - principal_id = azurerm_databricks_access_connector.unity.identity[0].principal_id -} diff --git a/examples/adb-uc/stage_3_spn_deploys_uc/terraform.tfvars b/examples/adb-uc/stage_3_spn_deploys_uc/terraform.tfvars deleted file mode 100644 index 356b0402..00000000 --- a/examples/adb-uc/stage_3_spn_deploys_uc/terraform.tfvars +++ /dev/null @@ -1 +0,0 @@ -databricks_resource_id = "/subscriptions/3e551f82-c624-4a3e-b8f6-e050a96a7004/resourceGroups/hwang-demo-rg/providers/Microsoft.Databricks/workspaces/demo-adb-ws" diff --git a/examples/adb-uc/stage_3_spn_deploys_uc/variables.tf b/examples/adb-uc/stage_3_spn_deploys_uc/variables.tf deleted file mode 100644 index 62246602..00000000 --- a/examples/adb-uc/stage_3_spn_deploys_uc/variables.tf +++ /dev/null @@ -1,3 +0,0 @@ -variable "databricks_resource_id" { - description = "The Azure resource ID for the databricks workspace deployment." -} diff --git a/modules/adb-overwatch-analysis/Makefile b/modules/adb-overwatch-analysis/Makefile deleted file mode 100644 index 653039d8..00000000 --- a/modules/adb-overwatch-analysis/Makefile +++ /dev/null @@ -1,7 +0,0 @@ -.PHONY: docs test_docs - -docs: - terraform-docs -c ../../.terraform-docs.yml . - -test_docs: - terraform-docs -c ../../.terraform-docs.yml --output-check . diff --git a/modules/adb-overwatch-analysis/README.md b/modules/adb-overwatch-analysis/README.md deleted file mode 100644 index 1e3d6f34..00000000 --- a/modules/adb-overwatch-analysis/README.md +++ /dev/null @@ -1,42 +0,0 @@ -# adb-overwatch-analysis - -This module deploys the following Databricks [python notebooks](./notebooks) on an existing **Overwatch** workspace. - ![Blank diagram](https://user-images.githubusercontent.com/103026825/233795155-566a9f1a-5ff2-4bfa-b940-4a4c5b898c6f.png) - - - - -## Requirements - -No requirements. - -## Providers - -| Name | Version | -|------|---------| -| [azurerm](#provider\_azurerm) | n/a | -| [databricks](#provider\_databricks) | n/a | - -## Modules - -No modules. - -## Resources - -| Name | Type | -|------|------| -| [databricks_notebook.overwatch_analysis](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/notebook) | resource | -| [azurerm_databricks_workspace.adb-ws](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/data-sources/databricks_workspace) | data source | -| [azurerm_resource_group.rg](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/data-sources/resource_group) | data source | - -## Inputs - -| Name | Description | Type | Default | Required | -|------|-------------|------|---------|:--------:| -| [overwatch\_ws\_name](#input\_overwatch\_ws\_name) | The name of the Overwatch workspace | `string` | n/a | yes | -| [rg\_name](#input\_rg\_name) | Resource group name | `string` | n/a | yes | - -## Outputs - -No outputs. - diff --git a/modules/adb-overwatch-analysis/main.tf b/modules/adb-overwatch-analysis/main.tf deleted file mode 100644 index a114dcda..00000000 --- a/modules/adb-overwatch-analysis/main.tf +++ /dev/null @@ -1,17 +0,0 @@ -data "azurerm_resource_group" "rg" { - name = var.rg_name -} - -data "azurerm_databricks_workspace" "adb-ws" { - name = var.overwatch_ws_name - resource_group_name = var.rg_name -} - -//Upload Databricks notebooks used to analyse the Overwatch results -resource "databricks_notebook" "overwatch_analysis" { - for_each = toset(["Cluster", "Helpers", "Jobs", "Notebook", "Readme", "Workspace"]) - source = "${path.module}/notebooks/${each.key}.py" - path = "/Overwatch/Analysis/${each.key}" - format = "SOURCE" - language = "PYTHON" -} \ No newline at end of file diff --git a/modules/adb-overwatch-analysis/notebooks/Cluster.py b/modules/adb-overwatch-analysis/notebooks/Cluster.py deleted file mode 100644 index 466d9495..00000000 --- a/modules/adb-overwatch-analysis/notebooks/Cluster.py +++ /dev/null @@ -1,897 +0,0 @@ -# Databricks notebook source -# MAGIC %md -# MAGIC %md -# MAGIC ## READ ME -# MAGIC > -# MAGIC - **Overwatch version - 07x** -# MAGIC - **Widgets are added to apply filters to the dashboards** -# MAGIC -# MAGIC Widgets Used: -# MAGIC | # | Widgets | Value | Default -# MAGIC | ----------- | ----------- | ----------- | ----------- | -# MAGIC | 1 | ETL Database Name | Your ETL Database Name | overwatch_etl -# MAGIC | 2 | Consumer DB Name | Your Consumer Database Name | overwatch -# MAGIC | 3 | Workspace Name | List of workspace (overwatch deployed) name | all -# MAGIC | 4 | Start Date | Start date for analysis | Current Date -# MAGIC | 5 | End Date | End date for analysis | 30 days from current -# MAGIC | 6 | Cluster tags | To filter by cluster tag names | None -# MAGIC | 7 | Include weekends | To record all days, include weekends | Yes | -# MAGIC | 8 | Only weekends | To record only weekends | No | - -# COMMAND ---------- - -# MAGIC %md -# MAGIC - **Use the widgets to apply filters in the dashboards** -# MAGIC - **Once the filters are applied, run the helper cmd (*""%run "./Helpers"""*) to reflect the filters in the master dataframe** -# MAGIC - **Go to View on topbar and select the *View* named as ClusterFinal under *Dashboards* to view the plots alone** - -# COMMAND ---------- - -# To remove all the widgets. Run only for the first time and comment it out after the first run. -# dbutils.widgets.removeAll() - -# COMMAND ---------- - -dbutils.widgets.text("etlDB", "overwatch_etl", "1. ETL Database Name") -dbutils.widgets.text("consumerDB", "overwatch", "2. Consumer DB Name") - -# COMMAND ---------- - -# MAGIC %md -# MAGIC #### Update Boxes 1 and 2 with your etl and consumer database names respectively -# MAGIC > -# MAGIC The following cells must know the Overwatch database names before they can execute. Please populate the cells before continuing. - -# COMMAND ---------- - -# To get the database names from the widgets -etlDB = str(dbutils.widgets.get("etlDB")) -consumerDB = str(dbutils.widgets.get("consumerDB")) - -# COMMAND ---------- - -# MAGIC %run "/Dashboards_Dev/In Progress/07x_rc_customer/Helpers" $etlDB = etlDB $consumerDB = consumerDB - -# COMMAND ---------- - -dbutils.widgets.text("tags", "all", "6. Cluster tags") -dbutils.widgets.dropdown("include_weekends", "Yes", ["Yes", "No"], "7. Include weekends") -dbutils.widgets.dropdown("only_weekends", "No", ["Yes", "No"], "8. Only weekends") -cluster_tags = str(dbutils.widgets.get("tags")) -include_weekends = dbutils.widgets.get("include_weekends") -only_weekends = dbutils.widgets.get("only_weekends") - -fetch_Name = spark.sql(f"select distinct workspace_name from {etlDB}.pipeline_report").rdd.flatMap(lambda x: x).collect()+["all"] -dbutils.widgets.multiselect("workspace_name","all",fetch_Name, "3. Workspace Name") -workspaceName = spark.sql(f"select distinct workspace_name from {etlDB}.pipeline_report").rdd.flatMap(lambda x: x).collect() if 'all' in dbutils.widgets.get("workspace_name").split(',') else dbutils.widgets.get("workspace_name").split(',') -dbutils.widgets.combobox("4. Start Date", f"{date.today() - timedelta(days=30)}", "") -dbutils.widgets.combobox("5. End Date", f"{date.today()}", "") -start_date = str(dbutils.widgets.get("4. Start Date")) -end_date = str(dbutils.widgets.get("5. End Date")) - -# COMMAND ---------- - -# MAGIC %md -# MAGIC #### Now that all the widgets are present and you can apply the filters wherever required - -# COMMAND ---------- - -masters = master(etlDB,consumerDB,workspaceName,start_date,end_date) - -clsf_master = masters.cluster_master_filter(includeWeekend = include_weekends,onlyWeekend = only_weekends) - -# COMMAND ---------- - -# MAGIC %md -# MAGIC ### Compute Overview -# MAGIC -# MAGIC > As a budget owner / admin I want to be able to quickly understand the cluster makeup in my environment their costs, etc. - -# COMMAND ---------- - -df = clsf_master\ -.select("cluster_category")\ -.distinct() - -display(df) - -# COMMAND ---------- - -dbu_spend = clsf_master\ -.groupBy("cluster_category", - "state_start_date", - "workspace_name")\ -.agg(round(sum(col("total_DBU_cost")),2).alias("total_DBU_cost(USD)"))\ -.toPandas() - -display(dbu_spend) - -# COMMAND ---------- - - -fig = px.box(dbu_spend, - x = "cluster_category", - y = "total_DBU_cost(USD)", - title = "DBU Spend by cluster category", - hover_data = ["state_start_date"], - points = "all", - color = "workspace_name") -##fig.update_traces(quartilemethod="exclusive") -fig.show() - -# COMMAND ---------- - -from pyspark.sql import SparkSession, Row -from pyspark.sql.window import Window -from pyspark.sql.functions import col, row_number - -daily_cluster_cost = clsf_master\ -.withColumn("date", explode("state_dates"))\ -.groupBy("date", "organization_id", "workspace_name", "days_in_state", "cluster_id", "cluster_name")\ -.agg(round((sum(col("total_DBU_cost"))/col("days_in_state")),2).alias("total_DBU_cost_(USD)"), - round((sum(col("total_compute_cost"))/col("days_in_state")),2).alias("total_compute_cost_(USD)"), - round((sum(col("total_cost"))/col("days_in_state")),2).alias("total_cost_(USD)"), - )\ -.orderBy(col("total_DBU_cost_(USD)").desc())\ -.distinct() - - -windowdf = Window.partitionBy(daily_cluster_cost["date"]).orderBy(daily_cluster_cost["total_DBU_cost_(USD)"].desc()) - -daily_cluster_cost = daily_cluster_cost\ -.withColumn("row", row_number().over(windowdf))\ -.filter(col("row") == 1 )\ -.distinct()\ -.toPandas() - -display(daily_cluster_cost) - -# COMMAND ---------- - -fig = px.bar(daily_cluster_cost, - x = "date", - y = "total_DBU_cost_(USD)", - title = "DBU spend by the most expensive cluster per day", - hover_data = ["organization_id", - "workspace_name", - "date", - "cluster_id", - "cluster_name", - "total_DBU_cost_(USD)", - "total_compute_cost_(USD)", - "total_cost_(USD)"], - color = "total_cost_(USD)") -fig.show() - -# COMMAND ---------- - -from pyspark.sql import SparkSession, Row -from pyspark.sql.window import Window -from pyspark.sql.functions import col, row_number - -daily_cluster_spent = clsf_master\ -.withColumn("date", explode("state_dates"))\ -.groupBy("date", - "days_in_state", - "cluster_id", - "cluster_name", - "organization_id", - "workspace_name")\ -.agg(round((sum(col("total_DBU_cost"))/col("days_in_state")),2).alias("total_DBU_cost_(USD)"), - round((sum(col("total_compute_cost"))/col("days_in_state")),2).alias("total_compute_cost_(USD)"), - round((sum(col("total_cost"))/col("days_in_state")),2).alias("total_cost_(USD)"))\ -.orderBy(col("date").asc())\ -.distinct() - -windowdf = Window.partitionBy(daily_cluster_spent["date"]).orderBy(daily_cluster_spent["total_DBU_cost_(USD)"].desc()) - -daily_cluster_spent = daily_cluster_spent\ -.withColumn("row", row_number().over(windowdf))\ -.filter(col("row") <= 20) - -daily_cluster_spent_grouped = daily_cluster_spent\ -.groupBy("date", - "organization_id", - "workspace_name" - )\ -.agg(round(sum("total_DBU_cost_(USD)"),2).alias("Total_DBU_cost_(USD)"))\ -.orderBy(col("date").desc())\ -.distinct()\ -.toPandas() - - -display(daily_cluster_spent_grouped) - -# COMMAND ---------- - -fig = px.bar(daily_cluster_spent_grouped, - x = "date", - y = "Total_DBU_cost_(USD)", - title = "Daily cluster spend chart (top 20)", - hover_data = ["organization_id", - "workspace_name", - "date", - "Total_DBU_cost_(USD)" - ], - color = "workspace_name") -fig.show() - -# COMMAND ---------- - -dbu_spend_without_autotermination = clsf_master\ -.filter(((col("auto_termination_minutes") == 0) | (col("auto_termination_minutes").isNull())) - & (col("cluster_category") == "Interactive") - - )\ -.fillna(0, subset=["core_hours"])\ -.withColumn("date", explode("state_dates"))\ -.groupBy("date", "organization_id", "workspace_name", "days_in_state", "cluster_id", "cluster_name")\ -.agg(round((sum(col("total_DBU_cost"))/col("days_in_state")),2).alias("DBU_cost_(USD)"), - round((sum(col("total_compute_cost"))/col("days_in_state")),2).alias("Compute_cost_(USD)"), - round((sum(col("total_cost"))/col("days_in_state")),2).alias("total_cost_(USD)"), - round((sum(col("core_hours"))/col("days_in_state")),2).alias("core_hours"))\ -.orderBy(col("DBU_cost_(USD)").desc())\ -.distinct() - -Windowdf = Window.partitionBy(dbu_spend_without_autotermination["date"]).orderBy(dbu_spend_without_autotermination["DBU_cost_(USD)"].desc()) - -dbu_spend_without_autotermination = dbu_spend_without_autotermination\ -.withColumn("row", row_number().over(Windowdf))\ -.filter(col("row") <= 3 ) - -dbu_spend_without_autotermination = dbu_spend_without_autotermination\ -.withColumn("rank", when(col("row") == 1, "Most expensive")\ - .when(col("row") == 2, "second expensive")\ - .when(col("row") == 3, "third expensive")\ - )\ -.distinct()\ -.toPandas() - - - -display(dbu_spend_without_autotermination) - -# COMMAND ---------- - -fig = px.scatter(dbu_spend_without_autotermination, - x = "date", - y = "DBU_cost_(USD)", - title = "DBU Spent by the top 3 expensive Interactive clusters (without auto-termination) per day", - color = "workspace_name", - hover_data = ["organization_id", - "workspace_name", - "cluster_id", - "cluster_name", - "rank", - "DBU_cost_(USD)", - "Compute_cost_(USD)", - "total_cost_(USD)", - "core_hours" - ]) -fig.update_traces(marker_size=15) -fig.show() - -# COMMAND ---------- - -# import plotly.graph_objects as go -# from plotly.subplots import make_subplots - -# def getValLabel(dataframe): -# ji = dataframe.select(["workspace_name","Number_of_clusters"]).rdd.flatMap(lambda x: x).collect() -# labels = [] -# values = [] -# for i in range(0, len(ji)): -# if i%2 == 0: -# labels.append(ji[i]) -# else: -# values.append(ji[i]) -# return labels, values - -# fig = make_subplots(2, 3, -# specs = [[{'type':'domain'}, {'type':'domain'}, {'type':'domain'}], -# [{'type':'domain'}, {'type':'domain'}, {'type':'domain'}]], -# subplot_titles = cluster_count.select("cluster_category").distinct().rdd.flatMap(lambda x: x).collect() -# ) - -# #name the dataframe -# cluster_category = cluster_count.select('cluster_category').distinct().rdd.flatMap(lambda x: x).collect() - -# dataframe_list = [] -# for i in cluster_category: -# dataframe = cluster_count\ -# .filter(col("cluster_category") == i)\ -# .select(col("Number_of_clusters"), col("workspace_name")) -# dataframe_list.append(dataframe) - - -# row = 1 -# col = 1 -# for i in dataframe_list: -# # split the labels and values in the dataframe -# labels, values = getValLabel(i) - -# fig.add_trace(go.Pie( -# labels = labels, -# values = values, -# name = str(i) -# ), row, col) - -# if col == 3: -# col = 1 -# row = row + 1 -# else: -# col = col + 1 - - -# fig.update_layout(title_text='Cluster count in each category') - -# fig.show() - - -# COMMAND ---------- - -cluster_count_SN = clsf_master\ -.groupBy("organization_id", "workspace_name", "cluster_category")\ -.agg(countDistinct("cluster_id").alias("Number_of_clusters"), - round(sum(col("total_DBU_cost")),2).alias("Total_DBU_Cost_(USD)") - )\ -.filter(clsf_master["cluster_category"] == "Single Node")\ -.toPandas() - - -display(cluster_count_SN) - -# COMMAND ---------- - -fig = px.pie(cluster_count_SN, - values = 'Number_of_clusters', - names = 'workspace_name', - title = 'Cluster count distribution in single node cluster category', - hover_data = ['Total_DBU_Cost_(USD)'], - color_discrete_sequence = px.colors.sequential.Bluered, - ) -fig.update_traces(textinfo='percent+label', textfont_size=12, pull = [0, 0, 0.1, 0], hole = 0.3) -fig.show() - -# COMMAND ---------- - -cluster_count_Interactive = clsf_master\ -.groupBy("organization_id", "workspace_name", "cluster_category")\ -.agg(countDistinct("cluster_id").alias("Number_of_clusters"), - round(sum(col("total_DBU_cost")),2).alias("Total_DBU_Cost_(USD)") - )\ -.filter(clsf_master["cluster_category"] == "Interactive")\ -.toPandas() - - -fig = px.pie(cluster_count_Interactive, - values = 'Number_of_clusters', - names = 'workspace_name', - title = 'Cluster count distribution in Interactive cluster category', - hover_data = ['Total_DBU_Cost_(USD)'], - color_discrete_sequence = px.colors.sequential.Bluered, - ) -fig.update_traces(textinfo='percent+label', textfont_size=12, pull = [0, 0, 0.1, 0], hole = 0.3) -fig.show() - -# COMMAND ---------- - -cluster_count_Automated = clsf_master\ -.groupBy("organization_id", "workspace_name", "cluster_category")\ -.agg(countDistinct("cluster_id").alias("Number_of_clusters"), - round(sum(col("total_DBU_cost")),2).alias("Total_DBU_Cost_(USD)") - )\ -.filter(clsf_master["cluster_category"] == "Automated")\ -.toPandas() - - -fig = px.pie(cluster_count_Automated, - values = 'Number_of_clusters', - names = 'workspace_name', - title = 'Cluster count distribution in Automated cluster category', - hover_data = ['Total_DBU_Cost_(USD)'], - color_discrete_sequence = px.colors.sequential.Bluered, - ) -fig.update_traces(textinfo='percent+label', textfont_size=12, pull = [0, 0, 0.1, 0], hole = 0.3) -fig.show() - -# COMMAND ---------- - -cluster_count_Warehouse = clsf_master\ -.groupBy("organization_id", "workspace_name", "cluster_category")\ -.agg(countDistinct("cluster_id").alias("Number_of_clusters"), - round(sum(col("total_DBU_cost")),2).alias("Total_DBU_Cost_(USD)") - )\ -.filter(clsf_master["cluster_category"] == "Warehouse")\ -.toPandas() - - -fig = px.pie(cluster_count_Warehouse, - values = 'Number_of_clusters', - names = 'workspace_name', - title = 'Cluster count distribution in Warehouse cluster category', - hover_data = ['Total_DBU_Cost_(USD)'], - color_discrete_sequence = px.colors.sequential.Bluered, - ) -fig.update_traces(textinfo='percent+label', textfont_size=12, pull = [0, 0, 0.1, 0], hole = 0.3) -fig.show() - -# COMMAND ---------- - -cluster_count_HC = clsf_master\ -.groupBy("organization_id", "workspace_name", "cluster_category")\ -.agg(countDistinct("cluster_id").alias("Number_of_clusters"), - round(sum(col("total_DBU_cost")),2).alias("Total_DBU_Cost_(USD)") - )\ -.filter(clsf_master["cluster_category"] == "High-Concurrency")\ -.toPandas() - - -fig = px.pie(cluster_count_HC, - values = 'Number_of_clusters', - names = 'workspace_name', - title = 'Cluster count distribution in High concurrency cluster category', - hover_data = ['Total_DBU_Cost_(USD)'], - color_discrete_sequence = px.colors.sequential.Bluered, - ) -fig.update_traces(textinfo='percent+label', textfont_size=12, pull = [0, 0, 0.1, 0], hole = 0.3) -fig.show() - -# COMMAND ---------- - -cluster_count_ST = clsf_master\ -.groupBy("organization_id", "workspace_name", "cluster_category")\ -.agg(countDistinct("cluster_id").alias("Number_of_clusters"), - round(sum(col("total_DBU_cost")),2).alias("Total_DBU_Cost_(USD)") - )\ -.filter(clsf_master["cluster_category"] == "Standard")\ -.toPandas() - - -fig = px.pie(cluster_count_ST, - values = 'Number_of_clusters', - names = 'workspace_name', - title = 'Cluster count distribution in Standard cluster category', - hover_data = ['Total_DBU_Cost_(USD)'], - color_discrete_sequence = px.colors.sequential.Bluered, - ) -fig.update_traces(textinfo='percent+label', textfont_size=12, pull = [0, 0, 0.1, 0], hole = 0.3) -fig.show() - -# COMMAND ---------- - -from pyspark.sql.functions import * -node_type_count_percent = clsf_master\ -.groupBy("organization_id","workspace_name","node_type_id")\ -.agg(countDistinct(col("cluster_id")).alias("cluster_count"))\ -.orderBy(col("cluster_count").desc())\ -.toPandas() - -node_type_count_percent.loc[((node_type_count_percent['cluster_count'] / node_type_count_percent['cluster_count'].sum())* 100) < 2 ,'node_type_id'] = 'Other Types' - -display(node_type_count_percent) - -# COMMAND ---------- - -fig = px.pie(node_type_count_percent, - values = 'cluster_count', - names = 'node_type_id', - title = 'Cluster node type breakdown', - hover_data = ['workspace_name'], - color_discrete_sequence = px.colors.sequential.Bluered, - ) -fig.update_traces(textinfo='percent+label', textfont_size=12, pull = [0, 0, 0.1, 0]) -fig.show() - -# COMMAND ---------- - -from pyspark.sql.functions import * -node_type_potential = clsf_master\ -.filter(col("worker_potential_core_H").isNotNull())\ -.groupBy("organization_id", "workspace_name", "node_type_id")\ -.agg(round(sum(col("worker_potential_core_H")),2).alias("Total_node_potential_hours"), - round(sum(col("total_worker_cost")),2).alias("Total_worker_cost(USD)") - )\ -.orderBy(col("Total_node_potential_hours").desc())\ -.toPandas() - -display(node_type_potential) - -# COMMAND ---------- - - -minimum_cluster_count = int(node_type_potential["Total_node_potential_hours"].sum()*0.05) - -node_type_potential.loc[node_type_potential['Total_node_potential_hours'] < minimum_cluster_count,'node_type_id'] = 'Other Types' - -fig = px.pie(node_type_potential, - values = 'Total_node_potential_hours', - names = 'node_type_id', - title = 'Cluster node type breakdown by potential', - hover_data = ['Total_worker_cost(USD)'], - color_discrete_sequence = px.colors.sequential.RdBu - ) - -fig.update_traces(textinfo='percent+label', textfont_size=12) -fig.show() - -# COMMAND ---------- - -node_type_potential_by_category = clsf_master\ -.filter(col("worker_potential_core_H").isNotNull())\ -.groupBy("organization_id", "workspace_name", "cluster_category")\ -.agg(round(sum(col("worker_potential_core_H")),2).alias("Total_node_potential_hours"), - round(sum(col("total_worker_cost")),2).alias("Total_worker_cost(USD)") - )\ -.orderBy(col("Total_node_potential_hours").desc())\ -.toPandas() - -display(node_type_potential_by_category) - -# COMMAND ---------- - -fig = px.pie(node_type_potential_by_category, - values = 'Total_node_potential_hours', - names = 'cluster_category', - title = 'Cluster node potential breakdown by cluster category', - hover_data = ['Total_worker_cost(USD)'], - color_discrete_sequence = px.colors.sequential.RdBu - ) -fig.update_traces(textinfo='percent+label', textfont_size=12, pull = [0.08, 0.08, 0.08, 0.08, 0.08]) -fig.show() - -# COMMAND ---------- - -from pyspark.sql import SparkSession, Row -from pyspark.sql.window import Window -from pyspark.sql.functions import col, row_number - -cluster_cost_per_category = ( - clsf_master.groupBy("state_start_date", "workspace_name", "cluster_category") - .agg( - round(sum(col("total_DBU_cost")), 2).alias("total_DBU_cost(USD)"), - round(sum(col("total_compute_cost")), 2).alias("Total_compute_cost(USD)"), - round(sum(col("total_cost")), 2).alias("Total_cost(USD)"), - ) - .orderBy(round(sum(col("total_cost")), 2).desc()) -) - -windowDept = Window.partitionBy(cluster_cost_per_category["cluster_category"]).orderBy( - cluster_cost_per_category["Total_cost(USD)"].desc() -) - -cluster_cost_per_category = ( - cluster_cost_per_category.withColumn("row", row_number().over(windowDept)) - .filter( - (col("row") <= 20) - & ( - (cluster_cost_per_category["cluster_category"] == ("Interactive")) - | (cluster_cost_per_category["cluster_category"] == ("Automated")) - | (cluster_cost_per_category["cluster_category"] == ("High-Concurrency")) - ) - ) - .toPandas() -) - -display(cluster_cost_per_category) - -# COMMAND ---------- - -fig = px.violin(cluster_cost_per_category, - x = "cluster_category", - y = "Total_cost(USD)", - title = "Total cost (dbu cost + compute cost) incurred by top 20 clusters per cluster category", - color = "workspace_name", - box = True, - points = "all", - hover_data = cluster_cost_per_category.columns) - -fig.show() - -# COMMAND ---------- - -# MAGIC %md -# MAGIC ## Autoscaling clusters - -# COMMAND ---------- - -# MAGIC %md -# MAGIC ### Auto-Scaling -# MAGIC > As an admin I want to understand how many clusters are utilizing auto-scaling, their scaling ranges, and their efficiencies - -# COMMAND ---------- - -autoscaling_cluster = clsf_master\ -.filter(col("autoscale").isNotNull())\ -.groupBy("organization_id", "workspace_name", "cluster_category")\ -.agg(countDistinct(col("cluster_id")).alias("cluster_count"), - round(sum(col("total_DBU_cost")),2).alias("total_DBU_cost(USD)"))\ -.orderBy(col("cluster_count").desc())\ -.toPandas() - -display(autoscaling_cluster) - -# COMMAND ---------- - -fig = px.pie(autoscaling_cluster, - values = 'cluster_count', - names = 'cluster_category', - title = 'Percentage of Autoscaling clusters per category', - hover_data = ['total_DBU_cost(USD)'] - ) -fig.update_traces(textinfo='percent+label', textfont_size=12, pull = [0.08, 0.08, 0.08, 0.08, 0.08]) -fig.show() - -# COMMAND ---------- - -scaleup_time_withoutPools = clsf_master\ -.filter( - (col("autoscale").isNotNull()) - & (col("state") == "RESIZING") - & (col("current_num_workers") < col("target_num_workers")) - & (col("instance_pool_id").isNull()) - - )\ -.groupBy("state_start_date", "organization_id", "workspace_name", "cluster_category")\ -.agg(round(avg("uptime_in_state_H"), 2).alias("average_scale_up_time(Hours)"))\ -.orderBy(col("average_scale_up_time(Hours)").desc())\ -.distinct()\ -.toPandas() - -display(scaleup_time_withoutPools) -# clusters with pools are not getting resized. - -# COMMAND ---------- - -fig = px.box(scaleup_time_withoutPools, - x = "cluster_category", - y = "average_scale_up_time(Hours)", - title = "Scale up time of clusters (Without pools) by cluster category", - points = "all", - color = "workspace_name") -##fig.update_traces(quartilemethod="exclusive") - -fig.show() - -# COMMAND ---------- - -scaleup_time_withPools = clsf_master\ -.filter( - (col("autoscale").isNotNull()) - & (col("state") == "RESIZING") - & (col("current_num_workers") < col("target_num_workers")) - & (col("instance_pool_id").isNotNull()) - )\ -.groupBy("state_start_date", "organization_id", "workspace_name", "cluster_category")\ -.agg(round(avg("uptime_in_state_H"), 2).alias("average_scale_up_time(Hours)"))\ -.orderBy(col("average_scale_up_time(Hours)").desc())\ -# .toPandas() - -display(scaleup_time_withPools) - -# COMMAND ---------- - -# fig = px.box(scaleup_time_withPools, -# x = "cluster_category", -# y = "average_scale_up_time(Hours)", -# title = "Scale up time of clusters (With pools) by cluster category", -# points = "all", -# color = "workspace_name") -# ##fig.update_traces(quartilemethod="exclusive") - -# fig.show() - -# COMMAND ---------- - -cost_of_autoscaling_clusters_per_category = clsf_master\ -.filter(col("autoscale").isNotNull())\ -.groupBy("state_start_date","workspace_name","cluster_category")\ -.agg( - round(sum(col("total_DBU_cost")),2).alias("total_DBU_cost(USD)"), - round(sum(col("total_compute_cost")),2).alias("Total_compute_cost(USD)"), - round(sum(col("total_cost")),2).alias("Total_cost(USD)") - )\ -.orderBy((col("Total_cost(USD)")).desc()) - -windowDept = Window.partitionBy(cost_of_autoscaling_clusters_per_category["cluster_category"]).orderBy( - cost_of_autoscaling_clusters_per_category["Total_cost(USD)"].desc() -) - -cost_of_autoscaling_clusters_per_category = ( - cost_of_autoscaling_clusters_per_category.withColumn("row", row_number().over(windowDept))\ - .filter( - (col("row") <= 20))\ - .drop("row")\ - .toPandas() -) - - -display(cost_of_autoscaling_clusters_per_category) - -# COMMAND ---------- - -fig = px.violin(cost_of_autoscaling_clusters_per_category, - x = "cluster_category", - y = "Total_cost(USD)", - title = "Total cost of autoscaling clusters (top 20) per cluster category", - color = "workspace_name", - box = True, - points = "all", - hover_data = cost_of_autoscaling_clusters_per_category.columns) - -fig.show() - -# COMMAND ---------- - -# MAGIC %md -# MAGIC ### Cluster Stability -# MAGIC -# MAGIC > As an admin I want to understand my cluster stability. I operate under the premise that unstable clusters are restarted when they become unstable -# MAGIC -# MAGIC -# MAGIC -# MAGIC -# MAGIC - -# COMMAND ---------- - -ClusterFailedCount = clsf_master\ -.where( - ((col("state") == "SPARK_EXCEPTION") - |(col("state") == "DRIVER_UNAVAILABLE") - |(col("state") == "DBFS_DOWN") - |(col("state") == "NODES_LOST") - |(col("state") == "DRIVER_NOT_RESPONDING") - |(col("state") == "METASTORE_DOWN") - ) & - (col("is_automated") == "false") - )\ -.groupBy(clsf_master["state"], clsf_master["node_type_id"])\ -.agg(countDistinct("cluster_id").alias("Count_ClusterID"))\ -.orderBy(col("Count_ClusterID").desc())\ -.limit(20)\ -.toPandas() - - -display(ClusterFailedCount) - - -# COMMAND ---------- - -fig = px.bar(ClusterFailedCount, - x = "state", - y = "Count_ClusterID", - color_continuous_scale = ["green", "red"], - color = "Count_ClusterID", - hover_data = ["node_type_id"], - title = "Cluster Failure States and count of failures") - -fig = fig.update_layout( - xaxis_title = "Failure states", - yaxis_title = "Count of failure", -) - - - -fig.show() - -# COMMAND ---------- - -ClusterFailedCountbyWorkspace = clsf_master\ -.where( - ((col("state") == "SPARK_EXCEPTION") - |(col("state") == "DRIVER_UNAVAILABLE") - |(col("state") == "DBFS_DOWN") - |(col("state") == "NODES_LOST") - |(col("state") == "DRIVER_NOT_RESPONDING") - |(col("state") == "METASTORE_DOWN") - ) & - (col("is_automated") == "false") - )\ -.groupBy(clsf_master["organization_id"], clsf_master["workspace_name"], clsf_master["state"], clsf_master["node_type_id"])\ -.agg(countDistinct("cluster_id").alias("Count_ClusterID"), - round(sum(col("total_cost")),2).alias("cost_of_failure") - )\ -.orderBy(col("cost_of_failure").desc())\ -.toPandas() - - -display(ClusterFailedCountbyWorkspace) - - -# COMMAND ---------- - -fig = px.bar(ClusterFailedCountbyWorkspace, - x = "state", - y = "cost_of_failure", - color_continuous_scale = ["green", "red"], - color = "workspace_name", - hover_data = ["organization_id", "workspace_name", "node_type_id", "Count_ClusterID"], - title = "Cost of cluster failures per Failure states per workspace") - -fig = fig.update_layout( - xaxis_title = "Failure states", - yaxis_title = "Cost of Failure", -) - -fig.show() - -# COMMAND ---------- - - -ClusterFailedCountViolin = clsf_master\ -.where( - ((col("state") == "SPARK_EXCEPTION") - |(col("state") == "DRIVER_UNAVAILABLE") - |(col("state") == "DBFS_DOWN") - |(col("state") == "NODES_LOST") - |(col("state") == "DRIVER_NOT_RESPONDING") - |(col("state") == "METASTORE_DOWN") - ) & (col("is_automated") == "false") -)\ -.groupBy(clsf_master["organization_id"], - clsf_master["workspace_name"], - clsf_master["cluster_category"], - clsf_master["state"], - clsf_master["node_type_id"] - )\ -.agg(countDistinct("cluster_id").alias("Count_ClusterID"))\ -.orderBy(col("Count_ClusterID").desc())\ -.limit(30)\ -.distinct()\ -.toPandas() - -display(ClusterFailedCountViolin) - -# COMMAND ---------- - -fig = px.violin(ClusterFailedCountViolin, - x = "state", - y = "Count_ClusterID", - title = "Cluster Failure States and failure count distribution", - color = "workspace_name", - box = True, - points = "all", - hover_data = ["organization_id", "workspace_name", "node_type_id", "cluster_category"] - ) - -fig = fig.update_layout( - xaxis_title = "Failure states", - yaxis_title = "Count of failure") - -fig.show() - -# COMMAND ---------- - -restart_count = clsf_master\ -.where((col("cluster_category") == "Interactive") - & (col("state") == "RESTARTING") - )\ -.groupBy("state_start_date", - "organization_id", - "cluster_id", - "cluster_name", - "workspace_name", - "state" - )\ -.agg(countDistinct("unixTimeMS_state_start").alias("cluster_restart_count"), - round(sum(col("total_cost")),2).alias("Restarting_cost_(USD)"), - round(sum(col("uptime_in_state_H")),2).alias("Uptime_in_state_Hours"))\ -.toPandas() - - -display(restart_count) - -# COMMAND ---------- - -fig = px.bar(restart_count, - x = "state_start_date", - y = "cluster_restart_count", - title = "Interactive cluster restarts per day per cluster", - color = "workspace_name", - hover_data = ["organization_id", "workspace_name", "cluster_id", "cluster_name", "Restarting_cost_(USD)", "Uptime_in_state_Hours"] - ) -fig.show() - -# COMMAND ---------- - diff --git a/modules/adb-overwatch-analysis/notebooks/Helpers.py b/modules/adb-overwatch-analysis/notebooks/Helpers.py deleted file mode 100644 index 4e4c6602..00000000 --- a/modules/adb-overwatch-analysis/notebooks/Helpers.py +++ /dev/null @@ -1,491 +0,0 @@ -# Databricks notebook source -from pyspark.sql.functions import * -from pyspark.sql.window import Window, WindowSpec -import plotly.express as px -from operator import add -from functools import reduce -import pyspark.sql.functions as F -from pyspark.sql.functions import collect_list, collect_set, when -from pyspark.sql.window import Window -from pyspark.sql.functions import rank, col -import plotly.graph_objects as go -import pyspark.sql.functions as func -from pyspark.sql.functions import UserDefinedFunction -from pyspark.sql.types import StringType -from datetime import date, timedelta -import pandas as pd -import pyspark -from pyspark.sql.functions import concat_ws - -# COMMAND ---------- - -class helpers: - - - def __init__(self, _etl_db, _consumer_db): - self.etl_db = _etl_db - self.consumer_db = _consumer_db -# try: -# if _etl_db == "" or _etl_db is null: -# print("add the database widget") -# except: -# print("add the database widget") - self.org_ids_lookup = table(f"{_etl_db}.pipeline_report")\ - .select(col("organization_id"), col("workspace_name"))\ - .distinct()\ - .cache() - self.org_ids_lookup.count() -# self.masters = new master(...) -# masters.clusterstatefact(...) - - def filter_workspaces(self, workspace_names) -> pyspark.sql.dataframe.DataFrame: - """ - Returns a dataframe filter by selected workspace name. - - Parameters: - workspace_names (str): Workspace Name - - Returns: - DataFrame: Filtered by selected workspace(From widgets) - - Example: - outputDF = inputDF.transform(object_name.filter_workspaces("workspace_names")) - """ - def inner(df): - org_ids = self.org_ids_lookup\ - .filter(col("workspace_name").isin(workspace_names))\ - .select(col("organization_id"))\ - .rdd.flatMap(lambda x: x).collect() - return df.filter(col("organization_id").isin(org_ids)) - return inner - - def filter_dates(self,dateColumn:str,start_date,end_date) -> pyspark.sql.dataframe.DataFrame: - """ - Returns a dataframe filter by dates(between selected start and end date). - - Parameters: - dateColumn (str): Date column name - start_date (date): Start date - end_date (date): End date - - Returns: - DataFrame: Data between selected dates - - Example: - outputDF = inputDF.transform(object_name.filter_dates(date_column_name,start_date,end_date)) - """ - def inner(df): - return df.filter(F.col(dateColumn).between(pd.to_datetime(start_date),pd.to_datetime(end_date))) - return inner - - def filter_by_weekdays(self,include_weekends,only_weekends) -> pyspark.sql.dataframe.DataFrame: - """ - Returns a dataframe filter by weekends and weekdays. - - Parameters: - include_weekends (str): yes/no - only_weekends (str): yes/no - - Returns: - DataFrame: filter by weekends and weekdays. - - Example: - outputDF = inputDF.transform(object_name.filter_dates(yes,no)) - """ - - def inner(df): - if include_weekends == 'Yes'and only_weekends == 'No': - return df - elif include_weekends == 'Yes' and only_weekends == 'Yes': - return df.filter(col('is_weekend') == 1) - elif include_weekends == 'No' and only_weekends == 'No': - return df.filter(col('is_weekend') == 0) - else: - raise Exception("Sorry, Please check the widget values (If Include weekends is 'NO' you cant keep Only weekends as 'Yes')") - return inner - - def filter_clusters(self,clusterTable) -> pyspark.sql.dataframe.DataFrame: - """ - Returns a dataframe filter with selected cluster_ids. - - Parameters: - clusterTable (DataFrame): Contains only selected clusters - - Returns: - DataFrame: Data selected clusters - - Example: - outputDF = inputDF.transform(object_name.filter_clusters(clusterTableName)) - """ - def inner(df): - data = clusterTable.join(df,on="cluster_id",how="inner")\ - .select(df["*"]) - return data - return inner - - # split_path - def partition_split(self,folder_level, consumerDB): - def inner(df): - if int(folder_level) < 1: - raise Exception("Please enter the folder depth level") - num = int(folder_level) + 1 - nb_df = df.withColumn("folder_path", concat_ws('/', slice(split(col('notebook_path'), '/'), 1, num))) - return nb_df - return inner - -# COMMAND ---------- - -class master(helpers): - - def __init__(self,_etl_db,_consumer_db,_workspace_name,_from_date,_until_date): - - self.etl_db = _etl_db - self.consumer_db = _consumer_db - self.start_date = _from_date - self.end_date = _until_date - self.workspace_name = _workspace_name - helpers.__init__(self, self.etl_db, self.consumer_db) - - - def spark_notebook_master(self, **kwargs): - self.include_weekend = kwargs.get("includeWeekend",True) - self.only_weekend = kwargs.get("onlyWeekend",False) - self.path_depth = kwargs.get("folder_level") - - sparkJob = spark.sql(f"select * from {self.consumer_db}.sparkJob").withColumn("is_weekend",dayofweek("date").isin([1,7]).cast("int")) - sparkTask = spark.sql(f"select * from {self.consumer_db}.sparkTask").withColumn("is_weekend",dayofweek("date").isin([1,7]).cast("int")) -# sparkTask = spark.sql(f"select * from {self.etl_db}.sparkTask_gold").withColumn("is_weekend",dayofweek("date").isin([1,7]).cast("int")) - - notebook = spark.sql("select * from {}.notebook".format(self.consumer_db)) - - SparkTask_master = sparkTask.join(sparkJob, - (sparkTask["cluster_id"] == sparkJob["cluster_id"]) & - (sparkTask["workspace_name"] == sparkJob["workspace_name"]) & - (sparkTask["timestamp"] == sparkJob["timestamp"]) & - (sparkTask["organization_id"] == sparkJob["organization_id"]) - ,"inner")\ - .withColumn('MemoryBytesSpilled', sparkTask.task_metrics['MemoryBytesSpilled'])\ - .withColumn('DiskBytesSpilled', sparkTask.task_metrics['DiskBytesSpilled'])\ - .withColumn("Execution_type", expr("case when db_job_id is null and db_id_in_job is null then 'Manual_notebook' else 'Job_notebook' end"))\ - .select(sparkTask["*"] - ,sparkJob["db_job_id"] - ,sparkJob["db_id_in_job"] - ,sparkJob["notebook_id"] - ,sparkJob["notebook_path"] - ,sparkJob["execution_id"] - ,sparkJob["job_runtime"] - ,sparkJob["job_result"] - ,sparkJob["user_email"] - ,explode(sparkJob["stage_ids"]).alias("stage_id") - ,"MemoryBytesSpilled" - ,"DiskBytesSpilled" - ,"Execution_type" - ) - -# sparkMaster = SparkTask_master.join(notebook, SparkTask_master["notebook_path"] == notebook["notebook_path"], "inner")\ -# .withColumn("Execution_type", expr("case when db_job_id is null and db_id_in_job is null then 'Manual_notebook' else 'Job_notebook' end"))\ -# .select(sparkTask["*"] -# ,sparkJob["db_job_id"] -# ,sparkJob["db_id_in_job"] -# ,sparkJob["execution_id"] -# ,sparkJob["job_runtime"] -# ,notebook["notebook_id"] -# ,notebook["notebook_path"] -# ,"Execution_type" -# ,"MemoryBytesSpilled" -# ,"DiskBytesSpilled" -# ) - df = SparkTask_master\ - .transform(helpers.filter_dates(self,"date", self.start_date, self.end_date))\ - .transform(helpers.filter_workspaces(self, self.workspace_name))\ - .transform(helpers.filter_by_weekdays(self, self.include_weekend, self.only_weekend))\ - .transform(helpers.partition_split(self, self.path_depth, self.consumer_db)) - return df - - def job_master_filter(self,**kwargs): - self.cluster_id = kwargs.get("clusterID","all") - self.tags = kwargs.get("tags","all") - self.include_weekend = kwargs.get("includeWeekend",True) - self.only_weekend = kwargs.get("onlyWeekend",False) - self.date_col = kwargs.get("dateColumn",False) - self.cluster_table = kwargs.get("clusterTable",False) - - - jrcp = spark\ - .sql(f"select *,DATE(task_runtime.startTS) as job_start_date from {self.consumer_db}.jobruncostpotentialfact")\ - .withColumn("is_weekend",dayofweek(self.date_col).isin([1,7]).cast("int")) - - job = spark\ - .sql(f"select * from {self.consumer_db}.job")\ - .select("job_id","tasks.notebook_task.notebook_path","created_by") - - jobrun = spark\ - .sql(f"select * from {self.consumer_db}.jobRun") - - - jrcp_master = jrcp\ - .join(jobrun, jrcp["run_id"] == jobrun["run_id"], "inner")\ - .join(job, jrcp["job_id"] == job["job_id"], "inner")\ - .select(jrcp["*"], - job["notebook_path"], - jobrun["cluster_type"] - )\ - .transform(helpers.filter_dates(self,self.date_col,self.start_date,self.end_date))\ - .transform(helpers.filter_workspaces(self,self.workspace_name))\ - .transform(helpers.filter_by_weekdays(self,self.include_weekend,self.only_weekend))\ - .transform(helpers.filter_clusters(self,self.cluster_table))\ - .select("organization_id", "workspace_name","job_start_date","job_id","run_id","job_name" - ,"task_runtime.startTS","task_runtime.endTS","task_runtime.runTimeH","cluster_id","cluster_name","cluster_type" - ,"terminal_state","worker_potential_core_H","total_compute_cost","task_type" - ,"total_dbu_cost","total_cost","is_weekend","notebook_path","created_by","last_edited_by","job_run_cluster_util") - return jrcp_master - - def expensive_jobs(self, **kwargs): - self.dataframe = kwargs.get("data",True) - expensive_jobs = self.dataframe\ - .groupby("job_name","job_start_date","workspace_name")\ - .agg(round(sum(col("total_dbu_cost")),2).alias("cost_in_USD"))\ - .orderBy(col("cost_in_USD").desc()) - - windowDate = Window.partitionBy("job_start_date").orderBy(col("cost_in_USD").desc()) - - top3_expensive_jobs = expensive_jobs.withColumn("row",row_number().over(windowDate)) - - top3expensive_jobs = top3_expensive_jobs.where(top3_expensive_jobs.row <=3)\ - .sort(["job_start_date","cost_in_USD"], ascending=False)\ - .sort(["row"], ascending=False) - - top3expensive_jobs_daily = top3expensive_jobs\ - .select(concat_ws(' : ',top3expensive_jobs.workspace_name,top3expensive_jobs.job_name,top3expensive_jobs.cost_in_USD)\ - .alias("Expensive_Jobs"),"job_start_date")\ - .sort(["job_start_date"], ascending=False)\ - .groupby("job_start_date")\ - .agg(F.collect_list("Expensive_Jobs").alias("top_3_expensive_jobs")) - - return top3expensive_jobs_daily - - def expensive_jobs_interactive_clusters(self,dataframe): - data = self.dataframe - expensive_int_jobs = self.dataframe\ - .filter(col("cluster_type") != "job_cluster")\ - .groupby("job_name","job_start_date","workspace_name")\ - .agg(round(sum(col("total_dbu_cost")),2).alias("cost_in_USD"))\ - .orderBy(col("cost_in_USD").desc()) - - windowDate = Window.partitionBy("job_start_date").orderBy(col("cost_in_USD").desc()) - - top3_expensive_int_jobs = expensive_int_jobs.withColumn("row",row_number().over(windowDate)) - - top3expensive_int_jobs = top3_expensive_int_jobs.where(top3_expensive_int_jobs.row <=3)\ - .sort(["job_start_date","cost_in_USD"], ascending=False)\ - .sort(["row"], ascending=False) - - top3expensive_jobs_int_daily = top3expensive_int_jobs\ - .select(concat_ws(' : ', - top3expensive_int_jobs.workspace_name, - top3expensive_int_jobs.job_name, - top3expensive_int_jobs.cost_in_USD)\ - .alias("Expensive_Jobs"),"job_start_date")\ - .sort(["job_start_date"], ascending=False)\ - .groupby("job_start_date")\ - .agg(F.collect_list("Expensive_Jobs").alias("top3expensive_int_jobs")) - - return top3expensive_jobs_int_daily - - def expensive_failure(self,dataframe): - data = self.dataframe - expensive_job_faliures = data\ - .where(col("terminal_state") == "Failed")\ - .groupby("job_name","job_start_date","workspace_name")\ - .agg(round(sum(col("total_dbu_cost")),2).alias("cost_in_USD"))\ - .orderBy(col("cost_in_USD").desc()) - windowDate = Window.partitionBy("job_start_date").orderBy(col("cost_in_USD").desc()) - - top3_expensive_fails= expensive_job_faliures.withColumn("row",row_number().over(windowDate)) - - top3_expensive_job_fails = top3_expensive_fails.where(top3_expensive_fails.row <=3) - - top3_expensive_job_faliures_daily = top3_expensive_job_fails\ - .select(concat_ws(' : ', - top3_expensive_job_fails.workspace_name, - top3_expensive_job_fails.job_name, - top3_expensive_job_fails.cost_in_USD)\ - .alias("Expensive_Jobs"),"job_start_date")\ - .sort(["job_start_date"], ascending=False)\ - .groupby("job_start_date")\ - .agg(F.collect_list("Expensive_Jobs").alias("top3_expensive_job_fails")) - return top3_expensive_job_faliures_daily - - - - def cluster_master_filter(self,**kwargs): - self.include_weekend = kwargs.get("includeWeekend",True) - self.only_weekend = kwargs.get("onlyWeekend",False) - - clusterstatefact = spark.sql(f"select * from {self.consumer_db}.clusterstatefact") - - - cluster = spark.sql(f"select * from {self.consumer_db}.cluster") - - clsf_master = clusterstatefact.join(cluster, clusterstatefact["cluster_id"] == cluster["cluster_id"], "inner")\ - .withColumn("cluster_category", - expr("""case when is_automated = 'true' and cluster_type not in ('Serverless','SQL Analytics','Single Node') then 'Automated' - when clusterstatefact.cluster_name like "dlt%" or cluster_type = 'Standard' then 'Standard' - when is_automated = 'false' and cluster_type not in ('Serverless','SQL Analytics','Single Node') then 'Interactive' - when (is_automated = 'false' or is_automated = 'true' or is_automated is null) and cluster_type = 'SQL Analytics' then 'Warehouse' - when (is_automated = 'false' or is_automated = 'true' or is_automated is null) and cluster_type = 'Serverless' then 'High-Concurrency' - when (is_automated = 'false' or is_automated = 'true' or is_automated is null) and cluster_type = 'Single Node' then 'Single Node' - else "Unidentified" - end"""))\ - .select(clusterstatefact["*"] - ,cluster["created_by"] - ,cluster["last_edited_by"] - ,cluster["deleted_by"] - ,cluster["driver_node_type"] - ,cluster["node_type"].alias("worker_node_type") - ,cluster["autoscale"] - ,cluster["is_automated"] - ,cluster["cluster_type"] - ,cluster["auto_termination_minutes"] - ,cluster["instance_pool_id"] - ,cluster["instance_pool_name"] - ,"cluster_category" - )\ - .withColumn("is_weekend",dayofweek("state_start_date").isin([1,7]).cast("int"))\ - .withColumn('SqlEndpointId', json_tuple(col("custom_tags"), "SqlEndpointId"))\ - .transform(helpers.filter_dates(self,"state_start_date",self.start_date,self.end_date))\ - .transform(helpers.filter_workspaces(self,self.workspace_name))\ - .transform(helpers.filter_by_weekdays(self,self.include_weekend,self.only_weekend)) - - return clsf_master - - def job_test_filter(self,**kwargs): - self.cluster_id = kwargs.get("clusterID","all") - self.tags = kwargs.get("tags","all") - self.include_weekend = kwargs.get("includeWeekend",True) - self.only_weekend = kwargs.get("onlyWeekend",False) - self.date_col = kwargs.get("dateColumn",False) - self.cluster_table = kwargs.get("clusterTable",False) - - - jrcp = spark\ - .sql(f"select *,DATE(task_runtime.startTS) as job_start_date from {self.consumer_db}.jobruncostpotentialfact")\ - .withColumn("is_weekend",dayofweek(self.date_col).isin([1,7]).cast("int")) - - job = spark\ - .sql(f"select * from {self.consumer_db}.job")\ - .select("job_id","tasks.notebook_task.notebook_path","created_by") - - jobrun = spark\ - .sql(f"select * from {self.consumer_db}.jobRun") - - - jrcp_master = jrcp\ - .join(jobrun, jrcp["run_id"] == jobrun["run_id"], "inner")\ - .join(job, jrcp["job_id"] == job["job_id"], "inner")\ - .select(jrcp["*"], - job["notebook_path"], - jobrun["cluster_type"] - )\ - .transform(helpers.filter_dates(self,self.date_col,self.start_date,self.end_date))\ - .transform(helpers.filter_workspaces(self,self.workspace_name))\ - .transform(helpers.filter_by_weekdays(self,self.include_weekend,self.only_weekend))\ - .select("organization_id", "workspace_name","job_start_date","job_id","run_id","job_name" - ,"task_runtime.startTS","task_runtime.endTS","task_runtime.runTimeH","cluster_id","cluster_name","cluster_type" - ,"terminal_state","worker_potential_core_H","total_compute_cost" - ,"total_dbu_cost","total_cost","is_weekend","notebook_path","created_by","last_edited_by","job_run_cluster_util", "job_trigger_type") -# .transform(helpers.filter_clusters(self,self.cluster_table))\ -# .select("organization_id", "workspace_name","job_start_date","job_id","run_id","job_name" -# ,"task_runtime.startTS","task_runtime.endTS","task_runtime.runTimeH","cluster_id","cluster_name","cluster_type" -# ,"terminal_state","worker_potential_core_H","total_compute_cost" -# ,"total_dbu_cost","total_cost","is_weekend","notebook_path","created_by","last_edited_by","job_run_cluster_util") - return jrcp_master - -# COMMAND ---------- - -# def job_master_filter(self,**kwargs): -# self.cluster_id = kwargs.get("clusterID","all") -# self.tags = kwargs.get("tags","all") -# self.include_weekend = kwargs.get("includeWeekend",True) -# self.only_weekend = kwargs.get("onlyWeekend",False) -# self.date_col = kwargs.get("dateColumn",False) -# self.cluster_table = kwargs.get("clusterTable",False) - - -# jrcp = spark\ -# .sql(f"select *,DATE(task_runtime.startTS) as job_start_date from {self.consumer_db}.jobruncostpotentialfact")\ -# .withColumn("is_weekend",dayofweek(self.date_col).isin([1,7]).cast("int")) - -# job = spark\ -# .sql(f"select * from {self.consumer_db}.job")\ -# .select("job_id","tasks.notebook_task.notebook_path","created_by") - -# jobrun = spark\ -# .sql(f"select * from {self.consumer_db}.jobRun") - - -# jrcp_master = jrcp\ -# .join(jobrun, jrcp["run_id"] == jobrun["run_id"], "inner")\ -# .join(job, jrcp["job_id"] == job["job_id"], "inner")\ -# .select(jrcp["*"], -# job["notebook_path"], -# jobrun["cluster_type"] -# )\ -# .transform(helpers.filter_dates(self,self.date_col,self.start_date,self.end_date))\ -# .transform(helpers.filter_workspaces(self,self.workspace_name))\ -# .transform(helpers.filter_by_weekdays(self,self.include_weekend,self.only_weekend))\ -# .transform(helpers.filter_clusters(self,self.cluster_table))\ -# .select("organization_id", "workspace_name","job_start_date","job_id","run_id","job_name" -# ,"task_runtime.startTS","task_runtime.endTS","task_runtime.runTimeH","cluster_id","cluster_name","cluster_type" -# ,"terminal_state","worker_potential_core_H","total_compute_cost" -# ,"total_dbu_cost","total_cost","is_weekend","notebook_path","created_by","last_edited_by","job_run_cluster_util") -# return jrcp_master - -# def expensive_jobs(self): -# jrcp_master = self.job_master_filter -# expensive_jobs = jrcp_master\ -# .groupby("job_name","job_start_date")\ -# .agg(round(sum(col("total_dbu_cost")),2).alias("cost_in_USD"))\ -# .orderBy(col("cost_in_USD").desc()).distinct() - -# windowDate = Window.partitionBy("job_start_date").orderBy(col("cost_in_USD").desc()) - -# top3_expensive_jobs = expensive_jobs.withColumn("row",row_number().over(windowDate)) - -# top3expensive_jobs = top3_expensive_jobs.where(top3_expensive_jobs.row <=3)\ -# .sort(["job_start_date","cost_in_USD"], ascending=False)\ -# .sort(["row"], ascending=False)\ -# .withColumnRenamed("job_start_date","start_date") - -# top3expensive_jobs_daily = top3expensive_jobs\ -# .select(concat_ws(' : ',top3expensive_jobs.job_name,top3expensive_jobs.cost_in_USD)\ -# .alias("Expensive_Jobs"),"start_date","cost_in_USD")\ -# .sort(["start_date","cost_in_USD"], ascending=False)\ -# .groupby("start_date")\ -# .agg(F.collect_list("Expensive_Jobs").alias("top_3_expensive_jobs"))\ - -# return top3expensive_jobs_daily - -# def expensive_failure(self): -# jrcp_master = self.job_master_filter -# expensive_job_faliures = jrcp_master\ -# .where(col("terminal_state") == "Failed")\ -# .groupby("job_name","job_start_date")\ -# .agg(round(sum(col("total_dbu_cost")),2).alias("cost_in_USD"))\ -# .orderBy(col("cost_in_USD").desc()).distinct() - -# top3_expensive_faliures = expensive_job_faliures.withColumn("row",row_number().over(windowDate)) - -# top3_expensive_job_faliures = top3_expensive_faliures.where(top3_expensive_faliures.row <=3)\ -# .sort(["job_start_date","cost_in_USD"], ascending=False)\ -# .sort(["row"], ascending=False)\ -# .withColumnRenamed("job_start_date","date") - -# top3_expensive_job_faliures_daily = top3_expensive_job_faliures\ -# .select(concat_ws(' : ',top3_expensive_job_faliures.job_name -# ,top3_expensive_job_faliures.cost_in_USD)\ -# .alias("Expensive_Jobs"),"date","cost_in_USD")\ -# .sort(["date","cost_in_USD"], ascending=False)\ -# .groupby("date")\ -# .agg(F.collect_list("Expensive_Jobs").alias("top_3_expensive_faliure_jobs")) -# return top3_expensive_job_faliures_daily \ No newline at end of file diff --git a/modules/adb-overwatch-analysis/notebooks/Jobs.py b/modules/adb-overwatch-analysis/notebooks/Jobs.py deleted file mode 100644 index 087375b5..00000000 --- a/modules/adb-overwatch-analysis/notebooks/Jobs.py +++ /dev/null @@ -1,379 +0,0 @@ -# Databricks notebook source -# MAGIC %md -# MAGIC # Read me -# MAGIC > -# MAGIC - **Replace the ETL DB and Consumer DB with your ETL database name and Consumer database name** -# MAGIC -# MAGIC - **As default the data will be filter:** -# MAGIC - For 30days period -# MAGIC - All workspaces included -# MAGIC - All clusters included -# MAGIC - Weekdays and Weekends included -# MAGIC -# MAGIC Widgets Used: -# MAGIC | # | Widgets | Value | Default -# MAGIC | ----------- | ----------- | ----------- | ----------- | -# MAGIC | 1 | ETL Database Name | Your ETL Database Name | None -# MAGIC | 2 | Consumer DB Name | Your Consumer Database Name | None -# MAGIC | 3 | Workspace Name | List of workspace (overwatch deployed) name | all -# MAGIC | 4 | Cluster | List of clusters in the above workspaces | all -# MAGIC | 5 | Job Tags | Custom tags of jobs | None -# MAGIC | 6 | Start Date | Start date for analysis | 30 days back from current -# MAGIC | 7 | End Date | End date for analysis | Current date -# MAGIC | 8 | Include weekends | To record all days, include weekends | Yes | -# MAGIC | 9 | Only weekends | To record only weekends | No | -# MAGIC > -# MAGIC - **Use the widgets to filter the data further** - -# COMMAND ---------- - -# MAGIC %md -# MAGIC - **%run in *cmd 2* replace the path with your path (*""/Dashboards_Dev/In Progress/07x/Helpers""*) to Helpers notebook** -# MAGIC - **Go to view on topbar and click the *View* under *Dashboard* to view the dashboards named:Job_overview or Job_micro_view** - -# COMMAND ---------- - -# MAGIC %run "/Dashboards_Dev/In Progress/07x_rc_customer/Helpers" $etlDB = etlDB $consumerDB = consumerDB - -# COMMAND ---------- - -# Run only for the first time and comment it out after the first run -# dbutils.widgets.removeAll() - -# COMMAND ---------- - -dbutils.widgets.text("etlDB", "overwatch_etl", "1.ETL DB") -dbutils.widgets.text("consumerDB", "overwatch", "2. Consumer DB") - -# COMMAND ---------- - -etlDB = str(dbutils.widgets.get("etlDB")) -consumerDB = str(dbutils.widgets.get("consumerDB")) - -# COMMAND ---------- - -fetch_workspace_name = spark.sql(f"select distinct workspace_name from {etlDB}.pipeline_report").rdd.flatMap(lambda x: x).collect()+["all"] - -# COMMAND ---------- - -dbutils.widgets.multiselect("workspace_name","all",fetch_workspace_name, "3. Workspace Name") -dbutils.widgets.text("cluster_id", "all", "4. Cluster ID") -dbutils.widgets.text("tags", "", "5. Job Tags") -dbutils.widgets.text("start_date", f"{date.today() - timedelta(days=30)}", "6. Start Date") -dbutils.widgets.text("end_date", f"{date.today()}", "7. End Date") -dbutils.widgets.dropdown("include_weekends", "Yes", ["Yes", "No"], "8. Include weekends") -dbutils.widgets.dropdown("only_weekends", "No", ["Yes", "No"], "9. Only weekends") - -workspace_name = spark.sql(f"select distinct workspace_name from {etlDB}.pipeline_report").rdd.flatMap(lambda x: x).collect() if 'all' in dbutils.widgets.get("workspace_name").split(',') else dbutils.widgets.get("workspace_name").split(',') - -# COMMAND ---------- - -job_tags = str(dbutils.widgets.get("tags")) -start_date = str(dbutils.widgets.get("start_date")) -end_date = str(dbutils.widgets.get("end_date")) -include_weekends = dbutils.widgets.get("include_weekends") -only_weekends = dbutils.widgets.get("only_weekends") -cluster_filter = spark.sql(f"select distinct cluster_id from {consumerDB}.jobrun").select("cluster_id").distinct() if 'all' in dbutils.widgets.get("cluster_id").split(',') else spark.createDataFrame(dbutils.widgets.get("cluster_id").replace(" ", "").split(','),"string").toDF("cluster_id") - -# COMMAND ---------- - -# MAGIC %md -# MAGIC ### Load Data - -# COMMAND ---------- - -# MAGIC %md -# MAGIC ### Master dataframe - -# COMMAND ---------- - -masters = master(etlDB,consumerDB,workspace_name,start_date,end_date) - -# COMMAND ---------- - -job = masters.job_master_filter(includeWeekend = include_weekends, - onlyWeekend = only_weekends, - dateColumn="job_start_date", - clusterTable = cluster_filter)\ - .distinct() -expensive_jobs = masters.expensive_jobs(data=job) -expensive_faliures = masters.expensive_failure(job) -expensive_jobs_int_clusters = masters.expensive_jobs_interactive_clusters(job) - -# COMMAND ---------- - -# MAGIC %md -# MAGIC # Macro View Begin - -# COMMAND ---------- - -# MAGIC %md -# MAGIC ## $DBUs by workflow by workspace by date -# MAGIC #### In the below dataframe and visulization: -# MAGIC ###### total_dbu_cost :- is the DBU spend in USD for the day per workspace -# MAGIC ###### top_3_expensive_jobs :- Provides the list of three top expensive jobs of the day across workspaces and this is by $DBUs by date. -# MAGIC ###### top_3_expensive_job_fails -- Provides the list of three top expensive jobs of the day across workspaces and this is by $DBUs by date. - -# COMMAND ---------- - -job_cost = job\ - .groupBy("job_start_date","workspace_name")\ - .agg(round(sum(col("total_dbu_cost")),2).alias("total_dbu_cost")) - -job_cost_master = job_cost\ - .join(expensive_jobs, ['job_start_date'])\ - .join(expensive_faliures, ['job_start_date']) - -dbu_cost = job_cost_master\ - .groupBy("job_start_date","workspace_name","top_3_expensive_jobs","top3_expensive_job_fails")\ - .agg(round(sum(col("total_dbu_cost")),2).alias("total_dbu_cost"))\ - .orderBy(col("job_start_date").asc())\ - .withColumn("top_1_expensive_job",col("top_3_expensive_jobs")[0])\ - .withColumn("top_2_expensive_job",col("top_3_expensive_jobs")[1])\ - .withColumn("top_3_expensive_job",col("top_3_expensive_jobs")[2])\ - .withColumn("top_1_expensive_fails",col("top3_expensive_job_fails")[0])\ - .withColumn("top_2_expensive_fails",col("top3_expensive_job_fails")[1])\ - .withColumn("top_3_expensive_fails",col("top3_expensive_job_fails")[2])\ - .withColumn('50%', F.expr('percentile(total_dbu_cost, 0.5)').over(Window.partitionBy('job_start_date')))\ - .withColumn('90%', F.expr('percentile(total_dbu_cost, 0.9)').over(Window.partitionBy('job_start_date')))\ - .withColumn('99%', F.expr('percentile(total_dbu_cost, 0.99)').over(Window.partitionBy('job_start_date')))\ - .withColumn('max', F.expr('percentile(total_dbu_cost, 1)').over(Window.partitionBy('job_start_date')))\ - .toPandas() -#compute -#filters job type -# jobs which are not runnu=ing for a period of time - -try: - display(job_cost_master) - fig = px.bar(dbu_cost - ,x=dbu_cost["job_start_date"] - ,y=dbu_cost["total_dbu_cost"] - ,hover_data=["top_1_expensive_job","top_2_expensive_job","top_3_expensive_job", - "top_1_expensive_fails","top_2_expensive_fails","top_3_expensive_fails", - "50%","90%","99%","max"] - ,color = "workspace_name" - ,color_discrete_sequence = px.colors.sequential.Electric - ,title="$DBUs by workflow by workspace by date" - ,labels={'job_start_date':'Job Start Date' - ,'total_dbu_cost':'Total DBU cost in USD(Daily/Workspace)' - ,'workspace_name':'Workspace Name'} - ) - - fig.update_layout(showlegend=True - ,xaxis_type='category') - - fig.show() -except ValueError: - print("Its an empty dataframe - Kindly check the job_cost_master dataframe") -except Exception as e: - print(f"An exception occurred : {e}") - -# COMMAND ---------- - -# MAGIC %md -# MAGIC ## Job Count by workspace - -# COMMAND ---------- - -job_count = job\ - .groupBy("workspace_name")\ - .agg(countDistinct("job_id").alias("job_count"), - round(sum((col("total_dbu_cost"))),2).alias("total_dbu_cost_USD"))\ - .toPandas() - -minimum_job_count = int(job_count["job_count"].sum()*0.2) -#The value collects 20% of total number jobs, any Workspace with job count less than this value will go to other category -job_count.loc[job_count['job_count'] < minimum_job_count,'workspace_name'] = 'Other Workspaces' -try: - fig = px.pie(job_count, values='job_count', names='workspace_name', - hover_data=['total_dbu_cost_USD'] , - color_discrete_sequence = px.colors.sequential.Bluered, - title='Job Count by workspace') - fig.update_traces(textinfo='percent+label', textfont_size=12,pull=[0.05,0.05,0.05,0.05,0.05]) - fig.show() - -except ValueError: - print("Its an empty dataframe - Kindly check the dataframe") -except Exception as e: - print(f"An exception occurred : {e}") - -# COMMAND ---------- - -# MAGIC %md -# MAGIC ## Number of jobs running on interactive clusters - -# COMMAND ---------- - -job_int_cost = job\ - .filter(col("cluster_type") != "job_cluster")\ - .groupBy("job_start_date","job_id","workspace_name","organization_id","created_by")\ - .agg(round(sum(col("total_dbu_cost")),2).alias("total_dbu_cost")) - -job_int_cost_master = job_int_cost\ - .join(expensive_jobs_int_clusters, ['job_start_date']) - -jobrun_interactive_cluster = job_int_cost_master\ - .groupby("organization_id","workspace_name","created_by","top3expensive_int_jobs")\ - .agg(countDistinct("job_id").alias("job_on_interactive_count") - ,round(sum(col("total_dbu_cost")),2).alias("total_dbu_cost"))\ - .sort(col("job_on_interactive_count").desc())\ - .withColumn("top_1_expensive_jobs",col("top3expensive_int_jobs")[0])\ - .withColumn("top_2_expensive_jobs",col("top3expensive_int_jobs")[1])\ - .withColumn("top_3_expensive_jobs",col("top3expensive_int_jobs")[2])\ - .fillna(value="Unknown", subset=["created_by"])\ - .limit(20)\ - .toPandas() -try: - display(jobrun_interactive_cluster) - fig = px.box(jobrun_interactive_cluster, x="workspace_name", y="job_on_interactive_count" - ,points="all",height = 650,width=1200 - ,hover_data=['organization_id','created_by','total_dbu_cost',"top_1_expensive_jobs","top_2_expensive_jobs","top_3_expensive_jobs"] - ,title="Jobs running in Interactive Clusters (Top 20 workspaces)" - ,labels={"workspace_name" : "Workspace name", - "job_on_interactive_count": "Jobs running on interactive Clusters(Count)"} - ,color_discrete_sequence = px.colors.sequential.gray) - - fig.update_traces(quartilemethod="linear") - fig.update_layout(hovermode="x" - ,showlegend=True - ,xaxis_type='category' - ,yaxis=dict(type='linear')) - fig.show() -except ValueError: - print("Its an empty dataframe - Kindly check the dataframe") -except Exception as e: - print(f"An exception occurred : {e}") - -# COMMAND ---------- - -# MAGIC %md -# MAGIC ### Jobs per day per status - -# COMMAND ---------- - -job_per_status = job\ - .select("terminal_state","job_start_date","run_id","job_id")\ - .where(col("terminal_state")!="null")\ - .groupby("terminal_state","job_start_date")\ - .agg(countDistinct("run_id").alias("number_of_runs"), - countDistinct("job_id").alias("number_of_jobs"))\ - .orderBy(col("job_start_date"))\ - .toPandas() - -jb_fail = job_per_status[job_per_status.terminal_state=="Failed"] - -jb_success = job_per_status[job_per_status.terminal_state=="Succeeded"] - -jb_cancel = job_per_status[job_per_status.terminal_state=="Cancelled"] - -jb_error = job_per_status[job_per_status.terminal_state=="Error"] -try: - fig = go.Figure() - - fig.add_trace(go.Scatter( - x=jb_success.job_start_date, y=jb_success.number_of_runs, - mode='lines', - line=dict(width=0.5, color='rgb(197, 83, 0)'), - stackgroup='one', - name='Succeeded', - groupnorm='percent' # sets the normalization for the sum of the stackgroup - )) - - fig.add_trace(go.Scatter( - x=jb_fail.job_start_date, y=jb_fail.number_of_runs, - mode='lines', - line=dict(width=0.5, color='rgb(0, 0, 255)'), - name='Failed', - stackgroup='one' - )) - - fig.add_trace(go.Scatter( - x=jb_cancel.job_start_date, y=jb_cancel.number_of_runs, - mode='lines', - line=dict(width=0.5, color='rgb(128,0,0)'), - name='Cancelled', - stackgroup='one' - )) - - fig.add_trace(go.Scatter( - x=jb_error.job_start_date, y=jb_error.number_of_runs, - mode='lines', - line=dict(width=0.5, color='rgb(165,42,42)'), - name='Error', - stackgroup='one' - )) - - fig.update_traces(mode="markers+lines") - fig.update_layout(title="Daily Job status distribution", - xaxis_title="Job start Date", - yaxis_title="Job run count", - legend_title="Terminal States", - hovermode="x unified", - showlegend=True, - xaxis_type='category', - yaxis=dict(type='linear' - ,range=[1, 100])) - fig.show() -except ValueError: - print("Its an empty dataframe - Kindly check the dataframe") -except Exception as e: - print(f"An exception occurred : {e}") - -# COMMAND ---------- - -# MAGIC %md -# MAGIC # Number of job Runs (Succeeded vs Failed) - -# COMMAND ---------- - -job_success_vs_failed = job\ - .select("terminal_state","workspace_name","run_id")\ - .where(col("terminal_state").isin(["Succeeded","Failed","Cancelled"]))\ - .groupby("workspace_name","terminal_state")\ - .agg(countDistinct("run_id").alias("number_of_runs"))\ - .toPandas() -try: - fig = px.bar(job_success_vs_failed, - x=job_success_vs_failed["workspace_name"], - y=job_success_vs_failed["number_of_runs"], - color = "terminal_state", - color_discrete_sequence = px.colors.sequential.Blackbody, - title="Number of job Runs (Succeeded vs Failed)", - labels={'number_of_runs':'Job run Count' - ,'workspace_name':'Workspace name'}, - height=500 - ) - fig.update_xaxes(type='category') - fig.show() -except ValueError: - print("Its an empty dataframe - Kindly check the dataframe") -except Exception as e: - print(f"An exception occurred : {e}") - -# COMMAND ---------- - -# MAGIC %md -# MAGIC ## Cost of failure By Workspace - -# COMMAND ---------- - -job_cost_faliure = job\ - .select("terminal_state","workspace_name","run_id","runTimeH")\ - .where(col("terminal_state") != "Succeeded")\ - .groupby("workspace_name")\ - .agg(countDistinct("run_id").alias("number_of_runs") - ,round(sum("runTimeH"),2).alias("Compute_timeH"))\ - .orderBy(col("number_of_runs").desc())\ - .toPandas() -try: - fig = px.bar(job_cost_faliure, x='workspace_name', y='Compute_timeH', - hover_data=['number_of_runs'], color='Compute_timeH', - color_continuous_scale = ["yellow","green","blue","red"], - title = "Impact of Failure By Workspace", - labels={'Compute_timeH':'Failed Compute time in Hours' - ,'workspace_name':'Workspace name'}, height=400) - fig.show() -except ValueError: - print("Its an empty dataframe - Kindly check the dataframe") -except Exception as e: - print(f"An exception occurred : {e}") \ No newline at end of file diff --git a/modules/adb-overwatch-analysis/notebooks/Notebook.py b/modules/adb-overwatch-analysis/notebooks/Notebook.py deleted file mode 100644 index c00ee29a..00000000 --- a/modules/adb-overwatch-analysis/notebooks/Notebook.py +++ /dev/null @@ -1,871 +0,0 @@ -# Databricks notebook source -# MAGIC %md -# MAGIC ## READ ME -# MAGIC > -# MAGIC - **Overwatch version - 07x** -# MAGIC - **Widgets are added to apply filters to the dashboards** -# MAGIC - **Please add your relevant ETL and consumer databases after the database widget appears** -# MAGIC - **By default, the data is filtered as:** -# MAGIC -# MAGIC | # | Widgets | Value | Default -# MAGIC | ----------- | ----------- | ----------- | ----------- | -# MAGIC | 1 | ETL Database Name | Your ETL Database Name | overwatch_etl -# MAGIC | 2 | Consumer DB Name | Your Consumer Database Name | overwatch -# MAGIC | 3 | Path Depth | Adjust folder/notebook path level | 2 -# MAGIC | 4 | Workspace Name | List of workspace (overwatch deployed) name | all -# MAGIC | 5 | Start Date | Start date for analysis | 30 days from current date -# MAGIC | 6 | End Date | End date for analysis | current date -# MAGIC | 7 | Include weekends | To record all days, include weekends | Yes | -# MAGIC | 8 | Only weekends | To record only weekends | No | -# MAGIC > -# MAGIC - **Use the widgets to apply filters in the dashboards** -# MAGIC - **Once the filters applied, run the helper cmd (*""%run "./Helpers"""*) to reflect the filter in the master dataframe** -# MAGIC - **Go to View on topbar and select the *View* named as Notebook under *Dashboards* to view the plots alone** - -# COMMAND ---------- - -# Removing all the widgets -# dbutils.widgets.removeAll() # Run only for the first time - -# COMMAND ---------- - -# Creating widget for getting database names -dbutils.widgets.text("etlDB", "overwatch_etl", "1. ETL Database Name") -dbutils.widgets.text("consumerDB", "overwatch", "2. Consumer DB Name") - -# COMMAND ---------- - -# MAGIC %md -# MAGIC # Add the database names to the widgets - -# COMMAND ---------- - -# To get the database names from widgets -etlDB = str(dbutils.widgets.get("etlDB")) -consumerDB = str(dbutils.widgets.get("consumerDB")) - -# COMMAND ---------- - -# MAGIC %run "/Dashboards_Dev/In Progress/07x_rc_customer/Helpers" $etlDB = etlDB $consumerDB = consumerDB $folder_level = folder_level - -# COMMAND ---------- - -fetch_Name = spark.sql(f"select distinct workspace_name from {etlDB}.pipeline_report").rdd.flatMap(lambda x: x).collect()+["all"] -dbutils.widgets.multiselect("workspace_name","all",fetch_Name, "4. Workspace Name") - -# COMMAND ---------- - -workspaceName = spark.sql(f"select distinct workspace_name from {etlDB}.pipeline_report").rdd.flatMap(lambda x: x).collect() if 'all' in dbutils.widgets.get("workspace_name").split(',') else dbutils.widgets.get("workspace_name").split(',') - -# COMMAND ---------- - -dbutils.widgets.combobox("5. Start Date", f"{date.today() - timedelta(days=30)}", "") -dbutils.widgets.combobox("6. End Date", f"{date.today()}", "") -dbutils.widgets.combobox("3. Path depth", "2", "") -dbutils.widgets.dropdown("include_weekends", "Yes", ["Yes", "No"], "7. Include weekends") -dbutils.widgets.dropdown("only_weekends", "No", ["Yes", "No"], "8. Only weekends") - -# COMMAND ---------- - -folder_level = int(dbutils.widgets.get("3. Path depth")) -include_weekends = dbutils.widgets.get("include_weekends") -only_weekends = dbutils.widgets.get("only_weekends") -start_date = str(dbutils.widgets.get("5. Start Date")) -end_date = str(dbutils.widgets.get("6. End Date")) - -# COMMAND ---------- - -# MAGIC %md -# MAGIC ### Once the filters applied, run the helper cmd (*""%run "./Helpers"""*) to reflect the filter in the master dataframe - -# COMMAND ---------- - -master = master(etlDB, consumerDB, workspaceName, start_date, end_date) -sparkMaster = master.spark_notebook_master(includeWeekend = include_weekends, onlyWeekend = only_weekends, folder_level = folder_level) - -notebook = spark.sql("select * from {}.notebook".format(consumerDB)) -notebook = notebook.withColumn("folder_path", concat_ws('/', slice(split(col('notebook_path'), '/'), 1, folder_level + 1))) - -# COMMAND ---------- - -# Data Intensive Notebooks (top 40 descending) -# Read + Shuffle + Write GBs (stacked bar) - -nb_throughput = sparkMaster\ -.groupBy(sparkMaster["folder_path"], sparkMaster["organization_id"], sparkMaster["workspace_name"])\ -.agg( - round((( - sum(sparkMaster.task_metrics.ShuffleReadMetrics.LocalBytesRead) - + sum(sparkMaster.task_metrics.ShuffleReadMetrics.RemoteBytesRead) - + sum(sparkMaster.task_metrics.ShuffleReadMetrics.RemoteBytesReadToDisk) - + sum(sparkMaster.task_metrics.ShuffleWriteMetrics.ShuffleBytesWritten) - + sum(sparkMaster.task_metrics.ShuffleWriteMetrics.ShuffleRecordsWritten) - + sum(sparkMaster.task_metrics.ShuffleWriteMetrics.ShuffleWriteTime) - )/1000000000),2).alias("TotalShuffle (GBs)") -, - round((( - sum(sparkMaster.task_metrics.InputMetrics.BytesRead) - + sum(sparkMaster.task_metrics.InputMetrics.RecordsRead) - )/1000000000), 2).alias("TotalReads (GBs)") -, - round((( - sum(sparkMaster.task_metrics.OutputMetrics.BytesWritten) - + (sum(sparkMaster.task_metrics.OutputMetrics.RecordsWritten)) - )/1000000000), 2).alias("TotalWrites (GBs)") -) - -Total_throughput = nb_throughput\ -.withColumn('TotalThroughput (GBs)', round((col("TotalShuffle (GBs)") + col("TotalReads (GBs)") + col("TotalWrites (GBs)")), 2))\ -.where(col("folder_path").isNotNull() & (col("folder_path") != ""))\ -.orderBy(col('TotalThroughput (GBs)').desc())\ -.limit(10)\ -.toPandas() - -fig = px.bar(Total_throughput, - x = "folder_path", - y = ["TotalShuffle (GBs)", "TotalReads (GBs)", "TotalWrites (GBs)"], - hover_data = ["organization_id", "workspace_name"], - title = "Data Throughput For Path Depth") - -fig = fig.update_layout( - xaxis_title = "Path", - yaxis_title = "Throughput (GB/s)", - legend_title = "Task_Metrics", -) - -fig.show() - -# COMMAND ---------- - -display(Total_throughput) - -# COMMAND ---------- - -# MAGIC %md -# MAGIC #### ResultSize - How much data pulled out into the notebook - -# COMMAND ---------- - -# sparkTask resultSize (total result size -- colored by avg result size for tasks with resultSize > 10KB) - -resultSize = sparkMaster\ -.groupBy(sparkMaster["folder_path"], sparkMaster["organization_id"], sparkMaster["workspace_name"])\ -.agg(round(avg(sparkMaster.task_metrics.ResultSize)/1000000,2).alias("ResultSize (MB)"))\ -.where(col("folder_path").isNotNull() & (col("folder_path") != ""))\ -.orderBy(col('ResultSize (MB)').desc())\ -.limit(10)\ -.toPandas() - -fig = px.bar(resultSize, - x = "folder_path", - y = "ResultSize (MB)", - hover_data = ["organization_id", "workspace_name"], - color_continuous_scale = ["green", "red"], - color = "ResultSize (MB)", - title = " Returning a lot of data to the UI (Top 40)") - -fig = fig.update_layout( - xaxis_title = "Path", - yaxis_title = "Total Result Size (MB)", -) - -fig.show() - -# COMMAND ---------- - -display(resultSize) - -# COMMAND ---------- - -# Spark executions (i.e. actions) Count - -sp_execution = sparkMaster\ -.groupBy(sparkMaster["folder_path"], - sparkMaster["organization_id"], - sparkMaster["workspace_name"])\ -.agg(countDistinct(col('execution_id')).alias('Execution_count') - ,round(sum(col('task_runtime.runTimeH')),2).alias("Execution_Runtime_Hrs") - )\ -.where(col("folder_path").isNotNull() & (col("folder_path") != ""))\ -.orderBy(col("Execution_count").desc())\ -.limit(10)\ -.toPandas() - -fig = px.bar(sp_execution, - x = "folder_path", - y = "Execution_count", - color_continuous_scale = ["green", "red"], - color = "Execution_Runtime_Hrs", - hover_data = ["organization_id", "workspace_name"], - title = "Spark Actions (count)") - -fig = fig.update_layout( - xaxis_title = "Path", - yaxis_title = "Count", -) - -fig.show() - -# COMMAND ---------- - -display(sp_execution) - -# COMMAND ---------- - -# largest records (1000s of records / MB) (higher is better -- meaning lower number of rec/mb means larger records) - -nb_records = sparkMaster\ -.groupBy(sparkMaster["folder_path"] - ,sparkMaster["organization_id"] - ,sparkMaster["workspace_name"] - )\ -.agg( - (round((( - sum(sparkMaster.task_metrics.ShuffleReadMetrics.LocalBytesRead) - + sum(sparkMaster.task_metrics.ShuffleReadMetrics.RemoteBytesRead) - + sum(sparkMaster.task_metrics.ShuffleReadMetrics.RemoteBytesReadToDisk) - + sum(sparkMaster.task_metrics.ShuffleWriteMetrics.ShuffleBytesWritten) - + sum(sparkMaster.task_metrics.ShuffleWriteMetrics.ShuffleRecordsWritten) - + sum(sparkMaster.task_metrics.ShuffleWriteMetrics.ShuffleWriteTime) - )/1024)/1000,2)).alias("TotalShuffle (MBs)") -, - (round((( - sum(sparkMaster.task_metrics.InputMetrics.BytesRead) - + sum(sparkMaster.task_metrics.InputMetrics.RecordsRead) - )/1024)/1000, 2)).alias("TotalReads (MBs)") -, - (round((( - sum(sparkMaster.task_metrics.OutputMetrics.BytesWritten) - + (sum(sparkMaster.task_metrics.OutputMetrics.RecordsWritten)) - )/1024)/1000, 2)).alias("TotalWrites (MBs)") -) - -NBlargestRecords = nb_records\ -.withColumn('TotalThroughput (MBs)', round((col("TotalShuffle (MBs)") + col("TotalReads (MBs)") + col("TotalWrites (MBs)")), 2))\ -.where(col("folder_path").isNotNull() & (col("folder_path") != ""))\ -.orderBy(col('TotalThroughput (MBs)').desc())\ -.limit(10)\ -.toPandas() - -fig = px.bar(NBlargestRecords, - x = "folder_path", - y = ["TotalShuffle (MBs)", "TotalReads (MBs)", "TotalWrites (MBs)"], - hover_data = ["organization_id", "workspace_name", "TotalThroughput (MBs)"], - title = "Largest records per path depth") - -fig = fig.update_layout( - xaxis_title = "Path", - yaxis_title = "1000s of records per MB", -) - -fig.show() - -# COMMAND ---------- - -display(NBlargestRecords) - -# COMMAND ---------- - -# Task count by task type (stacked bar of number of task's count group by path) - -cnt_cond = lambda cond: F.sum(F.when(cond, 1).otherwise(0)) - -SparkTask_type = sparkMaster\ -.groupBy(sparkMaster["folder_path"], sparkMaster["organization_id"], sparkMaster["workspace_name"])\ -.agg( - ( - cnt_cond(F.col('task_metrics.InputMetrics.BytesRead') > 0) - + cnt_cond(F.col('task_metrics.InputMetrics.RecordsRead') > 0) - ).alias('InputMetrics_count') - , - ( - cnt_cond(F.col('task_metrics.OutputMetrics.BytesWritten') > 0) - + cnt_cond(F.col('task_metrics.OutputMetrics.RecordsWritten') > 0) - ).alias('OutputMetrics_count') - , - ( - cnt_cond(F.col('task_metrics.ShuffleReadMetrics.RemoteBytesRead') > 0) - + cnt_cond(F.col('task_metrics.ShuffleReadMetrics.RemoteBytesReadToDisk') > 0) - + cnt_cond(F.col('task_metrics.ShuffleReadMetrics.LocalBytesRead') > 0) - + cnt_cond(F.col('task_metrics.ShuffleWriteMetrics.ShuffleBytesWritten') > 0) - + cnt_cond(F.col('task_metrics.ShuffleWriteMetrics.ShuffleRecordsWritten') > 0) - + cnt_cond(F.col('task_metrics.ShuffleWriteMetrics.ShuffleWriteTime') > 0) - ).alias('ShuffleMetrics_count') -) - -SparkTask_typeCount = SparkTask_type\ -.withColumn("Throughput_Count", - SparkTask_type["InputMetrics_count"] - + SparkTask_type["OutputMetrics_count"] - + SparkTask_type["ShuffleMetrics_count"])\ -.where(col("folder_path").isNotNull() & (col("folder_path") != ""))\ -.orderBy(col('Throughput_Count').desc())\ -.limit(10)\ -.toPandas() - -fig = px.bar(SparkTask_typeCount, - x = "folder_path", - y = ["InputMetrics_count", "OutputMetrics_count", "ShuffleMetrics_count"], - hover_data = ["organization_id", "workspace_name"], - title = "Task count by task type") - -fig = fig.update_layout( - xaxis_title = "Path", - yaxis_title = "Task's Count", -) - -fig.show() - -# COMMAND ---------- - -display(SparkTask_typeCount) - -# COMMAND ---------- - -# Notebook Efficiency (most inefficient i.e. sorted -- top 40) -# Large tasks (count of tasks > 400MB) (lower is better) - -spark_largeTask = sparkMaster\ -.groupBy(sparkMaster["folder_path"], sparkMaster["organization_id"], sparkMaster["workspace_name"])\ -.agg( - round((( - sum(sparkMaster.task_metrics.ShuffleReadMetrics.LocalBytesRead) - + sum(sparkMaster.task_metrics.ShuffleReadMetrics.RemoteBytesRead) - + sum(sparkMaster.task_metrics.ShuffleReadMetrics.RemoteBytesReadToDisk) - + sum(sparkMaster.task_metrics.ShuffleWriteMetrics.ShuffleBytesWritten) - + sum(sparkMaster.task_metrics.ShuffleWriteMetrics.ShuffleRecordsWritten) - + sum(sparkMaster.task_metrics.ShuffleWriteMetrics.ShuffleWriteTime) - )/1000000),2).alias("TotalShuffle (MBs)") -, - round((( - sum(sparkMaster.task_metrics.InputMetrics.BytesRead) - + sum(sparkMaster.task_metrics.InputMetrics.RecordsRead) - )/1000000), 2).alias("TotalReads (MBs)") -, - round((( - sum(sparkMaster.task_metrics.OutputMetrics.BytesWritten) - + (sum(sparkMaster.task_metrics.OutputMetrics.RecordsWritten)) - )/1000000), 2).alias("TotalWrites (MBs)") -) - -spark_largeTasks = spark_largeTask\ -.withColumn('TotalThroughput (MBs)', round((col("TotalShuffle (MBs)") + col("TotalReads (MBs)") + col("TotalWrites (MBs)")), 2))\ -.where((col("TotalShuffle (MBs)") > 400) | (col("TotalReads (MBs)") > 400) | (col("TotalWrites (MBs)") > 400))\ -.where(col("folder_path").isNotNull() & (col("folder_path") != ""))\ -.orderBy(col("TotalThroughput (MBs)").asc())\ -.limit(10)\ -.toPandas() - -fig = px.bar(spark_largeTasks, - x = "folder_path", - y = ["TotalShuffle (MBs)", "TotalReads (MBs)", "TotalWrites (MBs)"], - hover_data = ["organization_id", "workspace_name"], - title = " Large Tasks Count (> 400MB)") - -fig = fig.update_layout( - xaxis_title = "Path", - yaxis_title = "Large tasks", - legend_title = "Task_Metrics", -) - -fig.show() - -# COMMAND ---------- - -display(spark_largeTasks) - -# COMMAND ---------- - -# Compute Intensive Notebooks -# Notebooks with longest compute times - -longestNotebooks = sparkMaster\ -.groupBy(sparkMaster["folder_path"], sparkMaster["organization_id"], sparkMaster["workspace_name"], sparkMaster["Execution_type"])\ -.agg(round(sum("task_runtime.runTimeH"),2).alias("total_runtime (hrs)"))\ -.where(col("folder_path").isNotNull() & (col("folder_path") != ""))\ -.orderBy(col("total_runtime (hrs)").desc())\ -.limit(10)\ -.toPandas() - -fig = px.bar(longestNotebooks, - x = "folder_path", - y = "total_runtime (hrs)", - color_continuous_scale = ["green", "red"], - hover_data = ["organization_id", "workspace_name", "Execution_type"], - color = "Execution_type", - title = "Longest Running per path depth") - -fig = fig.update_layout( - xaxis_title = "Path", - yaxis_title = "Execution Runtime (Hrs)", -) - -fig.show() - -# COMMAND ---------- - -display(longestNotebooks) - -# COMMAND ---------- - -# # Jobs Executing on Notebooks (count) - -# JobsNotebook = sparkMaster.join(notebook, notebook.folder_path == sparkMaster.folder_path, "inner")\ -# .where(sparkMaster["db_job_id"].isNotNull() & sparkMaster["db_id_in_job"].isNotNull())\ -# .where((sparkMaster["folder_path"].isNotNull()) & (sparkMaster["folder_path"] != ""))\ -# .groupBy(sparkMaster["folder_path"], sparkMaster["organization_id"], sparkMaster["workspace_name"])\ -# .agg(countDistinct(notebook["notebook_id"]).alias("Notebook_Count"))\ -# .limit(10)\ -# .toPandas() - -# fig = px.bar(JobsNotebook, -# x = "folder_path", -# y = "Notebook_Count", -# color_continuous_scale = ["green", "red"], -# color = "Notebook_Count", -# hover_data = ["organization_id", "workspace_name"], -# title = " No. of notebooks executing via jobs") - -# fig = fig.update_layout( -# xaxis_title = "Path", -# yaxis_title = "Count", -# ) - -# fig.show() - - -# Jobs Executing on Notebooks (count) - -jobrun = spark.sql("select job_id from {}.jobrun where task_type in ('notebook','pipeline','python')".format(consumerDB)) - -jb = jobrun.join(sparkMaster, jobrun['job_id'] == sparkMaster['db_job_id'], "inner") - - -JobsNotebook = jb.join(notebook, notebook.folder_path == jb.folder_path, "inner")\ -.where(jb["db_job_id"].isNotNull() & jb["db_id_in_job"].isNotNull())\ -.where((jb["folder_path"].isNotNull()) & (jb["folder_path"] != ""))\ -.groupBy(jb["folder_path"], jb["organization_id"], jb["workspace_name"])\ -.agg(countDistinct(notebook["notebook_id"]).alias("Notebook_Count"))\ -.limit(10)\ -.toPandas() - -fig = px.bar(JobsNotebook, - x = "folder_path", - y = "Notebook_Count", - color_continuous_scale = ["green", "red"], - color = "Notebook_Count", - hover_data = ["organization_id", "workspace_name"], - title = "No. of notebooks executing via jobs per path depth") - -fig = fig.update_layout( - xaxis_title = "Path", - yaxis_title = "Count", -) - -fig.show() - -# COMMAND ---------- - -display(JobsNotebook) - -# COMMAND ---------- - -# Jobs Executing on Notebooks (count) which is not configured from workflow - -jobrun = spark.sql("select job_id from {}.jobrun where task_type not in ('notebook','pipeline','python')".format(consumerDB)) - -jb1 = jobrun.join(sparkMaster, jobrun['job_id'] == sparkMaster['db_job_id'], "inner") - - -JobsNotebook1 = jb1.join(notebook, notebook.folder_path == jb1.folder_path, "inner")\ -.where(jb1["db_job_id"].isNotNull() & jb1["db_id_in_job"].isNotNull())\ -.where((jb1["folder_path"].isNotNull()) & (jb1["folder_path"] != ""))\ -.groupBy(jb1["folder_path"], jb1["organization_id"], jb1["workspace_name"])\ -.agg(countDistinct(notebook["notebook_id"]).alias("Notebook_Count"))\ -.limit(10)\ -.toPandas() - -fig = px.bar(JobsNotebook1, - x = "folder_path", - y = "Notebook_Count", - color_continuous_scale = ["green", "red"], - color = "Notebook_Count", - hover_data = ["organization_id", "workspace_name"], - title = " No. of notebooks executing via jobs per path depth") - -fig = fig.update_layout( - xaxis_title = "Path", - yaxis_title = "Count", -) - -fig.show() - -# COMMAND ---------- - -# Notebook Efficiency (most inefficient i.e. sorted -- top 40) -# Disk / Memory spill (stacked bar by notebook) (lower is better) (desc) - -NotebookSpills = sparkMaster\ -.groupBy(sparkMaster["folder_path"], sparkMaster["organization_id"], sparkMaster["workspace_name"])\ -.agg((sum(col('MemoryBytesSpilled'))/1000000000).alias("MemorySpilled (GB)") - ,(sum(col('DiskBytesSpilled'))/1000000000).alias("DiskSpilled (GB)") - )\ -.where(col("folder_path") != "") - -NBTotalSpills = NotebookSpills\ -.withColumn("TotalSpills (GB)", (NotebookSpills["MemorySpilled (GB)"] + NotebookSpills["DiskSpilled (GB)"]))\ -.orderBy(col("TotalSpills (GB)").desc())\ -.limit(10)\ -.toPandas() - -fig = px.bar(NBTotalSpills, - x = "folder_path", - y = ["MemorySpilled (GB)","DiskSpilled (GB)"], - hover_data = ["organization_id", "workspace_name"], - title = "Total Spills per path depth") - -fig = fig.update_layout( - xaxis_title = "Path", - yaxis_title = "Total Spills", -) - -fig.show() - -# COMMAND ---------- - -display(NotebookSpills) - -# COMMAND ---------- - -# Notebook Efficiency (most inefficient i.e. sorted -- top 40) -# Processing speed (MB/sec) -- (read+shuffled+written) (mb) / taskRuntime (sec) (higher is better) (P0) - -ProcessSpeed = sparkMaster\ -.groupBy(sparkMaster["folder_path"], sparkMaster["organization_id"], sparkMaster["workspace_name"])\ -.agg( - round((( - sum(sparkMaster.task_metrics.ShuffleReadMetrics.LocalBytesRead) - + sum(sparkMaster.task_metrics.ShuffleReadMetrics.RemoteBytesRead) - + sum(sparkMaster.task_metrics.ShuffleReadMetrics.RemoteBytesReadToDisk) - + sum(sparkMaster.task_metrics.ShuffleWriteMetrics.ShuffleBytesWritten) - + sum(sparkMaster.task_metrics.ShuffleWriteMetrics.ShuffleRecordsWritten) - + sum(sparkMaster.task_metrics.ShuffleWriteMetrics.ShuffleWriteTime) - )/1000000),2).alias("TotalShuffle (MBs)") -, - round((( - sum(sparkMaster.task_metrics.InputMetrics.BytesRead) - + sum(sparkMaster.task_metrics.InputMetrics.RecordsRead) - )/1000000), 2).alias("TotalReads (MBs)") -, - round((( - sum(sparkMaster.task_metrics.OutputMetrics.BytesWritten) - + (sum(sparkMaster.task_metrics.OutputMetrics.RecordsWritten)) - )/1000000), 2).alias("TotalWrites (MBs)") -, - round((sum(sparkMaster.task_runtime.runTimeS)),2).alias("TaskRunTime (sec)") #take nanoseconds -) - -ProcessSpeedDF = ProcessSpeed\ -.withColumn('ProcessSpeed (MB/sec)', - round(((col("TotalShuffle (MBs)") + col("TotalReads (MBs)") + col("TotalWrites (MBs)"))/(col("TaskRunTime (sec)"))), 2) - )\ -.where((col("folder_path") != "") & - (col("ProcessSpeed (MB/sec)")>0) -)\ -.orderBy(col("ProcessSpeed (MB/sec)").asc())\ -.limit(10)\ -.toPandas()cccccbgfbkbrbiucnruhcjligndiutjkjrfffeicubek - - -fig = px.bar(ProcessSpeedDF, - x = "folder_path", - y = "ProcessSpeed (MB/sec)", - hover_data = ["organization_id", "workspace_name"], - title = "Processing Speed (MB/sec)") - -fig = fig.update_layout( - xaxis_title = "Path", - yaxis_title = "ProcessSpeed (MB/sec)" -) - -fig.show() - -# COMMAND ---------- - -display(ProcessSpeedDF) - -# COMMAND ---------- - -Notebook_failedJobs = sparkMaster\ -.where( - (sparkMaster["job_result.Result"] == "JobFailed") - & ((col("task_info.Failed") == True) | (col("task_info.Killed") == True)) - & ((col("task_info.speculative") == False)) - & ((col("folder_path") != "")) - )\ -.withColumn("Failed_Count", when(((col("task_info.Failed") == True) | (col("task_info.Killed") == True)), lit(1)).otherwise(lit(0))) - -JobRuntime = Notebook_failedJobs\ -.groupBy(Notebook_failedJobs["folder_path"] - ,Notebook_failedJobs["organization_id"] - ,Notebook_failedJobs["workspace_name"] - )\ -.agg( - round(avg(Notebook_failedJobs["task_runtime.runTimeH"]), 2).alias("AvgRunTimeH") - , round(sum(Notebook_failedJobs["Failed_Count"]), 2).alias("Failed_Count") - )\ -.orderBy(col("AvgRunTimeH").desc())\ -.limit(10)\ -.toPandas() - -fig = px.bar(JobRuntime, - x = "folder_path", - y = ["AvgRunTimeH", "Failed_Count"], - color = "AvgRunTimeH", - color_continuous_scale = ["green", "red"], - hover_data = ["organization_id", "workspace_name"], - title = "Longest Running Failed Spark Jobs") - -fig = fig.update_layout( - xaxis_title = "Path", - yaxis_title = "Job runtime (hrs)" -) - -fig.show() - -# COMMAND ---------- - -display(JobRuntime) - -# COMMAND ---------- - -# Notebook Efficiency (most inefficient i.e. sorted -- top 40) -# Serde time (stacked bar - ExecutorDeserializeTime + ResultSerializationTime)(minutes) (lower is better) (P0) - -SerdeTime = sparkMaster\ -.where(col("folder_path") != '')\ -.where(col("db_job_id").isNull())\ -.groupBy(sparkMaster["folder_path"], sparkMaster["organization_id"], sparkMaster["workspace_name"])\ -.agg( - round(sum("task_metrics.ExecutorDeserializeTime"), 2).alias("ExecutorDeserializeTime") - ,round(sum("task_metrics.ResultSerializationTime"), 2).alias("ResultSerializationTime") -)\ -.withColumn("Serde_Time (mins)", (col("ExecutorDeserializeTime") + col("ResultSerializationTime")))\ -.orderBy(col("Serde_Time (mins)").desc())\ -.limit(10)\ -.toPandas() - -fig = px.bar(SerdeTime, - x = "folder_path", - y = ["ExecutorDeserializeTime", "ResultSerializationTime"], - hover_data = ["organization_id", "workspace_name", "Serde_Time (mins)"], - title = "Serde time") - -fig = fig.update_layout( - xaxis_title = "Path", - yaxis_title = "Serde_Time (mins)", - legend_title = "Serde Time" -) - -fig.show() - -# COMMAND ---------- - -display(SerdeTime) - -# COMMAND ---------- - -# Most popular (distinct users) notebooks -- top 10 -- bar chart - -DistinctUserNB = sparkMaster\ -.where((col("folder_path") != '') - & (col("db_job_id").isNull()))\ -.groupBy(sparkMaster["folder_path"] - ,sparkMaster["organization_id"] - ,sparkMaster["workspace_name"])\ -.agg( - round(sum("task_runtime.runTimeH"), 2).alias("runTimeH") - ,countDistinct("notebook_id").alias("Notebook_Count") - ,countDistinct("user_email").alias("Distinct_Users") - )\ -.orderBy(col("Distinct_Users").desc())\ -.limit(10)\ -.toPandas() - -fig = px.bar(DistinctUserNB, - x = "folder_path", - y = "Distinct_Users", - color_continuous_scale = ["green", "red"], - color = "runTimeH", - hover_data = ["organization_id", "workspace_name", "Notebook_Count", "runTimeH"], - title = " Most popular (distinct users) notebooks per path depth") - -fig = fig.update_layout( - xaxis_title = "Path", - yaxis_title = "Distinct Users", -) - -fig.show() - -# COMMAND ---------- - -display(DistinctUserNB) - -# COMMAND ---------- - -NBComputeHrs = sparkMaster\ -.where((col("folder_path") != '') - & col("db_job_id").isNull())\ -.groupBy(sparkMaster["organization_id"], sparkMaster["workspace_name"], sparkMaster["date"])\ -.agg( - round(sum("task_runtime.runTimeH"), 2).alias("runTimeH") - )\ -.orderBy(col("runTimeH").desc())\ -.limit(50)\ -.toPandas() - -fig = px.box( - NBComputeHrs, - x = "workspace_name", - y = "runTimeH", - title = "Notebook Compute Hours per path depth", - points = "outliers", - height = 500, - width = 900, - color = "workspace_name") - -fig.show() - -# COMMAND ---------- - -display(NBComputeHrs) - -# COMMAND ---------- - -NBExecutionID = sparkMaster\ -.where((col("folder_path") != '') - & col("db_job_id").isNull())\ -.groupBy(sparkMaster["organization_id"], sparkMaster["folder_path"], sparkMaster["workspace_name"])\ -.agg(countDistinct("execution_id").alias("execution_id"))\ -.where(col("execution_id") > 1)\ -.orderBy(col("execution_id").desc())\ -.limit(10)\ -.toPandas() - -fig = px.bar(NBExecutionID, - x = "folder_path", - y = "execution_id", - color_continuous_scale = ["green", "red"], - color = "execution_id", - hover_data = ["organization_id", "workspace_name"], - title = "Spark Actions (count)") - -fig = fig.update_layout( - xaxis_title = "Path", - yaxis_title = "ExecutionID Count", -) - -fig.show() - -# COMMAND ---------- - -display(NBExecutionID) - -# COMMAND ---------- - -ShuffleExplosion = nb_throughput\ -.where(nb_throughput["folder_path"] != '')\ -.withColumn("Explosion_Ratio", (round((col("TotalWrites (GBs)")/col("TotalReads (GBs)")),2))) - -ExplosionRatio = ShuffleExplosion\ -.where(col("Explosion_Ratio").isNotNull() & (col("Explosion_Ratio") > 0))\ -.orderBy(col("Explosion_Ratio").desc())\ -.limit(10)\ -.toPandas() - -fig = px.bar(ExplosionRatio, - x = "folder_path", - y = "Explosion_Ratio", - color_continuous_scale = ["green", "red"], - color = "Explosion_Ratio", - hover_data = ["organization_id", "workspace_name"], - title = "Largest Shuffle Explosions") - -fig = fig.update_layout( - xaxis_title = "Path", - yaxis_title = "Explosion_Ratio", -) - -fig.show() - -# COMMAND ---------- - -display(ExplosionRatio) - -# COMMAND ---------- - -# MAGIC %md -# MAGIC ## TESTING - -# COMMAND ---------- - -# Jobs Executing on Notebooks (count) - -jobrun = spark.sql("select job_id from {}.jobrun where task_type in ('notebook','pipeline','python')".format(consumerDB)) - -jb = jobrun.join(sparkMaster, jobrun['job_id'] == sparkMaster['db_job_id'], "inner") - -JobsNotebook = jb.join(notebook, notebook.folder_path == jb.folder_path, "inner")\ -.where(jb["db_job_id"].isNotNull() & jb["db_id_in_job"].isNotNull())\ -.where((jb["folder_path"].isNotNull()) & (jb["folder_path"] != ""))\ -.groupBy(jb["folder_path"], jb["organization_id"], jb["workspace_name"])\ -.agg(countDistinct(notebook["notebook_id"]).alias("Notebook_Count"))\ -.limit(10)\ -.toPandas() - -fig = px.bar(JobsNotebook, - x = "folder_path", - y = "Notebook_Count", - color_continuous_scale = ["green", "red"], - color = "Notebook_Count", - hover_data = ["organization_id", "workspace_name"], - title = " No. of notebooks executing via jobs") - -fig = fig.update_layout( - xaxis_title = "Path", - yaxis_title = "Count", -) - -fig.show() - -# COMMAND ---------- - -# Jobs Executing on Notebooks (count) - -jobrun = spark.sql("select job_id from {}.jobrun where task_type not in ('notebook','pipeline','python')".format(consumerDB)) - -jb1 = jobrun.join(sparkMaster, jobrun['job_id'] == sparkMaster['db_job_id'], "inner") - - -JobsNotebook1 = jb1.join(notebook, notebook.folder_path == jb1.folder_path, "inner")\ -.where(jb1["db_job_id"].isNotNull() & jb1["db_id_in_job"].isNotNull())\ -.where((jb1["folder_path"].isNotNull()) & (jb1["folder_path"] != ""))\ -.groupBy(jb1["folder_path"], jb1["organization_id"], jb1["workspace_name"])\ -.agg(countDistinct(notebook["notebook_id"]).alias("Notebook_Count"))\ -.limit(10)\ -.toPandas() - -fig = px.bar(JobsNotebook1, - x = "folder_path", - y = "Notebook_Count", - color_continuous_scale = ["green", "red"], - color = "Notebook_Count", - hover_data = ["organization_id", "workspace_name"], - title = " No. of notebooks executing via jobs") - -fig = fig.update_layout( - xaxis_title = "Path", - yaxis_title = "Count", -) - -fig.show() \ No newline at end of file diff --git a/modules/adb-overwatch-analysis/notebooks/Readme.py b/modules/adb-overwatch-analysis/notebooks/Readme.py deleted file mode 100644 index 7ccebf86..00000000 --- a/modules/adb-overwatch-analysis/notebooks/Readme.py +++ /dev/null @@ -1,40 +0,0 @@ -# Databricks notebook source -# MAGIC %md -# MAGIC # Setup Widgets For Filtering The Data -# MAGIC > -# MAGIC - **Replace the ETL DB and Consumer DB with your ETL database name and Consumer database name** -# MAGIC - **As default the data will be filter:** -# MAGIC - For 30days period -# MAGIC - All workspaces included -# MAGIC - All clusters included -# MAGIC - Weekdays and Weekends included -# MAGIC > -# MAGIC - **Use the widgets to filter the data further** - -# COMMAND ---------- - -# MAGIC %md -# MAGIC ## READ ME -# MAGIC -# MAGIC Widgets Used: -# MAGIC | # | Widgets | Value | Default -# MAGIC | ----------- | ----------- | ----------- | ----------- | -# MAGIC | 1 | ETL Database Name | Your ETL Database Name | None -# MAGIC | 2 | Consumer DB Name | Your Consumer Database Name | None -# MAGIC | 3 | Path Depth | Adjust folder/notebook path level | 4 -# MAGIC | 4 | Workspace Name | List of workspace (overwatch deployed) name | all -# MAGIC | 5 | Start Date | Start date for analysis | 30 days from current date -# MAGIC | 6 | End Date | End date for analysis | current date -# MAGIC | 7 | Include weekends | To record all days, include weekends | Yes | -# MAGIC | 8 | Only weekends | To record only weekends | No | - -# COMMAND ---------- - -# MAGIC %md -# MAGIC ## Libraries Used: -# MAGIC -# MAGIC 1. Plotly -# MAGIC 2. Overwatch Latest Library "com.databricks.labs:overwatch_2.12:latest" - -# COMMAND ---------- - diff --git a/modules/adb-overwatch-analysis/notebooks/Workspace.py b/modules/adb-overwatch-analysis/notebooks/Workspace.py deleted file mode 100644 index ac42934b..00000000 --- a/modules/adb-overwatch-analysis/notebooks/Workspace.py +++ /dev/null @@ -1,855 +0,0 @@ -# Databricks notebook source -# MAGIC %md -# MAGIC # Read Me -# MAGIC > -# MAGIC - **Replace the ETL DB and Consumer DB with your ETL database name and Consumer database name** -# MAGIC - **Workspace names are picked automatically from your overwatch data once completing the run of cmd 5** -# MAGIC - **Start Date and End Date will be picked up with past 30 days by default** -# MAGIC -# MAGIC Widgets Used: -# MAGIC | # | Widgets | Value | Default -# MAGIC | ----------- | ----------- | ----------- | ----------- | -# MAGIC | 1 | ETL Database Name | Your ETL Database Name | None -# MAGIC | 2 | Consumer DB Name | Your Consumer Database Name | None -# MAGIC | 3 | Workspace Name | List of workspace (overwatch deployed) name | all -# MAGIC | 4 | Start Date | Start date for analysis | 30 days prior to the present -# MAGIC | 5 | End Date | End date for analysis | Current Date -# MAGIC | 6 | Include weekends | To record all days, include weekends | Yes | -# MAGIC | 7 | Only weekends | To record only weekends | No | -# MAGIC > -# MAGIC - **Use the widgets to filter the data based on your requirement** -# MAGIC - **Please skip the Azure code (27 & 28) if you are under AWS cloud and vice versa.** - -# COMMAND ---------- - -# Run only for the first time and comment it out after the first run -# dbutils.widgets.removeAll() - -# COMMAND ---------- - -dbutils.widgets.text("etlDB", "", "1. ETL Database Name") -dbutils.widgets.text("consumerDB", "", "2. Consumer DB Name") - -# COMMAND ---------- - -# MAGIC %md -# MAGIC ###Populate Widget Boxes 1 and 2 before proceeding -# MAGIC ##### The following cells must know the Overwatch database names before they can execute. Please populate the cells before continuing. - -# COMMAND ---------- - -etlDB = str(dbutils.widgets.get("etlDB")) -consumerDB = str(dbutils.widgets.get("consumerDB")) - -# COMMAND ---------- - -# MAGIC %run "./Helpers" $etlDB = etlDB $consumerDB = consumerDB - -# COMMAND ---------- - -fetch_Name = spark.sql(f"select distinct workspace_name from {etlDB}.pipeline_report").rdd.flatMap(lambda x: x).collect()+["all"] -dbutils.widgets.multiselect("workspace_name","all",fetch_Name, "4. Workspace Name") - -# COMMAND ---------- - -workspaceName = spark.sql(f"select distinct workspace_name from {etlDB}.pipeline_report").rdd.flatMap(lambda x: x).collect() if 'all' in dbutils.widgets.get("workspace_name").split(',') else dbutils.widgets.get("workspace_name").split(',') - -# COMMAND ---------- - -dbutils.widgets.combobox("5. Start Date", f"{date.today() - timedelta(days=30)}", "") -dbutils.widgets.combobox("6. End Date", f"{date.today()}", "") - -start_date = str(dbutils.widgets.get("5. Start Date")) -end_date = str(dbutils.widgets.get("6. End Date")) - -dbutils.widgets.dropdown("include_weekends", "Yes", ["Yes", "No"], "6. Include weekends") -dbutils.widgets.dropdown("only_weekends", "No", ["Yes", "No"], "7. Only weekends") -include_weekends = dbutils.widgets.get("include_weekends") -only_weekends = dbutils.widgets.get("only_weekends") - -# COMMAND ---------- - -# MAGIC %md -# MAGIC ### Once the filters applied, run the helper cmd (*""%run "./Helpers"""*) to reflect the filter in the master dataframe - -# COMMAND ---------- - -masters = master(etlDB, consumerDB, workspaceName,start_date,end_date) - -# COMMAND ---------- - -cluster_master = masters.cluster_master_filter(includeWeekend = include_weekends,onlyWeekend = only_weekends) -job_master = masters.job_test_filter(includeWeekend = include_weekends, - onlyWeekend = only_weekends, - dateColumn="job_start_date") - -# COMMAND ---------- - -# MAGIC %md -# MAGIC **What is the cost of each workspace ?** - -# COMMAND ---------- - -costByDate = cluster_master\ -.groupBy(["state_start_date", "organization_id", - "workspace_name", "cluster_id"])\ -.agg(round(sum(cluster_master["total_dbu_cost"]/cluster_master["days_in_state"]), 2).alias("cost_by_date"))\ -.groupBy(["state_start_date", "organization_id", - "workspace_name"])\ -.agg(round(sum(col("cost_by_date")), 2).alias("DBU_Cost (USD)"))\ -.orderBy(col('DBU_Cost (USD)').desc()) - -# costByDate.loc[costByDate['DBU_Cost (USD)'] < 3,'workspace_name'] = 'Other Types' - -windowDept = Window.partitionBy("state_start_date").orderBy(col("DBU_Cost (USD)").desc()) -costByDate_p = costByDate.withColumn("row",row_number().over(windowDept)) \ - .filter(col("row") <= 20) - -top20Workspaces = costByDate.join(costByDate_p, ((costByDate.state_start_date == costByDate_p.state_start_date) & (costByDate.organization_id == costByDate_p.organization_id) & (costByDate.workspace_name == costByDate_p.workspace_name)), 'left')\ -.select(costByDate['*'], col('row')) - -top20Workspaces_p = top20Workspaces.withColumn('organization_id', expr("case when row is null then 'Others' else organization_id end"))\ -.withColumn('workspace_name', expr("case when row is null then 'Others' else workspace_name end"))\ -.groupby('state_start_date', 'organization_id', 'workspace_name')\ -.agg(round(sum(col('DBU_Cost (USD)')), 2).alias('DBU_Cost (USD)'))\ -.orderBy(col('DBU_Cost (USD)').desc()) - -# Converting pyspark to pandas for visualization -costByDate_pandas = top20Workspaces_p.toPandas() - - -fig = px.bar(costByDate_pandas, - x = "state_start_date", - y = "DBU_Cost (USD)", - hover_data = ["organization_id", - "workspace_name"], - color = "workspace_name", - color_discrete_sequence = px.colors.sequential.Rainbow, - title = "Daily cluster spend chart") - -# # Visualizing the graph -fig.show() - -# COMMAND ---------- - -# MAGIC %md -# MAGIC **What is the cost of each workspace ?** - -# COMMAND ---------- - -costByOrg = cluster_master\ -.groupBy(["organization_id", - "workspace_name"])\ -.agg(round(sum(cluster_master["total_dbu_cost"]), 2).alias("DBU_Cost (USD)"))\ -.orderBy(col('DBU_Cost (USD)').desc()) - -top20 = costByOrg\ -.withColumn('row', lit('Nan'))\ -.limit(20) - -top20Workspaces = costByOrg.join(top20, top20['organization_id'] == costByOrg['organization_id'], 'left')\ -.select(costByOrg['*'], 'row') - -top20_p = top20Workspaces.withColumn('organization_id', expr("case when row is null then 'Others' else organization_id end"))\ -.withColumn('workspace_name', expr("case when row is null then 'Others' else workspace_name end"))\ -.groupby('organization_id', 'workspace_name')\ -.agg(round(sum(col('DBU_Cost (USD)')), 2).alias('DBU_Cost (USD)'))\ -.orderBy(col('DBU_Cost (USD)').desc()) - -# Converting pyspark to pandas for visualization -costByOrg_pandas = top20_p.toPandas() - - -# Plotting dataframe view using plotly library -fig = px.pie(costByOrg_pandas, - names = "workspace_name", - values = "DBU_Cost (USD)", - color_discrete_sequence = px.colors.sequential.Rainbow, - title = "Cluster spend on each workspace", - hole=.3) - -# # Visualizing the graph -fig.show() - -# COMMAND ---------- - -# MAGIC %md -# MAGIC **What is the dbu and compute cost of each workspace ?** - -# COMMAND ---------- - -costByType = cluster_master\ -.groupBy(["organization_id", - "workspace_name"])\ -.agg(round(sum(cluster_master["total_dbu_cost"]), 2).alias("DBU_Cost (USD)"), - round(sum(cluster_master["total_compute_cost"]), 2).alias("Compute_Cost (USD)"))\ -.orderBy(col('DBU_Cost (USD)').desc()) - -top20 = costByType\ -.withColumn('row', lit('Nan'))\ -.limit(20) - -top20Workspaces = costByType.join(top20, top20['organization_id'] == costByType['organization_id'], 'left')\ -.select(costByType['*'], 'row') - -top20_p = top20Workspaces.withColumn('organization_id', expr("case when row is null then 'Others' else organization_id end"))\ -.withColumn('workspace_name', expr("case when row is null then 'Others' else workspace_name end"))\ -.groupby('organization_id', 'workspace_name')\ -.agg(round(sum(col('DBU_Cost (USD)')), 2).alias('DBU_Cost (USD)'), - round(sum(col('Compute_Cost (USD)')), 2).alias('Compute_Cost (USD)')) - -costMap = top20_p.withColumn("costMap",create_map( - lit("DBU_Cost (USD)"),col("DBU_Cost (USD)"), - lit("Compute_Cost (USD)"),col("Compute_Cost (USD)") - )).drop("DBU_Cost (USD)","Compute_Cost (USD)")\ - .select( - "organization_id", - "workspace_name", - F.explode("costMap").alias("Cost Type", "Cost (USD)"), -)\ -.orderBy(col('DBU_Cost (USD)').desc()) - -# Converting pyspark to pandas for visualization -costMap_pandas = costMap.toPandas() - -# # Plotting dataframe view using plotly library -fig = px.bar(costMap_pandas, - x = "workspace_name", - y = "Cost (USD)", - hover_data = ["organization_id", - "workspace_name"], - color = "Cost Type", - color_discrete_sequence = px.colors.sequential.Blackbody, - title = "DBU Cost vs Compute Cost") - -# # Visualizing the graph -fig.show() - -# COMMAND ---------- - -# MAGIC %md -# MAGIC **What is the cost of each workspace by cluster type?** - -# COMMAND ---------- - -# Obtain the total cost of clusters by category on daily basis on each workspace -costByType = cluster_master\ -.withColumn('cluster_type', - expr("case when isAutomated = 'false' and SqlEndpointId is null then 'interactive'" - + "when isAutomated = 'true' then 'automated'" - + "when SqlEndpointId is not null then 'SQL' else cluster_name end"))\ -.groupBy(["state_start_date", - "organization_id", - "workspace_name", - "cluster_category", - "cluster_id"])\ -.agg(round(sum(cluster_master["total_dbu_cost"]/cluster_master["days_in_state"]), 2).alias("cost_by_date"))\ -.groupBy(["state_start_date", - "organization_id", - "workspace_name", - "cluster_category"])\ -.agg(round(sum(col("cost_by_date")), 2).alias("DBU_Cost"), - countDistinct('cluster_id').alias('cluster_count'))\ -.filter(col('cluster_category').isNotNull()) - -# Calculate the worksapce cost on every day -costByType_p = costByType.withColumn('costMap', create_map(col('cluster_category'), col('DBU_Cost')))\ -.withColumn('countMap', create_map(col('cluster_category'), col('cluster_count')))\ -.groupBy(["state_start_date", - "organization_id", - "workspace_name"])\ -.agg(round(sum(col("DBU_Cost")), 2).alias("DBU_Cost (USD)"), - collect_list(col('DBU_Cost')).alias('cost_by_type'), - collect_list(col('costMap')).alias('cost'), - collect_list(col('countMap')).alias('cluster_count'))\ -.orderBy(col('DBU_Cost (USD)').desc()) - -# Limit the workspaces to top 20 costing more -windowDept = Window.partitionBy("state_start_date")\ -.orderBy(col("DBU_Cost (USD)").desc()) - -costByDate_p = costByType_p\ -.withColumn("row",row_number().over(windowDept))\ -.filter(col("row") <= 20) - -# Grouping all the other workspaces apart from top 20 to Others category -top20Workspaces = costByType_p\ -.join(costByDate_p, ((costByType_p.state_start_date == costByDate_p.state_start_date) & (costByType_p.organization_id == costByDate_p.organization_id) & (costByType_p.workspace_name == costByDate_p.workspace_name)), 'left')\ -.select(costByType_p['*'], - col('row')) - -top20Workspaces_p = top20Workspaces\ -.withColumn('organization_id', expr("case when row is null then 'Others' else organization_id end"))\ -.withColumn('workspace_name', expr("case when row is null then 'Others' else workspace_name end"))\ -.select( - "state_start_date", - "organization_id", - "workspace_name", - explode(top20Workspaces.cost).alias("mapClusterTypes"), - "cluster_count" -) - -# Calculate the clusters cost by category -top20 = top20Workspaces_p\ -.withColumn('cluster_type', map_keys(col("mapClusterTypes"))[0])\ -.withColumn('Cost (USD)', map_values(col('mapClusterTypes'))[0])\ -.groupby('state_start_date', - 'organization_id', - 'workspace_name', - 'cluster_type')\ -.agg(round(sum(col('Cost (USD)')), 2).alias('DBU_Cost (USD)'))\ -.orderBy(col('DBU_Cost (USD)').desc()) - - -# Converting pyspark to pandas for visualization -costByType_pandas = top20.toPandas() - -# Plotting dataframe view using plotly library -fig = px.box(costByType_pandas, - x = "workspace_name", - y = "DBU_Cost (USD)", - color = 'cluster_type', hover_data=['state_start_date'], - title = "Cluster spend by type on each workspace", - points = "all") - -# # Visualizing the graph -fig.show() - -# COMMAND ---------- - -# MAGIC %md -# MAGIC **What is the count of clusters breakdown by type on each workspace ?** - -# COMMAND ---------- - -clusterCount = top20Workspaces\ -.withColumn('organization_id', expr("case when row is null then 'Others' else organization_id end"))\ -.withColumn('workspace_name', expr("case when row is null then 'Others' else workspace_name end"))\ -.select( - "state_start_date", - "organization_id", - "workspace_name", - explode(top20Workspaces.cluster_count).alias("mapClusterCount") -) - -clusterCount_p = clusterCount\ -.withColumn('cluster_type', map_keys(col("mapClusterCount"))[0])\ -.withColumn('Cluster Count', map_values(col('mapClusterCount'))[0])\ -.groupby('state_start_date', - 'organization_id', - 'workspace_name', - 'cluster_type')\ -.agg(round(sum(col('Cluster Count')), 2).alias('cluster_count'))\ -.orderBy(col('cluster_count').desc()) - - -# Converting pyspark to pandas for visualization -countByType_pandas = clusterCount_p.toPandas() - -# Plotting dataframe view using plotly library -fig = px.box(countByType_pandas, - x = "workspace_name", - y = "cluster_count", - color = 'cluster_type', hover_data=['state_start_date'], - title = "Cluster count by type on each workspace", - points = "all") - -# # Visualizing the graph -fig.show() - -# COMMAND ---------- - -scheduledJobs = job_master\ -.filter(col('job_trigger_type') == 'cron')\ -.groupBy(["job_start_date", - "organization_id", - "workspace_name"])\ -.agg(countDistinct(col('job_id')).alias('job_count'))\ -.orderBy(col('job_count').desc()) - -windowDept = Window.partitionBy("job_start_date")\ -.orderBy(col("job_count").desc()) - -costByDate_p = scheduledJobs\ -.withColumn("row",row_number().over(windowDept))\ -.filter(col("row") <= 20) - -top20Workspaces = scheduledJobs\ -.join(costByDate_p, ((scheduledJobs.job_start_date == costByDate_p.job_start_date) & (scheduledJobs.organization_id == costByDate_p.organization_id) & (scheduledJobs.workspace_name == costByDate_p.workspace_name)), 'left')\ -.select(scheduledJobs['*'], - col('row')) - -top20Workspaces_p = top20Workspaces\ -.withColumn('organization_id', expr("case when row is null then 'Others' else organization_id end"))\ -.withColumn('workspace_name', expr("case when row is null then 'Others' else workspace_name end"))\ -.groupby('job_start_date', - 'organization_id', - 'workspace_name')\ -.agg(round(sum(col('job_count')), 2).alias('Job Count'))\ -.orderBy(col('Job Count').desc()) - -# Converting pyspark to pandas for visualization -scheduledJobs_pandas = top20Workspaces_p.toPandas() - -# Plotting dataframe view using plotly library -fig = px.box(scheduledJobs_pandas, - x = "workspace_name", - y = "Job Count", - points = 'all', - hover_data=['job_start_date'], - color = "workspace_name", - title = "Count of scheduled jobs on each workspace") - -# # Visualizing the graph -fig.show() - -# COMMAND ---------- - -jobsComputeTime = job_master\ -.filter(col('job_trigger_type') == 'cron')\ -.groupBy(["job_start_date", - "organization_id", - "workspace_name"])\ -.agg(round(sum(job_master['runTimeH']), 2).alias('compute_time'))\ -.orderBy(col('compute_time').desc()) - -windowDept = Window.partitionBy("job_start_date")\ -.orderBy(col("compute_time").desc()) - -costByDate_p = jobsComputeTime\ -.withColumn("row",row_number().over(windowDept))\ -.filter(col("row") <= 20) - -top20Workspaces = jobsComputeTime\ -.join(costByDate_p, ((jobsComputeTime.job_start_date == costByDate_p.job_start_date) & (jobsComputeTime.organization_id == costByDate_p.organization_id) & (jobsComputeTime.workspace_name == costByDate_p.workspace_name)), 'left')\ -.select(jobsComputeTime['*'], - col('row')) - -jobComputeTime = top20Workspaces\ -.withColumn('organization_id', expr("case when row is null then 'Others' else organization_id end"))\ -.withColumn('workspace_name', expr("case when row is null then 'Others' else workspace_name end"))\ -.groupby('job_start_date', - 'organization_id', - 'workspace_name')\ -.agg(round(sum(col('compute_time')), 2).alias('Compute Time (hrs)'))\ -.orderBy(col('Compute Time (hrs)').desc()) - -# Converting pyspark to pandas for visualization -jobsComputeTime_pandas = jobComputeTime.toPandas() - -# Plotting dataframe view using plotly library -fig = px.box(jobsComputeTime_pandas, - x = "workspace_name", - y = "Compute Time (hrs)", - hover_data = ['job_start_date'], - points='all', - color = "workspace_name", - title = "Compute Time of scheduled jobs on each workspace") - -# # Visualizing the graph -fig.show() - -# COMMAND ---------- - -customTags = cluster_master.withColumn("job_id", (cluster_master["cluster_name"].substr(lit(1), instr(col("cluster_name"), 'run')-2)))\ -.withColumn("clusterName", expr("case when cluster_name like '%-run-%' then job_id else cluster_name end"))\ -.withColumn('JobId', json_tuple(col('custom_tags'), 'JobId'))\ -.withColumn('RunName', json_tuple(col('custom_tags'), 'RunName'))\ -.withColumn('KeepAlive', json_tuple(col('custom_tags'), 'KeepAlive'))\ -.withColumn('end_date ', json_tuple(col('custom_tags'), 'end_date '))\ -.withColumn('SqlEndpointId', json_tuple(col('custom_tags'), 'SqlEndpointId'))\ -.withColumn('dbsql-channel', json_tuple(col('custom_tags'), 'dbsql-channel'))\ -.withColumn('databricks-cloud', json_tuple(col('custom_tags'), 'databricks-cloud'))\ -.withColumn('databricks-cloud-priority', json_tuple(col('custom_tags'), 'databricks-cloud-priority'))\ -.withColumn('isTesting', json_tuple(col('custom_tags'), 'isTesting'))\ -.withColumn('test_new_tag', json_tuple(col('custom_tags'), 'test_new_tag'))\ -.withColumn('type', json_tuple(col('custom_tags'), 'type'))\ -.withColumn('OwnerEmail', json_tuple(col('custom_tags'), 'OwnerEmail'))\ -.withColumn('Owner', json_tuple(col('custom_tags'), 'Owner'))\ -.withColumn('cluster_type', json_tuple(col('custom_tags'), 'cluster_type')) - -costMap = customTags.withColumn("tagMap",create_map( - lit("JobId"),col("JobId"), - lit("RunName"),col("RunName"), - lit("KeepAlive"),col("KeepAlive"), - lit("SqlEndpointId"),col("SqlEndpointId"), - lit("dbsql-channel"),col("dbsql-channel"), - lit("databricks-cloud"),col("databricks-cloud"), - lit("databricks-cloud-priority"),col("databricks-cloud-priority"), - lit("isTesting"),col("isTesting"), - lit("test_new_tag"),col("test_new_tag"), - lit("type"),col("type"), - lit("OwnerEmail"),col("OwnerEmail"), - lit("Owner"),col("Owner"), - lit("cluster_type"),col("cluster_type") - ))\ - .select( - "organization_id", - "workspace_name", - "clusterName", - F.explode("tagMap").alias("Tag Type", "Tag Value") -) - -tagCount = costMap.where(col('Tag Value').isNotNull()).groupby("organization_id", "workspace_name", "Tag Type")\ -.agg(countDistinct(col('clusterName')).alias('Tag Count'))\ -.orderBy(col('workspace_name').desc(), col('Tag Count')) - -costByWorkspace = tagCount\ -.withColumn('tagMap', create_map(col('Tag Type'), col('Tag Count')))\ -.groupBy(["organization_id", - "workspace_name"])\ -.agg(sum(col('Tag Count')).alias('workspace_tag_count'), collect_list(col('tagMap')).alias('tagCountByType'))\ -.orderBy(col('workspace_tag_count').desc()) - -windowDept = Window\ -.orderBy(col("workspace_tag_count").desc()) - -top20workspace = costByWorkspace\ -.withColumn("row",row_number().over(windowDept))\ -.filter(col("row") <= 20) - -top20Workspaces = costByWorkspace\ -.join(top20workspace, ((costByWorkspace.organization_id == top20workspace.organization_id) & (costByWorkspace.workspace_name == top20workspace.workspace_name)), 'left')\ -.select(costByWorkspace['*'], - col('row')) - -jobComputeTime = top20Workspaces\ -.withColumn('organization_id', expr("case when row is null then 'Others' else organization_id end"))\ -.withColumn('workspace_name', expr("case when row is null then 'Others' else workspace_name end"))\ -.select( - "organization_id", - "workspace_name", - explode(top20Workspaces.tagCountByType).alias("mapTagTypes") -) -# Calculate the clusters cost by category -top20 = jobComputeTime\ -.withColumn('tag_type', map_keys(col("mapTagTypes"))[0])\ -.withColumn('tag_count', map_values(col('mapTagTypes'))[0])\ -.groupby('organization_id', - 'workspace_name', - 'tag_type')\ -.agg(sum(col('tag_count')).alias('Tag_Count'))\ -.orderBy(col('Tag_Count').desc()) - -tagCount_pandas = top20.toPandas() -new_df = tagCount_pandas.pivot(index='tag_type', columns='workspace_name')['Tag_Count'].fillna(0) - -fig = px.imshow(new_df, - labels=dict(x="Workspace Name", y="Tag Type", color="Tag Count"), - x=new_df.columns, - y=new_df.index, - title='Workspace Tags count by workspace', - aspect='auto') -# fig.update_xaxes(side="top") -fig.show() - -# COMMAND ---------- - -# MAGIC %md -# MAGIC # AZURE ONLY -# MAGIC Please skip the Azure code (24 & 25) if you are under AWS cloud and vice versa. - -# COMMAND ---------- - -try: - - splitByCloud = spark.sql(f"select * from {etlDB}.pipeline_report")\ - .withColumn('Cloud', expr("case when inputConfig.auditLogConfig.azureAuditLogEventhubConfig is null then 'AWS' else 'Azure' end"))\ - .select('organization_id', 'workspace_name', 'Cloud').distinct() - - node_count = cluster_master\ - .join(splitByCloud, cluster_master.organization_id == splitByCloud.organization_id, "left").select(cluster_master['*'], splitByCloud.Cloud)\ - .withColumn("job_id", (cluster_master["cluster_name"].substr(lit(1), instr(col("cluster_name"), 'run')-2)))\ - .withColumn("clusterName", expr("case when cluster_name like '%-run-%' then job_id else cluster_name end"))\ - .withColumn('node_type', cluster_master['worker_node_type'])\ - .where((splitByCloud['Cloud'] == 'Azure') & (col('node_type').isNotNull()))\ - .groupBy(["organization_id", - "workspace_name", - "node_type"])\ - .agg(countDistinct(col('clusterName')).alias('nodeType_count'))\ - .orderBy(col('nodeType_count').desc()) - - countByWorkspace = node_count\ - .withColumn('nodeMap', create_map(col('node_type'), col('nodeType_count')))\ - .groupBy(["organization_id", - "workspace_name"])\ - .agg(sum(col('nodeType_count')).alias('workspace_node_count'), collect_list(col('nodeMap')).alias('nodeCountByType'))\ - .orderBy(col('workspace_node_count').desc()) - - windowDept = Window\ - .orderBy(col("workspace_node_count").desc()) - - top20workspace = countByWorkspace\ - .withColumn("row",row_number().over(windowDept))\ - .filter(col("row") <= 20) - - - top20Workspaces = countByWorkspace\ - .join(top20workspace, ((countByWorkspace.organization_id == top20workspace.organization_id) & (countByWorkspace.workspace_name == top20workspace.workspace_name)), 'left')\ - .select(countByWorkspace['*'], - col('row')) - - jobComputeTime = top20Workspaces\ - .withColumn('organization_id', expr("case when row is null then 'Others' else organization_id end"))\ - .withColumn('workspace_name', expr("case when row is null then 'Others' else workspace_name end"))\ - .select( - "organization_id", - "workspace_name", - explode(top20Workspaces.nodeCountByType).alias("mapNodeTypes") - ) - - # Calculate the clusters cost by category - top20 = jobComputeTime\ - .withColumn('node_type', map_keys(col("mapNodeTypes"))[0])\ - .withColumn('node_count', map_values(col('mapNodeTypes'))[0])\ - .groupby('organization_id', - 'workspace_name', - 'node_type')\ - .agg(sum(col('node_count')).alias('Node Count'))\ - .orderBy(col('Node Count').desc()) - - - new_df_pandas = top20.toPandas() - new_df = new_df_pandas.pivot(index='node_type', columns='workspace_name')['Node Count'].fillna(0) - - fig = px.imshow(new_df, - labels=dict(x="Workspace Name", y="Node Type", color="NodeType Count"), - x=new_df.columns, - y=new_df.index, - aspect='auto', - title='Node type count by azure workspace') -# fig.update_xaxes(side="top") - fig.show() - -except ValueError: - print("Its an empty dataframe - There are no available Azure workspaces data.") -except Exception as e: - print(f"An exception occurred due to no available Azure workspaces data : {e}") - - -# COMMAND ---------- - -try: - - splitByCloud = spark.sql(f"select * from {etlDB}.pipeline_report")\ - .withColumn('Cloud', expr("case when inputConfig.auditLogConfig.azureAuditLogEventhubConfig is null then 'AWS' else 'Azure' end"))\ - .select('organization_id', 'workspace_name', 'Cloud').distinct() - - node_cost = cluster_master\ - .join(splitByCloud, cluster_master.organization_id == splitByCloud.organization_id, "left").select(cluster_master['*'], splitByCloud.Cloud)\ - .withColumn("job_id", (cluster_master["cluster_name"].substr(lit(1), instr(col("cluster_name"), 'run')-2)))\ - .withColumn("clusterName", expr("case when cluster_name like '%-run-%' then job_id else cluster_name end"))\ - .withColumn('node_type', cluster_master['worker_node_type'])\ - .where((splitByCloud['Cloud'] == 'Azure') & (col('node_type').isNotNull()))\ - .groupBy(["organization_id", - "workspace_name", - "node_type"])\ - .agg(round(sum(cluster_master['total_dbu_cost']), 2).alias('DBU Cost(USD)'))\ - .orderBy(col('DBU Cost(USD)').desc()) - - costByWorkspace = node_cost\ - .withColumn('nodeMap', create_map(col('node_type'), col('DBU Cost(USD)')))\ - .groupBy(["organization_id", - "workspace_name"])\ - .agg(sum(col('DBU Cost(USD)')).alias('workspace_node_cost'), collect_list(col('nodeMap')).alias('nodeCostByType'))\ - .orderBy(col('workspace_node_cost').desc()) - - windowDept = Window\ - .orderBy(col("workspace_node_cost").desc()) - - top20workspace = costByWorkspace\ - .withColumn("row",row_number().over(windowDept))\ - .filter(col("row") <= 20) - - top20Workspaces = costByWorkspace\ - .join(top20workspace, ((costByWorkspace.organization_id == top20workspace.organization_id) & (costByWorkspace.workspace_name == top20workspace.workspace_name)), 'left')\ - .select(costByWorkspace['*'], - col('row')) - - jobComputeTime = top20Workspaces\ - .withColumn('organization_id', expr("case when row is null then 'Others' else organization_id end"))\ - .withColumn('workspace_name', expr("case when row is null then 'Others' else workspace_name end"))\ - .select( - "organization_id", - "workspace_name", - explode(top20Workspaces.nodeCostByType).alias("mapNodeTypes") - ) - # Calculate the clusters cost by category - top20 = jobComputeTime\ - .withColumn('node_type', map_keys(col("mapNodeTypes"))[0])\ - .withColumn('node_cost', map_values(col('mapNodeTypes'))[0])\ - .groupby('organization_id', - 'workspace_name', - 'node_type')\ - .agg(sum(col('node_cost')).alias('Node Cost'))\ - .orderBy(col('Node Cost').desc()) - - new_df_pandas = top20.toPandas() - new_df = new_df_pandas.pivot(index='node_type', columns='workspace_name')['Node Cost'].fillna(0) - - fig = px.imshow(new_df, - labels=dict(x="Workspace Name", y="Node Type", color="DBU Cost (USD)"), - x=new_df.columns, - y=new_df.index, - title='Node type cost by azure workspace', - aspect='auto') - # fig.update_xaxes(side="top") - fig.show() - -except ValueError: - print("Its an empty dataframe - There are no available Azure workspaces data.") -except Exception as e: - print(f"An exception occurred due to no available Azure workspaces data : {e}") - -# COMMAND ---------- - -# MAGIC %md -# MAGIC # AWS ONLY - -# COMMAND ---------- - -try: - - splitByCloud = spark.sql(f"select * from {etlDB}.pipeline_report")\ - .withColumn('Cloud', expr("case when inputConfig.auditLogConfig.azureAuditLogEventhubConfig is null then 'AWS' else 'Azure' end"))\ - .select('organization_id', 'workspace_name', 'Cloud').distinct() - - node_count = cluster_master\ - .join(splitByCloud, cluster_master.organization_id == splitByCloud.organization_id, "left").select(cluster_master['*'], splitByCloud.Cloud)\ - .withColumn("job_id", (cluster_master["cluster_name"].substr(lit(1), instr(col("cluster_name"), 'run')-2)))\ - .withColumn("clusterName", expr("case when cluster_name like '%-run-%' then job_id else cluster_name end"))\ - .withColumn('node_type', cluster_master['worker_node_type'])\ - .where((splitByCloud['Cloud'] == 'AWS') & (col('node_type').isNotNull()))\ - .groupBy(["organization_id", - "workspace_name", - "node_type"])\ - .agg(countDistinct(col('clusterName')).alias('nodeType_count'))\ - .orderBy(col('nodeType_count').desc()) - - countByWorkspace = node_count\ - .withColumn('nodeMap', create_map(col('node_type'), col('nodeType_count')))\ - .groupBy(["organization_id", - "workspace_name"])\ - .agg(sum(col('nodeType_count')).alias('workspace_node_count'), collect_list(col('nodeMap')).alias('nodeCountByType'))\ - .orderBy(col('workspace_node_count').desc()) - - windowDept = Window\ - .orderBy(col("workspace_node_count").desc()) - - top20workspace = countByWorkspace\ - .withColumn("row",row_number().over(windowDept))\ - .filter(col("row") <= 1) - - top20Workspaces = countByWorkspace\ - .join(top20workspace, ((countByWorkspace.organization_id == top20workspace.organization_id) & (countByWorkspace.workspace_name == top20workspace.workspace_name)), 'left')\ - .select(countByWorkspace['*'], - col('row')) - - jobComputeTime = top20Workspaces\ - .withColumn('organization_id', expr("case when row is null then 'Others' else organization_id end"))\ - .withColumn('workspace_name', expr("case when row is null then 'Others' else workspace_name end"))\ - .select( - "organization_id", - "workspace_name", - explode(top20Workspaces.nodeCountByType).alias("mapNodeTypes") - ) - # Calculate the clusters cost by category - top20 = jobComputeTime\ - .withColumn('node_type', map_keys(col("mapNodeTypes"))[0])\ - .withColumn('node_count', map_values(col('mapNodeTypes'))[0])\ - .groupby('organization_id', - 'workspace_name', - 'node_type')\ - .agg(sum(col('node_count')).alias('Node Count'))\ - .orderBy(col('Node Count').desc()) - - new_df_pandas = top20.toPandas() - new_df = new_df_pandas.pivot(index='node_type', columns='workspace_name')['Node Count'].fillna(0) - fig = px.imshow(new_df, - labels=dict(x="Workspace Name", y="Node Type", color="NodeType Count"), - x=new_df.columns, - y=new_df.index, - aspect='auto', - title='Node type count by AWS workspace') - # fig.update_xaxes(side="top") - fig.show() - -except ValueError: - print("Its an empty dataframe - There are no available AWS workspaces data.") -except Exception as e: - print(f"An exception occurred due to no available AWS workspaces data : {e}") - - -# COMMAND ---------- - -try: - - splitByCloud = spark.sql(f"select * from {etlDB}.pipeline_report")\ - .withColumn('Cloud', expr("case when inputConfig.auditLogConfig.azureAuditLogEventhubConfig is null then 'AWS' else 'Azure' end"))\ - .select('organization_id', 'workspace_name', 'Cloud').distinct() - - node_cost = cluster_master\ - .join(splitByCloud, cluster_master.organization_id == splitByCloud.organization_id, "left").select(cluster_master['*'], splitByCloud.Cloud)\ - .withColumn("job_id", (cluster_master["cluster_name"].substr(lit(1), instr(col("cluster_name"), 'run')-2)))\ - .withColumn("clusterName", expr("case when cluster_name like '%-run-%' then job_id else cluster_name end"))\ - .withColumn('node_type', cluster_master['worker_node_type'])\ - .where((splitByCloud['Cloud'] == 'AWS') & (col('node_type').isNotNull()))\ - .groupBy(["organization_id", - "workspace_name", - "node_type"])\ - .agg(round(sum(cluster_master['total_dbu_cost']), 2).alias('DBU Cost(USD)'))\ - .orderBy(col('DBU Cost(USD)').desc()) - - costByWorkspace = node_cost\ - .withColumn('nodeMap', create_map(col('node_type'), col('DBU Cost(USD)')))\ - .groupBy(["organization_id", - "workspace_name"])\ - .agg(sum(col('DBU Cost(USD)')).alias('workspace_node_cost'), collect_list(col('nodeMap')).alias('nodeCostByType'))\ - .orderBy(col('workspace_node_cost').desc()) - - windowDept = Window\ - .orderBy(col("workspace_node_cost").desc()) - - top20workspace = costByWorkspace\ - .withColumn("row",row_number().over(windowDept))\ - .filter(col("row") <= 20) - - top20Workspaces = costByWorkspace\ - .join(top20workspace, ((costByWorkspace.organization_id == top20workspace.organization_id) & (costByWorkspace.workspace_name == top20workspace.workspace_name)), 'left')\ - .select(costByWorkspace['*'], - col('row')) - - jobComputeTime = top20Workspaces\ - .withColumn('organization_id', expr("case when row is null then 'Others' else organization_id end"))\ - .withColumn('workspace_name', expr("case when row is null then 'Others' else workspace_name end"))\ - .select( - "organization_id", - "workspace_name", - explode(top20Workspaces.nodeCostByType).alias("mapNodeTypes") - ) - # Calculate the clusters cost by category - top20 = jobComputeTime\ - .withColumn('node_type', map_keys(col("mapNodeTypes"))[0])\ - .withColumn('node_cost', map_values(col('mapNodeTypes'))[0])\ - .groupby('organization_id', - 'workspace_name', - 'node_type')\ - .agg(sum(col('node_cost')).alias('Node Cost'))\ - .orderBy(col('Node Cost').desc()) - - new_df_pandas = top20.toPandas() - new_df = new_df_pandas.pivot(index='node_type', columns='workspace_name')['Node Cost'].fillna(0) - - fig = px.imshow(new_df, - labels=dict(x="Workspace Name", y="Node Type", color="DBU Cost (USD)"), - x=new_df.columns, - y=new_df.index, - title='Node type cost by AWS workspace', - aspect='auto') - # fig.update_xaxes(side="top") - fig.show() - -except ValueError: - print("Its an empty dataframe - There are no available AWS workspaces data.") -except Exception as e: - print(f"An exception occurred due to no available AWS workspaces data : {e}") \ No newline at end of file diff --git a/modules/adb-overwatch-analysis/providers.tf b/modules/adb-overwatch-analysis/providers.tf deleted file mode 100644 index 1d847d2b..00000000 --- a/modules/adb-overwatch-analysis/providers.tf +++ /dev/null @@ -1,7 +0,0 @@ -terraform { - required_providers { - databricks = { - source = "databricks/databricks" - } - } -} \ No newline at end of file diff --git a/modules/adb-overwatch-analysis/variables.tf b/modules/adb-overwatch-analysis/variables.tf deleted file mode 100644 index 74d9dd98..00000000 --- a/modules/adb-overwatch-analysis/variables.tf +++ /dev/null @@ -1,9 +0,0 @@ -variable "rg_name" { - type = string - description = "Resource group name" -} - -variable "overwatch_ws_name" { - type = string - description = "The name of the Overwatch workspace" -} \ No newline at end of file diff --git a/modules/adb-overwatch-main-ws/Makefile b/modules/adb-overwatch-main-ws/Makefile deleted file mode 100644 index 653039d8..00000000 --- a/modules/adb-overwatch-main-ws/Makefile +++ /dev/null @@ -1,7 +0,0 @@ -.PHONY: docs test_docs - -docs: - terraform-docs -c ../../.terraform-docs.yml . - -test_docs: - terraform-docs -c ../../.terraform-docs.yml --output-check . diff --git a/modules/adb-overwatch-main-ws/README.md b/modules/adb-overwatch-main-ws/README.md deleted file mode 100644 index d786189d..00000000 --- a/modules/adb-overwatch-main-ws/README.md +++ /dev/null @@ -1,45 +0,0 @@ -# adb-overwatch-main-ws - -This module either creates a new workspace, or uses an existing one to deploy **Overwatch** - - -## Requirements - -No requirements. - -## Providers - -| Name | Version | -|------|---------| -| [azurerm](#provider\_azurerm) | n/a | -| [databricks.ow-main-ws](#provider\_databricks.ow-main-ws) | n/a | - -## Modules - -No modules. - -## Resources - -| Name | Type | -|------|------| -| [azurerm_databricks_workspace.adb-new-ws](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/databricks_workspace) | resource | -| [azurerm_databricks_workspace.adb-existing-ws](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/data-sources/databricks_workspace) | data source | -| [azurerm_resource_group.rg](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/data-sources/resource_group) | data source | -| [databricks_spark_version.latest_lts](https://registry.terraform.io/providers/databricks/databricks/latest/docs/data-sources/spark_version) | data source | - -## Inputs - -| Name | Description | Type | Default | Required | -|------|-------------|------|---------|:--------:| -| [overwatch\_ws\_name](#input\_overwatch\_ws\_name) | The name of an existing workspace, or the name to use to create a new one for Overwatch | `string` | n/a | yes | -| [rg\_name](#input\_rg\_name) | Resource group name | `string` | n/a | yes | -| [subscription\_id](#input\_subscription\_id) | Azure subscription ID | `string` | n/a | yes | -| [use\_existing\_ws](#input\_use\_existing\_ws) | A boolean that determines to either use an existing Databricks workspace for Overwatch, when it is set to true, or create a new one when it is set to false | `bool` | n/a | yes | - -## Outputs - -| Name | Description | -|------|-------------| -| [adb\_ow\_main\_ws\_url](#output\_adb\_ow\_main\_ws\_url) | Overwatch workspace url | -| [latest\_lts](#output\_latest\_lts) | The latest DBR LTS version | - \ No newline at end of file diff --git a/modules/adb-overwatch-main-ws/main.tf b/modules/adb-overwatch-main-ws/main.tf deleted file mode 100644 index 12243fed..00000000 --- a/modules/adb-overwatch-main-ws/main.tf +++ /dev/null @@ -1,30 +0,0 @@ -data "azurerm_resource_group" "rg" { - name = var.rg_name -} - -// Create a new workspace for Overwatch -resource "azurerm_databricks_workspace" "adb-new-ws" { - count = var.use_existing_ws ? 0 : 1 - name = var.overwatch_ws_name - resource_group_name = data.azurerm_resource_group.rg.name - location = data.azurerm_resource_group.rg.location - sku = "premium" - - tags = { - Environment = "Overwatch" - } -} - -// Use an existing workspace for Overwatch -data "azurerm_databricks_workspace" "adb-existing-ws" { - count = var.use_existing_ws ? 1 : 0 - name = var.overwatch_ws_name - resource_group_name = var.rg_name -} - - -data "databricks_spark_version" "latest_lts" { - provider = databricks.ow-main-ws - long_term_support = true - depends_on = [azurerm_databricks_workspace.adb-new-ws] -} \ No newline at end of file diff --git a/modules/adb-overwatch-main-ws/output.tf b/modules/adb-overwatch-main-ws/output.tf deleted file mode 100644 index e1d861fa..00000000 --- a/modules/adb-overwatch-main-ws/output.tf +++ /dev/null @@ -1,9 +0,0 @@ -output "adb_ow_main_ws_url" { - description = "Overwatch workspace url" - value = var.use_existing_ws ? one(data.azurerm_databricks_workspace.adb-existing-ws[*].workspace_url) : one(azurerm_databricks_workspace.adb-new-ws[*].workspace_url) -} - -output "latest_lts" { - description = "The latest DBR LTS version" - value = data.databricks_spark_version.latest_lts.id -} \ No newline at end of file diff --git a/modules/adb-overwatch-main-ws/providers.tf b/modules/adb-overwatch-main-ws/providers.tf deleted file mode 100644 index ef299183..00000000 --- a/modules/adb-overwatch-main-ws/providers.tf +++ /dev/null @@ -1,16 +0,0 @@ -terraform { - required_providers { - azurerm = { - source = "hashicorp/azurerm" - } - - databricks = { - source = "databricks/databricks" - } - } -} - -provider "databricks" { - alias = "ow-main-ws" - host = var.use_existing_ws ? one(data.azurerm_databricks_workspace.adb-existing-ws[*].workspace_url) : one(azurerm_databricks_workspace.adb-new-ws[*].workspace_url) -} \ No newline at end of file diff --git a/modules/adb-overwatch-main-ws/variables.tf b/modules/adb-overwatch-main-ws/variables.tf deleted file mode 100644 index 8d01ad16..00000000 --- a/modules/adb-overwatch-main-ws/variables.tf +++ /dev/null @@ -1,19 +0,0 @@ -variable "subscription_id" { - type = string - description = "Azure subscription ID" -} - -variable "rg_name" { - type = string - description = "Resource group name" -} - -variable "overwatch_ws_name" { - type = string - description = "The name of an existing workspace, or the name to use to create a new one for Overwatch" -} - -variable "use_existing_ws" { - type = bool - description = "A boolean that determines to either use an existing Databricks workspace for Overwatch, when it is set to true, or create a new one when it is set to false" -} \ No newline at end of file diff --git a/modules/adb-overwatch-mws-config/Makefile b/modules/adb-overwatch-mws-config/Makefile deleted file mode 100644 index 653039d8..00000000 --- a/modules/adb-overwatch-mws-config/Makefile +++ /dev/null @@ -1,7 +0,0 @@ -.PHONY: docs test_docs - -docs: - terraform-docs -c ../../.terraform-docs.yml . - -test_docs: - terraform-docs -c ../../.terraform-docs.yml --output-check . diff --git a/modules/adb-overwatch-mws-config/README.md b/modules/adb-overwatch-mws-config/README.md deleted file mode 100644 index 8d28922f..00000000 --- a/modules/adb-overwatch-mws-config/README.md +++ /dev/null @@ -1,69 +0,0 @@ -# adb-overwatch-mws-config - -This module deploys the required resources for a multi-workspace Overwatch deployment : -- Storage account and container to store Overwatch ETL output -- Role assignment of the SPN to the storage account created above -- Databricks secret scope backed with AKV to store the secrets needed on the main Overwatch workspace -- Databricks mount point to the container created above -- Databricks Overwatch [notebook runner](./notebooks/overwatch-runner.scala) -- Databricks job that will run Overwatch with the notebook above - - -## Requirements - -No requirements. - -## Providers - -| Name | Version | -|------|---------| -| [azuread](#provider\_azuread) | n/a | -| [azurerm](#provider\_azurerm) | n/a | -| [databricks](#provider\_databricks) | n/a | - -## Modules - -No modules. - -## Resources - -| Name | Type | -|------|------| -| [azurerm_role_assignment.data-contributor-role](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/role_assignment) | resource | -| [azurerm_storage_account.ow-sa](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/storage_account) | resource | -| [azurerm_storage_data_lake_gen2_filesystem.overwatch-db](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/storage_data_lake_gen2_filesystem) | resource | -| [databricks_job.overwatch](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/job) | resource | -| [databricks_mount.overwatch_db](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/mount) | resource | -| [databricks_notebook.overwatch_etl](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/notebook) | resource | -| [databricks_secret_scope.overwatch-akv](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/secret_scope) | resource | -| [azuread_service_principal.overwatch-spn](https://registry.terraform.io/providers/hashicorp/azuread/latest/docs/data-sources/service_principal) | data source | -| [azurerm_databricks_workspace.overwatch-ws](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/data-sources/databricks_workspace) | data source | -| [azurerm_key_vault.existing-kv](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/data-sources/key_vault) | data source | -| [azurerm_key_vault_secret.spn-key](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/data-sources/key_vault_secret) | data source | -| [azurerm_resource_group.rg](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/data-sources/resource_group) | data source | - -## Inputs - -| Name | Description | Type | Default | Required | -|------|-------------|------|---------|:--------:| -| [akv\_name](#input\_akv\_name) | Azure Key-Vault name | `string` | n/a | yes | -| [databricks\_secret\_scope\_name](#input\_databricks\_secret\_scope\_name) | Databricks secret scope name (backed by Azure Key-Vault) | `string` | n/a | yes | -| [latest\_dbr\_lts](#input\_latest\_dbr\_lts) | Latest DBR LTS version | `string` | n/a | yes | -| [overwatch\_spn\_app\_id](#input\_overwatch\_spn\_app\_id) | Azure SPN ID used to create the mount points | `string` | n/a | yes | -| [overwatch\_ws\_name](#input\_overwatch\_ws\_name) | Overwatch workspace name | `string` | n/a | yes | -| [ow\_sa\_name](#input\_ow\_sa\_name) | The name of the Overwatch ETL storage account | `string` | n/a | yes | -| [random\_string](#input\_random\_string) | Random string used as a suffix for the resources names | `string` | n/a | yes | -| [rg\_name](#input\_rg\_name) | Resource group name | `string` | n/a | yes | -| [tenant\_id](#input\_tenant\_id) | Azure Tenant ID | `string` | n/a | yes | -| [cron\_job\_schedule](#input\_cron\_job\_schedule) | Cron expression to schedule the Overwatch Job | `string` | `"0 0 8 * * ?"` | no | -| [cron\_timezone\_id](#input\_cron\_timezone\_id) | Timezone for the cron schedule | `string` | `"Europe/Brussels"` | no | -| [overwatch\_job\_notification\_email](#input\_overwatch\_job\_notification\_email) | Overwatch Job Notification Email | `string` | `"email@example.com"` | no | -| [overwatch\_version](#input\_overwatch\_version) | Overwatch library maven version | `string` | `"overwatch_2.12:0.7.1.0"` | no | - -## Outputs - -| Name | Description | -|------|-------------| -| [databricks\_mount\_db\_name](#output\_databricks\_mount\_db\_name) | Mount point name to the storage account where Overwatch will be writing the results | -| [etl\_storage\_prefix](#output\_etl\_storage\_prefix) | Overwatch ETL storage prefix, which represents a mount point to the ETL storage account | - \ No newline at end of file diff --git a/modules/adb-overwatch-mws-config/main.tf b/modules/adb-overwatch-mws-config/main.tf deleted file mode 100644 index 626ce1fa..00000000 --- a/modules/adb-overwatch-mws-config/main.tf +++ /dev/null @@ -1,151 +0,0 @@ -data "azurerm_resource_group" "rg" { - name = var.rg_name -} - -data "azurerm_databricks_workspace" "overwatch-ws" { - name = var.overwatch_ws_name - resource_group_name = var.rg_name -} - -// Storage Account -resource "azurerm_storage_account" "ow-sa" { - name = join("", [var.ow_sa_name, var.random_string]) - resource_group_name = data.azurerm_resource_group.rg.name - location = data.azurerm_resource_group.rg.location - account_tier = "Standard" - account_replication_type = "LRS" - is_hns_enabled = true - - identity { - type = "SystemAssigned" - } - - tags = { - source = "Databricks" - application = "Overwatch" - description = "Overwatch ETL database storage" - } -} - -resource "azurerm_storage_data_lake_gen2_filesystem" "overwatch-db" { - name = "overwatch-db" - storage_account_id = azurerm_storage_account.ow-sa.id -} - -// Role Assignment -data "azuread_service_principal" "overwatch-spn" { - application_id = var.overwatch_spn_app_id -} - -resource "azurerm_role_assignment" "data-contributor-role" { - scope = azurerm_storage_account.ow-sa.id - role_definition_name = "Storage Blob Data Contributor" - principal_id = data.azuread_service_principal.overwatch-spn.object_id -} - -// AKV data -data "azurerm_key_vault" "existing-kv" { - name = var.akv_name - resource_group_name = var.rg_name -} - -resource "databricks_secret_scope" "overwatch-akv" { - name = var.databricks_secret_scope_name - - keyvault_metadata { - resource_id = data.azurerm_key_vault.existing-kv.id - dns_name = data.azurerm_key_vault.existing-kv.vault_uri - } - -} - -data "azurerm_key_vault_secret" "spn-key" { - name = "spn-key" - key_vault_id = data.azurerm_key_vault.existing-kv.id -} - -resource "databricks_mount" "overwatch_db" { - name = "overwatch-etl-db" - - abfs { - tenant_id = var.tenant_id - client_id = var.overwatch_spn_app_id - client_secret_scope = var.databricks_secret_scope_name - client_secret_key = data.azurerm_key_vault_secret.spn-key.name - initialize_file_system = true - storage_account_name = azurerm_storage_account.ow-sa.name - container_name = azurerm_storage_data_lake_gen2_filesystem.overwatch-db.name - } -} - -locals { - etl_storage_prefix = "/mnt/${databricks_mount.overwatch_db.name}/ow_multi_ws" -} - -resource "databricks_job" "overwatch" { - name = "Overwatch ETL Job" - new_cluster { - autoscale { - min_workers = 1 - max_workers = 3 - } - spark_version = var.latest_dbr_lts - node_type_id = "Standard_DS3_v2" - - spark_conf = { - "fs.azure.account.auth.type" : "OAuth" - "fs.azure.account.oauth.provider.type" : "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider" - "fs.azure.account.oauth2.client.endpoint" : "https://login.microsoftonline.com/${var.tenant_id}/oauth2/token" - "fs.azure.account.oauth2.client.id" : var.overwatch_spn_app_id - "fs.azure.account.oauth2.client.secret" : "{{secrets/${var.databricks_secret_scope_name}/${data.azurerm_key_vault_secret.spn-key.name}}}" - "spark.hadoop.fs.azure.account.auth.type" : "OAuth" - "spark.hadoop.fs.azure.account.oauth.provider.type" : "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider" - "spark.hadoop.fs.azure.account.oauth2.client.endpoint" : "https://login.microsoftonline.com/${var.tenant_id}/oauth2/token" - "spark.hadoop.fs.azure.account.oauth2.client.id" : var.overwatch_spn_app_id - "spark.hadoop.fs.azure.account.oauth2.client.secret" : "{{secrets/${var.databricks_secret_scope_name}/${data.azurerm_key_vault_secret.spn-key.name}}}" - - } - } - notebook_task { - notebook_path = "/Overwatch/ETL/overwatch-runner" - base_parameters = { - "TempDir" : "/tmp/overwatch/", - "Parallelism" : 4, - "ETLStoragePrefix" : local.etl_storage_prefix, - "PathToCsvConfig" : "/mnt/${databricks_mount.overwatch_db.name}/config/overwatch_deployment_config.csv" - } - } - - library { - maven { - coordinates = "com.microsoft.azure:azure-eventhubs-spark_2.12:2.3.21" - exclusions = [] - } - } - library { - maven { - coordinates = "com.databricks.labs:${var.overwatch_version}" - exclusions = [] - } - } - email_notifications { - - on_failure = [var.overwatch_job_notification_email] - no_alert_for_skipped_runs = false - - } - - schedule { - quartz_cron_expression = var.cron_job_schedule - timezone_id = var.cron_timezone_id - pause_status = "PAUSED" - } -} - -//Upload Databricks notebook -resource "databricks_notebook" "overwatch_etl" { - source = "${path.module}/notebooks/overwatch-runner.scala" - path = "/Overwatch/ETL/overwatch-runner" - format = "SOURCE" - language = "SCALA" -} diff --git a/modules/adb-overwatch-mws-config/notebooks/overwatch-runner.scala b/modules/adb-overwatch-mws-config/notebooks/overwatch-runner.scala deleted file mode 100644 index 3a03b972..00000000 --- a/modules/adb-overwatch-mws-config/notebooks/overwatch-runner.scala +++ /dev/null @@ -1,61 +0,0 @@ -// Databricks notebook source -// MAGIC %md -// MAGIC ## 071.x Deployment Runner -// MAGIC This notebook is used to deploy Overwatch onto one or many workspaces -// MAGIC -// MAGIC ### Parameters -// MAGIC `PATHTOCSVCONFIG` = Path to where the CSV was uploaded -// MAGIC -// MAGIC `TEMPDIR` = Overwatch temporary working directory for intermediate / temporary files during the run -// MAGIC -// MAGIC `ETLSTORAGEPREFIX` = `etl_storage_prefix` from the config file -// MAGIC -// MAGIC `PARALLELISM` = Number of workspaces to load simultaneously -- Should == number of workspaces to deploy (up to a max of about 20, beyond that, larger drivers and high-throughput tuning may need to be implemented on the cluster) -// MAGIC -// MAGIC ### REPORTS -// MAGIC This deployment method provides user-friendly reports including a validation report (if validation is executed) and a deployment report. Both of these reports are stored in the `/report` folder -// MAGIC * Validation Report `/report/validationReport` stored as Delta -// MAGIC * Deployment Report `/report/deploymentReport` stored as Delta - -// COMMAND ---------- - -import com.databricks.labs.overwatch.MultiWorkspaceDeployment - -// COMMAND ---------- - -val TEMPDIR = dbutils.widgets.get("TempDir") -val PARALLELISM = dbutils.widgets.get("Parallelism").toInt -val ETLSTORAGEPREFIX = dbutils.widgets.get("ETLStoragePrefix") -val PATHTOCSVCONFIG = dbutils.widgets.get("PathToCsvConfig") - -// COMMAND ---------- - -display( - spark.read.option("header", "true").csv(PATHTOCSVCONFIG) -) - -// COMMAND ---------- - -MultiWorkspaceDeployment(PATHTOCSVCONFIG, TEMPDIR).validate(PARALLELISM) - -// COMMAND ---------- - -display( - spark.read.format("delta") - .load(s"${ETLSTORAGEPREFIX}/report/validationReport") -) - -// COMMAND ---------- - -MultiWorkspaceDeployment(PATHTOCSVCONFIG, TEMPDIR).deploy(PARALLELISM, "Bronze,Silver,Gold") - -// COMMAND ---------- - -display( - spark.read.format("delta") - .load(s"${ETLSTORAGEPREFIX}/report/deploymentReport") -) - -// COMMAND ---------- - - diff --git a/modules/adb-overwatch-mws-config/output.tf b/modules/adb-overwatch-mws-config/output.tf deleted file mode 100644 index 00092f63..00000000 --- a/modules/adb-overwatch-mws-config/output.tf +++ /dev/null @@ -1,9 +0,0 @@ -output "etl_storage_prefix" { - description = "Overwatch ETL storage prefix, which represents a mount point to the ETL storage account" - value = local.etl_storage_prefix -} - -output "databricks_mount_db_name" { - description = "Mount point name to the storage account where Overwatch will be writing the results" - value = databricks_mount.overwatch_db.name -} \ No newline at end of file diff --git a/modules/adb-overwatch-mws-config/providers.tf b/modules/adb-overwatch-mws-config/providers.tf deleted file mode 100644 index 1d847d2b..00000000 --- a/modules/adb-overwatch-mws-config/providers.tf +++ /dev/null @@ -1,7 +0,0 @@ -terraform { - required_providers { - databricks = { - source = "databricks/databricks" - } - } -} \ No newline at end of file diff --git a/modules/adb-overwatch-mws-config/variables.tf b/modules/adb-overwatch-mws-config/variables.tf deleted file mode 100644 index eba04b21..00000000 --- a/modules/adb-overwatch-mws-config/variables.tf +++ /dev/null @@ -1,67 +0,0 @@ -variable "overwatch_ws_name" { - type = string - description = "Overwatch workspace name" -} - -variable "rg_name" { - type = string - description = "Resource group name" -} - -variable "overwatch_spn_app_id" { - type = string - description = "Azure SPN ID used to create the mount points" -} - -variable "tenant_id" { - type = string - description = "Azure Tenant ID" -} - -variable "ow_sa_name" { - type = string - description = "The name of the Overwatch ETL storage account" -} - -variable "akv_name" { - type = string - description = "Azure Key-Vault name" -} - -variable "databricks_secret_scope_name" { - type = string - description = "Databricks secret scope name (backed by Azure Key-Vault)" -} - -variable "overwatch_job_notification_email" { - default = "email@example.com" - description = "Overwatch Job Notification Email" -} - -variable "cron_job_schedule" { - type = string - default = "0 0 8 * * ?" - description = "Cron expression to schedule the Overwatch Job" -} - -variable "cron_timezone_id" { - type = string - default = "Europe/Brussels" - description = "Timezone for the cron schedule" -} - -variable "overwatch_version" { - type = string - description = "Overwatch library maven version" - default = "overwatch_2.12:0.7.1.0" -} - -variable "random_string" { - type = string - description = "Random string used as a suffix for the resources names" -} - -variable "latest_dbr_lts" { - type = string - description = "Latest DBR LTS version" -} \ No newline at end of file diff --git a/modules/adb-overwatch-regional-config/Makefile b/modules/adb-overwatch-regional-config/Makefile deleted file mode 100644 index 653039d8..00000000 --- a/modules/adb-overwatch-regional-config/Makefile +++ /dev/null @@ -1,7 +0,0 @@ -.PHONY: docs test_docs - -docs: - terraform-docs -c ../../.terraform-docs.yml . - -test_docs: - terraform-docs -c ../../.terraform-docs.yml --output-check . diff --git a/modules/adb-overwatch-regional-config/README.md b/modules/adb-overwatch-regional-config/README.md deleted file mode 100644 index e9118296..00000000 --- a/modules/adb-overwatch-regional-config/README.md +++ /dev/null @@ -1,61 +0,0 @@ -# adb-overwatch-regional-config - -This module deploys the regional Azure required resources for a multi-workspace Overwatch deployment : -- Storage account to store the logs generated by the workspaces where Overwatch will be deployed -- Role assignment of the SPN to the storage account created above -- Eventhub namespace and Eventhub namespace authorization rule -- Azure Key-Vault with its access policy -- Azure Vault secret to store the SPN secret value - - -## Requirements - -No requirements. - -## Providers - -| Name | Version | -|------|---------| -| [azuread](#provider\_azuread) | n/a | -| [azurerm](#provider\_azurerm) | n/a | - -## Modules - -No modules. - -## Resources - -| Name | Type | -|------|------| -| [azurerm_eventhub_namespace.ehn](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/eventhub_namespace) | resource | -| [azurerm_eventhub_namespace_authorization_rule.ehn-ar](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/eventhub_namespace_authorization_rule) | resource | -| [azurerm_key_vault.kv](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/key_vault) | resource | -| [azurerm_key_vault_access_policy.kv-ap](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/key_vault_access_policy) | resource | -| [azurerm_key_vault_secret.spn-key](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/key_vault_secret) | resource | -| [azurerm_role_assignment.data-contributor-role-log](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/role_assignment) | resource | -| [azurerm_storage_account.log-sa](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/storage_account) | resource | -| [azuread_service_principal.overwatch-spn](https://registry.terraform.io/providers/hashicorp/azuread/latest/docs/data-sources/service_principal) | data source | -| [azurerm_client_config.current](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/data-sources/client_config) | data source | -| [azurerm_resource_group.rg](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/data-sources/resource_group) | data source | - -## Inputs - -| Name | Description | Type | Default | Required | -|------|-------------|------|---------|:--------:| -| [ehn\_name](#input\_ehn\_name) | Eventhubs namespace name | `any` | n/a | yes | -| [key\_vault\_prefix](#input\_key\_vault\_prefix) | AKV prefix to use when creating the resource | `string` | n/a | yes | -| [logs\_sa\_name](#input\_logs\_sa\_name) | Logs storage account name | `any` | n/a | yes | -| [overwatch\_spn\_app\_id](#input\_overwatch\_spn\_app\_id) | Azure SPN ID used to create the mount points | `string` | n/a | yes | -| [overwatch\_spn\_secret](#input\_overwatch\_spn\_secret) | Azure SPN secret | `string` | n/a | yes | -| [random\_string](#input\_random\_string) | Random string used as a suffix for the resources names | `string` | n/a | yes | -| [rg\_name](#input\_rg\_name) | Resource group name | `string` | n/a | yes | - -## Outputs - -| Name | Description | -|------|-------------| -| [akv\_name](#output\_akv\_name) | AKV name | -| [ehn\_ar\_name](#output\_ehn\_ar\_name) | Eventhubs namespace authorization rule name | -| [ehn\_name](#output\_ehn\_name) | Eventhubs namespace name | -| [logs\_sa\_name](#output\_logs\_sa\_name) | Logs storage account name | - \ No newline at end of file diff --git a/modules/adb-overwatch-regional-config/main.tf b/modules/adb-overwatch-regional-config/main.tf deleted file mode 100644 index 2369f731..00000000 --- a/modules/adb-overwatch-regional-config/main.tf +++ /dev/null @@ -1,101 +0,0 @@ -// Resource Group -data "azurerm_resource_group" "rg" { - name = var.rg_name -} - - -// Storage Account -resource "azurerm_storage_account" "log-sa" { - name = join("", [var.logs_sa_name, var.random_string]) - resource_group_name = data.azurerm_resource_group.rg.name - location = data.azurerm_resource_group.rg.location - account_tier = "Standard" - account_replication_type = "LRS" - is_hns_enabled = true - - identity { - type = "SystemAssigned" - } - - tags = { - source = "Databricks" - application = "Overwatch" - description = "Overwatch cluster logs storage" - } -} - - -// Role Assignment -data "azuread_service_principal" "overwatch-spn" { - application_id = var.overwatch_spn_app_id -} - -resource "azurerm_role_assignment" "data-contributor-role-log" { - scope = azurerm_storage_account.log-sa.id - role_definition_name = "Storage Blob Data Contributor" - principal_id = data.azuread_service_principal.overwatch-spn.object_id -} - - -// Eventhubs -resource "azurerm_eventhub_namespace" "ehn" { - name = join("-", [var.ehn_name, var.random_string]) - location = data.azurerm_resource_group.rg.location - resource_group_name = data.azurerm_resource_group.rg.name - sku = "Basic" - capacity = 1 - - tags = { - environment = "Overwatch" - } -} - -resource "azurerm_eventhub_namespace_authorization_rule" "ehn-ar" { - name = join("-", [var.ehn_name, "ar", var.random_string]) - namespace_name = azurerm_eventhub_namespace.ehn.name - resource_group_name = data.azurerm_resource_group.rg.name - listen = true - send = true - manage = true -} - -// AKV -data "azurerm_client_config" "current" { -} - -resource "azurerm_key_vault" "kv" { - name = join("-", [var.key_vault_prefix, var.random_string]) - location = data.azurerm_resource_group.rg.location - resource_group_name = var.rg_name - tenant_id = data.azurerm_client_config.current.tenant_id - sku_name = "standard" - purge_protection_enabled = false -} - - -resource "azurerm_key_vault_access_policy" "kv-ap" { - key_vault_id = azurerm_key_vault.kv.id - tenant_id = data.azurerm_client_config.current.tenant_id - object_id = data.azurerm_client_config.current.object_id - - key_permissions = [ - "Get", "List", "Update", "Create", "Import", "Delete", "Recover", "Backup", "Restore", "Purge" - ] - secret_permissions = [ - "Get", "List", "Set", "Delete", "Recover", "Backup", "Restore", "Purge" - ] - certificate_permissions = [ - "Get", "List", "Update", "Create", "Import", "Delete", "Recover", "Backup", "Restore", "Purge" - ] - - depends_on = [azurerm_key_vault.kv] -} - -resource "azurerm_key_vault_secret" "spn-key" { - name = "spn-key" - value = var.overwatch_spn_secret - expiration_date = "2030-12-31T23:59:59Z" - key_vault_id = azurerm_key_vault.kv.id - - depends_on = [azurerm_key_vault_access_policy.kv-ap] -} \ No newline at end of file diff --git a/modules/adb-overwatch-regional-config/output.tf b/modules/adb-overwatch-regional-config/output.tf deleted file mode 100644 index 61c5075d..00000000 --- a/modules/adb-overwatch-regional-config/output.tf +++ /dev/null @@ -1,19 +0,0 @@ -output "ehn_name" { - description = "Eventhubs namespace name" - value = azurerm_eventhub_namespace.ehn.name -} - -output "ehn_ar_name" { - description = "Eventhubs namespace authorization rule name" - value = azurerm_eventhub_namespace_authorization_rule.ehn-ar.name -} - -output "logs_sa_name" { - description = "Logs storage account name" - value = azurerm_storage_account.log-sa.name -} - -output "akv_name" { - description = "AKV name" - value = azurerm_key_vault.kv.name -} diff --git a/modules/adb-overwatch-regional-config/providers.tf b/modules/adb-overwatch-regional-config/providers.tf deleted file mode 100644 index 1d847d2b..00000000 --- a/modules/adb-overwatch-regional-config/providers.tf +++ /dev/null @@ -1,7 +0,0 @@ -terraform { - required_providers { - databricks = { - source = "databricks/databricks" - } - } -} \ No newline at end of file diff --git a/modules/adb-overwatch-regional-config/variables.tf b/modules/adb-overwatch-regional-config/variables.tf deleted file mode 100644 index 0b8c5e81..00000000 --- a/modules/adb-overwatch-regional-config/variables.tf +++ /dev/null @@ -1,32 +0,0 @@ -variable "rg_name" { - type = string - description = "Resource group name" -} - -variable "random_string" { - type = string - description = "Random string used as a suffix for the resources names" -} - -variable "logs_sa_name" { - description = "Logs storage account name" -} - -variable "overwatch_spn_app_id" { - type = string - description = "Azure SPN ID used to create the mount points" -} - -variable "overwatch_spn_secret" { - type = string - description = "Azure SPN secret" -} - -variable "ehn_name" { - description = "Eventhubs namespace name" -} - -variable "key_vault_prefix" { - type = string - description = "AKV prefix to use when creating the resource" -} \ No newline at end of file diff --git a/modules/adb-overwatch-ws-to-monitor/Makefile b/modules/adb-overwatch-ws-to-monitor/Makefile deleted file mode 100644 index 653039d8..00000000 --- a/modules/adb-overwatch-ws-to-monitor/Makefile +++ /dev/null @@ -1,7 +0,0 @@ -.PHONY: docs test_docs - -docs: - terraform-docs -c ../../.terraform-docs.yml . - -test_docs: - terraform-docs -c ../../.terraform-docs.yml --output-check . diff --git a/modules/adb-overwatch-ws-to-monitor/README.md b/modules/adb-overwatch-ws-to-monitor/README.md deleted file mode 100644 index 709caf51..00000000 --- a/modules/adb-overwatch-ws-to-monitor/README.md +++ /dev/null @@ -1,96 +0,0 @@ -# adb-overwatch-ws-to-monitor - -This module deploys the required resources for a given Databricks workspace to be monitored by Overwatch : -- Databricks PAT that will be used by Overwatch jobs -- Eventhub topic with its authorization rule -- Diagnostics settings -- AKV secrets to store the Databricks PAT, and the Eventhub primary connection string created above -- AKV-backed Databricks secret scope -- Container for the cluster logs in the existing log storage account -- Databricks mount to the container created above -- CSV file with all required parameters using [Overwatch deployment template](./overwatch_deployment_template.txt) - - -> **Note** -> For more details on the column description, please refer to [Overwatch Deployment Configuration](https://databrickslabs.github.io/overwatch/deployoverwatch/configureoverwatch/configuration/) - - -## Requirements - -No requirements. - -## Providers - -| Name | Version | -|------|---------| -| [azurerm](#provider\_azurerm) | n/a | -| [databricks](#provider\_databricks) | n/a | -| [null](#provider\_null) | n/a | -| [template](#provider\_template) | n/a | - -## Modules - -No modules. - -## Resources - -| Name | Type | -|------|------| -| [azurerm_eventhub.eh](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/eventhub) | resource | -| [azurerm_eventhub_authorization_rule.eh-ar](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/eventhub_authorization_rule) | resource | -| [azurerm_key_vault_secret.adb-pat](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/key_vault_secret) | resource | -| [azurerm_key_vault_secret.eh-conn-string](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/key_vault_secret) | resource | -| [azurerm_monitor_diagnostic_setting.dgs-ws](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/monitor_diagnostic_setting) | resource | -| [azurerm_storage_data_lake_gen2_filesystem.cluster-logs-fs](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/storage_data_lake_gen2_filesystem) | resource | -| [databricks_mount.cluster-logs-mount-ws](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/mount) | resource | -| [databricks_secret_scope.overwatch-akv](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/secret_scope) | resource | -| [databricks_token.pat-ws](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/token) | resource | -| [null_resource.local](https://registry.terraform.io/providers/hashicorp/null/latest/docs/resources/resource) | resource | -| [azurerm_databricks_workspace.adb-ws](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/data-sources/databricks_workspace) | data source | -| [azurerm_eventhub_namespace_authorization_rule.ehn-ar](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/data-sources/eventhub_namespace_authorization_rule) | data source | -| [azurerm_key_vault.existing-kv](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/data-sources/key_vault) | data source | -| [azurerm_key_vault_secret.spn-key](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/data-sources/key_vault_secret) | data source | -| [azurerm_monitor_diagnostic_categories.dgs-cat](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/data-sources/monitor_diagnostic_categories) | data source | -| [azurerm_resource_group.rg](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/data-sources/resource_group) | data source | -| [azurerm_storage_account.logs-sa](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/data-sources/storage_account) | data source | -| [template_cloudinit_config.local](https://registry.terraform.io/providers/hashicorp/template/latest/docs/data-sources/cloudinit_config) | data source | -| [template_file.ow-deployment-config](https://registry.terraform.io/providers/hashicorp/template/latest/docs/data-sources/file) | data source | - -## Inputs - -| Name | Description | Type | Default | Required | -|------|-------------|------|---------|:--------:| -| [active](#input\_active) | Whether or not the workspace should be validated / deployed | `bool` | n/a | yes | -| [adb\_ws\_name](#input\_adb\_ws\_name) | The name of an existing Databricks workspace that Overwatch will monitor | `string` | n/a | yes | -| [akv\_name](#input\_akv\_name) | Azure Key-Vault name | `string` | n/a | yes | -| [api\_waiting\_time](#input\_api\_waiting\_time) | API Tunable - Overwatch makes async api calls in parallel, api\_waiting\_time signifies the max wait time in case of no response received from the api call | `string` | n/a | yes | -| [auditlog\_prefix\_source\_path](#input\_auditlog\_prefix\_source\_path) | Location of auditlog (AWS/GCP Only) | `string` | n/a | yes | -| [automated\_dbu\_price](#input\_automated\_dbu\_price) | Contract price for automated DBUs | `number` | n/a | yes | -| [databricks\_secret\_scope\_name](#input\_databricks\_secret\_scope\_name) | Databricks secret scope name (backed by Azure Key-Vault) | `string` | n/a | yes | -| [ehn\_auth\_rule\_name](#input\_ehn\_auth\_rule\_name) | Eventhub namespace authorization rule name | `string` | n/a | yes | -| [ehn\_name](#input\_ehn\_name) | Eventhub namespace name | `string` | n/a | yes | -| [enable\_unsafe\_SSL](#input\_enable\_unsafe\_SSL) | API Tunable - Enables unsafe SSL | `string` | n/a | yes | -| [error\_batch\_size](#input\_error\_batch\_size) | API Tunable - Indicates the size of the error writer buffer containing API call errors | `string` | n/a | yes | -| [etl\_storage\_prefix](#input\_etl\_storage\_prefix) | Overwatch ETL storage prefix, which represents a mount point to the ETL storage account | `string` | n/a | yes | -| [excluded\_scopes](#input\_excluded\_scopes) | Scopes that should not be excluded from the pipelines | `string` | n/a | yes | -| [interactive\_dbu\_price](#input\_interactive\_dbu\_price) | Contract price for interactive DBUs | `number` | n/a | yes | -| [jobs\_light\_dbu\_price](#input\_jobs\_light\_dbu\_price) | Contract price for interactive DBUs | `number` | n/a | yes | -| [logs\_sa\_name](#input\_logs\_sa\_name) | Logs storage account name | `string` | n/a | yes | -| [max\_days](#input\_max\_days) | This is the max incremental days that will be loaded. Usually only relevant for historical loading and rebuilds | `number` | n/a | yes | -| [overwatch\_spn\_app\_id](#input\_overwatch\_spn\_app\_id) | Azure SPN used to create Databricks mounts | `string` | n/a | yes | -| [proxy\_host](#input\_proxy\_host) | Proxy url for the workspace | `string` | n/a | yes | -| [proxy\_password\_key](#input\_proxy\_password\_key) | Key which contains proxy password | `string` | n/a | yes | -| [proxy\_password\_scope](#input\_proxy\_password\_scope) | Scope which contains the proxy password key | `string` | n/a | yes | -| [proxy\_port](#input\_proxy\_port) | Proxy port for the workspace | `string` | n/a | yes | -| [proxy\_user\_name](#input\_proxy\_user\_name) | Proxy user name for the workspace | `string` | n/a | yes | -| [random\_string](#input\_random\_string) | Random string used as a suffix for the resources names | `string` | n/a | yes | -| [rg\_name](#input\_rg\_name) | Resource group name | `string` | n/a | yes | -| [sql\_compute\_dbu\_price](#input\_sql\_compute\_dbu\_price) | Contract price for DBSQL DBUs | `number` | n/a | yes | -| [success\_batch\_size](#input\_success\_batch\_size) | API Tunable - Indicates the size of the buffer on filling of which the result will be written to a temp location. This is used to tune performance in certain circumstance | `string` | n/a | yes | -| [tenant\_id](#input\_tenant\_id) | Azure tenant ID | `string` | n/a | yes | -| [thread\_pool\_size](#input\_thread\_pool\_size) | API Tunable - Max number of API calls Overwatch is allowed to make in parallel | `string` | n/a | yes | - -## Outputs - -No outputs. - diff --git a/modules/adb-overwatch-ws-to-monitor/main.tf b/modules/adb-overwatch-ws-to-monitor/main.tf deleted file mode 100644 index 3e038b1b..00000000 --- a/modules/adb-overwatch-ws-to-monitor/main.tf +++ /dev/null @@ -1,187 +0,0 @@ -// Resource Group -data "azurerm_resource_group" "rg" { - name = var.rg_name -} - - -// Azure Databricks ws -data "azurerm_databricks_workspace" "adb-ws" { - name = var.adb_ws_name - resource_group_name = var.rg_name -} - -resource "databricks_token" "pat-ws" { - comment = "Databricks PAT to be used by Overwatch jobs" -} - - -// EH topic -data "azurerm_eventhub_namespace_authorization_rule" "ehn-ar" { - name = var.ehn_auth_rule_name - resource_group_name = var.rg_name - namespace_name = var.ehn_name -} - -resource "azurerm_eventhub" "eh" { - name = "eh-overwatch-${data.azurerm_databricks_workspace.adb-ws.name}" - namespace_name = var.ehn_name - resource_group_name = var.rg_name - partition_count = 2 - message_retention = 1 -} - -resource "azurerm_eventhub_authorization_rule" "eh-ar" { - name = "eh-auth-rule-${data.azurerm_databricks_workspace.adb-ws.name}" - namespace_name = var.ehn_name - eventhub_name = azurerm_eventhub.eh.name - resource_group_name = data.azurerm_resource_group.rg.name - listen = true - send = true - manage = true -} - - -// Diagnostics Logs -data "azurerm_monitor_diagnostic_categories" "dgs-cat" { - resource_id = data.azurerm_databricks_workspace.adb-ws.id -} - -resource "azurerm_monitor_diagnostic_setting" "dgs-ws" { - name = "dgs-${data.azurerm_databricks_workspace.adb-ws.name}" - target_resource_id = data.azurerm_databricks_workspace.adb-ws.id - eventhub_name = azurerm_eventhub.eh.name - eventhub_authorization_rule_id = data.azurerm_eventhub_namespace_authorization_rule.ehn-ar.id - - dynamic "enabled_log" { - iterator = log_category_type - for_each = data.azurerm_monitor_diagnostic_categories.dgs-cat.log_category_types - content { - category = log_category_type.value - retention_policy { - enabled = false - } - } - } -} - - -// AKV -data "azurerm_key_vault" "existing-kv" { - name = var.akv_name - resource_group_name = var.rg_name -} - -data "azurerm_key_vault_secret" "spn-key" { - name = "spn-key" - key_vault_id = data.azurerm_key_vault.existing-kv.id -} - -resource "azurerm_key_vault_secret" "adb-pat" { - name = "pat-${data.azurerm_databricks_workspace.adb-ws.name}" - value = databricks_token.pat-ws.token_value - expiration_date = "2030-12-31T23:59:59Z" - key_vault_id = data.azurerm_key_vault.existing-kv.id -} - -resource "azurerm_key_vault_secret" "eh-conn-string" { - name = "eh-primary-conn-${data.azurerm_databricks_workspace.adb-ws.name}" - value = azurerm_eventhub_authorization_rule.eh-ar.primary_connection_string - expiration_date = "2030-12-31T23:59:59Z" - key_vault_id = data.azurerm_key_vault.existing-kv.id -} - -resource "databricks_secret_scope" "overwatch-akv" { - name = var.databricks_secret_scope_name - - keyvault_metadata { - resource_id = data.azurerm_key_vault.existing-kv.id - dns_name = data.azurerm_key_vault.existing-kv.vault_uri - } -} - -// Mount point to the Logs storage account -data "azurerm_storage_account" "logs-sa" { - name = var.logs_sa_name - resource_group_name = var.rg_name -} - -resource "azurerm_storage_data_lake_gen2_filesystem" "cluster-logs-fs" { - name = "cluster-logs-${data.azurerm_databricks_workspace.adb-ws.name}" - storage_account_id = data.azurerm_storage_account.logs-sa.id -} - -resource "databricks_mount" "cluster-logs-mount-ws" { - name = "cluster-logs" - - abfs { - tenant_id = var.tenant_id - client_id = var.overwatch_spn_app_id - client_secret_scope = var.databricks_secret_scope_name - client_secret_key = data.azurerm_key_vault_secret.spn-key.name - initialize_file_system = true - storage_account_name = data.azurerm_storage_account.logs-sa.name - container_name = azurerm_storage_data_lake_gen2_filesystem.cluster-logs-fs.name - } -} - -// add overwatch config that will be written to the csv file -data "template_file" "ow-deployment-config" { - template = file("${path.module}/overwatch_deployment_template.txt") - vars = { - workspace_name = data.azurerm_databricks_workspace.adb-ws.name - workspace_id = data.azurerm_databricks_workspace.adb-ws.workspace_id - workspace_url = "https://${data.azurerm_databricks_workspace.adb-ws.workspace_url}" - api_url = "https://${data.azurerm_databricks_workspace.adb-ws.location}.azuredatabricks.net" - cloud = "Azure" - primordial_date = formatdate("YYYY-MM-DD", timestamp()) - etl_storage_prefix = var.etl_storage_prefix - etl_database_name = "ow_etl_mws" - consumer_database_name = "overwatch_consumer_mws" - secret_scope = var.databricks_secret_scope_name - secret_key_dbpat = azurerm_key_vault_secret.adb-pat.name - auditlogprefix_source_path = var.auditlog_prefix_source_path - eh_name = azurerm_eventhub.eh.name - eh_scope_key = azurerm_key_vault_secret.eh-conn-string.name - interactive_dbu_price = var.interactive_dbu_price - automated_dbu_price = var.automated_dbu_price - sql_compute_dbu_price = var.sql_compute_dbu_price - jobs_light_dbu_price = var.jobs_light_dbu_price - max_days = var.max_days - excluded_scopes = var.excluded_scopes - active = var.active - proxy_host = var.proxy_host - proxy_port = var.proxy_port - proxy_user_name = var.proxy_user_name - proxy_password_scope = var.proxy_password_scope - proxy_password_key = var.proxy_password_key - success_batch_size = var.success_batch_size - error_batch_size = var.error_batch_size - enable_unsafe_SSL = var.enable_unsafe_SSL - thread_pool_size = var.thread_pool_size - api_waiting_time = var.api_waiting_time - } -} - -locals { - filename = "overwatch_deployment_config.csv" -} - -data "template_cloudinit_config" "local" { - gzip = false - base64_encode = false - - part { - filename = local.filename - content = data.template_file.ow-deployment-config.rendered - } -} - -resource "null_resource" "local" { - triggers = { - template = data.template_file.ow-deployment-config.rendered - } - - provisioner "local-exec" { - command = "echo \"${data.template_file.ow-deployment-config.rendered}\" >> \"${local.filename}\"" - } -} \ No newline at end of file diff --git a/modules/adb-overwatch-ws-to-monitor/overwatch_deployment_template.txt b/modules/adb-overwatch-ws-to-monitor/overwatch_deployment_template.txt deleted file mode 100644 index 99f55fbf..00000000 --- a/modules/adb-overwatch-ws-to-monitor/overwatch_deployment_template.txt +++ /dev/null @@ -1 +0,0 @@ -${workspace_name},${workspace_id},${workspace_url},${api_url},${cloud},${primordial_date},${etl_storage_prefix},${etl_database_name},${consumer_database_name},${secret_scope},${secret_key_dbpat},${auditlogprefix_source_path},${eh_name},${eh_scope_key},${interactive_dbu_price},${automated_dbu_price},${sql_compute_dbu_price},${jobs_light_dbu_price},${max_days},${excluded_scopes},${active},${proxy_host},${proxy_port},${proxy_user_name},${proxy_password_scope},${proxy_password_key},${success_batch_size},${error_batch_size},${enable_unsafe_SSL},${thread_pool_size},${api_waiting_time} \ No newline at end of file diff --git a/modules/adb-overwatch-ws-to-monitor/providers.tf b/modules/adb-overwatch-ws-to-monitor/providers.tf deleted file mode 100644 index 1d847d2b..00000000 --- a/modules/adb-overwatch-ws-to-monitor/providers.tf +++ /dev/null @@ -1,7 +0,0 @@ -terraform { - required_providers { - databricks = { - source = "databricks/databricks" - } - } -} \ No newline at end of file diff --git a/modules/adb-overwatch-ws-to-monitor/variables.tf b/modules/adb-overwatch-ws-to-monitor/variables.tf deleted file mode 100644 index 969bffe9..00000000 --- a/modules/adb-overwatch-ws-to-monitor/variables.tf +++ /dev/null @@ -1,144 +0,0 @@ -variable "adb_ws_name" { - type = string - description = "The name of an existing Databricks workspace that Overwatch will monitor" -} - -variable "rg_name" { - type = string - description = "Resource group name" -} - -variable "ehn_name" { - type = string - description = "Eventhub namespace name" -} - -variable "tenant_id" { - type = string - description = "Azure tenant ID" -} - -variable "overwatch_spn_app_id" { - type = string - description = "Azure SPN used to create Databricks mounts" -} - -variable "ehn_auth_rule_name" { - type = string - description = "Eventhub namespace authorization rule name" -} - -variable "logs_sa_name" { - type = string - description = "Logs storage account name" -} - -variable "random_string" { - type = string - description = "Random string used as a suffix for the resources names" -} - -variable "akv_name" { - type = string - description = "Azure Key-Vault name" -} - -variable "databricks_secret_scope_name" { - type = string - description = "Databricks secret scope name (backed by Azure Key-Vault)" -} - -variable "etl_storage_prefix" { - type = string - description = "Overwatch ETL storage prefix, which represents a mount point to the ETL storage account" -} - -variable "interactive_dbu_price" { - type = number - description = "Contract price for interactive DBUs" -} - -variable "automated_dbu_price" { - type = number - description = "Contract price for automated DBUs" -} - -variable "sql_compute_dbu_price" { - type = number - description = "Contract price for DBSQL DBUs" -} - -variable "jobs_light_dbu_price" { - type = number - description = "Contract price for interactive DBUs" -} - -variable "max_days" { - type = number - description = "This is the max incremental days that will be loaded. Usually only relevant for historical loading and rebuilds" -} - -variable "excluded_scopes" { - type = string - description = "Scopes that should not be excluded from the pipelines" -} - -variable "active" { - type = bool - description = "Whether or not the workspace should be validated / deployed" -} - -variable "proxy_host" { - type = string - description = "Proxy url for the workspace" -} - -variable "proxy_port" { - type = string - description = "Proxy port for the workspace" -} - -variable "proxy_user_name" { - type = string - description = "Proxy user name for the workspace" -} - -variable "proxy_password_scope" { - type = string - description = "Scope which contains the proxy password key" -} - -variable "proxy_password_key" { - type = string - description = "Key which contains proxy password" -} - -variable "success_batch_size" { - type = string - description = "API Tunable - Indicates the size of the buffer on filling of which the result will be written to a temp location. This is used to tune performance in certain circumstance" -} - -variable "error_batch_size" { - type = string - description = "API Tunable - Indicates the size of the error writer buffer containing API call errors" -} - -variable "enable_unsafe_SSL" { - type = string - description = "API Tunable - Enables unsafe SSL" -} - -variable "thread_pool_size" { - type = string - description = "API Tunable - Max number of API calls Overwatch is allowed to make in parallel" -} - -variable "api_waiting_time" { - type = string - description = "API Tunable - Overwatch makes async api calls in parallel, api_waiting_time signifies the max wait time in case of no response received from the api call" -} - -variable "auditlog_prefix_source_path" { - type = string - description = "Location of auditlog (AWS/GCP Only)" -} \ No newline at end of file