Support for Databricks CLI versions 0.205 and above

[mrmasterplan]

Expected Behavior

I expect the dbx tool to continue to support the lates and recommended versions of databricks tooling.
Databricks now recommends all users to migrate to the new Databricks CLI version 0.205 or above. (Link: https://learn.microsoft.com/en-us/azure/databricks/archive/dev-tools/cli/)
The version 0.18 is considered deprecated. DBX currently requires databricks-cli version 0.17. (See link to repo here: https://github.com/databrickslabs/dbx/blob/main/setup.py#L14)

Can we have a discussion how we continue to support the latests tool versions?

Current Behavior

DBX is incompatible with Databricks CLI version 0.205 or above.

Context

see above

Your Environment

• dbx version used: 0.8.18
• Databricks Runtime version: 13.3LTS

[mrmasterplan]

I see two ways to proceed:

Option 1: Use the databricks-sdk package which is available on pypi and depend on that instead of on the CLI tool.

Option 2: Depend on Databricks CLI version 0.205 and above which needs to be installed independently. It is not avialable on Pypi.

What do people prefer? I feel that option 2 is probably easier to code.