Support library development

[twsl]

One issue I have with databricks is the lack of support for proper python library develoment. As soon as I follow python dev best practices instead of just using notebooks, databricks support lacks a bit.
This is partially supported with artifacts and jobs in the databricks.yml.
Ideally I want to see a workflow, where I can use my vscode just like before and don't feel a difference.

Things I'd like to see:

• editable install of synced library code
• automatic availability/install on selected cluster
• Interactive notebooks

I'm not sure how well this is aligned with the vision for this extension, but I'll try to come up with a proposal PR to address my needs.

[TanishqDatabricks]

Hey @twsl, sorry for the delayed response! We've significantly ramped up support and resources to improve the Databricks IDE experience and are revisiting GitHub issues.

Everything you're asking for is now available in an SSH tunnel that allows you to connect directly to your Databricks compute and the workspace.

• You can install dependencies directly onto standard compute or set up a workspace base environment using serverless.
• We're going to make it easier to manage environments and dependencies with uv support and CLI commands that sync local dependencies with Databricks Runtime versions.
• Existing VS Code extension features will be integrated into this SSH tunnel shortly.

Do you have time to chat so you can share additional feedback on what improvements you'd like to see and give feedback on our roadmap? You can book a time to chat here or email me at tanishq.maheshwari@databricks.com to share your availability or feedback async.

[twsl]

Nice to see, this seems well aligned with my preferred workflow. I'll check if the feature is already enabled on my professional instances and will reach out after testing

[TanishqDatabricks]

@twsl glad to hear that it's aligned with your workflows, looking forward to hear back once you test it out!