Permanently Deleting a Cluster in Databricks
To permanently delete a cluster in Databricks, you need to use the Clusters API. This involves sending a POST request to the API endpoint for permanent deletion. Here’s how you can do it:
First, ensure you have the cluster ID of the cluster you want to delete. If you’re using the dbx launch command, you might need to find another way to obtain the cluster ID, as it’s not directly returned by the command.
Once you have the cluster ID, you can use the following command to permanently delete the cluster:
curl --netrc -X POST
https://dbc-a1b2345c-d6e7.cloud.databricks.com/api/2.0/clusters/permanent-delete
--data '{ "cluster_id": "1234-567890-frays123" }'
This command will permanently delete the specified cluster, removing its configuration and preventing any further use or restart.
Frequently Asked Questions
- Q: What happens if I terminate a cluster without pinning it?
A: If a cluster is terminated and not pinned, it will be automatically deleted 30 days later. - Q: Can I undo a cluster deletion?
A: No, deleting a cluster is a permanent action and cannot be undone. - Q: How do I pin a cluster in Databricks?
A: You can pin a cluster from the compute list or the compute detail page by clicking the pin icon. - Q: What is the purpose of pinning a cluster?
A: Pinning a cluster prevents it from being automatically deleted after termination, allowing you to keep its configuration for future use. - Q: Can I restart a terminated cluster?
A: Yes, you can restart a terminated cluster using the Clusters API or from the Databricks UI. - Q: How do I obtain a cluster ID for a job cluster created by dbx launch?
A: Currently, there is no direct method to obtain the cluster ID from the dbx launch command. You may need to check the Databricks UI or logs for the ID. - Q: Can I use the Clusters API to manage all types of compute resources?
A: Yes, the Clusters API allows you to manage compute resources programmatically, including starting, terminating, and deleting clusters.
Bottom Line: Permanently deleting a cluster in Databricks requires using the Clusters API with the cluster ID. Ensure you have the necessary permissions and understand that deletion is irreversible.