How To Check Memory During Databricks Job

 

 

 

BRIEF OVERVIEW

To check memory usage during a Databricks job, you can utilize several methods and tools provided by Databricks.

Here’s a comprehensive guide on how to monitor memory usage:

Using the Compute Metrics UI

Databricks offers a native compute metrics tool in the UI that provides real-time and historical data on hardware and Spark metrics, including memory usage.

1. Access the Compute Metrics UI:
   • Click on “Compute” in the sidebar
   • Select the compute resource you want to monitor
   • Click on the “Metrics” tab
2. View Memory Metrics:
   • By default, you’ll see hardware metrics, including memory usage
   • You can switch to Spark metrics for more detailed information
3. Filter by Time Period:
   • Use the date picker to select a specific time range
   • Metrics are collected every minute and stored for up to 30 days

Monitoring Job Cluster Utilization

For a more programmatic approach, you can query the Ganglia API on the Spark driver to obtain raw memory usage data:

This method allows you to capture memory usage at the system level, including both PySpark workloads in the JVM and Python workloads outside of it.

Using PySpark Memory Profiler

For more detailed memory profiling of PySpark User-Defined Functions (UDFs), you can use the PySpark Memory Profiler:

1. Enable Memory Profiling:
   • Install the Memory Profiler library on your cluster
   • Set the Spark configuration spark.python.profile.memory to true
2. Profile UDF Memory Usage:
   • Use the @profile decorator on your UDF
   • Run your job and collect the memory profile results

Example:

This will provide a detailed breakdown of memory usage by line of code within your UDF.

Best Practices for Memory Monitoring

1. Regular Monitoring: Implement routine checks of memory usage across your Databricks jobs.
2. Optimize Cluster Size: Use the memory usage data to right-size your clusters, avoiding over-provisioning or under-provisioning.
3. Identify Memory Leaks: Look for patterns of increasing memory usage over time that might indicate memory leaks.
4. Profile UDFs: Use the PySpark Memory Profiler to optimize memory-intensive UDFs.
5. Clear Caches: If memory usage is high, consider clearing Spark caches using spark.catalog.clearCache() or unmounting file systems with dbutils.fs.unmount().
6. Monitor Multiple Metrics: In addition to memory, keep an eye on CPU utilization, disk I/O, and network usage for a comprehensive view of your job’s performance.

 

Using these tools and practices, you can effectively monitor and optimize memory usage in your Databricks jobs, leading to better performance and cost-efficiency.