---
title: "How to Read Databricks Job Logs Programmatically"
description: "Learn how to programmatically read Databricks job logs using REST API, Python, DBFS, and CLI. Step-by-step guide for automation and debugging."
url: https://srinimf.com/2025/06/19/how-to-read-databricks-job-logs-programmatically/
date: 2025-06-19
modified: 2026-10-17
author: "Srini"
image: https://i0.wp.com/srinimf.com/wp-content/uploads/2025/06/databricks-logs-rest-api.png?fit=2240%2C1260&ssl=1
categories: ["databricks"]
tags: ["Automation", "databricks", "databricks api", "job logs"]
type: post
lang: en
---

# How to Read Databricks Job Logs Programmatically

If you’re working with production workloads [in Databricks](https://srinimf.com/2025/01/01/viewing-files-in-databricks-local-vs-dbfs-guide/), analyzing job logs is crucial for debugging, monitoring, and automation.

While the Databricks UI provides a convenient way to check logs, many data engineers and developers prefer to **access logs programmatically**—especially for automation, reporting, or alerts.

In this post, you’ll learn **[how to](https://srinimf.com/2026/10/17/databricks-find-job-status/) programmatically read Databricks job logs** using Python and REST APIs, along with a few additional methods.

## 🔧 Why Read Logs Programmatically?

- ✅ Automate error tracking or health monitoring
- ✅ Build dashboards for job status
- ✅ Integrate logs with monitoring tools
- ✅ Store logs in S3, Azure Blob, or internal systems
- ✅ Trigger actions based on job failures

## 🛠️ 1. Read Logs Using Databricks REST API in Python

The **Databricks REST API v2.1** provides powerful endpoints to access job run data, including logs.

### ✅ Step 1: Setup Authentication

Generate a **Personal Access Token** from your Databricks account settings. Use it like this:

```
import requests
DATABRICKS_INSTANCE = 'https://<your-databricks-instance>'  # e.g., https://adb-123456.7.azuredatabricks.net
TOKEN = 'dapiXXXXXXXXXXXXXXXXXXXX'
HEADERS = {'Authorization': f'Bearer {TOKEN}'}
```

### ✅ Step 2: List Job Runs

You can list past runs of a specific job by ID:

```
job_id = 12345  # Replace with your actual Job ID
response = requests.get(
    f'{DATABRICKS_INSTANCE}/api/2.1/jobs/runs/list?job_id={job_id}',
    headers=HEADERS
)
runs = response.json()
```

### ✅ Step 3: Get Logs and Notebook Output

To fetch the log output of a specific run:

```
run_id = runs['runs'][0]['run_id']  # Fetch latest run ID
log_response = requests.get(
    f'{DATABRICKS_INSTANCE}/api/2.1/jobs/runs/get-output?run_id={run_id}',
    headers=HEADERS
)
log_output = log_response.json()
print("Logs:", log_output.get("logs", "No logs available"))
print("Notebook Output:", log_output.get("notebook_output", {}).get("result", "No notebook output"))
```

You can also access:

- `error` → if the job failed
- `metadata` → status, cluster details, etc.

**Recommended Books**

- [Data Pipeline Pocket Reference](https://www.amazon.in/Data-Pipelines-Pocket-Reference-Processing/dp/8194722934/?tag=srinimf-21)
- [Querying Databricks SQL](https://www.amazon.in/dp/9355518013/?tag=srinmf-21)
- [Azure Databricks Cookbook](https://www.amazon.in/dp/1789809711/?tag=srinmf-21)

## 📂 2. Read Logs from DBFS (Databricks File System)

If you write logs to DBFS from your code (e.g., using `print()` or `logging`), you can read them like this:

```
# Within a Databricks notebook
dbutils.fs.head("dbfs:/logs/my-job-log.txt")
# From Python code
with open("/dbfs/logs/my-job-log.txt") as f:
    print(f.read())
```

To enable this, make sure your jobs log output explicitly:

```
import logging
logging.basicConfig(level=logging.INFO)
logging.info("This log will be stored and can be read later.")
```

## 💻 3. Optional: Use `databricks-cli` for Scripting

If you prefer shell scripts, install the Databricks CLI:

```
databricks configure --token
```

Then use:

```
databricks jobs runs get-output --run-id <run_id>
```

This CLI can be incorporated into CI/CD pipelines and monitoring scripts.

## 🧠 Best Practices

- Use `print()` or Python logging to generate readable logs
- Store logs in a consistent path in DBFS or a cloud bucket
- Automate log retrieval and integrate with alerting systems
- Combine with Databricks webhooks or monitoring tools like Prometheus or PagerDuty

## 📌 Conclusion

Reading Databricks job logs programmatically gives you full control over monitoring and automation. Whether you’re troubleshooting failures or building dashboards, the Databricks API, DBFS, and CLI offer flexible options.

If you’re running [production job](https://srinimf.com/2025/07/02/databricks-production-job-failures-a-complete-guide/)s at scale, log automation isn’t just a good practice—it’s essential.

### Share this:

-
- [Tweet](https://twitter.com/share)
- [![](https://i0.wp.com/assets.pinterest.com/images/pidgets/pinit_fg_en_rect_gray_20.png)](https://www.pinterest.com/pin/create/link/?url=https%3A%2F%2Fsrinimf.com%2F2025%2F06%2F19%2Fhow-to-read-databricks-job-logs-programmatically%2F&media=https%3A%2F%2Fi0.wp.com%2Fsrinimf.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fdatabricks-logs-rest-api.png%3Ffit%3D1200%252C675%26ssl%3D1&description=How%20to%20Read%20Databricks%20Job%20Logs%20Programmatically)
-
- [Share on Tumblr](https://www.tumblr.com/share)
- [Share on WhatsApp (Opens in new window) WhatsApp](https://srinimf.com/2025/06/19/how-to-read-databricks-job-logs-programmatically/?share=jetpack-whatsapp)
- [More](#)
-

- [Print (Opens in new window) Print](https://srinimf.com/2025/06/19/how-to-read-databricks-job-logs-programmatically/?share=print)
- [Share on Reddit (Opens in new window) Reddit](https://srinimf.com/2025/06/19/how-to-read-databricks-job-logs-programmatically/?share=reddit)
- [Share on Telegram (Opens in new window) Telegram](https://srinimf.com/2025/06/19/how-to-read-databricks-job-logs-programmatically/?share=telegram)
- [Email a link to a friend (Opens in new window) Email](mailto:?subject=%5BShared%20Post%5D%20How%20to%20Read%20Databricks%20Job%20Logs%20Programmatically&body=https%3A%2F%2Fsrinimf.com%2F2025%2F06%2F19%2Fhow-to-read-databricks-job-logs-programmatically%2F&share=email)
-

### Like this:

Like Loading…