Python requests: how do I retry a failing HTTP request with backoff?
Asked 7d agoby IT-QA·1 answer
pythonhttprequestsresilience
My script hits a flaky API that occasionally returns 503 or times out. I want it to retry a few times with increasing delay instead of failing on the first hiccup. What's the clean way with the requests library?
1 Answer
AIIT-QA Assistant·7d ago
*AI-drafted answer — reviewed for correctness. Please verify against your own environment and versions; corrections and better answers are welcome below.*
Don't hand-roll a retry loop — mount an `HTTPAdapter` with `urllib3`'s `Retry`, which handles backoff, status codes, and connection errors for you:
```python
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
retry = Retry(
total=5, # up to 5 attempts
backoff_factor=0.5, # 0.5s, 1s, 2s, 4s ... between tries
status_forcelist=[429, 500, 502, 503, 504], # retry on these
allowed_methods=["GET", "POST"], # opt POST in explicitly
respect_retry_after_header=True, # honor server's Retry-After
)
session = requests.Session()
session.mount("https://", HTTPAdapter(max_retries=retry))
session.mount("http://", HTTPAdapter(max_retries=retry))
resp = session.get("https://flaky.example.com/data", timeout=10)
resp.raise_for_status()
```
Key points:
- **`backoff_factor`** gives exponential delays: the nth retry waits `backoff_factor * (2 ** (n-1))` seconds, so 0.5 → 0.5s, 1s, 2s, 4s. This spacing is what makes retries help instead of hammering a struggling server.
- **Always set `timeout`** on the request itself — retries don't help if a single attempt hangs forever.
- **Only retry idempotent operations** by default. `POST` isn't safe to blindly retry unless the endpoint is idempotent (e.g. protected by an idempotency key); that's why `allowed_methods` is explicit.
- For per-call control or non-HTTP flakiness, the `tenacity` library gives a decorator (`@retry(wait=wait_exponential(), stop=stop_after_attempt(5))`).