Python HTTP Requests with the requests library

pip install requests. requests.get(url) for GET, .post(url, json=data) for POST. .json() parses the response body. .raise_for_status() raises on 4xx/5xx. Sessions for connection reuse and cookies.

The standard library has urllib.request, but requests is the de facto Python HTTP client. Cleaner API, sensible defaults.

A first GET request

import requests

response = requests.get("https://jsonplaceholder.typicode.com/users/1")

print(response.status_code)    # 200
print(response.ok)             # True (status < 400)

user = response.json()         # parse JSON body
print(user["name"])            # "Leanne Graham"

requests.get(url) returns a Response object. Key attributes:

• .status_code — HTTP status (200, 404, ...).
• .ok — convenience: status_code < 400.
• .text — body as a string.
• .json() — body parsed as JSON (raises if not valid JSON).
• .content — raw bytes (for binary responses).
• .headers — dict of response headers.

Query parameters

params = {"userId": 1, "_limit": 3}
response = requests.get(
  "https://jsonplaceholder.typicode.com/posts",
  params=params
)
# Resolves to: https://.../posts?userId=1&_limit=3

Pass a dict as params=. Requests handles URL encoding and assembly. Don't build URLs by string concatenation — params= handles &, ?, encoding, and special characters.

POST: sending JSON

new_post = {
  "title": "Quarterly Report",
  "body": "Revenue up 12%",
  "userId": 1
}

response = requests.post(
  "https://jsonplaceholder.typicode.com/posts",
  json=new_post
)

print(response.status_code)    # 201 Created
print(response.json())          # the created object with an id

json= serializes the dict to JSON, sets Content-Type: application/json, and sends it as the body. Cleaner than data=json.dumps(...).

PUT, PATCH, DELETE

# Replace
requests.put(url, json={...})

# Partial update
requests.patch(url, json={"title": "new title"})

# Delete
requests.delete(url)

Same shape — json= for the body, response object back.

Custom headers

headers = {
  "Accept": "application/json",
  "Authorization": "Bearer abc123",
  "X-Client": "myapp-v1.0",
}

response = requests.get(url, headers=headers)

Most APIs require headers — auth tokens, content negotiation, custom client IDs. The headers= dict merges with defaults.

Form data vs JSON

# JSON body (most APIs)
requests.post(url, json={"key": "value"})

# Form-encoded (older APIs, login forms)
requests.post(url, data={"username": "alice", "password": "..."})

# Raw bytes
requests.post(url, data=b"raw body")

json= for JSON. data= for form-encoded (or raw). Don't mix them.

File uploads

with open("photo.jpg", "rb") as f:
  files = {"upload": f}
  response = requests.post(url, files=files)

# Or with metadata
with open("photo.jpg", "rb") as f:
  files = {"upload": ("photo.jpg", f, "image/jpeg")}
  response = requests.post(url, files=files)

files= sends multipart/form-data — the standard for file upload forms.

Error handling

try:
  response = requests.get(url)
  response.raise_for_status()    # raises HTTPError for 4xx/5xx
  data = response.json()
except requests.exceptions.HTTPError as e:
  print(f"HTTP Error: {e}")
except requests.exceptions.ConnectionError:
  print("Connection failed")
except requests.exceptions.Timeout:
  print("Request timed out")
except requests.exceptions.RequestException as e:
  print(f"Other error: {e}")

raise_for_status() is the cleanest way to check for HTTP errors. Without it, a 404 still gives you a Response — you'd have to check response.ok manually.

RequestException is the parent of all requests exceptions; catch it for "any HTTP problem."

Timeouts

response = requests.get(url, timeout=5)         # 5 seconds total
response = requests.get(url, timeout=(3, 10))   # (connect, read)

Always set a timeout. The default is no timeout — your script can hang forever on a slow server.

For production code: timeout=30 is a reasonable upper bound for most APIs.

Sessions: reuse connections, persist state

session = requests.Session()
session.headers.update({"Authorization": "Bearer abc123"})

response = session.get(url1)
response = session.get(url2)    # reuses TCP connection, sends auth header

A Session:

• Reuses TCP connections (faster for multiple requests).
• Persists cookies across requests (login, then access protected pages).
• Lets you set defaults (headers, params, timeout).

For more than one request to the same domain, always use a session.

Streaming large responses

response = requests.get("https://example.com/huge.zip", stream=True)
with open("file.zip", "wb") as f:
  for chunk in response.iter_content(chunk_size=8192):
    f.write(chunk)

stream=True doesn't download the body until you iterate. Use for large files — avoids loading the whole thing into memory.

Authentication

# Basic auth
requests.get(url, auth=("user", "pass"))

# Bearer token
headers = {"Authorization": "Bearer abc123"}
requests.get(url, headers=headers)

# OAuth, digest auth, etc.
from requests.auth import HTTPDigestAuth
requests.get(url, auth=HTTPDigestAuth("user", "pass"))

For OAuth or complex flows, use requests-oauthlib or authlib.

Retries and backoff

from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

session = requests.Session()
retries = Retry(
  total=3,
  backoff_factor=1,
  status_forcelist=[429, 500, 502, 503, 504],
)
session.mount("https://", HTTPAdapter(max_retries=retries))

response = session.get(url)

For flaky APIs, this retries with exponential backoff. Worth setting up early.

For a higher-level interface, look at tenacity — clean retry decorators for any function.

Inspecting requests

response = requests.get(url, params={"q": "search"})

print(response.url)              # final URL with query string
print(response.request.method)   # GET
print(response.request.headers)  # what we sent
print(response.elapsed)          # how long it took

For debugging, the Response object has the request you made — useful for "did I send what I think I sent?"

httpx: a modern alternative

import httpx

# Same API as requests
r = httpx.get(url)

# Plus: async support
async with httpx.AsyncClient() as client:
  r = await client.get(url)

httpx is API-compatible with requests but adds async, HTTP/2, type hints. For new code that might go async, httpx is a safe choice.

For pure async, use aiohttp (covered briefly in the async lesson).

A real-world fetch

import requests

API_URL = "https://api.github.com"
TOKEN = "ghp_..."

session = requests.Session()
session.headers.update({
  "Authorization": f"token {TOKEN}",
  "Accept": "application/vnd.github.v3+json",
  "User-Agent": "myapp",
})

def get_user(username):
  r = session.get(f"{API_URL}/users/{username}", timeout=10)
  r.raise_for_status()
  return r.json()

def list_repos(username, per_page=30):
  r = session.get(
    f"{API_URL}/users/{username}/repos",
    params={"per_page": per_page, "sort": "updated"},
    timeout=10,
  )
  r.raise_for_status()
  return r.json()

Session, headers, timeout, raise_for_status — production-quality.

Common stumbles

Forgetting timeout=. Default is no timeout. Always set one.

Not checking status. response = requests.get(url) succeeds even for 404. Use raise_for_status() or check response.ok.

json= vs data=. json= sets JSON content-type. data= sends form-encoded. Don't mix.

Building URLs by concatenation. f"{base}?q={query}" breaks on special characters. Use params=.

Reading response.json() without checking content type. If the server returned HTML (like an error page), .json() raises JSONDecodeError.

Not using sessions for multiple requests. Each requests.get(...) opens a new TCP connection. Sessions reuse them.

Streaming response without consuming. stream=True keeps the connection open until you iterate or close(). Use a with block: with requests.get(url, stream=True) as r: ....

SSL verification disabled with verify=False. Disables HTTPS security — dangerous. Only for testing against self-signed certs.

What's next

Lesson 33: web scraping with BeautifulSoup. Parse HTML, extract data, navigate the DOM.

Recap

requests.get(url), .post(url, json=data), etc. Response: .status_code, .ok, .json(), .text, .headers. Always set timeout=. raise_for_status() for clean error handling. Sessions for connection reuse, cookies, default headers. params= for query strings (proper encoding). For streaming downloads, stream=True. Consider httpx for async support.

Next lesson: web scraping.