ReviseAlgo Logo

Concurrency

concurrent.futures

High-level concurrency API — ThreadPoolExecutor and ProcessPoolExecutor for simplified parallel execution with futures.

Interview: Practical concurrency — the easiest way to parallelize work in Python, frequently used in production code.

Last Updated: June 12, 2026 • 9 min read

The concurrent.futures module provides a high-level interface for asynchronously executing callables. It wraps threading and multiprocessing with a simple, uniform API based on executors and futures. This is the recommended way to parallelize work in most Python applications.

Executor Types

  • ThreadPoolExecutor: Uses threads — best for I/O-bound tasks (HTTP, file I/O, DB queries)
  • ProcessPoolExecutor: Uses processes — best for CPU-bound tasks (computation, data processing)
  • Both share the same interface — easy to switch between them
  • max_workers controls parallelism level (default: min(32, os.cpu_count() + 4) for threads)

Key Methods

  • executor.submit(fn, *args) — schedule a single callable, returns a Future
  • executor.map(fn, *iterables) — apply function to each item, returns results in order
  • concurrent.futures.as_completed(futures) — yields futures as they complete
  • concurrent.futures.wait(futures) — wait for all (or first) to complete

Future Objects

  • future.result(timeout) — block until result is ready
  • future.done() — check if the future has completed
  • future.cancel() — attempt to cancel (only works if not yet started)
  • future.exception() — get the exception if the callable raised one
  • future.add_done_callback(fn) — register a callback for when the future completes

Interview Insight

concurrent.futures is the "right way" to do parallel work in Python. It abstracts away thread/process management. Know when to use ThreadPoolExecutor (I/O-bound) vs ProcessPoolExecutor (CPU-bound). The map() method preserves input order; as_completed() processes results as they arrive.

Use Cases

Parallel HTTP requests — ThreadPoolExecutor for web scraping, API calls

Data processing — ProcessPoolExecutor for CPU-intensive transformations

File I/O — ThreadPoolExecutor for parallel file reading/writing

Map-reduce pattern — split data, process in parallel, aggregate results

Batch operations — sending emails, generating reports, image processing

Common Mistakes

Using ThreadPoolExecutor for CPU-bound work — GIL prevents true parallelism; use ProcessPoolExecutor

Using ProcessPoolExecutor for I/O-bound work — process creation overhead outweighs benefits

Not using if __name__ == "__main__" — required on Windows/macOS for ProcessPoolExecutor

Forgetting to call result() — exceptions in futures are silently ignored until accessed

Setting max_workers too high — too many threads/processes causes overhead; tune to your workload