Skip to main content

Key Concepts

Bacalhau is built around a few core ideas and terminologies. If you're new to Bacalhau, here's what you need to know:

Distributed Compute Orchestration​

Bacalhau coordinates computing workloads across a network of machines, intelligently matching jobs to resources.

  • Bacalhau acts as a dispatcher: You submit jobs (e.g., container workloads), and it finds the best node to run them based on available resources, data location, and constraints.

Bring the Compute to the Data​

Instead of moving data to compute, Bacalhau moves compute to where data lives, reducing network overhead and improving efficiency.

  • Traditionally, big data solutions shuffle large datasets across networks to a central compute cluster.
  • Bacalhau inverts this approach: it places compute tasks where the data already resides—whether in local storage, an S3 bucket, or other storage providers—reducing unnecessary data movement.

Jobs & Executions​

Bacalhau organizes work in a hierarchy that enables efficient resource allocation and parallelization.

  • A Job defines the overall workflow (e.g., "run a Docker image with these arguments").
  • A job can be broken into multiple Executions that run in parallel across different compute nodes.
  • Bacalhau optimizes these executions based on data locality and available resources.

Job Types​

Bacalhau supports various execution patterns to accommodate different workload requirements:

  • Batch Jobs: One-time execution of a workload, typically for data processing tasks that run to completion.
  • Ops Jobs: Administrative or operational tasks, often for system maintenance or monitoring.
  • Daemon Jobs: Long-running background processes that perform ongoing work.
  • Service Jobs: Web services or APIs that need to remain available and respond to requests.

Node Types​

The Bacalhau network consists of specialized components, each with specific responsibilities:

  • Orchestrator Node: Receives job submissions, schedules executions, and monitors state. Started with bacalhau serve --orchestrator.
  • Compute Node: Executes workloads locally, typically requiring Docker or another runtime. Started with bacalhau serve --compute.
  • Hybrid Node: Serves both roles at once—often used for local dev or small setups. Started with bacalhau serve --orchestrator --compute.

Execution Engines​

Bacalhau runs your code through pluggable runtime environments:

  • Bacalhau supports multiple execution engines through its modular architecture:
    • Docker: For container-based workloads
    • WebAssembly (WASM): For lightweight, sandboxed execution
  • The framework is designed to accommodate additional engines as needed.

Storage Providers​

Bacalhau can access data from various sources through a clean, extensible interface:

  • Bacalhau can mount data from various sources through its flexible storage provider interface:
    • S3-compatible storage
    • HTTP/HTTPS URLs
    • Local filesystems
    • IPFS
    • And more via storage provider plugins

Publisher​

After execution, Bacalhau ensures your results are accessible where you need them:

  • After a job finishes, its results can be published to a specific backend—like local disk, S3 or IPFS—so they're easy to retrieve.

Communication Layer​

A reliable messaging system allows Bacalhau components to coordinate effectively:

  • Bacalhau uses NATS.io as its communication backbone:
    • Orchestrators act as NATS servers
    • Compute nodes connect as NATS clients
    • This provides reliable, scalable messaging between components