PipeDream

Your pipeline.
Your infrastructure.
Room to scale.

A federated workflow coordinates work across independently managed compute and storage. PipeDream brings that orchestration to your high-performance computing (HPC) clusters, with managed capacity when you need it.

Add managed compute when you need it, with the same workflow API, policies, and run tracking.

Extend capacity: workflows across your HPC infrastructureAdd managed compute when you need it, with the same workflow API, policies, and run tracking. Applications submit workflow documents with steps, compute requirements, and input references. Input files are retrieved separately. Green pipes show logical data connections; PipeDream currently stages and relays compute transfers. The animation illustrates the architecture, not live activity.WorkflowData accessInputs & outputsYour applicationsCustomer A · Customer BTenant-scoped workflowsPipeDreamRoute, schedule, and trackPolicies & resource limitsYour HPC clusterYour infrastructureManaged computeAdd capacity as neededYour storageAuthorized inputs & outputsExtend capacity: workflows across your HPC infrastructureAdd managed compute when you need it, with the same workflow API, policies, and run tracking. Applications send workflow documents, while input files are retrieved separately from storage. Green pipes show logical data connections; PipeDream currently stages and relays compute transfers. The animation illustrates the architecture, not live activity.WorkflowYour applicationsCustomer A · Customer BPipeDreamRoute, schedule, and trackPolicies & resource limitsYour HPCclusterYour ownManagedcomputeAdd capacityYour storageAuthorized inputs & outputsAuthorized inputs& outputs
Workflow orchestrationData access & staging

Workflow documents carry steps, compute requirements, and input references. PipeDream retrieves input files separately and stages authorized data transfers.

Start with your HPC cluster.

Connect the high-performance computing infrastructure you already operate. Your application submits a workflow document; PipeDream coordinates preparation, execution, and results. Add managed compute destinations as your workload grows.

Keep control of your data.

Keep source datasets in storage you control. PipeDream stages authorized inputs for the selected compute, prepares the environment, and tracks the outputs. Access follows the storage connections and permissions you configure.

Built for your customers, too.

Give each customer its own organization and project context. Set cluster access and resource limits, then track jobs, logs, and artifacts by tenant. Policies and audit history make execution accountable across your application.

Your application owns the experience.
PipeDream handles the execution.

Open console