Google Dataflow API
Browse all Google APIs39
AI ReadinessNon-Ready (F)39/100
Official vendor OpenAPI document · agent-readyAnalyticsData Pipelinesoauth241 EndpointsREST
For Agents
Launch Apache Beam pipelines from templates, monitor job state and worker metrics, and drain streaming jobs so an agent can run and govern Dataflow workloads.
Use for: I need to launch a Dataflow job from a Flex Template, List all currently running streaming jobs in europe-west1, Retrieve the state and metrics of a specific job, Drain a streaming job before redeploying it
Not supported: Does not author Apache Beam code, store data, or schedule recurring runs - use only to launch, monitor, drain, and snapshot Dataflow jobs.
Google Cloud Dataflow API is the control plane for streaming and batch Apache Beam pipelines on Google Cloud. It exposes endpoints to launch jobs from Flex or classic templates, list and inspect jobs and their workers, drain or cancel running streaming jobs, and update the parameters of in-flight pipelines. The API also surfaces job metrics, debug snapshots, and template metadata so platform teams can build dashboards and self-service launchers without bespoke Beam code. Jobs run on managed worker VMs in the chosen region, with autoscaling controlled per job.
Install Jentic One Beta
Connect the Dataflow API to your agent
Jentic One is a self-hosted execution layer for AI agents. It lets your agent call the Dataflow API, or any other public or private API you need. You set the rules, the agent never sees your credentials, and every call is logged.
Two steps, two machines. Install the instance in a safe environment, then register your agent from wherever it runs.
1
Step 1: Jentic One Host machine
# On the machine that will host your Jentic One instance:
curl -fsSL "https://jentic.com/install.sh?src=apis&api=%2Fapis%2Fgoogleapis.com%2Fdataflow" | sh2
Step 2: Agent machine
# On the machine where your agent runs (keep this separate from the instance):
curl -fsSL "https://jentic.com/install.sh?src=apis&api=%2Fapis%2Fgoogleapis.com%2Fdataflow" | sh
jentic register # connects your agent to your Jentic One instanceJentic One is in public beta. The setup above keeps your agent separate from the instance, which is what you want before using real credentials: an agent running as the same OS user as Jentic One can read its stored keys directly. Just evaluating? A single local install is fine to start. See the secure deployment guide for the tiers.
Capabilities
What an agent can do with Dataflow API.
Launch a Dataflow job from a Flex Template or classic template with custom parameters
List jobs in a project and region with filters by state and creation time
Retrieve job graph, current state, and per-stage metrics
Drain or cancel a running streaming job to release resources gracefully
Update parameters or transformation graph of an in-flight pipeline
Snapshot a streaming job for replay or rollback
List supported Flex Templates and inspect their input parameter metadata
Use Cases
Patterns agents use Dataflow API for, with concrete tasks.
★ Self-Service Pipeline Launch
A data platform team publishes Flex Templates for common ETL shapes and a portal calls the Dataflow API to launch them with user-supplied parameters. Each launch returns a job ID and immediate state, then the portal polls the job for progress. End users start a pipeline without touching gcloud or Beam code.
Launch the Flex Template at gs://templates/etl-template-spec.json in us-central1 with parameters input='gs://raw/2026-06-10' and output='bq:proj.ds.facts'.
Streaming Job Lifecycle Management
A SRE runbook drains a streaming job, deploys a new version, and snapshots the previous state for rollback. The Dataflow API exposes drain and snapshot operations so the runbook completes without manual Console steps. Drains preserve in-flight messages instead of dropping them.
Drain streaming job 'jid-2026-06-10-abc' in us-central1 and create a snapshot named 'pre-deploy-snap' before launching the new version.
Failed Job Triage Bot
A Slack bot monitors Dataflow for jobs entering JOB_STATE_FAILED and posts the job ID, error log link, and last successful checkpoint. The Dataflow API supplies job state, current workers, and error messages so the bot can surface high-signal context. Engineers click through to logs only when needed.
List jobs in us-central1 with stateFilter=ACTIVE created in the last 24 hours, return any whose currentState is JOB_STATE_FAILED with their error messages.
Capacity and Cost Telemetry
A FinOps dashboard collects per-job worker counts, vCPU hours, and shuffled data from the Dataflow API metrics endpoint and aggregates them per team. Teams see cost drivers without opening individual job pages. Decisions to right-size workers are grounded in actual Dataflow telemetry.
Get the metrics for job 'jid-2026-06-10-xyz' and return totalVcpuTime and currentNumWorkers.
AI Agent Pipeline Operator
An on-call AI agent gets a 'job lagging' alert, asks Jentic for the Dataflow API operations needed, retrieves current watermark and worker count, and drains/relaunches the job with more workers. Jentic isolates the Google service account credential so raw keys never enter the agent context.
For job 'jid-2026-06-10-zzz', get its watermark and worker count, drain if watermark lag exceeds 5 minutes, and relaunch the same Flex Template with maxWorkers doubled.
Key Endpoints
41 endpoints — google cloud dataflow api is the control plane for streaming and batch apache beam pipelines on google cloud.
METHOD
PATH
DESCRIPTION
GET
/v1b3/projects/{projectId}/jobs
List jobs in a project
GET
/v1b3/projects/{projectId}/jobs/{jobId}
Get a specific job's state and graph
POST
/v1b3/projects/{projectId}/locations/{location}/flexTemplates:launch
Launch a job from a Flex Template
POST
/v1b3/projects/{projectId}/jobs/{jobId}/debug/sendCapture
Send a debug capture for a job
GET
/v1b3/projects/{projectId}/jobs/{jobId}/debug/getConfig
Get the debug config for a job
GET
/v1b3/projects/{projectId}/jobs
List jobs in a project
GET
/v1b3/projects/{projectId}/jobs/{jobId}
Get a specific job's state and graph
POST
/v1b3/projects/{projectId}/locations/{location}/flexTemplates:launch
Launch a job from a Flex Template
POST
/v1b3/projects/{projectId}/jobs/{jobId}/debug/sendCapture
Send a debug capture for a job
GET
/v1b3/projects/{projectId}/jobs/{jobId}/debug/getConfig
Get the debug config for a job
Jentic AI Readiness Score
This API is usable in Jentic One now. Its AI-readiness score against Jentic's framework shows where it stands today and where improvements would make it even easier for agents to use.
D
Dataflow API
- Non-Ready (F)39/100
41
Foundational Compliance56
Developer Experience & Jentic Compatibility20
AI-Readiness & Agent Experience94
Agent Usability63
Security47
AI DiscoverabilityPowered by JenticScoring Framework 1.0.0 | Scoring Engine 0.4.0
Show dimension breakdown
41
Foundational Compliance
Base layer of spec validity and structural soundness.
Grade: D-Signals: 4
62%
Lint Results
Aggregated quality score from linter diagnostics, weighted by severity.
100%
Resolution Completeness
Percentage of `$ref` references that resolve successfully.
0%
Specification Validity
Checks whether the API description parses successfully and conforms to its declared specification (e.g., OpenAPI).
0%
Structural Integrity
Structural correctness score based on schema issues using logarithmic dampening.
56
Developer Experience & Jentic Compatibility
Clarity, completeness, and ingestion readiness for developers and tooling.
Grade: CSignals: 4
0%
Example Density
How richly the API is illustrated with examples.
100%
Example Validity
Percentage of examples that conform to their schemas.
25%
Response Coverage
Percentage of operations with complete response definitions (success, client error, server error).
100%
Tooling Readiness
Health of API ingestion, bundling, and resolution within Jentic pipelines.
20
AI-Readiness & Agent Experience
Semantic breadth, depth, and agent comprehension for AI systems.
Grade: FSignals: 4
80%
Description Coverage
Coverage of descriptions across API elements.
0%
Error Standardization
Coverage of RFC 9457 Problem Details for error responses.
0%
OperationId Quality
Coverage, uniqueness, and casing consistency of operationIds for AI inference.
0%
Summary Coverage
Coverage of summaries across operations/tags/info.
94
Agent Usability
Functional utility, complexity comfort, and AI orchestration readiness.
Grade: A+Signals: 1
94%
Complexity Comfort
Agent comfort level based on API operational and structural complexity.
63
Security
Trust, risk posture, and security compliance.
Grade: B-Signals: 1
62%
Authentication Strength
Average quality of security schemes based on authentication method strength (weakest link for OAuth2).
47
AI Discoverability
Findability, semantic richness, and reasoning readiness.
Grade: DSignals: 1
47%
Descriptive Richness
Clarity and depth of descriptions across API elements.
Score it yourself
Every API in the directory is allowlisted, so you can re-score it with no key required.
npx @jentic/api-scorecard-cli score <openapi-url>Why Jentic?
What agents get from Jentic-routed access to this vendor.
Setup
Wiring the Dataflow API by hand means setting up Google OAuth, exchanging a service account for short-lived cloud-platform tokens instead of shipping a JSON key, and building project, location, and job paths against dataflow.googleapis.com. Through Jentic you install once, import the Dataflow API from the API Directory, store the Google credential once, and your agent calls it.
Permission scoping
The API puts the project, location, and job id in the URL path (/v1b3/projects/{projectId}/jobs/{jobId} and /v1b3/projects/{projectId}/locations/{location}/...), so a rule can pin your agent to one project, location, or job. You choose the operations it may call, so listing jobs and launching flex templates can be allowed while a job state change is not included unless you add it.
Credential isolation
Your Google service account credential is stored once, encrypted, by your own Jentic One instance and injected at execution time. It never enters the agent's prompt, logs, or context.
Intent-based discovery
Agents search Jentic by intent such as 'launch a Dataflow pipeline', and Jentic returns the matching Dataflow operation with its input schema, including parameter maps and requested state values, so the agent calls the right endpoint without browsing the reference docs.
Related APIs
Alternatives and complements available in the Jentic catalogue.
FAQs
Specific to using Dataflow API through Jentic.
What authentication does the Google Cloud Dataflow API use?
The Dataflow API uses Google OAuth 2.0 with the cloud-platform scope. Through Jentic the service account credentials are stored encrypted in your Jentic One instance and the agent receives scoped, short-lived access tokens per request.
Can I launch a Dataflow job from a Flex Template using only the Dataflow API?
Yes. Call /v1b3/projects/{projectId}/locations/{location}/flexTemplates:launch with the template GCS spec URI and a parameters map. The response contains the new job's ID and current state, and you can poll the job endpoint until it reaches JOB_STATE_RUNNING.
What are the rate limits for the Google Cloud Dataflow API?
Dataflow applies per-project quotas on jobs.create and jobs.list, plus per-region worker and vCPU quotas that limit how many concurrent jobs can run. Inspect the Cloud Console Quotas page for the precise limits in your project.
How do I drain a streaming job through Jentic?
Run pip install jentic, search Jentic for 'drain dataflow streaming job', load the schema for jobs.update on dataflow.googleapis.com, and execute it with the job ID and requestedState=JOB_STATE_DRAINED.
Does the Dataflow API let me write or compile Apache Beam code?
No. This API is a control plane: launching, monitoring, draining, and snapshotting jobs whose Beam graph is supplied as a template or compiled artifact. Beam pipeline development happens locally with the Beam SDK before publishing a template.
Can I limit what my agent is allowed to do with the Google Cloud Dataflow API?
Yes. Because you run Jentic One yourself, your own rules decide which Dataflow operations and credentials the agent may use. Since the API carries the project, location, and job id in the URL path (such as /v1b3/projects/{projectId}/jobs/{jobId} and /v1b3/projects/{projectId}/locations/{location}/...), you can pin the agent to a single project, location, or job. You also choose the operations it may call, so listing jobs and launching Flex Templates can be allowed while a job state change stays off unless you explicitly add it.
GET STARTED