Data Engineering Agent Scheduling and Publishing Guide
This guide covers how to use the Data Engineering Agent to configure Studio task scheduling, publish tasks to the scheduling system, and unpublish after publishing. It focuses on the distinction between "save scheduling configuration" and "publish to the scheduling system."
When to Use This
When a Studio task has completed draft development and is ready to run automatically on a schedule, use the Data Engineering Agent to help check, configure, or prepare for:
- Configuring cron scheduling intervals
- Configuring failure retry count and timeout
- Configuring or checking upstream/downstream dependencies
- Publishing tasks to the scheduling system
- Checking whether a task is published
- Querying the next scheduled run time
- Pausing or unpublishing tasks
Scheduling and publishing affect whether a task runs automatically — these are change operations. Whether they can be executed directly depends on current permissions, open tool capabilities, and workspace configuration. Before executing, have the Agent explain the impact scope and confirm with you.
Explore First, Then Enter Scheduling and Publishing
Scheduling and publishing are not suited to immediate execution.
The more natural approach is:
- Confirm whether the task already meets publishing conditions
- Determine whether what's missing is scheduling parameters, dependencies, or the actual publishing action
- Once configuration and impact scope are clear, actually save the schedule or publish
Better exploratory opening questions:
- Help me check whether this task currently meets publishing conditions.
- Help me check what this task is currently missing: cron, dependencies, VCluster, or retry and timeout config.
- Help me check whether this task is currently a draft, has saved scheduling, or is already published.
When these states are clear, saving the schedule, publishing, or unpublishing is more stable.
Three Phases of Scheduling and Publishing
Task Draft
A task draft is a task file in the Studio IDE. Drafts can contain SQL, Python, Shell, and other code, but they do not enter the scheduling system automatically.
A draft typically means:
- The task appears in the specified task directory
- Task code is saved
- Task is not published
- Will not run automatically on schedule
- Creating a draft does not produce run instances
Save Scheduling Configuration
Saving scheduling configuration writes cron, retry, timeout, compute cluster, and other settings into task metadata — but this does not mean the task is published. After saving, the task has scheduling parameters but still does not run automatically.
Only after the publishing action will the task enter the scheduling system.
Publish Task
Publishing submits the task to the scheduling system. After publishing, the task typically waits for subsequent cron triggers. Whether it runs immediately depends on the frontend response and scheduling system feedback.
Before publishing, confirm the SQL type, target table, compute cluster, scheduling time, retry strategy, timeout, and upstream/downstream dependencies.
Recommended Workflow
Confirm the Task Draft First
Before publishing, have the Agent check the task content.
Recommended question:
Key checks:
- Whether the task is in the correct directory
- Whether the SQL matches expectations
- Whether SQL will write or overwrite data
- Whether the schema is correct
- Whether the compute cluster is correct
- Whether upstream/downstream dependencies are needed
If this information is not yet clear, ask first:
Save Scheduling Configuration Without Publishing
Configure scheduling parameters first, then check the configuration result.
Recommended question:
The Agent should explain:
- Which task will be modified
- The cron expression
- Failure retry count
- Timeout
- VCluster
- Whether there are upstream/downstream dependencies
- Whether saving the configuration enters the scheduling system
- Whether run instances will be produced
After saving, confirm again:
Confirm Impact Before Publishing
Before publishing, have the Agent do one dedicated confirmation.
Recommended question:
Key checks in the pre-publishing confirmation:
- Whether the publish target is correct
- Whether the task SQL still matches expectations
- Whether the VCluster is correct
- Whether the cron matches expectations
- Whether it will run immediately
- Whether the next scheduled run time is reasonable
- How to pause or unpublish
Confirm and Publish
After confirming everything, have the Agent publish.
Recommended question:
After publishing, the task typically enters the scheduling system and waits for subsequent cron triggers. Whether run instances are immediately produced depends on the frontend response and scheduling system feedback. Confirm:
- Whether the task publish state has been updated
- Whether the next scheduled run time was returned
Unpublish or Pause Scheduling
After publishing, if subsequent automatic runs need to stop, choose to pause or unpublish.
Common operations:
| Operation | Typical meaning | When to use |
|---|---|---|
| pause | Typically pauses subsequent schedule triggers; actual state change depends on product feedback | Temporarily stop automatic runs |
| undeploy | Typically removes from the scheduling system; how to resume depends on product feedback | Cleaning up test tasks, taking tasks offline |
Recommended question:
After confirmation:
After unpublishing, whether the task has been removed from the scheduling system, whether it will still trigger automatically, and whether drafts and configuration are preserved — rely on backend return results and interface state, not just the action name.
FAQ
I saved a cron, but why isn't the task running?
Saving a cron only writes it into task configuration. The task must be published to the scheduling system before it runs automatically per the cron.
Will the task run immediately after publishing?
Typically not, but rely on the frontend response and scheduling system feedback. If you need immediate validation, execute the task separately or use a read-only query to verify — these are separate actions from publishing.
Can I unpublish a task after publishing?
Typically yes. You can confirm whether pause is supported to pause scheduling, or use undeploy to unpublish. The specific state changes and what is preserved depend on backend results and interface state.
Does unpublishing delete the task?
Typically it does not directly delete the task draft, but rely on backend results and interface state. Deleting a draft should be treated as a separate operation — do not conflate it with unpublishing.
Why should I verify the VCluster?
A scheduled task actually uses the compute cluster in its task configuration. Before publishing, verify the VCluster in both the task details returned by the Agent and the current Studio configuration, to avoid running the task on the wrong compute cluster.
How should test tasks be cleaned up?
For test tasks, follow this order:
- If published, unpublish first (
undeploy) - Confirm the task has been removed from the scheduling system and the next scheduled run is cancelled
- Then delete the draft from the task tree
- After deletion, confirm the task node no longer appears in the task directory