Activities
This guide provides a comprehensive overview of Temporal Activities.
In day-to-day conversations, the term Activity frequently denotes either an Activity Definition What is an Activity Definition? An Activity Definition is the code that defines the constraints of an Activity Task Execution. What is an Activity Type? An Activity Type is the mapping of a name to an Activity Definition. What is an Activity Execution? An Activity Execution is the full chain of Activity Task Executions.
An Activity is a normal function or object method that executes a single, well-defined action (either short or long running), such as calling another service, transcoding a media file, or sending an email message.
Workflow code orchestrates the execution of Activities, persisting the results. If an Activity Function Execution fails, any future execution starts from initial state (except Heartbeats). Therefore, an Activity function is allowed to contain any code without restrictions.
Activity Functions are executed by Worker Processes.
When the Activity Function returns, the Worker sends the results back to the Temporal Cluster as part of the ActivityTaskCompleted Event.
The Event is added to the Workflow Execution's Event History.
Activity Definition
An Activity Definition is the code that defines the constraints of an Activity Task Execution What is an Activity Task Execution? An Activity Task Execution is the execution of an Activity Type.
The term 'Activity Definition' is used to refer to the full set of primitives in any given language SDK that provides an access point to an Activity Function Definition——the method or function that is invoked for an Activity Task Execution What is an Activity Task Execution? An Activity Task Execution is the execution of an Activity Type.
Activity Definitions are named and referenced in code by their Activity Type What is an Activity Type? An Activity Type is the mapping of a name to an Activity Definition.
Activity Definition
Constraints
Activity Definitions are executed as normal functions.
In the event of failure, the function begins at its initial state when retried (except when Activity Heartbeats are established).
Therefore, an Activity Definition has no restrictions on the code it contains.
Parameters
An Activity Definition can support as many parameters as needed.
All values passed through these parameters are recorded in the Event History What is an Event History? An append log of Events that represents the full state a Workflow Execution.
Activity Definitions must contain the following parameters:
- Context: an optional parameter that provides Activity context within multiple APIs.
- Heartbeat: a notification from the Worker to the Temporal Cluster that the Activity Execution is progressing. Cancelations are allowed only if the Activity Definition permits Heartbeating.
- Timeouts: intervals that control the execution and retrying of Activity Task Executions.
Other parameters, such as Retry Policies What is a Retry Policy? A Retry Policy is a collection of attributes that instructs the Temporal Server how to retry a failure of a Workflow Execution or an Activity Task Execution.
Activity Type
An Activity Type is the mapping of a name to an Activity Definition.
Activity Types are scoped through Task Queues.
Activity Execution
An Activity Execution is the full chain of Activity Task Executions What is an Activity Task Execution? An Activity Task Execution is the execution of an Activity Type.
Activity Execution
An Activity Execution has no time limit. Activity Execution time limits and retries can be optimized for each situation within the Temporal Application.
If for any reason an Activity Execution does not complete (exhausts all retries), the error is returned to the Workflow, which decides how to handle it.
Request Cancellation
A Workflow can request to cancel an Activity Execution.
When an Activity Execution is canceled, or its Workflow Execution has completed or failed, the context passed into its function is canceled, which also sets its channel’s closed state to Done.
An Activity can use that to perform any necessary cleanup and abort its execution.
Cancellation requests are only delivered to Activity Executions that Heartbeat:
- The Heartbeat request fails with a special error indicating that the Activity Execution is canceled. Heartbeats can also fail when the Workflow Execution that spawned it is in a completed state.
- The Activity should perform all necessary cleanup and report when it is done.
- The Workflow can decide if it wants to wait for the Activity cancellation confirmation or proceed without waiting.
Activity Id
The identifier for an Activity Execution What is an Activity Execution? An Activity Execution is the full chain of Activity Task Executions.
An Activity Id can be used to complete the Activity asynchronously.
Schedule-To-Start Timeout
A Schedule-To-Start Timeout is the maximum amount of time that is allowed from when an Activity Task What is an Activity Task? An Activity Task contains the context needed to make an Activity Task Execution. What is a Worker? In day-to-day conversations, the term Worker is used to denote both a Worker Program and a Worker Process. Temporal documentation aims to be explicit and differentiate between them.
How to set a Schedule-To-Start Timeout
The moment that the Task is picked by the Worker from the Task Queue is considered to be the start of the Activity Task for the purposes of the Schedule-To-Start Timeout and associated metrics. This definition of "Start" avoids issues that a clock difference between the Temporal Cluster and a Worker might create.
Schedule-To-Start Timeout period
"Schedule" in Schedule-To-Start and Schedule-To-Close have different frequency guarantees.
The Schedule-To-Start Timeout is enforced for each Activity Task, whereas the Schedule-To-Close Timeout is enforced once per Activity Execution. Thus, "Schedule" in Schedule-To-Start refers to the scheduling moment of every Activity Task in the sequence of Activity Tasks that make up the Activity Execution, while "Schedule" in Schedule-To-Close refers to the first Activity Task in that sequence.
A Retry Policy What is a Retry Policy? A Retry Policy is a collection of attributes that instructs the Temporal Server how to retry a failure of a Workflow Execution or an Activity Task Execution.
Start-To-Close Timeout period with retries
This timeout has two primary use cases:
- Detect whether an individual Worker has crashed.
- Detect whether the fleet of Workers polling the Task Queue is not able to keep up with the rate of Activity Tasks.
The default Schedule-To-Start Timeout is ∞ (infinity).
If this timeout is used, we recommend setting this timeout to the maximum time a Workflow Execution is willing to wait for an Activity Execution in the presence of all possible Worker outages, and have a concrete plan in place to reroute Activity Tasks to a different Task Queue. This timeout does not trigger any retries regardless of the Retry Policy, as a retry would place the Activity Task back into the same Task Queue. We do not recommend using this timeout unless you know what you are doing.
In most cases, we recommend monitoring the temporal_activity_schedule_to_start_latency metric to know when Workers slow down picking up Activity Tasks, instead of setting this timeout.
Start-To-Close Timeout
A Start-To-Close Timeout is the maximum time allowed for a single Activity Task Execution What is an Activity Task Execution? An Activity Task Execution is the execution of an Activity Type.
The default Start-To-Close Timeout is the same as the default Schedule-To-Close Timeout What is a Schedule-To-Close Timeout? A Schedule-To-Close Timeout is the maximum amount of time allowed for the overall Activity Execution, from when the first Activity Task is scheduled to when the last Activity Task, in the chain of Activity Tasks that make up the Activity Execution, reaches a Closed status.
An Activity Execution must have either this timeout (Start-To-Close) or the Schedule-To-Close Timeout What is a Schedule-To-Close Timeout? A Schedule-To-Close Timeout is the maximum amount of time allowed for the overall Activity Execution, from when the first Activity Task is scheduled to when the last Activity Task, in the chain of Activity Tasks that make up the Activity Execution, reaches a Closed status. What is an Activity Heartbeat? An Activity Heartbeat is a ping from the Worker that is executing the Activity to the Temporal Cluster. Each ping informs the Temporal Cluster that the Activity Execution is making progress and the Worker has not crashed. What is a Heartbeat Timeout? A Heartbeat Timeout is the maximum time between Activity Heartbeats.
The main use case for the Start-To-Close timeout is to detect when a Worker crashes after it has started executing an Activity Task.
Start-To-Close Timeout period
A Retry Policy What is a Retry Policy? A Retry Policy is a collection of attributes that instructs the Temporal Server how to retry a failure of a Workflow Execution or an Activity Task Execution.
If the first Activity Task Execution returns an error the first time, then the full Activity Execution might look like this:
Start-To-Close Timeout period with retries
If this timeout is reached, the following actions occur:
- An ActivityTaskTimedOut Event is written to the Workflow Execution's mutable state.
- If a Retry Policy dictates a retry, the Temporal Cluster schedules another Activity Task.
- The attempt count increments by 1 in the Workflow Execution's mutable state.
- The Start-To-Close Timeout timer is reset.
Schedule-To-Close Timeout
A Schedule-To-Close Timeout is the maximum amount of time allowed for the overall Activity Execution What is an Activity Execution? An Activity Execution is the full chain of Activity Task Executions. What is an Activity Task? An Activity Task contains the context needed to make an Activity Task Execution.
Schedule-To-Close Timeout period
Example Schedule-To-Close Timeout period for an Activity Execution that has a chain Activity Task Executions:
Schedule-To-Close Timeout period with a retry
The default Schedule-To-Close Timeout is ∞ (infinity).
An Activity Execution must have either this timeout (Schedule-To-Close) or Start-To-Close What is a Start-To-Close Timeout? A Start-To-Close Timeout is the maximum time allowed for a single Activity Task Execution.
Activity Heartbeat
An Activity Heartbeat is a ping from the Worker that is executing the Activity to the Temporal Cluster. Each ping informs the Temporal Cluster that the Activity Execution is making progress and the Worker has not crashed.
Activity Heartbeats work in conjunction with a Heartbeat Timeout What is a Heartbeat Timeout? A Heartbeat Timeout is the maximum time between Activity Heartbeats.
Activity Heartbeats are implemented within the Activity Definition. Custom progress information can be included in the Heartbeat which can then be used by the Activity Execution should a retry occur.
An Activity Heartbeat can be recorded as often as needed (e.g. once a minute or every loop iteration). It is often a good practice to Heartbeat on anything but the shortest Activity Function Execution. Temporal SDKs control the rate at which Heartbeats are sent to the Cluster.
Heartbeating is not required from Local Activities What is a Local Activity? A Local Activity is an Activity Execution that executes in the same process as the Workflow Execution that spawns it.
For long-running Activities, we recommend using a relatively short Heartbeat Timeout and a frequent Heartbeat. That way if a Worker fails it can be handled in a timely manner.
A Heartbeat can include an application layer payload that can be used to save Activity Execution progress.
If an Activity Task Execution What is an Activity Task Execution? An Activity Task Execution is the execution of an Activity Type.
Activity Cancellations are delivered to Activities from the Cluster when they Heartbeat. Activities that don't Heartbeat can't receive a Cancellation. Heartbeat throttling may lead to Cancellation getting delivered later than expected.
Throttling
Heartbeats may not always be sent to the Cluster—they may be throttled by the Worker. The throttle interval is the smaller of the following:
- If
heartbeatTimeoutis provided,heartbeatTimeout * 0.8; otherwise,defaultHeartbeatThrottleInterval maxHeartbeatThrottleInterval
defaultHeartbeatThrottleInterval is 30 seconds by default, and maxHeartbeatThrottleInterval is 60 seconds by default.
Each can be set in Worker options.
Throttling is implemented as follows:
- After sending a Heartbeat, the Worker sets a timer for the throttle interval.
- The Worker stops sending Heartbeats, but continues receiving Heartbeats from the Activity and remembers the most recent one.
- When the timer fires, the Worker:
- Sends the most recent Heartbeat.
- Sets the timer again.
Which Activities should Heartbeat?
Heartbeating is best thought about not in terms of time, but in terms of "How do you know you are making progress?" For short-term operations, progress updates are not a requirement. However, checking the progress and status of Activity Executions that run over long periods is almost always useful.
Consider the following when setting Activity Hearbeats:
Your underlying task must be able to report definite progress. Note that your Workflow cannot read this progress information while the Activity is still executing (or it would have to store it in Event History). You can report progress to external sources if you need it exposed to the user.
Your Activity Execution is long-running, and you need to verify whether the Worker that is processing your Activity is still alive and has not run out of memory or silently crashed.
For example, the following scenarios are suitable for Heartbeating:
- Reading a large file from Amazon S3.
- Running a ML training job on some local GPUs.
And the following scenarios are not suitable for Heartbeating:
- Making a quick API call.
- Reading a small file from disk.
Heartbeat Timeout
A Heartbeat Timeout is the maximum time between Activity Heartbeats What is an Activity Heartbeat? An Activity Heartbeat is a ping from the Worker that is executing the Activity to the Temporal Cluster. Each ping informs the Temporal Cluster that the Activity Execution is making progress and the Worker has not crashed.
Heartbeat Timeout periods
If this timeout is reached, the Activity Task fails and a retry occurs if a Retry Policy What is a Retry Policy? A Retry Policy is a collection of attributes that instructs the Temporal Server how to retry a failure of a Workflow Execution or an Activity Task Execution.
Asynchronous Activity Completion
Asynchronous Activity Completion is a feature that enables an Activity Function to return without causing the Activity Execution to complete. The Temporal Client can then be used to both Heartbeat Activity Execution progress and eventually provide a result.
When to use Async Completion
The intended use-case for this feature is when an external system has the final result of a computation, started by an Activity.
Consider using Asynchronous Activities instead of Signals if the external process is unreliable and might fail to send critical status updates through a Signal.
Consider using Signals as an alternative to Asynchronous Activities to return data back to a Workflow Execution if there is a human in the process loop.
The reason is that a human in the loop means multiple steps in the process.
The first is the Activity Function that stores state in an external system and at least one other step where a human would “complete” the activity.
If the first step fails, you want to detect that quickly and retry instead of waiting for the entire process, which could be significantly longer when humans are involved.
Task Token
A Task Token is a unique identifier for an Activity Task Execution What is an Activity Task Execution? An Activity Task Execution is the execution of an Activity Type.
Asynchronous Activity Completion calls take either of the following as arguments:
- a Task Token, or
- an Activity Id, a Workflow Id, and optionally a Run Id.
What is a Workflow Id?
A Workflow Id is a customizable, application-level identifier for a Workflow Execution that is unique to an Open Workflow Execution within a Namespace.
Local Activity
A Local Activity is an Activity Execution What is an Activity Execution? An Activity Execution is the full chain of Activity Task Executions. What is a Workflow Execution? A Temporal Workflow Execution is a durable, scalable, reliable, and reactive function execution. It is the main unit of execution of a Temporal Application.
Some Activity Executions are very short-living and do not need the queuing semantic, flow control, rate limiting, and routing capabilities. For this case, Temporal supports the Local Activity feature.
The main benefit of Local Activities is that they use less Temporal service resources (e.g. lower state transitions) and have much lower latency overhead (because no need to roundtrip to the Cluster) compared to normal Activity Executions. However, Local Activities are subject to shorter durations and a lack of rate limiting.
Consider using Local Activities for functions that are the following:
- can be implemented in the same binary as the Workflow that calls them.
- do not require global rate limiting.
- do not require routing to a specific Worker or Worker pool.
- no longer than a few seconds, inclusive of retries (shorter than the Workflow Task Timeout, which is 10 seconds by default).
Using a Local Activity without understanding its limitations can cause various production issues. We recommend using regular Activities unless your use case requires very high throughput and large Activity fan outs of very short-lived Activities. More guidance in choosing between Local Activity vs Activity is available in our forums.