Configure Job Queue Autoscaling

Last Updated: October 19, 2026

When to use this runbook

Use for discrete JSON jobs handled by an HTTP application through the SaladCloud Job Queue Worker, when worker capacity should follow managed queue depth.

When not to use it

Do not use Job Queue autoscaling for direct gateway traffic, stateful jobs that cannot be retried, work that depends on instance-local durable state, or extremely long jobs that are incompatible with the documented interruption/retry model. Do not use it to attach a queue to an already-created group: queue_connection is absent from the current Container Group patch schema.

Required inputs

SALAD_API_KEY, SALAD_ORGANIZATION, SALAD_PROJECT, SALAD_QUEUE, and SALAD_CONTAINER_GROUP.
A unique queue name and group name, both resolved against live list operations.
Worker image containing the SaladCloud Job Queue Worker and application; application HTTP path and port.
Image/resources/priority, readiness behavior, min_replicas, max_replicas, and desired_queue_length.
Optional polling_period, max_upscale_per_minute, and max_downscale_per_minute.
Explicit user intent for test-job submission, job cancellation, scale-down, stop, group deletion, or queue deletion.

Authoritative sources

Purpose	Operation and canonical reference
Discover queues	Operation: `list_queues` — List Queues
Read queue	Operation: `get_queue` — Get Queue
Create queue	Operation: `create_queue` — Create Queue
Read/create group	Operations: `get_container_group`, `create_container_group` — Create Container Group
Update autoscaler	Operation: `update_container_group` — Update Container Group
Submit/read jobs	Operations: `create_queue_job`, `get_queue_job`, `list_queue_jobs` — Create Job
Cleanup	Operations: `delete_queue_job`, `delete_queue`, `delete_container_group` — Queue API

Use Job Queues, Job Queue Worker, Job Queue Autoscaling, and Autoscaling Settings for canonical behavior. Use Cold Starts and Iterating Quickly for test-loop tactics.

Dynamic values to retrieve

Exact queues and groups already present in the project.
Queue current_queue_length and current job states/events. To find the groups attached to a queue, read each group’s queue_connection.
Group queue_connection, queue_autoscaler, desired replicas, status/version, instances, and readiness.
Live quota headroom using the planned max_replicas, plus current hardware availability.
Worker and application logs for the relevant UTC window.

Preflight checks

Complete Discover Scope and Preflight, reserving quota headroom for max_replicas as required by canonical quota guidance.
Call list_queues and list_container_groups; compare exact names to prevent duplicates.
Confirm the application and worker are in the image, the application accepts/returns valid JSON, and the path/port match queue_connection.
Confirm jobs are idempotent and durable state/results are externalized before acknowledging success.
Validate bounds: desired queue length 1–100; minimum replicas 0–100; maximum replicas 1–500; polling period 15–1800 seconds when supplied; optional up/down rates 1–100. Require min_replicas <= max_replicas.
For scale from zero, accept cold-start latency and configure readiness so the worker does not receive jobs before the application is ready.

Procedure

Create or discover the queue

Operation: list_queues. If the exact queue name exists, call get_queue and reuse it only when that is the user’s intent.
If absent, Operation: create_queue. The minimal request is:

curl --fail-with-body --silent --show-error \
  --request POST \
  --url "https://api.salad.com/api/public/organizations/${SALAD_ORGANIZATION}/projects/${SALAD_PROJECT}/queues" \
  --header "Salad-Api-Key: ${SALAD_API_KEY}" \
  --header 'Content-Type: application/json' \
  --header 'Accept: application/json' \
  --data '{
    "name": "agent-example-queue",
    "display_name": "Agent Example Queue",
    "description": "Queue for an approved worker workload"
  }'

Replace the example with the approved SALAD_QUEUE; never treat the example as a real resource. Verify with get_queue.

Associate a new worker group and enable autoscaling

The actual association is queue_connection in Operation: create_container_group. The queue and group must be in the same supplied project. Create the group with queue_connection and the readiness probe, then set queue_autoscaler with Operation: update_container_group, then read the group back to confirm the values. This schema-derived scale-from-zero template uses a replica count of 0:

curl --fail-with-body --silent --show-error \
  --request POST \
  --url "https://api.salad.com/api/public/organizations/${SALAD_ORGANIZATION}/projects/${SALAD_PROJECT}/containers" \
  --header "Salad-Api-Key: ${SALAD_API_KEY}" \
  --header 'Content-Type: application/json' \
  --header 'Accept: application/json' \
  --data '{
    "name": "agent-example-workers",
    "container": {
      "image": "<registry>/<repository>:<immutable-tag-or-digest>",
      "resources": {
        "cpu": 4,
        "memory": 8192
      },
      "environment_variables": {
        "SALAD_LOG_LEVEL": "info"
      },
      "priority": "batch"
    },
    "autostart_policy": true,
    "restart_policy": "always",
    "replicas": 0,
    "queue_connection": {
      "path": "/process",
      "port": 8080,
      "queue_name": "agent-example-queue"
    },
    "readiness_probe": {
      "http": {
        "headers": [],
        "path": "/ready",
        "port": 8080,
        "scheme": "http"
      },
      "failure_threshold": 3,
      "initial_delay_seconds": 10,
      "period_seconds": 5,
      "success_threshold": 1,
      "timeout_seconds": 2
    }
  }'

Then enable autoscaling on the new group with a merge patch. This example scales from zero to a maximum of 2 replicas. The three optional fields (polling_period, max_upscale_per_minute, max_downscale_per_minute) may be left out, and they stay null:

curl --fail-with-body --silent --show-error \
  --request PATCH \
  --url "https://api.salad.com/api/public/organizations/${SALAD_ORGANIZATION}/projects/${SALAD_PROJECT}/containers/agent-example-workers" \
  --header "Salad-Api-Key: ${SALAD_API_KEY}" \
  --header 'Content-Type: application/merge-patch+json' \
  --header 'Accept: application/json' \
  --data '{
    "queue_autoscaler": {
      "desired_queue_length": 1,
      "min_replicas": 0,
      "max_replicas": 2,
      "polling_period": 30,
      "max_upscale_per_minute": 2,
      "max_downscale_per_minute": 1
    }
  }'

Replace all examples with approved values and a live hardware plan. For an existing group that already has the correct queue_connection, read it, then patch only queue_autoscaler with Operation: update_container_group and verify. When the worker image is not yet proven, create the group with an approved fixed replica count instead of 0, confirm a test job reaches succeeded (test-job submission needs explicit intent), and only then apply queue_autoscaler. The autoscaler sets the replica count from queue length, so it overrides the fixed count. See Cold Starts and Iterating Quickly.

Verify scaling behavior

Read the group and queue; verify both sides of the association.
Submit a test job only with explicit user intent. Operation: create_queue_job returns the job and generated ID.
Poll get_queue, get_container_group, list_container_group_instances, and get_queue_job within a bounded budget.
Verify the queue grows, desired/observed worker capacity changes within the configured boundaries, a current-version instance becomes ready, and the job reaches a terminal status.
After the queue drains, verify the group returns no lower than min_replicas; with minimum 0, expect cold starts on subsequent jobs.

Scaling from zero adds a cold start of a few minutes before the first job runs, while a new instance is allocated, pulls the image, and passes readiness. Plan the polling budget for it.

Worker, retry, and webhook behavior

The canonical queue documentation states that the worker forwards JSON to the configured HTTP application; 200 indicates job success and 500 indicates failure. A failed job can be retried up to three times (four total attempts), and an instance interruption counts as a failed attempt. Design handlers to be idempotent by job ID or application idempotency key, and commit output to external storage before returning success. The current local queue docs/OpenAPI do not define an agent-configurable acknowledgment timeout. Do not invent one. Observe job events and worker logs. When a job uses a webhook, validate the documented webhook-signature, webhook-id, and webhook-timestamp headers as described in Webhook Signatures. Never expose the webhook secret.

Decision rules

Situation	Action
Exact queue/group name already exists	Read and reconcile; do not create a duplicate.
Existing group has correct `queue_connection`	Patch autoscaler only after a read.
Existing group lacks or has wrong `queue_connection`	Stop; patch schema cannot attach/change it. Ask for an approved replacement strategy.
Minimum replicas is 0	Accept cold-start delay and require readiness before dispatch.
Maximum replicas exceeds live quota headroom	Stop or request an approved lower maximum/quota increase.
Queue has jobs but no scaling	Check association, autoscaler, quota/availability, group status, readiness, and worker logs in that order.
Job retries repeat the same application error	Fix the worker/application; do not reallocate nodes blindly.
Cleanup would cancel jobs, reduce capacity, or delete resources	Require explicit user intent.

Expected states and responses

Queue/group creation returns 201; update returns 200; job creation returns 201; delete/cancel operations return 202. Queue job statuses are pending, running, succeeded, cancelled, and failed. A queue response can include current_queue_length; treat an absent optional current length as unknown. Confirm an association from the group’s queue_connection. Canonical autoscaling guidance uses ceil(queue length / desired queue length) bounded by minimum/maximum and rate settings. Verify actual live behavior rather than calculating success from the formula alone.

Retry behavior

Reconcile any uncertain queue or group create by exact name before retrying. Reconcile a job create only by its returned job ID; if the outcome is unknown and no ID was returned, stop rather than submitting the job again. Retry reads within a bounded budget; honor Retry-After when present. Do not resubmit a failed job automatically: the handler may already have produced side effects. Use an application idempotency record and explicit retry intent.

Verification

Mutation	Read before	Read after	Success	Failure or unresolved
Create queue	`list_queues`	`get_queue`	Exact queue exists with intended metadata	Duplicate ambiguity, mismatch, or unresolved `404`
Create associated group	Queue/group lists	`get_container_group`	Group has the intended `queue_connection`; `queue_autoscaler` is applied by the following update	Mismatch, group failure, or `queue_connection` absent past budget
Update autoscaler	`get_container_group`	Same plus queue/instances	`queue_autoscaler` values match and scaling stays within bounds	Mismatch, quota/availability block, or no response to queued work past budget
Submit test job	`get_queue`	`get_queue_job` plus scaling reads	Job reaches intended terminal status and scaling predicate is observed	Failed/cancelled unexpectedly or pending/running past budget

Rollback or recovery

Restore only the previous autoscaler object with a fresh merge patch when it is known and authorized.
Do not detach a queue through an undocumented field; the current patch schema has no queue_connection.
Preserve a failed job and logs for diagnosis. Resubmit only after the cause is corrected and idempotency is proven.
Safe cleanup begins with get_queue, list_queue_jobs, get_container_group, and instance reads. The queue’s container_groups list can be empty even while a connected group is running, so check each group’s queue_connection to find groups attached to the queue. If any group is attached or any job is active, stop. Cancel jobs, reduce/stop/delete capacity, and delete the queue only with explicit intent, then verify each deletion in the same trusted scope.
Deleting a queue permanently removes its connection from every attached group, and the group keeps running with no queue. Recreating a queue with the same name does not reconnect it.
Stopping or deleting a group does not stop its worker at once, and the old container can take new jobs for a short time afterwards. Those jobs read running until the queue notices the worker is gone, which takes several minutes, and longer when no other worker is polling the queue. Stop job submission or let the queue drain first.
Cancelling a running job does not stop it. The instance runs the job to the end and the output is discarded.

Stop and escalation conditions

Stop for missing worker/image/path/port, non-idempotent state handling, duplicate ambiguity, unsupported association changes, insufficient quota/availability, missing destructive intent, repeated job failure, no scale response past the polling budget, or missing logs/events. Escalate with queue/group/job IDs, UTC times, association/autoscaler settings, queue length, instance readiness, worker logs, and dynamic quota/availability evidence.

Evidence to return to the user

Return validated scope and exact names; operation IDs/status classes; queue ID/current length; group version/status; redacted association and autoscaler settings; quota headroom; observed min/max/ready capacity; test job ID/status/events; polling window; retries; idempotency/external-state confirmation; cleanup performed or skipped; and unresolved or escalation status. Never return API, registry, or webhook secrets.

Overview

Container Engine

AI Gateway

Transcription APIs

Reference

Configure Job Queue Autoscaling

When to use this runbook

When not to use it

Required inputs

Authoritative sources

Dynamic values to retrieve

Preflight checks

Procedure

Create or discover the queue

Associate a new worker group and enable autoscaling

Verify scaling behavior

Worker, retry, and webhook behavior

Decision rules

Expected states and responses

Retry behavior

Verification

Rollback or recovery

Stop and escalation conditions

Evidence to return to the user

​When to use this runbook

​When not to use it

​Required inputs

​Authoritative sources

​Dynamic values to retrieve

​Preflight checks

​Procedure

​Create or discover the queue

​Associate a new worker group and enable autoscaling

​Verify scaling behavior

​Worker, retry, and webhook behavior

​Decision rules

​Expected states and responses

​Retry behavior

​Verification

​Rollback or recovery

​Stop and escalation conditions

​Evidence to return to the user

When to use this runbook

When not to use it

Required inputs

Authoritative sources

Dynamic values to retrieve

Preflight checks

Procedure

Create or discover the queue

Associate a new worker group and enable autoscaling

Verify scaling behavior

Worker, retry, and webhook behavior

Decision rules

Expected states and responses

Retry behavior

Verification

Rollback or recovery

Stop and escalation conditions

Evidence to return to the user