Scheduling ========== The scheduler is responsible for matching queued jobs to available bots. It runs background threads called **assigners** which periodically query for an unassigned job, select a suitable bot, and write the assignment back to the database. Multiple assigner instances can run concurrently for throughput, and each sleeps for a configurable interval between iterations (shortened when work is being actively assigned). Assigner Types -------------- Both assigner types support **preemption** when instance quotas are configured. If a job's instance is below its minimum quota and no bot is free, the scheduler can evict a lower-priority job from another instance in the same cohort to reclaim capacity. Preemption only triggers after the job has been queued longer than the configured ``preemption_delay``. PriorityAgeJobAssigner ~~~~~~~~~~~~~~~~~~~~~~ A single-threaded assigner that picks the next unassigned job ordered by priority (lower value = higher priority, following REAPI convention) and then by queue age. A ``priority_percentage`` parameter controls how often priority ordering is used versus pure age ordering, allowing occasional starvation relief for older, lower-priority jobs. When an assignment attempt fails (no healthy bot matches), the job is given a ``schedule_after`` backoff so it is not reconsidered immediately. CohortAssigner ~~~~~~~~~~~~~~ A cohort-aware assigner that spawns one thread per cohort. Each thread independently assigns jobs whose property label belongs to its cohort, which partitions the assignment workload by bot group. The key difference from ``PriorityAgeJobAssigner`` is instance-level prioritization: the ``CohortAssigner`` first looks for jobs in instances that are below their minimum quota, and only then considers instances that are below their maximum quota. This ensures that under-served instances receive capacity before opportunistic work is scheduled. Proactive Fetching ------------------ When a bot synchronizes its leases after completing work, the scheduler can fetch new jobs for it. The ``proactive_fetch_to_capacity`` flag controls how aggressively this happens. When **disabled** (the default), the scheduler only fetches as many new jobs as the bot just completed. It restricts candidates to jobs whose ``schedule_after`` is at or after the current time (``schedule_after >= now``), which includes newly created jobs and jobs still within their backoff window, but excludes jobs whose backoff has already elapsed. Those elapsed-backoff jobs are left for the assigner threads to pick up on their next iteration. When **enabled**, the scheduler fills all of the bot's spare capacity in a single synchronization, and the ``schedule_after`` filter is removed entirely so that all unassigned queued jobs are eligible regardless of their backoff state. Bot Assignment Strategy ----------------------- The assigner delegates bot selection to a pluggable strategy. The only strategy currently implemented is ``AssignByCapacity``, which selects a healthy bot (status OK, capacity > 0) whose platform capabilities match the job's requirements. By default, the first matching bot is locked and assigned. When **sampling** is enabled, the assigner uses a two-phase approach that can make better placement decisions under load. Bot Sampling ~~~~~~~~~~~~ When a ``SamplingConfig`` is attached to the strategy, the assigner reads a random sample of candidate bots *without* locking them, scores each candidate, and then attempts to lock the highest-scoring bot. If the chosen bot is no longer available (taken by another assigner), it falls back to locking any healthy bot. The ``max_attempts`` parameter controls how many sampling rounds are tried before falling back. Each candidate is scored with a weighted linear combination: .. code-block:: text score = priority * w_priority + capacity * w_capacity + locality_hit * w_locality where ``locality_hit`` is 1 if the bot has a locality hint matching the job and 0 otherwise. The weights and their defaults are: .. list-table:: :header-rows: 1 :widths: 20 15 65 * - Weight - Default - Effect * - ``priority`` - -10.0 - Lower bot priority value produces a higher score (per REAPI convention). * - ``locality`` - 5.0 - Bonus applied when the bot matches the job's locality hint. * - ``capacity`` - 1.0 - Bots with more spare capacity score higher, spreading load. With the default weights, priority dominates: a bot with a priority advantage of 1 outweighs a locality match. Adjusting the weights allows operators to tune placement toward locality-affinity or load-balancing as needed.