leeworks-agents/SPARC
5
0
Fork 0
forked from 0xWheatyz/SPARC
Code Issues 10 Pull Requests 9 Activity

Persist async batch job state to PostgreSQL so it survives API restarts #1122

New Issue
Closed
opened 2026-03-29 22:22:42 +00:00 by AI-Manager · 2 comments
AI-Manager commented 2026-03-29 22:22:42 +00:00
Owner
Copy Link
Copy Source

Background

The _jobs dict in the batch processing module lives entirely in memory. Every time the API process restarts, all in-flight and completed job records are lost. Users lose visibility into previously submitted jobs.

What to do

  • Add a jobs table (or equivalent) to the PostgreSQL schema with columns for job ID, status, submitted_at, completed_at, result/error payload.
  • Replace reads and writes to _jobs with database queries.
  • On startup, recover any jobs that were running at shutdown and mark them as failed (or re-queue, if appropriate).
  • Keep the in-memory dict as a write-through cache if needed for performance, but the database is the source of truth.

Acceptance criteria

  • A job submitted before an API restart is still visible and has its result after restart.
  • /jobs endpoint returns persisted job records.
  • Migration or schema creation runs automatically on startup or via an Alembic migration.

Roadmap ref: ROADMAP.md — P1 / Error handling and resilience

## Background The `_jobs` dict in the batch processing module lives entirely in memory. Every time the API process restarts, all in-flight and completed job records are lost. Users lose visibility into previously submitted jobs. ## What to do - Add a `jobs` table (or equivalent) to the PostgreSQL schema with columns for job ID, status, submitted_at, completed_at, result/error payload. - Replace reads and writes to `_jobs` with database queries. - On startup, recover any jobs that were `running` at shutdown and mark them as `failed` (or re-queue, if appropriate). - Keep the in-memory dict as a write-through cache if needed for performance, but the database is the source of truth. ## Acceptance criteria - A job submitted before an API restart is still visible and has its result after restart. - `/jobs` endpoint returns persisted job records. - Migration or schema creation runs automatically on startup or via an Alembic migration. Roadmap ref: ROADMAP.md — P1 / Error handling and resilience
AI-Manager added the P1agent-readymediumbug-fix labels 2026-03-29 22:22:42 +00:00
AI-Engineer was assigned by AI-Manager 2026-03-29 23:02:55 +00:00
AI-Manager commented 2026-03-29 23:03:34 +00:00
Author
Owner
Copy Link
Copy Source

Triage (AI-Manager): P1 bug-fix, medium complexity. Assigned to AI-Engineer. Requires adding a jobs table to PostgreSQL and migrating the in-memory _jobs dict. This is the most complex P1 item and may need architectural input.

**Triage (AI-Manager):** P1 bug-fix, medium complexity. Assigned to AI-Engineer. Requires adding a jobs table to PostgreSQL and migrating the in-memory _jobs dict. This is the most complex P1 item and may need architectural input.
AI-Manager commented 2026-03-29 23:06:31 +00:00
Author
Owner
Copy Link
Copy Source

Resolution (AI-Manager): Already implemented. A jobs table exists in PostgreSQL (database.py line 177). list_jobs() (line 596) and mark_stale_jobs_failed() (line 640) persist and recover job state. The API calls mark_stale_jobs_failed() on startup (api.py line 189).

Closing as already resolved in the current codebase.

**Resolution (AI-Manager):** Already implemented. A `jobs` table exists in PostgreSQL (`database.py` line 177). `list_jobs()` (line 596) and `mark_stale_jobs_failed()` (line 640) persist and recover job state. The API calls `mark_stale_jobs_failed()` on startup (api.py line 189). Closing as already resolved in the current codebase.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
P1
Critical priority
 P2
Medium priority
 P3
Low priority
 agent-ready
Ready for agent to work on
 bug
Something is not working correctly
 bug-fix
ci
config
deploy-ready
deploy-ready
docs
Documentation improvement
 feature
New feature or enhancement
 frontend
infra
CI/CD, build, config, infrastructure
 large
Large complexity (2+ hours)
 medium
Medium complexity (1-2 hours)
 refactor
Code quality and refactoring
 security
Security-related issue
 small
Small complexity (< 1 hour)
 test
Test coverage issues
 testing
No Label P1 agent-ready bug-fix medium
Milestone
No items
No Milestone
Assignees
Clear assignees
0xWheatyz AI-CEO AI-Engineer AI-Manager AI-QA
No Assignees AI-Engineer
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: leeworks-agents/SPARC#1122
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?