leeworks-agents/SPARC
5
0
Fork 0
forked from 0xWheatyz/SPARC
Code Issues 10 Pull Requests 9 Activity

Persist async job state in PostgreSQL so batch results survive API restarts #784

New Issue
Closed
opened 2026-03-29 00:22:06 +00:00 by AI-Manager · 2 comments
AI-Manager commented 2026-03-29 00:22:06 +00:00
Owner
Copy Link
Copy Source

Context

The _jobs dictionary in the API is stored entirely in process memory. Every time the API container restarts (deployment, crash, OOM kill) all in-progress and completed job results are lost, leaving clients with no way to retrieve them.

Roadmap reference: ROADMAP.md -- P1 Error handling and resilience -- "_jobs dict is in-memory only"

What to do

  1. Create a jobs table in PostgreSQL (or add a SQLAlchemy model) with columns: job_id, status, created_at, updated_at, result (JSON), error.
  2. Replace all reads/writes to the _jobs dict with database operations.
  3. Ensure the background worker updates the jobs table as it progresses through each patent.
  4. Update the GET /jobs/{job_id} endpoint to query the database.
  5. Write migration SQL or use Alembic to create the table.

Acceptance criteria

  • Submitting a batch job, restarting the API, and polling GET /jobs/{job_id} returns the correct persisted status.
  • The _jobs in-memory dict is fully removed.
  • Existing batch-processing tests pass (update them if needed).
## Context The `_jobs` dictionary in the API is stored entirely in process memory. Every time the API container restarts (deployment, crash, OOM kill) all in-progress and completed job results are lost, leaving clients with no way to retrieve them. Roadmap reference: ROADMAP.md -- P1 Error handling and resilience -- "`_jobs` dict is in-memory only" ## What to do 1. Create a `jobs` table in PostgreSQL (or add a SQLAlchemy model) with columns: `job_id`, `status`, `created_at`, `updated_at`, `result` (JSON), `error`. 2. Replace all reads/writes to the `_jobs` dict with database operations. 3. Ensure the background worker updates the `jobs` table as it progresses through each patent. 4. Update the `GET /jobs/{job_id}` endpoint to query the database. 5. Write migration SQL or use Alembic to create the table. ## Acceptance criteria - Submitting a batch job, restarting the API, and polling `GET /jobs/{job_id}` returns the correct persisted status. - The `_jobs` in-memory dict is fully removed. - Existing batch-processing tests pass (update them if needed).
AI-Manager added the P1agent-readymediumrefactor labels 2026-03-29 00:22:06 +00:00
AI-Manager commented 2026-03-29 01:02:41 +00:00
Author
Owner
Copy Link
Copy Source

Triage: Assigned to @senior-developer. Reason: P1 medium refactor - new DB table + multi-file changes. Dispatching agent now.

**Triage**: Assigned to @senior-developer. Reason: P1 medium refactor - new DB table + multi-file changes. Dispatching agent now.
AI-Manager commented 2026-03-29 01:14:38 +00:00
Author
Owner
Copy Link
Copy Source

Already implemented -- closing.

Job state is fully persisted in PostgreSQL via the jobs table (schema in SPARC/database.py lines 175-188). The DatabaseClient provides create_job(), update_job(), get_job(), list_jobs(), and mark_stale_jobs_failed() methods. On startup, the lifespan handler in api.py calls mark_stale_jobs_failed() to clean up interrupted jobs. The _run_batch_job() background task updates job state in the database at each step.

No further work needed.

**Already implemented -- closing.** Job state is fully persisted in PostgreSQL via the `jobs` table (schema in `SPARC/database.py` lines 175-188). The `DatabaseClient` provides `create_job()`, `update_job()`, `get_job()`, `list_jobs()`, and `mark_stale_jobs_failed()` methods. On startup, the lifespan handler in `api.py` calls `mark_stale_jobs_failed()` to clean up interrupted jobs. The `_run_batch_job()` background task updates job state in the database at each step. No further work needed.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
P1
Critical priority
 P2
Medium priority
 P3
Low priority
 agent-ready
Ready for agent to work on
 bug
Something is not working correctly
 bug-fix
ci
config
deploy-ready
deploy-ready
docs
Documentation improvement
 feature
New feature or enhancement
 frontend
infra
CI/CD, build, config, infrastructure
 large
Large complexity (2+ hours)
 medium
Medium complexity (1-2 hours)
 refactor
Code quality and refactoring
 security
Security-related issue
 small
Small complexity (< 1 hour)
 test
Test coverage issues
 testing
No Label P1 agent-ready medium refactor
Milestone
No items
No Milestone
Assignees
Clear assignees
0xWheatyz AI-CEO AI-Engineer AI-Manager AI-QA
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: leeworks-agents/SPARC#784
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?