Compare commits

..

1 Commits

Author SHA1 Message Date
agent-company fc942b2aa4 Add tests for tracked company admin endpoints and scheduler integration
20 test cases covering:
- GET/POST/DELETE /admin/tracked endpoints with admin auth enforcement
- GET /admin/alerts with limit parameter and auth
- scheduler.run_scheduled_analysis() for multi-company analysis, alert
  triggering on significant patent count changes, graceful failure handling

Closes leeworks-agents/SPARC#1656

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-20 19:14:29 +00:00
2 changed files with 464 additions and 118 deletions
+76 -118
View File
@@ -7,131 +7,86 @@ Semiconductor Patent & Analytics Report Core -- development priorities.
SPARC is a patent analysis platform with a working end-to-end pipeline:
Python/FastAPI backend, React/TypeScript frontend, PostgreSQL for persistence
and caching, Docker Compose for local development, and Gitea Actions CI/CD for
image builds and testing. Core features include patent retrieval via SerpAPI,
PDF parsing, LLM analysis via OpenRouter (multi-model: Claude, GPT-4o, Gemini,
Llama), batch processing, JWT authentication, analytics dashboard with patent
trend charts, scheduled recurring analysis with alerting, webhook notifications
(Slack/Discord), CSV and PDF export, S3/MinIO storage, side-by-side company
comparison, and dark mode.
---
## Completed
Items that have been implemented and merged into main.
### Security hardening
- ~~Rotate default JWT secret.~~ Startup check refuses to start with the
default secret in non-development environments.
- ~~CORS allow-origins are hardcoded.~~ Allowed origins are now configurable
via environment variable.
- ~~Database credentials in docker-compose.yml.~~ Compose references `.env`
for sensitive values.
### Error handling and resilience
- ~~`get_db_client()` creates a new `DatabaseClient` on every call.~~ Refactored
to a shared pooled singleton initialized at startup.
- ~~No rate limiting on auth endpoints.~~ Rate limiting middleware added to
`/auth/login` and `/auth/register`.
### Test coverage
- ~~API tests bypass authentication.~~ JWT auth integration tests added (33
cases covering registration, login, protected routes, token refresh, and
admin-only endpoints).
- ~~No test stage in CI.~~ Gitea Actions workflow now runs `pytest` and gates
the build.
- ~~No linting or type checking in CI.~~ `ruff` (Python) and `tsc --noEmit`
(TypeScript) added to CI pipeline.
### Backend
- ~~Add structured logging.~~ Python `logging` module used throughout.
- ~~Make LLM model configurable.~~ `MODEL` environment variable accepted;
multi-model support with per-analysis selection (GPT-4o, Gemini, Claude,
Llama).
- ~~SERP cache TTL hardcoded.~~ `SERP_CACHE_TTL_HOURS` exposed as env var.
- ~~Patent PDF storage.~~ S3/MinIO object storage backend added alongside
local filesystem. Volume mount requirement documented.
- ~~`analyze_single_patent` assumes local file.~~ Auto-download from cached
metadata link integrated.
- ~~`Patent.patent_id` typed as `int`.~~ Fixed to `str`.
### Frontend
- ~~No loading/error states.~~ Skeleton loaders and error states added to
Batch and Analytics pages.
- ~~No dark mode.~~ Full dark mode support with theme-aware chart colors.
- ~~Missing lockfile.~~ `package-lock.json` committed.
### Features (formerly P3)
- ~~Export analysis reports.~~ CSV and PDF export endpoints implemented.
- ~~Comparison view.~~ Side-by-side company patent portfolio comparison added.
- ~~Scheduled/recurring analysis.~~ APScheduler-based periodic re-analysis
with configurable interval and change-threshold alerting.
- ~~Webhook/notification support.~~ Slack, Discord, and generic HTTP POST
webhooks with retry logic.
- ~~Multi-model support.~~ Model picker in Analysis and Batch pages; backend
allow-list validation.
- ~~Patent trend charts.~~ Filing frequency and category distribution
visualizations added to Analytics page.
- ~~OpenAPI client generation.~~ TypeScript API client auto-generated from
FastAPI spec with CI freshness check.
image builds. Core features (patent retrieval via SerpAPI, PDF parsing, LLM
analysis via OpenRouter/Claude, batch processing, JWT authentication, analytics
dashboard) are all implemented and functional.
---
## P1 -- High Priority
These items address correctness, reliability, and coverage gaps that should be
These items address correctness, security, and reliability gaps that should be
resolved before broader production use.
### Resilience
### Security hardening
- **`_jobs` dict is in-memory only.** Job state is lost on API restart.
Persist job status in PostgreSQL or Redis so async batch results survive
restarts.
- **Rotate default JWT secret.** `auth.py` ships a fallback
`sparc-secret-key-change-in-production` that will be used if `JWT_SECRET` is
unset. Add a startup check that refuses to start with the default secret in
non-development environments.
- **CORS allow-origins are hardcoded.** `api.py` only permits
`localhost:3000` and `localhost:5173`. Make the allowed origins configurable
via environment variable so the dashboard works when deployed behind a real
domain.
- **Database credentials in docker-compose.yml.** The compose file embeds
`postgres:postgres` in plain text. Reference a `.env` file or Docker secrets
instead.
### Test coverage gaps
### Error handling and resilience
- **Export endpoint tests.** The CSV and PDF export endpoints (`/export/`)
lack test coverage. Add tests covering auth, success, 404, and edge cases.
*(Issue #1655)*
- **Tracked company admin endpoint tests.** The `/admin/tracked` CRUD
endpoints and scheduler integration lack test coverage. *(Issue #1656)*
- **`get_db_client()` in `auth.py` creates a new `DatabaseClient` on every
call.** This bypasses the connection pool and can exhaust database
connections under load. Refactor to share a single pooled client.
- **`_jobs` dict is in-memory only.** Job state is lost on API restart. Persist
job status in PostgreSQL or Redis so async batch results survive restarts.
- **No rate limiting on auth endpoints.** `/auth/login` and `/auth/register`
are unprotected against brute-force or abuse. Add rate limiting middleware.
### Test coverage for auth and admin
- The existing API tests (`tests/test_api.py`) bypass authentication entirely.
Add tests that exercise the JWT flow: registration, login, protected-route
access, token refresh, and admin-only endpoints.
---
## P2 -- Medium Priority
Improvements to reliability, test coverage, and code quality.
Improvements to usability, performance, and developer experience.
### Test coverage
### Backend
- **Webhook integration tests.** The retry logic, Slack/Discord payload
format, and multi-URL dispatch in `webhooks.py` need test coverage.
*(Issue #1657)*
- **S3/MinIO storage backend tests.** `storage.py` has local filesystem tests
but no unit tests for the S3 backend (read, write, exists, delete,
error handling). *(Issue #1660)*
- **`analyze_single_patent` auto-download path tests.** The auto-download
fallback (cache lookup, PDF download, FileNotFoundError) in
`analyzer.py` lacks test coverage. *(Issue #1661)*
- **Add structured logging.** Replace `print()` calls throughout `analyzer.py`,
`serp_api.py`, and `llm.py` with Python `logging` so log levels and
formatting are consistent.
- **Make LLM model configurable.** `llm.py` hardcodes
`anthropic/claude-3.5-sonnet`. Accept a `MODEL` environment variable to allow
switching models without code changes.
- **SERP cache TTL is hardcoded to 24 hours.** Expose `SERP_CACHE_TTL_HOURS`
as an environment variable in `config.py`.
- **Patent PDF storage.** PDFs are saved to a local `patents/` directory. For
containerized deployments, consider object storage (S3/MinIO) or at minimum
document the volume mount requirement more prominently.
- **`analyze_single_patent` assumes local file path.** The method constructs
`patents/{patent_id}.pdf` and reads from disk, but does not download the PDF
first. Either integrate the download step or document the prerequisite.
- **`Patent.patent_id` typed as `int` in `types.py` but used as `str`
everywhere.** Fix the type annotation to `str`.
### Code quality
### Frontend
- **Scheduler creates its own DatabaseClient.** `scheduler.py` bypasses the
application-level pooled client, creating a new connection on every tick.
Refactor to use `get_db_client()`. *(Issue #1658)*
- **No loading/error states on several pages.** The Batch and Analytics pages
would benefit from skeleton loaders and user-friendly error messages.
- **No dark mode.** Tailwind is configured but no dark variant is applied.
- **Missing `package-lock.json` or `pnpm-lock.yaml`.** The frontend has no
lockfile committed, leading to non-reproducible builds.
### API improvements
### CI/CD
- **API pagination.** The `/analyze/batch` and `/jobs` endpoints could benefit
from cursor-based pagination for large result sets.
- **Request validation improvements.** Add stricter input validation for
company names (disallow special characters, enforce length limits).
- **No test stage in the Gitea Actions workflow.** `build.yaml` builds and
pushes images but never runs `pytest`. Add a test job that gates the build.
- **No linting or type checking.** Add `ruff` (Python) and `tsc --noEmit`
(TypeScript) to CI.
---
@@ -139,20 +94,23 @@ Improvements to reliability, test coverage, and code quality.
Lower-urgency enhancements and future features.
- **Historical analysis diffing.** Show what changed between two analysis runs
for the same company, highlighting new patents and score shifts.
- **Patent classification tagging.** Automatically tag patents by technology
domain (AI, semiconductors, materials science) using LLM classification.
- **User-level API keys.** Allow users to generate personal API keys for
programmatic access without JWT token refresh.
- **Batch export.** Export analysis results for multiple companies at once as
a ZIP archive.
- **Rate limiting dashboard.** Surface rate limit status and usage statistics
in the admin panel.
- **Async webhook delivery.** Move webhook delivery to a background task queue
(e.g., Celery, arq) to avoid blocking the scheduler.
- **Multi-tenant support.** Scope analysis results and tracked companies per
user or organization.
- **Export analysis reports.** Allow users to download analysis results as PDF
or CSV from the dashboard.
- **Comparison view.** Side-by-side comparison of two companies' patent
portfolios.
- **Scheduled/recurring analysis.** Periodically re-analyze tracked companies
and alert on significant changes.
- **Webhook/notification support.** Send alerts (Slack, Discord, email) when
batch jobs complete or when a company's innovation score changes
significantly.
- **Multi-model support.** Let users choose between LLM providers per analysis
(e.g., GPT-4o, Gemini, Claude) and compare outputs.
- **Patent trend charts.** Visualize patent filing frequency and technology
category distribution over time in the Analytics page.
- **API pagination.** The `/analyze/batch` and `/jobs` endpoints could benefit
from cursor-based pagination for large result sets.
- **OpenAPI client generation.** Auto-generate the TypeScript API client from
the FastAPI OpenAPI spec to keep frontend types in sync.
---
+388
View File
@@ -0,0 +1,388 @@
"""Tests for tracked company admin endpoints and scheduler integration.
Covers issue #1656:
- GET /admin/tracked (list tracked companies)
- POST /admin/tracked (add a tracked company)
- DELETE /admin/tracked/{company_name} (remove a tracked company)
- GET /admin/alerts (list alerts)
- scheduler.run_scheduled_analysis() integration
All tests mock the database layer and use JWT auth fixtures.
"""
from datetime import datetime, timezone
from unittest.mock import MagicMock, patch, call
import pytest
from fastapi.testclient import TestClient
from SPARC.api import app
from SPARC.auth import create_access_token
@pytest.fixture
def client():
"""Create test client."""
return TestClient(app)
@pytest.fixture(autouse=True)
def mock_db():
"""Mock the database client used by admin and auth endpoints."""
db = MagicMock()
# Default admin user for auth
db.get_user_by_id.return_value = {
"id": 1,
"email": "admin@test.com",
"role": "admin",
"created_at": datetime(2025, 1, 1, tzinfo=timezone.utc),
}
with patch("SPARC.api.get_db_client", return_value=db), \
patch("SPARC.auth.get_db_client", return_value=db):
yield db
def _admin_header():
"""Create an Authorization header with a valid admin access token."""
token = create_access_token(1, "admin@test.com", "admin")
return {"Authorization": f"Bearer {token}"}
def _user_header():
"""Create an Authorization header with a regular user access token."""
token = create_access_token(2, "user@test.com", "user")
return {"Authorization": f"Bearer {token}"}
# ---------- GET /admin/tracked ----------
class TestListTrackedCompanies:
"""GET /admin/tracked"""
def test_list_tracked_returns_companies(self, client, mock_db):
"""Admin can list tracked companies."""
mock_db.list_tracked_companies.return_value = [
{"company_name": "NVIDIA", "last_patent_count": 120, "last_analyzed": "2025-06-15"},
{"company_name": "AMD", "last_patent_count": 80, "last_analyzed": "2025-06-14"},
]
response = client.get("/admin/tracked", headers=_admin_header())
assert response.status_code == 200
data = response.json()
assert len(data) == 2
assert data[0]["company_name"] == "NVIDIA"
def test_list_tracked_empty(self, client, mock_db):
"""Returns empty list when no companies are tracked."""
mock_db.list_tracked_companies.return_value = []
response = client.get("/admin/tracked", headers=_admin_header())
assert response.status_code == 200
assert response.json() == []
def test_list_tracked_requires_admin(self, client, mock_db):
"""Regular user cannot access tracked companies list."""
mock_db.get_user_by_id.return_value = {
"id": 2,
"email": "user@test.com",
"role": "user",
"created_at": datetime(2025, 1, 1, tzinfo=timezone.utc),
}
response = client.get("/admin/tracked", headers=_user_header())
assert response.status_code == 403
def test_list_tracked_unauthenticated(self, client):
"""Unauthenticated request returns 401."""
response = client.get("/admin/tracked")
assert response.status_code == 401
# ---------- POST /admin/tracked ----------
class TestAddTrackedCompany:
"""POST /admin/tracked"""
def test_add_tracked_company_success(self, client, mock_db):
"""Admin can add a company to tracking."""
mock_db.add_tracked_company.return_value = {
"company_name": "Intel",
"last_patent_count": 0,
"last_analyzed": None,
}
response = client.post(
"/admin/tracked",
json={"company_name": "Intel"},
headers=_admin_header(),
)
assert response.status_code == 200
data = response.json()
assert data["company_name"] == "Intel"
mock_db.add_tracked_company.assert_called_once_with("Intel")
def test_add_duplicate_returns_409(self, client, mock_db):
"""Adding an already-tracked company returns 409."""
mock_db.add_tracked_company.return_value = None
response = client.post(
"/admin/tracked",
json={"company_name": "NVIDIA"},
headers=_admin_header(),
)
assert response.status_code == 409
assert "already tracked" in response.json()["detail"].lower()
def test_add_tracked_requires_admin(self, client, mock_db):
"""Regular user cannot add tracked companies."""
mock_db.get_user_by_id.return_value = {
"id": 2,
"email": "user@test.com",
"role": "user",
"created_at": datetime(2025, 1, 1, tzinfo=timezone.utc),
}
response = client.post(
"/admin/tracked",
json={"company_name": "Intel"},
headers=_user_header(),
)
assert response.status_code == 403
def test_add_tracked_empty_name_rejected(self, client):
"""Empty company name is rejected by validation."""
response = client.post(
"/admin/tracked",
json={"company_name": ""},
headers=_admin_header(),
)
assert response.status_code == 422 # Pydantic validation error
# ---------- DELETE /admin/tracked/{company_name} ----------
class TestRemoveTrackedCompany:
"""DELETE /admin/tracked/{company_name}"""
def test_remove_tracked_company_success(self, client, mock_db):
"""Admin can remove a tracked company."""
mock_db.remove_tracked_company.return_value = True
response = client.delete(
"/admin/tracked/NVIDIA",
headers=_admin_header(),
)
assert response.status_code == 200
assert "Stopped tracking" in response.json()["message"]
mock_db.remove_tracked_company.assert_called_once_with("NVIDIA")
def test_remove_nonexistent_returns_404(self, client, mock_db):
"""Removing a non-tracked company returns 404."""
mock_db.remove_tracked_company.return_value = False
response = client.delete(
"/admin/tracked/UnknownCorp",
headers=_admin_header(),
)
assert response.status_code == 404
assert "not found" in response.json()["detail"].lower()
def test_remove_tracked_requires_admin(self, client, mock_db):
"""Regular user cannot remove tracked companies."""
mock_db.get_user_by_id.return_value = {
"id": 2,
"email": "user@test.com",
"role": "user",
"created_at": datetime(2025, 1, 1, tzinfo=timezone.utc),
}
response = client.delete(
"/admin/tracked/NVIDIA",
headers=_user_header(),
)
assert response.status_code == 403
# ---------- GET /admin/alerts ----------
class TestListAlerts:
"""GET /admin/alerts"""
def test_list_alerts_returns_data(self, client, mock_db):
"""Admin can list alerts."""
mock_db.list_alerts.return_value = [
{
"id": 1,
"company_name": "NVIDIA",
"alert_type": "patent_count_change",
"message": "Patent count increased by 25%",
"created_at": "2025-06-15T10:00:00Z",
},
]
response = client.get("/admin/alerts", headers=_admin_header())
assert response.status_code == 200
data = response.json()
assert len(data) == 1
assert data[0]["alert_type"] == "patent_count_change"
def test_list_alerts_with_limit(self, client, mock_db):
"""Custom limit parameter is passed to the database."""
mock_db.list_alerts.return_value = []
response = client.get("/admin/alerts?limit=10", headers=_admin_header())
assert response.status_code == 200
mock_db.list_alerts.assert_called_once_with(limit=10)
def test_list_alerts_requires_admin(self, client, mock_db):
"""Regular user cannot access alerts."""
mock_db.get_user_by_id.return_value = {
"id": 2,
"email": "user@test.com",
"role": "user",
"created_at": datetime(2025, 1, 1, tzinfo=timezone.utc),
}
response = client.get("/admin/alerts", headers=_user_header())
assert response.status_code == 403
# ---------- Scheduler integration ----------
class TestSchedulerIntegration:
"""Tests for scheduler.run_scheduled_analysis()."""
def test_no_tracked_companies_skips_analysis(self):
"""Scheduler does nothing when no companies are tracked."""
mock_db = MagicMock()
mock_db.list_tracked_companies.return_value = []
with patch("SPARC.scheduler.DatabaseClient", return_value=mock_db), \
patch("SPARC.scheduler.CompanyAnalyzer") as mock_analyzer_cls:
from SPARC.scheduler import run_scheduled_analysis
run_scheduled_analysis()
mock_analyzer_cls.assert_not_called()
def test_scheduler_analyzes_each_tracked_company(self):
"""Scheduler runs analysis for every tracked company."""
mock_db = MagicMock()
mock_db.list_tracked_companies.return_value = [
{"company_name": "NVIDIA", "last_patent_count": 100},
{"company_name": "AMD", "last_patent_count": 50},
]
mock_result_nvidia = MagicMock(success=True, patent_count=110)
mock_result_amd = MagicMock(success=True, patent_count=55)
mock_analyzer = MagicMock()
mock_analyzer._analyze_company_safe.side_effect = [mock_result_nvidia, mock_result_amd]
with patch("SPARC.scheduler.DatabaseClient", return_value=mock_db), \
patch("SPARC.scheduler.CompanyAnalyzer", return_value=mock_analyzer):
from SPARC.scheduler import run_scheduled_analysis
run_scheduled_analysis()
assert mock_analyzer._analyze_company_safe.call_count == 2
mock_db.update_tracked_company.assert_any_call("NVIDIA", 110)
mock_db.update_tracked_company.assert_any_call("AMD", 55)
def test_scheduler_triggers_alert_on_significant_change(self):
"""Scheduler stores an alert when patent count changes significantly."""
mock_db = MagicMock()
mock_db.list_tracked_companies.return_value = [
{"company_name": "Tesla", "last_patent_count": 100},
]
mock_result = MagicMock(success=True, patent_count=130) # 30% increase
mock_analyzer = MagicMock()
mock_analyzer._analyze_company_safe.return_value = mock_result
with patch("SPARC.scheduler.DatabaseClient", return_value=mock_db), \
patch("SPARC.scheduler.CompanyAnalyzer", return_value=mock_analyzer):
from SPARC.scheduler import run_scheduled_analysis
run_scheduled_analysis()
mock_db.store_alert.assert_called_once()
alert_kwargs = mock_db.store_alert.call_args
assert alert_kwargs[1]["company_name"] == "Tesla"
assert alert_kwargs[1]["alert_type"] == "patent_count_change"
assert alert_kwargs[1]["old_value"] == 100
assert alert_kwargs[1]["new_value"] == 130
def test_scheduler_no_alert_for_small_change(self):
"""Scheduler does not alert when change is below threshold."""
mock_db = MagicMock()
mock_db.list_tracked_companies.return_value = [
{"company_name": "Intel", "last_patent_count": 100},
]
mock_result = MagicMock(success=True, patent_count=105) # 5% increase
mock_analyzer = MagicMock()
mock_analyzer._analyze_company_safe.return_value = mock_result
with patch("SPARC.scheduler.DatabaseClient", return_value=mock_db), \
patch("SPARC.scheduler.CompanyAnalyzer", return_value=mock_analyzer):
from SPARC.scheduler import run_scheduled_analysis
run_scheduled_analysis()
mock_db.store_alert.assert_not_called()
def test_scheduler_handles_analysis_failure(self):
"""Scheduler continues when one company fails analysis."""
mock_db = MagicMock()
mock_db.list_tracked_companies.return_value = [
{"company_name": "FailCo", "last_patent_count": 50},
{"company_name": "SuccessCo", "last_patent_count": 30},
]
mock_fail_result = MagicMock(success=False, error="API timeout")
mock_ok_result = MagicMock(success=True, patent_count=35)
mock_analyzer = MagicMock()
mock_analyzer._analyze_company_safe.side_effect = [mock_fail_result, mock_ok_result]
with patch("SPARC.scheduler.DatabaseClient", return_value=mock_db), \
patch("SPARC.scheduler.CompanyAnalyzer", return_value=mock_analyzer):
from SPARC.scheduler import run_scheduled_analysis
run_scheduled_analysis()
# FailCo should not get updated, SuccessCo should
mock_db.update_tracked_company.assert_called_once_with("SuccessCo", 35)
def test_scheduler_handles_exception_in_analysis(self):
"""Scheduler continues even when analysis raises an exception."""
mock_db = MagicMock()
mock_db.list_tracked_companies.return_value = [
{"company_name": "CrashCo", "last_patent_count": 10},
{"company_name": "OKCo", "last_patent_count": 20},
]
mock_ok_result = MagicMock(success=True, patent_count=22)
mock_analyzer = MagicMock()
mock_analyzer._analyze_company_safe.side_effect = [
RuntimeError("unexpected error"),
mock_ok_result,
]
with patch("SPARC.scheduler.DatabaseClient", return_value=mock_db), \
patch("SPARC.scheduler.CompanyAnalyzer", return_value=mock_analyzer):
from SPARC.scheduler import run_scheduled_analysis
run_scheduled_analysis()
# OKCo should still be processed
mock_db.update_tracked_company.assert_called_once_with("OKCo", 22)
mock_db.close.assert_called_once()