Shepard AI
Engineering Teardown
A crypto movement-intelligence MVP built around one constraint: expensive ingestion must not sit on the user's request path. I separated market-data storage from account infrastructure, scheduled collection, and served dashboard panels from a database cache.
01 / Scheduling without request storms
The production Docker scheduler runs scanner, market overview, and snapshots every 900 seconds; RSS every 1,800 seconds. Jobs execute sequentially, use monotonic deadlines, and authenticate internal calls with a server-only cron secret. Dashboard loads read stored panels, not the external providers.
Trade-off: this is a single-instance, cron-equivalent scheduler, not a distributed queue. Deadlines reset after restart; a slow job can delay the next one. Failed jobs retry on their next scheduled cycle. Run records distinguish failure from a successful empty scan.
Code reviewed: deploy/app/scheduler.py
02 / RSS deduplication, precisely
Each item receives a fingerprint from provider:title:url: UTF-8 encode the first 500 characters, compute SHA-256, retain the first 12 bytes, and hex-encode them into a 24-character key. Aggregation uses a Map keyed by url || title_hash, so repeated URLs collapse within a sweep.
Trade-off: this is exact-identity deduplication, not semantic clustering. Syndicated copies with different URLs can survive. Provider-specific hashes and truncated input are deliberate MVP limitations; URL canonicalization and normalized-title matching are next.
Code reviewed: _shared/sentiment-engine.ts
03 / PostgreSQL: access paths & boundaries
Heavy snapshots and market cache live in Docker PostgreSQL, exposed through private PostgREST behind an HTTPS gateway. Supabase retains authentication, subscriptions, and usage enforcement. Composite B-tree indexes match symbol/timeframe lookups and latest-panel queries:
(symbol, timeframe, created_at DESC, id DESC)
(panel_type, created_at DESC, id DESC)PostgREST reuses its built-in database connection pool; the Compose configuration does not pin a custom pool size or deploy PgBouncer. Hourly bounded retention caps historical growth; PostgreSQL can reuse deleted space, but deletion does not immediately shrink files.
Code reviewed: retention_covering_indexes.sql / deploy/app/compose.yaml
04 / Rate limits: bounded recovery
The AI router uses timeouts and per-provider/model cooldowns. On HTTP 429, numeric Retry-After is clamped to 60 seconds–1 hour; other HTTP failures cool down for 30 seconds. It moves to the next eligible free provider instead of immediately hammering the failed one.
The separate Python/Celery engine enables exponential backoff and jitter for database/Redis task failures. That is not a universal API retry policy. Dashboard automation uses scheduled retry and cache serving; a failed mutating request is not blindly replayed. AI usage reservation makes identical retries idempotent.
Code reviewed: _shared/llm-router.ts / shepard_engine/worker.py