Open Sourceai

WatchNode

Open-source full-stack log monitoring with Hugging Face models detecting sequential, semantic, and statistical anomalies in production logs.

View GitHub

About the Project

WatchNode is an open-source observability platform that ingests application logs and surfaces sequential, semantic, and statistical anomalies using Hugging Face inference — helping teams catch issues before they become incidents.

Problem

Rule-based alerting drowns teams in noise and misses subtle patterns in high-volume log streams. Manual triage does not scale as systems grow.

Features I Handled

Described using the STAR method — Situation, Task, Action, Result.

Feature 1

Log Ingestion Pipeline

Situation

Customers emit high-volume, heterogeneous logs from multiple services with inconsistent formats.

Task

Build a reliable ingestion path that normalizes events and never blocks the write path.

Action

Designed an event-driven pipeline: ingestion API → BullMQ workers on Redis → persistence in MongoDB with tenant-scoped collections.

Result

Sustained ingestion under load with async processing decoupled from the API response cycle.

Feature 2

Hugging Face Anomaly Detection

Situation

Static thresholds failed to catch emerging failure patterns across different workloads and log semantics.

Task

Detect sequential, semantic, and statistical anomalies in log streams using production-grade ML inference.

Action

Integrated Hugging Face models via @huggingface/inference in worker processes — scoring temporal sequences, semantic embeddings, and statistical deviations before alert emission.

Result

Reduced false positives versus pure threshold rules and improved mean time to detect subtle log anomalies.

Feature 3

Multi-Tenant Dashboard

Situation

Multiple organizations needed isolated data, auth, and alerting configs on shared infrastructure.

Task

Deliver a secure multi-tenant product surface with JWT auth and per-tenant configuration.

Action

Built a Next.js dashboard with tenant-aware API routes, role-based access, and real-time alert views.

Result

Production-ready open-source observability at watchnode.io with isolated data and self-serve workflows.

Architecture & Technical Decisions

WatchNode is an open-source, event-driven full-stack platform with async workers — API and BullMQ workers share deployment units on AWS.

Open source?

Yes — full-stack on GitHub

Serverless?

No — persistent workers for ML + queues

Messaging

Redis + BullMQ

AI/ML

Hugging Face — sequential, semantic, statistical

Architecture pattern: Event-driven modular backend (API + BullMQ workers)
Hosting & deployment: AWS (containerized services, not fully serverless)
Data layer: MongoDB for metadata and log indexes; Redis for queues and caching
Integrations & services: BullMQ job queues; Hugging Face inference for anomaly detection; email/webhook alerting

flowchart LR
  ingest[LogIngestionAPI] --> queue[BullMQWorkers]
  queue --> ml[HuggingFaceInference]
  ml --> store[(MongoDB)]
  store --> api[RESTAPI]
  api --> dashboard[NextjsDashboard]
  api --> alerts[AlertService]