AI Pipeline

The Rosmarium AI Pipeline is driven by a separate Python background worker (rosmarium-ai-worker), ensuring that heavy machine learning tasks never block the core Node API.

Vector Embeddings

Vector embeddings are automatically generated in the background. For the pipeline to trigger, the Content Type must have aiIntelligence.enabled = true and the entry must be published.

When these conditions are met, the Node API pushes a job to BullMQ, and the Python worker executes the following:

Chunking

The text is broken down into semantic chunks using the spaCy sentencizer or our specialized HTML structure chunker.

Embedding

The chunks are embedded into high-dimensional vectors using a local Ollama instance running nomic-embed-text, or via OpenAI API.

Vector Storage

The resulting vectors are directly upserted back to PostgreSQL utilizing the pgvector extension for blazing fast HNSW indexing.

Multi-Source Content Ingestion

Rosmarium includes a fully autonomous content ingestion pipeline capable of extracting content from multiple sources and mapping it to your structured Content Types.

Sources supported in v2.1.0+:

Web Crawling: Recursively crawls websites, extracting clean Markdown while respecting rate limits, max depths, and pagination rules.
File System: Ingest local JSON arrays, XML, or raw text files directly from the server.
Databases: Execute SQL or MongoDB queries to import raw database rows into Rosmarium.
Cloud Storage: Ingest files directly from S3 or MinIO buckets.

When an ingestion job is triggered, the AI worker performs a sequence of intelligent operations:

✦

Pluggable Extraction

Uses specialized extractors for Web (crawl4ai), Files, Databases, and Cloud Storage to unify varied unstructured data into standard Markdown representations.

✦

LLM Classification

If no target content type is specified, the pipeline dynamically determines the most appropriate Content Type using semantic similarity between the page and your schema.

✦

Field Mapping

Uses structured LLM outputs to parse the raw Markdown and map it perfectly into the rigid schema fields required by the destination Content Type.

✦

Duplicate Prevention

Employs hybrid vector search (comparing embeddings and source URLs) to silently skip content that is semantically identical to already-ingested entries.

✦

AI Configuration

Choose your preferred classification model (e.g., gpt-4o-mini) based on hardware capability and override System/User Prompts to tailor the extraction.

Intelligence Operations

Rosmarium automatically enhances your unstructured data with the following operations on publish:

✦

Native RAG Retrieval

Powered by LlamaIndex, the platform provides parallel retrieval, time-weighted freshness decay, and Cohere reranker refinement right out of the box.

✦

Zero-Shot Auto-tagging

Uses a zero-shot cross-encoder NLI model to classify incoming content against your custom taxonomy without requiring manual tagging.

✦

Named Entity Recognition

Automatically extracts PERSON, ORG, and GPE entities via spaCy, feeding them directly into the Knowledge Graph.

✦

Semantic Deduplication

Scans the vector space on save to flag identical or heavily overlapping content pieces, keeping your repository pristine.