Platform > Data Layer > Real-Time Data Processing
Real-time data processing.
Process every data event the moment it arrives - per asset, per entity, at any scale - without batch delays, middleware, or separate processing infrastructure.
CAPABILITY OVERVIEW
From data arrival to action, instantly.
Rayven's real-time data processing engine handles data the moment it enters the platform.
Every incoming event - from an IoT sensor, an API call, a form submission or a file upload - triggers immediate workflow execution. Data is written to Cassandra, indexed by UID + timestamp, and simultaneously available to workflow logic, AI models + operational dashboards.
Processing runs per entity. Each asset, customer or transaction runs its own workflow instance independently, so a site with 10,000 sensors doesn't create a queue: each sensor's data is evaluated, stored + acted on in parallel.
Inbound triggers include:
-
Event-driven triggers (data arrival from any integration)
-
Scheduled triggers (time-based, configurable interval)
-
Threshold + condition-based triggers (rule-driven)
-
Iterative triggers (per-UID, per-Label, or per-record across Primary Tables)
KEY CAPABILITIES
What Real-Time Data Processing gives you.
Cassandra time-series engine
Apache Cassandra stores every workflow data event, indexed automatically by UID, Node ID + Timestamp. Optimised for high-frequency writes, horizontal scalability + low-latency reads - purpose-built for IoT telemetry, streaming data + operational event logs at scale.
Per-UID iterative processing
Workflows execute per entity UID - every asset, sensor, customer or record runs its own independent processing instance. A fleet of 10,000 devices processes in parallel with no queue contention. Logic is tailored per entity, not generalised across the whole dataset.
Instant event-driven execution
Workflows fire the moment data arrives. No batching, no polling delay, no intermediate queue. From ingestion to workflow completion - storage, evaluation, AI processing + automated action - in milliseconds.
Hybrid SQL + Cassandra architecture
Cassandra handles all time-series, event + workflow data. MySQL handles structured relational records, Primary + Secondary Tables + entity metadata. Both are query-able through the same platform interface - no separate data warehouse, no ETL pipeline between them.
Horizontal scalability, no performance ceiling
Cassandra scales horizontally across nodes. As data volume grows, performance holds. Replication across multiple nodes ensures high availability and fault tolerance - no single point of failure, consistent performance at any scale.
Real-time availability across the platform
Processed data is immediately available to dashboards (30-second auto-refresh), AI models, workflow logic, API endpoints + external systems. No lag between data arrival and downstream availability. The platform always operates on the latest state of your data.
HOW IT CONNECTS: EXPLAINER
Where Real-Time Data Processing fits in the Rayven Platform stack.
Real-time data processing is the engine at the centre of the platform. All data from the Integration Layer enters here first - before any logic runs, any dashboard updates, or any action fires.
Once data is ingested + processed:
-
The Execution Layer receives it for workflow logic, AI evaluation + automated actions
-
The Presentation Layer reads it for live dashboards, alerts + reports
-
API endpoints make it available to external systems on demand
-
AI models + ML pipelines consume it for training, inference + real-time predictionsems
Processing is not a separate step - it is part of every workflow from the moment data arrives.
USE CASES
How Real-Time Data Processing gets used.
Per-asset processing across a large industrial fleet
A mining operation runs 8,000 assets across multiple sites - each streaming telemetry at different frequencies via MQTT. Rayven processes each asset's data independently via per-UID iterative workflows. Each asset's readings are stored in Cassandra, evaluated against its own threshold rules + routed to the correct site dashboard - simultaneously, without any asset waiting on another.
Real-time transaction processing for financial services
A payments platform routes transaction events to Rayven via HTTP POST. Each event triggers a workflow immediately - evaluating against fraud detection rules, enriching with customer context from a Primary Table + routing the result to a review queue or confirmation output. Processing completes within milliseconds of event arrival, at transaction volume.
Partner delivering a real-time data platform to multiple clients
An MSP runs multiple clients on a single Rayven instance, with Label-based UID separation keeping each client's data isolated. Real-time processing pipelines for each client run independently. Each client sees their own live data in a branded portal - without the MSP needing to run separate infrastructure per client.
Rayven Custom Integrations FAQs:
Why does Rayven use Cassandra rather than a traditional SQL database for real-time data?
Cassandra is optimised for high-frequency, high-volume time-series writes - the kind generated by IoT sensors and streaming APIs. A single Cassandra table can absorb thousands of writes per second across thousands of UIDs without locking. Learn more about storage architecture at SQL + Cassandra Storage.
How fast does data move from ingestion to storage?
Data written to the Rayven platform via any Integration Layer connector is processed and stored within seconds. For IoT/MQTT streams and webhook triggers, the end-to-end latency from ingestion to dashboard update is typically under 30 seconds. See the Integration Layer.
Can Rayven process data from multiple sources simultaneously?
Yes. A single workflow can ingest from MQTT, a REST API, a file upload and a form submission simultaneously. Each source is a node; all outputs converge at the next stage. Explore the Execution Layer.
What triggers real-time processing in Rayven?
Any inbound data event - an MQTT message, a webhook POST, a form submission, or a scheduled poll returning new data - triggers the connected workflow immediately. See Workflows + Triggers for scheduling options.
How does Rayven handle spikes in real-time data volume?
Rayven's per-UID workflow architecture means each data entity runs its own independent workflow instance. Volume spikes on one UID do not affect processing for others. The platform scales horizontally to absorb load. See the Integration Layer.
Can real-time data feed AI models instantly?
Yes. Any real-time inbound data can route directly into an AI/LLM node, ML model or anomaly detection node within the same workflow - no intermediate storage required. Learn about AI Models + Training.
Is there a delay between data arrival and dashboard update?
Dashboard auto-refresh updates data displays within 30 seconds of new data being written. For critical monitoring, Rayven's alerting engine fires instantly upon threshold breach - independent of dashboard refresh cycles. See Dashboards + Visualisations.
How is real-time data stored for historical analysis?
All ingested data is written to Cassandra (time-series) and/or MySQL (structured records) depending on data type. Historical queries run against the same dataset used for real-time processing. Learn about SQL + Cassandra Storage.
Can real-time data trigger automated control actions?
Yes. A real-time data event can flow through a Conditional Filter, evaluate a rule, and output a control command to a Modbus device, MQTT broker or external API - all within the same workflow. Explore Control + Automation.
Does real-time processing require infrastructure management from my team?
No. Rayven manages all underlying infrastructure including Cassandra cluster maintenance, scaling and data retention. Your team configures workflows and views data - infrastructure operations are fully managed. Contact us for deployment specifics.
Also in the Data Layer:
Unified Data Tables
Structured Primary + Secondary Tables for entity records, metadata + relational data alongside Cassandra time-series.
Data Management
Configure retention policies, inspect workflow payloads, export raw data + manage data lifecycle across the platform.
Data Transformation
JavaScript, Advanced Function + Combine Data nodes for schema mapping, enrichment + normalisation within workflow processing chains.
File Parsing
Ingest + parse files from FTP, S3 + manual uploads into structured, real-time data available to workflows and AI models.
Calculation + Aggregation
Sum, average, count + aggregate across UID or Label over any defined time window - at the point of processing.
AI Models + Training
Train Python ML models on Cassandra time-series data + deploy predictions as real-time workflow steps.
SQL + Cassandra Data Storage
Hybrid storage architecture - MySQL for relational records, Cassandra for time-series + event data.
Join the Shift
Discover the easy way to do something new.
Book a free 30 minute assessment with our team and we'll scope your project, needs + what a solution might look like.