Platform > Integration Layer > Streaming Data Connectors
Streaming data connectors.
Ingest continuous, high-volume data streams from AWS Kinesis, MQTT brokers + AMQP queues directly into Rayven's time-series data engine - in real-time, at-scale.
CAPABILITY OVERVIEW
High-frequency data ingestion at-scale.
Rayven's streaming data connectors handle continuous, high-volume data flows from event streams, message queues + IoT telemetry sources.
Every incoming event is written to Cassandra, indexed by UID + timestamp, and immediately available to real-time workflow logic.
Designed for environments where data arrives continuously and latency matters - from sensor telemetry to financial transaction streams and app event logs.
Inbound connections include:
-
AWS Kinesis (stream ingestion, configurable polling)
-
MQTT (real-time broker subscription)
-
AMQP (message queue consumption)
-
HTTP streaming (real-time POST ingestion)
-
FTP polling (high-frequency scheduled file ingestion)
KEY CAPABILITIES
What Streaming Data Connectors give you.
AWS Kinesis
Ingest high-volume event streams from AWS Kinesis with configurable polling intervals. UID mapping from a nominated JSON key enables per-entity storage across large event volumes.
MQTT
Subscribe to MQTT brokers for real-time device + application event streams. Configurable topic filters, QoS levels + certificate-based auth for secure high-frequency ingestion.
AMQP
Consume messages from AMQP-compliant brokers including RabbitMQ + ActiveMQ. Each message triggers a workflow execution independently, supporting event-driven architectures at volume.
Cassandra time-series storage
All streaming data is written to Cassandra, auto-indexed by UID + timestamp. Optimised for high-frequency writes across thousands of concurrent streams without performance degradation.
Real-time workflow execution
Every ingested event fires the connected workflow immediately. No batching delay, no polling lag. Downstream AI, transformation + alert logic executes in real-time against each arriving event.
Per-UID iterative processing
Streaming data is processed per entity UID. Each device, customer or transaction runs through its own workflow instance independently - enabling fleet-scale operations without data collision.
HOW IT CONNECTS: EXPLAINER
Where Streaming Data Connectors fit in the Rayven Platform stack.
Streaming connectors sit at the base of the Integration Layer, feeding continuous data flows into the platform.
Once ingested, streaming data flows into:
-
The Data Layer where Cassandra stores each event indexed by UID + timestamp + aggregation nodes calculate rolling metrics
-
The Execution Layer where workflows evaluate each event in real-time, running ML models + triggering alerts on threshold breaches
-
The Presentation Layer for live operational dashboards refreshing every 30 seconds
Streaming + batch ingestion can run in the same workflow - combining real-time events with scheduled data pulls from other sources.
USE CASES
How Streaming Data gets used.
Industrial telemetry processing
A manufacturing site streams 5,000 machine readings per second via MQTT. Rayven stores each reading per-asset in Cassandra, runs a rolling anomaly detection algorithm + fires an alert if a reading deviates beyond threshold - without any batching or delay.
Financial transaction stream monitoring
AWS Kinesis delivers payment transaction events at high volume. Rayven ingests each event, evaluates against fraud detection rules + writes flagged transactions to a review queue - at transaction speed, in real-time.
Partner-built IoT monitoring platform
An MSP ingests client device telemetry via MQTT into Rayven with per-client UID separation. Each client accesses a branded portal showing their own real-time asset data - delivered as the partner's own monitoring product.
Streaming Data FAQs:
What streaming protocols does Rayven support?
AWS Kinesis, MQTT, AMQP, HTTP streaming + high-frequency FTP polling. These cover the primary event stream, message queue + device telemetry architectures across enterprise + industrial environments. See all Integration Layer options.
How is high-frequency streaming data stored?
All streaming events are written to Apache Cassandra, indexed by UID + timestamp. Cassandra's architecture is optimised for high-frequency time-series writes - thousands of concurrent streams is a standard configuration. Learn more about Data Layer storage.
Is there a maximum ingestion rate?
The platform is designed for high-volume streaming at scale. Ingestion rates depend on deployment configuration. Contact us to discuss specific throughput requirements for your use case.
How quickly does workflow logic execute after a streaming event arrives?
Immediately. Every ingested event triggers the connected workflow in real-time. There is no batching delay. Downstream processing, AI evaluation + alerts execute within milliseconds of event arrival. Explore the Execution Layer.
Can streaming data from multiple sources combine in one workflow?
Yes. A single workflow can ingest from MQTT + Kinesis simultaneously, or combine streaming data with scheduled API pulls or file ingestion. All data flows through the same transformation + logic nodes. See the Integration Layer.
How does Rayven handle MQTT authentication?
MQTT connections support certificate-based authentication + username/password credentials. Configurable TLS settings secure data in transit. Credentials are stored securely within the platform. See platform security for full details.
Can streaming data train or update ML models?
Yes. Cassandra-stored time-series data from streaming sources is available as a training dataset for Rayven's Python ML modeller. Models can be updated continuously as new data accumulates. See AI Models + Training.
What happens if a streaming source goes offline?
The workflow stops executing for that stream when events cease. The Detect Outage node can alert on gaps in expected data arrival, flagging stream interruptions in real-time. See Notifications + Alerts.
Can per-UID streaming data feed individual customer or asset dashboards?
Yes. Per-UID storage means each entity's streaming data is isolated + independently queryable. Label-based access control allows each user to see only their own entity's data. Learn how the Data Layer manages per-UID records.
Is AWS Kinesis the only cloud streaming service supported?
AWS Kinesis is the primary cloud streaming connector. For other cloud event services (Azure Event Hubs, Google Pub/Sub), use the HTTP streaming or AMQP nodes. Contact us about specific connector requirements.
Also in the Integration Layer:
Pre-Built IT/OT Connectors
150+ ready-to-use connectors across CRM, ERP, BI, AI/LLM services + industrial systems.
File Uploads
Ingest files directly via FTP, FTPS, SFTP + AWS S3, or through manual uploads within the platform.
Forms + Manual Input
Capture structured data from human input via configurable form widgets, feeding directly into workflows.
IoT Devices + Protocols
Native support for MQTT, Modbus, LoRaWAN, SNMP + Raw UDP for edge and industrial device connectivity.
API Endpoints
Expose processed platform data to external systems via authenticated GET endpoints + inbound POST hooks.
AI Connectors
Connect to OpenAI, Claude, Gemini, Cohere + more as workflow nodes for LLM-powered integration + processing.
Custom Integrations
Build bespoke two-way integrations via HTTP request nodes, REST API adapters + custom webhook endpoints.
Join the Shift
Discover the easy way to do something new.
Book a demo with our team and we'll show you exactly how Rayven can work for your environment.