Platform > Data Layer > Data Management
Data management.
A unified, real-time data management layer - no data warehouse, no data lake, no middleware between your data + the people who need it.
CAPABILITY OVERVIEW
All your data, managed in one place.
Rayven's data management capabilities give you complete visibility and control over all data flowing through the platform.
Inspect payload history at any workflow node, configure data retention policies, monitor stream quality in real-time + export raw data on demand. Every data point is stored, secured + accessible - without the cost and complexity of separate data warehousing infrastructure.
No ETL processes, no intermediary storage, no additional tooling.
Inbound triggers include:
Real-time data streams from any integration
File uploads + batch ingestion
Form submissions + manual entry
Workflow-generated data
API-ingested event data
KEY CAPABILITIES
What Data Management gives you.
Data Repository node
Configure data retention settings per workflow, controlling how long data is stored in Cassandra. Set uniqueness constraints by UID + timestamp to prevent duplicate records. Retention policies apply per node - giving granular lifecycle control across the platform.
Inspect Data tab
A per-node payload viewer showing exactly what data passed through each workflow step. Filter by UID, date range + content. Essential for troubleshooting, auditing + verifying data quality at any point in the processing chain.
Node Export
Download raw data for any workflow node over any selected time range. Returns the full payload as it existed at that node - no transformation applied. Used for compliance exports, data analysis + feeding external systems with raw operational data.
Real-time quality monitoring
Monitor data streams using custom logic and dashboards. Detect data gaps, anomalies + stream failures in real-time using the Detect Outage node + Rule Builder. Configure alerts to fire immediately when a source goes silent or returns unexpected values.
Tag + Label management
Tag incoming data streams with Labels to apply structure, context + access control as data enters the platform. Labels enable grouping by site, region or customer - used across workflows, dashboards + API endpoints to filter and segment data.
No data warehouse required
Rayven's hybrid SQL + Cassandra architecture replaces the need for a separate data warehouse or data lake. All data - structured records, time-series events + workflow payloads - is stored, queryable + accessible in real-time from one platform.
HOW IT CONNECTS: EXPLAINER
Where Data Management fits in the Rayven Platform stack.
Data management sits at the centre of the Data Layer, providing the control plane for all data stored across the platform.
-
Data enters via the Integration Layer from any source. The Data Management layer governs retention, quality monitoring + access to that data.
-
The Execution Layer reads managed data to power workflow logic + AI processing.
-
The Presentation Layer surfaces managed data in dashboards, reports + interfaces.
-
External systems access managed data via API endpoints on-demand.
USE CASES
How Data Management gets used.
Compliance data retention for a regulated industry
A utilities company configures retention policies per workflow node - operational telemetry retained for 90 days, compliance event logs for seven years. Node Export provides auditable, timestamped raw data extracts on demand. No separate compliance database required.
Real-time data quality monitoring for a retail operation
A retail chain monitors 200 store data feeds using Detect Outage nodes per feed. If any store's data stops arriving, an alert fires to the operations team within the detection window. Dashboard widgets surface data quality status per store in real-time.
Partner managing data lifecycle across multiple clients
An MSP configures per-client retention policies within a shared Rayven instance. Label-based segmentation keeps each client's data isolated. Node Export provides per-client data extracts when required. One platform, client-specific governance, no separate infrastructure per client.
Rayven Data Management FAQs:
Does Rayven replace the need for a data warehouse?
For operational data - real-time sensor feeds, workflow outputs, inspection records - yes. Rayven stores, structures and makes this data immediately queryable without a separate data warehouse layer. For strategic BI and long-term analytics it integrates with existing warehouse tools. See the Data Layer.
How is data governance enforced in Rayven?
Access to data is governed by label-based access control. Each UID is assigned labels; users and roles are assigned matching permissions. No user can query data outside their permission set, regardless of which interface they use. See Security.
Can I set data retention windows?
Yes. Retention periods are configurable per data type and table. Time-series Cassandra data can be set to auto-expire after a defined period. Structured MySQL records can be archived or deleted on schedule. Learn about SQL + Cassandra Storage.
Who manages the underlying database infrastructure?
Rayven manages all database infrastructure including MySQL and Cassandra clusters, backups, failover and scaling. Your team manages schema design, retention settings and access control within the platform interface. Contact us for SLA details.
Can I define my own table schemas?
Yes. Primary and Secondary Table schemas are fully configurable - define column names, data types, validation rules and links to UIDs. Schema changes can be applied without stopping active workflows. See Unified Data Tables.
How is data quality enforced at the point of ingestion?
Workflows include validation nodes - Conditional Filters, Rule Builders and JavaScript transformation nodes - that check incoming data before writing. Invalid records can be flagged, quarantined or rerouted without breaking the main pipeline. Explore Data Transformation.
Can data from different sources be merged into a single record?
Yes. UID-based architecture allows data from multiple Integration Layer sources to merge into the same entity record. CRM data, sensor readings and form submissions all link to the same UID. See Unified Data Tables.
Is historical data accessible for retrospective analysis?
Yes. All historical records remain queryable through workflow nodes, dashboard widgets and API endpoints. Retention limits are the only constraint. See Real-time Data Processing.
Can I audit who accessed or modified data?
Audit logging for data access and workflow execution is available within the platform. See the Security documentation for full details on logging, access tracking and compliance tooling.
Can Rayven integrate with an external data governance tool?
Yes. Data can be exported via API endpoints or SQL write nodes to external governance, cataloguing or BI platforms. Rayven does not lock data in - all stored data is accessible to authorised external systems. See the Integration Layer.
Also in the Data Layer:
Unified Data Tables
Structured Primary + Secondary Tables for entity records, metadata + relational data alongside Cassandra time-series.
Real-time Data Processing
Sub-second ingestion + processing of live sensor, device + event data with built-in deduplication + schema validation.
Data Transformation
JavaScript, Advanced Function + Combine Data nodes for schema mapping, enrichment + normalisation within workflow processing chains.
File Parsing
Ingest + parse files from FTP, S3 + manual uploads into structured, real-time data available to workflows and AI models.
Calculation + Aggregation
Sum, average, count + aggregate across UID or Label over any defined time window - at the point of processing.
AI Models + Training
Train Python ML models on Cassandra time-series data + deploy predictions as real-time workflow steps.
SQL + Cassandra Data Storage
Hybrid storage architecture - MySQL for relational records, Cassandra for time-series + event data.
Join the Shift
Discover the easy way to do something new.
Book a free 30 minute assessment with our team and we'll scope your project, needs + what a solution might look like.