Platform > Data Layer > Data Transformation
Data transformation.
Convert, clean, enrich + reshape any data as it moves through the platform - in real-time, without custom scripts, or separate ETL infrastructure.
CAPABILITY OVERVIEW
Clean, structured data at every every step.
Rayven's data transformation toolkit handles every schema conversion, enrichment + normalisation task within the same workflow builder used for ingestion and automation.
JavaScript for full custom logic, Advanced Function for prebuilt formulas, Combine Data for multi-source merges, Extract JSON Key for schema mapping + Conditional Filter for data routing.
Transformation happens inline as data flows through the workflow - no staging environment, no separate ETL pipeline, no transformation delay.
Inbound triggers include:
-
Raw data in any format (JSON, XML, CSV, String, binary)
-
Multi-source payloads requiring merging
-
Nested JSON structures requiring field extraction
-
Arrays requiring splitting into individual records
-
Data with missing or inconsistent field values
Outbound triggers include:
-
Clean, normalised JSON payloads for storage or API delivery
-
Merged multi-source datasets
-
Extracted + structured field sets
-
Format-converted outputs (JSON, CSV, XML, String)
KEY CAPABILITIES
What Data Transformation gives you.
JavaScript node
Write full custom JavaScript logic at any point in a workflow. Remap field names, convert data types, apply business rules, build conditional logic + construct any output format required. Full code control where prebuilt nodes are not enough.
Advanced Function node
A prebuilt formula library covering mathematical operations, time-series calculations, statistical functions + label-based aggregations. Apply complex logic without writing code - ideal for rolling averages, delta calculations + threshold evaluations.
Combine Data node
Merge payloads from multiple workflow sources or UIDs into a single unified payload. Used for joining data from different integrations, combining sensor readings with entity metadata, or consolidating multi-source inputs before downstream processing.
Extract JSON Key + schema mapping
Extract specific fields from nested JSON payloads using the Extract JSON Key node. Supports deep nesting + wildcard key selection. Combine with the JavaScript node for full schema remapping between source and destination formats.
Associate or Split Payload
Split array payloads into individual records for per-item workflow processing - essential when an API returns a batch of records needing individual evaluation. Associate Payload links records from different sources for combined downstream logic.
Automated ETL/ELT pipelines
Transform, extract + load as a native workflow function - no separate ETL tool required. Supports missing value handling (backfill), deduplication + format conversion (CSV to JSON, XML to JSON + more) within the same pipeline.
HOW IT CONNECTS: EXPLAINER
Where Data Transformation fits in the Rayven Platform stack.
Data transformation nodes sit in the Data Layer between ingestion and storage.
-
Raw data arrives from the Integration Layer in any format or schema.
-
Transformation nodes normalise, enrich + restructure data within the workflow.
-
Clean, structured outputs write to MySQL or Cassandra for storage.
-
The Execution Layer uses clean data for workflow logic, AI + automation.
-
Transformation also occurs at the output stage - converting stored data into the format required by downstream systems or API endpoints.
USE CASES
How Data Transformation gets used.
Schema normalisation across multiple industrial systems
A manufacturing site ingests data from three ERP systems with different field structures. A JavaScript node remaps all three to a unified schema before writing to Cassandra. Downstream dashboards and ML models always receive consistently structured data - regardless of which source it came from.
Financial data enrichment + aggregation
Transaction records arrive via API with nested JSON structures. An Extract JSON Key node pulls relevant fields, a JavaScript node calculates margin per transaction, and a Combine Data node merges results with customer metadata from a Primary Table. The enriched record writes to a dashboard table.
Partner normalising multi-client data formats
An MSP ingests exports from clients in CSV, XML + JSON formats. A transformation workflow normalises all three into a standard schema per client UID. Downstream dashboards + reporting workflows use one consistent data model regardless of client export format.
Rayven Data Transformation FAQs:
What transformation nodes are available in Rayven?
JavaScript Node, Advanced Function Node, Formula Builder, Conditional Filter, Rule Builder and Push/Pull Table Row nodes. These cover schema mapping, value calculation, conditional branching and data enrichment. Explore the full Execution Layer.
Can I transform data without writing code?
Yes. The Formula Builder and Conditional Filter nodes provide no-code transformation for common operations - field mapping, value substitution and threshold evaluation. For complex logic, the JavaScript Node provides full scripting capability. See the Execution Layer.
How does schema mapping work when sources differ?
The JavaScript Node allows arbitrary reshaping - rename fields, merge payloads, flatten nested JSON, extract sub-arrays. This normalises data from disparate sources into a consistent schema before storage. See Unified Data Tables.
Can transformation logic call external APIs?
Yes. The Output to HTTP node within a workflow can call an external API mid-workflow and feed the response into the next node. This supports data enrichment from third-party services within the same execution chain. Explore the Integration Layer.
Can AI be used as a transformation step?
Yes. An AI/LLM node can act as a transformation node - extracting structured fields from unstructured text, classifying records or generating calculated values. The output feeds downstream nodes like any other transformation result. See AI Models + Training.
How are transformation errors handled?
The Conditional Filter and Error Handler nodes route unexpected values or transformation failures to alternative paths. This prevents bad data from blocking the main workflow pipeline. Alerts can be fired on transformation failure via Notifications + Alerts.
Can transformation nodes access historical data from tables?
Yes. Pull Table Row nodes retrieve existing records by UID mid-workflow. This allows transformation logic to compare incoming data against stored history - for delta calculations, running totals or state tracking. See Unified Data Tables.
What is the Advanced Function Node used for?
The Advanced Function Node executes server-side Python or JavaScript for computationally intensive transformations - statistical calculations, image processing, regex parsing and data structure manipulation beyond what the Formula Builder supports. Explore the Execution Layer.
Can transformation outputs feed multiple downstream destinations simultaneously?
Yes. A single transformation node can branch into multiple output paths - storing to Cassandra, writing to a Secondary Table, calling an API and triggering an alert - all within the same workflow execution. Explore the Execution Layer.
Is there version control for transformation logic?
Workflow versions are managed in the workflow builder. Previous versions can be restored. Transformation nodes inherit the versioning of their parent workflow. See Workflows + Triggers for more.
Also in the Data Layer:
Unified Data Tables
Structured Primary + Secondary Tables for entity records, metadata + relational data alongside Cassandra time-series.
Data Management
Configure retention policies, inspect workflow payloads, export raw data + manage data lifecycle across the platform.
Real-time Data Processing
Sub-second ingestion + processing of live sensor, device + event data with built-in deduplication + schema validation.
File Parsing
Ingest + parse files from FTP, S3 + manual uploads into structured, real-time data available to workflows and AI models.
Calculation + Aggregation
Sum, average, count + aggregate across UID or Label over any defined time window - at the point of processing.
AI Models + Training
Train Python ML models on Cassandra time-series data + deploy predictions as real-time workflow steps.
SQL + Cassandra Data Storage
Hybrid storage architecture - MySQL for relational records, Cassandra for time-series + event data.
Join the Shift
Discover the easy way to do something new.
Book a free 30 minute assessment with our team and we'll scope your project, needs + what a solution might look like.