top of page

Input Database


Name

Description

Execute BigQuery

It execute the query in BigQuery and creates a DataFrame from it.

Execute Query In Snowflake

This node executes query in Snowflake.

Hive Incremental

This node is used to incrementally read data from Hive table.

Read JDBC

This node reads data from Relational Databases using JDBC and creates a DataFrame from it

JDBC Incremental Load

This node is used to load incremental data from RDBMS to Hive.

AutoIncrement

This node reads data from Relational Databases using JDBC and creates a DataFrame from it.

Read JDBC

This node writes data to databases using JDBC.

Query JDBC

This node executes query on Relational Databases using JDBC and creates a DataFrame from it.

Read BigQuery

It reads data from BigQuery table and creates a DataFrame from it.

Read Cassandra

This node reads data from Apache Cassandra.

Read Databricks Table

This node reads a table from Databricks.

Read DynamoDB

This node reads data from DynamoDB and gets the credentials from the instance profile.

Read Elastic Search

Reads data from ElasticSearch.

Read From Snowflake

This node reads a table from Snowflake.

Read HIVE Table

This node reads data from Apache HIVE table and creates a DataFrame from it.

Read MongoDB

Reads data from MongoDB.

Read Incorta

Reads the data from Incorta schema.


Input Structured Files


Name

Description

Read Avro

Dataset Node for Reading Apache Avro Files.

Create Dataset

Creates a dataset with the specified number of rows and nine pre-defined columns.

Read CSV

It reads in CSV files and creates a DataFrame from it.

Read Delta

Dataset Node for reading Apache Delta files.

Empty Dataset

It creates an empty DataFrame.

Read Excel

Dataset Node for Reading Excel Files.

ReadFlatFile

Creates a dataset with output schema from schema field with values extracted from fixedlength.

Read HANA CSV

It reads in HANA CSV files and creates a DataFrame from it.

Read JSON

Dataset Node for Reading JSON Files.

Read LIBSVM

It reads in LIBSVM files and creates a DataFrame from it.

Read Parquet

Dataset Node for reading Apache Parquet Files.

Read Shape File

It reads in Shape files and creates a DataFrame from it.

Dataset Structured

This Node creates a DataFrame by reading data from HDFS.

URL Text File Reader

Reads a text file from the given URL and creates a DataFrame from it. Each line in the file is a record in the DataFrame.

URL Single Record JSON Reader

It reads single record JSON from the given URL and creates a DataFrame from it.

Read XML

It reads in XML files and creates a DataFrame from it.


Input Unstructured Files


Name

Description

Binary Files

Reads in Binary Files from a given path and loads them as FileName/Content.

PDF

Reads in PDF Files from a given path and extracts the text content from them.

PDF Image OCR

Reads in PDF Files from a given path.

Text Files

Reads Text Files from a given path and loads each line as a separate row.

Tika

Reads in files from a given path and parses them with Apache Tika.

Whole Text Files

Reads Whole Text Files directory from a given path and loads each file as a separate Row with key (file name) and values (file content).


Input Streaming


Name

Description

Read Kafka Batch

Dataset Node for processing the batch.


Input SFTP


Name

Description

SFTP Read

This node reads data from SFTP location.

SFTP

Secure file transfer protocol.


Output Database


Name

Description

Insert Into HIVE Table

Saves the DataFrame into an Apache HIVE Table.

Save JDBC

This node writes data to databases using JDBC.

Save As HIVE Table

Saves the DataFrame into an Apache HIVE Table.

Save Cassandra

Saves the rows of the incoming DataFrame into Apache Cassandra.

Save Databricks Table

This node saves input data as a table in Databricks.

Save DynamoDB

Saves the rows of the incoming DataFrame into DynamoDB and gets the credentials from the instance profile.

Save ElasticSearch

Stores the rows of the incoming DataFrame into Elastic Search.

Save MongoDB

It Saves the incoming Dataframe into MongoDB.

Write To BigQuery


Update JDBC

This node update the data to selected columns.

Upsert JDBC

This node insert or update the data to databases using JDBC.

Write To Snowflake



Input Enterprise Applications


Name

Description

Read Marketo

This node reads data from Marketo Files.

Read Salesforce

This node reads data from Salesforce.



Output Structured Files


Name

Description

Save Avro

Saves the DataFrame into the specified location in Apache Avro Format.

Save CSV

Saves the DataFrame into the specified location in CSV Format.

Save Delta

Saves the DataFrame into the specified location in Delta Format. When running on Hadoop.

Save JDBC

This node writes data to databases using JDBC.

Save JSON

Saves the DataFrame into the specified location in JSON Format.

Save ORC

Saves the DataFrame into the specified location in ORC Format.

Save Parquet

Saves the DataFrame into the specified location in Parquet Format. When running on Hadoop.

Save Excel

Saves the DataFrame into the specified location in XLS Format.

Save Text

Saves the DataFrame into the specified location in Text Format.


Output Streaming


Name

Description

Kafka Producer

Write out the DataFrame to a specified Apache Kafka Topic.


Output SFTP


Name

Description

SFTP Write

This node save the data to SFTP location.


Real-Time Streaming


Name

Description

Streaming Kafka

Reads in streaming text from topics in Apache Kafka.

Streaming Socket Text Stream

Reads in streaming text from a socket.

Streaming Text File Stream

It monitors a specified directory for new files. It keeps reading in any new files created in the directory.


Others

Name

Description

CDC Using Full Table Merge

CDC Using Full Table Merge.

Columns Rename

This node creates a new DataFrame by renaming existing columns with the new name.

Count

This node counts the number of records in the incoming Dataframe and puts the count into result page.

DeltaDebeziumMerge


Explode

Explode the array of values into multiple rows with columnname_explode.

Flatten


Formula

test

Geo IP

This node converts IP to geo location.

Geo Point


JsonToEDI


Multi Window Analytics


Multi Window Ranking


DeltaDebeziumMerge


Recover Hive Partitions

Node to recover the partitions of external Hive table.

Register TempTable

This node registers the incoming DataFrame as a temporary table in Spark.

Round Value


Sample

Samples the incoming DataFrame.

SaveWaterMark

This node save the value in watermark variable in workflow to file.

Sort By

It sorts the incoming DataFrame on the fields specified.

Sort Columns

It sort the columns selection.

Transpose

This node transposes a dataframe without performing aggregation function by given column(transposeby). All Input columns to this node have to be of the same type.

Window Aggregation

This node calculates the moving values of selected functions for the field(input column).

Window Analytics


Window Ranking


Word Count



Structured Streaming


Name

Description

Structured Streaming Console Sink

It outputs the DataFrame to the console.

Structured Streaming CSV

It monitors a specified directory for new files. It keeps reading in any new files created in the directory.

Structured Streaming File Sink

It writes the DataFrame to files with Structured Streaming.

Structured Streaming JSON

It monitors a specified directory for new files. It keeps reading in any new files created in the directory.

Structured Streaming Kafka

Reads in streaming text from topics in Apache Kafka.

Structured Streaming Kinesis

Reads in streaming text from Kinesis stream.

Structured Streaming Socket

Reads in streaming text from a socket.

Structured Streaming Hive Sink

Saves the streaming data into a HIVE Table.

Structured Streaming Hive Sink2

Saves the streaming data into an Apache HIVE Table.


bottom of page