A number of new Processors have been added this week to Fire. They include:
JSON Parser Processor
Decision Processor
SortBy Processor
Empty Dataset Processor
JSON Parser Processor
The JSON Parser Processor allows parsing the JSON content in a column of the DataFrame.
Suppose we read in a file which has JSON string as the content of one of the columns. We would want to parse the JSON string into new columns for further processing.
Decision Processor
The Decision Processor allows specifying a condition. Based on the condition, processing stops or proceeds further along the workflow.
An example is if there are no records in the Dataset, do not proceed with further processing. The next stage of processing might be a HIVE QL node which fires a query. And we do not want that query to be fired if there are no incoming records in the Dataset.
SortBy Processor
The SortBy Processor allows sorting the DataFrame by various columns. Users can specify a number of columns for sorting, and for each column they can specify the sorting order too.
Empty Dataset Processor
There are times when we need an empty dataset. One scenario we ran into was that the user needed to run a HIVE QL. The HIVE QL Processor needs input from the previous Processor and it passes it along as it to the next Processor after executing the user provided HIVE QL.
Since the HIVE QL Processor has to receive input from a previous Processor, Empty Dataset Processor comes handy in this scenario.
