Jan 15, 20191 min
Fire Insights now provides building and executing Pipelines. Pipelines consist of workflows connected to each other.
The framework is flexible and would support more nodes types to be part of the Pipeline in the coming weeks.
Pipelines in Fire Insights is currently integrated with Apache Airflow. For execution, the pipeline is submitted to Airflow.
More Details of Pipelines in Fire Insights is available here:
http://docs.sparkflows.io/en/latest/installation-upgrading/configuring-pipeline.html
http://docs.sparkflows.io/en/latest/user-guide/pipelines.html
Fire uses Apache Airflow for executing Pipelines. Hence Airflow has to be installed on the same machine as Fire.
Below are the configurations needed in Fire.
Each Application now allows creating Pipelines.
Below is an example Pipeline with 3 Workflows.
Pipelines can be executed like workflows. When a Pipeline is executed, its execution is submitted to Airflow.
The Pipeline tab displays the list of Pipelines for the current Application.
Clicking on the Execute Action icon opens the Pipeline Execute Page.
Once a Pipeline is fired, its details are visible in Pipeline Executions.
The Pipeline Details page displays the list of workflow executions associated with the Pipeline.