BLOG | sparkflows

All Posts

Jul 3
1 min

Data Profiling In Fire Insights

Data Profiling is extremely helpful in understanding the data. Fire Insights provides a number of processors for users to profile their d...

22 viewsWrite a comment

May 16
1 min

Fire Insights is deeply integrated with Databricks

Overview Databricks is the leader in Apache Spark. With Databricks it is extremely easy to bring up and manage Apache Spark clusters and ...

23 viewsWrite a comment

Feb 7
1 min

Models in Sparkflows

ML Models in Fire Insights Overview Fire enables you to create ML models. Fire supports ML models using Spark ML, H2O, Scikit Learn, AWS ...

31 viewsWrite a comment

Dec 12, 2019
2 min

Export/ Import of Applications

Overview Fire enables you to export your Applications and download them to your computer. It then also enables you to import your Applica...

17 viewsWrite a comment

Aug 31, 2019
2 min

File Watcher with AWS & Sparkflows

Overview There are many use cases where we have to process the incoming files on S3. This document describes one way to achieve it with S...

67 viewsWrite a comment

Jul 8, 2019
1 min

Sparkflows is now integrated with H2O

Overview Sparkflows is now integrated with H2O. It has 8 processors for H2O which can be immediately used. Distributed Random Forest Grad...

45 viewsWrite a comment

May 10, 2019
1 min

Reading and Writing from MongoDB

MongoDB is a document database with the scalability and flexibility that you want with the querying and indexing that you need,

39 viewsWrite a comment

Feb 12, 2019
1 min

Interactive Dashboards in Fire

Overview Fire Insights now provides analyzing data with Interactive Dashboards. Interactive Dashboards allow slicing and dicing the data ...

53 viewsWrite a comment

Jan 15, 2019
1 min

Fire Insights now has Pipelines

Overview Fire Insights now provides building and executing Pipelines. Pipelines consist of workflows connected to each other. The framewo...

54 viewsWrite a comment

Jan 7, 2019
1 min

New Count and Assert Nodes in Fire

Overview There are use cases where we want to focus on Data Quality. One of the use cases is that if the number of records being processe...

29 viewsWrite a comment

Jan 7, 2019
1 min

Fire Insights is now Integrated with Amazon Sagemaker

Overview Fire Insights is now integrated with Amazon Sagemaker. The workflows can now use Sagemaker Processors in Fire to run Machine Lea...

71 viewsWrite a comment

Nov 29, 2018
1 min

Credentials Support in Fire

Overview Fire now supports defining credentials at the Applications/Project Level. Credentials are very important as sensitive data like ...

36 viewsWrite a comment

Nov 7, 2018
1 min

Structured Streaming in Fire

Overview We are excited to announce that Fire now supports Structured Streaming. You can connect with Kafka, make transforms, analytics a...

64 viewsWrite a comment

Oct 22, 2018
1 min

Applications in Fire

We are excited to release the new feature Applications. Applications are a natural way users and teams to work build Intelligent Applicat...

46 viewsWrite a comment

Sep 10, 2018
1 min

Triggering Workflows by Event

As Fire is powering more and more features on Big Data Analytics and Engineering, there are more and more Operational needs. Fire already...

63 viewsWrite a comment

Sep 5, 2018
1 min

New Processors - Decision / JSON Parser / SortBy / Empty Dataset

A number of new Processors have been added this week to Fire. They include: JSON Parser Processor Decision Processor SortBy Processor Emp...

47 viewsWrite a comment

Aug 2, 2018
2 min

Running Python Code in Sparkflows Fire Workflows

One of the things users want today is to be able to write and use native Python libraries when running on Big Data Platforms. The adoptio...

180 viewsWrite a comment

Jun 6, 2018
1 min

Powerful Data Validation with Sparkflows

One of the core needs of Data Processing is Data Validation. Sparkflows provides powerful capability in this regard. At a high level, it ...

154 viewsWrite a comment

Mar 20, 2018
1 min

Running Sparkflows on AWS EMR

Sparkflows can easily power your AWS EMR cluster. With its simplicity, power, ease of use and 200+ Processors available out of the box, i...

124 viewsWrite a comment

Jan 24, 2018
2 min

Powerful Data Prep & ETL with Sparkflows

Sparkflows allows you to quickly do very complex data preparation and ETL on Big Data. Sparkflows has a number of features for enabling i...

265 viewsWrite a comment

{"items":["5efecfb603ce9000175a5cbc","5ebfc5ed977c5e00170e78e2","5e3d48fe4494ca00173d527e","5df1f74bc3e8e8001705cd8e","5d6537f2fa8715001682c19c","5d2188e48e975d027eeb96db","5cd5bdf3fd3a43064a029a6c","5cce487d1c19bc08494f0f6f","5cce487d8b61e609e5a566c5","5cce487d27f0b2091b56c8ce","5cce487da9d7c20870c9effd","5cce487def4be2083ed4c341","5cce487df7e91708e621f355","5cce487df7e91708e621f357","5cce487de184cc08e48c5466","5cce487df7e91708e621f356","5cce487d2d320907e380b7d8","5cce487df7e91708e621f359","5cce487dfd99f70817bc59c9","5cce487d2ff22508c54f4298"],"styles":{"groupSize":1,"showArrows":true,"cubeImages":false,"cubeType":"fill","cubeRatio":1,"isVertical":true,"gallerySize":454,"collageAmount":0,"collageDensity":0,"groupTypes":"1","oneRow":false,"imageMargin":16,"galleryMargin":0,"floatingImages":0,"chooseBestGroup":true,"smartCrop":false,"hasThumbnails":false,"enableScroll":true,"isGrid":false,"isSlider":false,"isColumns":false,"isSlideshow":false,"cropOnlyFill":false,"fixedColumns":0,"enableInfiniteScroll":true,"isRTL":false,"minItemSize":120,"rotatingGroupTypes":"","rotatingCubeRatio":"","gallerySliderImageRatio":1.7777777777777777,"numberOfImagesPerRow":3,"numberOfImagesPerCol":1,"groupsPerStrip":0,"borderRadius":0,"boxShadow":0,"gridStyle":0,"mobilePanorama":false,"placeGroupsLtr":false,"viewMode":"preview","thumbnailSpacings":4,"galleryThumbnailsAlignment":"bottom","isMasonry":true,"isAutoSlideshow":false,"slideshowLoop":false,"autoSlideshowInterval":4,"useCustomButton":false,"bottomInfoHeight":0,"titlePlacement":"SHOW_BELOW","galleryHorizontalAlign":"center","galleryTextAlign":"center","galleryVerticalAlign":"center","scrollSnap":false,"itemClick":"expand","fullscreen":true,"allowSocial":true,"allowDownload":false,"allowTitle":true,"allowDescription":false,"loveButton":true,"loveCounter":false,"videoPlay":"hover","scrollAnimation":"NO_EFFECT","scrollDirection":0,"overlayAnimation":"FADE_IN","arrowsPosition":0,"arrowsSize":23,"watermarkOpacity":40,"watermarkSize":40,"useWatermark":true,"watermarkDock":{"top":"auto","left":"auto","right":0,"bottom":0,"transform":"translate3d(0,0,0)"},"loadMoreAmount":"all","defaultShowInfoExpand":1,"allowTitleExpand":true,"allowDescriptionExpand":true,"allowLinkExpand":true,"expandInfoPosition":0,"allowFullscreenExpand":true,"fullscreenLoop":false,"galleryAlignExpand":"left","addToCartBorderWidth":1,"addToCartButtonText":"","slideshowInfoSize":200,"playButtonForAutoSlideShow":false,"allowSlideshowCounter":false,"hoveringBehaviour":"NEVER_SHOW","thumbnailSize":120,"magicLayoutSeed":1,"imageHoverAnimation":"NO_EFFECT","calculateTextBoxHeightMode":"MANUAL","calculateTextBoxWidthMode":"PERCENT","textBoxHeight":320,"textBoxWidth":200,"textBoxWidthPercent":50,"textImageSpace":10,"textBoxBorderRadius":0,"textBoxBorderWidth":0,"textsVerticalPadding":-15,"textsHorizontalPadding":-30,"titleDescriptionSpace":6,"customButtonText":"","customButtonBorderWidth":1,"customButtonBorderRadius":0,"loadMoreButtonText":"","loadMoreButtonBorderWidth":1,"loadMoreButtonBorderRadius":0,"imageInfoType":"ATTACHED_BACKGROUND","itemBorderWidth":1,"itemBorderRadius":0,"itemEnableShadow":false,"itemShadowBlur":20,"itemShadowDirection":135,"itemShadowSize":10,"imageLoadingMode":"BLUR","expandAnimation":"NO_EFFECT","imageQuality":90,"usmToggle":false,"usm_a":0,"usm_r":0,"usm_t":0,"videoSound":false,"videoSpeed":"1","videoLoop":true,"gallerySizeType":"px","gallerySizePx":454,"itemBorderColor":{"themeName":"color_16","value":"rgba(255,255,255,1)"},"galleryLayout":1,"selectedLayout":"1|bottom|1|fill|true|0|true","layoutsVersion":2,"selectedLayoutV2":1,"isSlideshowFont":true,"externalInfoHeight":320,"externalInfoWidth":0},"container":{"width":-40,"galleryWidth":2592,"galleryHeight":1440,"scrollBase":0,"height":null}}

Innovate with Data