![]() Edit it to change scheduling to '5 secs'. ![]() ![]() Now add the reporting task to send the provenance events.Then on the root of the flow start all components. Enable the internal schema registry and all services. Select 'configure' on 'Provenance R/W' group.Enable the new provenance repository implementation by changing the property '' line to '=.WriteAheadProvenanceRepository'.Enable site to site in nifi.properties by setting the '.port' property to 8085 for example.This creates tremendous reuse, flexibility, and massive performance benefits all while still maintaining full fidelity of the provenance trail! Through the record abstraction it understands how to parse, demarcate, and reproduce the transformed output without this having to be a concern of the user or the processors themselves. Another great benefit here is we don't have to split data into individual records before the processors can operate on them. All this really helps highlight the power this makes available as we get to reuse the same components but plugin in separate concerns such as formats and schemas which we can reuse at various points in the flow. To do this we take advantage of the site to site reporting task for provenance, the new QueryRecord processor powered by Apache Calcite, various record readers and writers including a custom one built on the fly using Groovy all to read in the provenance stream while simultaneously writing it out in JSON, Avro, CSV, and XML. The provenance data it generates then becomes its own stream to experiment with. This example flow takes advantage of NiFi's ability to stream its own provenance data through the flow which it can then read, write, route, and transform for some interesting cases. Provenance Stream Record ReaderWriter XML AVRO JSON CSV LookupRecord (and its CSVRecordSetWriter) are configured to append this field (called "location") to the incoming records. The ScriptedLookupService contains a Groovy script which provides the lat/long values (and API Key) to the web service, and returns a String result corresponding to the geolocation. The ScriptedLookupService itself accepts a dynamic property 'googleApiKey' which is the user's Google API Key for performing lookups using the Google Reverse Lookup web API. The latitude and longitude values are in the incoming record, and dynamic properties in LookupRecord are provided to "find" the lat/long fields and pass them to the ScriptedLookupService. This example flow illustrates the use of a ScriptedLookupService in order to perform a latitude/longitude lookup to determine geographical location. NOTE: This template depends on features available in the next release of Apache NiFi (presumably 1.3.0) which is not released as of this writing. ReverseGeoLookup_ScriptedLookupService.xml Here, we have a collection of useful templates for learning about how to build DataFlows with the existing Processors. Please feel free to add any useful templates below. Templates received from others can then be imported into an instance of NiFi and dragged onto the canvas.įor more information on Templates, including how to import, export, and work with them, please see the Template Section of the User Guide. Once a DataFlow has been created, parts of it can be formed into a Template. This Template can then be dragged onto the canvas, or can be exported as an XML file and shared with others. A Template is a way of combining these basic building blocks into larger building blocks. At times, though, using these small building blocks can become tedious if the same logic needs to be repeated several times. To solve this issue, NiFi provides the concept of a Template. This is achieved by using the basic components: Processor, Funnel, Input/Output Port, Process Group, and Remote Process Group. These can be thought of as the most basic building blocks for constructing a DataFlow. Apache NiFi provides users the ability to build very large and complex DataFlows using NiFi.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |