MINIFICPP-2863 - Introduce Reporting Tasks - #2231
Conversation
There was a problem hiding this comment.
Pull request overview
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Introduces first-class Reporting Tasks to MiNiFi C++ by adding API abstractions, class-loading support, and flow parsing so reporting tasks can be defined and executed from flow configuration (including a Site-to-Site provenance reporting task).
Changes:
- Added reporting task API types (context/descriptor/factory/metadata) and wired them into the class loader/resource registration system.
- Replaced legacy “Provenance Reporting” flow section with generalized “Reporting Tasks” parsing (YAML + NiFi Flow JSON).
- Updated Site-to-Site provenance reporting implementation and tests/behave framework to use the new reporting task model.
Reviewed changes
Copilot reviewed 44 out of 44 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| minifi-api/include/minifi-cpp/provenance/Provenance.h | API change to provenance send() signature used by reporting/task flow changes |
| minifi-api/include/minifi-cpp/core/reporting/ReportingTaskMetadata.h | New reporting task metadata struct for factory construction |
| minifi-api/include/minifi-cpp/core/reporting/ReportingTaskFactory.h | New factory interface for reporting tasks |
| minifi-api/include/minifi-cpp/core/reporting/ReportingTaskDescriptor.h | New descriptor interface to declare supported properties |
| minifi-api/include/minifi-cpp/core/reporting/ReportingTaskContext.h | New context interface to access properties/services/repos/state for reporting tasks |
| minifi-api/include/minifi-cpp/core/reporting/ReportingTaskApi.h | New reporting task lifecycle API |
| minifi-api/include/minifi-cpp/core/ProcessContext.h | Makes ProcessContext implement ReportingTaskContext (shared context surface) |
| minifi-api/include/minifi-cpp/core/ClassLoader.h | Adds class loader registration overload for reporting task factories |
| minifi-api/include/minifi-cpp/agent/agent_docs.h | Adds ReportingTask to resource type taxonomy |
| minifi-api/include/minifi-api.h | Extends C API surface with reporting task metadata/callbacks/context property getter |
| libminifi/test/unit/LogMetricsPublisherTests.cpp | Adjusts fixture resource initialization/teardown to match repo directory requirements |
| libminifi/test/unit/JsonFlowSerializerTests.cpp | Adds JSON parsing test coverage for reportingTasks |
| libminifi/test/libtest/unit/ReportingTaskUtils.h | Adds helpers to instantiate reporting tasks for unit tests |
| libminifi/src/sitetosite/SiteToSiteClient.cpp | Replaces ProcessContext& yielding with YieldAction adapter |
| libminifi/src/sitetosite/RawSiteToSiteClient.cpp | Updates payload transmission to use YieldAction |
| libminifi/src/sitetosite/HttpSiteToSiteClient.cpp | Updates interface implementation to new YieldAction signature |
| libminifi/src/provenance/Provenance.cpp | Changes provenance commit error handling + updates send() behavior |
| libminifi/src/core/reporting/SiteToSiteProvenanceReportingTask.cpp | Migrates provenance reporting task to new reporting task APIs |
| libminifi/src/core/reporting/ReportingTask.cpp | Implements processor wrapper that runs ReportingTaskApi instances |
| libminifi/src/core/flow/StructuredConfiguration.cpp | Parses and instantiates reporting tasks from flow config |
| libminifi/src/core/flow/FlowSchema.cpp | Adds schema keys for reporting tasks and removes legacy provenance reporting keys |
| libminifi/src/core/FlowConfiguration.cpp | Removes legacy provenance report task factory method |
| libminifi/src/core/ClassLoader.cpp | Adds reporting task factory wrapper and registration plumbing |
| libminifi/src/RemoteProcessGroupPort.cpp | Adds useProtocol() helper + decouples protocol pool sizing from ProcessContext |
| libminifi/include/sitetosite/SiteToSiteClient.h | Introduces YieldAction type-erased yield adapter + updates transmit API |
| libminifi/include/sitetosite/RawSiteToSiteClient.h | Updates transmit signature to YieldAction |
| libminifi/include/sitetosite/HttpSiteToSiteClient.h | Updates transmit signature to YieldAction |
| libminifi/include/provenance/Provenance.h | Updates ProvenanceReporterImpl::send() override signature |
| libminifi/include/core/reporting/SiteToSiteProvenanceReportingTask.h | Converts provenance reporting to ReportingTaskBase and embeds an RPG port |
| libminifi/include/core/reporting/ReportingTask.h | Adds processor-facing wrapper for reporting tasks |
| libminifi/include/core/flow/StructuredConfiguration.h | Exposes parseReportingTasks() and removes legacy provenance reporting parser |
| libminifi/include/core/flow/FlowSchema.h | Adds schema fields for reporting tasks |
| libminifi/include/core/FlowConfiguration.h | Removes legacy provenance report task creation API |
| libminifi/include/RemoteProcessGroupPort.h | Adds reporting-context onSchedule overload + useProtocol() |
| extensions/standard-processors/tests/unit/ProcessorTests.cpp | Updates tests to build/report on new reporting-task wrapper |
| extensions/standard-processors/tests/features/reporting-tasks.feature | Adds behave feature for executing reporting tasks end-to-end |
| extensions/rocksdb-repos/tests/ProvenanceTests.cpp | Updates tests to assert append success (new return-value expectations) |
| core-framework/include/core/reporting/ReportingTaskFactoryImpl.h | Provides templated factory implementation for reporting tasks |
| core-framework/include/core/reporting/ReportingTaskBase.h | Provides base class with descriptor/property registration helpers |
| core-framework/include/core/Resource.h | Adds reporting task resource registration to static resource mechanism |
| behave_framework/src/minifi_behave/steps/flow_building_steps.py | Adds steps to assemble flows containing reporting tasks |
| behave_framework/src/minifi_behave/minifi/reporting_task.py | Adds reporting-task model + YAML serialization |
| behave_framework/src/minifi_behave/minifi/minifi_flow_definition.py | Emits “Reporting Tasks” section in generated YAML |
| behave_framework/src/minifi_behave/minifi/flow_definition.py | Tracks reporting tasks in the in-memory flow definition |
Suppressed comments (6)
minifi-api/include/minifi-cpp/core/reporting/ReportingTaskContext.h:1
ReportingTaskContext.his not self-contained: it usesProperty,PropertyReference,core::controller::ControllerServiceHandle, andStateManagerwithout including the required headers or providing forward declarations (and the inlinegetProperty(...)overloads require full definitions). This can break consumers that include this header directly. Add the appropriate includes (e.g., property headers and ControllerServiceHandle/StateManager declarations) or restructure to avoid inline member definitions that depend on incomplete types.
minifi-api/include/minifi-cpp/core/reporting/ReportingTaskContext.h:1ReportingTaskContext.his not self-contained: it usesProperty,PropertyReference,core::controller::ControllerServiceHandle, andStateManagerwithout including the required headers or providing forward declarations (and the inlinegetProperty(...)overloads require full definitions). This can break consumers that include this header directly. Add the appropriate includes (e.g., property headers and ControllerServiceHandle/StateManager declarations) or restructure to avoid inline member definitions that depend on incomplete types.
minifi-api/include/minifi-cpp/core/reporting/ReportingTaskContext.h:1ReportingTaskContext.his not self-contained: it usesProperty,PropertyReference,core::controller::ControllerServiceHandle, andStateManagerwithout including the required headers or providing forward declarations (and the inlinegetProperty(...)overloads require full definitions). This can break consumers that include this header directly. Add the appropriate includes (e.g., property headers and ControllerServiceHandle/StateManager declarations) or restructure to avoid inline member definitions that depend on incomplete types.
minifi-api/include/minifi-cpp/provenance/Provenance.h:1- This changes the public
ProvenanceReporterAPI by removing theforceparameter fromsend(...), which is a breaking change for downstream implementations/extensions. If backward compatibility is required, consider keeping the old signature (possibly deprecated) and forwarding to the new one (or providing a defaulted parameter), or clearly version/bump the API boundary to avoid silently breaking external implementers.
minifi-api/include/minifi-api.h:1 - Two documentation issues in the new C API additions: (1)
minifi_reporting_task_metadata.loggercomment refers to 'controller service' lifetime instead of 'reporting task'; (2) thefull_nameexample string is missing its closing quote, making the comment misleading for API consumers. Update these comments to match reporting-task terminology and provide a syntactically correct example.
minifi-api/include/minifi-api.h:1 - Two documentation issues in the new C API additions: (1)
minifi_reporting_task_metadata.loggercomment refers to 'controller service' lifetime instead of 'reporting task'; (2) thefull_nameexample string is missing its closing quote, making the comment misleading for API consumers. Update these comments to match reporting-task terminology and provide a syntactically correct example.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if (auto append_status = repo_->appendEvents(events_); !append_status) { | ||
| throw minifi::Exception(REPOSITORY_EXCEPTION, append_status.error()); | ||
| } |
| explicit SiteToSiteProvenanceReportingTask(ReportingTaskMetadata metadata) | ||
| : ReportingTaskBase{metadata}, | ||
| remote_port_{metadata.name, metadata.uuid, std::make_unique<RemoteProcessGroupPort>(metadata.name, "", Configure::create(), | ||
| metadata.uuid, sitetosite::TransferDirection::SEND, metadata.logger)} | ||
| { |
| auto scheduling_strategy = getOptionalField(reporting_task_node, schema_.scheduling_strategy, DEFAULT_SCHEDULING_STRATEGY); | ||
| if (scheduling_strategy == "TIMER_DRIVEN") { | ||
| reporting_task->setSchedulingStrategy(core::TIMER_DRIVEN); | ||
| } else { | ||
| reporting_task->setSchedulingStrategy(core::CRON_DRIVEN); | ||
| } | ||
| auto scheduling_period_str = getOptionalField(reporting_task_node, schema_.scheduling_period, DEFAULT_SCHEDULING_PERIOD_STR); | ||
| if (scheduling_strategy == "TIMER_DRIVEN") { | ||
| if (auto scheduling_period = utils::timeutils::StringToDuration<std::chrono::nanoseconds>(scheduling_period_str)) { | ||
| reporting_task->setSchedulingPeriod(*scheduling_period); | ||
| } | ||
| } else { | ||
| reporting_task->setCronPeriod(scheduling_period_str); | ||
| } |
| if (Node properties_node = reporting_task_node[schema_.processor_properties]) { | ||
| parsePropertiesNode(properties_node, *reporting_task, name, nullptr); | ||
| } |
| Node parameterContextsNode = root_node[schema_.parameter_contexts]; | ||
| Node parameterProvidersNode = root_node[schema_.parameter_providers]; | ||
| Node provenanceReportNode = root_node[schema_.provenance_reporting]; | ||
| Node reportingTasksNode = root_node[schema_.reporting_tasks]; |
| std::unique_ptr<core::ProcessGroup> root = parseRootProcessGroup(root_node); | ||
| parseProvenanceReporting(provenanceReportNode, root.get()); | ||
|
|
||
| parseReportingTasks(reportingTasksNode, root.get()); |
af3f98a to
f1b2dcd
Compare
f47e740 to
42b773f
Compare
Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
In order to streamline the review of the contribution we ask you to ensure the following steps have been taken:
For all changes:
Is there a JIRA ticket associated with this PR? Is it referenced in the commit message?
Does your PR title start with MINIFICPP-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
Has your PR been rebased against the latest commit within the target branch (typically main)?
Is your initial contribution a single, squashed commit?
For code changes:
For documentation related changes:
Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible.