Proposal to Adopt Semantic Operator as a Kubeflow Subproject - #1024
Proposal to Adopt Semantic Operator as a Kubeflow Subproject#1024vara-bonthu wants to merge 4 commits into
Conversation
Signed-off-by: vara-bonthu <vara.bonthu@gmail.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
andreyvelich
left a comment
andreyvelich
left a comment
There was a problem hiding this comment.
This looks great, thank you for this @vara-bonthu! I left a few thoughts.
I am super excited to see this moving forward!
/assign @kubeflow/kubeflow-steering-committee
| - A deterministic planner that emits one governed SQL statement per request | ||
| - Compile-time row, column, and metric authorization | ||
| - MCP, REST, and governed SQL-view interfaces | ||
| - StarRocks and Trino dialects and query clients |
There was a problem hiding this comment.
Do you have plans to integrate all dialects that Apache Ossie supports?
https://github.com/apache/ossie/blob/main/core-spec/spec.md#dialects
There was a problem hiding this comment.
We plan to expand dialect support based on community demand, but we should not commit to implementing every dialect immediately.
There are two related layers here. Apache Ossie dialects describe expressions in a semantic model, while Semantic Operator also needs an executable query-engine integration consisting of SQL emission, connectivity, schema introspection, and conformance testing. Today, Semantic Operator supports StarRocks and Trino query engines and consumes ANSI_SQL expressions, with a StarRocks specific extension where needed. This architecture makes additional engines pluggable through separate dialect and database-client interfaces.
|
|
||
| The core architecture is vendor-neutral. Query engines implement a dialect and database-client interface. Catalog sources and metadata enrichers use separate extension interfaces. StarRocks and Trino are implemented today. The design does not require a specific cloud provider, model provider, catalog, object store, or Kubernetes distribution. | ||
|
|
||
| AWS Glue is one optional catalog source. DataHub and engine information schemas are also supported. Additional engines and catalogs can be added without changing the semantic model or serving interfaces. |
There was a problem hiding this comment.
|
|
||
| https://kubedai.github.io/semantic-operator (Private repo) | ||
|
|
||
| The documentation can move or redirect to a Kubeflow-managed location after acceptance. |
There was a problem hiding this comment.
Do you have plans to create dedicated website similar to: https://spark.kubeflow.org/en/latest/ ?
There was a problem hiding this comment.
Yes. The project already has a dedicated documentation website with detailed docs, currently published through GitHub Pages at kubedai.github.io/semantic-operator. Its build and deployment workflow is maintained in the repository.
After transfer, we plan to publish it from Kubeflow managed infrastructure under a Kubeflow domain, similar to the Spark Operator documentation, and preserve or redirect the existing URLs.
|
|
||
| Documentation: https://kubedai.github.io/semantic-operator (Private repo) | ||
|
|
||
| ### Security Profile |
There was a problem hiding this comment.
@franciscojavierarceo @thesuperzapper @chasecadet @juliusvonkohout Do we want to have section which identifies Kubeflow maturity requirements for the new project (e.g. Experimental, Incubating, Graduated)?
Or we want to say that all newly joined projects will be Experimental first?
https://github.com/kubeflow/community/blob/master/subprojects/maturity_requirements.md#graduated
There was a problem hiding this comment.
We would be comfortable entering as an Experimental or Incubating project
| - Define a security response and release process | ||
| - Establish baseline project and adoption metrics | ||
|
|
||
| #### Kubeflow integration |
There was a problem hiding this comment.
Any potential interest to integrate this into Kubeflow Community Distribution?
cc @juliusvonkohout @tarekabouzeid
There was a problem hiding this comment.
Yes, we are interested in eventual integration with the Kubeflow Community Distribution.
We propose doing this in stages - first complete the repository transfer and establish a supported release, then add manifests and integration tests, and finally work with the distribution maintainers on inclusion.
The initial donation remains independently installable through Helm and does not assume immediate distribution inclusion.
|
|
||
| Semantic Operator is different from an LLM text-to-SQL gateway. The LLM is not trusted to construct joins, aggregations, policies, or SQL. It can select from certified semantic concepts exposed through MCP. The server validates that request and creates deterministic SQL from a versioned model. | ||
|
|
||
| The proposal also does not ask Kubeflow to own Apache Ossie. Semantic Operator is an independent Kubernetes implementation that consumes the Apache Ossie specification and adds Kubernetes lifecycle, schema drift checking, governance, serving protocols, and query-engine integrations. |
There was a problem hiding this comment.
Shall we say that the goal is to also work closely with Apache Ossie community to reduce number of breaking changes and to promote this operator as a reference architecture of the spec?
There was a problem hiding this comment.
Yes, agreed. We want to work closely with the Apache Ossie community, track specification changes, contribute implementation feedback, and reduce avoidable compatibility breaks.
We would also like Semantic Operator to serve as a Kubernetes reference implementation of the specification, subject to Apache Ossie community alignment. We will maintain a versioned compatibility matrix and keep Kubernetes specific governance and lifecycle extensions outside the Ossie document.
|
|
||
| ## Proposed Working Group | ||
|
|
||
| The maintainers propose the Kubeflow Data Working Group as the initial home because Semantic Operator governs access to analytical data and complements Spark Operator. The Steering Committee and Working Group chairs may identify a different or joint home based on the project's agent, serving, and metadata integrations. |
There was a problem hiding this comment.
As we discussed offline, I think newly formed Agents WG should cover the scope of this project: #1025
This could be a great candidate to move forward with this WG.
There was a problem hiding this comment.
Agreed. The Agents Working Group is a strong primary home because MCP based governed data access for agents is central to the project. I will update this wording
Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> Signed-off-by: Vara Bonthu <vara.bonthu@gmail.com>
Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> Signed-off-by: Vara Bonthu <vara.bonthu@gmail.com>
Signed-off-by: vara-bonthu <vara.bonthu@gmail.com>
2 participants
Summary
This PR proposes adopting Semantic Operator as a Kubeflow subproject.
Semantic Operator is a Kubernetes operator and stateless semantic server that operationalizes an Apache Ossie semantic layer on existing data platforms.
It allows organizations to define certified metrics, dimensions, relationships, and governance policies once as Kubernetes resources. AI agents select these certified business concepts instead of generating SQL. Semantic Operator validates each request, applies governance, and deterministically generates one SQL statement for StarRocks or Trino.
Why this belongs in Kubeflow
Semantic Operator adds a governed data-access layer for AI agents, applications, BI tools, and Kubeflow workloads.
Potential integrations include:
Kubeflow would provide vendor-neutral governance and a community home for developing this capability.
What this PR adds
Proposal links
Proposed destination
kubeflow/semantic-operatorWe welcome feedback from the Kubeflow Steering Committee, Working Group leads, maintainers, users, and potential contributors.