Repository navigation
V3 naksha 48 custom indices - #674
Conversation
- customIndexMapping model (lib-core models) and CustomIndexMappingCompiler that validates mappings and expands them into cmNN members / ciNN indices - CollectionIndexPolicy.toNativeCollection: shared preparation for lazy creation, collection writes and extensions; unmapped definitions unchanged - DefaultStorageHandler: mapped lazy creation, Handler collection precedence for mapped collection writes, property search translated to member queries - Mapped collections are created (or verified) when a Space or Handler is saved; unmapped collections stay lazy - Spaces must not define mappings when a Handler defines the collection - PgCollection.verifyNewHeadState rejects schema changes of existing collections with CONFLICT - QueryParameterDecoder: fix parsing of unquoted false Signed-off-by: Marcin-Here <ext-marcin.grabowski@here.com>
- Create or verify a Handler's mapped collection after authorization and validation, for every DefaultStorageHandler write (incl. UPSERT and batches); existing collections are verified when the mapping is removed or creation is disabled - Check hidden Space mappings before the Space collection is written - Only the built-in DefaultStorageHandler defines a collection for provisioning and validation - verifyNewHeadState compares member JSON paths, not only columns - Reject mappings overlapping any existing member path - Reject mapped Space writes that are not a single create or update Signed-off-by: Marcin-Here <ext-marcin.grabowski@here.com>
…ped paths - Space create and update check write access before the collection is written; AuthorizationEventHandler exposes the existing check - verifyNewHeadState compares member paths as typed segments - Document that changing hubSlimIndices requires a migration Signed-off-by: Marcin-Here <ext-marcin.grabowski@here.com>
…ed by the Hub) Signed-off-by: Marcin-Here <ext-marcin.grabowski@here.com>
…ings Handler collections are already created or verified when the Handler is saved. Signed-off-by: Marcin-Here <ext-marcin.grabowski@here.com>
Signed-off-by: Marcin-Here <ext-marcin.grabowski@here.com>
Keep the existing v3 handler existence check unchanged. This reverts commit b30d485. Signed-off-by: Marcin-Here <ext-marcin.grabowski@here.com>
… deprecated property query Signed-off-by: Marcin-Here <ext-marcin.grabowski@here.com>
…Util.box Signed-off-by: Marcin-Here <ext-marcin.grabowski@here.com>
…le field Signed-off-by: Marcin-Here <ext-marcin.grabowski@here.com>
…asses Handler saves create or verify the mapped collections returned by DefaultStorageHandler.configuredCollections(). Subclasses with other collection properties (e.g. HeteroStorageHandler) override it. Handlers are created with a placeholder Space, because a Handler save has no Space. Signed-off-by: Marcin-Here <ext-marcin.grabowski@here.com>
This comment was marked as off-topic.
This comment was marked as off-topic.
fe3d5d5 to
8c977f2
Compare
This comment was marked as off-topic.
This comment was marked as off-topic.
Signed-off-by: Marcin-Here <ext-marcin.grabowski@here.com>
This comment was marked as off-topic.
This comment was marked as off-topic.
Signed-off-by: Marcin-Here <ext-marcin.grabowski@here.com>
This comment was marked as off-topic.
This comment was marked as off-topic.
Signed-off-by: Marcin-Here <ext-marcin.grabowski@here.com>
This comment was marked as off-topic.
This comment was marked as off-topic.
| return executeDeleteSpace(writeRequest); | ||
| } else if (isUpdateSpaceRequest(writeRequest, spaceId)) { | ||
| return executeUpdateSpace(writeRequest); | ||
| } else if (isCreateSpaceRequest(writeRequest, spaceId)) { |
There was a problem hiding this comment.
We do not create collection upfront, when a space is created. It is possible that the one who creates a Space does not have Collection creation permission.
The Collection creation happens in a reactive manner when collection is not found while performing the Read/Write Features request.
I would suggest we retain the same approach, unless I miss something here.
| final Space space = ((Space) updateSpaceEntryReq.getWrites().get(0).getFeature()); | ||
| final SpaceProperties spaceProperties = space.getProperties(); | ||
| // The collection is written before the Space, so the Space write must be authorized first. | ||
| final ErrorResponse forbidden = new AuthorizationEventHandler(nakshaHub).checkWriteAccess(); |
There was a problem hiding this comment.
This approach is not able to check whether it has write request against a specific SpaceID or specific CollectionId or not. It only checks whether a broader "write" permission is allowed or not. And this will already happen as part of executeSingleCollectionWrite -> executeWriteToCustomSpaces.
So, I see this Auth check redundant. Let me know, if I miss something.
| } | ||
|
|
||
| private static @Nullable CustomIndexMappingList mappingList(NakshaCollection definition) { | ||
| Object raw = definition.getRaw("customIndexMapping"); |
There was a problem hiding this comment.
To get benefit of type-safety, it is better to define "customIndexMapping" data-type as part of NakshaCollection
There was a problem hiding this comment.
We only need to validate during Update EventHandler, that customIndexMapping has not changed in new EventHandler Properties
There was a problem hiding this comment.
We only need to validate during Update Space, that customIndexMapping has not changed in new Space Properties
| MemberType type = member.getDataType(); | ||
| AnyOp op = query.getOp(); | ||
| // Boolean REST predicates encode their value in the operation, not query.value. | ||
| if (MemberType.BOOLEAN.equals(type)) { |
There was a problem hiding this comment.
can we use some mechanism here, which should give me a warning or error in future, if some newly introduced MemberType is forgotten to be covered here?
| path: JsonPath = JsonPath("properties", "score"), | ||
| indexName: String = "score_idx" | ||
| ) = NakshaCollection(collectionId, catalog.id).apply { | ||
| members = MemberList(Member("score", type, path)) |
There was a problem hiding this comment.
Function name is rather "inhuman" ;) And moving member name to input param with default would make it more consistent with the other inputs.
xeus2001
left a comment
There was a problem hiding this comment.
From lib-psql side no concerns.
Signed-off-by: Marcin-Here <ext-marcin.grabowski@here.com>
Signed-off-by: Marcin-Here <ext-marcin.grabowski@here.com>
Signed-off-by: Marcin-Here <ext-marcin.grabowski@here.com>
Code Coverage
|
Signed-off-by: Marcin-Here <ext-marcin.grabowski@here.com>
Signed-off-by: Marcin-Here <ext-marcin.grabowski@here.com>
Code Coverage
|
| collectionForCreation.setId(operationData.getCollectionId()); | ||
| collectionForCreation.setCatalogId(operationData.getMapId()); | ||
| CollectionIndexPolicy.normalizeForHubCreation(collectionForCreation); | ||
| final NakshaCollection collectionForCreation = CollectionIndexPolicy.toNativeCollection( |
There was a problem hiding this comment.
What if we set members and indices as part of NakshaCollection instantiation itself?
That way, we will ALWAYS have members and indices available as part of NakshaCollection, irrespective of whether it is fetched from Storage or accepted in the input request.
And so, we don't have to call toNativeCollection from anywhere. Also, Extensions where NakshaCollection is referred, that should also automatically get benefit of having members and indices already available, without them having to explicitly invoke toNativeCollection (which most likely be missed).
Do you think of a specific challenge to talk about? OR I miss something?
There was a problem hiding this comment.
why not in NakshaCollections
NakshaCollection is Proxy over PlatformObject
So it's instantiation goes through:
Configs arrive as plain maps. A NakshaCollection only exists once someone asks for it:
'space.getProperties().getCollection()
-> JvmBoxingUtil.box(rawMap, NakshaCollection.class)
-> Platform.proxy(rawMap, NakshaCollection::class)
proxy = NakshaCollection(); // no-arg constructor, no data yet
proxy.bind(rawMap, symbol); // customIndexMapping becomes visible only here'
The constructor runs before any data is attached, so it can't expand anything. The proxy is also cached on the raw map, so every reader shares the same instance and the same map.
Other approaches could be exapanding through bind() or lazy getter:
Problem: It writes into the config. Expanding on first access stores cmNN/ciNN into the raw map of the Space/Handler.
So for example the admin validator reads the collection before the Space is saved, so the generated members would be persisted with the config and returned by GET /hub/spaces/{id}.
So instead of
'{ "id": "roads", "properties": { "collection": {
"id": "roads_data",
"customIndexMapping": [ { "jsonPath": ["properties","score"], "dataType": "int32" } ] } } }'
We would save: '
"collection": {
"id": "roads_data",
"customIndexMapping": [ { "jsonPath": ["properties","score"], "dataType": "int32" } ],
"members": [
...XYZ default members (useMembers() seeds XyzMembers.ALL)...,
{ "name": "cm01", "dataType": "int32", "path": ["properties","score"] }
],
"indices": [
{ "name": "tags", ... }, { "name": "geo", ... }, { "name": "fn_nv", ... },
{ "name": "ci01", "on": ["cm01"] }
]
}
Next problem that indices being null on NakshaCollection creation means "use the defaults" ('XyzIndices.all) in 'PgCollection'
also that is how we deciding to add hubSlim indices.
|
|
||
| public static @Nullable ReadFeatures adapt( | ||
| ReadFeatures request, NakshaCollection schema, NakshaCollection configured) { | ||
| // Member queries describe single HEAD states; history reads (e.g. the activity log) stay on the legacy path. |
There was a problem hiding this comment.
I would like to understand what could be the logical reason to skip custom index for historical queries?
once agreed, then we should also put reason as a comment for future reference
There was a problem hiding this comment.
- The reason was to preserve legacy behavior.
Example: we would havev1- score:51, v2 score: 10, v3(head) score: 10
Query:p.score=51 setVersions(2)
In legacy behavior with PropertyFilter
we would UNION HISTORY and HEAD and return 2 latest versions (v2,v3) and after that we would filter in memory
and in this case we would get empty result.
In member query the condition is part on WHERE in both tables, so we would get v1.
But if we agree that property queries on custom members should follow native member-query semantics,
we can remove the history and version-count guards. The request.getQueryMembers() != null guard stays, because an existing member query needs no translation.
| } | ||
| final NakshaCollection schema = CollectionIndexPolicy.toNativeCollection( | ||
| configured, operationData.getCollectionId(), operationData.getMapId()); | ||
| final ReadFeatures translated = PropertyMemberQueryTranslator.adapt(readFeatures, schema, configured); |
There was a problem hiding this comment.
It is not clear why we need to pass two objects schema and configured. Pls relook into simplifying this by passing only one, as in reality we will have only one collection version to work against.
There was a problem hiding this comment.
Agreed simplified to one.
| JsonPath path = query.getProperty().getPath(); | ||
| CustomIndexMapping mapping = mappingFor(path, configured); | ||
| if (mapping == null) return null; | ||
| Member member = CustomIndexMappingCompiler.memberForPath(schema, mapping.getJsonPath()); |
There was a problem hiding this comment.
It is confusing, why we are searching for mapping here using mappingFor function. Isn't querying for member directly using memberForPath sufficient? Or we will miss some situation?
There was a problem hiding this comment.
It was needed when we had 2 collection objects, after our simplification is no longer needed.
| if (member == null) return null; | ||
| MemberType type = member.getDataType(); | ||
| if (!TRANSLATED_TYPES.contains(type) && !LEGACY_ONLY_TYPES.contains(type)) { | ||
| logger.warn("Member type {} of {} is not supported by PropertyMemberQueryTranslator, using the legacy path", type, member.getName()); |
There was a problem hiding this comment.
also put a comment, what will be possibly impact of this warning.
There was a problem hiding this comment.
We can continue with this warning. But impact is higher, because consumer may get negative impression about member support due to possibly slower responses.
It is important to validate during Space / Handler Create/Update operation, to validate that only supported member datatypes are in use.
| if (collection == null || collection.getCustomIndexMapping() == null) { | ||
| return SUCCESSFUL_VALIDATION; | ||
| } | ||
| ReadFeatures getEventHandlersRequest = |
There was a problem hiding this comment.
We agreed not to tie space validation logic with handler and storage instantiation for now, to avoid config v/s runtime dependency. I would like to understand reasoning if we choose to continue with it.
| * @throws NakshaException with {@link NakshaError#CONFLICT}, if the existing collection does not match. | ||
| */ | ||
| public static void requireMatching(NakshaCollection stored, NakshaCollection definition) { | ||
| final NakshaCollection expected = expand(definition.copy(true)); |
There was a problem hiding this comment.
As I put comment somewhere else, try to see, if we can avoid creating copy of collection everytime, as I think all the required information should already be available in the given collection definition itself.
| final NakshaCollection expected = expand(definition.copy(true)); | ||
| for (CustomIndexMapping mapping : validatedMappings(definition)) { | ||
| final Member member = memberForPath(expected, mapping.getJsonPath()); | ||
| final boolean indexed = MemberType.BOOLEAN.equals(member.getDataType()) || isIndexed(stored, member.getName()); |
There was a problem hiding this comment.
I believe we can simplify this by only comparing the member definitions, without having to check presence of index. This is again to differentiate between config change v/s runtime behaviour change. If we miss something here, I would like to understand.
There was a problem hiding this comment.
No longer needed as we validate customIndexMapping not Collection itself.
Signed-off-by: Marcin-Here <ext-marcin.grabowski@here.com>
Signed-off-by: Marcin-Here <ext-marcin.grabowski@here.com>
Signed-off-by: Marcin-Here <ext-marcin.grabowski@here.com>
Signed-off-by: Marcin-Here <ext-marcin.grabowski@here.com>
Signed-off-by: Marcin-Here <ext-marcin.grabowski@here.com>
Code Coverage
|
| * Translates a property search on custom index mappings into member queries, so that the storage can use the | ||
| * generated indices. Returns the original request, when the collection has no mappings or the query can't be translated. | ||
| */ | ||
| private @NotNull ReadFeatures translatePropertyQuery(final @NotNull OperationData operationData) { |
There was a problem hiding this comment.
Ensure this translate function is generic enough so that it can be moved to lib-psql level at one point, so that we don't have to do this for every handler directly using lib-psql storage.
So, may be, it should take NakshaCollection and ReadRequest as input and it should return translated ReadRequest as outcome.
This will already benefit reusability in HeterogeneousStorageHandler
| collectionFromRequest.setId(collectionId); | ||
| collectionFromRequest.setCatalogId(mapId); | ||
| CollectionIndexPolicy.normalizeForHubCreation(collectionFromRequest); | ||
| write.setFeature(CollectionIndexPolicy.toNativeCollection(collectionFromRequest, collectionId, mapId)); |
There was a problem hiding this comment.
If we ensure members and indices are always populated whenever NakshaCollection object is created (either during JSON parsing OR while reading from Storage), then we will be able to avoid writing this custom wrapper logic in all the handlers directly using lib-psql storage.
There was a problem hiding this comment.
Custom members and indices creation moved to NakshaCollection
| public static @Nullable ReadFeatures adapt(ReadFeatures request, NakshaCollection configured) { | ||
| if (request.getQueryMembers() != null) return null; | ||
| // History windows (versions > 1) are applied after WHERE, translating would select different versions. | ||
| if (request.getQueryHistory() || request.getVersions() != 1) return null; |
There was a problem hiding this comment.
Recheck and allow request on history to be translated from prop-search to member-based query
| IPropertyQuery expression = request.getQuery().getProperties(); | ||
| if (expression == null) return null; | ||
| // All or nothing: one untranslatable predicate keeps the whole request on the legacy path. | ||
| Op propertyOp = translate(expression, CustomIndexMappingCompiler.generatedMembers(configured)); |
There was a problem hiding this comment.
At the end, we may not need to call generatedMembers here and simply pass the input collection as is
| if (query instanceof PAnd) return translateChildren((PAnd) query, schema, configured, true); | ||
| if (query instanceof POr) { | ||
| // REST sends "cs={...}" as the object OR the object wrapped into an array, for a tag-map only the object fits. | ||
| PQuery object = objectContainment((POr) query); |
There was a problem hiding this comment.
- Clarified.
- Open for rechecking.
| if (member == null) return null; | ||
| MemberType type = member.getDataType(); | ||
| if (!TRANSLATED_TYPES.contains(type) && !LEGACY_ONLY_TYPES.contains(type)) { | ||
| logger.warn("Member type {} of {} is not supported by PropertyMemberQueryTranslator, using the legacy path", type, member.getName()); |
There was a problem hiding this comment.
We can continue with this warning. But impact is higher, because consumer may get negative impression about member support due to possibly slower responses.
It is important to validate during Space / Handler Create/Update operation, to validate that only supported member datatypes are in use.
| * definition is normalized in place, as before. With mappings, a copy is returned that holds the generated members | ||
| * and indices, but not the mapping itself, so the configuration stays unchanged. | ||
| */ | ||
| public static @NotNull NakshaCollection toNativeCollection( |
There was a problem hiding this comment.
Remove, if this function remains unused at the end
…from NakshaCollection.kt Signed-off-by: Marcin-Here <ext-marcin.grabowski@here.com>
Code Coverage
|
| for (i in 0 until mappings.size) { | ||
| val field = "customIndexMapping[$i]" | ||
| val mapping = mappings[i] ?: throw invalid(field, "must not be null") | ||
| for (key in mapping.keys) { |
There was a problem hiding this comment.
Can we safely remove this check without any logical impact? Ideally, having additional field in an object should not cause rejection. It can be ignored instead (as unknown property).
Unless you think of some other reason, that suggests we should continue with it?
| if (indices.none { it?.name == indexName }) indices.add(Index(indexName, member.name)) | ||
| } | ||
| appliedMappings = rawMappings | ||
| appliedMembers = rawMembers()?.platformObject() |
There was a problem hiding this comment.
If we are only interested in knowing whether mapping has been already applied or not, don't we just need one single flag? I don't see benefit of having appliedMembers and appliedIndices as well. Do I miss something?
There was a problem hiding this comment.
moved and use same solution as was before in useMembers
| val generated = CustomIndexMappings.generatedMembers(this) | ||
| var members = rawMembers() | ||
| if (members == null) { | ||
| members = MemberList(XyzMembers.ALL) |
There was a problem hiding this comment.
As the standard list of members is fixed and static, I see it as an unwanted memory overhead if we add standard members in the list of members of each NakshaCollection.
Or perhaps there is a reason, that I miss?
There was a problem hiding this comment.
moved and use same solution as was before in useMembers
| if (expression == null) return null; | ||
| // All or nothing: one untranslatable predicate keeps the whole request on the legacy path. | ||
| Op propertyOp = translate(expression, CustomIndexMappingCompiler.generatedMembers(configured)); | ||
| Op propertyOp = translate(expression, collection.useMembers()); |
There was a problem hiding this comment.
It is confusing for me to understand purpose and differentiation between useMembers against getMembers.
Is useMembers supposed to return only the list of Standard members OR a combined list of Standard + Custom members?
Somehow, I think there is duplicate code between the two:
- If it is supposed to return the combined list, then
getMembersis already doing the job. In that caseuseMembersperhaps become obsolete? - If it is supposed to return only the standard member, then I think that is already available as
XyzMembers.ALL. So, stilluseMembersseem redundant to me.
Need to discuss this part further, depending on how you see?
There was a problem hiding this comment.
Changed as discussed.
…embers. Signed-off-by: Marcin-Here <ext-marcin.grabowski@here.com>
Code Coverage
|
Signed-off-by: Marcin-Here <ext-marcin.grabowski@here.com>
Code Coverage
|
No description provided.