16 v2 global embedding#49
Conversation
…predict. Added use_device to train
SarahAlidoost
left a comment
SarahAlidoost
left a comment
There was a problem hiding this comment.
@meiertgrootes thanks, the implementation looks good 👍 I left some comments. Most of them are related to code style. If something isnot clear, let me know.
Also, please consider running ruff as it can fixes things automatically and helps saving time in reviewing:
pip install ruff
ruff check --fix your_script.py # this fixes/shows errors
ruff format --check your_script.py --diff # this shows formatting issues
| "land_mask_patch": land_tensor, # (pH,pW) True=Land | ||
| "daily_timef_patch": daily_timef_tensor, #(M, T=31, 2) | ||
| "padded_days_mask": self.padded_days_tensor, # (M, T=31) True=padded | ||
| #"sh_geo_pos_patch": geo_pos_tensor, # (pH, pW, sh_embed_dim) |
There was a problem hiding this comment.
| #"sh_geo_pos_patch": geo_pos_tensor, # (pH, pW, sh_embed_dim) |
SarahAlidoost
left a comment
SarahAlidoost
left a comment
There was a problem hiding this comment.
@meiertgrootes thanks, the implementation looks good 👍 I left some comments. Most of them are related to code style. If something isnot clear, let me know.
Also, please consider running ruff as it can fixes things automatically and helps saving time in reviewing:
pip install ruff
ruff check --fix your_script.py # this fixes/shows errors
ruff format --check your_script.py --diff # this shows formatting issues
rogerkuou
left a comment
rogerkuou
left a comment
There was a problem hiding this comment.
Hi @meiertgrootes , besides Sarah's comments, I only have two very minor comments from my side. So I will already approve this PR. Please go ahead and merge after adapting Sarah's comment.
SarahAlidoost
left a comment
SarahAlidoost
left a comment
There was a problem hiding this comment.
@meiertgrootes thanks for addressing the comments 👍 . Let's merge it.
3 participants
This pull request replaces #48 which was corrupted by mal-configured local github setup
This pull request adds fully sphere aware geo position and scale encoding for patches.
Geo position is encoded using real-valued spherical harmonics as basis.
To create embeddings real-valued spherical harmonics at the lat/lon positions of pixels of the input data (i.e. native resolution) are calculated up to a user-defined order L. This results in an (L+1)^2 dimensional embedding vector. L ~ 10 should be fine.
Subsequently a sphere aware area-weighted PCA is performed on the native resolution SH embdding grid, with the requirment that the target diemension for the PCA (the sh_embed_dim) be smaller than (L+1)^2. The ranked PCA components up to sh_embed_dim are retained to be used as basis functions. The SH embedding vectors are then reprojected to this basis and scaled to zero mean and unit variance with tanh based soft-clipping at ~3 sigma to suppress pathological outliers in high order harmonics.
For each patch, a patch position embedding is then constructed as the area weighted mean of the token/pixel embeddings in the patch.
In addition for each patch a scale embedding is constructed consisting of: the patch physical extent in lat and lon directions, the patch area, the anisotropy of the patch extent, the pixel scale in phyical units [m] in lat/lon, anisotropy and isotropized linear scale, as well as finally effective harmonic order cutoff in lat/lon.
Both embeddings are precalculated once for all patches
The 10-dimensional scale embedding is concatenated with the geo position embedding for each patch. During training an trainable MLP is used to project the concatented embeddings into the desired embedding diemension for additive incorporation.
Note, it makes sense to choose the hidden diemension for the projection larger than bith the dimension of the concatenated embeddings, as well as the target embedding dimension.