Add java - sync emitter pipeline to regenerate TypeSpec SDKs#49762
Draft
XiaofeiCao wants to merge 18 commits into
Draft
Add java - sync emitter pipeline to regenerate TypeSpec SDKs#49762XiaofeiCao wants to merge 18 commits into
XiaofeiCao wants to merge 18 commits into
Conversation
Ports the post-publish SDK generation automation from Azure/autorest.java to this repo, adapted for the typespec-java emitter's new home in Azure/typespec-azure (packages/typespec-java). - eng/pipelines/post-publish-sdk.yaml: regenerates all TypeSpec-based SDKs and opens an automated draft PR. Uses the published @azure-tools/typespec-java npm package by default; when the TypeSpecJavaPRId parameter is set, builds the emitter dev package from that typespec-azure PR (turbo builds workspace deps, then Build-TypeSpec.ps1 builds and packs). - eng/pipelines/scripts/sync_sdk.py: regeneration script. - eng/pipelines/scripts/patches/: SDK patches applied after generation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace the single PR-id parameter with a DevPackage boolean (default false) plus a PRId string that defaults to the sentinel 'none' so the run panel never requires input for the default (published-package) path. PRId is validated at runtime only when DevPackage is true. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
DevPackage=false uses the published npm package. DevPackage=true builds the emitter from source: from the typespec-azure PR in PRId, or from main when PRId is left at the 'none' sentinel. Drops the previous PRId-required error. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The dev build failed because the 'core' submodule (microsoft/typespec) was not checked out: tspd and the http-client-java generator required by Build-Generator.ps1 live there. Initialize submodules after settling on the ref, only when DevPackage=true. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
tsp-client generate-lock-file failed with 'Unsupported URL Type catalog:' because New-EmitterPackageJson.ps1 was fed the typespec-azure monorepo source package.json, whose deps use the catalog:/workspace: protocols. Extract the packed tarball's package.json instead (pnpm pack resolves those protocols to concrete versions) and use it as the basis for emitter-package.json. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The default (DevPackage=false) route now pins eng/emitter-package.json to the published emitter given by TypeSpecJavaVersion, then regenerates - matching the post-publish-sdk intent. sync_sdk.py gains --emitter-version: it npm-packs the published package, seeds emitter-package.json from the tarball's resolved package.json, and regenerates the lock. DevPackage=true still builds from source (PRId or main). The typespec-azure clone/build steps are gated to the dev route, and the default route fails fast if no version is given. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The presence of a published version fully determines the route, so DevPackage is redundant. TypeSpecJavaVersion set = published route; empty = build emitter from source (PRId, or main). sync_sdk.py drops --dev-package and infers the route from --emitter-version. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Azure DevOps string parameters cannot be left truly empty in the run UI, so default TypeSpecJavaVersion to the 'none' sentinel (matching PRId). Pipeline conditions compare against 'none'; sync_sdk.py normalizes 'none' to empty and treats it as the dev (build-from-source) route. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Once the published tarball is downloaded via npm pack, its package.json has resolved dependency versions, so tsp-client generate-config-files can consume it directly to produce both emitter-package.json and its lock file - replacing the separate New-EmitterPackageJson.ps1 + generate-lock-file calls. The dev route still uses those two steps to inject the local .tgz path between them. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
pnpm pack resolves workspace:^ dependencies to caret ranges (e.g. ^0.69.1) in the packed package.json. New-EmitterPackageJson.ps1 pins peer deps from those devDependencies values, so carets leaked into emitter-package.json (^0.69.0 instead of the exact 0.69.2 that autorest.java produced). Strip the leading caret/tilde from the extracted package.json so peer deps are pinned to exact versions. Simple ranges like '>=x <y', 'workspace:', 'catalog:' are left as-is. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This reverts commit fc1b881.
Regenerating emitter-package.json from the emitter tarball pinned the peer/dev dependency versions from that tarball, which could be stale (e.g. an older typespec-client-generator-core than the one being released). Instead, only overwrite the @azure-tools/typespec-java dependency (published version or dev .tgz path) in the existing emitter-package.json and regenerate the lock file. Other dependency versions are managed elsewhere and updated before the pipeline runs. Removes the now-unused tarball extraction / config-file generation helpers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ports the post-publish SDK-regeneration automation from
Azure/autorest.javainto this repo, adapted for thetypespec-javaemitter's new home inAzure/typespec-azure(packages/typespec-java). This drives thejava - sync emitterpipeline (ADO definition 8274).What it does
Regenerates all TypeSpec-based SDKs with the
typespec-javaemitter and opens an automated draft PR with the results. Two routes, selected by theEmitter Versionparameter:Emitter Version= a version, e.g.0.45.4): pinseng/emitter-package.jsonto that published@azure-tools/typespec-javarelease (vianpm pack-> resolvedpackage.json), regenerates the lock file, and regenerates SDKs. This is the post-publish path and does not touch typespec-azure sources.Emitter Version=none): builds the emitter dev package from source inAzure/typespec-azure— from thePRIdPR, or frommainwhenPRIdisnone— then regenerates.Files
eng/pipelines/post-publish-emitter.yaml— the pipeline.eng/pipelines/scripts/sync_emitter.py— the regeneration script.eng/pipelines/scripts/patches/*.patch— SDK patches applied after generation.Notes
Azure/typespec-azureis public, so it isgit cloned directly (no GitHub service connection / repo resource needed). Submodules are initialized for the dev build (coreprovidestspd+ the http-client-java generator).npm pack, so a run can never pin an unpublished emitter version.sync_emitter.pyparses/--helpworks, all referenced templates resolve in-repo.How to verify
Run the
java - sync emitterpipeline (ADO def 8274) on this branch:Emitter Versionto a released version (e.g.0.45.4).Emitter Version=none, optionally setPRId.