diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..10212bc --- /dev/null +++ b/.gitattributes @@ -0,0 +1,2 @@ +# Shell scripts must retain LF endings on every platform. +*.sh text eol=lf diff --git a/.gitignore b/.gitignore index 044af7d..1e34738 100644 --- a/.gitignore +++ b/.gitignore @@ -38,4 +38,8 @@ hs_err_pid* *.iml #OSX -.DS_Store \ No newline at end of file +.DS_Store + +# The Maven credential provider is an opt-in for ingesting uncached packages. CI authenticates +# with MavenAuthenticate@0, and committing the extension would break anonymous restores. +.mvn/ diff --git a/README.md b/README.md index 4037c6c..f2168fd 100644 --- a/README.md +++ b/README.md @@ -17,6 +17,113 @@ This repo contains library for building Azure Java Functions. Visit the [complet ## Prerequisites * Java 8 +* [Apache Maven](https://maven.apache.org/) 3.0 or later + +## Package feed + +All Maven packages and plugins are restored from the `upstream-public` Azure Artifacts feed +(`https://pkgs.dev.azure.com/azfunc/public/_packaging/upstream-public/maven/v1`), which is configured +as the `central` repository in every `pom.xml` in this repository. + +The repository root also has a [`settings.xml`](settings.xml) that mirrors `central` to the same +feed. It exists because a `pom.xml` cannot cover everything: + +- Maven resolves build extensions and plugin prefixes *before* a pom's `` are honored, + so those requests would otherwise go straight to Maven Central. +- `MavenAuthenticate@0` and the credential provider key credentials off the Azure Artifacts *feed + name* (`upstream-public`), while the pom repository id must be `central` in order to override the + id Maven inherits from the Super POM. The mirror id bridges the two. + +CI installs this file to `~/.m2/settings.xml`. Locally you only need it when pulling a package or +version the feed has not cached yet, in which case pass it explicitly with `mvn -s settings.xml`. + +### Anonymous restore (default) + +The feed allows anonymous reads, so no credentials are required to build once a package version has +been saved to the feed. External contributors and fresh clones need no setup. `mvn` just works. +Never commit credentials or a `` entry to `settings.xml` in this repository because doing so +would force authentication on everyone. + +### Authenticating (Microsoft developers only) + +Authentication is only needed to *ingest* a package version that the feed has not cached yet. The +first restore of any new or upgraded dependency will fail anonymously with: + +> No local versions of package '...'; please provide authentication to access versions from upstream +> that have not yet been saved to your feed. + +When that happens, a Microsoft developer with access to the `azfunc/public` project must run the +restore once with credentials, which pulls the version from upstream and saves it to the feed. Every +subsequent anonymous restore then succeeds. + +The recommended way to authenticate is the `artifacts-maven-credprovider`, which acquires a token via +Entra ID so you do not have to manage a PAT. + +Run the helper script for your shell from the root of your clone. It installs the credential provider +into your local Maven repository if it is missing, then writes `.mvn/extensions.xml`. Both scripts +are idempotent, so re-running them is safe: + +```powershell +./eng/scripts/Install-MavenCredentialProvider.ps1 +``` + +```bash +./eng/scripts/install-maven-credprovider.sh +``` + +Pass `-Version` / `--version` to install a different release, and `-Force` / `--force` to reinstall or +to overwrite an `.mvn/extensions.xml` the script does not manage. + +If you would rather do it by hand, the equivalent steps are: + +1. Bootstrap the credential provider once per machine. Run this from a directory outside any Maven + project, such as your home directory. It downloads the extension from the public `AzureArtifacts` + tools feed, which needs no authentication: + + ```powershell + mvn dependency:get "-Dartifact=com.microsoft.azure:artifacts-maven-credprovider:3.2.1" "-DremoteRepositories=central::::https://pkgs.dev.azure.com/artifacts-public/PublicTools/_packaging/AzureArtifacts/maven/v1" + ``` + + Using the repository id `central` matters. Maven records the extension as having come from + `central`, which is the same id this repository's `pom.xml` files declare, so the cached copy + validates during later builds. + +2. Create `.mvn/extensions.xml` at the root of your clone: + + ```xml + + + com.microsoft.azure + artifacts-maven-credprovider + 3.2.1 + + + ``` + +`.mvn/` is deliberately listed in `.gitignore`. Do not commit it. The extension exits when it +detects a build context, and committing it would break anonymous restores for everyone else. + +If you would rather not use the credential provider, you can instead add a `` entry to your +user-level `~/.m2/settings.xml` (never to a file inside this repository), using an Azure DevOps +personal access token with Packaging read and write scope: + +```xml + + + + + central + azfunc + [PERSONAL_ACCESS_TOKEN] + + + +``` + +CI covers this automatically. The `MavenAuthenticate@0` task in the build templates authenticates the +`central` repository, so merged changes to dependency versions are ingested by the pipeline. The +credential provider is not used in pipelines. ## Parent POM diff --git a/build.ps1 b/build.ps1 index 6f098bd..a1701d8 100644 --- a/build.ps1 +++ b/build.ps1 @@ -135,9 +135,6 @@ Pop-Location -StackName "libraryDir" $ApplicationInsightsAgentVersion = '3.5.2' $ApplicationInsightsAgentFilename = "applicationinsights-agent-${ApplicationInsightsAgentVersion}.jar" -$ApplicationInsightsAgentUrl = "https://repo1.maven.org/maven2/com/microsoft/azure/applicationinsights-agent/${ApplicationInsightsAgentVersion}/${ApplicationInsightsAgentFilename}" - -# Download application insights agent from maven central $ApplicationInsightsAgentFile = "$currDir/$ApplicationInsightsAgentFilename" # local testing cleanup @@ -157,14 +154,15 @@ if (Test-Path -Path $oldExtract) { Remove-Item -Path $oldExtract -Recurse } -echo "Start downloading '$ApplicationInsightsAgentUrl' to '$currDir'" -try { - Invoke-WebRequest -Uri $ApplicationInsightsAgentUrl -OutFile $ApplicationInsightsAgentFile -} catch { - echo "An error occurred. Download fails" $ApplicationInsightsAgentFile - echo "Exiting" - exit 1 -} +Write-Host "Restoring '$ApplicationInsightsAgentFilename' through Maven" +$mavenArguments = @( + '--batch-mode' + 'org.apache.maven.plugins:maven-dependency-plugin:3.8.1:copy' + "-Dartifact=com.microsoft.azure:applicationinsights-agent:${ApplicationInsightsAgentVersion}:jar" + "-DoutputDirectory=$currDir" +) +& mvn @mavenArguments +StopOnFailedExecution if (-not(Test-Path -Path $ApplicationInsightsAgentFile)) { echo "$ApplicationInsightsAgentFile do not exist." @@ -214,4 +212,4 @@ Write-Host "Creating the functions.codeless file" New-Item -path $currDir\agent -type file -name "functions.codeless" Write-Host "Copying the unsigned Application Insights Agent to worker directory" -Copy-Item "$currDir/agent" "$currDir/azure-functions-java-worker/Azure.Functions.Cli/workers/java" -Recurse -Verbose -Force \ No newline at end of file +Copy-Item "$currDir/agent" "$currDir/azure-functions-java-worker/Azure.Functions.Cli/workers/java" -Recurse -Verbose -Force diff --git a/eng/ci/templates/jobs/build.yml b/eng/ci/templates/jobs/build.yml index 8ec1b2d..a3da951 100644 --- a/eng/ci/templates/jobs/build.yml +++ b/eng/ci/templates/jobs/build.yml @@ -13,6 +13,19 @@ jobs: inputs: workingFile: .npmrc + # Maven resolves plugins and extensions before a pom's repositories are honored. Install the + # mirror before MavenAuthenticate@0, which adds credentials to the same settings file. + - pwsh: | + $m2 = Join-Path $HOME '.m2' + New-Item -ItemType Directory -Path $m2 -Force | Out-Null + Copy-Item '$(Build.SourcesDirectory)/settings.xml' (Join-Path $m2 'settings.xml') -Force + displayName: 'Install Maven settings.xml' + + - task: MavenAuthenticate@0 + displayName: 'Authenticate Maven to CFS' + inputs: + artifactsFeeds: upstream-public + - pwsh: | Write-Host "Java_HOME: $JAVA_HOME" Get-Command mvn diff --git a/eng/ci/templates/official/jobs/build-and-test.yml b/eng/ci/templates/official/jobs/build-and-test.yml index 294e8f4..5eef6f5 100644 --- a/eng/ci/templates/official/jobs/build-and-test.yml +++ b/eng/ci/templates/official/jobs/build-and-test.yml @@ -24,6 +24,19 @@ jobs: - task: NuGetAuthenticate@1 displayName: 'Authenticate NuGet to CFS' + # Maven resolves plugins and extensions before a pom's repositories are honored. Install the + # mirror before MavenAuthenticate@0, which adds credentials to the same settings file. + - pwsh: | + $m2 = Join-Path $HOME '.m2' + New-Item -ItemType Directory -Path $m2 -Force | Out-Null + Copy-Item '$(Build.SourcesDirectory)/settings.xml' (Join-Path $m2 'settings.xml') -Force + displayName: 'Install Maven settings.xml' + + - task: MavenAuthenticate@0 + displayName: 'Authenticate Maven to CFS' + inputs: + artifactsFeeds: upstream-public + - pwsh: | Write-Host "Java_HOME: $env:JAVA_HOME" Get-Command mvn @@ -82,4 +95,4 @@ jobs: JAVA_HOME: $(JAVA_HOME_8_X64) displayName: 'Build & Run tests for java 8' condition: eq(${{ parameters.runEndToEndTests }}, true) - \ No newline at end of file + diff --git a/eng/scripts/Install-MavenCredentialProvider.ps1 b/eng/scripts/Install-MavenCredentialProvider.ps1 new file mode 100644 index 0000000..254508f --- /dev/null +++ b/eng/scripts/Install-MavenCredentialProvider.ps1 @@ -0,0 +1,156 @@ +#!/usr/bin/env pwsh + +<# +.SYNOPSIS + Bootstraps the Azure Artifacts Maven credential provider for local development. + +.DESCRIPTION + Maven packages for this repository are restored from an Azure Artifacts feed. Reads are + anonymous, so this script is only needed by Microsoft developers who have to ingest a package + version that the feed has not cached yet. + + The script: + 1. Verifies the credential provider is present in the local Maven repository, and downloads it + from the public AzureArtifacts tools feed if it is not. + 2. Writes '.mvn/extensions.xml' at the root of the repository so Maven loads the provider. + + '.mvn/' is intentionally listed in .gitignore. The extension exits when it detects a build + context, and committing it would force an authenticated restore on anonymous consumers. Azure + Pipelines uses the MavenAuthenticate@0 task instead. + +.PARAMETER Version + Version of the credential provider to install. Defaults to the version pinned by this script. + +.PARAMETER LocalRepositoryPath + Path to the local Maven repository. Defaults to '~/.m2/repository'. + +.PARAMETER Force + Overwrite an existing '.mvn/extensions.xml' even if it declares extensions this script does not + manage, and re-download the credential provider even when it is already installed. + +.EXAMPLE + ./eng/scripts/Install-MavenCredentialProvider.ps1 + +.LINK + https://eng.ms/docs/coreai/devdiv/one-engineering-system-1es/1es-docs/azure-artifacts/maven-credprovider +#> + +[CmdletBinding()] +param( + [string] $Version = '3.2.1', + [string] $LocalRepositoryPath, + [switch] $Force +) + +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' + +$groupId = 'com.microsoft.azure' +$artifactId = 'artifacts-maven-credprovider' +$bootstrapFeed = 'https://pkgs.dev.azure.com/artifacts-public/PublicTools/_packaging/AzureArtifacts/maven/v1' + +# Maven records the extension against this repository id. It must match the of the repositories +# declared in this repository's pom.xml files, otherwise resolution fails validation later. +$repositoryId = 'central' + +$repoRoot = (Resolve-Path (Join-Path $PSScriptRoot '..' '..')).Path + +if (-not (Get-Command mvn -ErrorAction SilentlyContinue)) { + throw "Maven ('mvn') was not found on PATH. Install Apache Maven 3.0 or above and try again." +} + +if (-not $LocalRepositoryPath) { + $LocalRepositoryPath = Join-Path $HOME '.m2' 'repository' +} + +$artifactDirectory = $LocalRepositoryPath +foreach ($segment in ($groupId.Split('.') + @($artifactId, $Version))) { + $artifactDirectory = Join-Path $artifactDirectory $segment +} + +$artifactPath = Join-Path $artifactDirectory "$artifactId-$Version.jar" + +if ((Test-Path $artifactPath) -and -not $Force) { + Write-Host "Credential provider $Version is already installed at '$artifactPath'." +} +else { + Write-Host "Installing credential provider $Version from the public tools feed..." + + # The bootstrap must run outside of any Maven project so that this repository's own repository + # and extension configuration does not take part in resolving the extension itself. + $workingDirectory = Join-Path ([IO.Path]::GetTempPath()) ('credprovider-bootstrap-' + [Guid]::NewGuid().ToString('n')) + New-Item -ItemType Directory -Path $workingDirectory -Force | Out-Null + + try { + Push-Location $workingDirectory + try { + $mvnArgs = @( + '--batch-mode' + 'dependency:get' + "-Dartifact=${groupId}:${artifactId}:${Version}" + "-DremoteRepositories=${repositoryId}::::${bootstrapFeed}" + ) + + if ($PSBoundParameters.ContainsKey('LocalRepositoryPath')) { + $mvnArgs += "-Dmaven.repo.local=$LocalRepositoryPath" + } + + & mvn @mvnArgs + if ($LASTEXITCODE -ne 0) { + throw "'mvn dependency:get' failed with exit code $LASTEXITCODE." + } + } + finally { + Pop-Location + } + } + finally { + Remove-Item $workingDirectory -Recurse -Force -ErrorAction SilentlyContinue + } + + if (-not (Test-Path $artifactPath)) { + throw "Bootstrap reported success but '$artifactPath' was not found. If a mirror is configured in your settings.xml, temporarily disable it and retry." + } + + Write-Host "Installed credential provider to '$artifactPath'." +} + +$extensionsDirectory = Join-Path $repoRoot '.mvn' +$extensionsPath = Join-Path $extensionsDirectory 'extensions.xml' + +if ((Test-Path $extensionsPath) -and -not $Force) { + $existing = Get-Content $extensionsPath -Raw + + if ($existing -notmatch [regex]::Escape($artifactId)) { + throw "'$extensionsPath' already exists and declares extensions this script does not manage. Review it manually, or re-run with -Force to overwrite it." + } + + if ($existing -match "\s*$([regex]::Escape($Version))\s*") { + Write-Host "'$extensionsPath' is already configured for version $Version." + Write-Host 'Done.' + return + } +} + +$extensionsContent = @" + + + + + $groupId + $artifactId + $Version + + +"@ + +New-Item -ItemType Directory -Path $extensionsDirectory -Force | Out-Null +Set-Content -Path $extensionsPath -Value $extensionsContent -Encoding utf8 + +Write-Host "Wrote '$extensionsPath' for version $Version." +Write-Host 'Done.' diff --git a/eng/scripts/install-maven-credprovider.sh b/eng/scripts/install-maven-credprovider.sh new file mode 100644 index 0000000..00135d4 --- /dev/null +++ b/eng/scripts/install-maven-credprovider.sh @@ -0,0 +1,161 @@ +#!/usr/bin/env bash +# +# Bootstraps the Azure Artifacts Maven credential provider for local development. +# +# Maven packages for this repository are restored from an Azure Artifacts feed. Reads are anonymous, +# so this script is only needed by Microsoft developers who have to ingest a package version that +# the feed has not cached yet. +# +# The script: +# 1. Verifies the credential provider is present in the local Maven repository, and downloads it +# from the public AzureArtifacts tools feed if it is not. +# 2. Writes '.mvn/extensions.xml' at the root of the repository so Maven loads the provider. +# +# '.mvn/' is intentionally listed in .gitignore. The extension exits when it detects a build context, +# and committing it would force an authenticated restore on anonymous consumers. Azure Pipelines +# uses the MavenAuthenticate@0 task instead. +# +# See https://eng.ms/docs/coreai/devdiv/one-engineering-system-1es/1es-docs/azure-artifacts/maven-credprovider + +set -euo pipefail + +GROUP_ID='com.microsoft.azure' +ARTIFACT_ID='artifacts-maven-credprovider' +BOOTSTRAP_FEED='https://pkgs.dev.azure.com/artifacts-public/PublicTools/_packaging/AzureArtifacts/maven/v1' + +# Maven records the extension against this repository id. It must match the of the repositories +# declared in this repository's pom.xml files, otherwise resolution fails validation later. +REPOSITORY_ID='central' + +version='3.2.1' +local_repository_path='' +force=false + +usage() { + cat <<'EOF' +Usage: install-maven-credprovider.sh [options] + +Options: + -v, --version Version of the credential provider to install. + -l, --local-repository Path to the local Maven repository. Defaults to ~/.m2/repository. + -f, --force Overwrite an unmanaged .mvn/extensions.xml and re-download the + credential provider even when it is already installed. + -h, --help Show this help text. +EOF +} + +while [[ $# -gt 0 ]]; do + case "$1" in + -v|--version) + [[ $# -ge 2 ]] || { echo "error: $1 requires a value" >&2; exit 1; } + version="$2" + shift 2 + ;; + -l|--local-repository) + [[ $# -ge 2 ]] || { echo "error: $1 requires a value" >&2; exit 1; } + local_repository_path="$2" + shift 2 + ;; + -f|--force) + force=true + shift + ;; + -h|--help) + usage + exit 0 + ;; + *) + echo "error: unknown argument '$1'" >&2 + usage >&2 + exit 1 + ;; + esac +done + +script_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" +repo_root="$(cd -- "$script_dir/../.." && pwd)" + +if ! command -v mvn >/dev/null 2>&1; then + echo "error: Maven ('mvn') was not found on PATH. Install Apache Maven 3.0 or above and try again." >&2 + exit 1 +fi + +local_repository_specified=true +if [[ -z "$local_repository_path" ]]; then + local_repository_specified=false + local_repository_path="$HOME/.m2/repository" +fi + +group_path="${GROUP_ID//./\/}" +artifact_path="$local_repository_path/$group_path/$ARTIFACT_ID/$version/$ARTIFACT_ID-$version.jar" + +if [[ -f "$artifact_path" && "$force" != true ]]; then + echo "Credential provider $version is already installed at '$artifact_path'." +else + echo "Installing credential provider $version from the public tools feed..." + + # The bootstrap must run outside of any Maven project so that this repository's own repository + # and extension configuration does not take part in resolving the extension itself. + working_directory="$(mktemp -d)" + cleanup() { rm -rf "$working_directory"; } + trap cleanup EXIT + + mvn_args=( + --batch-mode + dependency:get + "-Dartifact=${GROUP_ID}:${ARTIFACT_ID}:${version}" + "-DremoteRepositories=${REPOSITORY_ID}::::${BOOTSTRAP_FEED}" + ) + + if [[ "$local_repository_specified" == true ]]; then + mvn_args+=("-Dmaven.repo.local=$local_repository_path") + fi + + (cd "$working_directory" && mvn "${mvn_args[@]}") + + if [[ ! -f "$artifact_path" ]]; then + echo "error: bootstrap reported success but '$artifact_path' was not found." >&2 + echo "If a mirror is configured in your settings.xml, temporarily disable it and retry." >&2 + exit 1 + fi + + echo "Installed credential provider to '$artifact_path'." +fi + +extensions_directory="$repo_root/.mvn" +extensions_path="$extensions_directory/extensions.xml" + +if [[ -f "$extensions_path" && "$force" != true ]]; then + if ! grep -q "$ARTIFACT_ID" "$extensions_path"; then + echo "error: '$extensions_path' already exists and declares extensions this script does not manage." >&2 + echo "Review it manually, or re-run with --force to overwrite it." >&2 + exit 1 + fi + + if grep -qE "[[:space:]]*${version//./\\.}[[:space:]]*" "$extensions_path"; then + echo "'$extensions_path' is already configured for version $version." + echo 'Done.' + exit 0 + fi +fi + +mkdir -p "$extensions_directory" +cat >"$extensions_path" < + + + + $GROUP_ID + $ARTIFACT_ID + $version + + +EOF + +echo "Wrote '$extensions_path' for version $version." +echo 'Done.' diff --git a/pom.xml b/pom.xml index 8a0e26c..bf51a44 100644 --- a/pom.xml +++ b/pom.xml @@ -59,12 +59,12 @@ + - maven.snapshots - Maven Central Snapshot Repository - https://oss.sonatype.org/content/repositories/snapshots/ + central + https://pkgs.dev.azure.com/azfunc/public/_packaging/upstream-public/maven/v1 - false + true true @@ -72,6 +72,30 @@ + + + central + https://pkgs.dev.azure.com/azfunc/public/_packaging/upstream-public/maven/v1 + + true + + + true + + + + + sonatype-nexus-snapshots + https://pkgs.dev.azure.com/azfunc/public/_packaging/upstream-public/maven/v1 + + false + + + true + + + + com.microsoft.azure.functions diff --git a/settings.xml b/settings.xml new file mode 100644 index 0000000..23e083d --- /dev/null +++ b/settings.xml @@ -0,0 +1,35 @@ + + + + + + upstream-public + Azure Functions public upstream feed + https://pkgs.dev.azure.com/azfunc/public/_packaging/upstream-public/maven/v1 + central + + +