Skip to content

When using a PSRepository backed by Azure Devops Artifact feed that upstreams PSGallery, Install-Module fails on specific module versions #774

Description

@tristanbarcelon

Steps to reproduce

  1. Create an Azure Devops Artifact feed (org or team project scoped) called privatepsgallery and upstream powershellgallery.
  2. Generate a personal access token with Package read/write permission within the organization containing the Artifact feed
  3. Execute the following in PSCore, remembering to replace yourazdopat and yourazdoorganization placeholders with relevant values.
$VSCred = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList @('[email protected]', (ConvertTo-SecureString -String 'yourazdopat' -AsPlainText -Force))
Register-PSRepository -Name 'PrivatePSGallery' -SourceLocation 'https://pkgs.dev.azure.com/yourazdoorganization/_packaging/privatepsgallery/nuget/v2' -Credential $VSCred -InstallationPolicy Trusted
Find-Module -Name 'AzurePipelinesPS' -AllVersions -Repository psgallery -ErrorAction SilentlyContinue | Where-Object { $_.Version -inotlike '*-*' -and ([System.Version] $_.Version -ge [System.Version] '1.0.0' ) } | Select-Object -Property Name, Version, @{ label = 'SortableVersion'; expression = { [System.Version] $_.Version }} | Sort-Object -Property SortableVersion | Foreach-object {
         Write-Host "Installing module $($_.Name) and version $($_.Version)"
         Install-Module -Name $_.Name -RequiredVersion $_.Version -Repository privatepsgallery -Credential $vscred -Scope allusers -AllowClobber -SkipPublisherCheck
}

To reproduce the issue encountered with AWS.Tools.* modules, use this script below.

 [string[]] $Modules = @('AWS.Tools.ElasticLoadBalancingV2', 'AWS.Tools.Pricing', 'AWS.Tools.RDS', 'AWS.Tools.Route53', 'AWS.Tools.S3', 'AWS.Tools.SecurityToken', 'AWS.Tools.SimpleSystemsManagement')
 foreach ($Module in $Modules)
 {
    Find-Module -Name $Module -AllVersions -Repository psgallery -ErrorAction SilentlyContinue | Where-Object { $_.Version -inotlike '*-*' -and ( [System.Version] $_.Version -ge [System.Version] '4.0.0')} | Select-Object -Property Name, Version, @{ label = 'SortableVersion'; expression = { [System.Version] $_.Version }} | Sort-Object -Property SortableVersion | Foreach-object {
        Install-Module -Name $_.Name -RequiredVersion $_.Version -Repository privatepsgallery -Credential $vscred -Scope allusers -AllowClobber -SkipPublisherCheck
    }
 }

Expected behavior

When the module or requiredversion is not found in azure devops feed, it should fetch the module from powershellgallery and make it visible in Azure Devops.

Actual behavior

I get an error like this:

Install-Package: No match was found for the specified search criteria and module name 'AzurePipelinesPS'. Try Get-PSRepository to see all available registered module repositories.

Looking at Azure Devops, I can tell where it's failed, starting with version 3.1.42 of AzurePipelinesPS. I experience the same issues with SqlServerDSC on any version newer than 15.0.1 (e.g., 15.1.0, 15.1.1, 15.2.0). AWS.Tools.Common, AWS.Tools.EKS, AWS.Tools.EC2, etc all fail as well between versions 4.1.99 and 4.1.86, inclusive and 4.1.9.0. Due to the consistency of versions affected in AWS.Tools.* modules, I'm leaning towards corruption of version metadata stored in PSGallery's index/database when AWS pushed these versions to PSGallery.

image

Environment data

> $PSVersionTable
Name                           Value
----                           -----
PSVersion                      7.2.4
PSEdition                      Core
GitCommitId                    7.2.4
OS                             Microsoft Windows 10.0.22621
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0
> Get-Module -ListAvailable PowerShellGet,PackageManagement
ModuleType Version    PreRelease Name                                PSEdition ExportedCommands
---------- -------    ---------- ----                                --------- ----------------
Script     2.2.5                 PowerShellGet                       Desk      {Find-Command, Find-DSCResource, Find-M…

    Directory: C:\program files\powershell\7\Modules

ModuleType Version    PreRelease Name                                PSEdition ExportedCommands
---------- -------    ---------- ----                                --------- ----------------
Script     1.4.7                 PackageManagement                   Desk      {Find-Package, Get-Package, Get-Package…
Script     2.2.5                 PowerShellGet                       Desk      {Find-Command, Find-DSCResource, Find-M…

    Directory: C:\Program Files\WindowsPowerShell\Modules

ModuleType Version    PreRelease Name                                PSEdition ExportedCommands
---------- -------    ---------- ----                                --------- ----------------
Binary     1.0.0.1               PackageManagement                   Desk      {Find-Package, Get-Package, Get-Package…
Script     1.0.0.1               PowerShellGet                       Desk      {Install-Module, Find-Module, Save-Modu…
> Get-PackageProvider -ListAvailable
Name                     Version          DynamicOptions
----                     -------          --------------
NuGet                    3.0.0.1          Destination, ExcludeVersion, Scope, SkipDependencies, Headers, FilterOnTag, …
PowerShellGet            2.2.5.0          PackageManagementProvider, Type, Scope, AllowClobber, SkipPublisherCheck, In…
PowerShellGet            1.0.0.1

Activity

  1. SteveL-MSFT commented on Jun 10, 2022

    @SteveL-MSFT
    Member
  2. tristanbarcelon commented on Jun 10, 2022

    @tristanbarcelon
    Author

    I tried installing versions of dbatools >= 1.0 module in verbose mode

    Find-Module -Name dbatools -AllVersions -Repository PSGallery | Where-Object { [System.Version] $_.Version -ge [System.Version] '1.0' } | Foreach-Object {
         $_.Version
         Install-Module -Name $_.Name -RequiredVersion $_.Version -Scope AllUsers -SkipPublisherCheck -Repository PrivatePSGallery -Credential $VSCred -Verbose
    }

    log entry for nuget package v2 protocol uses FindPackagesById() seems correct. At the end of this logging statement, it is searching for something and it is empty. Is that the version and why is the total package yield 0? When I copy and paste https://pkgs.dev.azure.com/[redactedorg]/_packaging/privatepsgallery/nuget/v2/FindPackagesById()?id='dbatools' into my browser and authenticate to Azure Devops, this returns a proper result.

    VERBOSE: Repository details, Name = 'PrivatePSGallery', Location = 'https://pkgs.dev.azure.com/[redactedorg]/_packaging/privatepsgallery/nuget/v2'; IsTrusted = 'True'; IsRegistered = 'True'.
    VERBOSE: Using the provider 'PowerShellGet' for searching packages.
    VERBOSE: Using the specified source names : 'PrivatePSGallery'.
    VERBOSE: Getting the provider object for the PackageManagement Provider 'NuGet'.
    VERBOSE: The specified Location is 'https://pkgs.dev.azure.com/[redactedorg]/_packaging/privatepsgallery/nuget/v2' and PackageManagementProvider is 'NuGet'.
    VERBOSE: Searching repository 'https://pkgs.dev.azure.com/[redactedorg]/_packaging/privatepsgallery/nuget/v2/FindPackagesById()?id='dbatools'' for ''.
    VERBOSE: Total package yield:'0' for the specified package 'dbatools'
    

    When getting a specific version of dbatools using Nuget V2 protocol the rest query should look like this: https://pkgs.dev.azure.com/[redactedorg]/_packaging/privatepsgallery/nuget/v2/Packages(Id='dbatools',Version='0.8.952').

  3. anamnavi commented on Jun 23, 2022

    @anamnavi
    Member

    tristanbarcelon thanks for reaching out with this! We're looking into this scenario and seeing what's happening and will get back soon.

  4. tristanbarcelon commented on Jun 29, 2022

    @tristanbarcelon
    Author

    Hi Anam Navied (@anamnavi), I'm curious whether you've been able to reproduce this issue I'm observing. It started affecting newer versions of AWS.Tools.* modules starting with 4.1.116.

  5. anamnavi commented on Jun 30, 2022

    @anamnavi
    Member

    tristanbarcelon I'll test and get back on this early next week, sorry for the delay, just coming back from vacation.

  6. tristanbarcelon commented on Jul 20, 2022

    @tristanbarcelon
    Author

    Anam Navied (@anamnavi) , does your team collaborate with Azure Devops artifacts team? I noticed another pattern whenever these errors happen. Specifically, the errors happen within a few hours after a new version of a module is published in PSGallery. I don't know whether Azure Devops artifacts caches metadata from PSGallery rather than querying it directly for the presence of a module and version. Re-attempting the same script later works, though for some modules, the upstreaming behavior still remains broken (dbatools, sqlserverdsc, azurepipelinesps) and now it affects ImportExcel version 7.8.0.

  7. self-assigned this
    on Jan 18, 2023
  8. added this to the 3.0-Consider milestone on Jan 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions