Repository navigation
Find-Module -Command need perf improvement #31
Description
Activity
We can look at the speed but since people can register arbitrary repositories, they can be on flaky networks, etc etc the caller will definitely need to apply a timeout in order to ensure bounded latency.
We would probably limit the search to PSGallery by default and allow the user to add custom repositories if there's a need
Not sure if this is a proper place to suggest it, but I think it is a related functionality.
I think PowerShellGet should provide functionality similar to
apt-get updatewhich will simply download list of modules and their basic properties (i.e. version, commands list, etc). ThenFind-Module -Commandor its equivalent can just look into available cache and it will not depend on the network.Today, command
Find-Modulewith no arguments (or with-Repository PSGallery) provides the same data, but it is very slow (more than 2 minutes in my test). It gives about 30MB of data which can be compressed to 1MB, so network transfer time should not be the issue. This should be first enabled on repository side and then clients can get commands for it.Reacted by Steve Lee, John Zabroski, Cedric Ahlers, Matthew A Cargile and David RichmondIgor (@iricigor) Finally, someone who understands how package managers are supposed to work, back when we had 56k modems and debian key rings for package security.
Reacted by Igor and Matthew A CargileWe would probably limit the search to PSGallery by default and allow the user to add custom repositories if there's a need
Steve Lee (@SteveL-MSFT) Your ultimate design problem is going to be the fact you are tightly coupled to nuspec and have no way to declare to nuspect specific sub-package artifacts like commands. You can see your design flaw right here, where you accumulate PowerShell Module-specific entities in a general purpose "Tags" field":
Even with lucene on the backend, you will not be able to quickly search on an elasticsearch index. Plus, you're going to be decoding things, when you really just want a HDB document type that supports a list of commands as a field type.
I like the suggestion to adopt the
apt updatemodel. However, my original thinking is that the backend opens up the nupkg and puts the metadata into something that can be queried quickly (like AzureTable). There should not be any decoding during search time.Please see PowerShell/PowerShellGetv2#335 where I suggest breaking Publish-Module into smaller pieces. The smaller pieces can then be used to define a cleaner interface for metadata.
Have you used ElasticSearch? I ask because I don't understand how AzureTable solves your problems. To the best of my knowledge and searching, AzureTable doesn't support wildcard operators. But my searching was from 2013, so maybe it's changed, but if the currently supported operators are available here, then it's not supported and it's a really bad design choice. It looks like the best it can do is prefix matching.
Also, while AzureTable is highly available, I am unaware of it being used for fast querying.
Edit: To be clear, your SLA should be 10 milliseconds or effectively the time of your latency from pinging the remote server - ElasticSearch is effectively that fast.
I took a liberty and did proof-of-concept module psaptgetupdate. It finds information about commands, modules or scripts in about 20ms via local cache.
You can test it with:
Install-Module psaptgetupdate -Scope CurrentUser # test functionality Find-ModuleFromCache PlatyPS # check speed, it does 50 runs in under the second Measure-Command {1..50 | % {Find-ModuleFromCache PlatyPS}} | Select TotalMilliseconds
Local cache can be updated when needed (via command
psaptgetupdate) and the initial update is done when the module is imported. The whole update takes less than 5 seconds. The update is served as a pre-prepared package from Azure, as explained on the design page.This module also modifies the
CommandNotFoundActionerror handler, to suggest how to install missing command (see it in action). Also, module can be used to update all modules on your system. It runs for 5 seconds total, where 3-4s are spent on querying systerm for existing modules. It checks 20-30 modules per second! Try it with:Update-ModuleFromCache -WhatIf
Reacted by Steve Lee and Edwin YoungHi Igor, thanks very much for showing how much this can help. There's a lot of potential! We'll need to give it a bit of thought to see how best to incorporate in PSGet - things like what to do with multiple repositories. I noticed that Find-CommandFromCache get-azvm returns all the commands which start with get-azvm rather than just 'get-azvm', which is likely an easy change.
It will nice to have a public API for fast search from the local cache - we could use the API in PowerShell engine for Help and IntelliSense subsystems.
- Microsoft, How often do we have to explain this to you? Don't THINK... READ THE DOCUMENTATION... apt-get is literally 20 year old technology better than ANYTHING you have "thought to see how best to incorporate in PSGet". Perspective is worth 80 IQ points. Don't think "how can we improve PSGet"... think... jeez... apt-get toasted us on performance. What else are they doing better than us that we should get inspired by? It just so happens they dealt with multiple repositories over 2 decades ago, too.…On Fri, Nov 2, 2018 at 1:38 AM Ilya ***@***.***> wrote: It will nice to have a public API for fast search from the local cache - we could use the API in PowerShell engine for Help and IntelliSense subsystems. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <https://github.com/PowerShell/PowerShellGet/issues/287#issuecomment-435276672>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAbT_VB6Qw5bUU-jOlzi7KRzLmI17iYxks5uq9pigaJpZM4UNMLw> .
As part of PowerShell/PowerShell#1982, if the user types a command that doesn't exist, we want to be able to search PSGallery and inform the user to install a module. However, the current execution of
Find-Module -Command footakes ~4.5s, this needs to be closer to 100ms (or less).