Repository navigation
Is it time for "PowerShell vZeroTechnicalDebt" ("vNext") and/or for an opt-in mechanism into new/fixed features that break backward compatibility? #6745
Description
Note that the term technical debt is used loosely here to mean "accumulated broken behavior that can't be fixed without breaking backward-compatibility"; strictly speaking the term has a different, specific meaning.
Update: Rob Holt (@rjmholt) has started a discussion about how to implement and manage breaking changes: #13129.
Note: This issue, which arose out of #5551 (comment), is just to get the discussion started to see if there's a fundamental willingness to entertain such changes. Eventually, RFC(s) are needed.
PowerShell's steadfast commitment to backward compatibility has served the community very well over the years.
On the flip side, the inevitable by-product was the accumulation of technical debt that requires memorizing exceptions and is a barrier to newcomers.
Certain fundamental problems that exist today can only be solved at the expense of backward compatibility, and there are two - not mutually exclusive - ways to handle that:
-
Implementation of a "PowerShell vZeroTechnicalDebt" edition that sheds all technical debt, but forfeits backward compatibility (possibly with future versions using semantic versioning to indicate compatibility)
-
Integration of backward-compatibility-breaking features / fixes into the existing code base, available strictly on an opt-in basis only.
Which approaches, if any, are we willing to consider?
Once we have clarity on that, we can flesh out the processes.
Here are some of the existing fundamental problems that can only be solved at the expense of backward compatibility; I'm sure others will think of more:
-
The complexity and inconsistency of current error handling and problematic [lack of] integration with "native" (external) programs - Inconsistent handling of native command stderr #3996
-
Inconsistent, hard-to-predict preference-variable / common-parameter inheritance - Need straightforward way to honor the caller's preference-variable values / inherited common parameters in functions defined in script modules #4568
-
Performance issues due to
[object[]]being the fundamental collection type - PowerShell should support creating an List similar to how it supports arrays #5643 (comment)- While PowerShell understandably will never match the speed of Unix utilities, it is all the more important to make interop with them predictable and as painless as possible - see remaining issues re quoting and parsing.
-
Problematic dynamic features that should work lexically: Foreach loop using | % {} syntax doesn't support continue #3879 (comment) re
breakandcontinue, https://github.com/PowerShell/PowerShell-RFC/blob/master/1-Draft/RFC0003-Lexical-Strict-Mode.md reSet-StrictMode(though the latter may be fixed without breaking compatibility) -
[psobject]-related problems (though perhaps they can be fixed without breaking compatibility): Should all [psobject] cmdlet parameters be changed to [object]? #5551, Objects are situationally invisibly [psobject]-wrapped, sometimes causing unexpected behavior. #5579, -as operator does not recognize System.Management.Automation.PSCustomObject instances as such #4343, Preference variable $OutputEncoding in child scopes misbehaves with encodings created with New-Object / commands rather than expressions #5763 -
The unfortunate
-LiteralPath/-Pathsplit - brief rationale at Output-to-file cmdlets should not support wildcards with -Path #6714 (comment) and escaping woes at Output-to-file cmdlets should not support wildcards with -Path #6714 (comment) (not sure there's a good solution) -
Broken quoting for external programs - Invoking external command lines (utilities) is broken with respect to quoting on Unix #3734, Start-Process doesn't pass arguments with embedded whitespace and double quotes correctly #5576 (and others) and the related languishing RFC
-
-Commandand-FileCLI argument parsing - Command line arguments with a dollar sign #4024 (comment), CLI:pwsh -Command -andpwsh -File -unexpectedly exhibit pseudo-interactive behavior, lack support for argument passing #3223 - and general misalignment with the CLI of POSIX-like shells - Align PowerShell's CLI / startup behavior with POSIX-like shells such as Bash - command-line arguments #3743 - including user profiles getting loaded by default even in non-interactive (script) invocations - #!/usr/local/bin/powershell bang should not load profile #992 -
Inconsistent and surprising parsing of compound command-line arguments - Parsing of compound command-line tokens into arguments is surprising #6467 - and non-parameter tokens that look like parameters - Parameter parsing/passing: an unquoted argument that starts with a "-" (hyphen) is broken in two at the first "." (period) #6291, Parameter parsing/passing: an unquoted argument that looks like a named parameter/value pair separated with ":" (colon) is broken in two in positional binding #6292, and Parameter parsing/passing: unquoted tokens that look like named arguments with colon as the separator are broken in two when passed indirectly via $Args / @Args #6360
-
Broken handling of
ValueFromRemainingArgumentsparameters, as argued in Parameter binding problem with ValueFromRemainingArguments in PS functions #2035 (comment), with broken behavior shown in Write-Output -NoEnumerate outputs PSObject[] rather than Object[] and generally doesn't respect the input collection type #5955 and Write-Output -NoEnumerate wrap singular items into PSObject[] #5122 - going back to Fix Parameter Binder bug with Advanced Functions and ValueFromRemainingArguments #2038
Environment data
Written as of:
PowerShell Core v6.0.2
Activity
BurtHarris commentedon Apr 26, 2018
Interesting. I've always wondered why the
1in.ps1would be used to add a sort of semantic versioning to PowerShell scripts, allowing a script to indicate if it was adapted to possible breaking changes.Jaykul commentedon Apr 26, 2018
There expensive to fix problems which I think of as the technical debt of PowerShell not adopting new .NET concepts as they came along
Language syntax for type parameters for generics methods Enhancement: Ability to invoke generic methods with explicit type parameters when PowerShell cannot figure out <T> from the context #5146
Engine support for
Extension Methodsto automatically surface in the type system (like they do in compiled .Net languages). Add support for extension methods (LINQ) #2226Language support for async APIs Task-based Asynchronous Pattern #6716 (and RFC)
There are a few regrettable features:
Could we do something based on on type hints (like Kirk Munro (@KirkMunro)'s FormatPx) instead of cmdlets that output formatting objects? Asynchronous behavior of implicit Format-Table use can have surprising side effects - console output may appear out of order #4594 Get-FormatData doesn't report formatting data for some types for which formatting data ships with PowerShell. #4237 Feature Request: Add conditional colorization to Format-Table command #3886
Would you normalize the Registry provider to be content based -- instead of feeling like a demo of how a property provider could work? Registry provider: New-Item -Force shouldn't blindly recreate the leaf path component (registry key) #5444
Would you refactor PSProviders deeply to make them easier to write? It's a shame that it takes two layers of abstraction to arrive at "SHiPS" and finally give people a way to write providers that they're willing to use
How about a bigger question:
Would we be willing to reconsider the "many ways is better" approach and work on a "pit of success" approach? I mean, would we be willing to remove features that are considered "the easy way" but which are fundamentally worse, in favor of having only "the right way" to do things? E.g.:
List[PSObject]Dictionary[PSObject, PSObject](orDictionary[string, PSObject]😲)mklement0 commentedon Apr 27, 2018
All great suggestions, Joel Bennett (@Jaykul).
Re making
@()notation mean aList[PSObject]: I think we can take this further and use an efficiently extensible collection type as PowerShell's fundamental collection type, so that it is used wherever[object[]]currently is (and is used internally as well as when returning collections, with no type-conversion performance penalty); there are challenges around+use, but @PetSerAl has suggested a custom list implementation to deal with that.Re making
Process {}the default block in functions, as an aside: it's whatFiltercurrently does, but it is limited to the - implied -Processblock, and this function variant seemingly never really caught on; unifying the two in the context ofFunctionmakes sense to me.KirkMunro commentedon Apr 27, 2018
Funny, on the Azure PowerShell community standup call right now, and they're going to move from AzureRm to a new Az module that is cross platform (no separate AzureRm.Core), and with all command prefixes changed from AzureRm to Az. The two will be side by side for a while, but they're moving to the new command set.
Too bad PowerShell hasn't had an opportunity yet to create a new executable that would run side-by-side with Windows PowerShell, but that could break away from some of the legacy crud that drags it down.
bergmeister commentedon Apr 29, 2018
Just imagine what would happen should we go for it:
Realistically speaking it would take at least 6 months of focussed effort to come up with a 'no regrets' version and another 6 months to reiterate on it. Then 3rd party modules need to adapt to it as well in order for this version to be useful, which will take at least another 6 months to get a reasonable coverage (and bear in mind that some modules never will)... Then account for delay and unexpected problems, etc... So, no, I think it is only wishful thinking that one can get rid of all technical debt in one version in one go (and still develop and not just maintain the old version).
As much as I wished that such a version existed, I think it will only be possible to get to it slowly one breaking change at a time. With v6 a lot of breaking changes were already accepted but I think if one includes too many breaking changes in one version, it will become too complex to upgrade existing scripts/modules. It's good to discuss the most valuable breaking changes but until there isn't an LTS version of pwsh, I do not think it is time to think about having a 2nd train of pwsh with more substantial changes in parallel to the existing mainstream version.
BrucePay commentedon Apr 30, 2018
Christoph Bergmeister (@bergmeister) Agreed. However even relatively small changes on a core path can seriously hinder adoption. Look at Python 3. It took 10 years to really catch on. With much bigger changes, who knows how long it will take for Perl 6 to be dominant (and it took them 15 years to come up with their right stuff so 1.5 years for PowerShell++ seems optimistic :-)) On the other hand PHP seems to break things on a regular basis, possibly due to the way and what it's used for.
BurtHarris commentedon Apr 30, 2018
Python 3 is certainly the horror show, has it really caught on yet? I'm still running 2.7, and don't plan on upgrading any time soon. But I haven't heard much about Perl 6 recently either...
I think the lessons to learn from Python there are to separate breaking changes to the language from a version change to the engine. Hypothetically, a PS 7 engine could still run earlier scripts (.PS1) files in a no-breaking changes mode, while if the script were marked as 7 aware (say with a .PS7 extension) they could declare they have been updated and that the require at-least PS 7 to run. Hope that makes sense.
BurtHarris commentedon Apr 30, 2018
Perhaps the best success story is JavaScript/TypeScript/Babel. A transpiler (with source-map support) seems like the way-to-go for language evolution.
BrucePay commentedon Apr 30, 2018
Javascript is a special case. You're pretty much stuck with it so transpiling is really the only option. Typescript is "just" Javascript with extensions so it's easy for people to adopt. Any javascript program is a typescript program so you start with what you have and just add annotations from there. Dart, on the other hand, is it's own language but transpiles to either javascript or a native runtime in Chrome (at least that was the plan at one point). Dart doesn't seem to have picked up much adoption outside of Google, likely because it is its own language..
Burt Harris (@BurtHarris)
I was reading an article last week where the author was claiming critical mass had been achieved for Python 3. That all the core modules were available and now people where migrating in droves. We shall see..
WRT
.ps1, we reserved the right to change the extension in case we got the language completely wrong resulting in catastrophic changes to the runtime such that scripts for the previous version just wouldn't work. But changing the extension also leads to a huge pile of work because so many things in the ecosystem are tied to an extension. So it's not something to do lightly. And of course, being part of Windows, if we forked the extension, we'd still have to maintain both versions (kind of like Python 2/3).mklement0 commentedon Apr 30, 2018
Christoph Bergmeister (@bergmeister):
Valid concerns, but in the spirit of:
please share any that you may have in mind.
While a (lexically scoped) opt-in mechanism for incompatible changes is a solution, I'm concerned about two things:
Piecemeal introduction can lead to no one being able to remember which version is required for what feature; Perl (v5-) comes to mind.
The code base becoming bloated and hard to maintain, and the resulting binary being equally bloated, which hurts (at least) startup performance.
I've said it before: to me - and this is just a hunch - the v6 GA was an unfortunate compromise between making old-timers unhappy with breaking changes while carrying forward enough baggage to hinder adoption In the Unix[-like] world.
That said, given PowerShell's relative youth in the Unix[-like] world, perhaps there is (still) more of a willingness to work out problems by way of incompatible changes. As Bruce Payette (@BrucePay) states, Windows PowerShell will have to be maintained anyway, and it can remain the safe haven for backward compatibility.
alx9r commentedon May 4, 2018
I think the scale of the negative consequences of such breaking changes has already been covered, and I agree with most of those points. I am writing this because I am doubtful that, in the greater context, these changes would yield significant benefits in the first place. At least for the way I use PowerShell.
The OP includes the following statement:
The implied premise of this statement seems to be that making these various breaking changes would alleviate the burden of memorizing exceptions. Indeed that would be a great outcome. However, I am skeptical that that would be the result. PowerShell's behavior is deliberately rich. This makes it both expressive and unpredictable. Expressiveness and predictability seem to work against one another, at least amongst the languages I am familiar with.
While I agree that many of the breaking changes mentioned above would improve predictability somewhat in some cases, many unpredictable and surprising aspects of the language will remain. Despite spending years writing PowerShell, I am still frequently surprised by PowerShell behavior that seems to be by design and probably shouldn't be changed. Some recent examples that come to mind are as follows:
[scriptblock]arguments (Why does parameter binding impose the ScriptBlockArgumentNoInput restriction? #6419)[AutomationNull]::Valueto$nullduring parameter binding (Why is ,(1)[1] different from ,$null? #6357 is related)breakandcontinueon flow of control in the pipeline (break in the pipeline: Why does End run in some cases and not in others? #5811)$null(SO)$_(SO 1, SO 2)I expect that these examples are just a small fraction of the many other carefully-designed but surprising nuances I have not yet discovered. I selected these examples because
I'm fine with this. The overwhelming majority of surprising PowerShell behavior does not have lasting impact on my success with PowerShell. Surprising behavior is almost always caught immediately by testing during development. I learn from the things that slip through and use that to improve my testing strategy. This is true whether those surprising behaviors are the kind that could be eliminated or the kind that must remain.
Whether the proposed breaking changes above are made to or not, PowerShell will never become so predictable that I can significantly reduce test coverage. In other words, the way I use PowerShell, I don't think there's much of an upside that's even possible by making the breaking changes proposed above.
(BTW, thank you Michael Klement (@mklement0) for directly asking this question. This has been in the back of my mind for a while. It's good to see the opportunity for everyone to say their piece.)
mklement0 commentedon May 6, 2018
Thanks, alx9r.
I think it's important to distinguish between intrinsic complexity and extrinsic complexity:
Intrinsic complexity stems from the inherent complexity of the concepts being implemented, which in the case of PowerShell has two primary sources: internal complexity from introducing a new paradigm (object-based pipeline) and marrying two distinct worlds (shell syntax and programming-language syntax), and external complexity from interfacing with multiple, disparate outside worlds.
Intrinsic complexity is managed by documenting concepts and features properly and also given them names.
Why does parameter binding impose the ScriptBlockArgumentNoInput restriction? #6419 is a prime example: a powerful and widely useful feature underlies it, but it is
presently not documented at all, as far as I know, and it has no established name; see Document ad-hoc script-block arguments for end users MicrosoftDocs/PowerShell-Docs#2338 [since documented as delay-bind script blocks]The member enumeration feature similarly lacks an established name and focused documentation - see Update about_Properties to introduce the term member enumeration and to document the behavior with array-valued properties MicrosoftDocs/PowerShell-Docs#2198
Are the rules for conditional enumeration the same for comparison operators and the pipeline? #5832 is another candidate for proper conceptual documentation.
.Count, but also a.Lengthproperty and parameter-binding enumeration behavior being subtly different from pipeline/operator behavior.Extrinsic complexity stems from leaky abstractions and inconsistencies.
Such complexity should be eliminated .
If that is not an option due to backward-compatibility concerns, such complexity should be documented as known problems.
The script-module variable scoping behavior (which you reference in the context of
$_) is more than just a quirk: it is the root cause of a major problem, the previously mentioned Need straightforward way to honor the caller's preference-variable values / inherited common parameters in functions defined in script modules #4568.All other issues you mention strike me as falling into the extrinsic category (rather than being the result of careful design), because they all present as inconsistencies that have no obvious (documented) rationale or benefit:
$nullto pass no arguments rather than a positional$nullargument.[System.Management.Automation.Internal.AutomationNull]::Valuebe converted to$nullduring parameter binding, even though the type is preserved in direct variable assignment? See No any way to pass empty pipeline (AutomationNull) to function parameter #9150 (comment)breakandcontinue, across the call stack, with quiet termination if no enclosing loop is found?While the wealth of features and the joining of disparate worlds alone makes it hard to remember all requisite intrinsic complexity, minimizing the extrinsic one is still important.
Having to test your intended approach first without just knowing and trusting that it will work - or to have things break unexpectedly due to surprising behavior - is a serious productivity (and enjoyment) hindrance (even though PowerShell commendably makes it very easy to interactively test behavior).
It comes down to solid concepts (that don't contravene intuitive expectations), descriptive naming, and good documentation:
If I don't know it / not sure if I remember correctly, I need to know where to look it up, and have faith that known problems and edge cases are also documented (either as part of the regular help topics or via links from there).
So even if we decide that eliminating extrinsic complexity is not an option, we can at least document it systematically - and the discussion here can serve as the starting point for compiling a "pitfall gallery" (which may also include cases of unavoidable intrinsic complexity that may be surprising).
HumanEquivalentUnit commentedon Aug 6, 2018
Roman Kuzmin has been collecting such a gallery for a while, here: https://github.com/nightroman/PowerShellTraps
mklement0 commentedon Aug 6, 2018
Thanks, HumanEquivalentUnit - that looks like a great collection.
Along with the issues collected in this thread, it could form the basis for a gallery that is part of the official documentation, with each entry augmented with, as appropriate:
a design rationale that explains why the behavior, though perhaps surprising, is justified after all (a properly framed explanation may make the element of surprise may go away)
an acknowledgement of the problem, stating that:
59 remaining items
SeeminglyScience commentedon Nov 29, 2020
I think I saw it stated in a blog post that they intended to. No longer 100% sure.
You can follow the rules of semantic versioning without also providing an API around semantic versioning.
Would you increment a whole major version for those though? They seem pretty unlikely to affect a whole ton of folks, unlike some of the changes discussed ITT.
Either way, even if you make the assumption that a major version increase is enough warning for the existing user base, I would guess it doesn't address the big reasons they are adverse to them. Note that I'm not saying the changes here should or shouldn't be implemented, only that semantic versioning probably isn't a super big part of answering that.
AndrewSav commentedon Nov 29, 2020
'Adverse' or 'Averse'?
beedell-roke commentedon Oct 8, 2021
I support this. One of Microsoft's greatest problems for its regular consumers (although definitely not its business-customers) is the inconsistency noticed whenever utilizing its products because it attempts to retain backward-compatibility to such an impressive but ultimately detrimental extent.
dkaszews commentedon Jul 28, 2022
Isn't
Set-StrictModealready a "breaking" change for most scripts? Yet many people put it at the top of every file to avoid things like access to non-existent variables, while only applying to local scope without affecting other code. So just whack it under 4.0 and call it a day.It would be very valuable to provide tooling checking code for potential breaking changes. Especially for changes related to operator precedence, such tool should emit a warning for all cases where logic changes.
Set-StrictMode -Compatibility OperatorPrecedencesounds good to me, as it would also allow writing code compatible with both options.microsoft-github-policy-service commentedon Nov 16, 2023
This issue has not had any activity in 6 months, if this is a bug please try to reproduce on the latest version of PowerShell and reopen a new issue and reference this issue if this is still a blocker for you.
microsoft-github-policy-service commentedon Nov 16, 2023
This issue has not had any activity in 6 months, if this is a bug please try to reproduce on the latest version of PowerShell and reopen a new issue and reference this issue if this is still a blocker for you.
microsoft-github-policy-service commentedon Nov 16, 2023
This issue has not had any activity in 6 months, if this is a bug please try to reproduce on the latest version of PowerShell and reopen a new issue and reference this issue if this is still a blocker for you.
beedell-roke commentedon Nov 16, 2023
microsoft-github-policy-service commentedon Nov 24, 2023
This issue has been marked as "No Activity" as there has been no activity for 6 months. It has been closed for housekeeping purposes.
mklement0 commentedon Mar 25, 2024
Another undesirable behavior worth revisiting in the context of allowing breaking changes:
[string]-typed parameters unexpectedly and unhelpfully accept arrays in certain scenarios - see Make Join-Path parameter -ChildPath accept an array #21367 (comment) (bottom section).[-]Is it time for "PowerShell vZeroTechnicalDebt" and/or for an opt-in mechanism into new/fixed features that break backward compatibility?[/-][+]Is it time for "PowerShell vZeroTechnicalDebt" ("vNext") and/or for an opt-in mechanism into new/fixed features that break backward compatibility?[/+]