Skip to content

Powershell Hashtable Keys are case-sensitive on Ubuntu 18.04 #7761

Closed
#8566
softagram/PowerShell
#3
Closed
Powershell Hashtable Keys are case-sensitive on Ubuntu 18.04#7761
#8566
softagram/PowerShell
#3
Assignees
SteveL-MSFT
Labels
Issue-BugIssue has been identified as a bug in the productIssue-Discussionthe issue may not have a clear classification yet. The issue may generate an RFC or may be reclassifResolution-FixedThe issue is fixed.
Milestone
Future

Description

@dantraMSFT
Dan Travison (dantraMSFT)
opened on Sep 11, 2018

On a clean installation of Ubuntu 18.04 from Azure, creating a hashtable in script results in a hashtable with case-sensitive keys.

The LANG environment variable is set to 'C.UTF-8' and both CultureInfo.CurrentCulture and CultureInfo.CurrentUICulture report en-US-POSIX.

The result is scripts that are not case-strict when using hashtable keys will break.

Steps to reproduce

$h = @{HostName = 'host'}
$h.hostname
$h.HostName

Expected behavior

The value 'host' is returned for both $h.HostName and $h.hostname

Actual behavior

No value is returned for $h.HostName is 'host' while $h.hostname returns $null.

Environment data

> $PSVersionTable
 Name                           Value
----                           -----
PSVersion                      6.1.0-rc.1
PSEdition                      Core
GitCommitId                    6.1.0-rc.1
OS                             Linux 4.15.0-1022-azure #23-Ubuntu SMP Thu Aug 16 15:19:42 UTC 2018
Platform                       Unix
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Activity

  1. BrucePay commented on Sep 12, 2018

    @BrucePay
    Bruce Payette (BrucePay)
    on Sep 12, 2018
    Collaborator

    How odd. The code that constructs a hashtable is hard-coded to be case-insensitive:

                yield return Expression.Assign(temp,
                Expression.New(ordered ? CachedReflectionInfo.OrderedDictionary_ctor : CachedReflectionInfo.Hashtable_ctor,
                                ExpressionCache.Constant(keyValuePairs.Count),
                                ExpressionCache.CurrentCultureIgnoreCaseComparer.Cast(typeof(IEqualityComparer))));

    What happens when you manually create the hashtable?

    $h = [hashtable]::new([StringComparer]::CurrentCultureIgnoreCase)
$h.hostname = 123
$h.HostName
$h.hostname

  2. rjmholt commented on Sep 12, 2018

    @rjmholt
    Rob Holt (rjmholt)
    on Sep 12, 2018
    Collaborator

    James Truher (@JamesWTruher) found that this is due to Ubuntu 18.04's default LANG=C.UTF-8 (compare to 16.04's LANG=en-us.UTF-8 for our installs).

    Travis Plunk (@TravisEz13) made the point that this behaviour may be by design: the if the C.UTF-8 locale is supposed to be something along the lines of "culture invariant", then case-sensitivity may not mean anything and it just performs a strict codepoint-to-codepoint comparison.

    However, from the Debian manpage linked above, it sounds like the standard C locale should obey the casing rules of en_US, meaning C.UTF-8 should obey the casing rules of en_US.UTF-8. So this is a bug, probably in .NET Core.

    Also worth mentioning Travis Plunk (@TravisEz13)'s note of .NET's culture invariant mode: dotnet/announcements#20

  3. Ilya (iSazonov)
    added
    Issue-Discussionthe issue may not have a clear classification yet. The issue may generate an RFC or may be reclassif
    on Sep 12, 2018

  4. rjmholt commented on Sep 12, 2018

    @rjmholt
    Rob Holt (rjmholt)
    on Sep 12, 2018
    Collaborator

    I've managed a simple repro that doesn't depend on PowerShell and have opened an issue in corefx

  5. rjmholt commented on Sep 12, 2018

    @rjmholt
    Rob Holt (rjmholt)
    on Sep 12, 2018
    Collaborator

    CoreFX have designated the behaviour with that locale as by design as Travis Plunk (@TravisEz13) suggested, see https://github.com/dotnet/corefx/issues/32250#issuecomment-420749205.

    The options I see for our handling of it are:

    • This behaviour is by design for us too
    • We special-case handling of this locale
    • We change handling of hashtable keys to use CultureInvariantIgnoreCase or OrdinalIgnoreCase as I think Bruce Payette (@BrucePay) suggested

  6. rjmholt commented on Sep 13, 2018

    @rjmholt
    Rob Holt (rjmholt)
    on Sep 13, 2018
    Collaborator

    .NET team's suggested workaround: https://github.com/dotnet/corefx/issues/28611#issuecomment-386120096

    Official POSIX locale documentation from the Open Group: http://pubs.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap07.html#tag_07_02

  7. rjmholt commented on Nov 6, 2018

    @rjmholt
    Rob Holt (rjmholt)
    on Nov 6, 2018
    Collaborator

    Apparently there are issues with CultureInfo.InvariantCulture: dotnet/roslyn#29938 (comment)

  8. bergmeister commented on Dec 29, 2018

    @bergmeister
    Christoph Bergmeister (bergmeister)
    on Dec 29, 2018
    Contributor

    I suggest to use OrdinalIgnoreCase for the hashtable comparison please. It would be great if a fix for this gets shipped in the next patch of 6.1.

  9. Steve Lee (SteveL-MSFT)
    added
    Issue-BugIssue has been identified as a bug in the product
    on Dec 31, 2018
  10. Steve Lee (SteveL-MSFT)
    added this to the Future milestone on Dec 31, 2018
  11. Steve Lee (SteveL-MSFT)
    self-assigned this
    on Dec 31, 2018

  12. SteveL-MSFT commented on Dec 31, 2018

    @SteveL-MSFT
    Steve Lee (SteveL-MSFT)
    on Dec 31, 2018
    Member

    Christoph Bergmeister (@bergmeister) Setting LANG before starting pwsh is a viable workaround. I don't think this will be in next 6.1 patch.

  13. ffeldhaus commented on Dec 31, 2018

    @ffeldhaus
    Florian Feldhaus (ffeldhaus)
    on Dec 31, 2018
    Contributor

    I just checked that the issue also affects Mac OS if LANG is set to C.utf8.

    Florians-MacBook-Pro-3:~ florianfeldhaus$ export LANG=C.utf8
Florians-MacBook-Pro-3:~ florianfeldhaus$ pwsh
PowerShell 6.1.1
Copyright (c) Microsoft Corporation. All rights reserved.

https://aka.ms/pscore6-docs
Type 'help' to get help.

PS /Users/florianfeldhaus> $PSVersionTable

Name                           Value
----                           -----
PSVersion                      6.1.1
PSEdition                      Core
GitCommitId                    6.1.1
OS                             Darwin 18.2.0 Darwin Kernel Version 18.2.0: Mon Nov 12 20:24:46 PST 2018; root:xnu-4903.231.4~2/RELEASE_X86_64
Platform                       Unix
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

PS /Users/florianfeldhaus> Get-Culture

LCID             Name             DisplayName
----             ----             -----------
4096             en-US-POSIX      English (United States, Computer)

PS /Users/florianfeldhaus> $h=@{H=1}
PS /Users/florianfeldhaus> $h.h
PS /Users/florianfeldhaus> $h.H
1
  14. Ilya (iSazonov)
    closed this as completedin #8566on Jan 10, 2019
  15. Ilya (iSazonov)
    added
    Resolution-FixedThe issue is fixed.
    on Jan 19, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Issue-BugIssue has been identified as a bug in the productIssue-Discussionthe issue may not have a clear classification yet. The issue may generate an RFC or may be reclassifResolution-FixedThe issue is fixed.

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions