Question: not getting expected results for get-formatdata

This is a followup to issue #8063 which appears closed but which did not fix the originally reported problem.

Steps to reproduce

Hi Steve,

I have been anxiously awaiting this fix to test out the issues we discussed last year.

Unfortunately the problems do not appear to be fixed and things appear to be seriously broken still to the point that the results are unusable.

Run these commands

pwsh -noprofile
Get-FormatData -PowerShellVersion 6.2 -TypeName System.IO.DirectoryInfo | Export-FormatData -LiteralPath DirectoryInfo.ps1xml
Get-FormatData -PowerShellVersion 6.2 -TypeName System.IO.FileInfo | Export-FormatData -LiteralPath FileInfo.ps1xml
If you compare DirectoryInfo.ps1xml and FileInfo.ps1xml you will find the are exactly the same, however, the output for a directory item versus a file item should not be the same as a directory does not contain a length.

Ignoring that issue, the results returned are missing lots of XML code that was previously in FileSystem.format.ps1xml.

I won't detail them all here as you can compare the results produced of Get-FormatData with the old FileSystem.format.ps1xml file to see them all, however, I will point out a few.

FileSystem.format.ps1xml created a SelectionSet named FileSystemTypes and then used that and a CustomControlName FileSystemTypes-GroupingFormat for the view definitions. Those are all missing from the "Get-FormatData" results for System.IO.DirectoryInfo / System.IO.FileInfo.

Ignore those issues, I attempted to merge my changes into the results returned from Get-FormatData and I found that my changes were not displaying so I did this simple test.

pwsh -noprofile
Get-FormatData -PowerShellVersion 6.2 -TypeName System.IO.DirectoryInfo | Export-FormatData -LiteralPath DirectoryInfo.ps1xml

Edit the DirectoryInfo.ps1xml file and make the following change:

Look for <Label>LastWriteTime</Label> and change it to <Label>XXXXWriteTime</Label>

Now run

Update-FormatData -PrependPath .\DirectoryInfo.ps1xml

Now run Get-ChildItem and you will see that the label still says LastWriteTime instead of XXXXWriteTime

Now run this to get what the updated formatting info looks like

Get-FormatData -PowerShellVersion 6.2 -TypeName System.IO.DirectoryInfo | Export-FormatData -LiteralPath DirectoryInfo2.ps1xml

You will find that it does show the change, however. there is another section after it that still has the label as "LastWriteTime".
I believe all of these issues are because Get-FormatData is not returning the full and correct format data for these types.

If I run "Update-FormatData -Prepend My,FileSystem.format.ps1" which uses the modified version of the old FileSystem.format.ps1xml file that used to be included in PowerShell, then everything works correctly and it has my changes.

So we are back to the original problem.

There needs to be a way for me to obtain the format data being used (either via a file like it previously did or with Get-FormatData returning the correct formatting that is really being used).

Since PowerShell is no longer including the *.format.ps1xml files and Get-FormatData is broken and does not return the correct format data I have no way to determine what changes were made in new versions of PowerShell so that I can integrate them into my modified version.

I hope that this can be made a priority and get fixed soon.

Holler if you want any more details.

Thanks!

Joe

Joe S (@JoeSalmeri)

Joe S (@JoeSalmeri)
We actually do you one format definition to display both objects. The scripts embedded in the object take are of the issues of displaying or not displaying the length:

$format = get-formatdata -TypeName System.IO.FileInfo -PowerShellVersion 6.2
$format.FormatViewDefinition[0].Control.Rows[0].Columns[2].DisplayEntry.Value

results:

if ($_ -is [System.IO.DirectoryInfo]) { return '' }
if ($_.Attributes -band [System.IO.FileAttributes]::Offline)
{
    return '({0})' -f $_.Length
}
return $_.Length

Travis Plunk (@TravisEz13)

Travis Plunk (@TravisEz13)

That explains why Get-FormatData for System.IO.DirectoryInfo and Get-FormatData for System.IO.FileInfo return the same results.

I figured that you might be doing something like as part of the cleanup when the format.ps1xml files were integrated into the code, however, that does not address the main issue I reported that was supposed to be fixed.

Either Get-FormatData is not returning the data properly or Export-FormatData is not properly formatting it into XML (or I'm making a mistake but I don't believe that is the case).

First some quick background:

I have some customization that I have made to the old FileSystem.format.ps1xml file. With each new version of PS I have taken the newest version and integrated my changes into it to make sure that I picked up any changes that MS made.

PS Core removed the format files and integrated them into the C# code and the solution I was given was to use Get-FormatData and Export-FormatData to create the XML which I could then tweak and then apply using Update-FormatData -Prepend.

Get-FormatData had bugs which were to be fixed in PS Core 6.2 (by Steve)

With PS Core 6.2 available now I am testing out that functionality and the ability for me to re-integrate my changes.

Ok, now that you have the history here is a simple example that is not working properly:

pwsh -noprofile
Get-FormatData -PowerShellVersion 6.2 -TypeName System.IO.DirectoryInfo | Export-FormatData -LiteralPath DirectoryInfo.ps1xml

Edit the DirectoryInfo.ps1xml file and make the following change:

Look for LastWriteTime and change it to XXXXWriteTime
Now run

Update-FormatData -PrependPath .\DirectoryInfo.ps1xml

Now run Get-ChildItem and see that it is not displaying the XXXXWriteTime label it is displaying LastWriteTime still.

Clearly there appears to be something wrong with roundtrip between Get-FormatData, Export-FormatData, Update-FormatData as a simple change like that should have not had any issues.

Note: This is a simple change to illustrate that either Get-FormatData or Export-FormatData has a problem, my actual customization does a bunch of other things.

When I compare the DirectoryInfo.ps1xml file created from Get-FormatData and Export-FormatData there are many differences from the old FileSystem.Format.ps1xml file (I listed some in my earlier reply).

To me it seems like the issue is still with Get-FormatData not properly returning the correct format data for DirectoryInfo so when I modify it and then try to reapply it does not work.

If I am making a mistake, could you please point out to me how to properly do the roundtrip to get the format data for DirectoryInfo as XML that I can modify and then reapply?

Thanks very much!

Regards,

Joe

Travis Plunk (@TravisEz13)

Travis Plunk (@TravisEz13)

Hi Travis, never heard anything back on this.

With the removal of the format files, there really needs to be a way to get the format data, export it, modify the exported format data and then re-apply the update.

The proposed solution I was given does not seem to work because EITHER Get-FormatData is still not fixed to get ALL the format data OR more likely Export-FormatData is not properly exporting all the format data retrieved.

Here are the commands I was told to use:

Get-FormatData -PowerShellVersion 6.2 -TypeName System.IO.DirectoryInfo | Export-FormatData -LiteralPath DirectoryInfo.ps1xml

Modify DirectoryInfo.ps1xml using my editor/tools.

Update-FormatData -PrependPath .\DirectoryInfo.ps1xml

It is quite disappointing to see functionality removed when the replacement functionality does not appear to work properly

If I am incorrect and there are no more problems with Get-FormatData and Export-FormatData could you PLEASE provide some documentation or address my issue in that a roundtrip does not appear to be working?

This is on PowerShell Core 6.2 where it was all supposed to be fixed.

Thanks for your help.

Joe

To be honest we really need to have a cmdlet or something to create format data that hooks into the C# APIs so that it's a lot easier to work with. Would be great to have a DSL-style module to make it really simple to work with. A lot of the C# APIs do a lot of method chaining to get things done, might be interesting to try to expose that as a series of cmdlets for use in PS, and improve Update-FormatData to accept this kind of input.

Rain Sallow (/u/ta11ow) (@vexx32)

Rain Sallow (/u/ta11ow) (@vexx32)

I agree that would be a welcome change.

My issue is that the files are gone now and the methods that would allow me to modify still seem to be broken. I am still using my modified FileSystem.format.ps1xml file from PS 5. In the past I would check whenever that file was updated for any changes to make sure I kept my file updated with future PS changes.

Looking at my changes they really come down to changing the format for numbers (length attribute) and dates. The other changes are to adjust the width in output to handle the change in size for the number and date formats.

In Windows the formatting of Dates and Numbers is configured via Control Panel / Region and in Linux and also in other programming languages like Python they are configurable via locale.

Since those are user defined settings for the formatting we are discussing it seems like PS should be using those settings instead of what it currently does or maybe we just need something like

 SET-DEFAULT-NUMBER-FORMAT
 SET-DEFAULT-DATE-FORMAT
 SET-DEFAULT-DATETIME-FORMAT
 SET-DEFAULT-TIME-FORMAT

If those existed then I suspect that I would not need the width changes I make because PS would adjust it's output based on the widths defined in those defaults.

Is this even on the radar of any of the developers since we lost functionality and the suggestion methods are broken?

Thanks!

Number and datetime formats are typically controlled by culture settings, if I'm not mistaken? You might be able to adjust those from the objects Get/Set-Culture work with.

Hi Vexx32,

Sorry just saw this as I did not receive the normal email notification.

(Get-Culture).NumberFormat has the desired settings but they are not being used by the built in formats.

I would try modifying the settings with the objects you get from Get-Culture, and then pass them back into Set-Culture and/or $PSCulture/$PSUICulture

$Culture = Get-Culture
$Culture.NumberFormat = <# set number format #>
$Culture | Set-Culture
$PSUICulture = $PSCulture = $Culture

Rain Sallow (/u/ta11ow) (@vexx32)

Rain Sallow (/u/ta11ow) (@vexx32),

(Get-Culture).NumberFormat has

NumberGroupSizes : {3}
NumberGroupSeparator : ,

but the views defined for System.IO.DirectoryInfo and System.IO.FileInfo do not use the culture information when displaying the file sizes (length property).

In my custom format.sp1xml I just format them the way I want but it seems to me that the real issue is that the views defined do not use the culture settings. If the internal views were changed to use the culture settings then everyone would get the values displayed in their native format (or could change the culture settings as you suggested) and I would not need my custom formats.

Joe S (@JoeSalmeri)

I believe all of these issues are because Get-FormatData is not returning the full and correct format data for these types.

Joe S (@JoeSalmeri), just dropping in real quick as I haven't read this thread fully but did a search for 'scriptblock' and found nothing. You suspected it may be Get-FormatData causing issues. But that is being piped into Export-FormatData. That format type, DirectoryInfo, contains scripts, which if needed, would have to be explicitly requested by using the IncludeScriptBlock switch parameter for Export-FormatData.

Marc Kassay (@marckassay)

Marc Kassay (@marckassay)

Thanks for chiming in regarding this issue.

Just to summarize for you.

PowerShell no longer includes the *.ps1xml files that previously shipped with it.

Previously whenever a new PS version came out, I copied the ps1xml file and reapplied my customizations and then used Update-FormatData to load my changes. I did this so that my customizations always included any changes that MSFT made to the ps1xml files.

Once the files were removed I have been trying to use Get-FormatData to get the full source for DirectoryInfo so that I can apply my customizations to it.

Originally there was a bug (which is supposed to be fixed now) but I have tried the recommendations and still do not receive the full format data.

Here are the commands recommended that I am using (with your suggestion now included):

Get-FormatData -TypeName System.IO.DirectoryInfo -PowerShellVersion 7.0 |
Export-FormatData -LiteralPath DirectoryInfo.ps1xml -IncludeScriptBlock

If I issue that command AFTER I load my customizations then the DirectionInfo.ps1mxl file DOES contain the ScriptBlocks that I had defined in my ps1xml file.

If I start powershell without my profile (which loads my customizations) using

pwsh -noprofile

and then issue the above Get-FormatData command then the DirectoryInfo.ps1xml file DOES NOT contain any ScriptBlocks.

After initially reporting the issue about not being able to get the full format data for DirectoryInfo, I also asked how the "length" field information was handled using the new FormatData (where DirectoryInfo and FileFileInfo were merged) because only files have a length.

Above you can see Travis Plunk (@TravisEz13) reply where he showed the following:

$format = get-formatdata -TypeName System.IO.FileInfo -PowerShellVersion 6.2
$format.FormatViewDefinition[0].Control.Rows[0].Columns[2].DisplayEntry.Value

Which produced

if ($_ -is [System.IO.DirectoryInfo]) { return '' }
if ($.Attributes -band [System.IO.FileAttributes]::Offline)
{
return '({0})' -f $.Length
}
return $_.Length

However, using the Get-FormatData command above, that information does not appear in the ps1xml file.

To summarize, I am trying to get the full format data for DirectoryFileInfo so that I can export it customize to my liking and then Update-FormatData to load my customizations.

I use this approach so that if Microsoft ever makes changes to the format data then I will not lose those changes in my customizations.

If you issue the pwsh -noprofile command followed by the Get-FormatData command above and look at the resulting DirectoryInfo.ps1xml file you will see that no Script blocks exist and it does NOT include the ScriptBlock shown by Travis Plunk (@TravisEz13)

Interestingly the resulting XML contains the following:

<PropertyName>LengthString</PropertyName>

however, if you issue

dir | gm *

the resulting object does not contain a LengthString property.

If you issue the following commands from Travis Plunk (@TravisEz13) now on PS 7 rc 2

$format = get-formatdata -TypeName System.IO.FileInfo -PowerShellVersion 6.2
$format.FormatViewDefinition[0].Control.Rows[0].Columns[2].DisplayEntry.Value

you get
LengthString

So something else has changed again since Travis Plunk (@TravisEz13) reply

One of the formating changes that I make in my customizations is to add commas to the length property. Above you will see a recommendation to use Get-Culture to configure my desired format, however, that does now work because DirectoryInfo does not honor the culture settings.

So the end goal is to use Get-FormatData and Export-FormatData to produce the complete format data that is being used by DirectoryInfo now that the ps1xml files are removed.

It appears that this is still broken and not possible to do.

It is frustrating that the ps1xml files were removed before the replacement code to extract the format data was working properly.

This is not an intermittent problem and therefore it should be easy to debug why we can no longer get the complete formatdata for DirectoryInfo.

Please try the above commands and you will see what I am talking about.

Hopefully someone will investigate and either fix the problem or provide the correct commands to get the full format data.

Holler if you need more details.

Thanks again!

Joe S (@JoeSalmeri)

Joe S (@JoeSalmeri) thanks for the thorough explanation! I didn't realize that ps1xml are excluded in shipment until you mentioned, as it is still mentioned in the docs, and obviously still can be used in PowerShell. So it's just PowerShell today doesn't return all or correct ps1xml data, in the scenarios you mentioned.

Marc Kassay (@marckassay)

Marc Kassay (@marckassay) sure no problem!

You are correct (at least last time I had checked) that the documentation still references the ps1xml files.

It is really frustrating that the files were removed before the complete replacement solution was in place.

Unfortunately the issue does not seem to get any priority.

Travis Plunk (@TravisEz13)

Travis Plunk (@TravisEz13)

So with PS 7.2 released I decided to revisit the issues I reported here.

It looks like the Get-FormatData part works but Export-FormatData is still broken.

This can be seen by doing the following

Get-FormatData -TypeName System.IO.DirectoryInfo | Export-FormatData -Path fi.ps1xml -IncludeScriptBlock

followed by

Update-FormatData -PrePendPath

Then do a Get-ChildItem on a folder than contains other folders and also files and noticed the messed up output.

I have resolved the items that Export-FormatData forgot to export but I have one remaining issue that I could not resolve and was hoping that you help me out.

In most view it has the following:

PSParentPath

The issue is that PSParentPath has a value like this:

Microsoft.PowerShell.Core\FileSystem::/usr/bin

/usr/bin replaced with whatever directory you were displaying

Back in the days of the old format files it had the following code which stripped that out of PSParentPath

<Controls>
    <Control>
        <Name>FileSystemTypes-GroupingFormat</Name>
        <CustomControl>
            <CustomEntries>
                <CustomEntry>
                    <CustomItem>
                        <Frame>
                            <CustomItem>
                                <Text AssemblyName="System.Management.Automation" BaseName="FileSystemProviderStrings" ResourceId="DirectoryDisplayGrouping"/>
                                <ExpressionBinding>
                                    <ScriptBlock>
                                        $_.PSParentPath.Replace("Microsoft.PowerShell.Core\FileSystem::", "")
                                    </ScriptBlock>
                                </ExpressionBinding>
                            </CustomItem>
                        </Frame>
                    </CustomItem>
                </CustomEntry>
            </CustomEntries>
        </CustomControl>
    </Control>
</Controls>

I have been unable to figure out how the NEW formatting is removing the "Microsoft.PowerShell.Core\FileSystem::" as that custom control is no longer in the format data.

When ps is installed the NEW format data removes it even though I cannot find anything in the code that does it, however,
when I do the Get-FormatData | Export-FormatData and then Update-FormatData it starts displaying the
"Microsoft.PowerShell.Core\FileSystem::" again for each group.

How is that getting removed now that the control is no longer created? (or if it is I don't see it in the format data)
Obviously something else is missing the get / export update formatdata cycle but I have not been able to figure out what it is.

Can you please point me at how this is now done?

Thanks!

Ok, I went to the PS source code in in this file

PowerShell-7.2.0/src/System.Management.Automation/FormatAndOutput/DefaultFormatters/FileSystem_format_ps1xml.cs

The view definitions are created and I see that it creates a shared custom control to the string used for PSParent grouping.

I cannot find that definition in the Get-FormatData returned or in the Export-FormatData written to the file.

Obviously I can add the above Control code in my previous message to the ps1xml file but is there a way to extract that data so that get-formatdata and export-formatdata include it?

Joe S (@JoeSalmeri)

Joe S (@JoeSalmeri) Thanks for your investigations!

I'd ask you to reword the issue to get progress easily.

Pick up a type (which is not in FormatData already)
Create format XML file (or some files to demonstrate issues by one) for it
Do import/get/export to xml
Compare source and result xml files

Such process helps us to understand the issue(s) easily and write new tests for future PSs.

Also. it seems we had a discussion to add XML output for Get-FormatData or something like. This could helps too.

Then, you could review related issues #11596, #11308, #11307.

/cc Dongbo Wang (@daxian-dbw) Michael Klement (@mklement0) who could advice.

Ilya (@iSazonov)

Ilya (@iSazonov)

I cannot pick a type that is not in the FormatData because the entire issue is based on changing one of the built in format types
and the fact that doing a roundtrip ( Get-FormatData, Export-FormatData, Update-FormatData) does not work because the Get-FormatData and ExportData do not produce the complete existing format data that the PS source code creates.

It is easy to demonstrate this with the following example:

pwsh -noprofile

md ./test-dir
echo 'file-1' > ./test-dir/file-1
echo 'file-2' > ./test-dir/file-2
md ./test-dir/test-dir2
md ./test-dir/test-dir3

Get-ChildItem ./test-dir

    Directory: /home/joe/test-dir

UnixMode   User             Group                 LastWriteTime           Size Name
--------   ----             -----                 -------------           ---- ----
drwx------ joe              joe-denise         11/12/2021 09:08              0 test-dir2
drwx------ joe              joe-denise         11/12/2021 09:08              0 test-dir3
-rw------- joe              joe-denise         11/12/2021 09:08              7 file-1
-rw------- joe              joe-denise         11/12/2021 09:08              7 file-2

Get-FormatData -Type System.IO.DirectoryInfo | Export-FormatData -IncludeScriptBlock -Path DirectoryInfo.ps1xml

Update-FormatData -PrePend ./DirectoryInfo.ps1xml

Get-ChildItem ./test-dir

   PSParentPath: Microsoft.PowerShell.Core\FileSystem::/home/joe/test-dir

UnixMode   User             Group                 LastWriteTime           Size Name
--------   ----             -----                 -------------           ---- ----
drwx------ joe              joe-denise         11/12/2021 09:08              0 test-dir2
drwx------ joe              joe-denise         11/12/2021 09:08              0 test-dir3

LastWriteTime : 11/12/2021 9:08:14 AM
Length        : 7
Name          : file-1


LastWriteTime : 11/12/2021 9:08:14 AM
Length        : 7
Name          : file-2

No changes were made to the System.IO.DirectoryInfo that was retrieved, yet now the
Get-ChildItem output is messed up and has two problems.

The first issue that jumps right out is the fact that directories are being displayed
in table format whereas files are being displayed in list format.

"Get-ChildItem ./test-dir | Format-Table" still produces the messed up output.

"Get-ChildItem ./test-dir/ | Format-Table -View children" which is the view that
is the default view on Windows also has output that is messed up and has the
same two problems:

   PSParentPath: Microsoft.PowerShell.Core\FileSystem::/home/joe/test-dir

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d----          11/12/2021  9:08 AM                test-dir2
d----          11/12/2021  9:08 AM                test-dir3

LastWriteTime : 11/12/2021 9:16:01 AM
Length        : 7
Name          : file-1


LastWriteTime : 11/12/2021 9:16:01 AM
Length        : 7
Name          : file-2

The reason the first issue occurs is because Export-FormatData is not exporting
complete data and is missing values which were there when the
source code created them and are also in the Get-Format data produced.

PS Source code reference:

PowerShell-7.2.0/src/System.Management.Automation/FormatAndOutput/DefaultFormatters/FileSystem_format_ps1xml.cs

It is easy to fix the bad DirectoryInfo.ps1xml file that Export-FormatData created
by adding back the missing information in the file created.

The second issue is with the directory line.

When it is working that line says:

    Directory: /home/joe/test-dir

When the bad DirectoryInfo.ps1xml file that Export-FormatData created
is reloaded then it says:

    PSParentPath: Microsoft.PowerShell.Core\FileSystem::/home/joe/test-dir

That one took me a lot longer to figure out until I looked at the PS source code
referenced above.

In the old days when PS included the *.format.ps1xml files the custom control
was included in the files.

Those files were removed from the release and moved into the source code
where it has the code to add the custom control but Get-FormatData / Export-FormatData
is not including that custom control in the output they generate and therefore
you get the PSParentPath value above instead of the PSParent value that
is modified by the custom control in line 17 of the source code.

Although the above was done on Linux using the new childrenWithUnixStat, I
also showed that the children table view is also broke, and these same issues
occur on Windows 10 too ( with the exception of the childrenWithUnixStat view
which only exists on Linux).

For the first issue shown, the data is in the Get-FormatData output, the
problem is that Export-FormatData does not export everything and is missing
the additional values.

To fix that issue Export-FormatData needs to be fixed to include ALL values, not just the first one.

For the second issue shown, it looks like Get-FormatData output does not
include the custom control from the source code and therefore Export-FormatData
does not have the information to export.

To fix that issue Get-FormatData needs to also output the custom control that is created in the PS source code and Export-FormatData needs to export the custom control that would then be output by Get-FormatData.

This gives you a detailed example of exactly what is broken (with the source code references to show what should have been output) and also the what code is broken and causing the issues.

This is 100% reproducible using the example above and is an issue on BOTH Linux and Windows.

If you need anything else please let me know but with the details above and example it should be clear exactly what the problems are and what needs to be done to fix them.

Thanks!

Have a great day!

This issue has not had any activity in 6 months, if this is a bug please try to reproduce on the latest version of PowerShell and reopen a new issue and reference this issue if this is still a blocker for you.

This issue has been marked as "No Activity" as there has been no activity for 6 months. It has been closed for housekeeping purposes.

Navigation Menu

Repository navigation

Question: not getting expected results for get-formatdata #9300

Steps to reproduce

2 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Question: not getting expected results for get-formatdata #9300

Description

Steps to reproduce

Activity

TravisEz13 commented on Apr 5, 2019

JoeSalmeri commented on Apr 6, 2019

JoeSalmeri commented on May 28, 2019

vexx32 commented on May 28, 2019

JoeSalmeri commented on May 31, 2019

vexx32 commented on May 31, 2019

JoeSalmeri commented on Aug 20, 2019

vexx32 commented on Aug 20, 2019

JoeSalmeri commented on Aug 20, 2019

marckassay commented on Jan 15, 2020

JoeSalmeri commented on Jan 19, 2020

marckassay commented on Jan 22, 2020

JoeSalmeri commented on Jan 22, 2020

2 remaining items

JoeSalmeri commented on Nov 11, 2021

JoeSalmeri commented on Nov 12, 2021

iSazonov commented on Nov 12, 2021

JoeSalmeri commented on Nov 12, 2021

microsoft-github-policy-service commented on Nov 16, 2023

microsoft-github-policy-service commented on Nov 16, 2023

microsoft-github-policy-service commented on Nov 16, 2023

microsoft-github-policy-service commented on Nov 23, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions