Submitting ConvertFrom-Html RFC by MSAdministrator · Pull Request #137 · PowerShell/PowerShell-RFC

Josh Rickard (MSAdministrator) · 2018-07-15T01:44:05Z

Submitting ConvertFrom-Html RFC for consideration.

Mark Kraus (markekraus) · 2018-07-15T01:54:21Z

+    Method     = 'Post'
+}
+
+$dnsDumpsterObject = (Invoke-WebRequest @props).Content | ConvertFrom-Html


Acturally, this will work:

$dnsDumpsterObject = Invoke-WebRequest @props | ConvertFrom-Html

Paramater binding will convert BasicHtmlWebResponseObject to a string which will be the contents of Content.

Mark Kraus (@markekraus) Thanks for this, and this makes sense. I'll update the RFC.

Mark Kraus (markekraus) · 2018-07-15T01:56:04Z

+
+As a PowerShell Core user, I can convert HTML content to objects so that I can easily work with downloaded or local HTML content.
+
+As a IT Administrator, I can call `Invoke-WebRequest` and then use `ConvertFrom-Html` to convert the `RawContent` of my Web Request to an object so that I can easily work with HTML strings/content.


Hmm RawContent? Would that be supported by AngelSharp?

I actually meant Content in this one. Originally I thought RawContent should be supported but after further looking I think it should support Content instead.

Mark Kraus (markekraus) · 2018-07-15T02:00:59Z

What happens when more than one string is piped to the command?

$Html1, $Html2 | ConvertFrom-Html

Will parity be ensured such that

@([pscustomobject]@{a=1},[pscustomobject]@{a=2}) | ConvertTo-Html | ConvertFrom-Html

works as expected?

Ilya (iSazonov) · 2018-07-16T05:23:49Z

+
+# ConvertFrom-Html
+
+Currently the PowerShell Core Web CmdLets do not have access to the `HtmlWebResponseObject` and currently only contains the `BasicHtmlWebResponseObject` type.  Because of this, the capability to parse HTML using the `ParsedHtml` property of the `HtmlWebResponseObject` type does not exist within PowerShell Core.


We could add "The state of affairs" sub-title.

Ilya (iSazonov) · 2018-07-16T05:25:16Z

+RelationLink      Property   System.Collections.Generic.Dictionary[string,string] RelationLink {get;}
+StatusCode        Property   int StatusCode {get;}
+StatusDescription Property   string StatusDescription {get;}
+```


This is distracting. I think we should remove the block.

I will remove this. Thanks!

Ilya (iSazonov) · 2018-07-16T05:27:12Z

+StatusDescription Property   string StatusDescription {get;}
+```
+
+This RFC proposes that the creation of a new CmdLet named `ConvertFrom-Html`.  This CmdLet is to be implemented into PowerShell Core and should utilize the [AngelSharp](https://github.com/AngleSharp/AngleSharp) framework for converting strings into a PSCustomObject.


Please add sub-title "Proposition".

"for converting strings into ..." -> "for converting HTML strings into ...".

Here and below - what is output object type or types? We mention - "objects", "PSCustomObject" ...

Based on the RFC guidelines, i'm not sure if this is needed.

Ilya (iSazonov) · 2018-07-16T05:29:44Z

+#### CommonParameters
+
+This cmdlet supports the common parameters: -Debug, -ErrorAction, -ErrorVariable, -InformationAction, -InformationVariable, -OutVariable, -OutBuffer, -PipelineVariable, -Verbose, -WarningAction, and -WarningVariable. For more information, see about_CommonParameters (http://go.microsoft.com/fwlink/?LinkID=113216).
+


I believe we can remove this - we haven't requiriments for them in the RFC.

I will remove this.

Ilya (iSazonov) · 2018-07-16T05:35:21Z

+
+## Specification
+
+The proposal is to create a new 'ConvertFrom-Html' cmdlet that will convert Html content using PowerShell core.


Proposal was already above.

Ilya (iSazonov) · 2018-07-16T05:41:18Z

+### OUTPUTS
+
+#### PSCustomObject
+


What inputs and outputs we expect? Only strings and PSCustomObject ? I think it's useful to think about it now.

Can output be XML? Dom?

I believe that ConvertFrom-Html would output PSCustomObject. If we want to covert it after that, we can use ConvertTo CmdLets to the appropriate types.

I would prefer to see something closer to the HTML standard like that was in .ParsedHtml and .AllElements.
Mark Kraus (@markekraus) Thoughts?

Since the proposed cmdlet is ConvertFrom-Html and there is already a ConvertTo-Html the highest priority for the cmdlet is that it maintain parity with the existing cmdlet. Lack of conversion parity in converter cmdlets is something that constantly causes pain and frustration in the community. Whatever object is output by the new cmdlet, it needs to support

$html | ConvertFrom-Html | ConvertTo-Html

and

$obj | ConvertTo-Html | ConvertFrom-Html

in a manner that is expected by the user.

That object could contain methods and such which allow access to an object like 5.1's ParesdHtml.

An alternative is to use a different cmdlet name that way it is not associated with ConvertTo-Html. This could allow for returning a feature-rich object directly from AngleSharp. I'm starting to think this is the better idea. I'm not a huge fan of how the current ConvertTo-Html works and I'm no a fan of how limited ConvertFrom-Html would need to be as a result of how ConvertTo-Html works.

Wouldn't this just output a AngleSharp.Dom.Html.HtmlDocument object? ConvertTo-Html can be modified to accept this type and serialize back to html.

Yes. That would satisfy the issue. So long as users can convert back and forth reliably.

Ilya (iSazonov) · 2018-07-16T05:47:19Z

+Some considerations to keep in mind:
+
+- Converted Html may be piped to any number of CmdLets.  For example, ConverTo-Json, ConverTo-Csv, ConvertTo-Xml, and ConvertTo-Html
+- Based on conversations in #3267 and #2867, this CmdLet should use AngleSharp to parse Html


The first alternative about which we remember is .Net Core - here we should mention why we don't use it.

Josh Rickard (MSAdministrator) · 2018-07-17T02:17:49Z

Mark Kraus (@markekraus) Regarding your post above (I cannot seem to reply to it) but I think that each HTML String should be passed along independently through the pipeline. This would work similarly to how ConvertFrom-Json works now (i believe).

…ction

…ontent property

bviktor · 2018-08-28T16:59:33Z

Ah, this looks like a must have one, subscribing :)

bviktor · 2018-08-28T17:02:09Z

+
+Additionally, Windows PowerShell Web CmdLets utilize Internet Explorer to parse HTML content. Since non-Windows systems do not have Internet Explorer, PowerShell Core utilizes the `BasicHtmlWebResponseObject` which does not contain this property.
+
+This RFC proposes that the creation of a new CmdLet named `ConvertFrom-Html`.  This CmdLet is to be implemented into PowerShell Core and should utilize the [AngelSharp](https://github.com/AngleSharp/AngleSharp) framework for converting HTML strings into a PSCustomObject.


Typo: Angle vs Angel :)

Steve Lee (SteveL-MSFT) · 2018-10-06T05:05:29Z

If there are no other major concerns and we are past the feedback deadline, I can have @PowerShell/powershell-committee review this

bviktor · 2019-06-03T13:17:21Z

Any news? Or not making it into 7 either?

Fotis Gimian (fgimian) · 2019-06-17T23:53:30Z

This functionality would be quite essential if PowerShell 7 is to be a true replacement for Windows PowerShell 5. Any chance this can be moved forward and included as part of the v7 release? 🙏

Rain Sallow (/u/ta11ow) (vexx32) · 2019-06-17T23:58:14Z

Sorry, how is this essential as a replacement for PS 5? PS 5 never had a ConvertFrom-Html cmdlet. 😕

Fotis Gimian (fgimian) · 2019-06-18T04:43:56Z

Sorry, how is this essential as a replacement for PS 5? PS 5 never had a ConvertFrom-Html cmdlet. 😕

Because Invoke-WebRequest in PowerShell 5 provided full capabilities for manipulating the returned HTML due to the fact it was using the IE engine. PowerShell Core does not and returns only a basic HTTP response.

Some more info can be found here.

Joey Aiello (joeyaiello) · 2019-07-08T19:38:42Z

This inspired a meta-conversation with James Truher (@JamesWTruher) and Steve Lee (@SteveL-MSFT) about when and how we decide to take modules into PowerShell itself. At face value, this seems like the right approach, but we'd really prefer to see an implementation of this on the Gallery first so we can get some community usage that drives the quality and gives us an impression of how useful/popular this would be inbox. (See this comment on an AccountManagement module for similar reasoning.)

I recognize that this is to replace Windows PS 5 behavior in Invoke-WebRequest, and we're not dismissing the idea of shipping this in 7, but it's probably unlikely given the velocity we've been working at. But a working implementation would make it a lot more likely. 😄

Fotis Gimian (fgimian) · 2019-07-09T03:51:48Z

Thanks so much Joey Aiello (@joeyaiello), totally respect that. In case it helps, I've been using PowerHTML to perform this function and it works flawlessly.

Ilya (iSazonov) · 2019-07-09T05:49:06Z

we'd really prefer to see an implementation of this on the Gallery first so we can get some community usage

This again demonstrates the benefits of corefxlab model. MSFT team could create PowerShellLab repo (with prepared infrastructure to automatically publish nugets) and collect some new and experimental modules there. (I am ready to maintain it too :-) )

Akhil Thomas (akhilthomas011) · 2019-10-26T11:34:41Z

Any progress on this?

Joey Aiello (joeyaiello) · 2020-03-10T19:28:51Z

Given the relatively low number of downloads of PowerHTML, as well as the requirement of adding HtmlAgilityPack (or some equivalent), increasing both size and surface area of potential vulnerabilities, we feel that this is best served by the Gallery today.

In general, we're also raising the bar in terms of what we feel comfortable bringing into the PowerShell release. Module experimentation can and should continue in the Gallery.

Joey Aiello (joeyaiello) · 2020-03-10T19:31:04Z

Rejecting due to existing community implementation of PowerHTML and rationale above.

If there is significantly more demand for this cmdlet in the future, an issue should be opened in PowerShell/PowerShell to discuss inclusion in the PowerShell release itself.

Ilya (iSazonov) · 2020-03-11T05:11:54Z

Steve Lee (@SteveL-MSFT) mentioned that MSFT team is going to work on coloring in next milestone. In the case we need a color rendering. We already have simple rendering for MarkDown and we could consider to have more power rendering engine to support coloring and HTML look very attractive as color rendering language.

Miodrag Milić (majkinetor) · 2020-08-14T08:40:28Z

Rejecting due to existing community implementation of PowerHTML and rationale above.

This should work OTB, we live in HTML world. There is no good rationale for rejecting this.

Submitting ConvertFrom-Html RFC

4812a7f

Mark Kraus (markekraus) reviewed Jul 15, 2018

View reviewed changes

Ilya (iSazonov) reviewed Jul 16, 2018

View reviewed changes

Josh Rickard (rickardja) added 7 commits July 16, 2018 22:27

Added proposal statement to the top of RFC

9620a4a

Removed output from BasicHtmlWebResponseObject

9ce9081

Clarified output to be PSCustomObject in Motivation Section

7d404a4

Remove proposal and modified parameter definition in Specification se…

bfc0385

…ction

Removed CommonParameters from RFC

95e5e78

Modified example to accept parameter binding of WebRequest object - C…

0e062a4

…ontent property

Modified Alternate Proposals to clarify output is PSCustomObject

f0d5848

bviktor reviewed Aug 28, 2018

View reviewed changes

Steve Lee (SteveL-MSFT) added the Review - Committee label Jun 6, 2019

Prepare RFC0055 - ConvertFrom-Html for rejection

161baa4

Joey Aiello (joeyaiello) merged commit 26ca1f2 into PowerShell:master Mar 10, 2020


		As a PowerShell Core user, I can convert HTML content to objects so that I can easily work with downloaded or local HTML content.

		As a IT Administrator, I can call `Invoke-WebRequest` and then use `ConvertFrom-Html` to convert the `RawContent` of my Web Request to an object so that I can easily work with HTML strings/content.


		# ConvertFrom-Html

		Currently the PowerShell Core Web CmdLets do not have access to the `HtmlWebResponseObject` and currently only contains the `BasicHtmlWebResponseObject` type. Because of this, the capability to parse HTML using the `ParsedHtml` property of the `HtmlWebResponseObject` type does not exist within PowerShell Core.

		#### CommonParameters

		This cmdlet supports the common parameters: -Debug, -ErrorAction, -ErrorVariable, -InformationAction, -InformationVariable, -OutVariable, -OutBuffer, -PipelineVariable, -Verbose, -WarningAction, and -WarningVariable. For more information, see about_CommonParameters (http://go.microsoft.com/fwlink/?LinkID=113216).


		## Specification

		The proposal is to create a new 'ConvertFrom-Html' cmdlet that will convert Html content using PowerShell core.


		Additionally, Windows PowerShell Web CmdLets utilize Internet Explorer to parse HTML content. Since non-Windows systems do not have Internet Explorer, PowerShell Core utilizes the `BasicHtmlWebResponseObject` which does not contain this property.

		This RFC proposes that the creation of a new CmdLet named `ConvertFrom-Html`. This CmdLet is to be implemented into PowerShell Core and should utilize the [AngelSharp](https://github.com/AngleSharp/AngleSharp) framework for converting HTML strings into a PSCustomObject.

Conversation

Josh Rickard (MSAdministrator) commented Jul 15, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Mark Kraus (markekraus) commented Jul 15, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Josh Rickard (MSAdministrator) commented Jul 17, 2018

Uh oh!

bviktor commented Aug 28, 2018

Uh oh!

bviktor Aug 28, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Steve Lee (SteveL-MSFT) commented Oct 6, 2018

Uh oh!

bviktor commented Jun 3, 2019

Uh oh!

Fotis Gimian (fgimian) commented Jun 17, 2019

Uh oh!

Rain Sallow (/u/ta11ow) (vexx32) commented Jun 17, 2019

Uh oh!

Fotis Gimian (fgimian) commented Jun 18, 2019

Uh oh!

Joey Aiello (joeyaiello) commented Jul 8, 2019

Uh oh!

Fotis Gimian (fgimian) commented Jul 9, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Ilya (iSazonov) commented Jul 9, 2019

Uh oh!

Akhil Thomas (akhilthomas011) commented Oct 26, 2019

Uh oh!

Joey Aiello (joeyaiello) commented Mar 10, 2020

Uh oh!

Joey Aiello (joeyaiello) commented Mar 10, 2020

Uh oh!

Ilya (iSazonov) commented Mar 11, 2020

Uh oh!

Miodrag Milić (majkinetor) commented Aug 14, 2020

Uh oh!

Reviewers

Assignees

bviktor Aug 28, 2018 •

edited

Loading

Fotis Gimian (fgimian) commented Jul 9, 2019 •

edited

Loading