Here's a solution convert HTML tables to PSObjects, which you can then pipe to Export-CSV or do whatever you need to.
Please note: this is not a clean solution; it just about does the job for simple scenarios, but has a lot of issues:
- Can't cope with special characters (other than
, to get it to work you'll need to add new definitions to the DocType's entity map as required)
- Can't cope with
colspan or rowspan; assumes that all tables have the same number of columns in every row as they had in the header (there's a tweak to prevent errors if there's more columns than headers; but you may still get misalignment in that scenario).
- My technique for cleaning the HTML before converting to XML was to use a regex rather than a parsing library; so there could well be unexpected issues there.
function ConvertFrom-HtmlTableRow {
[CmdletBinding()]
param (
[Parameter(Mandatory = $true, ValueFromPipeline = $true)]
$htmlTableRow
,
[Parameter(Mandatory = $false, ValueFromPipeline = $false)]
$headers
,
[Parameter(Mandatory = $false, ValueFromPipeline = $false)]
[switch]$isHeader
)
process {
$cols = $htmlTableRow | select -expandproperty td
if($isHeader.IsPresent) {
0..($cols.Count - 1) | %{$x=$cols[$_] | out-string; if(($x) -and ($x.Trim() -gt [string]::Empty)) {$x} else {("Column_{0:0000}" -f $_)}} #clean the headers to ensure each col has a name
} else {
$colCount = ($cols | Measure-Object).Count - 1
$result = new-object -TypeName PSObject
0..$colCount | %{
$colName = if($headers[$_]){$headers[$_]}else{("Column_{0:00000} -f $_")} #in case we have more columns than headers
$colValue = $cols[$_]
$result | Add-Member NoteProperty $colName $colValue
}
write-output $result
}
}
}
function ConvertFrom-HtmlTable {
[CmdletBinding()]
param (
[Parameter(Mandatory = $true, ValueFromPipeline = $true)]
$htmlTable
)
process {
#currently only very basic <table><tr><td>...</td></tr></table> structure supported
#could be improved to better understand tbody, th, nested tables, etc
#$htmlTable.childNodes | ?{ $_.tagName -eq 'tr' } | ConvertFrom-HtmlTableRow
#remove anything tags that aren't td or tr (simplifies our parsing of the data
[xml]$cleanedHtml = ("<!DOCTYPE doctypeName [<!ENTITY nbsp ' '>]><root>{0}</root>" -f ($htmlTable | select -ExpandProperty innerHTML | %{(($_ | out-string) -replace '(</?t[rdh])[^>]*(/?>)|(?:<[^>]*>)','$1$2') -replace '(</?)(?:th)([^>]*/?>)','$1td$2'}))
[string[]]$headers = $cleanedHtml.root.tr | select -first 1 | ConvertFrom-HtmlTableRow -isHeader
if ($headers.Count -gt 0) {
$cleanedHtml.root.tr | select -skip 1 | ConvertFrom-HtmlTableRow -Headers $headers | select $headers
}
}
}
clear-host
[System.Uri]$url = 'https://en.wikipedia.org/wiki/List_of_countries_by_carbon_dioxide_emissions'
$rqst = Invoke-WebRequest $url
$rqst.ParsedHtml.getElementsByTagName('table') | ConvertFrom-HtmlTable
FYI: I've also published an earlier version of this code on CodeReview, so check there to see if anyone suggests any good improvements.
Set-Contentonly creates a text file with the extension .xml; it's not necessarily a real xml file. I guess you're just writing to an xml file to do the conversion; for that you can simply do it by converting the string to xml;[xml]$myXml = @($data.ParsedHtml.getElementsByTagName("table"))[0].OuterHTML | out-string. NB: This will not be robust; if you're not using XHTML there's no guarentee that the HTML will obey XML's syntax rules (e.g. HTML5 has void elements, so there<hr>should not have a closing tag; whilst in XML it must be iether<hr />or<hr></hr>.