Extract Domains from List of URLs using Powershell

Question

I have a list of thousands of URLs contained in a CSV file. They're full length, including folders, variables, etc. I'd like to produce a list of the domains for further analysis. I found this relevant SO thread, but I'm new to Powershell and I'm not seeing how to iterate this over the lines of a CSV.

My CSV has only one column:

http://something.net/prod/case_studies.asp
http://www.another.com/prod/group/gold/price_guarantee.asp
http://www.goodsite.co.uk/prod/case.asp?utm_source=google&utm_medium=search
http://wheel.net/prod/studious.asp
http://www.buystrop.com/gap/index.php?page_id=2345&group_id=9876

Here's what I'd like to end up with:

something.net
www.another.com
www.goodsite.co.uk
wheel.net
www.buystrop.com

I've tried several iterations of the code I found in the other thread, but nothing has worked yet. Either I get errors, or the output is blank.

Edit: Here's the code that I've tried so far:

$file = Get-Content 'file.csv'
$domains = ForEach ($p in $file) {select ([System.Uri]$p).Host}

That gives no error, the code runs but $domains is empty.

$domains = ForEach-Object {$file | select-object [System.Uri]$file.Host}
$domains = ForEach-Object [System.Uri]$file.Host

These both give an error indicating that it's looking at the entire file, not looping through each line. I've now tried a few dozen variations on these, I haven't yet figured out how parse the URL in each line.

The accepted answer to the question you linked has the pieces you need. Post the code you're attempting and where you're having trouble, and we can help you with it. — alroc
– alroc, Commented Sep 25, 2013 at 0:32
In your edit, you've misspelled the last $domains as $domians. — Anthony Neace
– Anthony Neace, Commented Sep 25, 2013 at 0:50
Thanks for the tip Hyper Anthony. I got excited when I thought it was something this simple, but alas, that's not it. Still trying... — Lenwood
– Lenwood, Commented Sep 25, 2013 at 0:53

Goyuix · Accepted Answer · 2013-09-25 03:00:20Z

Here is an answer that builds on what alroc showed, but adds making the list of hosts unique:

$domains = (gc file.csv)|%{(New-Object System.Uri $_).Host)|select -unique

Breaking this down:

$domains is the variable that is going to contain the list of unique hosts
gc is an alias for Get-Content - it reads in the file line by line
the | (pipe) character reads each line from gc and passes it to the next expression
% is an alias for ForEach-Object, and $_ is a special variable that holds the current value that each is being passed along the pipeline (one line of text from file.csv in this case)
New-Object creates a new instance of the Uri class with the URL from file.csv
The parenthesis () wrap the new object, allowing the property dereference operator . to be used to access the Host property of the Uri object. According to the MSDN docs this property contains just the "domain" part of the URL as you call it.
The resulting hosts are then piped to the select cmdlet (alias for Select-Object) that has a switch -unique that as you may guess, filters the values coming through it to be just the unique/distinct values.

I hope this helps!

alroc · Accepted Answer · 2013-09-25 01:05:39Z

2

You do need to loop through each item that you read from the file (which you're not doing in the last 2 examples; you're kind of doing it backwards).

$hosts = Get-Content file.csv;
$hosts|ForEach-Object {$url = New-Object System.Uri $_;$url.Host};

answered Sep 25, 2013 at 1:05

alroc

28.3k6 gold badges55 silver badges106 bronze badges

Collectives™ on Stack Overflow

Extract Domains from List of URLs using Powershell

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related