Have you ever wanted to browse the web via the command line? Yea. Me neither. But have you ever needed to pull information from a webpage, monitor a website or submit information via automation? I have and I use
Invoke-WebRequest to do it!
Invoke-WebRequest PowerShell cmdlet is the swiss army knife for the web. This cmdlet can send any HTTP verb to a web service along with common things like HTTP parameters, specify different HTTP headers and so on.
Invoke-WebRequest along with it's brother,
Invite-RestMethod are the two PowerShell cmdlets you'll want to familiarize yourself with if you need to do any kind of web automation.
Invoke-WebRequest cmdlet is a part of the Microsoft.PowerShell.Utility module that comes with Windows PowerShell and PowerShell Core. This cmdlet was included with PowerShell ever since v2 and it's one that is extremely powerful yet easy to use.
Invoke-WebRequest, PowerShell allows a developer to work with websites, web services and REST APIs in a lot of different ways.
At it's most basic, the
Invoke-WebRequest cmdlet sends an HTTP request method to an endpoint such as a URI or URL. The cmdlet supports all of the common request methods.
By far, the most common method is the GET method. This method reads information such as information from a website or maybe querying a REST API. The method is defined by using the
Method parameter. Since we need an endpoint to query, we'll also need a URI as well. To keep this easy, I'll pick any website. To shamelessly promote TechSnips, I'll choose techsnips.io.
Let's say I want to get a listing of all of the latest published videos as shown below.
I can get an HTML representation of this page by running
Invoke-WebRequest -Uri 'https://techsnips.io' -Method GET. When I do this,
Invoke-WebRequest downloads the entire web page and returns an output with various parsed information around the elements of the page.
To get the videos, I'll need to do some digging. When I look at the links property I see a commonality that all of the video links have a class of
ng-binding as shown below.
Once I know this, I can then find all of those elements and only return the
innerHTML property and voila!
Downloading Files with Invoke-WebRequest
We can also use
Invoke-WebRequest to download files from the web as well and it's really easy! We can download files by simply pointing
Invoke-WebRequest at a URI of a file and using the
OutFile parameter to tell the cmdlet to save the file to local disk.
As an example, below I'm downloading the SysInternals Handle utility and expanding the zip file once downloaded. It's really that easy!
Invoke-WebRequest -Uri 'https://download.sysinternals.com/files/Handle.zip' -OutFile C:\handle.zi Expand-Archive -Path C:\handle.zip
Submitting a Form and Working with Sessions
We can use
Invoke-WebRequest to also fill forms. To do this though, we commonly need to work with web sessions. HTTP is a naturally stateless protocol and your browser (in this case PowerShell) must be able to create a session which will be used to track things like cookies, for example. A common form is a login/password form so let's login to a fictional website!
Let's say our fictional login form is at the URL http://somewebsite.com. We'd first need to run
Invoke-WebRequest to download the HTML structure and create a session.
$response = Invoke-WebRequest -Uri 'http://somewebsite.com' -SessionVariable rb
Once we do this, the response will have a
Forms property we can then populate with a username and password. In this case, the username is represented by a field called
user and the password should be in a field called
password. This will depend on the webpage.
$form = $response.Forms $form.Fields["user"] = "username" $form.Fields["password"] = "password"
Once the form has been populated, we can then use
Invoke-WebRequest again but this time re-use the session we just created and automatically figure out the URI to send it to by reading the
Action property that's on the form as shown below.
$response = Invoke-WebRequest -Uri $form.Action -WebSession $rb -Method POST
If you've got all of the appropriate field names right and the webpage isn't doing any fancy, you should be logged in with the username and password inside of the
$rb web session. At this point, you can read various pages behind that authentication if you use the
$rb web session variable.
Resolving Short URIs
Finally, another great use of
Invoke-WebRequest is resolving short URIs. Perhaps you need to know what's behind that shortened URL but don't want to click on it to find out! No problem. Using
Invoke-WebRequest, we can read the
AbsoluteUri property from the parsed response it gives us!
Notice below I'm also using the
UseBasicParsing parameter. By default,
Invoke-WebRequest tries to use Internet Explorer (IE) to parse the HTML returned. This doesn't work on systems without IE. To get around that, we can use the
UseBasicParsing parameter to still download the content but only lightly parse it.
$Url = 'buff.ly/2sWvPOH' $Web = Invoke-WebRequest -Uri $Url -UseBasicParsing $Web.BaseResponse.ResponseUri.AbsoluteUri
Invoke-WebRequest cmdlet is one of the most versatile cmdlets that come with PowerShell. �If there's an action that can be performed via a typical graphical browser, the
Invoke-WebRequest cmdlet can do it too. You can find some examples of using this cmdlet by taking a look at this AWS article and one that I did on monitoring REST APIs.
Join the Jar Tippers on Patreon
It takes a lot of time to write detailed blog posts like this one. In a single-income family, this blog is one way I depend on to keep the lights on. I'd be eternally grateful if you could become a Patreon patron today!Become a Patron!
Subscribe to Adam the Automator
Get the latest posts delivered right to your inbox