NAME

HTTP::UserAgent - Web user agent class

SYNOPSIS

use HTTP::UserAgent;

my $ua = HTTP::UserAgent.new;
$ua.timeout = 10;

my $response = $ua.get("URL");

if $response.is-success {
    say $response.content;
}
else {
    die $response.status-line;
}

DESCRIPTION

This module provides functionality to crawling the web witha handling cookies and correct User-Agent value.

It has TLS/SSL support.

METHODS

method new

method new(HTTP::UserAgent:U: :$!useragent, Bool :$!throw-exceptions, :$!max-redirects = 5, :$!debug)

Default constructor.

There are four optional named arguments:

useragent

A string that specifies what will be provided in the User-Agent header in the request. A number of standard user agents are described in HTTP::UserAgent::Common, but a string that is not specified there will be used verbatim.

throw-exceptions

By default the request method will not throw an exception if the response from the server indicates that the request was unsuccesful, in this case you should check is-success to determine the status of the HTTP::Response returned. If this is specified then an exception will be thrown if the request was not a success, however you can still retrieve the response from the response attribute of the exception object.

max-redirects

This is the maximum number of redirects allowed for a single request, if this is exceeded then an exception will be thrown (this is not covered by no-exceptions above and will always be throw,) the default value is 5.

debug

It can etheir be a Bool like simply :debug or you can pass it a IO::Handle or a file name. Eg :debug($*ERR) will ouput on stderr :debug("mylog.txt") will ouput on the file.

method auth

method auth(HTTP::UserAgent:, Str $login, Str $password)

Sets username and password needed to HTTP Auth.

method get

multi method get(Str $url is copy, :bin?, *%headers)
multi method get(URI $uri, :bin?, *%headers)

Requests the $url site, returns HTTP::Response, except if throw-exceptions is set as described above whereby an exception will be thrown if the response indicates that the request wasn't successfull.

If the Content-Type of the response indicates that the content is text the content of the Response will be a decoded string, otherwise it will be left as a Blob.

If the ':bin' adverb is supplied this will force the response content to always be an undecoded Blob

Any additional named arguments will be applied as headers in the request.

method post

multi method post(URI $uri, %form, *%header ) -> HTTP::Response
multi method post(Str $uri, %form, *%header ) -> HTTP::Response

Make a POST request to the specified uri, with the provided Hash of %form data in the body encoded as "application/x-www-form-urlencoded" content. Any additional named style arguments will be applied as headers in the request.

An HTTP::Response will be returned, except if throw-exceptions has been set and the response indicates the request was not successfull.

If the Content-Type of the response indicates that the content is text the content of the Response will be a decoded string, otherwise it will be left as a Blob.

If the ':bin' adverb is supplied this will force the response content to always be an undecoded Blob

If greater control over the content of the request is required you should create an HTTP::Request directly and populate it as needed,

method request

method request(HTTP::Request $request, :bin?)

Performs the request described by the supplied HTTP::Request, returns a HTTP::Response, except if throw-exceptions is set as described above whereby an exception will be thrown if the response indicates that the request wasn't successful.

If the response has a 'Content-Encoding' header that indicates that the content was compressed, then it will attempt to inflate the data using Compress::Zlib, if the module is not installed then an exception will be thrown. If you do not have or do not want to install Compress::Zlib then you should be able to send an 'Accept-Encoding' header with a value of 'identity' which should cause a well behaved server to send the content verbatim if it is able to.

If the Content-Type of the response indicates that the content is text the content of the Response will be a decoded string, otherwise it will be left as a Blob. The content-types that are always considered to be binary (and thus left as a Blob ) are those with the major-types of 'image','audio' and 'video', certain 'application' types are considered to be 'text' (e.g. 'xml', 'javascript', 'json').

If the ':bin' adverb is supplied this will force the response content to always be an undecoded Blob

You can use the helper subroutines defined in HTTP::Request::Common to create the HTTP::Request for you or create it yourself if you have more complex requirements.

routine get :simple

sub get(Str $url) returns Str is export(:simple)

Like method get, but returns decoded content of the response.

routine head :simple

sub head(Str $url) returns Parcel is export(:simple)

Returns values of following header fields:

Content-Type
Content-Length
Last-Modified
Expires
Server

routine getstore :simple

sub getstore(Str $url, Str $file) is export(:simple)

Like routine get but writes the content to a file.

routine getprint :simple

sub getprint(Str $url) is export(:simple)

Like routine get but prints the content and returns the response code.

SUPPORT MODULES

This module encapsulates single HTTP Cookie.

use HTTP::Cookie;

my $cookie = HTTP::Cookie.new(:name<test_name>, :value<test_value>);
say ~$cookie;

The following methods are provided:

method new

my $c = HTTP::Cookie.new(:name<a_cookie>, :value<a_value>, :secure, fields => (a => b));

A constructor, it takes these named arguments:

key	description
name	name of a cookie
value	value of a cookie
secure	Secure param
httponly	HttpOnly param
fields	list of field Pairs (field => value)

method Str

Returns a cookie as a string in readable (RFC2109) form.

This module provides a bunch of methods to manage HTTP cookies.

use HTTP::Cookies;
my $cookies = HTTP::Cookies.new(
  :file<./cookies>,
  :autosave
);
$cookies.load;

method new

my $cookies = HTTP::Cookies.new(
  :file<./cookies.here>
  :autosave,
);

Constructor, takes named arguments:

key	description
file	where to write cookies
autosave	save automatically after every operation on cookies or not

my $cookies = HTTP::Cookies.new;
$cookies.set-cookie('Set-Cookie: name1=value1; HttpOnly');

Adds a cookie (passed as an argument $str of type Str) to the list of cookies.

method save

my $cookies = HTTP::Cookies.new;
$cookies.set-cookie('Set-Cookie: name1=value1; HttpOnly');
$cookies.save;

Saves cookies to the file ($.file).

method load

my $cookies = HTTP::Cookies.new;
$cookies.load;

Loads cookies from file specified at instantiation ($.file).

method extract-cookies

my $cookies = HTTP::Cookies.new;
my $response = HTTP::Response.new(Set-Cookie => "name1=value; Secure");
$cookies.extract-cookies($response);

Gets cookies ('Set-Cookie: ' lines) from the HTTP Response and adds it to the list of cookies.

my $cookies = HTTP::Cookies.new;
my $request = HTTP::Request.new;
$cookies.load;
$cookies.add-cookie-header($request);

Adds cookies fields ('Cookie: ' lines) to the HTTP Request.

method clear-expired

my $cookies = HTTP::Cookies.new;
$cookies.set-cookie('Set-Cookie: name1=value1; Secure');
$cookies.set-cookie('Set-Cookie: name2=value2; Expires=Wed, 09 Jun 2021 10:18:14 GMT');
$cookies.clear-expired; # contains 'name1' cookie only

Removes expired cookies.

method clear

my $cookies = HTTP::Cookies.new;
$cookies.load;   # contains something
$cookies.clear;  # will be empty after this action

Removes all cookies.

my $c = HTTP::Cookie.new(:name<a>, :value<b>, :httponly);
my $cookies = HTTP::Cookies.new;
$cookies.push-cookie: $c;

Pushes cookies (passed as an argument $c of type HTTP::Cookie) to the list of cookies.

method Str

Returns all cookies in human (and server) readable form.

HTTP::UserAgent::Common - the most commonly used User-Agents

This module provides a list of the most commonly used User-Agents.

use HTTP::UserAgent::Common;
say get-ua('chrome_linux');

routine get-ua

say get-ua('chrome_linux');

Returns correct UserAgent or unchaged passed argument if UserAgent could not be found.

Available UserAgents:

chrome_w7_64 firefox_w7_64 ie_w7_64 chrome_w81_64 firefox_w81_64 mob_safari_osx
safari_osx chrome_osx firefox_linux chrome_linux

HTTP::Header - class encapsulating HTTP message header

This module provides a class with a set of methods making us able to easily handle HTTP message headers.

use HTTP::Header;
my $h = HTTP::Header.new;
$h.field(Accept => 'text/plain');
say $h.field('Accept');
$h.remove-field('Accept');

method new

my $head = HTTP::Header.new(:h1<v1>, :h2<v2>);

A constructor. Takes name => value pairs as arguments.

method header

my $head = HTTP::Header.new(:h1<v1>, :h2<v2>);
say $head.header('h1');

my $head = HTTP::Header.new(:h1<v1>, :h2<v2>);
$head.header(:h3<v3>);

Gets/sets header field.

method init-field

my $head = HTTP::Header.new;
$head.header(:h1<v1>);
$head.init-header(:h1<v2>, :h2<v2>); # it doesn't change the value of 'h1'
say ~$head;

Initializes a header field: adds a field only if it does not exist yet.

method push-header

my $head = HTTP::Header.new;
$head.push-header( HTTP::Header::Field.new(:name<n1>, :value<v1>) );
say ~$head;

Pushes a new field. Does not check if exists.

method remove-header

my $head = HTTP::Header.new;
$head.header(:h1<v1>);
$head.remove-header('h1');

Removes a field of name $field.

method header-field-names

my $head = HTTP::Header.new(:h1<v1>, :h2<v2>);
my @names = $head.header-field-names;
say @names; # h1, h2

Returns a list of names of all fields.

method clear

my $head = HTTP::Header.new(:h1<v1>, :h2<v2>);
$head.clear;

Removes all fields.

method Str

Returns readable form of the whole header section.

method parse

my $head = HTTP::Header.new.parse("h1: v1\r\nh2: v2\r\n");
say $head.raku;

Parses the whole header section.

HTTP::Header::Field

This module provides a class encapsulating HTTP Message header field.

use HTTP::Header::Field;
my $header = HTTP::Header::Field.new(:name<Date>, values => (123, 456));

method new

Constructor. Takes these named arguments:

key	description
name	name of a header field
values	array of values of a header field

method Str

Stringifies an HTTP::Header::Field object. Returns a header field in a human (and server) readable form.

HTTP::Request - class encapsulating HTTP request message

Module provides functionality to easily manage HTTP requests.

use HTTP::Request;
my $request = HTTP::Request.new(GET => 'http://www.example.com/');

method new

A constructor, the first form takes parameters like:

method => URL, where method can be POST, GET ... etc.
field => values, header fields

my $req = HTTP::Request.new(:GET<example.com>, :h1<v1>);

The second form takes the key arguments as simple positional parameters and is designed for use in places where for example the request method may be calculated and the headers pre-populated.

method set-method

my $req = HTTP::Request.new;
$req.set-method: 'POST';

Sets a method of the request.

method uri

my $req = HTTP::Request.new;
$req.uri: 'example.com';

Sets URL to request.

method add-cookies

method add-cookies(HTTP::Cookies $cookies)

This will cause the appropriate cookie headers to be added from the supplied HTTP::Cookies object.

method add-form-data

multi method add-form-data(%data, :$multipart)
multi method add-form-data(:$multipart, *%data);
multi method add-form-data(Array $data, :$multipart)

Adds the form data, supplied either as a Hash, an Array of Pairs, or in a named parameter style, to the POST request (it doesn't make sense on most other request types).

The default is to use 'application/x-www-form-urlencoded' and 'multipart/form-data' can be used by providing the ':multipart' named argument. Alternatively a previously applied "content-type" header of either 'application/x-www-form-urlencoded' or 'multipart/form-data' will be respected and in the latter case any applied boundary marker will be retained.

As a special case for multipart data if the value for some key in the data is an Array of at least one item then it is taken to be a description of a file to be "uploaded" where the first item is the path to the file to be inserted, the second (optional) an alternative name to be used in the content disposition header and the third an optional Array of Pairs that will provide additional header lines for the part.

method Str

Returns stringified object.

method parse

method parse(Str $raw_request --> HTTP::Request:D)

Parses raw HTTP request. See HTTP::Message

HTTP::Request::Common - Construct common HTTP::Request objects

use HTTP::Request::Common;

my $ua       = HTTP::UserAgent.new;
my $response = $ua.request(GET 'http://google.com/');

This module provide functions that return newly created HTTP::Request objects. These functions are usually more convenient to use than the standard HTTP::Request constructor for the most common requests. The following functions are provided:

GET $url, Header => Value...

The GET function returns an HTTP::Request object initialized with the "GET" method and the specified URL.

HEAD $url, Header => Value,...

Like GET but the method in the request is "HEAD".

DELETE $url, Header => Value,...

Like GET but the method in the request is "DELETE".

`PUT $url, Header =` Value,..., content => $content>

Like GET but the method in the request is "PUT".

HTTP::Response - class encapsulating HTTP response message

use HTTP::Response;
my $response = HTTP::Response.new(200);
say $response.is-success; # it is

Module provides functionality to easily manage HTTP responses.

Response object is returned by the .get() method of HTTP::UserAgent.

method new

my $response = HTTP::Response.new(200, :h1<v1>);

A constructor, takes named arguments:

key	description
code	code of the response
fields	header fields (field_name => values)

method is-success

my $response = HTTP::Response.new(200);
say 'YAY' if $response.is-success;

Returns True if response is successful (status == 2xx), False otherwise.

method set-code

my $response = HTTP::Response.new;
$response.set-code: 200;

Sets code of the response.

method Str

Returns stringified object.

method parse

See HTTP::Message.

HTTP::Message - class encapsulating HTTP message

use HTTP::Message;
my $raw_msg = "GET / HTTP/1.1\r\nHost: somehost\r\n\r\n";
my $mess = HTTP::Message.new.parse($raw_msg);
say $mess;

This module provides a bunch of methods to easily manage HTTP message.

method new

my $msg = HTTP::Message.new('content', :field<value>);

A constructor, takes these named arguments:

key	description
content	content of the message (optional)
fields	fields of the header section

method add-content

my $msg = HTTP::Message.new('content', :field<value>);
$msg.add-content: 's';
say $msg.content; # says 'contents'

Adds HTTP message content. It does not remove the existing value, it concats to the existing content.

method decoded-content

my $msg = HTTP::Message.new();
say $msg.decoded-content;

Returns decoded content of the message (using Encode module to decode).

method field

See HTTP::Header.

method init-field

See HTTP::Header.

method push-field

See HTTP::Header.

method remove-field

See HTTP::Header.

method clear

my $msg = HTTP::Message.new('content', :field<value>);
$msg.clear;
say ~$msg; # says nothing

Removes the whole message, both header and content section.

method parse

my $msg = HTTP::Message.new.parse("GET / HTTP/1.1\r\nHost: example\r\ncontent\r\n");
say $msg.raku;

Parses the whole HTTP message.

It takes the HTTP message (with \r\n as a line separator) and obtains the header and content sections, creates a HTTP::Header object.

method Str

Returns HTTP message in a readable form.

AUTHOR

Filip Sergot

Source can be located at: https://github.com/raku-community-modules/HTTP-UserAgent . Comments and Pull Requests are welcome.

COPYRIGHT AND LICENSE

This library is free software; you can redistribute it and/or modify it under the MIT License.

NAME

SYNOPSIS

DESCRIPTION

METHODS

method new

method auth

method get

method post

method request

routine get :simple

routine head :simple

routine getstore :simple

routine getprint :simple

SUPPORT MODULES

HTTP::Cookie - HTTP cookie class

method new

method Str

HTTP::Cookies - HTTP cookie jars

method new

method set-cookie

method save

method load

method extract-cookies

method add-cookie-header

method clear-expired

method clear

method push-cookie

method Str

HTTP::UserAgent::Common - the most commonly used User-Agents

routine get-ua

HTTP::Header - class encapsulating HTTP message header

method new

method header

method init-field

method push-header

method remove-header

method header-field-names

method clear

method Str

method parse

HTTP::Header::Field

method new

method Str

HTTP::Request - class encapsulating HTTP request message

method new

method set-method

method uri

method add-cookies

method add-form-data

method Str

method parse

HTTP::Request::Common - Construct common HTTP::Request objects

GET $url, Header => Value...

HEAD $url, Header => Value,...

DELETE $url, Header => Value,...

PUT $url, Header = Value,..., content => $content>

HTTP::Response - class encapsulating HTTP response message

method new

method is-success

method set-code

method Str

method parse

HTTP::Message - class encapsulating HTTP message

method new

method add-content

method decoded-content

method field

method init-field

method push-field

method remove-field

method clear

method parse

method Str

AUTHOR

COPYRIGHT AND LICENSE

`PUT $url, Header =` Value,..., content => $content>