Raku LibCurl

Simple Examples Options Header Options Special Options Errors Info Received headers Content Proxies Multi

A Raku interface to libcurl.

libcurl is a free and easy-to-use client-side URL transfer
library, supporting DICT, FILE, FTP, FTPS, Gopher, HTTP, HTTPS,
IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP,
SMTP, SMTPS, Telnet and TFTP. libcurl supports SSL certificates,
HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload,
proxies, cookies, user+password authentication (Basic, Digest,
NTLM, Negotiate, Kerberos), file transfer resume, http proxy
tunneling and more!

libcurl is highly portable, it builds and works identically on
numerous platforms, including Solaris, NetBSD, FreeBSD, OpenBSD,
Darwin, HPUX, IRIX, AIX, Tru64, Linux, UnixWare, HURD, Windows,
Amiga, OS/2, BeOs, Mac OS X, Ultrix, QNX, OpenVMS, RISC OS, Novell
NetWare, DOS and more...

libcurl is free, thread-safe, IPv6 compatible, feature rich, well
supported, fast, thoroughly documented and is already used by many
known, big and successful companies and numerous applications.

Simple Examples

use LibCurl::Easy;

# GET
print LibCurl::Easy.new(URL => 'http://example.com').perform.content;

# GET (download a file)
LibCurl::Easy.new(URL => 'http://example.com/somefile',
                  download => 'somefile').perform;

# HEAD
say LibCurl::Easy.new(:nobody, URL => 'http://example.com')
    .perform.response-code;

# PUT (upload a file)
LibCurl::Easy.new(URL => 'http://example.com/somefile',
                  upload => 'somefile').perform;

# PUT (content from a string)
LibCurl::Easy.new(URL => 'http://example.com/somefile',
                  send => 'My Content').perform;

 # DELETE
LibCurl::Easy.new(URL => 'http://example.com/file-to-delete',
                  customrequest => 'DELETE').perform;

# POST
LibCurl::Easy.new(URL => 'http://example.com/form.html',
                  postfields => 'name=foo&opt=value').perform;

LibCurl::HTTP

If even those aren't easy enough, there is a tiny sub-class LibCurl::HTTP that adds aliases for the common HTTP methods:

use LibCurl::HTTP;

my $http = LibCurl::HTTP.new;

say $http.GET('http://example.com').perform.content;

say $http.GET('http://example.com', 'myfile').perform.response-code;

say $http.HEAD('http://example.com').perform.response-code;

$http.DELETE('http://example.com').perform;

$http.PUT('http://example.com', 'myfile').perform;

$http.POST('http://example.com/form.html', 'name=foo&opt=value').perform;

LibCurl::HTTP methods also enable failonerror by default, so any HTTP response >= 400 will throw an error.

You can even import very simple subroutines ala WWW:

use LibCurl::HTTP :subs;

# Just GET content (will return Failure on failure):
say get 'https://httpbin.org/get?foo=42&bar=x';

# GET and decode received data as JSON:
say jget('https://httpbin.org/get?foo=42&bar=x')<args><foo>;

# POST content (query args are OK; pass form as named args)
say post 'https://httpbin.org/post?foo=42&bar=x', :some<form>, :42args;

# And if you need headers, pass them inside a positional Hash:
say post 'https://httpbin.org/post?foo=42&bar=x', %(:Some<Custom-Header>),
    :some<form>, :42args;

# Same POST as above + decode response as JSON
say jpost('https://httpbin.org/post', :some<form-arg>)<form><some>;

Fancier Example

my $curl = LibCurl::Easy.new(:verbose, :followlocation);
$curl.setopt(URL => 'http://example.com', download => './myfile.html');
$curl.perform;
say $curl.success;
say $curl.Content-Type;
say $curl.Content-Length;
say $curl.Date;
say $curl.response-code;
say $curl.statusline;

Of course the full power of libcurl is available, so you aren't limited to HTTP URLs, you can use FTP, SMTP, TFTP, SCP, SFTP, and many, many more.

Options

Many of the libcurl options are available, mostly just skip the CURLOPT_, lowercase, and use '-' instead of '_'. For example, instead of CURLOPT_ACCEPT_ENCODING, use accept-encoding. When the options are really boolean (set to 1 to enable and 0 to disable), you can treat them like option booleans if you want, :option to enable, and :!option to disable.

Just like libcurl, the primary option is URL, and can take many forms depending on the desired protocol.

Options can be set on a handle either on initial creation with new(), or later with .setopt(). As a convenience, you can also just treat them like normal object methods. These are all equivalent:

my $curl = LibCurl::Easy.new(:verbose, :followlocation,
                             URL => 'http://example.com');

$curl.setopt(:verbose, followlocation => 1);
$curl.setopt(URL => 'http://example.com');

$curl.verbose(1);
$curl.followlocation(1);
$curl.URL('http://example.com');

postfields is actually CURLOPT_COPYPOSTFIELDS so it will always copy the fields.

Some of the normal options have _LARGE versions. LibCurl always maps the option to the _LARGE option where they exist, so you don't have to worry about overflows.

These are the current options (If you want one not in this list, let me know):

CAinfo CApath URL accepttimeout-ms accept-encoding address-scope append autoreferer buffersize certinfo cookie cookiefile cookiejar cookielist customrequest default-protocol dirlistonly failonerror followlocation forbid-reuse fresh-connect ftp-skip-pasv-ip ftp-use-eprt ftp-use-epsv ftpport header http-version httpauth httpget httpproxytunnel ignore-content-length) infilesize localport localportrange low-speed-limit low-speed-time maxconnects maxfilesize maxredirs max-send-speed max-recv-speed netrc nobody noprogress nosignal password path-as-is post postfields postfieldsize protocols proxy proxyauth proxyport proxytype proxyuserpwd proxy-ssl-verifypeer proxy-ssl-verifyhost range redir-protocols referer request-target resume-from ssl-verifyhost ssl-verifypeer tcp-keepalive tcp-keepidle tcp-keepintvl timecondition timeout timeout-ms timevalue unix-socket-path unrestricted-auth use-ssl useragent username userpwd verbose wildcardmatch xoauth2_bearer

Header Options

In addition to the normal libcurl special options that set headers (useragent, referer, cookie), there are some extra options for headers:

Content-MD5, Content-Type, Content-Length, Host, Accept, Expect, Transfer-Encoding.

$curl.Host('somewhere.com');  # or $curl.setopt(Host => 'somewhere.com')
$curl.Content-MD5('...');     # or $curl.setopt(Content-MD5 => '...')

You can also add any other headers you like:

$curl.set-header(X-My-Header => 'something', X-something => 'foo');

You can clear a standard header by setting the header to '', or send a header without content by setting the content to ';'

$curl.set-header(Accept => '');      # Don't send normal Accept header
$curl.set-header(Something => ';');  # Send empty header

If you are reusing the handle, you can also clear the set headers:

$curl.clear-header();

This only clears the 'extra' headers, not useragent/referer/cookie.

Special Options

In addition to the normal libcurl options, Raku LibCurl uses options for some special Raku functionality.

debugfunction replaces the libcurl CURLOPT_DEBUGFUNCTION callback, with one that looks like this:

sub debug(LibCurl::Easy $easy, CURL-INFO-TYPE $type, Buf $buf)
{...}

$curl.setopt(debugfunction => &debug);

xferinfo replaces the libcurl CURLOPT_XFERINFOFUNCTION (and CURLOPT_PROGRESSFUNCTION) with one that looks like this:

sub xferinfo(LibCurl::Easy $easy, $dltotal, $dlnow, $ultotal, $ulnow)
{...}

$curl.setopt(xferinfofunction => &xferinfo);

download specifies a filename to download into.

upload specifies a filename to upload from.

send specifies a Str or a Buf to send content from.

Finally there is a private option which replaces CURLOPT_PRIVATE, and you can safely store any object in it.

Errors

In most circumstances, errors from libcurl functions will result in a thrown X::LibCurl exception. You can catch these with CATCH. You can see the string error, or cast to Int to see the libcurl error code.

For HTTP transfers, you can access the response code with getinfo('response-code') or just .response-code. You can also check that the response code is in the 2xx range with .success.

You might find the failonerror option useful to force an error if the HTTP code is equal to or larger than 400. That will cause an exception in those cases.

On an error, you may find extra human readable error messages with the .error method.

$curl.perform;

CATCH {
    say "Caught!";
    when X::LibCurl {
        say "$_.Int() : $_";
        say $curl.response-code;
        say $curl.error;
    }
}

Info

After a transfer, you can retrieve internal information about the curl session with the .getinfo method.

You can explicitly request a single field, or a list of fields to get a hash, or just get all the fields as a hash. As in the options, there are also convenience methods for each info field.

say $curl.getinfo('effective-url');
say $curl.getinfo('response-code');

say $curl.getinfo(<effective-url response-code>);  # Hash with those keys

say $curl.getinfo;   # Hash of all info fields

say $curl.effective-url;
say $curl.response-code;

Fields currently defined are:

appconnect_time certinfo condition-unmet connect-time content-type cookielist effective-url ftp-entry-path header-size http-connectcode httpauth-avail lastsocket local-ip local-port namelookup-time num-connects os-errno pretransfer-time primary-ip primary-port proxyauth-avail redirect-url request-size response-code rtsp-client-cseq rtsp-cseq-recv rtsp-server-cseq rtsp-session-id size-download size-upload speed-download speed-upload ssl-engines total-time

Received header fields

You can retrieve the header fields in several ways as well.

say $curl.receiveheaders<Content-Length>;  # Hash of all headers

say $curl.get-header('Content-Length');

say $curl.Content-Length;

Content

If you did not specify the download option to download content into a file, the content will be stashed in memory in a Buf object you can access with the .buf() method.

If you understand that the content is decodable as a string, you can call the .content($encoding = 'utf-8') method which will decode the content into a Str, by default with the utf-8 encoding if not specified.

say "Got content", $curl.content;

Multi-part forms

Forms now uses the libcurl MIME capability, but requires verison 7.56 or greater for forms.

There is a special POST option for multipart/formdata.

my $curl = LibCurl::Easy.new(URL => 'http://...');

# normal field
$curl.formadd(name => 'fieldname', contents => 'something');

# Use a file as content, but not as a file upload
$curl.formadd(name => 'fieldname', filecontent => 'afile.txt');

# upload a file from disk, give optional filename or contenttype
$curl.formadd(name => 'fieldname', file => 'afile.txt',
              filename => 'alternate.name.txt',
              contenttype => 'image/jpeg');

# Send a Blob of contents, but as a file with a filename
$curl.formadd(name => 'fieldname', filename => 'some.file.name.txt',
              contents => "something".encode);

$curl.perform;

This will automatically cause LibCurl to POST the data.

Proxies

libcurl has great proxy support, and you should be able to specify anything needed as options to LibCurl to use them. The easiest for most common cases is to just set the proxy option.

By default, libcurl will also respect the environment variables http_proxy, ftp_proxy, all_proxy, etc. if any of those are set. Setting the proxy string to "" (an empty string) will explicitly disable the use of a proxy, even if there is an environment variable set for it.

A proxy host string can also include protocol scheme (http://) and embedded user + password.

Unix sockets

LibCurl can be used to communicate with a unix socket by setting the unix-socket-path option. You must still specify a host, but it is ignored. For example, you could use the docker REST API like this:

use LibCurl::Easy;
use JSON::Fast;

my $docker = LibCurl::Easy.new(unix-socket-path => '/var/run/docker.sock');

my $info = from-json($docker.URL("http://localhost/info").perform.content);
say $info<KernelVersion>;
say $info<OperatingSystem>;

Streaming

There is an experimental stream capability which can bind a Channel to the data stream. Instead of retrieving the buffered data at once, each time data comes in it is sent through to the Channel. This is useful for long-lived connections that periodically send more data.

You can access it with Something like this:

my $curl = LibCurl::Easy.new(URL => ...);

my $stream = $curl.stream-out;

start react whenever $stream -> $in {
    say "in: ", $in;
}

$curl.perform;

Suggestions for improving the interface for this capability welcome!

Multi

Raku LibCurl also supports the libcurl multi interface. You still have to construct LibCurl::Easy (or LibCurl::HTTP) handles for each transfer, but instead of calling .perform, just add the handle to a LibCurl::Multi.

use LibCurl::Easy;
use LibCurl::Multi;

my $curl1 = LibCurl::Easy.new(:verbose, :followlocation,
                     URL => 'http://example.com',
                     download => './myfile1.html');

my $curl2 = LibCurl::Easy.new(:verbose, :followlocation,
                     URL => 'http://example.com',
                     download => './myfile2.html');

my $multi = LibCurl:Multi.new;

$multi.add-handle($curl1);
$multi.add-handle($curl2);

$multi.perform;

say $curl1.statusline;
say $curl2.statusline;

You can also use an asynchronous callback to get a notification when an individual transfer has completed. The callback takes place in the same thread with all the transfers, so it should complete quickly (or start a new thread for heavy lifting as needed). You can add additional handles to the LibCurl::Multi at any time, even re-using completed LibCurl::Easy handles (after setting URL, etc. as needed).

use LibCurl::Easy;
use LibCurl::Multi;

my $curl1 = LibCurl::Easy.new(:followlocation,
                              URL => 'http://example.com',
                              download => 'myfile1.html');

my $curl2 = LibCurl::Easy.new(:followlocation,
                              URL => 'http://example.com',
                              download => 'myfile2.html');

sub callback(LibCurl::Easy $easy, Exception $e)
{
    die $e if $e;
    say $easy.response-code;
    say $easy.statusline;
}

my $multi = LibCurl::Multi.new(callback => &callback);

$multi.add-handle($curl1, $curl2);

$multi.perform;

INSTALL

LibCurl depends on libcurl, so you must install that prior to installing this module. It may already be installed on your system.

For debian or ubuntu: apt-get install libcurl4-openssl-dev
For alpine: apk add libcurl
For CentOS: yum install libcurl

LICENSE

Copyright © 2017 United States Government as represented by the Administrator of the National Aeronautics and Space Administration. No copyright is claimed in the United States under Title 17, U.S.Code. All Other Rights Reserved.

See NASA Open Source Agreement for more details.