Rand Stats




In brief

This repository is for a Raku package for the ingestion of different types of data from both URLs and files. Automatically deduces the data type from extensions.

Remark: The built-in sub slurp is overloaded by definitions of this package. The corresponding function data-import can be also used.

The format of the data of the URLs or files can be specified with the named argument "format". If format => Whatever then the format of the data is implied by the extension of the given URL or file name.

(Currently) the recognized formats are: CSV, HTML, JSON, Image (png, jpeg, jpg), PDF, Plaintext, Text, XML.

The functions slurp and data-import can work with:

Remark: Since "Text::CSV" is a "heavy" to install package, it is not included in the dependencies of this one.

Remark: Similarly, "PDF::Extract" requires additional, non-Raku installation, and it targets only macOS (currently.) That is why it is not included in the dependencies of "Data::Importers".


From Zef' ecosystem:

zef install Data::Importers

From GitHub:

zef install https://github.com/antononcube/Raku-Data-Importers.git

File examples

In order to use the slurp definitions of this package the named argument "format" has to be specified:

JSON file

use Data::Importers;

slurp($*CWD ~ '/resources/simple.json', format => 'json')
# {name => ingrid, value => 1}

Instead of slurp the function data-import can be used (no need to use "format"):

data-import($*CWD ~ '/resources/simple.json')
# {name => ingrid, value => 1}

CSV file

slurp($*CWD ~ '/resources/simple.csv', format => 'csv', headers => 'auto')
# [{X1 => 1, X2 => A, X3 => Cold} {X1 => 2, X2 => B, X3 => Warm} {X1 => 3, X2 => C, X3 => Hot}]

URLs examples


Import a JSON file:

my $url = 'https://raw.githubusercontent.com/antononcube/Raku-LLM-Prompts/main/resources/prompt-stencil.json';

my $res = data-import($url, format => Whatever);

# (Hash)

Here is the deduced type:

use Data::TypeSystem;

# Struct([Arity, Categories, ContributedBy, Description, Keywords, Name, NamedArguments, PositionalArguments, PromptText, Topics, URL], [Int, Hash, Str, Str, Array, Str, Array, Hash, Str, Hash, Str])

Using slurp instead of data-import:

# {Arity => 1, Categories => {Function Prompts => False, Modifier Prompts => False, Personas => False}, ContributedBy => Anton Antonov, Description => Write me!, Keywords => [], Name => Write me!, NamedArguments => [], PositionalArguments => {$a => VAL}, PromptText => -> $a='VAL' {"Something over $a."}, Topics => {AI Guidance => False, Advisor Bots => False, Character Types => False, Chats => False, Computable Output => False, Content Derived from Text => False, Education => False, Entertainment => False, Fictional Characters => False, For Fun => False, General Text Manipulation => False, Historical Figures => False, Linguistics => False, Output Formatting => False, Personalization => False, Prompt Engineering => False, Purpose Based => False, Real-World Actions => False, Roles => False, Special-Purpose Text Manipulation => False, Text Analysis => False, Text Generation => False, Text Styling => False, Wolfram Language => False, Writers => False, Writing Genres => False}, URL => None}

Image URL

Import an image:

my $imgURL = 'https://raw.githubusercontent.com/antononcube/Raku-WWW-OpenAI/main/resources/ThreeHunters.jpg';

data-import($imgURL, format => 'md-image').substr(^100)

Remark: Image ingestion is delegated to "Image::Markup::Utilities", [AAp1]. The format value 'md-image' can be used to display images in Markdown files or Jupyter notebooks.


Here we ingest a CSV file and show a table of a 10-rows sample:

use Data::Translators;

==> slurp(headers => 'auto') 
==> { $_.pick(10).sort({ $_<Package Item> }) }()
==> data-translation(field-names => <Package Item Title Rows Cols>)
EcdatDiamondPricing the C's of Diamond Stones3085
MASSGAGurineLevel of GAG in Urine of Children3142
Stat2DataCaterpillarsMeasurements of Manduca Sexta Caterpillars26718
Stat2DataFaithfulFacesFaithfulness from a Photo?1707
nycflights13flightsFlights data33677619
openintrocoast_starlightCoast Starlight Amtrak train163
openintrohsb2High School and Beyond survey20011
robustbasewoodModified Data on Wood Specific Gravity206
stevedataanes_partythermsMajor Party (Democrat, Republican) Thermometer Index Data (1978-2012)3383019
stevedatausa_computersPercentage of U.S. Households with Computer Access, by Year192


Here is an example of importing a PDF file into plain text:

my $txt = slurp('https://pdfobject.com/pdf/sample.pdf', format=>'text');

say text-stats($txt);
# (chars => 2851 words => 416 lines => 38)

Remark: The function text-stats is provided by this package, "Data::Importers".

Here is a sample of the imported text:

# Sample PDF
# This is a simple PDF file. Fun fun fun.
# Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Phasellus facilisis odio sed mi.
# Curabitur suscipit. Nullam vel nisi. Etiam semper ipsum ut lectus. Proin aliquam, erat eget
# pharetra commodo, eros mi condimentum quam, sed commodo justo quam ut velit.
# Integer a erat. Cras laoreet ligula cursus enim. Aenean scelerisque velit et tellus.



[AAp1] Anton Antonov, Image::Markup::Utilities Raku package, (2023), GitHub/antononcube.

[HMBp1] H. Merijn Brand, Text::CSV Raku package, (2015-2023), GitHub/Tux.

[SRp1] Steve Roe, PDF::Extract Raku package, (2023), GitHub/librasteve.