Data::Importers
In brief
This repository is for a Raku package for the ingestion of different types of data
from both URLs and files. Automatically deduces the data type from extensions.
Remark: The built-in sub slurp
is overloaded by definitions of this package.
The corresponding function data-import
can be also used.
The format of the data of the URLs or files can be specified with the named argument "format".
If format => Whatever
then the format of the data is implied by the extension of the given URL or file name.
(Currently) the recognized formats are: CSV, HTML, JSON, Image (png, jpeg, jpg), PDF, Plaintext, Text, XML.
The functions slurp
and data-import
can work with:
Remark: Since "Text::CSV" is a "heavy" to install package, it is not included in the dependencies of this one.
Remark: Similarly, "PDF::Extract" requires additional, non-Raku installation, and it targets only macOS (currently.)
That is why it is not included in the dependencies of "Data::Importers".
Installation
From Zef' ecosystem:
zef install Data::Importers
From GitHub:
zef install https://github.com/antononcube/Raku-Data-Importers.git
File examples
In order to use the slurp
definitions of this package the named argument "format"
has to be specified:
JSON file
use Data::Importers;
slurp($*CWD ~ '/resources/simple.json', format => 'json')
# {name => ingrid, value => 1}
Instead of slurp
the function data-import
can be used (no need to use "format"):
data-import($*CWD ~ '/resources/simple.json')
# {name => ingrid, value => 1}
CSV file
slurp($*CWD ~ '/resources/simple.csv', format => 'csv', headers => 'auto')
# [{X1 => 1, X2 => A, X3 => Cold} {X1 => 2, X2 => B, X3 => Warm} {X1 => 3, X2 => C, X3 => Hot}]
URLs examples
JSON URLs
Import a JSON file:
my $url = 'https://raw.githubusercontent.com/antononcube/Raku-LLM-Prompts/main/resources/prompt-stencil.json';
my $res = data-import($url, format => Whatever);
$res.WHAT;
# (Hash)
Here is the deduced type:
use Data::TypeSystem;
deduce-type($res);
# Struct([Arity, Categories, ContributedBy, Description, Keywords, Name, NamedArguments, PositionalArguments, PromptText, Topics, URL], [Int, Hash, Str, Str, Array, Str, Array, Hash, Str, Hash, Str])
Using slurp
instead of data-import
:
slurp($url)
# {Arity => 1, Categories => {Function Prompts => False, Modifier Prompts => False, Personas => False}, ContributedBy => Anton Antonov, Description => Write me!, Keywords => [], Name => Write me!, NamedArguments => [], PositionalArguments => {$a => VAL}, PromptText => -> $a='VAL' {"Something over $a."}, Topics => {AI Guidance => False, Advisor Bots => False, Character Types => False, Chats => False, Computable Output => False, Content Derived from Text => False, Education => False, Entertainment => False, Fictional Characters => False, For Fun => False, General Text Manipulation => False, Historical Figures => False, Linguistics => False, Output Formatting => False, Personalization => False, Prompt Engineering => False, Purpose Based => False, Real-World Actions => False, Roles => False, Special-Purpose Text Manipulation => False, Text Analysis => False, Text Generation => False, Text Styling => False, Wolfram Language => False, Writers => False, Writing Genres => False}, URL => None}
Image URL
Import an image:
my $imgURL = 'https://raw.githubusercontent.com/antononcube/Raku-WWW-OpenAI/main/resources/ThreeHunters.jpg';
data-import($imgURL, format => 'md-image').substr(^100)
# ![](data:image/jpeg;base64,/9j/4AAQSkZJRgABAQEASABIAAD/2wBDAAUEBAUEAwUFBAUGBgUGCA4JCAcHCBEMDQoOFBEVF
Remark: Image ingestion is delegated to
"Image::Markup::Utilities", [AAp1].
The format value 'md-image' can be used to display images in Markdown files or Jupyter notebooks.
CSV URL
Here we ingest a CSV file and show a table of a 10-rows sample:
use Data::Translators;
'https://raw.githubusercontent.com/antononcube/Raku-Data-ExampleDatasets/main/resources/dfRdatasets.csv'
==> slurp(headers => 'auto')
==> { $_.pick(10).sort({ $_<Package Item> }) }()
==> data-translation(field-names => <Package Item Title Rows Cols>)
Package | Item | Title | Rows | Cols |
---|
Ecdat | Diamond | Pricing the C's of Diamond Stones | 308 | 5 |
MASS | GAGurine | Level of GAG in Urine of Children | 314 | 2 |
Stat2Data | Caterpillars | Measurements of Manduca Sexta Caterpillars | 267 | 18 |
Stat2Data | FaithfulFaces | Faithfulness from a Photo? | 170 | 7 |
nycflights13 | flights | Flights data | 336776 | 19 |
openintro | coast_starlight | Coast Starlight Amtrak train | 16 | 3 |
openintro | hsb2 | High School and Beyond survey | 200 | 11 |
robustbase | wood | Modified Data on Wood Specific Gravity | 20 | 6 |
stevedata | anes_partytherms | Major Party (Democrat, Republican) Thermometer Index Data (1978-2012) | 33830 | 19 |
stevedata | usa_computers | Percentage of U.S. Households with Computer Access, by Year | 19 | 2 |
PDF URL
Here is an example of importing a PDF file into plain text:
my $txt = slurp('https://pdfobject.com/pdf/sample.pdf', format=>'text');
say text-stats($txt);
# (chars => 2851 words => 416 lines => 38)
Remark: The function text-stats
is provided by this package, "Data::Importers".
Here is a sample of the imported text:
$txt.lines[^6].join("\n")
# Sample PDF
# This is a simple PDF file. Fun fun fun.
# Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Phasellus facilisis odio sed mi.
# Curabitur suscipit. Nullam vel nisi. Etiam semper ipsum ut lectus. Proin aliquam, erat eget
# pharetra commodo, eros mi condimentum quam, sed commodo justo quam ut velit.
# Integer a erat. Cras laoreet ligula cursus enim. Aenean scelerisque velit et tellus.
TODO
- DONE Development
- TODO Unit tests
- TODO PDF ingestion
- Some initial tests are put in.
References
[AAp1] Anton Antonov,
Image::Markup::Utilities Raku package,
(2023),
GitHub/antononcube.
[HMBp1] H. Merijn Brand,
Text::CSV Raku package,
(2015-2023),
GitHub/Tux.
[SRp1] Steve Roe,
PDF::Extract Raku package,
(2023),
GitHub/librasteve.