Rand Stats

LLM::Functions

zef:antononcube

LLM::Functions

MacOS Linux Win64

In brief

This Raku package provides functions and function objects to access, interact, and utilize Large Language Models (LLMs), like OpenAI, [OAI1], PaLM, [ZG1], and MistralAI, [MAI1].

For more details how the concrete LLMs are accessed see the packages "WWW::OpenAI", [AAp2], "WWW::PaLM", [AAp3], and "WWW::MistralAI", [AAp9].

The LLM functions built by this package can have evaluators that use "sub-parsers" -- see "Text::SubParsers", [AAp4].

The primary motivation to have handy, configurable functions for utilizing LLMs came from my work on the packages "ML::FindTextualAnswer", [AAp6], and "ML::NLPTemplateEngine", [AAp7].

A very similar system of functionalities is developed by Wolfram Research Inc.; see the paclet "LLMFunctions", [WRIp1].

For well curated and instructive examples of LLM prompts see the Wolfram Prompt Repository. Many of those prompts (≈220) are available in Raku and Python -- see "LLM::Prompts", [AAp8], and "LLMPrompts", [AAp10], respectively.

The article "Generating documents via templates and LLMs", [AA1], shows an alternative way of streamlining LLMs usage. (Via Markdown, Org-mode, or Pod6 templates.)


Installation

Package installations from both sources use zef installer (which should be bundled with the "standard" Rakudo installation file.)

To install the package from Zef ecosystem use the shell command:

zef install LLM::Functions

To install the package from the GitHub repository use the shell command:

zef install https://github.com/antononcube/Raku-LLM-Functions.git

Design

"Out of the box" "LLM::Functions" uses "WWW::OpenAI", [AAp2], "WWW::PaLM", [AAp3], and "WWW::MistralAI", [AAp9]. Other LLM access packages can be utilized via appropriate LLM configurations.

Configurations:

New LLM functions are constructed with the function llm-function.

The function llm-function:

Here is a sequence diagram that follows the steps of a typical creation procedure of LLM configuration- and evaluator objects, and the corresponding LLM-function that utilizes them:

sequenceDiagram
  participant User
  participant llmfunc as llm-function
  participant llmconf as llm-configuration
  participant LLMConf as LLM configuration
  participant LLMEval as LLM evaluator
  participant AnonFunc as Anonymous function
  User ->> llmfunc: ・prompt<br>・conf spec
  llmfunc ->> llmconf: conf spec
  llmconf ->> LLMConf: conf spec
  LLMConf ->> LLMEval: wrap with
  LLMEval ->> llmfunc: evaluator object
  llmfunc ->> AnonFunc:  create with:<br>・evaluator object<br>・prompt
  AnonFunc ->> llmfunc: handle
  llmfunc ->> User: handle

Here is a sequence diagram for making a LLM configuration with a global (engineered) prompt, and using that configuration to generate a chat message response:

sequenceDiagram
  participant WWWOpenAI as WWW::OpenAI
  participant User
  participant llmfunc as llm-function
  participant llmconf as llm-configuration
  participant LLMConf as LLM configuration
  participant LLMChatEval as LLM chat evaluator
  participant AnonFunc as Anonymous function
  User ->> llmconf: engineered prompt
  llmconf ->> User: configuration object
  User ->> llmfunc: ・prompt<br>・configuration object
  llmfunc ->> LLMChatEval: configuration object
  LLMChatEval ->> llmfunc: evaluator object
  llmfunc ->> AnonFunc: create with:<br>・evaluator object<br>・prompt
  AnonFunc ->> llmfunc: handle
  llmfunc ->> User: handle
  User ->> AnonFunc: invoke with<br>message argument
  AnonFunc ->> WWWOpenAI: ・engineered prompt<br>・message
  WWWOpenAI ->> User: LLM response 

Configurations

OpenAI-based

Here is the default, OpenAI-based configuration:

use LLM::Functions;
.raku.say for llm-configuration('OpenAI').Hash;
# :prompts($[])
# :tool-request-parser(WhateverCode)
# :evaluator(Whatever)
# :max-tokens(300)
# :total-probability-cutoff(0.03)
# :module("WWW::OpenAI")
# :temperature(0.8)
# :format("values")
# :prompt-delimiter(" ")
# :argument-renames(${:api-key("auth-key"), :stop-tokens("stop")})
# :function(proto sub OpenAITextCompletion ($prompt is copy, :$model is copy = Whatever, :$suffix is copy = Whatever, :$max-tokens is copy = Whatever, :$temperature is copy = Whatever, Numeric :$top-p = 1, Int :$n where { ... } = 1, Bool :$stream = Bool::False, Bool :$echo = Bool::False, :$stop = Whatever, Numeric :$presence-penalty = 0, Numeric :$frequency-penalty = 0, :$best-of is copy = Whatever, :api-key(:$auth-key) is copy = Whatever, Int :$timeout where { ... } = 10, :$format is copy = Whatever, Str :$method = "tiny") {*})
# :images($[])
# :stop-tokens($[])
# :tools($[])
# :model("gpt-3.5-turbo-instruct")
# :name("openai")
# :api-key(Whatever)
# :tool-response-insertion-function(WhateverCode)
# :tool-prompt("")
# :api-user-id("user:339643454824")
# :examples($[])

Here is the ChatGPT-based configuration:

.say for llm-configuration('ChatGPT').Hash;
# max-tokens => 300
# model => gpt-3.5-turbo
# prompt-delimiter =>  
# api-user-id => user:117326557463
# api-key => (Whatever)
# tools => []
# prompts => []
# evaluator => (my \LLM::Functions::EvaluatorChat_4573845173584 = LLM::Functions::EvaluatorChat.new(context => "", examples => Whatever, user-role => "user", assitant-role => "assistant", system-role => "system", conf => LLM::Functions::Configuration.new(name => "chatgpt", api-key => Whatever, api-user-id => "user:117326557463", module => "WWW::OpenAI", model => "gpt-3.5-turbo", function => proto sub OpenAIChatCompletion ($prompt is copy, :$role is copy = Whatever, :$model is copy = Whatever, :$temperature is copy = Whatever, :$max-tokens is copy = Whatever, Numeric :$top-p = 1, Int :$n where { ... } = 1, Bool :$stream = Bool::False, :$stop = Whatever, Numeric :$presence-penalty = 0, Numeric :$frequency-penalty = 0, :@images is copy = Empty, :api-key(:$auth-key) is copy = Whatever, Int :$timeout where { ... } = 10, :$format is copy = Whatever, Str :$method = "tiny") {*}, temperature => 0.8, total-probability-cutoff => 0.03, max-tokens => 300, format => "values", prompts => [], prompt-delimiter => " ", examples => [], stop-tokens => [], tools => [], tool-prompt => "", tool-request-parser => WhateverCode, tool-response-insertion-function => WhateverCode, images => [], argument-renames => {:api-key("auth-key"), :stop-tokens("stop")}, evaluator => LLM::Functions::EvaluatorChat_4573845173584), formatron => "Str"))
# format => values
# name => chatgpt
# images => []
# module => WWW::OpenAI
# total-probability-cutoff => 0.03
# function => &OpenAIChatCompletion
# temperature => 0.8
# tool-prompt => 
# stop-tokens => []
# argument-renames => {api-key => auth-key, stop-tokens => stop}
# tool-response-insertion-function => (WhateverCode)
# tool-request-parser => (WhateverCode)
# examples => []

Remark: llm-configuration(Whatever) is equivalent to llm-configuration('OpenAI').

Remark: Both the "OpenAI" and "ChatGPT" configuration use functions of the package "WWW::OpenAI", [AAp2]. The "OpenAI" configuration is for text-completions; the "ChatGPT" configuration is for chat-completions.

PaLM-based

Here is the default PaLM configuration:

.say for llm-configuration('PaLM').Hash;
# images => []
# tool-request-parser => (WhateverCode)
# max-tokens => 300
# evaluator => (Whatever)
# total-probability-cutoff => 0
# tools => []
# api-user-id => user:194056758649
# tool-prompt => 
# argument-renames => {api-key => auth-key, max-tokens => max-output-tokens, stop-tokens => stop-sequences}
# tool-response-insertion-function => (WhateverCode)
# prompt-delimiter =>  
# function => &PaLMGenerateText
# examples => []
# stop-tokens => []
# api-key => (Whatever)
# temperature => 0.4
# name => palm
# format => values
# module => WWW::PaLM
# prompts => []
# model => text-bison-001

Basic usage of LLM functions

Textual prompts

Here we make a LLM function with a simple (short, textual) prompt:

my &func = llm-function('Show a recipe for:');
# -> $text, *%args { #`(Block|4573948390648) ... }

Here we evaluate over a message:

say &func('greek salad');
# Sure! Here's a recipe for Greek salad:
# 
# Ingredients:
# - 4 large ripe tomatoes, diced
# - 1 large cucumber, diced
# - 1 small red onion, thinly sliced
# - 1 green bell pepper, seeded and diced
# - 1 cup Kalamata olives, pitted
# - 1 cup feta cheese, crumbled
# - 1/4 cup extra virgin olive oil
# - 2 tablespoons red wine vinegar
# - 1 teaspoon dried oregano
# - Salt and black pepper to taste
# 
# Instructions:
# 1. In a large bowl, combine the diced tomatoes, cucumber, red onion, bell pepper, and Kalamata olives.
# 2. In a separate small bowl, whisk together the olive oil, red wine vinegar, dried oregano, salt, and black pepper to make the dressing.
# 3. Pour the dressing over the vegetable mixture and gently toss to combine.
# 4. Sprinkle the crumbled feta cheese over the salad.
# 5. Let the Greek salad sit at room temperature for about 10 minutes to allow the flavors to meld together.
# 6. Serve the salad as a side dish or as a main course with crusty bread.
# 
# Enjoy your delicious Greek salad!

Positional arguments

Here we make a LLM function with a function-prompt and numeric interpreter of the result:

my &func2 = llm-function(
        {"How many $^a can fit inside one $^b?"},
        form => Numeric,
        llm-evaluator => 'palm');
# -> **@args, *%args { #`(Block|4573979050352) ... }

Here were we apply the function:

my $res2 = &func2("tennis balls", "toyota corolla 2010");
# 48

Here we show that we got a number:

$res2 ~~ Numeric
# False

Named arguments

Here the first argument is a template with two named arguments:

my &func3 = llm-function(-> :$dish, :$cuisine {"Give a recipe for $dish in the $cuisine cuisine."}, llm-evaluator => 'palm');
# -> **@args, *%args { #`(Block|4573979104488) ... }

Here is an invocation:

&func3(dish => 'salad', cuisine => 'Russian', max-tokens => 300);
# **Russian Salad**
# 
# Ingredients:
# 
# * 2 pounds (900g) red potatoes, peeled and cubed
# * 1 pound (450g) carrots, peeled and cubed
# * 1 pound (450g) celery, thinly sliced
# * 1 cup (240ml) mayonnaise
# * 1/2 cup (120ml) sour cream
# * 1/4 cup (60ml) finely chopped fresh dill
# * 1/4 cup (60ml) finely chopped fresh parsley
# * Salt and pepper to taste
# 
# Instructions:
# 
# 1. In a large bowl, combine the potatoes, carrots, and celery.
# 2. In a small bowl, whisk together the mayonnaise, sour cream, dill, parsley, salt, and pepper.
# 3. Pour the dressing over the salad and toss to coat.
# 4. Serve immediately or chill for later.
# 
# **Tips:**
# 
# * To make the potatoes ahead of time, boil them in salted water until tender, then drain and cool completely. Once cool, store the potatoes in an airtight container in the refrigerator for up to 3 days.
# * To make the salad ahead of time, assemble it and then chill for up to 24 hours before serving.
# * If you don't have fresh dill or parsley, you can use 1/2 teaspoon dried

LLM example functions

The function llm-example-function can be given a training set of examples in order to generating results according to the "laws" implied by that training set.

Here a LLM is asked to produce a generalization:

llm-example-function([ 'finger' => 'hand', 'hand' => 'arm' ])('foot')
# leg

Here is an array of training pairs is used:

'Oppenheimer' ==> (["Einstein" => "14 March 1879", "Pauli" => "April 25, 1900"] ==> llm-example-function)()
# April 22, 1904

Here is defined a LLM function for translating WL associations into Python dictionaries:

my &fea = llm-example-function( '<| A->3, 4->K1 |>' => '{ A:3, 4:K1 }');
&fea('<| 23->3, G->33, T -> R5|>');
# { 23:3, G:33, T:R5 }

The function llm-example-function takes as a first argument:

Remark: The function llm-example-function is implemented with llm-function and suitable prompt.

Here is an example of using hints:

my &fec = llm-example-function(
        ["crocodile" => "grasshopper", "fox" => "cardinal"],
        hint => 'animal colors');

say &fec('raccoon');
# pigeon

Using predefined prompts

Using predefined prompts of the package "LLM::Prompts", [AAp8], can be very convenient in certain (many) cases.

Here is an example using "Fixed That For You" synthesis:

use LLM::Prompts;

llm-synthesize([llm-prompt('FTFY'), 'Wha is ther population?'])
# What is the population?

Using chat-global prompts

The configuration objects can be given prompts that influence the LLM responses "globally" throughout the whole chat. (See the second sequence diagram above.)

For detailed examples see the documents:


Chat objects

Here we create chat object that uses OpenAI's ChatGPT:

my $prompt = 'You are a gem expert and you give concise answers.';
my $chat = llm-chat(chat-id => 'gem-expert-talk', conf => 'ChatGPT', :$prompt);
# LLM::Functions::Chat(chat-id = gem-expert-talk, llm-evaluator.conf.name = chatgpt, messages.elems = 0)
$chat.eval('What is the most transparent gem?');
# The most transparent gem is generally considered to be diamond.
$chat.eval('Ok. What are the second and third most transparent gems?');
# The second most transparent gem is typically considered to be sapphire, while the third most transparent gem is often emerald.

Here are the prompt(s) and all messages of the chat object:

$chat.say
# Chat: gem-expert-talk
# ⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺
# Prompts: You are a gem expert and you give concise answers.
# ⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺
# role	user
# content	What is the most transparent gem?
# timestamp	2024-01-08T23:18:39.839978-05:00
# ⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺
# role	assistant
# content	The most transparent gem is generally considered to be diamond.
# timestamp	2024-01-08T23:18:40.748679-05:00
# ⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺
# role	user
# content	Ok. What are the second and third most transparent gems?
# timestamp	2024-01-08T23:18:40.771349-05:00
# ⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺
# role	assistant
# content	The second most transparent gem is typically considered to be sapphire, while the third most transparent gem is often emerald.
# timestamp	2024-01-08T23:18:41.893223-05:00

Potential problems

With PaLM with certain wrong configuration we get the error:

error => {code => 400, message => Messages must alternate between authors., status => INVALID_ARGUMENT}

TODO


References

Articles

[AA1] Anton Antonov, "Generating documents via templates and LLMs", (2023), RakuForPrediction at WordPress.

[ZG1] Zoubin Ghahramani, "Introducing PaLM 2", (2023), Google Official Blog on AI.

Repositories, sites

[MAI1] MistralAI team, MistralAI platform.

[OAI1] OpenAI team, OpenAI platform.

[WRIr1] Wolfram Research, Inc. Wolfram Prompt Repository.

Packages, paclets

[AAp1] Anton Antonov, LLM::Functions Raku package, (2023), GitHub/antononcube.

[AAp2] Anton Antonov, WWW::OpenAI Raku package, (2023), GitHub/antononcube.

[AAp3] Anton Antonov, WWW::PaLM Raku package, (2023), GitHub/antononcube.

[AAp4] Anton Antonov, Text::SubParsers Raku package, (2023), GitHub/antononcube.

[AAp5] Anton Antonov, Text::CodeProcessing Raku package, (2021), GitHub/antononcube.

[AAp6] Anton Antonov, ML::FindTextualAnswer Raku package, (2023), GitHub/antononcube.

[AAp7] Anton Antonov, ML::NLPTemplateEngine Raku package, (2023), GitHub/antononcube.

[AAp8] Anton Antonov, LLM::Prompts Raku package, (2023), GitHub/antononcube.

[AAp9] Anton Antonov, WWW::MistralAI Raku package, (2023), GitHub/antononcube.

[AAp10] Anton Antonov, LLMPrompts Python package, (2023), PyPI.org/antononcube.

[WRIp1] Wolfram Research, Inc. LLMFunctions paclet, (2023), Wolfram Language Paclet Repository.