Package {edgemodelr}


Type: Package
Title: Local Large Language Model Inference Engine
Version: 0.7.0
Description: Enables R users to run large language models locally using 'GGUF' model files and the 'llama.cpp' inference engine. Provides a complete R interface for loading models, generating text completions, extracting vector representations, and streaming responses in real-time. Includes grammar-constrained generation for structured output, text classification, and retrieval-augmented generation (RAG) pipelines. Models can also be served to many users at once as an HTTP API compatible with the 'OpenAI' chat completions API. Supports local inference without requiring cloud APIs or internet connectivity, ensuring complete data privacy and control. Based on the 'llama.cpp' project by Georgi Gerganov (2023) https://github.com/ggml-org/llama.cpp.
License: MIT + file LICENSE
URL: https://github.com/PawanRamaMali/edgemodelr
BugReports: https://github.com/PawanRamaMali/edgemodelr/issues
Encoding: UTF-8
Depends: R (≥ 4.0)
LinkingTo: Rcpp
Imports: Rcpp (≥ 1.0.0), parallel, utils, tools
Suggests: testthat (≥ 3.1.7), knitr, rmarkdown, curl, shiny, openssl, jsonlite, httr, plumber, promises, later, DBI, RSQLite, callr, ellmer, httpuv
SystemRequirements: GNU make or equivalent for building
Note: Package includes a self-contained 'llama.cpp' implementation for complete functionality without external dependencies.
Config/testthat/edition: 3
RoxygenNote: 7.3.3
NeedsCompilation: yes
Packaged: 2026-11-16 15:31:42 UTC; alpine
Author: Pawan Rama Mali [aut, cre, cph], Georgi Gerganov [aut, cph] (Author of llama.cpp and GGML library), The ggml authors [cph] (llama.cpp and GGML contributors), Jeffrey Quesnelle [ctb, cph] (YaRN RoPE implementation), Bowen Peng [ctb, cph] (YaRN RoPE implementation), pi6am [ctb] (DRY sampler from Koboldcpp), Ivan Yurchenko [ctb] (Z-algorithm implementation), Dirk Eddelbuettel [ctb, rev]
Maintainer: Pawan Rama Mali <prm@outlook.in>
Repository: CRAN
Date/Publication: 2026-11-17 07:50:37 UTC

edgemodelr: Local Large Language Model Inference Engine

Description

Enables R users to run large language models locally using 'GGUF' model files and the 'llama.cpp' inference engine. Provides a complete R interface for loading models, generating text completions, and streaming responses in real-time. Supports local inference without requiring cloud APIs or internet connectivity, ensuring complete data privacy and control. Based on the 'llama.cpp' project by Georgi Gerganov (2023) https://github.com/ggml-org/llama.cpp.

Details

The edgemodelr package provides R bindings for Local Large Language Model Inference Engine using llama.cpp and GGUF model files. This enables completely private, on-device text generation without requiring cloud APIs or internet connectivity.

Main Functions

edge_load_model

Load a GGUF model file

edge_completion

Generate text completions

edge_stream_completion

Stream text generation in real-time

edge_chat_stream

Interactive chat interface

edge_quick_setup

One-line model download and setup

edge_free_model

Release model memory

Model Management

edge_list_models

List available pre-configured models

edge_download_model

Download models from Hugging Face

Getting Started

Basic usage workflow:

  1. Download a model: setup <- edge_quick_setup("TinyLlama-1.1B")

  2. Generate text: edge_completion(setup$context, "Hello")

  3. Clean up: edge_free_model(setup$context)

For interactive chat:

setup <- edge_quick_setup("TinyLlama-1.1B")
edge_chat_stream(setup$context)

Examples

See comprehensive examples in the package:

Run examples:

# Getting started guide
source(system.file("examples/getting_started_example.R", package = "edgemodelr"))

# Data science assistant
source(system.file("examples/data_science_assistant_example.R", package = "edgemodelr"))

System Requirements

Privacy and Security

This package processes all data locally on your machine. No data is sent to external servers, ensuring complete privacy and control over your text generation workflows.

Author(s)

Pawan Rama Mali prm@outlook.in

See Also


Clean raw model output into an executable SQL statement

Description

Clean raw model output into an executable SQL statement

Usage

.clean_sql_output(raw)

Compute SHA-256 hash for a file (best-effort, no new hard dependencies)

Description

Compute SHA-256 hash for a file (best-effort, no new hard dependencies)

Usage

.compute_sha256(path)

Arguments

path

Path to the file

Value

Lowercase hex string or NA if computation not available


Download using curl command

Description

Download using curl command

Usage

.download_with_curl(
  url,
  destfile,
  hf_token = "",
  verbose = TRUE,
  max_retries = 3
)

Download using httr package (best for corporate networks)

Description

Download using httr package (best for corporate networks)

Usage

.download_with_httr(url, destfile, hf_token = "", verbose = TRUE)

Download using R's download.file with libcurl

Description

Download using R's download.file with libcurl

Usage

.download_with_r(url, destfile, hf_token = "", verbose = TRUE, max_retries = 3)

Download using wget command

Description

Download using wget command

Usage

.download_with_wget(
  url,
  destfile,
  hf_token = "",
  verbose = TRUE,
  max_retries = 3
)

Check if a file is a valid GGUF file

Description

Check if a file is a valid GGUF file

Usage

.is_valid_gguf_file(path)

Arguments

path

Path to the file

Value

TRUE if valid GGUF, FALSE otherwise


Check directory writeability

Description

Check directory writeability

Usage

.is_writable_dir(path)

Known hashes cache helpers (Trust-On-First-Use optional)

Description

Known hashes cache helpers (Trust-On-First-Use optional)

Usage

.known_hashes_path(cache_dir)

Linearise a record into a "field: value" block

Description

Linearise a record into a "field: value" block

Usage

.linearise_records(data)

Robust file download with retry and resume support

Description

Robust file download with retry and resume support

Usage

.robust_download(url, destfile, verbose = TRUE, max_retries = 3)

Arguments

url

URL to download

destfile

Destination file path

verbose

Print progress messages

max_retries

Maximum number of retry attempts

Value

TRUE if successful, FALSE otherwise


Build chat prompt from conversation history

Description

Build a formatted chat prompt from conversation history. When a model context is provided, uses the model's native chat template (e.g., ChatML for Qwen, Llama format for Llama models) for best results.

Usage

build_chat_prompt(history, ctx = NULL)

Arguments

history

List of conversation turns, each with $role and $content

ctx

Optional model context from edge_load_model(). If provided, uses the model's native chat template from GGUF metadata. If NULL, uses generic ChatML format.

Value

Formatted prompt string suitable for passing to edge_completion()


Chat with a local model through ellmer

Description

Returns an ellmer chat object backed by a local GGUF model, so the model works with ellmer's chat, streaming, tool calling and structured data functions. With model_path, the model is served by edge_serve in a background R process, which keeps running until the R session ends and is reused by later calls with the same model. With base_url, the chat connects to a server you started yourself.

Usage

chat_edge(
  model_path = NULL,
  system_prompt = NULL,
  base_url = NULL,
  n_ctx = 4096L,
  n_gpu_layers = 0L,
  port = NULL,
  params = NULL,
  echo = "none",
  embeddings = FALSE
)

Arguments

model_path

Path to a .gguf model file. Either this or base_url must be given.

system_prompt

Optional system prompt for the chat.

base_url

URL of a running edge_serve server, such as "http://127.0.0.1:8080".

n_ctx

Context size for a server started here (default: 4096).

n_gpu_layers

GPU layers for a server started here (default: 0).

port

Port for a server started here (default: a free one).

params

Optional ellmer::params() with sampling settings such as temperature.

echo

Passed to ellmer::chat_openai_compatible().

embeddings

Also serve POST /v1/embeddings from a server started here (default: FALSE), so the same server can embed texts, for example with ragnar::embed_openai(base_url = ). The model then keeps an output for every prompt token, which slows chat and takes more memory, and a chat model gives weaker embeddings than an embedding model. For embeddings inside R, embed_edge with an embedding model needs no server.

Details

Tool calling goes through edge_chat_completion: tool arguments may be strings, numbers, integers, booleans or enums, and the model makes one call per reply. Structured data works for flat objects of those types. Streamed replies arrive in one piece when generation ends (see edge_serve).

Value

An ellmer Chat object.

See Also

edge_serve

Examples

## Not run: 
chat <- chat_edge("gemma-3-1b-it-Q4_K_M.gguf",
                  system_prompt = "Answer in one sentence.")
chat$chat("What is the capital of France?")

# Tool calling with ellmer
get_weather <- ellmer::tool(
  function(city) "24 C and sunny",
  description = "Get the current weather for a city",
  arguments = list(city = ellmer::type_string("City name"))
)
chat$register_tool(get_weather)
chat$chat("What is the weather in Paris?")

## End(Not run)

Build an OpenAI-compatible API for many users

Description

Returns a plumber router that serves a local GGUF model to many users at once, for Posit Connect or any server that runs plumber APIs. The model is loaded once, and an edge_engine generates for up to n_slots requests at the same time with continuous batching; more requests wait in a queue, so a busy server answers later instead of failing. Request handlers return promises, so the R process keeps accepting requests while the engine works.

Usage

edge_api(
  model_path,
  n_slots = 8L,
  n_ctx = 2048L,
  n_threads = NULL,
  max_queue = 256L,
  cache_type = "q8_0",
  embeddings = FALSE,
  api_key = NULL,
  model_name = NULL,
  n_gpu_layers = 0L
)

Arguments

model_path

Path to a .gguf model file.

n_slots

Requests generated at once (default: 8).

n_ctx

Context per slot, in tokens (default: 2048): the longest prompt plus answer one request may use.

n_threads

CPU threads (default: one per core, within the container's CPU limit). The environment variable EDGEMODELR_THREADS overrides the default.

max_queue

Requests that may wait for a slot (default: 256).

cache_type

KV cache type for keys and values (default: "q8_0"); see edge_load_model.

embeddings

FALSE (default), TRUE to also serve /v1/embeddings from a second copy of the model loaded for embeddings, or the path of a .gguf embedding model to serve instead.

api_key

Optional key. When set, every endpoint but /health needs Authorization: Bearer <key> or X-API-Key: <key>. On Posit Connect, which uses Authorization for its own keys, use X-API-Key, or leave this unset and use Connect's access settings.

model_name

Name reported as the model (default: the file name).

n_gpu_layers

GPU layers (default: 0).

Details

Endpoints:

A full queue answers 503 with Retry-After.

Sizing: each slot gets n_ctx tokens of context, so the KV cache holds n_slots * n_ctx tokens (at most 32768); cache_type = "q8_0" halves its memory compared with "f16". On CPU, more slots give more tokens per second in total until the cores are saturated, typically at about 8 slots; each request then gets slower. Inside a container the default thread count follows the container's CPU limit.

Value

A plumber router. Run it with plumber::pr_run(), or return it from entrypoint.R on Posit Connect. The engine is in attr(router, "edge_engine").

See Also

edge_serve, edge_engine

Examples

## Not run: 
api <- edge_api("gemma-3-1b-it-Q4_K_M.gguf", n_slots = 8)
plumber::pr_run(api, port = 8080)

# entrypoint.R for Posit Connect
edgemodelr::edge_api(Sys.getenv("EDGEMODELR_MODEL"), n_slots = 8,
                     api_key = Sys.getenv("EDGEMODELR_API_KEY", unset = NA))

## End(Not run)

Ask a question using retrieval-augmented generation

Description

Combines semantic search with text generation: retrieves relevant context from the index, then generates an answer grounded in that context. This is the core RAG (Retrieval-Augmented Generation) function.

Usage

edge_ask(
  ctx,
  question,
  index,
  top_k = 3L,
  n_predict = 256L,
  temperature = 0.3,
  system_prompt = NULL,
  return_context = FALSE
)

Arguments

ctx

Model context from edge_load_model()

question

The question to answer

index

An edge_index object from edge_index_documents

top_k

Number of context chunks to retrieve (default: 3)

n_predict

Maximum tokens for the answer (default: 256)

temperature

Sampling temperature (default: 0.3)

system_prompt

Optional system-level instruction for the model

return_context

If TRUE, return both answer and retrieved context (default: FALSE)

Value

If return_context = FALSE: a character string with the answer. If return_context = TRUE: a list with answer, context, and prompt.

See Also

edge_index_documents, edge_search

Examples

## Not run: 
ctx <- edge_load_model("model.gguf")
index <- edge_index_documents("./reports/", ctx)

answer <- edge_ask(ctx, "What were Q3 revenues?", index)
cat(answer)

# With context for debugging
result <- edge_ask(ctx, "What were Q3 revenues?", index,
  return_context = TRUE)
cat(result$answer)
print(result$context)

## End(Not run)

Performance benchmarking for model inference

Description

Test inference speed and throughput with the current model to measure the effectiveness of optimizations.

Usage

edge_benchmark(
  ctx,
  prompt = "The quick brown fox",
  n_predict = 50,
  iterations = 3,
  track_memory = FALSE
)

Arguments

ctx

Model context from edge_load_model()

prompt

Test prompt to use for benchmarking (default: standard test)

n_predict

Number of tokens to generate for the test

iterations

Number of test iterations to average results

track_memory

If TRUE, attempt to report peak memory usage (best-effort)

Value

List with performance metrics

Examples

## Not run: 
setup <- edge_quick_setup("TinyLlama-1.1B")
if (!is.null(setup$context)) {
  ctx <- setup$context
  perf <- edge_benchmark(ctx)
  print(perf)
  edge_free_model(ctx)
}

## End(Not run)

Cache size information

Description

Cache size information

Usage

edge_cache_info(cache_dir = NULL)

Arguments

cache_dir

Cache directory path

Value

List with total_size_mb and file_count


Generate a chat completion using the model's native template

Description

Formats messages using the model's built-in chat template (read from GGUF metadata), generates a completion, and returns only the assistant's response. This is the recommended way to do multi-turn chat, as it uses the correct template format for the specific model (e.g., ChatML for Qwen, Llama format for Llama models).

Usage

edge_chat_completion(
  ctx,
  messages,
  n_predict = 256L,
  temperature = 0.7,
  top_p = 0.95,
  top_k = 0L,
  min_p = 0,
  repeat_penalty = 1,
  seed = NULL,
  stop = NULL,
  tools = NULL,
  tool_choice = "auto"
)

Arguments

ctx

Model context from edge_load_model()

messages

A list of message objects, each with role ("system", "user", or "assistant") and content (character string).

n_predict

Maximum tokens to generate (default: 256)

temperature

Sampling temperature (default: 0.7)

top_p

Nucleus sampling threshold (default: 0.95)

top_k

Sample only from the k most likely tokens (default: 0, off). 40 is a common choice.

min_p

Drop tokens less likely than this fraction of the most likely one (default: 0, off). 0.05 is a common choice.

repeat_penalty

Penalty for tokens used in the last 64 generated tokens (default: 1, off). Values from 1.1 to 1.3 stop small models from repeating themselves.

seed

Seed for sampling (default: NULL, drawn from R's random number generator, so set.seed() makes the text reproducible). Ignored when temperature = 0.

stop

Optional character vector of stop strings. Generation ends where the first one appears; the stop string itself is not returned.

tools

Optional list of tools from edge_tool that the model may call. See Details.

tool_choice

With tools: "auto" (default) lets the model either call a tool or answer in text, "required" makes it call one of the tools, a tool's name makes it call that tool, and "none" makes it answer in text. With "auto", a call identical to the last one, whose result is already in messages, is not returned; the model answers in text instead.

Details

With tools, the tools are described in the system message and generation is constrained by a grammar, so a tool call is always valid JSON naming a known tool, with every argument present and of the right type. The model writes a call as {"tool": "name", "arguments": {...}}. Run the tool yourself, append result$message and a message with role = "tool", the tool's name and its result as content, then call again so the model can use the result. Tool results are passed to the model as user messages, since not every chat template has a tool role.

The tool call is always generated greedily; temperature and the other sampling arguments apply to the text answer.

Value

Without tools, a character string containing only the assistant's response text. With tools, a list with content (the text answer, or NULL when the model called a tool), tool_calls (a list of calls, each with name and arguments, a named list) and message (the assistant message to append to messages before the next call).

See Also

build_chat_prompt, edge_completion, edge_tool

Examples

## Not run: 
ctx <- edge_load_model("model.gguf")

# Simple chat
answer <- edge_chat_completion(ctx, list(
  list(role = "user", content = "What is R?")
))
cat(answer)

# Multi-turn with system prompt
answer <- edge_chat_completion(ctx, list(
  list(role = "system", content = "You are a helpful R tutor."),
  list(role = "user", content = "What is a data.frame?"),
  list(role = "assistant", content = "A data.frame is a table..."),
  list(role = "user", content = "How do I filter rows?")
))

# Tool calling
weather <- edge_tool("get_weather", "Get the current weather for a city",
                     list(city = "string", unit = c("celsius", "fahrenheit")))
msgs <- list(list(role = "user", content = "Is it warm in Paris?"))
res <- edge_chat_completion(ctx, msgs, tools = list(weather), temperature = 0)
if (length(res$tool_calls) > 0) {
  call <- res$tool_calls[[1]]
  call$arguments$city  # "Paris"
  msgs <- c(msgs, list(res$message,
                       list(role = "tool", name = call$name, content = "24 C, sunny")))
  res <- edge_chat_completion(ctx, msgs, tools = list(weather), temperature = 0)
  cat(res$content)
}

edge_free_model(ctx)

## End(Not run)

Interactive chat session with streaming responses

Description

Interactive chat session with streaming responses

Usage

edge_chat_stream(
  ctx,
  system_prompt = NULL,
  max_history = 10,
  n_predict = 200L,
  temperature = 0.8,
  verbose = TRUE,
  top_p = 0.95,
  top_k = 0L,
  min_p = 0,
  repeat_penalty = 1,
  seed = NULL,
  stop = NULL
)

Arguments

ctx

Model context from edge_load_model()

system_prompt

Optional system prompt to set context

max_history

Maximum conversation turns to keep in context (default: 10)

n_predict

Maximum tokens per response (default: 200)

temperature

Sampling temperature (default: 0.8)

verbose

Whether to print status messages (default: TRUE)

top_p

Top-p sampling parameter (default: 0.95)

top_k

Sample only from the k most likely tokens (default: 0, off). 40 is a common choice.

min_p

Drop tokens less likely than this fraction of the most likely one (default: 0, off). 0.05 is a common choice.

repeat_penalty

Penalty for tokens used in the last 64 generated tokens (default: 1, off). Values from 1.1 to 1.3 stop small models from repeating themselves.

seed

Seed for sampling (default: NULL, drawn from R's random number generator, so set.seed() makes the text reproducible). Ignored when temperature = 0.

stop

Optional character vector of stop strings. Generation ends where the first one appears; the stop string itself is not returned.

Value

NULL (runs interactively)

Examples

## Not run: 
setup <- edge_quick_setup("TinyLlama-1.1B")
ctx <- setup$context

if (!is.null(ctx)) {
  # Start interactive chat with streaming
  # edge_chat_stream(ctx,
  #   system_prompt = "You are a helpful R programming assistant.")

  edge_free_model(ctx)
}

## End(Not run)

Classify text into predefined categories

Description

Classify text into predefined categories using grammar-constrained generation. The grammar constraint ensures the model can only output one of the provided categories.

Usage

edge_classify(
  ctx,
  text,
  categories,
  instruction = NULL,
  temperature = 0.1
)

Arguments

ctx

Model context from edge_load_model()

text

Text to classify (character string or vector for batch)

categories

Character vector of allowed categories

instruction

Optional classification instruction

temperature

Sampling temperature (default: 0.1)

Value

Character string (single text) or character vector (batch) with the predicted category. Output is guaranteed to be one of the specified categories.

See Also

edge_extract, edge_grammar_completion

Examples

## Not run: 
ctx <- edge_load_model("model.gguf")

# Single classification
result <- edge_classify(ctx,
  "I love this product!",
  categories = c("positive", "negative", "neutral"))

# Batch classification
texts <- c("Great product!", "Terrible experience", "It was okay")
results <- edge_classify(ctx, texts,
  categories = c("positive", "negative", "neutral"))

edge_free_model(ctx)

## End(Not run)

Clean up cache directory and manage storage

Description

Remove outdated model files from the cache directory to comply with CRAN policies about actively managing cached content and keeping sizes small. The CUDA backend in the cuda subdirectory and the local checksum store are never removed.

Usage

edge_clean_cache(
  cache_dir = NULL,
  max_age_days = getOption("edgemodelr.cache_max_age_days", 30),
  max_size_mb = getOption("edgemodelr.cache_max_size_mb", 5000),
  use_lru = TRUE,
  ask = TRUE,
  verbose = TRUE
)

Arguments

cache_dir

Cache directory path (default: user cache directory)

max_age_days

Maximum age of files to keep in days (default: option edgemodelr.cache_max_age_days or 30)

max_size_mb

Maximum total cache size in MB (default: option edgemodelr.cache_max_size_mb or 5000)

use_lru

If TRUE, evict least-recently-used files when size exceeds limit

ask

Whether to ask for user confirmation before deletion (only in interactive sessions)

verbose

Whether to print status messages (default: TRUE)

Value

Invisible list of deleted files

Examples

## Not run: 
# Clean cache files older than 30 days
edge_clean_cache()

# Clean cache with custom settings
edge_clean_cache(max_age_days = 7, max_size_mb = 100)

## End(Not run)

Generate text completion using loaded model

Description

Generate text completion using loaded model

Usage

edge_completion(
  ctx,
  prompt,
  n_predict = 128L,
  temperature = 0.8,
  top_p = 0.95,
  timeout_seconds = NULL,
  top_k = 0L,
  min_p = 0,
  repeat_penalty = 1,
  seed = NULL,
  stop = NULL
)

Arguments

ctx

Model context from edge_load_model()

prompt

Input text prompt

n_predict

Maximum tokens to generate (default: 128)

temperature

Sampling temperature (default: 0.8). 0 gives greedy decoding: the same prompt always gives the same text.

top_p

Top-p sampling parameter (default: 0.95)

timeout_seconds

Optional timeout in seconds for inference

top_k

Sample only from the k most likely tokens (default: 0, off). 40 is a common choice.

min_p

Drop tokens less likely than this fraction of the most likely one (default: 0, off). 0.05 is a common choice.

repeat_penalty

Penalty for tokens used in the last 64 generated tokens (default: 1, off). Values from 1.1 to 1.3 stop small models from repeating themselves.

seed

Seed for sampling (default: NULL, drawn from R's random number generator, so set.seed() makes the text reproducible). Ignored when temperature = 0.

stop

Optional character vector of stop strings. Generation ends where the first one appears; the stop string itself is not returned.

Value

Generated text as character string

Examples

## Not run: 
# Basic completion example
setup <- edge_quick_setup("TinyLlama-1.1B")
if (!is.null(setup$context)) {
  ctx <- setup$context

  # Simple completion
  result <- edge_completion(ctx, "The capital of France is", n_predict = 50)
  cat("Result:", result, "\n")

  # Completion with custom parameters
  creative_result <- edge_completion(
    ctx,
    "Write a short poem about data science:",
    n_predict = 100,
    temperature = 0.9,
    top_p = 0.8
  )
  cat("Creative result:", creative_result, "\n")

  # Reproducible sampling, less repetition, stop at the first blank line
  edge_completion(ctx, "List three colours:", seed = 42,
                  repeat_penalty = 1.2, stop = "\n\n")

  edge_free_model(ctx)
}

## End(Not run)

Return the default path for the installed CUDA backend DLL

Description

Return the default path for the installed CUDA backend DLL

Usage

edge_cuda_default_path()

Check whether a CUDA backend is installed and active

Description

Reports the installation and activation status of the GPU (CUDA) backend. Use this to verify that GPU inference is available before loading a model with n_gpu_layers > 0.

Usage

edge_cuda_info()

Value

A named list with three elements:

installed

Logical. TRUE if a CUDA backend DLL is present in the edgemodelr cache directory.

active

Logical. TRUE if the CUDA backend has been loaded into the current R session (i.e. edge_reload_cuda() was called).

path

Character. Path to the active CUDA DLL, the installed DLL (if not yet active), or NA if no backend is found.

See Also

edge_install_cuda, edge_install_cuda_toolkit, edge_reload_cuda

Examples

## Not run: 
info <- edge_cuda_info()
if (info$active) {
  cat("GPU inference active:", info$path, "\n")
} else if (info$installed) {
  cat("CUDA backend installed but not active. Call edge_reload_cuda().\n")
} else {
  cat("No CUDA backend. Run edge_install_cuda() to enable GPU support.\n")
}

## End(Not run)

Download a GGUF model from Hugging Face

Description

Download a GGUF model from Hugging Face

Usage

edge_download_model(
  model_id,
  filename = NULL,
  cache_dir = NULL,
  force_download = FALSE,
  verify_checksum = TRUE,
  expected_sha256 = NULL,
  trust_first_use = FALSE,
  verbose = TRUE
)

Arguments

model_id

Hugging Face model identifier (e.g., "TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF"), or a friendly model name from edge_list_models() (e.g., "Qwen3-0.6B", "mistral-7b")

filename

Specific GGUF file to download. Optional when using a friendly model name.

cache_dir

Directory to store downloaded models (default: "~/.cache/edgemodelr")

force_download

Force re-download even if file exists

verify_checksum

Verify SHA-256 checksum if available (default: TRUE)

expected_sha256

Optional expected SHA-256 hash for the model file

trust_first_use

Store a local hash if no known hash exists (default: FALSE)

verbose

Whether to print download progress messages

Value

Path to the downloaded model file

Examples

## Not run: 
# Download using friendly name (recommended)
model_path <- edge_download_model("Qwen3-0.6B")
model_path <- edge_download_model("mistral-7b")

# Download using HuggingFace repo ID
model_path <- edge_download_model(
  model_id = "TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF",
  filename = "tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf"
)

## End(Not run)

Download a model from a direct URL

Description

Downloads a GGUF model file from any URL. Supports resume and validates GGUF format. This function is useful for downloading models from GPT4All CDN or other direct sources that don't require authentication.

Usage

edge_download_url(
  url,
  filename,
  cache_dir = NULL,
  force_download = FALSE,
  verify_checksum = TRUE,
  expected_sha256 = NULL,
  trust_first_use = FALSE,
  verbose = TRUE
)

Arguments

url

Direct download URL for the model

filename

Local filename to save as

cache_dir

Directory to store downloaded models (default: user cache directory)

force_download

Force re-download even if file exists

verify_checksum

Verify SHA-256 checksum if available (default: TRUE)

expected_sha256

Optional expected SHA-256 hash for the file

trust_first_use

Store a local hash if no known hash exists (default: FALSE)

verbose

Whether to print progress messages

Value

Path to the downloaded model file

Examples

## Not run: 
# Download from GPT4All CDN (large file, not run in checks)
model_path <- edge_download_url(
  url = "https://gpt4all.io/models/gguf/mistral-7b-instruct-v0.1.Q4_0.gguf",
  filename = "mistral-7b.gguf"
)

## End(Not run)

Extract text embeddings from a model

Description

Computes dense vector embeddings for one or more text inputs using the loaded model. These embeddings can be used for semantic search, clustering, similarity comparison, and as input to downstream models.

Usage

edge_embeddings(ctx, texts, normalize = TRUE)

Arguments

ctx

Model context from edge_load_model()

texts

Character vector of texts to embed

normalize

Whether to L2-normalize the embeddings (default: TRUE). Normalized embeddings allow cosine similarity to be computed as a simple dot product.

Details

For best results, use a model designed for embeddings (e.g., nomic-embed-text). However, generative models can also produce useful embeddings from their hidden states.

The embedding dimension depends on the model architecture. Use edge_model_n_embd() to query the dimension.

Value

A numeric matrix with dimensions (n_texts x n_embd), where each row is the embedding vector for the corresponding input text.

See Also

edge_similarity, edge_similarity_matrix, edge_model_n_embd

Examples

## Not run: 
ctx <- edge_load_model("model.gguf")

# Single embedding
emb <- edge_embeddings(ctx, "The cat sat on the mat")
dim(emb)  # 1 x n_embd

# Multiple embeddings
texts <- c("cats are great", "dogs are loyal", "the stock market crashed")
embs <- edge_embeddings(ctx, texts)

# Compute similarity
edge_similarity(embs[1,], embs[2,])  # high (both about pets)
edge_similarity(embs[1,], embs[3,])  # low (different topics)

edge_free_model(ctx)

## End(Not run)

Run many requests at once on one model

Description

Starts a serving engine on a loaded model: a background thread that generates for several requests at the same time with continuous batching, the way llama.cpp's own server does. Each of n_slots slots holds one request; every step decodes the next token of each running request in one batch, so on CPU total throughput grows with the number of slots while each request gets a little slower. Requests beyond the slots wait in a queue, first come first served. A slot keeps its tokens after a request ends, and a later request that starts the same way (the same system prompt, or the next turn of a chat) reuses them instead of reading them again.

Usage

edge_engine(ctx, n_slots = NULL, max_queue = 256L)

edge_engine_submit(
  engine,
  prompt,
  n_predict = 256L,
  temperature = 0.7,
  top_p = 0.95,
  top_k = 0L,
  min_p = 0,
  repeat_penalty = 1,
  seed = NULL,
  stop = NULL,
  grammar = NULL
)

edge_engine_poll(engine, ids)

edge_engine_release(engine, ids)

edge_engine_cancel(engine, ids)

edge_engine_result(engine, id)

edge_engine_generate(
  engine,
  prompts,
  n_predict = 256L,
  temperature = 0.7,
  top_p = 0.95,
  top_k = 0L,
  min_p = 0,
  repeat_penalty = 1,
  seed = NULL,
  stop = NULL,
  grammar = NULL,
  timeout = Inf
)

edge_engine_stats(engine)

edge_engine_stop(engine)

Arguments

ctx

A model loaded with edge_load_model with n_parallel of at least n_slots and embeddings = FALSE. The context is shared by the slots: each request needs room for its prompt plus n_predict tokens, and waits in the queue until there is room.

n_slots

Requests generated at once (default: the model's n_parallel).

max_queue

Requests that may wait (default: 256). A submit beyond it fails with an error of class edge_busy.

engine

An engine from edge_engine().

prompt, prompts

Prompt text, already formatted (see build_chat_prompt).

n_predict

Most tokens to generate (default: 256).

temperature, top_p, top_k, min_p, repeat_penalty, seed, stop

Sampling, as in edge_completion. temperature = 0 is greedy.

grammar

Optional GBNF grammar with a root rule, such as one from edge_json_grammar.

ids, id

Request ids from edge_engine_submit().

timeout

Seconds edge_engine_generate() waits before it cancels what is left (default: no limit).

Details

The engine runs beside R: edge_engine_submit() returns at once with a request id, and R stays free while the request is generated. edge_engine_poll() reports progress and the text so far. A finished request is kept until edge_engine_release() frees it (an unfinished one is cancelled first), so a client that polls should release each request once it has read the result. edge_engine_result() gives a promise of the finished request (for plumber and Shiny) and edge_engine_generate() submits prompts and waits for all of them; both free the requests themselves.

While the engine runs, the model belongs to it: the direct completion functions refuse to use ctx until edge_engine_stop(). Tokenizing and chat templates still work. edge_api builds an HTTP API on an engine. An internal llama.cpp error on the engine's thread cannot become an R error and ends the R process, as it would end llama.cpp's own server; on Posit Connect the process is then restarted.

Value

edge_engine(): an edge_engine object. edge_engine_submit(): an integer request id. edge_engine_poll() and edge_engine_generate(): a data frame with one row per request: id, status ("queued", "running", "done" or "unknown"), text (so far, for an unfinished request), finish ("stop", "length", "cancelled" or "error"), error, prompt_tokens, completion_tokens, cached_tokens (prompt tokens reused from a slot), queue_position, queue_seconds, first_token_seconds and total_seconds. edge_engine_result(): a promise of a list with the same fields, rejected with an error of class edge_cancelled if the request is cancelled. edge_engine_stats(): a list of counters.

See Also

edge_api, edge_map

Examples

## Not run: 
ctx <- edge_load_model("gemma-3-1b-it-Q4_K_M.gguf", n_ctx = 8192, n_parallel = 4)
engine <- edge_engine(ctx)
prompts <- vapply(c("Rome", "Paris", "Oslo", "Lima"), function(city) {
  build_chat_prompt(list(list(role = "user",
    content = paste("One fact about", city))), ctx = ctx)
}, character(1))
out <- edge_engine_generate(engine, prompts, n_predict = 60, temperature = 0)
out$text
edge_engine_stats(engine)
edge_engine_stop(engine)

## End(Not run)

Extract structured data from text

Description

High-level function that combines prompt construction with grammar-constrained generation to extract structured data from text. Returns a parsed R list.

Usage

edge_extract(
  ctx,
  text,
  schema,
  instruction = NULL,
  n_predict = 512L,
  temperature = 0.2,
  draft = FALSE
)

Arguments

ctx

Model context from edge_load_model()

text

The input text to analyze

schema

A named list defining the extraction schema (see edge_json_grammar)

instruction

Optional instruction to guide extraction (default: auto-generated)

n_predict

Maximum tokens to generate (default: 512)

temperature

Sampling temperature (default: 0.2, very low for factual extraction)

draft

If TRUE, first let the model answer in free text, then run the grammar-constrained pass with that draft in the prompt (default: FALSE). It adds a second generation pass, so it takes two to four times as long.

Details

The grammar guarantees well-formed JSON, which is where most of its accuracy gain comes from. Forcing a very small model straight into a strict format can, however, cost answer accuracy. With draft = TRUE the model first reasons about the fields without constraints and the grammar is applied only to the final answer. Published results for this "draft, then constrain" pattern show gains for 4B-14B models (e.g. micro-F1 0.644 to 0.795 for Qwen3-14B on GenSIE 2026 zero-shot extraction). It is most worth trying with models under about 4B parameters, or when plain extraction leaves fields wrong.

Value

A named list with the extracted fields, or the raw JSON string if parsing fails. Requires the jsonlite package for automatic parsing.

See Also

edge_json_grammar, edge_grammar_completion, edge_classify, edge_extract_batch

Examples

## Not run: 
ctx <- edge_load_model("model.gguf")

result <- edge_extract(ctx,
  "I absolutely love this product! Best purchase ever.",
  schema = list(
    sentiment = c("positive", "negative", "neutral"),
    confidence = "number"
  ))
# result$sentiment == "positive"
# result$confidence == 0.95

edge_free_model(ctx)

## End(Not run)

Extract structured data from multiple texts

Description

Batch version of edge_extract that processes a vector of texts and returns a data frame.

Usage

edge_extract_batch(
  ctx,
  texts,
  schema,
  instruction = NULL,
  n_predict = 512L,
  temperature = 0.2,
  progress = TRUE,
  draft = FALSE
)

Arguments

ctx

Model context from edge_load_model()

texts

Character vector of texts to process

schema

A named list defining the extraction schema (see edge_json_grammar)

instruction

Optional instruction to guide extraction

n_predict

Maximum tokens to generate per text (default: 512)

temperature

Sampling temperature (default: 0.2)

progress

Show progress messages (default: TRUE)

draft

Passed to edge_extract: draft in free text first, then apply the grammar (default: FALSE)

Value

A data frame with one row per text and columns for each schema field

See Also

edge_extract, edge_map

Examples

## Not run: 
ctx <- edge_load_model("model.gguf")

reviews <- c("Love it!", "Worst purchase ever", "It's okay I guess")
results <- edge_extract_batch(ctx, reviews,
  schema = list(
    sentiment = c("positive", "negative", "neutral"),
    confidence = "number"
  ))
# results is a data frame:
#   sentiment confidence
# 1  positive       0.95
# 2  negative       0.90
# 3   neutral       0.60

edge_free_model(ctx)

## End(Not run)

Find and prepare GGUF models for use with edgemodelr

Description

This function finds compatible GGUF model files from various sources including Ollama installations, custom directories, or any folder containing GGUF files. It tests each model for compatibility with edgemodelr and creates organized copies or links for easy access.

Usage

edge_find_gguf_models(
  source_dirs = NULL,
  target_dir = NULL,
  create_links = TRUE,
  model_pattern = NULL,
  test_compatibility = TRUE,
  min_size_mb = 50,
  verbose = TRUE
)

Arguments

source_dirs

Vector of directories to search for GGUF files. If NULL, automatically searches common locations including Ollama installation.

target_dir

Directory where to create links/copies of compatible models. If NULL, creates a "local_models" directory in the current working directory.

create_links

Logical. If TRUE (default), creates symbolic links to save disk space. If FALSE, copies the files (uses more disk space but more compatible).

model_pattern

Optional pattern to filter model files by name.

test_compatibility

Logical. If TRUE (default), tests each GGUF file for compatibility with edgemodelr before including it.

min_size_mb

Minimum file size in MB to consider (default: 50MB). Helps filter out config files and focus on actual models.

verbose

Logical. Whether to print detailed progress information.

Details

This function performs the following steps:

  1. Searches specified directories (or auto-detects common locations)

  2. Identifies GGUF format files above the minimum size threshold

  3. Optionally tests each file for compatibility with edgemodelr

  4. Creates organized symbolic links or copies in the target directory

  5. Returns detailed information about working models

The function automatically searches these locations if no source_dirs specified:

Value

List containing information about compatible models, including paths and metadata

Examples

## Not run: 
# Basic usage - auto-detect and test all GGUF models
models_info <- edge_find_gguf_models()
if (!is.null(models_info) && length(models_info$models) > 0) {
  # Load the first compatible model
  ctx <- edge_load_model(models_info$models[[1]]$path)
  result <- edge_completion(ctx, "Hello", n_predict = 20)
  edge_free_model(ctx)
}

# Search specific directories
models_info <- edge_find_gguf_models(source_dirs = c("~/Downloads", "~/models"))

# Skip compatibility testing (faster but less reliable)
models_info <- edge_find_gguf_models(test_compatibility = FALSE)

# Copy files instead of creating links
models_info <- edge_find_gguf_models(create_links = FALSE)

# Filter for specific models
models_info <- edge_find_gguf_models(model_pattern = "llama")

## End(Not run)

Find and load Ollama models

Description

Utility functions to discover and work with locally stored Ollama models. Ollama stores models as SHA-256 named blobs which are GGUF files that can be used directly with edgemodelr.

Usage

edge_find_ollama_models(
  ollama_dir = NULL,
  test_compatibility = FALSE,
  max_size_gb = 10
)

Arguments

ollama_dir

Optional path to Ollama models directory. If NULL, will auto-detect.

test_compatibility

If TRUE, test if each model can be loaded successfully

max_size_gb

Maximum model size in GB to consider (default: 10)

Value

List with ollama_path and discovered models information

Examples

## Not run: 
# Find Ollama models
ollama_info <- edge_find_ollama_models()

if (!is.null(ollama_info) && length(ollama_info$models) > 0) {
  # Use first compatible model
  model_path <- ollama_info$models[[1]]$path
  ctx <- edge_load_model(model_path)
  result <- edge_completion(ctx, "Hello", n_predict = 10)
  edge_free_model(ctx)
}

## End(Not run)

Free model context and release memory

Description

Free model context and release memory

Usage

edge_free_model(ctx)

Arguments

ctx

Model context from edge_load_model()

Value

NULL (invisibly)

Examples

## Not run: 
# Requires a downloaded model (not run in checks)
model_path <- "model.gguf"
if (file.exists(model_path)) {
  ctx <- edge_load_model(model_path)
  # ... use model ...
  edge_free_model(ctx)  # Clean up
}

## End(Not run)

Generate text constrained by a GBNF grammar

Description

Uses llama.cpp's grammar-constrained sampling to force output to conform to a GBNF grammar specification. This ensures structured, parseable output.

Usage

edge_grammar_completion(
  ctx,
  prompt,
  grammar,
  grammar_root = "root",
  n_predict = 512L,
  temperature = 0.3,
  top_p = 0.95,
  jump_forward = TRUE,
  top_k = 0L,
  min_p = 0,
  repeat_penalty = 1,
  seed = NULL,
  stop = NULL
)

Arguments

ctx

Model context from edge_load_model()

prompt

The input prompt

grammar

A GBNF grammar string defining allowed output structure

grammar_root

The root rule name in the grammar (default: "root")

n_predict

Maximum tokens to generate (default: 512)

temperature

Sampling temperature (default: 0.3, lower for structured output). 0 gives greedy decoding: the same prompt always gives the same text.

top_p

Nucleus sampling threshold (default: 0.95)

jump_forward

If TRUE (default), text that the grammar allows in only one way (JSON keys, separators, the rest of an enum value once it is unambiguous) is added without sampling, and its tokens are decoded in one batch, in fewer decode steps. The forced text is tokenized the standard way, which can differ from how the model would split it, so the output can differ slightly from jump_forward = FALSE. It always matches the grammar. Set FALSE to sample every token.

top_k

Sample only from the k most likely tokens (default: 0, off). 40 is a common choice.

min_p

Drop tokens less likely than this fraction of the most likely one (default: 0, off). 0.05 is a common choice.

repeat_penalty

Penalty for tokens used in the last 64 generated tokens (default: 1, off). Values from 1.1 to 1.3 stop small models from repeating themselves.

seed

Seed for sampling (default: NULL, drawn from R's random number generator, so set.seed() makes the text reproducible). Ignored when temperature = 0.

stop

Optional character vector of stop strings. Generation ends where the first one appears; the stop string itself is not returned.

Details

GBNF (GGML BNF) is a format for defining formal grammars that constrain model output. This is useful for generating JSON, XML, or any structured format.

Jump-forward leaves the last token of each forced stretch for the model to sample, since that token may merge with the text after it (a model writes an opening brace and quote as one token). It saves the most with long JSON keys and edge_json_grammar(compact = TRUE), where the layout between fields is fixed and so is forced too: with four long keys a 0.6B model wrote the object about 3x faster. With short keys the saving is small.

Common GBNF patterns:

Value

Character string containing only the generated text (not the prompt)

See Also

edge_json_grammar, edge_extract, edge_classify

Examples

## Not run: 
ctx <- edge_load_model("model.gguf")

# Force yes/no output
grammar <- 'root ::= "yes" | "no"'
result <- edge_grammar_completion(ctx, "Is the sky blue? Answer:", grammar)

# Force JSON output
json_grammar <- edge_json_grammar(list(
  sentiment = c("positive", "negative", "neutral"),
  confidence = "number"
))
result <- edge_grammar_completion(ctx,
  "Analyze sentiment: 'I love this product'\nJSON:",
  json_grammar)

edge_free_model(ctx)

## End(Not run)

Build an embedding index from text documents

Description

Reads text files from a directory (or accepts text directly), splits into chunks, computes embeddings, and returns an index object for semantic search and retrieval-augmented generation.

Usage

edge_index_documents(
  source,
  ctx,
  chunk_size = 500L,
  chunk_overlap = 50L,
  file_pattern = "*.txt",
  normalize = TRUE,
  progress = TRUE
)

Arguments

source

Either a directory path containing text files, or a character vector of text chunks to index directly.

ctx

Model context from edge_load_model() (used for embedding)

chunk_size

Approximate number of characters per chunk (default: 500)

chunk_overlap

Number of characters of overlap between chunks (default: 50)

file_pattern

Glob pattern for files to read (default: "*.txt")

normalize

Normalize embeddings (default: TRUE)

progress

Show progress messages (default: TRUE)

Value

An edge_index object containing chunks, embeddings, and source metadata.

See Also

edge_search, edge_ask

Examples

## Not run: 
ctx <- edge_load_model("model.gguf")

# Index a directory
index <- edge_index_documents("./reports/", ctx)

# Or index text directly
index <- edge_index_documents(
  c("Revenue grew 15%", "New product launched"), ctx)

# Search the index
results <- edge_search(index, ctx, "revenue growth")

## End(Not run)

Install the CUDA backend for GPU-accelerated inference

Description

Downloads a pre-built GGML CUDA backend shared library and stores it in the edgemodelr cache. After installation, restart your R session (or call edge_reload_cuda()) so the GPU backend is picked up during model loading.

On Windows, the CUDA runtime libraries (cudart, cublas) must also be present. Run edge_install_cuda_toolkit() once after this function to install them automatically.

Usage

edge_install_cuda(cuda_version = "13.4", force = FALSE, llama_build = "b11960")

Arguments

cuda_version

CUDA version string. One of "12.4" or "13.4" (default). Use "13.4" for Blackwell GPUs (RTX 50 series, sm_120).

force

Re-download even if already installed.

llama_build

llama.cpp build number to pull the backend from (default: "b11960", the build of llama.cpp v0.8.0 bundled in this package).

Value

Invisibly returns the path to the installed CUDA backend DLL.

See Also

edge_install_cuda_toolkit, edge_reload_cuda, edge_cuda_info

Examples

## Not run: 
edge_install_cuda()                  # GPU backend installed
edge_install_cuda_toolkit()          # Install CUDA runtime DLLs (Windows)
ctx <- edge_load_model("model.gguf", n_gpu_layers = 35)

## End(Not run)

Install CUDA runtime libraries required for GPU inference

Description

The CUDA backend (ggml-cuda-XX.dll) requires two sets of runtime DLLs that are not included with the NVIDIA display driver:

After running this function, call edge_reload_cuda() and load a model with n_gpu_layers = -1 to run on your GPU.

This function is only needed on Windows. On Linux and macOS, CUDA support is enabled at build time via the EDGEMODELR_CUDA=1 environment variable.

Usage

edge_install_cuda_toolkit(cuda_version = "13.4", force = FALSE)

Arguments

cuda_version

CUDA major version string, e.g. "13.4" (default). Must match the version used with edge_install_cuda().

force

Reinstall even if the runtime DLLs are already present.

Value

Invisibly returns the cuda cache directory path.

See Also

edge_install_cuda, edge_reload_cuda, edge_cuda_info

Examples

## Not run: 
edge_install_cuda()          # install ggml-cuda GPU backend DLL (~140 MB)
edge_install_cuda_toolkit()  # install CUDA runtime DLLs (~400 MB, one-time)
edge_reload_cuda()           # activate in this R session
ctx <- edge_load_model("model.gguf", n_gpu_layers = -1)
result <- edge_completion(ctx, "Hello", n_predict = 20)

## End(Not run)

Generate a GBNF grammar for JSON output from a schema

Description

Converts a simple R list schema into a GBNF grammar string that constrains model output to valid JSON matching the schema.

Usage

edge_json_grammar(schema, compact = FALSE)

Arguments

schema

A named list where each element defines a field. Values can be:

  • "string" - any string value

  • "number" - a numeric value (integer or decimal)

  • "integer" - an integer value

  • "boolean" - true or false

  • A character vector - enum of allowed values, or I("x") for a single allowed value

compact

If TRUE, the layout is fixed to {"key": value, "key": value}, with one space after each colon and comma. Every key and separator is then forced, and edge_grammar_completion() adds them without sampling. If FALSE (default), the model may put no space, one space or a newline and indentation around each key and separator.

Value

A GBNF grammar string suitable for use with edge_grammar_completion

See Also

edge_grammar_completion, edge_extract

Examples

# Schema with enum and free-text fields
grammar <- edge_json_grammar(list(
  sentiment = c("positive", "negative", "neutral"),
  confidence = "number",
  explanation = "string"
))
cat(grammar)

List popular pre-configured models

Description

List popular pre-configured models

Usage

edge_list_models()

Value

Data frame with model information


Load a local GGUF model for inference

Description

Load a local GGUF model for inference

Usage

edge_load_model(
  model_path,
  n_ctx = 2048L,
  n_gpu_layers = 0L,
  n_threads = NULL,
  flash_attn = TRUE,
  embeddings = FALSE,
  prompt_cache = FALSE,
  lora = NULL,
  lora_scale = 1,
  n_parallel = 1L,
  cache_type_k = "f16",
  cache_type_v = "f16"
)

Arguments

model_path

Path to a .gguf model file

n_ctx

Maximum context length (default: 2048)

n_gpu_layers

Number of layers to offload to GPU (default: 0, CPU-only). Use -1 to offload all layers (requires CUDA backend via edge_install_cuda()).

n_threads

Number of CPU threads for inference (default: NULL = one per physical core). Extra hyperthreads slow generation down, often by half. Set to a lower value to leave cores free for other tasks.

flash_attn

Enable flash attention for faster inference (default: TRUE). Reduces memory usage and improves speed. Set to FALSE for maximum compatibility.

embeddings

Enable embedding extraction mode (default: FALSE). Must be TRUE to use edge_embeddings with this context. A model loaded with embeddings = TRUE can still be used for text generation.

prompt_cache

Keep the KV cache between calls and reuse the part of a new prompt that matches the start of the previous one, so only the rest is processed (default: FALSE). Much faster when prompts share a long fixed beginning, such as a system prompt. Each completion still depends only on its own prompt.

lora

Path(s) to LoRA adapters in GGUF format to apply on top of the model (default: NULL, none). The adapters must be made for this base model. A PEFT adapter can be converted with llama.cpp's convert_lora_to_gguf.py.

lora_scale

Scale for each adapter, recycled to the number of adapters (default: 1). 0 turns an adapter off. Change it later with edge_lora_scale.

n_parallel

How many prompts edge_map and edge_extract_batch generate at once (default: 1). Each step then decodes one token for every prompt in a single batch, which on CPU costs little more than one token, so batches of texts finish several times faster. The prompts share the context, so make n_ctx large enough for all of them with their answers; fewer run at once when they do not fit.

cache_type_k, cache_type_v

Number format of the keys and values in the KV cache (default: "f16"). One of "f16", "bf16", "q8_0", "q5_1", "q5_0", "q4_1", "q4_0" or "iq4_nl". "q8_0" for both halves the memory the cache takes, with almost no change in output, so a larger n_ctx fits; the 4-bit types save more and cost some quality. A quantized cache_type_v needs flash_attn = TRUE.

Value

External pointer to the loaded model context

Examples

## Not run: 
# Quick setup with automatic model download (downloads ~700MB)
setup <- edge_quick_setup("TinyLlama-1.1B")
if (!is.null(setup$context)) {
  ctx <- setup$context

  # Generate completion
  result <- edge_completion(ctx, "Explain R data.frame:", n_predict = 100)
  cat(result)

  # Free model when done
  edge_free_model(ctx)
}

# Manual model loading with threading control
model_path <- "path/to/your/model.gguf"
if (file.exists(model_path)) {
  ctx <- edge_load_model(model_path, n_ctx = 2048, n_threads = 4, flash_attn = TRUE)
  # ... use model ...
  edge_free_model(ctx)
}

## End(Not run)

Load an Ollama model by partial SHA-256 hash

Description

Find and load an Ollama model using a partial SHA-256 hash instead of the full path. This is more convenient than typing out the full blob path.

Usage

edge_load_ollama_model(partial_hash, n_ctx = 2048L, n_gpu_layers = 0L)

Arguments

partial_hash

First few characters of the SHA-256 hash

n_ctx

Maximum context length (default: 2048)

n_gpu_layers

Number of layers to offload to GPU (default: 0)

Value

Model context if successful, throws error if not found or incompatible

Examples

## Not run: 
# Load model using first 8 characters of SHA hash
# ctx <- edge_load_ollama_model("b112e727")
# result <- edge_completion(ctx, "Hello", n_predict = 10)
# edge_free_model(ctx)

## End(Not run)

Get or change the scale of loaded LoRA adapters

Description

Adapters are loaded with edge_load_model(lora = ...). Changing a scale takes effect on the next call and needs no reload, so one base model can switch between adapters, or turn them off with 0, from call to call.

Usage

edge_lora_scale(ctx, scale = NULL)

Arguments

ctx

Model context from edge_load_model

scale

NULL to only read the scales, or one number per loaded adapter, in the order they were given to edge_load_model()

Details

Changing a scale clears the context's KV cache, so with prompt_cache = TRUE the next prompt is processed in full.

Value

The adapter scales, a numeric vector (invisibly when setting)

Examples

## Not run: 
ctx <- edge_load_model("base.gguf", lora = c("terms.gguf", "labels.gguf"),
                       lora_scale = c(1, 0))
edge_completion(ctx, "Map 'Once daily' to a CDISC term:")
edge_lora_scale(ctx, c(0, 1))   # switch to the second adapter
edge_lora_scale(ctx)            # 0 1
edge_free_model(ctx)

## End(Not run)

Apply a prompt template to a vector of texts

Description

Maps a prompt template over a character vector, generating completions for each element. This is the primary function for batch LLM operations on data frames. A progress indicator is printed to the console.

Usage

edge_map(
  ctx,
  texts,
  prompt_template,
  n_predict = 128L,
  temperature = 0.7,
  top_p = 0.95,
  grammar = NULL,
  progress = TRUE,
  top_k = 0L,
  min_p = 0,
  repeat_penalty = 1,
  seed = NULL,
  stop = NULL
)

Arguments

ctx

Model context from edge_load_model()

texts

Character vector of input texts

prompt_template

A string with {text} as a placeholder for each input, or a function that takes a single text and returns a prompt string.

n_predict

Maximum tokens to generate per text (default: 128)

temperature

Sampling temperature (default: 0.7)

top_p

Nucleus sampling threshold (default: 0.95)

grammar

Optional GBNF grammar string to constrain output

progress

Show progress messages (default: TRUE)

top_k

Sample only from the k most likely tokens (default: 0, off). 40 is a common choice.

min_p

Drop tokens less likely than this fraction of the most likely one (default: 0, off). 0.05 is a common choice.

repeat_penalty

Penalty for tokens used in the last 64 generated tokens (default: 1, off). Values from 1.1 to 1.3 stop small models from repeating themselves.

seed

Seed for sampling (default: NULL, drawn from R's random number generator, so set.seed() makes the text reproducible). Ignored when temperature = 0.

stop

Optional character vector of stop strings. Generation ends where the first one appears; the stop string itself is not returned.

Value

Character vector of completions, same length as texts

See Also

edge_classify, edge_extract_batch

Examples

## Not run: 
ctx <- edge_load_model("model.gguf")

# Summarize a column
df <- data.frame(review = c("Great product!", "Terrible, broke on day 1"))
df$summary <- edge_map(ctx, df$review,
  "Summarize in 5 words: {text}")

# With a custom prompt function
df$analysis <- edge_map(ctx, df$review, function(text) {
  paste0("Product review: ", text, "\nOne-word sentiment:")
})

edge_free_model(ctx)

## End(Not run)

Get the embedding dimension of a loaded model

Description

Returns the size of the embedding vectors produced by the loaded model.

Usage

edge_model_n_embd(ctx)

Arguments

ctx

Model context from edge_load_model()

Value

Integer giving the embedding vector dimension

See Also

edge_embeddings

Examples

## Not run: 
ctx <- edge_load_model("model.gguf")
edge_model_n_embd(ctx)  # e.g., 4096
edge_free_model(ctx)

## End(Not run)

Generate an English narrative from structured data

Description

Turns a named list, single-row data.frame, or multi-row data.frame into one or more short English sentences describing the data. The function linearises the input as labelled "field: value" lines and asks the model to narrate them without inferring anything that is not in the input.

Usage

edge_narrate(
  ctx,
  data,
  instruction = NULL,
  max_words = 60L,
  n_predict = 200L,
  temperature = 0.3,
  progress = TRUE
)

Arguments

ctx

Model context from edge_load_model()

data

A named list, a data.frame, or a character vector of pre-formatted "field: value" blocks. Multi-row data.frames are processed row-by-row and the function returns a character vector of the same length.

instruction

Optional instruction overriding the default ("Write a single short paragraph describing this record. Use only the data provided. Do not infer causality or add information.")

max_words

Soft limit communicated in the prompt (default: 60). Not enforced; the model is asked to stay under this.

n_predict

Maximum tokens to generate per record (default: 200)

temperature

Sampling temperature (default: 0.3)

progress

Show progress messages for multi-row inputs (default: TRUE)

Details

This is a drafting helper. For regulated outputs (CSR, ICSR, PV) always run edge_verify_narrative or an equivalent back-check before surfacing the text to a human reviewer, and pass it the study's drug and adverse-event terms. Unchecked output is often wrong: in a published study of small models drafting case narratives from ICSRs, 60% of outputs had wrong drug information and 71% wrong or missing adverse events (Hakim et al., 2024, arXiv:2407.18322).

Output quality depends heavily on the model. Small models tend to restate the input fields rather than summarise them. A model of at least 7B parameters gives noticeably better narratives.

Value

Character vector of narratives, one per input record.

Examples

## Not run: 
ctx <- edge_load_model("Qwen3-1.7B-Q4_K_M.gguf", n_ctx = 2048L)

lab <- list(
  USUBJID = "ABC-001-042", LBTESTCD = "ALT",
  LBORRES = 285, LBORNRHI = 55,
  LBDY = 28, VISIT = "Week 4", LBNRIND = "HIGH"
)
edge_narrate(ctx, lab)

# Multi-row data.frame
df <- data.frame(
  subject = c("S001", "S002"),
  alt = c(285, 47),
  uln = c(55, 55),
  flag = c("HIGH", "NORMAL")
)
edge_narrate(ctx, df)

## End(Not run)

Quick setup for a popular model

Description

Quick setup for a popular model

Usage

edge_quick_setup(
  model_name,
  cache_dir = NULL,
  verify_checksum = TRUE,
  expected_sha256 = NULL,
  trust_first_use = FALSE,
  verbose = TRUE
)

Arguments

model_name

Name of the model from edge_list_models()

cache_dir

Directory to store downloaded models

verify_checksum

Verify SHA-256 checksum if available (default: TRUE)

expected_sha256

Optional expected SHA-256 hash for the model file

trust_first_use

Store a local hash if no known hash exists (default: FALSE)

verbose

Whether to print setup progress messages

Value

List with model path and context (if llama.cpp is available)

Examples

## Not run: 
# Quick setup with TinyLlama (downloads model, not run in checks)
setup <- edge_quick_setup("TinyLlama-1.1B")
ctx <- setup$context

if (!is.null(ctx)) {
  response <- edge_completion(ctx, "Hello!")
  cat("Response:", response, "\n")
  edge_free_model(ctx)
}

## End(Not run)

Activate an installed CUDA backend without restarting R

Description

Loads the CUDA backend DLL into the current R session. Must be called before the first edge_load_model() call in the session; otherwise a session restart is required for GPU support to take effect.

Usage

edge_reload_cuda(path = NULL)

Arguments

path

Path to the CUDA DLL. Defaults to the standard install location (the cuda sub-directory of the edgemodelr cache).

Value

Invisibly returns TRUE if activation succeeded.

See Also

edge_install_cuda, edge_install_cuda_toolkit, edge_cuda_info

Examples

## Not run: 
# After running edge_install_cuda() and edge_install_cuda_toolkit():
edge_reload_cuda()
ctx <- edge_load_model("model.gguf", n_gpu_layers = -1)

## End(Not run)

Resolve a model name to the canonical entry in edge_list_models()

Description

Resolve a model name to the canonical entry in edge_list_models()

Usage

edge_resolve_model_name(model_name, models = NULL)

Arguments

model_name

User-provided model name

models

Optional models data.frame

Value

Canonical model name or NA if not found


Save or load the prompt cache

Description

With prompt_cache = TRUE a context keeps the processed tokens of the last prompt and reuses the part a new prompt shares with it. edge_save_state() writes that cache to a file and edge_load_state() reads it back, in the same or a later R session, so a long system prompt or document is processed once and not again every session.

Usage

edge_save_state(ctx, path)

edge_load_state(ctx, path)

Arguments

ctx

Model context from edge_load_model, loaded with prompt_cache = TRUE

path

File to write or read

Details

The file holds the KV cache, so it is large: about n_tokens * n_layers * 2 * n_kv_heads * head_size * 2 bytes, for example 22 MB for 1000 tokens of a 1B model. It can only be loaded into a context of the same model, with the same LoRA adapters at the same scales; anything else is an error. The saved tokens must fit in the context's n_ctx.

To cache just a fixed beginning, run it once with a single token to generate, then save: edge_completion(ctx, prefix, n_predict = 1).

Value

The number of tokens saved or loaded, invisibly

Examples

## Not run: 
ctx <- edge_load_model("model.gguf", prompt_cache = TRUE)
edge_completion(ctx, long_document, n_predict = 1)
edge_save_state(ctx, "document.kv")

# Later, in a new session:
ctx <- edge_load_model("model.gguf", prompt_cache = TRUE)
edge_load_state(ctx, "document.kv")
edge_completion(ctx, paste0(long_document, "\nSummary:"))  # skips the document

## End(Not run)

Description

Finds the most similar text chunks to a query using cosine similarity over the embedding index.

Usage

edge_search(index, ctx, query, top_k = 5L)

Arguments

index

An edge_index object from edge_index_documents

ctx

Model context (same model used to build the index)

query

Query text string

top_k

Number of results to return (default: 5)

Value

A data frame with columns: chunk, score, source, index

See Also

edge_index_documents, edge_ask

Examples

## Not run: 
ctx <- edge_load_model("model.gguf")
index <- edge_index_documents("./docs/", ctx)
results <- edge_search(index, ctx, "quarterly revenue")
print(results)

## End(Not run)

Serve a model as a local OpenAI-compatible API

Description

Starts a local Plumber API server that exposes the loaded model through endpoints compatible with the OpenAI API format. This allows other applications (Python, JavaScript, curl) and OpenAI client libraries to use the model over HTTP. chat_edge uses it to connect the model to ellmer.

Usage

edge_serve(
  model_path,
  host = "127.0.0.1",
  port = 8080L,
  n_ctx = 2048L,
  n_gpu_layers = 0L,
  embeddings = FALSE,
  api_key = NULL,
  n_slots = 1L,
  ...
)

Arguments

model_path

Path to a .gguf model file

host

Host to bind to (default: "127.0.0.1" for local only)

port

Port number (default: 8080)

n_ctx

Context size (default: 2048)

n_gpu_layers

GPU layers (default: 0, use -1 for full GPU offload)

embeddings

Enable embeddings endpoint (default: FALSE)

api_key

Optional API key for authentication. If set, requests must include an Authorization: Bearer <key> or X-API-Key: <key> header.

n_slots

Requests generated at once (default: 1); see edge_api.

...

More arguments for edge_api, such as n_threads, max_queue or cache_type.

Details

Endpoints served:

Requests take max_tokens (or max_completion_tokens), temperature, top_p, seed and stop, and also top_k, min_p and repeat_penalty. usage counts tokens with the model's tokenizer.

Chat requests can pass tools and tool_choice in the OpenAI format. Tool arguments may be strings, numbers, integers, booleans or enums, and every argument is treated as required. A call comes back in message$tool_calls with finish_reason = "tool_calls"; send the result back as a message with role = "tool" and the call's tool_call_id. The model makes at most one call per reply. See edge_chat_completion for how calls are generated.

response_format may be {"type": "json_schema"} with a flat object schema of the same argument types; the reply is then constrained to JSON of that shape by a grammar.

With stream = true the response is sent as server-sent events in the OpenAI chunk format, ending with data: [DONE], and stream_options = {"include_usage": true} adds a usage chunk. The chunks are produced token by token but sent together when generation ends, so clients get the format but not the text as it is written.

The server is edge_api run with plumber::pr_run(): requests wait in a queue and n_slots of them are generated at once. It runs in the foreground; press Ctrl+C / Esc to stop. For many users, or Posit Connect, use edge_api directly.

Value

Does not return while the server runs.

See Also

edge_api, chat_edge

Examples

## Not run: 
# Serve a model locally
edge_serve("model.gguf", port = 8080)

# Then from another terminal or Python:
# curl http://localhost:8080/v1/chat/completions \
#   -H "Content-Type: application/json" \
#   -d '{"messages": [{"role": "user", "content": "Hello!"}]}'

## End(Not run)

Control llama.cpp logging verbosity

Description

Enable or disable verbose output from the underlying llama.cpp library. By default, all output except errors is suppressed to comply with CRAN policies.

Usage

edge_set_verbose(enabled = FALSE)

Arguments

enabled

Logical. If TRUE, enables verbose llama.cpp output. If FALSE (default), suppresses all output except errors.

Value

Invisible NULL

Examples

# Enable verbose output (not recommended for normal use)
edge_set_verbose(TRUE)

# Disable verbose output (default, recommended)
edge_set_verbose(FALSE)

Query SIMD optimization status

Description

Reports which SIMD (Single Instruction Multiple Data) features were enabled at compile time. This helps verify that the package is using CPU-optimized code paths for faster inference.

Usage

edge_simd_info()

Value

List with:

architecture

CPU architecture (e.g., "x86_64", "aarch64")

compiler_features

Character vector of compiler-detected SIMD features

ggml_features

Character vector of GGML-level optimization flags

is_generic

Logical; TRUE if compiled with generic (scalar) fallback

kernels

The quantized kernels in use: "avx2" when a generic build found AVX2 on this CPU and switched to its AVX2 kernels, "generic", or "native" for a build made with EDGEMODELR_SIMD. Set the environment variable EDGEMODELR_KERNELS=generic before loading the package to keep the generic kernels.

Examples

info <- edge_simd_info()
cat("Architecture:", info$architecture, "\n")
cat("SIMD features:", paste(info$compiler_features, collapse = ", "), "\n")
cat("Kernels:", info$kernels, "\n")

Compute cosine similarity between two embedding vectors

Description

Computes the cosine similarity between two embedding vectors.

Usage

edge_similarity(a, b)

Arguments

a

Numeric vector (embedding)

b

Numeric vector (embedding)

Value

Cosine similarity score between -1 and 1

See Also

edge_embeddings, edge_similarity_matrix

Examples

## Not run: 
ctx <- edge_load_model("model.gguf")
embs <- edge_embeddings(ctx, c("happy cat", "joyful kitten", "stock market"))

edge_similarity(embs[1,], embs[2,])  # high (similar meaning)
edge_similarity(embs[1,], embs[3,])  # low (different topics)

## End(Not run)

Compute a similarity matrix for a set of embeddings

Description

Efficiently computes all pairwise cosine similarities between embedding vectors using normalized matrix multiplication.

Usage

edge_similarity_matrix(embeddings)

Arguments

embeddings

A numeric matrix where each row is an embedding vector (as returned by edge_embeddings)

Value

A symmetric numeric matrix of pairwise cosine similarities

See Also

edge_embeddings, edge_similarity

Examples

## Not run: 
ctx <- edge_load_model("model.gguf")
embs <- edge_embeddings(ctx, c("cat", "kitten", "car", "automobile"))
sim_mat <- edge_similarity_matrix(embs)
# sim_mat[1,2] high (cat~kitten), sim_mat[1,3] low (cat~car)

## End(Not run)

Get optimized configuration for small language models

Description

Returns recommended parameters for loading and using small models (1B-3B parameters) to maximize inference speed on resource-constrained devices.

Usage

edge_small_model_config(
  model_size_mb = NULL,
  available_ram_gb = NULL,
  target = "laptop"
)

Arguments

model_size_mb

Model file size in MB (if known). If NULL, uses conservative defaults.

available_ram_gb

Available system RAM in GB. If NULL, uses conservative defaults.

target

Device target: "mobile", "laptop", "desktop", or "server" (default: "laptop")

Value

List with optimized parameters for edge_load_model() and edge_completion()

Examples

# Get optimized config for a 700MB model on a laptop
config <- edge_small_model_config(model_size_mb = 700, available_ram_gb = 8)

# Use the config to load a model
## Not run: 
model_path <- "path/to/tinyllama.gguf"
if (file.exists(model_path)) {
  ctx <- edge_load_model(
    model_path,
    n_ctx = config$n_ctx,
    n_gpu_layers = config$n_gpu_layers
  )

  result <- edge_completion(
    ctx,
    prompt = "Hello",
    n_predict = config$recommended_n_predict,
    temperature = config$recommended_temperature
  )

  edge_free_model(ctx)
}

## End(Not run)

Stream text completion with real-time token generation

Description

Stream text completion with real-time token generation

Usage

edge_stream_completion(
  ctx,
  prompt,
  callback,
  n_predict = 128L,
  temperature = 0.8,
  top_p = 0.95,
  timeout_seconds = NULL,
  top_k = 0L,
  min_p = 0,
  repeat_penalty = 1,
  seed = NULL,
  stop = NULL
)

Arguments

ctx

Model context from edge_load_model()

prompt

Input text prompt

callback

Function called for each generated token. Receives list with token info.

n_predict

Maximum tokens to generate (default: 128)

temperature

Sampling temperature (default: 0.8). 0 gives greedy decoding: the same prompt always gives the same text.

top_p

Top-p sampling parameter (default: 0.95)

timeout_seconds

Optional timeout in seconds for inference

top_k

Sample only from the k most likely tokens (default: 0, off). 40 is a common choice.

min_p

Drop tokens less likely than this fraction of the most likely one (default: 0, off). 0.05 is a common choice.

repeat_penalty

Penalty for tokens used in the last 64 generated tokens (default: 1, off). Values from 1.1 to 1.3 stop small models from repeating themselves.

seed

Seed for sampling (default: NULL, drawn from R's random number generator, so set.seed() makes the text reproducible). Ignored when temperature = 0.

stop

Optional character vector of stop strings. Generation ends where the first one appears; the stop string itself is not returned.

Details

With stop strings, text that could be the start of one is held back until the next token decides it, so a callback can receive several tokens' text at once and never sees the stop string.

Value

List with full response and generation statistics

Examples

## Not run: 
model_path <- file.path(tempdir(), "model.gguf")
if (file.exists(model_path)) {
  ctx <- edge_load_model(model_path)

  # Basic streaming with token display
  result <- edge_stream_completion(ctx, "Hello, how are you?",
    callback = function(data) {
      if (!data$is_final) {
        cat(data$token)
        utils::flush.console()
      } else {
        cat("\n[Done: ", data$total_tokens, " tokens]\n")
      }
      return(TRUE)  # Continue generation
    })

  edge_free_model(ctx)
}

## End(Not run)

Generate a SQL query from a natural language question

Description

Convenience wrapper that constructs a text-to-SQL prompt with the supplied schema, queries the model, and post-processes the output into a single executable SQL statement. Optionally executes the query in a DBI connection and returns both the SQL and its result.

Usage

edge_text_to_sql(
  ctx,
  question,
  schema,
  dialect = "sqlite",
  examples = NULL,
  con = NULL,
  n_predict = 256L,
  temperature = 0.1
)

Arguments

ctx

Model context from edge_load_model()

question

Natural language question (single character string)

schema

One or more CREATE TABLE statements describing the relevant tables. Pass a single string or a character vector: vectors are concatenated with newlines. Pre-filter the schema to only the tables that are likely relevant to the question; large schemas degrade accuracy.

dialect

SQL dialect label inserted into the prompt (default "sqlite"). Examples: "sqlite", "postgresql", "snowflake".

examples

Optional named list of question -> SQL pairs used as few-shot examples (e.g. list("How many subjects?" = "SELECT COUNT(*) FROM ADSL;")).

con

Optional DBI connection. When non-NULL the generated SQL is executed in a read-only sense using DBI::dbGetQuery; failures are captured rather than re-thrown.

n_predict

Maximum tokens to generate (default: 256)

temperature

Sampling temperature (default: 0.1, low for SQL)

Details

This is a drafting helper, not a sandbox. Always review generated SQL before executing it against any system of record. Pair this function with a read-only connection and dedicated validation logic for production use.

Output quality depends heavily on the model. Small general purpose models often produce syntactically valid SQL that answers a different question than the one asked. Use a model fine tuned for SQL, or a general model of at least 7B parameters, and review every statement before running it.

Value

If con is NULL, a single character string containing the SQL. If con is supplied, a list with elements sql, result, and error (NA on success).

Examples

## Not run: 
ctx <- edge_load_model("sqlcoder-7b-2.Q4_K_M.gguf", n_ctx = 4096L)

schema <- "
CREATE TABLE ADSL (USUBJID TEXT, AGE INTEGER, SEX TEXT, TRT01A TEXT, SAFFL TEXT);
CREATE TABLE ADAE (USUBJID TEXT, AETERM TEXT, AESER TEXT, AEREL TEXT);
"

sql <- edge_text_to_sql(ctx,
  "How many subjects on active arm had at least one serious related AE?",
  schema = schema, dialect = "sqlite")
cat(sql)

# With execution
con <- DBI::dbConnect(RSQLite::SQLite(), "study.db")
out <- edge_text_to_sql(ctx, "Count safety analysis subjects per arm",
                         schema = schema, con = con)
out$sql
out$result
DBI::dbDisconnect(con)

## End(Not run)

Tokenize text with a model's tokenizer

Description

Splits text into the model's tokens, the same way the completion functions read a prompt, so length(edge_tokenize(ctx, prompt)) is the number of tokens the prompt takes in the context.

Usage

edge_tokenize(ctx, text, add_special = TRUE)

Arguments

ctx

Model context from edge_load_model

text

A single character string

add_special

Add the model's start-of-text token, as the completion functions do (default: TRUE)

Value

An integer vector of token ids

Examples

## Not run: 
ctx <- edge_load_model("model.gguf")
length(edge_tokenize(ctx, "The quick brown fox"))
edge_free_model(ctx)

## End(Not run)

Define a tool the model can call

Description

A tool is a name, a description and typed arguments. Pass a list of tools to edge_chat_completion and the model can call them; the arguments always come back with the types given here.

Usage

edge_tool(name, description, parameters = list())

Arguments

name

Tool name: letters, digits and underscores, starting with a letter.

description

What the tool does, in one sentence. The model reads it to decide when to call the tool.

parameters

A named list of arguments, in the format of edge_json_grammar: "string", "number", "integer", "boolean" or a character vector of allowed values (use I("x") for a single allowed value). An argument can also be list(type = ..., description = ...) to tell the model what it means. Every argument is required. Default: no arguments.

Value

An edge_tool object

See Also

edge_chat_completion

Examples

weather <- edge_tool("get_weather", "Get the current weather for a city",
  list(city = list(type = "string", description = "City name"),
       unit = c("celsius", "fahrenheit")))
weather

Verify a generated narrative against source values

Description

Checks a generated narrative against the source record in two independent ways. The literal check needs no model: every numeric source value must appear in the narrative, and, when terms is given, every term from that dictionary (e.g. drug names, adverse-event terms) must occur in the narrative if and only if it occurs in the source. The model check re-extracts the named fields from the narrative with edge_extract and compares them to the source record.

Usage

edge_verify_narrative(
  ctx,
  narrative,
  expected,
  schema = NULL,
  tolerance = 1e-06,
  temperature = 0.1,
  method = c("both", "literal", "model"),
  terms = NULL
)

Arguments

ctx

Model context from edge_load_model(). May be NULL when method = "literal".

narrative

Character string produced by edge_narrate or any other generator.

expected

Named list of source field values. The names define which fields to re-extract from the narrative. Values define the source-of-truth.

schema

Optional named list overriding the schema passed to edge_extract. If NULL, types are inferred from expected (numeric -> "number", integer -> "integer", logical -> "boolean", character -> "string").

tolerance

Numeric tolerance for number fields (default: 1e-6). Values within tolerance are considered equal.

temperature

Sampling temperature for re-extraction (default: 0.1)

method

Which checks to run: "both" (default), "literal" (deterministic only, no model call) or "model" (re-extraction only).

terms

Optional character vector of controlled terms to guard, such as the drug names and MedDRA terms relevant to the study. Matching is case-insensitive on whole words. Ignored when method = "model".

Details

The model check re-extracts the fields with the model, so it inherits the model's limitations. A small model can miss a field that is present and report a mismatch that is not real, or confirm a value that is wrong. The literal check has no such failure mode, but it is strict: it does not recognise numbers written as words, and it reports a term as missing when the narrative uses a synonym or a misspelling. Treat any failure as a reason for a human to look rather than as proof of an error.

Unchecked generated narratives are often wrong. In a published study of small models drafting case narratives from ICSRs, 60% of outputs contained wrong drug information and 71% wrong or missing adverse events. A simple drug and adverse-event name comparison between source and output, which is what terms provides, flagged every hallucinated drug name in the sample that was checked (Hakim et al., 2024, arXiv:2407.18322).

Value

A list with:

Examples

## Not run: 
ctx <- edge_load_model("Qwen3-1.7B-Q4_K_M.gguf", n_ctx = 2048L)

rec <- list(subject = "S001", drug = "Drug X", event = "Hepatotoxicity",
            alt = 285, uln = 55, study_day = 28)
narrative <- edge_narrate(ctx, rec)
check <- edge_verify_narrative(ctx, narrative, expected = rec,
                               terms = c("Drug X", "Drug Y",
                                         "Hepatotoxicity", "Rash"))
if (!check$ok) {
  print(check$mismatches)
  print(check$literal[!check$literal$found, ])
  print(check$unsupported_terms)
}

# Deterministic check only, no model needed
edge_verify_narrative(NULL, narrative, rec, method = "literal")

## End(Not run)

Embed texts with a local model, for ragnar and other packages

Description

Turns texts into embedding vectors with a local GGUF embedding model, without a server. Called without x, it returns a function of the texts, which is what ragnar::ragnar_store_create(embed = ) takes. That function refers to the model only by its path, so a ragnar store saved with it embeds new texts and queries again in a later R session.

Usage

embed_edge(x, model_path, n_ctx = 2048L, normalize = TRUE)

Arguments

x

A character vector of texts, a data frame with a text column, or missing.

model_path

Path to a .gguf embedding model, such as Qwen3-Embedding-0.6B or nomic-embed-text.

n_ctx

Context size for the model (default: 2048). Longer texts are cut to the batch size.

normalize

Scale each vector to length 1 (default: TRUE), so the dot product of two vectors is their cosine similarity.

Details

The model is loaded the first time it is needed and kept for the rest of the session, one per path.

Value

For a character vector, a numeric matrix with one row per text. For a data frame, the data frame with an embedding matrix column added. Without x, a function that takes x.

See Also

edge_embeddings, chat_edge

Examples

## Not run: 
model <- "Qwen3-Embedding-0.6B-Q8_0.gguf"
m <- embed_edge(c("cats purr", "dogs bark"), model)
dim(m)

# A ragnar store that embeds locally
store <- ragnar::ragnar_store_create(
  "docs.duckdb",
  embed = embed_edge(model_path = model)
)

## End(Not run)

Check if model context is valid

Description

Check if model context is valid

Usage

is_valid_model(ctx)

Arguments

ctx

Model context to check

Value

Logical indicating if context is valid


Print method for edge_index objects

Description

Print method for edge_index objects

Usage

## S3 method for class 'edge_index'
print(x, ...)

Arguments

x

An edge_index object

...

Additional arguments (ignored)

Value

The object (invisibly)


Test if an Ollama model blob can be used with edgemodelr

Description

This function tries to load an Ollama GGUF blob with edgemodelr using a minimal configuration and then runs a very short completion. It is intended to quickly detect common incompatibilities (unsupported architectures, invalid or unsupported GGUF files, or models that cannot run inference) before you attempt to use the model in a longer session.

Usage

test_ollama_model_compatibility(model_path, verbose = FALSE)

Arguments

model_path

Path to the Ollama blob file (a GGUF file, typically named by its SHA-256 hash inside the Ollama models/blobs directory).

verbose

If TRUE, print human-readable diagnostics for models that fail the compatibility checks.

Details

A model is considered compatible if:

When verbose = TRUE, this function classifies common failure modes: unsupported model architecture, invalid GGUF file, unsupported GGUF version, or a generic error (first 80 characters reported with truncation indicator).

Value

Logical: TRUE if the model loads and can run a short completion successfully, FALSE otherwise.

Examples

## Not run: 
# Test an individual Ollama blob
# is_ok <- test_ollama_model_compatibility("/path/to/blob", verbose = TRUE)
#
# This function is also used internally by edge_find_ollama_models()
# when test_compatibility = TRUE.

## End(Not run)