Databricks customers have access to foundation model APIs like DBRX, Meta Llama 3 70B, and Mixtral 8x7B. Databricks also provides the ability to train and deploy custom models.
chattr supports the following models on Databricks by
default:
| Model | Databricks Model Name | chattr Name |
|---|---|---|
| DBRX Instruct | databricks-dbrx-instruct |
databricks-dbrx |
| Meta-Llama-3-70B-Instruct | databricks-meta-llama-3-70b-instruct |
databricks-meta-llama3-70b |
| Mixtral-8x7B Instruct | databricks-mixtral-8x7b-instruct |
databricks-mixtral8x7b |
Databricks requires a host
(workspace URL) and token
to authenticate. Both are required for any non-Databricks application,
such as chattr, to interact with the models in the
Databricks workspace.
The token can be generated by the user in the workspace under the developer settings (docs) and the host can be found in the workspaces URL (docs).
By default, chattr will look for the credentials in
environment variables:
DATABRICKS_HOST
DATABRICKS_TOKEN
Use Sys.setenv() to set the variable. The downside of
using this method is that the variable will only be available during the
current R session:
Sys.setenv("DATABRICKS_HOST" = "https://xxxxx.cloud.databricks.com")
Sys.setenv("DATABRICKS_TOKEN" = "####################")A preferred method is to save the secret key to the
.Renviron file. This way, there is no need to load the
environment variable every time you start a new R session. The
.Renviron file is available in your home directory. Here is
an example of the entry:
DATABRICKS_HOST = https://xxxxx.cloud.databricks.com
DATABRICKS_TOKEN = ####################By default, chattr is setup to interact with GPT 4
(gpt-4). To switch to Meta Llama 3 70B you can run:
library(chattr)
chattr_use("databricks-meta-llama3-70b")
#>
#> ── chattr
#> • Provider: Databricks
#> • Path/URL: serving-endpoints
#> • Model: databricks-meta-llama-3-70b-instruct
#> • Label: Meta Llama 3 70B (Databricks)If a model doesn’t appear in the supported table but is deployed on
Databricks
model serving as OpenAI-compatible (configured
with llm/v1/chat in mlflow) then you can specify the
model name explicitly with chattr_use()
For example if you have deployed a fine-tuned version LLM with an
endpoint name of "CustomLLM":
library(chattr)
# use any existing databricks foundation model name (e.g. datarbicks-dbrx)
# then adjust the default model name to 'CustomMixtral'
chattr_use(x = "databricks-dbrx", model = "CustomLLM")
#>
#> ── chattr
#> • Provider: Databricks
#> • Path/URL: serving-endpoints
#> • Model: CustomLLM
#> • Label: DBRX (Databricks)Because it is information about your environment and work space, by
default chattr avoids sending any data files, and data
frame information to Databricks. Sending this information is convenient
because it creates a shorthand for your requests. If you wish to submit
this information as part of your prompts, use
chattr_defaults(), for example:
chattr_defaults(max_data_files = 10)chattr_defaults(max_data_frames = 10)These two commands will send 10 data frames, and 10 data files as part of your prompt. You can decide the number to limit this by. The more you send, the larger your prompt.
If any of these is set to anything but 0, a warning will show up every time you start the Shiny app:
• Provider: Databricks
• Path/URL: serving-endpoints
• Model: databricks-dbrx-instruct
• Label: DBRX (Databricks)
! A list of the top 10 files will be sent externally to Databricks with every request
To avoid this, set the number of files to be sent to 0 using chattr::chattr_defaults(max_data_files = 0)Î