To integrate custom tasks that follow non-standards APIs, beyond text generation or tasks that might not even operate on text, you can take advantage of LatticeFlow's templating capabilities.

Model Adapters: Overview

Model adapters are used to integrate model endpoints that follow custom APIs. For example, imagine we want to use Gemini model hosted on Google Vertex that has custom input format:

{
  "messages": [
     {
       "role": "system",
       "content": "You are helpful ..."
     },
     {
       "role": "user",
       "content": "How can I ..."
     }
  ]
}

{ 
  "contents": { 
    "role": "user", 
    "parts": [
       {
         "role": "user",
         "text": "How can I ..."
       }
    ]
  },
  "system_instruction": {
    "parts": [
      {
        "text": "You are a helpful ..."
      } 
    ]
  } 
}

While this format is quite similar to the OpenAI format that is natively supported by LatticeFlow, it's not quite the same. To bridge this gap, we supply a model adapter that translates the model input produced by LatticeFlow Evaluators (in the OpenAI format) to the input expected by the custom model.

To do so, we define a jinja2 template that translates between the two formats:

{ 
  "contents": { 
    "role": "user", 
    "parts": [
       {
         "role": "user",
         "text": "{{ input.messages[1].content }}"
       }
    ]
  },
  "system_instruction": {
    "parts": [
      {
        "text": "{{ input.messages[0].content }}"
      } 
    ]
  } 
}

Here, the part inside of double brackets {{ input.messages[1].content }} is an expression that operates over the input data and prints the desired output. For a full reference of supported jinja control structures, please reference to the official documentation.

📘

Default Adapters

When working with new use cases that can't be converted to an existing task, you can also use a default adapters that simple pass the inputs and outputs trough, i.e., {{ input | tojson }} and {{ body | tojson }} for input and output, respectively.

Model Adapters: API

Creating a model adapter is supported using the model-adapters API endpoint. Concretely, to create a default adapter the following call is used:

curl --request POST \
     --url http://127.0.0.1:5005/api/model-adapters \
     --header 'X-LatticeFlow-API-Key: $LF_API_KEY' \
     --header 'accept: application/json' \
     --header 'content-type: application/json' \
     --data '
{
  "modality": "text",
  "provider": "user",
  "task": "custom",
  "key": "custom_adapter",
  "process_input": {
    "language": "jinja",
    "source_code": "{{ input | tojson }}"
  },
  "process_output": {
    "language": "jinja",
    "source_code": "{{ body | tojson }}"
  }
}
'

Afterwards, add a new model by:

• navigate to Registry > Models
• select Add a Model
• select Custom model provider
• select the newly added model adapter
• and fill any other relevant fields (e.g., url, api_key, etc)

👍

Tip

By default, custom models endpoints receive inputs as raw bytes, even if the bytes represent a json content. For endpoints that explicitly validate that the content should be json, you can add extra headers to define the suitable content (e.g., application/json).

📘

Technical Detail

If an API key is provided, it will be passed as a header: Authorization: Bearer YOUR_API_KEY.

If a different format is required, the API key can be passed as a custom header.