Notes for May 20-26

This week I spent a fair bit of time watching Microsoft Build recordings–partly because it has some impact on , and partly because it was brimming with stuff I can actually use.

And yes, you need to cherrypick. Here’s a few I am taking as inspiration:

The no-code stuff was also pretty impressive, but I can’t really use it in my own projects…

Prompting Phi-3 Successfully

I finally had some success at getting phi3:instruct to do function calling with ollama, which is great because it can be so much faster than llama3 on restricted hardware.

The system prompt structure I arrived at for moderately effective RAG is rather convoluted, though, and I had to cobble it together from various sources:

You are an AI assistant that can help the user with a variety of tasks. You have access to the functions provided by the schema below:

<|functions_schema|>
[
    {
        "name": "duckduckgo",
        "description": "searches the Internet",
        "parameters": [
            "name": "q"
            "type": "string"
        ],
        "required": [ "q" ],
        "returns": [
            {
                "name": "results",
                "type": "list[string]",
            }
        ]
    }
] 
<|end_functions_schema|>

When the user asks you a question, if you need to use functions, provide ONLY ALL OF THE function calls, ALL IN ONE PLACE, in the format:

<|function_calls|>
[
    { "name": "function_name", "kwargs": {"kwarg_1": "value_1", "kwarg_2": "value_2"}, "returns": ["output_1"]},
    { "name": "other_function_name", "kwargs": { "kwarg_3": "$output_1$"}, "returns": ["output_2", "output_3"]},
    ...
]
<|end_function_calls|>

IF AND ONLY IF you don't need to use functions, give your answer in between <|answer|> and <|end_answer|> blocks. For your thoughts and reasoning behind using or not using functions, place ALL OF THEM in between a SINGLE <|thoughts|> and <|end_thoughts|> block, before the <|function_calls|> and <|end_function_calls|> tags, like so:

<|thoughts|>
The user wants X, to do that, I should call the following functions:
1. function_name: Reasoning,
2. function_name_2: Reasoning2,
3. etc.
<|end_thoughts|>

Provide nothing else than the information in the <|function_calls|> & <|end_function_calls|>, <|answer|> & <|end_answer|> and <|thoughts|> & <|end_thoughts|> blocks.

phi3 still hallucinates every now and then, but the block approach made it easier to parse the outputs, and chaining function calls using contexts is relatively easy.

Still, I keep looking for ways to make prompt generation, function chaining and general workflows simpler, especially because this doesn’t really let me restrict outputs to a predefined set of options and other things that improve reliability when chaining actions.

Guidance

My prompting woes… prompted me (ahem) to take another look at guidance, which promises to solve a lot of those issues.

However, as it turns out I can’t really use it for my edge/ARM scenarios yet–this because ollama support is pretty much half-baked (in all fairness, guidance tries to do token manipulation directly, so it really relies on direct access to the model, not APIs).

But it is interesting for doing general purpose development, even with local models–there just isn’t really a lot of usable documentation, so it took me a bit to get it to work with Metal on macOS.

Here’s all you need to know, using their minimal sample:

from guidance import gen, select
from guidance.models import Transformers
from torch.backends import mps
from os import environ

MODEL_NAME = environ.get("MODEL_NAME", "microsoft/Phi-3-mini-4k-instruct")

device_map = None
if mps.is_available():
    device_map = "mps"

phi3 = Transformers(MODEL_NAME, device_map=device_map)

# capture our selection under the name 'answer'
lm = phi3 + f"Do you want a joke or a poem? A {select(['joke', 'poem'], name='answer')}.\n"

# make a choice based on the model's previous selection
if lm["answer"] == "joke":
    lm += f"Here is a one-line joke about cats: " + gen('output', stop='\n')
else:
    lm += f"Here is a one-line poem about dogs: " + gen('output', stop='\n')

And this is the minimal set of packages I needed to install to have a usable sandbox:

guidance==0.1.15
transformers==4.41.1
sentencepiece==0.2.0
torch==2.3.0
torchvision==0.18.0
accelerate==0.30.1
#litellm
#jupyter
#ipywidgets

I also had another go at using promptflow, but it too is tricky to use with local models.

But I have more fundamental things to solve–for instance, I’m still missing a decent in-process vector database for constrained environments. Can’t wait for sqlite-vec to ship.