I usually add context free grammar to make sure it always outputs valid JSON. Here’s the json.gbnf I use.

root   ::= object
value  ::= object | array | string | number | ("true" | "false" | "null") ws

object ::=
  "{" ws (
            string ":" ws value
    ("," ws string ":" ws value)*
  )? "}" ws

array  ::=
  "[" ws (
            value
    ("," ws value)*
  )? "]" ws

string ::=
  "\"" (
    [^"\\] |
    "\\" (["\\/bfnrt] | "u" [0-9a-fA-F] [0-9a-fA-F] [0-9a-fA-F] [0-9a-fA-F]) # escapes
  )* "\"" ws

number ::= ("-"? ([0-9] | [1-9] [0-9]*)) ("." [0-9]+)? ([eE] [-+]? [0-9]+)? ws

# Optional space: by convention, applied in this grammar after literal chars when allowed
ws ::= ([ \t\n] ws)?

I’ve been having quite a good time with Airoboros and combining the example prompts they provide in the README for both function calling and ReWOO style execution planning. I mostly use the provided prompts as-is, but have also been toying around with having it output the plan with JSON and adding a bit richer information about each execution plan step.

My rough approach is to first to a call with a ReWOO prompt where I list all the functions as tools (+ their descriptions; no params at the moment). Based on that I parse the plan, and do an additional calls where I only provide it with the shape of the selected function and the raw input it gets from prior plan steps (this is basically just an adapter that creates a function call in the correct shape).

One problem I haven’t been able to solve well so far are complex execution plan that involve e.g. executing single steps multiple times with different inputs (e.g. List N cities -> Get information X for each city -> Summarize all X). Would love to hear some input if anybody knows something about that. Apart from that I’m currently toying around with iterative plan prompting (asking for a new execution plan after each step of tool executions to allow for early exits or longer chains based on dynamically discovered information).