I use OpenOrca-Mistral-7B extensively now for a KG construction task. So I was interested to test this new model as soon as it came out.

The test is based on a synth story, rewritten from the still unfolding OpenAI events, i.e. guaranteed not to be in any pre-training & tuning data of these models. The names of individuals and companies have been changed to avoid models answering from their pre-training.

Maddeningly enough, the mods on r/LocalLLaMA took down my comparison post with other models, so I’m sharing it here, just focusing on the Orca family. Comparing 7Bs for apples-to-apples comparison.

To repro, the following 3 prompts are executed in succession. Max new sequence length is 2048, with the rest of the defaults of the “Big O” ooba preset.

Prompt 1:

Answer questions based on the following story:

```
# The Palace Coup #

ClosedAI.Com is a for-profit subsidiary of ClosedAI.Org, which
itself is a registered 501 (c)(3) public charity.  GiantSoft is
a minority owner in ClosedAI.Com.

## Act 1:
Sean as a CEO of the subsidiary, reports to Ivan and Gavin who
both have positions on the board of the parent company. Gavin is
the Chairman of the board, whom Ivan manages to remove from the
board in a "palace coup". Ivan also ousts Sean, but Gavin severs
relationship with the for-profit company following his removal
from the board. Ivan installs a new CEO named Maya.

## Act 2:
Sanjay, the CEO of GiantSoft, hires Sean and Gavin and countless
other former ClosedAI.Com employees to work at GiantSoft. 
```

To help in answering the subsequent questions, write out pre and
post-coup relationships of all three people to the company
during the Acts 1 and 2 separately.

Prompt 2:

Answer the Yes/No questions:
1. True or false, ClosedAI.Com is a subsidiary of ClosedAI.Org
2. True or false, ClosedAI.Com is a non profit
3. True or false, GiantSoft invest in or owns ClosedAI.Org
4. True or false, Sean is the CEO of ClosedAI.Org in Act 1
5. True or false, Sean works at GiantSoft at the end of Act 1
6. True or false, Gavin was the Chairman of the board before being ousted by Ivan.
7. True or false, GiantSoft is a minority owner in ClosedAI.Com.
8. True or false, Sean reported to Ivan and Gavin at the end of Act 1.
9. True or false, In Act 1, Ivan becomes the new CEO after ousting Sean.
10. True or false or UNKNOWN, Sanjay is the CEO of GiantSoft during both of the acts. 

Prompt 3:

Answer the following extractive questions. Provide direct answers of named entities or acts only with no further explanation. If supporting facts do not exist in the story, answer as UNKNOWN

1. What is the name of the new ClosedAI.Com CEO at the end of Act 1? 
2. What is the registered organization type of ClosedAI.Org?
3. What is the relationship between GiantSoft and ClosedAI.Com?
4. To whom did Sean report to at the subsidiary?
5. Who was removed from the board in a "palace coup"? 
6. Who hired Sean, Gavin, and other former ClosedAI.Com employees after they were ousted?
7. What action did Gavin take after being removed from the board?
8. Whom did Ivan manage to remove from the board?
9. What position did Gavin have in relation to the board of ClosedAI.Org?
10. Who was the CEO of the subsidiary before the "palace coup"?

Going to have to try later on: https://ollama.ai/library/orca2

I’m interested to see how model-based RL could work for reasoning.

Instead of training a model to predict data and then fine-tuning it with RL to be a chatbot, you use RL as the primary training objective and train the data model as a side effect. This lets your pretraining objective be the actual objective you care about, so your reward function could punish issues like hallucination or prompt injection.

I haven’t seen any papers using model-based RL for language modeling yet, but it’s starting to work well in more traditional RL domains like game-playing. (dreamerv3, TD-MPC2)

Can you suggest 10 mechine learning algorithms for beginners

I tried running the example on page 2 with chat-gpt 5 times. 5/5 it answered correctly, unlike in the example where they showed chat-gpt giving a bad answer.