Why does brainful have a plethora of models and providers?

• Models are quickly being replaced by new, more capable models.
• No single model is fit for all purposes. We have done the testing.

Essentially, we have gotten into the habit of sticking to a familiar model or set of models based on arbitrary criteria (or lack thereof) when we should rather care about choosing capabilities, not models.

We began work on a project at the start of 2025 that provides a high-level interface to talk to dozens of models and providers in an agnostic manner. The premise being that, this communication is based on desire of intent, not individual selection.

Our interface allows us to build powerful workflows by choosing the capabilities we care about, for example, the languages, skills, knowledge areas, speed, cost, latency, quality, relevance, and other nuances of the use case which often go completely unnoticed and unconsidered when sticking to a single model.

Now, addressing the provider part. The essential problem is that contrary to popular belief, model behaviour is not entirely dependent on the model!The efficacy of a model is just as dependent on the provider implementation as the inherent capability of the model itself.

To put this into perspective, our internal automated testing for one particular model amidst two providers revealed a world of difference in across all metrics.  Our standardised LLM intelligence test tests models against a wide set of capabilities (e.g. reasoning, math, classification, common sense) through sets of questions ranging from easy to hard difficulty. The results showed:

• Provider A of the LLM scored 5%, took 8.65 seconds to complete, and processed 90 tokens per second.
• Provider B of the LLM scored 17%, took 1.95 seconds to complete, and processed 380 tokens per second.

The shocking part was not just that provider B Pareto-dominated provider A, but did so at a 33% cheaper price.

In all, there are many other advantages to having a multi-model and multi-provider system:

1. To reduce vendor lock-in.

2. To provide greater model diversity with capabilities that may not be available from a single provider.

3. To provide a fallback mechanism in case a model is unavailable.

4. To load balance requests across multiple providers to maximise throughput.

5. To provide more nuanced model routing for better response quality.

6. To provide maximum inference speed by using the best provider implementation.

Playing such an integral part of brainful, we are in the process of opening access to our router through brainful.ai in the coming months.