This is an internal tool used by our engineers at Scalattice to benchmark our inference API for price and latency against rivals offering open models. We decided to release it because we believe it can further the development of inference networks and products, and help build a world of inference abundance.
The repo is public: github.com/scalattice/modelmeter.
What it is
Modelmeter is a small local web app. It pulls live list prices from several catalogs, lets you filter by provider and model family, estimates a bill for a given token mix, and runs the same prompt against multiple OpenAI-compatible chat endpoints in parallel so you can see latency, tokens per second, and approximate cost on one screen.
API tokens stay on your machine in a local SQLite file. Nothing is uploaded to Scalattice when you run Compare.
Why we needed it
When you operate a network, “is our rate competitive?” and “does this endpoint feel fast?” are not one spreadsheet. List prices move. Catalog IDs disagree. A model that is cheap on paper can be slow on the wire. We wanted one place where engineers could ask both questions without rebuilding a one-off notebook every week.
Modelmeter is that place. We use it when we change catalog rates, when we evaluate a new host, and when we want a quick A/B of the same prompt across Scalattice and another provider.
What you can do with it
- Search and filter live open-model catalogs
- Compare $/1M input and output, blend, tokens per dollar, and a simple bill estimate
- Run Compare / Try with optional system prompt, image, stream, and custom headers
- Add your own OpenAI-compatible hosts and named API tokens in Settings
Run it
git clone https://github.com/scalattice/modelmeter.git
cd modelmeter
./run.sh
# → http://127.0.0.1:8765./run.sh creates a venv, installs the package, and starts the server. Pass-through flags work: ./run.sh --host 0.0.0.0 --port 8765.
How this relates to Scalattice Cloud
Modelmeter is not a replacement for the Cloud dashboard or the production API. It is a lab. Your product traffic still goes to https://api.scalattice.cloud/v1 with an slt_ key. Modelmeter is for the quieter work of choosing rates, models, and hosts with eyes open.
If you are wiring the API into an app, start with the OpenAI SDK guide or live pricing. If you want the bench we use internally, clone Modelmeter.