← Blog

2026-11-16 · 5 min read

Setting Up vLLM and llama.cpp as OpenAI-Compatible Endpoints for Air Inference

Learn how to expose your vLLM or llama.cpp server as an OpenAI-compatible endpoint and list it on the Air Inference marketplace to reach a wider audience and monetize your AI models effectively.

Setting Up vLLM and llama.cpp as OpenAI-Compatible Endpoints for Air Inference

In the rapidly evolving landscape of artificial intelligence, developers and providers alike are seeking effective ways to connect and monetize their AI models. Air Inference offers a unique two-sided marketplace where developers can purchase access to AI inference services while providers can list their endpoints for a wider audience. This article will guide you through the process of setting up vLLM and llama.cpp as OpenAI-compatible endpoints, allowing you to expose your models to the Air Inference marketplace.

Understanding vLLM and llama.cpp

Before diving into the setup process, it's essential to grasp what vLLM and llama.cpp are.

  • vLLM: An efficient framework for serving large language models, vLLM optimizes memory usage and throughput, making it ideal for deploying models in production environments. Its ability to handle multiple requests concurrently can significantly enhance the user experience.

  • llama.cpp: A C++ implementation for running Meta’s LLaMA models, llama.cpp provides a lightweight and fast alternative to traditional Python-based frameworks. It’s particularly useful for developers looking to deploy models on systems with limited resources.

Both frameworks can be exposed as OpenAI-compatible endpoints, which means they can accept requests in the same format as the OpenAI API. This compatibility allows developers using Air Inference to seamlessly integrate your models into their applications.

Prerequisites

Before you begin, ensure you have the following prerequisites:

  1. Basic Understanding of APIs: Familiarity with RESTful APIs and how to make HTTP requests.
  2. Environment Setup: A server or local machine where you can run vLLM or llama.cpp.
  3. Installed Dependencies: Ensure you have Python, Docker, or any other required software installed for your chosen framework.
  4. Air Credits: If you are a developer looking to use the services, make sure to top up your Air Credits on Air Inference.

Step 1: Setting Up vLLM

1.1 Installation

To start with vLLM, you need to install it. You can do this via pip:

pip install vllm

1.2 Running the Server

Once installed, you can run the vLLM server. The following command starts the server and exposes the model on a specific port:

vllm serve --model <model_name> --port 8000

Replace <model_name> with the name of the model you wish to serve. This command sets up an inference server that listens on port 8000.

1.3 Creating an OpenAI-Compatible API

To expose your vLLM server as an OpenAI-compatible endpoint, you'll need to create a simple wrapper around it. This can be done using Flask, FastAPI, or any other web framework that allows you to handle HTTP requests. Below is a basic example using Flask:

from flask import Flask, request, jsonify
import requests

app = Flask(__name__)

@app.route('/api/v1/completions', methods=['POST'])
def completions():
    data = request.json
    model = data['model']
    prompt = data['prompt']
    max_tokens = data.get('max_tokens', 50)

    # Forward the request to the vLLM server
    response = requests.post(f'http://localhost:8000/generate', json={
        "model": model,
        "prompt": prompt,
        "max_tokens": max_tokens
    })

    return jsonify(response.json())

if __name__ == '__main__':
    app.run(port=5000)

In this example, your Flask app listens on port 5000 and forwards requests to the vLLM server running on port 8000.

1.4 Testing the Endpoint

You can test your endpoint using curl to ensure it's working correctly:

curl -X POST http://localhost:5000/api/v1/completions \
-H "Content-Type: application/json" \
-d '{"model": "your_model_name", "prompt": "Hello, world!", "max_tokens": 50}'

Replace "your_model_name" with the name of your vLLM model. You should receive a response from your vLLM server.

Step 2: Setting Up llama.cpp

2.1 Installation

To set up llama.cpp, you'll need to clone the repository and build the project. First, ensure you have CMake and a C++ compiler installed.

git clone https://github.com/your_username/llama.cpp.git
cd llama.cpp
mkdir build
cd build
cmake ..
make

2.2 Running the Server

After building, you can run the llama.cpp server. The command might look something like this:

./llama_server --model <model_path> --port 8001

Replace <model_path> with the path to your LLaMA model files.

2.3 Creating an OpenAI-Compatible API

Similar to vLLM, you’ll need to create a wrapper around the llama.cpp server. Here’s an example using Flask:

from flask import Flask, request, jsonify
import requests

app = Flask(__name__)

@app.route('/api/v1/completions', methods=['POST'])
def completions():
    data = request.json
    model = data['model']
    prompt = data['prompt']
    max_tokens = data.get('max_tokens', 50)

    # Forward the request to the llama.cpp server
    response = requests.post(f'http://localhost:8001/generate', json={
        "model": model,
        "prompt": prompt,
        "max_tokens": max_tokens
    })

    return jsonify(response.json())

if __name__ == '__main__':
    app.run(port=5000)

2.4 Testing the Endpoint

Just like with the vLLM setup, you can test the llama.cpp endpoint using curl:

curl -X POST http://localhost:5000/api/v1/completions \
-H "Content-Type: application/json" \
-d '{"model": "your_model_name", "prompt": "Hello, world!", "max_tokens": 50}'

Again, ensure you replace "your_model_name" with the appropriate model name.

Step 3: Listing on Air Inference Marketplace

Now that you have your OpenAI-compatible endpoints set up, it’s time to list them on the Air Inference marketplace. Here’s how to do it:

3.1 Create an Account

If you haven’t done so already, sign up for an account on Air Inference. After logging in, navigate to the provider dashboard.

3.2 Listing Your Endpoint

  1. Add a New Endpoint: Click on the "Add Endpoint" button.
  2. Fill in the Details: Provide the necessary information, including:
    • Endpoint Name: A descriptive name for your endpoint.
    • Description: Explain what your model does and any unique features.
    • API URL: Input the URL where your Flask app is running (e.g., http://your-server-ip:5000/api/v1/completions).
    • Pricing: Set your pricing structure. Remember that Air Inference charges a ~10% fee on off-platform payments.
  3. Submit for Review: Once you fill out the form, submit it for review. Air Inference will check your endpoint for compliance with their guidelines.

3.3 Promote Your Endpoint

After your endpoint goes live, you can promote it via social media, developer forums, or AI communities to attract developers looking for AI inference solutions.

Conclusion

By setting up vLLM and llama.cpp as OpenAI-compatible endpoints, you can effectively expose your AI models to a broader audience via the Air Inference marketplace. This not only helps you monetize your models but also contributes to the growing ecosystem of AI solutions. With the steps outlined in this guide, you should be well-equipped to get your models up and running on Air Inference.

Remember to keep your API documentation handy and be responsive to user feedback to improve your offerings continually. Happy coding!