Netra Logo
Pricing About

Learn / How to use Netra Runtime with LlamaIndex

How to use Netra Runtime with LlamaIndex

LlamaIndex is a data framework for LLM applications, best known for retrieval-augmented generation: it ingests your documents, indexes them, and answers questions over them. The generation side runs on any OpenAI-compatible endpoint through its OpenAILike class, so a model served by Netra Runtime can answer over your private data without that data leaving infrastructure you control.

What you need

  • The integration package: pip install llama-index llama-index-llms-openai-like (see the LlamaIndex docs).
  • A Netra Runtime API key from your dashboard at app.netraruntime.com. The API base URL is https://api.netraruntime.com/v1 (self-hosted deployments use their own endpoint URL instead).

Step 1: configure the LLM

Use OpenAILike rather than the plain OpenAI class. The plain class validates model names against OpenAI's catalog and rejects custom ids; OpenAILike accepts any OpenAI-compatible endpoint:

from llama_index.llms.openai_like import OpenAILike

llm = OpenAILike(
    model="qwen3.6-35b",
    api_base="https://api.netraruntime.com/v1",
    api_key="YOUR_NETRA_API_KEY",
    is_chat_model=True,
    context_window=128000,
)

print(llm.complete("Say hello from Netra Runtime."))

Set context_window to your deployed model's real window; LlamaIndex uses it to budget how much retrieved context fits into each prompt. Learn how context size drives memory on the serving side.

Step 2: make it the default

Register the LLM in global settings and every engine built afterwards uses it:

from llama_index.core import Settings, SimpleDirectoryReader, VectorStoreIndex

Settings.llm = llm

documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(documents)

query_engine = index.as_query_engine()
print(query_engine.query("What do these documents say about pricing?"))

Retrieval finds the relevant chunks, and your Netra-served model writes the answer. Chat engines, agents, and streaming (as_query_engine(streaming=True)) all work the same way.

Embeddings

Indexing needs an embedding model, and by default LlamaIndex looks for OpenAI. If your Netra deployment serves an embeddings endpoint, point the embedding class at the same base URL and key. Otherwise, a local embedding model (for example via llama-index-embeddings-huggingface) keeps the whole pipeline on infrastructure you control; only generation runs on your Netra endpoint.

Frequently asked questions

Why use OpenAILike instead of the OpenAI class in LlamaIndex?

The OpenAI class validates model names against OpenAI's own catalog, so custom model ids are rejected. OpenAILike skips that validation and works with any OpenAI-compatible endpoint, which is exactly the Netra Runtime case.

Can Netra Runtime power a full RAG pipeline in LlamaIndex?

Yes. Set the LLM in LlamaIndex Settings and every query engine and chat engine uses it. If your deployment also serves embeddings, configure the embedding model with the same base URL and key.

Do I need is_chat_model=True?

Yes, for chat-style models. It tells LlamaIndex to call the chat completions endpoint instead of legacy text completions, which is what modern instruction-tuned models expect.

Related