Working with ollama on a local GPU

[colltoaction]

Hi! I'm currently looking to set up a local environment with Ollama, running CUDA on my GPU. I have a few models running successfully including llama2 and nomic-embed-text.

I can run a chat session using the "LLM" mode, but I'm not able to use the rest. Different errors show up such as:

• "Query failed: shapes (7,1536) and (768,) not aligned: 1536 (dim 1) != 768 (dim 0)" (in "RAG" and "Graph-RAG")
• "Query failed: 'str' object has no attribute 'get'" (in Hyper-RAG).
• "Query failed: 'NoneType' object has no attribute 'get'" (in Hyper-RAG-Lite).

I'm using WSL2 on Windows and was able to get here by mounting the following settings.json file in docker-compose.yaml:

{
    "apiKey": "ollama",
    "modelProvider": "settings.custom_api",
    "modelName": "llama2",
    "baseUrl": "http://host.docker.internal:11434/v1",
    "selectedDatabase": "",
    "maxTokens": 2000,
    "temperature": 0.7,
    "embeddingModel": "nomic-embed-text",
    "embeddingDim": 768
}

Is there anything I should check to set up a complete environment? If you don't have access to this environment, let me know what tests you need me to perform.

Thank you!

[haoohu]

Thank you for your attention to Hyper-RAG. Your concerns can be referred to as follows:

• For the dim mismatch, you need to synchronously modify the default 1536 in backend/main.py to match different embedding models. You can also start the backend to view and locate the error.

• For object has no attribute 'get', the possible causes are as follows: 1. If you use your own documentation, check if test_insert_document() is commented out; 2. It is recommended to use a more powerful model to avoid format errors

In addition, it is recommended to give priority to experiencing hyperrag_demo.py to understand the component flow, which will further help you understand the web_ui.

[colltoaction]

Thanks Hao! I tested the demo and noticed the /embeddings endpoint as seen in the docs is not reached.

INFO:hyper_rag:[New Docs] inserting 1 docs
INFO:hyper_rag:[New Chunks] inserting 42 chunks
INFO:hyper_rag:Inserting 42 vectors to chunks
INFO:openai._base_client:Retrying request to /embeddings

If Ollama is not supported I would understand, given it is used through a compatibility layer. I tested several changes but haven't been able to make it work. When I use cURL or the ollama Python package the embeddings are returned:

np.array(ollama.embeddings(model='all-minilm', prompt="some text to embed")["embedding"])

I noticed the cURL API call uses /api/embeddings instead of /v1/embeddings but changing the base_url didn't work.

[colltoaction]

There were two problems:

1. To run the code outside docker I had to switch back to localhost
2. OpenAI compatibility for /v1/embeddings ollama/ollama-python#599

[colltoaction]

Sorry to spam you 🙏. I have this new error. Maybe it has to do with the embedding dimension, so I will check the backend/main.py as you suggested.

INFO:httpx:HTTP Request: POST http://localhost:11434/v1/embeddings "HTTP/1.1 200 OK"
INFO:hyper_rag:Writing hypergraph with 0 vertices, 0 hyperedges
Error occurred during insertion: index 0 is out of bounds for axis 0 with size 0. Retrying in 5 seconds...
Traceback (most recent call last):
  File "/home/colltoaction/GitHub/Hyper-RAG/examples/hyperrag_demo.py", line 72, in <module>
    insert_texts_with_retry(rag, texts)
  File "/home/colltoaction/GitHub/Hyper-RAG/examples/hyperrag_demo.py", line 43, in insert_texts_with_retry
    rag.insert(texts)
  File "/home/colltoaction/GitHub/Hyper-RAG/hyperrag/hyperrag.py", line 171, in insert
    return loop.run_until_complete(self.ainsert(string_or_strings))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/colltoaction/GitHub/Hyper-RAG/hyperrag/hyperrag.py", line 217, in ainsert
    await self.chunks_vdb.upsert(inserting_chunks)
  File "/home/colltoaction/GitHub/Hyper-RAG/hyperrag/storage.py", line 97, in upsert
    results = self._client.upsert(datas=list_data)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/colltoaction/GitHub/Hyper-RAG/.venv/lib/python3.12/site-packages/nano_vectordb/dbs.py", line 100, in upsert
    self.__storage["matrix"][i] = update_d[f_VECTOR].astype(Float)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^
IndexError: index 0 is out of bounds for axis 0 with size 0

[haoohu]

You might be able to first run the code using an OpenAI-type API, and then check where the problem lies when calling ollama.

[colltoaction]

I got a Gemini Pro account, thus having access to the API without rate-limiting. Note that Ollama also has an OpenAI compatible API just like Gemini, and I narrowed down the failure to missing HTTPS. Unfortunately it's not so simple, so I was wondering if Hyper-RAG could expose ssl-enforcing as a setting.

When running python examples/hyperrag_demo.py with the following settings:

LLM_BASE_URL = "https://generativelanguage.googleapis.com/v1beta/openai"
LLM_API_KEY = "XXXXXXXXXXXXXX"
LLM_MODEL = "gemini-3-pro-preview"

EMB_BASE_URL = "https://generativelanguage.googleapis.com/v1beta/openai"
EMB_API_KEY = "XXXXXXXXXXXXXX"
EMB_MODEL = "gemini-embedding-001"
EMB_DIM = 1536

I see:

Field(name=None,type=None,default=<dataclasses._MISSING_TYPE object at 0x7f3028571b80>,default_factory=<function HyperRAG.<lambda> at 0x7f302605e660>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),kw_only=<dataclasses._MISSING_TYPE object at 0x7f3028571b80>,_field_type=None)
INFO:hyper_rag:Logger initialized for working directory: caches/mock
INFO:hyper_rag:Load KV full_docs with 0 data
INFO:hyper_rag:Load KV text_chunks with 0 data
INFO:hyper_rag:Load KV llm_response_cache with 0 data
INFO:hyper_rag:Loaded hypergraph from caches/mock/hypergraph_chunk_entity_relation.hgdb with 0 vertices, 0 hyperedges
INFO:nano-vectordb:Load (0, 1536) data
INFO:nano-vectordb:Init {'embedding_dim': 1536, 'metric': 'cosine', 'storage_file': 'caches/mock/vdb_entities.json'} 0 data
INFO:nano-vectordb:Load (0, 1536) data
INFO:nano-vectordb:Init {'embedding_dim': 1536, 'metric': 'cosine', 'storage_file': 'caches/mock/vdb_relationships.json'} 0 data
INFO:nano-vectordb:Load (0, 1536) data
INFO:nano-vectordb:Init {'embedding_dim': 1536, 'metric': 'cosine', 'storage_file': 'caches/mock/vdb_chunks.json'} 42 data
INFO:hyper_rag:[New Docs] inserting 1 docs
INFO:hyper_rag:[New Chunks] inserting 42 chunks
INFO:hyper_rag:Inserting 42 vectors to chunks
INFO:httpx:HTTP Request: POST https://generativelanguage.googleapis.com/v1beta/openai/embeddings "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST https://generativelanguage.googleapis.com/v1beta/openai/embeddings "HTTP/1.1 200 OK"
INFO:hyper_rag:Writing hypergraph with 0 vertices, 0 hyperedges
Error occurred during insertion: index 0 is out of bounds for axis 0 with size 0. Retrying in 5 seconds...
INFO:hyper_rag:[New Docs] inserting 1 docs
INFO:hyper_rag:[New Chunks] inserting 42 chunks
INFO:hyper_rag:Inserting 42 vectors to chunks
INFO:httpx:HTTP Request: POST https://generativelanguage.googleapis.com/v1beta/openai/embeddings "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST https://generativelanguage.googleapis.com/v1beta/openai/embeddings "HTTP/1.1 200 OK"
INFO:hyper_rag:Writing hypergraph with 0 vertices, 0 hyperedges
Error occurred during insertion: index 0 is out of bounds for axis 0 with size 0. Retrying in 5 seconds...
INFO:hyper_rag:[New Docs] inserting 1 docs
INFO:hyper_rag:[New Chunks] inserting 42 chunks
INFO:hyper_rag:Inserting 42 vectors to chunks
INFO:httpx:HTTP Request: POST https://generativelanguage.googleapis.com/v1beta/openai/embeddings "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST https://generativelanguage.googleapis.com/v1beta/openai/embeddings "HTTP/1.1 200 OK"
INFO:hyper_rag:Writing hypergraph with 0 vertices, 0 hyperedges
Error occurred during insertion: index 0 is out of bounds for axis 0 with size 0. Retrying in 5 seconds...
Traceback (most recent call last):
  File "/home/colltoaction/GitHub/Hyper-RAG/examples/hyperrag_demo.py", line 71, in <module>
    insert_texts_with_retry(rag, texts)
  File "/home/colltoaction/GitHub/Hyper-RAG/examples/hyperrag_demo.py", line 50, in insert_texts_with_retry
    raise RuntimeError("Failed to insert texts after multiple retries.")
RuntimeError: Failed to insert texts after multiple retries.

After some debugging, namely removing the except block, I see:

  File "/home/colltoaction/GitHub/Hyper-RAG/hyperrag/hyperrag.py", line 217, in ainsert
    await self.chunks_vdb.upsert(inserting_chunks)
  File "/home/colltoaction/GitHub/Hyper-RAG/hyperrag/storage.py", line 97, in upsert
    results = self._client.upsert(datas=list_data)
  File "/home/colltoaction/GitHub/Hyper-RAG/.venv/lib/python3.12/site-packages/nano_vectordb/dbs.py", line 100, in upsert
    self.__storage["matrix"][i] = update_d[f_VECTOR].astype(Float)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^
IndexError: index 0 is out of bounds for axis 0 with size 0

Let me know if I can do anything else to help finding the issue.