Playground
# My OpenAI Key
import os
os.environ['OPENAI_API_KEY'] = "INSERT OPENAI KEY"
# Hide INFO logs regarding token usage, etc
import logging
logger = logging.getLogger()
logger.setLevel(logging.CRITICAL)
Setup
Generate some example Documents
from llama_index import download_loader
from llama_index.indices.vector_store import VectorStoreIndex
from llama_index.indices.tree.base import TreeIndex
WikipediaReader = download_loader("WikipediaReader")
loader = WikipediaReader()
documents = loader.load_data(pages=['Berlin'])
Create a list of any sort of indices (custom LLMs, custom embeddings, etc)
indices = [VectorStoreIndex.from_documents(documents), TreeIndex.from_documents(documents)]
INFO:root:> [build_index_from_documents] Total LLM token usage: 0 tokens
INFO:root:> [build_index_from_documents] Total embedding token usage: 18344 tokens
INFO:root:> Building index from nodes: 5 chunks
Using the Playground
Initialize with indices
from llama_index.playground import Playground
playground = Playground(indices=indices)
playground.compare("What is the population of Berlin?")
INFO:openai:error_code=None error_message='Rate limit reached for default-global-with-image-limits in organization org-ehTdCqs0FpsxuTTwsJIlNSdZ on requests per min. Limit: 60.000000 / min. Current: 110.000000 / min. Contact support@openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method.' error_param=None error_type=requests message='OpenAI API error received' stream_error=False
Query:
What is the population of Berlin?
Trying 10 combinations...
GPTVectorStoreIndex, mode = default
INFO:openai:error_code=None error_message='Rate limit reached for default-global-with-image-limits in organization org-ehTdCqs0FpsxuTTwsJIlNSdZ on requests per min. Limit: 60.000000 / min. Current: 90.000000 / min. Contact support@openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method.' error_param=None error_type=requests message='OpenAI API error received' stream_error=False
INFO:openai:error_code=None error_message='Rate limit reached for default-global-with-image-limits in organization org-ehTdCqs0FpsxuTTwsJIlNSdZ on requests per min. Limit: 60.000000 / min. Current: 90.000000 / min. Contact support@openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method.' error_param=None error_type=requests message='OpenAI API error received' stream_error=False
INFO:openai:error_code=None error_message='Rate limit reached for default-global-with-image-limits in organization org-ehTdCqs0FpsxuTTwsJIlNSdZ on requests per min. Limit: 60.000000 / min. Current: 80.000000 / min. Contact support@openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method.' error_param=None error_type=requests message='OpenAI API error received' stream_error=False
INFO:root:> [query] Total LLM token usage: 3545 tokens
INFO:root:> [query] Total embedding token usage: 7 tokens
The population of Berlin in 1949 was approximately 2.2 million inhabitants. After the fall of the Berlin Wall in 1989, the population of Berlin increased to approximately 3.7 million inhabitants.
GPTVectorStoreIndex, mode = embedding
INFO:root:> [query] Total LLM token usage: 3545 tokens
INFO:root:> [query] Total embedding token usage: 7 tokens
INFO:root:> Starting query: What is the population of Berlin?
The population of Berlin in 1949 was approximately 2.2 million inhabitants. After the fall of the Berlin Wall in 1989, the population of Berlin increased to approximately 3.7 million inhabitants.
GPTTreeIndex, mode = default
INFO:root:>[Level 0] Selected node: [1]/[1]
INFO:root:>[Level 1] Selected node: [3]/[3]
INFO:root:> [query] Total LLM token usage: 5168 tokens
INFO:root:> [query] Total embedding token usage: 0 tokens
INFO:root:> Starting query: What is the population of Berlin?
INFO:root:> Building index from nodes: 6 chunks
The population of Berlin is approximately 3.7 million people.
GPTTreeIndex, mode = summarize
INFO:root:> [query] Total LLM token usage: 21617 tokens
INFO:root:> [query] Total embedding token usage: 0 tokens
INFO:root:> Starting query: What is the population of Berlin?
The population of Berlin is approximately 3.7 million people.
GPTTreeIndex, mode = embedding
INFO:root:> [query] Total LLM token usage: 368 tokens
INFO:root:> [query] Total embedding token usage: 4598 tokens
INFO:root:> Starting query: What is the population of Berlin?
Approximately 3.7 million people.
GPTTreeIndex, mode = retrieve
INFO:root:> [query] Total LLM token usage: 1439 tokens
INFO:root:> [query] Total embedding token usage: 0 tokens
The population of Berlin is 3.75 million registered inhabitants.
Ran 6 combinations in total.
| Index | Mode | Output | Duration | LLM Tokens | Embedding Tokens | |
|---|---|---|---|---|---|---|
| 0 | VectorStoreIndex | default | \nThe population of Berlin in 1949 was approxi... | 52.319133 | 3545 | 7 |
| 1 | VectorStoreIndex | embedding | \nThe population of Berlin in 1949 was approxi... | 8.192025 | 3545 | 7 |
| 2 | TreeIndex | default | The population of Berlin is approximately 3.7 ... | 12.542335 | 5168 | 0 |
| 3 | TreeIndex | summarize | \nThe population of Berlin is approximately 3.... | 18.665586 | 21617 | 0 |
| 4 | TreeIndex | embedding | Approximately 3.7 million people. | 3.573458 | 368 | 4598 |
| 5 | TreeIndex | retrieve | \nThe population of Berlin is 3.75 million reg... | 2.269598 | 1439 | 0 |
Initialize with Documents
# Uses documents in a preset list of indices
playground = Playground.from_docs(documents=documents)