Callbacks

class llama_index.callbacks.AimCallback(repo: Optional[str] = None, experiment_name: Optional[str] = None, system_tracking_interval: Optional[int] = 1, log_system_params: Optional[bool] = True, capture_terminal_logs: Optional[bool] = True, event_starts_to_ignore: Optional[List[CBEventType]] = None, event_ends_to_ignore: Optional[List[CBEventType]] = None, run_params: Optional[Dict[str, Any]] = None)

AimCallback callback class.

Parameters

  • repo (str, optional) – Aim repository path or Repo object to which Run object is bound. If skipped, default Repo is used.

  • experiment_name (str, optional) – Sets Run’s experiment property. β€˜default’ if not specified. Can be used later to query runs/sequences.

  • system_tracking_interval (int, optional) – Sets the tracking interval in seconds for system usage metrics (CPU, Memory, etc.). Set to None to disable system metrics tracking.

  • log_system_params (bool, optional) – Enable/Disable logging of system params such as installed packages, git info, environment variables, etc.

  • capture_terminal_logs (bool, optional) – Enable/Disable terminal stdout logging.

  • event_starts_to_ignore (Optional[List[CBEventType]]) – list of event types to ignore when tracking event starts.

  • event_ends_to_ignore (Optional[List[CBEventType]]) – list of event types to ignore when tracking event ends.

end_trace(trace_id: Optional[str] = None, trace_map: Optional[Dict[str, List[str]]] = None) → None

Run when an overall trace is exited.

on_event_end(event_type: CBEventType, payload: Optional[Dict[str, Any]] = None, event_id: str = '', **kwargs: Any) → None
Parameters

  • event_type (CBEventType) – event type to store.

  • payload (Optional[Dict[str, Any]]) – payload to store.

  • event_id (str) – event id to store.

on_event_start(event_type: CBEventType, payload: Optional[Dict[str, Any]] = None, event_id: str = '', **kwargs: Any) → str
Parameters

  • event_type (CBEventType) – event type to store.

  • payload (Optional[Dict[str, Any]]) – payload to store.

  • event_id (str) – event id to store.

start_trace(trace_id: Optional[str] = None) → None

Run when an overall trace is launched.

class llama_index.callbacks.CBEvent(event_type: CBEventType, payload: Optional[Dict[str, Any]] = None, time: str = '', id_: str = '')

Generic class to store event information.

class llama_index.callbacks.CBEventType(value)

Callback manager event types.

CHUNKING

Logs for the before and after of text splitting.

NODE_PARSING

Logs for the documents and the nodes that they are parsed into.

EMBEDDING

Logs for the number of texts embedded.

LLM

Logs for the template and response of LLM calls.

QUERY

Keeps track of the start and end of each query.

RETRIEVE

Logs for the nodes retrieved for a query.

SYNTHESIZE

Logs for the result for synthesize calls.

TREE

Logs for the summary and level of summaries generated.

class llama_index.callbacks.CallbackManager(handlers: List[BaseCallbackHandler])

Callback manager that handles callbacks for events within LlamaIndex.

The callback manager provides a way to call handlers on event starts/ends.

Additionally, the callback manager traces the current stack of events. It does this by using a few key attributes. - trace_stack - The current stack of events that have not ended yet.

When an event ends, it’s remove from the stack.

  • trace_map - A mapping of event ids to their children events.

    On the start of events, the bottom of the trace stack is used as the current parent event for the trace map.

  • trace_id - A simple name for the current trace, usually denoting the

    entrypoint (query, index_construction, insert, etc.)

Parameters

handlers (List[BaseCallbackHandler]) – list of handlers to use.

add_handler(handler: BaseCallbackHandler) → None

Add a handler to the callback manager.

as_trace(trace_id: str) → Generator[None, None, None]

Context manager tracer for lanching and shutdown of traces.

end_trace(trace_id: Optional[str] = None, trace_map: Optional[Dict[str, List[str]]] = None) → None

Run when an overall trace is exited.

on_event_end(event_type: CBEventType, payload: Optional[Dict[str, Any]] = None, event_id: str = '', **kwargs: Any) → None

Run handlers when an event ends.

on_event_start(event_type: CBEventType, payload: Optional[Dict[str, Any]] = None, event_id: str = '', **kwargs: Any) → str

Run handlers when an event starts and return id of event.

remove_handler(handler: BaseCallbackHandler) → None

Remove a handler from the callback manager.

set_handlers(handlers: List[BaseCallbackHandler]) → None

Set handlers as the only handlers on the callback manager.

start_trace(trace_id: Optional[str] = None) → None

Run when an overall trace is launched.

class llama_index.callbacks.LlamaDebugHandler(event_starts_to_ignore: Optional[List[CBEventType]] = None, event_ends_to_ignore: Optional[List[CBEventType]] = None, print_trace_on_end: bool = True)

Callback handler that keeps track of debug info.

NOTE: this is a beta feature. The usage within our codebase, and the interface may change.

This handler simply keeps track of event starts/ends, separated by event types. You can use this callback handler to keep track of and debug events.

Parameters

  • event_starts_to_ignore (Optional[List[CBEventType]]) – list of event types to ignore when tracking event starts.

  • event_ends_to_ignore (Optional[List[CBEventType]]) – list of event types to ignore when tracking event ends.

end_trace(trace_id: Optional[str] = None, trace_map: Optional[Dict[str, List[str]]] = None) → None

Shutdown the current trace.

flush_event_logs() → None

Clear all events from memory.

get_event_pairs(event_type: Optional[CBEventType] = None) → List[List[CBEvent]]

Pair events by ID, either all events or a sepcific type.

get_events(event_type: Optional[CBEventType] = None) → List[CBEvent]

Get all events for a specific event type.

get_llm_inputs_outputs() → List[List[CBEvent]]

Get the exact LLM inputs and outputs.

on_event_end(event_type: CBEventType, payload: Optional[Dict[str, Any]] = None, event_id: str = '', **kwargs: Any) → None

Store event end data by event type.

Parameters

  • event_type (CBEventType) – event type to store.

  • payload (Optional[Dict[str, Any]]) – payload to store.

  • event_id (str) – event id to store.

on_event_start(event_type: CBEventType, payload: Optional[Dict[str, Any]] = None, event_id: str = '', **kwargs: Any) → str

Store event start data by event type.

Parameters

  • event_type (CBEventType) – event type to store.

  • payload (Optional[Dict[str, Any]]) – payload to store.

  • event_id (str) – event id to store.

print_trace_map() → None

Print simple trace map to terminal for debugging of the most recent trace.

start_trace(trace_id: Optional[str] = None) → None

Launch a trace.

class llama_index.callbacks.WandbCallbackHandler(run_args: Optional[WandbRunArgs] = None, event_starts_to_ignore: Optional[List[CBEventType]] = None, event_ends_to_ignore: Optional[List[CBEventType]] = None)

Callback handler that logs events to wandb.

NOTE: this is a beta feature. The usage within our codebase, and the interface may change.

Use the WandbCallbackHandler to log trace events to wandb. This handler is useful for debugging and visualizing the trace events. It captures the payload of the events and logs them to wandb. The handler also tracks the start and end of events. This is particularly useful for debugging your LLM calls.

The WandbCallbackHandler can also be used to log the indices and graphs to wandb using the persist_index method. This will save the indexes as artifacts in wandb. The load_storage_context method can be used to load the indexes from wandb artifacts. This method will return a StorageContext object that can be used to build the index, using load_index_from_storage, load_indices_from_storage or load_graph_from_storage functions.

Parameters

  • event_starts_to_ignore (Optional[List[CBEventType]]) – list of event types to ignore when tracking event starts.

  • event_ends_to_ignore (Optional[List[CBEventType]]) – list of event types to ignore when tracking event ends.

end_trace(trace_id: Optional[str] = None, trace_map: Optional[Dict[str, List[str]]] = None) → None

Run when an overall trace is exited.

finish() → None

Finish the callback handler.

load_storage_context(artifact_url: str, index_download_dir: Optional[str] = None) → StorageContext

Download an index from wandb and return a storage context.

Use this storage context to load the index into memory using load_index_from_storage, load_indices_from_storage or load_graph_from_storage functions.

Parameters

  • artifact_url (str) – url of the artifact to download. The artifact url will be of the form: entity/project/index_name:version and can be found in the W&B UI.

  • index_download_dir (Union[str, None]) – directory to download the index to.

log_trace_tree() → None

Log the trace tree to wandb.

on_event_end(event_type: CBEventType, payload: Optional[Dict[str, Any]] = None, event_id: str = '', **kwargs: Any) → None

Store event end data by event type.

Parameters

  • event_type (CBEventType) – event type to store.

  • payload (Optional[Dict[str, Any]]) – payload to store.

  • event_id (str) – event id to store.

on_event_start(event_type: CBEventType, payload: Optional[Dict[str, Any]] = None, event_id: str = '', **kwargs: Any) → str

Store event start data by event type.

Parameters

  • event_type (CBEventType) – event type to store.

  • payload (Optional[Dict[str, Any]]) – payload to store.

  • event_id (str) – event id to store.

persist_index(index: IndexType, index_name: str, persist_dir: Optional[str] = None) → None

Upload an index to wandb as an artifact. You can learn more about W&B artifacts here: https://docs.wandb.ai/guides/artifacts.

For the ComposableGraph index, the root id is stored as artifact metadata.

Parameters

  • index (IndexType) – index to upload.

  • index_name (str) – name of the index. This will be used as the artifact name.

  • persist_dir (Union[str, None]) – directory to persist the index. If None, a temporary directory will be created and used.

start_trace(trace_id: Optional[str] = None) → None

Launch a trace.