Open In Colab

Joint Tabular/Semantic QA over Tesla 10K#

In this example, we show how to ask questions over 10K with understanding of both the unstructured text as well as embedded tables.

We use Unstructured to parse out the tables, and use LlamaIndex recursive retrieval to index/retrieve tables if necessary given the user question.

If you’re opening this Notebook on colab, you will probably need to install LlamaIndex πŸ¦™.

!pip install llama-index
%load_ext autoreload
%autoreload 2
from pydantic import BaseModel
from unstructured.partition.html import partition_html
import pandas as pd

pd.set_option("display.max_rows", None)
pd.set_option("display.max_columns", None)
pd.set_option("display.width", None)
pd.set_option("display.max_colwidth", None)

Perform Data Extraction#

In these sections we use Unstructured to parse out the table and non-table elements.

Extract Elements#

We use Unstructured to extract table and non-table elements from the 10-K filing.

!wget "https://www.dropbox.com/scl/fi/mlaymdy1ni1ovyeykhhuk/tesla_2021_10k.htm?rlkey=qf9k4zn0ejrbm716j0gg7r802&dl=1" -O tesla_2021_10k.htm
!wget "https://www.dropbox.com/scl/fi/rkw0u959yb4w8vlzz76sa/tesla_2020_10k.htm?rlkey=tfkdshswpoupav5tqigwz1mp7&dl=1" -O tesla_2020_10k.htm
from llama_index.readers.file.flat_reader import FlatReader
from pathlib import Path

reader = FlatReader()
docs_2021 = reader.load_data(Path("tesla_2021_10k.htm"))
docs_2020 = reader.load_data(Path("tesla_2020_10k.htm"))
from llama_index.node_parser import (
    UnstructuredElementNodeParser,
)

node_parser = UnstructuredElementNodeParser()
import os
import pickle

if not os.path.exists("2021_nodes.pkl"):
    raw_nodes_2021 = node_parser.get_nodes_from_documents(docs_2021)
    pickle.dump(raw_nodes_2021, open("2021_nodes.pkl", "wb"))
else:
    raw_nodes_2021 = pickle.load(open("2021_nodes.pkl", "rb"))
100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 105/105 [14:59<00:00,  8.56s/it]
base_nodes_2021, node_mappings_2021 = node_parser.get_base_nodes_and_mappings(
    raw_nodes_2021
)
example_index_node = [b for b in base_nodes_2021 if isinstance(b, IndexNode)][
    20
]

# Index Node
print(
    f"\n--------\n{example_index_node.get_content(metadata_mode='all')}\n--------\n"
)
# Index Node ID
print(f"\n--------\nIndex ID: {example_index_node.index_id}\n--------\n")
# Referenceed Table
print(
    f"\n--------\n{node_mappings_2021[example_index_node.index_id].get_content()}\n--------\n"
)
--------
col_schema: Column: Type
Type: string
Summary: Type of net income (loss) per share calculation (basic or diluted)

Column: Amount
Type: string
Summary: Net income (loss) per share amount

Column: Weighted Average Shares
Type: string
Summary: Number of shares used in calculating net income (loss) per share

Summary of net income (loss) per share of common stock attributable to common stockholders
--------


--------
Index ID: id_617_table
--------


--------
                                                                                                                                                                                                                                             
0                                                                                                                       Year Ended December 31,                                                                                              
1                                                                                                                                          2021                      2020                      2019                                          
2                                                    Revenues                                                                                                                                                                                
3                                            Automotive sales                                                                                 $    44,125                    $    24,604                     $    19,358                     
4                               Automotive regulatory credits                                                                                       1,465                          1,580                             594                     
5                                          Automotive leasing                                                                                       1,642                          1,052                             869                     
6                                   Total automotive revenues                                                                                      47,232                         27,236                          20,821                     
7                               Energy generation and storage                                                                                       2,789                          1,994                           1,531                     
8                                          Services and other                                                                                       3,802                          2,306                           2,226                     
9                                              Total revenues                                                                                      53,823                         31,536                          24,578                     
10                                           Cost of revenues                                                                                                                                                                                
11                                           Automotive sales                                                                                      32,415                         19,696                          15,939                     
12                                         Automotive leasing                                                                                         978                            563                             459                     
13                          Total automotive cost of revenues                                                                                      33,393                         20,259                          16,398                     
14                              Energy generation and storage                                                                                       2,918                          1,976                           1,341                     
15                                         Services and other                                                                                       3,906                          2,671                           2,770                     
16                                     Total cost of revenues                                                                                      40,217                         24,906                          20,509                     
17                                               Gross profit                                                                                      13,606                          6,630                           4,069                     
18                                         Operating expenses                                                                                                                                                                                
19                                   Research and development                                                                                       2,593                          1,491                           1,343                     
20                        Selling, general and administrative                                                                                       4,517                          3,145                           2,646                     
21                                    Restructuring and other                                                                                           (   27    )                         β€”                              149               
22                                   Total operating expenses                                                                                       7,083                          4,636                           4,138                     
23                              Income (loss) from operations                                                                                       6,523                          1,994                               (    69       )       
24                                            Interest income                                                                                          56                             30                              44                     
25                                           Interest expense                                                                                           (  371    )                         (   748     )                       (  685    )  
26                                Other income (expense), net                                                                                         135                              (  122        )                      45               
27                          Income (loss) before income taxes                                                                                       6,343                          1,154                               (   665       )       
28                                 Provision for income taxes                                                                                         699                            292                             110                     
29                                          Net income (loss)                                                                                       5,644                            862                               (   775       )       
30    Net income attributable to noncontrolling interests and  redeemable noncontrolling interests in subsidiaries                                         125                            141                               87               
31      Net income (loss) attributable to common stockholders                                                                                 $     5,519                    $       721                     $         (   862       )       
32                                                                                                                                                                                                                                           
33                Net income (loss) per share of common stock                  attributable to common stockholders                                                                                                                           
34                                                      Basic                                                                                 $      5.60                    $      0.74                     $         (  0.98       )       
35                                                    Diluted                                                                                 $      4.90                    $      0.64                     $         (  0.98       )       
36              Weighted average shares used in computing net              income (loss) per share of common stock                                                                                                                           
37                                                      Basic                                                                                         986                            933                             887                     
38                                                    Diluted                                                                                       1,129                          1,083                             887                     
--------

Setup Recursive Retriever#

Now that we’ve extracted tables and their summaries, we can setup a recursive retriever in LlamaIndex to query these tables.

Construct Retrievers#

from llama_index.retrievers import RecursiveRetriever
from llama_index.query_engine import RetrieverQueryEngine
from llama_index import VectorStoreIndex
# construct top-level vector index + query engine
vector_index = VectorStoreIndex(base_nodes_2021)
vector_retriever = vector_index.as_retriever(similarity_top_k=1)
vector_query_engine = vector_index.as_query_engine(similarity_top_k=1)
from llama_index.retrievers import RecursiveRetriever

recursive_retriever = RecursiveRetriever(
    "vector",
    retriever_dict={"vector": vector_retriever},
    node_dict=node_mappings_2021,
    verbose=True,
)
query_engine = RetrieverQueryEngine.from_args(recursive_retriever)

Run some Queries#

response = query_engine.query("What was the revenue in 2020?")
print(str(response))
Retrieving with query id None: What was the revenue in 2020?
Retrieved node with id, entering: id_478_table
Retrieving with query id id_478_table: What was the revenue in 2020?
The revenue in 2020 was $31,536 million.
# compare against the baseline retriever
response = vector_query_engine.query("What was the revenue in 2020?")
print(str(response))
The revenue in 2020 was a number.
response = query_engine.query("What were the total cash flows in 2021?")
Retrieving with query id None: What were the total cash flows in 2021?
Retrieved node with id, entering: id_558_table
Retrieving with query id id_558_table: What were the total cash flows in 2021?

print(str(response))
The total cash flows in 2021 were $11,497 million.
response = vector_query_engine.query("What were the total cash flows in 2021?")
print(str(response))
The total cash flows in 2021 cannot be determined based on the given context information.
response = query_engine.query("What are the risk factors for Tesla?")
print(str(response))
Retrieving with query id None: What are the risk factors for Tesla?
Retrieving text node: Employees may leave Tesla or choose other employers over Tesla due to various factors, such as a very competitive labor market for talented individuals with automotive or technology experience, or any negative publicity related to us. In regions where we

19

have or will have operations, particularly significant engineering and manufacturing centers, there is strong competition for individuals with skillsets needed for our business, including specialized knowledge of electric vehicles, engineering and electrical and building construction expertise. Moreover, we may be impacted by perceptions relating to reductions in force that we have conducted in the past in order to optimize our organizational structure and reduce costs and the departure of certain senior personnel for various reasons. Likewise, as a result of our temporary suspension of various U.S. manufacturing operations in the first half of 2020, in April 2020, we temporarily furloughed certain hourly employees and reduced most salaried employees’ base salaries. We also compete with both mature and prosperous companies that have far greater financial resources than we do and start-ups and emerging companies that promise short-term growth opportunities.

Finally, our compensation philosophy for all of our personnel reflects our startup origins, with an emphasis on equity-based awards and benefits in order to closely align their incentives with the long-term interests of our stockholders. We periodically seek and obtain approval from our stockholders for future increases to the number of awards available under our equity incentive and employee stock purchase plans. If we are unable to obtain the requisite stockholder approvals for such future increases, we may have to expend additional cash to compensate our employees and our ability to retain and hire qualified personnel may be harmed.

We are highly dependent on the services of Elon Musk, Technoking of Tesla and our Chief Executive Officer.

We are highly dependent on the services of Elon Musk, Technoking of Tesla and our Chief Executive Officer. Although Mr. Musk spends significant time with Tesla and is highly active in our management, he does not devote his full time and attention to Tesla. Mr. Musk also currently serves as Chief Executive Officer and Chief Technical Officer of Space Exploration Technologies Corp., a developer and manufacturer of space launch vehicles, and is involved in other emerging technology ventures.

Our information technology systems or data, or those of our service providers or customers or users could be subject to cyber-attacks or other security incidents, which could result in data breaches, intellectual property theft, claims, litigation, regulatory investigations, significant liability, reputational damage and other adverse consequences.

We continue to expand our information technology systems as our operations grow, such as product data management, procurement, inventory management, production planning and execution, sales, service and logistics, dealer management, financial, tax and regulatory compliance systems. This includes the implementation of new internally developed systems and the deployment of such systems in the U.S. and abroad. While, we maintain information technology measures designed to protect us against intellectual property theft, data breaches, sabotage and other external or internal cyber-attacks or misappropriation, our systems and those of our service providers are potentially vulnerable to malware, ransomware, viruses, denial-of-service attacks, phishing attacks, social engineering, computer hacking, unauthorized access, exploitation of bugs, defects and vulnerabilities, breakdowns, damage, interruptions, system malfunctions, power outages, terrorism, acts of vandalism, security breaches, security incidents, inadvertent or intentional actions by employees or other third parties, and other cyber-attacks.

To the extent any security incident results in unauthorized access or damage to or acquisition, use, corruption, loss, destruction, alteration or dissemination of our data, including intellectual property and personal information, or our products or vehicles, or for it to be believed or reported that any of these occurred, it could disrupt our business, harm our reputation, compel us to comply with applicable data breach notification laws, subject us to time consuming, distracting and expensive litigation, regulatory investigation and oversight, mandatory corrective action, require us to verify the correctness of database contents, or otherwise subject us to liability under laws, regulations and contractual obligations, including those that protect the privacy and security of personal information. This could result in increased costs to us and result in significant legal and financial exposure and/or reputational harm.

We also rely on service providers, and similar incidents relating to their information technology systems could also have a material adverse effect on our business. There have been and may continue to be significant supply chain attacks. Our service providers, including our workforce management software provider, have been subject to ransomware and other security incidents, and we cannot guarantee that our or our service providers’ systems have not been breached or that they do not contain exploitable defects, bugs, or vulnerabilities that could result in a security incident, or other disruption to, our or our service providers’ systems. Our ability to monitor our service providers’ security measures is limited, and, in any event, malicious third parties may be able to circumvent those security measures.
The risk factors for Tesla include a highly competitive labor market for skilled individuals in the automotive and technology sectors, negative publicity, competition for individuals with specialized knowledge in electric vehicles and engineering, perceptions related to past reductions in force and departure of senior personnel, competition from companies with greater financial resources, dependence on the services of Elon Musk as CEO, potential cyber-attacks or security incidents leading to data breaches and reputational damage, and reliance on service providers who may be vulnerable to security incidents.
response = vector_query_engine.query("What are the risk factors for Tesla?")
print(str(response))
The risk factors for Tesla include strong competition for skilled individuals in the labor market, negative publicity, potential impacts from reductions in force and departure of senior personnel, competition from companies with greater financial resources, dependence on the services of Elon Musk, potential cyber-attacks or security incidents, and reliance on service providers who may be vulnerable to security breaches. These factors could disrupt Tesla's business, harm its reputation, result in legal and financial exposure, and impact its ability to retain and hire qualified personnel.

Try Table Comparisons#

In this setting we load in both the 2021 and 2020 10K filings, parse each into a hierarchy of tables/text objects, define a recursive retriever over each, and then compose both with a SubQuestionQueryEngine.

This allows us to execute document comparisons against both.

Define E2E Recursive Retriever Function#

import pickle
import os


def create_recursive_retriever_over_doc(docs, nodes_save_path=None):
    """Big function to go from document path -> recursive retriever."""
    node_parser = UnstructuredElementNodeParser()
    if nodes_save_path is not None and os.path.exists(nodes_save_path):
        raw_nodes = pickle.load(open(nodes_save_path, "rb"))
    else:
        raw_nodes = node_parser.get_nodes_from_documents(docs)
        if nodes_save_path is not None:
            pickle.dump(raw_nodes, open(nodes_save_path, "wb"))

    base_nodes, node_mappings = node_parser.get_base_nodes_and_mappings(
        raw_nodes
    )

    ### Construct Retrievers
    # construct top-level vector index + query engine
    vector_index = VectorStoreIndex(base_nodes)
    vector_retriever = vector_index.as_retriever(similarity_top_k=2)
    recursive_retriever = RecursiveRetriever(
        "vector",
        retriever_dict={"vector": vector_retriever},
        node_dict=node_mappings,
        verbose=True,
    )
    query_engine = RetrieverQueryEngine.from_args(recursive_retriever)
    return query_engine, base_nodes

Create Sub Question Query Engine#

import nest_asyncio

nest_asyncio.apply()
from llama_index.tools import QueryEngineTool, ToolMetadata
from llama_index.query_engine import SubQuestionQueryEngine
from llama_index import ServiceContext
from llama_index.llms import OpenAI

llm = OpenAI(model="gpt-4")

service_context = ServiceContext.from_defaults(llm=llm)
query_engine_2021, nodes_2021 = create_recursive_retriever_over_doc(
    docs_2021, nodes_save_path="2021_nodes.pkl"
)
query_engine_2020, nodes_2020 = create_recursive_retriever_over_doc(
    docs_2020, nodes_save_path="2020_nodes.pkl"
)
100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 89/89 [06:29<00:00,  4.38s/it]
# setup base query engine as tool
query_engine_tools = [
    QueryEngineTool(
        query_engine=query_engine_2021,
        metadata=ToolMetadata(
            name="tesla_2021_10k",
            description=(
                "Provides information about Tesla financials for year 2021"
            ),
        ),
    ),
    QueryEngineTool(
        query_engine=query_engine_2020,
        metadata=ToolMetadata(
            name="tesla_2020_10k",
            description=(
                "Provides information about Tesla financials for year 2020"
            ),
        ),
    ),
]

sub_query_engine = SubQuestionQueryEngine.from_defaults(
    query_engine_tools=query_engine_tools,
    service_context=service_context,
    use_async=True,
)

Try out some Comparisons#

response = sub_query_engine.query(
    "Can you compare and contrast the cash flow in 2021 with 2020?"
)
print(str(response))
In 2021, Tesla's cash flow was $11,497 million, which was significantly higher than in 2020, when it was $5.94 billion. This indicates a substantial increase in cash flow from one year to the next.
response = sub_query_engine.query(
    "Can you compare and contrast the R&D expenditures in 2021 vs. 2020?"
)
print(str(response))
In 2021, Tesla spent $2.593 billion on research and development (R&D), which was significantly higher than the $1.491 billion they spent in 2020. This indicates an increase in R&D expenditure from 2020 to 2021.
response = sub_query_engine.query(
    "Can you compare and contrast the risk factors in 2021 vs. 2020?"
)
print(str(response))
In 2021, Tesla faced risks such as competition for skilled labor, negative publicity, potential impacts from staff reductions and the departure of senior personnel, competition from financially stronger companies, dependence on Elon Musk, potential cyber-attacks or security incidents, competition in the energy generation and storage business, potential issues with components manufactured at their Gigafactories, risks associated with international operations, and the potential for product defects or delays in functionality.

In contrast, the risks in 2020 were largely influenced by the global COVID-19 pandemic, which affected macroeconomic conditions, government regulations, and social behaviors. This led to temporary suspensions of operations at manufacturing facilities, temporary employee furloughs and compensation reductions, and challenges in new vehicle deliveries, used vehicle sales, and energy product deployments. Global trade conditions and consumer trends, such as port congestion and microchip supply shortages, also posed risks to Tesla's business.

While both years presented unique challenges, the risks in 2021 were more related to competition, personnel, and manufacturing issues, whereas in 2020, the risks were largely driven by external factors such as the pandemic and global trade conditions.

Try Comparing against Baseline#

vector_index_2021 = VectorStoreIndex(nodes_2021)
vector_query_engine_2021 = vector_index_2021.as_query_engine(
    similarity_top_k=2
)
vector_index_2020 = VectorStoreIndex(nodes_2020)
vector_query_engine_2020 = vector_index_2020.as_query_engine(
    similarity_top_k=2
)
# setup base query engine as tool
query_engine_tools = [
    QueryEngineTool(
        query_engine=vector_query_engine_2021,
        metadata=ToolMetadata(
            name="tesla_2021_10k",
            description=(
                "Provides information about Tesla financials for year 2021"
            ),
        ),
    ),
    QueryEngineTool(
        query_engine=vector_query_engine_2020,
        metadata=ToolMetadata(
            name="tesla_2020_10k",
            description=(
                "Provides information about Tesla financials for year 2020"
            ),
        ),
    ),
]

base_sub_query_engine = SubQuestionQueryEngine.from_defaults(
    query_engine_tools=query_engine_tools,
    service_context=service_context,
    use_async=True,
)
response = base_sub_query_engine.query(
    "Can you compare and contrast the cash flow in 2021 with 2020?"
)
print(str(response))
Generated 2 sub questions.
[tesla_2021_10k] Q: What was the cash flow of Tesla in 2021?
[tesla_2020_10k] Q: What was the cash flow of Tesla in 2020?
[tesla_2020_10k] A: Tesla had a cash flow of $5.94 billion in 2020.
[tesla_2021_10k] A: The cash flow of Tesla in 2021 cannot be determined based on the given context information.
I'm sorry, but the cash flow of Tesla in 2021 is not specified, so a comparison with the 2020 cash flow of $5.94 billion cannot be made.