Introduction to LlamaIndex

LlamaIndex is an application data framework based on large language models (LLM), designed for context-enhanced applications. This type of LLM system is known as a RAG system, or "Retrieval-Augmented Generation". LlamaIndex provides the necessary abstractions to more easily ingest, construct, and access private or domain-specific data, so that this data can be securely and reliably injected into the LLM for more accurate text generation.

The key to data ingestion in LlamaIndex lies in loading and transforming. After loading documents, you can process them through transformation and output nodes. LlamaIndex can load data stored in cloud data warehouse Lakehouse through the Database Reader.

Example Code

from llama_index.readers.database import DatabaseReader

import streamlit as st

username = st.secrets.lakehouse.username
password = st.secrets.lakehouse.password
account = st.secrets.lakehouse.account
endpoint = st.secrets.lakehouse.endpoint
workspace = st.secrets.lakehouse.workspace
schema = st.secrets.lakehouse.schema
virtualcluster = st.secrets.lakehouse.virtualcluster

CONNECTION_STRING = (
    f"clickzetta://{username}:{password}@"
    f"{account}.{endpoint}/{workspace}?schema={schema}&virtualcluster={virtualcluster}"
)

db = DatabaseReader(uri=CONNECTION_STRING, schema=schema)

Available Methods for SQLDatabase

print(type(db.sql_database.from_uri))
print(type(db.sql_database.get_single_table_info))
print(type(db.sql_database.get_table_columns))
print(type(db.sql_database.get_usable_table_names))
print(type(db.sql_database.insert_into_table))
print(type(db.sql_database.run_sql))

query = f"""
    SELECT
        CONCAT(name, ' is ', age, ' years old.') AS text
    FROM public.users
    WHERE age >= 18
    """

documents = db.load_data(query=query)

index = VectorStoreIndex.from_documents(documents)

Reference Documentation

Llama-Index Database Reader