Implement Semantic Search with LangChain

In this tutorial, we’ll demonstrate how to use Upstash Vector with LangChain to perform a similarity search. We will upload a document about global warming and perform a search query to find the most semantically similar documents using embeddings generated automatically by Upstash.

Installation and Setup

First, we need to create a Vector Index in the Upstash Console. Once we have our index, we will copy the UPSTASH_VECTOR_REST_URL and UPSTASH_VECTOR_REST_TOKEN and paste them to our .env file. To learn more about index creation, you can check out this page. Add the following content to your .env file (replace with your actual URL and token):

UPSTASH_VECTOR_REST_URL=your_upstash_url
UPSTASH_VECTOR_REST_TOKEN=your_upstash_token

We now need to install the following libraries via PyPI:

pip install upstash-vector python-dotenv langchain langchain-community

Code

We will load our environment variables and initialize the index from the environment variables (URL and token).

from dotenv import load_dotenv
from langchain_community.vectorstores.upstash import UpstashVectorStore

load_dotenv()

# Create a vector store instance where embeddings are generated by Upstash
store = UpstashVectorStore(embedding=True)

Next, we will upload the global_warming.txt document to the Upstash Vector index.

from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import CharacterTextSplitter

# Load the document
loader = TextLoader("documents/global_warming.txt")
documents = loader.load()

# Split the document into chunks
text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
docs = text_splitter.split_documents(documents)

We will now insert the documents into the Upstash Vector index.

inserted_vectors = store.add_documents(docs)

Finally, we will perform a semantic search.

result = store.similarity_search("Technology's role in global warming.", k=5)
print(result)

Here’s the output:

[Document(metadata={'source': 'documents/global_warming.txt'}, page_content='Technology and innovation play a crucial role in advancing sustainable food systems and mitigating the impact of global warming. Precision agriculture, for example, uses data analytics, remote sensing, and GPS technology to optimize the use of resources such as water, fertilizer, and pesticides. By applying inputs more efficiently, precision agriculture reduces waste, lowers GHG emissions, and improves crop yields.\n\nIn the realm of food production, alternative proteins—such as plant-based meats, lab-grown meats, and insect-based proteins—offer promising solutions to reduce the environmental impact of livestock farming. These innovations require fewer resources to produce and generate lower GHG emissions compared to traditional animal agriculture. Additionally, vertical farming and hydroponics allow for the cultivation of crops in controlled environments, using less land and water while reducing the need for chemical inputs.\n\nConclusion'),
 Document(metadata={'source': 'documents/global_warming.txt'}, page_content='Global warming presents a formidable challenge to food production systems, but sustainable food practices offer a viable solution to mitigate its effects. By adopting regenerative agriculture, reducing food waste, and shifting toward plant-based diets, humanity can reduce the environmental impact of agriculture while ensuring food security for future generations. Furthermore, technology and innovation will continue to play a pivotal role in advancing sustainable food systems, allowing for more efficient and eco-friendly food production methods. In the face of climate change, the integration of sustainable food practices is not only an environmental imperative but also a pathway toward a healthier and more resilient future for both people and the planet.'),
 Document(metadata={'source': 'documents/global_warming.txt'}, page_content='Global Warming and Sustainable Foods: An Inextricable Connection'),
 Document(metadata={'source': 'documents/global_warming.txt'}, page_content='One of the key components of sustainable food systems is regenerative agriculture, which focuses on restoring soil health, enhancing biodiversity, and sequestering carbon. Regenerative farming practices include crop rotation, cover cropping, agroforestry, and reduced tillage, all of which contribute to building healthy soils that can absorb and store carbon. By improving soil health, regenerative agriculture also increases the resilience of crops to climate change, as healthy soils retain more water and nutrients, making crops more resistant to droughts and pests.'),
 Document(metadata={'source': 'documents/global_warming.txt'}, page_content='Numerous studies have shown that shifting to plant-based diets can significantly reduce GHG emissions and mitigate the effects of global warming. According to a report by the United Nations Food and Agriculture Organization (FAO), reducing the consumption of animal-based foods could reduce global agricultural emissions by up to 50%. Moreover, plant-based diets are associated with reduced land and water use, as growing crops for direct human consumption is more resource-efficient than growing feed for livestock.\n\nIn addition to their environmental benefits, plant-based diets offer numerous health advantages. They are rich in fiber, vitamins, and minerals, and have been linked to lower risks of chronic diseases such as heart disease, diabetes, and certain cancers. As global populations grow and the demand for food increases, promoting plant-based diets offers a sustainable solution to addressing both environmental and nutritional challenges.\n\nThe Role of Technology and Innovation')]

Notes

You can also query with score using similarity_search_with_score method.
Namespaces can be used to separate different types of documents. You can specify a namespace when creating the UpstashVectorStore instance.

store = UpstashVectorStore(embedding=True, namespace="my_namespace")

You can use OpenAI’s embeddings by setting embedding=OpenAIEmbeddings() in the UpstashVectorStore instance.

from langchain_openai import OpenAIEmbeddings
store = UpstashVectorStore(embedding=OpenAIEmbeddings())

To learn more about LangChain and its integration with Upstash Vector, you can visit the LangChain documentation.

​Installation and Setup

​Code

​Notes

Installation and Setup

Code

Notes