from langchain.document_loaders.generic import GenericLoader from langchain.document_loaders.parsers import OpenAIWhisperParser from langchain.document_loaders.blob_loaders.youtube_audio import YoutubeAudioLoader
# 调用 GenericLoader Class 的函数 load对视频的音频文件进行加载 pages = loader.load()
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
[youtube] Extracting URL: https://www.youtube.com/watch?v=_PHdzsQaDgw [youtube] _PHdzsQaDgw: Downloading webpage [youtube] _PHdzsQaDgw: Downloading ios player API JSON [youtube] _PHdzsQaDgw: Downloading android player API JSON [youtube] _PHdzsQaDgw: Downloading m3u8 information
WARNING: [youtube] Failed to download m3u8 information: HTTP Error 429: Too Many Requests
[info] _PHdzsQaDgw: Downloading 1 format(s): 140 [download] docs/youtube-zh//【2023年7月最新】ChatGPT注册教程，国内详细注册流程，支持中文使用，chatgpt 中国怎么用？.m4a has already been downloaded [download] 100% of 7.72MiB [ExtractAudio] Not converting audio docs/youtube-zh//【2023年7月最新】ChatGPT注册教程，国内详细注册流程，支持中文使用，chatgpt 中国怎么用？.m4a; file is already in target format m4a Transcribing part 1!
探索加载的数据
Youtube 音频文件加载得到的变量同上文类似，此处不再一一解释，通过类似代码可以展示加载数据：
1 2 3 4 5 6 7
print("Type of pages: ", type(pages)) print("Length of pages: ", len(pages))
Type of pages: <class 'list'> Length of pages: 1 Type of page: <class 'langchain.schema.document.Document'> Page_content: {"payload":{"allShortcutsEnabled":false,"fileTree":{"docs":{"items":[{"name":"ch02","path":"docs/ch02","contentType":"directory"},{"name":"ch03","path":"docs/ch03","contentType":"directory"},{"name":"ch05","path":"docs/ch05","contentType":"directory"},{"name":"ch06","path":"docs/ch06","contentType":"directory"},{"name":"ch08","path":"docs/ch08","contentType":"directory"},{"name":"ch09","path":"docs/ch09","contentType":"directory"},{"name":"ch10","path":"docs/ch10","contentType":"directory"},{"na Meta Data: {'source': 'https://github.com/datawhalechina/d2l-ai-solutions-manual/blob/master/docs/README.md'}
Type of pages: <class 'list'> Length of pages: 51 Type of page: <class 'langchain.schema.document.Document'> Page_content: # #letstalkaboutstress
Let’s talk about stress. Too much stress.
We know this can be a topic.
So let’s get this conversation going.
[Intro: two things you should know](#letstalkaboutstress%2064040a0733074994976118bbe0acc7fb/Intro%20two%20things%20you%20should%20know%20b5fd0c5393a9498b93396e79fe71e8bf.md)
[What is stress](#letstalkaboutstress%2064040a0733074994976118bbe0acc7fb/What%20is%20stress%20b198b685ed6a474ab14f6fafff7004b6.md)
[When is there too much stress?](#letstalkaboutstress%2 Meta Data: {'source': 'docs/Notion_DB/#letstalkaboutstress 64040a0733074994976118bbe0acc7fb.md'}
英文版
加载 PDF 文档
1 2 3 4 5 6 7
from langchain.document_loaders import PyPDFLoader
# 创建一个 PyPDFLoader Class 实例，输入为待加载的pdf文档路径 loader = PyPDFLoader("docs/cs229_lectures/MachineLearning-Lecture01.pdf")
# 调用 PyPDFLoader Class 的函数 load对pdf文件进行加载 pages = loader.load()
探索加载的数据
1 2 3 4 5 6 7
print("Type of pages: ", type(pages)) print("Length of pages: ", len(pages))
Type of pages: <class 'list'> Length of pages: 22 Type of page: <class 'langchain.schema.document.Document'> Page_content: MachineLearning-Lecture01 Instructor (Andrew Ng): Okay. Good morning. Welcome to CS229, the machine learning class. So what I wanna do today is ju st spend a little time going over the logistics of the class, and then we'll start to talk a bit about machine learning. By way of introduction, my name's Andrew Ng and I'll be instru ctor for this class. And so I personally work in machine learning, and I' ve worked on it for about 15 years now, and I actually think that machine learning i Meta Data: {'source': 'docs/cs229_lectures/MachineLearning-Lecture01.pdf', 'page': 0}
from langchain.document_loaders.generic import GenericLoader from langchain.document_loaders.parsers import OpenAIWhisperParser from langchain.document_loaders.blob_loaders.youtube_audio import YoutubeAudioLoader
Type of pages: <class 'list'> Length of pages: 1 Type of page: <class 'langchain.schema.document.Document'> Page_content: {"payload":{"allShortcutsEnabled":false,"fileTree":{"":{"items":[{"name":"37signals-is-you.md","path":"37signals-is-you.md","contentType":"file"},{"name":"LICENSE.md","path":"LICENSE.md","contentType":"file"},{"name":"README.md","path":"README.md","contentType":"file"},{"name":"benefits-and-perks.md","path":"benefits-and-perks.md","contentType":"file"},{"name":"code-of-conduct.md","path":"code-of-conduct.md","contentType":"file"},{"name":"faq.md","path":"faq.md","contentType":"file"},{"name":"ge Meta Data: {'source': 'https://github.com/basecamp/handbook/blob/master/37signals-is-you.md'}
37signals Is You Everyone working at 37signals represents 37signals. When a customer gets a response from Merissa on support, Merissa is 37signals. When a customer reads a tweet by Eron that our systems are down, Eron is 37signals. In those situations, all the other stuff we do to cultivate our best image is secondary. What’s right in front of someone in a time of need is what they’ll remember. That’s what we mean when we say marketing is everyone’s responsibility, and that it pays to spend the time to recognize that. This means avoiding the bullshit of outage language and bending our policies, not just lending your ears. It means taking the time to get the writing right and consider how you’d feel if you were on the other side of the interaction. The vast majority of our customers come from word of mouth and much of that word comes from people in our audience. This is an audience we’ve been educating and entertaining for 20 years and counting, and your voice is part of us now, whether you like it or not! Tell us and our audience what you have to say! This goes for tools and techniques as much as it goes for prose. 37signals not only tries to out-teach the competition, but also out-share and out-collaborate. We’re prolific open source contributors through Ruby on Rails, Trix, Turbolinks, Stimulus, and many other projects. Extracting the common infrastructure that others could use as well is satisfying, important work, and we should continue to do that. It’s also worth mentioning that joining 37signals can be all-consuming. We’ve seen it happen. You dig 37signals, so you feel pressure to contribute, maybe overwhelmingly so. The people who work here are some of the best and brightest in our industry, so the self-imposed burden to be exceptional is real. But here’s the thing: stop it. Settle in. We’re glad you love this job because we all do too, but at the end of the day it’s a job. Do your best work, collaborate with your team, write, read, learn, and then turn off your computer and play with your dog. We’ll all be better for it.
加载 Notion 文档
1 2 3
from langchain.document_loaders import NotionDirectoryLoader loader = NotionDirectoryLoader("docs/Notion_DB") pages = loader.load()
探索加载的数据
1 2 3 4 5 6 7
print("Type of pages: ", type(pages)) print("Length of pages: ", len(pages))
Type of pages: <class 'list'> Length of pages: 51 Type of page: <class 'langchain.schema.document.Document'> Page_content: # #letstalkaboutstress
Let’s talk about stress. Too much stress.
We know this can be a topic.
So let’s get this conversation going.
[Intro: two things you should know](#letstalkaboutstress%2064040a0733074994976118bbe0acc7fb/Intro%20two%20things%20you%20should%20know%20b5fd0c5393a9498b93396e79fe71e8bf.md)
[What is stress](#letstalkaboutstress%2064040a0733074994976118bbe0acc7fb/What%20is%20stress%20b198b685ed6a474ab14f6fafff7004b6.md)
[When is there too much stress?](#letstalkaboutstress%2 Meta Data: {'source': 'docs/Notion_DB/#letstalkaboutstress 64040a0733074994976118bbe0acc7fb.md'}
可以看到出现了提示”Created a chunk of size 23, which is longer than the specified 20”，意思是“创建了一个长度为23的块，这比指定的20要长。”。这是因为CharacterTextSplitter优先使用我们自定义的分隔符进行分割，所以在长度上会有较小的差距
page_content='Let’s talk about stress. Too much stress. \nWe know this can be a topic. \nSo let’s get this conversation going. \n[Intro: two things you should know](#letstalkaboutstress%2064040a0733074994976118bbe0acc7fb/Intro%20two%20things%20you%20should%20know%20b5fd0c5393a9498b93396e79fe71e8bf.md) \n[What is stress](#letstalkaboutstress%2064040a0733074994976118bbe0acc7fb/What%20is%20stress%20b198b685ed6a474ab14f6fafff7004b6.md) \n[When is there too much stress?](#letstalkaboutstress%2064040a0733074994976118bbe0acc7fb/When%20is%20there%20too%20much%20stress%20dc135b9a86a843cbafd115aa128c5c90.md) \n[What can I do](#letstalkaboutstress%2064040a0733074994976118bbe0acc7fb/What%20can%20I%20do%2009c1b13703ef42d4a889e2059c5b25fe.md) \n[What can Blendle do?](#letstalkaboutstress%2064040a0733074994976118bbe0acc7fb/What%20can%20Blendle%20do%20618ab89df4a647bf96e7b432af82779f.md) \n[Good reads](#letstalkaboutstress%2064040a0733074994976118bbe0acc7fb/Good%20reads%20e817491d84d549f886af972e0668192e.md) \nGo to **#letstalkaboutstress** on slack to chat about this topic' metadata={'Header 1': '#letstalkaboutstress'}
英文版
短句分割
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
#导入文本分割器 from langchain.text_splitter import RecursiveCharacterTextSplitter, CharacterTextSplitter
# 递归分割长段落 some_text = """When writing documents, writers will use document structure to group content. \ This can convey to the reader, which idea's are related. For example, closely related ideas \ are in sentances. Similar ideas are in paragraphs. Paragraphs form a document. \n\n \ Paragraphs are often delimited with a carriage return or two carriage returns. \ Carriage returns are the "backslash n" you see embedded in this string. \ Sentences have a period at the end, but also, have a space.\ and words are separated by space."""
['When writing documents, writers will use document structure to group content. This can convey to the reader, which idea\'s are related. For example, closely related ideas are in sentances. Similar ideas are in paragraphs. Paragraphs form a document. \n\n Paragraphs are often delimited with a carriage return or two carriage returns. Carriage returns are the "backslash n" you see embedded in this string. Sentences have a period at the end, but also,', 'have a space.and words are separated by space.']
递归字符分割器效果：
1 2 3
#分割结果 r_splitter.split_text(some_text)
1 2
["When writing documents, writers will use document structure to group content. This can convey to the reader, which idea's are related. For example, closely related ideas are in sentances. Similar ideas are in paragraphs. Paragraphs form a document.", 'Paragraphs are often delimited with a carriage return or two carriage returns. Carriage returns are the "backslash n" you see embedded in this string. Sentences have a period at the end, but also, have a space.and words are separated by space.']
["When writing documents, writers will use document structure to group content. This can convey to the reader, which idea's are related.", 'For example, closely related ideas are in sentances. Similar ideas are in paragraphs. Paragraphs form a document.', 'Paragraphs are often delimited with a carriage return or two carriage returns.', 'Carriage returns are the "backslash n" you see embedded in this string.', 'Sentences have a period at the end, but also, have a space.and words are separated by space.']
基于 Token 分割
1 2 3 4 5 6 7
# 使用token分割器进行分割， # 将块大小设为1，块重叠大小设为0，相当于将任意字符串分割成了单个Token组成的列 from langchain.text_splitter import TokenTextSplitter text_splitter = TokenTextSplitter(chunk_size=1, chunk_overlap=0) text1 = "foo bar bazzyfoo" text_splitter.split_text(text1)
from langchain.document_loaders import NotionDirectoryLoader#Notion加载器 from langchain.text_splitter import MarkdownHeaderTextSplitter#markdown分割器
markdown_document = """# Title\n\n \ ## Chapter 1\n\n \ Hi this is Jim\n\n Hi this is Joe\n\n \ ### Section \n\n \ Hi this is Lance \n\n ## Chapter 2\n\n \ Hi this is Molly"""
print("The first chunk") print(md_header_splits[0]) # Document(page_content='Hi this is Jim \nHi this is Joe', metadata={'Header 1': 'Title', 'Header 2': 'Chapter 1'}) print("The second chunk") print(md_header_splits[1]) # Document(page_content='Hi this is Lance', metadata={'Header 1': 'Title', 'Header 2': 'Chapter 1', 'Header 3': 'Section'})
1 2 3 4
The first chunk page_content='Hi this is Jim \nHi this is Joe \n### Section \nHi this is Lance' metadata={'Header 1': 'Title', 'Header 2': 'Chapter 1'} The second chunk page_content='Hi this is Molly' metadata={'Header 1': 'Title', 'Header 2': 'Chapter 2'}
分割数据库中的 Markdown 文档
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
#加载数据库的内容 loader = NotionDirectoryLoader("docs/Notion_DB") docs = loader.load() txt = ' '.join([d.page_content for d in docs])#拼接文档
page_content='Let’s talk about stress. Too much stress. \nWe know this can be a topic. \nSo let’s get this conversation going. \n[Intro: two things you should know](#letstalkaboutstress%2064040a0733074994976118bbe0acc7fb/Intro%20two%20things%20you%20should%20know%20b5fd0c5393a9498b93396e79fe71e8bf.md) \n[What is stress](#letstalkaboutstress%2064040a0733074994976118bbe0acc7fb/What%20is%20stress%20b198b685ed6a474ab14f6fafff7004b6.md) \n[When is there too much stress?](#letstalkaboutstress%2064040a0733074994976118bbe0acc7fb/When%20is%20there%20too%20much%20stress%20dc135b9a86a843cbafd115aa128c5c90.md) \n[What can I do](#letstalkaboutstress%2064040a0733074994976118bbe0acc7fb/What%20can%20I%20do%2009c1b13703ef42d4a889e2059c5b25fe.md) \n[What can Blendle do?](#letstalkaboutstress%2064040a0733074994976118bbe0acc7fb/What%20can%20Blendle%20do%20618ab89df4a647bf96e7b432af82779f.md) \n[Good reads](#letstalkaboutstress%2064040a0733074994976118bbe0acc7fb/Good%20reads%20e817491d84d549f886af972e0668192e.md) \nGo to **#letstalkaboutstress** on slack to chat about this topic' metadata={'Header 1': '#letstalkaboutstress'}
from langchain.llms import OpenAI from langchain.retrievers.self_query.base import SelfQueryRetriever from langchain.chains.query_constructor.base import AttributeInfo
metadata_field_info_chinese = [ AttributeInfo( name="source", description="The lecture the chunk is from, should be one of `docs/matplotlib/第一回：Matplotlib初相识.pdf`, `docs/matplotlib/第二回：艺术画笔见乾坤.pdf`, or `docs/matplotlib/第三回：布局格式定方圆.pdf`", type="string", ), AttributeInfo( name="page", description="The page from the lecture", type="integer", ), ]
/root/autodl-tmp/env/gpt/lib/python3.10/site-packages/langchain/chains/llm.py:275: UserWarning: The predict_and_parse method is deprecated, instead pass an output parser directly to LLMChain. warnings.warn(
from langchain.retrievers import SVMRetriever from langchain.retrievers import TFIDFRetriever from langchain.document_loaders import PyPDFLoader from langchain.text_splitter import RecursiveCharacterTextSplitter
# 加载PDF loader_chinese = PyPDFLoader("docs/matplotlib/第一回：Matplotlib初相识.pdf") pages_chinese = loader_chinese.load() all_page_text_chinese = [p.page_content for p in pages_chinese] joined_page_text_chinese = " ".join(all_page_text_chinese)
texts = [ """The Amanita phalloides has a large and imposing epigeous (aboveground) fruiting body (basidiocarp).""", """A mushroom with a large fruiting body is the Amanita phalloides. Some varieties are all-white.""", """A. phalloides, a.k.a Death Cap, is one of the most poisonous of all known mushrooms.""", ]
相似性检索： [Document(page_content='A mushroom with a large fruiting body is the Amanita phalloides. Some varieties are all-white.', metadata={}), Document(page_content='The Amanita phalloides has a large and imposing epigeous (aboveground) fruiting body (basidiocarp).', metadata={})] MMR 检索： [Document(page_content='一种具有大型子实体的蘑菇是毒鹅膏菌（Amanita phalloides）。某些品种全白。', metadata={}), Document(page_content='A. phalloides，又名死亡帽，是已知所有蘑菇中最有毒的一种。', metadata={})]
最大边际相关性
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
question = "what did they say about matlab?" docs_ss = vectordb.similarity_search(question,k=3)
from langchain.llms import OpenAI from langchain.retrievers.self_query.base import SelfQueryRetriever from langchain.chains.query_constructor.base import AttributeInfo
llm = OpenAI(temperature=0)
metadata_field_info = [ AttributeInfo( name="source", description="The lecture the chunk is from, should be one of `docs/cs229_lectures/MachineLearning-Lecture01.pdf`, `docs/cs229_lectures/MachineLearning-Lecture02.pdf`, or `docs/cs229_lectures/MachineLearning-Lecture03.pdf`", type="string", ), AttributeInfo( name="page", description="The page from the lecture", type="integer", ), ]
question = "what did they say about regression in the third lecture?"
docs = retriever.get_relevant_documents(question)
for d in docs: print(d.metadata)
1 2 3 4 5 6 7 8 9
/root/autodl-tmp/env/gpt/lib/python3.10/site-packages/langchain/chains/llm.py:275: UserWarning: The predict_and_parse method is deprecated, instead pass an output parser directly to LLMChain. warnings.warn(
question = "what did they say about matlab?" compressed_docs = compression_retriever.get_relevant_documents(question) pretty_print_docs(compressed_docs)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Document 1:
"MATLAB is I guess part of the programming language that makes it very easy to write codes using matrices, to write code for numerical routines, to move data around, to plot data. And it's sort of an extremely easy to learn tool to use for implementing a lot of learning algorithms." ---------------------------------------------------------------------------------------------------- Document 2:
"MATLAB is I guess part of the programming language that makes it very easy to write codes using matrices, to write code for numerical routines, to move data around, to plot data. And it's sort of an extremely easy to learn tool to use for implementing a lot of learning algorithms." ---------------------------------------------------------------------------------------------------- Document 3:
"And the student said, "Oh, it was the MATLAB." So for those of you that don't know MATLAB yet, I hope you do learn it. It's not hard, and we'll actually have a short MATLAB tutorial in one of the discussion sections for those of you that don't know it." ---------------------------------------------------------------------------------------------------- Document 4:
"And the student said, "Oh, it was the MATLAB." So for those of you that don't know MATLAB yet, I hope you do learn it. It's not hard, and we'll actually have a short MATLAB tutorial in one of the discussion sections for those of you that don't know it."
question = "what did they say about matlab?" compressed_docs = compression_retriever.get_relevant_documents(question) pretty_print_docs(compressed_docs)
1 2 3 4 5 6 7
Document 1:
"MATLAB is I guess part of the programming language that makes it very easy to write codes using matrices, to write code for numerical routines, to move data around, to plot data. And it's sort of an extremely easy to learn tool to use for implementing a lot of learning algorithms." ---------------------------------------------------------------------------------------------------- Document 2:
"And the student said, "Oh, it was the MATLAB." So for those of you that don't know MATLAB yet, I hope you do learn it. It's not hard, and we'll actually have a short MATLAB tutorial in one of the discussion sections for those of you that don't know it."
from langchain.retrievers import SVMRetriever from langchain.retrievers import TFIDFRetriever from langchain.document_loaders import PyPDFLoader from langchain.text_splitter import RecursiveCharacterTextSplitter
# 加载PDF loader = PyPDFLoader("docs/cs229_lectures/MachineLearning-Lecture01.pdf") pages = loader.load() all_page_text = [p.page_content for p in pages] joined_page_text = " ".join(all_page_text)
question = "What are major topics for this class?"# 这门课的主要主题是什么？ print("SVM:") docs_svm = svm_retriever.get_relevant_documents(question) print(docs_svm[0])
question = "what did they say about matlab?" print("TF-IDF:") docs_tfidf = tfidf_retriever.get_relevant_documents(question) print(docs_tfidf[0])
1 2 3 4
SVM: page_content="let me just check what questions you have righ t now. So if there are no questions, I'll just \nclose with two reminders, which are after class today or as you start to talk with other \npeople in this class, I just encourage you again to start to form project partners, to try to \nfind project partners to do your project with. And also, this is a good time to start forming \nstudy groups, so either talk to your friends or post in the newsgroup, but we just \nencourage you to try to star t to do both of those today, okay? Form study groups, and try \nto find two other project partners. \nSo thank you. I'm looking forward to teaching this class, and I'll see you in a couple of \ndays. [End of Audio] \nDuration: 69 minutes" metadata={} TF-IDF: page_content="Saxena and Min Sun here did, wh ich is given an image like this, right? This is actually a \npicture taken of the Stanford campus. You can apply that sort of cl ustering algorithm and \ngroup the picture into regions. Let me actually blow that up so that you can see it more \nclearly. Okay. So in the middle, you see the lines sort of groupi ng the image together, \ngrouping the image into [inaudible] regions. \nAnd what Ashutosh and Min did was they then applied the learning algorithm to say can \nwe take this clustering and us e it to build a 3D model of the world? And so using the \nclustering, they then had a lear ning algorithm try to learn what the 3D structure of the \nworld looks like so that they could come up with a 3D model that you can sort of fly \nthrough, okay? Although many people used to th ink it's not possible to take a single \nimage and build a 3D model, but using a lear ning algorithm and that sort of clustering \nalgorithm is the first step. They were able to. \nI'll just show you one more example. I like this because it's a picture of Stanford with our \nbeautiful Stanford campus. So again, taking th e same sort of clustering algorithms, taking \nthe same sort of unsupervised learning algor ithm, you can group the pixels into different \nregions. And using that as a pre-processing step, they eventually built this sort of 3D model of Stanford campus in a single picture. You can sort of walk into the ceiling, look" metadata={}
#import os #os.environ["LANGCHAIN_TRACING_V2"] = "true" #os.environ["LANGCHAIN_ENDPOINT"] = "https://api.langchain.plus" #os.environ["LANGCHAIN_API_KEY"] = "..." # replace dots with your api key
The course is about learning Matplotlib, a Python 2D plotting library used for creating publication-quality figures in various formats and platforms. Matplotlib can be used in Python scripts, IPython Shell, Jupyter notebook, web application servers, and various graphical user interface toolkits. The course will cover basic concepts of Matplotlib and gradually transition to advanced techniques. The course will also cover the components of a Matplotlib figure, including the Figure, Axes, and various sub-elements. The course will provide a general plotting template that can be used to create various types of plots. The template follows the Object-Oriented (OO) mode of Matplotlib, but the course will also cover the pyplot mode. Therefore, the course will involve learning Python and Matplotlib for data visualization using both OO and pyplot modes.
In terms of the advantages and disadvantages of the two modes, the OO mode is more flexible and powerful, allowing for more customization and control over the plot elements. However, it requires more code and can be more complex for beginners. On the other hand, the pyplot mode is simpler and easier to use, but it may not provide as much flexibility and control as the OO mode. The pyplot mode is more suitable for quick and simple plots, while the OO mode is more suitable for complex and customized plots.
Here is an example of a simple pyplot mode plotting template:
import matplotlib.pyplot as plt
import numpy as np
# Step 1: Prepare data
x = np.linspace(0, 2, 100)
y = x**2
# Step 2: Plot data
plt.plot(x, y)
# Step 3: Customize plot
plt.xlabel('x label')
plt.ylabel('y label')
plt.title('Simple Plot')
# Step 4: Show plot
plt.show()
1 2
This template can be used to create a simple plot with x and y labels and a title. Additional customization can be added using various pyplot functions.
# Build prompt template = """Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer. Use three sentences maximum. Keep the answer as concise as possible. Always say "thanks for asking!" at the end of the answer. {context} Question: {question} Helpful Answer:""" QA_CHAIN_PROMPT = PromptTemplate.from_template(template)
question = "Is probability a class topic?" result = qa_chain({"query": question}) print("ANswering:") print(result["result"]) print("Source document: ") print(result["source_documents"][0])
1 2 3 4
ANswering: Yes, probability is assumed to be a prerequisite for this class. The instructor assumes familiarity with basic probability and statistics, and will go over some of the prerequisites in the discussion sections as a refresher course. Thanks for asking! Source document: page_content="of this class will not be very program ming intensive, although we will do some \nprogramming, mostly in either MATLAB or Octa ve. I'll say a bit more about that later. \nI also assume familiarity with basic proba bility and statistics. So most undergraduate \nstatistics class, like Stat 116 taught here at Stanford, will be more than enough. I'm gonna \nassume all of you know what ra ndom variables are, that all of you know what expectation \nis, what a variance or a random variable is. And in case of some of you, it's been a while \nsince you've seen some of this material. At some of the discussion sections, we'll actually \ngo over some of the prerequisites, sort of as a refresher course under prerequisite class. \nI'll say a bit more about that later as well. \nLastly, I also assume familiarity with basi c linear algebra. And again, most undergraduate \nlinear algebra courses are more than enough. So if you've taken courses like Math 51, \n103, Math 113 or CS205 at Stanford, that would be more than enough. Basically, I'm \ngonna assume that all of you know what matrix es and vectors are, that you know how to \nmultiply matrices and vectors and multiply matrix and matrices, that you know what a matrix inverse is. If you know what an eigenvect or of a matrix is, that'd be even better. \nBut if you don't quite know or if you're not qu ite sure, that's fine, too. We'll go over it in \nthe review sections." metadata={'source': 'docs/cs229_lectures/MachineLearning-Lecture01.pdf', 'page': 4}
question = "Is probability a class topic?" result = qa_chain_mr({"query": question})
print(result["result"])
1
There is no clear answer to this question based on the given portion of the document. The document mentions statistics and algebra as topics that may be covered, but it does not explicitly state whether probability is a class topic.
基于 Refine 的检索式问答链
1 2 3 4 5 6 7 8
qa_chain_mr = RetrievalQA.from_chain_type( llm, retriever=vectordb.as_retriever(), chain_type="refine" ) question = "Is probability a class topic?" result = qa_chain_mr({"query": question}) print(result["result"])
1
The topic of probability is assumed to be a prerequisite and not a main topic of the class. The instructor assumes that students are familiar with basic probability and statistics, including random variables, expectation, variance, and basic linear algebra. The class will cover learning algorithms, including a discussion on overfitting and the probabilistic interpretation of linear regression. The instructor will use this probabilistic interpretation to derive the first classification algorithm, which will be discussed in the class. The classification problem involves predicting a discrete value, such as in medical diagnosis or housing sales. The class will not be very programming-intensive, but some programming will be done in MATLAB or Octave. The instructor will provide a refresher course on the prerequisites in some of the discussion sections. Additionally, the discussion sections will be used to cover extensions for the material taught in the main lectures, as machine learning is a vast field with many topics to explore.
question = "Is probability a class topic?" result = qa_chain({"query": question}) print("Q: ", question) print("A: ", result["result"])
question = "why are those prerequesites needed?" result = qa_chain({"query": question}) print("Q: ", question) print("A: ", result["result"])
1 2 3 4
Q: Is probability a class topic? A: Yes, probability is a topic that will be assumed to be familiar to students in this class. The instructor assumes that students have a basic understanding of probability and statistics, and will go over some of the prerequisites as a refresher course in the discussion sections. Q: why are those prerequesites needed? A: The prerequisites are needed because in this class, the instructor assumes that all students have a basic knowledge of computer science and knowledge of basic computer skills and principles. This includes knowledge of big-O notation and linear algebra, which are important concepts in machine learning. Without this basic knowledge, it may be difficult for students to understand the material covered in the class.
from langchain.embeddings.openai import OpenAIEmbeddings from langchain.text_splitter import CharacterTextSplitter, RecursiveCharacterTextSplitter from langchain.vectorstores import DocArrayInMemorySearch from langchain.document_loaders import TextLoader from langchain.chains import RetrievalQA, ConversationalRetrievalChain from langchain.memory import ConversationBufferMemory from langchain.chat_models import ChatOpenAI from langchain.document_loaders import TextLoader from langchain.document_loaders import PyPDFLoader
# 初始化一个 prompt 模板，创建一个检索 QA 链，然后传入一个问题并得到一个结果。 # 构建 prompt from langchain.prompts import PromptTemplate template = """Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer. Use three sentences maximum. Keep the answer as concise as possible. Always say "thanks for asking!" at the end of the answer. {context} Question: {question} Helpful Answer:""" QA_CHAIN_PROMPT = PromptTemplate(input_variables=["context", "question"],template=template,)
# 运行 chain from langchain.chains import RetrievalQA question = "Is probability a class topic?" qa_chain = RetrievalQA.from_chain_type(llm, retriever=vectordb.as_retriever(), return_source_documents=True, chain_type_kwargs={"prompt": QA_CHAIN_PROMPT})
result = qa_chain({"query": question}) print(result["result"])
1
Yes, probability is assumed to be a prerequisite for this class. The instructor assumes familiarity with basic probability and statistics, and will go over some of the prerequisites in the discussion sections as a refresher course. Thanks for asking!
question = "Is probability a class topic?" result = qa({"question": question}) print("Q: ", question) print("A: ", result['answer'])
question = "why are those prerequesites needed?" result = qa({"question": question}) print("Q: ", question) print("A: ", result['answer'])
1 2 3 4
Q: Is probability a class topic? A: Yes, probability is a topic that will be assumed to be familiar to students in this class. The instructor assumes that students have familiarity with basic probability and statistics, and that most undergraduate statistics classes will be more than enough. Q: why are those prerequesites needed? A: The reason for requiring familiarity with basic probability and statistics as prerequisites for this class is that the class assumes that students already know what random variables are, what expectation is, what a variance or a random variable is. The class will not spend much time reviewing these concepts, so students are expected to have a basic understanding of them before taking the class.
from langchain.embeddings.openai import OpenAIEmbeddings from langchain.text_splitter import CharacterTextSplitter, RecursiveCharacterTextSplitter from langchain.vectorstores import DocArrayInMemorySearch from langchain.document_loaders import TextLoader from langchain.chains import RetrievalQA, ConversationalRetrievalChain from langchain.memory import ConversationBufferMemory from langchain.chat_models import ChatOpenAI from langchain.document_loaders import TextLoader from langchain.document_loaders import PyPDFLoader
