题意:Weaviate OpenAi 导入数据速率限制错误
问题背景:
I have a book list in json file and i need to import them to weaviate db with opeai - ada module. I will ask questions to OpenAI through a web page about these, and I want it to find relevant answers from the description section of the books I have previously imported, and then return them to me in a meaningful and logical way.
“我有一个包含书籍列表的 JSON 文件,我需要将它们导入到 Weaviate 数据库中,并使用 OpenAI 的 Ada 模块。之后,我会通过网页向 OpenAI 提问,想让它从我之前导入的书籍描述部分中找到相关答案,并以有意义且合乎逻辑的方式返回给我。”
schema = {"classes": [{"class": "PersonalLibrary","description": "Information from Library Books","vectorizer": "text2vec-openai","moduleConfig": {"text2vec-openai": {"model": "ada", "type": "text"}},"properties": [{"dataType": ["text"],"description": "The information","moduleConfig": {"text2vec-openai": {"skip": "false", "vectorizePropertyName": False}},"name": "description",},{"dataType": ["text"],"description": "The title","moduleConfig": {"text2vec-openai": {"skip": "false", "vectorizePropertyName": False}},"name": "title",},{"dataType": ["text"],"description": "The author","moduleConfig": {"text2vec-openai": {"skip": "false", "vectorizePropertyName": False}},"name": "author",},{"dataType": ["text"],"description": "The url","moduleConfig": {"text2vec-openai": {"skip": "false", "vectorizePropertyName": False}},"name": "url",},],}]}
I have 5000+ book in my json and need to import them to weaveiate
“我的 JSON 文件中有5000多本书,需要将它们导入到 Weaviate 中。”
df = pandas.read_json("dummy4.json")
with client.batch as batch:for index, row in df.iterrows():properties = {'title': row['title'],'description': row['description'],'author': row['author'],'url': row['url'],'path': "https://books.mypage.com/explore/"+ str(row['id'])}batch.add_data_object(properties,"PersonalLibrary",)
Then i always take that error, {'error': [{'message': 'update vector: connection to: OpenAI API failed with status: 429 error: Rate limit reached for text-embedding-ada-002 in organization org-WWwo45yj2shuV173thd8l5S5 on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method.'}]} Just 3 books entered weaviate DB.
“然后我总是收到这样的错误:{'error': [{'message': '更新向量时:连接到 OpenAI API 失败,状态码为 429,错误信息为:对于组织 org-WWwo45yj2shuV173thd8l5S5,text-embedding-ada-002 的请求速率限制已达到每分钟 3 次。请在 20 秒后重试。如果问题仍然存在,请通过我们的帮助中心 help.openai.com 联系我们。请在您的账户中添加支付方式以增加速率限制。访问 https://platform.openai.com/account/billing 添加支付方式。'}]} 只有 3 本书成功导入到 Weaviate 数据库中。”
Can you give me advice, how can i import that datas for search with openai. I need to search some text in description then AI should give a meaningful answer
翻译为中文是:
“你能给我一些建议吗?我该如何导入这些数据以便通过 OpenAI 进行搜索。我需要在描述中搜索一些文本,然后 AI 应该给出一个有意义的回答。”
问题解决:
Duda from Weaviate here :)
You will need to throttle your data ingestion, as described here: text2vec-openai | Weaviate
“你需要按照这里描述的方式限制数据导入的速度:text2vec-openai | Weaviate”
Let me know if that helps.
“如果这对你有帮助,请告诉我。”
And thanks for using Weaviate ;)
“感谢你使用 Weaviate ;)”