doc (#1131)

doc

doc

doc

doc

http error response

remove upload file count token

Add more timeout

.github/workflows/helm-release.yaml

@@ -4,8 +4,7 @@ on:
   push:
     tags:
       - 'v*.*.*'
-    branches:
-      - master
+  workflow_dispatch:
 
 jobs:
   helm:
@@ -30,5 +29,6 @@ jobs:
             unset APP_VERSION
             unset HELM_VERSION
           fi
+          helm dependency update files/helm/fastgpt
           helm package files/helm/fastgpt --version ${HELM_VERSION}-helm --app-version ${APP_VERSION} -d bin
           helm push bin/fastgpt-${HELM_VERSION}-helm.tgz oci://${HELM_REPO}

.github/workflows/preview-image.yml

@@ -62,5 +62,6 @@ jobs:
         uses: actions/checkout@v3
       - name: Helm Check
         run: |
+          helm dependency update files/helm/fastgpt
           helm lint files/helm/fastgpt
           helm package files/helm/fastgpt

README.md

@@ -60,7 +60,7 @@ fastgpt.run 域名会弃用。
    - [x] 混合检索 & 重排
    - [x] Tool 模块
    - [ ] 嵌入 [Laf](https://github.com/labring/laf)，实现在线编写 HTTP 模块
-   - [ ] 插件封装功能
+   - [ ] 插件封装功能，支持低代码渲染
 
 `2` 知识库能力
    - [x] 多库复用，混用
@@ -68,9 +68,8 @@ fastgpt.run 域名会弃用。
    - [x] 支持知识库单独设置向量模型
    - [x] 源文件存储
    - [x] 支持手动输入，直接分段，QA 拆分导入
-   - [x] 支持 pdf，docx，txt，html，md，csv
+   - [x] 支持。txt， 。md， 。html， 。pdf， 。docx，pptx， 。csv， 。xlsx (有需要更多可 PR file loader)
    - [x] 支持 url 读取、CSV 批量导入
-   - [ ] 支持 PPT、Excel 导入
    - [ ] 支持文件阅读器
    - [ ] 更多的数据预处理方案
   
@@ -114,7 +113,7 @@ fastgpt.run 域名会弃用。
 * [多模型配置](https://doc.fastgpt.in/docs/development/one-api/)
 * [版本更新/升级介绍](https://doc.fastgpt.in/docs/development/upgrading)
 * [OpenAPI API 文档](https://doc.fastgpt.in/docs/development/openapi/)
-* [知识库结构详解](https://doc.fastgpt.in/docs/course/datasetengine/)
+* [知识库结构详解](https://doc.fastgpt.in/docs/course/dataset_engine/)
 
 <a href="#readme">
     <img src="https://img.shields.io/badge/-返回顶部-7d09f1.svg" alt="#" align="right">

docSite/content/docs/course/ai_settings.md

@@ -64,7 +64,7 @@ Tips: 可以通过点击上下文按键查看完整的上下文组成，便于
 
 FastGPT 知识库采用 QA 对(不一定都是问答格式，仅代表两个变量)的格式存储，在转义成字符串时候会根据**引用模板**来进行格式化。知识库包含多个可用变量： q, a, sourceId（数据的ID）, index(第n个数据), source(数据的集合名、文件名)，score(距离得分，0-1) 可以通过 {{q}} {{a}} {{sourceId}} {{index}} {{source}} {{score}} 按需引入。下面一个模板例子：
 
-可以通过 [知识库结构讲解](/docs/course/datasetEngine/) 了解详细的知识库的结构。
+可以通过 [知识库结构讲解](/docs/course/dataset_engine/) 了解详细的知识库的结构。
 
 #### 引用模板
 

docSite/content/docs/course/datasetEngine.md

@@ -1,93 +0,0 @@
----
-title: "知识库结构讲解"
-description: "本节会详细介绍 FastGPT 知识库结构设计，理解其 QA 的存储格式和多向量映射，以便更好的构建知识库。这篇介绍主要以使用为主，详细原理不多介绍。"
-icon: "dataset"
-draft: false
-toc: true
-weight: 102
----
-
-## 理解向量
-
-FastGPT 采用了 RAG 中的 Embedding 方案构建知识库，要使用好 FastGPT 需要简单的理解`Embedding`向量是如何工作的及其特点。
-
-人类的文字、图片、视频等媒介是无法直接被计算机理解的，要想让计算机理解两段文字是否有相似性、相关性，通常需要将它们转成计算机可以理解的语言，向量是其中的一种方式。
-
-向量可以简单理解为一个数字数组，两个向量之间可以通过数学公式得出一个`距离`，距离越小代表两个向量的相似度越大。从而映射到文字、图片、视频等媒介上，可以用来判断两个媒介之间的相似度。向量搜索便是利用了这个原理。
-
-而由于文字是有多种类型，并且拥有成千上万种组合方式，因此在转成向量进行相似度匹配时，很难保障其精确性。在向量方案构建的知识库中，通常使用`topk`召回的方式，也就是查找前`k`个最相似的内容，丢给大模型去做更进一步的`语义判断`、`逻辑推理`和`归纳总结`，从而实现知识库问答。因此，在知识库问答中，向量搜索的环节是最为重要的。
-
-影响向量搜索精度的因素非常多，主要包括：向量模型的质量、数据的质量（长度，完整性，多样性）、检索器的精度（速度与精度之间的取舍）。与数据质量对应的就是检索词的质量。
-
-检索器的精度比较容易解决，向量模型的训练略复杂，因此数据和检索词质量优化成了一个重要的环节。
-
-## FastGPT 中向量的结构设计
-
-FastGPT 采用了 `PostgresSQL` 的 `PG Vector` 插件作为向量检索器，索引为`HNSW`。且`PostgresSQL`仅用于向量检索，`MongoDB`用于其他数据的存取。
-
-在`MongoDB`的`dataset.datas`表中，会存储向量原数据的信息，同时有一个`indexes`字段，会记录其对应的向量ID，这是一个数组，也就是说，一组向量可以对应多组数据。
-
-在`PostgresSQL`的表中，设置一个 `index` 字段用于存储向量。在检索时，会先召回向量，再根据向量的ID，去`MongoDB`中寻找原数据内容，如果对应了同一组原数据，则进行合并，向量得分取最高得分。
-
-![](/imgs/datasetSetting1.png)
-
-### 多向量的目的和使用方式
-
-在一组向量中，内容的长度和语义的丰富度通常是矛盾的，无法兼得。因此，FastGPT 采用了多向量映射的方式，将一组数据映射到多组向量中，从而保障数据的完整性和语义的丰富度。
-
-你可以为一组较长的文本，添加多组向量，从而在检索时，只要其中一组向量被检索到，该数据也将被召回。
-
-### 提高向量搜索精度的方法
-
-1. 更好分词分段：当一段话的结构和语义是完整的，并且是单一的，精度也会提高。因此，许多系统都会优化分词器，尽可能的保障每组数据的完整性。
-2. 精简`index`的内容，减少向量内容的长度：当`index`的内容更少，更准确时，检索精度自然会提高。但与此同时，会牺牲一定的检索范围，适合答案较为严格的场景。
-3. 丰富`index`的数量，可以为同一个`chunk`内容增加多组`index`。
-4. 优化检索词：在实际使用过程中，用户的问题通常是模糊的或是缺失的，并不一定是完整清晰的问题。因此优化用户的问题（检索词）很大程度上也可以提高精度。
-5. 微调向量模型：由于市面上直接使用的向量模型都是通用型模型，在特定领域的检索精度并不高，因此微调向量模型可以很大程度上提高专业领域的检索效果。
-
-## FastGPT 构建知识库方案
-
-在 FastGPT 中，整个知识库由库、集合和数据 3 部分组成。集合可以简单理解为一个`文件`。一个`库`中可以包含多个`集合`，一个`集合`中可以包含多组`数据`。最小的搜索单位是`库`，也就是说，知识库搜索时，是对整个`库`进行搜索，而集合仅是为了对数据进行分类管理，与搜索效果无关。（起码目前还是）
-
-| 库 | 集合 | 数据 |
-| --- | --- | --- |
-| ![](/imgs/datasetEngine1.jpg) | ![](/imgs/datasetEngine2.jpg) | ![](/imgs/datasetEngine3.webp) |
-
-### 导入数据方案1 - 直接分段导入
-
-选择文件导入时，可以选择直接分段方案。直接分段会利用`句子分词器`对文本进行一定长度拆分，最终分割中多组的`q`。如果使用了直接分段方案，我们建议在`应用`设置`引用提示词`时，使用`通用模板`即可，无需选择`问答模板`。
-
-| 交互 | 结果 |
-| --- | --- |
-| ![](/imgs/datasetEngine4.webp) | ![](/imgs/datasetEngine5.webp) |
-
-
-### 导入数据方案2 - QA导入
-
-选择文件导入时，可以选择QA拆分方案。仍然需要使用到`句子分词器`对文本进行拆分，但长度比直接分段大很多。在导入后，会先调用`大模型`对分段进行学习，并给出一些`问题`和`答案`，最终问题和答案会一起被存储到`q`中。注意，新版的 FastGPT 为了提高搜索的范围，不再将问题和答案分别存储到 qa 中。
-
-| 交互 | 结果 |
-| --- | --- |
-| ![](/imgs/datasetEngine6.webp) | ![](/imgs/datasetEngine7.webp) |
-
-### 导入数据方案3 - 手动录入
-
-在 FastGPT 中，你可以在任何一个`集合`中点击右上角的`插入`手动录入知识点，或者使用`标注`功能手动录入。被搜索的内容为`q`，补充内容(可选)为`a`。
-
-|  |  |  |
-| --- | --- | --- |
-| ![](/imgs/datasetEngine8.jpg) | ![](/imgs/datasetEngine9.jpg) | ![](/imgs/datasetEngine10.jpg) |
-
-### 导入数据方案4 - CSV录入
-
-有些数据较为独特，可能需要单独的进行预处理分割后再导入 FastGPT，此时可以选择 csv 导入，可批量的将处理好的数据导入。
-
-![](/imgs/datasetEngine11.jpg)
-
-### 导入数据方案5 - API导入
-
-参考[FastGPT OpenAPI使用](/docs/development/openapi)。
-
-## QA的组合与引用提示词构建
-
-参考[引用模板与引用提示词示例](/docs/course/ai_settings/#示例)

docSite/content/docs/course/dataset_engine.md

@@ -0,0 +1,136 @@
+---
+title: '知识库搜索方案和参数'
+description: '本节会详细介绍 FastGPT 知识库结构设计，理解其 QA 的存储格式和多向量映射，以便更好的构建知识库。同时会介绍每个搜索参数的功能。这篇介绍主要以使用为主，详细原理不多介绍。'
+icon: 'language'
+draft: false
+toc: true
+weight: 106
+---
+
+## 理解向量
+
+FastGPT 采用了 RAG 中的 Embedding 方案构建知识库，要使用好 FastGPT 需要简单的理解`Embedding`向量是如何工作的及其特点。
+
+人类的文字、图片、视频等媒介是无法直接被计算机理解的，要想让计算机理解两段文字是否有相似性、相关性，通常需要将它们转成计算机可以理解的语言，向量是其中的一种方式。
+
+向量可以简单理解为一个数字数组，两个向量之间可以通过数学公式得出一个`距离`，距离越小代表两个向量的相似度越大。从而映射到文字、图片、视频等媒介上，可以用来判断两个媒介之间的相似度。向量搜索便是利用了这个原理。
+
+而由于文字是有多种类型，并且拥有成千上万种组合方式，因此在转成向量进行相似度匹配时，很难保障其精确性。在向量方案构建的知识库中，通常使用`topk`召回的方式，也就是查找前`k`个最相似的内容，丢给大模型去做更进一步的`语义判断`、`逻辑推理`和`归纳总结`，从而实现知识库问答。因此，在知识库问答中，向量搜索的环节是最为重要的。
+
+影响向量搜索精度的因素非常多，主要包括：向量模型的质量、数据的质量（长度，完整性，多样性）、检索器的精度（速度与精度之间的取舍）。与数据质量对应的就是检索词的质量。
+
+检索器的精度比较容易解决，向量模型的训练略复杂，因此数据和检索词质量优化成了一个重要的环节。
+
+
+### 提高向量搜索精度的方法
+
+1. 更好分词分段：当一段话的结构和语义是完整的，并且是单一的，精度也会提高。因此，许多系统都会优化分词器，尽可能的保障每组数据的完整性。
+2. 精简`index`的内容，减少向量内容的长度：当`index`的内容更少，更准确时，检索精度自然会提高。但与此同时，会牺牲一定的检索范围，适合答案较为严格的场景。
+3. 丰富`index`的数量，可以为同一个`chunk`内容增加多组`index`。
+4. 优化检索词：在实际使用过程中，用户的问题通常是模糊的或是缺失的，并不一定是完整清晰的问题。因此优化用户的问题（检索词）很大程度上也可以提高精度。
+5. 微调向量模型：由于市面上直接使用的向量模型都是通用型模型，在特定领域的检索精度并不高，因此微调向量模型可以很大程度上提高专业领域的检索效果。
+
+## FastGPT 构建知识库方案
+
+### 数据存储结构
+
+在 FastGPT 中，整个知识库由库、集合和数据 3 部分组成。集合可以简单理解为一个`文件`。一个`库`中可以包含多个`集合`，一个`集合`中可以包含多组`数据`。最小的搜索单位是`库`，也就是说，知识库搜索时，是对整个`库`进行搜索，而集合仅是为了对数据进行分类管理，与搜索效果无关。（起码目前还是）
+
+![](/imgs/dataset_tree.png)
+
+### 向量存储结构
+
+FastGPT 采用了`PostgresSQL`的`PG Vector`插件作为向量检索器，索引为`HNSW`。且`PostgresSQL`仅用于向量检索（该引擎可以替换成其它数据库），`MongoDB`用于其他数据的存取。
+
+在`MongoDB`的`dataset.datas`表中，会存储向量原数据的信息，同时有一个`indexes`字段，会记录其对应的向量ID，这是一个数组，也就是说，一组向量可以对应多组数据。
+
+在`PostgresSQL`的表中，设置一个`vector`字段用于存储向量。在检索时，会先召回向量，再根据向量的ID，去`MongoDB`中寻找原数据内容，如果对应了同一组原数据，则进行合并，向量得分取最高得分。
+
+![](/imgs/datasetSetting1.png)
+
+### 多向量的目的和使用方式
+
+在一组向量中，内容的长度和语义的丰富度通常是矛盾的，无法兼得。因此，FastGPT 采用了多向量映射的方式，将一组数据映射到多组向量中，从而保障数据的完整性和语义的丰富度。
+
+你可以为一组较长的文本，添加多组向量，从而在检索时，只要其中一组向量被检索到，该数据也将被召回。
+
+意味着，你可以通过标注数据块的方式，不断提高数据块的精度。
+
+### 检索方案
+
+1. 通过`问题优化`实现指代消除和问题扩展，从而增加连续对话的检索能力以及语义丰富度。
+2. 通过`Concat query`来增加`Rerank`连续对话的时，排序的准确性。
+3. 通过`RRF`合并方式，综合多个渠道的检索效果。
+4. 通过`Rerank`来二次排序，提高精度。
+
+![](/imgs/dataset_search_process.png)
+
+
+## 搜索参数
+| | | |
+| --- |---| --- |
+|![](/imgs/dataset_search_params1.png)| ![](/imgs/dataset_search_params2.png) | ![](/imgs/dataset_search_params3.png) |
+
+### 搜索模式
+
+#### 语义检索
+
+语义检索是通过向量距离，计算用户问题与知识库内容的距离，从而得出“相似度”，当然这并不是语文上的相似度，而是数学上的。
+
+优点：
+- 相近语义理解
+- 跨多语言理解（例如输入中文问题匹配英文知识点）
+- 多模态理解（文本，图片，音视频等）
+
+缺点：
+- 依赖模型训练效果
+- 精度不稳定
+- 受关键词和句子完整度影响
+
+#### 全文检索
+
+采用传统的全文检索方式。适合查找关键的主谓语等。
+
+#### 混合检索
+
+同时使用向量检索和全文检索，并通过 RRF 公式进行两个搜索结果合并，一般情况下搜索结果会更加丰富准确。
+
+由于混合检索后的查找范围很大，并且无法直接进行相似度过滤，通常需要进行利用重排模型进行一次结果重新排序，并利用重排的得分进行过滤。
+
+#### 结果重排
+
+利用`ReRank`模型对搜索结果进行重排，绝大多数情况下，可以有效提高搜索结果的准确率。不过，重排模型与问题的完整度（主谓语齐全）有一些关系，通常会先走问题优化后再进行搜索-重排。重排后可以得到一个`0-1`的得分，代表着搜索内容与问题的相关度，该分数通常比向量的得分更加精确，可以根据得分进行过滤。
+
+FastGPT 会使用 `RRF` 对重排结果、向量搜索结果、全文检索结果进行合并，得到最终的搜索结果。
+
+### 搜索过滤
+
+#### 引用上限
+
+每次搜索最多引用`n`个`tokens`的内容。
+
+之所以不采用`top k`，是发现在混合知识库（问答库、文档库）时，不同`chunk`的长度差距很大，会导致`top k`的结果不稳定，因此采用了`tokens`的方式进行引用上限的控制。
+
+#### 最低相关度
+
+一个`0-1`的数值，会过滤掉一些低相关度的搜索结果。
+
+该值仅在`语义检索`或使用`结果重排`时生效。
+
+### 问题优化
+
+#### 背景
+
+在 RAG 中，我们需要根据输入的问题去数据库里执行 embedding 搜索，查找相关的内容，从而查找到相似的内容（简称知识库搜索）。
+
+在搜索的过程中，尤其是连续对话的搜索，我们通常会发现后续的问题难以搜索到合适的内容，其中一个原因是知识库搜索只会使用“当前”的问题去执行。看下面的例子：
+
+![](/imgs/coreferenceResolution2.webp)
+
+用户在提问“第二点是什么”的时候，只会去知识库里查找“第二点是什么”，压根查不到内容。实际上需要查询的是“QA结构是什么”。因此我们需要引入一个【问题优化】模块，来对用户当前的问题进行补全，从而使得知识库搜索能够搜索到合适的内容。使用补全后效果如下：
+
+![](/imgs/coreferenceResolution3.webp)
+
+#### 实现方式
+
+在进行`数据检索`前，会先让模型进行`指代消除`与`问题扩展`，一方面可以可以解决指代对象不明确问题，同时可以扩展问题的语义丰富度。你可以通过每次对话后的对话详情，查看补全的结果。

docSite/content/docs/development/configuration.md

@@ -19,6 +19,9 @@ llm模型全部合并
 
 ```json
 {
+  "feConfigs": {
+    "lafEnv": "https://laf.dev" // laf环境
+  },
   "systemEnv": {
     "vectorMaxProcess": 15,
     "qaMaxProcess": 15,
@@ -164,7 +167,7 @@ llm模型全部合并
             "model": "bge-reranker-base", // 随意
             "name": "检索重排-base", // 随意
             "charsPointsPrice": 0,
-            "requestUrl": "{{host}}/api/v1/rerank",
+            "requestUrl": "{{host}}/v1/rerank",
             "requestAuth": "安全凭证，已自动补 Bearer"
         }
     ]

docSite/content/docs/development/custom-models/reranker.md

@@ -44,7 +44,7 @@ weight: 910
 
 ### docker 部署
 
-+ 镜像名: `luanshaotong/reranker:v0.2`  
++ 镜像名: `registry.cn-hangzhou.aliyuncs.com/fastgpt/rerank:v0.2`  
 + 端口号: 6006
 + 大小：约8GB
 
@@ -56,12 +56,12 @@ ACCESS_TOKEN=mytoken
 **运行命令示例**
 - 无需GPU环境，使用CPU运行
 ```sh
-docker run -d --name reranker -p 6006:6006 -e ACCESS_TOKEN=mytoken luanshaotong/reranker:v0.2
+docker run -d --name reranker -p 6006:6006 -e ACCESS_TOKEN=mytoken registry.cn-hangzhou.aliyuncs.com/fastgpt/rerank:v0.2
 ```
 
 - 需要CUDA 11.7环境
 ```sh
-docker run -d --gpus all --name reranker -p 6006:6006 -e ACCESS_TOKEN=mytoken luanshaotong/reranker:v0.2
+docker run -d --gpus all --name reranker -p 6006:6006 -e ACCESS_TOKEN=mytoken registry.cn-hangzhou.aliyuncs.com/fastgpt/rerank:v0.2
 ```
 
 **docker-compose.yml示例**
@@ -69,7 +69,7 @@ docker run -d --gpus all --name reranker -p 6006:6006 -e ACCESS_TOKEN=mytoken lu
 version: "3"
 services:
   reranker:
-    image: luanshaotong/reranker:v0.2
+    image: registry.cn-hangzhou.aliyuncs.com/fastgpt/rerank:v0.2
     container_name: reranker
     # GPU运行环境，如果宿主机未安装，将deploy配置隐藏即可
     deploy:

docSite/content/docs/development/docker.md

@@ -270,3 +270,7 @@ mongo连接失败，查看mongo的运行状态对应日志。
 ### 首次部署，root用户提示未注册
 
 日志会有错误提示。大概率是没有启动 Mongo 副本集模式。
+
+### 无法导出知识库、无法使用语音输入/播报
+
+没配置 SSL 证书，无权使用部分功能。

docSite/content/docs/development/upgrading/47.md

@@ -1,5 +1,5 @@
 ---
-title: 'V4.7(进行中)'
+title: 'V4.7'
 description: 'FastGPT V4.7更新说明'
 icon: 'upgrade'
 draft: false
@@ -26,7 +26,7 @@ curl --location --request POST 'https://{{host}}/api/admin/initv47' \
 
 ## 3. 升级 ReRank 模型
 
-4.7对ReRank模型进行了格式变动，兼容 cohere 的格式，可以直接使用 cohere 提供的 API。如果是本地的 ReRank 模型，需要修改镜像为：`luanshaotong/reranker:v0.2` 。
+4.7对ReRank模型进行了格式变动，兼容 cohere 的格式，可以直接使用 cohere 提供的 API。如果是本地的 ReRank 模型，需要修改镜像为：`registry.cn-hangzhou.aliyuncs.com/fastgpt/rerank:v0.2` 。
 
 cohere的重排模型对中文不是很好，感觉不如 bge 的好用，接入教程如下：
 

docSite/content/docs/development/upgrading/471.md

@@ -0,0 +1,31 @@
+---
+title: 'V4.7.1(进行中)'
+description: 'FastGPT V4.7.1 更新说明'
+icon: 'upgrade'
+draft: false
+toc: true
+weight: 825
+---
+
+## 初始化脚本
+
+从任意终端，发起 1 个 HTTP 请求。其中 {{rootkey}} 替换成环境变量里的 `rootkey`；{{host}} 替换成FastGPT的域名。
+
+```bash
+curl --location --request POST 'https://{{host}}/api/admin/clearInvalidData' \
+--header 'rootkey: {{rootkey}}' \
+--header 'Content-Type: application/json'
+```
+
+该请求会执行脏数据清理（清理无效的文件、清理无效的图片、清理无效的知识库集合、清理无效的向量）
+
+## V4.7.1 更新说明
+
+1. 新增 - Pptx 和 xlsx 文件读取。但所有文件读取都放服务端，会消耗更多的服务器资源，以及无法在上传时预览更多内容。
+2. 新增 - 集成 Laf 云函数，可以读取 Laf 账号中的云函数作为 HTTP 模块。
+3. 新增 - 定时器，清理垃圾数据。（采用小范围清理，会清理最近n个小时的，所以请保证服务持续运行，长时间不允许，可以继续执行 clearInvalidData 的接口进行全量清理。）
+4. 商业版新增 - 后台配置系统通知。
+5. 修改 - csv导入模板，取消 header 校验，自动获取前两列。
+6. 修复 - 工具调用模块连线数据类型校验错误。
+7. 修复 - 自定义索引输入时，解构数据失败。
+8. 修复 - rerank 模型数据格式。

docSite/content/docs/workflow/modules/tool.md

@@ -22,7 +22,7 @@ weight: 356
 
 ## 工具是如何运行的
 
-要了解工具如何允许，首先需要知道它的运行条件。
+要了解工具如何运行的，首先需要知道它的运行条件。
 
 1. 需要工具的介绍（或者叫描述）。这个介绍会告诉LLM，这个工具的作用是什么，LLM会根据上下文语义，决定是否需要调用这个工具。
 2. 工具的参数。有些工具调用时，可能需要一些特殊的参数。参数中有2个关键的值：`参数介绍`和`是否必须`。

files/deploy/fastgpt/docker-compose.yml

@@ -66,8 +66,8 @@ services:
         wait $!
   fastgpt:
     container_name: fastgpt
-    image: registry.cn-hangzhou.aliyuncs.com/fastgpt/fastgpt:v4.6.9 # git
-    # image: registry.cn-hangzhou.aliyuncs.com/fastgpt/fastgpt:v4.6.9 # 阿里云
+    image: registry.cn-hangzhou.aliyuncs.com/fastgpt/fastgpt:v4.7 # git
+    # image: registry.cn-hangzhou.aliyuncs.com/fastgpt/fastgpt:v4.7 # 阿里云
     ports:
       - 3000:3000
     networks:

files/helm/fastgpt/Chart.yaml

@@ -1,6 +1,6 @@
 apiVersion: v2
 name: fastgpt
-description: A Helm chart for Kubernetes
+description: A Helm chart for FastGPT
 
 # A chart can be either an 'application' or a 'library' chart.
 #

files/helm/fastgpt/README.md

@@ -0,0 +1,67 @@
+# fastgpt
+
+![Version: 0.1.0](https://img.shields.io/badge/Version-0.1.0-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: v4.6.6](https://img.shields.io/badge/AppVersion-v4.6.6-informational?style=flat-square)
+
+A Helm chart for FastGPT
+
+## Requirements
+
+| Repository | Name | Version |
+|------------|------|---------|
+| oci://registry-1.docker.io/bitnamicharts | mongodb | 15.0.1 |
+| oci://registry-1.docker.io/bitnamicharts | postgresql | 15.0.0 |
+
+## Values
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| affinity | object | `{}` |  |
+| autoscaling.enabled | bool | `false` |  |
+| autoscaling.maxReplicas | int | `100` |  |
+| autoscaling.minReplicas | int | `1` |  |
+| autoscaling.targetCPUUtilizationPercentage | int | `80` |  |
+| fullnameOverride | string | `""` |  |
+| image.pullPolicy | string | `"IfNotPresent"` |  |
+| image.repository | string | `"ghcr.io/labring/fastgpt"` |  |
+| image.tag | string | `""` |  |
+| imagePullSecrets | list | `[]` |  |
+| ingress.annotations | object | `{}` |  |
+| ingress.className | string | `""` |  |
+| ingress.enabled | bool | `false` |  |
+| ingress.hosts[0].host | string | `"chart-example.local"` |  |
+| ingress.hosts[0].paths[0].path | string | `"/"` |  |
+| ingress.hosts[0].paths[0].pathType | string | `"ImplementationSpecific"` |  |
+| ingress.tls | list | `[]` |  |
+| livenessProbe.httpGet.path | string | `"/"` |  |
+| livenessProbe.httpGet.port | string | `"http"` |  |
+| mongodb.architecture | string | `"replicaset"` |  |
+| mongodb.auth.rootPassword | string | `"123456"` |  |
+| mongodb.auth.rootUser | string | `"root"` |  |
+| mongodb.enabled | bool | `true` | Enable or disable the built-in MangoDB |
+| nameOverride | string | `""` |  |
+| nodeSelector | object | `{}` |  |
+| podAnnotations | object | `{}` |  |
+| podLabels | object | `{}` |  |
+| podSecurityContext | object | `{}` |  |
+| postgresql.enabled | bool | `true` | Enable or disable the built-in PostgreSQL |
+| postgresql.global.postgresql.auth.database | string | `"postgres"` | The default database of PostgreSQL |
+| postgresql.global.postgresql.auth.postgresPassword | string | `"postgres"` | The password of PostgreSQL, default username is `postgres` |
+| postgresql.image.repository | string | `"linuxsuren/pgvector"` | The PostgreSQL image which include the pgvector extension. See also the source code from https://github.com/LinuxSuRen/pgvector-docker |
+| postgresql.image.tag | string | `"v0.0.1"` |  |
+| readinessProbe.httpGet.path | string | `"/"` |  |
+| readinessProbe.httpGet.port | string | `"http"` |  |
+| replicaCount | int | `1` |  |
+| resources | object | `{}` |  |
+| securityContext | object | `{}` |  |
+| service.port | int | `3000` |  |
+| service.type | string | `"ClusterIP"` |  |
+| serviceAccount.annotations | object | `{}` |  |
+| serviceAccount.automount | bool | `true` |  |
+| serviceAccount.create | bool | `true` |  |
+| serviceAccount.name | string | `""` |  |
+| tolerations | list | `[]` |  |
+| volumeMounts | list | `[]` |  |
+| volumes | list | `[]` |  |
+
+----------------------------------------------
+Autogenerated from chart metadata using [helm-docs v1.13.1](https://github.com/norwoodj/helm-docs/releases/v1.13.1)

files/helm/fastgpt/values.yaml

@@ -112,6 +112,7 @@ tolerations: []
 affinity: {}
 
 mongodb:
+  # -- Enable or disable the built-in MangoDB
   enabled: true
   architecture: replicaset
   auth:
@@ -119,13 +120,17 @@ mongodb:
     rootPassword: "123456"
 
 postgresql:
+  # -- Enable or disable the built-in PostgreSQL
   enabled: true
   image:
 #    registry: 172.11.0.6:30002
+    # -- The PostgreSQL image which include the pgvector extension. See also the source code from https://github.com/LinuxSuRen/pgvector-docker
     repository: linuxsuren/pgvector
     tag: v0.0.1
   global:
     postgresql:
       auth:
+        # -- The password of PostgreSQL, default username is `postgres`
         postgresPassword: postgres
+        # -- The default database of PostgreSQL
         database: postgres

packages/global/common/error/code/common.ts

@@ -3,12 +3,17 @@ import { ErrType } from '../errorCode';
 /* dataset: 507000 */
 const startCode = 507000;
 export enum CommonErrEnum {
-  fileNotFound = 'fileNotFound'
+  fileNotFound = 'fileNotFound',
+  unAuthFile = 'unAuthFile'
 }
 const datasetErr = [
   {
     statusText: CommonErrEnum.fileNotFound,
     message: 'error.fileNotFound'
+  },
+  {
+    statusText: CommonErrEnum.unAuthFile,
+    message: 'error.unAuthFile'
   }
 ];
 export default datasetErr.reduce((acc, cur, index) => {

packages/global/common/file/icon.ts

@@ -1,5 +1,7 @@
 export const fileImgs = [
   { suffix: 'pdf', src: 'file/fill/pdf' },
+  { suffix: 'ppt', src: 'file/fill/ppt' },
+  { suffix: 'xlsx', src: 'file/fill/xlsx' },
   { suffix: 'csv', src: 'file/fill/csv' },
   { suffix: '(doc|docs)', src: 'file/fill/doc' },
   { suffix: 'txt', src: 'file/fill/txt' },

packages/global/common/file/tools.ts

@@ -11,5 +11,5 @@ export const formatFileSize = (bytes: number): string => {
 };
 
 export const detectFileEncoding = (buffers: string | Buffer) => {
-  return detect(buffers)?.encoding || 'utf-8';
+  return (detect(buffers)?.encoding || 'utf-8') as BufferEncoding;
 };

packages/global/common/string/textSplitter.ts

@@ -40,9 +40,9 @@ export const splitText2Chunks = (props: {
     { reg: /^(####\s[^\n]+)\n/gm, maxLen: chunkLen * 1.2 },
 
     { reg: /([\n]([`~]))/g, maxLen: chunkLen * 4 }, // code block
-    { reg: /([\n](?!\s*[\*\-|>0-9]))/g, maxLen: chunkLen * 2 }, // (?![\*\-|>`0-9]): markdown special char
+    { reg: /([\n](?!\s*[\*\-|>0-9]))/g, maxLen: chunkLen * 2 }, // 增大块，尽可能保证它是一个完整的段落。 (?![\*\-|>`0-9]): markdown special char
     { reg: /([\n])/g, maxLen: chunkLen * 1.2 },
-
+    // ------ There's no overlap on the top
     { reg: /([。]|([a-zA-Z])\.\s)/g, maxLen: chunkLen * 1.2 },
     { reg: /([！]|!\s)/g, maxLen: chunkLen * 1.2 },
     { reg: /([？]|\?\s)/g, maxLen: chunkLen * 1.4 },
@@ -56,7 +56,7 @@ export const splitText2Chunks = (props: {
   const checkIndependentChunk = (step: number) => step >= customRegLen && step <= 4 + customRegLen;
   const checkForbidOverlap = (step: number) => step <= 6 + customRegLen;
 
-  // if use markdown title split, Separate record title title
+  // if use markdown title split, Separate record title
   const getSplitTexts = ({ text, step }: { text: string; step: number }) => {
     if (step >= stepReges.length) {
       return [
@@ -97,6 +97,7 @@ export const splitText2Chunks = (props: {
       .filter((item) => item.text.trim());
   };
 
+  /* Gets the overlap at the end of a text as the beginning of the next block */
   const getOneTextOverlapText = ({ text, step }: { text: string; step: number }): string => {
     const forbidOverlap = checkForbidOverlap(step);
     const maxOverlapLen = chunkLen * 0.4;

packages/global/common/system/types/index.d.ts

@@ -55,7 +55,9 @@ export type FastGPTFeConfigsType = {
   customApiDomain?: string;
   customSharePageDomain?: string;
 
+  uploadFileMaxAmount?: number;
   uploadFileMaxSize?: number;
+  lafEnv?: string;
 };
 
 export type SystemEnvType = {

packages/global/core/dataset/api.d.ts

@@ -44,14 +44,18 @@ export type TextCreateDatasetCollectionParams = ApiCreateDatasetCollectionParams
 export type LinkCreateDatasetCollectionParams = ApiCreateDatasetCollectionParams & {
   link: string;
 };
+export type FileIdCreateDatasetCollectionParams = ApiCreateDatasetCollectionParams & {
+  fileId: string;
+};
 export type FileCreateDatasetCollectionParams = ApiCreateDatasetCollectionParams & {
-  name: string;
-  rawTextLength: number;
-  hashRawText: string;
-
   fileMetadata?: Record<string, any>;
   collectionMetadata?: Record<string, any>;
 };
+export type CsvTableCreateDatasetCollectionParams = {
+  datasetId: string;
+  parentId?: string;
+  fileId: string;
+};
 
 /* ================= data ===================== */
 export type PgSearchRawType = {

packages/global/core/dataset/constants.ts

@@ -73,6 +73,13 @@ export const DatasetCollectionSyncResultMap = {
 /* ------------ data -------------- */
 
 /* ------------ training -------------- */
+export enum ImportDataSourceEnum {
+  fileLocal = 'fileLocal',
+  fileLink = 'fileLink',
+  fileCustom = 'fileCustom',
+  csvTable = 'csvTable'
+}
+
 export enum TrainingModeEnum {
   chunk = 'chunk',
   auto = 'auto',

packages/global/core/module/node/constant.ts

@@ -61,7 +61,8 @@ export enum FlowNodeTypeEnum {
   pluginOutput = 'pluginOutput',
   queryExtension = 'cfr',
   tools = 'tools',
-  stopTool = 'stopTool'
+  stopTool = 'stopTool',
+  lafModule = 'lafModule'
 
   // abandon
 }

packages/global/core/module/template/constants.ts

@@ -20,6 +20,7 @@ import { AiQueryExtension } from './system/queryExtension';
 
 import type { FlowNodeTemplateType, moduleTemplateListType } from '../../module/type.d';
 import { FlowNodeTemplateTypeEnum } from '../../module/constants';
+import { lafModule } from './system/laf';
 
 /* app flow module templates */
 export const appSystemModuleTemplates: FlowNodeTemplateType[] = [
@@ -35,7 +36,8 @@ export const appSystemModuleTemplates: FlowNodeTemplateType[] = [
   ClassifyQuestionModule,
   ContextExtractModule,
   HttpModule468,
-  AiQueryExtension
+  AiQueryExtension,
+  lafModule
 ];
 /* plugin flow module templates */
 export const pluginSystemModuleTemplates: FlowNodeTemplateType[] = [
@@ -51,7 +53,8 @@ export const pluginSystemModuleTemplates: FlowNodeTemplateType[] = [
   ClassifyQuestionModule,
   ContextExtractModule,
   HttpModule468,
-  AiQueryExtension
+  AiQueryExtension,
+  lafModule
 ];
 
 /* all module */
@@ -73,7 +76,8 @@ export const moduleTemplatesFlat: FlowNodeTemplateType[] = [
   PluginInputModule,
   PluginOutputModule,
   RunPluginModule,
-  AiQueryExtension
+  AiQueryExtension,
+  lafModule
 ];
 
 export const moduleTemplatesList: moduleTemplateListType = [

packages/global/core/module/template/system/laf.ts

@@ -0,0 +1,86 @@
+import {
+  FlowNodeInputTypeEnum,
+  FlowNodeOutputTypeEnum,
+  FlowNodeTypeEnum
+} from '../../node/constant';
+import { FlowNodeTemplateType } from '../../type';
+import {
+  ModuleIOValueTypeEnum,
+  ModuleInputKeyEnum,
+  ModuleOutputKeyEnum,
+  FlowNodeTemplateTypeEnum
+} from '../../constants';
+import {
+  Input_Template_DynamicInput,
+  Input_Template_Switch,
+  Input_Template_AddInputParam
+} from '../input';
+import { Output_Template_Finish, Output_Template_AddOutput } from '../output';
+
+export const lafModule: FlowNodeTemplateType = {
+  id: FlowNodeTypeEnum.lafModule,
+  templateType: FlowNodeTemplateTypeEnum.externalCall,
+  flowType: FlowNodeTypeEnum.lafModule,
+  avatar: '/imgs/module/laf.png',
+  name: 'Laf 函数调用（测试）',
+  intro: '可以调用Laf账号下的云函数。',
+  showStatus: true,
+  isTool: true,
+  inputs: [
+    Input_Template_Switch,
+    {
+      key: ModuleInputKeyEnum.httpReqUrl,
+      type: FlowNodeInputTypeEnum.hidden,
+      valueType: ModuleIOValueTypeEnum.string,
+      label: '',
+      description: 'core.module.input.description.Http Request Url',
+      placeholder: 'https://api.ai.com/getInventory',
+      required: false,
+      showTargetInApp: false,
+      showTargetInPlugin: false
+    },
+    Input_Template_DynamicInput,
+    {
+      ...Input_Template_AddInputParam,
+      editField: {
+        key: true,
+        description: true,
+        dataType: true
+      },
+      defaultEditField: {
+        label: '',
+        key: '',
+        description: '',
+        inputType: FlowNodeInputTypeEnum.target,
+        valueType: ModuleIOValueTypeEnum.string
+      }
+    }
+  ],
+  outputs: [
+    {
+      key: ModuleOutputKeyEnum.httpRawResponse,
+      label: '原始响应',
+      description: 'HTTP请求的原始响应。只能接受字符串或JSON类型响应数据。',
+      valueType: ModuleIOValueTypeEnum.any,
+      type: FlowNodeOutputTypeEnum.source,
+      targets: []
+    },
+    {
+      ...Output_Template_AddOutput,
+      editField: {
+        key: true,
+        description: true,
+        dataType: true,
+        defaultValue: true
+      },
+      defaultEditField: {
+        label: '',
+        key: '',
+        description: '',
+        outputType: FlowNodeOutputTypeEnum.source,
+        valueType: ModuleIOValueTypeEnum.string
+      }
+    },
+    Output_Template_Finish
+  ]
+};

packages/global/core/module/type.d.ts

@@ -9,6 +9,7 @@ import { DispatchNodeResponseKeyEnum } from './runtime/constants';
 import { FlowNodeInputItemType, FlowNodeOutputItemType } from './node/type';
 import { UserModelSchema } from 'support/user/type';
 import {
+  ChatItemType,
   ChatItemValueItemType,
   ToolRunResponseItemType,
   UserChatItemValueItemType

packages/global/core/plugin/httpPlugin/utils.ts

@@ -41,7 +41,7 @@ export const str2OpenApiSchema = async (yamlStr = ''): Promise<OpenApiJsonSchema
               path,
               method,
               name: methodInfo.operationId || path,
-              description: methodInfo.description,
+              description: methodInfo.description || methodInfo.summary,
               params: methodInfo.parameters,
               request: methodInfo?.requestBody
             };

packages/global/package.json

@@ -2,18 +2,18 @@
   "name": "@fastgpt/global",
   "version": "1.0.0",
   "dependencies": {
+    "@apidevtools/swagger-parser": "^10.1.0",
     "axios": "^1.5.1",
     "dayjs": "^1.11.7",
     "encoding": "^0.1.13",
     "js-tiktoken": "^1.0.7",
-    "openapi-types": "^12.1.3",
-    "openai": "4.28.0",
-    "nanoid": "^4.0.1",
     "js-yaml": "^4.1.0",
-    "timezones-list": "^3.0.2",
-    "next": "13.5.2",
     "jschardet": "3.1.1",
-    "@apidevtools/swagger-parser": "^10.1.0"
+    "nanoid": "^4.0.1",
+    "next": "13.5.2",
+    "openai": "4.28.0",
+    "openapi-types": "^12.1.3",
+    "timezones-list": "^3.0.2"
   },
   "devDependencies": {
     "@types/js-yaml": "^4.0.9",

packages/global/support/user/team/controller.d.ts

@@ -1,5 +1,5 @@
 import { TeamMemberRoleEnum } from './constant';
-import { TeamMemberSchema } from './type';
+import { LafAccountType, TeamMemberSchema } from './type';
 
 export type AuthTeamRoleProps = {
   teamId: string;
@@ -10,12 +10,14 @@ export type CreateTeamProps = {
   name: string;
   avatar?: string;
   defaultTeam?: boolean;
+  lafAccount?: LafAccountType;
 };
 export type UpdateTeamProps = {
   teamId: string;
   name?: string;
   avatar?: string;
   teamDomain?: string;
+  lafAccount?: null | LafAccountType;
 };
 
 /* ------------- member ----------- */

packages/global/support/user/team/type.d.ts

@@ -1,5 +1,6 @@
 import type { UserModelSchema } from '../type';
 import type { TeamMemberRoleEnum, TeamMemberStatusEnum } from './constant';
+import { LafAccountType } from './type';
 
 export type TeamSchema = {
   _id: string;
@@ -13,6 +14,7 @@ export type TeamSchema = {
     lastExportDatasetTime: Date;
     lastWebsiteSyncTime: Date;
   };
+  lafAccount: LafAccountType;
 };
 export type tagsType = {
   label: string;
@@ -58,6 +60,7 @@ export type TeamItemType = {
   role: `${TeamMemberRoleEnum}`;
   status: `${TeamMemberStatusEnum}`;
   canWrite: boolean;
+  lafAccount?: LafAccountType;
 };
 
 export type TeamMemberItemType = {
@@ -74,3 +77,8 @@ export type TeamTagItemType = {
   label: string;
   key: string;
 };
+
+export type LafAccountType = {
+  token: string;
+  appid: string;
+};

packages/service/common/buffer/rawText/schema.ts

@@ -0,0 +1,33 @@
+import { connectionMongo, type Model } from '../../mongo';
+const { Schema, model, models } = connectionMongo;
+import { RawTextBufferSchemaType } from './type';
+
+export const collectionName = 'buffer.rawText';
+
+const RawTextBufferSchema = new Schema({
+  sourceId: {
+    type: String,
+    required: true
+  },
+  rawText: {
+    type: String,
+    default: ''
+  },
+  createTime: {
+    type: Date,
+    default: () => new Date()
+  },
+  metadata: Object
+});
+
+try {
+  RawTextBufferSchema.index({ sourceId: 1 });
+  //  20 minutes
+  RawTextBufferSchema.index({ createTime: 1 }, { expireAfterSeconds: 20 * 60 });
+} catch (error) {
+  console.log(error);
+}
+
+export const MongoRwaTextBuffer: Model<RawTextBufferSchemaType> =
+  models[collectionName] || model(collectionName, RawTextBufferSchema);
+MongoRwaTextBuffer.syncIndexes();

packages/service/common/buffer/rawText/type.d.ts

@@ -0,0 +1,8 @@
+export type RawTextBufferSchemaType = {
+  sourceId: string;
+  rawText: string;
+  createTime: Date;
+  metadata?: {
+    filename: string;
+  };
+};

packages/service/common/buffer/tts/schema.ts

@@ -2,7 +2,7 @@ import { connectionMongo, type Model } from '../../../common/mongo';
 const { Schema, model, models } = connectionMongo;
 import { TTSBufferSchemaType } from './type.d';
 
-export const collectionName = 'ttsbuffers';
+export const collectionName = 'buffer.tts';
 
 const TTSBufferSchema = new Schema({
   bufferId: {

packages/service/common/file/gridfs/controller.ts

@@ -4,6 +4,11 @@ import fsp from 'fs/promises';
 import fs from 'fs';
 import { DatasetFileSchema } from '@fastgpt/global/core/dataset/type';
 import { MongoFileSchema } from './schema';
+import { detectFileEncoding } from '@fastgpt/global/common/file/tools';
+import { CommonErrEnum } from '@fastgpt/global/common/error/code/common';
+import { ReadFileByBufferParams } from '../read/type';
+import { MongoRwaTextBuffer } from '../../buffer/rawText/schema';
+import { readFileRawContent } from '../read/utils';
 
 export function getGFSCollection(bucket: `${BucketNameEnum}`) {
   MongoFileSchema;
@@ -111,3 +116,106 @@ export async function getDownloadStream({
 
   return bucket.openDownloadStream(new Types.ObjectId(fileId));
 }
+
+export const readFileEncode = async ({
+  bucketName,
+  fileId
+}: {
+  bucketName: `${BucketNameEnum}`;
+  fileId: string;
+}) => {
+  const encodeStream = await getDownloadStream({ bucketName, fileId });
+  let buffers: Buffer = Buffer.from([]);
+  for await (const chunk of encodeStream) {
+    buffers = Buffer.concat([buffers, chunk]);
+    if (buffers.length > 10) {
+      encodeStream.abort();
+      break;
+    }
+  }
+
+  const encoding = detectFileEncoding(buffers);
+
+  return encoding as BufferEncoding;
+};
+
+export const readFileContentFromMongo = async ({
+  teamId,
+  bucketName,
+  fileId,
+  csvFormat = false
+}: {
+  teamId: string;
+  bucketName: `${BucketNameEnum}`;
+  fileId: string;
+  csvFormat?: boolean;
+}): Promise<{
+  rawText: string;
+  filename: string;
+}> => {
+  // read buffer
+  const fileBuffer = await MongoRwaTextBuffer.findOne({ sourceId: fileId }).lean();
+  if (fileBuffer) {
+    return {
+      rawText: fileBuffer.rawText,
+      filename: fileBuffer.metadata?.filename || ''
+    };
+  }
+
+  const [file, encoding, fileStream] = await Promise.all([
+    getFileById({ bucketName, fileId }),
+    readFileEncode({ bucketName, fileId }),
+    getDownloadStream({ bucketName, fileId })
+  ]);
+
+  if (!file) {
+    return Promise.reject(CommonErrEnum.fileNotFound);
+  }
+
+  const extension = file?.filename?.split('.')?.pop()?.toLowerCase() || '';
+
+  const fileBuffers = await (() => {
+    return new Promise<Buffer>((resolve, reject) => {
+      let buffers = Buffer.from([]);
+      fileStream.on('data', (chunk) => {
+        buffers = Buffer.concat([buffers, chunk]);
+      });
+      fileStream.on('end', () => {
+        resolve(buffers);
+      });
+      fileStream.on('error', (err) => {
+        reject(err);
+      });
+    });
+  })();
+
+  const params: ReadFileByBufferParams = {
+    teamId,
+    buffer: fileBuffers,
+    encoding,
+    metadata: {
+      relatedId: fileId
+    }
+  };
+
+  const { rawText } = await readFileRawContent({
+    extension,
+    csvFormat,
+    params
+  });
+
+  if (rawText.trim()) {
+    MongoRwaTextBuffer.create({
+      sourceId: fileId,
+      rawText,
+      metadata: {
+        filename: file.filename
+      }
+    });
+  }
+
+  return {
+    rawText,
+    filename: file.filename
+  };
+};

packages/service/common/file/image/controller.ts

@@ -14,7 +14,6 @@ export async function uploadMongoImg({
   teamId,
   expiredTime,
   metadata,
-
   shareId
 }: UploadImgProps & {
   teamId: string;
@@ -30,9 +29,8 @@ export async function uploadMongoImg({
     type,
     teamId,
     binary,
-    expiredTime: expiredTime,
+    expiredTime,
     metadata,
-
     shareId
   });
 

packages/service/common/file/image/schema.ts

@@ -25,13 +25,13 @@ const ImageSchema = new Schema({
     enum: Object.keys(mongoImageTypeMap),
     required: true
   },
-
   metadata: {
     type: Object
   }
 });
 
 try {
+  // tts expired
   ImageSchema.index({ expiredTime: 1 }, { expireAfterSeconds: 60 });
   ImageSchema.index({ type: 1 });
   ImageSchema.index({ createTime: 1 });

packages/service/common/file/multer.ts

@@ -37,7 +37,7 @@ export const getUploadModel = ({ maxSize = 500 }: { maxSize?: number }) => {
     async doUpload<T = Record<string, any>>(
       req: NextApiRequest,
       res: NextApiResponse,
-      originBuckerName?: `${BucketNameEnum}`
+      originBucketName?: `${BucketNameEnum}`
     ) {
       return new Promise<{
         file: FileType;
@@ -52,7 +52,7 @@ export const getUploadModel = ({ maxSize = 500 }: { maxSize?: number }) => {
           }
 
           // check bucket name
-          const bucketName = (req.body?.bucketName || originBuckerName) as `${BucketNameEnum}`;
+          const bucketName = (req.body?.bucketName || originBucketName) as `${BucketNameEnum}`;
           if (bucketName && !bucketNameMap[bucketName]) {
             return reject('BucketName is invalid');
           }

packages/service/common/file/read/csv.ts

@@ -0,0 +1,21 @@
+import Papa from 'papaparse';
+import { ReadFileByBufferParams, ReadFileResponse } from './type.d';
+import { readFileRawText } from './rawText';
+
+// 加载源文件内容
+export const readCsvRawText = async (params: ReadFileByBufferParams): Promise<ReadFileResponse> => {
+  const { rawText } = readFileRawText(params);
+
+  const csvArr = Papa.parse(rawText).data as string[][];
+
+  const header = csvArr[0];
+
+  const formatText = header
+    ? csvArr.map((item) => item.map((item, i) => `${header[i]}:${item}`).join('\n')).join('\n')
+    : '';
+
+  return {
+    rawText,
+    formatText
+  };
+};

packages/service/common/file/read/html.ts

@@ -0,0 +1,23 @@
+import { ReadFileByBufferParams, ReadFileResponse } from './type.d';
+import { initMarkdownText } from './utils';
+import { htmlToMarkdown } from '../../string/markdown';
+import { readFileRawText } from './rawText';
+
+export const readHtmlRawText = async (
+  params: ReadFileByBufferParams
+): Promise<ReadFileResponse> => {
+  const { teamId, metadata } = params;
+  const { rawText: html } = readFileRawText(params);
+
+  const md = await htmlToMarkdown(html);
+
+  const rawText = await initMarkdownText({
+    teamId,
+    md,
+    metadata
+  });
+
+  return {
+    rawText
+  };
+};

packages/service/common/file/read/markdown.ts

@@ -0,0 +1,18 @@
+import { ReadFileByBufferParams, ReadFileResponse } from './type.d';
+import { initMarkdownText } from './utils';
+import { readFileRawText } from './rawText';
+
+export const readMarkdown = async (params: ReadFileByBufferParams): Promise<ReadFileResponse> => {
+  const { teamId, metadata } = params;
+  const { rawText: md } = readFileRawText(params);
+
+  const rawText = await initMarkdownText({
+    teamId,
+    md,
+    metadata
+  });
+
+  return {
+    rawText
+  };
+};

packages/service/common/file/read/parseOffice.ts

@@ -0,0 +1,119 @@
+import { getNanoid } from '@fastgpt/global/common/string/tools';
+import fs from 'fs';
+import decompress from 'decompress';
+import { DOMParser } from '@xmldom/xmldom';
+import { clearDirFiles } from '../utils';
+import { addLog } from '../../system/log';
+
+const DEFAULTDECOMPRESSSUBLOCATION = '/tmp';
+
+function getNewFileName(ext: string) {
+  return `${DEFAULTDECOMPRESSSUBLOCATION}/${getNanoid()}.${ext}`;
+}
+
+const parseString = (xml: string) => {
+  let parser = new DOMParser();
+  return parser.parseFromString(xml, 'text/xml');
+};
+
+const parsePowerPoint = async ({
+  filepath,
+  decompressPath,
+  encoding
+}: {
+  filepath: string;
+  decompressPath: string;
+  encoding: BufferEncoding;
+}) => {
+  // Files regex that hold our content of interest
+  const allFilesRegex = /ppt\/(notesSlides|slides)\/(notesSlide|slide)\d+.xml/g;
+  const slidesRegex = /ppt\/slides\/slide\d+.xml/g;
+
+  /** The decompress location which contains the filename in it */
+
+  const files = await decompress(filepath, decompressPath, {
+    filter: (x) => !!x.path.match(allFilesRegex)
+  });
+
+  // Verify if atleast the slides xml files exist in the extracted files list.
+  if (
+    files.length == 0 ||
+    !files.map((file) => file.path).some((filename) => filename.match(slidesRegex))
+  ) {
+    return Promise.reject('解析 PPT 失败');
+  }
+
+  // Returning an array of all the xml contents read using fs.readFileSync
+  const xmlContentArray = files.map((file) =>
+    fs.readFileSync(`${decompressPath}/${file.path}`, encoding)
+  );
+
+  let responseArr: string[] = [];
+
+  xmlContentArray.forEach((xmlContent) => {
+    /** Find text nodes with a:p tags */
+    const xmlParagraphNodesList = parseString(xmlContent).getElementsByTagName('a:p');
+
+    /** Store all the text content to respond */
+    responseArr.push(
+      Array.from(xmlParagraphNodesList)
+        // Filter paragraph nodes than do not have any text nodes which are identifiable by a:t tag
+        .filter((paragraphNode) => paragraphNode.getElementsByTagName('a:t').length != 0)
+        .map((paragraphNode) => {
+          /** Find text nodes with a:t tags */
+          const xmlTextNodeList = paragraphNode.getElementsByTagName('a:t');
+          return Array.from(xmlTextNodeList)
+            .filter((textNode) => textNode.childNodes[0] && textNode.childNodes[0].nodeValue)
+            .map((textNode) => textNode.childNodes[0].nodeValue)
+            .join('');
+        })
+        .join('\n')
+    );
+  });
+
+  return responseArr.join('\n');
+};
+
+export const parseOffice = async ({
+  buffer,
+  encoding,
+  extension
+}: {
+  buffer: Buffer;
+  encoding: BufferEncoding;
+  extension: string;
+}) => {
+  // Prepare file for processing
+  // create temp file subdirectory if it does not exist
+  if (!fs.existsSync(DEFAULTDECOMPRESSSUBLOCATION)) {
+    fs.mkdirSync(DEFAULTDECOMPRESSSUBLOCATION, { recursive: true });
+  }
+
+  // temp file name
+  const filepath = getNewFileName(extension);
+  const decompressPath = `${DEFAULTDECOMPRESSSUBLOCATION}/${getNanoid()}`;
+  //   const decompressPath = `${DEFAULTDECOMPRESSSUBLOCATION}/test`;
+
+  // write new file
+  fs.writeFileSync(filepath, buffer, {
+    encoding
+  });
+
+  const text = await (async () => {
+    try {
+      switch (extension) {
+        case 'pptx':
+          return parsePowerPoint({ filepath, decompressPath, encoding });
+        default:
+          return Promise.reject('只能读取 .pptx 文件');
+      }
+    } catch (error) {
+      addLog.error(`Load ppt error`, { error });
+    }
+    return '';
+  })();
+
+  fs.unlinkSync(filepath);
+  clearDirFiles(decompressPath);
+  return text;
+};

packages/service/common/file/read/pdf.ts

@@ -1,5 +1,7 @@
-/* read file to txt */
-import * as pdfjsLib from 'pdfjs-dist';
+import * as pdfjs from 'pdfjs-dist/legacy/build/pdf.mjs';
+// @ts-ignore
+import('pdfjs-dist/legacy/build/pdf.worker.min.mjs');
+import { ReadFileByBufferParams, ReadFileResponse } from './type';
 
 type TokenType = {
   str: string;
@@ -11,9 +13,9 @@ type TokenType = {
   hasEOL: boolean;
 };
 
-export const readPdfFile = async ({ pdf }: { pdf: ArrayBuffer }) => {
-  pdfjsLib.GlobalWorkerOptions.workerSrc = '/js/pdf.worker.js';
-
+export const readPdfFile = async ({
+  buffer
+}: ReadFileByBufferParams): Promise<ReadFileResponse> => {
   const readPDFPage = async (doc: any, pageNo: number) => {
     const page = await doc.getPage(pageNo);
     const tokenizedText = await page.getTextContent();
@@ -51,14 +53,19 @@ export const readPdfFile = async ({ pdf }: { pdf: ArrayBuffer }) => {
       .join('');
   };
 
-  const doc = await pdfjsLib.getDocument(pdf).promise;
+  const loadingTask = pdfjs.getDocument(buffer.buffer);
+  const doc = await loadingTask.promise;
+
   const pageTextPromises = [];
   for (let pageNo = 1; pageNo <= doc.numPages; pageNo++) {
     pageTextPromises.push(readPDFPage(doc, pageNo));
   }
   const pageTexts = await Promise.all(pageTextPromises);
 
+  loadingTask.destroy();
+
   return {
-    rawText: pageTexts.join('')
+    rawText: pageTexts.join(''),
+    metadata: {}
   };
 };

packages/service/common/file/read/pptx.ts

@@ -0,0 +1,14 @@
+import { ReadFileByBufferParams, ReadFileResponse } from './type.d';
+// import { parseOfficeAsync } from 'officeparser';
+import { parseOffice } from './parseOffice';
+
+export const readPptxRawText = async ({
+  buffer,
+  encoding
+}: ReadFileByBufferParams): Promise<ReadFileResponse> => {
+  const result = await parseOffice({ buffer, encoding, extension: 'pptx' });
+
+  return {
+    rawText: result
+  };
+};

packages/service/common/file/read/rawText.ts

@@ -0,0 +1,10 @@
+import { ReadFileByBufferParams, ReadFileResponse } from './type.d';
+
+// 加载源文件内容
+export const readFileRawText = ({ buffer, encoding }: ReadFileByBufferParams): ReadFileResponse => {
+  const content = buffer.toString(encoding);
+
+  return {
+    rawText: content
+  };
+};

packages/service/common/file/read/type.d.ts

@@ -0,0 +1,12 @@
+export type ReadFileByBufferParams = {
+  teamId: string;
+  buffer: Buffer;
+  encoding: BufferEncoding;
+  metadata?: Record<string, any>;
+};
+
+export type ReadFileResponse = {
+  rawText: string;
+  formatText?: string;
+  metadata?: Record<string, any>;
+};

packages/service/common/file/read/utils.ts

@@ -0,0 +1,81 @@
+import { markdownProcess } from '@fastgpt/global/common/string/markdown';
+import { uploadMongoImg } from '../image/controller';
+import { MongoImageTypeEnum } from '@fastgpt/global/common/file/image/constants';
+import { addHours } from 'date-fns';
+import { ReadFileByBufferParams } from './type';
+import { readFileRawText } from '../read/rawText';
+import { readMarkdown } from '../read/markdown';
+import { readHtmlRawText } from '../read/html';
+import { readPdfFile } from '../read/pdf';
+import { readWordFile } from '../read/word';
+import { readCsvRawText } from '../read/csv';
+import { readPptxRawText } from '../read/pptx';
+import { readXlsxRawText } from '../read/xlsx';
+
+export const initMarkdownText = ({
+  teamId,
+  md,
+  metadata
+}: {
+  md: string;
+  teamId: string;
+  metadata?: Record<string, any>;
+}) =>
+  markdownProcess({
+    rawText: md,
+    uploadImgController: (base64Img) =>
+      uploadMongoImg({
+        type: MongoImageTypeEnum.collectionImage,
+        base64Img,
+        teamId,
+        metadata,
+        expiredTime: addHours(new Date(), 2)
+      })
+  });
+
+export const readFileRawContent = async ({
+  extension,
+  csvFormat,
+  params
+}: {
+  csvFormat?: boolean;
+  extension: string;
+  params: ReadFileByBufferParams;
+}) => {
+  switch (extension) {
+    case 'txt':
+      return readFileRawText(params);
+    case 'md':
+      return readMarkdown(params);
+    case 'html':
+      return readHtmlRawText(params);
+    case 'pdf':
+      return readPdfFile(params);
+    case 'docx':
+      return readWordFile(params);
+    case 'pptx':
+      return readPptxRawText(params);
+    case 'xlsx':
+      const xlsxResult = await readXlsxRawText(params);
+      if (csvFormat) {
+        return {
+          rawText: xlsxResult.formatText || ''
+        };
+      }
+      return {
+        rawText: xlsxResult.rawText
+      };
+    case 'csv':
+      const csvResult = await readCsvRawText(params);
+      if (csvFormat) {
+        return {
+          rawText: csvResult.formatText || ''
+        };
+      }
+      return {
+        rawText: csvResult.rawText
+      };
+    default:
+      return Promise.reject('Only support .txt, .md, .html, .pdf, .docx, pptx, .csv, .xlsx');
+  }
+};

packages/service/common/file/read/word.ts

@@ -0,0 +1,35 @@
+import mammoth from 'mammoth';
+import { htmlToMarkdown } from '../../string/markdown';
+import { ReadFileByBufferParams, ReadFileResponse } from './type';
+import { initMarkdownText } from './utils';
+
+/**
+ * read docx to markdown
+ */
+export const readWordFile = async ({
+  teamId,
+  buffer,
+  metadata = {}
+}: ReadFileByBufferParams): Promise<ReadFileResponse> => {
+  try {
+    const { value: html } = await mammoth.convertToHtml({
+      buffer
+    });
+
+    const md = await htmlToMarkdown(html);
+
+    const rawText = await initMarkdownText({
+      teamId,
+      md,
+      metadata
+    });
+
+    return {
+      rawText,
+      metadata: {}
+    };
+  } catch (error) {
+    console.log('error doc read:', error);
+    return Promise.reject('Can not read doc file, please convert to PDF');
+  }
+};

packages/service/common/file/read/xlsx.ts

@@ -0,0 +1,45 @@
+import { ReadFileByBufferParams, ReadFileResponse } from './type.d';
+import xlsx from 'node-xlsx';
+import Papa from 'papaparse';
+
+export const readXlsxRawText = async ({
+  buffer
+}: ReadFileByBufferParams): Promise<ReadFileResponse> => {
+  const result = xlsx.parse(buffer, {
+    skipHidden: false,
+    defval: ''
+  });
+
+  const format2Csv = result.map(({ name, data }) => {
+    return {
+      title: `#${name}`,
+      csvText: data.map((item) => item.join(',')).join('\n')
+    };
+  });
+
+  const rawText = format2Csv.map((item) => item.csvText).join('\n');
+  const formatText = format2Csv
+    .map((item) => {
+      const csvArr = Papa.parse(item.csvText).data as string[][];
+      const header = csvArr[0];
+
+      const formatText = header
+        ? csvArr
+            .map((item) =>
+              item
+                .map((item, i) => (item ? `${header[i]}:${item}` : ''))
+                .filter(Boolean)
+                .join('\n')
+            )
+            .join('\n')
+        : '';
+
+      return `${item.title}\n${formatText}`;
+    })
+    .join('\n');
+
+  return {
+    rawText: rawText,
+    formatText
+  };
+};

packages/service/common/file/utils.ts

@@ -1,4 +1,6 @@
+import { isProduction } from '../system/constants';
 import fs from 'fs';
+import path from 'path';
 
 export const removeFilesByPaths = (paths: string[]) => {
   paths.forEach((path) => {
@@ -33,12 +35,34 @@ export const clearDirFiles = (dirPath: string) => {
     return;
   }
 
-  fs.readdirSync(dirPath).forEach((file) => {
-    const curPath = `${dirPath}/${file}`;
-    if (fs.lstatSync(curPath).isDirectory()) {
-      clearDirFiles(curPath);
-    } else {
-      fs.unlinkSync(curPath);
+  fs.rmdirSync(dirPath, {
+    recursive: true
+  });
+};
+
+export const clearTmpUploadFiles = () => {
+  if (!isProduction) return;
+  const tmpPath = '/tmp';
+
+  fs.readdir(tmpPath, (err, files) => {
+    if (err) return;
+
+    for (const file of files) {
+      if (file === 'v8-compile-cache-0') continue;
+
+      const filePath = path.join(tmpPath, file);
+
+      fs.stat(filePath, (err, stats) => {
+        if (err) return;
+
+        // 如果文件是在2小时前上传的，则认为是临时文件并删除它
+        if (Date.now() - stats.mtime.getTime() > 2 * 60 * 60 * 1000) {
+          fs.unlink(filePath, (err) => {
+            if (err) return;
+            console.log(`Deleted temp file: ${filePath}`);
+          });
+        }
+      });
     }
   });
 };

packages/service/common/system/constants.ts

@@ -1 +1,3 @@
 export const FastGPTProUrl = process.env.PRO_URL ? `${process.env.PRO_URL}/api` : '';
+
+export const isProduction = process.env.NODE_ENV === 'production';

packages/service/common/system/timerLock/constants.ts

@@ -0,0 +1,15 @@
+export enum TimerIdEnum {
+  checkInValidDatasetFiles = 'checkInValidDatasetFiles',
+  checkInvalidDatasetData = 'checkInvalidDatasetData',
+  checkInvalidVector = 'checkInvalidVector',
+  clearExpiredSubPlan = 'clearExpiredSubPlan',
+  updateStandardPlan = 'updateStandardPlan'
+}
+
+export const timerIdMap = {
+  [TimerIdEnum.checkInValidDatasetFiles]: 'checkInValidDatasetFiles',
+  [TimerIdEnum.checkInvalidDatasetData]: 'checkInvalidDatasetData',
+  [TimerIdEnum.checkInvalidVector]: 'checkInvalidVector',
+  [TimerIdEnum.clearExpiredSubPlan]: 'clearExpiredSubPlan',
+  [TimerIdEnum.updateStandardPlan]: 'updateStandardPlan'
+};

packages/service/common/system/timerLock/schema.ts

@@ -0,0 +1,29 @@
+import { connectionMongo, type Model } from '../../mongo';
+import { timerIdMap } from './constants';
+const { Schema, model, models } = connectionMongo;
+import { TimerLockSchemaType } from './type.d';
+
+export const collectionName = 'systemtimerlocks';
+
+const TimerLockSchema = new Schema({
+  timerId: {
+    type: String,
+    required: true,
+    unique: true,
+    enum: Object.keys(timerIdMap)
+  },
+  expiredTime: {
+    type: Date,
+    required: true
+  }
+});
+
+try {
+  TimerLockSchema.index({ expiredTime: 1 }, { expireAfterSeconds: 5 });
+} catch (error) {
+  console.log(error);
+}
+
+export const MongoTimerLock: Model<TimerLockSchemaType> =
+  models[collectionName] || model(collectionName, TimerLockSchema);
+MongoTimerLock.syncIndexes();

packages/service/common/system/timerLock/type.d.ts

@@ -0,0 +1,5 @@
+export type TimerLockSchemaType = {
+  _id: string;
+  timerId: string;
+  expiredTime: Date;
+};

packages/service/common/system/timerLock/utils.ts

@@ -0,0 +1,25 @@
+import { TimerIdEnum } from './constants';
+import { MongoTimerLock } from './schema';
+import { addMinutes } from 'date-fns';
+
+/* 
+  利用唯一健，使得同一时间只有一个任务在执行，后创建的锁，会因唯一健创建失败，从而无法继续执行任务
+*/
+export const checkTimerLock = async ({
+  timerId,
+  lockMinuted
+}: {
+  timerId: `${TimerIdEnum}`;
+  lockMinuted: number;
+}) => {
+  try {
+    await MongoTimerLock.create({
+      timerId,
+      expiredTime: addMinutes(new Date(), lockMinuted)
+    });
+
+    return true;
+  } catch (error) {
+    return false;
+  }
+};

packages/service/core/dataset/collection/controller.ts

@@ -9,7 +9,6 @@ import {
   DatasetCollectionSchemaType
 } from '@fastgpt/global/core/dataset/type';
 import { MongoDatasetTraining } from '../training/schema';
-import { delay } from '@fastgpt/global/common/system/utils';
 import { MongoDatasetData } from '../data/schema';
 import { delImgByRelatedId } from '../../../common/file/image/controller';
 import { deleteDatasetDataVector } from '../../../common/vectorStore/controller';

packages/service/core/dataset/training/constants.ts

@@ -0,0 +1,6 @@
+export enum ImportDataSourceEnum {
+  fileLocal = 'fileLocal',
+  fileLink = 'fileLink',
+  fileCustom = 'fileCustom',
+  tableLocal = 'tableLocal'
+}

packages/service/core/dataset/training/controller.ts

@@ -1,14 +1,16 @@
-import { delay } from '@fastgpt/global/common/system/utils';
 import { MongoDatasetTraining } from './schema';
 import type {
   PushDatasetDataChunkProps,
   PushDatasetDataProps,
   PushDatasetDataResponse
 } from '@fastgpt/global/core/dataset/api.d';
-import { getCollectionWithDataset } from '../controller';
 import { TrainingModeEnum } from '@fastgpt/global/core/dataset/constants';
 import { simpleText } from '@fastgpt/global/common/string/tools';
 import { countPromptTokens } from '@fastgpt/global/common/string/tiktoken';
+import { ClientSession } from '../../../common/mongo';
+import { getLLMModel, getVectorModel } from '../../ai/model';
+import { addLog } from '../../../common/system/log';
+import { getCollectionWithDataset } from '../controller';
 
 export const lockTrainingDataByTeamId = async (teamId: string): Promise<any> => {
   try {
@@ -23,31 +25,52 @@ export const lockTrainingDataByTeamId = async (teamId: string): Promise<any> =>
   } catch (error) {}
 };
 
-export async function pushDataListToTrainingQueue({
-  teamId,
-  tmbId,
+export const pushDataListToTrainingQueueByCollectionId = async ({
   collectionId,
-  data,
-  prompt,
-  billId,
-  trainingMode = TrainingModeEnum.chunk
+  ...props
 }: {
   teamId: string;
   tmbId: string;
-} & PushDatasetDataProps): Promise<PushDatasetDataResponse> {
-  const vectorModelList = global.vectorModels;
-  const datasetModelList = global.llmModels;
-
+  session?: ClientSession;
+} & PushDatasetDataProps) => {
   const {
-    datasetId: { _id: datasetId, vectorModel, agentModel }
+    datasetId: { _id: datasetId, agentModel, vectorModel }
   } = await getCollectionWithDataset(collectionId);
+  return pushDataListToTrainingQueue({
+    ...props,
+    datasetId,
+    collectionId,
+    agentModel,
+    vectorModel
+  });
+};
 
+export async function pushDataListToTrainingQueue({
+  teamId,
+  tmbId,
+  datasetId,
+  collectionId,
+  agentModel,
+  vectorModel,
+  data,
+  prompt,
+  billId,
+  trainingMode = TrainingModeEnum.chunk,
+  session
+}: {
+  teamId: string;
+  tmbId: string;
+  datasetId: string;
+  agentModel: string;
+  vectorModel: string;
+  session?: ClientSession;
+} & PushDatasetDataProps): Promise<PushDatasetDataResponse> {
   const checkModelValid = async () => {
-    const agentModelData = datasetModelList?.find((item) => item.model === agentModel);
+    const agentModelData = getLLMModel(agentModel);
     if (!agentModelData) {
       return Promise.reject(`File model ${agentModel} is inValid`);
     }
-    const vectorModelData = vectorModelList?.find((item) => item.model === vectorModel);
+    const vectorModelData = getVectorModel(vectorModel);
     if (!vectorModelData) {
       return Promise.reject(`Vector model ${vectorModel} is inValid`);
     }
@@ -107,7 +130,7 @@ export async function pushDataListToTrainingQueue({
     const text = item.q + item.a;
 
     // count q token
-    const token = countPromptTokens(item.q);
+    const token = item.q.length;
 
     if (token > maxToken) {
       filterResult.overToken.push(item);
@@ -124,52 +147,43 @@ export async function pushDataListToTrainingQueue({
   });
 
   // insert data to db
-  const insertData = async (dataList: PushDatasetDataChunkProps[], retry = 3): Promise<number> => {
-    try {
-      const results = await MongoDatasetTraining.insertMany(
-        dataList.map((item, i) => ({
-          teamId,
-          tmbId,
-          datasetId,
-          collectionId,
-          billId,
-          mode: trainingMode,
-          prompt,
-          model,
-          q: item.q,
-          a: item.a,
-          chunkIndex: item.chunkIndex ?? 0,
-          weight: weight ?? 0,
-          indexes: item.indexes
-        }))
-      );
-      await delay(500);
-      return results.length;
-    } catch (error) {
-      if (retry > 0) {
-        await delay(500);
-        return insertData(dataList, retry - 1);
-      }
-      return Promise.reject(error);
-    }
-  };
+  const insertLen = filterResult.success.length;
+  const failedDocuments: PushDatasetDataChunkProps[] = [];
 
-  let insertLen = 0;
-  const chunkSize = 50;
-  const chunkList = filterResult.success.reduce(
-    (acc, cur) => {
-      const lastChunk = acc[acc.length - 1];
-      if (lastChunk.length < chunkSize) {
-        lastChunk.push(cur);
-      } else {
-        acc.push([cur]);
+  // 使用 insertMany 批量插入
+  try {
+    await MongoDatasetTraining.insertMany(
+      filterResult.success.map((item) => ({
+        teamId,
+        tmbId,
+        datasetId,
+        collectionId,
+        billId,
+        mode: trainingMode,
+        prompt,
+        model,
+        q: item.q,
+        a: item.a,
+        chunkIndex: item.chunkIndex ?? 0,
+        weight: weight ?? 0,
+        indexes: item.indexes
+      })),
+      {
+        session
       }
-      return acc;
-    },
-    [[]] as PushDatasetDataChunkProps[][]
-  );
-  for await (const chunks of chunkList) {
-    insertLen += await insertData(chunks);
+    );
+  } catch (error: any) {
+    addLog.error(`Insert error`, error);
+    // 如果有错误，将失败的文档添加到失败列表中
+    error.writeErrors.forEach((writeError: any) => {
+      failedDocuments.push(data[writeError.index]);
+    });
+    console.log('failed', failedDocuments);
+  }
+
+  // 对于失败的文档，尝试单独插入
+  for await (const item of failedDocuments) {
+    await MongoDatasetTraining.create(item);
   }
 
   delete filterResult.success;

packages/service/core/dataset/training/utils.ts

@@ -2,6 +2,7 @@ import { DatasetTrainingSchemaType } from '@fastgpt/global/core/dataset/type';
 import { addLog } from '../../../common/system/log';
 import { getErrText } from '@fastgpt/global/common/error/utils';
 import { MongoDatasetTraining } from './schema';
+import Papa from 'papaparse';
 
 export const checkInvalidChunkAndLock = async ({
   err,
@@ -39,3 +40,18 @@ export const checkInvalidChunkAndLock = async ({
   }
   return false;
 };
+
+export const parseCsvTable2Chunks = (rawText: string) => {
+  const csvArr = Papa.parse(rawText).data as string[][];
+
+  const chunks = csvArr
+    .map((item) => ({
+      q: item[0] || '',
+      a: item[1] || ''
+    }))
+    .filter((item) => item.q || item.a);
+
+  return {
+    chunks
+  };
+};

packages/service/core/workflow/dispatch/agent/runTool/toolChoice.ts

@@ -210,7 +210,6 @@ export const runToolWithToolChoice = async (
   ).filter(Boolean) as ToolRunResponseType;
 
   const flatToolsResponseData = toolsRunResponse.map((item) => item.moduleRunResponse).flat();
-
   if (toolCalls.length > 0 && !res.closed) {
     // Run the tool, combine its results, and perform another round of AI calls
     const assistantToolMsgParams: ChatCompletionAssistantToolParam = {

packages/service/core/workflow/dispatch/index.ts

@@ -37,6 +37,7 @@ import { dispatchRunTools } from './agent/runTool/index';
 import { ChatItemValueTypeEnum } from '@fastgpt/global/core/chat/constants';
 import { DispatchFlowResponse } from './type';
 import { dispatchStopToolCall } from './agent/runTool/stopTool';
+import { dispatchLafRequest } from './tools/runLaf';
 
 const callbackMap: Record<`${FlowNodeTypeEnum}`, Function> = {
   [FlowNodeTypeEnum.historyNode]: dispatchHistory,
@@ -56,6 +57,7 @@ const callbackMap: Record<`${FlowNodeTypeEnum}`, Function> = {
   [FlowNodeTypeEnum.queryExtension]: dispatchQueryExtension,
   [FlowNodeTypeEnum.tools]: dispatchRunTools,
   [FlowNodeTypeEnum.stopTool]: dispatchStopToolCall,
+  [FlowNodeTypeEnum.lafModule]: dispatchLafRequest,
 
   // none
   [FlowNodeTypeEnum.userGuide]: () => Promise.resolve()

packages/service/core/workflow/dispatch/tools/http468.ts

@@ -10,6 +10,7 @@ import { valueTypeFormat } from '../utils';
 import { SERVICE_LOCAL_HOST } from '../../../../common/system/tools';
 import { addLog } from '../../../../common/system/log';
 import { DispatchNodeResultType } from '@fastgpt/global/core/module/runtime/type';
+import { getErrText } from '@fastgpt/global/common/error/utils';
 
 type PropsArrType = {
   key: string;
@@ -61,7 +62,7 @@ export const dispatchHttp468Request = async (props: HttpRequestProps): Promise<H
     chatId,
     responseChatItemId,
     ...variables,
-    histories: histories.slice(0, 10),
+    histories: histories.slice(-10),
     ...body
   };
 
@@ -139,7 +140,8 @@ export const dispatchHttp468Request = async (props: HttpRequestProps): Promise<H
         body: Object.keys(requestBody).length > 0 ? requestBody : undefined,
         headers: Object.keys(headers).length > 0 ? headers : undefined,
         httpResult: { error: formatHttpError(error) }
-      }
+      },
+      [ModuleOutputKeyEnum.httpRawResponse]: getErrText(error)
     };
   }
 };
@@ -165,6 +167,7 @@ async function fetchData({
       'Content-Type': 'application/json',
       ...headers
     },
+    timeout: 120000,
     params: params,
     data: ['POST', 'PUT', 'PATCH'].includes(method) ? body : undefined
   });

packages/service/core/workflow/dispatch/tools/runLaf.ts

@@ -0,0 +1,212 @@
+import type { ModuleDispatchProps } from '@fastgpt/global/core/module/type.d';
+import {
+  DYNAMIC_INPUT_KEY,
+  ModuleInputKeyEnum,
+  ModuleOutputKeyEnum
+} from '@fastgpt/global/core/module/constants';
+import { DispatchNodeResponseKeyEnum } from '@fastgpt/global/core/module/runtime/constants';
+import axios from 'axios';
+import { valueTypeFormat } from '../utils';
+import { SERVICE_LOCAL_HOST } from '../../../../common/system/tools';
+import { addLog } from '../../../../common/system/log';
+import { DispatchNodeResultType } from '@fastgpt/global/core/module/runtime/type';
+
+type LafRequestProps = ModuleDispatchProps<{
+  [ModuleInputKeyEnum.httpReqUrl]: string;
+  [DYNAMIC_INPUT_KEY]: Record<string, any>;
+  [key: string]: any;
+}>;
+type LafResponse = DispatchNodeResultType<{
+  [ModuleOutputKeyEnum.failed]?: boolean;
+  [key: string]: any;
+}>;
+
+const UNDEFINED_SIGN = 'UNDEFINED_SIGN';
+
+export const dispatchLafRequest = async (props: LafRequestProps): Promise<LafResponse> => {
+  let {
+    appId,
+    chatId,
+    responseChatItemId,
+    variables,
+    module: { outputs },
+    histories,
+    params: { system_httpReqUrl: httpReqUrl, [DYNAMIC_INPUT_KEY]: dynamicInput, ...body }
+  } = props;
+
+  if (!httpReqUrl) {
+    return Promise.reject('Http url is empty');
+  }
+
+  const concatVariables = {
+    appId,
+    chatId,
+    responseChatItemId,
+    ...variables,
+    ...body
+  };
+
+  httpReqUrl = replaceVariable(httpReqUrl, concatVariables);
+
+  const requestBody = {
+    systemParams: {
+      appId,
+      chatId,
+      responseChatItemId,
+      histories: histories.slice(0, 10)
+    },
+    variables,
+    ...dynamicInput,
+    ...body
+  };
+
+  try {
+    const { formatResponse, rawResponse } = await fetchData({
+      method: 'POST',
+      url: httpReqUrl,
+      body: requestBody
+    });
+
+    // format output value type
+    const results: Record<string, any> = {};
+    for (const key in formatResponse) {
+      const output = outputs.find((item) => item.key === key);
+      if (!output) continue;
+      results[key] = valueTypeFormat(formatResponse[key], output.valueType);
+    }
+
+    return {
+      assistantResponses: [],
+      [DispatchNodeResponseKeyEnum.nodeResponse]: {
+        totalPoints: 0,
+        body: Object.keys(requestBody).length > 0 ? requestBody : undefined,
+        httpResult: rawResponse
+      },
+      [DispatchNodeResponseKeyEnum.toolResponses]: rawResponse,
+      [ModuleOutputKeyEnum.httpRawResponse]: rawResponse,
+      ...results
+    };
+  } catch (error) {
+    addLog.error('Http request error', error);
+    return {
+      [ModuleOutputKeyEnum.failed]: true,
+      [DispatchNodeResponseKeyEnum.nodeResponse]: {
+        totalPoints: 0,
+        body: Object.keys(requestBody).length > 0 ? requestBody : undefined,
+        httpResult: { error: formatHttpError(error) }
+      }
+    };
+  }
+};
+
+async function fetchData({
+  method,
+  url,
+  body
+}: {
+  method: string;
+  url: string;
+  body: Record<string, any>;
+}): Promise<Record<string, any>> {
+  const { data: response } = await axios({
+    method,
+    baseURL: `http://${SERVICE_LOCAL_HOST}`,
+    url,
+    headers: {
+      'Content-Type': 'application/json'
+    },
+    data: body
+  });
+
+  const parseJson = (obj: Record<string, any>, prefix = '') => {
+    let result: Record<string, any> = {};
+
+    if (Array.isArray(obj)) {
+      for (let i = 0; i < obj.length; i++) {
+        result[`${prefix}[${i}]`] = obj[i];
+
+        if (Array.isArray(obj[i])) {
+          result = {
+            ...result,
+            ...parseJson(obj[i], `${prefix}[${i}]`)
+          };
+        } else if (typeof obj[i] === 'object') {
+          result = {
+            ...result,
+            ...parseJson(obj[i], `${prefix}[${i}].`)
+          };
+        }
+      }
+    } else if (typeof obj == 'object') {
+      for (const key in obj) {
+        result[`${prefix}${key}`] = obj[key];
+
+        if (Array.isArray(obj[key])) {
+          result = {
+            ...result,
+            ...parseJson(obj[key], `${prefix}${key}`)
+          };
+        } else if (typeof obj[key] === 'object') {
+          result = {
+            ...result,
+            ...parseJson(obj[key], `${prefix}${key}.`)
+          };
+        }
+      }
+    }
+
+    return result;
+  };
+
+  return {
+    formatResponse:
+      typeof response === 'object' && !Array.isArray(response) ? parseJson(response) : {},
+    rawResponse: response
+  };
+}
+
+function replaceVariable(text: string, obj: Record<string, any>) {
+  for (const [key, value] of Object.entries(obj)) {
+    if (value === undefined) {
+      text = text.replace(new RegExp(`{{${key}}}`, 'g'), UNDEFINED_SIGN);
+    } else {
+      const replacement = JSON.stringify(value);
+      const unquotedReplacement =
+        replacement.startsWith('"') && replacement.endsWith('"')
+          ? replacement.slice(1, -1)
+          : replacement;
+      text = text.replace(new RegExp(`{{${key}}}`, 'g'), unquotedReplacement);
+    }
+  }
+  return text || '';
+}
+function removeUndefinedSign(obj: Record<string, any>) {
+  for (const key in obj) {
+    if (obj[key] === UNDEFINED_SIGN) {
+      obj[key] = undefined;
+    } else if (Array.isArray(obj[key])) {
+      obj[key] = obj[key].map((item: any) => {
+        if (item === UNDEFINED_SIGN) {
+          return undefined;
+        } else if (typeof item === 'object') {
+          removeUndefinedSign(item);
+        }
+        return item;
+      });
+    } else if (typeof obj[key] === 'object') {
+      removeUndefinedSign(obj[key]);
+    }
+  }
+  return obj;
+}
+function formatHttpError(error: any) {
+  return {
+    message: error?.message,
+    name: error?.name,
+    method: error?.config?.method,
+    baseURL: error?.config?.baseURL,
+    url: error?.config?.url,
+    code: error?.code,
+    status: error?.status
+  };
+}

packages/service/package.json

@@ -4,27 +4,36 @@
   "dependencies": {
     "@fastgpt/global": "workspace:*",
     "@node-rs/jieba": "1.10.0",
+    "@xmldom/xmldom": "^0.8.10",
     "axios": "^1.5.1",
     "cheerio": "1.0.0-rc.12",
     "cookie": "^0.5.0",
     "date-fns": "2.30.0",
     "dayjs": "^1.11.7",
+    "decompress": "^4.2.1",
     "encoding": "^0.1.13",
+    "file-type": "^19.0.0",
     "json5": "^2.2.3",
     "jsonwebtoken": "^9.0.2",
+    "mammoth": "^1.6.0",
     "mongoose": "^7.0.2",
     "multer": "1.4.5-lts.1",
     "next": "13.5.2",
     "nextjs-cors": "^2.1.2",
     "node-cron": "^3.0.3",
+    "node-xlsx": "^0.23.0",
+    "papaparse": "5.4.1",
+    "pdfjs-dist": "4.0.269",
     "pg": "^8.10.0",
     "tunnel": "^0.0.6"
   },
   "devDependencies": {
     "@types/cookie": "^0.5.2",
+    "@types/decompress": "^4.2.7",
     "@types/jsonwebtoken": "^9.0.3",
     "@types/multer": "^1.4.10",
     "@types/node-cron": "^3.0.11",
+    "@types/papaparse": "5.3.7",
     "@types/pg": "^8.6.6",
     "@types/tunnel": "^0.0.4"
   }

packages/service/support/permission/auth/file.ts

@@ -0,0 +1,42 @@
+import { AuthResponseType } from '@fastgpt/global/support/permission/type';
+import { AuthModeType } from '../type';
+import { DatasetFileSchema } from '@fastgpt/global/core/dataset/type';
+import { parseHeaderCert } from '../controller';
+import { getFileById } from '../../../common/file/gridfs/controller';
+import { BucketNameEnum } from '@fastgpt/global/common/file/constants';
+import { CommonErrEnum } from '@fastgpt/global/common/error/code/common';
+
+export async function authFile({
+  fileId,
+  per = 'owner',
+  ...props
+}: AuthModeType & {
+  fileId: string;
+}): Promise<
+  AuthResponseType & {
+    file: DatasetFileSchema;
+  }
+> {
+  const authRes = await parseHeaderCert(props);
+  const { teamId, tmbId } = authRes;
+
+  const file = await getFileById({ bucketName: BucketNameEnum.dataset, fileId });
+
+  if (!file) {
+    return Promise.reject(CommonErrEnum.fileNotFound);
+  }
+
+  if (file.metadata?.teamId !== teamId) {
+    return Promise.reject(CommonErrEnum.unAuthFile);
+  }
+  if (per === 'owner' && file.metadata?.tmbId !== tmbId) {
+    return Promise.reject(CommonErrEnum.unAuthFile);
+  }
+
+  return {
+    ...authRes,
+    isOwner: per === 'owner',
+    canWrite: per === 'owner',
+    file
+  };
+}

packages/service/support/user/team/controller.ts

@@ -10,7 +10,6 @@ import { MongoTeam } from './teamSchema';
 
 async function getTeamMember(match: Record<string, any>): Promise<TeamItemType> {
   const tmb = (await MongoTeamMember.findOne(match).populate('teamId')) as TeamMemberWithTeamSchema;
-
   if (!tmb) {
     return Promise.reject('member not exist');
   }
@@ -27,7 +26,8 @@ async function getTeamMember(match: Record<string, any>): Promise<TeamItemType>
     role: tmb.role,
     status: tmb.status,
     defaultTeam: tmb.defaultTeam,
-    canWrite: tmb.role !== TeamMemberRoleEnum.visitor
+    canWrite: tmb.role !== TeamMemberRoleEnum.visitor,
+    lafAccount: tmb.teamId.lafAccount
   };
 }
 

packages/service/support/user/team/teamSchema.ts

@@ -35,6 +35,14 @@ const TeamSchema = new Schema({
     lastWebsiteSyncTime: {
       type: Date
     }
+  },
+  lafAccount: {
+    token: {
+      type: String
+    },
+    appid: {
+      type: String
+    }
   }
 });
 

packages/web/common/file/read/csv.ts

@@ -1,40 +0,0 @@
-import Papa from 'papaparse';
-import { readFileRawText } from './rawText';
-
-/**
- * read csv to json
- * @response {
- *  header: string[],
- *  data: string[][]
- * }
- */
-export const readCsvContent = async ({ file }: { file: File }) => {
-  try {
-    const { rawText: textArr } = await readFileRawText(file);
-    const csvArr = Papa.parse(textArr).data as string[][];
-    if (csvArr.length === 0) {
-      throw new Error('csv 解析失败');
-    }
-
-    const header = csvArr.shift() as string[];
-
-    // add title to data
-    const rawText = csvArr
-      .map((item) =>
-        item.map((value, index) => {
-          if (!header[index]) return value;
-          return `${header[index]}: ${value}`;
-        })
-      )
-      .flat()
-      .join('\n');
-
-    return {
-      rawText,
-      header,
-      data: csvArr.map((item) => item)
-    };
-  } catch (error) {
-    return Promise.reject('解析 csv 文件失败');
-  }
-};

packages/web/common/file/read/html.ts

@@ -1,21 +0,0 @@
-import { htmlStr2Md } from '../../string/markdown';
-import { readFileRawText } from './rawText';
-import { markdownProcess } from '@fastgpt/global/common/string/markdown';
-
-export const readHtmlFile = async ({
-  file,
-  uploadImgController
-}: {
-  file: File;
-  uploadImgController?: (base64: string) => Promise<string>;
-}) => {
-  const { rawText } = await readFileRawText(file);
-  const md = htmlStr2Md(rawText);
-
-  const simpleMd = await markdownProcess({
-    rawText: md,
-    uploadImgController
-  });
-
-  return { rawText: simpleMd };
-};

packages/web/common/file/read/index.ts

@@ -1,49 +0,0 @@
-import { loadFile2Buffer } from '../utils';
-import { readCsvContent } from './csv';
-import { readHtmlFile } from './html';
-import { readMdFile } from './md';
-import { readPdfFile } from './pdf';
-import { readFileRawText } from './rawText';
-import { readWordFile } from './word';
-
-export const readFileRawContent = async ({
-  file,
-  uploadBase64Controller
-}: {
-  file: File;
-  uploadBase64Controller?: (base64: string) => Promise<string>;
-}): Promise<{
-  rawText: string;
-}> => {
-  const extension = file?.name?.split('.')?.pop()?.toLowerCase();
-
-  switch (extension) {
-    case 'txt':
-      return readFileRawText(file);
-    case 'md':
-      return readMdFile({
-        file,
-        uploadImgController: uploadBase64Controller
-      });
-    case 'html':
-      return readHtmlFile({
-        file,
-        uploadImgController: uploadBase64Controller
-      });
-    case 'csv':
-      return readCsvContent({ file });
-    case 'pdf':
-      const pdf = await loadFile2Buffer({ file });
-      return readPdfFile({ pdf });
-    case 'docx':
-      return readWordFile({
-        file,
-        uploadImgController: uploadBase64Controller
-      });
-
-    default:
-      return {
-        rawText: ''
-      };
-  }
-};

packages/web/common/file/read/md.ts

@@ -1,17 +0,0 @@
-import { markdownProcess } from '@fastgpt/global/common/string/markdown';
-import { readFileRawText } from './rawText';
-
-export const readMdFile = async ({
-  file,
-  uploadImgController
-}: {
-  file: File;
-  uploadImgController?: (base64: string) => Promise<string>;
-}) => {
-  const { rawText: md } = await readFileRawText(file);
-  const simpleMd = await markdownProcess({
-    rawText: md,
-    uploadImgController
-  });
-  return { rawText: simpleMd };
-};

packages/web/common/file/read/rawText.ts

@@ -1,36 +0,0 @@
-import { detectFileEncoding } from '@fastgpt/global/common/file/tools';
-
-/**
- * read file raw text
- */
-export const readFileRawText = (file: File) => {
-  return new Promise<{ rawText: string }>((resolve, reject) => {
-    try {
-      const reader = new FileReader();
-      reader.onload = () => {
-        //@ts-ignore
-        const encode = detectFileEncoding(reader.result);
-
-        // 再次读取文件，这次使用检测到的编码
-        const reader2 = new FileReader();
-        reader2.onload = () => {
-          resolve({
-            rawText: reader2.result as string
-          });
-        };
-        reader2.onerror = (err) => {
-          console.log('Error reading file with detected encoding:', err);
-          reject('Read file error with detected encoding');
-        };
-        reader2.readAsText(file, encode);
-      };
-      reader.onerror = (err) => {
-        console.log('error txt read:', err);
-        reject('Read file error');
-      };
-      reader.readAsBinaryString(file);
-    } catch (error) {
-      reject(error);
-    }
-  });
-};

packages/web/common/file/read/word.ts

@@ -1,28 +0,0 @@
-import { markdownProcess } from '@fastgpt/global/common/string/markdown';
-import { htmlStr2Md } from '../../string/markdown';
-import { loadFile2Buffer } from '../utils';
-import mammoth from 'mammoth';
-
-export const readWordFile = async ({
-  file,
-  uploadImgController
-}: {
-  file: File;
-  uploadImgController?: (base64: string) => Promise<string>;
-}) => {
-  const buffer = await loadFile2Buffer({ file });
-
-  const { value: html } = await mammoth.convertToHtml({
-    arrayBuffer: buffer
-  });
-  const md = htmlStr2Md(html);
-
-  const rawText = await markdownProcess({
-    rawText: md,
-    uploadImgController: uploadImgController
-  });
-
-  return {
-    rawText
-  };
-};

packages/web/components/common/Icon/constants.ts

@@ -101,6 +101,7 @@ export const iconPaths = {
   'core/dataset/mixedRecall': () => import('./icons/core/dataset/mixedRecall.svg'),
   'core/dataset/modeEmbedding': () => import('./icons/core/dataset/modeEmbedding.svg'),
   'core/dataset/rerank': () => import('./icons/core/dataset/rerank.svg'),
+  'core/dataset/splitLight': () => import('./icons/core/dataset/splitLight.svg'),
   'core/dataset/tableCollection': () => import('./icons/core/dataset/tableCollection.svg'),
   'core/dataset/websiteDataset': () => import('./icons/core/dataset/websiteDataset.svg'),
   'core/modules/basicNode': () => import('./icons/core/modules/basicNode.svg'),
@@ -124,7 +125,9 @@ export const iconPaths = {
   'file/fill/manual': () => import('./icons/file/fill/manual.svg'),
   'file/fill/markdown': () => import('./icons/file/fill/markdown.svg'),
   'file/fill/pdf': () => import('./icons/file/fill/pdf.svg'),
+  'file/fill/ppt': () => import('./icons/file/fill/ppt.svg'),
   'file/fill/txt': () => import('./icons/file/fill/txt.svg'),
+  'file/fill/xlsx': () => import('./icons/file/fill/xlsx.svg'),
   'file/html': () => import('./icons/file/html.svg'),
   'file/indexImport': () => import('./icons/file/indexImport.svg'),
   'file/manualImport': () => import('./icons/file/manualImport.svg'),

packages/web/components/common/Icon/icons/core/dataset/splitLight.svg

@@ -0,0 +1,6 @@
+<svg t="1711938287623" class="icon" viewBox="0 0 1024 1024" version="1.1" xmlns="http://www.w3.org/2000/svg"
+    p-id="5143">
+    <path
+        d="M153.6 153.6h716.8a51.2 51.2 0 0 1 0 102.4H153.6a51.2 51.2 0 1 1 0-102.4z m0 614.4h716.8a51.2 51.2 0 0 1 0 102.4H153.6a51.2 51.2 0 0 1 0-102.4z m0-307.2h131.6352a51.2 51.2 0 1 1 0 102.4H153.6a51.2 51.2 0 0 1 0-102.4z m292.5568 0h131.6864a51.2 51.2 0 0 1 0 102.4H446.1568a51.2 51.2 0 0 1 0-102.4z m292.608 0H870.4a51.2 51.2 0 0 1 0 102.4h-131.6352a51.2 51.2 0 0 1 0-102.4z"
+        p-id="5144"></path>
+</svg>

packages/web/components/common/Icon/icons/file/fill/ppt.svg

@@ -0,0 +1,11 @@
+<svg t="1712108366314" class="icon" viewBox="0 0 1024 1024" version="1.1" xmlns="http://www.w3.org/2000/svg" p-id="1576"
+    width="128" height="128">
+    <path
+        d="M145.6 0C100.8 0 64 36.8 64 81.6v860.8C64 987.2 100.8 1024 145.6 1024h732.8c44.8 0 81.6-36.8 81.6-81.6V324.8L657.6 0h-512z"
+        fill="#E34221" p-id="1577"></path>
+    <path d="M960 326.4v16H755.2s-100.8-20.8-99.2-108.8c0 0 4.8 92.8 97.6 92.8H960z" fill="#DC3119" p-id="1578"></path>
+    <path d="M657.6 0v233.6c0 25.6 17.6 92.8 97.6 92.8H960L657.6 0z" fill="#FFFFFF" p-id="1579"></path>
+    <path
+        d="M304 784h-54.4v67.2c0 6.4-4.8 11.2-11.2 11.2-6.4 0-12.8-4.8-12.8-11.2V686.4c0-9.6 8-17.6 17.6-17.6H304c38.4 0 59.2 25.6 59.2 57.6S340.8 784 304 784z m-3.2-94.4h-51.2v73.6h51.2c22.4 0 38.4-16 38.4-36.8 0-22.4-16-36.8-38.4-36.8zM480 784h-54.4v67.2c0 6.4-4.8 11.2-11.2 11.2-6.4 0-11.2-4.8-11.2-11.2V686.4c0-9.6 6.4-17.6 16-17.6H480c38.4 0 59.2 25.6 59.2 57.6S518.4 784 480 784z m-3.2-94.4h-49.6v73.6h49.6c22.4 0 38.4-16 38.4-36.8 0-22.4-16-36.8-38.4-36.8z m225.6 0h-52.8v161.6c0 6.4-4.8 11.2-11.2 11.2-6.4 0-12.8-4.8-12.8-11.2V689.6h-51.2c-6.4 0-11.2-4.8-11.2-11.2 0-4.8 4.8-9.6 11.2-9.6h128c6.4 0 11.2 4.8 11.2 11.2 0 4.8-4.8 9.6-11.2 9.6z"
+        fill="#FFFFFF" p-id="1580"></path>
+</svg>

packages/web/components/common/Icon/icons/file/fill/xlsx.svg

@@ -0,0 +1,11 @@
+<svg t="1712108380525" class="icon" viewBox="0 0 1024 1024" version="1.1" xmlns="http://www.w3.org/2000/svg" p-id="2654"
+    width="128" height="128">
+    <path
+        d="M145.6 0C100.8 0 64 36.8 64 81.6v860.8C64 987.2 100.8 1024 145.6 1024h732.8c44.8 0 81.6-36.8 81.6-81.6V324.8L657.6 0h-512z"
+        fill="#45B058" p-id="2655"></path>
+    <path
+        d="M374.4 862.4c-3.2 0-6.4-1.6-8-3.2l-59.2-80-60.8 80c-1.6 1.6-4.8 3.2-8 3.2-6.4 0-11.2-4.8-11.2-11.2 0-1.6 0-4.8 1.6-6.4l62.4-81.6-57.6-78.4c-1.6-1.6-3.2-3.2-3.2-6.4 0-4.8 4.8-11.2 11.2-11.2 4.8 0 8 1.6 9.6 4.8l56 73.6 54.4-73.6c1.6-3.2 4.8-4.8 8-4.8 6.4 0 12.8 4.8 12.8 11.2 0 3.2-1.6 4.8-1.6 6.4l-59.2 76.8 62.4 83.2c1.6 1.6 3.2 4.8 3.2 6.4 0 6.4-6.4 11.2-12.8 11.2z m160-1.6H448c-9.6 0-17.6-8-17.6-17.6V678.4c0-6.4 4.8-11.2 12.8-11.2 6.4 0 11.2 4.8 11.2 11.2v161.6h80c6.4 0 11.2 4.8 11.2 9.6 0 6.4-4.8 11.2-11.2 11.2z m112 3.2c-28.8 0-51.2-9.6-67.2-24-3.2-1.6-3.2-4.8-3.2-8 0-6.4 3.2-12.8 11.2-12.8 1.6 0 4.8 1.6 6.4 3.2 12.8 11.2 32 20.8 54.4 20.8 33.6 0 44.8-19.2 44.8-33.6 0-49.6-113.6-22.4-113.6-89.6 0-32 27.2-54.4 65.6-54.4 24 0 46.4 8 60.8 20.8 3.2 1.6 4.8 4.8 4.8 8 0 6.4-4.8 12.8-11.2 12.8-1.6 0-4.8-1.6-6.4-3.2-14.4-11.2-32-16-49.6-16-24 0-40 11.2-40 30.4 0 43.2 113.6 17.6 113.6 89.6 0 27.2-19.2 56-70.4 56z"
+        fill="#FFFFFF" p-id="2656"></path>
+    <path d="M960 326.4v16H755.2s-102.4-20.8-99.2-108.8c0 0 3.2 92.8 96 92.8h208z" fill="#349C42" p-id="2657"></path>
+    <path d="M656 0v233.6c0 25.6 19.2 92.8 99.2 92.8H960L656 0z" fill="#FFFFFF" p-id="2658"></path>
+</svg>

packages/web/components/common/Input/HttpInput/Editor.tsx

@@ -36,7 +36,7 @@ export default function Editor({
   hasVariablePlugin?: boolean;
   hasDropDownPlugin?: boolean;
   variables: EditorVariablePickerType[];
-  onChange?: (editorState: EditorState) => void;
+  onChange?: (editorState: EditorState, editor: LexicalEditor) => void;
   onBlur?: (editor: LexicalEditor) => void;
   value?: string;
   currentValue?: string;
@@ -119,9 +119,9 @@ export default function Editor({
         <HistoryPlugin />
         <FocusPlugin focus={focus} setFocus={setFocus} />
         <OnChangePlugin
-          onChange={(e) => {
+          onChange={(editorState: EditorState, editor: LexicalEditor) => {
             startSts(() => {
-              onChange?.(e);
+              onChange?.(editorState, editor);
             });
           }}
         />

packages/web/components/common/Input/HttpInput/index.tsx

@@ -32,15 +32,13 @@ const HttpInput = ({
 
   const [, startSts] = useTransition();
 
-  const onChangeInput = useCallback((editorState: EditorState) => {
-    const text = editorState.read(() => $getRoot().getTextContent());
-    const formatValue = text.replaceAll('\n\n', '\n').replaceAll('}}{{', '}} {{');
-    setCurrentValue(formatValue);
-    onChange?.(formatValue);
+  const onChangeInput = useCallback((editorState: EditorState, editor: LexicalEditor) => {
+    const text = editorStateToText(editor).replaceAll('}}{{', '}} {{');
+    onChange?.(text);
   }, []);
   const onBlurInput = useCallback((editor: LexicalEditor) => {
     startSts(() => {
-      const text = editorStateToText(editor).replaceAll('\n\n', '\n').replaceAll('}}{{', '}} {{');
+      const text = editorStateToText(editor).replaceAll('}}{{', '}} {{');
       onBlur?.(text);
     });
   }, []);

packages/web/components/common/MyDrawer/MyRightDrawer.tsx

@@ -0,0 +1,70 @@
+import React from 'react';
+import MyIcon from '../Icon';
+import {
+  Drawer,
+  DrawerBody,
+  DrawerHeader,
+  DrawerOverlay,
+  DrawerContent,
+  DrawerCloseButton,
+  DrawerContentProps,
+  Flex,
+  Image
+} from '@chakra-ui/react';
+import { useLoading } from '../../../hooks/useLoading';
+
+type Props = DrawerContentProps & {
+  onClose: () => void;
+  iconSrc?: string;
+  title?: any;
+  isLoading?: boolean;
+};
+
+const MyRightDrawer = ({
+  onClose,
+  iconSrc,
+  title,
+  maxW = ['90vw', '30vw'],
+  children,
+  isLoading,
+  ...props
+}: Props) => {
+  const { Loading } = useLoading();
+  return (
+    <Drawer isOpen placement="right" onClose={onClose}>
+      <DrawerOverlay />
+      <DrawerContent
+        maxW={maxW}
+        {...props}
+        h={'94%'}
+        mt={'2%'}
+        borderLeftRadius={'lg'}
+        overflow={'hidden'}
+      >
+        <DrawerCloseButton />
+        <DrawerHeader>
+          <Flex alignItems={'center'} pr={2}>
+            {iconSrc && (
+              <>
+                {iconSrc.startsWith('/') ? (
+                  <Image mr={3} objectFit={'contain'} alt="" src={iconSrc} w={'20px'} />
+                ) : (
+                  <MyIcon mr={3} name={iconSrc as any} w={'20px'} />
+                )}
+              </>
+            )}
+            {title}
+          </Flex>
+          <DrawerCloseButton zIndex={1} />
+        </DrawerHeader>
+
+        <DrawerBody>
+          {children}
+          <Loading loading={isLoading} fixed={false} />
+        </DrawerBody>
+      </DrawerContent>
+    </Drawer>
+  );
+};
+
+export default MyRightDrawer;

packages/web/components/common/MySelect/index.tsx

@@ -7,10 +7,13 @@ import {
   useDisclosure,
   MenuButton,
   Box,
-  css
+  css,
+  Flex
 } from '@chakra-ui/react';
 import type { ButtonProps, MenuItemProps } from '@chakra-ui/react';
 import { ChevronDownIcon } from '@chakra-ui/icons';
+import { useLoading } from '../../../hooks/useLoading';
+import MyIcon from '../Icon';
 
 export type SelectProps = ButtonProps & {
   value?: string;
@@ -20,14 +23,16 @@ export type SelectProps = ButtonProps & {
     label: string | React.ReactNode;
     value: string;
   }[];
+  isLoading?: boolean;
   onchange?: (val: any) => void;
 };
 
 const MySelect = (
-  { placeholder, value, width = '100%', list, onchange, ...props }: SelectProps,
+  { placeholder, value, width = '100%', list, onchange, isLoading = false, ...props }: SelectProps,
   selectRef: any
 ) => {
   const ref = useRef<HTMLButtonElement>(null);
+  const { Loading } = useLoading();
   const menuItemStyles: MenuItemProps = {
     borderRadius: 'sm',
     py: 2,
@@ -78,7 +83,10 @@ const MySelect = (
             : {})}
           {...props}
         >
-          {selectItem?.alias || selectItem?.label || placeholder}
+          <Flex alignItems={'center'}>
+            {isLoading && <MyIcon mr={2} name={'common/loading'} w={'16px'} />}
+            {selectItem?.alias || selectItem?.label || placeholder}
+          </Flex>
         </MenuButton>
 
         <MenuList

packages/web/components/common/Radio/LeftRadio.tsx

@@ -2,6 +2,8 @@ import React from 'react';
 import { Box, Flex, useTheme, Grid, type GridProps } from '@chakra-ui/react';
 import { useTranslation } from 'next-i18next';
 import MyTooltip from '../MyTooltip';
+import { QuestionOutlineIcon } from '@chakra-ui/icons';
+import QuestionTip from '../MyTooltip/QuestionTip';
 
 // @ts-ignore
 interface Props extends GridProps {
@@ -36,58 +38,59 @@ const LeftRadio = ({
   return (
     <Grid gridGap={[3, 5]} fontSize={['sm', 'md']} {...props}>
       {list.map((item) => (
-        <MyTooltip key={item.value} label={item.tooltip}>
-          <Flex
-            alignItems={item.desc ? align : 'center'}
-            cursor={'pointer'}
-            userSelect={'none'}
-            px={px}
-            py={py}
-            border={theme.borders.sm}
-            borderWidth={'1px'}
-            borderRadius={'md'}
-            position={'relative'}
-            {...(value === item.value
-              ? {
-                  borderColor: 'primary.400',
-                  bg: activeBg,
-                  boxShadow: 'focus'
+        <Flex
+          alignItems={item.desc ? align : 'center'}
+          key={item.value}
+          cursor={'pointer'}
+          userSelect={'none'}
+          px={px}
+          py={py}
+          border={theme.borders.sm}
+          borderWidth={'1px'}
+          borderRadius={'md'}
+          position={'relative'}
+          {...(value === item.value
+            ? {
+                borderColor: 'primary.400',
+                bg: activeBg,
+                boxShadow: 'focus'
+              }
+            : {
+                bg: defaultBg,
+                _hover: {
+                  borderColor: 'primary.300'
                 }
-              : {
-                  bg: defaultBg,
-                  _hover: {
-                    borderColor: 'primary.300'
-                  }
-                })}
-            onClick={() => onChange(item.value)}
+              })}
+          onClick={() => onChange(item.value)}
+        >
+          <Box
+            w={'18px'}
+            h={'18px'}
+            borderWidth={'2.4px'}
+            borderColor={value === item.value ? 'primary.015' : 'transparent'}
+            borderRadius={'50%'}
+            mr={3}
           >
-            <Box
-              w={'18px'}
-              h={'18px'}
-              borderWidth={'2.4px'}
-              borderColor={value === item.value ? 'primary.015' : 'transparent'}
+            <Flex
+              w={'100%'}
+              h={'100%'}
+              borderWidth={'1px'}
+              borderColor={value === item.value ? 'primary.600' : 'borderColor.high'}
+              bg={value === item.value ? 'primary.1' : 'transparent'}
               borderRadius={'50%'}
-              mr={3}
+              alignItems={'center'}
+              justifyContent={'center'}
             >
-              <Flex
-                w={'100%'}
-                h={'100%'}
-                borderWidth={'1px'}
-                borderColor={value === item.value ? 'primary.600' : 'borderColor.high'}
-                bg={value === item.value ? 'primary.1' : 'transparent'}
+              <Box
+                w={'5px'}
+                h={'5px'}
                 borderRadius={'50%'}
-                alignItems={'center'}
-                justifyContent={'center'}
-              >
-                <Box
-                  w={'5px'}
-                  h={'5px'}
-                  borderRadius={'50%'}
-                  bg={value === item.value ? 'primary.600' : 'transparent'}
-                ></Box>
-              </Flex>
-            </Box>
-            <Box flex={'1 0 0'}>
+                bg={value === item.value ? 'primary.600' : 'transparent'}
+              ></Box>
+            </Flex>
+          </Box>
+          <Box flex={'1 0 0'}>
+            <Flex alignItems={'center'}>
               <Box
                 color={'myGray.900'}
                 fontWeight={item.desc ? '500' : 'normal'}
@@ -95,15 +98,16 @@ const LeftRadio = ({
               >
                 {typeof item.title === 'string' ? t(item.title) : item.title}
               </Box>
-              {!!item.desc && (
-                <Box fontSize={'xs'} color={'myGray.500'} lineHeight={1.2}>
-                  {t(item.desc)}
-                </Box>
-              )}
-              {item?.children}
-            </Box>
-          </Flex>
-        </MyTooltip>
+              {!!item.tooltip && <QuestionTip label={item.tooltip} ml={1} color={'myGray.600'} />}
+            </Flex>
+            {!!item.desc && (
+              <Box fontSize={'xs'} color={'myGray.500'} lineHeight={1.2}>
+                {t(item.desc)}
+              </Box>
+            )}
+            {item?.children}
+          </Box>
+        </Flex>
       ))}
     </Grid>
   );

packages/web/components/common/Textarea/PromptEditor/index.tsx

@@ -1,10 +1,10 @@
 import { Button, ModalBody, ModalFooter, useDisclosure } from '@chakra-ui/react';
-import React, { useEffect, useState } from 'react';
+import React from 'react';
 import { editorStateToText } from './utils';
 import Editor from './Editor';
 import MyModal from '../../MyModal';
 import { useTranslation } from 'next-i18next';
-import { $getRoot, EditorState, type LexicalEditor } from 'lexical';
+import { EditorState, type LexicalEditor } from 'lexical';
 import { EditorVariablePickerType } from './type.d';
 import { useCallback, useTransition } from 'react';
 
@@ -34,16 +34,12 @@ const PromptEditor = ({
   const { t } = useTranslation();
 
   const onChangeInput = useCallback((editorState: EditorState, editor: LexicalEditor) => {
-    const stringifiedEditorState = JSON.stringify(editorState.toJSON());
-    const parsedEditorState = editor.parseEditorState(stringifiedEditorState);
-    const editorStateTextString = parsedEditorState.read(() => $getRoot().getTextContent());
-
-    const formatValue = editorStateTextString.replaceAll('\n\n', '\n').replaceAll('}}{{', '}} {{');
-    onChange?.(formatValue);
+    const text = editorStateToText(editor).replaceAll('}}{{', '}} {{');
+    onChange?.(text);
   }, []);
   const onBlurInput = useCallback((editor: LexicalEditor) => {
     startSts(() => {
-      const text = editorStateToText(editor).replaceAll('\n\n', '\n').replaceAll('}}{{', '}} {{');
+      const text = editorStateToText(editor).replaceAll('}}{{', '}} {{');
       onBlur?.(text);
     });
   }, []);

packages/web/components/common/Textarea/PromptEditor/plugins/OnBlurPlugin/index.tsx

@@ -1,6 +1,5 @@
 import { useEffect } from 'react';
 import { BLUR_COMMAND, COMMAND_PRIORITY_EDITOR, LexicalEditor } from 'lexical';
-import { mergeRegister } from '@lexical/utils';
 import { useLexicalComposerContext } from '@lexical/react/LexicalComposerContext';
 
 export default function OnBlurPlugin({ onBlur }: { onBlur?: (editor: LexicalEditor) => void }) {

packages/web/components/common/Textarea/PromptEditor/utils.ts

@@ -209,9 +209,8 @@ export function editorStateToText(editor: LexicalEditor) {
   const stringifiedEditorState = JSON.stringify(editor.getEditorState().toJSON());
   const parsedEditorState = editor.parseEditorState(stringifiedEditorState);
   const editorStateTextString = parsedEditorState.read(() => $getRoot().getTextContent());
-  const compressedText = editorStateTextString.replace(/\n+/g, '\n\n');
 
-  return compressedText;
+  return editorStateTextString;
 }
 
 const varRegex = /\{\{([a-zA-Z_][a-zA-Z0-9_]*)\}\}/g;

packages/web/package.json

@@ -12,31 +12,31 @@
     "@emotion/styled": "^11.11.0",
     "@fastgpt/global": "workspace:*",
     "@fingerprintjs/fingerprintjs": "^4.2.1",
+    "@lexical/react": "0.12.6",
+    "@lexical/text": "0.12.6",
+    "@lexical/utils": "0.12.6",
     "@monaco-editor/react": "^4.6.0",
-    "mammoth": "^1.6.0",
+    "@tanstack/react-query": "^4.24.10",
+    "date-fns": "2.30.0",
+    "dayjs": "^1.11.7",
     "i18next": "23.10.0",
     "joplin-turndown-plugin-gfm": "^1.0.12",
+    "lexical": "0.12.6",
+    "lodash": "^4.17.21",
+    "mammoth": "^1.6.0",
     "next-i18next": "15.2.0",
+    "papaparse": "^5.4.1",
     "pdfjs-dist": "4.0.269",
     "react": "18.2.0",
+    "react-day-picker": "^8.7.1",
     "react-dom": "18.2.0",
     "react-i18next": "13.5.0",
-    "turndown": "^7.1.2",
-    "lexical": "0.12.6",
-    "@lexical/react": "0.12.6",
-    "papaparse": "^5.4.1",
-    "@lexical/utils": "0.12.6",
-    "@lexical/text": "0.12.6",
-    "date-fns": "2.30.0",
-    "react-day-picker": "^8.7.1",
-    "lodash": "^4.17.21",
-    "@tanstack/react-query": "^4.24.10",
-    "dayjs": "^1.11.7"
+    "turndown": "^7.1.2"
   },
   "devDependencies": {
     "@types/lodash": "^4.14.191",
-    "@types/react": "18.2.0",
     "@types/papaparse": "^5.3.7",
+    "@types/react": "18.2.0",
     "@types/react-dom": "18.2.0",
     "@types/turndown": "^5.0.4"
   }

projects/app/data/config.json

@@ -1,4 +1,7 @@
 {
+  "feConfigs": {
+    "lafEnv": "https://laf.dev"
+  },
   "systemEnv": {
     "openapiPrefix": "fastgpt",
     "vectorMaxProcess": 15,
@@ -82,7 +85,7 @@
       "name": "Embedding-2",
       "avatar": "/imgs/model/openai.svg",
       "charsPointsPrice": 0,
-      "defaultToken": 700,
+      "defaultToken": 512,
       "maxToken": 3000,
       "weight": 100,
       "dbConfig": {},

projects/app/public/js/iframe.js

@@ -23,7 +23,7 @@ function embedChatbot() {
   const ChatBtn = document.createElement('div');
   ChatBtn.id = chatBtnId;
   ChatBtn.style.cssText =
-    'position: fixed; bottom: 1rem; right: 1rem; width: 40px; height: 40px; cursor: pointer; z-index: 2147483647; transition: 0;';
+    'position: fixed; bottom: 30px; right: 60px; width: 40px; height: 40px; cursor: pointer; z-index: 2147483647; transition: 0;';
 
   // btn icon
   const ChatBtnDiv = document.createElement('img');
@@ -39,7 +39,7 @@ function embedChatbot() {
   iframe.id = chatWindowId;
   iframe.src = botSrc;
   iframe.style.cssText =
-    'border: none; position: fixed; flex-direction: column; justify-content: space-between; box-shadow: rgba(150, 150, 150, 0.2) 0px 10px 30px 0px, rgba(150, 150, 150, 0.2) 0px 0px 0px 1px; bottom: 4rem; right: 1rem; width: 24rem; height: 40rem; max-width: 90vw; max-height: 85vh; border-radius: 0.75rem; display: flex; z-index: 2147483647; overflow: hidden; left: unset; background-color: #F3F4F6;';
+    'border: none; position: fixed; flex-direction: column; justify-content: space-between; box-shadow: rgba(150, 150, 150, 0.2) 0px 10px 30px 0px, rgba(150, 150, 150, 0.2) 0px 0px 0px 1px; bottom: 80px; right: 60px; width: 375px; height: 667px; max-width: 90vw; max-height: 85vh; border-radius: 0.75rem; display: flex; z-index: 2147483647; overflow: hidden; left: unset; background-color: #F3F4F6;';
   iframe.style.visibility = defaultOpen ? 'unset' : 'hidden';
 
   document.body.appendChild(iframe);

projects/app/public/locales/en/common.json

@@ -56,6 +56,7 @@
     }
   },
   "common": {
+    "Action": "Action",
     "Add": "Add",
     "Add New": "Add",
     "All": "All",
@@ -79,6 +80,7 @@
     "Create New": "Create",
     "Create Success": "Create Success",
     "Create Time": "Create time",
+    "Creating": "Creating",
     "Custom Title": "Custom Title",
     "Delete": "Delete",
     "Delete Failed": "Delete Failed",
@@ -146,6 +148,7 @@
     "Status": "Status",
     "Submit failed": "Submit failed",
     "Submit success": "Update Success",
+    "Sync success": "",
     "System version": "System version",
     "Team": "Team",
     "Team Tags Set": "Team Tags",
@@ -191,6 +194,7 @@
       "Empty file tip": "The file content is empty. The file may be unreadable or pure image file content.",
       "File Content": "File Content",
       "File Name": "File Name",
+      "File Size": "File Size",
       "File content can not be empty": "File content can not be empty",
       "Filename Can not Be Empty": "Filename Can not Be Empty",
       "Read File Error": "Read file error",
@@ -198,6 +202,7 @@
       "Select failed": "Select file failed",
       "Select file amount limit": "A maximum of {{max}} files can be selected",
       "Select file amount limit 100": "You can select a maximum of 100 files at a time",
+      "Some file count exceeds limit": "The number of files exceeding {{maxCount}} has been automatically intercepted",
       "Some file size exceeds limit": "Some files exceed: {{maxSize}}, have been filtered",
       "Support file type": "Support {{fileType}} files",
       "Support max count": "A maximum of {{maxCount}} files are supported.",
@@ -620,7 +625,7 @@
       "file": "File",
       "folder": "Folder",
       "import": {
-        "Auto mode Estimated Price Tips": "Enhanced processing calls the file processing model: {{price}} integral /1k Tokens",
+        "Auto mode Estimated Price Tips": "Need to call the file processing model, need to consume more Tokens: {{price}} credits /1k Tokens",
         "Auto process": "Auto",
         "Auto process desc": "Automatically set segmentation and preprocessing rules",
         "CSV Import": "CSV QA Import",
@@ -642,7 +647,7 @@
         "Data file progress": "Data upload progress",
         "Data process params": "Data process params",
         "Down load csv template": "Down load csv template",
-        "Embedding Estimated Price Tips": "Index billing: {{price}}/1k Tokens",
+        "Embedding Estimated Price Tips": "Use only the index model and consume a small amount of Tokens: {{price}} credits /1k Tokens",
         "Estimated Price": "Estimated Price: : {{amount}}{{unit}}",
         "Estimated Price Tips": "QA charges: {{charsPointsPrice}} points/1k Tokens",
         "Estimated points": "About {{points}} points",
@@ -657,15 +662,19 @@
         "Import Failed": "Import Failed",
         "Import Success Tip": "The {{num}} group data is imported successfully. Please wait for training.",
         "Import Tip": "This task cannot be terminated and takes some time to generate indexes. Please confirm the import. If the balance is insufficient, the unfinished task will be suspended and can continue after topping up.",
+        "Import success": "Import successful, please wait for training",
         "Link name": "Link name",
         "Link name placeholder": "Only static links are supported\nOne per line, up to 10 links at a time",
         "Local file": "Local file",
         "Local file desc": "Upload files in PDF, TXT, DOCX and other formats",
         "Only Show First 50 Chunk": "Show only part",
-        "Preview chunks": "Chunks",
-        "Preview raw text": "Preview file text (max show 10000 words)",
+        "Predicted chunk": "Predicted chunk",
+        "Predicted chunk amount": "Predicted chunks:{{amount}}",
+        "Predicted total chars": "Predicted chars: {{total}}",
+        "Preview chunks": "Preview chunks",
+        "Preview raw text": "Preview file text (max show 3000 words)",
         "Process way": "Process way",
-        "QA Estimated Price Tips": "QA billing: {{price}}/1k Tokens (including input and output)",
+        "QA Estimated Price Tips": "Need to call the file processing model, need to consume more Tokens: {{price}} credits /1k Tokens",
         "QA Import": "QA Split",
         "QA Import Tip": "According to certain rules, the text is broken into a larger paragraph, and the AI is invoked to generate a question and answer pair for the paragraph.",
         "Re Preview": "RePreview",
@@ -680,8 +689,8 @@
         "Total tokens": "Tokens",
         "Training mode": "Training mode",
         "Upload data": "Upload data",
-        "Upload file progress": "File upload progress",
-        "Upload status": "Upload status",
+        "Upload file progress": "Upload state",
+        "Upload status": "Status",
         "Upload success": "Upload success",
         "Web link": "Web link",
         "Web link desc": "Fetch static web content as a collection"
@@ -811,6 +820,7 @@
       "Http request props": "Request props",
       "Http request settings": "Request settings",
       "Input Type": "Input Type",
+      "Laf sync params": "Sync params",
       "Plugin output must connect": "Custom outputs must all be connected",
       "QueryExtension": {
         "placeholder": "Questions about python introduction and usage, etc. The current conversation is related to the game GTA5.",
@@ -904,6 +914,9 @@
         "target": "Target Data",
         "textarea": "Textarea"
       },
+      "laf": {
+        "Select laf function": ""
+      },
       "output": {
         "Add Output": "Add Output",
         "Output Number": "Output: {{length}}",
@@ -1218,19 +1231,26 @@
     "Set Public": "Set to public"
   },
   "plugin": {
+    "App": "Choose App",
     "Auth Header Prefix": "Auth header prefix",
     "Auth Method": "Auth method",
     "Auth Type": "Auth Type",
     "Confirm Delete": "Confirm to delete the plugin?",
     "Create Your Plugin": "Create Plugin",
+    "Currentapp": "CurrentApp",
     "Custom Plugin": "Custom plugin",
     "Description": "Description",
     "Edit Http Plugin": "Edit HTTP plugin",
+    "Enter Env": "Enter laf environment",
+    "Enter PAT": "Please enter personal access token (PAT)",
+    "Func": "Choose Function",
     "Get Plugin Module Detail Failed": "Get plugin detail failed",
     "HTTP Plugin": "HTTP plugin",
     "Import Plugin": "Import HTTP plugin",
     "Import from URL": "Import from URL. https://xxxx",
     "Intro": "Plugin Intro",
+    "Invalid Appid": "Invalid appid",
+    "Invalid Env": "Invalid Env",
     "Invalid Schema": "Invalid Schema",
     "Invalid URL": "Invalid URL",
     "Key": "Key",
@@ -1240,16 +1260,20 @@
     "No Intro": "This plugin is not introduced",
     "None": "None",
     "Path": "Path",
+    "Please bind laf accout first": "Please bind laf accout first",
     "Plugin List": "Plugin list",
     "Plugin Module": "Plugin",
     "Privacy Agreement": "privacy agreement",
     "Search plugin": "Search plugins",
     "Set Name": "Plugin Name",
+    "Synchronous app": "Sync App",
     "Synchronous version": "Sync Version",
     "To Edit Plugin": "To Edit",
     "Update Your Plugin": "Update Plugin",
     "Value": "Value",
-    "path": ""
+    "go to laf": "go to laf",
+    "path": "",
+    "update params": "update params"
   },
   "support": {
     "account": {
@@ -1295,6 +1319,9 @@
     "user": {
       "AI point standard": "AI points price",
       "Avatar": "Avatar",
+      "Go laf env": "Click to go to laf to get PAT certificate.",
+      "Laf account course": "Check out the laf account binding tutorial.",
+      "Laf account intro": "After binding your laf account, you will be able to use the laf module in your workflow to write code online.",
       "Need to login": "Please log in first",
       "Price": "Price",
       "User self info": "My info",
@@ -1348,6 +1375,7 @@
       "Pay error": "Pay error",
       "Pay success": "Pay success",
       "Plan expired time": "Plan expired time",
+      "Plan reset time": "Plan reset time",
       "Standard Plan Detail": "Standard Plan Detail",
       "To read plan": "Read plan",
       "bill": {
@@ -1486,10 +1514,13 @@
     "Bill Detail": "Bill Detail",
     "Change": "Change",
     "Copy invite url": "Copy invitation link",
+    "Current laf Env": "Current laf Env",
     "Edit name": "Click to modify nickname",
     "Invite Url": "Invite Url",
     "Invite url tip": "Friends who register through this link will be permanently bound to you, and you will get a certain balance reward when they recharge. In addition, when friends register with their mobile phone number, you will get 5 yuan reward immediately.",
+    "Laf Account Setting": "laf account setting",
     "Language": "Language",
+    "Learn More": "Learn More",
     "Member Name": "Name",
     "Notice": "Notice",
     "Old password is error": "Old password is error",
@@ -1504,6 +1535,7 @@
     "Promotion rate tip": "You will be rewarded with a percentage of the balance when your friends top up",
     "Recharge Record": "Recharge",
     "Replace": "Replace",
+    "Set Laf Account Failed": "set laf accout failed",
     "Set OpenAI Account Failed": "Set OpenAI account failed",
     "Sign Out": "Sign Out",
     "Source": "Source",

projects/app/public/locales/zh/common.json

@@ -56,6 +56,7 @@
     }
   },
   "common": {
+    "Action": "操作",
     "Add": "添加",
     "Add New": "新增",
     "All": "全部",
@@ -79,6 +80,7 @@
     "Create New": "新建",
     "Create Success": "创建成功",
     "Create Time": "创建时间",
+    "Creating": "创建中",
     "Custom Title": "自定义标题",
     "Delete": "删除",
     "Delete Failed": "删除失败",
@@ -146,6 +148,7 @@
     "Status": "状态",
     "Submit failed": "提交失败",
     "Submit success": "提交成功",
+    "Sync success": "同步成功",
     "System version": "系统版本",
     "Team": "团队",
     "Team Tags Set": "标签",
@@ -191,6 +194,7 @@
       "Empty file tip": "文件内容为空，可能该文件无法读取或为纯图片文件内容。",
       "File Content": "文件内容",
       "File Name": "文件名",
+      "File Size": "文件大小",
       "File content can not be empty": "文件内容不能为空",
       "Filename Can not Be Empty": "文件名不能为空",
       "Read File Error": "解析文件失败",
@@ -198,6 +202,7 @@
       "Select failed": "选择文件异常",
       "Select file amount limit": "最多选择 {{max}} 个文件",
       "Select file amount limit 100": "每次最多选择100个文件",
+      "Some file count exceeds limit": "超出{{maxCount}}个文件，已自动截取",
       "Some file size exceeds limit": "部分文件超出: {{maxSize}}，已被过滤",
       "Support file type": "支持 {{fileType}} 类型文件",
       "Support max count": "最多支持 {{maxCount}} 个文件。",
@@ -622,7 +627,7 @@
       "file": "文件",
       "folder": "目录",
       "import": {
-        "Auto mode Estimated Price Tips": "增强处理需调用文件处理模型: {{price}}积分/1k Tokens",
+        "Auto mode Estimated Price Tips": "需调用文件处理模型，需要消耗较多Tokens: {{price}}积分/1k Tokens",
         "Auto process": "自动",
         "Auto process desc": "自动设置分割和预处理规则",
         "CSV Import": "CSV 导入",
@@ -644,7 +649,7 @@
         "Data file progress": "数据上传进度",
         "Data process params": "数据处理参数",
         "Down load csv template": "点击下载 CSV 模板",
-        "Embedding Estimated Price Tips": "索引计费: {{price}}积分/1k Tokens",
+        "Embedding Estimated Price Tips": "仅使用索引模型，消耗少量Tokens: {{price}}积分/1k Tokens",
         "Estimated Price": "预估价格: {{amount}}{{unit}}",
         "Estimated Price Tips": "QA计费为\n输入: {{charsPointsPrice}}积分/1k Tokens",
         "Estimated points": "预估消耗 {{points}} 积分",
@@ -659,15 +664,19 @@
         "Import Failed": "导入文件失败",
         "Import Success Tip": "共成功导入 {{num}} 组数据，请耐心等待训练.",
         "Import Tip": "该任务无法终止，需要一定时间生成索引，请确认导入。如果余额不足，未完成的任务会被暂停，充值后可继续进行。",
+        "Import success": "导入成功，请等待训练",
         "Link name": "网络链接",
         "Link name placeholder": "仅支持静态链接，如果上传后数据为空，可能该链接无法被读取\n每行一个，每次最多 10 个链接",
         "Local file": "本地文件",
         "Local file desc": "上传 PDF, TXT, DOCX 等格式的文件",
         "Only Show First 50 Chunk": "仅展示部分",
-        "Preview chunks": "分段预览",
-        "Preview raw text": "预览源文本（最多展示10000字）",
+        "Predicted chunk": "预估分段",
+        "Predicted chunk amount": "预估分段:{{amount}}",
+        "Predicted total chars": "预估字数: {{total}}",
+        "Preview chunks": "预览分段（最多5段）",
+        "Preview raw text": "预览源文本（最多3000字）",
         "Process way": "处理方式",
-        "QA Estimated Price Tips": "QA计费为: {{price}}积分/1k Tokens(包含输入和输出)",
+        "QA Estimated Price Tips": "需调用文件处理模型，需要消耗较多Tokens: {{price}}积分/1k Tokens",
         "QA Import": "QA拆分",
         "QA Import Tip": "根据一定规则，将文本拆成一段较大的段落，调用 AI 为该段落生成问答对。有非常高的检索精度，但是会丢失很多内容细节。",
         "Re Preview": "重新生成预览",
@@ -683,7 +692,7 @@
         "Training mode": "训练模式",
         "Upload data": "上传数据",
         "Upload file progress": "文件上传进度",
-        "Upload status": "上传状态",
+        "Upload status": "状态",
         "Upload success": "上传成功",
         "Web link": "网页链接",
         "Web link desc": "读取静态网页内容作为数据集"
@@ -813,6 +822,7 @@
       "Http request props": "请求参数",
       "Http request settings": "请求配置",
       "Input Type": "输入类型",
+      "Laf sync params": "同步参数",
       "Plugin output must connect": "自定义输出必须全部连接",
       "QueryExtension": {
         "placeholder": "例如:\n关于 python 的介绍和使用等问题。\n当前对话与游戏《GTA5》有关。",
@@ -845,11 +855,11 @@
         "AppId": "应用的ID",
         "ChatId": "当前对话ID",
         "Current time": "当前时间",
-        "Histories": "历史纪录,最多取10条",
+        "Histories": "历史记录,最多取10条",
         "Key already exists": "Key 已经存在",
         "Key cannot be empty": "参数名不能为空",
         "Props name": "参数名",
-        "Props tip": "可以设置 Http 请求的相关参数\n可通过 {{key}} 来调用全局变了或外部参数输入，当前可使用变量:\n{{variable}}",
+        "Props tip": "可以设置 Http 请求的相关参数\n可通过 {{key}} 来调用全局变量或外部参数输入，当前可使用变量:\n{{variable}}",
         "Props value": "参数值",
         "ResponseChatItemId": "AI回复的ID",
         "Url and params have been split": "路径参数已被自动加入 Params 中",
@@ -906,6 +916,9 @@
         "target": "外部数据",
         "textarea": "段落输入"
       },
+      "laf": {
+        "Select laf function": "选择laf函数"
+      },
       "output": {
         "Add Output": "添加出参",
         "Output Number": "出参: {{length}}",
@@ -1220,19 +1233,26 @@
     "Set Public": "设为团队可用"
   },
   "plugin": {
+    "App": "选择应用",
     "Auth Header Prefix": "鉴权头部前缀",
     "Auth Method": "鉴权方法",
     "Auth Type": "鉴权类型",
     "Confirm Delete": "确认删除该插件？",
     "Create Your Plugin": "创建你的插件",
+    "Currentapp": "当前应用",
     "Custom Plugin": "自定义插件",
     "Description": "描述",
     "Edit Http Plugin": "编辑 HTTP 插件",
+    "Enter Env": "输入 laf 环境",
+    "Enter PAT": "请输入访问凭证（PAT）",
+    "Func": "选择函数",
     "Get Plugin Module Detail Failed": "获取插件信息异常",
     "HTTP Plugin": "HTTP 插件",
     "Import Plugin": "导入 HTTP 插件",
     "Import from URL": "从URL导入。https://xxxx",
     "Intro": "插件介绍",
+    "Invalid Appid": "appid 无效",
+    "Invalid Env": "laf 环境错误",
     "Invalid Schema": "Schema 无效",
     "Invalid URL": "URL 无效",
     "Key": "键",
@@ -1242,16 +1262,20 @@
     "No Intro": "这个插件没有介绍~",
     "None": "无",
     "Path": "路径",
+    "Please bind laf accout first": "请先绑定 laf 账号",
     "Plugin List": "插件列表",
     "Plugin Module": "插件模块",
     "Privacy Agreement": "隐私协议",
     "Search plugin": "搜索插件",
     "Set Name": "给插件取个名字",
+    "Synchronous app": "同步应用",
     "Synchronous version": "同步版本",
     "To Edit Plugin": "去编辑",
     "Update Your Plugin": "更新插件",
     "Value": "值",
-    "path": ""
+    "go to laf": "去编写",
+    "path": "",
+    "update params": "更新参数"
   },
   "support": {
     "account": {
@@ -1297,6 +1321,9 @@
     "user": {
       "AI point standard": "AI积分标准",
       "Avatar": "头像",
+      "Go laf env": "点击前往 laf 获取 PAT 凭证。",
+      "Laf account course": "查看绑定 laf 账号教程。",
+      "Laf account intro": "绑定你的laf账号后，你将可以在工作流中使用 laf 模块，实现在线编写代码。",
       "Need to login": "请先登录",
       "Price": "计费标准",
       "User self info": "个人信息",
@@ -1350,6 +1377,7 @@
       "Pay error": "支付失败",
       "Pay success": "支付成功",
       "Plan expired time": "套餐到期时间",
+      "Plan reset time": "套餐重置时间",
       "Standard Plan Detail": "套餐详情",
       "To read plan": "查看套餐",
       "bill": {
@@ -1407,7 +1435,7 @@
         "Standard update fail": "修改订阅套餐异常",
         "Standard update success": "变更订阅套餐成功！",
         "Sub plan": "订阅套餐",
-        "Sub plan tip": "免费使用 FastGPT 或升级更高的套餐",
+        "Sub plan tip": "免费使用 {{title}} 或升级更高的套餐",
         "Team plan and usage": "套餐与用量",
         "Training weight": "训练优先级: {{weight}}",
         "Update extra ai points": "额外AI积分",
@@ -1488,10 +1516,13 @@
     "Bill Detail": "账单详情",
     "Change": "变更",
     "Copy invite url": "复制邀请链接",
+    "Current laf Env": "当前 laf 环境",
     "Edit name": "点击修改昵称",
     "Invite Url": "邀请链接",
     "Invite url tip": "通过该链接注册的好友将永久与你绑定，其充值时你会获得一定余额奖励。\n此外，好友使用手机号注册时，你将立即获得 5 元奖励。\n奖励会发送到您的默认团队中。",
+    "Laf Account Setting": "laf 账号配置",
     "Language": "语言",
+    "Learn More": "查看文档",
     "Member Name": "昵称",
     "Notice": "通知",
     "Old password is error": "旧密码错误",
@@ -1506,6 +1537,7 @@
     "Promotion rate tip": "好友充值时你将获得一定比例的余额奖励",
     "Recharge Record": "支付记录",
     "Replace": "更换",
+    "Set Laf Account Failed": "设置 laf 账号异常",
     "Set OpenAI Account Failed": "设置 OpenAI 账号异常",
     "Sign Out": "登出",
     "Source": "来源",

projects/app/src/components/ChatBox/components/ChatItem.tsx

@@ -192,6 +192,7 @@ ${toolResponse}`}
               </Box>
             );
           }
+          return null;
         })}
       </Flex>
     );

projects/app/src/components/core/module/Flow/FlowProvider.tsx

@@ -214,9 +214,6 @@ export const FlowProvider = ({
         if (source?.flowType === FlowNodeTypeEnum.classifyQuestion && !type) {
           return ModuleIOValueTypeEnum.boolean;
         }
-        if (source?.flowType === FlowNodeTypeEnum.tools) {
-          return ModuleIOValueTypeEnum.tools;
-        }
         if (source?.flowType === FlowNodeTypeEnum.pluginInput) {
           return source?.inputs.find((input) => input.key === connect.sourceHandle)?.valueType;
         }

projects/app/src/components/core/module/Flow/ModuleTemplateList.tsx

@@ -52,6 +52,7 @@ const ModuleTemplateList = ({ isOpen, onClose }: ModuleTemplateListProps) => {
   const router = useRouter();
   const [currentParent, setCurrentParent] = useState<RenderListProps['currentParent']>();
   const [searchKey, setSearchKey] = useState('');
+  const { feConfigs } = useSystemStore();
 
   const {
     basicNodeTemplates,
@@ -64,7 +65,12 @@ const ModuleTemplateList = ({ isOpen, onClose }: ModuleTemplateListProps) => {
 
   const templates = useMemo(() => {
     const map = {
-      [TemplateTypeEnum.basic]: basicNodeTemplates,
+      [TemplateTypeEnum.basic]: basicNodeTemplates.filter((item) => {
+        if (item.flowType === FlowNodeTypeEnum.lafModule && !feConfigs.lafEnv) {
+          return false;
+        }
+        return true;
+      }),
       [TemplateTypeEnum.systemPlugin]: systemNodeTemplates,
       [TemplateTypeEnum.teamPlugin]: teamPluginNodeTemplates.filter((item) =>
         searchKey ? item.pluginType !== PluginTypeEnum.folder : true

projects/app/src/components/core/module/Flow/components/nodes/NodeHttp/index.tsx

@@ -88,7 +88,7 @@ enum TabEnum {
   headers = 'headers',
   body = 'body'
 }
-type PropsArrType = {
+export type PropsArrType = {
   key: string;
   type: string;
   value: string;
@@ -245,7 +245,7 @@ const RenderHttpMethodAndUrl = React.memo(function RenderHttpMethodAndUrl({
   );
 });
 
-function RenderHttpProps({
+export function RenderHttpProps({
   moduleId,
   inputs
 }: {

projects/app/src/components/core/module/Flow/components/nodes/NodeLaf.tsx

@@ -0,0 +1,260 @@
+import React, { useCallback, useMemo } from 'react';
+import { NodeProps } from 'reactflow';
+import NodeCard from '../render/NodeCard';
+import { FlowModuleItemType } from '@fastgpt/global/core/module/type.d';
+import Container from '../modules/Container';
+import { Box, Button, Center, Flex, useDisclosure } from '@chakra-ui/react';
+import { ModuleIOValueTypeEnum, ModuleInputKeyEnum } from '@fastgpt/global/core/module/constants';
+import { onChangeNode, useFlowProviderStore } from '../../FlowProvider';
+import { useTranslation } from 'next-i18next';
+import { getLafAppDetail } from '@/web/support/laf/api';
+import MySelect from '@fastgpt/web/components/common/MySelect';
+import { getApiSchemaByUrl } from '@/web/core/plugin/api';
+import { str2OpenApiSchema } from '@fastgpt/global/core/plugin/httpPlugin/utils';
+import { useUserStore } from '@/web/support/user/useUserStore';
+import { useSystemStore } from '@/web/common/system/useSystemStore';
+import { ChevronRightIcon } from '@chakra-ui/icons';
+import { useQuery } from '@tanstack/react-query';
+import dynamic from 'next/dynamic';
+import { FlowNodeInputTypeEnum } from '@fastgpt/global/core/module/node/constant';
+import { useToast } from '@fastgpt/web/hooks/useToast';
+import Divider from '../modules/Divider';
+import RenderToolInput from '../render/RenderToolInput';
+import RenderInput from '../render/RenderInput';
+import RenderOutput from '../render/RenderOutput';
+import { getErrText } from '@fastgpt/global/common/error/utils';
+
+const LafAccountModal = dynamic(() => import('@/components/support/laf/LafAccountModal'));
+
+const NodeLaf = (props: NodeProps<FlowModuleItemType>) => {
+  const { t } = useTranslation();
+  const { toast } = useToast();
+  const { feConfigs } = useSystemStore();
+  const { data, selected } = props;
+  const { moduleId, inputs } = data;
+
+  const requestUrl = inputs.find((item) => item.key === ModuleInputKeyEnum.httpReqUrl);
+
+  const { userInfo } = useUserStore();
+
+  const token = userInfo?.team.lafAccount?.token;
+  const appid = userInfo?.team.lafAccount?.appid;
+
+  // not config laf
+  if (!token || !appid) {
+    return (
+      <NodeCard minW={'350px'} selected={selected} {...data}>
+        <ConfigLaf />
+      </NodeCard>
+    );
+  }
+
+  const { data: lafData, isLoading: isLoadingFunctions } = useQuery(
+    ['getLafFunctionList'],
+    async () => {
+      // load laf app detail
+      const appDetail = await getLafAppDetail(appid);
+
+      // load laf app functions
+      const schemaUrl = `https://${appDetail?.domain.domain}/_/api-docs?token=${appDetail?.openapi_token}`;
+
+      const schema = await getApiSchemaByUrl(schemaUrl);
+      const openApiSchema = await str2OpenApiSchema(JSON.stringify(schema));
+      const filterPostSchema = openApiSchema.pathData.filter((item) => item.method === 'post');
+
+      return {
+        lafApp: appDetail,
+        lafFunctions: filterPostSchema.map((item) => ({
+          ...item,
+          requestUrl: `https://${appDetail?.domain.domain}${item.path}`
+        }))
+      };
+    },
+    {
+      onError(err) {
+        toast({
+          status: 'error',
+          title: getErrText(err, '获取Laf函数列表失败')
+        });
+      }
+    }
+  );
+
+  const lafFunctionSelectList = useMemo(
+    () =>
+      lafData?.lafFunctions.map((item) => ({
+        label: item.description ? `${item.name} (${item.description})` : item.name,
+        value: item.requestUrl
+      })) || [],
+    [lafData?.lafFunctions]
+  );
+
+  const selectedFunction = useMemo(
+    () => lafFunctionSelectList.find((item) => item.value === requestUrl?.value)?.value,
+    [lafFunctionSelectList, requestUrl?.value]
+  );
+
+  const onSyncParams = useCallback(() => {
+    const lafFunction = lafData?.lafFunctions.find((item) => item.requestUrl === selectedFunction);
+
+    if (!lafFunction) return;
+
+    const bodyParams =
+      lafFunction?.request?.content?.['application/json']?.schema?.properties || {};
+
+    const requiredParams =
+      lafFunction?.request?.content?.['application/json']?.schema?.required || [];
+
+    const allParams = [
+      ...Object.keys(bodyParams).map((key) => ({
+        name: key,
+        desc: bodyParams[key].description,
+        required: requiredParams?.includes(key) || false,
+        value: `{{${key}}}`,
+        type: 'string'
+      }))
+    ].filter((item) => !inputs.find((input) => input.key === item.name));
+
+    // add params
+    allParams.forEach((param) => {
+      onChangeNode({
+        moduleId,
+        type: 'addInput',
+        key: param.name,
+        value: {
+          key: param.name,
+          valueType: ModuleIOValueTypeEnum.string,
+          label: param.name,
+          type: FlowNodeInputTypeEnum.target,
+          required: param.required,
+          description: param.desc || '',
+          toolDescription: param.desc || '未设置参数描述',
+          edit: true,
+          editField: {
+            key: true,
+            name: true,
+            description: true,
+            required: true,
+            dataType: true,
+            inputType: true,
+            isToolInput: true
+          },
+          connected: false
+        }
+      });
+    });
+
+    toast({
+      status: 'success',
+      title: t('common.Sync success')
+    });
+  }, [inputs, lafData?.lafFunctions, moduleId, selectedFunction, t, toast]);
+
+  return (
+    <NodeCard minW={'350px'} selected={selected} {...data}>
+      <Container>
+        {/* select function */}
+        <MySelect
+          isLoading={isLoadingFunctions}
+          list={lafFunctionSelectList}
+          placeholder={t('core.module.laf.Select laf function')}
+          onchange={(e) => {
+            onChangeNode({
+              moduleId,
+              type: 'updateInput',
+              key: ModuleInputKeyEnum.httpReqUrl,
+              value: {
+                ...requestUrl,
+                value: e
+              }
+            });
+          }}
+          value={selectedFunction}
+        />
+        {/* auto set params and go to edit */}
+        {!!selectedFunction && (
+          <Flex justifyContent={'flex-end'} mt={2} gap={2}>
+            {/* <Button variant={'whiteBase'} size={'sm'} onClick={onSyncParams}>
+              {t('core.module.Laf sync params')}
+            </Button> */}
+            <Button
+              variant={'grayBase'}
+              size={'sm'}
+              onClick={() => {
+                const lafFunction = lafData?.lafFunctions.find(
+                  (item) => item.requestUrl === selectedFunction
+                );
+
+                if (!lafFunction) return;
+                const url = `${feConfigs.lafEnv}/app/${lafData?.lafApp?.appid}/function${lafFunction?.path}`;
+                window.open(url, '_blank');
+              }}
+            >
+              {t('plugin.go to laf')}
+            </Button>
+          </Flex>
+        )}
+      </Container>
+      {!!selectedFunction && <RenderIO {...props} />}
+    </NodeCard>
+  );
+};
+export default React.memo(NodeLaf);
+
+const ConfigLaf = () => {
+  const { t } = useTranslation();
+  const { userInfo } = useUserStore();
+  const { feConfigs } = useSystemStore();
+  const {
+    isOpen: isOpenLafConfig,
+    onOpen: onOpenLafConfig,
+    onClose: onCloseLafConfig
+  } = useDisclosure();
+
+  return !!feConfigs?.lafEnv ? (
+    <Center minH={150}>
+      <Button onClick={onOpenLafConfig} variant={'whitePrimary'}>
+        {t('plugin.Please bind laf accout first')} <ChevronRightIcon />
+      </Button>
+
+      {isOpenLafConfig && feConfigs?.lafEnv && (
+        <LafAccountModal defaultData={userInfo?.team.lafAccount} onClose={onCloseLafConfig} />
+      )}
+    </Center>
+  ) : (
+    <Box>系统未配置Laf环境</Box>
+  );
+};
+
+const RenderIO = ({ data, selected }: NodeProps<FlowModuleItemType>) => {
+  const { t } = useTranslation();
+  const { moduleId, inputs, outputs } = data;
+  const { splitToolInputs, hasToolNode } = useFlowProviderStore();
+  const { commonInputs, toolInputs } = splitToolInputs(inputs, moduleId);
+
+  return (
+    <>
+      {hasToolNode && (
+        <>
+          <Divider text={t('core.module.tool.Tool input')} />
+          <Container>
+            <RenderToolInput moduleId={moduleId} inputs={toolInputs} canEdit />
+          </Container>
+        </>
+      )}
+      <>
+        <Divider text={t('common.Input')} />
+        <Container>
+          <Box mb={3}>自定义Body参数</Box>
+          <RenderInput moduleId={moduleId} flowInputList={commonInputs} />
+        </Container>
+      </>
+      <>
+        <Divider text={t('common.Output')} />
+        <Container>
+          <RenderOutput moduleId={moduleId} flowOutputList={outputs} />
+        </Container>
+      </>
+    </>
+  );
+};

projects/app/src/components/core/module/Flow/components/render/NodeCard.tsx

@@ -164,7 +164,7 @@ const NodeCard = (props: Props) => {
             top={'-20px'}
             right={0}
             transform={'translateX(90%)'}
-            pl={'17px'}
+            pl={'20px'}
             pr={'10px'}
             pb={'20px'}
             pt={'20px'}

projects/app/src/components/core/module/Flow/components/render/RenderToolInput/constants.ts

@@ -1,11 +1,25 @@
 import { FlowNodeInputItemType } from '@fastgpt/global/core/module/node/type';
 import { FlowNodeInputTypeEnum } from '@fastgpt/global/core/module/node/constant';
+import { ModuleIOValueTypeEnum } from '@fastgpt/global/core/module/constants';
 
 export const defaultEditFormData: FlowNodeInputItemType = {
   valueType: 'string',
-  type: FlowNodeInputTypeEnum.hidden,
+  type: FlowNodeInputTypeEnum.target,
   key: '',
   label: '',
   toolDescription: '',
-  required: true
+  required: true,
+  edit: true,
+  editField: {
+    key: true,
+    description: true,
+    dataType: true
+  },
+  defaultEditField: {
+    label: '',
+    key: '',
+    description: '',
+    inputType: FlowNodeInputTypeEnum.target,
+    valueType: ModuleIOValueTypeEnum.string
+  }
 };