Snip test (#3204 )

* fix: index * fix: snapshot error; perf: snapshot diff compare * perf: init simple edit history
refactor: snapshot store to diff (#3155 )
2024-11-21 16:26:43 +08:00 · 2024-11-21 13:12:42 +08:00 · 2024-11-21 12:01:55 +08:00 · 2024-11-19 10:19:21 +08:00 · 2024-11-18 20:57:03 +08:00 · 2024-11-18 18:34:08 +08:00
65 changed files with 982 additions and 2508 deletions
--- a/docSite/content/zh-cn/docs/development/configuration.md
+++ b/docSite/content/zh-cn/docs/development/configuration.md
@@ -23,6 +23,7 @@ weight: 708
  "systemEnv": {
    "vectorMaxProcess": 15,
    "qaMaxProcess": 15,
+    "tokenWorkers": 50, // Token 计算线程保持数，会持续占用内存，不能设置太大。
    "pgHNSWEfSearch": 100 // 向量搜索参数。越大，搜索越精确，但是速度越慢。设置为100，有99%+精度。
  },
  "llmModels": [
--- a/docSite/content/zh-cn/docs/development/openapi/chat.md
+++ b/docSite/content/zh-cn/docs/development/openapi/chat.md
@@ -35,9 +35,10 @@ curl --location --request POST 'http://localhost:3000/api/v1/chat/completions' \
 --header 'Authorization: Bearer fastgpt-xxxxxx' \
 --header 'Content-Type: application/json' \
 --data-raw '{
-    "chatId": "abcd",
+    "chatId": "my_chatId",
    "stream": false,
    "detail": false,
+    "responseChatItemId": "my_responseChatItemId",
    "variables": {
        "uid": "asdfadsfasfd2323",
        "name": "张三"
@@ -104,6 +105,7 @@ curl --location --request POST 'http://localhost:3000/api/v1/chat/completions' \
  - 为 `undefined` 时（不传入），不使用 FastGpt 提供的上下文功能，完全通过传入的 messages 构建上下文。 不会将你的记录存储到数据库中，你也无法在记录汇总中查阅到。
  - 为`非空字符串`时，意味着使用 chatId 进行对话，自动从 FastGpt 数据库取历史记录，并使用 messages 数组最后一个内容作为用户问题。请自行确保 chatId 唯一，长度小于250，通常可以是自己系统的对话框ID。
 - messages: 结构与 [GPT接口](https://platform.openai.com/docs/api-reference/chat/object) chat模式一致。
+- responseChatItemId: string | undefined 。如果传入，则会将该值作为本次对话的响应消息的 ID，FastGPT 会自动将该 ID 存入数据库。请确保，在当前`chatId`下，`responseChatItemId`是唯一的。
 - detail: 是否返回中间值（模块状态，响应的完整结果等），`stream模式`下会通过`event`进行区分，`非stream模式`结果保存在`responseData`中。
 - variables: 模块变量，一个对象，会替换模块中，输入框内容里的`{{key}}`
 {{% /alert %}}
--- a/docSite/content/zh-cn/docs/development/upgrading/4813.md
+++ b/docSite/content/zh-cn/docs/development/upgrading/4813.md
@@ -1,5 +1,5 @@
 ---
-title: 'V4.8.13(进行中)'
+title: 'V4.8.13'
 description: 'FastGPT V4.8.13 更新说明'
 icon: 'upgrade'
 draft: false
@@ -13,13 +13,17 @@ weight: 811

 ### 2. 修改镜像

- 更新 FastGPT 镜像 tag: v4.8.13-beta
- 更新 FastGPT 商业版镜像 tag: v4.8.13-beta （fastgpt-pro镜像）
+- 更新 FastGPT 镜像 tag: v4.8.13-fix
+- 更新 FastGPT 商业版镜像 tag: v4.8.13-fix （fastgpt-pro镜像）
 - Sandbox 镜像，可以不更新

-### 3. 调整文件上传编排
+### 3. 添加环境变量

-虽然依然兼容旧版的文件上传编排，但是未来两个版本内将会去除兼容代码，请尽快调整编排，以适应最新的文件上传逻辑。尤其是嵌套应用的文件传递，未来将不会自动传递，必须手动指定传递的文件。
+- 给 fastgpt 和 fastgpt-pro 镜像添加环境变量：`FE_DOMAIN=http://xx.com`，值为 fastgpt 前端访问地址，注意后面不要加`/`。可以自动补齐相对文件地址的前缀。
+
+### 4. 调整文件上传编排
+
+虽然依然兼容旧版的文件上传编排，但是未来两个版本内将会去除兼容代码，请尽快调整编排，以适应最新的文件上传逻辑。尤其是嵌套应用的文件传递，未来将不会自动传递，必须手动指定传递的文件。具体内容可参考: [文件上传变更](/docs/guide/course/fileinput/#4813%E7%89%88%E6%9C%AC%E8%B5%B7%E5%85%B3%E4%BA%8E%E6%96%87%E4%BB%B6%E4%B8%8A%E4%BC%A0%E7%9A%84%E6%9B%B4%E6%96%B0)

 ## 更新说明

@@ -39,7 +43,9 @@ weight: 811
 14. 优化 - Markdown 组件自动空格，避免分割 url 中的中文。  
 15. 优化 - 工作流上下文拆分，性能优化。 
 16. 优化 - 语音播报，不支持 mediaSource 的浏览器可等待完全生成语音后输出。 
-17. 优化 - 对话引导 csv 读取，自动识别编码。
-18. 修复 - Dockerfile pnpm install 支持代理。 
-19. 修复 - BI 图表生成无法写入文件。同时优化其解析，支持数字类型数组。
-20. 修复 - 分享链接首次加载时，标题显示不正确。 
+17. 优化 - 对话引导 csv 读取，自动识别编码
+18. 优化 - csv 导入问题引导可能乱码
+19. 修复 - Dockerfile pnpm install 支持代理。。
+20. 修复 - Dockerfile pnpm install 支持代理。 
+21. 修复 - BI 图表生成无法写入文件。同时优化其解析，支持数字类型数组。
+22. 修复 - 分享链接首次加载时，标题显示不正确。 
--- a/docSite/content/zh-cn/docs/development/upgrading/4814.md
+++ b/docSite/content/zh-cn/docs/development/upgrading/4814.md
@@ -0,0 +1,19 @@
+---
+title: 'V4.8.14(进行中)'
+description: 'FastGPT V4.8.14 更新说明'
+icon: 'upgrade'
+draft: false
+toc: true
+weight: 810
+---
+
+## 更新预告
+
+1. 
+2. 新增 - 工作流支持进入聊天框/点击开始对话后，自动触发一轮对话。
+3. 新增 - 重写 chatContext，对话测试也会有日志，并且刷新后不会丢失对话。
+4. 新增 - 分享链接支持配置是否允许查看原文。
+5. 优化 - 工作流 ui 细节。
+6. 优化 - 应用编辑记录采用 diff 存储，避免浏览器溢出。
+7. 修复 - 分块策略，四级标题会被丢失。 同时新增了五级标题的支持。
+8. 修复 - MongoDB 知识库集合唯一索引。
--- a/docSite/content/zh-cn/docs/use-cases/external-integration/official_account.md
+++ b/docSite/content/zh-cn/docs/use-cases/external-integration/official_account.md
@@ -39,44 +39,45 @@ weight: 506
 海外版用户（cloud.tryfastgpt.ai)可以填写下面的 IP 白名单：

 ```
-34.87.20.17
-35.247.161.35
-34.87.51.146
-34.87.110.152	
-35.247.163.68
-34.126.163.205
-34.87.20.189
-34.87.102.86
 35.240.227.100
-35.198.192.104
-34.143.149.171
-34.87.152.33
 34.124.237.188
-35.197.149.75
-34.87.44.74
-34.124.189.116
-34.87.79.202
-34.87.173.252
 34.143.240.160
-34.87.180.104
+34.87.51.146
+34.87.79.202
+35.247.163.68
+34.87.102.86
+35.198.192.104
+34.126.163.205
+34.124.189.116
+34.143.149.171
+34.87.173.252
 34.142.157.52
+34.87.180.104
+34.87.20.189
+34.87.110.152
+34.87.44.74
+34.87.152.33
+35.197.149.75
+35.247.161.35
 ```

 国内版用户（fastgpt.cn)可以填写下面的 IP 白名单：

 ```
-47.97.59.172
-121.43.108.48
-121.41.75.88
+47.97.1.240
+121.43.105.217
 121.41.178.7
 121.40.65.187
-121.196.235.183
-120.55.195.90
-120.55.193.112
-120.26.229.115
-112.124.41.79
+47.97.59.172
 101.37.205.32
+120.55.195.90
+120.26.229.115
+120.55.193.112
 47.98.190.173
+112.124.41.79
+121.196.235.183
+121.41.75.88
+121.43.108.48
 ```

 ## 4. 获取AES Key，选择加密方式
--- a/files/docker/docker-compose-milvus.yml
+++ b/files/docker/docker-compose-milvus.yml
@@ -139,8 +139,6 @@ services:
      - OPENAI_BASE_URL=http://oneapi:3000/v1
      # AI模型的API Key。（这里默认填写了OneAPI的快速默认key，测试通后，务必及时修改）
      - CHAT_API_KEY=sk-fastgpt
-      # 是否将图片转成 base64 传递给模型，本地开发和内网环境使用共有模型时候需要设置为 true
-      - MULTIPLE_DATA_TO_BASE64=false
      # 数据库最大连接数
      - DB_MAX_LINK=30
      # 登录凭证密钥
--- a/files/docker/docker-compose-pgvector.yml
+++ b/files/docker/docker-compose-pgvector.yml
@@ -97,8 +97,6 @@ services:
      - OPENAI_BASE_URL=http://oneapi:3000/v1
      # AI模型的API Key。（这里默认填写了OneAPI的快速默认key，测试通后，务必及时修改）
      - CHAT_API_KEY=sk-fastgpt
-      # 是否将图片转成 base64 传递给模型，本地开发和内网环境使用共有模型时候需要设置为 true
-      - MULTIPLE_DATA_TO_BASE64=false
      # 数据库最大连接数
      - DB_MAX_LINK=30
      # 登录凭证密钥
--- a/files/docker/docker-compose-zilliz.yml
+++ b/files/docker/docker-compose-zilliz.yml
@@ -77,8 +77,6 @@ services:
      - OPENAI_BASE_URL=http://oneapi:3000/v1
      # AI模型的API Key。（这里默认填写了OneAPI的快速默认key，测试通后，务必及时修改）
      - CHAT_API_KEY=sk-fastgpt
-      # 是否将图片转成 base64 传递给模型，本地开发和内网环境使用共有模型时候需要设置为 true
-      - MULTIPLE_DATA_TO_BASE64=false
      # 数据库最大连接数
      - DB_MAX_LINK=30
      # 登录凭证密钥
--- a/packages/global/common/string/markdown.ts
+++ b/packages/global/common/string/markdown.ts
@@ -1,5 +1,6 @@
 import { batchRun } from '../fn/utils';
-import { simpleText } from './tools';
+import { getNanoid, simpleText } from './tools';
+import type { ImageType } from '../../../service/worker/readFile/type';

 /* Delete redundant text in markdown */
 export const simpleMarkdownText = (rawText: string) => {
@@ -92,3 +93,25 @@ export const markdownProcess = async ({

  return simpleMarkdownText(imageProcess);
 };
+
+export const matchMdImgTextAndUpload = (text: string) => {
+  const base64Regex = /"(data:image\/[^;]+;base64[^"]+)"/g;
+  const imageList: ImageType[] = [];
+  const images = Array.from(text.match(base64Regex) || []);
+  for (const image of images) {
+    const uuid = `IMAGE_${getNanoid(12)}_IMAGE`;
+    const mime = image.split(';')[0].split(':')[1];
+    const base64 = image.split(',')[1];
+    text = text.replace(image, uuid);
+    imageList.push({
+      uuid,
+      base64,
+      mime
+    });
+  }
+
+  return {
+    text,
+    imageList
+  };
+};
--- a/packages/global/common/string/textSplitter.ts
+++ b/packages/global/common/string/textSplitter.ts
@@ -99,7 +99,7 @@ ${mdSplitString}
  5. 标点分割：重叠
 */
 const commonSplit = (props: SplitProps): SplitResponse => {
-  let { text = '', chunkLen, overlapRatio = 0.2, customReg = [] } = props;
+  let { text = '', chunkLen, overlapRatio = 0.15, customReg = [] } = props;

  const splitMarker = 'SPLIT_HERE_SPLIT_HERE';
  const codeBlockMarker = 'CODE_BLOCK_LINE_MARKER';
@@ -113,6 +113,8 @@ const commonSplit = (props: SplitProps): SplitResponse => {
  text = text.replace(/(\r?\n|\r){3,}/g, '\n\n\n');

  // The larger maxLen is, the next sentence is less likely to trigger splitting
+  const markdownIndex = 4;
+  const forbidOverlapIndex = 8;
  const stepReges: { reg: RegExp; maxLen: number }[] = [
    ...customReg.map((text) => ({
      reg: new RegExp(`(${replaceRegChars(text)})`, 'g'),
@@ -122,9 +124,11 @@ const commonSplit = (props: SplitProps): SplitResponse => {
    { reg: /^(##\s[^\n]+\n)/gm, maxLen: chunkLen * 1.4 },
    { reg: /^(###\s[^\n]+\n)/gm, maxLen: chunkLen * 1.6 },
    { reg: /^(####\s[^\n]+\n)/gm, maxLen: chunkLen * 1.8 },
+    { reg: /^(#####\s[^\n]+\n)/gm, maxLen: chunkLen * 1.8 },

    { reg: /([\n]([`~]))/g, maxLen: chunkLen * 4 }, // code block
-    { reg: /([\n](?!\s*[\*\-|>0-9]))/g, maxLen: chunkLen * 2 }, // 增大块，尽可能保证它是一个完整的段落。 (?![\*\-|>`0-9]): markdown special char
+    { reg: /([\n](?=\s*[0-9]+\.))/g, maxLen: chunkLen * 2 }, // 增大块，尽可能保证它是一个完整的段落。 (?![\*\-|>`0-9]): markdown special char
+    { reg: /(\n{2,})/g, maxLen: chunkLen * 1.6 },
    { reg: /([\n])/g, maxLen: chunkLen * 1.2 },
    // ------ There's no overlap on the top
    { reg: /([。]|([a-zA-Z])\.\s)/g, maxLen: chunkLen * 1.2 },
@@ -136,8 +140,9 @@ const commonSplit = (props: SplitProps): SplitResponse => {

  const customRegLen = customReg.length;
  const checkIsCustomStep = (step: number) => step < customRegLen;
-  const checkIsMarkdownSplit = (step: number) => step >= customRegLen && step <= 3 + customRegLen;
-  const checkForbidOverlap = (step: number) => step <= 6 + customRegLen;
+  const checkIsMarkdownSplit = (step: number) => step >= customRegLen && step <= markdownIndex;
+  +customReg.length;
+  const checkForbidOverlap = (step: number) => step <= forbidOverlapIndex + customReg.length;

  // if use markdown title split, Separate record title
  const getSplitTexts = ({ text, step }: { text: string; step: number }) => {
@@ -231,7 +236,7 @@ const commonSplit = (props: SplitProps): SplitResponse => {
      // use slice-chunkLen to split text
      const chunks: string[] = [];
      for (let i = 0; i < text.length; i += chunkLen - overlapLen) {
-        chunks.push(`${parentTitle}${text.slice(i, i + chunkLen)}`);
+        chunks.push(text.slice(i, i + chunkLen));
      }
      return chunks;
    }
@@ -241,7 +246,6 @@ const commonSplit = (props: SplitProps): SplitResponse => {

    const maxLen = splitTexts.length > 1 ? stepReges[step].maxLen : chunkLen;
    const minChunkLen = chunkLen * 0.7;
-    // console.log(splitTexts, stepReges[step].reg);

    const chunks: string[] = [];
    for (let i = 0; i < splitTexts.length; i++) {
@@ -249,12 +253,34 @@ const commonSplit = (props: SplitProps): SplitResponse => {

      const lastTextLen = lastText.length;
      const currentText = item.text;
-      const currentTextLen = currentText.length;
      const newText = lastText + currentText;
-      const newTextLen = lastTextLen + currentTextLen;
+      const newTextLen = newText.length;
+
+      // Markdown 模式下，会强制向下拆分最小块，并再最后一个标题时候，给小块都补充上所有标题（包含父级标题）
+      if (isMarkdownStep) {
+        // split new Text, split chunks must will greater 1 (small lastText)
+        const innerChunks = splitTextRecursively({
+          text: newText,
+          step: step + 1,
+          lastText: '',
+          parentTitle: parentTitle + item.title
+        });
+
+        const lastChunk = innerChunks[innerChunks.length - 1];
+        if (!lastChunk) continue;
+
+        chunks.push(
+          ...innerChunks.map(
+            (chunk) =>
+              step === markdownIndex + customRegLen ? `${parentTitle}${item.title}${chunk}` : chunk // 合并进 Markdown 分块时，需要补标题
+          )
+        );
+
+        continue;
+      }

      // newText is too large(now, The lastText must be smaller than chunkLen)
-      if (newTextLen > maxLen || isMarkdownStep) {
+      if (newTextLen > maxLen) {
        // lastText greater minChunkLen, direct push it to chunks, not add to next chunk. (large lastText)
        if (lastTextLen > minChunkLen) {
          chunks.push(lastText);
@@ -278,15 +304,6 @@ const commonSplit = (props: SplitProps): SplitResponse => {

        if (!lastChunk) continue;

-        if (forbidConcat) {
-          chunks.push(
-            ...innerChunks.map(
-              (chunk) => (step === 3 + customRegLen ? `${parentTitle}${chunk}` : chunk) // 合并进 Markdown 分块时，需要补标题
-            )
-          );
-          continue;
-        }
-
        // last chunk is too small, concat it to lastText(next chunk start)
        if (lastChunk.length < minChunkLen) {
          chunks.push(...innerChunks.slice(0, -1));
@@ -304,11 +321,11 @@ const commonSplit = (props: SplitProps): SplitResponse => {
        continue;
      }

-      // new text is small
+      // New text is small

      // Not overlap
      if (forbidConcat) {
-        chunks.push(`${parentTitle}${item.title}${item.text}`);
+        chunks.push(item.text);
        continue;
      }

--- a/packages/global/common/system/types/index.d.ts
+++ b/packages/global/common/system/types/index.d.ts
@@ -56,6 +56,7 @@ export type FastGPTFeConfigsType = {
    github?: string;
    google?: string;
    wechat?: string;
+    microsoft?: string;
  };
  limit?: {
    exportDatasetLimitMinutes?: number;
--- a/packages/global/support/user/constant.ts
+++ b/packages/global/support/user/constant.ts
@@ -14,5 +14,6 @@ export const userStatusMap = {
 export enum OAuthEnum {
  github = 'github',
  google = 'google',
-  wechat = 'wechat'
+  wechat = 'wechat',
+  microsoft = 'microsoft'
 }
--- a/packages/plugins/register.ts
+++ b/packages/plugins/register.ts
@@ -5,18 +5,7 @@ import { cloneDeep } from 'lodash';
 import { WorkerNameEnum, runWorker } from '@fastgpt/service/worker/utils';

 // Run in main thread
-const staticPluginList = [
-  'getTime',
-  'fetchUrl',
-  'Doc2X',
-  'Doc2X/URLPDF2text',
-  'Doc2X/URLImg2text',
-  `Doc2X/FilePDF2text`,
-  `Doc2X/FileImg2text`,
-  'feishu',
-  'google',
-  'bing'
-];
+const staticPluginList = ['getTime', 'fetchUrl', 'feishu', 'google', 'bing'];
 // Run in worker thread (Have npm packages)
 const packagePluginList = [
  'mathExprVal',
@@ -28,7 +17,9 @@ const packagePluginList = [
  'drawing',
  'drawing/baseChart',
  'wiki',
-  'databaseConnection'
+  'databaseConnection',
+  'Doc2X',
+  'Doc2X/PDF2text'
 ];

 export const list = [...staticPluginList, ...packagePluginList];
@@ -55,6 +46,8 @@ export const getCommunityPlugins = () => {
 };

 export const getSystemPluginTemplates = () => {
+  if (!global.systemPlugins) return [];
+
  const oldPlugins = global.communityPlugins ?? [];
  return [...oldPlugins, ...cloneDeep(global.systemPlugins)];
 };
@@ -96,7 +89,3 @@ export const getCommunityCb = async () => {
    {}
  );
 };
-
-export const getSystemPluginCb = async () => {
-  return global.systemPluginCb;
-};
--- a/packages/plugins/src/Doc2X/FileImg2text/index.ts
+++ b/packages/plugins/src/Doc2X/FileImg2text/index.ts
@@ -1,172 +0,0 @@
-import { delay } from '@fastgpt/global/common/system/utils';
-import { addLog } from '@fastgpt/service/common/system/log';
-
-type Props = {
-  apikey: string;
-  files: Array<string>;
-  img_correction: boolean;
-  formula: boolean;
-};
-
-type Response = Promise<{
-  result: string;
-  failreason: string;
-  success: boolean;
-}>;
-
-const main = async ({ apikey, files, img_correction, formula }: Props): Response => {
-  // Check the apikey
-  if (!apikey) {
-    return {
-      result: '',
-      failreason: `API key is required`,
-      success: false
-    };
-  }
-
-  let real_api_key = apikey;
-  if (!apikey.startsWith('sk-')) {
-    const response = await fetch('https://api.doc2x.noedgeai.com/api/token/refresh', {
-      method: 'POST',
-      headers: {
-        Authorization: `Bearer ${apikey}`
-      }
-    });
-    if (response.status !== 200) {
-      return {
-        result: '',
-        failreason: `Get token failed: ${await response.text()}`,
-        success: false
-      };
-    }
-    const data = await response.json();
-    real_api_key = data.data.token;
-  }
-
-  let final_result = '';
-  let fail_reason = '';
-  let flag = false;
-  //Process each file one by one
-  for await (const url of files) {
-    // Fetch the image and check its content type
-    const imageResponse = await fetch(url);
-    if (!imageResponse.ok) {
-      fail_reason += `\n---\nFile:${url} \n<Content>\nFailed to fetch image from URL\n</Content>\n`;
-      flag = true;
-      continue;
-    }
-
-    const contentType = imageResponse.headers.get('content-type');
-    const fileName = url.match(/read\?filename=([^&]+)/)?.[1] || 'unknown.png';
-    if (!contentType || !contentType.startsWith('image/')) {
-      fail_reason += `\n---\nFile:${url} \n<Content>\nThe provided URL does not point to an image: ${contentType}\n</Content>\n`;
-      flag = true;
-      continue;
-    }
-
-    const blob = await imageResponse.blob();
-    const formData = new FormData();
-    formData.append('file', blob, fileName);
-    formData.append('img_correction', img_correction ? '1' : '0');
-    formData.append('equation', formula ? '1' : '0');
-
-    let upload_url = 'https://api.doc2x.noedgeai.com/api/platform/async/img';
-    if (real_api_key.startsWith('sk-')) {
-      upload_url = 'https://api.doc2x.noedgeai.com/api/v1/async/img';
-    }
-
-    let uuid;
-    let upload_flag = true;
-    const uploadAttempts = [1, 2, 3];
-    for await (const attempt of uploadAttempts) {
-      const upload_response = await fetch(upload_url, {
-        method: 'POST',
-        headers: {
-          Authorization: `Bearer ${real_api_key}`
-        },
-        body: formData
-      });
-
-      if (!upload_response.ok) {
-        // Rate limit, wait for 10s and retry at most 3 times
-        if (upload_response.status === 429 && attempt < 3) {
-          await delay(10000);
-          continue;
-        }
-        fail_reason += `\n---\nFile:${fileName}\n<Content>\nFailed to upload file: ${await upload_response.text()}\n</Content>\n`;
-        flag = true;
-        upload_flag = false;
-        break;
-      }
-      if (!upload_flag) {
-        continue;
-      }
-
-      const upload_data = await upload_response.json();
-      uuid = upload_data.data.uuid;
-      break;
-    }
-
-    // Get the result by uuid
-    let result_url = 'https://api.doc2x.noedgeai.com/api/platform/async/status?uuid=' + uuid;
-    if (real_api_key.startsWith('sk-')) {
-      result_url = 'https://api.doc2x.noedgeai.com/api/v1/async/status?uuid=' + uuid;
-    }
-
-    let required_flag = true;
-    const maxAttempts = 100;
-    // Wait for the result, at most 100s
-    for await (const _ of Array(maxAttempts).keys()) {
-      const result_response = await fetch(result_url, {
-        headers: {
-          Authorization: `Bearer ${real_api_key}`
-        }
-      });
-      if (!result_response.ok) {
-        fail_reason += `\n---\nFile:${fileName}\n<Content>\nFailed to get result: ${await result_response.text()}\n</Content>\n`;
-        flag = true;
-        required_flag = false;
-        break;
-      }
-      const result_data = await result_response.json();
-      if (['ready', 'processing'].includes(result_data.data.status)) {
-        await delay(1000);
-      } else if (result_data.data.status === 'pages limit exceeded') {
-        fail_reason += `\n---\nFile:${fileName}\n<Content>\nFailed to get result: pages limit exceeded\n</Content>\n`;
-        flag = true;
-        required_flag = false;
-        break;
-      } else if (result_data.data.status === 'success') {
-        let result;
-        try {
-          result = result_data.data.result.pages[0].md;
-          result = result.replace(/\\[\(\)]/g, '$').replace(/\\[\[\]]/g, '$$');
-        } catch {
-          // no pages
-          final_result += `\n---\nFile:${fileName}\n<Content>\n \n</Content>\n`;
-          required_flag = false;
-        }
-        final_result += `\n---\nFile:${fileName}\n<Content>\n${result}\n</Content>\n`;
-        required_flag = false;
-        break;
-      } else {
-        fail_reason += `\n---\nFile:${fileName}\n<Content>\nFailed to get result: ${result_data.data.status}\n</Content>\n`;
-        flag = true;
-        required_flag = false;
-        break;
-      }
-    }
-    if (required_flag) {
-      fail_reason += `\n---\nFile:${fileName}\n<Content>\nTimeout waiting for result\n</Content>\n`;
-      flag = true;
-    }
-  }
-
-  return {
-    result: final_result,
-    failreason: fail_reason,
-    success: !flag
-  };
-};
-
-export default main;
--- a/packages/plugins/src/Doc2X/FileImg2text/template.json
+++ b/packages/plugins/src/Doc2X/FileImg2text/template.json
@@ -1,500 +0,0 @@
-{
-  "author": "Menghuan1918",
-  "version": "488",
-  "name": "Doc2X 图像(文件)识别",
-  "avatar": "plugins/doc2x",
-  "intro": "将上传的图片文件发送至Doc2X进行解析，返回带LaTeX公式的markdown格式的文本",
-  "courseUrl": "https://fael3z0zfze.feishu.cn/wiki/Rkc5witXWiJoi5kORd2cofh6nDg?fromScene=spaceOverview",
-  "showStatus": true,
-  "weight": 10,
-
-  "isTool": true,
-  "templateType": "tools",
-
-  "workflow": {
-    "nodes": [
-      {
-        "nodeId": "pluginConfig",
-        "name": "common:core.module.template.system_config",
-        "intro": "",
-        "avatar": "core/workflow/template/systemConfig",
-        "flowNodeType": "pluginConfig",
-        "position": {
-          "x": -90.53591960393504,
-          "y": -17.580286776561252
-        },
-        "version": "4811",
-        "inputs": [],
-        "outputs": []
-      },
-      {
-        "nodeId": "pluginInput",
-        "name": "插件开始",
-        "intro": "可以配置插件需要哪些输入，利用这些输入来运行插件",
-        "avatar": "core/workflow/template/workflowStart",
-        "flowNodeType": "pluginInput",
-        "showStatus": false,
-        "position": {
-          "x": 368.6800424053505,
-          "y": -17.580286776561252
-        },
-        "version": "481",
-        "inputs": [
-          {
-            "renderTypeList": ["input"],
-            "selectedTypeIndex": 0,
-            "valueType": "string",
-            "canEdit": true,
-            "key": "apikey",
-            "label": "apikey",
-            "description": "Doc2X的验证密匙，对于个人用户可以从Doc2X官网 - 个人信息 - 身份令牌获得",
-            "required": true,
-            "toolDescription": "",
-            "defaultValue": ""
-          },
-          {
-            "renderTypeList": ["reference"],
-            "selectedTypeIndex": 0,
-            "valueType": "arrayString",
-            "canEdit": true,
-            "key": "files",
-            "label": "files",
-            "description": "待处理图片文件",
-            "required": true,
-            "toolDescription": "待处理图片文件"
-          },
-          {
-            "renderTypeList": ["switch"],
-            "selectedTypeIndex": 0,
-            "valueType": "boolean",
-            "canEdit": true,
-            "key": "img_correction",
-            "label": "img_correction",
-            "description": "是否启用图形矫正功能",
-            "required": true,
-            "toolDescription": "",
-            "defaultValue": false
-          },
-          {
-            "renderTypeList": ["switch"],
-            "selectedTypeIndex": 0,
-            "valueType": "boolean",
-            "canEdit": true,
-            "key": "formula",
-            "label": "formula",
-            "description": "是否开启纯公式识别(仅适用于图片内容仅有公式时)",
-            "required": true,
-            "toolDescription": "",
-            "defaultValue": false
-          }
-        ],
-        "outputs": [
-          {
-            "id": "apikey",
-            "valueType": "string",
-            "key": "apikey",
-            "label": "apikey",
-            "type": "hidden"
-          },
-          {
-            "id": "url",
-            "valueType": "arrayString",
-            "key": "files",
-            "label": "files",
-            "type": "hidden"
-          },
-          {
-            "id": "img_correction",
-            "valueType": "boolean",
-            "key": "img_correction",
-            "label": "img_correction",
-            "type": "hidden"
-          },
-          {
-            "id": "formula",
-            "valueType": "boolean",
-            "key": "formula",
-            "label": "formula",
-            "type": "hidden"
-          }
-        ]
-      },
-      {
-        "nodeId": "pluginOutput",
-        "name": "插件输出",
-        "intro": "自定义配置外部输出，使用插件时，仅暴露自定义配置的输出",
-        "avatar": "core/workflow/template/pluginOutput",
-        "flowNodeType": "pluginOutput",
-        "showStatus": false,
-        "position": {
-          "x": 1796.2235867744578,
-          "y": 6.419713223438748
-        },
-        "version": "481",
-        "inputs": [
-          {
-            "renderTypeList": ["reference"],
-            "valueType": "string",
-            "canEdit": true,
-            "key": "result",
-            "label": "result",
-            "description": "处理结果(或者是报错信息)",
-            "value": ["zHG5jJBkXmjB", "xWQuEf50F3mr"]
-          },
-          {
-            "renderTypeList": ["reference"],
-            "valueType": "string",
-            "canEdit": true,
-            "key": "failreason",
-            "label": "failreason",
-            "description": "文件处理失败原因，由文件名以及报错组成，多个文件之间由横线分隔开",
-            "value": ["zHG5jJBkXmjB", "jbv4nVZvmFXm"]
-          },
-          {
-            "renderTypeList": ["reference"],
-            "valueType": "boolean",
-            "canEdit": true,
-            "key": "success",
-            "label": "success",
-            "description": "是否全部文件都处理成功，如有没有处理成功的文件，失败原因将会输出在failreason中",
-            "value": ["zHG5jJBkXmjB", "k46cjNulVk5Y"]
-          }
-        ],
-        "outputs": []
-      },
-      {
-        "nodeId": "zHG5jJBkXmjB",
-        "name": "HTTP 请求",
-        "intro": "可以发出一个 HTTP 请求，实现更为复杂的操作（联网搜索、数据库查询等）",
-        "avatar": "core/workflow/template/httpRequest",
-        "flowNodeType": "httpRequest468",
-        "showStatus": true,
-        "position": {
-          "x": 1081.967607938733,
-          "y": -426.08028677656125
-        },
-        "version": "481",
-        "inputs": [
-          {
-            "key": "system_addInputParam",
-            "renderTypeList": ["addInputParam"],
-            "valueType": "dynamic",
-            "label": "",
-            "required": false,
-            "description": "common:core.module.input.description.HTTP Dynamic Input",
-            "customInputConfig": {
-              "selectValueTypeList": [
-                "string",
-                "number",
-                "boolean",
-                "object",
-                "arrayString",
-                "arrayNumber",
-                "arrayBoolean",
-                "arrayObject",
-                "arrayAny",
-                "any",
-                "chatHistory",
-                "datasetQuote",
-                "dynamic",
-                "selectApp",
-                "selectDataset"
-              ],
-              "showDescription": false,
-              "showDefaultValue": true
-            },
-            "debugLabel": "",
-            "toolDescription": ""
-          },
-          {
-            "key": "system_httpMethod",
-            "renderTypeList": ["custom"],
-            "valueType": "string",
-            "label": "",
-            "value": "POST",
-            "required": true,
-            "debugLabel": "",
-            "toolDescription": ""
-          },
-          {
-            "key": "system_httpTimeout",
-            "renderTypeList": ["custom"],
-            "valueType": "number",
-            "label": "",
-            "value": 30,
-            "min": 5,
-            "max": 600,
-            "required": true,
-            "debugLabel": "",
-            "toolDescription": ""
-          },
-          {
-            "key": "system_httpReqUrl",
-            "renderTypeList": ["hidden"],
-            "valueType": "string",
-            "label": "",
-            "description": "common:core.module.input.description.Http Request Url",
-            "placeholder": "https://api.ai.com/getInventory",
-            "required": false,
-            "value": "Doc2X/FileImg2text",
-            "debugLabel": "",
-            "toolDescription": ""
-          },
-          {
-            "key": "system_httpHeader",
-            "renderTypeList": ["custom"],
-            "valueType": "any",
-            "value": [],
-            "label": "",
-            "description": "common:core.module.input.description.Http Request Header",
-            "placeholder": "common:core.module.input.description.Http Request Header",
-            "required": false,
-            "debugLabel": "",
-            "toolDescription": ""
-          },
-          {
-            "key": "system_httpParams",
-            "renderTypeList": ["hidden"],
-            "valueType": "any",
-            "value": [],
-            "label": "",
-            "required": false,
-            "debugLabel": "",
-            "toolDescription": ""
-          },
-          {
-            "key": "system_httpJsonBody",
-            "renderTypeList": ["hidden"],
-            "valueType": "any",
-            "value": "{\n  \"apikey\": \"{{apikey}}\",\n  \"files\": {{files}},\n  \"img_correction\": {{img_correction}},\n  \"formula\": {{formula}}\n}",
-            "label": "",
-            "required": false,
-            "debugLabel": "",
-            "toolDescription": ""
-          },
-          {
-            "key": "system_httpFormBody",
-            "renderTypeList": ["hidden"],
-            "valueType": "any",
-            "value": [],
-            "label": "",
-            "required": false,
-            "debugLabel": "",
-            "toolDescription": ""
-          },
-          {
-            "key": "system_httpContentType",
-            "renderTypeList": ["hidden"],
-            "valueType": "string",
-            "value": "json",
-            "label": "",
-            "required": false,
-            "debugLabel": "",
-            "toolDescription": ""
-          },
-          {
-            "renderTypeList": ["reference"],
-            "valueType": "string",
-            "canEdit": true,
-            "key": "apikey",
-            "label": "apikey",
-            "customInputConfig": {
-              "selectValueTypeList": [
-                "string",
-                "number",
-                "boolean",
-                "object",
-                "arrayString",
-                "arrayNumber",
-                "arrayBoolean",
-                "arrayObject",
-                "arrayAny",
-                "any",
-                "chatHistory",
-                "datasetQuote",
-                "dynamic",
-                "selectApp",
-                "selectDataset"
-              ],
-              "showDescription": false,
-              "showDefaultValue": true
-            },
-            "required": true,
-            "value": ["pluginInput", "apikey"]
-          },
-          {
-            "renderTypeList": ["reference"],
-            "valueType": "arrayString",
-            "canEdit": true,
-            "key": "files",
-            "label": "files",
-            "customInputConfig": {
-              "selectValueTypeList": [
-                "string",
-                "number",
-                "boolean",
-                "object",
-                "arrayString",
-                "arrayNumber",
-                "arrayBoolean",
-                "arrayObject",
-                "arrayAny",
-                "any",
-                "chatHistory",
-                "datasetQuote",
-                "dynamic",
-                "selectApp",
-                "selectDataset"
-              ],
-              "showDescription": false,
-              "showDefaultValue": true
-            },
-            "required": true,
-            "value": ["pluginInput", "url"]
-          },
-          {
-            "renderTypeList": ["reference"],
-            "valueType": "boolean",
-            "canEdit": true,
-            "key": "img_correction",
-            "label": "img_correction",
-            "customInputConfig": {
-              "selectValueTypeList": [
-                "string",
-                "number",
-                "boolean",
-                "object",
-                "arrayString",
-                "arrayNumber",
-                "arrayBoolean",
-                "arrayObject",
-                "arrayAny",
-                "any",
-                "chatHistory",
-                "datasetQuote",
-                "dynamic",
-                "selectApp",
-                "selectDataset"
-              ],
-              "showDescription": false,
-              "showDefaultValue": true
-            },
-            "required": true,
-            "value": ["pluginInput", "img_correction"]
-          },
-          {
-            "renderTypeList": ["reference"],
-            "valueType": "boolean",
-            "canEdit": true,
-            "key": "formula",
-            "label": "formula",
-            "customInputConfig": {
-              "selectValueTypeList": [
-                "string",
-                "number",
-                "boolean",
-                "object",
-                "arrayString",
-                "arrayNumber",
-                "arrayBoolean",
-                "arrayObject",
-                "arrayAny",
-                "any",
-                "chatHistory",
-                "datasetQuote",
-                "dynamic",
-                "selectApp",
-                "selectDataset"
-              ],
-              "showDescription": false,
-              "showDefaultValue": true
-            },
-            "required": true,
-            "value": ["pluginInput", "formula"]
-          }
-        ],
-        "outputs": [
-          {
-            "id": "error",
-            "key": "error",
-            "label": "workflow:request_error",
-            "description": "HTTP请求错误信息，成功时返回空",
-            "valueType": "object",
-            "type": "static"
-          },
-          {
-            "id": "httpRawResponse",
-            "key": "httpRawResponse",
-            "required": true,
-            "label": "workflow:raw_response",
-            "description": "HTTP请求的原始响应。只能接受字符串或JSON类型响应数据。",
-            "valueType": "any",
-            "type": "static"
-          },
-          {
-            "id": "system_addOutputParam",
-            "key": "system_addOutputParam",
-            "type": "dynamic",
-            "valueType": "dynamic",
-            "label": "",
-            "customFieldConfig": {
-              "selectValueTypeList": [
-                "string",
-                "number",
-                "boolean",
-                "object",
-                "arrayString",
-                "arrayNumber",
-                "arrayBoolean",
-                "arrayObject",
-                "any",
-                "chatHistory",
-                "datasetQuote",
-                "dynamic",
-                "selectApp",
-                "selectDataset"
-              ],
-              "showDescription": false,
-              "showDefaultValue": false
-            }
-          },
-          {
-            "id": "xWQuEf50F3mr",
-            "valueType": "string",
-            "type": "dynamic",
-            "key": "result",
-            "label": "result"
-          },
-          {
-            "id": "jbv4nVZvmFXm",
-            "valueType": "string",
-            "type": "dynamic",
-            "key": "failreason",
-            "label": "failreason"
-          },
-          {
-            "id": "k46cjNulVk5Y",
-            "valueType": "boolean",
-            "type": "dynamic",
-            "key": "success",
-            "label": "success"
-          }
-        ]
-      }
-    ],
-    "edges": [
-      {
-        "source": "pluginInput",
-        "target": "zHG5jJBkXmjB",
-        "sourceHandle": "pluginInput-source-right",
-        "targetHandle": "zHG5jJBkXmjB-target-left"
-      },
-      {
-        "source": "zHG5jJBkXmjB",
-        "target": "pluginOutput",
-        "sourceHandle": "zHG5jJBkXmjB-source-right",
-        "targetHandle": "pluginOutput-target-left"
-      }
-    ]
-  }
-}
--- a/packages/plugins/src/Doc2X/FilePDF2text/index.ts
+++ b/packages/plugins/src/Doc2X/FilePDF2text/index.ts
@@ -1,165 +0,0 @@
-import { delay } from '@fastgpt/global/common/system/utils';
-import { addLog } from '@fastgpt/service/common/system/log';
-import { result } from 'lodash';
-
-type Props = {
-  apikey: string;
-  files: Array<string>;
-  ocr: boolean;
-};
-
-// Response type same as HTTP outputs
-type Response = Promise<{
-  result: string;
-  failreason: string;
-  success: boolean;
-}>;
-
-const main = async ({ apikey, files, ocr }: Props): Response => {
-  // Check the apikey
-  if (!apikey) {
-    return {
-      result: '',
-      failreason: `API key is required`,
-      success: false
-    };
-  }
-
-  let real_api_key = apikey;
-  if (!apikey.startsWith('sk-')) {
-    const response = await fetch('https://api.doc2x.noedgeai.com/api/token/refresh', {
-      method: 'POST',
-      headers: {
-        Authorization: `Bearer ${apikey}`
-      }
-    });
-    if (response.status !== 200) {
-      return {
-        result: '',
-        failreason: `Get token failed: ${await response.text()}`,
-        success: false
-      };
-    }
-    const data = await response.json();
-    real_api_key = data.data.token;
-  }
-
-  let final_result = '';
-  let fail_reason = '';
-  let flag = false;
-  //Process each file one by one
-  for await (const url of files) {
-    //Fetch the pdf and check its contene type
-    const PDFResponse = await fetch(url);
-    if (!PDFResponse.ok) {
-      fail_reason += `\n---\nFile:${url} \n<Content>\nFailed to fetch PDF from URL\n</Content>\n`;
-      flag = true;
-      continue;
-    }
-
-    const contentType = PDFResponse.headers.get('content-type');
-    const file_name = url.match(/read\?filename=([^&]+)/)?.[1] || 'unknown.pdf';
-    if (!contentType || !contentType.startsWith('application/pdf')) {
-      fail_reason += `\n---\nFile:${file_name}\n<Content>\nThe provided file does not point to a PDF: ${contentType}\n</Content>\n`;
-      flag = true;
-      continue;
-    }
-
-    const blob = await PDFResponse.blob();
-    const formData = new FormData();
-    formData.append('file', blob, file_name);
-    formData.append('ocr', ocr ? '1' : '0');
-
-    let upload_url = 'https://api.doc2x.noedgeai.com/api/platform/async/pdf';
-    if (real_api_key.startsWith('sk-')) {
-      upload_url = 'https://api.doc2x.noedgeai.com/api/v1/async/pdf';
-    }
-
-    let uuid;
-    let upload_flag = true;
-    const uploadAttempts = [1, 2, 3];
-    for await (const attempt of uploadAttempts) {
-      const upload_response = await fetch(upload_url, {
-        method: 'POST',
-        headers: {
-          Authorization: `Bearer ${real_api_key}`
-        },
-        body: formData
-      });
-      if (!upload_response.ok) {
-        // Rate limit, wait for 10s and retry at most 3 times
-        if (upload_response.status === 429 && attempt < 3) {
-          await delay(10000);
-          continue;
-        }
-        fail_reason += `\n---\nFile:${file_name}\n<Content>\nFailed to upload file: ${await upload_response.text()}\n</Content>\n`;
-        flag = true;
-        upload_flag = false;
-      }
-      if (!upload_flag) {
-        continue;
-      }
-      const upload_data = await upload_response.json();
-      uuid = upload_data.data.uuid;
-      break;
-    }
-
-    // Get the result by uuid
-    let result_url = 'https://api.doc2x.noedgeai.com/api/platform/async/status?uuid=' + uuid;
-    if (real_api_key.startsWith('sk-')) {
-      result_url = 'https://api.doc2x.noedgeai.com/api/v1/async/status?uuid=' + uuid;
-    }
-
-    let required_flag = true;
-    let result = '';
-    // Wait for the result, at most 100s
-    const maxAttempts = 100;
-    for await (const _ of Array(maxAttempts).keys()) {
-      const result_response = await fetch(result_url, {
-        headers: {
-          Authorization: `Bearer ${real_api_key}`
-        }
-      });
-      if (!result_response.ok) {
-        fail_reason += `\n---\nFile:${file_name}\n<Content>\nFailed to get result: ${await result_response.text()}\n</Content>\n`;
-        flag = true;
-        required_flag = false;
-        break;
-      }
-      const result_data = await result_response.json();
-      if (['ready', 'processing'].includes(result_data.data.status)) {
-        await delay(1000);
-      } else if (result_data.data.status === 'pages limit exceeded') {
-        fail_reason += `\n---\nFile:${file_name}\n<Content>\nPages limit exceeded\n</Content>\n`;
-        flag = true;
-        required_flag = false;
-        break;
-      } else if (result_data.data.status === 'success') {
-        result = await Promise.all(
-          result_data.data.result.pages.map((page: { md: any }) => page.md)
-        ).then((pages) => pages.join('\n'));
-        result = result.replace(/\\[\(\)]/g, '$').replace(/\\[\[\]]/g, '$$');
-        final_result += `\n---\nFile:${file_name}\n<Content>\n${result}\n</Content>\n`;
-        required_flag = false;
-        break;
-      } else {
-        fail_reason += `\n---\nFile:${file_name}\n<Content>\nFailed to get result: ${result_data.data.status}\n</Content>\n`;
-        flag = true;
-        required_flag = false;
-        break;
-      }
-    }
-    if (required_flag) {
-      fail_reason += `\n---\nFile:${file_name}\n<Content>\nTimeout after 100s for uuid ${uuid}\n</Content>\n`;
-      flag = true;
-    }
-  }
-
-  return {
-    result: final_result,
-    failreason: fail_reason,
-    success: !flag
-  };
-};
-
-export default main;
--- a/packages/plugins/src/Doc2X/PDF2text/index.ts
+++ b/packages/plugins/src/Doc2X/PDF2text/index.ts
@@ -0,0 +1,157 @@
+import { delay } from '@fastgpt/global/common/system/utils';
+import axios from 'axios';
+import { getErrText } from '@fastgpt/global/common/error/utils';
+
+type Props = {
+  apikey: string;
+  files: string[];
+};
+
+// Response type same as HTTP outputs
+type Response = Promise<{
+  result: string;
+  success: boolean;
+  error?: Record<string, any>;
+}>;
+
+const main = async ({ apikey, files }: Props): Response => {
+  // Check the apikey
+  if (!apikey) {
+    return Promise.reject(`API key is required`);
+  }
+  const successResult = [];
+  const failedResult = [];
+
+  const axiosInstance = axios.create({
+    timeout: 30000 // 30 seconds timeout
+  });
+
+  //Process each file one by one
+  for await (const url of files) {
+    try {
+      //Fetch the pdf and check its content type
+      const PDFResponse = await axiosInstance.get(url, { responseType: 'arraybuffer' });
+      if (PDFResponse.status !== 200) {
+        throw new Error(
+          `File:${url} \n<Content>\nFailed to fetch PDF from URL: ${PDFResponse.statusText}\n</Content>`
+        );
+      }
+
+      const contentType = PDFResponse.headers['content-type'];
+      const file_name = url.match(/read\/([^?]+)/)?.[1] || 'unknown.pdf';
+      if (!contentType || !contentType.startsWith('application/pdf')) {
+        throw new Error(
+          `File:${file_name}\n<Content>\nThe provided file does not point to a PDF: ${contentType}\n</Content>`
+        );
+      }
+
+      const blob = new Blob([PDFResponse.data], { type: 'application/pdf' });
+      // Get pre-upload URL first
+      const preupload_response = await axiosInstance.post(
+        'https://v2.doc2x.noedgeai.com/api/v2/parse/preupload',
+        null,
+        {
+          headers: {
+            Authorization: `Bearer ${apikey}`
+          }
+        }
+      );
+
+      if (preupload_response.status !== 200) {
+        throw new Error(
+          `File:${file_name}\n<Content>\nFailed to get pre-upload URL: ${preupload_response.statusText}\n</Content>`
+        );
+      }
+
+      const preupload_data = preupload_response.data;
+      if (preupload_data.code !== 'success') {
+        throw new Error(
+          `File:${file_name}\n<Content>\nFailed to get pre-upload URL: ${JSON.stringify(preupload_data)}\n</Content>`
+        );
+      }
+
+      const upload_url = preupload_data.data.url;
+      const uid = preupload_data.data.uid;
+      // Upload file to pre-signed URL with binary stream
+
+      const response = await axiosInstance.put(upload_url, blob, {
+        headers: {
+          'Content-Type': 'application/pdf'
+        }
+      });
+      if (response.status !== 200) {
+        throw new Error(`Upload failed with status ${response.status}: ${response.statusText}`);
+      }
+
+      // Get the result by uid
+
+      // Wait for the result, at most 90s
+      const checkResult = async (retry = 30) => {
+        if (retry <= 0)
+          return Promise.reject(
+            `File:${file_name}\n<Content>\nFailed to get result (uid: ${uid}): Get result timeout\n</Content>`
+          );
+
+        try {
+          const result_response = await axiosInstance.get(
+            `https://v2.doc2x.noedgeai.com/api/v2/parse/status?uid=${uid}`,
+            {
+              headers: {
+                Authorization: `Bearer ${apikey}`
+              }
+            }
+          );
+
+          const result_data = result_response.data;
+          if (!['ok', 'success'].includes(result_data.code)) {
+            return Promise.reject(
+              `File:${file_name}\n<Content>\nFailed to get result (uid: ${uid}): ${JSON.stringify(result_data)}\n</Content>`
+            );
+          }
+
+          if (['ready', 'processing'].includes(result_data.data.status)) {
+            await delay(3000);
+            return checkResult(retry - 1);
+          }
+
+          if (result_data.data.status === 'success') {
+            const result = (
+              await Promise.all(
+                result_data.data.result.pages.map((page: { md: any }) => page.md)
+              ).then((pages) => pages.join('\n'))
+            )
+              // Do some post-processing
+              .replace(/\\[\(\)]/g, '$')
+              .replace(/\\[\[\]]/g, '$$')
+              .replace(/<img\s+src="([^"]+)"(?:\s*\?[^>]*)?(?:\s*\/>|>)/g, '![img]($1)');
+
+            return `File:${file_name}\n<Content>\n${result}\n</Content>`;
+          }
+
+          await delay(100);
+          return checkResult(retry - 1);
+        } catch (error) {
+          await delay(100);
+          return checkResult(retry - 1);
+        }
+      };
+
+      const result = await checkResult();
+      successResult.push(result);
+    } catch (error) {
+      failedResult.push(
+        `File:${url} \n<Content>\nFailed to fetch image from URL: ${getErrText(error)}\n</Content>`
+      );
+    }
+  }
+
+  return {
+    result: successResult.join('\n******\n'),
+    error: {
+      message: failedResult.join('\n******\n')
+    },
+    success: failedResult.length === 0
+  };
+};
+
+export default main;
--- a/packages/plugins/src/Doc2X/FilePDF2text/template.json
+++ b/packages/plugins/src/Doc2X/FilePDF2text/template.json
@@ -1,9 +1,9 @@
 {
  "author": "Menghuan1918",
  "version": "488",
-  "name": "Doc2X PDF文件(文件)识别",
+  "name": "PDF识别",
  "avatar": "plugins/doc2x",
-  "intro": "将上传的PDF文件发送至Doc2X进行解析，返回带LaTeX公式的markdown格式的文本",
+  "intro": "将PDF文件发送至Doc2X进行解析，返回结构化的LaTeX公式的文本(markdown)，支持传入String类型的URL或者流程输出中的文件链接变量",
  "courseUrl": "https://fael3z0zfze.feishu.cn/wiki/Rkc5witXWiJoi5kORd2cofh6nDg?fromScene=spaceOverview",
  "showStatus": true,
  "weight": 10,
@@ -13,30 +13,16 @@

  "workflow": {
    "nodes": [
-      {
-        "nodeId": "pluginConfig",
-        "name": "common:core.module.template.system_config",
-        "intro": "",
-        "avatar": "core/workflow/template/systemConfig",
-        "flowNodeType": "pluginConfig",
-        "position": {
-          "x": -30.474351356537454,
-          "y": -101.45216221730038
-        },
-        "version": "4811",
-        "inputs": [],
-        "outputs": []
-      },
      {
        "nodeId": "pluginInput",
-        "name": "插件开始",
+        "name": "自定义插件输入",
        "intro": "可以配置插件需要哪些输入，利用这些输入来运行插件",
        "avatar": "core/workflow/template/workflowStart",
        "flowNodeType": "pluginInput",
        "showStatus": false,
        "position": {
-          "x": 407.2817920483865,
-          "y": -101.45216221730038
+          "x": -137.96875104510553,
+          "y": -90.9968973555371
        },
        "version": "481",
        "inputs": [
@@ -47,33 +33,25 @@
            "canEdit": true,
            "key": "apikey",
            "label": "apikey",
-            "description": "Doc2X的验证密匙，对于个人用户可以从Doc2X官网 - 个人信息 - 身份令牌获得",
+            "description": "Doc2X的API密匙，可以从Doc2X开放平台获得",
            "required": true,
-            "toolDescription": "",
-            "defaultValue": ""
+            "defaultValue": "",
+            "list": []
          },
          {
-            "renderTypeList": ["reference"],
+            "renderTypeList": ["fileSelect"],
            "selectedTypeIndex": 0,
            "valueType": "arrayString",
            "canEdit": true,
            "key": "files",
            "label": "files",
-            "description": "待处理的PDF文件",
+            "description": "需要处理的PDF地址",
            "required": true,
-            "toolDescription": "待处理的PDF文件"
-          },
-          {
-            "renderTypeList": ["switch"],
-            "selectedTypeIndex": 0,
-            "valueType": "boolean",
-            "canEdit": true,
-            "key": "ocr",
-            "label": "ocr",
-            "description": "是否开启对PDF文件内图片的OCR识别，建议开启",
-            "required": true,
-            "toolDescription": "",
-            "defaultValue": true
+            "list": [],
+            "canSelectFile": true,
+            "canSelectImg": false,
+            "maxFiles": 14,
+            "defaultValue": ""
          }
        ],
        "outputs": [
@@ -90,26 +68,19 @@
            "key": "files",
            "label": "files",
            "type": "hidden"
-          },
-          {
-            "id": "formula",
-            "valueType": "boolean",
-            "key": "ocr",
-            "label": "ocr",
-            "type": "hidden"
          }
        ]
      },
      {
        "nodeId": "pluginOutput",
-        "name": "插件输出",
+        "name": "自定义插件输出",
        "intro": "自定义配置外部输出，使用插件时，仅暴露自定义配置的输出",
        "avatar": "core/workflow/template/pluginOutput",
        "flowNodeType": "pluginOutput",
        "showStatus": false,
        "position": {
-          "x": 1842.070888321717,
-          "y": -101.45216221730038
+          "x": 1505.494975310334,
+          "y": -4.14668564643415
        },
        "version": "481",
        "inputs": [
@@ -124,12 +95,13 @@
          },
          {
            "renderTypeList": ["reference"],
-            "valueType": "string",
+            "valueType": "object",
            "canEdit": true,
-            "key": "failreason",
-            "label": "failreason",
-            "description": "文件处理失败原因，由文件名以及报错组成，多个文件之间由横线分隔开",
-            "value": ["zHG5jJBkXmjB", "yDxzW5CFalGw"]
+            "key": "error",
+            "label": "error",
+            "description": "",
+            "value": ["zHG5jJBkXmjB", "httpRawResponse"],
+            "isToolOutput": true
          },
          {
            "renderTypeList": ["reference"],
@@ -138,7 +110,8 @@
            "key": "success",
            "label": "success",
            "description": "是否全部文件都处理成功，如有没有处理成功的文件，失败原因将会输出在failreason中",
-            "value": ["zHG5jJBkXmjB", "m6CJJj7GFud5"]
+            "value": ["zHG5jJBkXmjB", "m6CJJj7GFud5"],
+            "isToolOutput": false
          }
        ],
        "outputs": []
@@ -151,8 +124,8 @@
        "flowNodeType": "httpRequest468",
        "showStatus": true,
        "position": {
-          "x": 1077.7986740892777,
-          "y": -496.9521622173004
+          "x": 619.0661933308237,
+          "y": -472.91377894611503
        },
        "version": "481",
        "inputs": [
@@ -202,7 +175,7 @@
            "renderTypeList": ["custom"],
            "valueType": "number",
            "label": "",
-            "value": 30,
+            "value": 300,
            "min": 5,
            "max": 600,
            "required": true,
@@ -217,7 +190,7 @@
            "description": "common:core.module.input.description.Http Request Url",
            "placeholder": "https://api.ai.com/getInventory",
            "required": false,
-            "value": "Doc2X/FilePDF2text",
+            "value": "Doc2X/PDF2text",
            "debugLabel": "",
            "toolDescription": ""
          },
@@ -247,7 +220,7 @@
            "key": "system_httpJsonBody",
            "renderTypeList": ["hidden"],
            "valueType": "any",
-            "value": "{\n  \"apikey\": \"{{apikey}}\",\n  \"files\": {{files}},\n  \"ocr\": {{ocr}}\n}",
+            "value": "{\n  \"apikey\": \"{{apikey}}\",\n  \"files\": {{files}}\n}",
            "label": "",
            "required": false,
            "debugLabel": "",
@@ -331,37 +304,7 @@
              "showDefaultValue": true
            },
            "required": true,
-            "value": ["pluginInput", "url"]
-          },
-          {
-            "renderTypeList": ["reference"],
-            "valueType": "boolean",
-            "canEdit": true,
-            "key": "ocr",
-            "label": "ocr",
-            "customInputConfig": {
-              "selectValueTypeList": [
-                "string",
-                "number",
-                "boolean",
-                "object",
-                "arrayString",
-                "arrayNumber",
-                "arrayBoolean",
-                "arrayObject",
-                "arrayAny",
-                "any",
-                "chatHistory",
-                "datasetQuote",
-                "dynamic",
-                "selectApp",
-                "selectDataset"
-              ],
-              "showDescription": false,
-              "showDefaultValue": true
-            },
-            "required": true,
-            "value": ["pluginInput", "formula"]
+            "value": [["pluginInput", "url"]]
          }
        ],
        "outputs": [
@@ -422,30 +365,42 @@
            "type": "dynamic",
            "key": "success",
            "label": "success"
-          },
-          {
-            "id": "yDxzW5CFalGw",
-            "valueType": "string",
-            "type": "dynamic",
-            "key": "failreason",
-            "label": "failreason"
          }
        ]
      }
    ],
    "edges": [
-      {
-        "source": "pluginInput",
-        "target": "zHG5jJBkXmjB",
-        "sourceHandle": "pluginInput-source-right",
-        "targetHandle": "zHG5jJBkXmjB-target-left"
-      },
      {
        "source": "zHG5jJBkXmjB",
        "target": "pluginOutput",
        "sourceHandle": "zHG5jJBkXmjB-source-right",
        "targetHandle": "pluginOutput-target-left"
+      },
+      {
+        "source": "pluginInput",
+        "target": "zHG5jJBkXmjB",
+        "sourceHandle": "pluginInput-source-right",
+        "targetHandle": "zHG5jJBkXmjB-target-left"
      }
-    ]
+    ],
+    "chatConfig": {
+      "questionGuide": false,
+      "ttsConfig": {
+        "type": "web"
+      },
+      "whisperConfig": {
+        "open": false,
+        "autoSend": false,
+        "autoTTSResponse": false
+      },
+      "chatInputGuide": {
+        "open": false,
+        "textList": [],
+        "customUrl": ""
+      },
+      "instruction": "",
+      "variables": [],
+      "welcomeText": ""
+    }
  }
 }
--- a/packages/plugins/src/Doc2X/URLImg2text/index.ts
+++ b/packages/plugins/src/Doc2X/URLImg2text/index.ts
@@ -1,166 +0,0 @@
-import { delay } from '@fastgpt/global/common/system/utils';
-import { addLog } from '@fastgpt/service/common/system/log';
-
-type Props = {
-  apikey: string;
-  url: string;
-  img_correction: boolean;
-  formula: boolean;
-};
-
-type Response = Promise<{
-  result: string;
-  success: boolean;
-}>;
-
-const main = async ({ apikey, url, img_correction, formula }: Props): Response => {
-  // Check the apikey
-  if (!apikey) {
-    return {
-      result: `API key is required`,
-      success: false
-    };
-  }
-
-  let real_api_key = apikey;
-  if (!apikey.startsWith('sk-')) {
-    const response = await fetch('https://api.doc2x.noedgeai.com/api/token/refresh', {
-      method: 'POST',
-      headers: {
-        Authorization: `Bearer ${apikey}`
-      }
-    });
-    if (response.status !== 200) {
-      return {
-        result: `Get token failed: ${await response.text()}`,
-        success: false
-      };
-    }
-    const data = await response.json();
-    real_api_key = data.data.token;
-  }
-
-  let imageResponse;
-  // Fetch the image and check its content type
-  try {
-    imageResponse = await fetch(url);
-  } catch (e) {
-    return {
-      result: `Failed to fetch image from URL: ${url} with error: ${e}`,
-      success: false
-    };
-  }
-
-  if (!imageResponse.ok) {
-    return {
-      result: `Failed to fetch image from URL: ${url}`,
-      success: false
-    };
-  }
-
-  const contentType = imageResponse.headers.get('content-type');
-  if (!contentType || !contentType.startsWith('image/')) {
-    return {
-      result: `The provided URL does not point to an image: ${contentType}`,
-      success: false
-    };
-  }
-
-  const blob = await imageResponse.blob();
-  const formData = new FormData();
-  const fileName = url.split('/').pop()?.split('?')[0] || 'image';
-  formData.append('file', blob, fileName);
-  formData.append('img_correction', img_correction ? '1' : '0');
-  formData.append('equation', formula ? '1' : '0');
-
-  let upload_url = 'https://api.doc2x.noedgeai.com/api/platform/async/img';
-  if (real_api_key.startsWith('sk-')) {
-    upload_url = 'https://api.doc2x.noedgeai.com/api/v1/async/img';
-  }
-
-  let uuid;
-  const uploadAttempts = [1, 2, 3];
-  for await (const attempt of uploadAttempts) {
-    const upload_response = await fetch(upload_url, {
-      method: 'POST',
-      headers: {
-        Authorization: `Bearer ${real_api_key}`
-      },
-      body: formData
-    });
-
-    if (!upload_response.ok) {
-      // Rate limit, wait for 10s and retry at most 3 times
-      if (upload_response.status === 429 && attempt < 3) {
-        await delay(10000);
-        continue;
-      }
-      return {
-        result: `Failed to upload image: ${await upload_response.text()}`,
-        success: false
-      };
-    }
-
-    const upload_data = await upload_response.json();
-    uuid = upload_data.data.uuid;
-    break;
-  }
-
-  // Get the result by uuid
-  let result_url = 'https://api.doc2x.noedgeai.com/api/platform/async/status?uuid=' + uuid;
-  if (real_api_key.startsWith('sk-')) {
-    result_url = 'https://api.doc2x.noedgeai.com/api/v1/async/status?uuid=' + uuid;
-  }
-  const maxAttempts = 100;
-  // Wait for the result, at most 100s
-  for await (const _ of Array(maxAttempts).keys()) {
-    const result_response = await fetch(result_url, {
-      headers: {
-        Authorization: `Bearer ${real_api_key}`
-      }
-    });
-    if (!result_response.ok) {
-      return {
-        result: `Failed to get result: ${await result_response.text()}`,
-        success: false
-      };
-    }
-    const result_data = await result_response.json();
-    if (['ready', 'processing'].includes(result_data.data.status)) {
-      await delay(1000);
-    } else if (result_data.data.status === 'pages limit exceeded') {
-      return {
-        result: 'Doc2X Pages limit exceeded',
-        success: false
-      };
-    } else if (result_data.data.status === 'success') {
-      let result;
-      try {
-        result = result_data.data.result.pages[0].md;
-        result = result.replace(/\\[\(\)]/g, '$').replace(/\\[\[\]]/g, '$$');
-      } catch {
-        // no pages
-        return {
-          result: '',
-          success: true
-        };
-      }
-      return {
-        result: result,
-        success: true
-      };
-    } else {
-      return {
-        result: `Failed to get result: ${await result_data.text()}`,
-        success: false
-      };
-    }
-  }
-
-  return {
-    result: 'Timeout waiting for result',
-    success: false
-  };
-};
-
-export default main;
--- a/packages/plugins/src/Doc2X/URLImg2text/template.json
+++ b/packages/plugins/src/Doc2X/URLImg2text/template.json
@@ -1,484 +0,0 @@
-{
-  "author": "Menghuan1918",
-  "version": "488",
-  "name": "Doc2X 图像(URL)识别",
-  "avatar": "plugins/doc2x",
-  "intro": "从URL下载图片并发送至Doc2X进行解析，返回带LaTeX公式的markdown格式的文本",
-  "courseUrl": "https://fael3z0zfze.feishu.cn/wiki/Rkc5witXWiJoi5kORd2cofh6nDg?fromScene=spaceOverview",
-  "showStatus": true,
-  "weight": 10,
-
-  "isTool": true,
-  "templateType": "tools",
-
-  "workflow": {
-    "nodes": [
-      {
-        "nodeId": "pluginInput",
-        "name": "插件开始",
-        "intro": "可以配置插件需要哪些输入，利用这些输入来运行插件",
-        "avatar": "core/workflow/template/workflowStart",
-        "flowNodeType": "pluginInput",
-        "showStatus": false,
-        "position": {
-          "x": 353.91678143999377,
-          "y": -75.09744210499466
-        },
-        "version": "481",
-        "inputs": [
-          {
-            "renderTypeList": ["input"],
-            "selectedTypeIndex": 0,
-            "valueType": "string",
-            "canEdit": true,
-            "key": "apikey",
-            "label": "apikey",
-            "description": "Doc2X的验证密匙，对于个人用户可以从Doc2X官网 - 个人信息 - 身份令牌获得",
-            "required": true,
-            "toolDescription": "",
-            "defaultValue": ""
-          },
-          {
-            "renderTypeList": ["reference"],
-            "selectedTypeIndex": 0,
-            "valueType": "string",
-            "canEdit": true,
-            "key": "url",
-            "label": "url",
-            "description": "待处理图片的URL",
-            "required": true,
-            "toolDescription": "待处理图片的URL"
-          },
-          {
-            "renderTypeList": ["switch"],
-            "selectedTypeIndex": 0,
-            "valueType": "boolean",
-            "canEdit": true,
-            "key": "img_correction",
-            "label": "img_correction",
-            "description": "是否启用图形矫正功能",
-            "required": true,
-            "toolDescription": "",
-            "defaultValue": false
-          },
-          {
-            "renderTypeList": ["switch"],
-            "selectedTypeIndex": 0,
-            "valueType": "boolean",
-            "canEdit": true,
-            "key": "formula",
-            "label": "formula",
-            "description": "是否开启纯公式识别(仅适用于图片内容仅有公式时)",
-            "required": true,
-            "toolDescription": "",
-            "defaultValue": false
-          }
-        ],
-        "outputs": [
-          {
-            "id": "apikey",
-            "valueType": "string",
-            "key": "apikey",
-            "label": "apikey",
-            "type": "hidden"
-          },
-          {
-            "id": "url",
-            "valueType": "string",
-            "key": "url",
-            "label": "url",
-            "type": "hidden"
-          },
-          {
-            "id": "img_correction",
-            "valueType": "boolean",
-            "key": "img_correction",
-            "label": "img_correction",
-            "type": "hidden"
-          },
-          {
-            "id": "formula",
-            "valueType": "boolean",
-            "key": "formula",
-            "label": "formula",
-            "type": "hidden"
-          }
-        ]
-      },
-      {
-        "nodeId": "pluginOutput",
-        "name": "插件输出",
-        "intro": "自定义配置外部输出，使用插件时，仅暴露自定义配置的输出",
-        "avatar": "core/workflow/template/pluginOutput",
-        "flowNodeType": "pluginOutput",
-        "showStatus": false,
-        "position": {
-          "x": 1703.581616889916,
-          "y": -14.097442104994656
-        },
-        "version": "481",
-        "inputs": [
-          {
-            "renderTypeList": ["reference"],
-            "valueType": "string",
-            "canEdit": true,
-            "key": "result",
-            "label": "result",
-            "description": "处理结果(或者是报错信息)",
-            "value": ["zHG5jJBkXmjB", "xWQuEf50F3mr"]
-          },
-          {
-            "renderTypeList": ["reference"],
-            "valueType": "boolean",
-            "canEdit": true,
-            "key": "success",
-            "label": "success",
-            "description": "是否处理成功",
-            "value": ["zHG5jJBkXmjB", "m6CJJj7GFud5"]
-          }
-        ],
-        "outputs": []
-      },
-      {
-        "nodeId": "zHG5jJBkXmjB",
-        "name": "HTTP 请求",
-        "intro": "可以发出一个 HTTP 请求，实现更为复杂的操作（联网搜索、数据库查询等）",
-        "avatar": "core/workflow/template/httpRequest",
-        "flowNodeType": "httpRequest468",
-        "showStatus": true,
-        "position": {
-          "x": 1000.6685388413375,
-          "y": -457.0974421049947
-        },
-        "version": "481",
-        "inputs": [
-          {
-            "key": "system_addInputParam",
-            "renderTypeList": ["addInputParam"],
-            "valueType": "dynamic",
-            "label": "",
-            "required": false,
-            "description": "common:core.module.input.description.HTTP Dynamic Input",
-            "customInputConfig": {
-              "selectValueTypeList": [
-                "string",
-                "number",
-                "boolean",
-                "object",
-                "arrayString",
-                "arrayNumber",
-                "arrayBoolean",
-                "arrayObject",
-                "arrayAny",
-                "any",
-                "chatHistory",
-                "datasetQuote",
-                "dynamic",
-                "selectApp",
-                "selectDataset"
-              ],
-              "showDescription": false,
-              "showDefaultValue": true
-            },
-            "debugLabel": "",
-            "toolDescription": ""
-          },
-          {
-            "key": "system_httpMethod",
-            "renderTypeList": ["custom"],
-            "valueType": "string",
-            "label": "",
-            "value": "POST",
-            "required": true,
-            "debugLabel": "",
-            "toolDescription": ""
-          },
-          {
-            "key": "system_httpTimeout",
-            "renderTypeList": ["custom"],
-            "valueType": "number",
-            "label": "",
-            "value": 30,
-            "min": 5,
-            "max": 600,
-            "required": true,
-            "debugLabel": "",
-            "toolDescription": ""
-          },
-          {
-            "key": "system_httpReqUrl",
-            "renderTypeList": ["hidden"],
-            "valueType": "string",
-            "label": "",
-            "description": "common:core.module.input.description.Http Request Url",
-            "placeholder": "https://api.ai.com/getInventory",
-            "required": false,
-            "value": "Doc2X/URLImg2text",
-            "debugLabel": "",
-            "toolDescription": ""
-          },
-          {
-            "key": "system_httpHeader",
-            "renderTypeList": ["custom"],
-            "valueType": "any",
-            "value": [],
-            "label": "",
-            "description": "common:core.module.input.description.Http Request Header",
-            "placeholder": "common:core.module.input.description.Http Request Header",
-            "required": false,
-            "debugLabel": "",
-            "toolDescription": ""
-          },
-          {
-            "key": "system_httpParams",
-            "renderTypeList": ["hidden"],
-            "valueType": "any",
-            "value": [],
-            "label": "",
-            "required": false,
-            "debugLabel": "",
-            "toolDescription": ""
-          },
-          {
-            "key": "system_httpJsonBody",
-            "renderTypeList": ["hidden"],
-            "valueType": "any",
-            "value": "{\n  \"apikey\": \"{{apikey}}\",\n  \"url\": \"{{url}}\",\n  \"img_correction\": {{img_correction}},\n  \"formula\": {{formula}}\n}",
-            "label": "",
-            "required": false,
-            "debugLabel": "",
-            "toolDescription": ""
-          },
-          {
-            "key": "system_httpFormBody",
-            "renderTypeList": ["hidden"],
-            "valueType": "any",
-            "value": [],
-            "label": "",
-            "required": false,
-            "debugLabel": "",
-            "toolDescription": ""
-          },
-          {
-            "key": "system_httpContentType",
-            "renderTypeList": ["hidden"],
-            "valueType": "string",
-            "value": "json",
-            "label": "",
-            "required": false,
-            "debugLabel": "",
-            "toolDescription": ""
-          },
-          {
-            "renderTypeList": ["reference"],
-            "valueType": "string",
-            "canEdit": true,
-            "key": "apikey",
-            "label": "apikey",
-            "customInputConfig": {
-              "selectValueTypeList": [
-                "string",
-                "number",
-                "boolean",
-                "object",
-                "arrayString",
-                "arrayNumber",
-                "arrayBoolean",
-                "arrayObject",
-                "arrayAny",
-                "any",
-                "chatHistory",
-                "datasetQuote",
-                "dynamic",
-                "selectApp",
-                "selectDataset"
-              ],
-              "showDescription": false,
-              "showDefaultValue": true
-            },
-            "required": true,
-            "value": ["pluginInput", "apikey"]
-          },
-          {
-            "renderTypeList": ["reference"],
-            "valueType": "string",
-            "canEdit": true,
-            "key": "url",
-            "label": "url",
-            "customInputConfig": {
-              "selectValueTypeList": [
-                "string",
-                "number",
-                "boolean",
-                "object",
-                "arrayString",
-                "arrayNumber",
-                "arrayBoolean",
-                "arrayObject",
-                "arrayAny",
-                "any",
-                "chatHistory",
-                "datasetQuote",
-                "dynamic",
-                "selectApp",
-                "selectDataset"
-              ],
-              "showDescription": false,
-              "showDefaultValue": true
-            },
-            "required": true,
-            "value": ["pluginInput", "url"]
-          },
-          {
-            "renderTypeList": ["reference"],
-            "valueType": "boolean",
-            "canEdit": true,
-            "key": "img_correction",
-            "label": "img_correction",
-            "customInputConfig": {
-              "selectValueTypeList": [
-                "string",
-                "number",
-                "boolean",
-                "object",
-                "arrayString",
-                "arrayNumber",
-                "arrayBoolean",
-                "arrayObject",
-                "arrayAny",
-                "any",
-                "chatHistory",
-                "datasetQuote",
-                "dynamic",
-                "selectApp",
-                "selectDataset"
-              ],
-              "showDescription": false,
-              "showDefaultValue": true
-            },
-            "required": true,
-            "value": ["pluginInput", "img_correction"]
-          },
-          {
-            "renderTypeList": ["reference"],
-            "valueType": "boolean",
-            "canEdit": true,
-            "key": "formula",
-            "label": "formula",
-            "customInputConfig": {
-              "selectValueTypeList": [
-                "string",
-                "number",
-                "boolean",
-                "object",
-                "arrayString",
-                "arrayNumber",
-                "arrayBoolean",
-                "arrayObject",
-                "arrayAny",
-                "any",
-                "chatHistory",
-                "datasetQuote",
-                "dynamic",
-                "selectApp",
-                "selectDataset"
-              ],
-              "showDescription": false,
-              "showDefaultValue": true
-            },
-            "required": true,
-            "value": ["pluginInput", "formula"]
-          }
-        ],
-        "outputs": [
-          {
-            "id": "error",
-            "key": "error",
-            "label": "workflow:request_error",
-            "description": "HTTP请求错误信息，成功时返回空",
-            "valueType": "object",
-            "type": "static"
-          },
-          {
-            "id": "httpRawResponse",
-            "key": "httpRawResponse",
-            "required": true,
-            "label": "workflow:raw_response",
-            "description": "HTTP请求的原始响应。只能接受字符串或JSON类型响应数据。",
-            "valueType": "any",
-            "type": "static"
-          },
-          {
-            "id": "system_addOutputParam",
-            "key": "system_addOutputParam",
-            "type": "dynamic",
-            "valueType": "dynamic",
-            "label": "",
-            "customFieldConfig": {
-              "selectValueTypeList": [
-                "string",
-                "number",
-                "boolean",
-                "object",
-                "arrayString",
-                "arrayNumber",
-                "arrayBoolean",
-                "arrayObject",
-                "any",
-                "chatHistory",
-                "datasetQuote",
-                "dynamic",
-                "selectApp",
-                "selectDataset"
-              ],
-              "showDescription": false,
-              "showDefaultValue": false
-            }
-          },
-          {
-            "id": "xWQuEf50F3mr",
-            "valueType": "string",
-            "type": "dynamic",
-            "key": "result",
-            "label": "result"
-          },
-          {
-            "id": "m6CJJj7GFud5",
-            "valueType": "boolean",
-            "type": "dynamic",
-            "key": "success",
-            "label": "success"
-          }
-        ]
-      },
-      {
-        "nodeId": "sWEDDSeuI9ar",
-        "name": "系统配置",
-        "intro": "",
-        "avatar": "core/workflow/template/systemConfig",
-        "flowNodeType": "pluginConfig",
-        "position": {
-          "x": -117.03701176267538,
-          "y": -75.09744210499466
-        },
-        "version": "4811",
-        "inputs": [],
-        "outputs": []
-      }
-    ],
-    "edges": [
-      {
-        "source": "pluginInput",
-        "target": "zHG5jJBkXmjB",
-        "sourceHandle": "pluginInput-source-right",
-        "targetHandle": "zHG5jJBkXmjB-target-left"
-      },
-      {
-        "source": "zHG5jJBkXmjB",
-        "target": "pluginOutput",
-        "sourceHandle": "zHG5jJBkXmjB-source-right",
-        "targetHandle": "pluginOutput-target-left"
-      }
-    ]
-  }
-}
--- a/packages/plugins/src/Doc2X/URLPDF2text/index.ts
+++ b/packages/plugins/src/Doc2X/URLPDF2text/index.ts
@@ -1,156 +0,0 @@
-import { delay } from '@fastgpt/global/common/system/utils';
-import { addLog } from '@fastgpt/service/common/system/log';
-
-type Props = {
-  apikey: string;
-  url: string;
-  ocr: boolean;
-};
-
-// Response type same as HTTP outputs
-type Response = Promise<{
-  result: string;
-  success: boolean;
-}>;
-
-const main = async ({ apikey, url, ocr }: Props): Response => {
-  // Check the apikey
-  if (!apikey) {
-    return {
-      result: `API key is required`,
-      success: false
-    };
-  }
-
-  let real_api_key = apikey;
-  if (!apikey.startsWith('sk-')) {
-    const response = await fetch('https://api.doc2x.noedgeai.com/api/token/refresh', {
-      method: 'POST',
-      headers: {
-        Authorization: `Bearer ${apikey}`
-      }
-    });
-    if (response.status !== 200) {
-      return {
-        result: `Get token failed: ${await response.text()}`,
-        success: false
-      };
-    }
-    const data = await response.json();
-    real_api_key = data.data.token;
-  }
-
-  //Fetch the pdf and check its contene type
-  let PDFResponse;
-  try {
-    PDFResponse = await fetch(url);
-  } catch (e) {
-    return {
-      result: `Failed to fetch PDF from URL: ${url} with error: ${e}`,
-      success: false
-    };
-  }
-  if (!PDFResponse.ok) {
-    return {
-      result: `Failed to fetch PDF from URL: ${url}`,
-      success: false
-    };
-  }
-
-  const contentType = PDFResponse.headers.get('content-type');
-  if (!contentType || !contentType.startsWith('application/pdf')) {
-    return {
-      result: `The provided URL does not point to a PDF: ${contentType}`,
-      success: false
-    };
-  }
-
-  const blob = await PDFResponse.blob();
-  const formData = new FormData();
-  const fileName = url.split('/').pop()?.split('?')[0] || 'pdf';
-  formData.append('file', blob, fileName);
-  formData.append('ocr', ocr ? '1' : '0');
-
-  let upload_url = 'https://api.doc2x.noedgeai.com/api/platform/async/pdf';
-  if (real_api_key.startsWith('sk-')) {
-    upload_url = 'https://api.doc2x.noedgeai.com/api/v1/async/pdf';
-  }
-
-  let uuid;
-  const uploadAttempts = [1, 2, 3];
-  for await (const attempt of uploadAttempts) {
-    const upload_response = await fetch(upload_url, {
-      method: 'POST',
-      headers: {
-        Authorization: `Bearer ${real_api_key}`
-      },
-      body: formData
-    });
-    if (!upload_response.ok) {
-      if (upload_response.status === 429 && attempt < 3) {
-        await delay(10000);
-        continue;
-      }
-      return {
-        result: `Failed to upload file: ${await upload_response.text()}`,
-        success: false
-      };
-    }
-    const upload_data = await upload_response.json();
-    uuid = upload_data.data.uuid;
-    break;
-  }
-
-  // Get the result by uuid
-  let result_url = 'https://api.doc2x.noedgeai.com/api/platform/async/status?uuid=' + uuid;
-  if (real_api_key.startsWith('sk-')) {
-    result_url = 'https://api.doc2x.noedgeai.com/api/v1/async/status?uuid=' + uuid;
-  }
-
-  let result = '';
-  // Wait for the result, at most 100s
-  const maxAttempts = 100;
-  for await (const _ of Array(maxAttempts).keys()) {
-    const result_response = await fetch(result_url, {
-      headers: {
-        Authorization: `Bearer ${real_api_key}`
-      }
-    });
-    if (!result_response.ok) {
-      return {
-        result: `Failed to get result: ${await result_response.text()}`,
-        success: false
-      };
-    }
-    const result_data = await result_response.json();
-    if (['ready', 'processing'].includes(result_data.data.status)) {
-      await delay(1000);
-    } else if (result_data.data.status === 'pages limit exceeded') {
-      return {
-        result: 'Doc2X Pages limit exceeded',
-        success: false
-      };
-    } else if (result_data.data.status === 'success') {
-      result = await Promise.all(
-        result_data.data.result.pages.map((page: { md: any }) => page.md)
-      ).then((pages) => pages.join('\n'));
-      result = result.replace(/\\[\(\)]/g, '$').replace(/\\[\[\]]/g, '$$');
-      return {
-        result: result,
-        success: true
-      };
-    } else {
-      return {
-        result: `Failed to get result: ${await result_data.text()}`,
-        success: false
-      };
-    }
-  }
-
-  return {
-    result: 'Timeout waiting for result',
-    success: false
-  };
-};
-
-export default main;
--- a/packages/plugins/src/Doc2X/URLPDF2text/template.json
+++ b/packages/plugins/src/Doc2X/URLPDF2text/template.json
@@ -1,435 +0,0 @@
-{
-  "author": "Menghuan1918",
-  "version": "488",
-  "name": "Doc2X PDF文件(URL)识别",
-  "avatar": "plugins/doc2x",
-  "intro": "从URL下载PDF文件，并发送至Doc2X进行解析，返回带LaTeX公式的markdown格式的文本",
-  "courseUrl": "https://fael3z0zfze.feishu.cn/wiki/Rkc5witXWiJoi5kORd2cofh6nDg?fromScene=spaceOverview",
-  "showStatus": true,
-  "weight": 10,
-
-  "isTool": true,
-  "templateType": "tools",
-
-  "workflow": {
-    "nodes": [
-      {
-        "nodeId": "pluginInput",
-        "name": "插件开始",
-        "intro": "可以配置插件需要哪些输入，利用这些输入来运行插件",
-        "avatar": "core/workflow/template/workflowStart",
-        "flowNodeType": "pluginInput",
-        "showStatus": false,
-        "position": {
-          "x": 388.243055058894,
-          "y": -75.09744210499466
-        },
-        "version": "481",
-        "inputs": [
-          {
-            "renderTypeList": ["input"],
-            "selectedTypeIndex": 0,
-            "valueType": "string",
-            "canEdit": true,
-            "key": "apikey",
-            "label": "apikey",
-            "description": "Doc2X的验证密匙，对于个人用户可以从Doc2X官网 - 个人信息 - 身份令牌获得",
-            "required": true,
-            "toolDescription": "",
-            "defaultValue": ""
-          },
-          {
-            "renderTypeList": ["reference"],
-            "selectedTypeIndex": 0,
-            "valueType": "string",
-            "canEdit": true,
-            "key": "url",
-            "label": "url",
-            "description": "待处理PDF文件的URL",
-            "required": true,
-            "toolDescription": "待处理PDF文件的URL"
-          },
-          {
-            "renderTypeList": ["switch"],
-            "selectedTypeIndex": 0,
-            "valueType": "boolean",
-            "canEdit": true,
-            "key": "ocr",
-            "label": "ocr",
-            "description": "是否开启对PDF文件内图片的OCR识别，建议开启",
-            "required": true,
-            "toolDescription": "",
-            "defaultValue": true
-          }
-        ],
-        "outputs": [
-          {
-            "id": "apikey",
-            "valueType": "string",
-            "key": "apikey",
-            "label": "apikey",
-            "type": "hidden"
-          },
-          {
-            "id": "url",
-            "valueType": "string",
-            "key": "url",
-            "label": "url",
-            "type": "hidden"
-          },
-          {
-            "id": "formula",
-            "valueType": "boolean",
-            "key": "ocr",
-            "label": "ocr",
-            "type": "hidden"
-          }
-        ]
-      },
-      {
-        "nodeId": "pluginOutput",
-        "name": "插件输出",
-        "intro": "自定义配置外部输出，使用插件时，仅暴露自定义配置的输出",
-        "avatar": "core/workflow/template/pluginOutput",
-        "flowNodeType": "pluginOutput",
-        "showStatus": false,
-        "position": {
-          "x": 1665.6420513111314,
-          "y": -40.597442104994656
-        },
-        "version": "481",
-        "inputs": [
-          {
-            "renderTypeList": ["reference"],
-            "valueType": "string",
-            "canEdit": true,
-            "key": "result",
-            "label": "result",
-            "description": "处理结果(或者是报错信息)",
-            "value": ["zHG5jJBkXmjB", "xWQuEf50F3mr"]
-          },
-          {
-            "renderTypeList": ["reference"],
-            "valueType": "boolean",
-            "canEdit": true,
-            "key": "success",
-            "label": "success",
-            "description": "是否处理成功",
-            "value": ["zHG5jJBkXmjB", "m6CJJj7GFud5"]
-          }
-        ],
-        "outputs": []
-      },
-      {
-        "nodeId": "zHG5jJBkXmjB",
-        "name": "HTTP 请求",
-        "intro": "可以发出一个 HTTP 请求，实现更为复杂的操作（联网搜索、数据库查询等）",
-        "avatar": "core/workflow/template/httpRequest",
-        "flowNodeType": "httpRequest468",
-        "showStatus": true,
-        "position": {
-          "x": 966.3422652224374,
-          "y": -446.5974421049947
-        },
-        "version": "481",
-        "inputs": [
-          {
-            "key": "system_addInputParam",
-            "renderTypeList": ["addInputParam"],
-            "valueType": "dynamic",
-            "label": "",
-            "required": false,
-            "description": "common:core.module.input.description.HTTP Dynamic Input",
-            "customInputConfig": {
-              "selectValueTypeList": [
-                "string",
-                "number",
-                "boolean",
-                "object",
-                "arrayString",
-                "arrayNumber",
-                "arrayBoolean",
-                "arrayObject",
-                "arrayAny",
-                "any",
-                "chatHistory",
-                "datasetQuote",
-                "dynamic",
-                "selectApp",
-                "selectDataset"
-              ],
-              "showDescription": false,
-              "showDefaultValue": true
-            },
-            "debugLabel": "",
-            "toolDescription": ""
-          },
-          {
-            "key": "system_httpMethod",
-            "renderTypeList": ["custom"],
-            "valueType": "string",
-            "label": "",
-            "value": "POST",
-            "required": true,
-            "debugLabel": "",
-            "toolDescription": ""
-          },
-          {
-            "key": "system_httpTimeout",
-            "renderTypeList": ["custom"],
-            "valueType": "number",
-            "label": "",
-            "value": 30,
-            "min": 5,
-            "max": 600,
-            "required": true,
-            "debugLabel": "",
-            "toolDescription": ""
-          },
-          {
-            "key": "system_httpReqUrl",
-            "renderTypeList": ["hidden"],
-            "valueType": "string",
-            "label": "",
-            "description": "common:core.module.input.description.Http Request Url",
-            "placeholder": "https://api.ai.com/getInventory",
-            "required": false,
-            "value": "Doc2X/URLPDF2text",
-            "debugLabel": "",
-            "toolDescription": ""
-          },
-          {
-            "key": "system_httpHeader",
-            "renderTypeList": ["custom"],
-            "valueType": "any",
-            "value": [],
-            "label": "",
-            "description": "common:core.module.input.description.Http Request Header",
-            "placeholder": "common:core.module.input.description.Http Request Header",
-            "required": false,
-            "debugLabel": "",
-            "toolDescription": ""
-          },
-          {
-            "key": "system_httpParams",
-            "renderTypeList": ["hidden"],
-            "valueType": "any",
-            "value": [],
-            "label": "",
-            "required": false,
-            "debugLabel": "",
-            "toolDescription": ""
-          },
-          {
-            "key": "system_httpJsonBody",
-            "renderTypeList": ["hidden"],
-            "valueType": "any",
-            "value": "{\n  \"apikey\": \"{{apikey}}\",\n  \"url\": \"{{url}}\",\n  \"ocr\": {{ocr}}\n}",
-            "label": "",
-            "required": false,
-            "debugLabel": "",
-            "toolDescription": ""
-          },
-          {
-            "key": "system_httpFormBody",
-            "renderTypeList": ["hidden"],
-            "valueType": "any",
-            "value": [],
-            "label": "",
-            "required": false,
-            "debugLabel": "",
-            "toolDescription": ""
-          },
-          {
-            "key": "system_httpContentType",
-            "renderTypeList": ["hidden"],
-            "valueType": "string",
-            "value": "json",
-            "label": "",
-            "required": false,
-            "debugLabel": "",
-            "toolDescription": ""
-          },
-          {
-            "renderTypeList": ["reference"],
-            "valueType": "string",
-            "canEdit": true,
-            "key": "apikey",
-            "label": "apikey",
-            "customInputConfig": {
-              "selectValueTypeList": [
-                "string",
-                "number",
-                "boolean",
-                "object",
-                "arrayString",
-                "arrayNumber",
-                "arrayBoolean",
-                "arrayObject",
-                "arrayAny",
-                "any",
-                "chatHistory",
-                "datasetQuote",
-                "dynamic",
-                "selectApp",
-                "selectDataset"
-              ],
-              "showDescription": false,
-              "showDefaultValue": true
-            },
-            "required": true,
-            "value": ["pluginInput", "apikey"]
-          },
-          {
-            "renderTypeList": ["reference"],
-            "valueType": "string",
-            "canEdit": true,
-            "key": "url",
-            "label": "url",
-            "customInputConfig": {
-              "selectValueTypeList": [
-                "string",
-                "number",
-                "boolean",
-                "object",
-                "arrayString",
-                "arrayNumber",
-                "arrayBoolean",
-                "arrayObject",
-                "arrayAny",
-                "any",
-                "chatHistory",
-                "datasetQuote",
-                "dynamic",
-                "selectApp",
-                "selectDataset"
-              ],
-              "showDescription": false,
-              "showDefaultValue": true
-            },
-            "required": true,
-            "value": ["pluginInput", "url"]
-          },
-          {
-            "renderTypeList": ["reference"],
-            "valueType": "boolean",
-            "canEdit": true,
-            "key": "ocr",
-            "label": "ocr",
-            "customInputConfig": {
-              "selectValueTypeList": [
-                "string",
-                "number",
-                "boolean",
-                "object",
-                "arrayString",
-                "arrayNumber",
-                "arrayBoolean",
-                "arrayObject",
-                "arrayAny",
-                "any",
-                "chatHistory",
-                "datasetQuote",
-                "dynamic",
-                "selectApp",
-                "selectDataset"
-              ],
-              "showDescription": false,
-              "showDefaultValue": true
-            },
-            "required": true,
-            "value": ["pluginInput", "formula"]
-          }
-        ],
-        "outputs": [
-          {
-            "id": "error",
-            "key": "error",
-            "label": "workflow:request_error",
-            "description": "HTTP请求错误信息，成功时返回空",
-            "valueType": "object",
-            "type": "static"
-          },
-          {
-            "id": "httpRawResponse",
-            "key": "httpRawResponse",
-            "required": true,
-            "label": "workflow:raw_response",
-            "description": "HTTP请求的原始响应。只能接受字符串或JSON类型响应数据。",
-            "valueType": "any",
-            "type": "static"
-          },
-          {
-            "id": "system_addOutputParam",
-            "key": "system_addOutputParam",
-            "type": "dynamic",
-            "valueType": "dynamic",
-            "label": "",
-            "customFieldConfig": {
-              "selectValueTypeList": [
-                "string",
-                "number",
-                "boolean",
-                "object",
-                "arrayString",
-                "arrayNumber",
-                "arrayBoolean",
-                "arrayObject",
-                "any",
-                "chatHistory",
-                "datasetQuote",
-                "dynamic",
-                "selectApp",
-                "selectDataset"
-              ],
-              "showDescription": false,
-              "showDefaultValue": false
-            }
-          },
-          {
-            "id": "xWQuEf50F3mr",
-            "valueType": "string",
-            "type": "dynamic",
-            "key": "result",
-            "label": "result"
-          },
-          {
-            "id": "m6CJJj7GFud5",
-            "valueType": "boolean",
-            "type": "dynamic",
-            "key": "success",
-            "label": "success"
-          }
-        ]
-      },
-      {
-        "nodeId": "rZmLfANEyyJe",
-        "name": "系统配置",
-        "intro": "",
-        "avatar": "core/workflow/template/systemConfig",
-        "flowNodeType": "pluginConfig",
-        "position": {
-          "x": -93.55061402342784,
-          "y": -55.907069101622824
-        },
-        "version": "4811",
-        "inputs": [],
-        "outputs": []
-      }
-    ],
-    "edges": [
-      {
-        "source": "pluginInput",
-        "target": "zHG5jJBkXmjB",
-        "sourceHandle": "pluginInput-source-right",
-        "targetHandle": "zHG5jJBkXmjB-target-left"
-      },
-      {
-        "source": "zHG5jJBkXmjB",
-        "target": "pluginOutput",
-        "sourceHandle": "zHG5jJBkXmjB-source-right",
-        "targetHandle": "pluginOutput-target-left"
-      }
-    ]
-  }
-}
--- a/packages/service/common/file/gridfs/controller.ts
+++ b/packages/service/common/file/gridfs/controller.ts
@@ -36,6 +36,7 @@ export async function uploadFile({
  path,
  filename,
  contentType,
+  encoding,
  metadata = {}
 }: {
  bucketName: `${BucketNameEnum}`;
@@ -44,6 +45,7 @@ export async function uploadFile({
  path: string;
  filename: string;
  contentType?: string;
+  encoding: string;
  metadata?: Record<string, any>;
 }) {
  if (!path) return Promise.reject(`filePath is empty`);
@@ -52,7 +54,7 @@ export async function uploadFile({
  const stats = await fsp.stat(path);
  if (!stats.isFile()) return Promise.reject(`${path} is not a file`);

-  const { stream: readStream, encoding } = await stream2Encoding(fs.createReadStream(path));
+  const readStream = fs.createReadStream(path);

  // Add default metadata
  metadata.teamId = teamId;
--- a/packages/service/common/file/read/utils.ts
+++ b/packages/service/common/file/read/utils.ts
@@ -4,16 +4,17 @@ import FormData from 'form-data';

 import { WorkerNameEnum, runWorker } from '../../../worker/utils';
 import fs from 'fs';
-import { detectFileEncoding } from '@fastgpt/global/common/file/tools';
 import type { ReadFileResponse } from '../../../worker/readFile/type';
 import axios from 'axios';
 import { addLog } from '../../system/log';
 import { batchRun } from '@fastgpt/global/common/fn/utils';
 import { addHours } from 'date-fns';
+import { matchMdImgTextAndUpload } from '@fastgpt/global/common/string/markdown';

 export type readRawTextByLocalFileParams = {
  teamId: string;
  path: string;
+  encoding: string;
  metadata?: Record<string, any>;
 };
 export const readRawTextByLocalFile = async (params: readRawTextByLocalFileParams) => {
@@ -22,13 +23,12 @@ export const readRawTextByLocalFile = async (params: readRawTextByLocalFileParam
  const extension = path?.split('.')?.pop()?.toLowerCase() || '';

  const buffer = fs.readFileSync(path);
-  const encoding = detectFileEncoding(buffer);

  const { rawText } = await readRawContentByFileBuffer({
    extension,
    isQAImport: false,
    teamId: params.teamId,
-    encoding,
+    encoding: params.encoding,
    buffer,
    metadata: params.metadata
  });
@@ -53,6 +53,7 @@ export const readRawContentByFileBuffer = async ({
  encoding: string;
  metadata?: Record<string, any>;
 }) => {
+  // Custom read file service
  const customReadfileUrl = process.env.CUSTOM_READ_FILE_URL;
  const customReadFileExtension = process.env.CUSTOM_READ_FILE_EXTENSION || '';
  const ocrParse = process.env.CUSTOM_READ_FILE_OCR || 'false';
@@ -78,6 +79,7 @@ export const readRawContentByFileBuffer = async ({
      data: {
        page: number;
        markdown: string;
+        duration: number;
      };
    }>(customReadfileUrl, data, {
      timeout: 600000,
@@ -89,10 +91,12 @@ export const readRawContentByFileBuffer = async ({
    addLog.info(`Use custom read file service, time: ${Date.now() - start}ms`);

    const rawText = response.data.markdown;
+    const { text, imageList } = matchMdImgTextAndUpload(rawText);

    return {
-      rawText,
-      formatText: rawText
+      rawText: text,
+      formatText: rawText,
+      imageList
    };
  };

@@ -119,6 +123,9 @@ export const readRawContentByFileBuffer = async ({
        }
      });
      rawText = rawText.replace(item.uuid, src);
+      if (formatText) {
+        formatText = formatText.replace(item.uuid, src);
+      }
    });
  }

@@ -127,7 +134,7 @@ export const readRawContentByFileBuffer = async ({
    if (isQAImport) {
      rawText = rawText || '';
    } else {
-      rawText = formatText || '';
+      rawText = formatText || rawText;
    }
  }

--- a/packages/service/core/chat/utils.ts
+++ b/packages/service/core/chat/utils.ts
@@ -109,7 +109,7 @@ export const loadRequestMessages = async ({
    }
    return Promise.all(
      messages.map(async (item) => {
-        if (item.type === 'image_url' && process.env.MULTIPLE_DATA_TO_BASE64 === 'true') {
+        if (item.type === 'image_url') {
          // Remove url origin
          const imgUrl = (() => {
            if (origin && item.image_url.url.startsWith(origin)) {
--- a/packages/service/core/dataset/collection/schema.ts
+++ b/packages/service/core/dataset/collection/schema.ts
@@ -118,7 +118,7 @@ try {
    {
      unique: true,
      partialFilterExpression: {
-        externalFileId: { $exists: true }
+        externalFileId: { $exists: true, $ne: '' }
      }
    }
  );
--- a/packages/service/core/dataset/training/schema.ts
+++ b/packages/service/core/dataset/training/schema.ts
@@ -51,6 +51,11 @@ const TrainingDataSchema = new Schema({
    type: Date,
    default: () => new Date('2000/1/1')
  },
+  retryCount: {
+    type: Number,
+    default: 5
+  },
+
  model: {
    // ai model
    type: String,
@@ -97,7 +102,7 @@ try {
  // lock training data(teamId); delete training data
  TrainingDataSchema.index({ teamId: 1, datasetId: 1 });
  // get training data and sort
-  TrainingDataSchema.index({ mode: 1, lockTime: 1, weight: -1 });
+  TrainingDataSchema.index({ mode: 1, retryCount: 1, lockTime: 1, weight: -1 });
  TrainingDataSchema.index({ expireAt: 1 }, { expireAfterSeconds: 7 * 24 * 60 * 60 }); // 7 days
 } catch (error) {
  console.log(error);
--- a/packages/service/core/workflow/dispatch/chat/oneapi.ts
+++ b/packages/service/core/workflow/dispatch/chat/oneapi.ts
@@ -256,11 +256,6 @@ export const dispatchChatCompletion = async (props: ChatProps): Promise<ChatResp
      history: chatCompleteMessages
    };
  } catch (error) {
-    addLog.warn(`LLM response error`, {
-      baseUrl: user.openaiAccount?.baseUrl,
-      requestBody
-    });
-
    if (user.openaiAccount?.baseUrl) {
      return Promise.reject(`您的 OpenAI key 出错了: ${getErrText(error)}`);
    }
--- a/packages/service/core/workflow/dispatch/tools/http468.ts
+++ b/packages/service/core/workflow/dispatch/tools/http468.ts
@@ -18,7 +18,6 @@ import {
  textAdaptGptResponse,
  replaceEditorVariable
 } from '@fastgpt/global/core/workflow/runtime/utils';
-import { getSystemPluginCb } from '../../../../../plugins/register';
 import { ContentTypes } from '@fastgpt/global/core/workflow/constants';
 import { uploadFileFromBase64Img } from '../../../../common/file/gridfs/controller';
 import { ReadFileBaseUrl } from '@fastgpt/global/common/file/constants';
@@ -209,7 +208,8 @@ export const dispatchHttp468Request = async (props: HttpRequestProps): Promise<H

  try {
    const { formatResponse, rawResponse } = await (async () => {
-      const systemPluginCb = await getSystemPluginCb();
+      const systemPluginCb = global.systemPluginCb;
+      console.log(systemPluginCb, '-=', httpReqUrl);
      if (systemPluginCb[httpReqUrl]) {
        const pluginResult = await replaceSystemPluginResponse({
          response: await systemPluginCb[httpReqUrl](requestBody),
--- a/packages/service/worker/htmlStr2Md/utils.ts
+++ b/packages/service/worker/htmlStr2Md/utils.ts
@@ -1,5 +1,6 @@
 import TurndownService from 'turndown';
 import { ImageType } from '../readFile/type';
+import { matchMdImgTextAndUpload } from '@fastgpt/global/common/string/markdown';
 // @ts-ignore
 const turndownPluginGfm = require('joplin-turndown-plugin-gfm');

@@ -24,23 +25,10 @@ export const html2md = (
    turndownService.remove(['i', 'script', 'iframe', 'style']);
    turndownService.use(turndownPluginGfm.gfm);

-    const base64Regex = /"(data:image\/[^;]+;base64[^"]+)"/g;
-    const imageList: ImageType[] = [];
-    const images = Array.from(html.match(base64Regex) || []);
-    for (const image of images) {
-      const uuid = crypto.randomUUID();
-      const mime = image.split(';')[0].split(':')[1];
-      const base64 = image.split(',')[1];
-      html = html.replace(image, uuid);
-      imageList.push({
-        uuid,
-        base64,
-        mime
-      });
-    }
+    const { text, imageList } = matchMdImgTextAndUpload(html);

    return {
-      rawText: turndownService.turndown(html),
+      rawText: turndownService.turndown(text),
      imageList
    };
  } catch (error) {
--- a/packages/service/worker/readFile/extension/rawText.ts
+++ b/packages/service/worker/readFile/extension/rawText.ts
@@ -18,9 +18,17 @@ const rawEncodingList = [

 // 加载源文件内容
 export const readFileRawText = ({ buffer, encoding }: ReadRawTextByBuffer): ReadFileResponse => {
-  const content = rawEncodingList.includes(encoding)
-    ? buffer.toString(encoding as BufferEncoding)
-    : iconv.decode(buffer, 'gbk');
+  const content = (() => {
+    try {
+      if (rawEncodingList.includes(encoding)) {
+        return buffer.toString(encoding as BufferEncoding);
+      }
+
+      return iconv.decode(buffer, encoding);
+    } catch (error) {
+      return buffer.toString('utf-8');
+    }
+  })();

  return {
    rawText: content
--- a/packages/web/components/common/Icon/constants.ts
+++ b/packages/web/components/common/Icon/constants.ts
@@ -50,6 +50,7 @@ export const iconPaths = {
  'common/list': () => import('./icons/common/list.svg'),
  'common/loading': () => import('./icons/common/loading.svg'),
  'common/logLight': () => import('./icons/common/logLight.svg'),
+  'common/microsoft': () => import('./icons/common/microsoft.svg'),
  'common/monitor': () => import('./icons/common/monitor.svg'),
  'common/navbar/pluginFill': () => import('./icons/common/navbar/pluginFill.svg'),
  'common/navbar/pluginLight': () => import('./icons/common/navbar/pluginLight.svg'),
--- a/packages/web/components/common/Icon/icons/common/microsoft.svg
+++ b/packages/web/components/common/Icon/icons/common/microsoft.svg
@@ -0,0 +1 @@
+<svg t="1731513229844" class="icon" viewBox="0 0 1024 1024" version="1.1" xmlns="http://www.w3.org/2000/svg" p-id="5087" width="200" height="200"><path d="M493.0048 492.9536H128.0512V128H493.056v364.9536z" fill="#F1511B" p-id="5088"></path><path d="M895.9488 492.9536H530.944V128H896v364.9536z" fill="#80CC28" p-id="5089"></path><path d="M493.0048 896H128v-364.9024H493.056V896z" fill="#00ADEF" p-id="5090"></path><path d="M895.8976 896h-364.9024v-364.9024h364.9024V896z" fill="#FBBC09" p-id="5091"></path></svg>
--- a/packages/web/i18n/en/common.json
+++ b/packages/web/i18n/en/common.json
@@ -997,6 +997,7 @@
  "support.user.login.Email": "Email",
  "support.user.login.Github": "GitHub Login",
  "support.user.login.Google": "Google Login",
+  "support.user.login.Microsoft": "Microsoft Login",
  "support.user.login.Password": "Password",
  "support.user.login.Password login": "Password Login",
  "support.user.login.Phone": "Phone Login",
--- a/packages/web/i18n/zh/common.json
+++ b/packages/web/i18n/zh/common.json
@@ -18,9 +18,6 @@
  "FAQ.switch_package_a": "套餐使用规则为优先使用更高级的套餐，因此，购买的新套餐若比当前套餐更高级，则新套餐立即生效：否则将继续使用当前套餐。",
  "FAQ.switch_package_q": "是否切换订阅套餐？",
  "Folder": "文件夹",
-  "just_now": "刚刚",
-  "yesterday": "昨天",
-  "yesterday_detail_time": "昨天 {{time}}",
  "Login": "登录",
  "Move": "移动",
  "Name": "名称",
@@ -125,6 +122,7 @@
  "common.Documents": "文档",
  "common.Done": "完成",
  "common.Edit": "编辑",
+  "common.Error": "错误",
  "common.Exit": "退出",
  "common.Exit Directly": "直接退出",
  "common.Expired Time": "过期时间",
@@ -194,7 +192,6 @@
  "common.Update Successful": "更新成功",
  "common.Username": "用户名",
  "common.Waiting": "等待中",
-  "common.Error": "错误",
  "common.Warning": "警告",
  "common.Website": "网站",
  "common.all_result": "完整结果",
@@ -551,6 +548,7 @@
  "core.dataset.import.Chunk Range": "范围：{{min}}~{{max}}",
  "core.dataset.import.Chunk Split": "直接分段",
  "core.dataset.import.Chunk Split Tip": "将文本按一定的规则进行分段处理后，转成可进行语义搜索的格式，适合绝大多数场景。不需要调用模型额外处理，成本低。",
+  "core.dataset.import.Continue upload": "继续上传",
  "core.dataset.import.Custom process": "自定义规则",
  "core.dataset.import.Custom process desc": "自定义设置数据处理规则",
  "core.dataset.import.Custom prompt": "自定义提示词",
@@ -579,11 +577,10 @@
  "core.dataset.import.Select source": "选择来源",
  "core.dataset.import.Source name": "来源名",
  "core.dataset.import.Sources list": "来源列表",
-  "core.dataset.import.Continue upload": "继续上传",
-  "core.dataset.import.Upload complete": "完成上传",
  "core.dataset.import.Start upload": "开始上传",
  "core.dataset.import.Total files": "共 {{total}} 个文件",
  "core.dataset.import.Training mode": "训练模式",
+  "core.dataset.import.Upload complete": "完成上传",
  "core.dataset.import.Upload data": "确认上传",
  "core.dataset.import.Upload file progress": "文件上传进度",
  "core.dataset.import.Upload status": "状态",
@@ -894,6 +891,7 @@
  "is_using": "正在使用",
  "item_description": "字段描述",
  "item_name": "字段名",
+  "just_now": "刚刚",
  "key_repetition": "key 重复",
  "move.confirm": "确认移动",
  "navbar.Account": "账号",
@@ -998,6 +996,7 @@
  "support.user.login.Email": "邮箱",
  "support.user.login.Github": "GitHub 登录",
  "support.user.login.Google": "Google 登录",
+  "support.user.login.Microsoft": "微软登录",
  "support.user.login.Password": "密码",
  "support.user.login.Password login": "密码登录",
  "support.user.login.Phone": "手机号登录",
@@ -1214,5 +1213,7 @@
  "user.type": "类型",
  "verification": "验证",
  "xx_search_result": "{{key}} 的搜索结果",
-  "yes": "是"
+  "yes": "是",
+  "yesterday": "昨天",
+  "yesterday_detail_time": "昨天 {{time}}"
 }
--- a/pnpm-lock.yaml
+++ b/pnpm-lock.yaml
@@ -22,7 +22,7 @@ importers:
        version: 13.3.0
      next-i18next:
        specifier: 15.3.0
-        version: 15.3.0(i18next@23.11.5)(next@14.2.5(@babel/core@7.24.9)(babel-plugin-macros@3.1.0)(react-dom@18.3.1(react@18.3.1))(react@18.3.1)(sass@1.77.8))(react-i18next@14.1.2(i18next@23.11.5)(react-dom@18.3.1(react@18.3.1))(react@18.3.1))(react@18.3.1)
+        version: 15.3.0(i18next@23.11.5)(next@14.2.5(react-dom@18.3.1(react@18.3.1))(react@18.3.1)(sass@1.77.8))(react-i18next@14.1.2(i18next@23.11.5)(react-dom@18.3.1(react@18.3.1))(react@18.3.1))(react@18.3.1)
      prettier:
        specifier: 3.2.4
        version: 3.2.4
@@ -61,7 +61,7 @@ importers:
        version: 4.0.2
      next:
        specifier: 14.2.5
-        version: 14.2.5(@babel/core@7.24.9)(babel-plugin-macros@3.1.0)(react-dom@18.3.1(react@18.3.1))(react@18.3.1)(sass@1.77.8)
+        version: 14.2.5(react-dom@18.3.1(react@18.3.1))(react@18.3.1)(sass@1.77.8)
      openai:
        specifier: 4.61.0
        version: 4.61.0(encoding@0.1.13)
@@ -201,7 +201,7 @@ importers:
        version: 1.4.5-lts.1
      next:
        specifier: 14.2.5
-        version: 14.2.5(@babel/core@7.24.9)(babel-plugin-macros@3.1.0)(react-dom@18.3.1(react@18.3.1))(react@18.3.1)(sass@1.77.8)
+        version: 14.2.5(react-dom@18.3.1(react@18.3.1))(react@18.3.1)(sass@1.77.8)
      nextjs-cors:
        specifier: ^2.2.0
        version: 2.2.0(next@14.2.5(react-dom@18.3.1(react@18.3.1))(react@18.3.1)(sass@1.77.8))
@@ -277,7 +277,7 @@ importers:
        version: 2.1.1(@chakra-ui/system@2.6.1(@emotion/react@11.11.1(@types/react@18.3.1)(react@18.3.1))(@emotion/styled@11.11.0(@emotion/react@11.11.1(@types/react@18.3.1)(react@18.3.1))(@types/react@18.3.1)(react@18.3.1))(react@18.3.1))(react@18.3.1)
      '@chakra-ui/next-js':
        specifier: 2.1.5
-        version: 2.1.5(@chakra-ui/react@2.8.1(@emotion/react@11.11.1(@types/react@18.3.1)(react@18.3.1))(@emotion/styled@11.11.0(@emotion/react@11.11.1(@types/react@18.3.1)(react@18.3.1))(@types/react@18.3.1)(react@18.3.1))(@types/react@18.3.1)(framer-motion@9.1.7(react-dom@18.3.1(react@18.3.1))(react@18.3.1))(react-dom@18.3.1(react@18.3.1))(react@18.3.1))(@emotion/react@11.11.1(@types/react@18.3.1)(react@18.3.1))(next@14.2.5(@babel/core@7.24.9)(babel-plugin-macros@3.1.0)(react-dom@18.3.1(react@18.3.1))(react@18.3.1)(sass@1.77.8))(react@18.3.1)
+        version: 2.1.5(@chakra-ui/react@2.8.1(@emotion/react@11.11.1(@types/react@18.3.1)(react@18.3.1))(@emotion/styled@11.11.0(@emotion/react@11.11.1(@types/react@18.3.1)(react@18.3.1))(@types/react@18.3.1)(react@18.3.1))(@types/react@18.3.1)(framer-motion@9.1.7(react-dom@18.3.1(react@18.3.1))(react@18.3.1))(react-dom@18.3.1(react@18.3.1))(react@18.3.1))(@emotion/react@11.11.1(@types/react@18.3.1)(react@18.3.1))(next@14.2.5(react-dom@18.3.1(react@18.3.1))(react@18.3.1)(sass@1.77.8))(react@18.3.1)
      '@chakra-ui/react':
        specifier: 2.8.1
        version: 2.8.1(@emotion/react@11.11.1(@types/react@18.3.1)(react@18.3.1))(@emotion/styled@11.11.0(@emotion/react@11.11.1(@types/react@18.3.1)(react@18.3.1))(@types/react@18.3.1)(react@18.3.1))(@types/react@18.3.1)(framer-motion@9.1.7(react-dom@18.3.1(react@18.3.1))(react@18.3.1))(react-dom@18.3.1(react@18.3.1))(react@18.3.1)
@@ -340,7 +340,7 @@ importers:
        version: 4.17.21
      next-i18next:
        specifier: 15.3.0
-        version: 15.3.0(i18next@23.11.5)(next@14.2.5(@babel/core@7.24.9)(babel-plugin-macros@3.1.0)(react-dom@18.3.1(react@18.3.1))(react@18.3.1)(sass@1.77.8))(react-i18next@14.1.2(i18next@23.11.5)(react-dom@18.3.1(react@18.3.1))(react@18.3.1))(react@18.3.1)
+        version: 15.3.0(i18next@23.11.5)(next@14.2.5(react-dom@18.3.1(react@18.3.1))(react@18.3.1)(sass@1.77.8))(react-i18next@14.1.2(i18next@23.11.5)(react-dom@18.3.1(react@18.3.1))(react@18.3.1))(react@18.3.1)
      papaparse:
        specifier: ^5.4.1
        version: 5.4.1
@@ -486,6 +486,9 @@ importers:
      json5:
        specifier: ^2.2.3
        version: 2.2.3
+      jsondiffpatch:
+        specifier: ^0.6.0
+        version: 0.6.0
      jsonwebtoken:
        specifier: ^9.0.2
        version: 9.0.2
@@ -700,7 +703,7 @@ importers:
        version: 6.3.4
      ts-jest:
        specifier: ^29.1.0
-        version: 29.2.2(@babel/core@7.24.9)(@jest/transform@29.7.0)(@jest/types@29.6.3)(babel-jest@29.7.0(@babel/core@7.24.9))(jest@29.7.0(@types/node@20.14.11)(babel-plugin-macros@3.1.0))(typescript@5.5.3)
+        version: 29.2.2(@babel/core@7.24.9)(@jest/transform@29.7.0)(@jest/types@29.6.3)(babel-jest@29.7.0(@babel/core@7.24.9))(jest@29.7.0(@types/node@20.14.11)(babel-plugin-macros@3.1.0)(ts-node@10.9.2(@types/node@20.14.11)(typescript@5.5.3)))(typescript@5.5.3)
      ts-loader:
        specifier: ^9.4.3
        version: 9.5.1(typescript@5.5.3)(webpack@5.92.1)
@@ -3177,8 +3180,8 @@ packages:
  '@tanstack/react-query@4.36.1':
    resolution: {integrity: sha512-y7ySVHFyyQblPl3J3eQBWpXZkliroki3ARnBKsdJchlgt7yJLRDUcf4B8soufgiYt3pEQIkBWBx1N9/ZPIeUWw==}
    peerDependencies:
-      react: ^16.8.0 || ^17.0.0 || ^18.0.0
-      react-dom: ^16.8.0 || ^17.0.0 || ^18.0.0
+      react: 18.3.1
+      react-dom: 18.3.1
      react-native: '*'
    peerDependenciesMeta:
      react-dom:
@@ -3331,6 +3334,9 @@ packages:
  '@types/decompress@4.2.7':
    resolution: {integrity: sha512-9z+8yjKr5Wn73Pt17/ldnmQToaFHZxK0N1GHysuk/JIPT8RIdQeoInM01wWPgypRcvb6VH1drjuFpQ4zmY437g==}

+  '@types/diff-match-patch@1.0.36':
+    resolution: {integrity: sha512-xFdR6tkm0MWvBfO8xXCSsinYxHcqkQUlcHeSpMC2ukzOb6lwQAfDmW+Qt0AvlGd8HpsS28qKsB+oPeJn9I39jg==}
+
  '@types/eslint-scope@3.7.7':
    resolution: {integrity: sha512-MzMFlSLBqNF2gcHWO0G1vP/YQyfvrxZ0bF+u7mzUdZ1/xK4A4sru+nraZz5i3iEIk1l1uyicaDVTB4QbbEkAYg==}

@@ -4848,6 +4854,9 @@ packages:
  dezalgo@1.0.4:
    resolution: {integrity: sha512-rXSP0bf+5n0Qonsb+SVVfNfIsimO4HEtmnIpPHY8Q1UCzKlQrDMfdobr8nJOOsRgWCyMRqeSBQzmWUMq7zvVig==}

+  diff-match-patch@1.0.5:
+    resolution: {integrity: sha512-IayShXAgj/QMXgB0IWmKx+rOPuGMhqm5w6jvFxmVenXKIzRqTAAsbBPT3kWQeGANj3jGgvcvv4yK6SxqYmikgw==}
+
  diff-sequences@29.6.3:
    resolution: {integrity: sha512-EjePK1srD3P08o2j4f0ExnylqRs5B9tJjcp9t1krH2qRi8CCdsYfwe9JgSLurFBWwq4uOlipzfk5fHNvwFKr8Q==}
    engines: {node: ^14.15.0 || ^16.10.0 || >=18.0.0}
@@ -5139,6 +5148,7 @@ packages:
  eslint@8.56.0:
    resolution: {integrity: sha512-Go19xM6T9puCOWntie1/P997aXxFsOi37JIHRWI514Hc6ZnaHGKY9xFhrU65RT6CcBEzZoGG1e6Nq+DT04ZtZQ==}
    engines: {node: ^12.22.0 || ^14.17.0 || >=16.0.0}
+    deprecated: This version is no longer supported. Please see https://eslint.org/version-support for other options.
    hasBin: true

  espree@9.6.1:
@@ -6308,6 +6318,11 @@ packages:
  jsonc-parser@3.3.1:
    resolution: {integrity: sha512-HUgH65KyejrUFPvHFPbqOY0rsFip3Bo5wb4ngvdi1EpCYWUQDC5V+Y7mZws+DLkr4M//zQJoanu1SP+87Dv1oQ==}

+  jsondiffpatch@0.6.0:
+    resolution: {integrity: sha512-3QItJOXp2AP1uv7waBkao5nCvhEv+QmJAd38Ybq7wNI74Q+BBmnLn4EDKz6yI9xGAIQoUF87qHt+kc1IVxB4zQ==}
+    engines: {node: ^18.0.0 || >=20.0.0}
+    hasBin: true
+
  jsonfile@6.1.0:
    resolution: {integrity: sha512-5dgndWOriYSm5cnYaJNhalLNDKOqFwyDB/rr1E9ZsGciGvKPs8R2xYGCacuf3z6K1YKDz182fd+fY3cn3pMqXQ==}

@@ -10462,6 +10477,14 @@ snapshots:
      next: 14.2.5(@babel/core@7.24.9)(babel-plugin-macros@3.1.0)(react-dom@18.3.1(react@18.3.1))(react@18.3.1)(sass@1.77.8)
      react: 18.3.1

+  '@chakra-ui/next-js@2.1.5(@chakra-ui/react@2.8.1(@emotion/react@11.11.1(@types/react@18.3.1)(react@18.3.1))(@emotion/styled@11.11.0(@emotion/react@11.11.1(@types/react@18.3.1)(react@18.3.1))(@types/react@18.3.1)(react@18.3.1))(@types/react@18.3.1)(framer-motion@9.1.7(react-dom@18.3.1(react@18.3.1))(react@18.3.1))(react-dom@18.3.1(react@18.3.1))(react@18.3.1))(@emotion/react@11.11.1(@types/react@18.3.1)(react@18.3.1))(next@14.2.5(react-dom@18.3.1(react@18.3.1))(react@18.3.1)(sass@1.77.8))(react@18.3.1)':
+    dependencies:
+      '@chakra-ui/react': 2.8.1(@emotion/react@11.11.1(@types/react@18.3.1)(react@18.3.1))(@emotion/styled@11.11.0(@emotion/react@11.11.1(@types/react@18.3.1)(react@18.3.1))(@types/react@18.3.1)(react@18.3.1))(@types/react@18.3.1)(framer-motion@9.1.7(react-dom@18.3.1(react@18.3.1))(react@18.3.1))(react-dom@18.3.1(react@18.3.1))(react@18.3.1)
+      '@emotion/cache': 11.11.0
+      '@emotion/react': 11.11.1(@types/react@18.3.1)(react@18.3.1)
+      next: 14.2.5(react-dom@18.3.1(react@18.3.1))(react@18.3.1)(sass@1.77.8)
+      react: 18.3.1
+
  '@chakra-ui/number-input@2.1.1(@chakra-ui/system@2.6.1(@emotion/react@11.11.1(@types/react@18.3.1)(react@18.3.1))(@emotion/styled@11.11.0(@emotion/react@11.11.1(@types/react@18.3.1)(react@18.3.1))(@types/react@18.3.1)(react@18.3.1))(react@18.3.1))(react@18.3.1)':
    dependencies:
      '@chakra-ui/counter': 2.1.0(react@18.3.1)
@@ -12393,6 +12416,8 @@ snapshots:
    dependencies:
      '@types/node': 22.7.8

+  '@types/diff-match-patch@1.0.36': {}
+
  '@types/eslint-scope@3.7.7':
    dependencies:
      '@types/eslint': 8.56.10
@@ -13203,7 +13228,7 @@ snapshots:

  axios@1.7.7:
    dependencies:
-      follow-redirects: 1.15.9(debug@4.3.7)
+      follow-redirects: 1.15.9
      form-data: 4.0.1
      proxy-from-env: 1.1.0
    transitivePeerDependencies:
@@ -14182,6 +14207,8 @@ snapshots:
      asap: 2.0.6
      wrappy: 1.0.2

+  diff-match-patch@1.0.5: {}
+
  diff-sequences@29.6.3: {}

  diff@4.0.2: {}
@@ -14982,6 +15009,8 @@ snapshots:

  follow-redirects@1.15.6: {}

+  follow-redirects@1.15.9: {}
+
  follow-redirects@1.15.9(debug@4.3.4):
    optionalDependencies:
      debug: 4.3.4
@@ -15044,7 +15073,7 @@ snapshots:
    dependencies:
      react: 18.3.1
      react-dom: 18.3.1(react@18.3.1)
-      tslib: 2.7.0
+      tslib: 2.8.0
    optionalDependencies:
      '@emotion/is-prop-valid': 0.8.8

@@ -16141,6 +16170,12 @@ snapshots:

  jsonc-parser@3.3.1: {}

+  jsondiffpatch@0.6.0:
+    dependencies:
+      '@types/diff-match-patch': 1.0.36
+      chalk: 5.3.0
+      diff-match-patch: 1.0.5
+
  jsonfile@6.1.0:
    dependencies:
      universalify: 2.0.1
@@ -17323,6 +17358,18 @@ snapshots:
      react: 18.3.1
      react-i18next: 14.1.2(i18next@23.11.5)(react-dom@18.3.1(react@18.3.1))(react@18.3.1)

+  next-i18next@15.3.0(i18next@23.11.5)(next@14.2.5(react-dom@18.3.1(react@18.3.1))(react@18.3.1)(sass@1.77.8))(react-i18next@14.1.2(i18next@23.11.5)(react-dom@18.3.1(react@18.3.1))(react@18.3.1))(react@18.3.1):
+    dependencies:
+      '@babel/runtime': 7.24.8
+      '@types/hoist-non-react-statics': 3.3.5
+      core-js: 3.37.1
+      hoist-non-react-statics: 3.3.2
+      i18next: 23.11.5
+      i18next-fs-backend: 2.3.1
+      next: 14.2.5(react-dom@18.3.1(react@18.3.1))(react@18.3.1)(sass@1.77.8)
+      react: 18.3.1
+      react-i18next: 14.1.2(i18next@23.11.5)(react-dom@18.3.1(react@18.3.1))(react@18.3.1)
+
  next@14.2.5(@babel/core@7.24.9)(babel-plugin-macros@3.1.0)(react-dom@18.3.1(react@18.3.1))(react@18.3.1)(sass@1.77.8):
    dependencies:
      '@next/env': 14.2.5
@@ -17349,10 +17396,36 @@ snapshots:
      - '@babel/core'
      - babel-plugin-macros

+  next@14.2.5(react-dom@18.3.1(react@18.3.1))(react@18.3.1)(sass@1.77.8):
+    dependencies:
+      '@next/env': 14.2.5
+      '@swc/helpers': 0.5.5
+      busboy: 1.6.0
+      caniuse-lite: 1.0.30001669
+      graceful-fs: 4.2.11
+      postcss: 8.4.31
+      react: 18.3.1
+      react-dom: 18.3.1(react@18.3.1)
+      styled-jsx: 5.1.1(react@18.3.1)
+    optionalDependencies:
+      '@next/swc-darwin-arm64': 14.2.5
+      '@next/swc-darwin-x64': 14.2.5
+      '@next/swc-linux-arm64-gnu': 14.2.5
+      '@next/swc-linux-arm64-musl': 14.2.5
+      '@next/swc-linux-x64-gnu': 14.2.5
+      '@next/swc-linux-x64-musl': 14.2.5
+      '@next/swc-win32-arm64-msvc': 14.2.5
+      '@next/swc-win32-ia32-msvc': 14.2.5
+      '@next/swc-win32-x64-msvc': 14.2.5
+      sass: 1.77.8
+    transitivePeerDependencies:
+      - '@babel/core'
+      - babel-plugin-macros
+
  nextjs-cors@2.2.0(next@14.2.5(react-dom@18.3.1(react@18.3.1))(react@18.3.1)(sass@1.77.8)):
    dependencies:
      cors: 2.8.5
-      next: 14.2.5(@babel/core@7.24.9)(babel-plugin-macros@3.1.0)(react-dom@18.3.1(react@18.3.1))(react@18.3.1)(sass@1.77.8)
+      next: 14.2.5(react-dom@18.3.1(react@18.3.1))(react@18.3.1)(sass@1.77.8)

  nextjs-node-loader@1.1.5(webpack@5.92.1):
    dependencies:
@@ -18410,7 +18483,7 @@ snapshots:
    dependencies:
      chokidar: 3.6.0
      immutable: 4.3.6
-      source-map-js: 1.2.0
+      source-map-js: 1.2.1

  sax@1.4.1: {}

@@ -18755,6 +18828,11 @@ snapshots:
      '@babel/core': 7.24.9
      babel-plugin-macros: 3.1.0

+  styled-jsx@5.1.1(react@18.3.1):
+    dependencies:
+      client-only: 0.0.1
+      react: 18.3.1
+
  stylis@4.2.0: {}

  stylis@4.3.2: {}
@@ -18956,6 +19034,25 @@ snapshots:

  ts-dedent@2.2.0: {}

+  ts-jest@29.2.2(@babel/core@7.24.9)(@jest/transform@29.7.0)(@jest/types@29.6.3)(babel-jest@29.7.0(@babel/core@7.24.9))(jest@29.7.0(@types/node@20.14.11)(babel-plugin-macros@3.1.0)(ts-node@10.9.2(@types/node@20.14.11)(typescript@5.5.3)))(typescript@5.5.3):
+    dependencies:
+      bs-logger: 0.2.6
+      ejs: 3.1.10
+      fast-json-stable-stringify: 2.1.0
+      jest: 29.7.0(@types/node@20.14.11)(babel-plugin-macros@3.1.0)(ts-node@10.9.2(@types/node@20.14.11)(typescript@5.5.3))
+      jest-util: 29.7.0
+      json5: 2.2.3
+      lodash.memoize: 4.1.2
+      make-error: 1.3.6
+      semver: 7.6.3
+      typescript: 5.5.3
+      yargs-parser: 21.1.1
+    optionalDependencies:
+      '@babel/core': 7.24.9
+      '@jest/transform': 29.7.0
+      '@jest/types': 29.6.3
+      babel-jest: 29.7.0(@babel/core@7.24.9)
+
  ts-jest@29.2.2(@babel/core@7.24.9)(@jest/transform@29.7.0)(@jest/types@29.6.3)(babel-jest@29.7.0(@babel/core@7.24.9))(jest@29.7.0(@types/node@20.14.11)(babel-plugin-macros@3.1.0))(typescript@5.5.3):
    dependencies:
      bs-logger: 0.2.6
--- a/projects/app/.env.template
+++ b/projects/app/.env.template
@@ -33,7 +33,7 @@ MILVUS_TOKEN=133964348b00b4b4e4b51bef680a61350950385c8c64a3ec16b1ab92d3c67dcc4e0
 SANDBOX_URL=http://localhost:3001
 # 商业版地址
 PRO_URL=
-# 页面的地址，用于自动补全相对路径资源的 domain
+# 页面的地址，用于自动补全相对路径资源的 domain，注意后面不要跟 /
 FE_DOMAIN=http://localhost:3000
 # 二级路由，需要打包时候就确定
 # NEXT_PUBLIC_BASE_URL=/fastai
--- a/projects/app/package.json
+++ b/projects/app/package.json
@@ -42,6 +42,7 @@
    "jest": "^29.5.0",
    "js-yaml": "^4.1.0",
    "json5": "^2.2.3",
+    "jsondiffpatch": "^0.6.0",
    "jsonwebtoken": "^9.0.2",
    "lodash": "^4.17.21",
    "mermaid": "^10.2.3",
--- a/projects/app/src/components/core/chat/ChatContainer/ChatBox/index.tsx
+++ b/projects/app/src/components/core/chat/ChatContainer/ChatBox/index.tsx
@@ -323,7 +323,7 @@ const ChatBox = (
              })
            };
          } else if (event === SseResponseEventEnum.updateVariables && variables) {
-            variablesForm.reset(variables);
+            variablesForm.setValue('variables', variables);
          } else if (event === SseResponseEventEnum.interactive) {
            const val: AIChatItemValueItemType = {
              type: ChatItemValueTypeEnum.interactive,
@@ -408,7 +408,7 @@ const ChatBox = (
      isInteractivePrompt = false
    }) => {
      variablesForm.handleSubmit(
-        async ({ variables }) => {
+        async ({ variables = {} }) => {
          if (!onStartChat) return;
          if (isChatting) {
            toast({
@@ -435,7 +435,7 @@ const ChatBox = (
          // Only declared variables are kept
          const requestVariables: Record<string, any> = {};
          allVariableList?.forEach((item) => {
-            requestVariables[item.key] = variables[item.key] || '';
+            requestVariables[item.key] = variables[item.key];
          });

          const responseChatId = getNanoid(24);
--- a/projects/app/src/components/support/user/inform/SystemMsgModal.tsx
+++ b/projects/app/src/components/support/user/inform/SystemMsgModal.tsx
@@ -1,12 +1,12 @@
 import React, { useCallback } from 'react';
 import MyModal from '@fastgpt/web/components/common/MyModal';
 import { useUserStore } from '@/web/support/user/useUserStore';
-import { useQuery } from '@tanstack/react-query';
 import { Button, ModalBody, ModalFooter, useDisclosure } from '@chakra-ui/react';
 import { useTranslation } from 'next-i18next';
 import { LOGO_ICON } from '@fastgpt/global/common/system/constants';
 import { getSystemMsgModalData } from '@/web/support/user/inform/api';
 import dynamic from 'next/dynamic';
+import { useRequest2 } from '@fastgpt/web/hooks/useRequest';
 const Markdown = dynamic(() => import('@/components/Markdown'), { ssr: false });

 const SystemMsgModal = ({}: {}) => {
@@ -15,7 +15,9 @@ const SystemMsgModal = ({}: {}) => {

  const { isOpen, onOpen, onClose } = useDisclosure();

-  const { data } = useQuery(['initSystemMsgModal', systemMsgReadId], getSystemMsgModalData, {
+  const { data } = useRequest2(getSystemMsgModalData, {
+    refreshDeps: [systemMsgReadId],
+    manual: false,
    onSuccess(res) {
      if (res?.content && (!systemMsgReadId || res.id !== systemMsgReadId)) {
        onOpen();
--- a/projects/app/src/pages/api/common/file/upload.ts
+++ b/projects/app/src/pages/api/common/file/upload.ts
@@ -51,6 +51,7 @@ async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
      path: file.path,
      filename: file.originalname,
      contentType: file.mimetype,
+      encoding: file.encoding,
      metadata: metadata
    });

--- a/projects/app/src/pages/api/core/dataset/collection/create/localFile.ts
+++ b/projects/app/src/pages/api/core/dataset/collection/create/localFile.ts
@@ -67,6 +67,7 @@ async function handler(req: NextApiRequest, res: NextApiResponse<any>): CreateCo
    const { rawText } = await readRawTextByLocalFile({
      teamId,
      path: file.path,
+      encoding: file.encoding,
      metadata: {
        ...fileMetadata,
        relatedId: relatedImgId
@@ -81,6 +82,7 @@ async function handler(req: NextApiRequest, res: NextApiResponse<any>): CreateCo
      path: file.path,
      filename: file.originalname,
      contentType: file.mimetype,
+      encoding: file.encoding,
      metadata: fileMetadata
    });

--- a/projects/app/src/pages/api/core/dataset/update.ts
+++ b/projects/app/src/pages/api/core/dataset/update.ts
@@ -9,7 +9,7 @@ import {
 } from '@fastgpt/global/support/permission/constant';
 import { CommonErrEnum } from '@fastgpt/global/common/error/code/common';
 import type { ApiRequestProps, ApiResponseType } from '@fastgpt/service/type/next';
-import { DatasetTypeEnum } from '@fastgpt/global/core/dataset/constants';
+import { DatasetTypeEnum, TrainingModeEnum } from '@fastgpt/global/core/dataset/constants';
 import { ClientSession } from 'mongoose';
 import { parseParentIdInMongo } from '@fastgpt/global/common/parentFolder/utils';
 import { mongoSessionRun } from '@fastgpt/service/common/mongo/sessionRun';
@@ -21,6 +21,7 @@ import {
 import { authUserPer } from '@fastgpt/service/support/permission/user/auth';
 import { TeamWritePermissionVal } from '@fastgpt/global/support/permission/user/constant';
 import { DatasetErrEnum } from '@fastgpt/global/common/error/code/dataset';
+import { MongoDatasetTraining } from '@fastgpt/service/core/dataset/training/schema';

 export type DatasetUpdateQuery = {};
 export type DatasetUpdateResponse = any;
@@ -84,6 +85,12 @@ async function handler(

  const isFolder = dataset.type === DatasetTypeEnum.folder;

+  updateTraining({
+    teamId: dataset.teamId,
+    datasetId: id,
+    agentModel: agentModel?.model
+  });
+
  const onUpdate = async (session?: ClientSession) => {
    await MongoDataset.findByIdAndUpdate(
      id,
@@ -137,3 +144,29 @@ async function handler(
  }
 }
 export default NextAPI(handler);
+
+async function updateTraining({
+  teamId,
+  datasetId,
+  agentModel
+}: {
+  teamId: string;
+  datasetId: string;
+  agentModel?: string;
+}) {
+  if (!agentModel) return;
+
+  await MongoDatasetTraining.updateMany(
+    {
+      teamId,
+      datasetId,
+      mode: { $in: [TrainingModeEnum.qa, TrainingModeEnum.auto] }
+    },
+    {
+      $set: {
+        model: agentModel,
+        retryCount: 5
+      }
+    }
+  );
+}
--- a/projects/app/src/pages/app/detail/components/Plugin/Header.tsx
+++ b/projects/app/src/pages/app/detail/components/Plugin/Header.tsx
@@ -34,6 +34,7 @@ import {
  WorkflowInitContext
 } from '../WorkflowComponents/context/workflowInitContext';
 import { WorkflowEventContext } from '../WorkflowComponents/context/workflowEventContext';
+import { getAppConfigByDiff } from '@/web/core/app/diff';

 const Header = () => {
  const { t } = useTranslation();
@@ -51,16 +52,19 @@ const Header = () => {

  const nodes = useContextSelector(WorkflowInitContext, (v) => v.nodes);
  const edges = useContextSelector(WorkflowNodeEdgeContext, (v) => v.edges);
-  const {
-    flowData2StoreData,
-    flowData2StoreDataAndCheck,
-    setWorkflowTestData,
-    past,
-    future,
-    setPast,
-    onSwitchTmpVersion,
-    onSwitchCloudVersion
-  } = useContextSelector(WorkflowContext, (v) => v);
+
+  const flowData2StoreData = useContextSelector(WorkflowContext, (v) => v.flowData2StoreData);
+  const flowData2StoreDataAndCheck = useContextSelector(
+    WorkflowContext,
+    (v) => v.flowData2StoreDataAndCheck
+  );
+  const setWorkflowTestData = useContextSelector(WorkflowContext, (v) => v.setWorkflowTestData);
+  const past = useContextSelector(WorkflowContext, (v) => v.past);
+  const future = useContextSelector(WorkflowContext, (v) => v.future);
+  const setPast = useContextSelector(WorkflowContext, (v) => v.setPast);
+  const onSwitchTmpVersion = useContextSelector(WorkflowContext, (v) => v.onSwitchTmpVersion);
+  const onSwitchCloudVersion = useContextSelector(WorkflowContext, (v) => v.onSwitchCloudVersion);
+
  const showHistoryModal = useContextSelector(WorkflowEventContext, (v) => v.showHistoryModal);
  const setShowHistoryModal = useContextSelector(
    WorkflowEventContext,
@@ -76,15 +80,20 @@ const Header = () => {
        [...future].reverse().find((snapshot) => snapshot.isSaved) ||
        past.find((snapshot) => snapshot.isSaved);

+      const initialState = past[past.length - 1]?.state;
+      const savedSnapshotState = getAppConfigByDiff(initialState, savedSnapshot?.diff);
+
      const val = compareSnapshot(
+        // nodes of the saved snapshot
        {
-          nodes: savedSnapshot?.nodes,
-          edges: savedSnapshot?.edges,
-          chatConfig: savedSnapshot?.chatConfig
+          nodes: savedSnapshotState?.nodes,
+          edges: savedSnapshotState?.edges,
+          chatConfig: savedSnapshotState?.chatConfig
        },
+        // nodes of the current canvas
        {
-          nodes: nodes,
-          edges: edges,
+          nodes,
+          edges,
          chatConfig: appDetail.chatConfig
        }
      );
@@ -132,8 +141,6 @@ const Header = () => {

  const onBack = useCallback(async () => {
    try {
-      localStorage.removeItem(`${appDetail._id}-past`);
-      localStorage.removeItem(`${appDetail._id}-future`);
      router.push({
        pathname: '/app/list',
        query: {
@@ -142,7 +149,7 @@ const Header = () => {
        }
      });
    } catch (error) {}
-  }, [appDetail._id, appDetail.parentId, lastAppListRouteType, router]);
+  }, [appDetail.parentId, lastAppListRouteType, router]);

  const Render = useMemo(() => {
    return (
--- a/projects/app/src/pages/app/detail/components/SimpleApp/Edit.tsx
+++ b/projects/app/src/pages/app/detail/components/SimpleApp/Edit.tsx
@@ -17,16 +17,44 @@ import styles from './styles.module.scss';
 import { useSystem } from '@fastgpt/web/hooks/useSystem';
 import { useTranslation } from 'next-i18next';
 import { onSaveSnapshotFnType, SimpleAppSnapshotType } from './useSnapshots';
+import { getAppConfigByDiff, getAppDiffConfig } from '@/web/core/app/diff';
+import { formatTime2YMDHMS } from '@fastgpt/global/common/string/time';
+
+const convertOldFormatHistory = (past: SimpleAppSnapshotType[]) => {
+  const baseState = past[past.length - 1].appForm;
+
+  return past.map((item, index) => {
+    if (index === past.length - 1) {
+      return {
+        title: item.title,
+        isSaved: item.isSaved,
+        state: baseState
+      };
+    }
+
+    const currentState = item.appForm;
+
+    const diff = getAppDiffConfig(baseState, currentState);
+
+    return {
+      title: item.title || formatTime2YMDHMS(new Date()),
+      isSaved: item.isSaved,
+      diff
+    };
+  });
+};

 const Edit = ({
  appForm,
  setAppForm,
  past,
+  setPast,
  saveSnapshot
 }: {
  appForm: AppSimpleEditFormType;
  setAppForm: React.Dispatch<React.SetStateAction<AppSimpleEditFormType>>;
  past: SimpleAppSnapshotType[];
+  setPast: (value: React.SetStateAction<SimpleAppSnapshotType[]>) => void;
  saveSnapshot: onSaveSnapshotFnType;
 }) => {
  const { isPc } = useSystem();
@@ -39,9 +67,26 @@ const Edit = ({
    // show selected dataset
    loadAllDatasets();

+    if (appDetail.version !== 'v2') {
+      return setAppForm(
+        appWorkflow2Form({
+          nodes: v1Workflow2V2((appDetail.modules || []) as any)?.nodes,
+          chatConfig: appDetail.chatConfig
+        })
+      );
+    }
+
    // Get the latest snapshot
-    if (past?.[0]?.appForm) {
-      return setAppForm(past[0].appForm);
+    if (past?.[0]?.diff) {
+      const pastState = getAppConfigByDiff(past[past.length - 1].state, past[0].diff);
+
+      return setAppForm(pastState);
+    } else if (past && past.length > 0 && past?.every((item) => item.appForm)) {
+      // 格式化成 diff
+      const newPast = convertOldFormatHistory(past);
+
+      setPast(newPast);
+      return setAppForm(getAppConfigByDiff(newPast[newPast.length - 1].state, newPast[0].diff));
    }

    const appForm = appWorkflow2Form({
@@ -59,15 +104,6 @@ const Edit = ({
    }

    setAppForm(appForm);
-
-    if (appDetail.version !== 'v2') {
-      setAppForm(
-        appWorkflow2Form({
-          nodes: v1Workflow2V2((appDetail.modules || []) as any)?.nodes,
-          chatConfig: appDetail.chatConfig
-        })
-      );
-    }
  });

  return (
--- a/projects/app/src/pages/app/detail/components/SimpleApp/Header.tsx
+++ b/projects/app/src/pages/app/detail/components/SimpleApp/Header.tsx
@@ -1,4 +1,4 @@
-import React, { useCallback, useState } from 'react';
+import React, { useCallback, useMemo, useState } from 'react';
 import { useContextSelector } from 'use-context-selector';
 import { AppContext } from '../context';
 import FolderPath from '@/components/common/folder/Path';
@@ -29,6 +29,7 @@ import {
 } from './useSnapshots';
 import PublishHistories from '../PublishHistoriesSlider';
 import { AppVersionSchemaType } from '@fastgpt/global/core/app/version';
+import { getAppConfigByDiff } from '@/web/core/app/diff';

 const Header = ({
  forbiddenSaveSnapshot,
@@ -48,7 +49,11 @@ const Header = ({
  const { t } = useTranslation();
  const { isPc } = useSystem();
  const router = useRouter();
-  const { appId, onSaveApp, currentTab } = useContextSelector(AppContext, (v) => v);
+  const appId = useContextSelector(AppContext, (v) => v.appId);
+  const onSaveApp = useContextSelector(AppContext, (v) => v.onSaveApp);
+  const currentTab = useContextSelector(AppContext, (v) => v.currentTab);
+  const appLatestVersion = useContextSelector(AppContext, (v) => v.appLatestVersion);
+
  const { lastAppListRouteType } = useSystemStore();
  const { allDatasets } = useDatasetStore();

@@ -102,9 +107,19 @@ const Header = ({
  const [isShowHistories, { setTrue: setIsShowHistories, setFalse: closeHistories }] =
    useBoolean(false);

+  const initialAppForm = useMemo(
+    () =>
+      appWorkflow2Form({
+        nodes: appLatestVersion?.nodes || [],
+        chatConfig: appLatestVersion?.chatConfig || {}
+      }),
+    [appLatestVersion]
+  );
+
  const onSwitchTmpVersion = useCallback(
    (data: SimpleAppSnapshotType, customTitle: string) => {
-      setAppForm(data.appForm);
+      const pastState = getAppConfigByDiff(initialAppForm, data.diff);
+      setAppForm(pastState);

      // Remove multiple "copy-"
      const copyText = t('app:version_copy');
@@ -112,11 +127,11 @@ const Header = ({
      const title = customTitle.replace(regex, `$1`);

      return saveSnapshot({
-        appForm: data.appForm,
+        appForm: pastState,
        title
      });
    },
-    [saveSnapshot, setAppForm, t]
+    [initialAppForm, saveSnapshot, setAppForm, t]
  );
  const onSwitchCloudVersion = useCallback(
    (appVersion: AppVersionSchemaType) => {
@@ -143,7 +158,8 @@ const Header = ({
  useDebounceEffect(
    () => {
      const savedSnapshot = past.find((snapshot) => snapshot.isSaved);
-      const val = compareSimpleAppSnapshot(savedSnapshot?.appForm, appForm);
+      const pastState = getAppConfigByDiff(initialAppForm, savedSnapshot?.diff);
+      const val = compareSimpleAppSnapshot(pastState, appForm);
      setIsPublished(val);
    },
    [past, allDatasets],
--- a/projects/app/src/pages/app/detail/components/SimpleApp/index.tsx
+++ b/projects/app/src/pages/app/detail/components/SimpleApp/index.tsx
@@ -49,7 +49,13 @@ const SimpleEdit = () => {
        saveSnapshot={saveSnapshot}
      />
      {currentTab === TabEnum.appEdit ? (
-        <Edit appForm={appForm} setAppForm={setAppForm} past={past} saveSnapshot={saveSnapshot} />
+        <Edit
+          appForm={appForm}
+          setAppForm={setAppForm}
+          past={past}
+          setPast={setPast}
+          saveSnapshot={saveSnapshot}
+        />
      ) : (
        <Box flex={'1 0 0'} h={0} mt={[4, 0]}>
          {currentTab === TabEnum.publish && <PublishChannel />}
--- a/projects/app/src/pages/app/detail/components/SimpleApp/useSnapshots.tsx
+++ b/projects/app/src/pages/app/detail/components/SimpleApp/useSnapshots.tsx
@@ -3,14 +3,19 @@ import { SetStateAction, useEffect, useRef } from 'react';
 import { formatTime2YMDHMS } from '@fastgpt/global/common/string/time';
 import { AppSimpleEditFormType } from '@fastgpt/global/core/app/type';
 import { isEqual } from 'lodash';
+import { getAppDiffConfig } from '@/web/core/app/diff';

 export type SimpleAppSnapshotType = {
-  appForm: AppSimpleEditFormType;
+  diff?: Record<string, any>;
  title: string;
  isSaved?: boolean;
+  state?: AppSimpleEditFormType;
+
+  // old format
+  appForm?: AppSimpleEditFormType;
 };
 export type onSaveSnapshotFnType = (props: {
-  appForm: AppSimpleEditFormType;
+  appForm: AppSimpleEditFormType; // Current edited app form data
  title?: string;
  isSaved?: boolean;
 }) => Promise<boolean>;
@@ -56,7 +61,7 @@ export const compareSimpleAppSnapshot = (

 export const useSimpleAppSnapshots = (appId: string) => {
  const forbiddenSaveSnapshot = useRef(false);
-  const [past, setPast] = useLocalStorageState<SimpleAppSnapshotType[]>(`${appId}-past-simple`, {
+  const [past, setPast] = useLocalStorageState<SimpleAppSnapshotType[]>(`${appId}-past`, {
    defaultValue: []
  }) as [SimpleAppSnapshotType[], (value: SetStateAction<SimpleAppSnapshotType[]>) => void];

@@ -66,26 +71,47 @@ export const useSimpleAppSnapshots = (appId: string) => {
      return false;
    }

-    const pastState = past[0];
+    if (past.length === 0) {
+      setPast([
+        {
+          title: title || formatTime2YMDHMS(new Date()),
+          isSaved,
+          state: appForm
+        }
+      ]);
+      return true;
+    }

-    const isPastEqual = compareSimpleAppSnapshot(pastState?.appForm, appForm);
-    if (isPastEqual) return false;
+    const lastPast = past[past.length - 1];
+    if (!lastPast?.state) return false;

-    setPast((past) => [
-      {
-        appForm,
+    // Get the diff between the current app form data and the initial state
+    const diff = getAppDiffConfig(lastPast.state, appForm);
+
+    // If the diff is the same as the previous snapshot, do not save
+    if (past[0].diff && isEqual(past[0].diff, diff)) return false;
+
+    setPast((past) => {
+      const newPast = {
+        diff,
        title: title || formatTime2YMDHMS(new Date()),
        isSaved
-      },
-      ...past.slice(0, 199)
-    ]);
+      };
+
+      if (past.length >= 100) {
+        return [newPast, ...past.slice(0, 98), lastPast];
+      }
+      return [newPast, ...past];
+    });
    return true;
  });

  // remove other app's snapshot
  useEffect(() => {
    const keys = Object.keys(localStorage);
-    const snapshotKeys = keys.filter((key) => key.endsWith('-past-simple'));
+    const snapshotKeys = keys.filter(
+      (key) => key.endsWith('-past') || key.endsWith('-past-simple')
+    );
    snapshotKeys.forEach((key) => {
      const keyAppId = key.split('-')[0];
      if (keyAppId !== appId) {
@@ -94,6 +120,20 @@ export const useSimpleAppSnapshots = (appId: string) => {
    });
  }, [appId]);

+  // 旧的编辑记录，直接重置到新的变量中
+  const [oldPast, setOldPast] = useLocalStorageState<SimpleAppSnapshotType[]>(
+    `${appId}-past-simple`,
+    {}
+  );
+  useEffect(() => {
+    if (oldPast && oldPast.length > 0) {
+      setPast(past);
+      setOldPast([]);
+      // refresh page
+      window.location.reload();
+    }
+  }, [oldPast]);
+
  return { forbiddenSaveSnapshot, past, setPast, saveSnapshot };
 };

--- a/projects/app/src/pages/app/detail/components/Workflow/Header.tsx
+++ b/projects/app/src/pages/app/detail/components/Workflow/Header.tsx
@@ -34,6 +34,7 @@ import {
  WorkflowInitContext
 } from '../WorkflowComponents/context/workflowInitContext';
 import { WorkflowEventContext } from '../WorkflowComponents/context/workflowEventContext';
+import { getAppConfigByDiff } from '@/web/core/app/diff';

 const Header = () => {
  const { t } = useTranslation();
@@ -55,16 +56,19 @@ const Header = () => {

  const nodes = useContextSelector(WorkflowInitContext, (v) => v.nodes);
  const edges = useContextSelector(WorkflowNodeEdgeContext, (v) => v.edges);
-  const {
-    flowData2StoreData,
-    flowData2StoreDataAndCheck,
-    setWorkflowTestData,
-    past,
-    future,
-    setPast,
-    onSwitchTmpVersion,
-    onSwitchCloudVersion
-  } = useContextSelector(WorkflowContext, (v) => v);
+
+  const flowData2StoreData = useContextSelector(WorkflowContext, (v) => v.flowData2StoreData);
+  const flowData2StoreDataAndCheck = useContextSelector(
+    WorkflowContext,
+    (v) => v.flowData2StoreDataAndCheck
+  );
+  const setWorkflowTestData = useContextSelector(WorkflowContext, (v) => v.setWorkflowTestData);
+  const past = useContextSelector(WorkflowContext, (v) => v.past);
+  const future = useContextSelector(WorkflowContext, (v) => v.future);
+  const setPast = useContextSelector(WorkflowContext, (v) => v.setPast);
+  const onSwitchTmpVersion = useContextSelector(WorkflowContext, (v) => v.onSwitchTmpVersion);
+  const onSwitchCloudVersion = useContextSelector(WorkflowContext, (v) => v.onSwitchCloudVersion);
+
  const showHistoryModal = useContextSelector(WorkflowEventContext, (v) => v.showHistoryModal);
  const setShowHistoryModal = useContextSelector(
    WorkflowEventContext,
@@ -81,11 +85,14 @@ const Header = () => {
        [...future].reverse().find((snapshot) => snapshot.isSaved) ||
        past.find((snapshot) => snapshot.isSaved);

+      const initialState = past[past.length - 1]?.state;
+      const savedSnapshotState = getAppConfigByDiff(initialState, savedSnapshot?.diff);
+
      const val = compareSnapshot(
        {
-          nodes: savedSnapshot?.nodes,
-          edges: savedSnapshot?.edges,
-          chatConfig: savedSnapshot?.chatConfig
+          nodes: savedSnapshotState?.nodes,
+          edges: savedSnapshotState?.edges,
+          chatConfig: savedSnapshotState?.chatConfig
        },
        {
          nodes: nodes,
@@ -137,8 +144,6 @@ const Header = () => {

  const onBack = useCallback(async () => {
    try {
-      localStorage.removeItem(`${appDetail._id}-past`);
-      localStorage.removeItem(`${appDetail._id}-future`);
      router.push({
        pathname: '/app/list',
        query: {
@@ -147,7 +152,7 @@ const Header = () => {
        }
      });
    } catch (error) {}
-  }, [appDetail._id, appDetail.parentId, lastAppListRouteType, router]);
+  }, [appDetail.parentId, lastAppListRouteType, router]);

  const Render = useMemo(() => {
    return (
--- a/projects/app/src/pages/app/detail/components/WorkflowComponents/Flow/components/ContextMenu.tsx
+++ b/projects/app/src/pages/app/detail/components/WorkflowComponents/Flow/components/ContextMenu.tsx
@@ -5,7 +5,6 @@ import { useTranslation } from 'react-i18next';
 import { nodeTemplate2FlowNode } from '@/web/core/workflow/utils';
 import { CommentNode } from '@fastgpt/global/core/workflow/template/system/comment';
 import { useContextSelector } from 'use-context-selector';
-import { WorkflowContext } from '../../context';
 import { useReactFlow } from 'reactflow';
 import { WorkflowNodeEdgeContext } from '../../context/workflowInitContext';
 import { WorkflowEventContext } from '../../context/workflowEventContext';
--- a/projects/app/src/pages/app/detail/components/WorkflowComponents/Flow/nodes/NodePluginIO/InputEditModal.tsx
+++ b/projects/app/src/pages/app/detail/components/WorkflowComponents/Flow/nodes/NodePluginIO/InputEditModal.tsx
@@ -215,6 +215,7 @@ const FieldEditModal = ({
  );
  const onSubmitError = useCallback(
    (e: Object) => {
+      console.log('e', e);
      for (const item of Object.values(e)) {
        if (item.message) {
          toast({
--- a/projects/app/src/pages/app/detail/components/WorkflowComponents/Flow/nodes/NodePluginIO/InputTypeConfig.tsx
+++ b/projects/app/src/pages/app/detail/components/WorkflowComponents/Flow/nodes/NodePluginIO/InputTypeConfig.tsx
@@ -511,22 +511,14 @@ const InputTypeConfig = ({
              <FormLabel flex={'0 0 132px'} fontWeight={'medium'}>
                {t('app:document_upload')}
              </FormLabel>
-              <Switch
-                {...register('canSelectFile', {
-                  required: true
-                })}
-              />
+              <Switch {...register('canSelectFile')} />
            </Flex>
            <Box w={'full'} minH={'40px'}>
              <Flex alignItems={'center'}>
                <FormLabel flex={'0 0 132px'} fontWeight={'medium'}>
                  {t('app:image_upload')}
                </FormLabel>
-                <Switch
-                  {...register('canSelectImg', {
-                    required: true
-                  })}
-                />
+                <Switch {...register('canSelectImg')} />
              </Flex>
              <Flex color={'myGray.500'}>
                <Box fontSize={'xs'}>{t('app:image_upload_tip')}</Box>
--- a/projects/app/src/pages/app/detail/components/WorkflowComponents/Flow/nodes/render/Handle/ConnectionHandle.tsx
+++ b/projects/app/src/pages/app/detail/components/WorkflowComponents/Flow/nodes/render/Handle/ConnectionHandle.tsx
@@ -6,6 +6,7 @@ import { NodeOutputKeyEnum } from '@fastgpt/global/core/workflow/constants';
 import { useContextSelector } from 'use-context-selector';
 import { WorkflowContext } from '../../../../context';
 import { WorkflowNodeEdgeContext } from '../../../../context/workflowInitContext';
+import { FlowNodeItemType } from '@fastgpt/global/core/workflow/type/node';

 export const ConnectionSourceHandle = ({
  nodeId,
@@ -141,15 +142,23 @@ export const ConnectionTargetHandle = React.memo(function ConnectionTargetHandle
  const { connectingEdge, nodeList } = useContextSelector(WorkflowContext, (ctx) => ctx);

  const { LeftHandle, rightHandle, topHandle, bottomHandle } = useMemo(() => {
-    const node = nodeList.find((node) => node.nodeId === nodeId);
-    const connectingNode = nodeList.find((node) => node.nodeId === connectingEdge?.nodeId);
+    let node: FlowNodeItemType | undefined = undefined,
+      connectingNode: FlowNodeItemType | undefined = undefined;
+    for (const item of nodeList) {
+      if (item.nodeId === nodeId) {
+        node = item;
+      }
+      if (item.nodeId === connectingEdge?.nodeId) {
+        connectingNode = item;
+      }
+      if (node && (connectingNode || !connectingEdge?.nodeId)) break;
+    }

-    const connectingNodeSourceNodeIdMap = new Map<string, number>();
    let forbidConnect = false;
-    edges.forEach((edge) => {
-      if (edge.target === connectingNode?.nodeId) {
-        connectingNodeSourceNodeIdMap.set(edge.source, 1);
-      } else if (edge.target === nodeId) {
+    for (const edge of edges) {
+      if (forbidConnect) break;
+
+      if (edge.target === nodeId) {
        // Node has be connected tool, it cannot be connect by other handle
        if (edge.targetHandle === NodeOutputKeyEnum.selectedTools) {
          forbidConnect = true;
@@ -163,7 +172,7 @@ export const ConnectionTargetHandle = React.memo(function ConnectionTargetHandle
          forbidConnect = true;
        }
      }
-    });
+    }

    const showHandle = (() => {
      if (forbidConnect) return false;
@@ -178,9 +187,6 @@ export const ConnectionTargetHandle = React.memo(function ConnectionTargetHandle
      // Not the same parent node
      if (connectingNode && connectingNode?.parentNodeId !== node?.parentNodeId) return false;

-      // Unable to connect to the source node
-      if (connectingNodeSourceNodeIdMap.has(nodeId)) return false;
-
      return true;
    })();

--- a/projects/app/src/pages/app/detail/components/WorkflowComponents/Flow/nodes/render/NodeCard.tsx
+++ b/projects/app/src/pages/app/detail/components/WorkflowComponents/Flow/nodes/render/NodeCard.tsx
@@ -115,6 +115,16 @@ const NodeCard = (props: Props) => {
      }
    },
    {
+      onSuccess(res) {
+        if (!res) return;
+        // Execute forcibly updates the courseUrl field
+        onChangeNode({
+          nodeId,
+          type: 'attr',
+          key: 'courseUrl',
+          value: res?.courseUrl
+        });
+      },
      manual: false
    }
  );
--- a/projects/app/src/pages/app/detail/components/WorkflowComponents/context/index.tsx
+++ b/projects/app/src/pages/app/detail/components/WorkflowComponents/context/index.tsx
@@ -1,7 +1,7 @@
 import { postWorkflowDebug } from '@/web/core/workflow/api';
 import {
  checkWorkflowNodeAndConnection,
-  compareSnapshot,
+  simplifyWorkflowNodes,
  storeEdgesRenderEdge,
  storeNode2FlowNode
 } from '@/web/core/workflow/utils';
@@ -37,10 +37,11 @@ import { useDisclosure } from '@chakra-ui/react';
 import { uiWorkflow2StoreWorkflow } from '../utils';
 import { useTranslation } from 'next-i18next';
 import { formatTime2YMDHMS, formatTime2YMDHMW } from '@fastgpt/global/common/string/time';
-import { cloneDeep } from 'lodash';
+import { cloneDeep, isEqual } from 'lodash';
 import { AppVersionSchemaType } from '@fastgpt/global/core/app/version';
 import WorkflowInitContextProvider, { WorkflowNodeEdgeContext } from './workflowInitContext';
 import WorkflowEventContextProvider from './workflowEventContext';
+import { getAppConfigByDiff, getAppDiffConfig } from '@/web/core/app/diff';

 /* 
  Context
@@ -67,14 +68,22 @@ export const ReactFlowCustomProvider = ({
  );
 };

-type OnChange<ChangesType> = (changes: ChangesType[]) => void;
-
 export type WorkflowSnapshotsType = {
+  diff?: any;
+  title: string;
+  isSaved?: boolean;
+  state?: WorkflowStateType;
+
+  // old format
+  nodes?: Node[];
+  edges?: Edge[];
+  chatConfig?: AppChatConfigType;
+};
+
+export type WorkflowStateType = {
  nodes: Node[];
  edges: Edge[];
-  title: string;
  chatConfig: AppChatConfigType;
-  isSaved?: boolean;
 };

 type WorkflowContextType = {
@@ -751,7 +760,7 @@ const WorkflowContextProvider = ({
    defaultValue: []
  }) as [WorkflowSnapshotsType[], (value: SetStateAction<WorkflowSnapshotsType[]>) => void];

-  const resetSnapshot = useMemoizedFn((state: Omit<WorkflowSnapshotsType, 'title' | 'isSaved'>) => {
+  const resetSnapshot = useMemoizedFn((state: WorkflowStateType) => {
    setNodes(state.nodes);
    setEdges(state.edges);
    setAppDetail((detail) => ({
@@ -759,70 +768,60 @@ const WorkflowContextProvider = ({
      chatConfig: state.chatConfig
    }));
  });
-  const pushPastSnapshot = useMemoizedFn(
-    ({
-      pastNodes,
-      pastEdges,
-      customTitle,
-      chatConfig,
-      isSaved
-    }: {
-      pastNodes: Node[];
-      pastEdges: Edge[];
-      customTitle?: string;
-      chatConfig: AppChatConfigType;
-      isSaved?: boolean;
-    }) => {
-      if (!pastNodes || !pastEdges || !chatConfig) return false;

+  const pushPastSnapshot = useMemoizedFn(
+    ({ pastNodes, pastEdges, chatConfig, customTitle, isSaved }) => {
+      if (!pastNodes || !pastEdges || !chatConfig) return false;
      if (forbiddenSaveSnapshot.current) {
        forbiddenSaveSnapshot.current = false;
        return false;
      }

-      const pastState = past[0];
-      const isPastEqual = compareSnapshot(
-        {
-          nodes: pastNodes,
-          edges: pastEdges,
-          chatConfig: chatConfig
-        },
-        {
-          nodes: pastState?.nodes,
-          edges: pastState?.edges,
-          chatConfig: pastState?.chatConfig
-        }
-      );
+      // Get initial state
+      const lastSnapshot = past[past.length - 1];
+      if (!lastSnapshot?.state) return false;

-      if (isPastEqual) return false;
+      // Create current state object
+      const newState = {
+        nodes: simplifyWorkflowNodes(pastNodes),
+        edges: pastEdges,
+        chatConfig
+      };
+
+      // Calculate diff from initial state
+      const diff = getAppDiffConfig(lastSnapshot.state, newState);
+      if (past[0].diff && isEqual(past[0].diff, diff)) return false;

      setFuture([]);
-      setPast((past) => [
-        {
-          nodes: pastNodes,
-          edges: pastEdges,
+      setPast((past) => {
+        const newPast = {
+          diff,
          title: customTitle || formatTime2YMDHMS(new Date()),
-          chatConfig,
          isSaved
-        },
-        ...past.slice(0, 199)
-      ]);
+        };
+        if (past.length >= 100) {
+          return [newPast, ...past.slice(0, 98), lastSnapshot];
+        }
+        return [newPast, ...past];
+      });

      return true;
    }
  );
+
  const onSwitchTmpVersion = useMemoizedFn((params: WorkflowSnapshotsType, customTitle: string) => {
    // Remove multiple "copy-"
    const copyText = t('app:version_copy');
    const regex = new RegExp(`(${copyText}-)\\1+`, 'g');
    const title = customTitle.replace(regex, `$1`);
+    const pastState = getAppConfigByDiff(past[past.length - 1].state, params.diff);

-    resetSnapshot(params);
+    resetSnapshot(pastState);

    return pushPastSnapshot({
-      pastNodes: params.nodes,
-      pastEdges: params.edges,
-      chatConfig: params.chatConfig,
+      pastNodes: pastState.nodes,
+      pastEdges: pastState.edges,
+      chatConfig: pastState.chatConfig,
      customTitle: title
    });
  });
@@ -848,15 +847,19 @@ const WorkflowContextProvider = ({
    if (past[1]) {
      setFuture((future) => [past[0], ...future]);
      setPast((past) => past.slice(1));
-      resetSnapshot(past[1]);
+      const pastState = getAppConfigByDiff(past[past.length - 1].state, past[1].diff);
+      resetSnapshot(pastState);
    }
  });
  const redo = useMemoizedFn(() => {
-    const futureState = future[0];
+    if (!future[0]) return;
+
+    const futureState = getAppConfigByDiff(past[past.length - 1].state, future[0].diff);

    if (futureState) {
      setPast((past) => [future[0], ...past]);
      setFuture((future) => future.slice(1));
+
      resetSnapshot(futureState);
    }
  });
@@ -874,44 +877,79 @@ const WorkflowContextProvider = ({
  }, [appId]);

  const initData = useCallback(
-    async (e: Parameters<WorkflowContextType['initData']>[0], isInit?: boolean) => {
-      // Refresh web page, load init
+    async (
+      e: {
+        nodes: StoreNodeItemType[];
+        edges: StoreEdgeItemType[];
+        chatConfig?: AppChatConfigType;
+      },
+      isInit?: boolean
+    ) => {
+      const nodes = e.nodes?.map((item) => storeNode2FlowNode({ item, t })) || [];
+      const edges = e.edges?.map((item) => storeEdgesRenderEdge({ edge: item })) || [];
+
+      const initialState = {
+        nodes: simplifyWorkflowNodes(nodes),
+        edges,
+        chatConfig: e.chatConfig || appDetail.chatConfig
+      };
+
      if (isInit && past.length > 0) {
-        return resetSnapshot(past[0]);
+        // new format
+        if (past[0].diff && past[past.length - 1].state) {
+          const targetState = getAppConfigByDiff(
+            past[past.length - 1].state,
+            past[0].diff
+          ) as WorkflowStateType;
+
+          setNodes(targetState.nodes);
+          setEdges(targetState.edges);
+          setAppDetail((state) => ({
+            ...state,
+            chatConfig: targetState.chatConfig
+          }));
+          return;
+        }
+
+        // 适配旧的编辑记录（4.8.15去除）
+        if (past.every((item) => item.nodes)) {
+          const newPast = convertOldFormatHistory(past);
+
+          setPast(newPast);
+
+          const latestState = getAppConfigByDiff(
+            newPast[newPast.length - 1].state,
+            newPast[0].diff
+          ) as WorkflowStateType;
+
+          setNodes(latestState.nodes);
+          setEdges(latestState.edges);
+          setAppDetail((state) => ({
+            ...state,
+            chatConfig: latestState.chatConfig
+          }));
+          return;
+        }
      }
-      // If it is the initial data, save the initial snapshot
+
+      setNodes(nodes);
+      setEdges(edges);
+      if (e.chatConfig) {
+        setAppDetail((state) => ({ ...state, chatConfig: e.chatConfig as AppChatConfigType }));
+      }
+
      if (isInit && past.length === 0) {
-        pushPastSnapshot({
-          pastNodes: e.nodes?.map((item) => storeNode2FlowNode({ item, t })) || [],
-          pastEdges: e.edges?.map((item) => storeEdgesRenderEdge({ edge: item })) || [],
-          customTitle: t(`app:app.version_initial`),
-          chatConfig: appDetail.chatConfig,
-          isSaved: true
-        });
+        setPast([
+          {
+            title: t(`app:app.version_initial`),
+            isSaved: true,
+            state: initialState
+          }
+        ]);
        forbiddenSaveSnapshot.current = true;
      }
-
-      setNodes(e.nodes?.map((item) => storeNode2FlowNode({ item, t })) || []);
-      setEdges(e.edges?.map((item) => storeEdgesRenderEdge({ edge: item })) || []);
-
-      const chatConfig = e.chatConfig;
-      if (chatConfig) {
-        setAppDetail((state) => ({
-          ...state,
-          chatConfig
-        }));
-      }
    },
-    [
-      appDetail.chatConfig,
-      past,
-      resetSnapshot,
-      pushPastSnapshot,
-      setAppDetail,
-      setEdges,
-      setNodes,
-      t
-    ]
+    [appDetail.chatConfig, past, setAppDetail, setEdges, setNodes, setPast, t]
  );

  const value = useMemo(
@@ -997,3 +1035,36 @@ const WorkflowContextProvider = ({
  );
 };
 export default React.memo(WorkflowContextProvider);
+
+// Convert old history format to new format
+const convertOldFormatHistory = (past: WorkflowSnapshotsType[]) => {
+  const baseState = {
+    nodes: past[past.length - 1].state?.nodes || [],
+    edges: past[past.length - 1].state?.edges || [],
+    chatConfig: past[past.length - 1].state?.chatConfig || {}
+  };
+
+  return past.map((item, index) => {
+    if (index === past.length - 1) {
+      return {
+        title: item.title,
+        isSaved: item.isSaved,
+        state: baseState
+      };
+    }
+
+    const currentState = {
+      nodes: item.nodes || [],
+      edges: item.edges || [],
+      chatConfig: item.chatConfig || {}
+    };
+
+    const diff = getAppDiffConfig(baseState, currentState);
+
+    return {
+      title: item.title || formatTime2YMDHMS(new Date()),
+      isSaved: item.isSaved,
+      diff
+    };
+  });
+};
--- a/projects/app/src/pages/chat/share.tsx
+++ b/projects/app/src/pages/chat/share.tsx
@@ -385,7 +385,7 @@ const Render = (props: Props) => {
  const [isLoaded, setIsLoaded] = useState(false);

  const contextParams = useMemo(() => {
-    return { shareId, outLinkUid: authToken || localUId || customUid };
+    return { shareId, outLinkUid: authToken || customUid || localUId };
  }, [authToken, customUid, localUId, shareId]);

  useMount(() => {
--- a/projects/app/src/pages/dataset/detail/components/Import/Context.tsx
+++ b/projects/app/src/pages/dataset/detail/components/Import/Context.tsx
@@ -254,8 +254,6 @@ const DatasetImportContextProvider = ({ children }: { children: React.ReactNode
              icon={<MyIcon name={'common/backFill'} w={'14px'} />}
              aria-label={''}
              size={'smSquare'}
-              w={'26px'}
-              h={'26px'}
              borderRadius={'50%'}
              variant={'whiteBase'}
              mr={2}
--- a/projects/app/src/pages/login/components/LoginForm/components/FormLayout.tsx
+++ b/projects/app/src/pages/login/components/LoginForm/components/FormLayout.tsx
@@ -59,6 +59,16 @@ const FormLayout = ({ children, setPageType, pageType }: Props) => {
          }
        ]
      : []),
+    ...(feConfigs?.oauth?.microsoft
+      ? [
+          {
+            label: t('common:support.user.login.Microsoft'),
+            provider: OAuthEnum.microsoft,
+            icon: 'common/microsoft',
+            redirectUrl: `https://login.microsoftonline.com/common/oauth2/v2.0/authorize?client_id=${feConfigs?.oauth?.microsoft}&response_type=code&redirect_uri=${redirectUri}&response_mode=query&scope=https%3A%2F%2Fgraph.microsoft.com%2Fuser.read&state=${state.current}`
+          }
+        ]
+      : []),
    ...(pageType !== LoginPageTypeEnum.passwordLogin
      ? [
          {
--- a/projects/app/src/service/common/system/volumnMongoWatch.ts
+++ b/projects/app/src/service/common/system/volumnMongoWatch.ts
@@ -1,4 +1,4 @@
-import { getSystemPlugins } from '@/service/core/app/plugin';
+import { getSystemPluginCb } from '@/service/core/app/plugin';
 import { initSystemConfig } from '.';
 import { createDatasetTrainingMongoWatch } from '@/service/core/dataset/training/utils';
 import { MongoSystemConfigs } from '@fastgpt/service/common/system/config/schema';
@@ -29,7 +29,7 @@ const refetchSystemPlugins = () => {
  changeStream.on('change', async (change) => {
    setTimeout(() => {
      try {
-        getSystemPlugins(true);
+        getSystemPluginCb(true);
      } catch (error) {}
    }, 5000);
  });
--- a/projects/app/src/service/core/app/plugin.ts
+++ b/projects/app/src/service/core/app/plugin.ts
@@ -11,7 +11,8 @@ const getCommercialPlugins = () => {
  return GET<SystemPluginTemplateItemType[]>('/core/app/plugin/getSystemPlugins');
 };
 export const getSystemPlugins = async (refresh = false) => {
-  if (isProduction && global.systemPlugins && !refresh) return cloneDeep(global.systemPlugins);
+  if (isProduction && global.systemPlugins && global.systemPlugins.length > 0 && !refresh)
+    return cloneDeep(global.systemPlugins);

  try {
    if (!global.systemPlugins) {
@@ -54,17 +55,22 @@ const getCommercialCb = async () => {
    {}
  );
 };
-export const getSystemPluginCb = async () => {
-  if (isProduction && global.systemPluginCb) return global.systemPluginCb;
+
+export const getSystemPluginCb = async (refresh = false) => {
+  if (
+    isProduction &&
+    global.systemPluginCb &&
+    Object.keys(global.systemPluginCb).length > 0 &&
+    !refresh
+  )
+    return global.systemPluginCb;

  try {
-    await getSystemPlugins();
    global.systemPluginCb = {};
+    await getSystemPlugins();
    global.systemPluginCb = FastGPTProUrl ? await getCommercialCb() : await getCommunityCb();
    return global.systemPluginCb;
  } catch (error) {
-    //@ts-ignore
-    global.systemPluginCb = undefined;
    return Promise.reject(error);
  }
 };
--- a/projects/app/src/service/events/generateQA.ts
+++ b/projects/app/src/service/events/generateQA.ts
@@ -39,11 +39,13 @@ export async function generateQA(): Promise<any> {
    try {
      const data = await MongoDatasetTraining.findOneAndUpdate(
        {
-          lockTime: { $lte: addMinutes(new Date(), -6) },
-          mode: TrainingModeEnum.qa
+          mode: TrainingModeEnum.qa,
+          retryCount: { $gte: 0 },
+          lockTime: { $lte: addMinutes(new Date(), -6) }
        },
        {
-          lockTime: new Date()
+          lockTime: new Date(),
+          $inc: { retryCount: -1 }
        }
      )
        .select({
--- a/projects/app/src/service/events/generateVector.ts
+++ b/projects/app/src/service/events/generateVector.ts
@@ -39,10 +39,12 @@ export async function generateVector(): Promise<any> {
      const data = await MongoDatasetTraining.findOneAndUpdate(
        {
          mode: TrainingModeEnum.chunk,
-          lockTime: { $lte: addMinutes(new Date(), -1) }
+          retryCount: { $gte: 0 },
+          lockTime: { $lte: addMinutes(new Date(), -6) }
        },
        {
-          lockTime: new Date()
+          lockTime: new Date(),
+          $inc: { retryCount: -1 }
        }
      ).select({
        _id: 1,
--- a/projects/app/src/service/mongo.ts
+++ b/projects/app/src/service/mongo.ts
@@ -32,8 +32,9 @@ export function connectToDatabase() {
      systemStartCb();

      //init system config；init vector database；init root user
-      await Promise.all([getInitConfig(), getSystemPluginCb(), initVectorStore(), initRootUser()]);
+      await Promise.all([getInitConfig(), initVectorStore(), initRootUser()]);

+      getSystemPluginCb();
      startMongoWatch();
      // cron
      startCron();
--- a/projects/app/src/web/core/app/diff.ts
+++ b/projects/app/src/web/core/app/diff.ts
@@ -0,0 +1,23 @@
+import { create } from 'jsondiffpatch';
+
+const createWorkflowDiffPatcher = () =>
+  create({
+    objectHash: (obj: any) => obj.id || obj.nodeId || obj._id,
+    propertyFilter: (name: string) => name !== 'selected'
+  });
+
+const diffPatcher = createWorkflowDiffPatcher();
+
+export const getAppDiffConfig = <T extends Record<string, unknown>>(
+  initialState?: T,
+  newState?: T
+) => {
+  return diffPatcher.diff(initialState, newState);
+};
+
+export const getAppConfigByDiff = <T extends Record<string, unknown>>(
+  initialState?: T,
+  diff?: ReturnType<typeof diffPatcher.diff>
+) => {
+  return diffPatcher.patch(structuredClone(initialState), diff) as T;
+};
--- a/projects/app/src/web/core/workflow/utils.ts
+++ b/projects/app/src/web/core/workflow/utils.ts
@@ -631,3 +631,13 @@ export const compareSnapshot = (

  return isEqual(node1, node2);
 };
+
+// remove node size
+export const simplifyWorkflowNodes = (nodes: Node[]) => {
+  return nodes.map((node) => ({
+    id: node.id,
+    type: node.type,
+    position: node.position,
+    data: node.data
+  }));
+};
Author	SHA1	Message	Date
Archer	019bf67e2d	Snip test (#3204 ) * fix: index * fix: snapshot error; perf: snapshot diff compare * perf: init simple edit history	2024-11-21 16:26:43 +08:00
heheer	9b2c3b242a	refactor: snapshot store to diff (#3155 ) * refactor: snapshot store to diff * change initial state position * fix old snapshot format * encapsulate json diff	2024-11-21 13:12:42 +08:00
Archer	4f55025906	rFix textspliter (#3200 ) * fix: text splitter * perf: splitter	2024-11-21 12:01:55 +08:00
Archer	489bb076a3	更新 configuration.md (#3188 )	2024-11-19 10:19:21 +08:00
Archer	a9db5b57c5	feat: add training retry time (#3187 ) * feat: add training retry time * remoce log	2024-11-18 20:57:03 +08:00
Archer	fdb3720b41	Update 4813.md (#3185 )	2024-11-18 18:34:08 +08:00
Archer	00641a8652	Update official_account.md (#3181 )	2024-11-18 14:54:35 +08:00
Archer	5c56b375c7	Update official_account.md (#3180 )	2024-11-18 14:28:01 +08:00
Archer	d8d9b936c4	fix: custom uid (#3177 ) * chat api doc * fix: custom uid	2024-11-18 10:27:52 +08:00
Archer	b237a3ec55	更新 4813.md (#3174 )	2024-11-16 10:32:27 +08:00
Archer	af894680cb	Update official_account.md (#3169 )	2024-11-15 15:26:28 +08:00
Archer	58745f8c35	4.8.14 test (#3164 ) * perf: match base 64 image * perf: register plugins	2024-11-15 10:35:04 +08:00
Archer	f699061dea	更新 plugin.ts (#3163 )	2024-11-14 23:11:33 +08:00
Archer	3f72f88591	perf: doc2x plugins (#3162 )	2024-11-14 21:56:13 +08:00
Menghuan1918	be59c2f6a7	更新Doc2X插件：适配新接口 (#3159 ) * fix: doc2x now not hava the picture API * fix: 适配doc2x V2 API * Update to axios to request doc2x * Add time out	2024-11-14 20:55:37 +08:00
Archer	795904a357	remove log (#3161 )	2024-11-14 20:31:01 +08:00
Archer	e8824987fa	4.8.14 test (#3160 ) * perf: remove base64 check * perf: update doc	2024-11-14 18:33:43 +08:00
Archer	b6d650adfb	Update 4813.md (#3158 )	2024-11-14 16:46:02 +08:00
Archer	710fa37847	fix: chat variable update (#3156 ) * perf: file encoding * fix: chat variable update	2024-11-14 15:50:47 +08:00
Archer	e22031ca6c	update doc (#3151 )	2024-11-14 10:45:05 +08:00
heheer	0c9e10dd2b	feat: add microsoft oauth (#3152 )	2024-11-14 10:33:49 +08:00
				`@@ -0,0 +1 @@`
				<svg t="1731513229844" class="icon" viewBox="0 0 1024 1024" version="1.1" xmlns="http://www.w3.org/2000/svg" p-id="5087" width="200" height="200"><path d="M493.0048 492.9536H128.0512V128H493.056v364.9536z" fill="#F1511B" p-id="5088"></path><path d="M895.9488 492.9536H530.944V128H896v364.9536z" fill="#80CC28" p-id="5089"></path><path d="M493.0048 896H128v-364.9024H493.056V896z" fill="#00ADEF" p-id="5090"></path><path d="M895.8976 896h-364.9024v-364.9024h364.9024V896z" fill="#FBBC09" p-id="5091"></path></svg>