Feat: Images dataset collection (#4941)

* New pic (#4858)

* 更新数据集相关类型,添加图像文件ID和预览URL支持;优化数据集导入功能,新增图像数据集处理组件;修复部分国际化文本;更新文件上传逻辑以支持新功能。

* 与原先代码的差别

* 新增 V4.9.10 更新说明,支持 PG 设置`systemEnv.hnswMaxScanTuples`参数,优化 LLM stream 调用超时,修复全文检索多知识库排序问题。同时更新数据集索引,移除 datasetId 字段以简化查询。

* 更换成fileId_image逻辑,并增加训练队列匹配的逻辑

* 新增图片集合判断逻辑,优化预览URL生成流程,确保仅在数据集为图片集合时生成预览URL,并添加相关日志输出以便调试。

* Refactor Docker Compose configuration to comment out exposed ports for production environments, update image versions for pgvector, fastgpt, and mcp_server, and enhance Redis service with a health check. Additionally, standardize dataset collection labels in constants and improve internationalization strings across multiple languages.

* Enhance TrainingStates component by adding internationalization support for the imageParse training mode and update defaultCounts to include imageParse mode in trainingDetail API.

* Enhance dataset import context by adding additional steps for image dataset import process and improve internationalization strings for modal buttons in the useEditTitle hook.

* Update DatasetImportContext to conditionally render MyStep component based on data source type, improving the import process for non-image datasets.

* Refactor image dataset handling by improving internationalization strings, enhancing error messages, and streamlining the preview URL generation process.

* 图片上传到新建的 dataset_collection_images 表,逻辑跟随更改

* 修改了除了controller的其他部分问题

* 把图片数据集的逻辑整合到controller里面

* 补充i18n

* 补充i18n

* resolve评论:主要是上传逻辑的更改和组件复用

* 图片名称的图标显示

* 修改编译报错的命名问题

* 删除不需要的collectionid部分

* 多余文件的处理和改动一个删除按钮

* 除了loading和统一的imageId,其他都resolve掉的

* 处理图标报错

* 复用了MyPhotoView并采用全部替换的方式将imageFileId变成imageId

* 去除不必要文件修改

* 报错和字段修改

* 增加上传成功后删除临时文件的逻辑以及回退一些修改

* 删除path字段,将图片保存到gridfs内,并修改增删等操作的代码

* 修正编译错误

---------

Co-authored-by: archer <545436317@qq.com>

* perf: image dataset

* feat: insert image

* perf: image icon

* fix: training state

---------

Co-authored-by: Zhuangzai fa <143257420+ctrlz526@users.noreply.github.com>
This commit is contained in:
Archer
2025-06-03 16:30:59 +08:00
committed by archer
parent 9fb5d05865
commit 92c38d9d2f
104 changed files with 2341 additions and 693 deletions

View File

@@ -26,16 +26,21 @@
"collection.training_type": "處理模式",
"collection_data_count": "資料量",
"collection_metadata_custom_pdf_parse": "PDF 增強解析",
"collection_name": "數據集名稱",
"collection_not_support_retraining": "此集合類型不支援重新調整參數",
"collection_not_support_sync": "該集合不支援同步",
"collection_sync": "立即同步",
"collection_sync_confirm_tip": "確認開始同步資料?\n系統將會拉取最新資料進行比較如果內容不相同則會建立一個新的集合並刪除舊的集合請確認",
"collection_tags": "集合標籤",
"common.dataset.data.Input Error Tip": "[圖片數據集] 處理過程錯誤:",
"common.error.unKnow": "未知錯誤",
"common_dataset": "通用資料集",
"common_dataset_desc": "通過導入文件、網頁鏈接或手動錄入形式構建知識庫",
"condition": "條件",
"config_sync_schedule": "設定定時同步",
"confirm_import_images": "共 {{num}} 張圖片 | 確認創建",
"confirm_to_rebuild_embedding_tip": "確定要為資料集切換索引嗎?\n切換索引是一個重要的操作需要對您資料集內所有資料重新建立索引可能需要較長時間請確保帳號內剩餘點數充足。\n\n此外您還需要注意修改使用此資料集的應用程式避免與其他索引模型資料集混用。",
"core.dataset.Image collection": "圖片數據集",
"core.dataset.import.Adjust parameters": "調整參數",
"custom_data_process_params": "自訂",
"custom_data_process_params_desc": "自訂資料處理規則",
@@ -88,6 +93,7 @@
"image_auto_parse": "圖片自動索引",
"image_auto_parse_tips": "呼叫 VLM 自動標註文件裡的圖片,並生成額外的檢索索引",
"image_training_queue": "圖片處理排隊",
"images_creating": "正在創建",
"immediate_sync": "立即同步",
"import.Auto mode Estimated Price Tips": "需呼叫文字理解模型,將消耗較多 AI 點數:{{price}} 點數 / 1K tokens",
"import.Embedding Estimated Price Tips": "僅使用索引模型,消耗少量 AI 點數:{{price}} 點數 / 1K tokens",
@@ -102,6 +108,8 @@
"index_size": "索引大小",
"index_size_tips": "向量化時內容的長度,系統會自動按該大小對分塊進行進一步的分割。",
"input_required_field_to_select_baseurl": "請先輸入必填信息",
"insert_images": "新增圖片",
"insert_images_success": "新增圖片成功,需等待訓練完成才會展示",
"is_open_schedule": "啟用定時同步",
"keep_image": "保留圖片",
"loading": "加載中...",
@@ -133,6 +141,7 @@
"process.Image_Index": "圖片索引生成",
"process.Is_Ready": "已就緒",
"process.Is_Ready_Count": "{{count}} 組已就緒",
"process.Parse_Image": "圖片解析中",
"process.Parsing": "內容解析中",
"process.Vectorizing": "索引向量化",
"process.Waiting": "排隊中",
@@ -174,11 +183,13 @@
"the_knowledge_base_has_indexes_that_are_being_trained_or_being_rebuilt": "資料集有索引正在訓練或重建中",
"total_num_files": "共 {{total}} 個文件",
"training.Error": "{{count}} 組異常",
"training.Image mode": "圖片處理",
"training.Normal": "正常",
"training_mode": "分段模式",
"training_ready": "{{count}} 組",
"vector_model_max_tokens_tip": "每個分塊資料,最大長度為 3000 tokens",
"vllm_model": "圖片理解模型",
"vlm_model_required_warning": "需要圖片理解模型",
"website_dataset": "網站同步",
"website_dataset_desc": "通過爬蟲,批量爬取網頁數據構建知識庫",
"website_info": "網站資訊",