Feat: Images dataset collection (#4941)

* New pic (#4858)

* 更新数据集相关类型,添加图像文件ID和预览URL支持;优化数据集导入功能,新增图像数据集处理组件;修复部分国际化文本;更新文件上传逻辑以支持新功能。

* 与原先代码的差别

* 新增 V4.9.10 更新说明,支持 PG 设置`systemEnv.hnswMaxScanTuples`参数,优化 LLM stream 调用超时,修复全文检索多知识库排序问题。同时更新数据集索引,移除 datasetId 字段以简化查询。

* 更换成fileId_image逻辑,并增加训练队列匹配的逻辑

* 新增图片集合判断逻辑,优化预览URL生成流程,确保仅在数据集为图片集合时生成预览URL,并添加相关日志输出以便调试。

* Refactor Docker Compose configuration to comment out exposed ports for production environments, update image versions for pgvector, fastgpt, and mcp_server, and enhance Redis service with a health check. Additionally, standardize dataset collection labels in constants and improve internationalization strings across multiple languages.

* Enhance TrainingStates component by adding internationalization support for the imageParse training mode and update defaultCounts to include imageParse mode in trainingDetail API.

* Enhance dataset import context by adding additional steps for image dataset import process and improve internationalization strings for modal buttons in the useEditTitle hook.

* Update DatasetImportContext to conditionally render MyStep component based on data source type, improving the import process for non-image datasets.

* Refactor image dataset handling by improving internationalization strings, enhancing error messages, and streamlining the preview URL generation process.

* 图片上传到新建的 dataset_collection_images 表,逻辑跟随更改

* 修改了除了controller的其他部分问题

* 把图片数据集的逻辑整合到controller里面

* 补充i18n

* 补充i18n

* resolve评论:主要是上传逻辑的更改和组件复用

* 图片名称的图标显示

* 修改编译报错的命名问题

* 删除不需要的collectionid部分

* 多余文件的处理和改动一个删除按钮

* 除了loading和统一的imageId,其他都resolve掉的

* 处理图标报错

* 复用了MyPhotoView并采用全部替换的方式将imageFileId变成imageId

* 去除不必要文件修改

* 报错和字段修改

* 增加上传成功后删除临时文件的逻辑以及回退一些修改

* 删除path字段,将图片保存到gridfs内,并修改增删等操作的代码

* 修正编译错误

---------

Co-authored-by: archer <545436317@qq.com>

* perf: image dataset

* feat: insert image

* perf: image icon

* fix: training state

---------

Co-authored-by: Zhuangzai fa <143257420+ctrlz526@users.noreply.github.com>
This commit is contained in:
Archer
2025-06-03 16:30:59 +08:00
committed by archer
parent 9fb5d05865
commit 92c38d9d2f
104 changed files with 2341 additions and 693 deletions

View File

@@ -220,9 +220,11 @@ export const iconPaths = {
import('./icons/core/dataset/feishuDatasetOutline.svg'),
'core/dataset/fileCollection': () => import('./icons/core/dataset/fileCollection.svg'),
'core/dataset/fullTextRecall': () => import('./icons/core/dataset/fullTextRecall.svg'),
'core/dataset/imageFill': () => import('./icons/core/dataset/imageFill.svg'),
'core/dataset/manualCollection': () => import('./icons/core/dataset/manualCollection.svg'),
'core/dataset/mixedRecall': () => import('./icons/core/dataset/mixedRecall.svg'),
'core/dataset/modeEmbedding': () => import('./icons/core/dataset/modeEmbedding.svg'),
'core/dataset/otherDataset': () => import('./icons/core/dataset/otherDataset.svg'),
'core/dataset/questionExtension': () => import('./icons/core/dataset/questionExtension.svg'),
'core/dataset/rerank': () => import('./icons/core/dataset/rerank.svg'),
'core/dataset/searchfilter': () => import('./icons/core/dataset/searchfilter.svg'),
@@ -230,7 +232,6 @@ export const iconPaths = {
'core/dataset/tableCollection': () => import('./icons/core/dataset/tableCollection.svg'),
'core/dataset/tag': () => import('./icons/core/dataset/tag.svg'),
'core/dataset/websiteDataset': () => import('./icons/core/dataset/websiteDataset.svg'),
'core/dataset/otherDataset': () => import('./icons/core/dataset/otherDataset.svg'),
'core/dataset/websiteDatasetColor': () => import('./icons/core/dataset/websiteDatasetColor.svg'),
'core/dataset/websiteDatasetOutline': () =>
import('./icons/core/dataset/websiteDatasetOutline.svg'),
@@ -379,10 +380,12 @@ export const iconPaths = {
fullScreen: () => import('./icons/fullScreen.svg'),
help: () => import('./icons/help.svg'),
history: () => import('./icons/history.svg'),
image: () => import('./icons/image.svg'),
infoRounded: () => import('./icons/infoRounded.svg'),
kbTest: () => import('./icons/kbTest.svg'),
key: () => import('./icons/key.svg'),
keyPrimary: () => import('./icons/keyPrimary.svg'),
loading: () => import('./icons/loading.svg'),
menu: () => import('./icons/menu.svg'),
minus: () => import('./icons/minus.svg'),
'modal/AddClb': () => import('./icons/modal/AddClb.svg'),