Compare commits

...

17 Commits

Author SHA1 Message Date
Archer
15dc7b220e perf: snap code (#3350)
* perf: readme

* perf: snap code
2024-12-09 18:27:40 +08:00
heheer
c64d629a6a fix: loop node snapshot & infinite reload (#3349) 2024-12-09 18:18:33 +08:00
Archer
1a294c1fd3 perf: load plugin groups code;perf: system plugin schema;fix: special variables replace;perf: retry cron app job (#3347)
* perf: load plugin groups code

* perf: system plugin schema

* feat: retry cron app job

* fix: special variables replace
2024-12-09 17:18:07 +08:00
Archer
a4f6128a89 Update 4814.md (#3345) 2024-12-09 14:50:54 +08:00
heheer
021ec0595d fix: plugin group api position & team selector optional display (#3339) 2024-12-09 09:55:10 +08:00
YeYuheng
6ceee7cb5e marker doc (#3335)
* marker doc

* marker doc
2024-12-07 20:28:40 +08:00
Archer
90d7d2a164 perf: workflow snapshots;fix: extrat node selected (#3334)
* doc

* perf: workflow snapshots

* fix: extrat node selected

* refresh page reset snapshot
2024-12-06 18:08:52 +08:00
imgbot[bot]
0c308fcf8b [ImgBot] Optimize images (#3333)
*Total -- 16,249.30kb -> 13,885.08kb (14.55%)

/docSite/assets/imgs/translate7.png -- 238.17kb -> 92.79kb (61.04%)
/docSite/assets/imgs/image-16.png -- 138.74kb -> 72.25kb (47.92%)
/docSite/assets/imgs/api-dataset-1.png -- 96.68kb -> 56.44kb (41.62%)
/docSite/assets/imgs/integration1.png -- 2.11kb -> 1.23kb (41.47%)
/docSite/assets/imgs/image-17.png -- 222.24kb -> 138.71kb (37.59%)
/docSite/assets/imgs/image-19.png -- 167.16kb -> 108.02kb (35.38%)
/docSite/assets/imgs/document_analysis1.png -- 37.93kb -> 24.90kb (34.37%)
/docSite/assets/imgs/image-9.png -- 82.66kb -> 54.82kb (33.68%)
/docSite/assets/imgs/knowledge_merge1.png -- 25.64kb -> 17.10kb (33.29%)
/docSite/assets/imgs/flow-tool1.png -- 54.42kb -> 36.44kb (33.04%)
/docSite/assets/imgs/image-18.png -- 157.53kb -> 105.89kb (32.78%)
/docSite/assets/imgs/image-8.png -- 116.99kb -> 79.61kb (31.95%)
/docSite/assets/imgs/image-12.png -- 179.21kb -> 122.00kb (31.92%)
/docSite/assets/imgs/image-10.png -- 81.53kb -> 55.61kb (31.79%)
/docSite/assets/imgs/image-13.png -- 172.11kb -> 117.71kb (31.61%)
/docSite/assets/imgs/image-7.png -- 416.11kb -> 287.61kb (30.88%)
/docSite/assets/imgs/image-6.png -- 471.28kb -> 329.27kb (30.13%)
/docSite/assets/imgs/image-5.png -- 371.50kb -> 263.09kb (29.18%)
/docSite/assets/imgs/document_analysis2.png -- 26.52kb -> 18.92kb (28.65%)
/docSite/assets/imgs/htmlRendering1.png -- 70.58kb -> 50.45kb (28.52%)
/.github/imgs/image.png -- 20.82kb -> 15.08kb (27.54%)
/docSite/assets/imgs/flow-tool6.png -- 15.63kb -> 11.35kb (27.39%)
/docSite/assets/imgs/variable_update1.png -- 19.61kb -> 14.98kb (23.59%)
/docSite/assets/imgs/translate10.png -- 84.00kb -> 65.14kb (22.45%)
/docSite/assets/imgs/google_search_plugin4.png -- 50.35kb -> 39.31kb (21.92%)
/docSite/assets/imgs/translate8.png -- 68.96kb -> 54.29kb (21.27%)
/docSite/assets/imgs/bing_search_plugin5.png -- 54.46kb -> 42.89kb (21.24%)
/docSite/assets/imgs/form_input1.png -- 52.12kb -> 41.63kb (20.12%)
/docSite/assets/imgs/translate9.png -- 106.28kb -> 86.35kb (18.75%)
/docSite/assets/imgs/flow-tool7.png -- 36.86kb -> 30.04kb (18.51%)
/docSite/assets/imgs/image-11.png -- 819.07kb -> 669.89kb (18.21%)
/docSite/assets/imgs/template_submission3.png -- 283.37kb -> 232.65kb (17.9%)
/docSite/assets/imgs/doc2x_plugin3.png -- 121.64kb -> 100.78kb (17.15%)
/docSite/assets/imgs/spellcheck5.png -- 36.60kb -> 30.87kb (15.65%)
/docSite/assets/imgs/knowledge_merge3.png -- 179.32kb -> 151.42kb (15.56%)
/docSite/assets/imgs/RAG1.png -- 381.41kb -> 322.90kb (15.34%)
/docSite/assets/imgs/image-15.png -- 257.77kb -> 219.79kb (14.74%)
/docSite/assets/imgs/image-14.png -- 599.17kb -> 511.20kb (14.68%)
/docSite/assets/imgs/spellcheck3.png -- 48.21kb -> 41.18kb (14.58%)
/docSite/assets/imgs/doc2x_plugin4.png -- 62.40kb -> 53.33kb (14.53%)
/docSite/assets/imgs/spellcheck2.png -- 128.33kb -> 110.10kb (14.2%)
/docSite/assets/imgs/spellcheck1.png -- 63.44kb -> 54.54kb (14.02%)
/docSite/assets/imgs/spellcheck4.png -- 153.90kb -> 132.41kb (13.97%)
/docSite/assets/imgs/translate12.png -- 176.14kb -> 151.87kb (13.78%)
/docSite/assets/imgs/translate11.png -- 237.58kb -> 206.44kb (13.11%)
/docSite/assets/imgs/knowledge_merge2.png -- 123.49kb -> 107.44kb (13%)
/docSite/assets/imgs/translate4.png -- 174.85kb -> 153.32kb (12.31%)
/docSite/assets/imgs/flow-tool3.png -- 163.79kb -> 143.71kb (12.26%)
/docSite/assets/imgs/user-selection1.png -- 60.21kb -> 52.85kb (12.23%)
/docSite/assets/imgs/plugin_submission3.png -- 182.66kb -> 161.34kb (11.67%)
/docSite/assets/imgs/plugin_submission5.png -- 188.61kb -> 169.78kb (9.99%)
/docSite/assets/imgs/RAG3.png -- 77.61kb -> 69.96kb (9.86%)
/docSite/assets/imgs/flow-tool4.png -- 152.11kb -> 138.04kb (9.25%)
/docSite/assets/imgs/form_input3.png -- 173.80kb -> 158.12kb (9.02%)
/docSite/assets/imgs/translate15.png -- 126.36kb -> 115.04kb (8.95%)
/docSite/assets/imgs/translate2.png -- 315.54kb -> 287.55kb (8.87%)
/docSite/assets/imgs/translate14.png -- 216.02kb -> 196.93kb (8.84%)
/docSite/assets/imgs/translate13.png -- 257.41kb -> 235.13kb (8.66%)
/docSite/assets/imgs/google_search_plugin3.png -- 207.39kb -> 189.77kb (8.49%)
/docSite/assets/imgs/flow-tool5.png -- 337.84kb -> 309.44kb (8.4%)
/docSite/assets/imgs/translate3.png -- 341.09kb -> 312.54kb (8.37%)
/docSite/assets/imgs/form_input2.png -- 147.56kb -> 136.13kb (7.75%)
/docSite/assets/imgs/bing_search_plugin2.png -- 222.07kb -> 205.64kb (7.4%)
/docSite/assets/imgs/translate1.png -- 261.13kb -> 242.17kb (7.26%)
/docSite/assets/imgs/translate20.png -- 433.50kb -> 403.10kb (7.01%)
/docSite/assets/imgs/bing_search_plugin3.png -- 145.72kb -> 135.72kb (6.86%)
/docSite/assets/imgs/translate18.png -- 441.89kb -> 411.76kb (6.82%)
/docSite/assets/imgs/variable_update2.png -- 179.43kb -> 167.96kb (6.39%)
/docSite/assets/imgs/google_search_plugin2.png -- 84.19kb -> 78.98kb (6.18%)
/docSite/assets/imgs/variable_update3.png -- 189.57kb -> 178.13kb (6.03%)
/docSite/assets/imgs/translate5.png -- 463.44kb -> 436.15kb (5.89%)
/docSite/assets/imgs/plugin_submission4.png -- 156.23kb -> 147.34kb (5.69%)
/docSite/assets/imgs/points1.png -- 146.54kb -> 138.21kb (5.69%)
/docSite/assets/imgs/google_search_plugin1.png -- 51.28kb -> 48.38kb (5.66%)
/docSite/assets/imgs/flow-tool2.png -- 70.64kb -> 66.65kb (5.65%)
/docSite/assets/imgs/translate61.png -- 590.71kb -> 558.60kb (5.44%)
/docSite/assets/imgs/bing_search_plugin1.png -- 101.07kb -> 95.63kb (5.38%)
/docSite/assets/imgs/translate16.png -- 88.25kb -> 83.76kb (5.1%)
/docSite/assets/imgs/user-selection2.png -- 162.55kb -> 154.34kb (5.05%)
/docSite/assets/imgs/user-selection3.png -- 179.79kb -> 170.84kb (4.98%)
/docSite/assets/imgs/template_submission1.png -- 177.66kb -> 169.88kb (4.38%)
/docSite/assets/imgs/translate19.png -- 464.75kb -> 448.85kb (3.42%)
/docSite/assets/imgs/bing_search_plugin4.png -- 91.43kb -> 88.42kb (3.3%)
/docSite/assets/imgs/variable_update5.png -- 185.49kb -> 179.44kb (3.26%)
/docSite/assets/imgs/variable_update4.png -- 174.67kb -> 169.14kb (3.17%)
/docSite/assets/imgs/translate21.png -- 66.15kb -> 64.78kb (2.07%)
/docSite/assets/imgs/plugin_submission1.png -- 209.81kb -> 206.46kb (1.6%)
/docSite/assets/imgs/htmlRendering3.png -- 58.24kb -> 57.57kb (1.16%)
/docSite/assets/imgs/RAG2.png -- 44.26kb -> 43.84kb (0.93%)
/docSite/assets/imgs/translate17.png -- 284.26kb -> 281.84kb (0.85%)
/docSite/assets/imgs/htmlRendering2.png -- 91.74kb -> 91.13kb (0.66%)

Signed-off-by: ImgBotApp <ImgBotHelp@gmail.com>
Co-authored-by: ImgBotApp <ImgBotHelp@gmail.com>
2024-12-06 11:13:35 +08:00
Archer
1aebe5f185 V4.8.15 feature (#3331)
* feat: add customize toolkit (#3205)

* chaoyang

* fix-auth

* add toolkit

* add order

* plugin usage

* fix

* delete console:

* Fix: Fix fullscreen preview top positioning and improve Markdown rendering logic (#3247)

* 完成任务:修复全屏预览顶部固定问题,优化 Markdown 渲染逻辑

* 有问题修改

* 问题再修改

* 修正问题

* fix: plugin standalone display issue (#3254)

* 4.8.15 test (#3246)

* o1 config

* perf: system plugin code

* 调整系统插件代码。增加html 渲染安全配置。 (#3258)

* perf: base64 picker

* perf: list app or dataset

* perf: plugin config code

* 小窗适配等问题 (#3257)

* 小窗适配等问题

* git问题

* 小窗剩余问题

* feat: system plugin auth and lock version (#3265)

* feat: system plugin auth and lock version

* update comment

* 4.8.15 test (#3267)

* tmp log

* perf: login direct

* perf: iframe html code

* remove log

* fix: plugin standalone display (#3277)

* refactor: 页面拆分&i18n拆分 (#3281)

* refactor: account组件拆成独立页面

* script: 新增i18n json文件创建脚本

* refactor: 页面i18n拆分

* i18n: add en&hant

* 4.8.15 test (#3285)

* tmp log

* remove log

* fix: watch avatar refresh

* perf: i18n code

* fix(plugin): use intro instead of userguide (#3290)

* Universal SSO (#3292)

* tmp log

* remove log

* feat: common oauth

* readme

* perf: sso provider

* remove sso code

* perf: refresh plugins

* feat: add api dataset (#3272)

* add api-dataset

* fix api-dataset

* fix api dataset

* fix ts

* perf: create collection code (#3301)

* tmp log

* remove log

* perf: i18n change

* update version doc

* feat: question guide from chatId

* perf: create collection code

* fix: request api

* fix: request api

* fix: tts auth and response type (#3303)

* perf: md splitter

* fix: tts auth and response type

* fix: api file dataset (#3307)

* perf: api dataset init (#3310)

* perf: collection schema

* perf: api dataset init

* refactor: 团队管理独立页面 (#3302)

* ui: 团队管理独立页面

* 代码优化

* fix

* perf: sync collection and ui check (#3314)

* perf: sync collection

* remove script

* perf: update api server

* perf: api dataset parent

* perf: team ui

* perf: team 18n

* update team ui

* perf: ui check

* perf: i18n

* fix: debug variables & cronjob & system plugin callback load (#3315)

* fix: debug variables & cronjob & system plugin callback load

* fix type

* fix

* fix

* fix: plugin dataset quote;perf: system variables init (#3316)

* fix: plugin dataset quote

* perf: system variables init

* perf: node templates ui;fix: dataset import ui (#3318)

* fix: dataset import ui

* perf: node templates ui

* perf: ui refresh

* feat:套餐改名和套餐跳转配置 (#3309)

* fixing:except Sidebar

* 去除了多余的代码

* 修正了套餐说明的代码

* 修正了误删除的show_git代码

* 修正了名字部分等代码

* 修正了问题,遗留了其他和ui讨论不一致的部分

* 4.8.15 test (#3319)

* remove log

* pref: bill ui

* pref: bill ui

* perf: log

* html渲染文档 (#3270)

* html渲染文档

* 文档有点小问题

* feat: doc (#3322)

* 集合重训练 (#3282)

* rebaser

* 一点补充

* 小问题

* 其他问题修正,删除集合保留文件的参数还没找到...

* reTraining

* delete uesless

* 删除了一行错误代码

* 集合重训练部分

* fixing

* 删除console代码

* feat: navbar item config (#3326)

* perf: custom navbar code;perf: retraining code;feat: api dataset and dataset api doc (#3329)

* feat: api dataset and dataset api doc

* perf: retraining code

* perf: custom navbar code

* fix: ts (#3330)

* fix: ts

* fix: ts

* retraining ui

* perf: api collection filter

* perf: retrining button

---------

Co-authored-by: heheer <heheer@sealos.io>
Co-authored-by: Jiangween <145003935+Jiangween@users.noreply.github.com>
Co-authored-by: papapatrick <109422393+Patrickill@users.noreply.github.com>
2024-12-06 10:56:53 +08:00
Archer
b188544386 Update 4814.md (#3325) 2024-12-05 17:43:42 +08:00
heheer
7faa427e84 microsoft oauth tenantId & custom button name (#3300) 2024-12-03 15:28:56 +08:00
Archer
cb56d1e53e fix: ts (#3296) 2024-12-03 00:11:38 +08:00
Archer
f28b7e41a8 fix: auto send prompt runtime (#3295) 2024-12-03 00:02:56 +08:00
heheer
c506442993 fix simple snapshot init & auto excute when change app (#3293) 2024-12-02 21:40:31 +08:00
Archer
1cef206c13 Fix share page feedback auth (#3284)
* fix: share page mark auth

* perf: lang check

* perf: load share
2024-12-01 21:09:13 +08:00
gallonyin
d0e8c9c62e Fix: Sandbox Dockerfile (#3280)
修复打包异常,异常信息:
Step 17/28 : COPY package.json pnpm-workspace.yaml /app
When using COPY with more than one source file, the destination must be a directory and end with a /
2024-11-30 11:09:51 +08:00
Archer
a4a8b7909c Update README.md (#3276) 2024-11-29 16:51:43 +08:00
420 changed files with 8162 additions and 4790 deletions

BIN
.github/imgs/image.png vendored

Binary file not shown.

Before

Width:  |  Height:  |  Size: 21 KiB

After

Width:  |  Height:  |  Size: 15 KiB

View File

@@ -27,9 +27,6 @@ FastGPT 是一个基于 LLM 大语言模型的知识库问答系统,提供开
<a href="/#-%E7%9B%B8%E5%85%B3%E9%A1%B9%E7%9B%AE">
<img height="21" src="https://img.shields.io/badge/相关项目-7d09f1?style=flat-square" alt="project">
</a>
<a href="https://github.com/labring/FastGPT/blob/main/LICENSE">
<img height="21" src="https://img.shields.io/badge/License-Apache--2.0-ffffff?style=flat-square&labelColor=d4eaf7&color=7d09f1" alt="license">
</a>
</p>
https://github.com/labring/FastGPT/assets/15308462/7d3a38df-eb0e-4388-9250-2409bd33f6d4
@@ -118,7 +115,7 @@ https://github.com/labring/FastGPT/assets/15308462/7d3a38df-eb0e-4388-9250-2409b
## 🏘️ 社区交流群
扫码加入飞书话题群 (新开,逐渐弃用微信群)
扫码加入飞书话题群:
![](https://oss.laf.run/otnvvf-imgs/fastgpt-feishu1.png)
@@ -126,6 +123,11 @@ https://github.com/labring/FastGPT/assets/15308462/7d3a38df-eb0e-4388-9250-2409b
<img src="https://img.shields.io/badge/-返回顶部-7d09f1.svg" alt="#" align="right">
</a>
## 🏘️ 加入我们
我们正在寻找志同道合的小伙伴,加速 FastGPT 的发展。你可以通过 [FastGPT 2025 招聘](https://fael3z0zfze.feishu.cn/wiki/P7FOwEmPziVcaYkvVaacnVX1nvg)了解 FastGPT 的招聘信息。
## 💪 相关项目
- [Laf3 分钟快速接入三方应用](https://github.com/labring/laf)
@@ -171,7 +173,7 @@ https://github.com/labring/FastGPT/assets/15308462/7d3a38df-eb0e-4388-9250-2409b
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://next.ossinsight.io/widgets/official/compose-org-active-contributors/thumbnail.png?activity=active&period=past_28_days&owner_id=102226726&repo_ids=605673387&image_size=2x3&color_scheme=dark">
<img alt="Active participants of labring - past 28 days" src="https://next.ossinsight.io/widgets/official/compose-org-active-contributors/thumbnail.png?activity=active&period=past_28_days&owner_id=102226726&repo_ids=605673387&image_size=2x3&color_scheme=light">
</picture>
</picture>****
</td>
<td rowspan="2">
<picture>

Binary file not shown.

Before

Width:  |  Height:  |  Size: 381 KiB

After

Width:  |  Height:  |  Size: 323 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 44 KiB

After

Width:  |  Height:  |  Size: 44 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 78 KiB

After

Width:  |  Height:  |  Size: 70 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 56 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 101 KiB

After

Width:  |  Height:  |  Size: 96 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 222 KiB

After

Width:  |  Height:  |  Size: 206 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 146 KiB

After

Width:  |  Height:  |  Size: 136 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 91 KiB

After

Width:  |  Height:  |  Size: 88 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 54 KiB

After

Width:  |  Height:  |  Size: 43 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 122 KiB

After

Width:  |  Height:  |  Size: 101 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 62 KiB

After

Width:  |  Height:  |  Size: 53 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 38 KiB

After

Width:  |  Height:  |  Size: 25 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 26 KiB

After

Width:  |  Height:  |  Size: 19 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 54 KiB

After

Width:  |  Height:  |  Size: 36 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 71 KiB

After

Width:  |  Height:  |  Size: 67 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 164 KiB

After

Width:  |  Height:  |  Size: 144 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 152 KiB

After

Width:  |  Height:  |  Size: 138 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 338 KiB

After

Width:  |  Height:  |  Size: 309 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 16 KiB

After

Width:  |  Height:  |  Size: 11 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 37 KiB

After

Width:  |  Height:  |  Size: 30 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 52 KiB

After

Width:  |  Height:  |  Size: 42 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 148 KiB

After

Width:  |  Height:  |  Size: 136 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 174 KiB

After

Width:  |  Height:  |  Size: 158 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 51 KiB

After

Width:  |  Height:  |  Size: 48 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 84 KiB

After

Width:  |  Height:  |  Size: 79 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 207 KiB

After

Width:  |  Height:  |  Size: 190 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 50 KiB

After

Width:  |  Height:  |  Size: 39 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 50 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 91 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 58 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 56 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 670 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 122 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 118 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 511 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 220 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 72 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 139 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 106 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 108 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 156 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 166 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 85 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 113 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 72 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 372 KiB

After

Width:  |  Height:  |  Size: 263 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 471 KiB

After

Width:  |  Height:  |  Size: 329 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 416 KiB

After

Width:  |  Height:  |  Size: 288 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 117 KiB

After

Width:  |  Height:  |  Size: 80 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 83 KiB

After

Width:  |  Height:  |  Size: 55 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 2.1 KiB

After

Width:  |  Height:  |  Size: 1.2 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 26 KiB

After

Width:  |  Height:  |  Size: 17 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 124 KiB

After

Width:  |  Height:  |  Size: 107 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 179 KiB

After

Width:  |  Height:  |  Size: 151 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 210 KiB

After

Width:  |  Height:  |  Size: 206 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 183 KiB

After

Width:  |  Height:  |  Size: 161 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 156 KiB

After

Width:  |  Height:  |  Size: 147 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 189 KiB

After

Width:  |  Height:  |  Size: 170 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 146 KiB

After

Width:  |  Height:  |  Size: 138 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 63 KiB

After

Width:  |  Height:  |  Size: 54 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 128 KiB

After

Width:  |  Height:  |  Size: 110 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 48 KiB

After

Width:  |  Height:  |  Size: 41 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 154 KiB

After

Width:  |  Height:  |  Size: 132 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 37 KiB

After

Width:  |  Height:  |  Size: 31 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 178 KiB

After

Width:  |  Height:  |  Size: 170 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 283 KiB

After

Width:  |  Height:  |  Size: 233 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 261 KiB

After

Width:  |  Height:  |  Size: 242 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 84 KiB

After

Width:  |  Height:  |  Size: 65 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 238 KiB

After

Width:  |  Height:  |  Size: 206 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 176 KiB

After

Width:  |  Height:  |  Size: 152 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 257 KiB

After

Width:  |  Height:  |  Size: 235 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 216 KiB

After

Width:  |  Height:  |  Size: 197 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 126 KiB

After

Width:  |  Height:  |  Size: 115 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 88 KiB

After

Width:  |  Height:  |  Size: 84 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 284 KiB

After

Width:  |  Height:  |  Size: 282 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 442 KiB

After

Width:  |  Height:  |  Size: 412 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 465 KiB

After

Width:  |  Height:  |  Size: 449 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 316 KiB

After

Width:  |  Height:  |  Size: 288 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 434 KiB

After

Width:  |  Height:  |  Size: 403 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 66 KiB

After

Width:  |  Height:  |  Size: 65 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 341 KiB

After

Width:  |  Height:  |  Size: 312 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 175 KiB

After

Width:  |  Height:  |  Size: 153 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 463 KiB

After

Width:  |  Height:  |  Size: 436 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 591 KiB

After

Width:  |  Height:  |  Size: 559 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 238 KiB

After

Width:  |  Height:  |  Size: 93 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 69 KiB

After

Width:  |  Height:  |  Size: 54 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 106 KiB

After

Width:  |  Height:  |  Size: 86 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 60 KiB

After

Width:  |  Height:  |  Size: 53 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 162 KiB

After

Width:  |  Height:  |  Size: 154 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 180 KiB

After

Width:  |  Height:  |  Size: 171 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 20 KiB

After

Width:  |  Height:  |  Size: 15 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 179 KiB

After

Width:  |  Height:  |  Size: 168 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 190 KiB

After

Width:  |  Height:  |  Size: 178 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 175 KiB

After

Width:  |  Height:  |  Size: 169 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 186 KiB

After

Width:  |  Height:  |  Size: 179 KiB

View File

@@ -43,7 +43,7 @@ weight: 708
"usedInExtractFields": true, // 是否用于内容提取务必保证至少有一个为true
"usedInToolCall": true, // 是否用于工具调用务必保证至少有一个为true
"usedInQueryExtension": true, // 是否用于问题优化务必保证至少有一个为true
"toolChoice": true, // 是否支持工具选择(分类,内容提取,工具调用会用到。目前只有gpt支持
"toolChoice": true, // 是否支持工具选择(分类,内容提取,工具调用会用到。)
"functionCall": false, // 是否支持函数调用(分类,内容提取,工具调用会用到。会优先使用 toolChoice如果为false则使用 functionCall如果仍为 false则使用提示词模式
"customCQPrompt": "", // 自定义文本分类提示词(不支持工具和函数调用的模型
"customExtractPrompt": "", // 自定义内容提取提示词
@@ -95,9 +95,7 @@ weight: 708
"customExtractPrompt": "",
"defaultSystemChatPrompt": "",
"defaultConfig": {
"temperature": 1,
"max_tokens": null,
"stream": false
"temperature": 1
}
},
{
@@ -122,9 +120,7 @@ weight: 708
"customExtractPrompt": "",
"defaultSystemChatPrompt": "",
"defaultConfig": {
"temperature": 1,
"max_tokens": null,
"stream": false
"temperature": 1
}
}
],

View File

@@ -0,0 +1,66 @@
---
title: '接入 Marker PDF 文档解析'
description: '使用 Marker 解析 PDF 文档,可实现图片提取和布局识别'
icon: 'api'
draft: false
toc: true
weight: 909
---
## 背景
PDF 是一个相对复杂的文件格式,在 FastGPT 内置的 pdf 解析器中,依赖的是 pdfjs 库解析,该库基于逻辑解析,无法有效的理解复杂的 pdf 文件。所以我们在解析 pdf 时候,如果遇到图片、表格、公式等非简单文本内容,会发现解析效果不佳。
市面上目前有多种解析 PDF 的方法,比如使用 [Marker](https://github.com/VikParuchuri/marker),该项目使用了 Surya 模型,基于视觉解析,可以有效提取图片、表格、公式等复杂内容。为了可以让 Marker 快速接入 FastGPT我们做了一个自定义解析的拓展 Demo。
在 FastGPT 4.8.15 版本中,你可以通过增加一个环境变量,来替换掉 FastGPT 系统内置解析器,实现自定义的文档解析服务。该功能只是 Demo 阶段,后期配置模式和交互规则会发生改动。
## 使用教程
### 1. 按照 Marker
参考文档 [Marker 安装教程](https://github.com/labring/FastGPT/tree/main/python/pdf-marker),安装 Marker 模型。封装的 API 已经适配了 FastGPT 自定义解析服务。
这里介绍快速 Docker 安装的方法:
```dockerfile
docker pull crpi-h3snc261q1dosroc.cn-hangzhou.personal.cr.aliyuncs.com/marker11/marker_images:latest
docker run --gpus all -itd -p 7231:7231 --name model_pdf_v1 crpi-h3snc261q1dosroc.cn-hangzhou.personal.cr.aliyuncs.com/marker11/marker_images:latest
```
### 2. 添加 FastGPT 环境变量
```
CUSTOM_READ_FILE_URL=http://xxxx.com/v1/parse/file
CUSTOM_READ_FILE_EXTENSION=pdf
```
* CUSTOM_READ_FILE_URL - 自定义解析服务的地址, host改成解析服务的访问地址path 不能变动。
* CUSTOM_READ_FILE_EXTENSION - 支持的文件后缀,多个文件类型,可用逗号隔开。
### 3. 测试效果
通过知识库上传一个 pdf 文件,并确认上传,可以在日志中看到 LOG LOG_LEVEL需要设置 info 或者 debug
```
[Info] 2024-12-05 15:04:42 Parsing files from an external service
[Info] 2024-12-05 15:07:08 Custom file parsing is complete, time: 1316ms
```
然后你就可以发现,通过 Marker 解析出来的 pdf 会携带图片链接:
![alt text](/imgs/image-10.png)
## 效果展示
以清华的 [ChatDev Communicative Agents for Software Develop.pdf](https://arxiv.org/abs/2307.07924) 为例,展示 Marker 解析的效果:
| | | |
| --- | --- | --- |
| ![alt text](/imgs/image-11.png) | ![alt text](/imgs/image-12.png) | ![alt text](/imgs/image-13.png) |
| ![alt text](/imgs/image-14.png) | ![alt text](/imgs/image-15.png) | ![alt text](/imgs/image-16.png) |
上图是分块后的结果,下图是 pdf 原文。整体图片、公式、表格都可以提取出来,效果还是杠杠的。
不过要注意的是,[Marker](https://github.com/VikParuchuri/marker) 的协议是`GPL-3.0 license`,请在遵守协议的前提下使用。

View File

@@ -145,7 +145,7 @@ curl --location --request POST 'https://<oneapi_url>/v1/chat/completions' \
"usedInExtractFields": true, // 是否用于内容提取务必保证至少有一个为true
"usedInToolCall": true, // 是否用于工具调用务必保证至少有一个为true
"usedInQueryExtension": true, // 是否用于问题优化务必保证至少有一个为true
"toolChoice": true, // 是否支持工具选择(分类,内容提取,工具调用会用到。目前只有gpt支持
"toolChoice": true, // 是否支持工具选择(分类,内容提取,工具调用会用到。)
"functionCall": false, // 是否支持函数调用(分类,内容提取,工具调用会用到。会优先使用 toolChoice如果为false则使用 functionCall如果仍为 false则使用提示词模式
"customCQPrompt": "", // 自定义文本分类提示词(不支持工具和函数调用的模型
"customExtractPrompt": "", // 自定义内容提取提示词

Some files were not shown because too many files have changed in this diff Show More