Compare commits

...

5 Commits

Author SHA1 Message Date
Archer
84cf6b5658 v4.6.4 (#585)
* perf: md format

* add systemConfig schema (#2)

* fix: markdown

* fix: root

* fix: root

---------

Co-authored-by: heheer <71265218+newfish-cmyk@users.noreply.github.com>
2023-12-08 16:33:15 +08:00
Archer
b58249fc3a 4.6.4-alpha (#582) 2023-12-08 15:01:11 +08:00
angular-moon
54d52d8d25 fixed: Adjust the api authentication sequence to preferentially process authentication to avoid incorrect authentication caused by cookies (#572) 2023-12-07 18:19:18 +08:00
Carson Yang
f298b90b69 Docs: update theme version (#571)
Signed-off-by: Carson Yang <yangchuansheng33@gmail.com>
2023-12-07 18:07:57 +08:00
Archer
e01c38efe0 4.6.4-alpha (#569) 2023-12-07 13:43:08 +08:00
140 changed files with 2439 additions and 1655 deletions

View File

@@ -1,94 +1,175 @@
---
title: '分享链接鉴权'
description: 'FastGPT 分享链接鉴权'
title: '分享链接身份鉴权'
description: 'FastGPT 分享链接身份鉴权'
icon: 'share'
draft: false
toc: true
weight: 860
---
## 介绍
在 FastGPT V4.6.4 中,我们修改了分享链接的数据读取方式,为每个用户生成一个 localId用于标识用户从云端拉取对话记录。但是这种方式仅能保障用户在同一设备同一浏览器中使用如果切换设备或者清空浏览器缓存则会丢失这些记录。这种方式存在一定的风险因此我们仅允许用户拉取近`30天``20条`记录。
分享链接身份鉴权设计的目的在于,将 FastGPT 的对话框快速、安全的接入到你现有的系统中,仅需 2 个接口即可实现。
## 使用说明
分享链接鉴权设计的目的在于,将 FastGPT 的对话框安全的接入你现有的系统中。
免登录链接配置中,你可以选择填写`身份验证`栏。这是一个`POST`请求的根地址。在填写该地址后,分享链接的初始化、开始对话以及对话结束都会向该地址的特定接口发送一条请求。下面以`host`来表示`凭身份验证根地址`。服务器接口仅需返回是否校验成功即可,不需要返回其他数据,格式如下:
免登录链接配置中,增加了`凭证校验服务器`后,使用分享链接时会向服务器发起请求,校验链接是否可用,并在每次对话结束后,向服务器发送对话结果。下面以`host`来表示`凭证校验服务器`。服务器接口仅需返回是否校验成功即可,不需要返回其他数据,格式如下:
### 接口统一响应格式
```json
{
"success": true,
"message": "错误提示",
"msg": "同message, 错误提示"
"msg": "同message, 错误提示",
"uid": "用户唯一凭证"
}
```
`FastGPT` 将会判断`success`是否为`true`决定是允许用户继续操作。`message``msg`是等同的,你可以选择返回其中一个,当`success`不为`true`时,将会提示这个错误。
`uid`是用户的唯一凭证,将会用于拉取对话记录以及保存对话记录。可参考下方实践案例。
### 触发流程
![](/imgs/sharelinkProcess.png)
## 配置校验地址和校验token
### 1. 配置校验地址的`BaseURL`、
## 配置教程
### 1. 配置身份校验地址
![](/imgs/share-setlink.jpg)
配置校验地址后,在每次分享链接使用时,都会向对应的地址发起校验和上报请求。
{{% alert icon="🤖" %}}
这里仅需配置根地址,无需具体到完整请求路径。
{{% /alert %}}
### 2. 分享链接中增加额外 query
在分享链接的地址中,增加一个额外的参数: authToken。例如
原始的链接https://fastgpt.run/chat/share?shareId=648aaf5ae121349a16d62192
完整链接: https://fastgpt.run/chat/share?shareId=648aaf5ae121349a16d62192&authToken=userid12345
原始的链接:`https://fastgpt.run/chat/share?shareId=648aaf5ae121349a16d62192`
这个`token`通常是你系统生成的在发出校验请求时FastGPT 会在`body`中携带 token={{authToken}} 的参数。
完整链接: `https://fastgpt.run/chat/share?shareId=648aaf5ae121349a16d62192&authToken=userid12345`
## 聊天初始化校验
这个`authToken`通常是你系统生成的用户唯一凭证Token之类的。FastGPT 会在鉴权接口的`body`中携带 token={{authToken}} 的参数。
**FastGPT 发出的请求**
### 3. 编写聊天初始化校验接口
{{< tabs tabTotal="3" >}}
{{< tab tabName="请求示例" >}}
{{< markdownify >}}
```bash
curl --location --request POST '{{host}}/shareAuth/init' \
--header 'Content-Type: application/json' \
--data-raw '{
"token": "sintdolore"
"token": "{{authToken}}"
}'
```
**响应示例**
{{< /markdownify >}}
{{< /tab >}}
{{< tab tabName="鉴权成功" >}}
{{< markdownify >}}
```json
{
"success": true,
"uid": "username123",
}
```
系统会拉取该分享链接下uid 为 username123 的对话记录。
{{< /markdownify >}}
{{< /tab >}}
{{< tab tabName="鉴权失败" >}}
{{< markdownify >}}
```json
{
"success": false,
"message": "分享链接无效",
"message": "身份错误",
}
```
## 对话前校验
{{< /markdownify >}}
{{< /tab >}}
{{< /tabs >}}
**FastGPT 发出的请求**
### 4. 编写对话前校验接口
{{< tabs tabTotal="3" >}}
{{< tab tabName="请求示例" >}}
{{< markdownify >}}
```bash
curl --location --request POST '{{host}}/shareAuth/start' \
--header 'Content-Type: application/json' \
--data-raw '{
"token": "sintdolore",
"token": "{{authToken}}",
"question": "用户问题",
}'
```
**响应示例**
{{< /markdownify >}}
{{< /tab >}}
{{< tab tabName="鉴权成功" >}}
{{< markdownify >}}
```json
{
"success": true
"success": true,
"uid": "username123",
}
```
## 对话结果上报
{{< /markdownify >}}
{{< /tab >}}
{{< tab tabName="鉴权失败" >}}
{{< markdownify >}}
```json
{
"success": false,
"message": "身份验证失败",
}
```
```json
{
"success": false,
"message": "存在违规词",
}
```
{{< /markdownify >}}
{{< /tab >}}
{{< /tabs >}}
### 5. 编写对话结果上报接口(可选)
该接口无规定返回值。
响应值与[chat 接口格式相同](/docs/development/openapi/chat/#响应),仅多了一个`token`
可以重点关注`responseData`里的`price`值,`price`与实际价格的倍率为`100000`,即 100000=1元。
```bash
curl --location --request POST '{{host}}/shareAuth/finish' \
--header 'Content-Type: application/json' \
--data-raw '{
"token": "sint dolore",
"token": "{{authToken}}",
"responseData": [
{
"moduleName": "KB Search",
@@ -156,18 +237,18 @@ curl --location --request POST '{{host}}/shareAuth/finish' \
}'
```
响应值与 chat 接口相同,增加了一个 token。可以重点关注`responseData`里的值price 与实际价格的倍率为`100000`
**此接口无需响应值**
## 使用示
## 实践案
我们以[Laf作为服务器为例](https://laf.dev/),展示这 3 个接口的使用方式。
我们以[Laf作为服务器为例](https://laf.dev/)简单展示这 3 个接口的使用方式。
### 1. 创建3个Laf接口
![](/imgs/share-auth1.jpg)
{{< tabs tabTotal="3" >}}
{{< tab tabName="/shareAuth/init" >}}
{{< markdownify >}}
@@ -179,13 +260,15 @@ import cloud from '@lafjs/cloud'
export default async function (ctx: FunctionContext) {
const { token } = ctx.body
// 此处省略 token 解码过程
if (token === 'fastgpt') {
return { success: true }
return { success: true, data: { uid: "user1" } }
}
return { success: false,message: "身份错误" }
return { success: false,message:"身份错误" }
}
```
{{< /markdownify >}}
@@ -201,8 +284,8 @@ import cloud from '@lafjs/cloud'
export default async function (ctx: FunctionContext) {
const { token, question } = ctx.body
console.log(token, question, 'start')
// 此处省略 token 解码过程
if (token !== 'fastgpt') {
return { success: false, message: "身份错误" }
@@ -212,8 +295,9 @@ export default async function (ctx: FunctionContext) {
return { success: false, message: "内容不合规" }
}
return { success: true }
return { success: true, data: { uid: "user1" } }
}
```
{{< /markdownify >}}
@@ -229,7 +313,12 @@ import cloud from '@lafjs/cloud'
export default async function (ctx: FunctionContext) {
const { token, responseData } = ctx.body
console.log(token,responseData,'=====')
const total = responseData.reduce((sum,item) => sum + item.price,0)
const amount = total / 100000
// 省略数据库操作
return { }
}
```
@@ -241,17 +330,24 @@ export default async function (ctx: FunctionContext) {
### 2. 配置校验地址
我们随便复制3个地址中一个接口https://d8dns0.laf.dev/shareAuth/finish , 去除 /shareAuth/finish 后填入 FastGPT 中: https://d8dns0.laf.dev
我们随便复制3个地址中一个接口: `https://d8dns0.laf.dev/shareAuth/finish`, 去除`/shareAuth/finish`后填入`身份校验`:`https://d8dns0.laf.dev`
![](/imgs/share-auth2.jpg)
### 3. 修改分享链接参数
源分享链接:[https://fastgpt.run/chat/share?shareId=64be36376a438af0311e599c](https://fastgpt.run/chat/share?shareId=64be36376a438af0311e599c)
源分享链接:`https://fastgpt.run/chat/share?shareId=64be36376a438af0311e599c`
修改后:[https://fastgpt.run/chat/share?shareId=64be36376a438af0311e599c&authToken=fastgpt](https://fastgpt.run/chat/share?shareId=64be36376a438af0311e599c&authToken=fastgpt)
修改后:`https://fastgpt.run/chat/share?shareId=64be36376a438af0311e599c&authToken=fastgpt`
### 4. 测试效果
1. 打开源链接或者`authToken`不等于 `fastgpt`的链接会提示身份错误。
2. 发送内容中包含你字,会提示内容不合规。
1. 打开源链接或者`authToken`不等于`fastgpt`的链接会提示身份错误。
2. 发送内容中包含你字,会提示内容不合规。
## 使用场景
这个鉴权方式通常是帮助你直接嵌入`分享链接`到你的应用中,在你的应用打开分享链接前,应做`authToken`的拼接后再打开。
除了对接已有系统的用户外,你还可以对接`余额`功能,通过`结果上报`接口扣除用户余额,通过`对话前校验`接口检查用户的余额。

View File

@@ -4,5 +4,5 @@ go 1.21
require (
github.com/colinwilson/lotusdocs v0.1.0 // indirect
github.com/gohugoio/hugo-mod-bootstrap-scss/v5 v5.20300.20003 // indirect
github.com/gohugoio/hugo-mod-bootstrap-scss/v5 v5.20300.20200 // indirect
)

View File

@@ -10,5 +10,8 @@ github.com/colinwilson/lotusdocs v0.1.0 h1:oTC8pAYQp9XDNaUwE4SEY+id3ByNELxIIFrkt
github.com/colinwilson/lotusdocs v0.1.0/go.mod h1:9zu2REJDi+zdPRcR5/bRYSUR7gkNF4NQLvV38SEoCP8=
github.com/gohugoio/hugo-mod-bootstrap-scss/v5 v5.20300.20003 h1:pt/JGVD5YYRsVVijOHPZI6YKTUvbR4e0hgV9B0S6rbI=
github.com/gohugoio/hugo-mod-bootstrap-scss/v5 v5.20300.20003/go.mod h1:mvM05r93HiefwoaxQTaYiJxtJAhTebwQtU1Xh/J+Okk=
github.com/gohugoio/hugo-mod-bootstrap-scss/v5 v5.20300.20200 h1:SmpwwN3DNzJWbV+IT8gaFu07ENUFpCvKou5BHYUKuVs=
github.com/gohugoio/hugo-mod-bootstrap-scss/v5 v5.20300.20200/go.mod h1:kx8MBj9T7SFR8ZClWvKZPmmUxBaltkoXvnWlZZcSnYA=
github.com/gohugoio/hugo-mod-jslibs-dist/popperjs/v2 v2.21100.20000/go.mod h1:mFberT6ZtcchrsDtfvJM7aAH2bDKLdOnruUHl0hlapI=
github.com/twbs/bootstrap v5.3.0+incompatible/go.mod h1:fZTSrkpSf0/HkL0IIJzvVspTt1r9zuf7XlZau8kpcY0=
github.com/twbs/bootstrap v5.3.2+incompatible/go.mod h1:fZTSrkpSf0/HkL0IIJzvVspTt1r9zuf7XlZau8kpcY0=

View File

@@ -109,6 +109,11 @@ defaultContentLanguage = 'zh-cn'
listDescTrunc = 100 # Number of characters by which to truncate the list card description
# Link behaviour
intLinkTooltip = true # Enable a tooltip for internal links that displays info about the destination? default false
# extLinkNewTab = false # Open external links in a new Tab? default true
# logoLinkURL = "" # Set a custom URL destination for the top header logo link.
[params.flexsearch] # Parameters for FlexSearch
# enabled = true
# tokenize = "full"

View File

@@ -11,9 +11,9 @@
{{ end -}}
{{ if .Page.Store.Get "hasMermaid" }}
{{ $mermaid := resources.Get (printf "%s/%s" ($.Scratch.Get "pathName") "js/mermaid.js") }}
{{ $mermaid := resources.Get (printf "%s/%s" ($.Scratch.Get "pathName") "js/mermaid.min.js") }}
{{ if hugo.IsProduction }}
{{ $mermaid = $mermaid | minify | fingerprint "sha384" }}
{{ $mermaid = $mermaid | fingerprint "sha384" }}
{{ end }}
<script src="{{ $mermaid.RelPermalink }}" {{ if hugo.IsProduction }}integrity="{{ $mermaid.Data.Integrity }}"{{ end }}></script>
<script>

View File

@@ -2,7 +2,7 @@
<nav id="sidebar" class="sidebar-wrapper">
<div class="sidebar-brand">
<!-- change -->
<a href='{{ relLangURL "" }}' aria-label="HomePage" alt="HomePage" style="text-transform: unset;">
<a href='{{ with .Site.Params.docs.logoLinkURL }}{{ . }}{{ else }}{{ relLangURL "" }}{{ end }}' aria-label="HomePage" alt="HomePage" style="text-transform: unset;">
{{ with resources.Get "images/logos/logo.svg" }}
{{ .Content | safeHTML }}
&nbsp;

View File

@@ -2,7 +2,7 @@
<div id="top-header" class="top-header d-print-none">
<div class="header-bar d-flex justify-content-between">
<div class="d-flex align-items-center">
<a href='{{ relLangURL "" }}' class="logo-icon me-3" aria-label="HomePage" alt="HomePage">
<a href='{{ with .Site.Params.docs.logoLinkURL }}{{ . }}{{ else }}{{ relLangURL "" }}{{ end }}' class="logo-icon me-3" aria-label="HomePage" alt="HomePage">
<div class="small">
{{ with resources.Get "images/logos/mark.svg" }}
{{ .Content | safeHTML }}

View File

@@ -1,4 +1,5 @@
# 非 host 版本, 不使用本机代理
# (不懂 Docker 的,只需要关心 OPENAI_BASE_URL 和 CHAT_API_KEY 即可!)
version: '3.3'
services:
pg:
@@ -47,7 +48,7 @@ services:
environment:
# root 密码,用户名为: root
- DEFAULT_ROOT_PSW=1234
# 中转地址,如果是用官方号,不需要管
# 中转地址,如果是用官方号,不需要管。务必加 /v1
- OPENAI_BASE_URL=https://api.openai.com/v1
- CHAT_API_KEY=sk-xxxx
- DB_MAX_LINK=5 # database max link

View File

@@ -0,0 +1,24 @@
import { ErrType } from '../errorCode';
/* dataset: 507000 */
const startCode = 507000;
export enum CommonErrEnum {
fileNotFound = 'fileNotFound'
}
const datasetErr = [
{
statusText: CommonErrEnum.fileNotFound,
message: 'error.fileNotFound'
}
];
export default datasetErr.reduce((acc, cur, index) => {
return {
...acc,
[cur.statusText]: {
code: startCode + index,
statusText: cur.statusText,
message: cur.message,
data: null
}
};
}, {} as ErrType<`${CommonErrEnum}`>);

View File

@@ -13,23 +13,23 @@ export enum DatasetErrEnum {
const datasetErr = [
{
statusText: DatasetErrEnum.unAuthDataset,
message: '无权操作该知识库'
message: 'core.dataset.error.unAuthDataset'
},
{
statusText: DatasetErrEnum.unAuthDatasetCollection,
message: '无权操作该数据集'
message: 'core.dataset.error.unAuthDatasetCollection'
},
{
statusText: DatasetErrEnum.unAuthDatasetData,
message: '无权操作该数据'
message: 'core.dataset.error.unAuthDatasetData'
},
{
statusText: DatasetErrEnum.unAuthDatasetFile,
message: '无权操作该文件'
message: 'core.dataset.error.unAuthDatasetFile'
},
{
statusText: DatasetErrEnum.unCreateCollection,
message: '无权创建数据集'
message: 'core.dataset.error.unCreateCollection'
},
{
statusText: DatasetErrEnum.unLinkCollection,

View File

@@ -4,7 +4,9 @@ import { ErrType } from '../errorCode';
export enum OutLinkErrEnum {
unExist = 'unExist',
unAuthLink = 'unAuthLink',
linkUnInvalid = 'linkUnInvalid'
linkUnInvalid = 'linkUnInvalid',
unAuthUser = 'unAuthUser'
}
const errList = [
{
@@ -19,6 +21,10 @@ const errList = [
code: 501,
statusText: OutLinkErrEnum.linkUnInvalid,
message: '分享链接无效'
},
{
statusText: OutLinkErrEnum.unAuthUser,
message: '身份校验失败'
}
];
export default errList.reduce((acc, cur, index) => {

View File

@@ -6,6 +6,7 @@ import pluginErr from './code/plugin';
import outLinkErr from './code/outLink';
import teamErr from './code/team';
import userErr from './code/user';
import commonErr from './code/common';
export const ERROR_CODE: { [key: number]: string } = {
400: '请求失败',
@@ -96,5 +97,6 @@ export const ERROR_RESPONSE: Record<
...outLinkErr,
...teamErr,
...userErr,
...pluginErr
...pluginErr,
...commonErr
};

View File

@@ -1,3 +1,10 @@
export type UploadImgProps = {
base64Img: string;
expiredTime?: Date;
metadata?: Record<string, any>;
shareId?: string;
};
export type UrlFetchParams = {
urlList: string[];
selector?: string;

View File

@@ -49,7 +49,14 @@ export const cheerioToHtml = ({
}
});
return $(selector || 'body').html();
const html = $(selector || 'body')
.map((item, dom) => {
return $(dom).html();
})
.get()
.join('\n');
return html;
};
export const urlsFetch = async ({
urlList,

View File

@@ -26,10 +26,14 @@ export const simpleMarkdownText = (rawText: string) => {
rawText = rawText.replace(/\\\\n/g, '\\n');
// Remove headings and code blocks front spaces
['####', '###', '##', '#', '```', '~~~'].forEach((item) => {
['####', '###', '##', '#', '```', '~~~'].forEach((item, i) => {
const isMarkdown = i <= 3;
const reg = new RegExp(`\\n\\s*${item}`, 'g');
if (reg.test(rawText)) {
rawText = rawText.replace(new RegExp(`\\n\\s*(${item})`, 'g'), '\n$1');
rawText = rawText.replace(
new RegExp(`(\\n)\\s*(${item})`, 'g'),
isMarkdown ? '\n$1$2' : '$1$2'
);
}
});

View File

@@ -12,12 +12,13 @@ export const splitText2Chunks = (props: {
text: string;
chunkLen: number;
overlapRatio?: number;
customReg?: string[];
}): {
chunks: string[];
tokens: number;
overlapRatio?: number;
} => {
let { text = '', chunkLen, overlapRatio = 0.2 } = props;
let { text = '', chunkLen, overlapRatio = 0.2, customReg = [] } = props;
const splitMarker = 'SPLIT_HERE_SPLIT_HERE';
const codeBlockMarker = 'CODE_BLOCK_LINE_MARKER';
const overlapLen = Math.round(chunkLen * overlapRatio);
@@ -29,22 +30,29 @@ export const splitText2Chunks = (props: {
// The larger maxLen is, the next sentence is less likely to trigger splitting
const stepReges: { reg: RegExp; maxLen: number }[] = [
{ reg: /^(#\s[^\n]+)\n/gm, maxLen: chunkLen * 1.4 },
{ reg: /^(##\s[^\n]+)\n/gm, maxLen: chunkLen * 1.4 },
{ reg: /^(###\s[^\n]+)\n/gm, maxLen: chunkLen * 1.4 },
{ reg: /^(####\s[^\n]+)\n/gm, maxLen: chunkLen * 1.4 },
...customReg.map((text) => ({ reg: new RegExp(`([${text}])`, 'g'), maxLen: chunkLen * 1.4 })),
{ reg: /^(#\s[^\n]+)\n/gm, maxLen: chunkLen * 1.2 },
{ reg: /^(##\s[^\n]+)\n/gm, maxLen: chunkLen * 1.2 },
{ reg: /^(###\s[^\n]+)\n/gm, maxLen: chunkLen * 1.2 },
{ reg: /^(####\s[^\n]+)\n/gm, maxLen: chunkLen * 1.2 },
{ reg: /([\n](`))/g, maxLen: chunkLen * 4 }, // code block
{ reg: /([\n](?![\*\-|>0-9]))/g, maxLen: chunkLen * 1.8 }, // (?![\*\-|>`0-9]): markdown special char
{ reg: /([\n])/g, maxLen: chunkLen * 1.4 },
{ reg: /([\n]([`~]))/g, maxLen: chunkLen * 4 }, // code block
{ reg: /([\n](?!\s*[\*\-|>0-9]))/g, maxLen: chunkLen * 2 }, // (?![\*\-|>`0-9]): markdown special char
{ reg: /([\n])/g, maxLen: chunkLen * 1.2 },
{ reg: /([。]|([a-zA-Z])\.\s)/g, maxLen: chunkLen * 1.4 },
{ reg: /([]|!\s)/g, maxLen: chunkLen * 1.4 },
{ reg: /([]|\?\s)/g, maxLen: chunkLen * 1.6 },
{ reg: /([]|;\s)/g, maxLen: chunkLen * 1.8 },
{ reg: /([。]|([a-zA-Z])\.\s)/g, maxLen: chunkLen * 1.2 },
{ reg: /([]|!\s)/g, maxLen: chunkLen * 1.2 },
{ reg: /([]|\?\s)/g, maxLen: chunkLen * 1.4 },
{ reg: /([]|;\s)/g, maxLen: chunkLen * 1.6 },
{ reg: /([]|,\s)/g, maxLen: chunkLen * 2 }
];
const customRegLen = customReg.length;
const checkIsCustomStep = (step: number) => step < customRegLen;
const checkIsMarkdownSplit = (step: number) => step >= customRegLen && step <= 3 + customRegLen;
const checkIndependentChunk = (step: number) => step >= customRegLen && step <= 4 + customRegLen;
const checkForbidOverlap = (step: number) => step <= 6 + customRegLen;
// if use markdown title split, Separate record title title
const getSplitTexts = ({ text, step }: { text: string; step: number }) => {
if (step >= stepReges.length) {
@@ -55,11 +63,13 @@ export const splitText2Chunks = (props: {
}
];
}
const isMarkdownSplit = step <= 3;
const isMarkdownSplit = checkIsMarkdownSplit(step);
const independentChunk = checkIndependentChunk(step);
const { reg } = stepReges[step];
const splitTexts = text
.replace(reg, isMarkdownSplit ? `${splitMarker}$1` : `$1${splitMarker}`)
.replace(reg, independentChunk ? `${splitMarker}$1` : `$1${splitMarker}`)
.split(`${splitMarker}`)
.filter((part) => part.trim());
@@ -76,7 +86,7 @@ export const splitText2Chunks = (props: {
};
const getOneTextOverlapText = ({ text, step }: { text: string; step: number }): string => {
const forbidOverlap = step <= 6;
const forbidOverlap = checkForbidOverlap(step);
const maxOverlapLen = chunkLen * 0.4;
// step >= stepReges.length: Do not overlap incomplete sentences
@@ -114,7 +124,8 @@ export const splitText2Chunks = (props: {
lastText: string;
mdTitle: string;
}): string[] => {
const isMarkdownSplit = step <= 3;
const independentChunk = checkIndependentChunk(step);
const isCustomStep = checkIsCustomStep(step);
// mini text
if (text.length <= chunkLen) {
@@ -134,12 +145,14 @@ export const splitText2Chunks = (props: {
return chunks;
}
const { maxLen } = stepReges[step];
const minChunkLen = chunkLen * 0.7;
// split text by special char
const splitTexts = getSplitTexts({ text, step });
const maxLen = splitTexts.length > 1 ? stepReges[step].maxLen : chunkLen;
const minChunkLen = chunkLen * 0.7;
const miniChunkLen = 30;
// console.log(splitTexts, stepReges[step].reg);
const chunks: string[] = [];
for (let i = 0; i < splitTexts.length; i++) {
const item = splitTexts[i];
@@ -170,8 +183,8 @@ export const splitText2Chunks = (props: {
mdTitle: currentTitle
});
const lastChunk = innerChunks[innerChunks.length - 1];
// last chunk is too small, concat it to lastText
if (!isMarkdownSplit && lastChunk.length < minChunkLen) {
// last chunk is too small, concat it to lastText(next chunk start)
if (!independentChunk && lastChunk.length < minChunkLen) {
chunks.push(...innerChunks.slice(0, -1));
lastText = lastChunk;
} else {
@@ -189,10 +202,14 @@ export const splitText2Chunks = (props: {
lastText = newText;
// markdown paragraph block: Direct addition; If the chunk size reaches, add a chunk
if (isMarkdownSplit || newTextLen >= chunkLen) {
if (
isCustomStep ||
(independentChunk && newTextLen > miniChunkLen) ||
newTextLen >= chunkLen
) {
chunks.push(`${currentTitle}${lastText}`);
lastText = isMarkdownSplit ? '' : getOneTextOverlapText({ text: lastText, step });
lastText = getOneTextOverlapText({ text: lastText, step });
}
}

View File

@@ -0,0 +1,13 @@
export enum SystemConfigsTypeEnum {
fastgpt = 'fastgpt',
fastgptPro = 'fastgptPro'
}
export const SystemConfigsTypeMap = {
[SystemConfigsTypeEnum.fastgpt]: {
label: 'fastgpt'
},
[SystemConfigsTypeEnum.fastgptPro]: {
label: 'fastgptPro'
}
};

View File

@@ -0,0 +1,8 @@
import { SystemConfigsTypeEnum } from "./constants";
export type SystemConfigsType = {
_id: string;
type: `${SystemConfigsTypeEnum}`;
value: Record<string, any>;
createTime: Date;
};

View File

@@ -24,7 +24,7 @@ export const getDefaultAppForm = (templateId = 'fastgpt-universal'): AppSimpleEd
dataset: {
datasets: [],
similarity: 0.4,
limit: 5,
limit: 1500,
searchEmptyText: '',
searchMode: DatasetSearchModeEnum.embedding
},

View File

@@ -1,33 +0,0 @@
import { ModuleItemType } from '../module/type';
import { AdminFbkType, ChatItemType, moduleDispatchResType } from './type';
export type UpdateHistoryProps = {
chatId: string;
customTitle?: string;
top?: boolean;
};
export type AdminUpdateFeedbackParams = AdminFbkType & {
chatItemId: string;
};
export type InitChatResponse = {
chatId: string;
appId: string;
app: {
userGuideModule?: ModuleItemType;
chatModels?: string[];
name: string;
avatar: string;
intro: string;
canUse?: boolean;
};
title: string;
variables: Record<string, any>;
history: ChatItemType[];
};
export type ChatHistoryItemResType = moduleDispatchResType & {
moduleType: `${FlowNodeTypeEnum}`;
moduleName: string;
};

View File

@@ -44,8 +44,16 @@ export const ChatSourceMap = {
}
};
export enum ChatStatusEnum {
loading = 'loading',
running = 'running',
finish = 'finish'
}
export const HUMAN_ICON = `/icon/human.svg`;
export const LOGO_ICON = `/icon/logo.svg`;
export const IMG_BLOCK_KEY = 'img-block';
export const FILE_BLOCK_KEY = 'file-block';
export const MARKDOWN_QUOTE_SIGN = 'QUOTE SIGN';

View File

@@ -1,6 +1,6 @@
import { ClassifyQuestionAgentItemType } from '../module/type';
import { SearchDataResponseItemType } from '../dataset/type';
import { ChatRoleEnum, ChatSourceEnum } from './constants';
import { ChatRoleEnum, ChatSourceEnum, ChatStatusEnum } from './constants';
import { FlowNodeTypeEnum } from '../module/node/constant';
import { ModuleOutputKeyEnum } from '../module/constants';
import { AppSchema } from '../app/type';
@@ -20,7 +20,7 @@ export type ChatSchema = {
variables: Record<string, any>;
source: `${ChatSourceEnum}`;
shareId?: string;
isInit: boolean;
outLinkUid?: string;
content: ChatItemType[];
};
@@ -51,6 +51,7 @@ export type AdminFbkType = {
a?: string;
};
/* --------- chat item ---------- */
export type ChatItemType = {
dataId?: string;
obj: ChatItemSchema['obj'];
@@ -61,11 +62,12 @@ export type ChatItemType = {
};
export type ChatSiteItemType = ChatItemType & {
status: 'loading' | 'running' | 'finish';
status: `${ChatStatusEnum}`;
moduleName?: string;
ttsBuffer?: Uint8Array;
};
/* ---------- history ------------- */
export type HistoryItemType = {
chatId: string;
updateTime: Date;
@@ -77,10 +79,10 @@ export type ChatHistoryItemType = HistoryItemType & {
top: boolean;
};
// response data
/* ------- response data ------------ */
export type moduleDispatchResType = {
moduleLogo?: string;
price: number;
price?: number;
runningTime?: number;
tokens?: number;
model?: string;
@@ -112,3 +114,8 @@ export type moduleDispatchResType = {
// plugin output
pluginOutput?: Record<string, any>;
};
export type ChatHistoryItemResType = moduleDispatchResType & {
moduleType: `${FlowNodeTypeEnum}`;
moduleName: string;
};

View File

@@ -6,9 +6,9 @@ import type { LLMModelItemType } from '../ai/model.d';
export type DatasetUpdateBody = {
id: string;
parentId?: string;
tags?: string[];
name?: string;
avatar?: string;
intro?: string;
permission?: DatasetSchemaType['permission'];
agentModel?: LLMModelItemType;
websiteConfig?: DatasetSchemaType['websiteConfig'];

View File

@@ -54,17 +54,10 @@ export const DatasetSearchModule: FlowModuleTemplateType = {
{
key: ModuleInputKeyEnum.datasetLimit,
type: FlowNodeInputTypeEnum.hidden,
label: '单次搜索上限',
description: '最多取 n 条记录作为本次问题引用',
value: 5,
label: '引用上限',
description: '单次搜索最大的 Tokens 数量中文约1字=1.7Tokens英文约1字=1Tokens',
value: 1500,
valueType: ModuleDataTypeEnum.number,
min: 1,
max: 20,
step: 1,
markList: [
{ label: '1', value: 1 },
{ label: '20', value: 20 }
],
showTargetInApp: false,
showTargetInPlugin: false
},

View File

@@ -1,27 +1,12 @@
import type { HistoryItemType, ChatSiteItemType } from '../../core/chat/type.d';
import type { InitChatResponse } from '../../core/chat/api.d';
import { OutLinkSchema } from '@fastgpt/global/support/outLink/type';
export type InitShareChatResponse = {
userAvatar: string;
app: InitChatResponse['app'];
};
/* one page type */
export type ShareChatType = InitShareChatResponse & {
history: ShareChatHistoryItemType;
};
/* history list item type */
export type ShareChatHistoryItemType = HistoryItemType & {
shareId: string;
variables?: Record<string, any>;
chats: ChatSiteItemType[];
};
export type AuthLinkChatProps = { ip?: string | null; authToken?: string; question: string };
export type AuthLinkLimitProps = AuthLinkChatProps & { outLink: OutLinkSchema };
export type AuthShareChatInitProps = {
authToken?: string;
export type AuthOutLinkInitProps = {
outLinkUid: string;
tokenUrl?: string;
};
export type AuthOutLinkChatProps = { ip?: string | null; outLinkUid: string; question: string };
export type AuthOutLinkLimitProps = AuthOutLinkChatProps & { outLink: OutLinkSchema };
export type AuthOutLinkResponse = {
uid: string;
};

View File

@@ -3,7 +3,6 @@ import { OutLinkTypeEnum } from './constant';
export type OutLinkSchema = {
_id: string;
shareId: string;
userId: string;
teamId: string;
tmbId: string;
appId: string;

View File

@@ -45,4 +45,4 @@ export const TeamMemberStatusMap = {
color: 'red.600'
}
};
export const leaveStatus = { $ne: TeamMemberStatusEnum.leave };
export const notLeaveStatus = { $ne: TeamMemberStatusEnum.leave };

View File

@@ -1,5 +1,5 @@
import { UserModelSchema } from '../type';
import { TeamMemberRoleEnum, TeamMemberStatusEnum } from './constant';
import type { UserModelSchema } from '../type';
import type { TeamMemberRoleEnum, TeamMemberStatusEnum } from './constant';
export type TeamSchema = {
_id: string;
@@ -22,6 +22,16 @@ export type TeamMemberSchema = {
defaultTeam: boolean;
};
export type TeamMemberWithUserSchema = TeamMemberSchema & {
userId: UserModelSchema;
};
export type TeamMemberWithTeamSchema = TeamMemberSchema & {
teamId: TeamSchema;
};
export type TeamMemberWithTeamAndUserSchema = TeamMemberWithTeamSchema & {
userId: UserModelSchema;
};
export type TeamItemType = {
userId: string;
teamId: string;

View File

@@ -3,6 +3,7 @@ import { BucketNameEnum } from '@fastgpt/global/common/file/constants';
import fsp from 'fs/promises';
import fs from 'fs';
import { DatasetFileSchema } from '@fastgpt/global/core/dataset/type';
import { delImgByFileIdList } from '../image/controller';
export function getGFSCollection(bucket: `${BucketNameEnum}`) {
return connectionMongo.connection.db.collection(`${bucket}.files`);
@@ -69,24 +70,65 @@ export async function getFileById({
_id: new Types.ObjectId(fileId)
});
if (!file) {
return Promise.reject('File not found');
}
// if (!file) {
// return Promise.reject('File not found');
// }
return file;
return file || undefined;
}
export async function delFileById({
export async function delFileByFileIdList({
bucketName,
fileId
fileIdList,
retry = 3
}: {
bucketName: `${BucketNameEnum}`;
fileId: string;
fileIdList: string[];
retry?: number;
}): Promise<any> {
try {
const bucket = getGridBucket(bucketName);
await Promise.all(fileIdList.map((id) => bucket.delete(new Types.ObjectId(id))));
} catch (error) {
if (retry > 0) {
return delFileByFileIdList({ bucketName, fileIdList, retry: retry - 1 });
}
}
}
// delete file by metadata(datasetId)
export async function delFileByMetadata({
bucketName,
datasetId
}: {
bucketName: `${BucketNameEnum}`;
datasetId?: string;
}) {
const bucket = getGridBucket(bucketName);
await bucket.delete(new Types.ObjectId(fileId));
return true;
const files = await bucket
.find(
{
...(datasetId && { 'metadata.datasetId': datasetId })
},
{
projection: {
_id: 1
}
}
)
.toArray();
const idList = files.map((item) => String(item._id));
// delete img
await delImgByFileIdList(idList);
// delete file
await delFileByFileIdList({
bucketName,
fileIdList: idList
});
}
export async function getDownloadStream({

View File

@@ -1,3 +1,4 @@
import { UploadImgProps } from '@fastgpt/global/common/file/api';
import { imageBaseUrl } from './constant';
import { MongoImage } from './schema';
@@ -9,11 +10,10 @@ export const maxImgSize = 1024 * 1024 * 12;
export async function uploadMongoImg({
base64Img,
teamId,
expiredTime
}: {
base64Img: string;
expiredTime,
metadata
}: UploadImgProps & {
teamId: string;
expiredTime?: Date;
}) {
if (base64Img.length > maxImgSize) {
return Promise.reject('Image too large');
@@ -24,7 +24,8 @@ export async function uploadMongoImg({
const { _id } = await MongoImage.create({
teamId,
binary: Buffer.from(base64Data, 'base64'),
expiredTime
expiredTime: expiredTime,
metadata
});
return getMongoImgUrl(String(_id));
@@ -37,3 +38,9 @@ export async function readMongoImg({ id }: { id: string }) {
}
return data?.binary;
}
export async function delImgByFileIdList(fileIds: string[]) {
return MongoImage.deleteMany({
'metadata.fileId': { $in: fileIds.map((item) => String(item)) }
});
}

View File

@@ -5,13 +5,17 @@ const { Schema, model, models } = connectionMongo;
const ImageSchema = new Schema({
teamId: {
type: Schema.Types.ObjectId,
ref: TeamCollectionName
ref: TeamCollectionName,
required: true
},
binary: {
type: Buffer
},
expiredTime: {
type: Date
},
metadata: {
type: Object
}
});
@@ -21,7 +25,7 @@ try {
console.log(error);
}
export const MongoImage: Model<{ teamId: string; binary: Buffer }> =
export const MongoImage: Model<{ teamId: string; binary: Buffer; metadata?: Record<string, any> }> =
models['image'] || model('image', ImageSchema);
MongoImage.syncIndexes();

View File

@@ -26,9 +26,13 @@ export async function connectMongo({
bufferCommands: true,
maxConnecting: Number(process.env.DB_MAX_LINK || 5),
maxPoolSize: Number(process.env.DB_MAX_LINK || 5),
minPoolSize: Number(process.env.DB_MAX_LINK || 10) * 0.5,
minPoolSize: Math.min(10, Number(process.env.DB_MAX_LINK || 10)),
connectTimeoutMS: 60000,
waitQueueTimeoutMS: 60000
waitQueueTimeoutMS: 60000,
socketTimeoutMS: 60000,
maxIdleTimeMS: 300000,
retryWrites: true,
retryReads: true
});
console.log('mongo connected');

View File

@@ -82,7 +82,7 @@ export const sseErrRes = (res: NextApiResponse, error: any) => {
} else if (error?.response?.data?.error?.message) {
msg = error?.response?.data?.error?.message;
} else if (error?.error?.message) {
msg = error?.error?.message;
msg = `${error?.error?.code} ${error?.error?.message}`;
}
addLog.error(`sse error: ${msg}`, error);

View File

@@ -0,0 +1,32 @@
import { SystemConfigsType } from '@fastgpt/global/common/system/config/type';
import { connectionMongo, type Model } from '../../../common/mongo';
import { SystemConfigsTypeMap } from '@fastgpt/global/common/system/config/constants';
const { Schema, model, models } = connectionMongo;
const collectionName = 'systemConfigs';
const systemConfigSchema = new Schema({
type: {
type: String,
required: true,
enum: Object.keys(SystemConfigsTypeMap)
},
value: {
type: Object,
required: true
},
createTime: {
type: Date,
default: () => new Date()
}
})
try {
systemConfigSchema.index({ createTime: -1 }, { expireAfterSeconds: 90 * 24 * 60 * 60 });
} catch (error) {
console.log(error);
}
export const MongoSystemConfigs: Model<SystemConfigsType>=
models[collectionName] || model(collectionName, systemConfigSchema);
MongoSystemConfigs.syncIndexes();

View File

@@ -50,10 +50,6 @@ const ChatSchema = new Schema({
top: {
type: Boolean
},
variables: {
type: Object,
default: {}
},
source: {
type: String,
enum: Object.keys(ChatSourceMap),
@@ -62,9 +58,17 @@ const ChatSchema = new Schema({
shareId: {
type: String
},
isInit: {
type: Boolean,
default: false
outLinkUid: {
type: String
},
variables: {
type: Object,
default: {}
},
metadata: {
//For special storage
type: Object,
default: {}
},
content: {
type: [
@@ -89,9 +93,10 @@ const ChatSchema = new Schema({
});
try {
ChatSchema.index({ tmbId: 1 });
ChatSchema.index({ updateTime: -1 });
ChatSchema.index({ appId: 1 });
ChatSchema.index({ tmbId: 1 });
ChatSchema.index({ shareId: 1 });
ChatSchema.index({ updateTime: -1 });
} catch (error) {
console.log(error);
}

View File

@@ -0,0 +1,22 @@
import type { ChatItemType } from '@fastgpt/global/core/chat/type';
import { MongoChatItem } from './chatItemSchema';
export async function getChatItems({
chatId,
limit = 30,
field
}: {
chatId?: string;
limit?: number;
field: string;
}): Promise<{ history: ChatItemType[] }> {
if (!chatId) {
return { history: [] };
}
const history = await MongoChatItem.find({ chatId }, field).sort({ _id: -1 }).limit(limit);
history.reverse();
return { history };
}

View File

@@ -1,11 +1,11 @@
import { MongoDatasetData } from './schema';
import { deletePgDataById } from './pg';
import { MongoDatasetTraining } from '../training/schema';
import { delFileById } from '../../../common/file/gridfs/controller';
import { delFileByFileIdList, delFileByMetadata } from '../../../common/file/gridfs/controller';
import { BucketNameEnum } from '@fastgpt/global/common/file/constants';
import { MongoDatasetCollection } from '../collection/schema';
import { delDatasetFiles } from '../file/controller';
import { delay } from '@fastgpt/global/common/system/utils';
import { delImgByFileIdList } from '../../../common/file/image/controller';
/* delete all data by datasetIds */
export async function delDatasetRelevantData({ datasetIds }: { datasetIds: string[] }) {
@@ -17,9 +17,11 @@ export async function delDatasetRelevantData({ datasetIds }: { datasetIds: strin
});
// delete related files
await Promise.all(datasetIds.map((id) => delDatasetFiles({ datasetId: id })));
await Promise.all(
datasetIds.map((id) => delFileByMetadata({ bucketName: BucketNameEnum.dataset, datasetId: id }))
);
await delay(1000);
await delay(500);
// delete pg data
await deletePgDataById(`dataset_id IN ('${datasetIds.join("','")}')`);
@@ -49,17 +51,16 @@ export async function delCollectionRelevantData({
collectionId: { $in: collectionIds }
});
// delete file
await Promise.all(
filterFileIds.map((fileId) => {
return delFileById({
bucketName: BucketNameEnum.dataset,
fileId
});
// delete file and imgs
await Promise.all([
delImgByFileIdList(filterFileIds),
delFileByFileIdList({
bucketName: BucketNameEnum.dataset,
fileIdList: filterFileIds
})
);
]);
await delay(1000);
await delay(500);
// delete pg data
await deletePgDataById(`collection_id IN ('${collectionIds.join("','")}')`);

View File

@@ -1,9 +0,0 @@
import { BucketNameEnum } from '@fastgpt/global/common/file/constants';
import { getGFSCollection } from '../../../common/file/gridfs/controller';
export async function delDatasetFiles({ datasetId }: { datasetId: string }) {
const db = getGFSCollection(BucketNameEnum.dataset);
await db.deleteMany({
'metadata.datasetId': String(datasetId)
});
}

View File

@@ -12,10 +12,6 @@ const OutLinkSchema = new Schema({
type: String,
required: true
},
userId: {
type: Schema.Types.ObjectId,
ref: 'user'
},
teamId: {
type: Schema.Types.ObjectId,
ref: TeamCollectionName,

View File

@@ -22,15 +22,15 @@ export const updateOutLinkUsage = async ({
};
export const pushResult2Remote = async ({
authToken,
outLinkUid,
shareId,
responseData
}: {
authToken?: string;
outLinkUid?: string; // raw id, not parse
shareId?: string;
responseData?: any[];
}) => {
if (!shareId || !authToken || !global.systemEnv.pluginBaseUrl) return;
if (!shareId || !outLinkUid || !global.systemEnv.pluginBaseUrl) return;
try {
const outLink = await MongoOutLink.findOne({
shareId
@@ -42,7 +42,7 @@ export const pushResult2Remote = async ({
baseURL: outLink.limit.hookUrl,
url: '/shareAuth/finish',
data: {
token: authToken,
token: outLinkUid,
responseData
}
});

View File

@@ -19,6 +19,7 @@ export async function authApp({
AuthResponseType & {
teamOwner: boolean;
app: AppDetailType;
role: `${TeamMemberRoleEnum}`;
}
> {
const result = await parseHeaderCert(props);
@@ -65,6 +66,7 @@ export async function authApp({
return {
...result,
app,
role,
isOwner,
canWrite,
teamOwner: role === TeamMemberRoleEnum.owner

View File

@@ -1,67 +0,0 @@
import { AuthResponseType } from '@fastgpt/global/support/permission/type';
import { AuthModeType } from '../type';
import type { ChatWithAppSchema } from '@fastgpt/global/core/chat/type';
import { parseHeaderCert } from '../controller';
import { MongoChat } from '../../../core/chat/chatSchema';
import { ChatErrEnum } from '@fastgpt/global/common/error/code/chat';
import { getTeamInfoByTmbId } from '../../user/team/controller';
import { TeamMemberRoleEnum } from '@fastgpt/global/support/user/team/constant';
import { PermissionTypeEnum } from '@fastgpt/global/support/permission/constant';
export async function authChat({
chatId,
per = 'owner',
...props
}: AuthModeType & {
chatId: string;
}): Promise<
AuthResponseType & {
chat: ChatWithAppSchema;
}
> {
const { userId, teamId, tmbId } = await parseHeaderCert(props);
const { role } = await getTeamInfoByTmbId({ tmbId });
const { chat, isOwner, canWrite } = await (async () => {
// get chat
const chat = (await MongoChat.findOne({ chatId, teamId })
.populate('appId')
.lean()) as ChatWithAppSchema;
if (!chat) {
return Promise.reject('Chat is not exists');
}
const isOwner = role === TeamMemberRoleEnum.owner || String(chat.tmbId) === tmbId;
const canWrite = isOwner;
if (
per === 'r' &&
role !== TeamMemberRoleEnum.owner &&
chat.appId.permission !== PermissionTypeEnum.public
) {
return Promise.reject(ChatErrEnum.unAuthChat);
}
if (per === 'w' && !canWrite) {
return Promise.reject(ChatErrEnum.unAuthChat);
}
if (per === 'owner' && !isOwner) {
return Promise.reject(ChatErrEnum.unAuthChat);
}
return {
chat,
isOwner,
canWrite
};
})();
return {
userId,
teamId,
tmbId,
chat,
isOwner,
canWrite
};
}

View File

@@ -12,7 +12,7 @@ export const authCert = async (props: AuthModeType) => {
canWrite: true
};
};
export async function authCertAndShareId({
export async function authCertOrShareId({
shareId,
...props
}: AuthModeType & { shareId?: string }) {
@@ -20,11 +20,11 @@ export async function authCertAndShareId({
return authCert(props);
}
const { app } = await authOutLinkValid({ shareId });
const { shareChat } = await authOutLinkValid({ shareId });
return {
teamId: String(app.teamId),
tmbId: String(app.tmbId),
teamId: String(shareChat.teamId),
tmbId: String(shareChat.tmbId),
authType: AuthUserTypeEnum.outLink,
apikey: '',
isOwner: false,

View File

@@ -14,6 +14,7 @@ import {
import { getFileById } from '../../../common/file/gridfs/controller';
import { BucketNameEnum } from '@fastgpt/global/common/file/constants';
import { getTeamInfoByTmbId } from '../../user/team/controller';
import { CommonErrEnum } from '@fastgpt/global/common/error/code/common';
export async function authDatasetByTmbId({
teamId,
@@ -167,6 +168,10 @@ export async function authDatasetFile({
const file = await getFileById({ bucketName: BucketNameEnum.dataset, fileId });
if (!file) {
return Promise.reject(CommonErrEnum.fileNotFound);
}
if (file.metadata.teamId !== teamId) {
return Promise.reject(DatasetErrEnum.unAuthDatasetFile);
}

View File

@@ -78,21 +78,19 @@ export async function authOutLinkCrud({
};
}
/* outLink exist and it app exist */
export async function authOutLinkValid({ shareId }: { shareId?: string }) {
if (!shareId) {
return Promise.reject(OutLinkErrEnum.linkUnInvalid);
}
const shareChat = await MongoOutLink.findOne({ shareId });
if (!shareChat) {
return Promise.reject(OutLinkErrEnum.linkUnInvalid);
}
const app = await MongoApp.findById(shareChat.appId);
if (!app) {
return Promise.reject(AppErrEnum.unExist);
}
return {
app,
appId: shareChat.appId,
shareChat
};
}

View File

@@ -115,6 +115,18 @@ export async function parseHeaderCert({
{}) as ReqHeaderAuthType;
const { uid, teamId, tmbId, appId, openApiKey, authType } = await (async () => {
if (authApiKey && authorization) {
// apikey from authorization
const authResponse = await parseAuthorization(authorization);
return {
uid: authResponse.uid,
teamId: authResponse.teamId,
tmbId: authResponse.tmbId,
appId: authResponse.appId,
openApiKey: authResponse.apikey,
authType: AuthUserTypeEnum.apikey
};
}
if (authToken && (cookie || token)) {
// user token(from fastgpt web)
const res = await authCookieToken(cookie, token);
@@ -139,6 +151,7 @@ export async function parseHeaderCert({
authType: AuthUserTypeEnum.root
};
}
// apikey: abandon
if (authApiKey && apikey) {
// apikey
const parseResult = await authOpenApiKey({ apikey });
@@ -152,32 +165,8 @@ export async function parseHeaderCert({
};
}
if (authApiKey && authorization) {
// apikey from authorization
const authResponse = await parseAuthorization(authorization);
return {
uid: authResponse.uid,
teamId: authResponse.teamId,
tmbId: authResponse.tmbId,
appId: authResponse.appId,
openApiKey: authResponse.apikey,
authType: AuthUserTypeEnum.apikey
};
}
return {
uid: '',
teamId: '',
tmbId: '',
appId: '',
openApiKey: '',
authType: AuthUserTypeEnum.token
};
})();
// not rootUser and no uid, reject request
if (!rootkey && !uid && !teamId && !tmbId) {
return Promise.reject(ERROR_ENUM.unAuthorization);
}
})();
return {
userId: String(uid),

View File

@@ -1,4 +1,8 @@
import { UserType } from '@fastgpt/global/support/user/type';
import { MongoUser } from './schema';
import { getTeamInfoByTmbId, getUserDefaultTeam } from './team/controller';
import { ERROR_ENUM } from '@fastgpt/global/common/error/errorCode';
import { UserErrEnum } from '@fastgpt/global/common/error/code/user';
export async function authUserExist({ userId, username }: { userId?: string; username?: string }) {
if (userId) {
@@ -9,3 +13,56 @@ export async function authUserExist({ userId, username }: { userId?: string; use
}
return null;
}
export async function getUserDetail({
tmbId,
userId
}: {
tmbId?: string;
userId?: string;
}): Promise<UserType> {
const team = await (async () => {
if (tmbId) {
return getTeamInfoByTmbId({ tmbId });
}
if (userId) {
return getUserDefaultTeam({ userId });
}
return Promise.reject(ERROR_ENUM.unAuthorization);
})();
const user = await MongoUser.findById(team.userId);
if (!user) {
return Promise.reject(ERROR_ENUM.unAuthorization);
}
return {
_id: user._id,
username: user.username,
avatar: user.avatar,
balance: user.balance,
timezone: user.timezone,
promotionRate: user.promotionRate,
openaiAccount: user.openaiAccount,
team
};
}
export async function getUserAndAuthBalance({
tmbId,
minBalance
}: {
tmbId: string;
minBalance?: number;
}) {
const user = await getUserDetail({ tmbId });
if (!user) {
return Promise.reject(UserErrEnum.unAuthUser);
}
if (minBalance !== undefined && user.team.balance < minBalance) {
return Promise.reject(UserErrEnum.balanceNotEnough);
}
return user;
}

View File

@@ -1,35 +1,15 @@
import { TeamItemType } from '@fastgpt/global/support/user/team/type';
import { connectionMongo, Types } from '../../../common/mongo';
import { TeamItemType, TeamMemberWithTeamSchema } from '@fastgpt/global/support/user/team/type';
import { Types } from '../../../common/mongo';
import {
TeamMemberRoleEnum,
TeamMemberStatusEnum,
TeamCollectionName,
TeamMemberCollectionName,
leaveStatus
notLeaveStatus
} from '@fastgpt/global/support/user/team/constant';
import { MongoTeamMember } from './teamMemberSchema';
import { MongoTeam } from './teamSchema';
async function getTeam(match: Record<string, any>): Promise<TeamItemType> {
const db = connectionMongo?.connection?.db;
const TeamMember = db.collection(TeamMemberCollectionName);
const results = await TeamMember.aggregate([
{
$match: match
},
{
$lookup: {
from: TeamCollectionName,
localField: 'teamId',
foreignField: '_id',
as: 'team'
}
},
{
$unwind: '$team'
}
]).toArray();
const tmb = results[0];
const tmb = (await MongoTeamMember.findOne(match).populate('teamId')) as TeamMemberWithTeamSchema;
if (!tmb) {
return Promise.reject('member not exist');
@@ -37,17 +17,17 @@ async function getTeam(match: Record<string, any>): Promise<TeamItemType> {
return {
userId: String(tmb.userId),
teamId: String(tmb.teamId),
teamName: tmb.team.name,
teamId: String(tmb.teamId._id),
teamName: tmb.teamId.name,
memberName: tmb.name,
avatar: tmb.team.avatar,
balance: tmb.team.balance,
avatar: tmb.teamId.avatar,
balance: tmb.teamId.balance,
tmbId: String(tmb._id),
role: tmb.role,
status: tmb.status,
defaultTeam: tmb.defaultTeam,
canWrite: tmb.role !== TeamMemberRoleEnum.visitor,
maxSize: tmb.team.maxSize
maxSize: tmb.teamId.maxSize
};
}
@@ -57,7 +37,7 @@ export async function getTeamInfoByTmbId({ tmbId }: { tmbId: string }) {
}
return getTeam({
_id: new Types.ObjectId(tmbId),
status: leaveStatus
status: notLeaveStatus
});
}
@@ -83,12 +63,8 @@ export async function createDefaultTeam({
balance?: number;
maxSize?: number;
}) {
const db = connectionMongo.connection.db;
const Team = db.collection(TeamCollectionName);
const TeamMember = db.collection(TeamMemberCollectionName);
// auth default team
const tmb = await TeamMember.findOne({
const tmb = await MongoTeamMember.findOne({
userId: new Types.ObjectId(userId),
defaultTeam: true
});
@@ -97,7 +73,7 @@ export async function createDefaultTeam({
console.log('create default team', userId);
// create
const { insertedId } = await Team.insertOne({
const { _id: insertedId } = await MongoTeam.create({
ownerId: userId,
name: teamName,
avatar,
@@ -105,7 +81,7 @@ export async function createDefaultTeam({
maxSize,
createTime: new Date()
});
await TeamMember.insertOne({
await MongoTeamMember.create({
teamId: insertedId,
userId,
name: 'Owner',
@@ -116,16 +92,11 @@ export async function createDefaultTeam({
});
} else {
console.log('default team exist', userId);
await Team.updateOne(
{
_id: new Types.ObjectId(tmb.teamId)
},
{
$set: {
...(balance !== undefined && { balance }),
maxSize
}
await MongoTeam.findByIdAndUpdate(tmb.teamId, {
$set: {
...(balance !== undefined && { balance }),
maxSize
}
);
});
}
}

View File

@@ -0,0 +1,51 @@
import { connectionMongo, type Model } from '../../../common/mongo';
const { Schema, model, models } = connectionMongo;
import { TeamMemberSchema as TeamMemberType } from '@fastgpt/global/support/user/team/type.d';
import { userCollectionName } from '../../user/schema';
import {
TeamMemberRoleMap,
TeamMemberStatusMap,
TeamMemberCollectionName,
TeamCollectionName
} from '@fastgpt/global/support/user/team/constant';
const TeamMemberSchema = new Schema({
teamId: {
type: Schema.Types.ObjectId,
ref: TeamCollectionName,
required: true
},
userId: {
type: Schema.Types.ObjectId,
ref: userCollectionName,
required: true
},
name: {
type: String,
default: 'Member'
},
role: {
type: String,
enum: Object.keys(TeamMemberRoleMap)
},
status: {
type: String,
enum: Object.keys(TeamMemberStatusMap)
},
createTime: {
type: Date,
default: () => new Date()
},
defaultTeam: {
type: Boolean,
default: false
}
});
try {
} catch (error) {
console.log(error);
}
export const MongoTeamMember: Model<TeamMemberType> =
models[TeamMemberCollectionName] || model(TeamMemberCollectionName, TeamMemberSchema);

View File

@@ -0,0 +1,41 @@
import { connectionMongo, type Model } from '../../../common/mongo';
const { Schema, model, models } = connectionMongo;
import { TeamSchema as TeamType } from '@fastgpt/global/support/user/team/type.d';
import { userCollectionName } from '../../user/schema';
import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant';
import { PRICE_SCALE } from '@fastgpt/global/support/wallet/bill/constants';
const TeamSchema = new Schema({
name: {
type: String,
required: true
},
ownerId: {
type: Schema.Types.ObjectId,
ref: userCollectionName
},
avatar: {
type: String,
default: '/icon/logo.svg'
},
createTime: {
type: Date,
default: () => Date.now()
},
balance: {
type: Number,
default: 2 * PRICE_SCALE
},
maxSize: {
type: Number,
default: 5
}
});
try {
} catch (error) {
console.log(error);
}
export const MongoTeam: Model<TeamType> =
models[TeamCollectionName] || model(TeamCollectionName, TeamSchema);

View File

@@ -283,6 +283,11 @@
"Speaking": "I'm listening...",
"Stop Speak": "Stop Speak",
"Type a message": "Input problem",
"markdown": {
"Edit Question": "Edit Question",
"Quick Question": "Ask the question immediately",
"Send Question": "Send Question"
},
"quote": {
"Quote Tip": "Only the actual reference content is displayed here. If the data is updated, it will not be updated in real time",
"Read Quote": "Read Quote",
@@ -290,6 +295,9 @@
},
"tts": {
"Stop Speech": "Stop"
},
"error": {
"Messages empty": "Interface content is empty, maybe the text is too long ~"
}
},
"dataset": {
@@ -313,7 +321,6 @@
"Name": "Name",
"Quote Length": "Quote Length",
"Read Dataset": "Read Dataset",
"Search Top K": "Top K",
"Set Empty Result Tip": ",Response empty text",
"Set Website Config": "Configuring Website",
"Similarity": "Similarity",
@@ -327,12 +334,12 @@
"QA Prompt": "QA Prompt",
"Start Sync Tip": "Are you sure to start synchronizing data? The old data will be deleted and then re-acquired, please confirm!",
"Sync": "Data Sync",
"Sync Collection": "Data Sync",
"Website Create Success": "Created successfully, data is being synchronized",
"Website Empty Tip": "No associated website yet",
"Website Link": "Website Link",
"Website Sync": "Website",
"id": "Id",
"Sync Collection": "Data Sync",
"metadata": {
"Chunk Size": "Chunk Size",
"Createtime": "Create Time",
@@ -372,6 +379,12 @@
"id": "Data ID"
},
"error": {
"unAuthDataset": "No access to this knowledge base ",
"unAuthDatasetCollection": "Not authorized to manipulate this data set ",
"unAuthDatasetData": "Not authorized to manipulate this data ",
"unAuthDatasetFile": "No permission to manipulate this file ",
"unCreateCollection": "No permission to manipulate this data ",
"unLinkCollection": "not a network link collection ",
"Start Sync Failed": "Start Sync Failed"
},
"file": "File",
@@ -403,8 +416,11 @@
},
"link": "Link",
"search": {
"Dataset Search Params": "Dataset Search Params",
"Empty result response": "Empty Response",
"Empty result response Tips": "If you fill in the content, if no suitable content is found, you will directly reply to the content.",
"Max Tokens": "Max Tokens",
"Max Tokens Tips": "The maximum number of Tokens in a single search, about 1 word in Chinese =1.7Tokens, about 1 word in English =1 tokens",
"Min Similarity": "Min Similarity",
"Min Similarity Tips": "The similarity of different index models is different, please use the search test to select the appropriate value",
"Params Setting": "Params Setting",
@@ -510,10 +526,13 @@
"variable options": "Options"
},
"variable add option": "Add Option"
},
"shareChat": {
"Init Error": "Init Chat Error",
"Init History Error": "Init History Error"
}
},
"dataset": {
"Chunk Length": "Chunk Length",
"Confirm move the folder": "Confirm Move",
"Confirm to delete the data": "Confirm to delete the data?",
"Confirm to delete the file": "Are you sure to delete the file and all its data?",
@@ -581,13 +600,10 @@
"import csv tip": "Ensure that the CSV is in UTF-8 format; otherwise, garbled characters will be displayed",
"test": {
"noResult": "Search results are empty"
},
"website": {
"Base Url": "BaseUrl",
"Selector": "Selector"
}
},
"error": {
"fileNotFound": "File not found ~",
"team": {
"overSize": "Team member exceeds limit"
}

View File

@@ -283,6 +283,11 @@
"Speaking": "我在听,请说...",
"Stop Speak": "停止录音",
"Type a message": "输入问题",
"markdown": {
"Edit Question": "编辑问题",
"Quick Question": "点我立即提问",
"Send Question": "发送问题"
},
"quote": {
"Quote Tip": "此处仅显示实际引用内容,若数据有更新,此处不会实时更新",
"Read Quote": "查看引用",
@@ -290,6 +295,9 @@
},
"tts": {
"Stop Speech": "停止"
},
"error": {
"Messages empty": "接口内容为空,可能文本超长了~"
}
},
"dataset": {
@@ -313,10 +321,9 @@
"Name": "知识库名称",
"Quote Length": "引用内容长度",
"Read Dataset": "查看知识库详情",
"Search Top K": "单次搜索数量",
"Set Empty Result Tip": ",未搜索到内容时回复指定内容",
"Set Website Config": "开始配置网站信息",
"Similarity": "相度",
"Similarity": "相度",
"Sync Time": "最后更新时间",
"Virtual File": "虚拟文件",
"Website Dataset": "Web 站点同步",
@@ -327,12 +334,12 @@
"QA Prompt": "QA 拆分引导词",
"Start Sync Tip": "确认开始同步数据?将会删除旧数据后重新获取,请确认!",
"Sync": "同步数据",
"Sync Collection": "数据同步",
"Website Create Success": "创建成功,正在同步数据",
"Website Empty Tip": "还没有关联网站",
"Website Link": "Web 站点地址",
"Website Sync": "Web 站点同步",
"id": "集合ID",
"Sync Collection": "数据同步",
"metadata": {
"Chunk Size": "分割大小",
"Createtime": "创建时间",
@@ -372,7 +379,13 @@
"id": "数据ID"
},
"error": {
"Start Sync Failed": "开始同步失败"
"Start Sync Failed": "开始同步失败",
"unAuthDataset": "无权操作该知识库",
"unAuthDatasetCollection": "无权操作该数据集",
"unAuthDatasetData": "无权操作该数据",
"unAuthDatasetFile": "无权操作该文件",
"unCreateCollection": "无权操作该数据",
"unLinkCollection": "不是网络链接集合"
},
"file": "文件",
"folder": "目录",
@@ -403,19 +416,22 @@
},
"link": "链接",
"search": {
"Dataset Search Params": "搜索参数",
"Empty result response": "空搜索回复",
"Empty result response Tips": "若填写该内容,没有搜索到合适内容时,将直接回复填写的内容。",
"Min Similarity": "最低相似度",
"Min Similarity Tips": "不同索引模型的相似度有区别,请通过搜索测试来选择合适的数值",
"Max Tokens": "引用上限",
"Max Tokens Tips": "单次搜索最大的 Tokens 数量中文约1字=1.7Tokens英文约1字=1Tokens",
"Min Similarity": "最低相关度",
"Min Similarity Tips": "不同索引模型的相关度有区别,请通过搜索测试来选择合适的数值,使用 ReRank 时,相关度可能会很低。",
"Params Setting": "搜索参数设置",
"Top K": "单次搜索上限",
"mode": {
"embFullTextReRank": "混合检索",
"embFullTextReRank desc": "使用向量检索与全文检索混合结果进行 Rerank 进行重排,通常效果最佳",
"embFullTextReRank desc": "使用向量检索与全文检索混合结果进行 Rerank 进行重排,相关度通常差异明显,推荐。",
"embedding": "语义检索",
"embedding desc": "直接进行向量 topk 相关性查询",
"embeddingReRank": "增强语义检索",
"embeddingReRank desc": "超额进行向量 topk 查询后再使用 Rerank 进行排序"
"embeddingReRank desc": "超额进行向量 topk 查询后再使用 Rerank 进行排序,相关度通常差异明显。"
},
"search mode": "检索模式"
},
@@ -510,10 +526,13 @@
"variable options": "选项"
},
"variable add option": "添加选项"
},
"shareChat": {
"Init Error": "初始化对话框失败",
"Init History Error": "初始化聊天记录失败"
}
},
"dataset": {
"Chunk Length": "数据总量",
"Confirm move the folder": "确认移动到该目录",
"Confirm to delete the data": "确认删除该数据?",
"Confirm to delete the file": "确认删除该文件及其所有数据?",
@@ -581,13 +600,10 @@
"import csv tip": "请确保CSV为UTF-8格式否则会乱码",
"test": {
"noResult": "搜索结果为空"
},
"website": {
"Base Url": "",
"Selector": ""
}
},
"error": {
"fileNotFound": "文件找不到了~",
"team": {
"overSize": "团队成员超出上限"
}

View File

@@ -69,78 +69,85 @@ const MessageInput = ({
maxCount: 10
});
const uploadFile = async (file: FileItemType) => {
if (file.type === FileTypeEnum.image) {
try {
const src = await compressImgFileAndUpload({
file: file.rawFile,
maxW: 4329,
maxH: 4329,
maxSize: 1024 * 1024 * 5,
// 30 day expired.
expiredTime: addDays(new Date(), 30)
});
setFileList((state) =>
state.map((item) =>
item.id === file.id
? {
...item,
src: `${location.origin}${src}`
}
: item
)
);
} catch (error) {
setFileList((state) => state.filter((item) => item.id !== file.id));
console.log(error);
const uploadFile = useCallback(
async (file: FileItemType) => {
if (file.type === FileTypeEnum.image) {
try {
const src = await compressImgFileAndUpload({
file: file.rawFile,
maxW: 4329,
maxH: 4329,
maxSize: 1024 * 1024 * 5,
// 30 day expired.
expiredTime: addDays(new Date(), 30),
shareId
});
setFileList((state) =>
state.map((item) =>
item.id === file.id
? {
...item,
src: `${location.origin}${src}`
}
: item
)
);
} catch (error) {
setFileList((state) => state.filter((item) => item.id !== file.id));
console.log(error);
toast({
status: 'error',
title: t('common.Upload File Failed')
});
toast({
status: 'error',
title: t('common.Upload File Failed')
});
}
}
}
};
const onSelectFile = useCallback(async (files: File[]) => {
if (!files || files.length === 0) {
return;
}
const loadFiles = await Promise.all(
files.map(
(file) =>
new Promise<FileItemType>((resolve, reject) => {
if (file.type.includes('image')) {
const reader = new FileReader();
reader.readAsDataURL(file);
reader.onload = () => {
const item = {
},
[shareId, t, toast]
);
const onSelectFile = useCallback(
async (files: File[]) => {
if (!files || files.length === 0) {
return;
}
const loadFiles = await Promise.all(
files.map(
(file) =>
new Promise<FileItemType>((resolve, reject) => {
if (file.type.includes('image')) {
const reader = new FileReader();
reader.readAsDataURL(file);
reader.onload = () => {
const item = {
id: nanoid(),
rawFile: file,
type: FileTypeEnum.image,
name: file.name,
icon: reader.result as string
};
uploadFile(item);
resolve(item);
};
reader.onerror = () => {
reject(reader.error);
};
} else {
resolve({
id: nanoid(),
rawFile: file,
type: FileTypeEnum.image,
type: FileTypeEnum.file,
name: file.name,
icon: reader.result as string
};
uploadFile(item);
resolve(item);
};
reader.onerror = () => {
reject(reader.error);
};
} else {
resolve({
id: nanoid(),
rawFile: file,
type: FileTypeEnum.file,
name: file.name,
icon: 'pdf'
});
}
})
)
);
icon: 'pdf'
});
}
})
)
);
setFileList((state) => [...state, ...loadFiles]);
}, []);
setFileList((state) => [...state, ...loadFiles]);
},
[uploadFile]
);
const handleSend = useCallback(async () => {
const textareaValue = TextareaDom.current?.value || '';

View File

@@ -1,5 +1,5 @@
import React, { useMemo, useState } from 'react';
import type { ChatHistoryItemResType } from '@fastgpt/global/core/chat/api.d';
import type { ChatHistoryItemResType } from '@fastgpt/global/core/chat/type.d';
import type { ChatItemType } from '@fastgpt/global/core/chat/type';
import { Flex, BoxProps, useDisclosure, Image, useTheme, Box } from '@chakra-ui/react';
import { useTranslation } from 'next-i18next';

View File

@@ -1,6 +1,6 @@
import React, { useMemo, useState } from 'react';
import { Box, useTheme, Flex, Image } from '@chakra-ui/react';
import type { ChatHistoryItemResType } from '@fastgpt/global/core/chat/api.d';
import type { ChatHistoryItemResType } from '@fastgpt/global/core/chat/type.d';
import { useTranslation } from 'next-i18next';
import { moduleTemplatesFlat } from '@/web/core/modules/template/system';
import Tabs from '../Tabs';
@@ -12,12 +12,21 @@ import { formatPrice } from '@fastgpt/global/support/wallet/bill/tools';
import Markdown from '../Markdown';
import { DatasetSearchModeMap } from '@fastgpt/global/core/dataset/constant';
function Row({ label, value }: { label: string; value?: string | number }) {
function Row({
label,
value,
rawDom
}: {
label: string;
value?: string | number;
rawDom?: React.ReactNode;
}) {
const theme = useTheme();
const val = value || rawDom;
const strValue = `${value}`;
const isCodeBlock = strValue.startsWith('~~~json');
return value !== undefined && value !== '' && value !== 'undefined' ? (
return val !== undefined && val !== '' && val !== 'undefined' ? (
<Box mb={3}>
<Box fontSize={['sm', 'md']} mb={isCodeBlock ? 0 : 1} flex={'0 0 90px'}>
{label}:
@@ -29,7 +38,8 @@ function Row({ label, value }: { label: string; value?: string | number }) {
? { transform: 'translateY(-3px)' }
: { px: 3, py: 1, border: theme.borders.base })}
>
<Markdown source={strValue} />
{value && <Markdown source={strValue} />}
{rawDom}
</Box>
</Box>
) : null;
@@ -113,12 +123,28 @@ const WholeResponseModal = ({
<Row label={t('chat.response.module maxToken')} value={activeModule?.maxToken} />
<Row
label={t('chat.response.module historyPreview')}
value={(() => {
if (!activeModule?.historyPreview) return '';
return activeModule.historyPreview
.map((item, i) => `**${item.obj}**\n${item.value}`)
.join('\n\n---\n\n');
})()}
rawDom={
activeModule.historyPreview ? (
<>
{activeModule.historyPreview?.map((item, i) => (
<Box
key={i}
_notLast={{
borderBottom: '1px solid',
borderBottomColor: 'myWhite.700',
mb: 2
}}
pb={2}
>
<Box fontWeight={'bold'}>{item.obj}</Box>
<Box whiteSpace={'pre-wrap'}>{item.value}</Box>
</Box>
))}
</>
) : (
''
)
}
/>
{activeModule.quoteList && activeModule.quoteList.length > 0 && (
<Row

View File

@@ -12,7 +12,7 @@ import Script from 'next/script';
import { throttle } from 'lodash';
import type { ExportChatType } from '@/types/chat.d';
import type { ChatItemType, ChatSiteItemType } from '@fastgpt/global/core/chat/type.d';
import type { ChatHistoryItemResType } from '@fastgpt/global/core/chat/api.d';
import type { ChatHistoryItemResType } from '@fastgpt/global/core/chat/type.d';
import { useToast } from '@/web/common/hooks/useToast';
import { useAudioPlay } from '@/web/common/utils/voice';
import { getErrText } from '@fastgpt/global/common/error/utils';
@@ -30,7 +30,7 @@ import {
Textarea
} from '@chakra-ui/react';
import { feConfigs } from '@/web/common/system/staticData';
import { eventBus } from '@/web/common/utils/eventbus';
import { EventNameEnum, eventBus } from '@/web/common/utils/eventbus';
import { adaptChat2GptMessages } from '@fastgpt/global/core/chat/adapt';
import { useMarkdown } from '@/web/common/hooks/useMarkdown';
import { ModuleItemType } from '@fastgpt/global/core/module/type.d';
@@ -48,7 +48,7 @@ import type { AdminMarkType } from './SelectMarkCollection';
import MyIcon from '@/components/Icon';
import Avatar from '@/components/Avatar';
import Markdown from '@/components/Markdown';
import Markdown, { CodeClassName } from '@/components/Markdown';
import MySelect from '@/components/Select';
import MyTooltip from '../MyTooltip';
import ChatBoxDivider from '@/components/core/chat/Divider';
@@ -64,6 +64,7 @@ import { splitGuideModule } from '@fastgpt/global/core/module/utils';
import type { AppTTSConfigType } from '@fastgpt/global/core/module/type.d';
import MessageInput from './MessageInput';
import { ModuleOutputKeyEnum } from '@fastgpt/global/core/module/constants';
import { FlowNodeTypeEnum } from '@fastgpt/global/core/module/node/constant';
const nanoid = customAlphabet('abcdefghijklmnopqrstuvwxyz1234567890', 24);
@@ -80,7 +81,7 @@ export type StartChatFnProps = {
};
export type ComponentRef = {
getChatHistory: () => ChatSiteItemType[];
getChatHistories: () => ChatSiteItemType[];
resetVariables: (data?: Record<string, any>) => void;
resetHistory: (history: ChatSiteItemType[]) => void;
scrollToBottom: (behavior?: 'smooth' | 'auto') => void;
@@ -132,9 +133,10 @@ const ChatBox = (
const ChatBoxRef = useRef<HTMLDivElement>(null);
const theme = useTheme();
const router = useRouter();
const { shareId } = router.query as { shareId?: string };
const { t } = useTranslation();
const { toast } = useToast();
const { isPc } = useSystemStore();
const { isPc, setLoading } = useSystemStore();
const TextareaDom = useRef<HTMLTextAreaElement>(null);
const chatController = useRef(new AbortController());
const questionGuideController = useRef(new AbortController());
@@ -258,7 +260,7 @@ const ChatBox = (
const result = await postQuestionGuide(
{
messages: adaptChat2GptMessages({ messages: history, reserveId: false }).slice(-6),
shareId: router.query.shareId as string
shareId
},
abortSignal
);
@@ -270,7 +272,7 @@ const ChatBox = (
}
} catch (error) {}
},
[questionGuide, scrollToBottom, router.query.shareId]
[questionGuide, scrollToBottom, shareId]
);
/**
@@ -323,7 +325,6 @@ const ChatBox = (
setTimeout(() => {
scrollToBottom();
}, 100);
try {
// create abort obj
const abortSignal = new AbortController();
@@ -415,15 +416,20 @@ const ChatBox = (
async (index: number) => {
if (!onDelMessage) return;
const delHistory = chatHistory.slice(index);
setChatHistory((state) => (index === 0 ? [] : state.slice(0, index)));
await Promise.all(
delHistory.map((item, i) => onDelMessage({ contentId: item.dataId, index: index + i }))
);
setLoading(true);
sendPrompt(variables, delHistory[0].value, chatHistory.slice(0, index));
try {
await Promise.all(
delHistory.map((item, i) => onDelMessage({ contentId: item.dataId, index: index + i }))
);
setChatHistory((state) => (index === 0 ? [] : state.slice(0, index)));
sendPrompt(variables, delHistory[0].value, chatHistory.slice(0, index));
} catch (error) {}
setLoading(false);
},
[chatHistory, onDelMessage, sendPrompt, variables]
[chatHistory, onDelMessage, sendPrompt, setLoading, variables]
);
// delete one message
const delOneMessage = useCallback(
@@ -439,7 +445,7 @@ const ChatBox = (
// output data
useImperativeHandle(ref, () => ({
getChatHistory: () => chatHistory,
getChatHistories: () => chatHistory,
resetVariables(e) {
const defaultVal: Record<string, any> = {};
variableModules?.forEach((item) => {
@@ -513,16 +519,22 @@ const ChatBox = (
}
};
window.addEventListener('message', windowMessage);
eventBus.on('guideClick', ({ text }: { text: string }) => {
eventBus.on(EventNameEnum.sendQuestion, ({ text }: { text: string }) => {
if (!text) return;
handleSubmit((data) => sendPrompt(data, text))();
});
eventBus.on(EventNameEnum.editQuestion, ({ text }: { text: string }) => {
if (!text) return;
resetInputVal(text);
});
return () => {
eventBus.off('guideClick');
eventBus.off(EventNameEnum.sendQuestion);
eventBus.off(EventNameEnum.editQuestion);
window.removeEventListener('message', windowMessage);
};
}, [handleSubmit, sendPrompt]);
}, [handleSubmit, resetInputVal, sendPrompt]);
return (
<Flex flexDirection={'column'} h={'100%'}>
@@ -752,40 +764,30 @@ const ChatBox = (
<Box textAlign={'left'} mt={['6px', 2]}>
<Card bg={'white'} {...MessageCardStyle}>
<Markdown
source={item.value}
source={(() => {
const text = item.value as string;
// replace quote tag: [source1] 标识第一个来源需要提取数字1从而去数组里查找来源
const quoteReg = /\[source:(.+)\]/g;
const replaceText = text.replace(quoteReg, `[QUOTE SIGN]($1)`);
// question guide
if (
index === chatHistory.length - 1 &&
!isChatting &&
questionGuides.length > 0
) {
return `${replaceText}\n\`\`\`${
CodeClassName.questionGuide
}\n${JSON.stringify(questionGuides)}`;
}
return replaceText;
})()}
isChatting={index === chatHistory.length - 1 && isChatting}
/>
<ResponseTags responseData={item.responseData} />
{/* question guide */}
{index === chatHistory.length - 1 &&
!isChatting &&
questionGuides.length > 0 && (
<Box mt={2}>
<ChatBoxDivider
icon="core/chat/QGFill"
text={t('chat.Question Guide Tips')}
/>
<Flex alignItems={'center'} flexWrap={'wrap'} gap={2}>
{questionGuides.map((item) => (
<Button
key={item}
borderRadius={'md'}
variant={'outline'}
colorScheme={'gray'}
size={'xs'}
whiteSpace={'pre-wrap'}
h={'auto'}
py={1}
onClick={() => {
resetInputVal(item);
}}
>
{item}
</Button>
))}
</Flex>
</Box>
)}
{/* admin mark content */}
{showMarkIcon && item.adminFeedback && (
<Box>

View File

@@ -0,0 +1,11 @@
<?xml version="1.0" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"><svg t="1701930523211"
class="icon" viewBox="0 0 1024 1024" version="1.1" xmlns="http://www.w3.org/2000/svg" p-id="9398"
xmlns:xlink="http://www.w3.org/1999/xlink" width="128" height="128">
<path
d="M742.4 179.2H281.6a128 128 0 0 0-128 128v596.7104L298.496 768H742.4a128 128 0 0 0 128-128V307.2a128 128 0 0 0-128-128zM281.6 230.4h460.8a76.8 76.8 0 0 1 76.8 76.8v332.8a76.8 76.8 0 0 1-76.8 76.8H278.272L204.8 785.664V307.2a76.8 76.8 0 0 1 76.8-76.8z"
p-id="9399"></path>
<path
d="M534.016 525.2608v-0.1792c0-66.9952 36.3008-129.1008 95.6416-163.6352a22.3744 22.3744 0 0 1 30.1312 7.2192 20.8384 20.8384 0 0 1-7.424 29.1584 150.7584 150.7584 0 0 0-59.8528 64.0768c27.0848-2.9184 53.248 10.624 65.7664 34.048a62.0288 62.0288 0 0 1-9.3952 71.6032 67.328 67.328 0 0 1-72.4992 17.0752c-25.472-9.3696-42.3424-32.9984-42.368-59.392z m-175.616 0v-0.1792c0-66.9952 36.3008-129.1008 95.6416-163.6352a22.3744 22.3744 0 0 1 30.1568 7.2192 20.8384 20.8384 0 0 1-7.4496 29.1584 150.7584 150.7584 0 0 0-59.8528 64.0768c27.1104-2.9184 53.248 10.624 65.792 34.048a62.0288 62.0288 0 0 1-9.4208 71.6032 67.328 67.328 0 0 1-72.4736 17.0752c-25.4976-9.3696-42.3424-32.9984-42.3936-59.392z"
p-id="9400"></path>
</svg>

After

Width:  |  Height:  |  Size: 1.3 KiB

View File

@@ -0,0 +1,8 @@
<?xml version="1.0" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"><svg t="1701927696489"
class="icon" viewBox="0 0 1024 1024" version="1.1" xmlns="http://www.w3.org/2000/svg" p-id="5076"
xmlns:xlink="http://www.w3.org/1999/xlink" width="128" height="128">
<path
d="M1015.485274 476.463913c-7.599113-15.198226-20.197642-27.796755-35.395868-35.395868L114.790411 8.418547c-18.997782-9.498891-40.495273-10.998716-60.592927-4.299498S17.901721 25.01661 8.40283 43.914404C-1.496015 63.812081-2.695875 87.009374 5.303191 107.806946l155.381863 404.252812L5.303191 916.212582c-7.599113 19.797689-6.999183 41.395168 1.599814 60.692915 8.598996 19.397736 24.297164 34.196008 43.994864 41.795122 9.198926 3.499591 18.797806 5.299381 28.496674 5.299381 12.198576 0 24.397152-2.799673 35.495856-8.299031L980.089406 582.951483c39.095436-19.597712 54.993581-67.392133 35.395868-106.48757zM79.094578 944.509279l151.182353-392.954131h310.363771c21.797456 0 39.49539-17.697934 39.49539-39.49539s-17.697934-39.49539-39.49539-39.49539H230.276931L79.494531 79.210284l865.199007 432.649497L79.094578 944.509279z"
p-id="5077"></path>
</svg>

After

Width:  |  Height:  |  Size: 1.2 KiB

View File

@@ -103,6 +103,8 @@ const iconPaths = {
'core/app/tts': () => import('./icons/core/app/tts.svg'),
'core/app/headphones': () => import('./icons/core/app/headphones.svg'),
'common/playLight': () => import('./icons/common/playLight.svg'),
'core/chat/quoteSign': () => import('./icons/core/chat/quoteSign.svg'),
'core/chat/sendLight': () => import('./icons/core/chat/sendLight.svg'),
'core/chat/sendFill': () => import('./icons/core/chat/sendFill.svg'),
'core/chat/recordFill': () => import('./icons/core/chat/recordFill.svg'),
'core/chat/stopSpeechFill': () => import('./icons/core/chat/stopSpeechFill.svg'),

View File

@@ -296,7 +296,7 @@ const CodeLight = ({
}) => {
const { copyData } = useCopyData();
if (!inline && match) {
if (!inline) {
return (
<Box my={3} borderRadius={'md'} overflow={'overlay'} backgroundColor={'#222'}>
<Flex
@@ -315,7 +315,7 @@ const CodeLight = ({
</Flex>
</Flex>
<SyntaxHighlighter style={codeLight as any} language={match?.[1]} PreTag="pre">
{String(children)}
{String(children).replace(/&nbsp;/g, ' ')}
</SyntaxHighlighter>
</Box>
);

View File

@@ -5,7 +5,7 @@ import RemarkGfm from 'remark-gfm';
import RemarkMath from 'remark-math';
import RehypeKatex from 'rehype-katex';
import RemarkBreaks from 'remark-breaks';
import { eventBus } from '@/web/common/utils/eventbus';
import { EventNameEnum, eventBus } from '@/web/common/utils/eventbus';
import 'katex/dist/katex.min.css';
import styles from '../index.module.scss';
@@ -27,7 +27,7 @@ function MyLink(e: any) {
textDecoration={'underline'}
cursor={'pointer'}
onClick={() => {
eventBus.emit('guideClick', { text });
eventBus.emit(EventNameEnum.sendQuestion, { text });
}}
>
{text}

View File

@@ -0,0 +1,92 @@
import React, { useMemo } from 'react';
import { Box, Flex, useTheme } from '@chakra-ui/react';
import 'katex/dist/katex.min.css';
import ChatBoxDivider from '@/components/core/chat/Divider';
import { useTranslation } from 'next-i18next';
import { EventNameEnum, eventBus } from '@/web/common/utils/eventbus';
import MyTooltip from '@/components/MyTooltip';
import MyIcon from '@/components/Icon';
const QuestionGuide = ({ text }: { text: string }) => {
const theme = useTheme();
const { t } = useTranslation();
const questionGuides = useMemo(() => {
try {
const json = JSON.parse(text);
if (Array.isArray(json) && !json.find((item) => typeof item !== 'string')) {
return json as string[];
}
return [];
} catch (error) {
return [];
}
}, [text]);
return questionGuides.length > 0 ? (
<Box mt={2}>
<ChatBoxDivider icon="core/chat/QGFill" text={t('chat.Question Guide Tips')} />
<Flex alignItems={'center'} flexWrap={'wrap'} gap={2}>
{questionGuides.map((text) => (
<Flex
key={text}
alignItems={'center'}
flexWrap={'wrap'}
fontSize={'sm'}
border={theme.borders.sm}
py={'1px'}
px={3}
borderRadius={'md'}
_hover={{
'.controller': {
display: 'flex'
}
}}
overflow={'hidden'}
position={'relative'}
>
<Box className="textEllipsis" flex={'1 0 0'}>
{text}
</Box>
<Box
className="controller"
display={['flex', 'none']}
pr={2}
position={'absolute'}
right={0}
left={0}
justifyContent={'flex-end'}
alignItems={'center'}
h={'100%'}
lineHeight={0}
bg={`linear-gradient(to left, white,white min(60px,100%),rgba(255,255,255,0) 80%)`}
>
<MyTooltip label={t('core.chat.markdown.Edit Question')}>
<MyIcon
name={'edit'}
w={'14px'}
cursor={'pointer'}
_hover={{
color: 'green.600'
}}
onClick={() => eventBus.emit(EventNameEnum.editQuestion, { text })}
/>
</MyTooltip>
<MyTooltip label={t('core.chat.markdown.Send Question')}>
<MyIcon
ml={4}
name={'core/chat/sendLight'}
w={'14px'}
cursor={'pointer'}
_hover={{ color: 'myBlue.600' }}
onClick={() => eventBus.emit(EventNameEnum.sendQuestion, { text })}
/>
</MyTooltip>
</Box>
</Flex>
))}
</Flex>
</Box>
) : null;
};
export default React.memo(QuestionGuide);

View File

@@ -1,64 +0,0 @@
import React, { useMemo } from 'react';
import { Box, useTheme } from '@chakra-ui/react';
import { getFileAndOpen } from '@/web/core/dataset/utils';
import { useToast } from '@/web/common/hooks/useToast';
import { getErrText } from '@fastgpt/global/common/error/utils';
type QuoteItemType = {
file_id?: string;
filename: string;
};
const QuoteBlock = ({ code }: { code: string }) => {
const theme = useTheme();
const { toast } = useToast();
const quoteList = useMemo(() => {
try {
return JSON.parse(code) as QuoteItemType[];
} catch (error) {
return [];
}
}, [code]);
return (
<Box mt={3} pt={2} borderTop={theme.borders.base}>
{quoteList.length > 0 ? (
<>
<Box>:</Box>
<Box as={'ol'}>
{quoteList.map((item, i) => (
<Box
key={i}
as={'li'}
{...(item.file_id
? {
textDecoration: 'underline',
color: 'myBlue.800',
cursor: 'pointer'
}
: {})}
onClick={async () => {
if (!item.file_id) return;
try {
await getFileAndOpen(item.file_id);
} catch (error) {
toast({
status: 'warning',
title: getErrText(error, '打开文件失败')
});
}
}}
>
{item.filename}
</Box>
))}
</Box>
</>
) : (
<Box></Box>
)}
</Box>
);
};
export default QuoteBlock;

View File

@@ -46,7 +46,7 @@ const MdImage = ({ src }: { src?: string }) => {
/>
<Modal isOpen={isOpen} onClose={onClose} isCentered>
<ModalOverlay />
<ModalContent maxW={'auto'} w="auto" bg={'transparent'}>
<ModalContent boxShadow={'none'} maxW={'auto'} w="auto" bg={'transparent'}>
<Image
borderRadius={'md'}
src={src}

View File

@@ -308,8 +308,9 @@
}
.markdown code,
.markdown tt {
border: 1px solid #eaeaea;
border-radius: 3px 3px 3px 3px;
border: 1px solid #dee0e2;
background-color: #f4f6f8;
border-radius: 3px;
margin: 0 2px;
padding: 0 5px;
}

View File

@@ -9,17 +9,26 @@ import 'katex/dist/katex.min.css';
import styles from './index.module.scss';
import dynamic from 'next/dynamic';
import CodeLight from './CodeLight';
import { Link, Button } from '@chakra-ui/react';
import MyTooltip from '../MyTooltip';
import { useTranslation } from 'next-i18next';
import { EventNameEnum, eventBus } from '@/web/common/utils/eventbus';
import MyIcon from '../Icon';
import { getFileAndOpen } from '@/web/core/dataset/utils';
import { MARKDOWN_QUOTE_SIGN } from '@fastgpt/global/core/chat/constants';
const CodeLight = dynamic(() => import('./CodeLight'));
const MermaidCodeBlock = dynamic(() => import('./img/MermaidCodeBlock'));
const MdImage = dynamic(() => import('./img/Image'));
const ChatGuide = dynamic(() => import('./chat/Guide'));
const EChartsCodeBlock = dynamic(() => import('./img/EChartsCodeBlock'));
const QuoteBlock = dynamic(() => import('./chat/Quote'));
const ChatGuide = dynamic(() => import('./chat/Guide'));
const QuestionGuide = dynamic(() => import('./chat/QuestionGuide'));
const ImageBlock = dynamic(() => import('./chat/Image'));
export enum CodeClassName {
guide = 'guide',
questionGuide = 'questionGuide',
mermaid = 'mermaid',
echarts = 'echarts',
quote = 'quote',
@@ -37,12 +46,12 @@ function Code({ inline, className, children }: any) {
if (codeType === CodeClassName.guide) {
return <ChatGuide text={String(children)} />;
}
if (codeType === CodeClassName.questionGuide) {
return <QuestionGuide text={String(children)} />;
}
if (codeType === CodeClassName.echarts) {
return <EChartsCodeBlock code={String(children)} />;
}
if (codeType === CodeClassName.quote) {
return <QuoteBlock code={String(children)} />;
}
if (codeType === CodeClassName.img) {
return <ImageBlock images={String(children)} />;
}
@@ -55,6 +64,52 @@ function Code({ inline, className, children }: any) {
function Image({ src }: { src?: string }) {
return <MdImage src={src} />;
}
function A({ children, ...props }: any) {
const { t } = useTranslation();
// empty href link
if (!props.href && typeof children?.[0] === 'string') {
const text = useMemo(() => String(children), [children]);
return (
<MyTooltip label={t('core.chat.markdown.Quick Question')}>
<Button
variant={'base'}
size={'xs'}
borderRadius={'md'}
my={1}
onClick={() => eventBus.emit(EventNameEnum.sendQuestion, { text })}
>
{text}
</Button>
</MyTooltip>
);
}
// quote link
if (children?.length === 1 && typeof children?.[0] === 'string') {
const text = String(children);
if (text === MARKDOWN_QUOTE_SIGN && props.href) {
return (
<MyTooltip label={props.href}>
<MyIcon
name={'core/chat/quoteSign'}
transform={'translateY(-2px)'}
w={'18px'}
color={'myBlue.600'}
cursor={'pointer'}
_hover={{
color: 'myBlue.800'
}}
onClick={() => getFileAndOpen(props.href)}
/>
</MyTooltip>
);
}
}
return <Link {...props}>{children}</Link>;
}
const Markdown = ({ source, isChatting = false }: { source: string; isChatting?: boolean }) => {
const components = useMemo(
@@ -62,14 +117,16 @@ const Markdown = ({ source, isChatting = false }: { source: string; isChatting?:
img: Image,
pre: 'div',
p: 'div',
code: Code
code: Code,
a: A
}),
[]
);
const formatSource = source
.replace(/\\n/g, '\n&nbsp;')
.replace(/(http[s]?:\/\/[^\s。]+)([。,])/g, '$1 $2');
.replace(/(http[s]?:\/\/[^\s。]+)([。,])/g, '$1 $2')
.replace(/\n*(\[QUOTE SIGN\]\(.*\))/g, '$1');
return (
<ReactMarkdown

View File

@@ -82,7 +82,7 @@ const MyRadio = ({
<Box pr={2}>
<Box>{t(item.title)}</Box>
{!!item.desc && (
<Box fontSize={'sm'} color={'myGray.500'}>
<Box fontSize={['xs', 'sm']} color={'myGray.500'}>
{t(item.desc)}
</Box>
)}

View File

@@ -18,6 +18,7 @@ type DatasetParamsProps = {
limit?: number;
searchMode: `${DatasetSearchModeEnum}`;
searchEmptyText?: string;
maxTokens?: number;
};
const DatasetParamsModal = ({
@@ -25,6 +26,7 @@ const DatasetParamsModal = ({
limit,
similarity,
searchMode = DatasetSearchModeEnum.embedding,
maxTokens = 3000,
onClose,
onSuccess
}: DatasetParamsProps & { onClose: () => void; onSuccess: (e: DatasetParamsProps) => void }) => {
@@ -52,8 +54,8 @@ const DatasetParamsModal = ({
isOpen={true}
onClose={onClose}
iconSrc="/imgs/modal/params.svg"
title={'搜索参数调整'}
minW={['90vw', '500px']}
title={t('core.dataset.search.Dataset Search Params')}
w={['90vw', '550px']}
h={['90vh', 'auto']}
overflow={'unset'}
isCentered={searchEmptyText !== undefined}
@@ -78,36 +80,42 @@ const DatasetParamsModal = ({
<QuestionOutlineIcon ml={1} />
</MyTooltip>
</Box>
<MySlider
markList={[
{ label: '0', value: 0 },
{ label: '1', value: 1 }
]}
min={0}
max={1}
step={0.01}
value={getValues(ModuleInputKeyEnum.datasetSimilarity) || 0.5}
onChange={(val) => {
setValue(ModuleInputKeyEnum.datasetSimilarity, val);
setRefresh(!refresh);
}}
/>
<Box flex={1} mx={4}>
<MySlider
markList={[
{ label: '0', value: 0 },
{ label: '1', value: 1 }
]}
min={0}
max={1}
step={0.01}
value={getValues(ModuleInputKeyEnum.datasetSimilarity) ?? 0.5}
onChange={(val) => {
setValue(ModuleInputKeyEnum.datasetSimilarity, val);
setRefresh(!refresh);
}}
/>
</Box>
</Box>
)}
{limit !== undefined && (
<Box display={['block', 'flex']} py={8}>
<Box flex={'0 0 100px'} mb={[8, 0]}>
{t('core.dataset.search.Top K')}
{t('core.dataset.search.Max Tokens')}
<MyTooltip label={t('core.dataset.search.Max Tokens Tips')} forceShow>
<QuestionOutlineIcon ml={1} />
</MyTooltip>
</Box>
<Box flex={1}>
<Box flex={1} mx={4}>
<MySlider
markList={[
{ label: '1', value: 1 },
{ label: '30', value: 30 }
{ label: '300', value: 300 },
{ label: maxTokens, value: maxTokens }
]}
min={1}
max={30}
value={getValues(ModuleInputKeyEnum.datasetLimit) || 5}
min={300}
max={maxTokens}
step={10}
value={getValues(ModuleInputKeyEnum.datasetLimit) ?? 1000}
onChange={(val) => {
setValue(ModuleInputKeyEnum.datasetLimit, val);
setRefresh(!refresh);

View File

@@ -17,7 +17,7 @@ import {
Grid,
Switch
} from '@chakra-ui/react';
import { FlowNodeInputTypeEnum } from '@fastgpt/global/core/module/node/constant';
import { FlowNodeInputTypeEnum, FlowNodeTypeEnum } from '@fastgpt/global/core/module/node/constant';
import { QuestionOutlineIcon } from '@chakra-ui/icons';
import dynamic from 'next/dynamic';
import { onChangeNode, useFlowProviderStore } from '../../FlowProvider';
@@ -37,6 +37,7 @@ import { useQuery } from '@tanstack/react-query';
import type { EditFieldModeType, EditFieldType } from '../modules/FieldEditModal';
import { feConfigs } from '@/web/common/system/staticData';
import { DatasetSearchModeEnum } from '@fastgpt/global/core/dataset/constant';
import { ModuleInputKeyEnum } from '@fastgpt/global/core/module/constants';
const FieldEditModal = dynamic(() => import('../modules/FieldEditModal'));
const SelectAppModal = dynamic(() => import('../../SelectAppModal'));
@@ -635,9 +636,12 @@ const SelectAppRender = React.memo(function SelectAppRender({ item, moduleId }:
});
const SelectDatasetParamsRender = React.memo(function SelectDatasetParamsRender({
item,
inputs = [],
moduleId
}: RenderProps) {
const { nodes } = useFlowProviderStore();
const { t } = useTranslation();
const [data, setData] = useState({
searchMode: DatasetSearchModeEnum.embedding,
@@ -645,6 +649,23 @@ const SelectDatasetParamsRender = React.memo(function SelectDatasetParamsRender(
similarity: 0.5
});
const tokenLimit = useMemo(() => {
let maxTokens = 3000;
nodes.forEach((item) => {
if (item.type === FlowNodeTypeEnum.chatNode) {
const model =
item.data.inputs.find((item) => item.key === ModuleInputKeyEnum.aiModel)?.value || '';
const quoteMaxToken =
chatModelList.find((item) => item.model === model)?.quoteMaxToken || 3000;
maxTokens = Math.max(maxTokens, quoteMaxToken);
}
});
return maxTokens;
}, [nodes]);
const { isOpen, onOpen, onClose } = useDisclosure();
useEffect(() => {
@@ -671,6 +692,7 @@ const SelectDatasetParamsRender = React.memo(function SelectDatasetParamsRender(
{isOpen && (
<DatasetParamsModal
{...data}
maxTokens={tokenLimit}
onClose={onClose}
onSuccess={(e) => {
for (let key in e) {

View File

@@ -4,7 +4,7 @@ import type { OutLinkEditType } from '@fastgpt/global/support/outLink/type.d';
export const defaultApp: AppDetailType = {
_id: '',
userId: 'userId',
name: '模型加载中',
name: '应用加载中',
type: 'simple',
simpleTemplateId: 'fastgpt-universal',
avatar: '/icon/logo.svg',

View File

@@ -1,7 +1,75 @@
import type { AppTTSConfigType } from '@fastgpt/global/core/module/type.d';
import { ModuleItemType } from '../module/type';
import { AdminFbkType, ChatItemType, moduleDispatchResType } from '@fastgpt/global/core/chat/type';
export type GetChatSpeechProps = {
ttsConfig: AppTTSConfigType;
input: string;
shareId?: string;
};
/* ---------- chat ----------- */
export type InitChatProps = {
appId?: string;
chatId?: string;
};
export type InitOutLinkChatProps = {
chatId?: string;
shareId?: string;
outLinkUid?: string;
};
export type InitChatResponse = {
chatId?: string;
appId: string;
userAvatar?: string;
title: string;
variables: Record<string, any>;
history: ChatItemType[];
app: {
userGuideModule?: ModuleItemType;
chatModels?: string[];
name: string;
avatar: string;
intro: string;
canUse?: boolean;
};
};
/* ---------- history ----------- */
export type getHistoriesProps = {
appId?: string;
// share chat
shareId?: string;
outLinkUid?: string; // authToken/uid
};
export type UpdateHistoryProps = {
chatId: string;
customTitle?: string;
top?: boolean;
shareId?: string;
outLinkUid?: string;
};
export type DelHistoryProps = {
chatId: string;
shareId?: string;
outLinkUid?: string;
};
export type ClearHistoriesProps = {
appId?: string;
shareId?: string;
outLinkUid?: string;
};
/* -------- chat item ---------- */
export type DeleteChatItemProps = {
chatId: string;
contentId?: string;
shareId?: string;
outLinkUid?: string;
};
export type AdminUpdateFeedbackParams = AdminFbkType & {
chatItemId: string;
};

View File

@@ -0,0 +1,15 @@
import { InitChatResponse } from './api';
export const defaultChatData: InitChatResponse = {
chatId: '',
appId: '',
app: {
name: 'Loading',
avatar: '/icon/logo.svg',
intro: '',
canUse: false
},
title: '新对话',
variables: {},
history: []
};

View File

@@ -4,22 +4,42 @@ export const Prompt_QuoteTemplateList: PromptTemplateItem[] = [
{
title: '标准模板',
desc: '标准提示词,用于结构不固定的知识库。',
value: `{{q}}\n{{a}}`
value: `<data>
{{q}}
{{a}}
</data>`
},
{
title: '问答模板',
desc: '适合 QA 问答结构的知识库,或大部分核心介绍位于 a 的知识库。',
value: `{instruction:"{{q}}",output:"{{a}}"}`
desc: '适合 QA 问答结构的知识库,可以让AI较为严格的按预设内容回答',
value: `<QA>
<问题>
{{q}}
</问题>
<答案>
{{a}}
</答案>
</QA>`
},
{
title: '标准严格模板',
desc: '在标准模板基础上,对模型的回答做更严格的要求。',
value: `{{q}}\n{{a}}`
value: `<data>
{{q}}
{{a}}
</data>`
},
{
title: '严格问答模板',
desc: '在问答模板基础上,对模型的回答做更严格的要求。',
value: `{question:"{{q}}",answer:"{{a}}"}`
value: `<QA>
<问题>
{{q}}
</问题>
<答案>
{{a}}
</答案>
</QA>`
}
];
@@ -27,56 +47,70 @@ export const Prompt_QuotePromptList: PromptTemplateItem[] = [
{
title: '标准模板',
desc: '',
value: `你的知识:
"""
value: `使用 <data></data> 标记中的内容作为你的知识:
{{quote}}
"""
回答要求:
1. 优先使用知识库内容回答问题
2. 你可以回答我不知道
3. 不要提及你是从知识库获取的知识
4. 知识库包含 markdown 内容时,按 markdown 格式返回
我的问题是:"{{question}}"`
- 如果你不清楚答案,你需要澄清
- 避免提及你是从 data 获取的知识
- 保持答案与 data 中描述的一致
- 使用 Markdown 语法优化回答格式
- 使用与问题相同的语言回答。
问题:"{{question}}"`
},
{
title: '问答模板',
desc: '',
value: `你的知识库:
"""
value: `使用 <QA></QA> 标记中的问答对进行回答。
{{quote}}
"""
回答要求:
1. 优先使用知识库内容回答问题,其中 instruction 是相关介绍output 是预期回答或补充
2. 你可以回答我不知道
3. 不要提及你是从知识库获取的知识
4. 知识库包含 markdown 内容时,按 markdown 格式返回
我的问题是:"{{question}}"`
- 选择其中一个或多个问答对进行回答
- 回答的内容应尽可能与 <答案></答案> 中的内容一致
- 如果没有相关的问答对,你需要澄清
- 避免提及你是从 QA 获取的知识,只需要回复答案
问题:"{{question}}"`
},
{
title: '标准严格模板',
desc: '',
value: `你的知识:
"""
value: `忘记你已有的知识,仅使用 <data></data> 标记中的内容作为你的知识:
{{quote}}
"""
思考流程:
1. 判断问题是否与 <data></data> 标记中的内容有关。
2. 如果有关,你按下面的要求回答。
3. 如果无关,你直接拒绝回答本次问题。
回答要求:
1. 仅使用知识库内容回答问题
2. 与知识库无关的问题,你直接回答我不知道
3. 不要提及你是从知识库获取的知识
4. 知识库包含 markdown 内容时,按 markdown 格式返回
我的问题是:"{{question}}"`
- 避免提及你是从 data 获取的知识
- 保持答案与 data 中描述的一致
- 使用 Markdown 语法优化回答格式
- 使用与问题相同的语言回答
问题:"{{question}}"`
},
{
title: '严格问答模板',
desc: '',
value: `你的知识库:
"""
value: `忘记你已有的知识,仅使用 <QA></QA> 标记中的问答对进行回答。
{{quote}}
"""
回答要求
1. 从知识库中选择一个合适的答案进行回答,其中 instruction 是相关问题answer 是已知答案
2. 与知识库无关的问题,你直接回答我不知道
3. 不要提及你是从知识库获取的知识
我的问题是:"{{question}}"`
思考流程
1. 判断问题是否与 <QA></QA> 标记中的内容有关
2. 如果无关,你直接拒绝回答本次问题
3. 判断是否有相近或相同的问题
4. 如果有相同的问题,直接输出对应答案。
5. 如果只有相近的问题,请把相近的问题和答案一起输出。
最后,避免提及你是从 QA 获取的知识,只需要回复答案。
问题:"{{question}}"`
}
];

View File

@@ -1,8 +1,10 @@
export const Prompt_AgentQA = {
description: `我会给你一段文本,学习它们,并整理学习成果,要求为:
1. 提出问题并给出每个问题的答案。
2. 每个答案都要详细完整,给出相关原文描述,答案可以包含普通文字、链接、代码、表格、公示、媒体链接等 markdown 元素。
3. 最多提出 30 个问题
description: `<context></context> 标记中是一段文本,学习它们,并整理学习成果
学习要求:
- 提出问题并给出每个问题的答案
- 答案需详细完整,给出相关原文描述。答案可以包含普通文字、链接、代码、表格、公示、媒体链接等 markdown 元素。
- 最多提出 30 个问题。
`,
fixedText: `最后,你需要按下面的格式返回多个问题和答案:
Q1: 问题。
@@ -11,7 +13,10 @@ Q2:
A2:
……
我的文本:"""{{text}}"""`
<context>
{{text}}
<context/>
`
};
export const Prompt_ExtractJson = `你可以从 "对话记录" 中提取指定信息,并返回一个 JSON 对象JSON 对象要求:

View File

@@ -54,7 +54,7 @@ const PayModal = ({ onClose }: { onClose: () => void }) => {
onSuccess(res) {
if (!res) return;
toast({
title: '充值成功',
title: res,
status: 'success'
});
router.reload();

View File

@@ -2,7 +2,10 @@ import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo';
import { authCert } from '@fastgpt/service/support/permission/auth/common';
import { delFileById, getGFSCollection } from '@fastgpt/service/common/file/gridfs/controller';
import {
delFileByFileIdList,
getGFSCollection
} from '@fastgpt/service/common/file/gridfs/controller';
import { addLog } from '@fastgpt/service/common/mongo/controller';
import { MongoDatasetCollection } from '@fastgpt/service/core/dataset/collection/schema';
import { delay } from '@fastgpt/global/common/system/utils';
@@ -77,7 +80,7 @@ export async function checkFiles(start: Date, end: Date, limit: number) {
// 3. if not found, delete file
if (hasCollection === 0) {
await delFileById({ bucketName: 'dataset', fileId: String(_id) });
await delFileByFileIdList({ bucketName: 'dataset', fileIdList: [String(_id)] });
console.log('delete file', _id);
deleteFileAmount++;
}

View File

@@ -4,6 +4,7 @@ import { connectToDatabase } from '@/service/mongo';
import { authFileToken } from '@fastgpt/service/support/permission/controller';
import { detect } from 'jschardet';
import { getDownloadStream, getFileById } from '@fastgpt/service/common/file/gridfs/controller';
import { CommonErrEnum } from '@fastgpt/global/common/error/code/common';
export default async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
try {
@@ -22,6 +23,10 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse<
getDownloadStream({ bucketName, fileId })
]);
if (!file) {
return Promise.reject(CommonErrEnum.fileNotFound);
}
// get encoding
let buffers: Buffer = Buffer.from([]);
for await (const chunk of encodeStream) {

View File

@@ -1,21 +1,22 @@
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo';
import { authCert } from '@fastgpt/service/support/permission/auth/common';
import { authCertOrShareId } from '@fastgpt/service/support/permission/auth/common';
import { uploadMongoImg } from '@fastgpt/service/common/file/image/controller';
type Props = { base64Img: string; expiredTime?: Date };
import { UploadImgProps } from '@fastgpt/global/common/file/api';
export default async function handler(req: NextApiRequest, res: NextApiResponse) {
try {
await connectToDatabase();
const { teamId } = await authCert({ req, authToken: true });
const { base64Img, expiredTime } = req.body as Props;
const { base64Img, expiredTime, metadata, shareId } = req.body as UploadImgProps;
const { teamId } = await authCertOrShareId({ req, shareId, authToken: true });
const data = await uploadMongoImg({
teamId,
base64Img,
expiredTime
expiredTime,
metadata
});
jsonRes(res, { data });

View File

@@ -1,32 +1,14 @@
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo';
import { MongoDatasetTraining } from '@fastgpt/service/core/dataset/training/schema';
import { startQueue } from '@/service/utils/tools';
import { authCert } from '@fastgpt/service/support/permission/auth/common';
export default async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
try {
await connectToDatabase();
const { userId } = await authCert({ req, authToken: true });
await unlockTask(userId);
await authCert({ req, authToken: true });
startQueue();
} catch (error) {}
jsonRes(res);
}
async function unlockTask(userId: string) {
try {
await MongoDatasetTraining.updateMany(
{
userId
},
{
lockTime: new Date('2000/1/1')
}
);
startQueue();
} catch (error) {
unlockTask(userId);
}
}

View File

@@ -4,13 +4,14 @@ import { connectToDatabase } from '@/service/mongo';
import type { CreateQuestionGuideParams } from '@/global/core/ai/api.d';
import { pushQuestionGuideBill } from '@/service/support/wallet/bill/push';
import { createQuestionGuide } from '@fastgpt/service/core/ai/functions/createQuestionGuide';
import { authCertAndShareId } from '@fastgpt/service/support/permission/auth/common';
import { authCertOrShareId } from '@fastgpt/service/support/permission/auth/common';
export default async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
try {
await connectToDatabase();
const { messages, shareId } = req.body as CreateQuestionGuideParams;
const { tmbId, teamId } = await authCertAndShareId({
const { tmbId, teamId } = await authCertOrShareId({
req,
authToken: true,
shareId

View File

@@ -5,6 +5,7 @@ import { MongoChat } from '@fastgpt/service/core/chat/chatSchema';
import { MongoApp } from '@fastgpt/service/core/app/schema';
import { MongoOutLink } from '@fastgpt/service/support/outLink/schema';
import { authApp } from '@fastgpt/service/support/permission/auth/app';
import { MongoChatItem } from '@fastgpt/service/core/chat/chatItemSchema';
/* 获取我的模型 */
export default async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
@@ -20,6 +21,9 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse<
await authApp({ req, authToken: true, appId, per: 'owner' });
// 删除对应的聊天
await MongoChatItem.deleteMany({
appId
});
await MongoChat.deleteMany({
appId
});

View File

@@ -381,7 +381,7 @@ function datasetTemplate({
key: 'similarity',
value: 0.4,
type: FlowNodeInputTypeEnum.slider,
label: '相度',
label: '相度',
connected: true
},
{

View File

@@ -289,7 +289,7 @@ function datasetTemplate(formData: AppSimpleEditFormType): ModuleItemType[] {
key: 'similarity',
value: formData.dataset.similarity,
type: FlowNodeInputTypeEnum.slider,
label: '相度',
label: '相度',
connected: false
},
{

View File

@@ -4,6 +4,9 @@ import { connectToDatabase } from '@/service/mongo';
import { MongoApp } from '@fastgpt/service/core/app/schema';
import type { AppUpdateParams } from '@fastgpt/global/core/app/api';
import { authApp } from '@fastgpt/service/support/permission/auth/app';
import { FlowNodeTypeEnum } from '@fastgpt/global/core/module/node/constant';
import { ModuleInputKeyEnum } from '@fastgpt/global/core/module/constants';
import { getChatModel } from '@/service/core/ai/model';
/* 获取我的模型 */
export default async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
@@ -20,6 +23,36 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse<
// 凭证校验
await authApp({ req, authToken: true, appId, per: permission ? 'owner' : 'w' });
// check modules
// 1. dataset search limit, less than model quoteMaxToken
if (modules) {
let maxTokens = 3000;
modules.forEach((item) => {
if (item.flowType === FlowNodeTypeEnum.chatNode) {
const model =
item.inputs.find((item) => item.key === ModuleInputKeyEnum.aiModel)?.value || '';
const chatModel = getChatModel(model);
const quoteMaxToken = chatModel.quoteMaxToken || 3000;
maxTokens = Math.max(maxTokens, quoteMaxToken);
}
});
modules.forEach((item) => {
if (item.flowType === FlowNodeTypeEnum.datasetSearchNode) {
item.inputs.forEach((input) => {
if (input.key === ModuleInputKeyEnum.datasetLimit) {
const val = input.value as number;
if (val > maxTokens) {
input.value = maxTokens;
}
}
});
}
});
}
// 更新模型
await MongoApp.updateOne(
{

View File

@@ -8,8 +8,9 @@ import { pushChatBill } from '@/service/support/wallet/bill/push';
import { BillSourceEnum } from '@fastgpt/global/support/wallet/bill/constants';
import type { ChatItemType } from '@fastgpt/global/core/chat/type';
import { authApp } from '@fastgpt/service/support/permission/auth/app';
import { authUser } from '@/service/support/permission/auth/user';
import { dispatchModules } from '@/service/moduleDispatch';
import { authCert } from '@fastgpt/service/support/permission/auth/common';
import { getUserAndAuthBalance } from '@fastgpt/service/support/user/controller';
export type Props = {
history: ChatItemType[];
@@ -40,15 +41,20 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse)
}
/* user auth */
const [{ teamId, tmbId }, { user }] = await Promise.all([
const [_, { teamId, tmbId }] = await Promise.all([
authApp({ req, authToken: true, appId, per: 'r' }),
authUser({
authCert({
req,
authToken: true,
minBalance: 0
authToken: true
})
]);
// auth balance
const user = await getUserAndAuthBalance({
tmbId,
minBalance: 0
});
/* start process */
const { responseData } = await dispatchModules({
res,

View File

@@ -0,0 +1,58 @@
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo';
import { authCert } from '@fastgpt/service/support/permission/auth/common';
import { MongoChat } from '@fastgpt/service/core/chat/chatSchema';
import { MongoChatItem } from '@fastgpt/service/core/chat/chatItemSchema';
import { ClearHistoriesProps } from '@/global/core/chat/api';
import { authOutLink } from '@/service/support/permission/auth/outLink';
import { ChatSourceEnum } from '@fastgpt/global/core/chat/constants';
/* clear chat history */
export default async function handler(req: NextApiRequest, res: NextApiResponse) {
try {
await connectToDatabase();
const { appId, shareId, outLinkUid } = req.query as ClearHistoriesProps;
const match = await (async () => {
if (shareId && outLinkUid) {
const { uid } = await authOutLink({ shareId, outLinkUid });
return {
shareId,
outLinkUid: uid
};
}
if (appId) {
const { tmbId } = await authCert({ req, authToken: true });
return {
appId,
tmbId,
source: ChatSourceEnum.online
};
}
return Promise.reject('Param are error');
})();
console.log(match);
// find chatIds
const list = await MongoChat.find(match, 'chatId').lean();
const idList = list.map((item) => item.chatId);
await MongoChatItem.deleteMany({
chatId: { $in: idList }
});
await MongoChat.deleteMany({
chatId: { $in: idList }
});
jsonRes(res);
} catch (err) {
jsonRes(res, {
code: 500,
error: err
});
}
}

View File

@@ -0,0 +1,38 @@
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo';
import { MongoChat } from '@fastgpt/service/core/chat/chatSchema';
import { MongoChatItem } from '@fastgpt/service/core/chat/chatItemSchema';
import { DelHistoryProps } from '@/global/core/chat/api';
import { autChatCrud } from '@/service/support/permission/auth/chat';
/* clear chat history */
export default async function handler(req: NextApiRequest, res: NextApiResponse) {
try {
await connectToDatabase();
const { chatId, shareId, outLinkUid } = req.query as DelHistoryProps;
await autChatCrud({
req,
authToken: true,
chatId,
shareId,
outLinkUid,
per: 'w'
});
await MongoChatItem.deleteMany({
chatId
});
await MongoChat.findOneAndRemove({
chatId
});
jsonRes(res);
} catch (err) {
jsonRes(res, {
code: 500,
error: err
});
}
}

View File

@@ -1,54 +0,0 @@
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo';
import { authCert } from '@fastgpt/service/support/permission/auth/common';
import { MongoChat } from '@fastgpt/service/core/chat/chatSchema';
import { MongoChatItem } from '@fastgpt/service/core/chat/chatItemSchema';
import { ChatSourceEnum } from '@fastgpt/global/core/chat/constants';
type Props = {
chatId?: string;
appId?: string;
};
/* clear chat history */
export default async function handler(req: NextApiRequest, res: NextApiResponse) {
try {
await connectToDatabase();
const { chatId, appId } = req.query as Props;
const { tmbId } = await authCert({ req, authToken: true });
if (chatId) {
await MongoChatItem.deleteMany({
chatId,
tmbId
});
await MongoChat.findOneAndRemove({
chatId,
tmbId
});
}
if (appId) {
const chats = await MongoChat.find({
appId,
tmbId,
source: ChatSourceEnum.online
}).select('_id');
const chatIds = chats.map((chat) => chat._id);
await MongoChatItem.deleteMany({
chatId: { $in: chatIds }
});
await MongoChat.deleteMany({
_id: { $in: chatIds }
});
}
jsonRes(res);
} catch (err) {
jsonRes(res, {
code: 500,
error: err
});
}
}

View File

@@ -2,7 +2,7 @@ import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo';
import { authCert } from '@fastgpt/service/support/permission/auth/common';
import type { AdminUpdateFeedbackParams } from '@fastgpt/global/core/chat/api.d';
import type { AdminUpdateFeedbackParams } from '@/global/core/chat/api.d';
import { MongoChatItem } from '@fastgpt/service/core/chat/chatItemSchema';
/* 初始化我的聊天框,需要身份验证 */

View File

@@ -0,0 +1,62 @@
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo';
import { MongoChat } from '@fastgpt/service/core/chat/chatSchema';
import type { ChatHistoryItemType } from '@fastgpt/global/core/chat/type.d';
import { ChatSourceEnum } from '@fastgpt/global/core/chat/constants';
import { getHistoriesProps } from '@/global/core/chat/api';
import { authOutLink } from '@/service/support/permission/auth/outLink';
import { authCert } from '@fastgpt/service/support/permission/auth/common';
export default async function handler(req: NextApiRequest, res: NextApiResponse) {
try {
await connectToDatabase();
const { appId, shareId, outLinkUid } = req.body as getHistoriesProps;
const limit = shareId && outLinkUid ? 20 : 30;
const match = await (async () => {
if (shareId && outLinkUid) {
const { uid } = await authOutLink({ shareId, outLinkUid });
return {
shareId,
outLinkUid: uid,
source: ChatSourceEnum.share,
updateTime: {
$gte: new Date(new Date().setDate(new Date().getDate() - 30))
}
};
}
if (appId) {
const { tmbId } = await authCert({ req, authToken: true });
return {
appId,
tmbId,
source: ChatSourceEnum.online
};
}
return Promise.reject('Params are error');
})();
const data = await MongoChat.find(match, 'chatId title top customTitle appId updateTime')
.sort({ top: -1, updateTime: -1 })
.limit(limit);
jsonRes<ChatHistoryItemType[]>(res, {
data: data.map((item) => ({
chatId: item.chatId,
updateTime: item.updateTime,
appId: item.appId,
customTitle: item.customTitle,
title: item.title,
top: item.top
}))
});
} catch (err) {
jsonRes(res, {
code: 500,
error: err
});
}
}

View File

@@ -1,24 +1,20 @@
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo';
import { MongoChatItem } from '@fastgpt/service/core/chat/chatItemSchema';
import type { InitChatResponse } from '@fastgpt/global/core/chat/api.d';
import type { ChatItemType } from '@fastgpt/global/core/chat/type.d';
import { authApp } from '@fastgpt/service/support/permission/auth/app';
import { getGuideModule } from '@fastgpt/global/core/module/utils';
import { getChatModelNameListByModules } from '@/service/core/app/module';
import { authChat } from '@fastgpt/service/support/permission/auth/chat';
import { ModuleOutputKeyEnum } from '@fastgpt/global/core/module/constants';
import type { InitChatProps, InitChatResponse } from '@/global/core/chat/api.d';
import { MongoChat } from '@fastgpt/service/core/chat/chatSchema';
import { getChatItems } from '@fastgpt/service/core/chat/controller';
import { ChatErrEnum } from '@fastgpt/global/common/error/code/chat';
/* 初始化我的聊天框,需要身份验证 */
export default async function handler(req: NextApiRequest, res: NextApiResponse) {
try {
await connectToDatabase();
let { appId, chatId } = req.query as {
appId: string;
chatId: '' | string;
};
let { appId, chatId } = req.query as InitChatProps;
if (!appId) {
return jsonRes(res, {
@@ -27,57 +23,44 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse)
});
}
// 校验使用权限
const [{ app }, autChatResult] = await Promise.all([
// auth app permission
const [{ app, tmbId }, chat] = await Promise.all([
authApp({
req,
authToken: true,
appId,
per: 'r'
}),
chatId
? authChat({
req,
authToken: true,
chatId,
per: 'r'
})
: undefined
chatId ? MongoChat.findOne({ chatId }) : undefined
]);
// get app and history
const { history = [] }: { history?: ChatItemType[] } = await (async () => {
if (chatId) {
// auth chatId
const history = await MongoChatItem.find(
{
chatId
},
`dataId obj value adminFeedback userFeedback ${ModuleOutputKeyEnum.responseData}`
)
.sort({ _id: -1 })
.limit(30);
// auth chat permission
if (!app.canWrite && String(tmbId) !== String(chat?.tmbId)) {
throw new Error(ChatErrEnum.unAuthChat);
}
history.reverse();
return { history };
}
return {};
})();
// get app and history
const { history } = await getChatItems({
chatId,
limit: 30,
field: `dataId obj value adminFeedback userFeedback ${ModuleOutputKeyEnum.responseData}`
});
jsonRes<InitChatResponse>(res, {
data: {
chatId,
appId,
title: chat?.title || '新对话',
userAvatar: undefined,
variables: chat?.variables || {},
history,
app: {
userGuideModule: getGuideModule(app.modules),
chatModels: getChatModelNameListByModules(app.modules),
name: app.name,
avatar: app.avatar,
intro: app.intro
},
title: autChatResult?.chat?.title || '新对话',
variables: autChatResult?.chat?.variables || {},
history
}
}
});
} catch (err) {

View File

@@ -0,0 +1,42 @@
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo';
import { MongoChat } from '@fastgpt/service/core/chat/chatSchema';
import { ChatSourceEnum } from '@fastgpt/global/core/chat/constants';
/* clear chat history */
export default async function handler(req: NextApiRequest, res: NextApiResponse) {
try {
await connectToDatabase();
const { outLinkUid, chatIds } = req.body as {
outLinkUid: string;
chatIds: string[];
};
if (!outLinkUid) {
throw new Error('shareId or outLinkUid is required');
}
const sliceIds = chatIds.slice(0, 50);
await MongoChat.updateMany(
{
chatId: { $in: sliceIds },
source: ChatSourceEnum.share,
outLinkUid: { $exists: false }
},
{
$set: {
outLinkUid
}
}
);
jsonRes(res);
} catch (err) {
jsonRes(res, {
code: 500,
error: err
});
}
}

View File

@@ -2,14 +2,26 @@ import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo';
import { MongoChatItem } from '@fastgpt/service/core/chat/chatItemSchema';
import { authChat } from '@fastgpt/service/support/permission/auth/chat';
import { autChatCrud } from '@/service/support/permission/auth/chat';
import type { DeleteChatItemProps } from '@/global/core/chat/api.d';
export default async function handler(req: NextApiRequest, res: NextApiResponse) {
try {
await connectToDatabase();
const { chatId, contentId } = req.query as { chatId: string; contentId: string };
const { chatId, contentId, shareId, outLinkUid } = req.query as DeleteChatItemProps;
await authChat({ req, authToken: true, chatId, per: 'w' });
if (!contentId || !chatId) {
return jsonRes(res);
}
await autChatCrud({
req,
authToken: true,
chatId,
shareId,
outLinkUid,
per: 'w'
});
await MongoChatItem.deleteOne({
dataId: contentId,

View File

@@ -4,7 +4,7 @@ import { connectToDatabase } from '@/service/mongo';
import { GetChatSpeechProps } from '@/global/core/chat/api.d';
import { text2Speech } from '@fastgpt/service/core/ai/audio/speech';
import { pushAudioSpeechBill } from '@/service/support/wallet/bill/push';
import { authCertAndShareId } from '@fastgpt/service/support/permission/auth/common';
import { authCertOrShareId } from '@fastgpt/service/support/permission/auth/common';
import { authType2BillSource } from '@/service/support/wallet/bill/utils';
import { getAudioSpeechModel } from '@/service/core/ai/model';
import { MongoTTSBuffer } from '@fastgpt/service/common/buffer/tts/schema';
@@ -25,7 +25,7 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse)
throw new Error('model or voice not found');
}
const { teamId, tmbId, authType } = await authCertAndShareId({ req, authToken: true, shareId });
const { teamId, tmbId, authType } = await authCertOrShareId({ req, authToken: true, shareId });
const ttsModel = getAudioSpeechModel(ttsConfig.model);
const voiceData = ttsModel.voices?.find((item) => item.value === ttsConfig.voice);

View File

@@ -1,43 +0,0 @@
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo';
import { MongoChat } from '@fastgpt/service/core/chat/chatSchema';
import type { ChatHistoryItemType } from '@fastgpt/global/core/chat/type.d';
import { ChatSourceEnum } from '@fastgpt/global/core/chat/constants';
import { authApp } from '@fastgpt/service/support/permission/auth/app';
/* 获取历史记录 */
export default async function handler(req: NextApiRequest, res: NextApiResponse) {
try {
await connectToDatabase();
const { appId } = req.body as { appId: string };
const { tmbId } = await authApp({ req, authToken: true, appId, per: 'r' });
const data = await MongoChat.find(
{
appId,
tmbId,
source: ChatSourceEnum.online
},
'chatId title top customTitle appId updateTime'
)
.sort({ top: -1, updateTime: -1 })
.limit(20);
jsonRes<ChatHistoryItemType[]>(res, {
data: data.map((item) => ({
chatId: item.chatId,
updateTime: item.updateTime,
appId: item.appId,
customTitle: item.customTitle,
title: item.title,
top: item.top
}))
});
} catch (err) {
jsonRes(res, {
code: 500,
error: err
});
}
}

View File

@@ -0,0 +1,85 @@
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo';
import type { InitChatResponse, InitOutLinkChatProps } from '@/global/core/chat/api.d';
import { getGuideModule } from '@fastgpt/global/core/module/utils';
import { getChatModelNameListByModules } from '@/service/core/app/module';
import { ModuleOutputKeyEnum } from '@fastgpt/global/core/module/constants';
import { getChatItems } from '@fastgpt/service/core/chat/controller';
import { MongoTeamMember } from '@fastgpt/service/support/user/team/teamMemberSchema';
import { authOutLink } from '@/service/support/permission/auth/outLink';
import { MongoApp } from '@fastgpt/service/core/app/schema';
import { selectShareResponse } from '@/utils/service/core/chat';
import { AppErrEnum } from '@fastgpt/global/common/error/code/app';
import { MongoChat } from '@fastgpt/service/core/chat/chatSchema';
import { ChatErrEnum } from '@fastgpt/global/common/error/code/chat';
export default async function handler(req: NextApiRequest, res: NextApiResponse) {
try {
await connectToDatabase();
let { chatId, shareId, outLinkUid } = req.query as InitOutLinkChatProps;
// auth link permission
const { shareChat, uid, appId } = await authOutLink({ shareId, outLinkUid });
// auth app permission
const [tmb, chat, app] = await Promise.all([
MongoTeamMember.findById(shareChat.tmbId, '_id userId').populate('userId', 'avatar').lean(),
MongoChat.findOne({ chatId, shareId }).lean(),
MongoApp.findById(appId).lean()
]);
if (!app) {
throw new Error(AppErrEnum.unExist);
}
// auth chat permission
if (chat && chat.outLinkUid !== uid) {
throw new Error(ChatErrEnum.unAuthChat);
}
const { history } = await getChatItems({
chatId,
limit: 30,
field: `dataId obj value userFeedback ${
shareChat.responseDetail ? `adminFeedback ${ModuleOutputKeyEnum.responseData}` : ''
} `
});
// pick share response field
history.forEach((item) => {
item.responseData = selectShareResponse({ responseData: item.responseData });
});
jsonRes<InitChatResponse>(res, {
data: {
chatId,
appId: app._id,
title: chat?.title || '新对话',
//@ts-ignore
userAvatar: tmb?.userId?.avatar,
variables: chat?.variables || {},
history,
app: {
userGuideModule: getGuideModule(app.modules),
chatModels: getChatModelNameListByModules(app.modules),
name: app.name,
avatar: app.avatar,
intro: app.intro
}
}
});
} catch (err) {
jsonRes(res, {
code: 500,
error: err
});
}
}
export const config = {
api: {
responseLimit: '10mb'
}
};

View File

@@ -1,17 +1,24 @@
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo';
import { UpdateHistoryProps } from '@fastgpt/global/core/chat/api.d';
import { UpdateHistoryProps } from '@/global/core/chat/api.d';
import { MongoChat } from '@fastgpt/service/core/chat/chatSchema';
import { authChat } from '@fastgpt/service/support/permission/auth/chat';
import { autChatCrud } from '@/service/support/permission/auth/chat';
/* 更新聊天标题 */
/* update chat top, custom title */
export default async function handler(req: NextApiRequest, res: NextApiResponse) {
try {
await connectToDatabase();
const { chatId, customTitle, top } = req.body as UpdateHistoryProps;
const { chatId, shareId, outLinkUid, customTitle, top } = req.body as UpdateHistoryProps;
await authChat({ req, authToken: true, chatId });
await autChatCrud({
req,
authToken: true,
chatId,
shareId,
outLinkUid,
per: 'w'
});
await MongoChat.findOneAndUpdate(
{ chatId },

View File

@@ -24,13 +24,13 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse<
});
// find all delete id
const collections = await findCollectionAndChild(collectionId, '_id metadata');
const collections = await findCollectionAndChild(collectionId, '_id fileId');
const delIdList = collections.map((item) => item._id);
// delete
await delCollectionRelevantData({
collectionIds: delIdList,
fileIds: collections.map((item) => item.metadata?.fileId).filter(Boolean)
fileIds: collections.map((item) => item?.fileId || '').filter(Boolean)
});
// delete collection

View File

@@ -4,7 +4,6 @@ import { connectToDatabase } from '@/service/mongo';
import { authDatasetCollection } from '@fastgpt/service/support/permission/auth/dataset';
import { loadingOneChunkCollection } from '@fastgpt/service/core/dataset/collection/utils';
import { delCollectionRelevantData } from '@fastgpt/service/core/dataset/data/controller';
import { createOneCollection } from '@fastgpt/service/core/dataset/collection/controller';
import { MongoDatasetCollection } from '@fastgpt/service/core/dataset/collection/schema';
import { DatasetCollectionTypeEnum } from '@fastgpt/global/core/dataset/constant';
import { DatasetErrEnum } from '@fastgpt/global/common/error/code/dataset';

View File

@@ -8,15 +8,18 @@ import { authDataset } from '@fastgpt/service/support/permission/auth/dataset';
export default async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
try {
await connectToDatabase();
const { id, parentId, name, avatar, tags, permission, agentModel, websiteConfig, status } =
const { id, parentId, name, avatar, intro, permission, agentModel, websiteConfig, status } =
req.body as DatasetUpdateBody;
if (!id) {
throw new Error('缺少参数');
}
// 凭证校验
await authDataset({ req, authToken: true, datasetId: id, per: 'owner' });
if (permission) {
await authDataset({ req, authToken: true, datasetId: id, per: 'owner' });
} else {
await authDataset({ req, authToken: true, datasetId: id, per: 'w' });
}
await MongoDataset.findOneAndUpdate(
{
@@ -26,11 +29,11 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse<
...(parentId !== undefined && { parentId: parentId || null }),
...(name && { name }),
...(avatar && { avatar }),
...(tags && { tags }),
...(permission && { permission }),
...(agentModel && { agentModel: agentModel.model }),
...(websiteConfig && { websiteConfig }),
...(status && { status })
...(status && { status }),
...(intro && { intro })
}
);

Some files were not shown because too many files have changed in this diff Show More