Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
D
dify
Project
Project
Details
Activity
Releases
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Board
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
ai-tech
dify
Commits
4d502ea4
Unverified
Commit
4d502ea4
authored
Jan 03, 2024
by
crazywoola
Committed by
GitHub
Jan 03, 2024
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
fix: openai embedding list out of bound (#1879)
parent
66b35888
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
4 additions
and
4 deletions
+4
-4
text_embedding.py
...e/model_providers/openai/text_embedding/text_embedding.py
+4
-4
No files found.
api/core/model_runtime/model_providers/openai/text_embedding/text_embedding.py
View file @
4d502ea4
...
@@ -68,7 +68,7 @@ class OpenAITextEmbeddingModel(_CommonOpenAI, TextEmbeddingModel):
...
@@ -68,7 +68,7 @@ class OpenAITextEmbeddingModel(_CommonOpenAI, TextEmbeddingModel):
for
i
in
_iter
:
for
i
in
_iter
:
# call embedding model
# call embedding model
embeddings
,
embedding_used_tokens
=
self
.
_embedding_invoke
(
embeddings
_batch
,
embedding_used_tokens
=
self
.
_embedding_invoke
(
model
=
model
,
model
=
model
,
client
=
client
,
client
=
client
,
texts
=
tokens
[
i
:
i
+
max_chunks
],
texts
=
tokens
[
i
:
i
+
max_chunks
],
...
@@ -76,7 +76,7 @@ class OpenAITextEmbeddingModel(_CommonOpenAI, TextEmbeddingModel):
...
@@ -76,7 +76,7 @@ class OpenAITextEmbeddingModel(_CommonOpenAI, TextEmbeddingModel):
)
)
used_tokens
+=
embedding_used_tokens
used_tokens
+=
embedding_used_tokens
batched_embeddings
+=
[
data
for
data
in
embeddings
]
batched_embeddings
+=
embeddings_batch
results
:
list
[
list
[
list
[
float
]]]
=
[[]
for
_
in
range
(
len
(
texts
))]
results
:
list
[
list
[
list
[
float
]]]
=
[[]
for
_
in
range
(
len
(
texts
))]
num_tokens_in_batch
:
list
[
list
[
int
]]
=
[[]
for
_
in
range
(
len
(
texts
))]
num_tokens_in_batch
:
list
[
list
[
int
]]
=
[[]
for
_
in
range
(
len
(
texts
))]
...
@@ -87,7 +87,7 @@ class OpenAITextEmbeddingModel(_CommonOpenAI, TextEmbeddingModel):
...
@@ -87,7 +87,7 @@ class OpenAITextEmbeddingModel(_CommonOpenAI, TextEmbeddingModel):
for
i
in
range
(
len
(
texts
)):
for
i
in
range
(
len
(
texts
)):
_result
=
results
[
i
]
_result
=
results
[
i
]
if
len
(
_result
)
==
0
:
if
len
(
_result
)
==
0
:
embeddings
,
embedding_used_tokens
=
self
.
_embedding_invoke
(
embeddings
_batch
,
embedding_used_tokens
=
self
.
_embedding_invoke
(
model
=
model
,
model
=
model
,
client
=
client
,
client
=
client
,
texts
=
[
""
],
texts
=
[
""
],
...
@@ -95,7 +95,7 @@ class OpenAITextEmbeddingModel(_CommonOpenAI, TextEmbeddingModel):
...
@@ -95,7 +95,7 @@ class OpenAITextEmbeddingModel(_CommonOpenAI, TextEmbeddingModel):
)
)
used_tokens
+=
embedding_used_tokens
used_tokens
+=
embedding_used_tokens
average
=
embeddings
[
0
]
average
=
embeddings
_batch
[
0
]
else
:
else
:
average
=
np
.
average
(
_result
,
axis
=
0
,
weights
=
num_tokens_in_batch
[
i
])
average
=
np
.
average
(
_result
,
axis
=
0
,
weights
=
num_tokens_in_batch
[
i
])
embeddings
[
i
]
=
(
average
/
np
.
linalg
.
norm
(
average
))
.
tolist
()
embeddings
[
i
]
=
(
average
/
np
.
linalg
.
norm
(
average
))
.
tolist
()
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment