🖼️ fix: Correct ToolMessage Response Format for Agent-Mode Image Tools (#12310)

* fix: Set response format for agent tools in DALLE3, FluxAPI, and StableDiffusion classes

- Added logic to set `responseFormat` to 'content_and_artifact' when `isAgent` is true in DALLE3.js, FluxAPI.js, and StableDiffusion.js.

* test: Add regression tests for image tool agent mode in imageTools-agent.spec.js

- Introduced a new test suite for DALLE3, FluxAPI, and StableDiffusion classes to verify that the invoke() method returns a ToolMessage with base64 in artifact.content, ensuring it is not serialized into content.
- Validated that responseFormat is set to 'content_and_artifact' when isAgent is true, and confirmed the correct handling of base64 data in the response.

* fix: handle agent error paths and generateFinetunedImage in image tools

- StableDiffusion._call() was returning a raw string on API error, bypassing
  returnValue() and breaking the content_and_artifact contract when isAgent is true
- FluxAPI.generateFinetunedImage() had no isAgent branch; it would call
  processFileURL (unset in agent context) instead of fetching and returning
  the base64 image as an artifact tuple
- Add JSDoc to all three responseFormat assignments clarifying why LangChain
  requires this property for correct ToolMessage construction

* test: expand image tool agent mode regression suite

- Add env var save/restore in beforeEach/afterEach to prevent test pollution
- Add error path tests for all three tools verifying ToolMessage content and
  artifact are correctly populated when the upstream API fails
- Add generate_finetuned action test for FluxAPI covering the new agent branch
  in generateFinetunedImage

* chore: fix lint errors in FluxAPI and imageTools-agent spec

* chore: fix import ordering in imageTools-agent spec
This commit is contained in:
Danny Avila 2026-03-19 15:33:46 -04:00 committed by GitHub
parent 93952f06b4
commit a88bfae4dd
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
4 changed files with 338 additions and 8 deletions

View file

@ -43,6 +43,10 @@ class StableDiffusionAPI extends Tool {
this.returnMetadata = fields.returnMetadata ?? false;
/** @type {boolean} */
this.isAgent = fields.isAgent;
if (this.isAgent) {
/** Ensures LangChain maps [content, artifact] tuple to ToolMessage fields instead of serializing it into content. */
this.responseFormat = 'content_and_artifact';
}
if (fields.uploadImageBuffer) {
/** @type {uploadImageBuffer} Necessary for output to contain all image metadata. */
this.uploadImageBuffer = fields.uploadImageBuffer.bind(this);
@ -115,7 +119,7 @@ class StableDiffusionAPI extends Tool {
generationResponse = await axios.post(`${url}/sdapi/v1/txt2img`, payload);
} catch (error) {
logger.error('[StableDiffusion] Error while generating image:', error);
return 'Error making API request.';
return this.returnValue('Error making API request.');
}
const image = generationResponse.data.images[0];