🖼️ fix: Correct ToolMessage Response Format for Agent-Mode Image Tools (#12310)

* fix: Set response format for agent tools in DALLE3, FluxAPI, and StableDiffusion classes

- Added logic to set `responseFormat` to 'content_and_artifact' when `isAgent` is true in DALLE3.js, FluxAPI.js, and StableDiffusion.js.

* test: Add regression tests for image tool agent mode in imageTools-agent.spec.js

- Introduced a new test suite for DALLE3, FluxAPI, and StableDiffusion classes to verify that the invoke() method returns a ToolMessage with base64 in artifact.content, ensuring it is not serialized into content.
- Validated that responseFormat is set to 'content_and_artifact' when isAgent is true, and confirmed the correct handling of base64 data in the response.

* fix: handle agent error paths and generateFinetunedImage in image tools

- StableDiffusion._call() was returning a raw string on API error, bypassing
  returnValue() and breaking the content_and_artifact contract when isAgent is true
- FluxAPI.generateFinetunedImage() had no isAgent branch; it would call
  processFileURL (unset in agent context) instead of fetching and returning
  the base64 image as an artifact tuple
- Add JSDoc to all three responseFormat assignments clarifying why LangChain
  requires this property for correct ToolMessage construction

* test: expand image tool agent mode regression suite

- Add env var save/restore in beforeEach/afterEach to prevent test pollution
- Add error path tests for all three tools verifying ToolMessage content and
  artifact are correctly populated when the upstream API fails
- Add generate_finetuned action test for FluxAPI covering the new agent branch
  in generateFinetunedImage

* chore: fix lint errors in FluxAPI and imageTools-agent spec

* chore: fix import ordering in imageTools-agent spec
This commit is contained in:
Danny Avila 2026-03-19 15:33:46 -04:00 committed by GitHub
parent 93952f06b4
commit a88bfae4dd
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
4 changed files with 338 additions and 8 deletions

View file

@ -113,6 +113,10 @@ class FluxAPI extends Tool {
/** @type {boolean} **/
this.isAgent = fields.isAgent;
if (this.isAgent) {
/** Ensures LangChain maps [content, artifact] tuple to ToolMessage fields instead of serializing it into content. */
this.responseFormat = 'content_and_artifact';
}
this.returnMetadata = fields.returnMetadata ?? false;
if (fields.processFileURL) {
@ -524,10 +528,40 @@ class FluxAPI extends Tool {
return this.returnValue('No image data received from Flux API.');
}
// Try saving the image locally
const imageUrl = resultData.sample;
const imageName = `img-${uuidv4()}.png`;
if (this.isAgent) {
try {
const fetchOptions = {};
if (process.env.PROXY) {
fetchOptions.agent = new HttpsProxyAgent(process.env.PROXY);
}
const imageResponse = await fetch(imageUrl, fetchOptions);
const arrayBuffer = await imageResponse.arrayBuffer();
const base64 = Buffer.from(arrayBuffer).toString('base64');
const content = [
{
type: ContentTypes.IMAGE_URL,
image_url: {
url: `data:image/png;base64,${base64}`,
},
},
];
const response = [
{
type: ContentTypes.TEXT,
text: displayMessage,
},
];
return [response, { content }];
} catch (error) {
logger.error('[FluxAPI] Error processing finetuned image for agent:', error);
return this.returnValue(`Failed to process the finetuned image. ${error.message}`);
}
}
try {
logger.debug('[FluxAPI] Saving finetuned image:', imageUrl);
const result = await this.processFileURL({
@ -541,12 +575,6 @@ class FluxAPI extends Tool {
logger.debug('[FluxAPI] Finetuned image saved to path:', result.filepath);
// Calculate cost based on endpoint
const endpointKey = endpoint.includes('ultra')
? 'FLUX_PRO_1_1_ULTRA_FINETUNED'
: 'FLUX_PRO_FINETUNED';
const cost = FluxAPI.PRICING[endpointKey] || 0;
// Return the result based on returnMetadata flag
this.result = this.returnMetadata ? result : this.wrapInMarkdown(result.filepath);
return this.returnValue(this.result);
} catch (error) {