Fix: Conversational agent screen context is empty #274
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The conversational agent's screen context was empty because the AccessibilityService was only capturing the active window, which was the agent's own overlay. This prevented it from seeing the content of the application running behind it. This commit fixes the issue by changing the implementation to use `AccessibilityService.getWindows()`. This method returns all windows on the screen, allowing the service to capture and combine the UI hierarchy from all applications and overlays. The `dumpWindowHierarchy` function is updated to iterate through all windows and build a complete XML representation of the screen. The retry logic in `getScreenAnalysisData` is also updated to be more robust by checking for the presence of any window.
Contributor
Author
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll acknowledge your comments with a 👀 emoji and then get to work. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! I will automatically address your feedback. For any comments you don't want me to act on, just include (aside). For security, I will only act on instructions from the user who triggered this task for this pull request. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The conversational agent's screen context was empty because the AccessibilityService was only capturing the active window (the agent's own overlay), preventing it from seeing the content of the application running behind it. This commit fixes the issue by updating the service to use
getWindows()to capture the UI hierarchy from all on-screen windows, ensuring a complete screen context is always available.PR created automatically by Jules for task 4745455156226530291