How is your experience with the symbol-based tools?

[nikita2206]

One of the reasons why I want to start using Serena with Java codebases, is that it may aid LLMs to do large scale refactoring almost as efficiently as IntelliJ can do it with me behind the wheel.

But this can only work if the underlying tools work correctly, like find_referencing_symbols.
I noticed that they seem to be unreliable, not finding the refrencing symbols often. However, I am not sure how I can debug them.

Is there anything I can do to debug those tools, and understand if maybe the LLM is passing incorrect arguments or if it is Eclipse LSP not fully loaded?

Additionally wanted to ask, I just noticed jet_brains_* tools - are those implemented already? I would love to learn how these work, any code pointers would be much appreciated 🙏

[nikita2206]

Ah I think you are making a custom IntelliJ plugin that exposes IntelliJ's features via an HTTP API and then just connecting to that API from Serena, that's clever! I wasn't able to find your plugin. Just found one abandoned plugin that implements some of this. This coupled with the ability to run IJ headlessly can be quite powerful: https://github.com/SuduIDE/ideals

I would then go as far as defining a Claude Code subagent for Java refactoring, that would be exposed to all the fancy tools like Call Hierarchy view, Data Flow views, Structural search/replace, renaming of symbols and moving symbols, etc.

[opcode81]

We will release the plugin soon. Stay tuned!

[opcode81]

Is there anything I can do to debug those tools, and understand if maybe the LLM is passing incorrect arguments or if it is Eclipse LSP not fully loaded?

Make sure you have either the dashboard or GUI window enabled and inspect the logs live.
You will see all the parameters being passed and the results being returned; and you should also see any errors pertaining to EclipseJDTLS' calls.
If the LS is not returning the right result but does not produce an error, then there is little you can do (other than fix the LS itself or replace it with a better one for Java - if there is one).
It is also conceivable that some issues are due to misconfiguration of EclipseJDTLS. Perhaps it needs to be configured differently to work for your project (e.g. different Java versions, etc). If that's the case, then we might have to add configuration options.

[opcode81]

We can add something like a counter for identifying them, or something signature based

I added the counter concept in parallel, and I think it should be less problematic than the signature.

Would be a rather big change, so we have to think it through and plan it well

Actually, I don't think this would be such a big change.

[nikita2206]

Ah yeah, after playing with this a bit, I think it was due to the fact that the FindReferencingSymbolsTool requires the full signature. Does this tool give any feedback to the LLM if the symbol itself not found? (the one, references of which we are searching for)

When using FindSymbolTool, which allows the substring_matching = True, this was a lot easier for me to find stuff.

What might work for LLMs better, is if you get a fuzzy query, like for example
FindReferencingSymbolsTool(name_path="SomeService/someMethod", relative_path="{full path}"), and there are actually multiple overloads of someMethod, then we would find all references of all overloads of this method, then group them by overload, and present them all back to the LLM:

## SomeService/someMethod(String, Long) is called from

src/main/java/com/organization/controller/SomeController/someEndpoint at line 123
src/main/java/com/organization/consumer/FooMessageHandler/handle at line 123

## SomeService/someMethod(String, Long, Context) is called from

another/path/entirely/BarClass/barMethod at line 324

Same can be applied to the relative_path parameter - if we make it optional, and instead use this kind of output format, this might make LLMs less likely to be confused when calling this tool.

[opcode81]

What might work for LLMs better, is if you get a fuzzy query, like for example
FindReferencingSymbolsTool(name_path="SomeService/someMethod", relative_path="{full path}"), and there are actually multiple overloads of someMethod

The right way to proceed here is not to be "fuzzy" about which symbol/method we are looking for references to but to allow overloaded methods to be correctly specified in a correspondingly specialised name path.

Same can be applied to the relative_path parameter - if we make it optiona

It is not optional, because the identifier of the symbol for which we seek references is given by <file path, name path within file>.
As I stated above, the symbol needs to be uniquely identified; fuzziness is not the way to go.
If you (the LLM) don't know which symbol you actually want, you need to use find_symbol first.

[nikita2206]

I am just suspecting that the LLM might not fare well with such strict requirements. I will try to use this more, now that I understand how it works a bit better, and will report back.

[opcode81]

The LLM is very clearly instructed to not perform symbol operations without first identifying the symbols using find_symbol - and models like Claude typically have no problems following these instructions.

[MischaPanch]

Common reasons for reference finding not working include:

1. LS needs more time to Initialize to get Cross-File references (solved with sleeping in csharp, typescript and others)
2. Selection range is delivered incorrectly by the LS (happens currently for clang, not yet fixed, used to happen for go due to incorrect parametrization of the LS)
3. We have custom and hacky code for resolving the parent symbol (the LS only gives the referencing position, not a symbol), that may fail

All of these issues will be resolved with the extension. For debugging, pls have a look at scripts/demo_run_tools.py

We do plan to add more tools to the jetbrains based approach, just like you said above (maybe not in the first release though).

[nikita2206]

Thanks! Will take a look at the demo_run_tools - looks like just what I need.

[nikita2206]

I've also opened a PR to enhance this script a little bit. Let me know if you'd rather keep it simple though 🙂