KDevelop syntax parser is barely usable

[Aleksey Midenkov]

It takes about 30 minutes to reparse project (as large as MySQL
sources). More often than not I can't jump to a symbol because symbol
database is broken and is reparsing. And it breaks too much (several
times a day). Moreover often it works not very well: shows duplicate
entries, jumps to declaration but can't jump to definition, don't see
macro definitions, etc.

I was happy with how kdevelop 3 worked with ctags: it was not very
precise, but it worked always and it worked fast. Now kate has ctags
plugin. Is it possible to utilize this plugin in kdevelop as a helper
to syntax parser and make symbol jumping always available?

[René J.V. Bertin]

Post by Aleksey Midenkov
database is broken and is reparsing. And it breaks too much (several
times a day).

That shouldn't happen, except if you cause KDevelop to crash and then NOT tell it to preserve the cache when restarting. Or if you somehow open the same session in different versions (an older one from your distro and a more recent AppImage version for instance).

Post by Aleksey Midenkov
precise, but it worked always and it worked fast. Now kate has ctags
plugin. Is it possible to utilize this plugin in kdevelop as a helper
to syntax parser and make symbol jumping always available?

That should be possible, but you'd have to patch the ctags plugin json or .desktop file to declare itself as a kdevelop plugin too, I think (it should show something like `X-KDE-ServiceTypes=KTextEditor/Plugin,KDevelop/Plugin`).
Note however that you will probably NOT see the plugin's settings dialog in KDevelop so you'd have to configure it via Kate.

R.

[Steve the Fiddle]

Just some related feedback,

I work with the Audacity source code (0.6k lines of code
https://www.audacityteam.org/).
Kdevelop 5.2.4, Ubuntu 16.04.

Kdevelop rarely crashes for me, so the database does not need to reparse
from scratch very often, but I still often see KDevelop reparsing very
slowly due to code changes outside of KDevelop (such as when Git bisecting).

I have also been struggling with the problems of KDevelop not seeing
macros, templates, not finding definitions, not finding declarations ... In
short, parsing only works for about 50% of the Audacity code, whereas
KDevelop 3 gave about 90% coverage. The problem may be in part, due to
Audacity now making more use of C++11, and a little C++14, with more
templates and broader use of "modern" C++ idioms. Regardless of the
reasons, the problem for me remains that KDevelop 5 syntax parser does not
work at all well for Audacity source code.

The case for Audacity is complicated due to it using WxWidgets and
automake, but KDevelop 3 could mostly handle that reasonably well with a
bit of effort setting it up. I've been struggling with KDevelop 5 since it
was first released, and have managed to get it from "unusable" to "just
about usable".

Steve

Post by René J.V. Bertin

Post by Aleksey Midenkov
database is broken and is reparsing. And it breaks too much (several
times a day).

That shouldn't happen, except if you cause KDevelop to crash and then NOT
tell it to preserve the cache when restarting. Or if you somehow open the
same session in different versions (an older one from your distro and a
more recent AppImage version for instance).

Post by Aleksey Midenkov
precise, but it worked always and it worked fast. Now kate has ctags
plugin. Is it possible to utilize this plugin in kdevelop as a helper
to syntax parser and make symbol jumping always available?

That should be possible, but you'd have to patch the ctags plugin json or
.desktop file to declare itself as a kdevelop plugin too, I think (it
should show something like
`X-KDE-ServiceTypes=KTextEditor/Plugin,KDevelop/Plugin`).
Note however that you will probably NOT see the plugin's settings dialog
in KDevelop so you'd have to configure it via Kate.
R.

[Sven Brauch]

Hi,

Post by Steve the Fiddle
The problem may be in part, due to
Audacity now making more use of C++11, and a little C++14, with more
templates and broader use of "modern" C++ idioms.

FWIW, this is almost always a setup problem, usually include paths and/or
preprocessor defines not being set up correctly. For CMake, KDevelop is very
good at setting this up automatically, for automake much less so.

If include paths, compiler flags and definitions are set correctly, KDevelop's
parser will be very close to perfect, since it uses clang behind the scenes.
It doesn't matter how complex your template code is.

Performance is a different topic, but correctness is very good these days.

Best,
Sven

[Steve the Fiddle]

Post by Sven Brauch
Hi,

Post by Steve the Fiddle
The problem may be in part, due to
Audacity now making more use of C++11, and a little C++14, with more
templates and broader use of "modern" C++ idioms.

FWIW, this is almost always a setup problem, usually include paths and/or
preprocessor defines not being set up correctly. For CMake, KDevelop is very
good at setting this up automatically, for automake much less so.

It would be really really nice if there were better documentation for
setting up. I've been struggling with this ever since KDevelop 5 was first
released, and I've managed to improve parsing from "totally useless" to
"barely usable". Despite all of my efforts I have been unable to improve
further.

I did find some documentation re. setting up KDevelop for WxWidgets
applications, but it was many years out of date, so not really helpful.

If this is just a setup problem, and anyone can help me get KDevelop to to
parse the Audacity code reasonably well (
https://github.com/audacity/audacity), then I'd be happy to write up some
documentation for KDevelop (I write some of the documentation for Audacity).

Steve

Post by Sven Brauch
If include paths, compiler flags and definitions are set correctly, KDevelop's
parser will be very close to perfect, since it uses clang behind the scenes.
It doesn't matter how complex your template code is.
Performance is a different topic, but correctness is very good these days.
Best,
Sven

[Sven Brauch]

Hi,

Post by Steve the Fiddle
It would be really really nice if there were better documentation for
setting up. I've been struggling with this ever since KDevelop 5 was first
released, and I've managed to improve parsing from "totally useless" to
"barely usable". Despite all of my efforts I have been unable to improve
further.

There shouldn't be that much to it. Right-click your project, click "Open
configuration", then go to the "Language support" tab. There, configure the
correct includes paths, compiler defines and compiler options. That should be
it.

Best,
Sven

[René J.V. Bertin]

Post by Steve the Fiddle
If this is just a setup problem, and anyone can help me get KDevelop to to
parse the Audacity code reasonably well (
https://github.com/audacity/audacity), then I'd be happy to write up some
documentation for KDevelop (I write some of the documentation for Audacity).

It depends considerably on your local set-up, and your platform because of wxW . I have these defines configured:

WXUSINGDLL=1
_FILE_OFFSET_BITS=64
__WXGTK__=1

and the following custom include paths:

1=/opt/local/libexec/wxWidgets/3.0/include/wx-3.0
2=/opt/local/libexec/wxWidgets/3.0/lib/wx/include/gtk2-unicode-3.0

(all copied directly from my audacity/.kdev4/audacity.kdev4 settings file).

You can set these via the project settings dialog in KDevelop.

I'm guessing that you could get the wx-related defines and paths from wx-config, so if KDevelop supported (supports?) shell expressions in its settings files you could (almost) write a template .kdev4 file. Almost, because typically you have to know which wx-config script to use...

R.

[Steve the Fiddle]

Post by Steve the Fiddle

Post by Steve the Fiddle
If this is just a setup problem, and anyone can help me get KDevelop to to
parse the Audacity code reasonably well (
https://github.com/audacity/audacity), then I'd be happy to write up some
documentation for KDevelop (I write some of the documentation for

Audacity).
It depends considerably on your local set-up, and your platform because of
WXUSINGDLL=1
_FILE_OFFSET_BITS=64
__WXGTK__=1
1=/opt/local/libexec/wxWidgets/3.0/include/wx-3.0
2=/opt/local/libexec/wxWidgets/3.0/lib/wx/include/gtk2-unicode-3.0
(all copied directly from my audacity/.kdev4/audacity.kdev4 settings file).
You can set these via the project settings dialog in KDevelop.
I'm guessing that you could get the wx-related defines and paths from
wx-config, so if KDevelop supported (supports?) shell expressions in its
settings files you could (almost) write a template .kdev4 file. Almost,
because typically you have to know which wx-config script to use...
R.

Just a quick update,

Thanks to help from René, I've now got KDevelop working much better with
the Audacity code. It's still not perfect, but it is now much more usable.
The improvements came from modifying the "defines" and "custom includes
paths" (as René suggested). I'm still not sure that I really understand how
to use these properly, and the manual is rather sparse about this feature.
Setting this up was extremely time consuming because it takes so long to
re-parse the project before the effect of the changes can be seen.

I'll have another bash at configuring the project when I update Ubuntu, but
it's good enough for now :-)

Thanks for your support guys.

Steve

[Sven Brauch]

Hi,

Post by Steve the Fiddle
I'm still not sure that I really understand how
to use these properly, and the manual is rather sparse about this feature.

they are really only the compiler includes and defines (-DFOO and -Ifoo). What
you need to set here compeletely depends on your project, usually you want the
same here like your build system passes to your actual compiler. Maybe "make
VERBOSE=1" and copying from the compiler command line helps.

KDevelop is mostly optimized around being used with CMake or qmake, where
these can be queried from the build system automatically, and you don't need
to configure anything.

Best,
Sven

[René J.V. Bertin]

Post by Sven Brauch
KDevelop is mostly optimized around being used with CMake or qmake, where
these can be queried from the build system automatically, and you don't need
to configure anything.

Not so perfectly with qmake as with cmake, btw, in my experience. Maybe that's on Mac only, but I think that even on Linux I often have to add at least the path to the Qt5 header directory because otherwise certain headerfiles aren't found by the parser.

R.

[René J.V. Bertin]

paths" (as René suggested). I'm still not sure that I really understand how
to use these properly, and the manual is rather sparse about this feature.

Don't blame the manual, blame the people who're not trying to keep it complete ;)

The idea with "these" is simple. You shouldn't need them with cmake or qmake-based projects, but when your project is based on a Makefile you need to copy all non-standard header include files set in Makefile(s) and those preprocessor macros that are not set via a header file (config.h often).

Setting this up was extremely time consuming because it takes so long to
re-parse the project before the effect of the changes can be seen.

The trick there is to take it 1 file at a time; the Problem toolview has a reload/update button you can use to reparse just that file. Deactivate the full project parse feature or the "reparse project after changing" (in the defines/include paths settings dialog) so that you don't trigger a full reparse accidentally.

I find that disabling automatic full parsing at project load works well enough for me. Files are parsed when you open them, which includes all the headerfiles they include. So when file1.cpp from a project refers to a function from libfile2.cpp of that same project I will probably only get direct access to the function declaration from libfile2.h *at first*, but from there I can use a shortcut to open libfile2.cpp which will then be parsed. In the end I'll have a parse database representing only the parts of the project that are relevant to me. Even with audacity that can make a considerable difference.
FWIW, I also reduce the number of background parser threads to 2 at the most (I have only 4 cores in my system) because otherwise I find I have too little foreground reactivity left once the parser kicks in.

Cheers,
R.

[Axel Kellermann]

I'm not sure if I experience exactly the same problems as Steve and
Aleksey, but the symptoms seem similar enough that I feel I have to
chime in.
One of the largeish projects I occasionally work on is a cmake-based
C/C++ library with quite a deep directory structure and a ton of C/C++
and Python source files. Like in Steve's case, stability is not an issue
for me. Since the switch to clang I haven't had a single crash in the
parser. But at the same time kdevelop's parser seems to consistently
miss some of the projects header files, macros, templates, and function
declarations. I'm pretty sure the project isn't misconfigured, as
multiple people work on it on multiple platforms and with a variety of
IDEs, and I never heard complaints about parsing problems.
What's really weird is that only kdevelop installed via my systems
package manager seems to be affected (currently kdevelop 5.3 on Arch).
If I build kdevelop myself via kdesrc-build, the parsing seems to work
like a charm. At least all of the files I know of that previously made
trouble are processed correctly with my own build. Could that fact help
in pinpointing the trouble we seem to have? If so, what info could I
provide to help with that?

Axel

Post by Sven Brauch
Hi,

Post by Steve the Fiddle
The problem may be in part, due to
Audacity now making more use of C++11, and a little C++14, with more
templates and broader use of "modern" C++ idioms.

FWIW, this is almost always a setup problem, usually include paths and/or
preprocessor defines not being set up correctly. For CMake, KDevelop is very
good at setting this up automatically, for automake much less so.
If include paths, compiler flags and definitions are set correctly, KDevelop's
parser will be very close to perfect, since it uses clang behind the scenes.
It doesn't matter how complex your template code is.
Performance is a different topic, but correctness is very good these days.

[René J.V. Bertin]

Post by Steve the Fiddle
Kdevelop rarely crashes for me, so the database does not need to reparse
from scratch very often, but I still often see KDevelop reparsing very
slowly due to code changes outside of KDevelop (such as when Git bisecting).

For that reason I always set my config NOT to parse the entire project after opening, and I patch KDevelop so I have a "reparse entire project" action in the project manager context menu.

It also helps to "exclude from project" (= set up a filter) on build directories and patterns matching autoconf/automake temporary files.

Quit KDevelop explicitly before you log off, that way it's less likely to crash, and your next log in will be faster.

R

[Aleksey Midenkov]

Post by René J.V. Bertin

Post by Aleksey Midenkov
database is broken and is reparsing. And it breaks too much (several
times a day).

That shouldn't happen, except if you cause KDevelop to crash and then NOT tell it to preserve the cache when restarting. Or if you somehow open the same session in different versions (an older one from your distro and a more recent AppImage version for instance).

That shouldn't happen but it happens. KDevelop crashes f.ex. when you
change git branch or when you do KDE shutdown: that's not me, who does
something illegal, I guess. In most cases database gets broken when
git branch is changed. I mean symbols become dull (unjumpable) and it
even doesn't start to reparse, you have to do it manually. Sometimes
it works correctly, but rarely (I would not call it satisfactory).
Branch with relatively big changes breaks database for sure. The truth
is: KDevelop is not production-ready for serious development.