dir_filter for a HUGE number of directories
Mike D
incredibly large project where it's just not feasible (or necessary)
to parse through all the files and folders in the projects directory
hierarchy using the current auto-generated scripts. Unfortunately,
the project directory structure contains thousands of folders, and
tens of thousands of source and header files (not something that I
have the power to change).
Fortunately, a large number of these folders can be filtered out
because they're not needed for my purposes. The tricky part is
filtering them out correctly in a way where I can run the autogen
scripts to build all my tag, filelists, id-gen databases, etc for only
the files/folders i'm interested in.
In the past, I've successfully been using both cscope and ctags by
generating a file containing the entire list of absolute paths to all
the files I'm interested in, and feeding it into ctags and cscope, and
it's worked great. I haven't found any way to pass this sort of info
to exVim's autogen scripts though, it mainly seems to work with file-
type filters and folder filters. Folder filters seems like it could
do the trick, so I've slightly modified my script to spit out the full
list of folders that I'd like excluded, and then tried manually
inserting that into the autogen script that's generated through
(leader-C) once I've opened the exVim project file. When I try to run
the script however, it complains that the list is too long, and fails
to execute. I'm guessing there's probably some limit to the size of
the dir_filters list, and since I'm trying to filter out thousands of
folders i've probably passed that limit. Does anyone have any ideas
to work around this issue? Somehow I think I need to either filter
out all the folders I don't want, or pass in a list of all the folders
(or exact files) I do want... but in either case the list is going to
be huge.
Thanks in advance!
Wu Jie
Can you show me your dir_filter list? I think this problem is
caused by id-utils. Since id-utils have a limitation command line
length, if the dir_filter is longer than its limitation (problem 512
or 1024 character), it would report and error like you mentioned.
(also this could be happened by updating filename list. )
So we could confirm this problem by running the the commands below
in exVim.
:Update ID
:Update filenamelist
:Update tag
:Update symbol
:Update inherit
:Update cscope
If there one of these commands not successfully execute, then we
can know which one is criminal.
That's said you've done this and found ID have the wrong things, in
current state, may be you have to disable ID in your project. I've
thinking about re-writing id-utils so that it can support read
filename-list from a file instead of filter directories, but I really
don't have too much time to start this project, so I have to bear this
until I have more time.
If the wrong information caused by filenamelist, That will be some
big problem. (well, that's see the result first.)
By the way: The dir_filter only effect the first level directory,
so when I design exVim, I never expect people would have thousands of
folders in first level (but now it happened).
Any way, tell me your dir_filter and test result, then we can find
some solution.
Mike D
I didn't realize the dir_filter list only effect the first level
directory, so that could be part of the problem. The hierarchy of our
project can look something like this:
root->Internal---->PackagaA--->Version 1.0
--->Version 2.0
->External--->PackageB--->1.1
--->1.5
So 1) we have multiple versions of particular packages, usually
because someone is integrating a different version at some point, but
I only care about the one that's being linked against, and 2) Packages
are organized based on their purpose (internal, external, shared, etc)
so we end up with a crazy amount of sub-folders before you get to any
of the source files. I just experimented a bit by manually putting in
a custom "dir_filter" by issuing <leader-C> in the project window, and
I notice if I put in any folders that are more than one level deep, it
won't update any files in the project.
ex: <this works> dir_filter = c:\root\main
<this doesn't work> dir_filter = c:\root\main\external
So I guess the real problem is that I'm trying to use dir_filter
incorrectly based on it's original purpose? In that case, I supposed
I could create multiple .vimentry files, one for each major subfolder
(we have about 4), and then have the main .vimentry reference those
other vimentry files. That wouldn't solve the problem though that
each "package" often has multiple versions, and I would be getting
symbols for all versions of the package instead of the one that's
getting linked against. So, here is a quick example of a shorter
version of my dir_filter that I don't think will work:
dir_filter = ""c:/p4/test/main/ion/dev", "c:/p4/test/main/assert/
10.00.07", "c:/p4/test/Acme/1.02.00", "c:/p4/test/external/aemsupgpl/
1.10.00", "c:/p4/test/internal/AFK/dev""
Any idea how I can work around this? Ideally I would just send the
programs the exact list of files I want (so no need to filter, just
use the complete list as the files I am interested in) but I can see
that being a lot of work since each individual program (id-utils)
needs to support that I think?
Wu Jie
First of all, We need correct the syntax of the dir_filter. The
dir_filter won't receive the full-path, instead it just receive the
name of the folder (which in first level). So if you are working in c:
\root\main, and you have directory like
c:\root\main
c:\root\main\external
c:\root\main\src
c:\root\main\prj_1
c:\root\main\prj_2
c:\root\main\prj_3
and you only want the external and src be your directory, you can use
dir_filter as:
dir_filter = external,src
-------------------------------------------------------------------------------------------------------------------
I see your problem. Unfortunately, there is no perfect solution in
current version of exVim. I think I should add a feature to let the
user can use their own custom filenamelist.(But this not support yet,
and it won't works for the id-utils since this program not receive a
file that contains files for parsing :( )
After I review the code, I found it is really difficult to generate
project tree by custom file name list. It won't be an easy task
(probably take me 1-2 weeks.) But currently I don't have too much
time
to write this feature. So I put it to my task list until I have time
to do it. (it will be a long time, could wait for half yeas.) I'm
sorry about that.
Here is a temp-solution by using vimentry-refs:
First you create those project you want to involve to your work.
That's said you have the directory hierarchy like this:
p4
|- test
|- main
|- assert
|- 10.00.07
|- 10.00.08
|- acme
|- 1.02.00
|- 1.03.00
|- external
|- aemsupgpl
|- 1.10.00
|- 1.20.00
And you wish to use the main/assert.10.00.07, acme/1.02.00, external/
aemsupgpl/1.10.00, you have to create a vimentry to receive all the
info from sub-vimentry and also could manipulate the files. So we
create a vimentry in the root directory. that's said in p4/test/main/
my_project.vimentry . and then we have to create sub-vimentry in each
folder you want. for example:
main/assert.10.00.07/assert.vimentry
acme/1.02.00/acme.vimentry
external/aemsupgpl/1.10.00/aemsupgpl.vimentry
then refresh the project in each vimentry by /C or /R, and update
them. After this is done, we back to my_project.vimentry. in the
vimentry file, add the full path of sub-vimentry file by rewriting
the vimentryRefs value as below:
vimentryRefs+=c:/p4/main/assert.10.00.07/assert.vimentry
vimentryRefs+=c:/p4/acme/1.02.00/acme.vimentry
vimentryRefs+=c:/p4/external/aemsupgpl/1.10.00/
aemsupgpl.vimentry
Now that we have vimentryRefs, we could update the
my_project.vimentry
to let it load the project information from sub-vimentry files. But
since our main vimentry is put in the root directory, doing a full
update will make if parse all the files, tags, symbols and IDs. We
can't avoid update file list, IDs. Since they are update depends on
directory filter. So we do a full-update on them by refresh the
project (/R or /C) and typing Up IDs. After that, we have to update
the symbols, inherits, cscopes manually by typing:
:Up symbol
:Up inherit
:Up cscope
The tricks of this solution is let us avoid the tags update, but load
the symbols, tags, inherits from the sub-vimentry file. The
disadvantage is you still need to update the full file list in the
project and bear the long time update for IDs. (:()
Multi-project always have trouble against exVim. Since exVim is not
design in process HUGE project and some external tools used in it not
fit with filename list.
Mike D
for the time being, and maybe just live with only having the symbols
for the important packages/folders that I really need to eliminate
some of the update time.
Another idea that could work, although only on operating systems that
support symbolic links, would be for me
1) grab the entire list of folders that I'm interested in with full
paths
2) Use that list to create symbolic links at the base level package
(in a script of course)
So you would end up with something like this:
p4
|- test
|- main
|- assert
|- 10.00.07
|- 10.00.08
|- acme
|- 1.02.00
|- 1.03.00
|-10.00.08 <symlink>
|-1.03.00 <symlink>
If I write a script to automatically create those symlinks, then I
could use dir_filters like this
dir_filter = 10.00.08,1.03.00
The only problem is that it wouldn't work on Windows since it doesn't
support symbolic links, and also if any of the packages had the same
version then there would be a conflict.
If I have some free time sometime I will take a look at the idutils
source and see if I can create a patch that allows better filters to
be passed to it.
> ...
>
> read more »
Wu Jie
That's why we remove some platform-dependence feature.
If you read your quick_gen_project_xxx_autogen file, you would notice
that at the last section, the script try to find file
quick_gen_project_pre_custom.bat/sh
quick_gen_project_post_custom.bat/sh
and execute them before and after the quick_gen_project.bat/sh been
process.
Basically this is used for some project that have cpp-boost library.
The boost library have the problem in parsing ctags, so before parsing
it, it need to rename some file in xxx_pre_custom.bat/sh, and rename
back in xxx_post_custom.bat/sh. So if you already have your filename
list, and want to use it than the one generated by
quick_gen_project.bat/sh. You can use these two files to help you hack
the process. Here are the things you need to do.
1: Read the quick_gen_project.bat/sh in toolkit/quickgen/batch/
quick_gen_project.bat (in Windows is bat, Unix like system use sh).
That's use "sh" one as example. In the gen_filenamelist() function,
you can learn how exVim generate filenamelist by dir_filter and
file_filter. All you need to do is write your own replace them. The
filenamelist_cwd is used in generate cscope, and filenamelist_vimfiles
is used in generate ctags. All the rest file generation are based on
cscope/ctags file except IDs. So you can read the exact
filenamelist_(cwd/vimfiles) to learn the syntax in it, so that you can
generate them with your own script. Basically they just different
between the relate path. filenamelist_cwd use '.' and
filenamelist_vimfiles use '..' You still need to generate
filenametags which will bee used in Lookup-File plugin (F5 or gf to
trigger it. ). But it is trivial work once you generate the
filenamelist_xxx, you can copy the last section in gen_filenamelist()
to achieve this.
2: Once you write the script in generate filenamelist, you have to put
it in quick_gen_project_post_custom.bat/sh. NOTE: it is in post, keep
reading you will know why. Cause there is no way to let the
exVim :Update command to parse multiple option. You can only choose
update one option each time. So you only can use
:Update all
:Update IDs
:Update tags ....
but you can't do
:Update tags|IDs|cscope .... (yeah it is my fault :P )
Base on this, if we want to let custom filenamelist replace the
original one, we have to trigger quick_gen_project_xxx_autogen script
first, then make it process in our rule. By doing this in exVim we
use :Update ID instead of the :Update. The tricks here is no matter
how, the ID still use its own rule, so trigger it means we process the
script in quick_gen_project_xxx_auto like:
sh .\%vimfiles_path%\quick_gen_project_pre_custom.sh
sh "%toolkit_path%\quickgen\batch\quick_gen_project.sh" ID
sh .\%vimfiles_path%\quick_gen_project_post_custom.sh
Now suppose everything goes well, we are in
quick_gen_project_post_custom.sh script. In this script we write the
code below:
gen_custom_filenamelist # this function will be define by you.
# the reset function could be found and copied from
quick_gen_project.sh file.
gen_tag
gen_symbols
gen_inherits
gen_cscope
gen_id
Or may be a more lazy way is:
gen_custom_filenamelist # this function will be define by you.
sh "%toolkit_path%\quickgen\batch\quick_gen_project.sh" tag
sh "%toolkit_path%\quickgen\batch\quick_gen_project.sh" symbol
sh "%toolkit_path%\quickgen\batch\quick_gen_project.sh" inherit
sh "%toolkit_path%\quickgen\batch\quick_gen_project.sh" cscope
Now you could use the custom filename list in tag,symbol,inherit and
cscope. But the project-tree and IDs still use dir_filter.
Hope you can understand this, though it was another hack solution.
Any problem let me know.
Good Luck & Best Wishes :)
> ...
>
> read more »
Mike D
steps for the scripts, I think what you're suggesting will work for
me! It should be pretty straightforward for me to generate my own
file lists in the same format, just a few small changes to my existing
scripts. Thanks so much for all the help, exVim is a great product
and I'm slowly turning it into my new primary tool for working on a
daily basis :-)