Can bazel provide a rule for copying recursively from one directory to another?

[Leo Yin]

Hi,

I'm working a Java project of about 500,000 lines of code and changing the ant build.xml to bazel BUILD file. 

The build.xml has many tasks of copying files in directory trees. For example,

     <target name="copy_static">

              ...

              <copy todir="${Common_Template}/Runtime/LegoRuntime/resources/report/dataExport/exportTemplate/lib/lib" overwrite="true">

<fileset dir="${reporterProject}/WebRoot/lib">

<include name="echarts/**/*" />

<exclude name="**/*.svn" />

</fileset>

<fileset dir="${reporterProject}/WebRoot/lib/">

<include name="jquery-mousewheel/**/*" />

<exclude name="**/*.svn" />

</fileset>

<fileset dir="${reporterProject}/WebRoot/lib/">

<include name="tiny/**/*" />

<exclude name="**/*.svn" />

</fileset>

<fileset dir="${reporterProject}/WebRoot/lib/">

<include name="tiny-extra/**/*" />

<exclude name="**/*.svn" />

</fileset>

<fileset dir="${reporterProject}/WebRoot/lib">

<include name="echarts.source.js" />

<include name="require.js" />

</fileset>

</copy>

                ...

I know genrule() can be used to copy files. However, for copying a large number of files in some directory tree genrule() is too low level 

and it is tedious to write code for such genrule() 's srcs/outs/cmd. -- think about listing hundreds and thousands of files in genrule()'s srcs/outs/cmds.

Can bazel provide a rule for such purpose? The rule may has the form as follows:

recursive_copy_dir(

    name = "copy1",

    srcs = glob(

                   ["dir1/dir2/**/*.java"],                                          

                   exclude = ["**/dir5/**"],                                       # can exclude files we don't need

    ) +

    glob(["dir3/**/*.java"]),                                                       # can have multiple directories in srcs

    dst_dir = "dst_dir",                                                             # should only have one dst directory.

)

The outs of above rule should be as follows:

            dst_dir/**/*.java                                                         # files from dir1/dir2 and dir3

':copy1' may be used to refer to the rule's outs; $(location :copy1) may be used to refer to the directory 'dst_dir' (so $(location :copy1)/a.java can refer to a specific file in the outs of the rule).

Thank you very much.

-Leo

[Leo Yin]

It seems to me that bazel does not have any rule which can do the job of <copy todir=...> in ant in a convenient way (as I have explained, genrule() is too low level and not suitable for a large number of file copying). 

Since ant is used widely to build Java projects, the rule may be necessary and very helpful: It will be an easy job to map <copy todir=...> in ant to bazel BUILD.

regards,

-Leo

在 2016年4月21日星期四 UTC+8下午8:34:31，Leo Yin写道：

[Damien Martin-guillerez]

Can you tell me a bit more on why you want to copy files around?

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/0b68ea23-f87b-4831-855c-d8f0f32fd61e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Leo Yin]

The project is a web service program running in Tomcat. It uses many third party libraries and text files. Its source code consists of many xml files, javascript files, and java files.

The reason for copying source javascript files is that there is only one copy of those files in the svn server and they are used in different directories: 1) used by Tomcat and 2) used by a browser daemon process to generate xml files and reporter tables. 

The reason for copying third party libraries text files is that our company requires all third party files should be put into a specific directory in the svn server for easy management (for example, performing open source scanning by using blackduck software); at the same time these third party files need be copy to directories of particular structures to be packed to form Tomcat jar files.

Finally, the project has linux version and windows version. So common files need being copied to directories for linux version and windows version respectively, to produce corresponding releases.

regards,

-Leo

在 2016年4月21日星期四 UTC+8下午8:57:10，Damien Martin-guillerez写道：

[Damien Martin-guillerez]

What would be the problem with using a pkg_tar? (taring the files)

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/22fa49e5-7b30-4c8f-8d93-afc2aa14fef8%40googlegroups.com.

[Leo Yin]

pkg_tar() is not suitable for the job. 

Considering the following case:

       There are already files in dir2. The job is copying some other files from dir1 to dir2. 

       Using pkg_tar(), we need pack dir1/* into a tar file, copy it to dir2, and finally extract files from the tar file under dir2.

       The final action of extracting files from the tar file should be implemented in genrule(), which needs to list all those files in genrule()'s outs and cmd, which is inconvenient when the number of files is big.

It may has practical reasons that Ant provide the <copy todir=...> action. Could bazel provide a counterpart rule()?

在 2016年4月22日星期五 UTC+8下午7:05:33，Damien Martin-guillerez写道：

[Damien Martin-guillerez]

Sorry I still don't understand what is wrong with taring. In appengine rules we do what you describe without any problem: https://google.com/bazelbuild/rules_appengine

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/88699dff-848f-47af-a465-e2c22a7b7fba%40googlegroups.com.