custom-tools plugin: adding support for 7z-files

[Martin Hjelmqvist]

Hello,

I'm currently working on my bachelor thesis; Improve the Jenkins custom tools installer. There is a need for a file format able to contain up to 100GB of installation files, since tar.gz and zip has a size limit of ~8GB and ~4GB size limit respectively. The idea is to add support for the file format 7z which currently has a size limit of 16 exabytes and also is faster to unpack than a zip file.

Anything I should keep in mind when integrating the extension for this plugin? So far I have developed code for unpacking a 7z-file through a string file path, but not sure that will work in the end.
Any thoughts?

Sincerely,
Martin Hjelmqvist

Bachelor Thesis at HMS Industrial Networks, Sweden

[Oleg Nenashev]

Hi Martin,

Great to hear that you're working on this plugin. I'm it's maintainer, so I will try to help where I can. If you consider a long-term project ("if you graduate on the next year"), maybe you would be interested to make a proposal for Google Summer Of Code. I think we can find mentors for this project

Regarding your question...

Actually, "ToolInstallation" and "ToolInstaller" are different entities in Jenkins. Custom Tools Plugin provides the ToolInstallation implementation, but it uses standard ToolInstallers being provided by other Jenkins plugins.

If you want to add a 7zip ToolInstaller, I think Extra Tool Installers Plugin is the best destination for such feature. You can find examples of extension point implementations there.

Best regards,
Oleg Nenashev

вторник, 22 марта 2016 г., 10:50:10 UTC+1 пользователь Martin Hjelmqvist написал:

[Martin Hjelmqvist]

Hi Oleg!

Good to hear! This project is planned to be finished by the end of May this year though. 

Aha, I need to approach this in another way then. Are you proposing that I add this feature in the Extra Tool Installers Plugin that extends AbstractExtraToolInstaller (which in turn extends ToolInstaller)? By extension points, you mean like BatchCommandInstaller and SharedDirectoryInstaller? Trying to get a picture of the structure. :)

Sincerely,
Martin Hjelmqvist

[Oleg Nenashev]

Hi Martin,

Yes, GSoC is definitely not an option for this year bachelor thesis.

Are you proposing that I add this feature in the Extra Tool Installers Plugin that extends AbstractExtraToolInstaller (which in turn extends ToolInstaller)? By extension points, you mean like BatchCommandInstaller and SharedDirectoryInstaller?

Something like that. Generic Tool installers definitely do not belong to Custom Tools Plugin from my PoV.

Just a note on the terminology:

• Extention Point - ToolInstaller (it extends the ExtensionPoint interface)
• Extension Point implementation - BatchCommandInstaller, SharedDirectoryInstaller, etc.

BR, Oleg

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/74aZNSlm3Yg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/15de6fd5-4403-40ce-9140-8b5b3b13b071%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Martin Hjelmqvist]

Hello again, Oleg

I could use some more advice about how to integrate this feature.
If this 7zip-installer is added to ExtraToolInstallers, where in custom tools plugin do you think I should call the 7zip-installer? 
So far the tracks lead me to hudson.tools.FilePath where an if statement checks whether the archive ends with zip or tar.gz.
Source (line 821 and 847): 
https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/FilePath.java

Should I do any changes there as well or is there any other way to do this?

Sincerely,
Martin Hjelmqvist 

[Boris Brodski]

Sorry for interfering. Can 7-Zip-JBinding be helpful in some way here?

http://sevenzipjbind.sourceforge.net/

[Martin Hjelmqvist]

I have code for a simple 7z-extractor using Apache's SevenZ classes, but 7-Zip-JBinding might be better to use instead.

I'll have a closer look at it later. Thanks for the tip. :)

[Vincent Daburon]

Hi,
In the javadoc, you could see :
https://commons.apache.org/proper/commons-compress/apidocs/index.html?org/apache/commons/compress/archivers/sevenz/package-tree.html

"The format is very Windows/Intel specific, so it uses little-endian byte order, doesn't store user/group or permission bits, and represents times using NTFS timestamps (100 nanosecond units since 1 January 1601). Hence the official tools recommend against using it for backup purposes on *nix, and recommend .tar.7z or .tar.lzma or .tar.xz instead."

Not sure, it will be a good feature for *nix OS.

Regards.
Vincent D.

[Oleg Nenashev]

If this 7zip-installer is added to ExtraToolInstallers, where in custom tools plugin do you think I should call the 7zip-installer? 

In the case of ToolInstaller you don't need to modify Custom Tools at all.
When Custom Tool plugin requests a tool home directory, all the installation happens automagically. It happens here.

BR, Oleg

четверг, 24 марта 2016 г., 14:24:03 UTC+1 пользователь Vincent Daburon написал:

[Martin Hjelmqvist]

Hello again.

Been away for a while, but I'm back at it again.

Got a couple of questions also.

So, using the format 7z might cause a problem due to the endianness of the format.

1. How does the current extraction of tar.gz/zip files handle this issue? As far as I know the extraction is done in hudson.tools.FilePath, but not sure how it handles different endianness.

2. Does the extractor code need to find out what kind of endian, big or little, that is being used and then extract the tool differently depending on the endianness? Or is this handled automatically somehow?

3. Should I consider a different format entirely, considering that 7z doesn't store user/group or permission bits etc? (See [3.1] below)
If yes, Is RAR a better alternative?

[3.1] SevenZ
"The format is very Windows/Intel specific, so it uses little-endian byte order, doesn't store user/group or permission bits, and represents times using NTFS timestamps (100 nanosecond units since 1 January 1601). Hence the official toolsrecommend against using it for backup purposes on *nix, and recommend .tar.7z or .tar.lzma or .tar.xz instead."


Best regards,
Martin Hjelmqvist

[Oleg Nenashev]

Hi Martin,

Let me apologize for the response delay. And thanks for pinging me via a direct message. I was snowed under GSoC activities and missed the notice. If there is no response within 2 days, please do not hesitate to ping me again.

Please find my feedback below...

How does the current extraction of .tar.gz/.zip files handle this issue? As far as I know the extraction is done in hudson.tools.FilePath, but not sure how it handles different endianness.

You are right, the extraction happens in FilePath. In the Jenkins project we rely on the implementation in org.apache.commons.compress.archivers. It supports endian bits at least for unarchive operations.

2. Does the extractor code need to find out what kind of endian, big or little, that is being used and then extract the tool differently depending on the endianness? Or is this handled automatically somehow?

Yes and no, we rely on the library. So for the Jenkins project it's "automatic"

3. Should I consider a different format entirely, considering that 7z doesn't store user/group or permission bits etc? (See [3.1] below)

If yes, is RAR a better alternative perhaps?

RAR has its own issues. For your case I'd recommend to stick to a particular format and to implement it. Not every tool needs permission bits. The advantage is that there is a big number of Windows tools being delivered by RAR. Hence such installer provides much more added alue than 7zip.

I would also like to mention that Jenkins' FilePath implementation supports files with size >8Gb (starting from 1.627) via LONGFILE_GNU extension. The most of the bits have been implemented here: https://github.com/jenkinsci/jenkins/commit/b6c7b83e7ba1a7538382d220d044626f2b2e65be . I would expect the tar installer to work on big files, but it does not decrease the potential added value of adding new installers , especially in the case of RAR files.

Hopefully it helps,
Oleg

2016-04-16 16:10 GMT+02:00 Martin Hjelmqvist <martin-h...@hotmail.com>:

Hello, Oleg!

Hope everything's good with you.

I have a few questions about this project I'm working on (adding support for 7z-files) I was hoping you could answer.

So, using the format 7z might cause a problem due to the endianness of the format.

1. How does the current extraction of .tar.gz/.zip files handle this issue? As far as I know the extraction is done in hudson.tools.FilePath, but not sure how it handles different endianness.

2. Does the extractor code need to find out what kind of endian, big or little, that is being used and then extract the tool differently depending on the endianness? Or is this handled automatically somehow?

3. Should I consider a different format entirely, considering that 7z doesn't store user/group or permission bits etc? (See [3.1] below)

If yes, is RAR a better alternative perhaps?

[3.1] SevenZ javadoc

"The format is very Windows/Intel specific, so it uses little-endian byte order, doesn't store user/group or permission bits, and represents times using NTFS timestamps (100 nanosecond units since 1 January 1601). Hence the official toolsrecommend against using it for backup purposes on *nix, and recommend .tar.7z or .tar.lzma or .tar.xz instead."


Best regards,
Martin Hjelmqvist

вторник, 12 апреля 2016 г., 14:54:58 UTC+2 пользователь Martin Hjelmqvist написал:

[Martin Hjelmqvist]

Thanks for the reply. Figured you were busy with something. :)

 

You are right, the extraction happens in FilePath. In the Jenkins project we rely on the implementation in org.apache.commons.compress.archivers. It supports endian bits at least for unarchive operations. 

Yes and no, we rely on the library. So for the Jenkins project it's "automatic"

I'll look up the endian compatibility with SevenZ and JUnrar (libraries for .7z and .rar respectively).

I've been looking into what you mentioned about Extra Tool Installers, but not quite sure how do approach it anyway. Please do correct the following proposals if they are wrong. Either way, I need to extend hudson's FilePath.java in order to recognize the format of the installation files. The following code is from FilePath.java in installIfNecessary(...), where the blue code is where I think I'll add the format check. Next I think I'll try to imitate the unzip methods' structure in that class as much as possible when implementing the rar extractor. (Adding "unrar" methods where the extraction is done).

	try { if(archive.toExternalForm().endsWith(".zip")){
	unzipFrom(cis); }else if(archive.toExternalForm().endsWith(".rar")){ unrarFrom(cis);
	}else{
	untarFrom(cis,GZIP); }
	} catch (IOException e) {
	throw new IOException(String.format("Failed to unpack %s (%d bytes read of total %d)",
	archive,cis.getByteCount(),con.getContentLength()),e);
	}

 

What do you think about this approach? Any other ideas?

RAR has its own issues. For your case I'd recommend to stick to a particular format and to implement it. Not every tool needs permission bits. The advantage is that there is a big number of Windows tools being delivered by RAR. Hence such installer provides much more added alue than 7zip.

I'm able to extract both .rar and .7z files, so I think I'm quite flexible on that point. If RAR is preferred, I could implement support for that format.

 

I would also like to mention that Jenkins' FilePath implementation supports files with size >8Gb (starting from 1.627) via LONGFILE_GNU extension. The most of the bits have been implemented here: https://github.com/jenkinsci/jenkins/commit/b6c7b83e7ba1a7538382d220d044626f2b2e65be . I would expect the tar installer to work on big files, but it does not decrease the potential added value of adding new installers , especially in the case of RAR files.

Aha, interesting. How does that work? Is there a way to select some sort of long name option before compressing files? Not familiar with it. Good to hear that the work won't be in vain at least.

[Oleg Nenashev]

Aha, interesting. How does that work? Is there a way to select some sort of long name option before compressing files? Not familiar with it. Good to hear that the work won't be in vain at least.

See http://commons.apache.org/proper/commons-compress/tar.html . The library supports "star/GNU tar/BSD" and "POSIX/PAX" extensions for tar specification, which allow handling big files.

Either way, I need to extend hudson's FilePath.java in order to recognize the format of the installation files. ... What do you think about this approach? Any other ideas?

Most likely you do not need it, because you can implement a standalone ToolInstaller. But such switch is a good option for generic installers as well.

BR, Oleg
 

воскресенье, 1 мая 2016 г., 23:40:14 UTC+2 пользователь Martin Hjelmqvist написал:

[Martin Hjelmqvist]

Most likely you do not need it, because you can implement a standalone ToolInstaller. But such switch is a good option for generic installers as well.

Aha, so making a standalone ToolInstaller should work then?

What needs to be done in a standalone in order for this to work? 

Should I copy the behavior of FilePath's installIfNecessary(...) into my performInstallation(...) regarding connections and so on?

Will this ultimately end up like a copy of FilePath of some sort?

Here is a link to my Github repository with my current RAR extractor if you want to see.

It uses String file paths to point where the RAR file is and where to extract the content on the local computer.

How should the extractor know where to extract a file in this standalone installer? Using strings won't work, will it?

So many questions, haha.

Thanks again for helping me.

Sincerely,
Martin Hjelmqvist

[Martin Hjelmqvist]

Bump :)

[Oleg Nenashev]

> Aha, so making a standalone ToolInstaller should work then?

Yes

> What needs to be done in a standalone in order for this to work? Should I copy the behavior of FilePath's installIfNecessary(...) into my performInstallation(...) regarding connections and so on? Will this ultimately end up like a copy of FilePath of some sort?

ToolInstallers are pretty simple. So generally you check availability of the tool in the destination, and then unarchive the stuff if there is no unpacked data there.

--

You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/74aZNSlm3Yg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/28a4e8a7-391b-4118-9765-25492e17f029%40googlegroups.com.

[Martin Hjelmqvist]

Been some time since last post, but here's an update.
A .7z extractor is no longer to be developed, instead it will be a .rar extractor using the JUnrar library.
A pull request* on Github to the extra-tool-installers-plugin repo has been made and the progress can be seen here.

However, when I package the plugin in Netbeans, the rar extraction installer won't show in the Global Tool Configuration view in Jenkins. This happens when a rar Archive is initialized in the code. If only declared, the installer can be seen in the view. The dependency for JUnrar is listed in the pom.xml and is found during build. The pom.xml can be seen in the pull request.
My question is, what makes the installer disappear if a JUnrar Archive is initialized, and what must I do in order to make it work?

*Keep in mind that the code in the installer class is still under development.

All feedback is welcome.