SDCH dev tools

[Greg Howett]

Hi All,

 

I work for jetNEXUS and we spent a fair amount of time on SDCH. We wrote a product internally we call jetSTREAM that uses SDCH but if the client supports it will upgrade to jetSTREAM. It has many improvement over SDCH in terms of dictionary selection, security and clients compatibility.

 

We also played about and wrote an ISAPI  for IIS (old I know) for SDCH and jetSTREAM

 

In order to develop this we built a number of “internal” test tools.

 

I have uploaded a couple as they may be of use. I have not uploaded the source code just the object code for windows.

 

Dictionary test and delta test tool

https://github.com/jetNEXUS/jetNEXUS-vcdifftester

 

SDCH, jetSTREAM Test client

https://github.com/jetNEXUS/jetNEXUS-SDCHHTTPClient

 

We are looking to adding this technology to our load balancer ADC this year.

 

Good luck with your projects and ill post any other tools that maybe useful.

 

Kind regards

Greg

 

From: SD...@googlegroups.com [mailto:SD...@googlegroups.com] On Behalf Of ernest.w....@gmail.com
Sent: 05 January 2016 05:27
To: SDCH <SD...@googlegroups.com>
Subject: Re: SDCH for templated server response

 

Thanks again Linkoln, it still wasnt clear what "aligned block" means but line 268 gives excellent example which explains it all. 

On Monday, January 4, 2016 at 7:41:45 PM UTC+2, openvcdiff wrote:

Hi Ernest:

1. I've encountered somewhat odd behavior, I see relatively short string, say 10 bytes are not replaced. Even when I encode the dictionary itself using the same dictionary, which theoretically would leave me with file free of any string and just delta instruction, is it correct?

Please read the description of kBlockSize in the open-vcdiff source code:

https://github.com/google/open-vcdiff/blob/master/src/blockhash.h

 

Checked the kBlockSize, as I understand this block defines minimal substring to encode however i have following data in dictionary 

var ebPtcl="http://",ebBigS="

and server response has a string 

var ebPtcl="http://",ebBigS="blah-blah some long string"

 

however the string before the "blah-blah" stuff is not being encoded. is it expected behavior?

 

Please reread the description of kBlockSize from the link I mentioned above,

in particular the comment on lines 49-53.

In this case, kBlockSize is 16 and your match has size 29, so yes, this

behavior is possible depending on the alignment of the match within the

target text.

 

Cheers,

Lincoln 

--
You received this message because you are subscribed to the Google Groups "SDCH" group.
To unsubscribe from this group and stop receiving emails from it, send an email to SDCH+uns...@googlegroups.com.
To post to this group, send email to SD...@googlegroups.com.
Visit this group at https://groups.google.com/group/SDCH.
For more options, visit https://groups.google.com/d/optout.

[Gregor]

Hi All,

 

I work for jetNEXUS and we spent a fair amount of time on SDCH. We wrote a product internally we call jetSTREAM that uses SDCH but if the client supports it will upgrade to jetSTREAM. It has many improvement over SDCH in terms of dictionary selection, security and clients compatibility.

 

We also played about and wrote an ISAPI  for IIS (old I know) for SDCH and jetSTREAM

 

In order to develop this we built a number of “internal” test tools.

 

I have uploaded a couple as they may be of use. I have not uploaded the source code just the object code for windows.

 

Dictionary test and delta test tool

https://github.com/jetNEXUS/jetNEXUS-vcdifftester

 

SDCH, jetSTREAM Test client

https://github.com/jetNEXUS/jetNEXUS-SDCHHTTPClient

 

We are looking to adding this technology to our load balancer ADC this year.

 

Good luck with your projects and ill post any other tools that maybe useful as we wrote a dictionary creator programme also 

 

Kind regards

Greg

 

[ernest.w....@gmail.com]

Hi Greg,

thanks for sharing! BTW, do you use ISAPI filter or extension? I POCed SDCH in our old ISAPI filter with success however I have a difficulty to add "sdch" to content encoding http header - it simply cancels IIS (7.5) build-in dynamic compression. did you encounter such a problem?

would you like to share a little bit more? what took most time of below "fair amount of time"? I did it in one day and I'm afraid that I have wrong estimation of "work done" since everyone speaking about time it took to implement. I have a feeling that I miss something

[Greg Howett]

Hi Ernest,

 

It’s been such a long time since I last looked at it J

 

We did end up with a working ISAPI and I do remember the compression content type encoding  was a nightmare to solve but we did end up with a working product.

 

We ended up building our own product (based on principle of SDCH) that uses a JavaScript client as the end point so it can run on any browser (well almost any)

 

This product runs (will run J ) on our ADC load balancing platform.

 

I have enclosed some of the dev notes that may of use.

 

We did write some windows maenad components that could be used in other projects.

 

Cheers

[ernest.w....@gmail.com]

Thanks Greg, will check it.

[Christopher O'Connell]

Hello Everyone,

We're just putting together a blog post about our experiences with SDCH. As promised, we've put a few of our scripts up on GitHub here https://github.com/mikandi/sdch-tools

In addition to our tools for compiling dictionaries, I've built a simple tool that downloads 30 pages from Wikipedia, trains on 10 of them and then shows the differences in size between raw VCDiff, GZip and SDCH (VCDiff+GZip).

Please feel free to pull the repo and try it out. Also, any feedback if there's any minor problems with paths, versions of programs, is most welcome.

All the best,

~ Christopher

[Chris Bentzel]

Cool! Looking forward to the blog post and thanks for open sourcing the dictionary tools.