[Differential] D6689: tcp/lro: Implement hash table for LRO entries.

hselasky (Hans Petter Selasky)

unread,

Jun 2, 2016, 10:35:16 AM6/2/16

to

_______________________________________________
freeb...@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net...@freebsd.org"

sepherosa_gmail.com (Sepherosa Ziehau)

unread,

Jun 2, 2016, 12:09:52 PM6/2/16

to

sepherosa_gmail.com (Sepherosa Ziehau)

unread,

Jun 2, 2016, 12:24:22 PM6/2/16

to

sepherosa_gmail.com (Sepherosa Ziehau)

unread,

Jun 2, 2016, 12:26:07 PM6/2/16

to

D6689.17214.patch

hselasky (Hans Petter Selasky)

unread,

Jun 2, 2016, 12:28:33 PM6/2/16

to

sepherosa_gmail.com (Sepherosa Ziehau)

unread,

Jun 2, 2016, 12:35:52 PM6/2/16

to

hselasky (Hans Petter Selasky)

unread,

Jun 2, 2016, 12:36:04 PM6/2/16

to

sepherosa_gmail.com (Sepherosa Ziehau)

unread,

Jun 6, 2016, 4:40:18 AM6/6/16

to

sepherosa_gmail.com updated this revision to Diff 17347.
sepherosa_gmail.com added a comment.

Address hps's cache pollution concern for queue_mbuf path. Similar approach was in the patch obtained from rrs.

CHANGES SINCE LAST UPDATE
https://reviews.freebsd.org/D6689?vs=17214&id=17347

REVISION DETAIL
https://reviews.freebsd.org/D6689

AFFECTED FILES
sys/netinet/tcp_lro.c
sys/netinet/tcp_lro.h

EMAIL PREFERENCES
https://reviews.freebsd.org/settings/panel/emailpreferences/

To: sepherosa_gmail.com, rrs, gallatin, glebius, gnn, bz, rwatson, #transport, hselasky
Cc: freebsd-net-list

D6689.17347.patch

sepherosa_gmail.com (Sepherosa Ziehau)

unread,

Jun 6, 2016, 4:41:45 AM6/6/16

to

sepherosa_gmail.com added a comment.

In https://reviews.freebsd.org/D6689#141127, @hselasky wrote:

> > Well, as I said the VM does not have the luxury to do sorting :)
>
> Can you test the existing "tcp_lro_flush_all()" in combination with "tcp_lro_queue_mbuf()" and compare it to your version? Sorting is not as slow as you think.
>
> --HPS

I will use the updated patch to test the hash and the queue_mbuf in Hyper-V. Thanks!

REVISION DETAIL
https://reviews.freebsd.org/D6689

EMAIL PREFERENCES
https://reviews.freebsd.org/settings/panel/emailpreferences/

To: sepherosa_gmail.com, rrs, gallatin, glebius, gnn, bz, rwatson, #transport, hselasky
Cc: freebsd-net-list

gallatin (Andrew Gallatin)

unread,

Jun 7, 2016, 9:16:32 AM6/7/16

to

gallatin accepted this revision.
gallatin added a comment.

Looks good in terms of not killing perf. for the sorted case, and I'm fine with it as-is. However, maybe an else would be better than a goto?

REVISION DETAIL
https://reviews.freebsd.org/D6689

EMAIL PREFERENCES
https://reviews.freebsd.org/settings/panel/emailpreferences/

To: sepherosa_gmail.com, rrs, glebius, gnn, bz, rwatson, #transport, hselasky, gallatin

sepherosa_gmail.com (Sepherosa Ziehau)

unread,

Jun 7, 2016, 9:26:02 PM6/7/16

to

sepherosa_gmail.com added a comment.

In https://reviews.freebsd.org/D6689#142118, @gallatin wrote:

> Looks good in terms of not killing perf. for the sorted case, and I'm fine with it as-is. However, maybe an else would be better than a goto?

Yeah, sure, I can move the bucket assignment into the {}, and use if-elseif-else to avoid goto.

And as for performance, I am running the test using hash table currently. The result of nginx+wrk test is quite promising (web object sizes range from 1KB~40KB; for 4reqs/conn, 4~10K concurrent connection, I am getting 12%~14% performance improvement :). 14reqs/conn nginx+wrk test is still running.

hselasky (Hans Petter Selasky)

unread,

Jun 8, 2016, 8:17:38 AM6/8/16

to

hselasky added a comment.

Hi,

Were you able to test the performance using tcp_lro_queue_mbuf() ?

Better name for function?? tcp_lro_rx2() -> tcp_lro_rx_sub()

--HPS

hselasky (Hans Petter Selasky)

unread,

Jun 8, 2016, 9:29:22 AM6/8/16

to

hselasky added a comment.

@sepherosa_gmail.com

Regarding performance. Is it possible to get the Hyper-V to sort the IP-packets before they enter the FreeBSD network stack in the VM?

sepherosa_gmail.com (Sepherosa Ziehau)

unread,

Jun 12, 2016, 9:40:00 PM6/12/16

to

sepherosa_gmail.com (Sepherosa Ziehau)

unread,

Jun 12, 2016, 9:41:25 PM6/12/16

to

sepherosa_gmail.com (Sepherosa Ziehau)

unread,

Jun 13, 2016, 10:10:42 PM6/13/16

to

D6689.17569.patch

rrs (Randall Stewart)

unread,

Jul 5, 2016, 9:28:07 AM7/5/16

to

rrs accepted this revision.
rrs added a comment.

These look fine and familiar ;-)

I think a hash table as an option to sorting is probably a good thing :D

REVISION DETAIL
https://reviews.freebsd.org/D6689

EMAIL PREFERENCES
https://reviews.freebsd.org/settings/panel/emailpreferences/

To: sepherosa_gmail.com, glebius, gnn, bz, rwatson, gallatin, hselasky, #transport, rrs
Cc: freebsd-net-list

sepherosa_gmail.com (Sepherosa Ziehau)

unread,

Jul 5, 2016, 9:44:07 PM7/5/16

to

sepherosa_gmail.com added a comment.

In https://reviews.freebsd.org/D6689#148252, @rrs wrote:

> These look fine and familiar ;-)
>
> I think a hash table as an option to sorting is probably a good thing :D

Yeah, and I got pretty exiting performance improvement and noticeable latency reduction with both methods in Azure for nginx 1KB~40KB web objects and 4/14 reqs/conn :)). I will post the result once 128/256 mbuf queue depth for sorting method is done (512/1024/2048/4096 have been completed so far).

sepherosa_gmail.com (Sepherosa Ziehau)

unread,

Jul 7, 2016, 9:48:36 PM7/7/16

to

sepherosa_gmail.com added a comment.

In https://reviews.freebsd.org/D6689#148473, @sepherosa_gmail.com wrote:

> In https://reviews.freebsd.org/D6689#148252, @rrs wrote:
>
> > These look fine and familiar ;-)
> >
> > I think a hash table as an option to sorting is probably a good thing :D
>
>
> Yeah, and I got pretty exiting performance improvement and noticeable latency reduction with both methods in Azure for nginx 1KB~40KB web objects and 4/14 reqs/conn :)). I will post the result once 128/256 mbuf queue depth for sorting method is done (512/1024/2048/4096 have been completed so far).

If you are interested, I have the measurement result posted here:
results <https://people.freebsd.org/~sephe/lro_pic/>

For nginx workload, both latency and performance are measured. As you can see both performance and latency got improved, almost for all nginx workloads we have tested so far.

Both methods provide significant improvement for some kinds of workload:
1KB obj 14reqs/conn <https://people.freebsd.org/~sephe/lro_pic/1K_14.png>
8KB obj 14reqs/conn <https://people.freebsd.org/~sephe/lro_pic/8K_14.png>

Phabricator

unread,

Aug 2, 2016, 2:37:30 AM8/2/16

to

This revision was automatically updated to reflect the committed changes.
Closed by commit rS303656: tcp/lro: Implement hash table for LRO entries. (authored by sephe).

CHANGED PRIOR TO COMMIT
https://reviews.freebsd.org/D6689?vs=17569&id=18953#toc

REPOSITORY
rS FreeBSD src repository

CHANGES SINCE LAST UPDATE
https://reviews.freebsd.org/D6689?vs=17569&id=18953

REVISION DETAIL
https://reviews.freebsd.org/D6689

AFFECTED FILES
head/sys/netinet/tcp_lro.c
head/sys/netinet/tcp_lro.h

D6689.18953.patch