Shallalist vs Urlblacklist vs Squidblacklist
3,688 views
Skip to first unread message
Squidblacklist org
Jun 15, 2017, 9:06:06 PM6/15/17
to e2guardian
Greetz e2Guardian!
The importance of well maintained blacklist data has serious implications for Squidguard and Dansguardian based web filtering solutions, as they tend to rely heavily on blacklists which typically come from one of the providers we will discuss in this brief article. These affected web filtering platforms are deployed at various locations worldwide, from medical facilities, libraries, educational facilities, military establishments, banks, hotels, the deployment is widespread. If your web filtering strategies depend on blacklists, then you should pay particular attention to the results of this simple analysis.
We invite you to download the data sets of dead domains that we used to generate the chart above, See for yourself that these dead domains are in fact, listed in their blacklists. http://www.squidblacklist.org/shalla-deadpool.zip and http://www.squidblacklist.org/urlbl-deadpool.zip.
Here we have analyzed the porn blacklists produced by both of these entities, and we have as a result, determined the exact number of dead domains actually exceeds the number of live domains in both blacklists. Dead domains defined as, expired, parked, redirected, or otherwise dead websites. And therefore, we have concluded that the maintainers of these lists are either incompetent or simply dont care.
Since 2012, we we have come a long way. We have developed a unique domain name management and blacklist production system that allows us to produce and manage our blacklists in a way that allows us to bring you a higher level of quality unseen before. You need the resources and tools you can rely on. And if you are using Dansguardian or Squidguard, then you need consider the very real opportunity to breath new life in your existing web filtering platforms and applications by switching your blacklist provider to Squidblacklist.org. Additionally we hope to empower new development of partnerships and solutions through our innovations. We intend to keep evolving our technology to continue pushing forward, to pave a future in which poorly maintained blacklists are a thing of the past.
Footnote: We would have much rather have co-operated with the operators of shalla and urlblacklist in a productive manner, but they opted to ignore our emails.
Signed,Benjamin E. Nichols
isoi...@gmail.com
Jul 19, 2017, 9:35:36 PM7/19/17
to e2guardian
On Thursday, June 15, 2017 at 7:06:06 PM UTC-6, Squidblacklist org wrote:
> Greetz e2Guardian!
>
>
> The importance of well maintained blacklist data has serious implications for Squidguard and Dansguardian based web filtering solutions, as they tend to rely heavily on blacklists which typically come from one of the providers we will discuss in this brief article. These affected web filtering platforms are deployed at various locations worldwide, from medical facilities, libraries, educational facilities, military establishments, banks, hotels, the deployment is widespread. If your web filtering strategies depend on blacklists, then you should pay particular attention to the results of this simple analysis.
>
>
>
> Greetz e2Guardian!
>
>
> The importance of well maintained blacklist data has serious implications for Squidguard and Dansguardian based web filtering solutions, as they tend to rely heavily on blacklists which typically come from one of the providers we will discuss in this brief article. These affected web filtering platforms are deployed at various locations worldwide, from medical facilities, libraries, educational facilities, military establishments, banks, hotels, the deployment is widespread. If your web filtering strategies depend on blacklists, then you should pay particular attention to the results of this simple analysis.
>
>
>
> We invite you to download the data sets of dead domains that we used to generate the chart above, See for yourself that these dead domains are in fact, listed in their blacklists. http://www.squidblacklist.org/shalla-deadpool.zip and http://www.squidblacklist.org/urlbl-deadpool.zip.
>
>
> Here we have analyzed the porn blacklists produced by both of these entities, and we have as a result, determined the exact number of dead domains actually exceeds the number of live domains in both blacklists. Dead domains defined as, expired, parked, redirected, or otherwise dead websites. And therefore, we have concluded that the maintainers of these lists are either incompetent or simply dont care.
> Since 2012, we we have come a long way. We have developed a unique domain name management and blacklist production system that allows us to produce and manage our blacklists in a way that allows us to bring you a higher level of quality unseen before. You need the resources and tools you can rely on. And if you are using Dansguardian or Squidguard, then you need consider the very real opportunity to breath new life in your existing web filtering platforms and applications by switching your blacklist provider to Squidblacklist.org. Additionally we hope to empower new development of partnerships and solutions through our innovations. We intend to keep evolving our technology to continue pushing forward, to pave a future in which poorly maintained blacklists are a thing of the past.
>
>
> Footnote: We would have much rather have co-operated with the operators of shalla and urlblacklist in a productive manner, but they opted to ignore our emails.
>
>
>
>
> Signed,
>
>
> Benjamin E. Nichols
> http://www.squidblacklist.org
Currently low on budget. What about the free lists, how effective are those compared to the paid lists? I appreciate, and understand that the reason for the fee is for list maintenance. I am merely a... bum. >
>
> Here we have analyzed the porn blacklists produced by both of these entities, and we have as a result, determined the exact number of dead domains actually exceeds the number of live domains in both blacklists. Dead domains defined as, expired, parked, redirected, or otherwise dead websites. And therefore, we have concluded that the maintainers of these lists are either incompetent or simply dont care.
> Since 2012, we we have come a long way. We have developed a unique domain name management and blacklist production system that allows us to produce and manage our blacklists in a way that allows us to bring you a higher level of quality unseen before. You need the resources and tools you can rely on. And if you are using Dansguardian or Squidguard, then you need consider the very real opportunity to breath new life in your existing web filtering platforms and applications by switching your blacklist provider to Squidblacklist.org. Additionally we hope to empower new development of partnerships and solutions through our innovations. We intend to keep evolving our technology to continue pushing forward, to pave a future in which poorly maintained blacklists are a thing of the past.
>
>
> Footnote: We would have much rather have co-operated with the operators of shalla and urlblacklist in a productive manner, but they opted to ignore our emails.
>
>
>
>
> Signed,
>
>
> Benjamin E. Nichols
> http://www.squidblacklist.org
Benjamin E. Nichols
Jul 20, 2017, 12:02:21 AM7/20/17
to e2gua...@googlegroups.com
> "Currently low on budget. What about the free lists, how effective are those compared to the paid lists? I appreciate, and understand that the reason for the fee is for list maintenance. I am merely a... bum."
>
that. And Squidblacklist.org was originally a free service, its in our
roots, and I would actually prefer to give the blacklists away, but
unfortunately free is not a sustainable business model.
Regardless, I still want to give something back to the community and I
strongly believe that the world should have a valid free option for the
community, which is why we work with closely with UT-1. Also sometimes a
white listed environment is the optimal choice for a higher degree of
control. In the future we will be expanding our free whitelist
offerings to include ported formats and a master whitelist, which will
enable easy deployment. And I think that we may well make this an
entirely new area of our website which should be free.
--
--
1-405-397-1360 - Call Anytime.
fide...@gmail.com
Aug 29, 2019, 10:51:00 AM8/29/19
to e2guardian
Hello Benjamin, I made a Paypal subscription last friday (6 days ago) and got no answer from several mails, mail ven...@tenaxsoft.com
thanks
Darren Critchley
Sep 3, 2019, 10:10:11 AM9/3/19
to e2guardian
I am guessing that you did not hear. It appears the Benjamin passed away back in March.
The site currently appears to be running on autopilot. We have tried for months to contact someone to see if anyone is taking over. It appears not at this moment. For months the lists have been broken, the porn list in particular was truncated at some 20,000 records, it should have well over 1 million in it.
You need to contact Paypal immediately and try to get your payment back.
For anyone else that was using squidblacklists, what have you replaced them with?
shiv...@peplink.com
Sep 3, 2019, 10:00:45 PM9/3/19
to e2guardian
Hi All,
I was also using squidblacklist for content filtering and I'm now looking for a replacement as well.
UT-1 blacklists seem to be an option. https://dsi.ut-capitole.fr/blacklists/index_en.php
Does anyone know if openDNS provides its blacklists via a subscription ?
Thanks
Shivani
Darren Critchley
Sep 3, 2019, 11:50:49 PM9/3/19
to shiv...@peplink.com, e2guardian
OPenDNS was bought by Cisco and it is
not a blacklist based service. It is a DNS filter. I also do not
think it works for the small guy any more.
As for the UT-1 blacklists, sadly the porn part of their lists was actually from the squidblacklists, so there is not a lot of updates any more.
Hopefully someone steps up to the plate soon.
Darren
As for the UT-1 blacklists, sadly the porn part of their lists was actually from the squidblacklists, so there is not a lot of updates any more.
Hopefully someone steps up to the plate soon.
Darren
--
E2guardian:
https://groups.google.com/d/forum/e2guardian
Github:
https://github.com/e2guardian/e2guardian
Follow us on twitter:
https://twitter.com/e2guardian
---
You received this message because you are subscribed to the Google Groups "e2guardian" group.
To unsubscribe from this group and stop receiving emails from it, send an email to e2guardian+...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/e2guardian/51897653-f955-4a91-a1f1-f3f309ade69e%40googlegroups.com.
fide...@gmail.com
Sep 4, 2019, 12:41:03 AM9/4/19
to e2guardian
oh! so UT1 is not having updates?? I am using that one.
I compared http://squidguard.mesd.k12.or.us/blacklists.tgz and UT-1
Tested about 100 domains that where not in common in the lists. My conclusion is that MESD has a lot more dead domains and many useless IPs.
UT-1 only domains where dead many times but about 1/3 where real sites MESD was missing. I only found one case where MESD had a porn site that UT-1 had not.
I talked with https://urlfilterdb.com . Their list is encrypted and available only for their solution ufdbGuard for Squid.
Shalla porn domains quantity seems too small, about 70% of the others but I will test it anyway. Has anyone made a real compare?
Tested about 100 domains that where not in common in the lists. My conclusion is that MESD has a lot more dead domains and many useless IPs.
UT-1 only domains where dead many times but about 1/3 where real sites MESD was missing. I only found one case where MESD had a porn site that UT-1 had not.
I talked with https://urlfilterdb.com . Their list is encrypted and available only for their solution ufdbGuard for Squid.
Shalla porn domains quantity seems too small, about 70% of the others but I will test it anyway. Has anyone made a real compare?
Darren Critchley
Sep 4, 2019, 10:04:30 AM9/4/19
to fide...@gmail.com, e2guardian
No, I did not say that UT-1 is not
having updates. It is definitely updated every few days, but the
porn section (if you are running in a school) is not updated as
often as squidblacklist.org where.
I see that squidblacklist.org is truncating lists again:
https://www.squidblacklist.org/downloads.html
The porn database should have over 1 million entries and right now has 86077
Shallalist is dead, has been since 2016. Go into their downloads and look at the update dates and sizes of files. No changes since 2016
The http://squidguard.mesd.k12.or.us/blacklists.tgz is out of date by years, uses the UT-1 as their source and here is a note from their site.
This page has not been updated in many years, but I'll leave it up as the information may still be useful. We don't use squidGuard much these days, but we still use the blacklists. The definitive place to fetch the blacklists is: http://dsi.ut-capitole.fr/blacklists/index_en.php
I did find some open source lists, but the frequency of the updates is not nearly good enough to be considered.
So at this point, the only list still working is UT-1 which is what we are moving to, we have no option. I have set up a script before I went on vacation to monitor the changes in the lists day to day and I do see some changes in the lists.
The only other viable option is to change the filter you are using to a DNS filter.
Benjamin was very good at what he did and provided some excellent lists. Unfortunately he was a one man show and there is no sign of anyone taking over. His site is registered for years. I do not know how long his domain hosting will be around, I suspect when the bill is not paid, that the website will disappear.
Did you ask https://urlfilterdb.com if they would sell you an unencrypted list?
It's funny, we have all been to the same places. For us, this is the second black list supplier to disappear in about four years or less. We where with urlblacklists for a few years, and then moved over to squidblacklist.
Regards
Darren
I see that squidblacklist.org is truncating lists again:
https://www.squidblacklist.org/downloads.html
The porn database should have over 1 million entries and right now has 86077
Shallalist is dead, has been since 2016. Go into their downloads and look at the update dates and sizes of files. No changes since 2016
The http://squidguard.mesd.k12.or.us/blacklists.tgz is out of date by years, uses the UT-1 as their source and here is a note from their site.
This page has not been updated in many years, but I'll leave it up as the information may still be useful. We don't use squidGuard much these days, but we still use the blacklists. The definitive place to fetch the blacklists is: http://dsi.ut-capitole.fr/blacklists/index_en.php
I did find some open source lists, but the frequency of the updates is not nearly good enough to be considered.
So at this point, the only list still working is UT-1 which is what we are moving to, we have no option. I have set up a script before I went on vacation to monitor the changes in the lists day to day and I do see some changes in the lists.
The only other viable option is to change the filter you are using to a DNS filter.
Benjamin was very good at what he did and provided some excellent lists. Unfortunately he was a one man show and there is no sign of anyone taking over. His site is registered for years. I do not know how long his domain hosting will be around, I suspect when the bill is not paid, that the website will disappear.
Did you ask https://urlfilterdb.com if they would sell you an unencrypted list?
It's funny, we have all been to the same places. For us, this is the second black list supplier to disappear in about four years or less. We where with urlblacklists for a few years, and then moved over to squidblacklist.
Regards
Darren
--
E2guardian:
https://groups.google.com/d/forum/e2guardian
Github:
https://github.com/e2guardian/e2guardian
Follow us on twitter:
https://twitter.com/e2guardian
---
You received this message because you are subscribed to the Google Groups "e2guardian" group.
To unsubscribe from this group and stop receiving emails from it, send an email to e2guardian+...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/e2guardian/a679d9e7-50ac-45f8-b496-02a49cb09c12%40googlegroups.com.
fide...@gmail.com
Sep 4, 2019, 12:27:37 PM9/4/19
to e2guardian
very good data, thanks.
Looking at yerterday shalla list I see the porn list was updated the day before, september 2th (I use D/M/Y):
I found this lists site, do you know it?
2,154,345 domains updated yesterday
Darren Critchley
Sep 8, 2019, 11:00:01 AM9/8/19
to fide...@gmail.com, e2guardian
Download another and look. Compare file
sizes from day to day and you will see that they are not changing.
Yes the date is changing, because they are automatically
generated.
I was not aware of https://blocklist.site/app/index.php
But look at the "Last Updated" on each list. The only list that is close to being updated is the Ads one "Last updated: 08/21/19"
The majority of the lists have not been updated since February or March. Even the sponsored ones.
Regards
Darren
I was not aware of https://blocklist.site/app/index.php
But look at the "Last Updated" on each list. The only list that is close to being updated is the Ads one "Last updated: 08/21/19"
The majority of the lists have not been updated since February or March. Even the sponsored ones.
Regards
Darren
--
E2guardian:
https://groups.google.com/d/forum/e2guardian
Github:
https://github.com/e2guardian/e2guardian
Follow us on twitter:
https://twitter.com/e2guardian
---
You received this message because you are subscribed to the Google Groups "e2guardian" group.
To unsubscribe from this group and stop receiving emails from it, send an email to e2guardian+...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/e2guardian/ccb1f4c3-113c-4341-a402-6f30f6129e9f%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages