IMPORTANT: Spark mailing lists moving to Apache by September 1st

[Matei Zaharia]

As part of Spark's recent move to Apache, we are planning to migrate the mailing lists to Apache infrastructure this month, so that the current Google groups will become read-only on September 1, 2013. To keep receiving updates about Spark or to participate in development discussions, please subscribe to the following lists:

Get help using Spark or contribute to the project on our mailing lists:

• us...@spark.incubator.apache.org — for usage questions, help, and announcements. (subscribe) (archives)
• d...@spark.incubator.apache.org — for people who want to contribute code to Spark. (subscribe) (archives)

Most users will probably want the User list, but individuals interested in contributing code to the project should also subscribe to the Dev list.

After September 1st, we'll try to make the Google Groups mirror the Apache list so you can still read updates on them, but it's not yet clear whether this will be possible. Sorry for the inconvenience and thanks for bearing with us.

Matei

[Marco Didonna]

Too bad ... apache ML have a terrible web interface 

[Evan Chan]

I'm not able to subscribe:

Delivery to the following recipient failed permanently:

     dev-sc.1374707752.eciikkjecnggoomplmjm-ev=ooyala.com@spark.incubator.apache.org

Technical details of permanent failure:
Google tried to deliver your message, but it was rejected by the server for the recipient domain spark.incubator.apache.org by mx1.eu.apache.org. [192.87.106.230].

The error that the other server returned was:
552 spam score (6.3) exceeded threshold (HTML_IMAGE_ONLY_16,HTML_MESSAGE,HTML_SHORT_LINK_IMG_2,RCVD_IN_DNSWL_LOW,SPF_PASS,URIBL_BLACK

----- Original message -----

DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=ooyala.com; s=google;
        h=mime-version:date:message-id:subject:from:to:content-type;
        bh=PrDuISf258qmaqMIfG/tRn4b1ct9rHBDov1H10W61e0=;
        b=C7junSPFbgsOomy9XZqcZlf+j7nPhRjJbDEskbi9uumVn5lnH/sZx7SRPfAD0xT1nH
         wSE0NU9Y6ou7BgVHe0jq385T1CtCrG2guK/XiOIp/rIdVNAfaxqmJc9Dkuo5t6Owmk7b
         bQjDpQ4L2EV/vnUoifvPz5yU70gJvjjmIjlb4=
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=google.com; s=20120113;
        h=mime-version:date:message-id:subject:from:to:content-type
         :x-gm-message-state;
        bh=PrDuISf258qmaqMIfG/tRn4b1ct9rHBDov1H10W61e0=;
        b=pQG74K0muZD/t15GOHUgFczFXhK1rFb2glkhgqg/Asup1/5viWurMv3XdPtGNLmS8D
         /A7a0XO2+MsfZoY1AbPQtlCgxWpv+EtPpYj8SeNfafZC4s59ShbVs8K90+d/Nx3Z6yqP
         nOWsJbFvDK17yBd9SkInEzEJeuFqW7sLASMUBdDLi3hcEz6NIT84PUUMp4fDy3lm0S7M
         eZK+451zZR7d4AiJLqfq4Pfs1joS/u9lhF7aQYv9efUliCf41QAkwCYtXUbZ0P307S80
         yyw6+Rra1ym9T1bjjZ3cUuJLu5qysg90UxNFiMQ7Xok7BrhkJ4YQJbXalQs9uAjRNTdW
         7nsg==

[Barry Kaplan]

This is a BIG downgrade.

[Ryan LeCompte]

Can you elaborate? 

On Aug 17, 2013, at 11:02 AM, Barry Kaplan <mem...@gmail.com> wrote:

This is a BIG downgrade.

--
You received this message because you are subscribed to the Google Groups "Spark Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spark-users...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Barry Kaplan]

On Saturday, August 17, 2013 11:39:06 AM UTC-7, Ryan LeCompte wrote:

Can you elaborate? 

Really? Its just a matter of usability. Or does the apache ml's have a nice interface like google groups? Some kind for forum system?

[Alex Boisvert]

Agree it's a bit of a downgrade ... but several 3rd-parties index Apache mailing lists.  My favorite is http://markmail.org/ that provides good browsing/search capabilities.  The main downside is that you can't easily post directly from there... so it requires community participants to subscribe to the mailing lists.

--

[Evan Chan]

I still can't post to d...@spark.incubator.apache.org:

In general though, besides the spam filtering, there are many ways that the apache list is much worse:

- No central site to go to to search messages

- Sending messages and searching is separated

- There may be some archival sites for messages, but you cannot respond to old messages or revive an old thread, unless you have subscribed to the mailing list

I can personally attest that I have never succeeded in emailing Apache projects consistently.   I doubt I'm the only one that is constantly frustrated by the Apache mailing lists.

Delivery to the following recipient failed permanently:

     d...@spark.incubator.apache.org

Technical details of permanent failure:

Google tried to deliver your message, but it was rejected by the server for the recipient domain spark.incubator.apache.org by mx1.us.apache.org. [140.211.11.136].

The error that the other server returned was:

552 spam score (6.0) exceeded threshold (HTML_IMAGE_ONLY_20,HTML_MESSAGE,HTML_SHORT_LINK_IMG_3,RCVD_IN_DNSWL_LOW,SPF_PASS,T_REMOTE_IMAGE,URIBL_BLACK

----- Original message -----

DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=ooyala.com; s=google;
        h=mime-version:date:message-id:subject:from:to:content-type;

        bh=IkzyCF7ywXdnashWGOvZuLnkbAd5dtEdcqDn/CW0Q/k=;
        b=RgiIBNAc0n1ZvlIuefMSe42UVf/XVvkkqDnQjsWl7BhsEsMTjHq8+XJmzmlf+pgU6L
         7REJNE+a68L4Whc2ks7pRz0n77TTED47QGM1qvqlozZjRcBfYXoOGh/V2ua6Q+qUffDg
         AbDW1X021re7mL9Q3FQhlPQFZxFJsd0ugiovQ=

X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=google.com; s=20120113;

        h=x-gm-message-state:mime-version:date:message-id:subject:from:to
         :content-type;
        bh=IkzyCF7ywXdnashWGOvZuLnkbAd5dtEdcqDn/CW0Q/k=;
        b=QrZhsysqbtoAUCRUIJ2M45www0rk7qZeVd1eAyOcpAOuFXekGXHQc7JbRPFHEPAKKz
         FhlVjMRd56BtSnYjU2rtpg0RrwReJJD4rKqe7wM3VNUW5rpGcQ8NKdSaRCBBPJuTTp+f
         aKwXMs2z4dHEkdeFL5HVi0/74+AnCWStbrdX9ER7FYPYKC+oRxSk5zHgYvdR6Cut3wWH
         UU9rZbfH7CSgAgmx/alahr9s5n0Y9uph4ATPfpvjcOItPadi9f/mcS+DWQyBy2KCyWiP
         t9hfzbTPyZJ5pbNkxxAS9UrFRMIsZ2TzJ8nL0L17h6cIlwFEMWjL5FiseoiIFXmDE+tW
         buIg==
X-Gm-Message-State: ALoCoQlAokryOEFN4hMwbAqogX0dnOV0IZnZM7YkGEiWYL9X78IGcQ51ysElBHsesPPVqd8boAH3
MIME-Version: 1.0
X-Received: by 10.152.19.1 with SMTP id a1mr13764224lae.8.1376953841136; Mon,
 19 Aug 2013 16:10:41 -0700 (PDT)
Received: by 10.112.35.131 with HTTP; Mon, 19 Aug 2013 16:10:41 -0700 (PDT)
Date: Mon, 19 Aug 2013 16:10:41 -0700
Message-ID: <CADWPM3gPTWxEJW9N=aV9Gp-kvtNzBiNg6q=7o7LSuJLKX0QSDg@mail.gmail.com>
Subject: Moving to Scala 2.10
From: Evan Chan <e...@ooyala.com>
To: "d...@spark.incubator.apache.org" <d...@spark.incubator.apache.org>
Content-Type: multipart/alternative; boundary=089e01493e4292c51304e4550c9c

<http://www.ooyala.com/>
<http://www.facebook.com/ooyala><http://www.linkedin.com/company/ooyala><http://www.twitter.com/ooyala>

On Saturday, August 17, 2013 11:39:06 AM UTC-7, Ryan LeCompte wrote:

[Matei Zaharia]

Evan, did you subscribe to the list by emailing dev-su...@incubator.apache.org?

It also seems to dislike HTML emails.. maybe you can set it to email this in plain text.

Unfortunately I don't believe we have an option to use another type of list because Apache wants to own its infrastructure. One thing that may work is mirroring the messages onto this one, but I'll have to look into it.

Matei

You received this message because you are subscribed to the Google Groups "Spark Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spark-develope...@googlegroups.com.

[Andy Konwinski]

Or could we just subscribe that list to this one so that all of the messages are mirrored onto it?

Andy

[Evan Chan]

The plain-text mode was what did it. Looks like HTML emails just sets
off the spam detector.

Yes, if the google list could somehow mirror the apache one, and if
replies to the google list automatically go to the apache one, that
would make things work really well.

thanks!
Evan

On Mon, Aug 19, 2013 at 10:10 PM, Matei Zaharia <matei....@gmail.com> wrote:

> <CADWPM3gPTWxEJW9N=aV9Gp-kvtNzBiNg6q=7o7LSuJ...@mail.gmail.com>

> Subject: Moving to Scala 2.10
> From: Evan Chan <e...@ooyala.com>
> To: "d...@spark.incubator.apache.org" <d...@spark.incubator.apache.org>
> Content-Type: multipart/alternative; boundary=089e01493e4292c51304e4550c9c
> <http://www.ooyala.com/>
> <http://www.facebook.com/ooyala><http://www.linkedin.com/company/ooyala><http://www.twitter.com/ooyala>
>
> On Saturday, August 17, 2013 11:39:06 AM UTC-7, Ryan LeCompte wrote:
>>
>> Can you elaborate?
>>
>> On Aug 17, 2013, at 11:02 AM, Barry Kaplan <mem...@gmail.com> wrote:
>>
>> This is a BIG downgrade.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Spark Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to spark-users...@googlegroups.com.
>> For more options, visit https://groups.google.com/groups/opt_out.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Spark Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to spark-develope...@googlegroups.com.
>
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
> --

> You received this message because you are subscribed to a topic in the

> Google Groups "Spark Developers" group.

> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/spark-developers/vtg-5db8JWY/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to

> spark-develope...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

--
--
Evan Chan
Staff Engineer
e...@ooyala.com |

[Henry Saputra]

From http://www.apache.org/foundation/mailinglists.html looks like the spam filter will reject HTML-formatted emails than the plain ones.

- Henry

[Victor Hooi]

Hi,

I have to say I also prefer the Google Groups interface as well...

Cheers,

Victor

Message-ID: <CADWPM3gPTWxEJW9N=aV9Gp-kvtNzBiNg6q=7o7LSuJ...@mail.gmail.com>

[Brenton Partridge]

Any updates on which list is the correct one to be using? I'm seeing new activity on both lists, including the summit announcement which was posted here after the read-only "deadline" on September 1. I'm new to Spark (but planning on using it on a 70TB dataset in academia) so I would really appreciate knowing where the community will actually be posting!

Thanks,

Brenton

[Andy Konwinski]

The official lists are definitely the ones listed on the website at http://spark.incubator.apache.org/mailing-lists.html -- namely:

us...@spark.incubator.apache.org and d...@spark.incubator.apache.org

Some of us have still cc the old list just out of habit but we should probably knock that off since the move has officially happened.

Andy

--

You received this message because you are subscribed to the Google Groups "Spark Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spark-users...@googlegroups.com.

[Matei Zaharia]

Yup, though we haven’t shut down the old user list yet, we strongly encourage people to move over. Maybe we can pick a date like Jan 1st to shut it down. Unfortunately it wasn’t possible to reliably mirror messages from here on the Apache list.

Matei

[Mike Potts]

I notice that there are still a lot of active topics in this group: and also activity on the apache mailing list (which is a really horrible experience!).  Is it a firm policy on apache's front to disallow external groups?  I'm going to be ramping up on spark, and I really hate the idea of having to rely on the apache archives and my mail client.  Also: having to search for topics/keywords both in old threads (here) as well as new threads in apache's (clunky) archive, is going to be a pain!  I almost feel like I must be missing something because the current solution seems unfeasibly awkward!

[Andy Konwinski]

Hey Mike,

As you probably noticed when you CC'd spark-de...@googlegroups.com, that list has already be reconfigured so that it no longer allows posting (and bounces emails sent to it).

We will be doing the same thing to the spark...@googlegroups.com list too (we'll announce a date for that soon).

That may sound very frustrating, and you are *not* alone feeling that way. We've had a long conversation with our mentors about this, and I've felt very similar to you, so I'd like to give you background.

As I'm coming to see it, part of becoming an Apache project is moving the community *fully* over to Apache infrastructure, and more generally the Apache way of organizing the community.

This applies in both the nuts-and-bolts sense of being on apache infra, but possibly more importantly, it is also a guiding principle and way of thinking.

In various ways, moving to apache Infra can be a painful process, and IMO the loss of all the great mailing list functionality that comes with using Google Groups is perhaps the most painful step. But basically, the de facto mailing lists need to be the Apache ones, and not Google Groups. The underlying reason is that Apache needs to take full accountability for recording and publishing the mailing lists, it has to be able to institutionally guarantee this. This is because discussion on mailing lists is one of the core things that defines an Apache community. So at a minimum this means Apache owning the master copy of the bits.

All that said, we are discussing the possibility of having a google group that subscribes to each list that would provide an easier to use and prettier archive for each list (so far we haven't gotten that to work).

I hope this was helpful. It has taken me a few years now, and a lot of conversations with experienced (and patient!) Apache mentors, to internalize some of the nuance about "the Apache way". That's why I wanted to share.

Andy

On Thu, Dec 19, 2013 at 6:28 PM, Mike Potts <masp...@gmail.com> wrote:

I notice that there are still a lot of active topics in this group: and also activity on the apache mailing list (which is a really horrible experience!).  Is it a firm policy on apache's front to disallow external groups?  I'm going to be ramping up on spark, and I really hate the idea of having to rely on the apache archives and my mail client.  Also: having to search for topics/keywords both in old threads (here) as well as new threads in apache's (clunky) archive, is going to be a pain!  I almost feel like I must be missing something because the current solution seems unfeasibly awkward!

[Mike Potts]

Thanks very much for the prompt and comprehensive reply!  I appreciate the overarching desire to integrate with apache: I'm very happy to hear that there's a move to use the existing groups as mirrors: that will overcome all of my objections: particularly if it's bidirectional! :)

[Nick Pentreath]

One option that is 3rd party that works nicely for the Hadoop project and it's related projects is http://search-hadoop.com - managed by sematext. Perhaps we can plead with Otis to add Spark lists to search-spark.com, or the existing site?

Just throwing it out there as a potential solution to at least searching and navigating the Apache lists

Sent from my iPad

> On 20 Dec 2013, at 6:46 AM, Aaron Davidson <ilik...@gmail.com> wrote:
>
> I'd be fine with one-way mirrors here (Apache threads being reflected in
> Google groups) -- I have no idea how one is supposed to navigate the Apache
> list to look for historic threads.

[Andy Konwinski]

I've set up two new unofficial google groups to mirror the Apache Spark user and dev lists:

https://groups.google.com/forum/#!forum/apache-spark-dev-mirror

https://groups.google.com/forum/#!forum/apache-spark-user-mirror

Basically these lists each subscribe to the corresponding Apache list.

They do not allow folks to subscribe directly to them. Getting emails from the Google Group would offer no advantages that I can think of and we really want to encourage folks to sign up for the official mailing list instead.

The lists do allow the public to send email to them, which I think might be necessary since the "from:" field for all emails that get distributed via the Apache mailing list is set to the author of the email.

I think this might be a great compromise. At least we can try this out and see how it goes.

Matei, can you confirm that Jan 1 is the date we want to turn off the existing spark-users google group?

We could consider using the existing spark-developers and spark-users google groups instead of the two new ones I just created but I think that it is much more obvious to have the lists include the word mirror in their names.

The dev list mirror seems to be working, because I see the last couple emails from this thread in it already. I'll confirm and ensure that the user list mirror is working too.

Thoughts?

Andy

P.S. Thanks to Patrick for suggesting this to me originally.

On Thu, Dec 19, 2013 at 8:46 PM, Aaron Davidson <ilik...@gmail.com> wrote:

I'd be fine with one-way mirrors here (Apache threads being reflected in Google groups) -- I have no idea how one is supposed to navigate the Apache list to look for historic threads.

[Matei Zaharia]

Yes, I agree that we should close down the existing Google group on Jan 1st. While it’s more convenient to use, it’s created confusion. I hope that we can get the ASF to support better search interfaces in the future too. I think we just have to drive this from within.

The Google Group should be a nice way to make the content searchable from the web. We should also see what it takes to make it mirrored on Nabble (http://www.nabble.com). I’ve found a lot of information about other projects there, and other Apache projects do use it.

Matei

[Mike Potts]

This is great!  Could the new *-mirror groups start off with a complete copy of the (closed) original groups and the apache lists?  (So as to avoid having to search 3 different sources for historical information.)

[Andy Konwinski]

That would be really awesome. I'm not familiar with any Google Groups functionality that supports that but I'll look.

That's an argument for maybe just changing the names of the existing groups to something with mirror in them instead of using newly created ones.

On Dec 20, 2013 8:18 AM, "Mike Potts" <masp...@gmail.com> wrote:

This is great!  Could the new *-mirror groups start off with a complete copy of the (closed) original groups and the apache lists?  (So as to avoid having to search 3 different sources for historical information.)

--

[Mike Potts]

I actually prefer that, but I didn't want my preference to get in the way of creating mirror groups, one way or the other :)  (My argument would be that since the old groups would be closing anyway, re-purposing them as mirrors is fair use: and less work/confusing than creating new *-mirror groups instead.)

[Patrick Wendell]

Andy and Mike,

I'd also prefer to just convert the old groups into mirrors. That way
people who are still subscribed to them will continue to get e-mails
(and most people on the list are read-only users).

Ideally we'd have the behavior that users who try to e-mail the google
group get a bounce back saying "this is now a read only mirror".

That said I have *no idea* of this is possible to set-up nicely within
google groups. I defer to Andy! Having the new mirror groups also
seems like a decent solution as well...

- Patrick

[Andy Konwinski]

Per Matei's suggestion, I've set up two nabble archive lists, one to archive the apache dev list and one to archive the apache user list.

user list archive: http://apache-spark-user-list.1001560.n3.nabble.com

dev list archive: http://apache-spark-developers-list.1001551.n3.nabble.com

Between these and whatever solution we end up with for the google group mirrors, we should have decent enough alternatives to reading via the apache list archives going forward.

[Mike Potts]

Thanks!!  Is is straightforward to pre-populate it with all the existing threads from apache?

[Andy Konwinski]

Unfortunately, I don't see that as an option for either these nabble lists or the google groups.

For the google groups, we we could subscribe the old spark google groups to the apache lists (instead of using the new mirror groups I created).

However the tradeoff is that as far as I can tell, we wouldn't be able to have posting disabled or even subscriber only for those groups (since the emails from the apache lists would then get bounced). In this case, I'm concerned that if it remains possible to post to the google groups, then people will continue posting to them, which would be confusing to everybody.

We could rename the old groups to make it clear they are the mirrors, however then the people who are subscribed to them and have rules setup for dealing with them would have to change their rules.

Finally, we could unsubscribe everybody from the old google groups and then rename them to have the word mirror in them which would preserve history and be no worse than if they just became silent and activity moved to the new mirror groups i created (which as I mentioned, must allow posting by anybody). So I guess this last option is what I propose we do.

Andy

On Sun, Dec 22, 2013 at 3:51 PM, Mike Potts <masp...@gmail.com> wrote:

Thanks!!  Is is straightforward to pre-populate it with all the existing threads from apache?

[Mike Potts]

Just a thought: if the mirroring was bidirectional (between apache lists and google groups) -- meaning that you could read/write via either channel and everything would be mirrored in both groups -- would that be a bad thing?  The apache lists would be comprehensive and available as a first-class communications channel; and so would the google groups: I would have thought the benefits of maintaining both channels (fully mirrored) would outweigh any drawbacks... but again I'm probably missing something :)

[Andy Konwinski]

It would be a good thing, I'm just not sure how to achieve it, or if it's possible. AFAIK, you cannot simply subscribe the lists to each other. Have you heard of a setup like this being used before?

On Dec 22, 2013 7:12 PM, "Mike Potts" <masp...@gmail.com> wrote:

Just a thought: if the mirroring was bidirectional (between apache lists and google groups) -- meaning that you could read/write via either channel and everything would be mirrored in both groups -- would that be a bad thing?  The apache lists would be comprehensive and available as a first-class communications channel; and so would the google groups: I would have thought the benefits of maintaining both channels (fully mirrored) would outweigh any drawbacks... but again I'm probably missing something :)

[Mike Potts]

Haha, no: I’m afraid there my reach exceeds my grasp :(

> You received this message because you are subscribed to a topic in the Google Groups "Spark Users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/spark-users/vtg-5db8JWY/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to spark-users...@googlegroups.com.

[Evan Chan]

Thanks Andy, at first glance nabble seems great, it allows search plus posting new topics, so it appears to be bidirectional.    Now just have to register an account on there.

--

--

Evan Chan

Staff Engineer
e...@ooyala.com  | 

[Patrick Wendell]

Hey Andy - these Nabble groups look great! Thanks for setting them up.