Syncing unsubscribes from Mailchimp to our company db

[chrisb]

We'd like to keep our company db in sync with Mailchimp, specifically
ensuring that unsubscribes get reflected in our customer records. Our
situation is perhaps unusual in that we create a new Mailchimp list
every time we do a mass mail, so we need to ensure we process
unsubscribes before the next list gets created.

Does this approach look sound? It seems to work from my quick tests,
but I'm a bit hazy on Mailchimp terminology (lists vs campaigns) so
would appreciate comments.


My planned approach:

Every hour (or other configured period), sync the latest unsubscribes
down from Mailchimp to keep our company db up to date.

1. Get lists:
-------------

First need to get all lists defined on our account

Request:

http://api.mailchimp.com/1.2/?method=lists&apikey=xxxxxxx&output=xml

(Need to use correct apikey for production account)

Output:

<MCAPI type="array">
<struct key="0" type="array">
<id type="string">12345678</id>
<web_id type="integer">106793</web_id>
<name type="string">Chris test list</name>
<date_created type="string">2009-06-02 14:03:58</date_created>
<member_count type="double">3</member_count>
<unsubscribe_count type="double">1</unsubscribe_count>
<cleaned_count type="double">0</cleaned_count>
</struct>
</MCAPI>

2. Get unsubscribes for each list:
----------------------------------

Loop through all lists returned, getting the unsubscribes for each one
since the last run. Mark each user as 'opted out' in our company db.

Request (for each list):


http://api.mailchimp.com/1.2/?method=listMembers&apikey=xxxxxxx&id=12345678&since=2009-06-01+09%3A00%3A00&status=unsubscribed&limit=15000&output=xml

(Need to use the correct apikey; also populate 'since' with the last
run time; and populate 'id' with each list's id as we loop through
lists from the previous request)

Outputs:

<MCAPI type="array">
<struct key="0" type="array">
<email type="string">chr...@example.com</email>
<timestamp type="string">2009-06-02 14:16:51</timestamp>
</struct>
</MCAPI>

For each such user, mark them as unsubscribed in our company db

Et voila!

TIA,

Chris

[obfuscode]

To the core of your question, why do you create a new list for every
mass mail?

> http://api.mailchimp.com/1.2/?method=listMembers&apikey=xxxxxxx&id=12...

>
>         (Need to use the correct apikey; also populate 'since' with the last
> run time; and populate 'id' with each list's id as we loop through
> lists from the previous request)
>
> Outputs:
>
>         <MCAPI type="array">
>         <struct key="0" type="array">

>         <email type="string">chri...@example.com</email>

[chrisb]

I'd have to ask our marketing guys, but a couple of reasons I can
think of:

- they try to target each mailing quite specifically by data mining
our db (we have quite a lot of customer data at our disposal). The
criteria may be different for each mailing.

- customers may have opted out by logging in to our website and
changing their settings. This marks them as 'no mass mailing' in our
own db but doesn't sync that data up to Mailchimp.

[obfuscode]

Well from what you describe there are a few things that could make
your life infinitely easier...

a) Use interest groups to segment your "master list" into groups you
can target specific marketing campaigns at
http://www.mailchimp.com/blog/sending-targeted-emails-with-interest-groups/

b) Sync your local DB to the mailchimp one for unsubscribes instead of
the other way around

c) Brush up on campaigns... lists are just that, a list of email
addresses and other information, campaigns are extremely powerful ways
of sending to that list. You can send to a subset of the list (using
interest groups), you can do A/B split testing if you want to see what
kind of information generates the best response and you can track
statistics per campaign. Which is much harder to do when you're
creating new lists.

But in the event you want to continue with your current method, it
looks like it'll work fine although I'm curious what language/system
you're using to do your calls as the XML method seems much more
difficult than doing direct API calls with the MCAPI code (PHP or
equivalent)?

[chrisb]

Thanks for the advice.

a, c) Yes - these both sound like good ideas, and more useful than
creating a new list each time. I'll talk to the marketing guys about
this. It'd involve more development work though, since we'd want to
manage interest groups programatically based on the results of the
data mining.

b) Maybe - although then the 'unsubscribe' link at the bottom of our
mails would have to point to our own server. We'd have to think a
little bit about security and making sure it's easy for Marketing to
include the link in the correct format.

As for our technology, it's a Java app. There doesn't seem to be a
Mailchimp API, but making http(s) requests and parsing an XML response
is not difficult (we'll use Jersey or the Apache Commons Http client
to make the request and JAXB to unmarshall the XML).

[obfuscode]

I don't have any experience with the XML transactions with MailChimp
but if you can do anything you can do with the standard API then you
can do a dual sync. Download the list of unsubscribes from the MC
system then loop through your local system updating MC with any unsubs
that weren't in the list you just downloaded.

For the interest groups, again if the XML can do the same as the API
then you can create campaigns and segment the interest groups
programatically. But it does look like you'll have quite a bit of
work ahead of you.

Good luck :D

[chrisb]

Yes, much fun to come. Thanks for your help.

[jesse]

Chris -
Just wanted to toss my 2 cents in here and add to the things
obfuscode mentioned, all of which are pretty much spot on.

As he suggested, if you can not create new lists everytime, life can
get *a lot* easier. In addition to segmenting on Interest Groups, you
can also segment on Merge Vars - that gives you up to 30 fields you
can populate. If you can work your marketing folks requirements into
that, you'll probably also make them very happy as you could either
sync that data nightly (and/or when users change data) so it's always
in the MailChimp system when they want to send new campaigns. That
will do 2 things they'll like:
* keep them from having to go through the list setup steps and
export/import new data
* give you more aggregate list stats

One thing to consider there is setting some fields to "hidden" in our
app so that users who hit their profile update page can't change some
fields.

There are some general examples of ways you might want to implement
that sync in #3 & #4 here:
http://www.mailchimp.com/api/examples/

It's kind of hidden at the bottom of #4 (need to rework that
documentation), but be sure to take a look at the meta redirect notes
there. That's particularly important because you are not going to be
able to remove our Unsub link and point them directly to your site.

Finally, like obfuscode said, even without doing that extra work, your
initial method for grabbing all unsubs will work - you could also
consider setting up Webhooks for your lists to automatically receive
that data - they should get copied over when you replicate lists,
which is what I assume your marketing folks do now.
http://www.mailchimp.com/api/webhooks/

jesse