413 FULL head error on Streaming API

308 views
Skip to first unread message

Julien

unread,
Nov 12, 2009, 2:49:08 AM11/12/09
to Twitter Development Talk
Hi,

I work for Superfeedr. We do feed parsing on-demand. We don't care
what feeds people are giving us to fetch. As a matter of fact, we
don't even care about the content. Yet, some users have given us RSS/
Atom feeds from Twitter to fetch.

We bumped into the polling limits a few months ago : we asked to be
white-listed and "gained" a few month. We now have about 15k Twitter
feeds, and polling is not suitable anymore. We implemented your
streaming API to make everybody happy.
Except that it deosn't work. : we get "413 FULL head" errors. I asked
Ryan Sarver for help, he granted us a "shadow" role, which is really
not what we need.

What can we do?

Thanks

John Kalucki

unread,
Nov 12, 2009, 9:39:41 AM11/12/09
to Twitter Development Talk
413 usually means "too long" on the Streaming API. Too many
predicates, or perhaps a URL of crazy length. This is documented in
the wiki.

First, be sure that you are using a POST parameter and not encoding
your predicates in the URL. Second, look at the text message that is
returned with the 413. It will tell you what you are doing wrong. Most
likely, you are attempting to follow more users than the Shadow role
allows.

Detail your use case, and perhaps then we can give some advice on the
availability of the data and, if available, the most practical way to
obtain the data. We do not make every arbitrary materialization of the
data available.

-John Kalucki
http://twitter.com/jkalucki
Services, Twitter Inc.

Julien

unread,
Nov 12, 2009, 11:16:12 AM11/12/09
to Twitter Development Talk
Thanks John, I'll review all that and we'll post more info soon.

Thanks for listening!

On Nov 12, 6:39 am, John Kalucki <jkalu...@gmail.com> wrote:
> 413 usually means "too long" on the Streaming API. Too many
> predicates, or perhaps a URL of crazy length. This is documented in
> the wiki.
>
> First, be sure that you are using a POST parameter and not encoding
> your predicates in the URL. Second, look at the text message that is
> returned with the 413. It will tell you what you are doing wrong. Most
> likely, you are attempting to follow more users than the Shadow role
> allows.
>
> Detail your use case, and perhaps then we can give some advice on the
> availability of the data and, if available, the most practical way to
> obtain the data. We do not make every arbitrary materialization of the
> data available.
>
> -John Kaluckihttp://twitter.com/jkalucki

Julien

unread,
Nov 12, 2009, 12:35:18 PM11/12/09
to Twitter Development Talk
For some reason, my previous post didn't show up :/
So, we don't get the error when we stay in the limitations of the
default access level (200 track and 400 userids).

If we go beyond that, we get : HTTP/1.1 413 Request Entity Too Large

Could it be that our username hasn't been approved despite what the
interface says? Our username is "superfeedr_trac"

Thanks,

Julien

Julien

unread,
Nov 12, 2009, 12:21:59 PM11/12/09
to Twitter Development Talk
John,

This is exactly what we post to your servers (I just hid the
Authorization) :

POST /1/statuses/filter.json HTTP/1.1
Host: stream.twitter.com
User-agent: TwitterStream
Authorization: Basic XXXXX==
Content-type: application/x-www-form-urlencoded
Content-length: 13609

follow=80706782,80706249,59397117,813276,1750351,1497,14303772,15640990,10630,1581511,794545,12,16718628,15351161,55033131,22207903,956501,16028823,982721,14100016,4411621,14615776,648563,26383279,21161414,10239,811350,9843812,16540939,20,25806143,15889416,10128922,13955972,31953,7337062,32653,1042361,26020994,1714051,7544012,309073,32063,14098218,33205299,1392281,664153,15328268,6717392,611823,15913,21169368,6892002,731253,7281962,14956807,5469512,13885092,36502008,15503041,5381582,26566923,2876271,14083455,18029701,1091741,8256162,34174320,16418101,17544169,22784949,5529162,813286,7136992,47685641,20935486,15864599,14712874,469163,14134376,13370272,1501,5973812,15133601,24338507,14454247,1748581,12858,13331942,1799511,10425502,24103,428333,10365,15279944,14229661,27663774,14692721,10226672,14363100,25694882,9536542,14418757,14114731,11900,15352135,18399826,22449663,7889672,25963448,5563012,66713497,1786041,19643660,17919393,14090674,14956888,25808898,59194896,2048141,15111776,16426292,45733,20201051,65973559,17781981,52680804,64409399,1367531,31132926,16450330,23853857,14075928,43856654,28156155,40197535,15773675,41112544,28174228,20526944,63407440,22802398,67899674,6017542,18120198,52589720,24855923,17829758,33233205,15945424,15954704,45075974,44888804,17409240,22797985,14323759,26072066,614133,11881852,21655440,37879306,1652541,6368672,29925623,31353077,6821102,15876379,14634720,37405382,15838599,14790192,711303,23753940,59726910,17819913,18866407,17113231,17124895,1947301,5741722,27456531,17171111,42033487,11856032,53372599,11857072,31114028,15996830,41075473,65904217,759251,23981347,14516048,27500565,16890969,16970248,15880642,16129920,42402819,45369197,31329385,5402612,13838562,14803917,8468872,14372478,65632985,14292132,80427603,38400130,22910295,77877070,48840381,3216921,612473,80713872,42056836,20153343,1705201,20924090,81391288,19438765,104203,14089573,4700381,7400702,33684457,37874853,14496232,7235522,7411552,71508034,15072502,33584794,15237935,81025521,22297824,7692692,18824787,18949452,7161542,22301030,27565488,14439930,1975321,1830491,46376581,19945693,23235489,14239496,3850041,22021097,43417156,12942,14162945,18159833,13827442,21645304,15708435,794928,15572963,11894422,456413,5579402,5990942,20399599,16297742,8085962,38961973,20631151,41375788,16416613,60334210,16705726,14305706,3128561,6893672,2860101,19039063,63810996,22651121,42860263,937961,50270097,36951816,1186,14613750,18799530,12974922,22543092,18991180,15251361,7408842,56458858,23530011,5567,48403197,586,15204168,17028306,2318591,20389644,7314182,14704003,14438790,42096373,45891150,15237382,60098511,27096075,19059514,14438014,4234581,15097328,50751569,246,54911372,770289,44011351,20194429,20365897,22540123,24203163,25489250,15533871,14166131,35136170,80766621,15915632,11268812,14768955,16901185,15888142,20132337,45537669,754978,19562029,32552821,33292459,14629669,14740219,1922861,4432431,19329393,16629431,31237480,41167154,7668952,16031518,28329367,81912095,26907579,15269523,15227791,15234407,41017380,21925979,21026837,11490952,16651978,15744222,17382456,14781982,22836661,22193651,19658936,24729040,14491138,17461978,44371159,32838392,36078825,3222731,24007848,29678507,15315785,22280679,23970108,41380057,34574635,42009631,59129082,35538270,66357651,66538952,38336343,21899223,25667617,78859670,67601095,67768214,14301002,68747982,16271343,16270679,19594811,40203798,22757239,46594911,80293100,13900252,23832022,19052717,71315517,78823450,33855080,43990220,17990812,45806824,9141622,27104736,32706664,39988586,16381061,20258912,74157758,52053604,74227685,43351209,76671588,59415820,21829544,37388407,15224126,26227858,18248645,38689604,65185176,21611204,22147157,14582043,13180972,47326062,61516762,33461621,14293310,76728008,72937694,14638581,69043094,34805346,16191323,75689974,30072246,45827487,75347542,74425788,80291874,80271800,71601293,51650188,56746527,18477121,22253558,16004911,82080333,24952459,33367205,15066838,6107422,972651,52725585,43062502,46277588,7076492,51509917,51413735,14994303,21879024,19394186,19058681,15494380,19697415,47326510,23135963,5793822,8921852,17133441,14302335,29014787,15633653,22205534,15192712,36116095,25098243,23484039,36751886,16896485,32943325,5515802,46400077,24609329,51457672,43913584,51469543,43705420,43798591,43585102,51459322,33478756,33479453,33480681,33480845,14440647,19704522,16625255,17636405,17774041,40275680,14933478,69889578,54591680,55241169,53926207,15856061,56307587,47401736,41773608,69399502,18876589,55300646,48367849,54550364,20275150,29471630,34940839,16560657,23855299,9473532,21568952,28568411,20065061,2367911,25110374,17487702,17453292,38735244,37409663,14480405,60442463,2557521,16348255,19709389,18716535,10068172,20432797,19861054,14821352,15301503,16202520,26827762,28874360,36640797,34578472,45316559,37423356,35557028,50306597,10330772,26742335,19356277,55176469,60956213,48413801,8204342,56482694,15439395,20172983,82633117,54590165,62599860,15480278,82568633,61220731,62534573,62613431,15417909,14342883,1143031,16651779,14511951,76979058,85951718,30962015,39063364,43339470,23332237,25547701,14612371,14875897,43286678,68675334,9599642,68132270,55250737,41036546,69040373,15436933,27201512,17745929,16409683,20302398,816653,22199141,37764422,54682125,15426659,35865630,16886396,15728419,78934317,18124790,45172332,1340521,21143108,30973,7282592,16910058,41117241,31397466,14801539,42262856,48658417,84553621,48697966,48689560,19764472,16973333,48463419,67995848,796955,20381538,13058552,87105253,19402238,53550244,59929520,60224553,42005012,25038085,26934903,66156073,40928638,39442498,39442877,36018499,803468,20574484,19643352,50539213,21586418,22745779,24291858,48651329,48654029,7466622,11778,86391789,32642765,27324578,23940534,88672023,86763679,17208949,16026621,89294978,76847458,21683908&track=remedie,caldeiraodohuck,twitpic,fantastico,carasebocas,Ultima,qipit,globoreporter,qlikview,pequelia,lucianohuck,twitter,tvglobo,domingaodofaustao,gappproxy,globo,protip,musasdobrasileirao,iphone,livetechdocs,globoesporte,clothingswap,Zilok,jornalnacional,jeffboudier,harry,tvrecord,blogestudio,termtter,palin,rederecord,pubsubhubbub,bfrench,caminhodasindias,jornaldarecord,chrome,webhooks,track,illinois,atkins,liveink,hawaii,harper3,glenn,jackson,mexico2,threadless,iyermatter,steelers,fukung,ubuntu,kevron,essec,sentiment,santa,copepodo,eagles,Track,twitterapi,asdfghjklqwertyuiop,yankees,banana,sounders,awesome,bucketworks,gmail,twollar,tokbox,sushubh,ipolitics,briandorsey,graph,massage,cardinals,
99acres,vuelta,firefox,Manchester,tracking,Lithuania,adwordsfr,
0eRPG,Remove,lazio,drupal,Malawi,steve,amazing,waynesutton,djangocon,opengov,downtown,liberzon,kansas,voice,notre,michael,digitalstrategy,illini,makaan,ihatewhengirlssay,broadbandforum,pancheros,linux,juliensharp,exclaimtrack,html5,feltham,indivibes,sleep,liverpoolinternets,breakingsfnews,paychex,michigan,crowdcompass,twitteroauth,haynes,linkedprocess,sushraju,
912dc,museum,greasemonkey,affiliate,milwaukee,maine,excla,dotclear,techcrunch,karen26,genedelibero,nyjets,Museum,remove,dc912,crowd,adwords,arturogoga,football,zeroidiots,eurobasket,amyloo,magicbricks,entangledstate,iwish,israel,techwhack,TRACK,aberystwyth,twollars,patientsknowbest,sleepfm,wcr09,chiefs,earthquake,motogp,siliconindia,sirius,Fallen,augmentedreality,ashlawn,startup,gtanuel,nickreynolds,pocketsmith,perth,hello,hiring,mattwarren,There,showrss,sitespect,skynewstopstory,voxeo,texas,stanford,singapore,greetz,ericrygh,coimbra,youaroevisual,bunny,huffington,wigsgiw,ivivek,information,jifdjis90,snobbyslice,hackday,juliansharp,galactica,socrates,citibank,freemium,nebraska,bankoti,android,stevegillmor,agraquest,python,ericjhansen,packers,hangzhou,skip1,china,mahtweets,ninyou,springs,weekend,xhtmlized,xerox,phillies,wordpress,maemo,ThereDotCom,dropbox,eyelite,leanne,breaking,nokia,kunming,website,rabbit,reddit,garyvee,sonyou,Metro,radarla,alexandra,money,leipzig,benseese,geophysics,brisbane,psd2html,green,janet,malibu,montessori,monkeysplat,blackberry,usopen,nerduo,youaroenvisual,karen,socialmedia,Glenn,pygame,dspattison,beagleboard,tracks,jetairways,zipcar,hosting,textwise,Tolkien,table,memcache,bobov,adamac,django,tomad,exclaim,kiryas,encryption,joshnunn,kayne,finance,pinax,india,bentsai,franklin,eageles,gluten,hayashih,skype,urasandesu,apple,realtor,dropio,miduhima,yamashiro,secude,vdhguy,security,jjudge,memcached,social,jaredwsmith,techcrunch50,google,swayze,textme,nevis,Hahn_Alex,satmar,state,savedforlater,angels,icici,grails,Taylor,ihone,aberfoyle,melon,worstpirate,yuiseki,XCode,boston,webhosting,iPhone,templeton,appengine,dougbarth,sociable,yedagandhi,remover,Comptia,scotland,csavage,email,ongeval,yeshiva,strasbourg,kularika,harvard,github,whitmore,akiba,monsey,meanjoepaterno,melontech,userstorylab,hashlove,search,exposur3,googleappengine,wipro,bluecigs,omnilife,vgtweetup,johnpwood,warenosyo,digicorp,facebook,tacos,Virgin,salesforce,soo_mei,cotapata,astadia,foodjammers,nstein,maako,l2wireless,mygirlbetty,european,jeffrey,ACRON,oracle,omniture,georg,barroso,inetium,nacin,nikolic,product,mpilgrim,brendadronkers,ghana,kathryn,mobile,kersey,kevin,brenda,trend,ansbach,economy,tombloo,thril,odesk,hatzolah,District,puzzle,jackass,japan,corus,romangeeko,isaac,ostendorf,george,miljoenennota,kevingrose,komal,elance,apotheek,cigarettes,thrivingink,predict,infosys,pizza,hemeon,kou_mikami,ravcasleygera,umelabo,budweiser,spikeekips,
839863,eargang,50101,misaki,cocoa,
75309,fundtech,absarhaq,blucigs,thrill,digradio,scholarsavenue,postrank,simplecap,jimiyo,invent,invite,innovation,vimeo,kevinrose,Realtor,forecast,scholars,charousset,creativity,weather,vosizneias,isharetunes,alaska,thegling,orbis,accenture,tumblr,baynote,gling,actocom,testcricket,sufjan,download,mypageo,eleicoes2009,invest,JRDNLive,bermuda,catchoffer,nicovideo,roadhouse,coral,customer,masons,hasidic,mason,tofubeats,lifepass,frugal,geneity,durga,wisdom,openbet,market,vosiznies,giants,ballet,choicestream,bills,monza,Emmys,jewish,swine,rehava,dinamo,Feedback,glutenfreefaces,junkma,atigeo,droidcon,wisdon,inmyhood,betfair,cadsoft,discover,tweetDC,agent,nata2,esperanto,chinese,pixiv,historychannel,funny,nonoriri,hafizulazeez,erlang,gonbuto,yugui,oquno,rakusai,tsunami,gaogaogao,ZxAxTxSxUxOxNxN,barcampchs,bengals,ksasao,hitode909,rhythmsift,watch_akiba,trapezoid,DJWILDPARTY,kusigahama,wild_man,Shako_Pani,kuzuha,belleville,kalamazoo,MobileHackerz,hironica,yteppei,hkhumanoid,koseijin,postlets,a_kodama,enkiv2,yanbe,hajimehoshi,daikichi_blue,Philippines,sfc_orf,mat2uken,wilfue,deutsch,buchmesse,sakushin,rackspace,yusukezzz,boonies,tacke,memebomb,sj_tga,nojiri_h,H_D_D,quinte,kevinleversee,FollowFriday,lesondes,teraiman,removeall,liepzig,smartfade,cloud,aviary,kensuzuki,takimo,yummly,nepholologist,doodoo_jap,optimost,monae,nanhai,hoverhell,topdesblogs,naka3,REMOVE,paras,shokai,takaoka,genshi,shimobayashi,dai_okabe,takano32,bedbak,nintendo,automobile,Thomas,paulorcf,photos,hydrogeology,Krishna,music,tiichan,cricket,xlamp,statistics,rasmiranjan,Fitmitjulia,steelman,avila,foobar,browns,nuvaring,clt20,Monson,karthiktrue,blair,windows,ura_g7,arpan,
11eyes,Obama,nooblast,President,cck09,zilverenkruis,korea,zilverenkruisachmea,rajiv,s12bt,achmea,chicagokarl,science,bvalani,frozenworm,takkkun,fredbvalani,server,ignite,photoshop,quecy,smitu,jaisankar,serkantoto,sabu_mikawaya,kotobuki,goahead256,pubsubhubbud,koizuka,madrid2016,genschi,natant,MindTouch,zorgverzekering,risenberg,killerstartups,terebe,corkd,brown2020,bosschaang,rajivloveyou21,scrapbook,Ilayaraja,isuca,zilveren,tmura,verobus,rer2009,kanekure,rubyenrails,philosophy,macarthur,statitics,wanna,mangalore,shashi,global,singing,instrumentation,royal,hametsu,beatcancer,Mozilla,mallu,akshay,stephankaag,rubyonrails,sprint,telugu,dogeza,webbie,cookeville,cybershot,singer,psychiatry,kalmadi,keyword,WiMAX,economics,Buddhism,glocom,entomology,memeo,randomationword,kaleidico,sports,rolyal,Follow,climate,holderhq,koalition,cigarcook,health,paranormal,torrent,kanji,mogra,mexico,chicago,opera,illlinois,nevada,boulder,ryankanno,harper2,superfeedr,rails,russia,pubsub,monkeys,david,pkitano,atkinsm,california,hoffman,indiana,happiness,lentil,julien51,harper,thomasknoll,colorado,detour1999,spain,ejabberd,pongpaet,obama,twtmycard,parislemon,foursquare,gadfly,hardstyle,mindstorms,blackhat,guwahati,toronto,testing,kanye,elvis,lawyersclub,nixonmcinnes,Hamilton,musharraf,raptors,hansakochcom,yoonew,larok,bigbroafrica,hacking,napier,surajkala,bears,collectdesign,machine,Israeli,beyonce,wistia,ixigo,tajim,bicycle,federer,mbaclub,Arenys,budgeting,GoogleCodeJam,syndeomedia,teamtaylor,hagos,pandu,traeblain,chrisbrogan,edinburgh,shophtml,dingman,egypt,caclubindia,hansakoch,caclub,amandachapel,syndeolabs,brady,croatia,Soicla,profermics,miage,buffalo,hanskoch,arduino,additude,asshole,musichackday,rakhisawant,imparatta,suraj,Bulgaria,slept,geekandpoke,gwhatchet,cerevo,marcec,gitorious,spore,stafon,todesking,kettlemonkey,wildparty,Halloween,bookmyshow,HadekanX2,mygenius,bdutt,invitation,indian,Yossi,assam,gauhati,holga,tvduell,googlewave,artificial,jboss,umezawakazuki,thong,Searched,TravelCenters,public,leiberman,healthcare,StantonStreet,option,senate,BRAVIA,elrowan,boeing,sternbr,tickets,scotsbob,epona,anahi,tunisie,ubuntu9,ubbin,magic,rer20009,chumby,afghanistan,makayama,traffic,trafficsupport,jonikahara,byouinsan,paeae,cleartrip,ironman,cchsm,commontown,video,PenultimosDias,hvelarde,BrianWancho,flightstats,flight,quelle,YoaniSanchez,bandcontest,Xperia,sha_feng,discord,phweet,stock,Finland,Kotka,yahoo,dewet,centrelink,siegburg,tqubed,moosecox,podcast,googlevoice,gooberdlx,kindohm

And here is the response that we get :
413. HTTP/1.1 413 Request Entity Too Large


I am using to make that query our account "superfeedr_trac", which is
authorized. (When I try to authorize again, I get the following
message : "You're already set to access the Streaming API."

Thanks for your help.
Julien



On Nov 12, 8:16 am, Julien <julien.genest...@gmail.com> wrote:

Cameron Kaiser

unread,
Nov 12, 2009, 1:52:07 PM11/12/09
to twitter-deve...@googlegroups.com
> For some reason, my previous post didn't show up :/

That's because you're a new poster and the volunteer mods have to approve
them first. Give us a chance :)

--
------------------------------------ personal: http://www.cameronkaiser.com/ --
Cameron Kaiser * Floodgap Systems * www.floodgap.com * cka...@floodgap.com
-- It's lonely at the top, but the food is better. ----------------------------

Julien

unread,
Nov 12, 2009, 2:05:39 PM11/12/09
to Twitter Development Talk
Indeed... sorry about that!

On Nov 12, 10:52 am, Cameron Kaiser <spec...@floodgap.com> wrote:
> > For some reason, my previous post didn't show up :/
>
> That's because you're a new poster and the volunteer mods have to approve
> them first. Give us a chance :)
>
> --
> ------------------------------------ personal:http://www.cameronkaiser.com/--
>   Cameron Kaiser * Floodgap Systems *www.floodgap.com* ckai...@floodgap.com

Julien

unread,
Nov 14, 2009, 2:58:18 AM11/14/09
to Twitter Development Talk
John, Cameron... Any clue?

John Kalucki

unread,
Nov 15, 2009, 10:01:23 AM11/15/09
to Twitter Development Talk
I believe I answered this question on Nov 12 (message 2) in this
thread.

http://groups.google.com/group/twitter-development-talk/tree/browse_frm/thread/c6046a1df338f0ae/1c4bf379d27abb32?rnum=1&q=413&_done=%2Fgroup%2Ftwitter-development-talk%2Fbrowse_frm%2Fthread%2Fc6046a1df338f0ae%2Fb886df7e063a8c44%3Flnk%3Dgst%26q%3D413%26#doc_d9c83205c406586b

There is no single role in the Streaming API that will allow that many
follows and track parameters.

-John




On Nov 12, 9:21 am, Julien <julien.genest...@gmail.com> wrote:
> John,
>
> This is exactly what we post to your servers (I just hid the
> Authorization) :
>
> POST /1/statuses/filter.json HTTP/1.1
> Host: stream.twitter.com
> User-agent: TwitterStream
> Authorization: Basic XXXXX==
> Content-type: application/x-www-form-urlencoded
> Content-length: 13609
>
> follow=80706782,80706249,59397117,813276,1750351,1497,14303772,15640990,10630,1581511,794545,12,16718628,15351161,55033131,22207903,956501,16028823,982721,14100016,4411621,14615776,648563,26383279,21161414,10239,811350,9843812,16540939,20,25806143,15889416,10128922,13955972,31953,7337062,32653,1042361,26020994,1714051,7544012,309073,32063,14098218,33205299,1392281,664153,15328268,6717392,611823,15913,21169368,6892002,731253,7281962,14956807,5469512,13885092,36502008,15503041,5381582,26566923,2876271,14083455,18029701,1091741,8256162,34174320,16418101,17544169,22784949,5529162,813286,7136992,47685641,20935486,15864599,14712874,469163,14134376,13370272,1501,5973812,15133601,24338507,14454247,1748581,12858,13331942,1799511,10425502,24103,428333,10365,15279944,14229661,27663774,14692721,10226672,14363100,25694882,9536542,14418757,14114731,11900,15352135,18399826,22449663,7889672,25963448,5563012,66713497,1786041,19643660,17919393,14090674,14956888,25808898,59194896,2048141,15111776,16426292,45733,20201051,65973559,17781981,52680804,64409399,1367531,31132926,16450330,23853857,14075928,43856654,28156155,40197535,15773675,41112544,28174228,20526944,63407440,22802398,67899674,6017542,18120198,52589720,24855923,17829758,33233205,15945424,15954704,45075974,44888804,17409240,22797985,14323759,26072066,614133,11881852,21655440,37879306,1652541,6368672,29925623,31353077,6821102,15876379,14634720,37405382,15838599,14790192,711303,23753940,59726910,17819913,18866407,17113231,17124895,1947301,5741722,27456531,17171111,42033487,11856032,53372599,11857072,31114028,15996830,41075473,65904217,759251,23981347,14516048,27500565,16890969,16970248,15880642,16129920,42402819,45369197,31329385,5402612,13838562,14803917,8468872,14372478,65632985,14292132,80427603,38400130,22910295,77877070,48840381,3216921,612473,80713872,42056836,20153343,1705201,20924090,81391288,19438765,104203,14089573,4700381,7400702,33684457,37874853,14496232,7235522,7411552,71508034,15072502,33584794,15237935,81025521,22297824,7692692,18824787,18949452,7161542,22301030,27565488,14439930,1975321,1830491,46376581,19945693,23235489,14239496,3850041,22021097,43417156,12942,14162945,18159833,13827442,21645304,15708435,794928,15572963,11894422,456413,5579402,5990942,20399599,16297742,8085962,38961973,20631151,41375788,16416613,60334210,16705726,14305706,3128561,6893672,2860101,19039063,63810996,22651121,42860263,937961,50270097,36951816,1186,14613750,18799530,12974922,22543092,18991180,15251361,7408842,56458858,23530011,5567,48403197,586,15204168,17028306,2318591,20389644,7314182,14704003,14438790,42096373,45891150,15237382,60098511,27096075,19059514,14438014,4234581,15097328,50751569,246,54911372,770289,44011351,20194429,20365897,22540123,24203163,25489250,15533871,14166131,35136170,80766621,15915632,11268812,14768955,16901185,15888142,20132337,45537669,754978,19562029,32552821,33292459,14629669,14740219,1922861,4432431,19329393,16629431,31237480,41167154,7668952,16031518,28329367,81912095,26907579,15269523,15227791,15234407,41017380,21925979,21026837,11490952,16651978,15744222,17382456,14781982,22836661,22193651,19658936,24729040,14491138,17461978,44371159,32838392,36078825,3222731,24007848,29678507,15315785,22280679,23970108,41380057,34574635,42009631,59129082,35538270,66357651,66538952,38336343,21899223,25667617,78859670,67601095,67768214,14301002,68747982,16271343,16270679,19594811,40203798,22757239,46594911,80293100,13900252,23832022,19052717,71315517,78823450,33855080,43990220,17990812,45806824,9141622,27104736,32706664,39988586,16381061,20258912,74157758,52053604,74227685,43351209,76671588,59415820,21829544,37388407,15224126,26227858,18248645,38689604,65185176,21611204,22147157,14582043,13180972,47326062,61516762,33461621,14293310,76728008,72937694,14638581,69043094,34805346,16191323,75689974,30072246,45827487,75347542,74425788,80291874,80271800,71601293,51650188,56746527,18477121,22253558,16004911,82080333,24952459,33367205,15066838,6107422,972651,52725585,43062502,46277588,7076492,51509917,51413735,14994303,21879024,19394186,19058681,15494380,19697415,47326510,23135963,5793822,8921852,17133441,14302335,29014787,15633653,22205534,15192712,36116095,25098243,23484039,36751886,16896485,32943325,5515802,46400077,24609329,51457672,43913584,51469543,43705420,43798591,43585102,51459322,33478756,33479453,33480681,33480845,14440647,19704522,16625255,17636405,17774041,40275680,14933478,69889578,54591680,55241169,53926207,15856061,56307587,47401736,41773608,69399502,18876589,55300646,48367849,54550364,20275150,29471630,34940839,16560657,23855299,9473532,21568952,28568411,20065061,2367911,25110374,17487702,17453292,38735244,37409663,14480405,60442463,2557521,16348255,19709389,18716535,10068172,20432797,19861054,14821352,15301503,16202520,26827762,28874360,36640797,34578472,45316559,37423356,35557028,50306597,10330772,26742335,19356277,55176469,60956213,48413801,8204342,56482694,15439395,20172983,82633117,54590165,62599860,15480278,82568633,61220731,62534573,62613431,15417909,14342883,1143031,16651779,14511951,76979058,85951718,30962015,39063364,43339470,23332237,25547701,14612371,14875897,43286678,68675334,9599642,68132270,55250737,41036546,69040373,15436933,27201512,17745929,16409683,20302398,816653,22199141,37764422,54682125,15426659,35865630,16886396,15728419,78934317,18124790,45172332,1340521,21143108,30973,7282592,16910058,41117241,31397466,14801539,42262856,48658417,84553621,48697966,48689560,19764472,16973333,48463419,67995848,796955,20381538,13058552,87105253,19402238,53550244,59929520,60224553,42005012,25038085,26934903,66156073,40928638,39442498,39442877,36018499,803468,20574484,19643352,50539213,21586418,22745779,24291858,48651329,48654029,7466622,11778,86391789,32642765,27324578,23940534,88672023,86763679,17208949,16026621,89294978,76847458,21683908&track=remedie,caldeiraodohuck,twitpic,fantastico,carasebocas,Ultima,qipit,globoreporter,qlikview,pequelia,lucianohuck,twitter,tvglobo,domingaodofaustao,gappproxy,globo,protip,musasdobrasileirao,iphone,livetechdocs,globoesporte,clothingswap,Zilok,jornalnacional,jeffboudier,harry,tvrecord,blogestudio,termtter,palin,rederecord,pubsubhubbub,bfrench,caminhodasindias,jornaldarecord,chrome,webhooks,track,illinois,atkins,liveink,hawaii,harper3,glenn,jackson,mexico2,threadless,iyermatter,steelers,fukung,ubuntu,kevron,essec,sentiment,santa,copepodo,eagles,Track,twitterapi,asdfghjklqwertyuiop,yankees,banana,sounders,awesome,bucketworks,gmail,twollar,tokbox,sushubh,ipolitics,briandorsey,graph,massage,cardinals,
> 99acres,vuelta,firefox,Manchester,tracking,Lithuania,adwordsfr,
> 0eRPG,Remove,lazio,drupal,Malawi,steve,amazing,waynesutton,djangocon,opengov,downtown,liberzon,kansas,voice,notre,michael,digitalstrategy,illini,makaan,ihatewhengirlssay,broadbandforum,pancheros,linux,juliensharp,exclaimtrack,html5,feltham,indivibes,sleep,liverpoolinternets,breakingsfnews,paychex,michigan,crowdcompass,twitteroauth,haynes,linkedprocess,sushraju,
> 912dc,museum,greasemonkey,affiliate,milwaukee,maine,excla,dotclear,techcrunch,karen26,genedelibero,nyjets,Museum,remove,dc912,crowd,adwords,arturogoga,football,zeroidiots,eurobasket,amyloo,magicbricks,entangledstate,iwish,israel,techwhack,TRACK,aberystwyth,twollars,patientsknowbest,sleepfm,wcr09,chiefs,earthquake,motogp,siliconindia,sirius,Fallen,augmentedreality,ashlawn,startup,gtanuel,nickreynolds,pocketsmith,perth,hello,hiring,mattwarren,There,showrss,sitespect,skynewstopstory,voxeo,texas,stanford,singapore,greetz,ericrygh,coimbra,youaroevisual,bunny,huffington,wigsgiw,ivivek,information,jifdjis90,snobbyslice,hackday,juliansharp,galactica,socrates,citibank,freemium,nebraska,bankoti,android,stevegillmor,agraquest,python,ericjhansen,packers,hangzhou,skip1,china,mahtweets,ninyou,springs,weekend,xhtmlized,xerox,phillies,wordpress,maemo,ThereDotCom,dropbox,eyelite,leanne,breaking,nokia,kunming,website,rabbit,reddit,garyvee,sonyou,Metro,radarla,alexandra,money,leipzig,benseese,geophysics,brisbane,psd2html,green,janet,malibu,montessori,monkeysplat,blackberry,usopen,nerduo,youaroenvisual,karen,socialmedia,Glenn,pygame,dspattison,beagleboard,tracks,jetairways,zipcar,hosting,textwise,Tolkien,table,memcache,bobov,adamac,django,tomad,exclaim,kiryas,encryption,joshnunn,kayne,finance,pinax,india,bentsai,franklin,eageles,gluten,hayashih,skype,urasandesu,apple,realtor,dropio,miduhima,yamashiro,secude,vdhguy,security,jjudge,memcached,social,jaredwsmith,techcrunch50,google,swayze,textme,nevis,Hahn_Alex,satmar,state,savedforlater,angels,icici,grails,Taylor,ihone,aberfoyle,melon,worstpirate,yuiseki,XCode,boston,webhosting,iPhone,templeton,appengine,dougbarth,sociable,yedagandhi,remover,Comptia,scotland,csavage,email,ongeval,yeshiva,strasbourg,kularika,harvard,github,whitmore,akiba,monsey,meanjoepaterno,melontech,userstorylab,hashlove,search,exposur3,googleappengine,wipro,bluecigs,omnilife,vgtweetup,johnpwood,warenosyo,digicorp,facebook,tacos,Virgin,salesforce,soo_mei,cotapata,astadia,foodjammers,nstein,maako,l2wireless,mygirlbetty,european,jeffrey,ACRON,oracle,omniture,georg,barroso,inetium,nacin,nikolic,product,mpilgrim,brendadronkers,ghana,kathryn,mobile,kersey,kevin,brenda,trend,ansbach,economy,tombloo,thril,odesk,hatzolah,District,puzzle,jackass,japan,corus,romangeeko,isaac,ostendorf,george,miljoenennota,kevingrose,komal,elance,apotheek,cigarettes,thrivingink,predict,infosys,pizza,hemeon,kou_mikami,ravcasleygera,umelabo,budweiser,spikeekips,
> 839863,eargang,50101,misaki,cocoa, ...
>
> read more »

Julien

unread,
Nov 15, 2009, 12:47:09 PM11/15/09
to Twitter Development Talk
Thanks John,

I am not sure I understand when you say "There is no single role in
the Streaming API that will allow that many follows and track
parameters.", your documentation says :
"The default access level allows up to 200 track keywords and 400
follow userids. Increased access levels allow 80,000 follow userids
("shadow" role), 400,000 follow userids ("birddog" role), 10,000 track
keywords ("restricted track" role), and 200,000 track keywords
("partner track" role). Increased track access levels also pass a
higher proportion of statuses before limiting the stream."

Ryan Sarver said he granted us the "shadow" role yet (80000 usersid),
we can track more than 400 userid...

What are our remaining options? Polling doesn't work (you block our
requests, and you say, (-but your doc says something different) that
we can use your streaming API.

Julien



On Nov 15, 7:01 am, John Kalucki <jkalu...@gmail.com> wrote:
> I believe I answered this question on Nov 12 (message 2) in this
> thread.
>
> http://groups.google.com/group/twitter-development-talk/tree/browse_f...
>
> There is no single role in the Streaming API that will allow that many
> follows and track parameters.
>
> -John
>
> On Nov 12, 9:21 am, Julien <julien.genest...@gmail.com> wrote:
>
>
>
> > John,
>
> > This is exactly what we post to your servers (I just hid the
> > Authorization) :
>
> > POST /1/statuses/filter.json HTTP/1.1
> > Host: stream.twitter.com
> > User-agent: TwitterStream
> > Authorization: Basic XXXXX==
> > Content-type: application/x-www-form-urlencoded
> > Content-length: 13609
>
> > follow=80706782,80706249,59397117,813276,1750351,1497,14303772,15640990,106 30,1581511,794545,12,16718628,15351161,55033131,22207903,956501,16028823,98 2721,14100016,4411621,14615776,648563,26383279,21161414,10239,811350,984381 2,16540939,20,25806143,15889416,10128922,13955972,31953,7337062,32653,10423 61,26020994,1714051,7544012,309073,32063,14098218,33205299,1392281,664153,1 5328268,6717392,611823,15913,21169368,6892002,731253,7281962,14956807,54695 12,13885092,36502008,15503041,5381582,26566923,2876271,14083455,18029701,10 91741,8256162,34174320,16418101,17544169,22784949,5529162,813286,7136992,47 685641,20935486,15864599,14712874,469163,14134376,13370272,1501,5973812,151 33601,24338507,14454247,1748581,12858,13331942,1799511,10425502,24103,42833 3,10365,15279944,14229661,27663774,14692721,10226672,14363100,25694882,9536 542,14418757,14114731,11900,15352135,18399826,22449663,7889672,25963448,556 3012,66713497,1786041,19643660,17919393,14090674,14956888,25808898,59194896 ,2048141,15111776,16426292,45733,20201051,65973559,17781981,52680804,644093 99,1367531,31132926,16450330,23853857,14075928,43856654,28156155,40197535,1 5773675,41112544,28174228,20526944,63407440,22802398,67899674,6017542,18120 198,52589720,24855923,17829758,33233205,15945424,15954704,45075974,44888804 ,17409240,22797985,14323759,26072066,614133,11881852,21655440,37879306,1652 541,6368672,29925623,31353077,6821102,15876379,14634720,37405382,15838599,1 4790192,711303,23753940,59726910,17819913,18866407,17113231,17124895,194730 1,5741722,27456531,17171111,42033487,11856032,53372599,11857072,31114028,15 996830,41075473,65904217,759251,23981347,14516048,27500565,16890969,1697024 8,15880642,16129920,42402819,45369197,31329385,5402612,13838562,14803917,84 68872,14372478,65632985,14292132,80427603,38400130,22910295,77877070,488403 81,3216921,612473,80713872,42056836,20153343,1705201,20924090,81391288,1943 8765,104203,14089573,4700381,7400702,33684457,37874853,14496232,7235522,741 1552,71508034,15072502,33584794,15237935,81025521,22297824,7692692,18824787 ,18949452,7161542,22301030,27565488,14439930,1975321,1830491,46376581,19945 693,23235489,14239496,3850041,22021097,43417156,12942,14162945,18159833,138 27442,21645304,15708435,794928,15572963,11894422,456413,5579402,5990942,203 99599,16297742,8085962,38961973,20631151,41375788,16416613,60334210,1670572 6,14305706,3128561,6893672,2860101,19039063,63810996,22651121,42860263,9379 61,50270097,36951816,1186,14613750,18799530,12974922,22543092,18991180,1525 1361,7408842,56458858,23530011,5567,48403197,586,15204168,17028306,2318591, 20389644,7314182,14704003,14438790,42096373,45891150,15237382,60098511,2709 6075,19059514,14438014,4234581,15097328,50751569,246,54911372,770289,440113 51,20194429,20365897,22540123,24203163,25489250,15533871,14166131,35136170, 80766621,15915632,11268812,14768955,16901185,15888142,20132337,45537669,754 978,19562029,32552821,33292459,14629669,14740219,1922861,4432431,19329393,1 6629431,31237480,41167154,7668952,16031518,28329367,81912095,26907579,15269 523,15227791,15234407,41017380,21925979,21026837,11490952,16651978,15744222 ,17382456,14781982,22836661,22193651,19658936,24729040,14491138,17461978,44 371159,32838392,36078825,3222731,24007848,29678507,15315785,22280679,239701 08,41380057,34574635,42009631,59129082,35538270,66357651,66538952,38336343, 21899223,25667617,78859670,67601095,67768214,14301002,68747982,16271343,162 70679,19594811,40203798,22757239,46594911,80293100,13900252,23832022,190527 17,71315517,78823450,33855080,43990220,17990812,45806824,9141622,27104736,3 2706664,39988586,16381061,20258912,74157758,52053604,74227685,43351209,7667 1588,59415820,21829544,37388407,15224126,26227858,18248645,38689604,6518517 6,21611204,22147157,14582043,13180972,47326062,61516762,33461621,14293310,7 6728008,72937694,14638581,69043094,34805346,16191323,75689974,30072246,4582 7487,75347542,74425788,80291874,80271800,71601293,51650188,56746527,1847712 1,22253558,16004911,82080333,24952459,33367205,15066838,6107422,972651,5272 5585,43062502,46277588,7076492,51509917,51413735,14994303,21879024,19394186 ,19058681,15494380,19697415,47326510,23135963,5793822,8921852,17133441,1430 2335,29014787,15633653,22205534,15192712,36116095,25098243,23484039,3675188 6,16896485,32943325,5515802,46400077,24609329,51457672,43913584,51469543,43 705420,43798591,43585102,51459322,33478756,33479453,33480681,33480845,14440 647,19704522,16625255,17636405,17774041,40275680,14933478,69889578,54591680 ,55241169,53926207,15856061,56307587,47401736,41773608,69399502,18876589,55 300646,48367849,54550364,20275150,29471630,34940839,16560657,23855299,94735 32,21568952,28568411,20065061,2367911,25110374,17487702,17453292,38735244,3 7409663,14480405,60442463,2557521,16348255,19709389,18716535,10068172,20432 797,19861054,14821352,15301503,16202520,26827762,28874360,36640797,34578472 ,45316559,37423356,35557028,50306597,10330772,26742335,19356277,55176469,60 956213,48413801,8204342,56482694,15439395,20172983,82633117,54590165,625998 60,15480278,82568633,61220731,62534573,62613431,15417909,14342883,1143031,1 6651779,14511951,76979058,85951718,30962015,39063364,43339470,23332237,2554 7701,14612371,14875897,43286678,68675334,9599642,68132270,55250737,41036546 ,69040373,15436933,27201512,17745929,16409683,20302398,816653,22199141,3776 4422,54682125,15426659,35865630,16886396,15728419,78934317,18124790,4517233 2,1340521,21143108,30973,7282592,16910058,41117241,31397466,14801539,422628 56,48658417,84553621,48697966,48689560,19764472,16973333,48463419,67995848, 796955,20381538,13058552,87105253,19402238,53550244,59929520,60224553,42005 012,25038085,26934903,66156073,40928638,39442498,39442877,36018499,803468,2 0574484,19643352,50539213,21586418,22745779,24291858,48651329,48654029,7466 622,11778,86391789,32642765,27324578,23940534,88672023,86763679,17208949,16 026621,89294978,76847458,21683908&track=remedie,caldeiraodohuck,twitpic,fan tastico,carasebocas,Ultima,qipit,globoreporter,qlikview,pequelia,lucianohuc k,twitter,tvglobo,domingaodofaustao,gappproxy,globo,protip,musasdobrasileir ao,iphone,livetechdocs,globoesporte,clothingswap,Zilok,jornalnacional,jeffb oudier,harry,tvrecord,blogestudio,termtter,palin,rederecord,pubsubhubbub,bf rench,caminhodasindias,jornaldarecord,chrome,webhooks,track,illinois,atkins ,liveink,hawaii,harper3,glenn,jackson,mexico2,threadless,iyermatter,steeler s,fukung,ubuntu,kevron,essec,sentiment,santa,copepodo,eagles,Track,twittera pi,asdfghjklqwertyuiop,yankees,banana,sounders,awesome,bucketworks,gmail,tw ollar,tokbox,sushubh,ipolitics,briandorsey,graph,massage,cardinals,
> > 99acres,vuelta,firefox,Manchester,tracking,Lithuania,adwordsfr,
> > 0eRPG,Remove,lazio,drupal,Malawi,steve,amazing,waynesutton,djangocon,opengo v,downtown,liberzon,kansas,voice,notre,michael,digitalstrategy,illini,makaa n,ihatewhengirlssay,broadbandforum,pancheros,linux,juliensharp,exclaimtrack ,html5,feltham,indivibes,sleep,liverpoolinternets,breakingsfnews,paychex,mi chigan,crowdcompass,twitteroauth,haynes,linkedprocess,sushraju,
>
> ...
>
> read more »

John Kalucki

unread,
Nov 15, 2009, 2:29:51 PM11/15/09
to Twitter Development Talk
The default access levels are track 200 and follow 400.

Shadow increases follow to 80k, but leaves track at the default of
200.
TrackRestricted increases track keywords to 10k but leaves follow at
the default of 400.

All of the other mentioned roles do the same, they increase access on
one dimension, but leave the other dimension at the default. There is
no role currently available that increases both the track and the
follow limit. One could successfully argue that there should be such a
role. In the mean time, we suggest that the rare service that requires
both increased track and increased follow access obtain elevated roles
on two different accounts and connect twice. While this initially
appears annoying, it allows functionally partitioned predicate refresh
periodicity.

-John Kalucki
http://twitter.com/jkalucki
Services, Twitter Inc.


Julien

unread,
Nov 15, 2009, 6:35:20 PM11/15/09
to Twitter Development Talk
Thanks a lot John for this clear explaination.
Is it possible to have the following settings:
- superfeedr_foll as Shadow
- superfeedr_trac as TrackRestricted

That would truly be awesome. I will change our implementation so that
we use 2 different accounts for the 2 purposes.
Again, thx.

Julien


On Nov 15, 11:29 am, John Kalucki <jkalu...@gmail.com> wrote:
> The default access levels are track 200 and follow 400.
>
> Shadow increases follow to 80k, but leaves track at the default of
> 200.
> TrackRestricted increases track keywords to 10k but leaves follow at
> the default of 400.
>
> All of the other mentioned roles do the same, they increase access on
> one dimension, but leave the other dimension at the default. There is
> no role currently available that increases both the track and the
> follow limit. One could successfully argue that there should be such a
> role. In the mean time, we suggest that the rare service that requires
> both increased track and increased follow access obtain elevated roles
> on two different accounts and connect twice. While this initially
> appears annoying, it allows functionally partitioned predicate refresh
> periodicity.
>
> -John Kaluckihttp://twitter.com/jkalucki

John Kalucki

unread,
Nov 15, 2009, 9:34:43 PM11/15/09
to Twitter Development Talk
Please request the additional access by whatever means you requested
your initial access.

-John
> ...
>
> read more »

Julien

unread,
Nov 15, 2009, 11:26:24 PM11/15/09
to Twitter Development Talk
Ok I sent an email, but Ryan takes forever to respond to my emails :/
Isn't there anything you could do?

thanks,
> ...
>
> read more »
Reply all
Reply to author
Forward
0 new messages