The documentation may be a bit out-dated. Right now the limit for all
searches is pretty much 7 days b/c of performance/storage reasons.
They are working on extending that window, but there is not a specific
date for having that accomplished.
-Chad
Twitter claims to have all its data stored in disk-based
databases from what I understand ... yet without access to
this data it is worthless.
It seems to me they could allow searches of this historical
data via a new "History API" then let us cache the results
on our own servers. Most of the services I've conceived
would do this infrequently -- never in real time -- and
would not impact their existing "cached" server data because
this historical data would exist on separate data storage
servers ... theoretically anyways.
Owkaye
The load on a server to extract a big dataset once a month
would be minimal, and both you and I can see the value in
this approach. But I'm not sure the folks at Twitter do, or
if they do maybe they just don't have the people who can
(and will) get things like this implemented. Is a shortage
of competent staff the cause of this type of problem?
Even though I have the capabilities I do not have the
'resume' to get a job there and help them deal with some of
this stuff, nor do I have the contacts within the Twitter
organization to put a good word in for me and help me get
hired so I could do good things for them.
I'm 52 years old too, and my age seems to be a negative to
most of the Web 2.x companies hiring these days. This is
kind of a shame considering that people like me frequently
have broader-based experience and insights that are
sometimes lacking in younger people, and because of this we
can add a lot more value in the areas of planning and
structural development than people half our age. Our coding
skills are honed after so many years of experience too, not
to mention the thousands of code snippets we have collected
over the years to contribute to making us even faster.
But since jobs like this are basically not open to me and
many other folks my age, my alternative is to remain self-
employed and try to build something on top of their existing
available source data and API's ... and then deal with the
issues and frustrations created when building a service on
top of a 'moving target' that sometimes seems to be moving
in funny directions.
I hear about Twitter having lots of money to work with, and
I'm probably wrong here but it almost seems like there's too
little of this money being dedicated to paying new talent
with long term views of some of these issues, and who will
implement wise policies to help support and encourage rapid
growth in the areas that are lacking. But once again this
might just be due to a shortage of the right staff.
Obviously we cannot do anything from the outside except
point out these issues and ask questions, or beg and plead
for changes, but it sure would be great if a few of us could
actually get in there as employees and implement a couple of
the new features we really need -- such as a new Historical
Search API for example. Then developers like you and I
could proceed with some of our plans now, instead of months
or years from now ... or possibly never. I would love to
lead a team on a project like this, or even be one of its
members, but until it happens I'll focus on building my own
little space in the Twitter universe and continue to hope
for the best.
:)
Owkaye
My thought is that there may be no one at Twitter actually
planning or developing a plan for historical data access,
and if this is true then hiring someone with the skills and
the desire to implement this in a practical manner would go
a very long way towards providing people like us with a
workable solution now.
Having said this, I agree that in the absence of enough
people in the company who can be trusted to make wise
decisions and accomplish a wide variety of projects all at
the same time, it ends up becoming a priority issue. When
there are too few people available to actually take charge
and make progress on projects like the one we've been
discussing in this thread, it all comes down to priorities
-- and when those priorities focus on things we do not need,
the things we really want are set aside and ignored, with no
progress being made.
In other companies money is a significant limiting factor,
but I tend to question this at Twitter given all the reports
of their financial condition, so I really think it's a
priority issue in Twitter's case.
Now, if only someone at Twitter could see how important
historical data access can be to real businesses, and how
these businesses might be willing to pay for this data, then
all it would take is to hire the right person to implement
it. Twitter simply needs the money, the current ability to
recognize the future value of such a project, and the
commitment to "make it happen" ... and then they hire a
leader who gets it done.
Easier said than done of course, but there are excellent
people available who can accomplish such goals when given
the chance -- and the support they need from within the
company of course.
Then again, if these people are already working on it (as
you may have suggested) then it's going to happen one of
these days anyways ... :)
Owkaye