|In PHP <= 5.1.6 trying to decode an integer value that's > PHP_INT_MAX
will result in an intger of PHP_INT_MAX.
In PHP 5.2+ decoding an integer > PHP_INT_MAX will cause a conversion to
a float.
Neither behaviour is perfect, capping at PHP_INT_MAX is marginally
worse, but the float conversion loses precision.
If you expect to deal with large numbers at all, let alone in JSON,
ensure you're using a 64-bit system.
------
my thoughts:
if moving to a 64-bit system is not feasible and you aren't interesting
in hacking the (php itself) source (num_as_string json_decode() patch at
http://bugs.php.net/bug.php?id=46363), you could try xml or writing a
quick-n-dirty twitter-specific decoder. other alternatives are looking
at a hosting provider or paying someone (ahem) to patch your PHP
instance <plug>(I do Linux only, not BSD or 'doze, but I've been doing
Linux for a long time and I'm pretty good at it)</plug>.
thanks!
Joseph Cheek
jos...@cheek.com, www.cheek.com
www.twitter.com/cheekdotcom
As Joseph points out, PHP on a 64-bit system can handle these numbers.
If you really want this data as a string, you could write a regex in
PHP to alter the json string to wrap the digits in quotes before
sending it through json_decode(), but that would be a pretty gnarly
kludge.
-Chad
Joseph Cheek
jos...@cheek.com, www.cheek.com
twitter: http://twitter.com/cheekdotcom
This goes for any large numbers, including tweet ids. As far as I am
concerned they can output everything in JSON as strings.
For http://twxlate.com, I use Google's GSON library
(http://code.google.com/p/google-gson/), and, if I had typed my
variables (actually, Java class fields) as int, Integer, long, or Long,
then the change you suggest would break my code.
Of course the code would also have been broken when the values exceeded
2**31 or 2**63. Anticipating this and also possibly non-numeric values,
I typed the fields as Strings. So I have lucked out on the impact of
Twitpocalylses.
But others may not have been or be so lucky.
On a higher plane, to my mind this is an issue of versioning in the API
syntax. Twitter does not support explicit versioning (i.e., the
inclusion of an API version parameter in API requests) as other RESTful
interfaces do. For example, most Google API RESTful interface methods
include a v= parameter to indicate which version of the method is being
used.
If twitter included this, then to receive large numeric values encoded
as strings the requester could simply include, e.g., v=1.1, on the
request. Given that twitter specified in advance that if you request
version 1.1 responses then large numeric values would be encoded as
strings, then any request made with v=1.1 would have known and had to be
prepared to accept string encoded values. Without v=1.1, you would still
get integer encoded values, and nothing would break.
There have been, and I suspect will continue to be, changes to the API
specification that were or will be not backward compatible without a
version parameter, and for this reason I strongly urge twitter to adopt
this mechanism for all non-backward compatible API changes.
Version negotiation could also be done via HTTP headers, but I prefer
method parameters.
Comments expected and welcome.
Jim Renkel
This has already happened. Tweet ids are now bigger than a 32 bit int
can store. That's what Twitpocalypse 2 was all about.
-Chad
i know this is akin to COBOL programmers in the fifties saying "hmm, i
wonder if we should concern ourselves with what will happen to our apps
in the year 2000?" and giving it no further thought, and i apologize,
but i'm ok with that.
Joseph
--
Chad Etzel wrote:
> This has already happened. Tweet ids are now bigger than a 32 bit int
> can store. That's what Twitpocalypse 2 was all about.
>
> -Chad
>
--
They have passed 32-bit *un*signed. My latest tweet id is 4345383616
(which is > 2^32 - 1)
Twitpocalypse 2 was about overflowing the 32 bit int space.
-Chad
What do you get on your server if you do:
echo PHP_INT_MAX;
also, what version of PHP are you using?
-Chad
"As mentioned previously, the Twitter operations team will artificially
increase the maximum status ID to 4294967296 this coming Friday,
September 11th. This action is part of routine database upgrades and
maintenance. "
http://groups.google.com/group/twitter-development-talk/browse_thread/thread/8fde4804c5a75fb2
it appears to me that tw2k9 (<-- my new abbreviation, sorry, have typed
twitpocalypse 2 too many times) is about using the full 32-bit int
space. agreed, it didn't make much sense to announce that you were
going to now utilize the full address space, but that *is* how it was
announced - increase the new *maximum* status ID [emphasis mine] to
4294967296.
$ echo $[ 65536 * 65536 ]
4294967296
thanks!
Joseph Cheek
jos...@cheek.com, www.cheek.com
twitter: http://twitter.com/cheekdotcom
and using json_decode on the next_cursor field of an ids list works just fine.
I'm using PHP 5.2.10-0.dotdeb.1 (though PHP 5.2.11-0.dotdeb.1 is available)
fwiw,
-Chad
How do I know my php can handle it? I'm running on an older PPC
machine, which is 64 bit, but I have no idea if I built it as 64, I
just supplied standard configure arguments.
If php sees a 64 bit integer come in as JSON what happens on 32 bit?
Does it just truncate it or completly ruin it's representation?
Why can't it just be treated as a string?don't these ID's end up in a
database, or maybe just passed as a URL argument? As a URL arg you
pass to Twitter, so it will work fine.
Any database can store a 64 bit int as a string, which gives your
ability to get the string back to post to a URL.
Can a 32 bit build of a database store 64 bit ints? That to me seems
the bigger issue.
I'm not seeing where there is a need for math on a 64 bit int ditectly
in php. You get the value from JSON, treat it as a string, store that
string, and use it for your lookups and sorts.
Inneficient on lookups and storage, is that the core of this?
What key aspect of this problem am I missing?
--
Scott
Iphone says hello.
--
Scott
Iphone says hello.
Take it to 1000.00 or so, and you can built out your own 1U server
that is going to hold it's own very very well. Or, rent a 1U, they
are a few hundred bucks a month, you get to pick the OS, pick the
bit'ness, etc.
If you are a startup, on a shoestring, part of the fun is being
creative, and making due with what you have.
And don't laugh at that MacMini, you can easily get to the SATA port,
and plug it right into a SATA array elsewhere, getting SATA II on full
size dives. I just pegged the drives to a cork board, its not pretty,
but it works. I have a second on standby in case that one craps. I
just had them laying around, and made them up from parts.
While I am not going to mention the site, I can tell you, it is php on
the front, apache2 with a good deal of mod_rewriting going on, and
substantial database work, all happening on the MacMini, handling
500,000+ average unique visitors, which is a lot more requests when
you add in images and the rest. (this site is not twitter related)
If I was sloppy, that database would fall on it's face under that
load. I have added in some caching where needed, so pages are
rendered to disk at the expense of a php include of the data.
I am in this thread to learn about the issue, but so far, getting to
64 bits should not be a barrier. Isn't this really a php issue
anyway? Why can't php handle a 64 bit int in a 32 bit system? Maybe
they need a bug report or a patch. MySql seems to handle 64 bit on my
32 bit test system here just fine. Maybe someone can write a wrapper
for the json encode/decode that fixes this in that part of php?
--
Scott * If you contact me off list replace talklists@ with scott@ *
Heck, Perl does it fine. I run TTYtter on a 32-bit system that runs a Perl
that's over 10 years old, and it's perfectly happy with bigints.
--
------------------------------------ personal: http://www.cameronkaiser.com/ --
Cameron Kaiser * Floodgap Systems * www.floodgap.com * cka...@floodgap.com
-- The less we know, the better we feel. -- David Bowie, "Miracle Goodnight" --
There seem to be acceptable workarounds, solid proposed workarounds,
etc. I guess I am not getting it, JSON is just a string returned,
yes, it can represent type of data, but it is still just a string. I
can not see it being that huge a performance hit to massage that
string a bit once you get ahold of it.
--
Scott * If you contact me off list replace talklists@ with scott@ *
Precisely. If you really want to turn the numbers into strings in PHP,
here are 2 workaround examples. Please note, I use PHP personally, so
I am not apathetic to this issue. There are programmatic ways of
dealing with it, and that's half the fun of programming, right?
Example 1 (cursor specific example);
<?
/* normally use curl, but using file_get_contents for simplicity of example */
$json = file_get_contents("http://twitter.com/followers/ids/barackobama.json?cursor=-1");
echo $json;
$pattern = "/\"next_cursor\":([0-9]+),/";
$replace = "\"next_cursor\":\"$1\",";
$new_json = preg_replace($pattern, $replace, $json);
$pattern = "/\"previous_cursor\":([0-9]+),/";
$replace = "\"previous_cursor\":\"$1\",";
/* note $new_json in third parameter */
$new_json = preg_replace($pattern, $replace, $new_json);
echo $new_json;
var_dump(json_decode($new_json));
?>
Example 2 (generalized json "naked" numbers example):
<?
/* generic helper function to turn all number types in json into strings */
function stringify_json_numbers($json)
{
$pattern = "/([:[]?)([-]?[0-9]+)([,}\]])/";
$replace = "$1\"$2\"$3";
$new_json = preg_replace($pattern, $replace, $json);
return $new_json;
}
/* normally use curl, but using file_get_contents for simplicity of example */
$json = file_get_contents("http://twitter.com/followers/ids/barackobama.json?cursor=-1");
echo $json;
$new_json = stringify_json_numbers($json);
echo $new_json;
echo "<pre>";
var_dump(json_decode($new_json));
echo "</pre>";
?>
Can't we all just get along?
-Chad
Or, if that is too much maintenance, how about
api.twitter.com/?bitfix=32 or whatever.
--
Scott * If you contact me off list replace talklists@ with scott@ *
Jim Renkel
On Sep 25, 2009, at 8:08 PM, "Jim Renkel" <james....@gmail.com>
wrote:
2009/9/26, Michael Steuer <mst...@gmail.com>: