Issue 324 in protobuf: Feature: Add a new primitive type for a DateTime format
prot...@googlecode.com
Owner: liuj...@google.com
Labels: Type-Defect Priority-Medium
New issue 324 by Grigand: Feature: Add a new primitive type for a DateTime
format
http://code.google.com/p/protobuf/issues/detail?id=324
Problem Statement:
Protobuffers is lacking in one major way: Date/Time support. Without a
defined standard way of transporting date+time information, we are left
with inventing our own. Then we are required to document how this data is
stored/interpreted. This makes for an ugly public API when 99% of the
interchange format is well defined, and 1% is not.
Frankly it's getting very painful, especially when we use proto-buffers to
define our wire format and convert these proto structures to
json/bson/xml. Both bson and xml require knowledge that the field is of
type date+time. Json, unfortunately, is in the same boat protobuffers is,
and the current 'hack' is using new Date(...). While writing these various
formats we are currently in the position of defining date+time as an Int64
and recognizing it by the field name suffix 'RawDateTime'. Lots of ugly
code, but I think you get the idea of why this is important.
Solution(s):
I am told google transmits date+time as an Int64 offset from the unix epoch
(00:00:00 UTC on 1 January 1970). This would be more than sufficient for
most uses. I also admit there is a whole 'bag of worms' that goes along
with date+time. I'm not interested in solving world hunger, but a simple,
well defined, standard date+time format is a must for many projects. I
really don't care what standard is used, anything is better than nothing at
this point ;)
If you really want to overkill the problem ...
struct DateTime64
{
union {
struct {
sint64 Year : 16; // +/-32767 (16 bits)
uint64 Month : 4; // 1~12 (20 bits)
uint64 Day : 5; // 1~31 (25 bits)
uint64 Hour : 5; // 0~23 (30 bits)
uint64 Minute : 6; // 0~59 (36 bits)
uint64 Second : 6; // 0~59 (42 bits)
uint64 Milli : 10; // 0~999 (52 bits)
uint64 IsDST : 1; // 0~1 (53 bits) - Daylight
Saving Time
sint64 UtcOffset : 11; // -720~840 (64 bits) - UTC offset
minutes
} ;
uint64 Int64;
};
};
prot...@googlecode.com
Comment #1 on issue 324 by ask...@gmail.com: Feature: Add a new primitive
http://code.google.com/p/protobuf/issues/detail?id=324
I think adding a binary datetime type, especially based on the aging and
awful C conventions, is a terrible idea. It's just not adequate for a host
of time representations. Moreover, I don't how it will save the bug filer
any work whatsoever, since not all of his target formats use the same
date/time representation anyway. XML "uses" RFC3339 strings, and JSON uses
whatever the author wants (though RFC3339 is common).
prot...@googlecode.com
Comment #2 on issue 324 by Grigand: Feature: Add a new primitive type for a
DateTime format
http://code.google.com/p/protobuf/issues/detail?id=324
>> I think adding a binary datetime type, especially based on the aging and
>> awful C conventions, is a terrible idea. It's just not adequate for a
>> host of time representations.
This is of course the 'can-o-worms' I spoke of. Though I agree it's
incomplete information, it does transfer the date+time value. If your
specific application has need of more, another field can specify the UTC
Offset, Zone Id, Olson ID, or whatever serves your specific requirements.
Example:
message Document {
required datetime utc_modified_date = 1;
required datetime utc_created_date = 2;
optional int32 utc_offset_minutes = 3;
}
This isn't as 'ideal' as being able to define both in one field; however,
it is unambiguous enough for a public facing API. I don't need to document
in what format the date+time part should be specified. The use of
utc_offset_minutes should also be obvious to any consumer. This, IMO,
get's us "close enough" without getting tied up in months of debate about
the perfect date+time representation. I'd even be OK with moving date+time
as ISO8601/RFC3339 string data, anything just something standard :)
My preference is still the later option, a packed field based 'observed
time', rather than using an 'incremental time'. Incremental time does a
great job at sorting and calculating (add, subtract, etc) date+time. An
observed time on the other hand is horrible thing to add/subtract and may
even be difficult to sort. Yet the observed time is still superior for
transporting date+time which is the problem protobuffers is designed to
solve.
>> Moreover, I don't how it will save the bug filer any work whatsoever,
>> since not all of his target formats ...
This is somewhat off topic. I don't want this issue to get bogged down in
transcoding of protobuffers into other protocols. The *real problem* is
that a public API built on protobuffers can not support date+time values in
a standard way.
>> ..., since not all of his target formats use the same date/time
>> representation anyway.
The time/energy saved has to do with the fact that the descriptor and
datatype of the proto do not reflect whether the field is a date+time or
not. This means that serializers that are crawling field values encounter
a string, or uint64, not a language specific date+time. They must either
guess by convention or define field extensions that supply the additional
information.
You are correct that I have varying requirements in target formats, this is
the nutshell of my issue. I need to convert the date+time values into
various formats (ISO8601/RFC3339, JSON Date(), or BSON Unix epoch +
milliseconds) to comply with other protocols. This means I must know it's
a date+time, how to extract it into a language-specific type, and then
convert it into the appropriate form for the target protocol.
Again, transcoding between protocols is just a very small part of the
overall problem.
prot...@googlecode.com
Comment #3 on issue 324 by cms...@gmail.com: Feature: Add a new primitive
http://code.google.com/p/protobuf/issues/detail?id=324
Can we just have a 64 bit type for 'milliseconds since 1970', as both Java
and Javascript like this format?
prot...@googlecode.com
Comment #4 on issue 324 by tom.ritc...@gmail.com: Feature: Add a new
http://code.google.com/p/protobuf/issues/detail?id=324
If you use a signed 64-bit integer for "milliseconds since the epoch", that
gets you to almost 300 million years in the past, and 300 million years in
the future. It is fairly likely that I'll be deceased by then, and almost
certain I shan't be using protocol buffers in developing software. :-D
I see zero advantage to having the complex data structure. It's inevitable
that you will have to convert whatever data structure the Protobuff team
comes up with to the format that your own development system is already
using, and I've never run into any time or date system that didn't
understand "time since the epoch" (even if it's sometimes "seconds since
1970" instead of milliseconds).
I'm already using signed 64-bit ints as times in my protocol buffers simply
because it's the "native format" of the cross-platform toolkit I'm using
(JUCE) and I personally would not take the time to use any such format.
I "vote" that this bug be marked "Will Not Fix" (not that this is a
democracy, but hey).
Happy New Year, btw!
prot...@googlecode.com
Comment #5 on issue 324 by lukera...@gmail.com: Feature: Add a new
The issue is not about the best representation of the datetime as it is
serialised (ms since epoch seems fine to me) it is whether the datetime
should be a supported native type, so that the generated classes can have
properties that don't require extra handling to convert to a native datetime
prot...@googlecode.com
Status: Invalid
Comment #6 on issue 324 by liu...@google.com: Feature: Add a new primitive
Protobuf library is designed to provides fundamental low-level primitive
types. You can use int64 or even bytes for arbitrary data in application
layer.
prot...@googlecode.com
Comment #7 on issue 324 by apenneba...@42six.com: Feature: Add a new
Yes, please! Datetimes are very common fields, historically implemented
with a slew of incompatible formats. Protocol Buffers is intended to help
data travel between systems in a language-agnostic way, so the current
solution of rolling your own format is antithetical.
--
You received this message because this project is configured to send all
issue notifications to this address.
You may adjust your notification preferences at:
https://code.google.com/hosting/settings
prot...@googlecode.com
Comment #8 on issue 324 by wouter.b...@gmail.com: Feature: Add a new
Temporenc (http://temporenc.org) is a comprehensive binary encoding format
for dates and times that tries to solve the problem discussed in this
issue. It is designed to be embedded into protobufs (or other embedding
formats).