case study for beaming of transactions via QR
Andreas Schildbach
released a new testnet version 1.43 of Bitcoin Wallet. You'll need to
install that version on both the sending and the receiving phone.
If you long-click on a tx in the tx list, you'll now have the option to
view the transaction details. This also includes a QR code that contains
the whole transaction! You can currently scan the QR code only with an
external app like ZXing or Goggle, which allows you to view the details
of the same transaction on a second phone.
In order to leverage the Android intent mechanism once again, I invented
a new URI format "btctx:<base64-encoded-bitcoin-serialized-tx>". A bit
more elaborate:
- tx is serialized using standard Bitcoin serialization
- resulting bytes are base64-encoded, without padding or wrapping
- "btctx:" scheme is prepended
- QR code type URI is generated from that
Some issues I found on the way:
Typical tx sizes of around 500 bytes seem to work quite well. However, I
have one 1200 byte tx which obviously busts even a "version 40" QR code.
I think there is some room for optimization, base64 and QR codes does
not seem to be the best fit. The reason I did not chose a "binary" QR
code is that ZXing does not seem to support this, and the intent
mechanism would probably not work either.
After the tx has been beamed, the date is missing. This is expected,
since its not part of the Bitcoin serialization.
After the tx has been beamed, the amount is zero. This is probably due
to the fact that BitCoinJ does not have access to the matching wallet.
Is there any workaround to this problem? Is the amount(s) not contained
in the tx and could be read from there? Anyway, I would expect BitCoinJ
to throw an exception rather than returning the wrong value from
tx.amount(wallet).
Currently, I cannot feed the tx into the validation process, so the
beaming of transactions in not of any particular use for the user.
I will extend my study to NFC and perhaps Bluetooth and/or Wifi direct.
Let's see which limitations we will hit there.
Jim
Is it:
1) Read QR code / de-base64 - deserialise
2) <I should have a valid tx now>
3) call peer.broadcastTransaction(tx);
wallet#confirmSend(tx);
(ok broadcastTransaction is not public but that's what
crowbars are for)
Andreas <- could you put an example tx as a QR code somewhere on a
web page so I can grab it ? I don't suppose it matters if I
double spend it, it would just get thrown out by the miners.
I presume I can forward the tx in a 'dumb' fashion i.e I
do not need to know anything else.
Jim
p.s. Love the phrase "beaming" !
--
http://multibit.org Money, reinvented
Andreas Schildbach
> What would I have to do to forward the tx to the bitcoin network ?
> Is it:
> 1) Read QR code / de-base64 - deserialise
Yes.
> 2) <I should have a valid tx now>
Syntactically valid, yes I guess so.
> 3) call peer.broadcastTransaction(tx);
> wallet#confirmSend(tx);
> (ok broadcastTransaction is not public but that's what
> crowbars are for)
I'd say first BitCoinJ should check if the tx is already known (dupe
cache, transaction pools, perhaps blockchain on a fully validating
node). If known, dump.
Then I think everything possible should be checked if the tx is not only
syntactically valid but also valid in the sense of double spending etc.
I know this is probably only possible on a fully validating node.
Finally, I think the tx should be added to the pending pool like any
other pending tx that comes over the bitcoin network. From there, it can
be re-broadcast easily.
To sum up, I think pending tx received should be handled all the same,
no matter if they arrive via the Bitcoin network, QR code, NFC...
> Andreas <- could you put an example tx as a QR code somewhere on a
> web page so I can grab it ? I don't suppose it matters if I
> double spend it, it would just get thrown out by the miners.
I mailed you a screenshot directly. The tx has yet to be confirmed (on
testnet).
> p.s. Love the phrase "beaming" !
Well, Google has been using that term for NFC-based communication
lately, so I thought I'd borrow it...
nibor
Andreas Schildbach
> Typical tx sizes of around 500 bytes seem to work quite well. However, I
> have one 1200 byte tx which obviously busts even a "version 40" QR code.
> I think there is some room for optimization, base64 and QR codes does
> not seem to be the best fit. The reason I did not chose a "binary" QR
> code is that ZXing does not seem to support this, and the intent
> mechanism would probably not work either.
I managed to cram the 1200 bytes transaction into a QR code and read it
with the ZXing scanner on a Galaxy Nexus by changing the following:
- Error correction down from L (30%) to Q (25%).
- Made sure the QR code is type Alphanumeric and make use of as much as
possible of the available codespace:
0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ$*+-./:
Characters space and % are left out because they have special meaning
and render the URL invalid. I'm not sure about the + sign, but all my
tests went well.
I derived a "Base43" variant from the Base58 implementation.
I guess 1200 byte transactions are near the sensible maximum. Goggles
already chokes at detecting it properly, while ZXing still can manage.
How common are those big transactions? The vast majority I encountered
were below 500 bytes. But what about the future, if Bitcoin grows?
What do you think of the algorithm at all? Should we propose this as a
"standard" for transactions in a QR code?
Jonny Heggheim
Andreas Schildbach
(java.util.zip.GZIPOutputStream) and BZip2
(org.apache.commons.compress.compressors.bzip2.BZip2CompressorOutputStream).
Generally, BZip2 seems to be very bad at compressing transactions. It
was only able to squeeze 1% from the 1200 byte tx.
GZip however, achieved ratios of 24% to -10%, with smaller transactions
having less compression potential than larger.
If we decide to compress, we should make this optional. I'd suggest the
first char after the scheme to serve as an compression identifier:
- = none
G = GZip
What do you think?
Jim
Andreas's transaction QR codes.
See log excerpt:
11:17:00.189 [AWT-EventQueue-0] INFO o.m.qrcode.QRCodeEncoderDecoder -
Decoded image successfully, result was :
'btctx:AQAAAAHIiNvvsBKucQi8TFzDapDZHUbowhf1320mYEsDQjS7sgAAAACLSDBFAiEA2DskY6VTOQ+crXrrxV3NBpTO2WlJqj0PuH8SQ8FVAQkCIB1klM6zShmDGYJUP+bm5rmwUU4vtk6kwqmgFD/tNLOeAUEEXJz7xrzLdlHvAaEH9enfNCnfN4rTryxNLuhWK8R2Vu+v/673f/ZBgPuKEufEi19ySpJECZJvcr6oEbJDLN09Lv////8CAFElAgAAAAAZdqkUVSVtFbjnROSYZdZPycPXyHzw9PuIrKDY8hsAAAAAGXapFMSMkCol15OWaPijoRlBec97reC5iKwAAAAA'
AbstractTradePanel#decoded 258 bytes.
01 00 00 00 01 c8 88 db ef b0 12 ae 71 08 bc 4c
5c c3 6a 90 d9 1d 46 e8 c2 17 f5 df 6d 26 60 4b
03 42 34 bb b2 00 00 00 00 8b 48 30 45 02 21 00
d8 3b 24 63 a5 53 39 0f 9c ad 7a eb c5 5d cd 06
94 ce d9 69 49 aa 3d 0f b8 7f 12 43 c1 55 01 09
02 20 1d 64 94 ce b3 4a 19 83 19 82 54 3f e6 e6
e6 b9 b0 51 4e 2f b6 4e a4 c2 a9 a0 14 3f ed 34
b3 9e 01 41 04 5c 9c fb c6 bc cb 76 51 ef 01 a1
07 f5 e9 df 34 29 df 37 8a d3 af 2c 4d 2e e8 56
2b c4 76 56 ef af ff ae f7 7f f6 41 80 fb 8a 12
e7 c4 8b 5f 72 4a 92 44 09 92 6f 72 be a8 11 b2
43 2c dd 3d 2e ff ff ff ff 02 00 51 25 02 00 00
00 00 19 76 a9 14 55 25 6d 15 b8 e7 44 e4 98 65
d6 4f c9 c3 d7 c8 7c f0 f4 fb 88 ac a0 d8 f2 1b
00 00 00 00 19 76 a9 14 c4 8c 90 2a 25 d7 93 96
68 f8 a3 a1 19 41 79 cf 7b ad e0 b9 88 ac 00 00
00 00
AbstractTradePanel#created transaction
0306a2e4cc3f2438a49078114f63a5616bc1a4bfd59f1ab8c11a695dd849b145
from mySD89iqpmptrK3PhHFW9fa7BXiP7ANy3Y
to moHAZmhbhEoJx9yPcC5EhqFD1H4tjFmm5A 0.36 BTC
to mySD89iqpmptrK3PhHFW9fa7BXiP7ANy3Y 4.68 BTC
AbstractTradePanel#result of broadcastTransaction was true
It certainly looks like it has been sent successfully.
Also, mainly for fun, I have converted the 'btctx:' string into
a sound file using the Java codec I have been working on.
(http://jdigi.net)
Here is what the bitcoin tx sounds like:
http://multibit.org/jdigi/sample_transaction.au (6.9 seconds)
I will add the transaction send code into the MultiBit code
base - you just drag a QR code onto the send swatch target and
it sends it. You have to look in the logs to see what happened -
there is no user feedback or anything.
Andreas Schildbach
> Anyway, I would expect BitCoinJ
> to throw an exception rather than returning the wrong value from
> tx.amount(wallet).
I just found out this method is part of my delta, so this is not an
issue with main BitCoinJ. Sorry about that.
Jonny Heggheim
Thilo Planz
Jim-Beam
:-)
Gary Rowe
Jim
But which one is the most bitcoin friendly ?
I am thinking speedy processor, Bluetooth, NFC, easy to program.
Preferably not pink. :-)
Recommendations ?
Jonny Heggheim
Gary Rowe
On a side note, to get start with Android programming I'd recommend "Hello, Android!" as a good introduction.
Andreas Schildbach
(NFC, Wifi Direct, Android 4 aka Ice Cream Sandwich), is the official
developer phone (will receive upgrades), has a wonderful, big and
high-resolution screen and is reasonably priced (�520 in Germany).
Jim
Thanks for your suggestions.
On 7 Dec 2011, at 14:59, Andreas Schildbach <and...@schildbach.de> wrote:
> I'd recommend the Galaxy Nexus. It has all the features you can imagine
> (NFC, Wifi Direct, Android 4 aka Ice Cream Sandwich), is the official
> developer phone (will receive upgrades), has a wonderful, big and
> high-resolution screen and is reasonably priced (€520 in Germany).
Miron Cuperman
I guess 1200 byte transactions are near the sensible maximum. Gogglesalready chokes at detecting it properly, while ZXing still can manage.
How common are those big transactions? The vast majority I encountered
were below 500 bytes. But what about the future, if Bitcoin grows?
Andreas Schildbach
for NFC.
The Android NFC stack allows for binary data being inserted into the
message, together with a mime type. The intent filtering mechanism can
listen for that mime type and dispatch any transaction directly to the
client application. Thus, I made up the following proposal:
- Bitcoin-serialize the transaction
- put these bytes as payload into an NDEF record with tnf=MIME_MEDIA
type=application/x-btctx
- put this record as the first record into an NDEF message (more records
can be added in future and should be ignored by readers for now)
- either write this message to an NFC tag or push it to a second device
Note that I skipped any compression in this case, because NDEF record
payloads can be as large as 2^32 bytes (in theory).
If you push messages, be aware that Android currently supports these two
protocols:
- com.android.npp NDEF push protocol [Gingerbread/2.3 and later]
- NFC Forum's SNEP (Simple NDEF Exchange Protocol) [Ice Cream
Sandwich/4.0 and later]
(I expect all NFC-capable Gingerbread devices to be updated to ICS
anyway. AFAIK its only Nexus S and some Galaxy S2.)
If you write messages to a tag, I can provide you with a simple Android
app to do so (incl. source). I did not find a tag writer in Android
Market that is capable of writing binary data.
As for supported tag technologies:
http://developer.android.com/reference/android/nfc/tech/TagTechnology.html
I know all of this is quite Android specific. However, I have no idea
how NFC support and messages dispatching looks like in other OSes. I
appreciate any input to make this proposal as "compatible to the world"
as possible.
Mike Hearn
You'll probably have one before me :) but yes, galaxy nexus is probably one of the best phones on the market right now, as it's one of the newest . Android 4 is great!
Mike Hearn
Yes, this is very cool work. Some things we need to bear in mind: it may be that you want to provide several transactions in a chain because your spends dependencies are not included in the chain yet. For example because they were never broadcast, as the sender was trusted to not double spend. It may also be that the fee included is not enough and the recipient is expected to provide the fee instead. So for these reasons I think nfc and/or insecure Bluetooth is the way to go .... unless you use multiple qrcodes we'll reach the limit of what can be done really fast.
Re: knowing the amount. The getValueSentToMe method should do what you want, I think. To know the input sizes means resolving the input dependencies .... a bit more work.
Andreas Schildbach
NF/QR-code initiated Bluetooth or Wifi Direct connections will dominate
this usecase as soon as the technology is more widespread. I will
investigate into the Wifi Direct scenario very soon.
As for multiple transactions, I guess this will never fit into one QR
(even a single transaction can be too large).
NFC fits well. We can put multiple Ndef records into one Ndef message,
each containing one transaction.
Would it make sense to expose a method in BitCoinJ that - for a given
transaction - tells which other transactions should probably be
transferred as well?
About the "fee provided by recipient": This concept is still very
unclear to me. Do you have any pointers for info? I don't understand how
a signed transaction can be altered by the receiver. Does the receiver
offer the fee to the sender beforehand?
On 12/13/2011 10:36 AM, Mike Hearn wrote:
> Yes, this is very cool work. Some things we need to bear in mind: it may
> be that you want to provide several transactions in a chain because your
> spends dependencies are not included in the chain yet. For example
> because they were never broadcast, as the sender was trusted to not
> double spend. It may also be that the fee included is not enough and the
> recipient is expected to provide the fee instead. So for these reasons I
> think nfc and/or insecure Bluetooth is the way to go .... unless you use
> multiple qrcodes we'll reach the limit of what can be done really fast.
>
> Re: knowing the amount. The getValueSentToMe method should do what you
> want, I think. To know the input sizes means resolving the input
> dependencies .... a bit more work.
>
> On Dec 7, 2011 5:51 PM, "Miron Cuperman" <mi...@google.com
Mike Hearn
About the "fee provided by recipient": This concept is still very
unclear to me. Do you have any pointers for info? I don't understand how
a signed transaction can be altered by the receiver. Does the receiver
offer the fee to the sender beforehand?
Andreas Schildbach
Ah I see. Thanks for the explanation.
(Do I understand correctly: The receiver would need to pay double the
fee on average, because its two transactions rather than one,
potentially taking twice the amount of space in the block.)
For the QR usecase however, I don't see the connection. The QR code is
only for getting the tx from the sender to receiver quickly (e.g. POS).
If the receiver puts another transaction on top (paying to himself),
that's his own thing. It does not need to go to any particular client
quickly, Bitcoin (re-)broadcast should be enough to get it confirmed by
the network.
Thinking about how the "receiver pays fees" could be integrated into
wallets:
1) Pending transactions that have no fee or a "maybe too small" fee
attached should get some attention marker in the GUI. The logic should
take any following pending transactions into account. As this logic is
probably non-trivial, it should be contained in BitCoinJ I think.
2) From the transaction details screen or the context menu of a
transaction, it should be possible to "Pay fee", which creates a
transaction crafted exactly for this purpose. The magic should be again
contained in BitCoinJ I think. As soon as this fee transaction is
amongst the pending transactions, the marker from step 1 will
automatically disappear.
I wonder if it will be possible for the sender to pay a fee for the
already broadcasted tx without a fee (in case he changes his mind)?
Difficult, because he does not own the addresses of the transaction
outputs. Possible solution: He could intentionally double-spend the same
outputs - identical transaction but this time paying a larger fee. Two
transactions would race for getting included into the block chain -
transaction with fee attached will probably win. Hopefully the sender
would not be punished for double spending.
Cheers,
Andreas
Mike Hearn
(Do I understand correctly: The receiver would need to pay double the
fee on average, because its two transactions rather than one,
potentially taking twice the amount of space in the block.)
- Avoid DoS attacks that bloat the size of the chain by letting transactions be priority ordered when blocks start getting full.
- More importantly, provide an incentive for mining when inflation falls. This is the reason called out in the paper.
For the QR usecase however, I don't see the connection.
1) Pending transactions that have no fee or a "maybe too small" fee
attached should get some attention marker in the GUI. The logic should
take any following pending transactions into account. As this logic is
probably non-trivial, it should be contained in BitCoinJ I think.
Alessandro Polverini
> (Do I understand correctly: The receiver would need to pay double the
> fee on average, because its two transactions rather than one,
> potentially taking twice the amount of space in the block.)
only A has fees, miners should understand that they have to include both
in the block if they want to collect A's fees since A alone is invalid.
Cheers,
Alex
Mike Hearn
For what I understand, if transaction A depends on transaction B and only A has fees, miners should understand that they have to include both in the block if they want to collect A's fees since A alone is invalid.
Andreas Schildbach
> service, for a price, which is optional. Whether I'm willing to pay
> miners not depends on the prices they set and the calculated risk of a
> double spend.
Wow, thanks for explanation! This puts whole Bitcoin into a new light
for me.
> Therefore when making a payment, you may need to include an arbitrary
> number of other transactions along with the new one.
This should be no problem with NFC. I'm looking forward to implement this.