BitCoinJ as a getwork source

[shads]

I'm looking at using bitcoinj as a source for getwork requests (part of a java pool server) but I'm having trouble wrapping my head around Generating the hash1 and midstate strings. 

I'm testing by pulling a getwork request from bitcoind, parsing the work into a subclass of Block like this:

    public MyBlock(NetworkParameters params, String data) throws ProtocolException {
        super(params, swapBytes(Hex.decode(data), 80));
        this.dataString = data;
    }   

    public static byte[] swapBytes(byte[] bytes, int trimLength) {
        byte[] rev = new byte[trimLength != -1 && bytes.length > trimLength ? trimLength : bytes.length];
        for (int i = 0; i < rev.length; i += 4) {
            byte[] chunk = Arrays.copyOfRange(bytes, i ,  i  + 4);
            chunk = Utils.reverseBytes(chunk);
            System.arraycopy(chunk, 0, rev, i , 4);
        }
        return rev;
    }

Once the block is parsed I can verify the various parts of the block header are all correct and then take the bytes and turn it back into the original data string

    public byte[] getBytes() {
        if (originalBytes == null) {
            originalBytes = new byte[80];
            Utils.uint32ToByteArrayLE(getVersion(), originalBytes, 0);
            System.arraycopy(Utils.reverseBytes(getPrevBlockHash()), 0, originalBytes, 4, 32);
            System.arraycopy(Utils.reverseBytes(getMerkleRoot()), 0, originalBytes, 36, 32);
            Utils.uint32ToByteArrayLE(getTime(), originalBytes, 68);
            Utils.uint32ToByteArrayLE(getDifficultyTarget(), originalBytes, 72);
            Utils.uint32ToByteArrayLE(getNonce(), originalBytes, 76);   
        }
        return originalBytes;
    }

        byte[] bytes = block.getBytes();
        bytes = swapBytes(bytes, 80);
        byte[] padded = block.padForSHA256(bytes);
        L.println(Utils.bytesToHexString(padded));   

I've tried the midstate calculation using two different methods (SUN and bouncy castle implementations).  Both produce an identical result but neither matches the midstate from the original getwork no matter how much I swap endianess or reverse bytes or otherwise mess with them:

    public byte[] getMidstate1() {
        MessageDigest digest;
            try {
                digest = MessageDigest.getInstance("SHA-256");
                digest.getProvider();
                digest.update(getBytes(), 0, 64);
                byte[] midstate = digest.digest();
            return midstate;
            } catch (NoSuchAlgorithmException e) {
                e.printStackTrace();
                return null;
            }
    }

    public byte[] getMidstate() {
        SHA256Digest sha = new SHA256Digest();
        sha.update(getBytes(), 0, 64);
        sha.update(getBytes(), 64, 16);
        byte[] midstate = new byte[32];
        int finalResult = sha.doFinal(midstate, 0);
        return midstate;
    }

Any help would be greatly appreciated.  Also I don't know where to start with hash1.  It always seems to be the same string:
"00000000000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000000010000"

I've looked through the original daemon code but it doesn't really give many hints.  Doesn't help that I don't know c++
https://github.com/bitcoin/bitcoin/blob/master/src/main.cpp

[shads]

Just to clarify in that last getMidstate() method this line :

sha.update(getBytes(), 64, 16);

should be there.  I've tried without it, it was just something I was messing with when I happened to copy and paste it.

[Mike Hearn]

I'm not sure what you're trying to achieve here. BitCoinJ is not capable of mining so it can't answer getwork requests. If you only care about receiving/parsing them I'm not sure why you care about reconstructing the midstate?

What would a java pool server do exactly?

--
Google Switzerland GmbH

[shads]

Hi Mike,

The pool server itself is basically java version of pushpool.  It simply provides miner clients with work and keeps track of submitted shares.  I'm using my subclass of Block to verify returned work meets a lowered difficulty target before sending upstream.  However I'd really like to be able to use BitCoinJ to generate work internally and avoid all the overhead of HTTP connections to a bitcoin daemon.  I realize there's a lot more to it than just calculating midstate from an existing block but I'm plodding through it step by step and this is the step I've made it to so far.

BTW If there's anything in the roadmap along the lines of what I'm doing let me know as I'd be happy to contribute to code to the project.

[Mike Hearn]

It's way too early to be trying to implement getwork serving. BitCoinJ is a long way from being able to replace the C++ server I'm afraid. It's months of work, at minimum. If you have any non-trivial amount of mining taking place against a BitCoinJ based node you open up the network to attack and could cause large splits in the chain.

--
Google Switzerland GmbH

[Alex]

It's way too early to be trying to implement getwork serving. BitCoinJ is a long way from being able to replace the C++ server I'm afraid. It's months of work, at minimum.

Hello Mike,
I'm interested in studying the bitcoinj sources to better understand the protocol, and I'm interested too in a getWork() implementation.
Just to better understand the status of the project, would be you so kind to explain which kind of steps bitcoinj would need to correctly implement a getWork() method?
I.e.: if someone would like to implement it, what would be the requisites?

 

If you have any non-trivial amount of mining taking place against a BitCoinJ based node you open up the network to attack and could cause large splits in the chain.

Why that?

Thanks,
Alex

[John Sample]

>> If you have any non-trivial amount of mining taking place against a
>> BitCoinJ based node you open up the network to attack and could cause large
>> splits in the chain.
>
> Why that?

Since bitcoinj only stores the small subset of transactions that apply
to the user you have no way of knowing whether the transactions you
construct a block from are double spends or otherwise invalid.
If you include invalid spends full nodes will reject your block but
other simple mode nodes might not. These nodes in turn will reject
mainline blocks and be on a forked chain.

[Mark Tolley]

Firstly, hi all.

> BitCoinJ is a long way from being able to replace the C++ server I'm afraid. It's months of work, at minimum.

I realise that bitcoinj is currently targeted at mobile applications, but I'm another person interested in getting bitcoinj fleshed out and working as a more capable network node.

> Just to better understand the status of the project, would be you so kind to explain which kind of steps bitcoinj would need to correctly implement a getWork() method?

It seems like storing the complete blockchain & history would be a good first step in this direction.

What I'm wondering is if the main developers are happy to assist if a few of us want to go down this road; are the people on this list willing to help with getting us (me at least) up to speed on the code, advise on the design & crucially accept patches to this end?

If I can get a positive response I'm hoping to be able start a new thread and rough out a few ideas I've been thinking about. However I'd understand if the developers would rather keep the core bitcoinj project focused on the mobile stuff, in which case I'll fork a seperate project to work on it.

Thanks,
Mark

[John Sample]

I don't think anyone is opposed to bitcoinj eventually reaching full
node status.
It is certainly something I want to see happen.... in fact that was
why I started contributing. I'm not (usually) a mobile developer I
just want to see core bitcoin tools being developed in more languages.
However, it is a daunting task and not something that will happen overnight.
Mike has a post somewhere describing how dangerous a non-satoshi full
node could be to the network if it was even slightly off.

In the long run it would be nice if the developer had the option of
using bitcoinj in full or spv mode. A generalized API that could be
used either way would be a boon to the community.

Storing the full block chain would be a good start. Its also going to
be tough since it also deals with how transactions are stored.
Right now the transactions are kind of locked to being represented in
serialized objects and quite a bit of plumbing needs to be swapped to
change that.

I would think as long as making the API fuller featured doesn't hurt
mobile capabilities it could only be a good thing.

[Jeffrey Perz]

I would also like to see a full Java implementation, but I'd settle
for a basic Java GUI to start if it could be done in parallel to the
people working on more pressing issues. I've started to roll my own
for a project I'm working on that uses BitcoinJ, but I'd rather
leverage the bits I need or submit patches rather than re-invent the
wheel. I find the inline documentation on this project leaps and
bounds better than anything I saw in the mainline source, but maybe
there are other documents that I should have been looking at. Hell, it
still bothers me that I couldn't figure out how to compile the
mainline client, never mind changing it.

[Jim]

@Jeff re: a Java GUI front end to bitcoinj. I have been working on a
gui front end in a spin-off project called 'MultiBit'.
(www.multibit.org). This uses the bitcoinj code as the back end and is
in Swing.

This week I have hooked up the bitcoinj block downloading and will start
on the wallet display/ send/ receive soon so it is quite near to being a
minimal viable product. You might want to have a look at that.

--
multibit.org : coming soon : bitcoin desktop client : powered by bitcoinj

[Mike Hearn]

Yes, we don't want to see a fork. There has been persistent long term interest in reimplementing the full node, so I suppose it's going to happen eventually. I'm happy to assist and advise even if I don't work on that goal directly.

There are several tasks that are shared between lightweight/mobile and full modes. It makes sense to knock them off first as everyone using the library will benefit regardless of their goals. A few I can think of:

• Transaction fees: We need a getFee() method on Transaction, a way to set fees when creating spends (these are easy), and less easy, the Wallet needs to know how to create spends with automatically selected fees such that the network will always relay them and it should be able to minimize the fees paid through smart coin selection. Along with unit tests for all of that of course. Once that's done the Block checking code can implement the inflation formula and then verify the value on the coinbase tx. Potentially in the long term, BitCoinJ could automatically split up or combine your outputs to balance the desire for low fees with privacy (eg if you make a payment to a friend with a huge output they can find out that you are rich, which isn't what you want).
  
  
• Value capping: there has been a long-standing TODO to introduce a value abstraction (currently we use BigIntegers). As part of doing that, we'd implement the MAX_MONEY checks to ensure we never end up with negative value or more than 21 million coins. Currently if we get a transaction with an invalid value field we wouldn't notice. Checking this doesn't hurt mobile users and is required for a full implementation.
  
  
• Rule audit: the protocol rules are very complex, fortunately Hal has done a great job of documenting them. Go over the rules and ensure all the ones that are implementable with todays code are actually done and covered by unit tests. Obviously some require full storage of the chain, but others don't, eg transactions included in blocks should be checked for finality. We skip quite a few today because we're relying on difficulty as a proxy for validity.
  
  
• Tx simplification: refactor and flesh out the equivalent of the C++ SignatureHash function. Currently only SIGHASH_ALL is supported in BitCoinJ and only for signing. The Transaction class should know how to clone itself into a simplified form given the set of SIGHASH flags, ready for signing and/or verification. This work is a core part of OP_CHECKSIG and is needed to support more exotic contracts, which some other people (including myself) are interested in playing with once the mobile apps are more solid.
  
  
• Script editing API: right now the Script class is only half baked. It knows how to parse scripts, but it's not very easy to build up new scripts or do pattern matching on them. Editing scripts is needed for interesting contracts.
  
  
• Better wallet: either we need a keypool so backups don't have to be done continuously, or we need to implement deterministic wallets so you only need to back up a single key. We need to store creation times of keys so we can always efficiently rebuild the wallet.
  
  
• Misc features: wallet encryption (being worked on by John), better multiple wallets support, storage/usage of addr structures, alert broadcasts, etc.

Those things are all shared goals. Separately, to do a full implementation you'd need to tackle these tasks as well, but they would come later:

• Script execution: implement all the opcodes, along with unit tests for each one, tests for error handling and other things. BitcoinJS (stefans javascript implementation) has some example unit tests you could start with.
  
  
• Fully checking block store: a BlockStore is given the full block to work with, not just the headers. You'd start by implementing a database backed block store that just did the headers: there is a Derby implementation on a branch (may have bitrotted a little) which shows how to do this. Then you'd add storage of the full transactions. Then you'd implement checking of the transactions dependencies to ensure they exist and the outputs aren't spent. Then you'd run the scripts. Then you'd have to implement full re-org handling. Todays re-org handling only manages transactions in the wallet. You'd have to ensure all the spent pointers in your store are updated too.
  
  
• Start relaying: announce valid blocks and transactions to peers that didn't get them yet.
  
  
• Memory pool: we'll be grabbing relevant transactions off the wire and storing them in the wallet long before anyone gets to this, but the memory pool is a bit different. You need to:
• store all transactions broadcast
• verify them
• ideally cache their validity so we don't have to re-verify them once they appear in a block
• implement the finalization rules
• implement tx replacement via sequence numbers
• implement the ability to put together a new block that drains the pool up to the block size limits, selecting based on (ideally pluggable) fee policies
• make it overload proof, ie, if you start running out of memory, drop or save to disk transactions that won't confirm for a long time or are very tiny.

There are certainly other tasks I forgot about.

And of course, keep up with any features and protocol changes implemented in the Satoshi client.

Now you see why I say it's a lot of work :-)

[Mark Tolley]

On 28/07/2011 12:21, Mike Hearn wrote:

Now you see why I say it's a lot of work :-)

Oh wow, you weren't kidding! Thank you for taking the time to make such a detailed reply.

• Transaction fees: We need a getFee() method on Transaction, a way to set fees when creating spends (these are easy), and less easy, the Wallet needs to know how to create spends with automatically selected fees such that the network will always relay them and it should be able to minimize the fees paid through smart coin selection. Along with unit tests for all of that of course. Once that's done the Block checking code can implement the inflation formula and then verify the value on the coinbase tx. [...]
  

Top of the list seems like a good place to start. I'll try to have a look into that this evening then.

• Value capping: there has been a long-standing TODO to introduce a value abstraction (currently we use BigIntegers). As part of doing that, we'd implement the MAX_MONEY checks to ensure we never end up with negative value or more than 21 million coins. Currently if we get a transaction with an invalid value field we wouldn't notice. Checking this doesn't hurt mobile users and is required for a full implementation.

Looks like work has been started on this here: https://github.com/hegjon/bitcoinj/commit/61a2c0a7c2715f772a481da58a1c575c67bebbc0
Is there a reason this has been neglected for a couple of months? Although the sanity checks you mention don't seem to have been done yet, this should be a decent foundation to build on.

Cheers

[Andreas Schildbach]

On 07/28/2011 02:39 PM, Mark Tolley wrote:

>> * *Transaction fees:* We need a getFee() method on Transaction, a

>> way to set fees when creating spends (these are easy), and less
>> easy, the Wallet needs to know how to create spends with

>> automatically selected fees such that the network will always...

>>
> Top of the list seems like a good place to start. I'll try to have a
> look into that this evening then.

FYI: The "setting fees" part is implemented in my branch. Have a look at:

http://code.google.com/p/bitcoinj/issues/detail?id=45#c5

>> * *Value capping:* there has been a long-standing TODO to

>> introduce a value abstraction (currently we use BigIntegers). As
>> part of doing that, we'd implement the MAX_MONEY checks to
>> ensure we never end up with negative value or more than 21
>> million coins. Currently if we get a transaction with an invalid
>> value field we wouldn't notice. Checking this doesn't hurt
>> mobile users and is required for a full implementation.
>>
> Looks like work has been started on this here:
> https://github.com/hegjon/bitcoinj/commit/61a2c0a7c2715f772a481da58a1c575c67bebbc0
> Is there a reason this has been neglected for a couple of months?
> Although the sanity checks you mention don't seem to have been done yet,
> this should be a decent foundation to build on.

About the BigInteger replacement: will this actually replace BigInteger
or will it encapsulate BigInteger? I am a bit ancious that instantiating
another couple of thousand objects will hit mobile devices too bad.
There are already memory problems, although I currently can't tell if
they are the result of a bug.

So if it is possible at all, I'd vote for not encapsulating any objects
into the future "Bitcoin" object.

[Mark Tolley]

On 28/07/2011 14:38, Andreas Schildbach wrote:
> FYI: The "setting fees" part is implemented in my branch. Have a look at:
>
> http://code.google.com/p/bitcoinj/issues/detail?id=45#c5

Thanks, this helped get me started.

> About the BigInteger replacement: will this actually replace BigInteger
> or will it encapsulate BigInteger? I am a bit ancious that instantiating
> another couple of thousand objects will hit mobile devices too bad.
> There are already memory problems, although I currently can't tell if
> they are the result of a bug.
>
> So if it is possible at all, I'd vote for not encapsulating any objects
> into the future "Bitcoin" object.

My understanding is that a Bitcoin class would wrap BigInteger; make
sure it was constrained to acceptable values, and provide convenience
methods for creating, printing, etc bitcoin values. If the only field in
this class was a reference to the encapsulated BigInteger, that's only
what, 8 bytes overhead per value?

I guess if performance is really a problem then most, if not all, of
these convenience methods could be made static and the class would
simply operate on existing BigIntegers.

[Jeffrey Perz]

Doh, found it on the Team page. My last reply was supposed to be for Jim. Anyway, thanks again!

On Friday, 29 July 2011 02:47:04 UTC-4, Jeffrey Perz wrote:

Thanks, looks like a good start. Is there a public repo I can use? I only need read-only at this point.

[Jim]

@Jeffrey. No problem ! Let me know if I have missed putting anything
into git or if you have any problems building it.
I have spent a bit of time splitting it out into a model/ view/
controller structure in the style of a Struts web app so there are a few
small 'glue' classes to navigate.

Once I have the Swing rendition done and reasonably stable I plan to add
another set of views rendering to vanilla HTML (Tomcat + Struts + JSPs
probably) so that there is a JEE web application for people to use too.

[Mike Hearn]

My understanding is that a Bitcoin class would wrap BigInteger

It'd probably just wrap a long actually. BigInteger is overkill given the value caps.

 

make sure it was constrained to acceptable values, and provide convenience methods for creating, printing, etc bitcoin values.

Yes, exactly.

The cost of GC load depends a lot on device version. Older Androids have fairly weak mark/sweep collectors. Modern devices have much more advanced (generational concurrent) collectors. I agree we need to keep an eye on this. The API should probably be phrased in terms of objects with a long accessor as an option. Inside TransactionInput/Output we can just use longs and avoid the object cost until a user asks for one.

Right now though I suspect any GC/memory issues are caused elsewhere.

--
Google Switzerland GmbH

[Mike Hearn]

Looks like work has been started on this here: https://github.com/hegjon/bitcoinj/commit/61a2c0a7c2715f772a481da58a1c575c67bebbc0

I don't remember it being submitted, but maybe I lost track of it.

 

Is there a reason this has been neglected for a couple of months?

There were higher priorities and I wasn't sure if that was quite the best way to do it.

--
Google Switzerland GmbH

[Jonny Heggheim]

Hi Andreas!

About the BigInteger replacement: will this actually replace BigInteger
or will it encapsulate BigInteger? I am a bit ancious that instantiating
another couple of thousand objects will hit mobile devices too bad.
There are already memory problems, although I currently can't tell if
they are the result of a bug.

So if it is possible at all, I'd vote for not encapsulating any objects
into the future "Bitcoin" object.

This was just a refactor that used the code that was already there.

I understand that there are conserns of memory limits on Android, is it possible to give some numbers?

Btw, having a trivial Bitcoin class does not mean that it must be used everywhere, if we run into memory problems (like having a milion Bitcoin objects in an array), then that code could store the values as long instead of Bitcoin objects.

Jonny

[Andreas Schildbach]

> About the BigInteger replacement: will this actually replace BigInteger
> or will it encapsulate BigInteger? I am a bit ancious that instantiating
> another couple of thousand objects will hit mobile devices too bad.
> There are already memory problems, although I currently can't tell if
> they are the result of a bug.
>
> So if it is possible at all, I'd vote for not encapsulating any objects
> into the future "Bitcoin" object.
>

> Btw, having a trivial Bitcoin class does not mean that it must be used
> everywhere, if we run into memory problems (like having a milion Bitcoin
> objects in an array), then that code could store the values as long
> instead of Bitcoin objects.

Yes, you are right. And since Mike asserted that we can use long instead
of BigInteger, my objections are off the table.

Should we start collecting what operations we'd need on a "CoinsAmount"
object?

[Jonny Heggheim]

Should we start collecting what operations we'd need on a "CoinsAmount"
object?

Normally I like that the code and tests design what is needed. Looking back at the commit located here https://github.com/hegjon/bitcoinj/commit/61a2c0a7c2715f772a481da58a1c575c67bebbc0, all methods that were created in the "Bitcoin" class were in use. I would start with the Bitcoin and BitcoinTest code and remove/add code as we move on.