Possibly a stupid question, but has anybody here talked 1-on-1 with one of the big pool operators?

[Chris Wilmer]

I just assume these conversations... say, between Gavin (or Mike) and the appropriate person at 21 inc, or KnC Miner must have happened (let's put aside the Chinese pools where there may be additional communication obstacles)... but maybe they haven't? I was *really* surprised to see that 21 inc wasn't supporting BIP101... has anyone made a phone call to Balaji (CEO of 21) about this? He's clearly advocating for lots and lots of tiny transactions... what's his view on how that would actually work?

I'd volunteer to make these phone calls myself, but (understandably) nobody would care what I think, but I assume people would at least talk to Gavin.

Again, maybe this has all happened already and I'm in the dark. It's just something I've been wondering a lot about.

[Mike Hearn]

I haven't talked with any of the pool operators except Slush.

I don't know how many pool operators Gavin talked to, if any. They are not exactly easy to get hold of or even find. There is a summary here from some time ago:

https://bitcoinmagazine.com/articles/major-mining-pools-make-stand-bitcoin-xt-fork-support-bip-100-grows-1440537258

--
You received this message because you are subscribed to the Google Groups "bitcoin-xt" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bitcoin-xt+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[NxtChg]

>I haven't talked with any of the pool operators except Slush.

Wasn't the next step "to win the minds of miners and pools"?

We know you, guys, are busy, but as Chris said, nobody's gonna listen to average Joe.

[Chris Wilmer]

@Mike:

I have Balaji's email address... I even emailed him a year or so ago to congratulate him on co-founding the Stanford Bitcoin Research Group and he responded. Why don't you email him? 

Just so you know, I feel terrible saying "why don't you do X" because you are already doing way more than me for the cause, so-to-speak, but I beyond running an XT node at home, I'm not sure what else I can do myself at the moment.

[Joseph Gleason ⑈]

There is value to even a relative nobody reaching out with a message along the lines of "Are you interested in bitcoin xt?  Are there any questions I can get answered for you?  If I don't know the answers, I'll find out and get back to you.  We would like pool operators support, please let me know if you have any concerns."

I have no idea what the concerns might be, but lets say it is about elephants.

Lets say I am a pool operator and I am concerned that XT might not allow me to keep my elephants blue.  Being not a noob, I don't just want to email a list or question IRC.  I want do my non-noob asking good questions search first.  So I search.  And I find a 200 post discussion about elephant colors.  I find some people talking about encoding pictures of grandma into the color patterns.  I find others saying that will destroy the world because grandma is mean.  Luke Jr. will have posted that grandma is fundamental or antithetical to bitcoin in some way.  Someone will have said that the future of everything is purple.

So I don't have my question answered and give up because I have things to do and don't care that much and catching up on the discussion enough to ask is hard work.  But maybe if someone emailed me and I could just ask them to figure it out and summarize for me, that would be grand.

(Elephants should be all colors simultaneously)

[Tom]

On Monday 28. September 2015 23.56.20 Joseph Gleason ⑈ wrote:
> I have no idea what the concerns might be, but lets say it is about
> elephants.

Why elephants, why not bike-sheds? :P

[Mike Hearn]

Yeah, there's definitely an element of this problem. Miner communication is pretty poor. It says on the XT website that miners should feel free to email us here and ask any questions, get advice on optimisation etc, but I suspect most miners haven't read it.

[NxtChg]

>It says on the XT website that miners should feel free to
>email us here and ask any questions, get advice on optimisation
>etc, but I suspect most miners haven't read it.

Wow, this is our strategy to win? "Call us, maybe"?

I think, more people should use this approach:
* make a new spreadsheet app
* put email address on the site
* wait for people to write and ask why your app is better than Excel
* ???
* PROFIT!!!

This sure instills confidence...

[Mike Hearn]

I didn't say that was the strategy, it was just an observation.

I'll be away for the rest of this week.  When I'm back I'll go talk to some mining pools. Originally Gavin said he'd do this after the XT launch but I think both of us ended up being so busy that we didn't do it. Also some of the pools laid out specific objections. So far their objections seem to fall into the following categories:

• We are afraid of higher orphan rates. Both me and Gavin plan to look at protocol upgrades to reduce orphan rates. This may or may not help, as it's unclear to me to what extent this argument is data driven e.g. if there's some level of orphaning due to block size they'd accept, or if they will have the same concerns no matter how optimised the protocol gets.
  
  
• We are afraid of being DDoSd. Nothing we can do about this one except encourage them to work with their ISPs to sink the packet floods.
  
  
• We are afraid of social instability and/or price falls if there is a hard fork that some people disagree with. This is a tough one. It boils down to "we like decentralisation on the assumption that everyone always agrees", although decentralisation is actually about mechanisms to resolve disagreements amongst groups of people. There's no such thing as a decentralised system where everyone always agrees on everything just through spontaneous intellectualism.
  
  I think this one just has to be tackled by laying out the arguments for why it's not true. For instance P2SH was controversial and triggered with a 55% voting threshold. The sky didn't fall. But most people don't realise that.
  
  
• We are waiting for Bitcoin Core to do something.
  
  It seems unlikely that any proposal will get committed and tested by the 0.12 release now given the timeline Wladimir proposed. Indeed this is why Gavin and I started the debate back in April. Given that Core 0.13 would presumably not come out for another six months, that'd place the earliest Core does anything around June of next year (unless they decide to make major changes in a point release).
  
  Not much we can do about this one either.

So as you can see, whilst we'll go and talk to some miners, the only objection that might actually be resolvable through action is lower orphan rates.

[Tom Zander]

On Tuesday 29. September 2015 14.59.23 Mike Hearn wrote:
> So as you can see, whilst we'll go and talk to some miners, the only
> objection that might actually be resolvable through action is lower orphan
> rates.

Any technical objections, yes. I fully agree with your assessment.

The part that is not technical is arguably more important. It is the feelings
that humans crave when choosing sides;
Do Mike and Gavin seem sure of themselves.
Do Mike and Gavin seem trustworthy and do I feel they won't cheat me or let me
down.

All emotional points, and arguably that’s how most of the business decisions
get made.

So, yeah, I feel that if you avoid falling in the main trap that many
engineers fall into; showing only the bugs and problems and forgetting to
demonstrate the good parts, you're talking to miners will surely create some
good will.


ps. Todd is quite good at some of these points. But he falls down at others.

[Gavin Andresen]

On Tue, Sep 29, 2015 at 8:08 AM, NxtChg <nxt...@hush.com> wrote:

Wow, this is our strategy to win?

Can we please stop the "us versus them / winners and losers" rhetoric? Lets try to 'keep calm and bitcoin on.'

Here's what's going on with me:

I'm going to ignore the debate for a couple of weeks and write some code (pre-announcement of 'weak blocks' to optimize block propagation), mostly for my mental sanity, partly because showing insanely fast propagation of very large blocks eliminates the last technical objections.

Then I'm going to try to work with Greg/Pieter/Wlad/Jeff/peanut gallery to try to get a BIP based on resource usage that gets rough consensus. The BIP101 code includes resource usage constraint code that needs to get written into a BIP anyway.

If that succeeds, fantastic, we've got a path forward-- implement in XT, port to Core (and maybe backport six times for miners who insist on running old code), we all move forward.

If it is STILL impossible to get rough consensus, then another round of talking to exchanges and merchants and miners and either convincing them that BIP101 is the best choice or finding out what they want to do.

--

--
Gavin Andresen

[NxtChg]

>Can we please stop the "us versus them / winners and losers" rhetoric?

I would agree if the block limit was the only issue.

Alas, it's just a manifestation of a deeper conflict of what Bitcoin is and where it should go.

[Jonathan Toomim (Toomim Bros)]

• We are afraid of higher orphan rates. Both me and Gavin plan to look at protocol upgrades to reduce orphan rates. This may or may not help, as it's unclear to me to what extent this argument is data driven e.g. if there's some level of orphaning due to block size they'd accept, or if they will have the same concerns no matter how optimised the protocol gets.
  

If their concern is not data-driven, we can show them data. 

One thing that you can bring up is that when orphan rates go up for everybody, mining difficulty goes down. That is, average blocks mined successfully will not change. Larger blocks will make the reward gradient steeper for orphan rates, but it will not punish the average miner.

• We are afraid of being DDoSd. Nothing we can do about this one except encourage them to work with their ISPs to sink the packet floods.
  

DDoS attacks seem to have died down. 

Suggest they put the XT/BIP101 pools on a separate IP address (and ideally a separate machine) rather than simply on a separate port. That way, DDoS attacks will be easier to filter out, and will be less likely to take down their whole infrastructure.

• We are afraid of social instability and/or price falls if there is a hard fork that some people disagree with. This is a tough one. It boils down to "we like decentralisation on the assumption that everyone always agrees", although decentralisation is actually about mechanisms to resolve disagreements amongst groups of people. There's no such thing as a decentralised system where everyone always agrees on everything just through spontaneous intellectualism.
  

Stability => stagnation => obsolescence. If we want bitcoin to stay relevant, we need to be able to make changes, even when only 75% of current users support them.

---

There is also the "Large blocks can be used to assist selfish mining attacks" objection. It might be possible to disprove this on testnet. 

Also, the "CreateNewBlock's performance is crap with big blocks anyway" argument. This is mostly an incorrect objection, since CreateNewBlock's current problems are with large mempools, not large blocks, and large blocks reduce mempool size. However, I expect most miners won't distinguish between the two, as they probably saw their CNB/GBT latency increase during the stress tests when both blocks and backlogs were big. Still, even though it's an invalid, it's still a real motivation, and we can fix it. I bet miners would be more likely to join XT and BIP101 if they gained an immediate revenue benefit from slightly lower orphan rates with a faster CNB.

[Mike Hearn]

Also, the "CreateNewBlock's performance is crap with big blocks anyway" argument

I did some profiling of this weeks ago. It looks like it should be fairly straightforward to make it much faster. The slowness does not appear to be fundamental.

[Tom Harding]

On 9/29/2015 6:32 AM, Gavin Andresen wrote:
>
> If it is STILL impossible to get rough consensus, then another round
> of talking to exchanges and merchants and miners and either convincing
> them that BIP101 is the best choice or finding out what they want to do.

Gavin, as an operator of one of the network of 600 XT nodes out there,
can you clarify for me:

- Does anything you have seen or heard in the last 2 months make YOU
feel that changes are needed BEFORE BIP 101 is adopted? Or, are the
weak blocks work and additional meetings just to convince others, with
the actual need for changes far in the future? In your view.

- Do you encourage miners to produce BIP 101 compatible blocks right now?

- Do you encourage full node operators to switch to Bitcoin XT or
another BIP 101 compatible build right now?

- In your view, does BIP 101 need to be changed before being
activated? Or has all the discussion just strengthened your beliefs?
Or somewhere in between?

A statement on the "state of BIP 101" would help me.

[Gavin Andresen]

On Tue, Sep 29, 2015 at 3:29 PM, Tom Harding <to...@thinlink.com> wrote:

On 9/29/2015 6:32 AM, Gavin Andresen wrote:

If it is STILL impossible to get rough consensus, then another round of talking to exchanges and merchants and miners and either convincing them that BIP101 is the best choice or finding out what they want to do.

Gavin, as an operator of one of the network of 600 XT nodes out there, can you clarify for me:

 - Does anything you have seen or heard in the last 2 months make YOU feel that changes are needed BEFORE BIP 101 is adopted?  Or, are the weak blocks work and additional meetings just to convince others, with the actual need for changes far in the future?  In your view.

The only change I can think of that is strictly needed:

  + Modifying Matt's fast relay network so it will handle >1MB blocks.

However, if we actually want miners to be willing to produce >1MB blocks then optimizations like weak blocks are necessary.

 - Do you encourage miners to produce BIP 101 compatible blocks right now?

Mike and I may disagree on this, but I have consistently said we have to have rough consensus with all the big exchanges and bitcoin-accepting businesses before asking miners to switch. So I haven't been asking miners to switch. I think it is sad to see miners "voting" for BIP100 in their coinbase transactions (for example) when there isn't even an implementation of BIP100 available yet.

The big exchanges and bitcoin-accepting businesses generally just want the developers to come to consensus. They want the block size limit to grow (or go away entirely), but they haven't been willing to dive into the technical debate on exactly how and when.

 

 - Do you encourage full node operators to switch to Bitcoin XT or another BIP 101 compatible build right now?

Yes. Ideally, we'd see 90% of full nodes running BIP101 (just about everybody except the miners).

 - In your view, does BIP 101 need to be changed before being activated?   Or has all the discussion just strengthened your beliefs?  Or somewhere in between?

I don't think BIP101 needs to change, but I do need to write a second BIP that describes the resource limits (I've asked twice for collaboration from the other increase-the-blocksize authors, I'll try again in a week or so).

--

--
Gavin Andresen

[Jonathan Toomim (Toomim Bros)]

On Sep 29, 2015, at 8:30 AM, Mike Hearn <he...@vinumeris.com> wrote:

Also, the "CreateNewBlock's performance is crap with big blocks anyway" argument

I did some profiling of this weeks ago. It looks like it should be fairly straightforward to make it much faster. The slowness does not appear to be fundamental.

Agree. Tons of low-hanging fruit in CNB. 

You probably have a pretty good idea of how to fix it already, but in the unlikely event that I saw something you didn't, here are my ideas:

- Sort by transaction fee and/or priority in CNB before checking inputs. Only check inputs on a txn when preparing to add it to a block; don't check inputs on txns that can't fit in the block anyway due to low priority and congestion. Reading a txn's fee is a lot faster than checking its inputs. We may need to create two priority queues (one by fee, one by bitcoin days destroyed) in order to implement the 5% fee-free reserved space, but that will still probably be a lot faster.

- Use 

#pragma omp parallel for 

or something like it to parallelize some of the loops.

- Pre-sort or pre-filter outside of CNB.

I'm not quite competent enough to safely implement these changes myself yet. Sorry. Maybe soon.

[Mike Hearn]

You probably have a pretty good idea of how to fix it already, but in the unlikely event that I saw something you didn't, here are my ideas:

Thanks.

The conclusion I reached is that if you profile getblocktemplate, virtually all the CPU time is being taken up by checks for edge cases, because the quickest way to check for the edge case was to apply a general and massive hammer that does a lot of unnecessary work. In other words, it applies a massive hammer to a couple of (what look like) tiny nails.

I haven't looked at it more in depth, but it looks to me like replacing those massive-hammer checks with some more specific and tightly written code could eliminate almost all the overhead, without any parallelisation or other more general techniques being necessary.

[Mike Hearn]

I broadly agree with Gavin, modulo these two aspects:

 

However, if we actually want miners to be willing to produce >1MB blocks then optimizations like weak blocks are necessary.

I think >1mb blocks work fine today even without optimisations. The issue is in the definition of the word "fine". Miners dislike orphans, but optimising the orphan rate to be as close as possible to the theoretical limit (i.e. speed of light bound) is a potentially endless project. It isn't something that BIP101 should be seen as depending on. Rather, optimising orphans should just proceed in parallel and the sizes of blocks should depend on user demand.

Mike and I may disagree on this, but I have consistently said we have to have rough consensus with all the big exchanges and bitcoin-accepting businesses before asking miners to switch.

I agree and disagree. I agree that getting businesses to agree first is a good thing to do, but I think that's already done. So far they all either support BIP101 or have the more general stance of "we want the limit to rise but we also want everyone to agree first". 

Well, we all like motherhood and apple pie. It's not like Gavin and I are standing around shouting "DISAGREEMENT IS AWESOME!". Disagreement is fundamental in any project once it passes a certain scale, all you can do is manage it.

So we can't do any better than this. The job is done, from the business/exchange perspective. We may as well move on to talking to miners.

[Tom Zander]

On Wednesday 30. September 2015 13.39.57 Mike Hearn wrote:
> So we can't do any better than this. The job is done, from the
> business/exchange perspective. We may as well move on to talking to miners.

I'm a big proponent of that approach.

Gavin says he wants to take a break from politics, and he wants to wait for
the other devs to work on the promises they made in Canada.
Fine, that’s his prerogative. Although I'm sceptical because I've seen so many
delay tactics, this just looks like another. How many second chances can you
give?

The point here is that we are moving forward, and some core devs are
struggling very hard, but are now finally moving.
To stop pushing because you see movement is not a strategy I understand.

I think talking to miners is by far the best strategy. Show you know what you
are doing, use that show of movement from some core devs as an indication that
with the miners support you can get everyone on one page.

[Chris Wilmer]

I still don't really understand what the obstacles are to talking to 21 Inc about their support (or not) for BIP101. If you are going to talk to miners, start with the lowest hanging fruit (easiest people to find and talk to that have meaningful hash power). 

in...@21.co

Contact info for Balaji is right here:

http://balajis.com/about/

I know he's a busy guy but I imagine he'd be delighted to spend time talking to either of you (Mike/Gavin). If he doesn't support BIP101 please let us know why!

-Chris

[Cypher Doc]

great question Chris.

[Mike Hearn]

FWIW I am now in the process of talking to miners. I've emailed Balaji but not heard back yet.

--

[Chris Wilmer]

Thanks Mike!

[Cypher Doc]

just to throw in my 2 cents on the orphan issue.

why do orphans need to be viewed negatively in the case of big blocks or no limit?  according to Peter_R's paper and my own views, orphaning is a natural check against attacking bloat blocks.  miners need to be educated that this is a good thing; if a rogue miner or spammer tries to attack the network by sending a bloat block, he will get punished.

On Monday, September 28, 2015 at 2:03:23 PM UTC-7, Chris Wilmer wrote:

[Jonathan Toomim (Toomim Bros)]

It's mostly political. Currently the largest pools (Antpool, F2pool, BTCChina, BW.com) are also the pools with the highest orphan rates. They don't want to be put at a disadvantage. 

This is, unfortunately, one of the problems with using miner voting. People in power choose rules that keep them in power.

Oh well. I can't think of a better way to do it.

[Chris Wilmer]

See, I'd rather not focus on that problem at the moment. We should be wondering why 21 Inc, KnC Miner, and BitFury are not going with BIP101. I don't think they have any connectivity problems.

[Yu Meitian]

While I was several times trying to note some issues for instance in the Bitcoin-Core IRC channel, I got harsh answers based on the motto "when average Joe doesn't understand he should not use bitcoins". In my opinion this is a terrible, unprofessional attitude and I personally hope we will have some more professionalism with Bitcoin-XT.

It is simply not enough to be a really good programmer, but to NOT understand economic basics of how an internet based product being successful. And it seems like a part of the programmer don't understand beside Gavin, Mike, Jeff as far as I can judge based on their behavior.

So what do you think about following:

1. GIT-Admin-Access for well known economist. The idea is to have someone in the team to beacon from another point of view than programmer do.

2. Poll like integration for Bitcoin clients. A possibility for everyone with Bitcoin to vote on important decisions. Vote faking could be prevented with a stake like system. So your vote will be proportional to your stake. Similar to share holder system.

3. Possibility to reward Developer, maybe through a fund.

4. Letter of Manifest which have to be signed from every member with GIT-Admin-Access. Which includes stuff like: A - Programmer and all guys with GIT-Access will always respect the opinion of the Bitcoin-User majority. B - The absolute number of bitcoins ever produced can't be changed.

5. Bitcoin-XT client should ask on installing if you want the full version or the light version. The light version should work without the need of downloading the complete blockchain. In general the whole system should be

[NxtChg]

>GIT-Admin-Access for well known economist.

Having professional economist on the team is good, and we already have a couple on /r/bitcoinxt.

But why does he need git admin access?

>Poll like integration for Bitcoin clients. A possibility for everyone with Bitcoin to vote on important decisions.

People are developing third-party systems for such voting.

Integrating it into all clients would make it easier to vote, but it's a major change; plus no common voting system yet exists.

And this still leaves the problem of enforcing voting decisions.

>Possibility to reward Developer, maybe through a fund.

That's always a good idea.


>Letter of Manifest which have to be signed from every member...

In general, it seems you're looking at the governance problem from a wrong angle.

It's grossly inefficient, if not outright impossible, to manage a project as a democracy where everyone has a vote.

Much better system is several alternative implementations, each run by a single person in charge, and users instead deciding which one they prefer.

[Gavin Andresen]

On Thu, Oct 8, 2015 at 3:17 PM, Jonathan Toomim (Toomim Bros) <j...@toom.im> wrote:

It's mostly political. Currently the largest pools (Antpool, F2pool, BTCChina, BW.com) are also the pools with the highest orphan rates. They don't want to be put at a disadvantage. 

Put at a disadvantage how?

If you are on a high-latency or bandwidth-constrained connection, then until the p2p networking code gets optimized you should produce smaller blocks; you will NOT be at a disadvantage to miners on better network connections until the block subsidy drops and transaction fees become significant.

And the p2p networking code will be optimized long before then, so you won't be at a disadvantage then, either (all of this assuming you're not trying to mine behind a dial-up modem or something completely ridiculous).

That's what cypherdoc is saying: fear of higher orphan rates should NOT be an issue. Produce smaller blocks if your orphan rate is higher than the rest of the network; simulations show that it doesn't matter what the other miners do (it can be more complicated than that, but it only starts to get complicated if a miner gets more than 33% of hash rate AND has some control over other miners' connectivity so they can control exactly how new block announcements propagate to other miners-- neither of which are true in practice).

--

--
Gavin Andresen

[Jonathan Toomim (Toomim Bros)]

On Oct 8, 2015, at 4:09 PM, Gavin Andresen <gavina...@gmail.com> wrote:
> Put at a disadvantage how?
>
> If you are on a high-latency or bandwidth-constrained connection, then until the p2p networking code gets optimized you should produce smaller blocks

Gavin, I've made most of those arguments to other people before. I agree with your viewpoint on the technical aspects of large blocks, but the point I was trying to make was a bit different.

> That's what cypherdoc is saying: fear of higher orphan rates should NOT be an issue.

Fear is always an issue when it comes to politics and money. A higher orphan rate for large blocks currently isn't an issue. Fear of higher orphan rates shouldn't be an issue either, but it is.

[Cypher Doc]

the great thing about lifting the cap is that it allows competition across wider geographic areas.  smaller or new pools who want to get in the market to compete can explore the upper limits of progressively larger block sizes, at some risk of orphaning, to try and capture more fees per larger block. this will attract more hashers to their pool if successful allowing them to grow.  pools outside of China can try to leverage their better bandwidth access to do this.  we'll get more pools and more decentralization. 

[Jonathan Toomim (Toomim Bros)]

Pools can move their full nodes pretty easily. Pools that serve mostly mainland China clients can put their full node outside the mainland and use something like the stratum protocol (sending candidate block headers plus Merkle root) to their pool server in the mainland. That way, they only have to send about 10 kB per minute across the Great Firewall regardless of how big the blocks are.

I don't see large block sizes causing more pools to show up. It might change which pools are big, but I doubt it will change how many large pools there are. Miners tend to prefer to mine on large (> 5%) pools because it reduces their payout variance. As long as that preference is not addressed (e.g. with a variant of p2pool that doesn't have performance and scaling issues), mining will be dominated by a small number of large pools.

[Chris Wilmer]

Hi Gavin,

I feel like we're all preaching to the choir here. It seems clear to us that the blocksize should be raised... so what *are* Antpool, F2pool, and BTCChina (and 21 inc, and KinC Miner, and so on), saying about this issue? The suspense is killing me. I understand that some pool operators are hard to reach (and perhaps some might be irrational)... but three pools in particular, 21 Inc, KncMiner, and BTCChina, I would think would make reasonable arguments one way or another... so what are they?

Apologies in advance if there's a resource online that explains what those 3 companies think about BIP101 already.

-Chris

[Peter Tschipper]

Hi All,

I just put in PR #84, for a dynamically adjusting spam blocker. I'd
like to get feedback on it when you have the chance. It's simple and works
well at choking off spam without affecting other transactions while
keeping the memory pool small and network activity low.


If you want to build and run it you can pull from my repository, it's
based on the latest master branch.

https://github.com/ptschip/bitcoinxt/tree/spamblocker

thanks and have a great weekend :)

[Mike Hearn]

Hi Peter,

Let's try and debate design decisions on the mailing list before implementing them. You are free to write whatever code you like of course, but it helps avoid redundant effort if there's agreement beforehand.

I tend to agree with Thomas that defining "spam" as "low fee" is an absolute last resort. The problem is that once you believe that per-tx fees are unlikely to pay for hashing power directly any time soon (or at least, wouldn't pay for very much) then there's no longer any reason to want fees to be high. High fees always reflect some kind of design problem, in such a worldview. In theory even low fees do - when I first used Bitcoin all transactions were free, and that was pretty nice.

In this case the design problem is a kind of lazyness around how to prioritise transactions. Trying to fix flooding attacks by simply raising fees is like trying to fix email spam by making everyone solve a CAPTCHA for every recipient they send to. Sure it might "work" for some very weak definition of work, in that it'd get rid of most (not all) email spam. But it'd also get rid of most of the users. That is a pyrrhic victory.

The actual way ESPs solved spam is through a combination of really good prioritisation based on the signals available, and brute force ... having sufficient compute capacity to actually swallow all the mail and sort through it.

The mempool limiting approach we have now feels good to me. It can use refinement, as we have discussed in the past, but the basic principle of random selection is sound. RAM sizes vs block sizes are so disproportionate that you can swallow a whole lot of spam before needing to start shooting things, and when you do start, you're very very likely to delete spam transactions just due to the basic percentages involved.

That leaves the problem as being one of prioritisation. Again, I understand the appeal of "just make users pay more". It's very simple, and simple is good. But:

• Users won't like it.
• It is taken as axiomatic that legit users are always willing to pay more than attackers. I have seen nobody attempt to justify this assumption.
• Transactions sent before the attack will have low fees. So attackers can cause trouble by just flooding in waves; each time the min fee drops again, start the next wave and boot out all the transactions sent recently.

A long time ago Gavin invented the idea of priority, and Satoshi was very happy with this idea. The reason I wrote Mempool Explorer is to let us quickly iterate and prototype on better spam filtering algorithms. Of course it's very hard because we have to do it all in the open, and we have very few signals. But even so there's plenty of low hanging fruit.

[Tom Zander]

On Monday 12. October 2015 17.55.26 Mike Hearn wrote:
> In this case the design problem is a kind of lazyness around how to
> prioritise transactions. Trying to fix flooding attacks by simply raising
> fees is like trying to fix email spam by making everyone solve a CAPTCHA
> for every recipient they send to. Sure it might "work" for some very weak
> definition of work, in that it'd get rid of most (not all) email spam. But
> it'd also get rid of most of the users.

Exactly.

A "spam attack" is really just a customer sending a lot of transactions. In
this case to show that there are some bugs in the system.

According to economy, it is always more profitable for all parties if you can
serve *all* customers instead of turning away customers by telling them they
can't enter your mempool.

The direct effect here was;
Around 10% of the nodes went down because they didn't have a memory
protection. XT solved that (although we have some ideas for improvement).

We know that the malleability issues can cause more pain, and they need to be
stopped. Not easy, would likely require a hard-fork to properly fix.

We know of some issues with slowness in the GBT json call on big mempools.


Bottom line for me is that we should learn from these issues and make sure we
handle these loads better, fix the issues that caused the big loads in the
first place and use this time to strengthen the system.

In short, even spam is a legit transaction that should be handled. I'd argue
that anyone raising the min-relay-fee is censoring a legit customer. I'll
definitely not be configuring my node in such a fashion.

[Peter Tschipper]

Firstly I'd like to apologize if I seem to be trying to push through something too quickly.  I really didn't even intend to write this code, just woke up in the middle of the night a few nights ago and starting trying out a few things and by morning it was done, and it seemed that with the current spamming going on it would be good to get it out there so as others could try it out. 

But on to your other points:

"High fees always reflect some kind of design problem, in such a worldview. In theory even low fees do - when I first used Bitcoin all transactions were free, and that was pretty nice."  

That's actually why I wrote it the way I did.  In a counter-intuitive way, not to "raise" fees but to actually "reduce" them and allow more free transactions through.  I'm a beleiver in micro transactions and free transactions and obviously a supporter of big blocks: I don't like fees or the idea of centrally planning a fee market.   So, currently the minrelaytxfee is set to .00001 and over on the core project they just bumped it to 0.00005.  That means that "all" transactions are subject to these limits.  However, in this pull, fees are not increased "until" the memory pool is full (or we could make it until it's 5 times full or any amount agreed upon).  So in this implemenation, we can have any number of free transactions until the pool really starts to fill up in an abnormal way.  Then the lowest fee transations are slowly choked off but never completely and as soon as an attack is over the fees are removed automatically.  

"The mempool limiting approach we have now feels good to me. It can use refinement, as we have discussed in the past, but the basic principle of random selection is sound. " 

I think it is sound also, but I beleive both these approaches work in a complimentary fashion.  This implementation would be a first hurdle for the spammers to get over, while at the same time entirely removing fees under normal conditions.  While the second approach "mempool limiting" would be the final wall that protects the nodes from some other kind attack, perhaps from some entity with very deep pockets.

"Transactions sent before the attack will have low fees. So attackers can cause trouble by just flooding in waves; each time the min fee drops again, start the next wave and boot out all the transactions sent recently."

I'm not sure what you mean here.  I don't know how transactions can be "booted" out once they've been accepted into the mempool and forwarded to other nodes?  Do you mean just keeping the memory pool almost full?  If that were the case, then wouldn't those blocks just get mined as any normal block.  I don't know what they could acheive there in terms of causing nodes to crash or disrupting the system.

"A long time ago Gavin invented the idea of priority, and Satoshi was very happy with this idea. The reason I wrote Mempool Explorer is to let us quickly iterate and prototype on better spam filtering algorithms. Of course it's very hard because we have to do it all in the open, and we have very few signals. But even so there's plenty of low hanging fruit."

What I like about this implementation is the simplicity.  It doesn't really create that much new functionality, just automates what is already there in the code and there is very little a spammer can do IMO to get around it,  even by analyizing the code. Unless they want to pay up...which I don't think the miners would mind at all.

[Peter Tschipper]

On 12/10/2015 9:31 AM, Tom Zander wrote:

According to economy, it is always more profitable for all parties if you can 
serve *all* customers instead of turning away customers by telling them they 
can't enter your mempool.

Without getting into a debate about economics, I'm not sure there are any large businesses that can serve *all* their customers
during times of stress.  Services are usually cut back, limited or fees are raised.  However, my point here is not to say that I like
the idea of rasing fees.  I'm acutally against it, and that's why I wrote the code so that we have more "free" transactions and no raising of fees
unless the system is under stress.

The direct effect here was;
Around 10% of the nodes went down because they didn't have a memory 
protection. XT solved that (although we have some ideas for improvement).

We know that the malleability issues can cause more pain, and they need to be 
stopped. Not easy, would likely require a hard-fork to properly fix.

We know of some issues with slowness in the GBT json call on big mempools.


Bottom line for me is that we should learn from these issues and make sure we 
handle these loads better, fix the issues that caused the big loads in the 
first place and use this time to strengthen the system.

In short, even spam is a legit transaction that should be handled. I'd argue 
that anyone raising the min-relay-fee is censoring a legit customer.  I'll 
definitely not be configuring my node in such a fashion

I agree, spam is a legit transaction, I'm only using the term loosly.  Without having to say "low fee legit transactions"
in every sentence.  But to my point, currently, XT limits transactions using the -relaytxfee of 0.00001 , so all transactions
currently are limited in some way.  So without trying to repeat myself too much, this implementation does
away with fee limiting under normal conditions and therefore allows for more, not less, free transactions under
normal conditions.

[Tom Zander]

On Monday 12. October 2015 10.00.28 Peter Tschipper wrote:
> That's actually why I wrote it the way I did. In a counter-intuitive
> way, not to "raise" fees but to actually "reduce" them and allow more
> free transactions through.

The difference here is that you started with the assumption that we have a
minrelaytxfree set that is pretty significant.

The reality is that we only have that because a good memory-limiting wasn't
there and some people had the foolish idea that a fee market would be good
this early on in the lifetime of Bitcoin.

The solution in my opinion, however, is not to refine how we stop transactions
based on their fee, but to stop doing so.
Which makes your solution go in the wrong direction.

> I think it is sound also, but I beleive both these approaches work in a
> complimentary fashion.

The problem is that they don't actually work complimentary. One hurts the
other. Because one is based on single-machine view and the other on
distributed computing concepts.
Disallowing transactions, even with a dynamically adjusted threshold, is
consistent throughout the system. So a free transaction in a busy system
essentially has no chance anywhere.

The random eviction concept has the advantage that a certain transaction can
be evicted from 90% of the pools, but it will continue to be propagated and
has a good chance of eventually being mined.

See, this is a core difference that you didn't account for. And your answers
show you are not understanding or acknowledging this.
Your chance doesn't remove the limit, it merely uses it mildly better.

From your other email;

> So without trying to repeat myself
> too much, this implementation does
> away with fee limiting under normal conditions

I don't think this goes far enough. Just do away with this limitation at all.
The dust limit should be enough.

[Mike Hearn]

I don't agree that it should be entirely up to me and Gavin to talk to miners. I think there's an assumption here that we have some level of special access to them that we just don't. 

If miners hear from lots of people consistently that they should reconsider, that'll have a lot more effect than just me trying to find out their email addresses and set up calls. I'm doing this anyway but it's not easy.

From talking to a couple of miners so far and reviewing the public statements of others, their big issues are:

• They made huge investments and are terrified of anything that might hurt the price, in particular, media coverage of a war or split of some kind. They are, so far, more scared of this than they are of Core not doing anything.
  
  
• Many of them don't realise that Core is effectively run by just two guys, both of whom are dead set against any rise of the block size limit. They believe the rhetoric about debate, consensus, etc, even though it is not really true (see Gregory Maxwell's recent admission that agreement doesn't actually matter for changes to be made in Core). In short they believe that eventually Core will give them what they want.
  
  
• They think BIP100 will happen. They don't realise that Jeff isn't actually working on it right now, and Maxwell/van der Laan have never agreed to it anyway, so it wouldn't get merged no matter what.
  
  
• Because of all the FUD that has been spread around, their default position is "let's wait for the second conference and re-evaluate after that". These conferences are a very clever PR move, and have effectively frozen a lot of decision makers in place, because they are desperately trying to avoid the conclusion that the Core maintainers don't know what they are doing.
  
  
• They don't know much about XT (probably due to the censorship), and in some cases, believe things about it that are completely false.

So those are the issues.

[Chris Wilmer]

Great perspective, thanks! 

[Peter Tschipper]

Hi Tom,

Thanks for your response, you certainly come up with good points and make me think more deeply on the issues.  However, I still don't agree with you
on the following.

On 12/10/2015 10:51 AM, Tom Zander wrote:

The problem is that they don't actually work complimentary. One hurts the 
other.  Because one is based on single-machine view and the other on 
distributed computing concepts.
Disallowing transactions, even with a dynamically adjusted threshold, is 
consistent throughout the system. So a free transaction in a busy system 
essentially has no chance anywhere.

Yes, that's the main point of this implementation..."having no chance anywhere" an attacker can not attack and therefore will not attack.  Why would they bother to make the attempt???.  And therefore the mempool should always be in a good state where free transactions are possible.  Attacks have always been in the low fee area to my knowledge...however that's not to say an attacker with deep enough pockets wouldn't mount an attack to disrupt the system using higher fee transactions...So that's where, in my mind, the current random mempool eviction comes in.  That's why I say , these two approaches are complimentary in terms of closing of the avenues of attack.

The random eviction concept has the advantage that a certain transaction can 
be evicted from 90% of the pools, but it will continue to be propagated and 
has a good chance of eventually being mined.

Yes, with mempool pruning, all transactions should still live..that's good.

See, this is a core difference that you didn't account for. And your answers 
show you are not understanding or acknowledging this.
Your chance doesn't remove the limit, it merely uses it mildly better.

See answer below...

>From your other email;

> So without trying to repeat myself
> too much, this implementation does
> away with fee limiting under normal conditions

I don't think this goes far enough. Just do away with this limitation at all. 
The dust limit should be enough.

Here is the main problem I have with your "evict approach only"...and I may be wrong so please correct me.
Doesn't doing away with the *possibility* of imposing a fee on those that want to abuse the system open
up an attack on creating UTXO bloat?  With no rate limiting of any kind possible an attacker can just endlessly send
transactions to himself thus bloating the UTXO while costing him nothing.  

So then are we not back to needing some sort of rate limiting at some point, in addition to mempool eviction?

[Peter Tschipper]

Come to think of it I think the UTXO bloat attack vector isn't really that big of a deal, yes they can create more bloat
for no fees but It would take a great deal of BTC.  But but the ability to constantly send and resend transactions
without any loss for the attacker is a problem. With a high enough transaction rate they could tie up the memory
pool and cause the eviction of the majority of good transactions from every node.  Certainly the  good random transactions that
remain will be resent over the network eventually and make their way to to mined , however an attacker could keep this up
indefinitely with no downside since they have to pay nothing, slowing down and hobbling the network.   Without
the ability to introduce a fee at some point then I still don't see how you could stop this attack from happening
with a mempool eviction strategy only.
  

[Robin]

We discussed this in another thread on this list. A possible solution to the resending attacks is to use a bloom filter that stops previously evicted TX from being accepted again.


Peter Tschipper <peter.t...@gmail.com> schreef op 13 oktober 2015 07:31:01 CEST:

[Mike Hearn]

There's always some payment for a transaction needed, the question is really to what extent is that payment made in BTC vs made in coin-age (i.e. priority).

[Nick ODell]

Couple comments on the patch:

• When it raises the relay limit, it won't kick out any transactions. Correct? So it seems like this doesn't need reinsertion prevention. OTOH, it doesn't seem like this is actually limited to 3MB.
  
• If the mempool goes below 3MB again, it will allow low value transactions again, correct?
  
• Multiplier has no Y.
  

>Here is the main problem I have with your "evict approach only"...and I may be wrong so please correct me.

>Doesn't doing away with the *possibility* of imposing a fee on those that want to abuse the system open

>up an attack on creating UTXO bloat?  With no rate limiting of any kind possible an attacker can just endlessly send

>transactions to himself thus bloating the UTXO while costing him nothing.   

Well, the transactions still need to get into the blockchain, with the fee/priority/size requirements that has. We're just talking about resource limits on the mempool right now.

[berrtus]

I have a bit of familiarity with the Chinese and I have some advice:

Don't worry so much about whatever they said their objections are.  Instead:

Contact each miner individually and ask them what they would like.  It might be something simple like some improvement here or there that makes something easier for them.  Show them you are doing something they asked for and they will use XT.  Invite them to trial XT for some benefit.  It is that simple.

So contacting miners is a great idea. 

How to contact miners?  It is not that complicated they will all have sales agents that use Skype.  Someone important and busy does not have to waste time on this unless it is a really big operation.  The initial contacts and groundwork could be done by a salesperson type.  Anyhow this is a great idea contacting the miners and the simple strategy is to ask them what we could do to make things easier.  In China everything is a negotiation.  Sorry if I am not 100% perfectly worded here.

[chaosgrid]

I don't understand why neither Gavin nor Mike want to attend the conference in Hong Kong. This seems like a missed opportunity to talk to miners directly, face to face..

[HostFat]

There is an high probability that they are already doing it during these months, but still, the nodes are the ones that choose the deepest rules of the network and not the miners.

[Jonathan Toomim]

I will be going.

[berrtus]

It's my newbie understanding that this is based on blocks discovered which would say miners have to do with it and not necessarily nodes per say that might not ever discover a block.  Anyway, I also volunteer to call miners in particular I volunteer to try contacting Chinese miners especially if we could work out something to offer them for installing XT.

Robert

[berrtus]

but I add we would be accused of 'bribing' miners so maybe have to think the details through of any benefits offered.

[Chris Wheeler]

I don't think we need to 'offer' miners anything other than a working solution to the current 1MB block size limit which they are facing - i.e. BIP101.

[Robert B]

There could be a software benefit that might make things more convenient for them such as: "more easily set transaction fee limits'' whatever it is "connect more quickly when you've found a block".  Anyway I am all for at least approaching them.

--
You received this message because you are subscribed to a topic in the Google Groups "bitcoin-xt" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bitcoin-xt/SkNO-gnKiX0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bitcoin-xt+...@googlegroups.com.

[Mike Hearn]

I already talked to most of the miners.

[shangzhou wu]

这样做真的好吗？

作为一个程序员不应该用code来说服人吗，太不专业了。或者你根本就没有这个能力。

gmaxwell sipa wumpus ...他们的努力有目共睹

 

“Bitcoin was meant to be both technically and socially robust.”

在 2015年11月24日星期二 UTC+8上午12:15:06，Mike Hearn写道：

[Mike Hearn]

Please use English here.

--

You received this message because you are subscribed to the Google Groups "bitcoin-xt" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bitcoin-xt+...@googlegroups.com.