[ Removed by reddit in response to a copyright notice. ] submitted by
You can check it out on the forums here
. Here's a copypasta: Surae's End of November (2017!) Update
Hello, everyone! Sarang posted his update a few days ago to give the community time to review his work before the end of the month. I was hoping to finish multisig off before the end of this month... so I held off on writing this update until then... but it looks like I'm somewhere between 2 days and a week behind on that estimate. MRL Announcements Meetings.
We are holding weekly meetings on Mondays at 17:00 UTC. Logs are to be posted on my github soon(tm). Usually we alternate between "office hours" and "research meetings." At office hours, we want members of the community to come in and be able to ask questions, so we are considering opening up a relay to the freenode channel during office hours times, unless things get out of hand. POW-Difficulty Replacement Contest.
Some time in December, I am going to formalize an FFS "idea" to open up a multiple-round contest for possible replacements for our proof of work game. The first round would have a 3- or 6-month deadline. Personally, I would love it if this FFS could have an unbounded reward amount. If the community is extremely generous, we could easily whip up a large enough reward to spur lots and lots of interest across the world.
The Bitcoin POW game uses SHA256 to find nonces that produce hashes with sufficiently small digests according to the Bitcoin difficulty metric. Our current POW game uses CryptoNight to find nonces that produce hashes with sufficiently small digests according to the CryptoNote difficulty metric. The winner need not be proof of work. My current thoughts are roughly this:
All submissions will be public. Submissions that minimize incentives for centralized mining (or maximize disincentives) will be preferred over submissions that do not. Submissions that are elegant will be preferred over submissions that are not. Submissions that have provable claims about desirable properties will be preferred over submissions that do not (e.g. for either the Bitcoin or the Monero POW games, the necessary and sufficient network conditions for these games to produce blocks in a Poisson process have not been identified, to my understanding). Submissions that have a smaller environmental impact will be preferred over submissions that have a larger impact. And so on. I would like as many ideas as possible about a judging rubric for the first round. Especially if a large amount of money will be put up as a prize.
The details of the next round would be announced along with the winners of the first round. The reward funds should be released when a set of judges agree on a winner. MRL and Monero Core should each have representation on the panel of judges, and there ought to be at least one independent judge not directly associated with the Monero Project, like Peter Todd, Tim Ruffing, or someone along those lines. But, again, this is just an idea. If the community doesn't like it, we can drop it. Here is a rundown for November Multisig.
Almost done. I know, I know, it's been forever. We, as a community, have recently come to see how important it is to carefully and formally ensure the correctness of our schemes before proceeding. Multisig is a delicate thing because a naively implemented multisig can reveal information about the participants.
I'm finishing vetting key creation today, finishing signatures tomorrow and the next day. Then I'm passing the result off to moneromooo and luigi to ensure that my description of their code is accurate up to their understanding. Then onto Sarang for final reviews before submission, hopefully by the end of the month. I have my life until Sunday evening blocked off to finish this. A copy of the document will be made available to the community ASAP (an older version is on my github), after more checking and writing is completed.
This whitepaper on multisig will be broken into two papers: one will be intended for peer review describing multi-ring signatures, and one will be a Monero Standard.
More about that later... RTRS RingCT column-linkability and amortization.
You may say "what? I thought we were putting RTRS RingCT on the back burner?" Well, I'm still think ing about amortization of signatures. I'm thinking it will be possible (although perhaps not feasible) for miners to include amortized signatures upon finding new blocks. This would allow users to cite an amortized signature for fast verification, but has some possible drawbacks. But more exciting, I'm also chatting with Tim Ruffing, one of the authors on the RTRS RingCT papers: he thinks he has a solution to our "linkability by columns" problem with MLSAG and RingCT. Currently we try to avoid using more than one ring signature per recipient. This avoids linking distinct outputs based on bundling of these ring signatures. Ruffing believes RTRS RingCT can be tweaked to prove several commitments in a vector of commitments; this would allow a single RTRS RingCT to be computed and checked for each output being spent.
Once all the details are checked, I'll write up a document and make a copy of it available to the community. If it works, of course. Consequences of bulletproofs.
In my last end-of-month update I hinted at issues with an exponential space-time trade-off in RTRS RingCT. Due to the speed and space savings with bulletproofs, it may now be feasible to implement RTRS RingCT. With improved verification time savings with bulletproofs we can relax our requirements for verification times for signatures. This will allow the slightly longer verification times of RTRS RingCT to be counter-acted. Solving the problem "what ring sizes can we really get away with?" involves some modeling and solving some linear programming problems (linear programming, or linear optimization, is an anachronistically named area of applied mathematics involved with optimizing logistic problems... see here
for more information).
Hence, we will be inserting bulletproofs into Monero with low friction, and then we will look into the logistics of moving to RTRS RingCT. Monero Standards.
Right now, we don't have a comprehensive list of how Monero works, all the various primitives and how they all fit together. Sarang and I have begun working on some Monero Standards that are similar to the original Cryptonote Standards (see here
for more information). For each standard, from our hash function on upward, we will describe the standard, provide a justification for Monero's choices in those standards (complete with references), as well as a list of possible replacement standards. For example, our Monero RingCT Standard should describe the RingCT scheme described by shen, which is essentially a ring signature with linear combinations of signing keys + amount commitments. Under the "possible replacements" section, we would describe both the RTRS RingCT scheme and the doubly efficient zk-snark technology as two separate options.
These standards may take awhile to complete, and will be living documents as we change the protocol over the years. In the meantime, it will make it dramatically easier for future researchers to step into MRL and pick up where previous researchers have left off. Hierarchical view keys.
Exploiting the algebra we currently use for computing one-time keys, the sub-address scheme plays with view keys in a certain way, allowing a user to have one single view key for many wallets. Similarly, we may split a view key into several shares, where each subset of shares can be used to grant partial view access to the wallet. A receiver can request that a sender use a particular basepoint in their transaction key where different subsets of shares of the view key grant access to transactions with different basepoints in their transaction keys. None of these are protocol-level observations, they are wallet-level observations. Moreover, these require only that a receiver optionally specify a basepoint.
In other words: hierarchical view keys are a latent feature of our one-time address scheme that has not seen specific development yet. It's a rather low priority compared to the other projects under development; it grants users fine-grained control over their legal compliance, but Monero Standards will have great long-term impact on development and research at Monero. Criticisms.
Monero has suffered some recent criticisms about our hash function. I want to briefly address them.
First, I believe part of the criticism came from a confusion between Keccak3, SHA-3, and Keccak: we have never claimed to use SHA-3 as our hash function, we have only used the Keccak3 hash function, which is a legacy choice inherited from the original CryptoNote reference code. Many developers confuse the two, but Keccak3 was the hash function on which SHA-3 is based. In particular, the Keccak sponge construction can be used to fashion lots and lots of primitives, all of which could fairly be called "Keccak:" both Keccak3 and SHA-3 are Keccak constructions. This may be a subtle nomenclature issue, but it's important because a good portion of our criticisms say "Hey, they aren't using SHA-3!"
Second, I believe part of the criticism also comes from our choice of library, which in my opinion isn't a big deal as long as the library does what it says on the tin. In this case, our hash function is a valid implementation of Keccak3 according to the Keccak3 documentation. The most important criticism, from my point of view, is our choice of pre-SHA-3 Keccak3 as our hash function. Keccak3 underwent lots of analysis during the SHA contest, and Keccak3 is a well-vetted hash funtion. However, it has not been chosen as an international standard. There is a sentiment in the cryptocurrency community to distrust standards, which is probably a healthy sentiment. In this case, however, it means that our choice of hash function is not likely to be supported in common, well-vetted libraries in the future. Moreover, since SHA-3 is an international standard, it shall be undergoing heavy stress testing over the coming decades, a benefit Keccak3 shall not enjoy.
Last month, after some discussions, we made changes to our choice of PRNG in Monero to match the PRNG for Bitcoin. There has since been some discussions instantiated by anonimal about this choice of PRNG. We at MRL are doing our best to assist the core team in weighing the relative costs and benefits of switching to a library like crypto++, and so we believe these criticisms fall into the same category. We intend to address these issues and make formal recommendations in the aforementioned Monero Standards. Sorry for using the word aforementioned. Things that didn't move much
include a) educational outreach, b) SPECTRE, c) anti-ASIC roadmap, d) refund transactions. Most of which was on hold to complete multisig.
As far as educational outreach, I contacted a few members of a few math/cs depts at universities around me, but I haven't gotten anything hopeful yet. I wanted to go local (with respect to me) to make it easier to organize, but that's looking less likely. No matter how enthusiastic of a department we find, garnering participation from faculty members, beginning an application process for new students, squirelling up funding, working out logistics of getting teachers or lecturers/speakers from point A to point B, where to stash students, etc would be a challenge to finish before, say, July. And some schools start their fall semesters in mid-August. So I'm thinking that Summer 2019 is reasonable as the first Monero Summer School... and would be a real fun way to finish off a two-year post-doc! December plan.
I am going to finish multisig, and then finish the zk-lit review with Jeffrey Quesnelle, since these are both slam dunks. Any other time in December I have will be devoted to a) looking into the logistics of using the bulletproofs + RTRS RingCT set-up, b) reading the new zk-stark paper and assessing its importance for Monero, c) beginning work on Monero Standards, which includes addressing our hash function criticisms, our PRNG, etc. Thank you again!
This is an incredible opportunity, and this community is filled with some smart cookies. Every day is a challenge, and I couldn't ask for a more fun thing to be doing with my life right now. I'm hoping that my work ends up making Monero better for you.
Why do both sides of the debate seem “right” to me? submitted by
I know, I know, a healthy debate is healthy and all - and maybe I'm just not used to the tumult and jostling which would be inevitable in a real live open major debate about something as vital as Bitcoin.
And I really do agree with the starry-eyed idealists who say Bitcoin is vital
. Imperfect as it may be, it certainly does seem to represent the first real chance we've had in the past few hundred years to try to steer our civilization and our planet away from the dead-ends and disasters which our government-issued debt-based currencies keep dragging us into.
But this particular debate, about the blocksize, doesn't seem to be getting resolved at all.
Pretty much every time I read one of the long-form major arguments contributed by Bitcoin "thinkers" who I've come to respect over the past few years, this weird thing happens: I usually end up finding myself nodding my head and agreeing
with whatever particular piece I'm reading!
But that should be impossible - because a lot of these people vehemently disagree!
So how can both sides sound so convincing to me, simply depending on whichever piece I currently
happen to be reading?
Does anyone else feel this way? Or am I just a gullible idiot? Just Do It?
When you first look at it or hear about it, increasing the size seems almost like a no-brainer: The "big-block" supporters say just increase the blocksize to 20 MB or 8 MB, or do some kind of scheduled or calculated regular increment which tries to take into account the capabilities of the infrastructure and the needs of the users. We do have the bandwidth and the memory to at least increase the blocksize now, they say - and we're probably gonna continue to have more bandwidth and memory in order to be able to keep increasing the blocksize for another couple decades - pretty much like everything else computer-based we've seen over the years (some of this stuff is called by names such as "Moore's Law").
On the other hand, whenever the "small-block" supporters warn about the utter catastrophe that a failed hard-fork would mean, I get totally freaked by their possible doomsday scenarios, which seem totally plausible and terrifying - so I end up feeling that the only way I'd want to go with a hard-fork would be if there was some pre-agreed "triggering" mechanism where the fork itself would only actually "switch on" and take effect provided that some "supermajority" of the network (of who? the miners? the full nodes?) had signaled (presumably via some kind of totally reliable p2p trustless software-based voting system?) that they do indeed "pre-agree" to actually adopt the pre-scheduled fork (and thereby avoid any
possibility whatsoever of the precious blockchain somehow tragically splitting into two and pretty much killing this cryptocurrency off in its infancy).
So in this "conservative" scenario, I'm talking about wanting at least 95% pre-adoption agreement - not the mere 75% which I recall some proposals call for, which seems like it could easily lead to a 75/25 blockchain split.
But this time, with this long drawn-out blocksize debate, the core devs, and several other important voices who have become prominent opinion shapers over the past few years, can't seem to come to any real agreement on this. Weird split among the devs
As far as I can see, there's this weird split: Gavin and Mike seem to be the only people among the devs who really want a major blocksize increase - and all the other devs seem to be vehemently against them.
But then on the other hand, the users
seem to be overwhelmingly in favor of a major increase.
And there are meta-questions about governance, about about why this didn't come out as a BIP, and what the availability of Bitcoin XT means.
And today or yesterday there was this really cool big-blockian exponential graph based on doubling the blocksize every two years for twenty years, reminding us of the pure mathematical
fact that 210
is indeed about 1000 - but not really addressing any of the game-theoretic
points raised by the small-blockians. So a lot of the users seem to like it, but when so few devs say anything positive about it, I worry: is this just yet more exponential chart porn?
On the one hand, Gavin's and Mike's blocksize increase proposal initially seemed like a no-brainer to me.
And on the other hand, all the other devs seem to be against them. Which is weird - not what I'd initially expected at all (but maybe I'm just a fool who's seduced by exponential chart porn?).
Look, I don't mean to be rude to any of the core devs, and I don't want to come off like someone wearing a tinfoil hat - but it has to cross people's minds that the powers that be (the Fed and the other central banks and the governments that use their debt-issued money to run this world into a ditch) could very well be much more scared shitless than they're letting on. If we assume that the powers that be are using their usual playbook and tactics, then it could be worth looking at the book "Confessions of an Economic Hitman" by John Perkins, to get an idea of how they might try to attack Bitcoin. So, what I'm saying is, they do have a track record of sending in "experts" to try to derail projects and keep everyone enslaved to the Creature from Jekyll Island. I'm just saying. So, without getting ad hominem - let's just make sure that our ideas can really stand scrutiny on their own - as Nick Szabo says, we need to make sure there is "more computer science, less noise" in this debate.
When Gavin Andresen first came out with the 20 MB thing - I sat back and tried to imagine if I could download 20 MB in 10 minutes (which seems to be one of the basic mathematical and technological constraints here - right?)
I figured, "Yeah, I could download that" - even with my crappy internet connection.
And I guess the telecoms might
be nice enough to continue to double our bandwidth every two years for the next couple decades – if we ask them politely?
On the other hand - I think we should be careful about entrusting the financial freedom of the world into the greedy hands of the telecoms companies - given all their shady shenanigans over the past few years in many countries. After decades of the MPAA and the FBI trying to chip away at BitTorrent, lately PirateBay has been hard to access. I would say it's quite likely that certain persons at institutions like JPMorgan and Goldman Sachs and the Fed might be very, very motivated to see Bitcoin fail - so we shouldn't be too sure about scaling plans which depend on the willingness of companies Verizon and AT&T to double our bandwith every two years. Maybe the real important hardware buildout challenge for a company like 21 (and its allies such as Qualcomm) to take on now would not be "a miner in every toaster" but rather "Google Fiber Download and Upload Speeds in every Country, including China".
I think I've read all the major stuff on the blocksize debate from Gavin Andresen, Mike Hearn, Greg Maxwell, Peter Todd, Adam Back, and Jeff Garzick and several other major contributors - and, oddly enough, all
their arguments seem reasonable - heck even Luke-Jr seems reasonable to me on the blocksize debate, and I always thought he was a whackjob overly influenced by superstition and numerology - and now today I'm reading the article by Bram Cohen - the inventor of BitTorrent - and I find myself agreeing with him too!
I say to myself: What's going on with me? How can I possibly agree with all
of these guys, if they all have such vehemently opposing viewpoints?
I mean, think back to the glory days of a couple of years ago, when all we were hearing was how this amazing unprecedented grassroots innovation called Bitcoin was going to benefit everyone from all walks of life, all around the world:
- wealthy individuals trying to preserve and transport their wealth across space and across time
- iPhone and Android users who want to buy a latte on their smartphone at Starbucks
- Venezuelans and Argentinians and Cypriots and Russian oligarchs and Greeks and anyone else whose state-backed currency sucks
- unbanked Africans who will someday be texting around money via SMS messages on their cellphones
- online content providers who will finally be able to get paid via micropayments
- smart contracts and stock brokering and lawyering and land deeding and the refrigerator calling out to order more milk and distributed anonymous corporations (DACs) automatically negotiating and adjusting driverless taxicab fares in the Uber-future of the Internet of Things
...basically the entire human race transacting everything into the blockchain.
(Although let me say that I think that people's focus on ideas like driverless cabs creating realtime fare markets based on supply and demand seems to be setting our sights a bit low as far as Bitcoin's abilities to correct the financial world's capital-misallocation problems which seem to have been made possible by infinite debt-based fiat. I would have hoped that a Bitcoin-based economy would solve much more noble, much more urgent capital-allocation problems than driverless taxicabs creating fare markets or refrigerators ordering milk on the internet of things. I was thinking more along the lines that Bitcoin would finally strangle dead-end debt-based deadly-toxic energy industries like fossil fuels and let profitable clean energy industries like Thorium LFTRs take over - but that's another topic. :=) Paradoxes in the blocksize debate
Let me summarize the major paradoxes I see here:
(1) Regarding the people (the majority of the core devs) who are against
a blocksize increase: Well, the small-blocks arguments do seem kinda weird, and certainly not very "populist", in the sense that: When on earth have end-users ever heard of a computer technology whose capacity didn't grow pretty much exponentially year-on-year? All the cool new technology we've had - from hard drives to RAM to bandwidth - started out pathetically tiny and grew to unimaginably huge over the past few decades - and all our software has in turn gotten massively powerful and big and complex (sometimes bloated) to take advantage of the enormous new capacity available.
But now suddenly, for the first time in the history of technology, we seem to have a majority of the devs, on a major p2p
project - saying: "Let's not scale the system up. It could be dangerous. It might break the whole system (if the hard-fork fails)."
I don't know, maybe I'm missing something here, maybe someone else could enlighten me, but I don't think I've ever seen this sort of thing happen in the last few decades of the history of technology - devs arguing against
scaling up p2p technology to take advantage of expected growth in infrastructure capacity.
(2) But... on the other hand... the dire warnings of the small-blockians about what could happen if a hard-fork were to fail
- wow, they do seem really dire! And these guys are pretty much all heavyweight, experienced programmers and/or game theorists and/or p2p open-source project managers.
I must say, that nearly all of the long-form arguments I've read - as well as many, many of the shorter comments I've read from many users in the threads, whose names I at least have come to more-or-less recognize over the past few months and years on reddit and bitcointalk - have been amazingly impressive in their ability to analyze all aspects of the lifecycle and management of open-source software projects, bringing up lots of serious points which I could never have come up with, and which seem to come from long experience with programming and project management - as well as dealing with economics and human nature (eg, greed - the game-theory stuff).
So a lot of really smart and experienced people with major expertise in various areas ranging from programming to management to game theory to politics to economics have been making some serious, mature, compelling arguments.
But, as I've been saying, the only problem to me is: in many of these cases, these arguments are vehemently in opposition to each other! So I find myself agreeing with pretty much all of them, one by one - which means the end result is just a giant contradiction.
I mean, today we have Bram Cohen, the inventor of BitTorrent, arguing (quite cogently and convincingly to me), that it would be dangerous to increase the blocksize. And this seems to be a guy who would know a few things about scaling out a massive global p2p network - since the protocol which he invented, BitTorrent, is now apparently responsible for like a third of the traffic on the internet (and this despite the long-term concerted efforts of major evil players such as the MPAA and the FBI to shut the whole thing down). Was the BitTorrent analogy too "glib"?
By the way - I would like to go on a slight tangent here and say that one of the main reasons why I felt so "comfortable" jumping on the Bitcoin train back a few years ago, when I first heard about it and got into it, was the whole rough analogy I saw with BitTorrent.
I remembered the perhaps paradoxical fact that when a torrent is more
popular (eg, a major movie release that just came out last week), then it actually becomes faster
to download. More people want it, so more people have a few pieces of it, so more people are able to get it from each other. A kind of self-correcting economic feedback loop, where more demand directly leads to more supply.
(BitTorrent manages to pull this off by essentially adding a certain structure to the file being shared, so that it's not simply like an append-only list
of 1 MB blocks, but rather more like an random-access or indexed array
of 1 MB chunks. Say you're downloading a film which is 700 MB. As soon as your "client" program has downloaded a single 1-MB chunk - say chunk #99 - your "client" program instantly turns into a "server" program as well - offering that chunk #99 to other clients. From my simplistic understanding, I believe the Bitcoin protocol does something similar, to provide a p2p architecture. Hence my - perhaps naïve - assumption that Bitcoin already had the right algorithms / architecture / data structure to scale.)
The efficiency of the BitTorrent network seemed to jive with that "network law" (Metcalfe's Law?) about fax machines. This law states that the more fax machines there are, the more valuable the network of fax machines becomes. Or the value of the network grows on the order of the square of the number of nodes.
This is in contrast with other technology like cars, where the more
you have, the worse
things get. The more cars there are, the more traffic jams you have, so things start going downhill. I guess this is because highway space is limited - after all, we can't pave over the entire countryside, and we never did get those flying cars we were promised, as David Graeber laments in a recent essay in The Baffler magazine :-)
And regarding the "stress test" supposedly happening right now in the middle of this ongoing blocksize debate, I don't know what worries me more: the fact that it apparently is taking only $5,000 to do a simple kind of DoS on the blockchain - or the fact that there are a few rumors swirling around saying that the unknown company doing the stress test shares the same physical mailing address with a "scam" company?
Or maybe we should just be worried that so much of this debate is happening on a handful of forums which are controlled by some guy named theymos who's already engaged in some pretty "contentious" or "controversial" behavior like blowing a million dollars on writing forum software (I guess he never heard that reddit.com software is open-source)?
So I worry that the great promise of "decentralization" might be more fragile than we originally thought. Scaling
Anyways, back to Metcalfe's Law: with virtual stuff, like torrents and fax machines, the more the merrier. The more people downloading a given movie, the faster it arrives - and the more people own fax machines, the more valuable the overall fax network.
So I kindof (naïvely?) assumed that Bitcoin, being "virtual" and p2p, would somehow scale up the same magical way BitTorrrent did. I just figured that more people using it would somehow automatically make it stronger and faster.
But now a lot of devs have started talking in terms of the old "scarcity" paradigm, talking about blockspace being a "scarce resource" and talking about "fee markets" - which seems kinda scary, and antithetical to much of the earlier rhetoric we heard about Bitcoin (the stuff about supporting our favorite creators with micropayments, and the stuff about Africans using SMS to send around payments).
Look, when some asshole is in line in front of you at the cash register and he's holding up the line so they can run his credit card
to buy a bag of Cheeto's, we tend to get pissed off at the guy - clogging up our expensive global electronic payment infrastructure to make a two-dollar purchase. And that's on a fairly efficient centralized system - and presumably after a year or so, VISA and the guy's bank can delete or compress the transaction in their SQL databases.
Now, correct me if I'm wrong, but if some guy buys a coffee on the blockchain, or if somebody pays an online artist $1.99 for their work - then that transaction, a few bytes or so, has to live on the blockchain forever?
Or is there some "pruning" thing that gets rid of it after a while?
And this could lead to another question: Viewed from the perspective of double-entry bookkeeping, is the blockchain "world-wide ledger" more like the "balance sheet" part of accounting, i.e. a snapshot
assets and liabilities? Or is it more like the "cash flow" part of accounting, i.e. a journal
revenues and expenses?
When I think of thousands of machines around the globe having to lug around multiple identical copies of a multi-gigabyte file containing some asshole's coffee purchase forever and ever... I feel like I'm ideologically drifting in one direction (where I'd end up also being against really cool stuff like online micropayments and Africans banking via SMS)... so I don't want to go there.
But on the other hand, when really experienced and battle-tested veterans with major experience in the world of open-souce programming and project management (the "small-blockians") warn of the catastrophic consequences of a possible failed hard-fork, I get freaked out and I wonder if Bitcoin really was destined to be a settlement layer for big transactions. Could the original programmer(s) possibly weigh in?
And I don't mean to appeal to authority - but heck, where the hell is Satoshi Nakamoto in all this? I do understand that he/she/they would want to maintain absolute anonymity - but on the other hand, I assume SN wants Bitcoin to succeed (both for the future of humanity - or at least for all the bitcoins SN allegedly holds :-) - and I understand there is a way that SN can cryptographically sign a message - and I understand that as the original developer of Bitcoin, SN had some very specific opinions about the blocksize... So I'm kinda wondering of Satoshi could weigh in from time to time. Just to help out a bit. I'm not saying "Show us a sign" like a deity or something - but damn it sure would be fascinating and possibly very helpful if Satoshi gave us his/hetheir 2 satoshis worth at this really confusing juncture. Are we using our capacity wisely?
I'm not a programming or game-theory whiz, I'm just a casual user who has tried to keep up with technology over the years.
It just seems weird to me that here we have this massive supercomputer (500 times more powerful than the all the supercomputers in the world combined) doing fairly straightforward "embarassingly parallel" number-crunching operations to secure a p2p world-wide ledger called the blockchain to keep track of a measly 2.1 quadrillion tokens spread out among a few billion addresses - and a couple of years ago you had people like Rick Falkvinge saying the blockchain would someday be supporting multi-million-dollar letters of credit for international trade and you had people like Andreas Antonopoulos saying the blockchain would someday allow billions of "unbanked" people to send remittances around the village or around the world dirt-cheap - and now suddenly in June 2015 we're talking about blockspace as a "scarce resource" and talking about "fee markets" and partially centralized, corporate-sponsored "Level 2" vaporware like Lightning Network and some mysterious company is "stess testing" or "DoS-ing" the system by throwing away a measly $5,000 and suddenly it sounds like the whole system could eventually head right back into PayPal and Western Union territory again, in terms of expensive fees.
When I got into Bitcoin, I really was heavily influenced by vague analogies with BitTorrent: I figured everyone would just have tiny little like utorrent-type program running on their machine (ie, Bitcoin-QT or Armory or Mycelium etc.).
I figured that just like anyone can host a their own blog or webserver, anyone would be able to host their own bank.
Yeah, Google and and Mozilla and Twitter and Facebook and WhatsApp did come along and build stuff on top of TCP/IP, so I did expect a bunch of companies to build layers on top of the Bitcoin protocol as well. But I still figured the basic unit of bitcoin client software powering the overall system would be small and personal and affordable and p2p - like a bittorrent client - or at the most, like a cheap server hosting a blog or email server.
And I figured there would be a way at the software level, at the architecture level, at the algorithmic level, at the data structure level - to let the thing scale - if not infinitely, at least fairly massively and gracefully - the same way the BitTorrent network has.
Of course, I do also understand that with BitTorrent, you're sharing a read-only object (eg, a movie) - whereas with Bitcoin, you're achieving distributed trustless consensus and appending it to a write-only (or append-only) database.
So I do understand that the problem which BitTorrent solves is much simpler than the problem which Bitcoin sets out to solve.
But still, it seems that there's got
to be a way to make this thing scale. It's p2p and it's got 500 times more computing power than all the supercomputers in the world combined - and so many brilliant and motivated and inspired people want this thing to succeed! And Bitcoin could be our civilization's last chance to steer away from the oncoming debt-based ditch of disaster we seem to be driving into!
It just seems that Bitcoin has got
to be able to scale somehow - and all these smart people working together should be able to come up with a solution which pretty much everyone can agree - in advance - will
Right? Right? A (probably irrelevant) tangent on algorithms and architecture and data structures
I'll finally weigh with my personal perspective - although I might be biased due to my background (which is more on the theoretical side of computer science).
My own modest - or perhaps radical - suggestion would be to ask whether we're really looking at all the best possible algorithms and architectures and data structures out there.
From this perspective, I sometimes worry that the overwhelming majority of the great minds working on the programming and game-theory stuff might come from a rather specific, shall we say "von Neumann" or "procedural" or "imperative" school of programming (ie, C and Python and Java programmers).
It seems strange to me that such a cutting-edge and important computer project would have so little participation from the great minds at the other
end of the spectrum of programming paradigms - namely, the "functional" and "declarative" and "algebraic" (and co-algebraic!) worlds.
For example, I was struck in particular by statements I've seen here and there (which seemed rather hubristic or lackadaisical to me - for something as important as Bitcoin
), that the specification
of Bitcoin and the blockchain doesn't really exist in any form other than the reference implementation(s)
languages such as C or Python?). Curry-Howard anyone?
I mean, many computer scientists are aware of the Curry-Howard isomorophism, which basically says that the relationship between a theorem and its proof is equivalent to the relationship between a specification and its implementation. In other words, there is a long tradition in mathematics (and in computer programming) of:
- separating the compact (and easy-to-check) statement of a theorem from the messy (and hard-to-check) details of its proof(s);
- separating the specification of a system from its implementation(s); and
- being able to prove that an implementation does indeed satisfy its specification.
And it's not exactly "turtles all the way down" either: a specification is generally simple and compact enough that a good programmer can usually simply visually inspect it to determine if it is indeed "correct" - something which is very difficult, if not impossible, to do with a program written in a procedural, implementation-oriented language such as C or Python or Java.
So I worry that we've got this tradition, from the open-source github C/Java programming tradition, of never actually writing our "specification", and only writing the "implementation". In mission-critical military-grade programming projects (which often use languages like Ada or Maude) this is simply not allowed. It would seem that a project as mission-critical as Bitcoin - which could literally be crucial for humanity's continued survival - should also use this kind of military-grade software development approach.
And I'm not saying rewrite the implementations in these kind of theoretical languages. But it might be helpful if the C/Python/Java programmers in the Bitcoin imperative programming world could build some bridges to the Maude/Haskell/ML programmers of the functional and algebraic programming worlds to see if any kind of useful cross-pollination might take place - between specifications and implementations.
For example, the JavaFAN formal analyzer for multi-threaded Java programs (developed using tools based on the Maude language) was applied to the Remote Agent AI program aboard NASA's Deep Space 1 shuttle, written in Java - and it took only a few minutes using formal mathematical reasoning to detect a potential deadlock which would have occurred years later during the space mission when the damn spacecraft was already way out around Pluto.
And "the Maude-NRL (Naval Research Laboratory) Protocol Analyzer (Maude-NPA) is a tool used to provide security proofs of cryptographic protocols and to search for protocol flaws and cryptosystem attacks."
These are open-source formal reasoning tools developed by DARPA and used by NASA and the US Navy to ensure that program implementations satisfy their specifications. It would be great if some of the people involved in these kinds of projects could contribute to help ensure the security and scalability of Bitcoin.
But there is a wide abyss between the kinds of programmers who use languages like Maude and the kinds of programmers who use languages like C/Python/Java - and it can be really hard to get the two worlds to meet. There is a bit of rapprochement between these language communities in languages which might be considered as being somewhere in the middle, such as Haskell and ML. I just worry that Bitcoin might be turning into being an exclusively C/Python/Java project (with the algorithms and practitioners traditionally of that community), when it could be more advantageous if it also had some people from the functional and algebraic-specification and program-verification community involved as well. The thing is, though: the theoretical practitioners are big on "semantics" - I've heard them say stuff like "Yes but a C / C++ program has no easily identifiable semantics". So to get them involved, you really have to first be able to talk about what
your program does (specification) - before proceeding to describe how
it does it (implementation). And writing high-level specifications is typically very hard using the syntax and semantics of languages like C and Java and Python - whereas specs are fairly easy to write in Maude - and not only that, they're executable, and you state and verify properties about them - which provides for the kind of debate Nick Szabo was advocating ("more computer science, less noise"). Imagine if we had an executable algebraic specification of Bitcoin in Maude, where we could formally reason about and verify certain crucial game-theoretical properties - rather than merely hand-waving and arguing and deploying and praying.
And so in the theoretical programming community you've got major research on various logics such as Girard's Linear Logic (which is resource-conscious) and Bruni and Montanari's Tile Logic (which enables "pasting" bigger systems together from smaller ones in space and time), and executable algebraic specification languages such as Meseguer's Maude (which would be perfect for game theory modeling, with its functional modules for specifying the deterministic parts of systems and its system modules for specifiying non-deterministic parts of systems, and its parameterized skeletons for sketching out the typical architectures of mobile systems, and its formal reasoning and verification tools and libraries which have been specifically applied to testing and breaking - and fixing - cryptographic protocols).
And somewhat closer to the practical hands-on world, you've got stuff like Google's MapReduce and lots of Big Data database languages developed by Google as well. And yet here we are with a mempool growing dangerously big for RAM on a single machine, and a 20-GB append-only list as our database - and not much debate on practical results from Google's Big Data databases.
(And by the way: maybe I'm totally ignorant for asking this, but I'll ask anyways: why the hell does the mempool have to stay in RAM? Couldn't it work just as well if it were stored temporarily on the hard drive?)
And you've got CalvinDB out of Yale which apparently provides an ACID layer on top of a massively distributed database.
Look, I'm just an armchair follower cheering on these projects. I can barely manage to write a query in SQL, or read through a C or Python or Java program. But I would argue two points here: (1) these languages may be too low-level and "non-formal" for writing and modeling and formally reasoning about and proving properties of mission-critical specifications
- and (2) there seem to be some Big Data tools already deployed by institutions such as Google and Yale which support global petabyte-size databases on commodity boxes with nice properties such as near-real-time and ACID - and I sometimes worry that the "core devs" might be failing to review the literature (and reach out to fellow programmers) out there to see if there might be some formal program-verification and practical Big Data tools out there which could be applied to coming up with rock-solid, 100% consensus proposals to handle an issue such as blocksize scaling, which seems to have become much more intractable than many people might have expected.
I mean, the protocol solved the hard stuff: the elliptical-curve stuff and the Byzantine General stuff. How the heck can we be falling down on the comparatively "easier" stuff - like scaling the blocksize?
It just seems like defeatism to say "Well, the blockchain is already 20-30 GB and it's gonna be 20-30 TB ten years from now - and we need 10 Mbs bandwidth now and 10,000 Mbs bandwidth 20 years from - assuming the evil Verizon and AT&T actually give us that - so let's just become a settlement platform and give up on buying coffee or banking the unbanked or doing micropayments, and let's push all that stuff into some corporate-controlled vaporware without even a whitepaper yet."
So you've got Peter Todd doing some possibly brilliant theorizing and extrapolating on the idea of "treechains" - there is a Let's Talk Bitcoin podcast from about a year ago where he sketches the rough outlines of this idea out in a very inspiring, high-level way - although the specifics have yet to be hammered out. And we've got Blockstream also doing some hopeful hand-waving about the Lightning Network.
Things like Peter Todd's treechains - which may be similar to the spark in some devs' eyes called Lightning Network - are examples of the kind of algorithm or architecture which might
manage to harness the massive computing power of miners and nodes in such a way that certain kinds of massive and graceful scaling become possible.
It just seems like a kindof tiny dev community working on this stuff. Being a C or Python or Java programmer should not be a pre-req to being able to help contribute to the specification (and formal reasoning and program verification) for Bitcoin and the blockchain.
XML and UML are crap modeling and specification languages, and C and Java and Python are even worse (as specification
languages - although as implementation
languages, they are of course fine).
But there are
serious modeling and specification languages out there, and they could be very helpful at times like this - where what we're dealing with is questions of modeling and specification (ie, "needs and requirements").
One just doesn't often see the practical, hands-on world of open-source github implementation-level programmers and the academic, theoretical world of specification-level programmers meeting very often. I wish there were some way to get these two worlds to collaborate on Bitcoin.
Maybe a good first step to reach out to the theoretical people would be to provide a modular executable algebraic specification of the Bitcoin protocol in a recognized, military/NASA-grade specification language such as Maude - because that's something the theoretical community can actually wrap their heads around, whereas it's very hard to get them to pay attention to something written only
as a C / Python / Java implementation (without an accompanying specification in a formal language).
They can't check whether the program does what it's supposed to do - if you don't provide a formal mathematical definition of what the program is supposed to do. Specification : Implementation :: Theorem : Proof
You have to remember: the theoretical community is very
aware of the Curry-Howard isomorphism. Just like it would be hard to get a mathematician's attention by merely showing them a proof
without telling also telling them what theorem
the proof is proving - by the same token, it's hard to get the attention of a theoretical computer scientist by merely showing them an implementation
without showing them the specification
that it implements.
Bitcoin is currently confronted with a mathematical or "computer science" problem: how to secure the network while getting high enough transactional throughput, while staying within the limited RAM, bandwidth and hard drive space limitations of current and future infrastructure. The problem only becomes a political and economic problem if we give up on trying to solve it as a mathematical and "theoretical computer science" problem.
There should be a plethora of whitepapers out now proposing algorithmic solutions to these scaling issues. Remember, all we have to do is apply the Byzantine General consensus-reaching procedure to a worldwide database which shuffles 2.1 quadrillion tokens among a few billion addresses. The 21 company has emphatically pointed out that racing to compute a hash to add a block is an "embarrassingly parallel" problem - very easy to decompose among cheap, fault-prone, commodity boxes, and recompose into an overall solution - along the lines of Google's highly successful MapReduce.
I guess what I'm really saying is (and I don't mean to be rude here), is that C and Python and Java programmers might not be the best qualified people to develop and formally prove the correctness of (note I do not say: "test", I say "formally prove the correctness of") these kinds of algorithms.
I really believe in the importance of getting the algorithms and architectures right - look at Google Search itself, it uses some pretty brilliant algorithms and architectures (eg, MapReduce, Paxos) which enable it to achieve amazing performance - on pretty crappy commodity hardware. And look at BitTorrent, which is truly p2p, where more demand leads to more supply.
So, in this vein, I will close this lengthy rant with an oddly specific link - which may or may not be able to make some interesting contributions to finding suitable algorithms, architectures and data structures which might help Bitcoin scale massively. I have no idea if this link could be helpful - but given the near-total lack of people from the Haskell and ML and functional worlds in these Bitcoin specification debates, I thought I'd be remiss if I didn't throw this out - just in case there might be something here which could help us channel the massive computing power of the Bitcoin network in such a way as to enable us simply sidestep this kind of desperate debate where both sides seem right because the other side seems wrong. https://personal.cis.strath.ac.uk/neil.ghani/papers/ghani-calco07
The above paper is about "higher dimensional trees". It uses a bit of category theory (not a whole lot) and a bit of Haskell (again not a lot - just a simple data structure called a Rose tree, which has a wikipedia page) to develop a very expressive and efficient data structure which generalizes from lists to trees to higher dimensions.
I have no idea if this kind of data structure could be applicable to the current scaling mess we apparently are getting bogged down in - I don't have the game-theory skills to figure it out.
I just thought that since the blockchain is like a list, and since there are some tree-like structures which have been grafted on for efficiency (eg Merkle trees) and since many of the futuristic scaling proposals seem to also involve generalizing from list-like structures (eg, the blockchain) to tree-like structures (eg, side-chains and tree-chains)... well, who knows, there might be some nugget of algorithmic or architectural or data-structure inspiration there. So... TL;DR:
(1) I'm freaked out that this blocksize debate has splintered the community so badly and dragged on so long, with no resolution in sight, and both sides seeming so right (because the other side seems so wrong).
(2) I think Bitcoin could gain immensely by using high-level formal, algebraic and co-algebraic program specification and verification languages (such as Maude including Maude-NPA, Mobile Maude parameterized skeletons, etc.) to specify (and possibly also, to some degree, verify) what
Bitcoin does - before translating to low-level implementation
languages such as C and Python and Java saying how
Bitcoin does it. This would help to communicate and reason about programs with much more mathematical certitude - and possibly obviate the need for many political and economic tradeoffs which currently seem dismally inevitable - and possibly widen the collaboration on this project.
(3) I wonder if there are some Big Data approaches out there (eg, along the lines of Google's MapReduce and BigTable, or Yale's CalvinDB), which could be implemented to allow Bitcoin to scale massively and painlessly - and to satisfy all stakeholders, ranging from millionaires to micropayments, coffee drinkers to the great "unbanked".
I couldn’t verify Craig’s math, not my specialty area as I’m an algebra & calculus guy, but I knew enough to understand the boundary conditions of the problem. Rizun et al didn’t. Craig’s answer, just like his answer to Szabo (who for a short time I thought was Satoshi, until I read his California seashells blog and realized Szabo didn’t understand money) on Turing Completeness was ... The Electric Coin Company discovered a new way to scale blockchains with “recursive proof composition," a proof to verify the entirety of a blockchain in one function. Despite knowing that proof of work (or proof of stake) can be an inevitable necessity to having a cryptocurrency like Bitcoin, I'd like to fully understand where other methods fail. The first problem Bitcoin addresses from my understanding is signatures with each transaction. This is seen as one of the biggest flaws in a proof of work-based cryptocurrency: centralized mining power opens the possibility of a 51% attack. 4 out of the top 5 bitcoin mining pools come from a single country (China) and 80% of the world’s bitcoin mining hashpower is provided by the top 5 mining pools. Furthermore, 70% of bitcoin mining hardware (ASICs) is built by a single manufacturer ... If you want to ask questions, share interesting math, or discuss videos, take a look at the 3blue1brown subreddit.People have also shared projects they’re working on here, like their own videos, animations, and interactive lessons.
3Blue1Brown series S1 • E7 Inverse matrices, column space and null space Essence of linear algebra, chapter 7 - Duration: 12:09. 3Blue1Brown Recommended for you 12:09 Khan Academy is a 501(c)(3) nonprofit organization with the mission of providing a free, world-class education for anyone, anywhere. Our interactive practice... Digital currency, how does it work, what's a data miner and will Bitcoin last? We asked Professor Ross Anderson of the University of Cambridge Computer Labor... Bitcoin and cryptocurrency mining explained with the Byzantine Generals Problem. We use it to explain the essence of cryptocurrency mining. https://www.udemy... In this Bitcoin protocol tutorial video, the notion of Proof of Work is discussed. The idea of proof of work, normally, is to protect against spam and DoS attacks. With bitcoin, however, it is ...