Hacker News from Y Combinator

Syndicate content
Links for the intellectually curious, ranked by readers. // via fulltextrssfeed.com
Updated: 3 hours 25 min ago

The Email I Received from Google in 2007 When They Wanted to Buy Zilo (2012)

3 hours 25 min ago

Little funny story:

In March 2007, we signed a term sheet with Mangrove who wanted to invest in Zlio..A few days after, I met a Google Executive who decided to buy the company.

I just found in my mailbox, the email I received from it. Please see it below. As you can see, I received this email before I signed an NDA. That’s why I’m putting it here ;-). Finally, Google needed more than 1 month to close the deal. After waiting for it 30 days, we decided to accept the Mangrove investment proposal.

For the little story, Zlio, became blacklisted/sandboxed by Google 6 months after…. It killed the company…

Do you think I regret it ? Clearly not ! See why here

PS: You can follow me on Twitter

Email received from A.H. on March 22, 2007.


Thanks for this.  I have had internal conversations now. We would like to move on this urgently as an acquisition. We realise you are late in your cycle and we are early in ours but frankly your idea and personal background is a good indication of strength in our mind and we have already got the internal senior nod to proceed. Add to this the fact that we share your vision and would want to execute on something similar anyway means we would like to move forward discussing with you. We believe the combination of your vision, leadership and first mover advantage with our ability to create Googlely products, our consumer and partner relationships and distribution, our checkout product and our scale infrastructure will lead to a great solution that enables you as an entrepreneur to reach your goals more efficiently. We have a pretty smooth acquisition process here at Google to make deals happen but here is how you can help us be quick:

1) sign the nda so we can start to exchange info – this is our standard nda. You should realise that we make absolutely sure that no engineers working on similar areas in Google are involved in the assessment of this deal at this stage so we can avoid tainting
2) send me captable (with names) and loan/liabilities info (Jerry tells me there are some complexities around ownership?), CVs of your team (we are especially keen on good engineering and prod people and in Israel we are already building out our team. You can delete the names on the top of the CVs if you want to protect anonymity)
3) consider location – Israel is great for your prd and eng people but relationship, partnership and business team may have to relocate to UK or US but we do have a Paris office that may be expanded – I just cant guarantee that right now.
4) consider valuation – if we are to act as quickly as you want we need strong guidance. You can give me a range and I will respond. You have indications of value from your VC termsheet and I can tell you that we will give strong monetary incentives to the people who join us in terms of bonuses etc on top of consideration for the equity of the company. Naturally, we believe we take away the risk of successful execution through an acquisition so value of the equity should reflect this.
5) send me financials
6) send me IPR/patent details
7) send me something that describes how difficult this is to do from an engineering challenge and arrange a conference call between your lead engineer and one of our deal team engineer members to have a discussion around this

I will take all this and the corporate presentation you have given me and create a presentation for internal use to our executive board (Larry, Sergey and Eric) for approval to negotiate then issue a termsheet to you

We will want to do some DD post then just like a VC, normally we front end some of this via a visit to meet you and the team but in the interests of time we can back end this but you need to be helpful with the 7 points above instead.

I hope this is exciting news for you as it is for us….

Let me know and also pls send me your mobile phone as it is not on your business card

Share on FacebookShare on Google+Share on Linkedin

A tiling window manager written in Rust

3 hours 25 min ago

Dissent – Accountable anonymous group communication

3 hours 25 min ago

The Dissent project is a research collaboration between Yale University and UT Austin to create a powerful, practical anonymous group communication system offering strong, provable security guarantees with reasonable efficiency. Dissent's technical approach differs in two fundamental ways from the traditional relay-based approaches used by systems such as Tor:

  • Dissent builds on dining cryptographers and verifiable shuffle algorithms to offer provable anonymity guarantees, even in the face of traffic analysis attacks, of the kinds likely to be feasible for authoritarian governments and their state-controlled ISPs for example.

  • Dissent seeks to offer accountable anonymity, giving users strong guarantees of anonymity while also protecting online groups or forums from anonymous abuse such as spam, Sybil attacks, and sockpuppetry. Unlike other systems, Dissent can guarantee that each user of an online forum gets exactly one bandwidth share, one vote, or one pseudonym, which other users can block in the event of misbehavior.

Dissent offers an anonymous communication substrate intended primarily for applications built on a broadcast communication model: for example, bulletin boards, wikis, auctions, or voting. Users of an online group obtain cryptographic guarantees of sender and receiver anonymity, message integrity, disruption resistance, proportionality, and location hiding.

See our CCS '10, OSDI '12, and USENIX Security '13 papers describing the experimental protocols underlying Dissent. Also feel free to check out the source code at the link below, but please keep in mind that it is an experimental prototype that is not yet ready for widespread deployment by normal users.

Communication Model Dissent conceptually builds a "shuffled send" anonymous multicast primitive
Anytrust cloud architecture A multi-provider, “anytrust” cloud architecture offers scalability to large anonymity sets

Obama to Announce Easing in U.S.-Cuba Relations

3 hours 25 min ago

Dec. 17 (Bloomberg) -- President Barack Obama is set to announce steps towards normalization of U.S.-Cuba relations following the release of American Alan Gross who has been hold captive by the Cuban government for five years. Bloomberg’s Phil Mattingly reports on “In The Loop.”

President Barack Obama will end the U.S. isolation of Cuba that has persisted for more than a half century, initiating talks to resume diplomatic relations, opening a U.S. embassy in Havana and loosening trade and travel restrictions on the nation.

The steps effectively end an embargo that has been one of the most durable elements of U.S. foreign policy. Travelers will be able to use credit cards in Cuba and Americans will be able to legally bring home up to $100 in previously illegal Cuban cigars treasured by aficionados.

Obama is scheduled to deliver remarks from the White House at 12:01 p.m. Washington time, about the same time that Raul Castro is set to speak in Cuba. The two spoke yesterday about the deal, an administration official said.

“Decades of U.S. isolation of Cuba have failed to accomplish our enduring objective of promoting the emergence of a democratic, prosperous, and stable Cuba,” the White House said in a fact sheet e-mailed to reporters. “We know from hard-learned experience that it is better to encourage and support reform than to impose policies that will render a country a failed state.”

The embargo is one of the last remnants of the Cold War, sacrosanct in U.S. domestic politics because of the influence of the Cuban-American exile community in Florida. Generational shifts have reduced support for the embargo, though Obama’s moves drew criticism from some Cuban-American lawmakers.

Photographer: Spencer Platt/Getty Images

A woman walks under a Cuban flag in Santiago de Cuba, Cuba. Close

A woman walks under a Cuban flag in Santiago de Cuba, Cuba.


Open Photographer: Spencer Platt/Getty Images

A woman walks under a Cuban flag in Santiago de Cuba, Cuba.

The White House announced the steps after Cuba released American Alan Gross on humanitarian grounds. Following high-level talks between the governments since the spring, the U.S. and Cuba also made a parallel prisoner exchange of three Cuban intelligence agents for a U.S. intelligence asset who has been imprisoned for more than 20 years, according to administration officials who briefed reporters on condition of anonymity before Obama speaks.

Cuba also agreed to release 53 people the U.S. considers political prisoners, some of whom have already been released, the officials said.

Obama, Castro Speak

Pope Francis, the Catholic Church’s first Latin-American pontiff, played a role in the talks, sending a letter to Cuban leader Raul Castro and Obama urging negotiation. The Vatican hosted delegations from two countries for talks.

Limits on Cuban-Americans’ remittances to relatives in their homeland will jump to $8,000 from $2,000 annually. U.S. companies will be permitted to export an expansive list of goods including building materials and allowed to build telecommunications infrastructure on the island.

Photographer: Alex Wong/Getty Images

Supporters of U.S. citizen Alan Gross at the Lafayette Park outside the White House Dec. 3, 2013. Close

Supporters of U.S. citizen Alan Gross at the Lafayette Park outside the White House Dec. 3, 2013.


Open Photographer: Alex Wong/Getty Images

Supporters of U.S. citizen Alan Gross at the Lafayette Park outside the White House Dec. 3, 2013.

Senate Foreign Relations Chairman Robert Menendez, who gives up his gavel to a Republican in January, criticized Obama’s action, saying it “sets an extremely dangerous precedent” that “invites dictatorial and rogue regimes to use Americans serving overseas as bargaining chips.”


“President Obama’s actions have vindicated the brutal behavior of the Cuban government,” said the New Jersey Democrat, whose parents fled Cuba during Fidel Castro’s reign. “I fear that today’s actions will put at risk the thousands of Americans that work overseas to support civil society, advocate for access to information, provide humanitarian services, and promote democratic reforms.”

Senator Richard Durbin, an Illinois Democrat close to Obama, said the action is long overdue.

“Opening the door with Cuba for trade, travel, and the exchange of ideas will create a force for positive change in Cuba that more than 50 years of our current policy of exclusion could not achieve,” Durbin, who visited Gross in Cuba in 2012, said today in an e-mailed statement.

Gross, a 65-year-old American, left Cuba on a U.S. government plane this morning to fly to the U.S., said an administration official familiar with the release.

Gross was arrested by Cuban officials while working to expand Internet access for Havana’s Jewish community. He was accused of undermining the Cuban state and in December 2009 was sentenced to 15 years in prison.

Cuba Embargo

Trade with Cuba has been severely restricted for half a century due to a Cold War freeze.

Under President John F. Kennedy, the Central Intelligence Agency backed an armed invasion of the island along with repeated attempts to assassinate Cuban President Fidel Castro, Raul’s older brother.

The U.S. has maintained an embargo on most trade with the island, a policy that periodically has come under pressure from American business interests that see potential profit there. The World Bank, citing 2011 data, pegs the island’s gross domestic product at more than $68 billion -- about what the U.S. produced that year in a day and a half, according to data compiled by Bloomberg.

Under Raul Castro, who took over from his brother, Fidel, as acting president in 2006 and succeeded him two years later, the Marxist nation has taken tentative steps away from central planning. In July, Cuba lowered its economic growth forecast for this year to 1.4 percent.

Earlier this week, Moody’s said the economy on the island about 90 miles off the coast of Florida was slowing, citing “decreased reform momentum.”

To contact the reporters on this story: Angela Greiling Keane in Washington at agreilingkea@bloomberg.net; Mike Dorning in Washington at mdorning@bloomberg.net

To contact the editors responsible for this story: Steven Komarow at skomarow1@bloomberg.net Joe Sobczyk

Press spacebar to pause and continue. Press esc to stop.

Blatant CSRF in Doorkeeper, most popular OAuth2 gem

3 hours 25 min ago
I read a post about CSRF on DigitalOcean (in Russian) by Sergey Belove. My first reaction was, obviously, how come? DigitalOcean is not kind of a team that would have lame "skip_before_action :verify_authenticity_token".

DigitalOcean uses Doorkeeper, the most popular OAuth Provider library for rails apps and it manages clients, tokens, scopes and validations out of box.
Then I looked into Doorkeeper's commit history... it turns out Doorkeeper's endpoints never had CSRF protection, because they inherit directly from ActionController::Base, not ApplicationController.

Which means any HTML page on the Internet can get your access_token with arbitrary scope (such as "email", "dialogs" or "withdraw_money") from any Doorkeeper-compatible Rails app you are logged in. Example:

    <form action="https://cloud.digitalocean.com/v1/oauth/authorize" method="POST">
      <input type="hidden" name="client_id" value="EVIL_APP_ID" />
      <input type="hidden" name="redirect_uri" value="http://CALLBACK" />
      <input type="hidden" name="response_type" value="code" />
      <input type="hidden" name="scope" value="ANY SCOPE" />

This is a big deal. You must upgrade Doorkeeper NOW.

P.S. It's funny that Sergey is not a Rails developer so he simply tried to send a request without authenticity_token. Frankly, I wouldn't try that - Rails has built-in CSRF protection everywhere, why even bother? That's why.

What Happened When Marissa Mayer Tried to Be Steve Jobs

3 hours 25 min ago

Credit Illustration by Matt Dorfman. Photographs by Getty Images. Continue reading the main story Share This Page Continue reading the main story Continue reading the main story Related Coverage

Eric Jackson was sitting in his hotel room on Sea Island, Ga., watching his kids splash around in the pool, when he clicked “publish” on his latest blog post for Forbes.com. Jackson, an influential hedge-fund manager, had become fixated on Yahoo and the efforts of its chief executive, Marissa Mayer, to turn around the enormous yet floundering Internet company. It was July 21, 2014, almost exactly two years to the day since Mayer took over, arriving at Yahoo’s headquarters to an unfurled purple carpet and Sherpard Fairley-style “HOPE” posters bearing her face. During those 24 months, Mayer eliminated dozens of products and rebooted others. She acquired 41 start-ups and even hired Katie Couric. But just one week earlier, Mayer announced the company’s lowest quarterly earnings in a decade. Jackson argued in his post that Yahoo no longer made sense as an independent entity. Instead, it might be a nice takeover target for one of the tech industry’s Big Four: Apple, Facebook, Amazon or Google.

Jackson’s conclusion wasn’t based simply on a discouraging quarter. It was a result of an eye-opening calculation he had performed — what’s known on Wall Street as a sum-of-the-parts valuation. Yahoo had a market value of $33 billion at the time, but that figure owed largely to its stake in Alibaba, the Chinese Internet conglomerate. According to Jackson’s valuation, Yahoo’s stake in Alibaba was worth roughly $37 billion. But if you subtracted that position, the entirety of Yahoo’s core business, all its web products and content sites, actually had a market valuation of negative $4 billion. A conquering company could theoretically buy Yahoo, sell off its Asian assets and absorb its business units free. This sort of sale would make a lot of money for Yahoo’s shareholders, Jackson wrote, even if it meant gutting the company and losing Mayer as C.E.O. after only two years.

A day after his post, Jackson received an unusual email. A major Yahoo shareholder had written to explain that he and many other investors, along with numerous employees and advertisers, had themselves become extremely frustrated with Mayer. Her turnaround plan, he said, had failed. The start-ups she acquired (most notably the social blogging platform Tumblr, which Yahoo bought for $1.1 billion in 2013) had failed to revive the company’s flat revenues of roughly $5 billion per year. Nor had Mayer succeeded, despite her track record overseeing Google’s search engine, in turning any of Yahoo’s many products into an industry leader. There were also a number of embarrassing management setbacks. The best outcome for Yahoo, the shareholder said, might be to sell the company.

One day later, on July 23, Jackson published a subsequent Forbes column that outlined parts of the shareholder’s argument. It went viral within the investment community, and in the following days, Jackson fielded several calls from other significant Yahoo investors, including managers of large mutual funds and hedge funds, who encouraged him to continue his campaign. Jackson’s own fund didn’t have the capital to mount an offensive, which would require buying a large stake in Yahoo and using it as leverage to effect changes within management. But he knew someone who might. Jeffrey Smith, who ran an activist fund called Starboard Value, had recently led such a campaign against AOL, leading the company to eliminate its money-losing local news network, Patch. While still on vacation, Jackson looked up Smith’s email on his Bloomberg terminal. Within hours, the two men were on the phone with a handful of associates from Starboard.

Jackson’s dire calculation of Yahoo’s value would soon be reinforced in the markets. On Sept. 19, Alibaba went public on the New York Stock Exchange, closing at $93.89 per share. But as Alibaba’s stock soared, Yahoo’s dropped, an indication that the market seemed to concur with Jackson’s analysis: Yahoo’s core business was worth less than zero dollars. A week later, Smith published an open letter calling for Yahoo to divest itself of its Alibaba assets, return the money to its shareholders and then merge with AOL. Redundancies could be eliminated, thousands of people could be fired and two former Internet superpowers would be downsized into a single and steady (if uninspiring) entity that sold ads against its collective online properties — news, blogs and Web products like email, maps and weather. “We trust the board and management will do the right thing for shareholders, even if this may mean accepting AOL as the surviving entity,” Smith wrote.

Dynamic and wildly profitable Internet companies like Facebook and Google may get most of the attention, but Silicon Valley is littered with firms that just get by doing roughly the same thing year after year — has-beens like Ask.com, a search engine that no longer innovates but happily takes in $400 million in annual revenue, turning a profit in the process. Mayer, who is 39, was hired to keep Yahoo from suffering this sort of fate. She believed it could again become a top-tier tech firm that enjoyed enormous growth and competed for top talent. And two years in, Mayer, who has a tendency to compare herself with Steve Jobs, wasn’t about to abandon her turnaround plan. On the afternoon of Oct. 21, she entered a web TV studio on Yahoo’s garrisonlike campus to present the company’s latest quarterly results. But the presentation effectively became a response to Starboard’s campaign. Even though Yahoo’s revenue had decreased in five of the past six quarters, Mayer attested that she had “great confidence in the strength of our business.”

Mayer’s resolve was consistent with other remarks she had made at the time, in both public and private. She highlighted various signs of promise. Yahoo’s mobile revenues, while still small, had doubled from the previous year. Display advertising revenue was down 6 percent, but the number of ads sold had actually increased by 24 percent. Yahoo was engaging more mobile users than ever before. Mayer didn’t bother talking about a potential AOL takeover. Her goal was nothing less than to return her company to the level of the Big Four. “We believe deeply in the future potential of Yahoo,” she said into the camera, “and the transformation we are pursuing to bring an iconic company back to greatness.”

Generally speaking, there are only a few ways to make money on the Internet. There are e-commerce companies and marketplaces — think Amazon, eBay and Uber — that profit from transactions occurring on their platforms. Hardware companies, like Apple or Fitbit, profit from gadgets. For everyone else, though, it more or less comes down to advertising. Social-media companies, like Facebook or Twitter, may make cool products that connect their users, but they earn revenue by selling ads against the content those users create. Innovative media companies, like Vox or Hulu, make money in much the same way, except that they’re selling ads against content created by professionals. Google, which has basically devoured the search business, still makes a vast majority of its fortune by selling ads against our queries.

Yahoo essentially invented the online-advertising business. In 1994, two graduate students at Stanford, Jerry Yang and David Filo, dreamed up a way to help early users navigate the web. They picked URLs that they each liked — beginning with around 100 links, including one for Nerf toys and one dedicated to armadillos — and listed them on a page called “Jerry and David’s Guide to the World Wide Web.” Within a year, their guide had to be divided into 19 categories (art, business, etc.) and was generating one million clicks a day. In 1995, the year Yahoo started selling ads, a former company executive estimated that the entire market was about $20 million. By 1997, Yahoo’s ad revenues alone were $70.4 million. The next year, they were $203 million.

To keep up with the growth, Yahoo quickly expanded beyond its directory to create a multitude of ad-supported products. The company aimed to be all things to all web users, and for most of a decade, it was a wildly successful strategy. In 1997, Yahoo added chat rooms, classified ads and an email service. In 1998, it introduced sports, games, movies, real estate, a calendar, file sharing, auctions, shopping and an address book. Even during the crash of the Internet bubble, a profusion of more traditional advertisers began to migrate from print to digital. The search business, in particular, was growing enormously. In 2002, Yahoo’s first full year monetizing search results with attendant ads, its revenues reached $953 million. In 2003, they eclipsed $1.6 billion. In 2004, they grew again to $3.5 billion. At its peak, Yahoo’s market capitalization reached $128 billion. It was $20 billion larger than Berkshire Hathaway, Warren Buffett’s holding company.

But this growth obscured a looming problem. While Yahoo was busy enlarging its portfolio, a new generation of start-ups was focusing on perfecting one single product. Soon enough, Yahoo was losing out to eBay in auctions, Google in search and Craigslist in classifieds. Then Facebook came along, replacing Yahoo as the home page for millions of people. The advertising dollars soon followed, and Yahoo’s revenue flattened. Between 2007 and 2012, the company churned through four C.E.O.s. The last of them, Scott Thompson, resigned in disgrace after five months when a large activist shareholder, Dan Loeb, published an open letter accusing him of fabricating a computer-science degree. After Thompson’s resignation, in May 2012, Yahoo was worth less than $20 billion on the public markets.

As an ad-supported business, Yahoo had only two ways to increase its revenue. It could display more ads by attracting more people to its products — a plan that would require inventing (or acquiring) new products, improving old products or some combination. Alternately, it could elevate ad prices by upgrading its content. In the opinion of Thompson’s interim successor, Ross Levinsohn, Yahoo would be best positioned as such a “premium” content company. In Levinsohn’s vision, Yahoo had fallen so far behind its competitors in building successful back-end technology, like real-time advertising auctions and search, that it should cede most of those businesses altogether. In the process, the company could also shed more than half of its 15,000 employees, and home in on its best asset: reach. Some 700 million people still visited Yahoo’s home page every month, making it almost seven times as large than the combined online audiences of The New York Times, The Daily Mail and The Washington Post. Levinsohn believed that offering this audience better content could raise Yahoo’s earnings by up to $2 billion in two years.


But in Silicon Valley, the big money, and most of the prestige, is in making cool products. This is partly because product businesses, based on technology, are easier to scale than content ones, which require more human labor. It also reflects a cultural bias. The Valley’s greatest companies, from Hewlett-Packard onward, have been built around technological ingenuity. Tech executives know how to hire engineers and designers; they’re less adroit at recruiting editors or producers. When Loeb joined the Yahoo board, he recruited Michael J. Wolf, the former president of MTV, and the two consulted the noted venture capitalist Marc Andreessen about who should become the next C.E.O. of Yahoo. Andreessen made the case for a product executive.

Loeb’s decision was facilitated by another factor too. At the time, Google was valued at $250 billion; Facebook was worth $100 billion. Loeb opted to refashion Yahoo in their image. And so, in the spring of 2012, Loeb and Wolf began coveting a product C.E.O. The two board members asked the executive recruiter Jim Citrin of Spencer Stuart to approach Marissa Mayer, the wunderkind engineer who oversaw the user interface of Google’s search engine. Citrin cautioned that Mayer, who was among the first 25 people to join Google back in 1999, appeared to be a lifer. She had already earned hundreds of millions of dollars following Google’s 2004 I.P.O. and presumably had her pick of job opportunities. Still, he said he’d reach out to her.

Unknown to Citrin, however, Mayer was interested in pursuing her own turnaround of sorts. A couple of years earlier, she lost a turf battle to a powerful engineer within Google and was quietly reassigned to oversee Google Maps and other so-called local products. Mayer tried to spin the move positively, but that became harder after Larry Page, one of the company’s founders, regained the role of C.E.O. and removed Mayer from the group of executives reporting to him. According to one friend, Mayer had been observing the Yahoo vacancy for months. After Citrin called her cellphone, she told him she was interested.

The challenge of redirecting Yahoo was immense, but the next C.E.O. would have one tremendous advantage. In 2005, Yahoo invested $1 billion for a 40 percent stake in a little-known company called Alibaba. It turned out to be a remarkably prescient bet. Alibaba is commonly referred to as the Google of China, but it’s something more akin to the country’s version of Google, eBay and Amazon all in one — a web portal that provides e-commerce and business to business services. Weeks before Mayer was hired in July 2012, Yahoo sold half its 40 percent stake back to Alibaba for $7.1 billion. As a part of that deal, Alibaba agreed to hold an initial public offering sometime before the end of 2014. Suddenly the easiest way for Wall Street to make a bet on Alibaba — a hot start-up in a hot market, guaranteed an I.P.O. — was through an investment in Yahoo.

The arrangement ensured that Yahoo’s stock, for the next two years, would be tied to the performance of Alibaba rather than that of its own core business. This was a tremendous benefit to an incoming C.E.O., essentially offering a two-year air cover. Without having to manage the company’s stock price, inevitably one of a chief executive’s most distracting tasks, Mayer could focus on acquiring start-ups, jump-starting products and making strategic changes. Moreover, in two years she would be able to use the Alibaba cash to reinvest in her putative growth. When Citrin offered the job, she accepted.

Turning around a technology company has been historically rare. Tech companies invent new ways of doing things, but as they expand, often metastatically, they tend to shift their focus toward protecting their booming business rather than investing in new disruptive ones. Inevitably some newer company, usually financed by a wealthy venture-capital firm, beats them to it. It’s a cycle that happens in all industries, but everything moves faster in technology — too fast, usually, to allow for turnarounds. Steve Jobs may have resurrected Apple, and IBM was able to reinvent itself from a P.C. company into a business-services firm. But the next best example is probably Jeffery Boyd’s deliverance of Priceline — not exactly a titan of the industry. And his plan was almost a total restart, including dialing down the “name your price” pitch line and switching the company’s strategy from air travel to hotel bookings.

In order to revive Yahoo as a product company, Mayer would try to treat it as a giant start-up itself. Hours after entering Yahoo’s complex on the morning of July 17, 2012, she set up her computer to log into the company’s code base so she could personally make changes, much like the founder of a tiny tech firm might do. During her second week, she devised a weekly all-staff update, called “F.Y.I.,” that she would host at URL’s, a cafeteria on the Yahoo campus. (Employees pronounce it “Earl’s.”) She also tried to make the company a more desirable place to work. Yahoo employees, until then confined largely to BlackBerrys, were given iPhones or Samsungs as their company phones. All meals at URL’s would henceforth be free. For years, the partitions between the bathroom stalls didn’t go all the way to the wall. People hung toilet paper to try to fill the gaps. Soon after Mayer’s arrival, new partitions were installed.

Mayer saw her plan as a return, in a sense, to Yahoo’s original mission. Yahoo grew in popularity and value during the late 1990s, when it was the most user-friendly way to peruse the World Wide Web. Now, Mayer believed, it could ride the shift from P.C.s to smartphones and make the mobile web-browsing experience more user-friendly too. Yahoo, in other words, would need to become a really great apps company. Mayer wanted to narrow its product portfolio down to approximately a dozen from more than 100. She and her C.M.O., Kathy Savitt, did some market research and found a list of common user activities on mobile devices. She called this list the “Daily Habits,” and they included news-reading, checking weather, reading email and photo-sharing. Mayer was determined to ensure that Yahoo had the best mobile app for each.

This was going to be difficult. Previous Yahoo C.E.O.s had underinvested in mobile-app development, plowing money into advertising technology and web tools instead. A couple of days into the job, Mayer was having lunch at URL’s when an employee walked up to her and introduced himself as Tony. “I’m a mobile engineer,” Tony said. “I’m on the mobile team.”

Mayer responded to Tony, “Great, how big is our mobile team?” After some back and forth, Tony replied that there were “maybe 60” engineers. Mayer was dumbfounded. Facebook, for instance, had a couple of thousand people working on mobile. When she queried the engineering management department, it responded that Yahoo had roughly 100. “Like an actual hundred,” Mayer responded, “or like 60 rounded up to 100 to make me feel better?” The department responded that it was more like 60.

Companies like Facebook and Google are known for their fast-paced product updates. Yahoo, by contrast, was sluggish. Yahoo Mail, with its 30 billion emails a day, was arguably the company’s most important product. But despite the decline in desktop email use, Yahoo hadn’t built mail apps for smartphones. It had simply made the Yahoo Mail website usable on smaller mobile screens. Now an app was going to be designed for four separate platforms, including both Apple’s and Google’s mobile operating systems. During her first meeting with the executive in charge of Mail and Messenger, Vivek Sharma, Mayer said that she wanted it done by December. (Mayer declined to comment for this article.)

Mayer subsequently immersed herself in the redesign. Months into her tenure, she was meeting with Sharma’s team regularly in a conference room that started to look more like a design studio: projectors hung from the ceiling, rendering screens displayed on the wall. All around, dozens of foam core boards were pinned with ideas. Mayer would regularly interrogate designers about the minutest details of display and user experience. By early December, one day before Yahoo Mail was set to release, she convened a meeting at Phish Food, a conference room in the executive building of Yahoo’s campus, to talk about the product’s color. For months, the team had settled on blue and gray. If users were going to read emails on their phones all day long, the thinking went, it was best to choose the most subtly contrasting hues. But now, Mayer explained, she wanted to change the colors to various shades of purple, which she believed better suited Yahoo’s brand.

Some around the table were encouraged that their C.E.O. refused to release a product that she was less than fully satisfied with. If changing a few pixels led to an increase of 0.01 percent more users, that could translate to millions of dollars in ad revenue. Others, however, were visibly furious. According to one senior executive, Sharma’s body language changed the moment Mayer issued her request. He looked deflated. Altering the color of such an intricate product would require that members of his team spend all night adjusting colors in thousands of places. He slumped off and prepared to tell his staff the bad news.

In reality, Yahoo needed to move fast. And Mayer, who had begun her tenure while six months pregnant, tried to lead by example; she often slept only four hours a night and thrived on the breakneck pace. In her first months on the job, she unveiled a new version of Flickr, Yahoo’s photo-sharing social network, and a new Yahoo home page. Then Mayer overhauled the same products again, even as the company released new apps like Yahoo Weather and Yahoo News Digest. She outbid Facebook by a couple hundred million dollars to buy Tumblr. In the three quarters before Mayer’s arrival, Yahoo’s home-page team tested five new looks. In Mayer’s first two months, she prototyped 37.

Mayer also announced that she wanted to repair Yahoo’s search engine, her specialty at Google. “I don’t see a reason why our search share should fall below 15 percent, which is where it roughly is today,” she told employees at an F.Y.I. “I also don’t see a reason why it can’t climb back up to the 20 percent that we had.” According to one executive, she told a group of six vice presidents that she wanted a redesigned search product unveiled by the end of the year. “Tell me if you can do it,” she instructed the group. “Otherwise, I’ll find people who can.”

Largely owing to Mayer’s age and Google pedigree, her hiring had initially brought her overwhelmingly positive press during her first months at Yahoo. An article by Dan Frommer, an influential beat reporter for the industry news site ReadWrite, summed up a popular view: “This is a great move for Yahoo, which has stewed in mediocrity for years.” And by the beginning of 2013, just six months into the job, Mayer’s turnaround seemed to be ahead of schedule. In March, Yahoo’s stock rose to $22 per share, and Mayer added a tool to the company’s internal network that allowed employees to view their stock compensation.


Credit Illustration by Matt Dorfman

At a board meeting in April, Mayer admitted that she had not yet identified a “breakthrough product,” but she reminded those in attendance that Steve Jobs didn’t come up with the iPod until five years into his second tenure at Apple. At an F.Y.I. around that time, she read a speech that Jobs gave to Apple employees at the beginning of his turnaround. Afterward, channeling Jobs, Mayer told hundreds of employees sitting at URL’s, “Our purpose is to inspire and delight our users, to build beautiful services, things that people love to use and enjoy using every day, and that’s our opportunity.” She continued: “We are the world’s largest start-up. We have $5 billion in revenue, but it can and will go in the blink of an eye if we don’t do our jobs.”

At Google, Mayer made her name as the executive who helped determine how products would look and work. She was less sure, however, about how to monetize them. At Yahoo, her most important hire was supposed to be someone who could figure all that out for her while also running Yahoo’s 1,000-plus-person sales force. Mayer did not have to search far. Within weeks of becoming C.E.O., she received an email from Henrique de Castro, the fashionable Portuguese president of Google’s media, mobile and platforms businesses. De Castro asked if she was available for a dinner and suggested a place. Mayer countered with a quieter venue and even arranged to get a table in the back.

Over dinner, de Castro impressed Mayer with his knowledge of Yahoo’s business and his specific proposals for building it. For several mornings in a row, the two exchanged emails to negotiate de Castro’s salary. Every night, Mayer would make an offer, only to wake up to a reply with a list of more conditions. Eventually de Castro negotiated himself a contract worth around $60 million, depending on the value of Yahoo stock. The Yahoo board was aghast, but Mayer argued that de Castro had to be compensated for the unvested Google stock he was abandoning. Anyway, she said, his talent ensured that Yahoo would reap many multiples of his salary in return. Henrique de Castro would pay for himself.

Mayer’s turnaround plan may have been predicated on building irresistible products to sell ads against. But hundreds of employees at Yahoo already worked on a side of the business that generated advertising revenues the more traditional way — by creating or licensing content, from beauty tips and Kardashian stories to daily videos on the financial markets and scripted comedies. For most of her first year at Yahoo, Mayer hardly paid any attention to this group, which generated $1.5 billion in annual revenue. This began to change in the spring of 2013, however, when Mayer attended the NewFronts, the annual event in New York where digital-content businesses display their coming programming for ad-agency buyers. Mayer kicked off the show by reading a corporate script from a prompter in an office-casual cardigan. Her talk felt woefully out-of-place at a splashy event whose other attendees included Ed Helms, the actor, and the Lumineers, the popular rock group. Still, the glamour of the event seemed to ignite Mayer’s interest in content, and within a month she asked that all programming decisions be run by her.

While some at the company favored upgrading Yahoo’s content, there was a fear that Mayer, who preferred to read Town and Country and wear Oscar de la Renta couture, might undermine the company’s middle-American brand. To some, she also seemed to lack the instincts of a media executive. During a breakfast with Anna Wintour, the editor in chief of Vogue, Mayer asked if there might be any partnership opportunities between the magazine and Shine, Yahoo’s site for women. According to Mayer’s own telling of the story to top Yahoo executives, Wintour looked appalled. Shine, with its 500 million monthly page views, appealed to a mass audience, not a narrow and affluent one. Nevertheless, Mayer quickly became infatuated with the idea that Yahoo could attract more sophisticated consumers. She began pushing for deputies to commission high-quality shows, the way Netflix was doing with “House of Cards” and “Orange Is the New Black.” One Yahoo executive was forced to explain that only a company that sold subscriptions to consumers could expect to make money off such expensive productions.

Reared in Google’s data-obsessed culture, Mayer tended to require countless tests about user preferences before making an important product decision. But when it came to media strategy, she seemed perfectly comfortable going with her gut. As a teenager in Wisconsin, she grew up sneaking into the living room to watch “Saturday Night Live” and occasionally recited sketches during meetings; in April 2013, Yahoo paid an estimated $10 million per year for the “S.N.L.” archives. Even though the actress Gwyneth Paltrow had created a best-selling cookbook and popular lifestyle blog, Mayer, who habitually asked deputies where they attended college, balked at hiring her as a contributing editor for Yahoo Food. According to one executive, Mayer disapproved of the fact that Paltrow did not graduate college.

Over the summer, Mayer greenlighted a plan to hire Katie Couric, the former anchor of “CBS Evening News” and former co-host of the “Today” show. As was the case with de Castro, Couric put the idea in Mayer’s head herself after the two shared a stage at an advertising event in the Turks and Caicos. Couric, who was then hosting a failing daytime talk show on ABC, told Mayer she wanted to do something big for Yahoo. Couric had previously worked with the company to produce a video series, “Katie’s Take,” in which she interviewed experts on topics like health and parenting. Despite Couric’s star power, users didn’t click on her videos, no matter how prominently editors positioned them on the page. Mayer ignored those metrics, and in mid-2013, she named Couric Yahoo’s “global anchor” in a deal worth more than $5 million a year.

Yahoo also lured a slate of expensive journalists to helm a series of new “digital magazines.” Mayer hired David Pogue, The New York Times’s gadget columnist, to be the editor of Yahoo Tech. She personally recruited Joe Zee, the creative director of Elle, telling him to consider Yahoo his “playground.” Mayer’s team asked the former Page Six editor Paula Froelich to run Yahoo Travel; the makeup star Bobbi Brown would oversee Yahoo Beauty. And Yahoo Shine, with its $45 million in yearly revenue, was shot down.

For the NewFronts event in June 2014, Mayer wore a designer dress and introduced a slate of buzzy new shows that she personally approved. Behind the scenes, though, her hands-on strategy was backfiring. “I just think it is a strategic mistake to take on big media where they are strongest,” Jonah Peretti, the C.E.O. of Buzzfeed, wrote me in an email earlier this year, referring to her focus on stars, scripted shows and glossy content. “Especially when there is a huge open space at the intersection of media and tech where it is hard for other big companies to compete.” Couric, meanwhile, had done several interviews with high-profile figures, including former Secretary of Defense Robert Gates, but Yahoo’s users weren’t clicking on the videos. In May, Yahoo Tech had just nine million visitors, placing it in seventh place among competitors, far behind rivals like CNET and Gizmodo. Yahoo Tech would sometimes go weeks without running a single ad. Yahoo Food was 12th in its market. Yahoo’s display advertising revenues dipped 7 percent during the second quarter of 2014.

One of the Yahoo board’s hesitations upon hiring Mayer was her relative lack of experience as a manager. While running search at Google, she oversaw 250 people. Mayer liked to spin her demotion by saying that it left her in charge of more than 1,000 staff members, but a majority of them were contractors. Either way, in her haste to turn around Yahoo, this relative inexperience began to surface. Some on the board had hoped that Ross Levinsohn would stay on as C.O.O., but any hope was jettisoned when Mayer asked him to fly from L.A. for a meeting and then stood him up. Mayer’s refusal to delegate became a sticking point, too. She insisted on personally approving every hire. One executive complained to a friend that Mayer spent as much time deliberating Yahoo’s parking policies as she did strategizing over the sale of its Alibaba stock.

Mayer also had a habit of operating on her own time. Every Monday at 3 p.m. Pacific, she asked her direct reports to gather for a three-hour meeting. Mayer demanded all of her staff across the world join the call, so executives from New York, where it was 6 p.m., and Europe, where it was 11 p.m. or later, would dial in, too. Invariably, Mayer herself would be at least 45 minutes late; some calls were so delayed that Yahoo executives in Europe couldn’t hang up till after 3 a.m. In theory, Mayer kept up with her direct reports through weekly individual meetings. In practice, she often went weeks without seeing them.

This delinquency eventually became a problem outside Yahoo. At a major advertising event in the South of France, Mayer sat for an interview with Martin Sorrell, the C.E.O. of WPP, one of the world’s largest agencies. In front of a filled auditorium, Sorrell asked Mayer why she did not return his emails. Sheryl Sandberg, he said, always got back to him. Later, Mayer was scheduled for dinner with executives from the ad agency IPG. The 8:30 p.m. meal was inconvenient for the firm’s C.E.O., Michael Roth, but he shuffled his calendar so he could accommodate it. Mayer didn’t show up until 10.

At F.Y.I.s, Mayer liked to tell employees that she believed in taking risks and that she was unafraid to admit failure. This philosophy worked well for web products but not for strategic hires. Despite the board’s urging, Mayer opted against vetting Henrique de Castro. As a result, she was unaware that de Castro had a poor reputation among his colleagues in Google’s advertising business. Many had derisively called him the Most Interesting Man in the World, in reference to the satirically fatuous spokesman for Dos Equis beer. De Castro had a tendency to make grand, awkwardly worded pronouncements. He was the inspiration for the Twitter handle @HdCYouKnowMe, which posted tweets that straddled the line between reality and parody: “To incentivize the sales force, you need to hit them with the carrot” and “Product is like snakes . . . slippery — we need someone with a big hammer.” De Castro’s new Yahoo colleagues got a full dose of his strange locution at the company’s annual sales meeting, in early 2013, when he berated his sales force with a rangy, pedantic speech. (De Castro did not respond to requests for comment.)

De Castro’s plan for growing Yahoo revenues focused on user-generated content, like the videos available on YouTube or Instagram. The only problem was that Yahoo did not have access to enough user-generated content to support this plan. (Its attempt to acquire Daily Motion, a YouTube clone, had fallen apart.) As Yahoo’s ad revenues continued to decline, de Castro began to alienate his staff and fellow executives. After one of his direct reports gave a presentation about Yahoo’s business in front of some 40 senior executives, de Castro humiliated the person, saying: “I think your strategy’s more of a fantasy. You make it up. You just make it up.” More important, advertising revenue declined in every quarter since he was hired. Within a year, Mayer had personally taken control of Yahoo’s ad team. De Castro would leave the company in January 2014. For about 15 months of work, he would be paid $109 million.

Mayer’s largest management problem, however, related to the start-up culture she had tried to instill. Early on, she banned working from home. This policy affected only 164 employees, but it was initiated months after she constructed an elaborate nursery in her office suite so that her son, Macallister, and his nanny could accompany her to work each day. Mayer also favored a system of quarterly performance reviews, or Q.P.R.s, that required every Yahoo employee, on every team, be ranked from 1 to 5. The system was meant to encourage hard work and weed out underperformers, but it soon produced the exact opposite. Because only so many 4s and 5s could be allotted, talented people no longer wanted to work together; strategic goals were sacrificed, as employees did not want to change projects and leave themselves open to a lower score.

One of the uglier parts of the process was a series of quarterly “calibration meetings,” in which managers would gather with their bosses and review all the employees under their supervision. In practice, the managers would use these meetings to conjure reasons that certain staff members should get negative reviews. Sometimes the reason would be political or superficial. Mayer herself attended calibration meetings where these kinds of arbitrary judgments occurred. The senior executives who reported to Mayer would join her in a meeting at Phish Food and hold up spreadsheets of names and ratings. During the revamping of Yahoo Mail, for instance, Kathy Savitt, the C.M.O., noted that Vivek Sharma was bothering her. “He just annoys me,” she said during the meeting. “I don’t want to be around him.” Sharma’s rating was reduced. Shortly after Yahoo Mail went live, he departed for Disney. (Savitt disputes this account.)

As concerns with Q.P.R.s escalated, employees asked if an entire F.Y.I. could be devoted to anonymous questions on the topic. One November afternoon, Mayer took the stage at URL’s as hundreds of Yahoo employees packed the cafeteria. Mayer explained that she had sifted through the various questions on the internal network, but she wanted to begin instead with something else. Mayer composed herself and began reading from a book, “Bobbie Had a Nickel,” about a little boy who gets a nickel and considers all the ways he can spend it.

“Bobbie had a nickel all his very own,” Mayer read. “Should he buy some candy or an ice cream cone?”

Mayer paused to show everyone the illustrations of a little boy in red hair and blue shorts choosing between ice cream and candy. “Should he buy a bubble pipe?” she continued. “Or a boat of wood?” At the end of the book, Bobby decides to spend his nickel on a carousel ride. Mayer would later explain that the book symbolized how much she valued her roving experiences thus far at Yahoo. But few in the room seemed to understand the connection. By the time she closed the book, URL’s had gone completely silent.

Mayer’s plan to restore Yahoo to the ranks of the tech giants had been premised on producing apps that hundreds of millions of people wanted to use. But as the Alibaba I.P.O. approached this fall, Yahoo’s new and updated apps weren’t getting enough traction. The digital-magazine strategy had not taken off, either; few of the strategic acquisitions seemed poised to break out; and Yahoo’s search business, which Mayer hoped to grow to a 20 percent market share, had dropped to around 10 percent. Mayer’s plan to sell a new kind of advertising in apps had been undermined by her trouble building relationships with clients, like Sorrell and Roth. Despite her efforts, Yahoo was still not growing. On July 15, the company reported dismal second-quarter revenues. Weeks later, Alibaba went public, and Jeff Smith pounced.

Since Starboard began its campaign to force an AOL merger, Mayer has been meeting with shareholders to reassure them about her strategy. Many public-company C.E.O.s become belligerent when activist investors come charging, but Mayer appears to be open to a compromise. Most Yahoo observers expect that, in the coming weeks, she will announce a plan to transfer almost all of the company’s Alibaba gains to shareholders while simultaneously resisting any merger with AOL. Mayer often compares her situation with the one Jobs inherited, and many expect her to insist that she needs at least the five years he received before his turnaround began to succeed with the release of the iPod.

But Mayer may not be able to buy herself the time. Major Yahoo shareholders have recently begun collaborating on a series of spreadsheets that calculate that AOL and Yahoo are worth between 70 and 80 percent more when combined than they are apart. Some investors are further attracted to the merger because it will bring AOL’s chief executive, Tim Armstrong, into Yahoo. Like Mayer, Armstrong made his career as an early Google employee — but he was in charge of its sales force. After a rough start, Armstrong has managed to get AOL’s stock going again, not by inventing some new consumer product but by optimizing its ad and media assets.

Armstrong, who has seen a version of this analysis, appears willing to consider a deal. Doing so, of course, would provide enormous personal benefit; Armstrong, who owns 5 percent of AOL, could stand to gain tens if not hundreds of millions of dollars. For Mayer, the calculus is less enticing. It’s unlikely that her personal turnaround plan included shrinking a $30 billion company into a $5 billion one, all to combine it with a $3 billion company and realize $1 billion in cost-savings.

But it’s also unclear what, if any, other options she has. Turning Yahoo into a growing products company and appeasing its activist shareholders are now almost mutually exclusive tasks. If Yahoo were to sell off its stake in Alibaba, it would become an entirely different sort of entity. Its market capitalization could drop to $5 billion from $50 billion. Efficiencies that once seemed small would become, in the language of investors, “material.” Starboard might pressure Mayer to sell Yahoo’s real estate or to lay off 10,000 employees. Acquisitions would get harder, as smaller bets would become much bigger relative to Yahoo’s market capitalization.

Aswath Damodaran, a professor at N.Y.U.'s Stern School of Business, has long argued about the danger of companies that try to return to the growth stage of their life cycle. These technology companies, he said, are run by people afflicted with something he calls the Steve Jobs syndrome. “We have created an incentive structure where C.E.O.s want to be stars,” Damodaran explained. “To be a star, you’ve got to be the next Steve Jobs — somebody who has actually grown a company to be a massive, large-market cap company.” But, he went on, “it’s extremely dangerous at companies when you focus on the exception rather than the rule.” He pointed out that “for every Apple, there are a hundred companies that tried to do what Apple did and fell flat on their faces.”

In many ways, Yahoo’s decline from a $128 billion company to one worth virtually nothing is entirely natural. Yahoo grew into a colossus by solving a problem that no longer exists. And while Yahoo’s products have undeniably improved, and its culture has become more innovative, it’s unlikely that Mayer can reverse an inevitability unless she creates the next iPod. All breakthrough companies, after all, will eventually plateau and then decline. U.S. Steel was the first billion-dollar company in 1901, but it was worth about the same in 1991. Kodak, which once employed nearly 80,000 people, now has a market value below $1 billion. Packard and Hudson ruled the roads for more than 40 years before disappearing. These companies matured and receded over the course of generations, in some cases even a century. Yahoo went through the process in 20 years. In the technology industry, things move fast.

“Sometimes,” Damodaran told me, “companies have to act their age.” For Yahoo, embracing its maturity means settling for a business that earns close to $1 billion in profit every year. It has outlasted other formerly iconic Internet portals, from AltaVista to Excite, and even dwarfs more recent web sensations like Myspace and Ask.com. For a company that started out as “Jerry and David’s Guide to the World Wide Web,” that’s not a bad way to grow old.

Nicholas Carlson is chief correspondent for Business Insider. This article is adapted from  "Marissa Mayer and the Fight to Save Yahoo!” to be published next month by Twelve.

A version of this article appears in print on December 21, 2014, on page MM2 of the Sunday Magazine with the headline: No Results .

Farewell, Dr. Dobb's

15 hours 25 min ago

This year, our website will deliver almost 10.3 million page views, which is an unprecedented number for Dr. Dobb's. It's up from 9 million last year and 8 million three years ago. That kind of growth is somewhat unusual for a site that has not changed its look or its mission, nor indulged in tawdry tricks like click-bait headlines or slideshows promising 9 quick tips for choosing a coding style. The numbers confirm that there is a deep thirst in the programmer community for long-form technical content featuring algorithms and code, as well as strong demand for explanations of new developer technologies and reliable reviews of books and tools.

If I were so inclined, this might be the right time for me to move on, and so leave, as they say in sports, "at the top of my game." And indeed I will be leaving Dr. Dobb's at the end of the year. But it would be more accurate to say that it is Dr. Dobb's that is leaving: Our parent company, United Business Media (UBM), has decided to sunset Dr. Dobb's. "Sunset" sounds like a marketing euphemism to avoid saying "closing down," but in this context, it has a specific meaning that "closing" does not convey. That is, that there will be no new content after year end; however, all current content will be accessible and links to existing Dr. Dobb's articles will continue to work correctly. It is the equivalent of a product coming to end of life. It still runs, but no new features will be added.

Over the years, my editorials have frequently analyzed market forces operating on different segments of the developer universe, so it would be wrong for me not to do the same for an event as personal and close to home as this.


Why would a well-known site, dearly loved by its readers and coming off a year of record page views, be sunset by its owner?

In one word, revenue. Four years ago, when I came to Dr. Dobb's, we had close to $3 million in revenue, almost all of it from advertising. Despite our excellent growth on the editorial side, our revenue declined such that today it's barely 30% of what it was when I started. While some of this drop is undoubtedly due to turnover in our sales staff, even if the staff had been stable and executed perfectly, revenue would be much the same and future prospects would surely point to upcoming losses. This is because in the last 18 months, there has been a marked shift in how vendors value website advertising. They've come to realize that website ads tend to be less effective than they once were. Given that I've never bought a single item by clicking on an ad on a website, this conclusion seems correct in the small.

So vendors have redeployed their advertising dollars into more fruitful options. This is not a Dr. Dobb's-only phenomenon. Our direct competitors, BZ Media (parent of SD Times) and c4Media (InfoQ), are experiencing the same pressures. They have responded by putting on small conferences, which generate much of their revenue. Dr. Dobb's could do the same, but for the fact that our parent company is geared to large tradeshows, rather that many small events. (It owns Black Hat and Interop, among many other events.) Unfortunately, the software market today is so highly segmented that aside from vendor-sponsored events (JavaOne, Google IO, etc.), most successful programmer conferences are small, often very small. UBM argues (correctly, I believe): Why should we tie up resources starting a series of niche events that are unlikely to grow much, when we could put all that time, effort, and management attention into the bigger tradeshows and move the revenue up more quickly? The logic is unassailable.

So rather than continue with Dr. Dobb's until it actually loses money, they've decided to sunset the site — a sudden end to remarkably robust and wondrous journey that began 38 years ago.


No amount of analysis and explanation can mask the deep, personal sadness I feel at writing about this decision. Like many of you, I grew up reading Dr. Dobb's. For me, as I suspect it was for many of you, Dr. Dobb's Journal was the lifeline to a thorough understanding of programming. I recall that when the magazine appeared in my mailbox, all other activity for the day came to a sudden stop and the remaining hours were spent blissfully poring over article after article, soaking in the information. I learned C from Allen Holub's C Chest column, operating systems from the 18-part series on 386BSD, video programming from Michael Abrash's Black Book, and data compression from Mark Nelson. And so on — each month brought new, enabling insights and explanations of often arcane topics.

Having this deep, passionate connection, I felt lifted in ways not often encountered in one's career when I was approached about succeeding Jonathan Erickson, the editor who steered the magazine through its glory days in print. The honor of this position has fueled me every day, renewed by conversations in person with developers whose eyes would light up when I'd mention I worked on Dr. Dobb's.

Putting aside my feelings, I should note that recent events fulfill the original vision of Dr. Dobb's. The founders, Bob Albrecht and Dennis Allison, first put together a newsletter in 1976 with the specific aim of making programming information more accessible. It was an experiment in sharing.

Dr. Dobb's subsequent popularity meant that it became a worldwide means of sharing curated, high-quality programming info. The advent of the Web, which offered a vast array of new information sources, meant that Dr. Dobb's was no longer the central access point — a complicated transition for the team, but one wholly in keeping with the original mission. With the advent of Hacker News and Proggit and other aggregators, developers themselves began curating content from numerous sources, and in a certain way, our mission is now complete.

This should not suggest that there is no role anymore for Dr. Dobb's. As our page views show, the need for an independent site with in-depth articles, code, algorithms, and reliable product reviews is still very much present. And I will dearly miss that content. I wish I could point you to another site that does similar work, but alas, I know of none.

To the previous editors, especially Jon Erickson and Mike Swaine, to the many contributors, columnists, and bloggers (especially Al Stevens, Al Williams, Allen Holub, Andrew Koenig, Eric Bruno, Gastón Hillar, Herb Sutter, Mark Nelson, Pablo Santos, Scott Ambler, and Walter Bright), and to all of you, our dear readers, who sent us comments in the true spirit of sharing rather than admonishment, who helped us up if we slipped, and who gloried in our triumphs, allow me to quote Octavio Paz: Let me say "two words that all men have uttered since the dawn of humanity: thank you."

Paz goes on to say, "The word gratitude has equivalents in every language and in each tongue the range of meanings is abundant." Perhaps, none more abundant than in the sense I mean it today, as I thank you for so many blessings and contributions to Dr. Dobb's.

— Andrew Binstock
Editor in Chief
Twitter: platypusguy

P.S. Our managing editor, Deirdre Blake (dblakenew@gmail.com), who has toiled for two decades at Dr. Dobb's, will be looking for similar work after a short break. I will be returning to my former work of writing white papers and doing market analysis for technology vendors. If you want to stay in touch, please follow me on Twitter at @platypusguy or feel free to email me at my personal address, which is my first name at pacificdataworks.com.

The Elf on the Shelf is preparing your child to live in a future police state

15 hours 25 min ago
By Peter Holley December 16 at 1:32 PM
The Elf on the Shelf balloon floats down Sixth Avenue — with eyes wide open — during the Macy’s Thanksgiving Day Parade in New York. (Andrew Kelly/Reuters)

For some, the Elf on the Shelf doll, with its doe-eyed gaze and cherubic face, has become a whimsical holiday tradition — one that helpfully reminds children to stay out of trouble in the lead-up to Christmas.

For others — like, say, digital technology professor Laura Pinto — the Elf on the Shelf is “a capillary form of power that normalizes the voluntary surrender of privacy, teaching young people to blindly accept panoptic surveillance and” [deep breath] “reify hegemonic power.”

I mean, obvs, right?

The latter perspective is detailed in “Who’s the Boss,” a paper published by the Canadian Centre for Policy Alternatives, in which Pinto and co-author Selena Nemorin argue that the popular seasonal doll is preparing a generation of children to uncritically accept “increasingly intrusive (albeit whimsically packaged) modes of surveillance.”

Before you burst out laughing, know that Pinto comes across as extremely friendly and not at all paranoid on the phone. She’s also completely serious.

“The Elf on the Shelf” is both a book and a doll. The former is a soft pixie scout elf that parents are instructed to hide around the house. The accompanying book, written in rhyme, tells a Christmas-themed story that explains how Santa Claus keeps tabs on who is naughty and who is nice.

The book describes elves hiding in children’s homes each day during the holidays to monitor their behavior before returning to the North Pole each night with a report for “the boss.”

Because we live in a world grappling with corporate smartphone surveillance, behavior management apps in the classroom and private communication interceptions by various governments, Pinto —  a digital technology professor at the University of Ontario Institute of Technology — sees the Elf on the Shelf dolls as one development among many threatening our collective definition of privacy.

If she’s right, in all likelihood she’s fighting a losing battle. The Elf on the Shelf book sold over 6 million copies and joined the Macy’s Thanksgiving parade last year, according to the Daily Mail.

“I don’t think the elf is a conspiracy and I realize we’re talking about a toy,” Pinto told The Post. “It sounds humorous, but we argue that if a kid is okay with this bureaucratic elf spying on them in their home, it normalizes the idea of surveillance and in the future restrictions on our privacy might be more easily accepted.”

Until the introduction of Elf on the Shelf, Santa’s mythological helpers had always been relegated to the toy workshop, Pinto said. After the story and toy were introduced by Chanda Bell, a onetime Atlanta reading teacher, the traditional narrative changed to include the hiding, surveillance and back-and-forth travel, Pinto said.

“As evidenced by the millions of books and dolls sold,” the Toronto Star writes, “the story has become a cultural phenomenon, with parents littering their social media feeds with photos of the elf in strange places.”

Facebook, in fact, is how Pinto said she originally became aware of the doll. Last December, she began to see her Facebook friends who have kids posting Elf on the Shelf photos.

The more she read about the doll and the rules that accompany it, the more she began to feel like the game that resonated with the purpose of the infamous panopticon, Jeremy Bentham’s 18th Century design for a model prison (a central tower in a circular structure, surrounded by cells that made it impossible for prisoners to know if they were being watched).

She writes:

Elf on the Shelf presents a unique (and prescriptive) form of play that blurs the distinction between play time and real life. Children who participate in play with The Elf on the Shelf doll have to contend with rules at all times during the day: they may not touch the doll, and they must accept that the doll watches them at all times with the purpose of reporting to Santa Claus. This is different from more conventional play with dolls, where children create play-worlds born of their imagination, moving dolls and determining interactions with other people and other dolls. Rather, the hands-off “play” demanded by the elf is limited to finding (but not touching!) The Elf on the Shelf every morning, and acquiescing to surveillance during waking hours under the elf’s watchful eye. The Elf on the Shelf controls all parameters of play, who can do and touch what, and ultimately attempts to dictate the child’s behavior outside of time used for play.

“The whole thing with panopticonism under the Jeremy Bentham structure,” Pinto said, “is that you never quite knew if you were being watched or not and that forced you into behaving in a certain way. The elf is the same way.”

Pinto said she’s not the first person to be troubled by Elf on the Shelf’s surveilling. She’s said parents routinely contact her to say they changed the rules of the game after it made their families uneasy. And many kids, she said, often intuitively feel like spying and being a tattletale is wrong.

“A mom e-mailed me and told me that the first day they read the elf book and put the elf out, her daughter woke up crying because she was being watched by the elf,” Pinto recounted. “They changed the game so it wouldn’t scare the child.”

Emma Waverman, a blogger with Today’s Parent, told the Star that the idea of the elf watching someone all the time is “creepy.”

“It makes the motivation to behave something that’s external,” she told the paper. “If I’m not around or if the elf is not around, do they act crazy?”

Translated into academia-speak, Pinto and Nemorin make a similar point in Who’s the Boss?

“What is troubling,” they write, “is what The Elf on the Shelf represents and normalizes: anecdotal evidence reveals that children perform an identity that is not only for caretakers, but for an external authority (The Elf on the Shelf), similar to the dynamic between citizen and authority in the context of the surveillance state.”

For all you skeptics out there, take a look at this …

And here is a video explanation of the paper:

Peter Holley is a general assignment reporter at The Washington Post. He can be reached at peter.holley@washpost.com.

PhpBB suffers massive security compromise

15 hours 25 min ago
Update #3 17-12-2014 - 01:10

At this time we are proceeding with recovery efforts and have some additional important information.

We have confirmed that initial entry was made via a team member's compromised login details and not as the result of a vulnerability in the phpBB software. The phpBB download packages were never altered.

The attackers were able to obtain access to the phpBB.com and area51 databases, meaning that user information, including hashed salted passwords, was compromised. Additionally, all logins on area51 between Dec. 12th and Dec. 15th were logged in plaintext. While the hashing algorithm utilized in phpBB will make it difficult to obtain those passwords, you should not take any chances. If you were using your phpBB.com or area51 passwords anywhere else, you must change them.

We will provide full details, including the steps we have taken since the compromise, once we are back in operation.

Update #2 15-12-2014 - 23:30

On Sunday Dec. 14th, several of the web servers powering phpBB.com were compromised. Upon discovering the ongoing attack, we immediately took our network offline to perform a thorough investigation, which is continuing.

At this time, we would like to ask everyone to follow basic security protocol. If you were using your www.phpBB.com or area51.phpBB.com passwords anywhere else, please change them to unique ones.

Your personal phpBB Forums are NOT affected by the compromise of our servers.

We will be rebuilding our systems from the ground up and verifying the integrity of all data prior to coming back online. This process will likely take several days.

Further updates will be posted here when we have additional information.

- The phpBB Team

If you need urgent assistance, please make use of the #phpbb IRC channel on Freenode. A web-based client is available at http://webchat.freenode.net.

Leaving stuff lying around in logs on pastebins

15 hours 25 min ago
A look inside Facebook's source code - Sinthetic Labs

Sinthetic Labs | Blog

Welcome to our static blog. We use a thing called HTML. Crazy, right?

4th October, 2014 — Nathan Malcolm

Note: None of the code in this post was obtained illegally, nor given to me directly by any Facebook employee at any time.

I've always been a fan of Facebook from a technical point of view. They contribute a lot to the open source community and often open source their internal software too. Phabricator, libphutil, and XHP are great examples of that. For a while I contributed a bit to both Phabricator and XHP, and I ended up finding out a lot more about Facebook's internals than I intended. Read on...

It was mid-2013 and I was busy fixing a few bugs I had encountered while using Phabricator. If my memory serves me correctly the application was throwing a PhutilBootloaderException. I didn't have much knowledge of how Phabricator worked at the time so I googled the error message. As you'd expect I came across source code and references, but one specific link stood out. It was a Pastebin link.

Of course, this intrigued me. This is what I found...

[emir@dev3003 ~/devtools/libphutil] arc diff --trace >>> [0] <conduit> conduit.connect() <<< [0] <conduit> 98,172 us >>> [1] <exec> $ (cd '/home/emir/devtools/libphutil'; git rev-parse --show-cdup) <<< [1] <exec> 13,629 us >>> [2] <exec> $ (cd '/home/emir/devtools/libphutil/'; git rev-parse --verify HEAD^) <<< [2] <exec> 17,024 us >>> [3] <exec> $ (cd '/home/emir/devtools/libphutil/'; git diff --no-ext-diff --no-textconv --raw 'HEAD^' --) >>> [4] <exec> $ (cd '/home/emir/devtools/libphutil/'; git diff --no-ext-diff --no-textconv --raw HEAD --) >>> [5] <exec> $ (cd '/home/emir/devtools/libphutil/'; git ls-files --others --exclude-standard) >>> [6] <exec> $ (cd '/home/emir/devtools/libphutil/'; git ls-files -m) <<< [5] <exec> 73,004 us <<< [6] <exec> 74,084 us <<< [4] <exec> 77,907 us <<< [3] <exec> 80,606 us >>> [7] <exec> $ (cd '/home/emir/devtools/libphutil/'; git log --first-parent --format=medium 'HEAD^'..HEAD) <<< [7] <exec> 16,390 us >>> [8] <conduit> differential.parsecommitmessage() <<< [8] <conduit> 106,631 us Linting... >>> [9] <exec> $ (cd '/home/emir/devtools/libphutil'; git rev-parse --show-cdup) <<< [9] <exec> 9,976 us >>> [10] <exec> $ (cd '/home/emir/devtools/libphutil/'; git merge-base 'HEAD^' HEAD) <<< [10] <exec> 13,472 us >>> [11] <exec> $ (cd '/home/emir/devtools/libphutil/'; git diff --no-ext-diff --no-textconv --raw '00645a0aec09edc7f0f1f573032991ae94faa01b' --) >>> [12] <exec> $ (cd '/home/emir/devtools/libphutil/'; git diff --no-ext-diff --no-textconv --raw HEAD --) >>> [13] <exec> $ (cd '/home/emir/devtools/libphutil/'; git ls-files --others --exclude-standard) >>> [14] <exec> $ (cd '/home/emir/devtools/libphutil/'; git ls-files -m) <<< [11] <exec> 19,092 us <<< [14] <exec> 15,219 us <<< [12] <exec> 21,602 us <<< [13] <exec> 43,139 us >>> [15] <exec> $ (cd '/home/emir/devtools/libphutil/'; git diff --no-ext-diff --no-textconv -M -C --no-color --src-prefix=a/ --dst-prefix=b/ -U32767 '00645a0aec09edc7f0f1f573032991ae94faa01b' --) <<< [15] <exec> 28,318 us >>> [16] <exec> $ '/home/engshare/devtools/libphutil/src/parser/xhpast/bin/xhpast' --version <<< [16] <exec> 11,420 us >>> [17] <exec> $ '/home/engshare/devtools/arcanist/scripts/phutil_analyzer.php' '/home/emir/devtools/libphutil/src/markup/engine/remarkup/markuprule/hyperlink' <<< [17] <exec> 490,196 us >>> [18] <exec> $ '/home/engshare/devtools/arcanist/scripts/phutil_analyzer.php' '/home/engshare/devtools/libphutil/src/markup' >>> [19] <exec> $ '/home/engshare/devtools/arcanist/scripts/phutil_analyzer.php' '/home/engshare/devtools/libphutil/src/markup/engine/remarkup/markuprule/base' >>> [20] <exec> $ '/home/engshare/devtools/arcanist/scripts/phutil_analyzer.php' '/home/engshare/devtools/libphutil/src/parser/uri' >>> [21] <exec> $ '/home/engshare/devtools/arcanist/scripts/phutil_analyzer.php' '/home/engshare/devtools/libphutil/src/utils' <<< [18] <exec> 498,899 us <<< [19] <exec> 497,710 us <<< [20] <exec> 517,740 us <<< [21] <exec> 556,267 us >>> [22] <exec> $ '/home/engshare/devtools/libphutil/src/parser/xhpast/bin/xhpast' <<< [22] <exec> 10,066 us LINT OKAY No lint problems. Running unit tests... HipHop Fatal error: Uncaught exception exception 'PhutilBootloaderException' with message 'The phutil library '' has not been loaded!' in /home/engshare/devtools/libphutil/src/__phutil_library_init__.php:124\nStack trace:\n#0 /home/engshare/devtools/libphutil/src/__phutil_library_init__.php(177): PhutilBootloader->getLibraryRoot()\n#1 /home/engshare/devtools/arcanist/src/unit/engine/phutil/PhutilUnitTestEngine.php(53): PhutilBootloader->moduleExists()\n#2 /home/engshare/devtools/arcanist/src/workflow/unit/ArcanistUnitWorkflow.php(113): PhutilUnitTestEngine->run()\n#3 /home/engshare/devtools/arcanist/src/workflow/diff/ArcanistDiffWorkflow.php(1172): ArcanistUnitWorkflow->run()\n#4 /home/engshare/devtools/arcanist/src/workflow/diff/ArcanistDiffWorkflow.php(225): ArcanistDiffWorkflow->runUnit()\n#5 /home/engshare/devtools/arcanist/scripts/arcanist.php(257): ArcanistDiffWorkflow->run()\n#6 {main}

Okay — so this isn't exactly source code. It's just some command line output. But it does tell us some interesting information.

  • The person who, likely, posted this was "emir". This may be the person's first name, or it could be their first initial and then their surname (E. Mir). It's clear this output was intended to be seen by another engineer at Facebook, so posting it on Pastebin probably wasn't the smartest move. This person may have made other slip ups which could make them a target if an attacker sees an opportunity.

  • "dev3003" is the name of the machine emir was working on at the time. This tells us Facebook has at least 3,000 machines reversed for development (assuming "3003" increments from 1, which I'm quite sure it does).

  • `/home/engshare/devtools/` is the path where libphutil and arcanist are installed. `/home/engshare/` is shared between the development machines via NFS if I remember correctly. Nothing overly interesting here, but there are likely other internal scripts located in that directory.

  • There's also some information about execution times and Git hashes which could be of use but nothing I'd personally look in to.

After this find, I went ahead and tried to similiar pastes which had to been made. I was not disappointed.

[25/10/2013] Promoting The Meme Bank (1/1) - Campaign Update Failed: Campaign 6009258279237: Value cannot be null (Value given: null) TAAL[BLAME_files,www/flib/core/utils/enforce.php,www/flib/core/utils/EnforceBase.php]

Now, this looks to be an exception which was caught and logged. What's interesting here is it shows us file names and paths. "flib" (Facebook Library) is an internal library which contains useful utilities and functions to help with the development. Let's go deeper..

[ksalas@dev578 ~/www] ./scripts/intl/intl_string.php scan . Loading modules, hang on... Analyzing directory `.' Error: Command `ulimit -s 65536 && /mnt/vol/engshare/tools/fbt_extractor -tasks 32 '/data/users/ksalas/www-hg'' failed with error #2: stdout: stderr: warning: parsing problem in /data/users/ksalas/www-hg/flib/intern/third-party/phpunit/phpunit/Tests/TextUI/dataprovider-log-xml-isolation.phpt warning: parsing problem in /data/users/ksalas/www-hg/flib/intern/third-party/phpunit/phpunit/Tests/TextUI/dataprovider-log-xml.phpt warning: parsing problem in /data/users/ksalas/www-hg/flib/intern/third-party/phpunit/phpunit/Tests/TextUI/log-xml.phpt warning: parsing problem in /data/users/ksalas/www-hg/scripts/sandcastle/local_testing/script_for_test_commits.php warning: parsing problem in /data/users/ksalas/www-hg/lib/arcanist/lint/linter/__tests__/hphpast/php-tags-script.lint-test LEXER: unrecognised symbol, in token rule:' warning: parsing problem in /data/users/ksalas/www-hg/scripts/intern/test/test.php warning: parsing problem in /data/users/ksalas/www-hg/scripts/intern/test/test2.php Fatal error: exception Common.Todo Fatal error: exception Sys_error("Broken pipe") Type intl_string.php --help to get more information about how to use this script.

Now we're getting to the good stuff. We have ksalas on dev578 running what seems to be a string parser. `intl_string.php` tries to run `/mnt/vol/engshare/tools/fbt_extractor`, so we know for sure there are some other scripts in `/mnt/vol/engshare/`. We can also see they use PHP Unit for unit testing, and "www-hg" shouts Mercurial to me. It's well known they moved from Subversion to Git -- I'd put money on it that they've been expiermenting with Mercurial too at some point.

"That's still not god damn source code!" I hear you cry. Don't worry, someone posted some on Pastebin too.

Index: flib/core/db/queryf.php =================================================================== --- flib/core/db/queryf.php +++ flib/core/db/queryf.php @@ -1104,11 +1104,12 @@ * @author rmcelroy */ function mysql_query_all($sql, $ok_sql, $conn, $params) { + FBTraceDB::rqsend($ok_sql); switch (SQLQueryType::parse($sql)) { case SQLQueryType::READ: $t_start = microtime(true); $result = mysql_query_read($ok_sql, $conn); $t_end = microtime(true); $t_delta = $t_end - $t_start; if ($t_delta > ProfilingThresholds::$queryReadDuration) { ProfilingThresholds::recordDurationError('mysql.queryReadDuration',

The file in question is `flib/core/db/queryf.php`. At first glance we can tell it's a diff of a file which contains a bunch of MySQL-related functions. The function we can see here, `mysql_query_all()`, was written by rmcelroy. From what I can see in the code it's pretty much a simple function which executes a query, with a little custom logging code. It may be more complex but unfortunately we may never know. :(

I'll post a few more example of code I've found, all of which (and more) can be downloaded from the bottom of this post.

diff --git a/flib/entity/user/personal/EntPersonalUser.php b/flib/entity/user/personal/EntPersonalUser.php index 4de7ad8..439c162 100644 --- a/flib/entity/user/personal/EntPersonalUser.php +++ b/flib/entity/user/personal/EntPersonalUser.php @@ -306,13 +306,15 @@ class EntPersonalUser extends EntProfile public function prepareFriendIDs() { require_module_lazy('friends'); - // TODO: add privacy checks! DT('ReciprocalFriends')->add($this->id); return null; } public function getFriendIDs() { - return DT('ReciprocalFriends')->get($this->id); + if ($this->canSeeFriends()) { + return DT('ReciprocalFriends')->get($this->id); + } + return array(); } /** @@ -397,6 +399,7 @@ class EntPersonalUser extends EntProfile $this->viewerCanSee, array( PrivacyConcepts::EXISTENCE, + PrivacyConcepts::FRIENDS, // Note that we're fetching GENDER here because it's PAI // so it's cheap and because we don't want to add a prepareGender // call here if we don't have to. @@ -418,6 +421,10 @@ class EntPersonalUser extends EntProfile return must_prepare($this->viewerCanSee)->canSee(); } + protected function canSeeFriends() { + return must_prepare($this->viewerCanSee)->canSeeFriends(); + } + # update your local master branch git checkout master git pull --rebase # never do any work on master branch # create & switch to new branch instead git checkout -b my_branch # rebase 'my_branch' onto master git checkout my_branch git rebase master # list branches git branch # delete 'my_branch' branch $ git branch -d my_branch # shows status $ git status stage file, also remove conflict $ git add <file> revert file to head revision $ git checkout -- <file> commit change $ git commit -a --amend -a stages all modified files --amend overwrites last commit show all local history (amend commits, branch changes, etc.) $ git reflog show history (there is lot of options) $ git log $ git log --pretty=oneline --abbrev-commit --author=plamenko $ git log -S"text to search" show last commit (what is about to be send for diff) $ git show get the version of the file from the given commit $ git checkout <commit> path/to/file fetch & merge $ git pull --rebase resolving conflicts: use ours: $ git checkout --ours index.html use theirs: $ git checkout --theirs index.html commit author: $ git config --global user.name "Ognjen Dragoljevic" $ git config --global user.email [email protected] After doing this, you may fix the identity used for this commit with: $ git commit --amend --reset-author commit template: /mnt/vol/engshare/admin/scripts/templates/git-commit-template.txt rename a branch: $ git branch -m old_branch new_branch interactive rebase $ git rebase -i master pick edit make changes ... $ git commit -a --amend $ git rebase --continue exec $ arc diff $ arc amend $ git push --dry-run origin HEAD:master // remove dry-run to do actual push ... to update commit message in phabricator $ arc diff --verbatim #!/bin/bash # # Creates a new www sandbox managed by git. # # Usage: git-clone-www [dirname] # # dirname defaults to "www-git". # DIRNAME=${1:-www-git} NFS_REPO=/home/engshare/git/tfb # Are we running on a machine that has a local shared copy of the git repo? if [ -d /data/git/tfb ]; then # Yes. Reuse its objects directory. echo "Cloning the local host's shared www repository..." PARENT=/data/git/tfb SHARE=-s else # Nope, copy the NFS server's objects locally so as not to be dog slow. echo "Copying from the shared www repository on the NFS server..." PARENT=$NFS_REPO SHARE= fi if [ ! -d $HOME/local ]; then echo "You don't seem to have a 'local' symlink in your home directory." echo "Fix that and try again." exit 1 fi cd $HOME/local if [ -d "$DIRNAME" ]; then echo "You already have a $DIRNAME directory; won't overwrite it." echo "Aborting." exit 1 fi # We clone the shared repository here rather than running "git svn clone" # because it's much, much more efficient. And the clone has some options: # # -n = Don't check out working copy yet. # -s = Reference the origin's .git/objects directory rather than copying. # Saves gobs of disk space and makes the clone nearly instantaneous. # We don't do this if there's no local-disk shared repo. git clone $SHARE -n "$PARENT" "$DIRNAME" cd "$DIRNAME" # If we're sharing a local repository's objects, use the NFS server as a # fallback so stuff doesn't break if we use this repo from another host # that doesn't have a /data/git/tfb directory. ALTERNATES=.git/objects/info/alternates if [ -s $ALTERNATES ]; then echo $NFS_REPO/.git/objects >> $ALTERNATES fi # We want to use the same remote branch name ("remotes/trunk") for git-svn # and for fetches from the shared git repo, so set that up explicitly. git config remote.origin.url "file://$PARENT/.git" git config remote.origin.fetch refs/remotes/trunk:refs/remotes/trunk git config --remove-section branch.master # Enable the standard commit template git config commit.template /home/engshare/admin/scripts/templates/git-commit-template.txt # Enable recording of rebase conflict resolutions git config rerere.enabled true # Now fetch from the shared repo. This mostly just creates the new "trunk" # branch since we already have the objects thanks to the initial "git clone". git fetch origin # Blow away the "origin/" branches created by "git clone" -- we don't need them. rm -rf .git/refs/remotes/origin # Now it's time to turn this plain old git repo into a git-svn repo. Really # all we need is the svn-remote configuration (installed above) and a # metadata file with some version information. git-svn is smart enough to # rebuild the other stuff it needs. echo "" echo "Synchronizing with svn..." git svn init -itrunk svn+ssh://tubbs/svnroot/tfb/trunk/www # Now tweak the git-svn config a little bit so it's easier for someone to # go add more "fetch" lines if they want to track svn-side branches in # addition to trunk. This doesn't affect any of the existing history. git config svn-remote.svn.url svn+ssh://tubbs/svnroot git config svn-remote.svn.fetch tfb/trunk/www:refs/remotes/trunk # Let git-svn update its mappings and fetch the latest revisions. This can # spew lots of uninteresting output so suppress it. git svn fetch > /dev/null echo "" echo "Checking out working copy..." # We use git reset here because the git svn fetch might have advanced trunk # to a newer revision than the master branch created by git clone. git reset --hard trunk if [ ! -d "$HOME/$DIRNAME" ]; then echo "" echo "Making home dir symlink: $HOME/$DIRNAME" ln -s "local/$DIRNAME" "$HOME/$DIRNAME" else echo "" echo "$HOME/$DIRNAME already exists; leaving it alone." fi echo "" echo "All done. To make this your new main sandbox directory, run" echo "" echo " rm -rf ~/www" echo " ln -s ~/$DIRNAME ~/www" echo ""

Lastly, I wanted to share something which I found quite amusing. Facebook's MySQL password. This came from what seems to be a `print_r()` of an array which made its way in to production a few years ago.

array ( 'ip' => '', 'db_name' => 'insights', 'user' => 'mark', 'pass' => 'e5p0nd4', 'mode' => 'r', 'port' => 3306, 'cleanup' => false, 'num_retries' => 3, 'log_after_num_retries' => 4, 'reason' => 'insights', 'cdb' => true, 'flags' => 0, 'is_shadow' => false, 'backoff_retry' => false, ) Host: (Private IP) Database Name: insights User: mark Password: e5p0nd4

Okay, so it's not the most secure password. But Facebook's database servers are heavily firewalled. Though if you do manage to break in to Facebook's servers, there's the password.

Edit: Mark Zuckerberg was an officer at the Jewish fraternity Alpha Epsilon Pi. The motto on their coat of arms is "ESPONDA". :-)

So what have we learnt today? I think the main thing to take away from this is you shouldn't use public services such as Pastebin to post internal source code. Some creepy guy like me is going to collect it all and write about it. Another thing is to make sure debug information is never pushed to production. I didn't put much effort in to this but there will be more of Facebook's source code floating around out there.

Again I'd like to stress that everything I have posted here was already available on the Internet. All I needed to do was search for it. And here's the download:

URL: facebook_source_code.zip
Password: sintheticlabs.com

If you enjoyed this post and want to see more, follow @SintheticLabs on Twitter.

This is a static site. There is no server side logic. Please point your scanners elsewhere.

Home | Blog | Tools | Twitter | Github | Contact© 2014

Donations towards research is greatly appreciated:
BTC: 14W72Ktt7zgfoRhg48ftqdUw938y5vBLWr A look inside Facebook's source code - Sinthetic Labs

Sinthetic Labs | Blog

Welcome to our static blog. We use a thing called HTML. Crazy, right?

4th October, 2014 — Nathan Malcolm

Note: None of the code in this post was obtained illegally, nor given to me directly by any Facebook employee at any time.

I've always been a fan of Facebook from a technical point of view. They contribute a lot to the open source community and often open source their internal software too. Phabricator, libphutil, and XHP are great examples of that. For a while I contributed a bit to both Phabricator and XHP, and I ended up finding out a lot more about Facebook's internals than I intended. Read on...

It was mid-2013 and I was busy fixing a few bugs I had encountered while using Phabricator. If my memory serves me correctly the application was throwing a PhutilBootloaderException. I didn't have much knowledge of how Phabricator worked at the time so I googled the error message. As you'd expect I came across source code and references, but one specific link stood out. It was a Pastebin link.

Of course, this intrigued me. This is what I found...

[emir@dev3003 ~/devtools/libphutil] arc diff --trace >>> [0] <conduit> conduit.connect() <<< [0] <conduit> 98,172 us >>> [1] <exec> $ (cd '/home/emir/devtools/libphutil'; git rev-parse --show-cdup) <<< [1] <exec> 13,629 us >>> [2] <exec> $ (cd '/home/emir/devtools/libphutil/'; git rev-parse --verify HEAD^) <<< [2] <exec> 17,024 us >>> [3] <exec> $ (cd '/home/emir/devtools/libphutil/'; git diff --no-ext-diff --no-textconv --raw 'HEAD^' --) >>> [4] <exec> $ (cd '/home/emir/devtools/libphutil/'; git diff --no-ext-diff --no-textconv --raw HEAD --) >>> [5] <exec> $ (cd '/home/emir/devtools/libphutil/'; git ls-files --others --exclude-standard) >>> [6] <exec> $ (cd '/home/emir/devtools/libphutil/'; git ls-files -m) <<< [5] <exec> 73,004 us <<< [6] <exec> 74,084 us <<< [4] <exec> 77,907 us <<< [3] <exec> 80,606 us >>> [7] <exec> $ (cd '/home/emir/devtools/libphutil/'; git log --first-parent --format=medium 'HEAD^'..HEAD) <<< [7] <exec> 16,390 us >>> [8] <conduit> differential.parsecommitmessage() <<< [8] <conduit> 106,631 us Linting... >>> [9] <exec> $ (cd '/home/emir/devtools/libphutil'; git rev-parse --show-cdup) <<< [9] <exec> 9,976 us >>> [10] <exec> $ (cd '/home/emir/devtools/libphutil/'; git merge-base 'HEAD^' HEAD) <<< [10] <exec> 13,472 us >>> [11] <exec> $ (cd '/home/emir/devtools/libphutil/'; git diff --no-ext-diff --no-textconv --raw '00645a0aec09edc7f0f1f573032991ae94faa01b' --) >>> [12] <exec> $ (cd '/home/emir/devtools/libphutil/'; git diff --no-ext-diff --no-textconv --raw HEAD --) >>> [13] <exec> $ (cd '/home/emir/devtools/libphutil/'; git ls-files --others --exclude-standard) >>> [14] <exec> $ (cd '/home/emir/devtools/libphutil/'; git ls-files -m) <<< [11] <exec> 19,092 us <<< [14] <exec> 15,219 us <<< [12] <exec> 21,602 us <<< [13] <exec> 43,139 us >>> [15] <exec> $ (cd '/home/emir/devtools/libphutil/'; git diff --no-ext-diff --no-textconv -M -C --no-color --src-prefix=a/ --dst-prefix=b/ -U32767 '00645a0aec09edc7f0f1f573032991ae94faa01b' --) <<< [15] <exec> 28,318 us >>> [16] <exec> $ '/home/engshare/devtools/libphutil/src/parser/xhpast/bin/xhpast' --version <<< [16] <exec> 11,420 us >>> [17] <exec> $ '/home/engshare/devtools/arcanist/scripts/phutil_analyzer.php' '/home/emir/devtools/libphutil/src/markup/engine/remarkup/markuprule/hyperlink' <<< [17] <exec> 490,196 us >>> [18] <exec> $ '/home/engshare/devtools/arcanist/scripts/phutil_analyzer.php' '/home/engshare/devtools/libphutil/src/markup' >>> [19] <exec> $ '/home/engshare/devtools/arcanist/scripts/phutil_analyzer.php' '/home/engshare/devtools/libphutil/src/markup/engine/remarkup/markuprule/base' >>> [20] <exec> $ '/home/engshare/devtools/arcanist/scripts/phutil_analyzer.php' '/home/engshare/devtools/libphutil/src/parser/uri' >>> [21] <exec> $ '/home/engshare/devtools/arcanist/scripts/phutil_analyzer.php' '/home/engshare/devtools/libphutil/src/utils' <<< [18] <exec> 498,899 us <<< [19] <exec> 497,710 us <<< [20] <exec> 517,740 us <<< [21] <exec> 556,267 us >>> [22] <exec> $ '/home/engshare/devtools/libphutil/src/parser/xhpast/bin/xhpast' <<< [22] <exec> 10,066 us LINT OKAY No lint problems. Running unit tests... HipHop Fatal error: Uncaught exception exception 'PhutilBootloaderException' with message 'The phutil library '' has not been loaded!' in /home/engshare/devtools/libphutil/src/__phutil_library_init__.php:124\nStack trace:\n#0 /home/engshare/devtools/libphutil/src/__phutil_library_init__.php(177): PhutilBootloader->getLibraryRoot()\n#1 /home/engshare/devtools/arcanist/src/unit/engine/phutil/PhutilUnitTestEngine.php(53): PhutilBootloader->moduleExists()\n#2 /home/engshare/devtools/arcanist/src/workflow/unit/ArcanistUnitWorkflow.php(113): PhutilUnitTestEngine->run()\n#3 /home/engshare/devtools/arcanist/src/workflow/diff/ArcanistDiffWorkflow.php(1172): ArcanistUnitWorkflow->run()\n#4 /home/engshare/devtools/arcanist/src/workflow/diff/ArcanistDiffWorkflow.php(225): ArcanistDiffWorkflow->runUnit()\n#5 /home/engshare/devtools/arcanist/scripts/arcanist.php(257): ArcanistDiffWorkflow->run()\n#6 {main}

Okay — so this isn't exactly source code. It's just some command line output. But it does tell us some interesting information.

  • The person who, likely, posted this was "emir". This may be the person's first name, or it could be their first initial and then their surname (E. Mir). It's clear this output was intended to be seen by another engineer at Facebook, so posting it on Pastebin probably wasn't the smartest move. This person may have made other slip ups which could make them a target if an attacker sees an opportunity.

  • "dev3003" is the name of the machine emir was working on at the time. This tells us Facebook has at least 3,000 machines reversed for development (assuming "3003" increments from 1, which I'm quite sure it does).

  • `/home/engshare/devtools/` is the path where libphutil and arcanist are installed. `/home/engshare/` is shared between the development machines via NFS if I remember correctly. Nothing overly interesting here, but there are likely other internal scripts located in that directory.

  • There's also some information about execution times and Git hashes which could be of use but nothing I'd personally look in to.

After this find, I went ahead and tried to similiar pastes which had to been made. I was not disappointed.

[25/10/2013] Promoting The Meme Bank (1/1) - Campaign Update Failed: Campaign 6009258279237: Value cannot be null (Value given: null) TAAL[BLAME_files,www/flib/core/utils/enforce.php,www/flib/core/utils/EnforceBase.php]

Now, this looks to be an exception which was caught and logged. What's interesting here is it shows us file names and paths. "flib" (Facebook Library) is an internal library which contains useful utilities and functions to help with the development. Let's go deeper..

[ksalas@dev578 ~/www] ./scripts/intl/intl_string.php scan . Loading modules, hang on... Analyzing directory `.' Error: Command `ulimit -s 65536 && /mnt/vol/engshare/tools/fbt_extractor -tasks 32 '/data/users/ksalas/www-hg'' failed with error #2: stdout: stderr: warning: parsing problem in /data/users/ksalas/www-hg/flib/intern/third-party/phpunit/phpunit/Tests/TextUI/dataprovider-log-xml-isolation.phpt warning: parsing problem in /data/users/ksalas/www-hg/flib/intern/third-party/phpunit/phpunit/Tests/TextUI/dataprovider-log-xml.phpt warning: parsing problem in /data/users/ksalas/www-hg/flib/intern/third-party/phpunit/phpunit/Tests/TextUI/log-xml.phpt warning: parsing problem in /data/users/ksalas/www-hg/scripts/sandcastle/local_testing/script_for_test_commits.php warning: parsing problem in /data/users/ksalas/www-hg/lib/arcanist/lint/linter/__tests__/hphpast/php-tags-script.lint-test LEXER: unrecognised symbol, in token rule:' warning: parsing problem in /data/users/ksalas/www-hg/scripts/intern/test/test.php warning: parsing problem in /data/users/ksalas/www-hg/scripts/intern/test/test2.php Fatal error: exception Common.Todo Fatal error: exception Sys_error("Broken pipe") Type intl_string.php --help to get more information about how to use this script.

Now we're getting to the good stuff. We have ksalas on dev578 running what seems to be a string parser. `intl_string.php` tries to run `/mnt/vol/engshare/tools/fbt_extractor`, so we know for sure there are some other scripts in `/mnt/vol/engshare/`. We can also see they use PHP Unit for unit testing, and "www-hg" shouts Mercurial to me. It's well known they moved from Subversion to Git -- I'd put money on it that they've been expiermenting with Mercurial too at some point.

"That's still not god damn source code!" I hear you cry. Don't worry, someone posted some on Pastebin too.

Index: flib/core/db/queryf.php =================================================================== --- flib/core/db/queryf.php +++ flib/core/db/queryf.php @@ -1104,11 +1104,12 @@ * @author rmcelroy */ function mysql_query_all($sql, $ok_sql, $conn, $params) { + FBTraceDB::rqsend($ok_sql); switch (SQLQueryType::parse($sql)) { case SQLQueryType::READ: $t_start = microtime(true); $result = mysql_query_read($ok_sql, $conn); $t_end = microtime(true); $t_delta = $t_end - $t_start; if ($t_delta > ProfilingThresholds::$queryReadDuration) { ProfilingThresholds::recordDurationError('mysql.queryReadDuration',

The file in question is `flib/core/db/queryf.php`. At first glance we can tell it's a diff of a file which contains a bunch of MySQL-related functions. The function we can see here, `mysql_query_all()`, was written by rmcelroy. From what I can see in the code it's pretty much a simple function which executes a query, with a little custom logging code. It may be more complex but unfortunately we may never know. :(

I'll post a few more example of code I've found, all of which (and more) can be downloaded from the bottom of this post.

diff --git a/flib/entity/user/personal/EntPersonalUser.php b/flib/entity/user/personal/EntPersonalUser.php index 4de7ad8..439c162 100644 --- a/flib/entity/user/personal/EntPersonalUser.php +++ b/flib/entity/user/personal/EntPersonalUser.php @@ -306,13 +306,15 @@ class EntPersonalUser extends EntProfile public function prepareFriendIDs() { require_module_lazy('friends'); - // TODO: add privacy checks! DT('ReciprocalFriends')->add($this->id); return null; } public function getFriendIDs() { - return DT('ReciprocalFriends')->get($this->id); + if ($this->canSeeFriends()) { + return DT('ReciprocalFriends')->get($this->id); + } + return array(); } /** @@ -397,6 +399,7 @@ class EntPersonalUser extends EntProfile $this->viewerCanSee, array( PrivacyConcepts::EXISTENCE, + PrivacyConcepts::FRIENDS, // Note that we're fetching GENDER here because it's PAI // so it's cheap and because we don't want to add a prepareGender // call here if we don't have to. @@ -418,6 +421,10 @@ class EntPersonalUser extends EntProfile return must_prepare($this->viewerCanSee)->canSee(); } + protected function canSeeFriends() { + return must_prepare($this->viewerCanSee)->canSeeFriends(); + } + # update your local master branch git checkout master git pull --rebase # never do any work on master branch # create & switch to new branch instead git checkout -b my_branch # rebase 'my_branch' onto master git checkout my_branch git rebase master # list branches git branch # delete 'my_branch' branch $ git branch -d my_branch # shows status $ git status stage file, also remove conflict $ git add <file> revert file to head revision $ git checkout -- <file> commit change $ git commit -a --amend -a stages all modified files --amend overwrites last commit show all local history (amend commits, branch changes, etc.) $ git reflog show history (there is lot of options) $ git log $ git log --pretty=oneline --abbrev-commit --author=plamenko $ git log -S"text to search" show last commit (what is about to be send for diff) $ git show get the version of the file from the given commit $ git checkout <commit> path/to/file fetch & merge $ git pull --rebase resolving conflicts: use ours: $ git checkout --ours index.html use theirs: $ git checkout --theirs index.html commit author: $ git config --global user.name "Ognjen Dragoljevic" $ git config --global user.email [email protected] After doing this, you may fix the identity used for this commit with: $ git commit --amend --reset-author commit template: /mnt/vol/engshare/admin/scripts/templates/git-commit-template.txt rename a branch: $ git branch -m old_branch new_branch interactive rebase $ git rebase -i master pick edit make changes ... $ git commit -a --amend $ git rebase --continue exec $ arc diff $ arc amend $ git push --dry-run origin HEAD:master // remove dry-run to do actual push ... to update commit message in phabricator $ arc diff --verbatim #!/bin/bash # # Creates a new www sandbox managed by git. # # Usage: git-clone-www [dirname] # # dirname defaults to "www-git". # DIRNAME=${1:-www-git} NFS_REPO=/home/engshare/git/tfb # Are we running on a machine that has a local shared copy of the git repo? if [ -d /data/git/tfb ]; then # Yes. Reuse its objects directory. echo "Cloning the local host's shared www repository..." PARENT=/data/git/tfb SHARE=-s else # Nope, copy the NFS server's objects locally so as not to be dog slow. echo "Copying from the shared www repository on the NFS server..." PARENT=$NFS_REPO SHARE= fi if [ ! -d $HOME/local ]; then echo "You don't seem to have a 'local' symlink in your home directory." echo "Fix that and try again." exit 1 fi cd $HOME/local if [ -d "$DIRNAME" ]; then echo "You already have a $DIRNAME directory; won't overwrite it." echo "Aborting." exit 1 fi # We clone the shared repository here rather than running "git svn clone" # because it's much, much more efficient. And the clone has some options: # # -n = Don't check out working copy yet. # -s = Reference the origin's .git/objects directory rather than copying. # Saves gobs of disk space and makes the clone nearly instantaneous. # We don't do this if there's no local-disk shared repo. git clone $SHARE -n "$PARENT" "$DIRNAME" cd "$DIRNAME" # If we're sharing a local repository's objects, use the NFS server as a # fallback so stuff doesn't break if we use this repo from another host # that doesn't have a /data/git/tfb directory. ALTERNATES=.git/objects/info/alternates if [ -s $ALTERNATES ]; then echo $NFS_REPO/.git/objects >> $ALTERNATES fi # We want to use the same remote branch name ("remotes/trunk") for git-svn # and for fetches from the shared git repo, so set that up explicitly. git config remote.origin.url "file://$PARENT/.git" git config remote.origin.fetch refs/remotes/trunk:refs/remotes/trunk git config --remove-section branch.master # Enable the standard commit template git config commit.template /home/engshare/admin/scripts/templates/git-commit-template.txt # Enable recording of rebase conflict resolutions git config rerere.enabled true # Now fetch from the shared repo. This mostly just creates the new "trunk" # branch since we already have the objects thanks to the initial "git clone". git fetch origin # Blow away the "origin/" branches created by "git clone" -- we don't need them. rm -rf .git/refs/remotes/origin # Now it's time to turn this plain old git repo into a git-svn repo. Really # all we need is the svn-remote configuration (installed above) and a # metadata file with some version information. git-svn is smart enough to # rebuild the other stuff it needs. echo "" echo "Synchronizing with svn..." git svn init -itrunk svn+ssh://tubbs/svnroot/tfb/trunk/www # Now tweak the git-svn config a little bit so it's easier for someone to # go add more "fetch" lines if they want to track svn-side branches in # addition to trunk. This doesn't affect any of the existing history. git config svn-remote.svn.url svn+ssh://tubbs/svnroot git config svn-remote.svn.fetch tfb/trunk/www:refs/remotes/trunk # Let git-svn update its mappings and fetch the latest revisions. This can # spew lots of uninteresting output so suppress it. git svn fetch > /dev/null echo "" echo "Checking out working copy..." # We use git reset here because the git svn fetch might have advanced trunk # to a newer revision than the master branch created by git clone. git reset --hard trunk if [ ! -d "$HOME/$DIRNAME" ]; then echo "" echo "Making home dir symlink: $HOME/$DIRNAME" ln -s "local/$DIRNAME" "$HOME/$DIRNAME" else echo "" echo "$HOME/$DIRNAME already exists; leaving it alone." fi echo "" echo "All done. To make this your new main sandbox directory, run" echo "" echo " rm -rf ~/www" echo " ln -s ~/$DIRNAME ~/www" echo ""

Lastly, I wanted to share something which I found quite amusing. Facebook's MySQL password. This came from what seems to be a `print_r()` of an array which made its way in to production a few years ago.

array ( 'ip' => '', 'db_name' => 'insights', 'user' => 'mark', 'pass' => 'e5p0nd4', 'mode' => 'r', 'port' => 3306, 'cleanup' => false, 'num_retries' => 3, 'log_after_num_retries' => 4, 'reason' => 'insights', 'cdb' => true, 'flags' => 0, 'is_shadow' => false, 'backoff_retry' => false, ) Host: (Private IP) Database Name: insights User: mark Password: e5p0nd4

Okay, so it's not the most secure password. But Facebook's database servers are heavily firewalled. Though if you do manage to break in to Facebook's servers, there's the password.

Edit: Mark Zuckerberg was an officer at the Jewish fraternity Alpha Epsilon Pi. The motto on their coat of arms is "ESPONDA". :-)

So what have we learnt today? I think the main thing to take away from this is you shouldn't use public services such as Pastebin to post internal source code. Some creepy guy like me is going to collect it all and write about it. Another thing is to make sure debug information is never pushed to production. I didn't put much effort in to this but there will be more of Facebook's source code floating around out there.

Again I'd like to stress that everything I have posted here was already available on the Internet. All I needed to do was search for it. And here's the download:

URL: facebook_source_code.zip
Password: sintheticlabs.com

If you enjoyed this post and want to see more, follow @SintheticLabs on Twitter.

This is a static site. There is no server side logic. Please point your scanners elsewhere.

Home | Blog | Tools | Twitter | Github | Contact© 2014

Donations towards research is greatly appreciated:



How Disney Was Hustled into Making the Trippiest Movie About Computers Ever

15 hours 25 min ago

When you think about the early 80s, a few things probably come to mind: gauzy impressions of synthesizers, Hollywood blockbusters, and computers, all filtered through the saturated lens of degraded VHS tape. Computers are People, Too!, a Disney documentary about computer art released in 1982, might just be the defining artifact of this period. At the very least, it’s certainly the strangest.

Computers are People, Too! was initially released as a TV movie in the months leading up to the theatrical release of Tron, Hollywood’s first major foray into computer animation. Part compendium of early computer art from analog pioneers like Lee Harrison and John Whitney, part treatise on the immense promise of digital technology, and part Hollywood marketing stunt, the documentary captures the enthusiasm of an age gone by in an ecstatic kaleidoscope of vintage graphics and buckets of camp.

Everyone was experimenting. It was a celebration. Computers were part of that

The set-up revolves around actress Elaine Joyce, who asks if she’s going to be replaced by machines. An artificially intelligent supercomputer from the future proceeds to demonstrate why she’s wrong with a hallucinogenic display of computer art from the past, as well as the cutting edge of 1982. The computer intones, in the voice of Hollywood veteran Joseph Campanella, “We are on the verge of a beautiful partnership.”

Now forgotten to all but a few die-hard fans, Computers are People, Too! has been given a second life online. The documentary has been uploaded to YouTube, and has garnered over two thousand views. A website with the URL computersarepeopletoo.com plays the movie on a loop.

To find out more about how the hell such a bizarre document was made, I called up Mike Bonifer, who co-wrote and produced the documentary while working as a publicist for Tron in the early 80s. He now runs a business consulting firm based in Los Angeles.

As Bonifer tells it, the story of Computers are People, Too! is one of unbridled enthusiasm in an age of technological promise, crushing disillusionment, and the carnival of youth.

Motherboard: Tell me a little bit about the history of Computers are People, Too!—why did Disney want to produce it?
Michael Bonifer:
There was a wave of interest at Disney in computers that was spurred by Tron. Once Tron hit the lot, it was this computer hysteria that was going on. All the young people at the studio at the time were super hungry to find out about computers. 

What was interesting about that time at Disney was that the senior management was occupied with EPCOT. There was this sense of young people running amok in a creative, good way. You got to do things that you wouldn’t normally have gotten to do if senior management had been back in Burbank, minding every little thing that happened. Everyone was experimenting. It was a celebration. Computers were part of that.

He was on the wrong path. He was analog

You were the publicist for Tron, and Computers are People, Too! was somewhat of a tie-in, right?
I was the publicist for Tron, so I bit off as much as I could chew off that project. I wrote the book The Art of Tron, and I got the green light to make a TV show about computers that would have a significant chunk of Tron in it. That was the justification: marketing for Tron. I was just playing along. I had the opportunity, like a lot of young people did, to get a budget for this sort of thing. And once you got a budget, you could play; nobody was watching out for what you did. You could do what you wanted. That’s how we made Computers are People Too!

It was playful. Everybody wanted to play with computers. Everybody was looking at each other’s machines. it must’ve been like what cars were like when everyone started getting cars. Checking them out, rigging them, and fixing them. You could talk to people in accounting and they’d be talking about the modems that were going into the New York office. It was company-wide, but it was also worldwide at that point. Tron was the lightning rod. It was a beautiful game. And Disney, at the time, was at the epicenter of it.

Elaine Joyce, the show’s host, tries to show up her computer counterpart by dancing like a total maniac.

Why the title? Declaring that computers are people seems like a bold move. 
Jim Fanning gave it its name. He was an intern. There was a show called Kids are People, Too!on at the time, and Jim came into my office and said, “Mike, I have the name of the movie. Computers are People, Too!” And I said, yeah, that sounds good.

We never looked beyond that or had to think about it. Something intuitively told me that Jim was right. It was how our humanity is reflected in computers that would be most interesting, and it was aligned with the theme of Tron, and it was something we could explore in the TV show, too.

How do you think humanity is reflected in computers?
It’s all John Henry versus the Inky-Poo when you talk about technology. I associate more with John Henry than with the Inky-Poo. We were looking into the machine for our humanity, and now—progress lurches, it’s not a smooth curve—this wave has washed up on top of us now, and we’re trying to maintain our humanity in the midst of it all. That’s the fresh challenge.

All the young people at the studio at the time were super hungry to find out about computers

The next wave is going to be how we make our peace with it. At the beginning, it was all celebration and light—bringing things to light. Now, it’s keeping things light and not having the darkness overwhelm us. The darkness of the surveillance state.

Gideon Ariel used primitive motion capture technology to digitize the movements of athletes (pictured: a shot putter) and analyze them.

So the documentary came from a place of immense enthusiasm, and now you think we’ve had a few decades to see the darker side of computers?
Right. Let’s look at Gideon Ariel; he’s got these wireframe figures of dancers and athletes, and it was wonderful. There was no downside to it. Now, you look at the same thing and think, wow, every baseball pitcher is going to pitch exactly the same way.

The challenge is, can any athlete look idiosyncratic at this point? Because they’re all inside the mind of the machine now, and the machine is telling them what to do and making the adjustments, and the machine is mathematically perfect. Is that the only option we have? Mathematical perfection? This, to me, is the flip side of where we were when we made Computers and People, Too!

What was the biggest challenge in producing the documentary?
You had to find where all the computer graphics were in the world. Computers are People, Too! is probably one of the best compilations of CG; better than Siggraph at that stage. I had to go to Denver to see Lee Harrison, who was one of the analog pioneers. Harrison literally had computers that were built of blocks of wood and nails and copper wires, and he’d made this Mr. Noise thing.

Lee Harrison’s Mr. Noise used analog circuits to produce sound-activated graphics.

He was so emotional, and I didn’t know what to say to him. He took me out for drinks, and he really thought this was his moment. Disney had come calling. And I was just a frickin’ publicist who had swung this gig to produce a TV show. He started crying, and I started crying. It was so emotional, and I felt so deeply for him. I loved everything about him, but I wasn’t the guy from Disney that was going to elevate him to the pantheon. He was on the wrong path. He was analog. And I knew, even at that time, that everything was going to be digital.

What was once seen as playful is now seen as pervasive and threatening

And then there was Art Swerdloff, our editor. His mentor was Slavko Vorkapić, who basically invented the montage. We were editing this show, and Art would stop for two hours and talk about the history of an edit, and Slavko Vorkapić, and the birth of montage editing, and how to do the Crimean war in five minutes. It was the least efficient way you could ever make a movie. We bought our own equipment because there was no editing bay in town with the newest equipment, so we were all learning how to edit, too. We were all like kids together.

Art was a wonderful man, and he became a friend for the rest of his life. If we were ever to re-release Computers are People, Too!, I would make sure it was dedicated to Art Swerdloff, because he was a really cool guy.

An early demo of a computer-generated character, referred to in the documentary as “mathematical plastic surgery.”

What’s your favourite segment in the documentary?
My favourite segment is the compilation of the CG of the era. People were telling me that people were playing that sequence in gay clubs in LA. Almost before the show was out, I was hearing, ‘Yeah, I saw that thing from your show. It was in this club I was at last night!’

Really? That’s wild.
Yeah. That’s when I thought, okay, there’s something to this.

Michael Iceberg, a Disney performer who played frenetic synth compositions from inside a giant pyramid, is featured prominently in the documentary. Without context, it’s completely bizarre. Why?
We all got swept up in the Iceberg thing. We were all fascinated. He travelled around in a bus, he had a really fun crew, he had a super attractive wife, and he’d come up out of this pyramid. There was a whole mini Iceberg frenzy during production when we saw this guy.

It was a lot to take in. Art got swept away and was obsessed with the guy. Even though he was terrible with music, he was convinced that Iceberg was somehow a totem for the show. I don’t think it was filler, I just think we got carried away with this carnival quality of the thing. We were like kids at a carnival, and it was our carnival music.

Is that the only option we have? Mathematical perfection?

How do you think attitudes about technology have changed since 1982?
We were like Mickey at the beginning of The Sorcerer’s Apprentice—we had some magic. That was the Tron era. Now, we’re swimming in it. The pages of the magic are being swept away, and the magic is out of control. it’s really going to take a new kind of ethos almost; a new kind of understanding with how we engage with each other.

Look at the Sony hack. At Disney, when computers were first coming in, we were saying, "Yes! More computers!" I’m sure Sony people are saying, "I will not put anything on email, I’m going to do everything on parchment. Give me some quills!" That’s the real difference: Disney then and Sony today.

What was once seen as play is now seen as pervasive and threatening. We have to find out what the new line is. It can’t just run everywhere like water, like it did in the beginning. But I’m not some prophet of gloom. It’s just a different kind of excitement, and we need to be attentive to what we’re doing now like we did back then. There’s still plenty to celebrate, it’s just a different kind of struggle.

Michael Iceberg plays synthesizers inside a giant pyramid. The former Disney performer fascinated the documentary’s producers.

Bitcoin Developer Guide

16 December 2014 - 2:00pm
BETA: This documentation has not been extensively reviewed by Bitcoin experts and so likely contains numerous errors. Please use the Issue and Edit links on the bottom left menu to help us improve. Click here to close this disclaimer. X

The Developer Guide aims to provide the information you need to understand Bitcoin and start building Bitcoin-based applications. To make the best use of this documentation, you may want to install the current version of Bitcoin Core, either from source or from a pre-compiled executable.

Questions about Bitcoin development are best asked in one of the Bitcoin development communities. Errors or suggestions related to documentation on Bitcoin.org can be submitted as an issue or posted to the bitcoin-documentation mailing list.

In the following documentation, some strings have been shortened or wrapped: “[…]” indicates extra data was removed, and lines ending in a single backslash “\” are continued below. If you hover your mouse over a paragraph, cross-reference links will be shown in blue. If you hover over a cross-reference link, a brief definition of the term will be displayed in a tooltip.

Block Chain Edit | History | Report Issue | Discuss

The block chain provides Bitcoin’s public ledger, an ordered and timestamped record of transactions. This system is used to protect against double spending and modification of previous transaction records.

Each full node in the Bitcoin network independently stores a block chain containing only blocks validated by that node. When several nodes all have the same blocks in their block chain, they are considered to be in consensus. The validation rules these nodes follow to maintain consensus are called consensus rules. This section describes many of the consensus rules used by Bitcoin Core.

Block Chain Overview Edit | History | Report Issue | Discuss

The illustration above shows a simplified version of a block chain. A block of one or more new transactions is collected into the transaction data part of a block. Copies of each transaction are hashed, and the hashes are then paired, hashed, paired again, and hashed again until a single hash remains, the merkle root of a merkle tree.

The merkle root is stored in the block header. Each block also stores the hash of the previous block’s header, chaining the blocks together. This ensures a transaction cannot be modified without modifying the block that records it and all following blocks.

Transactions are also chained together. Bitcoin wallet software gives the impression that satoshis are sent from and to wallets, but bitcoins really move from transaction to transaction. Each transaction spends the satoshis previously received in one or more earlier transactions, so the input of one transaction is the output of a previous transaction.

A single transaction can create multiple outputs, as would be the case when sending to multiple addresses, but each output of a particular transaction can only be used as an input once in the block chain. Any subsequent reference is a forbidden double spend—an attempt to spend the same satoshis twice.

Outputs are tied to transaction identifiers (TXIDs), which are the hashes of signed transactions.

Because each output of a particular transaction can only be spent once, the outputs of all transactions included in the block chain can be categorized as either Unspent Transaction Outputs (UTXOs) or spent transaction outputs. For a payment to be valid, it must only use UTXOs as inputs.

Satoshis cannot be left in a UTXO after a transaction or they will be irretrievably lost, so any difference between the number of satoshis in a transaction’s inputs and outputs is given as a transaction fee to the Bitcoin miner who creates the block containing that transaction. For example, in the illustration above, each transaction spends 10,000 satoshis fewer than it receives from its combined inputs, effectively paying a 10,000 satoshi transaction fee.

Proof Of Work Edit | History | Report Issue | Discuss

The block chain is collaboratively maintained by anonymous peers on the network, so Bitcoin requires that each block prove a significant amount of work was invested in its creation to ensure that untrustworthy peers who want to modify past blocks have to work harder than honest peers who only want to add new blocks to the block chain.

Chaining blocks together makes it impossible to modify transactions included in any block without modifying all following blocks. As a result, the cost to modify a particular block increases with every new block added to the block chain, magnifying the effect of the proof of work.

The proof of work used in Bitcoin takes advantage of the apparently random nature of cryptographic hashes. A good cryptographic hash algorithm converts arbitrary data into a seemingly-random number. If the data is modified in any way and the hash re-run, a new seemingly-random number is produced, so there is no way to modify the data to make the hash number predictable.

To prove you did some extra work to create a block, you must create a hash of the block header which does not exceed a certain value. For example, if the maximum possible hash value is 2256 − 1, you can prove that you tried up to two combinations by producing a hash value less than 2255.

In the example given above, you will produce a successful hash on average every other try. You can even estimate the probability that a given hash attempt will generate a number below the target threshold. Bitcoin assumes a linear probability that the lower it makes the target threshold, the more hash attempts (on average) will need to be tried.

New blocks will only be added to the block chain if their hash is at least as challenging as a difficulty value expected by the consensus protocol. Every 2,016 blocks, the network uses timestamps stored in each block header to calculate the number of seconds elapsed between generation of the first and last of those last 2,016 blocks. The ideal value is 1,209,600 seconds (two weeks).

  • If it took fewer than two weeks to generate the 2,016 blocks, the expected difficulty value is increased proportionally (by as much as 300%) so that the next 2,016 blocks should take exactly two weeks to generate if hashes are checked at the same rate.

  • If it took more than two weeks to generate the blocks, the expected difficulty value is decreased proportionally (by as much as 75%) for the same reason.

(Note: an off-by-one error in the Bitcoin Core implementation causes the difficulty to be updated every 2,016 blocks using timestamps from only 2,015 blocks, creating a slight skew.)

Because each block header must hash to a value below the target threshold, and because each block is linked to the block that preceded it, it requires (on average) as much hashing power to propagate a modified block as the entire Bitcoin network expended between the time the original block was created and the present time. Only if you acquired a majority of the network’s hashing power could you reliably execute such a 51 percent attack against transaction history (although, it should be noted, that even less than 50% of the hashing power still has a good chance of performing such attacks).

The block header provides several easy-to-modify fields, such as a dedicated nonce field, so obtaining new hashes doesn’t require waiting for new transactions. Also, only the 80-byte block header is hashed for proof-of-work, so including a large volume of transaction data in a block does not slow down hashing with extra I/O, and adding additional transaction data only requires the recalculation of the ancestor hashes in the merkle tree.

Block Height And Forking Edit | History | Report Issue | Discuss

Any Bitcoin miner who successfully hashes a block header to a value below the target threshold can add the entire block to the block chain (assuming the block is otherwise valid). These blocks are commonly addressed by their block height—the number of blocks between them and the first Bitcoin block (block 0, most commonly known as the genesis block). For example, block 2016 is where difficulty could have first been adjusted.

Multiple blocks can all have the same block height, as is common when two or more miners each produce a block at roughly the same time. This creates an apparent fork in the block chain, as shown in the illustration above.

When miners produce simultaneous blocks at the end of the block chain, each node individually chooses which block to accept. In the absence of other considerations, discussed below, nodes usually use the first block they see.

Eventually a miner produces another block which attaches to only one of the competing simultaneously-mined blocks. This makes that side of the fork stronger than the other side. Assuming a fork only contains valid blocks, normal peers always follow the the most difficult chain to recreate and throw away stale blocks belonging to shorter forks. (Stale blocks are also sometimes called orphans or orphan blocks, but those terms are also used for blocks without a known parent block.)

Long-term forks are possible if different miners work at cross-purposes, such as some miners diligently working to extend the block chain at the same time other miners are attempting a 51 percent attack to revise transaction history.

Since multiple blocks can have the same height during a block chain fork, block height should not be used as a globally unique identifier. Instead, blocks are usually referenced by the hash of their header (often with the byte order reversed, and in hexadecimal).

Transaction Data Edit | History | Report Issue | Discuss

Every block must include one or more transactions. The first one of these transactions must be a coinbase transaction, also called a generation transaction, which should collect and spend the block reward (comprised of a block subsidy and any transaction fees paid by transactions included in this block).

The UTXO of a coinbase transaction has the special condition that it cannot be spent (used as an input) for at least 100 blocks. This temporarily prevents a miner from spending the transaction fees and block reward from a block that may later be determined to be stale (and therefore the coinbase transaction destroyed) after a block chain fork.

Blocks are not required to include any non-coinbase transactions, but miners almost always do include additional transactions in order to collect their transaction fees.

All transactions, including the coinbase transaction, are encoded into blocks in binary rawtransaction format.

The rawtransaction format is hashed to create the transaction identifier (txid). From these txids, the merkle tree is constructed by pairing each txid with one other txid and then hashing them together. If there are an odd number of txids, the txid without a partner is hashed with a copy of itself.

The resulting hashes themselves are each paired with one other hash and hashed together. Any hash without a partner is hashed with itself. The process repeats until only one hash remains, the merkle root.

For example, if transactions were merely joined (not hashed), a five-transaction merkle tree would look like the following text diagram:

ABCDEEEE .......Merkle root / \ ABCD EEEE / \ / AB CD EE .......E is paired with itself / \ / \ / A B C D E .........Transactions

As discussed in the Simplified Payment Verification (SPV) subsection, the merkle tree allows clients to verify for themselves that a transaction was included in a block by obtaining the merkle root from a block header and a list of the intermediate hashes from a full peer. The full peer does not need to be trusted: it is expensive to fake block headers and the intermediate hashes cannot be faked or the verification will fail.

For example, to verify transaction D was added to the block, an SPV client only needs a copy of the C, AB, and EEEE hashes in addition to the merkle root; the client doesn’t need to know anything about any of the other transactions. If the five transactions in this block were all at the maximum size, downloading the entire block would require over 500,000 bytes—but downloading three hashes plus the block header requires only 140 bytes.

Note: If identical txids are found within the same block, there is a possibility that the merkle tree may collide with a block with some or all duplicates removed due to how unbalanced merkle trees are implemented (duplicating the lone hash). Since it is impractical to have separate transactions with identical txids, this does not impose a burden on honest software, but must be checked if the invalid status of a block is to be cached; otherwise, a valid block with the duplicates eliminated could have the same merkle root and block hash, but be rejected by the cached invalid outcome, resulting in security bugs such as CVE-2012-2459.

Consensus Rule Changes Edit | History | Report Issue | Discuss

To maintain consensus, all full nodes validate blocks using the same consensus rules. However, sometimes the consensus rules are changed to introduce new features or prevent network abuse. When the new rules are implemented, there will likely be a period of time when non-upgraded nodes follow the old rules and upgraded nodes follow the new rules, creating two possible ways consensus can break:

  1. A block following the new consensus rules is accepted by upgraded nodes but rejected by non-upgraded nodes. For example, a new transaction feature is used within a block: upgraded nodes understand the feature and accept it, but non-upgraded nodes reject it because it violates the old rules.

  2. A block violating the new consensus rules is rejected by upgraded nodes but accepted by non-upgraded nodes. For example, an abusive transaction feature is used within a block: upgraded nodes reject it because it violates the new rules, but non-upgraded nodes accept it because it follows the old rules.

In the first case, rejection by non-upgraded nodes, mining software which gets block chain data from those non-upgraded nodes refuses to build on the same chain as mining software getting data from upgraded nodes. This creates permanently divergent chains—one for non-upgraded nodes and one for upgraded nodes—called a hard fork.

In the second case, rejection by upgraded nodes, it’s possible to keep the block chain from permanently diverging if upgraded nodes control a majority of the hash rate. That’s because, in this case, non-upgraded nodes will accept as valid all the same blocks as upgraded nodes, so the upgraded nodes can build a stronger chain that the non-upgraded nodes will accept as the best valid block chain. This is called a soft fork.

Although a fork is an actual divergence in block chains, changes to the consensus rules are often described by their potential to create either a hard or soft fork. For example, “increasing the block size above 1 MB requires a hard fork.” In this example, an actual block chain fork is not required—but it is a possible outcome.

Resources: BIP16, BIP30, and BIP34 were implemented as changes which might have lead to soft forks. BIP50 describes both an accidental hard fork, resolved by temporary downgrading the capabilities of upgraded nodes, and an intentional hard fork when the temporary downgrade was removed. A document from Gavin Andresen outlines how future rule changes may be implemented.

Detecting Forks Edit | History | Report Issue | Discuss

Non-upgraded nodes may use and distribute incorrect information during both types of forks, creating several situations which could lead to financial loss. In particular, non-upgraded nodes may relay and accept transactions that are considered invalid by upgraded nodes and so will never become part of the universally-recognized best block chain. Non-upgraded nodes may also refuse to relay blocks or transactions which have already been added to the best block chain, or soon will be, and so provide incomplete information.

Bitcoin Core includes code that detects a hard fork by looking at block chain proof of work. If a node receives block chain headers demonstrating six blocks more proof of work than the best chain this node considers valid, the node reports an error in the getinfo RPC results and runs the -alertnotify command if set.

Full nodes can also check block and transaction version numbers. If the block or transaction version numbers seen in several recent blocks are higher than the version numbers the node uses, it can assume it doesn’t use the current consensus rules. Future versions of Bitcoin Core (>0.9.3) will likely report this situation through the getinfo RPC and -alertnotify command if set.

In either case, data should not be relied upon if it comes from a node that apparently isn’t using the current consensus rules.

SPV clients which connect to full nodes can detect a likely hard fork by connecting to several full nodes and ensuring that they’re all on the same chain with the same block height, plus or minus several blocks to account for transmission delays and stale blocks. If there’s a divergence, the client can disconnect from nodes with weaker chains.

SPV clients should also monitor for block and transaction version number increases to ensure they process received transactions and create new transactions using the current consensus rules.

Transactions Edit | History | Report Issue | Discuss

Transactions let users spend satoshis. Each transaction is constructed out of several parts which enable both simple direct payments and complex transactions. This section will describe each part and demonstrate how to use them together to build complete transactions.

To keep things simple, this section pretends coinbase transactions do not exist. Coinbase transactions can only be created by Bitcoin miners and they’re an exception to many of the rules listed below. Instead of pointing out the coinbase exception to each rule, we invite you to read about coinbase transactions in the block chain section of this guide.

The figure above shows the main parts of a Bitcoin transaction. Each transaction has at least one input and one output. Each input spends the satoshis paid to a previous output. Each output then waits as an Unspent Transaction Output (UTXO) until a later input spends it. When your Bitcoin wallet tells you that you have a 10,000 satoshi balance, it really means that you have 10,000 satoshis waiting in one or more UTXOs.

Each transaction is prefixed by a four-byte transaction version number which tells Bitcoin peers and miners which set of rules to use to validate it. This lets developers create new rules for future transactions without invalidating previous transactions.

An output has an implied index number based on its location in the transaction—the first output is output zero. The output also has an amount in satoshis which it pays to a conditional pubkey script. Anyone who can satisfy the conditions of that pubkey script can spend up to the amount of satoshis paid to it.

An input uses a transaction identifier (txid) and an output index number (often called “vout” for output vector) to identify a particular output to be spent. It also has a signature script which allows it to provide data parameters that satisfy the conditionals in the pubkey script. (The sequence number and locktime are related and will be covered together in a later subsection.)

The figures below help illustrate how these features are used by showing the workflow Alice uses to send Bob a transaction and which Bob later uses to spend that transaction. Both Alice and Bob will use the most common form of the standard Pay-To-Public-Key-Hash (P2PKH) transaction type. P2PKH lets Alice spend satoshis to a typical Bitcoin address, and then lets Bob further spend those satoshis using a simple cryptographic key pair.

Bob must first generate a private/public key pair before Alice can create the first transaction. Bitcoin uses the Elliptic Curve Digital Signature Algorithm (ECDSA) with the secp256k1 curve; secp256k1 private keys are 256 bits of random data. A copy of that data is deterministically transformed into an secp256k1 public key. Because the transformation can be reliably repeated later, the public key does not need to be stored.

The public key (pubkey) is then cryptographically hashed. This pubkey hash can also be reliably repeated later, so it also does not need to be stored. The hash shortens and obfuscates the public key, making manual transcription easier and providing security against unanticipated problems which might allow reconstruction of private keys from public key data at some later point.

Bob provides the pubkey hash to Alice. Pubkey hashes are almost always sent encoded as Bitcoin addresses, which are base58-encoded strings containing an address version number, the hash, and an error-detection checksum to catch typos. The address can be transmitted through any medium, including one-way mediums which prevent the spender from communicating with the receiver, and it can be further encoded into another format, such as a QR code containing a bitcoin: URI.

Once Alice has the address and decodes it back into a standard hash, she can create the first transaction. She creates a standard P2PKH transaction output containing instructions which allow anyone to spend that output if they can prove they control the private key corresponding to Bob’s hashed public key. These instructions are called the pubkey script or scriptPubKey.

Alice broadcasts the transaction and it is added to the block chain. The network categorizes it as an Unspent Transaction Output (UTXO), and Bob’s wallet software displays it as a spendable balance.

When, some time later, Bob decides to spend the UTXO, he must create an input which references the transaction Alice created by its hash, called a Transaction Identifier (txid), and the specific output she used by its index number (output index). He must then create a signature script—a collection of data parameters which satisfy the conditions Alice placed in the previous output’s pubkey script. Signature scripts are also called scriptSigs.

Pubkey scripts and signature scripts combine secp256k1 pubkeys and signatures with conditional logic, creating a programable authorization mechanism.

For a P2PKH-style output, Bob’s signature script will contain the following two pieces of data:

  1. His full (unhashed) public key, so the pubkey script can check that it hashes to the same value as the pubkey hash provided by Alice.

  2. An secp256k1 signature made by using the ECDSA cryptographic formula to combine certain transaction data (described below) with Bob’s private key. This lets the pubkey script verify that Bob owns the private key which created the public key.

Bob’s secp256k1 signature doesn’t just prove Bob controls his private key; it also makes the non-signature-script parts of his transaction tamper-proof so Bob can safely broadcast them over the peer-to-peer network.

As illustrated in the figure above, the data Bob signs includes the txid and output index of the previous transaction, the previous output’s pubkey script, the pubkey script Bob creates which will let the next recipient spend this transaction’s output, and the amount of satoshis to spend to the next recipient. In essence, the entire transaction is signed except for any signature scripts, which hold the full public keys and secp256k1 signatures.

After putting his signature and public key in the signature script, Bob broadcasts the transaction to Bitcoin miners through the peer-to-peer network. Each peer and miner independently validates the transaction before broadcasting it further or attempting to include it in a new block of transactions.

P2PKH Script Validation Edit | History | Report Issue | Discuss

The validation procedure requires evaluation of the signature script and pubkey script. In a P2PKH output, the pubkey script is:


The spender’s signature script is evaluated and prefixed to the beginning of the script. In a P2PKH transaction, the signature script contains an secp256k1 signature (sig) and full public key (pubkey), creating the following concatenation:


The script language is a Forth-like stack-based language deliberately designed to be stateless and not Turing complete. Statelessness ensures that once a transaction is added to the block chain, there is no condition which renders it permanently unspendable. Turing-incompleteness (specifically, a lack of loops or gotos) makes the script language less flexible and more predictable, greatly simplifying the security model.

To test whether the transaction is valid, signature script and pubkey script arguments are pushed to the stack one item at a time, starting with Bob’s signature script and continuing to the end of Alice’s pubkey script. The figure below shows the evaluation of a standard P2PKH pubkey script; below the figure is a description of the process.

If false is not at the top of the stack after the pubkey script has been evaluated, the transaction is valid (provided there are no other problems with it).

P2SH Scripts Edit | History | Report Issue | Discuss

Pubkey scripts are created by spenders who have little interest in the long-term security or usefulness of the particular satoshis they’re currently spending. Receivers do care about the conditions imposed on the satoshis by the pubkey script and, if they want, they can ask spenders to use a particular pubkey script. Unfortunately, custom pubkey scripts are less convenient than short Bitcoin addresses and more difficult to secure than P2PKH pubkey hashes.

To solve these problems, pay-to-script-hash (P2SH) transactions were created in 2012 to let a spender create a pubkey script containing a hash of a second script, the redeem script.

The basic P2SH workflow, illustrated below, looks almost identical to the P2PKH workflow. Bob creates a redeem script with whatever script he wants, hashes the redeem script, and provides the redeem script hash to Alice. Alice creates a P2SH-style output containing Bob’s redeem script hash.

When Bob wants to spend the output, he provides his signature along with the full (serialized) redeem script in the signature script. The peer-to-peer network ensures the full redeem script hashes to the same value as the script hash Alice put in her output; it then processes the redeem script exactly as it would if it were the primary pubkey script, letting Bob spend the output if the redeem script does not return false.

The hash of the redeem script has the same properties as a pubkey hash—so it can be transformed into the standard Bitcoin address format with only one small change to differentiate it from a standard address. This makes collecting a P2SH-style address as simple as collecting a P2PKH-style address. The hash also obfuscates any public keys in the redeem script, so P2SH scripts are as secure as P2PKH pubkey hashes.

Standard Transactions Edit | History | Report Issue | Discuss

After the discovery of several dangerous bugs in early versions of Bitcoin, a test was added which only accepted transactions from the network if their pubkey scripts and signature scripts matched a small set of believed-to-be-safe templates, and if the rest of the transaction didn’t violate another small set of rules enforcing good network behavior. This is the IsStandard() test, and transactions which pass it are called standard transactions.

Non-standard transactions—those that fail the test—may be accepted by nodes not using the default Bitcoin Core settings. If they are included in blocks, they will also avoid the IsStandard test and be processed.

Besides making it more difficult for someone to attack Bitcoin for free by broadcasting harmful transactions, the standard transaction test also helps prevent users from creating transactions today that would make adding new transaction features in the future more difficult. For example, as described above, each transaction includes a version number—if users started arbitrarily changing the version number, it would become useless as a tool for introducing backwards-incompatible features.

As of Bitcoin Core 0.9, the standard pubkey script types are:

Pay To Public Key Hash (P2PKH) Edit | History | Report Issue | Discuss

P2PKH is the most common form of pubkey script used to send a transaction to one or multiple Bitcoin addresses.

Pubkey script: OP_DUP OP_HASH160 <PubKeyHash> OP_EQUALVERIFY OP_CHECKSIG Signature script: <sig> <pubkey> Pay To Script Hash (P2SH) Edit | History | Report Issue | Discuss

P2SH is used to send a transaction to a script hash. Each of the standard pubkey scripts can be used as a P2SH redeem script, but in practice only the multisig pubkey script makes sense until more transaction types are made standard.

Pubkey script: OP_HASH160 <Hash160(redeemScript)> OP_EQUAL Signature script: <sig> [sig] [sig...] <redeemScript> Multisig Edit | History | Report Issue | Discuss

Although P2SH multisig is now generally used for multisig transactions, this base script can be used to require multiple signatures before a UTXO can be spent.

In multisig pubkey scripts, called m-of-n, m is the minimum number of signatures which must match a public key; n is the number of public keys being provided. Both m and n should be op codes OP_1 through OP_16, corresponding to the number desired.

Because of an off-by-one error in the original Bitcoin implementation which must be preserved for compatibility, OP_CHECKMULTISIG consumes one more value from the stack than indicated by m, so the list of secp256k1 signatures in the signature script must be prefaced with an extra value (OP_0) which will be consumed but not used.

The signature script must provide signatures in the same order as the corresponding public keys appear in the pubkey script or redeem script. See the desciption in OP_CHECKMULTISIG for details.

Pubkey script: <m> <A pubkey> [B pubkey] [C pubkey...] <n> OP_CHECKMULTISIG Signature script: OP_0 <A sig> [B sig] [C sig...]

Although it’s not a separate transaction type, this is a P2SH multisig with 2-of-3:

Pubkey script: OP_HASH160 <Hash160(redeemScript)> OP_EQUAL Redeem script: <OP_2> <A pubkey> <B pubkey> <C pubkey> <OP_3> OP_CHECKMULTISIG Signature script: OP_0 <A sig> <C sig> <redeemScript> Pubkey Edit | History | Report Issue | Discuss

Pubkey scripts are a simplified form of the P2PKH pubkey script, but they aren’t as secure as P2PKH, so they generally aren’t used in new transactions anymore.

Pubkey script: <pubkey> OP_CHECKSIG Signature script: <sig> Null Data Edit | History | Report Issue | Discuss

Null data pubkey scripts let you add a small amount of arbitrary data to the block chain in exchange for paying a transaction fee, but doing so is discouraged. (Null data is a standard pubkey script type only because some people were adding data to the block chain in more harmful ways.)

Pubkey Script: OP_RETURN <0 to 40 bytes of data> (Null data scripts cannot be spent, so there's no signature script.) Non-Standard Transactions Edit | History | Report Issue | Discuss

If you use anything besides a standard pubkey script in an output, peers and miners using the default Bitcoin Core settings will neither accept, broadcast, nor mine your transaction. When you try to broadcast your transaction to a peer running the default settings, you will receive an error.

If you create a redeem script, hash it, and use the hash in a P2SH output, the network sees only the hash, so it will accept the output as valid no matter what the redeem script says. This allows payment to non-standard pubkey script almost as easily as payment to standard pubkey scripts. However, when you go to spend that output, peers and miners using the default settings will check the redeem script to see whether or not it’s a standard pubkey script. If it isn’t, they won’t process it further—so it will be impossible to spend that output until you find a miner who disables the default settings.

Note: standard transactions are designed to protect and help the network, not prevent you from making mistakes. It’s easy to create standard transactions which make the satoshis sent to them unspendable.

As of Bitcoin Core 0.9, standard transactions must also meet the following conditions:

  • The transaction must be finalized: either its locktime must be in the past (or less than or equal to the current block height), or all of its sequence numbers must be 0xffffffff.

  • The transaction must be smaller than 100,000 bytes. That’s around 200 times larger than a typical single-input, single-output P2PKH transaction.

  • Each of the transaction’s inputs must be smaller than 500 bytes. That’s large enough to allow 3-of-3 multisig transactions in P2SH. Multisig transactions which require more than 3 public keys are currently non-standard.

  • The transaction’s signature script must only push data to the script evaluation stack. It cannot push new OP codes, with the exception of OP codes which solely push data to the stack.

  • The transaction must not include any outputs which receive fewer than the defined minimum number of satoshis, currently 546.

Signature Hash Types Edit | History | Report Issue | Discuss

OP_CHECKSIG extracts a non-stack argument from each signature it evaluates, allowing the signer to decide which parts of the transaction to sign. Since the signature protects those parts of the transaction from modification, this lets signers selectively choose to let other people modify their transactions.

The various options for what to sign are called signature hash types. There are three base SIGHASH types currently available:

The base types can be modified with the SIGHASH_ANYONECANPAY (anyone can pay) flag, creating three new combined types:

Because each input is signed, a transaction with multiple inputs can have multiple signature hash types signing different parts of the transaction. For example, a single-input transaction signed with NONE could have its output changed by the miner who adds it to the block chain. On the other hand, if a two-input transaction has one input signed with NONE and one input signed with ALL, the ALL signer can choose where to spend the satoshis without consulting the NONE signer—but nobody else can modify the transaction.

Locktime And Sequence Number Edit | History | Report Issue | Discuss

One thing all signature hash types sign is the transaction’s locktime. (Called nLockTime in the Bitcoin Core source code.) The locktime indicates the earliest time a transaction can be added to the block chain.

Locktime allows signers to create time-locked transactions which will only become valid in the future, giving the signers a chance to change their minds.

If any of the signers change their mind, they can create a new non-locktime transaction. The new transaction will use, as one of its inputs, one of the same outputs which was used as an input to the locktime transaction. This makes the locktime transaction invalid if the new transaction is added to the block chain before the time lock expires.

Care must be taken near the expiry time of a time lock. The peer-to-peer network allows block time to be up to two hours ahead of real time, so a locktime transaction can be added to the block chain up to two hours before its time lock officially expires. Also, blocks are not created at guaranteed intervals, so any attempt to cancel a valuable transaction should be made a few hours before the time lock expires.

Previous versions of Bitcoin Core provided a feature which prevented transaction signers from using the method described above to cancel a time-locked transaction, but a necessary part of this feature was disabled to prevent denial of service attacks. A legacy of this system are four-byte sequence numbers in every input. Sequence numbers were meant to allow multiple signers to agree to update a transaction; when they finished updating the transaction, they could agree to set every input’s sequence number to the four-byte unsigned maximum (0xffffffff), allowing the transaction to be added to a block even if its time lock had not expired.

Even today, setting all sequence numbers to 0xffffffff (the default in Bitcoin Core) can still disable the time lock, so if you want to use locktime, at least one input must have a sequence number below the maximum. Since sequence numbers are not used by the network for any other purpose, setting any sequence number to zero is sufficient to enable locktime.

Locktime itself is an unsigned 4-byte integer which can be parsed two ways:

  • If less than 500 million, locktime is parsed as a block height. The transaction can be added to any block which has this height or higher.

  • If greater than or equal to 500 million, locktime is parsed using the Unix epoch time format (the number of seconds elapsed since 1970-01-01T00:00 UTC—currently over 1.395 billion). The transaction can be added to any block whose block time is greater than the locktime.

Transaction Fees And Change Edit | History | Report Issue | Discuss

Transactions typically pay transaction fees based on the total byte size of the signed transaction. The transaction fee is given to the Bitcoin miner, as explained in the block chain section, and so it is ultimately up to each miner to choose the minimum transaction fee they will accept.

By default, miners reserve 50 KB of each block for high-priority transactions which spend satoshis that haven’t been spent for a long time. The remaining space in each block is typically allocated to transactions based on their fee per byte, with higher-paying transactions being added in sequence until all of the available space is filled.

As of Bitcoin Core 0.9, transactions which do not count as high-priority transactions need to pay a minimum fee (currently 1,000 satoshis) to be broadcast across the network. Any transaction paying only the minimum fee should be prepared to wait a long time before there’s enough spare space in a block to include it. Please see the verifying payment section for why this could be important.

Since each transaction spends Unspent Transaction Outputs (UTXOs) and because a UTXO can only be spent once, the full value of the included UTXOs must be spent or given to a miner as a transaction fee. Few people will have UTXOs that exactly match the amount they want to pay, so most transactions include a change output.

Change outputs are regular outputs which spend the surplus satoshis from the UTXOs back to the spender. They can reuse the same P2PKH pubkey hash or P2SH script hash as was used in the UTXO, but for the reasons described in the next subsection, it is highly recommended that change outputs be sent to a new P2PKH or P2SH address.

Avoiding Key Reuse Edit | History | Report Issue | Discuss

In a transaction, the spender and receiver each reveal to each other all public keys or addresses used in the transaction. This allows either person to use the public block chain to track past and future transactions involving the other person’s same public keys or addresses.

If the same public key is reused often, as happens when people use Bitcoin addresses (hashed public keys) as static payment addresses, other people can easily track the receiving and spending habits of that person, including how many satoshis they control in known addresses.

It doesn’t have to be that way. If each public key is used exactly twice—once to receive a payment and once to spend that payment—the user can gain a significant amount of financial privacy.

Even better, using new public keys or unique addresses when accepting payments or creating change outputs can be combined with other techniques discussed later, such as CoinJoin or merge avoidance, to make it extremely difficult to use the block chain by itself to reliably track how users receive and spend their satoshis.

Avoiding key reuse can also provide security against attacks which might allow reconstruction of private keys from public keys (hypothesized) or from signature comparisons (possible today under certain circumstances described below, with more general attacks hypothesized).

  1. Unique (non-reused) P2PKH and P2SH addresses protect against the first type of attack by keeping ECDSA public keys hidden (hashed) until the first time satoshis sent to those addresses are spent, so attacks are effectively useless unless they can reconstruct private keys in less than the hour or two it takes for a transaction to be well protected by the block chain.

  2. Unique (non-reused) private keys protect against the second type of attack by only generating one signature per private key, so attackers never get a subsequent signature to use in comparison-based attacks. Existing comparison-based attacks are only practical today when insufficient entropy is used in signing or when the entropy used is exposed by some means, such as a side-channel attack.

So, for both privacy and security, we encourage you to build your applications to avoid public key reuse and, when possible, to discourage users from reusing addresses. If your application needs to provide a fixed URI to which payments should be sent, please see the bitcoin: URI section below.

Transaction Malleability Edit | History | Report Issue | Discuss

None of Bitcoin’s signature hash types protect the signature script, leaving the door open for a limited denial of service attack called transaction malleability. The signature script contains the secp256k1 signature, which can’t sign itself, allowing attackers to make non-functional modifications to a transaction without rendering it invalid. For example, an attacker can add some data to the signature script which will be dropped before the previous pubkey script is processed.

Although the modifications are non-functional—so they do not change what inputs the transaction uses nor what outputs it pays—they do change the computed hash of the transaction. Since each transaction links to previous transactions using hashes as a transaction identifier (txid), a modified transaction will not have the txid its creator expected.

This isn’t a problem for most Bitcoin transactions which are designed to be added to the block chain immediately. But it does become a problem when the output from a transaction is spent before that transaction is added to the block chain.

Bitcoin developers have been working to reduce transaction malleability among standard transaction types, but a complete fix is still only in the planning stages. At present, new transactions should not depend on previous transactions which have not been added to the block chain yet, especially if large amounts of satoshis are at stake.

Transaction malleability also affects payment tracking. Bitcoin Core’s RPC interface lets you track transactions by their txid—but if that txid changes because the transaction was modified, it may appear that the transaction has disappeared from the network.

Current best practices for transaction tracking dictate that a transaction should be tracked by the transaction outputs (UTXOs) it spends as inputs, as they cannot be changed without invalidating the transaction.

Best practices further dictate that if a transaction does seem to disappear from the network and needs to be reissued, that it be reissued in a way that invalidates the lost transaction. One method which will always work is to ensure the reissued payment spends all of the same outputs that the lost transaction used as inputs.

Contracts Edit | History | Report Issue | Discuss

Contracts are transactions which use the decentralized Bitcoin system to enforce financial agreements. Bitcoin contracts can often be crafted to minimize dependency on outside agents, such as the court system, which significantly decreases the risk of dealing with unknown entities in financial transactions.

The following subsections will describe a variety of Bitcoin contracts already in use. Because contracts deal with real people, not just transactions, they are framed below in story format.

Besides the contract types described below, many other contract types have been proposed. Several of them are collected on the Contracts page of the Bitcoin Wiki.

Escrow And Arbitration Edit | History | Report Issue | Discuss

Charlie-the-customer wants to buy a product from Bob-the-businessman, but neither of them trusts the other person, so they use a contract to help ensure Charlie gets his merchandise and Bob gets his payment.

A simple contract could say that Charlie will spend satoshis to an output which can only be spent if Charlie and Bob both sign the input spending it. That means Bob won’t get paid unless Charlie gets his merchandise, but Charlie can’t get the merchandise and keep his payment.

This simple contract isn’t much help if there’s a dispute, so Bob and Charlie enlist the help of Alice-the-arbitrator to create an escrow contract. Charlie spends his satoshis to an output which can only be spent if two of the three people sign the input. Now Charlie can pay Bob if everything is ok, Bob can refund Charlie’s money if there’s a problem, or Alice can arbitrate and decide who should get the satoshis if there’s a dispute.

To create a multiple-signature (multisig) output, they each give the others a public key. Then Bob creates the following P2SH multisig redeem script:

OP_2 [A's pubkey] [B's pubkey] [C's pubkey] OP_3 OP_CHECKMULTISIG

(Op codes to push the public keys onto the stack are not shown.)

OP_2 and OP_3 push the actual numbers 2 and 3 onto the stack. OP_2 specifies that 2 signatures are required to sign; OP_3 specifies that 3 public keys (unhashed) are being provided. This is a 2-of-3 multisig pubkey script, more generically called a m-of-n pubkey script (where m is the minimum matching signatures required and n in the number of public keys provided).

Bob gives the redeem script to Charlie, who checks to make sure his public key and Alice’s public key are included. Then he hashes the redeem script to create a P2SH redeem script and pays the satoshis to it. Bob sees the payment get added to the block chain and ships the merchandise.

Unfortunately, the merchandise gets slightly damaged in transit. Charlie wants a full refund, but Bob thinks a 10% refund is sufficient. They turn to Alice to resolve the issue. Alice asks for photo evidence from Charlie along with a copy of the redeem script Bob created and Charlie checked.

After looking at the evidence, Alice thinks a 40% refund is sufficient, so she creates and signs a transaction with two outputs, one that spends 60% of the satoshis to Bob’s public key and one that spends the remaining 40% to Charlie’s public key.

In the signature script Alice puts her signature and a copy of the unhashed serialized redeem script that Bob created. She gives a copy of the incomplete transaction to both Bob and Charlie. Either one of them can complete it by adding his signature to create the following signature script:

OP_0 [A's signature] [B's or C's signature] [serialized redeem script]

(Op codes to push the signatures and redeem script onto the stack are not shown. OP_0 is a workaround for an off-by-one error in the original implementation which must be preserved for compatibility. Note that the signature script must provide signatures in the same order as the corresponding public keys appear in the redeem script. See the description in OP_CHECKMULTISIG for details.)

When the transaction is broadcast to the network, each peer checks the signature script against the P2SH output Charlie previously paid, ensuring that the redeem script matches the redeem script hash previously provided. Then the redeem script is evaluated, with the two signatures being used as input data. Assuming the redeem script validates, the two transaction outputs show up in Bob’s and Charlie’s wallets as spendable balances.

However, if Alice created and signed a transaction neither of them would agree to, such as spending all the satoshis to herself, Bob and Charlie can find a new arbitrator and sign a transaction spending the satoshis to another 2-of-3 multisig redeem script hash, this one including a public key from that second arbitrator. This means that Bob and Charlie never need to worry about their arbitrator stealing their money.

Resource: BitRated provides a multisig arbitration service interface using HTML/JavaScript on a GNU AGPL-licensed website.

Micropayment Channel Edit | History | Report Issue | Discuss

Alice also works part-time moderating forum posts for Bob. Every time someone posts to Bob’s busy forum, Alice skims the post to make sure it isn’t offensive or spam. Alas, Bob often forgets to pay her, so Alice demands to be paid immediately after each post she approves or rejects. Bob says he can’t do that because hundreds of small payments will cost him thousands of satoshis in transaction fees, so Alice suggests they use a micropayment channel.

Bob asks Alice for her public key and then creates two transactions. The first transaction pays 100 millibitcoins to a P2SH output whose 2-of-2 multisig redeem script requires signatures from both Alice and Bob. This is the bond transaction. Broadcasting this transaction would let Alice hold the millibitcoins hostage, so Bob keeps this transaction private for now and creates a second transaction.

The second transaction spends all of the first transaction’s millibitcoins (minus a transaction fee) back to Bob after a 24 hour delay enforced by locktime. This is the refund transaction. Bob can’t sign the refund transaction by himself, so he gives it to Alice to sign, as shown in the illustration below.

Alice checks that the refund transaction’s locktime is 24 hours in the future, signs it, and gives a copy of it back to Bob. She then asks Bob for the bond transaction and checks that the refund transaction spends the output of the bond transaction. She can now broadcast the bond transaction to the network to ensure Bob has to wait for the time lock to expire before further spending his millibitcoins. Bob hasn’t actually spent anything so far, except possibly a small transaction fee, and he’ll be able to broadcast the refund transaction in 24 hours for a full refund.

Now, when Alice does some work worth 1 millibitcoin, she asks Bob to create and sign a new version of the refund transaction. Version two of the transaction spends 1 millibitcoin to Alice and the other 99 back to Bob; it does not have a locktime, so Alice can sign it and spend it whenever she wants. (But she doesn’t do that immediately.)

Alice and Bob repeat these work-and-pay steps until Alice finishes for the day, or until the time lock is about to expire. Alice signs the final version of the refund transaction and broadcasts it, paying herself and refunding any remaining balance to Bob. The next day, when Alice starts work, they create a new micropayment channel.

If Alice fails to broadcast a version of the refund transaction before its time lock expires, Bob can broadcast the first version and receive a full refund. This is one reason micropayment channels are best suited to small payments—if Alice’s Internet service goes out for a few hours near the time lock expiry, she could be cheated out of her payment.

Transaction malleability, discussed above in the Transactions section, is another reason to limit the value of micropayment channels. If someone uses transaction malleability to break the link between the two transactions, Alice could hold Bob’s 100 millibitcoins hostage even if she hadn’t done any work.

For larger payments, Bitcoin transaction fees are very low as a percentage of the total transaction value, so it makes more sense to protect payments with immediately-broadcast separate transactions.

Resource: The bitcoinj Java library provides a complete set of micropayment functions, an example implementation, and a tutorial all under an Apache license.

CoinJoin Edit | History | Report Issue | Discuss

Alice is concerned about her privacy. She knows every transaction gets added to the public block chain, so when Bob and Charlie pay her, they can each easily track those satoshis to learn what Bitcoin addresses she pays, how much she pays them, and possibly how many satoshis she has left.

Alice isn’t a criminal, she just wants plausible deniability about where she has spent her satoshis and how many she has left, so she starts up the Tor anonymity service on her computer and logs into an IRC chatroom as “AnonGirl.”

Also in the chatroom are “Nemo” and “Neminem.” They collectively agree to transfer satoshis between each other so no one besides them can reliably determine who controls which satoshis. But they’re faced with a dilemma: who transfers their satoshis to one of the other two pseudonymous persons first? The CoinJoin-style contract, shown in the illustration below, makes this decision easy: they create a single transaction which does all of the spending simultaneously, ensuring none of them can steal the others’ satoshis.

Each contributor looks through their collection of Unspent Transaction Outputs (UTXOs) for 100 millibitcoins they can spend. They then each generate a brand new public key and give UTXO details and pubkey hashes to the facilitator. In this case, the facilitator is AnonGirl; she creates a transaction spending each of the UTXOs to three equally-sized outputs. One output goes to each of the contributors’ pubkey hashes.

AnonGirl then signs her inputs using SIGHASH_ALL to ensure nobody can change the input or output details. She gives the partially-signed transaction to Nemo who signs his inputs the same way and passes it to Neminem, who also signs it the same way. Neminem then broadcasts the transaction to the peer-to-peer network, mixing all of the millibitcoins in a single transaction.

As you can see in the illustration, there’s no way for anyone besides AnonGirl, Nemo, and Neminem to confidently determine who received which output, so they can each spend their output with plausible deniability.

Now when Bob or Charlie try to track Alice’s transactions through the block chain, they’ll also see transactions made by Nemo and Neminem. If Alice does a few more CoinJoins, Bob and Charlie might have to guess which transactions made by dozens or hundreds of people were actually made by Alice.

The complete history of Alice’s satoshis is still in the block chain, so a determined investigator could talk to the people AnonGirl CoinJoined with to find out the ultimate origin of her satoshis and possibly reveal AnonGirl as Alice. But against anyone casually browsing block chain history, Alice gains plausible deniability.

The CoinJoin technique described above costs the participants a small amount of satoshis to pay the transaction fee. An alternative technique, purchaser CoinJoin, can actually save them satoshis and improve their privacy at the same time.

AnonGirl waits in the IRC chatroom until she wants to make a purchase. She announces her intention to spend satoshis and waits until someone else wants to make a purchase, likely from a different merchant. Then they combine their inputs the same way as before but set the outputs to the separate merchant addresses so nobody will be able to figure out solely from block chain history which one of them bought what from the merchants.

Since they would’ve had to pay a transaction fee to make their purchases anyway, AnonGirl and her co-spenders don’t pay anything extra—but because they reduced overhead by combining multiple transactions, saving bytes, they may be able to pay a smaller aggregate transaction fee, saving each one of them a tiny amount of satoshis.

Resource: An alpha-quality (as of this writing) implementation of decentralized CoinJoin is CoinMux, available under the Apache license.

Wallets Edit | History | Report Issue | Discuss

A Bitcoin wallet can refer to either a wallet program or a wallet file. Wallet programs create public keys to receive satoshis and use the corresponding private keys to spend those satoshis. Wallet files store private keys and (optionally) other information related to transactions for the wallet program.

Wallet programs and wallet files are addressed below in separate subsections, and this document attempts to always make it clear whether we’re talking about wallet programs or wallet files.

Wallet Programs Edit | History | Report Issue | Discuss

Permitting receiving and spending of satoshis is the only essential feature of wallet software—but a particular wallet program doesn’t need to do both things. Two wallet programs can work together, one program distributing public keys in order to receive satoshis and another program signing transactions spending those satoshis.

Wallet programs also need to interact with the peer-to-peer network to get information from the block chain and to broadcast new transactions. However, the programs which distribute public keys or sign transactions don’t need to interact with the peer-to-peer network themselves.

This leaves us with three necessary, but separable, parts of a wallet system: a public key distribution program, a signing program, and a networked program. In the subsections below, we will describe common combinations of these parts.

Note: we speak about distributing public keys generically. In many cases, P2PKH or P2SH hashes will be distributed instead of public keys, with the actual public keys only being distributed when the outputs they control are spent.

Full-Service Wallets Edit | History | Report Issue | Discuss

The simplest wallet is a program which performs all three functions: it generates private keys, derives the corresponding public keys, helps distribute those public keys as necessary, monitors for outputs spent to those public keys, creates and signs transactions spending those outputs, and broadcasts the signed transactions.

As of this writing, almost all popular wallets can be used as full-service wallets.

The main advantage of full-service wallets is that they are easy to use. A single program does everything the user needs to receive and spend satoshis.

The main disadvantage of full-service wallets is that they store the private keys on a device connected to the Internet. The compromise of such devices is a common occurrence, and an Internet connection makes it easy to transmit private keys from a compromised device to an attacker.

To help protect against theft, many wallet programs offer users the option of encrypting the wallet files which contain the private keys. This protects the private keys when they aren’t being used, but it cannot protect against an attack designed to capture the encryption key or to read the decrypted keys from memory.

Signing-Only Wallets Edit | History | Report Issue | Discuss

To increase security, private keys can be generated and stored by a separate wallet program operating in a more secure environment. These signing-only wallets work in conjunction with a networked wallet which interacts with the peer-to-peer network.

Signing-only wallets programs typically use deterministic key creation (described in a later subsection) to create parent private and public keys which can create child private and public keys.

When first run, the signing-only wallet creates a parent private key and transfers the corresponding parent public key to the networked wallet.

The networked wallet uses the parent public key to derive child public keys, optionally helps distribute them, monitors for outputs spent to those public keys, creates unsigned transactions spending those outputs, and transfers the unsigned transactions to the signing-only wallet.

Often, users are given a chance to review the unsigned transactions’ details (particularly the output details) using the signing-only wallet.

After the optional review step, the offline wallet uses the parent private key to derive the appropriate child private keys and signs the transactions, giving the signed transactions back to the networked wallet.

The networked wallet then broadcasts the signed transactions to the peer-to-peer network.

The following subsections describe the two most common variants of signing-only wallets: offline wallets and hardware wallets.

Offline Wallets Edit | History | Report Issue | Discuss

Several full-service wallets programs will also operate as two separate wallets: one program instance acting as a signing-only wallet (often called an “offline wallet”) and the other program instance acting as the networked wallet (often called an “online wallet” or “watching-only wallet”).

The offline wallet is so named because it is intended to be run on a device which does not connect to any network, greatly reducing the number of attack vectors. If this is the case, it is usually up to the user to handle all data transfer using removable media such as USB drives. The user’s workflow is something like:

  1. (Offline) Disable all network connections on a device and install the wallet software. Start the wallet software in offline mode to create the parent private and public keys. Copy the parent public key to removable media.

  2. (Online) Install the wallet software on another device, this one connected to the Internet, and import the parent public key from the removable media. As you would with a full-service wallet, distribute public keys to receive payment. When ready to spend satoshis, fill in the output details and save the unsigned transaction generated by the wallet to removable media.

  3. (Offline) Open the unsigned transaction in the offline instance, review the output details to make sure they spend the correct amount to the correct address. This prevents malware on the online wallet from tricking the user into signing a transaction which pays an attacker. After review, sign the transaction and save it to removable media.

  4. (Online) Open the signed transaction in the online instance so it can broadcast it to the peer-to-peer network.

The primary advantage of offline wallets is their possibility for greatly improved security over full-service wallets. As long as the offline wallet is not compromised (or flawed) and the user reviews all outgoing transactions before signing, the user’s satoshis are safe even if the online wallet is compromised.

The primary disadvantage of offline wallets is hassle. For maximum security, they require the user dedicate a device to only offline tasks. The offline device must be booted up whenever funds are to be spent, and the user must physically copy data from the online device to the offline device and back.

Hardware Wallets Edit | History | Report Issue | Discuss

Hardware wallets are devices dedicated to running a signing-only wallet. Their dedication lets them eliminate many of the vulnerabilities present in operating systems designed for general use, allowing them to safely communicate directly with other devices so users don’t need to transfer data manually. The user’s workflow is something like:

  1. (Hardware) Create parent private and public keys. Connect hardware wallet to a networked device so it can get the parent public key.

  2. (Networked) As you would with a full-service wallet, distribute public keys to receive payment. When ready to spend satoshis, fill in the transaction details, connect the hardware wallet, and click Spend. The networked wallet will automatically send the transaction details to the hardware wallet.

  3. (Hardware) Review the transaction details on the hardware wallet’s screen. Some hardware wallets may prompt for a passphrase or PIN number. The hardware wallet signs the transaction and uploads it to the networked wallet.

  4. (Networked) The networked wallet receives the signed transaction from the hardware wallet and broadcasts it to the network.

The primary advantage of hardware wallets is their possibility for greatly improved security over full-service wallets with much less hassle than offline wallets.

The primary disadvantage of hardware wallets is their hassle. Even though the hassle is less than that of offline wallets, the user must still purchase a hardware wallet device and carry it with them whenever they need to make a transaction using the signing-only wallet.

An additional (hopefully temporary) disadvantage is that, as of this writing, very few popular wallet programs support hardware wallets—although almost all popular wallet programs have announced their intention to support at least one model of hardware wallet.

Distributing-Only Wallets Edit | History | Report Issue | Discuss

Wallet programs which run in difficult-to-secure environments, such as webservers, can be designed to distribute public keys (including P2PKH or P2SH addresses) and nothing more. There are two common ways to design these minimalist wallets:

Neither method adds a significant amount of overhead, especially if a database is used anyway to associate each incoming payment with a separate public key for payment tracking. See the Payment Processing section for details.

Wallet Files Edit | History | Report Issue | Discuss

Bitcoin wallets at their core are a collection of private keys. These collections are stored digitally in a file, or can even be physically stored on pieces of paper.

Private Key Formats Edit | History | Report Issue | Discuss

Private keys are what are used to unlock satoshis from a particular address. In Bitcoin, a private key in standard format is simply a 256-bit number, between the values:

0x01 and 0xFFFF FFFF FFFF FFFF FFFF FFFF FFFF FFFE BAAE DCE6 AF48 A03B BFD2 5E8C D036 4140, representing nearly the entire range of 2256-1 values. The range is governed by the secp256k1 ECDSA encryption standard used by Bitcoin.

Wallet Import Format (WIF) Edit | History | Report Issue | Discuss

In order to make copying of private keys less prone to error, Wallet Import Format may be utilized. WIF uses base58Check encoding on an private key, greatly decreasing the chance of copying error, much like standard Bitcoin addresses.

  1. Take a private key.

  2. Add a 0x80 byte in front of it for mainnet addresses or 0xef for testnet addresses.

  3. Append a 0x01 byte after it if it should be used with compressed public keys (described in a later subsection). Nothing is appended if it is used with uncompressed public keys.

  4. Perform a SHA-256 hash on the extended key.

  5. Perform a SHA-256 hash on result of SHA-256 hash.

  6. Take the first four bytes of the second SHA-256 hash; this is the checksum.

  7. Add the four checksum bytes from point 5 at the end of the extended key from point 2.

  8. Convert the result from a byte string into a Base58 string using Base58Check encoding.

The process is easily reversible, using the Base58 decoding function, and removing the padding.

Mini Private Key Format Edit | History | Report Issue | Discuss

Mini private key format is a method for encoding a private key in under 30 characters, enabling keys to be embedded in a small physical space, such as physical bitcoin tokens, and more damage-resistant QR codes.

  1. The first character of mini keys is ‘S’.

  2. In order to determine if a mini private key is well-formatted, a question mark is added to the private key.

  3. The SHA256 hash is calculated. If the first byte produced is a `00’, it is well-formatted. This key restriction acts as a typo-checking mechanism. A user brute forces the process using random numbers until a well-formatted mini private key is produced.

  4. In order to derive the full private key, the user simply takes a single SHA256 hash of the original mini private key. This process is one-way: it is intractable to compute the mini private key format from the derived key.

Many implementations disallow the character ‘1’ in the mini private key due to its visual similarity to ‘l’.

Resource: A common tool to create and redeem these keys is the Casascius Bitcoin Address Utility.

Public Key Formats Edit | History | Report Issue | Discuss

Bitcoin ECDSA public keys represent a point on a particular Elliptic Curve (EC) defined in secp256k1. In their traditional uncompressed form, public keys contain an identification byte, a 32-byte X coordinate, and a 32-byte Y coordinate. The extremely simplified illustration below shows such a point on the elliptic curve used by Bitcoin, x2 = y3 + 7, over a field of contiguous numbers.

(Secp256k1 actually modulos coordinates by a large prime, which produces a field of non-contiguous integers and a significantly less clear plot, although the principles are the same.)

An almost 50% reduction in public key size can be realized without changing any fundamentals by dropping the Y coordinate. This is possible because only two points along the curve share any particular X coordinate, so the 32-byte Y coordinate can be replaced with a single bit indicating whether the point is on what appears in the illustration as the “top” side or the “bottom” side.

No data is lost by creating these compressed public keys—only a small amount of CPU is necessary to reconstruct the Y coordinate and access the uncompressed public key. Both uncompressed and compressed public keys are described in official secp256k1 documentation and supported by default in the widely-used OpenSSL library.

Because they’re easy to use, and because they reduce almost by half the block chain space used to store public keys for every spent output, compressed public keys are the default in Bitcoin Core and are the recommended default for all Bitcoin software.

However, Bitcoin Core prior to 0.6 used uncompressed keys. This creates a few complications, as the hashed form of an uncompressed key is different than the hashed form of a compressed key, so the same key works with two different P2PKH addresses. This also means that the key must be submitted in the correct format in the signature script so it matches the hash in the previous output’s pubkey script.

For this reason, Bitcoin Core uses several different identifier bytes to help programs identify how keys should be used:

  • Private keys meant to be used with compressed public keys have 0x01 appended to them before being Base-58 encoded. (See the private key encoding section above.)

  • Uncompressed public keys start with 0x04; compressed public keys begin with 0x03 or 0x02 depending on whether they’re greater or less than the midpoint of the curve. These prefix bytes are all used in official secp256k1 documentation.

Hierarchical Deterministic Key Creation Edit | History | Report Issue | Discuss

The hierarchical deterministic key creation and transfer protocol (HD protocol) greatly simplifies wallet backups, eliminates the need for repeated communication between multiple programs using the same wallet, permits creation of child accounts which can operate independently, gives each parent account the ability to monitor or control its children even if the child account is compromised, and divides each account into full-access and restricted-access parts so untrusted users or programs can be allowed to receive or monitor payments without being able to spend them.

The HD protocol takes advantage of the ECDSA public key creation function, point(), which takes a large integer (the private key) and turns it into a graph point (the public key):

point(private_key) == public_key

Because of the way point() functions, it’s possible to create a child public key by combining an existing (parent) public key with another public key created from any integer (i) value. This child public key is the same public key which would be created by the point() function if you added the i value to the original (parent) private key and then found the remainder of that sum divided by a global constant used by all Bitcoin software (G):

point( (parent_private_key + i) % G ) == parent_public_key + point(i)

This means that two or more independent programs which agree on a sequence of integers can create a series of unique child key pairs from a single parent key pair without any further communication. Moreover, the program which distributes new public keys for receiving payment can do so without any access to the private keys, allowing the public key distribution program to run on a possibly-insecure platform such as a public web server.

Child public keys can also create their own child public keys (grandchild public keys) by repeating the child key derivation operations:

point( (child_private_key + i) % G ) == child_public_key + point(i)

Whether creating child public keys or further-descended public keys, a predictable sequence of integer values would be no better than using a single public key for all transactions, as anyone who knew one child public key could find all of the other child public keys created from the same parent public key. Instead, a random seed can be used to deterministically generate the sequence of integer values so that the relationship between the child public keys is invisible to anyone without that seed.

The HD protocol uses a single root seed to create a hierarchy of child, grandchild, and other descended keys with unlinkable deterministically-generated integer values. Each child key also gets a deterministically-generated seed from its parent, called a chain code, so the compromising of one chain code doesn’t necessary compromise the integer sequence for the whole hierarchy, allowing the master chain code to continue being useful even if, for example, a web-based public key distribution program gets hacked.

As illustrated above, HD key derivation takes four inputs:

In the normal form shown in the above illustration, the parent chain code, the parent public key, and the index number are fed into a one-way cryptographic hash (HMAC-SHA512) to produce 512 bits of deterministically-generated-but-seemingly-random data. The seemingly-random 256 bits on the righthand side of the hash output are used as a new child chain code. The seemingly-random 256 bits on the lefthand side of the hash output are used as the integer value to be combined with either the parent private key or parent public key to, respectively, create either a child private key or child public key:

point( (parent_private_key + lefthand_hash_output) % G ) == child_public_key point(child_private_key) == parent_public_key + point(lefthand_hash_output)

Specifying different index numbers will create different unlinkable child keys from the same parent keys. Repeating the procedure for the child keys using the child chain code will create unlinkable grandchild keys.

Because creating child keys requires both a key and a chain code, the key and chain code together are called the extended key. An extended private key and its corresponding extended public key have the same chain code. The (top-level parent) master private key and master chain code are derived from random data, as illustrated below.

A root seed is created from either 128 bits, 256 bits, or 512 bits of random data. This root seed of as little as 128 bits is the the only data the user needs to backup in order to derive every key created by a particular wallet program using particular settings.

Warning: As of this writing, HD wallet programs are not expected to be fully compatible, so users must only use the same HD wallet program with the same HD-related settings for a particular root seed.

The root seed is hashed to create 512 bits of seemingly-random data, from which the master private key and master chain code are created (together, the master extended private key). The master public key is derived from the master private key using point(), which, together with the master chain code, is the master extended public key. The master extended keys are functionally equivalent to other extended keys; it is only their location at the top of the hierarchy which makes them special.

Hardened Keys Edit | History | Report Issue | Discuss

Hardened extended keys fix a potential problem with normal extended keys. If an attacker gets a normal parent chain code and parent public key, he can brute-force all chain codes deriving from it. If the attacker also obtains a child, grandchild, or further-descended private key, he can use the chain code to generate all of the extended private keys descending from that private key, as shown in the grandchild and great-grandchild generations of the illustration below.

Perhaps worse, the attacker can reverse the normal child private key derivation formula and subtract a parent chain code from a child private key to recover the parent private key, as shown in the child and parent generations of the illustration above. This means an attacker who acquires an extended public key and any private key descended from it can recover that public key’s private key and all keys descended from it.

For this reason, the chain code part of an extended public key should be better secured than standard public keys and users should be advised against exporting even non-extended private keys to possibly-untrustworthy environments.

This can be fixed, with some tradeoffs, by replacing the the normal key derivation formula with a hardened key derivation formula.

The normal key derivation formula, described in the section above, combines together the index number, the parent chain code, and the parent public key to create the child chain code and the integer value which is combined with the parent private key to create the child private key.

The hardened formula, illustrated above, combines together the index number, the parent chain code, and the parent private key to create the data used to generate the child chain code and child private key. This formula makes it impossible to create child public keys without knowing the parent private key. In other words, parent extended public keys can’t create hardened child public keys.

Because of that, a hardened extended private key is much less useful than a normal extended private key—however, hardened extended private keys create a firewall through which multi-level key derivation compromises cannot happen. Because hardened child extended public keys cannot generate grandchild chain codes on their own, the compromise of a parent extended public key cannot be combined with the compromise of a grandchild private key to create great-grandchild extended private keys.

The HD protocol uses different index numbers to indicate whether a normal or hardened key should be generated. Index numbers from 0x00 to 0x7fffffff (0 to 231-1) will generate a normal key; index numbers from 0x80000000 to 0xffffffff will generate a hardened key. To make descriptions easy, many developers use the prime symbol to indicate hardened keys, so the first normal key (0x00) is 0 and the first hardened key (0x80000000) is 0´.

(Bitcoin developers typically use the ASCII apostrophe rather than the unicode prime symbol, a convention we will henceforth follow.)

This compact description is further combined with slashes prefixed by m or M to indicate hierarchy and key type, with m being a private key and M being a public key. For example, m/0’/0/122’ refers to the 123rd hardened private child (by index number) of the first normal child (by index) of the first hardened child (by index) of the master private key. The following hierarchy illustrates prime notation and hardened key firewalls.

Wallets following the BIP32 HD protocol only create hardened children of the master private key (m) to prevent a compromised child key from compromising the master key. As there are no normal children for the master keys, the master public key is not used in HD wallets. All other keys can have normal children, so the corresponding extended public keys may be used instead.

The HD protocol also describes a serialization format for extended public keys and extended private keys. For details, please see the wallet section in the developer reference or BIP32 for the full HD protocol specification.

Storing Root Seeds Edit | History | Report Issue | Discuss

Root seeds in the HD protocol are 128, 256, or 512 bits of random data which must be backed up precisely. To make it more convenient to use non-digital backup methods, such as memorization or hand-copying, BIP39 defines a method for creating a 512-bit root seed from a pseudo-sentence (mnemonic) of common natural-language words which was itself created from 128 to 256 bits of entropy and optionally protected by a password.

The number of words generated correlates to the amount of entropy used:

Entropy Bits Words 128 12 160 15 192 18 224 21 256 24

The passphrase can be of any length. It is simply appended to the mnemonic pseudo-sentence, and then both the mnemonic and password are hashed 2,048 times using HMAC-SHA512, resulting in a seemingly-random 512-bit seed. Because any input to the hash function creates a seemingly-random 512-bit seed, there is no fundamental way to prove the user entered the correct password, possibly allowing the user to protect a seed even when under duress.

For implementation details, please see BIP39.

Loose-Key Wallets Edit | History | Report Issue | Discuss

Loose-Key wallets, also called “Just a Bunch Of Keys (JBOK)”, are a deprecated form of wallet that originated from the Bitcoin Core client wallet. The Bitcoin Core client wallet would create 100 private key/public key pairs automatically via a Pseudo-Random-Number Generator (PRNG) for later use.

These unused private keys are stored in a virtual “key pool”, with new keys being generated whenever a previously-generated key was used, ensuring the pool maintained 100 unused keys. (If the wallet is encrypted, new keys are only generated while the wallet is unlocked.)

This created considerable difficulty in backing up one’s keys, considering backups have to be run manually to save the newly-generated private keys. If a new key pair set is generated, used, and then lost prior to a backup, the stored satoshis are likely lost forever. Many older-style mobile wallets followed a similar format, but only generated a new private key upon user demand.

This wallet type is being actively phased out and discouraged from being used due to the backup hassle.

Payment Processing Edit | History | Report Issue | Discuss

Payment processing encompasses the steps spenders and receivers perform to make and accept payments in exchange for products or services. The basic steps have not changed since the dawn of commerce, but the technology has. This section will explain how receivers and spenders can, respectively, request and make payments using Bitcoin—and how they can deal with complications such as refunds and recurrent rebilling.

Bitcoin payment processing is being actively developed at the moment, so each subsection below attempts to describe what’s widely deployed now, what’s new, and what might be coming before the end of 2014.

The figure above illustrates payment processing using Bitcoin from a receiver’s perspective, starting with a new order. The following subsections will each address the three common steps and the three occasional or optional steps.

It is worth mentioning that each of these steps can be outsourced by using third party APIs and services.

Pricing Orders Edit | History | Report Issue | Discuss

Because of exchange rate variability between satoshis and national currencies (fiat), many Bitcoin orders are priced in fiat but paid in satoshis, necessitating a price conversion.

Exchange rate data is widely available through HTTP-based APIs provided by currency exchanges. Several organizations also aggregate data from multiple exchanges to create index prices, which are also available using HTTP-based APIs.

Any applications which automatically calculate order totals using exchange rate data must take steps to ensure the price quoted reflects the current general market value of satoshis, or the applications could accept too few satoshis for the product or service being sold. Alternatively, they could ask for too many satoshis, driving away potential spenders.

To minimize problems, your applications may want to collect data from at least two separate sources and compare them to see how much they differ. If the difference is substantial, your applications can enter a safe mode until a human is able to evaluate the situation.

You may also want to program your applications to enter a safe mode if exchange rates are rapidly increasing or decreasing, indicating a possible problem in the Bitcoin market which could make it difficult to spend any satoshis received today.

Exchange rates lie outside the control of Bitcoin and related technologies, so there are no new or planned technologies which will make it significantly easier for your program to correctly convert order totals from fiat into satoshis.

Because the exchange rate fluctuates over time, order totals pegged to fiat must expire to prevent spenders from delaying payment in the hope that satoshis will drop in price. Most widely-used payment processing systems currently expire their invoices after 10 to 20 minutes.

Shorter expiration periods increase the chance the invoice will expire before payment is received, possibly necessitating manual intervention to request an additional payment or to issue a refund. Longer expiration periods increase the chance that the exchange rate will fluctuate a significant amount before payment is received.

Requesting Payments Edit | History | Report Issue | Discuss

Before requesting payment, your application must create a Bitcoin address, or acquire an address from another program such as Bitcoin Core. Bitcoin addresses are described in detail in the Transactions section. Also described in that section are two important reasons to avoid using an address more than once—but a third reason applies especially to payment requests:

Using a separate address for each incoming payment makes it trivial to determine which customers have paid their payment requests. Your applications need only track the association between a particular payment request and the address used in it, and then scan the block chain for transactions matching that address.

The next subsections will describe in detail the following four compatible ways to give the spender the address and amount to be paid. For increased convenience and compatibility, providing all of these options in your payment requests is recommended.

  1. All wallet software lets its users paste in or manually enter an address and amount into a payment screen. This is, of course, inconvenient—but it makes an effective fallback option.

  2. Almost all desktop wallets can associate with bitcoin: URIs, so spenders can click a link to pre-fill the payment screen. This also works with many mobile wallets, but it generally does not work with web-based wallets unless the spender installs a browser extension or manually configures a URI handler.

  3. Most mobile wallets support scanning bitcoin: URIs encoded in a QR code, and almost all wallets can display them for accepting payment. While also handy for online orders, QR Codes are especially useful for in-person purchases.

  4. Recent wallet updates add support for the new payment protocol providing increased security, authentication of a receiver’s identity using X.509 certificates, and other important features such as refunds.

Warning: Special care must be taken to avoid the theft of incoming payments. In particular, private keys should not be stored on web servers, and payment requests should be sent over HTTPS or other secure methods to prevent man-in-the-middle attacks from replacing your Bitcoin address with the attacker’s address.

Plain Text Edit | History | Report Issue | Discuss

To specify an amount directly for copying and pasting, you must provide the address, the amount, and the denomination. An expiration time for the offer may also be specified. For example:

(Note: all examples in this section use testnet addresses.)

Pay: mjSk1Ny9spzU2fouzYgLqGUD8U41iR35QN Amount: 100 BTC You must pay by: 2014-04-01 at 23:00 UTC

Indicating the denomination is critical. As of this writing, popular Bitcoin wallet software defaults to denominating amounts in either bitcoins (BTC) , millibitcoins (mBTC) or microbitcoins (uBTC, “bits”). Choosing between each unit is widely supported, but other software also lets its users select denomination amounts from some or all of the following options:

Bitcoins Unit (Abbreviation) 1.0 bitcoin (BTC) 0.01 bitcent (cBTC) 0.001 millibitcoin (mBTC) 0.000001 microbitcoin (uBTC, “bits”) 0.00000001 satoshi bitcoin: URI Edit | History | Report Issue | Discuss

The bitcoin: URI scheme defined in BIP21 eliminates denomination confusion and saves the spender from copying and pasting two separate values. It also lets the payment request provide some additional information to the spender. An example:


Only the address is required, and if it is the only thing specified, wallets will pre-fill a payment request with it and let the spender enter an amount. The amount specified is always in decimal bitcoins (BTC).

Two other parameters are widely supported. The label parameter is generally used to provide wallet software with the recipient’s name. The message parameter is generally used to describe the payment request to the spender. Both the label and the message are commonly stored by the spender’s wallet software—but they are never added to the actual transaction, so other Bitcoin users cannot see them. Both the label and the message must be URI encoded.

All four parameters used together, with appropriate URI encoding, can be seen in the line-wrapped example below.

bitcoin:mjSk1Ny9spzU2fouzYgLqGUD8U41iR35QN\ ?amount=0.10\ &label=Example+Merchant\ &message=Order+of+flowers+%26+chocolates

The URI scheme can be extended, as will be seen in the payment protocol section below, with both new optional and required parameters. As of this writing, the only widely-used parameter besides the four described above is the payment protocol’s r parameter.

Programs accepting URIs in any form must ask the user for permission before paying unless the user has explicitly disabled prompting (as might be the case for micropayments).

QR Codes Edit | History | Report Issue | Discuss

QR codes are a popular way to exchange bitcoin: URIs in person, in images, or in videos. Most mobile Bitcoin wallet apps, and some desktop wallets, support scanning QR codes to pre-fill their payment screens.

The figure below shows the same bitcoin: URI code encoded as four different Bitcoin QR codes at four different error correction levels. The QR code can include the label and message parameters—and any other optional parameters—but they were omitted here to keep the QR code small and easy to scan with unsteady or low-resolution mobile cameras.

The error correction is combined with a checksum to ensure the Bitcoin QR code cannot be successfully decoded with data missing or accidentally altered, so your applications should choose the appropriate level of error correction based on the space you have available to display the code. Low-level damage correction works well when space is limited, and quartile-level damage correction helps ensure fast scanning when displayed on high-resolution screens.

Payment Protocol Edit | History | Report Issue | Discuss

Bitcoin Core 0.9 supports the new payment protocol. The payment protocol adds many important features to payment requests:

  • Supports X.509 certificates and SSL encryption to verify receivers’ identity and help prevent man-in-the-middle attacks.

  • Provides more detail about the requested payment to spenders.

  • Allows spenders to submit transactions directly to receivers without going through the peer-to-peer network. This can speed up payment processing and work with planned features such as child-pays-for-parent transaction fees and offline NFC or Bluetooth-based payments.

Instead of being asked to pay a meaningless address, such as “mjSk1Ny9spzU2fouzYgLqGUD8U41iR35QN”, spenders are asked to pay the Common Name (CN) description from the receiver’s X.509 certificate, such as “www.bitcoin.org”.

To request payment using the payment protocol, you use an extended (but backwards-compatible) bitcoin: URI. For example:

bitcoin:mjSk1Ny9spzU2fouzYgLqGUD8U41iR35QN\ ?amount=0.10\ &label=Example+Merchant\ &message=Order+of+flowers+%26+chocolates\ &r=https://example.com/pay/mjSk1Ny9spzU2fouzYgLqGUD8U41iR35QN

None of the parameters provided above, except r, are required for the payment protocol—but your applications may include them for backwards compatibility with wallet programs which don’t yet handle the payment protocol.

The r parameter tells payment-protocol-aware wallet programs to ignore the other parameters and fetch a PaymentRequest from the URL provided. The browser, QR code reader, or other program processing the URI opens the spender’s Bitcoin wallet program on the URI.

The Payment Protocol is described in depth in BIP70, BIP71, and BIP72. An example CGI program and description of all the parameters which can be used in the Payment Protocol is provided in the Developer Examples Payment Protocol subsection. In this subsection, we will briefly describe in story format how the Payment Protocol is typically used.

Charlie, the client, is shopping on a website run by Bob, the businessman. Charlie adds a few items to his shopping cart and clicks the “Checkout With Bitcoin” button.

Bob’s server automatically adds the following information to its invoice database:

  • The details of Charlie’s order, including items ordered and shipping address.

  • An order total in satoshis, perhaps created by converting prices in fiat to prices in satoshis.

  • An expiration time when that total will no longer be acceptable.

  • A pubkey script to which Charlie should send payment. Typically this will be a P2PKH or P2SH pubkey script containing a unique (never before used) secp256k1 public key.

After adding all that information to the database, Bob’s server displays a bitcoin: URI for Charlie to click to pay.

Charlie clicks on the bitcoin: URI in his browser. His browser’s URI handler sends the URI to his wallet program. The wallet is aware of the Payment Protocol, so it parses the r parameter and sends an HTTP GET to that URL looking for a PaymentRequest message.

The PaymentRequest message returned may include private information, such as Charlie’s mailing address, but the wallet must be able to access it without using prior authentication, such as HTTP cookies, so a publicly-accessible HTTPS URL with a guess-resistant part is typically used. The unique public key created for the payment request can be used to create a unique identifier. This is why, in the example URI above, the PaymentRequest URL contains the P2PKH address: https://example.com/pay/mjSk1Ny9spzU2fouzYgLqGUD8U41iR35QN

After receiving the HTTP GET to the URL above, the PaymentRequest-generating CGI program on Bob’s webserver takes the unique identifier from the URL and looks up the corresponding details in the database. It then creates a PaymentDetails message with the following information:

  • The amount of the order in satoshis and the pubkey script to be paid.

  • A memo containing the list of items ordered, so Charlie knows what he’s paying for. It may also include Charlie’s mailing address so he can double-check it.

  • The time the PaymentDetails message was created plus the time it expires.

  • A URL to which Charlie’s wallet should send its completed transaction.

That PaymentDetails message is put inside a PaymentRequest message. The payment request lets Bob’s server sign the entire Request with the server’s X.509 SSL certificate. (The Payment Protocol has been designed to allow other signing methods in the future.) Bob’s server sends the payment request to Charlie’s wallet in the reply to the HTTP GET.

Charlie’s wallet receives the PaymentRequest message, checks its signature, and then displays the details from the PaymentDetails message to Charlie. Charlie agrees to pay, so the wallet constructs a payment to the pubkey script Bob’s server provided. Unlike a traditional Bitcoin payment, Charlie’s wallet doesn’t necessarily automatically broadcast this payment to the network. Instead, the wallet constructs a Payment message and sends it to the URL provided in the PaymentDetails message as an HTTP POST. Among other things, the Payment message contains:

  • The signed transaction in which Charlie pays Bob.

  • An optional memo Charlie can send to Bob. (There’s no guarantee that Bob will read it.)

  • A refund address (pubkey script) which Bob can pay if he needs to return some or all of Charlie’s satoshis.

Bob’s server receives the Payment message, verifies the transaction pays the requested amount to the address provided, and then broadcasts the transaction to the network. It also replies to the HTTP POSTed Payment message with a PaymentACK message, which includes an optional memo from Bob’s server thanking Charlie for his patronage and providing other information about the order, such as the expected arrival date.

Charlie’s wallet sees the PaymentACK and tells Charlie that the payment has been sent. The PaymentACK doesn’t mean that Bob has verified Charlie’s payment—see the Verifying Payment subsection below—but it does mean that Charlie can go do something else while the transaction gets confirmed. After Bob’s server verifies from the block chain that Charlie’s transaction has been suitably confirmed, it authorizes shipping Charlie’s order.

In the case of a dispute, Charlie can generate a cryptographically-proven receipt out of the various signed or otherwise-proven information.

If a refund needs to be issued, Bob’s server can safely pay the refund-to pubkey script provided by Charlie. (Note: a proposal has been discussed to give refund-to addresses an implicit expiration date so users and software don’t need to worry about payments being sent to addresses which are no longer monitored.) See the Refunds section below for more details.

Verifying Payment Edit | History | Report Issue | Discuss

As explained in the Transactions and Block Chain sections, broadcasting a transaction to the network doesn’t ensure that the receiver gets paid. A malicious spender can create one transaction that pays the receiver and a second one that pays the same input back to himself. Only one of these transactions will be added to the block chain, and nobody can say for sure which one it will be.

Two or more transactions spending the same input are commonly referred to as a double spend.

Once the transaction is included in a block, double spends are impossible without modifying block chain history to replace the transaction, which is quite difficult. Using this system, the Bitcoin protocol can give each of your transactions an updating confidence score based on the number of blocks which would need to be modified to replace a transaction. For each block, the transaction gains one confirmation. Since modifying blocks is quite difficult, higher confirmation scores indicate greater protection.

0 confirmations: The transaction has been broadcast but is still not included in any block. Zero confirmation transactions (unconfirmed transactions) should generally not be trusted without risk analysis. Although miners usually confirm the first transaction they receive, fraudsters may be able to manipulate the network into including their version of a transaction.

1 confirmation: The transaction is included in the latest block and double-spend risk decreases dramatically. Transactions which pay sufficient transaction fees need 10 minutes on average to receive one confirmation. However, the most recent block gets replaced fairly often by accident, so a double spend is still a real possibility.

2 confirmations: The most recent block was chained to the block which includes the transaction. As of March 2014, two block replacements were exceedingly rare, and a two block replacement attack was impractical without expensive mining equipment.

6 confirmations: The network has spent about an hour working to protect the transaction against double spends and the transaction is buried under six blocks. Even a reasonably lucky attacker would require a large percentage of the total network hashing power to replace six blocks. Although this number is somewhat arbitrary, software handling high-value transactions, or otherwise at risk for fraud, should wait for at least six confirmations before treating a payment as accepted.

Bitcoin Core provides several RPCs which can provide your program with the confirmation score for transactions in your wallet or arbitrary transactions. For example, the listunspent RPC provides an array of every satoshi you can spend along with its confirmation score.

Although confirmations provide excellent double-spend protection most of the time, there are at least three cases where double-spend risk analysis can be required:

  1. In the case when the program or its user cannot wait for a confirmation and wants to accept unconfirmed payments.

  2. In the case when the program or its user is accepting high value transactions and cannot wait for at least six confirmations or more.

  3. In the case of an implementation bug or prolonged attack against Bitcoin which makes the system less reliable than expected.

An interesting source of double-spend risk analysis can be acquired by connecting to large numbers of Bitcoin peers to track how transactions and blocks differ from each other. Some third-party APIs can provide you with this type of service.

For example, unconfirmed transactions can be compared among all connected peers to see if any UTXO is used in multiple unconfirmed transactions, indicating a double-spend attempt, in which case the payment can be refused until it is confirmed. Transactions can also be ranked by their transaction fee to estimate the amount of time until they’re added to a block.

Another example could be to detect a fork when multiple peers report differing block header hashes at the same block height. Your program can go into a safe mode if the fork extends for more than two blocks, indicating a possible problem with the block chain. For more details, see the Detecting Forks subsection.

Another good source of double-spend protection can be human intelligence. For example, fraudsters may act differently from legitimate customers, letting savvy merchants manually flag them as high risk. Your program can provide a safe mode which stops automatic payment acceptance on a global or per-customer basis.

Issuing Refunds Edit | History | Report Issue | Discuss

Occasionally receivers using your applications will need to issue refunds. The obvious way to do that, which is very unsafe, is simply to return the satoshis to the pubkey script from which they came. For example:

  • Alice wants to buy a widget from Bob, so Bob gives Alice a price and Bitcoin address.

  • Alice opens her wallet program and sends some satoshis to that address. Her wallet program automatically chooses to spend those satoshis from one of its unspent outputs, an output corresponding to the Bitcoin address mjSk1Ny9spzU2fouzYgLqGUD8U41iR35QN.

  • Bob discovers Alice paid too many satoshis. Being an honest fellow, Bob refunds the extra satoshis to the mjSk… address.

This seems like it should work, but Alice is using a centralized multi-user web wallet which doesn’t give unique addresses to each user, so it has no way to know that Bob’s refund is meant for Alice. Now the refund is a unintentional donation to the company behind the centralized wallet, unless Alice opens a support ticket and proves those satoshis were meant for her.

This leaves receivers only two correct ways to issue refunds:

As discussed in the Payment section, refund_to addresses may come with implicit expiration dates, so you may need to revert to contacting the spender directly if the refund is being issued a long time after the original payment was made.

Disbursing Income (Limiting Forex Risk) Edit | History | Report Issue | Discuss

Many receivers worry that their satoshis will be less valuable in the future than they are now, called foreign exchange (forex) risk. To limit forex risk, many receivers choose to disburse newly-acquired payments soon after they’re received.

If your application provides this business logic, it will need to choose which outputs to spend first. There are a few different algorithms which can lead to different results.

  • A merge avoidance algorithm makes it harder for outsiders looking at block chain data to figure out how many satoshis the receiver has earned, spent, and saved.

  • A last-in-first-out (LIFO) algorithm spends newly acquired satoshis while there’s still double spend risk, possibly pushing that risk on to others. This can be good for the receiver’s balance sheet but possibly bad for their reputation.

  • A first-in-first-out (FIFO) algorithm spends the oldest satoshis first, which can help ensure that the receiver’s payments always confirm, although this has utility only in a few edge cases.

Merge Avoidance Edit | History | Report Issue | Discuss

When a receiver receives satoshis in an output, the spender can track (in a crude way) how the receiver spends those satoshis. But the spender can’t automatically see other satoshis paid to the receiver by other spenders as long as the receiver uses unique addresses for each transaction.

However, if the receiver spends satoshis from two different spenders in the same transaction, each of those spenders can see the other spender’s payment. This is called a merge, and the more a receiver merges outputs, the easier it is for an outsider to track how many satoshis the receiver has earned, spent, and saved.

Merge avoidance means trying to avoid spending unrelated outputs in the same transaction. For persons and businesses which want to keep their transaction data secret from other people, it can be an important strategy.

A crude merge avoidance strategy is to try to always pay with the smallest output you have which is larger than the amount being requested. For example, if you have four outputs holding, respectively, 100, 200, 500, and 900 satoshis, you would pay a bill for 300 satoshis with the 500-satoshi output. This way, as long as you have outputs larger than your bills, you avoid merging.

More advanced merge avoidance strategies largely depend on enhancements to the payment protocol which will allow payers to avoid merging by intelligently distributing their payments among multiple outputs provided by the receiver.

Last In, First Out (LIFO) Edit | History | Report Issue | Discuss

Outputs can be spent as soon as they’re received—even before they’re confirmed. Since recent outputs are at the greatest risk of being double-spent, spending them before older outputs allows the spender to hold on to older confirmed outputs which are much less likely to be double-spent.

There are two closely-related downsides to LIFO:

  • If you spend an output from one unconfirmed transaction in a second transaction, the second transaction becomes invalid if transaction malleability changes the first transaction.

  • If you spend an output from one unconfirmed transaction in a second transaction and the first transaction’s output is successfully double spent to another output, the second transaction becomes invalid.

In either of the above cases, the receiver of the second transaction will see the incoming transaction notification disappear or turn into an error message.

Because LIFO puts the recipient of secondary transactions in as much double-spend risk as the recipient of the primary transaction, they’re best used when the secondary recipient doesn’t care about the risk—such as an exchange or other service which is going to wait for six confirmations whether you spend old outputs or new outputs.

LIFO should not be used when the primary transaction recipient’s reputation might be at stake, such as when paying employees. In these cases, it’s better to wait for transactions to be fully verified (see the Verification subsection above) before using them to make payments.

First In, First Out (FIFO) Edit | History | Report Issue | Discuss

The oldest outputs are the most reliable, as the longer it’s been since they were received, the more blocks would need to be modified to double spend them. However, after just a few blocks, a point of rapidly diminishing returns is reached. The original Bitcoin paper predicts the chance of an attacker being able to modify old blocks, assuming the attacker has 30% of the total network hashing power:

Blocks Chance of successful modification 5 17.73523% 10 4.16605% 15 1.01008% 20 0.24804% 25 0.06132% 30 0.01522% 35 0.00379% 40 0.00095% 45 0.00024% 50 0.00006%

FIFO does have a small advantage when it comes to transaction fees, as older outputs may be eligible for inclusion in the 50,000 bytes set aside for no-fee-required high-priority transactions by miners running the default Bitcoin Core codebase. However, with transaction fees being so low, this is not a significant advantage.

The only practical use of FIFO is by receivers who spend all or most of their income within a few blocks, and who want to reduce the chance of their payments becoming accidentally invalid. For example, a receiver who holds each payment for six confirmations, and then spends 100% of verified payments to vendors and a savings account on a bi-hourly schedule.

Rebilling Recurring Payments Edit | History | Report Issue | Discuss

Automated recurring payments are not possible with decentralized Bitcoin wallets. Even if a wallet supported automatically sending non-reversible payments on a regular schedule, the user would still need to start the program at the appointed time, or leave it running all the time unprotected by encryption.

This means automated recurring Bitcoin payments can only be made from a centralized server which handles satoshis on behalf of its spenders. In practice, receivers who want to set prices in fiat terms must also let the same centralized server choose the appropriate exchange rate.

Non-automated rebilling can be managed by the same mechanism used before credit-card recurring payments became common: contact the spender and ask them to pay again—for example, by sending them a PaymentRequest bitcoin: URI in an HTML email.

In the future, extensions to the payment protocol and new wallet features may allow some wallet programs to manage a list of recurring transactions. The spender will still need to start the program on a regular basis and authorize payment—but it should be easier and more secure for the spender than clicking an emailed invoice, increasing the chance receivers get paid on time.

Operating Modes Edit | History | Report Issue | Discuss

Currently there are two primary methods of validating the block chain as a client: Full nodes and SPV clients. Other methods, such as server-trusting methods, are not discussed as they are not recommended.

Full Node Edit | History | Report Issue | Discuss

The first and most secure model is the one followed by Bitcoin Core, also known as a “thick” or “full chain” client. This security model assures the validity of the block chain by downloading and validating blocks from the genesis block all the way to the most recently discovered block. This is known as using the height of a particular block to verify the client’s view of the network.

For a client to be fooled, an adversary would need to give a complete alternative block chain history that is of greater difficulty than the current “true” chain, which is impossible due to the fact that the longest chain is by definition the true chain. After the suggested six confirmations, the ability to fool the client become intractable, as only a single honest network node is needed to have the complete state of the block chain.

Simplified Payment Verification (SPV) Edit | History | Report Issue | Discuss

An alternative approach detailed in the original Bitcoin paper is a client that only downloads the headers of blocks during the initial syncing process and then requests transactions from full nodes as needed. This scales linearly with the height of the block chain at only 80 bytes per block header, or up to 4.2MB per year, regardless of total block size.

As described in the white paper, the merkle root in the block header along with a merkle branch can prove to the SPV client that the transaction in question is embedded in a block in the block chain. This does not guarantee validity of the transactions that are embedded. Instead it demonstrates the amount of work required to perform a double-spend attack.

The block’s depth in the block chain corresponds to the cumulative difficulty that has been performed to build on top of that particular block. The SPV client knows the merkle root and associated transaction information, and requests the respective merkle branch from a full node. Once the merkle branch has been retrieved, proving the existence of the transaction in the block, the SPV client can then look to block depth as a proxy for transaction validity and security. The cost of an attack on a user by a malicious node who inserts an invalid transaction grows with the cumulative difficulty built on top of that block, since the malicious node alone will be mining this forged chain.

Potential SPV Weaknesses Edit | History | Report Issue | Discuss

If implemented naively, an SPV client has a few important weaknesses.

First, while the SPV client can not be easily fooled into thinking a transaction is in a block when it is not, the reverse is not true. A full node can simply lie by omission, leading an SPV client to believe a transaction has not occurred. This can be considered a form of Denial of Service. One mitigation strategy is to connect to a number of full nodes, and send the requests to each node. However this can be defeated by network partitioning or Sybil attacks, since identities are essentially free, and can be bandwidth intensive. Care must be taken to ensure the client is not cut off from honest nodes.

Second, the SPV client only requests transactions from full nodes corresponding to keys it owns. If the SPV client downloads all blocks and then discards unneeded ones, this can be extremely bandwidth intensive. If they simply ask full nodes for blocks with specific transactions, this allows full nodes a complete view of the public addresses that correspond to the user. This is a large privacy leak, and allows for tactics such as denial of service for clients, users, or addresses that are disfavored by those running full nodes, as well as trivial linking of funds. A client could simply spam many fake transaction requests, but this creates a large strain on the SPV client, and can end up defeating the purpose of thin clients altogether.

To mitigate the latter issue, Bloom filters have been implemented as a method of obfuscation and compression of block data requests.

Bloom Filters Edit | History | Report Issue | Discuss

A Bloom filter is a space-efficient probabilistic data structure that is used to test membership of an element. The data structure achieves great data compression at the expense of a prescribed false positive rate.

A Bloom filter starts out as an array of n bits all set to 0. A set of k random hash functions are chosen, each of which output a single integer between the range of 1 and n.

When adding an element to the Bloom filter, the element is hashed k times separately, and for each of the k outputs, the corresponding Bloom filter bit at that index is set to 1.

Querying of the Bloom filter is done by using the same hash functions as before. If all k bits accessed in the bloom filter are set to 1, this demonstrates with high probability that the element lies in the set. Clearly, the k indices could have been set to 1 by the addition of a combination of other elements in the domain, but the parameters allow the user to choose the acceptable false positive rate.

Removal of elements can only be done by scrapping the bloom filter and re-creating it from scratch.

Application Of Bloom Filters Edit | History | Report Issue | Discuss

Rather than viewing the false positive rates as a liability, it is used to create a tunable parameter that represents the desired privacy level and bandwidth trade-off. A SPV client creates their Bloom filter and sends it to a full node using the message filterload, which sets the filter for which transactions are desired. The command filteradd allows addition of desired data to the filter without needing to send a totally new Bloom filter, and filterclear allows the connection to revert to standard block discovery mechanisms. If the filter has been loaded, then full nodes will send a modified form of blocks, called a merkle block. The merkle block is simply the block header with the merkle branch associated with the set Bloom filter.

An SPV client can not only add transactions as elements to the filter, but also public keys, data from signature scripts and pubkey scripts, and more. This enables P2SH transaction finding.

If a user is more privacy-conscious, he can set the Bloom filter to include more false positives, at the expense of extra bandwidth used for transaction discovery. If a user is on a tight bandwidth budget, he can set the false-positive rate to low, knowing that this will allow full nodes a clear view of what transactions are associated with his client.

Resources: BitcoinJ, a Java implementation of Bitcoin that is based on the SPV security model and Bloom filters. Used in most Android wallets.

Bloom filters were standardized for use via BIP37. Review the BIP for implementation details.

Future Proposals Edit | History | Report Issue | Discuss

There are future proposals such as Unspent Transaction Output (UTXO) commitments in the block chain to find a more satisfactory middle-ground for clients between needing a complete copy of the block chain, or trusting that a majority of your connected peers are not lying. UTXO commitments would enable a very secure client using a finite amount of storage using a data structure that is authenticated in the block chain. These type of proposals are, however, in very early stages, and will require soft forks in the network.

Until these types of operating modes are implemented, modes should be chosen based on the likely threat model, computing and bandwidth constraints, and liability in bitcoin value.

Resources: Original Thread on UTXO Commitments, Authenticated Prefix Trees BIP Proposal

P2P Network Edit | History | Report Issue | Discuss

The Bitcoin network protocol allows full nodes (peers) to collaboratively maintain a peer-to-peer network for block and transaction exchange. Many SPV clients also use this protocol to connect to full nodes.

Consensus rules do not cover networking, so Bitcoin programs may use alternative networks and protocols, such as the high-speed block relay network used by some miners and the dedicated transaction information servers used by some wallets that provide SPV-level security.

To provide practical examples of the Bitcoin peer-to-peer network, this section uses Bitcoin Core as a representative full node and BitcoinJ as a representative SPV client. Both programs are flexible, so only default behavior is described. Also, for privacy, actual IP addresses in the example output below have been replaced with RFC5737 reserved IP addresses.

Peer Discovery Edit | History | Report Issue | Discuss

When started for the first time, programs don’t know the IP addresses of any active full nodes. In order to discover some IP addresses, they query one or more DNS names (called DNS seeds) hardcoded into Bitcoin Core and BitcoinJ. The response to the lookup should include one or more DNS A records with the IP addresses of full nodes that may accept new incoming connections. For example, using the Unix dig command:

;; QUESTION SECTION: ;seed.bitcoin.sipa.be. IN A ;; ANSWER SECTION: seed.bitcoin.sipa.be. 60 IN A seed.bitcoin.sipa.be. 60 IN A seed.bitcoin.sipa.be. 60 IN A [...]

The DNS seeds are maintained by Bitcoin community members: some of them provide dynamic DNS seed servers which automatically get IP addresses of active nodes by scanning the network; others provide static DNS seeds that are updated manually and are more likely to provide IP addresses for inactive nodes. In either case, nodes are added to the DNS seed if they run on the default Bitcoin ports of 8333 for mainnet or 18333 for testnet.

DNS seed results are not authenticated and a malicious seed operator or network man-in-the-middle attacker can return only IP addresses of nodes controlled by the attacker, isolating a program on the attacker’s own network and allowing the attacker to feed it bogus transactions and blocks. For this reason, programs should not rely on DNS seeds exclusively.

Once a program has connected to the network, its peers can begin to send it addr (address) messages with the IP addresses and port numbers of other peers on the network, providing a fully decentralized method of peer discovery. Bitcoin Core keeps a record of known peers in a persistent on-disk database which usually allows it to connect directly to those peers on subsequent startups without having to use DNS seeds.

However, peers often leave the network or change IP addresses, so programs may need to make several different connection attempts at startup before a successful connection is made. This can add a significant delay to the amount of time it takes to connect to the network, forcing a user to wait before sending a transaction or checking the status of payment.

To avoid this possible delay, BitcoinJ always uses dynamic DNS seeds to get IP addresses for nodes believed to be currently active. Bitcoin Core also tries to strike a balance between minimizing delays and avoiding unnecessary DNS seed use: if Bitcoin Core has entries in its peer database, it spends up to 11 seconds attempting to connect to at least one of them before falling back to seeds; if a connection is made within that time, it does not query any seeds.

Both Bitcoin Core and BitcoinJ also include a hardcoded list of IP addresses and port numbers to several dozen nodes which were active around the time that particular version of the software was first released. Bitcoin Core will start attempting to connect to these nodes if none of the DNS seed servers have responded to a query within 60 seconds, providing an automatic fallback option.

As a manual fallback option, Bitcoin Core also provides several command-line connection options, including the ability to get a list of peers from a specific node by IP address, or to make a persistent connection to a specific node by IP address. See the -help text for details. BitcoinJ can be programmed to do the same thing.

Resources: Bitcoin Seeder, the program run by several of the seeds used by Bitcoin Core and BitcoinJ. The Bitcoin Core DNS Seed Policy. The hardcoded list of IP addresses used by Bitcoin Core and BitcoinJ is generated using the makeseeds script.

Connecting To Peers Edit | History | Report Issue | Discuss

Connecting to a peer is done by sending a version message, which contains your version number, block, and current time to the remote node. The remote node responds with its own version message. Then both nodes send a verack message to the other node to indicate the connection has been established.

Once connected, the client can send to the remote node getaddr and addr messages to gather additional peers.

In order to maintain a connection with a peer, nodes by default will send a message to peers before 30 minutes of inactivity. If 90 minutes pass without a message being received by a peer, the client will assume that connection has closed.

Block Broadcasting Edit | History | Report Issue | Discuss

At the start of a connection with a peer, both nodes send getblocks messages containing the hash of the latest known block. If a peer believes they have newer blocks or a longer chain, that peer will send an inv message which includes a list of up to 500 hashes of newer blocks, stating that it has the longer chain. The receiving node would then request these blocks using the command getdata, and the remote peer would reply via block messages. After all 500 blocks have been processed, the node can request another set with getblocks, until the node is caught up with the network. Blocks are only accepted when validated by the receiving node.

New blocks are also discovered as miners publish their found blocks, and these messages are propagated in a similar manner. Through previously established connections, an inv message is sent with the new block hashed, and the receiving node requests the block via the getdata message.

Transaction Broadcasting Edit | History | Report Issue | Discuss

In order to send a transaction to a peer, an inv message is sent. If a getdata response message is received, the transaction is sent using tx. The peer receiving this transaction also forwards the transaction in the same manner, given that it is a valid transaction.

Memory Pool Edit | History | Report Issue | Discuss

Full peers may keep track of unconfirmed transactions which are eligible to be included in the next block. This is essential for miners who will actually mine some or all of those transactions, but it’s also useful for any peer who wants to keep track of unconfirmed transactions, such as peers serving unconfirmed transaction information to SPV clients.

Because unconfirmed transactions have no permanent status in Bitcoin, Bitcoin Core stores them in non-persistent memory, calling them a memory pool or mempool. When a peer shuts down, its memory pool is lost except for any transactions stored by its wallet. This means that never-mined unconfirmed transactions tend to slowly disappear from the network as peers restart or as they purge some transactions to make room in memory for others.

Transactions which are mined into blocks that are later orphaned may be added back into the memory pool. These re-added transactions may be re-removed from the pool almost immediately if the replacement blocks include them. This is the case in Bitcoin Core, which removes orphaned blocks from the chain one by one, starting with the tip (highest block). As each block is removed, its transactions are added back to the memory pool. After all of the orphaned blocks are removed, the replacement blocks are added to the chain one by one, ending with the new tip. As each block is added, any transactions it confirms are removed from the memory pool.

SPV clients don’t have a memory pool for the same reason they don’t relay transactions. They can’t independently verify that a transaction hasn’t yet been included in a block and that it only spends UTXOs, so they can’t know which transactions are eligible to be included in the next block.

Misbehaving Nodes Edit | History | Report Issue | Discuss

Take note that for both types of broadcasting, mechanisms are in place to punish misbehaving peers who take up bandwidth and computing resources by sending false information. If a peer gets a banscore above the -banscore=<n> threshold, he will be banned for the number of seconds defined by -bantime=<n>, which is 86,400 by default (24 hours).

Alerts Edit | History | Report Issue | Discuss

In case of a bug or attack, the Bitcoin Core developers provide a Bitcoin alert service with an RSS feed and users of Bitcoin Core can check the error field of the getinfo RPC results to get currently active alerts for their specific version of Bitcoin Core.

These messages are aggressively broadcast using the alert message, being sent to each peer upon connect for the duration of the alert.

These messages are signed by a specific ECDSA private key that only a small number of developers control.

Resource: More details about the structure of messages and a complete list of message types can be found at the Protocol Specification page of the Bitcoin Wiki.

Mining Edit | History | Report Issue | Discuss

Mining adds new blocks to the block chain, making transaction history hard to modify. Mining today takes on two forms:

  • Solo mining, where the miner attempts to generate new blocks on his own, with the proceeds from the block reward and transaction fees going entirely to himself, allowing him to receive large payments with a higher variance (longer time between payments)

  • Pooled mining, where the miner pools resources with other miners to find blocks more often, with the proceeds being shared among the pool miners in rough correlation to the amount of hashing power they each contributed, allowing the miner to receive small payments with a lower variance (shorter time between payments).

Solo Mining Edit | History | Report Issue | Discuss

As illustrated below, solo miners typically use bitcoind to get new transactions from the network. Their mining software periodically polls bitcoind for new transactions using the getblocktemplate RPC, which provides the list of new transactions plus the public key to which the coinbase transaction should be sent.

The mining software constructs a block using the template (described below) and creates a block header. It then sends the 80-byte block header to its mining hardware (an ASIC) along with a target threshold (difficulty setting). The mining hardware iterates through every possible value for the block header nonce and generates the corresponding hash.

If none of the hashes are below the threshold, the mining hardware gets an updated block header with a new merkle root from the mining software; this new block header is created by adding extra nonce data to the coinbase field of the coinbase transaction.

On the other hand, if a hash is found below the target threshold, the mining hardware returns the block header with the successful nonce to the mining software. The mining software combines the header with the block and sends the completed block to bitcoind to be broadcast to the network for addition to the block chain.

Pool Mining Edit | History | Report Issue | Discuss

Pool miners follow a similar workflow, illustrated below, which allows mining pool operators to pay miners based on their share of the work done. The mining pool gets new transactions from the network using bitcoind. Using one of the methods discussed later, each miner’s mining software connects to the pool and requests the information it needs to construct block headers.

In pooled mining, the mining pool sets the target threshold a few orders of magnitude higher (less difficult) than the network difficulty. This causes the mining hardware to return many block headers which don’t hash to a value eligible for inclusion on the block chain but which do hash below the pool’s target, proving (on average) that the miner checked a percentage of the possible hash values.

The miner then sends to the pool a copy of the information the pool needs to validate that the header will hash below the target and that the the block of transactions referred to by the header merkle root field is valid for the pool’s purposes. (This usually means that the coinbase transaction must pay the pool.)

The information the miner sends to the pool is called a share because it proves the miner did a share of the work. By chance, some shares the pool receives will also be below the network target—the mining pool sends these to the network to be added to the block chain.

The block reward and transaction fees that come from mining that block are paid to the mining pool. The mining pool pays out a portion of these proceeds to individual miners based on how many shares they generated. For example, if the mining pool’s target threshold is 100 times lower than the network target threshold, 100 shares will need to be generated on average to create a successful block, so the mining pool can pay 1/100th of its payout for each share received. Different mining pools use different reward distribution systems based on this basic share system.

Block Prototypes Edit | History | Report Issue | Discuss

In both solo and pool mining, the mining software needs to get the information necessary to construct block headers. This subsection describes, in a linear way, how that information is transmitted and used. However, in actual implementations, parallel threads and queuing are used to keep ASIC hashers working at maximum capacity,

getwork RPC Edit | History | Report Issue | Discuss

The simplest and earliest method was the now-deprecated Bitcoin Core getwork RPC, which constructs a header for the miner directly. Since a header only contains a single 4-byte nonce good for about 4 gigahashes, many modern miners need to make dozens or hundreds of getwork requests a second. Solo miners may still use getwork, but most pools today discourage or disallow its use.

getblocktemplate RPC Edit | History | Report Issue | Discuss

An improved method is the Bitcoin Core getblocktemplate RPC. This provides the mining software with much more information:

  1. The information necessary to construct a coinbase transaction paying the pool or the solo miner’s bitcoind wallet.

  2. A complete dump of the transactions bitcoind or the mining pool suggests including in the block, allowing the mining software to inspect the transactions, optionally add additional transactions, and optionally remove non-required transactions.

  3. Other information necessary to construct a block header for the next block: the block version, previous block hash, and bits (target).

  4. The mining pool’s current target threshold for accepting shares. (For solo miners, this is the network target.)

Using the transactions received, the mining software adds a nonce to the coinbase extra nonce field and then converts all the transactions into a merkle tree to derive a merkle root it can use in a block header. Whenever the extra nonce field needs to be changed, the mining software rebuilds the necessary parts of the merkle tree and updates the time and merkle root fields in the block header.

Like all bitcoind RPCs, getblocktemplate is sent over HTTP. To ensure they get the most recent work, most miners use HTTP longpoll to leave a getblocktemplate request open at all times. This allows the mining pool to push a new getblocktemplate to the miner as soon as any miner on the peer-to-peer network publishes a new block or the pool wants to send more transactions to the mining software.

Stratum Edit | History | Report Issue | Discuss

A widely used alternative to getblocktemplate is the Stratum mining protocol. Stratum focuses on giving miners the minimal information they need to construct block headers on their own:

  1. The information necessary to construct a coinbase transaction paying the pool.

  2. The parts of the merkle tree which need to be re-hashed to create a new merkle root when the coinbase transaction is updated with a new extra nonce. The other parts of the merkle tree, if any, are not sent, effectively limiting the amount of data which needs to be sent to (at most) about a kilobyte at current transaction volume.

  3. All of the other non-merkle root information necessary to construct a block header for the next block.

  4. The mining pool’s current target threshold for accepting shares.

Using the coinbase transaction received, the mining software adds a nonce to the coinbase extra nonce field, hashes the coinbase transaction, and adds the hash to the received parts of the merkle tree. The tree is hashed as necessary to create a merkle root, which is added to the block header information received. Whenever the extra nonce field needs to be changed, the mining software updates and re-hashes the coinbase transaction, rebuilds the merkle root, and updates the header merkle root field.

Unlike getblocktemplate, miners using Stratum cannot inspect or add transactions to the block they’re currently mining. Also unlike getblocktemplate, the Stratum protocol uses a two-way TCP socket directly, so miners don’t need to use HTTP longpoll to ensure they receive immediate updates from mining pools when a new block is broadcast to the peer-to-peer network.

Resources: The GPLv3 BFGMiner mining software and AGPLv3 Eloipool mining pool software are widely-used among miners and pools. The libblkmaker C library and python-blkmaker library, both MIT licensed, can interpret GetBlockTemplate for your programs.

Announcing a redesign of the Django websites

16 December 2014 - 2:00pm
Posted by Jannis Leidel and Russell Keith-Magee on December 15, 2014

The Django project is excited to announce that after many years, we're launching a redesign of our primary website, our documentation site and our issue tracker.

Django's website has been largely unchanged since the project was launched back in 2005, so you can imagine how excited we are to update it. The original design was created by Wilson Miner while he was working at the Lawrence Journal-World, the newspaper at which Django was created.. Wilson's design has held up incredibly well over the years, but new design aesthetics, and technologies such as mobile devices, web fonts, HTML5 and CSS3 have drastically changed the way websites are built.

The old design was also focused on introducing a new web framework to the world. Django is now a well-established framework, so the website has a much broader audience -- not just new Django users, but established users, managers, and people new to programming. This redesign also allows us to shine a spotlight on areas of the community that have historically been hidden, such as the Django Software Foundation (DSF), the community of projects that support Django developers (such as people.djangoproject.com and djangopackages.com), and the various educational and consulting resources that exist in our community.

This redesign is the result of multiple attempts and the collaboration of a number of groups and individuals. Work on the redesign started in 2010. Initially, a number of people (including Christian Metts and Julian Phalip) tried to produce a new design as individual efforts; however, these efforts stalled due to a lack of momentum. In 2012, the DSF developed a design brief and put out a call for a volunteer team to redesign the site. The DSF received a number of applicants and selected interactive agency Threespot to complete the design task. For a number of reasons (almost entirely the DSF's fault), this design got most of the way to completion, but not 100% complete.

Earlier this year, Andrew McCarthy took on the task of completing the design work including a style guide for future expansions. The design was then handed over to the DSF's website working group to convert that website into working code.

Since everyone is a volunteer on this team we'd like to name them individually: Adrian Holovaty, Audrey Roy, Aymeric Augustin, Baptiste Mispelon, Daniel Roy Greenfeld, Elena Williams, Jannis Leidel, Ola Sitarska, Ola Sendecka, Russell Keith-Magee, Tomek Paczkowski and Trey Hunner. One of the DSF's current fellows Tim Graham also helped by finding bugs and reviewing tickets. Of course we couldn't have done it without the backing of the DSF board of directors over the years.

Now we'd like to invite you to share in the result of our efforts and help us making it even better. Please test-drive the site and let us know what you think.

If you find a bug -- which we're sure some will -- open a ticket on the website's issue tracker. If you want to contribute directly to the site's code please don't hesitate to join us on Freenode in the channel #django-websites.

We also wouldn't mind if you'd tell us about your experience on Twitter using the hashtag #10YearsLater, or by tweeting at @djangoproject.

So now, without further ado, please check out the new site djangoproject.com, the documentation docs.djangoproject.com and our issue tracker code.djangoproject.com.

That's all for now. Happy coding, everyone!

...and we'll see you all again in 2023 when we launch our next redesign :-)

Back to Top

Millennium Falcon's Hyperdrive SFX Demonstrated by Sound Designer Ben Burtt

16 December 2014 - 2:00pm

Ratings have been disabled for this video.

Rating is available when the video has been rented.

This feature is not available right now. Please try again later.

Published on Dec 11, 2014

Also watch: "Fantastic 10-minute SFX documentary w/ Sound Designer Ben Burtt on Star Wars, Indy, E.T. & more!" https://www.youtube.com/watch?v=mWiuK...
The 'comical' sounds that make the Millennium Falcon's hyperdrive malfunction, as demonstrated by Star Wars sound designer Ben Burtt in this 1980 clip.

Show more Show less

The most popular web sites every year since 1996

16 December 2014 - 2:00pm
By Philip Bump December 15 at 10:56 AM

Our goal is not to confuse or alarm you, but we must, as agents of the news media, speak the truth. And so we say, with all due solemnity, that if it were 16 years ago, you would right now be reading this article at Excite.com.

That’s really one of the best case scenarios. You could also be at Infoseek, or Tripod (what) or Xoom (what?). Because the web in December 1998 was a much different, much lamer place.

We like to think of sites like Google, Facebook and Amazon as immutable — parts of the web as it exists now and has always existed. This is not the case, however. Sixteen years ago, only Amazon (the CEO of which owns The Post) was a popular site; it was the 16th most popular site on the web according to Media Metrix (which later was absorbed into comScore). Infoseek and Hotbot were more popular than Google (which, that December, looked like this) and Facebook (which didn’t exist).

Sites like AOL and Yahoo did exist — and were popular. But the easiest way to make that point is to share with you this graphic, which shows the 20 most popular sites in December of each year, according to comScore. More interestingly, what it shows is when certain sites became and then stopped being popular.

We will be the first to admit that this is too small to read (though you can click on it, which improves things). So we broke it out into smaller clusters.

1996 to 2000

This section of the graph displays the heart of the original dot-com boom, the era in which investors figured that the world couldn’t resist groceries delivered to ones home or pet supplies ordered online. How wrong they were!

The first year here is 1996, when the web was … young. Several of the top 20 sites were college sites, thanks to colleges having invested early in the internet. AOL and Yahoo were there too, as they have been ever since. Mostly, however, the list is garbage nonsense like “GNN” and “Teleport,” which we don’t even know what they are.

Notice the ascent of Excite and CNET. They’ll be interesting in the next part of the graph.

2000 to 2004

Excite plummets; CNET is bouncing around in the middle. (Stay tuned!)

EBay, you’ll notice, is in the top five, and Amazon is holding strong. Those paying close attention to the evolution of the media industry are currently nodding and saying, “Hmmm” suggestively.

What’s really interesting, though, is that top tier: Microsoft/MSN. Yahoo, after absorbing Geocities. And, out of the blue, Google — which replaced other search engines (Altavista and Webcrawler, for example) as the go-to for web users.

This is the natural point at which we note that comScore includes every site associated with a company. So Go and ESPN and so on are part of “Disney.” That holds for the rest of the listing.

2004 to 2009

This is the point at which things start to stabilize. In 2007, Facebook appears, thanks to its having expanded beyond only accommodating college students.

But the top tier, which includes Time Warner (after it absorbed AOL) and then AOL (after that collapsed) contains companies that largely occupy the upper levels today.

There are a lot of other ups-and-downs here. Viacom, Walmart, and Ask Jeeves/Ask are ones to keep an eye on.

EBay is starting to fade.

2009 to 2013

Welcome to the modern era. Remember, this is data from comScore (links to the individual years are below). So other metrics may have other rankings.

We say that mostly because we have never heard of “Glam Media,” apparently the seventh biggest web network last year. Sure, why not?

But the top five continues to consist of the same sites: Google, Yahoo, Microsoft, Facebook and AOL. The interesting maneuvers are lower on the chart: Apple flitting around, and the appearance of LinkedIn, which we all hope doesn’t rise any higher. (Opinions expressed here are not necessarily those of people who are looking for work and therefore updating their LinkedIn profiles.)


What’s interesting about this data is that, despite its reflecting December (with noted exceptions), it’s pretty static. Walmart and Target don’t appear to do better because it’s the holiday season; Wikipedia (and Wikimedia, its umbrella organization) does fine despite school being out.

So what happens this year? We shall see. We do have a prediction though: Xoom is set for a comeback, even though it is now a money-sending site (?) instead of an “instant software source.”

You heard it here first.

Data sources: 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013.

Philip Bump writes about politics for The Fix. He is based in New York City.

HireArt (YC W12) is looking for a full-stack engineer

16 December 2014 - 2:00pm
HireArt is looking for a full-stack engineer to join our product team. We're a small, nimble team of three engineers and a designer, looking for another engineer to join us.

We are trying to solve one of the biggest problems in hiring: how to effectively measure a person's fit for a job. We want to do it in a way that is intuitive and transparent for both employers and job-seekers, and which scales to support organizations of any size. We believe that the right solution requires both algorithms and human judgement, and we are developing technologies which incorporate both. We believe that by building a product around new standards for assessment in recruiting, we can dramatically improve the accuracy of hiring decisions, as well as the job-seeking process.

We've already helped build teams at hundreds of companies, including, Facebook, Google, and airbnb, and helped hundreds of thousands of applicants in their job searches. You would help build pieces of the employer- and job seeker-facing apps, as well as internal tools we use in our back-end operation. You would also contribute to our R&D effort, working with domain experts to incorporate their knowledge and experience into our web application.

Some of our current projects include: - a data model for assessment that includes human- and computer-generated input to render decisions about job applicants - testing our assessment model using data about hiring and retention - optimizing and automating applicant sourcing to yield high-quality candidates - creating actionable feedback for job applicants - integration with applicant tracking systems

Ideally you: - are interested in the hiring process and excited about improving it - can communicate clearly and precisely about technology and product - are comfortable in the full stack, and very strong in a few areas - have at least one year of Rails experience - are located in or willing to move to NYC

Our technology stack includes Rails, Postgres, Git, and Heroku. We each own specific product verticals, and enlist each other’s help to get them shipped. We like to have thoughtful discussions together on a range of topics, from software architecture to company strategy. We make sure every voice is heard and every opinion is considered.

Please introduce yourself to developers@hireart.com (with a resume or GitHub url, if you have it) to get started.

BPG Image Comparison

16 December 2014 - 2:00am
BPG Image Comparison Pont de Québec at NightMercado dos LavradoresVallée de Colca77 Bombay StreetMachu PicchuAbandoned FactoryAdventure with the WindmillsAir Force Academy ChapelBas-reliefBodhi BaumBuenos AiresCamelCatedral de ToledoCitroenClaudetteClovisfestEaglefairyEndeavor and ColumbiaErfurt CathedralBrewer's BlackbirdBallet ExerciseLe VibronFruitsAvenchesIrish MannequinIsle of SkyeCologne CathedralKrøyerStadiumMoscowMrioNan Lian GardenNymphProductionVintage CarSoccer PlayersDirt RoadMascotSaint CatherineSkingSouthern PacificSteinwaySwallowtailFort de RoppeTennisKavicaOld BookWashington MonumentWaterfallZoo Bird Head BPG-x265MozjpegJP2K-KakaduWebPOriginalLargeMediumSmallTiny --- vs --- BPG-x265MozjpegJP2K-KakaduWebPOriginalLargeMediumSmallTiny

bpgenc-0.9.1 -m 8, with x265-1.4+189 --aq-mode 1 --deblock -1:-1 --crf $Q

mozjpeg-3.0 -dc-scan-opt 2 -quality $Q

kdu_compress-7.4 -quiet -no_info -full -precise -slope $Q

cwebp-0.4.2 -m 6 -pass 10 -size $Q

Medium images were first encoded with bpgenc at 19 CRF filesizes.
Medium is 60% of Large. Small is 60% of Medium. Tiny is 60% of Small. Everything else was matched to +/- 5% filesize.

BPG, JPEG 2000, and WebP decoded in javascript when needed.

This page is based on Xiph.org's Daala comparison page. A list of sources for the images can be found in this text file.

Good News You May Have Missed in 2014

16 December 2014 - 2:00am

Looking Up

Good News You May Have Missed in 2014

ByBill Gates on December 15, 2014

Looking Up

I ended 2013 by compiling something slightly unusual: a list of some of the good news you might have missed. I thought it was a pretty good note to end the year on, and people seemed to like reading about some of the ways the world is becoming a better place. This year, I thought I’d do it again.

Of course, we can’t ignore the fact that it’s been a turbulent year, in the United States and many other countries. But it’s worth taking a moment to celebrate some of the good news too. More children are surviving than ever before. We’re making progress against some of the world’s deadliest diseases. These are some of the most fundamental ways to measure the world’s progress—and by that measure, 2014 was definitely another good year.


More Fifth Birthdays Than Ever Before

To me, one of the best ways to measure progress is to look at how many children are dying of preventable causes. And today, more kids are living to see their fifth birthday than ever before. This year, for at least the 42nd year in a row, the child mortality rate has fallen. And it’s not just moving in the right direction—it’s falling faster than anyone expected. The Economist ran a great article about this in September, where it estimated that just since 2001, the world has saved 13.6 million children’s lives. It’s hard to think of a better sign the world is improving.


We Hit a Big Milestone in Fighting AIDS

The world has done an impressive job of providing treatment to people living with HIV. But for years we were falling behind, because for all the people who started getting treatment, even more would become infected. Not anymore, though. New data released this month show that 2013 was the first year when more people started getting treatment than became infected with HIV. Why does that matter? Because treating people not only keeps them alive, it also dramatically reduces the odds that they will pass the virus on to anyone else. As the epidemiologists say, we can start to bend the curve of the disease. We still have a long way to go before we can declare the end of AIDS, but this is a big milestone.


Rotavirus Vaccine is Reaching More Kids Than Ever

When I read an article in the late 1990s that mentioned a diarrheal disease called rotavirus killed hundreds of thousands of kids a year, I couldn’t believe something I’d never even heard of was killing that many children. But rotavirus doesn’t get much press because it’s almost never deadly in rich countries—and the world tends to ignore diseases that only affect the world’s poorest people. In many ways, rotavirus was a catalyst for my commitment to global health—in fact, one of our foundation’s first grants supported efforts against rotavirus. Since then, the number of kids dying from this illness has been cut nearly in half thanks to a cheap and effective vaccine. And today, that vaccine is reaching more kids than ever before. For example in India, where rota kills nearly 80,000 children a year, the government decided this year to deliver the vaccine for free to poor children. And manufacturers there are working on a more affordable vaccine that could reach even more children in the coming years.


A Tuberculosis Breakthrough—Finally

The world is way overdue for a better tuberculosis treatment. TB is one of the world’s leading causes of death, and our existing treatments are inadequate—especially for drug resistant forms of the disease. But efforts to improve them have been stalled for decades. So it’s a big deal that earlier this year, scientists announced that a new TB treatment regimen has proven effective in early-phase research. From here, the drug regimen goes on to a large clinical trial to confirm the results. If this new treatment regimen pans out, it could dramatically reduce the time it takes to cure drug-resistant TB and save poor countries billions of dollars in health-care costs.


Nigeria’s Fight Against Polio Helped Its Fight Against Ebola

A lot of the media coverage about Nigeria this year focused on two things: Ebola and terrorism. Both are frightening, and they masked the fact that from a global health perspective, Nigeria actually had a pretty good year. Although it’s one of only three countries that have never been free from polio (Pakistan and Afghanistan are the other two), I don’t think it will be on that list for long. Nigeria has reported only six cases of polio this year, compared to more than 50 last year. What’s more, the infrastructure Nigeria has built to fight polio actually made it easier for them to swiftly contain Ebola. The fact that Nigeria is now Ebola free is a great example of how doing the work to fight things like fighting polio also leaves countries better prepared to deal with outbreaks of other diseases.


Looking Ahead

One more thing: this January, Melinda and I will publish our annual letter. This year, we’re looking ahead to 2030. We’ll be writing about a few areas—from health to farming and banking—where life will really change, especially for people in some of the world’s poorest places. (Spoiler alert: we think there’s a lot more progress ahead.) If you’d like to get an e-mail notice when the letter is out, you can sign up below.

How You Know

16 December 2014 - 2:00am

December 2014

I've read Villehardouin's chronicle of the Fourth Crusade at least two times, maybe three. And yet if I had to write down everything I remember from it, I doubt it would amount to much more than a page. Multiply this times several hundred, and I get an uneasy feeling when I look at my bookshelves. What use is it to read all these books if I remember so little from them?

A few months ago, as I was reading Constance Reid's excellent biography of Hilbert, I figured out if not the answer to this question, at least something that made me feel better about it. She writes:

Hilbert had no patience with mathematical lectures which filled the students with facts but did not teach them how to frame a problem and solve it. He often used to tell them that "a perfect formulation of a problem is already half its solution."

That has always seemed to me an important point, and I was even more convinced of it after hearing it confirmed by Hilbert.

But how had I come to believe in this idea in the first place? A combination of my own experience and other things I'd read. None of which I could at that moment remember! And eventually I'd forget that Hilbert had confirmed it too. But my increased belief in the importance of this idea would remain something I'd learned from this book, even after I'd forgotten I'd learned it.

Reading and experience train your model of the world. And even if you forget the experience or what you read, its effect on your model of the world persists. Your mind is like a compiled program you've lost the source of. It works, but you don't know why.

The place to look for what I learned from Villehardouin's chronicle is not what I remember from it, but my mental models of the crusades, Venice, medieval culture, seige warfare, and so on. Which doesn't mean I couldn't have read more attentively, but at least the harvest of reading is not so miserably small as it might seem.

This is one of those things that seem obvious in retrospect. But it was a surprise to me and presumably would be to anyone else who felt uneasy about (apparently) forgetting so much they'd read.

Realizing it does more than make you feel a little better about forgetting, though. There are specific implications.

For example, reading and experience are usually "compiled" at the time they happen, using the state of your brain at that time. The same book would get compiled differently at different points in your life. Which means it is very much worth reading important books multiple times. I always used to feel some misgivings about rereading books. I unconsciously lumped reading together with work like carpentry, where having to do something again is a sign you did it wrong the first time. Whereas now the phrase "already read" seems almost ill-formed.

Intriguingly, this implication isn't limited to books. Technology will increasingly make it possible to relive our experiences. When people do that today it's usually to enjoy them again (e.g. when looking at pictures of a trip) or to find the origin of some bug in their compiled code (e.g. when Stephen Fry succeeded in remembering the childhood trauma that prevented him from singing). But as technologies for recording and playing back your life improve, it may become common for people to relive experiences without any goal in mind, simply to learn from them again as one might when rereading a book.

Eventually we may be able not just to play back experiences but also to index and even edit them. So although not knowing how you know things may seem part of being human, it may not be.

Thanks to Sam Altman, Jessica Livingston, and Robert Morris for reading drafts of this.

New Zealand’s crusade to rid itself of mammals

16 December 2014 - 2:00am
Rats and other invasive mammals are destroying New Zealand’s native fauna. A quarter of native birds are extinct. The kiwi is threatened. What can be done? “Conservation is all about killing things,” a volunteer coördinator said. Credit Photograph by Stephen Dupont

In the days—perhaps weeks—it had spent in the trap, the stoat had lost most of its fur, so it looked as if it had been flayed. Its exposed skin was the deep, dull purple of a bruise, and it was coated in an oily sheen, like a sausage. Stoat traps are often baited with eggs, and this one contained an empty shell. Kevin Adshead, who had set the trap, poked at the stoat with a screwdriver. It writhed and squirmed, as if attempting to rise from the dead. Then it disgorged a column of maggots.

“Look at those teeth,” Adshead said, pointing with his screwdriver at the decomposing snout.

Adshead, who is sixty-four, lives about an hour north of Auckland. He and his wife, Gill, own a thirty-five-hundred-acre farm, where for many years they raised cows and sheep. About a decade ago, they decided they’d had enough of farming and left to do volunteer work in the Solomon Islands. When they returned, they began to look at the place differently. They noticed that many of the trees on the property, which should have been producing cascades of red flowers around Christmastime, instead were stripped bare. That was the work of brushtail possums. To save the trees, the Adsheads decided to eliminate the possums, a process that involved dosing them with cyanide.

One thing led to another, and soon the Adsheads were also going after rats. With them, the preferred poison is an anticoagulant that causes internal hemorrhaging. Next came the stoats, or, as Americans would say, short-tailed weasels. To dispatch these, the Adsheads lined their farm with powerful traps, known as DOC 200s, which feature spring-controlled kill bars. DOC 200s are also helpful against ferrets, but the opening is too small for cats, so the Adsheads bought cat traps, which look like rural mailboxes, except that inside, where the letters would go, there’s a steel brace that delivers an uppercut to the jaw.

The Adsheads put out about four hundred traps in all, and they check them on a regular rotation. When I visited, on a bright blue day toward the end of the Southern Hemisphere winter, they offered to show me how it was done. They packed a knapsack of supplies, including some eggs and kitty treats, and we set off.

As we tromped along, Kevin explained his trapping philosophy. Some people are fastidious about cleaning their traps of bits of rotted stoat. “But I’m not,” he said. “I like the smell in there; it attracts things.” Often, he experiments with new techniques; recently he’d learned about a kind of possum bait made from flour, molasses, and cinnamon, and Gill had whipped up a batch, which was now in the knapsack. For cats, he’d found that the best bait was Wiener schnitzel.

“I slice it thin and I tie it over the trigger,” he told me. “And what happens with that is it starts to dry out and they still go for it.”

I’d come to watch the Adsheads poke at decaying stoats because they are nature lovers. So are most New Zealanders. Indeed, on a per-capita basis, New Zealand may be the most nature-loving nation on the planet. With a population of just four and a half million, the country has some four thousand conservation groups. But theirs is, to borrow E. O. Wilson’s term, a bloody, bloody biophilia. The sort of amateur naturalist who in Oregon or Oklahoma might track butterflies or band birds will, in Otorohanga, poison possums and crush the heads of hedgehogs. As the coördinator of one volunteer group put it to me, “We always say that, for us, conservation is all about killing things.”

The reasons for this are in one sense complicated—the result of a peculiar set of geological and historical accidents—and in another quite simple. In New Zealand, anything with fur and beady little eyes is an invader, brought to the country by people—either Maori or European settlers. The invaders are eating their way through the native fauna, producing what is, even in an age of generalized extinction, a major crisis. So dire has the situation become that schoolchildren are regularly enlisted as little exterminators. (A recent blog post aimed at hardening hearts against cute little fuzzy things ran under the headline “Mrs. Tiggy-Winkle, Serial Killer.”)

Not long ago, New Zealand’s most prominent scientist issued an emotional appeal to his countrymen to wipe out all mammalian predators, a project that would entail eliminating hundreds of millions, maybe billions, of marsupials, mustelids, and rodents. To pursue this goal—perhaps visionary, perhaps quixotic—a new conservation group was formed this past fall. The logo of the group, Predator Free New Zealand, shows a kiwi with a surprised expression standing on the body of a dead rat.

New Zealand can be thought of as a country or as an archipelago or as a small continent. It consists of two major islands—the North Island and the South Island, which together are often referred to as the mainland—and hundreds of minor ones. It’s a long way from anywhere, and it’s been that way for a very long while. The last time New Zealand was within swimming distance of another large landmass was not long after it broke free from Australia, eighty million years ago. The two countries are now separated by the twelve-hundred-mile-wide Tasman Sea. New Zealand is separated from Antarctica by more than fifteen hundred miles and from South America by five thousand miles of the Pacific.

As the author David Quammen has observed, “Isolation is the flywheel of evolution.” In New Zealand, the wheel has spun in both directions. The country is home to several lineages that seem impossibly outdated. Its frogs, for example, never developed eardrums, but, as if in compensation, possess an extra vertebra. Unlike frogs elsewhere, which absorb the impact of a jump with their front legs, New Zealand frogs, when they hop, come down in a sort of belly flop. (As a recent scientific paper put it, this “saltational” pattern shows that “frogs evolved jumping before they perfected landing.”) Another “Lost World” holdover is the tuatara, a creature that looks like a lizard but is, in fact, the sole survivor of an entirely separate order—the Rhynchocephalia—which thrived in the early Mesozoic. The order was thought to have vanished with the dinosaurs, and the discovery that a single species had somehow managed to persist has been described as just as surprising to scientists as the capture of a live Tyrannosaurus rex would have been.

At the same time, New Zealand has produced some of nature’s most outlandish innovations. Except for a few species of bats, the country has no native mammals. Why this is the case is unclear, but it seems to have given other groups more room to experiment. Weta, which resemble giant crickets, are some of the largest insects in the world; they scurry around eating seeds and smaller invertebrates, playing the part that mice do almost everywhere else. Powelliphanta are snails that seem to think they’re wrens; each year, they lay a clutch of hard-shelled eggs. Powelliphanta, too, are unusually big—the largest measure more than three and a half inches across—and, in contrast to most other snails, they’re carnivores, and hunt down earthworms, which they slurp up like spaghetti.

New Zealand’s iconic kiwi is such an odd bird that it is sometimes referred to as an honorary mammal. When it was first described to English naturalists, in 1813, they thought it was a hoax. Kiwi are covered in long, shaggy feathers that look like hair and their extended, tapered beaks have nostrils on the end. They are around the size of chickens but lay eggs that are ten times as large, and it usually falls to the male, Horton-like, to hatch them.

“Everybody gets a doughnut in lieu of everything they ever wanted.”
Buy the print »

New Zealand’s biggest oddballs were the moa, which, in a feathers-for-fur sort of way, stood in for elephants and giraffes. The largest of them, the South Island giant moa, weighed five hundred pounds, and with its neck outstretched could reach a height of twelve feet. Moa fed on New Zealand’s native plants, which, since there were no mammalian browsers, developed a novel set of defenses. For instance, some New Zealand plants have thin, tough leaves when they are young, but when they mature—and grow taller than a moa could chomp on—they put out leaves that are wider and less leathery. The Australian naturalist Tim Flannery has described New Zealand’s avian-dominated landscape as a “completely different experiment in evolution.” It shows, he has written, “what the world might have looked like if mammals as well as dinosaurs had become extinct 65 million years ago, leaving the birds to inherit the globe.” Jared Diamond once described the country’s native fauna as the nearest we’re likely to come to “life on another planet.”

With about half the population of New Jersey, New Zealand is the sort of place where everyone seems to know everyone else. One day, without quite understanding how the connection had been made for me, I found myself in a helicopter with Nick Smith, who was then the country’s minister for conservation.

Smith, who is forty-nine, has a ruddy face and straw-colored hair. He is a member of the country’s center-right National Party, and calls himself a “Bluegreen.” (Blue is the National Party’s color.) He got interested in politics back in the nineteen-seventies, when, as an exchange student in Delaware, he met Joe Biden. We set off from Smith’s district office, in the city of Nelson, on the northern tip of the South Island, and drove out to the helicopter pad in his electric car.

“When the first settlers came here, they tried to create another England,” Smith told me. “We were Little Britain. The comment about us was, we were more British than the British. And, as part of the maturing of New Zealand, there’s the question, What do you connect your nationhood to? You know, for America it’s very much the flag, the Constitution, those sorts of things. The connection with species that are unique to New Zealand is increasingly part of our national identity. It’s what we are as New Zealanders, and I make no bones of the fact that the government is keen to encourage that. You need some things for a country to hold together.”

He went on, “I say to people, If you want your grandkids to see kiwi only in sanctuaries, well, that’s where we’re headed. And that’s why we need to use pretty aggressive tools to try to turn this around.”

My visit happened to coincide with the application of one of these aggressive tools. The country’s Department of Conservation was conducting a massive aerial drop of a toxin known as 1080. (The key ingredient in 1080, sodium fluoroacetate, interferes with energy production on a cellular level, inflicting what amounts to a heart attack.) New Zealand, which has roughly one-tenth of one per cent of the world’s land, uses eighty per cent of its 1080. This year’s drop—the department was planning to spread 1080 over nearly two million acres—had been prompted by an unusually warm winter, which had produced an exceptionally large supply of beech seed, which in turn had produced an explosion in the number of rats and stoats. When the beech seed ran out, the huge cohort of predators was expected to turn its attention to the native fauna. Smith had approved the 1080 operation, which had been dubbed Battle for Our Birds, but the timing of it troubled him; owing to the exigencies of rat biology, the drop had to take place right around the time of a national election.

“If you ask the cynical politics of it, people don’t like poisons but they like rats even less,” he told me. “And so I’ve been doing a few quite deliberate photo opportunities with buckets of rats.”

On this particular day, Smith was attending a more cheerful sort of photo op—one with live animals. When we arrived at the helicopter pad, three other people were already there, all associated with a privately funded effort to restore one of the country’s most popular national parks, named for Abel Tasman. (In 1642, Tasman, a Dutch explorer, was the first European to reach New Zealand—though he didn’t quite reach it, as four of his sailors were killed by Maori before they could land.) Smith and I got into the helicopter next to the pilot, the other three climbed into the back, and we took off. We flew over Tasman Bay, and then over the park, which was studded with ghostly white trees.

“We like to see all those dead trees,” Devon McLean, the director of the restoration project, announced cheerfully into his headset. He explained that the trees were invasive pines, known in New Zealand as “wilding conifers.” I had a brief vision of scrawny seedlings rampaging through the forest. Each dead tree, McLean said, had been individually sprayed with herbicide. He was also happy to report that the park had recently been doused with 1080.

After about half an hour, we landed on a small island named Adele, where we were greeted by a large sign: “Have You Checked for Rats, Mice and Seeds?” A few years ago, after an intensive campaign of poisoning and trapping, Adele was declared “pest-free.” The arrival of a single pregnant rat could undo all that work; hence the hortatory signage. In another helicopter, the conservation department was going to deliver two or three dozen representatives of one of New Zealand’s rarest species, the South Island saddleback. The birds would be released onto Adele, where, it was hoped, in the absence of rats, they would multiply.

More people began to arrive by boat—reporters, representatives of several regional conservation groups, members of the local Maori iwi, or tribe. By this point, it was drizzling, but there was a festive mood on the beach, as if everyone were waiting for a celebrity. “Hardly any New Zealanders have ever seen a saddleback,” a woman whispered to me.

In anticipation of the birds’ appearance, speeches were offered in Maori and English. “It has taken us Pakeha New Zealanders a little while to gain an appreciation of what is special here and to really be committed to its protection and survival,” Smith said, using the Maori word for European. “It’s kind of scary to think of South Island saddlebacks—there are only about six hundred that exist on the planet.”

Finally, the birds arrived, in a helicopter that had been loaned for the occasion by a wealthy businessman. Three crates were unloaded onto the beach, and Smith and a pair of local dignitaries were given the honor of opening them. South Island saddlebacks are glossy black, with patches of rust-colored feathers around their middles and little orange wattles that make them look as though they’re smiling. They are another example of an ancient lineage that persists in New Zealand, and they have no close relatives anywhere else in the world. The birds hopped out of their crates, flew into the bush, and were gone.

There are two quick ways to tell a Norway rat from a ship rat. One is to look at the ears. Ship rats have large ears that stick out from their heads; Norway rats’ ears are shorter and less fleshy. The other is to look at their tails. Again, Norway rats’ are shorter. With a ship rat, if you take its tail and fold it over its body—here it obviously helps if the rat is dead—it will extend beyond its nose.

These and many other facts about rats I learned from James Russell, an ecologist at the University of Auckland. Russell’s office is filled with vials containing rat body parts in various stages of decomposition, and he also keeps a couple of dead rats at home in his freezer. Wherever he goes, Russell asks people to send him the tails of rats that they have trapped, and often they oblige. Russell then has the rats’ DNA sequenced. Eventually, he hopes to be able to tell how all of New Zealand’s rat populations are related.

“I would be inclined to say rats are our biggest problem,” Russell told me. “But I have colleagues who spend their career on stoats, and colleagues who spend their career on cats. And they open all their talks with ‘Stoats are the biggest problem,’ or ‘Cats are the biggest problem.’ ”

“It’s an adjustment being a handshake guy in a fist-bump company.”
Buy the print »

Russell, who is thirty-five, is a slight man with tousled brown hair and a cheerful, let’s-get-on-with-it manner which I eventually came to see as very New Zealand. I ended up spending a lot of time with him, because he volunteered to guide me along some of the country’s windier back roads.

“New Zealand was the last large landmass on earth to be colonized,” he told me one day, as we zipped along through the midsection of the North Island. “And so what we saw was the tragedy of human history playing out over a short amount of time. We’re only ten years behind a lot of these things; that’s as compared to countries where you’re hundreds or thousands of years behind the catastrophe.”

New Zealand’s original settlers were the Maori, Polynesians who came around the year 1300, probably from somewhere near the Society Islands. By that point, people had already been living in Australia for some fifty thousand years. They’d been in continental North America for at least ten thousand years, and in Hawaii, which is even more remote than New Zealand, for more than five hundred years. In each case, it’s now known, the arrival of humans precipitated a wave of extinctions; it’s just that, as Russell points out, these “tragedies” were not recorded by the people who produced them.

When the Maori showed up, there were nine species of moa in New Zealand, and it was also home to the world’s largest eagle—the Haast’s eagle—which preyed on them. Within a century or two, the Maori had hunted down all of the moa, and the Haast’s eagle, too, was gone. A Maori saying—“Ka ngaro i te ngaro a te moa”—translates as “Lost like the moa is lost.”

In their ships, the Maori also brought with them Pacific rats, or kiore. These were New Zealand’s first introduced mammals (unless you count the people who brought them). The Maori intended to eat the kiore, but the rats multiplied and spread far faster than they could be consumed, along the way feasting on weta, young tuatara, and the eggs of ground-nesting birds. In what, in evolutionary terms, amounted to no time at all, several species of New Zealand’s native ducks, a couple of flightless rails, and two species of flightless geese were gone.

The arrival of British settlers, in the middle of the nineteenth century, brought in more—many more—new invaders. Some of them, like the Norway rat and the ship rat, were stowaways. Others were introduced deliberately, in an effort to make New Zealand feel more like home. The “importation of those animals and birds, not native to New Zealand,” an 1861 act of the colonial Parliament declared, would both “contribute to the pleasure and profit of the inhabitants” and help maintain “associations with the Old Country.” What were known as “acclimatization societies” sprang up in every region. Among the many creatures the societies tried to “acclimatize” were red deer, fallow deer, white-tailed deer, sika deer, tahr, chamois, moose, elk, hedgehogs, wallabies, turkeys, pheasants, partridges, quail, mallards, house sparrows, blackbirds, brown trout, Atlantic salmon, herring, whitefish, and carp. Brushtail possums were specially imported from Australia, in an attempt to start a fur industry.

Not all the new arrivals took; others took all too well. By the perverse logic of such affairs, some of the most disastrous introductions were made in an effort to control previous disastrous introductions. Within a few decades of their importation, European rabbits had overrun the countryside, and in 1876 an act was passed to “provide for the Destruction of Rabbits in New Zealand.” The act had no perceptible impact, so stoats and ferrets were released into the bush in the hope that they would be more effective.

“Our forebears tried most experiments that could possibly be conceived and some that would be difficult for anyone with any knowledge of ecology to seriously contemplate,” Robert McDowall, a New Zealand naturalist, has written in a history of introduction efforts. The combined effect of all these “experiments,” particularly the introduction of predators, like stoats, has been ongoing devastation. Roughly a quarter of New Zealand’s native bird species are now extinct, and many of those which remain are just barely hanging on. If current trends continue, it is predicted that within a generation or two the land of Kiwis will be without kiwi.

“The defence of isolation for remote islands has no fallback position,” John McLennan, a New Zealand ecologist, has written. “It is all or nothing, akin to virginity, with no intermediate state.”

Sirocco, a sexually dysfunctional kakapo, normally lives alone on his own private island. On occasion, though, he is brought, with great fanfare, to the mainland, and this is where James Russell and I set out to meet him, at a special mountaintop reserve ringed by a twenty-nine-mile fence. The fence is seven feet high and made of steel mesh with openings so narrow an adult can’t stick a pinkie through. At the base of the fence, an eighteen-inch apron prevents rats from tunnelling under; on top, an outwardly curved metal lip stops possums and feral cats from clambering over. To get inside, human visitors have to pass through two sets of gates, an arrangement that made me think of a maximum-security prison turned inside out.

Kakapo—the name comes from the Maori, meaning “parrot of the night”—are nocturnal, so all audiences with Sirocco take place after dark. Russell and I joined a group of about twenty other visitors, who had paid forty dollars apiece to get a peek at the bird. Sirocco was hopping around in a dimly lit plate-glass enclosure. He was large—about the size of an osprey—with bright green-and-brown feathers and a bulbous, vaguely comical beak. He gazed through the glass at us and gave a sharp cry.

“He’s very intense, isn’t he?” the woman next to me said.

Alone among parrots, kakapo are flightless, and, alone among flightless birds, they’re what’s known as lek breeders. During mating season, a male will hollow out a little amphitheatre for himself, puff up his chest, and let out a “boom” that sounds like a foghorn. Kakapo, which can live to eighty, breed only irregularly, in years when their favorite foods are in good supply.

Kakapo once were everywhere in New Zealand. In the late nineteenth century, they were still plentiful in rugged areas; Charlie Douglas, an explorer who climbed some of the steepest mountains of the South Island, described them standing “in dozens round the camp, screeching and yelling like a lot of demons.” But then their numbers crashed. By the nineteen-seventies, there was just one small population remaining, and it was threatened by feral cats. In the nineteen-eighties, every individual that could be caught was captured and “translocated.” Today, there are a hundred and twenty-six kakapo left, and all of them, save Sirocco, live on three remote, predator-free islands, Little Barrier, Codfish, and Anchor.

Sirocco’s chaperone, a Department of Conservation ranger named Alisha Sherriff, had brought along a little metal container, which she passed among the visitors. Inside was half a cup’s worth of Sirocco’s shit.

“Have a good sniff,” she suggested.

“It’s earthy!” one woman exclaimed.

“I think it smells smoky, with notes of honey,” Sherriff said. When the container came to me, I couldn’t detect any honey, but the bouquet did strike me as earthy, with hints of newly mown hay. Sherriff had also brought along a ziplock bag with some of Sirocco’s feathers. These, too, had a strong, sweetish scent.

New Zealand birds tend to smell, which was not a problem when the islands’ top predators were avian, since birds hunt by sight. But, as mammals hunt with their noses, it’s become yet another liability. (A few years ago, a Christchurch biologist was awarded a four-hundred-thousand-dollar grant to investigate the possibility of developing some sort of “deodorant” for ground-nesting birds.)

March 1, 2004“My bark is worse than my bite, and my bark is sort of a yap.”
Buy the print »

Sherriff explained that Sirocco had been born on Codfish Island in 1997, but as a chick he had come down with a respiratory infection, and so had been removed from his mother and raised in isolation. By the time he was well enough to rejoin the other kakapo, he’d decided he preferred people. During breeding season, he’d try to mate with the rangers on Codfish while they were walking to the outhouse. A special barrier was built to try to prevent the encounters, but Sirocco turned out to be too determined.

“When you’ve got a three-kilo bird crashing through the bush in the middle of the night, there’s quite a high risk of people lashing out unintentionally,” Sherriff said. It had been decided that, for Sirocco’s own safety, he’d have to be moved. He now lives hundreds of miles from Codfish, on an island whose name the Department of Conservation won’t disclose. For the sake of genetic diversity, Sirocco’s semen had been collected, by means of what Sherriff described as a “delicate massage,” but his sperm count had proved too low to attempt artificial insemination.

After about half an hour with Sirocco, we were hustled out to make room for the next tour group. Russell and I passed back through the gates and drove down to a restaurant at the base of the mountain, where we’d arranged to have dinner with Matt Cook, the reserve’s natural-heritage manager. Cook told us that it had taken teams of exterminators three years to eliminate mammals from inside the fence and that they’d never managed to finish off the mice. The entire perimeter was wired so that when, say, a section of fence was hit by a falling branch, a call automatically went out to a maintenance crew.

“This always happens at 3 A.M. on a Saturday morning,” he said. If the fence was breached, Cook reckoned, the crew had about ninety minutes to repair it before rats would find the opening and sweep back in.

Today, invasive species are everywhere. No matter where you are reading this, almost certainly you are surrounded by them. In the northeastern United States, common invasive plants include burdock, garlic mustard, purple loosestrife, and multiflora rose; invasive birds include starlings, rock pigeons, and house sparrows; and invasive insects include Japanese beetles, gypsy moths, and hemlock woolly adelgid. Texas has more than eight hundred nonnative plant species, California at least a thousand. Even as New Zealand has been invaded by nonnative species, its native species have invaded elsewhere. Potamopyrgus antipodarum, or, as it is more commonly known, the New Zealand mud snail, is a tiny aquatic snail about the size of a grain of rice. It can now be found in Europe, Asia, Australia, the Middle East, and the American West, and it has proved such a successful transplant that in some parts of the world it reaches densities of half a million snails per square yard. In an irony perhaps only Kiwis can appreciate, a recent study in Utah found that mud snails were threatening populations of rainbow trout, a fish imported at great expense to New Zealand in the eighteen-eighties and now considered an invader there.

The project of reshuffling the world’s flora and fauna, which began slowly with the spread of species like the Pacific rat and sped up thanks to the efforts of acclimatization societies, has now, with global trade and travel, accelerated to the point that, on any given day, something like ten thousand species are being moved around just in the ballast water of supertankers. Such is the scale of this remix that biologists have compared it to reassembling the continents. Two hundred million years ago, all of the world’s landmasses were squished together into a single giant supercontinent, Pangaea. We are, it’s been suggested, creating a “new Pangaea.”

One response is simply to accept this as the planet’s destiny. Yes, the invaders will, inevitably, choke out some local species, and there will be losses, especially on islands, where, unfortunately, much of the world’s diversity resides. But people aren’t going to stop shipping goods and they aren’t going to stop travelling; therefore, we’re just going to have to learn to live in a Pangaea of our own making. Meanwhile, who even knows at this point what’s native and what’s not? Many species that people think of as a natural part of the landscape are really introductions that occurred before recent memory. For instance, the ring-necked pheasant, the official state bird of South Dakota, is an import from China.

“It is impractical to try to restore ecosystems to some ‘rightful’ historical state,” a group of American researchers wrote a few years ago, in Nature. “We must embrace the fact of ‘novel ecosystems’ and incorporate many alien species into management plans, rather than try to achieve the often impossible goal of eradicating them.”

New Zealanders are nothing if not practical. They like to describe the national mind-set as “the No. 8 wire mentality”; for much of the country’s history, No. 8 wire was used to fix livestock fences and just about everything else. Nevertheless, Kiwis refuse to “embrace” novel ecosystems. In the past few decades, they have cleared mammalian predators from a hundred and seventeen offshore islands. The earliest efforts involved tiny specks, like Mokopuna, or, as it is also known, Leper Island, which is about the size of Gramercy Park. But, more recently, they’ve successfully de-ratted much larger islands, like Campbell, which is the size of Nantucket. With its predator-free islands and its fenced-in reserves and its massive poison drops from the air, New Zealand has managed to bring back from the very edge of oblivion several fantastic birds, including the kakapo, the South Island saddleback, the Campbell Island teal, and the black robin. At its lowest point, the black robin was down to just five individuals, only one of which—a bird named Old Blue—was a fertile female. (When Old Blue died, her passing was announced in Parliament.)

Meanwhile, by tackling larger and larger areas, New Zealanders have expanded the boundaries of what seems possible, and they increasingly find their skills in demand. When, for example, Australia decided to try to get rid of invasive rodents on Macquarie Island, roughly halfway between Tasmania and Antarctica, it hired a New Zealander to lead the effort, and when the U.S. National Park Service decided to get rid of pigs on Santa Cruz Island, off the coast of Southern California, it hired Kiwis to shoot them. The largest rat-eradication effort ever attempted is now in progress on South Georgia Island, a British territory in the South Atlantic with an area of nearly a million acres. A New Zealand helicopter pilot was brought in to fly the bait-dropping missions. One day, when I was driving around with James Russell, he got an e-mail from Brazil: the government wanted to hire him to help it get rid of rats on the Fernando de Noronha archipelago, off Recife. David Bellamy, a British environmentalist and TV personality, has observed that New Zealand is the only country in the world that has succeeded in turning pest eradication into an export industry.

The idea of ridding all of New Zealand of its mammalian predators was proposed by Paul Callaghan, a world-renowned physicist, in a speech delivered in Wellington, in February, 2012. In scientific circles, Callaghan was celebrated for his work on nuclear magnetic resonance; to Kiwis he was probably best known for having recently been named New Zealander of the Year. At the time he gave the speech, Callaghan was dying of cancer, and everyone who heard it realized that it was one of the last he would deliver. (He died the following month.)

“Let’s get rid of the lot,” Callaghan said. “Let’s get rid of all the predators—all the damned mustelids, all the rats, all the possums—from the mainland.”

“It’s crazy,” he continued, referring to his own proposal. But, he went on, “I think it might be worth a shot. I think it’s our great challenge.” Callaghan compared the project to the moon landing. It could be, he said, “New Zealand’s Apollo program.”

December 3, 2001“And that’s a huge colander.”
Buy the print »

I listened to Callaghan’s speech, via YouTube, on one of my last evenings in New Zealand. It seemed to me that what he was proposing was more like New Zealand’s Manhattan Project than its Apollo program, though I could see why he hadn’t framed it that way.

New Zealand’s North Island is roughly forty-four thousand square miles. That means that it’s nearly a thousand times bigger than the largest offshore island from which predators have, at this point, been eradicated. James Russell serves as an adviser to Predator Free New Zealand, the group that was formed to pursue Callaghan’s vision. I asked him about the feasibility of scaling up by such an enormous amount. In response, he showed me a graph: the size of the islands from which predators have been successfully removed has been increasing by roughly an order of magnitude each decade.

“It is a daunting scale that we’re talking about,” Russell told me. “But, then, you see the rate at which we have scaled up.”

“Some people think scientists should just be objective,” he went on. “They sit in the lab, they report their results, and that’s it. But you can’t separate your private life from your work life. So I do this science and then I go home and think, Wouldn’t it be great if New Zealand had birds everywhere and we didn’t have to worry about rats? And so that’s the world I imagine.”

Listening to Callaghan on YouTube also reminded me of a point that Nick Smith had made the day of the saddleback release: in New Zealand, killing small mammals brings people together. During my travels around the country, I found that extermination, weird as it may sound, really is a grassroots affair. I met people like the Adsheads, who had decided to clear their own land, and also people like Annalily van den Broeke, who every few weeks goes out to reset traps in a park near her home, in the suburbs of Auckland. In Wellington, I met a man named Kelvin Hastie, who works for a 3-D mapping company. He had divided his neighborhood into a grid and was organizing the community to get a rat trap into every hundred-square-metre block.

“Most of the neighbors are pretty into it,” he told me.

Just about everyone I spoke to, including Hastie, expressed excitement about the latest breakthrough in extermination technology, a device designed by a company called Goodnature. And so, on my last day in the country, I went to visit the company’s offices. They were situated in a drab stretch of anonymous buildings, not far from one of Wellington’s best surfing beaches. When I arrived, I noticed a dead stoat in a plastic bag by the door. Apparently, it had been killed quite recently, because it was still very much intact.

Robbie van Dam, one of the company’s founders, showed me around. In the first room, several young people were working at standing desks, on MacBooks propped up on cardboard boxes. A small kitchen was stocked with candy and half-empty bottles of wine. In a back room, bins of plastic parts were being assembled into an L-shaped machine that resembled a portable hair dryer. Van Dam pulled out one of the finished products, known as the A24. At one end, there was a CO2 cannister of the sort used in bicycle pumps. Van Dam uncapped the other end and pulled out a plastic tube. It was filled with brown goo. In the crook of the L was a hole, and in the hole there was a wire. Van Dam gingerly touched the wire, and a piston came flying out.

The A24 is designed to be screwed to a tree. The idea is that a rat, smelling the goo, which is mostly ground nuts, will stick its head into the hole, trip the wire, and be killed instantly. The rat then falls to the ground, and the device—this is the beauty part—automatically resets itself. No need to fish out rotten eggs or decaying flesh. Each CO2 cannister is supposed to be good for two dozen rats or, alternatively, stoats—hence the name. (For stoats, there’s different bait, made of preserved rabbit.) Van Dam also showed me a slightly larger machine, the A12, designed for possums.

“The humaneness problem was probably the hardest part,” he told me. In the case of possums, it had turned out that a blow to the head wasn’t enough to bring about quick death. For that reason, the A12 is designed to fill the animal’s skull with carbon dioxide and emulsify its brain. Goodnature also sells cannisters of possum bait, laced with cinnamon. I picked up a tube. “12 out of 12 possums choose this as their final meal,” the label said.

Even taking the long view—the very long view—the threat posed to New Zealand’s fauna by invasive species is extraordinary. It may be unprecedented in the eighty million years that New Zealand has existed. But we live in an age of unprecedented crises. We’re all aware of them, and mostly we just feel paralyzed, incapable of responding. New Zealanders aren’t just wringing their hands, nor are they are telling themselves consolatory tales about the birth of “novel ecosystems.” They’re dividing their neighborhoods into grids and building better possum traps—ones that deliver CO2 directly to the brain.

A couple of miles from Goodnature’s headquarters is a rocky beach where little blue penguins sometimes nest. The beach is infested with rats, which can attack the nests, so Goodnature has installed some A24s along it, and van Dam took me down to have a look. It was a beautiful windy day, and the surf was high. Under the first A24, which was attached to a gnarly bush, there was nothing. Often, van Dam explained, cats or other rats drag off dead animals that have dropped from the A24, so it’s not always possible to know what, if anything, has been accomplished. This had proved to be something of a sales problem.

“If people didn’t find something dead under there, they were really disappointed,” he said. Goodnature now offers a digital counter that attaches to the A24 and records how many times the piston has been released.

Under the second A24, there was a dead mouse; under the third and fourth, nothing. Under the fifth were two ship rats, one freshly killed and the other just a clump of matted fur with a very long tail. ♦