Your First Conference

Do you remember the first programming conference you ever attended?

I do.

It was an OSCON, in Portland. Something like a thousand people attended it.

Looking back it was objectively terrible compared to any of the events I've attended in the last year. Hosted in a convention center, terrible food, worse coffee, parallel tracks with no coherent narrative. I remember the party was run by Sun Microsystems, not officially part of the conference, and in the open air parking garage of the DoubleTree. Standing around a keg and talking loudly over the echo of the other attendees none of that mattered. I loved it, it was amazing.

I enjoy the intimate community run conferences around the world I attend now. But when I remember what it was like my first time I have to admit that none of these could have been my first conference.

I didn't know anyone back then and the anonymity you get by being one in a sea of people was comforting. The intimate setting of a conference like NodeConf, which I've run for the last 3 years, would have been terrifying.

In the last few years I've seen my community grow at an astonishing rate and yet we haven't created much that is accessible enough to draw most of that community in. For those who are just dipping their toes in the water of JavaScript I want to create an event that draws them in to the deep end.

I have high hopes for JSFest, and I hope that you'll join me.

The Bridge

If I take a moment to look back on my life I see only the bridges. They connect me from here to there, from where I was to where I needed to go.

When I was young each bridge was clear to me, where it went, what I needed to do. One foot in front of the other I crossed the bridge.

So many others decided not to cross the bridge with me. They stayed where they were, they found their place, and I was left alone to cross more bridges until I found my own place to stay.

Eventually the bridges I crossed lost their clarity. The steps I took, the places I need to go, they were unclear to me until I was walking them. At first I could see nothing but with each step across the bridge I could see it more clearly, where I was going, what I needed to do, where I would end up and what it would be like when I arrived.

But now it is different. Now I am many miles across the longest bridge and I still cannot see it.

Along the way I've met others who couldn't find their place, who kept crossing bridges and never stayed where they were and we came to realize that maybe we will never find a place that is ours, a place we stay.

This new bridge, the longest one yet, I'm not alone as I walk across it. Many friends are on it and I can see most of those whom inspire me. Nobody knows what this bridge is, what to do, where it's going, none of us can see it.

There seem to be people behind us, following us, across the bridge. I used to think they followed us because we could see it more clearly but now I think they follow us because we're crossing it even though we can't see it.

One foot in front of the other, with each step we may fall, we're finding our place, crossing the bridge.

FA Podcast: Community Chat w/ @izs and @mikeal

Listen on soundcloud.

Show Notes:

  • A quick chat about community with Isaac Schlueter, head of the node.js project.

Subscribe to the podcast feed.

FA Podcast: governance vs. Governance w/ @selenamarie and @mikeal

Listen on soundcloud.

Show Notes:

  • A chat with Selena Deckelmann about community.

Subscribe to the podcast feed.

FA Podcast: NodeBases Chat (Replication) w/ @maxogden @rvagg @mikeal

Listen on soundcloud.

Show Notes:

  • A chat about replication in the node database world.

Subscribe to the podcast feed.

FA Podcast: NodeBases Chat (Locks and Mutexes) w/ @dominictarr @mikeal

Listen on soundcloud.

Show Notes:

  • A conversation with @dominictarr about node database locks, mutexes, and other fun stuff.

Subscribe to the podcast feed.

Thank You All

This year's NodeConf was, by far, the greatest conference I've ever been a part of.

Last year I put everything I had into NodeConf and this year I did the same. I can only do so much and what brought this year to a higher level was that I asked more of the speakers and staff than any conference ever has.

In creating the format for this year I thought mainly about the experience of the attendees. Only after I had committed to splitting the crowd of 300 up into smaller (~40 person) hands-on sessions did I realize the toll it would take on those presenting. Nobody has ever asked more of their staff and speakers than NodeConf and they deserve all of the credit for making this year so amazing. And so, I'm going to call each of them out individually for some much needed recognition and thank them one at a time.

First of all, to my wife Anna Maier, who has spent her lunch breaks for months handling many of the logistics and payments including a grueling ordeal with the bus company. And my mother, Terry Helmka, for helping with the SO track and running craft workshops for the kids.

To my brother John Akin for documenting such an amazing experience.

To Emily Tanaka-Delgado for handling the attendee intake at Joyent and getting everyone on the buses.

To Alyssa Ravasio and Nicole Sullivan for helping with registration and cleanup.

To Paul Campbell, for his wonderful songs and glowing presence.

To @visnup and @dshaw who led the effort to keep people hydrated in the morning and de-hydrated with beers every evening, overcoming the dwindling resources of cargo van space and recycling receptacles.

To Jessica Lord for designing the map and tshirts.

Dan Finlay, Daniel Erickson and Hannah Boone who drove attendees into town during medical emergencies.

To Forrest Norvell and Domenic Denicola who taught more people domains in a single day than had ever understood them before.

To Emily Rose, Adam Ulvi and Ben Acker who flew, powered, and repaired drones all day and made it so accessible that even my mother and my wife (who have never written a line of code) were able to fly them with JavaScript.

To Rick Waldron, Elijah Insua, and Raquel Velez who opened people up to manipulating the physical world with node.

To Isaac Schlueter and Bert Belder who demystified core and created new committers.

To Max Bruning, TJ Fontaine, and Josh Clulow who made DTrace accessible.

To Eran Hammer and Mike Cantelon who, 8 times in a row, contrasted the differences in web frameworks without arguing with one another :)

To @raynos and Dominic Tarr who got peoples code talking to other peoples code in a distributed system they wrote in less than an hour.

To Max Ogden and @substack, who guided people along a stream adventure. And again to @substack, who was sick the whole conference but still ran all his sessions to the high praise and acclaim of all the attendees.

And finally, to Claudia, Jim, Michael and all the rest of the staff at Walker Creek Ranch for making this such a magical experience.

FA Podcast: Routine

Listen on soundcloud.

Show Notes:

  • Freedom, Routine and Ritual

Twyla Tharp and the Creative Habit http://www.amazon.com/dp/B000SEOWBG/?tag=httpwwwfutu0b-20

Coffee Ritual

Bona Vita Temp Controlled Kettle http://www.amazon.com/dp/B005YR0F40/?tag=httpwwwfutu0b-20

Hario v60 Ceramic Cone ht

Subscribe to the podcast feed.

FA Podcast: Episode One

Listen on soundcloud.

Show Notes:

One Question w/ Visnu Pitiyanuvath, Chris Williams and Isaac Schlueter.

How i stopped being fat.

Subscribe to the podcast feed.

No Builds

When we first started writing code at Getable I set a simple contraint; no builds.

Nobody is allowed to checkin a script that we run either before deployment or before checkin. As our application grew we were tempted more than once to solve a problem with a build step. Instead, we resisted and found what was ultimately a better solution.

Anything that would be a build step is written as normal code in node.js, without the writing to disc part. The routes we write for resources that would normally be the output of a build process we generate the first time they are requested and cache them in memory indefinitely which is easy since all of our routes use jaws.

We have two scripts to run the application, one for debugging and one for production. Debug sets a single global that we check in if statements to enable source maps. The debug script also watches the files in the working directory for changes and flushes the cache, obviating the need for "dynamic rebuilds."

And we "build" a lot of stuff this way. All the front-end resources use browserify, the css is built using rework, and a bunch of other js files (jQuery, engine.io, etc) are concatenated together, minified, and loaded before the rest of the application code.

Deploying is as simple as pushing to git and restarting a process. We check our dependencies in to git and try to avoid compiled dependencies.

Tests can run without any rebuilding and they use the same code as the production app but with sourcemaps enabled since they set the debug global.

That's it. Short post :) Simple ideas are simple, create constraints to keep them that way.

The Fiction Of Leadership

Node core has one creator, one dictator, and a short list of committers. I am not the creator, the dictator, or on the committer list.

I'm the dictator of several tiny GitHub projects and a conference. I'm the CTO of a company with a small but amazing engineering team.

I make decisions, I have opinions, and I agree and disagree with many people about many things in public.

People listen to me because I've gained some amount of credibility with them over time. Other people don't listen to me and probably for similar if opposing reasons.

People who write node listen to me more than people who don't. I was around in the earliest days of the platform and have written node nearly every day since. Since node is fairly young experience is a scarce resource but every year that resource grows and people who were there early matter less.

I don't make decisions for people. Substack doesn't make decisions for people. Isaac doesn't make decisions for people, except in core. We all write code, we state our opinions, and people choose which to listen to. Isaac, Substack and I almost never all agree about any one thing.

There is no "node leadership" in the larger community in that there is nobody with the authority or even the credibility to end, remove or push out another project or person. There's no illuminati, no committee, and the effect of decisions made by core, which is a dictatorship, are nearly over.

And yet, people sometimes claim that the "node leadership" are stifling their ideas.

Several people are influencers in the node community. People are influenced by them because they have built up a fair amount of credibility for their ideas and solutions. Nearly all of those influencers disagree with underscore and async, the two most depended on libraries in the node ecosystem, which gives some insight in to the ability of so called leadership to stifle ideas or projects.

All of these influencers, especially me, are often wrong. I was very dismissive of underscore and async, which I now use regularly.

No matter what your idea is you will get no more than 50% of the node influencers to agree with it so it's best not to even try. The success of your idea does not hang on me, or Substack, or Isaac endorsing or using it. Most of what the influencers actually produce is dismissed by nearly all other influencers. We sometimes don't even agree that we all like node.js.

I wrote a post that encouraged people with certain ideas to pursue them outside of node's walls not because I disagree with those ideas but because I truly believe they will be more successful that way. Before coming to node I spent time in Python's alternate concurrency systems (Twisted, asyncore, stackless) and the reason why node has succeeded where they failed has a lot to do with node not trying to make an existing community happy and creating a new one instead.

One day someone will write something much better than node and I'll use it, but I don't believe that will happen inside of node's ecosystem because I've never seen it happen before. Grand new projects succeed when they build new communities not seek the approval of people already invested in something else.

If it makes me a negative person to suggest that people will have more success following the path of successful projects than failed ones then I guess I'm an asshole.

It is unlikely that I will stop having opinions, and expressing them, nor will other people in the fictitious "node leadership" but your success and the acceptance of your ideas isn't dependent on our approval and any great idea we didn't already think of we're likely to dismiss out of ignorance, arrogance, or both, so it's best to just ignore what we think and have fun building software.

No idea is served well by complaining about the opinions of people who disagree with it and nobody is going to be convinced your detractors are wrong by complaining about them saying it instead of refuting their ideas with your own.

Momentum

At Mozilla and OSAF I worked on ancillary projects, projects that fed in to the development of the main product without being part of the main product itself. The most frustrating part for me was that while both the main product and my ancillary projects are open source they were hugely different in their needs and community. The main project is well funded, there are many people paid to work on it, and this injects continual investment in to that project, while my ancillary project is usually just me and in order to attract contributors I needed to understand how momentum works in community driven projects outside the walls of my employer.

This made me more attentive to other projects, to what drives momentum and what discourages it. How do you grow a project and what are the limitations? When node.js came along I actively tried to steer things in a direction I thought was positive and would encourage growth but only now, after years of reflection, do I begin to understand why it was so successful and why it now appears to be maintaining an absurd growth rate well above its contemporaries.

The earliest months of a project are the most important. After initial release you have nowhere to go but up, if you release something on your own you're just one developer and if you get 10 people to invest in your project that's 10x growth. From what I've seen there is no real limit to the early growth of an open source project, there are an unbelievable amount of people who will choose to invest in even the newest and least proven technology. You don't need to hitch your wagon to an already growing community and you also don't need to attack the competition to get attention. You need to reduce negativity, that means reaching out to people who are likely to think your project is interesting and distancing yourself from any predictable negativity until you've grown large enough you can handle it. Negativity early on is a killer, it decreases early investment and a new project must avoid it.

Negativity comes in two flavors, internal and external. Internal negativity is attacking the competition; their concurrency patterns suck, you can't write real programs, etc. This attracts the wrong kind of people, you want people who are going to sit on IRC and answer questions in their free time not troll every newcomer. In the early days you should hope that anyone you would attack decides to ignore you, completely. Human beings make investments and any project that is incompatible with their current investments is a potential threat. This is where external negativity comes from and the best way to avoid it is distance.

Every platform comes with a set of established patterns for compatibility. Sometimes these patterns are incredibly limiting, some of them may even lead to the eventual demise of the platform itself. Every platform has compatibility forks that attempt to overcome those limitations and in every case I know of the fork fails to overtake the established pattern. Anyone who's written Twisted or EventMachine knows what I'm talking about. These forks get some momentum early on but are eventually crushed by the negativity from the rest of the community they try to make their home.

If you write a library in Twisted you can grab some value from the rest of the Python community (the language, a portion of the libraries) but a lot of it isn't compatible with what Twisted is trying to do and what you publish using Twisted isn't usable by the rest of the Python community at all unless they adopt Twisted, which leads to some predictable negativity from the community at large.

But, the real question should be "why was Twisted trying to be compatible with Python at all?"

Promises to Fibers

At this point in time Promise libraries for node are entirely compatible with all established patterns in node.js and are only incompatible if the author chooses not to expose a compatible API. If you use a Promise library in your module it's at your discretion whether or not you expose an API as a Promise or as a callback which you then internally wrap up in a Promise. In many ways Promises are just another flow control library, in node at least, and when authors choose to expose a callback API they are opting out of being a compatibility fork. This means two things 1) that Promises are highly unlikely to grow their own community that is incompatible with most of the node ecosystem and 2) the usage or Promises as an external API will never eclipse node's currently established patterns.

If the goal of Promise libraries is to overcome some limitation in node's patterns then they are, currently, limiting the scope of that fix to people who actively invest additional resources in supporting Promises. They are not building a significant ecosystem for their pattern because they are hampered by compatibility. This isn't necessarily a bad thing, this could be the goal of Promise libraries, to provide a flow control pattern for people that seek one out and not to create something potentially bigger than node that is free of the limitations they see in node.

Fibers and streamline are another bag of worms. Their advocates are small but loud and live on the node.js mailing list. Both libraries are attempting to overcome limitations they see in node.js and both do it in ways that limit compatibility with the rest of the ecosystem. Both are met with a fair amount of negativity, which is unfortunately predictable, and it has slowed their growth rate to a fraction of the rest of node's. I have the same question for these libraries as I do for Twisted; "why be compatible with node.js at all?"

Myths of growth

Embedded deep within our thoughts on this subject is a false belief. We all seem to be under the impression that the available growth to open source ecosystems is finite. The reason why people view new challenges to their investment as a threat is the idea that if it were to take off the project would reduce existing investment compatible with established patterns.

The reason partially incompatible forks try so hard to be at least partially compatible is so that they can avoid having to create their own community and gain their own momentum because they believe the available investment is finite and that its better to push their way in to an established and growing community than create a new one.

What I've found is that the available investment in open source ecosystems is not finite. Investment in new projects appears to have little to no impact on existing projects. What I do find is that there are limits to momentum. Quarter over quarter percent growth is dramatic in the earliest growth period (there's so much room to grow!) but once that growth, as a percent of the whole, reduces it rarely if ever grows more than 5% above the previous quarter.

Momentum is not a series of curves that dip and swing, momentum is a snowball on a mountain. The snowball experiences dramatic growth as it starts down the mountain but once it begins to flatten its potential is limited to maintaining its speed or slowing down.

Releasing a new project is the top of the mountain, the more speed you can give it in the earliest phases, the longer you can keep it from slowing down, the greater the potential for that project in the long term.

This is why I think partial compatibility is a mistake. Partial compatibility exposes you to more negativity, which limits early growth. Partial compatibility also limits the amount of potential investment. Node.js could have been a non-blocking fork of Narwhal, it would have had the "advantage" of many existing js modules and the potential to bind to the huge world of Java libraries. Node took a different path.

Node is one of the least compatible platforms that has ever been released. It was compatible with almost no existing JavaScript code since nearly all existing code used the DOM (it would be almost a year before jsdom and others created node versions of the DOM). It was incompatible with most C libraries that nearly all other platforms lean on, like database bindings, which meant that new libraries had to be written from scratch where they hadn't been for platforms like Ruby and Python. There was almost nothing written that worked on node and nothing published that could be used to write a node application. The Fibers and Twisteds of the world consider this incompatibility a liability, a huge barrier to investment, when the opposite is actually true.

In the earlier days of node you could write anything. The advantage of nothing being compatible is that there is a much greater opportunity to attract people interested in writing virtually anything. That means a greater potential for early investment, which creates early momentum, which you can only lose over time so maximizing that early growth is the most important thing.

Node made very few decisions, it limited the size of its core much more than other platforms, but decisions were made. One of those decisions will eventually hold it back from being suitable for a future class of applications. I'm confident in this because I've seen it so many times before. Whatever takes off for that new class of applications to eclipse node will not take off inside the walls of node's ecosystem. Attempting to innovate against the grain of node's current patterns for compatibility will insure failure the way other compatibility forks in other languages failed.

If you take anything from the success of node's ecosystem it's that if you want to create something that overcomes node's limitations you can't do it within node's ecosystem.

Idols

We had never spoken at a conference before but that didn't keep us from submitting a talk on something we had yet to finish and waiting until the last moment to get it all done. The night before our talk Adam and I couldn't help but go out, it was our first chance to drink with some of our idols. In a hotel parking garage filled with kegs and neckbeards we had our fill of drink and conversation before heading back to the hotel and opening our laptops.

Well after midnight we did the last few checkins and passed out. As the sun came up I awoke in fear that our live demo would break or that we'd get too nervous to walk through it consistently. We decided to record the demo ahead of time as a video and Adam had to walk with his elbow between the keyboard and screen of his laptop from the Courtyard Marriott to the Oregon Convention Center while it rendered.

I was nervous, Adam was nervous, and our room in the convention center was so packed they had to turn some people away. If I were to watch that presentation today I'm sure I would be mortified but at the time people loved it and were excited about what we had released. Adam and I were interviewed for a podcast, treated well at parties, and felt like we'd made a good impression on many of our idols.

This was the new peak of my professional life. The community, the conversation, and the recognition I knew I didn't deserve, it all lifted me up to a high I was afraid to fall down from. I thought it was the best conference in the entire world and I told Adam I'd never miss another one for as long as I lived. That was OSCON 2007.

As time rolled on I gave more talks, I learned to speak better, to tell a narrative. I learned to write less like the author of pharmaceutical instructions. More importantly, the open source world began to change. People started making new conferences from what used to be pieces of conferences like OSCON. They made conferences without talks, without schedules, without sponsors and without exhibit halls. Conferences in theaters, bars and parking lots. I started to understand what I liked about my experience at OSCON and it had nothing to do with the manufactured landscape created by O'Reilly but with the experience of flowing through the rivers that fed in to it -- the hallways, bars and bakeries where the community would congregate.

Today I'm at at odds with nearly all my former idols. Sometimes for their grey bearded misogyny or their adherence to tradition and process over their own stated morals and beliefs. And there seems to be a lot of people listening to me, enough that it can get to my head sometimes. And I'm sure they look up to me the same way I did the men (they were unfortunately all men) in the Doubletree Parking Garage at OSCON 2007.

I hope those who read me now will one day see me as I see my former idols. I hope they feel like my conference was, by new standards, impersonal and rigid. That my views are misguided and outdated and that my style of speech and performance lacking theatrics and emotion.

I hope that the future has better ideas than mine and that I might stay open minded enough to listen to them. If I can see past my own history and the investments I've made to line up behind those that will surpass me and my contemporaries then I might be truly rare and truly deserving of all the praise I've never deserved since my first presentation at OSCON 2007.

The Tolerance Of Maturity

Walking around the Ferry Building on a sunny San Francisco day Tim Caswell told me about his idea for a better middleware system. He was experimenting with a continuation passing style that would become connect and I told him it was a worthy experiment, as did many other node developers. These were the unstable days of node, the 0.1.x “Ryan breaks all our modules every other release” days.

When Promises were removed from core in favor of the error-first callback pattern nobody thought it would define compatibility in node's ecosystem the way the Promise API had. The idea was that a base callback API would be a better opportunity for the ecosystem to do a better Promise than core had or that some alternative continuation passing system would emerge and that might be where a standard would form and people would build bigger ecosystems on top. That didn't happen.

Looking at connect you can see what a fork in compatibility looks like. connect was released early enough in node's history and express had such a big user base that people wrote, and even continue to write, a lot of modules in this pattern which conflicts with node's standard callback pattern. For people who don't use connect these modules are worthless. A search for connect on npmjs shows where most of the effort in a compatibility fork goes; tying together modules written for the dominant pattern (in this case for node's callback pattern and sometimes streams) in to a style compatible with connect which wouldn't be necessary if it were using the dominant pattern.

When Tim announced connect the community didn't take a big shit on him the way they treat people who fork compatibility now. Node was immature and it had attracted the kind of people who solve problems like this through experimentation and iteration, and so we all applauded the effort and even those of us who didn't invest in the experiment were eager to see its results.

Nobody thought node could mature as fast as it did. This had a lot to do with its popularity, which attracted contributors, and with limiting the scope of the platform, which encourages people to invest more in the ecosystem. The decisions that were made that enforce compatibility were locked earlier than anyone anticipated by the weight of a giant module ecosystem built on their API. Attempts to change even superficial parts of node core, like the names of standard libraries, were quickly reversed because the cost of breaking compatibility became too high.

One day, long before anyone imagined, node.js became a bad place for experiments like the one Tim had done just a year or so earlier. The greater the module ecosystem the less accepting the community was of ideas that conflicted with the patterns they had already invested in.

Today, node is a phenomenal place for innovation outside of its core patterns: error-first callbacks, streams and the node module system. As embracing as the community is of new ideas on top of the patterns established in core it is hostile to ideas that conflict with established patterns of compatibility. Mature communities don't usually tolerate forks in compatibility, this is true across many platforms and the less academic a community is the less tolerant it will be of ideas of for ideas sake and the node community is anything but academic.

As advantageous as it is for someone to build value in the node ecosystem by leveraging its community and modules it is just as disadvantaged for those that wish to conflict with established patterns that describe compatibility. You cant really have one without the other, the pressure of the ecosystem can propel a project forward when it is compatible as hard as it crushes competing ideas.

If your preferred problem to solve is one that node is done solving then node is not a good place for your experiment. Your ideas would be much better served by distancing them from node. Good luck with your experiment but the time for some experiments are over for us, we've moved on to new problems, new ideas, new experiments in new places, and don't have the capacity as a community to uproot or destroy the immense value that has been created in the ecosystem.

Open Source Ecosystem Growth

I've been trying to quantify the growth of various open source ecosystems. The other day I published a story to Perspective about it. If you have an iOS device you should watch the Perspective story since it allows you to interact with the data in a way you can't with the static charts I'll be showing in this article.

Using data from modulecounts.com we can look at the size of different package repositories on a quarter by quarter basis for the last few years.

This is how people tend to visualize this data. In fact, this chart is very similar to the way the data is shown on modulecounts.com which is good for comparing the absolute number of packages per repository. But, this doesn't illustrate the growth of each repository very well. Rather than compare the absolute size we can look at the number of new packages added each quarter.

This already shows us a lot we couldn't see before. For one quarter CPAN added a huge number of modules, as many at Ruby, but this was nearly invisible in the previous graph. We also see that Ruby and Python are a little closer in growth than the previous chart would have suggested, with Ruby about double Python. This also illustrates pretty well that the first full quarter we have for node's growth it's already larger than Python and that, in terms of growth, it overtakes Ruby by the last quarter of 2012 and takes a big lead by the first quarter of this year.

The biggest criticism you can make of the previous charts it that they tend to visualize a package in each repository as being equal. People tend to dismiss any package system data as an indicator of growth or popularity on the basis that one community publishes "smaller" or "less significant" packages compared to another. One way we can overcome this is by displaying the growth of each repository relative to its own size the previous quarter. This makes the size of individual packages irrelevant since we're looking at growth as a percentage of the repository and rather than comparing size of a repository or new packages added we can compare the percentage growth quarter over quarter.

To me, this is the most interesting data. Many claims were made last year that some of these platforms were losing developers and that the ecosystems were dying but what we actually see is that the growth rate of all these platforms is relatively stable. If these platforms were losing developers, or developers were becoming less engaged, or even if these platforms failed to attract new developers we'd see a bigger decline in growth. Every quarter the size of the repository grows and all of these platforms have to fight harder to maintain a stable growth percentage. Where some might claim that the result of node's incredible growth is a divestment in existing platforms this data would seem to indicate that this is not true, that the available growth of open source platforms is not finite and that each of these platforms is attracting new people and are less in competition with each other than people claim.

It would be my instinct to write off node's first quarter growth as being easy due to the relatively small size of the repository but the fact that it maintains that growth and that by the end of the first quarter of this year it's nearly the same size as Perl and Python's repositories while it actually increases its growth rate it looks more like node's ~25% growth rate is about as stable as the other platforms we're tracking.

I've heard people claim that people are "leaving Ruby for node.js" and while this may be anecdotally true Ruby is attracting enough new developers to maintain a healthy growth rate and node.js explosive growth (100% year over year) can't be explained by any people "switching" from other platforms to node.js. This data would seem to indicate that node.js is attracting a huge following of people who are not already invested in the platforms listed above.

What is also surprising is how close this data shows Ruby and Python. Both are growing at a rate, relative to their existing size, of within a few percentage points.

Now that we're pretty sure that all these platforms are healthy and that this year they grew at about the same rate as last year let's take the average growth rate for last year and predict the absolute growth for the next few years.

If this were the current growth chart it would lead to a lot of divestment in platforms below the middle line, which would be foolish and irrational. This chart assumes that everyone maintains their current, rather healthy, growth rate but because it compares the absolute values it gives the impression that there is divestment below a certain line. This is how we usually look at this data and I hope this illustrates how many false impressions it gives.

The San Francisco Safety Net

There's this story, it's of a young entrepreneur-developer who, rather than accept the traditional road in front of them, takes glorious personal risk to build a business of great importance in their newly adopted city of San Francisco.

It's a great story, I enjoy it every time I hear it, but it's absolute horseshit.

If you're a developer in San Francisco the last thing you're ever in is a position of great personal risk. For you, this is the safest place in the world.

San Francisco is a bleeding city. It may be the worst run city government in America despite its massive budget. The rent prices are increasingly unaffordablefor most of the city's residents and the murder rates in some popular neighborhoods make Oakland start to sound safe. Despite all this San Francisco offers a rare luxury in today's economy to a portion of its residents: near guaranteed employment at a prevailing wage.

When your friends from Austin or Portland or Boston start telling you about how their city is "the next Silicon Valley" (which they think means San Francisco) the polite thing to do is shake your head, congratulate them, and completely disregard their claim. You do this because you don't want to be a dick, they have a nice city -- it smells far less of urine than San Francisco -- it's great that they love it so much, but you know it will never be the next San Francisco.

Startups spread through San Francisco faster than hepatitis in the Tenderloin. Once you spend a month in one company you can, if you like, send a few emails and be making more money in a matter of days. This web of small companies, however temporary their existence might be, form a social safety net for the privileged class of developers who make the city their home.

This helps mitigate the risk of joining a company that might fail, as most startups do. None of the developers will be out of work more than a few weeks full of interviews surviving on their "bonus" of all the vacation time nobody ever takes. This means companies can take bigger risks, that the culture doesn't bat an eye at spending a few million doing something crazy, cause after it all falls apart everyone is going to land on their feet and have another year of cutting edge work stacked onto their resume.

The riskiest thing to do here is take a job at a big boring company that's been around a decade or so. How many opportunities will there be to learn something new, to become one of a dozen new world experts, at an IBM or a Microsoft? There's opportunity to learn new things at companies big and small but a startup's risky nature also affects their technology choices and you'll easily find yourself using libraries that are today "unproven" and tomorrow will drive all the job postings for "ninjas" and "rockstars."

This safety net is what keeps me here. It's not that San Francisco is more creative than Portland or its culture more exciting than New York or its cocktails better than Boston. I'm here because I was already a crazy person with very little aversion to risk who is bad at saving money. I could live almost anywhere but I'd eventually put myself into a bad situation where I'd have to take a job I didn't like or suffer through a prolonged period of unemployment finding the right place.

I'm not alone. Many of my friends are similarly crazy and poor at financial planning. This is how the startup ecosystem feeds itself and grows faster than other cities. Startups need talent and San Francisco offers a reduction in risk and a culture for that talent to thrive in. The more startups born that call San Francisco their home the greater the safety net.

The longer you live here the harder it is to imagine living anywhere else.

Broken Promises

James Coglan wrote a lengthy article about Promises in node.js. In his article James quotes the first of many talks I gave last year tracing the causes of node's success. James makes a convincing argument that Promises do, in fact, have features and that node does not ship with several of those features.

The value of a platform is measured by the value you can pull from the ecosystem to write your program. More than any other metric the success of the node.js ecosystem has been the measure of node's success overall. That success is measured in two parts: the absolute number of modules and the degree of compatibility between those modules.

A platform should provide enough API to ensure compatibility, and no more. A healthy community is a productive one. A platform with more people creating value than that which ships with the platform itself is the mark of a healthy community. Although counter-intuitive, doing less encourages more value to be created so long as people are confident that what they create will be compatible and useful to the largest number of people.

People seem to forget that node originally had Promises. We can argue all day about whether or not those were the right Promises, there was certainly plenty of that before Ryan took them out of core, but the effect of removing them was that flow control was no longer defined by node. This lead to several Promise libraries being published along with a host of other flow control approaches.

The proliferation of JavaScript flow control styles wouldn't have happened if node took Promises in to core. Ryan Dahl, on his best day, would not have been able to design a suitable flow control library because Ryan Dahl writes servers in C for fun. Node's success has as much to do with what Ryan and Isaac have been smart enough to leave out than what they put in.

In the time since Promises were removed from node, a greenfield of opportunity for flow control, the ecosystem has mostly rejected Promises and other similar abstractions in favor of "callback managers" like async. Why? Simple, if you stay at the callback layer you retain the most compatibility with the rest of the ecosystem.

Think of the node module ecosystem as a market experiment. For many years node.js has not shipped with flow control determined by node. The module system has exploded and is on track to eclipse Python's by the end of the year. Many flow control libraries are in the npm registry and none have seen significant adoption because the compatibility trade off is too great. Flow control, as a problem that needs to be solved, is not greater than compatibility and people have managed the problem with the least intrusive approach available which is callback management like async.

An ecosystem is built by multiplying the value of one module by another. Python and Ruby have incredibly interesting approaches to alternate concurrency but their biggest failing is that they have several interesting approaches to concurrency and compatibility between modules written on top of them vary. The majority of value available to a Ruby or Python developer is written for the platform's default IO pattern which is blocking. To choose an alternative concurrency pattern is to limit the amount of value available to you. Similar, although not quite as severe, penalties arise when you build libraries in node that use an alternative to API patterns provided by node core.

A lot can be said, and opinions can differ widely, on what the best API is and what is simpler by any individual's definition of simplicity. I think that my views are best summed up by James' own example comparing a Promise to async.

async.map(inputs, fn, function(error, results) {});

is equivalent to:

list(inputs.map(promisify(fn))).then(
   function(results) {},
   function(error) {}
);

These are both fine APIs but if you walk through the code you do need to understand more about the Promise's API than async's. Without even knowing what async.map does I can assume that this array of inputs is manipulated by this function and then, finally, I will receive the final callback and results. To understand the second API I need to know what inputs.map returns and then what list does to that return value. I can figure out what .then() and its callbacks do without learning all about Promises but inputs.map and list require me to do some research. This means that if I publish a library I can expect that anyone who wants to contribute will need to learn a bit about the Promise API but if I used async there's a good chance they could figure out how things worked without reading through that project's documentation.

I don't think people will have a very hard time learning the second API but hiding the semantics behind syntax complicates it and adds to the cognitive load I need to retain in my own head every time I sit down to write a line of code.

In the very early days of node I recall Ryan saying something that stuck with me: "Things that happen in the future should look like they happen in the future." The best thing about node's callback API is that it requires little to no explanation. Passing a function implies it will be called in the future where composing objects and stacking them together do not.

Two kinds of developers will take syntax > semantics, brand new developers and developers who build the abstractions that hide those semantics. A new developer doesn't understand the semantics of either approach and therefor cannot evaluate what is visible and what is not. Most developers, once comfortable with an API, will take a syntax that hides less of the semantics they'll have to keep in their head to write code effectively.

An abstraction should remove the visible semantics when those semantics don't need to be understood by the end developer. But flow control fails this test because the semantics which might be removed, literally the flow that is being controlled, are operations provided by the final consumer of that API.

The person who writes an API is in the worst possible position to evaluate its simplicity. The author necessarily understands all the semantic complexity hidden by their library and can rarely separate what they are hiding that needs to be understood by the consumer of their library and what doesn't.

The platform must make decisions that encourage compatibility and discourage incompatibility in the ecosystem. There's a list of people I see that attack node for having a "mono-culture" whenever they are given a chance. What they take issue with are decisions that core has made or patterns being advocated in the community that encourage compatibility. Compatibility, in many cases, is at odds with diversity but a culture that encourages incompatibility is no longer a productive one.

Being compatible means that people who have a different opinion about the API or pattern you use for compatibility get pushed out. The key to node's success is to make as few of these decisions as possible and let the community work the rest out. So far, a higher level API for flow control hasn't been important enough for the community to get traction around an alternative to callback management. It turns out that this particular problem is not big enough to build an ecosystem on top of.

Individuals that invested in solving the problems node has standardized will always take issue with those decisions. The fact remains that node standardized far less than its contemporaries and ships with a standard library a fraction the size of any competing platform. What has been standardized is certainly different from what other platforms define but the volume is far less and the diversity of libraries in the community for what would have been the standard library is much greater.

Complaining about callbacks in node is like complaining about significant whitespace in Python. It's a decision that's been made, it's part of the platform, and node's growth seems to indicate that people are doing well with what is available and have less of a problem than James and others would suggest. I see little evidence that this is holding back adoption and the failure of alternative styles of flow control to gain wider adoption supports the decision not to include it as part of core.

If Promises were a "missed opportunity" I'd wonder how much more opportunity node could have handled while it grew faster than any platform I've ever seen. Instead, I think that Promises are one of those things that a vocal minority of intelligent people like to talk about while everyone else continues to build a healthy and diverse ecosystem just fine without.

Put Down That Feed Reader

Feed Readers have been around a long time, some of the earliest ones doubled as Usenet readers, and just like email they have persisted through various technology innovations and shakeups.

The problem typically solved by a Feed Reader is aggregation. More specifically, it's aggregation of medium to long form articles.

Even casual usage of a Feed Reader leads to information processing and triage issues. Feed Readers are not a place where you see only the articles you want to read, you add feeds that you're mostly or even casually interested in. For me, the information a Feed Reader can tell you (publishing site, title, author, picture) isn't always enough information to triage what I'm willing to take the time to read.

A few years ago I stopped using a Feed Reader when I noticed that nearly every article I wanted to read I'd already read after being linked to it on Twitter. Twitter is a great news aggregator. There are tons of people I have some kind of trust relationship with filtering content for me and when they post it it comes with their endorsement of the article. Another great part of Twitter is that there isn't an unread count. I slip in to the stream whenever I like and I'm fine with what I don't see when I'm not engaged.

I've also been using Flipboard for well over a year. Flipboard is a much better aggregator than a Feed Reader ever could be but is not always a great reader.

Feed Readers double as an optimized "reader" applications. This was somewhat frustrating to me because I'd have to context switch too often from triaging what to read and taking the time to actually read long form articles. But when I dumped the Feed Reader I was left with lots and lots of open tabs waiting to be read -- a system I don't recommend and is known to cause anxiety.

Enter Pocket. Pocket is fucking awesome. Anywhere I find an article I want to read (Twitter, Flipboard, Firefox, Chrome) I just send it to Pocket and forget about it until some time in the future when I context switch in to reading mode. I even use the Desktop app for when I feel like taking a break and reading but I'm not on my iPad or iPhone.

Separating aggregation from the reader is my favorite accidental innovation in this whole process. It limits context switching and allows me to accomplish both tasks much faster. A big thanks to Daniel Erickson for telling me about Pocket and their Desktop app which got me started on this.

The system still had one problem. There are a few feeds, about 6, that I genuinely want to read every article. These fell through the cracks the last few years until I found IFTTT. IFTTT is an express train to mashup town so it can be a little confusing to figure out what the hell you're suppose to do with it. One thing it does, very well, is send every new article in a feed to your Pocket account.

There it is, my life is complete :)

PS

These are the feeds IFTTT posts to Pocket: David Simon, Charlie LeDuff's articles on Fox Detroit, Paul Carr @ Pando (I also read his NSFWCorp but with less frequency so I let them come in from Twitter), Clay Shirky's Blog (and god damn do I wish he had an aggregated feed like David Simon does), Asymco, and Tim Ferris.

Hiring

My first task in my new role has been hiring. I've been involved in hiring for a long time but now that its the most important thing for me to get right it's a good time to write about it.

Don't build a team, build a culture. Teams expand, contract, change process and leadership at a decent pace in any startup and the culture is what can hold it together through all that change. Culture is the expression of shared values and interests between people. Culture is the heartbeat of a community. So the central question must be: what are the values and interests of your culture? Hiring is a selection process and you have to identify what you want the culture to be to do it right.

Open Source communities form around projects and technology, a common and shared goal is almost a given, and the more focused a community is on accomplishing their shared goal the more diverse the people that make up that community tend to be provided the leadership doesn't do anything stupid and exclusionary. It's pretty simple when you think about it, a node.js bug might be worked on and tested by a half dozen people from different parts of the world. You can find issues where a mormon, a jew, a bi-sexual and a Canadian all collaborated together without a single interpersonal squabble. That's because focusing on a shared goal makes us temporarily blind to unrelated differences.

If you're creating a culture, and hiring is how you'll build it, you must decide what the shared goals are.

We all get some amount of fulfillment from our work, intellectually and emotionally. A person's temperament and history have a lot to do with what part of their work they derive that fulfillment from. For instance, people who come from a long history of being removed from the product learn to derive fulfillment from the part of their job they control, their code, rather than from anything related to finished product. People who seem abrasive excel at their individual work but that's because they prefer doing things themselves rather than interacting with people which makes them poor collaborators.

Lot's of people like to tackle "hard problems." These people are dangerous. The problem with enjoying hard problems is that you rarely find simple solutions. People who want the word "architect" or "scientist" in their engineering title should be avoided unless you're building a bridge or an artificial limb. "Smart People" are overrated, I once sat in a room with people 10x smarter than me, original Macintosh team smart, and all we did was write a lot more code in our failure to make a usable product than a dumber team could have, a failure so spectacular they wrote a book about it.

Products are built to solve a problem for users. The scope and method of problem solving changes as the product evolves, what can sustain that development process and help keep the product consistent is a culture that values solving the problems of users. People that like finishing, people that like shipping, and people who feel a sense of pride about the product they put in someone's hand more than the lines of code they wrote to get there.

There are plenty of startups where culture means ping pong and free beer and most of the time it's because they share less interest in the product than beer. If you lock people up together to build stuff they'll find a way to identify with each other, if they don't actually share a common interest in making a product they'll find unrelated and personal interests to identify with and your group will grow homogenous as people look to hire more people that seem like them personally. At the same time people will get emotional about changes to their code, solve problems you don't even have, and read Reddit instead of work together to build something great.

Gather

A little over a year ago Max Ogden and I started a company. With some seed funding from Atlas Ventures we went about creating an application that would "bring people together." Our initial release was a little rocky but around the end of the year we got the product to where we had envisioned it.

Its been a great experience. We stayed focused on making a simple product and ended up building and cutting several ideas that distracted from our core value of bringing people together for real life interactions. During this time Atlas Ventures has been nothing short of amazing, they supported us when we needed it and have continued to believe in us even now.

With the money in the bank starting to run out we have some hard decisions. At this time the product hasn't gotten enough users to justify a new round of investment. We still believe in the product and want to see it live on. The current hosting costs are minimal and the money we have can keep it up more than a year if we stop paying ourselves.

Gather will stay up at least through the end of the year and I'm looking at ways to cut the costs even further if we need to. Max and I will seek out new opportunities. I'm already excited about a few and, in case you didn't notice, Max has rocked the world of voxel gaming with less than a month's effort.