052 JSJ Node & NPM with Isaac Schlueter

by woody2shoes on March 22, 2013

Use this link and code JAVAJAB to get 20% off your registration for FluentConf 2013!

Panel

Discussion

01:33 – Isaac Schlueter Introduction

02:33 – Node Backstory

05:37 – Node and New Features

13:30 – Language Accommodations

22:32 – C++

25:19 – New Streams API

30:37 – Semantic Versioning

  • Experimental versions

33:01 – NPM

39:30 – Issac’s Future

41:06 – Discovery

  • Recommendation Engine
  • Exposing Quality of Modules
  • Code Quality

47:18 – Advice for Adopting Node

Picks

Next Week

Software Team Dynamics

Transcript

CHUCK:  You all ready?

JAMISON:  Super ready.

AJ:  So ready. 

JOE:  I was born ready.

MERRICK:  I was molded by ready.

[Laughter]

CHUCK:  Alright.

[Hosting and bandwidth provided by the Blue Box Group. Check them out at  Bluebox.net.]

[This episode is sponsored by Component One, makers of Wijmo. If you need stunning UI elements or awesome graphs and charts, then go to Wijmo.com and check them out.]

CHUCKHey everybody and welcome to Episode 52 of the JavaScript Jabber Show. This week on our panel, we have Joe Eames.

JOE:  Hey there.

CHUCK:  We also have Merrick Christensen.

MERRICK:  What up?

CHUCK:  AJ O’Neal.

AJ:  How do you decide the order each week?

CHUCK:  I just make it up.

AJ:  Okay. It’s only random.

CHUCK:  And Jamison Dance.

JAMISON:  Hey guys.

CHUCK:  I’m Charles Max Wood from DevChat.tv and we have a special guess that’s Isaac. I know I’m going to destroy your last name. Let me see if I can say it… You say it.

ISAAC:  Schlueter.

CHUCK:  Schlueter!

ISAAC:  Yeah.

AJ:  That’s so much easier than I’d ever imagined.

[Laughter]

ISAAC:  I wanted to hear Chuck keep going on that.

JOE:  Yeah, it’s pretty good.

CHUCK:  It has extra constantans in it, it throws me off. And then extra vowels.

MERRICK:  I heard him just crying, “Shu…shu…” [Laughs]

ISSAC:  I have relatives that can’t say it right and it’s their name so…

[Laughter]

CHUCK:  Alright. Well, do you want to introduce yourself real quickly since you haven’t been on the show?

ISAAC:  Sure. I am the author of NPM and I’ve been maintaining Node for the last — Jesus! It’s been almost a year and a half now, a year or so.

CHUCK:  So just a couple small projects that nobody’s heard of, right?

[Laughter]

ISAAC:  Yeah, a handful of little things on GitHub.

CHUCK:  Is there anything else we have to know about you?

ISAAC:  I enjoy changing my Twitter avatar to things that are funny or disturbing or preferably both.

[Laughter]

ISAAC:  And, I don’t know.

CHUCK:  Alright. Well, we really appreciate you coming on the show.

AJ:  That is pretty disturbing dude. You’ve got your face on a really overweight cat. That is pretty gnarly.

[Laughter]

CHUCK:  Yeah, when this comes out in a week, will it have changed?

ISAAC:  Probably. I think it’s the red and white glow that actually makes it even worse.

JAMISON:  Yeah, I was going to say that creepy gradient.

AJ:  Pretty gnarly. So, can you tell us a little bit about your back story with Node?

ISAAC:  Sure. I got into Node very early on. I had been messing around with server-side JS for quite a while because I wanted to stop writing PHP on the server. The fact that I kept switching languages to do client server stuff seemed just like a waste of brain power.

So, I poked around with v8 and messed around with some Spider Monkey stuff. And when I stumbled across Node, I was like, “Okay, this is what I’ve been trying to figure out that I need.” And Ryan had already done it. So, I just kind of jumped on that bandwagon right away. Since then, it was gaining a lot of traction and people were sending these really interesting things to the mailing list that they were doing with Node. But it was always a huge pain to install their module and play with it. So, I wrote a package manager so I could, basically so that I could mess around with the things that people were sharing on the mailing list.

And that ended up kind of becoming somewhat popular and like a fixture in the Node ecosystem. And next thing I knew, I was working at Joyent and helping out with Node itself. And then, Ryan got burnt out on it and wanted to go do other things. So, I stepped up and took over the lead maintenance of the project.

JOE:  Did you go on with Joyent because of your work with NPM then?

ISAAC:  Yeah. Actually, I was looking at another possible job. The job I was at, I wasn’t super happy with. I was kind of planning on quitting regardless. And around the same time, my girlfriend and I were going to move to Oakland because she got a job in Berkeley. And I mentioned to Ryan, “Hey, I’m going to be starting this new start-up with these guys or something.” And he was like, “Oh, no, no, no. Don’t do that. Come to Joyent and you’ll get a pay check.” And I was like, “Oh, that actually sounds kind of nice.”

[Laughter]

AJ:  That’s awesome. 

CHUCK:  So, you get paid to work on Node then?

ISAAC:  I do, yes.

CHUCK:  That is cool of your employer.

ISAAC:  I think of it more as that I get paid to be myself and Node is just something that I do. They are paying me just to be Isaac and I would actually do that for free. But don’t tell them that. But if I wasn’t getting paid, I would have to not be Isaac for a part of that time. And so, I guess they’re just paying me to not waste my time at some other job.

JAMISON:  It seems like you’ve achieved the programmer dream. That seems like one of the most fulfilling things you could do with your career is just do what you want and get paid for it.

ISAAC:  Yeah. You know, it is a tremendous amount of work though, you have no idea when it’s like, “Oh, this thing that you were doing once in a while on the evenings and weekends, what if you did that all day long?” It sounds great except, well I didn’t realize until I was in this position is that you end up just working 24 hours a day if you’re not careful. So, it’s definitely been an interesting good problem to have though, for sure.

JAMISON:  Sure. So, one thing I’ve heard you talk about is how Node wants to hit 1.0 and kind of be done with new features. Can you talk about why? I mean, there are lots of frameworks and tools that are obsessed with new features and love new versions and fixing stuff and adding new things and stuff. Why do you want Node or am I mischaracterizing that?

ISAAC:   No. I don’t think you’re mischaracterizing it at all. I think that some people put it a little more strongly than it actually is. Once we get to 1.0, it won’t be like Node is never getting a new feature again. I’m sure that there still will be. In the real world, we’re going to hit cases that we didn’t anticipate and have to do things to change around. And also when we get new, when we pull in new versions of v8, there may be options or opportunities for optimizing things differently or fixing other JavaScript bugs or what have you. So, there still will be new versions after 1.0.

But the API surface is actually already pretty close to finished. There’s few areas that are still a little bit kind of in motion but they’re pretty much all settling. And certainly our HDP implementation is kind of just a pile of garbage. It works pretty well but it’s got some real unsavory code in there that I’d like to rip out and redo for 0.12.

Anyway, get back to your question. The reason why we want to do this is because really the philosophy of Node is that the core should be rather small. It should do sort of the base level things that you need in order to write really good network programs. And so, what that means is we need TCP, HTTP and datagram servers and clients, DNS servers and clients. Actually, we don’t have a DNS server in core right now, we might not ever. But there’s already a couple in NPM that you could use if you wanted to.

And then above that, even our HTTP implementation is rather low level. Node is not Rails, it’s not even Apache. It’s a very low level HTTP implementation that you can use to write your own server logic. The benefit of doing it that way is that we can get a lot of innovation up above the core in userland, in NPM registry. And the way I look at it is every feature that we add into core is a hard-coded decision that nobody gets to change. You know what I mean? And nobody really gets to innovate on. Unless you want to go the route of Python where you have these very expansive batteries included standard library which is good, in some cases, because you know, you have an XML parser right away.

The problem is that the people working on the core project, there’s always going to be fewer people working on the core project than there are working on all of the modules in NPM, right? And everything that we put in the core, everybody gets. So like, you don’t actually give a damn about XML parsers. What benefit is it to you that Python has an XML parser built in, or even worse that it has three? Also, there ends up being this political battle because if you have a really full-featured core library, there’s no incentive to actually do things in userland in the module space. Since there’s no incentive to do things in the module space, the module space kind of rots away. And if you can’t get your pet module into core, effectively, you can’t use it.

So, we’ve tried to avoid that with Node and with NPM. I think we’ve done a pretty good job. We’ve definitely uncovered new problems, problems that Python doesn’t have. And I mean, I’m picking on Python a little bit here. But I think to varying degrees, it’s true of most platforms. I mean, they all have to set this bar somewhere and wherever you set it, you have a different set of problems.

So with Node, we’ve chosen the problems of discovery. And having a little bit of a more difficult time to figure out which modules are good and which ones I should use. On the flip side, we have a much smaller binary and you don’t get any modules that you’re not using. So, there’s trade-offs there. We also have been able to move a lot more quickly and get a lot more innovation in some of these areas because of that.

CHUCK:  So just to clarify, when you’re talking about modules and adding things to core, these are things that you are sticking on top of what you already get from v8?

ISAAC:  Right. Let me back up a bit then. When I say Node core, what I mean is all of the functionality that’s in the Node binary file. So, all the JavaScript libraries that are bundled in that, v8, libuv, the open SSL bindings that we have. The basic set of stuff that’s covered in the Node API database. When I say modules or user space, what I’m talking about is the tens of thousands of modules that are on the NPM registry that users have created and published. There’s very little ceremony around getting things into that space. So as a result, if you look up and you find that there’s three XML parsers and none of them are really doing what you want, you could just write a new one and publish it and you don’t have to ask anyone’s permission.

As you’d expect, that means we have a lot more copying of things. But on the other hand, we also have a lot more different ideas or different takes on things. There might be one that uses Promises, one that’s written in CoffeeScript, you name it. And even though I think that some of them are bad, like they still get to be used. You know what I mean? And sometimes, if there’s something that I, personally as one of the Node maintainers, if I think, “Well, that’s a terrible idea. And no, I’m not going to bundle that in core. That’s awful, what are you doing?” Like, people should still be able to kind of experiment with that even if I think it’s stupid.

AJ:  Yeah. And I loved seeing the new streams API in NPM. I thought it was really cool seeing some of the core things in Node kind of being able to innovate and experiment inside of the userland stuff too.

ISAAC:  Yeah. And it’s way more flexible in that way because now we can take this — if you want to use the new streams API, you don’t have to wait for 0.10. Let’s say in 0.10, we change some API and that really messes up your day for whatever reason. And it’s going to take you a few months to refactor everything to work right or whatever, like we try to avoid that but it’s kind of unavoidable sometimes. And if you’re in that state and you still want to use the new streams API, you can because it’s in NPM. You just pull it in.

I really want to continue this pattern. It’s been very effective. Fedor Indutny is also working on a new TLS implementation which is called tlsnappy and that’s also in NPM. It’s its own project on Github. And at some point, we’re probably going to pull that project into Node core. But in the meantime, if you want faster TLS, if you want to use that experimental stuff, you can totally do that.

CHUCK:  So, do you usually write your own packages into core or do you pull something that already exists in the core?

ISAAC:  That’s kind of a complicated question, I guess. I do write a lot of NPM modules and maintain a bunch of them. It’s relatively recently that we’ve been able to really experiment or innovate with new core libraries in NPM. But certainly, it’s very valuable when it is possible because you can get more people using it, more people looking at it, you know?

MERRICK:  Yeah.

I think a lot of people sort of dismiss unstable releases of Node. You know, they look at it and say, “Well, that’s not stable. I’m not going to bother testing it out.” But they’re a little more open to experimenting with stuff that’s on NPM.

MERRICK:  Yeah, That’s a good point. So, when you talk about kind of locking down Node to the core functionality, how is Node going to respond to language innovations, like you see a lot of, for example, modules in ECMAScript 6 classes, things like that. Is Node going to try and update to accommodate the new language kind of features?

ISAAC:  Well, I think that’s really hard to say. I mean, there is definitely a bit of a chicken and egg problem in JavaScript language implementations because you have this new language feature let’s say, and it does roughly the same thing as the old language feature. And at first, it’s maybe a little bit slower because it’s not worth their effort to optimize it until some people are using it, right?

MERRICK:  Sure.

ISAAC:  But nobody wants to use it until it’s fast. So, if it’s doing just the same thing as the old language feature but slower, like no. Node, of course, will never ever pull that in. Now over time, this gradual iteration gets to a point where classes are faster than using just regular functions in the new keyword, or where modules are faster than running code that isn’t module code. But that’s going to take a while. And I try not to plan more than six or eight months out. So, I don’t know. I mean, sure. Maybe but I wouldn’t really bet the farm on it.

I think that in userland, of course, people will start using new language features as soon as they’re available. They already do this, right? But sort of the point of keeping Node really small and very, very stable and getting to the place where we can keep it very, very stable is that people can innovate on it and kind of have Node be a known quantity. If we’re swapping it out and saying like, “Oh, don’t use require anymore, use import.” Yeah, all those Node programs you’ve already written are now broken, right? That’s completely, like we could never do that. That would make Python 3.0 look like a patch update by comparison.

[Laughter]

ISAAC:  I feel bad, Python’s a great language. Every language is the greatest. Every program is wonderful and every programmer does great work. I’m not trying to pick on Python here. But I think that was a mistake. Making big changes to a platform should just never happen. And same thing with Perl 6, right? It’s a different language. It has different semantics around certain things, like we can’t do that. If we ever do that, we’ll just call it something different. It won’t be called Node. It will be called…

[Crosstalk]

[Laughter]

JAMISON:  That brings up a question that’s kind of related to that it’s about Node and TC39. I know some of the people in the Node community are very down with TC39. I think you’re not one of them, I think you’re a little bit more open from the stuff I’ve seen you say. But is there anyone on the Node core team, or that works closely with the Node, that is also on TC39?

ISAAC:  I don’t think so. A lot of people who are on TC39 are kind of into JavaScript, as I imagine. And so, a bunch of them do use Node occasionally, at least whenever they want to write little server programs or whatever. But I don’t think there’s any like Node community all-stars or really serious contributors who are working on TC39.

AJ:  That’s too bad.

ISAAC:  Well, there are some wheels turning. I think we’re looking into whether or not it might be worthwhile to have a Node person on TC39, or a Joyent person, or something like that just to kind of keep things going in a direction that’s good for us. But really, I mean, we’ve tried hard to not really care too much about the language. Like one of the benefits of Node, one of the reasons that we’ve been able to get to the level of stability that we have so much faster than a platform like Python or Perl or Ruby is that we’re not messing around with the language. That’s a huge, huge bunch of work that goes into making a good platform that we just don’t have to do. The language is handed to us, and all we have to do is figure out how to like express this asynchronous IO paradigm in that language. It would basically be a whole rewrite but you could basically take and write bind libuv to some other language with open SSL and a TCP parser, or sorry an HTTP parser, and have something very, very similar to Node in a different language.

MERRICK:  Luvit?

ISAAC:  Exactly, Luvit’s a good example. There was also some other stuff that was sort or similar. Oh gosh, I think I’m going to get this wrong. Daria I think is the name of the language that’s binding with libev. I’m sure I’m getting that wrong; some woman’s name language. Anyway, there’s a few others. And they’re kind of really very Node-ish because really Node is like libuv in JavaScript.

JOE:  I find that to be kind of a crazy statement, eye opening for sure and revealing but crazy. Well, you know, Node was like inspired by how JavaScript handled IO, right? It was the, “Hey, JavaScript handles IO this way, let’s embrace that and put that on the server side.” And the fact that the paradigm really isn’t specific to JavaScript, just inspired by it?

ISAAC:  Well, I think that’s actually less true than you imagine. So before…

JOE:  Edumacate me.

ISAAC:  [Chuckles] Before writing Node, Ryan wrote a lot of Ruby.

AJ:  Wasn’t the HTTP parser for event machine? Isn’t that what he wrote it for?

ISAAC:  I don’t remember exactly what he wrote it for. And there’s, depending on who you ask, he stole it from Zedd Shaw.

[Laughter]

ISAAC:  Completely, completely, patently untrue. So, he wrote the sort of what ended up becoming Node on top of libev and libio, becoming somewhat frustrated with always having to write HTTP server stuff in C because writing it in Ruby was too slow. And every time you’d cross between Ruby and C, you take this big performance hit. And actually, I think his first choice of language that he looked into was Haskell. And the idea was like every request would be this, like an HTTP server would be one function that takes a request and returns a response and does some asynchronous stuff in the middle.

JOE:  My jaw just hit the floor.

MERRICK:  Yeah. It probably wouldn’t have been as popular if he went with Haskell.

ISAAC:  No.

[Laughter]

ISAAC:  Again, every language is great and every program is…

[Laughter]

[Crosstalk]

JOE:  Ryan was setting out to change the world when he did this, right?

ISAAC:  Yeah. But his goal, he wasn’t really into JavaScript, he basically stumbled upon JavaScript because v8 had just recently been open sourced and was this really advanced high speed virtual machine that he could use to do his stuff that he was already doing in C. And like, if you just use libuv, you can write a program that’s very, very similar to a Node program but is completely in C.

Now, it’s going to be like a hundred times as many lines and it’s going to be a much bigger pain in the ass to go through and write and read and make changes to because JavaScript’s just a much more expressive language. But you can do callbacks in C, they’re just called function pointers. You know what I mean? And you have to be very careful about how you set everything up and what kind of arguments you’re passing, and so on and so forth. But it’s really very similar to Node and that’s really where Node came from. But the fact that JavaScript was chosen, was mostly owing to the fact that JavaScript did not already have an IO paradigm. So, it was feasible to invent something completely new for it. Whereas if you were to write Node in Ruby or Python, or if you use EventMachine, or what’s the other one? Twisted, Tornado, and there’s AnyEvent in Perl, it’s great but it’s a little bit like being on a Paleo Diet and going to McDonald’s. You had to be very careful what you order.

So, it’s really easy to say, “Okay, I need a module that reads a comma-delimited files or something. And the next thing you know, you’re blocking every request to read your config files because they’re being read synchronously. So in Node, the idea was to make it the default to do things the right way.

CHUCK:  I have a question on kind of a different tactic. And that is, we’re talking to you about an engine that we use to run our JavaScript on our servers. And you probably spend most of your time writing C++ in order to make it work. Do you feel that there’s a disconnect at all between the two?

ISAAC:  I actually write mostly JavaScript.

CHUCK:  Oh, really?

ISAAC:  Yeah. Node is written mostly in JavaScript. I guess that’s not true. If you count lines of code in the project, it’s mostly C++ but that’s because it’s mostly v8. But Node itself, we have a source folder which has some wrappers around libuv constructs and a few things that you just kind of have to do in C++ and then we try to get into the JavaScript layer as quickly fast as possible though.

JOE:  Yeah. I saw that in your talk. What’s like the percentage if you took out the v8? Or if you were excluding that and not considering that, what’s the ratio of JavaScript code to C++ code?

ISAAC:  Let me see. If I had to guess, I would say that there’s about 28314 lines of C++ in the source folder.

[Laughter]

ISAAC:  And…

JOE:  Just roughly?

ISSAC:  Just off the top of my head, there’s about 21265 lines of JavaScript in the lib folder.

JOE:  Awesome.

ISAAC:  That’s just a guestimate. Don’t quote me on that, I don’t know.

[Laughter]

CHUCK:  So, how much of that C and C++ code is external libraries that you depend on?

ISAAC:  Oh no, that’s just the source folder. Everything that’s in depths is external stuff.

AJ:  And remember that one line of JavaScript is roughly equivalent to 25 lines of C.

ISAAC:  Is it 25? That sounds about right. But is there some data or something?

MERRICK:  I wish.

ISAAC:  Oh, well.

AJ:  I wrote this thing LibEVN with a friend that is Evented Networking in C and just followed the Node API and I remember it being about that much.

ISAAC:  Yeah, that sounds about right to me, probably about 25 to one. Also, most Node users are not C++ programmers. So, most Node users are much more comfortable digging through like tracking down bugs and sending us patches if we do more of our work in JavaScript. Plus v8 is really good at optimizing JavaScript. So, if you write it, if you write really small functions, if you try to make sure that you always send the same types of things around, then v8 will do a really great job optimizing your JS.

JAMISON:  One thing I think would be cool to talk about is the New Streams API because that’s one of the big changes in Node 0.10. What motivated that change? I guess we should talk about what they are first, for people that don’t really know.

ISAAC:  So basically, we created a unified streams API. There kind of wasn’t one. There was sort of this pattern that we were using and there was a stream base class but you had to implement everything about a stream yourself. So, a stream is a sequence of data that goes from a start to a finish. And they might be, the data might be separated in time like a flow of data from like a TCP connection or if you’re reading a file start to finish, or if you are doing things like this. And then, there’s also writable streams which are the same thing but you’re sending it to someone else.

So, we had this pattern where a readable stream would emit data events. And you could call pause to tell it, “Hey, hold on a second. Don’t give me anymore data just yet.” And you could call resume to tell it, “Okay, now I’m ready to receive more.” Then on the writable side, you would call write and pass in some data and then write would return true if the other side wanted you to send more data or it would return false if it wanted you to slow down.

Then, we added this thing called pipe which is like, you could take a readable thing and say, “All the data that comes out of this, write it into the writable side and pause and resume at the right times based on the return from the right method.” That was kind of good. The problem that we found, there was a handful of problems that we found. The first is that pause is actually only advisory. And so, even if you call pause, you might still get one or two other data events coming through. And this mostly was owing to the way that our HTTP and TCP interaction worked. Because TCP has its own framing but then HTTP is framed messages on top of that. So in one TCP packet, you may get the headers, you may get like the end of one message and then the headers and then the start of the next. Like all in one TCP chunk.

AJ:  Yeah.

ISAAC:  And so, with HTTP then, we needed to omit the end on the first one, start a new message, and then we have this little snippet of body. So, what we would do is just send it right away. Ended up working around that but still it kind of sucks that you were getting like, you get an HTTP request and like it just starts throwing data at you right away before you ever even tell it that you want to start reading from it.

So, if you want to get like, let’s say look at the headers and see if the authorization is valid, and you need to go look in a database to see that. Okay, so you go look in the database and you come back and now you’ve missed the first bit of the response, or the first bit of the request body. So, that sucks. And also, what I found a lot is if you’re writing these parsers that are taking data in chunks, sometimes what you really want to say is like, “Give me ten bytes of data.”

AJ:  Yeah, because you want to be able to predictably…

ISAAC:  Yeah, that’s so hard to do with data events because now what you need to do is you get a data event, you see if it has enough. If it doesn’t have enough, you store it. Then you wait for there to be more and then you’re going to get too much. Then you just kind of have to like re-emit it or something with the rest? I don’t know. It’s just kind of a mess.

Also, every stream in Node had its own like top to bottom implementation of this stuff. Some of them were doing buffering differently, some of them had set encodings, some of them didn’t. For some of them, pause was advisory. For some of them, it was a strict start buffering if you need to. And I was constantly finding bugs with one that didn’t exist in another. It was a huge, huge, huge pain.

So, we have long known that we needed a consistent implementation of this. And especially because writing streams in userland is really hard like we had all these talks saying, “Streams are great! You should use streams!” And then people would sit down and be like, “Okay. Well, I’m going to write this thing as a stream,” and then look at the API and there’s just no help. It’s very difficult. You know what I mean? There’s a lot of moving parts. You have to get them all exactly right.

AJ:  Yeah. What I love about the new one is that it’s easy to do the right thing as opposed to the old one it was easy to do the wrong thing or to skip doing the right thing.

ISAAC:  Yeah. And then, of course, everything’s easy until you deal with TCP and especially TLS where there’s lots of these kind of very specific semantics around how you do things. But I’m pretty happy with where it ended up.

MERRICK:  Yeah, it’s great.

AJ:  I’ve read over and I love it. I think that it solves all my pain points.

ISAAC:  Cool.

CHUCK:  So how much refinement do you think it’s going to go through between now and the next version?

ISAAC:  Streams API, I don’t think it will go through much. I think that’s pretty solid. We may find some edge cases. There’s already a few sort of minor bugs that don’t actually change any semantics but things that people have found that are probably going to go into 0.10.1. Between now and .012, I don’t expect any changes there though.

CHUCK:  Nice. So, do you follow semantic versioning on Node?

ISAAC:  Sort of. I mean, our versions have a certain semantics sort of built into them. I think that stuff like Semver.org is kind of okay but that’s like one thing that you could communicate with version numbers. I think what’s more important is that version numbers can carry semantics but it’s really up to humans to put those semantics there, right?

So, in Node, every even numbered minor version is a stable family, I guess. So, from 0.10.0 up through 0.10.35, we’re going to have a consistent API, consistent binary interface, the whole bit. We will fix bugs but go very, very far out of our way to avoid adding any API, even if it means that we keep a bug around. Once we go to the next stable branch, then that’s where you’ll see actual bug fixes, changes to API’s, things that are kind of redesigned or reconsidered. And then, so like 0.11 is what we’re going to be hacking on until it becomes stable. And then we bless it, I write a long blog post, and we call it 0.12.

CHUCK:  How many people use 0.11 as you’re working on it? How common is it for people to use the experimental version?

ISAAC:  I think people have gotten a little bit more friendly with using experimental versions, at least to test things out. But certainly, nobody really uses it in production in a serious way. So, it’s mostly people who are like writing modules. I use it, like I pretty much live on Node master as my real local Node install for the whole time we’re developing it. Even if that’s sometimes rather painful, I try to just kind of feel the pain and fix it rather than fall back to the stable version.

CHUCK:  Yeah, it seems like a good way to go to figure out where the problems are.

ISAAC:  Yeah. And especially because stuff like, I use Node for a lot of things that I do. I write some Node modules and do stuff with NPM, obviously. And so, when we break something that breaks NPM, at least, I notice right away which should pretty much never do. That always means that something went wrong.

JAMISON:  Can we talk a little bit about NPM?

ISAAC:  Sure.

JAMISON:  I think we’ve whacks poetic about NPM before on this show. But it’s so good. Oh, man! It’s so good that it makes using other languages worse simply because of their package management. Even if the language is something I enjoy more than JavaScript, it’s painful to not be able to easily install third party stuff.

ISAAC:  Oh! Thanks.

JAMISON:  It seems like NPM is pretty stable. I mean, are you still iterating on NPM? Are you still trying to add features and things?

ISAAC:  I think, I mean, NPM is a little bit more of a playground than Node. You know, it’s more easy to justify adding a feature to it because it’s not something that’s going to be running on your production HTTP server. And if there’s something that will make your life a little more pleasant, yeah sure, we should add that to NPM. That being said, I think the main reason why it hasn’t been iterated on as much, is because I’ve been busy with Node. There are some definite like….

JAMISON:  What are the pain points? I mean, what would you want to iterate on with NPM? As just a casual user, it just seems like it works and I can’t imagine more stuff that it would do besides install packages.

ISAAC:  Well, it probably shouldn’t be doing more stuff, that’s for sure. But the way it’s actually implemented is pretty awful. And I mean, I suppose I have a different view of NPM than most people. Like the code base is a complete and just a god-awful, whole — it’s like everything about NPM is great except for the code.

[Laughter]

ISAAC:  The patterns are great, the community is great, the way it’s been taken up has been awesome. But like, it’s something that’s just kind of evolved and it originally was developed before we had NPM, obviously. So, some of the decisions that went into it are kind of like stupid. In terms of like how its modules are broken up and how the code is organized, it’s just kind of a mess.

AJ:  Have you never rewritten from scratch?

ISAAC:  I’ve rewritten it from scratch a few times but not in the last couple of years. So, in the first few iterations, yeah, I basically rewrote it from scratch about every couple of months. But yes, since it’s become a stable thing, it really kind of hurts to break stuff. A lot of people get really upset. So, that makes a rewrite a lot more difficult.

There is this one file though, the file that actually — you’re going to laugh. But like the file that does two things basically, it takes a package descriptor, like [inaudible] version, or a URL, or a path or what have you, it takes that and it puts it in the cache folder. Then it also takes whatever is in the cache folder and puts it in a particular location.

That file is like 1200 lines which is like roughly ten times what it probably should be. I mean, it should be its own package that lives apart from everything else and it’s just in the NPM registry. And the fact that it’s not and that it’s all kind of like tightly woven into everything else is just kind of awful. So, there’s a lot of stuff like that.

AJ:  That’s a cool idea. You hear about that with compilers a lot, like self-hosted compilers. But a self-hosted package manager, where NPM is made up of packages that you install with NPM.

ISAAC:  Well, it kind of already is that way. It’s just not done as well as it could be. Once upon a time, the first thing that NPM always did was install it self. And it’s only since the mid 0.6 releases where it builds with Node.

AJ:  So basically, it sounds like there’s a lot of internal clean up that you would like to do that wouldn’t necessarily change behavior but that would make it easier to read and understand and stuff?

ISAAC:  Yeah. Also, it would enable us to make the shrink wrap command actually do like bi-level consistency. So you could say, like if I shrink wrap a particular version number but if that version number changes or if the check sum changes, then abort. Call that an error. The way it’s implemented right now, it was originally written using the system Tar and Gzip programs because there was no JavaScript Tar implementation and there was no Gzip in Node. So, it’s still doing a lot of like stupid things where it’s copying files more frequently than it needs to and into more places than it needs to. And so, really, all of that stuff can be really streamlined. And I think that we can make it so that installing a package takes, does about a third as much file systems work than it currently does which is a big portion of it’s time that it’s spending now. So, we can make it a lot faster, more reusable, better abstracted chunks and so on.

CHUCK:  Now that you’re maintaining Node, how much time do you get to spend on NPM?

ISAAC:  It depends on how much time I’m spending on Node, basically the rest, but not that often. Every few weeks, I’ll sit down and go through a bunch of bugs and pull wrecks and try to kind of clean things up or keep stuff from completely falling apart. Lately, I’ve gotten a lot of help from a few people who are going through and sort of testing bugs and making sure that they’re still valid and stuff like that. So, that’s been great. But yeah, it’s definitely a little bit of a number two priority at the moment for me.

CHUCK:  Yeah, we should to make Jamison take over maintaining NPM for you.

JAMISON:  Oh yeah, you’d want that for sure.

[Laughter]

MERRICK:  You would want that, Jamison’s amazing. Everything he touches turns to gold or algorithms.

ISAAC:  Really, what I want is somebody to just like rewrite it.

AJ:  He’ll deliver it to you all packaged up and beautiful.

ISAAC:  Yeah, rewrite a completely new thing called NPM2 and I’ll take a look at it and be like, “Yep, this is better. It’s yours now.”

[Laughter]

ISAAC:  I don’t know. It would have to be exactly the same functional API though because otherwise, it would be a nightmare to get everybody to start using it.

CHUCK:  Yeah.

JAMISON:  Alright. I’ll keep that in mind when I’m considering all the free time I have and what projects I want to take on.

[Laughter]

ISAAC:  I think it’s sort of like a solved problem, like why would you spend your — waste your day doing that?

AJ:  To make you feel better, obviously.

[Laughter]

JAMISON:  So, you said you don’t want to talk, or you don’t plan too much about the future but what does the future hold for you? Do you want to be doing Node like five years from now? Do you imagine it will just be done and you’ll be moved on to other things or what?

ISAAC:  I sure hope that maintaining Node is not a full time job in five years. If it is, I hope it’s not mine. I don’t think I would last that long. My hope is that we can automate more of what goes into building releases and sort of managing the project. There’s a guy who’s recently started at Joyent called, his name is TJ Fontaine and he’s been doing stuff with Jenkins and automated continuous integration stuff. And eventually, we’re planning on — we’re going to get Nightlies out soon. And eventually, just automate the entire release process.

CHUCK:  That would be really, really cool.

ISAAC:  Oh, yeah! Life would be about the same for everybody else. But for me, it would just be so nice.

[Laughter]

ISAAC:  There’s so much manual process, so many manual steps that go into that. And every time we add a new kind of installer or a new kind of binary, it’s like just another manual step, right? So, it’s gotten kind of out of hand. And also, if we’re not changing the API much, then that means that we could just have new releases that are just kind of pegged to a new v8 version and not have to worry too much about that stuff breaking. Stuff won’t break in new ways unless you change it.

CHUCK:  Yeah. And so then, you’re only adding new packages to core when you absolutely need them. Otherwise, you’re just sticking with v8 and everything else that you already have.

ISAAC:  Yep. Pretty much.

JAMISON:  One thing you mentioned with Node and with NPM and the philosophy of keeping everything in userland, that’s a problem is discovery. And I know the way I do it is just look at the number of stars on Github or whatever, like Google Node, whatever this module is that I need and then find the one with the most stars. Is NPM thinking about doing anything to help with discovery? What ideas do you have to make discovery easier beyond just what’s popular on GitHub?

ISAAC:  So, I think we can do a little bit better. We have the data to do a little bit better than just what’s popular on Github. What we don’t have is like the templates in implementation and code to crunch the data. So, what I’d like to do is actually augment what’s popular on Github with what’s popular in NPM. And in particular, what’s popular with module authors that you think are good authorities on the subject.

JAMISON:  Kind of weighted based on certain people’s reputation or something.

ISAAC:  Right. I think just to paint in really broad strokes, from what I’ve seen, there is a couple of different kinds of schools of thinking about how you should do modules and how you should organize Node programs. And I sort of think of these as like the Substack school, the TJ Holowaychuk School of Thinking, and the Nodejitsu style of doing things. And they’re all very good in their own way. But there will be a very different reaction to a particular module based on whether or not it’s the kind of style that you like.

So, what I’d like to do is somehow figure out a way to sort of algorithmically do that. So if you — and also to let new approaches sort of emerge out of this ecosystem, right? Since that’s how these things came about. But if you tend to like modules, like the things that Substack is writing, then maybe this other thing that is liked by other people who like the same thing, or is from a module author who likes those same modules, you’ll see that as more of a higher ranking option.

CHUCK:  Interesting. So, you’re talking about sort of a recommendation engine, sort of like what you’d get on like Netflix or something.

ISAAC:  Yeah, kind of. And I think that it’s something that we can conceivably do in a deterministic, algorithmic way. Another thing that I’d like to start doing is exposing quality of modules. So, does this thing actually build on this or that version of this OS? And then, maybe you can filter so that you can say, “Only show me modules that all their tests pass on Linux or something.”

AJ:  To that end, what about something like a generic JSHint that doesn’t check style, just checks like code quality type stuff that you’d get a thumbs up or a thumbs down.

JAMISON:  Like is there an objective answer to code quality that you could include in a code parser or something? I don’t know…

AJ:  No. Like if you just take the JSHint options that aren’t based on styling, so you’re not like indenting or whether you do comma first or whatever. Don’t check on those things but check on the things that actually affect whether or not the code will run as expected. Like if functions are being declared inside of ‘if’ blocks and that kind of stuff where if you don’t understand how JavaScript works, or even if you do and you’re doing that, you’re making potential mistakes kind of thing. A Goldstar system like this passes these options of JSHint or doesn’t.

ISAAC:  I think CPAN is actually the definite leader in this area. They have a measure of quality which is spelled with a K to not tell you that it’s…[Crosstalk] Yeah, for that, but also because it’s not just strictly quality. It’s like a measure of a few different things. Some of which you might care about and some of which you might not. But overall, it’s a heuristic and they don’t try to say it’s anything other than a heuristic. But it includes stuff like whether or not this thing has a build script, whether or not it’s executable, whether or not it actually builds on different operating systems if the test passed, and so and so forth, or if it’s using any deprecated APIs, that kind of thing.

And I think that that’s definitely something we could do. Like, if you have a module that works with Node 0.6 but doesn’t work with 0.8 or 0.10, it’d be nice to know that right away. Like this thing hasn’t been touched but not because it’s finished, because it’s just broken and old and nobody cares about it. On the other hand, if there’s a module that hasn’t been touched because it’s done and it works and everybody’s using it, then maybe that’s kind of a good thing.

CHUCK:  Alright. Well, we’re almost out of time. Are there any other topics that we would be remiss in not covering?

AJ:  I was just going to say I’m just super excited that now, you can run Node in strict mode and all the core modules are strict mode compliant. That makes me so happy because now you’ve got jQuery, Node, Grunt and I’m sure there’s another dozen other big name projects that are all strict mode now.

ISAAC:  Yeah. We don’t enforce it but we do definitely enable it. And it’s nice that there’s that v8 flag that can turn it on. I mean, that might be another thing in quality like if it fails to run in strict mode, then there’s some kind of problem. The thing that, really honestly, I’m still so upset about is the octal number literals. I accepted that most JavaScript programs don’t really care about octal literals. But like Node programs care a lot about those. We have so many octal literals, it’s ridiculous. And now, we have to do parser and give a string or something.

JAMISON:  Every time AJ mentions the words use strict, octal literals are his strongman, I don’t know. They are just his hot blood topic.

[Laughter]

AJ:  I wanted to ask one question and I know we don’t have a ton of time to spend on this. But what’s your advice for people who need some encouragement to get it adopted in their enterprise stack? I mean, a lot of people have sort of like, a lot of the Node strategies for deployment are restarting the process because people still aren’t, there’s not a lot of documentation on finding memory leaks for example or debugging. I know that Node has come a long way with that. Is there anything that you can do to try to encourage people for the adopting Node?

ISAAC:  There’s actually four companies that I mentioned in the blog post. And I’m very grateful to all four of them for what they’re doing for the Node project. But in particular, if you want to use Node in your company, they’re all kind of worth mentioning.

The first is Joyent, who is my employer. They provide really good hosting and they’ve actually made a lot of tools that make it, at least, reasonable to track down. I mean, tracking down memory release in C programs is also kind of tricky and our tooling is not quite as good as Valgrind but there’s a lot of stuff with DTrace and MDB where you can actually take a core and see where your memory, profile your memory and do these other really interesting things to trace how your program is running.

The next one up is The Node Firm which does a lot of consulting and support and they’re not like a normal consulting firm. They’re not there to kind of run in and build a Node program for you and then run out. Their goal is to actually train up your staff and talk to the people who can make decisions and make them feel a little better about using Node. And also, provide support for when things go sideways.

The third one is this new company StrongLoop which is actually, they’re actually building a Node distro that they’re supporting their particular distro of Node, their distribution which has Node and it’s bundled with a handful of modules that they’re verifying actually work with that build of Node.

The last is Iris Couch which actually is the company that runs the NPM registry. So, if you’re using the registry internal in your organization, they have something called Iris NPM which you can use to have exactly the same kind of high availability stuff that the main registry uses but in your firewall, in your data center.

MERRICK:  Oh, cool!

ISAAC:  And I think that in this space, it’s really growing quickly. A lot of companies are starting to kind of see Node as a real thing. They see Microsoft using it. They see Joyent using it. They see Voxer becoming this big popular thing and those sites all kind of seem to be staying up and not getting hacked too badly. So, maybe we can try this. I think that a lot of the benefits of writing network programs relatively easily is worth taking a little bit of a risk and that risk is lowering as more services come into this space.

CHUCK:  Alright. Well, let’s go ahead and get into the picks. I’m sorry to cut this off. I’m really excited about all the stuff that we’ve talked about. But yeah, we’re up on our time. I’m going to go ahead and make Joe start us off with picks.

JOE:  Alright. So, I’ve got just one main pick that I’m going to pick here. It’s a book called ‘Wild at Heart’. It’s kind of a very introspective book for men and it’s about the nature of being a man. It’s written from a Christian point of view. It’s a really good book. It’s a little different on typical self-help book that I read but I really liked it quite a bit. So, I’m going to pick that.

JAMISON:  If you read that book, do you just instantly grow a mustache?

[Laughter]

JOE:  Yes.

MERRICK:  I could use a book like that.

AJ:  I have a book on how to become a hipster.

MERRICK:  You know, I have a hormone disorder.

[Laughter]

CHUCK:  Alright. Jamison, what are your picks?

JAMISON:  I have two picks. One pick is the band, the Aquabats. I’ve been to a few of their shows and it was along time ago. I was just feeling sad and I Googled a bunch of their videos on YouTube, and they’re amazing. It’s a wonderful childlike Ska music. It’s so good.

CHUCK:  Yey! The world is right again. Jamison picked music!

JAMISON:  Yeah, it’s been awhile. I found that there’s a lot of Aquabats fans around here too, I guess. It makes sense. I think they’re from Utah.

My other pick is actually a blog post by Isaac, not to stroke your ego too much or anything. But he’s talking about user feedback and how to correctly respond to user feedback that seems crazy to you by thinking about how — it’s probably not crazy. There’s probably something behind that even if the actual request seems weird. So, it’s a good one. Check it out.

CHUCK:  Alright. Merrick, what are your picks?

MERRICK:  So, I actually have three picks. The first one is going to be O’Reilly’s Fluent Conference. They draw in all sorts of different, just great speakers that are really big names like Brendan Eich and Paul Irish, these kinds of guys. They cover a lot of stuff, like Node, HTML 5. It’s a big conference. You can go to FluentConf.com for the schedule, happens May 28th to the 30th. It’s at the Hilton Union Square in San Francisco. And for our listeners, you can actually get 20% off on your ticket, using JAVAJAB. Unfortunately, that’s the code we got and that gives you 20% off on the registration. So, hopefully you guys, I’ll see you there.

The second pick is ‘So Good They Can’t Ignore You’ by Cal Newport. It’s a book. I look at people like Isaac or Ryan and these guys. We all kind of joke, they have the programmer dream. The book ‘So Good They Can’t Ignore You’ is a heavy focus on skills rather than passion to create work that you have passion about and stories like theirs, I kind of see emulated in the book.

The last one is ‘Heart of the Swarm’. Star Craft II expansion came out. It’s awesome. And that’s it.

CHUCK:  Awesome. AJ, what are your picks?

AJ:  Oh, boy! All of them, actually. I have everything you can pick, I’ve got today. So quick, GruntJS, amazing. I don’t know why I didn’t take the time to learn it before. Muchas gracias to Merrick because he sat down with me for about 40 minutes and just showed me how to convert my project over. And then, I went and I made a screen cast on how to do it as well.

JSHint, I might have picked that before but I love it. It just makes life so much easier. Gimp, for all the times when Photoshop is too complicated.

MERRICK:  You just want that programmer UI.

AJ:  I couldn’t figure out how to do something in Illustrator. And so, I opened up Gimp and I got it done in like two seconds and I was like, “Hurrah!” Okay, maybe not two seconds because Gimp is still kind of complicated.

But anyway, Hitchhikers Guide to the Galaxy. There’s another book, it’s called ‘And Another Thing…’. I just read the first page of it, seems hilarious.

CHUCK:  Who wrote it? Because didn’t Donald Adams pass away?

JAMISON:  Douglas Adams.

AJ:  Yeah. So this guy is — I don’t know how to pronounce it. Eoin Colfer. I think it’s just Owen, it’s just spelled weird. I think what they did was, he had already had some manuscripts or something that he had started for the book. And so, they raided his computer and got the gist of what the last book was going to be. And then they handed it off to this guy to let him kind of take some creative liberty and work through it and finish it.

And then, let’s see. Last.fm has a free download section. And one of their free downloads is a song that’s probably way too not cool for anyone to like but me, but it’s called Hawaii and it’s by Blackbird and Blackbird. So, I’ll post all that in the notes.

CHUCK:  Alright. I’ll go ahead and go next. My first pick, I had this problem where Mac OS kept telling me that my boot hard drive was filling up. And so, I started looking around to see where all of the space was taken up and moved a bunch of stuff off of my SSD onto the other drives I have in my Mac Pro. And it was still complaining that it was close to full. I realized that my downloads directory was full of stuff. And it was full of some stuff that I wanted to keep and some stuff that I wanted to not keep. And so, I looked around and I picked a program called Hazel and I heard about it on the Mac Power Users Podcast. So, I’m probably going to pick that one too.

But what it does is you can set up rules for it to watch a particular folder or a set of folders. And then, just apply those rules to that folder. So, for example, if I download something that is a .zip, then Hazel will automatically unzip it for me. And then, it has other rules like if it’s a .app, then it will move it into my applications folder. If it’s a .dmg, then it will move it into my installers folder and it moves it off of my main disc so that I can just deal with it. I’m really, really happy with it, really loving it.

And when I set up the rules, it actually did a whole bunch of other stuff just to get that stuff out of the way. So now, I have everything neatly organized off on one of my internal drives, that’s not my main SSD drive. Hazel is my first pick and I’ll put a link to that in the show notes.

My second pick is the Mac Power Users Podcast, like I said before. It’s just a great show. It’s done by David Sparks and oh, I forget her name. The only reason I remember his name is because he goes by MacSparky online. But anyway, they go in-depth on all kinds of things related to running a Mac. And I just really, really like the show. They do such a great job in covering all of the different angles. They’re both attorneys and so, they’re both pretty thorough in their approach to the way they use their computers. And it’s a terrific show. It was picked up by FiveByFive.tv which is Dan Benjamin’s podcast network a while back. So anyway, he’s been helping move them along that way, can’t say enough good things.

So anyway, those are my two picks. And we’ll throw it over to Isaac, what are your picks?

ISAAC:  So, the first pick in that blog post about Giraffes and Ladders, my first pick is a book that has kind of changed my life quite a bit. It’s called ‘Nonviolent Communication’. And it’s one of these things that’s basically advanced common sense. And so, the hard part is realizing you don’t already know this stuff. But it’s powerful. Powerful stuff, you should get this book, read it several times, practice it. It will change how you work with people.

JAMISON:  Is that the source of the ‘All programming languages are great, all programs are wonderful’ comment?

ISAAC:  No, no, no. That was total violence. That was snarky backhanded stuff. I’m just kidding. Basically, I want to not cause more drama than I need to. But when somebody comes at you screaming in your face, it’s like how do you deal with that without just screaming back? And a lot of therapists use it. And I think that most of real world software development is effectively like dealing with crazy people including your self. So, it’s a really powerful tool to have in your toolbox. And that’s it for now. I mean, I pretty much talked about everything else I liked.

CHUCK:  It’s funny that you mentioned that book because I have had it come up in so many different conversations. I just haven’t gotten around to reading it yet. But when it comes across my radar that many times, I think I really need to pick it up and figure out what it’s about.

ISAAC:  Yeah. It’s very, very powerful. That’s all I can say. It will give you powers.

CHUCK:  Awesome.

ISAAC:  I mean, it’s one of these things where like somebody comes at you screaming about what an awful, terrible person you are. And you can either turn that into a fight or you can leave with them like thanking you.

CHUCK:  Well, that makes sense. That is a super power.

[Laughter]

MERRICK:  Does it teach you how to leave with their wallet?

ISAAC:  No. So, that’s another book which is also really powerful called ‘How to Win Friends and Influence People’.

[Laughter]

ISAAC:  Which is basically like, how to be an effective long term sociopath and not make enemies.

[Laughter]

JAMISON:  Probably the best description of that book I’ve ever heard.

ISAAC:  ‘Nonviolent Communication’ is extremely powerful.

CHUCK:  Alright. We’ll check them out. Well, thanks for coming on the show, Isaac. We really, really appreciate you being here.

ISAAC:  No problem. Thank you for having me.

CHUCK:  We’ll have to have you back sometime.

ISAAC:  Cool.

CHUCK:  Anyway, we’ll wrap up the show. Thanks for listening. We’ll catch you all in a week. And don’t forget to use that code to get into Fluent Conf.

0 comments

Previous post:

Next post: