The parking problem

If you are a web developer also involved in managing the servers hosting your apps, then the idea of “serverless” app hosting needs little introduction.

Using AWS Lambda in combination with AWS API Gateway is catching as a combination for deploying simple web apps. Besides the lack of a server to maintain, scaling is handled automatically, and you only pay for the requests actually made to the app.

My task was to review options suitable for serving a simple mix of static and dynamic content using Node.js. We’re used to developing with the Express web server, so the new solution needed to be easy to transition to, since we would be maintaining this new service side-by-side with other Express apps.

For an excellent introduction to running an Express-style app on AWS Lambda and API Gateway, I recommend the post Server-free Express.js.

Legedemain and dpl

The tool I tried first was Legedemain. You can read an introduction to Legedmain by the author.

On the face, it sounds like a great fit: Legedemain helps you run Express apps on AWS Lambda. I soon found it wasn’t a complete solution. It doesn’t handle deployment to AWS Lambda and AWS Gateway currently.

Not to be immediately deterred, I tried pairing it with aws-lambda-deploy. But this only updates Lambda and not API Gateway.

Trying this approach made me realize I wanted an integrated approach that could manage API Gateway and Lambda together.

serverless

After some more looking I found serverless, which is a very popular framework for managing both AWS Lambda and API Gateway together. Although there a lot of docs, I ran into problems.

As a first impression, I crashed the project setup tool simply by pressing the Enter key.

Then I found the CLI tool can’t even print out basic help information with compiling whole project.

serverless is not meant to run Express apps. Instead, you are encouraged to write more “native” logic. In theory, I like this idea. It’s simpler with less abstraction and data translation. serverless has it’s own Plugin and Hook system which can work like Express Middleware. API API Gateway and AWS Lambda already handle the translation of HTTP request to and from JavaScript objects, so they replace the role of Express. I was willing to adapt to this.

Serverless advocates very small Lambda functions that handle maybe just one route endpoint, or maybe a small collection handle the CRUD actions for a single type of entity. There are some good reasons for this. Small Lambda functions would start up faster. Since they spend less time executing, they cost less. Also, you can upgrade some functions without touching the others, further increasing uptime. serverless also let’s you manage several Lambda functions a single project so you don’t have to be concerned with how many Lambdas are stored and running on AWS.

Serverless has a name for an Express-style app with lots of routes served by a single server. They call that a “Monolithic app” and don’t have many docs for monolithic apps. I opened the linked issue to suggest they improved such docs and my issue got tagged with “priority: low”.

While Serverless has a lot of positives, the fact that the model is very different from Express-style apps was a downside for my team. Another downside is that much of the configuration involves editting somewhat complex JSON files. Here their otherwise verbose docs fail with some handwaving that you should be very familiar with the docs for API Gateway and Lambda configuration. So, rather than abstracting away API Gateway and Lambda behind a single interface and set of documentation, now you have three sites of documentation to reference. For something we may touch infrequently, there’s too much required context to make changes that would be simple with Express.

The JSON structure includes such keys as name and customName. Since name is already a variable that be set to custom volumes, would you care to guess what the customName is? Better names would have been serverlessName and lambdaName which highlight the difference better.

While Serverless is popular and appears it will have a bright future, I found it would not be a good fit for us. For developers who work with Express 95% of the time, serverless has too many differences from traditional Express development to make it an easy addition to a workflow.

There seems to be a lot of momentum behind the serverless ecosystem and I expect to checking on it again in the future.

aglex

aglex is an acronym for Api Gateway Lambda EXpress.

While I’m afraid I may be the #2 user of this tool in the world, I can heartily suggest that perhaps you should be the #3.

By this point I had a clear sense of what I was looking for and aglex has it. Using this simple CLI tool, I was able to quickly deploy a (fairly) normal Express app. aglex than handled configuring and deploying both API Gateway and Lambda to bring the app online.

The documentation is a simple one-page README and it was beautifully sufficient.

After I set up project and it will be straightforward and familiar for co-workers to add a few more Express routes to the app when we need to and deploy the related changes to API Gateway and Lambda. No deep dive into complex JSON structures or the diving into the depths of the API Gateway and Lambda docs is required.

That said, as newer, less popular project aglex has it’s own rough edges which I hope get smoothed out soon:

I also had difficulty serving a static JSON file, but was eventually able to do so. It appears primarily designed to return JSON. The configuration file explicitly mentions serving static files, so it appears this is a supported use-case that I expect to improve in the future.

Girls with dolls on a cargo bike

So you made a “WIP” git commit that you didn’t intend to push and then you made another commit after it.

Now you’d like to undo the WIP commit but it’s no longer the most recent commit.

Here’s how you can re-order the last two git commits to make the WIP commit the most recent again.

(This is only safe to do locally, before you push).

# Interactively rebase the last two commits.
git rebase -i HEAD~2

An editor will pop-up with two “pick” statements. Swap the order of them. Save.

That’s it.

One of the follow-ups to the Great NPM Unpublishing was to blame small modules.

I’m with Sindre Sorhus that small modules are easy to reason about, but there are better ways to avoid the cascading effect that unpublishing can have besides increasing the amount of copy/paste coding in world.

For corporate projects, third-party modules stored on third-party module repos should not be a dependency for building and deploying. Storing third-party dependencies locally in some form solves the problem of third-party modules on npm which disappear. Check the third-party code into source control or use a private NPM repo. Using npm shrinkwrap to require precise versions is a good idea.

For open source modules, I would like to see distribution options that include downloading the entire dependency tree in a single tarball. The tarballs would contain a stack that has been tested and approved by the module maintainer, so you don’t end up downloading a combination of dependencies that no one ever tested or intended you to use. A module being unpublished would not affect this case, as an approved version would continue to remain in the tarball.

Open source projects don’t need to wait for module repos to offer this feature, they can upload and link to their own copies.

The project tarball could itself contain vetted tarballs of dependencies inside, ready for npm install. And yes, signed packages would help make sure that you all packages you are getting are from the authors you expect.

Npm, Inc has responded with their solution. They take some responsiblity for the difficulties encountered, but they also put themselves at the center of solving the problem. Their solutions alone would only make the main NPM repo a larger single dependency and single-point-of-failure for the JavaScript community.

We’ll be a more resilient community if we all take care to make sure that key dependencies for our projects are locally available so that further disruptions at npmjs.org are not felt so widely.

The Perfect Day

What can you do to make today perfect? Of the parts that you can control, what will you do to make the day the best day possible?

On the day I took this photo, I was playing the Perfect Day game. I work up early, forced myself out of bed and squeezed in a 20 mile bike ride on a old favorite route when the air was still cool on what would be a hot summer day.

Frankly, I don’t remember much else from that day, but I remember this part. I remember this S curve between corn field and horse pasture, just before the road makes a sharp right turn into the woods and down Abington hill. I remember my eyes tearing up as the speedometer flirted with 50 and the forest blurred into green. Then just as fast, the hill flattening out again and safety returning, like a calculated roller coaster.

I’ve forgotten that Friday, but I remember this small perfection.

Bicycling saves lives

Perl’s prove is a great tool for running TAP-based test suites. I use it for a large project with about 20,000 tests. Our test suite runs multiple times per day. At that scale and frequency, the run time is a concern.

To address that, prove has -j option, allowing you to run tests in parallel. Unfortunately, in our large test suite not all tests are “parallel ready”. I needed a way to run most tests in parallel, but mark a few to be run in sequence. Ideally, I’d like these tests to be part of the same run as the parallel tests.

After much digging around, I found that prove has had support for adding exceptions to parallel tests runs since in 2008. The feature has just never been documented in prove, making it difficult to discover. Here’s an example which works with the currently released version of prove, 3.25:

# All tests are allowed to run in parallel, except those starting with "p"
--rules='seq=t/p*.t' --rules='par=**'

For more details and documentation, you can see my related pull request. You can leave a comment there to advocate for or against merging the the patch, or you can download the raw patch directly to apply to your own copy of the Test-Harness-3.25 distribution.

**UPDATE 9/14/2012: ** It turns out it doesn’t work quite as I thought. I tried tweaking some of the internals of prove further, but hit a bug. See the details in my follow-up to the Perl-QA list. I changed approached and started working on making my exceptional tests parallel-safe. Perhaps I can cover some of the techniques I used for that in a future post.