Npm on blog.iankulin.com

npm ERR! Exit handler never called!

Mon, 21 Oct 2024 00:00:00 +0000

I quite like GitHub scanning all my code and sending me security advisories. Here’s today’s:

With these, and my dependabot alerts, fixing them is usually just a matter of pulling down the project, running an npm update, building any artifacts, then pushing it back up. But today, not so:

package-lock.json

It’s probably worth revisiting what the package-lock.json does. It contains all the versions of any packages you’ve imported, and their dependencies. The idea is that this will make the build reproducible. We don’t commit the node_modules folder (that actually contains all that package code), but npm can reproduce it exactly by using the version information in the package-lock.json file. Here’s a snippet where you can see all those versions:

 "node_modules/body-parser": {
 "version": "1.20.2",
 "resolved": "https://registry.npmjs.org/body-parser/-/body-parser-1.20.2.tgz",
 "integrity": "sha512-ml9pReCu3M61kGlqoTm2umSXTlRTuGTx0bfYj+uIUKKYycG5NtSbeetV3faSU6R7ajOPw0g/J1PvK4qNy7s5bA==",
 "dependencies": {
 "bytes": "3.1.2",
 "content-type": "~1.0.5",
 "debug": "2.6.9",
 "depd": "2.0.0",
 "destroy": "1.2.0",
 "http-errors": "2.0.0",
 "iconv-lite": "0.4.24",
 "on-finished": "2.4.1",
 "qs": "6.11.0",
 "raw-body": "2.5.2",
 "type-is": "~1.6.18",
 "unpipe": "1.0.0"
 },
 "engines": {
 "node": ">= 0.8",
 "npm": "1.2.8000 || >= 1.4.16"
 }
 },

For me, I don’t really care that I’m using “iconv-lite” version 0.4.24, but if I’m working on a project with someone else, it might be important that we’re using the same version so we’re not chasing our tails trying to sort out a bug.

npm update

There are some rules about how the versions of packages are entered in package.json; when we run npm update, it uses those rules to look in the npm registry to find the most recent version of all the packages it’s allowed. Then it updates them in package-lock.json, and downloads the code into the node_modules directory.

This is potentially a substantial change to your app, so you’d definitely want to be running your testing process again afterwards.

The Error

npm ERR! Exit handler never called!

npm ERR! This is an error with npm itself. Please report this error at:
npm ERR! <https://github.com/npm/cli/issues>

This sounds quite serious, but before you head off to report it, try this:

npm install --no-package-lock

This just runs the update ignoring the package-lock.json file - as if you’d just deleted it. If that works, it was a problem with the package-lock.json file, which in this context of just wanting all the latest versions we don’t care about. We do want to rebuild the package-lock.json file though, so go ahead and delete it and run npm install to create a nice new one.

Now your project will have a couple of version changes in those package files. You’ll need to redo all your testing and rebuild any Docker images etc, and then you’re all up to date and secure again!

Code reuse by publishing to NPM

Mon, 14 Oct 2024 00:00:00 +0000

If you find yourself copying over a source file from one Node project to another because it’s a handy utility you wrote and are used to using, you’re only doing it half right. A better way to do this is to publish your utility to the Node Package Manager (NPM). That way you can just import your utility where ever you need it, it will live in the node_modules of any project that uses it, and most importantly, updates are sorted out automatically - because that’s what package managers are good at.

By the time you are even thinking about this, you’ve already gotten used starting your Node projects with npm init and installing packages with npm install express when you have your own packages on npm they are handled exactly like that.

So, how do we get our code up to npm?

NPM Account

You need an account on npm. It doesn’t cost anything to host your packages there, though they will be public on the free plan. If you don’t have an account, create one. It’s 2024 so use 2FA.

Create your project

Make a directory with the same name as your package. All lower case, no spaces, hyphens are allowed. While we’re at the command line, let’s sign into npm

mkdir is-even
cd is-even
npm login

Almost every straightforward name you can think of for your utility code will have been used already on npm. I’m not trying to be twitter famous or to get 96,000 downloads of my package per week - I just want to reuse my code conveniently, so I’ll scope it to my user name. So my package won’t be called is-even on npm (famously, that’s already taken), it will be called @iankulin/is-even

This is called scoping it to our username. When we do that, the project is initialised like this:

npm init --scope=iankulin

You’ll get asked the questions in a similar way as init-ing a regular node project. index.js is fine for your file name. You’ll end up with something that looks like this in your package.json. Note that I’ve added "type": "module", since I’m all about the ESM this week.

Now we’d better write some code. Here’s my index.js

function isEven(num) {
 if (typeof num === 'number' && Number.isInteger(num)) {
 return num % 2 === 0;
 }
 // we're counting all non-integers and non-numbers as odd
 return false;
}

export { isEven };

Publishing

After extensive testing and refinement, you can push it up to npm:

npm publish --access public

And the exciting moment - we can see our package on npm, just like a real coder.

Use your package

Using the package once it’s published is exactly the same as using any npm package: Start a new project, and install the package.

mkdir test-is-even
cd test-is-even
npm init
npm install @iankulin/is-even

Again, because I’m using ESM, I’ve added "type": "module", to my package.json. And some test code in my index.js

Sorting out Node package dependencies when cloning old repos

Wed, 06 Sep 2023 00:00:00 +0000

If you clone an old node project and npm install it, you’ll most likely get a bunch of errors and warning messages. If you just decide to yolo it and run the project, you’ll get a bunch more.

I’ve been doing this exact thing. I want to add some auth to my app, and I’ve been following WebDevSimplified’s video about using passport. I was building into my app without really understanding what I was doing, ran into problems and decided just to clone his repo and integrate the code into my app. The repo is four years old.

The reason this is a problem is that npm uses package.json and package-lock.json to specify the versions of different packages. This is great, since it means you can clone a repo and know you are using the exact same versions of each package that everyone else using the repo is using. However, given enough time it also becomes a problem - packages are updated to address security vulnerabilities in their own code, or in their dependencies all the time.

To untangle this mess, it’s worth understanding what’s going on with these two files.

package.json

The package.json file doesn’t just store package versions, it has a heap of other project configuration stuff - like the starts script, project name and other meta data that we’re not really interested in here. What we’re interested in is the dependancies, so let’s have a look at a sample.

"dependencies": {
 "lodash": "^4.17.21",
 "express": "~4.17.1",
 "axios": "2.6.0"
}

Unsurprisingly, we’re looking at some JSON. In this case, key value pairs consisting of the package name, and then a version, but the version sometimes has some punctuation in front of it. The actual number is the version that was pulled down when we said something like npm install lodash. In the case above, that was version 4.17.21

The caret ^ in front of it, means this version, or any future version up to but not including the next major version. So npm install is free to grab whatever the current version is - maybe 4.18.34 or 4.99.99 - but not 5.0.0 or anything after that.

This is a sensible restriction. In most projects a major version denotes a breaking (not backward compatible) change, so it makes sense to allow any future improvements and bug fixes, but to not allow breaking changes. For this reason, this is the default, so if you don’t manually edit your dependencies, this is what they will be set to.

If you want to be slightly stricter, you use the tilde ~ in front of the version number as shown above for the express package. In this case, you’re specifying the minimum version of the package, but allowing only patches, and not minor version changes. So in the case of ~4.17.1 it would be fine to install 4.17.2 or 4.17.9 but not 4.18.0

The last case is no punctuation in front of the version number, in which case we are locked into that version. This is what’s happening with the axios package above. npm install will only fetch 2.6.0, even if there’s a bug fix 2.6.1 available.

For a long time, package.json was all that was available, and beautiful thing that it is, there was still an issue.

package-lock.json

Even though, in the example of "axios": "2.6.0" we’ve firmly locked axios to version 2.6.0 by putting it in the package.json file with no prefix on the version number, some changes are still possible - how so?

Most non-trivial packages you use will themselves depend on other packages. These are called transitional dependancies. In the case of axios (which is a http client to pull web pages into node) it depends on seven other packages that do more specialised things such as handling streams, understanding mime types and so on.

If you want code bases to be completely reproducible, then we also need to lock all the versions of the transitive dependencies. To do this, package-lock.json was introduced in Node v5.0 in 2017. Here’s a snippet out of the file for an app using axios.

 "node_modules/mime-types": {
 "version": "2.1.35",
 "resolved": "https://registry.npmjs.org/mime-types/-/mime-types-2.1.35.tgz",
 "integrity": "sha512-ZDY+bPm5zTTF+YpCrAU9nK0UgICYPT0QtT1NZWFv4s++TNkcgVaT0g6+4R2uI4MjQjzysHB1zxuWL50hzaeXiw==",
 "dependencies": {
 "mime-db": "1.52.0"
 },
 "engines": {
 "node": ">= 0.6"
 }
 },

The Mess

Running npm install causes npm to look at both of those files to work out what packages to download. It creates the node_modules folder and puts all those packages in there so we can require them. When I tried that with this four year old project, this is the first of three pages of error messages I got.

Most of the rest were errors from bcrypt - and you don’t really want to run old cryptology code.

So, we’re in a bit of a bind here. The package version specified by the package developers doesn’t work any more. We can (and will shortly) ignore those, but of course then we’re risking that some breaking change in one of the packages will break the app code in some other way. Nevertheless, that’s what we need to do, but we’ll do it starting from the least risky to the most risky.

Delete package-lock.json - if you trust the developers of the packages being used (and you shouldn’t be running them if you do not) then letting them decide the relative risks with updating the transitive dependencies is probably a reasonable bet. We can achieve that by deleting or renaming package-lock.json and re-running npm install.
Edit package.json to allow minor version changes - maybe all of the package versions in here have the caret ^ in front of their version number to allow them to be updated to the latest minor version and patch without changing the major version. If they do not, then try adding the caret to each one.
Allow the latest version - an option we didn’t talk about when adding carets or tildes is that we can actually tell npm to just download the latest version. You don’t often see this, but if you put in a wildcard it will just grab the most recent. This is a bit more of a nuclear option, so it’s probably worth having a look at the 2000 lines of errors I had, and seeing if you can make an intelligent guess about which package is troublesome, and starting from there, doing them one at a time.

"dependencies": {
 "axios": "*"
}

In my case there was lots of bcrypt sounding errors, so that was my first try - I set that to the wildcard version, and the number of lines of warning/error output dropped from 2249 to 14. Also the process actually completed this time - I had a node_modules folder and a new package-lock.json. Included in the 14 lines of output was advice that there was a number of security vulnerabilities that could be fixed by running npm audit fix --force

npm audit will let you know of any known security vulnerabilities in your installed packages. If you add the fix --force option it will update them to the minimum version to address those vulnerabilities. This is basically just doing what we did in the previous step, but a bit smarter. In general, it’s going to be safer to have to fix some code than to ship code with known vulnerabilities, so if you are offered this choice, go ahead and run that.

Test everything. You’ve just pulled a heap of new code in this project. It needs tested.

How to deploy a Node.js app

Wed, 05 Jul 2023 00:00:00 +0000

This is one of those things that is simple once you know it. I had my tiny Node service working on my MacBook, but how do I run it on the server?

Native or Container

Obviously I need Node.js installed on the server, should I have it in a Docker container, or native on the machine. There’s no clear answer here - in a container set up with Docker Compose might be more in line with my ideology of treating machines as disposable, but a native install is simpler, and I probably want to make life simpler at this stage when I’m learning everything.

Installing Node

This took me down a bigger rabbit hole than I was expecting. My VPS is Unbuntu LTS 22.04.2, so I spun one of those up in a VM on the homelab to try things out.

A quick google search suggested the NodeSource binary distributions. That involves curling a big script (when I pasted the script into ChatGPT it said it wasn’t malicious). I could chose the Node version, so I grabbed 20.x That was as painless as you’d expect.

Then I started wondering why I couldn’t just apt install it. If I could do that, it would reduce the chance of a supply chain attack since I’d have the power of Canonical on my side. So I rolled the previous install back (thank you Proxmox backups of VMs), and tried:

apt install nodejs

That worked fine - Node is in the Ubuntu packages, but the version is quite old - v12.22.9. This is on the current Ubuntu LTS 22.04.2. I don’t think it will matter for my purposes, but it explains why you’d do something other than just this.

I’m also going to need npm, so lets get that with:

apt install npm

That seemed to download a heap more stuff that the node install.

Deploying your project

Again, the first search result was more complicated than I needed. The advice was to clone my repository onto the server where I wanted to deploy. This is such a minor project, I hadn’t pushed it up to GitHub. So that seemed excessive. You know, not everything has to be DevOps CI/CD! I mean, we ain’t talking about a very complicated project here:

I’ve got this tiny source file, and the text file I want to serve. All the dependencies (just Express) are in the package.json, so presumably that’s all I need on the server to get going.

I scp’d those from my laptop to a directory on the folder:

Once they are there, I need to install the packages from the package.json, so we do that with:

npm install

That installed 59 packages (presumably Express plus 58 of it’s dependencies). Then I started the app with:

node .

and it worked!

Insomnia

I should probably explain what you’re looking at above. I could have tested this little node server by going to the api address in a browser and checked that I got back the text file I was expecting. And in Chrome (and I assume Firefox) there are developer tools that would show the return code etc. However, most of the REST API videos I’ve watched use a better tool - mostly Postman. These sort of tools give you a heap of other capabilities, none of which I really need for this simple project, but will be very handy for more complex APIs where there is a body to the request.

The only reason I’m using Insomnia instead of Postman is that when I tried Postman, it straightaway wanted some of my data to make it work. Insomnia hasn’t forced me to do that yet.

Expired Packages Part II

Tue, 31 Jan 2023 00:00:00 +0000

Following on from the previous post…

I went the nuclear route - deleted the node_modules folder, package-lock.json and installed the packages from packages.json. I still had some errors, but the react app at least ran correctly. Also, the messages are a bit more intelligible, and all of them cascade from this one.

# npm audit report

nth-check <2.0.1
Severity: high
Inefficient Regular Expression Complexity in nth-check - https://github.com/advisories/GHSA-rp65-9cf3-cjxr
fix available via `npm audit fix --force`
Will install react-scripts@2.1.3, which is a breaking change
node_modules/svgo/node_modules/nth-check

From my, admittedly ignorant, viewpoint, there’s a couple of weird things going on here.

The first is how the hell is installing react-scripts@2.1.3 a good idea, when the current version is 5.0.1. That does not seem like a good solution.

The second is that is that the currently installed version of nth-check seems like it is 2.1.1 which is the current version, and certainly >2.0.1 which is the complaint. My basis for this claim is this encouraging part of package-lock.json:

But if I check the installed version using npm list nth-check, I get this bad news:

So one version of css-select is using an old version of nth-check, likely this is the source of my troubles.

As far as I can make out, the package-lock.json file’s purpose is to lock in particular versions of packages. If it’s committed with the rest of your code, it guarantees that when you rebuild the app, it will be the same as the one committed, without the need commit all the node modules you are depending on. Generally I don’t think you are meant to directly edit it, but it suddenly seems like a good idea.

There’s about five dependencies on this package in package-lock.json, and I notice all of them except this one, start with the ^caret.

"nth-check": {
 "version": "1.0.2",
 "resolved": "https://registry.npmjs.org/nth-check/-/nth-check-1.0.2.tgz",
 "integrity": "sha512-WeBOdju8SnzPN5vTUJYxYUxLeXpCaVP5i5e0LF8fg7WORF2Wd7wFX/pk0tYZk7s8T+J7VLy0Da6J1+wCT0AtHg==",
 "requires": {
 "boolbase": "~1.0.0"
 }
}

In general, I think I should be doing this in packages.json, but the nth-check is not in there.

After fruitlessly googling around and asking in a couple of discords, I check with ChatGPT to see what she thought.

Well, I’d done all that, and a couple of humans had already told me not to jigger with package-lock.json, so that was my next stop - I edited it to add the caret and tried npm install - it just changes it back, which make sense.

There’s another install script npm ci which is supposed to use the package-lock.json rather than doing it’s own check, that didn’t help either.

I went back to googling, focusing on the react-scripts (which is basically responsible for building the template React app) and found this issue on github.

Basically, it’s claimed to be a problem in how npm audit works, and can’t be a real vulnerability since it’s just a tool being used during dev. That still doesn’t answer why they don’t just update their code to use the newer version on nth-check, but in any case, it can be safely ignored.

For people using CI tools that depend on an error free build, the react-scripts can be moved to a different section of the packages.json file and an argument passed to npm audit to ignore those dependencies.

So, what have I learned from this? I should have done more looking for answers for the exact error, instead of logically coming up with solutions and then searching for and pursuing them. But also, I have a clear idea of what packages.json and package-lock.json do now!