What is the difference between packages.dhall and spago.dhall files?

Question

spago docs state:

packages.dhall: this file is meant to contain the totality of the packages available to your project (that is, any package you might want to import).

In practice it pulls in the official package-set as a base, and you are then able to add any package that might not be in the package set, or override existing ones.

spago.dhall: this is your project configuration. It includes the above package set, the list of your dependencies, the source paths that will be used to build, and any other project-wide setting that spago will use. (my emphasis)

Why do both files have the notion/concept of dependencies? Example: packages.dhall and spago.dhall from the ebook.

spago.dhall dependencies can be found in the project .spago folder. But I cannot locate the ones from packages.dhall. Others are common like aff. A different perspective:

[...] what you choose is a "snapshot", which is a collection of certain versions of all available packages that are guaranteed to compile and work together.

The snapshot is defined in your packages.dhall file, and then you specify the specific packages that you want to use in spago.dhall. The version for each package comes from the snapshot.

That sounds, like spago.dhall is an excerpt of packages from packages.dhall.The note about versions is a bit confusing, as there aren't version specifiers in both files.

So, why two files? What is the mental model for someone coming from npm ecosystem with package.json (which might be present as well)?

Answer 1

The mental model is that of a Haskell developer, which is what most PureScript developers used to be, and many still are. :-)

But more seriously, the mental model is having multiple "projects" in a "solution", which is the model of Haskell's de-facto standard package manager, Stack. In Haskell this situation is very common, in PureScript - much less so, but still not unheard of.

In a situation like this it's usually beneficial to have all the "projects" to share a common set of packages, which are all guaranteed to be "compatible" with each other, which simply means that they all compile together and their tests pass. In Haskell Stack this common set of packages is defined in stack.yaml. In Spago - it's packages.dhall.

Once you have this common base set of packages established, each individual project may pick and choose the particular packages that it uses. In Haskell Stack this is specified either in package.yaml or in <project-name>.cabal (the latter being phased out). In Spago - it's spago.dhall.

But of course, when you have just the one project, having both packages.dhall to establish the "base set" of packages and then, separately, spago.dhall to pick some particular packages from that set - may seem a bit redundant. And indeed, it's possible to do without the packages.dhall file completely: just specify the URL of the package set directly in spago.dhall, as the value of the packages property:

{ name = "my-project"
, dependencies = [ ... ]
, license = "..."
, packages = https://github.com/purescript/package-sets/releases/download/psc-0.13.8-20201223/packages.dhall
, repository = "..."
, sources = [ "src/**/*.purs" ]
}

This will work, but there is one important caveat: hashing. When the URL of the package set is specified in packages.dhall, running spago install will compute a hash of that package set and put it inside packages.dhall, right next to the URL. Here's what mine looks like:

let upstream =
      https://github.com/purescript/package-sets/releases/download/psc-0.13.8-20201222/packages.dhall sha256:620d0e4090cf1216b3bcbe7dd070b981a9f5578c38e810bbd71ece1794bfe13b

Then, if maintainers of the package set become evil and change the contents of that file, Spago will be able to notice that, recompute the hash, and reinstall the packages.

If you put the URL directly in spago.dhall, this doesn't happen, and you're left with the slight possibility of your dependencies getting out of sync.

---

Now to address this point separately:

Why do both files have the notion/concept of dependencies? Example: packages.dhall and spago.dhall from the ebook.

If you look closer at the examples you linked, you'll see that these are not the same dependencies. The ones in spago.dhall are dependencies of your package - the one where spago.dhall lives.

But dependencies in packages.dhall are dependencies of the test-unit package, which is being added to the package set as an override, presumably because we want to use the special version stackless-default, which isn't present in the official package set. When you override a package like this, you can override any fields specified in that package's own spago.dhall, and in this case we're overriding dependencies, repo, and version.