A new build setup for Volto

What if I told you that a project-less build in Volto is possible? What if we unified the development experience while working on a project or an standalone add-on? What if the project artifact is no longer necessary, and the burden of maintaining the boilerplate, along with utilities configuration, etc is gone? How about if updating Volto have become changing the version number in one file? or that this build will continue working independently of future changes that may happen in how Volto is being built?

Show me, or it didn't happen

I've been toying around with the idea of a project-less setup since we moved Volto core to a monorepo. I'm still impressed how well pnpm workspaces works and fits into the JavaScript ecosystem.

The Volto Team have this dream since long time ago that at some point we would be able to be able to get rid of the "Volto project" abstraction, and use Volto directly to run any kind of project. We put lots of efforts during the last years to be able to move all the code and configuration from the project level into add-ons. The project became expendable by moving all the project setup and configuration out of it.

project setup === add-on setup

A project setup, as we know it, it consists in a boilerplate that uses Volto as a library, plus a (policy) add-on. An isolated add-on setup needs a project around to work. We came to the realisation that both scenarios are the same if we remove the project artifact.

So what if we come up with a setup that uses directly vanilla Volto in all the possible ways, instead of having a man in the middle (project)?

The main idea is being able to make everything work, dev server, production build, utilities, tests, etc just using plain vanilla Volto plus an add-on (or add-ons).

To accomplish this, we will make extensive use of pnpm workspaces, of course, like we do in Volto core monorepo. We also had to make small changes, fix some bugs, and add some features here and there.

mrs-developer

If we need vanilla Volto, the best way to check it out it is via mrs-developer. We could do it using a plain git command, but it's convenient to keep all our required checkouts in one place using mrs.developer.json. We do not need it for keeping any jsconfig.json or tsconfig.json anymore, because that's pnpm job now.

Originally mrs-developer output all the checkouts to the same folder. We extended it by adding a new feature that allows having specific optional outputs per repository. So we will configure it to checkout Volto like this:

{
  "core": {
    "output": "./",
    "package": "@plone/volto",
    "url": "git@github.com:plone/volto.git",
    "https": "https://github.com/plone/volto.git",
    "tag": "18.0.0-alpha.25"
  }
}

So when we run mrs-developer, we get Volto checked out in the core folder, and pin it to use a specific tag (or branch, if required).

pnpm workspaces

After this, we need to make pnpm know about the code in the setup. We use pnpm-workspace.yaml file with this configuration:

packages:
  - 'core/packages/*'
  - 'packages/*'

We declare all Volto core packages as workspaces. Then we declare another possible source of workspaces in the packages folder.

package.json

In order to round up the trick, we should make the top package.json a full blown pnpm monorepo. Let's take a look at it:

{
  "name": "@kitconcept/volto-light-theme-dev",
  "scripts": {
    "preinstall": "npx only-allow pnpm",
    "start": "pnpm build:deps && VOLTOCONFIG=$(pwd)/volto.config.js pnpm --filter @plone/volto start",
    "start:prod": "pnpm --filter @plone/volto start:prod",
    "build": "pnpm build:deps && VOLTOCONFIG=$(pwd)/volto.config.js pnpm --filter @plone/volto build",
    "build:deps": "pnpm --filter @plone/registry --filter @plone/components build",
    "i18n": "pnpm --filter @kitconcept/volto-light-theme i18n",
    "test": "RAZZLE_JEST_CONFIG=$(pwd)/jest-addon.config.js pnpm --filter @plone/volto test",
    "lint": "eslint --max-warnings=0 'packages/**/src/**/*.{js,jsx,ts,tsx}'",
    "lint:fix": "eslint --fix 'packages/**/src/**/*.{js,jsx,ts,tsx}'",
    "prettier": "prettier --check 'packages/**/src/**/*.{js,jsx,ts,tsx}'",
    "prettier:fix": "prettier --write 'packages/**/src/**/*.{js,jsx,ts,tsx}' ",
    "stylelint": "stylelint 'packages/**/src/**/*.{css,scss,less}' --allow-empty-input",
    "stylelint:fix": "stylelint 'packages/**/src/**/*.{css,scss,less}' --fix --allow-empty-input",
    "dry-release": "pnpm --filter @kitconcept/volto-light-theme dry-release",
    "release": "pnpm --filter @kitconcept/volto-light-theme release",
    "release-major-alpha": "pnpm --filter @kitconcept/volto-light-theme release-major-alpha",
    "release-alpha": "pnpm --filter @kitconcept/volto-light-theme release-alpha",
    "storybook": "pnpm build:deps && VOLTOCONFIG=$(pwd)/volto.config.js pnpm --filter @plone/volto storybook dev -p 6006 -c $(pwd)/.storybook",
    "build-storybook": "pnpm build:deps && VOLTOCONFIG=$(pwd)/volto.config.js pnpm --filter @plone/volto build-storybook -c $(pwd)/.storybook"
  },
  "dependencies": {
    "@plone/volto": "workspace:*",
    "@plone/registry": "workspace:*",
    "@kitconcept/volto-light-theme": "workspace:*"
  },
  "devDependencies": {
    "mrs-developer": "^2.2.0"
  },
  "packageManager": "pnpm@8.15.4"
}

The key are the dependencies: @plone/volto and @plone/registry, then our add-on (either if it's an standalone add-on or the policy add-on of our project) or add-ons for our setup. All of them, using their versions inside the pnpm monorepo ("workspace: *")

Then a lonely devDependency: mrs-developer >= 2.2.0

The last thing are the scripts. Please notice that they are pointing to Volto, so Volto itself is the one running these processes, under Volto core environment itself. This is done thanks to the pnpm --filter feature.

How about tooling?

Dealing with tooling is one of the hardest things due to the different nature of how they behave, and the assumptions that every tool does. So we have to make happy every single one of them. In some cases it is easy, in others are a bit more difficult. Also, tools behave different if you run them through the command line or used via IDEs extensions.

Remember that we have the requirement that this setup should use the dependencies and configuration from Volto itself to reduce maintenance burden.

Tools mainly make the assumption that they are available in the root repository (as they expect it to be hoisted in a flat node_modules environment). Using pnpm workspaces this is not always true, since it uses a symlinked node_modules structure, every dependency is installed by the package that it's requiring it.

Fortunately, pnpm has yet another feature: public-hoist-pattern[] This allows you to "lift up" dependencies from the different workspaces to the main root one. This is key for tooling and specially IDEs which expect them to be in the root repository. So we will hoist all the tooling dependencies up to the top level. This is done by specifying them in .npmrc file.

public-hoist-pattern[]=*eslint*
public-hoist-pattern[]=*prettier*
public-hoist-pattern[]=*stylelint*
public-hoist-pattern[]=*cypress*
public-hoist-pattern[]=*process*
public-hoist-pattern[]=*parcel*

By doing this, IDEs have access to the tooling packages and the linters IDE extensions will be happy. Command line tools will also find the right dependencies in the root.

volto.config.js

There's one missing link to cover, which is how we tell Volto which add-ons should it load?

Luckily, we have in Volto a way to specify the add-ons and the theme programmatically, using volto.config.js file:

const addons = [
  '@kitconcept/volto-light-theme',
];
const theme = '@kitconcept/volto-light-theme';

module.exports = {
  addons,
  theme,
};

And thanks to a recent addition to @plone/registry we can provide this list via and environment variable:

$ VOLTOCONFIG=$(pwd)/volto.config.js pnpm --filter @plone/volto start

Makefile

We worked in some convenience Makefile commands to make life easier to developers.

help                             Show this help
install                          Installs the add-on in a development environment
start                            Starts Volto, allowing reloading of the add-on during development
build                            Build a production bundle for distribution of the project with the add-on
i18n                             Sync i18n
ci-i18n                          Check if i18n is not synced
format                           Format codebase
lint                             Lint, or catch and remove problems, in code base
release                          Release the add-on on npmjs.org
release-dry-run                  Dry-run the release of the add-on on npmjs.org
test                             Run unit tests
ci-test                          Run unit tests in CI
backend-docker-start             Starts a Docker-based backend for development
storybook-start                  Start Storybook server on port 6006
storybook-build                  Build Storybook
acceptance-frontend-dev-start    Start acceptance frontend in development mode
acceptance-frontend-prod-start   Start acceptance frontend in production mode
acceptance-backend-start         Start backend acceptance server
ci-acceptance-backend-start      Start backend acceptance server in headless mode for CI
acceptance-test                  Start Cypress in interactive mode
ci-acceptance-test               Run cypress tests in headless mode for CI
build-image                      Build Docker Image

The most interesting one is make install which will bootstrap the environment the first time. This is required because there's a chicken-egg problem that we should solve when we still haven't installed the environment and mrs-developer is not around yet.

make install will bootstrap the environment, being the first command that should be run. Every time you change mrs.developer.json you must also run it as well.

Conclusion

After this, we have closed the circle, and we have a full blown setup that covers all the bases. Dev server, production build, linters, i18n, unit testing, Cypress, Storybook, releases.

We have the same layout and developer experience for developing a standalone add-on or a full blown project setup.

In addition, we can also use mrs-developer to add more external add-ons to the build, or even improve Volto at the same time that we develop our project (a long time missed feature).

We have reduced the boilerplate fingerprint to a buch of files, the vast majority being placeholders that would not require to be updated over time, because we are using the Volto configuration directly. If something changes in the future, it will happen in Volto, not in our setup.

A single place to specify the Volto version that we want to use. No more duplicated dependencies or devDependencies.

There's already a generator that you can try if you want to test drive it in your upcoming add-on or project. Feedback is welcome! The plans is that it will become the default if the results are as expected. You can try it by running:

$ pipx run cookiecutter gh:plone/cookiecutter-volto

You can take a look at some of the first add-ons that are using it already:

volto-light-theme

volto-button-block

volto-social-blocks