Mono & Msa : Front 2. Monorepo


Posted on Wed, Nov 9, 2022 architecture turbo devops node yarn

Monorepo Architecture

A monorepository is an architectural concept or code management practice whereby distinct codes are united together to form one repository. In a nutshell, this is just like combining monolithic and microservice architecture. Sounds cool, right 😍?

This architecture approach can greatly help your project's organization many levels! In monorepos, scaling needs specialized tooling. This avoids unnecessary complexity encountered with vanilla git. Unlike poly repos, where scaling needs specialized coordination for code changes, packaging, testing, and release.

We’ll talk more about this architecture model Including its benefits, drawbacks, and most importantly how to implement it.

Combining everything to form this architecture may look impossible. But with tooling, we can get a proper setup. For instance, Workspaces are the starting points of monorepo as they encapsulate all applications you might have in a project. We’ll be using yarn workspaces, but npm and pnpm have decent workspace technologies as well.

Workspaces connect to your architectures and enable your project to run scripts globally with ease, orchestrate packages and cache executed scripts locally/remotely.

That being said, you may be wondering if Monorepos are actually true monoliths, the truth is they aren't. Code in a monorepo is only brought together to form one single repository rather than multiple repositories (polyrepo). However, with git submodules, you can achieve full seperation between repositories while still be inside of a monorepo context.

There are many benefits included in monorepo technologies.

So, Why Use Monorepo?

The need for a monorepo setup may not seem necessary at first and it may not even be important if you are writing a small program. However, if you plan to scale your application, you'll start considering it as an alternative. Let's say you start without a monorepo and build your codebase to a point where your IDE can't stand the ram needed to index your application? What do you do now?

If you have a lot of couplage in your code, your team will take longer to separate the logic in your codebase. Since microservices are the goal to deal with the separation of a massive codebase.

It will be easier for your team to refactor your code if you have already implemented a monorepo simply because its nature is to decouple your application if you follow good practices.

In this case, I recommend following good refactoring principles, this link is a perfect resource! 😇

Refactoring.Guru

When you look at code from this angle, organization becomes much more of a priority! So the big question is this : “Should you start with building with monorepo mindset right away?” To me, that's a yes if you plan on building a big application. F.A.N.G Companies are already using Monorepos because of their large codebases.

Without wasting much time on theory and to keep a neutral viewpoint on monorepos, let's look at some advantages and disadvantages of Monorepos.

Let's start with the advantages :

  1. Concurrency and partial releases 💢

    Monorepos promotes concurrency, while micro frontends enable teams to work on a project with different frameworks and technologies. This will increase development, speed and of course improve the overall application's performance.

  2. Cost effective repo, CI/CD and deployment setup 💰

    Setting up and deploying an application with a monorepo and micro frontend setup is cheaper than most of the architecture options out there. This is simply because you can easily add new nodes to your application with this type of setup.

    Remember the differences between scaling horizontally vs scaling vertically? With monorepos, you can still have the choice to scale vertically or horizontally which can definitely affect the cost of your application when it's hosted on a server.

    • Vertically : by adding new nodes to your server and including git submodules at the root level of your monorepo and splitting your services.

    • Horizontally : by scrapping git submodules and sticking to having the root of your application as the endpoint of your application

  3. Improved Collaboration 👪

    Monorepos removes some of the barriers you can find between teams. By enabling them to design and build microservices that work together.

  4. Easier to Refactor🥳

    Moving source code between various folders and subfolders is much easier in the monorepo ecosystem. This is simply because you have direct access to all the services in your codebase.

  5. Single Source of Truth 💢

    A monorepo has automatic build, run and deployment statistics that can be shown to your team to monitor every dependency.

  6. Code Sharing 🔗

    Duplicating code for various microservices in your development workspace causes additional engineering overhead.

    If common models, shared libraries, and helper code are all stored in a monorepo via a registry, teams can easily access them among the different packages and services inside the monorepo.

    You can either access them directly via the code source and gain from hot reload technologies or include them in your package.json dependencies to stick to a decoupling codebase. It all depends on how you manage your imports 🤩

The Monorepos Drawbacks ⚠️

Although monorepos have many advantages over polyrepo. However, that doesn't mean it comes without drawbacks. Below are some disadvantages of using a monorepo.

  1. Build Pipelines 🧫

    Making a change in the monorepo codebases is very slow. You've to spend a whole lot of time waiting for the build up system to finish appending changes. However, this problem is now subtle as many monorepo tools are now putting that into consideration. You can run isolated pipelines with the use of gitsubmodules.

  2. Limitations Around Access Control 🎛️

    Development teams having access to the entire codebase can be an advantage. But it becomes a disadvantage when the company decides to protect some part of the codebase.

    This is one of the problems often found in monorepos but can be solved with your team's adequate architecture planning. However, with submodules you can give access specific packages to a developer without giving access to the full code.

    To read more on git submodules feel free to click on this link : git-submodules

Monorepo Folder Structure

From the introduction, you should know that a monorepo acts as a container for multiple applications. Each application has access to a shared set of packages. The folder structure of a monorepo is straightforward. You have a folder that houses applications, and other packages.

Monorepo folder looks like this :

- apps/
--- app-one
--- app-two
- packages/
--- package-one
--- package-two
--- package-three

A package can be anything from UI components over functions (e.g. utilities) to configuration (e.g. ESLint, TypeScript).

Monorepo folder with contents :

- apps/
--- React-app
--- Angular-app
- packages/
--- ui
--- utilities
--- eslint-config
--- ts-config

A package can depend on another package in the packages folder. For example, the ui package can interact with the utilities package. And Both, ui and utilities package, may use configuration from the *-config packages.

Applications on the other hand do not work like that. They’re not dependent on each other. They can be deployed independently unlike packages.

Workspaces in Monorepo

Monorepo has many applications and packages working together, hence it needs tooling for everything to work. Workspace is one of the most essential enablers of monorepo. A workspace is a local package made up from your own sources from that same project.

They enable us to create a project structure where apps can use packages as dependencies. Many workspaces exist, we will be using yarn workspace.

You can also use npm or pnpm. Monorepos will be almost impossible to achieve without workspaces. They can be seen as the lifewire of monorepo setup.

Hoisting : Handling Redundancies in Monorepo

Package managers like yarn and npm have employed hoisting in their workspace algorithm.

Hoisting helps in managing redundancies as it scans your package.json files across your workspaces and figures out what the most common versions of dependencies are.

For instance, If you have 40 packages, and 35 of them are using Angular 14.19.0, but 5 are using Angular 13.01.0. It'll "hoist" the common version of Angular, 14.19.0 to the top level ./node_modules directory. This will save space and build time as common versions of dependencies are maintained.

Turborepo Overview

Turborepo is a monorepo tool that eliminates build pipelines issues present in monorepos. By providing an efficient build mechanism and package management. You can make changes across many codebases easily with the help of Turborepo.

source: @turborepo.com

According to their website : 💡 "Turborepo is a high-performance build system for JavaScript and TypeScript codebases." from : Turborepo.com. Turborepo charges your monorepo by reducing build time and giving you additional devtools. That will help you with building and continuously integrating your application.

🤷 Why use Turborepo?

Many monorepo tools exist. Tooling, automation and decoupling are Monorepo's projects core strengths. Who wouldn't like to automate particular build or run tasks that stops a developer from doing what he loves best. Monorepo tools like Lage, Bazel, Turborepo, Lerna, etc exist. They all come with one feature or the order.

The image below compares popular monorepo tools :

🔥 Turborepo being one one of the newest tools on the market, flourishes with features and sets out to compete with build tools such as webpack. Their innovation towards optimization puts them high above their competitions.

Below are some cool features of Turborepo 😎

  1. Parallel Execution ‼️

    Turborepo utilizes multiple CPU powers with modern parallel techniques. Ensuring that no CPU is left idle in a running process.

  2. Zero-Runtime Overhead ☔️

    Turborepo isolates itself from the runtime process of your code. This will ensure optimal performance of your system, as only the necessary things are used during build processes.

  3. Incremental builds 🪜

    Turborepo follows a do-not-repeat yourself approach. Turborepo will remember things you’ve built and skip the stuff that's already been computed. This will improve execution speed and faster performance.

  4. Remote Caching 🏪

    Remote caching allows you to store and retrieve the results of process execution to and from a remote server. With Turborepo, you can share a remote build cache with your team and CI/CD for faster builds.

Building a Monorepo With Turborepo

We’ve talked a lot about monorepos but lets test our theoretical knowledge of monorepos by building a practical monorepo application with Turborepo.

First and foremost, install yarn package manager. you can also use npm and pnpm too! To install yarn, run the code below :

npm install --global yarn

1. Run Turborepo CLI

The fastest way to get a feel of how Turborepo works is to run their CLi. This CLI scaffolds a monorepo with apps (docs, web) and packages (design system and shared configs (eslint, tsconfig)). You can check it out with the command below :

yarn dlx create-turbo@latest

You’ll be prompted to install Turborepo’s CLI after running the code. Enter “y” to accept.

You will be asked to also choose package managers. We’ll be using yarn, you can also use npm, yarn or pnpm.

After calling your project turbo-demo and If the process is successful, your workspace will look like the one in the image below. You can explore them individually to see what you've got.

2. Exploring the package.json and turbo.json file

From the package.json file, the entry points to our application is “turbo”. You can use this turbo together with run and then pass other custom turbo flags to accomplish most processes in Turborepo.

This is how package.js file looks like :

{
  "name": "turbo-demo",
  "scripts": {
    "build": "turbo run build",
    "test": "turbo run test"
  },
  "devDependencies": {
    "turbo": "1.2.5"
  }
}

Turbo syntax :

# command to run anything accross your packages
turbo run <task>

on your terminal, run turbo run build and then pass the flag's name. You can access all flags allowed in Turborepo from the official turbo CLI Reference page.

You can check turbo.json to see turbo dependencies. It generally looks like the following :

{
  "$schema": "<https://turborepo.org/schema.json>",
  "pipeline": {
    "build": {
      "dependsOn": ["^build"],
      "outputs": []
    },
    "test": {
      "dependsOn": ["^build"],
      "outputs": []
    }
  }
}

We'll make use of this file in the continuous integration section.

3. Running our dev scripts

Since we already have all the files we need to get Turborepo up and running from their demo sample.

# running our application at root
yarn run dev

⚠️ Both dev scripts run simultaneously, starting your Next.js apps on ports 3000 / 3001.

You can run only one of the apps using the --filter flag.

# filtering which package (app) you would like to run
yarn dev --filter docs

You'll notice that it now only runs docs:dev

You can set whether to cache your dev by going to the turbo.json and passing either true or false values.

Versioning With Monorepos

Applying versions to monorepo packages is quite different from how it's done in poly repos. This is so because packages act as dependencies for applications.

We can apply versioning to our packages with the help of our registry and changesets. With the help of npm registry, you can configure npm to publish packages to GitHub Packages and to use the packages stored on GitHub Packages as dependencies in an npm project.

In this way you can easily share packages between workspaces with the package version of choice.

Changesets is a powerful tool for versioning in monorepos.

⚠️ Before moving on go ahead and install changesets in your monorepo with the following command :

# adding 
yarn add -D changeset --ignore-workspace-root-check

add a .changeset folder at the root of the project and in it create a config.json file

You can now add the code below to your package.json scripts :

"scripts": {
  ...
  "changeset-create": "changeset",
  "changeset-apply": "changeset version",
  "release": "turbo run build && changeset publish"
},

From scripts code in package.json file, you can see that versioning includes publishing to a registry. We'll be using the npm registry.

There are some information we need to gather before we can proceed with versioning.

Below is a breakdown of our following steps :

1. Create npm Organization ✨

Go to npmjs official website to create an npm organization account.

This account is very necessary as we're going to use the login details to login in our local machine through CLI.

2. Login to a local machine 🖥️

Lets now login with the same details from npmjs. Run the code below and follow the prompts :

# logging in to your account
npm login

3. Verifying setup 👁️

you can check to see if everything goes well by running :

# installing and running dev
yarn install && yarn dev

We're almost done with the prerequisites needed for versioning.

You can make changes to any of the packages. Our goal here is to have the change reflected in our new version which gets published to npm.

Finally, It's time to version our packages with changesets. 🚀

Below are the steps to versioning in monorepo with changesets :

1. Create a changelog 🏗️

Changelog holds the logs of our changed packages.

Run the code below to create changelog with changesets :

yarn changeset-create

Follow the prompts after running the code.

You can check to see packages that have changed so far with the git command below :

# to check branches status
git status 

# or 
git diff

To see the difference from the first log and the changelog we created now.

2. Apply changes 💣

We created a changelog to view modified packages in step 1. In this step, we'll apply the changes and versions to the targeted packages.

Run the code below to apply changes :

yarn changeset-apply

You can view the changes once again with the git commands in step 1. That's: git status and git show

3. Publish to npm registry 📤

If everything goes as expected. It's now time to publish our updated packages to npm.

This can be done with the command below :

yarn release

After running this command, you can verify on npm to see if your package got a new version.

Pretty cool, right?

Continuous Integration (CI) With Monorepos 🌟

CI in monorepo can be done in many ways. It can be done with most remote repository providers (i.e Gitlab, GitHub, etc.).

For simplicity, we'll be using GitHub Actions and Turborepo for Monorepo CI.

We're using GitHub because it's one of the largest and most used remote repository providers in the world. The steps of accomplishing CI with Turborepo is quite straightforward.

Below is a breakdown of the steps :

1. Create a github/workflows.release.yml file

Turborepo uses GitHub Actions for CI by default.

Open the .github/workflows.release.yml and add the following code to work with GitHub Actions:

name: Release

on:
  push:
    branches:
      - main

concurrency: ${{ github.workflow }}-${{ github.ref }}

jobs:
  release:
    name: Release
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Repository
        uses: actions/checkout@v2
        with:
          fetch-depth: 0

      - name: Setup Node.js 16.x
        uses: actions/setup-node@v2
        with:
          node-version: 16.x

      - name: Install Dependencies
        run: yarn install

      - name: Create Release Pull Request or Publish to npm
        id: changesets
        uses: changesets/action@v1
        with:
          publish: yarn release
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          NPM_TOKEN: ${{ secrets.NPM_TOKEN }}

Note ⚠️ : To enable this workflow on your own GitHub repository, you have to create a NPM_TOKEN on npm and use it as a repository secret on GitHub.

Go to your GitHub repository settings and click on the Secrets, and then Actions tab. Create a new secret called NPM_TOKEN and enter the value of your Scoped Access Token from npm.

That's technically another way of versioning in monorepos 😇

From there the CI with GitHub actions takes over for your monorepo's packages. If the CI succeeds, it creates a new pull request with the increased version and changelog. Once this Pull request gets merged, CI runs again and releases the package to npm.

After doing that, we can proceed to pushing our changes to GitHub.

yarn changeset-create

Create a changelog for the packages with this code :

Make changes to any package of choice. Use git show or git diff to see altered files.

2. Makes changes to packages and publish

We aren't done yet. Hang on 😅

Remember to enable the "Allow GitHub Actions to create and approve pull requests" for your organization/repository. Access to packages can also be controlled with this same methodology.

Conclusion

Decoupled systems have many advantages over coupled systems. This is simply because of the rise in the need for collaboration between teams, and the need for an increase in performance with minimal system configurations.

The monolithic architecture may seem to be obsolete, but it still has some practical application in this data-driven age. For instance, a monolithic architecture is an ideal setup if you're building an inelastic app that requires small maintenance and no scaling.

Knowing when to choose any of the system architectures and approaches in this article will go a long way toward improving your overall product and team performance.

Thanks for staying to this point! 🚀