The Discipline of Coding

Next-Generation Configuration for NodeJS and ReactJS: wj-config

One Configuration Package for All Your JavaScript

José Pablo Ramírez Vargas's photo
José Pablo Ramírez Vargas
·

14 min read

Table of contents

Hello everyone. For those of you that don't know who I am, here is my presentation in a nutshell in the context of what I'll explain here: I am a senior software developer specialized in .Net, C# and generally speaking, very large enterprise systems for very large and global companies.

If you are not fond of .Net, you probably don't know that Microsoft included a very cool application configuration system in .Net Core. It is no longer that weird-working configuration XML system. It is now a key-value pair system that supports hierarchies. It sounds complex, but is really not, and most importantly: It is gorgeous. Those who know it will easily follow this article.

Enter wj-config: A Configuration System for the JavaScript World inspired on .Net Core's Configuration System

The wj-config NPM package is my first entry to the JavaScript world and came to be from the lack of really good and cross-technology (within JavaScript) configuration systems that could stop the madness created by a team of 25 junior developers that could not make a clean use of the .env configuration system in React. I also had a NodeJS Express project that needed something (not related to the mentioned dev team), so I decided I needed to tackle both simultaneously.

For my company I created something quick and I liked the result, and v1.x versions of wj-config closely resemble my first approach. It was not very clean, I needed it fast and it just had what I needed to get by.

Because it was a largely successful addition to this big microservices project, I decided I wanted to see it evolve, so here comes v2. This article covers v2 only as v1 was largely an experiment.

wj-config Overview

This is a package that exports a function. This function, when evaluated, creates a builder object. This builder object has the ability to gather information about your configuration sources as well as a few other preferences based on the features you want in your final configuration object. As you may have guessed by the previous sentence, the builder's final result will be a configuration object.

The Configuration Object

The goal of wj-config is to provide all of your configuration values in a single object. This object will contain whatever hierarchy you define in your data sources. If your data sources are hierarchical in nature, the final configuration object will also be like that.

Let's pose the first realistic example. Just like in .Net, the idea here is to create the configuration object by merging at least 2 configuration JSON files: One master configuration, and one environment-specific configuration. The master file will contain most of the configuration values, while the environment-specific one will tend to contain only value overrides specific to that environment.

NOTE: wj-config can actually work with just one data source. It is just not a common scenario.

This would be the master configuration file:

{
    "logging": {
        "minLevel": "information",
        "logFile": "/usr/shared/logs/myapplog.txt"
    },
    "app": {
        "title": "My Application",
        "version": "1.0.0"
    }
}

It contains the settings that are common to all or most of the environments (development, pre-production, production, qa, test, etc.).

But some values are not suited for development in the above configuration. The most typical one is the logging's minimum level. We usually want this value set to "debug" when we develop. Furthermore, if you're like me, your PC is Windows, so that log file path is not going to work. So here comes the environment-specific JSON file:

{
    "logging": {
        "minLevel": "debug",
        "logFile": "~/myApp/logs/myapplog.txt"
    }
}

The idea is that your configuration values will be the result of merging of the 2 JSON's above, so you will still have the app section while running in the development environment. You just didn't have to override any of those values so you didn't specify them in the second file.

The end result of merging the two JSON's above will be:

{
    "logging": {
        "minLevel": "debug",
        "logFile": "~/myApp/logs/myapplog.txt"
    },
    "app": {
        "title": "My Application",
        "version": "1.0.0"
    }
}

That would be the object result of the configuration building process: The end result of merging the data sources.

This is a very simple example. More complex scenarios are supported, including bringing configuration values from environment variables.

wj-config Features

I'll at least mention all of them here, so bear with me. I promise it is worth it.

1. Capable of using 6 different sources of configuration

The builder object has helper functions for 6 different data sources: Objects (mostly used for loaded JSON files), fetched objects, environment variables, dictionaries, single values and JSON strings. The most common one is the first one, and uses the addObject() function.

The following is code for NodeJS to import two JSON files as explained in the Overview. It is written as a module to be able to import it cleanly in as many modules as you require configuration. I name this one config.js all the time.

import wjConfig, { Environment } form 'wj-config';
import mainConfig from "./config.json" assert {type: 'json'};

// A utilitarian function to only load the environment-specific file.
const loadJsonFile = (fileName, isRequired) => {
    const fileExists = fs.existsSync(fileName);
    if (fileExists) {
        const data = fs.readFileSync(fileName);
        return JSON.parse(data);
    }
    else if (isRequired) {
        throw new Error(`Configuration file ${fileName} is required but was not found.`);
    }
    // Return an empty object.
    return {};
};

/*
wj-config provides a handy environment object.  It is built by passing the 
current environment name and an optional list of possible environment 
names.  Not explained yet, but providing the names pays dividends and 
avoids a nasty error:  Your provided environment name must be one of 
the names in the list.
*/
const env = new Environment(process.env.NODE_ENV /*
    , ['MyDev', 'MyTest', 'MyPreProd', 'MyProd']*/);

const config = wjConfig()
    .addObject(mainConfig) // Main config file.
    .addObject(loadJsonFile(`./config.${env.value}.json`)) // Environment-specific config file.
    .build();

export default await config; //The build() function is asynchronous, so await.

Pretty simple, right? Call the wjConfig() function, then add the two data sources, then build and await. Voilá! You can consume your config.js module anywhere in your NodeJS application, like this Express router example:

import config from './config.js'; // <-- Import your config module.
import express from 'express';

const router = express.Router();

/* GET home page. */
router.get('/', function (req, res, next) {
  res.render('index', { title: config.app.title }); // <-- Configuration consumed here!
});

export default router;

2. Custom data sources can be created

I won't fully elaborate this one here, but know that just like there are addXXX() functions in the builder, there is also an add() function. By creating a custom data source (all explained in detail in the package's ReadMe @ NPMJS or the project's home) you can import configuration data from anywhere your imagination allows by simply passing your data source to add().

3. It can take care of URL building for you, automatically with URL encoding

This is BY FAR the most original and amazing feature of this configuration package, at least in my opinion: If you tell it to, you can have URL-building functions created inside the configuration object that automatically create URL's for you based on the configuration data within, all with URL encoding included. Curious? This is how.

URL Building

The builder has this function, createUrlFunctions(), that will search the merged configuration data for a property named ws (or the given name or names as this is just the default name), and if this property is an object, it will recursively search in it a root node that signals the start of URL configurations.

This root node is defined as a node (a node is a non-leaf property) that has either a host property or a rootPath property.

Once the root node is found, it will replace any string property with a function that, on execution, appends the parent node's root path (including host, port and scheme) and the value found in the replaced property. So the once string property will now be a function property.

This function can accept value replacement data as well as query string data.

All this could be very confusing as plain text reading, so an example is probably worth a lot more.

Let's say you define this configuration:

{
    "app": {
        "title": "My Application",
        "version": "1.0.0"
    },
    "ws": {
        "gateway": {
            "host": "myapp.example.com",
            "scheme": "https",
            "rootPath": "/api/v1",
            "products": {
                "rootPath": "/products",
                "all": "",
                "single": "/{productId}"
            }
        }
    }
}

The above intends to define 2 REST-compliant URL's:

1. https://myapp.example.com/api/v1/products
2. https://myapp.example.com/api/v1/products/{productId}

Why so chopped, you wonder? Why defined in sooo many properties? Two URL's can be specified with 2 properties, right? Well, that's right. But what happens when you want to override something because in some environment of yours (development most commonly), you maybe want http and not https, or want localhost instead of a domain name? Well, if you go the simple way, your override will have to provide the full URL's in both the master configuration and the environment-specific configuration, and I don't endorse code repetition or configuration repetition. DRY (don't repeat yourself) applies to code and configuration.

If you didn't have wj-config to help you out, chopping the URL's like this would become a concatenating nightmare in your consuming code. For example, our micro frontends in React have over 100 URL's defined for its gateway API. Without wj-config, it would be over 100 concatenations or over 100 environment-specific overrides. Choose your poison, right? Well, no! wj-config is here as the antidote to both poisons.

By separating the host, port, scheme and the ability to set root paths you can simply override the bit of URL you need to override, just once. This enables you to define configuration data in just one place and also be able to override in just one place.

The concatenation part is given by the URL-building functions I mentioned at the beginning.

Back to the example JSON, how do we obtain the full URL we so desperately need? Simple. First, re-write your config.js like this (spoiler: It is just one extra line compared to the previous example):

import wjConfig, { Environment } form 'wj-config';
import mainConfig from "./config.json" assert {type: 'json'};

const loadJsonFile = (fileName, isRequired) => {
    const fileExists = fs.existsSync(fileName);
    if (fileExists) {
        const data = fs.readFileSync(fileName);
        return JSON.parse(data);
    }
    else if (isRequired) {
        throw new Error(`Configuration file ${fileName} is required but was not found.`);
    }
    // Return an empty object.
    return {};
};

const env = new Environment(process.env.NODE_ENV);

const config = wjConfig()
    .addObject(mainConfig)
    .addObject(loadJsonFile(`./config.${env.value}.json`))
    .createUrlFunctions() // <-- This is new!
    .build();

export default await config;

Now just consume config.js wherever you need the URL's:

import config from './config.js';

const allProductsUrl = config.ws.gateway.products.all(); // That's it.
// Outputs something like https://myapp.example.com/api/v1/products
console.log(allProducts);

Do you like it? Let me know in the Comments section.

Advanced URL Building

The above just showed you the very basics of URL building. There's more:

  1. Define placeholders that can be replaced to create more dynamic URL's.
  2. Create query strings dynamically.
  3. Create fully dynamic URL's from any node with a rootPath or host property.

By default, URL's can have replaceable pieces defined as {name}, and the name part is used to ask for a replacement value. See the project's ReadMe for details, but quickly here's how.

import config from './config.js';

const singleProductUrl = config.ws.gateway.products.single({ productId: 123 });
// Outputs something like https://myapp.example.com/api/v1/products/123
console.log(singleProductUrl);

NOTE: Any value used to replace is URL encoded for your convenience.

You can also append query strings dynamically by using the function's 2nd parameter:

import config from './config.js';

const singleProductUrl = config.ws.gateway.products.single({ productId: 123 }, { format: 'compact' });
// Outputs something like https://myapp.example.com/api/v1/products/123?format=compact
console.log(singleProductUrl);

Finally, if you only have, say, a host name and the rest you get dynamically somehow, you can use the buildUrl() function in the gateway node:

import config from './config.js';

const relUrl = calculatePartialUrlSomehow() // say, /my/dyn/path/{someId}/categories
const dynUrl = config.ws.gateway.buildUrl(relUrl, { someId: 456 }, { maxRecords: 50 });
// Outputs something like https://myapp.example.com/api/v1/my/dyn/path/456/categories?maxRecords=50
console.log(dynUrl);

4. Obtain easy access to the environment name with the Environment class

Just as I like .Net Core configuration, I also like the IHostEnvironment interface. This interface allows code to access the current environment name and even comes with a couple of Boolean functions to test if you are, for example, running in the Production environment.

Well, I also wanted that. And I got it. You have seen it. See the examples above: They create an env object from the imported Environment class.

This object contains:

  1. A value property with the current environment's name.
  2. A names property that is the array of possible environment names. If a list is not provided during construction, the list will be Development, PreProduction and Production. Yes, capitalized names. Deal with it javascripters! I come from .Net, a Pascal-cased world. :-)
  3. One isXXX() function per environment name that returns true if the system is running under that environment.

The object can be instantiated whenever is needed, but why do so? It can be added to the configuration object. See this revised version of the config.js module:

import wjConfig, { Environment } form 'wj-config';
import mainConfig from "./config.json" assert {type: 'json'};

const loadJsonFile = (fileName, isRequired) => {
    const fileExists = fs.existsSync(fileName);
    if (fileExists) {
        const data = fs.readFileSync(fileName);
        return JSON.parse(data);
    }
    else if (isRequired) {
        throw new Error(`Configuration file ${fileName} is required but was not found.`);
    }
    // Return an empty object.
    return {};
};

const env = new Environment(process.env.NODE_ENV);

const config = wjConfig()
    .addObject(mainConfig)
    .addObject(loadJsonFile(`./config.${env.value}.json`))
    .includeEnvironment(env) // <-- This is new!
    .createUrlFunctions()
    .build();

export default await config;

By adding the new line you see above, the resulting configuration object will have an environment property that will contain the env object for your convenience.

5. Capable of using environment variables to add or override configuration data

Again, just like .Net Configuration, you can use especially crafted environment variable names to inject configuration data. Just like .Net Configuration, hierarchical names are separated with double underscores (__); unlike .Net Configuration, the environment variable name must start with a prefix.

Imagine you want to provide the application's version through environment variable. Remember? We defined a version property in our configuration examples, under the app property.

This is a realistic example, because JS doesn't really bundle or compile in a way where we can access version data. It is not like a DLL whose file and product versions are accessed through the Assembly class. Maybe you have a CI/CD routine that creates the version environment variable on deployment. If that's the case, and to stop talking, you would name the environment variable OPT_app__version, and then set its value to the version number the CI/CD determined.

Now it is time to ammend config.js one more time:

import wjConfig, { Environment } form 'wj-config';
import mainConfig from "./config.json" assert {type: 'json'};

const loadJsonFile = (fileName, isRequired) => {
    const fileExists = fs.existsSync(fileName);
    if (fileExists) {
        const data = fs.readFileSync(fileName);
        return JSON.parse(data);
    }
    else if (isRequired) {
        throw new Error(`Configuration file ${fileName} is required but was not found.`);
    }
    // Return an empty object.
    return {};
};

const env = new Environment(process.env.NODE_ENV);

const config = wjConfig()
    .addObject(mainConfig)
    .addObject(loadJsonFile(`./config.${env.value}.json`))
    .addEnvironment(process.env) // <-- This is new!
    .includeEnvironment(env)
    .createUrlFunctions()
    .build();

export default await config;

That's it. OPT_ is the default expected prefix, so no need to specify a different prefix when calling addEnvironment(), but you can if you want to.

6. Where did this value come from?? Solve configuration messes easily with Value Tracing

If you ever suffer because you are getting a configuration value that should not be there, no worries, I got your back. wj-config can create a complete value trace of all configuration values in your object.

Just pass true as the first argument of the builder's build() function and upon consumption of the configuration object you will be greeted with the special _trace property. This property will tell you, for each configuration value, the data source that set it last. It actually gives you 2 values:

  1. An index numerical property. It starts with 0 goes up. Index 0 means the first configured data source set the value. You can probably figure out the rest. :-)
  2. A name string property. Every data source must provide a name. This provided name is what you'll see here.

WARNING: Not all data sources are capable of providing unique names! But I also got your back here.

Already-long story short: Let's ammend config.js one more time:

import wjConfig, { Environment } form 'wj-config';
import mainConfig from "./config.json" assert {type: 'json'};

const loadJsonFile = (fileName, isRequired) => {
    const fileExists = fs.existsSync(fileName);
    if (fileExists) {
        const data = fs.readFileSync(fileName);
        return JSON.parse(data);
    }
    else if (isRequired) {
        throw new Error(`Configuration file ${fileName} is required but was not found.`);
    }
    // Return an empty object.
    return {};
};

const env = new Environment(process.env.NODE_ENV);

const config = wjConfig()
    .addObject(mainConfig) // Non-unique name warning
    .name('Main Configuration') // So let's rename it.
    .addObject(loadJsonFile(`./config.${env.value}.json`)) // Ditto
    .name(`${env.value} Configuration`) // Ditto Ditto
    .addEnvironment(process.env)
    .includeEnvironment(env)
    .createUrlFunctions()
    .build(true); // Or .build(env.isDevelopment()); if you feel classy.

export default await config;

That's it. Output config as an object somewhere and examine the _trace property and uncover the culprit in your configuration dilemma.

import config from './config.js';

console.log('Config: %o', config); // or config._trace

Conclusion

wj-config is here to help you code and configure the right way. It is especially helpful when configuring URL's and help you stay DRY.

It allows for an enormous flexibility and programming and deployment styles. It is particularly well-suited for microservices or micro frontends.

While it is not mentioned here, the project's home has useful information including a Deployment folder with scripts to automate environment variable importation into ReactJS or other browser-based technologies/frameworks.

Happy coding! Stay DRY and SOLID.

Node.jsReactconfigurationEnvironment variablesEnvironment