😎Node Js (John Smilga - Udemy)

Node, Express, Mongo DB, Mongoose and PROJECTS! PROJECTS! PROJECTS! 👊🏻

😀 What is Node JS

Environment to run JS outside the browser.

Globals

There's no window in Node unlike in vanilla JS (browser). However, there are global variables that can be accessed from anywhere in the code. Here's some of them (there are more and not listing everything)

// Globals
__dirname  - path to current directory // /Users/sandeepamarnath/Desktop/node_tutorial  
__filename - file name
require    - function to use modules
module     - info about current module
process    - info about env where our code is executed

😀 Modules

Node uses a common js module pattern. Every js file in the node is a module. One module can be imported by the other module and we can choose what we would like to export or allow other modules to import.

Note: when you use your own modules, then to require them, use require(./mymodule). See how we are starting with ./ even though they are in the same folder. No need to use ./ if it's a built-in module.

// Code without modules

//// app.js
const secret = "SECRET"
const peter = "Peter"
const john = "John"
const sayHi = (name) => {
    console.log(`Hi ${name}`)
}

sayHi("Susan")
sayHi(john)


// ------------------------------------
// With modules

//// names.js (we need to export what we want so that other modules can use them)


// dont share this
const secret = "SECRET"

//share them
const john = "John"
const peter = "Peter"

// exports is an object in module
module.exports = { john, peter } // This is ES6 syntax, same as writing {john:john, peter:peter}

// NOTE: If we export only one, then we can say module.exports = john, and now this will just be a string and not an object

//// app.js

const names = require('./names') // see we are using ./ even though it is in same folder as explained above
console.log(names) // {john:john, peter:peter}

//// utils.js

const sayHi = (name) => {
    console.log(`Hi ${name}`)
}

module.exports = sayHi // same as 'export default sayHi' in ES6

//// app.js 

const names = require('./names')
const sayHi = require('./utils') // sayHi function

sayHi(names.john) // Hi John



// Other ways of exporting

module.exports.items = ['item1', 'item2']; 

const person = {
    name: 'San'
}

module.exports.singlePerson = person

// Other way of importing (destructuring)
const { items, singlePerson } = require('./multipleexpprt')

The tricky part of the modules

// Let's say we export or don't export (that doesn't matter), but IF WE HAVE 
// A FUNCTION CALL IN A MODULE, the module requirng it will call that function

// test.js (not exporting anything, same result even if we export something) 
 
const num1 = 5;
const num2 = 10;

function addValues() {
    console.log(`The sum is ${num1 + num2}`)
}

addValues()  // LOOK WE ARE CALLING THE FUNCTION


// app.js 

require('./test') // THIS WILL RUN THE addValues() 
// in the test.js even if we dont export from test

Built-in modules

Some popular built-in modules are

• os

• http

• path

• fs (file system)

  • sync

  • async

OS module

Provides useful methods and props to interact with OS and the server

// Basic os module related code

const os = require('os') // no ./, just 'os' as it is a built in module

// info about current user
console.log(os.userInfo())


// system uptime in seconds (goes to 100 seconds and not 60)
console.log(os.uptime())

const currentOs = {
    name: os.type(),
    release: os.release(),
    totalMem: os.totalmem(),
    freeMem: os.freemem()
}

console.log(currentOs)

Path module

Provides useful methods such as getting relative path, absolute path and so on

// Basic path module related code

const path = require('path')
console.log(path.sep) // gives / or \ depending on your OS. It's path separator

const filepath = path.join("/", 'content', "test.txt")
console.log(filepath) // gives /content/test.txt

console.log(path.basename(filepath)) // test.txt

// get absolute path (full path)
const absolute = path.resolve(__dirname, "content", "test.txt")
console.log(absolute)

FS module (file system)

const { readFileSync, writeFileSync } = require('fs')
const os = require('path')

// to read file, we need to provide two things
// * file path 
// * encoding (like utf-8)

const first = readFileSync('./content/first.txt', 'utf8')
const second = readFileSync('./content/second.txt', 'utf8')
// reads the content of the file in sync way
console.log(first) // Hello this is first text file
console.log(second) // Hello this is second file

writeFileSync('./content/result-sync.txt',
    `Here's the result : ${first} ${second}`,
    { flag: 'a' }) // flag 'a' means append. It will not overwrite the data, instead will accumulate it

const { readFileSync, readFile } = require('fs') // readFile is Async

//SYNC VERSION
const first = readFileSync('./content/first.txt', 'utf8')
console.log(first)


//ASYNC VERSION
readFile('./content/first.txt', 'utf8', function (err, result) {
    if (err) {
        console.log(err)
        return
    }
    console.log(result)
})


// The above is same as 

//ASYNC VERSION
readFile('./content/first.txt', 'utf8', cb)


const cb = function (err, result) {
    if (err) {
        console.log(err)
        return
    }
    console.log(result)
}

Async file write

//ASYNC VERSION
readFile('./content/first.txt', 'utf8', function (err, result) {
    if (err) {
        console.log(err)
        return
    }
    const first = result
    console.log("Now getting second")
    readFile('./content/second.txt', 'utf8', (err, result) => {
        if (err)
            return console.log(err)
        const second = result
        console.log("Writing the result into a file")
        writeFile('./content/result-async.txt', second, () => { }) // not capturing result and error in callback
        
    })
})
// THIS IS CALLBACK HELL

// result from callback will be undefined even if we capture in the callback just like this

 writeFile('./content/result-async.txt', second, (err, res) => {
            if (err) return console.log(err)
            console.log(res)
        })
        
        
// appending by giving flag 'a'
 writeFile('./content/result-async.txt', second, { flag: 'a' }, (err, res) => {
            if (err) return console.log(err)
            console.log(res)
        })

Sync vs Async file reads and writes

Sync file read/write

const first = readFileSync('./content/first.txt', 'utf8')
const second = readFileSync('./content/second.txt', 'utf8')
// reads the content of the file in sync way

console.log("Started");

writeFileSync('./content/result-sync.txt',
    `Here's the result : ${first} ${second}`,
    { flag: 'a' }) // flag 'a' means append. It will not overwrite the data, instead will accumulate it
    
    
console.log('done with this task')
console.log('starting with the next task')


// OTUPUT

// Started
// done with this task
// starting with the next task

// If write file is big then it takes lot of time and blocks the next lines of code
// so no other user can do anything with the app if one user is writing a big
// file

Async file read/write

const { readFileSync, readFile, writeFile } = require('fs') // readFile is Async

//SYNC VERSION
const first = readFileSync('./content/first.txt', 'utf8')
console.log("Task 1 started")


//ASYNC VERSION
readFile('./content/first.txt', 'utf8', function (err, result) {
    if (err) {
        console.log(err)
        return
    }
    const first = result
    readFile('./content/second.txt', 'utf8', (err, result) => {
        if (err)
            return console.log(err)
        const second = result
        writeFile('./content/result-async.txt', second, { flag: 'a' }, (err, res) => {
            if (err) return console.log(err)
            console.log("End of Task 1")  // Appears at the end in console
        })
    })
})

console.log("Task 2 started")


// OUTPUT
//Task 1 started
//Task 2 started
//End of Task 1

// You see the writing file task isn't blocking

HTTP module

HTTP module is all related to server-side coding in node. We can use node for back-end(server-side) because of HTTP module. HTTP does many things and few are listed here

Using HTTP,

• We can create a server

• That server can take request and send back the response

• That server can define a port on which the client can send request

// setting up simple server

const http = require('http');

// server takes a callback function that exposes request and response.
// When we send any request (as a client) to 3000 port, and server responds 
// with res.write(). We can see this response in the browser when we type localhost:3000


const server = http.createServer((req, res) => {
    res.write("Hi sandeep, this is my server sdfa")
    res.end() // end of response. If this is not included, then browser will not stop querying the server and never get a response from server
})

server.listen(3000) // server is listening on port 3000. This can be any port

// There are lot of other methods and variations but this is the basic setup of a server





// Another example

const http = require('http')

const server = http.createServer((req, res) => {
    console.log("Request received... Here's your response")
    res.end("My response")
})

server.listen(5000, () => {
    console.log("I'm the server who is listening to the requests on port 5000")
})


// server.listen can take a callback. This will be executed soon after the 
// server has been setup, even before client sends the request to server

The server that can handle basic requests

const http = require('http');
// simple server

// const server = http.createServer((req, res) => {
//     res.write("Hi sandeep, this is my server sdfa")
//     res.end()
// })


const server = http.createServer((req, res) => {
    if (req.url === "/") {

        // res.write("Hi sandeep, this is my server sdfa")
        res.end("Hi sandeep, this is my server sdfa")
        return
    }
    if (req.url === "/a") {
        // res.write("Hi sandeep, abput page")
        res.end("Hi sandeep, abput page")
        return
    }

    res.end("This is no good")
})


server.listen(3000)

NPM (Node package manager)

When we make a change to the file, we need to run node app.js command to rerun the code. To avoid that Install nodemon to watch the changes made to the file.

// package.json
"scripts": {
    "dev": "nodemon app.js"
  },
  
 // In command line
 npm run dev
 
 // You can also do 
 "scripts": {
    "start": "nodemon app.js"
  },
  
  // In commandline
  
  npm start // npm run start also works but you can omit run. Not possible for all commands, for example 'npm run dev' is no possible.
  
  
  // I cannot do this in cmd line
  nodemon app.js // gives error because I have installed nodemon locally with command - npm i nodemon
  
  // To work gloabally (any folder and command line) then use
  npm install nodemon -g // installs globally
  
  // Now you can run this is cmd line
  nodemon app.js // works fine now.

NPX (x - execute)

npx offcially means node package runner. Why do we need it?

• Let's say you need to run app.js. How to do it?

• // In cmdline type 
  node app.js
  OR
  node app

• Now what if you want to keep track of changes? We install nodemon

• // Installing locally as I don't need it globally
  npm i nodemon

• How to run nodemon? Write a command in script tag and then use it

•  "scripts": {
      "start": "nodemon app.js"
    },
    
    // In cmd line
    npm start

• Can I run nodemon app.js in command line instead of putting this in scripts and then calling that in cmd line like npm start? No you can't. For that you need to install nodemon globally.

• // To install nodemon globally
  npm i nodemon -g

• But what if I don't want to install nodemon globally but still want to run nodemon app.js in command line? Is it possible?

• Absolutely possible with npx. npx is introduced in npm version 5.2 to solve this issue. It is introduced to run a package from command line without having to install it globally.

• // To install nodemon locally
  npm i nodemon
  
  //To run package in command line
  npx nodemon app.js
  
  OR 
  
  npx nodemon app

util

It provides the utility methods that could be very useful. For example

• promisify - returns back a promise so we can avoid the call back hell. Will see this later in action.

😀 Blocking code for all the users/ resources

Let's say we have 3 resources

• /

• /about

• /unknown

If there is a huge for-loop in about section like shown below and there are 10 users visiting our site. The site runs smoothly for everyone until one of the users visits /about. The moment he visits /about where it takes a lot of time to execute for-loop and since Node is single-threaded, all other users who visits any page will be waiting for the page to load and all the pages will be buffering for all the users until the for-loop in the about page has completed.

const http = require('http')

const server = http.createServer((req, res) => {
    if (req.url === '/') {
        return res.end("Home page")
        // return
    }
    if (req.url === '/about') {
        for (let i = 0; i < 10000000; i++) { console.log("For loop", i) }
        return res.end("About page")

    }

    res.end("My response")
})

server.listen(5000, () => {
    console.log("I'm the server who is listening to the requests on port 5000")
})

Hence this is a bad design due to the blocking code.

😀 Promisifying callbacks for ease (To avoid cb hell)

What the heck is promisifying? Why do we need Promise all of a sudden?

Let's see the below example we already know.

The first example is simple but the second example is complex. First file read becomes the input to the second file read and second file read becomes the input to file write. This leads to the call back hell (one call back inside another inside another and so on). We can avoid callback hell by promisifying this. The consumption of the data will then be linear even though one depends on the other. Let's see how.

//Example 1 :  Simple Async file read code. (Everything handled inside callback function)

fs.readFile("./content/first.txt", 'utf8', (err, data) => {
    if (err) return console.log(err)
    console.log(data)
})


//Example 2 :  A bit complex file read, file read and file write code. (Everything handled inside callback function)
// The dependencies on one another below lead to callback hell
fs.readFile("./content/first.txt", 'utf8', (err, data1) => {
    if (err) {
        return console.log(err)
    }
    console.log(data1)
    const writeData = data1

    // 2nd file read below depends on first file read data
    fs.readFile('./content/second.txt', 'utf8', (err, data2) => {
        if (err) return console.log(err)
        console.log(data2)

        // file write below depends on second file read data

        fs.writeFile("./content/writedata.txt", `My data is ${data1} and ${data2}`, { flag: 'a' }, (err, data) => {
            console.log("The data has been written ")
        })

    })
})

Promisifying the above examples

The idea is,

const asyncFunction = ()=> { // this function returns a promise
  return new Promise((resolve,reject)=>{
    
    // your async code that you have written above (but slightly modified) and instead of returning the 
    // error and data or console logging it, you just resolve the data and reject 
    // the error
  
  }
}

// Then console log or do anything with that error and data outside here
asyncFunction().then(resp => clg(resp).catch(err=>clg(err))

// Example 1

const fs = require('fs')

// getText is an async function as it returns a promise. Hence .then() can be called on getText()
const getText = () => {
    return new Promise((resolve, reject) => {
        fs.readFile("./content/first.txt", 'utf8', (err, data) => {
            if (err) reject(err)
            resolve(data)
        })
    })
}

getText().then((res) => console.log(res))
        .catch(err => console.log(err))

//-----------------------------------------------------------

// Example 2 (Considering only 2 reads (one depends on the other and not consiering write))

// Notice that we are writing the callback only for one read file and we
// can use it twice

const getText = (path) => {
    return new Promise((resolve, reject) => {

        fs.readFile(path, 'utf8', (err, data) => {
            if (err) {
                reject(err)
            } else {
                resolve(data)
            }
        })

    })
}


//Method 1 (then-catch, cb hell) -  utilizing the example 2 with promise then-catch (cb hell still forms)
// this still forms callback hell because in LINE 1 we are not returning the getText call so we can chain then-catch outside

getText("./content/first.txt").then(res => console.log(res))
    .catch(err => console.log(`1st link error`))
    .then(getText("../content/second.txt") // LINE 1
        .then(res => console.log(res))
        .catch(err => console.log(`2nd link error`)))

//-----------------------------------------------------------

//Method 2 (then-catch,  no cb hell) -  utilizing the example 2 with promise then-catch (cb hell doesn't form)
// this doesn't form cb hell because in LINE 1 we are returning the getText() call, so we can chain outside

getText("./content/first.txt").then(res => console.log(res))
    .catch(errr => console.log(`1st err`))
    .then(() => getText("./content/second.txt")) // LINE 1
    .then((res) => console.log(res))
    .catch(err => console.log(err))

// METHOD 2 is good but still there is a better approach using async-await shown below in method 3

//-----------------------------------------------------------

// Method 3 (async -await, no cb hell, no then-catch) - Better than then-catch outer chain (method 2)
// But for async-await to work, the function in which async await is used must be async function

// notice this is an async function as we are using async await inside it
 async function method3() {
    try {
        const first = await getText("./content/first.txt");
        const second = await getText("./content/second.txt");
        console.log(first, "From method 3")
        console.log(second, "From method 3")
    } catch (err) {
        console.log(`Error from method 3`)
    }
}

method3() 

We didn't take the example of write. Meaning, I considered only second read depends on first read and didn't consider write depending on second read as we considered in the Example 2 above. That's because it would become very complex to implement that because we can only resolve or reject something which is then called in then or catch block. Let's consider it now below.

// RECAP : EXAMPLE 2 FROM ABOVE

fs.readFile("./content/first.txt", 'utf8', (err, data1) => {
    if (err) {
        return console.log(err)
    }
    console.log(data1)
    const writeData = data1

    // 2nd file read below depends on first file read data
    fs.readFile('./content/second.txt', 'utf8', (err, data2) => {
        if (err) return console.log(err)
        console.log(data2)

        // file write below depends on second file read data

        fs.writeFile("./content/writedata.txt", `My data is ${data1} and ${data2}`, { flag: 'a' }, (err, data) => {
            console.log("The data has been written ")
        })

    })
})

Promisifying the above example

// FOR READ
const getText = (path) => {
    return new Promise((resolve, reject) => {
        fs.readFile(path, 'utf8', (err, data) => {
            if (err) {
                reject(err)
            } else {
                resolve(data)
            }
        })

    })
}

// FOR WRITE
const writeText = (path) => {
    return new Promise((resolve, reject) => {
        fs.writeFile(path, "This is normal text written", (err, data) => resolve("Done writing"))
    })
}

Let's see how to use them to escape the call back hell using two methods

• Method 1 - then-catch

• Method 2 - async-await

// Method 1 - then-catch
getText("./content/first.txt").then(res => console.log(res))
    .catch(errr => console.log(`1st err`))
    .then(() => getText("./content/second.txt")) // LINE 1
    .then((res) => {
        console.log(res)
        return writeText("./content/wtext.txt")
    })
    .catch(err => console.log(err))
    .then(res => console.log(res)) // Done Writing
    .catch(err => console.log(err))

//Method 2 - aync-await
async function method2() {
    try {
        const first = await getText("./content/first.txt");
        const second = await getText("./content/second.txt");
        console.log(first, "From method 3")
        console.log(second, "From method 3")
        const writefile = await writeText("./content/first.txt")
        console.log(writefile)
    } catch (err) {
        console.log(`Error from method 3`)
    }
}
method2()

// Method 2 is the best as it keeps the code cleaner without then and catch. 
// It's syntax is almost same as normal sync code

More simplified version

// This is a function that returns a promise (promisify)
// this entire boiler plate code is based on fs.readFile

const getText = (path) => {
    return new Promise((resolve, reject) => {
        fs.readFile(path, 'utf8', (err, data) => {
            if (err) {
                reject(err)
            } else {
                resolve(data)
            }
        })

    })
}

// We can simplify this using builtin promisify which does the same as above. 

Built in promisify

// The above code can be simplified using util module that provides promisify functionality
const util = require('util')
const readFilePromise = util.promisify(readFile) // does the same as above code
const writeFilePromise = util.promisify(fs.writeFile)


// using them

async function consumePromise() {
    try {
        // const first = await getText("./content/first.txt");
        // const second = await getText("./content/second.txt");


        const first = await readFilePromise("./content/first.txt", 'utf8')
        const second = await readFilePromise("./content/second.txt", 'utf8')

        console.log(first, "From consme promises")
        console.log(second, "From consume promises")
        await writeFilePromise("./content/sandeep.txt", "SANDEEP")
    } catch (err) {
        console.log(`Error`)
    }
}

consumePromise()

Can this be more simplified?

Yes, this can be more simplified like below. We don't even need utli but just use require('fs').promises

const { readFile, writeFile } = require('fs').promises


async function mostSimplifiedReadAndWrite() {
    try {

        // METHOD 1 - Complex method using our own promises

        // const first = await getText("./content/first.txt");
        // const second = await getText("./content/second.txt");

        // METHOD 2 - Simple method using util

        // const first = await readFilePromise("./content/first.txt", 'utf8')
        // const second = await readFilePromise("./content/second.txt", 'utf8')

        // METHOD 3 - MOST SIMPLIFIED method using require('fs').promises

        const first = await readFile("./content/first.txt", 'utf8')
        const second = await readFile("./content/second.txt", 'utf8')

        console.log(first, "From method 3")
        console.log(second, "From method 3")
        await writeFile("./content/sandeep.txt", "NODE IS GOOD")
    } catch (err) {
        console.log(`Error from method 3`)
    }
}

mostSimplifiedReadAndWrite()

Remember: To avoid the call back hell, we could

• Create our own promises

• Use utli promisify

• Use require('fs').promises which is the above code

😀 Event-driven programming

When a certain event happens, the callback related to that event will fire. This is called event-driven programming. Node is event-driven.

event on and emit

multiple on on single emit

events with arguments

no args for 2nd one

Even though you don't write your own events always, the events are the core building block of the node. A lot of built-in modules rely on them so we are using them anyway.

For example,

Built-in example of EventEmiiter

😀 Streams

• Used to write or read sequentially

• When we have to deal with a large file, streams come in handy

• 4 types of. stream in node

  • writeable - used to write data sequentially

  • readable - used to read data sequentially

  • duplex - used to both read and write sequentially

  • transform -. data can be modified while writing or reading

• Streams extend event emitter class

Need of streams

In both sync and async, when we are reading a file in sync or async way, we are reading the whole file at once and place them into a variable. But if the file is too big then we get an error saying we can't place everything in a string. So the solution would be readStream.

need for stream

read stream

The read stream also has options where we can set how much size the file should be and what format the data should be

Let's see another example as to why we need streams

The client (browser) is requesting the data which is in the server (on a file called big.txt - which is 2MB). Sending the large file from server to browser in one shot (one chunk) is not a. good idea because what if the internet drops and the file might not reach. It's always good to send the files in streams (chunks)

Large file in one chunk

Let's do the same thing in streams now

reading data in streams which is sent to client

read in streams

sending in streams

Writing in chunks using pipe method on the file stream

pipe

DON'T WORRY ABOUT LARGE SIZE HERE, CHECK THE NEXT IMAGE

ALWAYS SEE IF DATA IS CHUNKED IF THE RESPONSE IS LARGE

Need for Express Frame work

Express is built on http module. In order to better understand the need for Express, we need to understand a few more things about http module.

HTTP continued

basic server response

client (browser) received the response sent by server

Currently there are two problems,

• We are not setting the headers on the server (like status code and what type of data the server is serving)

• If we navigate to any resource,say localhost:3000/anything, we still get the same Hello page

Let's handle first problem of handling headers and status code. Note that status code will be set by default (atleast that's what I saw. Even if I didn't specify, it was present)

Example 2

Example 3

Example 4

All good, then what's wrong with http module and why we need express?

If we have have an HTML, then obviously will have CSS and JS and other resources liked to it like images and so on. We need to rewuest for every single resoruce and handle that individually. That's a. lot of overhead if app grows.

Let's handle all the requests using http

Handling all the requests using http

handling all client requests using http

Express

Express - Node.js web application framework

Express is a fast, minimalist web frame work for node.

Install

 npm install express --save

// Earlier if you omit --save then the package would just save it to your local but 
// not in package.json so that when you push to git then next person couldn't 
// get it. 

// But now that problem doesn't exist anymore

// To install older version just type
 npm install express@4.17.1 --save //or any other version u like 

Basic setup

Similar to what we did in http module.

const express = require('express');
const app = express();

app.get("/", (req, res) => {
    console.log("User hits the resource")
    res.send("HOME PAGE")
})

app.get("/about", (req, res) => {
    console.log("User hits the resource")
    res.send("About PAGE")
})

app.listen(5000, () => {
    console.log("Server started listening")
})

In express, we generally do these operations

We can send back the status before sending the response. Also, we can handle all other requests using all method.

const express = require('express');
const app = express();

app.get("/", (req, res) => {
    console.log("User hits the resource")
    res.status(200).send("HOME PAGE") // we can chain these methods like this
})

app.get("/about", (req, res) => {
    console.log("User hits the resource")
    res.status(200).send("About PAGE")
})

// for all others
app.all('*', (req, res) => {
    res.status(404).status(404).send('<h1>Resource not found</h1>') 
})

app.listen(5000, () => {
    console.log("Server started listening")
})

First Express App

Sending HTML file to the client

const express = require('express')
const path = require('path')
const app = express()

app.get("/", (req, res) => {
    res.sendFile(path.resolve(__dirname, "02-express-tutorial/navbar-app/index.html"))
})

app.listen(5000, () => {
    console.log("Server is listening")
})

We get the same error (not having access to css and js) like we got when using http module.

To solve this we use a middleware like this

Now, time to think🤔 I said that if you place static files in the middleware then the server will get access to those files like styles and js and so on. So I first wrote, res.sendFile() and pointed an HTML inside sendFile() so that when user/client requests localhost:5000/, then the server gets access to HTML page we are sending and then for styles it can lookup the middleware folder and serve those as well to the client.

But the question is, isn't HTML page which is placed inside sendFile() isn't static as well 🤔

Definitely it is. So instead of sending this HTML file through sendFile(), we can add this HTML file to the public folder itself where middleware is pointing to. That will do the same job and we will be in good shape. Let's see how.

So the first option to send HTML file to client was through sendFile but mostly for HTML file sending we don't use this much. The second option was by placing HTML file inside middleware (adding HTML file into the static folder). And the third option is by using SSR(Server side rendering) where we use template engine.

Why Express is used? Use cases, in other words

• To setup APIs -> RES.JSON()

• Setup templates with Server Side Rendering. We send back entire HTML, CSS and JS ourselves to the client using RES.RENDER()

1. Let's build API using Express

A simple API

Where to find in DOCS?

Express Docs

JSON output on the browser

Enhancing the app a bit

At this point, let's enhance what we are sending (JSON) a bit.

When you see any website, if they display products, then probably they display only a few things about the product like name, image and price. Once we click on the product then it displays all the details like description, make, expiry date, seller and so on.

So the idea is, when the client requests /products lets send only id, name, image first. Later, when the client requests for a specific product (product/3) then we can send all the details related to that product.

Sending only a few details in each product when request comes for all the products

To get a single product we use route parameters

Route parameters

The problem here is we are hardcoding 1. What if we need product 2? So let's use route params.

But what if I access the product that doesn't exist? It returns undefined and we need to display that.

FYI, the route params can be more complex than the above illustrated one. Let's see an example

complex route params

Query-string parameters OR URL parameters

It's a way to send small amounts of information to the server through the URL. These params are not part of the URL but they are generally used to query the database or sort the results and something like that. The person who sets up the server (Node dev or backend team) decides which params will be accepted.

To give you an example, let's navigate to the hacker news API below

HN Search powered by AlgoliaHN Search

There's a small mistake in the above screenshot. ?query= is not "Where". Actually, until /search it is URL as mentioned and ? is the where. What I'm saying above is, in the above URL which is until /search, get me query = foo and tags = story.

So, upon logging req.query we get {query:'foo',tags:'story'}.

This is the general convention where we use ourDomain.com/api/version/something and here v1 is the version.

Let's design the server in such a way that it accepts params called search and limit. It means, the user is searching for a specific product and limiting the results to a certain number.

Example 1 - The limit is 2 and search is alb

Example 2 - The search is ent

Example 3 - limit is 3

Example 4 - Some random param which is not taken care of by the server (We give back all the results)

Example 5 - Not giving any param will also result in same above (will get back everything)

Example 6 - What if no results exist for the search parameters passed?

By default if we don't handle then

See we are sending back an empty array as no results were found for the search where name starts with p. Wouldn't it be nice if we can send some message saying -> "Your search URL is correct but unfortunately we didn't find anything for this match"

To tell the first part, we send 200 status code that says -> "Your search URL is correct"

For the second part we can send the data like below

Or I can also do this

It's up to us (backend devs) to what we send back if the data is not found

The gotcha to keep in mind. We have to send one and only one response in a request. If we are using if condition to specify the response then we need to return the response like below

DON'T Do this

This is like sending two responses which will result in error

Error caused by the above code

DO this

Middleware in Express js

I found Kyle's video interesting about middleware, so please watch this before you proceed

middleware

Need for the middleware

We can do something like this

We can do better than this like below

middleware passing the control back

You see that the page is growing bigger so it would be nice to have the getData in a separate file. Let's call that logger here after. Also, wouldn't it be nice if we have a function that can add the logger / getData to all the routes by default instead of we adding them manually? We'll we have such a functionality. Let's take a look.

• Let's first move the logger to a separate file

logger.js

using logger from a different file

• Let's now add the logger programmatically to all the URLs without us adding manually. In that way, if logger (variable name of logger) changes later, we don't need to change in all the routes.

App.use()

The above is achieved using app.use(logger)

We pass the logger into app.use() and then that is equivalent of passing middleware in all the routes.

Rules of app.use()

• In order for accessing app.use(logger) in all the routes, the app.use() must be placed at the top of all the routes. Order matters here

• Let's say we need to apply the logger to only the routes starting with /api (or something) we can define that as the first parameter and logger as the second parameter.

rules of app.use()

Multiple middlewares

multiple middlewares

Let's mimic authorization here (real authorization functionality might look different).

authorize 1

authorize 2

authorize 2 (continued)

this authorization was for demonstration but in real authorization, we check for JSON Web token and if the token exists then we communicate with the DB and get the user (all this stuff will come up later)

Now that we have the middle wares setup, we can now access that info in any route. For example, in authorize middleware, the req.user = {name:'john',id:3}. We can access this in any route as every route will have access to this middleware in our current setup. To prove that let's add it in /items and will see how we can access it there:

Now let's say, like before, I don't want to apply the middleware to all my routes. For example, I only need to check for authorized users in /api/items

using middleware only in /api/items

Options of middleware

• Our own middleware what we saw till now

• Express provided middleware

• Third party middleware

Express provided middleware - example

In this example, all the static assets are placed in public folder and we are letting express know that using this above express middleware so that while rendering HTML page, it can include this public folder where all the assets are placed

Third party middleware

Let's install a logger middleware (3rd party) called morgan

npm: morgannpm

This helps in logging just like logger we used our own

Http methods

Get (default method the browser performs)

To get all the data

get

Post (add/insert data to server)

Before going to POST, jog your memory here Render static files

In order to explain the POST request, let's start with adding static resources and rendering it into the browser

Example 1 - using index.html

The reason we do this is because we cannot simply configure our browser to perform a POST request. We need a simple app like this or a tool like POST MAN which we will see later.

So let's take a look at our index.html which is currently displaying in the browser and understand what's going on

method - post and action - /login

We know this anyway that we won't get the data as we are not handling /login in our express app yet but the point to notice here is below. Once you do the post request, got to Network tab and then see it's a post request.

Also, to see what data we passed in the body of POST scroll to the bottom after clicking the Network tab and see below as shown below

post data (body)

Let's handle the post data

handling post request

At this point I will not be able to access the body (data) sent to the server (in other words inside post method) as I will not have access to name = 'john' inside post method. Hence I will not be able to add the user john to my list of people. This is where the middleware comes into picture. We use express middleware (builtin) like this.

Note that the req doesn't have a body property initially, so req.body will be undefined before defining the middleware.

parsing body

Now, what is app.use(express.urlencoded({ extended: false }))

https://stackoverflow.com/questions/23259168/what-are-express-json-and-express-urlencoded

In the early versions of express, the body parser ({extended:false} in this case) came only upon installing an npm package as explained in above article. Now it comes by default. But what exactly is this body parser option?

encoded

I know it's not very clear at this point why we need extended. I'll update this once I get clarity.

We could also install body parser and do this as explained in the video below

https://www.youtube.com/watch?v=vKlybue_yMQ&ab_channel=codedamn

Under the hood, the urlencoded (new one in express) uses the body-parser (old one)

Small task with POST request we learned so far

TASK: If name is provided, then welcome that user, else give the 401 error and ask him to provide credentials

Example 2 - using javascript.html

Here, we are submitting the form using axios package in Javascript unlike directly using html form in previous example. axios is similar to fetch and they both behave the same way. axios is chosen here as it has better error handling and stuff but the concept remains same.

form submission using javascript

server handling post data submitted by javascript form

For forms submitted using js, the content type is application/json

post body

The point to notice here is, when the html form is submitted and the post is handled in backend using app.use(express.urlencoded({ extended: false })), when the server sends the data using res.send("Success"), the front-end page form disappears and this message appears to the client in the browser.

On the other hand, when the JS form is submitted using fetch or axios, the post is handled in backend using app.use(express.json()) , when the server sends the data using res.send("Success") or res.json({message:'Success'}) and so on, the front-end page doesn't disappear but the data is reached to client and can be handled by client's JS as shown in the above example to display something on the screen.

TASK: If name is not provided then handle that when the data is submitted using JS form

handling in post data if body is not provided in JS form post

At this point, please navigate to Post man tool, understand it and come back here

Put (Update the data)

We use route parameter. Let's say we need to update the data of order 2, then

PUT

This is a convention. Technically, there are multiple other ways we can set this up. We will cover the official name of this convention and more details about it later.

Let's first see the console log data in PUT method sent through POSTMAN.

put method access in post man

PUT basic implementation

put method basic implementation

PUT implementation to update the new data provided

PUT implementation

Success case where person id exists

PUT success

Failure case where person id doesn't exist

PUT failure

Put VS Patch

Patch is used to update only the required fields. Let's say we pass the id and then would like to update only the name and nothing else, then we do it with patch.

Whereas in Put, the ones we are passing only will get updated and rest will not be retained but will be removed.

Think of it like, in react, we copy the existing values and then update the required ones -> This is what patch does

For put, think that we are only passing the values we need and not copying the other ones so that will be removed -> This is what Put does

Delete

We don't need to pass in the body for the delete request

delete

post man delete

Post-man tool

You saw that to test the get and post methods we had to setup a front-end app. That is so time consuming. Post-man allows us to write backend without worrying about front-end

Get

Post

we sent this using front-end JS form

We can send the same thing like this in post-man

post data

this body what we are sending is captured by express.json() middleware and adds it to req.body in the server.

TASK : Add another post method and handle /api/postman/people

At this point, navigate back to here, take a look at PUT and DELETE methods and then come back

Express Router

Basic router setup

Our complete app until now. We have only a couple of routes and you can see how big the app.js file already. So the solution is to use express router where we can group the routes together and as far as the functionality, we can set them up as a separate controllers.

Later when we talk about database, we will cover the common convention which is MVC pattern. That's not a rule but most used pattern that structures our code.

app.js

const express = require('express')
const app = express()
let { people } = require('./data')
//static assets
app.use(express.static('./methods-public'))

//parse the html form data
app.use(express.urlencoded({ extended: false }))

app.get("/api/people", (req, res) => {
    res.status(200).json({ success: true, data: people })
})
// parse javascript form data
app.use(express.json())

app.post("/api/people", (req, res) => {
    const { name } = req.body
    if (!name) {
        return res.status(400).json({ success: 'false', msg: 'please provide name value' })
    }
    res.status(201).json({ message: "success", person: req.body.name })
})

app.post("/api/postman/people", (req, res) => {
    const { name } = req.body
    if (!name) {
        return res.status(400).json({ success: 'false', msg: 'please provide name value' })
    }
    res.status(201).json({ success: true, data: [...people, { name: req.body.name }] })
})

app.post("/login", (req, res) => {
    console.log(req.body)
    if (!req.body.name) {
        return res.status(404).send("Please give us the credentials")
    }
    res.send(`The name is ${req.body.name}`)
})

app.put("/api/people/:id", (req, res) => {
    // we should get two data. 
    // One is the id we need to update
    // Two is we need the new data to be updated to
    const { id } = req.params // this will be string
    const { name } = req.body

    // first see if that ID exists. 
    // converting str to the number as per people array. +id is same as Number(id)
    const person = people.find(person => person.id === +id)
    if (!person) {
        return res.status(404).json({ success: false, msg: `No person with id ${id}` })
    }

    const newPeople = people.map(person => {
        if (person.id === +id) {
            person.name = name
        }
        return person
    })
    res.status(200).json({ success: true, data: newPeople })
})

app.delete("/api/people/:id", (req, res) => {
    const { id } = req.params
    const person = people.find(person => person.id === +id)
    if (!person) {
        return res.status(404).json({ success: false, msg: `No person found with id ${id}` })
    }
    const newPeople = people.filter(person => person.id !== +id)
    return res.status(200).json({ success: true, data: newPeople })
})

app.listen(5000, () => {
    console.log("Server started listening on 5000")
})

You see we have a common type of code /api/people, /api/people/postman, /api/people/:id. Don't you think we can somehow group them?

Let's move all the routes starting with /api/people to people.js.

Steps to turn normal app into express router

• In people.js import router from express.Router

• Import all the above marked routes starting with /api/people from app.js to people.js

• Replace all the app to router in people.js

• Cut and paste any related data like people import from app.js to people.js

• Import people routes from people.js

• Write middleware app.use('/api/people',people) in app.js to setup the base path so that when we hit /api/people from postman, it hits this middleware and then takes us to people.js routes

• In people.js, remove /api/people in all routes as we are defining them in app.js middleware as said in above point.

• Test all the routes in postman now.

app.js

people.js

const express = require('express')
const router = express.Router()
let { people } = require('../data')

router.get("/", (req, res) => {
    res.status(200).json({ success: true, data: people })
})
router.get("/:id", (req, res) => {
    const personId = +req.params.id
    const person = people.find(person => person.id === personId)
    if (!person) {
        return res.status(404)
        .json({ success: false, msg: `No person found with the id ${personId}`})
    }
    return res.status(200).json({ success: true, data: { ...person } })
})
router.post("/", (req, res) => {
    const { name } = req.body
    if (!name) {
        return res.status(400)
        .json({ success: 'false', msg: 'please provide name value' })
    }
    res.status(201).json({ message: "success", person: req.body.name })
})
router.post("/postman", (req, res) => {
    const { name } = req.body
    if (!name) {
        return res.status(400)
        .json({ success: 'false', msg: 'please provide name value' })
    }
    res.status(201)
    .json({ success: true, data: [...people, { name: req.body.name }] })
})
router.put("/:id", (req, res) => {
    // we should get two data. 
    // One is the id we need to update
    // Two is we need the new data to be updated to
    const { id } = req.params // this will be string
    const { name } = req.body

    // first see if that ID exists. 
    // converting str to the number as per people array. +id is same as Number(id)
    const person = people.find(person => person.id === +id)
    if (!person) {
        return res.status(404)
        .json({ success: false, msg: `No person with id ${id}` })
    }

    const newPeople = people.map(person => {
        if (person.id === +id) {
            person.name = name
        }
        return person
    })
    res.status(200).json({ success: true, data: newPeople })
})

router.delete("/:id", (req, res) => {
    const { id } = req.params
    const person = people.find(person => person.id === +id)
    if (!person) {
        return res.status(404).json({ success: false, msg: `No person found with id ${id}` })
    }
    const newPeople = people.filter(person => person.id !== +id)
    return res.status(200).json({ success: true, data: newPeople })
})

module.exports = router // exporting router so that it can be imported as people or something else

Let's now do the same thing for login in auth file

auth.js

app.js

Controller

You see that we are now using router to separate the functionality but still when you look at people.js above, it's still a big file with so many methods and callback functions inside it. Wouldn't it be nicer and cleaner if we can separate the callback functions into a separate file?

controller

Yes we can do that and call them controllers. Controllers are the functions that are present inside GET, POST and other http methods. This can be called controllers as we are now slowly transforming this into MVC ( Model View Controller) pattern. Let's leave auth as is as it is small file and convert people.js

Setting up controller folder and inside it the file people.js

people.js(controller)

let { people } = require('../data')

const getPeople = (req, res) => {
    res.status(200).json({ success: true, data: people })
}

const createPerson = (req, res) => {
    const { name } = req.body
    if (!name) {
        return res.status(400).json({ success: 'false', msg: 'please provide name value' })
    }
    res.status(201).json({ message: "success", person: req.body.name })
}

const getPerson = (req, res) => {
    const personId = +req.params.id
    const person = people.find(person => person.id === personId)
    if (!person) {
        return res.status(404).json({ success: false, msg: `No person found with the id ${personId}` })
    }
    return res.status(200).json({ success: true, data: { ...person } })
}

const createPersonPostman = (req, res) => {
    const { name } = req.body
    if (!name) {
        return res.status(400).json({ success: 'false', msg: 'please provide name value' })
    }
    res.status(201).json({ success: true, data: [...people, { name: req.body.name }] })
}

const updatePerson = (req, res) => {
    // we should get two data. 
    // One is the id we need to update
    // Two is we need the new data to be updated to
    const { id } = req.params // this will be string
    const { name } = req.body

    // first see if that ID exists. 
    // converting str to the number as per people array. +id is same as Number(id)
    const person = people.find(person => person.id === +id)
    if (!person) {
        return res.status(404).json({ success: false, msg: `No person with id ${id}` })
    }

    const newPeople = people.map(person => {
        if (person.id === +id) {
            person.name = name
        }
        return person
    })
    res.status(200).json({ success: true, data: newPeople })
}

const deletePerson = (req, res) => {
    const { id } = req.params
    const person = people.find(person => person.id === +id)
    if (!person) {
        return res.status(404).json({ success: false, msg: `No person found with id ${id}` })
    }
    const newPeople = people.filter(person => person.id !== +id)
    return res.status(200).json({ success: true, data: newPeople })
}

module.exports = {
    getPeople,
    getPerson,
    createPersonPostman,
    createPerson,
    updatePerson,
    deletePerson
}

Setting up routes : 1st way

first way of setting up routes

Setting up routes : 2nd way (choose 1st or 2nd way as per your convenience)

second way of setting up routes

Mongo DB

It's a NoSql DB. Atlas is a cloud platform that enables you to host your DB

Get Started with Atlasmongodb

Mongoose

It's a library used on top of MongoDB. This takes up all the heavy lifting and makes the db part easy.

Basic MongoDB and Mongoose setup

To test this we just require this file in app.js

connect.js

const mongoose = require('mongoose')

const connectionString =
    'mongodb+srv://sandeep:[e-mail]/03-TASK-MANAGER?retryWrites=true&w=majority'

mongoose
    .connect(connectionString, {
        useNewUrlParser: true,
        useCreateIndex: true,
        useFindAndModify: false,
        useUnifiedTopology: true
    })
    .then(() => console.log('Connected to the DB...'))
    .catch((err) => console.log(err))

Now if you observe the console, then we see, 1st the server is connected and then the DB. Think about it. What's the point in connecting to the server 1st without a DB? All the data must be first retrieved. Meaning, first the DB must be connected and then the server

So we should not invoke mongoose.connect as shown in the connect.js. We can set it up as a function and then invoke it in the app.js.

Refactoring the connect.js and app.js where first the DB should be invoked before the server

connect.js

app.js

Code for app and connect

app.js

const express = require('express')
const app = express()
const tasks = require('./routes/tasks')
const connectDB = require('./db/connect')

// Middleware
app.use(express.json())

// Routes
app.get('/hello', (req, res) => {
    res.send('Task Manager App')
})

app.use('/api/v1/tasks', tasks)

const port = 3000

const start = async () => {
    try {
        await connectDB()
        app.listen(port, console.log(`Server is listeneing on port ${port}...`))
    } catch (error) {
        console.log(error)
    }
}

start()

connect.js

const mongoose = require('mongoose')

const connectionString =
    'mongodb+srv://sandeep:[e-mail]/03-TASK-MANAGER?retryWrites=true&w=majority'

const connectDB = (url) => {
    console.log("Connecting to the DB...")
    return mongoose
        .connect(connectionString, {
            useNewUrlParser: true,
            useCreateIndex: true,
            useFindAndModify: false,
            useUnifiedTopology: true
        })
}

module.exports = connectDB

Route not found

Unknown route

Env

Don't you think we should not push the database credentials to the git. Yes, we should not expose it. So we need to put it in env and ignore it in git file. We can install dotenv package in our application and then we can access the secret variables in our application.

npm install dotenv

using dotenv

env variables

.gitignore

Modal

Modal is the representation for the collection.

Think of documents as rows in RDBMS and collection as table

Below, it shows the documents at right side for the Products collection.

mongo db screenshot that shows we can have different structure for each document which is not good practice

This is where we use mongoose schema and setup the structure for all the documents that we will have in our collection.

Schema types can be found here

Mongoose v6.1.8: SchemaTypes

mongoose Schema types

Based on this let's setup simple schema type for a Task collection

Task collection's schema

Now we can go to the controller and start using the model.

Before you proceed, read this a bit

Mongoose v6.1.8: Models

3 approaches for inserting the document into a collection

Create operation in DB

Step 1 : Defining a schema for a collection (model) also think it as a table

defining schema

different types in defining schema

Only the properties we set in the schema will be passed on to the database. Everything else will be ignored.

only the properties we set in the schema will be passed on to the database

Step 2 : Create a new document by receiving the req.body and add it to the collection

I showed you three approaches above to create a document. Let's pick the second approach and create a document. We get the incoming request (req.body) and then before sending it as a response, we add it to the DB using approach 2 in the controller.

create operation

Validation

We can send the empty object or empty fields to the DB without validation. Let's setup the validation to schema so that the values exist before sending to the DB. Validation is pretty big and we cover them as and when we progress.

Basic validation

More on validation can be found in Mongoose docs below

Mongoose v6.1.8: Validation

Error handling

At this point, if the validation error occurs, then we are not sending any response and the postman keeps requesting and doesn't get any response back.

not sending the response when error occurs

In other words, we are not handling the error gracefully. The reason for that is we have async operation like below

unhandled

Let's handle this using try-catch block. Later will find more ways to improve this and see how we can avoid this boiler plate of try-catch, but for now let's use this

handled

Async Wrappers

The below explained stuff is a bit complicated, so I'm writing this line in future where I thoroughly understood the concept and wrote a blog on this. Please refer this first and then proceed

https://sandeepamarnath.medium.com/wrapper-in-javascript-to-avoid-writing-try-catch-multiple-times-b3640dce34e4

Since we are using async functions, we wrap them with try-catch block. Doing that in each controller will become redundant as we are repeating same code.

Creating our own async wrapper

This is bit tricky to understand first. Imagine as I have explained below.

In this project we are creating the async wrapper on our own. However, in the upcoming projects we will use npm package for this and I'll explain that then

res has status and json already before asyncWrapper is called

fn is nothing but the whole block marked on left

result

Now we can replace all the controllers by this asyncWrapper that will result in below code

const Task = require('../models/Task')
const asyncWrapper = require('../middleware/asyncWraper')

const getAllTasks = asyncWrapper(async (req, res) => {
    const tasks = await Task.find({})
    res.status(200).json({ tasks })
})

const createTask = asyncWrapper(async (req, res) => {
    const task = await Task.create(req.body)
    res.status(201).json({ task })
})

const getTask = asyncWrapper(async (req, res) => {
    // res.status(200).json(req.params)
    console.log("Get single task")
    const { id: taskId } = req.params
    const task = await Task.findById(taskId)
    if (!task) {
        return res.status(404).json({ msg: `No task with id: ${taskId}` })
    }
    res.status(200).json({ task })
})

const deleteTask = asyncWrapper(async (req, res) => {
    const { id: taskId } = req.params
    const task = await Task.findOneAndDelete({ _id: taskId });
    if (!task) {
        return res.status(404).json({ msg: `No task with id: ${taskId}` })
    }
    const tasks = await Task.find({})
    res.status(200).json({ tasks })
})

const updateTask = asyncWrapper(async (req, res) => {
    const { id: taskId } = req.params
    const task = await Task.findByIdAndUpdate({ _id: taskId }, req.body, { new: true, runValidators: true })
    if (!task) {
        return res.status(404).json({ msg: `No task with id: ${taskId}` })
    }
    res.status(200).json({ task })
})
const editTask = asyncWrapper(async (req, res) => {
    const { id: taskId } = req.params
    const task = await Task.findByIdAndUpdate({ _id: taskId }, req.body, { new: true, runValidators: true })
    if (!task) {
        return res.status(404).json({ msg: `No task with id: ${taskId}` })
    }
    res.status(200).json({ task })
})

module.exports = { getAllTasks, createTask, getTask, updateTask, deleteTask, editTask }

But at the moment we are not handling error anymore yet as we are giving the error to next middleware like this

Let's handle this error which has been passed to the next middleware.

Error handling with Express middleware passed into next

Now I'll ask you a question. In the above image, I said we get an error in post man that is not handled yet as we are not handling the error passed into next. Agree? Then where is the error html in the post man coming from? We didn't write that code.

Well, express does the default error handling for us. Look at the docs as shown below

type error and search for Error Handling and scroll below

Default error handling express docs

Express error handling

Writing our own error handler

our own error handler

Express error handling

The status must be 500 but I didn't define it so it says 200. We will define it below.

Let's write the error handler in in a separate file

Custom error handling class for 404

For handling these in an elegant way

these should be handled by our Error Handling class

Simple implementation of Error class

error-handler.js will handle the next(error)

Lets create a new folder and call it errors.

Creating custom error class

using the above Custom Error class

error-handler.js where we handle 404 error and also other error

500 message

404 message

Error handling and Validation are the two topics we will return to later in some other project. This is a general setup to give an overall idea.

We get a npm package for this async wrapper and we don't need to write asyncWrappers ourselves. https://www.npmjs.com/package/express-async-errors

😎 PROJECTS 🚀🚀🚀🚀

Now that we have gained enough knowledge, let's build some cool projects.

I will make notes only where I feel it is important. Please navigate through Udemy here for projects if you want to follow along.

https://www.udemy.com/course/nodejs-tutorial-and-projects-course/learn/lecture/27016786#questionswww.udemy.com

Convention

Our server might serve different things like /index and then /api and so on. So we could use this. /api/v1 means this servers for /api requests of version 1 so that in future you can add more versions keeping the old ones active.

convention

✅ 1. Task Manager App

This App involves

• Setting up the controllers for GET, CREATE, PATCH, DELETE operations

• Setting up middle wares for error handling, and async wrapper to avoid writing try-catch in each controller

• Setting up postman for creating the routes easily

• Setting up mongo db and mongoose

• Using a simple front-end to perform the actions

• Final code - https://github.com/john-smilga/node-express-course/tree/main/03-task-manager/final
• This app is not deployed as we still didn't cover authentication and security

Deployment

PORT

In local we use 5000 or 3000 or whatever we feel is good, but in the platform which we deploy the same port may not be available, so we need to let the platform choose the port number. Let's set that up using process.env.PORT variable.

port

✅ 2. JWT

There are two parts to this project. The below fig shows how we use JWT to access /dashboard route. Since the dashboard route is protected, we need to verify the token that we gave to client (jwt sign) during login/register. If the client gives us back the same token to access /dashboard route then we need to provide the proper data from /dashboard route.

verify token functionality embedded into /dashboard route

The token verification functionality is embedded into /dashboard above. But what if we need to protect some other route. We need to provide this token verification functionality to that route as well so we might have to repeat the same token verification code. Instead of repeating we can put the token verification code into its own middleware and use the next functionality. Watch the video for further details.

✅ 3. Jobs API (Project Steps)

Let's classify our projects' steps broadly

Step 1 (Basic setup)

• Create app.js file. Import express and start the server

• Define basic middle wares like express.json (to get access to the req.body) and express.static (if app has in built html css and JS)

• Define and test basic route. app.get('/', req res CallBack) and see if that works by calling this in URL

const express = require('express')
const app = express()

// middlewares

app.use(express.json()) // to get req.body
app.use(express.static('public')) // to serve static files

// routes

app.get('/', (req, res) => {
  res.send('Home Page')
})

const port = process.env.PORT || 3000

app.listen(3000, () => {
  console.log('Listening on Port 3000')
})

Routes and Controllers

• We need auth route and controllers for register and login

• We need jobs route and controllers for create, read, update and delete operations. Boiler plate code with a console log statement in each controller should be enough at this point

app.js

const express = require('express')
const app = express()
const authRouter = require('./routes/auth')

// middlewares

app.use(express.json()) // to get req.body
app.use(express.static('public')) // to serve static files

// routes

app.get('/', (req, res) => {
  res.send('Home Page')
})

app.use('/auth', authRouter)

const port = process.env.PORT || 3000

app.listen(port, () => {
  console.log('Listening on Port 3000')
})

routes/auth.js

const express = require('express')
const router = express.Router()
const { register, login, logout } = require('../controllers/authController')

router.route('/register').get(register)
router.route('/login').get(login)
router.route('/logout').get(logout)

module.exports = router

controllers/authController.js

const register = async (req, res) => {
  res.send('Register controller')
}

const login = async (req, res) => {
  res.send('Login controller')
}

const logout = async (req, res) => {
  res.send('Logout controller')
}

module.exports = { register, login, logout }

Step 2 (Error Handlers)

• Setup error handler middle wares

• One for notFound and the other for all ErrorHandler which covers other errors like BadRequest and UnAuthenticated

• One of the ways done in 06JobsAPI is

  • Created error folder and then create CustomErrorAPI which is the main class that extends Error class

CustomErrorAPI

error/CustomAPIError.js

class CustomAPIError extends Error {
  constructor(message) {
    super(message)
  }
}

module.exports = CustomAPIError

• Then I'll use this class (extend this class) to create NotFoundError class, BadRequestError class and UnAuthenticatedError class (and any other class if u want)

BadRequestError

NotFoundError

We can import all these errors in the index file so that to throw these errors in the controllers, we can import from this index file

Index Page for errors folder

Error middlewares

Now we have all these errors implemented, let's define error middlewares that we can use in app js

We will have notFound Middleware and ErrorHandler middleware. Not found one is used when a route doesn't exist. The error handler one is used when something else goes wrong. This something else can be Authentication error or BadRequest error and so on which are defined above. (We have already implemented them above)

Error handler middleware

notFound middleware

Use these two mw in app js

• Not found mw

• Error Handler mw

  • This includes BadRequest Error

  • Authentication Error

  • Not Found Error (This error occurs when we throw NotFoundError in any controller). If the end point which user entered doesn't exist then automatically it goes to Not Found mw above in first step. If it is any other kind of error except notFound one (non-existent route) then errorHandlerMW will handle that error.

app js

Step 3 (Register)

• We need to define a post route for register

• Then define a controller for register

• Then create a Model for user and define UserSchema. Also, define UserSchema validations. If u will be sending name, email and password then define the validations for these fields as shown below.

const UserSchema = new mongoose.Schema({
  name: {
    type: String,
    required: [true, 'Please provide name'],
    maxlength: 50,
    minlength: 3,
  },
  email: {
    type: String,
    required: [true, 'Please provide email'],
    match: [
      /^(([^<>()[\]\\.,;:\s@"]+(\.[^<>()[\]\\.,;:\s@"]+)*)|(".+"))@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\])|(([a-zA-Z\-0-9]+\.)+[a-zA-Z]{2,}))$/,
      'Please provide a valid email',
    ],
    unique: true,
  },
  password: {
    type: String,
    required: [true, 'Please provide password'],
    minlength: 6,
  },
})

module.exports = mongoose.model('User', UserSchema)

• In the register controller write code to put the incoming fields into to db

const user = await User.create({ ...req.body })

• This creates the user in the db but if u think about it then we are saving the passwords in plain text. So before we do that we can hash the password using bcryptjs package. We can do this here in the controller itself but the mongoose provides a pre hook which executes before saving the data into db. So you can use this instead of cluttering your controller like this

// in the User model

// this executes after User.create but before going into DB. So we can hash the password here
UserSchema.pre('save', async function () {
  const salt = await bcrypt.genSalt(10)
  this.password = await bcrypt.hash(this.password, salt) // this refers to the document
})

// we can access like this
// this.username, this.password and so on

• Once the user gets created in the db with his password hashed, we then need to create and send back the token. We can do this also in the controller but then mongoose gives us a nice little helper methods functionality where, similar to pre hook, we can create a method on mongoose and then use it in the controller like so

UserSchema.methods.generateToken = function () {
  return jwt.sign(
    { userId: this._id, name: this.name },
    process.env.JWT_SECRET,
    {
      expiresIn: process.env.JWT_LIFETIME,
    }
  )
}

Now the auth controller's register method looks like this

const register = async (req, res) => {
  const user = await User.create({ ...req.body })
  const token = user.createJWT()
  res.status(StatusCodes.CREATED).json({ user: { name: user.name }, token })
}

Step 4 (Login)

• We need to get username (email in this case) and password

• See if this email exists in the db

• If yes, then compare that password string with the hashed password in the db

• If they match, that means the user is legit and he is the one in the db so we can send him back the token like how we sent back in the reg call

Auth Controller for login

UserSchema - User model

Step 5 (Verify token - Auth middleware)

Once we have created the token and sent in register and login route, now it's time to think why we need this token. We have different end points for jobs like GetJob, GetAllJobs, CreateJob, UpdateJob and DeleteJob. All these end points of job are secure. Meaning, the user should be logged in to access these functionality. How do we differentiate a login and logged out user? Well with the token. The logged in user will have the token we sent him back when he registered or logged in. So if the user sends us back the token to access any of these end points then that means that user is logged in.

So what's the next step? We get the token and we need to verify that using jsonwebtoken package (same package that helped us create the token). If they match then we can allow the user to access any of the job routes.

We can create this token verification functionality in all the job related endpoints mentioned above. But it's better to write it once. So let's write in in the Auth middleware

• Write Auth mw where we implement the token checking functionality

• We can stick this method/ function to all the routes we need to be secure

auth mw to verify the token

Securing getAllJobs end point as an example here. Without the token this endpoint can't be accessed.

attach auth mw to make getAlljobs endpoint secure

As you see we are adding authMiddleware for getAllJobs route here. We can add it to all routes here but instead of doing that we can also add it to jobs route in the app.js file as shown below

app.use('/api/v1/jobs', authMiddleware, jobsRouter) 

After passing the auth mw, it will reach here in jobs controller

Step 6 (Jobs - Schema setup)

Once the token is verified then we can move to jobs part. First we need to define Job Schema with validations followed by Job model. We can say a job object or document would have these fields

• company

• position

• status

• createdBy (User) - So we need to link the User object ID (user ID) to this - Reference : https://mongoosejs.com/docs/populate.html
• Also, we can define timestamps so we know when was each doc created and updated. We can then use them to sort

model/Job.js

const mongoose = require('mongoose')
const JobSchema = new mongoose.Schema(
  {
    company: {
      type: String,
      required: [true, 'Please provide company name'],
      maxLength: 20,
    },
    position: {
      type: String,
      required: [true, 'Please provide the position'],
      maxLength: 120,
    },
    status: {
      type: String,
      enum: ['interview', 'declined', 'pending'],
      default: 'pending',
    },
    // Reference - https://mongoosejs.com/docs/populate.html
    createdBy: {
      type: mongoose.Types.ObjectId,
      ref: 'User',
      required: [true, 'Please provide a user'],
    },
  },
  { timestamps: true }
)

module.exports = mongoose.model('Jobs', JobSchema)

Step 7 (Create and Get Job)

Now we have our db in place for Job, let's first create a job. We need to setup functionality in controllers

controller/job

const createJob = async (req, res) => {
  req.body.createdBy = req.user.userId
  const job = await Job.create(req.body)
  res.status(StatusCodes.CREATED).json({ job })
}

const getAllJobs = async (req, res) => {
  // we need to get job related to the user who is logged in
  const jobs = await Job.find({ createdBy: req.user.userId }).sort('createdAt')
  res.status(StatusCodes.OK).json({ jobs, count: jobs.length })
}

Step 8 (Postman - Dynamically set logged in user's token in authorization header)

Currently, to create a job for a user,

• We login or register a particular user

• Get the token and copy that token

• Manually paste it in the Authorization Header by writing Bearer <token>

• Then create a job

• Go to the GetAllJobs route, paste the token here as well in the Authorization header

• GetAllJobs for that user

This copy paste thing gets annoying quite fast. So it would be better if we could automatically set the Authorization header to the user who is currently logged in. Let's do that.

Automate token setup in Tests in postman

Navigate to Tests in postman. When we click login we get back the token correct? So we need to write a few lines of code here to use that token

Tests in postman

Write Tests to store token

Don't forget to write the tests for Register endpoint as well

Getting the token set while log in or register

Now to test this, create two users (register) UserA and UserB. Login as UserA, create 2 jobs. Login as UserB and create 5 jobs. Now, when you login as UserA and get all jobs you should see only 2 jobs and as userB you should see 5 jobs.

Step 9 (Get a single job)

Now it's time to get a job. we pass our request from postman as {{URL}}/jobs/<JobID> and the JobID is accepted as id in the GetJob router as shown below.

req.params

For getting a single job for that particular user we need to test whether the logged in user is requesting that particular job and also if the id passed exists in the db. Once these two conditions satisfy then we get the job from db and we can pass this response.

const getJob = async (req, res) => {
  const {
    user: { userId },
    params: { id: jobId },
  } = req

  const job = await Job.findOne({ _id: jobId, createdBy: userId }) // matching both jobID (if exists) and logged in user
  if (!job) throw new NotFoundError(`No job with id ${jobId}`)
  res.status(StatusCodes.OK).json({ job })
}

Note : In the request param, if we pass an ID that matches it's format but the ID doesn't exist then we get job as null. In this case we throw NotFound error meaning - The ID format u passed is right but the job doesn't exist.

Now if you pass the wrong ID format (for example, if u remove last character of ID, then the acceptable format is wrong and it throws some other error. We will handle this later)

Step 10 (Update a job)

Similar to Get Single Job, we need to pass jobId and userId in the request so that mongoose can first find that particular job related to the logged in user.

We also need to pass company and position in the request to which the new values get updated.

update job

Step 11 (Delete a job)

Let's now take a look at how to delete a job. Similar to what we did in update we do the same in delete. findByIdAndRemove or findOneAndRemove both does the trick of deleting a job.

We need to find if the job with an ID and user with that ID (createdBy) exists, then we can delete and it gives back the deleted job. If no job comes back then it means that it didn't delete it. In that case we send 404 (job not found), else we can just send 200 and leave the response empty.

Delete a job

Step 12 (Making Errors more friendly)

Currently in our setup we have 3 errors

• Validation error - We throw BadRequest error when some field is missing

• Duplicate error (for email) - Mongoose validation throws an error because of unique set to true

• Cast Error - If syntax of our request is not good. For example while sending ID in query params if we mess up the number of characters of an ID (if we decrease or increase one letter in ID) then this error is thrown

The goal here is to handle all these better. At the moment, for a duplicate email, this is what our response looks like

mongoose error

For this let's work in error-handler middleware. Currently it looks like this

06-jobs-api/starter/middleware/error-handler.js

Current error-middleware

Notice that whatever we throw like BadRequestError or NotFoundError, that will be caught in line number 6. Rest all the errors like thrown by Mongoose or any internal code errors will be caught at line 9. The error occurring at line 9 is all of the errors which we are not throwing and it's not very informative as to what exactly went wrong.

For example, let's say there's a duplicate error thrown by mongoose for unique = true field like email. Since we are not throwing and mongoose is throwing that, we catch in line 9 and throw as 'Something went wrong'. Now the user doesn't know that it already exists.

The work around here is to get the mongoose error, and show that to user in a useful way. The bottomline is, we need to cater for all the other errors which we are not throwing like Validation Error, Duplicate Error and Cast Error thrown by mongoose. Let's change the code like this below

Duplicate Error

customError message and statusCode set

Also, you can remove line 11 to 13 in the above code as we don't have to check if err is instance of CustomAPIError. We are concerned only about the message and status code that err param gives in line 3 and then we are setting them anyways in line 7 and 8 as err.statusCode and err.message

Validation Error

When does a validation error occur?

Validation error occurs at MODEL level. Meaning, while creating a model we define what fields must be defined and how they should be. If we don't adhere to those rules while creating a document (When we create a document then it checks Model and see if all the validations pass for this DOCUMENT as defined in the MODEL) then the validation error occurs.

validation rules defined in User Schema (User model)

error while creating the document

We can make use of name prop ValidationError, and send some useful message and statusCode. Let's check if this error is thrown and then if yes, we need to iterate through the fields within err.errors and then pick message prop for each and display them like this

validation error handling

and we get the below response

response for validation error

Cast Error

When we tamper the format of the parameter we are sending in the request then this error occurs. For example, ID format when requesting for a single job.

Let's say we find all jobs and I want the details about single job so I send get request to GetSingleJob route with ID 62eb2fdae7df6711285626f2 this is what I get

ID format correct

Now let's say I mess up the ID format and send a string which is not of proper length, then I get cast error. Notice I have removed 2 at the end in the previous request like this 62eb2fdae7df6711285626f

ID format wrong so we get cast error

cast error code

improved cast error

Step 13 (Security packages)

So far our apps were small and cute where we just worked in local, the security wasn't a concern. We are now going to host this app on cloud (Heroku) so we need to worry about some bad hackers trying to hack our API. Meaning, a hacker can try to access our secured routes (jobs in this case) without proper authentication and do more damage than we can think of like getting our logged in user data / registered users email and so on.

Luckily there are many npm packages to our rescue. The packages we are going to use are:

• helmet

  • Sets various HTTP headers to prevent numerous possible attacks. This is very popular

• cors - Cross Origin Resource Sharing

  • Ensures that our AP is accessible from different domains

  • If you don't have cors installed, your app (server) will only be accessible by same domain like we did in our other apps where we used public folder to write our front-end within the same app in javascript file where we send request to our own end-points from our own apps' public folder

  • CORS is a mechanism to allow or restrict requested resources on a web-server depending on where the HTTP request was initiated

  • By installing and implementing the CORS package essentially we make our API accessible to the public

CORS explained

• xss-clean

  • Sanitizes the user input in req.body, req.query and req.params as a result it protects us from cross-site-scripting attacks where the attacker tries to inject some malicious code

• express-rate-limit

  • We (server) can limit the amount of request a client can make

The app.js looks like this below after installing security packages.

app.js

require('dotenv').config()
require('express-async-errors')

const express = require('express')
const app = express()

// extra security-packages
const helmet = require('helmet')
const cors = require('cors')
const xss = require('xss-clean')
const rateLimiter = require('express-rate-limit')

// CONNECT DB
const connectDB = require('./db/connect')

// ROUTER
const authRouter = require('./routes/auth')
const jobsRouter = require('./routes/jobs')

// error handler
const notFoundMiddleware = require('./middleware/not-found')
const errorHandlerMiddleware = require('./middleware/error-handler')

// token verification to access secured route
const authMiddleware = require('./middleware/authentication')

app.use(express.json())

// for heroku we need to do this
app.set('trust proxy', 1)
app.use(
  rateLimiter({
    windowMs: 15 * 60 * 1000, // 15 minutes
    max: 100, // max 100 requests in 15 mins
  })
)
app.use(helmet())
app.use(cors())
app.use(xss())

// routes
app.use('/api/v1/auth', authRouter)
app.use('/api/v1/jobs', authMiddleware, jobsRouter)
app.get('/api/v1', (req, res) => {
  const x = 5
  console.log('The value of x is', xy)
  res.send('home')
})

app.use(notFoundMiddleware)
app.use(errorHandlerMiddleware)

const port = process.env.PORT || 3000

const start = async () => {
  try {
    await connectDB(process.env.MONGO_URI)
    app.listen(port, () =>
      console.log(`Server is listening on port ${port}...`)
    )
  } catch (error) {
    console.log(error)
  }
}
start()

Step 14 (Deploy - Heroku or any other cloud provider)

Great! Congrats. You've come a long way. It's now time to deploy our server. Let's use heroku for this. It's free (it has a free tier so don't worry)

Let's deploy 🚀🚀🚀🚀🚀

• Make a copy of jobs-api/starter folder and put it on to desktop. I've called to project copy as 06-jobs-api-deployable

• Open VSCODE with this project.

• Go to https://dashboard.heroku.com/apps and login with [e-mail] / Heroku@123
• Go to documentation and Node JS and Deploying Node Js Apps on Heroku https://devcenter.heroku.com/articles/deploying-nodejs
• Go back to vscode and remove git folder if there's any already by typing rm -rf .git

• Make sure you have process.env.PORT setup in app.js - const port = process.env.PORT || 3000

• Setup a dummy route so that we know our app is deployed once it's done by accessing that route

// dummy route
app.get('/', (req, res) => {
  res.send('Jobs API')
})

• Now follow the steps in the documentation https://devcenter.heroku.com/articles/deploying-nodejs
• In package.json setup node version in engines object

"engines": {
    "node": "14.x"
  },

• In package.json in start script, change nodemon to node as we are now in prod and not development anymore

  "scripts": {
    "start": "node app.js" // change nodemon to node
  },

• Then specify a PROC file - Heroku first looks at proc file - https://devcenter.heroku.com/articles/deploying-nodejs#specifying-a-start-script

  • Create a file and name it Procfile (no extension needed)

  • Add this line

web: node app.js

• Follow John's video 202. Deploy on Heroku for more info

• Create routes in postman for login and get all jobs using deployed url. Once they work fine, you're good to go

Step 15 (Setup the documentation - Swagger UI)

Doing directly from swagger takes a lot more effort. Instead, we can do it from postman.

• Go to postman and make sure all the routes have same URL

• Click on these 3 dots and export (u can change the file name but should be .json)

Postman export

• Now we cannot directly pass exported postman docs into swagger UI. The data from postman needs to be parsed in a way that the swagger understands. For this we use APIMATIC. We need to sign up for this https://www.apimatic.io/dashboard with mrs [e-mail] / Apimatic@123
• In APIMATIC, click on import json file from postman export. Don't pay attention to warnings you get while importing

• Then click Edit and edit how you need. Watch video 207. APIMATIC Setup

Step 16 (Add swagger to our app)

Once the swagger UI is working, then you need to have two packages in your package.json

• swagger-ui-express - provides swagger to our app

• yamljs - converts our postman json to something swagger can understand

Once you have installed these 2 packages, create a file having the extension .yaml. In my case I'll create swagger.yaml and then paste the yaml file you did in step 15 (yaml that you got from APIMATIC Export) and paste it in that file swagger.yaml

Back in our app.js we need to do the following

• require swagger-ui-express, yamljs packages

• Load the yaml file and then pass it on to swagger UI

• Set up a route to show the documentation like this

app.js (partial)

// swagger
const swaggerUI = require('swagger-ui-express')
const YAML = require('yamljs')
const swaggerDocument = YAML.load('./swagger.yaml')

// Home route
app.get('/', (req, res) => {
  res.send('<h1>Jobs API</h1><a href ="/api-docs">API Documentation</a>') // uses below route
})

// Document route
app.use('/api-docs', swaggerUI.serve, swaggerUI.setup(swaggerDocument))

• Run these commands to update PROD (Heroku)

git add -A
git commit -m "swagger docs added" 
git push heroku master

Step 17 (PAT YOUR BACK)

Great! Congrats! You've completed the project. So whenever in future you want to come back to node-express learning and do some projects, I suggest you go through this entire page notes once (optional), take a look at previous project notes JWT project and this project (Jobs). You should be good.

✅ 4. File upload project

In this project, let's upload the files to our server or cloud called cloudinary. We also use express-file-upload library here.

Upload images project

Step 1 - Product Model

Let's define 3 fields in Product Model, name, price and then product image.

model/Product.js

const mongoose = require('mongoose')
const ProductSchema = new mongoose.Schema({
  name: {
    type: String,
    required: true,
  },
  price: {
    type: Number,
    required: true,
  },
  image: {
    type: String,
    required: true,
  },
})

module.exports = mongoose.model('Product', ProductSchema)

Step 2 - Product and Upload Controller

Let's setup two controllers. ProductController and UploadController.

Controller/productController.js.js

const Product = require('../models/Product')
const { StatusCodes } = require('http-status-codes')

const createProduct = async (req, res) => {
  const product = await Product.create(req.body)
  res.status(StatusCodes.OK).json({ product })
}

const getAllProducts = async (req, res) => {
  const products = await Product.find({})
  res.status(StatusCodes.OK).json({ products })
} 

module.exports = {
  createProduct,
  getAllProducts,
}

Controller/uploadController.js

const { StatusCodes } = require('http-status-codes')

const uploadProductImage = async (req, res) => {
  res.send('create product')
}

module.exports = {
  uploadProductImage,
}

Now why do we need uploadController.js file and uploadProductImage controller? Before we answer this question, lets first setup ProductsRoute

Step 3 - Product Route

route/productRoute.js

const express = require('express')
const router = express.Router()
const {
  getAllProducts,
  createProduct,
} = require('../controllers/productController')
const { uploadProductImage } = require('../controllers/uploadsController')

router.route('/').get(getAllProducts).post(createProduct)
router.route('/uploads').post(uploadProductImage)

module.exports = router

Step 4 - App.js setup

app.js

require('dotenv').config()
require('express-async-errors')

const express = require('express')
const app = express()

app.use(express.json()) // to have access to req.body

// database
const connectDB = require('./db/connect')

// product router
const productRouter = require('./routes/productRoutes')

// error handler
const notFoundMiddleware = require('./middleware/not-found')
const errorHandlerMiddleware = require('./middleware/error-handler')

app.get('/', (req, res) => {
  res.send('<h1>File Upload Starter</h1>')
})

app.use('/api/v1/products', productRouter)

// middleware
app.use(notFoundMiddleware)
app.use(errorHandlerMiddleware)

const port = process.env.PORT || 3000

const start = async () => {
  try {
    await connectDB(process.env.MONGO_URI)

    app.listen(port, () =>
      console.log(`Server is listening on port ${port}...`)
    )
  } catch (error) {
    console.log(error)
  }
}

start()

Step 5 - Postman setup

Once the basic setup is done let's now setup the routes in postman. We will have 3 routes

• For Image upload (POST)

• For creation of product (POST)

• To get all products (GET)

Step 6 - Why we need uploadImage controller?

Ok it's time to answer this question. What do we send image path string in the Product? We don't know the path of the image we will be uploading correct?

The idea is, we need to put an image to the server OR cloudinary (store image on cloud) and then get the path of that image and add that path as Product image string.

So for uploading it to the cloud or server first we need this uploadImage controller/route.

Steps

• Upload an image to the server or cloud (cloudinary) and get the path for this image

• Use this path to create product

Step 7 - Upload image

Now let's focus on /products/uploads route to upload the image and get the path before creating the product.

Let's upload an image / file from postman

File Upload

Now if I upload image like shown above and click on Send button in Postman, in our controller if we console log req, we see nothing in body. So how can we get image that we uploaded then? We need to use additional package called express-fileupload to grab that image data and parse it.

express-fileupload package

req.files

Now that we have access to the file that got uploaded, we need to do two things

• We need to move this file to our server (any folder) Or cloudinary

• Also, we need this image to be publicly available (we need to store in a folder that is publicly available - example public folder)

make a folder publicly available

• Also, on the response, not that we have mv function (last key in image) that helps us to move the image to any other folder

Upload image to publicly accessible folder

publicly accessible

path of image on public folder

Images or files publicly accessible

So now, how we create products?

• Upload image in Upload Image path

• Once the image is uploaded, you will get the path

• Copy that path and create the product

Steps in uploading image and getting path

Steps in creating the product

Step 8 - Test on Front-end

front-end explanation

So when you click Choose File, the /products/uploads route is called giving us back the path of the image that we can use when we Create Product (when we click Add Product)

Choose file - /uploads endpoint is called giving us back the path

Image path in the response

Step 9 - Error Checks

Before we explore cloudinary, let's first check if the uploaded file is an image, it has a proper size and so on.

controller/UploadProductImage.js

const { StatusCodes } = require('http-status-codes')
const path = require('path')
const CustomError = require('../errors')
const uploadProductImage = async (req, res) => {
  if (!req.files) {
    throw new CustomError.BadRequestError('No File uploaded')
  }
  const productImage = req.files.image
  if (!productImage.mimetype.startsWith('image')) {
    throw new CustomError.BadRequestError('Please upload an image')
  }
  const maxSize = 1024 * 1024 // 1000 Mb
  if (productImage.size > maxSize) {
    throw new CustomError.BadRequestError(
      'Please upload an image smaller than 1Mb'
    )
  }

  const imagePath = path.join(
    __dirname,
    '../public/uploads/' + `${req.files.image.name}`
  )
  await productImage.mv(imagePath)
  res
    .status(StatusCodes.OK)
    .json({ image: { src: `/uploads/${productImage.name}` } })
}

module.exports = {
  uploadProductImage,
}

Step 10 - Cloudinary Setup

Instead of storing the images on the server like we did till now, we can use a popular cloud option like cloudinary https://cloudinary.com/ to store the images on cloud. The benefit is that the cloud can be located in many geographical locations and will be very faster to get the image as a client.

Details

[e-mail] / Cloudinary@123

  cloud_name: 'staar', 
  api_key: '834572319122985', 
  api_secret: 'lkTEVctI21BirqEbFr_YeX2BuNg' 

Setup cloudinary in app.js

npm install cloudinary

app.js

require('dotenv').config()
require('express-async-errors')

const express = require('express')
const app = express()
const fileUpload = require('express-fileupload') // to access files that are uploaded
const cloudinary = require('cloudinary').v2 // store images on cloud

cloudinary.config({
  cloud_name: process.env.CLOUD_NAME,
  api_key: process.env.CLOUD_API_KEY,
  api_secret: process.env.CLOUD_API_SECRET,
  secure: true,
})

app.use(express.static('./public')) // make public folder publicly available
app.use(express.json()) // to have access to req.body- This don't give files that are uploaded

app.use(fileUpload())

// database
const connectDB = require('./db/connect')

// product router
const productRouter = require('./routes/productRoutes')

// error handler
const notFoundMiddleware = require('./middleware/not-found')
const errorHandlerMiddleware = require('./middleware/error-handler')

app.get('/', (req, res) => {
  res.send('<h1>File Upload Starter</h1>')
})

app.use('/api/v1/products', productRouter)

// middleware
app.use(notFoundMiddleware)
app.use(errorHandlerMiddleware)

const port = process.env.PORT || 5000

const start = async () => {
  try {
    await connectDB(process.env.MONGO_URI)

    app.listen(port, () =>
      console.log(`Server is listening on port ${port}...`)
    )
  } catch (error) {
    console.log(error)
  }
}

start()

How to put images to the cloudinary?

const result = await cloudinary.uploader.upload(IMG_PATH)

Now the image path (param to cloud upload) is /public/uploads folder. We get the image uploaded from /public/uploads and add it to cloud using above line of code. However, we have second option to replace /public/uploads folder. express-fileupload package will also provide us the temp folder for storing the image on server. How to enable that? Well, enable useTempFiles:true in express-fileupload package like this. This would create a tmp folder (left side highlighted)

changing code to introduce temp (tmp) folder

Now tmp folder is enabled

Now the uploaded file will be available right away in tmp folder. We need to give this path to cloudinary.

File Upload to cloudinary

Where can user access image he just uploaded? OR in other words, what do we send back as image URL?

image URL to be sent back

When we send the response with secure_url like this, this is what we see in postman

const uploadProductImage = async (req, res) => {
  console.log(req.files.image)
  const result = await cloudinary.uploader.upload(
    req.files.image.tempFilePath,
    { use_filename: true, folder: '07-File-Upload' }
  )
  res.status(StatusCodes.OK).json({ image: { src: result.secure_url } })
}

cloudinary URL

Step 11 - Clear temp folder after upload

Notice that, after each upload from front-end or postman, we will have temp files pile up on server which was created before each push to cloudinary. We need to cleanup after we put that image to cloudinary

controller/uploadController.js

const { StatusCodes } = require('http-status-codes')
const path = require('path')
const CustomError = require('../errors')
const cloudinary = require('cloudinary').v2
const fs = require('fs')


const uploadProductImage = async (req, res) => {
  console.log(req.files.image)
  const result = await cloudinary.uploader.upload(
    req.files.image.tempFilePath,
    { use_filename: true, folder: '07-File-Upload' }
  )
  // this will delete any files in this path
  fs.unlinkSync(req.files.image.tempFilePath)
  res.status(StatusCodes.OK).json({ image: { src: result.secure_url } })
}

module.exports = {
  uploadProductImage,
}

👏 Well done! Completed the File Upload Project.

✅ 5. Stripe API Project

This is the project where we will work on accepting payments with stripe. This has become a goto platform for everything related to online payments.

Step 1 (General overview of how online payments work)

stripe payment page front-end

In any E-Commerce, Once we add the items to the cart and click on checkout we come to this payment page as shown above. The idea is, when we click pay, we cannot just directly communicate with stripe from the front-end to accept $ amount. That would be highly insecure.

Front-end ---> to stripe = Insecure

Instead, once the Pay button is clicked, we need to communicate with our backend where we send our payment intent.

In John's react course, we build this logic of communicating with backend using Serverless functions (as there's no real backend that we manage there in react course). But here in node course, since we have our own backend we communicate with it and get the payment intent from stripe and then send it to front-end

Frontend ---> Our backend (OR serverless function) ---> Stripe

---> Stripe sends back payment intent to backend ---> We can then proceed with this payment.

So basically the overview is, in order for front-end to communicate with stripe it should have the payment intent from stripe.

Let's create a stripe account - [e-mail] / Rememberstripepassword@123

Stripe creds

Step 2 (Understand Stripe Code - Docs)

API keys for front-end and backend

Create .env file on starter project in backend and paste this Secret Key and restart the server. Once you're done, let's go to docs and see how to accept online payments https://stripe.com/docs/checkout/quickstart

docs

Select the below shown. server.js is the node js code we need in our controller.

node js code

Go to checkout.html file and there is the script tags you can use for your html. That way you can get access to stripe in client side js file.

You can use checkout.js page to write your client side js file. There you have to use publishable key.

client checkout.js

The above is the checkout.js file. Notice that we have items (cart)

const items = [{ id: "xl-tshirt" }];

and we are sending this to create-payment-intent route (to our backend where we add code given by stripe in server.js)

checkout.js partial file

  const response = await fetch("/create-payment-intent", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ items }),
  });

In server.js we have code for that create-payment-intent route where we send back clientSecret to front-end (checkout.js). Once checkout.js gets back the client secret then it can communicate with stripe.

server.js

app.post("/create-payment-intent", async (req, res) => {
  const { items } = req.body;

  // Create a PaymentIntent with the order amount and currency
  const paymentIntent = await stripe.paymentIntents.create({
    amount: calculateOrderAmount(items),
    currency: "cad",
    automatic_payment_methods: {
      enabled: true,
    },
  });

  res.send({
    clientSecret: paymentIntent.client_secret,
  });
});

The key shown in the server.js is our real key. So put that in env file and don't share that with anyone

Step 3 (Using stripe code in our code)

Once you understand what is server.js and checkout.js and checkout.html. Then let's move on to our code to implement these.

3.1 Our front-end in public folder

Go to public/browser-app.js paste your public key from stripe

publishable key in public/browser-app.js

3.2 Stripe Controller

Let's work on our controller now in controllers/StripeController.js

We have to setup a route in app.js first. That route's name should match the route in browser-app.js as we call this route from browse-app.js and the request comes to our app.js (server). Let's call that /server in browser.js

route we need to call from browser-app.js

require('dotenv').config()
require('express-async-errors')
const express = require('express')
const app = express()

// controller
const stripeController = require('./controllers/stripeController')

// error handler
const notFoundMiddleware = require('./middleware/not-found')
const errorHandlerMiddleware = require('./middleware/error-handler')

app.use(express.json())
app.use(express.static('./public'))

// stripe
app.post('/stripe', stripeController)
app.use(notFoundMiddleware)
app.use(errorHandlerMiddleware)

const port = process.env.PORT || 3000

const start = async () => {
  try {
    app.listen(port, () =>
      console.log(`Server is listening on port ${port}...`)
    )
  } catch (error) {
    console.log(error)
  }
}

start()

controller/stripeController.js

const stripe = async (req, res) => {
  console.log(req.body)
  res.send('Stripe post route')
}

module.exports = stripe

Start your server and go to localhost:3000 in your browser, you will then hit /stripe route which invokes stripeController as shown

browser-app.js and stripe controller communicating

Note that the amount is in cents here and not dollars because stripe needs the smallest unit of currency. For example, t-shirt's price is 1999 which means 19.99$. But stripe needs this in cents so 19.99 * 100 will be 1999.

Now in the stripeController (backend) we need to do two things

• Verify if the cost of the items is actually what front-end is saying

• Communicate with stripe and get the client secret because the front-end can make the payment directly to stripe if it has the client-secret as shown below

front-end uses client secret to pay to stripe

Normally in the stripe controller (our backend), we will get our cart (purchase in this case) from front-end and we need to take the ID of each item and check it's price in DB and confirm if the total is matching up with what front-end sent us. This is because front-end can modify the values and we can't rely on that so we always need to check in the backend and see if the prices are matching. If they are matching then we communicate with stripe and create a payment intent and get the client secret.

Once we get the client secret, we can send it to front-end where front-end can make the payment directly to stripe. Without this client-secret in front-end it cannot make a payment to stripe.

controllers/sripeController.js

const stripe = require('stripe')(process.env.STRIPE_API_KEY)

const stripeController = async (req, res) => {
  const { purchase, total_amount, shipping_fee } = req.body

  const calcOrderAmount = () => {
    // IMPORTANT NOTEs
    // normally here, we will communicate with our database and get the price of each item and
    // verify if what front-end is saying the cost of each item is, it's actually true (because front-end can manipulate the items and cost)
    // since this is just a demo project we will not do the verification here by calling db
    // here we will just combine total_amount and shipping_fee that will give me total amount
    return total_amount + shipping_fee
  }

  const paymentIntent = await stripe.paymentIntents.create({
    amount: calcOrderAmount(),
    currency: 'usd',
  })

  res.json({ clientSecret: paymentIntent.client_secret })
}

module.exports = stripeController

Once this is setup then navigate to localhost:3000 in your browser and make a payment 😎🚀

Pay with stripe once you have the client secret from our back-end stripe controller

Good job! Done with stipe project

✅ 6. E-Commerce API 😍

Buckle up your seats for yet another very large project E-commerce API. This involves several features like

• Role based authorization

• Sending JWT via cookie

• Users

• Orders

• Reviews

• Documentation using docgen package

• What is MongoDB aggregator and how to use it

• And much more

The final documentation looks like this https://e-commerce-api-10.herokuapp.com/

Let's write some steps on how to build this app.

Step 1 (Basic Setup)

• Get the starter file from John's repo

• It already has the error handling code (Step2 and Step 12 of previous project, Jobs API)

• In the app js file, define express and then app

• Setup the port

• Import dotenv and require('dotenv').config()

• Setup basic route for '/' and check in the browser if it works

• Setup the notFound and errorHandler middlewares

• Require express.json()

• Setup the DB by defining Mongo_URL in .env file (create this file)

• Define Mongoose connect.js. Note that in version 6, we don't need the optional parameters, it can just be like this below

connect mongodb to our app using mongoose

• Require this in the app js and call it like this

mongoose connect in start

• At this point, the app.js file looks like this

const express = require('express')
const app = express()
const connectDB = require('./db/connect')
require('dotenv').config()

// make sure this express-async-errors is above authRouter import. Else the code in the controllers (inside authRouters) are not covered so it wont work.
require('express-async-errors') // for avoiding writing try-catch in controllers

const notFoundMW = require('./middleware/not-found')
const errorHandlerMW = require('./middleware/error-handler')

//// Middlewares and Routes
app.use(express.json()) // used to get req.body data for post reqs

// Routes

// Basic Route
app.get('/', (req, res) => {
  res.send('E-Commerce API Home page')
})

// app.use('/api/v1/auth',)

app.use(notFoundMW)
app.use(errorHandlerMW)

////

const port = process.env.PORT || 5000

const start = async () => {
  try {
    // connect to db
    await connectDB(process.env.MONGO_URL)
    app.listen(port)
    console.log('Server is listening on port', port)
  } catch (err) {
    console.log(err)
  }
}

start()

Step 2 (Define logging middleware)

Till now, we didn't do logging. We can use Morgan package https://www.npmjs.com/package/morgan to handle this in our app

• Require morgan package

• Define a middleware and use this with option tiny to start with

• We'll explore different options soon

Here's how it looks like in app.js

const morgan = require('morgan')
app.use(morgan('tiny'))

So once you add this package like this then you call any route, for example go to localhost:5000/ which makes a get request to route '/', this is what you see

morgan log

This helps you debug nicely where it will show what route is being hit in console. Let's say you hit a non existing route /apples, then it will show it right away and you know what's going on

morgan log

Step 3 (Auth setup)

In this project, we will be sending JWT in cookies. Also, we will have role based authentication. Only an ADMIN user can CRUD Products. We have USERs who CRUD Orders, and Reviews.

3.1 Create User Model

• Create models folder and User.js file

• Create schema with name,email, password (all type:String)

• Export mongoose model

model/User.js

const mongoose = require('mongoose')

const UserSchema = new mongoose.Schema({
  name: {
    type: String,
    required: [true, 'Please provide na,e property'],
    minLength: 2,
    maxLength: 50,
  },
  email: {
    type: String,
    unique: true,
    required: [true, 'Email address is required'],
    match: [
      /^\w+([\.-]?\w+)*@\w+([\.-]?\w+)*(\.\w{2,3})+$/,
      'Please fill a valid email address',
    ], // we will use a validator package later to validate this
  },
  password: {
    type: String,
    required: [true, 'Please provide password'],
    minLength: 3,
  },
  role: {
    enum: ['admin', 'user'],
    default: 'user',
  },
})

const UserModel = mongoose.model('User', UserSchema)

module.exports = UserModel

Setup Custom validator

As you can see above, in the email, we are using match option and doing the email validation manually. In mongoose, we can do custom validation (write custom validators if the builtin ones are not enough)

validate is a property which we can use on one of the fields. This can have two properties

• Validate function

• Message

For the validate function, we can use our own validation but instead we can use a npm package called validator

So, let's modify the above code like this for User model

const mongoose = require('mongoose')
const validator = require('validator')
const UserSchema = new mongoose.Schema({
  name: {
    type: String,
    required: [true, 'Please provide na,e property'],
    minLength: 2,
    maxLength: 50,
  },
  email: {
    type: String,
    unique: true,
    required: [true, 'Email address is required'],
    validate: {
      validator: validator.isEmail,
      message: 'Please provide a valid email',
    },
  },
  password: {
    type: String,
    required: [true, 'Please provide password'],
    minLength: 3,
    maxLength: 40,
  },
  roles: {
    type: String,
    enum: ['admin', 'user'],
    default: 'user',
  },
})

const UserModel = mongoose.model('User', UserSchema)

module.exports = UserModel

3.2 Create Auth Controller

• Create controllers folder add authController file

• Export (register,login,logout) functions

• res.send('some string value')

auth/authController.js

const register = async (req, res) => {
  res.send('Register User')
}

const login = async (req, res) => {
  res.send('Login User')
}

const logout = async (req, res) => {
  res.send('Logout User')
}

module.exports = { register, login, logout }

3.3 Create Auth Routes

• Create routes folder

• Setup authRoutes file and import all controllers

• Setup three routes - post('/register') post('/login') get('/logout')

auth/authRoutes.js

const { register, login, logout } = require('../controllers/authController')

const express = require('express')
const router = express.Router()

// showing both ways here but I prefer router.post syntax when there's a single route
router.route('/register').post(register)
router.post('/login', login)
router.route('/logout').get(logout)

module.exports = router

3.4 Postman - Test these routes created

Before we test these routes, in the last project, you might have noticed after deploying the app when you wanted to test the deployed app, you had to change local URL to PROD URL in all the routes which is not the best way to do. Instead we can create a new Environment and create variables there. If we want to change to production, we will have prod environment and that way we don't have to manually change URL or any other values.

postman auth routes

3.5 Register Controller

• Create user

  • Send response with entire user (only while testing)

  • Check if email already in use. We can check in two ways (Schema level and controller level). Let's do both here,

    • To check in schema, we already added unique - true which does it for us

    • To check in controller, we need to check if user already exists with this email.

controller

const register = async (req, res) => {
  const { email } = req.body
  const emailAlreadyExists = await User.findOne({ email })
  if (emailAlreadyExists) {
    throw new customError.BadRequestError('Email already exists')
  }
  const user = await User.create(req.body)
  // only while testing we send this entire user object
  res.status(StatusCodes.CREATED).json({ user })
}

User Role

We have two types of user roles

• User

• Admin - Have more privileges like

  • View all users' account, modify them if necessary

  • Do all CRUD operations on Products whereas user can only view the products but not modify them

In our current setup, we can send the role as admin or user from the postman. But that's not a good way of doing it. Why? Because, if you think about it, we don't want anybody out there to be a admin who gets more privileges.

Let's take this reg page as an example, there's no user role selection option anywhere.

Registration UI

We don't want people to register as admins from this page. Not only front-end should be secure by not providing the user role option but also the backend should be secure. If you notice in the current implementation, the reg controller looks like this where we have User.create(req.body)

current regController

 const register = async (req, res) => {
  const { email } = req.body
  const emailAlreadyExists = await User.findOne({ email })
  if (emailAlreadyExists) {
    throw new customError.BadRequestError('Email already exists')
  }
  const user = await User.create(req.body) // What ever we pass here get's into DB in req.body
  // only while testing we send this entire user object
  res.status(StatusCodes.CREATED).json({ user })
}

We need to restrict and not put the role into DB even if user provides it via front-end or postman. We can modify our code like this

const register = async (req, res) => {
  const { name,email,password } = req.body
  const emailAlreadyExists = await User.findOne({ email })
  if (emailAlreadyExists) {
    throw new customError.BadRequestError('Email already exists')
  }
  const user = await User.create({name,email,password}) // now the role will not be sent to DB and the default User role will be used and sent to db
  // only while testing we send this entire user object
  res.status(StatusCodes.CREATED).json({ user })
} 

Setting the very first user as admin

We can also use another strategy here where we can set up our first user as admin and all other users as users. We can use a method called countDocuments on User object where it will count all documents and if it's 0 then the first user can be admin.

register controller

const register = async (req, res) => {
  const { name, email, password } = req.body
  const emailAlreadyExists = await User.findOne({ email })
  if (emailAlreadyExists) {
    throw new customError.BadRequestError('Email already exists')
  }
  // let's setup first user as admin
  const isFirstUser = (await User.countDocuments({})) === 0
  const role = isFirstUser ? 'admin' : 'user'
  const user = await User.create({ name, email, password, role })
  // only while testing we send this entire user object
  res.status(StatusCodes.CREATED).json({ user }

We can take multiple approaches to make a user admin

• We manually go to our DB and change the role to some users as 'admin'

• We can set first user as admin as shown above programmatically

• We let front-end/postman send us the role (BAD PRACTICE)

3.6 Hash passwords

Let's hash the passwords and store it to DB. As you know we will be using bcryptjs package to do this. In our previous project, we first did it in the register controller and once you got a grasp of how to do it there, we then made use of pre-commit hook called pre on the Schema to hash the password. Let's directly do pre in this case

models/User.js

UserSchema.pre('save', async function () {
  const salt = await bcrypt.genSalt(10)
  this.password = await bcrypt.hash(this.password, salt)
})

3.7 Compare Passwords

We've setup hashing password functionality. While we are here, let's also write functionality to compare the passwords. We can write the instance method (a method/function on the User Schema) to compare the password. By doing this, we can use this method in our Login controller later.

model/User.js

UserSchema.methods.comparePasswords = async function (candidatePassword) {
  const isMatch = bcrypt.compare(candidatePassword, this.password)
  return isMatch
}

// the above method can be used in login controller like this 
// User.comparePassword(passwordStr) which we will see shortly 

So the entire model/User.js looks like this

model/User.js

const mongoose = require('mongoose')
const validator = require('validator')
const bcrypt = require('bcryptjs')
const UserSchema = new mongoose.Schema({
  name: {
    type: String,
    required: [true, 'Please provide name property'],
    minLength: 2,
    maxLength: 50,
  },
  email: {
    type: String,
    unique: true,
    required: [true, 'Email address is required'],
    validate: {
      validator: validator.isEmail,
      message: 'Please provide a valid email',
    },
  },
  password: {
    type: String,
    required: [true, 'Please provide password'],
    minLength: 3,
    maxLength: 40,
  },
  role: {
    type: String,
    enum: ['admin', 'user'],
    default: 'user',
  },
})

// pre-commit hook to save the password
// this pre will be run before committing to the DB
UserSchema.pre('save', async function () {
  const salt = await bcrypt.genSalt(10)
  this.password = await bcrypt.hash(this.password, salt)
})

// this is the instance method we can create on a schema. 
// Later this can be used in controller as UserSchema.comparePasswords(passwordStr)
UserSchema.methods.comparePasswords = async function (candidatePassword) {
  const isMatch = bcrypt.compare(candidatePassword, this.password)
  return isMatch
}

const UserModel = mongoose.model('User', UserSchema)

module.exports = UserModel

3.8 Issue JWT (Json Web Token)

Once the user gets registered, we need to send back the token so that, using that token the user can further query the protected routes. If you take a look at Jobs API (previous project), we used to send back the token directly in response Step 3 (Register) in Register and Login controller.

Then the client (front-end/postman) used to send this token in subsequent requests to access the protected route (Jobs) Step 5 (Verify token - Auth middleware)

In Jobs API, we had setup the methods to sign the token and verify token on the user schema directly, but here in this project, lets create a utils folder and create a token and verify here. Obviously we can do it in multiple ways but let's do this here in utils folder

Steps we are going to take

• [] require 'jsonwebtoken' package

• [] create jwt - jwt.sign(payload,secret,options)

• [] verify jwt - jwt.verify(token,secret)

• [] add variables in .env JWT_SECRET=jwtSecret and JWT_LIFETIME=1d

• [] restart the server !!!!

• [] refactor code, create jwt functions in utils

• [] refactor cookie code

• [] setup func attachCookiesToResponse

• [] accept payload(res, tokenUser)

• [] create token, setup cookie

• [] optionally send back the response

Let's initially do this setup in register controller and then move this logic of token creation into utils folder

Controllers/authController.js

const register = async (req, res) => {
  const { name, email, password } = req.body
  const emailAlreadyExists = await User.findOne({ email })
  if (emailAlreadyExists) {
    throw new customError.BadRequestError('Email already exists')
  }
  // let's setup first user as admin
  const isFirstUser = (await User.countDocuments({})) === 0
  const role = isFirstUser ? 'admin' : 'user'
  const user = await User.create({ name, email, password, role })

  // generate jwt token. TODO: Move this logic to utils folder soon
  const tokenUser = { userID: user._id, name: user.name, role: user.role }
  const token = jwt.sign(tokenUser, 'jwtSecret', { expiresIn: '1d' })

  res.status(StatusCodes.CREATED).json({ tokenUser, token })
}

User registered response

Let's move our token creation code to the utils folder and also setup the token verification which we can later use it in authMiddleware.

utils/jwt.js

const jwt = require('jsonwebtoken')

const createJWT = ({ payload }) => {
  const token = jwt.sign(payload, process.env.JWT_SECRET, {
    expiresIn: process.env.JWT_LIFETIME,
  })
  return token
}

// we will use this token validation later in auth mw when we start querying protected routes
const isTokenValid = ({ token }) => jwt.verify(token, process.env.JWT_SECRET)

module.exports = { createJWT, isTokenValid }

Let's setup the index.js so that we can import by targeting utils folder

const { createJWT, isTokenValid } = require('./jwt')

module.exports = { createJWT, isTokenValid }

Let's now see how to modified authController looks like for register function

controller/authController.js

const register = async (req, res) => {
  const { name, email, password } = req.body
  const emailAlreadyExists = await User.findOne({ email })
  if (emailAlreadyExists) {
    throw new customError.BadRequestError('Email already exists')
  }
  // let's setup first user as admin
  const isFirstUser = (await User.countDocuments({})) === 0
  const role = isFirstUser ? 'admin' : 'user'
  const user = await User.create({ name, email, password, role })
  const tokenUser = {
    userId: user._id,
    name: user.name,
    role: user.role,
  }
  const token = createJWT({ payload: tokenUser }) // this is now called to create token

  res.status(StatusCodes.CREATED).json({ user:tokenUser, token })
}

3.9 Send JWT cookie

Once we have seen the creation of JWT, let's now look at an alternative method of how we can send our JWT.

In Jobs API (previous project) we saw that we send token in our response which is then stored in local storage on front-end. But we can also send our jwt on a cookie that gets attached to response and on the front-end, we don't have to store in the local storage, instead it gets directly attached in cookies in front-end and we can access from our browser. And also in the next request, the browser will automatically send the jwt from the cookie to us without having us to manually send from front-end for the next requests.

We will see soon the gotchas to both the approaches but first let's see how to send jwt in cookie. So we will create a cookie, attach jwt to that cookie and attach the cookie in our response.

Reference - https://expressjs.com/en/5x/api.html#res.cookie

Express Cookie

const register = async (req, res) => {
  const { name, email, password } = req.body
  const emailAlreadyExists = await User.findOne({ email })
  if (emailAlreadyExists) {
    throw new customError.BadRequestError('Email already exists')
  }
  // let's setup first user as admin
  const isFirstUser = (await User.countDocuments({})) === 0
  const role = isFirstUser ? 'admin' : 'user'
  const user = await User.create({ name, email, password, role })
  const tokenUser = {
    userId: user._id,
    name: user.name,
    role: user.role,
  }
  const token = createJWT({ payload: tokenUser })

  const oneDayinMillisec = 24 * 60 * 60 * 1000

  // send token via cookie
  res.cookie('token', token, {
    httpOnly: true,
    expiresIn: new Date(Date.now() + oneDayinMillisec),
  })

  res.status(StatusCodes.CREATED).json({ user:tokenUser }) // removed token from response here
}

jwt cookie

3.10 Cookie parsing

We saw how we can send the token in the cookie, now let's take a look at how we can parse the cookie we get back from client / postman in our request. We will check if front-end really attaches the cookie having token to it's requests.

As far as adding a cookie to the response we saw in 3.9, it's very easy that we do res.cookie. But to get the cookie that gets attached in a request by client we need to install an extra package named cookie-parser.

const express = require('express')
const app = express()

// rest of the packages
const morgan = require('morgan')
const cookieParser = require('cookie-parser') // COOKIE PARSER
const connectDB = require('./db/connect')
require('dotenv').config()
require('express-async-errors') // for avoiding writing try-catch in controllers
const notFoundMW = require('./middleware/not-found')
const errorHandlerMW = require('./middleware/error-handler')
const authRouter = require('./routes/authRoute')

//// Middlewares and Routes

app.use(morgan('tiny'))

app.use(express.json()) // used to get req.body data for post reqs
app.use(cookieParser()) // used to parse the cookies sent from the client(front-end) or postman

// Routes

// Basic Route
app.get('/', (req, res) => {
  console.log(req.cookies) // this is avaiable because of cookie-parser package
  res.send('E-Commerce API Home page')
})

// auth router
app.use('/api/v1/auth', authRouter)

app.use(notFoundMW)
app.use(errorHandlerMW)

////

const port = process.env.PORT || 5000

const start = async () => {
  try {
    // connect to db
    await connectDB(process.env.MONGO_URL)
    app.listen(port)
    console.log('Server is listening on port', port)
  } catch (err) {
    console.log(err)
  }
}

start()

cookie has the token in request we send from postman

And the other point to note is, like I said, once we send token in cookie from server, the cookie will be stored on cookies section in postman and cookies section in browser (if client is sending from front-end).

Then, in each request, the postman/ client will send us that cookie automatically. So once we send cookie from server, the browser or postman does the rest.

3.11 Refactor cookie code (3.10)

Once we are familiar with attaching the token in the cookie and parsing that which we get it in request, let's tweak the code we saw in 3.10 and add that code of attaching thee token in cookie in utils/jwt.js

adding code to attach cookie in utils/jwt.js

authController

3.12 Secure and signed flags

secured and signed flag in express

Let's think about sending cookies from browser to server. We are currently sending via HTTP which is not secure. What this cookie contains? It has a token. If browser sends the cookie containing a token and once browser sends it, before it reaches the server, if a hacker gets that cookie then he gets access to token and he can then use this token to send the request to server's protected routes on behalf of original client.

To avoid this we need a secured way (over HTTPS) to send cookies from browser to server. So we need to use this secure option in production environment. It doesn't matter in dev env.

If signed option is enabled then the server can detect if client modified the cookie the server had sent it. We can then do res.signedCookies to access it.

// in app.js

app.use(cookieParser(process.env.JWT_SECRET)) 

// in app.js to access it
// Basic Route
app.get('/', (req, res) => {
  console.log(req.signedCookies) // this is avaiable because of cookie-parser package
  res.send('E-Commerce API Home page')
})

// in jwt.js
const attachCookiesToResponse = ({ res, user }) => {
  const token = createJWT({ payload: user })
  const oneDayinMillisec = 24 * 60 * 60 * 1000
  res.cookie('token', token, {
    httpOnly: true,
    expires: new Date(Date.now() + oneDayinMillisec),
    secure: process.env.NODE_ENV === 'production',
    signed: true, // this will sign the cookie
  })
}

3.13 Login Route

Steps we need to take

• [] check if email and password exist, if one missing return 400 - Bad Request

• [] find user, if no user return 401 - Unauthorized or unauthenticated

• [] check password, if does not match return 401

• [] if everything is correct, attach cookie and send back the same response as in register

authController.js

const login = async (req, res) => {
  const { email, password } = req.body
  if (!email || !password)
    throw new customError.BadRequestError('Please provide email and password')
  const user = await User.findOne({ email })
  if (!user) throw new customError.UnauthenticatedError('Invalid credentials.')
  const isPasswordCorrect = await user.comparePasswords(password)
  if (!isPasswordCorrect)
    throw new customError.UnauthenticatedError('Invalid credentials.')
  const tokenUser = { userId: user._id, name: user.name, role: user.role }
  attachCookiesToResponse({ res, user: tokenUser })
  res.status(StatusCodes.OK).json({ user:tokenUser })
}

3.14 Logout Route

Having cookie with token means that the user is logged in. So in the logout route we need to send some other string in cookie instead of token, so that the user would not send us the token in the next request being logged out.

What if he already noted down the token somewhere and sends that token in the cookie after being logged out? To solve this we also need to invalidate the token (expire the token immediately so that when user users the same token in the next route it should not be valid)

Steps

• [] set token cookie equal to some string value - maybe logout

• [] set expires:new Date(Date.now())

const logout = async (req, res) => {
  res.cookie('token', 'logout', {
    httpOnly: true,
    expires: new Date(Date.now()),
  })
  res.status(StatusCodes.OK).json({ msg: 'user logged out' })
}

3.15 Create-React-APP with express

In Udemy John's Video 262. Cookies - Big picture and "Gotchas", he has explained how to connect CRA with node-express.

Issues when using Create-React-APP with Express

• Both are on different domains. Create-React-APP is on localhost:3000 and express is on localhost:5000

• When we try to access a route from CRA, it gives CORS error. You know that the way to solve this is by adding cors() to the server so that the resources on the server will be available

• The other issue that occurs when dealing with different domains is the cookies that we send in the response from our server. Token in this case. This cookie from our express server won't show up in the CRA front-end. The way to get around it is by proxy configuration in CRA package.json file.

  • We need to setup proxy pointing to localhost:5000, our server in CRA package.json. "proxy" : "http:localhost:5000"

  • Now where ever we try to access the server route, for example, http:localhost:5000/api/v1/xyz then we can just do this way in CRA,

      const fetchRoute = async () => {
            url = `/api/v1/xyz` // no need to write localhost:5000 prefix. It's specifieds in proxy already
            await fetch(url)
      }

Step 4 (Auth Middleware)

FYI, before Step 4, I built step 5 and then came to step 4. It's ok for you to go in order but don't get confused!! 😕

Time to implement the authentication middleware to give access to protected routes. Next, we will be implementing th User routes and, as an admin, we might have to get all the users info. For this the user who is requesting this info (all users or single user) must be an admin. How can we test that? Well we need a middleware (auth mw) to test whether the token is valid and the user who is requesting is actually an admin. So, two parts to this:

• authenticationMW should implement a method to check the token if valid, and if it is valid then pass the payload we get from validating the token to next route

• Along with checking the token is valid we also need to check if the token belongs to admin user and not the normal user

4.1 Authenticate User (Validate token)

Authentication MW - Validating the token and passing the payload to next route

middleware/authentication.js

const customErrors = require('../errors')
const { isTokenValid } = require('../utils')

const authenticateUser = async (req, res, next) => {
  const { token } = req.signedCookies
  console.log('The token is', token)
  if (!token)
    throw new customErrors.UnauthenticatedError('Authentication Invalid')
  try {
    // we get payload (that we stored in token while creating jwt) if token is valid
    const { name, userId, role } = isTokenValid({ token })
    req.user = { name, userId, role }
    next()
  } catch (error) {
    throw new customErrors.UnauthenticatedError('Authentication Invalids')
  }
}

module.exports = authenticateUser

4.2 Authorize permissions

Authentication MW - Check if user (attached in req in above route authenticateUser) is admin

const authorizePermissions = async (req, res, next) => {
  console.log('Coming from authorize permissions', req.user)
  if (req.user.role !== 'admin') {
    throw new customErrors.UnAuthorizedError( // creating this error below
      'Unauthorized to access this route'
    )
  }
  next()
}

4.3 Unauthorized error

We created Unauthorized error class which didn't exist before (similar to other errors)

errors/unauthorized.js

const { StatusCodes } = require('http-status-codes')
const CustomAPIError = require('./custom-api')

class UnAuthorizedError extends CustomAPIError {
  constructor(message) {
    super(message)
    this.statusCode = StatusCodes.FORBIDDEN
  }
}

module.exports = UnAuthorizedError

Let's now use this authenticate and authorized permissions as shown in the User route. We will create User route in next section, so you can come back to this after setting up (section 5 below) User routes

I did section 5 first and then section 4, so I already had user routes below. Doesn't really matter but don't get confused if you're studying this in the future (which I believe is now, 😂 )

4.4 Using authentication MW

routes/UserRoute.js

const express = require('express')
const router = express.Router()

const {
  getAllUsers,
  getSingleUser,
  showCurrentUser,
  updateUser,
  updateUserPassword,
} = require('../controllers/userController')

const {
  authenticateUser,
  authorizePermissions,
} = require('../middleware/authentication')

router.route('/').get(authenticateUser, authorizePermissions, getAllUsers) // first authenticates user, then checks if admin and then allows to proceed to access protected route
router.route('/showMe').get(showCurrentUser)
router.route('/updateUser').patch(updateUser)
router.route('/updateUserPassword').post(updateUserPassword)

// this rout with :id must be placed last so that it won't interfer with previous routes mentioned above this
router.route('/:id').get(authenticateUser, getSingleUser)

module.exports = route

4.4 Authorise permissions for multiple roles

In middlewares/authentication.js, we have this code which checks if the role is admin.

if (req.user.role !== 'admin')

What if we have multiple roles, then we need to hardcode every role and check? No, we can pass this as a param from user router and check against req.user.role like this

middleware/authentication.js

const authorizePermissions = (roles) => {
// we are now returning a function 
  return async (req, res, next) => {
    if (!roles.includes(req.user.role)) {
      throw new customErrors.UnAuthorizedError(
        'Unauthorized to access this route'
      )
    }
    next()
  }
}

And it's called in the user router like this. The commented code was used earlier

routes/UserRouter.js

const express = require('express')
const router = express.Router()

const {
  getAllUsers,
  getSingleUser,
  showCurrentUser,
  updateUser,
  updateUserPassword,
} = require('../controllers/userController')

const {
  authenticateUser,
  authorizePermissions,
} = require('../middleware/authentication')
// router.route('/').get(authenticateUser, authorizePermissions, getAllUsers)
router
  .route('/')
  .get(authenticateUser, authorizePermissions(['admin']), getAllUsers) // we can add multiple roles here, like admin, ownwer and so on.
router.route('/showMe').get(showCurrentUser)
router.route('/updateUser').patch(updateUser)
router.route('/updateUserPassword').post(updateUserPassword)

// this rout with :id must be placed last so that it won't interfer with previous routes mentioned above this
router.route('/:id').get(authenticateUser, getSingleUser)

module.exports = router

Now we are not hardcoding the role. Tomorrow if I have more roles like owner, superuser and so on, I can include it in array and don't have to change my authorizePermissions middleware

Step 5 (User Routes)

Uff! We completed the auth part. Now let's think from an admin perspective! What if you are an admin, wouldn't you like to have read/write access to all the Users we just created? Of course. So being an admin, we should be able to

• Get All Users

• Get Single User

• Show Current logged in user

• Update User info

• Update User Password

Let's set up some boiler plate for the same and these are the steps we will take

Steps

• [] add userController file

• [] export (getAllUsers,getSingleUser,showCurrentUser,updateUser,updateUserPassword) functions

• [] res.send('some string value')

• [] setup userRoutes file

• [] import all controllers

• [] setup just one route - router.route('/').get(getAllUsers);

• [] import userRoutes as userRouter in the app.js

• [] setup app.use('/api/v1/users', userRouter)

Code

controllers/userController.js

const getAllUsers = async (req, res) => {
  res.send('Get All Users route')
}

const getSingleUser = async (req, res) => {
  res.send('Get Single User route')
}

const showCurrentUser = async (req, res) => {
  res.send('Show current user route')
}

const updateUser = async (req, res) => {
  res.send('Update user route')
}

const updateUserPassword = async (req, res) => {
  res.send('update User Password route')
}

module.exports = {
  getAllUsers,
  getSingleUser,
  showCurrentUser,
  updateUser,
  updateUserPassword,
}

routes/userRoute.js

const express = require('express')
const router = express.Router()

const {
  getAllUsers,
  getSingleUser,
  showCurrentUser,
  updateUser,
  updateUserPassword,
} = require('../controllers/userController')

router.route('/').get(getAllUsers)
router.route('/showMe').get(showCurrentUser)
router.route('/updateUser').patch(updateUser)
router.route('/updateUserPassword').post(updateUserPassword)

// this rout with :id must be placed last so that it won't interfer with previous routes mentioned above this
router.route('/:id').get(getSingleUser)

module.exports = router

partial app.js for this example

// auth router
app.use('/api/v1/auth', authRouter)

app.js

const express = require('express')
const app = express()

// rest of the packages
const morgan = require('morgan')
const cookieParser = require('cookie-parser')
const connectDB = require('./db/connect')
require('dotenv').config()
require('express-async-errors') // for avoiding writing try-catch in controllers
const notFoundMW = require('./middleware/not-found')
const errorHandlerMW = require('./middleware/error-handler')
const authRouter = require('./routes/authRoute')
const userRouter = require('./routes/userRoute')

//// Middlewares and Routes

app.use(morgan('tiny'))

app.use(express.json()) // used to get req.body data for post reqs
app.use(cookieParser(process.env.JWT_SECRET)) // used to parse the cookies sent from the client(front-end) or postman

// Routes

// Basic Route
app.get('/', (req, res) => {
  console.log(req.signedCookies) // this is avaiable because of cookie-parser package
  res.send('E-Commerce API Home page')
})

// auth router
app.use('/api/v1/auth', authRouter)

// user router
app.use('/api/v1/users', userRouter)

app.use(notFoundMW)
app.use(errorHandlerMW)

const port = process.env.PORT || 5000

const start = async () => {
  try {
    // connect to db
    await connectDB(process.env.MONGO_URL)
    app.listen(port)
    console.log('Server is listening on port', port)
  } catch (err) {
    console.log(err)
  }
}

start()

5.1 Get All Users and Get Single User

Steps

• [] Get all users where role is 'user' and remove password

• [] Get Single User where id matches id param and remove password

• [] If no user 404

controllers/userController

const { StatusCodes } = require('http-status-codes')
const customErrors = require('../errors')
const User = require('../models/User')

const getAllUsers = async (req, res) => {
  const users = await User.find({ role: 'user' }).select('-password')
  res.status(StatusCodes.OK).json({ users })
}

const getSingleUser = async (req, res) => {
  const { id } = req.params
  const user = await User.findOne({ _id: id }).select('-password')
  if (!user)
    throw new customErrors.NotFoundError(`User not found with ID ${id}`)
  res.status(StatusCodes.OK).json({ user })
}

// we will work on the below later
const showCurrentUser = async (req, res) => {
  res.send('Show current user route')
}

const updateUser = async (req, res) => {
  res.send('Update user route')
}

const updateUserPassword = async (req, res) => {
  res.send('update User Password route')
}

module.exports = {
  getAllUsers,
  getSingleUser,
  showCurrentUser,
  updateUser,
  updateUserPassword,
}

5.2 Show Current User

Let's say a user logs in, we send him the token. With that he can further access the protected route. We check the token and if it's valid we allow him to access it.

What if he refreshes the page, he would still be sending the token in a cookie, and we have our authentication mw to check the token and give us back the user. We can send this user back without querying the DB (we would be checking if token is valid in auth mw like I said and that gets us the current user) and the client can display the username and so on the front-end using this route. He must of course be authenticated to access this route.

// router/userRouter.js (partial file content)
router.route('/showMe').get(authenticateUser, showCurrentUser)

// the above route hits this code (authenticate mw)

// middleware/authenticate.js (partial file content)
const authenticateUser = async (req, res, next) => {
  const { token } = req.signedCookies
  if (!token)
    throw new customErrors.UnauthenticatedError('Authentication Invalid')
  try {
    // we get payload (that we stored in token while creating jwt) if token is valid
    const { name, userId, role } = isTokenValid({ token })
    req.user = { name, userId, role }
    next()
  } catch (error) {
    throw new customErrors.UnauthenticatedError('Authentication Invalids')
  }
}

// if the above is successful where token would be present(meaning the user is logged in)
// , then we get the req.user passed to next request which would be showMe route
// to show the current user. This is the first route that hits when user refreshes the page

// controllers/UserController.js (partial file content)
const showCurrentUser = async (req, res) => {
  res.status(StatusCodes.OK).json({ user: req.user })
}

5.3 Update Password

Let's say a logged-in user needs to update his password, then we need this route.

Steps to Update User Password

• [] almost identical to login user

• [] add authenticateUser middleware in the route

• [] check for oldPassword and newPassword in the body

• [] if one missing 400

• [] look for user with req.user.userId

• [] check if oldPassword matches with user.comparePassword

• [] if no match 401

• [] if everything good set user.password equal to newPassword

• [] await user.save()

The save method validates the Schema before saving. If new password length is less or more than the one mentioned in the validation then it complains.

// routes/userRoute.js (partial file)
router.route('/updateUserPassword').patch(authenticateUser, updateUserPassword)

// controllers/userController.js (partial file)
const updateUserPassword = async (req, res) => {
  const { oldPassword, newPassword } = req.body
  if (!oldPassword || !newPassword) {
    throw new customErrors.BadRequestError('Old or new password is missing')
  }
  const user = await User.findOne({ _id: req.user.userId })
  const isMatch = await user.comparePasswords(oldPassword, user.password)
  if (!isMatch) {
    throw new customErrors.UnauthenticatedError('Invalid Credentials')
  }
  user.password = newPassword
  await user.save() // user.save will run the pre hook (which hashes the password) before saving to db
  // save() method also runs DB validation
  res.status(StatusCodes.OK).json({ message: 'Updated password successfully' })
}

5.4 Create TokenUser function

In the authController, we have a code to create the tokenUser object. We are repeating this code in both register and login functions and later need this in other place. So let's put this code in utils..

create tokenUser

Steps to createTokenUser in Utils

• [] create a file in utils (createTokenUser)

• [] setup a function that accepts user object and returns userToken object

• [] export as default

• [] setup all the correct imports/exports and refactor existing code

// Replacing this below code in authController register and login functions

const { attachCookiesToResponse, createTokenUser } = require('../utils')

// const tokenUser = { userId: user._id, name: user.name, role: user.role }
const tokenUser = createTokenUser(user)

/************************************************/

// utils/createTokenUser.js

const createTokenUser = (user) => {
  // this function returns userToken object
  return { userId: user._id, name: user.name, role: user.role }
}

module.exports = createTokenUser

/************************************************/

// utils/index.js
const { createJWT, isTokenValid, attachCookiesToResponse } = require('./jwt')
const createTokenUser = require('../utils/createTokenUser')
module.exports = {
  createJWT,
  isTokenValid,
  attachCookiesToResponse,
  createTokenUser,
}

5.5 Update User

Let's update the user now. What properties can we update. The role shouldn't be updated directly. The password needs to be updated with updatePassword route we have already seen. The remaining fields are email and password.

We have two update options

• findOneAndUpdate / findByIdAndUpdate

• save()

Let's see both

5.5.1 findOneAndUpdate

updateUser with User.findOneAndUpdate()

• [] add authenticateUser middleware in the route

• [] check for name and email in the body

• [] if one is missing, send 400 (optional)

• [] use findOneAndUpdate()

• [] create token user, attachCookiesToResponse and send back the tokenUser

Notice the last point. Once we update the properties like email and pass, it would be nicer to also update the tokenUser , attach to cookies and send back the new token user in response. This is because, lets say if the user updated the username / name that was displayed in his welcome message on the screen, then after the update he should be seeing the new name

// controllers/userController.js (partial file)

const updateUser = async (req, res) => {
  const { name, email } = req.body
  if (!name || !email) {
    throw new customErrors.BadRequestError('Please provide name and email')
  }
  const user = await User.findOneAndUpdate(
    { _id: req.user.userId },
    { name, email },
    { new: true, runValidators: true }
  )
  const tokenUser = createTokenUser(user)
  attachCookiesToResponse({ res, user: tokenUser })
  res.status(StatusCodes.OK).json({ user: tokenUser })
}

// routes/userRoutes.js (partial)
router.route('/updateUser').patch(authenticateUser, updateUser)

5.5.2 save()

We can update the user using save method as well like we did for update password. The steps here would be

• find the user using User.findOne()

• update the values on this object. Example, user.email = email and so on

• Then do User.save()

This works for sure and updates the email and name but would break the password and you will not be able to login again.

There are some gotchas with this and knowing that would fix the issue

First, let's see what happens on save(). Let's write code for this

const updateUser = async (req, res) => {
  const { name, email } = req.body
  if (!name || !email) {
    throw new customErrors.BadRequestError('Please provide name and email')
  }

  // findOneAndUpdate doesn't invoke pre save hook

  // const user = await User.findOneAndUpdate(
  //   { _id: req.user.userId },
  //   { name, email },
  //   { new: true, runValidators: true }
  // )

  // save() invokes/calls the pre save hook
  const user = await User.find({ _id: req.user.userId })
  user.name = name
  user.email = email
  
  await user.save() // this calls pre-hook that runs first and then is saved to db

  const tokenUser = createTokenUser(user)
  attachCookiesToResponse({ res, user: tokenUser })
  res.status(StatusCodes.OK).json({ user: tokenUser })
}

/* pre-save hook for User that's called on create user and on user.save()  */
// pre-commit hook to save the password
// this pre will be run before committing to the DB on save() and while creating new user
UserSchema.pre('save', async function () {
  const salt = await bcrypt.genSalt(10)
  this.password = await bcrypt.hash(this.password, salt)
})

When you reach the line user.save(), then before saving into database, it runs the 'pre save' hook as shown above. This will create a new salt and hash the password again because of which the existing password becomes invalid and the user cannot login anymore after this update. This is very important to understand and remember.

So what's the fix here?

We have a method called modifiedPaths() on the user object. This would give us the array of values that we modified/updated. In this case, we updated email and name. Password wasn't updated so, this.modifiedPaths() give us ['name','email']

UserSchema.pre('save', async function () {
  console.log('shows what we are modifying', this.modifiedPaths()) // ['name','email']
  const salt = await bcrypt.genSalt(10)
  this.password = await bcrypt.hash(this.password, salt)
})

And more interestingly, I can see if the property was modified or not using

console.log(this.isModified('password)) // we get false as this was not modified

So, the below code can save us from updating the password

UserSchema.pre('save', async function () {
  // console.log('shows what we are modifying', this.modifiedPaths()) // ['name','email']
  // if we didn't modify the password then we just return here
  if (!this.isModified('password')) return 
  const salt = await bcrypt.genSalt(10)
  this.password = await bcrypt.hash(this.password, salt)
})

In line 4 above, we say that if password is not modified (which is true in this case as we are just changing email and name) then don't proceed further to create salt and hash, just return.

Also, what about while creating the User?

While creating the user, line number 2, this.modifiedPaths() would log all the values as all the values which are new when registering the user are considered being changed so line 4 will be false and the password will be hashed. So nothing to worry there as well.

5.6 Check Permissions

Things look great so far but there is one pothole. In getSingleUser route, currently one user who is logged in can get the details of other user if the logged in user knows the ID of other user as shown below. We need to avoid this.

get single user security pothole

We will have to set up a util function to avoid this and this can be reusable later. In this function we will check for the permissions. For params for this function, we will pass in req user (getting from auth user mw) and also we pass id property that we get from the model. This can be user model (in this case _id) or review model for future and so on.

// controllers/userController.js
const getSingleUser = async (req, res) => {
  const { id } = req.params
  const user = await User.findOne({ _id: id }).select('-password')
  if (!user)
    throw new customErrors.NotFoundError(`User not found with ID ${id}`)
  // what are we checking in checkPermissions?
  // Our authenticated user (req.user) and user._id got from DB, if they both are same

  // if they both are same, that means the logged in user is accessing his own info and we can very well give him

  // if they both are not same, that means the logged in user is accessing others' user id. We can give the user back only if the user requesting is an admin. If not we will throw error

  // both user._id and req.param.id will be same but user._id will be in form of object. we need toString() on that
  checkPermissions(req.user, user._id)
  res.status(StatusCodes.OK).json({ user })
}

// utils/checkPermissions.js
const customError = require('../errors')

const checkPermissions = (reqUser, resourceUserID) => {
  //   console.log(reqUser) // { name: 'john', userId: '63044830c70600e17b5ac3f3', role: 'user' }
  //   console.log(resourceUserID) //new ObjectId("6304483ac70600e17b5ac3f7")
  //   console.log(typeof resourceUserID) // object
  //   console.log(resourceUserID.toString()) // 6304483ac70600e17b5ac3f7

  // if we are using return, that means we are not stopping the request, we simply return to caller (getSingleUser controller) and then proceed to give back the response
  if (reqUser.userId === resourceUserID.toString()) return
  if (reqUser.role === 'admin') return
    
  // if one of the above two conditions meet then we throw error as it would be unauthorized route
  throw new customError.UnAuthorizedError('Not authorized to access this route')
}

module.exports = checkPermissions

Step 6 (Products)

Let's now work on creating some products. It should be straight forward from here as we have already learnt most of the stuff. Let's start with Products model.

6.1 Products Model

Steps

• [] create Product.js in models folder

• [] create Schema

• [] name : {type:String}

• [] price: {type:Number}

• [] description: {type:String}

• [] image: {type:String}

• [] category: {type:String}

• [] company: {type:String}

• [] colors: {type:[]}

• [] featured: {type:Boolean}

• [] freeShipping: {type:Boolean}

• [] inventory:{type:Number}

• [] averageRating:{type:Number}

• [] user

• [] set timestamps

• [] export Product model

Note

How to setup enum values?

// until now we have setup this way
  category: {
    type: String,
    required: [true, 'Please provide product category'],
    enum: ['office', 'kitchen', 'bedroom'],
  },
  
  // We can also setup like this below. We can use this when we need to 
  // provide back message saying the VALUE provided is not supported
  company: {
    type: String,
    required: [true, 'Please provide company category'],
    enum: {
      values: ['ikea', 'liddy', 'marcos'],
      message: '{VALUE} is not supported',
    },
  },

models/Product.js

const mongoose = require('mongoose')
const ProductSchema = new mongoose.Schema(
  {
    name: {
      type: String,
      trim: true,
      required: [true, 'Please provide product name'],
      maxLength: [100, 'Name cannot be more than 100 characters'],
    },
    price: {
      type: Number,
      required: [true, 'Please provide price'],
      default: 0,
    },
    description: {
      type: String,
      required: [true, 'Please provide product descritption'],
      maxLength: [1000, 'Description cannot be more than 1000 characters'],
    },
    image: {
      type: String,
      default: '/uploads/example.jpeg',
    },
    category: {
      type: String,
      required: [true, 'Please provide product category'],
      enum: ['office', 'kitchen', 'bedroom'],
    },
    company: {
      type: String,
      required: [true, 'Please provide company category'],
      enum: {
        values: ['ikea', 'liddy', 'marcos'],
        message: '{VALUE} is not supported',
      },
    },
    // we will see how this array of String works when we send data from postman
    colors: {
      type: [String],
      required: true,
    },
    featured: {
      type: Boolean,
      default: false,
    },
    freeShipping: {
      type: Boolean,
      default: false,
    },
    inventory: {
      type: Number,
      required: tue,
      default: 15,
    },
    // we will calculate this rating and num of reviews as we build up
    averageRating: {
      type: Number,
      default: 0,
    },
    numOfReviews: {
      type: Number,
      default: 0,
    },
    user: {
      type: mongoose.Types.ObjectId,
      ref: 'User',
      required: [true, 'Please provide a user'],
    },
  },
  { timestamps: true }
)

module.exports = mongoose.model('Product', ProductSchema)

Note : We will see how to calculate averageRating and numOfReviews later herer -> Step 10 (Average rating and number of Reviews on Product)

6.2 Products Structure

Steps

• [] add productController file in controllers

• [] export (createProduct, getAllProducts, getSingleProduct, updateProduct, deleteProduct, uploadImage) functions

• [] res.send('function name')

• [] setup productRoutes file in routes

• [] import all controllers

• [] only getAllProducts and getSingleProduct accessible to public

• [] rest only by admin (setup middlewares)

• [] typical setup

• [] router.route('/uploadImage').post(uploadImage)

• [] import productRoutes as productRouter in the app.js

• [] setup app.use('/api/v1/products', productRouter)

controllers/ProductController.js

const createProduct = async (req, res) => {
  res.send('Create Product')
}
const getAllProducts = async (req, res) => {
  res.send('Get All Product')
}
const getSingleProduct = async (req, res) => {
  res.send('Get Single Product')
}
const updateProduct = async (req, res) => {
  res.send('Update Product')
}
const deleteProduct = async (req, res) => {
  res.send('Delete Product')
}
const uploadImage = async (req, res) => {
  res.send('Upload image')
}

module.exports = {
  createProduct,
  getAllProducts,
  getSingleProduct,
  updateProduct,
  deleteProduct,
  uploadImage,
}

routes/productRoute.js

const express = require('express')
const router = express.Router()
const {
  createProduct,
  getAllProducts,
  getSingleProduct,
  updateProduct,
  deleteProduct,
  uploadImage,
} = require('../controllers/productController')
const {
  authenticateUser,
  authorizePermissions,
} = require('../middleware/authentication')

router
  .route('/')
  .get(getAllProducts)
  .post(authenticateUser, authorizePermissions(['admin']), createProduct)
router
  .route('/uploadImage')
  .post(authenticateUser, authorizePermissions(['admin']), uploadImage)
router
  .route('/:id')
  .get(getSingleProduct)
  .patch(authenticateUser, authorizePermissions(['admin']), updateProduct)
  .delete(authenticateUser, authorizePermissions(['admin']), deleteProduct)

module.exports = router

app.js

const express = require('express')
const app = express()

// rest of the packages
const morgan = require('morgan')
const cookieParser = require('cookie-parser')
const connectDB = require('./db/connect')
require('dotenv').config()
require('express-async-errors') // for avoiding writing try-catch in controllers
const notFoundMW = require('./middleware/not-found')
const errorHandlerMW = require('./middleware/error-handler')
const authRouter = require('./routes/authRoute')
const userRouter = require('./routes/userRoute')
const productRouter = require('./routes/productRoute')

//// Middlewares and Routes

app.use(morgan('tiny'))

app.use(express.json()) // used to get req.body data for post reqs
app.use(cookieParser(process.env.JWT_SECRET)) // used to parse the cookies sent from the client(front-end) or postman

// Routes

// Basic Route
app.get('/', (req, res) => {
  console.log(req.signedCookies) // this is avaiable because of cookie-parser package
  res.send('E-Commerce API Home page')
})

// auth router
app.use('/api/v1/auth', authRouter)

// user router
app.use('/api/v1/users', userRouter)

// product router
app.use('/api/v1/products', productRouter)

app.use(notFoundMW)
app.use(errorHandlerMW)

const port = process.env.PORT || 5000

const start = async () => {
  try {
    // connect to db
    await connectDB(process.env.MONGO_URL)
    app.listen(port)
    console.log('Server is listening on port', port)
  } catch (err) {
    console.log(err)
  }
}

start()

6.3 Postman setup for products

Let's setup all the routes in postman for products

• Get All Products (GET)

• Get Single Product (GET)

• Create Product (POST) - Admin route

• Update Product (PATCH) - Admin route

• Delete Product (DELETE) - Admin route

• Upload Image (POST) - Admin route

Configure these routes in postman and login as admin. You should be able to see the appropriate responses from controllers. Just to test, login as user and all the Admin routes should give you message 'unauthorized to access this route'

6.4 Product CRUD

6.4.1 Create Product

Once we have all the basic setup working, let's focus on Creating product. We get all the details in req.body except for the user. Remember, the product model has got User property to show createdBy. We get this similar to Jobs API. req.user (getting from authentication middleware) will have the user prop and we will attach that user to req.body and then create the product.

Getting user prop from auth mw

user from auth mw

// controllers/productController.js - partial file

const createProduct = async (req, res) => {
  req.body.user = req.user.userId // req.uesr.uesrID comes from auth mw
  const product = await Product.create(req.body)
  res.status(StatusCodes.CREATED).json({ product })
}

Reference - https://mongoosejs.com/docs/populate.html

6.4.2 Get All Products

const getAllProducts = async (req, res) => {
  const products = await Product.find({})
  res.status(StatusCodes.OK).json({ products, count: products.length })
}

6.4.3 Get Single Product

const getSingleProduct = async (req, res) => {
  const { id: productId } = req.params

  // We can use both findById and findOne as shown below

  // const product = await Product.findById(productId)
  const product = await Product.findOne({ _id: productId })
  if (!product) {
    throw new CustomErrors.NotFoundError(`No product with ID ${productId}`)
  }
  res.status(StatusCodes.OK).json({ product })
}

6.4.4 Update Product

const updateProduct = async (req, res) => {
  const { id: productId } = req.params
  const product = await Product.findOneAndUpdate({ _id: productId }, req.body, {
    new: true,
    runValidators: true,
  })
  if (!product) {
    throw new CustomErrors.NotFoundError(`No product with ID ${productId}`)
  }
  res.status(StatusCodes.OK).json({ product })
}

6.4.5 Delete Product

We are not doing findOneAndDelete here. Instead we first find the product using findOne and then remove it. This will be helpful later when we do Reviews.

const deleteProduct = async (req, res) => {
  const { id: productId } = req.params
  const product = await Product.findOne({ _id: productId })
  if (!product) {
    throw new CustomErrors.NotFoundError(`No product with ID ${productId}`)
  }
  // we are not doing findOneAndDelete here because we will do some other functionality later
  // after finding the product and before deleting it here
  await product.remove()
  res.status(StatusCodes.OK).json({ msg: 'Success! Product removed' })
}

6.4.6 Upload Image for Product

Let's store images on server this time rather than cloudinary. We will store it in /public/uploads on our server

• Make a new folder public under root and under public make another folder uploads. So it should be /public/uploads

• Go to app.js and require package express-fileupload

• Then in app.js enable public folder to be publicly available

• Then in app.js invoke fileUpload

app.js (partial)

const fileUpload = require('express-fileupload')

// enable public folder to be publicly available
app.use(express.static('./public'))
app.use(fileUpload())

• If you want to setup some default image (if user doesn't give image URL while creating product) then add your image into public folder. Example /public/uploads/example.jpeg. Remember this is our default we had setup in Product Model

• Ok now when we upload an image from postman in uploadsImage route, in the uploadsImage controller we will have access to uploaded image in req.files

• We will throw error if

  • req.files not present

  • image on req.files is not an image and some other file (determined by mimetype)

  • image on req.files is larger than mentioned size

• Once we have our uploaded image pass all the conditions then we need to move our image to specific path on our computer which is /public/uploads/. For this we need to construct full path for destination

• Then on the uploaded image, we have mv function that helps moving the image. We can move that image to destination path mentioned above.

• Finally we will send back image path (no need to send full path) and we send /uploads/{imagename}. This path would be sufficient instead of full path as user will know about /public which we exposed in step 3 above where we made /public folder publicly available

controller/productController.js partial

const uploadImage = async (req, res) => {
  if (!req.files) {
    throw new CustomErrors.BadRequestError('No file uploaded')
  }
  const productImage = req.files.image
  if (!productImage.mimetype.startsWith('image')) {
    throw new CustomErrors.BadRequestError('Please upload an image file')
  }
  const maxSize = 1024 * 1024
  if (productImage.size > maxSize) {
    throw new CustomErrors.BadRequestError(
      'Please upload an image of size less than 1Mb'
    )
  }
  // the image path will be the complete path (DESTINATION PATH we need to move) of image on our computer
  const imagePath = path.join(
    __dirname,
    '../public/uploads/',
    `${productImage.name}`
  )
  // moving our uploaded image to the above mentioned path
  productImage.mv(imagePath)
  res
    .status(StatusCodes.OK)
    .json({ img: { src: `/uploads/${productImage.name}` } })
}

Step 7 (Reviews)

Let's now work on Reviews.

7.1 Review Model

Steps

• [] create Review.js in models folder

• [] create Schema

• [] rating : {type:Number}

• [] title: {type:String}

• [] comment: {type:String}

• [] user

• [] product

• [] set timestamps

• [] export Review model

Also, one user should give only one review per product. We can achieve this two ways.

• By writing some code in controller (which we will see later)

• By using mongoose index. Remember we have already used unique=true for email. This means that all users will have unique email. Meaning, when a user needs to get created, the email prop he provides should not be there already in db for other emails. This is the indexing that is done by mongoose.

  • So in our Review model now, 1 user per 1 product should have 1 review and not more. So we need to set index for both user and product. This is a compound index (index includes multiple fields - User and Product in this case for Review Schama)

ReviewSchema.index({product:1,user:1},{unique:true})

model/Review.js

const mongoose = require('mongoose')
const ReviewSchema = new mongoose.Schema(
  {
    rating: {
      type: Number,
      min: 1,
      max: 5,
      required: [true, 'Please provide a rating'],
    },
    title: {
      type: String,
      trim: true,
      required: [true, 'Please provide review title'],
      maxLength: 100,
    },
    comment: {
      type: String,
      required: [true, 'Please provide review text'],
    },
    user: {
      type: mongoose.Types.ObjectId,
      ref: 'User',
      required: [true, 'Please provide a user'],
    },
    product: {
      type: mongoose.Types.ObjectId,
      ref: 'Product',
      required: [true, 'Please provide a product'],
    },
  },
  { timestamps: true }
)

// one user - one review per one product
ReviewSchema.index({ product: 1, user: 1 }, { unique: true })

module.exports = mongoose.model('Review', ReviewSchema)

7.2 Review Structure

Steps

• [] add reviewController file in controllers

• [] export (createReview, getAllReviews, getSingleReview, updateReview, deleteReview) functions

• [] res.send('function name')

• [] setup reviewRoutes file in routes

• [] import all controllers

• [] only getAllReviews and getSingleReview accessible to public

• [] rest only to users (setup middleware)

• [] typical REST setup

• [] import reviewRoutes as reviewRouter in the app.js

• [] setup app.use('/api/v1/reviews', reviewRouter)

controllers/reviewController.js

const createReview = async (req, res) => {
  res.send('Create Review')
}

const getAllReviews = async (req, res) => {
  res.send('Get All Review')
}

const getSingleReview = async (req, res) => {
  res.send('Get Single Review')
}

const updateReview = async (req, res) => {
  res.send('Update Review')
}

const deleteReview = async (req, res) => {
  res.send('Delete Review')
}

module.exports = {
  createReview,
  getAllReviews,
  getSingleReview,
  updateReview,
  deleteReview,
}

const express = require('express')
const router = express.Router()

const {
  createReview,
  getAllReviews,
  getSingleReview,
  updateReview,
  deleteReview,
} = require('../controllers/reviewController')

const { authenticateUser } = require('../middleware/authentication')

router.route('/').get(getAllReviews).post(authenticateUser, createReview)
router
  .route('/:id')
  .get(getSingleReview)
  .patch(authenticateUser, updateReview)
  .delete(authenticateUser, deleteReview)

module.exports = router

app.js

const express = require('express')
const app = express()

// rest of the packages
const morgan = require('morgan')
const cookieParser = require('cookie-parser')
const connectDB = require('./db/connect')
require('dotenv').config()
require('express-async-errors') // for avoiding writing try-catch in controllers
const notFoundMW = require('./middleware/not-found')
const errorHandlerMW = require('./middleware/error-handler')
const authRouter = require('./routes/authRoute')
const userRouter = require('./routes/userRoute')
const productRouter = require('./routes/productRoute')
const reviewRouter = require('./routes/reviewRoute')
const fileUpload = require('express-fileupload')

//// Middlewares and Routes

app.use(morgan('tiny'))
app.use(express.json()) // used to get req.body data for post reqs
app.use(cookieParser(process.env.JWT_SECRET)) // used to parse the cookies sent from the client(front-end) or postman

// enable public folder to be publicly available
app.use(express.static('./public'))
app.use(fileUpload())
// Routes

// Basic Route
app.get('/', (req, res) => {
  console.log(req.signedCookies) // this is avaiable because of cookie-parser package
  res.send('E-Commerce API Home page')
})

// auth router
app.use('/api/v1/auth', authRouter)

// user router
app.use('/api/v1/users', userRouter)

// product router
app.use('/api/v1/products', productRouter)

// review router
app.use('/api/v1/reviews', reviewRouter)

app.use(notFoundMW)
app.use(errorHandlerMW)

const port = process.env.PORT || 5000

const start = async () => {
  try {
    // connect to db
    await connectDB(process.env.MONGO_URL)
    app.listen(port)
    console.log('Server is listening on port', port)
  } catch (err) {
    console.log(err)
  }
}

start()

7.3 Postman setup for Reviews

Let's setup all the routes in postman for products

• Get All Reviews (GET)

• Get Single Review (GET)

• Create Review (POST) - Authenticated route

• Update Review (PATCH) - Authenticated route

• Delete Review (DELETE) - Authenticated route

7.4 Review CRUD

Once we have basic controller and routers for Review setup, let's now focus on getting Reviews working.

7.4.1 Create Review

Steps

• [] check for product in the req.body

• [] attach user property (set it equal to req.user.userId) on to req.body

• [] create review

• [] don't test yet

First we need to check if product that we are getting in req.body from postman/front-end is a valid product or not

controller/ReviewController.js partial

const createReview = async (req, res) => {
  const { product: productId } = req.body

  // req.body contains product.
  // First we need to validate if product sent in req.body actually exists
  // and also check that product is valid

  const isValidProduct = await Product.findOne({ _id: productId })
  if (!isValidProduct) {
    throw new CustomError.NotFoundError(`No product with id : ${productId}`)
  }

  req.body.user = req.user.userId
  const review = await Review.create(req.body)
  res.status(StatusCodes.CREATED).json({ review })
}

We can also check from controller here that if the user already left a review for this product or not. The first way we checked it was through Review.index in model. This is the second way how we can check that in the controller.

controller/ReviewController.js partial

const createReview = async (req, res) => {
  const { product: productId } = req.body

  // req.body contains product.
  // First we need to validate if product sent in req.body actually exists
  // and also check that product is valid

  const isValidProduct = await Product.findOne({ _id: productId })
  if (!isValidProduct) {
    throw new CustomError.NotFoundError(`No product with id : ${productId}`)
  }

  // second way of checking if user already submitted a review for this product.
  // The first way was in the Review model - ReviewSchema.index({ product: 1, user: 1 }, { unique: true })

  const alreadySubmitted = await Review.findOne({
    product: productId,
    user: req.user.userId,
  })

  // if user already submitted review for this product then he shouldn't be able to create review again
  if (alreadySubmitted) {
    throw new CustomError.BadRequestError(
      `This user with id ${req.user.userId} already left a review for this product id ${productId}`
    )
  }

  req.body.user = req.user.userId
  const review = await Review.create(req.body)
  res.status(StatusCodes.CREATED).json({ review })
}

Let's now test our create Review functionality

Create Review by logging in as a user

Create Review

Create review again with the same user

Creating review second time gives error

7.4.2 Get All Reviews

const getAllReviews = async (req, res) => {
  const reviews = await Review.find()
  res.status(StatusCodes.OK).json({ reviews, count: reviews.length })
}

7.4.3 Get Single Review

const getSingleReview = async (req, res) => {
  const { id: reviewId } = req.params
  const review = await Review.findOne({ _id: reviewId })
  if (!review) {
    throw new CustomError.NotFoundError(`No review found with id ${reviewId}`)
  }
  res.status(StatusCodes.OK).json({ review })
}

7.4.4 Delete Review

Steps

• [] get id from req.params

• [] check if review exists

• [] if no review, 404

• [] check permissions (req.user, review.user)

• [] use await review.remove()

• [] send back 200

Like we did in Jobs API, only a logged in user must be able to delete his own review. So here we check if I'm deleting somebody else's review then throw an error

const deleteReview = async (req, res) => {
  const { id: reviewId } = req.params
  const review = await Review.findOne({ _id: reviewId })
  if (!review) {
    throw new CustomError.NotFoundError(`No review found with id ${reviewId}`)
  }
  // check if user who created the review is actually requesting it. If not throw unauthorized error
  // if (review.user.toString() !== req.user.userId) {
  //   throw new CustomError.UnAuthorizedError(
  //     'Not authorized to access this review'
  //   )
  // }

  // below code does same thing as above commmented code
  checkPermissions(req.user, review.user)

  await review.remove()
  res.status(StatusCodes.OK).json({ msg: 'Successfully deleted the review' })
}

We will later learn why we are using review.remove() instead of findOneAndDelete. remove() has got some advantages over findOneAndDelete in this scenario which we will see it shortly. Similarly for Update Review below in 7.4.5 we use review.save() instead if findOneAndUpdate(). We will also see why that is important in this case shortly.

Why do we use remove and save methods (Click these and read)---> Step 9 (Remove associations) and Step 10 (Average rating and number of Reviews on Product)

7.4.5 Update Review

Let's go with save() method instead of findOneAndUpdate().

const updateReview = async (req, res) => {
  const { id: reviewId } = req.params
  const { rating, title, comment } = req.body
  const review = await Review.findOne({ _id: reviewId })
  if (!review) {
    throw new CustomError.NotFoundError(`No review found with id ${reviewId}`)
  }
  checkPermissions(req.user, review.user)
  // since we are setting all 3 props here, even if one is missing we get an error.
  // Optionally, if u want to set any one, you can also do that using if statemets
  // if title is present in req.body then set that, if comment is present then set that and so on
  review.title = title
  review.rating = rating
  review.comment = comment
  await review.save()
  res.status(StatusCodes.OK).json({ review })
}

Step 8 (Additional Capabilities to get data - Goodies)

We are now in a good shape but mongoose provides some more important features. Let's start with populate method. If we need to reference documents in other collection then we use populate method https://mongoosejs.com/docs/populate.html

If you remember, we have already used this several times to reference user in Product model, and also reference user and product in Review model. Let's explore more about this.

8.1 Populate()

Now let's say I am requesting for reviews as shown below. If I want to see my product details for which this review is associated, I currently only see product (product ID highlighted below). Wouldn't it be nice to see more details about the product like name, category and so on?

get all reviews

We need to use populate method for this. When to use populate?

When another model(product) is associated with this model(review), then we can use populate to populate more info.

const getAllReviews = async (req, res) => {
  const reviews = await Review.find({}).populate('product') // this will give all props of producct
  res.status(StatusCodes.OK).json({ reviews, count: reviews.length })
}

The above code will give all props of product

populate product in review

If you want only certain fields to be populated

const getAllReviews = async (req, res) => {
  const reviews = await Review.find({}).populate({
    path: 'product',
    select: 'name company price',
  })
  res.status(StatusCodes.OK).json({ reviews, count: reviews.length })
}

populate specific fields

We can do it for user as well.

const getAllReviews = async (req, res) => {
  const reviews = await Review.find({})
    .populate({
      path: 'product',
      select: 'name company price',
    })
    .populate({
      path: 'user',
      select: 'name',
    })
  res.status(StatusCodes.OK).json({ reviews, count: reviews.length })
}

populate user and product

populate visual representation

8.2 Mongoose Virtuals

Ok, now that we got to know about populate method, let's talk about this use case where, while getting a single product like below, I also want to get all the reviews associated with that product.

Note that previously, with populate method, we just expanded review to have all other product props. Before using populate we just had product ID in review and later after using populate we had other product props along with product ID.

Here, now we are talking what if we need all reviews associated with product. If you take a look, there's no review or reviews on our product model (but we had product on review model because of which populate was possible)

I cannot do below as product model doesn't have review model as it's field inside product model

const getAllProducts = async (req, res) => {
  const products = await Product.find({}).populate('review') // NOT POSSIBLE
  res.status(StatusCodes.OK).json({ products, count: products.length })
}

But after creating a virtual the above code is possible. Let's see how

In order to accomplish this we need to use mongoose virtuals. Mongoose virtuals are used to get properties that don't persist or exist on db and they only exist logically which needs to be generated on the fly

product can now accept virtuals

Refer here to docs attached below - Go to Populate section in below page. Same as what John says

Mongoose v6.5.4: Mongoose Tutorials: Mongoose Virtuals

Virtuals Tutorials in docs

The virtual will now work after above code is modified

controllers/ProductController.js

const getSingleProduct = async (req, res) => {
  const { id: productId } = req.params

  // populate will now work as we have added virtuals - true to product model
  const product = await Product.findOne({ _id: productId }).populate('reviews')
  if (!product) {
    throw new CustomErrors.NotFoundError(`No product with ID ${productId}`)
  }
  res.status(StatusCodes.OK).json({ product })
}

model/Product.js

const mongoose = require('mongoose')
const ProductSchema = new mongoose.Schema(
  {
    name: {
      type: String,
      trim: true,
      required: [true, 'Please provide product name'],
      maxLength: [100, 'Name cannot be more than 100 characters'],
    },
    price: {
      type: Number,
      required: [true, 'Please provide price'],
      default: 0,
    },
    description: {
      type: String,
      required: [true, 'Please provide product descritption'],
      maxLength: [1000, 'Description cannot be more than 1000 characters'],
    },
    image: {
      type: String,
      default: '/uploads/example.jpeg',
    },
    category: {
      type: String,
      required: [true, 'Please provide product category'],
      enum: ['office', 'kitchen', 'bedroom'],
    },
    company: {
      type: String,
      required: [true, 'Please provide company'],
      enum: {
        values: ['ikea', 'liddy', 'marcos'],
        message: '{VALUE} is not supported',
      },
    },
    // we will see how this array of String works when we send data from postman
    colors: {
      type: [String],
      required: true,
    },
    featured: {
      type: Boolean,
      default: false,
    },
    freeShipping: {
      type: Boolean,
      default: false,
    },
    inventory: {
      type: Number,
      required: true,
      default: 15,
    },
    // we will calculate this rating as we build up
    averageRating: {
      type: Number,
      default: 0,
    },
    user: {
      type: mongoose.Types.ObjectId,
      ref: 'User',
      required: [true, 'Please provide a user'],
    },
  },
  // setting virtuals here to true. Meaning, product model will now accept virtuals
  { timestamps: true, toJSON: { virtuals: true }, toObject: { virtuals: true } }
)

// using the name 'reviews' as we used the same name in populate() in getSingleProduct controller
ProductSchema.virtual('reviews', {
  ref: 'Review',
  localField: '_id', // this is the Product id
  foreignField: 'product', // field in the Review that reference Product Model
  justOne: false, // want more documents of reviews not just one document
})

module.exports = mongoose.model('Product', ProductSchema)

now getting all reviews for single product

We can also match certain documents. Let's say I want to get reviews only whose rating is 3 in single product

// using the name 'reviews' as we used the same name in populate() in getSingleProduct controller
ProductSchema.virtual('reviews', {
  ref: 'Review',
  localField: '_id', // this is the Product id
  foreignField: 'product', // field in the Review that reference Product Model
  justOne: false, // want more documents of reviews not just one document
  match: { rating: 3 }, // ADDED THIS MATCH HERE
})

getting review only whose rating in 3

This is a virtual property that we added. Meaning, this is not a real property which we can query for certain things. No! We can't query on review virtual property on product. We get all properties in review (virtual prop) by default. Hence, let's look at an alternative approach we can take to get reviews of a single product.

8.3 Virtuals Alternative approach

Let's see alternative way of how to implement reviews to appear in products without virtuals.

Steps

• First we will implement a controller in reviews called getSingleProductReviews

• Then we will import it into products route

Note that you can also implement this controller in Product to get all reviews, but we are going to implement this in review controller and import into product routes because, in the review controller we already have access to review model. Not a big deal, we can actually import Review model into product controller as well, but I prefer this way in implementing in Review Controller.

reviewController - getSingleProductReviews

routes/productRoute

single product reviews

Step 9 (Remove associations)

Like I said before that we will see later why we use save and remove methods instead of findOneAndUpdate() and findOneAndDelete(), now is the time we will see why actually we use save() and remove()

Ok, let's see this use case. If we remove a product, the reviews associated with that product should automatically be removed which makes sense. But think about it, in our current implementation that's not the case. If we remove a product, the reviews related to that product are not removed.

So if we have a post method that runs after removing the product, that post method should remove all the reviews associated with that product. We can have such post method that we can run after remove and save methods. That's not possible with findOneAndUpdate and findOneAndDelete method. That is the reason why we use save and remove.

models/Product.js partial

ProductSchema.post('remove', async function () {
  // Even though this is product schema, I can access other model (like Review)
  await this.model('Review').deleteMany({ product: this._id }) // product is the prop on Review, so we delete that

  // the above means, Remove all reviews associated with product having _id
})

models/Product.js

const mongoose = require('mongoose')
const ProductSchema = new mongoose.Schema(
  {
    name: {
      type: String,
      trim: true,
      required: [true, 'Please provide product name'],
      maxLength: [100, 'Name cannot be more than 100 characters'],
    },
    price: {
      type: Number,
      required: [true, 'Please provide price'],
      default: 0,
    },
    description: {
      type: String,
      required: [true, 'Please provide product descritption'],
      maxLength: [1000, 'Description cannot be more than 1000 characters'],
    },
    image: {
      type: String,
      default: '/uploads/example.jpeg',
    },
    category: {
      type: String,
      required: [true, 'Please provide product category'],
      enum: ['office', 'kitchen', 'bedroom'],
    },
    company: {
      type: String,
      required: [true, 'Please provide company'],
      enum: {
        values: ['ikea', 'liddy', 'marcos'],
        message: '{VALUE} is not supported',
      },
    },
    // we will see how this array of String works when we send data from postman
    colors: {
      type: [String],
      required: true,
    },
    featured: {
      type: Boolean,
      default: false,
    },
    freeShipping: {
      type: Boolean,
      default: false,
    },
    inventory: {
      type: Number,
      required: true,
      default: 15,
    },
    // we will calculate this rating and num of reviews as we build up
    averageRating: {
      type: Number,
      default: 0,
    },
    numOfReviews: {
      type: Number,
      default: 0,
    },
    user: {
      type: mongoose.Types.ObjectId,
      ref: 'User',
      required: [true, 'Please provide a user'],
    },
  },
  // setting virtuals here to true. Meaning, product model will now accept virtuals
  { timestamps: true, toJSON: { virtuals: true }, toObject: { virtuals: true } }
)

// using the name 'reviews' as we used the same name in populate() in getSingleProduct controller
// ProductSchema.virtual('reviews', {
//   ref: 'Review',
//   localField: '_id', // this is the Product id
//   foreignField: 'product', // field in the Review that reference Product Model
//   justOne: false, // want more documents of reviews not just one document
//   match: { rating: 3 },
// })

ProductSchema.post('remove', async function () {
  // Even though this is product schema, I can access other model (like Review)
  await this.model('Review').deleteMany({ product: this._id }) // product is the prop on Review, so we delete that

  // the above means, Remove all reviews associated with product having _id
})

module.exports = mongoose.model('Product', ProductSchema)

To test this

• Create a product

• Create some reviews associated with that product

• Remove that product

• Check the reviews associated with that product and you should not see any reviews

Step 10 (Average rating and number of Reviews on Product)

Ok, it's time to calculate Average Rating and Number of Reviews on a product. Review is a different Model which is linked to product (review has product), but product model has no association with review directly. Even then we need to know how many reviews (numOfReviews) a product has and what is the average rating (rating is a prop on review) for a product. Average rating is all ratings combined for a product / number of reviews for that particular product. In this case we use mongodb aggregation.

The idea is, we need to calculate this numOfReviews on a product when some review changes on the DB (when any review gets updated or created). So basically, when a new review is created for a product or existing review is updated or review is deleted then we need to calculate the numOfReviews on a product for which this review is associated. We can use post save and post remove hook for this and this is the reason why we chose save() over findOneAndUpdate() and remove over findOneAndRemove()

Static Method on Schema

While we are on implementing average rating and number of reviews, let's also discuss about Schema static methods. Remember we implemented instance method for comparing the password on user document #3.7-compare-passwords. This was an instance method, meaning, every user document was able to access comparePassword method that was written on Schema. UserSchema.methods.comparePassword is what we had done so that all user documents (instances of User) had access to this comparePassword method.

Unlike instance method, what if we want to access a method directly on Schema rather than accessing on instance? For example, let's say I don't want user.comparePassword() but I want UserSchema.comparePassword(), then I should setup static method on Schema which we will do in this case on ReviewSchema.

models/Review.js partial

// creating a static method (review instance can't access this but we can access directly access it on ReviewSchema)
ReviewSchema.statics.calculateAverageRating = async function (productId) {
  // Here, I need to calculate average rating on this product id.
  // I need to take all the reviews associated with this product id, and then get average of them and
  // then attach it to this product (of productId given).
  console.log(productId)
}

average rating yet to be calculated

Now to test this setup, create a review/update that review and delete that review. You should see the console log for that review on that product passed as param.

10.1 Aggregation pipeline

Now that we have setup post save and post remove hooks and what should happen in this hook (calculate average rating and number of reviews both), let's work on calculateAverageRating static method. Even though the method says calculateAverageRating we will calculate both average rating and numOfRaviews by setting aggregation pipeline.

First we will setup this aggregation pipeline in mongodb atlas interface and then get an idea how to do it through code.

Idea of aggregation

So what are we going to do here? What we want to achieve here? We want two things

1. Calculate number of reviews

Here, we can do this without even using aggregate pipeline. Aggregate pipeline is for doing some steps like matching certain queries, grouping them and then returning it to wherever it needs to go. This is all done on mongodb side and not in express JS in our code. In our example, if we need number of reviews, we can do it the other way in our code. We can actually get all the reviews for a specific product and calculate the length of that and attach it to specific product, this is one option. But here we are going to do that using aggregate pipeline on mongodb side itself. And I am assuming this would be faster. Let's look at the next case, calculate average rating, which makes more sense

2. Calculate average rating

This is also something which we can do it on express side. Get all reviews, iterate them one by one and store the rating in a variable and then calculate the average and attach it to specific product. But think about it, if we have a million or a billion reviews, we need to get all of them (which takes a lot of time and processing) and then iterate through them (which takes some more time) and then separate out according to product and then attach it to product. For an example and better understanding, consider udemy website which has got a million courses and each course has got a million reviews. So total reviews would be million * million.

For this reason it would be nice to do it on mongodb side. On mongodb, let's say we need to know how many 1 rating, 2 ratings, 3 ratings, 4 ratings, and 5 ratings for a course. Then we need to pick that course (match that course), group by rating prop so we get an array of each rating and then we can attach an amount prop that keeps count of number of separate ratings. This is what we are going to do now. We won't group by separate ratings, but all we do it calculate number of reviews for a product and it's average rating. So we are not grouping by anything here, so we can define group id to be null. Group id defines what we are grouping on like rating for example.

Two steps here

• First we need to go to Atlas account (Mongodb), choose our project E-commerce, go to review collection (because we are going to aggregate here), and choose Aggregation

Aggregation playground

matching and grouping our docs

• Once we do it here and get all documents we are looking for then we can copy the code by clicking here

export the aggregation to code

Export to node

10.2 MongoDB aggregation

Ok, now you know what is aggregation and how we do it, let's jump back to Atlas UI and do the matching and grouping to get caculateAverageRating and numOfReviews. Then in next step 10.3, we will write this in code.

aggregation to get avg rating and num of reviews

Let's export this code

export code

Let's copy this code and put it in temp file. This is what we want in our code, but let's write from scratch

model/temp.js

[
  {
    '$match': {
      'product': new ObjectId('63156f49d6daad5d68142a3b')
    }
  }, {
    '$group': {
      '_id': null, 
      'averageRating': {
        '$avg': '$rating'
      }, 
      'numOfReviews': {
        '$sum': 1
      }
    }
  }
]

10.3 MongoDB aggregation in code

Now that we have setup our aggregation in GUI and have the code snippet for aggregation, let's see how we can add this code in ReviewSchema.statics.calculateAverageRating()function to get average rating and number of reviews for passed product and then attach it to product model dynamically.

reviews aggregation code console logging

Above, we just added the code (temp file shown on right side is the actual code snippet we got from atlas in our previous step). We are adding a console log to see the average rating and number of reviews. We tested it by updating an existing review. This will log even if we create or delete the review.

Let's now attach this result on to our product. So every time someone creates or updates or deletes a review for this product, then we dynamically calculate them and add it to that product again.

code snapshot for aggregation to get averageRating and numOfReviews for this product

models/Review.js

const mongoose = require('mongoose')
const ReviewSchema = new mongoose.Schema(
  {
    rating: {
      type: Number,
      min: 1,
      max: 5,
      required: [true, 'Please provide a rating'],
    },
    title: {
      type: String,
      trim: true,
      required: [true, 'Please provide review title'],
      maxLength: 100,
    },
    comment: {
      type: String,
      required: [true, 'Please provide review text'],
    },
    user: {
      type: mongoose.Types.ObjectId,
      ref: 'User',
      required: [true, 'Please provide a user'],
    },
    product: {
      type: mongoose.Types.ObjectId,
      ref: 'Product',
      required: [true, 'Please provide a product'],
    },
  },
  { timestamps: true }
)

// one user - one review per one product
ReviewSchema.index({ product: 1, user: 1 }, { unique: true })

// creating a static method (review instance can't access this but we can access directly access it on ReviewSchema)
ReviewSchema.statics.calculateAverageRating = async function (productId) {
  // Here, I need to calculate average rating on this product id.
  // I need to take all the reviews associated with this product id, and then get average of them and
  // then attach it to this product (of productId given).
  const result = await this.aggregate([
    { $match: { product: productId } },
    {
      $group: {
        _id: null,
        averageRating: { $avg: '$rating' },
        numOfReviews: { $sum: 1 },
      },
    },
  ])
  console.log('The result is', result) // [ { _id: null, averageRating: 2.5, numOfReviews: 2 } ]
  // notice that the result is an array
  // if we delete all reviews for this product then this array will just be empty, so we need to check if array has atleast one element
  // we will do that through optional chaining

  try {
    // All we are doing here is - go to product and add average rating and numOfReviews
    // If array is empty (when no reviews or all reviews get deleted), then set averageRating and numOfReviews to 0
    await this.model('Product').findOneAndUpdate(
      { _id: productId },
      {
        averageRating: Math.ceil(result[0]?.averageRating || 0),
        numOfReviews: result[0]?.numOfReviews || 0,
      }
    )
  } catch (error) {
    console.log(error)
  }
}

// when a review is modified (created or updated or deleted) we need to calculate number of reviews (numOfReviews)
// for a product with which this review is associated and also calculate avg rating. So we use post save and remove hooks

ReviewSchema.post('save', async function () {
  console.log(
    'Adding or updating the review.... Average rating is calculating for the product...'
  )
  await this.constructor.calculateAverageRating(this.product) // this is how we access static method
})
ReviewSchema.post('remove', async function () {
  console.log(
    'Deleting the review.... Average rating is calculating for the product...'
  )
  await this.constructor.calculateAverageRating(this.product) // this is how we access static method
})

module.exports = mongoose.model('Review', ReviewSchema)

Step 11 (Orders)

We are almost there and this is going to be our last model. To understand this order, let's look at John's comfy sloth store project here https://react-course-comfy-sloth-store.netlify.app/cart. When it comes to order, a big part is cart (The items user wants to buy). So we have got two options here

• Option 1 - We have got all the cart data in front-end and is persisted in the local storage (like in comfy sloth project). We will cover this option in our project now. We will assume there's a front-end that is sending us all the cart info (in our case all the cart info will be sent by postman ofcourse).

• Option 2 - We can store the cart data directly in the database. In the Yelp clone project (future project in John's course - watch out I might do it below), we will setup cart data in the database and we can then see how to do it with express sessions.

Cart

Checkout the cart

So once the user is ready with all the products and click checkout we will do two things

• First, we will communicate with stripe (to get the client secret key)

• Second, we will setup the initial order. Before we even communicate with stripe, we want to double check whether the data that is coming from the front-end / postman actually makes sense (if the ID exists for the product and if it does then what is the correct price). Then we will communicate with stripe (first point) and then we will setup the order. Once the payment is complete then we will get additional data that we will add it to our order.

11.1 Order Model

Steps

Order Schema

• [] create Order.js in models folder

• [] create Schema

• [] tax : {type:Number}

• [] shippingFee: {type:Number}

• [] subtotal: {type:Number}

• [] total: {type:Number}

• [] orderItems:[]

• [] status:{type:String}

• [] user

• [] clientSecret:{type:String}

• [] paymentId:{type:String}

• [] set timestamps

• [] export Order model

Note that for cartItems in the OrderSchema, we have setup as a separate schema. We can technically set this up directly as a simple object in cartItems prop in OrderSchema, but this is the clean way to setup as we will get the validation for the cart item as well.

model/Order.js

const mongoose = require('mongoose')

// we can technically set this up directly as a simple object 
// in cartItems prop in OrderSchema, but this is the clean way 
// to setup as we will get the validation for the cart item as well

const SingleCartItemSchema = mongoose.Schema({
  name: {
    type: String,
    required: true,
  },
  image: {
    type: String,
    required: true,
  },
  price: {
    type: Number,
    required: true,
  },
  // amount is the quantity of this product
  amount: {
    type: Number,
    required: true,
  },
  product: {
    type: mongoose.Types.ObjectId,
    ref: 'Product',
    required: true,
  },
})

const OrderSchema = new mongoose.Schema(
  {
    tax: {
      type: Number,
      required: true,
    },
    shippingFee: {
      type: Number,
      required: true,
    },
    // this is the total for all cart items where for each item, 
    // we multiply price by quantity
    subtotal: {
      type: Number,
      required: true,
    },
    // total = subtotal + tax + shippingFee
    total: {
      type: Number,
      required: true,
    },
    // we can techincally set this up directly as a simple object 
    // in cartItems here, but this is the clean way to setup as a 
    // different schema as we will get the validation for SingleCartItemSchema as well

    cartItems: [SingleCartItemSchema],
    status: {
      type: String,
      enum: ['pending', 'failed', 'paid', 'delivered', 'cancelled'],
      default: 'pending',
    },
    user: {
      type: mongoose.Types.ObjectId,
      ref: 'User',
      required: [true, 'Please provide a user'],
    },
    clientSecret: {
      type: String,
      required: true,
    },
    paymentIntentId: {
      type: String,
    },
  },
  { timestamps: true }
)

const Order = mongoose.model('Order', OrderSchema)

module.exports = Order

/*

The above two lines is same as
module.exports = mongoose.model('Order', OrderSchema)

*/

11.2 Order Structure

Steps

• [] add orderController file in controllers

• [] export (getAllOrders, getSingleOrder, getCurrentUserOrders, createOrder, updateOrder) functions

• [] res.send('function name')

• [] setup orderRoutes file in routes

• [] import all controllers

• [] authenticate user in all routes

• [] getAllOrders admin only

• [] typical REST setup

• [] router.route('/showAllMyOrders').get(getCurrentUserOrders)

• [] import orderRoutes as orderRouter in the app.js

• [] setup app.use('/api/v1/orders', orderRouter)

controllers/orderController.js

const createOrder = async (req, res) => {
  res.send('Create Order')
}

const getAllOrders = async (req, res) => {
  res.send('Get All Orders')
}

const getSingleOrder = async (req, res) => {
  res.send('Get Single Order')
}

const updateOrder = async (req, res) => {
  res.send('Update Order')
}
const getCurrentUserOrders = async (req, res) => {
  res.send('Get current ussers Orders')
}

module.exports = {
  getAllOrders,
  getSingleOrder,
  getCurrentUserOrders,
  createOrder,
  updateOrder,
}

routes/orderRoute.js

const express = require('express')
const router = express.Router()

const {
  getAllOrders,
  getSingleOrder,
  getCurrentUserOrders,
  createOrder,
  updateOrder,
} = require('../controllers/OrderController')

const {
  authenticateUser,
  authorizePermissions,
} = require('../middleware/authentication')

router
  .route('/')
  .get(authenticateUser, authorizePermissions(['admin']), getAllOrders)
  .post(authenticateUser, createOrder)

router.route('/showAllMyOrders').get(authenticateUser, getCurrentUserOrders)

router
  .route('/:id')
  .patch(authenticateUser, updateOrder)
  .get(authenticateUser, getSingleOrder)

module.exports = router

app.js

const express = require('express')
const app = express()

// rest of the packages
const morgan = require('morgan')
const cookieParser = require('cookie-parser')
const connectDB = require('./db/connect')
require('dotenv').config()
require('express-async-errors') // for avoiding writing try-catch in controllers
const notFoundMW = require('./middleware/not-found')
const errorHandlerMW = require('./middleware/error-handler')
const authRouter = require('./routes/authRoute')
const userRouter = require('./routes/userRoute')
const productRouter = require('./routes/productRoute')
const reviewRouter = require('./routes/reviewRoute')
const orderRouter = require('./routes/orderRoute')
const fileUpload = require('express-fileupload')

//// Middlewares and Routes

app.use(morgan('tiny'))
app.use(express.json()) // used to get req.body data for post reqs
app.use(cookieParser(process.env.JWT_SECRET)) // used to parse the cookies sent from the client(front-end) or postman

// enable public folder to be publicly available
app.use(express.static('./public'))
app.use(fileUpload())
// Routes

// Basic Route
app.get('/', (req, res) => {
  console.log(req.signedCookies) // this is avaiable because of cookie-parser package
  res.send('E-Commerce API Home page')
})

// auth router
app.use('/api/v1/auth', authRouter)

// user router
app.use('/api/v1/users', userRouter)

// product router
app.use('/api/v1/products', productRouter)

// review router
app.use('/api/v1/reviews', reviewRouter)

// order router
app.use('/api/v1/orders', orderRouter)

app.use(notFoundMW)
app.use(errorHandlerMW)

const port = process.env.PORT || 5000

const start = async () => {
  try {
    // connect to db
    await connectDB(process.env.MONGO_URL)
    app.listen(port)
    console.log('Server is listening on port', port)
  } catch (err) {
    console.log(err)
  }
}

start()

11.3 Postman setup for Orders

Let's setup these routes in postman now

• Get All Orders (Admin route)

• All below routes must be authenticated

  • Create order

  • Update order

  • Get Current user's order

  • Get Single order

Order routes in postman

11.4 Order CRUD

Let's now do some CRUD operations on Order

11.4.1 Create Order

I would suggest to go read stripe project before proceeding with this task --->#5.-stripe-api-project Before we set this up, let's see what to expect from the front-end. We will be expecting to get

• the shipping fee,

• tax

• and also the cart items array.

  • In cart items array we will have bunch of objects (each object represents a product). So each object will have a product-name, price, product-image, quantity, product id (very important)

So before we setup any functionality let's setup proper request in the postman

Setup Create Order in postman

You can use mock data John has prepared. It will be in starter/mockData/products.json file. Make sure you create one or two products and when you create order two things must match - product ID and the amount (quantity of each product) in the array.

creating an order in postman

Let's first get the req.body and destructure items, shippingFee and tax, and validate them.

controllers/orderController.js partial

const Order = require('../models/Order')
const Product = require('../models/Product')
const { StatusCodes } = require('http-status-codes')
const CustomError = require('../errors')
const checkPermissions = require('../utils/checkPermissions')

const createOrder = async (req, res) => {
  const { items: cartItems, tax, shippingFee } = req.body
  if (!cartItems || cartItems.length < 1) {
    throw new CustomError.BadRequestError('No cart items provided')
  }

  if (!tax || !shippingFee) {
    throw new CustomError.BadRequestError('Please provide tax and shipping fee')
  }

  res.send('Create Order')
}

Once this works fine then we can start testing the products. Now, this product is an array like this in the request

"items": [
      {
        "name": "product 1",
        "price": 2599,
        "image": "https://dl.airtable.com/.attachmentThumbnails/e8bc3791196535af65f40e36993b9e1f/438bd160",
        "amount": 34,
        "product": "6316666d1c1538a1ad5f5c58"
      },
      
      {
        "name": "product 2",
        "price": 2599,
        "image": "https://dl.airtable.com/.attachmentThumbnails/e8bc3791196535af65f40e36993b9e1f/438bd160",
        "amount": 34,
        "product": "6316666d1c1538a1ad5f5c57"
      },
      
]

Notice that we have two products in cart. The idea in the create order controller is to loop through this items array and get each product ID, get that product from the db and get the price of it and keep adding cost of each product to a variable.

Then finally check if total amount, shipping fee and tax matches to what front-end sent us and if yes then contact stripe and create a client secret (we will fake this here as we don't have a front-end and actually front-end must exist to test stripe as front-end needs to communicate with stripe after getting client secret from backend, please refer to stripe project) and send back to front-end.

Now, let's step back a second and see how to loop through cart items. We need to use for of loop as, inside for of loop, we will be using await to get each product from db. forEach or map wouldn't work with await.

See the below image to know when should these checks happen that is present in the create order controller

Proceed to checkout

Once we get back clientSecret then we see this stripe payment page

Creating an order in the controller covering all above concepts

controller/orderController.js partial

const Order = require('../models/Order')
const Product = require('../models/Product')
const { StatusCodes } = require('http-status-codes')
const { BadRequestError, NotFoundError } = require('../errors')

const fakeStripeAPI = async ({ amount, currency }) => {
  const client_secret = 'someRandomValue'
  return { client_secret, amount } // mimicing the response of real stripe function
}

const createOrder = async (req, res) => {
  const { items: cartItems, tax, shippingFee } = req.body
  if (!cartItems || cartItems.length < 1) {
    throw new BadRequestError('No cart items provided')
  }

  if (!tax || !shippingFee) {
    throw new BadRequestError('Please provide tax and shipping fee')
  }

  let orderItems = [] // used below to create the actual Order after stipe payment
  let subtotal = 0 // price * qty for each product

  for (let item of cartItems) {
    const dbProduct = await Product.findOne({ _id: item.product })
    if (!dbProduct) {
      throw new NotFoundError(`No product with id ${item.product}`)
    }

    const { name, image, price, _id } = dbProduct // no need of _id here as it is same as item.product,
    // but that's ok lets have it here, no harm

    const singleOrderItem = {
      name,
      price,
      image,
      amount: item.amount, // this is coming from front-end
      product: _id,
    }

    // add this singleOrderItem to orderItems array
    orderItems = [...orderItems, singleOrderItem]

    // I also need to calculate subtotal of my cart
    subtotal += item.amount * price // note that we are choosing price (from db, not from front-end) and amount (from front-end)
  }

  // once we have orderItems and subtotal calcualated on the items fetched
  // from db, we can setup stripe (fake stripe in this case) to
  // get the client_secret and send it to front-end

  const total = subtotal + tax + shippingFee

  // get client secret (from stripe, but fake stripe in this case)
  // we will setup fakeStripeAPI function to mimic real stripe function
  const paymentIntent = await fakeStripeAPI({
    amount: total,
    currency: 'usd',
  })

  // look at the Order schema for all required values
  const order = await Order.create({
    cartItems: orderItems,
    total,
    subtotal,
    tax,
    shippingFee,
    clientSecret: paymentIntent.client_secret,
    user: req.user.userId,
  })

  res
    .status(StatusCodes.CREATED)
    .json({ order, clientSecret: order.clientSecret })
}

11.4.2 Get All Orders

This is an admin only route.

controller/orderController.js partial

const getAllOrders = async (req, res) => {
  const orders = await Order.find({})
  res.status(StatusCodes.OK).json({ orders })
}

11.4.3 Get Single Order

You need to check permissions here as each user should access his/her own order only and not others' orders.

controller/orderController.js partial

const getSingleOrder = async (req, res) => {
  const { id: orderId } = req.params
  const order = await Order.findOne({ _id: orderId })
  if (!order) {
    throw new NotFoundError(`No order with id ${orderId}`)
  }
  
  // note: Don't populate above while getting order, else order.user will have 
  // addtional field attached from populate and bleow checkPermissions will fail
  // I learnt this after some trial and error, so don't fall into this trap again. 
  // Removed populate in this controller here so don't worry
  checkPermissions(req.user, order.user)
  res.status(StatusCodes.OK).json({ order })
}

11.4.4 Get Current User Orders

Get only the orders of the currently logged in user

controller/orderController.js partial

const getCurrentUserOrders = async (req, res) => {
  // populating here to check who the user is, for ease
  const orders = await Order.find({ user: req.user.userId }).populate({
    path: 'user',
    select: 'name',
  })
  console.log('req user userid', req.user.userId)
  res.status(StatusCodes.OK).json({ orders })
}

11.4.5 Update Order

• Once user adds items to cart and clicks Proceed to Checkout then we hit create order controller.

• After creating the order (the status of order will be pending), we will send back clientSecret from back-end to front-end.

• Once front-end has client secret he can proceed to below shown page where he can enter his card details and make payment to stripe

• Once payment is made to stripe, stripe will send back payment Intent id to front-end

• Front-end then calls update order and sends us this payment intent id that stripe sent

• Now we are in update controller. We need to store this payment intent id for this order and update the order status from 'pending' to 'paid'

once payment is made we get back payment intent id from stripe

Steps to update order

• [] get order id

• [] get paymentIntentId (req.body)

• [] get order

• [] if does not exist - 404

• [] check permissions

• [] set paymentIntentId and status as 'paid'

• [] order.save()

controller/orderController.js partial

const updateOrder = async (req, res) => {
  const { id: orderId } = req.params
  const { paymentIntentId } = req.body
  if (!paymentIntentId) {
    throw new BadRequestError('Please provide payment intenet ID')
  }
  const order = await Order.findOne({ _id: orderId })
  if (!order) {
    throw new NotFoundError(`No order with id ${orderId}`)
  }
  // we don't want Susan to update Peter's order so checking permissions
  checkPermissions(req.user, order.user)

  // save payment intenet id and also the status
  order.paymentIntentId = paymentIntentId
  order.status = 'paid'

  await order.save()

  res.status(StatusCodes.OK).json({ order })
}

order updated successfully

Order controller full code

controller/OrderController.js

const Order = require('../models/Order')
const Product = require('../models/Product')
const { StatusCodes } = require('http-status-codes')
const { BadRequestError, NotFoundError } = require('../errors')
const { checkPermissions } = require('../utils')

const fakeStripeAPI = async ({ amount, currency }) => {
  const client_secret = 'someRandomValue'
  return { client_secret, amount } // mimicing the response of real stripe function
}

const createOrder = async (req, res) => {
  const { items: cartItems, tax, shippingFee } = req.body
  if (!cartItems || cartItems.length < 1) {
    throw new BadRequestError('No cart items provided')
  }

  if (!tax || !shippingFee) {
    throw new BadRequestError('Please provide tax and shipping fee')
  }

  let orderItems = [] // used below to create the actual Order after stipe payment
  let subtotal = 0 // price * qty for each product

  for (let item of cartItems) {
    const dbProduct = await Product.findOne({ _id: item.product })
    if (!dbProduct) {
      throw new NotFoundError(`No product with id ${item.product}`)
    }

    const { name, image, price, _id } = dbProduct // no need of _id here as it is same as item.product,
    // but that's ok lets have it here, no harm

    const singleOrderItem = {
      name,
      price,
      image,
      amount: item.amount, // this is coming from front-end
      product: _id,
    }

    // add this singleOrderItem to orderItems array
    orderItems = [...orderItems, singleOrderItem]

    // I also need to calculate subtotal of my cart
    subtotal += item.amount * price // note that we are choosing price (from db, not from front-end) and amount (from front-end)
  }

  // once we have orderItems and subtotal calcualated on the items fetched
  // from db, we can setup stripe (fake stripe in this case) to
  // get the client_secret and send it to front-end

  const total = subtotal + tax + shippingFee

  // get client secret (from stripe, but fake stripe in this case)
  // we will setup fakeStripeAPI function to mimic real stripe function
  const paymentIntent = await fakeStripeAPI({
    amount: total,
    currency: 'usd',
  })

  // look at the Order schema for all required values
  const order = await Order.create({
    cartItems: orderItems,
    total,
    subtotal,
    tax,
    shippingFee,
    clientSecret: paymentIntent.client_secret,
    user: req.user.userId,
  })

  res
    .status(StatusCodes.CREATED)
    .json({ order, clientSecret: order.clientSecret })
}

const getAllOrders = async (req, res) => {
  const orders = await Order.find({}).populate({ path: 'user', select: 'name' })
  res.status(StatusCodes.OK).json({ orders })
}

const getSingleOrder = async (req, res) => {
  const { id: orderId } = req.params
  const order = await Order.findOne({ _id: orderId })
  if (!order) {
    throw new NotFoundError(`No order with id ${orderId}`)
  }
  checkPermissions(req.user, order.user)
  res.status(StatusCodes.OK).json({ order })
}

const updateOrder = async (req, res) => {
  const { id: orderId } = req.params
  const { paymentIntentId } = req.body
  if (!paymentIntentId) {
    throw new BadRequestError('Please provide payment intenet ID')
  }
  const order = await Order.findOne({ _id: orderId })
  if (!order) {
    throw new NotFoundError(`No order with id ${orderId}`)
  }
  // we don't want Susan to update Peter's order so checking permissions
  checkPermissions(req.user, order.user)

  // save payment intenet id and also the status
  order.paymentIntentId = paymentIntentId
  order.status = 'paid'

  await order.save()

  res.status(StatusCodes.OK).json({ order })
}
const getCurrentUserOrders = async (req, res) => {
  // populating here to check who the user is, for ease
  const orders = await Order.find({ user: req.user.userId }).populate({
    path: 'user',
    select: 'name',
  })
  res.status(StatusCodes.OK).json({ orders })
}

module.exports = {
  getAllOrders,
  getSingleOrder,
  getCurrentUserOrders,
  createOrder,
  updateOrder,
}

Step 12 (Documentation)

Finally we are here after lot of hard work. Instead of swagger UI which we used in jobs api for documentation, let's use a better one here which is docgen

GitHub - thedevsaddam/docgen: Transform your postman collection to HTML/Markdown documentationGitHub

docgen

To install this, do as shown in documentation.

curl https://raw.githubusercontent.com/thedevsaddam/docgen/v3/install.sh -o install.sh \
&& sudo chmod +x install.sh \
&& sudo ./install.sh \
&& rm install.sh

12.1 Create documentation

Steps

• [] Export Postman Collection

• [] Replace the URL to localhost:5000 or whatever port it's running on(later to actual URL once deployed to heroku

• [] docgen build -i fileName.json -o index.html

• [] create index.html in public

• In postman, click on export and save the collection in a folder anywhere

export docs

• Open that saved file in vscode and change the {{URL}} variable to localhost:5000 (what ever app your port is running on). Later we will change this again to the actual URL once deployed to heroku.

• Run this command below so that your docs, filename.json, will be converted to html, filename.html

// docgen build -i <postman exported filename>.json -o index.html
docgen build -i docs.json -o index.html  

• This would generate docs.html, move this html file into public folder

• Note that since we have a line in our code to expose public folder as shown below, docs will now be available on http://localhost:5000/

app.js partial

// enable public folder to be publicly available
app.use(express.static('./public'))

• If you uncomment above line then the route / will be hit defined below and that will be run instead of docs as shown below

app.js partial

// Basic Route
app.get('/', (req, res) => {
  console.log(req.signedCookies) // this is avaiable because of cookie-parser package
  res.send('E-Commerce API Home page')
})

• Now we have our docs deployed to localhost:5000

our docs in localhost

Step 13 (Security Packages)

Since we will deploy our apps to heroku, it's good to consider some security packages to secure our app.

Steps

• [] express-rate-limiter - to limit requests made from each IP Address

• [] helmet - to set security response headers

• [] xss-clean - to sanitize user input

• [] express-mongo-sanitize - to protect against mongodb injection

• [] cors (cookies!!!!) - to allow access from different domains. Note that we are sending JWTs with cookies and remember they will only work if the front-end is on the same domain. To solve this issue we use cors

app.js

const express = require('express')
const app = express()

// rest of the packages
const morgan = require('morgan')
const cookieParser = require('cookie-parser')
const connectDB = require('./db/connect')
require('dotenv').config()
require('express-async-errors') // for avoiding writing try-catch in controllers
const notFoundMW = require('./middleware/not-found')
const errorHandlerMW = require('./middleware/error-handler')
const authRouter = require('./routes/authRoute')
const userRouter = require('./routes/userRoute')
const productRouter = require('./routes/productRoute')
const reviewRouter = require('./routes/reviewRoute')
const orderRouter = require('./routes/orderRoute')
const fileUpload = require('express-fileupload')

// Security packages
const rateLimiter = require('express-rate-limit')
const helmet = require('helmet')
const xss = require('xss-clean')
const cors = require('cors')
const expressMongoSanitize = require('express-mongo-sanitize')

//// Middlewares and Routes

// for rate limiter if it's behind the proxy then we need to set that as well
app.set('turst proxy', 1)
app.use(
  rateLimiter({
    windowsMs: 15 * 60 * 1000,
    max: 60,
  })
)
app.use(helmet())
app.use(xss())
app.use(cors())
app.use(expressMongoSanitize())

app.use(morgan('tiny'))
app.use(express.json()) // used to get req.body data for post reqs
app.use(cookieParser(process.env.JWT_SECRET)) // used to parse the cookies sent from the client(front-end) or postman

// enable public folder to be publicly available
app.use(express.static('./public'))
app.use(fileUpload())
// Routes

// Basic Route
app.get('/', (req, res) => {
  console.log(req.signedCookies) // this is avaiable because of cookie-parser package
  res.send('E-Commerce API Home page')
})

// auth router
app.use('/api/v1/auth', authRouter)

// user router
app.use('/api/v1/users', userRouter)

// product router
app.use('/api/v1/products', productRouter)

// review router
app.use('/api/v1/reviews', reviewRouter)

// order router
app.use('/api/v1/orders', orderRouter)

app.use(notFoundMW)
app.use(errorHandlerMW)

const port = process.env.PORT || 5000

const start = async () => {
  try {
    // connect to db
    await connectDB(process.env.MONGO_URL)
    app.listen(port)
    console.log('Server is listening on port', port)
  } catch (err) {
    console.log(err)
  }
}

start()

Step 14 (Deploy to Heroku)

For Heroku setup, credentials and CLI, refer to Jobs API here ---> Step 14 (Deploy - Heroku or any other cloud provider)

Steps

• [] heroku account and heroku cli

• [] remove/copy from the main repo

• [] add dev command "nodemon app.js"

• [] change start to "node app.js"

• [] setup node version in package.json

• [] "engines": {"node": "14.x"}

• [] Procfile "web: node app.js"

• [] remove existing git repo

• [] rm -rf .git - mac,

• [] git init

• [] git add .

• [] git commit -m "initial commit"

• [] heroku login

• [] heroku create "App Name"

• [] git remote -v

• [] setup env vars in GUI

• [] git push heroku master/main

• Move your starter folder (you worked on till now) to your desktop and rename folder to what you want

• Open this project with vscode

• Change the start script in package.json by replacing start:nodemon app.js to start:node app.js. Rename the old start to 'dev'. If you want to run this in dev from now, do npm run dev

package.json partial

"scripts": {
    "start": "node app.js",
    "dev": "nodemon app.js"
}

• Also add this to your package json

"engines": {
    "node": "14.x"
  }

• Create a file called Procfile in root and add this "web: node app.js" Make sure the spacing is correct here

Proc file

• Remove existing git repo

rm -rf .git

• Setup git

git init
git add -A
git commit -m "initial commit"
heroku login // press any key and they'll take us to login
heroku create "APP NAME" // heroku create "e-commerce-api-node-project" -> If you omit name they will give a random name

• Check if remote is pointing to heroku

git remote -v

• Check if our project is created on heroku

deployed on heroku

• Now click on this app on heroku and setup env variables

env variables

• Take your keys and values in .env and set them up one by one here. In the process of doing so I noticed that we are still using jwtSecret as string for JWT_SECRET. For this let's generate more complex string.

  • Go to https://www.allkeysgenerator.com/ and select Encryption key and click on 256-bit (keep clicking on this 256-bit to generate new keys) and copy this and paste it for JWT_SECRET.

  

  all key generator

• Now you have your configs on heroku

env vars on heroku

• Once this is done you can push the project to heroku

git push heroku master

• While this is building you can check the logs on

View logs and open app

• Once the Build is successful, open app by clicking the Open App button next to More as shown in the above image and it should open the app

• Cool 😎 our project is now deployed finally in https://e-commerce-api-node-project.herokuapp.com/

14.1 Final fixes

• Notice that when we click on any route, we get this error and will not be able to open any of the routes. This is caused because docgen is using inline javascript which is forbidden by helmet package. Let's see in a second how to solve this problem by doing other steps first.

helmet package causes this error

• Replace http://localhost:5000 with our app's URL https://e-commerce-api-node-project.herokuapp.com/api/v1 in docs.json and republish it

docgen build -i docs.json -o index.html

• Move the resulting index.html file to public folder again and replace the existing one

• Ok, now we have updated our index.html, let's solve this problem of javascript shown in above screenshot

  • Currently we have index.html which is the result of docgen command above. This is a minified file (so first we need to right click and format this file), we have embedded javascript within this file inside <script> tag. We need to separate this file, put it into another javascript file and then call that file in this index.html file like this <script src = "that-file.js"></script> that solves the issue

  • Restart the server and run npm run dev to check if everything is working fine

• Now once this issue is solved, finally re-publish the app with working docs

git add -A
git commit -m "added few fixes"
git push heroku master

The doc without errors is published now https://e-commerce-api-node-project.herokuapp.com/

• Now let's create a git repo for this project and push our code

rm -rf .git // remove all git configs
AND Create a repository  

Awesome 👏 😎 Finally completed the project, https://github.com/sandeep194920/E-Commerce-API-Node

✅ 7. Send Emails

In bigger projects we will have to send emails in several places. For example, after user registration, we might want to email user to verify email. So in this project we will see how to send the email.

Step 1 (General Project Structure)

/app.js

require('dotenv').config()
require('express-async-errors')

const express = require('express')
const app = express()
const sendEmail = require('./controllers/sendEmail')

// error handler
const notFoundMiddleware = require('./middleware/not-found')
const errorHandlerMiddleware = require('./middleware/error-handler')

app.use(express.json())

// routes
app.get('/', (req, res) => {
  res.send(`<h1>Email Project</h1> <a href='send'>Send Email</a>`)
})

app.get('/send', sendEmail)

app.use(notFoundMiddleware)
app.use(errorHandlerMiddleware)

const port = process.env.PORT || 3000

const start = async () => {
  try {
    app.listen(port, () =>
      console.log(`Server is listening on port ${port}...`)
    )
  } catch (error) {
    console.log(error)
  }
}

start()

controller/sendEmail.js

const sendEmail = async (req, res) => {
  res.send('Email controller')
}

module.exports = sendEmail

Step 2 (Email Sending Functionality)

2.1 Nodemailer package

We have a package to send emails in node and that is nodemailer. This is the most popular option for sending emails through node.

Nodemailer :: Nodemailer

Nodemailer

With nodemailer, we need to create a test account first. Then nodemailer uses a transport service (another 3rd party) to send our emails.

Nodemailer - Will setup the logic to send email

3rd Party - Will do the actual email sending

2.2 Email Transport Service

Now who is that 3rd party? The transporter who actually sends the email. Nodemailer will use this transport service to send the email.

As far as our options,

• For Testing we can use services like Ethereal https://ethereal.email/ or mailtrap https://mailtrap.io/
• For Production we can use Sendgrid https://sendgrid.com/ or Mailgun https://cloud.google.com/compute/docs/tutorials/sending-mail/using-mailgun.
• We can also use normal gmail account for this but we won't do that here

• In this project let's go with Ethereal as the setup is easy and faster

Ethereal Create User

Account created

To check the incoming emails u need these values. You can create as many accounts as you want just by clicking "Create Ethereal Account" button.

nodemailer config

You will setup this host in .env as you don't want to share with others (but not in this project). Once we have this Ethereal account created then we can send emails.

Let's navigate back to nodemailer docs and follow the steps.

nodemailer

controller/sendEmailController.js

const nodemailer = require('nodemailer')
const sendEmail = async (req, res) => {
  let testAccount = await nodemailer.createTestAccount()
  // create reusable transporter object using the default SMTP transport

  // You can get this createTransport code from ethereal docs as well as shown in one
  // of the above image
  let transporter = nodemailer.createTransport({
    host: 'smtp.ethereal.email',
    port: 587,
    secure: false, // true for 465, false for other ports
    auth: {
      user: '[e-mail]', // generated ethereal user
      pass: 'bSRSmZcCMmcC1gkWwC', // generated ethereal password
    },
  })

  // send mail with defined transport object
  let info = await transporter.sendMail({
    from: '"Sandeep Amarnath" <[e-mail]>', // sender address
    to: '[e-mail], [e-mail]', // list of receivers
    subject: 'Sending Emails with node js', // Subject line
    text: 'Hello Email', // plain text body
    html: '<b>Hello Anand and Paxy</b>', // html body
  })

  res.status(200).json(info)
}

module.exports = sendEmail

Got the email

2.3 Sendgrid - Send Emails in Production

Once we know how to send emails with Ethereal, let's try to do the same thing with Sendgrid. This is one of the most popular email providers for the production.

Just like Ethereal, sendgrid also has nice and clean - easy to understand UI. Mailgun and other choice are comparatively confusing.

SendGrid

Documentation

[e-mail] / Sendgridmail@123

Click on Verify a single sender button and create sender email.

Click on Verify sender and fill the form

I've given [e-mail] as a sender email

Email verified after filling the form

Once this is set up then we will create API key. Navigate to https://app.sendgrid.com/guide/integrate/langs/nodejs

Copy this API key and put in env file

Copy this created API key and put in env file and install this npm install --save @sendgrid/mail

.env

SENDGRID_API_KEY=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

Copy this code present in the below documentation and paste this in sendEmailController file and try it out

SendGrid

sendgrid documentation - copy the code from here

controller/sendEmailController.js

const nodemailer = require('nodemailer')
const sgMail = require('@sendgrid/mail')

// Sending Email through Ethereal transporter

const sendEmailEthereal = async (req, res) => {
  let testAccount = await nodemailer.createTestAccount()
  // create reusable transporter object using the default SMTP transport

  // You can get this createTransport code from ethereal docs as well as shown in one
  // of the above image
  let transporter = nodemailer.createTransport({
    host: 'smtp.ethereal.email',
    port: 587,
    secure: false, // true for 465, false for other ports
    auth: {
      user: '[e-mail]', // generated ethereal user
      pass: 'bSRSmZcCMmcC1gkWwC', // generated ethereal password
    },
  })

  // send mail with defined transport object
  let info = await transporter.sendMail({
    from: '"Sandeep Amarnath" <[e-mail]>', // sender address
    to: '[e-mail], [e-mail]', // list of receivers
    subject: 'Sending Emails with node js', // Subject line
    text: 'Hello Email', // plain text body
    html: '<b>Hello Anand and Paxy</b>', // html body
  })

  res.status(200).json(info)
}

// Sending Email through Sendgrid transporter

const sendEmailSendgrid = async (req, res) => {
  sgMail.setApiKey(process.env.SENDGRID_API_KEY)
  const msg = {
    to: '[e-mail]', // Change to your recipient
    from: '[e-mail]', // Change to your verified sender
    subject: 'Test Email',
    text: 'Test Email from Sendgrid',
    html: '<strong>Test Email from Sendgrid</strong>',
  }
  try {
    const info = await sgMail.send(msg)
    res.status(200).json(info)
  } catch (er) {
    console.log(er)
  }
}

module.exports = sendEmailSendgrid

Done with this Email Sending Project. Good job!!

✅ 8. Auth Workflow

In this project we will learn the following:

• How to validate emails

• Create refresh tokens

• Setup reset password functionality

Step 1 (Project Overview)

When the app loads, the front-end initially calls /showMe route that we implemented in backend. This route checks if the user has already logged in and gives back 401 error (UnAuthorized) if not logged in (if no token is present in the request - cookie from front-end).

app initial load

For recalling about /showMe route and getting a complete understanding , refer how we implemented it in E-Commerce project #5.2-show-current-user

Now since you don't have an account yet, you can click register and enter creds to register.

registering user

Once you register the user, you get back 201, saying user created and you get back a message to verify the email, hence you still can't access the application before verification. You can't login yet and if you do, you get a message saying "Please verify your email".

Once you verify your email, by clicking on the verification link present in the email, the verification will be successful and that verification link will redirect you back to the front-end screen (with a token in the URL - more about this later) that says, "Email Confirmed, Please login".

Now we can login with that email and it works where it logs you in successfully. When you login you see the refresh token cookie (In this project you will see two cookies - access token and refresh token - More on this later)

So to reiterate this, the user will login, get the refresh token stored in the cookie and also gets back a user from /showMe route. So even when user refreshes the page, the refresh token exists in the cookie (this token in the cookie will not go away on refreshing the page 😄) and with this cookie the /showMe route is called (this is the first thing that happens when page loads or reloads) and gets back the user.

When the application is deployed on heroku, sometimes if this node app is not used much, heroku will put it sleep and it takes time to load when used after long time. Just FYI, this is the annoying part of Heroku free service.

Step 2 (Starter files overview)

To start with we will use the starter files provided by John. We have two folders /front-end (don't worry about this yet), /server. The /server contains most of the familiar files. It is actually the E-commerce project we just completed. This is because just to showcase that we can add functionality to any existing project (No need to worry much here). We are not going to change anything major in existing code in E-commerce part. They are just used so that it will make us comfortable an existing project with larger codebase.

We are not going to alter too much in e-commerce project part but may be a very little. Will see how it goes.

Install Project and run it

• Navigate to starter/server and do npm install

• Setup .env file with Mongo URI

• Run npm start

Step 3 (Understand the workflow)

Before we continue, let's try to understand what we need to achieve. Unlike Jobs API or our E-commerce API where, when user registers he used to get a token back, we shouldn't send a token in this project as soon as the user registers.

So, when a user registers with username and password, we create a user in the db, but there's an additional step where the user needs to verify the email. So when the user registers, first we need to send that verification email.

In E-commerce project, in register controller, these are the steps we did

1. Checked if Name, Email and Password are provided

2. Checked if this email already exists

3. If email didn't exist then we can created the user

4. Once the user got created, we sent back the token

In this project, we need to change the 4th point where, after creating the user we shouldn't send him the token, but instead send the verification email + fake token must be also be sent (will see in some time why we need this fake token).

Step 4 (Modify current code to implement email verification)

Let's modify existing e-commerce code to implement the email verification

4.1 Modify User Model

Steps to Update User Model

• [] add following three properties

• [] verificationToken - String

• [] isVerified - {type:Boolean, default:false}

• [] verified - Date

models/UserModel.js

models/UserModel.js

const mongoose = require('mongoose')
const validator = require('validator')
const bcrypt = require('bcryptjs')

const UserSchema = new mongoose.Schema({
  name: {
    type: String,
    required: [true, 'Please provide name'],
    minlength: 3,
    maxlength: 50,
  },
  email: {
    type: String,
    unique: true,
    required: [true, 'Please provide email'],
    validate: {
      validator: validator.isEmail,
      message: 'Please provide valid email',
    },
  },
  password: {
    type: String,
    required: [true, 'Please provide password'],
    minlength: 6,
  },
  role: {
    type: String,
    enum: ['admin', 'user'],
    default: 'user',
  },
  // Auth Workflow Project
  verificationToken: String,
  isVerified: {
    type: Boolean,
    default: false,
  },
  verified: {
    type: Date,
  },
})

UserSchema.pre('save', async function () {
  // console.log(this.modifiedPaths());
  // console.log(this.isModified('name'));
  if (!this.isModified('password')) return
  const salt = await bcrypt.genSalt(10)
  this.password = await bcrypt.hash(this.password, salt)
})

UserSchema.methods.comparePassword = async function (canditatePassword) {
  const isMatch = await bcrypt.compare(canditatePassword, this.password)
  return isMatch
}

module.exports = mongoose.model('User', UserSchema)

4.2 Modify Register Controller

As I said in Step 4, the user when registers, will not get back the token but instead will get back the verification email. Hence we need to remove the code after createUser and add some code.

Steps for Update Register Controller

• [] setup fake verificationToken - 'fake token' - We will later use crypto library for doing this

• [] remove everything after User.create()

• [] send back success message and token

register controller. We are sending back fake token for now for testing. Will send email later

controllers/authController.js partial

const register = async (req, res) => {
  const { email, name, password } = req.body

  const emailAlreadyExists = await User.findOne({ email })
  if (emailAlreadyExists) {
    throw new CustomError.BadRequestError('Email already exists')
  }

  // first registered user is an admin
  const isFirstAccount = (await User.countDocuments({})) === 0
  const role = isFirstAccount ? 'admin' : 'user'

  /* E-Commerce CODE where we used to send token after registration was successful. 
     Commenting this below part out in this Auth Workflow setup*/

  // const tokenUser = createTokenUser(user);
  // attachCookiesToResponse({ res, user: tokenUser });
  // res.status(StatusCodes.CREATED).json({ user: tokenUser });

  /* In Auth Workflow project, let's now implement this new functionality below of 
     sending verification email */

  const verificationToken = 'fake token'

  const user = await User.create({
    name,
    email,
    password,
    role,
    verificationToken,
  })

  // actually we need to send an email here, but let's send verification token back just for testing it in postman!!!

  res.status(StatusCodes.CREATED).json({
    msg: 'Success! Please check your email to verify the account',
    verificationToken: user.verificationToken, // we could have directly done verification token. But just seeing if user got created and has verificationToken on user object
  })
}

4.2.1 Postman test register

we get back the response (token + message) for register call

So thee user got registered in our DB but not yet verified.

registered but not yet verified

So at this point, if user tries to login, he should not be able to. He must get back a message saying "Please verify your email", so let's modify our login controller to handle this.

4.3 Modify Login Controller

Steps to Update Login Controller

• [] right after isPasswordCorrect

• [] check if user.isVerified, if not 401

Currently in our E-commerce app (and this auth app since it's the same app as e-commerce), we are sending back the token to the user once he logs in. This shouldn't be the case anymore and before sending this token, we need another check.

So, after we check if the provided password is correct, we need to also check if the user is verified or not. If not we need to send back 401 (Unauthorized) response. Let's implement that.

login controller code screenshot

controllers/authController.js partial

const login = async (req, res) => {
  const { email, password } = req.body

  if (!email || !password) {
    throw new CustomError.BadRequestError('Please provide email and password')
  }
  const user = await User.findOne({ email })

  if (!user) {
    throw new CustomError.UnauthenticatedError('Invalid Credentials')
  }
  const isPasswordCorrect = await user.comparePassword(password)
  if (!isPasswordCorrect) {
    throw new CustomError.UnauthenticatedError('Invalid Credentials')
  }

  /* In Auth Workflow project, let's now implement this new functionality below of 
     sending 401 (Unauthorized) if user is not verified */
  if (!user.isVerified) {
    throw new CustomError.UnauthenticatedError('Please verify your email')
  }
  /* The above is implemented in AuthWorkflow project*/

  const tokenUser = createTokenUser(user)
  attachCookiesToResponse({ res, user: tokenUser })
  res.status(StatusCodes.OK).json({ user: tokenUser })
}

4.4 Setup proper token

Now we are sending a fake token in the register controller, but when it comes to production, sending fake token is silly. We need unique token for each user.

• User registers

• He/She gets the unique token

• Proceed to the next steps

To generate this unique token for each user, there are multiple options. A clean option would be to use crypto library built into Node and you don't have to install anything extra.

const crypto = require('crypto')
const verificationToken = crypto.randomBytes(40).toString('hex')

Note: crypto.randomBytes(<any number>) - This is going to create a buffer with random bytes. Since this is a buffer we want turn this into a string, and on this randomBytes buffer we have a method called toString() which will turn buffer into a string where each byte will be encoded as 2 hexadecimal character. Since we have specified buffer number as 40, converting that to hex will give us back 80 characters.

Modified this token line in auth controller for register (line 23 below)

controllers/authController.js partial

const register = async (req, res) => {
  const { email, name, password } = req.body

  const emailAlreadyExists = await User.findOne({ email })
  if (emailAlreadyExists) {
    throw new CustomError.BadRequestError('Email already exists')
  }

  // first registered user is an admin
  const isFirstAccount = (await User.countDocuments({})) === 0
  const role = isFirstAccount ? 'admin' : 'user'

  /* E-Commerce CODE where we used to send token after registration was successful. 
     Commenting this below part out in this Auth Workflow setup*/

  // const tokenUser = createTokenUser(user);
  // attachCookiesToResponse({ res, user: tokenUser });
  // res.status(StatusCodes.CREATED).json({ user: tokenUser });

  /* In Auth Workflow project, let's now implement this new functionality below of 
     sending verification email */

  const verificationToken = crypto.randomBytes(40).toString('hex')

  const user = await User.create({
    name,
    email,
    password,
    role,
    verificationToken,
  })

  // actually we need to send an email here, but let's send verification token back just for testing it in postman!!!

  res.status(StatusCodes.CREATED).json({
    msg: 'Success! Please check your email to verify the account',
    verificationToken: user.verificationToken, // we could have directly done verification token. But just seeing if user got created and has verificationToken on user object
  })
}

getting 80 character string in token

4.5 Verify Email big picture

• User registers ---> /register - this will send him the verificationToken

• User verifies his email (body has verificationToken) ---> /verify-email - this will check if verificationToken sent by user is correct or not and if correct then user is verified

• User logs in ---> /login - gets back the normal token like previous projects and he can access any route

When user registers,

• User provides name, email and password during registration

• These are sent to registration controller via /register route

• Registration controller does the following

  • Checks if email exists in DB already. If yes it throws an error

  • Checks length of password and hashes the password

  • Creates a verificationToken (in this project, we are using crypto library provided by node to do this)

  • Stores name, email, hashed password, verificationToken to DB

  • Sends back the verificationToken and a message ("please verify email") to the user (now doing this through postman, but generally we will send an email to the user's email with these and expect the user to click the link - we will do this later in some time)

• Note that isVerifed is also set to false by userModel already in the DB at this point

/register - when user registers this is what happens

When user clicks the email sent by registration controller

If this is done through postman then the user can't click the email but need to do a POST request to /verify-email (same thing happens when clicked on email. i.e POST request is done to /verify-email)

• User sends a POST request to /verify-email with email and verificationToken

• This was the same verificationToken that was sent by registration controller in previous step

• /verify-email controller verifies this verificationToken sent by user against the one stored for this user in DB

• Once they match the /verification-email controller

  • Sets isVerified to true

  • Sets verified to Date.now()

  • Sets verificationToken to empty string to avoid duplicate verification

  • Once these are set in DB, it sends a success message "Email successfully verified" back to the user

/verify-email POST route called by the user when he needs to verify email

4.5.1 Verify Email

Let's now implement above steps to our code to verify email. Initially we will verify the email through postman and then later we will see how to send the email to the user

Steps Verify Email Controller

• [] create verifyEmail in authController

• [] get verificationToken and email from req.body

• [] setup a '/verify-email' route in authRoutes

• [] test in a Postman

• [] check for user using email

• [] if no user 401

• [] if token does not match user token 401

• [] if correct set

• [] user.isVerified = true

• [] user.verified = Date.now()

• [] user.verificationToken = ''

• [] save use with instance method

• [] return msg:'email verified'

/verify-email controller

Line 47, verifyEmail controller below

controllers/authController.js

const User = require('../models/User')
const { StatusCodes } = require('http-status-codes')
const CustomError = require('../errors')
const { attachCookiesToResponse, createTokenUser } = require('../utils')
const crypto = require('crypto')

const register = async (req, res) => {
  const { email, name, password } = req.body

  const emailAlreadyExists = await User.findOne({ email })
  if (emailAlreadyExists) {
    throw new CustomError.BadRequestError('Email already exists')
  }

  // first registered user is an admin
  const isFirstAccount = (await User.countDocuments({})) === 0
  const role = isFirstAccount ? 'admin' : 'user'

  /* E-Commerce CODE where we used to send token after registration was successful. 
     Commenting this below part out in this Auth Workflow setup*/

  // const tokenUser = createTokenUser(user);
  // attachCookiesToResponse({ res, user: tokenUser });
  // res.status(StatusCodes.CREATED).json({ user: tokenUser });

  /* In Auth Workflow project, let's now implement this new functionality below of 
     sending verification email */

  const verificationToken = crypto.randomBytes(40).toString('hex')

  const user = await User.create({
    name,
    email,
    password,
    role,
    verificationToken,
  })

  // actually we need to send an email here, but let's send verification token back just for testing it in postman!!!

  res.status(StatusCodes.CREATED).json({
    msg: 'Success! Please check your email to verify the account',
    verificationToken: user.verificationToken, // we could have directly done verification token. But just seeing if user got created and has verificationToken on user object
  })
}

const verifyEmail = async (req, res) => {
  const { verificationToken, email } = req.body
  if (!email || !verificationToken) {
    throw new CustomError.BadRequestError('Please provide email and token')
  }
  const user = await User.findOne({ email })
  if (!user) {
    throw new CustomError.UnauthenticatedError('Verification failed')
  }
  if (verificationToken !== user.verificationToken) {
    throw new CustomError.UnauthenticatedError('Verification failed')
  }

  user.isVerified = true
  user.verified = Date.now()
  // to avoid duplicate, setting verificationToken to ''. If user clicks on verify email again, then he will get 'verification failed'
  user.verificationToken = ''
  user.save()
  res.status(StatusCodes.OK).json({ msg: 'Email verified' })
}

const login = async (req, res) => {
  const { email, password } = req.body

  if (!email || !password) {
    throw new CustomError.BadRequestError('Please provide email and password')
  }
  const user = await User.findOne({ email })

  if (!user) {
    throw new CustomError.UnauthenticatedError('Invalid Credentials')
  }
  const isPasswordCorrect = await user.comparePassword(password)
  if (!isPasswordCorrect) {
    throw new CustomError.UnauthenticatedError('Invalid Credentials')
  }

  /* In Auth Workflow project, let's now implement this new functionality below of 
     sending 401 (Unauthorized) if user is not verified */
  if (!user.isVerified) {
    throw new CustomError.UnauthenticatedError('Please verify your email')
  }
  /* The above is implemented in AuthWorkflow project*/

  const tokenUser = createTokenUser(user)
  attachCookiesToResponse({ res, user: tokenUser })
  res.status(StatusCodes.OK).json({ user: tokenUser })
}
const logout = async (req, res) => {
  res.cookie('token', 'logout', {
    httpOnly: true,
    expires: new Date(Date.now() + 1000),
  })
  res.status(StatusCodes.OK).json({ msg: 'user logged out!' })
}

module.exports = {
  register,
  login,
  logout,
  verifyEmail,
}

4.6 Send Email

Since we have set up the functionality for email verification (/verify-email) where we are currently sending POST request to verify email (/verify-email) after register, let's now do this /verify-email POST request through email.

So the idea is, when the user registers, instead of sending him verificationToken and message through json object, let's now send email which will have the verificationToken, so that when user clicks that email, automatically a POST request must happen to /verify-email.

Before we re-learn how to send email, please re-read #7.-send-emails and familiarize yourself to know how we send the emails in dev and prod environments.

So let's now do it step by step

4.6.1 Email Setup

Email Setup

• [] re-read project #7.-send-emails

• [] ethereal credentials (create account/login)

• [] install nodemailer

• [] create (nodemailerConfig, sendEmail, sendResetPasswordEmail, sendVerficationEmail) files in utils

npm install nodemailer

Let's create these files in utils folder

/utils/nodemailerConfig.js
/utils/sendEmail.js
/utils/sendResetPasswordEmail.js
/utils/sendVerficationEmail.js

Note that we are going to split the email sending functionality into different files as we need to re-user them for sending forgot/reset password email as well.

Send Email

Let's now write some code in /utils/sendEmail.js file. First let's hardcode sending email part - what we did in #2.2-email-transport-service. Later we can tweak this to re-use this for sending password reset email and verification email.

/utils/sendEmail.js

const nodemailer = require('nodemailer')

// Sending Email through Ethereal transporter

const sendEmail = async (req, res) => {
  // Not using testAccount in this case. 
  // If it was used then user should have been testAccount.user instead of 
  // '[e-mail]' and 
  // pass should have been testAccount.pass instead of 'fRYdcDUhv1kUEp5A8F'

  let testAccount = await nodemailer.createTestAccount()

  // create reusable transporter object using the default SMTP transport
  // You can get this createTransport code from ethereal account
  const transporter = nodemailer.createTransport({
    host: 'smtp.ethereal.email',
    port: 587,
    auth: {
      user: '[e-mail]',
      pass: 'fRYdcDUhv1kUEp5A8F',
    },
  })

  // send mail with defined transport object
  let info = await transporter.sendMail({
    from: '"Fred Foo 👻" <[e-mail]>', // sender address
    to: '[e-mail], [e-mail]', // list of receivers
    subject: 'Hello ✔', // Subject line
    text: 'Hello world?', // plain text body
    html: '<b>Hello world?</b>', // html body
  })

  res.status(200).json(info)
}

module.exports = sendEmail

Note that the above code will be used in authController in register. Later we will this code in some more places like sending reset password email and resending verification email (so we will later modify the above sendEmail code to accept arguments).

controllers/authController.js partial

const register = async (req, res) => {
  const { email, name, password } = req.body

  const emailAlreadyExists = await User.findOne({ email })
  if (emailAlreadyExists) {
    throw new CustomError.BadRequestError('Email already exists')
  }

  // first registered user is an admin
  const isFirstAccount = (await User.countDocuments({})) === 0
  const role = isFirstAccount ? 'admin' : 'user'

  /* E-Commerce CODE where we used to send token after registration was successful. 
     Commenting this below part out in this Auth Workflow setup*/

  // const tokenUser = createTokenUser(user);
  // attachCookiesToResponse({ res, user: tokenUser });
  // res.status(StatusCodes.CREATED).json({ user: tokenUser });

  /* In Auth Workflow project, let's now implement this new functionality below of 
     sending verification email */

  const verificationToken = crypto.randomBytes(40).toString('hex')

  const user = await User.create({
    name,
    email,
    password,
    role,
    verificationToken,
  })

  await sendEmail() // SENDING EMAIL HERE. NOT YET INCLUDING THE TOKEN. WILL DO IT SHORTLY

  res.status(StatusCodes.CREATED).json({
    msg: 'Success! Please check your email to verify the account',

    // NOW COMMENTING THIS BELOW AS WEE ARE NOW SENDING EMAIL IN LINE (33)-await sendEmail()

    // verificationToken: user.verificationToken, // we could have directly done verification token. But just seeing if user got created and has verificationToken on user object
  })
}

Now we are actually sending email during the registration process but not yet sending the verification token in that email. The above is to showcase how we are first sending the email and then sending the response back to the user (Success message). We will send this verification token in the email shortly.

steps in registration controller

Also, one more thing to remember is that currently, in our register controller, we are sending email to hardcoded email (receivers below), but this will change later. We will later be sending to the actual user who is registering.

  // send mail with defined transport object
  let info = await transporter.sendMail({
    from: '"Fred Foo 👻" <[e-mail]>', // sender address
    to: '[e-mail], [e-mail]', // list of receivers
    subject: 'Hello ✔', // Subject line
    text: 'Hello world?', // plain text body
    html: '<b>Hello world?</b>', // html body
  })

4.6.2 Send Email from front-end

Now that we have setup the code for sending email (without sending proper token in the email yet of course), let's try to send this email from thee front-end.

We know that the register functionality works as we did it in postman in section 4.6.1 above, we can try doing the same by clicking register button in the front-end. Navigate to starter/front-end in the project and run

npm install && npm start

You will see the react-app spin up. Do the registration process and you should see it send an email.

navigate to front-end react app

register the user

got the success email

we should get the email for from and to these emails mentioned

got the email that we have in send email utils above (send email utils is used by send email controller)

4.6.3 Sending Verification link in email

Steps to Send Verification Link

• [] refactor sendEmail

• [] setup sendVerificationEmail.js

• [] pass arguments

Let's work on refactoring sendEmail part. Let's put the configuration of createTransport in nodeMailerConfig.js. If we are using send grid or any other service for production then it's good to put the values of host, port, user and pass into .env and then use it in nodeMailerConfig.js

sendEmail config part must be put into nodeMailerConfig

Once we do this the code looks like this

//11-auth-workflow/starter/server/utils/nodemailerConfig.js

module.exports = {
  host: 'smtp.ethereal.email',
  port: 587,
  auth: {
    user: '[e-mail]',
    pass: 'rvvbPUutr2MZBuM5jq',
  },
}

/* -------------------------------------------------------- */


//11-auth-workflow/starter/server/utils/sendEmail.js

const nodemailer = require('nodemailer')
const nodeMailerConfig = require('./nodemailerConfig')
// Sending Email through Ethereal transporter

const sendEmail = async (req, res) => {
  // Not using testAccount in this case. If it was used then 
  // user should have been testAccount.user instead of '[e-mail]' 
  // and pass should have been testAccount.pass instead of 'fRYdcDUhv1kUEp5A8F'

  let testAccount = await nodemailer.createTestAccount()

  // create reusable transporter object using the default SMTP transport
  // You can get this createTransport code from ethereal account

  const transporter = nodemailer.createTransport(nodeMailerConfig)

  // send mail with defined transport object
  let info = await transporter.sendMail({
    from: '"Fred Foo 👻" <[e-mail]>', // sender address
    to: '[e-mail], [e-mail]', // list of receivers
    subject: 'Hello ✔', // Subject line
    text: 'Hello world?', // plain text body
    html: '<b>Hello world?</b>', // html body
  })
}

module.exports = sendEmail

Once we do this let's pass in to, subject and html as params. Also, we can return transport sendMail() in line 16 below and call it in place we are using it. No need to return with await keyword as sendEmail function (line 5 below) is async and it returns promise by default so we can omit await in line 16 below.

pass params

After refactor it looks like this,

refactored to accept params

Ok, now we have a better understanding of how we can make sendEmail function to accept params. Currently, sendEmail is being called in registerController. But we have to use this send email functionality in resetPassword as well later.

So let's have a separate function called sendVerificationEmail to send email during registration and sendResetPasswordEmail to send reset password email later. Both of these functions will call sendEmail.

The setup we currently have (Setup A)

current setup to send emails (SETUP A)

The setup we need (Setup B)

setup we are going to implement (SETUP B)

We need to implement setup b where the sendVerificationEmail function uses sendEmail. We will then use sendVerificationEmail in register controller.

The files for setup b looks like this

utils/sendEmail.js

const nodemailer = require('nodemailer')
const nodeMailerConfig = require('./nodemailerConfig')
// Sending Email through Ethereal transporter

const sendEmail = async ({ to, subject, html }) => {
  // Not using testAccount in this case. If it was used then user should have been testAccount.user instead of '[e-mail]' and pass should have been testAccount.pass instead of 'fRYdcDUhv1kUEp5A8F'

  let testAccount = await nodemailer.createTestAccount()

  // create reusable transporter object using the default SMTP transport
  // You can get this createTransport code from ethereal account

  const transporter = nodemailer.createTransport(nodeMailerConfig)

  // send mail with defined transport object. This return
  return transporter.sendMail({
    from: '"Sandeep Amarnath" <[e-mail]>', // sender address
    to, // list of receivers
    subject, // Subject line
    html, // html body
  })
}

module.exports = sendEmail

utils/sendEmail.js

const sendEmail = require('./sendEmail')

/* origin is going to be the URL for the front-end (to specify where the user 
should navigate once he clicks the link). More on this later */

// prettier-ignore
const sendVerificationEmail = async({ name,email,verificationToken,origin}) => {

    const link = 'https://google.com'

    // we will add the confirmation link shortly
    const message = `<p>Please confirm your email by clicking on the following link: <a href='${link}'>Take me to Google</a> </p>`

    // sendEmail is a promise and we are directly returning without await. 
    // It will be awaited in the controller its  called in
    return sendEmail({
        to:email,
        subject:'Email Confirmation',
        html:`<h1>Hello ${name}</h1>   
             ${message}
        `
    })
}

module.exports = sendVerificationEmail

controllers/authController.js partial

const register = async (req, res) => {
  const { email, name, password } = req.body

  const emailAlreadyExists = await User.findOne({ email })
  if (emailAlreadyExists) {
    throw new CustomError.BadRequestError('Email already exists')
  }

  // first registered user is an admin
  const isFirstAccount = (await User.countDocuments({})) === 0
  const role = isFirstAccount ? 'admin' : 'user'

  /* E-Commerce CODE where we used to send token after registration was successful. 
     Commenting this below part out in this Auth Workflow setup*/

  // const tokenUser = createTokenUser(user);
  // attachCookiesToResponse({ res, user: tokenUser });
  // res.status(StatusCodes.CREATED).json({ user: tokenUser });

  /* In Auth Workflow project, let's now implement this new functionality below of 
     sending verification email */

  const verificationToken = crypto.randomBytes(40).toString('hex')

  const user = await User.create({
    name,
    email,
    password,
    role,
    verificationToken,
  })

  /* await sendEmail() -  Not used anymore since we are using sendVerificationEmail here.
  sendVerificationEmail internally uses sendEmail */

  // await sendEmail()

  /* origin is URL to specify where the user 
      should navigate once he clicks the link. More on this later */
  const origin = 'http://localhost:3000'
  await sendVerificationEmail({
    name: user.name,
    email: user.email,
    verificationToken: user.verificationToken,
    origin,
  })

  res.status(StatusCodes.CREATED).json({
    msg: 'Success! Please check your email to verify the account',

    // NOW COMMENTING THIS BELOW AS WEE ARE SENDING EMAIL IN LINE  - await sendEmail() above

    // verificationToken: user.verificationToken, // we could have directly done verification token. But just seeing if user got created and has verificationToken on user object
  })
}

At this point we can send verification email and user gets it in ethereal. In sendVerificationEmail, where we are sending the email, we are including a link and on clicking that, it takes us to google.com. So the current flow is

• User registers

• The request hits registerController

• In registerController, sendVerificationEmail function is called

• In sendVerificationEmail function, sendEmail function is called which sends the email to the user. The sendVerification email includes the link (<a href), which when clicked by the user, goes to google.com (Using google.com link just for understanding here. Actually we will replace this link with the actual verification link. So when the user clicks this link he should be verified and navigated back to other screen (We will see how in a moment)

• The registerController also sends to success message

4.6.3.1 Sending the actual verification link

Now that we know we can include link in the email (we are doing it in sendVerificationEmail and it currently takes user to google), let's extend on this knowledge and build a verification link. So, the idea is

• We send a verification link in the email when user registers

• This link, when clicked

  • It should navigate user to front-end route /user/verify-email

  • This link should also have query params (verificationToken, and email) so that, In this process, while navigating to this route from front-end, the front-end route reads these query params and sends a POST request to /api/v1/auth/verify-email as we did from the postman {{URL}}/verify-email . The POST request will also include verificationToken and email that is passed in query params from backend and this verifies the email.

  • After this, it also displays that the verification process is successful.

Front-end code

Notice that in react app below, react router is configured for /user/verify-email route. So the idea is, when the link is clicked from email, it should navigate to this route. For that to happen, the href in that link should be pointing to this route /user/verify-email. Since we need full URL, it should be http://localhost:3000/user/verify-email with query params verificationToken and email. Let's see why we need these query params to be present in this link.

react-app route to navigate when the verification link is clicked

Let's look at <Verify/> Component

verify email front-end code

Pages/Verify.js

import React, { useState, useEffect } from 'react';
import { useLocation, Link } from 'react-router-dom';
import styled from 'styled-components';
import { useGlobalContext } from '../context';
import axios from 'axios';
function useQuery() {
  return new URLSearchParams(useLocation().search);
}

const VerifyPage = () => {
  const [error, setError] = useState(false);
  const [loading, setLoading] = useState(false);
  const { isLoading } = useGlobalContext();
  const query = useQuery();

  const verifyToken = async () => {
    setLoading(true);
    try {
      const { data } = await axios.post('/api/v1/auth/verify-email', {
        verificationToken: query.get('token'),
        email: query.get('email'),
      });
    } catch (error) {
      // console.log(error.response);
      setError(true);
    }
    setLoading(false);
  };

  useEffect(() => {
    if (!isLoading) {
      verifyToken();
    }
  }, []);

  if (loading) {
    return (
      <Wrapper className='page'>
        <h2>Loading...</h2>
      </Wrapper>
    );
  }

  if (error) {
    return (
      <Wrapper className='page'>
        <h4>There was an error, please double check your verification link </h4>
      </Wrapper>
    );
  }

  return (
    <Wrapper className='page'>
      <h2>Account Confirmed</h2>
      <Link to='/login' className='btn'>
        Please login
      </Link>
    </Wrapper>
  );
};

const Wrapper = styled.section``;

export default VerifyPage;

So now we know that we need to send query params -> verificationToken, and email. Let's see the full href in the link (when clicked, this is what the URL should be with query params which then hits Verify Page in react app)

// href of that link is (when it is clicked it should navigate to)
http://localhost:3000/user/verify-email?token=<token>&email=<email>

So the sendVerificationEmail controller looks like this

sendVerificationEmail.js

controller/sendVerificationEmail.js

const sendEmail = require('./sendEmail')

/* origin is going to be the URL for the front-end (to specify where the user 
should navigate once he clicks the link). More on this later */

// prettier-ignore
const sendVerificationEmail = async({ name,email,verificationToken,origin}) => {

    // origin is https://localhost:3000 for dev (passing this from registerController)
    // so the full URL should be - http://localhost:3000/user/verify-email?verificationToken=<token>&email=<email>
    const link = `${origin}/user/verify-email?token=${verificationToken}&email=${email}`

    // we will add the confirmation link shortly
    const message = `<p>Please confirm your email by clicking on the following link: 
    <a href='${link}'>Click to verify</a> </p>`

    // sendEmail is a promise and we are directly returning without await. 
    // It will be awaited in the controller its  called in
    return sendEmail({
        to:email,
        subject:'Email Confirmation',
        html:`<h1>Hello ${name}</h1>   
             ${message}
        `
    })
}

module.exports = sendVerificationEmail

Full flow of sending verification email

full flow of sending verification email

4.6.4 Origin info

We used origin = http://localhost:3000in sendVerificationEmail but what about production? We can do this the same way. We can first deploy our app to prod and then use this prod URL as origin and that is the easy way.

Now what if we need this to be dynamic. I mean, if there is some other front-end using our app then it's tedious that we need to know the deployed URL of that app which is unnecessary headache (may be it's fine in this case). So let's see how we can generate this origin.

The origin can be generated by the req object. We have access to this req. If we console log req object it looks very large. We mainly want headers here. Here are a few important things of req object.

req headers

x-forwarded-host
x-forwarded-proto

The above 2 form the actual origin (because that is the actual sender of reequest). 
We can use them as origin like this

const origin = req.get(x-forwarded-proto) + req.get(x-forwarded-host)

// another way of getting fields from req headers is below
// const origin = req.headers['x-forwarded-proto'] + req.headers['x-forwarded-host']

Note that we use get method (req.get) to get the headers

................................................................................

The origin shown inside headers is https://localhost:5000. This is because we include
our server URL as proxy to allow port forwarding for CORS. 

referrer : This is the URL where the request came from

Step 5 (Refresh token)

Making this Step 5 and I didn't prefer doing it as 4.7. Technically, we are still using the same E-commerce code (as Step 4 heading specifies). Just making it as step 5 for better differentiation.

5.1 Need of Refresh token

These are some of the points I am drafting based on the resources (stack overflow and youtube) attached below

• In real world, it's better to have separate servers called authServer and resourceServer/s

  • authServer - used only for authentication and authorization. This server's responsibility is to issue refresh token, access token and also logout the users

  • resourceServer - this server (can also be multiple servers) provides protected data like products, reviews and so on (like we did in e-commerce project)

• One of the uses of refresh token is, we don't have to send username and password (credentials) over the wire (from front-end to back-end) every time when we need a new access token. The refresh token will get you the new access token from authServer so you can continue to make requests to your protected resourceServer

• The other main use of refresh_token is that, let's say your authServer is very protected compared to resourceServer in real world (third party services like auth0, okta, azure and so on or your own implementation). You will only send your access_token to the resourceServer (to get data) and you will never have to send refresh_token to resourceServer . So there is a good chance that your access_token when sent to resourceServer, there is a hacker intercepting your resourceServer (since it's not that secure) who gets access to your short lived access_token.

  • For this reason, the access_token is has short life span (like 30 minutes). Remember, when this access_token expires, you will send refresh_token to authServer (which is very secure than resourceServer) to get a new access_token. Since you are not sending the refresh_token to the resourceServer at any time, there is no way that the hacker who is intercepting the resourceServer gets your refresh_token. If you, as a developer, still get a doubt that your users' refresh_token might also be hacked, then you can logout all users (make refresh_token invalid for all users) and that way the users will login again to get a new refresh_token and things will get on track again.

Resources

Why Does OAuth v2 Have Both Access and Refresh Tokens?Stack Overflow

Stack overflow

workflow with and without access token

Nice tutorial that explains why we need refresh token and how to implement with and without refresh token

Why Does OAuth v2 Have Both Access and Refresh Tokens?Stack Overflow

My answer

5.2 Access and Refresh token in current app

This is what we need to achieve in current app

• User logs in

• Then he/she gets back both access-token and refresh-token

• Access-token will have short lifetime and refresh token will have longer life time

• Once access-token expires the refresh token is used to get back a new access token

• User can continue to access the app with new access-token

5.3 Modify Setup

5.3.1 Token Model

We are creating this token model here as we are going to store the refresh token in the db.

Token Model

• [] create Token.js in models

• [] refreshToken,ip,userAgent - all String and required

• [] isValid - Boolean, default:true

• [] ref user

• [] timestamps true

models/Token.js

const mongoose = require('mongoose')
const TokenSchema = new mongoose.Schema(
  {
    refreshToken: {
      type: String,
      required: true,
    },

    /* We are not going to use the ip and UserAgent fields in this project 
  (they are here just to showcase where we can get them on the request) */

    // to specify where we are getting the request from 
    ip: {
      type: String,
      required: true,
    },

    // to specify what device is accessing our project
    userAgent: {
      type: String,
      required: true,
    },
    isValid: {
      type: Boolean,
      default: true,
    },
    user: {
      type: mongoose.Types.ObjectId,
      ref: 'User',
      required: true,
    },
  },
  { timestamps: true }
)

const tokenModel = mongoose.model('Token', TokenSchema)
module.exports = tokenModel

5.3.2 Create refresh token in login controller

We currently have code to create a token (this was just an access token as per our previous setup in e-commerce ) project, but now let's setup refresh token.

Before I do that, let me show you the current code for login controller before modifying it. Note that Lines 17 - 22 were added during modifying the register controller.

controller/authController.js partial

const login = async (req, res) => {
  const { email, password } = req.body

  if (!email || !password) {
    throw new CustomError.BadRequestError('Please provide email and password')
  }
  const user = await User.findOne({ email })

  if (!user) {
    throw new CustomError.UnauthenticatedError('Invalid Credentials')
  }
  const isPasswordCorrect = await user.comparePassword(password)
  if (!isPasswordCorrect) {
    throw new CustomError.UnauthenticatedError('Invalid Credentials')
  }

  /* In Auth Workflow project, let's now implement this new functionality below of 
     sending 401 (Unauthorized) if user is not verified */
  if (!user.isVerified) {
    throw new CustomError.UnauthenticatedError('Please verify your email')
  }
  /* The above is implemented in AuthWorkflow project*/

  const tokenUser = createTokenUser(user)
  attachCookiesToResponse({ res, user: tokenUser })
  res.status(StatusCodes.OK).json({ user: tokenUser })
}

Modified login controller

We are creating a refresh token here, let's see how it looks. For now, we are just creating the refresh token, adding it to DB, and sending it back to the user just to check if it works. We will tweak this eventually and add more functionality.

login controller - refresh token introduced

controllers/authController.js partial

const login = async (req, res) => {
  const { email, password } = req.body

  if (!email || !password) {
    throw new CustomError.BadRequestError('Please provide email and password')
  }
  const user = await User.findOne({ email })

  if (!user) {
    throw new CustomError.UnauthenticatedError('Invalid Credentials')
  }
  const isPasswordCorrect = await user.comparePassword(password)
  if (!isPasswordCorrect) {
    throw new CustomError.UnauthenticatedError('Invalid Credentials')
  }

  /* In Auth Workflow project, let's now implement this new functionality below of 
     sending 401 (Unauthorized) if user is not verified */
  if (!user.isVerified) {
    throw new CustomError.UnauthenticatedError('Please verify your email')
  }
  /* The above is implemented in AuthWorkflow project*/

  const tokenUser = createTokenUser(user)

  // create refresh token - and tie this to a user
  let refreshToken = '' // we will see why this is let and why setting this to empty later
  // let's set all the fields the token model needs and tie this refresh token to a user
  refreshToken = crypto.randomBytes(40).toString('hex')

  const userAgent = req.headers['user-agent'] // another way is  - req.get('user-agent')

  const ip = req.ip

  const userToken = { refreshToken, ip, userAgent, user: user._id }

  const token = await Token.create(userToken)

  // check for existing refresh token (we will do this later, and we will see later why this is important)

  /* commenting below code of attachCookiesToResponse for now*/
  //attachCookiesToResponse({ res, user: tokenUser })

  res.status(StatusCodes.OK).json({ user: tokenUser, token }) // just to showcase, let's send back the token
}

5.3.2.1 Send two cookies (access + refresh token)

Earlier, in register and login controller, we just sent one cookie which was called token after user logged in or registered. Now we need to modify this to send two cookies (one for access + one for refresh token)

Current setup we have with one token

utils/jwt.js

const jwt = require('jsonwebtoken')

const createJWT = ({ payload }) => {
  // we can also remove expiresIn in this token and set it in the cookie on which we send this token below
  const token = jwt.sign(payload, process.env.JWT_SECRET, {
    expiresIn: process.env.JWT_LIFETIME,
  })
  return token
}

const isTokenValid = ({ token }) => jwt.verify(token, process.env.JWT_SECRET)

const attachCookiesToResponse = ({ res, user }) => {
  const token = createJWT({ payload: user })

  const oneDay = 1000 * 60 * 60 * 24

  res.cookie('token', token, {
    httpOnly: true,
    expires: new Date(Date.now() + oneDay),
    secure: process.env.NODE_ENV === 'production',
    signed: true,
  })
}

module.exports = {
  createJWT,
  isTokenValid,
  attachCookiesToResponse,
}

After modification for two cookies.

We use attachCookiesToResponse in login controller, so let's change there a bit

controllers/authController.js partial

const login = async (req, res) => {
  const { email, password } = req.body

  if (!email || !password) {
    throw new CustomError.BadRequestError('Please provide email and password')
  }
  const user = await User.findOne({ email })

  if (!user) {
    throw new CustomError.UnauthenticatedError('Invalid Credentials')
  }
  const isPasswordCorrect = await user.comparePassword(password)
  if (!isPasswordCorrect) {
    throw new CustomError.UnauthenticatedError('Invalid Credentials')
  }

  /* In Auth Workflow project, let's now implement this new functionality below of 
     sending 401 (Unauthorized) if user is not verified */
  if (!user.isVerified) {
    throw new CustomError.UnauthenticatedError('Please verify your email')
  }
  /* The above is implemented in AuthWorkflow project*/

  const tokenUser = createTokenUser(user)

  // create refresh token - and tie this to a user
  let refreshToken = '' // we will see why this is let and why setting this to empty later
  // let's set all the fields the token model needs and tie this refresh token to a user
  refreshToken = crypto.randomBytes(40).toString('hex')

  const userAgent = req.headers['user-agent'] // another way is  - req.get('user-agent')

  const ip = req.ip

  const userToken = { refreshToken, ip, userAgent, user: user._id }

  await Token.create(userToken)

  // check for existing refresh token (we will do this later, and we will see later why this is important)

  attachCookiesToResponse({ res, user: tokenUser, refreshToken }) // adding refreshToken here so we can send access + refresh token in attachCookiesToResponse

  res.status(StatusCodes.OK).json({ user: tokenUser })
}

utils/jwt.js

const jwt = require('jsonwebtoken')

const createJWT = ({ payload }) => {
  // we can also remove expiresIn in this token and set it in the cookie on which we send this token below
  // const token = jwt.sign(payload, process.env.JWT_SECRET, {
  //   expiresIn: process.env.JWT_LIFETIME,
  // })

  const token = jwt.sign(payload, process.env.JWT_SECRET)

  return token
}

const isTokenValid = ({ token }) => jwt.verify(token, process.env.JWT_SECRET)

const attachCookiesToResponse = ({ res, user, refreshToken }) => {
  // in accessTokenJWT we will have only the user

  // in refreshTokenJWT we will have the user + refeshToken string value
  const accessTokenJWT = createJWT({ payload: { user } })
  const refreshTokenJWT = createJWT({ payload: { user, refreshToken } })

  // accessTokenJWT can be short term like 15 mins, where as refreshTokenJWT can be longer like one day or 60 days

  const oneDay = 1000 * 60 * 60 * 24

  res.cookie('accessToken', accessTokenJWT, {
    httpOnly: true,
    // expires: new Date(Date.now() + oneDay), // we can give expiresIn or maxAge
    secure: process.env.NODE_ENV === 'production',
    signed: true,
    maxAge: 1000, // 1000 = 1s
  })
  res.cookie('refreshToken', refreshTokenJWT, {
    httpOnly: true,
    expires: new Date(Date.now() + oneDay),
    secure: process.env.NODE_ENV === 'production',
    signed: true,
  })
}

// const OLD_attachCookiesToResponse = ({ res, user }) => {

//   const token = createJWT({ payload: user })
//   const oneDay = 1000 * 60 * 60 * 24
//   res.cookie('token', token, {
//     httpOnly: true,
//     expires: new Date(Date.now() + oneDay),
//     secure: process.env.NODE_ENV === 'production',
//     signed: true,
//   })

// }

module.exports = {
  createJWT,
  isTokenValid,
  attachCookiesToResponse,
}

We modified attachTokenToCookies to attach two cookies to response. One for access and one for refresh.

getting back access and refresh tokens when we login

5.3.2.2 Check if refresh token exists in DB already

check if refresh token exists in db or not

Since we are creating the refresh token and adding it to DB every time when the user logs in, it creates multiple refresh tokens for same user which is not good.

Instead, we need to check if the refresh token exists in DB for this user already, and if it does then we need to check if it is valid. If valid then we can use this token and attach it to cookie.

We can also make this token invalid if we find any suspicious login. So that user will then have to provide credentials once again which will then be a safe approach.

Let's add the code to check if refresh token exists.

checking for existing token

controllers/authController.js - partial

const login = async (req, res) => {
  const { email, password } = req.body

  if (!email || !password) {
    throw new CustomError.BadRequestError('Please provide email and password')
  }
  const user = await User.findOne({ email })

  if (!user) {
    throw new CustomError.UnauthenticatedError('Invalid Credentials')
  }
  const isPasswordCorrect = await user.comparePassword(password)
  if (!isPasswordCorrect) {
    throw new CustomError.UnauthenticatedError('Invalid Credentials')
  }

  /* In Auth Workflow project, let's now implement this new functionality below of 
     sending 401 (Unauthorized) if user is not verified */
  if (!user.isVerified) {
    throw new CustomError.UnauthenticatedError('Please verify your email')
  }
  /* The above is implemented in AuthWorkflow project*/

  const tokenUser = createTokenUser(user)

  // create refresh token - and tie this to a user
  let refreshToken = ''

  const existingToken = await Token.findOne({ user: user._id })
  console.log('existing token', existingToken)

  // IF Refresh Token is already present in DB
  if (existingToken) {
    // if u find any suspicious activity on this refresh token, u can go and invalidate in DB for this user
    if (!existingToken.isValid) {
      throw new CustomError.UnauthenticatedError('Invalid credentials')
    }
    refreshToken = existingToken.refreshToken
    attachCookiesToResponse({ res, user: tokenUser, refreshToken })
    res.status(StatusCodes.OK).json({ user: tokenUser })
    return // adding return here so that the below code doesn't have to execute if token is already present in DB
  }

  // IF Refresh Token is NOT present in DB
  console.log('REACHED NOT BLOCK')

  refreshToken = crypto.randomBytes(40).toString('hex')

  const userAgent = req.headers['user-agent'] // another way is  - req.get('user-agent')

  const ip = req.ip

  const userToken = { refreshToken, ip, userAgent, user: user._id }

  await Token.create(userToken)

  attachCookiesToResponse({ res, user: tokenUser, refreshToken }) // adding refreshToken here so we can send access + refresh token in attachCookiesToResponse

  res.status(StatusCodes.OK).json({ user: tokenUser })
}

5.3.3 Authentication middleware modification

Remember what authentication middleware does? Just to remind - we use this middleware in front of protected routes. Let's say we want to create a new Order, and for this the user must already be logged in.

this auth mw checks the token if exists in the request

This authentication mw checks if the request has the token attached and if yes then only it will allow us to create the order.

Currently we have this in auth mw

current auth mw

Now since we don't have token but now have access-token and refresh-token, we need to modify this auth mw code.

auth middleware with explanation

middleware/authentication.js partial

const authenticateUser = async (req, res, next) => {
  const { accessToken, refreshToken } = req.signedCookies
  try {
    if (accessToken) {
      const payload = isTokenValid(accessToken)
      req.user = payload.user
      return next()
    }
    const payload = isTokenValid(refreshToken)
    const existingToken = await Token.findOne({
      user: payload.user.userId,
      refreshToken: payload.refreshToken,
    })
    if (!existingToken || !existingToken.isValid) {
      throw new CustomError.UnauthenticatedError('Authentication Invalid')
    }
    req.user = payload.user
    attachCookiesToResponse({
      res,
      user: payload.user,
      refreshToken: existingToken.refreshToken,
    })
    next()
  } catch (e) {
    throw new CustomError.UnauthenticatedError(
      'Authentication Invalid, error occured'
    )
  }
  // OLD IMPLEMENTATION BELOW
  // const token = req.signedCookies.token;
  // if (!token) {
  //   throw new CustomError.UnauthenticatedError('Authentication Invalid');
  // }
  // try {
  //   const { name, userId, role } = isTokenValid({ token });
  //   req.user = { name, userId, role };
  //   next();
  // } catch (error) {
  //   throw new CustomError.UnauthenticatedError('Authentication Invalid');
  // }
}

utils/jwt.js

const jwt = require('jsonwebtoken')

const createJWT = ({ payload }) => {
  // we can also remove expiresIn in this token and set it in the cookie on which we send this token below
  // const token = jwt.sign(payload, process.env.JWT_SECRET, {
  //   expiresIn: process.env.JWT_LIFETIME,
  // })

  const token = jwt.sign(payload, process.env.JWT_SECRET)

  return token
}

const isTokenValid = (token) => jwt.verify(token, process.env.JWT_SECRET)
/*
On verifying, we get back same thing what we signed
- For access token we get 
{ payload: { user } }

- For refresh token we get 
{ payload: { user, refreshToken } }
*/

const attachCookiesToResponse = ({ res, user, refreshToken }) => {
  // in accessTokenJWT we will have only the user

  // in refreshTokenJWT we will have the user + refeshToken string value
  const accessTokenJWT = createJWT({ payload: { user } })
  const refreshTokenJWT = createJWT({ payload: { user, refreshToken } })

  // accessTokenJWT can be short term like 15 mins, where as refreshTokenJWT can be longer like one day or 60 days

  const oneDay = 1000 * 60 * 60 * 24

  res.cookie('accessToken', accessTokenJWT, {
    httpOnly: true,
    // expires: new Date(Date.now() + oneDay), // we can give expiresIn or maxAge
    secure: process.env.NODE_ENV === 'production',
    signed: true,
    maxAge: 1000 * 60 * 60 * 60, // 1000 = 1s
  })
  res.cookie('refreshToken', refreshTokenJWT, {
    httpOnly: true,
    expires: new Date(Date.now() + oneDay),
    secure: process.env.NODE_ENV === 'production',
    signed: true,
  })
}

// const OLD_attachCookiesToResponse = ({ res, user }) => {

//   const token = createJWT({ payload: user })
//   const oneDay = 1000 * 60 * 60 * 24
//   res.cookie('token', token, {
//     httpOnly: true,
//     expires: new Date(Date.now() + oneDay),
//     secure: process.env.NODE_ENV === 'production',
//     signed: true,
//   })
// }

module.exports = {
  createJWT,
  isTokenValid,
  attachCookiesToResponse,
}

Step 6 (Logout)

Let's now implement the logout functionality

6.1 Remove the tokens

We can logout the users by

• Setting access and refresh token cookies to timeout immediately

• Removing the token for the logged in user from the database

So the idea is,

• The user is already logged in before, so he can be logged out

• By this, we know that we can get access to the req.user in logout controller. We can achieve this by adding the authentication middleware in logout route before it hits logout controller (making logout route as protected route like showCurrentUser route). Also, the logout route can be a delete route

routes/authRoute.js partial

router.delete('/logout', authenticateUser, logout) 

Here's the complete code for authRoutes

routes/authRoute.js

const express = require('express')
const router = express.Router()

const {
  register,
  login,
  logout,
  verifyEmail,
} = require('../controllers/authController')

const { authenticateUser } = require('../middleware/authentication')

router.post('/register', register)
router.post('/login', login)
router.delete('/logout', authenticateUser, logout)
router.post('/verify-email', verifyEmail)

module.exports = router

Logout controller

authController - logout

Logout controller code below - Line 151

controllers/authController.js

const User = require('../models/User')
const Token = require('../models/Token')
const { StatusCodes } = require('http-status-codes')
const CustomError = require('../errors')
const {
  attachCookiesToResponse,
  createTokenUser,
  sendVerificationEmail,
} = require('../utils')
const crypto = require('crypto')
// const sendEmail = require('../utils/sendEmail') // Not used anymore since we are using sendVerificationEmail here.
// sendVerificationEmail internally uses sendEmail

const register = async (req, res) => {
  const { email, name, password } = req.body

  const emailAlreadyExists = await User.findOne({ email })
  if (emailAlreadyExists) {
    throw new CustomError.BadRequestError('Email already exists')
  }

  // first registered user is an admin
  const isFirstAccount = (await User.countDocuments({})) === 0
  const role = isFirstAccount ? 'admin' : 'user'

  /* E-Commerce CODE where we used to send token after registration was successful. 
     Commenting this below part out in this Auth Workflow setup*/

  // const tokenUser = createTokenUser(user);
  // attachCookiesToResponse({ res, user: tokenUser });
  // res.status(StatusCodes.CREATED).json({ user: tokenUser });

  /* In Auth Workflow project, let's now implement this new functionality below of 
     sending verification email */

  const verificationToken = crypto.randomBytes(40).toString('hex')

  const user = await User.create({
    name,
    email,
    password,
    role,
    verificationToken,
  })

  /* await sendEmail() -  Not used anymore since we are using sendVerificationEmail here.
  sendVerificationEmail internally uses sendEmail */

  // await sendEmail()

  /* origin is URL to specify where the user 
      should navigate once he clicks the link. More on this later */

  const origin = 'http://localhost:3000'
  await sendVerificationEmail({
    name: user.name,
    email: user.email,
    verificationToken: user.verificationToken,
    origin,
  })

  res.status(StatusCodes.CREATED).json({
    msg: 'Success! Please check your email to verify the account',

    // NOW COMMENTING THIS BELOW AS WEE ARE SENDING EMAIL IN LINE  - await sendEmail() above

    // verificationToken: user.verificationToken, // we could have directly done verification token. But just seeing if user got created and has verificationToken on user object
  })
}

const verifyEmail = async (req, res) => {
  const { verificationToken, email } = req.body
  if (!email || !verificationToken) {
    throw new CustomError.BadRequestError('Please provide email and token')
  }
  const user = await User.findOne({ email })
  if (!user) {
    throw new CustomError.UnauthenticatedError('Verification failed')
  }
  if (verificationToken !== user.verificationToken) {
    throw new CustomError.UnauthenticatedError('Verification failed')
  }

  user.isVerified = true
  user.verified = Date.now()
  // to avoid duplicate, setting verificationToken to ''. If user clicks on verify email again, then he will get 'verification failed'
  user.verificationToken = ''
  user.save()
  res.status(StatusCodes.OK).json({ msg: 'Email verified' })
}

const login = async (req, res) => {
  const { email, password } = req.body

  if (!email || !password) {
    throw new CustomError.BadRequestError('Please provide email and password')
  }
  const user = await User.findOne({ email })

  if (!user) {
    throw new CustomError.UnauthenticatedError('Invalid Credentials')
  }
  const isPasswordCorrect = await user.comparePassword(password)
  if (!isPasswordCorrect) {
    throw new CustomError.UnauthenticatedError('Invalid Credentials')
  }

  /* In Auth Workflow project, let's now implement this new functionality below of 
     sending 401 (Unauthorized) if user is not verified */
  if (!user.isVerified) {
    throw new CustomError.UnauthenticatedError('Please verify your email')
  }
  /* The above is implemented in AuthWorkflow project*/

  const tokenUser = createTokenUser(user)

  // create refresh token - and tie this to a user
  let refreshToken = ''

  const existingToken = await Token.findOne({ user: user._id })

  // IF Refresh Token is already present in DB
  if (existingToken) {
    // if u find any suspicious activity on this refresh token, u can go and invalidate in DB for this user
    if (!existingToken.isValid) {
      throw new CustomError.UnauthenticatedError('Invalid credentials')
    }
    refreshToken = existingToken.refreshToken
    attachCookiesToResponse({ res, user: tokenUser, refreshToken })
    res.status(StatusCodes.OK).json({ user: tokenUser })
    return // adding return here so that the below code doesn't have to execute if token is already present in DB
  }

  // IF Refresh Token is NOT present in DB

  refreshToken = crypto.randomBytes(40).toString('hex')

  const userAgent = req.headers['user-agent'] // another way is  - req.get('user-agent')

  const ip = req.ip

  const userToken = { refreshToken, ip, userAgent, user: user._id }

  await Token.create(userToken)

  attachCookiesToResponse({ res, user: tokenUser, refreshToken }) // adding refreshToken here so we can send access + refresh token in attachCookiesToResponse

  res.status(StatusCodes.OK).json({ user: tokenUser })
}

const logout = async (req, res) => {
  // We need to remove that refreshToken stored in the db for this user
  await Token.findOneAndDelete({ user: req.user.userId })

  // Setting the expiration time to now so that the tokens expires immediately
  res.cookie('accessToken', 'logout', {
    httpOnly: true,
    expires: new Date(Date.now()),
  })

  res.cookie('refreshToken', 'logout', {
    httpOnly: true,
    expires: new Date(Date.now()),
  })

  // leaving the message here just so we can see in postman
  res.status(StatusCodes.OK).json({ msg: 'user logged out!' })
}

module.exports = {
  register,
  login,
  logout,
  verifyEmail,
}

After implementing this, you can test the flow

• In front-end or postman, login a user

• Make sure you got back both the cookies

• If you are on postman, then you can easily test showCurrentUser (/showMe) route

• Delete the access token (not refresh token), and call the showMe route again. You should get back new access token. This shows that the refresh token functionality works fine

• After confirming that the refresh token works fine, let's test the logout functionality by deleting both the tokens

• Make a request to showMe route and it should give you back 401 (Invalid Credentials)

• Login again and then you should get back both the cookies (both tokens)

• Hurray! Everything works. Awesome 👏

Step 7 (Password Reset Functionality)

7.1 User Model modification

Steps

Forgot/Reset Password Functionality

• User Model

• [] passwordToken {type:String}

• [] passwordTokenExpirationDate {type:Date}

Let's add two fields to User Model

• passwordToken

• passwordTokenExpirationDate

Lines 43 - 48

models/User.js

const mongoose = require('mongoose')
const validator = require('validator')
const bcrypt = require('bcryptjs')

const UserSchema = new mongoose.Schema({
  name: {
    type: String,
    required: [true, 'Please provide name'],
    minlength: 3,
    maxlength: 50,
  },
  email: {
    type: String,
    unique: true,
    required: [true, 'Please provide email'],
    validate: {
      validator: validator.isEmail,
      message: 'Please provide valid email',
    },
  },
  password: {
    type: String,
    required: [true, 'Please provide password'],
    minlength: 2,
  },
  role: {
    type: String,
    enum: ['admin', 'user'],
    default: 'user',
  },
  // Auth Workflow Project
  verificationToken: String,
  isVerified: {
    type: Boolean,
    default: false,
  },
  verified: {
    type: Date,
  },

  // Reset / forgot password functionality support

  passwordToken: {
    type: String,
  },
  passwordTokenExpirationDate: {
    type: Date,
  },
})

UserSchema.pre('save', async function () {
  // console.log(this.modifiedPaths());
  // console.log(this.isModified('name'));
  if (!this.isModified('password')) return
  const salt = await bcrypt.genSalt(10)
  this.password = await bcrypt.hash(this.password, salt)
})

UserSchema.methods.comparePassword = async function (canditatePassword) {
  const isMatch = await bcrypt.compare(canditatePassword, this.password)
  return isMatch
}

module.exports = mongoose.model('User', UserSchema)

7.2 Auth controller and auth router modifications

Steps

• authController

• [] forgotPassword and resetPassword

• [] authRoutes

• [] post '/forgot-password' '/reset-password

• Test these two routes in postman

authController - forgot and reset password added

authRoutes forgot and reset password added

7.3 Forgot Password functionality

Same as send verification email functionality. Refer #4.5-verify-email-big-picture before you

proceed. The idea is,

• User forgot the password and he clicks on "Reset/Forgot Password" link in front-end

• He gets an email and he need to reset his password by clicking on the link sent to his email within certain time (passwordTokenExpirationDate)

• Then he will be redirected to a page where he can give new password and that will be save to db

In this 7.3 section let's work on sending that email with reset password link when thee user clicks on Forgot/Reset password button in front-end.

Front-end

On Front-end, we have a route for forgot password

ForgotPassword in front-end

ForgotPassword component front-end

Server

We will work in forgotPassword controller

• Check for email. If not provided then throw (401) error

• Check if user exists, and if he does then create a passwordToken (we will be sending this in the email) and also set passwordTokenExpirationDate to know how long this email is valid

• Send the email to the user (if user exists) - We will do it in 7.3.1 section below

• If user exists or not, send him the success message saying - "Success - Please check your email to reset the password"

  • Even though the user doesn't exist, why do we send this success email? Well, because, that way, the attacker/hacker will not know if that user email exists in DB or not if he tries to randomly reset passwords and wants to know if the email exists in DB or not.

  • Hence if we send this success message for the user who exists and not exists then hacker won't know if that user is in DB or not

forgot password controller

Small correction in line 182 above. It should be

const passwordTokenExpirationDate = new Date(Date.now() + tenMinutes)

const forgotPassword = async (req, res) => {
  const { email } = req.body
  if (!email) {
    throw new CustomError.BadRequestError('Please provide a valid email')
  }
  const user = await User.findOne({ email })
  if (user) {
    const passwordToken = crypto.randomBytes(70).toString('hex')
    // send email for password reset
    const origin = 'http://localhost:3000'
    sendResetPasswordEmail({
      name: user.name,
      email: user.email,
      token: passwordToken,
      origin,
    })

    const tenMinutes = 1000 * 60 * 10
    const passwordTokenExpirationDate = new Date(Date.now() + tenMinutes)
    user.passwordToken = passwordToken
    user.passwordTokenExpirationDate = passwordTokenExpirationDate
    await user.save()
  }

  // we will send this success response if the user exists or not for security reasons as explained above
  res
    .status(StatusCodes.OK)
    .json({ msg: 'Success! Please check you email for password reset link' })
}

postman for forgot password - valid email

email sent to valid email

If hacker tries to send reset password email to invalid email, he still gets a success message so that he can't know if that user exists in DB or not as shown below.

postman - invalid email - but still gets success message

7.3.1 Send forgot password email

index.js

authController.js forgotPassword - sendResetPasswordEmail

sendResetPasswordEmail

sendResetPasswordEmail.js

utils/sendResetPasswordEmail.js

const sendEmail = require('./sendEmail')

/* origin is going to be the URL for the front-end (to specify where the user should navigate once he clicks the link).*/

const sendResetPasswordEmail = async ({ name, email, token, origin }) => {
  // origin is https://localhost:3000 for dev

  const passwordResetLink = `${origin}/user/reset-password?token=${token}&email=${email}`

  const message = `<p>Please reset your password by clicking on the following link: 
     <a href='${passwordResetLink}'>Reset Password</a> </p>`

  return sendEmail({
    to: email,
    subject: 'Password Reset',
    html: `<h1>Hello ${name}</h1>   
        ${message}`,
  })
}

module.exports = sendResetPasswordEmail

sendResetPasswordEmail uses sendEmail

utils/sendEmail.js

const nodemailer = require('nodemailer')
const nodeMailerConfig = require('./nodemailerConfig')
// Sending Email through Ethereal transporter

const sendEmail = async ({ to, subject, html }) => {
  // Not using testAccount in this case. If it was used then user should have been testAccount.user instead of '[e-mail]' and pass should have been testAccount.pass instead of 'fRYdcDUhv1kUEp5A8F'

  let testAccount = await nodemailer.createTestAccount()

  // create reusable transporter object using the default SMTP transport
  // You can get this createTransport code from ethereal account

  const transporter = nodemailer.createTransport(nodeMailerConfig)

  // send mail with defined transport object. This return
  return transporter.sendMail({
    from: '"Sandeep Amarnath" <[e-mail]>', // sender address
    to, // list of receivers
    subject, // Subject line
    html, // html body
  })
}

module.exports = sendEmail

Let's see the flow now

reset password button clicked

enter email and click

got the success message. This will be the case even if email doesn't exist in db

got the email in ethereal

got the email in ethereal

when clicked on the Reset Password link in the email (previous image), we get redirected to this front-end

7.3.1.1 Front-end code

Let's now look at how, on clicking the Reset Password link in the email, it redirects to front-end.

We already have written the redirected link in sendResetPasswordEmail.js file like this

reset password link

So on-clicking this, it will be redirected to this URL

http://localhost:3000/user/reset-password?token=c7ed6dc701639b6cf1945077aa2ab77a8dd899a99768d20c14261e43763799691f5bbaae108559992fa277f9f66c311fa70a8cfc6f5f355bbd116bb2da3f553d1a61a9fd100d&email=[e-mail]

reset password

In line 47 we have <ResetPassword/> component which is that page that shows up on clicking reset password link in email

reset-password controller

This way we call reset password controller from front-end, where we send query params back to our resetPassword controller so we can verify if they match what we had sent them in forgotPassword controller. Along with query params, we also pass new password (line 33 above). Let's now implement resetPassword controller in the server now so that we can enter new password from front-end and then save it in DB for that user's email

7.3.2 Reset Password

User clicks on reset password link in email and then redirects to front-end. He then enters new password and now it hits controllers/authController.js - resetPassword

controllers/authController.js partial

const resetPassword = async (req, res) => {
  const { token, email, password } = req.body
  if (!email || !token || !password) {
    throw new CustomError.BadRequestError('Please provide all values')
  }
  const user = await User.findOne({ email })
  if (user) {
    const currentDate = new Date(Date.now())
    if (
      user.passwordToken === token &&
      user.passwordTokenExpirationDate > currentDate
    ) {
      user.password = password
      user.passwordToken = null
      user.passwordTokenExpirationDate = null
      await user.save()
    }
  }
  res.status(StatusCodes.OK).json({ msg: 'Password reset Successful' })
}

setting new password

this message appears briefly and then shows below page

now entering new password

logged in with new password

Step 8 (Hash token)

passwordToken

When user clicks on "Reset Password" button in front-end, the forgotPassword controller is hit

passwordToken

• In this controller, in line 179 above,

  • the passwordToken is generated

  • and in line 182, it is sent to the user - SECURE as now one can see this email (and token inside that) except the legit user himself

  • and in line 191, it is saved in the DB - NOT SECURE (in this section we will see why and how we need to hash this)

8.1 Why to hash passwordToken before sending to DB

Let's say John ([e-mail]) wants to do a password reset so he clicks on "Forgot Password" in front-end. Now the forgotPassword controller creates a passwordToken, saves (un hashed passwordToken) in DB and also sends the same to John's email.

After the email is sent, let's say John didn't open his email, and meanwhile a hacker hacked the DATABASE and got access to his un hashed passwordToken from DB. Now this hacker can construct the URL with the passwordToken he stole from DB and enters that URL in browser. This way he can now reset John's password and keep using that. So hacker will now become John.

If passwordToken is hashed before stored in the DB, the hacker cannot construct the URL with token so he can't reset John's password. That's the reason we need to hash the passwordToken before sending to DB.

Once the legit user clicks on email link sent by server to reset his password, in the resetPassword controller, we can get the un hashed passwordToken and hash it and then compare it with DB passwordToken (which is already hashed by forgotPassword controller). Explained below in 8.2.

Now you may ask, we didn't hash verifyEmaiToken, so what about that.

If you think about it, it's not required because, if hacker gets that un hashed verifyEmailToken from DB, he can construct URL with that and with that, the user gets verified. There's no use for hacker by verifying users email.

8.2 How to hash passwordToken before sending to DB

We will again use crypto library for hashing passwordToken.

Let's create a file called createHash.js in utils folder. Here, we will have function that takes in normal string (un hashed token string) and returns a md5 hashed value. This is one way and can't be reversed. If we need to later compare the user sent string (un hashed passwordToken from front-end ) to resetPassword controller then we can hash user sent string and then compare to hashed one in DB.

utils/createHash.js

utils/createHash.js

const crypto = require('crypto')

// hashing is one way. Once we hash something then we can't reverse that.
// We can only compare it with a hashed value. There is no way to access original

const hashString = (unhashedToken) =>
  crypto.createHash('md5').update(unhashedToken).digest('hex')

module.exports = hashString

utils index.js

forgot-password controller

controllers/authController.js partial

const forgotPassword = async (req, res) => {
  const { email } = req.body
  if (!email) {
    throw new CustomError.BadRequestError('Please provide a valid email')
  }
  const user = await User.findOne({ email })
  if (user) {
    const passwordToken = crypto.randomBytes(70).toString('hex')
    // send email for password reset
    const origin = 'http://localhost:3000'
    sendResetPasswordEmail({
      name: user.name,
      email: user.email,
      token: passwordToken,
      origin,
    })

    const tenMinutes = 1000 * 60 * 10
    const passwordTokenExpirationDate = new Date(Date.now() + tenMinutes)

    // hashing the passwordToken before sending it to DB
    // But sending the unhashed passwordToken to the user. DB is what we need to secure
    user.passwordToken = createHash(passwordToken)
    user.passwordTokenExpirationDate = passwordTokenExpirationDate
    await user.save()
  }

  // we will send this success response if the user exists or not for security reasons as explained above
  res
    .status(StatusCodes.OK)
    .json({ msg: 'Success! Please check you email for password reset link' })
}

Comparing hash in resetPassword controller

compare hash in resetPassword controller

controllers/authController.js - partial

const resetPassword = async (req, res) => {
  const { token, email, password } = req.body
  if (!email || !token || !password) {
    throw new CustomError.BadRequestError('Please provide all values')
  }
  console.log('The email, token and pass is', email, token, password)
  const user = await User.findOne({ email })
  if (user) {
    console.log('Before user update', user)
    const currentDate = new Date(Date.now())
    console.log('The current date is', currentDate)
    console.log('The expiration date is', user.passwordTokenExpirationDate)
    console.log(user.passwordTokenExpirationDate > currentDate)
    if (
      // since token sent by user is plain string and user.password(stored in db) is hashed one, we can't compare directly
      // first we need to hash the user sent one and then compare. We can't un hash the user.passwordToken because reversing is not possible as I explained
      user.passwordToken === createHash(token) &&
      user.passwordTokenExpirationDate > currentDate
    ) {
      user.password = password
      user.passwordToken = null
      user.passwordTokenExpirationDate = null
      console.log('The new user is', user)
      await user.save()
    }
  }
  res.status(StatusCodes.OK).json({ msg: 'Password reset Successful' })
}

Congrats on completing this project 😎🚀