How to Use Socket.io with Express

socket.io allows us to make a connection between a client and a server but have that connection persist.

Normally, the client and server would have a simple transaction: the client would get data from the server then close the connection. If the client needs 10 pieces of information they have to make 10 requests. With WebSockets, the client can connect to the server and have that connection stay open. So the client can make 10 requests with just 1 connection.

Socket.IO is composed of two parts:

  • A server that integrates with (or mounts on) the Node.JS HTTP Server
  • A client library that loads on the browser side

Set up Socket.io on Express Server

Install dependencies: The only packages you need are Express and Socket.io

npm install express socket.io

Import packages

const http = require('http');
const express = require('express');
const socketio = require('socket.io');

Create io object

const app = express();
const server = http.createServer(app);
const io = socketio(server);

http.createServer is used by Express under the hood but we need to access it directly

Socket.io Methods on the Server

This is a function that runs every time a client connects to our server. In the callback, we create a socket instance for each one of the clients

We can access the socket.id, a random ID that is automatically assigned to them when they join the server.

// Run when client connects
io.on('connection', socket => {
   console.log('New WS Connection...');
});

socket.on listens for events from the client

The string "sendMessage" is a custom event emitted from the client whenever a new message is sent. socket.on allows us to access that event and any data sent with it (message) on the server.

socket.on("sendMessage", message => {
   console.log(message)
})

io.emit sends an event down to every client connected to the server

In this example, we are taking the message that was sent to the server with "sendMessage" and emitting it to all the connected sockets through an event called "recieveMessage".

socket.on("sendMessage", message => {
   io.emit("recieveMessage", message)
})

In order to send a message to every socket except the socket sending the message, use socket.broadcast.emit

socket.broadcast.emit("recieveMessage", message)

Emit message to a room with socket.to in front of the emit command

socket.on("sendMessage", (message, room) => {
   socket.to(room).emit("receiveMessage", message)
})

To join a room, use the socket.join method. You can only join a room from the server, not the client.

socket.on("joinRoom", room => {
   socket.join(room)
})

Send a callback function to the server.

On the client, pass the callback function as the last parameter.

socket.emit("joinRoom", room, message => {
   displayMessage(message)
})

On the server, use the callback function. Basically, you’re doing this: displayMessage(`Joined ${room}`)

socket.on("joinRoom", (room, cb) => {
   socket.join(room)
   cb(`Joined ${room}`)
})

Disconnect from the server

socket.on('disconnect', () => {
   //...
});

Socket.io on the Client

In this tutorial, we are assuming you are using static HTML pages as the frontend. So no JavaScript frameworks like React or Vue.

If Socket.io is installed in the root directory, you can use the client library in our public files by simply adding this line to your main HTML file

<script src="/socket.io/socket.io.js"></script>

You can create a Socket object with this command:

const socket = io()

If Socket.io is NOT installed in the root directory, you must install the socket.io-client package in your client folder.

npm i socket.io-client

Then, in a JavaScript file you can import io from the library

import { io } from 'socket.io-client'

Then, create a socket on the frontend by calling the io function and passing in the address of the server.

const socket = io('http://localhost:3000')

Also, on the server, make sure you add the following parameters to the require('socket.io') function. This will prevent CORS from blocking the frontend from connecting to the backend. The URL should be the URL that your frontend application is running on.

const io = socketio(server, {
  cors: {
    origin: ['http://localhost:8080']
  }
});

Use socket.on to receive events from the server

// Receive message from server and output it to screen
socket.on('message', (message) => {
  outputMessage(message);
});

// Example 2
socket.on('roomUsers', ({ room, users }) => {
  outputRoomName(room);
  outputUsers(users);
});

Use socket.emit to send events to the server

// Tell server you want to join a chatroom
socket.emit('joinRoom', { username, room });

Serverless Lambda Functions with Netlify

What is Serverless?

Serverless computing refers to a backend architecture in which you are not responsible for maintaining your own server to run server-side code.

Instead, you write functions, upload them to a Serverless computing service (ex. AWS Lambda), and the service will run the functions when called.

The service you are using (ex. AWS Lambda) takes care of all the infrastructure for you. So “Serverless” does not mean no servers are involved. Instead, it means you don’t have to deal with the headache of taking care of the server(s) yourself.

Netlify Lambda

Netlify Lambda is powered by AWS Lambda, but just much simpler.

We are going to be using Netlify Lambda to build a simple application.

Build a simple application

Note: must have Node.js Version 10 or later

1) Create a new React app

2) Install the Netlify CLI

You have 2 options for installing the CLI:

Option 1: Install it globally

npm install netlify-cli -g

Option 2: Install it as a dev dependency in project

npm install netlify-cli --save-dev

3) Create a functions folder and make a new .js file inside

This is where we store all of our Lambda functions

I will call my file sayHello.js

4) Inside the file, write this basic boilerplate

Each JavaScript file holds one Lambda function.

The function must export a handler method with the following general syntax:

exports.handler = async function(context, event) {
}

Netlify provides the event and context parameters when the Serverless function is called.

When you call the Serverless function endpoint, the handler receives an event object.

The context parameter includes info about the context in which the Serverless function was called.

Because async functions return a Promise, Netlify recommends returning a response with HTTP status code instead of allow function to time out

exports.handler = async function(context, event) {
   return {
        statusCode: 200,
        message: JSON.stringify({msg: "Hello world"})
    }
}

5) Create netlify.toml file at the root level of the directory

This is a configuration file on how Netlify will build and deploy the site.

command: the command that starts the dev server

functions: where we store our Serverless functions

[build]
command = 'npm run build'
functions = 'functions'
publish = 'build'

6) Run the project

In the project directory, run the command netlify dev. This starts a local dev server

If you append /.netlify/functions/sayHello to the URL, the server will send the message defined earlier ({msg: “Hello world”})

The full URL:

localhost:4000/.netlify/functions/sayHello

7) Call function in frontend

This is an example of how you could access the data with axios. You would probably use React Hooks to asynchronously get the data and put it in a view.

const results = await axios.get('/.netlify/functions/sayHello')

As you can see, you use a local URL in the GET request

All About Logging

What is an application log?

There are multiple different types of logging including system logs, server logs, garbage collector logs, and application logs.

An application log is a file that contains information about events that have occurred within a software application. Every application produces logs which hold information about what the application is doing.

What type of information is included in a log? It varies by the application. The developers of the software application control what goes into the log file, but there are some commonly included components:

  • context info: Background information that provides insight into the state of the application at the time of the message
  • timestamps: A specific piece of contextual information for tracking and correlating issues
  • Log levels: Labels that help you calculate the level of importance for the entries in your log file. Frequently used levels include INFO, WARN, and ERROR

Why are logs important?

The main use for logs is diagnostics. Logs hold information that can help with

  • debugging
  • analytics
  • compliance
  • security

Logging Agent vs Logging Library

A logging agent is a separate service that reads logs from one location and sends them to another location. Think of it like a funnel.

Some benefits of logging agents include:

  1. minimal setup & config
  2. multi-threading & failure safety
  3. no code
  4. scalable (single agent instance can log for almost any number of applications)
  5. modify/update/replace agents w/o having to take down application
  6. format-agnostic

Some drawbacks of logging agents include:

  1. require interaction w/ host (not good w/ Serverless)
  2. requires CPU compute usage to track log files

A logging library (or logging framework) is code that you embed into your application to create and manage log events.

Some benefits of logging libraries include:

Some drawbacks of logging libraries include:

  • usually synchronous (application must wait for the library to finish writing a log event before continuing)
  • no failure safety (libraries only run as long as your application is running)
  • not scalable (each application runs its own instance of the library)

Logging in Node with Winston

What is Winston?

Winston is a simple and universal logging library.

Why use a logging library?

When developing an application, sometimes we need to get information about how the application is running. We can use functions like console.log, console.error, console.warn to accomplish this. A logging library, such as Winston, allows us to output similar types of data except it can send this data (log data) to locations other than just the console. For example, an external file.

The basic idea of logging

Basically, you have a logger that writes to an arbitrary amount of transports. A transport is essentially a storage device for your logs. Some common transports include:

  • stdout (console)
  • file
  • HTTP
  • Datadog (log aggregator)

Log levels

Log levels allow us to categorize the entries in a log file by urgency. The higher up it is, the more severe it is. As you can see, fatal is the most severe.

These are npm log levels:

{
   fatal: 0,
   error: 0,
   info: 0,
   http: 0,
   verbose: 0,
   debug: 0,
}

Setting up a simple logger in Winston

The recommended way to use winston is to create your own logger. The simplest way to do this is using winston.createLogger.

Install Winston in project

npm install winston

In a new file, create a logger with winston.createLogger

const winston = require('winston');

const logger = winston.createLogger({

});

module.exports = logger;

Add custom properties to logger

const winston = require('winston');

const logger = winston.createLogger({
   format: winston.format.simple(),
   transports: [new winston.transports.Console()],
});

module.exports = logger;

The format property controls how the logs are displayed visually.

transports is an array of the places (transports) to send the log data to. In the above example, our only transport is the console.

Use the logger

const logger = require('./logger')   // access the file you created earlier

logger.info('text info');
logger.warn('text warn');
logger.error('text error');
logger.debug('text debug');

The output should look something like this:

Why is the debug log not showing up? The default log level for a Winston logger is info. This means that Winston will only log the info level and all levels ABOVE it (warn, error, etc.)

Creating custom log format in Winston

Destructure Winston import

This is just for our own ease of use. No change to the logic.

const { createLogger, format, transports } = require('winston');
const { timestamp, combine, printf } = format;

const logger = createLogger({
  format: format.simple(),
  transports: [new transports.Console()]
});

Create custom format

const myFormat = printf(({ level, message, timestamp }) => {
  return `${timestamp} ${level}: ${message}`;
});

Add the format to your logger

const logger = createLogger({
  format: combine(timestamp(), myFormat),
  transports: [new transports.Console()]
});

Final code

const { createLogger, format, transports } = require('winston');

const myFormat = printf(({ level, message, timestamp }) => {
  return `${timestamp} ${level}: ${message}`;
});

const logger = createLogger({
  format: combine(timestamp(), myFormat),
  transports: [new transports.Console()]
});

Here is another custom log format

const logger = createLogger({
  format: combine(
       format.colorize(),
       timestamp({ format: 'YYYY-MM-DD HH:mm:ss' }), 
       myFormat
  ),
  transports: [new transports.Console()]
});

How to Embed Google Form in Website

This is how you embed a Google Form in an HTML page so that users can interact with the form directly in the page rather than having to press a link and navigate to the form in a separate window.

In the Google Form page, press the “Send” button at the top right

On the “Send form” popup, navigate to the Embed HTML tab.

Update the width and height of the form to fit the window you are going to put it in

Copy the iframe code

Paste it into the code for the HTML page you are putting the form in.

<body>

<h1>This is a form</h1>

<iframe src="https://docs.google.com/forms/d/e/1FAIpQLSf9JDsOyHEaDHD5PXQ2wzyt-th_dzUQmoCB7ad346zez1PYnA/viewform?embedded=true" width="640" height="1395" frameborder="0" marginheight="0" marginwidth="0">Loading…</iframe>

</body>

Django Error While Trying to Create Superuser

Error

When I tried to run this command: python manage.py createsuperuser in my Django project, it gave me a super long error which included the following:

File “manage.py”, line 22, in
main()
File “manage.py”, line 18, in main
execute_from_command_line(sys.argv)
File “/Users/username/.local/share/virtualenvs/django-learn-csZU2bYZ/lib/python3.8/site-packages/django/core/management/init.py”, line 364, in execute_from_command_line
utility.execute()

….

ImportError: cannot import name ‘path’ from ‘django.urls’ (/Users/username/.local/share/virtualenvs/django-learn-csZU2bYZ/lib/python3.8/site-packages/django/urls/init.py)

Solution

The problem in my case was that I had the wrong version of Django installed. In order to run the python manage.py createsuperuser command you need at least Django 2.

Run this command to check the version of Django you are currently working with:

python -m django --version 

Run this command to install the latest version of Django.

pip install --upgrade django

Run the python -m django --version command again and make sure your new Django version is 2.x or above. If so, you should be able to run python manage.py createsuperuser now

How to Set Up a New Django Project

Django is a high-level Python web framework that enables rapid development of secure and maintainable websites. It is one of the most popular Python backend development frameworks. It is known for its security, simplicity, and reliability. It is without a doubt one of the best frameworks to learn.

Note: You must have Python 3 or later installed on your computer to complete this tutorial.

Create a new empty project. The first thing we need to do is install Django in our new project. You might be tempted to do it using the command pip install django, but I would advise against this because it installs the Django package on your global system rather than just in this specific project.

Instead, you should use a tool called pipenv, which allows you to set up a virtual environment for each Python project that you create on your system. Install it globally by running this command:

pip install pipenv

With pipenv installed, run this command in your project folder to create a virtual environment.

pipenv shell

Any packages we install, including Django, will be installed into this virtual environment rather than on our global system. This command will also generate a Pipfile in your project directory. This Pipfile will list any packages that you install in the virtual environment.

Run this command to install Django in the virtual environment

pipenv install django

In Django, a project is basically the overall website/application. Then, you have the concept of apps; there are usually multiple apps in each project.

Run this command to create a new project (put your project’s name in place of project-name):

django-admin startproject project-name 
cd project-name 

Inside your new project folder, you should see a second folder with the same project name and a manage.py file. You rarely change any of the code in the manage.py file, but you will use this file all the time. We use this file to run the server, create migrations, and carry out many other commands.

Once inside the project folder we just created, run this command to run the server on http: 127.0.0.1:8000 (port 8000):

python manage.py runserver
# ctrl+c to stop

To run the server on a different port

python manage.py runserver 8081

You will probably get the below error. The cause of the error is that there are migrations ready to create the default, necessary database tables that Django uses that have not been applied.

To apply them, we run this command:

python manage.py migrate

Now, your server should be up and running with no errors. If you type localhost:8000 (or whichever port your project is running on) in your browser, you should see this default Django page.

You’ve officially created your first Django project. The next step is to start adding apps to your project. Stay tuned for a tutorial on that.

Python Tutorial: Web Scraping with BeautifulSoup and Requests

Web scraping is the process of automatically extracting data from a website.

You will need an understanding of basic HTML page structure in order to grasp this tutorial. Web scraping with BeautifulSoup to get data/text from a page is done by referencing specific HTML semantic tags, classes, and ids on that page and getting the data from within them. You will better understand what I mean as we continue.

BeautifulSoup is the Python package we are going to use to do the web scraping. Requests is a package that allows you to send HTTP/1.1 requests extremely easily.

In this tutorial, we are going to be parsing the information from the daily forecast element on the Orlando, FL weather page on weather.com (this is the link). We will transfer the information to a structured, table format using Pandas.

The picture below shows the div element which holds the daily forecast information for the week. When you view this tutorial, the numbers will probably not be the same but, other than that, you should see something that looks like this. This is the element we are going to be scraping.

In Terminal, create a new folder with mkdir, cd into the folder, and create a .py file. I will call mine scraper.py. This file will contain the code for the scraper.

In the next lines, we are installing some necessary dependencies using pip, the Python package manager, into our project folder.

mkdir project-folder
cd project-folder
touch scraper.py
pip install beautifulsoup4
pip install requests
pip install pandas

Open up the scraper.py file in a Code Editor.

First, we will add the import statements to our file for the packages we just installed using pip:

from bs4 import BeautifulSoup
import requests

Ok, let’s start writing the code. First, write the following line:

page = requests.get('https://weather.com/weather/today/l/9ca5fcd4263a24d4d3aaea0c6ab0aea6bea876cfce908ee624588a8f269f6fa1')

This line uses the request package to get the HTML page of the URL in quotes. In this case, we are using a page on the site.

Next, write this line:

soup = BeautifulSoup(page.content, 'html.parser')

This line initializes BeautifulSoup in our project. Now, we can use the soup variable to access the functions within the BeautifulSoup package. The first parameter in the statement is what you want to scrape (in this case, the content of the page we got earlier) and the second parameter just tells BeautifulSoup that it will be parsing HTML.

Now, if you are using Google Chrome, you can Inspect the page and find the id of the div element which holds the data you want to scrape.

As you can see from the above image, I found that the div containing the Daily Forecast information had an id of “WxuDailyWeatherCard-main-bb1a17e7-dc20-421a-b1b8-c117308c6626”

After getting the id of the div element which holds the data we want, we are defining a reference to the div element with the following line:

week = soup.find(id='WxuDailyWeatherCard-main-bb1a17e7-dc20-421a-b1b8-c117308c6626')

Now we can use the variable week to reference the Daily Forecast div.

Using Inspect again, I found that the actual content within the div element is stored in a elements with a class of “Column–innerWrapper–3K14X”.

We can get access to all 5 (one for each day of the week) of these a tag items using the find_all function.

items = week.find_all(class_='Column--innerWrapper--3K14X')

Now, we can get access to the data within the items array. We will make an array of the dates within the items array using this line:

date = [item.find('h3', class_='Column--label--L3RrD').get_text() for item in items]

Using the same process as before, I found that all the dates were stored in a h3 with a class of ‘Column–label–L3RrD’. The above line creates an array of dates by running the find() method on every item within the items array.

Now, we can do this for the temperature and chance of rain data stored in the items array.

temp = [item.find(class_='Column--temp--2v_go').get_text() for item in items]

chanceOfRain = [item.find(class_='Column--precip--2H5Iw').get_text() for item in items]

Now, you can access the date, temperature, and chance of rain data with the variables we just created!

Pandas Addition

As an extra, you can put the weather data we just scraped into a simple Panda’s data frame so that it looks better.

First, install the Pandas package and import Pandas at the top of your scraper.py file.

import pandas as pd

Under the data definitions, write this line. This line creates a data frame in the variable weather_info with the data.

weather_info = pd.DataFrame({
    'Date': date,
    'Temp': temp,
    'Chance Of Rain': chanceOfRain
})

You can print the weather info to see your data outputted in a table format

print(weather_info)

And that’s it! You’re done. Here is the full code:

Here is the expected output– a Pandas data frame. You can do some research and see how to make it look better:

Helpful Terminal Commands

Using the Terminal or Command Line will prove to be a super handy and important aspect of your software development process. Here are a few commands that will most definitely be useful for you when using the Terminal. I’m using a Mac.

Make a new folder

mkdir folder-name

Go into folder (only up 1 level)

cd folder-name

To cd into a folder with a space in the name, use the backslash character between the words

cd Software\ Development

Go backwards by one level in the folder system

cd ..

Go to your computer’s root directory

cd ~

Clear the Terminal window (just for declutter)

clear

See the contents of a folder

ls

See the hidden files in a folder

ls -a

Copy file to a different directory. This line copies the file file-name.txt to the directory/file one above the current directory/file.

cp file-name.txt ../

Rename a file

mv file-name.txt NEW-FILE-NAME.txt

Remove file from current directory

rm file-name.txt

Shows where an application lies in your file system. In this example, we are seeing the location of the Git application. This command will either return the file system location if the application exists on your machine or no output if there is no application with that name on your machine

which application-name
which git

Shows where you are currently in your file system

pwd

Basically a print statement but in the Terminal

echo hello

Make a new file and put the defined text inside of it. In this case, we put the text hello in a file called hello.txt

echo hello > hello.txt

Setting up a Free Postgres Database in Heroku w/ PgAdmin

Step 1: Log into Heroku

Step 2: Create a New App

Create a new Heroku app by either clicking “Create a new app” in the center of the screen or “New” at the top right of the screen.

Choose an App Name and the Region (either the US or Europe) for your new app. The name can be anything you want. Then, just click the Create app button.

Step 3: Add a PostgreSQL Database

To attach a PostgreSQL database to the blank app you just made, you need to go to the Resources tab in the header of the new app’s dahsboard. Then type Heroku Postgres into the Add-ons search field. When shown, select the suggested Heroku Postgres add-on from the dropdown:

The next popup asks you to choose a pricing plan for the database. Select the Hobby Dev – Free plan and click Provision.

Now your PostgreSQL database is up !

Step 4: Get Heroku DB Information

To find the credentials and the connection URL for the PostgreSQL database, make sure you’re still in the Resources tab and click on the Heroku Postgres resource we just added.

That should bring you to this screen:

Select the Settings tab in the header of that screen. You will be navigated to a page where Here, you can click the View Credentials button to see the credentials of your PostgreSQL database.

You will need these values to input into your PgAdmin interface

Step 5: Configure PgAdmin w/ Heroku DB Credentials

Open up PgAdmin

Right-click on Servers at the very top of the left-side panel

Choose Create > Serve

Fill out the new server form with all of the corresponding information from the Heroku View Credentials page.

After you save the new server, it will become populated with a huge list of databases. You will not have access to any of them except the one with your Database name from Heroku. Scroll through and find this one.

Then, as you would with any other PgAdmin database, go to Schemas > public > Create new Table. Populate this table with the fields you want.

You’re all set.