Working Through Rust Web Programming pt 3: Persisting Data With PostgreSQL

Introduction

In this series, I will be working through this book on Rust Web Programming. The book is quite large and in depth, but in these articles I will be distilling the big picture pieces of the book into bite size tutorials and walk throughs.

In this section, we will be overhauling the JSON state management we hacked together in part 1 in favor of a more production-ized postgreSQL database. PostgreSQL manages concurrent database actions for us, enables us to use the functionality of SQL on our database and decouples the database itself from the web app.

The backend work will be tracked in this github project https://github.com/mattmacf98/web_app. This specific work is achieved in this commit.

Setting up our Postgres Service in Docker

Docker

The first section of this chapter does a very nice overview of Docker. We go over some must know Docker commands like…

• docker compose up (to spin up our docker-compose file)
• docker container ls -a (to see the list of running containers)
• docker exec -it <container id> psql -U <username> <db> (to access psql on our docker instance

After we are finished exploring Docker, we ultimately create this docker-compose yml

version: "3.7"

services:
  postgres:
    container_name: "to-do-postgres"
    image: 'postgres:11.2'
    restart: always
    ports:
      - '5432:5432'
    environment:
      - 'POSTGRES_USER=username'
      - 'POSTGRES_DB=to_do'
      - 'POSTGRES_PASSWORD=password'
      - 'POSTGRES_HOST_AUTH_METHOD=trust'

This yml creates a single postgres service named “to-do-postgres” using the starting point image postgres:11.2 from docker hub. Then we expose port 5432 — postgres’s default port– in the postgres container as port 5432 in our local host. Finally, we provide a bunch of environment variables to allow use to access postgres inside the container.

Diesel

The book recommends Diesel to manage our database schemas and migrations. Diesel handled migrations very well in a straightforward way, it would create up and down sql files where we could isolate exactly what changed between different database versions.

up.sql

CREATE TABLE to_do (
    id SERIAL PRIMARY KEY,
    title VARCHAR NOT NULL,
    status VARCHAR NOT NULL,
    date timestamp NOT NULL  DEFAULT NOW()
)

down.sql

DROP TABLE to_do

Diesel relies on a .env file in your root directory to tell it what the database url is (the commit I referenced has it in src which is wrong, I fix this in a later commit).

DATABASE_URL=postgres://username:password@localhost/to_do

The piece with Diesel I struggled with was actually just the setup for the cli. I eventually got it working by setting an environment variable for Dynamic Linked Libraries. If you get an error like Symbol Not Found _PQClear, check out my Stack Overflow answer here.

Summary

• Docker is a nice & easy way to spin up services for things like postgres
• Diesel can be used to manage postgres databases in Rust

Connecting our App with our Postgres Service

The book sets up data models, then creates configuration and then uses a connection pool to properly connect to the database. I believe this is done to highlight the step by step improvements configuration and database pools make on our service. However, I am going to structure this walkthrough in a way that makes logical sense for actually implementing this all together: config -> database pool -> data models.

Configuration

We are going to want to have some things easily accessible and configurable in our web app, now is a good time to set up a config store to manage these keys and values.

config.yml

DB_URL: postgres://username:password@localhost/to_do

config.rs

use std::collections::HashMap;
use std::env;

use serde_yaml;

pub struct Config {
    pub map: HashMap<String, serde_yaml::Value>
}

impl Config {
    pub fn new() -> Config {
        let args: Vec<String> = env::args().collect();
        let file_path = &args[args.len() - 1];

        let file = std::fs::File::open(file_path).unwrap();
        let map: HashMap<String, serde_yaml::Value> = serde_yaml::from_reader(file).unwrap();
        return Config {map};
    }
}

Our config is just going to have a map of the key in the config and it’s value. When we create a new config, we take in the last argument (i.e. if we ran cargo run config.yml, the last arg would be config.yml) and we use that as our config file.

We open this file and generate our key-value store for all our config values, right now we just have DB_URL.

Database Pool

In the book, this was originally set up so that every action created a new database connection on request, this is not efficient and could lead to overwhelming our postgres server. I am going to start out with the database pool.

use diesel::prelude::*;
use crate::config::Config;

use actix_web::dev::Payload;
use actix_web::error::ErrorServiceUnavailable;
use actix_web::{Error, FromRequest, HttpRequest};
use futures::future::{Ready, ok, err};
use diesel::{
    r2d2::{Pool, ConnectionManager, PooledConnection},
    pg::PgConnection
};
use lazy_static::lazy_static;

type PgPool = Pool<ConnectionManager<PgConnection>>;

pub struct DbConnection {
    pub db_connection: PgPool
}

lazy_static! {
    pub static ref DBCONNECTION: DbConnection = {
        let connection_string = Config::new().map.get("DB_URL").unwrap().as_str().unwrap().to_string();

        DbConnection {
            db_connection: PgPool::builder().max_size(8)
            .build(ConnectionManager::new(connection_string))
            .expect("Failed to create db connection pool")
        }
    };
}

pub fn establish_connection() -> PooledConnection<ConnectionManager<PgConnection>> {
    return DBCONNECTION.db_connection.get().unwrap();
}

pub struct DB {
    pub connection: PooledConnection<ConnectionManager<PgConnection>>
}

impl FromRequest for DB {
    type Error = Error;
    type Future = Ready<Result<DB, Error>>;

    fn from_request(_: &HttpRequest, _: &mut Payload) -> Self::Future {
        return match DBCONNECTION.db_connection.get() {
            Ok(connection) => ok(DB { connection }),
            Err(_) => err(ErrorServiceUnavailable("Could not make connection to database"))
        }
    }
}

We have two structs here, the DbConnection holds our connection pool and our DB is a wrapper which asks our DbConnection for a connection from its pool.

DbConnection has one parameter which is the postgres pool. We have a single static DB_CONNECTION of type DbConnection, this value is a lazy static meaning it only gets constructed when we need it to be (so if we never request a db connection, we never waste compute power to instantiate it).

When it is needed, we use our config to get our DB_URL and then use that to set up a connection pool to our postgres service with max 8 connections at a time.

DB has a single connection to the postgres service, we implement FromRequest for DB. If you remember we did this for JWT in part 1, this allows us to automatically generate a DB item from an HTTPRequest passed into our views. It’s construction simply asks our connection pool manger for an available connection to the postgres service. If none are available, we get an error.

Data Models

Now we can get into creating our data models and using them in our views to interact with our postgres service.

Diesel will automatically generate a schema.rs file for us when we create tables.

// @generated automatically by Diesel CLI.

diesel::table! {
    to_do (id) {
        id -> Int4,
        title -> Varchar,
        status -> Varchar,
        date -> Timestamp,
    }
}

We will reference this schema in our data models.

The book splits the to do items into two structs: new_item and item, the main difference is that item is just for querying and has a serial id generated by postgres, while new_item is strictly for insertion and does not have the id. I am not sure if I would always opt for this separation of structs — I am sure you could get by with just one struct– but this does allow for extreme specificity and restrictions on what you can insert vs query from the table. Below is the Item schema.

use chrono::{NaiveDateTime};
use crate::schema::to_do;

#[derive(Queryable, Identifiable)]
#[table_name="to_do"]
pub struct Item {
    pub id: i32,
    pub title: String,
    pub status: String,
    pub date: NaiveDateTime
}

We define its capabilities (Queryable and Identifiable) using Diesel macros. Importantly we use crate::schema::to_do and apply the table_name macro to indicate this data model maps to the todo schema. Rust compilation will not catch it if you leave out the use for the schema, but you will get runtime problems if it is not there.

Now that we have our schema and data models set up, we can use them inside our views. Below is an example of how create uses the to_do item models.

use crate::diesel;
use diesel::prelude::*;
use actix_web::{HttpRequest, HttpResponse};
use crate::database::{DB};
use crate::json_serialization::to_do_items::ToDoItems;
use crate::models::item::item::Item;
use crate::schema::to_do;
use crate::models::item::new_item::NewItem;

pub async fn create(req: HttpRequest, db:DB) -> HttpResponse {
    let title: String = req.match_info().get("title").unwrap().to_string();

    let items = to_do::table
        .filter(to_do::columns::title.eq(&title.as_str()))
        .order(to_do::columns::id.asc())
        .load::<Item>(&db.connection)
        .unwrap();

    if items.len() == 0 {
        let new_post = NewItem::new(title);
        let _ = diesel::insert_into(to_do::table).values(&new_post).execute(&db.connection);
    }

    return HttpResponse::Ok().json(ToDoItems::get_state());
}

There are few things to take not of here

1. We can instantly grab our DB object since it implements FromRequest
2. We use crate::schema::to_do and then do a filter on the table for entries where the to do item title matches the one we are trying to create and reference our Item data model when we load the matches into items
3. We use diesel to create a new item — if there is not a conflicting one– with the help of our db connection.

This is the basic flow for all our views: Get DB -> Query the table -> Preform some action and/or return data. (I know the get view violates this pattern, I update it in a following chapter).

Summary

• Use a config file and struct to easily manage global variables we might want to change
• Diesel generates a schema.rs file for our tables, we must then create Rust structs to reference that schema so we can interact with the table
• In our views, we use: the db connection, our data models and diesel to interact with our postgres server

Conclusion

After completing this chapter, we now have a good idea of how to create scalable data management for production grade apps. We are well on our way to developing a great framework to build highly scalable projects in Rust. In the next section, we are going to dive into Authentication and Authorization.