NAV Navbar
Biff
A web framework + self-hosted deployment solution for Clojure
Jacob O'Bryant

Introduction

Biff is designed to make web development with Clojure as easy as possible without compromising on simplicity. The main target audience is early stage startups and hobbyists, where speed in the beginning really matters. Biff is also made to be easy to take apart: as your project grows and your requirements become more complex, you can peel back Biff's layers until you have the level of flexibility you need.

Biff is still fairly young, and there may be breaking changes. I've been using it in production for Findka since May 2020. To help Biff grow and to help me discover what needs improvement, I'm giving free one-on-one mentoring (pair programming, code review, design help, etc) to anyone who wants to learn Clojure web dev with Biff (as my schedule allows). If you're interested, fill out this quick survey.

Core features (a few of these were inspired by Firebase):

Resources

Contributing

The most helpful way to contribute is to use Biff and let me know what problems you run into. You can also write tutorials or blog about your experience. I'd be happy to list your articles under Resources and promote them myself, not that I have a large following.

PRs are welcome too, especially if you want to tackle some of the current issues. If you're planning something significant, you might want to bring it up in #biff on Clojurians Slack.

The easiest way to hack on Biff is to start a new project (see Getting Started) and then change the Biff dependency in deps.edn to {:local/root "/path/to/cloned/biff/repo" ...}. Then just run ./task init; ./task dev. Eval (biff.core/refresh) as needed.

Documentation

You'll need Ruby; then run:

cd slate
gem install bundler
bundle install
cd ..

After that, you can run ./task docs-dev and edit slate/source/index.html.md to work on the documentation. See the Slate README.

Getting started

Requirements:

Run this command to create a new Biff project:

bash <(curl -s https://raw.githubusercontent.com/jacobobryant/biff/master/new-project.sh)

That script will create a minimal, working CRUD app which demonstrates most of Biff's features. You'll be guided through the process of starting the app, trying it out, and exploring the code. You can refer back to the documentation here as needed. When you're ready to deploy, check out Deployment.

Overview

Project structure

Here's an example of running the new-project.sh script above:

$ bash <(curl -s https://raw.githubusercontent.com/jacobobryant/biff/master/new-project.sh)
Creating a new Biff project. Available project types:

  1. SPA (single-page application). Includes ClojureScript, React, and
     Biff's subscribable queries. Good for highly interactive applications.

  2. MPA (multi-page application). Uses server-side rendering instead of
     React etc. Good for simpler applications.

Choose a project type ([spa]/mpa): spa
Creating a SPA project.
Fetching latest Biff version...
Enter name for project directory: example
Enter main namespace (e.g. example.core): example.core
Enter the domain you plan to use in production (e.g. example.com),
or leave blank to choose later: example.com

Your project is ready. Run the following commands to get started:

  cd example
  git init
  ./task init
  ./task dev

Run `./task help` for more info.
$ cd example/
$ tree
.
├── all-tasks
│   ├── 10-biff
│   └── 20-example
├── config
│   ├── deploy-key
│   ├── main.edn
│   ├── ssh-public-key
│   └── task.env
├── deps.edn
├── infra
│   ├── provisioners
│   │   ├── 00-config
│   │   ├── 10-wait
│   │   ├── 20-dependencies
│   │   ├── 30-users
│   │   ├── 40-app
│   │   ├── 50-systemd
│   │   ├── 60-nginx
│   │   └── 70-firewall
│   ├── run-provisioners.sh
│   ├── system.tf.json
│   └── webserver.json
├── resources
│   └── www
│       └── js
│           ├── ensure-signed-in.js
│           └── ensure-signed-out.js
├── shadow-cljs.edn
├── src
│   └── example
│       ├── client
│       │   ├── app
│       │   │   ├── components.cljs
│       │   │   ├── db.cljs
│       │   │   ├── mutations.cljs
│       │   │   └── system.cljs
│       │   └── app.cljs
│       ├── core.clj
│       ├── handlers.clj
│       ├── jobs.clj
│       ├── routes.clj
│       ├── rules.clj
│       ├── static.clj
│       └── triggers.clj
├── tailwind.config.js
├── tailwind.css
└── task

Tasks

all-tasks/ contains Bash scripts which define project tasks as functions. For example:

all-tasks/10-biff
init () {
  if [ -f package.json ]; then
    npm install
  else
  ...
}

You can run these functions with ./task <name of function>. all-tasks/10-biff contains tasks provided by Biff. You can define new tasks in other files, such as all-tasks/20-example.

Static resources

./task dev starts your app on localhost:8080. Your app will serve files from www/ and www-dev/, which are populated from several sources:

In production, only the first two points apply. Before deploying, you'll use ./task build-assets to add your production CLJS and CSS to resources/www/ so it can be checked into your git repository and deployed from there. (Alternatively, you can download assets from a CI server. I'll make Biff do this by default later.)

Configuration

config/main.edn is read when your app starts. It contains configuration and secrets. The contents of config/ are git-ignored, but Terraform will copy config/main.edn to production when you deploy. (Later Biff will use Hashicorp Vault.)

config/main.edn
{:prod {; Standalone topology is only recommended for development.
        :biff.crux/topology :standalone
        ; Uncomment to use jdbc in production:
        ;:biff.crux/topology :jdbc
        ;:biff.crux.jdbc/dbname "..."
        ;:biff.crux.jdbc/user "..."
        ;:biff.crux.jdbc/password "..."
        ;:biff.crux.jdbc/host "..."
        ;:biff.crux.jdbc/port ...
        :biff/host "example.com"}
 :dev {:inherit [:prod]
       :biff/dev true}}

config/task.env contains configuration needed by ./task. config/deploy-key and config/ssh-public-key are needed for deployment. You'll need to update all of these files before deploying.

Infrastructure

Before deploying, you'll create a VM image (via Packer) with ./task build-image. That task will read from infra/webserver.json, and it will run the scripts under infra/provisioners/. If you need to customize the image, you can add more scripts and re-run ./task build-image.

After that, you can create a server and deploy your app (via Terraform) with ./task tf apply. You'll need to commit and push first. When the server starts, it will fetch the latest commit from your git repository and run your app from that.

You can use ./task deploy instead of ./task tf apply for subsequent deploys, as long as you haven't made any infrastructure changes. ./task deploy will simply restart the app process on the server, causing it to fetch the latest commit again.

App entrypoint

Your app is started by running the -main function from your app's main namespace, e.g. example.core/-main.

src/example/core.clj
(defn start [first-start]
  (let [sys (biff.core/start-system
              {:biff/first-start first-start
               ...}
              biff.core/default-spa-components)]
    (when (:biff/dev sys)
      (biff.project/update-spa-files sys))
    (println "System started.")))

(defn -main []
  (start true))

Some of the files discussed in the previous section are managed by Biff (specifically, all-tasks/10-biff and everything under infra/). When your app starts, the biff.project/update-spa-files will write to those files. This means that when you update Biff (by changing the :sha value in deps.edn), those non-Clojure files will also get updated. You shouldn't change any of those files by hand, because your changes will get overwritten.

biff.core/start-system takes a system map and passes it through a number of component functions. It's kind of like passing a Ring request through middleware functions. The system map includes all the configuration values, using flat, namespaced keys. It also includes any resources or values that components choose to pass on.

Biff's default components do the following:

Each component function receives the system map and then returns a modified version. For example, here's the component which starts Jetty:

biff.components
(defn start-web-server [{:biff.web/keys [handler host port] :as sys}]
  (let [server (jetty/run-jetty handler
                 {:host host
                  :port port
                  :join? false
                  :websockets {"/api/chsk" handler}
                  :allow-null-path-info true})]
    (update sys :biff/stop conj #(jetty/stop-server server))))

When all components have finished, the result is stored in biff.core/system (an atom). During development, you can reload the system by calling (biff.core/refresh) (I recommend binding an editor shortcut to that). That will call all the functions in :biff/stop, reload Clojure files with tools.deps.namespace.repl, and then restart your app with biff.core/start-system.

Decomposing

To a degree, you can modify the behavior of Biff by passing in certain configuration values. When you need more flexibility, you can decompose Biff. For example, you can replace default-spa-components like so:

src/example/core.clj
(require '[biff.components :as c])

(defn start [first-start]
  (let [sys (biff.core/start-system {...}
              ; Add or remove components as needed.
              [c/init
               c/set-defaults
               c/start-crux
               c/start-sente
               c/start-tx-listener
               c/start-event-router
               c/set-auth-route
               c/set-http-handler
               c/start-web-server
               c/write-static-resources
               c/start-jobs
               c/print-spa-help])]
    ...))

And you can replace biff.project/update-spa-files with its body:

(require '[biff.project.infra :as infra])

(defn start [first-start]
  (let [sys (biff.core/start-system {...}
              ...)]
    (when (:biff/dev sys)
      (let [opts (assoc sys ...)]
          (biff.project/copy-files "biff/project/base/{{dir}}/"
            (assoc opts
              :files #{"all-tasks/10-biff"
                       "infra/provisioners/10-wait"
                       "infra/provisioners/20-dependencies"
                       "infra/provisioners/30-users"
                       "infra/provisioners/40-app"
                       "infra/provisioners/50-systemd"
                       "infra/provisioners/60-nginx"
                       "infra/provisioners/70-firewall"
                       "infra/run-provisioners.sh"}))
          (spit "infra/webserver.json"
            (cheshire/generate-string
              infra/default-packer-config {:pretty true}))
          (spit "infra/system.tf.json"
            (cheshire/generate-string
              (infra/default-terraform-config opts) {:pretty true}))))
    (println "System started.")))

This should give you the flexibility you need.

The rest of this documentation covers Biff's individual features in-depth. The fastest way to learn Biff is probably to create a new project and then experiment. You can refer back here when you need more information.

Here's a demonstration of adding a feature to a Biff application (short version):

Configuration

Configuration can be set in code (by passing it in to biff.core/start-system) and in config/main.edn. When Biff reads in config/main.edn, it will merge the nested maps according to the current environment and the value of :inherit. The result is merged into the system map.

The default environment is :prod. This can be overridden by setting the BIFF_ENV environment variable:

BIFF_ENV=dev clj -m example.core

So this:

{:prod {:foo 1
        :bar 2}
 :dev {:inherit [:prod]
       :foo 3}}

would become this:

{:foo 3
 :bar 2}

Here is a complete list of configuration options and their default values. See the following sections for a deeper explanation.

:biff/host "localhost"  ; The hostname this app will be served on, e.g. "example.com" for prod or
                        ; "localhost" for dev.
:biff/rules nil         ; An authorization rules data structure. To allow late binding, this can
                        ; optionally be a var or a 0-arg function.
:biff/triggers nil      ; A database triggers data structure. As with :biff/rules, this can
                        ; optionally be a var or a function.
:biff/send-email nil    ; A function which receives the system map merged with the following
                        ; keys: :to, :template, :data. Used for sending sign-in emails.
:biff/static-pages nil  ; A map from paths to Rum data structures, e.g.
                        ; {"/hello/" [:html [:body [:p {:style {:color "red"}} "hello"]]]}
:biff/fn-whitelist nil  ; Collection of fully-qualified function symbols to allow in
                        ; Crux queries sent from the frontend. Functions in clojure.core
                        ; need not be qualified. For example: '[map? example.core/frobulate]
:biff/routes nil        ; A vector of Reitit routes.
:biff/event-handler nil ; A Sente event handler function.
:biff/jobs nil          ; A vector of job data structures to schedule.
:biff/after-refresh     ; A fully-qualified symbol that specifies a function for biff.core/refresh
                        ; to call.

:biff.init/start-nrepl true
:biff.init/start-shadow false

:biff.auth/on-signup "/signin-sent" ; A redirect route.
:biff.auth/on-signin-request "/signin-sent"
:biff.auth/on-signin-fail "/signin-fail"
:biff.auth/on-signin "/app"
:biff.auth/on-signout "/"

:biff.crux/topology :standalone ; One of #{:jdbc :standalone}
; Ignored if :biff.crux/topology isn't :jdbc.
:biff.crux.jdbc/dbname nil
:biff.crux.jdbc/user nil
:biff.crux.jdbc/password nil
:biff.crux.jdbc/host nil
:biff.crux.jdbc/port nil

:biff.http/not-found-path "/404.html"
:biff.http/spa-path "/app/index.html" ; If set, takes precedence over :biff.http/not-found-path and
                                      ; sets http status to 200 instead of 404, unless the
                                      ; requested file path is prefixed by one of
                                      ; :biff.http/asset-paths.
:biff.http/asset-paths #{"/cljs/" "/js/" "/css/"} ; See :biff.http/spa-path.
:biff.http/secure-defaults true ; Whether to use ring.middleware.defaults/secure-site-defaults
                                ; or just site-defaults.

:biff.web/host "localhost" ; Host that the web server will listen on. localhost is used in
                           ; production because requests are reverse-proxied through nginx.
:biff.web/port 8080        ; Port that the web server will listen on.

:biff/dev false ; When true, changes the defaults for the following keys:
                :biff.init/start-shadow true
                :biff.init/start-nrepl false ; shadow-cljs has its own nrepl server.
                ; Also overrides values for these keys:
                :biff/host "localhost"
                :biff.crux/topology :standalone
                :biff.http/secure-defaults false
                :biff.web/host "0.0.0.0"

The following keys are added to the system map:

Biff merges the system map into incoming Ring requests and Sente events. It also adds :biff/db (a Crux DB value) on each new request/event.

Static resources

See Overview > Static resources.

Relevant config:

:biff/static-pages nil  ; A map from paths to Rum data structures, e.g.
                        ; {"/hello/" [:html [:body [:p "hello"]]]}

As mentioned, Biff serves your static resources from www/. In production, www/ is a symlink to /var/www/ and is served directly by Nginx (so static files will be served even if your JVM process goes down, e.g. during deployment).

Here's a larger example for :biff/static-pages:

(ns example.static
  (:require
    [rum.core :as rum]))

(def signin-form
  (rum/fragment
    [:p "Email address:"]
    [:form {:action "/api/signin-request" :method "post"}
     [:input {:name "email" :type "email" :placeholder "Email"}]
     [:button {:type "submit"} "Sign up/Sign in"]]))

(def home
  [:html
   [:head
    [:meta {:charset "utf-8"}]
    [:script {:src "/js/ensure-signed-out.js"}]]
   [:body
    signin-form]])

(def signin-sent
  [:html [:head [:meta {:charset "utf-8"}]]
   [:body
    [:p "Sign-in link sent, please check your inbox."]
    [:p "(Just kidding: click on the sign-in link that was printed to the console.)"]]])

(def signin-fail
  [:html [:head [:meta {:charset "utf-8"}]]
   [:body
    [:p "Invalid sign-in token."]
    signin-form]])

(def app
  [:html
   [:head
    [:meta {:charset "utf-8"}]
    [:script {:src "/js/ensure-signed-in.js"}]]
   [:body
    [:#app {:style {:font-weight "bold"}} "Loading..."]
    [:script {:src "/cljs/app/main.js"}]]])

(def not-found
  [:html [:head [:meta {:charset "utf-8"}]]
   [:body
    [:p "Not found."]]])

(def pages
  {"/" home
   "/signin/sent/" signin-sent
   "/signin/fail/" signin-fail
   "/app/" app
   "/404.html" not-found})

Authentication

Relevant config:

:biff/send-email nil ; A function which receives the system map merged with the following keys:
                     ; :to, :template, :data. Used for sending sign-in emails.
:biff.auth/on-signup "/signin-sent" ; A redirect route.
:biff.auth/on-signin-request "/signin-sent"
:biff.auth/on-signin-fail "/signin-fail"
:biff.auth/on-signin "/app"
:biff.auth/on-signout "/"

Biff currently provides email link authentication. The user clicks a link (which contains a JWT) in an email to sign in, and then Biff stores their user ID in an encrypted cookie. Password and SSO authentication are on the roadmap.

After a user is signed in, you can authenticate them on the backend from an event/request handler like so:

(require '[ring.middleware.anti-forgery :refer [wrap-anti-forgery]])

(defn handler [{:keys [session/uid biff/db]}]
  (if (some? uid)
    (do
      (prn (crux.api/entity db {:user/id uid}))
      ; => {:crux.db/id {:user/id #uuid "..."}
      ;     :user/id #uuid "..." ; duplicated for query convenience
      ;     :user/email "alice@example.com"}
      {:body "Hello, authenticated user."
       :headers/Content-Type "text/plain"})
    (do
      (println "User not authenticated.")
      ; Redirect the user to the login page
      {:status 302
       :headers/Location "/login"}
      ; If this is an API endpoint, you can just return a 403:
      ; {:status 403
      ;  :body "Forbidden."
      ;  :headers/Content-Type "text/plain"}
      )))

(def routes
  [["/foo" {:post handler
            :name ::foo
            ; You must include this for any endpoint which uses :session/uid.
            :middleware [wrap-anti-forgery]}]
   ...])

Biff provides a set of HTTP endpoints for authentication:

Sign up

Sends the user an email with a sign-in link. The token included in the link will expire after 24 hours. Redirects to the value of :biff.auth/on-signup.

HTTP Request

POST /api/signup

Form Parameters

Parameter Description
email The user's email address.

Example Usage

[:p "Email address:"]
[:form {:action "/api/signup" :method "post"}
 [:input {:name "email" :type "email" :placeholder "Email"}]
 [:button {:type "submit"} "Sign up"]]

The :biff.auth/send-email function will be called with the following options:

(send-email (merge
              ring-request
              {:to "alice@example.com"
               :template :biff.auth/signup
               :data {:biff.auth/link "https://example.com/api/signin?token=..."}}))

You will have to provide a send-email function which generates an email from the template data and sends it. (The example app just prints the template data to the console). If you use Mailgun, you can do something like this:

(def templates
  {:biff.auth/signup
   (fn [{:keys [biff.auth/link to]}]
     {:subject "Thanks for signing up"
      :html (rum.core/render-static-markup
              [:div
               [:p "We received a request to sign up with Findka using this email address."]
               [:p [:a {:href link} "Click here to sign up."]]
               [:p "If you did not request this link, you can ignore this email."]])})
   :biff.auth/signin ...})

(defn send-email* [api-key opts]
  (http/post (str "https://api.mailgun.net/v3/mail.example.com/messages")
    {:basic-auth ["api" api-key]
     :form-params (assoc opts :from "Example <contact@mail.example.com>")}))

(defn send-email [{:keys [to text template data mailgun/api-key] :as sys}]
  (if (some? template)
    (if-some [template-fn (get templates template)]
      (send-email* api-key
        (assoc (template-fn (assoc data :to to)) :to to))
      (println "Email template not found:" template))
    (send-email* api-key (select-keys sys [:to :subject :text :html]))))

Dealing with bots

The above example is susceptible to abuse from bots. An account isn't created until the user clicks the confirmation link, but it's better not to send emails to bots/spam victims in the first place. You'll need to use your own method for deciding if signups come from bots (I use recaptcha v3). The map passed to send-email includes the Ring request specifically so you can check the form parameters and make that decision.

If you render the login form with JS, you may not need to deal with this for a while. If you render it statically (like in the example app), you'll have to deal with it sooner.

Request sign-in

Sends the user an email with a sign-in link. This is the same as Sign up, except:

Sign in

Verifies the given JWT and adds a :uid key to the user's session (expires in 90 days). Also sets a :csrf cookie, the value of which must be included in the X-CSRF-Token header for any HTTP requests that require authentication.

If this is the first sign in, creates a user document in Crux with the email and a random user ID (a UUID). For example:

{:crux.db/id {:user/id #uuid "some-uuid"}
 :user/id #uuid "some-uuid" ; duplicated for query convenience.
 :user/email "abc@example.com"}

Redirects to the value of :biff.auth/on-signin (or :biff.auth/on-signin-fail if the token is incorrect or expired).

This endpoint is used by the link generated by Sign up and Request sign-in, so you typically won't need to generate a link for this endpoint yourself.

HTTP Request

GET /api/signin

URL Parameters

Parameter Description
token A JWT

Sign out

Clears the user's session and :csrf cookie. Redirects to the value of :biff.auth/on-signout.

HTTP Request

GET /api/signout

Example Usage

[:a {:href "/api/signout"} "sign out"]

Check if signed in

Returns status 200 if the user is authenticated, else 403.

HTTP Request

GET /api/signed-in

Example Usage

Include this on your landing page:

fetch("/api/signed-in").then(response => {
  if (response.status == 200) {
    document.location = "/app";
  }
});

Include this on your app page:

fetch("/api/signed-in").then(response => {
  if (response.status != 200) {
    document.location = "/";
  }
});

HTTP routes

Relevant config:

:biff/routes nil ; A vector of Reitit routes.
:biff.http/not-found-path "/404.html"
:biff.http/spa-path "/app/index.html" ; If set, takes precedence over :biff.http/not-found-path and
                                      ; sets http status to 200 instead of 404, unless the
                                      ; requested file path is prefixed by one of
                                      ; :biff.http/asset-paths.
:biff.http/asset-paths #{"/cljs/" "/js/" "/css/"} ; See :biff.http/spa-path.
:biff.http/secure-defaults true ; Whether to use ring.middleware.defaults/secure-site-defaults
                                ; or just site-defaults.
:biff/dev false ; When true, overrides values for these keys:
                :biff.http/secure-defaults false
                ...

The value of :biff/routes will be wrapped with some default middleware which, among other things:

(ns example.routes
  (:require
    [biff.util :as bu]
    ...))

(defn echo [req]
  {:headers/Content-Type "application/edn"
   :body (prn-str
           (merge
             (select-keys req [:params :body-params])
             (u/select-ns req 'params)))})

(def routes
  [["/echo" {:get echo
             :post echo
             :name ::echo}]
   ...])

$ curl -XPOST localhost:8080/echo?foo=1 -F bar=2
{:params {:foo "1", :bar "2"}, :params/bar "2", :params/foo "1"}
$ curl -XPOST localhost:8080/echo -d '{:foo 1}' -H "Content-Type: application/edn"
{:params {:foo "1"}, :params/foo "1", :body-params {:foo "1"}}

Endpoints that require authentication must be wrapped in anti-forgery middleware. See Authentication. When POSTing to such endpoints, you must include the value of the csrf cookie in the X-CSRF-Token header:

(cljs-http.client/post "/foo" {:headers {"X-CSRF-Token" (biff.client/csrf)}})

For SPA apps, you can usually communicate over web sockets instead.

Web sockets

Relevant config:

:biff/event-handler nil ; A Sente event handler function.

Example:

(defmulti api :id)

(defmethod api :default
  [{:keys [id]} _]
  (biff.util/anom :not-found (str "No method for " id)))

(defmethod api :example/do-something
  [{:keys [biff/db session/uid] :as sys} {:keys [foo bar]}]
  ...)

(defmethod api :example/echo
  [{:keys [client-id biff/send-event]} arg]
  (send-event client-id [:example/print ":example/echo called"])
  ; arg will be sent to the client. If you don't want to return anything,
  ; return nil explicitly.
  arg)

(def event-handler #(api % (:?data %)))

Biff provides a helper function for initializing the web socket connection on the frontend:

(defonce system (atom {}))

(defmulti api (comp first :?data))

(defmethod api :default
  [{[event-id] :?data} data]
  (println "unhandled event:" event-id))

(defmethod api :biff/error
  [_ anom]
  (pprint anom))

(defmethod api :example/print
  [_ arg]
  (prn arg))

(defn api-send [& args]
  (apply (:api-send @system) args))

(defn ^:export init []
  (reset! system
    (biff.client/init-sub {:handler api
                           ...}))
  ...)

(comment
  (go
    (<! (api-send [:example/echo {:foo "bar"}]))
    ; => {:foo "bar"}
    ; => ":example/echo called"
    ))

Transactions

You can send arbitrary transactions from the frontend. They will be submitted only if they pass certain authorization rules which you define (see Rules). Transactions look like this:

(defn set-display-name [display-name]
  (api-send
    [:biff/tx
     {[:users {:user/id @db/uid}]
      {:db/update true
       :display-name display-name}}]))

The transaction is a map from idents to documents. The first element of an ident is a table, such as :games. Tables are defined by your rules, and they specify which rules a document write must pass in order to be allowed.

The second element, if present, is a document ID. If omitted, it means we're creating a new document and we want the server to set the ID to a random UUID:

{[:messages] {:text "hello"}}

If you want to create multiple documents in the same table with random IDs, use nested vectors instead of a map.

[[[:messages] {:text "a"}]
 [[:messages] {:text "b"}]]

:db/current-time is replaced by the server with the current time.

{[:events] {:timestamp :db/current-time
            ...}}

If :db/update is true, the given document will be merged with an existing document, failing if the document doesn't exist. There's also :db/merge which simply creates the document if it doesn't exist (i.e. upsert).

{[:chatrooms {:chatroom/id #uuid "some-uuid"}]
 {:db/update true
  :title "Existing chatroom"}

 [:chatrooms {:chatroom/id #uuid "another-uuid"}]
 {:db/merge true
  :title "New or existing chatroom"}}

You can dissoc document keys by setting them to :db/remove. You can delete whole documents by setting them to nil.

{[:users {:user/id #uuid "my-id"}]
 {:db/update true
  :display-name :db/remove}

 [:orders {:order/id #uuid "some-order-id"}]
 nil}

You can add or remove an element to/from a set by using :db/union and :db/disj, respectively:

{[:games {:game/id #uuid "old-game-uuid"}]
 {:db/update true
  :users [:db/disj "my-uid"]}

 [:games {:game/id #uuid "new-game-uuid"}]
 {:db/update true
  :users [:db/union "my-uid"]}}

Using maps as document IDs lets you specify composite IDs. In addition, all keys in the document ID will be duplicated in the document itself. This allows you to use document ID keys in your queries.

{[:user-item {:user #uuid "some-user-id"
              :item #uuid "some-item-id"}]
 {:rating :like}}

; Expands to:
[:crux.tx/put
 {:crux.db/id {:user #uuid "some-user-id"
               :item #uuid "some-item-id"}
  :user #uuid "some-user-id"
  :item #uuid "some-item-id"
  :rating :like}]

Transactions on the back end

On the back end, you can use biff.crux/submit-tx:

(biff.crux/submit-tx sys
  {[:users {:user/id #uuid "some-uuid"}]
   {:db/update true
    :display-name "alice"}})

This will bypass the write authorization functions defined in :biff/rules, but it will throw an exception if any documents don't conform to the specs for their respective tables. For example, if the value for :user/id above is correct, the transaction above would succeed given these rules:

(require '[biff.util :as bu])

(bu/sdefs
  :user/id uuid?
  :user/email string?
  ::display-name string?
  :ref/user (bu/only-keys :req [:user/id])
  ::user (bu/only-keys
           :req [:user/email]
           :opt-un [::display-name]))

(def rules
  {:users {:specs [:ref/user ::user]
           :write (constantly false)}})

But if the :user/id value was incorrect (and thus refers to a non-existent user), the transaction would fail. It would also fail if you set :display-name 123 or :display-name nil instead of :display-name "alice" in the transaction.

You can also use bypass Biff's transactions and use Crux's API directly:

(let [{:keys [biff/node biff/db]} sys
      doc-id {:user/id #uuid "some-uuid"}
      user (crux.api/entity db doc-id)]
  (crux.api/submit-tx node
    [[:crux.tx/put (merge user
                     {:crux.db/id doc-id
                      :user/id #uuid "some-uuid"
                      :display-name "alice"})]]))

But if you do this, Biff won't be able to check the transaction against your specs.

Queries

Biff allows you to subscribe to Crux queries from the frontend with one major caveat: cross-entity joins are not allowed. Basically, this means all the where clauses in the query have to be for the same entity.

; OK
'{:find [doc]
  :where [[doc :foo 1]
          [doc :bar "hey"]]}

; Not OK
'{:find [doc]
  :where [[user :name "Tilly"]
          [doc :user user]]}

So to be clear, Biff's subscribable "queries" are not datalog at all. They're just predicates that can take advantage of Crux's indices. Biff makes this restriction so that it can provide query updates to clients efficiently without having to solve a hard research problem first. However, it turns out that we can go quite far even with this restriction.

On the frontend, use biff.client/init-sub to initialize a websocket connection that handles query subscriptions for you:

(def default-subscriptions
  #{[:biff/sub {:table :users
                :args {'name "Ben"}
                :where '[[:name name]
                         [:age age]
                         [(<= 18 age)]
                         [(yourapp.core/likes-cheese? doc)]]}]})

(def subscriptions (atom default-subscriptions))
(def sub-results (atom {}))

(biff.client/init-sub
  {:subscriptions subscriptions
   :sub-results sub-results
   ...})

If you want to subscribe to a query, swap! it into subscriptions. If you want to unsubscribe, swap! it out. Biff will populate sub-results with the results of your queries and remove old data when you unsubscribe. You can then use the contents of that atom to drive your UI. The contents of sub-results is a map of the form subscription->table->id->doc, for example:

{[:biff/sub '{:table :users
              :where ...}]
 {:users
  {{:user/id #uuid "some-uuid"} {:name "Sven"
                                 :age 250
                                 ...}}}}

Note the subscription format again:

[:biff/sub {:table :users
            :args {'name "Ben"}
            :where '[[:name name]
                     [:age age]
                     [(<= 18 age)]
                     [(yourapp.core/likes-cheese? doc)]]}]

The first element is a Sente event ID. The query map (the second element) omits the entity variable in the where clauses since it has to be the same for each clause anyway. But it will be bound to doc in case you want to use it in e.g. a predicate function. :find is similarly omitted.

The :table value is connected to authorization rules which you define on the backend (see Rules). When a client subscribes to this query, it will be rejected unless you define rules for that table which allow the query. You also have to whitelist any predicate function calls (like yourapp.core/likes-cheese?), though the comparison operators (like <=) are whitelisted for you.

I haven't yet added support for or, not, etc. clauses in subscriptions. See #9.

You can also subscribe to individual documents:

[:biff/sub '{:table :users
             :id {:user/id #uuid "some-uuid"}}]

All this is most powerful when you make the subscriptions atom a derivation of sub-results:

(ns example.client.app.db
  (:require
    [biff.rum :as br]))

(br/defatoms
  sub-results {}
  message-cutoff (js/Date.)
  route {})

; defderivations lets you use rum.core/derived-atom without the boilerplate.
(br/defderivations
  ; data is an atom that contains a map of table->id->doc. It will be updated
  ; whenever sub-results changes.
  data (apply merge-with merge (vals @sub-results))

  uid (get-in @data [:uid nil :uid])
  user (get-in @data [:users {:user/id @uid}])
  email (:user/email @user)
  foo (:foo @user)
  bar (:bar @user)
  messages (->> @data
             :messages
             vals
             (sort-by :timestamp #(compare %2 %1)))

  tab (get-in @route [:data :name] :crud)

  subscriptions (disj #{[:biff/sub :uid]
                        [:biff/sub {:table :messages
                                    :args {'t0 @message-cutoff}
                                    :where '[[:timestamp t]
                                             [(< t0 t)]]}]
                        (when @uid
                          [:biff/sub {:table :users
                                      :id {:user/id @uid}}])}
                  nil))

When a user signs into this app, they will subscribe to their user ID ([:biff/sub :uid], a special subscription) and any messages that are sent after the page loaded. When the user's ID is received from the back end and loaded into sub-results, it will cause subscriptions to update. The client will then subscribe to the document for the current user. subscriptions will also be updated if message-cutoff changes.

This is what I meant when I said that we can go pretty far without cross-entity joins: using this method, we can declaratively load all the relevant data and perform joins on the client. This should be sufficient for many situations.

However, it won't work if you need an aggregation of a set of documents that's too large to send to the client (not to mention each client), or if the client isn't allowed to see the individual documents. There will also be increased latency since you have to wait for a network hop between joins.

To remedy that, I was previously working on a Materialize integration, though it's no longer a priority for me at the moment.

Queries on the back end

You can use Crux's API:

(let [{:keys [biff/db]} sys]
  (crux.api/q db
    {:find '[user]
     :full-results? true
     :args [{'name "Ben"}]
     :where '[[user :name name]
              ...]}))

Rules

Relevant config:

:biff/rules nil         ; An authorization rules data structure.
:biff/fn-whitelist nil  ; Collection of fully-qualified function symbols to allow in
                        ; Crux queries sent from the frontend. Functions in clojure.core
                        ; need not be qualified. For example: '[map? example.core/frobulate]

Your app's rules define what transactions and subscriptions will be accepted from the frontend (see Transactions and Subscriptions).

The value of :biff/rules is a map of table->rules, for example:

(ns example.rules
  (:require
    [biff.util :as bu]
    [clojure.spec.alpha :as s]))

; Same as (do (s/def ...) ...)
(bu/sdefs
  :user/id uuid?
  ; like s/keys, but only allows specified keys.
  ::user-ref (bu/only-keys :req [:user/id])
  ::user (bu/only-keys :req [:user/email])
  ...)

(def rules
  {:users {:spec [::user-ref ::user]
           :get (fn [{:keys [session/uid] {:keys [user/id]} :doc}]
                  (= uid id))}
   ...})

Tables

The table is used in transactions and subscriptions to specify which rules should be used. The rules above authorize us to subscribe to this:

[:biff/sub {:table :users
            :id {:user/id #uuid "some-uuid"}}]

And for transactions:

{:games {:spec [::game-ref ::game]
         :create (fn [env] ...)}}
; Authorizes:
[:biff/tx {[:games {:game/id "ABCD"}]
           {:users #{#uuid "some-uuid"}}}]

Specs

For each document in the query result or transaction, authorization has two steps. First, the document ID and the document are checked with s/valid? against the two elements in :specs, respectively. For example, the specs for the :users table above would authorize a read or write operation on the following document:

{:crux.db/id {:user/id #uuid "some-uuid"}
 :user/id #uuid "some-uuid"
 :user/email "email@example.com"}

Note that during this check, the document will not include the ID or any keys in the ID (for map IDs). (Also recall that map ID keys are automatically duplicated in the document when using Biff transactions).

For write operations, the document must pass the spec before and/or after the transaction, depending on whether the document is being created, updated or deleted.

Operations

If the specs pass, then the document must also pass a predicate specified by the operation. There are five operations: :create, :update, :delete, :query, :get.

{:messages {:specs ...
            :create (fn [env] ...)
            :get (fn [env] ...)}}

You can use the same predicate for multiple operations like so:

{:messages {:specs ...
            [:create :update] (fn [env] ...)}}

There are several aliases:

Alias Expands to
:read [:query :get]
:write [:create :update :delete]
:rw [:query :get :create :update :delete]

For example:

{:messages {:specs ...
            :write (fn [env] ...)}}

:get refers to subscriptions for individual documents while :query is for multiple documents:

; get
[:biff/sub {:table :users
            :id {:user/id #uuid "some-uuid"}}]
; query
[:biff/sub {:table :games
            :where [[:users #uuid "some-uuid"]]}]

Predicates

Predicates receive the system map merged with some additional keys, depending on the operation:

Key Operations Description
:session/uid :rw The ID of the user who submitted the query/transaction. nil if they're unauthenticated.
:biff/db :rw The Crux DB value before this operation occurred.
:doc :rw The document being operated on.
:doc-before :write The previous value of the document being operated on.
:current-time :write The inst used to replace any occurrences of :db/current-time (see Transactions).
:generated-id :create true iff a random UUID was generated for this document's ID.

Some examples:

(def rules
  {:public-users {:spec [::user-public-ref ::user-public]
                  ; Returns false iff :session/uid is nil.
                  :get biff.rules/authenticated?
                  :write (fn [{:keys [session/uid] {:keys [user.public/id]} :doc}]
                           (= uid id))}
   :users {:spec [::user-ref ::user]
           :get (fn [{:keys [session/uid] {:keys [user/id]} :doc}]
                  (= uid id))}
   :games {:spec [::game-ref ::game]
           :query (fn [{:keys [session/uid] {:keys [users]} :doc}]
                    (contains? users uid))
           [:create :update] (fn [{:keys [session/uid doc doc-before]
                                   {:keys [users]} :doc}]
                               (and
                                 (some #(contains? (:users %) uid) [doc doc-before])
                                 ; Checks that no keys other than :users have changed
                                 ; (supports varargs).
                                 (biff.rules/only-changed-keys? doc doc-before :users)
                                 ; Checks that the value of :users (a set) hasn't changed except
                                 ; for the addition/removal of uid (supports varargs).
                                 (biff.rules/only-changed-elements? doc doc-before :users uid)))}})

(def rules
  {:events {:spec [uuid? ::event]
            :create (fn [{:keys [session/uid current-time generated-id]
                          {:keys [timestamp user]} :doc}]
                      (and
                        (= uid (:user/id user))
                        ; Make sure that :timestamp was set by the server, not the client.
                        (= current-time timestamp)
                        ; Make sure that the ID was set by the server, not the client.
                        generated-id))}}

Triggers

Relevant config:

:biff/triggers nil ; A database triggers data structure.

Triggers let you run code in response to document writes. You must define a map of table->operation->fn, for example:

(defn assign-players [{:keys [biff/node doc]
                       {:keys [users x o]} :doc :as env}]
  ; When a user joins or leaves a game, re-assign users to X and O as needed.
  ; Delete the game document if everyone has left.
  (let [new-doc ... ; Same as doc but maybe with different :x and :o values
        op (cond
             (empty? users) [:crux.tx/delete (:crux.db/id doc)]
             (not= doc new-doc) [:crux.tx/put new-doc])]
    (when op
      (crux/submit-tx node
        [[:crux.tx/match (some :crux.db/id [doc new-doc]) doc]
         op]))))

(def triggers
  {:games {[:create :update] assign-players}})

See Tables and Operations. The function will receive the system map merged with the following keys:

Key Description
:doc The document that was written.
:doc-before The document's value before being written.
:db The Crux DB value after this operation occurred.
:db-before The Crux DB value before this operation occurred.
:op One of #{:create :update :delete}.

Jobs

Relevant config:

:biff/jobs []

Each element of :biff/jobs is a map with three keys. For example:

(defn some-job [sys]
  (println "This function will run every 2 minutes,")
  (println "beginning 1 minute after your app starts."))

(def jobs
  [{:offset-minutes 1
    :period-minutes 2
    :job-fn #'some-job}])

Deployment

See Overview > Infrastructure.

1. Set up DigitalOcean

Biff comes with Terraform config for DigitalOcean. You can write your own config if you want to use a different provider (see Overview > Decomposing), but for now I'll assume you're using DigitalOcean. If you don't already have an account, you can sign up with my referral link which will give you $100 of credit for 60 days (and $25 for me if you stick with them).

You'll also need a domain that points to DigitalOcean's nameservers.

2. Update config

In config/main.edn, make sure :biff/host is set to the domain you want to use for your production app (e.g. myapp.example.com). If you've changed this since creating your Biff project, run ./task dev (or (biff.core/refresh)) to make sure the Terraform config file (infra/system.tf.json) is up-to-date.

In config/task.env, update the following environment variables:

Put your personal SSH public key in config/ssh-public-key. For example: cp ~/.ssh/id_rsa.pub config/ssh-public-key. This will let Terraform (and you) run commands on the server after it's provisioned.

Run ./task generate-deploy-key. This will write a new SSH private key to config/deploy-key, which will let the server download your code from git (assuming you're using a private repo. If not, you can ignore this step). The public key will be in config/deploy-key.pub. You'll need to give that key read access to your git repo. If you're using Github, you can do this at https://github.com/your_username/your_repo/settings/keys -> Add deploy key.

3. Create an image

Run ./task build-image. It'll take 3-5 minutes. Some of the output will be red; this is (probably) OK. If successful, the command will write the new image ID to config/task.env, for example:

$ grep IMAGE_ID config/task.env
export IMAGE_ID=12345 # Managed by Biff.

4. Update repo

Build your CSS and ClojureScript for production with ./task build-assets. They'll be written to resources/www/css/main.css and resources/www/cljs/app/main.js. Commit those files (and all other changes) to your repo and push. Whenever your app starts on the server, it will fetch the latest commit from your repo and run that.

5. Deploy with Terraform

If you've already added your domain to DigitalOcean (i.e. it shows up under Networking > Domains), you'll need to import it into Terraform. For example, if your app's domain is foo.example.com, then you'll need to run ./task tf import digitalocean_domain.default example.com.

You might also need to do the following before proceeding:

Run ./task tf apply. Terraform will show you the changes to be made, and it'll ask for confirmation before it does anything. After the command finishes, watch the logs with ./task logs. You should eventually see System started. Once you do, your app is live!

6. Future deploys

For future deploys, simply push the changes to your repo and then run ./task deploy. This will restart your app's process on the server, which will cause it to re-fetch the latest commit.

If you make any infrastructure changes, you can re-run ./task tf apply. If you made image changes, re-run ./task build-image first.

7. Cleanup

You can remove the resources provisioned by Terraform with ./task tf destroy. However, that will also remove the domain from DigitalOcean which you may not want. Instead, you can delete resources manually from the DigitalOcean web console. While you're there, you can delete the image(s) you created (these won't be deleted by ./task tf destroy).

FAQ

Comparison to Firebase

Basically, if you like Firebase and you like Clojure backend dev, you might enjoy using Biff for your next side project. Same if you like the idea of Firebase but in practice you have issues with it. If you want something mature or you like having a Node/ClojureScript backend, Firebase is a great choice. Here's a non-trivial example of using Firebase with ClojureScript.

Some shared features:

Some differences:

Comparison to Fulcro

Similarities:

Differences:

Why Crux and not Datomic?

Short answer: Like Vim, Arch Linux, and Clojure itself, Crux is one of those pieces of software that sparks joy.

I used Datomic pretty heavily in my own projects for about a year prior to switching to Firestore and then Crux. My opinion on Datomic vs. Crux is that Datomic is more powerful and maybe can scale better, but Crux is easier to get started with and has a lot less operational overhead for small projects (in terms of developer time). I've had many headaches from my time using Datomic (and AWS, which Datomic Cloud is coupled to). On the other hand, using Crux has been smooth—and you can use DigitalOcean instead of AWS (yay). Since Biff prioritizes the solo-developer / early-stage / rapid-prototyping use-case, I think Crux is a much better fit. Whereas if I was in a situation with many developers/delivering an application that I knew would have scale once released, Datomic Cloud Ions would be worth considering (but even then, I personally would probably stick with Crux—I just love Crux).

Off the top of my head, a few more reasons:

Clojure