what if my git host were a static site generator?

i have been running several personal git forges for, at this point, almost half my life :o i like running my own dev infrastructure, not only because i’m almost always 120ms+ away from us-east-1, but because sysadmin is just plain fun :3 in 2015 i had a Gogs instance which became a Gitea instance which became a Forgejo instance, and i’ve also deployed GitLab/Forgejo several other times for various groups i’ve been a member of. i like the communal collaborative git forge, and Forgejo is great at this!

but my Forgejo server keeps running out of disk space (from crashing while repacking git repos that haven’t updated) and falling over / OOMing under ambient scraper load. for my needs it’s clear that this is just the wrong size of thing: on the tiny machines i use for personal infrastructure, the software can’t stand up to the internet’s cosmic microwave background radiation.

i also kinda wanna simplify my experience by only exposing features i’ll actually use: Forgejo and its ilk do way more than i need them to: issues, PRs, releases, wikis - a bunch of GitHub feature-compatibility that i don’t care about, and pay some sort of cost for anyway :(

publishing to the open web

the usual antidote prescribed for Forgejo resource exhaustion is to block scrapers via a web application firewall like Anubis, which aims to gate access to the webapp behind a JavaScript proof of work challenge.

but this is counter to, like, the philosophy of the open web, right? the browser, ostensibly the “user agent”, is coerced into user-unfriendly behavior, executing near-useless code that taxes the user’s device (the point of the challenge is to spin!) - were it to refuse, no user-relevant information could be displayed at all. alternative browsers that don’t support JavaScript (or just don’t support JITted JavaScript) are either completely blocked off or locked behind a truly intrusive wait time. this deepens the oligoculture of the modern web, which i think is a bad thing.

additionally, deployment of such a thing is an admission of defeat - we surrender to the assumption that the fronted application does not work correctly when met with real-world internet traffic: isn’t this kind of ridiculous when we have a workload where reads so heavily outnumber writes? serving write-sparse data ought to be super cheap in practice: all of github pages ran on one machine for years!! why not have a git host where everything is static files?

git repo views with minimal server compute

at its core, sorcery is shaped like a static site generator: when it receives an update to a git repo, it will rebuild a bunch of on-disk HTML for that repo - an overview page, the directory tree the tip commit of each branch, and syntax-highlighted source code renderings for each file in the tips. this allows us to pay a fixed upfront cost for serving many future requests, which means we are resilient against scraper load (because a sendfile-and-forget has basically negligible cost). however, since it would be expensive to render out HTML ahead of time for every revision of every file, we choose not to serve static historical views of the repo.

repo history viewing is an integral feature of a git web interface, though, so we serve the .git directory directly, implement a basic read-only git client in JavaScript, and then client-side render all the “rich views” of the repository - the repo site generator does also need to emit some supplementary JSON data to aid the git client, since we can’t reliably list directories in the git repo, but that’s still static!

since browsing around history can mean many fetches to different objects (commits, trees [i.e. repo directory listings], blobs [i.e. file contents]), a high-latency connection can cause direct object fetching and traversal to feel really slow. even moreso when git stores these objects compressed in delta-encoded packfiles: a naïve fetch of a packfile index in linux.git to view the diff of one commit would use over 400MiB of bandwidth!! and smartly scanning ranges would kill cache hit rates and also waterfall out to a bunch of requests that depend on the data in prior requests.

so, as a non-static optimization¹, we also provide a serverside route to fetch specific git objects by a list of object IDs (QUERY /<user>/<repo>/obj), returning a simple binary “git object bundle” format which gets trivially parsed on the client. this alleviates the burden of navigating packfiles on the client. additionally, even for loose object repos, we can still optimize roundtrips by providing “smart fetch” modes which traverse for referenced oids for a given access pattern (e.g. traversing commit history via commit.parent->parent->parent->…, or accessing all blobs in the trees of a pair of commits in order to diff them).

non-features

so sorcery is a “git repo viewer” and not a “git forge” because it omits user accounts, ssh/gpg key management, and issues+patches entirely. in fact, sorcery proper is entirely read-only! repos are only ever written via git over ssh, which is separate to sorcery. you can run sorcery-ssh as your git user’s ssh ForceCommand, and it will provide “autocreate repo on first push” + the ability to edit repo descriptions. the benefit of this setup is that your public deployment is as secure as its sshd, which is reassuring in big 26 :)

historical views do require JavaScript, but i’m not super into blanket js allergy on the web (because unless you live in Ashburn, running UI code on-device is basically always better!) - sorcery uses my own frontend microframework + a bunch of built-in web platform affordances + intentional codesplitting to create a rich clientside experience in a super lightweight manner! e.g. loading a project overview page transfers about 9kb of gzipped JS to support recent commit pagination / language filtering / links through to commit diffs - the largest part of the site ends up being the syntax highlighting grammars; i think i want to try writing a pure-javascript executor for tree-sitter grammars and queries so that we can shed some of the WASM weight (especially from the code that gets repackaged in each highlight language’s WASM bundle).

so, yeah

i still like the communal git forge!! for my personal projects, i mostly just want somewhere to push code that i can browse from my phone / link to people: i don’t need collaborative features, and it’s much more lightweight this way. reading my code should never involve a ceremony of proof to the server that you’re worthy of receiving hypertext.

in the near-term i want to add support for CI annotations (which i am opinionated about, and will talk about in a future blog post!! smash that RSS button) & maybe in the future i’ll extend the repo viewer into a discrete, more batteries-included forge project that i can use with my friends (once we really distill down to what’s most important to us and what is superfluous…)

anyway, check it out!