I'm seeing an increasing trend of pushback against this norm. An early example was David Crawshaw's one-process programming notes [1]. Running the database in the same process as the application server, using SQLite, is getting more popular with the rise of Litestream [2]. Earlier this year, I found the post "One machine can go pretty far if you build things properly" [3] quite refreshing.
Most of us can ignore FAANG-scale problems and keep right on using POSIX on a handful of machines.
One trend in OS research (what little exists) is the idea of the database OS. Taking that as an inspiration I think there's a better way to structure things to get that same simplicity and in fact even more, but without many of the downsides. I'm planning to write about it more at some point on my company blog (https://hydraulic.software/blog.html) but here's a quick summary. See what you think.
---
In a traditional 3-tier CRUD web app you have the RDBMS, then stateless web servers, then JavaScript and HTML in the browser running a pseudo-stateless app. Because browsers don't understand load balancing you probably also have an LB in there so you can scale and upgrade the web server layer without user-visible downtime. The JS/HTML speaks an app specific ad-hoc RPC protocol that represents RPCs as document fetches, and your web server (mostly) translates back and forth between this protocol and whatever protocol your RDBMS speaks layering access control on top (because the RDBMS doesn't know who is logged in).
This approach is standard and lets people use web browsers which have some advantages, but creates numerous problems. It's complex, expensive, limiting for the end user, every app requires large amounts of boilerplate glue code, and it's extremely error prone. XSS, XSRF and SQL injection are all bugs that are created by this choice of architecture.
These problems can be fixed by using "two tier architecture". In two tier architecture you have your RDBMS cluster directly exposed to end users, and users log in directly to their RDBMS account using an app. The app ships the full database driver and uses it to obtain RPC services. Ordinary CRUD/ACL logic can be done with common SQL features like views, stored procedures and row level security [1][2][3]. Any server-side code that isn't neatly expressible with SQL is implemented as RDBMS server plugins.
At a stroke this architecture solves the following problems:
1. SQL injection bugs disappear by design because the RDBMS enforces security, not a highly privileged web app. By implication you can happily give power users like business analysts direct SQL query access to do obscure/one-off things that might otherwise turn into abandoned backlog items.
2. XSS, XSRF and all the other escaping bugs go away, because you're not writing a web app anymore - data is pulled straight from the database's binary protocol into your UI toolkit's data structures. Buffer lengths are signalled OOB across the entire stack.
3. You don't need a hardware/DNS load balancer anymore because good DB drivers can do client-side load balancing.
4. You don't need to design ad-hoc JSON/REST protocols that e.g. frequently suck at pagination, because you can just invoke server-side procedures directly. The DB takes care of serialization, result streaming, type safety, access control, error reporting and more.
5. The protocol gives you batching for free, so if you have some server logic written in e.g. JavaScript, Python, Kotlin, Java etc then it can easily use query results as input or output and you can control latency costs. With some databases like PostgreSQL you get server push/notifications.
6. You can use whatever libraries and programming languages you want.
This architecture lacks popularity today because to make it viable you need a few things that weren't available until very recently (and a few useful things still aren't yet). At minimum:
1. You need a way to distribute and update GUI desktop apps that isn't incredibly painful, ideally one that works well with JVM apps because JDBC drivers tend to have lots of features. Enter my new company, stage left (yes! that's right! this whole comment is a giant ad for our product). Hydraulic Conveyor was launched in July and makes distributing and updating desktop apps as easy as with a web app [4].
2. You're more dependent on having a good RDBMS. PostgreSQL only got RLS recently and needs extra software to scale client connections well. MS SQL Server is better but some devs would feel "weird" buying a database (it's not that expensive though). Hosted DBs usually don't let you install arbitrary extensions.
3. You need solid UI toolkits with modern themes. JetBrains has ported the new Android UI toolkit to the desktop [5] allowing lots of code sharing. It's reactive and thus has a Kotlin language dependency. JavaFX is a more traditional OOP toolkit with CSS support, good business widgets and is accessible from more languages for those who prefer that; it also now has a modern GitHub-inspired SASS based style pack that looks great [6] (grab the sampler app here [7]). For Lispers there's a reactive layer over the top [8].
4. There's some smaller tools that would be useful e.g. for letting you log into your DB with OAuth, for ensuring DB traffic can get through proxies.
Downsides?
1. Migrating between DB vendors is maybe harder. Though, the moment you have >1 web server you have the problem of doing a 'live' migration anyway, so the issues aren't fundamentally different, it'd just take longer.
2. Users have install your app. That's not hard and in a managed IT environment the apps can be pushed out centrally. Developers often get hung up on this point but the success of the installed app model on mobile, popularity of Electron and the whole video game industry shows users don't actually care much, as long as they plan to use the app regularly.
3. To do mobile/tablet you'd want to ship the DB driver as part of your app. There might be oddities involved, though in theory JDBC drivers could run on Android and be compiled to native for iOS using GraalVM.
4. Skills, hiring, etc. You'd want more senior devs to trailblaze this first before asking juniors to learn it.
[1] https://www.postgresql.org/docs/current/ddl-rowsecurity.html
[2] https://docs.microsoft.com/en-us/sql/relational-databases/se...
[3] https://docs.oracle.com/database/121/TDPSG/GUID-72D524FF-5A8...
[4] https://hydraulic.software/
[5] https://www.jetbrains.com/lp/compose-mpp/
[6] https://github.com/mkpaz/atlantafx
[7] https://downloads.hydraulic.dev/atlantafx/sampler/download.h...
Web applications can come close to directly accessing the database by using GraphQL with something like Hasura or PostGraphile. PostGraphile even uses Postgres's row-level security. A colleague and I once did a project using Hasura with a SPA-style JavaScript front-end and a separate back-end service driven by Hasura webhooks for doing the actual computation, and we ended up being unhappy with that approach. Some of our problems were related to the SPA architecture, but some were related to our use of GraphQL and Hasura.
We ended up starting over with a typical server-rendered web application, where the server itself accesses the database and communicates with the computation-heavy back-end service over gRPC, using a minimum of client-side JavaScript. I remain happy with that architecture, though I continue to explore different ways of integrating modest amounts of client-side JavaScript for interactivity and real-time updates. And to bring it back to the topic of my previous comment, if we assume that there has to be a server rendering HTML anyway, then I think it often makes sense to reduce complexity by bringing the database into that server process. I haven't yet tried that in production, though.
I think my preference is to use HTTP primarily as it was originally intended, for fetching HTML, as well as for form submissions. For finer-grained interactivity, I think it's better to use WebSockets as opposed to REST-ish requests and responses. I'm not dogmatic on that, though.
On web apps versus packaged desktop apps, I'm still inclined to go with web whenever feasible, and only develop a packaged desktop app if a web app just won't work. Being able to use an application without an installation step is really powerful for pre-sales demos or trials, for onboarding new users, and for applications that may only be used occasionally by any given user. Even for an application that you use all the time, a web app can be fine, as the popularity of Google Docs demonstrates. For example, if you just want to get the browser chrome out of the way, desktop browsers support "installing" a web app as an OS-level app with no browser chrome. IMO, Hydraulic's Eton demo app could just as well be a web app.
I look forward to your blog post, though even your preliminary HN comment offers a lot to think about.
Yes, for transient apps that are only used occasionally or where the user isn't committed e.g. social networks, the combination of sandboxing + no integration/unintegration steps, automatic deletion from the cache etc is really useful. Of course there's no rule that says only web browsers can supply these features. Other kinds of browser-like thing can do so too.
It's also worth thinking a bit outside the box. Although we claim web apps don't have installation steps, that's not really true most of the time. The lack of any explicit integration step ("install") means the browser doesn't know if the user values anything they do on the site. So you have to provide data persistence, and that in turn means you need a signup / account creation flow. It also means you're on the hook to store, replicate and back up any data any user ever creates, even if they only used it once and then never return.
Well, a lot of users really hate creating yet another account, especially if they aren't committed yet. It's tedious, they think you'll spam them with marketing emails and they're probably right, plus they don't want to make another password. Or they make one, abandon for a year, then can't remember how to log in.
You might think that's so fundamental it can't be any other way, but it's really just a side effect of how browsers have evolved. Think about how you might do demos outside the browser. You could just have a trial mode where the app spins up a local in-process RDBMS like H2 that writes the data into the app's private data area (on Windows) or home directory on macOS/Linux. No accounts necessary - just one or two clicks to download+trigger app install, and you're done. If/when the user decides to graduate, then they create an account and the app uploads the data from the local DB to the real remote DB. If they abandon it and don't care, it's not your problem, it costs you nothing. If they run low on disk space the OS will suggest they delete old unused apps at that point and you'll get tossed along with the rest.
Mostly though, this is about making developers more productive. If the primary determinant of your success is feature throughput and not shaving a few seconds off your onboarding, e.g. you're making specialised apps, internal apps, enterprise software, then optimizing for better dev environments can make sense. Installation just isn't that bad.