I'm seeing an increasing trend of pushback against this norm. An early example was David Crawshaw's one-process programming notes [1]. Running the database in the same process as the application server, using SQLite, is getting more popular with the rise of Litestream [2]. Earlier this year, I found the post "One machine can go pretty far if you build things properly" [3] quite refreshing.
Most of us can ignore FAANG-scale problems and keep right on using POSIX on a handful of machines.
One trend in OS research (what little exists) is the idea of the database OS. Taking that as an inspiration I think there's a better way to structure things to get that same simplicity and in fact even more, but without many of the downsides. I'm planning to write about it more at some point on my company blog (https://hydraulic.software/blog.html) but here's a quick summary. See what you think.
---
In a traditional 3-tier CRUD web app you have the RDBMS, then stateless web servers, then JavaScript and HTML in the browser running a pseudo-stateless app. Because browsers don't understand load balancing you probably also have an LB in there so you can scale and upgrade the web server layer without user-visible downtime. The JS/HTML speaks an app specific ad-hoc RPC protocol that represents RPCs as document fetches, and your web server (mostly) translates back and forth between this protocol and whatever protocol your RDBMS speaks layering access control on top (because the RDBMS doesn't know who is logged in).
This approach is standard and lets people use web browsers which have some advantages, but creates numerous problems. It's complex, expensive, limiting for the end user, every app requires large amounts of boilerplate glue code, and it's extremely error prone. XSS, XSRF and SQL injection are all bugs that are created by this choice of architecture.
These problems can be fixed by using "two tier architecture". In two tier architecture you have your RDBMS cluster directly exposed to end users, and users log in directly to their RDBMS account using an app. The app ships the full database driver and uses it to obtain RPC services. Ordinary CRUD/ACL logic can be done with common SQL features like views, stored procedures and row level security [1][2][3]. Any server-side code that isn't neatly expressible with SQL is implemented as RDBMS server plugins.
At a stroke this architecture solves the following problems:
1. SQL injection bugs disappear by design because the RDBMS enforces security, not a highly privileged web app. By implication you can happily give power users like business analysts direct SQL query access to do obscure/one-off things that might otherwise turn into abandoned backlog items.
2. XSS, XSRF and all the other escaping bugs go away, because you're not writing a web app anymore - data is pulled straight from the database's binary protocol into your UI toolkit's data structures. Buffer lengths are signalled OOB across the entire stack.
3. You don't need a hardware/DNS load balancer anymore because good DB drivers can do client-side load balancing.
4. You don't need to design ad-hoc JSON/REST protocols that e.g. frequently suck at pagination, because you can just invoke server-side procedures directly. The DB takes care of serialization, result streaming, type safety, access control, error reporting and more.
5. The protocol gives you batching for free, so if you have some server logic written in e.g. JavaScript, Python, Kotlin, Java etc then it can easily use query results as input or output and you can control latency costs. With some databases like PostgreSQL you get server push/notifications.
6. You can use whatever libraries and programming languages you want.
This architecture lacks popularity today because to make it viable you need a few things that weren't available until very recently (and a few useful things still aren't yet). At minimum:
1. You need a way to distribute and update GUI desktop apps that isn't incredibly painful, ideally one that works well with JVM apps because JDBC drivers tend to have lots of features. Enter my new company, stage left (yes! that's right! this whole comment is a giant ad for our product). Hydraulic Conveyor was launched in July and makes distributing and updating desktop apps as easy as with a web app [4].
2. You're more dependent on having a good RDBMS. PostgreSQL only got RLS recently and needs extra software to scale client connections well. MS SQL Server is better but some devs would feel "weird" buying a database (it's not that expensive though). Hosted DBs usually don't let you install arbitrary extensions.
3. You need solid UI toolkits with modern themes. JetBrains has ported the new Android UI toolkit to the desktop [5] allowing lots of code sharing. It's reactive and thus has a Kotlin language dependency. JavaFX is a more traditional OOP toolkit with CSS support, good business widgets and is accessible from more languages for those who prefer that; it also now has a modern GitHub-inspired SASS based style pack that looks great [6] (grab the sampler app here [7]). For Lispers there's a reactive layer over the top [8].
4. There's some smaller tools that would be useful e.g. for letting you log into your DB with OAuth, for ensuring DB traffic can get through proxies.
Downsides?
1. Migrating between DB vendors is maybe harder. Though, the moment you have >1 web server you have the problem of doing a 'live' migration anyway, so the issues aren't fundamentally different, it'd just take longer.
2. Users have install your app. That's not hard and in a managed IT environment the apps can be pushed out centrally. Developers often get hung up on this point but the success of the installed app model on mobile, popularity of Electron and the whole video game industry shows users don't actually care much, as long as they plan to use the app regularly.
3. To do mobile/tablet you'd want to ship the DB driver as part of your app. There might be oddities involved, though in theory JDBC drivers could run on Android and be compiled to native for iOS using GraalVM.
4. Skills, hiring, etc. You'd want more senior devs to trailblaze this first before asking juniors to learn it.
[1] https://www.postgresql.org/docs/current/ddl-rowsecurity.html
[2] https://docs.microsoft.com/en-us/sql/relational-databases/se...
[3] https://docs.oracle.com/database/121/TDPSG/GUID-72D524FF-5A8...
[4] https://hydraulic.software/
[5] https://www.jetbrains.com/lp/compose-mpp/
[6] https://github.com/mkpaz/atlantafx
[7] https://downloads.hydraulic.dev/atlantafx/sampler/download.h...
There were other issues of course, it wasn't just about distribution. Too bad so many downsides came along with the upsides. The real goal for OS research should be IMHO to find ways to combine what people like about web dev with what people like about desktop/mobile dev. All the action is above the POSIX layer.
What do you think drove this? Presumably plenty of people in the dark mass of 9-to-5 devs were happy with VB/Delphi. Jonathan Edwards has written [1] that VB came from "a more civilized age. Before the dark times… before the web." Did nerdy devs like me, with our adolescent anti-Microsoft attitude (speaking for myself anyway; I was born in 1980), ruin everything?
There is revisionist history, and then there is that statement. That statement ... is almost offensive.
... Why did the "golden age" end? Because Microsoft sucked in so so so so so so so many ways. That statement is a bald faced lie, a whitewashing attempt to forgive Microsoft from inflicting millions of man years in damage to the IT industry over three decades of anticompetitive practices to keep their substandard software in a position of market dominance.
But for anyone beyond the MCSE factory programmer (aka NOT the thought leaders of the industry) aside from those profiting handsomely from the Microsoft evil empire, did not like Microsoft.
In the days of DOS, you had DOS or UNIX. Which was better?
In the days of Windows in the 3.1 and even pretty much in the windows 95, you didn't have preemptive multitasking (something that non-windows had for 20 years at that point). It crashed CONSTANTLY, had no security, required restarts for practically anything that was installed.
Meanwhile the UNIX people would brag about machines not being rebooted for years. This was before the constant patch cycle was a thing.
Microsoft's apis were slapdash and disorganized, and frequently went out of favor.
During this time Microsoft would constantly crush and screw over competitors and engage in constant anticompetitive behavior. If you didn't suck on the teat of Microsoft, your platform was under unrelenting assault, not by pure technical achievement, but by the full smorgasborg of corporate dirty tricks, FUD, bribery/kickbacks, lobbying, lies, secret apis, golf schmoozing with nontechnical CIOs to close deals, no or undermined standards, etc.
The graveyard is long: Sybase, Lotus 123, Netscape, Novell.
The Microsoft times were a time of being stuck with one option: and OS that crashed constantly or is utterly porous to attackers. A browser that has incompatible web apis and a disbanded developer team (IE 6) that gets corporate mandated and is a thorn in your side in the entire IT stack for two decades. Databases stolen from companies (Sybase signed SUCH a bad deal, it astonishes me to this day) running on platforms that can't stay up. Office software with inaccessible file formats and byzantine and closed programmatic apis for accessing it. A substandard desktop UI.
If you used Microsoft software with an ounce of historical knowledge or awareness, you could see the prison. You had no practical choices. All the executives in your company were bought and paid for. Microsoft had sales forces that tracked non-Microsoft systems and targeted them within companies by any means necessary. Every new piece of software written in MIcrosoft had to pay the "sociopath management tax" and go through extensive reviews on how it could be used to further or maintain Microsoft's empire and control.
Their software was deliberately dumped in beta form onto the market to crowd out the competitors.
None of this is an "adolescent" attitude. I'll repeat myself: MILLIONS OF MAN HOURS OF DAMAGE. You know, probably billions. Decades x tens of millions of workers.
This isn't just IT programmer frustration. This is bad applications forced on non-programmer users. This is better companies, better software, better IT industry denied proper funding and profits. Instead, Microsoft took trillions of dollars in revenues from them. This is undermining a free market, free ideas, and freedom for Microsoft's profit.
Also, VMS and other OSes were the ones that ran for years without rebooting. Unix at the time was not so stable. Before Tcl, John Ousterhout wrote a log-structured file system for Unix because their Unix systems crashed often enough and took so long to boot that a possible loss of data but fast boot-up was deemed better than the lengthier downtimes with the existing file system.
So the PC market went with Microsoft and its encompassing environment, much to everyone's detriment. Fortunately, we've all moved on to the web and JavaScript and everything is now sunshine and roses. :-)