>>DonHop+(OP)
When I saw this title, I first thought it was about understanding the cockroach-like aliens in Factorio, but it is actually about queuing and congestion and parallel processing with conveyor belts and factories!

With the caveat that Factorio is EXTREMELY addictive, properly described as "programmer crack", here are some other Factorio related discussions:

Factorio – a game where you can automate basically anything (factorio.com)

https://news.ycombinator.com/item?id=11266471

http://www.factorio.com/

"This game is like crack for programmers." -kentonv (Tech lead for Cloudflare Workers, author of Sandstorm.io, Cap'n Proto)

https://news.ycombinator.com/item?id=11269098

Raycasting engine in Factorio game – Facto-RayO v1.0 [video] (youtube.com)

https://news.ycombinator.com/item?id=19878688

https://www.youtube.com/watch?v=7lVAFcDX4eM

Factorio to talk about horizontal scaling of software services (medium.com)

https://news.ycombinator.com/item?id=19095271

https://medium.com/@chrisfauerbach/gamification-of-horizonta...

Mapping Factorio with Leaflet (cloudflare.com)

https://news.ycombinator.com/item?id=18185058

https://blog.cloudflare.com/mapping-factorio-with-leaflet/

>>DonHop+(OP)
For those who don't know factorio is truly a masterpiece of a game. It's described as "programming the game". Strongly recommended to everyone here.

>>DonHop+(OP)
A better way to understand Kafka is to fill a room with rakes, turn of the lights, and walk around.

>>gnulin+tq
"Oh, s##t, I have to wake up in 2 hours"

-- said at least once by anyone who has played it...

>>gnulin+tq
And high quality network multiplayer! "Pair programming" was never this much fun!

>>briffl+Rs
Oh man, all those hours wasted trying to get bots so that I can automate iron outpost building only to get destroyed by a huge wave of biters. Epic memories.

>>gnulin+tq
So I must be the only person on HN who doesn't like it too much. I love the concept and the factory building - but I found the necessity to move a player character manually with direct controls to be incredibly detracting from the process of building factories. I have no idea why they couldn't just include a mode where you have your typical RTS-like mode where you just go and click on things that you want to do without playing a game of "will this 5 pixels tall character fit through this gap between two machines or not". Also, this might be a personal preference - I found the game's style to be hard to read, it's far too dark even when played at maximum brightness...it's just an endless sea of dark brown.

>>gambit+yA
Same. I need it to be a God game without shooting and maybe a less photo realistic, more contrasting texture set.

>>gambit+yA
In the true spirit of Factorio, there are mods to let you remove the player character and allow you to move arbitrarily around the map.

>>DonHop+(OP)
I think the analogy works better with queues and distributed services in general, rather than just Kafka.

>>gambit+yA
I think there is a "sandbox" mode in factorio which works like you describe. It has been some time since I last played but I recall it had something like that. You don't have a character in that mode.

>>gambit+yA
I find having to actually move to be a pretty engaging part of the game. You have to find resources, and that might be dangerous. Vehicles and concrete help you maneuver more quickly, but have their own risks/drawbacks. Defending your base and setting up an outpost starts out as a manual task until you know how to do it automatically.

In simpler terms, having to do things the slow and manual way makes the fast and automatic solutions even more gratifying.

>>DonHop+(OP)
The analogy helped me understand, but in factorio belts are required primarily because everything takes physical space. What are real world use cases for kafka? Is it primarily if your data is too much to simply store in a db?

>>chrisc+GC
> What are real world use cases for kafka?

He covers it under the introduction "Why bother with async messaging?".

Basically in the analogy the belts are queues. Factorio doesn't require queues, but it quickly leads to problems with resource management, as you essentially have to lock the system until your item is done processing.

>>DonHop+f
Some random Factorio-themed things of my own:

https://imgur.com/c51pTnP - here I tried to invent a way to solve the accessibility issues of moving walkways; https://news.ycombinator.com/item?id=17703131 for context.

https://mastodon.technology/@temporal/100646861775747986 - Factorio-inspired tootstorm about what I'd like to see in an IDE/debugger.

>>chrisc+GC
Push architectures put all of the control on the producer, but become quite painful in the face of an unreliable consumer. Pull architectures put all of the control on the consumer, but become quite painful in the face of an unreliable producer.

Message queues allow the producer to think it's participating in a push architecture, while the consumer believes it's a pull-based arch. Because the message queue itself does very little work, it gets to play a very well-behaved consumer insofar as the producer is concerned, while behaving like a well-behaved producer for the consumer. The queue's buffering then allows both the producer and the consumer to misbehave to some degree while the queue itself keeps the illusion of good behaviour.

Kafka specifically falls under the category of log-oriented message queues, which are eminently useful for distributing any workload that looks like "tail a log and process each line as it comes in" across a large number of nodes.

>>DonHop+f
Pretty off topic, but I'd appreciate it if you didn't use "____ crack" as a way to describe something that is addictive.

>>gambit+yA
My solution to this is just a mod for being able to interact with anything on the screen, and another for being able to move through all the gaps in structures you can't normally squeeze through. I think they are "Long Reach" and "Squeak Through", respectively. You still end up having to move around, but its much closer to what you're looking for, especially once you start getting player speed upgrades.

>>DonHop+(OP)
I wrote this article a couple of months ago and didn't expect it to appear on my news feed all of the sudden. Thanks for the love everyone. Any questions are welcome of course!

>>DonHop+(OP)
You lost me at the part about partitions. It might help if those gifs were labeled.

>>pdpi+aG
This. With the slight nuance that it works better if you don't think of it as queues, since that suggests messages being removed when processed.

The big win when introducing Kafka is that (apart from the schema) it completely decouples producers and consumers of messages. That in turn reduces the required coordination between independent teams within an organisation.

>>Lowkey+HI
Yeah, I started out labeling them, but that turned out to be extremely tedious. Does anyone know of tools that make annotating gifs easy?

>>X6S1x6+DG
Why? Isn't crack famously (or infamously) addictive?

>>realit+Kr
This sounds like an idea founded on experience, but without including any details about how you arrived at this sentiment the comment doesn't bring any value to the thread.

>>X6S1x6+DG
I coined the nickname "Deep Crack" for the EFF's DES cracker. I hope that's not objectionable. The first key's free!

    Coming into Los Angeles
    Bringing in a couple of keys
    But don't touch my bags if you please
    Mister Customs Man

https://en.wikipedia.org/wiki/EFF_DES_cracker

https://www.donhopkins.com/home/archive/humor/deep-crack.txt

>>gambit+yA
By mid-game your actions should be pretty divorced from the Player Character. Once you have a bot network you can place down items anywhere you can see on the map (i.e. have Radar coverage), allowing you to build anywhere you want without having to be nearby. You can even use the bot network to expand the bot network & radar coverage.

>>DonHop+(OP)
Isn't this a repost?

>>DonHop+f
Oh my, can someone ping Gwern to add Factorio to this list. https://www.gwern.net/Turing-complete

>>kingbi+yM
Additionally, maintaining the stock of material for your bot adds yet another fascinating layer of complexity to the game.

>>saint-+NQ
There's neither surprise nor accident in Factorio's Turing completeness. The developers were aware of and fans of the existence of proofs of games on that list (Transport Tycoon is a particular example that they intentionally modeled train mechanics from)

>>rs2329+1O
Dang suggested reposting it since it didn't get much attention the first time.

>>DonHop+f
Pro-tip to others who think 'Programmer Crack' is hyperbole- it is not.

Factorio does have a rather steep learning curve, and the UI is not the most intuitive. But. Once how it works clicks, it clicks, and suddenly morning. And I'm in my 40's, I'd thought my days of unexpected overnights while working on fascinating problems were mostly behind me. Highly recommended, but be aware.

>>root_a+ly
I used to play this with co-workers and we would end up having a standup meeting about it once we got back to the office in the morning.

>>pul+NH
How do you like the .17 update? :D

More seriously, I am curious about what people who haven't played Factorio think of the article. Since I've sunk $bignum hours in to it, I found your explanations clear, but I'd like to hear what those who haven't think.

>>Gauntl+9U
Obviously the circuit system is Turing complete on purpose, but I believe the belts and splitters may be Turing complete on their own, which would definitely be accidental and surprising to most.

>>chrisc+GC
A database is a huge API to share with another team, and queries from other people add load with a principle-agent problem [1]

Give people a firehouse of events or deltas, though, and if they want to query it, performance is now their problem - they build the database, update it, index it, etc.

This is part of scaling organisationally, not just removing the database as a bottleneck.

Events are also a route to making cache invalidation - one of the hardest problems in CS, as we know - tractable. Build your caches as services that consume the firehouse, and invalidate based on a join between events and the cached data.

[1] https://en.m.wikipedia.org/wiki/Principal%E2%80%93agent_prob...

>>jpk+rL
Not GP, but early versions of Kafka had a lot of poorly-chosen default settings and sharp edges. eg, it was tuned for speed and availability over reliability, so it could lose your data if you weren't careful (not dissimilar to early versions of MongoDB that way). It was possible/easy to set up topics with no replication, so you'd lose data on your first server outage. KStreams had no real error handling - it would just explode and not even really tell you why. I wasn't as close to the operational side, but there were a lot of problems there too.

IME it has gotten better over time, but it's still pretty easy to shoot yourself in the foot if you don't understand what you're doing with it.

>>Baeocy+9V
I had to uninstall it last weekend because I came to realize my playing it was no longer healthy. I mean, that sounds funny, but I'm being totally serious. I've had a few games like that in my life, and I've come to recognize a few signs that what I'm doing is not good for me anymore.

But wow, what a great game.

>>chrisc+GC
Let's say you have a few tens of millions of people using your webapp and you are collecting analytics/usage data from them. You definitely don't want them all trying to authenticate + write directly to a db; you need an intermediate layer to process all the data coming from tons of different sources.

You could potentially do that with a separate microservice you communicate with via http, but this requires a "liveness" of the microservice that isn't really necessary; you will often lose events if the microservice isn't able to keep up with the incoming load, and you need to process the events just as fast as they come in. The data flow is really just unidirectional so the response is unnecessary, you just need to reliably transmit the data.

Kafka lets you handle unidirectional data flows in a way that is lazier. The data producers just write to a service and the consumers connect to the service. In between, Kafka just behaves like a distributed message queue. Obviously this is a huge benefit over directly writing to a db or any other kind of offline storage since it can greatly reduce the connection overhead. The main benefit over using a microservice is that it relaxes the constraint that all the data is processed/handled exactly as it comes in. It makes non-critical data flow more redundant by adding this queue

(I don't think the linked article does a great job explaining why you would use kafka / what the alternatives are)

>>gambit+yA
I don't like it because it feels too much like work. For my job I already do a lot of ETL so why would I do ETL in my free time if they aren't even going to pay me

>>Baeocy+9V
> Factorio does have a rather steep learning curve

Not... at all? I mean, Vim has a steep learning curve[1]. Factorio is a game meant for the masses.

Of course any game is hard to master, but in that case any game that is not made for 3 year olds has a steep learning curve. You won't know how to set up and manage a 1kspm factory from day one, but that's okay, you can start by exploring and have a lot of fun doing that.

[1] (if you ignore that you can just start by hitting "i" and save with "<esc>:wq" and you have an ordinary editor, but that's not the point.)

>>lucb1e+761
The fact that you would even consider comparing the controls of a video game to vim probably means it’s not the most intuitive.

>>DonHop+(OP)
There's a similar web game - http://www.factoryidle.com, for those who can't/won't play FactorIO, with a similarly addictive quality.

>>mabbo+Z31
Whenever they release a new version, I'll boot it up and waste the better part of a weekend to get another mega-base running. Always satisfying

>>seaish+6X
with filter splitters being a thing now, I think they would be, yes.

>>fragme+591
I'm getting 403 bad request responses from cloudfront on that url

>>Baeocy+0W
I haven’t played Factorio, and have just barely used Kafka. Therefore, if there was any subtlety, it was lost on me. Nonetheless, this article was clear. I thought this cute and I liked it, but I’m not sure that I gained any new insights into Kafka. This is not a criticism!

However, one of the main advantages of Kafka, I thought, was that it’s so much more than a queue, since data is replayable, etc. The article does mention this as a shortcoming of the analogy, but isn’t this one of the main selling points vs other message queues?

>>vlunkr+A61
I think the parent was assuming "steep learning curve" here was qualified as "for a piece of software used by programmers"; while the people that downvoted them assume "steep learning curve" here was qualified as "for a game." It is certainly easy to learn compared to a code-base; it is certainly hard to learn compared to Bejeweled.

Mind you, compared to other “programming games” like e.g. Shenzhen I/O, the main gameplay loop of Factorio is quite a bit simpler to wrap your head around. It's only when you try to ultra-optimize everything that it reaches that level of complexity. (Or when you "program in Factorio", ala "programming in Minecraft.")

>>Baeocy+9V
I tried factorio several times and it didn't click for me as a 'crack' game. And I'm a software engineer, and I've loved computers since I was 5, and I spend most of my free time on my computer.

I didn't learn to write code until I was in my late teens although. Ultimately I've found factorio a rather tedious game, that kind of reminds me of my job. I also tend to not code things for fun, seeing how it will ultimately turn into a large project I won't have time to maintain. Kind of like how mechanics don't like maintaining their own cars.

It makes me wonder sometimes.

>>DonHop+(OP)
That's crazy! I installed it yesterday for the first time to try out the steam proton thing and was literally up till 3 am. I didn't realise the programmer crack angle till I read this. I'm screwed.

>>dEnigm+pL
My understanding is that it's similar to people taking offense at the term "grammar nazi" because it trivializes an atrocity

>>novok+cf1
What do you find motivating in software engineering?

I initially found Factorio interesting, mostly for the parts that encourage spreadsheet calculation, e.g. balancing the various rates of production and resource intake. Some other aspects appealed too, such as finding drill placements to fully cover ore deposits.

Where I lost interest is when the game became mostly about refactoring large transport belt networks. That reminds me of the non-fun aspects of programming.

>>Baeocy+9V
Factorio is also the only game on Steam that i know of, that has thousands of reviews and its rating is"Overwhelmingly Positive". It just might be the highest rated game on Steam and it hasn't even left Early Access.

>>xmonke+ok1
There's a Linux native version, no need for Proton.

>>lucb1e+761
Pfff! The only vi command anyone ever needs to learn is :q! ;)

>>xmonke+ok1
You're screwed.

Also press Alt

And use "Q" while hovering over an object to put it in hand (if it's in your inventory). Press "Q" again to drop whats in hand. I played waaayyy to long without knowing this and it is unbelievably useful, saves mountains of time digging through your inventory.

>>mabbo+Z31
I've only had 2 games like that in my life, EverQuest (also called EverCrack) and Factorio (also called Programmer Crack). I had to go cold turkey or I was skipping the gym and meals and not sleeping!

>>rishav+yq1
> It just might be the highest rated game on Steam

Indeed the second best positive percentage according to steamdb: https://steamdb.info/stats/gameratings/

No 29 if you just sort by positive reviews.

>>Baeocy+9V
0.17 made the UI not crap. Gotta enable it through the game beta opt-ins.

>>Baeocy+9V
I think the reason for this is that it pushes all the buttons your programmer brain wants pushed: designing building and debugging systems with a strong focus on optimisation. However it doesn't contain meetings, sprints, language oddities, politics or any of the stuff that makes normal programming less fun.

>>Baeocy+9V
I can't stand Factorio and find it too tedious and exhausting, which is probably related to why I find coding to be a chore, despite being a talented software engineer.

>>bluebl+yn1
I never particularly cared about belt optimization past 'good enough'. I spent most of my time trying to break things and find edge-cases in the control system. Working around the (purposefully nerfed to keep them from being too powerful) logistics bots issues was probably my favorite thing to tackle.

(That, and building vast quantities of artillery in the middle of almost fully red zones, then flipping the power switch for the loaders and watching the fun. :D)

>>DonHop+pU
Wow, I ask a valid question about a old post and I'm punished with multiple downvotes. Lesson learnt.

>>jjjens+4w1
One thing I credit Factorio for (and I am not joking at all when I say this, despite how ridiculous the statement might seem) is learning to be a little more understanding about folks that deal with addictions. I've never had a problem not doing something to excess, so I never really understood people who do. But when I first discovered Factorio a few years back, there were enough times where I didn't do $ActuallyRatherImportantThingX because I was allllmost done with some aspect of a megabase to give me pause.

There are lots of games I can play for a specified amount of time and be fine with it. Pretty much all of them, really. Factorio, I get annoyed if pulled away in the middle of a project. So now I know to not even start unless it's free time for the rest of the night.

>>Itsdij+pt1
Also, Z to drop something on the ground if you find your bots a little to assertive about filling your inventory back up and you find yourself stuck. :D

>>dEnigm+pL
I agree with the (apparently unpopular) notion and I avoid describing things as "x crack" and wince a bit when people use it as it seems a bit tone-deaf and childish, like referring to something you don't like that there's too much of as "programmer AIDS" or a bad design as a "UI Holocaust" or a failed project as an "abortion" or a bad sports defeat as being "raped", all of which make a similar kind of allegorical sense but are more obviously tacky.

For whatever reason crack allusions are a bad taste joke that people (unlike most of the above) don't seem to realise is bad taste, possibly because they consider crack addicts as either other or lesser to them, rather than unfortunate victims with a shared humanity.

>>derefr+ud1
> I think the parent was assuming "steep learning curve" here was qualified as "for a piece of software used by programmers";

No. For example, my cousin plays it, someone who typically plays games like the Sims (and other games popular amongst women), has no university degree. She has no trouble with Factorio. I am honestly confused how anyone would call Factorio a difficult game. It's nigh impossible to lose unless you change the default settings to generate a more difficult world with much more aggressive biters. It's a building game, not a shooter that requires any kind of skill or specialised software that requires a manual.

Edit: asked said cousin whether she thought it has a steep learning curve. Answer was "lol, yes! And the curve just continues, there is still lots I could do better". I suppose having nearly only programmers as friends skewed my view of how many people had no trouble getting into it.

>>jjjens+4w1
EverQuest was actually the game that taught me what the bad signs were.

In high school, my friend was into drugs and partying. I was into EverQuest. I think we'd have both been better long term if he'd gotten me into drugs instead of me getting him into EverQuest.

>>pul+NH
I really enjoyed reading this, thank you very much.

Please check the picture with the label "All microservice instances consume all messages". From my understanding it is more of a depiction of a consumer group.

As factorio really consumes the messages (they are gone once consumed) you cannot really implement "All microservice instances consume all messages".

Maybe this is BS, i have no Kafka expertise.

>>ZeroGr+PF1
I think that kind of word use is inevitable and actually a sign of something good. It for me shows that people know about these things but they are percieved to be so unbelievably horrible that they feel unreal and as such can serve as metaphors.

I know these things happened, happen and most likely will happen and such metaphors are unpleasant for people affected, but I think it's natural language evolution.

People still get murdered occasionally and families of murder victims might be uneasy when they hear about someone that "he killed her with this joke!" but it won't prevent people from using that phrase because murders are so rare for most people that the word gradually loses its literal meaning.

>>pdpi+aG
It’s a good theory, and definitely something you should be aware of but... in my experience if the producer fails, you get nothing (of course), and if the consumer fails, the messages just back up until the queue overflows and of course if the queue itself fails - which can happen due to hardware or network as well as software issues - you have a new point of failure that’s unrelated to the producers or consumers. So the whole thing is only as good as the responsiveness of your monitoring team... which is effectively the case with synchronous message delivery, only with the added complexity of troubleshooting the queue in the middle.

>>jand+mQ1
You're absolutely right. There's no correct way to visualize this in Factorio.

>>fredle+pB1
What if there was a package manager for Factorio?

>>realit+Kr
Doesn't Kafka also support mouse traps, too?

>>twic+tb2
Factorio packages are Blueprints and Blueprint Books! And the equivalent of npmjs.com would be https://factorioprints.com .

https://wiki.factorio.com/Blueprint

https://wiki.factorio.com/Blueprint_book

https://factorioprints.com/

https://www.reddit.com/r/factorio/comments/7vnkze/blueprint_...

One of the stages of evolution of a Factorio programmer is learning how to make scalable modular blueprints that you can stamp down one next to another, with their inputs and output routed so they all plug together easily and run in parallel, without having to be manually wired together.

>>root_a+ly
Katherine of Sky has some great "pair programming" videos where she teams up with other experienced players. It's amazing to see how diverse different people's styles can be! You really get insight into how their minds works.

I can't remember which ones are multi player, but here are some of her playlists -- awesome stuff:

Factorio Tutorials Playlist

https://www.youtube.com/playlist?list=PL4o6UvJIdPNpm9szThfI-...

Factorio 0.17: Entry Level to Megabase 2

https://www.youtube.com/watch?v=el9Ih1gg3i4&list=PL4o6UvJIdP...

Factorio 0.17: Pyanodon with Aven1017

https://www.youtube.com/playlist?list=PL4o6UvJIdPNpUZ230345G...

Factorio Landfill Avalanche! (MP with Aven!)

https://www.youtube.com/playlist?list=PL4o6UvJIdPNrEWhjZOzfC...

Factorio ASMR

https://www.youtube.com/watch?v=P7dasB1a5J0&list=PL4o6UvJIdP...

>Rabbi Tricky: OMG KoS, your voice, your choice of words, it makes me really relaxed. I saw it mentioned before, but its like Bob Ross. Happy little trees are now in my mind, together with happy little miners and such. Please don't make this a one time video, i really love this. Much love

>mithrane: KOS - You really are the Female Bob Ross of Factorio. Love your videos and love this concept.

>>xmonke+ok1
Oh no, what have I wrought? Another addict. Well hopefully it served as a warning to more people than the number of people it tempted into getting hooked. It’s only because I haven’t set up my PC since I moved that I have had any time in my life not playing Factorio myself. I still have the dreams, though.

>>rs2329+rE1
You were downvoted because you did not RTF FAQ.

>>ZeroGr+PF1
I was always kind of unsure why it made me feel weird until someone pointed out how weird it would be to say "OMG these fries are like meth". I think for many people there is a racial component to it as well.

>>dEnigm+pL
Not any more than cocaine. The major difference between the two is mostly who does them

>>NullPr+hG2
But you're happy to abuse me. So that is acceptable then.

>>DonHop+hp2
yeah at this point I consider a game like that solved and my interest in it disappears along with any craving to explore more. It is solved, move on.

>>serpix+PP4
That's a bit like saying programming is solved because package managers exist.

Just like NPM packages, blueprints just let you get into much more trouble much faster!

>>rs2329+6P4
How did you came to the conclusion that I am abusing you by pointing out that you did not read the FAQ?

My message was not even unsolicited. You were asking why and I gave you an answer.