Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- What does the (currently latest, https://github.com/systemd/systemd/issues/40262#issuecomment...) comment mean? Who are the "large folio people"?by micw
- by pseudalopex
- Someone should implement a new operating system that can efficiently handle text processing.
It could have some simple tools that let you generate reports, display them on screen, and compose tools for that sort of thing in a natural way.
We could call it UNIX.
by hedora - Ooh, mmapped writes. I make that mistake once, years ago. :) I posted a comment in that GH issue.by amluto
- Say more? Sounds like a good storyby zbentley
- How do you try to copy Windows NT's Event Log — which is essentially unchanged from the 1990s when systems ran on 32MB of RAM or less — and fail so spectacularly?
The first thing I do on a Linux system is install a proper syslog daemon.
by pudgywalsh - Which log daemon do you use?
- https://utcc.utoronto.ca/~cks/space/blog/linux/SystemdJourna...
"The systemd journal doesn't force you to not have plain text logs" -- Chris Siebenmann (2024-06-30)
- One of the first things I do on win10 is disable most of excess logging.by rasz
- by ValdikSS
- Thanks for that comment. Out of curiosity how did you come up with that setup? For me most of that test suite sounds alien
- I recently looked into disk usage of journald and was also shocked. My next step towards peace of mind is https://www.devuan.org/os/init-freedom
Will try it out as next distro for my Debian system, longtime experience with Void Linux (runit) on another box is great.
by smartmic - I wish Qubes Domain-0 was a customized Gentoo with OpenRC. Fedora with systemd was a poor choice to base off. Nobody should have let Poettering have the influence he was given over userland, systemd is an almost irrevocable mistake.by rustcleaner
- I’ve been using devuan more or less since day one. I highly recommend it.
FreeBSD isn’t too shabby these days either.
by hedora - systemd-journald has one of the most deranged log file formats I have ever dealt with, and one of the worse user interfaces, too.
I am not again binary logs, or logs in a database. It's just yet another time I deal with good ideas implemented horribly, horribly badly when it comes to systemd.
by p_l - Many applications hammer the disk even if the developers don't believe this is an issue, not only journald, unfortunately.
It's my third attempt to make my regular Linux desktop less disk-chatty. This is a huge issue for btrfs and for COW FS in general, because they have massive write amplification for small and frequent writes (38,7 TB written to my idle desktop SSD in 2 years).
If you're interested, here are my findings this time so far:
- workrave: 60 second stat sync https://github.com/rcaelers/workrave/pull/717 - kde klipper: saves to disk on every copy, even if permanent storage is disabled https://bugs.kde.org/show_bug.cgi?id=501030 - kde plasmashell: saves qt shader cache each time notification popup disappears https://bugs.kde.org/show_bug.cgi?id=523805 - bitwarden firefox extension: tries to connect to desktop application every 10 seconds, writes about every failure to browser's WebStorage 14+ KB https://github.com/bitwarden/clients/issues/22192 - firefox datareporting/glean: very chatty .mozilla/firefox/xxx/datareporting/glean/db/data.safe - ipfs: writes every received DHT announce to disk, 20 GB in 3 hours https://discuss.ipfs.tech/t/constant-writes-to-datastore-log/20316 - mailcow: redis saves data every 5 minutes https://github.com/mailcow/mailcow-dockerized/pull/7405by ValdikSS - journald is IMO the worst part of the systemd ecosystem. You're better off using it only as a router and not storing any logs in it. The indexing system it uses is slow and provides no control over chatty subsystems - you cannot truncate the logs for just a single identifier. For all the use indexing is doing you will get better performance out of a modern grep like ag or rg. Structure is worth something but it's better off somewhere other than journald.by barrkel
- Systemd is touted as being highly modular. So it should be easy enough to replace the logging module journald.
Why hasn't this been done if it's that terrible?
by magicalhippo - I recently put a lot of effort into reducing logging because of excessive writes. It was so much easier when everything had its own log and you could just look at which files were growing.by graemep
- I would much rather that they had used an existing database file format. Sqlite3 is robust and already present in the default installation of most Linux distributions. Querying system logs with SQL would be cool and likely faster than using the sd_journal API with all it's weird quirks.by e2le
- The cherry on the cake is that you practically cannot filter journald. The only option is limiting by severity (e.g. errors and higher) or switch to non persistent journald storage and forward to rsyslog and filter there.
Am a bit vague on the details but sometimes a driver goes bezerk and starts logging many times per second, e.g. a bug in amdgpu after resume from suspend. Took a while to get that filtered which luckily was only possible because it were kernel messages (dmesg), but for a while I had to disae persistent kernel logging which is dat from ideal.
I get that for certain core parts simplicity is more important than features. But journald is just too basic to enable persistent storage but I also don't want to switch it off.
by jck86 - If you have systemd>=253 you can make use of LogFilterPatterns[0] (in .service files), but it's really unpredictable, cumbersome to work with, and does not work with user services or non-service log sources.
[0]: https://www.freedesktop.org/software/systemd/man/latest/syst...
by mzajc - journald is awful for many reasons, but what makes it worse is that everything running on your machine thinks it has any rights to dump all the logs it wants unprompted. Open a file picker and kio will decide it's a good idea to spam tens or hundreds of thousands of entries into it a day, listing every single file you have in a directory with some log such as "No node found for item that was just removed" and that has zero impact to the user whatsoever. You almost need to keep a script tracking all the journal floods for every new service to make sure it's not treating your system log as its dumping ground. To be fair, the kernel and usb peripherals can also have a bad day and spam 3 million lines an hour into it, think input irq status -75.
It's too much of a chore to keep up with all the program-level configs (if they have them) and service files, but LogFilterPatterns in systemd can help in an unintended way: you can make one log blacklist with a .conf file in /etc/systemd/system/service.d/, and put in there all the patterns that spam your journal one by one, don't even have to chase misattributed loglevels. It just looks something like:
[Service]
LogFilterPatterns=~I am a completely useless log entry
LogFilterPatterns=~I am another useless log entry
But it doesn't pick up on identifiers and doesn't do anything for kernel spam. It's only great to make some messages shut up. Also, I'd consider any btrfs install that does not have nocow on cache, journal etc. to be defective.
by 0x_rs - I'm using a certain object storage implementation post minio enshitification. The software itself is great don't get me wrong but I've noticed their logs are basically unreadable. Its metrics and traces in json data meant to be rendered on a dashboard instead of being read by humans. It's also extremely verbose even at an INFO level.
Maybe just get these on an open telemetry endpoint instead? I also don't get why people send by default json logs to journald as it's clearly meant to be a replacement to syslog which is already a good standard.
by irusensei - systemd-journald also has rate limits that you can configure ;-)by micw
- > I'd consider any btrfs install that does not have nocow on...to be defective.
You're getting COW on the extents if you're snapshotting anyway.
by quotemstr - That was the task for years for syslog services that dealt with it without issue.by greatgib
- I completely agree with one of the comments from there:
> But the fundamental conclusion is: the design was wrong. It should not have used mmapped writes. pwrite would have been far better.
It really does not make any sense to use memory-mapped files when writing logs.
Not even pwrite makes sense, because logs should normally be written by opening and using the log files as append-only sequential files.
Only when reading logs, to search for problems, accessing them as read-only memory-mapped files is OK.
Actually not only for logs, but almost always, read-write memory-mapped files are either inefficient or too complex to use (i.e. to avoid problems you must carefully use msync and/or madvise, which eliminates the simplicity that makes memory-mapped files preferable to using pread/pwrite). It is better to use memory-mapped files only for read-only accesses, using the appropriate option flags in open and mmap.
by adrian_b - I wonder if something like https://github.com/open-telemetry/otel-arrow would be a better fit instead
- great and clear summary thank you!
I would laso add that if my design decisions or development actions lead to an issue affecting multiple linux distro defaults I would feel responsible and rush for a solid fix instead of this https://github.com/systemd/systemd/issues/15292#issuecomment...
by giov4