Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Has anyone played with this and SQLite? I have no data, just a hunch, but I’d think this is a recipe for corruption if you’re doing lots of writes.by jherdman
- PG till the wheels fall off babyby the_sleaze_
- My concern is number of database connections. In the example it's 100 fibers per worker, at that rate you are going to exhaust db connections sooner. Happy to be wrong.by swe_dima
- I don't know Solid Queue or the rails environment, but I expect that not every worker will create it's own connection, there should be a connection pool in-betweenby looperhacks
- I think you want to make sure you don't hold a database connection while waiting on other slow async work (like outgoing HTTP requests). Then you can more feasibly have more workers than pool size. It's just very tricky to do this in Rails...by resonious
- Carmine (which coded this Fibers update) wrote about this in his blog. See the section 'The database connection math'. And no, you won't have one connection per fiber.
The difference is staggering when you compare to threaded mode: it requires 1,320 database connections to run the same benchmark that the fiber mode runs with 60.
https://paolino.me/solid-queue-doesnt-need-a-thread-per-job/
by pqdbr - This looks fantastic for a common async workflow I use. I often use one job to fan out multiple individual http request jobs. The reason I prefer jobs for this is easy and consistent retry logic, and durability. I want to make sure those HTTP requests eventually go through. Fibers would be much better suited for this. So much of work that goes onto work queues is IO bound, and fibers are a great fit for that.by mrinterweb
- Didn't EventMachine solve these types of issues way back when?
- EventMachine provided a event-reactor based concurrency model, but its API was so different and imposed a different style of programming (namely, callback-based) that it didn't really mesh with the "Ruby way" and most importantly was incompatible with most of the Ruby ecosystem (and especially Rails).by ciconia
- This is a nice update.
Is it possible to either have multiple ractors dispatching jobs with fibres or to set up multiple queues with different strategies?
E.g. one for IO bound and one for CPU bound?
With Sidekiq I’ve had luck having workers running on Truffleruby but generally don’t use it for my main rails apps.
by Lio - You can, and Carmine (who coded this update) wrote exactly about this on this blog post: https://paolino.me/solid-queue-doesnt-need-a-thread-per-job/
From his article:
One backend, two modes
Fiber mode isn’t universally better. CPU-bound jobs get nothing from it, and blocking libraries or C extensions that do not cooperate with Ruby’s fiber scheduler stall the reactor. And that’s fine – you don’t have to pick one.
As Trevor Turk pointed out in the PR discussion, that’s the whole point: separately configured worker pools. Here’s what Chat with Work actually runs in production:
workers: - queues: [ chat ] fibers: 10 processes: 2 polling_interval: 0.1 - queues: [ turbo ] fibers: 10 processes: 1 polling_interval: 0.05 - queues: [ notifications, default, maintenance ] fibers: 5 processes: 1 polling_interval: 0.2 - queues: [ cpu ] threads: 1 processes: 1
by pqdbr - So fibers are a lot like threads but they're more scoped to a task that can be paused and resumed, that's kinda coolby ramon156
- Some programmers find it easier to write concurrent programs that use "light-weight threads", "fibers", "goroutines", "coroutines" or other variants of this idea.
This feature is obviously intended for them and it might enhance their productivity.
Nonetheless, no program that uses a great number of any variant of the "light-weight threads" can ever be as efficient as a thread pool that is dimensioned to have the same number of threads as the number of hardware threads of a SMT CPU, or a slightly greater number of threads than the number of hardware threads of a non-SMT CPU.
For maximum performance, the use of a correctly-sized thread pool remains the best solution, but writing an efficient program that uses it can be significantly more difficult, because good methods of communication and synchronization must be implemented, while the run-time library of a language with "light-weight threads"/"fibers"/etc. already takes care of such problems so the programmer does not need to think about them.
by adrian_b - It’s more like goroutines or other lightweight concurrency mechanisms. If threads are OS-level concurrency primitives, fibers are scheduled within Ruby itself, which makes them much more efficient than threads.
In fact, I got the following results in HTTP benchmark tests:
Go:
* Latency under stable load: p95 0.25–5.32 ms, p99 2.20–9.92 ms * Memory: 23–31 MB RSS across HTTP scenarios
Ruby:
* Latency under stable load: p95 1.03–6.45 ms, p99 2.32–8.30 ms * Memory: 84–295 MB RSS, depending on the scenario
Fibers can also handle WebSockets better because WebSocket workloads involve more I/O waiting.
A typical Falcon setup uses N workers, one thread per worker, and many fibers. Since the fibers are cooperatively scheduled within a single thread, this avoids much of the context-switching overhead associated with OS threads. Multiple workers can still run in parallel across CPU cores.
by vdombr - As someone who spent 15+ years doing Ruby/Rails, it’s nice to see this land.
That said, these days you’ll pry the BEAM from my cold, dead hands. It’s hard to go back to any other concurrency story.
by symfoniq - What's BEAM?by thunderbong
- Same path & agreed. The amount of things you do not have to care about with BEAM/Elixir compared to Rails is really interesting.
- How has agentic coding been with rails? What has your experience been like?
Took me sometime to get used to the magic imports and metaclasses (ruby I know not rails). Curious how well llms navigate this
by tt_dev - Ironically, looking into Ruby Fiber led me to BEAM/Elixir. You are right, there is no going back!by ajx1001