samuel@oriontransfer.org wrote:

Thanks for your detailed reply. It's impressive and useful

that you have such a good knowledge of these issues.

No problem.

I spent some time just thinking about this issue, and how this

feature tries to solve the problem in Ruby. On the one hand, I'm fundamentally opposed to increasing the

surface area of Ruby when it could be done by writing a gem.

This has a massive upstream cost, affecting both JRuby and

Rubinius. While I appreciate what you are saying w.r.t.

maximising usage, I feel like building this into Ruby will

cause stagnation of progress long term - one solution for all

problems isn't always ideal. Seeing initiatives like

stdgems.org only reinforces how I feel about this.

I understood something it was already decided by matz and ko1 to

do something along the lines of auto-Fiber. Though I can't find

ko1's original message in the archives, it's mostly quoted in in

my reply to him:

http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/80531

I should note some languages like Go, Erlang, Haskell, and the

afore-mentioned Crystal all have lightweight threading along

these lines in the core language.

In their current state, Fibers are much less useful than the

equivalents in those languages; while native Threads are too

expensive. Something in between Fibers and Threads seems

desirable; maybe we can give auto-Fiber another (short) name;

but I'm not sure it's necessary.

I was also influenced to explore lightweight threading in a

rack-devel thread and the responses James Tucker wrote to me:

Subject: big responses to slow clients: Rack vs PSGI

It's somewhere in https://groups.google.com/group/rack-devel but

that requires JS; so I can't view or link to it using w3m :<

Generally speaking - I really appreciate the work that's been

done here. I also feel like you've reinvented nio4r, async and

a bunch of other stuff, at a very low level, without as much

testing, compatibility, etc.

That's a fair point about less testing and compatibility.

But, I think there is more code using normal Ruby stdlib

that can automatically take advantage of these changes

so we'll be able to nail down any problems quickly.

On a technical level, I consider the design of libev (used by

nio4r and async) too limited in that it does not take advantage

of thread-safety baked into kqueue and epoll. Thinking in terms

of "events vs. threads" too limiting. As I've said before;

combining them is advantageous because both have their uses.

kqueue is an thread-friendly queue, so is epoll.

This feels like the microkernel vs monolithic kernel debate,

too. On one level, isolation and compartmentalization provided

by micro-kernels is appealing; but the ease-of-development of a

monolith allowed Linux to become the kernel for nearly

everything, from tiny IoT devices to giant supercomputers.

And that doesn't preclude things like loadable modules and FUSE

for userspace filesystems from being useful, despite core

filesystem drivers being bundled with Linux. So I think async

can still be supported as an alternative for Ruby; but the

bundled implementation can benefit more from tighter integration

into the core.

A more recent example might be git; which included high-level

non-essential "porcelain" tools early on in addition to the core

"plumbing". Initially, it was intended that separately

maintained wrappers such as "cogito", would implement the

porcelain UI bits and git would remain low-level plumbing. That

ended up making both development and usage more complicated.

Eventually git swallowed up most of the cogito functionality and

cogito was abandoned.

git also ended up with bundled functionality that would've been

separately packaged in other VCSes, including import/export

tools for email, CVS, SVN, etc.

The most relevant example from git might be the bundling of

libxdiff in git, allowing optimizations and tweaks not possible

with an external diff. However, GIT_EXTERNAL_DIFF still remains

supported for less-common use cases.

On a non-technical level:

Finally, this (ruby-core) is one of the few places I can still

contribute to in the Ruby world. All other relevant Ruby

projects requires running non-Free software (including JS) and

having to abide accept Terms-of-Service set by a corporation.

Fwiw, I agree with Rubinius philosophy of implementing more of

Ruby in Ruby and would rather contribute to that; but the above

is a huge factor in why I went on to work on C Ruby, instead.

(the other major factor is I strongly prefer C to C++).

Ideally, we could move all socket related code into a gem -

perhaps that's already on the cards e.g. stdgems. Once that's

done, fixing issues like exceptions: false would be easier

since it can be versioned.

Maybe that'll be done, too, but not my call.

But what about IO.pipe, backtick , and IO.popen?

I was thinking about how we could expose this to Ruby - and

ideally, I think we should add two functions: IO.wait_for_single_fd and IO.wait_for_pid . The C functions

rb_wait_for_single_fd and rb_waitpid would invoke these

functions, and these functions would implement the current

logic of the current C functions. It probably makes sense to

think in more detail how these functions should work - e.g.

wait_for_multiple_fds (or select ), or something more

elaborate.

Maybe.. I guess we already have IO#wait_able in io/wait; and

Process.wait/IO.select is already possible to override and that

would have the same effect. We'd also have to expose the

optional read/write buffering + encoding conversion and make

that accessible to pure Ruby.

It would make C Ruby feel closer to Rubinius and that would be

nice :) I'm not sure how feasable it would be; to introduce

more Ruby-visible APIs to implement this.

And I think exposing more APIs to handle FDs directly is a

mistake in the presence of native threads. My proposed C API

prefers "int *fd" and "rb_io_t" to deal with close notification

handling. Multithreaded programs recycle FDs frequently and

internal APIs need to be prepared to deal with that.

The implementation I proposed also takes advantage of some

C-only optimizations such as reading/writing to memory across

Fiber stack boundaries: something which cannot be done with

higher-level APIs . Similar optimizations already landed for

thread_sync.c (Mutex/Queue) as well as IO#close in trunk.

Again, designing user-visible APIs is most difficult and

ruby-core have to think most about long-term support and

consequences.

So the difficulty of changing/adding APIs is:

1) internal C API (easiest) 2) public C API (difficult) 3) Ruby API (most difficult)

So, I've mainly done 1) and made minimal additions to 3).

Only changes to 2) are to internal behavior, so use from C

extensions remains the same.

Then, we could allow things like async and auto-fibers to

extend Ruby's IO system to provide a policy for blocking IO.

auto-fibers could be implemented as a gem with a C

extension. What do you think?

I guess this is meant for matz and ko1.

We could actually have that today; and I guess you already have

that with async . All the IO methods are well-documented and

you can even ignore/override the existing IO buffering if you

override all the methods by monkey patching core classes.

Heck, you may even go as far as to never allocate rb_io_t

if you override IO.open/IO.pipe/*Socket.new/... and replace

them with your own class.

What I think is (or at least ought to be) irrelevant.

I only give matz and ko1 another option to choose from. We can wait

for matz and ko1 to decide what to do, maybe they'll discuss

this at: https://bugs.ruby-lang.org/projects/ruby/wiki/DevelopersMeeting20170616Japan

I certainly won't attend meetings or try to influence anybody

using anything besides plain-text messages, here.