tornado: disable epoll and add a test that shows why.
Here's a simple test program that shows the problem with epoll.
import socket, select, os
s1, s2 = socket.socketpair()
fd = os.dup(s1.fileno())
e = select.epoll()
e.register(s1, select.EPOLLIN)
e.poll(1)
s2.close()
e.poll(1)
e.poll(1)
os.close(3)
os.close(fd)
e.poll(1)
libcurl apparently still has a bug where it will sometimes close and replace
its underlying file descriptor *before* telling epoll (or not even telling
epoll at all). select() would have no problem with this, since the fd is
the same before and after, but epoll remembers the identity of the fd, which
persists even if the file no longer exists in this process (like if it was
inherited by a subprocess). Unfortunately, it keeps returning the original
fd number, not the new one, which is both useless and confusing. And to
make things worse, it doesn't poll on the new file descriptor that inherited
the old number!
This causes these possible symptoms:
1. if a subprocess inherits our libcurl fd, we'll keep getting epoll events
on it under some conditions, even after we think we've closed it. When
the remote http server dies, we'll just get endless EOF events on a fd
we can't possibly read anymore since it doesn't even exist in this
process.
2. if the remote server *doesn't* close our
connection right away, catawampus might just sit forever (or until
a local timeout if one exists) waiting for the server to respond on the
new socket. The server might in fact respond, but we'd never get an
event about it. This could explain strange catawampus delays. This could
happen regardless of whether a subprocess has a copy of the original fd
or not; the main thing is epoll is not listening on the new fd.
The "right" fix would be to overhaul libcurl to always unregister its fds
before closing or replacing them, but that's too hard (because libcurl's fd
logic is very confusing). Instead, let's just bludgeon it by switching to
good old-fashioned select(), which is more resilient to such things.
select() is less scalable than epoll() if you have zillions of fds (which is
what tornado was originally built to handle), but our use case doesn't need
this.
b/17575492
Change-Id: I1d57a7d8defa48e0b003e10b59afa474d4c4e2bd
1 file changed
