分享|写代码的时候发现的 bug,感觉很有意思 整理出来给大家
26
发布于 江西

间歇性挂起的 HTTPS 服务器

背景

本地测试用的简易 HTTPS 服务器,非阻塞 socket + epoll,任何请求都返回 403。会话分两个 map 管理:

· ssl_init:刚 accept,握手进行中

· ready:握手完成,可以收发应用数据

测试中发现一个稳定复现的规律:

延迟插入位置 时长 结果

不插入 — 正常返回 403

Ssl_accept() 与 test_drop_read() 之间 10ms / 100ms / 1s 客户端挂起

test_drop_read() 之后 10ms / 100ms / 1s 正常返回 403

只有位置敏感,时长无关。

---

材料 A:相关代码

```cpp

// ============ Client_session ============

class Client_session {

SSL* ssl;

std::optional<Client_info> cinfo;

// ...

public:

// std::expected:has_value() == true 表示成功,返回 {};

// has_value() == false 表示失败,返回 std::unexpected(...)

std::expected<void, Ssl_error_struct> Ssl_accept(void)

{

if (cinfo.has_value() == false)

return std::unexpected(Ssl_error_struct("Client_session::Ssl_init: client info is invalid\n"));

if (ssl == nullptr)

return std::unexpected(Ssl_error_struct("Client_session::Ssl_init: SSL* is nullptr\n"));

if (int res = SSL_accept(ssl); res <= 0)

return std::unexpected(Collect_all_ssl_errors());

return {};

}

inline void test_drop_read(std::ofstream& log) {

char buffer[1024]{};

if (int num = SSL_read(ssl, buffer, 1023); num > 0) {

log << "----------\n" << buffer << "\n----------\n";

log.flush();

}

}

void test_send_403() {

const char* resp = "HTTP/1.1 403 Forbidden\r\nContent-Length: 0\r\n\r\n";

SSL_write(ssl, resp, strlen(resp));

}

};

```

```cpp

// ============ 主循环 ============

while (true) {

int n = epoll_wait(epfd, events, MAX, -1);

for (int i = 0; i < n; ++i) {

if (server.Is_my_fd(events[i].data.fd)) {

auto cinfo = std::move(server.Accept4(SOCK_CLOEXEC | SOCK_NONBLOCK)).value();

int fd = cinfo.fd;

Client_session cs(std::move(cinfo), server.Get_ssl());

cs.Ssl_set_fd();

server.Register_client_epoll(cs, EPOLLIN | EPOLLRDHUP); // ADD

Map_ssl_init.emplace(fd, std::move(cs));

continue;

}

if (Map_ssl_init.contains(events[i].data.fd)) {

auto res = Map_ssl_init.find(events[i].data.fd);

if (events[i].events & EPOLLRDHUP) {

server.Register_client_epoll(res->second, 0, EPOLL_CTL_DEL);

Map_ssl_init.extract(events[i].data.fd);

continue;

}

if (events[i].events & EPOLLIN) {

auto accept_res = res->second.Ssl_accept();

// usleep(100000); // ← 待测延迟位置 A

res->second.test_drop_read(log);

// usleep(100000); // ← 待测延迟位置 B

if (accept_res.has_value()) { // has_value() == true → 成功

server.Register_client_epoll(res->second, EPOLLOUT, EPOLL_CTL_MOD);

Map_ready.emplace(events[i].data.fd, std::move(res->second));

Map_ssl_init.extract(events[i].data.fd);

}

}

}

if (Map_ready.contains(events[i].data.fd)) {

if (events[i].events & EPOLLOUT) {

auto res = Map_ready.find(events[i].data.fd);

res->second.test_send_403();

server.Register_client_epoll(res->second, 0, EPOLL_CTL_DEL);

Map_ready.extract(events[i].data.fd);

}

}

}

}

```

---

材料 B:两次完整运行日志

运行 ①(延迟在位置 B,客户端正常收到 403)

```

Server wakeup

Server: listen success, port: 8118

[epoll_ctl] epfd=5 op=1 fd=4 events=0x2001 data.fd=6 ret=0 errno=0

listen

[epoll_ctl] epfd=5 op=1 fd=4 events=0x2001 data.fd=7 ret=0 errno=0

listen

----------

GET / HTTP/1.1

Host: localhost:8118

Connection: keep-alive

User-Agent: Mozilla/5.0 ... Edg/155.0.0.0

Accept: text/html,...

Accept-Encoding: gzip, deflate, br, zstd

----------

[epoll_ctl] epfd=5 op=3 fd=4 events=0x4 data.fd=6 ret=0 errno=11

Into ready map

get epollout!

249

[epoll_ctl] epfd=5 op=2 fd=4 events=0x0 data.fd=6 ret=0 errno=0

Client_info::Destructor: fd "6" is free

did ssl have errors?:

Not collect ssl_error.

[epoll_ctl] epfd=5 op=3 fd=4 events=0x4 data.fd=7 ret=0 errno=11

Into ready map

get epollout!

249

[epoll_ctl] epfd=5 op=2 fd=4 events=0x0 data.fd=7 ret=0 errno=0

Client_info::Destructor: fd "7" is free

did ssl have errors?:

Not collect ssl_error.

```

运行 ②(延迟在位置 A,客户端挂起)

```

Server wakeup

Server: listen success, port: 8118

[epoll_ctl] epfd=5 op=1 fd=4 events=0x2001 data.fd=6 ret=0 errno=0

listen

[epoll_ctl] epfd=5 op=1 fd=4 events=0x2001 data.fd=7 ret=0 errno=0

listen

----------

GET / HTTP/1.1

Host: localhost:8118

Connection: keep-alive

User-Agent: Mozilla/5.0 ... Edg/155.0.0.0

Accept: text/html,...

Accept-Encoding: gzip, deflate, br, zstd

----------

[epoll_ctl] epfd=5 op=1 fd=4 events=0x2001 data.fd=8 ret=0 errno=0

listen

----------

GET /favicon.ico HTTP/1.1

Host: localhost:8118

...

----------

[epoll_ctl] epfd=5 op=2 fd=4 events=0x0 data.fd=7 ret=0 errno=0

Client_info::Destructor: fd "7" is free

[epoll_ctl] epfd=5 op=2 fd=4 events=0x0 data.fd=6 ret=0 errno=0

Client_info::Destructor: fd "6" is free

[epoll_ctl] epfd=5 op=2 fd=4 events=0x0 data.fd=8 ret=0 errno=0

Client_info::Destructor: fd "8" is free

```

---

材料 C:test_drop_read 内部插桩

在 SSL_read 之后立即打印以下内容:

运行 ① 每次 test_drop_read:

```

ssl_read ret=785 err=0 ssl_pending=0 has_pending=0 outq=0 poll_r=1 revents=4

ssl_read ret=-1 err=2 ssl_pending=0 has_pending=0 outq=249 poll_r=1 revents=4

```

运行 ② 每次 test_drop_read:

```

ssl_read ret=785 err=0 ssl_pending=0 has_pending=0 outq=510 poll_r=1 revents=4

```

字段说明:

字段 来源

ret SSL_read 返回值

err SSL_get_error 返回值(0=NONE,2=WANT_READ)

ssl_pending SSL_pending(ssl)

has_pending SSL_has_pending(ssl)

outq ioctl(fd, SIOCOUTQ)

poll_r / revents poll(fd, POLLOUT, 0)(4 = POLLOUT)

---

材料 D:已排除项

· epoll 是 LT 模式,不是 ET

· Register_client_epoll 成功时返回空 std::optional,失败时有值(已读源码确认)

· epoll_ctl 的 epfd 与 epoll_wait 是同一个

· 客户端抓包显示 TLS 握手完整、HTTP 请求已发出

· poll 说 socket 可写,has_pending=0 说明 OpenSSL 内部没有待发数据

---

问题

1. 给出完整因果链

2. 写出修改后的关键代码,说明为什么修改后延迟放在任何位置都不会再挂起。

如果你觉得这里看的不清楚可以去我的服务器看HTML版,8 133 19 192(过年之后就过期了,我不保证续费哦,别打😭)

评论 (0)