今天在做一次推送消息的时候,发现好多数据推送不成功,nginx 的日志文件里,返回很多以下错误,我用的环境是 nginx + php7
2018/09/20 16:55:53 [error] 22050#0: *6840 upstream prematurely closed connection while reading response header from upstream, client: 127.0.0.1, server: localhost, request: "GET /1.php HTTP/1.0", upstream: "fastcgi://127.0.0.1:9000", host: "127.0.0.1"
2018/09/20 16:55:55 [error] 22050#0: *6835 upstream prematurely closed connection while reading response header from upstream, client: 127.0.0.1, server: localhost, request: "GET /1.php HTTP/1.0", upstream: "fastcgi://127.0.0.1:9000", host: "127.0.0.1"
2018/09/20 16:56:01 [error] 22052#0: *6132 upstream prematurely closed connection while reading response header from upstream, client: 127.0.0.1, server: localhost, request: "GET /1.php HTTP/1.0", upstream: "fastcgi://127.0.0.1:9000", host: "127.0.0.1"
2018/09/20 16:56:01 [error] 22052#0: *7067 upstream prematurely closed connection while reading response header from upstream, client: 127.0.0.1, server: localhost, request: "GET /1.php HTTP/1.0", upstream: "fastcgi://127.0.0.1:9000", host: "127.0.0.1"
2018/09/20 16:12:51 [error] 5395#0: *44785 upstream timed out (110: Connection timed out) while connecting to upstream, client: 127.0.0.1, server: localhost, request: "GET /1.php HTTP/1.0", upstream: "fastcgi://127.0.0.1:9000", host: "127.0.0.1"
2018/09/20 16:12:51 [error] 5395#0: *44786 upstream timed out (110: Connection timed out) while connecting to upstream, client: 127.0.0.1, server: localhost, request: "GET /1.php HTTP/1.0", upstream: "fastcgi://127.0.0.1:9000", host: "127.0.0.1"
错误信息大概是“上游服务关闭,读取数据失败”,当消息小的时候,却不会有这样的错误,可能是并发造成的。于是做一下 ab 测试
ab -n 1000 -c 200 http://127.0.0.1/1.php
好吧,200个并发时,出现了大量的错误信息,总体上是两个错误,即:
# 错误1
upstream prematurely closed connection while reading response header from upstream
# 错误2
upstream timed out (110: Connection timed out) while connecting to upstream
解决错误1
首先想到的,既然是并发才会出现此问题,那是不是系统已经到达瓶颈了呢?(本地环境,从来没有做过优化),vmstat 一下,如图:
发现什么内存、IO、CPU没有什么压力的,vmstat 命令的使用,可以去 vmstat 查看
既然不是服务器本身的问题,就是软件配置或代码的问题了,问题是“上游提前关闭”,应该是出在 php-fpm 上,打开 php-fpm.log 日志文件(默认在PHP的安装目录下的 var/log/php-fpm.log 位置)
( 难道对CGI来说 就是 修改php的 set_time_limit(0); ini_set("max_execution_time", 0); 就不会上游提前关闭了 )
[20-Sep-2018 16:55:28] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 13 total children [20-Sep-2018 16:55:29] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 14 total children [20-Sep-2018 16:55:30] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 15 total children [20-Sep-2018 16:55:31] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 16 total children [20-Sep-2018 16:56:20] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 0 idle, and 20 total children [20-Sep-2018 16:56:21] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 16 children, there are 0 idle, and 21 total children [20-Sep-2018 16:56:22] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 22 total children
有好多这种警告,虽然警告,但是也是很严重的,说是让修改 start_servers 这些参数,先修改看下再说,打开 etc/php-fpm.d/www.conf
; This value sets the limit on the number of simultaneous requests that will be
; served. Equivalent to the ApacheMaxClients directive with mpm_prefork.
; Equivalent to the PHP_FCGI_CHILDREN environment variable in the original PHP
; CGI. The below defaults are based on a server without much resources. Don't
; forget to tweak pm.* to fit your needs.
; Note: Used when pm is set to 'static', 'dynamic' or 'ondemand'
; Note: This value is mandatory.
pm.max_children = 1024 ; 修改成 1024 根据实际情况来修改,我这是本地环境,先随便增加
; The number of child processes created on startup.
; Note: Used only when pm is set to 'dynamic'
; Default Value: min_spare_servers + (max_spare_servers - min_spare_servers) / 2
pm.start_servers = 512 ; 修改成 512 根据实际情况来修改,我这是本地环境,先随便增加
; The desired minimum number of idle server processes.
; Note: Used only when pm is set to 'dynamic'
; Note: Mandatory when pm is set to 'dynamic'
pm.min_spare_servers = 4 ; 修改成 4 根据实际情况来修改,我这是本地环境,先随便增加
; The desired maximum number of idle server processes.
; Note: Used only when pm is set to 'dynamic'
; Note: Mandatory when pm is set to 'dynamic'
pm.max_spare_servers = 512 ; 修改成 512 根据实际情况来修改,我这是本地环境,先随便增加
然后重启 php-fpm,再后再做一次 ab 压力测试,发现上面的问题已经不存在了,看来真的是这儿出问题了,那这些参数是什么意思呢
php-fpm 进程池开启进程有两种方式:
static,直接开启指定数量的php-fpm进程,不再增加或者减少。
dynamic,开始时开启一定数量的php-fpm进程,当请求量变大时,动态的增加php-fpm进程数到上限,当空闲时自动释放空闲的进程数到一个下限
要用到的一些参数,分别是pm、pm.max_children、pm.start_servers、pm.min_spare_servers和pm.max_spare_servers。
pm 表示使用那种方式,有两个值可以选择,就是static(静态)或者dynamic(动态)
pm.max_children:静态方式下开启的php-fpm进程数量,在动态方式下他限定php-fpm的最大进程数(这里要注意pm.max_spare_servers的值只能小于等于pm.max_children)
pm.start_servers:动态方式下的起始php-fpm进程数量。
pm.min_spare_servers:动态方式空闲状态下的最小php-fpm进程数量。
pm.max_spare_servers:动态方式空闲状态下的最大php-fpm进程数量。
如果pm设置为static,那么其实只有pm.max_children这个参数生效。系统会开启参数设置数量的php-fpm进程。
如果pm设置为dynamic,4个参数都生效。系统会在php-fpm运行开始时启动pm.start_servers个php-fpm进程,然后根据系统的需求动态在pm.min_spare_servers和pm.max_spare_servers之间调整php-fpm进程数。
特别说一下,这些参数要根据实际情况来修改,增大之后会增加CPU的压力,如图
解决错误2
这种错误是经常出现的,一般出现,nginx 都会返回 502 或 504。但是,导致这种错误的原因也很多,以我现在的情况为例,把第1个问题解决好之后,这个也没有出现了。
网上也有这种答案,比如告诉要配置,以下参数
...
location / {
proxy_connect_timeout 159s;
proxy_send_timeout 600;
proxy_read_timeout 600;
proxy_buffer_size 64k;
proxy_buffers 16 32k;
proxy_busy_buffers_size 64k;
proxy_temp_file_write_size 64k;
proxy_pass_header Set-Cookie;
proxy_redirect off;
proxy_hide_header Vary;
proxy_set_header Accept-Encoding '';
proxy_ignore_headers Cache-Control Expires;
proxy_set_header Referer $http_referer;
proxy_set_header Host $host;
proxy_set_header Cookie $http_cookie;
proxy_set_header X-Real-IP $remote_addr;
....
}
...
另外,可以把 PHP 的慢日志暂时打开,可以看一些哪些程序执行慢,我也是第一次用到PHP中的慢日志,修改
; The log file for slow requests
; Default Value: not set
; Note: slowlog is mandatory if request_slowlog_timeout is set
slowlog = log/$pool.log.slow ; 这一行打开,日志的存放位置,如果不存在 log 目录,需要手动创建
; The timeout for serving a single request after which a PHP backtrace will be
; dumped to the 'slowlog' file. A value of '0s' means 'off'.
; Available units: s(econds)(default), m(inutes), h(ours), or d(ays)
; Default Value: 0
request_slowlog_timeout = 25s ; 请求时间,超过此时间的都会被记录下
未经允许不得转载:易读小屋 » 记录一次 upstream prematurely closed connection while reading response header from upstream