魏长东

weichangdong

正在查看: 2016年10月日志归档(第 1 页 / 共 18 篇)

php swoole小试牛刀

一直觉得swoole很神奇,终于自己研究了一把,果然很牛逼。之前我实现队列,简单的就是while true死循环,一直查询数据库(mysql或者redis),如果有新数据就处理。稍微高级一点的就是,php fork子进程来处理,每处理完成一个就exit。

但是发现swoole实现起来,很简单。

»»阅读全文

nginx和elastic

那么使用nginx有什么好处呢?

1. 记录每个API访问请求的日志。(ElasticSearch本身不支持这个功能,只有slowLog和服务日志)

2. 支持大量的客户端连接。ES官方的blog中推荐使用keep-alives,在nginx和ES之间使用长连接。我理解是因为在通常情况下,ES都是架构中的底层,访问它的一般是固定的上层服务,这种情况是适用于使用keep-alive的。(实际上不管用不用keep-alive,nginx都可以起到支持更大量客户端连接的作用)

3. 负载均衡的请求Elasticsearch服务器。

4. 缓存数据,减少同一内容再次请求Elasticsearch服务器。

5. 提供主动健康检测(仅nginx plus),不断检测后端Elasticsearch服务器是否正常,并主动的进行切换。(当某台ES挂掉的时候,nginx不分发请求到此结点,当结点重新恢复正常时,自动归位)

6. 报告丰富的监控指标(仅nginx plus),提供监控和管理。

7. 安全验证。只让持有账户名密码的客户端访问到ES集群。

8. 对特殊接口如"_shutdown"限制访问。(这个功能相当实用)

9. 带角色的访问控制(比如user角色拥有数据访问权限,admin角色拥有集群管控权限)

====我是配置例子的分割线====

一个简单的nginx配置如下:

upstream elasticsearch_servers {
    zone elasticsearch_servers 64K;
    server 192.168.187.132:9200;
    server 192.168.187.133:9200;
    keepalive 40 ;
}
match statusok {
    status 200;
    header Content-Type ~ "application/json";
    body ~ '"status" : 200';
}
server {
    listen 9200;
    status_zone elasticsearch;
    location / {
        proxy_pass http://elasticsearch_servers;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_cache elasticsearch;
        proxy_cache_valid 200 302 10m;
        proxy_cache_valid 404 1m;
        proxy_connect_timeout 5s;
        proxy_read_timeout 10s;
        proxy_set_header Connection "Keep-Alive";
        proxy_set_header Proxy-Connection "Keep-Alive";
        health_check interval=5s fails=1 passes=1 uri=/ match=statusok;

    } 
    # redirect server error pages to the static page /50x.html
    error_page 500 502 503 504 /50x.html;
    location = /50x.html {
        root /usr/share/nginx/html;
    }
    access_log logs/es_access.log combined;
}
server {
    listen 8080;
    root /usr/share/nginx/html;
    location / {
        index status.html;
    }
    location =/status {
        status;
    }
}

长连接、负载均衡、对有效的请求缓存10分钟、主动的健康监测、状态收集。

====我是安全验证配置的分割线====

一个带安全验证的配置如下:

events {
  worker_connections  1024;
}

http {

  upstream elasticsearch {
    server 127.0.0.1:9200;
  }

  server {
    listen 8080;

    auth_basic "Protected Elasticsearch";
    auth_basic_user_file passwords;

    location / {
      proxy_pass http://elasticsearch;
      proxy_redirect off;
    }
  }
}

passwords文件和nginx.conf在同一目录,里面的格式是按行的"用户名:crypt(3)加密后的密码串":

$ printf "john:$(openssl passwd -crypt s3cr3t)n" > passwords

做完以上配置后重启nginx,则直接访问服务会被禁止:

$ curl -i localhost:8080
# HTTP/1.1 401 Unauthorized
# ...

通过正确的用户名密码则可顺利访问:

$ curl -i john:s3cr3t@localhost:8080
# HTTP/1.1 200 OK
# ...

====我是访问限制配置的分割线====

location / {
  if ($request_filename ~ _shutdown) {
    return 403;
    break;
  }

  proxy_pass http://elasticsearch;
  proxy_redirect off;
}

做了此配置之后,直接访问_shutdown会被拒绝:

$ curl -i -X POST john:s3cr3t@localhost:8080/_cluster/nodes/_shutdown
# HTTP/1.1 403 Forbidden
# ....


 

针对我目前的项目,上层应用仅需要访问ES中的数据,所以cluster和node等API接口都应拒绝上层应用的访问。同时,对不应被删除的资源进行-DELETE也应禁止。这对ES集群是一种安全保证,否则轻易就可以被修改集群配置或删除大量数据。
 

====我是多角色配置的分割线====

events {
  worker_connections  1024;
}

http {

  upstream elasticsearch {
      server 127.0.0.1:9200;
  }

  # Allow access to /_search and /_analyze for authenticated "users"
  #
  server {
      listen 8081;

      auth_basic           "Elasticsearch Users";
      auth_basic_user_file users;

      location / {
        return 403;
      }

      location ~* ^(/_search|/_analyze) {
        proxy_pass http://elasticsearch;
        proxy_redirect off;
      }
  }

  # Allow access to anything for authenticated "admins"
  #
  server {
      listen 8082;

      auth_basic           "Elasticsearch Admins";
      auth_basic_user_file admins;

      location / {
        proxy_pass http://elasticsearch;
        proxy_redirect off;
      }
  }

}

区分admins和users两种权限,admins可以访问一切API,而users只允许访问_search和_analyze接口。

多角色访问限制的代价是每个角色使用不同的端口号访问集群,这在架构上是合理的——一个客户端只需要拥有一种角色,也对应一个访问端口。

使用lua可以进行更细致的url权限控制,nginx对lua的嵌入也支持得很好很简洁,此处不做更多深入的探究。有兴趣可以了解。

参考文档:

http://www.ttlsa.com/nginx/nginx-elasticsearch/

https://www.elastic.co/blog/playing-http-tricks-nginx
 

版权声明:本文为博主原创文章,未经博主允许不得转载。

 

以上就介绍了ElasticSearch:Nginx可以给ElasticSearch集群带来什么福利?,包括了方面的内容,希望对PHP教程有兴趣的朋友有所帮助。

elastic search小知识

curl  localhost:9200/wcdweather/_count?pretty
curl -XGET localhost:9200/_mapping?pretty
几个es默认的分词器
curl -XPOST localhost:9200/_analyze?analyzer=standard -d '{"text":"New York"}'
curl -XPOST localhost:9200/_analyze?analyzer=whitespace -d '{"text":"New York"}'
curl -XPOST localhost:9200/_analyze?analyzer=simple -d '{"text":"New York"}'

nginx没有access日志

开始这么写,没有access日志
access_log  /var/log/php-fpm/5.6/weather.log  main;
root   "/work/wcd/";

调整了下,restart了就ok了
root   "/work/wcd/";
access_log  /var/log/php-fpm/5.6/weather.log  main;

access日志加上执行时间

加上 $request_time
log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for" $request_time';

用过的Bucket(生命不止,折腾不息)

最开始在SAE上使用Storage,通过php的sdk操作上传图片文件之类的,最后又玩Bucket,但是没有深入.

接下来工作用到aws的s3存储,用到了命令行的上传下载,以及php sdk的上传.

然后中间用到了七牛的图片存储,现在本站就是用的这个存储,直接php sdk上传,并且用到了他的cdn.

现在又买了阿里云的oss存储.计划存储图片,也是要用到php的sdk,因为现在一个网站正在备案的路上,所以暂时还没成型.将来会用到阿里云的cdn和oss结合.期待中........

elastic search 集群小知识

curl  localhost:9200/360weather/_count?pretty

curl  http://localhost:9200/_cat/indices?v
curl  http://localhost:9200/_cat/nodes?v
curl  http://localhost:9200/_cat/health?v

curl  http://localhost:9200/_cluster/health
curl  http://localhost:9200/_cluster/state

curl  http://localhost:9200/_nodes/process/stats

时区大全

http://www.zeitverschiebung.net/en/difference/timezone/Australia/Broken_Hill/timezone/asia--shanghai
时区和utc时区相差小时数
    [Asia/Riyadh] => 3
    [Asia/Tehran] => 3.5
    [Asia/Yerevan] => 4
    [Africa/Lusaka] => 2
    [Africa/Nairobi] => 3
    [Africa/Kigali] => 2
    [Africa/Lubumbashi] => 2
    [Africa/Kinshasa] => 1
    [Africa/Djibouti] => 3
    [Africa/Kampala] => 3
    [Africa/Bangui] => 1
    [Africa/Ndjamena] => 1
    [Asia/Amman] => 3
    [Africa/Blantyre] => 2
    [Asia/Nicosia] => 3
    [Africa/Luanda] => 1
    [Africa/Dar_es_Salaam] => 3
    [Asia/Baku] => 4
    [Indian/Mahe] => 4
    [Asia/Ashgabat] => 5
    [Asia/Damascus] => 3
    [Europe/Athens] => 3
    [Africa/Mogadishu] => 3
    [Asia/Aden] => 3
    [Africa/Tripoli] => 2
    [Asia/Baghdad] => 3
    [Africa/Addis_Ababa] => 3
    [Africa/Asmara] => 3
    [Africa/Cairo] => 2
    [Africa/Khartoum] => 3
    [Asia/Jerusalem] => 3
    [Asia/Kuwait] => 3
    [Asia/Qatar] => 3
    [Asia/Muscat] => 4
    [Europe/Riga] => 3
    [Europe/Moscow] => 3
    [Europe/Volgograd] => 3
    [Europe/Tirane] => 2
    [Asia/Yekaterinburg] => 5
    [Europe/Ulyanovsk] => 4
    [Europe/Samara] => 4
    [Europe/Astrakhan] => 4
    [Europe/Kaliningrad] => 2
    [Europe/Istanbul] => 3
    [Africa/Bujumbura] => 2
    [Asia/Beirut] => 3
    [Asia/Gaza] => 3
    [Asia/Dubai] => 4
    [Asia/Bahrain] => 3
    [Europe/Zaporozhye] => 3
    [Europe/Simferopol] => 3
    [Europe/Tallinn] => 3
    [Asia/Tbilisi] => 4
    [Europe/Chisinau] => 3
    [Europe/Minsk] => 3
    [Europe/Vilnius] => 3
    [Asia/Samarkand] => 5
    [Europe/Stockholm] => 2
    [Asia/Aqtobe] => 5
    [Asia/Aqtau] => 5
    [Asia/Almaty] => 6
    [Asia/Oral] => 5
    [Europe/Helsinki] => 3
    [Europe/Kiev] => 3
    [Europe/Uzhgorod] => 3
    [Europe/Bucharest] => 3
    [Europe/Bratislava] => 2
    [Europe/Sofia] => 3
    [Europe/Budapest] => 2
    [Europe/Skopje] => 2
    [Europe/Belgrade] => 2
    [Europe/Oslo] => 2
    [Europe/Warsaw] => 2
    [Europe/Podgorica] => 2
    [Asia/Kabul] => 4.5
    [Asia/Bangkok] => 7
    [Asia/Karachi] => 5
    [Asia/Dushanbe] => 5
    [Asia/Kuala_Lumpur] => 8
    [Asia/Colombo] => 5.5
    [Africa/Johannesburg] => 2
    [Asia/Bishkek] => 6
    [Asia/Thimphu] => 6
    [Asia/Kolkata] => 5.5
    [Africa/Maseru] => 2
    [Africa/Gaborone] => 2
    [Africa/Harare] => 2
    [Africa/Maputo] => 2
    [Indian/Antananarivo] => 3
    [Indian/Mayotte] => 3
    [Indian/Comoro] => 3
    [Indian/Mauritius] => 4
    [Africa/Mbabane] => 2
    [Indian/Reunion] => 4
    [Africa/Windhoek] => 2
    [Asia/Dhaka] => 6
    [Asia/Jakarta] => 7
    [Asia/Pontianak] => 7
    [Asia/Tashkent] => 5
    [Asia/Makassar] => 8
    [Indian/Maldives] => 5
    [Asia/Krasnoyarsk] => 7
    [Asia/Barnaul] => 7
    [Asia/Novokuznetsk] => 7
    [Asia/Novosibirsk] => 6
    [Asia/Irkutsk] => 8
    [Asia/Omsk] => 6
    [Asia/Tomsk] => 7
    [Asia/Urumqi] => 6
    [Asia/Kashgar] => 6
    [Asia/Kathmandu] => 5.75
    [Asia/Rangoon] => 6.5
    [Asia/Chongqing] => 8
    [Asia/Qyzylorda] => 6
    [Asia/Manila] => 8
    [Pacific/Palau] => 9
    [Asia/Ho_Chi_Minh] => 7
    [Asia/Jayapura] => 9
    [Indian/Cocos] => 6.5
    [Asia/Ulaanbaatar] => 8
    [Indian/Kerguelen] => 5
    [Asia/Vientiane] => 7
    [Asia/Taipei] => 8
    [Asia/Shanghai] => 8
    [Pacific/Pohnpei] => 11
    [Asia/Dili] => 9
    [Asia/Kuching] => 8
    [Asia/Harbin] => 8
    [Asia/Pyongyang] => 8.5
    [Asia/Tokyo] => 9
    [Asia/Seoul] => 9
    [Asia/Yakutsk] => 9
    [Asia/Vladivostok] => 10
    [Asia/Khandyga] => 9
    [Pacific/Port_Moresby] => 10
    [Pacific/Guadalcanal] => 11
    [Pacific/Tarawa] => 12
    [Pacific/Efate] => 11
    [Pacific/Noumea] => 11
    [Australia/Melbourne] => 11
    [Australia/Lindeman] => 10
    [Australia/Sydney] => 11
    [Australia/Brisbane] => 10
    [Pacific/Majuro] => 12
    [Asia/Kamchatka] => 12
    [Asia/Magadan] => 11
    [Asia/Ust-Nera] => 10
    [Asia/Sakhalin] => 11
    [Asia/Anadyr] => 12
    [Australia/Currie] => 11
    [Australia/Hobart] => 11
    [Australia/Adelaide] => 10.5
    [Pacific/Auckland] => 13
    [Australia/Darwin] => 9.5
    [Australia/Perth] => 8
    [Indian/Christmas] => 7
    [Pacific/Kosrae] => 11
    [Asia/Singapore] => 8
    [Pacific/Funafuti] => 12
    [Asia/Choibalsan] => 8
    [Asia/Brunei] => 8
    [Asia/Macau] => 8
    [Asia/Phnom_Penh] => 7
    [Asia/Hong_Kong] => 8
    [Africa/Dakar] => 0
    [Africa/Brazzaville] => 1
    [Australia/Broken_Hill] => 10.5
    [Europe/Lisbon] => 1
    [Atlantic/Madeira] => 1
    [Africa/Monrovia] => 0
    [Africa/Tunis] => 1
    [Africa/Algiers] => 1
    [Africa/Abidjan] => 0
    [Africa/Accra] => 0
    [Africa/Malabo] => 1
    [Africa/Douala] => 1
    [Pacific/Fiji] => 12
    [Europe/Madrid] => 2
    [Africa/Ceuta] => 2
    [Africa/Lagos] => 1
    [Africa/Ouagadougou] => 0
    [Africa/Lome] => 0
    [Africa/Bissau] => 0
    [Africa/Nouakchott] => 0
    [Africa/Porto-Novo] => 1
    [Africa/Libreville] => 1
    [Africa/Freetown] => 0
    [Africa/Sao_Tome] => 0
    [Africa/Banjul] => 0
    [Atlantic/Canary] => 1
    [Africa/Niamey] => 1
    [Africa/Bamako] => 0
    [Africa/El_Aaiun] => 1
    [Atlantic/St_Helena] => 0
    [Africa/Conakry] => 0
    [Europe/Gibraltar] => 2
    [Africa/Casablanca] => 1
    [Europe/Malta] => 2
    [Europe/Rome] => 2
    [Europe/Copenhagen] => 2
    [Atlantic/Faroe] => 1
    [Europe/Vienna] => 2
    [Europe/London] => 1
    [Atlantic/Reykjavik] => 0
    [Europe/Dublin] => 1
    [Europe/Zurich] => 2
    [Arctic/Longyearbyen] => 2
    [Europe/Amsterdam] => 2
    [Europe/Berlin] => 2
    [Europe/Brussels] => 2
    [Europe/Paris] => 2
    [Europe/Luxembourg] => 2
    [Europe/Monaco] => 2
    [Europe/Andorra] => 2
    [Europe/Vaduz] => 2
    [Europe/Jersey] => 1
    [Europe/Guernsey] => 1
    [Europe/Prague] => 2
    [Europe/Isle_of_Man] => 1
    [Europe/San_Marino] => 2
    [Europe/Ljubljana] => 2
    [Europe/Sarajevo] => 2
    [Europe/Zagreb] => 2
    [Atlantic/Azores] => 0
    [America/Barbados] => -4
    [America/Maceio] => -3
    [America/Fortaleza] => -3
    [America/Belem] => -3
    [America/Araguaina] => -3
    [America/Manaus] => -4
    [America/Recife] => -3
    [America/Godthab] => -2
    [Atlantic/Cape_Verde] => -1
    [America/Guyana] => -4
    [America/Cayenne] => -3
    [America/Paramaribo] => -3
    [America/Asuncion] => -3
    [America/Montevideo] => -3
    [America/Miquelon] => -2
    [Atlantic/South_Georgia] => -2
    [Atlantic/Stanley] => -3
    [America/Argentina/Buenos_Aires] => -3
    [America/Argentina/Tucuman] => -3
    [America/La_Paz] => -4
    [America/Sao_Paulo] => -3
    [America/Bahia] => -3
    [America/Campo_Grande] => -4
    [America/Cuiaba] => -4
    [America/Argentina/Cordoba] => -3
    [America/Martinique] => -4
    [America/Nassau] => -4
    [America/Port_of_Spain] => -4
    [America/St_Kitts] => -4
    [America/Dominica] => -4
    [America/Havana] => -4
    [America/Anguilla] => -4
    [America/Antigua] => -4
    [America/St_Lucia] => -4
    [Atlantic/Bermuda] => -3
    [America/St_Vincent] => -4
    [America/Montserrat] => -4
    [America/Guadeloupe] => -4
    [America/Grenada] => -4
    [America/Cayman] => -5
    [America/Belize] => -6
    [America/El_Salvador] => -6
    [America/Guatemala] => -6
    [America/Aruba] => -4
    [America/Grand_Turk] => -4
    [America/Santo_Domingo] => -4
    [America/Curacao] => -4
    [America/Lower_Princes] => -4
    [America/Mexico_City] => -5
    [America/Merida] => -5
    [America/Tegucigalpa] => -6
    [America/Managua] => -6
    [America/Monterrey] => -5
    [America/Caracas] => -4
    [America/Jamaica] => -5
    [America/Cancun] => -5
    [America/Costa_Rica] => -6
    [America/Tortola] => -4
    [America/Panama] => -5
    [America/Port-au-Prince] => -5
    [America/Lima] => -5
    [America/Bogota] => -5
    [America/Thule] => -3
    [America/Argentina/Jujuy] => -3
    [America/Argentina/Mendoza] => -3
    [America/Santiago] => -3
    [America/Argentina/Ushuaia] => -3
    [America/Rio_Branco] => -5
    [America/Porto_Velho] => -4
    [America/Argentina/San_Juan] => -3
    [America/Mazatlan] => -6
    [America/Tijuana] => -7
    [America/Guayaquil] => -5
    [America/Argentina/Rio_Gallegos] => -3
    [America/Hermosillo] => -7
    [Pacific/Galapagos] => -6
    [America/Boa_Vista] => -4
    [America/Argentina/Catamarca] => -3
    [America/Argentina/La_Rioja] => -3
    [America/Eirunepe] => -5
    [America/Chihuahua] => -6
    [Pacific/Fakaofo] => 13
    [Pacific/Tongatapu] => 13
    [Pacific/Tahiti] => -10
    [Pacific/Wallis] => 12
    [Pacific/Apia] => 14
    [Pacific/Pitcairn] => -8
    [America/Indiana/Knox] => -5
    [America/Chicago] => -5
    [America/New_York] => -4
    [America/Kentucky/Monticello] => -4
    [Pacific/Rarotonga] => -10
    [Pacific/Niue] => -11
    [Pacific/Guam] => 10
    [Pacific/Saipan] => 10
    [America/Indiana/Marengo] => -4
    [America/Indiana/Vevay] => -4
    [Pacific/Gambier] => -9
    [America/Indiana/Petersburg] => -4
    [Pacific/Easter] => -5
    [America/North_Dakota/New_Salem] => -5
    [America/Kentucky/Louisville] => -4
    [America/Detroit] => -4
    [America/Indiana/Indianapolis] => -4
    [America/Indiana/Vincennes] => -4
    [America/Indiana/Tell_City] => -5
    [America/Puerto_Rico] => -4
    [America/St_Thomas] => -4
    [America/Menominee] => -5
    [America/North_Dakota/Center] => -5
    [America/Los_Angeles] => -7
    [America/Phoenix] => -7
    [America/Denver] => -6
    [America/Shiprock] => -6
    [America/Boise] => -6
    [America/Juneau] => -8
    [America/Yakutat] => -8
    [America/Edmonton] => -6
    [America/Dawson_Creek] => -7
    [America/Regina] => -6
    [America/Swift_Current] => -6
    [America/Montreal] => -4
    [America/Vancouver] => -7
    [America/Resolute] => -5
    [America/Goose_Bay] => -3
    [America/Toronto] => -4
    [America/Moncton] => -3
    [America/Iqaluit] => -4
    [America/St_Johns] => -2.5
    [America/Winnipeg] => -5
    [America/Halifax] => -3
    [America/Nipigon] => -4
    [America/Glace_Bay] => -3
    [America/Rankin_Inlet] => -5
    [America/Whitehorse] => -7
    [America/Rainy_River] => -5
    [America/Anchorage] => -8
    [America/Nome] => -8
    [Pacific/Pago_Pago] => -11
    [America/Blanc-Sablon] => -4
    [America/Inuvik] => -6
    [America/Yellowknife] => -6
    [America/Pangnirtung] => -4
    [America/Cambridge_Bay] => -6
    [America/Adak] => -9
    [Pacific/Honolulu] => -10
    [America/Dawson] => -7
    [America/Thunder_Bay] => -4
    [Europe/Vatican] => 2
    [Antarctica/McMurdo] => 13