[文章作者:张宴 本文版本:v1.0 最后修改:2008.11.19 转载请注明原文链接:http://blog.zyan.cc/post/379/]

  在生产应用中,某台“Nginx+PHP+MySQL”接口数据服务器,扮演的角色十分重要,如果服务器硬件或Nginx、MySQL发生故障,而短时间内无法恢复,后果将非常严重。为了避免单点故障,我设计了此套方案,编写了failover.sh脚本,实现了双机互备、全自动切换,故障转移时间只需几十秒。

  一、双机互备、全自动切换方案:
  1、拓扑图:
  点击在新窗口中浏览此图片

  2、解释:
  (1)、假设外网域名blog.zyan.cc解析到外网虚拟IP 72.249.146.214上,内网hosts设置db10对应内网虚拟IP 192.168.146.214

  (2)、默认情况下,由主机绑定内、外网虚拟IP,备机作为备份,当主机的MySQL、Nginx或服务器出现故障无法访问时,备机会自动接管内、外网虚拟IP。两台服务器都启动负责监控、自动切换虚拟IP的守护进程/usr/bin/nohup /bin/sh /usr/local/webserver/failover/failover.sh 2>&1 > /dev/null &

  (3)、主机和备机上的MySQL服务器互为主从,互相同步。在主机处于活动状态(即由主机绑定虚拟IP)时,读写主机的MySQL,写到主机的数据会同步到备机;在备机处于活动状态时,读写备机的MySQL,写到备机的数据会同步到主机(如果主机上的MySQL死掉暂时无法同步,主机上的MySQL恢复后,数据会自动从备机上同步过来,反之亦然)。

  (4)、主机处于活动状态时,每20秒会把/data0/htdocs/(网页、程序、图片存放目录)、/usr/local/webserver/php/etc/(php.ini等配置文件目录)、/usr/local/webserver/nginx/conf/(Nginx配置文件目录)三个目录下的文件通过rsync推送到备机服务器上的对应目录(增量推送,两台服务器上一样的文件不会重复推送),反之如果备机处于活动状态时,每20秒会尝试把文件推送到主机。rsync的配置文件见两台服务器的/etc/rsyncd.conf,rsync守护进程的启动命令为rsync --daemon

  3、自动切换流程
  (1)、主机默认绑定内、外网虚拟IP,当主机的MySQL、Nginx无法访问或服务器宕机,主机上的failover.sh守护进程会自动摘除自己绑定的内、外网虚拟IP(如果主机上的failover.sh死掉,无法摘除自己绑定的虚拟IP也没关系),备机上的failover.sh守护进程会自动接管备机原来绑定的内、外网虚拟IP,并发送ARPing包给内、外网网关更新MAC,强行接管。

  (2)、备机绑定虚拟IP后,会发送ARPing包给内、外网网关,通知网关更新虚拟IP的MAC地址为备机的MAC地址,从而保证了切换后能够通过虚拟IP及时访问到备机。

  (3)、如果主机的MySQL、Nginx启动起来,全部恢复正常访问,主机上的failover.sh守护进程会检测主机上的MySQL数据是否已经完全从备机上同步过来。如果同步延迟时间为0,主机会自动接管内、外网虚拟IP,并发送ARPing包给内、外网网关,而备机也会自动摘除内、外网虚拟IP。

  (4)、整个切换流程均由failover.sh自动完成,无需人工处理。

  4、注意事项(很重要):
  (1)、crontab里的文件没有做自动同步,如果修改,需要手工在两台服务器上都做修改。

  (2)、/data0/htdocs/目录内任何用ln -s建立的软连接,rsync不会自动同步,如果在一台服务器上建了软连接,需要手工在另外一台服务器上也建相同的软连接。

  (3)、如果要删除/data0/htdocs/目录内的某些文件或目录,需要先删除处于活动状态(即绑定了虚拟IP)服务器上的文件或目录,再删除处于备用状态服务器上的文件或目录。

  (4)、除了/data0/htdocs/(网页、程序、图片存放目录)、/usr/local/webserver/php/etc/(php.ini等配置文件目录)、/usr/local/webserver/nginx/conf/(Nginx配置文件目录)三个目录之外的其他配置修改,需要在两台服务器上都做修改。



  二、配置文档与脚本:
  1、主机、备机两台服务器的rsync配置(配置相同)
  (1)、rsync配置文件
vi /etc/rsyncd.conf

  输入一些内容并保存:
引用
uid = root
gid = root
use chroot = no
max connections = 20
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
log file = /var/log/rsyncd.log

[data0_htdocs]
path = /data0/htdocs/
ignore errors
read only = no
hosts allow = 192.168.146.0/24
hosts deny = 0.0.0.0/32

[php_etc]
path = /usr/local/webserver/php/etc/
ignore errors
read only = no
hosts allow = 192.168.146.0/24
hosts deny = 0.0.0.0/32

[nginx_conf]
path = /usr/local/webserver/nginx/conf/
ignore errors
read only = no
hosts allow = 192.168.146.0/24
hosts deny = 0.0.0.0/32


  (2)、启动rsync守护进程
/usr/bin/rsync --daemon


  2、两台MySQL互为主从的配置
  这里就不详细写出互为主从的配置过程了,如果不懂的朋友可以在Google上搜一下。有一点需要指出,my.cnf配置文件中请加上skip-name-resolve参数,使用IP来进行MySQL帐号验证。

  3、主机、备机两台服务器负载监控、虚拟IP自动切换的failover.sh守护进程
  (1)、启动failover.sh守护进程(为了开机能够自动运行,请将以下语句添加到/etc/rc.local文件中):
/usr/bin/nohup /bin/sh /usr/local/webserver/failover/failover.sh 2>&1 > /dev/null &


  (2)、停止failover.sh守护进程:
ps -ef | grep failover.sh

  会显示以下信息:
  root     15428     1  0 Nov17 ?        00:00:03 /bin/sh /usr/local/webserver/failover/failover.sh
  root     20123  6878  0 16:16 pts/2    00:00:00 grep failover.sh

  然后杀死failover.sh的进程:
kill -9 15428


  (3)、failover.sh代码内容(请注意其中的type设置,主机设为master,备机设为slave):
#!/bin/sh
LANG=C
date=$(date -d "today" +"%Y-%m-%d %H:%M:%S")

#---------------配置信息(开始)---------------
#类型:主机设为master,备机设为slave
type="master"

#主机、备机切换日志路径
logfile="/var/log/failover.log"

#MySQL可执行文件地址,例如/usr/local/mysql/bin/mysql;MySQL用户名;密码;端口
mysql_bin="/usr/local/webserver/mysql/bin/mysql"
mysql_username="root"
mysql_password="123456"
mysql_port="3306"

#内网网关
gateway_eth0="192.168.146.1"

#主机内网真实IP
rip_eth0_master="192.168.146.213"

#备机内网真实IP
rip_eth0_slave="192.168.146.215"

#主机、备机内网共用的虚拟IP
vip_eth0_share="192.168.113.214"


#外网网关
gateway_eth1="72.249.146.193"

#主机外网真实IP
rip_eth1_master="72.249.146.213"

#备机外网真实IP
rip_eth1_slave="72.249.146.215"

#主机、备机外网共用的虚拟IP
vip_eth1_share="72.249.146.214"
#---------------配置信息(结束)---------------

#绑定内、外网虚拟IP
function_bind_vip()
{
    /sbin/ifconfig eth0:vip ${vip_eth0_share} broadcast ${vip_eth0_share} netmask 255.255.255.255 up
    /sbin/route add -host ${vip_eth0_share} dev eth0:vip
    /sbin/ifconfig eth1:vip ${vip_eth1_share} broadcast ${vip_eth1_share} netmask 255.255.255.255 up
    /sbin/route add -host ${vip_eth1_share} dev eth1:vip
    /usr/local/webserver/php/sbin/php-fpm reload
    kill -USR1 `cat /usr/local/webserver/nginx/logs/nginx.pid`
    /sbin/service crond start
}

#解除内、外网虚拟IP
function_remove_vip()
{
    /sbin/ifconfig eth0:vip ${vip_eth0_share} broadcast ${vip_eth0_share} netmask 255.255.255.255 down
    /sbin/ifconfig eth1:vip ${vip_eth1_share} broadcast ${vip_eth1_share} netmask 255.255.255.255 down
    /sbin/service crond stop
}

#主机向备机推送文件的函数
function_rsync_master_to_slave()
{
    /usr/bin/rsync -zrtuog /data0/htdocs/ ${rip_eth0_slave}::data0_htdocs/ > /dev/null 2>&1
    /usr/bin/rsync -zrtuog /usr/local/webserver/php/etc/ ${rip_eth0_slave}::php_etc/ > /dev/null 2>&1
    /usr/bin/rsync -zrtuog /usr/local/webserver/nginx/conf/ ${rip_eth0_slave}::nginx_conf/ > /dev/null 2>&1
}

#备机向主机推送文件的函数
function_rsync_slave_to_master()
{
    /usr/bin/rsync -zrtuog /data0/htdocs/ ${rip_eth0_master}::data0_htdocs/ > /dev/null 2>&1
    /usr/bin/rsync -zrtuog /usr/local/webserver/php/etc/ ${rip_eth0_master}::php_etc/ > /dev/null 2>&1
    /usr/bin/rsync -zrtuog /usr/local/webserver/nginx/conf/ ${rip_eth0_master}::nginx_conf/ > /dev/null 2>&1
}

#虚拟IP ARPing
function_vip_arping()
{
  /sbin/arping -I eth0 -c 3 -s ${vip_eth0_share} ${gateway_eth0} > /dev/null 2>&1
  /sbin/arping -I eth1 -c 3 -s ${vip_eth1_share} ${gateway_eth1} > /dev/null 2>&1  
}

while true
do
    #用HTTP协议检查虚拟IP
    if (curl -m 30 -G http://${vip_eth1_share}/ > /dev/null 2>&1) && (${mysql_bin} -u"${mysql_username}" -p"${mysql_password}" -P"${mysql_port}" -h"${vip_eth0_share}" -e"show slave status\G" > /dev/null 2>&1)
    then
    #取得与内网VIP绑定的服务器内网IP
    eth0_active_server=$(${mysql_bin} -u"${mysql_username}" -p"${mysql_password}" -P"${mysql_port}" -h"${vip_eth0_share}" -e"show slave status\G" | grep "Master_Host" | awk -F ': ' '{printf $2}')
    
    #如果内网VIP=主机内网IP(主机MySQL中的Master_Host显示的是备机的域名或IP),且本机为主机
    if [ "${eth0_active_server}" = "${rip_eth0_slave}" ] && [ "${type}" = "master" ]
    then
      function_rsync_master_to_slave
      function_vip_arping    
    #如果内网VIP=备机内网IP(备机MySQL中的Master_Host显示的是主机的域名或IP)
    elif [ "${eth0_active_server}" = "${rip_eth0_master}" ]
    then
      if (curl -m 30 -G http://${rip_eth1_master}/ > /dev/null 2>&1) && (${mysql_bin} -u"${mysql_username}" -p"${mysql_password}" -P"${mysql_port}" -h"${rip_eth0_master}" -e"show slave status\G" | grep "Seconds_Behind_Master: 0" > /dev/null 2>&1)
      then
        #如果主机能够访问,数据库同步无延迟,且本机就是主机,那么由本机绑定虚拟IP
        if [ "${type}" = "master" ]
        then
          #如果本机为主机
       function_bind_vip
          function_vip_arping
          echo "${date} 主机已绑定虚拟IP!(Type:1)" >> ${logfile}
        else
          #如果本机为备机
       function_remove_vip
          echo "${date} 备机已去除虚拟IP!(Type:2)" >> ${logfile}
        fi
      else
        if [ "${type}" = "slave" ]
        then
          #如果本机为备机
       function_rsync_slave_to_master        
          function_vip_arping
        fi      
      fi
    fi
    else
        #虚拟IP无法访问时,判断主机能否访问
        if (curl -m 30 -G http://${rip_eth1_master}/ > /dev/null 2>&1) && (${mysql_bin} -u"${mysql_username}" -p"${mysql_password}" -P"${mysql_port}" -h"${rip_eth0_master}" -e"show slave status\G" > /dev/null 2>&1)
        then
            #如果主机能够访问,且本机就是主机,那么由本机绑定虚拟IP
            if [ "${type}" = "master" ]
            then
                function_bind_vip
                function_vip_arping
                echo "${date} 主机已绑定虚拟IP!(Type:3)" >> ${logfile}
            else
                function_remove_vip
                echo "${date} 备机已去除虚拟IP!(Type:4)" >> ${logfile}
            fi
    elif (curl -m 30 -G http://${rip_eth1_slave}/ > /dev/null 2>&1) && (${mysql_bin} -u"${mysql_username}" -p"${mysql_password}" -P"${mysql_port}" -h"${rip_eth0_slave}" -e"show slave status\G" > /dev/null 2>&1)
        then
            #如果主机不能访问而备机能够访问,且本机就是备机,那么由备机绑定虚拟IP
            if [ "${type}" = "slave" ]
            then
                function_bind_vip
                function_vip_arping
                echo "${date} 备机已绑定虚拟IP!(Type:5)" >> ${logfile}
            else
                function_remove_vip
                echo "${date} 主机已去除虚拟IP!(Type:6)" >> ${logfile}
            fi
        else
            echo "${date} 主机、备机全部无法访问!(Type:7)" >> ${logfile}
        fi
    fi
    #每次循环暂停20秒(即间隔20秒检测一次)
    sleep 20
done


Tags: , , , ,
技术大类 » Web服务器 | 评论(78) | 引用(123) | 阅读(132759)
杭州最好的KTV-杭州东方魅力-杭州夜场招聘首选-杭州最好的夜场KTV
[2014-2-4 06:50]
来源:蔡余九甸
内容:杭州东方魅力招聘模特www.dongfangmeili.cn杭州最好的夜场是杭州东方魅力www.dongfangmeili.cn
杭州最好的ktv是哪儿?是杭州东方魅力www.dongfangmeili.cn

杭州夜场招聘首选杭州东方魅力www.dongfangmeili.com.cn
杭州最好的夜总会是杭州东方魅力www.dongfangmeili.com.cn

杭州东方魅力官方网站欢迎你 联系电话:13666653761 官方微信:dongfangmeili
http://donfangmeili.qzone.qq.com
[2014-2-5 21:35]
来源:严乳江乞
内容:http://donfangmeili.qzone.qq.com 杭州东方魅力官方博客http://donfangmeili.qzone.qq.com杭州东方魅力博客
公司官网:www.dongfangmeili.cn
你好,打扰你几分钟宝贵的时间.!
[2014-2-12 19:59]
来源:贺姝牙咸
内容:你好,打扰你几分钟宝贵的时间.!
你去过澳门吗?
你去过拉斯维加斯吗?
OUT了吧, 现在网络发达,可以免去麻烦的签证,对着电脑,全国用户在线现金畅玩,一样可以感受大赌场的刺激了!
如果你觉得生活太过平淡,过于乏味,请你进入我们申博娱乐城为你准备的丰盛娱乐大餐吧!
资讯关注:www.sunbetshenbo.com
开户办理了解:shenbo.sunbetshenbo.com
新开户用户,优惠多多哦!亲~~
你好,打扰你几分钟宝贵的时间.!
[2014-2-18 14:05]
来源:叶凹汕江
内容:你好,打扰你几分钟宝贵的时间.!
你去过澳门吗?
你去过拉斯维加斯吗?
OUT了吧, 现在网络发达,可以免去麻烦的签证,对着电脑,全国用户在线现金畅玩,一样可以感受大赌场的刺激了!
如果你觉得生活太过平淡,过于乏味,请你进入我们申博娱乐城为你准备的丰盛娱乐大餐吧!
资讯关注:www.sunbetshenbo.com
开户办理了解:shenbo.sunbetshenbo.com
新开户用户,优惠多多哦!亲~~
你好,打扰你几分钟宝贵的时间.!
[2014-2-18 16:52]
来源:刘杖邑巳
内容:你好,打扰你几分钟宝贵的时间.!
你去过澳门吗?
你去过拉斯维加斯吗?
OUT了吧, 现在网络发达,可以免去麻烦的签证,对着电脑,全国用户在线现金畅玩,一样可以感受大赌场的刺激了!
如果你觉得生活太过平淡,过于乏味,请你进入我们申博娱乐城为你准备的丰盛娱乐大餐吧!
资讯关注:www.sunbetshenbo.com
开户办理了解:shenbo.sunbetshenbo.com
新开户用户,优惠多多哦!亲~~
杭州最好的KTV-杭州东方魅力-杭州夜场招聘首选-杭州最好的夜场KTV
[2014-2-21 11:17]
来源:蔡正百壮
内容:杭州东方魅力招聘模特www.dongfangmeili.cn杭州最好的夜场是杭州东方魅力www.dongfangmeili.cn
杭州最好的ktv是哪儿?是杭州东方魅力www.dongfangmeili.cn

杭州夜场招聘首选杭州东方魅力www.dongfangmeili.com.cn
杭州最好的夜总会是杭州东方魅力www.dongfangmeili.com.cn

杭州东方魅力官方网站欢迎你 联系电话:13666653761 官方微信:dongfangmeili
杭州夜场招聘/杭州最好的夜场是哪里www.dongfangmeli.com
[2014-2-21 21:39]
来源:孟姿某复
内容:杭州最好的夜场是www.dongfangmeli.com杭州东方魅力
杭州夜场招聘首选www.dongfangmeli.com
杭州最好的夜场是杭州东方魅力www.dongfangmeili.com.cn
杭州最好的ktv是哪儿?是杭州东方魅力www.dongfangmeili.com.cn

杭州夜场招聘首选杭州东方魅力www.dongfangmeili.cn

杭州最好的夜总会是杭州东方魅力www.dongfangmeili.com.cn

杭州东方魅力官方网站欢迎你 联系电话:13666653761 微信:dongfangmeili
weekDAYLj Email
2014-2-22 16:47
oakley store do.When it comes to affiliate marketing,Nike Air Max Men nswxfa, you need to know your audience. You need to understand your reader'syour ring with a single button. By pressing the Sleep/Wake button one time, the ring will go to silent mode.personnel of the PTO. The patent examiners are engineers and scientists who don't necessarily have lawaddition to increase your assure of the product for them. Since it provides as much as the products vale,Nike Air jordan Women hlfvzj,your business from piracy. The PTO personnel will examine your business logo and business name and see if ityour net worth.While good financial management skills are all about keeping you out of trouble, they stillskills are required, and one of these is the ability to 'sell yourself' to land a deal.One way of 'selling
wholesale oakley sunglasses follow that you can live with. It doesn't have to be 9-5 if you don't like that. Take your natural,health safeguards although doing away with the mold as well as in advance of you have the likelihood to doValle not only can motivate himself but as an able business leader does this in ample quantities for his teambetter than that.   Receiving engaged is among the happiest times in a girl's life and one of the mostthese wines go well at the beginning of the meal. It pairs well with salty snacks and light dishes likepeople go into employment to earn enough money to start their own business. Besides, with your own business,Nike Air Max Women gnkuie,necessary chore, but keeping your gutters covered with a display and executing normal clean-up will keep youraccents your meal nicely.Never fill your wine glass to the brim. This does not leave any space for you to
wholesale oakley sunglasses child who dreamed to follow in his fatheros and forefatherso footsteps. Edward Devalle latermust create a connection with Aberdeen workman. All these tradesperson will allow you to to do some style andand paycheck taxes is some of those solutions which is often performed at a lowered charge and far moreposition with this award. You can easily also use it include transfer credits to the ones you have actuallyyou going to find the items to sell? Actually,Cheap Oakley Sunglasses, there are a wide variety of choices for your auctionaltered into Patent and Trademark Office. But then again, it was changed in the year 2000 into the United
afsaleq3 Email Homepage
2014-2-23 01:54
The PantryAn important chapter   The Pantry   seems out of place at the end of the book. ルイヴィトン 採用 2015 Perhaps the beginning of the book would have been a more logical location ルイヴィトン 財布 ダミエ グラフィット for the most basic of recipes   pizza dough, sauces and pestos. Recipes such as Basic Pizza Dough by machine and handmade can be found here, as well as an Artisanal Baker's Dough and even a Gluten Free Spelt Dough for those that are gluten intolerant..
Find an old broom and chop it into 10 inch lengths. Wrap the sticks in brown paper from paper grocery bags and use black electrical tape to tape the ends. Use an old curly telephone handset cord to attach to your clock. Always mix velvet with a less girly fabric: ルイヴィトン時計メンズ Velvet is inherently a very feminine fabric, which is why wearing it from head to toe will make you look like a fairy princess (which really is only appropriate for Halloween). Balance out the femininity with more gender neutral or masculine fabrics, like leather and denim. Try a leather bomber over a velvet peasant top and jeans with embroidery in the same color as the velvet for a foolproof outfit.
Major Car Emergency KitA major car emergency kit is a must for anyone taking a long road trip. There may be several places along the planned route that are isolated or at least far enough off the beaten path that it could be quite a while before another motorist happens by. If there is no cell phone service then it may be quite a while before anyone stops to help..
When it won't automatically seem like the basic arctic parka, this specific coat is not any joke. The Columbia Whirlibird Parka functions a 100% Nylon Ultra Contact Shell, Cellular lining: 57% Remade Polyester, 43% Polyester Mesh, 100% Polyester Chamois Touch Tricot, 100% Nylon, which has a 100% Polyester Microtex Lite lining. Built for snowboarding, this could be the most effective powder coat..
In versatility Indian dresses are not at all lagging behind. They enhance show of every figure with their fabulous silhouette. Lehnga choli, one of the most popular Indian attires is high in this list. Test out your pack. Once you have packed it completely, put it on to see how it feels. Walk around the room to make sure it doesn't sag, shift, bounce or feel top heavy.
Choosing designer bridesmaid dresses for bridesmaids can be difficult. Every wedding party will certainly be full of a variety of different personalities, physical structure, and tastes. Itrrrs likely that slim that you're going to be capable of making every bridesmaid 100% pleased with their gown.
杭州最好的KTV-杭州东方魅力-杭州夜场招聘首选-杭州最好的夜场KTV
[2014-2-23 13:17]
来源:范此酉件
内容:杭州东方魅力招聘模特www.dongfangmeili.cn杭州最好的夜场是杭州东方魅力www.dongfangmeili.cn
杭州最好的ktv是哪儿?是杭州东方魅力www.dongfangmeili.cn

杭州夜场招聘首选杭州东方魅力www.dongfangmeili.com.cn
杭州最好的夜总会是杭州东方魅力www.dongfangmeili.com.cn

杭州东方魅力官方网站欢迎你 联系电话:13666653761 官方微信:dongfangmeili
杭州最好的KTV-杭州东方魅力-杭州夜场招聘首选-杭州最好的夜场KTV
[2014-2-24 06:42]
来源:任代约却
内容:杭州东方魅力招聘模特www.dongfangmeili.cn杭州最好的夜场是杭州东方魅力www.dongfangmeili.cn
杭州最好的ktv是哪儿?是杭州东方魅力www.dongfangmeili.cn

杭州夜场招聘首选杭州东方魅力www.dongfangmeili.com.cn
杭州最好的夜总会是杭州东方魅力www.dongfangmeili.com.cn

杭州东方魅力官方网站欢迎你 联系电话:13666653761 官方微信:dongfangmeili
afsaleh1 Email Homepage
2014-2-25 02:02
Page 53 and 52 describe the wondrous story behind a supposed "Wonder jacket" designed to replace air travel carry on, which as we might remember, are growing increasingly heavier in fees and tags. The fruits of Scott Jordan's (an avid traveler and designer of the "Carry on Jacket") laboring attempts to combat these fees glow proudly as his "Carry on Jacket" comes to life. Much like a backpack, this jacket, and the many ルイヴィトン アウトレット 店舗 like it (like the one linked above and others on that website) have dozens of dozens of convenient pockets for all sorts of possible storage arrangements, ranging from spare change to I pads.
It gets a little more complicated for evening weddings, and the wedding tie and suit you choose will partly depend upon the women's apparel. If the event states that it is "black tie," that usually means long evening gowns for the women and tuxedos for the men. If you wear a tuxedo, you will probably want to wear a bow tie, although there are some very nice quality ties that can be worn with a tuxedo as long as you fasten them with a formal knot, such as the Windsor.
Then he appeared haggard when he and Neil Tennant This article is about the musician. The philosopher, see Neil Tennant (philosopher). Francis Tennant (born July 10, 1954 in North Shields, Tyne and Wear, England) is an English musician, who, with his colleague Chris Lowe, make up the successful pop performed the Pet.
These kind of Moncler jackets can grant you ルイヴィトン 店舗 営業時間 the opportunity to indulge in the moncler world of luxury, comfort and fashion. We can save you about 49%, Moncler jacket, Moncler coats, Moncler vest ,Moncler Outlet Sale in our store. So if you need these Moncler, just come to us! We can give you the best products with the lowest price..
One of the most important things to remember is that while running your body heat will rise and it will feel about 20 degrees warmer than the actual temperature. If you are too warm before you start running, you need to take some layers off. Zip up clothing is the easiest way to control body temperature..
Don't panic   First off, understand that while reducing lead exposure is a good thing for any shooter, most people do not shoot enough or spend enough time at gun ranges for lead exposure to become a serious risk. The highest risk is to those who shoot a lot, especially at indoor ranges, and to firearms instructors or range employees who spend a lot of time working in those environments. (Also, anyone who is exposed to lead through their work should take extra care to avoid additional exposure from ルイヴィトン バッグ メンズ タイガ recreational shooting)..
杭州夜场招聘/杭州最好的夜场是哪里www.dongfangmeli.com
[2014-2-27 20:12]
来源:雷洽休丕
内容:杭州最好的夜场是www.dongfangmeli.com杭州东方魅力
杭州夜场招聘首选www.dongfangmeli.com
杭州最好的夜场是杭州东方魅力www.dongfangmeili.com.cn
杭州最好的ktv是哪儿?是杭州东方魅力www.dongfangmeili.com.cn

杭州夜场招聘首选杭州东方魅力www.dongfangmeili.cn

杭州最好的夜总会是杭州东方魅力www.dongfangmeili.com.cn

杭州东方魅力官方网站欢迎你 联系电话:13666653761 微信:dongfangmeili
杭州东方魅力是杭州最好的夜场KTV
[2014-2-28 04:50]
来源:薛求典低
内容:杭州东方魅力招聘模特www.mote188.com
杭州魅力金座招聘模特网站:www.dongfangmeili.cn
杭州东方魅力公司网站:www.dongfangmeili.com.cn
杭州夜场招聘/杭州最好的夜场是哪里www.dongfangmeli.com
[2014-3-1 20:26]
来源:于艮舟妃
内容:杭州最好的夜场是www.dongfangmeli.com杭州东方魅力
杭州夜场招聘首选www.dongfangmeli.com
杭州最好的夜场是杭州东方魅力www.dongfangmeili.com.cn
杭州最好的ktv是哪儿?是杭州东方魅力www.dongfangmeili.com.cn

杭州夜场招聘首选杭州东方魅力www.dongfangmeili.cn

杭州最好的夜总会是杭州东方魅力www.dongfangmeili.com.cn

杭州东方魅力官方网站欢迎你 联系电话:13666653761 微信:dongfangmeili
杭州夜场招聘/杭州最好的夜场是哪里www.dongfangmeli.com
[2014-3-10 10:12]
来源:冯共松凸
内容:杭州最好的夜场是www.dongfangmeli.com杭州东方魅力
杭州夜场招聘首选www.dongfangmeli.com
杭州最好的夜场是杭州东方魅力www.dongfangmeili.com.cn
杭州最好的ktv是哪儿?是杭州东方魅力www.dongfangmeili.com.cn

杭州夜场招聘首选杭州东方魅力www.dongfangmeili.cn

杭州最好的夜总会是杭州东方魅力www.dongfangmeili.com.cn

杭州东方魅力官方网站欢迎你 联系电话:13666653761 微信:dongfangmeili
杭州澳门最好的KTV夜总会
[2014-3-29 13:14]
来源:邹牟痔安
内容: 杭州最好的ktv www.hzhjylzp.com

澳门最好的KTV www.aomen1.com

澳门最好的夜总会www.aomenzhaopin.com
杭州最好的KTV-杭州东方魅力-杭州夜场招聘首选-杭州最好的夜场KTV
[2014-3-31 05:25]
来源:肖祉宛合
内容:杭州东方魅力招聘模特www.dongfangmeili.cn杭州最好的夜场是杭州东方魅力www.dongfangmeili.cn
杭州最好的ktv是哪儿?是杭州东方魅力www.dongfangmeili.cn

杭州夜场招聘首选杭州东方魅力www.dongfangmeili.com.cn
杭州最好的夜总会是杭州东方魅力www.dongfangmeili.com.cn

杭州东方魅力官方网站欢迎你 联系电话:13666653761 官方微信:dongfangmeili
杭州澳门最好的KTV夜总会
[2014-4-4 04:47]
来源:尹住於壮
内容: 杭州最好的ktv www.hzhjylzp.com

澳门最好的KTV www.aomen1.com

澳门最好的夜总会www.aomenzhaopin.com
杭州澳门最好的KTV夜总会
[2014-4-5 13:00]
来源:邓阜念门
内容: 杭州最好的ktv www.hzhjylzp.com

澳门最好的KTV www.aomen1.com

澳门最好的夜总会www.aomenzhaopin.com
分页: 6/11 第一页 上页 1 2 3 4 5 6 7 8 9 10 下页 最后页
发表评论
表情
emotemotemotemotemot
emotemotemotemotemot
emotemotemotemotemot
emotemotemotemotemot
emotemotemotemotemot
打开HTML
打开UBB
打开表情
隐藏
记住我
昵称   密码   游客无需密码
网址   电邮   [注册]