[文章作者:张宴 本文版本:v1.0 最后修改:2008.12.09 转载请注明原文链接:http://blog.zyan.cc/post/385/]

  曾经在七月,写过一篇文章──《基于Sphinx+MySQL的千万级数据全文检索(搜索引擎)架构设计》,前公司的分类信息搜索基于此架构,效果明显,甚至将很大一部分带Where条件的MySQL SQL查询,都改用了Sphinx+MySQL搜索。但是,这套架构仍存在局限:一是MySQL本身的并发能力有限,在200~300个并发连接下,查询和更新就比较慢了;二是由于MySQL表的主键与Sphinx索引的ID一一对应,从而无法跨多表建立整站查询,而且新增加类别还得修改配置文件,比较麻烦;三是因为和MySQL集成,无法发挥出Sphinx的优势。

  最近,我设计出了下列这套最新的搜索引擎架构,目前已经写出“搜索查询接口”和“索引更新接口”的beta版。经测试,在一台“奔腾四 3.6GHz 双核CPU、2GB内存”的普通PC机,7000万条索引记录的条件下,“搜索查询接口”平均查询速度为0.0XX秒(查询速度已经达到百度、谷歌、搜狗、中国雅虎等搜索引擎的水平,详见文章末尾的“附2”),并且能够支撑高达5000的并发连接;而“索引更新接口”进行数据分析、入队列、返回信息给用户的全过程,高达1500 Requests/Sec。

  “队列控制器”这一部分是核心,它要控制队列读取,更新MySQL主表与增量表,更新搜索引擎数据存储层Tokyo Tyrant,准实时(1分钟内)完成更新Sphinx增量索引,定期合并Sphinx索引。我预计在这周写出beta版。

点击在新窗口中浏览此图片

  图示说明:
  1、搜索查询接口:
  ①、Web应用服务器通过HTTP POST/GET方式,将搜索关键字等条件,传递给搜索引擎服务器的search.php接口;
  ②③、search.php通过Sphinx的API(我根据最新的Sphinx 0.9.9-rc1 API,改写了一个C语言的PHP扩展sphinx.so),查询Sphinx索引服务,取得满足查询条件的搜索引擎唯一ID(15位搜索唯一ID:前5位类别ID+后10位原数据表主键ID)列表;
  ④⑤、search.php将这些ID号作为key,通过Memcache协议一次性从Tokyo Tyrant中mget取回ID号对应的文本数据。
  ⑥⑦、search.php将搜索结果集,按查询条件,进行摘要和关键字高亮显示处理,以JSON格式或XML格式返回给Web应用服务器。

  2、索引更新接口:
  ⑴、Web应用服务器通过HTTP POST/GET方式,将要增加、删除、更新的内容告知搜索服务器的update.php接口;
  ⑵、update.php将接收到的信息处理后,写入TT高速队列(我基于Tokyo Tyrant做的一个队列系统);
  注:这两步的速度可达到1500次请求/秒以上,可应对6000万PV的搜索索引更新调用。

  3、搜索索引与数据存储控制:
  ㈠、“队列控制器”守护进程从TT高速队列中循环读取信息(每次50条,直到末尾);
  ㈡、“队列控制器”将读取出的信息写入搜索引擎数据存储层Tokyo Tyrant;
  ㈢、“队列控制器”将读取出的信息异步写入MySQL主表(这张主表按500万条记录进行分区,仅作为数据永久性备份用);
  ㈣、“队列控制器”将读取出的信息写入MySQL增量表;
  ㈤、“队列控制器”在1分钟内,触发Sphinx更新增量索引,Sphinx的indexer会将MySQL增量表作为数据源,建立增量索引。Sphinx的增量索引和作为数据源的MySQL增量表成对应关系;
  ㈥、“队列控制器”每间隔3小时,短暂停止从TT高速队列中读取信息,并触发Sphinx将增量索引合并入主索引(这个过程非常快),同时清空MySQL增量表(保证了MySQL增量表的记录数始终只有几千条至几十万条,大大加快Sphinx增量索引更新速度),然后恢复从TT高速队列中取出数据,写入MySQL增量表。

  本架构使用的开源软件:
  1、Sphinx 0.9.9-rc1
  2、Tokyo Tyrant 1.1.9
  3、MySQL 5.1.30
  4、Nginx 0.7.22
  5、PHP 5.2.6

  本架构自主研发的程序:
  1、搜索查询接口(search.php)
  2、索引更新接口(update.php)
  3、队列控制器
  4、Sphinx 0.9.9-rc1 API的PHP扩展(sphinx.so)
  5、基于Tokyo Tyrant的高速队列系统



  附1:MySQL FullText、Lucene搜索、Sphinx搜索的第三方对比结果:
  1、查询速度:
  MySQL FullText最慢,Lucene、Sphinx查询速度不相上下,Sphinx稍占优势。
  点击在新窗口中浏览此图片

  2、建索引速度:
  Sphinx建索引速度是最快的,比Lucene快9倍以上。因此,Sphinx非常适合做准实时搜索引擎。

  3、详细对比数据见以下PDF文档:  



  附2:国内各大中文搜索引擎搜索速度分析:
  以“APMServ张宴”为关键字,比较在各大中文搜索引擎的搜索速度:
  1、百度:
  ①、第一次搜索
  点击在新窗口中浏览此图片

  ②、第二次搜索
  点击在新窗口中浏览此图片

  分析:百度对第一次搜索的搜索结果做了Cache,所以第二次查询非常快。



  2、谷歌:
  ①、第一次搜索
  点击在新窗口中浏览此图片

  ②、第二次搜索
  点击在新窗口中浏览此图片

  分析:谷歌也对第一次搜索的搜索结果做了Cache,但两次查询跟百度同比,都要慢一些。



  3、搜狗:
  ①、第一次搜索
  点击在新窗口中浏览此图片

  ②、第二次搜索
  点击在新窗口中浏览此图片

  ③、第三次搜索
  点击在新窗口中浏览此图片

  分析:搜狗疑似对第一次搜索的搜索结果做了短暂的Cache,第二次搜索速度非常快,第三次搜索的速度比第二次搜索的速度慢。搜狗第一次搜索的速度跟百度差不多。



  4、中国雅虎:
  ①、第一次搜索
  点击在新窗口中浏览此图片

  ②、第二次搜索
  点击在新窗口中浏览此图片

  分析:搜索结果没有做Cache。中国雅虎的搜索速度跟百度第一次搜索的速度差不多。



  5、网易有道:
  ①、第一次搜索
  点击在新窗口中浏览此图片

  ②、第二次搜索
  点击在新窗口中浏览此图片

  分析:有道对第一次搜索的搜索结果做了Cache。但是,跟谷歌一样,两次搜索同比都要较百度、搜狗、中国雅虎慢一些。



技术大类 » 搜索引擎技术 | 评论(87) | 引用(1) | 阅读(133943)
过客
2012-4-18 11:10
文章好读,歌曲好听
Louis Vuitton Neverfull Email Homepage
2012-4-21 17:34
www.lvbagsclassic.com are authorized authentic Louis Vuitton handbags outlet store. All the items at our site are 100% authentic. All our Louis Vuitton handbags will come with the authenticity card, serial Number, dust bag and care booklet. We promise you will be 100% satisfied when you get such cheap authentic Louis Vuitton handbags from us!
Louis Vuitton Neverfull
Hogan
2012-5-10 18:10
Mentre youre gestione di problemi complessi tra cui PAYE insieme con le imprese del prelievo, l'idea è sicuramente intelligente per acquisire specializzata fare qualcosa di sicuro si sta usando la vostra organizzazione. Self-employmentSelf lavoratori uomini e donne potrebbe anche fare pieno uso Hogan dei ragionieri in tutta Stansted, quando hanno l'effetto della loro prelievo unico e sì, è spesso sconcertante. Stare di fronte a una buona varietà prelievo può essere schiacciante, di conseguenza, il
coach factory outlet Email Homepage
2012-5-17 11:52
Today, following half a century, mentor leather-bases coach factory outlet continues to be the delicate craft of leather-based master is accountable for,Would you like to meet more friends, or go with the times? If yes, coach factory online is opening welcome doors to you.in the market you definitely can find various colorways that are designed in as well as the high quality that applied in. For most of you would like to come. So just come to our coach factory outlet online store to choose one.
louis vuitton sale Email Homepage
2012-5-17 11:52
Louis Vuitton belt at louis vuitton sale is one kind of fashion accessory with high cost performance among the Louis Vuitton accessories.Offering quality LV products with favorable prices, louis vuitton outlet store is at your service. Hurry up, or you can not seize the chance.in fact, louis vuitton is one of the most famous fashion design master.he opened the fist suitcase shop called after his name.
coach outlet online Email Homepage
2012-5-17 11:52
I heard of coach outlet online through the advertisement when I was shopping. And now I often brow the webpage and buy Coach bags online.It is a symbol regarding position not to mention nature.Here I would like to launch a excellent bags pertaining to business men.Which may be coach outlet store.coach outlet has always been simple,durable style features to win consumers.The products are more flexible,with easy bleaching,wear characteristics,and simply use a damp cloth.
coach outlet, Email Homepage
2012-5-17 11:53
Remember the coach outlet provide coach bags which won't be deteriorated into its overall styles by any means. It will maintain its looks, colors, and uniqueness for long time.coach outlet store online has been voted by Hour Detroit magazine readers as the Best of Detroit in their 12th annual readers'poll.Lots of women like which usually amount normally include a coach outlet online ,it provides coziness to many girls that don't even think it is a great bushel of great interest directly to them.
xujie777 Email
2012-5-18 16:52
We aim to make all our customers satisfy with our products. You will find a variety of Men's fashion louis vuitton uk, fashion Women's cheap Louis Vuitton bags in our store at affordable price.Thinking of interesting ways to cost a milestone birthday? louis vuitton online shop had one of the most distinctive distinctive celebrations.bakery along with living room operated by means of about three moment louis vuitton online Most effective Pastry Chef’s of the year Rammy Nominee Chef’s.
xujie777 Email
2012-5-18 16:52
The choices are likely to be basically countless seeing that louis vuitton outlet occurs with the help of completely new and also incredible concepts once in a while.Louis vuitton Wholesale Monogram Canvas HandbagsLouis Vuitton Collection Beach Handbags louis vuitton bags outlet Damier Canvas HandbagsLouis vuitton Mahina HandbagsLouis Vuitton Monogram Mini Lin HandbagsLouis Vuitton Monogram Multicolore HandbagsLouis vuitton Monogram Vernis HandbagsLouis Vuitton Wholesale Epi Leather HandbagsLouis Vuitton For Men HandbagsLouis Vuitton Damier Canvas WalletsLouis Vuitton Epi Leather WalletsLouis Vuitton Monogram Canvas WalletsLouis Vuitton Monogram Vernis WalleLouis Vuitton ShoesLouis Vuitton Men wallets.As the Authentic Louis Vuitton are so high-priced, so came the louis vuitton handbags outlet.
xujie777 Email
2012-5-18 16:52
coach outlet has become a popular shopping experience for consumers around the world, and a desirable distribution channel for manufacturer's and retailers.Getting your hands on coach outlet store online can be a hefty investment of hundreds of dollars.But do not despair,the Coach Outlet store could be the answer to your prayers.coach outlet online is your smart choice when you want to get the discount Coach accessory. You can find the exact Coach Bags and other accessory you want at a low price that's right for you. My dear friends, let the coach outlet online pave your way into the world of high fashion with their ultimate fashion factory.
潇潇易水
2012-5-19 19:22
张老师,能开源大家学习一下吗,谢谢
xujie123
2012-5-21 11:27
Our online store offers you discounted Designer louis vuitton replica wallet at present. You could find them in desirable quality and price. If you don't mind high class louis vuitton uk, have a good time here.louis vuitton Store Online Handbags can also bring great accuracy as well as practical applicability and fashionable.Have you ever dreamed of being as charming as Madonna? Have you ever thought of becoming an envy of all your friends? If so, come to louis vuitton outlet.<br/>
xujie555
2012-5-21 13:53
coach factory outlet uses graceful accessories to match the classical logo of coach, which is the best combination of coach. The handmade coach products?can make you more charming and graceful.coach factory online provides people many coach goods. If you wish to snatch the coach handbag, then this best method is made for that you like for coach discount.coach factory outlet online is a fashion brand to ensure its quality. With designer coach shoulder bags, you will always attract people's attention. The bags will emphasize your personal style and taste.<br/>
xujie333
2012-5-21 15:17
coach outlet online Store guarantee that all the coach handbags offered are own high quality. In addition , all of them are sold at an unexpected low price.If you want to purchase, just visit their website.coach factory outlet is really sizzling kinds of shopping way for you. With the usage of the replica designer coach bags, you can surely be able to change your individual looks in a stunning manner.Coach bags on sale from the coach outlet are cheap or discount prices that you certainly will stand out from the crowd on your next camping trip!<br/>
coach88888888 Email
2012-8-13 17:54
If you desire to go to Coach Factory Outlet, but have no idea in which to go, you can research online. It is no doubt that there is drastically information and details about it for the reference.Coach Factory Online are loved by many people, when you walk in the street, you could see many people take coach styles.hat experts claim Coach Outlet Online shopping is in the changes they are available in.Coach Outlet will send you a coupon in the post to use in their upon only after you type a achieve. The Coach bags are customarily somewhat many than the ones in the stores.
baozang
2012-9-22 11:57
Lamo - Like Sebago, Louis Vuitton Canada  there are only about 40 styles of Lamo Footwear.  Most of their footwear uses a suede leather traditional in True Religion Jeans  moccasins, though only about 1/3 of the footwear collection is a traditional moccasin look. True Religion Brand Jeans The rest of the collection is slippers and UGG styled boots.Louis Vuitton Artsy The traditional moccasins do have a soft rubber sole making them great deck True Religion Sale shoes as well, however. Expect to pay between $50 and $60 for Lamo moccasins.
Coach Email Homepage
2012-11-3 14:32
"The Coach Outletengines that will likely power the J-31 we do know a bit mor.Coach Outlet  Those engines were actually revealed at the Zhuhai showCoach Outlet in 2008," Fisher said referring to an annual China air show. He beCoach Outletlieves the new J-31 engine is undergoing preliminary
有泉快充——网络充值平台
[2013-3-19 18:58]
来源:付向力宦
内容:中国移动手机话费充值http://shop68418126.taobao.com/category-383550299.htm
中国联通手机话费充值http://shop68418126.taobao.com/category-383550300.htm
中国电信手机话费充值http://shop68418126.taobao.com/category-383550301.htm
中国购物搜索联盟:http://sounion.gouwuke.com
分页: 4/5 第一页 上页 1 2 3 4 5 下页 最后页
发表评论
表情
emotemotemotemotemot
emotemotemotemotemot
emotemotemotemotemot
emotemotemotemotemot
emotemotemotemotemot
打开HTML
打开UBB
打开表情
隐藏
记住我
昵称   密码   游客无需密码
网址   电邮   [注册]