[文章作者:张宴 本文版本:v1.0 最后修改:2008.12.09 转载请注明原文链接:http://blog.zyan.cc/post/385/]

  曾经在七月,写过一篇文章──《基于Sphinx+MySQL的千万级数据全文检索(搜索引擎)架构设计》,前公司的分类信息搜索基于此架构,效果明显,甚至将很大一部分带Where条件的MySQL SQL查询,都改用了Sphinx+MySQL搜索。但是,这套架构仍存在局限:一是MySQL本身的并发能力有限,在200~300个并发连接下,查询和更新就比较慢了;二是由于MySQL表的主键与Sphinx索引的ID一一对应,从而无法跨多表建立整站查询,而且新增加类别还得修改配置文件,比较麻烦;三是因为和MySQL集成,无法发挥出Sphinx的优势。

  最近,我设计出了下列这套最新的搜索引擎架构,目前已经写出“搜索查询接口”和“索引更新接口”的beta版。经测试,在一台“奔腾四 3.6GHz 双核CPU、2GB内存”的普通PC机,7000万条索引记录的条件下,“搜索查询接口”平均查询速度为0.0XX秒(查询速度已经达到百度、谷歌、搜狗、中国雅虎等搜索引擎的水平,详见文章末尾的“附2”),并且能够支撑高达5000的并发连接;而“索引更新接口”进行数据分析、入队列、返回信息给用户的全过程,高达1500 Requests/Sec。

  “队列控制器”这一部分是核心,它要控制队列读取,更新MySQL主表与增量表,更新搜索引擎数据存储层Tokyo Tyrant,准实时(1分钟内)完成更新Sphinx增量索引,定期合并Sphinx索引。我预计在这周写出beta版。

点击在新窗口中浏览此图片

  图示说明:
  1、搜索查询接口:
  ①、Web应用服务器通过HTTP POST/GET方式,将搜索关键字等条件,传递给搜索引擎服务器的search.php接口;
  ②③、search.php通过Sphinx的API(我根据最新的Sphinx 0.9.9-rc1 API,改写了一个C语言的PHP扩展sphinx.so),查询Sphinx索引服务,取得满足查询条件的搜索引擎唯一ID(15位搜索唯一ID:前5位类别ID+后10位原数据表主键ID)列表;
  ④⑤、search.php将这些ID号作为key,通过Memcache协议一次性从Tokyo Tyrant中mget取回ID号对应的文本数据。
  ⑥⑦、search.php将搜索结果集,按查询条件,进行摘要和关键字高亮显示处理,以JSON格式或XML格式返回给Web应用服务器。

  2、索引更新接口:
  ⑴、Web应用服务器通过HTTP POST/GET方式,将要增加、删除、更新的内容告知搜索服务器的update.php接口;
  ⑵、update.php将接收到的信息处理后,写入TT高速队列(我基于Tokyo Tyrant做的一个队列系统);
  注:这两步的速度可达到1500次请求/秒以上,可应对6000万PV的搜索索引更新调用。

  3、搜索索引与数据存储控制:
  ㈠、“队列控制器”守护进程从TT高速队列中循环读取信息(每次50条,直到末尾);
  ㈡、“队列控制器”将读取出的信息写入搜索引擎数据存储层Tokyo Tyrant;
  ㈢、“队列控制器”将读取出的信息异步写入MySQL主表(这张主表按500万条记录进行分区,仅作为数据永久性备份用);
  ㈣、“队列控制器”将读取出的信息写入MySQL增量表;
  ㈤、“队列控制器”在1分钟内,触发Sphinx更新增量索引,Sphinx的indexer会将MySQL增量表作为数据源,建立增量索引。Sphinx的增量索引和作为数据源的MySQL增量表成对应关系;
  ㈥、“队列控制器”每间隔3小时,短暂停止从TT高速队列中读取信息,并触发Sphinx将增量索引合并入主索引(这个过程非常快),同时清空MySQL增量表(保证了MySQL增量表的记录数始终只有几千条至几十万条,大大加快Sphinx增量索引更新速度),然后恢复从TT高速队列中取出数据,写入MySQL增量表。

  本架构使用的开源软件:
  1、Sphinx 0.9.9-rc1
  2、Tokyo Tyrant 1.1.9
  3、MySQL 5.1.30
  4、Nginx 0.7.22
  5、PHP 5.2.6

  本架构自主研发的程序:
  1、搜索查询接口(search.php)
  2、索引更新接口(update.php)
  3、队列控制器
  4、Sphinx 0.9.9-rc1 API的PHP扩展(sphinx.so)
  5、基于Tokyo Tyrant的高速队列系统



  附1:MySQL FullText、Lucene搜索、Sphinx搜索的第三方对比结果:
  1、查询速度:
  MySQL FullText最慢,Lucene、Sphinx查询速度不相上下,Sphinx稍占优势。
  点击在新窗口中浏览此图片

  2、建索引速度:
  Sphinx建索引速度是最快的,比Lucene快9倍以上。因此,Sphinx非常适合做准实时搜索引擎。

  3、详细对比数据见以下PDF文档:  



  附2:国内各大中文搜索引擎搜索速度分析:
  以“APMServ张宴”为关键字,比较在各大中文搜索引擎的搜索速度:
  1、百度:
  ①、第一次搜索
  点击在新窗口中浏览此图片

  ②、第二次搜索
  点击在新窗口中浏览此图片

  分析:百度对第一次搜索的搜索结果做了Cache,所以第二次查询非常快。



  2、谷歌:
  ①、第一次搜索
  点击在新窗口中浏览此图片

  ②、第二次搜索
  点击在新窗口中浏览此图片

  分析:谷歌也对第一次搜索的搜索结果做了Cache,但两次查询跟百度同比,都要慢一些。



  3、搜狗:
  ①、第一次搜索
  点击在新窗口中浏览此图片

  ②、第二次搜索
  点击在新窗口中浏览此图片

  ③、第三次搜索
  点击在新窗口中浏览此图片

  分析:搜狗疑似对第一次搜索的搜索结果做了短暂的Cache,第二次搜索速度非常快,第三次搜索的速度比第二次搜索的速度慢。搜狗第一次搜索的速度跟百度差不多。



  4、中国雅虎:
  ①、第一次搜索
  点击在新窗口中浏览此图片

  ②、第二次搜索
  点击在新窗口中浏览此图片

  分析:搜索结果没有做Cache。中国雅虎的搜索速度跟百度第一次搜索的速度差不多。



  5、网易有道:
  ①、第一次搜索
  点击在新窗口中浏览此图片

  ②、第二次搜索
  点击在新窗口中浏览此图片

  分析:有道对第一次搜索的搜索结果做了Cache。但是,跟谷歌一样,两次搜索同比都要较百度、搜狗、中国雅虎慢一些。






技术大类 » 搜索引擎技术 | 评论(99) | 引用(1) | 阅读(156320)
悠悠我心 Email
2013-4-26 16:28
shy学习了!~
磨延城 Email Homepage
2013-10-21 21:09
磨途歌学习了
Maxwell
2014-7-14 15:31
铭铭中来到了学长的博客,我也是民大的,今年刚毕业的,现在做数据挖掘,偶遇大牛学长,留言点赞!!!!zanzanzan学长不介意的话加个好友链接哈~~blog.csdn.net/yeweiouyang,非常感谢!
Steven Email Homepage
2015-3-19 02:59
Thank you for this post Weight Loss Tea
bird
2017-10-9 21:46
是否还回复?数据写入 tt队列,是否可以保证事务提交成功?如何进行事务回滚?
John32 Email Homepage
2019-5-27 08:04
Looking for the best writing service online but lost in the number of fake reviews and don’t know who to believe? We at ScamFighter publish only genuine reviews on all writing service from the web. Look at the last review on: https://scamfighter.net/review/speedypaper.com
PhilR Email
2019-6-30 02:18
Whether you’re planning to explore all opportunities, consider the job you really wanted in the first place, or just move up the career ladder,  we may provide you "whitbread review" on Resume101.org to help you make the best choice.
Bill Email
2019-7-11 06:21
The nursing thesis or nursing dissertation is the most timeconsuming and voluminous among the research works of various nursing programs. Therefore, if you search "nursing school admission essays" Welcome to nursingessaywriting.com. Let us facilitate your life and save your time and good mood for you, for when you order your nursing research papers, you will receive the top quality work. A well-organized teamwork assures of the best result possible.
Bill Email
2019-7-11 06:21
The nursing thesis or nursing dissertation is the most timeconsuming and voluminous among the research works of various nursing programs. Therefore, if you search "nursing school admission essays" Welcome to nursingessaywriting.com. Let us facilitate your life and save your time and good mood for you, for when you order your nursing research papers, you will receive the top quality work. A well-organized teamwork assures of the best result possible.
Bill Email
2019-7-11 06:23
The nursing thesis or nursing dissertation is the most timeconsuming and voluminous among the research works of various nursing programs. Therefore, if you search "nursing school admission essays" Welcome to nursingessaywriting.com. Let us facilitate your life and save your time and good mood for you, for when you order your nursing research papers, you will receive the top quality work. A well-organized teamwork assures of the best result possible.
johnmathew Email
2019-11-12 14:19
What an amazing post you are sharing here. I am really wonder reading this and welcome method of writing an article. You are always sharing some quality content for us like this post, that why we are always like your posts. I hope you are continuing writing fabulous and informative article for us. I would like you to recommend high quality essay content from custom essay writing service; it is helpful to students who are studying in higher studies. Once again I thankful to your post.
Computer Science Writing Services Homepage
2020-1-28 13:15
You are sure to get the best quality computer science writing services in the shortest period when you seek online computer science writing services and computer science writitng help services.
Customized essay help Email Homepage
2020-6-19 05:43
We provide assignment help with best quality and at cheap rates, our allassignmenthelp.com is a well-known as best website for the assignment.assignment help
ALANCOOPER Email
2020-7-8 13:46
Online Assignment is your academic caretaker, Best homework help assistance is one of the most sought after in the Uk and other countries and you should give us a chance to assist you.Visit: https://www.allhomeworkhelp.co.ukVisit: https://www.makemyassignments.co.ukVisit: https://www.allhomeworkhelp.co.uk/math-homework-help.htmlVisit: https://www.makemyassignments.co.uk/buy-assignments-online.html
pay someone to do my online class Email Homepage
2020-8-13 20:19
Can I pay someone to do my online class place approaching me? Granted, we're a stable free grammar check providers joining you among material teachers to get your education & experiments.  https://www.allassignmenthelp.com/check-grammarhttps://www.allassignmenthelp.com/digital-electronics-assignment-help.htm...
MAXWELL Email Homepage
2020-9-12 23:11
Professional Help with assignment from experienced and qualified academic writers.We provide best pay someone to do my math homework assistance to university students.pay someone to do my math homework
programming helpers Email Homepage
2020-10-23 22:49
cs assignment help project from able and dedicated authorities for learners fighting with their study in one programming communication. programming helpers on the lowest price all over the world. please try us. we will not disappoint you.https://www.programmingassignmenthelpers.com/programming-experts.html
分页: 5/5 第一页 上页 1 2 3 4 5 最后页
发表评论
表情
emotemotemotemotemot
emotemotemotemotemot
emotemotemotemotemot
emotemotemotemotemot
emotemotemotemotemot
打开HTML
打开UBB
打开表情
隐藏
记住我
昵称   密码   游客无需密码
网址   电邮   [注册]