[文章/程序 作者:张宴 本文版本:v1.3 最后修改:2009.07.06 转载请注明原文链接:http://blog.zyan.cc/phpcws_v100/]

  注:最新的分词系统 HTTPCWS 已经发布,用来取代 PHPCWS。

  请点击以下网址下载 HTTPCWS:


  http://code.google.com/p/httpcws

  原来的 PHPCWS 停止更新。




  名称:PHPCWS(PHP中文分词扩展)
  协议:New BSD License
  作者:张宴
  网址:http://code.google.com/p/phpcws/
  SVN:http://code.google.com/p/phpcws/source/browse/#svn/trunk/phpcws

  一、PHPCWS 简介

  1、什么是 PHPCWS ?
  PHPCWS 是一款开源的PHP中文分词扩展,目前仅支持Linux/Unix系统。

  PHPCWS 先使用“ICTCLAS 3.0 共享版中文分词算法”的API进行初次分词处理,再使用自行编写的“逆向最大匹配算法”对分词和进行词语合并处理,并增加标点符号过滤功能,得出分词结果。

  ICTCLAS(Institute of Computing Technology, Chinese Lexical Analysis System)是中国科学院计算技术研究所在多年研究工作积累的基础上,基于多层隐马模型研制出的汉语词法分析系统,主要功能包括中文分词;词性标注;命名实体识别;新词识别;同时支持用户词典。ICTCLAS经过五年精心打造,内核升级6次,目前已经升级到了ICTCLAS3.0,分词精度98.45%,各种词典数据压缩后不到3M。ICTCLAS在国内973专家组组织的评测中活动获得了第一名,在第一届国际中文处理研究机构SigHan组织的评测中都获得了多项第一名,是当前世界上最好的汉语词法分析器。

  ICTCLAS 3.0 商业版是收费的,而免费提供的 ICTCLAS 3.0 共享版不开源,词库是根据人民日报一个月的语料得出的,很多词语不存在。所以本人对ICTCLAS分词后的结果,再采用逆向最大匹配算法,根据自己补充的一个9万条词语的自定义词库(与ICTCLAS词库中的词语不重复),对ICTCLAS分词结果进行合并处理,输出最终分词结果。

  由于 ICTCLAS 3.0 共享版只支持GBK编码,因此,如果是UTF-8编码的字符串,可以先用PHP的iconv函数转换成GBK编码,再用phpcws_split函数进行分词处理,最后转换回UTF-8编码。

  2、PHPCWS 中文分词在线演示

  演示网址:http://blog.zyan.cc/demo/phpcws/

  3、PHPCWS 分词速度及用途

  初次使用时,Apache 或 php-cgi(FastCGI) 进程,需要加载一次词库到内存中,需要0.0X秒。58字节的一句话——“2009年2月13日,我编写了一款PHP中文分词扩展:PHPCWS 1.0.0。”,分词速度只需0.0003秒。

  PHPCWS 属于《亿级数据的高并发通用搜索引擎架构设计》的一部分,用作“搜索查询接口”的关键字分词处理。在此架构中,Sphinx索引引擎对于CJK(中日韩)语言支持一元切分,假设【反恐行动是国产主视角射击网络游戏】这段文字,Sphinx会将其切成【反 恐 行 动 是 国 产 主 视 角 射 击 网 络 游 戏】,然后对每个字建立反向索引。如果用这句话中包含的字组成一个不存在的词语,例如【恐动】,也会被搜索到,所以搜索时,需要加引号,例如搜索【"反恐行动"】,就能完全匹配连在一起的四个字,不连续的【"恐动"】就不会被搜索到。但是,这样还有一个问题,搜索【"反恐行动游戏"】或【"国产网络游戏"】就会搜索不到。所以,我在搜索层写了个PHP中文分词扩展,搜索“反恐行动游戏”、“国产网络游戏”,会被PHPCWS中文分词函数分别切分为“反恐行动 游戏”、“国产 网络游戏”,这时候,用PHP函数给以空格分隔的词语加上引号,去搜索【"反恐行动" "游戏"】或【"国产" "网络游戏"】,就能搜索到这条记录了。由于PHPCWS位于搜索层,中文分词词库发生增、删、改,只需平滑重启一次Web服务器或php-cgi进程即可,无需重建搜索索引。

  根据上述情况,对于那些采用二元交叉切分的搜索引擎,PHPCWS用在前端搜索层对用户输入的搜索关键字、短语进行分词处理,同样适合。PHPCWS开发的目的正在于此,对于短句、小文本中文分词切分,速度非常之快。

  4、自定义词库

  自定义词库名称为userdict.tch,格式为 Tokyo Cabinet DBM 的 Abstract key-value 内存哈希数据库(key为GBK编码的词语名词,value为词频。目前词频均填1,暂时用不上)。自定义词库的修改在安装步骤中会详细介绍。



  二、phpcws 1.0.1 安装步骤

  1、安装 Tokyo Cabinet 数据库:
wget http://tokyocabinet.sourceforge.net/tokyocabinet-1.4.17.tar.gz
tar zxvf tokyocabinet-1.4.17.tar.gz
cd tokyocabinet-1.4.17/
./configure --prefix=/usr/local/tokyocabinet-1.4.17
make && make install
mkdir -p /usr/local/tokyocabinet-1.4.17/lib/static/
cp -f /usr/local/tokyocabinet-1.4.17/lib/*.a /usr/local/tokyocabinet-1.4.17/lib/static/
echo "/usr/local/tokyocabinet-1.4.17/lib" >> /etc/ld.so.conf
/sbin/ldconfig
cd ../


  2、安装 PHPCWS 扩展:
wget http://phpcws.googlecode.com/files/phpcws-1.0.1.tar.gz
tar zxvf phpcws-1.0.1.tar.gz
cd phpcws-1.0.1/
/usr/local/webserver/php/bin/phpize
./configure --with-php-config=/usr/local/webserver/php/bin/php-config --with-tc=/usr/local/tokyocabinet-1.4.17
make && make install
cd ../

  注:其中/usr/local/webserver/php/为您的PHP安装目录。

  3、安装 ICTCLAS 3.0 词库和 PHPCWS 自定义词库(词库汉字编码均为GBK):
cd /usr/local/
wget http://phpcws.googlecode.com/files/phpcws-dict-1.0.0.tar.gz
tar zxvf phpcws-dict-1.0.0.tar.gz
mv phpcws-dict-1.0.0 phpcws


  您可以往自定义词库内增加自己的词语,方法如下:
cd /usr/local/phpcws/

#如果userdict.tch已经被PHP加载,请拷贝一个备份
cp userdict.tch userdict_new.tch

#添加新词语
/usr/local/tokyocabinet-1.4.11/bin/tcamgr put userdict_new.tch 词语一 1
/usr/local/tokyocabinet-1.4.11/bin/tcamgr put userdict_new.tch 词语二 1

查询某个词语是否已经在词库内
/usr/local/tokyocabinet-1.4.11/bin/tcamgr get userdict_new.tch 词语名词

#删除一个词语
/usr/local/tokyocabinet-1.4.11/bin/tcamgr out userdict_new.tch 词语名称

#将修改后的新词库覆盖回原词库
mv userdict_new.tch userdict.tch

#重启您的Apache服务器,或php-cgi(FastCGI)进程。
#例如重启php-cgi(FastCGI)进程:
/usr/local/webserver/php/sbin/php reload

  您可以从搜狗细胞词库网站下载自己需要的行业分类文本词库,整合到PHPCWS自定义词库中。

  4、修改您的php.ini文件,增加以下几行内容:
[phpcws]
extension = "phpcws.so"
phpcws.dict_path = "/usr/local/phpcws"


  5、重启您的Apache服务器,或php-cgi(FastCGI)进程

  6、查看您的phpinfo信息,如果搜索到以下内容,则表明安装成功:

  点击在新窗口中浏览此图片



  三、phpcws 1.0.1 调用方式

  1、PHPCWS扩展拥有一个PHP函数:
  string phpcws_split ( string $text [, string $interpunction ] )

  参数说明:
  $text 为GBK编码的文本内容;

  $interpunction 为可选参数,用来控制是否过滤标点符号。
  ①、值为空时,表示不过滤;

  ②、值为"default"时,过滤掉默认的标点符号,即以下标点符号:
  ,.!!"#¥%&`'()〔〕〈〉《》「」『』〖〗【】.*+,-.。、?…—·ˉˇ¨‘’“”々~‖∶"/:;|〃<=>?@[\]^_`{|} ̄

  ③、值为其他字符串时,表示过滤掉自定义的标点符号,例如值为",.!,。!"即过滤掉半角、全角的逗号、句号、感叹号。

  2、PHP中文分词实例:

  ①、对GBK编码的字符串进行中文分词处理(example_gbk.php):

  ②、对UTF-8编码的字符串进行中文分词处理(example_utf8.php):

  输出结果为:
分词结果为:
2009年 2月 13日 , 我 编写 了 一款 PHP 中文 分词 扩展 : PHPCWS 1.0.0 。

过滤掉默认标点符号的分词结果为:
2009年 2月 13日 我 编写 了 一款 PHP 中文 分词 扩展 PHPCWS 1.0.0

过滤掉全角逗号与冒号的分词结果为:
2009年 2月 13日 我 编写 了 一款 PHP 中文 分词 扩展 PHPCWS 1.0.0 。


  以上两个实例PHP程序文件可以从SVN中获取:




技术大类 » 搜索引擎技术 | 评论(112) | 引用(0) | 阅读(98063)
游客
2010-9-9 19:52
在应用级别上研究的不错 我想问个问题 假如说词库中没有“中国” 对“中国”怎么分? 一个词还是两个字?如果是两个字,后续会怎样处理这个合理的词?
lingyunzi
2011-5-10 10:41
这个程序每运行一两个月后就会自动失效.需要重新杀掉进程重启才能继续运行.不知道怎么回事...您的演示也不管用了.估计也是这个原因.
phper
2011-6-5 03:45
目前在项目里需要用到,感谢张宴老师的共享精神,研究下
photoshop Email Homepage
2011-7-18 17:57
正好遇到这个问题,头疼之际,看到了这篇文章,很不错!!!!zan
dresses Email Homepage
2011-7-26 10:09
Here are a variety of the empire <a href="http://www.romancearound.com/">dresses</a>, I want to introduce yourself. The first shoulder chiffon <a href="http://www.romancearound.com/bridesmaid-dresses-c-197.html">bridesmaid dresses</a> empire style wedding will <a href="http://www.romancearound.com/wedding-dresses-trumpet-wedding-dresses-c-180_253.html">trumpet wedding dresses</a>.This Tia cowries, dressed in her wedding day.Chiffon of the empire the best clothes, because they are the size of the drum as complete motion from the wind the site makes it easy to create a plan impression.One romantic flowers floating above shoulder function and help catch the eyes of the cut of the chest and neck problems, so that more comprehensive coverage of physical illusion bust.Clothing do not like big long torso, short legs, or pear-shaped<a href="http://www.romancearound.com/bridesmaid-dresses-ankle-length-bridesmaid-dresses-c-197_198.html">ankle length bridesmaid dresses</a> figure.special is also suitable for pregnant brides since the mobility as a dress can accommodate more Many of the abdomen.
dresses Email Homepage
2011-7-26 10:15
Here are a variety of the empire <a href="http://www.romancearound.com/">dresses</a>, I want to introduce yourself. The first shoulder chiffon <a href="http://www.romancearound.com/bridesmaid-dresses-c-197.html">bridesmaid dresses</a> empire style wedding will <a href="http://www.romancearound.com/wedding-dresses-trumpet-wedding-dresses-c-180_253.html">trumpet wedding dresses</a>.This Tia cowries, dressed in her wedding day.Chiffon of the empire the best clothes, because they are the size of the drum as complete motion from the wind the site makes it easy to create a plan impression.One romantic flowers floating above shoulder function and help catch the eyes of the cut of the chest and neck problems, so that more comprehensive coverage of physical illusion bust.Clothing do not like big long torso, short legs, or pear-shaped<a href="http://www.romancearound.com/bridesmaid-dresses-ankle-length-bridesmaid-dresses-c-197_198.html">ankle length bridesmaid dresses</a> figure.special is also suitable for pregnant brides since the mobility as a dress can accommodate more Many of the abdomen.
Tizen论坛 Email Homepage
2011-10-18 17:08
出个utf8和中英文混合的吧,UTF-8转码PHP的iconv()函数有时候会出现问题。转换会出错。
Tizen论坛 Email Homepage
2011-10-18 17:11
很好,Tizen论坛的服务器已经使用你的中文分词。非常感激。
louis vuitton uk Email Homepage
2011-11-22 17:12
This louis vuitton uk for sale belongs to the sounding just what are termed as Louis Vuitton vintage best sellers, many other products and services for the reason that range appearing companies.You will easily notice the unfold zippers of this coach outlet store online. That is the decoration. There are some inside pockets for you as well. They are easy to match your clothes and to carry.Let us inspire your inner beauty with fine christian louboutin sale. Purse the elegance in bridal wedding. Enjoy the fashion.
armani watches Email Homepage
2011-11-23 10:13
If you are looking for armani Bags, our armani watches Handbags Canada outlet store is your first choice. We promise Original Packing and Best Discount,3-5 Workdays To Your Door!If you buy bags and purses of the latest new designs from the louis vuitton outlet now, you can enjoy special discounts. What else are you waiting for?After you choose the right kind of plants and the size pot for the plant.
qyhdt Email
2012-3-1 16:18
64位centos5.6系统
安装 phpcws时候 make时候报错
cannot find -lICTCLAS30
Hogan
2012-5-10 18:06
Ci sono diversi motivi principali per cui si può sicuramente trovare sul proprio volere le soluzioni contabili che coinvolgono tutta Stansted, anche se alcune persone potrebbero uomini e donne, probabilmente si dovrà disporre di soluzioni contabili rispetto con la gente. Stai con me per scoprire di più su molti dei motivi per prendere in considerazione l'impiego contabili in tutta Hogan Stansted. Costruire un businessIf vi capita di essere in funzione, è abbastanza possibile che si può fare uso di soluzioni contabili. Non date per scontato tutte le società hanno abbastanza soldi per utilizzare a tempo pieno il personale fiscali, così utilizzando una conditi insieme con l'agenzia affidabile che coinvolge tutta commercialisti Stansted potrebbe rendere meno difficili problemi.
hogan280 Email
2012-5-10 18:13
Focalizzando l'attenzione sull'elemento gara di questo reato - qualcosa suscitato dal comportamento oltraggioso della Difesa League inglese e BNP al di fuori del tribunale di Liverpool - è facile perdere di vista il fatto che Hogan questo è un racconto ampio di abuso e non è univoco ad una comunità. Marai Larasi, direttore della Imkaan e co-presidente del End Violence Against Women Coalition, avverte: "Una eccessiva attenzione su alcuni casi di sfruttamento sessuale con una concentrazione di etnie, piuttosto che lo sfruttamento della stessa è fuorviante e gli atteggiamenti razzisti dei combustibili che in ultima analisinon aiuta le donne e le ragazze. "
coach factory outlet Email Homepage
2012-5-17 13:12
Today, following half a century, mentor leather-bases coach factory outlet continues to be the delicate craft of leather-based master is accountable for,Would you like to meet more friends, or go with the times? If yes, coach factory online is opening welcome doors to you.in the market you definitely can find various colorways that are designed in as well as the high quality that applied in. For most of you would like to come. So just come to our coach factory outlet online store to choose one.
louis vuitton sale Email Homepage
2012-5-17 13:13
Louis Vuitton belt at louis vuitton sale is one kind of fashion accessory with high cost performance among the Louis Vuitton accessories.Offering quality LV products with favorable prices, louis vuitton outlet store is at your service. Hurry up, or you can not seize the chance.in fact, louis vuitton is one of the most famous fashion design master.he opened the fist suitcase shop called after his name.
coach outlet online Email Homepage
2012-5-17 13:13
I heard of coach outlet online through the advertisement when I was shopping. And now I often brow the webpage and buy Coach bags online.It is a symbol regarding position not to mention nature.Here I would like to launch a excellent bags pertaining to business men.Which may be coach outlet store.coach outlet has always been simple,durable style features to win consumers.The products are more flexible,with easy bleaching,wear characteristics,and simply use a damp cloth.
coach outlet, Email Homepage
2012-5-17 13:13
Remember the coach outlet provide coach bags which won't be deteriorated into its overall styles by any means. It will maintain its looks, colors, and uniqueness for long time.coach outlet store online has been voted by Hour Detroit magazine readers as the Best of Detroit in their 12th annual readers'poll.Lots of women like which usually amount normally include a coach outlet online ,it provides coziness to many girls that don't even think it is a great bushel of great interest directly to them.
xujie777 Email
2012-5-19 08:52
We aim to make all our customers satisfy with our products. You will find a variety of Men's fashion louis vuitton uk, fashion Women's cheap Louis Vuitton bags in our store at affordable price.Thinking of interesting ways to cost a milestone birthday? louis vuitton online shop had one of the most distinctive distinctive celebrations.bakery along with living room operated by means of about three moment louis vuitton online Most effective Pastry Chef’s of the year Rammy Nominee Chef’s.
xujie777 Email
2012-5-19 08:52
The choices are likely to be basically countless seeing that louis vuitton outlet occurs with the help of completely new and also incredible concepts once in a while.Louis vuitton Wholesale Monogram Canvas HandbagsLouis Vuitton Collection Beach Handbags louis vuitton bags outlet Damier Canvas HandbagsLouis vuitton Mahina HandbagsLouis Vuitton Monogram Mini Lin HandbagsLouis Vuitton Monogram Multicolore HandbagsLouis vuitton Monogram Vernis HandbagsLouis Vuitton Wholesale Epi Leather HandbagsLouis Vuitton For Men HandbagsLouis Vuitton Damier Canvas WalletsLouis Vuitton Epi Leather WalletsLouis Vuitton Monogram Canvas WalletsLouis Vuitton Monogram Vernis WalleLouis Vuitton ShoesLouis Vuitton Men wallets.As the Authentic Louis Vuitton are so high-priced, so came the louis vuitton handbags outlet.
xujie777 Email
2012-5-19 08:52
coach outlet has become a popular shopping experience for consumers around the world, and a desirable distribution channel for manufacturer's and retailers.Getting your hands on coach outlet store online can be a hefty investment of hundreds of dollars.But do not despair,the Coach Outlet store could be the answer to your prayers.coach outlet online is your smart choice when you want to get the discount Coach accessory. You can find the exact Coach Bags and other accessory you want at a low price that's right for you. My dear friends, let the coach outlet online pave your way into the world of high fashion with their ultimate fashion factory.
分页: 5/6 第一页 上页 1 2 3 4 5 6 下页 最后页
发表评论
表情
emotemotemotemotemot
emotemotemotemotemot
emotemotemotemotemot
emotemotemotemotemot
emotemotemotemotemot
打开HTML
打开UBB
打开表情
隐藏
记住我
昵称   密码   游客无需密码
网址   电邮   [注册]