发布版本:
httpcws 1.0.0 (最新版本:2009-08-10发布)
程序网址:http://code.google.com/p/httpcws
安装使用手册:http://blog.zyan.cc/httpcws_v100/
下载地址(32位版):http://httpcws.googlecode.com/files/httpcws-1.0.0-i386-bin.tar.gz
下载地址(64位版):http://httpcws.googlecode.com/files/httpcws-1.0.0-x86_64-bin.tar.gz
中文分词在线演示:http://blog.zyan.cc/demo/httpcws/
PHP演示程序下载:http://blog.zyan.cc/demo/httpcws/httpcws-php-demo.zip
httpcws 中文简介
1、什么是 httpcws ?
HTTPCWS 是一款基于HTTP协议的开源中文分词系统,目前仅支持Linux系统。HTTPCWS 使用“ICTCLAS 3.0 2009共享版中文分词算法”的API进行分词处理,得出分词结果。HTTPCWS 将取代本人之前开发的 PHPCWS 中文分词扩展。
ICTCLAS(Institute of Computing Technology, Chinese Lexical Analysis System)是中国科学院计算技术研究所在多年研究工作积累的基础上,基于多层隐马模型研制出的汉语词法分析系统,主要功能包括中文分词;词性标注;命名实体识别;新词识别;同时支持用户词典。ICTCLAS经过五年精心打造,内核升级6次,目前已经升级到了ICTCLAS3.0,分词精度98.45%,各种词典数据压缩后不到3M。ICTCLAS在国内973专家组组织的评测中活动获得了第一名,在第一届国际中文处理研究机构SigHan组织的评测中都获得了多项第一名,是当前世界上最好的汉语词法分析器。
ICTCLAS 3.0 商业版是收费的,而免费提供的 ICTCLAS 3.0 共享版不开源,词库是根据人民日报一个月的语料得出的,很多词语不存在。所以本人补充的一个19万条词语的自定义词库,对ICTCLAS分词结果进行合并处理,输出最终分词结果。
由于 ICTCLAS 3.0 2009 共享版只支持GBK编码,因此,如果是UTF-8编码的字符串,可以先用iconv函数转换成GBK编码,再用httpcws进行分词处理,最后转换回UTF-8编码。
HTTPCWS 软件自身(包括httpcws.cpp源文件、dict/httpcws_dict.txt自定义词库)采用NewBSD开源协议,可以自由修改。HTTPCWS 使用的 ICTCLAS 共享版 API 及 dict/Data/ 目录内的语料库,版权及著作权归中国科学院计算技术研究所、ictclas.org所有,使用需遵循其相关协议。
2、httpcws 中文分词在线演示
演示网址:http://blog.zyan.cc/demo/httpcws/
3、httpcws 中文分词下载安装
32位版:
64位版:
命令行启动参数:
4、httpcws 使用方法
GET方法(文本长度受URL的长度限制,需要分词的文本为GBK编码,最好采用urlencode对文本进行编码):
POST方法(文本长度无限制,适用于大文本分词,需要分词的文本为GBK编码,最好采用urlencode对文本进行编码):
PHP 调用 HTTPCWS 示例:
①、对GBK编码的字符串进行中文分词处理(HTTP POST方式):
②、对UTF-8编码的字符串进行中文分词处理(HTTP POST方式):
③、对GBK编码的字符串进行中文分词处理(HTTP GET方式):
④、对UTF-8编码的字符串进行中文分词处理(HTTP GET方式):
5、httpcws 分词速度及用途
局域网内 HTTPCWS 接口中文分词平均处理速度(Wait时间):0.001秒。HTTPCWS 基于 libevent + epoll 网络IO模型开发,经测试,每秒可处理5000~20000次请求。
HTTPCWS 属于《[http://blog.zyan.cc/post/385.htm 亿级数据的高并发通用搜索引擎架构设计]》的一部分,用作“搜索查询接口”的关键字分词处理。在此架构中,Sphinx索引引擎对于CJK(中日韩)语言支持一元切分,假设【反恐行动是国产主视角射击网络游戏】这段文字,Sphinx会将其切成【反 恐 行 动 是 国 产 主 视 角 射 击 网 络 游 戏】,然后对每个字建立反向索引。如果用这句话中包含的字组成一个不存在的词语,例如【恐动】,也会被搜索到,所以搜索时,需要加引号,例如搜索【"反恐行动"】,就能完全匹配连在一起的四个字,不连续的【"恐动"】就不会被搜索到。但是,这样还有一个问题,搜索【"反恐行动游戏"】或【"国产网络游戏"】就会搜索不到。所以,我在搜索层写了个PHP中文分词扩展,搜索“反恐行动游戏”、“国产网络游戏”,会被httpcws中文分词函数分别切分为“反恐行动 游戏”、“国产 网络游戏”,这时候,用PHP函数给以空格分隔的词语加上引号,去搜索【"反恐行动" "游戏"】或【"国产" "网络游戏"】,就能搜索到这条记录了。由于httpcws位于搜索层,中文分词词库发生增、删、改,只需重启httpcws进程即可,无需重建搜索索引。
根据上述情况,对于那些采用二元交叉切分的搜索引擎,httpcws用在前端搜索层对用户输入的搜索关键字、短语进行分词处理,同样适合。httpcws开发的目的正在于此,对于短句、小文本中文分词切分,速度非常之快。
6、自定义词库
修改dict/httpcws_dict.txt文件,可以自由增加自己需要的词语。重启httpcws即可生效。
httpcws 1.0.0 (最新版本:2009-08-10发布)
程序网址:http://code.google.com/p/httpcws
安装使用手册:http://blog.zyan.cc/httpcws_v100/
下载地址(32位版):http://httpcws.googlecode.com/files/httpcws-1.0.0-i386-bin.tar.gz
下载地址(64位版):http://httpcws.googlecode.com/files/httpcws-1.0.0-x86_64-bin.tar.gz
中文分词在线演示:http://blog.zyan.cc/demo/httpcws/
PHP演示程序下载:http://blog.zyan.cc/demo/httpcws/httpcws-php-demo.zip
httpcws 中文简介
1、什么是 httpcws ?
HTTPCWS 是一款基于HTTP协议的开源中文分词系统,目前仅支持Linux系统。HTTPCWS 使用“ICTCLAS 3.0 2009共享版中文分词算法”的API进行分词处理,得出分词结果。HTTPCWS 将取代本人之前开发的 PHPCWS 中文分词扩展。
ICTCLAS(Institute of Computing Technology, Chinese Lexical Analysis System)是中国科学院计算技术研究所在多年研究工作积累的基础上,基于多层隐马模型研制出的汉语词法分析系统,主要功能包括中文分词;词性标注;命名实体识别;新词识别;同时支持用户词典。ICTCLAS经过五年精心打造,内核升级6次,目前已经升级到了ICTCLAS3.0,分词精度98.45%,各种词典数据压缩后不到3M。ICTCLAS在国内973专家组组织的评测中活动获得了第一名,在第一届国际中文处理研究机构SigHan组织的评测中都获得了多项第一名,是当前世界上最好的汉语词法分析器。
ICTCLAS 3.0 商业版是收费的,而免费提供的 ICTCLAS 3.0 共享版不开源,词库是根据人民日报一个月的语料得出的,很多词语不存在。所以本人补充的一个19万条词语的自定义词库,对ICTCLAS分词结果进行合并处理,输出最终分词结果。
由于 ICTCLAS 3.0 2009 共享版只支持GBK编码,因此,如果是UTF-8编码的字符串,可以先用iconv函数转换成GBK编码,再用httpcws进行分词处理,最后转换回UTF-8编码。
HTTPCWS 软件自身(包括httpcws.cpp源文件、dict/httpcws_dict.txt自定义词库)采用NewBSD开源协议,可以自由修改。HTTPCWS 使用的 ICTCLAS 共享版 API 及 dict/Data/ 目录内的语料库,版权及著作权归中国科学院计算技术研究所、ictclas.org所有,使用需遵循其相关协议。
2、httpcws 中文分词在线演示
演示网址:http://blog.zyan.cc/demo/httpcws/
3、httpcws 中文分词下载安装
32位版:
cd /usr/local/
wget http://httpcws.googlecode.com/files/httpcws-1.0.0-i386-bin.tar.gz
tar zxvf httpcws-1.0.0-i386-bin.tar.gz
rm -f httpcws-1.0.0-i386-bin.tar.gz
cd httpcws-1.0.0-i386-bin/
ulimit -SHn 65535
/usr/local/httpcws-1.0.0-i386-bin/httpcws -d -x /usr/local/httpcws-1.0.0-i386-bin/dict/
wget http://httpcws.googlecode.com/files/httpcws-1.0.0-i386-bin.tar.gz
tar zxvf httpcws-1.0.0-i386-bin.tar.gz
rm -f httpcws-1.0.0-i386-bin.tar.gz
cd httpcws-1.0.0-i386-bin/
ulimit -SHn 65535
/usr/local/httpcws-1.0.0-i386-bin/httpcws -d -x /usr/local/httpcws-1.0.0-i386-bin/dict/
64位版:
cd /usr/local/
wget http://httpcws.googlecode.com/files/httpcws-1.0.0-x86_64-bin.tar.gz
tar zxvf httpcws-1.0.0-x86_64-bin.tar.gz
rm -f httpcws-1.0.0-x86_64-bin.tar.gz
cd httpcws-1.0.0-x86_64-bin/
ulimit -SHn 65535
/usr/local/httpcws-1.0.0-x86_64-bin/httpcws -d -x /usr/local/httpcws-1.0.0-x86_64-bin/dict/
wget http://httpcws.googlecode.com/files/httpcws-1.0.0-x86_64-bin.tar.gz
tar zxvf httpcws-1.0.0-x86_64-bin.tar.gz
rm -f httpcws-1.0.0-x86_64-bin.tar.gz
cd httpcws-1.0.0-x86_64-bin/
ulimit -SHn 65535
/usr/local/httpcws-1.0.0-x86_64-bin/httpcws -d -x /usr/local/httpcws-1.0.0-x86_64-bin/dict/
命令行启动参数:
4、httpcws 使用方法
GET方法(文本长度受URL的长度限制,需要分词的文本为GBK编码,最好采用urlencode对文本进行编码):
http://192.168.8.42:1985/?w=有人的地方就有江湖
http://192.168.8.42:1985/?w=%D3%D0%C8%CB%B5%C4%B5%D8%B7%BD%BE%CD%D3%D0%BD%AD%BA%FE
http://192.168.8.42:1985/?w=%D3%D0%C8%CB%B5%C4%B5%D8%B7%BD%BE%CD%D3%D0%BD%AD%BA%FE
POST方法(文本长度无限制,适用于大文本分词,需要分词的文本为GBK编码,最好采用urlencode对文本进行编码):
curl -d "有人的地方就有江湖" http://192.168.8.42:1985
curl -d "%D3%D0%C8%CB%B5%C4%B5%D8%B7%BD%BE%CD%D3%D0%BD%AD%BA%FE" http://192.168.8.42:1985
curl -d "%D3%D0%C8%CB%B5%C4%B5%D8%B7%BD%BE%CD%D3%D0%BD%AD%BA%FE" http://192.168.8.42:1985
PHP 调用 HTTPCWS 示例:
①、对GBK编码的字符串进行中文分词处理(HTTP POST方式):
<?php
@header('Content-Type: text/html; charset=gb2312');
$text = "有人的地方就有江湖";
$text = urlencode($text);
$opts = array(
'http'=>array(
'method'=>"POST",
'header'=>"Content-type: application/x-www-form-urlencoded\r\n".
"Content-length:".strlen($data)."\r\n" .
"Cookie: foo=bar\r\n" .
"\r\n",
'content' => $text,
)
);
$context = stream_context_create($opts);
$result = file_get_contents("http://127.0.0.1:1985", false, $context);
echo $result;
?>
@header('Content-Type: text/html; charset=gb2312');
$text = "有人的地方就有江湖";
$text = urlencode($text);
$opts = array(
'http'=>array(
'method'=>"POST",
'header'=>"Content-type: application/x-www-form-urlencoded\r\n".
"Content-length:".strlen($data)."\r\n" .
"Cookie: foo=bar\r\n" .
"\r\n",
'content' => $text,
)
);
$context = stream_context_create($opts);
$result = file_get_contents("http://127.0.0.1:1985", false, $context);
echo $result;
?>
②、对UTF-8编码的字符串进行中文分词处理(HTTP POST方式):
<?php
@header('Content-Type: text/html; charset=utf-8');
$text = "有人的地方就有江湖";
$text = iconv("UTF-8", "GBK//IGNORE", $text);
$text = urlencode($text);
$opts = array(
'http'=>array(
'method'=>"POST",
'header'=>"Content-type: application/x-www-form-urlencoded\r\n".
"Content-length:".strlen($data)."\r\n" .
"Cookie: foo=bar\r\n" .
"\r\n",
'content' => $text,
)
);
$context = stream_context_create($opts);
$result = file_get_contents("http://127.0.0.1:1985", false, $context);
$result = iconv("GBK", "UTF-8//IGNORE", $result);
echo $result;
?>
@header('Content-Type: text/html; charset=utf-8');
$text = "有人的地方就有江湖";
$text = iconv("UTF-8", "GBK//IGNORE", $text);
$text = urlencode($text);
$opts = array(
'http'=>array(
'method'=>"POST",
'header'=>"Content-type: application/x-www-form-urlencoded\r\n".
"Content-length:".strlen($data)."\r\n" .
"Cookie: foo=bar\r\n" .
"\r\n",
'content' => $text,
)
);
$context = stream_context_create($opts);
$result = file_get_contents("http://127.0.0.1:1985", false, $context);
$result = iconv("GBK", "UTF-8//IGNORE", $result);
echo $result;
?>
③、对GBK编码的字符串进行中文分词处理(HTTP GET方式):
<?php
@header('Content-Type: text/html; charset=gb2312');
$text = "有人的地方就有江湖";
$text = urlencode($text);
$result = file_get_contents("http://127.0.0.1:1985/?w=".$text);
echo $result;
?>
@header('Content-Type: text/html; charset=gb2312');
$text = "有人的地方就有江湖";
$text = urlencode($text);
$result = file_get_contents("http://127.0.0.1:1985/?w=".$text);
echo $result;
?>
④、对UTF-8编码的字符串进行中文分词处理(HTTP GET方式):
<?php
@header('Content-Type: text/html; charset=utf-8');
$text = "有人的地方就有江湖";
$text = iconv("UTF-8", "GBK//IGNORE", $text);
$text = urlencode($text);
$result = file_get_contents("http://127.0.0.1:1985/?w=".$text);
$result = iconv("GBK", "UTF-8//IGNORE", $result);
echo $result;
?>
@header('Content-Type: text/html; charset=utf-8');
$text = "有人的地方就有江湖";
$text = iconv("UTF-8", "GBK//IGNORE", $text);
$text = urlencode($text);
$result = file_get_contents("http://127.0.0.1:1985/?w=".$text);
$result = iconv("GBK", "UTF-8//IGNORE", $result);
echo $result;
?>
5、httpcws 分词速度及用途
局域网内 HTTPCWS 接口中文分词平均处理速度(Wait时间):0.001秒。HTTPCWS 基于 libevent + epoll 网络IO模型开发,经测试,每秒可处理5000~20000次请求。
HTTPCWS 属于《[http://blog.zyan.cc/post/385.htm 亿级数据的高并发通用搜索引擎架构设计]》的一部分,用作“搜索查询接口”的关键字分词处理。在此架构中,Sphinx索引引擎对于CJK(中日韩)语言支持一元切分,假设【反恐行动是国产主视角射击网络游戏】这段文字,Sphinx会将其切成【反 恐 行 动 是 国 产 主 视 角 射 击 网 络 游 戏】,然后对每个字建立反向索引。如果用这句话中包含的字组成一个不存在的词语,例如【恐动】,也会被搜索到,所以搜索时,需要加引号,例如搜索【"反恐行动"】,就能完全匹配连在一起的四个字,不连续的【"恐动"】就不会被搜索到。但是,这样还有一个问题,搜索【"反恐行动游戏"】或【"国产网络游戏"】就会搜索不到。所以,我在搜索层写了个PHP中文分词扩展,搜索“反恐行动游戏”、“国产网络游戏”,会被httpcws中文分词函数分别切分为“反恐行动 游戏”、“国产 网络游戏”,这时候,用PHP函数给以空格分隔的词语加上引号,去搜索【"反恐行动" "游戏"】或【"国产" "网络游戏"】,就能搜索到这条记录了。由于httpcws位于搜索层,中文分词词库发生增、删、改,只需重启httpcws进程即可,无需重建搜索索引。
根据上述情况,对于那些采用二元交叉切分的搜索引擎,httpcws用在前端搜索层对用户输入的搜索关键字、短语进行分词处理,同样适合。httpcws开发的目的正在于此,对于短句、小文本中文分词切分,速度非常之快。
6、自定义词库
修改dict/httpcws_dict.txt文件,可以自由增加自己需要的词语。重启httpcws即可生效。
สล็อตเครดิตฟรี
2021-10-11 17:33
สล็อตเครดิตฟรี Your article is very useful and interesting. i will follow you forever
แจกเครดิตฟรี
2021-10-11 17:34
[[https://pgauto789.com/%E0%B9%80%E0%B8%84%E0%B8%A3%E0%B8%94%E0%B8%B4%E0%B8%95%E0%B8%9F%E0%B8%A3%E0%B8%B5/|แจกเครดิตฟรี]] I am happy to find your distinguished way of writing the post. Now you make it easy for me to understand and implement the concept. Thank you for the post.
pgslot.art
2021-10-16 03:23
I have been surfing online more than 2 hours today, yet I never found any interesting article like yours.It is pretty worth enough for me. Personally,if all web owners and bloggers made good content asyou did, the web will be a lot more useful than ever before.pg slot auto 24/7 Visit herehttps://pgslot.art/pg slot auto
Bb123123
2021-10-16 03:25
great good wonderful.https://pgslot.art/https://pgslot.art/
123
2021-10-24 22:26
<a href="https://joker-autogame.com/คาสิโนวอเล็ต/">คาสิโนวอเล็ต</a>
pgslotauto888 pg slot
2021-11-7 05:40
1111 Thank you for allowing me to leave a comment hereIt’s a really good article and website.I want to recommend one websiteThis website is good for someone who wants to make money online.There are more than 500 games including ‘pg soft’ and ‘joker game’You can register easily, play 24 hours. If you are interested, please visit PGSLOTAUTO888https://pgslotauto888.com/pg-slot/
PGGAMESLOT
2021-11-9 17:56
pg The latest with a game system designed for direct pg slot players because the game format is very advanced, can play pg slot auto via ios and android systems, supports playing pg slots via mobile Deposit-withdraw automatically in just 8 seconds
PGGAMESLOT
2021-11-9 17:56
ฝาก50รับ150 No need to turn or call each other easily understood is a promotion, deposit 50, get 150 wallet, is another promotion that many people like and think that it is a worthwhile promotion.
PGGAMESLOT
2021-11-9 17:56
เครดิตฟรี No deposit required is something that online gambling sites. free credit slots pg There is a reward for the members of the web. free credit slots no deposit will be able to play free slots all within the web
PGGAMESLOT
2021-11-9 17:56
joker game Direct web slots, not through agents online gambling games The most legendary casino game Stuck in one of the 5 most popular slot game camps. With more than 150 games to choose from, Joker Slots
PGGAMESLOT
2021-11-9 17:56
โจ๊กเกอร์ Direct web slots, not through agents online gambling games The most legendary casino game Stuck in one of the 5 most popular slot game camps. With more than 150 games to choose from, Joker Slots
เกมยิงปลา
2021-11-11 12:28
เกมยิงปลา<a href="https://starvip789.com/เกมยิงปลา/">เกมยิงปลา</a> Earn real money without investment 2021 new dimension of making money from online gambling games Play free fish shooting games.
เกมยิงปลา
2021-11-11 12:28
<a href="https://starvip789.com/เกมยิงปลา/">เกมยิงปลา</a> Earn real money without investment 2021 new dimension of making money from online gambling games Play free fish shooting games.
เกมยิงปลา
2021-11-11 12:28
<a href="https://starvip789.com/เกมยิงปลา/">เกมยิงปลา</a> Earn real money without investment 2021 new dimension of making money from online gambling games Play free fish shooting games.
ไฮโล
2021-11-11 12:29
<a href="https://starvip789.com/ไฮโลออนไลน์/">ไฮโล</a> Slot games from the popular roma xo slots and free roma credit slots 2021 You can try it for free now.
โปรโมชั่นสล็อต
2021-11-11 12:29
<a href="https://starvip789.com/โปรโมชั่น/">โปรโมชั่นสล็อต</a> Promotion of new web slots 2021 that is becoming popular right now. inevitable promotion New member 100% bonus
เล่นสล็อต
2021-11-11 12:30
<a href="https://pg-gameslot.com/เล่นสล็อต/">เล่นสล็อต</a> Free Trial PG and Joker is to play slots for free without having to pay a deposit first. There are currently playing slots. mostly through the mobile internet without having to go to the casino to play slots
Super Slot 888
2021-11-11 12:31
<a href="https://pg-gameslot.com/super-slot-888/">Super Slot 888</a> The number 1 online slot game in Thailand, slot 888 online that includes the slot 888 game camp to play more than 300 games, slot 888 auto, open for deposit-withdrawal service with an automatic system.
pg slot เว็บใหม่
2021-11-11 12:33
<a href="https://pg-gameslot.com/pg-slot-เว็บใหม่/">pg slot เว็บใหม่</a> The hottest slot game camp now in 2021, including all slots, all game camps in one website, welcomes PG, a new website, free credit, no deposit required, easy to play, earn real money.
ฝาก30รับ100
2021-11-11 12:39
<a href="https://allslot878.com/ฝาก30รับ100/">ฝาก30รับ100</a> Latest 2021 Promotion Hits Slots Can play all game camps, new members, deposit 30, get 100, give away free credit, no need to deposit, no need to share Promotion deposit 30 get 100 unlimited withdrawal No minimum deposit
分页: 8/21 3 4 5 6 7 8 9 10 11 12