第1章 开始驾驭文本 ...............................................................................1
1.1 驾驭文本重要的原因 ...............................................................................................2
1.2 预览：一个基于事实的问答系统 ...........................................................................4
1.2.1 嗨，弗兰肯斯坦医生 ...................................................................................5
1.3 理解文本很困难 .......................................................................................................8
1.4 驾驭的文本 .............................................................................................................11
1.5 文本及智能应用：搜索及其他 .............................................................................13
1.5.1 搜索和匹配 .................................................................................................13
1.5.2 抽取信息 .....................................................................................................14
1.5.3 对信息分组 .................................................................................................15
1.5.4 一个智能应用 .............................................................................................15
1.6 小结 .........................................................................................................................15
1.7 相关资源 .................................................................................................................16
第2章 驾驭文本的基础 ..........................................................................17
2.1 语言基础知识 .........................................................................................................18
2.1.1 词语及其类别 .............................................................................................19
2.1.2 短语及子句 .................................................................................................20
2.1.3 词法 .............................................................................................................21
2.2 文本处理常见工具 .................................................................................................23
2.2.1 字符串处理工具 .........................................................................................23
2.2.2 词条及切词 .................................................................................................23
2.2.3 词性标注 .....................................................................................................25
2.2.4 词干还原 .....................................................................................................27
2.2.5 句子检测 .....................................................................................................29
2.2.6 句法分析和文法 .........................................................................................31
2.2.7 序列建模 .....................................................................................................33
2.3 从常见格式文件中抽取内容并做预处理 .............................................................34
2.3.1 预处理的重要性 .........................................................................................35
2.3.2 利用Apache Tika抽取内容 ........................................................................37
2.4 小结 .........................................................................................................................39
2.5 相关资源 .................................................................................................................40
第3章 搜索 ............................................................................................41
3.1 搜索和多面示例：Amazon.com ............................................................................42
3.2 搜索概念入门 .........................................................................................................44
3.2.1 索引内容 .....................................................................................................45
3.2.2 用户输入 .....................................................................................................47
3.2.3 利用向量空间模型对文档排名 .................................................................51
3.2.4 结果展示 .....................................................................................................54
3.3 Apache Solr搜索服务器介绍 .................................................................................57
3.3.1 首次运行Solr ..............................................................................................58
3.3.2 理解Solr中的概念 ......................................................................................59
3.4 利用Apache Solr对内容构建索引 .........................................................................63
3.4.1 使用XML构建索引 ....................................................................................64
3.4.2 利用Solr和Apache Tika对内容进行抽取和索引 ......................................66
3.5 利用Apache Solr来搜索内容 .................................................................................69
3.5.1 Solr查询输入参数 ......................................................................................71
3.5.2 抽取内容的多面展示 .................................................................................74
3.6 理解搜索性能因素 .................................................................................................77
3.6.1 数量判定 .....................................................................................................77
3.6.2 判断数量 .....................................................................................................81
3.7 提高搜索性能 .........................................................................................................82
3.7.1 硬件改进 .....................................................................................................82
3.7.2 分析的改进 .................................................................................................83
3.7.3 提高查询性能 .............................................................................................85
3.7.4 其他评分模型 .............................................................................................88
3.7.5 提升Solr性能的技术 ..................................................................................89
3.8 其他搜索工具 .........................................................................................................91
3.9 小结 .........................................................................................................................93
3.10 相关资源 ...............................................................................................................93
第4章 模糊字符串匹配 ..........................................................................94
4.1 模糊字符串匹配方法 .............................................................................................96
4.1.1 字符重合度度量方法 .................................................................................96
4.1.2 编辑距离 .....................................................................................................99
4.1.3 n元组编辑距离 .........................................................................................102
4.2 寻找模糊匹配串 ...................................................................................................105
4.2.1 在Solr中使用前缀来匹配 ........................................................................105
4.2.2 利用trie树进行前缀匹配 .........................................................................106
4.2.3 使用n元组进行匹配 ..................................................................................111
4.3 构建模糊串匹配应用 ...........................................................................................112
4.3.1 在搜索中加入提前输入功能 ...................................................................113
4.3.2 搜索中的查询拼写校正 ...........................................................................117
4.3.3 记录匹配 ...................................................................................................122
4.4 小结 .......................................................................................................................127
4.5 相关资源 ...............................................................................................................128
第5章 命名实体识别 ...........................................................................129
5.1 命名实体的识别方法 ...........................................................................................131
5.1.1 基于规则的实体识别 ...............................................................................131
5.1.2 基于统计分类器的实体识别 ...................................................................132
5.2 基于OpenNLP的基本实体识别 ...........................................................................133
5.2.1 利用OpenNLP寻找人名 ...........................................................................134
5.2.2 OpenNLP识别的实体解读 .......................................................................136
5.2.3 基于概率过滤实体 ...................................................................................137
5.3 利用OpenNLP进行深度命名实体识别 ...............................................................137
5.3.1 利用OpenNLP识别多种实体类型 ...........................................................138
5.3.2 OpenNLP识别实体的背后机理 ...............................................................141
5.4 OpenNLP的性能 ...................................................................................................143
5.4.1 结果的质量 ...............................................................................................144
5.4.2 运行性能 ...................................................................................................145
5.4.3 OpenNLP的内存使用 ...............................................................................146
5.5 对新领域定制OpenNLP实体识别 .......................................................................147
5.5.1 训练模型的原因和方法 ...........................................................................147
5.5.2 训练OpenNLP模型 ...................................................................................148
5.5.3 改变建模输入 ...........................................................................................150
5.5.4 对实体建模的新方法 ...............................................................................152
5.6 小结 .......................................................................................................................154
5.7 进一步阅读材料 ...................................................................................................155
第6章 文本聚类 ..................................................................................156
6.1 Google News中的文档聚类 .................................................................................157
6.2 聚类基础 ...............................................................................................................158
6.2.1 三种聚类的文本类型 ...............................................................................158
6.2.2 选择聚类算法 ...........................................................................................160
6.2.3 确定相似度 ...............................................................................................161
6.2.4 给聚类结果打标签 ...................................................................................162
6.2.5 聚类结果的评估 .......................................................................................163
6.3 搭建一个简单的聚类应用 ...................................................................................165
6.4 利用Carrot2对搜索结果聚类 ...............................................................................166
6.4.1 使用Carrot2API ........................................................................................166
6.4.2 使用Carrot2对Solr的搜索结果聚类 ........................................................168
6.5 利用Apache Mahout对文档集聚类 ......................................................................171
6.5.1 对聚类的数据进行预处理 .......................................................................172
6.5.2 K-means聚类 ............................................................................................175
6.6 利用Apache Mahout进行主题建模 ......................................................................180
6.7 考察聚类性能 .......................................................................................................183
6.7.1 特征选择与特征约简 ...............................................................................183
6.7.2 Carrot2的性能和质量 ...............................................................................186
6.7.3 Mahout基准聚类算法 ..............................................................................187
6.8 致谢 .......................................................................................................................192
6.9 小结 .......................................................................................................................192
6.10 参考文献 .............................................................................................................193
第7章 分类及标注 ...............................................................................195
7.1 分类及归类概述 ...................................................................................................197
7.2 分类过程 ...............................................................................................................200
7.2.1 选择分类机制 ...........................................................................................201
7.2.2 识别文本分类中的特征 ...........................................................................202
7.2.3 训练数据的重要性 ...................................................................................203
7.2.4 评估分类器性能 .......................................................................................206
7.2.5 将分类器部署到生产环境 .......................................................................208
7.3 利用Apache Lucene构建文档分类器 ..................................................................209
7.3.1 利用Lucene对文本进行分类 ...................................................................210
7.3.2 为MoreLikeThis分类器准备训练数据 ....................................................212
7.3.3 训练MoreLikeThis分类器 ........................................................................214
7.3.4 利用MoreLikeThis分类器对文档进行分类 ............................................217
7.3.5 测试MoreLikeThis分类器 ........................................................................220
7.3.6 将MoreLikeThis投入生产环境 ................................................................223
7.4 利用Apache Mahout训练朴素贝叶斯分类器 ......................................................223
7.4.1 利用朴素贝叶斯算法进行文本分类 .......................................................224
7.4.2 准备训练数据 ...........................................................................................225
7.4.3 留存测试数据 ...........................................................................................229
7.4.4 训练分类器 ...............................................................................................229
7.4.5 测试分类器 ...............................................................................................231
7.4.6 改进自举过程 ...........................................................................................232
7.4.7 将Mahout贝叶斯分类器集成到Solr ........................................................234
7.5 利用OpenNLP进行文档分类 ...............................................................................238
7.5.1 回归模型及最大熵文档分类 ...................................................................239
7.5.2 为最大熵文档分类器准备训练数据 .......................................................241
7.5.3 训练最大熵文档分类器 ...........................................................................242
7.5.4 测试最大熵文档分类器 ...........................................................................248
7.5.5 生产环境下的最大熵文档分类器 ...........................................................249
7.6 利用Apache Solr构建标签推荐系统 ...................................................................250
7.6.1 为标签推荐收集训练数据 .......................................................................253
7.6.2 准备训练数据 ...........................................................................................255
7.6.3 训练Solr标签推荐系统 ............................................................................256
7.6.4 构建推荐标签 ...........................................................................................258
7.6.5 对标签推荐系统进行评估 .......................................................................261
7.7 小结 .......................................................................................................................263
7.8 参考文献 ...............................................................................................................265
第8章 构建示例问答系统 ....................................................................266
8.1 问答系统基础知识 ...............................................................................................268
8.2 安装并运行QA代码 .............................................................................................270
8.3 一个示例问答系统的架构 ...................................................................................271
8.4 理解问题并产生答案 ...........................................................................................274
8.4.1 训练答案类型分类器 ...............................................................................275
8.4.2 对查询进行组块分析 ...............................................................................279
8.4.3 计算答案类型 ...........................................................................................280
8.4.4 生成查询 ...................................................................................................283
8.4.5 对候选段落排序 .......................................................................................285
8.5 改进系统的步骤 ...................................................................................................287
8.6 本章小结 ...............................................................................................................287
8.7 相关资源 ...............................................................................................................288
第9章 未驾驭的文本：探索未来前沿 ..................................................289
9.1 语义、篇章和语用：探索高级NLP ....................................................................290
9.1.1 语义 ...........................................................................................................291
9.1.2 篇章 ...........................................................................................................292
9.1.3 语用 ...........................................................................................................294
9.2 文档及文档集自动摘要 .......................................................................................295
9.3 关系抽取 ...............................................................................................................298
9.3.1 关系抽取方法综述 ...................................................................................299
9.3.2 评估 ...........................................................................................................302
9.3.3 关系抽取工具 ...........................................................................................303
9.4 识别重要内容和人物 ...........................................................................................303
9.4.1 全局重要性及权威度 ...............................................................................304
9.4.2 个人重要性 ...............................................................................................305
9.4.3 与重要性相关的资源及位置 ...................................................................306
9.5 通过情感分析来探测情感 ...................................................................................306
9.5.1 历史及综述 ...............................................................................................307
9.5.2 工具及数据需求 .......................................................................................308
9.5.3 一个基本的极性算法 ...............................................................................309
9.5.4 高级话题 ...................................................................................................311
9.5.5 用于情感分析的开源库 ...........................................................................312
9.6 跨语言检索 ...........................................................................................................313
9.7 本章小结 ...............................................................................................................315
9.8 相关资源 ...............................................................................................................315
· · · · · · (
收起)
