NLP

NLP 基础概念全景图

• 重点内容：
• 什么是 NLP，NLP 应用场景（情感分析、命名实体识别、聊天机器人、法规文档处理等）
• NLP vs 计算机视觉 vs Tabular ML 的不同
• 传统 NLP vs 深度学习 NLP（BoW/TF-IDF vs Word2Vec/BERT）

文本预处理核心方法

• 重点内容：
• Tokenization（分词）
• Stopwords removal（去除停用词）
• Lemmatization vs Stemming（词干还原 vs 词形还原）
• Lowercasing、去标点、处理 HTML、表情符号

• Python实战（用 nltk/spacy）
• 用 nltk 对金融新闻进行清洗和分词
• 输出一篇金融报告的高频词云

---

文本预处理核心方法（NLP基础）

技术	说明	举例/方法
Tokenization	将文本切分为一个个词/子词	nltk.word_tokenize()
Stopwords Removal	去除如 "the", "is", "and" 等无实际语义的高频词	from nltk.corpus import stopwords
Lowercasing	转小写，统一格式	'Finance' → 'finance'
Punctuation Removal	去除标点符号	re.sub(r'[^\w\s]', '', text)
HTML Cleaning	去除网页文本中的 <p> 等标签	from bs4 import BeautifulSoup
Emoji Removal	去除表情符号	re.sub(emoji_pattern, '', text)
Stemming	提取词干（如 dealing → deal）	from nltk.stem import PorterStemmer
Lemmatization	词形还原（如 better → good）	from nltk.stem import WordNetLemmatizer

Python 实战：清洗 + 分词（使用 nltk）

🔧 1. 安装必要包（仅第一次执行）

pip install nltk wordcloud beautifulsoup4

📥 2. 下载资源包（nltk）

import nltk
nltk.download('punkt')
nltk.download('stopwords')
nltk.download('wordnet')

---

📰 3. 金融新闻清洗与分词

import re
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
from nltk.stem import WordNetLemmatizer
from bs4 import BeautifulSoup

# 示例金融文本（你可以替换为真实新闻数据）
text = """<p>Stocks rallied today as the Federal Reserve signaled a potential pause in interest rate hikes.
Markets reacted positively to the dovish tone.</p> 🚀💵"""

# Step 1: 去除 HTML 和表情符号
soup = BeautifulSoup(text, "html.parser")
text = soup.get_text()
text = re.sub(r'[^\w\s]', '', text)  # 去标点
text = re.sub(r'\d+', '', text)      # 去数字
text = text.lower()                  # 小写
text = re.sub(r'[^\x00-\x7F]+', '', text)  # 去除非ASCII字符（emoji）

# Step 2: 分词 + 去停用词 + 词形还原
tokens = word_tokenize(text)
stop_words = set(stopwords.words('english'))
lemmatizer = WordNetLemmatizer()

cleaned_tokens = [lemmatizer.lemmatize(w) for w in tokens if w not in stop_words and len(w) > 2]

print("🧹 Clean Tokens:\n", cleaned_tokens)

---

☁️ 4. 输出高频词云图（WordCloud）

from wordcloud import WordCloud
import matplotlib.pyplot as plt

# 合并所有词汇为一个字符串
clean_text = ' '.join(cleaned_tokens)

# 生成词云
wordcloud = WordCloud(width=800, height=400, background_color='white').generate(clean_text)

# 可视化
plt.figure(figsize=(10, 5))
plt.imshow(wordcloud, interpolation='bilinear')
plt.axis('off')
plt.title("💼 Financial News Word Cloud", fontsize=16)
plt.show()

---

✅ 最终效果

你会得到：

• 一份干净的词汇表 cleaned_tokens

• 一张金融词云图，展示新闻中出现频率最高的关键词（如：fed, market, rate, pause, inflation）

NLP 常用正则表达式大全（含金融实战场景）

功能	正则表达式	说明	示例
移除HTML标签	<.*?>	删除 <p>, <div> 等标签	re.sub(r'<.*?>', '', text)
去除标点符号	[^\w\s]	保留字母、数字、空格	re.sub(r'[^\w\s]', '', text)
去除数字	\d+	匹配所有数字	re.sub(r'\d+', '', text)
去除非ASCII字符	[^\x00-\x7F]+	处理 emoji、中文、特殊符号	re.sub(r'[^\x00-\x7F]+', '', text)
保留字母数字组合	\b[A-Za-z0-9]+\b	提取词语如 risk123	re.findall(r'\b[A-Za-z0-9]+\b', text)
检测 Email	[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}	提取邮箱地址	re.findall(...)
检测 URL	https?://[^\s]+	提取网页链接	re.findall(...)
提取日期（简化版）	\d{4}-\d{2}-\d{2}	匹配如 2023-10-01	re.findall(...)
匹配特定金融词	`\b(risk	loan	default
多个空格合并成一个	\s+	用一个空格替换多个空格/tab	re.sub(r'\s+', ' ', text)
只保留字母	[^a-zA-Z\s]	删除数字、符号，仅保留英文单词	re.sub(r'[^a-zA-Z\s]', '', text)

金融文本清洗函数库

import re
import pandas as pd

class FinancialTextCleaner:
    def __init__(self):
        pass

    def clean_html(self, text):
        return re.sub(r'<.*?>', '', text)

    def remove_non_ascii(self, text):
        return re.sub(r'[^        return re.sub(r'[^\x00-        return re.sub(r'[^\x00-\x7F]+', '', text)

    def remove_digits(self, text):
        return re.sub(r'\d+', '', text)

    def remove_punctuation(self, text):
        return re.sub(r'[^\w\s]', '', text)

    def remove_multiple_spaces(self, text):
        return re.sub(r'\s+', ' ', text).strip()

    def lower_case(self, text):
        return text.lower()

    def clean_text(self, text):
        text = self.clean_html(text)
        text = self.remove_non_ascii(text)
        text = self.remove_digits(text)
        text = self.remove_punctuation(text)
        text = self.remove_multiple_spaces(text)
        text = self.lower_case(text)
        return text

    def clean_dataframe_column(self, df, column_name, new_column_name=None):
        if new_column_name is None:
            new_column_name = column_name + '_cleaned'
        df[new_column_name] = df[column_name].astype(str).apply(self.clean_text)
        return df

    def extract_keywords(self, text, keywords):
        pattern = r'\\b(' + '|'.join(map(re.escape, keywords)) + r')\\b'
        return re.findall(pattern, text.lower())

    def extract_money_amounts(self, text):
        return re.findall(r'\$\d+(\.\d+)?\s?(million|billion)?', text.lower())

    def extract_dates(self, text):
        return re.findall(r'\d{4}-\d{2}-\d{2}', text)

    def extract_ratings(self, text):
        return re.findall(r'(AAA|AA\+?|A\+?|BBB\+?|BB\+?|B\+?)', text)


# Example usage:
if __name__ == "__main__":
    cleaner = FinancialTextCleaner()
    sample_text = "<p>As of 2023-12-31, the Company had $25.3 million in credit risk exposure. 📉</p>"
    print("Cleaned Text:", cleaner.clean_text(sample_text))
    print("Keywords:", cleaner.extract_keywords(sample_text, ['credit', 'risk', 'exposure']))
    print("Amounts:", cleaner.extract_money_amounts(sample_text))
    print("Dates:", cleaner.extract_dates(sample_text))
    print("Ratings:", cleaner.extract_ratings("The firm was rated AAA and later downgraded to BBB+"))

文本表示方法（BoW、TF-IDF、词向量）

docs = [
    "The interest rate will rise next quarter due to inflation pressure.",
    "The central bank is expected to keep the interest rate unchanged.",
    "Stock market volatility increased following policy changes."
]

from sklearn.feature_extraction.text import CountVectorizer

bow_vectorizer = CountVectorizer()
bow_matrix = bow_vectorizer.fit_transform(docs)

print("BoW Feature Names:\n", bow_vectorizer.get_feature_names_out())
print("BoW Matrix:\n", bow_matrix.toarray())

from sklearn.feature_extraction.text import TfidfVectorizer

tfidf_vectorizer = TfidfVectorizer()
tfidf_matrix = tfidf_vectorizer.fit_transform(docs)

print("TF-IDF Feature Names:\n", tfidf_vectorizer.get_feature_names_out())
print("TF-IDF Matrix:\n", tfidf_matrix.toarray())

import gensim.downloader as api
from gensim.utils import simple_preprocess
import numpy as np

model = api.load("word2vec-google-news-300")  # 需要联网，建议本地已有模型

def get_avg_vector(sentence):
    tokens = simple_preprocess(sentence)
    vectors = [model[word] for word in tokens if word in model]
    return np.mean(vectors, axis=0) if vectors else np.zeros(300)

word2vec_matrix = np.array([get_avg_vector(doc) for doc in docs])
print("Word2Vec Shape:", word2vec_matrix.shape)

Classic NLP Models

应用层级结构

text
复制编辑
输入文本 → 文本清洗 → 向量表示（BoW / TF-IDF）→ 训练分类模型（Naive Bayes / SVM / LR）→ 预测情感 / 主题

Sentiment Analysis

Raw Text → Cleaned Text → Feature Vectors → Model → Sentiment Label