Python develops text summary system based on natural language processing

1. Project Overview

Natural language processing (NLP) is an important research direction in the field of artificial intelligence, and text abstracts, as an important application of NLP, are of great significance in the era of information explosion. This project aims to develop a Python-based text summary system that can automatically extract key information from long texts and generate concise and comprehensive summary, helping users quickly obtain the core content of the document.

1.1 Project background

With the development of the Internet, people face massive text information every day, such as news reports, academic papers, product reviews, etc. It becomes a challenge to quickly get the core content of this information. Text summary technology can automatically analyze long texts, extract key information, and generate concise summary, greatly improving the efficiency of information acquisition.

1.2 Project Objectives

Develop a summary system that can handle Chinese and English texts

Supports two methods: extracted digest and generated digest

Provides a web interface for user convenience

Supports input in multiple text formats (TXT, PDF, Word, etc.)

Provide summary quality assessment function

1.3 Technical Route

This project adopts Python as the main development language, combines multiple NLP libraries and deep learning frameworks to realize the text summary function. The main technical routes include:

Traditional NLP method: Extracted abstract based on TF-IDF, TextRank and other algorithms

Deep learning method: Generative abstract based on Seq2Seq, Transformer and other models

Pre-trained model: Use pre-trained models such as BERT and GPT to improve the quality of the summary

2. System design

2.1 System Architecture

The text summary system adopts a modular design, mainly including the following modules:

• Data preprocessing module: responsible for text cleaning, word segmentation, stop word removal and other preprocessing work
• Abstract generation module: contains two submodules: extracted abstract and generated abstract.
• Evaluation module: Responsible for quality assessment of generated summary
• Web interface module: Provides user-friendly interactive interface
• File processing module: supports reading and processing of files in multiple formats

The system architecture diagram is as follows:

+------------------+     +------------------+     +------------------+
|                  |     |                  |     |                  |
| File Processing Module     |---->| Data Preprocessing Module |---->| Abstract Generation Module     |
|                  |     |                  |     |                  |
+------------------+     +------------------+     +--------|---------+
                                                          |
                                                          v
+------------------+     +------------------+     +------------------+
|                  |     |                  |     |                  |
|    Web interface module  |<----|    Evaluation module     |<-----|  Summary result output    |
|                  |     |                  |     |                  |
+------------------+     +------------------+     +------------------+

2.2 Module design

2.2.1 Data preprocessing module

The data preprocessing module is mainly responsible for cleaning and standardizing the input text, including:

• Text cleaning: Remove HTML tags, special characters, etc.
• Text participle: Use jieba (Chinese) or NLTK (English) for word segmentation
• Go to stop words: Remove common stop words such as "the", "yes", "the", "is", etc.
• Part-of-speech marking: marking the part-of-speech of words to provide support for subsequent processing
• Sentence segmentation: dividing text into sentence units

2.2.2 Summary Generation Module

The summary generation module is the core of the system and contains two summary methods:

Extraction summary:

• TF-IDF method: Calculate sentence importance based on word frequency-inverse document frequency
• TextRank algorithm: Use graph algorithm to calculate sentence importance
• LSA (Latent Semantic Analysis): Extracting text topics using matrix decomposition

Generative summary:

• Seq2Seq Model: Generate summary using encoder-decoder architecture
• Transformer model: Using self-attention mechanism to improve summary quality
• Pre-trained model fine-tuning: fine-tuning based on pre-trained models such as BERT and GPT

2.2.3 Evaluation module

The evaluation module is responsible for quality evaluation of the generated summary, mainly including:

• ROUGE score: Calculate the overlap between the generated summary and the reference summary
• BLEU Rating: Assess the fluency and accuracy of generated summary
• Manual evaluation interface: supports users to evaluate the quality of the summary

2.2.4 Web interface module

The web interface module provides user-friendly interactive interface, and its main functions include:

• Text input: Supports direct text input or uploading files
• Parameter settings: Allow users to set parameters such as summary length, algorithm selection, etc.
• Result display: Display the generated summary results
• Evaluation Feedback: Allows users to evaluate the quality of the summary

2.2.5 File Processing Module

The file processing module supports the reading and processing of files in multiple formats, including:

• TXT file: directly read text content
• PDF file: Extract text using PyPDF2 or pdfminer
• Word file: Extract text using python-docx
• HTML file: Extract text content using BeautifulSoup

3. System implementation

3.1 Development Environment

Operating system: Windows/Linux/MacOS

Programming Language: Python 3.8+

Main dependency library:

NLP processing: NLTK, jieba, spaCy

Deep Learning: PyTorch, Transformers

Web Framework: Flask

File processing: PyPDF2, python-docx, BeautifulSoup

Data processing: NumPy, Pandas

3.2 Core algorithm implementation

3.2.1 TextRank algorithm implementation

TextRank is a graph-based sorting algorithm similar to Google's PageRank algorithm. In the text summary, we treat each sentence as a node in the graph, and the similarity between sentences is the weight of the edges.

def textrank_summarize(text, ratio=0.2):
    """
    useTextRankAlgorithm generates text summary
    
    parameter:
        text (str): Enter text
        ratio (float): The percentage of abstracts to original text
        
    return:
        str: Generated summary
    """
    # Text preprocessing    sentences = text_to_sentences(text)
    
    # Construct a sentence similarity matrix    similarity_matrix = build_similarity_matrix(sentences)
    
    # Calculate TextRank scores using NetworkX library    import networkx as nx
    nx_graph = nx.from_numpy_array(similarity_matrix)
    scores = (nx_graph)
    
    # Choose important sentences based on scores    ranked_sentences = sorted(((scores[i], s) for i, s in enumerate(sentences)), reverse=True)
    
    # Choose the number of sentences based on the proportion    select_length = int(len(sentences) * ratio)
    
    # Order the selected sentences in the original order    selected_sentences = sorted(
        [ranked_sentences[i][1] for i in range(select_length)],
        key=lambda s: (s))
    
    # Generate summary    summary = ' '.join(selected_sentences)
    
    return summary

3.2.2 Seq2Seq model implementation

The Seq2Seq (sequence to sequence) model is a generative summary method based on neural networks, including two parts: encoder and decoder.

import torch
import  as nn
import  as optim

class Encoder():
    def __init__(self, input_dim, emb_dim, hid_dim, n_layers, dropout):
        super().__init__()
         = (input_dim, emb_dim)
         = (emb_dim, hid_dim, n_layers, dropout=dropout)
         = (dropout)
        
    def forward(self, src):
        # src = [src_len, batch_size]
        embedded = ((src))
        # embedded = [src_len, batch_size, emb_dim]
        outputs, (hidden, cell) = (embedded)
        # outputs = [src_len, batch_size, hid_dim * n_directions]
        # hidden = [n_layers * n_directions, batch_size, hid_dim]
        # cell = [n_layers * n_directions, batch_size, hid_dim]
        return hidden, cell

class Decoder():
    def __init__(self, output_dim, emb_dim, hid_dim, n_layers, dropout):
        super().__init__()
        self.output_dim = output_dim
         = (output_dim, emb_dim)
         = (emb_dim, hid_dim, n_layers, dropout=dropout)
        self.fc_out = (hid_dim, output_dim)
         = (dropout)
        
    def forward(self, input, hidden, cell):
        # input = [batch_size]
        # hidden = [n_layers * n_directions, batch_size, hid_dim]
        # cell = [n_layers * n_directions, batch_size, hid_dim]
        
        input = (0)
        # input = [1, batch_size]
        
        embedded = ((input))
        # embedded = [1, batch_size, emb_dim]
        
        output, (hidden, cell) = (embedded, (hidden, cell))
        # output = [1, batch_size, hid_dim * n_directions]
        # hidden = [n_layers * n_directions, batch_size, hid_dim]
        # cell = [n_layers * n_directions, batch_size, hid_dim]
        
        prediction = self.fc_out((0))
        # prediction = [batch_size, output_dim]
        
        return prediction, hidden, cell

​​​​​​​class Seq2Seq():
    def __init__(self, encoder, decoder, device):
        super().__init__()
         = encoder
         = decoder
         = device
        
    def forward(self, src, trg, teacher_forcing_ratio=0.5):
        # src = [src_len, batch_size]
        # trg = [trg_len, batch_size]
        
        batch_size = [1]
        trg_len = [0]
        trg_vocab_size = .output_dim
        
        # Store prediction results for each step        outputs = (trg_len, batch_size, trg_vocab_size).to()
        
        # Encoder forward propagation        hidden, cell = (src)
        
        # The first input is the <SOS> tag        input = trg[0,:]
        
        for t in range(1, trg_len):
            # Decoder forward propagation            output, hidden, cell = (input, hidden, cell)
            
            #Storing prediction results            outputs[t] = output
            
            # Decide whether to use teacher forcing            teacher_force = () &lt; teacher_forcing_ratio
            
            # Get the most likely word            top1 = (1)
            
            # If you use teacher forcing, the next input is the real tag            # Otherwise, use the model to predict the results            input = trg[t] if teacher_force else top1
            
        return outputs

3.2.3 Abstract implementation based on Transformer

Use Hugging Face's Transformers library to implement the summary function based on pretrained models:

from transformers import pipeline

​​​​​​​def transformer_summarize(text, max_length=150, min_length=30):
    """
    Use pre-trainedTransformerModel generation summary
    
    parameter:
        text (str): Enter text
        max_length (int): Summary Maximum Length
        min_length (int): Summary Minimum Length
        
    return:
        str: Generated summary
    """
    # Initialize summary pipeline    summarizer = pipeline("summarization", model="facebook/bart-large-cnn")
    
    # Generate summary    summary = summarizer(text, max_length=max_length, min_length=min_length, do_sample=False)
    
    return summary[0]['summary_text']

3.3 Web interface implementation

Implementing the web interface using the Flask framework:

from flask import Flask, render_template, request, jsonify
from  import secure_filename
import os
from summarizer import TextRankSummarizer, Seq2SeqSummarizer, TransformerSummarizer
from file_processor import process_file

app = Flask(__name__)
['UPLOAD_FOLDER'] = 'uploads/'
['MAX_CONTENT_LENGTH'] = 16 * 1024 * 1024  # Limit uploaded file size to 16MB
# Make sure the upload directory exists(['UPLOAD_FOLDER'], exist_ok=True)

@('/')
def index():
    return render_template('')

@('/summarize', methods=['POST'])
def summarize():
    # Get parameters    text = ('text', '')
    file = ('file')
    method = ('method', 'textrank')
    ratio = float(('ratio', 0.2))
    max_length = int(('max_length', 150))
    min_length = int(('min_length', 30))
    
    # If a file is uploaded, process the file contents    if file and  != '':
        filename = secure_filename()
        file_path = (['UPLOAD_FOLDER'], filename)
        (file_path)
        text = process_file(file_path)
        (file_path)  # Delete the file after processing is completed    
    # Check if the text is empty    if not text:
        return jsonify({'error': 'Please provide text content or upload files'}), 400
    
    # Generate summary according to the selected method    if method == 'textrank':
        summarizer = TextRankSummarizer()
        summary = (text, ratio=ratio)
    elif method == 'seq2seq':
        summarizer = Seq2SeqSummarizer()
        summary = (text, max_length=max_length)
    elif method == 'transformer':
        summarizer = TransformerSummarizer()
        summary = (text, max_length=max_length, min_length=min_length)
    else:
        return jsonify({'error': 'Unsupported summary method'}), 400
    
    return jsonify({'summary': summary})

​​​​​​​if __name__ == '__main__':
    (debug=True)

3.4 File processing module implementation

import os
import PyPDF2
import docx
from bs4 import BeautifulSoup

def process_file(file_path):
    """
    Process files according to file type，Extract text content
    
    parameter:
        file_path (str): File path
        
    return:
        str: Extracted text content
    """
    file_ext = (file_path)[1].lower()
    
    if file_ext == '.txt':
        return process_txt(file_path)
    elif file_ext == '.pdf':
        return process_pdf(file_path)
    elif file_ext == '.docx':
        return process_docx(file_path)
    elif file_ext in ['.html', '.htm']:
        return process_html(file_path)
    else:
        raise ValueError(f"Unsupported file types: {file_ext}")

def process_txt(file_path):
    """Processing TXT files"""
    with open(file_path, 'r', encoding='utf-8') as f:
        return ()

def process_pdf(file_path):
    """Processing PDF files"""
    text = ""
    with open(file_path, 'rb') as f:
        pdf_reader = (f)
        for page_num in range(len(pdf_reader.pages)):
            page = pdf_reader.pages[page_num]
            text += page.extract_text()
    return text

def process_docx(file_path):
    """Processing DOCX files"""
    doc = (file_path)
    text = ""
    for para in :
        text +=  + "\n"
    return text

​​​​​​​def process_html(file_path):
    """Processing HTML files"""
    with open(file_path, 'r', encoding='utf-8') as f:
        soup = BeautifulSoup((), '')
        # Remove script and style elements        for script in soup(["script", "style"]):
            ()
        # Get text        text = soup.get_text()
        # Handle extra whitespace characters        lines = (() for line in ())
        chunks = (() for line in lines for phrase in ("  "))
        text = '\n'.join(chunk for chunk in chunks if chunk)
    return text

4. System testing and evaluation

4.1 Test dataset

To evaluate the performance of the text summary system, we tested using the following dataset:

Chinese dataset:

• LCSTS (Large Scale Chinese Short Text Summarization) dataset
• News summary data set (collected from news websites such as Sina and NetEase)

English dataset:

• CNN/Daily Mail Dataset
• XSum dataset
• Reddit TIFU dataset

4.2 Evaluation indicators

We use the following metrics to evaluate the quality of the summary:

ROUGE（Recall-Oriented Understudy for Gisting Evaluation）：

• ROUGE-1: Overlapping of single words
• ROUGE-2: Overlapping of two consecutive words
• ROUGE-L: The longest common subsequence

BLEU（Bilingual Evaluation Understudy）：

Evaluate the n-gram exact match between generated text and reference text

Manual evaluation:

• Information Integrity: Whether the summary contains the main information of the original text
• Continuity: Whether the abstract is coherent and logically clear
• Readability: Is the summary easy to understand

4.3 Test results

Test results on LCSTS dataset:

Test results on CNN/Daily Mail dataset:

4.4 Performance Analysis

From the test results, we can see:

Generative summary vs extracted summary:

• Generative digests (Seq2Seq, Transformer) are better than extracted digests (TF-IDF, TextRank) in all indicators.
• Generative summary produces smoother, coherent text, and extracted summary sometimes has coherence problems

Performance of different models:

• Transformer-based models perform best, thanks to their powerful self-attention mechanism
• TextRank performs better in the extraction method and is suitable for scenarios with limited computing resources.

Differences in Chinese and English processing:

• The ROUGE-2 score of Chinese abstract is generally lower than in English, which may be related to the challenge of Chinese word segmentation
• English abstracts perform better in terms of coherence, which is related to language characteristics

5. System deployment and use

5.1 Deployment Requirements

Hardware requirements:

• CPU: 4 cores or more
• Memory: 8GB or more (more than 16GB is recommended when using deep learning models)
• Hard disk: 10GB free space

Software requirements:

• Python 3.8 or later
• Dependency library: See
• Operating system: Windows/Linux/MacOS

5.2 Installation steps

Cloning the project warehouse:

git clone /username/
cd text-summarization-system

Create a virtual environment:

python -m venv venv
source venv/bin/activate  # Linux/MacOS
venv\Scripts\activate  # Windows

Installation dependencies:

pip install -r 

Download the pretrained model (optional for generative summary):

python download_models.py

Start the web service:

python 

Visit the web interface:

Open in the browser http://localhost:5000

5.3 Instructions for use

Web interface use:

• Enter or paste the text you want to abstract in the text box
• Or upload files in TXT, PDF, Word, HTML formats
• Select summary method (TextRank, Seq2Seq, Transformer)
• Set summary parameters (proportion, length, etc.)
• Click the "Generate Summary" button
• View the generated summary results

Command line use:

python  --input  --method transformer --output 

API usage:

import requests

url = "http://localhost:5000/summarize"
data = {
    "text": "This is a long text that needs a summary...",
    "method": "transformer",
    "max_length": 150,
    "min_length": 30
}

response = (url, data=data)
summary = ()["summary"]
print(summary)

6. Project Summary and Prospect

6.1 Project Summary

This project successfully developed a Python-based text summary system, which has the following characteristics:

• Various abstract methods: support for extracted digests (TF-IDF, TextRank) and generated digests (Seq2Seq, Transformer)
• Multilingual support: Supports summary generation of Chinese and English texts
• Multi-format support: Supports TXT, PDF, Word, HTML and other file formats
• User-friendly interface: Provides web interface and API interface for user convenience
• High-quality abstract: Especially based on Transformer models, high-quality abstracts can be generated

6.2 Insufficient projects

Despite some results, the project still has the following shortcomings:

• Computational resource requirements: Deep learning models (especially Transformers) require higher computing resources
• Long text processing: For ultra-long text (such as the entire book), the system processing capability is limited
• Area-specific adaptation: For texts in specific areas (such as medicine, law), the quality of abstracts needs to be improved
• Multilingual support is limited: mainly supports Chinese and English, and support for other languages ​​is limited

6.3 Future Outlook

In the future, the system can be improved from the following aspects:

Model optimization:

• Introduce more advanced pre-trained models (such as T5, BART)
• Optimize model parameters and reduce computing resource requirements
• Explore model distillation technology to improve inference speed

Feature extension:

• Text summary that supports more languages
• Added multi-document summary function
• Added keyword extraction and topic analysis functions

Improved user experience:

• Optimize the web interface to provide a more friendly user experience
• Add batch processing function
• Provide summary results comparison function

Field adaptation:

• Training special abstract models for specific fields (such as medicine, law, technology)
• Add domain knowledge base and improve the quality of summary of professional texts

This is the end of this article about Python's text summary system based on natural language processing. For more related content of Python's natural language processing text summary, please search for my previous articles or continue browsing the related articles below. I hope everyone will support me in the future!