SoFunction
Updated on 2025-03-04

Python uses Transformers to implement machine translation functions

introduction

In recent years, machine translation technology has developed rapidly. From traditional rule-based translation to statistical machine translation, to the popular neural network translation model today, especially the model based on Transformer architecture, the translation effect has made a qualitative leap. The Transformers library is launched by Hugging Face and is one of the most popular natural language processing libraries. It provides a variety of pre-trained language models that can be used for tasks such as text classification, text generation, machine translation, etc. This article will explain in detail how to use the Transformers library to implement a machine translation model.

1. Preparation

Before you start, make sure you have the Transformers library and the PyTorch or TensorFlow framework installed. The following are the installation commands:

pip install transformers torch

2. Select the model and data set

The Transformers library provides a variety of pre-trained models for machine translation, such as:

  • Helsinki-NLP/opus-mt-* Series: Covering multiple language pairs.
  • facebook/wmt19-* series: Model based on WMT19 dataset.

You can choose the appropriate model by accessing Hugging Face's model library. For example, if you want to implement English to French translation, you can use the Helsinki-NLP/opus-mt-en-fr model.

3. Implement machine translation steps

1. Load pretrained models and word participle

First, load the translation model and word participle from the Transformers library. Word participle is used to convert text into input formats that the model can understand.

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

# Select a modelmodel_name = "Helsinki-NLP/opus-mt-en-fr"  # English to French translation modeltokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)

Here we choseHelsinki-NLP/opus-mt-en-frModel for translation of English into French. For different language pairs, just select different models.

2. Write translation functions

On this basis, we can write a simple translation function that translates the input text into the target language. This function will encode the input text using a word participle, pass the encoded text to the model, and then decode the output of the model to generate the translated text.

def translate(text, tokenizer, model):
    # Encode the input text into the model input format    inputs = (text, return_tensors="pt", truncation=True)
    # Generate translation using the model    outputs = (inputs, max_length=40, num_beams=4, early_stopping=True)
    # Decode the generated tensor as text    translated_text = (outputs[0], skip_special_tokens=True)
    return translated_text

3. Conduct a test

After writing the translation function, we can test some English sentences to see the translation effect of the model.

english_text = "Hello, how are you?"
translated_text = translate(english_text, tokenizer, model)
print("Translated text:", translated_text)

After running the code, you will get a translated French text.

4. Adjust the translation effect

In machine translation, the quality of the generated translation text may be affected by the generated parameters. existIn the method, you can optimize the effect by adjusting the following parameters:

  • max_length: Controls the maximum length of the generated translated text to prevent the text from being too long.
  • num_beams: Set the size of beam search. Larger values ​​can improve translation quality, but increase calculation volume.
  • early_stopping: Set toTrueThe generation process can be stopped at the right time.

For example, you cannum_beamsSet to 8 to improve translation results or reducemax_lengthTo speed up generation.

outputs = (inputs, max_length=50, num_beams=8, early_stopping=True)

5. Batch translation and post-processing

If there are multiple texts that need to be translated, you can use batch translation to improve efficiency. At the same time, sometimes the output of the model may contain redundant punctuation marks or spaces, which can be post-processed after the output.

Batch translation

def batch_translate(texts, tokenizer, model):
    inputs = tokenizer(texts, return_tensors="pt", padding=True, truncation=True)
    outputs = (**inputs, max_length=40, num_beams=4, early_stopping=True)
    return [(output, skip_special_tokens=True) for output in outputs]

Post-processing

Sometimes the output of the model may contain extra spaces or punctuation marks, which can be cleaned up after generation:

import re

def clean_translation(text):
    # Remove excess spaces    text = (r"\s+", " ", text)
    # Remove extra punctuation marks at the end of the sentence    text = (r"\s([?.!"](?:\s|$))", r"\1", text)
    return text

VI. Other advanced operations

1. Use a custom vocabulary

In certain professional fields (such as law, medicine, etc.), specific vocabulary is required. Transformers supports loading custom vocabulary to enhance the professionalism of translations.

2. Fine-tune the model

If the existing pretrained models cannot meet the needs of a specific task, the model can be fine-tuned with a small amount of domain-specific data to improve the translation effect. Hugging Face provides a Trainer class that allows for easy fine-tuning operations.

7. Suggestions

The above describes how to use the Transformers library to quickly build a machine translation system and use a pre-trained translation model to implement the English to French translation function. For situations where you need to translate other languages, just replace the appropriate model. By appropriately adjusting parameters and performing post-processing, the translation effect can be further improved. If there are higher requirements, the model can also be fine-tuned and trained to suit specific areas.

8. Complete code example

For easy understanding and application, here is a complete code example, from the loading of the model to the processing of translated text. The code also includes batch translation and simple post-processing, which is easy for you to use in actual projects.

import re
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

# 1. Loading the model and word participlemodel_name = "Helsinki-NLP/opus-mt-en-fr"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)

# 2. Define translation functionsdef translate(text, tokenizer, model):
    inputs = (text, return_tensors="pt", truncation=True)
    outputs = (inputs, max_length=50, num_beams=8, early_stopping=True)
    translated_text = (outputs[0], skip_special_tokens=True)
    return clean_translation(translated_text)

# 3. Define batch translation functionsdef batch_translate(texts, tokenizer, model):
    inputs = tokenizer(texts, return_tensors="pt", padding=True, truncation=True)
    outputs = (**inputs, max_length=50, num_beams=8, early_stopping=True)
    return [clean_translation((output, skip_special_tokens=True)) for output in outputs]

# 4. Define the translated post-processing functiondef clean_translation(text):
    text = (r"\s+", " ", text)  # Remove excess spaces    text = (r"\s([?.!"](?:\s|$))", r"\1", text)  # Remove extra spaces at the end of the sentence    return text

# Test individual translationsenglish_text = "Hello, how are you?"
translated_text = translate(english_text, tokenizer, model)
print("Single translation result:", translated_text)

# Test batch translationtexts = ["Hello, how are you?", "I am learning machine translation.", "Transformers library is amazing!"]
translated_texts = batch_translate(texts, tokenizer, model)
print("Batch translation results:", translated_texts)

9. Challenges and future developments of machine translation

Although using the Transformers library can quickly build a translation system, the effectiveness of machine translation is limited by many factors:

  1. Model limitations: Pre-trained general models may not translate accurately enough complex syntactic structures or domain-specific vocabulary.
  2. Data quality: The translation effect of the model is closely related to the quality of the training data. Multilingual models have limited effectiveness when translating certain low-resource languages.
  3. Long text processing: Existing models may experience text incoherence and missed information when translating long text.

Future development direction

As research deepens, machine translation is still evolving, and there are several key directions in the future that may bring better translation results:

  • Large-scale pre-training multitasking model: For example, multilingual and multitasking pre-training can enable the model to be better generalized and improve the translation effect of low-resource languages.
  • Small sample fine adjustment: Fine-tuning the model through a small amount of specific domain data, its performance in specific domains can be enhanced.
  • Enhance language context understanding: Combined with the latest developments in deep learning (such as context perception, graph neural networks, etc.), it may allow machines to better understand the context.

10. Summary

Using the Transformers library for machine translation is relatively simple and effective, and is especially suitable for quickly building and testing translation functions in projects. Through this tutorial, you can easily access the basic implementation of mobile phone translation and understand how to optimize the generated translation. In the future, with the continuous development of natural language processing technology, the application prospects of machine translation will be broader. Hope this article helps you achieve a smoother translation experience in your project!

The above is the detailed content of Python using Transformers to implement machine translation functions. For more information about Python Transformers machine translation, please follow my other related articles!