SoFunction
Updated on 2025-03-02

Golang developed string and slice problem record

background

In the project, we use mysql to store data information, where the label table records the tag-related information. The table structure is as follows:

CREATE TABLE `label` (
  `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT COMMENT 'id',
  `name` varchar(190) COLLATE utf8mb4_unicode_ci DEFAULT 'label name',
  PRIMARY KEY (`id`),
  UNIQUE KEY `uniq_n` (`name`)
) ENGINE=InnoDB AUTO_INCREMENT=7050965 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci COMMENT='label'

Where the name field is varchar type, 190 representsCharacter length, and is the only index

Why does the name field set the maximum character length to 190?

name is type varchar and is the unique index. MySQL stipulates that when varchar type is indexed, the maximum length is 767 bytes

The encoding format of the name field is utf8mb4_unicode_ci. A character is represented by up to 4 bytes. 767 / 4 = 191.75, so you only need to limit the maximum character length to be less than 191.75, and take 190 as the limit in the project.

The business code logic is very simple, and there are two main steps:

  • Query the label table by name. If the label information is found, it will be returned directly.
  • If no label information is found, try to create label information

pseudocode:

const (
   // Maximum character length   LabelNameLengthLimit = 190
)
func GetOrCreateLabel(ctx , name string) (label , err error) {
   var err error
   // Query label information based on name   label, err = db_reader.GetLabel(name)
   // No corresponding label information was found and the string length exceeded 190, then query again after slicing   if IsRecordNotFoundError(err) && len(name) > LabelNameLengthLimit {
      label, err = db_reader.GetLabel(name[:LabelNameLengthLimit])
   }
   // Or an error is reported, try to create label information   if err != nil {
       = name
      err = db_writer.CreateLabel(ctx, &label)
      // Report a unique key conflict error, which may be due to the problem caused by concurrent creation. Please provide a guarantee to query again      if IsKeyConflict(err) {
         label, err = db_reader.GetLabel(ctx, name)
         if err != nil {
            return label, err
         }
      }
      if err != nil {
         return label, err
      }
   }
   return label, nil
}

question

Some cases, the first execution of business code is successful, and the subsequent execution of business code is constantly reported.

case1：

labelName := "© 2015 Charanga Sí o Ké - Chema Muñoz, Perfecto Artola, Pablo Guerrero, Antonio Flores, Los Gipsy King, Chambao, Adele, La Pegatina, El Chaval de la Peca, Los Manolos © 2015 Charanga Sí o Ké - Sones de Rumba"

implement:

  • The first time the business code is executed successfully
  • Subsequent execution of business code, 22 lines of stable reproduction report error

analyze

When executing business code in the future

  • According to name, the corresponding label information cannot be checked, and the error of err is reported RecordNotFoundError
  • According to name[:LabelNameLengthLimit], the corresponding label information cannot be checked, and the error ReportNotFoundError is reported by err
var err error
// Query label information based on namelabel, err = db_reader.GetLabel(name)
// No corresponding label information was found and the string length exceeded 190, then query again after slicingif IsRecordNotFoundError(err) && len(name) > LabelNameLengthLimit {
   label, err = db_reader.GetLabel(name[:LabelNameLengthLimit])
}

When err != nil, try to create label information, and a unique key conflict error is reported.

Online may be the only key conflict caused by concurrent creation. If you query it once, the query will still report an error RecordNotFoundError

err = db_writer.CreateLabel(ctx, &label)
// Report a unique key conflict error, which may be due to the problem caused by concurrent creation. Please provide a guarantee to query againif IsKeyConflict(err) {
   label, err = db_reader.GetLabel(ctx, name)
   if err != nil {
      return label, err
   }
}

Since the character length exceeds 190, the main problem that is located is:When querying based on the name [:LabelNameLengthLimit] after slice, no result was found, but when creating label information based on name, a unique key conflict was reported, indicating that the query value is inconsistent with the actual stored value.

golang string underlying implementation go/src/reflect/

// StringHeader is the runtime representation of a string.
// It cannot be used safely or portably and its representation may
// change in a later release.
// Moreover, the Data field is not sufficient to guarantee the data
// it references will not be garbage collected, so programs must keep
// a separate, correctly typed pointer to the underlying data.
type StringHeader struct {
   Data uintptr
   Len  int
}

in:

  • Data points to the first address of []byte at the bottom layer, and the bottom layer of string is actually []byte
  • Len represents the length of the byte slice, avoiding repeated calculations when multiple times when obtaining the string length.
  • Built-in function len(string), which gets the string byte length
  • Slice the string labelName[:LabelNameLengthLimit], which is sliced ​​according to the number of bytes.

So the question is:When querying and storing, the standards for intercepting strings are different.

  • When the length of the string character exceeds 190, the query is intercepted by bytes.
  • When the length of the string character exceeds 190, it is intercepted according to the characters when storing.

Test code:

labelName := "© 2015 Charanga Sí o Ké - Chema Muñoz, Perfecto Artola, Pablo Guerrero, Antonio Flores, Los Gipsy King, Chambao, Adele, La Pegatina, El Chaval de la Peca, Los Manolos © 2015 Charanga Sí o Ké - Sones de Rumba"
("Original string content:", labelName)
("Byte length:", len(labelName))
("Seave content by bytes:", labelName[:LabelNameLengthLimit])
("Character length:", len([]rune(labelName)))
("Seave content by character:", string([]rune(labelName)[:LabelNameLengthLimit]))

Output:

Original string content: © 2015 Charanga Sí o Ké - Chema Muñoz, Perfecto Artola, Pablo Guerrero, Antonio Flores, Los Gipsy King, Chambao, Adele, La Pegatina, El Chaval de la Peca, Los Manolos © 2015 Charanga Sí o Ké - Sones de Rumba
Byte length: 214
Contents are captured by bytes: © 2015 Charanga Sí o Ké - Chema Muñoz, Perfecto Artola, Pablo Guerrero, Antonio Flores, Los Gipsy King, Chambao, Adele, La Pegatina, El Chaval de la Peca, Los Manolos © 2015 Charanga S�
Character length: 207
Contents are captured by characters: © 2015 Charanga Sí o Ké - Chema Muñoz, Perfecto Artola, Pablo Guerrero, Antonio Flores, Los Gipsy King, Chambao, Adele, La Pegatina, El Chaval de la Peca, Los Manolos © 2015 Charanga Sí o Ké

solve

Solution:

  • First turn string into rune slice, and use character slices to represent strings
  • Determine whether the rune slice length exceeds the limit. If it exceeds the limit, slice it according to the character length.
const (
   // Maximum character length   LabelNameLengthLimit = 190
)
func GetOrCreateLabel(ctx , name string) (label , err error) {
   if rs := []rune(name); len(rs) > LabelNameLengthLimit {
      name = string(rs[:LabelNameLengthLimit])
   }
   var err error
   // Query label information based on name   label, err = db_reader.GetLabel(name)
   // Or an error is reported, try to create label information   if err != nil {
       = name
      err = db_writer.CreateLabel(ctx, &label)
      // Report a unique key conflict error, which may be due to the problem caused by concurrent creation. Please provide a guarantee to query again      if IsKeyConflict(err) {
         label, err = db_reader.GetLabel(ctx, name)
         if err != nil {
            return label, err
         }
      }
      if err != nil {
         return label, err
      }
   }
   return label, nil
}

This is the article about the story of the string and slice problem in Golang development. This is all about this article. For more related Go string slice content, please search for my previous articles or continue browsing the related articles below. I hope everyone will support me in the future!