In some Linux systems, the glibc version itself comes with is too low, causing rpm to be unable to be installed. If you directly update the system's glibc version, it will cause the system to crash. Even if you compile and install the glibc library to a non-system directory, you cannot set the environment variables and these 2 files. Once you set the glibc library pointing to the new version, the system will crash in minutes. You cannot install this new library directly into the system directory. The author unfortunately encountered this problem recently.
glibc introduction
What is glibc? Why is the risk of escalation so high? What is the function of glibc?
glibc is a libc library released by GNU, that is, a c runtime library. glibc is the lowest-level API in the Linux system, and almost any other runtime library will depend on glibc. In addition to encapsulating the system services provided by the linux operating system, glibc itself also provides many other necessary functional services implementations. It takes a long time for the kernel to implement a function. Since glibc and the kernel are not developed together, glibc needs to be compatible with different versions of kernels, and the kernel also needs to be compatible with different versions of glibc. Both parties have too much historical burden.
A simple understanding is actually similar to the http API interface. The difference is that glibc calls the kernel to operate on the underlying disk, memory, network card, etc., and http API operates on the application-level app. For example, listening to a song or downloading a movie is actually calling the http API interface.
The problem occurred
Trying to upgrade the glibc library version of Linux server by compiling and installing. After install fails, most commands in the shell (ls, cat, rm, cp, ln, scp, vi, yum, etc.) are executed and the connection is rejected when trying a new ssh connection.
The command error is similar:
# ls ls: relocation error: /usr/lib64/.6: symbol _dl_starting_up, version GLIBC_PRIVATE not defined in file .2 with link time reference
or
bash: /usr/bin/ls: /lib64/.2: bad ELF interpreter: No such file or directory
Remedial operation space
There is an ssh client connected to this server and has not disconnected the original low-version glibc library file. The cd, pwd, export, echo, sln, and chmod commands are available. Enter ls+ twice to prompt all files in the directory.
Start trying to fix it
Step 1
glibc is the underlying library of Linux system. Many shell commands and even bash themselves rely on this set of dynamic link libraries.
The above problems are/lib64（/usr/lib64）The soft connection in the directory is corrupt, or the connected so library file version is not unified, or there is a problem with the connected so library file itself.
The solution is to restore all the following soft connections to point to the original version (lower version).
The premise is to arrive first/lib64In the directory, use the ls+two TAB keys to confirm the system.libc-<>.soWhich version is the minimum version number (that is, the easy version).
lrwxrwxrwx. 1 root root 10 Jul 4 16:55 .2 -> ld-2. lrwxrwxrwx. 1 root root 14 Jul 4 16:55 .1 -> libanl-2. lrwxrwxrwx. 1 root root 23 Jul 4 16:55 .1 -> libBrokenLocale-2. lrwxrwxrwx. 1 root root 15 Jul 4 16:55 .1 -> libcidn-2. lrwxrwxrwx. 1 root root 16 Jul 4 16:55 .1 -> libcrypt-2. lrwxrwxrwx. 1 root root 12 Jul 4 16:55 .6 -> libc-2. lrwxrwxrwx. 1 root root 13 Jul 4 16:55 .2 -> libdl-2. lrwxrwxrwx. 1 root root 12 Jul 4 16:55 .6 -> libm-2. lrwxrwxrwx. 1 root root 14 Jul 4 16:55 .1 -> libnsl-2. lrwxrwxrwx. 1 root root 21 Jul 4 16:55 libnss_compat.so.2 -> libnss_compat-2. lrwxrwxrwx. 1 root root 17 Jul 4 16:55 libnss_db.so.2 -> libnss_db-2. lrwxrwxrwx. 1 root root 18 Jul 4 16:55 libnss_dns.so.2 -> libnss_dns-2. lrwxrwxrwx. 1 root root 20 Jul 4 16:55 libnss_files.so.2 -> libnss_files-2. lrwxrwxrwx. 1 root root 21 Jul 4 16:55 libnss_hesiod.so.2 -> libnss_hesiod-2. lrwxrwxrwx. 1 root root 22 Jul 4 16:55 libnss_nisplus.so.2 -> libnss_nisplus-2. lrwxrwxrwx. 1 root root 18 Jul 4 16:55 libnss_nis.so.2 -> libnss_nis-2. lrwxrwxrwx. 1 root root 18 Jul 4 16:55 .0 -> libpthread-2. lrwxrwxrwx. 1 root root 17 Jul 4 16:55 .2 -> libresolv-2. lrwxrwxrwx. 1 root root 13 Jul 4 16:55 .1 -> librt-2. lrwxrwxrwx. 1 root root 15 Jul 4 16:55 .1 -> libutil-2.
Step 2
Since the ln command is no longer available, you can use the sln command to create/repair soft connections. Command format:sln <file pointed to> <soft connection name>. Assuming that you need to fall back to version number XXX, you can fix it with just the following command.
cd /lib64 sln .2 sln .1 sln .1 sln .1 sln .1 sln .6 sln .2 sln .6 sln .1 sln libnss_compat libnss_compat.so.2 sln libnss_db libnss_db.so.2 sln libnss_dns libnss_dns.so.2 sln libnss_files libnss_files.so.2 sln libnss_hesiod libnss_hesiod.so.2 sln libnss_nisplus libnss_nisplus.so.2 sln libnss_nis libnss_nis.so.2 sln .0 sln .2 sln .1 sln .1
At this point, if there are no operation errors, key commands such as ls, including ssh connections should be able to be used normally and the repair is completed.
However, due to the author's error during operation ("sln xxx yyy" written as "sln yyy xxx"), the ld-2. original library file was overwritten into a soft connection file, so further remediation is needed.
Secondary remediation after misoperation
The solution is to restore accidentally damaged ld-2. files, so you need a copy of available ld-2. file data. Since the author uses a server cluster, the original files can be obtained from other nodes. If it is a stand-alone server, you may need to use the Internet to obtain the ld-2. original file.
The biggest obstacle at present is that commands such as scp, mount, wget cannot be used. You need to consider how to put the obtained original file on the disk of the problem server - the solution is echo command + redirect the output file, which is as follows:
Open the original file in binary mode in a text editor (for example, using EmEditor on Windows), and select all to copy the original byte content of the file, as follows:
7F 45 4C 46 02 01 01 00 00 00 00 00 00 00 00 00 03 00 3E 00 01 00 00 00 20 11 00 00 00 00 00 00 40 00 00 00 00 00 00 00 48 77 02 00 00 00 00 00 00 00 00 00 40 00 38 00 07 00 40 00 1C 00 1B 00 01 00 00 00 05 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 A0 18 02 00 00 00 00 00 A0 18 02 00 00 00 00 00 00 00 20 00 00 00 00 00 01 00 00 00 06 00 00 00 40 1B 02 00 00 00 00 00 40 1B 22 00 00 00 00 00 40 1B 22 00 00 00 00 00 38 14 00 00 00 00 00 00 10 16 00 00 00 00 00 00 00 00 20 00 00 00 00 00 02 00 00 00 06 00 00 00 00 1E 02 00 00 00 00 00 ......(common5107OK，slightly)......
Continue to edit the text, replace "(1 space) and "(2 spaces) in the copy content with "\x", and also insert "\x" at the beginning of each line, as follows:
\x7F\x45\x4C\x46\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x03\x00\x3E\x00\x01\x00\x00\x00\x20\x11\x00\x00\x00\x00\x00\x00 \x40\x00\x00\x00\x00\x00\x00\x00\x48\x77\x02\x00\x00\x00\x00\x00\x00\x00\x00\x00\x40\x00\x38\x00\x07\x00\x40\x00\x1C\x00\x1B\x00 \x01\x00\x00\x00\x05\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 \xA0\x18\x02\x00\x00\x00\x00\x00\xA0\x18\x02\x00\x00\x00\x00\x00\x00\x00\x20\x00\x00\x00\x00\x00\x01\x00\x00\x00\x06\x00\x00\x00 \x40\x1B\x02\x00\x00\x00\x00\x00\x40\x1B\x22\x00\x00\x00\x00\x00\x40\x1B\x22\x00\x00\x00\x00\x00\x38\x14\x00\x00\x00\x00\x00\x00 \x10\x16\x00\x00\x00\x00\x00\x00\x00\x00\x20\x00\x00\x00\x00\x00\x02\x00\x00\x00\x06\x00\x00\x00\x00\x1E\x02\x00\x00\x00\x00\x00 ......(common5107OK，slightly)......
Merge all behaviors in one line and remove all spaces as follows:
\x7F\x45\x4C\x46\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x03\x00\x3E\x00\x01\x00\x00\x00\x20\x11\x00\x00\x00\x00\x00\x00\x40\x00\x00\x00\x00\x00\x00\x00\x48\x77\x02......(slightly)......
Compose the edited hexadecimal data with the echo command parametersecho -e "Edited Hexadecimal Data" > ~/In the form, paste it to the ssh terminal connected to the problem server and execute it. After a long wait (remember in hours, because the echo is very slow. You can flexibly use the functions similar to CommandWindow in the shell client to enter commands in the input box to save time), and the generated~/The file is used as a replacement for the damaged ld-2. file. The method of repairing soft connections was reused using the above sln method, and the repair was finally successful.
Summarize
You can only use third-party tools to introduce glibc into your project. The first is to let the rpm installation package come with this library. The second is to use yum or other third-party tool libraries for installation. The third is to change to an updated system. The new system comes with a higher version of glibc. I suggest changing the system directly if there is no mandatory requirement for the system version.
This is the article about how to deal with the failure to upgrade GLIBC on Linux server, causing the unavailability of shell commands. For more related contents related to the failure to upgrade GLIBC on Linux, please search for my previous articles or continue browsing the related articles below. I hope everyone will support me in the future!