Common misunderstandings in broadband router performance evaluation
Liu Min
Compared with narrowband, broadband is mainly called "fast", so when purchasing broadband routers, people are very concerned about their performance. In layman's terms, it is "fast or not". So many people have put forward various evidence to prove that their broadband routers have good performance and are very fast. But many evidences are often one-sided, and it is too late to regret when you buy them and find that the performance is insufficient. This article starts from a professional perspective and analyzes several "evidences" that people often quote but are easily misleading everyone's judgment, telling everyone how to view these evidence objectively and how to correctly judge the performance of a broadband router.
Evidence 1: Processor main frequency
Said: "The processor is 100M main frequency, and has strong performance." "The processor is 133M main frequency, and has much better performance than the 100M one." "Using a professional RISC CPU for communications."
Misconception: First of all, the processor is just a component of the router, and the special thing is that it is the most core device. The processor is poor and the router performance is not good, but the router performance is not necessarily good if the processor is good. Just like cooking, you can never make good braised pork without fat and lean fresh pork, but with fresh pork with fat and lean fresh pork, you can still not make good braised pork without good soy sauce and good chef. Secondly, the main frequency of the processor is only a performance indicator of the processor. To know the performance of the processor, you must also understand its bus width (16-bit or 32-bit), cache capacity and structure, internal bus structure, whether it is a single-CPU or multi-CPU distributed processing, computing mode, etc. These will greatly affect the performance of the processor and are not more important than the main frequency at all. The so-called "adopting a professional RISC CPU for communication" is equivalent to saying nothing, just using professional terms to show off, and I don't dare to measure the details, because almost all routers use a professional RISC CPU for communication. The key is to see what core the CPU uses and what internal structure is.
User advancement: Generally speaking, 100M or below are lower main frequency, 100M~200M are medium, and 200M or above are higher main frequency. Also, I want to ask what core the processor is, is it 80186, ARM7, ARM9, MIPS or Intel Xscale? How big is the cache capacity? Is it a single CPU or a multi-CPU distributed processing? Be careful not to ask the unscrupulous businessmen who don’t tell the truth. 80186. ARM7 core processor is a typical configuration of the first generation broadband router, with low performance and no mainstream manufacturers have used it. ARM9 and MIPS core processors are currently the mainstream. The Intel Xscale architecture is an advanced network processor for high-end products. Cache capacity of 8K or less is small, 16K is common, and 32K or more is large. Generally, processors are single CPUs, and advanced processors are distributed using multi-CPUs, with high performance. If you are a professional or like to go to the bottom of the matter, you can also go into the details of whether the ARM9 is the ordinary 920T/922T/940T or the enhanced 926E/946E/966E, whether the MIPS is 2K, 3K, 4K, or 5K, and the performance and structure of different models will be greatly different. You can enter the processor model to Google and search it, then go to the chip manufacturer's website to see what others say about this chip (the reviewer has different levels of quality, and the purpose is different, so you cannot trust it all).
Evidence 2: Memory capacity
Said: "My memory is 16M, and you see that one is only 8M, and the performance is much better than it."
Misunderstanding: There is one problem with this sentence, two traps. One problem is: memory is also just a component of the router, but only an important component. Like processors, the performance of the router using small memory cannot be good, but the performance of the router using large memory may not be good. Memory is used to store all data during the calculation process, and using memory is like laying out a room. The smart Shanghai woman is good at "making a temple in a snail shell". Useless things will never be placed in the room. Every thing is placed reasonably. Every corner of the room, from the floor to the ceiling, is fully utilized. A one-bedroom with a size of 10 or 20 square meters can place all the furniture and appliances needed by the whole family, and there is even room to place one or two potted plants to decorate. But if it were a lazy bachelor, you would give him a two-bedroom apartment and the same amount of furniture and appliances to make sure he couldn't put it in. In a few days, there was garbage everywhere in the room and there was no place to stand. Therefore, it is unreasonable to absolutely judge the performance of the router based on the size of the memory used. Scientific use of memory is equally important. High-level software design can plan and use memory well. Low-level software even does not have the ability to design it yourself. If you directly use unoptimized reference software provided by processor chip manufacturers, the memory cannot be effectively planned and used. Two traps are: memory units and memory bus. Memory can be used as units bytes or Bits. The difference between the two sounds is 8 times the capacity difference (1 Byte = 8 Bit). Generally, uppercase B is used to represent Byte and lowercase b represents Bit. However, some people with ulterior motives will do their tricks here. Some of them do not label the unit confuses the public like the above statement, and some of them deliberately use B to conceal the public. The memory bus is wide and narrow. It needs to store or retrieve 4Byte data in memory. If it is 16-bit memory, it needs to be operated twice. If it is 32-bit memory, it only needs to be operated once, which is twice the efficiency. Therefore, even if the bus width of the same capacity is different, its impact on router performance is significant.
User advancement: Generally speaking, 1M~4M Byte is smaller, 8M Byte is medium, and 16M Byte or above is larger. In addition, if someone only tells you how much M is, you must ask clearly whether it is Byte or Bit, 16-bit or 32-bit. Of course, you need to find a trustworthy partner and never be deceived. If you are a professional or like to follow the truth, enter the memory model on Google and search it, and then go to the chip manufacturer's website to see what it is, you will never be wrong, unless criminals use polishing chips to fake chip screen printing. Therefore, it is important to choose an honest and trustworthy manufacturer product.
Evidence 3: Flash capacity
Said: "My Flash has a large capacity and better performance than it."
Misunderstanding: There is no reason. Flash is used to store operating systems and applications, and its size mainly depends on the operating system used, the efficiency of application writing and the fancyness of the user interface. If you choose a high-efficiency real-time operating system (such as VxWorks, Ecos, Nucleus, etc.), the designer has a deep understanding of the operating system, can tailor it reasonably, and the designer's application writing efficiency is also very efficient, so you can use small-capacity Flash. If an inefficient operating system (such as Linux, etc.), the designer does not understand the operating system, the application writing efficiency is inefficient, or simply copy the unoptimized reference software provided by the chip manufacturer, and can only use large-capacity Flash. From this perspective, the smaller Flash is, the higher the software level is, the more trustworthy the product. Of course, the product has many functions and a fancy user interface (such as many high-definition pictures), which are relatively larger using Flash.
User advancement: Ordinary users don’t need to understand how big Flash is. Just look at whether the functions meet the needs, and at most, check whether the user interface looks good or not, and whether it suits your taste (if you have this preference) is enough. If you are a professional or like to get to the bottom of it, you can try to ask the manufacturer what operating system they use, whether the software is written by yourself or copied, but basically you will be disappointed, which is the manufacturer's secret.
Evidence 4: Throughput
Said: "Super performance, Throughput up to 97M." "64Byte small package Throughput reaches line speed."
Misconception: The working process of a router is very similar to that of a post office parcel business. The process of sending packages at the post office is that everyone handed over the items to be sent, the sender, recipient and item information to the post office. The post office wrapped the items in standard boxes of different sizes and pasted a formatted package sheet on the box. After checking, it was delivered to the destination. The opposite process was made. The router is basically the same, except that the things sent and received are data. Throughput represents the amount of data that the router can process per second, which is equivalent to the package processing capacity per post office unit time, and is an intuitive reflection of the router's performance. Everyone must pay attention to the fact that there are the most hidden names behind this data.
First of all, it should be stated that the so-called router Throughput must be the Throughput of LAN-to-WAN. Only when data flows out or into the LAN requires the router to process it, so that this represents the performance of the router. Instead of LAN-to-LAN, this represents the performance of the small switches inside the router. Normally, it should be wire speed, and it has no meaning at all. This is like the parcel processing capacity of a post office unit time, which must refer to the amount of parcels shipped out and into the post office, rather than the amount of parcels moved from one room to another in the post office. But some people still use LAN-to-LAN data as router Throughput to promote it, and even mark it on product packaging. Of course, they will not write it on LAN-to-LAN, so everyone should be careful.
In addition, router Throughput should generally be the test data obtained when NAT is turned on and firewall is turned off. This is because NAT is the most basic and core function of a broadband router. If NAT is not enabled, it will not be a broadband router. Moreover, although the NAT function of different product designs is the same, the quality of software design directly affects NAT efficiency and router performance, so the Throughput enabled by NAT is meaningful. As for firewalls, they should be considered as the accompanying functions and advanced functions of broadband routers. Some products have many firewall rules and are very complicated and can filter a lot of things, while some products have few and simple rules. If there are many rules and complex, the CPU will use it to filter data for a long time. If there are few rules and simple rules, the CPU will use it to filter data for a short time. This has a great impact on Throughput test data. To be fair, it is reasonable to turn off the firewall when testing the router Throughput, especially when comparing the performance of different products. This is like after the anthrax mail attack in the United States, the post office has been much stricter in checking the postal bags, which will have a great impact on the postal bag processing capacity within the time of the postal office. However, other countries do not have such complex inspection procedures. If we assert that the US postal office is inefficient based on this, it will be unfair. As for the judgment of firewalls, they are generally placed in function comparison rather than performance comparison. Of course, if it is not a horizontal comparison of different products, only the performance of the specific product itself can be examined. If the Throughput data can be obtained in both the firewall shutdown and the firewall turn on (it needs to indicate which filtering rules are enabled), everyone can understand this product more thoroughly.
There are two common Throughput testing methods: one is Smartbits testing and the other is Chariot testing. Smartbits Testing uses the world's most authoritative network equipment testing instrument Smartbits2000 or 6000, and is equipped with its included professional testing software SmartApplication. It uses continuous UDP packets to measure the number of packets that routers of different sizes can process per second. When a router processes data packets, the main time it spends on processing packets at the beginning and end of the packets, so for packets of different sizes, the number of packets that the router can process per second will not be too different. This is like the post office processing packages. The main time spent processing sender, recipient and item information and inspections. The heavy or light package will not have much impact on processing speed. For example, 128Byte packets can process 10,000 per second, but it cannot handle 20,000 per second. Instead, it is only a little more than 10,000, such as 10,100. This gives some dishonest manufacturers an opportunity to take advantage of. For example, when his router processes the largest 1518Byte packets, the limit value of 100M linear speed of 1518Byte packets is 8127 according to the theoretical calculation, so the converted Throughput is 100M*8000/8127=98.44M, so he proudly announced that my router Throughput is as high as 98.44M. However, it turns out that this router is 11,000 per second when processing the smallest 64Byte packets, and the limit value of 100M linear speed of 64Byte packets is 148,810, so the converted Throughput is only 100M*11000/148810=7.39M, which is more than 13 times the difference between the two. It all depends on how we look at throughput, how many packets are processed per second or how many megadata is processed per second, just like how many packets are processed per day by post office or how many kilograms of goods. From the perspective of post office capabilities, it should be more reasonable to judge how many packets are processed every day, because this data is relatively stable and less affected by the weight of the packet. But how many kilograms of goods per day are fundamentally affected by the weight of the parcel. If all the mailings are books, the calculated kilograms of goods processed every day will be very large, and if all of them are cotton, it will be very small. In actual situations, there must be books, cotton and other various things, but what proportion is it reasonable to allocate these goods when you evaluate them? No one can explain it clearly unless someone sets a standard to specify the proportion of various things, and everyone calculates it according to this standard. Since there is no standard, some people use books to calculate, some people use cotton to calculate, and those with the rules will explain what I use to calculate, and those who are not in the rules will be happy to catch up with troubled waters. In addition, judging from the hardware commonly used by broadband routers, even the best performance is not enough to achieve the 64Byte packet Throughput line speed, but some people claim that my product is so good, and they come up with Smartbits test data to prove it. This type of product uses the so-called NAT hardware acceleration technology. Throughput is very good-looking, but the price is that all data does not pass through the CPU, data filtering cannot be done, the portal is open, the firewall does not work, and the router does not have any management functions.
Chariot test uses two servers to install NETIQ's popular software Chariot, one connected to the router's WAN port and the other connected to the LAN port. By counting the time when a script file of a predetermined length and format is transmitted from one server to another without error, the router's Throughput is calculated. This is a lot of knowledge. First, server performance. If the performance is poor, it will become a bottleneck in testing, and the performance of the router cannot be reflected. Therefore, dishonest manufacturers use high-performance servers when testing their own products, for fear that their own product performance will not be performed. However, when testing other people's products, they use low-performance servers to deliberately belittle the data. The second is to establish the number of connections. If the router software is written poorly, the Throughput will plummet when the number of connections increases, while a good router will be very stable and will even rise slightly with the increase of the number of connections. If the router is not performing enough, the test will fail even because it is unable to establish so many connections at the same time. Therefore, a bad router only dares to use one pair of connections to obtain test data, while a good router is not afraid of multiple connections. Third, Chariot tests are not testing UDP "packages" like Smartbits tests, but data "stream". The amount of TCP packets its data is encapsulated is related to the server operating system, but they are generally encapsulated as 1518Byte packages, so its test data will be much better than the data used to test 64Byte packages with Smartbits, which is very confusing. Fourth, the result of Chriot test Throughput is effective data load, excluding TCP protocol loss, frame interval, response and Chriot's own system loss. The typical loss in this part is calculated by theory about 6M, that is, even if you connect to a switch that can forward wire speed, not a router, the measured Throughput can only be around 94M. This is the theoretical limit. If someone claims that my product Chriot tests Throughput 97M and 98M, it is undoubtedly a fool.
User advancement: Throughput is the Throughput of LAN-to-WAN. Generally, it should be the test data obtained when NAT is turned on and the firewall is turned off. There are two methods: Smartbits test and Chariot test, and the results can be very different. According to the common hardware capabilities nowadays, Smartbits package test data can easily reach linear speed, just like when you test the intelligence of college students, the question is 1+1=? , everyone can answer, it can't compare. So you need to make an objective assessment. The question should be more difficult, compare the 64Byte packet test data, and make the difference immediately. Of course, you have to be careful whether there is NAT hardware acceleration, and if so, turn it off and then test it. As for Chariot testing, the performance of different products should be fair to compare under the same test environment and methods and the same number of connections. Moreover, in actual network applications, there are always multiple connections and almost impossible to be single connections. In order to find a bad router, Chariot testing is best performed under multiple connections. Generally, you can choose 100 pairs of connections to basically see the difference. In short, a single Throughput data is meaningless. It must explain what method the data is measured to be useful. Comparison of mutual performance must be fair and meaningful under the same testing environment and methods. The best way is to watch Smartbits test NAT to enable LAN-to-WAN Throughput with 64 Byte packets.
Evidence 5: Number of aircraft
Said: "The router has 200 units of the engine." The maximum allowable engine capacity of this router is 253 units.
Misconception: This statement is inaccurate. The busyness of each network is very different. Everyone in the Internet cafe is busy chatting and gaming online, and almost all the data is passed through the router WAN port, so the load is very heavy. But if it is an enterprise network, most people are busy designing, writing reports, and making plans. At the same time, only a small number of people are using the network, and most of the data flows within the enterprise network, so the router load is very light. In an enterprise with 200 PCs, a router that can be used in an Internet cafe may not be able to carry 50 PCs. Estimating the average data traffic of each PC on a network cannot be accurate. This is like the cafeteria staff can only eat four taels per person on average. This pot of four pounds is about 10 people. But it is very likely that one day everyone has exercise and their appetite is surprisingly good. One pot of rice is only enough for 6 people to eat, or one day when there is a cold, everyone has a bad appetite, and 5 people will be enough for them to eat. The master of the miner's canteen must estimate that he eats six taels per person on average, while the master of the female model canteen may only dare to estimate that he eats one tael per person on average. Therefore, a more objective statement should indicate which type of network the number of the engine is for, and the quantity is a range estimated based on typical situations. For example, "The Internet cafe has 150~250 units (typical value)", which is much more responsible. The second statement of maximum allowable number of engines is bluffing. Its basis is not the performance of the router, but the maximum number of IP addresses that DHCP can allocate. 254 are reduced by 253.
User advancement: The number of drivers is only an estimated value and experience value. It must be viewed in combination with the actual situation of the network. The accurate performance depends on the test data. Smartbits tests how many pps (packet per second) of the LAN-to-WAN Throughput of the Smartbits test NAT to enable 64 Byte packets. However, the number of carry-ons is very intuitive and easy to understand for ordinary users. When referring to this data, you must pay attention to the several misunderstandings mentioned above. In addition, if this data comes from a large manufacturer or a reputable manufacturer, it is generally said to be more rigorous and highly credible. If it comes from a small manufacturer or a miscellaneous brand, the general statement is vague and has poor credibility.
Evidence 5: Number of aircraft carriers Evidence 6: Number of WAN ports
Said: "This router has a dual WAN port, and its performance is twice that of a single WAN port." "Single WAN port has 100 units, and dual WAN port has 200 units."
Misconception: This statement confuses the concept. After the basic hardware and software of a router are determined, its processing capabilities or performance will be determined and will not change significantly with the increase or decrease of the number of WAN ports. There is a situation: the router's own processing capacity is surplus compared to the WAN port exit bandwidth. For example, the router's processing capacity is 40M, and the WAN port exit bandwidth is 10M per line. Due to the limitation of the exit bandwidth, a single WAN port router can only have a throughput of 10M, while a dual WAN port router can have a throughput of 20M. From this perspective, it seems that the network performance has doubled, but this is just a question of whether your network configuration is reasonable. The router performance is always 40M, and it has not changed. Conversely, if the router itself has only 5M processing capacity, it is possible to have only 5M throughput, whether it is a single WAN port or a dual WAN port. For example, a factory can make 10,000 products a day, but only one truck can transport 5,000 products a day. At this time, adding a truck will be very effective. But if the daily output is only 3,000 pieces and two trucks are equipped, it will be of no use and will only increase the cost.
User advancement: There are now some multi-WAN interface routers of different brands on the market, but the performance is uneven. When a router uses multiple WAN ports, it must first be based on the premise that the router itself must have strong performance. It has surplus of processing capabilities compared to the egress bandwidth router. If its processing capacity is limited, multiple WAN ports are purely a decoration. When choosing a multi-WAN interface router, you must carefully examine its performance. One thing for your reference: If the router uses an ARM7 core or a comparable performance processor, and the main frequency is less than 100M, it can basically be determined that its performance is not enough to use multiple WAN interfaces.
How to correctly judge performance
The previous analysis was made on several "evidences" that people often quote but are likely to mislead everyone's judgment, and told everyone how to view these "evidences". Everyone may ask: You say this doesn’t count, and that doesn’t count, so tell me how to judge. Here I would like to tell you a four-step judgment method of Step-by-Step for your reference.
1. Selecting a brand: To a large extent, brand represents the quality, reputation, service, etc. of manufacturers and products. Choosing a product from a certain level is choosing a manufacturer and brand. The previous article also talked about the importance of choosing an honest and trustworthy manufacturer. We can understand the capabilities of a manufacturer and whether it is honest through the description of product specifications and performance. Currently, there are many brands in the broadband router market in China, which can be roughly divided into three categories. One type is imported well-known brands, such as Linksys, Netgear, and DrayTek. Most of them are purchased by major OEM/ODM companies in *. The product quality is generally guaranteed, unless there is ISP compatibility problem in the use area and cannot be interoperable with the local government (this is caused by the regional separation between * and the mainland, and it is generally difficult to solve). The manufacturer's reputation is generally guaranteed, and the disadvantage is that it is expensive and has an English interface (mostly). The second category is well-known local brands, such as TP-LINK, which have independent research and development and manufacturing capabilities. The localized design makes the products very in line with the needs and usage habits of domestic users. It has good ISP compatibility, guaranteed quality and reputation, and has a very high performance and price ratio. It is currently the most dazzling brand in the Chinese broadband router market. The industry generally believes that its market share has exceeded 50%. The third category is small brands and miscellaneous brands. They do not have the ability to develop and are forced to pay cost pressure, so they can only buy some cheap and low-quality boards from some small factories in *, put a shell on them and label them, and then sell them. Some illegal people even copy the designs of small factories in * to make products, including Copy hardware and copying software, and the quality and reputation. Since this type of manufacturer knows little about routers, the ignorant is fearless and wants to seek benefits through improper means, these manufacturers are often the ones who call it the most fiercely and the most powerfully. The first and second categories of brands mentioned above are optional, while the third category is not advisable.
2. Exclusion method: The product is a complex and a certain "evidence" cannot be viewed in isolation. The performance of a broadband router depends on the hardware foundation and software design. Hardware is the guarantee, software is the key, and it is expressed through indicators such as Throughput and number of belt drives. As mentioned earlier, you must be good at cooking ingredients, auxiliary ingredients and chefs to make good dishes. The same is true for broadband routers. The processor is the main material, other hardware is the auxiliary material, and the design engineer is the chef, which is indispensable. However, there are many kinds of products on the market, and users can learn relatively little information about each product. What should we do under limited information conditions? We can adopt reverse thinking. Since it is difficult to fully prove that the product has good performance, we might as well find out the bad ones first. It is very simple to prove that it is not good. Just find a problem. This is the way to eliminate it. For example, if we know that a certain router uses an ARM7 core processor, we don’t have to look at other parameters anymore, and the performance will definitely not be good. The key to elimination is to learn as much information as possible, and based on the information you have mastered and the methods of analyzing and identifying the information told before, dig out the implicit content, distinguish the authenticity of it, and find out the bad ones. In this way, through the second step, we greatly narrowed the scope of choice.
3. Inquire more: Listening to both sides will make it clear, listening to the side will make it dark. Listening to what others say is a way to understand a product that is both trouble-saving and effective. It is important to inquire with whom. One of the best ways is to find horizontal evaluation data of reputable and professional media. Pay attention to the premise that it is credible and professional, and can be objective and fair and measure ideas. For example, the "18 broadband router comprehensive test" in the 2nd issue of "Online Online" magazine (2004 not long ago is a horizontal evaluation with a relatively comprehensive product/brand, and a more professional, objective and comprehensive test, which is very valuable for reference. I believe that based on the current market position and development speed of broadband router products, many authoritative media will conduct similar horizontal evaluations this year, so everyone can pay attention to it. For the first and second-class brand products mentioned in the first step, you can call or send an email to the manufacturer directly about unknown places, and you can generally get more objective answers.
4. Try it: After the above three steps, everyone can basically make judgments and choices. If possible, it would be safer to try it out. Non-professionals can directly install the router to the network for use. Be careful to try it during the busiest period of the network to see that the sex is not enough. For example, Internet cafes should be basically full, and it is best for everyone to test it while playing "Legend". If professionals have equipment and software, they can do Smartbits tests and Chariot tests, focusing on the following items: Smartbits test NAT to enable LAN-to-WANThroughput with 64Byte packets, the strictest and most objective performance data; the maximum number of concurrent connections and connection establishment speeds greatly affect the belt drive capability; Chariot tests throughput with multiple connections can also better reflect performance.
[1]
Article entry: csh Editor in charge: csh