Preface
The previous article introduced why the front-end needs to have a monitoring system? What is the significance of the front-end monitoring system? Some friends left a message after reading it and wanted to hear some detailed implementation. So in this article, we will start to introduce how to implement front-end monitoring.
If you don't understand why, what is the use of surveillance, it is recommended to read the previous article first:Why can't the front-end be without a monitoring system?
Before starting to implement it, you must first have a overall context in your mind and understand the specific process steps for building front-end monitoring. Because the front-end monitoring system is actually a complete full-stack project, not just the front-end, and even the main implementations are centered around data.
Of course, there is another point that the implementation of this article is mainly aimed at ordinary business and self-developed by small and medium-sized factories. I have seen monitoring systems made by large manufacturers, which are very complex and have strong capabilities. They can often have billions of data, and finally reach the direction of big data. I will only introduce how to implement the main functions and how to solve the problem.
The construction process of front-end monitoring is divided into the following stages:
- Acquisition phase: data acquisition
- API stage: build API application and receive collected data
- Data storage stage: API application connects to the database and stores the collected data
- Query statistics stage: query, statistics, and analyze the collected data
- Visualization stage: The front-end querys statistics through API and performs visual display
- Alarm phase: API docking alarm notification services, such as DingTalk
- Deployment phase: The overall application deployment is online
Next, I will sort out the key implementation ideas for each stage.
Collection stage: What data should be collected?
The first step in monitoring is to collect data. With data, the prerequisite for realizing monitoring.
The meaning of collecting data is to record the real operations of users during the use of products. Combined with our analysis in the previous article, the data generated by real operations can be divided into two categories:Exception dataandBehavioral data。
Let's analyze the abnormal data first. Exceptions in the project can be divided into two categories: one is front-end exception and the other is interface exception.
Front-end exception
In summary, front-end exceptions can be roughly divided into:
- JS code execution exception
- Promise exception
- Static resource loading exception
- abnormal
- Cross-domain exceptions
The most important one, and the one we encounter the most, is the execution exceptions of various js code. For example, type errors, reference errors, etc. Most of these exceptions are caused by our poor encoding. Therefore, collecting such exceptions is conducive to improving the encoding quality.
Then there is the Promise exception. Promise is one of the most important attributes of ES6. It tests our js asynchronous programming capabilities and is concentrated in interface requests. Therefore, exception capture in these two parts is very critical.
In addition, static resource loading exceptions generally refer to referring to some image addresses, third-party js addresses, etc. in html. If various reasons cannot be loaded normally, this must be monitored.
Exception is usually used to use a third-party front-end framework. Some errors are customized in it and can use it.If thrown out, this type of exception is also necessary to catch.
As for cross-domain exceptions, we often encounter this, which can generally be discovered during the joint debugging stage during front-end and back-end development. However, it is possible that the backend suddenly changed its configuration online, resulting in the frontend cross-domain, so it also needs to monitor it for security.
There are probably five types of abnormal collection on the front end, which basically cover more than 90% of abnormal situations on the front end.
Interface exception
Interface exceptions are backend exceptions, but interface exceptions can directly lead to front-end page errors. Therefore, this type of exception is an important basis for us to judge the root cause of online problems. Interface exceptions can be classified according to the response result:
- Not responded/timeout response exception
- 4xx request exception
- 5xx Server exception
- Insufficient permissions
Sometimes due to network problems or server problems, the front-end does not receive a response after initiating the request, and the request is suspended. This is an unresponsive/timeout response exception. We can set the maximum request time for this type of exception, actively disconnect the request after the timeout, and add an interface timeout record.
In addition, we can use other types of interface exceptions to use the HTTP status code or the specified fields returned by the backend, such aserror_codeLet's make a judgment.
Whether using status codes or other judgment methods, as long as you can distinguish the exception types, there is no strict requirement for this.
4xxThe exception type is a request exception, which is generally a parameter problem passed by the front end, or an interface verification parameter problem. The key to handling such exceptions is to save the request parameters, which can facilitate the front-end troubleshooting.
5xxErrors are exceptions processed internally by the server. The key information of this type of exception is the error time and the returned exception description. Saving these can facilitate the backend to find logs.
I think insufficient permissions are also an important mistake. Because the permission design of some management systems is relatively complicated now, sometimes the interface cannot be adjusted inexplicably, which affects the user's next operation, which also requires recording and tracking.
Behavioral data
The behavioral data is relatively broad, and we can define any meaningful operation of the user as behavioral data.
For example, clicking a button, how long it takes to stay, the click rate of the new function, when to use it, etc. One of the major advantages of self-developed monitoring systems is their flexibility. Any useful information you need can be designed at this stage.
This stage is very critical and is the core of monitoring system design, so I wrote it in detail. You should also consider which data you collect at this stage. The subsequent stages are based on the specific implementation of this design.
API stage: Build an API interface for reporting data
The previous stage has prepared a plan to collect data. After collecting data, the next step is toData reporting。
To put it bluntly, data reporting is to pass the data over and then store it in the database by calling an API interface. Therefore, the task of this stage is to build an API interface to report data.
As a glorious front-end engineer, developing interfaces naturally, you must choose ones belonging to the same JS family.Now. There are also many frameworks at present. I prefer lightweight and simple ones, and I need to install them myself, so I chose a simple and classic Express framework.
What to do in building an API application is:
- Directory structure design
- Routing design
- Authentication and authentication
- Parameter verification
- Request response encapsulation
- Error handling
There are also some details to deal with. This stage is a very good time to learn for students with weak back-end foundations.
I highly recommend that front-end friends master some basic back-end knowledge, at least understand what is going on in terms of simple principles. This stage mainly involves understanding how API applications are built, why each part needs to be done this, and what problems can be solved, so that your back-end basic knowledge will be established.
After the framework is built, the main thing is to design the interface URL and then write processing logic to ensure that the interface designed in this step can be adjusted and data can be received.
Data storage stage: interface docking database
In the previous step, we have built an API interface and received the collected data. Then our step is to connect to the database and store the collected data in the database.
For databases, choose the document database that is the most friendly front-end and belongs to the NoSQL family.MongoDB。
The biggest feature of this database is that the stored data format is similar to JSON. It operates like calling functions in JS and combining JOSN data. It is very easy to understand and get started with our front-end, and you can experience its elegance in the actual combat process.
The data storage stage mainly introduces the basic information and operations of the database, including the following aspects:
- How to connect to the database
- How to design fields
- How to verify
- How to write
- How to query
The most important thing in this stage is data verification. After designing the database fields, we hope that all the written data must meet the data format we want. If the compliance does not comply after verification, we can supplement or modify the data fields, or directly refuse to write, which can ensure the reliability of the data and avoid unnecessary data cleaning.
After doing a good job in data writing, we also need to add some simple query and modification functions. Because after writing data, you need to see if the execution is successful, you can check a list to see the results.
Modifying the function is also necessary. A very common requirement in front-end monitoring is:Calculate the user's page stay time. My plan is to create a record when the user enters a certain page, and then modify the record when leaving, add a field of end time, which requires the modification function.
Finally, I would like to mention how many people are talking about how to do itData cleaning. In fact, this depends on how you did the verification when storing data. If it is indeed possible to store invalid data, you can write an interface to clear the data, write your own cleaning logic, and then execute it regularly.
Query statistics stage: data query and statistical analysis
After a series of preparations, we completed the API interface and data writing functions. Assuming that we have collected enough data and stored it in the database, this stage is time to make good use of this data.
The main task of this stage is to search and statistically analyze the data, which is basically a "query" operation.
The query here is not just a matter of checking. How to check it specifically depends on whether the data we collect can be effectively used. My idea starts from these two aspects:
- Behavioral data: Overall statistical query, look at the trends of a certain period of time
- Exception data: single query, precise positioning, and troubleshoot specific errors
Of course this is just in general. The behavioral data will also be queried in a single line. For example, I want to see what a user does at a certain time, which is an accurate search. There are also statistics on the exception data, such as the ranking of the trigger frequency of the exception interface.
The amount of behavioral data will be very large, and will be frequently generated and written to the database during the user's use of the system. Therefore, most of the cases of this type of data are made by aggregation query from multiple dimensions such as pages and time, and finally some percentage conclusions are drawn. These statistics can roughly reflect the actual use of the product.
There is an optimization point here. Because frequent requests will increase the burden on the interface, so the data can also be stored locally first, and then requested the interface after reaching a certain amount, and stored in one go.
Exception data is very important to developers and is a divine assistance for us to locate and resolve bugs. Unlike multiple statistics of behavioral data, we are more concerned about the detailed information of each individual record for abnormal data, so that we can see errors clearly at a glance.
Exception data query is also relatively simple. Just like ordinary list query, you can return the latest exception data. Of course, after we troubleshoot the problem, we should also mark the handled exceptions as processed, which can prevent repeated troubleshooting.
It can be seen that the most important thing in this stage is to make statistical interfaces to prepare for the visual chart display in the next stage.
Visualization phase: The final data graph display
In the previous stage, we developed a statistical interface and found the desired data results. Unfortunately, these results can only be understood by programmers, and others may not be able to understand them. So in the end, in order to respond to data more intuitively, we need to use front-end visualization charts to make these data come alive.
At this stage, we finally returned to the most familiar front-end field. The task in this stage is relatively simple and easy. Based on React, a new front-end application is built, connected to the statistical interface in the previous step, and then integrated the front-end chart library to display the statistical results in a chart.
This new application is a front-end monitoring system that is truly to be displayed to the outside world, and is for use by developers or product students within the team, so that they can view the data generated by the product in real time, thereby solving their own problems.
There is actually no key issue to talk about at this stage. The main thing is to choose a useful chart library and connect to the interface. There are also many types of charts. You should consider which data is suitable for which chart and judge based on actual conditions.
Finally, the front-end page and interface data of the monitoring system must not be viewed by everyone, so there must be basic login pages and functions. If you do this, the task of this stage will be over.
Alarm phase: Immediately call the alarm immediately
In the previous stage, after the front-end of the monitoring system was built and the statistics were displayed as a chart, the entire monitoring system was basically available.
But there is another situation where the user suddenly reported an error when using our product, and the error message was also written to the database. If you do not actively refresh the page at this time, in fact, you cannot keep refreshing, then we don’t know this error at all.
If this is a very fatal bug, the impact is very wide, and we don’t even know that the bug happens, it will cause us a lot of losses.
Therefore, in order to ensure that we solve bugs in a timely manner, the function of an alarm notification is very important. Its function is to push it to the developer as soon as possible when an exception occurs, so that everyone can immediately discover the problem and then solve it as quickly as possible to avoid omissions.
Alarm notifications, generally the common solution is to connect with DingTalk or enterprise WeChat robots, and we use DingTalk here. The specific platform to use depends on which platform your subject is on. For example, if the main body of my team is DingTalk, then when sending an alarm notification, you can use your mobile phone number to @ any of your team members to achieve more accurate reminders.
This part is a supplement to the API application. After applying for DingTalk developer permissions, access relevant code in the API.
Deployment phase: Everything is ready and only wait for it to go online
In the previous stages, we completed data collection, API application construction, data storage, front-end visualization, and monitoring and alarms, and the functions of the entire front-end monitoring system are fully complete. The last step is to deploy all front-end and back-end databases online for everyone to access.
The deployment area mainly involves nginx parsing, https configuration, database installation, and nodejs application deployment, etc. The content at this stage will be more operational and maintenance. But don't worry, I will also give a detailed introduction to key operations here.
After this system is online, you can try to save the collected data through the API in any of your front-end projects according to the first collection method, and then you can log in to the monitoring system to view the real usage data.
When this part is completed, congratulations, a small front-end monitoring system has been built. In the future, the functions can be continuously expanded based on this to gradually make this self-developed monitoring system more powerful.
Summarize
This article introduces the construction process of the front-end monitoring system, divides the overall process into several stages, briefly describes what to do in each stage and what key issues are, and helps you clarify the ideas for building a monitoring system. For more information about the construction of front-end monitoring of JavaScript architecture, please pay attention to my other related articles!