Research and development performance hardcore technical documents, AI algorithm improvement stage

Tsinghua University postdoctoral fellow

Core member of Fang Yun Intelligent Team, expert in AI algorithm application.

Fang Yun's founding team has rich experience in technology R&D and enterprise management. Relying on long-term industry accumulation and deep understanding of digital industry, the R&D team is evaluated in a digital way, which drives enterprises to accurately measure the work efficiency of R&D organizations and individuals and rationally allocate R&D resources. Help technical decision-makers to accurately measure the organizational performance (easy for upward reporting and peer communication) and personal performance (easy for downward management) of R&D. Looking back on 2020, we made a lot of attempts in data analysis based on actual user data, and achieved remarkable results, and turned the research results into practical applications, which deeply improved the product capabilities.

(A) algorithm research process

The foundation of algorithm research is data. Whether it is modeling analysis based on mathematics and empirical knowledge, or data analysis based on statistics and machine learning, it needs to rely on data.

In the first step of algorithm research, we established an independent data index system, and based on this index system, we carried out the follow-up research. The indicator system consists of three levels, the first level is the most basic metadata, the second level is calculated by the first level indicator, and the third level indicator is calculated by the second level indicator and the first level indicator. Generally speaking, advanced indicators have higher information density, and can also achieve a deeper information transmission effect when expressing information. On the other hand, the more advanced the indicators you choose, the more effective they are. It is necessary to select the necessary indicators at all levels according to the specific scenarios and algorithm requirements in order to achieve the required analysis results. For example, in kmeans algorithm, the classification effect of low-level indicators is better, while in SVM algorithm, high-level indicators are needed.

The second step of research, Kmeans. In view of the relatively complete collection of metadata and the small amount of data, combined with sklearn's algorithm to select the guide map, we choose Kmeans algorithm to unsupervised cluster the employee's behavior data.

While selecting some basic index data, the idea of RFM is introduced, and the work freshness (R), work frequency (F) and workload (M) of employees in a specific period are used as clustering indicators, and the algorithm clustering is carried out together, which has achieved very obvious classification results. The core here is that we not only evaluate employees' work result data through basic indicators, but also evaluate employees' work process data through RFM method. By combining these two types of data, employees can be well classified and characterized. The interpretation of classification results can be directly explained according to the meaning of indicators.

The third step of research, SVM. On the basis of good clustering results, we believe that the data quality is reliable, which means that we have a good objective data set. On this basis, we propose that enterprise managers grade employees' performance and form a label, so that we can get a training set of supervised learning, so as to predict employees' behavior under supervised learning. We made many attempts in this work, and finally selected the most effective 15 index as the representative index of employee behavior through feature engineering.

Here, we review the research process as an empirical reference for future research. In the preliminary analysis of SVM, we selected more than 60 indicators for supervised learning, but the learning effect is not good, and the discrimination between categories is very low, mainly because SVM algorithm can not clearly find the boundaries between categories because of too many indicators. So we use some feature engineering methods to reduce the dimension. Firstly, through Pearson correlation analysis, we divide a large number of indicators into 24 categories according to the degree of correlation, and the indicators in each category are highly correlated. Therefore, one of the most representative indicators can be selected from each category. In this selection process, our research group selected 24 most representative indicators according to the actual situation. Secondly, there are still too many 24 indexes of SVM. We use RFE algorithm to judge which indicators have the greatest influence on learning accuracy, and then choose the most effective indicators. In the process of RFE, we use Lasso, Ridge, Logistic, RFClassifier and linerSVM as filters to get the most effective features under each algorithm. Then, we choose those features that are considered "effective" by more algorithms, such as the average completion time of tasks. These features are all considered effective in the five filters, so this feature is a good feature for us to do supervised learning.

In addition, feature screening also needs to consider a question, that is, whether the filter and classifier should have the same algorithm paradigm. For example, if the classification will use SVM, then the filter employment should choose SVM class. Only in this way can we ensure that the selected features are the most effective under the corresponding classification algorithm.

The fourth step is data distribution fitting. Although we have made some achievements in the first three steps, through careful examination of the existing data, we find that there are still two problems in the data. First, some data still have the problems of missing and wrong filling, which belongs to the problem of data error. Second, there are some extreme data in the relatively complete data, which are not necessarily wrong data, but may also be caused by the abnormal behavior of individual employees. No matter what kind of situation leads to abnormal data (provided that the missing values have been preprocessed), we can judge the distribution of data by fitting the distribution of data and find those abnormal values.

In the study of data distribution fitting, we tried a variety of distribution functions, and finally put forward four common distribution functions that can be used to fit employee behavior data: normal distribution, F distribution, chi-square distribution and gamma distribution. Taking the normal distribution as an example, if we fit an index to the normal distribution, then we can think that the data within the left and right 5% is normal behavior, and the data beyond the left and right 5% is abnormal behavior. And through further analysis, we find that the data between 5% and11000 on one side are sometimes reasonable behaviors, while the data over11000 on one side are most likely to be called abnormal behaviors. Through this analysis, the abnormal behavior data of employees can be found through data distribution fitting, and the corresponding management strategies can be put forward.

In addition, we also put forward that when fitting, we need significant fitting to think that the data conforms to a certain distribution. However, if we judge this way, we find that some data do not meet the requirements of significance, but the data itself does have strong practical information, so we suggest that significance is not necessary as the premise of analysis. In fact, this also shows that in the digital age, more practical analysis methods are needed to analyze data and guide business. Instead of sticking to overly academic or rigid analytical standards.

To sum up, under these four main research ideas, we have carried out a series of standard algorithms research on employee behavior data of cooperative customers, such as feature engineering, unsupervised learning, supervised learning, data distribution fitting and so on. Then, combined with practical application scenarios, the research results are transformed into specific applications. Next, it summarizes the specific application.

(2) Product transformation results

The transformation of research results into products is a process of continuous accumulation, and quantitative change leads to qualitative change. In the initial research, we will study on several points, but we are not sure which research results can be transformed into practical applications in the end. With the increase of research, achievements that can be transformed into actual product functions will appear, which are reflected in three levels. The first level, some good research points and some solutions for specific scenarios can be transformed into actual product functions. On the second level, a single function point seems to be of little value, but when a typical function point appears, we will realize that other seemingly useless function points are effective supplements to this typical function point. On the third level, some characteristics of many researches can be transformed into product ideas and product models, which are more valuable than single-point product functions. This idea of turning research into products is rooted in practice, refined and summarized, which is of great reference significance.

After exploring several research points, we constantly think about how to turn the research points into practical functions, combining customer needs and our own design of user pain points and product functions. In the study of 2020, the main line we have been doing is employee behavior portrait. Whether supervised learning or unsupervised learning, it is to choose a set of appropriate indicators and weights to achieve the ranking of employees. In this way, we integrated a variety of ranking algorithms, and finally proposed to let users choose their own ranking mode. In different ranking modes, we provide users with different algorithms or ranking methods, which is equivalent to our intelligent way at the back end and meets the diverse needs of front-end users. This is also the embodiment that products in the digital age provide users with personalized functions in an intelligent way. Specifically, we provide users with four optional modes to rank employees.

Mode 1, the best practice in the industry, with the help of the existing cases of mature users, a set of indicators and corresponding weights are formulated. Users choose the desired case category, and we calculate the corresponding ranking results according to their actual data. There are two scoring modes here, one is to give products by yourself, and the other is to confirm the excellent degree of different categories according to the existing scoring ranking, and then use regression tree to deduce the index weight.

Mode 2: AI clustering algorithm, in which the system performs kmeans clustering on the natural state of employees for three or more times, adjusting the indicator type and weight each time, and then the customer selects a clustering result that meets the expectation, so that the customer's choice corresponds to the indicator type and weight.

Mode 3: AI supervised learning, in which employees get N categories through kmeans clustering, and customers rank the N categories according to the degree of excellence. Next, according to the scoring situation, the system judges the importance of different indicators through RFE algorithm (the estimator chooses decision tree regression or decision tree classification).

Mode 4: AI-aided customization (purely manual), where the user specifies n indicators and determines the weights for the n indicators, and the employees are ranked by the system. Optional algorithms are: weighted sum, RandomForestRegressor and GradientBoostingRegressor. Note that the latter two specific methods are: according to the weighted sum score, Y is obtained, and X is the input weighted index. Then train to get the model.

Fang Yun Intelligent's various AI performance evaluation methods have been verified by practice and realized productization.

(3)? Accuracy analysis of the algorithm

When analyzing data, the results generally need to be accurate, so it can be said that the algorithm solves the problem to some extent. In the process of digital transformation, we don't need to judge the algorithm with absolute prediction accuracy. This is because when we evaluate employees' behavior, the annotation of training set or people's cognition is extremely subjective, and this subjectivity will change dynamically, so what the algorithm can capture may sometimes be objective laws, but sometimes it may only be managers' temporary emotions. We should evaluate the algorithm from practice. It is a good algorithm that conforms to cognition and laws, but the algorithm that can explain or capture users' short-term attitudes is also reliable. Specifically, according to the existing research, we give the following summary about accuracy.

First, Kmeans is unsupervised learning, which is inaccurate, but it can explain our findings about Mr. Niu Lao and Mr. Nanguo, which is in line with the common sense of management.

SVM forecast, we first get a key conclusion, that is, the management strictness is high, medium and low, which corresponds to the medium, high and low employee performance. This conclusion is in line with the law of common sense, so it can also be inferred that the algorithm is effective.

Secondly, according to the SVM training of employee data+labels in the past, at first, our prediction accuracy was only 60%, but after sample screening and parameter optimization, the accuracy could reach 93%.

Thirdly, in the analysis of data rationality, employees within 95% are screened out by fitting different distribution of employee behavior data, and then employees between 95% and 0.00 1 are further screened out to accurately screen employees with data problems. The concrete practice results show that we have indeed grasped the extreme point of behavior, and also grasped the point that exceeds 5% but the behavior is reasonable.

(4) Research summary and next step plan

The purpose of algorithm research and data analysis is ultimately to discover new user needs and develop new product functions. The second part summarizes the train of thought from research to product actual function transformation. First, good research points are directly transformed into actual product functions. Second, some low-value function points support typical function points. Third, the * * * thinking embodied in the research is transformed into product thinking and product model.

Next, our research is also devoted to exploring more product functions and product models from these three aspects. At present, the main idea is:

The first is to implant the knowledge and process of project management into products to help enterprise managers complete project management simply and efficiently. It is a typical function to dynamically assign different tasks to people. On this basis, the analysis and ranking of employees' behavior will become a good auxiliary function, and we can assign them different tasks according to their behavior characteristics.

The second is to deepen the single-point function. When we train the model with SVM, we find that every month's model is predicted in the next month or other months, and the accuracy is unstable. The possible reason is that the evaluation criteria fluctuate every month. Then we can train models on long-term data every month and get multiple models. On this basis, the data of the next month is predicted based on the model of the past few months, so that the data of one month will have different evaluations under the model of multiple months, which can reflect the fluctuation of evaluation standards every month.

The third is the upgrade of product model. We can adopt a lightweight front end, collect some simple and necessary data, and put all complex analysis into the back end to realize. Functionally, users make some personalized data and mode choices at the front end, and the system can make diversified analysis for users at the back end, presenting users with intelligent operation interface (such as intelligent process and templated process) and analysis results (ranking, radar chart, behavior space mapping, etc.). ), even customized processes, data and algorithms. The system provides the analysis results.

Free trial address: FarCloud| Fang Yun | Fang Yun Data Intelligence R&D Performance |30 days to improve efficiency by 30%.