Calculating model accuracy is a critical part of any machine learning project yet many data science tools make it difficult or impossible to assess the true accuracy of a model.
It is clear that validating machine learning models is a very difficult task. For instance, how would one validate an advanced artificial intelligence (AI) system? The AI might be able to answer many questions based on its vast knowledge and experience. However, it may not always be possible for a human to understand the methodologies used by the AI in arriving at certain conclusions or predictions.
In machine learning, model validation is referred to as the process where a trained model is evaluated with a testing data set. The testing data set is a separate portion of the same data set from which the training set is derived.
The basic recipe for applying a supervised machine learning model are:
* Choose a class of model
* Choose model hyper parameters
* Fit the model to the training data
* Use the model to predict labels for new data
From Python Data Science Handbook by Jake VanderPlas
In my opinion, cross-validation is the best technique for evaluating the efficiency of an individual data set on a statistical prediction model. The goal is to make sure that the model and the data fit together well. During the training process, cross-validation is performed where the user can determine whether the model is prone to underfitting or overfitting the data. For the goal variable, the data to be used for cross-validation must be from the same distribution, or else we will confuse ourselves on how the model will behave in real life.
The best way to validate Machine Learning models is through real-time data.
As the ML model learns from data, it will keep on updating itself to reflect new information.
If we have real-time data from the environment, the ML model should be able to predict what is going to happen next.
If the model can make predictions, then we validate it.
If the predictions turn out to be inaccurate, then we can change the ML model and make it more accurate.
Imagine that there is a situation where we want to predict the future stock prices of Google, Microsoft, and Apple or consider a simple machine learning model that predicts the movement of a ball as it drops.
Given its very limited experience with such predictions, one might say that this particular machine learning model is only correct 50% of the time. However, if this same model were used to make predictions regarding the flight path of an airplane in real-time, then such inaccuracy could have serious consequences.
It is also worth mentioning that the human mind itself can only make predictions based on its own experiences. And yet, a human might think that his or her predictions are always accurate. In fact, we often hear people say things like, ‘I knew it all along’ or ‘it had to happen this way.’ However, these behaviors do not indicate true knowledge and understanding at all.
In summary, humans often do not fully understand the nature of knowledge and belief. They can never really know if their conclusions are correct or incorrect.
An advanced AI system may be able to deduce certain conclusions and make predictions with greater accuracy than a human ever could. However, if such an AI were to claim certainty in its knowledge, one might say that it is only correct 80% of the time.
In fact, we might say that humans are correct 10% of the time, while an advanced AI is only correct 100% of the time. Of course, this assumes there is no chance for error in the system’s data or algorithms.
I think it is rather difficult to assess exactly how accurate models are. We can use standard statistical tools for assessing model accuracy of course, but I feel that these do not account for all the relevant considerations.
I am not sure whether all those considerations are really relevant to assessing model accuracy and I would like to know what other people think about this.