{"id":176,"date":"2024-07-22T12:23:36","date_gmt":"2024-07-22T16:23:36","guid":{"rendered":"https:\/\/www.econai.tech\/?page_id=176"},"modified":"2024-08-23T09:14:26","modified_gmt":"2024-08-23T13:14:26","slug":"neural-networks","status":"publish","type":"page","link":"https:\/\/tomomitanaka.ai\/?page_id=176","title":{"rendered":"Neural Networks"},"content":{"rendered":"\n
Safety by Design Expert’s Note<\/div>\n

Neural Networks are crucial for safety experts due to their:<\/p>\n\n\n\n

    \n
  1. Ability to handle complex, high-dimensional data<\/li>\n\n\n\n
  2. Automatic feature learning for subtle pattern detection<\/li>\n\n\n\n
  3. Adaptability to new scenarios<\/li>\n\n\n\n
  4. Scalability for growing safety systems<\/li>\n<\/ol>\n\n\n\n

    However, their “black box” nature presents challenges for interpretability and accountability in safety-critical applications.<\/p>\n<\/div>\n\n\n\n

    Neural Networks have revolutionized the field of machine learning, enabling breakthroughs in areas such as image recognition, natural language processing, and game playing. <\/p>\n\n\n\n

    Known for their ability to learn complex patterns and representations from data, Neural Networks have become a cornerstone of modern artificial intelligence.<\/p>\n\n\n\n

    In this post, we’ll explore the fundamentals of Neural Networks, with a particular focus on deep learning architectures and how they handle high-dimensional data. <\/p>\n\n\n\n

    We’ll also implement a simple Neural Network for our ongoing house price prediction task to see how it performs in practice.<\/p>\n\n\n\n

    You can find the complete code<\/a> in my GitHub repository.<\/p>\n\n\n\n

    Contents<\/h4>\n\n\n\n
      \n
    1. Understanding Neural Networks<\/li>\n\n\n\n
    2. The Power of Deep Learning<\/li>\n\n\n\n
    3. Neural Networks in High-Dimensional Spaces<\/li>\n\n\n\n
    4. Implementing a Neural Network for House Price Prediction<\/li>\n\n\n\n
    5. Comparing Neural Networks with Other Models<\/li>\n\n\n\n
    6. Conclusion<\/li>\n<\/ol>\n\n\n\n
      <\/div><\/span>
      \n

      Neural Networks<\/strong>
      Neural Networks are a type of machine learning model inspired by the human brain. They consist of layers of interconnected nodes (neurons) that process input data to make predictions or decisions. Each layer transforms the data, allowing the network to learn complex patterns.<\/p>\n<\/div><\/div><\/div>\n\n\n\n

      <\/div><\/span>
      \n

      Interpretability<\/strong>
      Interpretability is the ability to understand and explain how a machine learning model makes its predictions. It is crucial in ensuring that the model’s decisions can be trusted, especially in critical applications where understanding the decision-making process is essential.<\/p>\n<\/div><\/div><\/div>\n\n\n\n

      1. Understanding Neural Networks<\/h3>\n\n\n\n

      Neural Networks are a class of machine learning algorithms inspired by the structure and function of the human brain. <\/p>\n\n\n\n

      They consist of interconnected nodes (neurons) organized in layers:<\/p>\n\n\n\n

        \n
      • Input Layer: Receives the initial data<\/li>\n\n\n\n
      • Hidden Layers: Process the data through a series of transformations<\/li>\n\n\n\n
      • Output Layer: Produces the final prediction or classification<\/li>\n<\/ul>\n\n\n\n

        <\/p>\n\n\n\n

        Key aspects of Neural Networks<\/div>\n

        Weights and Biases<\/strong>: Adjustable parameters that determine the strength of connections between neurons<\/p>\n\n\n\n

        Activation Functions<\/strong>: Non-linear functions that introduce complexity and enable the network to learn non-linear relationships<\/p>\n\n\n\n

        Backpropagation<\/strong>: The algorithm used to update weights based on the error of predictions<\/p>\n\n\n\n

        Gradient Descent<\/strong>: The optimization technique used to minimize the error during training<\/p>\n<\/div>\n\n\n\n

        <\/div><\/span>
        \n

        Backpropagation<\/strong>
        Backpropagation is the algorithm used in Neural Networks to update the model’s weights based on the error of predictions. It works by propagating the error backward through the network, allowing the model to learn and improve over time.<\/p>\n<\/div><\/div><\/div>\n\n\n\n

        <\/div><\/span>
        \n

        Activation Function<\/strong>
        An activation function is a mathematical function used in Neural Networks to introduce non-linearity into the model. It helps the network learn complex patterns by allowing neurons to be activated or “fired” based on the input they receive.<\/p>\n<\/div><\/div><\/div>\n\n\n\n

        2. The Power of Deep Learning<\/h3>\n\n\n\n

        Deep Learning refers to Neural Networks with multiple hidden layers. This depth allows the network to learn hierarchical representations of the data:<\/p>\n\n\n\n

          \n
        • Lower layers learn simple features<\/li>\n\n\n\n
        • Higher layers combine these to form more complex, abstract features<\/li>\n<\/ul>\n\n\n\n

          This hierarchical learning is particularly powerful for tasks involving high-dimensional data like images, text, or complex numerical datasets.<\/p>\n\n\n\n

          3. Neural Networks in High-Dimensional Spaces<\/h3>\n\n\n\n

          Neural Networks excel at handling high-dimensional data due to several key characteristics:<\/p>\n\n\n\n

          \n

          Automatic Feature Extraction<\/strong>: Deep networks learn relevant features directly from the data, reducing the need for manual feature engineering<\/p>\n\n\n\n

          Non-linear Transformations<\/strong>: Multiple layers with non-linear activation functions can capture complex relationships in the data<\/p>\n\n\n\n

          Dimensionality Reduction<\/strong>: Hidden layers can act as a form of non-linear dimensionality reduction, learning compact representations of the input data<\/p>\n\n\n\n

          Scalability<\/strong>: Neural Networks can be scaled to handle extremely large datasets and high-dimensional inputs<\/p>\n<\/div>\n\n\n\n

          These properties make Neural Networks particularly well-suited for tasks like image classification, natural language processing, and complex regression problems.<\/p>\n\n\n\n

          4. Implementing a Neural Network for House Price Prediction<\/h3>\n\n\n\n

          Let’s implement a simple Neural Network for our house price prediction task using TensorFlow and Keras:<\/p>\n\n\n\n

          Python<\/span><\/path><\/path><\/svg><\/span>
          # Create and train the model<\/span><\/span>\nnn_model = MLPRegressor(<\/span>hidden_layer_sizes<\/span>=(<\/span>64<\/span>, <\/span>32<\/span>, <\/span>16<\/span>), <\/span><\/span>\n                        <\/span>activation<\/span>=<\/span>'relu'<\/span>, <\/span><\/span>\n                        <\/span>solver<\/span>=<\/span>'adam'<\/span>, <\/span><\/span>\n                        <\/span>alpha<\/span>=<\/span>0.0001<\/span>,<\/span><\/span>\n                        <\/span>batch_size<\/span>=<\/span>32<\/span>, <\/span><\/span>\n                        <\/span>learning_rate_init<\/span>=<\/span>0.001<\/span>,<\/span><\/span>\n                        <\/span>max_iter<\/span>=<\/span>1000<\/span>,<\/span><\/span>\n                        <\/span>random_state<\/span>=<\/span>42<\/span>,<\/span><\/span>\n                        <\/span>verbose<\/span>=<\/span>True<\/span>)<\/span><\/span>\n<\/span>\nnn_model.fit(X_train, y_train)<\/span><\/span>\n<\/span>\n# Make predictions<\/span><\/span>\ny_pred = nn_model.predict(X_test)<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
          5. Model Performance and Evaluation<\/h3>\n\n\n\n
          Let’s evaluate our Neural Network model and compare it with our previous models.<\/p>\n\n\n\n
          <\/td>MAE
          (thousand)<\/td>          		MSE 
          (million)<\/td>          		RMSE
          (thousand)<\/td>          		MAPE<\/td>MedAE
          (thousand)<\/td>          		R\u00b2<\/td><\/tr>
          Linear<\/td>18.6
          (\u00b12.6)<\/td>          		1,377 
          (\u00b1982)<\/td>          		36.4
          (\u00b114.4)<\/td>          		11.04% 
          (\u00b10.98%)<\/td>          		11.6 
          (\u00b11.2)<\/td>          		0.780 
          (\u00b10.178)<\/td><\/tr>
          Ridge<\/td>16.8
           (\u00b12.6)<\/td>          		853
          (\u00b1615)<\/td>          		28.7
          (\u00b110.6)<\/td>          		9.80% 
          (\u00b11.09%)<\/td>          		11.5
          (\u00b11.8)<\/td>          		0.867 
          (\u00b10.073)<\/td><\/tr>
          Elastic Net<\/td>16.6
          (\u00b12.4)<\/td>          		834
          (\u00b1611)<\/td>          		28.4
          (\u00b110.7)<\/td>          		9.86% 
          (\u00b11.13%)<\/td>          		11.3
          (\u00b11.6)<\/td>          		0.870 <\/strong>
          (\u00b10.074)<\/td><\/tr>
          Random Forest<\/td>17.5
          (\u00b12.0)<\/td>          		927
          (\u00b1506)<\/td>          		30.2
          (\u00b1 8.1)<\/td>          		0.10% 
          (\u00b10.01%)<\/td>          		11.1
          (\u00b12.0)<\/td>          		0.853 
          (\u00b10.065)<\/td><\/tr>
          GBDT<\/td>16.8
          (\u00b12.4)<\/td>          		953 
          (\u00b1595)<\/td>          		30.5
          (\u00b19.9)<\/td>          		0.09%<\/strong> 
          (\u00b10.02%)<\/td>          		10.3
          (\u00b11.8)<\/td>          		0.839 
          (\u00b10.094)<\/td><\/tr>
          XGBoost<\/td>17.8
          (\u00b12.4)<\/td>          		1,177 (\u00b1748)<\/td>33.9
           (\u00b111.1)<\/td>          		0.10% 
          (\u00b10.02%)<\/td>          		10.8 
          (\u00b12.2)<\/td>          		0.806
          (\u00b10.078)<\/td><\/tr>
          LightGBM<\/td>16.7
          (\u00b13.4)<\/td>          		838 <\/strong>
          (\u00b1513)<\/td>          		28.6
           (\u00b19.1)<\/td>          		0.09%<\/strong> 
          (\u00b10.02%)<\/td>          		10.8
          (\u00b10.5)<\/td>          		0.860 
          (\u00b10.071)<\/td><\/tr>
          SVM 
          after kernel
          selection<\/td>          		15.9<\/strong>
          (\u00b11.2)<\/td>          		800<\/strong>
          (\u00b1296)<\/td>          		28.3<\/strong>
          (\u00b15.2)<\/td>          		9.19%
          (\u00b10.85%)<\/td>          		9.9 <\/strong>
          (\u00b10.8)<\/td>          		0.869
          (\u00b10.037)<\/td><\/tr>
          Neural
          Network<\/td>          		16.5 (\u00b10.9)<\/td>1,005 (\u00b1318)<\/td>30.1 
          (\u00b14.5)<\/td>          		9.61% (\u00b10.64%)<\/td>10.8 (\u00b10.4)<\/td>0.834
          (\u00b10.060)<\/td><\/tr><\/tbody><\/table>

          <\/figcaption><\/figure>\n\n\n\n
          Mean Absolute Error (MAE)<\/h5>\n\n\n\n
          The Neural Network performs well with an MAE of 16.5, which is better than most models except for SVM (15.9) and Elastic Net (16.6). It’s particularly better than the simpler Linear model and tree-based models like Random Forest and XGBoost.<\/p>\n\n\n\n
          Mean Squared Error (MSE) and Root Mean Squared Error (RMSE)<\/h5>\n\n\n\n
          The Neural Network’s performance is middle-of-the-pack here. With an MSE of 1,005 million and RMSE of 30.1, it’s better than Linear and XGBoost, but not as good as Ridge, Elastic Net, LightGBM, or SVM. This suggests that while it’s generally accurate, it might be making some larger errors on certain predictions.<\/p>\n\n\n\n
          Mean Absolute Percentage Error (MAPE)<\/h5>\n\n\n\n
          At 9.61%, the Neural Network’s MAPE is competitive with the linear models (Ridge and Elastic Net) and better than the simple Linear model. However, it’s notably higher than the tree-based models (GBDT, XGBoost, LightGBM).<\/p>\n\n\n\n
          Median Absolute Error (MedAE)<\/h5>\n\n\n\n
          The Neural Network’s MedAE of 10.8 is better than the Linear, Ridge, and Elastic Net models, but slightly worse than the tree-based models and SVM. This suggests that for at least half of the predictions, the Neural Network is quite accurate.<\/p>\n\n\n\n
          R-squared (R\u00b2)<\/h5>\n\n\n\n
          With an R\u00b2 of 0.834, the Neural Network explains a good amount of variance in the data. It outperforms the simple Linear model and XGBoost but falls short of the top performers like Elastic Net, Ridge, and SVM.<\/p>\n\n\n\n
          Standard Errors<\/h5>\n\n\n\n
          The Neural Network generally has lower standard errors compared to most other models, indicating more consistent performance across different subsets of the data.<\/p>\n\n\n\n
          Overall observations<\/h4>\n\n\n\n
          The Neural Network performs competitively across all metrics, consistently beating the simple Linear model and often outperforming tree-based methods like Random Forest and XGBoost.<\/p>\n\n\n\n
          It doesn’t quite reach the top performance of models like Elastic Net, Ridge, or SVM after kernel selection, but it’s not far behind.<\/p>\n\n\n\n
          The Neural Network shows more consistent performance (lower standard errors) compared to most other models, which could be valuable in real-world applications.<\/p>\n\n\n\n
          I tried to further optimize the Neural Network using Grid Search and RandomizedSearchCV. However, the improvements were marginal, and the model still fell slightly short compared to the top performers like Elastic Net and SVM after kernel selection.<\/p>\n\n\n\n
          For this particular problem, simpler models like Elastic Net or well-tuned SVMs seem to perform slightly better, suggesting that the relationship between features and house prices might be relatively linear or that the neural network might benefit from further tuning.<\/p>\n\n\n\n
          Conclusion<\/h3>\n\n\n\n
          Our exploration of Neural Networks, in the context of house price prediction, has revealed both the power and the nuances of this advanced machine learning technique.<\/p>\n\n\n\n
          Key Takeaways<\/h4>\n\n\n\n
          Versatility<\/strong>: Neural Networks demonstrated competitive performance across various metrics, showcasing their ability to handle complex, high-dimensional data without extensive feature engineering.<\/p>\n\n\n\n
          Consistency<\/strong>: The lower standard errors in the Neural Network’s performance metrics indicate a more stable and reliable model across different subsets of the data.<\/p>\n\n\n\n
          Comparative Performance<\/strong>: While not decisively outperforming all other models, the Neural Network held its own against both simpler linear models and more complex tree-based algorithms.<\/p>\n\n\n\n
          Complexity vs. Performance<\/strong>: In this specific house price prediction task, simpler models like Elastic Net and well-tuned SVMs slightly edged out the Neural Network, suggesting that the underlying relationships in the data might be more linear than initially assumed.<\/p>\n\n\n\n
          Optimization Challenges<\/strong>: Notably, attempts to further optimize the Neural Network using Grid Search and RandomizedSearchCV did not yield significant improvements. This underscores the robustness of our initial model and highlights the challenges in fine-tuning neural networks for certain types of data.<\/p>\n\n\n\n
          Reflections<\/h4>\n\n\n\n
          The performance of Neural Networks in this task, along with the results from our optimization attempts, underscores an important principle in machine learning: the most complex model is not always the best choice, and sometimes, initial configurations can be surprisingly effective. <\/p>\n\n\n\n
          While Neural Networks excel in capturing intricate patterns, especially in very large and complex datasets, they may not always provide significant advantages in more structured, tabular data scenarios like our house price prediction task.<\/p>\n\n\n\n
          This study also highlights the importance of comprehensive model evaluation and the value of exploring various optimization techniques. By comparing the Neural Network against a variety of other algorithms and attempting different optimization strategies, we gained valuable insights into the nature of our data, the relative strengths of different approaches, and the limitations of model tuning in certain contexts.<\/p>\n\n\n\n
          \n
          While Neural Networks didn’t dramatically outperform other models in this specific task, and further optimization proved challenging, their strong and consistent performance makes them a valuable tool in any data scientist’s toolkit.<\/p>\n\n\n\n
          This experience reinforces the importance of understanding the nature of your data and the problem at hand when selecting and tuning machine learning models. <\/p>\n\n\n\n
          It also highlights the value of thorough experimentation and the need to balance model complexity with actual performance gains.<\/p>\n<\/div>\n\n\n\n
          <\/i><\/div>
          \n
          Understanding and Managing the “Black Box” Nature of Neural Networks<\/strong><\/p>\n\n\n\n
          The Challenge:<\/strong> Neural Networks, while powerful, are often criticized for their “black box” nature\u2014meaning their decision-making processes are not easily interpretable. This lack of transparency can pose significant challenges in safety-critical applications, where understanding the reasoning behind a model’s predictions is crucial for accountability and risk management.<\/p>\n\n\n\n
          Improving Interpretability:<\/strong> To address this, techniques like SHAP (Shapley Additive Explanations) and LIME (Local Interpretable Model-agnostic Explanations) can be employed. SHAP provides a way to explain the output of any machine learning model by attributing the contribution of each feature to the final prediction. LIME, on the other hand, approximates the model locally with an interpretable model, offering insights into the reasons behind individual predictions.<\/p>\n\n\n\n
          Managing Risks:<\/strong> In safety-critical applications, where decisions could have significant consequences, it\u2019s essential to apply these interpretability techniques. Doing so allows for more transparent model audits, facilitates the identification of potential biases, and supports more informed decision-making processes. By making Neural Networks more interpretable, we can ensure that these models are not only effective but also align with ethical and safety standards.<\/p>\n<\/div><\/div>\n\n\n\n
          <\/p>\n","protected":false},"excerpt":{"rendered":"
          Neural Networks have revolutionized the field of machine learning, enabling breakthroughs in areas such as image recognition, natural language processing, and game playing. Known for their ability to learn complex patterns and representations from data, Neural Networks have become a cornerstone of modern artificial intelligence. In this post, we#8217;ll explore the fundamentals of Neural Networks,<\/p>\n","protected":false},"author":1,"featured_media":21,"parent":107,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-176","page","type-page","status-publish","has-post-thumbnail","hentry"],"_links":{"self":[{"href":"https:\/\/tomomitanaka.ai\/index.php?rest_route=\/wp\/v2\/pages\/176","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/tomomitanaka.ai\/index.php?rest_route=\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/tomomitanaka.ai\/index.php?rest_route=\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/tomomitanaka.ai\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/tomomitanaka.ai\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=176"}],"version-history":[{"count":75,"href":"https:\/\/tomomitanaka.ai\/index.php?rest_route=\/wp\/v2\/pages\/176\/revisions"}],"predecessor-version":[{"id":5514,"href":"https:\/\/tomomitanaka.ai\/index.php?rest_route=\/wp\/v2\/pages\/176\/revisions\/5514"}],"up":[{"embeddable":true,"href":"https:\/\/tomomitanaka.ai\/index.php?rest_route=\/wp\/v2\/pages\/107"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/tomomitanaka.ai\/index.php?rest_route=\/wp\/v2\/media\/21"}],"wp:attachment":[{"href":"https:\/\/tomomitanaka.ai\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=176"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}