AI

• The recommended approach
  • ML
  • • Start with a simple algorithm that you can implement quickly. Implement it and test it on your cross-validation data.
    • Plot learning curves to decide if more data, more features, etc.
    • Error analysis
      • Manually examine the examples (in cross-validation set) that your algorithm made errors on. See if you spot any systematic trend in what type of examples it is making errors on.
  • DL
    • Try to have the same number of hidden units in every layer
      • Usually the more the better, but computational
    • ReLU variants
      • Softplus, leaky relu, prelu, relu6, elu
  • Common

    • Learning rate

    • < 1/sqrt(num_features)

    • Learning rate automation
      • https://www.tensorflow.org/api_docs/python/tf/keras/optimizers
        • Some optimizers support it
          
      • https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/schedules

    • Continue to verify and monitor your data since it may change for many reasons in reality

    • Stop learning in particular circumstances

      • E.g. service failure -> only a few users can access the service -> incomplete/incorrect data

  • Transfer learning
    • Where to cut?
    • By convention, we cut the source network after the convolutional layers and append a number of fully connected layers of our own. This is consistent with the view that convolutional layers are excellent feature extractors for the image domain
    • Do I make the source models weights trainable, as in allowing to change values during subsequent model training or do I make them constant? 
    • Leaving them constant, effectively treats the source model as a feature extractor. If your new data set is small, this is the recommended approach, at the risk of overfitting your data. 
    • The larger your data set is, the more confident you can be that letting the source network continue to train will not result in overfitting
    • Whether to make the pre-trained embeddings trainable or not. 
    • the primary factor to consider when making this decision is dataset size. The larger your dataset is, the less likely that letting the embeddings be trainable, will result in over-fitting.
    • Pretrained embedding
      • Tensorflow hub
        • E.g.
          
          • https://github.com/mungeol/training-data-analyst/blob/master/courses/machine_learning/deepdive/09_sequence/reusable-embeddings.ipynb
    • Pretrained model
      • Tensorflow hub
        • https://www.tensorflow.org/hub
      • Keras applications
        • https://keras.io/applications/

• Rules of Machine Learning: Best Practices for ML Engineering

• https://developers.google.com/machine-learning/guides/rules-of-ml

• http://martin.zinkevich.org/rules_of_ml/rules_of_ml.pdf

• See also

  • Do we Need Hundreds of Classifiers to Solve Real World Classification Problems?

  • http://jmlr.org/papers/volume15/delgado14a/delgado14a.pdf

  • Choosing the right estimator

  • https://scikit-learn.org/stable/tutorial/machine_learning_map/index.html