AT06_Multi_layer_Perceptron_Treinamento_e_Teste.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Multi-layer Perceptron - Treinamento e Teste.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/barauna-lo/CAP4213-Deep-Learning/blob/main/Multi_layer_Perceptron_Treinamento_e_Teste.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1i9n4mmcVybX"
      },
      "source": [
        "<img src=https://raw.githubusercontent.com/barauna-lo/CAP4213-Deep-Learning/main/logoinpe.png>\n",
        "\n",
        "\n",
        "# CAP-421-3: Aprendizado Profundo\n",
        "## Atividade2: Treinamento e Teste - MLP (Multi-layer Perceptron)\n",
        "\n",
        "* Luan Orion Baraúna \n",
        "\n",
        "* Renato Maximiniano \n",
        "\n",
        "* Vinicius Monego\n",
        "\n",
        "Multilayer perceptron neural networks are neural networks that follow the logic of perceptron networks proposed in 1958 by Frank Rosenblatt in which the perceptron is a binary classifier that maps its input $x$ (a real value vector) to an output value $f ( x )$ (a simple binary value).\n",
        "\n",
        "$$\n",
        "f(x)= \\begin{cases}1&{\\text{if }}w\\cdot x+b\\geq 0\\\\0&{\\text{else}}\n",
        "\\end{cases}\n",
        "$$\n",
        "\n",
        "\n",
        "Where $w$ is a real weight vector and $w \\cdot x $ is the dot product (which computes a sum with weights) and $b$ is the bias (\"bias\"), a constant term that does not depend on any input value and is added for training control. The main difference is associated with the amount of hidden layers between the input layer and the output layer.\n",
        "\n",
        "\n",
        "<img src = \"https://res-2.cloudinary.com/ht54oxr6q/image/upload/q_auto/v1/ghost-blog-images/title.png\" width =500>\n",
        "\n",
        "This notebook was built based on standard course [code](https://colab.research.google.com/drive/1B8e6R8J6T7-ahBte6NFXWiAX1Hun4MGO?usp=sharing). Here, we search for an optimal set of hyperparameters to improve the accuracy of the model prediction. For this, we use the [Optuna](https://optuna.org/) hyperparameter optimization library. Optuna uses a Bayesian optimization algorithm called [TPE](https://towardsdatascience.com/a-conceptual-explanation-of-bayesian-model-based-hyperparameter-optimization-for-machine-learning-b8172278050f).\n",
        "\n",
        "\n",
        "## _Importing Libraries_\n",
        "\n",
        "<img src='https://pandas.pydata.org/static/img/pandas_white.svg  ' width = 300>\n",
        "\n",
        "<img src='https://user-images.githubusercontent.com/1217238/65354639-dd928f80-dba4-11e9-833b-bc3e8c6a737d.png' width = 300>\n",
        "\n",
        "<img src='https://scipy-lectures.org/_images/scikit-learn-logo.png' width =300>\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "collapsed": true,
        "id": "QQoSlJ0MbLXr",
        "jupyter": {
          "outputs_hidden": true
        }
      },
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "from sklearn.neural_network import MLPClassifier\n",
        "from sklearn.metrics import classification_report, confusion_matrix\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "# Dowloadin the dataset\n",
        "#!wget https://www.dropbox.com/sh/ew55wdqbc519ayd/AAA7UQqs0opzEH_Rf7GLWEeea/tabela_painters.csv?dl=0\n",
        "\n",
        "#Importing all the data    \n",
        "#dados=pd.read_csv('tabela_painters.csv?dl=0')\n",
        "dados=pd.read_csv('https://www.dropbox.com/sh/ew55wdqbc519ayd/AAA7UQqs0opzEH_Rf7GLWEeea/tabela_painters.csv?dl=1') \n",
        "\n",
        "#Select some data\n",
        "dados2=dados.drop(dados.index[[0,1]])\n",
        "\n",
        "#Specifing the train and target collums\n",
        "x_treino = np.array(dados2[dados2.columns[1:2048]])\n",
        "y_treino = np.array(dados2[dados2.columns[-1]])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G6vrz6Dti1r-"
      },
      "source": [
        "## Function ```train_test_split```\n",
        "\n",
        "É uma função de seleção de **modelo do Sklearn** para dividir matrizes de dados em **dois subconjuntos**: dados de treinamento e dados de teste. Por padrão, essa função faz partições aleatórias para os dois conjuntos. Entretanto, é possível especificar um estado alaetório para a operação."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "80ARw_aGtrNs",
        "jupyter": {
          "source_hidden": true
        }
      },
      "source": [
        "```\n",
        "-------------------------------------------------------------------------------\n",
        "Função train_test_split(X, y, train-size=None, test_size=None,\n",
        "                        random_state=None, shuffle = True, stratify = None)\n",
        "-------------------------------------------------------------------------------\n",
        "Entrada: | X e y      --> São o conjunto de dados que serão utilizados. \n",
        "         | train_size --> Define o tamanho do conjunto de dados de treinamento.\n",
        "         |                Existem 3 opções: \n",
        "         |                     None --> padrão\n",
        "         |                     Int --> requer o número exato de amostras\n",
        "         |                     Float --> varua de 0.1 a 1.0.\n",
        "         | test_size --> Especifica o tamanho do conjunto de dados de teste. O  \n",
        "         |               estado padrão se adapta ao tamanho do treinamento.\n",
        "         |               Será definido como 0.25 se o tamanho do treinamento\n",
        "         |               for definido como padrão.\n",
        "         | radom_state --> O modo padrão executa uma divisão aleatória usando\n",
        "         |                 **np.random**. Como alternativa, você pode adicionar \n",
        "         |                 um número inteiro usando um número exato.\n",
        "         | shuffle --> Se deve ou não embaralhar os dados antes de dividir. \n",
        "         |             Se shuffle = False, stratify deve ser igual a None.\n",
        "         | stratify --> Se for igual a None, os dados são divididos de forma\n",
        "         | estratificados  \n",
        "---------|--------------------------------------------------------------------- \n",
        "Saída    | retorna os conjuntos de treinamento e teste\n",
        "         |      --> X_train, X_test, y_train e y_test\n",
        "-------------------------------------------------------------------------------\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Yei6Xu5wa7PR"
      },
      "source": [
        "x_train, x_test, y_train, y_test = train_test_split(x_treino, y_treino, stratify=y_treino, random_state=1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IN_UaV9PFMMK"
      },
      "source": [
        "<img src='https://optuna.org/assets/img/optuna-logo@2x.png' width =300>\n",
        "\n",
        "## Using Optuna \n",
        "\n",
        "https://optuna.org/\n",
        "\n",
        "Here we use Optuna to search the optimal hyperparameters of our model within a predetermined range.\n",
        "\n",
        "Optuna optimizes a function that returns the model accuracy (here called `objective`), and receives as input `trial` object which represents a set of hyperparameters. The model is retrained for every set of hyperparameters generated."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OvwQ5f2aBmrl"
      },
      "source": [
        "#%pip install optuna\n",
        "import optuna\n",
        "\n",
        "def objective(trial):\n",
        "    #Escolhendo os hiperâmetros                    #Seus nomes          #Intevalo que irão variar\n",
        "    actv               = trial.suggest_categorical('actv'              ,['relu', 'logistic', 'tanh'])\n",
        "    solvers            = trial.suggest_categorical('solvers'           ,['adam', 'sgd'])\n",
        "    alpha              = trial.suggest_float(      'alpha'             ,1e-4, 1e-2)\n",
        "    batch_size         = trial.suggest_categorical('batch_size'        ,[1, 16, 32, 64, 128])\n",
        "    learning_rate_init = trial.suggest_float(      'learning_rate_init', 1e-4, 1e-2)\n",
        "    max_iter           = trial.suggest_int(        'max_iter'          ,40, 400)\n",
        "    momentum           = trial.suggest_float(      'momentum'          ,1e-2, 0.9)\n",
        "    nl                 = trial.suggest_int(        'number_layers'     ,1, 5)\n",
        "    #loop para a variação do nº de neurônios escondidos\n",
        "    hn = []\n",
        "    for l in range(nl):\n",
        "        hn.append(trial.suggest_int('hidden_neurons_{}'.format(l), 50, 200))\n",
        "\n",
        "    net = MLPClassifier(hidden_layer_sizes=hn, \n",
        "                        activation=actv, \n",
        "                        solver=solvers, \n",
        "                        alpha=alpha, \n",
        "                        batch_size=batch_size, # learning_rate='constant', \n",
        "                        learning_rate_init=learning_rate_init, \n",
        "                        max_iter=max_iter,  \n",
        "                        verbose=True,  \n",
        "                        momentum=momentum, \n",
        "                        early_stopping=True, \n",
        "                        validation_fraction=0.1, \n",
        "                        n_iter_no_change=10)\n",
        "\n",
        "    net.fit(x_train,y_train)\n",
        "\n",
        "    predict = net.predict(x_test)\n",
        "\n",
        "    # print('Acurácia: {}%' .format(round(net.score(x_test, y_test) * 100)))\n",
        "\n",
        "    ac = []\n",
        "    accuracy = round(net.score(x_test, y_test) * 100,4)\n",
        "    ac.append(accuracy)\n",
        "    return accuracy"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KH5bRGuoFMML"
      },
      "source": [
        "### Executando o Modelo\n",
        "\n",
        "A `study` object is a collection of `trial`. It contains information related to the evolution of the optimization, including attributes such as the best parameters and best value. This object also contains the `optimize` method which starts the optimization of the function passed as parameter. \n",
        "\n",
        "During the application of Optuna the direction parameter was defined as maximize, so that the search is always for the highest accuracy value and the number of models to be generated was defined as 100.\n",
        "\n",
        "We also import the ```time``` module to inform how long these code runs. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "collapsed": true,
        "id": "zDRqdiWKECXs",
        "jupyter": {
          "outputs_hidden": true
        },
        "outputId": "3211a949-4b82-47f4-a333-4d08e8a2f688"
      },
      "source": [
        "import time\n",
        "\n",
        "inicio = time.time()\n",
        "#study = optuna.create_study()\n",
        "study = optuna.create_study(direction='maximize')\n",
        "study.optimize(objective, n_trials=100)\n",
        "\n",
        "fim = time.time()\n",
        "print('the code runs for ',round(((fim- inicio)/60/60),3),'horas')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 08:39:06,176]\u001b[0m A new study created in memory with name: no-name-a80d1f5f-31ad-45f2-bedf-a56733616dd2\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.78764134\n",
            "Validation score: 0.769231\n",
            "Iteration 2, loss = 0.69567040\n",
            "Validation score: 0.769231\n",
            "Iteration 3, loss = 0.64043004\n",
            "Validation score: 0.769231\n",
            "Iteration 4, loss = 0.61166524\n",
            "Validation score: 0.769231\n",
            "Iteration 5, loss = 0.58945427\n",
            "Validation score: 0.769231\n",
            "Iteration 6, loss = 0.57856497\n",
            "Validation score: 0.769231\n",
            "Iteration 7, loss = 0.56765149\n",
            "Validation score: 0.769231\n",
            "Iteration 8, loss = 0.56107349\n",
            "Validation score: 0.769231\n",
            "Iteration 9, loss = 0.55832305\n",
            "Validation score: 0.769231\n",
            "Iteration 10, loss = 0.55729052\n",
            "Validation score: 0.769231\n",
            "Iteration 11, loss = 0.55762154\n",
            "Validation score: 0.769231\n",
            "Iteration 12, loss = 0.55415889\n",
            "Validation score: 0.769231\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 08:40:50,267]\u001b[0m Trial 0 finished with value: 76.7808 and parameters: {'actv': 'logistic', 'solvers': 'sgd', 'alpha': 0.003460613828701539, 'batch_size': 1, 'learning_rate_init': 0.006437299724302034, 'max_iter': 181, 'momentum': 0.7243658748647253, 'number_layers': 4, 'hidden_neurons_0': 92, 'hidden_neurons_1': 117, 'hidden_neurons_2': 109, 'hidden_neurons_3': 86}. Best is trial 0 with value: 76.7808.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.42764793\n",
            "Validation score: 0.910769\n",
            "Iteration 2, loss = 0.20436612\n",
            "Validation score: 0.923077\n",
            "Iteration 3, loss = 0.19721677\n",
            "Validation score: 0.892308\n",
            "Iteration 4, loss = 0.20719517\n",
            "Validation score: 0.904615\n",
            "Iteration 5, loss = 0.19377064\n",
            "Validation score: 0.895385\n",
            "Iteration 6, loss = 0.19559398\n",
            "Validation score: 0.910769\n",
            "Iteration 7, loss = 0.17969026\n",
            "Validation score: 0.923077\n",
            "Iteration 8, loss = 0.17988162\n",
            "Validation score: 0.923077\n",
            "Iteration 9, loss = 0.19089752\n",
            "Validation score: 0.898462\n",
            "Iteration 10, loss = 0.18001849\n",
            "Validation score: 0.923077\n",
            "Iteration 11, loss = 0.17567525\n",
            "Validation score: 0.923077\n",
            "Iteration 12, loss = 0.17818660\n",
            "Validation score: 0.920000\n",
            "Iteration 13, loss = 0.17781119\n",
            "Validation score: 0.923077\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 08:40:59,451]\u001b[0m Trial 1 finished with value: 91.2118 and parameters: {'actv': 'tanh', 'solvers': 'adam', 'alpha': 0.0008393313138610084, 'batch_size': 32, 'learning_rate_init': 0.0016756675376930757, 'max_iter': 102, 'momentum': 0.48661661016446206, 'number_layers': 1, 'hidden_neurons_0': 190}. Best is trial 1 with value: 91.2118.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.35804057\n",
            "Validation score: 0.916923\n",
            "Iteration 2, loss = 0.25046895\n",
            "Validation score: 0.904615\n",
            "Iteration 3, loss = 0.27336631\n",
            "Validation score: 0.904615\n",
            "Iteration 4, loss = 0.25378062\n",
            "Validation score: 0.873846\n",
            "Iteration 5, loss = 0.25866194\n",
            "Validation score: 0.916923\n",
            "Iteration 6, loss = 0.24923201\n",
            "Validation score: 0.916923\n",
            "Iteration 7, loss = 0.25441777\n",
            "Validation score: 0.892308\n",
            "Iteration 8, loss = 0.27119840\n",
            "Validation score: 0.929231\n",
            "Iteration 9, loss = 0.25990774\n",
            "Validation score: 0.898462\n",
            "Iteration 10, loss = 0.25346197\n",
            "Validation score: 0.926154\n",
            "Iteration 11, loss = 0.25674224\n",
            "Validation score: 0.895385\n",
            "Iteration 12, loss = 0.25325311\n",
            "Validation score: 0.913846\n",
            "Iteration 13, loss = 0.24035834\n",
            "Validation score: 0.910769\n",
            "Iteration 14, loss = 0.24446266\n",
            "Validation score: 0.923077\n",
            "Iteration 15, loss = 0.24793193\n",
            "Validation score: 0.926154\n",
            "Iteration 16, loss = 0.25573411\n",
            "Validation score: 0.929231\n",
            "Iteration 17, loss = 0.25155699\n",
            "Validation score: 0.901538\n",
            "Iteration 18, loss = 0.23496660\n",
            "Validation score: 0.920000\n",
            "Iteration 19, loss = 0.25292672\n",
            "Validation score: 0.913846\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 08:41:06,189]\u001b[0m Trial 2 finished with value: 91.7669 and parameters: {'actv': 'tanh', 'solvers': 'adam', 'alpha': 0.006896394459207602, 'batch_size': 32, 'learning_rate_init': 0.006395017907646915, 'max_iter': 363, 'momentum': 0.42729160503401536, 'number_layers': 2, 'hidden_neurons_0': 82, 'hidden_neurons_1': 78}. Best is trial 2 with value: 91.7669.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.56668172\n",
            "Validation score: 0.769231\n",
            "Iteration 2, loss = 0.47232610\n",
            "Validation score: 0.916923\n",
            "Iteration 3, loss = 0.26391193\n",
            "Validation score: 0.929231\n",
            "Iteration 4, loss = 0.22640864\n",
            "Validation score: 0.929231\n",
            "Iteration 5, loss = 0.21491553\n",
            "Validation score: 0.916923\n",
            "Iteration 6, loss = 0.20050716\n",
            "Validation score: 0.923077\n",
            "Iteration 7, loss = 0.20711874\n",
            "Validation score: 0.929231\n",
            "Iteration 8, loss = 0.19175755\n",
            "Validation score: 0.941538\n",
            "Iteration 9, loss = 0.19378680\n",
            "Validation score: 0.938462\n",
            "Iteration 10, loss = 0.18922275\n",
            "Validation score: 0.932308\n",
            "Iteration 11, loss = 0.19589932\n",
            "Validation score: 0.941538\n",
            "Iteration 12, loss = 0.19876021\n",
            "Validation score: 0.913846\n",
            "Iteration 13, loss = 0.19350935\n",
            "Validation score: 0.932308\n",
            "Iteration 14, loss = 0.18465954\n",
            "Validation score: 0.916923\n",
            "Iteration 15, loss = 0.19730063\n",
            "Validation score: 0.929231\n",
            "Iteration 16, loss = 0.18618269\n",
            "Validation score: 0.941538\n",
            "Iteration 17, loss = 0.17362177\n",
            "Validation score: 0.932308\n",
            "Iteration 18, loss = 0.17515132\n",
            "Validation score: 0.938462\n",
            "Iteration 19, loss = 0.17896282\n",
            "Validation score: 0.861538\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 08:41:10,051]\u001b[0m Trial 3 finished with value: 91.6744 and parameters: {'actv': 'logistic', 'solvers': 'adam', 'alpha': 0.0017724558159625723, 'batch_size': 128, 'learning_rate_init': 0.006997855747604874, 'max_iter': 357, 'momentum': 0.7922241536897467, 'number_layers': 4, 'hidden_neurons_0': 59, 'hidden_neurons_1': 169, 'hidden_neurons_2': 170, 'hidden_neurons_3': 56}. Best is trial 2 with value: 91.7669.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.38764843\n",
            "Validation score: 0.907692\n",
            "Iteration 2, loss = 0.25624886\n",
            "Validation score: 0.898462\n",
            "Iteration 3, loss = 0.22312330\n",
            "Validation score: 0.935385\n",
            "Iteration 4, loss = 0.21358414\n",
            "Validation score: 0.935385\n",
            "Iteration 5, loss = 0.21128717\n",
            "Validation score: 0.941538\n",
            "Iteration 6, loss = 0.20634153\n",
            "Validation score: 0.910769\n",
            "Iteration 7, loss = 0.20433317\n",
            "Validation score: 0.938462\n",
            "Iteration 8, loss = 0.19944907\n",
            "Validation score: 0.935385\n",
            "Iteration 9, loss = 0.19899003\n",
            "Validation score: 0.843077\n",
            "Iteration 10, loss = 0.20041735\n",
            "Validation score: 0.926154\n",
            "Iteration 11, loss = 0.19634599\n",
            "Validation score: 0.926154\n",
            "Iteration 12, loss = 0.19531851\n",
            "Validation score: 0.800000\n",
            "Iteration 13, loss = 0.19729089\n",
            "Validation score: 0.941538\n",
            "Iteration 14, loss = 0.19174029\n",
            "Validation score: 0.941538\n",
            "Iteration 15, loss = 0.19065481\n",
            "Validation score: 0.935385\n",
            "Iteration 16, loss = 0.19174386\n",
            "Validation score: 0.932308\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 08:41:15,919]\u001b[0m Trial 4 finished with value: 92.5069 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.0019880095067829713, 'batch_size': 32, 'learning_rate_init': 0.006140199221189514, 'max_iter': 261, 'momentum': 0.6186060911085853, 'number_layers': 4, 'hidden_neurons_0': 61, 'hidden_neurons_1': 139, 'hidden_neurons_2': 153, 'hidden_neurons_3': 155}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.46040205\n",
            "Validation score: 0.873846\n",
            "Iteration 2, loss = 0.31299075\n",
            "Validation score: 0.867692\n",
            "Iteration 3, loss = 0.27760910\n",
            "Validation score: 0.913846\n",
            "Iteration 4, loss = 0.25530978\n",
            "Validation score: 0.913846\n",
            "Iteration 5, loss = 0.24324133\n",
            "Validation score: 0.923077\n",
            "Iteration 6, loss = 0.23701421\n",
            "Validation score: 0.926154\n",
            "Iteration 7, loss = 0.23650307\n",
            "Validation score: 0.904615\n",
            "Iteration 8, loss = 0.23049845\n",
            "Validation score: 0.926154\n",
            "Iteration 9, loss = 0.22726187\n",
            "Validation score: 0.938462\n",
            "Iteration 10, loss = 0.22652102\n",
            "Validation score: 0.929231\n",
            "Iteration 11, loss = 0.22253553\n",
            "Validation score: 0.923077\n",
            "Iteration 12, loss = 0.21950686\n",
            "Validation score: 0.932308\n",
            "Iteration 13, loss = 0.21664678\n",
            "Validation score: 0.926154\n",
            "Iteration 14, loss = 0.22112605\n",
            "Validation score: 0.876923\n",
            "Iteration 15, loss = 0.21848448\n",
            "Validation score: 0.929231\n",
            "Iteration 16, loss = 0.21616648\n",
            "Validation score: 0.935385\n",
            "Iteration 17, loss = 0.21356495\n",
            "Validation score: 0.907692\n",
            "Iteration 18, loss = 0.21513455\n",
            "Validation score: 0.923077\n",
            "Iteration 19, loss = 0.21321580\n",
            "Validation score: 0.935385\n",
            "Iteration 20, loss = 0.20946141\n",
            "Validation score: 0.926154\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 08:41:30,461]\u001b[0m Trial 5 finished with value: 92.2294 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.003137675399525761, 'batch_size': 32, 'learning_rate_init': 0.005898402774404304, 'max_iter': 257, 'momentum': 0.31202497094823173, 'number_layers': 5, 'hidden_neurons_0': 164, 'hidden_neurons_1': 186, 'hidden_neurons_2': 88, 'hidden_neurons_3': 91, 'hidden_neurons_4': 107}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.34550984\n",
            "Validation score: 0.916923\n",
            "Iteration 2, loss = 0.28964893\n",
            "Validation score: 0.920000\n",
            "Iteration 3, loss = 0.27703251\n",
            "Validation score: 0.950769\n",
            "Iteration 4, loss = 0.26825250\n",
            "Validation score: 0.947692\n",
            "Iteration 5, loss = 0.26340714\n",
            "Validation score: 0.935385\n",
            "Iteration 6, loss = 0.26082809\n",
            "Validation score: 0.947692\n",
            "Iteration 7, loss = 0.25981601\n",
            "Validation score: 0.950769\n",
            "Iteration 8, loss = 0.25495374\n",
            "Validation score: 0.944615\n",
            "Iteration 9, loss = 0.25619427\n",
            "Validation score: 0.947692\n",
            "Iteration 10, loss = 0.25301261\n",
            "Validation score: 0.944615\n",
            "Iteration 11, loss = 0.25532876\n",
            "Validation score: 0.947692\n",
            "Iteration 12, loss = 0.24932949\n",
            "Validation score: 0.935385\n",
            "Iteration 13, loss = 0.24805802\n",
            "Validation score: 0.892308\n",
            "Iteration 14, loss = 0.24731827\n",
            "Validation score: 0.944615\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 08:41:42,088]\u001b[0m Trial 6 finished with value: 91.3969 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.0069275168930882786, 'batch_size': 16, 'learning_rate_init': 0.009328010874133569, 'max_iter': 194, 'momentum': 0.15371376553365101, 'number_layers': 1, 'hidden_neurons_0': 163}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.37350080\n",
            "Validation score: 0.910769\n",
            "Iteration 2, loss = 0.27390217\n",
            "Validation score: 0.916923\n",
            "Iteration 3, loss = 0.25146474\n",
            "Validation score: 0.916923\n",
            "Iteration 4, loss = 0.23750311\n",
            "Validation score: 0.923077\n",
            "Iteration 5, loss = 0.22592133\n",
            "Validation score: 0.929231\n",
            "Iteration 6, loss = 0.21795581\n",
            "Validation score: 0.932308\n",
            "Iteration 7, loss = 0.21238400\n",
            "Validation score: 0.929231\n",
            "Iteration 8, loss = 0.20940755\n",
            "Validation score: 0.926154\n",
            "Iteration 9, loss = 0.20746138\n",
            "Validation score: 0.920000\n",
            "Iteration 10, loss = 0.20302826\n",
            "Validation score: 0.923077\n",
            "Iteration 11, loss = 0.20120146\n",
            "Validation score: 0.929231\n",
            "Iteration 12, loss = 0.20072167\n",
            "Validation score: 0.929231\n",
            "Iteration 13, loss = 0.19778095\n",
            "Validation score: 0.929231\n",
            "Iteration 14, loss = 0.19795303\n",
            "Validation score: 0.935385\n",
            "Iteration 15, loss = 0.19573085\n",
            "Validation score: 0.929231\n",
            "Iteration 16, loss = 0.19439675\n",
            "Validation score: 0.923077\n",
            "Iteration 17, loss = 0.19066999\n",
            "Validation score: 0.926154\n",
            "Iteration 18, loss = 0.19170115\n",
            "Validation score: 0.935385\n",
            "Iteration 19, loss = 0.19258012\n",
            "Validation score: 0.935385\n",
            "Iteration 20, loss = 0.19184085\n",
            "Validation score: 0.932308\n",
            "Iteration 21, loss = 0.18730906\n",
            "Validation score: 0.923077\n",
            "Iteration 22, loss = 0.18675196\n",
            "Validation score: 0.935385\n",
            "Iteration 23, loss = 0.18793132\n",
            "Validation score: 0.932308\n",
            "Iteration 24, loss = 0.18805706\n",
            "Validation score: 0.935385\n",
            "Iteration 25, loss = 0.18850120\n",
            "Validation score: 0.935385\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 08:41:50,431]\u001b[0m Trial 7 finished with value: 91.9519 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.002934555726472646, 'batch_size': 64, 'learning_rate_init': 0.00822444234552473, 'max_iter': 92, 'momentum': 0.44592918908218065, 'number_layers': 2, 'hidden_neurons_0': 170, 'hidden_neurons_1': 177}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.34653741\n",
            "Validation score: 0.864615\n",
            "Iteration 2, loss = 0.27011895\n",
            "Validation score: 0.892308\n",
            "Iteration 3, loss = 0.26914096\n",
            "Validation score: 0.895385\n",
            "Iteration 4, loss = 0.31038013\n",
            "Validation score: 0.880000\n",
            "Iteration 5, loss = 0.27926863\n",
            "Validation score: 0.898462\n",
            "Iteration 6, loss = 0.29062871\n",
            "Validation score: 0.876923\n",
            "Iteration 7, loss = 0.30268084\n",
            "Validation score: 0.855385\n",
            "Iteration 8, loss = 0.30472753\n",
            "Validation score: 0.892308\n",
            "Iteration 9, loss = 0.29668845\n",
            "Validation score: 0.855385\n",
            "Iteration 10, loss = 0.33813843\n",
            "Validation score: 0.864615\n",
            "Iteration 11, loss = 0.37526900\n",
            "Validation score: 0.867692\n",
            "Iteration 12, loss = 0.35241535\n",
            "Validation score: 0.861538\n",
            "Iteration 13, loss = 0.34877242\n",
            "Validation score: 0.904615\n",
            "Iteration 14, loss = 0.31884882\n",
            "Validation score: 0.873846\n",
            "Iteration 15, loss = 0.34144864\n",
            "Validation score: 0.873846\n",
            "Iteration 16, loss = 0.32461490\n",
            "Validation score: 0.886154\n",
            "Iteration 17, loss = 0.31316520\n",
            "Validation score: 0.873846\n",
            "Iteration 18, loss = 0.29764456\n",
            "Validation score: 0.923077\n",
            "Iteration 19, loss = 0.31114745\n",
            "Validation score: 0.880000\n",
            "Iteration 20, loss = 0.33309861\n",
            "Validation score: 0.861538\n",
            "Iteration 21, loss = 0.31909777\n",
            "Validation score: 0.910769\n",
            "Iteration 22, loss = 0.33714084\n",
            "Validation score: 0.883077\n",
            "Iteration 23, loss = 0.30437649\n",
            "Validation score: 0.870769\n",
            "Iteration 24, loss = 0.29865388\n",
            "Validation score: 0.907692\n",
            "Iteration 25, loss = 0.31068698\n",
            "Validation score: 0.861538\n",
            "Iteration 26, loss = 0.31889447\n",
            "Validation score: 0.883077\n",
            "Iteration 27, loss = 0.30949938\n",
            "Validation score: 0.913846\n",
            "Iteration 28, loss = 0.30091669\n",
            "Validation score: 0.901538\n",
            "Iteration 29, loss = 0.30524598\n",
            "Validation score: 0.910769\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 08:41:59,640]\u001b[0m Trial 8 finished with value: 91.3043 and parameters: {'actv': 'tanh', 'solvers': 'adam', 'alpha': 0.0054830690866607985, 'batch_size': 32, 'learning_rate_init': 0.009883002266742309, 'max_iter': 248, 'momentum': 0.49012049704505806, 'number_layers': 1, 'hidden_neurons_0': 86}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 1.67950171\n",
            "Validation score: 0.916923\n",
            "Iteration 2, loss = 1.42102169\n",
            "Validation score: 0.910769\n",
            "Iteration 3, loss = 1.22627556\n",
            "Validation score: 0.916923\n",
            "Iteration 4, loss = 1.06024346\n",
            "Validation score: 0.916923\n",
            "Iteration 5, loss = 0.92742883\n",
            "Validation score: 0.916923\n",
            "Iteration 6, loss = 0.81188345\n",
            "Validation score: 0.916923\n",
            "Iteration 7, loss = 0.71660602\n",
            "Validation score: 0.916923\n",
            "Iteration 8, loss = 0.63502102\n",
            "Validation score: 0.913846\n",
            "Iteration 9, loss = 0.56575889\n",
            "Validation score: 0.920000\n",
            "Iteration 10, loss = 0.50939616\n",
            "Validation score: 0.920000\n",
            "Iteration 11, loss = 0.46350048\n",
            "Validation score: 0.913846\n",
            "Iteration 12, loss = 0.42054532\n",
            "Validation score: 0.913846\n",
            "Iteration 13, loss = 0.38484627\n",
            "Validation score: 0.916923\n",
            "Iteration 14, loss = 0.35807246\n",
            "Validation score: 0.916923\n",
            "Iteration 15, loss = 0.33183697\n",
            "Validation score: 0.916923\n",
            "Iteration 16, loss = 0.31180902\n",
            "Validation score: 0.883077\n",
            "Iteration 17, loss = 0.29348275\n",
            "Validation score: 0.913846\n",
            "Iteration 18, loss = 0.27892825\n",
            "Validation score: 0.920000\n",
            "Iteration 19, loss = 0.26631280\n",
            "Validation score: 0.923077\n",
            "Iteration 20, loss = 0.25710641\n",
            "Validation score: 0.923077\n",
            "Iteration 21, loss = 0.25069080\n",
            "Validation score: 0.920000\n",
            "Iteration 22, loss = 0.23947615\n",
            "Validation score: 0.920000\n",
            "Iteration 23, loss = 0.23560695\n",
            "Validation score: 0.920000\n",
            "Iteration 24, loss = 0.22889456\n",
            "Validation score: 0.916923\n",
            "Iteration 25, loss = 0.22501483\n",
            "Validation score: 0.923077\n",
            "Iteration 26, loss = 0.22210034\n",
            "Validation score: 0.926154\n",
            "Iteration 27, loss = 0.21963905\n",
            "Validation score: 0.923077\n",
            "Iteration 28, loss = 0.21623383\n",
            "Validation score: 0.923077\n",
            "Iteration 29, loss = 0.21403617\n",
            "Validation score: 0.926154\n",
            "Iteration 30, loss = 0.21193478\n",
            "Validation score: 0.920000\n",
            "Iteration 31, loss = 0.20995134\n",
            "Validation score: 0.920000\n",
            "Iteration 32, loss = 0.20957610\n",
            "Validation score: 0.916923\n",
            "Iteration 33, loss = 0.20569454\n",
            "Validation score: 0.920000\n",
            "Iteration 34, loss = 0.20656317\n",
            "Validation score: 0.916923\n",
            "Iteration 35, loss = 0.20821945\n",
            "Validation score: 0.920000\n",
            "Iteration 36, loss = 0.20505819\n",
            "Validation score: 0.916923\n",
            "Iteration 37, loss = 0.20395293\n",
            "Validation score: 0.923077\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 08:50:47,030]\u001b[0m Trial 9 finished with value: 91.3969 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.009255107067747221, 'batch_size': 1, 'learning_rate_init': 0.0028930551066064496, 'max_iter': 271, 'momentum': 0.08880064618519395, 'number_layers': 1, 'hidden_neurons_0': 183}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.52547916\n",
            "Validation score: 0.836923\n",
            "Iteration 2, loss = 0.32611840\n",
            "Validation score: 0.883077\n",
            "Iteration 3, loss = 0.26741270\n",
            "Validation score: 0.898462\n",
            "Iteration 4, loss = 0.24094310\n",
            "Validation score: 0.916923\n",
            "Iteration 5, loss = 0.22102261\n",
            "Validation score: 0.920000\n",
            "Iteration 6, loss = 0.20861278\n",
            "Validation score: 0.916923\n",
            "Iteration 7, loss = 0.20077881\n",
            "Validation score: 0.920000\n",
            "Iteration 8, loss = 0.19578382\n",
            "Validation score: 0.916923\n",
            "Iteration 9, loss = 0.19285440\n",
            "Validation score: 0.916923\n",
            "Iteration 10, loss = 0.19108782\n",
            "Validation score: 0.916923\n",
            "Iteration 11, loss = 0.18626557\n",
            "Validation score: 0.916923\n",
            "Iteration 12, loss = 0.18274809\n",
            "Validation score: 0.907692\n",
            "Iteration 13, loss = 0.18451685\n",
            "Validation score: 0.907692\n",
            "Iteration 14, loss = 0.18500053\n",
            "Validation score: 0.923077\n",
            "Iteration 15, loss = 0.18026447\n",
            "Validation score: 0.920000\n",
            "Iteration 16, loss = 0.18014678\n",
            "Validation score: 0.913846\n",
            "Iteration 17, loss = 0.17923018\n",
            "Validation score: 0.913846\n",
            "Iteration 18, loss = 0.17934824\n",
            "Validation score: 0.913846\n",
            "Iteration 19, loss = 0.17859243\n",
            "Validation score: 0.913846\n",
            "Iteration 20, loss = 0.18032268\n",
            "Validation score: 0.910769\n",
            "Iteration 21, loss = 0.17833256\n",
            "Validation score: 0.913846\n",
            "Iteration 22, loss = 0.17581058\n",
            "Validation score: 0.910769\n",
            "Iteration 23, loss = 0.17634441\n",
            "Validation score: 0.913846\n",
            "Iteration 24, loss = 0.17754717\n",
            "Validation score: 0.916923\n",
            "Iteration 25, loss = 0.17339113\n",
            "Validation score: 0.913846\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 08:50:55,181]\u001b[0m Trial 10 finished with value: 91.8594 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.0007087488200437895, 'batch_size': 64, 'learning_rate_init': 0.0043297144483182325, 'max_iter': 309, 'momentum': 0.6580467220731413, 'number_layers': 5, 'hidden_neurons_0': 131, 'hidden_neurons_1': 126, 'hidden_neurons_2': 192, 'hidden_neurons_3': 188, 'hidden_neurons_4': 189}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.61342953\n",
            "Validation score: 0.818462\n",
            "Iteration 2, loss = 0.40865896\n",
            "Validation score: 0.907692\n",
            "Iteration 3, loss = 0.32969608\n",
            "Validation score: 0.920000\n",
            "Iteration 4, loss = 0.29413569\n",
            "Validation score: 0.935385\n",
            "Iteration 5, loss = 0.27006624\n",
            "Validation score: 0.941538\n",
            "Iteration 6, loss = 0.25640615\n",
            "Validation score: 0.941538\n",
            "Iteration 7, loss = 0.24605951\n",
            "Validation score: 0.950769\n",
            "Iteration 8, loss = 0.24078041\n",
            "Validation score: 0.953846\n",
            "Iteration 9, loss = 0.23549102\n",
            "Validation score: 0.916923\n",
            "Iteration 10, loss = 0.23364228\n",
            "Validation score: 0.907692\n",
            "Iteration 11, loss = 0.23343401\n",
            "Validation score: 0.938462\n",
            "Iteration 12, loss = 0.23052598\n",
            "Validation score: 0.920000\n",
            "Iteration 13, loss = 0.23030938\n",
            "Validation score: 0.923077\n",
            "Iteration 14, loss = 0.22833891\n",
            "Validation score: 0.938462\n",
            "Iteration 15, loss = 0.22706655\n",
            "Validation score: 0.944615\n",
            "Iteration 16, loss = 0.22468867\n",
            "Validation score: 0.938462\n",
            "Iteration 17, loss = 0.22682776\n",
            "Validation score: 0.941538\n",
            "Iteration 18, loss = 0.22347758\n",
            "Validation score: 0.944615\n",
            "Iteration 19, loss = 0.22245916\n",
            "Validation score: 0.916923\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 08:51:04,218]\u001b[0m Trial 11 finished with value: 91.7669 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.00362585630339854, 'batch_size': 32, 'learning_rate_init': 0.004737308147215586, 'max_iter': 164, 'momentum': 0.2779723477881824, 'number_layers': 5, 'hidden_neurons_0': 135, 'hidden_neurons_1': 194, 'hidden_neurons_2': 53, 'hidden_neurons_3': 142, 'hidden_neurons_4': 74}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.35240047\n",
            "Validation score: 0.892308\n",
            "Iteration 2, loss = 0.23365490\n",
            "Validation score: 0.913846\n",
            "Iteration 3, loss = 0.21548159\n",
            "Validation score: 0.898462\n",
            "Iteration 4, loss = 0.21241607\n",
            "Validation score: 0.920000\n",
            "Iteration 5, loss = 0.20941708\n",
            "Validation score: 0.898462\n",
            "Iteration 6, loss = 0.20205097\n",
            "Validation score: 0.907692\n",
            "Iteration 7, loss = 0.20623378\n",
            "Validation score: 0.920000\n",
            "Iteration 8, loss = 0.19927452\n",
            "Validation score: 0.907692\n",
            "Iteration 9, loss = 0.19740165\n",
            "Validation score: 0.907692\n",
            "Iteration 10, loss = 0.19773776\n",
            "Validation score: 0.913846\n",
            "Iteration 11, loss = 0.19423985\n",
            "Validation score: 0.907692\n",
            "Iteration 12, loss = 0.19559583\n",
            "Validation score: 0.910769\n",
            "Iteration 13, loss = 0.19219048\n",
            "Validation score: 0.920000\n",
            "Iteration 14, loss = 0.18966894\n",
            "Validation score: 0.910769\n",
            "Iteration 15, loss = 0.19052318\n",
            "Validation score: 0.910769\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 08:51:12,242]\u001b[0m Trial 12 finished with value: 91.1193 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.0023448619005761178, 'batch_size': 32, 'learning_rate_init': 0.003467637954474437, 'max_iter': 297, 'momentum': 0.8877689659648531, 'number_layers': 4, 'hidden_neurons_0': 150, 'hidden_neurons_1': 141, 'hidden_neurons_2': 112, 'hidden_neurons_3': 128}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.48394141\n",
            "Validation score: 0.901538\n",
            "Iteration 2, loss = 0.33627290\n",
            "Validation score: 0.901538\n",
            "Iteration 3, loss = 0.30585214\n",
            "Validation score: 0.923077\n",
            "Iteration 4, loss = 0.29958998\n",
            "Validation score: 0.916923\n",
            "Iteration 5, loss = 0.29664066\n",
            "Validation score: 0.923077\n",
            "Iteration 6, loss = 0.28712092\n",
            "Validation score: 0.840000\n",
            "Iteration 7, loss = 0.28737842\n",
            "Validation score: 0.913846\n",
            "Iteration 8, loss = 0.28369208\n",
            "Validation score: 0.913846\n",
            "Iteration 9, loss = 0.28213449\n",
            "Validation score: 0.910769\n",
            "Iteration 10, loss = 0.28087499\n",
            "Validation score: 0.913846\n",
            "Iteration 11, loss = 0.28158278\n",
            "Validation score: 0.898462\n",
            "Iteration 12, loss = 0.27771027\n",
            "Validation score: 0.913846\n",
            "Iteration 13, loss = 0.27469150\n",
            "Validation score: 0.876923\n",
            "Iteration 14, loss = 0.27397056\n",
            "Validation score: 0.913846\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 08:51:22,474]\u001b[0m Trial 13 finished with value: 91.2118 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.004806825389376383, 'batch_size': 16, 'learning_rate_init': 0.005617921788297891, 'max_iter': 233, 'momentum': 0.3249979489223206, 'number_layers': 5, 'hidden_neurons_0': 112, 'hidden_neurons_1': 157, 'hidden_neurons_2': 72, 'hidden_neurons_3': 169, 'hidden_neurons_4': 112}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.74728413\n",
            "Validation score: 0.233846\n",
            "Iteration 2, loss = 0.72010748\n",
            "Validation score: 0.396923\n",
            "Iteration 3, loss = 0.69434355\n",
            "Validation score: 0.646154\n",
            "Iteration 4, loss = 0.67025652\n",
            "Validation score: 0.806154\n",
            "Iteration 5, loss = 0.64847762\n",
            "Validation score: 0.867692\n",
            "Iteration 6, loss = 0.62911617\n",
            "Validation score: 0.873846\n",
            "Iteration 7, loss = 0.61185390\n",
            "Validation score: 0.852308\n",
            "Iteration 8, loss = 0.59627610\n",
            "Validation score: 0.833846\n",
            "Iteration 9, loss = 0.58203303\n",
            "Validation score: 0.815385\n",
            "Iteration 10, loss = 0.56883503\n",
            "Validation score: 0.809231\n",
            "Iteration 11, loss = 0.55648125\n",
            "Validation score: 0.809231\n",
            "Iteration 12, loss = 0.54476340\n",
            "Validation score: 0.812308\n",
            "Iteration 13, loss = 0.53352121\n",
            "Validation score: 0.821538\n",
            "Iteration 14, loss = 0.52266902\n",
            "Validation score: 0.830769\n",
            "Iteration 15, loss = 0.51213233\n",
            "Validation score: 0.849231\n",
            "Iteration 16, loss = 0.50188958\n",
            "Validation score: 0.852308\n",
            "Iteration 17, loss = 0.49194982\n",
            "Validation score: 0.858462\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 08:51:24,783]\u001b[0m Trial 14 finished with value: 83.7188 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.004884548136128474, 'batch_size': 128, 'learning_rate_init': 0.00025770389254133313, 'max_iter': 394, 'momentum': 0.6050989179413, 'number_layers': 3, 'hidden_neurons_0': 61, 'hidden_neurons_1': 95, 'hidden_neurons_2': 145}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.55317817\n",
            "Validation score: 0.769231\n",
            "Iteration 2, loss = 0.54799105\n",
            "Validation score: 0.769231\n",
            "Iteration 3, loss = 0.54825309\n",
            "Validation score: 0.769231\n",
            "Iteration 4, loss = 0.54863025\n",
            "Validation score: 0.769231\n",
            "Iteration 5, loss = 0.54857275\n",
            "Validation score: 0.769231\n",
            "Iteration 6, loss = 0.54715502\n",
            "Validation score: 0.769231\n",
            "Iteration 7, loss = 0.54859435\n",
            "Validation score: 0.769231\n",
            "Iteration 8, loss = 0.54835996\n",
            "Validation score: 0.769231\n",
            "Iteration 9, loss = 0.54843065\n",
            "Validation score: 0.769231\n",
            "Iteration 10, loss = 0.54803211\n",
            "Validation score: 0.769231\n",
            "Iteration 11, loss = 0.54825135\n",
            "Validation score: 0.769231\n",
            "Iteration 12, loss = 0.54850027\n",
            "Validation score: 0.769231\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 08:51:33,068]\u001b[0m Trial 15 finished with value: 76.7808 and parameters: {'actv': 'logistic', 'solvers': 'sgd', 'alpha': 0.0013661437752339237, 'batch_size': 32, 'learning_rate_init': 0.008015075825946839, 'max_iter': 311, 'momentum': 0.23293747708704138, 'number_layers': 4, 'hidden_neurons_0': 200, 'hidden_neurons_1': 198, 'hidden_neurons_2': 91, 'hidden_neurons_3': 101}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.33803923\n",
            "Validation score: 0.910769\n",
            "Iteration 2, loss = 0.22756380\n",
            "Validation score: 0.920000\n",
            "Iteration 3, loss = 0.20267476\n",
            "Validation score: 0.895385\n",
            "Iteration 4, loss = 0.19829423\n",
            "Validation score: 0.932308\n",
            "Iteration 5, loss = 0.18784432\n",
            "Validation score: 0.916923\n",
            "Iteration 6, loss = 0.18788688\n",
            "Validation score: 0.901538\n",
            "Iteration 7, loss = 0.18330410\n",
            "Validation score: 0.926154\n",
            "Iteration 8, loss = 0.18180059\n",
            "Validation score: 0.907692\n",
            "Iteration 9, loss = 0.17981835\n",
            "Validation score: 0.923077\n",
            "Iteration 10, loss = 0.17688700\n",
            "Validation score: 0.926154\n",
            "Iteration 11, loss = 0.17889839\n",
            "Validation score: 0.926154\n",
            "Iteration 12, loss = 0.17730764\n",
            "Validation score: 0.926154\n",
            "Iteration 13, loss = 0.17452470\n",
            "Validation score: 0.920000\n",
            "Iteration 14, loss = 0.17453967\n",
            "Validation score: 0.926154\n",
            "Iteration 15, loss = 0.17172162\n",
            "Validation score: 0.904615\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 08:51:39,036]\u001b[0m Trial 16 finished with value: 91.1193 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.0001535572117085729, 'batch_size': 32, 'learning_rate_init': 0.007705326374577373, 'max_iter': 147, 'momentum': 0.5860328524154411, 'number_layers': 3, 'hidden_neurons_0': 115, 'hidden_neurons_1': 148, 'hidden_neurons_2': 144}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.52305489\n",
            "Validation score: 0.858462\n",
            "Iteration 2, loss = 0.31537358\n",
            "Validation score: 0.870769\n",
            "Iteration 3, loss = 0.27099268\n",
            "Validation score: 0.858462\n",
            "Iteration 4, loss = 0.25417556\n",
            "Validation score: 0.901538\n",
            "Iteration 5, loss = 0.24604559\n",
            "Validation score: 0.898462\n",
            "Iteration 6, loss = 0.24229966\n",
            "Validation score: 0.907692\n",
            "Iteration 7, loss = 0.24015646\n",
            "Validation score: 0.892308\n",
            "Iteration 8, loss = 0.23762899\n",
            "Validation score: 0.904615\n",
            "Iteration 9, loss = 0.23494636\n",
            "Validation score: 0.910769\n",
            "Iteration 10, loss = 0.23284342\n",
            "Validation score: 0.876923\n",
            "Iteration 11, loss = 0.23271779\n",
            "Validation score: 0.904615\n",
            "Iteration 12, loss = 0.22984247\n",
            "Validation score: 0.920000\n",
            "Iteration 13, loss = 0.23181313\n",
            "Validation score: 0.904615\n",
            "Iteration 14, loss = 0.22938563\n",
            "Validation score: 0.920000\n",
            "Iteration 15, loss = 0.22485855\n",
            "Validation score: 0.923077\n",
            "Iteration 16, loss = 0.22347874\n",
            "Validation score: 0.920000\n",
            "Iteration 17, loss = 0.22182496\n",
            "Validation score: 0.916923\n",
            "Iteration 18, loss = 0.22393851\n",
            "Validation score: 0.898462\n",
            "Iteration 19, loss = 0.22599349\n",
            "Validation score: 0.892308\n",
            "Iteration 20, loss = 0.22444409\n",
            "Validation score: 0.904615\n",
            "Iteration 21, loss = 0.22005421\n",
            "Validation score: 0.913846\n",
            "Iteration 22, loss = 0.22199444\n",
            "Validation score: 0.916923\n",
            "Iteration 23, loss = 0.21877212\n",
            "Validation score: 0.913846\n",
            "Iteration 24, loss = 0.21827411\n",
            "Validation score: 0.910769\n",
            "Iteration 25, loss = 0.21634241\n",
            "Validation score: 0.916923\n",
            "Iteration 26, loss = 0.21620352\n",
            "Validation score: 0.926154\n",
            "Iteration 27, loss = 0.21462313\n",
            "Validation score: 0.923077\n",
            "Iteration 28, loss = 0.21601977\n",
            "Validation score: 0.916923\n",
            "Iteration 29, loss = 0.21306356\n",
            "Validation score: 0.913846\n",
            "Iteration 30, loss = 0.21739984\n",
            "Validation score: 0.889231\n",
            "Iteration 31, loss = 0.21701162\n",
            "Validation score: 0.926154\n",
            "Iteration 32, loss = 0.21171941\n",
            "Validation score: 0.876923\n",
            "Iteration 33, loss = 0.21195457\n",
            "Validation score: 0.920000\n",
            "Iteration 34, loss = 0.21049795\n",
            "Validation score: 0.920000\n",
            "Iteration 35, loss = 0.21070878\n",
            "Validation score: 0.892308\n",
            "Iteration 36, loss = 0.21264851\n",
            "Validation score: 0.923077\n",
            "Iteration 37, loss = 0.21124841\n",
            "Validation score: 0.886154\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 08:51:58,906]\u001b[0m Trial 17 finished with value: 91.6744 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.004049810284695638, 'batch_size': 32, 'learning_rate_init': 0.0054368380106460685, 'max_iter': 217, 'momentum': 0.35915133435418056, 'number_layers': 5, 'hidden_neurons_0': 147, 'hidden_neurons_1': 109, 'hidden_neurons_2': 140, 'hidden_neurons_3': 160, 'hidden_neurons_4': 151}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.36782047\n",
            "Validation score: 0.947692\n",
            "Iteration 2, loss = 0.24192496\n",
            "Validation score: 0.947692\n",
            "Iteration 3, loss = 0.22236598\n",
            "Validation score: 0.938462\n",
            "Iteration 4, loss = 0.21761982\n",
            "Validation score: 0.870769\n",
            "Iteration 5, loss = 0.22692976\n",
            "Validation score: 0.950769\n",
            "Iteration 6, loss = 0.21899118\n",
            "Validation score: 0.944615\n",
            "Iteration 7, loss = 0.21849635\n",
            "Validation score: 0.941538\n",
            "Iteration 8, loss = 0.20268451\n",
            "Validation score: 0.947692\n",
            "Iteration 9, loss = 0.20003014\n",
            "Validation score: 0.950769\n",
            "Iteration 10, loss = 0.20241733\n",
            "Validation score: 0.892308\n",
            "Iteration 11, loss = 0.19922389\n",
            "Validation score: 0.944615\n",
            "Iteration 12, loss = 0.20130072\n",
            "Validation score: 0.953846\n",
            "Iteration 13, loss = 0.21577351\n",
            "Validation score: 0.944615\n",
            "Iteration 14, loss = 0.21472637\n",
            "Validation score: 0.944615\n",
            "Iteration 15, loss = 0.20086309\n",
            "Validation score: 0.926154\n",
            "Iteration 16, loss = 0.19797321\n",
            "Validation score: 0.944615\n",
            "Iteration 17, loss = 0.20416852\n",
            "Validation score: 0.941538\n",
            "Iteration 18, loss = 0.18949214\n",
            "Validation score: 0.941538\n",
            "Iteration 19, loss = 0.17947439\n",
            "Validation score: 0.947692\n",
            "Iteration 20, loss = 0.18297822\n",
            "Validation score: 0.929231\n",
            "Iteration 21, loss = 0.18633078\n",
            "Validation score: 0.938462\n",
            "Iteration 22, loss = 0.19828554\n",
            "Validation score: 0.944615\n",
            "Iteration 23, loss = 0.17846937\n",
            "Validation score: 0.953846\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 08:52:05,662]\u001b[0m Trial 18 finished with value: 91.7669 and parameters: {'actv': 'tanh', 'solvers': 'adam', 'alpha': 0.00237735274580333, 'batch_size': 64, 'learning_rate_init': 0.0036039709810587258, 'max_iter': 271, 'momentum': 0.1892353473545429, 'number_layers': 4, 'hidden_neurons_0': 107, 'hidden_neurons_1': 58, 'hidden_neurons_2': 88, 'hidden_neurons_3': 104}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.90117874\n",
            "Validation score: 0.769231\n",
            "Iteration 2, loss = 0.87087072\n",
            "Validation score: 0.769231\n",
            "Iteration 3, loss = 0.84236727\n",
            "Validation score: 0.769231\n",
            "Iteration 4, loss = 0.81831091\n",
            "Validation score: 0.769231\n",
            "Iteration 5, loss = 0.79488634\n",
            "Validation score: 0.769231\n",
            "Iteration 6, loss = 0.77276424\n",
            "Validation score: 0.769231\n",
            "Iteration 7, loss = 0.75321616\n",
            "Validation score: 0.769231\n",
            "Iteration 8, loss = 0.73677726\n",
            "Validation score: 0.769231\n",
            "Iteration 9, loss = 0.71945562\n",
            "Validation score: 0.769231\n",
            "Iteration 10, loss = 0.70678181\n",
            "Validation score: 0.769231\n",
            "Iteration 11, loss = 0.69133914\n",
            "Validation score: 0.769231\n",
            "Iteration 12, loss = 0.68003122\n",
            "Validation score: 0.769231\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 08:53:10,637]\u001b[0m Trial 19 finished with value: 76.7808 and parameters: {'actv': 'logistic', 'solvers': 'sgd', 'alpha': 0.0063223674149656, 'batch_size': 1, 'learning_rate_init': 0.002342110246735192, 'max_iter': 135, 'momentum': 0.026735760596803104, 'number_layers': 3, 'hidden_neurons_0': 51, 'hidden_neurons_1': 178, 'hidden_neurons_2': 165}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.57846468\n",
            "Validation score: 0.769231\n",
            "Iteration 2, loss = 0.39491087\n",
            "Validation score: 0.861538\n",
            "Iteration 3, loss = 0.31984721\n",
            "Validation score: 0.876923\n",
            "Iteration 4, loss = 0.29188991\n",
            "Validation score: 0.880000\n",
            "Iteration 5, loss = 0.27383399\n",
            "Validation score: 0.886154\n",
            "Iteration 6, loss = 0.25779208\n",
            "Validation score: 0.892308\n",
            "Iteration 7, loss = 0.24531280\n",
            "Validation score: 0.904615\n",
            "Iteration 8, loss = 0.23417282\n",
            "Validation score: 0.904615\n",
            "Iteration 9, loss = 0.22428304\n",
            "Validation score: 0.904615\n",
            "Iteration 10, loss = 0.21668842\n",
            "Validation score: 0.910769\n",
            "Iteration 11, loss = 0.21110535\n",
            "Validation score: 0.904615\n",
            "Iteration 12, loss = 0.20908237\n",
            "Validation score: 0.910769\n",
            "Iteration 13, loss = 0.20777102\n",
            "Validation score: 0.901538\n",
            "Iteration 14, loss = 0.20621857\n",
            "Validation score: 0.907692\n",
            "Iteration 15, loss = 0.20367556\n",
            "Validation score: 0.907692\n",
            "Iteration 16, loss = 0.20258010\n",
            "Validation score: 0.910769\n",
            "Iteration 17, loss = 0.19841265\n",
            "Validation score: 0.904615\n",
            "Iteration 18, loss = 0.19929333\n",
            "Validation score: 0.916923\n",
            "Iteration 19, loss = 0.19861413\n",
            "Validation score: 0.910769\n",
            "Iteration 20, loss = 0.19779121\n",
            "Validation score: 0.907692\n",
            "Iteration 21, loss = 0.19633372\n",
            "Validation score: 0.910769\n",
            "Iteration 22, loss = 0.19747131\n",
            "Validation score: 0.907692\n",
            "Iteration 23, loss = 0.19545184\n",
            "Validation score: 0.916923\n",
            "Iteration 24, loss = 0.19292131\n",
            "Validation score: 0.907692\n",
            "Iteration 25, loss = 0.19252779\n",
            "Validation score: 0.920000\n",
            "Iteration 26, loss = 0.19275893\n",
            "Validation score: 0.910769\n",
            "Iteration 27, loss = 0.19448252\n",
            "Validation score: 0.910769\n",
            "Iteration 28, loss = 0.19154012\n",
            "Validation score: 0.907692\n",
            "Iteration 29, loss = 0.19305499\n",
            "Validation score: 0.907692\n",
            "Iteration 30, loss = 0.19249243\n",
            "Validation score: 0.920000\n",
            "Iteration 31, loss = 0.19002548\n",
            "Validation score: 0.904615\n",
            "Iteration 32, loss = 0.19373745\n",
            "Validation score: 0.910769\n",
            "Iteration 33, loss = 0.18857609\n",
            "Validation score: 0.916923\n",
            "Iteration 34, loss = 0.19052351\n",
            "Validation score: 0.904615\n",
            "Iteration 35, loss = 0.18740395\n",
            "Validation score: 0.907692\n",
            "Iteration 36, loss = 0.18830075\n",
            "Validation score: 0.913846\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 08:53:15,322]\u001b[0m Trial 20 finished with value: 91.8594 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.008806008111512055, 'batch_size': 128, 'learning_rate_init': 0.005942088399305558, 'max_iter': 57, 'momentum': 0.5659448067042001, 'number_layers': 5, 'hidden_neurons_0': 72, 'hidden_neurons_1': 137, 'hidden_neurons_2': 118, 'hidden_neurons_3': 64, 'hidden_neurons_4': 50}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.37490832\n",
            "Validation score: 0.889231\n",
            "Iteration 2, loss = 0.27878290\n",
            "Validation score: 0.895385\n",
            "Iteration 3, loss = 0.25619512\n",
            "Validation score: 0.904615\n",
            "Iteration 4, loss = 0.24142968\n",
            "Validation score: 0.916923\n",
            "Iteration 5, loss = 0.22968737\n",
            "Validation score: 0.920000\n",
            "Iteration 6, loss = 0.21971881\n",
            "Validation score: 0.923077\n",
            "Iteration 7, loss = 0.21682307\n",
            "Validation score: 0.929231\n",
            "Iteration 8, loss = 0.21117564\n",
            "Validation score: 0.923077\n",
            "Iteration 9, loss = 0.20943536\n",
            "Validation score: 0.938462\n",
            "Iteration 10, loss = 0.20447615\n",
            "Validation score: 0.935385\n",
            "Iteration 11, loss = 0.20357987\n",
            "Validation score: 0.938462\n",
            "Iteration 12, loss = 0.20229431\n",
            "Validation score: 0.938462\n",
            "Iteration 13, loss = 0.20049329\n",
            "Validation score: 0.935385\n",
            "Iteration 14, loss = 0.19948967\n",
            "Validation score: 0.941538\n",
            "Iteration 15, loss = 0.19681934\n",
            "Validation score: 0.935385\n",
            "Iteration 16, loss = 0.19732210\n",
            "Validation score: 0.938462\n",
            "Iteration 17, loss = 0.19610951\n",
            "Validation score: 0.929231\n",
            "Iteration 18, loss = 0.19627757\n",
            "Validation score: 0.935385\n",
            "Iteration 19, loss = 0.19649521\n",
            "Validation score: 0.929231\n",
            "Iteration 20, loss = 0.19402717\n",
            "Validation score: 0.923077\n",
            "Iteration 21, loss = 0.19412339\n",
            "Validation score: 0.941538\n",
            "Iteration 22, loss = 0.19473786\n",
            "Validation score: 0.938462\n",
            "Iteration 23, loss = 0.18886531\n",
            "Validation score: 0.941538\n",
            "Iteration 24, loss = 0.19104045\n",
            "Validation score: 0.944615\n",
            "Iteration 25, loss = 0.19012611\n",
            "Validation score: 0.926154\n",
            "Iteration 26, loss = 0.19134194\n",
            "Validation score: 0.920000\n",
            "Iteration 27, loss = 0.18985770\n",
            "Validation score: 0.938462\n",
            "Iteration 28, loss = 0.18720178\n",
            "Validation score: 0.941538\n",
            "Iteration 29, loss = 0.18681178\n",
            "Validation score: 0.929231\n",
            "Iteration 30, loss = 0.18849217\n",
            "Validation score: 0.938462\n",
            "Iteration 31, loss = 0.18886725\n",
            "Validation score: 0.932308\n",
            "Iteration 32, loss = 0.18718785\n",
            "Validation score: 0.938462\n",
            "Iteration 33, loss = 0.18580615\n",
            "Validation score: 0.941538\n",
            "Iteration 34, loss = 0.18885927\n",
            "Validation score: 0.941538\n",
            "Iteration 35, loss = 0.18641467\n",
            "Validation score: 0.932308\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 08:53:26,311]\u001b[0m Trial 21 finished with value: 91.9519 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.002848621732043671, 'batch_size': 64, 'learning_rate_init': 0.008447237656856075, 'max_iter': 64, 'momentum': 0.39840415404096574, 'number_layers': 2, 'hidden_neurons_0': 172, 'hidden_neurons_1': 176}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.34982607\n",
            "Validation score: 0.883077\n",
            "Iteration 2, loss = 0.26945393\n",
            "Validation score: 0.898462\n",
            "Iteration 3, loss = 0.24396454\n",
            "Validation score: 0.901538\n",
            "Iteration 4, loss = 0.22612150\n",
            "Validation score: 0.901538\n",
            "Iteration 5, loss = 0.21681679\n",
            "Validation score: 0.901538\n",
            "Iteration 6, loss = 0.21004926\n",
            "Validation score: 0.901538\n",
            "Iteration 7, loss = 0.20498174\n",
            "Validation score: 0.904615\n",
            "Iteration 8, loss = 0.20144890\n",
            "Validation score: 0.910769\n",
            "Iteration 9, loss = 0.19876484\n",
            "Validation score: 0.910769\n",
            "Iteration 10, loss = 0.19694167\n",
            "Validation score: 0.907692\n",
            "Iteration 11, loss = 0.19266442\n",
            "Validation score: 0.898462\n",
            "Iteration 12, loss = 0.19038198\n",
            "Validation score: 0.913846\n",
            "Iteration 13, loss = 0.19157865\n",
            "Validation score: 0.910769\n",
            "Iteration 14, loss = 0.18587768\n",
            "Validation score: 0.904615\n",
            "Iteration 15, loss = 0.18684010\n",
            "Validation score: 0.913846\n",
            "Iteration 16, loss = 0.18602500\n",
            "Validation score: 0.904615\n",
            "Iteration 17, loss = 0.18407356\n",
            "Validation score: 0.910769\n",
            "Iteration 18, loss = 0.18334335\n",
            "Validation score: 0.916923\n",
            "Iteration 19, loss = 0.18253010\n",
            "Validation score: 0.913846\n",
            "Iteration 20, loss = 0.18320415\n",
            "Validation score: 0.910769\n",
            "Iteration 21, loss = 0.18351609\n",
            "Validation score: 0.910769\n",
            "Iteration 22, loss = 0.17941158\n",
            "Validation score: 0.913846\n",
            "Iteration 23, loss = 0.18231457\n",
            "Validation score: 0.913846\n",
            "Iteration 24, loss = 0.17942102\n",
            "Validation score: 0.892308\n",
            "Iteration 25, loss = 0.17784335\n",
            "Validation score: 0.913846\n",
            "Iteration 26, loss = 0.17665485\n",
            "Validation score: 0.904615\n",
            "Iteration 27, loss = 0.17738801\n",
            "Validation score: 0.916923\n",
            "Iteration 28, loss = 0.18033509\n",
            "Validation score: 0.913846\n",
            "Iteration 29, loss = 0.17565170\n",
            "Validation score: 0.916923\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 08:53:35,119]\u001b[0m Trial 22 finished with value: 91.8594 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.0025965141004118085, 'batch_size': 64, 'learning_rate_init': 0.008829627561014011, 'max_iter': 210, 'momentum': 0.38123499853483156, 'number_layers': 2, 'hidden_neurons_0': 169, 'hidden_neurons_1': 164}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.38492530\n",
            "Validation score: 0.895385\n",
            "Iteration 2, loss = 0.27971881\n",
            "Validation score: 0.916923\n",
            "Iteration 3, loss = 0.25389464\n",
            "Validation score: 0.920000\n",
            "Iteration 4, loss = 0.23913869\n",
            "Validation score: 0.926154\n",
            "Iteration 5, loss = 0.22538926\n",
            "Validation score: 0.935385\n",
            "Iteration 6, loss = 0.21964647\n",
            "Validation score: 0.938462\n",
            "Iteration 7, loss = 0.21402945\n",
            "Validation score: 0.947692\n",
            "Iteration 8, loss = 0.20985146\n",
            "Validation score: 0.929231\n",
            "Iteration 9, loss = 0.20583634\n",
            "Validation score: 0.947692\n",
            "Iteration 10, loss = 0.20497314\n",
            "Validation score: 0.938462\n",
            "Iteration 11, loss = 0.20368171\n",
            "Validation score: 0.935385\n",
            "Iteration 12, loss = 0.20097077\n",
            "Validation score: 0.941538\n",
            "Iteration 13, loss = 0.19982086\n",
            "Validation score: 0.920000\n",
            "Iteration 14, loss = 0.19813214\n",
            "Validation score: 0.938462\n",
            "Iteration 15, loss = 0.19593453\n",
            "Validation score: 0.932308\n",
            "Iteration 16, loss = 0.19441778\n",
            "Validation score: 0.938462\n",
            "Iteration 17, loss = 0.19604982\n",
            "Validation score: 0.944615\n",
            "Iteration 18, loss = 0.19265176\n",
            "Validation score: 0.938462\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 08:53:40,445]\u001b[0m Trial 23 finished with value: 91.2118 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.00416960638576644, 'batch_size': 64, 'learning_rate_init': 0.007247301685963553, 'max_iter': 280, 'momentum': 0.5201929109676289, 'number_layers': 2, 'hidden_neurons_0': 156, 'hidden_neurons_1': 186}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.49479365\n",
            "Validation score: 0.873846\n",
            "Iteration 2, loss = 0.32408050\n",
            "Validation score: 0.880000\n",
            "Iteration 3, loss = 0.27680548\n",
            "Validation score: 0.886154\n",
            "Iteration 4, loss = 0.25424755\n",
            "Validation score: 0.892308\n",
            "Iteration 5, loss = 0.23739192\n",
            "Validation score: 0.901538\n",
            "Iteration 6, loss = 0.22480520\n",
            "Validation score: 0.907692\n",
            "Iteration 7, loss = 0.21560459\n",
            "Validation score: 0.916923\n",
            "Iteration 8, loss = 0.20913973\n",
            "Validation score: 0.913846\n",
            "Iteration 9, loss = 0.20380777\n",
            "Validation score: 0.916923\n",
            "Iteration 10, loss = 0.19945268\n",
            "Validation score: 0.920000\n",
            "Iteration 11, loss = 0.19720690\n",
            "Validation score: 0.913846\n",
            "Iteration 12, loss = 0.19432034\n",
            "Validation score: 0.910769\n",
            "Iteration 13, loss = 0.19115326\n",
            "Validation score: 0.916923\n",
            "Iteration 14, loss = 0.18954043\n",
            "Validation score: 0.916923\n",
            "Iteration 15, loss = 0.19075853\n",
            "Validation score: 0.913846\n",
            "Iteration 16, loss = 0.18704472\n",
            "Validation score: 0.913846\n",
            "Iteration 17, loss = 0.18522949\n",
            "Validation score: 0.916923\n",
            "Iteration 18, loss = 0.18434323\n",
            "Validation score: 0.907692\n",
            "Iteration 19, loss = 0.18597156\n",
            "Validation score: 0.907692\n",
            "Iteration 20, loss = 0.18473412\n",
            "Validation score: 0.907692\n",
            "Iteration 21, loss = 0.18558568\n",
            "Validation score: 0.916923\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 08:53:48,059]\u001b[0m Trial 24 finished with value: 91.3969 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.0014160670439839988, 'batch_size': 64, 'learning_rate_init': 0.004664197787530067, 'max_iter': 49, 'momentum': 0.3081798838893128, 'number_layers': 4, 'hidden_neurons_0': 180, 'hidden_neurons_1': 156, 'hidden_neurons_2': 92, 'hidden_neurons_3': 143}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.30407276\n",
            "Validation score: 0.913846\n",
            "Iteration 2, loss = 0.25705916\n",
            "Validation score: 0.916923\n",
            "Iteration 3, loss = 0.24456818\n",
            "Validation score: 0.916923\n",
            "Iteration 4, loss = 0.24482553\n",
            "Validation score: 0.916923\n",
            "Iteration 5, loss = 0.23747216\n",
            "Validation score: 0.932308\n",
            "Iteration 6, loss = 0.23963364\n",
            "Validation score: 0.932308\n",
            "Iteration 7, loss = 0.23669366\n",
            "Validation score: 0.929231\n",
            "Iteration 8, loss = 0.23066352\n",
            "Validation score: 0.910769\n",
            "Iteration 9, loss = 0.22989445\n",
            "Validation score: 0.938462\n",
            "Iteration 10, loss = 0.22760184\n",
            "Validation score: 0.923077\n",
            "Iteration 11, loss = 0.22569125\n",
            "Validation score: 0.932308\n",
            "Iteration 12, loss = 0.22711040\n",
            "Validation score: 0.935385\n",
            "Iteration 13, loss = 0.22215088\n",
            "Validation score: 0.929231\n",
            "Iteration 14, loss = 0.22354354\n",
            "Validation score: 0.926154\n",
            "Iteration 15, loss = 0.22081483\n",
            "Validation score: 0.920000\n",
            "Iteration 16, loss = 0.22031685\n",
            "Validation score: 0.920000\n",
            "Iteration 17, loss = 0.21771006\n",
            "Validation score: 0.938462\n",
            "Iteration 18, loss = 0.21716884\n",
            "Validation score: 0.941538\n",
            "Iteration 19, loss = 0.21305470\n",
            "Validation score: 0.926154\n",
            "Iteration 20, loss = 0.21623639\n",
            "Validation score: 0.938462\n",
            "Iteration 21, loss = 0.20948916\n",
            "Validation score: 0.938462\n",
            "Iteration 22, loss = 0.21369104\n",
            "Validation score: 0.913846\n",
            "Iteration 23, loss = 0.20986067\n",
            "Validation score: 0.941538\n",
            "Iteration 24, loss = 0.20723961\n",
            "Validation score: 0.941538\n",
            "Iteration 25, loss = 0.20691088\n",
            "Validation score: 0.926154\n",
            "Iteration 26, loss = 0.20841198\n",
            "Validation score: 0.926154\n",
            "Iteration 27, loss = 0.20889836\n",
            "Validation score: 0.916923\n",
            "Iteration 28, loss = 0.20960131\n",
            "Validation score: 0.938462\n",
            "Iteration 29, loss = 0.20326934\n",
            "Validation score: 0.926154\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 08:54:10,060]\u001b[0m Trial 25 finished with value: 92.0444 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.0031616953909307647, 'batch_size': 16, 'learning_rate_init': 0.008547296484008211, 'max_iter': 241, 'momentum': 0.6824399187961935, 'number_layers': 3, 'hidden_neurons_0': 144, 'hidden_neurons_1': 182, 'hidden_neurons_2': 52}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.36252176\n",
            "Validation score: 0.926154\n",
            "Iteration 2, loss = 0.29845739\n",
            "Validation score: 0.926154\n",
            "Iteration 3, loss = 0.28916351\n",
            "Validation score: 0.923077\n",
            "Iteration 4, loss = 0.28496336\n",
            "Validation score: 0.910769\n",
            "Iteration 5, loss = 0.27821924\n",
            "Validation score: 0.895385\n",
            "Iteration 6, loss = 0.27636462\n",
            "Validation score: 0.901538\n",
            "Iteration 7, loss = 0.26852743\n",
            "Validation score: 0.929231\n",
            "Iteration 8, loss = 0.26948392\n",
            "Validation score: 0.932308\n",
            "Iteration 9, loss = 0.26575283\n",
            "Validation score: 0.929231\n",
            "Iteration 10, loss = 0.26190598\n",
            "Validation score: 0.923077\n",
            "Iteration 11, loss = 0.26402497\n",
            "Validation score: 0.907692\n",
            "Iteration 12, loss = 0.26018609\n",
            "Validation score: 0.926154\n",
            "Iteration 13, loss = 0.26122566\n",
            "Validation score: 0.913846\n",
            "Iteration 14, loss = 0.25651714\n",
            "Validation score: 0.926154\n",
            "Iteration 15, loss = 0.25620920\n",
            "Validation score: 0.916923\n",
            "Iteration 16, loss = 0.25505771\n",
            "Validation score: 0.913846\n",
            "Iteration 17, loss = 0.25239171\n",
            "Validation score: 0.916923\n",
            "Iteration 18, loss = 0.25262533\n",
            "Validation score: 0.916923\n",
            "Iteration 19, loss = 0.24952161\n",
            "Validation score: 0.913846\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 08:54:24,233]\u001b[0m Trial 26 finished with value: 90.9343 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.005651652398674068, 'batch_size': 16, 'learning_rate_init': 0.006925951275960574, 'max_iter': 247, 'momentum': 0.6957969514530776, 'number_layers': 3, 'hidden_neurons_0': 141, 'hidden_neurons_1': 190, 'hidden_neurons_2': 64}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.42237728\n",
            "Validation score: 0.898462\n",
            "Iteration 2, loss = 0.43190623\n",
            "Validation score: 0.769231\n",
            "Iteration 3, loss = 0.45748311\n",
            "Validation score: 0.889231\n",
            "Iteration 4, loss = 0.39834126\n",
            "Validation score: 0.747692\n",
            "Iteration 5, loss = 0.44818185\n",
            "Validation score: 0.889231\n",
            "Iteration 6, loss = 0.47401367\n",
            "Validation score: 0.769231\n",
            "Iteration 7, loss = 0.44834003\n",
            "Validation score: 0.843077\n",
            "Iteration 8, loss = 0.43359360\n",
            "Validation score: 0.769231\n",
            "Iteration 9, loss = 0.44068582\n",
            "Validation score: 0.769231\n",
            "Iteration 10, loss = 0.47954078\n",
            "Validation score: 0.772308\n",
            "Iteration 11, loss = 0.59641011\n",
            "Validation score: 0.809231\n",
            "Iteration 12, loss = 0.59147105\n",
            "Validation score: 0.880000\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 08:54:35,461]\u001b[0m Trial 27 finished with value: 88.5291 and parameters: {'actv': 'tanh', 'solvers': 'adam', 'alpha': 0.001969504655080518, 'batch_size': 16, 'learning_rate_init': 0.009964774366507726, 'max_iter': 324, 'momentum': 0.8117899588321917, 'number_layers': 3, 'hidden_neurons_0': 125, 'hidden_neurons_1': 97, 'hidden_neurons_2': 73}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.56300652\n",
            "Validation score: 0.769231\n",
            "Iteration 2, loss = 0.56146265\n",
            "Validation score: 0.769231\n",
            "Iteration 3, loss = 0.56140455\n",
            "Validation score: 0.769231\n",
            "Iteration 4, loss = 0.56047014\n",
            "Validation score: 0.769231\n",
            "Iteration 5, loss = 0.56092607\n",
            "Validation score: 0.769231\n",
            "Iteration 6, loss = 0.55947540\n",
            "Validation score: 0.769231\n",
            "Iteration 7, loss = 0.56115882\n",
            "Validation score: 0.769231\n",
            "Iteration 8, loss = 0.56035641\n",
            "Validation score: 0.769231\n",
            "Iteration 9, loss = 0.56153941\n",
            "Validation score: 0.769231\n",
            "Iteration 10, loss = 0.56058459\n",
            "Validation score: 0.769231\n",
            "Iteration 11, loss = 0.56132519\n",
            "Validation score: 0.769231\n",
            "Iteration 12, loss = 0.56141476\n",
            "Validation score: 0.769231\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 08:54:42,821]\u001b[0m Trial 28 finished with value: 76.7808 and parameters: {'actv': 'logistic', 'solvers': 'sgd', 'alpha': 0.003375291345032761, 'batch_size': 16, 'learning_rate_init': 0.006228693906460166, 'max_iter': 253, 'momentum': 0.6426403184896066, 'number_layers': 4, 'hidden_neurons_0': 96, 'hidden_neurons_1': 131, 'hidden_neurons_2': 52, 'hidden_neurons_3': 197}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 1.28026105\n",
            "Validation score: 0.950769\n",
            "Iteration 2, loss = 0.80229660\n",
            "Validation score: 0.950769\n",
            "Iteration 3, loss = 0.56011302\n",
            "Validation score: 0.953846\n",
            "Iteration 4, loss = 0.41897031\n",
            "Validation score: 0.944615\n",
            "Iteration 5, loss = 0.34375904\n",
            "Validation score: 0.929231\n",
            "Iteration 6, loss = 0.30172255\n",
            "Validation score: 0.950769\n",
            "Iteration 7, loss = 0.27994138\n",
            "Validation score: 0.938462\n",
            "Iteration 8, loss = 0.27179028\n",
            "Validation score: 0.941538\n",
            "Iteration 9, loss = 0.26604668\n",
            "Validation score: 0.950769\n",
            "Iteration 10, loss = 0.25886229\n",
            "Validation score: 0.938462\n",
            "Iteration 11, loss = 0.26190240\n",
            "Validation score: 0.947692\n",
            "Iteration 12, loss = 0.25633579\n",
            "Validation score: 0.935385\n",
            "Iteration 13, loss = 0.25239931\n",
            "Validation score: 0.950769\n",
            "Iteration 14, loss = 0.25597253\n",
            "Validation score: 0.895385\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 08:57:29,878]\u001b[0m Trial 29 finished with value: 92.0444 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.003972659287939226, 'batch_size': 1, 'learning_rate_init': 0.007557083252576561, 'max_iter': 183, 'momentum': 0.7056528310685979, 'number_layers': 4, 'hidden_neurons_0': 124, 'hidden_neurons_1': 200, 'hidden_neurons_2': 130, 'hidden_neurons_3': 111}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.89822343\n",
            "Validation score: 0.769231\n",
            "Iteration 2, loss = 0.75234695\n",
            "Validation score: 0.769231\n",
            "Iteration 3, loss = 0.67486306\n",
            "Validation score: 0.769231\n",
            "Iteration 4, loss = 0.62410933\n",
            "Validation score: 0.769231\n",
            "Iteration 5, loss = 0.59449899\n",
            "Validation score: 0.769231\n",
            "Iteration 6, loss = 0.57749846\n",
            "Validation score: 0.769231\n",
            "Iteration 7, loss = 0.57098595\n",
            "Validation score: 0.769231\n",
            "Iteration 8, loss = 0.56395497\n",
            "Validation score: 0.769231\n",
            "Iteration 9, loss = 0.55902195\n",
            "Validation score: 0.769231\n",
            "Iteration 10, loss = 0.55999226\n",
            "Validation score: 0.769231\n",
            "Iteration 11, loss = 0.55756589\n",
            "Validation score: 0.769231\n",
            "Iteration 12, loss = 0.55591950\n",
            "Validation score: 0.769231\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 08:59:26,026]\u001b[0m Trial 30 finished with value: 76.7808 and parameters: {'actv': 'logistic', 'solvers': 'sgd', 'alpha': 0.004328711170154433, 'batch_size': 1, 'learning_rate_init': 0.005252687664086009, 'max_iter': 185, 'momentum': 0.74696260665051, 'number_layers': 4, 'hidden_neurons_0': 96, 'hidden_neurons_1': 200, 'hidden_neurons_2': 130, 'hidden_neurons_3': 107}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 1.30108526\n",
            "Validation score: 0.904615\n",
            "Iteration 2, loss = 0.84964842\n",
            "Validation score: 0.947692\n",
            "Iteration 3, loss = 0.60665085\n",
            "Validation score: 0.960000\n",
            "Iteration 4, loss = 0.46036856\n",
            "Validation score: 0.941538\n",
            "Iteration 5, loss = 0.38087081\n",
            "Validation score: 0.947692\n",
            "Iteration 6, loss = 0.32885622\n",
            "Validation score: 0.944615\n",
            "Iteration 7, loss = 0.30153305\n",
            "Validation score: 0.947692\n",
            "Iteration 8, loss = 0.28001513\n",
            "Validation score: 0.929231\n",
            "Iteration 9, loss = 0.26996502\n",
            "Validation score: 0.944615\n",
            "Iteration 10, loss = 0.26473787\n",
            "Validation score: 0.926154\n",
            "Iteration 11, loss = 0.26265965\n",
            "Validation score: 0.910769\n",
            "Iteration 12, loss = 0.25386725\n",
            "Validation score: 0.904615\n",
            "Iteration 13, loss = 0.26271408\n",
            "Validation score: 0.923077\n",
            "Iteration 14, loss = 0.25365335\n",
            "Validation score: 0.953846\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 09:02:27,937]\u001b[0m Trial 31 finished with value: 92.0444 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.003453668967552586, 'batch_size': 1, 'learning_rate_init': 0.007444772101602385, 'max_iter': 207, 'momentum': 0.7169653335131206, 'number_layers': 5, 'hidden_neurons_0': 122, 'hidden_neurons_1': 183, 'hidden_neurons_2': 168, 'hidden_neurons_3': 81, 'hidden_neurons_4': 115}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 1.18636934\n",
            "Validation score: 0.892308\n",
            "Iteration 2, loss = 0.58584812\n",
            "Validation score: 0.938462\n",
            "Iteration 3, loss = 0.38653392\n",
            "Validation score: 0.901538\n",
            "Iteration 4, loss = 0.35367927\n",
            "Validation score: 0.895385\n",
            "Iteration 5, loss = 0.32692574\n",
            "Validation score: 0.929231\n",
            "Iteration 6, loss = 0.33296357\n",
            "Validation score: 0.923077\n",
            "Iteration 7, loss = 0.32952411\n",
            "Validation score: 0.920000\n",
            "Iteration 8, loss = 0.34212164\n",
            "Validation score: 0.904615\n",
            "Iteration 9, loss = 0.31695550\n",
            "Validation score: 0.926154\n",
            "Iteration 10, loss = 0.31842523\n",
            "Validation score: 0.913846\n",
            "Iteration 11, loss = 0.34320299\n",
            "Validation score: 0.926154\n",
            "Iteration 12, loss = 0.38701920\n",
            "Validation score: 0.769231\n",
            "Iteration 13, loss = 0.58814696\n",
            "Validation score: 0.769231\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 09:06:36,822]\u001b[0m Trial 32 finished with value: 91.3043 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.003399044145295904, 'batch_size': 1, 'learning_rate_init': 0.006704827762625896, 'max_iter': 231, 'momentum': 0.8849395411069669, 'number_layers': 5, 'hidden_neurons_0': 157, 'hidden_neurons_1': 181, 'hidden_neurons_2': 165, 'hidden_neurons_3': 80, 'hidden_neurons_4': 116}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.36058097\n",
            "Validation score: 0.935385\n",
            "Iteration 2, loss = 0.25262106\n",
            "Validation score: 0.880000\n",
            "Iteration 3, loss = 0.24371776\n",
            "Validation score: 0.947692\n",
            "Iteration 4, loss = 0.23406306\n",
            "Validation score: 0.944615\n",
            "Iteration 5, loss = 0.23345004\n",
            "Validation score: 0.947692\n",
            "Iteration 6, loss = 0.22938421\n",
            "Validation score: 0.947692\n",
            "Iteration 7, loss = 0.22881155\n",
            "Validation score: 0.926154\n",
            "Iteration 8, loss = 0.22428242\n",
            "Validation score: 0.947692\n",
            "Iteration 9, loss = 0.22202874\n",
            "Validation score: 0.941538\n",
            "Iteration 10, loss = 0.22154670\n",
            "Validation score: 0.910769\n",
            "Iteration 11, loss = 0.22650631\n",
            "Validation score: 0.935385\n",
            "Iteration 12, loss = 0.21627002\n",
            "Validation score: 0.901538\n",
            "Iteration 13, loss = 0.21733090\n",
            "Validation score: 0.947692\n",
            "Iteration 14, loss = 0.21331338\n",
            "Validation score: 0.938462\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 09:06:48,066]\u001b[0m Trial 33 finished with value: 91.5819 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.0029949465363014625, 'batch_size': 32, 'learning_rate_init': 0.006313926645876833, 'max_iter': 214, 'momentum': 0.7682617194874686, 'number_layers': 5, 'hidden_neurons_0': 145, 'hidden_neurons_1': 155, 'hidden_neurons_2': 180, 'hidden_neurons_3': 80, 'hidden_neurons_4': 97}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.66954411\n",
            "Validation score: 0.769231\n",
            "Iteration 2, loss = 0.59623024\n",
            "Validation score: 0.769231\n",
            "Iteration 3, loss = 0.59825190\n",
            "Validation score: 0.769231\n",
            "Iteration 4, loss = 0.62111724\n",
            "Validation score: 0.769231\n",
            "Iteration 5, loss = 0.58472297\n",
            "Validation score: 0.769231\n",
            "Iteration 6, loss = 0.57397929\n",
            "Validation score: 0.769231\n",
            "Iteration 7, loss = 0.71574014\n",
            "Validation score: 0.769231\n",
            "Iteration 8, loss = 0.63381467\n",
            "Validation score: 0.769231\n",
            "Iteration 9, loss = 0.67824590\n",
            "Validation score: 0.769231\n",
            "Iteration 10, loss = 0.62742886\n",
            "Validation score: 0.769231\n",
            "Iteration 11, loss = 0.63314775\n",
            "Validation score: 0.769231\n",
            "Iteration 12, loss = 0.63159299\n",
            "Validation score: 0.769231\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 09:07:04,807]\u001b[0m Trial 34 finished with value: 76.7808 and parameters: {'actv': 'tanh', 'solvers': 'adam', 'alpha': 0.001827963504806413, 'batch_size': 16, 'learning_rate_init': 0.008951262024875, 'max_iter': 282, 'momentum': 0.5351043235306677, 'number_layers': 5, 'hidden_neurons_0': 74, 'hidden_neurons_1': 167, 'hidden_neurons_2': 150, 'hidden_neurons_3': 127, 'hidden_neurons_4': 151}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.33644569\n",
            "Validation score: 0.895385\n",
            "Iteration 2, loss = 0.25478979\n",
            "Validation score: 0.923077\n",
            "Iteration 3, loss = 0.24075123\n",
            "Validation score: 0.920000\n",
            "Iteration 4, loss = 0.23511342\n",
            "Validation score: 0.926154\n",
            "Iteration 5, loss = 0.23170047\n",
            "Validation score: 0.864615\n",
            "Iteration 6, loss = 0.23185314\n",
            "Validation score: 0.898462\n",
            "Iteration 7, loss = 0.22926699\n",
            "Validation score: 0.932308\n",
            "Iteration 8, loss = 0.22424656\n",
            "Validation score: 0.932308\n",
            "Iteration 9, loss = 0.22342144\n",
            "Validation score: 0.926154\n",
            "Iteration 10, loss = 0.21979149\n",
            "Validation score: 0.920000\n",
            "Iteration 11, loss = 0.22071177\n",
            "Validation score: 0.935385\n",
            "Iteration 12, loss = 0.21328347\n",
            "Validation score: 0.932308\n",
            "Iteration 13, loss = 0.21896929\n",
            "Validation score: 0.926154\n",
            "Iteration 14, loss = 0.21372924\n",
            "Validation score: 0.926154\n",
            "Iteration 15, loss = 0.21143551\n",
            "Validation score: 0.926154\n",
            "Iteration 16, loss = 0.20961921\n",
            "Validation score: 0.926154\n",
            "Iteration 17, loss = 0.21078800\n",
            "Validation score: 0.923077\n",
            "Iteration 18, loss = 0.20850393\n",
            "Validation score: 0.932308\n",
            "Iteration 19, loss = 0.20827230\n",
            "Validation score: 0.923077\n",
            "Iteration 20, loss = 0.20852312\n",
            "Validation score: 0.920000\n",
            "Iteration 21, loss = 0.20622428\n",
            "Validation score: 0.929231\n",
            "Iteration 22, loss = 0.20112628\n",
            "Validation score: 0.932308\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 09:07:20,122]\u001b[0m Trial 35 finished with value: 91.9519 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.004475608934827702, 'batch_size': 32, 'learning_rate_init': 0.0076706851176678555, 'max_iter': 161, 'momentum': 0.8256802701603255, 'number_layers': 4, 'hidden_neurons_0': 136, 'hidden_neurons_1': 190, 'hidden_neurons_2': 103, 'hidden_neurons_3': 116}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.64041414\n",
            "Validation score: 0.892308\n",
            "Iteration 2, loss = 0.36562694\n",
            "Validation score: 0.753846\n",
            "Iteration 3, loss = 0.52898996\n",
            "Validation score: 0.769231\n",
            "Iteration 4, loss = 0.54661008\n",
            "Validation score: 0.769231\n",
            "Iteration 5, loss = 0.54406139\n",
            "Validation score: 0.769231\n",
            "Iteration 6, loss = 0.54443350\n",
            "Validation score: 0.769231\n",
            "Iteration 7, loss = 0.54377552\n",
            "Validation score: 0.769231\n",
            "Iteration 8, loss = 0.54212836\n",
            "Validation score: 0.769231\n",
            "Iteration 9, loss = 0.54319350\n",
            "Validation score: 0.769231\n",
            "Iteration 10, loss = 0.54289973\n",
            "Validation score: 0.769231\n",
            "Iteration 11, loss = 0.54329443\n",
            "Validation score: 0.769231\n",
            "Iteration 12, loss = 0.54246093\n",
            "Validation score: 0.769231\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 09:12:52,561]\u001b[0m Trial 36 finished with value: 90.6568 and parameters: {'actv': 'relu', 'solvers': 'adam', 'alpha': 0.0037698235129903284, 'batch_size': 1, 'learning_rate_init': 0.007549574605094835, 'max_iter': 197, 'momentum': 0.7194235471522108, 'number_layers': 4, 'hidden_neurons_0': 120, 'hidden_neurons_1': 199, 'hidden_neurons_2': 128, 'hidden_neurons_3': 89}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.41470123\n",
            "Validation score: 0.876923\n",
            "Iteration 2, loss = 0.26671081\n",
            "Validation score: 0.883077\n",
            "Iteration 3, loss = 0.23835909\n",
            "Validation score: 0.907692\n",
            "Iteration 4, loss = 0.21709912\n",
            "Validation score: 0.923077\n",
            "Iteration 5, loss = 0.20739717\n",
            "Validation score: 0.913846\n",
            "Iteration 6, loss = 0.19923983\n",
            "Validation score: 0.923077\n",
            "Iteration 7, loss = 0.19425232\n",
            "Validation score: 0.935385\n",
            "Iteration 8, loss = 0.19054998\n",
            "Validation score: 0.941538\n",
            "Iteration 9, loss = 0.18809332\n",
            "Validation score: 0.932308\n",
            "Iteration 10, loss = 0.18481627\n",
            "Validation score: 0.920000\n",
            "Iteration 11, loss = 0.18582968\n",
            "Validation score: 0.929231\n",
            "Iteration 12, loss = 0.18160083\n",
            "Validation score: 0.926154\n",
            "Iteration 13, loss = 0.18213002\n",
            "Validation score: 0.932308\n",
            "Iteration 14, loss = 0.17985781\n",
            "Validation score: 0.938462\n",
            "Iteration 15, loss = 0.17903087\n",
            "Validation score: 0.932308\n",
            "Iteration 16, loss = 0.18037193\n",
            "Validation score: 0.938462\n",
            "Iteration 17, loss = 0.17692232\n",
            "Validation score: 0.938462\n",
            "Iteration 18, loss = 0.17722750\n",
            "Validation score: 0.935385\n",
            "Iteration 19, loss = 0.17771299\n",
            "Validation score: 0.923077\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 09:12:57,686]\u001b[0m Trial 37 finished with value: 91.3043 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.0007508886491724063, 'batch_size': 128, 'learning_rate_init': 0.009372254153135582, 'max_iter': 339, 'momentum': 0.6803969625980968, 'number_layers': 3, 'hidden_neurons_0': 103, 'hidden_neurons_1': 113, 'hidden_neurons_2': 79}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.34583319\n",
            "Validation score: 0.923077\n",
            "Iteration 2, loss = 0.29089687\n",
            "Validation score: 0.920000\n",
            "Iteration 3, loss = 0.27259287\n",
            "Validation score: 0.904615\n",
            "Iteration 4, loss = 0.27146643\n",
            "Validation score: 0.913846\n",
            "Iteration 5, loss = 0.26904825\n",
            "Validation score: 0.920000\n",
            "Iteration 6, loss = 0.26688007\n",
            "Validation score: 0.926154\n",
            "Iteration 7, loss = 0.26273272\n",
            "Validation score: 0.935385\n",
            "Iteration 8, loss = 0.26092475\n",
            "Validation score: 0.923077\n",
            "Iteration 9, loss = 0.26181627\n",
            "Validation score: 0.932308\n",
            "Iteration 10, loss = 0.25804660\n",
            "Validation score: 0.920000\n",
            "Iteration 11, loss = 0.25815080\n",
            "Validation score: 0.932308\n",
            "Iteration 12, loss = 0.25659697\n",
            "Validation score: 0.929231\n",
            "Iteration 13, loss = 0.25584880\n",
            "Validation score: 0.886154\n",
            "Iteration 14, loss = 0.25557045\n",
            "Validation score: 0.895385\n",
            "Iteration 15, loss = 0.25301343\n",
            "Validation score: 0.916923\n",
            "Iteration 16, loss = 0.25424452\n",
            "Validation score: 0.938462\n",
            "Iteration 17, loss = 0.25190850\n",
            "Validation score: 0.935385\n",
            "Iteration 18, loss = 0.25027458\n",
            "Validation score: 0.932308\n",
            "Iteration 19, loss = 0.24789339\n",
            "Validation score: 0.898462\n",
            "Iteration 20, loss = 0.25459537\n",
            "Validation score: 0.901538\n",
            "Iteration 21, loss = 0.24901077\n",
            "Validation score: 0.935385\n",
            "Iteration 22, loss = 0.24624000\n",
            "Validation score: 0.904615\n",
            "Iteration 23, loss = 0.24746784\n",
            "Validation score: 0.901538\n",
            "Iteration 24, loss = 0.24764740\n",
            "Validation score: 0.938462\n",
            "Iteration 25, loss = 0.24437872\n",
            "Validation score: 0.938462\n",
            "Iteration 26, loss = 0.24646360\n",
            "Validation score: 0.926154\n",
            "Iteration 27, loss = 0.24248730\n",
            "Validation score: 0.870769\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 09:13:22,919]\u001b[0m Trial 38 finished with value: 91.3043 and parameters: {'actv': 'tanh', 'solvers': 'sgd', 'alpha': 0.00802440789367342, 'batch_size': 32, 'learning_rate_init': 0.008470074067713449, 'max_iter': 125, 'momentum': 0.47450781886397275, 'number_layers': 3, 'hidden_neurons_0': 159, 'hidden_neurons_1': 171, 'hidden_neurons_2': 156}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.67462727\n",
            "Validation score: 0.769231\n",
            "Iteration 2, loss = 0.54522138\n",
            "Validation score: 0.769231\n",
            "Iteration 3, loss = 0.54312904\n",
            "Validation score: 0.769231\n",
            "Iteration 4, loss = 0.54246849\n",
            "Validation score: 0.769231\n",
            "Iteration 5, loss = 0.54324366\n",
            "Validation score: 0.769231\n",
            "Iteration 6, loss = 0.54300575\n",
            "Validation score: 0.769231\n",
            "Iteration 7, loss = 0.54338411\n",
            "Validation score: 0.769231\n",
            "Iteration 8, loss = 0.54322084\n",
            "Validation score: 0.769231\n",
            "Iteration 9, loss = 0.54313670\n",
            "Validation score: 0.769231\n",
            "Iteration 10, loss = 0.54258812\n",
            "Validation score: 0.769231\n",
            "Iteration 11, loss = 0.54330357\n",
            "Validation score: 0.769231\n",
            "Iteration 12, loss = 0.54314542\n",
            "Validation score: 0.769231\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 09:17:03,461]\u001b[0m Trial 39 finished with value: 76.7808 and parameters: {'actv': 'relu', 'solvers': 'adam', 'alpha': 0.00528736994862391, 'batch_size': 1, 'learning_rate_init': 0.006750507573884519, 'max_iter': 108, 'momentum': 0.6218865740898344, 'number_layers': 5, 'hidden_neurons_0': 87, 'hidden_neurons_1': 76, 'hidden_neurons_2': 182, 'hidden_neurons_3': 72, 'hidden_neurons_4': 141}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.98779156\n",
            "Validation score: 0.769231\n",
            "Iteration 2, loss = 0.69530230\n",
            "Validation score: 0.769231\n",
            "Iteration 3, loss = 0.60657247\n",
            "Validation score: 0.769231\n",
            "Iteration 4, loss = 0.57498693\n",
            "Validation score: 0.769231\n",
            "Iteration 5, loss = 0.56523422\n",
            "Validation score: 0.769231\n",
            "Iteration 6, loss = 0.56229985\n",
            "Validation score: 0.769231\n",
            "Iteration 7, loss = 0.56383063\n",
            "Validation score: 0.769231\n",
            "Iteration 8, loss = 0.56073558\n",
            "Validation score: 0.769231\n",
            "Iteration 9, loss = 0.56164869\n",
            "Validation score: 0.769231\n",
            "Iteration 10, loss = 0.55979662\n",
            "Validation score: 0.769231\n",
            "Iteration 11, loss = 0.56099498\n",
            "Validation score: 0.769231\n",
            "Iteration 12, loss = 0.55898625\n",
            "Validation score: 0.769231\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 09:19:39,885]\u001b[0m Trial 40 finished with value: 76.7808 and parameters: {'actv': 'logistic', 'solvers': 'sgd', 'alpha': 0.005853369286539067, 'batch_size': 1, 'learning_rate_init': 0.005740453328958501, 'max_iter': 169, 'momentum': 0.8294651923968446, 'number_layers': 5, 'hidden_neurons_0': 121, 'hidden_neurons_1': 164, 'hidden_neurons_2': 134, 'hidden_neurons_3': 95, 'hidden_neurons_4': 89}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 1.17451958\n",
            "Validation score: 0.923077\n",
            "Iteration 2, loss = 0.79540863\n",
            "Validation score: 0.907692\n",
            "Iteration 3, loss = 0.57978653\n",
            "Validation score: 0.883077\n",
            "Iteration 4, loss = 0.45098267\n",
            "Validation score: 0.916923\n",
            "Iteration 5, loss = 0.37379539\n",
            "Validation score: 0.926154\n",
            "Iteration 6, loss = 0.32441408\n",
            "Validation score: 0.883077\n",
            "Iteration 7, loss = 0.29426595\n",
            "Validation score: 0.910769\n",
            "Iteration 8, loss = 0.26889025\n",
            "Validation score: 0.910769\n",
            "Iteration 9, loss = 0.26436372\n",
            "Validation score: 0.916923\n",
            "Iteration 10, loss = 0.25562016\n",
            "Validation score: 0.916923\n",
            "Iteration 11, loss = 0.24753997\n",
            "Validation score: 0.913846\n",
            "Iteration 12, loss = 0.25277187\n",
            "Validation score: 0.916923\n",
            "Iteration 13, loss = 0.24642608\n",
            "Validation score: 0.913846\n",
            "Iteration 14, loss = 0.24431626\n",
            "Validation score: 0.880000\n",
            "Iteration 15, loss = 0.24341530\n",
            "Validation score: 0.913846\n",
            "Iteration 16, loss = 0.24624396\n",
            "Validation score: 0.916923\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 09:25:05,167]\u001b[0m Trial 41 finished with value: 91.5819 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.003118752318836021, 'batch_size': 1, 'learning_rate_init': 0.007365774218626038, 'max_iter': 200, 'momentum': 0.7237382578574678, 'number_layers': 4, 'hidden_neurons_0': 165, 'hidden_neurons_1': 182, 'hidden_neurons_2': 102, 'hidden_neurons_3': 115}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 1.36367922\n",
            "Validation score: 0.929231\n",
            "Iteration 2, loss = 0.95218140\n",
            "Validation score: 0.920000\n",
            "Iteration 3, loss = 0.69647636\n",
            "Validation score: 0.932308\n",
            "Iteration 4, loss = 0.53846635\n",
            "Validation score: 0.923077\n",
            "Iteration 5, loss = 0.43651239\n",
            "Validation score: 0.916923\n",
            "Iteration 6, loss = 0.36516587\n",
            "Validation score: 0.867692\n",
            "Iteration 7, loss = 0.31799953\n",
            "Validation score: 0.929231\n",
            "Iteration 8, loss = 0.29127752\n",
            "Validation score: 0.920000\n",
            "Iteration 9, loss = 0.26950832\n",
            "Validation score: 0.929231\n",
            "Iteration 10, loss = 0.25977135\n",
            "Validation score: 0.920000\n",
            "Iteration 11, loss = 0.25666364\n",
            "Validation score: 0.935385\n",
            "Iteration 12, loss = 0.24590710\n",
            "Validation score: 0.929231\n",
            "Iteration 13, loss = 0.24530166\n",
            "Validation score: 0.923077\n",
            "Iteration 14, loss = 0.24232395\n",
            "Validation score: 0.935385\n",
            "Iteration 15, loss = 0.24020455\n",
            "Validation score: 0.920000\n",
            "Iteration 16, loss = 0.23977027\n",
            "Validation score: 0.932308\n",
            "Iteration 17, loss = 0.23934686\n",
            "Validation score: 0.923077\n",
            "Iteration 18, loss = 0.23342399\n",
            "Validation score: 0.932308\n",
            "Iteration 19, loss = 0.24007774\n",
            "Validation score: 0.920000\n",
            "Iteration 20, loss = 0.24099426\n",
            "Validation score: 0.932308\n",
            "Iteration 21, loss = 0.23740116\n",
            "Validation score: 0.926154\n",
            "Iteration 22, loss = 0.23706997\n",
            "Validation score: 0.929231\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 09:30:40,450]\u001b[0m Trial 42 finished with value: 92.0444 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.003740900320879785, 'batch_size': 1, 'learning_rate_init': 0.0063653749212582615, 'max_iter': 251, 'momentum': 0.6777511280321065, 'number_layers': 4, 'hidden_neurons_0': 131, 'hidden_neurons_1': 189, 'hidden_neurons_2': 155, 'hidden_neurons_3': 138}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 1.29626371\n",
            "Validation score: 0.904615\n",
            "Iteration 2, loss = 0.79618108\n",
            "Validation score: 0.901538\n",
            "Iteration 3, loss = 0.54521948\n",
            "Validation score: 0.901538\n",
            "Iteration 4, loss = 0.40805811\n",
            "Validation score: 0.901538\n",
            "Iteration 5, loss = 0.33664305\n",
            "Validation score: 0.904615\n",
            "Iteration 6, loss = 0.29415041\n",
            "Validation score: 0.910769\n",
            "Iteration 7, loss = 0.27557814\n",
            "Validation score: 0.910769\n",
            "Iteration 8, loss = 0.26174806\n",
            "Validation score: 0.916923\n",
            "Iteration 9, loss = 0.25459964\n",
            "Validation score: 0.910769\n",
            "Iteration 10, loss = 0.25066417\n",
            "Validation score: 0.913846\n",
            "Iteration 11, loss = 0.24717981\n",
            "Validation score: 0.904615\n",
            "Iteration 12, loss = 0.24795104\n",
            "Validation score: 0.907692\n",
            "Iteration 13, loss = 0.24884085\n",
            "Validation score: 0.910769\n",
            "Iteration 14, loss = 0.24859881\n",
            "Validation score: 0.907692\n",
            "Iteration 15, loss = 0.24888914\n",
            "Validation score: 0.910769\n",
            "Iteration 16, loss = 0.24604874\n",
            "Validation score: 0.843077\n",
            "Iteration 17, loss = 0.24827058\n",
            "Validation score: 0.907692\n",
            "Iteration 18, loss = 0.24566394\n",
            "Validation score: 0.910769\n",
            "Iteration 19, loss = 0.24438633\n",
            "Validation score: 0.910769\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 09:35:20,283]\u001b[0m Trial 43 finished with value: 91.3043 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.0037620681180779453, 'batch_size': 1, 'learning_rate_init': 0.0061473532854655, 'max_iter': 263, 'momentum': 0.7799095797156492, 'number_layers': 4, 'hidden_neurons_0': 131, 'hidden_neurons_1': 191, 'hidden_neurons_2': 155, 'hidden_neurons_3': 144}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.35826200\n",
            "Validation score: 0.904615\n",
            "Iteration 2, loss = 0.28350230\n",
            "Validation score: 0.935385\n",
            "Iteration 3, loss = 0.27095363\n",
            "Validation score: 0.941538\n",
            "Iteration 4, loss = 0.26239797\n",
            "Validation score: 0.907692\n",
            "Iteration 5, loss = 0.25369375\n",
            "Validation score: 0.923077\n",
            "Iteration 6, loss = 0.25601608\n",
            "Validation score: 0.916923\n",
            "Iteration 7, loss = 0.25050984\n",
            "Validation score: 0.920000\n",
            "Iteration 8, loss = 0.24870278\n",
            "Validation score: 0.858462\n",
            "Iteration 9, loss = 0.24914787\n",
            "Validation score: 0.941538\n",
            "Iteration 10, loss = 0.24552282\n",
            "Validation score: 0.916923\n",
            "Iteration 11, loss = 0.24374707\n",
            "Validation score: 0.938462\n",
            "Iteration 12, loss = 0.24205915\n",
            "Validation score: 0.941538\n",
            "Iteration 13, loss = 0.24416903\n",
            "Validation score: 0.938462\n",
            "Iteration 14, loss = 0.23666200\n",
            "Validation score: 0.935385\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 09:35:44,830]\u001b[0m Trial 44 finished with value: 92.0444 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.002181428013363468, 'batch_size': 16, 'learning_rate_init': 0.007914507110541629, 'max_iter': 231, 'momentum': 0.5699723265012147, 'number_layers': 5, 'hidden_neurons_0': 178, 'hidden_neurons_1': 122, 'hidden_neurons_2': 198, 'hidden_neurons_3': 165, 'hidden_neurons_4': 131}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.58521512\n",
            "Validation score: 0.861538\n",
            "Iteration 2, loss = 0.33893782\n",
            "Validation score: 0.876923\n",
            "Iteration 3, loss = 0.28398740\n",
            "Validation score: 0.883077\n",
            "Iteration 4, loss = 0.25238548\n",
            "Validation score: 0.895385\n",
            "Iteration 5, loss = 0.23426695\n",
            "Validation score: 0.889231\n",
            "Iteration 6, loss = 0.22594852\n",
            "Validation score: 0.913846\n",
            "Iteration 7, loss = 0.21806810\n",
            "Validation score: 0.855385\n",
            "Iteration 8, loss = 0.21907029\n",
            "Validation score: 0.913846\n",
            "Iteration 9, loss = 0.21137994\n",
            "Validation score: 0.898462\n",
            "Iteration 10, loss = 0.21508713\n",
            "Validation score: 0.901538\n",
            "Iteration 11, loss = 0.21165002\n",
            "Validation score: 0.910769\n",
            "Iteration 12, loss = 0.20843564\n",
            "Validation score: 0.916923\n",
            "Iteration 13, loss = 0.20873941\n",
            "Validation score: 0.916923\n",
            "Iteration 14, loss = 0.20487899\n",
            "Validation score: 0.907692\n",
            "Iteration 15, loss = 0.20436243\n",
            "Validation score: 0.913846\n",
            "Iteration 16, loss = 0.20698564\n",
            "Validation score: 0.889231\n",
            "Iteration 17, loss = 0.20764610\n",
            "Validation score: 0.916923\n",
            "Iteration 18, loss = 0.20138852\n",
            "Validation score: 0.892308\n",
            "Iteration 19, loss = 0.20271393\n",
            "Validation score: 0.750769\n",
            "Iteration 20, loss = 0.20626069\n",
            "Validation score: 0.883077\n",
            "Iteration 21, loss = 0.20226199\n",
            "Validation score: 0.916923\n",
            "Iteration 22, loss = 0.20007781\n",
            "Validation score: 0.907692\n",
            "Iteration 23, loss = 0.19696143\n",
            "Validation score: 0.920000\n",
            "Iteration 24, loss = 0.19652760\n",
            "Validation score: 0.910769\n",
            "Iteration 25, loss = 0.19643611\n",
            "Validation score: 0.913846\n",
            "Iteration 26, loss = 0.19794393\n",
            "Validation score: 0.907692\n",
            "Iteration 27, loss = 0.19416551\n",
            "Validation score: 0.910769\n",
            "Iteration 28, loss = 0.19676577\n",
            "Validation score: 0.913846\n",
            "Iteration 29, loss = 0.19190258\n",
            "Validation score: 0.907692\n",
            "Iteration 30, loss = 0.19483723\n",
            "Validation score: 0.904615\n",
            "Iteration 31, loss = 0.19557230\n",
            "Validation score: 0.910769\n",
            "Iteration 32, loss = 0.19182920\n",
            "Validation score: 0.904615\n",
            "Iteration 33, loss = 0.19098764\n",
            "Validation score: 0.910769\n",
            "Iteration 34, loss = 0.19131411\n",
            "Validation score: 0.889231\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 09:36:18,564]\u001b[0m Trial 45 finished with value: 91.6744 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.0021858949817464284, 'batch_size': 32, 'learning_rate_init': 0.00475307815543196, 'max_iter': 292, 'momentum': 0.4395566799245517, 'number_layers': 5, 'hidden_neurons_0': 189, 'hidden_neurons_1': 118, 'hidden_neurons_2': 197, 'hidden_neurons_3': 167, 'hidden_neurons_4': 130}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 1.34210818\n",
            "Validation score: 0.920000\n",
            "Iteration 2, loss = 0.90494600\n",
            "Validation score: 0.910769\n",
            "Iteration 3, loss = 0.65050207\n",
            "Validation score: 0.880000\n",
            "Iteration 4, loss = 0.49233527\n",
            "Validation score: 0.895385\n",
            "Iteration 5, loss = 0.39422288\n",
            "Validation score: 0.923077\n",
            "Iteration 6, loss = 0.33153058\n",
            "Validation score: 0.858462\n",
            "Iteration 7, loss = 0.29394464\n",
            "Validation score: 0.910769\n",
            "Iteration 8, loss = 0.27253248\n",
            "Validation score: 0.861538\n",
            "Iteration 9, loss = 0.25878524\n",
            "Validation score: 0.901538\n",
            "Iteration 10, loss = 0.25019200\n",
            "Validation score: 0.889231\n",
            "Iteration 11, loss = 0.24479078\n",
            "Validation score: 0.876923\n",
            "Iteration 12, loss = 0.24409057\n",
            "Validation score: 0.916923\n",
            "Iteration 13, loss = 0.23682594\n",
            "Validation score: 0.913846\n",
            "Iteration 14, loss = 0.23896691\n",
            "Validation score: 0.892308\n",
            "Iteration 15, loss = 0.23599688\n",
            "Validation score: 0.913846\n",
            "Iteration 16, loss = 0.23739850\n",
            "Validation score: 0.907692\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 09:38:48,613]\u001b[0m Trial 46 finished with value: 91.8594 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.004447373344033808, 'batch_size': 1, 'learning_rate_init': 0.006582207520611087, 'max_iter': 177, 'momentum': 0.6425341225082918, 'number_layers': 4, 'hidden_neurons_0': 77, 'hidden_neurons_1': 192, 'hidden_neurons_2': 175, 'hidden_neurons_3': 154}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.29504284\n",
            "Validation score: 0.935385\n",
            "Iteration 2, loss = 0.23736580\n",
            "Validation score: 0.947692\n",
            "Iteration 3, loss = 0.22214532\n",
            "Validation score: 0.938462\n",
            "Iteration 4, loss = 0.21850041\n",
            "Validation score: 0.938462\n",
            "Iteration 5, loss = 0.21511391\n",
            "Validation score: 0.947692\n",
            "Iteration 6, loss = 0.21625397\n",
            "Validation score: 0.956923\n",
            "Iteration 7, loss = 0.20675006\n",
            "Validation score: 0.947692\n",
            "Iteration 8, loss = 0.20428139\n",
            "Validation score: 0.953846\n",
            "Iteration 9, loss = 0.20298732\n",
            "Validation score: 0.944615\n",
            "Iteration 10, loss = 0.20046859\n",
            "Validation score: 0.929231\n",
            "Iteration 11, loss = 0.19780287\n",
            "Validation score: 0.953846\n",
            "Iteration 12, loss = 0.20139956\n",
            "Validation score: 0.947692\n",
            "Iteration 13, loss = 0.19929190\n",
            "Validation score: 0.947692\n",
            "Iteration 14, loss = 0.19615493\n",
            "Validation score: 0.855385\n",
            "Iteration 15, loss = 0.19582054\n",
            "Validation score: 0.950769\n",
            "Iteration 16, loss = 0.19617838\n",
            "Validation score: 0.950769\n",
            "Iteration 17, loss = 0.19075121\n",
            "Validation score: 0.920000\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 09:39:11,348]\u001b[0m Trial 47 finished with value: 91.9519 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.0012564826307727458, 'batch_size': 16, 'learning_rate_init': 0.00947714124107249, 'max_iter': 231, 'momentum': 0.5664336785166472, 'number_layers': 3, 'hidden_neurons_0': 177, 'hidden_neurons_1': 145, 'hidden_neurons_2': 118}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.43812139\n",
            "Validation score: 0.886154\n",
            "Iteration 2, loss = 0.25163451\n",
            "Validation score: 0.910769\n",
            "Iteration 3, loss = 0.21890167\n",
            "Validation score: 0.904615\n",
            "Iteration 4, loss = 0.21042866\n",
            "Validation score: 0.901538\n",
            "Iteration 5, loss = 0.20613261\n",
            "Validation score: 0.907692\n",
            "Iteration 6, loss = 0.20276794\n",
            "Validation score: 0.910769\n",
            "Iteration 7, loss = 0.20039781\n",
            "Validation score: 0.895385\n",
            "Iteration 8, loss = 0.20417158\n",
            "Validation score: 0.892308\n",
            "Iteration 9, loss = 0.19648286\n",
            "Validation score: 0.923077\n",
            "Iteration 10, loss = 0.19573956\n",
            "Validation score: 0.913846\n",
            "Iteration 11, loss = 0.19185522\n",
            "Validation score: 0.913846\n",
            "Iteration 12, loss = 0.19212221\n",
            "Validation score: 0.907692\n",
            "Iteration 13, loss = 0.19130663\n",
            "Validation score: 0.916923\n",
            "Iteration 14, loss = 0.18797559\n",
            "Validation score: 0.910769\n",
            "Iteration 15, loss = 0.18610833\n",
            "Validation score: 0.901538\n",
            "Iteration 16, loss = 0.18812478\n",
            "Validation score: 0.913846\n",
            "Iteration 17, loss = 0.18912377\n",
            "Validation score: 0.913846\n",
            "Iteration 18, loss = 0.18464979\n",
            "Validation score: 0.926154\n",
            "Iteration 19, loss = 0.18403910\n",
            "Validation score: 0.913846\n",
            "Iteration 20, loss = 0.18000899\n",
            "Validation score: 0.920000\n",
            "Iteration 21, loss = 0.18080541\n",
            "Validation score: 0.907692\n",
            "Iteration 22, loss = 0.17823230\n",
            "Validation score: 0.907692\n",
            "Iteration 23, loss = 0.18033467\n",
            "Validation score: 0.913846\n",
            "Iteration 24, loss = 0.17653424\n",
            "Validation score: 0.907692\n",
            "Iteration 25, loss = 0.17343055\n",
            "Validation score: 0.923077\n",
            "Iteration 26, loss = 0.17653419\n",
            "Validation score: 0.904615\n",
            "Iteration 27, loss = 0.17139267\n",
            "Validation score: 0.920000\n",
            "Iteration 28, loss = 0.17293983\n",
            "Validation score: 0.913846\n",
            "Iteration 29, loss = 0.17157849\n",
            "Validation score: 0.907692\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 09:39:20,191]\u001b[0m Trial 48 finished with value: 91.2118 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.0026977449383986884, 'batch_size': 32, 'learning_rate_init': 0.003970652528951729, 'max_iter': 191, 'momentum': 0.8613082985566501, 'number_layers': 5, 'hidden_neurons_0': 66, 'hidden_neurons_1': 172, 'hidden_neurons_2': 60, 'hidden_neurons_3': 90, 'hidden_neurons_4': 169}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.43926522\n",
            "Validation score: 0.855385\n",
            "Iteration 2, loss = 0.33144231\n",
            "Validation score: 0.861538\n",
            "Iteration 3, loss = 0.35192319\n",
            "Validation score: 0.904615\n",
            "Iteration 4, loss = 0.32487131\n",
            "Validation score: 0.929231\n",
            "Iteration 5, loss = 0.28729503\n",
            "Validation score: 0.913846\n",
            "Iteration 6, loss = 0.33209449\n",
            "Validation score: 0.929231\n",
            "Iteration 7, loss = 0.32875355\n",
            "Validation score: 0.910769\n",
            "Iteration 8, loss = 0.47424878\n",
            "Validation score: 0.769231\n",
            "Iteration 9, loss = 0.55582936\n",
            "Validation score: 0.873846\n",
            "Iteration 10, loss = 0.38391762\n",
            "Validation score: 0.938462\n",
            "Iteration 11, loss = 0.33266690\n",
            "Validation score: 0.920000\n",
            "Iteration 12, loss = 0.31622133\n",
            "Validation score: 0.889231\n",
            "Iteration 13, loss = 0.32152573\n",
            "Validation score: 0.926154\n",
            "Iteration 14, loss = 0.30376870\n",
            "Validation score: 0.938462\n",
            "Iteration 15, loss = 0.48101421\n",
            "Validation score: 0.910769\n",
            "Iteration 16, loss = 0.56443149\n",
            "Validation score: 0.886154\n",
            "Iteration 17, loss = 0.42890176\n",
            "Validation score: 0.941538\n",
            "Iteration 18, loss = 0.39405934\n",
            "Validation score: 0.769231\n",
            "Iteration 19, loss = 0.37763533\n",
            "Validation score: 0.938462\n",
            "Iteration 20, loss = 0.34761887\n",
            "Validation score: 0.929231\n",
            "Iteration 21, loss = 0.35752002\n",
            "Validation score: 0.920000\n",
            "Iteration 22, loss = 0.34728360\n",
            "Validation score: 0.926154\n",
            "Iteration 23, loss = 0.34886570\n",
            "Validation score: 0.901538\n",
            "Iteration 24, loss = 0.30529252\n",
            "Validation score: 0.920000\n",
            "Iteration 25, loss = 0.33247897\n",
            "Validation score: 0.916923\n",
            "Iteration 26, loss = 0.32321176\n",
            "Validation score: 0.938462\n",
            "Iteration 27, loss = 0.32989048\n",
            "Validation score: 0.929231\n",
            "Iteration 28, loss = 0.31573734\n",
            "Validation score: 0.926154\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 09:39:38,479]\u001b[0m Trial 49 finished with value: 91.4894 and parameters: {'actv': 'tanh', 'solvers': 'adam', 'alpha': 0.004852719712461645, 'batch_size': 32, 'learning_rate_init': 0.0051944802572711345, 'max_iter': 246, 'momentum': 0.6656421392759356, 'number_layers': 4, 'hidden_neurons_0': 138, 'hidden_neurons_1': 174, 'hidden_neurons_2': 138, 'hidden_neurons_3': 133}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.27350934\n",
            "Validation score: 0.898462\n",
            "Iteration 2, loss = 0.21087787\n",
            "Validation score: 0.923077\n",
            "Iteration 3, loss = 0.20067465\n",
            "Validation score: 0.926154\n",
            "Iteration 4, loss = 0.19677983\n",
            "Validation score: 0.892308\n",
            "Iteration 5, loss = 0.19454215\n",
            "Validation score: 0.941538\n",
            "Iteration 6, loss = 0.18947245\n",
            "Validation score: 0.932308\n",
            "Iteration 7, loss = 0.18443823\n",
            "Validation score: 0.876923\n",
            "Iteration 8, loss = 0.18573437\n",
            "Validation score: 0.913846\n",
            "Iteration 9, loss = 0.18351755\n",
            "Validation score: 0.929231\n",
            "Iteration 10, loss = 0.18385712\n",
            "Validation score: 0.907692\n",
            "Iteration 11, loss = 0.18169479\n",
            "Validation score: 0.913846\n",
            "Iteration 12, loss = 0.17651519\n",
            "Validation score: 0.938462\n",
            "Iteration 13, loss = 0.17660315\n",
            "Validation score: 0.920000\n",
            "Iteration 14, loss = 0.17609241\n",
            "Validation score: 0.910769\n",
            "Iteration 15, loss = 0.17397793\n",
            "Validation score: 0.929231\n",
            "Iteration 16, loss = 0.17198670\n",
            "Validation score: 0.907692\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 09:40:00,651]\u001b[0m Trial 50 finished with value: 91.3969 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.00026143914808648863, 'batch_size': 16, 'learning_rate_init': 0.008756615579418436, 'max_iter': 231, 'momentum': 0.5169917687971719, 'number_layers': 3, 'hidden_neurons_0': 195, 'hidden_neurons_1': 102, 'hidden_neurons_2': 121}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.33095600\n",
            "Validation score: 0.858462\n",
            "Iteration 2, loss = 0.24373067\n",
            "Validation score: 0.910769\n",
            "Iteration 3, loss = 0.22822919\n",
            "Validation score: 0.913846\n",
            "Iteration 4, loss = 0.22235392\n",
            "Validation score: 0.873846\n",
            "Iteration 5, loss = 0.22001101\n",
            "Validation score: 0.932308\n",
            "Iteration 6, loss = 0.21677933\n",
            "Validation score: 0.913846\n",
            "Iteration 7, loss = 0.21284293\n",
            "Validation score: 0.883077\n",
            "Iteration 8, loss = 0.20907758\n",
            "Validation score: 0.904615\n",
            "Iteration 9, loss = 0.20845930\n",
            "Validation score: 0.913846\n",
            "Iteration 10, loss = 0.20477820\n",
            "Validation score: 0.880000\n",
            "Iteration 11, loss = 0.20333006\n",
            "Validation score: 0.929231\n",
            "Iteration 12, loss = 0.20391316\n",
            "Validation score: 0.913846\n",
            "Iteration 13, loss = 0.20461546\n",
            "Validation score: 0.923077\n",
            "Iteration 14, loss = 0.19841473\n",
            "Validation score: 0.926154\n",
            "Iteration 15, loss = 0.20061372\n",
            "Validation score: 0.923077\n",
            "Iteration 16, loss = 0.19773889\n",
            "Validation score: 0.923077\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 09:40:18,472]\u001b[0m Trial 51 finished with value: 92.0444 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.0014278961742406815, 'batch_size': 16, 'learning_rate_init': 0.008049705355368178, 'max_iter': 261, 'momentum': 0.5783580237136451, 'number_layers': 4, 'hidden_neurons_0': 151, 'hidden_neurons_1': 121, 'hidden_neurons_2': 185, 'hidden_neurons_3': 179}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.36392412\n",
            "Validation score: 0.886154\n",
            "Iteration 2, loss = 0.28085023\n",
            "Validation score: 0.901538\n",
            "Iteration 3, loss = 0.26706826\n",
            "Validation score: 0.904615\n",
            "Iteration 4, loss = 0.26166608\n",
            "Validation score: 0.876923\n",
            "Iteration 5, loss = 0.26133488\n",
            "Validation score: 0.895385\n",
            "Iteration 6, loss = 0.25852210\n",
            "Validation score: 0.901538\n",
            "Iteration 7, loss = 0.24993207\n",
            "Validation score: 0.864615\n",
            "Iteration 8, loss = 0.24996122\n",
            "Validation score: 0.895385\n",
            "Iteration 9, loss = 0.24611569\n",
            "Validation score: 0.907692\n",
            "Iteration 10, loss = 0.24665840\n",
            "Validation score: 0.895385\n",
            "Iteration 11, loss = 0.24202534\n",
            "Validation score: 0.910769\n",
            "Iteration 12, loss = 0.24365631\n",
            "Validation score: 0.907692\n",
            "Iteration 13, loss = 0.23951036\n",
            "Validation score: 0.913846\n",
            "Iteration 14, loss = 0.23686334\n",
            "Validation score: 0.898462\n",
            "Iteration 15, loss = 0.23943962\n",
            "Validation score: 0.898462\n",
            "Iteration 16, loss = 0.23910280\n",
            "Validation score: 0.913846\n",
            "Iteration 17, loss = 0.23441494\n",
            "Validation score: 0.904615\n",
            "Iteration 18, loss = 0.23367376\n",
            "Validation score: 0.907692\n",
            "Iteration 19, loss = 0.22952229\n",
            "Validation score: 0.895385\n",
            "Iteration 20, loss = 0.22809236\n",
            "Validation score: 0.898462\n",
            "Iteration 21, loss = 0.22810475\n",
            "Validation score: 0.907692\n",
            "Iteration 22, loss = 0.22788403\n",
            "Validation score: 0.904615\n",
            "Iteration 23, loss = 0.22862982\n",
            "Validation score: 0.907692\n",
            "Iteration 24, loss = 0.22361273\n",
            "Validation score: 0.889231\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 09:40:49,684]\u001b[0m Trial 52 finished with value: 91.8594 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.003117925951188674, 'batch_size': 16, 'learning_rate_init': 0.0071510553439186825, 'max_iter': 242, 'momentum': 0.6050058252292306, 'number_layers': 5, 'hidden_neurons_0': 174, 'hidden_neurons_1': 129, 'hidden_neurons_2': 168, 'hidden_neurons_3': 54, 'hidden_neurons_4': 111}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.50858405\n",
            "Validation score: 0.889231\n",
            "Iteration 2, loss = 0.33044649\n",
            "Validation score: 0.904615\n",
            "Iteration 3, loss = 0.28454791\n",
            "Validation score: 0.910769\n",
            "Iteration 4, loss = 0.25600105\n",
            "Validation score: 0.920000\n",
            "Iteration 5, loss = 0.23512789\n",
            "Validation score: 0.929231\n",
            "Iteration 6, loss = 0.22350682\n",
            "Validation score: 0.926154\n",
            "Iteration 7, loss = 0.21437283\n",
            "Validation score: 0.935385\n",
            "Iteration 8, loss = 0.20923346\n",
            "Validation score: 0.932308\n",
            "Iteration 9, loss = 0.20279993\n",
            "Validation score: 0.932308\n",
            "Iteration 10, loss = 0.19631356\n",
            "Validation score: 0.932308\n",
            "Iteration 11, loss = 0.19887684\n",
            "Validation score: 0.923077\n",
            "Iteration 12, loss = 0.19478565\n",
            "Validation score: 0.923077\n",
            "Iteration 13, loss = 0.19623994\n",
            "Validation score: 0.929231\n",
            "Iteration 14, loss = 0.19269497\n",
            "Validation score: 0.935385\n",
            "Iteration 15, loss = 0.19147670\n",
            "Validation score: 0.935385\n",
            "Iteration 16, loss = 0.19574202\n",
            "Validation score: 0.920000\n",
            "Iteration 17, loss = 0.18969280\n",
            "Validation score: 0.923077\n",
            "Iteration 18, loss = 0.18628157\n",
            "Validation score: 0.935385\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 09:40:54,849]\u001b[0m Trial 53 finished with value: 91.1193 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.0015393895111950248, 'batch_size': 128, 'learning_rate_init': 0.007976676286402997, 'max_iter': 263, 'momentum': 0.5534567278053769, 'number_layers': 5, 'hidden_neurons_0': 163, 'hidden_neurons_1': 119, 'hidden_neurons_2': 190, 'hidden_neurons_3': 184, 'hidden_neurons_4': 131}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.42150551\n",
            "Validation score: 0.895385\n",
            "Iteration 2, loss = 0.29090921\n",
            "Validation score: 0.901538\n",
            "Iteration 3, loss = 0.26101017\n",
            "Validation score: 0.920000\n",
            "Iteration 4, loss = 0.25268689\n",
            "Validation score: 0.913846\n",
            "Iteration 5, loss = 0.24753848\n",
            "Validation score: 0.916923\n",
            "Iteration 6, loss = 0.24700294\n",
            "Validation score: 0.920000\n",
            "Iteration 7, loss = 0.23989368\n",
            "Validation score: 0.913846\n",
            "Iteration 8, loss = 0.24049747\n",
            "Validation score: 0.920000\n",
            "Iteration 9, loss = 0.23641718\n",
            "Validation score: 0.916923\n",
            "Iteration 10, loss = 0.23752181\n",
            "Validation score: 0.923077\n",
            "Iteration 11, loss = 0.23617067\n",
            "Validation score: 0.932308\n",
            "Iteration 12, loss = 0.23137174\n",
            "Validation score: 0.910769\n",
            "Iteration 13, loss = 0.23373296\n",
            "Validation score: 0.935385\n",
            "Iteration 14, loss = 0.23306836\n",
            "Validation score: 0.910769\n",
            "Iteration 15, loss = 0.22997983\n",
            "Validation score: 0.910769\n",
            "Iteration 16, loss = 0.22854970\n",
            "Validation score: 0.932308\n",
            "Iteration 17, loss = 0.23069692\n",
            "Validation score: 0.923077\n",
            "Iteration 18, loss = 0.22798547\n",
            "Validation score: 0.929231\n",
            "Iteration 19, loss = 0.22812969\n",
            "Validation score: 0.880000\n",
            "Iteration 20, loss = 0.22337394\n",
            "Validation score: 0.932308\n",
            "Iteration 21, loss = 0.22174048\n",
            "Validation score: 0.926154\n",
            "Iteration 22, loss = 0.22327596\n",
            "Validation score: 0.926154\n",
            "Iteration 23, loss = 0.22221296\n",
            "Validation score: 0.932308\n",
            "Iteration 24, loss = 0.22068678\n",
            "Validation score: 0.935385\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 09:41:25,149]\u001b[0m Trial 54 finished with value: 91.4894 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.002403048057153097, 'batch_size': 16, 'learning_rate_init': 0.005805043421891211, 'max_iter': 298, 'momentum': 0.25096131045416487, 'number_layers': 4, 'hidden_neurons_0': 184, 'hidden_neurons_1': 134, 'hidden_neurons_2': 111, 'hidden_neurons_3': 154}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 1.67670708\n",
            "Validation score: 0.916923\n",
            "Iteration 2, loss = 1.53702091\n",
            "Validation score: 0.935385\n",
            "Iteration 3, loss = 1.44997400\n",
            "Validation score: 0.941538\n",
            "Iteration 4, loss = 1.38062777\n",
            "Validation score: 0.926154\n",
            "Iteration 5, loss = 1.31368799\n",
            "Validation score: 0.923077\n",
            "Iteration 6, loss = 1.25073780\n",
            "Validation score: 0.941538\n",
            "Iteration 7, loss = 1.18857002\n",
            "Validation score: 0.932308\n",
            "Iteration 8, loss = 1.13566591\n",
            "Validation score: 0.935385\n",
            "Iteration 9, loss = 1.07818620\n",
            "Validation score: 0.938462\n",
            "Iteration 10, loss = 1.03281412\n",
            "Validation score: 0.926154\n",
            "Iteration 11, loss = 0.98503002\n",
            "Validation score: 0.929231\n",
            "Iteration 12, loss = 0.94018243\n",
            "Validation score: 0.932308\n",
            "Iteration 13, loss = 0.90150302\n",
            "Validation score: 0.929231\n",
            "Iteration 14, loss = 0.86037984\n",
            "Validation score: 0.935385\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 09:44:08,083]\u001b[0m Trial 55 finished with value: 91.7669 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.0038306351113125163, 'batch_size': 1, 'learning_rate_init': 0.0007278622207787045, 'max_iter': 212, 'momentum': 0.7146016864707618, 'number_layers': 5, 'hidden_neurons_0': 108, 'hidden_neurons_1': 185, 'hidden_neurons_2': 158, 'hidden_neurons_3': 116, 'hidden_neurons_4': 91}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.63160148\n",
            "Validation score: 0.910769\n",
            "Iteration 2, loss = 0.51344105\n",
            "Validation score: 0.916923\n",
            "Iteration 3, loss = 0.45955010\n",
            "Validation score: 0.901538\n",
            "Iteration 4, loss = 0.41654245\n",
            "Validation score: 0.910769\n",
            "Iteration 5, loss = 0.38014938\n",
            "Validation score: 0.904615\n",
            "Iteration 6, loss = 0.34916693\n",
            "Validation score: 0.913846\n",
            "Iteration 7, loss = 0.32126439\n",
            "Validation score: 0.923077\n",
            "Iteration 8, loss = 0.30843732\n",
            "Validation score: 0.904615\n",
            "Iteration 9, loss = 0.28854204\n",
            "Validation score: 0.901538\n",
            "Iteration 10, loss = 0.27930847\n",
            "Validation score: 0.932308\n",
            "Iteration 11, loss = 0.26694720\n",
            "Validation score: 0.892308\n",
            "Iteration 12, loss = 0.25922008\n",
            "Validation score: 0.913846\n",
            "Iteration 13, loss = 0.25446923\n",
            "Validation score: 0.916923\n",
            "Iteration 14, loss = 0.24204202\n",
            "Validation score: 0.913846\n",
            "Iteration 15, loss = 0.23837402\n",
            "Validation score: 0.932308\n",
            "Iteration 16, loss = 0.23196330\n",
            "Validation score: 0.932308\n",
            "Iteration 17, loss = 0.23793418\n",
            "Validation score: 0.932308\n",
            "Iteration 18, loss = 0.23792704\n",
            "Validation score: 0.929231\n",
            "Iteration 19, loss = 0.22597824\n",
            "Validation score: 0.895385\n",
            "Iteration 20, loss = 0.23185161\n",
            "Validation score: 0.920000\n",
            "Iteration 21, loss = 0.22634234\n",
            "Validation score: 0.926154\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 09:50:01,885]\u001b[0m Trial 56 finished with value: 91.6744 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.0011732769255888426, 'batch_size': 1, 'learning_rate_init': 0.006966617150702263, 'max_iter': 261, 'momentum': 0.7640024593500212, 'number_layers': 4, 'hidden_neurons_0': 150, 'hidden_neurons_1': 81, 'hidden_neurons_2': 188, 'hidden_neurons_3': 133}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 1.36997421\n",
            "Validation score: 0.861538\n",
            "Iteration 2, loss = 1.13495854\n",
            "Validation score: 0.910769\n",
            "Iteration 3, loss = 0.97560297\n",
            "Validation score: 0.923077\n",
            "Iteration 4, loss = 0.84295769\n",
            "Validation score: 0.923077\n",
            "Iteration 5, loss = 0.73601949\n",
            "Validation score: 0.898462\n",
            "Iteration 6, loss = 0.64725248\n",
            "Validation score: 0.920000\n",
            "Iteration 7, loss = 0.57388924\n",
            "Validation score: 0.901538\n",
            "Iteration 8, loss = 0.51192565\n",
            "Validation score: 0.883077\n",
            "Iteration 9, loss = 0.46363436\n",
            "Validation score: 0.892308\n",
            "Iteration 10, loss = 0.41902517\n",
            "Validation score: 0.926154\n",
            "Iteration 11, loss = 0.38862907\n",
            "Validation score: 0.907692\n",
            "Iteration 12, loss = 0.35579037\n",
            "Validation score: 0.898462\n",
            "Iteration 13, loss = 0.33473345\n",
            "Validation score: 0.913846\n",
            "Iteration 14, loss = 0.31202178\n",
            "Validation score: 0.913846\n",
            "Iteration 15, loss = 0.29664094\n",
            "Validation score: 0.926154\n",
            "Iteration 16, loss = 0.28250439\n",
            "Validation score: 0.913846\n",
            "Iteration 17, loss = 0.27298077\n",
            "Validation score: 0.913846\n",
            "Iteration 18, loss = 0.26042735\n",
            "Validation score: 0.898462\n",
            "Iteration 19, loss = 0.25420367\n",
            "Validation score: 0.916923\n",
            "Iteration 20, loss = 0.24843623\n",
            "Validation score: 0.932308\n",
            "Iteration 21, loss = 0.24174878\n",
            "Validation score: 0.898462\n",
            "Iteration 22, loss = 0.23821999\n",
            "Validation score: 0.907692\n",
            "Iteration 23, loss = 0.23207464\n",
            "Validation score: 0.895385\n",
            "Iteration 24, loss = 0.22983630\n",
            "Validation score: 0.920000\n",
            "Iteration 25, loss = 0.22840943\n",
            "Validation score: 0.892308\n",
            "Iteration 26, loss = 0.22301124\n",
            "Validation score: 0.923077\n",
            "Iteration 27, loss = 0.22645946\n",
            "Validation score: 0.889231\n",
            "Iteration 28, loss = 0.22490013\n",
            "Validation score: 0.916923\n",
            "Iteration 29, loss = 0.21905743\n",
            "Validation score: 0.916923\n",
            "Iteration 30, loss = 0.22038596\n",
            "Validation score: 0.883077\n",
            "Iteration 31, loss = 0.22107658\n",
            "Validation score: 0.913846\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 09:56:31,288]\u001b[0m Trial 57 finished with value: 91.6744 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.0034794465172526103, 'batch_size': 1, 'learning_rate_init': 0.00774733350990319, 'max_iter': 221, 'momentum': 0.15504025343112632, 'number_layers': 5, 'hidden_neurons_0': 113, 'hidden_neurons_1': 197, 'hidden_neurons_2': 146, 'hidden_neurons_3': 70, 'hidden_neurons_4': 74}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.56674015\n",
            "Validation score: 0.769231\n",
            "Iteration 2, loss = 0.56578206\n",
            "Validation score: 0.769231\n",
            "Iteration 3, loss = 0.56405406\n",
            "Validation score: 0.769231\n",
            "Iteration 4, loss = 0.56351142\n",
            "Validation score: 0.769231\n",
            "Iteration 5, loss = 0.56489016\n",
            "Validation score: 0.769231\n",
            "Iteration 6, loss = 0.56538450\n",
            "Validation score: 0.769231\n",
            "Iteration 7, loss = 0.56369606\n",
            "Validation score: 0.769231\n",
            "Iteration 8, loss = 0.56513009\n",
            "Validation score: 0.769231\n",
            "Iteration 9, loss = 0.56540973\n",
            "Validation score: 0.769231\n",
            "Iteration 10, loss = 0.56469533\n",
            "Validation score: 0.769231\n",
            "Iteration 11, loss = 0.56469908\n",
            "Validation score: 0.769231\n",
            "Iteration 12, loss = 0.56363026\n",
            "Validation score: 0.769231\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 09:56:44,687]\u001b[0m Trial 58 finished with value: 76.7808 and parameters: {'actv': 'logistic', 'solvers': 'sgd', 'alpha': 0.0021336589212677917, 'batch_size': 16, 'learning_rate_init': 0.008252738756471785, 'max_iter': 311, 'momentum': 0.6156276901527343, 'number_layers': 5, 'hidden_neurons_0': 155, 'hidden_neurons_1': 149, 'hidden_neurons_2': 196, 'hidden_neurons_3': 178, 'hidden_neurons_4': 105}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.28602846\n",
            "Validation score: 0.910769\n",
            "Iteration 2, loss = 0.22541072\n",
            "Validation score: 0.867692\n",
            "Iteration 3, loss = 0.21649341\n",
            "Validation score: 0.907692\n",
            "Iteration 4, loss = 0.21182238\n",
            "Validation score: 0.901538\n",
            "Iteration 5, loss = 0.20788886\n",
            "Validation score: 0.913846\n",
            "Iteration 6, loss = 0.20850142\n",
            "Validation score: 0.910769\n",
            "Iteration 7, loss = 0.20285774\n",
            "Validation score: 0.913846\n",
            "Iteration 8, loss = 0.20150302\n",
            "Validation score: 0.904615\n",
            "Iteration 9, loss = 0.19898842\n",
            "Validation score: 0.910769\n",
            "Iteration 10, loss = 0.19800289\n",
            "Validation score: 0.913846\n",
            "Iteration 11, loss = 0.19490232\n",
            "Validation score: 0.916923\n",
            "Iteration 12, loss = 0.19187164\n",
            "Validation score: 0.861538\n",
            "Iteration 13, loss = 0.19359919\n",
            "Validation score: 0.910769\n",
            "Iteration 14, loss = 0.19256949\n",
            "Validation score: 0.913846\n",
            "Iteration 15, loss = 0.19011526\n",
            "Validation score: 0.916923\n",
            "Iteration 16, loss = 0.18827682\n",
            "Validation score: 0.910769\n",
            "Iteration 17, loss = 0.18667374\n",
            "Validation score: 0.910769\n",
            "Iteration 18, loss = 0.18844758\n",
            "Validation score: 0.913846\n",
            "Iteration 19, loss = 0.18499341\n",
            "Validation score: 0.907692\n",
            "Iteration 20, loss = 0.18231729\n",
            "Validation score: 0.920000\n",
            "Iteration 21, loss = 0.18423642\n",
            "Validation score: 0.913846\n",
            "Iteration 22, loss = 0.18132827\n",
            "Validation score: 0.910769\n",
            "Iteration 23, loss = 0.18394394\n",
            "Validation score: 0.920000\n",
            "Iteration 24, loss = 0.18055309\n",
            "Validation score: 0.913846\n",
            "Iteration 25, loss = 0.17822113\n",
            "Validation score: 0.913846\n",
            "Iteration 26, loss = 0.17919340\n",
            "Validation score: 0.920000\n",
            "Iteration 27, loss = 0.17547221\n",
            "Validation score: 0.910769\n",
            "Iteration 28, loss = 0.17801857\n",
            "Validation score: 0.901538\n",
            "Iteration 29, loss = 0.17574024\n",
            "Validation score: 0.926154\n",
            "Iteration 30, loss = 0.17528135\n",
            "Validation score: 0.898462\n",
            "Iteration 31, loss = 0.17289495\n",
            "Validation score: 0.910769\n",
            "Iteration 32, loss = 0.17252495\n",
            "Validation score: 0.926154\n",
            "Iteration 33, loss = 0.17427646\n",
            "Validation score: 0.904615\n",
            "Iteration 34, loss = 0.17193911\n",
            "Validation score: 0.907692\n",
            "Iteration 35, loss = 0.16971017\n",
            "Validation score: 0.913846\n",
            "Iteration 36, loss = 0.17057990\n",
            "Validation score: 0.916923\n",
            "Iteration 37, loss = 0.16891758\n",
            "Validation score: 0.910769\n",
            "Iteration 38, loss = 0.17018675\n",
            "Validation score: 0.904615\n",
            "Iteration 39, loss = 0.17302021\n",
            "Validation score: 0.929231\n",
            "Iteration 40, loss = 0.16386270\n",
            "Validation score: 0.913846\n",
            "Iteration 41, loss = 0.16347822\n",
            "Validation score: 0.913846\n",
            "Iteration 42, loss = 0.16524692\n",
            "Validation score: 0.913846\n",
            "Iteration 43, loss = 0.16121497\n",
            "Validation score: 0.901538\n",
            "Iteration 44, loss = 0.16691469\n",
            "Validation score: 0.913846\n",
            "Iteration 45, loss = 0.15897614\n",
            "Validation score: 0.913846\n",
            "Iteration 46, loss = 0.15772406\n",
            "Validation score: 0.916923\n",
            "Iteration 47, loss = 0.15701042\n",
            "Validation score: 0.916923\n",
            "Iteration 48, loss = 0.15496598\n",
            "Validation score: 0.858462\n",
            "Iteration 49, loss = 0.15618447\n",
            "Validation score: 0.916923\n",
            "Iteration 50, loss = 0.15652389\n",
            "Validation score: 0.876923\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 09:57:29,538]\u001b[0m Trial 59 finished with value: 91.4894 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.0016750799286213618, 'batch_size': 16, 'learning_rate_init': 0.009091968784830085, 'max_iter': 279, 'momentum': 0.47665839059822773, 'number_layers': 2, 'hidden_neurons_0': 169, 'hidden_neurons_1': 108}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.33207890\n",
            "Validation score: 0.889231\n",
            "Iteration 2, loss = 0.23768646\n",
            "Validation score: 0.910769\n",
            "Iteration 3, loss = 0.21715582\n",
            "Validation score: 0.916923\n",
            "Iteration 4, loss = 0.20518312\n",
            "Validation score: 0.910769\n",
            "Iteration 5, loss = 0.20149479\n",
            "Validation score: 0.901538\n",
            "Iteration 6, loss = 0.19562716\n",
            "Validation score: 0.904615\n",
            "Iteration 7, loss = 0.19643876\n",
            "Validation score: 0.910769\n",
            "Iteration 8, loss = 0.19476138\n",
            "Validation score: 0.910769\n",
            "Iteration 9, loss = 0.19115366\n",
            "Validation score: 0.923077\n",
            "Iteration 10, loss = 0.19110972\n",
            "Validation score: 0.920000\n",
            "Iteration 11, loss = 0.18576201\n",
            "Validation score: 0.920000\n",
            "Iteration 12, loss = 0.18671278\n",
            "Validation score: 0.920000\n",
            "Iteration 13, loss = 0.18446313\n",
            "Validation score: 0.920000\n",
            "Iteration 14, loss = 0.18370521\n",
            "Validation score: 0.920000\n",
            "Iteration 15, loss = 0.18267852\n",
            "Validation score: 0.920000\n",
            "Iteration 16, loss = 0.18234381\n",
            "Validation score: 0.920000\n",
            "Iteration 17, loss = 0.18233534\n",
            "Validation score: 0.907692\n",
            "Iteration 18, loss = 0.18026077\n",
            "Validation score: 0.892308\n",
            "Iteration 19, loss = 0.18010359\n",
            "Validation score: 0.910769\n",
            "Iteration 20, loss = 0.18020690\n",
            "Validation score: 0.883077\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 09:57:34,108]\u001b[0m Trial 60 finished with value: 92.0444 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.0026603553678046744, 'batch_size': 32, 'learning_rate_init': 0.006402168435361328, 'max_iter': 225, 'momentum': 0.683298974861529, 'number_layers': 3, 'hidden_neurons_0': 53, 'hidden_neurons_1': 162, 'hidden_neurons_2': 64}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.39629693\n",
            "Validation score: 0.916923\n",
            "Iteration 2, loss = 0.24652751\n",
            "Validation score: 0.932308\n",
            "Iteration 3, loss = 0.23337503\n",
            "Validation score: 0.889231\n",
            "Iteration 4, loss = 0.22145566\n",
            "Validation score: 0.920000\n",
            "Iteration 5, loss = 0.22025732\n",
            "Validation score: 0.929231\n",
            "Iteration 6, loss = 0.21739663\n",
            "Validation score: 0.941538\n",
            "Iteration 7, loss = 0.21552209\n",
            "Validation score: 0.935385\n",
            "Iteration 8, loss = 0.21020795\n",
            "Validation score: 0.932308\n",
            "Iteration 9, loss = 0.20698231\n",
            "Validation score: 0.907692\n",
            "Iteration 10, loss = 0.20817988\n",
            "Validation score: 0.926154\n",
            "Iteration 11, loss = 0.21227969\n",
            "Validation score: 0.941538\n",
            "Iteration 12, loss = 0.20539059\n",
            "Validation score: 0.941538\n",
            "Iteration 13, loss = 0.20531560\n",
            "Validation score: 0.895385\n",
            "Iteration 14, loss = 0.20126943\n",
            "Validation score: 0.938462\n",
            "Iteration 15, loss = 0.20364450\n",
            "Validation score: 0.932308\n",
            "Iteration 16, loss = 0.19838034\n",
            "Validation score: 0.910769\n",
            "Iteration 17, loss = 0.19745639\n",
            "Validation score: 0.907692\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 09:57:40,109]\u001b[0m Trial 61 finished with value: 92.0444 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.00268041208313838, 'batch_size': 32, 'learning_rate_init': 0.006475401868621897, 'max_iter': 200, 'momentum': 0.7406085431252851, 'number_layers': 5, 'hidden_neurons_0': 52, 'hidden_neurons_1': 186, 'hidden_neurons_2': 175, 'hidden_neurons_3': 165, 'hidden_neurons_4': 127}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.35120647\n",
            "Validation score: 0.920000\n",
            "Iteration 2, loss = 0.23577167\n",
            "Validation score: 0.929231\n",
            "Iteration 3, loss = 0.21649886\n",
            "Validation score: 0.932308\n",
            "Iteration 4, loss = 0.20561853\n",
            "Validation score: 0.713846\n",
            "Iteration 5, loss = 0.20778195\n",
            "Validation score: 0.920000\n",
            "Iteration 6, loss = 0.20276857\n",
            "Validation score: 0.947692\n",
            "Iteration 7, loss = 0.19843661\n",
            "Validation score: 0.944615\n",
            "Iteration 8, loss = 0.19622139\n",
            "Validation score: 0.947692\n",
            "Iteration 9, loss = 0.19787471\n",
            "Validation score: 0.938462\n",
            "Iteration 10, loss = 0.19292962\n",
            "Validation score: 0.941538\n",
            "Iteration 11, loss = 0.19209071\n",
            "Validation score: 0.938462\n",
            "Iteration 12, loss = 0.18967705\n",
            "Validation score: 0.941538\n",
            "Iteration 13, loss = 0.18926120\n",
            "Validation score: 0.932308\n",
            "Iteration 14, loss = 0.18785763\n",
            "Validation score: 0.950769\n",
            "Iteration 15, loss = 0.18690368\n",
            "Validation score: 0.944615\n",
            "Iteration 16, loss = 0.18543165\n",
            "Validation score: 0.655385\n",
            "Iteration 17, loss = 0.19214540\n",
            "Validation score: 0.944615\n",
            "Iteration 18, loss = 0.18451648\n",
            "Validation score: 0.947692\n",
            "Iteration 19, loss = 0.17943518\n",
            "Validation score: 0.947692\n",
            "Iteration 20, loss = 0.18141703\n",
            "Validation score: 0.944615\n",
            "Iteration 21, loss = 0.18052529\n",
            "Validation score: 0.901538\n",
            "Iteration 22, loss = 0.18199438\n",
            "Validation score: 0.947692\n",
            "Iteration 23, loss = 0.17760780\n",
            "Validation score: 0.935385\n",
            "Iteration 24, loss = 0.17785156\n",
            "Validation score: 0.935385\n",
            "Iteration 25, loss = 0.18112373\n",
            "Validation score: 0.944615\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 09:57:48,019]\u001b[0m Trial 62 finished with value: 91.3043 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.0009628696077073518, 'batch_size': 32, 'learning_rate_init': 0.008500769676494808, 'max_iter': 159, 'momentum': 0.5899255566208146, 'number_layers': 4, 'hidden_neurons_0': 65, 'hidden_neurons_1': 126, 'hidden_neurons_2': 179, 'hidden_neurons_3': 177}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 1.53832843\n",
            "Validation score: 0.889231\n",
            "Iteration 2, loss = 0.96087367\n",
            "Validation score: 0.920000\n",
            "Iteration 3, loss = 0.64799813\n",
            "Validation score: 0.923077\n",
            "Iteration 4, loss = 0.47365504\n",
            "Validation score: 0.913846\n",
            "Iteration 5, loss = 0.37245479\n",
            "Validation score: 0.858462\n",
            "Iteration 6, loss = 0.31853806\n",
            "Validation score: 0.861538\n",
            "Iteration 7, loss = 0.28473118\n",
            "Validation score: 0.843077\n",
            "Iteration 8, loss = 0.27028845\n",
            "Validation score: 0.929231\n",
            "Iteration 9, loss = 0.26447099\n",
            "Validation score: 0.852308\n",
            "Iteration 10, loss = 0.25364874\n",
            "Validation score: 0.886154\n",
            "Iteration 11, loss = 0.25002214\n",
            "Validation score: 0.907692\n",
            "Iteration 12, loss = 0.25070485\n",
            "Validation score: 0.923077\n",
            "Iteration 13, loss = 0.24418471\n",
            "Validation score: 0.750769\n",
            "Iteration 14, loss = 0.25185665\n",
            "Validation score: 0.852308\n",
            "Iteration 15, loss = 0.25076244\n",
            "Validation score: 0.821538\n",
            "Iteration 16, loss = 0.24813073\n",
            "Validation score: 0.935385\n",
            "Iteration 17, loss = 0.24751367\n",
            "Validation score: 0.901538\n",
            "Iteration 18, loss = 0.24653661\n",
            "Validation score: 0.923077\n",
            "Iteration 19, loss = 0.24507965\n",
            "Validation score: 0.913846\n",
            "Iteration 20, loss = 0.24460345\n",
            "Validation score: 0.913846\n",
            "Iteration 21, loss = 0.24887841\n",
            "Validation score: 0.916923\n",
            "Iteration 22, loss = 0.24829701\n",
            "Validation score: 0.907692\n",
            "Iteration 23, loss = 0.25076429\n",
            "Validation score: 0.932308\n",
            "Iteration 24, loss = 0.24396653\n",
            "Validation score: 0.923077\n",
            "Iteration 25, loss = 0.24802996\n",
            "Validation score: 0.926154\n",
            "Iteration 26, loss = 0.24157226\n",
            "Validation score: 0.916923\n",
            "Iteration 27, loss = 0.24825481\n",
            "Validation score: 0.907692\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 10:04:04,420]\u001b[0m Trial 63 finished with value: 91.4894 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.004705743005979755, 'batch_size': 1, 'learning_rate_init': 0.007338588664989586, 'max_iter': 257, 'momentum': 0.6502710800340347, 'number_layers': 4, 'hidden_neurons_0': 129, 'hidden_neurons_1': 180, 'hidden_neurons_2': 149, 'hidden_neurons_3': 152}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 1.64413640\n",
            "Validation score: 0.932308\n",
            "Iteration 2, loss = 1.09531770\n",
            "Validation score: 0.858462\n",
            "Iteration 3, loss = 0.77187207\n",
            "Validation score: 0.916923\n",
            "Iteration 4, loss = 0.57079702\n",
            "Validation score: 0.920000\n",
            "Iteration 5, loss = 0.44990231\n",
            "Validation score: 0.926154\n",
            "Iteration 6, loss = 0.36807045\n",
            "Validation score: 0.935385\n",
            "Iteration 7, loss = 0.32542211\n",
            "Validation score: 0.926154\n",
            "Iteration 8, loss = 0.29367713\n",
            "Validation score: 0.926154\n",
            "Iteration 9, loss = 0.27727652\n",
            "Validation score: 0.916923\n",
            "Iteration 10, loss = 0.26102670\n",
            "Validation score: 0.920000\n",
            "Iteration 11, loss = 0.25884073\n",
            "Validation score: 0.923077\n",
            "Iteration 12, loss = 0.25092036\n",
            "Validation score: 0.926154\n",
            "Iteration 13, loss = 0.24632813\n",
            "Validation score: 0.910769\n",
            "Iteration 14, loss = 0.25077116\n",
            "Validation score: 0.923077\n",
            "Iteration 15, loss = 0.24849993\n",
            "Validation score: 0.920000\n",
            "Iteration 16, loss = 0.24652022\n",
            "Validation score: 0.920000\n",
            "Iteration 17, loss = 0.24688944\n",
            "Validation score: 0.938462\n",
            "Iteration 18, loss = 0.24313952\n",
            "Validation score: 0.938462\n",
            "Iteration 19, loss = 0.24428059\n",
            "Validation score: 0.932308\n",
            "Iteration 20, loss = 0.24461497\n",
            "Validation score: 0.938462\n",
            "Iteration 21, loss = 0.24548813\n",
            "Validation score: 0.926154\n",
            "Iteration 22, loss = 0.24198486\n",
            "Validation score: 0.938462\n",
            "Iteration 23, loss = 0.24429539\n",
            "Validation score: 0.895385\n",
            "Iteration 24, loss = 0.24720985\n",
            "Validation score: 0.920000\n",
            "Iteration 25, loss = 0.24329266\n",
            "Validation score: 0.932308\n",
            "Iteration 26, loss = 0.24038143\n",
            "Validation score: 0.916923\n",
            "Iteration 27, loss = 0.24771896\n",
            "Validation score: 0.910769\n",
            "Iteration 28, loss = 0.24671221\n",
            "Validation score: 0.929231\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 10:12:51,919]\u001b[0m Trial 64 finished with value: 91.8594 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.004012833705171148, 'batch_size': 1, 'learning_rate_init': 0.005965773090809555, 'max_iter': 244, 'momentum': 0.7070113344900641, 'number_layers': 5, 'hidden_neurons_0': 143, 'hidden_neurons_1': 195, 'hidden_neurons_2': 161, 'hidden_neurons_3': 97, 'hidden_neurons_4': 165}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.34836576\n",
            "Validation score: 0.916923\n",
            "Iteration 2, loss = 0.23835714\n",
            "Validation score: 0.926154\n",
            "Iteration 3, loss = 0.21843188\n",
            "Validation score: 0.926154\n",
            "Iteration 4, loss = 0.20844532\n",
            "Validation score: 0.923077\n",
            "Iteration 5, loss = 0.20590119\n",
            "Validation score: 0.932308\n",
            "Iteration 6, loss = 0.20022228\n",
            "Validation score: 0.935385\n",
            "Iteration 7, loss = 0.19782290\n",
            "Validation score: 0.938462\n",
            "Iteration 8, loss = 0.19729319\n",
            "Validation score: 0.935385\n",
            "Iteration 9, loss = 0.19958550\n",
            "Validation score: 0.926154\n",
            "Iteration 10, loss = 0.19397544\n",
            "Validation score: 0.932308\n",
            "Iteration 11, loss = 0.19468131\n",
            "Validation score: 0.923077\n",
            "Iteration 12, loss = 0.19084935\n",
            "Validation score: 0.913846\n",
            "Iteration 13, loss = 0.19233223\n",
            "Validation score: 0.938462\n",
            "Iteration 14, loss = 0.18822204\n",
            "Validation score: 0.858462\n",
            "Iteration 15, loss = 0.19164431\n",
            "Validation score: 0.923077\n",
            "Iteration 16, loss = 0.18567805\n",
            "Validation score: 0.910769\n",
            "Iteration 17, loss = 0.18716349\n",
            "Validation score: 0.950769\n",
            "Iteration 18, loss = 0.18249412\n",
            "Validation score: 0.941538\n",
            "Iteration 19, loss = 0.18373920\n",
            "Validation score: 0.950769\n",
            "Iteration 20, loss = 0.18198690\n",
            "Validation score: 0.947692\n",
            "Iteration 21, loss = 0.18150157\n",
            "Validation score: 0.932308\n",
            "Iteration 22, loss = 0.17917808\n",
            "Validation score: 0.932308\n",
            "Iteration 23, loss = 0.17721984\n",
            "Validation score: 0.950769\n",
            "Iteration 24, loss = 0.17678083\n",
            "Validation score: 0.916923\n",
            "Iteration 25, loss = 0.17865525\n",
            "Validation score: 0.947692\n",
            "Iteration 26, loss = 0.17706079\n",
            "Validation score: 0.932308\n",
            "Iteration 27, loss = 0.17497792\n",
            "Validation score: 0.941538\n",
            "Iteration 28, loss = 0.17227472\n",
            "Validation score: 0.944615\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 10:12:58,301]\u001b[0m Trial 65 finished with value: 91.8594 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.0025829380006168294, 'batch_size': 32, 'learning_rate_init': 0.005412161273063384, 'max_iter': 178, 'momentum': 0.7499119431217741, 'number_layers': 3, 'hidden_neurons_0': 50, 'hidden_neurons_1': 161, 'hidden_neurons_2': 57}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.36567214\n",
            "Validation score: 0.876923\n",
            "Iteration 2, loss = 0.24435240\n",
            "Validation score: 0.907692\n",
            "Iteration 3, loss = 0.21868577\n",
            "Validation score: 0.806154\n",
            "Iteration 4, loss = 0.21334774\n",
            "Validation score: 0.935385\n",
            "Iteration 5, loss = 0.19887370\n",
            "Validation score: 0.932308\n",
            "Iteration 6, loss = 0.19460681\n",
            "Validation score: 0.876923\n",
            "Iteration 7, loss = 0.19438645\n",
            "Validation score: 0.923077\n",
            "Iteration 8, loss = 0.19141576\n",
            "Validation score: 0.929231\n",
            "Iteration 9, loss = 0.19117435\n",
            "Validation score: 0.923077\n",
            "Iteration 10, loss = 0.18928600\n",
            "Validation score: 0.932308\n",
            "Iteration 11, loss = 0.18764340\n",
            "Validation score: 0.929231\n",
            "Iteration 12, loss = 0.18772581\n",
            "Validation score: 0.932308\n",
            "Iteration 13, loss = 0.18387602\n",
            "Validation score: 0.852308\n",
            "Iteration 14, loss = 0.18714343\n",
            "Validation score: 0.935385\n",
            "Iteration 15, loss = 0.18142031\n",
            "Validation score: 0.935385\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 10:13:02,406]\u001b[0m Trial 66 finished with value: 92.3219 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.0019300239325546207, 'batch_size': 32, 'learning_rate_init': 0.004995829862558102, 'max_iter': 208, 'momentum': 0.7400935030899899, 'number_layers': 3, 'hidden_neurons_0': 55, 'hidden_neurons_1': 142, 'hidden_neurons_2': 69}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.30452695\n",
            "Validation score: 0.907692\n",
            "Iteration 2, loss = 0.24566568\n",
            "Validation score: 0.926154\n",
            "Iteration 3, loss = 0.25790783\n",
            "Validation score: 0.898462\n",
            "Iteration 4, loss = 0.26899330\n",
            "Validation score: 0.920000\n",
            "Iteration 5, loss = 0.27315020\n",
            "Validation score: 0.926154\n",
            "Iteration 6, loss = 0.25861889\n",
            "Validation score: 0.932308\n",
            "Iteration 7, loss = 0.23871289\n",
            "Validation score: 0.929231\n",
            "Iteration 8, loss = 0.26404607\n",
            "Validation score: 0.923077\n",
            "Iteration 9, loss = 0.26266862\n",
            "Validation score: 0.904615\n",
            "Iteration 10, loss = 0.27423473\n",
            "Validation score: 0.929231\n",
            "Iteration 11, loss = 0.23531746\n",
            "Validation score: 0.932308\n",
            "Iteration 12, loss = 0.24572777\n",
            "Validation score: 0.910769\n",
            "Iteration 13, loss = 0.27901968\n",
            "Validation score: 0.889231\n",
            "Iteration 14, loss = 0.25030233\n",
            "Validation score: 0.923077\n",
            "Iteration 15, loss = 0.23287862\n",
            "Validation score: 0.916923\n",
            "Iteration 16, loss = 0.23402525\n",
            "Validation score: 0.880000\n",
            "Iteration 17, loss = 0.25687866\n",
            "Validation score: 0.916923\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 10:13:08,858]\u001b[0m Trial 67 finished with value: 91.7669 and parameters: {'actv': 'tanh', 'solvers': 'adam', 'alpha': 0.0019436332489615766, 'batch_size': 32, 'learning_rate_init': 0.004857822239369917, 'max_iter': 241, 'momentum': 0.40966813239867417, 'number_layers': 3, 'hidden_neurons_0': 66, 'hidden_neurons_1': 138, 'hidden_neurons_2': 79}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.57738600\n",
            "Validation score: 0.769231\n",
            "Iteration 2, loss = 0.54287216\n",
            "Validation score: 0.769231\n",
            "Iteration 3, loss = 0.54190074\n",
            "Validation score: 0.769231\n",
            "Iteration 4, loss = 0.54035268\n",
            "Validation score: 0.769231\n",
            "Iteration 5, loss = 0.53944728\n",
            "Validation score: 0.769231\n",
            "Iteration 6, loss = 0.53798967\n",
            "Validation score: 0.769231\n",
            "Iteration 7, loss = 0.53659424\n",
            "Validation score: 0.769231\n",
            "Iteration 8, loss = 0.53529244\n",
            "Validation score: 0.769231\n",
            "Iteration 9, loss = 0.53377765\n",
            "Validation score: 0.769231\n",
            "Iteration 10, loss = 0.53221210\n",
            "Validation score: 0.769231\n",
            "Iteration 11, loss = 0.53063125\n",
            "Validation score: 0.769231\n",
            "Iteration 12, loss = 0.52885190\n",
            "Validation score: 0.769231\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 10:13:10,877]\u001b[0m Trial 68 finished with value: 76.7808 and parameters: {'actv': 'logistic', 'solvers': 'sgd', 'alpha': 0.003155699072315875, 'batch_size': 128, 'learning_rate_init': 0.0041929974734931305, 'max_iter': 273, 'momentum': 0.7923144106742082, 'number_layers': 2, 'hidden_neurons_0': 59, 'hidden_neurons_1': 141}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.29527847\n",
            "Validation score: 0.926154\n",
            "Iteration 2, loss = 0.23127637\n",
            "Validation score: 0.904615\n",
            "Iteration 3, loss = 0.21297785\n",
            "Validation score: 0.932308\n",
            "Iteration 4, loss = 0.20257198\n",
            "Validation score: 0.938462\n",
            "Iteration 5, loss = 0.20066206\n",
            "Validation score: 0.916923\n",
            "Iteration 6, loss = 0.20001075\n",
            "Validation score: 0.935385\n",
            "Iteration 7, loss = 0.19433798\n",
            "Validation score: 0.932308\n",
            "Iteration 8, loss = 0.19295036\n",
            "Validation score: 0.935385\n",
            "Iteration 9, loss = 0.19172028\n",
            "Validation score: 0.929231\n",
            "Iteration 10, loss = 0.18903532\n",
            "Validation score: 0.907692\n",
            "Iteration 11, loss = 0.18889935\n",
            "Validation score: 0.926154\n",
            "Iteration 12, loss = 0.18619449\n",
            "Validation score: 0.935385\n",
            "Iteration 13, loss = 0.18503573\n",
            "Validation score: 0.935385\n",
            "Iteration 14, loss = 0.18427449\n",
            "Validation score: 0.923077\n",
            "Iteration 15, loss = 0.18335893\n",
            "Validation score: 0.938462\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 10:13:14,177]\u001b[0m Trial 69 finished with value: 91.1193 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.002769776660328952, 'batch_size': 32, 'learning_rate_init': 0.0055378012621058416, 'max_iter': 149, 'momentum': 0.7433749521515951, 'number_layers': 1, 'hidden_neurons_0': 54}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.46838562\n",
            "Validation score: 0.889231\n",
            "Iteration 2, loss = 0.30156260\n",
            "Validation score: 0.907692\n",
            "Iteration 3, loss = 0.26800959\n",
            "Validation score: 0.883077\n",
            "Iteration 4, loss = 0.24649532\n",
            "Validation score: 0.913846\n",
            "Iteration 5, loss = 0.23117842\n",
            "Validation score: 0.907692\n",
            "Iteration 6, loss = 0.22188141\n",
            "Validation score: 0.895385\n",
            "Iteration 7, loss = 0.21686236\n",
            "Validation score: 0.913846\n",
            "Iteration 8, loss = 0.21079542\n",
            "Validation score: 0.901538\n",
            "Iteration 9, loss = 0.21007346\n",
            "Validation score: 0.923077\n",
            "Iteration 10, loss = 0.20820770\n",
            "Validation score: 0.904615\n",
            "Iteration 11, loss = 0.20702183\n",
            "Validation score: 0.904615\n",
            "Iteration 12, loss = 0.20609519\n",
            "Validation score: 0.895385\n",
            "Iteration 13, loss = 0.20310426\n",
            "Validation score: 0.907692\n",
            "Iteration 14, loss = 0.20241765\n",
            "Validation score: 0.923077\n",
            "Iteration 15, loss = 0.19960936\n",
            "Validation score: 0.904615\n",
            "Iteration 16, loss = 0.20127553\n",
            "Validation score: 0.913846\n",
            "Iteration 17, loss = 0.20013511\n",
            "Validation score: 0.913846\n",
            "Iteration 18, loss = 0.20122365\n",
            "Validation score: 0.916923\n",
            "Iteration 19, loss = 0.20037726\n",
            "Validation score: 0.923077\n",
            "Iteration 20, loss = 0.19755026\n",
            "Validation score: 0.913846\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 10:13:19,589]\u001b[0m Trial 70 finished with value: 91.2118 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.0041223023885885906, 'batch_size': 32, 'learning_rate_init': 0.00447380377925581, 'max_iter': 222, 'momentum': 0.3537420870763817, 'number_layers': 3, 'hidden_neurons_0': 57, 'hidden_neurons_1': 152, 'hidden_neurons_2': 72}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.33493659\n",
            "Validation score: 0.916923\n",
            "Iteration 2, loss = 0.23801891\n",
            "Validation score: 0.923077\n",
            "Iteration 3, loss = 0.21769531\n",
            "Validation score: 0.913846\n",
            "Iteration 4, loss = 0.20797642\n",
            "Validation score: 0.916923\n",
            "Iteration 5, loss = 0.20175048\n",
            "Validation score: 0.892308\n",
            "Iteration 6, loss = 0.19988375\n",
            "Validation score: 0.929231\n",
            "Iteration 7, loss = 0.19568738\n",
            "Validation score: 0.929231\n",
            "Iteration 8, loss = 0.19480275\n",
            "Validation score: 0.916923\n",
            "Iteration 9, loss = 0.19194922\n",
            "Validation score: 0.926154\n",
            "Iteration 10, loss = 0.19399552\n",
            "Validation score: 0.892308\n",
            "Iteration 11, loss = 0.18909476\n",
            "Validation score: 0.873846\n",
            "Iteration 12, loss = 0.18968865\n",
            "Validation score: 0.944615\n",
            "Iteration 13, loss = 0.18352128\n",
            "Validation score: 0.880000\n",
            "Iteration 14, loss = 0.18708809\n",
            "Validation score: 0.910769\n",
            "Iteration 15, loss = 0.18561708\n",
            "Validation score: 0.941538\n",
            "Iteration 16, loss = 0.18212224\n",
            "Validation score: 0.932308\n",
            "Iteration 17, loss = 0.18167623\n",
            "Validation score: 0.944615\n",
            "Iteration 18, loss = 0.18331537\n",
            "Validation score: 0.916923\n",
            "Iteration 19, loss = 0.17772815\n",
            "Validation score: 0.929231\n",
            "Iteration 20, loss = 0.18193661\n",
            "Validation score: 0.886154\n",
            "Iteration 21, loss = 0.17898583\n",
            "Validation score: 0.873846\n",
            "Iteration 22, loss = 0.18414477\n",
            "Validation score: 0.913846\n",
            "Iteration 23, loss = 0.17709713\n",
            "Validation score: 0.947692\n",
            "Iteration 24, loss = 0.17431454\n",
            "Validation score: 0.935385\n",
            "Iteration 25, loss = 0.17746415\n",
            "Validation score: 0.883077\n",
            "Iteration 26, loss = 0.17441375\n",
            "Validation score: 0.907692\n",
            "Iteration 27, loss = 0.17283311\n",
            "Validation score: 0.941538\n",
            "Iteration 28, loss = 0.17305989\n",
            "Validation score: 0.935385\n",
            "Iteration 29, loss = 0.17059992\n",
            "Validation score: 0.938462\n",
            "Iteration 30, loss = 0.16927921\n",
            "Validation score: 0.941538\n",
            "Iteration 31, loss = 0.16822275\n",
            "Validation score: 0.920000\n",
            "Iteration 32, loss = 0.16755297\n",
            "Validation score: 0.935385\n",
            "Iteration 33, loss = 0.16657032\n",
            "Validation score: 0.923077\n",
            "Iteration 34, loss = 0.16750625\n",
            "Validation score: 0.938462\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 10:13:30,113]\u001b[0m Trial 71 finished with value: 91.7669 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.0023204834529910114, 'batch_size': 32, 'learning_rate_init': 0.006133764112987737, 'max_iter': 206, 'momentum': 0.6794203362996298, 'number_layers': 3, 'hidden_neurons_0': 80, 'hidden_neurons_1': 144, 'hidden_neurons_2': 67}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.32704411\n",
            "Validation score: 0.941538\n",
            "Iteration 2, loss = 0.24025277\n",
            "Validation score: 0.941538\n",
            "Iteration 3, loss = 0.22813052\n",
            "Validation score: 0.963077\n",
            "Iteration 4, loss = 0.22274727\n",
            "Validation score: 0.963077\n",
            "Iteration 5, loss = 0.21822688\n",
            "Validation score: 0.963077\n",
            "Iteration 6, loss = 0.21410328\n",
            "Validation score: 0.947692\n",
            "Iteration 7, loss = 0.21046626\n",
            "Validation score: 0.960000\n",
            "Iteration 8, loss = 0.21113664\n",
            "Validation score: 0.956923\n",
            "Iteration 9, loss = 0.21045214\n",
            "Validation score: 0.953846\n",
            "Iteration 10, loss = 0.20784829\n",
            "Validation score: 0.960000\n",
            "Iteration 11, loss = 0.20431633\n",
            "Validation score: 0.950769\n",
            "Iteration 12, loss = 0.20456598\n",
            "Validation score: 0.966154\n",
            "Iteration 13, loss = 0.20424034\n",
            "Validation score: 0.966154\n",
            "Iteration 14, loss = 0.19859891\n",
            "Validation score: 0.963077\n",
            "Iteration 15, loss = 0.20090925\n",
            "Validation score: 0.950769\n",
            "Iteration 16, loss = 0.20013530\n",
            "Validation score: 0.963077\n",
            "Iteration 17, loss = 0.19705882\n",
            "Validation score: 0.963077\n",
            "Iteration 18, loss = 0.19921572\n",
            "Validation score: 0.938462\n",
            "Iteration 19, loss = 0.19586347\n",
            "Validation score: 0.950769\n",
            "Iteration 20, loss = 0.19507254\n",
            "Validation score: 0.963077\n",
            "Iteration 21, loss = 0.19110308\n",
            "Validation score: 0.935385\n",
            "Iteration 22, loss = 0.19315451\n",
            "Validation score: 0.953846\n",
            "Iteration 23, loss = 0.18935613\n",
            "Validation score: 0.953846\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 10:13:40,952]\u001b[0m Trial 72 finished with value: 91.9519 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.0033819780336776547, 'batch_size': 32, 'learning_rate_init': 0.006787906050557399, 'max_iter': 203, 'momentum': 0.8019611556692853, 'number_layers': 3, 'hidden_neurons_0': 123, 'hidden_neurons_1': 188, 'hidden_neurons_2': 50}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.35261627\n",
            "Validation score: 0.926154\n",
            "Iteration 2, loss = 0.23619093\n",
            "Validation score: 0.920000\n",
            "Iteration 3, loss = 0.21384508\n",
            "Validation score: 0.953846\n",
            "Iteration 4, loss = 0.20405325\n",
            "Validation score: 0.960000\n",
            "Iteration 5, loss = 0.20244011\n",
            "Validation score: 0.895385\n",
            "Iteration 6, loss = 0.20006600\n",
            "Validation score: 0.947692\n",
            "Iteration 7, loss = 0.20020853\n",
            "Validation score: 0.944615\n",
            "Iteration 8, loss = 0.19966499\n",
            "Validation score: 0.907692\n",
            "Iteration 9, loss = 0.19492091\n",
            "Validation score: 0.935385\n",
            "Iteration 10, loss = 0.19109881\n",
            "Validation score: 0.916923\n",
            "Iteration 11, loss = 0.19118467\n",
            "Validation score: 0.953846\n",
            "Iteration 12, loss = 0.18681884\n",
            "Validation score: 0.963077\n",
            "Iteration 13, loss = 0.18673577\n",
            "Validation score: 0.944615\n",
            "Iteration 14, loss = 0.18244965\n",
            "Validation score: 0.953846\n",
            "Iteration 15, loss = 0.18311013\n",
            "Validation score: 0.947692\n",
            "Iteration 16, loss = 0.18542070\n",
            "Validation score: 0.956923\n",
            "Iteration 17, loss = 0.18174732\n",
            "Validation score: 0.963077\n",
            "Iteration 18, loss = 0.17995612\n",
            "Validation score: 0.861538\n",
            "Iteration 19, loss = 0.18523901\n",
            "Validation score: 0.913846\n",
            "Iteration 20, loss = 0.18222775\n",
            "Validation score: 0.923077\n",
            "Iteration 21, loss = 0.18168089\n",
            "Validation score: 0.963077\n",
            "Iteration 22, loss = 0.17832140\n",
            "Validation score: 0.960000\n",
            "Iteration 23, loss = 0.17372045\n",
            "Validation score: 0.963077\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 10:13:56,532]\u001b[0m Trial 73 finished with value: 91.9519 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.0005585067605999414, 'batch_size': 32, 'learning_rate_init': 0.007956233026573986, 'max_iter': 292, 'momentum': 0.5248805097858524, 'number_layers': 4, 'hidden_neurons_0': 149, 'hidden_neurons_1': 124, 'hidden_neurons_2': 187, 'hidden_neurons_3': 170}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.42360952\n",
            "Validation score: 0.892308\n",
            "Iteration 2, loss = 0.25285041\n",
            "Validation score: 0.916923\n",
            "Iteration 3, loss = 0.22318340\n",
            "Validation score: 0.873846\n",
            "Iteration 4, loss = 0.21908654\n",
            "Validation score: 0.910769\n",
            "Iteration 5, loss = 0.20173041\n",
            "Validation score: 0.920000\n",
            "Iteration 6, loss = 0.20378455\n",
            "Validation score: 0.907692\n",
            "Iteration 7, loss = 0.20065707\n",
            "Validation score: 0.935385\n",
            "Iteration 8, loss = 0.19900749\n",
            "Validation score: 0.935385\n",
            "Iteration 9, loss = 0.19586807\n",
            "Validation score: 0.932308\n",
            "Iteration 10, loss = 0.19280397\n",
            "Validation score: 0.932308\n",
            "Iteration 11, loss = 0.19222832\n",
            "Validation score: 0.929231\n",
            "Iteration 12, loss = 0.19091522\n",
            "Validation score: 0.941538\n",
            "Iteration 13, loss = 0.19015860\n",
            "Validation score: 0.855385\n",
            "Iteration 14, loss = 0.19354758\n",
            "Validation score: 0.916923\n",
            "Iteration 15, loss = 0.18934009\n",
            "Validation score: 0.938462\n",
            "Iteration 16, loss = 0.18810627\n",
            "Validation score: 0.932308\n",
            "Iteration 17, loss = 0.18555266\n",
            "Validation score: 0.941538\n",
            "Iteration 18, loss = 0.18366225\n",
            "Validation score: 0.938462\n",
            "Iteration 19, loss = 0.18365679\n",
            "Validation score: 0.941538\n",
            "Iteration 20, loss = 0.18192759\n",
            "Validation score: 0.935385\n",
            "Iteration 21, loss = 0.18222889\n",
            "Validation score: 0.938462\n",
            "Iteration 22, loss = 0.18245151\n",
            "Validation score: 0.938462\n",
            "Iteration 23, loss = 0.17818862\n",
            "Validation score: 0.941538\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 10:14:04,922]\u001b[0m Trial 74 finished with value: 92.2294 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.001573785069954077, 'batch_size': 32, 'learning_rate_init': 0.006402178157565337, 'max_iter': 224, 'momentum': 0.6365153741476665, 'number_layers': 5, 'hidden_neurons_0': 55, 'hidden_neurons_1': 124, 'hidden_neurons_2': 173, 'hidden_neurons_3': 162, 'hidden_neurons_4': 129}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.52725729\n",
            "Validation score: 0.803077\n",
            "Iteration 2, loss = 0.32537033\n",
            "Validation score: 0.904615\n",
            "Iteration 3, loss = 0.26068124\n",
            "Validation score: 0.916923\n",
            "Iteration 4, loss = 0.22960046\n",
            "Validation score: 0.926154\n",
            "Iteration 5, loss = 0.21995221\n",
            "Validation score: 0.932308\n",
            "Iteration 6, loss = 0.20508997\n",
            "Validation score: 0.941538\n",
            "Iteration 7, loss = 0.20096342\n",
            "Validation score: 0.938462\n",
            "Iteration 8, loss = 0.19924806\n",
            "Validation score: 0.886154\n",
            "Iteration 9, loss = 0.19892832\n",
            "Validation score: 0.938462\n",
            "Iteration 10, loss = 0.19389280\n",
            "Validation score: 0.944615\n",
            "Iteration 11, loss = 0.19405260\n",
            "Validation score: 0.932308\n",
            "Iteration 12, loss = 0.19302404\n",
            "Validation score: 0.935385\n",
            "Iteration 13, loss = 0.19393306\n",
            "Validation score: 0.938462\n",
            "Iteration 14, loss = 0.19190986\n",
            "Validation score: 0.935385\n",
            "Iteration 15, loss = 0.18874851\n",
            "Validation score: 0.935385\n",
            "Iteration 16, loss = 0.18859822\n",
            "Validation score: 0.935385\n",
            "Iteration 17, loss = 0.18743932\n",
            "Validation score: 0.941538\n",
            "Iteration 18, loss = 0.18850638\n",
            "Validation score: 0.932308\n",
            "Iteration 19, loss = 0.18598462\n",
            "Validation score: 0.941538\n",
            "Iteration 20, loss = 0.18441582\n",
            "Validation score: 0.932308\n",
            "Iteration 21, loss = 0.18654908\n",
            "Validation score: 0.932308\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 10:14:10,822]\u001b[0m Trial 75 finished with value: 91.8594 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.0017074358268044643, 'batch_size': 64, 'learning_rate_init': 0.007139182491237228, 'max_iter': 271, 'momentum': 0.6315744458917987, 'number_layers': 5, 'hidden_neurons_0': 62, 'hidden_neurons_1': 122, 'hidden_neurons_2': 171, 'hidden_neurons_3': 194, 'hidden_neurons_4': 120}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.35180194\n",
            "Validation score: 0.889231\n",
            "Iteration 2, loss = 0.23922926\n",
            "Validation score: 0.910769\n",
            "Iteration 3, loss = 0.21695571\n",
            "Validation score: 0.920000\n",
            "Iteration 4, loss = 0.21187153\n",
            "Validation score: 0.898462\n",
            "Iteration 5, loss = 0.20671700\n",
            "Validation score: 0.923077\n",
            "Iteration 6, loss = 0.20401813\n",
            "Validation score: 0.923077\n",
            "Iteration 7, loss = 0.19825427\n",
            "Validation score: 0.916923\n",
            "Iteration 8, loss = 0.19860656\n",
            "Validation score: 0.916923\n",
            "Iteration 9, loss = 0.19379388\n",
            "Validation score: 0.926154\n",
            "Iteration 10, loss = 0.19298461\n",
            "Validation score: 0.926154\n",
            "Iteration 11, loss = 0.19592074\n",
            "Validation score: 0.920000\n",
            "Iteration 12, loss = 0.18766460\n",
            "Validation score: 0.880000\n",
            "Iteration 13, loss = 0.18734717\n",
            "Validation score: 0.916923\n",
            "Iteration 14, loss = 0.18649571\n",
            "Validation score: 0.923077\n",
            "Iteration 15, loss = 0.18602217\n",
            "Validation score: 0.923077\n",
            "Iteration 16, loss = 0.18451404\n",
            "Validation score: 0.926154\n",
            "Iteration 17, loss = 0.17974612\n",
            "Validation score: 0.929231\n",
            "Iteration 18, loss = 0.18139574\n",
            "Validation score: 0.920000\n",
            "Iteration 19, loss = 0.18066444\n",
            "Validation score: 0.923077\n",
            "Iteration 20, loss = 0.17864501\n",
            "Validation score: 0.904615\n",
            "Iteration 21, loss = 0.18115612\n",
            "Validation score: 0.926154\n",
            "Iteration 22, loss = 0.17696270\n",
            "Validation score: 0.913846\n",
            "Iteration 23, loss = 0.17538042\n",
            "Validation score: 0.920000\n",
            "Iteration 24, loss = 0.17538799\n",
            "Validation score: 0.929231\n",
            "Iteration 25, loss = 0.17469109\n",
            "Validation score: 0.929231\n",
            "Iteration 26, loss = 0.17463328\n",
            "Validation score: 0.929231\n",
            "Iteration 27, loss = 0.16964881\n",
            "Validation score: 0.913846\n",
            "Iteration 28, loss = 0.17379292\n",
            "Validation score: 0.923077\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 10:14:49,013]\u001b[0m Trial 76 finished with value: 91.5819 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.0009150078123284244, 'batch_size': 16, 'learning_rate_init': 0.00499646586619792, 'max_iter': 253, 'momentum': 0.6038331643943697, 'number_layers': 4, 'hidden_neurons_0': 152, 'hidden_neurons_1': 133, 'hidden_neurons_2': 83, 'hidden_neurons_3': 160}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.32712912\n",
            "Validation score: 0.916923\n",
            "Iteration 2, loss = 0.23824962\n",
            "Validation score: 0.913846\n",
            "Iteration 3, loss = 0.24079221\n",
            "Validation score: 0.926154\n",
            "Iteration 4, loss = 0.21952202\n",
            "Validation score: 0.926154\n",
            "Iteration 5, loss = 0.21505013\n",
            "Validation score: 0.929231\n",
            "Iteration 6, loss = 0.20004344\n",
            "Validation score: 0.920000\n",
            "Iteration 7, loss = 0.19893195\n",
            "Validation score: 0.929231\n",
            "Iteration 8, loss = 0.18732650\n",
            "Validation score: 0.923077\n",
            "Iteration 9, loss = 0.19992739\n",
            "Validation score: 0.916923\n",
            "Iteration 10, loss = 0.20664933\n",
            "Validation score: 0.926154\n",
            "Iteration 11, loss = 0.20196550\n",
            "Validation score: 0.923077\n",
            "Iteration 12, loss = 0.19011173\n",
            "Validation score: 0.910769\n",
            "Iteration 13, loss = 0.18397568\n",
            "Validation score: 0.929231\n",
            "Iteration 14, loss = 0.18912727\n",
            "Validation score: 0.920000\n",
            "Iteration 15, loss = 0.20279694\n",
            "Validation score: 0.935385\n",
            "Iteration 16, loss = 0.18599350\n",
            "Validation score: 0.932308\n",
            "Iteration 17, loss = 0.19267682\n",
            "Validation score: 0.932308\n",
            "Iteration 18, loss = 0.19526145\n",
            "Validation score: 0.929231\n",
            "Iteration 19, loss = 0.18041326\n",
            "Validation score: 0.867692\n",
            "Iteration 20, loss = 0.17810614\n",
            "Validation score: 0.923077\n",
            "Iteration 21, loss = 0.17618804\n",
            "Validation score: 0.926154\n",
            "Iteration 22, loss = 0.18821395\n",
            "Validation score: 0.889231\n",
            "Iteration 23, loss = 0.19305675\n",
            "Validation score: 0.926154\n",
            "Iteration 24, loss = 0.17822379\n",
            "Validation score: 0.929231\n",
            "Iteration 25, loss = 0.17911144\n",
            "Validation score: 0.920000\n",
            "Iteration 26, loss = 0.17659643\n",
            "Validation score: 0.926154\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 10:15:02,951]\u001b[0m Trial 77 finished with value: 91.5819 and parameters: {'actv': 'relu', 'solvers': 'adam', 'alpha': 0.002026937899744316, 'batch_size': 32, 'learning_rate_init': 0.007564493223951998, 'max_iter': 189, 'momentum': 0.5021235200338621, 'number_layers': 5, 'hidden_neurons_0': 71, 'hidden_neurons_1': 112, 'hidden_neurons_2': 200, 'hidden_neurons_3': 176, 'hidden_neurons_4': 139}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 1.29803847\n",
            "Validation score: 0.889231\n",
            "Iteration 2, loss = 0.84230808\n",
            "Validation score: 0.932308\n",
            "Iteration 3, loss = 0.58635663\n",
            "Validation score: 0.923077\n",
            "Iteration 4, loss = 0.44302887\n",
            "Validation score: 0.929231\n",
            "Iteration 5, loss = 0.35441098\n",
            "Validation score: 0.916923\n",
            "Iteration 6, loss = 0.30406056\n",
            "Validation score: 0.901538\n",
            "Iteration 7, loss = 0.27144552\n",
            "Validation score: 0.892308\n",
            "Iteration 8, loss = 0.25261587\n",
            "Validation score: 0.833846\n",
            "Iteration 9, loss = 0.24601067\n",
            "Validation score: 0.926154\n",
            "Iteration 10, loss = 0.23349242\n",
            "Validation score: 0.935385\n",
            "Iteration 11, loss = 0.23415798\n",
            "Validation score: 0.895385\n",
            "Iteration 12, loss = 0.23275135\n",
            "Validation score: 0.907692\n",
            "Iteration 13, loss = 0.23288386\n",
            "Validation score: 0.920000\n",
            "Iteration 14, loss = 0.22485709\n",
            "Validation score: 0.929231\n",
            "Iteration 15, loss = 0.23024284\n",
            "Validation score: 0.898462\n",
            "Iteration 16, loss = 0.22845234\n",
            "Validation score: 0.935385\n",
            "Iteration 17, loss = 0.22604366\n",
            "Validation score: 0.901538\n",
            "Iteration 18, loss = 0.22671222\n",
            "Validation score: 0.923077\n",
            "Iteration 19, loss = 0.23224567\n",
            "Validation score: 0.932308\n",
            "Iteration 20, loss = 0.22549307\n",
            "Validation score: 0.932308\n",
            "Iteration 21, loss = 0.22963080\n",
            "Validation score: 0.926154\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 10:19:56,077]\u001b[0m Trial 78 finished with value: 91.9519 and parameters: {'actv': 'tanh', 'solvers': 'sgd', 'alpha': 0.005150046254263052, 'batch_size': 1, 'learning_rate_init': 0.005921055315060713, 'max_iter': 211, 'momentum': 0.6653235805064386, 'number_layers': 3, 'hidden_neurons_0': 133, 'hidden_neurons_1': 168, 'hidden_neurons_2': 94}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.40750917\n",
            "Validation score: 0.898462\n",
            "Iteration 2, loss = 0.25207588\n",
            "Validation score: 0.895385\n",
            "Iteration 3, loss = 0.21968113\n",
            "Validation score: 0.849231\n",
            "Iteration 4, loss = 0.21355501\n",
            "Validation score: 0.935385\n",
            "Iteration 5, loss = 0.21001214\n",
            "Validation score: 0.929231\n",
            "Iteration 6, loss = 0.20553922\n",
            "Validation score: 0.938462\n",
            "Iteration 7, loss = 0.20221329\n",
            "Validation score: 0.910769\n",
            "Iteration 8, loss = 0.20323880\n",
            "Validation score: 0.913846\n",
            "Iteration 9, loss = 0.19751173\n",
            "Validation score: 0.920000\n",
            "Iteration 10, loss = 0.19818186\n",
            "Validation score: 0.929231\n",
            "Iteration 11, loss = 0.19821647\n",
            "Validation score: 0.932308\n",
            "Iteration 12, loss = 0.19543033\n",
            "Validation score: 0.932308\n",
            "Iteration 13, loss = 0.19316310\n",
            "Validation score: 0.941538\n",
            "Iteration 14, loss = 0.19027805\n",
            "Validation score: 0.932308\n",
            "Iteration 15, loss = 0.18954698\n",
            "Validation score: 0.941538\n",
            "Iteration 16, loss = 0.18971765\n",
            "Validation score: 0.910769\n",
            "Iteration 17, loss = 0.18985503\n",
            "Validation score: 0.910769\n",
            "Iteration 18, loss = 0.18628397\n",
            "Validation score: 0.916923\n",
            "Iteration 19, loss = 0.18622874\n",
            "Validation score: 0.926154\n",
            "Iteration 20, loss = 0.18925978\n",
            "Validation score: 0.941538\n",
            "Iteration 21, loss = 0.18317021\n",
            "Validation score: 0.920000\n",
            "Iteration 22, loss = 0.18397421\n",
            "Validation score: 0.916923\n",
            "Iteration 23, loss = 0.18907812\n",
            "Validation score: 0.920000\n",
            "Iteration 24, loss = 0.18131934\n",
            "Validation score: 0.864615\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 10:20:05,010]\u001b[0m Trial 79 finished with value: 91.6744 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.0014206454917860978, 'batch_size': 32, 'learning_rate_init': 0.0065110667927066376, 'max_iter': 227, 'momentum': 0.5703477279649131, 'number_layers': 5, 'hidden_neurons_0': 55, 'hidden_neurons_1': 128, 'hidden_neurons_2': 182, 'hidden_neurons_3': 187, 'hidden_neurons_4': 128}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.44069454\n",
            "Validation score: 0.907692\n",
            "Iteration 2, loss = 0.38868615\n",
            "Validation score: 0.941538\n",
            "Iteration 3, loss = 0.37759917\n",
            "Validation score: 0.932308\n",
            "Iteration 4, loss = 0.37104085\n",
            "Validation score: 0.932308\n",
            "Iteration 5, loss = 0.36526919\n",
            "Validation score: 0.938462\n",
            "Iteration 6, loss = 0.36058580\n",
            "Validation score: 0.941538\n",
            "Iteration 7, loss = 0.35441964\n",
            "Validation score: 0.941538\n",
            "Iteration 8, loss = 0.34829461\n",
            "Validation score: 0.938462\n",
            "Iteration 9, loss = 0.34127008\n",
            "Validation score: 0.941538\n",
            "Iteration 10, loss = 0.33969431\n",
            "Validation score: 0.947692\n",
            "Iteration 11, loss = 0.33444651\n",
            "Validation score: 0.950769\n",
            "Iteration 12, loss = 0.33095676\n",
            "Validation score: 0.941538\n",
            "Iteration 13, loss = 0.33055312\n",
            "Validation score: 0.950769\n",
            "Iteration 14, loss = 0.32399560\n",
            "Validation score: 0.935385\n",
            "Iteration 15, loss = 0.32174187\n",
            "Validation score: 0.932308\n",
            "Iteration 16, loss = 0.31733192\n",
            "Validation score: 0.932308\n",
            "Iteration 17, loss = 0.31358196\n",
            "Validation score: 0.935385\n",
            "Iteration 18, loss = 0.30869795\n",
            "Validation score: 0.926154\n",
            "Iteration 19, loss = 0.30563708\n",
            "Validation score: 0.947692\n",
            "Iteration 20, loss = 0.30238057\n",
            "Validation score: 0.929231\n",
            "Iteration 21, loss = 0.30466454\n",
            "Validation score: 0.923077\n",
            "Iteration 22, loss = 0.30052305\n",
            "Validation score: 0.932308\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 10:25:23,950]\u001b[0m Trial 80 finished with value: 91.7669 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.0005341570516815678, 'batch_size': 1, 'learning_rate_init': 0.005258223432379049, 'max_iter': 169, 'momentum': 0.04494257283203884, 'number_layers': 4, 'hidden_neurons_0': 116, 'hidden_neurons_1': 176, 'hidden_neurons_2': 162, 'hidden_neurons_3': 139}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.43880282\n",
            "Validation score: 0.901538\n",
            "Iteration 2, loss = 0.27284040\n",
            "Validation score: 0.926154\n",
            "Iteration 3, loss = 0.24705368\n",
            "Validation score: 0.883077\n",
            "Iteration 4, loss = 0.24186211\n",
            "Validation score: 0.920000\n",
            "Iteration 5, loss = 0.23633761\n",
            "Validation score: 0.916923\n",
            "Iteration 6, loss = 0.23220467\n",
            "Validation score: 0.923077\n",
            "Iteration 7, loss = 0.22812017\n",
            "Validation score: 0.907692\n",
            "Iteration 8, loss = 0.23060902\n",
            "Validation score: 0.935385\n",
            "Iteration 9, loss = 0.22766831\n",
            "Validation score: 0.907692\n",
            "Iteration 10, loss = 0.22612851\n",
            "Validation score: 0.910769\n",
            "Iteration 11, loss = 0.22211585\n",
            "Validation score: 0.929231\n",
            "Iteration 12, loss = 0.22021016\n",
            "Validation score: 0.944615\n",
            "Iteration 13, loss = 0.21974413\n",
            "Validation score: 0.938462\n",
            "Iteration 14, loss = 0.21739775\n",
            "Validation score: 0.938462\n",
            "Iteration 15, loss = 0.21672060\n",
            "Validation score: 0.926154\n",
            "Iteration 16, loss = 0.21699139\n",
            "Validation score: 0.938462\n",
            "Iteration 17, loss = 0.21522165\n",
            "Validation score: 0.916923\n",
            "Iteration 18, loss = 0.21803994\n",
            "Validation score: 0.929231\n",
            "Iteration 19, loss = 0.21141961\n",
            "Validation score: 0.929231\n",
            "Iteration 20, loss = 0.21230769\n",
            "Validation score: 0.935385\n",
            "Iteration 21, loss = 0.20810954\n",
            "Validation score: 0.910769\n",
            "Iteration 22, loss = 0.21181632\n",
            "Validation score: 0.932308\n",
            "Iteration 23, loss = 0.20796466\n",
            "Validation score: 0.929231\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 10:25:38,928]\u001b[0m Trial 81 finished with value: 91.8594 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.003542108649249975, 'batch_size': 32, 'learning_rate_init': 0.006224344039605627, 'max_iter': 192, 'momentum': 0.7311371158881564, 'number_layers': 5, 'hidden_neurons_0': 128, 'hidden_neurons_1': 184, 'hidden_neurons_2': 169, 'hidden_neurons_3': 148, 'hidden_neurons_4': 100}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.34651869\n",
            "Validation score: 0.910769\n",
            "Iteration 2, loss = 0.24289724\n",
            "Validation score: 0.895385\n",
            "Iteration 3, loss = 0.22036941\n",
            "Validation score: 0.916923\n",
            "Iteration 4, loss = 0.21191473\n",
            "Validation score: 0.904615\n",
            "Iteration 5, loss = 0.21034911\n",
            "Validation score: 0.941538\n",
            "Iteration 6, loss = 0.20101582\n",
            "Validation score: 0.929231\n",
            "Iteration 7, loss = 0.20102963\n",
            "Validation score: 0.941538\n",
            "Iteration 8, loss = 0.19713152\n",
            "Validation score: 0.867692\n",
            "Iteration 9, loss = 0.19826805\n",
            "Validation score: 0.941538\n",
            "Iteration 10, loss = 0.19412700\n",
            "Validation score: 0.929231\n",
            "Iteration 11, loss = 0.19317667\n",
            "Validation score: 0.895385\n",
            "Iteration 12, loss = 0.18830764\n",
            "Validation score: 0.904615\n",
            "Iteration 13, loss = 0.19359188\n",
            "Validation score: 0.944615\n",
            "Iteration 14, loss = 0.19090536\n",
            "Validation score: 0.938462\n",
            "Iteration 15, loss = 0.18743595\n",
            "Validation score: 0.907692\n",
            "Iteration 16, loss = 0.18832857\n",
            "Validation score: 0.935385\n",
            "Iteration 17, loss = 0.18231744\n",
            "Validation score: 0.920000\n",
            "Iteration 18, loss = 0.18460919\n",
            "Validation score: 0.926154\n",
            "Iteration 19, loss = 0.18579924\n",
            "Validation score: 0.932308\n",
            "Iteration 20, loss = 0.18540896\n",
            "Validation score: 0.929231\n",
            "Iteration 21, loss = 0.18268829\n",
            "Validation score: 0.935385\n",
            "Iteration 22, loss = 0.17991972\n",
            "Validation score: 0.920000\n",
            "Iteration 23, loss = 0.18490860\n",
            "Validation score: 0.932308\n",
            "Iteration 24, loss = 0.17947915\n",
            "Validation score: 0.938462\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 10:25:45,314]\u001b[0m Trial 82 finished with value: 91.3969 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.0028387488506479534, 'batch_size': 32, 'learning_rate_init': 0.006449014206230199, 'max_iter': 221, 'momentum': 0.6800543147156407, 'number_layers': 3, 'hidden_neurons_0': 51, 'hidden_neurons_1': 178, 'hidden_neurons_2': 65}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.38217497\n",
            "Validation score: 0.901538\n",
            "Iteration 2, loss = 0.23692333\n",
            "Validation score: 0.920000\n",
            "Iteration 3, loss = 0.22299100\n",
            "Validation score: 0.916923\n",
            "Iteration 4, loss = 0.22265369\n",
            "Validation score: 0.883077\n",
            "Iteration 5, loss = 0.21912102\n",
            "Validation score: 0.913846\n",
            "Iteration 6, loss = 0.21030004\n",
            "Validation score: 0.901538\n",
            "Iteration 7, loss = 0.20926383\n",
            "Validation score: 0.907692\n",
            "Iteration 8, loss = 0.20743060\n",
            "Validation score: 0.858462\n",
            "Iteration 9, loss = 0.21018458\n",
            "Validation score: 0.892308\n",
            "Iteration 10, loss = 0.20625167\n",
            "Validation score: 0.876923\n",
            "Iteration 11, loss = 0.20418994\n",
            "Validation score: 0.916923\n",
            "Iteration 12, loss = 0.20063863\n",
            "Validation score: 0.910769\n",
            "Iteration 13, loss = 0.20130538\n",
            "Validation score: 0.907692\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 10:25:51,648]\u001b[0m Trial 83 finished with value: 91.9519 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.002467816090469628, 'batch_size': 32, 'learning_rate_init': 0.0069189522441936435, 'max_iter': 236, 'momentum': 0.7672625693720039, 'number_layers': 5, 'hidden_neurons_0': 61, 'hidden_neurons_1': 200, 'hidden_neurons_2': 174, 'hidden_neurons_3': 162, 'hidden_neurons_4': 123}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.48933708\n",
            "Validation score: 0.864615\n",
            "Iteration 2, loss = 0.30497129\n",
            "Validation score: 0.870769\n",
            "Iteration 3, loss = 0.25010574\n",
            "Validation score: 0.889231\n",
            "Iteration 4, loss = 0.22330636\n",
            "Validation score: 0.901538\n",
            "Iteration 5, loss = 0.20993669\n",
            "Validation score: 0.883077\n",
            "Iteration 6, loss = 0.20484954\n",
            "Validation score: 0.886154\n",
            "Iteration 7, loss = 0.19985254\n",
            "Validation score: 0.913846\n",
            "Iteration 8, loss = 0.19844661\n",
            "Validation score: 0.867692\n",
            "Iteration 9, loss = 0.19617713\n",
            "Validation score: 0.898462\n",
            "Iteration 10, loss = 0.19379215\n",
            "Validation score: 0.889231\n",
            "Iteration 11, loss = 0.19156098\n",
            "Validation score: 0.916923\n",
            "Iteration 12, loss = 0.18759838\n",
            "Validation score: 0.913846\n",
            "Iteration 13, loss = 0.18967563\n",
            "Validation score: 0.910769\n",
            "Iteration 14, loss = 0.18775235\n",
            "Validation score: 0.910769\n",
            "Iteration 15, loss = 0.18674472\n",
            "Validation score: 0.886154\n",
            "Iteration 16, loss = 0.18682653\n",
            "Validation score: 0.926154\n",
            "Iteration 17, loss = 0.18293610\n",
            "Validation score: 0.880000\n",
            "Iteration 18, loss = 0.18598522\n",
            "Validation score: 0.913846\n",
            "Iteration 19, loss = 0.18306471\n",
            "Validation score: 0.809231\n",
            "Iteration 20, loss = 0.18649712\n",
            "Validation score: 0.916923\n",
            "Iteration 21, loss = 0.18334600\n",
            "Validation score: 0.910769\n",
            "Iteration 22, loss = 0.18209938\n",
            "Validation score: 0.913846\n",
            "Iteration 23, loss = 0.17920469\n",
            "Validation score: 0.916923\n",
            "Iteration 24, loss = 0.18070854\n",
            "Validation score: 0.926154\n",
            "Iteration 25, loss = 0.17817300\n",
            "Validation score: 0.883077\n",
            "Iteration 26, loss = 0.17911289\n",
            "Validation score: 0.920000\n",
            "Iteration 27, loss = 0.18032093\n",
            "Validation score: 0.926154\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 10:26:02,233]\u001b[0m Trial 84 finished with value: 91.6744 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.0011467680554582812, 'batch_size': 32, 'learning_rate_init': 0.005587004631906773, 'max_iter': 235, 'momentum': 0.5476870287415385, 'number_layers': 5, 'hidden_neurons_0': 54, 'hidden_neurons_1': 137, 'hidden_neurons_2': 185, 'hidden_neurons_3': 170, 'hidden_neurons_4': 140}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.67166314\n",
            "Validation score: 0.910769\n",
            "Iteration 2, loss = 0.54124243\n",
            "Validation score: 0.830769\n",
            "Iteration 3, loss = 0.47158726\n",
            "Validation score: 0.920000\n",
            "Iteration 4, loss = 0.41763192\n",
            "Validation score: 0.926154\n",
            "Iteration 5, loss = 0.36863345\n",
            "Validation score: 0.938462\n",
            "Iteration 6, loss = 0.34159681\n",
            "Validation score: 0.935385\n",
            "Iteration 7, loss = 0.31405842\n",
            "Validation score: 0.944615\n",
            "Iteration 8, loss = 0.29709836\n",
            "Validation score: 0.926154\n",
            "Iteration 9, loss = 0.28300367\n",
            "Validation score: 0.941538\n",
            "Iteration 10, loss = 0.27057900\n",
            "Validation score: 0.932308\n",
            "Iteration 11, loss = 0.25593365\n",
            "Validation score: 0.932308\n",
            "Iteration 12, loss = 0.25239702\n",
            "Validation score: 0.944615\n",
            "Iteration 13, loss = 0.24810392\n",
            "Validation score: 0.938462\n",
            "Iteration 14, loss = 0.24287834\n",
            "Validation score: 0.947692\n",
            "Iteration 15, loss = 0.23760728\n",
            "Validation score: 0.926154\n",
            "Iteration 16, loss = 0.24069659\n",
            "Validation score: 0.947692\n",
            "Iteration 17, loss = 0.23657004\n",
            "Validation score: 0.904615\n",
            "Iteration 18, loss = 0.23774332\n",
            "Validation score: 0.861538\n",
            "Iteration 19, loss = 0.23204238\n",
            "Validation score: 0.941538\n",
            "Iteration 20, loss = 0.23189467\n",
            "Validation score: 0.935385\n",
            "Iteration 21, loss = 0.22645822\n",
            "Validation score: 0.904615\n",
            "Iteration 22, loss = 0.22686653\n",
            "Validation score: 0.932308\n",
            "Iteration 23, loss = 0.22875117\n",
            "Validation score: 0.904615\n",
            "Iteration 24, loss = 0.23406975\n",
            "Validation score: 0.935385\n",
            "Iteration 25, loss = 0.22670090\n",
            "Validation score: 0.947692\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 10:29:34,262]\u001b[0m Trial 85 finished with value: 91.8594 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.001736515935471602, 'batch_size': 1, 'learning_rate_init': 0.007507525623598818, 'max_iter': 378, 'momentum': 0.6956224462447462, 'number_layers': 4, 'hidden_neurons_0': 71, 'hidden_neurons_1': 159, 'hidden_neurons_2': 140, 'hidden_neurons_3': 110}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.56462575\n",
            "Validation score: 0.769231\n",
            "Iteration 2, loss = 0.56025279\n",
            "Validation score: 0.769231\n",
            "Iteration 3, loss = 0.55992092\n",
            "Validation score: 0.769231\n",
            "Iteration 4, loss = 0.56016658\n",
            "Validation score: 0.769231\n",
            "Iteration 5, loss = 0.55969102\n",
            "Validation score: 0.769231\n",
            "Iteration 6, loss = 0.56032519\n",
            "Validation score: 0.769231\n",
            "Iteration 7, loss = 0.56029244\n",
            "Validation score: 0.769231\n",
            "Iteration 8, loss = 0.56029863\n",
            "Validation score: 0.769231\n",
            "Iteration 9, loss = 0.55998516\n",
            "Validation score: 0.769231\n",
            "Iteration 10, loss = 0.55994335\n",
            "Validation score: 0.769231\n",
            "Iteration 11, loss = 0.56016945\n",
            "Validation score: 0.769231\n",
            "Iteration 12, loss = 0.55948738\n",
            "Validation score: 0.769231\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 10:29:51,062]\u001b[0m Trial 86 finished with value: 76.7808 and parameters: {'actv': 'logistic', 'solvers': 'sgd', 'alpha': 0.0022889593691758792, 'batch_size': 16, 'learning_rate_init': 0.006077027518927515, 'max_iter': 253, 'momentum': 0.6300237822406556, 'number_layers': 5, 'hidden_neurons_0': 164, 'hidden_neurons_1': 107, 'hidden_neurons_2': 58, 'hidden_neurons_3': 157, 'hidden_neurons_4': 77}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.30805196\n",
            "Validation score: 0.929231\n",
            "Iteration 2, loss = 0.23107569\n",
            "Validation score: 0.947692\n",
            "Iteration 3, loss = 0.21773500\n",
            "Validation score: 0.938462\n",
            "Iteration 4, loss = 0.21237399\n",
            "Validation score: 0.950769\n",
            "Iteration 5, loss = 0.21299723\n",
            "Validation score: 0.907692\n",
            "Iteration 6, loss = 0.20959969\n",
            "Validation score: 0.938462\n",
            "Iteration 7, loss = 0.20353274\n",
            "Validation score: 0.956923\n",
            "Iteration 8, loss = 0.20455894\n",
            "Validation score: 0.956923\n",
            "Iteration 9, loss = 0.20112068\n",
            "Validation score: 0.944615\n",
            "Iteration 10, loss = 0.20175872\n",
            "Validation score: 0.950769\n",
            "Iteration 11, loss = 0.19927542\n",
            "Validation score: 0.941538\n",
            "Iteration 12, loss = 0.19486930\n",
            "Validation score: 0.920000\n",
            "Iteration 13, loss = 0.19730556\n",
            "Validation score: 0.923077\n",
            "Iteration 14, loss = 0.19284011\n",
            "Validation score: 0.926154\n",
            "Iteration 15, loss = 0.19106638\n",
            "Validation score: 0.956923\n",
            "Iteration 16, loss = 0.18830075\n",
            "Validation score: 0.941538\n",
            "Iteration 17, loss = 0.18801750\n",
            "Validation score: 0.913846\n",
            "Iteration 18, loss = 0.18818865\n",
            "Validation score: 0.938462\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 10:29:58,788]\u001b[0m Trial 87 finished with value: 91.5819 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.0029485596069706316, 'batch_size': 32, 'learning_rate_init': 0.006645196469574867, 'max_iter': 198, 'momentum': 0.8300726959290053, 'number_layers': 3, 'hidden_neurons_0': 103, 'hidden_neurons_1': 194, 'hidden_neurons_2': 97}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.53456028\n",
            "Validation score: 0.892308\n",
            "Iteration 2, loss = 0.26408941\n",
            "Validation score: 0.926154\n",
            "Iteration 3, loss = 0.22377988\n",
            "Validation score: 0.892308\n",
            "Iteration 4, loss = 0.22460442\n",
            "Validation score: 0.929231\n",
            "Iteration 5, loss = 0.20751474\n",
            "Validation score: 0.929231\n",
            "Iteration 6, loss = 0.20468982\n",
            "Validation score: 0.923077\n",
            "Iteration 7, loss = 0.19619566\n",
            "Validation score: 0.923077\n",
            "Iteration 8, loss = 0.21555089\n",
            "Validation score: 0.907692\n",
            "Iteration 9, loss = 0.22196352\n",
            "Validation score: 0.923077\n",
            "Iteration 10, loss = 0.20840115\n",
            "Validation score: 0.932308\n",
            "Iteration 11, loss = 0.18560705\n",
            "Validation score: 0.935385\n",
            "Iteration 12, loss = 0.18522725\n",
            "Validation score: 0.926154\n",
            "Iteration 13, loss = 0.20984050\n",
            "Validation score: 0.932308\n",
            "Iteration 14, loss = 0.19435783\n",
            "Validation score: 0.904615\n",
            "Iteration 15, loss = 0.19100085\n",
            "Validation score: 0.916923\n",
            "Iteration 16, loss = 0.19670250\n",
            "Validation score: 0.920000\n",
            "Iteration 17, loss = 0.18603265\n",
            "Validation score: 0.932308\n",
            "Iteration 18, loss = 0.17446265\n",
            "Validation score: 0.932308\n",
            "Iteration 19, loss = 0.17341817\n",
            "Validation score: 0.935385\n",
            "Iteration 20, loss = 0.17140405\n",
            "Validation score: 0.920000\n",
            "Iteration 21, loss = 0.19166247\n",
            "Validation score: 0.898462\n",
            "Iteration 22, loss = 0.17460965\n",
            "Validation score: 0.929231\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 10:30:05,755]\u001b[0m Trial 88 finished with value: 92.0444 and parameters: {'actv': 'relu', 'solvers': 'adam', 'alpha': 0.009635158203828039, 'batch_size': 128, 'learning_rate_init': 0.008149962123341068, 'max_iter': 179, 'momentum': 0.7016117842971291, 'number_layers': 5, 'hidden_neurons_0': 139, 'hidden_neurons_1': 182, 'hidden_neurons_2': 176, 'hidden_neurons_3': 80, 'hidden_neurons_4': 196}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.50787146\n",
            "Validation score: 0.775385\n",
            "Iteration 2, loss = 0.34027225\n",
            "Validation score: 0.904615\n",
            "Iteration 3, loss = 0.28296075\n",
            "Validation score: 0.886154\n",
            "Iteration 4, loss = 0.25390169\n",
            "Validation score: 0.920000\n",
            "Iteration 5, loss = 0.23284682\n",
            "Validation score: 0.923077\n",
            "Iteration 6, loss = 0.22255889\n",
            "Validation score: 0.880000\n",
            "Iteration 7, loss = 0.21521969\n",
            "Validation score: 0.935385\n",
            "Iteration 8, loss = 0.21396943\n",
            "Validation score: 0.932308\n",
            "Iteration 9, loss = 0.21078820\n",
            "Validation score: 0.920000\n",
            "Iteration 10, loss = 0.20997988\n",
            "Validation score: 0.929231\n",
            "Iteration 11, loss = 0.20714490\n",
            "Validation score: 0.901538\n",
            "Iteration 12, loss = 0.20656689\n",
            "Validation score: 0.873846\n",
            "Iteration 13, loss = 0.20414329\n",
            "Validation score: 0.929231\n",
            "Iteration 14, loss = 0.20401902\n",
            "Validation score: 0.907692\n",
            "Iteration 15, loss = 0.20234893\n",
            "Validation score: 0.910769\n",
            "Iteration 16, loss = 0.20226374\n",
            "Validation score: 0.929231\n",
            "Iteration 17, loss = 0.19607806\n",
            "Validation score: 0.929231\n",
            "Iteration 18, loss = 0.19766863\n",
            "Validation score: 0.920000\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 10:30:13,308]\u001b[0m Trial 89 finished with value: 90.9343 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.002603495116735719, 'batch_size': 32, 'learning_rate_init': 0.003320516972087063, 'max_iter': 322, 'momentum': 0.5795846152014179, 'number_layers': 5, 'hidden_neurons_0': 64, 'hidden_neurons_1': 120, 'hidden_neurons_2': 191, 'hidden_neurons_3': 167, 'hidden_neurons_4': 158}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.32985230\n",
            "Validation score: 0.892308\n",
            "Iteration 2, loss = 0.24590462\n",
            "Validation score: 0.932308\n",
            "Iteration 3, loss = 0.23077536\n",
            "Validation score: 0.852308\n",
            "Iteration 4, loss = 0.22197334\n",
            "Validation score: 0.920000\n",
            "Iteration 5, loss = 0.21578513\n",
            "Validation score: 0.923077\n",
            "Iteration 6, loss = 0.21714783\n",
            "Validation score: 0.901538\n",
            "Iteration 7, loss = 0.21423766\n",
            "Validation score: 0.920000\n",
            "Iteration 8, loss = 0.20986555\n",
            "Validation score: 0.923077\n",
            "Iteration 9, loss = 0.20793403\n",
            "Validation score: 0.932308\n",
            "Iteration 10, loss = 0.20667713\n",
            "Validation score: 0.923077\n",
            "Iteration 11, loss = 0.20343836\n",
            "Validation score: 0.883077\n",
            "Iteration 12, loss = 0.20250265\n",
            "Validation score: 0.901538\n",
            "Iteration 13, loss = 0.20034320\n",
            "Validation score: 0.920000\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 10:30:29,726]\u001b[0m Trial 90 finished with value: 91.5819 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.0018962120467957504, 'batch_size': 16, 'learning_rate_init': 0.0057150489421570515, 'max_iter': 216, 'momentum': 0.45758073321121095, 'number_layers': 3, 'hidden_neurons_0': 160, 'hidden_neurons_1': 114, 'hidden_neurons_2': 55}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.41681573\n",
            "Validation score: 0.898462\n",
            "Iteration 2, loss = 0.27057528\n",
            "Validation score: 0.920000\n",
            "Iteration 3, loss = 0.23285114\n",
            "Validation score: 0.923077\n",
            "Iteration 4, loss = 0.21903869\n",
            "Validation score: 0.916923\n",
            "Iteration 5, loss = 0.20998447\n",
            "Validation score: 0.923077\n",
            "Iteration 6, loss = 0.20241665\n",
            "Validation score: 0.926154\n",
            "Iteration 7, loss = 0.20036062\n",
            "Validation score: 0.926154\n",
            "Iteration 8, loss = 0.20330912\n",
            "Validation score: 0.901538\n",
            "Iteration 9, loss = 0.19471196\n",
            "Validation score: 0.923077\n",
            "Iteration 10, loss = 0.19599985\n",
            "Validation score: 0.926154\n",
            "Iteration 11, loss = 0.19002639\n",
            "Validation score: 0.932308\n",
            "Iteration 12, loss = 0.19217908\n",
            "Validation score: 0.929231\n",
            "Iteration 13, loss = 0.18937106\n",
            "Validation score: 0.910769\n",
            "Iteration 14, loss = 0.19168658\n",
            "Validation score: 0.867692\n",
            "Iteration 15, loss = 0.19214040\n",
            "Validation score: 0.932308\n",
            "Iteration 16, loss = 0.19457630\n",
            "Validation score: 0.916923\n",
            "Iteration 17, loss = 0.18403772\n",
            "Validation score: 0.892308\n",
            "Iteration 18, loss = 0.18625093\n",
            "Validation score: 0.920000\n",
            "Iteration 19, loss = 0.18120108\n",
            "Validation score: 0.929231\n",
            "Iteration 20, loss = 0.18304568\n",
            "Validation score: 0.923077\n",
            "Iteration 21, loss = 0.18619226\n",
            "Validation score: 0.932308\n",
            "Iteration 22, loss = 0.18281014\n",
            "Validation score: 0.920000\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 10:30:38,217]\u001b[0m Trial 91 finished with value: 92.0444 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.0031810765334798477, 'batch_size': 64, 'learning_rate_init': 0.007228636867644256, 'max_iter': 285, 'momentum': 0.6561150473013948, 'number_layers': 4, 'hidden_neurons_0': 143, 'hidden_neurons_1': 189, 'hidden_neurons_2': 153, 'hidden_neurons_3': 84}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.40940583\n",
            "Validation score: 0.895385\n",
            "Iteration 2, loss = 0.26821695\n",
            "Validation score: 0.880000\n",
            "Iteration 3, loss = 0.23296808\n",
            "Validation score: 0.916923\n",
            "Iteration 4, loss = 0.21297264\n",
            "Validation score: 0.929231\n",
            "Iteration 5, loss = 0.20992897\n",
            "Validation score: 0.904615\n",
            "Iteration 6, loss = 0.20315380\n",
            "Validation score: 0.916923\n",
            "Iteration 7, loss = 0.20066190\n",
            "Validation score: 0.932308\n",
            "Iteration 8, loss = 0.19748870\n",
            "Validation score: 0.901538\n",
            "Iteration 9, loss = 0.19648829\n",
            "Validation score: 0.910769\n",
            "Iteration 10, loss = 0.19458248\n",
            "Validation score: 0.932308\n",
            "Iteration 11, loss = 0.19078964\n",
            "Validation score: 0.926154\n",
            "Iteration 12, loss = 0.19320747\n",
            "Validation score: 0.913846\n",
            "Iteration 13, loss = 0.18809020\n",
            "Validation score: 0.932308\n",
            "Iteration 14, loss = 0.18687969\n",
            "Validation score: 0.926154\n",
            "Iteration 15, loss = 0.18810555\n",
            "Validation score: 0.938462\n",
            "Iteration 16, loss = 0.18657496\n",
            "Validation score: 0.923077\n",
            "Iteration 17, loss = 0.18730120\n",
            "Validation score: 0.920000\n",
            "Iteration 18, loss = 0.18633893\n",
            "Validation score: 0.898462\n",
            "Iteration 19, loss = 0.18765044\n",
            "Validation score: 0.910769\n",
            "Iteration 20, loss = 0.18139459\n",
            "Validation score: 0.913846\n",
            "Iteration 21, loss = 0.18434128\n",
            "Validation score: 0.944615\n",
            "Iteration 22, loss = 0.18302004\n",
            "Validation score: 0.920000\n",
            "Iteration 23, loss = 0.18054085\n",
            "Validation score: 0.926154\n",
            "Iteration 24, loss = 0.17807116\n",
            "Validation score: 0.929231\n",
            "Iteration 25, loss = 0.17434216\n",
            "Validation score: 0.904615\n",
            "Iteration 26, loss = 0.17960373\n",
            "Validation score: 0.941538\n",
            "Iteration 27, loss = 0.17654427\n",
            "Validation score: 0.944615\n",
            "Iteration 28, loss = 0.17658350\n",
            "Validation score: 0.932308\n",
            "Iteration 29, loss = 0.17709641\n",
            "Validation score: 0.947692\n",
            "Iteration 30, loss = 0.17260726\n",
            "Validation score: 0.935385\n",
            "Iteration 31, loss = 0.17059904\n",
            "Validation score: 0.935385\n",
            "Iteration 32, loss = 0.17421584\n",
            "Validation score: 0.923077\n",
            "Iteration 33, loss = 0.17216909\n",
            "Validation score: 0.935385\n",
            "Iteration 34, loss = 0.17121260\n",
            "Validation score: 0.916923\n",
            "Iteration 35, loss = 0.17014388\n",
            "Validation score: 0.944615\n",
            "Iteration 36, loss = 0.16857743\n",
            "Validation score: 0.923077\n",
            "Iteration 37, loss = 0.16854454\n",
            "Validation score: 0.923077\n",
            "Iteration 38, loss = 0.17018834\n",
            "Validation score: 0.923077\n",
            "Iteration 39, loss = 0.16497961\n",
            "Validation score: 0.916923\n",
            "Iteration 40, loss = 0.17106429\n",
            "Validation score: 0.923077\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 10:30:45,797]\u001b[0m Trial 92 finished with value: 91.9519 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.0033025764881101224, 'batch_size': 64, 'learning_rate_init': 0.007145250635725059, 'max_iter': 283, 'momentum': 0.7329166575450357, 'number_layers': 4, 'hidden_neurons_0': 57, 'hidden_neurons_1': 186, 'hidden_neurons_2': 165, 'hidden_neurons_3': 62}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.43175991\n",
            "Validation score: 0.883077\n",
            "Iteration 2, loss = 0.27566777\n",
            "Validation score: 0.907692\n",
            "Iteration 3, loss = 0.24205609\n",
            "Validation score: 0.920000\n",
            "Iteration 4, loss = 0.22331783\n",
            "Validation score: 0.923077\n",
            "Iteration 5, loss = 0.21865037\n",
            "Validation score: 0.910769\n",
            "Iteration 6, loss = 0.20771303\n",
            "Validation score: 0.932308\n",
            "Iteration 7, loss = 0.20712167\n",
            "Validation score: 0.935385\n",
            "Iteration 8, loss = 0.20150481\n",
            "Validation score: 0.935385\n",
            "Iteration 9, loss = 0.20313150\n",
            "Validation score: 0.929231\n",
            "Iteration 10, loss = 0.20053111\n",
            "Validation score: 0.932308\n",
            "Iteration 11, loss = 0.20091974\n",
            "Validation score: 0.916923\n",
            "Iteration 12, loss = 0.19573520\n",
            "Validation score: 0.929231\n",
            "Iteration 13, loss = 0.19700610\n",
            "Validation score: 0.929231\n",
            "Iteration 14, loss = 0.19796020\n",
            "Validation score: 0.926154\n",
            "Iteration 15, loss = 0.19175044\n",
            "Validation score: 0.923077\n",
            "Iteration 16, loss = 0.19301074\n",
            "Validation score: 0.916923\n",
            "Iteration 17, loss = 0.19627040\n",
            "Validation score: 0.929231\n",
            "Iteration 18, loss = 0.19039511\n",
            "Validation score: 0.929231\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 10:30:51,581]\u001b[0m Trial 93 finished with value: 92.1369 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.003723176101176948, 'batch_size': 64, 'learning_rate_init': 0.006849468257024891, 'max_iter': 266, 'momentum': 0.6528441687189137, 'number_layers': 4, 'hidden_neurons_0': 118, 'hidden_neurons_1': 192, 'hidden_neurons_2': 153, 'hidden_neurons_3': 76}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.90360462\n",
            "Validation score: 0.873846\n",
            "Iteration 2, loss = 0.27582357\n",
            "Validation score: 0.926154\n",
            "Iteration 3, loss = 0.25249708\n",
            "Validation score: 0.916923\n",
            "Iteration 4, loss = 0.21899901\n",
            "Validation score: 0.923077\n",
            "Iteration 5, loss = 0.21468833\n",
            "Validation score: 0.923077\n",
            "Iteration 6, loss = 0.21137190\n",
            "Validation score: 0.932308\n",
            "Iteration 7, loss = 0.20275163\n",
            "Validation score: 0.935385\n",
            "Iteration 8, loss = 0.21454877\n",
            "Validation score: 0.913846\n",
            "Iteration 9, loss = 0.21508927\n",
            "Validation score: 0.923077\n",
            "Iteration 10, loss = 0.19823457\n",
            "Validation score: 0.929231\n",
            "Iteration 11, loss = 0.19277603\n",
            "Validation score: 0.923077\n",
            "Iteration 12, loss = 0.20142229\n",
            "Validation score: 0.929231\n",
            "Iteration 13, loss = 0.20131316\n",
            "Validation score: 0.938462\n",
            "Iteration 14, loss = 0.18867400\n",
            "Validation score: 0.926154\n",
            "Iteration 15, loss = 0.18533538\n",
            "Validation score: 0.929231\n",
            "Iteration 16, loss = 0.18504087\n",
            "Validation score: 0.926154\n",
            "Iteration 17, loss = 0.20314810\n",
            "Validation score: 0.923077\n",
            "Iteration 18, loss = 0.21500832\n",
            "Validation score: 0.920000\n",
            "Iteration 19, loss = 0.19600661\n",
            "Validation score: 0.926154\n",
            "Iteration 20, loss = 0.19895402\n",
            "Validation score: 0.929231\n",
            "Iteration 21, loss = 0.18774432\n",
            "Validation score: 0.935385\n",
            "Iteration 22, loss = 0.18716529\n",
            "Validation score: 0.929231\n",
            "Iteration 23, loss = 0.18698461\n",
            "Validation score: 0.929231\n",
            "Iteration 24, loss = 0.18550018\n",
            "Validation score: 0.935385\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 10:30:57,960]\u001b[0m Trial 94 finished with value: 91.3969 and parameters: {'actv': 'relu', 'solvers': 'adam', 'alpha': 0.008694007362631756, 'batch_size': 128, 'learning_rate_init': 0.008219756868757786, 'max_iter': 263, 'momentum': 0.6029561039205309, 'number_layers': 4, 'hidden_neurons_0': 138, 'hidden_neurons_1': 116, 'hidden_neurons_2': 178, 'hidden_neurons_3': 78}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.94098153\n",
            "Validation score: 0.913846\n",
            "Iteration 2, loss = 0.23633888\n",
            "Validation score: 0.941538\n",
            "Iteration 3, loss = 0.23278103\n",
            "Validation score: 0.932308\n",
            "Iteration 4, loss = 0.20907985\n",
            "Validation score: 0.932308\n",
            "Iteration 5, loss = 0.20372819\n",
            "Validation score: 0.923077\n",
            "Iteration 6, loss = 0.19397321\n",
            "Validation score: 0.932308\n",
            "Iteration 7, loss = 0.19417141\n",
            "Validation score: 0.926154\n",
            "Iteration 8, loss = 0.19482155\n",
            "Validation score: 0.920000\n",
            "Iteration 9, loss = 0.19018198\n",
            "Validation score: 0.932308\n",
            "Iteration 10, loss = 0.18934348\n",
            "Validation score: 0.935385\n",
            "Iteration 11, loss = 0.18269127\n",
            "Validation score: 0.941538\n",
            "Iteration 12, loss = 0.18494006\n",
            "Validation score: 0.926154\n",
            "Iteration 13, loss = 0.17850806\n",
            "Validation score: 0.920000\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 10:31:01,988]\u001b[0m Trial 95 finished with value: 91.9519 and parameters: {'actv': 'relu', 'solvers': 'adam', 'alpha': 0.006969852714720612, 'batch_size': 128, 'learning_rate_init': 0.00777470435791318, 'max_iter': 240, 'momentum': 0.6908692466142587, 'number_layers': 4, 'hidden_neurons_0': 124, 'hidden_neurons_1': 193, 'hidden_neurons_2': 159, 'hidden_neurons_3': 75}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.44430881\n",
            "Validation score: 0.901538\n",
            "Iteration 2, loss = 0.24291513\n",
            "Validation score: 0.932308\n",
            "Iteration 3, loss = 0.22141265\n",
            "Validation score: 0.941538\n",
            "Iteration 4, loss = 0.22024281\n",
            "Validation score: 0.950769\n",
            "Iteration 5, loss = 0.21025496\n",
            "Validation score: 0.935385\n",
            "Iteration 6, loss = 0.20491623\n",
            "Validation score: 0.947692\n",
            "Iteration 7, loss = 0.22737764\n",
            "Validation score: 0.889231\n",
            "Iteration 8, loss = 0.23169500\n",
            "Validation score: 0.941538\n",
            "Iteration 9, loss = 0.20401970\n",
            "Validation score: 0.953846\n",
            "Iteration 10, loss = 0.20226852\n",
            "Validation score: 0.947692\n",
            "Iteration 11, loss = 0.19902015\n",
            "Validation score: 0.941538\n",
            "Iteration 12, loss = 0.19389631\n",
            "Validation score: 0.947692\n",
            "Iteration 13, loss = 0.18741722\n",
            "Validation score: 0.950769\n",
            "Iteration 14, loss = 0.18367446\n",
            "Validation score: 0.938462\n",
            "Iteration 15, loss = 0.18616142\n",
            "Validation score: 0.950769\n",
            "Iteration 16, loss = 0.19598998\n",
            "Validation score: 0.953846\n",
            "Iteration 17, loss = 0.18534348\n",
            "Validation score: 0.947692\n",
            "Iteration 18, loss = 0.18200492\n",
            "Validation score: 0.953846\n",
            "Iteration 19, loss = 0.19048823\n",
            "Validation score: 0.935385\n",
            "Iteration 20, loss = 0.18841293\n",
            "Validation score: 0.947692\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 10:31:11,636]\u001b[0m Trial 96 finished with value: 92.0444 and parameters: {'actv': 'relu', 'solvers': 'adam', 'alpha': 0.003670004532828052, 'batch_size': 64, 'learning_rate_init': 0.00860692073741573, 'max_iter': 303, 'momentum': 0.6520730950088863, 'number_layers': 5, 'hidden_neurons_0': 140, 'hidden_neurons_1': 183, 'hidden_neurons_2': 145, 'hidden_neurons_3': 85, 'hidden_neurons_4': 198}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.71326614\n",
            "Validation score: 0.910769\n",
            "Iteration 2, loss = 0.25003650\n",
            "Validation score: 0.916923\n",
            "Iteration 3, loss = 0.23299965\n",
            "Validation score: 0.916923\n",
            "Iteration 4, loss = 0.21506206\n",
            "Validation score: 0.923077\n",
            "Iteration 5, loss = 0.21547124\n",
            "Validation score: 0.916923\n",
            "Iteration 6, loss = 0.21453876\n",
            "Validation score: 0.926154\n",
            "Iteration 7, loss = 0.20162581\n",
            "Validation score: 0.926154\n",
            "Iteration 8, loss = 0.20784490\n",
            "Validation score: 0.913846\n",
            "Iteration 9, loss = 0.19203975\n",
            "Validation score: 0.923077\n",
            "Iteration 10, loss = 0.20184929\n",
            "Validation score: 0.920000\n",
            "Iteration 11, loss = 0.19219812\n",
            "Validation score: 0.932308\n",
            "Iteration 12, loss = 0.19070835\n",
            "Validation score: 0.920000\n",
            "Iteration 13, loss = 0.18647998\n",
            "Validation score: 0.923077\n",
            "Iteration 14, loss = 0.18605691\n",
            "Validation score: 0.910769\n",
            "Iteration 15, loss = 0.18383353\n",
            "Validation score: 0.926154\n",
            "Iteration 16, loss = 0.18410981\n",
            "Validation score: 0.923077\n",
            "Iteration 17, loss = 0.18804584\n",
            "Validation score: 0.920000\n",
            "Iteration 18, loss = 0.18270536\n",
            "Validation score: 0.923077\n",
            "Iteration 19, loss = 0.19532883\n",
            "Validation score: 0.916923\n",
            "Iteration 20, loss = 0.18023025\n",
            "Validation score: 0.916923\n",
            "Iteration 21, loss = 0.17049601\n",
            "Validation score: 0.920000\n",
            "Iteration 22, loss = 0.17774753\n",
            "Validation score: 0.923077\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 10:31:20,924]\u001b[0m Trial 97 finished with value: 91.5819 and parameters: {'actv': 'relu', 'solvers': 'adam', 'alpha': 0.0046077656846468164, 'batch_size': 64, 'learning_rate_init': 0.008692923731764713, 'max_iter': 300, 'momentum': 0.6507922570892035, 'number_layers': 5, 'hidden_neurons_0': 118, 'hidden_neurons_1': 196, 'hidden_neurons_2': 152, 'hidden_neurons_3': 85, 'hidden_neurons_4': 195}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.80981031\n",
            "Validation score: 0.769231\n",
            "Iteration 2, loss = 0.48878709\n",
            "Validation score: 0.907692\n",
            "Iteration 3, loss = 0.37781954\n",
            "Validation score: 0.935385\n",
            "Iteration 4, loss = 0.39327825\n",
            "Validation score: 0.769231\n",
            "Iteration 5, loss = 0.34916471\n",
            "Validation score: 0.941538\n",
            "Iteration 6, loss = 0.32493024\n",
            "Validation score: 0.944615\n",
            "Iteration 7, loss = 0.30071742\n",
            "Validation score: 0.910769\n",
            "Iteration 8, loss = 0.40107669\n",
            "Validation score: 0.769231\n",
            "Iteration 9, loss = 0.44482814\n",
            "Validation score: 0.926154\n",
            "Iteration 10, loss = 0.45336408\n",
            "Validation score: 0.809231\n",
            "Iteration 11, loss = 0.42608605\n",
            "Validation score: 0.870769\n",
            "Iteration 12, loss = 0.37138922\n",
            "Validation score: 0.926154\n",
            "Iteration 13, loss = 0.37786216\n",
            "Validation score: 0.901538\n",
            "Iteration 14, loss = 0.38494331\n",
            "Validation score: 0.926154\n",
            "Iteration 15, loss = 0.34409438\n",
            "Validation score: 0.910769\n",
            "Iteration 16, loss = 0.42242506\n",
            "Validation score: 0.889231\n",
            "Iteration 17, loss = 0.40427167\n",
            "Validation score: 0.920000\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 10:31:28,687]\u001b[0m Trial 98 finished with value: 91.7669 and parameters: {'actv': 'tanh', 'solvers': 'adam', 'alpha': 0.009755043268634658, 'batch_size': 64, 'learning_rate_init': 0.009119539603175408, 'max_iter': 184, 'momentum': 0.7030814727971504, 'number_layers': 4, 'hidden_neurons_0': 140, 'hidden_neurons_1': 181, 'hidden_neurons_2': 125, 'hidden_neurons_3': 66}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.39980376\n",
            "Validation score: 0.913846\n",
            "Iteration 2, loss = 0.30699602\n",
            "Validation score: 0.901538\n",
            "Iteration 3, loss = 0.29579980\n",
            "Validation score: 0.923077\n",
            "Iteration 4, loss = 0.28901607\n",
            "Validation score: 0.929231\n",
            "Iteration 5, loss = 0.28620763\n",
            "Validation score: 0.932308\n",
            "Iteration 6, loss = 0.28362635\n",
            "Validation score: 0.913846\n",
            "Iteration 7, loss = 0.28076288\n",
            "Validation score: 0.923077\n",
            "Iteration 8, loss = 0.28007094\n",
            "Validation score: 0.935385\n",
            "Iteration 9, loss = 0.27457518\n",
            "Validation score: 0.941538\n",
            "Iteration 10, loss = 0.27207642\n",
            "Validation score: 0.932308\n",
            "Iteration 11, loss = 0.27139034\n",
            "Validation score: 0.916923\n",
            "Iteration 12, loss = 0.27008090\n",
            "Validation score: 0.935385\n",
            "Iteration 13, loss = 0.26685893\n",
            "Validation score: 0.895385\n",
            "Iteration 14, loss = 0.26574067\n",
            "Validation score: 0.907692\n",
            "Iteration 15, loss = 0.26469543\n",
            "Validation score: 0.916923\n",
            "Iteration 16, loss = 0.26318668\n",
            "Validation score: 0.913846\n",
            "Iteration 17, loss = 0.25922679\n",
            "Validation score: 0.907692\n",
            "Iteration 18, loss = 0.25899359\n",
            "Validation score: 0.910769\n",
            "Iteration 19, loss = 0.26095955\n",
            "Validation score: 0.941538\n",
            "Iteration 20, loss = 0.25588184\n",
            "Validation score: 0.938462\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-10-07 10:32:00,334]\u001b[0m Trial 99 finished with value: 91.7669 and parameters: {'actv': 'relu', 'solvers': 'sgd', 'alpha': 0.003947204241374226, 'batch_size': 16, 'learning_rate_init': 0.006874482780305113, 'max_iter': 250, 'momentum': 0.5879650835214342, 'number_layers': 4, 'hidden_neurons_0': 183, 'hidden_neurons_1': 103, 'hidden_neurons_2': 194, 'hidden_neurons_3': 163}. Best is trial 4 with value: 92.5069.\u001b[0m\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HO61bU2_kK8A"
      },
      "source": [
        "# ORGINAL HYPER PARAMETERS\n",
        "# net = MLPClassifier(hidden_layer_sizes=(100, ), activation='relu', solver='adam', \n",
        "#                    alpha=0.0001, batch_size='auto', learning_rate='constant', \n",
        "#                     learning_rate_init=0.001, max_iter=200,  \n",
        "#                     verbose=True,  \n",
        "#                     momentum=0.9, early_stopping=True, \n",
        "#                     validation_fraction=0.1, n_iter_no_change=10)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hwPYfDSYFMMO"
      },
      "source": [
        "## Results\n",
        "Printing The Best Parameters\n",
        "\n",
        "After the 100 interactions, the model's hyperparameters with the best accuracy can be seen below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5vpr27X1FMMO",
        "outputId": "3418a8d6-7512-4d7a-fad7-b4610ffcd158"
      },
      "source": [
        "#df['value'].idxmax() #THE MAXIMUM INDEX\n",
        "\n",
        "#Defining DataFrame from optuna \n",
        "df = study.trials_dataframe()\n",
        "\n",
        "#Selecting the row with the maximum acuraccy\n",
        "best_fit = df.iloc[df['value'].idxmax(),:]\n",
        "\n",
        "#Alocating New Parameters\n",
        "new_params_actv = best_fit['params_actv']\n",
        "new_params_alpha = best_fit['params_alpha']\n",
        "new_params_batch_size = best_fit['params_batch_size']\n",
        "params_learning_rate_init = best_fit['params_learning_rate_init']\n",
        "new_params_max_iter = best_fit['params_max_iter']\n",
        "new_params_momentum = best_fit['params_momentum']\n",
        "new_params_number_layers = best_fit['params_number_layers']\n",
        "new_params_solvers = best_fit['params_solvers']\n",
        "\n",
        "new_params_hidden_neurons_0 = best_fit['params_hidden_neurons_0']\n",
        "new_params_hidden_neurons_1 = best_fit['params_hidden_neurons_1']\n",
        "new_params_hidden_neurons_2 = best_fit['params_hidden_neurons_2']\n",
        "new_params_hidden_neurons_3 = best_fit['params_hidden_neurons_3']\n",
        "new_params_hidden_neurons_4 = best_fit['params_hidden_neurons_4']\n",
        "\n",
        "\n",
        "def layer_numerbers(n4,n3,n2,n1,n0):\n",
        "    if type(n4) == int and float:\n",
        "        return ((int(new_params_hidden_neurons_4),int(new_params_hidden_neurons_3),int(new_params_hidden_neurons_2),int(new_params_hidden_neurons_1),int(new_params_hidden_neurons_0)))\n",
        "    elif type(n3) == int or float:\n",
        "        return ((int(new_params_hidden_neurons_3),int(new_params_hidden_neurons_2),int(new_params_hidden_neurons_1),int(new_params_hidden_neurons_0)))\n",
        "    elif type(n2) == int or float:\n",
        "        return ((int(new_params_hidden_neurons_2),int(new_params_hidden_neurons_1),int(new_params_hidden_neurons_0)))\n",
        "    elif type(n1) == int or float:\n",
        "        return ((int(new_params_hidden_neurons_1),int(new_params_hidden_neurons_0)))\n",
        "    elif type(n0) == int or float:\n",
        "        return ((int(new_params_hidden_neurons_0)))\n",
        "neuron_number = layer_numerbers(new_params_hidden_neurons_4,new_params_hidden_neurons_3,new_params_hidden_neurons_2,new_params_hidden_neurons_1,new_params_hidden_neurons_0)\n",
        "\n",
        "\n",
        "\n",
        "print(best_fit)\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "number                                                4\n",
            "value                                           92.5069\n",
            "datetime_start               2021-10-07 08:41:10.052518\n",
            "datetime_complete            2021-10-07 08:41:15.919270\n",
            "duration                         0 days 00:00:05.866752\n",
            "params_actv                                        relu\n",
            "params_alpha                                   0.001988\n",
            "params_batch_size                                    32\n",
            "params_hidden_neurons_0                              61\n",
            "params_hidden_neurons_1                           139.0\n",
            "params_hidden_neurons_2                           153.0\n",
            "params_hidden_neurons_3                           155.0\n",
            "params_hidden_neurons_4                             NaN\n",
            "params_learning_rate_init                       0.00614\n",
            "params_max_iter                                     261\n",
            "params_momentum                                0.618606\n",
            "params_number_layers                                  4\n",
            "params_solvers                                      sgd\n",
            "state                                          COMPLETE\n",
            "Name: 4, dtype: object\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "collapsed": true,
        "jupyter": {
          "outputs_hidden": true
        },
        "id": "rCZncbWjFMMQ",
        "outputId": "14e48807-5762-490f-f074-f25e0e0a031d"
      },
      "source": [
        "#Testing the best hyperparameters\n",
        "\n",
        "best_net = MLPClassifier(hidden_layer_sizes=neuron_number,\n",
        "                    activation=          str(new_params_actv),\n",
        "                    solver=              str(new_params_solvers),\n",
        "                    alpha=               new_params_alpha,\n",
        "                    batch_size=          new_params_batch_size,\n",
        "                    learning_rate=       'constant',\n",
        "                    learning_rate_init = params_learning_rate_init, \n",
        "                    max_iter=            new_params_max_iter,  \n",
        "                    verbose=             True,  \n",
        "                    momentum=            new_params_momentum, \n",
        "                    early_stopping=      True, \n",
        "                    validation_fraction= 0.1, \n",
        "                    n_iter_no_change=10)\n",
        "\n",
        "\n",
        "best_net.fit(x_train,y_train)\n",
        "print('Accuracy: {}%' .format(round(best_net.score(x_test, y_test) * 100,4)))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration 1, loss = 0.35872287\n",
            "Validation score: 0.907692\n",
            "Iteration 2, loss = 0.25295974\n",
            "Validation score: 0.923077\n",
            "Iteration 3, loss = 0.22809400\n",
            "Validation score: 0.926154\n",
            "Iteration 4, loss = 0.21935897\n",
            "Validation score: 0.926154\n",
            "Iteration 5, loss = 0.21743021\n",
            "Validation score: 0.920000\n",
            "Iteration 6, loss = 0.21663729\n",
            "Validation score: 0.926154\n",
            "Iteration 7, loss = 0.21109465\n",
            "Validation score: 0.920000\n",
            "Iteration 8, loss = 0.20819079\n",
            "Validation score: 0.941538\n",
            "Iteration 9, loss = 0.20778098\n",
            "Validation score: 0.944615\n",
            "Iteration 10, loss = 0.20606220\n",
            "Validation score: 0.923077\n",
            "Iteration 11, loss = 0.20166471\n",
            "Validation score: 0.929231\n",
            "Iteration 12, loss = 0.19866906\n",
            "Validation score: 0.935385\n",
            "Iteration 13, loss = 0.20045846\n",
            "Validation score: 0.941538\n",
            "Iteration 14, loss = 0.19991857\n",
            "Validation score: 0.935385\n",
            "Iteration 15, loss = 0.20026292\n",
            "Validation score: 0.901538\n",
            "Iteration 16, loss = 0.19754443\n",
            "Validation score: 0.858462\n",
            "Iteration 17, loss = 0.19523548\n",
            "Validation score: 0.938462\n",
            "Iteration 18, loss = 0.19410664\n",
            "Validation score: 0.938462\n",
            "Iteration 19, loss = 0.19300038\n",
            "Validation score: 0.932308\n",
            "Iteration 20, loss = 0.19173955\n",
            "Validation score: 0.938462\n",
            "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n",
            "Acurácia: 91.8594%\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N8Ia5UiDFMMQ"
      },
      "source": [
        "# _**Final Results**_"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_Sw0Cv8y6Atf",
        "outputId": "d0720371-e2d7-48ab-fd4b-018c49637069"
      },
      "source": [
        "predict = best_net.predict(x_test)\n",
        "print('Accuracy: {}%' .format(round(best_net.score(x_test, y_test) * 100,4)))\n",
        "print('')\n",
        "print(confusion_matrix(y_test, predict))\n",
        "print('')\n",
        "print(classification_report(y_test, predict))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Acurácia: 92%\n",
            "\n",
            "[[789  41]\n",
            " [ 50 201]]\n",
            "\n",
            "                   precision    recall  f1-score   support\n",
            "\n",
            "      non-logging       0.94      0.95      0.95       830\n",
            "selective-logging       0.83      0.80      0.82       251\n",
            "\n",
            "         accuracy                           0.92      1081\n",
            "        macro avg       0.89      0.88      0.88      1081\n",
            "     weighted avg       0.91      0.92      0.92      1081\n",
            "\n"
          ]
        }
      ]
    }
  ]
}