{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3b68d7f5",
   "metadata": {},
   "source": [
    "# How to use Umlaut\n",
    "\n",
    "## Import desired metrics\n",
    "To use the measurement methods in a pipeline, the desired metrics have to be imported from the Umlaut package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "55a5f4d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from umlaut import Benchmark,\\\n",
    "                     ConfusionMatrixTracker,\\\n",
    "                     HyperparameterTracker,\\\n",
    "                     BenchmarkSupervisor,\\\n",
    "                     TimeMetric,\\\n",
    "                     MemoryMetric,\\\n",
    "                     PowerMetric,\\\n",
    "                     EnergyMetric,\\\n",
    "                     ThroughputMetric,\\\n",
    "                     LatencyMetric,\\\n",
    "                     TTATracker,\\\n",
    "                     LossTracker,\\\n",
    "                     CPUMetric\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from keras import optimizers\n",
    "\n",
    "def prep_data():\n",
    "    fashion_mnist = tf.keras.datasets.fashion_mnist\n",
    "\n",
    "    (train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data()\n",
    "    class_names = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat',\n",
    "                   'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']\n",
    "\n",
    "    train_images = train_images / 255.0\n",
    "    test_images = test_images / 255.0\n",
    "\n",
    "    return train_images, test_images, train_labels, test_labels, class_names"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a287f1b",
   "metadata": {},
   "source": [
    "## Initializing the benchmark object\n",
    "To start include the measurement functions into the pipeline, a benchmark object has to be created. A benchmark object manages the database entries for the measured metrics. It provides functions to log and access measurement data. The object receives the database file in which the results should be stored and a description as input values.\n",
    "\n",
    "## Two kinds of metrics\n",
    "The Umlaut framework distinguishes two different kinds of metrics. The supervised metrics and the valued metrics. Both kinds of metrics have to be used different.\n",
    "\n",
    "### Supervised metrics\n",
    "The supervised metrics include:\n",
    "- throughput\n",
    "- latency\n",
    "- time\n",
    "- memory\n",
    "- power\n",
    "- energy\n",
    "- cpu"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "2d3abf2e",
   "metadata": {},
   "outputs": [],
   "source": [
    "bm = Benchmark('mnist_db', description=\"mnist_clothing\")\n",
    "\n",
    "mnist_metrics = {\n",
    "    \"throughput\": ThroughputMetric('mnist throughput'),\n",
    "    \"latency\": LatencyMetric('mnist latency'),\n",
    "    \"time\": TimeMetric('mnist time'),\n",
    "    \"memory\": MemoryMetric('mnist memory', interval=0.1),\n",
    "    \"power\": PowerMetric('mnist power'),\n",
    "    \"energy\": EnergyMetric('mnist energy'),\n",
    "    \"cpu\": CPUMetric('mnist' cpu', interval=0.1)\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9ba1084",
   "metadata": {},
   "source": [
    "To include the measurement of supervised metrics the desired metrics as well as the benchmark object have to be passed to a @BenchmarkSupervisor decorator. The appropriate BenchmarkSupervisor object manages the tracking of all supervised metrics.\n",
    "\n",
    "### Valued metrics\n",
    "The valued metrics include:\n",
    "- confusion matrix\n",
    "- time to accuracy\n",
    "- loss\n",
    "\n",
    "To collect measurements for the valued metrics, each one has a track function which has to be called with the corresponding input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "3da2510e",
   "metadata": {},
   "outputs": [],
   "source": [
    "@BenchmarkSupervisor(bloat_metrics.values(), bm)\n",
    "def train(train_images, train_labels):\n",
    "    model = tf.keras.Sequential([\n",
    "        tf.keras.layers.Flatten(input_shape=(28, 28)),\n",
    "        tf.keras.layers.Dense(128, activation='relu'),\n",
    "        tf.keras.layers.Dense(10)\n",
    "    ])\n",
    "\n",
    "    model.compile(optimizer=\"Adam\",\n",
    "                  loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
    "                  metrics=['accuracy'])\n",
    "\n",
    "    history = model.fit(train_images, train_labels, epochs=10, batch_size=256)\n",
    "\n",
    "    mnist_metrics[\"throughput\"].track(num_entries=len(train_images))\n",
    "    mnist_metrics[\"latency\"].track(num_entries=len(train_images))\n",
    "\n",
    "    TTATracker(bm).track(accuracies=history.history[\"accuracy\"],  description=\"TTA\")\n",
    "    LossTracker(bm).track(loss_values=history.history['loss'], description=\"loss\")\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "410a0f5b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def test(model, test_images, test_labels, class_names):\n",
    "\n",
    "    model.evaluate(test_images, test_labels)\n",
    "    pred_test = model.predict_classes(test_images)\n",
    "    con_mat = confusion_matrix(test_labels, pred_test)\n",
    "    ConfusionMatrixTracker(bm).track(con_mat, class_names, \"confusion matrix\")\n",
    "    \n",
    "    bm.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3850d0b1",
   "metadata": {},
   "source": [
    "## Collect measurements\n",
    "When all desired metrics are applied to the pipeline it can be executed and measurements are collected during the runtime."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "343df58f",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2021-07-26 13:40:53.128730: W tensorflow/core/framework/cpu_allocator_impl.cc:80] Allocation of 188160000 exceeds 10% of free system memory.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.8690 - accuracy: 0.7114\n",
      "Epoch 2/10\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.4414 - accuracy: 0.8489\n",
      "Epoch 3/10\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.3959 - accuracy: 0.8628\n",
      "Epoch 4/10\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.3678 - accuracy: 0.8720\n",
      "Epoch 5/10\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.3469 - accuracy: 0.8776\n",
      "Epoch 6/10\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.3349 - accuracy: 0.8809\n",
      "Epoch 7/10\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.3165 - accuracy: 0.8867\n",
      "Epoch 8/10\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.3068 - accuracy: 0.8899\n",
      "Epoch 9/10\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.3026 - accuracy: 0.8914\n",
      "Epoch 10/10\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.2821 - accuracy: 0.8985\n"
     ]
    }
   ],
   "source": [
    "train_images, test_images, train_labels, test_labels, class_names = prep_data()\n",
    "model = train(train_images, train_labels)\n",
    "test(model, test_images, test_labels, class_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0dc8107c",
   "metadata": {},
   "source": [
    "## Visualize results\n",
    "To evaluate the measurement results, Umlaut provides a command line interface as visualization frontend. To use this interface **umlaut-cli mnist_db.db** has to be executed. The CLI prompts the user to insert a valid combination for the UUID, the measurement type and the measurement description to visualize the corresponding metrics."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d0aa0b8",
   "metadata": {},
   "source": [
    "\n",
    "![Time](https://imgur.com/QfXlA8C.png)\n",
    "![Memory](https://imgur.com/fOh7gIL.png)\n",
    "![CPU](https://i.imgur.com/pSj1NDn.png)\n",
    "![Energy](https://imgur.com/3cuhcdo.png)\n",
    "![Latency](https://imgur.com/SVdR0EG.png)\n",
    "![Loss](https://imgur.com/8X041nq.png)\n",
    "![Power](https://imgur.com/z59U8aT.png)\n",
    "![Thrpughput](https://imgur.com/NzSGUgr.png)\n",
    "![TTA](https://imgur.com/31gHh8E.png)\n",
    "![Matrix](https://imgur.com/7WEJ4Ta.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f678f431",
   "metadata": {},
   "source": [
    "## Hyperparameter Tracking\n",
    "\n",
    "Umlaut also includes measuring hyperparameters. Similar to valued metrics, trackers are used to collect hyperparameter measurements. The user can decide which parameters and which values for them to track."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "0717361a",
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(train_images, train_labels):\n",
    "    ht = HyperparameterTracker(bm, \"hyper params of sample pipeline\", ['lr', 'num_epochs', 'batch_size'],\n",
    "                               'loss')\n",
    "    for lr in [0.1, 0.01, 0.001]:\n",
    "        for num_epochs in [5, 10, 20]:\n",
    "            for batch_size in [64, 128, 256]:\n",
    "\n",
    "                model = tf.keras.Sequential([\n",
    "                    tf.keras.layers.Flatten(input_shape=(28, 28)),\n",
    "                    tf.keras.layers.Dense(128, activation='relu'),\n",
    "                    tf.keras.layers.Dense(10)\n",
    "                ])\n",
    "\n",
    "                optimizer = optimizers.Adam(lr=lr)\n",
    "\n",
    "                model.compile(optimizer=optimizer,\n",
    "                              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
    "                              metrics=['accuracy'])\n",
    "\n",
    "                history = model.fit(train_images, train_labels, epochs=num_epochs, batch_size=batch_size)\n",
    "                ht.track({'lr': lr, 'num_epochs': num_epochs, 'batch_size': batch_size, 'loss': history.history[\"accuracy\"][-1]})\n",
    "    ht.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1227030",
   "metadata": {},
   "source": [
    "![Hyperparameter](https://imgur.com/vk1kejk.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2fcc2d7f",
   "metadata": {},
   "source": [
    "# Troubleshooting "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7bee6879",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## PyRAPL (Intel) - No Access\n",
    "\n",
    "Run **sudo chmod -R a+r /sys/class/powercap/intel-rapl**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}