.. _sma-ex2:

Ex. 2: Optimization
-------------------

This example demonstrates the optimization of the
`Ackley <https://en.wikipedia.org/wiki/Ackley_function>`_ function, which is
commonly used to test the performance of optimization algorithms. The function
has many local minima and one global minimum at :math:`f(0,0)=0`.

.. math::

   f(x,y)=-20exp[{-0.2\sqrt{0.5(x^2+y^2)}}] - exp[{0.5(cos 2 \pi x + cos 2 \pi y)}] + e + 20

.. figure:: ./images/ackley.jpg
   :align: center
   :scale: 50%
   :target: https://en.wikipedia.org/wiki/Ackley_function


Populate the Nodes
++++++++++++++++++

The :ref:`sma-wizard` conveniently provides functionality to populate all the
nodes and provides a collection of optimization test functions, including the
Ackley function. To construct the workflow quickly and focus on the optimization
aspect, use the wizard by:

 1. Click on the |wiz| button to launch the wizard.
 2. Click on the ``Optimization`` tab and set the ``Function to be optimized``
    to ``Ackley`` and level the ``Method`` as ``surrogate``.
 3. Click ``Populate Nodes``

You should now have a collection of nodes including:

 1. a :ref:`sma-doe` node with two variables, using the hammersly method with
    1000 samples
 2. a ``Code`` node with the Ackley function
 3. a :ref:`sma-rsm` node with the ``radial basis function`` selected, and
 4. a :ref:`sma-optimizer` node

 that looks like this:

 .. figure:: ./images/ackley_workflow.png
    :align: center
    :scale: 50%

Local Optimization
++++++++++++++++++

To demonstrate the challenge with black-box optimization, we will first try
using a local minimization method:

1. On the :ref:`sma-optimizer`, change the number of attempts to ``10``.
2. Select ``local`` from the ``Method`` combo-box. The ``Local Method``
   combo-box should appear with the ``Nelder-Mead`` method selected.
3. Run all the nodes by pressing |play| button on the application toolbar.

After all the nodes have been run, the results of the 10 minimization attempts
are displayed. Notice how each of the 10 attempts seems to provide a different
"optimum". This happens because each attempt has an initial condition which is
randomly picked from inside the ranges of the variables specified in the
:ref:`sma-doe` node. This local minimization method frequently gets stuck in the
many local minima of the Ackley function.

The attempts can be visualized on the ``Plot`` tab of the :ref:`sma-optimizer`
node, where the surface is the sampled function and the dots are the
optimization attempts. Note how these particular attempts never found the global
minimum:

.. figure:: ./images/ackley_local.png
   :align: center

Global Optimization
+++++++++++++++++++

Global optimization routines try to avoid getting stuck in local minima. One
method provided in the :ref:`sma-optimizer` node that works well is
`Differential Evolution <https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.differential_evolution.html>`_.

To use this method:

1. select ``differential evolution`` from the ``Method`` combo-box
2. press the ``Optimize`` button on the :ref:`sma-optimizer` node

After the node has run, all 10 optimization attempts should be roughly the same
value, :math:`f(-0.02, -0.02)=0.19`. The differential evolution algorithm did a
good job of avoiding the local minimas and finding the true minimum.
Unfortunately, this value is not precisely the true minimum, which should be
:math:`f(0,0)=0`. The problem is that there are not enough samples at or around
the true minimum for the surrogate model, constructed in the :ref:`sma-rsm`, to
accurately represent the sharp minimum of the Ackley function.

.. figure:: ./images/ackley_global.png
   :align: center

Response Surface Refinement
+++++++++++++++++++++++++++

After an optimum has been found, the response surface can be refined around that
optimum to get closer to the true optimum. In this particular case, lets see if
we can get closer to the mathematical optimum. To do this, we will leave the
original samples in-place and add new samples around the optimum:

1. Duplicate the :ref:`sma-doe` node and ``Code`` node by

    a. Selecting the :ref:`sma-doe` node by clicking on the title.
    b. While holding ``Ctrl`` on the keyboard, select the ``Code`` node by
       clicking on its title.
    c. Copy the selected nodes by either pressing ``Ctrl`` + ``c`` on the
       keyboard or ``right-click`` and select ``Copy`` from the menu.
    d. ``left-click`` to the left of the :ref:`sma-rsm` node.
    e. Paste the nodes by pressing ``Ctrl`` + ``v`` on the keyboard or
       ``right-click`` and select ``Paste`` from the menu.

2. Add a ``Matrix/Response`` terminal to the :ref:`sma-rsm` node by pressing
   the |add| button below the existing ``Matrix/Response`` terminal.

3. Connect the terminals of the copied nodes by

    a. Connecting the ``Selected Output`` terminal of the copied :ref:`sma-doe`
       node to the ``matrix`` terminal of the copied ``Code`` node.
    b. Connecting the ``returnOut`` terminals of the copied ``Code`` node to the
       new ``Matrix/Response 1`` terminal of the :ref:`sma-rsm` node.
    c. Connecting the ``DOE Matrix`` terminal of the copied :ref:`sma-doe` node
       to the new ``Matrix/Response 1`` terminal of the :ref:`sma-rsm` node.

4. Adjust the variable ranges of the copied :ref:`sma-doe` node by

    a. Select the ``variable`` tab
    b. Select variable ``a`` and changed the ``from`` value to ``-0.1`` and the
       ``to`` value to ``0.1``.
    c. Select variable ``b`` and changed the ``from`` value to ``-0.1`` and the
       ``to`` value to ``0.1``.
    d. On the ``Design`` tab, change the number of ``Samples`` to ``100``.
    e. On the ``Design`` tab, press the ``Build`` button to build the samples.

5. Adjust the response surface in the :ref:`sma-rsm` node node by

    a. Select the ``Model`` tab
    b. Select the ``radial basis function`` from the table.
    c. Change the ``Function`` from ``multiquadric`` to ``cubic``

6. Run all the nodes by pressing |play| button on the application toolbar.

By adding the additional samples, the found optimum
(:math:`f(-0.0007, 0.001)=0.02`) is closer to the analytic solution
(:math:`f(0,0)=0`). You should have a workflow that looks like this:

.. figure:: ./images/ackley_refine.png
   :align: center
   :scale: 50%


.. Note:: See how close you can get to the true solution of :math:`f(0,0)=0` by
   changing the samples and response surface model settings.

.. |play| image:: ../../../nodeworks/images/play.svg
.. |add| image:: ../../../nodeworks/images/add.svg
.. |wiz| image:: ../../../nodeworks/images/wand.svg