.. _sma-ex2: Ex. 2: Optimization ------------------- This example demonstrates the optimization of the `Ackley `_ function, which is commonly used to test the performance of optimization algorithms. The function has many local minima and one global minimum at :math:`f(0,0)=0`. .. math:: f(x,y)=-20exp[{-0.2\sqrt{0.5(x^2+y^2)}}] - exp[{0.5(cos 2 \pi x + cos 2 \pi y)}] + e + 20 .. figure:: ./images/ackley.jpg :align: center :scale: 50% :target: https://en.wikipedia.org/wiki/Ackley_function Populate the Nodes ++++++++++++++++++ The :ref:`sma-wizard` conveniently provides functionality to populate all the nodes and provides a collection of optimization test functions, including the Ackley function. To construct the workflow quickly and focus on the optimization aspect, use the wizard by: 1. Click on the |wiz| button to launch the wizard. 2. Click on the ``Optimization`` tab and set the ``Function to be optimized`` to ``Ackley`` and level the ``Method`` as ``surrogate``. 3. Click ``Populate Nodes`` You should now have a collection of nodes including: 1. a :ref:`sma-doe` node with two variables, using the hammersly method with 1000 samples 2. a ``Code`` node with the Ackley function 3. a :ref:`sma-rsm` node with the ``radial basis function`` selected, and 4. a :ref:`sma-optimizer` node that looks like this: .. figure:: ./images/ackley_workflow.png :align: center :scale: 50% Local Optimization ++++++++++++++++++ To demonstrate the challenge with black-box optimization, we will first try using a local minimization method: 1. On the :ref:`sma-optimizer`, change the number of attempts to ``10``. 2. Select ``local`` from the ``Method`` combo-box. The ``Local Method`` combo-box should appear with the ``Nelder-Mead`` method selected. 3. Run all the nodes by pressing |play| button on the application toolbar. After all the nodes have been run, the results of the 10 minimization attempts are displayed. Notice how each of the 10 attempts seems to provide a different "optimum". This happens because each attempt has an initial condition which is randomly picked from inside the ranges of the variables specified in the :ref:`sma-doe` node. This local minimization method frequently gets stuck in the many local minima of the Ackley function. The attempts can be visualized on the ``Plot`` tab of the :ref:`sma-optimizer` node, where the surface is the sampled function and the dots are the optimization attempts. Note how these particular attempts never found the global minimum: .. figure:: ./images/ackley_local.png :align: center Global Optimization +++++++++++++++++++ Global optimization routines try to avoid getting stuck in local minima. One method provided in the :ref:`sma-optimizer` node that works well is `Differential Evolution `_. To use this method: 1. select ``differential evolution`` from the ``Method`` combo-box 2. press the ``Optimize`` button on the :ref:`sma-optimizer` node After the node has run, all 10 optimization attempts should be roughly the same value, :math:`f(-0.02, -0.02)=0.19`. The differential evolution algorithm did a good job of avoiding the local minimas and finding the true minimum. Unfortunately, this value is not precisely the true minimum, which should be :math:`f(0,0)=0`. The problem is that there are not enough samples at or around the true minimum for the surrogate model, constructed in the :ref:`sma-rsm`, to accurately represent the sharp minimum of the Ackley function. .. figure:: ./images/ackley_global.png :align: center Response Surface Refinement +++++++++++++++++++++++++++ After an optimum has been found, the response surface can be refined around that optimum to get closer to the true optimum. In this particular case, lets see if we can get closer to the mathematical optimum. To do this, we will leave the original samples in-place and add new samples around the optimum: 1. Duplicate the :ref:`sma-doe` node and ``Code`` node by a. Selecting the :ref:`sma-doe` node by clicking on the title. b. While holding ``Ctrl`` on the keyboard, select the ``Code`` node by clicking on its title. c. Copy the selected nodes by either pressing ``Ctrl`` + ``c`` on the keyboard or ``right-click`` and select ``Copy`` from the menu. d. ``left-click`` to the left of the :ref:`sma-rsm` node. e. Paste the nodes by pressing ``Ctrl`` + ``v`` on the keyboard or ``right-click`` and select ``Paste`` from the menu. 2. Add a ``Matrix/Response`` terminal to the :ref:`sma-rsm` node by pressing the |add| button below the existing ``Matrix/Response`` terminal. 3. Connect the terminals of the copied nodes by a. Connecting the ``Selected Output`` terminal of the copied :ref:`sma-doe` node to the ``matrix`` terminal of the copied ``Code`` node. b. Connecting the ``returnOut`` terminals of the copied ``Code`` node to the new ``Matrix/Response 1`` terminal of the :ref:`sma-rsm` node. c. Connecting the ``DOE Matrix`` terminal of the copied :ref:`sma-doe` node to the new ``Matrix/Response 1`` terminal of the :ref:`sma-rsm` node. 4. Adjust the variable ranges of the copied :ref:`sma-doe` node by a. Select the ``variable`` tab b. Select variable ``a`` and changed the ``from`` value to ``-0.1`` and the ``to`` value to ``0.1``. c. Select variable ``b`` and changed the ``from`` value to ``-0.1`` and the ``to`` value to ``0.1``. d. On the ``Design`` tab, change the number of ``Samples`` to ``100``. e. On the ``Design`` tab, press the ``Build`` button to build the samples. 5. Adjust the response surface in the :ref:`sma-rsm` node node by a. Select the ``Model`` tab b. Select the ``radial basis function`` from the table. c. Change the ``Function`` from ``multiquadric`` to ``cubic`` 6. Run all the nodes by pressing |play| button on the application toolbar. By adding the additional samples, the found optimum (:math:`f(-0.0007, 0.001)=0.02`) is closer to the analytic solution (:math:`f(0,0)=0`). You should have a workflow that looks like this: .. figure:: ./images/ackley_refine.png :align: center :scale: 50% .. Note:: See how close you can get to the true solution of :math:`f(0,0)=0` by changing the samples and response surface model settings. .. |play| image:: ../../../nodeworks/images/play.svg .. |add| image:: ../../../nodeworks/images/add.svg .. |wiz| image:: ../../../nodeworks/images/wand.svg