Skip to main content

Sinuoidal Functions II

We will continue with sinusoidal type functions for the examples we will present, but now from R2→R\mathbb{R}^2\to\mathbb{R}. For the examples, we will use the following libraries,

# Libraries
import matplotlib.pyplot as plt
import numpy as np
import tensorflow as tf

and we're going to have to have Ω⊂R2\Omega\sub\mathbb{R^2} is compact (closed and bounded).

Implementation in more dimensions, now for sin⁡(x+y)\sin(x+y)​

Let Ω=[−2π,2π]×[−2π,2π].\Omega=[-2\pi, 2\pi]\times [-2\pi, 2\pi]. The idea is to approximate a neural network Φ:Ω→R\Phi:\Omega\to\mathbb{R} to the continuous function f:Ω→Rf: \Omega\to\mathbb{R}, f(x,y)=sin⁡(x+y)f(x,y)=\sin(x+y).

To calculate the points of the domain and the function in the domain,

# Omega domain
num_points = 256
omega_x = np.linspace(-2*np.pi, 2*np.pi, num_points, dtype="float64")
omega_y = np.linspace(-2*np.pi, 2*np.pi, num_points, dtype="float64")
omega_x, omega_y = np.meshgrid(omega_x, omega_y) 
omega = np.stack([omega_x.ravel(), omega_y.ravel()], axis=1)
f_omega = np.sin(omega[:,0]+ omega[:,1]) # function sin(x+y) "implementation"

omega_x_tf = tf.convert_to_tensor(omega[:,0][:, np.newaxis])
omega_y_tf = tf.convert_to_tensor(omega[:,1][:, np.newaxis])
f_omega_tf = tf.convert_to_tensor(f_omega[:, np.newaxis])

construction of a neural network of R2→R\mathbb{R}^2\to\mathbb{R}

# Single hidden Layer Neural Network Phi wit M=400 hidden units
M = 400
weight_x = tf.Variable(tf.random.normal([1,M], dtype=tf.float64))
weight_y = tf.Variable(tf.random.normal([1,M], dtype=tf.float64))
beta = tf.Variable(tf.random.normal([M], dtype=tf.float64))
alpha = tf.Variable(tf.random.normal([M,1], dtype=tf.float64))

@tf.function
def Phi(x, y):
    return tf.matmul(
        tf.keras.activations.tanh(
            tf.add(
                tf.add(tf.matmul(x, weight_x), tf.matmul(y, weight_y)),
                beta
            )
        ),
        alpha
    )

training,

# training by gradient descent
learning_rate = 0.1
training_epochs = 4000

optimizer = tf.keras.optimizers.Adam(learning_rate)
mse = tf.keras.losses.MeanSquaredError()

for epoch in range(training_epochs):
    with tf.GradientTape() as tape:
        loss = mse(Phi(omega_x_tf, omega_y_tf), f_omega_tf)
    gradients = tape.gradient(loss, [weight_x, weight_y, beta, alpha])
    optimizer.apply_gradients(zip(gradients, [weight_x, weight_y, beta, alpha]))

For the visual element, we only plot the contours of the original function, the neural network and its absolute error,

# Graphic
f_omega = f_omega_tf.numpy().reshape((num_points, num_points))
f_omega_nn = Phi(omega_x_tf, omega_y_tf).numpy().reshape((num_points, num_points))

fig, axs = plt.subplots(1, 3, figsize=(18, 5))

cf1 = axs[0].contourf(omega_x, omega_y, f_omega, cmap='viridis')
axs[0].set_title("$\\sin(x^2+y^2)$")
plt.colorbar(cf1, ax=axs[0])

cf2 = axs[1].contourf(omega_x, omega_y, f_omega_nn, cmap='viridis')
axs[1].set_title("$\\Phi(x,y)$")
plt.colorbar(cf2, ax=axs[1])

cf3 = axs[2].contourf(omega_x, omega_y, np.abs(f_omega - f_omega_nn), cmap='inferno')
axs[2].set_title("$|\\sin(x^2+y^2) - \\Phi(x,y)|$")
plt.colorbar(cf3, ax=axs[2])

for ax in axs:
    ax.set_xlabel("$x$")
    ax.set_ylabel("$y$")

plt.tight_layout()
plt.show()

function approximation sin(x+y)

The image shows how the neural network approximated the function, and you can see they have the same nature. The approximation can be improved by adding more neurons and more epochs. There are also other options, such as applying training strategies and adding more layers. We'll briefly review all of this in the next section.

Another style of implementation, now for sin⁡(x2+y2)\sin(x^2+y^2)​

As you can see, when we added a dimension, it became a bit more difficult to implement the functions, even though we only used a single layer. For this, there are equivalent ways to implement neural networks with multiple layers and outputs. In this case, a two-layer function with one output will be implemented.

Let Ω=[−2π,2π]×[−2π,2π].\Omega=[-2\pi, 2\pi]\times [-2\pi, 2\pi]. The idea is to approximate a neural network Φ:Ω→R\Phi:\Omega\to\mathbb{R} to the continuous function f:Ω→Rf: \Omega\to\mathbb{R}, f(x,y)=sin⁡(x2+y2)f(x,y)=\sin(x^2+y^2).

For this other type of implementation, we must use the following libraries,

from tensorflow.keras import models, layers
from tensorflow.keras.callbacks import ReduceLROnPlateau
from tensorflow.keras.optimizers import Adam

to calculate the points of the domain and the function in the domain,

num_points = 1024
omega_x = np.linspace(-2*np.pi, 2*np.pi, num_points, dtype="float64")
omega_y = np.linspace(-2*np.pi, 2*np.pi, num_points, dtype="float64")
omega_x, omega_y = np.meshgrid(omega_x, omega_y)
omega = np.stack([omega_x.ravel(), omega_y.ravel()], axis=1)
f_omega = np.sin(omega[:,0]**2+ omega[:,1]**2)

new construction of a neural network of R2→R\mathbb{R}^2\to\mathbb{R},

model = models.Sequential([
    layers.Input(shape=(2,)),
    layers.Dense(128, activation="tanh"),
    layers.Dense(128, activation="tanh"),
    layers.Dense(1)
])

this is less verbose and more compact, and it is easier to add layers as dimensions. Whenever possible, you should try to use this form, although there may be problems which would arise if you had to use the extended form.

New training settings,

model.compile(optimizer="adam", loss="mse")

reduce_lr = ReduceLROnPlateau(
    monitor='loss',       # o 'val_loss' si tienes validación
    factor=0.5,           # reduce a la mitad la tasa de aprendiSzaje
    patience=5,          # espera 10 epochs sin mejora
    min_lr=1e-7,          # no baja más de esta tasa
    verbose=1             # imprime cuando cambia
)

as you can see, it's easier to configure the optimizer and the loss. It's also easier to add a strategy to reduce the learning rate.

Training,

model.fit(omega, f_omega, epochs=350, verbose=1, callbacks=[reduce_lr])

For the visual element, we only plot the contours of the original function, the neural network and its absolute error,

f_omega_tf = tf.convert_to_tensor(f_omega, dtype=tf.float64)
omega_x_tf = tf.convert_to_tensor(omega_x, dtype=tf.float64)
omega_y_tf = tf.convert_to_tensor(omega_y, dtype=tf.float64)

# Red neuronal evaluada
f_omega_nn = model.predict(omega).reshape((num_points, num_points))

# Reestructuramos ground truth para graficar
f_omega = f_omega.reshape((num_points, num_points))

fig, axs = plt.subplots(1, 3, figsize=(18, 5))

cf1 = axs[0].contourf(omega_x, omega_y, f_omega, cmap='viridis')
axs[0].set_title("$\\sin(x^2+y^2)$")
plt.colorbar(cf1, ax=axs[0])

cf2 = axs[1].contourf(omega_x, omega_y, f_omega_nn, cmap='viridis')
axs[1].set_title("$\\Phi(x, y)$")
plt.colorbar(cf2, ax=axs[1])

cf3 = axs[2].contourf(omega_x, omega_y, np.abs(f_omega - f_omega_nn), cmap='inferno')
axs[2].set_title("$|\\sin(x^2+y^2) - \\Phi|$")
plt.colorbar(cf3, ax=axs[2])

for ax in axs:
    ax.set_xlabel("$x$")
    ax.set_ylabel("$y$")

plt.tight_layout()
plt.show()

function approximation sin(x+y)

It can be noted that with the changes, the neural network better approximated the function, considering its more difficult nature to reach. In fact, it's a very good approximation; the difficulty lies in the corners, but these can be improved with more neurons, perhaps another layer, and more epochs.