Last week we have developed the federated Keras model, now we can test some hyperparameters to improve our performances:
Testing many combinations of Learning rates for the server and the client, the best settings seems to be:
With a number of records equal to 100.000 we reach a sparse categorical accuracy of 7.0%
Now it's time to try to change the layers. First, I will change the weights initialization of the dense layers to 'he_uniform' because it seems to work better with relu activations. (More on this on the last week)
Using the he_uniform initialization in 10 rounds we reach 9.4% of sparse categorical accuracy!
Let's try now to change the number of neurons in the hidden layers from 128 to 164 we reach 9.49% of sparse categorical accuracy, not a big difference considering that we are incrementing the number of weights and so the dimension of the model...
So now let's get back the original architecture which let us achieve 9.4% and try to change the embedding size from 64 to 128.
After our 10 rounds we reach the 8.7%.. nevermind.
So try now to reduce that value instead, switch from 64 to 32: Ok it does not work.
Until now we had two hidden layers, so let's try to remove one dense layer.. It improves the performance, after just 3 round we get 12% but then the accuracy seems to stop growing even if the loss still decrease slowly.