November 2022 – Parameter-free Learning and Optimization Algorithms
1 post published by bremen79 during November 2022| Parameter-free Learning and Optimization Algorithms
1 post published by bremen79 during November 2022| Parameter-free Learning and Optimization Algorithms
This time we will introduce the Weighted Average Algorithm (WAA). I will do it my way: I am allergic to present for each algorithm a different analysis! From my blog it should be clear that we only have two main algorithms in online learning: OMD and FTRL. So, 99% of the online algorithms are instantiations […]| Parameter-free Learning and Optimization Algorithms
In this post, we introduce yet another way to quantify the ability of online learning algorithms to compete with a different comparators, besides the dynamic regret that we saw last time. 1. Strongly Adaptive Regret We introduce the concept of strongly adaptive regret: This definition captures the fact that we want the performance of the […]| Parameter-free Learning and Optimization Algorithms
In this post, we will see how to extend the notion of regret to a sequence of comparators instead of a single one. We saw that the definition of regret makes sense as a direct generalization of both the stochastic setting and the offline optimization. However, in some cases, we know that the environment is […]| Parameter-free Learning and Optimization Algorithms
We will continue with black-box reductions, this time solving the problem of sleeping experts. 1. Sleeping Experts Consider now the setting of learning with experts where only a subset of the experts is active in each round. In particular we have that is expert is active at time , and if the expert is inactive, […]| Parameter-free Learning and Optimization Algorithms
This is the first post on a new topic: how to reduce one online learning problem into another in a black-box way. That is, we will use an online convex optimization algorithm to solve a problem different from what it was meant to solve, without looking at its internal working in any way. The only […]| Parameter-free Learning and Optimization Algorithms
A few recent Arxiv papers and some recent conversations during my lectures made me realize that some optimization people might not be fully aware of important details on SGD when used on functions where the minimizer can be arbitrarily far from the initialization or even in the case when the minimizer does not exist. So, […]| Parameter-free Learning and Optimization Algorithms
This post is about Nesterov’s universal algorithm and how delicate is to claim that an algorithm is “universal”, “parameter-free”, “adaptive”, or any other similar word to denote the fact that the algorithm does not need prior knowledge of the characteristics of a function to converge at its best rate. This post was born from a […]| Parameter-free Learning and Optimization Algorithms
Disclaimer: I deliberated extensively on whether writing this blog post was a good idea. Some kind of action was necessary because this award was just too unfair. I consulted with many senior people in my field. Ultimately, I concluded that this path was the most appropriate one to take because critiques are an integral part […]| Parameter-free Learning and Optimization Algorithms
I am back! I finally found some spare time to write something interesting. A lot of people told me that they like this blog because I present “new” results. Sometimes the results I present are so “new” that people cite me in papers. This makes me happy: Not only you learned something reading my posts, […]| Parameter-free Learning and Optimization Algorithms
In the latest posts, we saw that it is possible to solve convex/concave saddle-point optimization problems using two online convex optimization algorithms playing against each other. We obtained a …| Parameter-free Learning and Optimization Algorithms
Because life is too short to tune learning rates. By Francesco Orabona| Parameter-free Learning and Optimization Algorithms
EDIT 4/25/23This blog post went viral in 2020 and this idea is now widely accepted by the deep learning community. In fact, this is not only the most read post on my blog, but I might say that this…| Parameter-free Learning and Optimization Algorithms
