Is this you? As a journalist, you can create a free Muck Rack account to customize your profile, list your contact preferences, and upload a portfolio of your best work.
Claim your profile
Articles by Wouter van Heeswijk
Mathematics > Optimization and Control
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
[2512.11187] Deep Learning--Accelerated Multi-Start Large Neighborhood Search for Real-time Freight Bundling
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Unwrapping the Power of Python: A Quick Guide to Wrappers and Decorators
Unwrapping the Power of Python: A Quick Guide to Wrappers and Decorators The popularity of Python programming language can be largely explained due to its facilitation of clean, concise, and efficient code. Wrappers and decorators are among the functionalities that enhance code readability and reusability in Python.
A Marriage of Machine Learning and Optimization Algorithms
How pattern detection and pattern exploitation might elevate each other to a new level Although most of us don’t see it, optimization algorithms (OAs) are at work everywhere. They plan shelve stocking for our grocery stores, create airport schedules, and give us the shortest route to our holiday destination. Exact algorithms in particular do very well at exploiting known structures — e.g., convex structures — finding solutions even in massive decision spaces with many constraints.
Entropy-Regularized Reinforcement Learning Explained
Learn more reliable, robust, and transferable policies by adding entropy bonuses to your algorithm Entropy is a concept associated with a state of disorder, randomness, or uncertainty. It can be considered as a measure of information for random variables. Traditionally, it is associated with fields such as thermodynamics, but the term found its way to many other domains. In 1948, Claude Shannon introduced the notion of entropy in information theory.
Ten Lessons In Time Management
Being successful doesn't make you manage your time well. Managing your time well makes you successful. - Randy Pausch In 2006, time must have stood still for Randy Pausch, the moment he heard he had a terminal form of pancreatic cancer. With a remaining life expectancy of mere months, the computer science professor decided to get the most out of what time he had left on earth.
Five Ways To Handle Large Action Spaces in Reinforcement Learning
Action spaces, particularly in combinatorial optimization problems, may grow unwieldy in size. This article discusses five strategies to handle them. Handling large action spaces remains a fairly open problem in Reinforcement Learning. Researchers have made great strides in terms of handling large state spaces, with convolutional networks and transformers being some recent high-profile examples. However, there are three so-called curses of dimensionality: state, outcome, and action [1].
Nailing DataCamp's Data Scientist Professional Certification
By Wouter van Heeswijk, PhD Prepare yourself for the exams DS101, DS201 and the practical exam Photo by Vasily Koloda on Unsplash DataCamp is a great way to build a data science foundation. This article explains how to complete the certification trajectory and solidify that foundation.
Deep Deterministic Policy Gradients Explained
A gradient-based reinforcement learning algorithm to learn deterministic policies for continuous action spaces This article introduces Deep Deterministic Policy Gradient (DDPG) — a Reinforcement Learning algorithm suitable for deterministic policies applied in continuous action spaces. By combining the actor-critic paradigm with deep neural networks, continuous action spaces can be tackled without resorting to stochastic policies.
Solving The Taxi Environment With Q-Learning - A Tutorial
The goal of the Taxi Environment in OpenAI’s Gym — yes, from the company behind ChatGPT and Dall⋅E — is simple and straightforward, making for an excellent introduction to the field of Reinforcement Learning (RL). This article provides a step-to-step guide to implement the environment, learn a policy using tabular Q-learning, and visualize the learned behavior in animations. If you are looking to make your first strides in RL, this tutorial might be just for you.
When Stochastic Policies Are Better Than Deterministic Ones
Why we let randomness dictate our action selection in Reinforcement Learning If you are used to deterministic decision-making policies (e.g., as in Deep Q-learning), the need for and use of stochastic policies might elude you. After all, deterministic policies offer a convenient state-action mapping π:s ↦ a, ideally even the optimal mapping (that is, if all the Bellman equations are learned to perfection).
When Stochastic Policies Are Better Than Deterministic Ones
1 day ago If you were accustomed to a time when Twitter — while far from perfect — was a place where you could dependably digest a wide range of breaking news, politics, celebrity gossip, or personal musings, it’s time to accept a new reality. Twitter is becoming a degraded product. In the four months since …
Three Fundamental Flaws In Common Reinforcement Learning Algorithms (And How To Fix Them)
Harness yourself against these shortcomings encountered in everyday RL algorithms Reinforcement Learning algorithms such as Q-learning and REINFORCE have been around for decades and their textbook implementations are still widely used. Unfortunately, they exhibit some fundamental flaws, which greatly increase the struggle of learning a good policy. This article addresses three major shortcomings of classical Reinforcement Learning algorithms, along with solutions to overcome them.
Rainbow DQN - The Best Reinforcement Learning Has to Offer?
Reinforcement Learning has come a long way since the era of classical tabular Q-learning. Deep Q-Learning Networks (DQN) drove a revolution in the field, enabling powerful generalizations of states. By replacing a Q-table (storing the values of all state-action pair) with a powerful neural network (theoretically able to transform any input state to an output value per action), we could handle problems with massive state spaces, signifying a breakthrough in the field.
Proximal Policy Optimization (PPO) Explained
The journey from REINFORCE to the go-to algorithm in continuous control Proximal Policy Optimization (PPO) is presently considered state-of-the-art in Reinforcement Learning. The algorithm, introduced by OpenAI in 2017, seems to strike the right balance between performance and comprehension. It is empirically competitive with quality benchmarks, even vastly outperforming them on some tasks.
Mass Layoffs in Tech — Is The AI Winter Coming?
Until very recently, companies were fighting to attract and retain quality staff in data science. Online business thrived during times of lockdown, with the world suddenly relying on parcel deliveries, cloud environments, online meeting spaces, and virtual pastimes. Tech giants reported record profits, funneling their excess cash into ambitious AI projects and -innovations [1].
Mass Layoffs in Tech
Until very recently, companies were fighting to attract and retain quality staff in data science. Online business thrived during times of lockdown, with the world suddenly relying on parcel deliveries, cloud environments, online meeting spaces, and virtual pastimes. Tech giants reported record profits, funneling their excess cash into ambitious AI projects and -innovations [1].
Were Abstract Painters The First Encoders?
The journey to capture reality with a bare minimum of shapes and colors Encoding may be seen as the art of abstracting the most salient patterns from reality. Such abstractions are indispensable for the purpose of generalization. For instance, consider the textbook example of recognizing cats in pictures. A trained neural network should be able to detect cats even in previously unseen images. This is only possible if the network can extrapolate its training observations.
Were Abstract Painters The First Encoders?
Encoding may be seen as the art of abstracting the most salient patterns from reality. Such abstractions are indispensable for the purpose of generalization. For instance, consider the textbook example of recognizing cats in pictures. A trained neural network should be able to detect cats even in previously unseen images. This is only possible if the network can extrapolate its training observations.
Robotic Process Automation — Automation Without the Hassle?
Automate boring and repetitive processes without touching the underlying systems. A look into the fastest-growing branch of enterprise software. Humanity has always been driven by a desire to make things easier and more comfortable. From the first axe to aqueducts and from web scripts to machine learning algorithms — we always strive to find solutions for repetitive tasks.
Trust Region Policy Optimization (TRPO) Explained
The Reinforcement Learning algorithm TRPO builds upon natural policy gradient algorithms, ensuring updates remain within ‘trustworthy’ policy regions The paper describing OpenAI’s Trust Region Policy Optimization (TRPO) algorithm, authored by Schulman et al. (2015), is foundational in modern Reinforcement Learning. It is rooted in early works on natural policy gradients by Amari (1998) and Kakade (2001), as well as improvements made since.
The Alberta Plan: Sutton’s Research Vision for Artificial Intelligence
For anyone familiar with Reinforcement Learning, it is hard not to know who Richard Sutton is. The Sutton & Barto textbook is considered canonical in the field. I always find it highly inspirational to study the views of genuine thought leaders. Thus, when they present a new research vision, I’m primed to listen.
Sutton's Research Vision for Artificial Intelligence
For anyone familiar with Reinforcement Learning, it is hard not to know who Richard Sutton is. The Sutton & Barto textbook is considered canonical in the field. I always find it highly inspirational to study the views of genuine thought leaders. Thus, when they present a new research vision, I'm primed to listen.
Natural Policy Gradients In Reinforcement Learning Explained
Policy gradient algorithms are at the root of modern Reinforcement Learning. The idea is that, by simply following the gradient (i.e., vector of partial derivatives) of the objective function, we ultimately end up at an optimum. It is a clever way to (i) directly optimize policies (rather than learn indirect value functions) and (ii) let the reward function guide the search. However, policy gradients have fundamental flaws.
Python’s Map(), Filter(), and Reduce() Functions Explained
Demonstrating the building blocks of functional programming in Python with code examples, mathematics, and…emojis At its roots, Python is an object-oriented programming language, built upon an imperative programming paradigm.
Is “Small Data” The Next Big Thing In Data Science?
For the past two decades or so, we have lived in an era of ‘Big Data’. With storage capacity and computational power becoming increasingly cheaper, we could store and process massive amounts of data to generate new insights. Fueled by the successes of Google, Amazon and Facebook, substantial breakthroughs in large-scale data analysis have been made, with data-driven decision-making becoming a top priority for many enterprises.
Avoid This Pitfall When Using LASSO and Ridge Regression
Your regulation penalties might target the wrong variables In linear regression, LASSO and Ridge regularization are commonly applied to combat overfitting and generate more robust models. However, when applied on incorrectly scaled variables, the results can be disastrous. Suppose we have a very simple linear model, in which we predict life expectancy y based on marital status x_1 ∈ {0,1} (1 means married) and annual salary x_2 ∈ R^+ (real nonnegative number).
Seven Exploration Strategies In Reinforcement Learning You Should Know
Pure exploration and -exploitation, ϵ-greedy, Boltzmann exploration, optimistic initialization, confidence intervals, knowledge gradients The exploration-exploitation dilemma is omnipresent in everyday life. Once you found a restaurant you like, you might decide to visit that very same restaurant for the rest of your life, exploiting your positive experience. However, there is a certain appeal in exploring new venues as well.
Five Tips For Writing A Great Data Science Thesis
Write for your reader, not for yourself In this article, I will share some tips on how to improve your Data Science thesis. Over the years, I have supervised my share of Data Science thesis projects, ranging from Big Four firms to local SMEs and from multinational banks to software consultancies.
Structure Your Data Science Project - The Double Diamond Model
The Double Diamond model prescribes divergent and convergent thinking for both the research and design phase. With up to 85% data science projects failing, structured design processes are necessary more than ever. Data Science takes place within the complex intersection of programming, mathematics and business. The proverbial unicorn is a master in all of these areas, yet in practice being a master in one domain and proficient in the others is the best you can hope for.
A Deep Dive into Problem States
Define appropriate problem states for your Markov Decision Process. Learn about higher-order MDPs and belief state variables to boost your decisions. If you ever tried to build a Markov Decision Process (MDP) model, the first step was probably to define the state variable s. After all, the state is used to describe the system, determine feasible actions, compute rewards, and govern time transitions. Without a state, you can’t do very much.
Speed Up Your Simulations By Deploying These Sampling Strategies
Six strategies to sample random variables more efficiently. Learn about antithetic variables, importance sampling, moment matching, control variates, stratified sampling, and quasi-random sequences. Although many real-world systems (and all their uncertainties) can be formulated as a mathematical problems, we often lack the ability to acquire analytical solutions.
Policy Gradients In Reinforcement Learning Explained
Learn all about policy gradient algorithms based on likelihood ratios (REINFORCE): the intuition, the derivation, the ‘log trick’, and update rules for Gaussian and softmax policies. When I first studied policy gradient algorithms, I did not find them particularly easy to fathom.
The Five Building Blocks of Markov Decision Processes
Define and communicate your Reinforcement Learning models by mastering the foundational principles of a Markov Decision Process Photo by Mourizal Zativa on Unsplash The ability to properly formulate a Markov Decision Process (MDP) is imperative for successful Reinforcement Learning (RL) practitioners.
Why Reinforcement Learning Doesn’t Need Bellman’s Equation
Richard Bellman founded dynamic programming and his famous recursive equation while working at RAND Corporation, based in Santa Monica. Photo by Sung Shin on Unsplash In academic circles, it’s often boilerplate to pair a Reinforcement Learning (RL) algorithm with a Markov Decision Process (MDP) formulation and the famed Bellman equation.
Regulate Your Regression Model With Ridge, LASSO and ElasticNet
Photo by Mike Cox on Unsplash Linear models have a wide appeal. Even with a basic understanding of Excel, it is possible to create a model that explains patterns in data. After attaching weights (coefficients) to explanatory variables (features), it is easy to assess the importance of individual variables when explaining the data.
Polynomial Regression - An Alternative For Neural Networks?
Example of polynomial regression performed on a data set, using three different degrees of polynomials. Theoretically, polynomials can achieve infinitely close fits. [image from WikiMedia by Shobha] To start: this is more a discussion piece than anything else. Although I will attempt to compare polynomials and neural networks, there is not a great deal of mathematical rigor or literature support to back up this article.
Precision and Recall - A Comprehensive Guide With Practical Examples
Suppose my job is to develop a credit card fraud detection model for a bank. In reality, let’s say only 0.1% of transactions is fraudulent. I cleverly assume that no transactions are fraudulent, and boast a 99.9% model accuracy (i.e., the % of correct classifications). I pat myself on the back and crack open a cold one. However, my employer not happy. Despite being a highly accurate model, it does not do anything to detect fraud, and as such does not address the underlying business problem.
Bring Your Childhood Drawings To Life Within Seconds- A Demo Of Meta’s Creative AI
With the holidays in close proximity, it’s time for some light-hearted fun with AI. As a child, you probably fantasized about your drawings coming to life — moving, dancing, running… Thanks to Meta AI Research, that no longer needs to remain a fantasy! In the Meta Demo Lab, it takes mere seconds to transform a drawing into a full-fledged animated character. The proceedings are exceedingly simple; it is literally child’s play. Let’s check it out! Draw something Well… no words needed.
Implement Policy Iteration in Python - A Minimal Working Example
A few days ago I wrote an article on value iteration (Richard Bellman, 1957), today it is time for policy iteration (Ronald Howard, 1960). Policy iteration is an exact algorithm to solve Markov Decision Process models, being guaranteed to find an optimal policy. Compared to value iteration, a benefit is having a clear stopping criterion — once the policy is stable, it is provably optimal. However, it often has a higher computational burden for problems with many states.
Zen and the Art of Data Science - How To Avoid Gumption Traps
To me, Robert Pirsig’s bestseller Zen and the Art of Motorcycle Maintenance (1974) was definitely one of the more engaging reads I’ve come across. The aim of this article is not to give a synopsis on the Metaphysics of Quality, but to distill some insights that might aid data scientists when getting discouraged with their project.
Deep Policy Gradient For Cliff Walking
The last article in this Reinforcement Learning series on the cliff walking problem focused on the vanilla policy gradient algorithm (REINFORCE). Although bearing strong similarities, there are some important nuances in the deep learning variant — using a neural network representation — that require attention.
Need Help Making Decisions? Ask Your Digital Twin!
Let’s face it: making decisions is hard. Just consider the following hurdles to overcome: You need to figure out how to actually evaluate decisions. What performance metrics truly matter, how would you rank or weight them? Once you know what constitutes the quality of decisions, you need to estimate how various decisions score on the defined metrics. The consequences of decisions typically reveal themselves only in the future. There is no instant feedback mechanism.
Why Discount Future Rewards In Reinforcement Learning?
Although discount rates are an integral part of Markov decision problems and Reinforcement Learning (RL), we often select γ=0.9 or γ=0.99 without thinking twice. Surely, when asked, we have some intuitions like ‘rewards today are worth more than rewards tomorrow’ or ‘compensating for uncertainty’. When pressed, can you defend why those intuitions hold though? Why do you pick γ=0.8 instead of γ=0.9? Isn’t uncertainty already incorporated in the expected value?
Cliff-Walking Problem With The Discrete Policy Gradient Algorithm
The textbook cliff walking problem continues to fascinate. Although exceedingly simple, it elucidates many interesting aspects of reinforcement learning algorithms. After treating some value-based implementation (SARSA and Q-learning here, Deep Q-learning here), now it is time to move to a policy-based implementation. Although many libraries exist nowadays, we manually implement the procedure in Python to demonstrate the mechanisms behind the discrete policy gradient algorithm.
Cliff-Walking Problem With The Discrete Policy Gradient Algorithm
Cliff-Walking Problem With The Discrete Policy Gradient Algorithm A full implementation of the REINFORCE algorithm in Python. The steps are performed manually to illustrate the inner workings of the procedure. Photo by Jennifer Capel on Unsplash The textbook cliff walking problem continues to fascinate. Although exceedingly simple, it elucidates many…
Data Platforms - Liberate Data From Your Legacy Systems
Stating that data radically transforms the way businesses operate is hardly a revelation. Whether it is high-frequency sensor data, real-time stock market prices or detailed user logs — we track, collect and store data at a scale unprecedented in history. The reason: these vast piles of data hide value. Data-driven decision-making is not just a buzz word — quality data truly allows taking evidence-backed actions that propel businesses forwards.
Five Things to Consider for Reinforcement Learning in Business Environments
Even those completely unfamiliar with Reinforcement Learning (RL) probably know how DeepMind’s AlphaGo managed to beat world champion Lee Sedol in the ancient board game Go. For many, RL has become synonymous with playing games, be it chess, Super Mario, or simply crossing a frozen lake. Games are fun, intuitive, unambiguously defined, and often contain distinct properties that make them challenging to learning algorithms.
Deep Q-Learning for the Cliff Walking Problem
At first glance, moving from vanilla Q-learning to deep Q-learning seems like a minor step. Just replace the lookup table with a neural network and you’re done. There’s more to it than that though — even for the simplest of problems deep Q-learning might struggle to achieve results. To show how several common pitfalls can be avoided, this article treats a TensorFlow 2.0 implementation of deep Q-learning to solve the well-known cliff walking problem.
Three Baseline Policies Your Reinforcement Learning Algorithm Absolutely Should Outperform
Being a subdomain of Machine Learning, Reinforcement Learning (RL) is often likened to a black box. You try a couple of actions, feed the resulting observations into a neural network, and out roll some values — an esoteric policy telling you what to do in any given circumstance. When traversing a frozen lake or playing a videogame, you will see soon enough whether that policy is of any use.
Show More
loading
Actions
Is this you?
As a journalist, you can create a free Muck Rack account to customize your profile, list your contact preferences, and upload a portfolio of your best work.Get in touch with Wouter
Contact Wouter, search articles and posts on X, monitor coverage, and track replies from one place.
Learn more about Muck Rack