Using unavailable data at prediction time and mixing magic numbers with real numbers Welcome back to another edition in this series on easily missed mistakes in machine learning workflows! For those who haven’t read the first one, this is part of a series that focuses predominantly on procedural errors that may not always be very obvious but have a very high potential of deteriorating model performance if they do end up slipping into our development pipeline.
Thomas A Dorfer
As seen in:
Illumination,
Medium,
The Startup,
Flipboard,
Towards Data Science,
Die Presse,
KDnuggets,
Meinbezirk
Is this you? As a journalist, you can create a free Muck Rack account to customize your profile, list your contact preferences, and upload a portfolio of your best work.
Claim your profile
Articles by Thomas A Dorfer
Avoid These Easily Missed Mistakes in Machine Learning Workflows - Part 1
Misusing identifiers, incorrect data splits, and ignoring rare feature values One greatly enjoyable thing about having been involved in the domain of machine learning for as long as I have is the opportunity to always learn something new.
Visualizing XGBoost Parameters: A Data Scientist’s Guide To Better Models
Why understanding parameters is critical for building robust models After leaving neuroscience behind and commencing on a data science path a number of years ago, I’ve had the privilege of working on numerous real-world machine learning projects. One thing that stands out across industries and use cases — whether it’s predicting customer churn, forecasting sales, or optimizing supply chains — is how often XGBoost dominates when working with tabular data.
Ten Years of AI in Review
Image by the Author. The last decade has been a thrilling and eventful ride for the field of artificial intelligence (AI). Modest explorations of the potential of deep learning turned into an explosive proliferation of a field that now includes everything from recommender systems in e-commerce to object detection for autonomous vehicles and generative models that can create everything from realistic images to coherent text.
Ten Years of AI in Review
1 day ago Developed by a firm that works closely with Ikea, this flat-pack couch upends conventional design. The couch is the most comfortable seat in most homes, but it’s also the most uncomfortable seat to get into your house. Heavy, bulky, and requiring a team of geometry PhD students to navigate up a …
Thomas A Dorfer - KDnuggets
Thomas A Dorfer Thomas A Dorfer is a Data & Applied Scientist at Microsoft. Prior to his current role, he worked as a data scientist in the biotech industry and as a researcher in the domain of neurofeedback. He holds a Master's degree in Integrative Neuroscience and, in his spare time, also writes technical blog posts on Medium on the subjects of data science, machine learning, and AI.
Ten Years of AI in Review
From image classification to chatbot therapy The last decade has been a thrilling and eventful ride for the field of artificial intelligence (AI). Modest explorations of the potential of deep learning turned into an explosive proliferation of a field that now includes everything from recommender systems in e-commerce to object detection for autonomous vehicles and generative models that can create everything from realistic images to coherent text.
Ten Years of AI in Review
The last decade has been a thrilling and eventful ride for the field of artificial intelligence (AI). Modest explorations of the potential of deep learning turned into an explosive proliferation of a field that now includes everything from recommender systems in e-commerce to object detection for autonomous vehicles and generative models that can create everything from realistic images to coherent text.
YOLO-NAS: How to Achieve the Best Performance on Object Detection Tasks
A foundational model generated through neural architecture search, innovative quantization blocks, and a robust pre-training paradigm In the domain of object detection, YOLO (You Only Look Once) has become a household name. Since the release of the first model in 2015, the YOLO family has been growing steadily, with each new model outperforming its predecessor in mean average precision (mAP) and inference latency.
How to Achieve the Best Performance on Object Detection Tasks
In the domain of object detection, YOLO ( You Only Look Once) has become a household name. Since the release of the first model in 2015, the YOLO family has been growing steadily, with each new model outperforming its predecessor in mean average precision (mAP) and inference latency. Two weeks ago, the YOLO family has welcomed yet another member: YOLO-NAS, a novel and foundational model developed by the deep learning company Deci.
Quick and Easy Time-Series Forecasting in PowerBI: A Practical Guide
How to set up and configure a forecasting system in just a few clicks Time-series forecasting has become a ubiquitous tool for businesses, governments, and individuals alike. It is nowadays being applied in almost every domain imaginable: in finance to predict stocks and interest rates, in healthcare to predict hospital bed capacity, in transportation to plan routes and traffic patterns, in energy to predict power supply and demand, and the list goes on.
A Practical Guide
Time-series forecasting has become a ubiquitous tool for businesses, governments, and individuals alike. It is nowadays being applied in almost every domain imaginable: in finance to predict stocks and interest rates, in healthcare to predict hospital bed capacity, in transportation to plan routes and traffic patterns, in energy to predict power supply and demand, and the list goes on.
How to Choose the Best Evaluation Metric for Regression Problems
A comprehensive guide covering the most commonly used evaluation metrics used in regression and their utility in different scenarios Before building a regression model, it’s worth taking a moment to carefully think about how to evaluate it. A variety of factors will fall into that decision, including whether or not large errors should be punished more than small ones, or how comprehensible and intuitive the metric needs to be for stakeholders.
How to Choose the Best Evaluation Metric for Regression Problems
Before building a regression model, it's worth taking a moment to carefully think about how to evaluate it. A variety of factors will fall into that decision, including whether or not large errors should be punished more than small ones, or how comprehensible and intuitive the metric needs to be for stakeholders. This article will cover the most commonly used evaluation metrics for regression problems.
How to Choose the Best Evaluation Metric for Classification Problems
A comprehensive guide covering the most commonly used evaluation metrics for supervised classification and their utility in different scenarios In order to properly evaluate a classification model, it is important to carefully consider which evaluation metric is the most suitable. This article will cover the most commonly used evaluation metrics for classification tasks, including relevant example cases, and will provide you with the information necessary to help you choose among them.
How to Choose the Best Evaluation Metric for Classification Problems
In order to properly evaluate a classification model, it is important to carefully consider which evaluation metric is the most suitable. This article will cover the most commonly used evaluation metrics for classification tasks, including relevant example cases, and will provide you with the information necessary to help you choose among them. A classification problem is characterized by the prediction of the category or class of a given observation based on its corresponding features.
Auto-Sklearn: How To Boost Performance and Efficiency Through Automated Machine Learning
Learn how to leverage AutoML to maximize the outcome of your machine learning workflows Many of us are familiar with the challenge of selecting a suitable machine learning model for a specific prediction task, given the vast number of models to choose from. On top of that, we also need to find optimal hyperparameters in order to maximize our model’s performance. These challenges can largely be overcome through automated machine learning, or AutoML.
Advanced Time-Series Anomaly Detection with Deep Learning in PowerBI
Introduction With an ever-increasing number of applications and services deployed worldwide, time-series anomaly detection has become a ubiquitous and indispensable tool for capturing metric regressions. However, setting up an anomaly detection system is certainly not straightforward and often requires a fair amount of domain expertise. In PowerBI, the set-up can be done in just a few clicks, enabling the implementation of a state-of-the-art anomaly detection system in no time.
How To Use Argument Parsing for Greater Efficiency in Machine Learning Workflows
A complete guide on how to use argparse for your command line applications and how it can be leveraged for machine learning projects If you’ve spent some time roaming around in the world of data science or software engineering, you have most likely come across some applications that require you to use your command-line interface, or CLI. Common examples include Azure CLI for managing Azure resources or Git for version control and source code management.
How To Use Argument Parsing for Greater Efficiency in Machine Learning Workflows
If you've spent some time roaming around in the world of data science or software engineering, you have most likely come across some applications that require you to use your command-line interface, or CLI. Common examples include Azure CLI for managing Azure resources or Git for version control and source code management. The same type of functionality and program interactivity can be achieved with your own custom Python application.
Enhanced Object Detection: How To Effectively Implement YOLOv8
A practical guide to object detection in images, videos, and real-time webcam feed using both CLI and Python Object detection, a subfield of computer vision, is primarily concerned with the identification and localization of objects in images or videos with a certain degree of confidence. An identified object is generally annotated with a bounding box, which provides information to the viewer about the object’s nature and location in the scene.
How To Effectively Implement YOLOv8
Object detection, a subfield of computer vision, is primarily concerned with the identification and localization of objects in images or videos with a certain degree of confidence. An identified object is generally annotated with a bounding box, which provides information to the viewer about the object's nature and location in the scene.
Comparing List Comprehensions vs. Built-In Functions in Python: Which Is Better?
An in-depth analysis of syntax, readability, and performance Have you ever found yourself scrolling through Netflix on a rainy day, feeling overwhelmed by the endless choices of movies and shows to watch? In programming, the paradox of choice can be just as overwhelming. With so many libraries and frameworks available, providing countless different ways of achieving the same goal, it’s easy to get lost in a sea of options.
Which Is Better?
Have you ever found yourself scrolling through Netflix on a rainy day, feeling overwhelmed by the endless choices of movies and shows to watch? In programming, the paradox of choice can be just as overwhelming. With so many libraries and frameworks available, providing countless different ways of achieving the same goal, it's easy to get lost in a sea of options.
Gegrillter Störbauch und Stosuppe
Ein Ausschnitt aus einem hochkomplexen Gericht rund um den Stör. Ein Rezept von Thomas Dorfer ist selbst für ambitionierte Hobbyköche eine Herausforderung. Hier ein Ausschnitt aus seinem Gericht rund um den gegrillten Störbauch. Fisch, Stosuppe und Röstzwiebel-Teriyaki serviert er mit im Wurzelsud gekochter Kalbszunge, in feinen Streifen und zu Röllchen gedreht, mit Rahmrettich und einer Kartoffel-Sauerrahmmousseline. Stosuppe.
Enhanced Debugging in Python: Tracebacks Just Got a Major Upgrade
How location-enriched tracebacks facilitate the debugging experience in Python 3.11 In Python, a traceback is a report that is displayed when an exception occurs, along with a (hopefully) helpful error message that can guide the user to the problem. Up until now, tracebacks only showed the line in which the exception occurred, but did not provide any location-specific information about which part of that line is affected.
Tracebacks Just Got a Major Upgrade
In Python, a traceback is a report that is displayed when an exception occurs, along with a (hopefully) helpful error message that can guide the user to the problem. Up until now, tracebacks only showed the line in which the exception occurred, but did not provide any location-specific information about which part of that line is affected. To illustrate this, let's assume we have a dictionary object containing some summary statistics about some random data.
Effective Data Visualization: 9 Valuable Tips to Increase the Quality of Your Charts
How minor modifications can make a major difference to the communicative power of your visualizations Look at that poor guy above. He’s been grinding for days to extract some meaningful insights from his data only to now fail to effectively communicate them to his superiors. While the deadline for his big presentation is only hours away and closing in fast, he remains unaware that the charts he prepared are more confusing than insightful.
9 Valuable Tips to Increase the Quality of Your Charts
Look at that poor guy above. He's been grinding for days to extract some meaningful insights from his data only to now fail to effectively communicate them to his superiors. While the deadline for his big presentation is only hours away and closing in fast, he remains unaware that the charts he prepared are more confusing than insightful. The pie chart was a poor choice as it requires the reader to estimate angles, which we humans are really bad at.
How to Effectively Use Lambda Functions in Python as a Data Scientist
An introduction to their syntax, capabilities, and applicability in data science Lambda functions in Python are small, anonymous (or nameless) functions characterized by a more concise syntax compared to regular Python functions. Within the realm of data science, lambda functions are frequently used in conjunction with higher-order functions. These are functions that take one or more functions as arguments, or return a function as their result. Common examples include map(), filter(), and reduce().
How to Effectively Use Lambda Functions in Python as a Data Scientist
Lambda functions in Python are small, anonymous (or nameless) functions characterized by a more concise syntax compared to regular Python functions. Within the realm of data science, lambda functions are frequently used in conjunction with higher-order functions. These are functions that take one or more functions as arguments, or return a function as their result. Common examples include map(), filter(), and reduce().
Data Science in Small and Big Companies
Data science has become an indispensable part for businesses and has penetrated into almost every domain, from e-commerce to autonomous driving and healthcare. Within each domain, data science teams are working hard to produce valuable, data-driven insights that can subsequently be used to make informed business decisions. The underlying process that governs the derivation of those insights, however, can vary extensively.
Why Simple Models Are Often Better Original
The significance of Occam’s Razor in data science and machine learning In data science and machine learning, simplicity is an important concept that can have significant impact on model characteristics such as performance and interpretability. Over-engineered solutions tend to adversely affect these characteristics by increasing the likelihood of overfitting, decreasing computational efficiency, and lowering the transparency of the model’s output.
Can ChatGPT Explain Code?
An analysis of its ability to understand and explain code using Python functions from low to high complexity Since OpenAI launched ChatGPT in November last year, its potential applications have been explored extensively. Within the tech community, there has been a growing interest in evaluating whether ChatGPT can assist with coding or even outperform its human counterparts.
How To Stay on Top of the Latest AI Research
How To Stay on Top of the Latest AI Research Strategies on how to keep up with a rapidly advancing field Photo by Dmitry Ratushny on Unsplash Artificial Intelligence (AI) is a disruptive and fast-moving field whose developmental trajectory is accelerating rapidly. In fact, the number of publications in this space has been rising dramatically in recent years. Stanford’s annual Artificial Intelligence Index Report shows that the…
The Utility of Regular Expressions in Data Science
An illustration of common applications with Python Time and again, data scientists find themselves in a situation where they have to determine whether particular fields in their data conform to the required text format or whether a particular string is present or absent. In other situations, they may have to replace a particular string in their data with another. To achieve this, they utilize what has become the norm for these types of problems: regular expressions.
The Value of Diversity in Tech
The Value of Diversity in Tech Why diverse teams will boost the joint intellectual potential of your organization Photo by sharonmccutcheon on Unsplash Introduction Information Technology is a collaborative sector that caters to billions of diverse customers scattered across the globe. Yet, the products and services created by this sector are conceived by relatively homogeneous teams. This…
The Value of Diversity in Tech
Information Technology is a collaborative sector that caters to billions of diverse customers scattered across the globe. Yet, the products and services created by this sector are conceived by relatively homogeneous teams. This is true in particular for the domains of software development, software engineering, and data science.
The Expressive Power of the Scatter Plot
The Expressive Power of the Scatter Plot How to Visualize Multidimensional Data in 2D Photo by roguewild on Unsplash Introduction Data visualization is an important aspect of data science that enables its practitioner to obtain a graphical representation of the data at hand, detect anomalies, or recognize patterns and trends. While there are multiple ways of performing graphical data analysis, the…
The Expressive Power of the Scatter Plot
Data visualization is an important aspect of data science that enables its practitioner to obtain a graphical representation of the data at hand, detect anomalies, or recognize patterns and trends. While there are multiple ways of performing graphical data analysis, the scatter plot is a widely known and frequently used tool to visualize relationships among variables.
The Case Against the Pie Chart
Why its use is almost never justified The visualization of quantitative data through charts and graphs has the purpose of making the data easier to understand and to derive valuable insights from it. Pie charts, however, tend to do the opposite and obscure the underlying quantities of the data presented. This is mainly because humans are much better at interpreting lengths and heights rather than estimating quantities from angles, which is the skill needed to read a pie chart.
The Case Against the Pie Chart
The visualization of quantitative data through charts and graphs has the purpose of making the data easier to understand and to derive valuable insights from it. Pie charts, however, tend to do the opposite and obscure the underlying quantities of the data presented. This is mainly because humans are much better at interpreting lengths and heights rather than estimating quantities from angles, which is the skill needed to read a pie chart.
DBSCAN vs. HDBSCAN
Cluster Analysis is a pertinent domain in data science that enables the grouping of similar objects into distinct subgroups. While there are different families of clustering algorithms, the most widely known is K-Means. This is a centroid-based algorithm, meaning that objects in the data are clustered by being assigned to the nearest centroid. However, a major pitfall of K-Means is its lack of detecting outliers, or noisy data points, which leads them to be classified incorrectly.
How to Evaluate Clustering Performance without Ground Truth Labels
An illustrative guide to choosing an appropriate validity metric based on cluster conformation There are plenty of methods available from the supervised learning playbook to evaluate clustering performance for situations where the ground truth labels are known. However, performance evaluation is not as straightforward when the ground truth is not available. In this case, we need to rely on validity metrics that give us an indication of how well clusters are defined.
How to Evaluate Clustering Performance without Ground Truth Labels
There are plenty of methods available from the supervised learning playbook to evaluate clustering performance for situations where the ground truth labels are known. However, performance evaluation is not as straightforward when the ground truth is not available. In this case, we need to rely on validity metrics that give us an indication of how well clusters are defined. These validity metrics rely heavily on the underlying conformation of the clusters in the data.
On Tyranny: A Brief Summary
By Thomas A Dorfer A reminder of our duties as democratic citizens Studying European history of the twentieth century can be a somber task. During the 1920s and '30s, democracies across the continent were dismantled by tyrants and collapsed into fascism and right-wing authoritarianism. Not long thereafter, in the 1940s, communism took over Eastern Europe.
Clustering on Mixed Data Types
Utilizing Gower dissimilarity and HDBSCAN Clustering is an unsupervised machine learning technique which aims to group similar data points into distinct subgroups. Typically, the distance metric used for this grouping is Euclidean distance for numerical data and Jaccard distance for categorical data. The majority of clustering algorithms are also designed explicitly for either numerical or categorical data, but not both.
Spittal: autArK begleitet Menschen in besonderen Lebenssituationen. Ein Interview. SPITTAL/DRAU. Seit 25 Jahren begleitet autArK Menschen in besonderen Lebenssituationen auf ihrem Weg zur größtmöglichen beruflichen und persönlichen Eigenständigkeit. In Spittal hat Elke Setz die Projektleitung Jugendcoaching über. Ein Interview. Wofür steht autArK? Es ist normal verschieden zu sein! Das ist unser Leitsatz und ein Zitat Richard von Weizsäckers. Der Satz sagt schon viel aus.
Neue Förderungen: Ölkesselfreie Stadt Spittal
Spittal: Umstieg auf erneuerbare Energie wird wieder gefördert. SPITTAL. Das Programm „Ölkesselfreie Gemeinden und Städte“ läuft auch in der Stadt Spittal sehr erfolgreich. Die Aktion geht in der Lieserstadt nuh bereits in Runde 2. In Förderperiode 1, die von der Landes-Umweltabteilung mit insgesamt 40.000 Euro unterstützt wurde, wurden knapp 30 Förderanträge positiv abgewickelt.
Sport/Fußball: Morgensterne wecken Interesse
Sandro Morgenstern (15) ist der neue Elfmeterspezialist des FC Lendorf. Austria Klagenfurt klopfte an, auch bei Cousin Michael (16). Der FC Lendorf hat mit Sandro Morgenstern einen neuen Elfmeterspezialisten. Der 15-Jährige versenkte die letzten beiden Strafstöße vom Punkt eiskalt.
Show More
loading
Actions
Is this you?
As a journalist, you can create a free Muck Rack account to customize your profile, list your contact preferences, and upload a portfolio of your best work.Get in touch with Thomas A
Contact Thomas A, search articles and posts on X, monitor coverage, and track replies from one place.
Learn more about Muck Rack