An introduction to Q-Learning: Reinforcement Learning (Transitioning to Q-Learning)

Transitioning to Q-Learning

By now, we have got the following equation which gives us a value of going to a particular state (form now on, we will refer to the rooms as states) taking the stochasticity of the environment into the account:
V(s)=maxa(R(s,a)+γ∑s′P(s,a,s′)V(s′))
We have also learned very briefly about the idea of living penalty which deals with associating each move of the robot with a reward.
Q-Learning poses an idea of assessing the quality of an action that is taken to move to a state rather than determining the possible value of the state (value footprint) being moved to.
Earlier, we had:
An environment with an agent (with possible value footprints)
If we incorporate the idea of assessing the quality of actions for moving to a certain state s′.
An environment with an agent (with quality of actions)
The robot now has four different states to choose from and along with that, there are four different actions also for the current state it is in. So how do we calculate Q(s, a) i.e. the cumulative quality of the possible actions the robot might take? Let’s break down.
From this equationV(s)=maxa(R(s,a)+γ∑s′P(s,a,s′)V(s′)), if we discard the max() function, we get:
R(s,a)+γ∑s′(P(s,a,s′)V(s′))
Essentially, in the equation that produces V(s), we are considering all possible actions and all possible states (from the current state the robot is in) and then we are taking the maximum value caused by taking a certain action. The above equation produces a value footprint is for just one possible action. In fact, we can think of it as the quality of the action:
Q(s,a)=R(s,a)+γ∑s′(P(s,a,s′)V(s′))
Now that we have got an equation to quantify the quality of a particular action we are going to make a little adjustment in the above equation. We can now say that V(s) is the maximum of all the possible values of Q(s, a). Let’s utilize this fact and replace V(s′) as a function of Q():
Q(s,a)=R(s,a)+γ∑s′(P(s,a,s′)maxa′Q(s′,a′))
But why would we do that?
To ease our calculations. Because now, we have only one function Q() (which is also at the core of the dynamic programming paradigm) to calculate and R(s, a) is a quantified metric which produces rewards of moving to a certain state. The qualities of the actions are called Q-values. And from now on, we will refer the value footprints as the Q-values.
We now have the last piece of the puzzle remaining i.e. temporal difference before we jump to the implementation part and we are going to study that in the next section.

Comments

Post a Comment

Popular posts from this blog

Maxpooling vs minpooling vs average pooling

Image
Pooling is performed in neural networks to reduce variance and computation complexity. Many times, beginners blindly use a pooling method without knowing the reason for using it. Here is a comparison of three basic pooling methods that are widely used. The three types of pooling operations are: Max pooling: The maximum pixel value of the batch is selected. Min pooling: The minimum pixel value of the batch is selected. Average pooling: The average value of all the pixels in the batch is selected. The batch here means a group of pixels of size equal to the filter size which is decided based on the size of the image. In the following example, a filter of 9x9 is chosen. The output of the pooling method varies with the varying value of the filter size. The operations are illustrated through the following figures. Average, Max and Min pooling of size 9x9 applied on an image We cannot say that a particular pooling method is better over others generally. The choice of pooling operation is made...
Read more

Generative AI - Prompting with purpose: The RACE framework for data analysis

Image
  Prompting with purpose: The RACE framework for data analysis We've already established that the quality of your prompts directly influences the quality of your AI-generated outputs. A well-crafted prompt acts as a guiding light, leading the AI toward insightful and actionable results. But how can we consistently achieve this level of clarity and precision in our prompts? Let’s look into the RACE framework—an invaluable tool to elevate your prompts, ensuring they are clear, insightful, and evidence-based. This framework offers a structured approach to prompt engineering, helping you break down complex tasks into manageable steps and communicate your intentions effectively to the AI. This reading will assist you with developing a powerful methodology with generative AI for your data analysis tasks, leading to more accurate, relevant, and impactful results. What is the RACE framework? RACE is an acronym that stands for Research, Answer, Cite, and Expand. It serves as a valuable fram...
Read more

Best Practices for Storing and Loading JSON Objects from a Large SQL Server Table Using .NET Core

 Best Practices for Storing and Loading JSON Objects from a Large SQL Server Table Using .NET Core Introduction JSON (JavaScript Object Notation) is widely used for storing structured data in SQL Server. However, when dealing with large tables, inefficient handling of JSON data can lead to performance bottlenecks, high storage costs, and slow queries. In this article, we will cover best practices for saving and loading JSON objects from large SQL Server tables using .NET Core. We will focus on storage strategies, indexing, efficient querying, and optimized data retrieval using Entity Framework Core (EF Core) and Dapper. --- 1. Choosing the Right Storage Strategy SQL Server supports JSON natively, but choosing the correct storage method depends on your use case. Option 1: Store JSON as NVARCHAR(MAX) (Best for Flexibility & Simplicity) ✔ Ideal for semi-structured or frequently changing data. ✔ Suitable when you need to store the entire JSON object but rarely query individual fiel...
Read more