1

In Q-learning, am I the one who will define the way in which actions allow the agent to interact with the environment, so that the way in which actions allow the agent to interact with the environment can vary greatly from the problem in question?

For example, in this article: https://www.learndatasci.com/tutorials/reinforcement-q-learning-scratch-python-openai-gym/, which explains Q-learnig, it teaches the Smartcab problem, it only has 6 actions (walk up, down, right and left, pickup and dropoff). The action of moving upwards makes the agent add +1 to Y, advancing its state. In this Smartcab example, the states are positions X and Y representing where the agent is.

But the way in which actions allow the agent to interact with the scenario can be something much broader depending on the problem?, instead of being movement actions (such as walking up, down, right and left), instead of Furthermore, are they more complex actions, which could make the agent change state in a very different way than this Smartcab example?

In Q-learning the way in which the actions will make the agent interact with the environment will depend greatly on the problem in question, so that each problem can have its own rules for the agent to interact with the environment, rules that I myself Can I set it according to my needs?

will The J
  • 267
  • 6

2 Answers2

3

Agent has the free will to choose one action out of its action space (many environments may pose continuously valued infinite action space and state space unlike your referenced simple grid environment) at a timestep according to the agent's current state following some probability distribution during training or even possibly as a final converged optimal stochastic policy $\pi^*$ depending on your problem setup, so clearly each problem has its own rule for defining action space for the agent to interact with the environment, but the mechanism of the action space defining rule itself is not determined by the agent, but totally determined by the environment.

cinch
  • 2,082
  • 1
  • 4
  • 9
1

The action space for a reinforcement learning (RL) task is mostly determined by the problem you are applying RL to. As such, you get to define it in the following ways:

  • By choosing what problem to work on
  • By writing a formal description or code for the environment
  • By encoding a suitable representation of the action space for the agent code to interact with

The second and third items here are constrained by the ones above them. Other than that, RL as a whole is a high level and generalised description of learning by trial and error that applies to a very broad number of cases. Q-learning specifically may not be applicable to all of them, and is one of many solution methods.

Understanding which RL methods are suitable for a specific environment takes a bit of study. One of the factors that affects which methods work best is the nature of the action space. However, if you are just concerned about defining a similar number of discrete actions to TaxiCab, that happen to do something entirely different in a new problem of a similar size, then Q-learning should still apply, and you can absolutely decide on a different set of actions with entirely different consequences to the environment.

Neil Slater
  • 32,068
  • 3
  • 43
  • 64