data science
advanced
10 sample questions
Reinforcement Learning MCQ Practice Test
Study of decision-making processes where agents learn optimal behaviors through trial and error.
Q1. In a deep Q-network (DQN), what is the primary purpose of the target network?
-
A. To store the current policy
-
B. To store the current Q-values
-
C. To stabilize the Q-value updates by using a frozen network lagging behind the online network ✓
-
D. To optimize the Q-function using a different architecture
Explanation: The target network in a DQN is used to stabilize the Q-value updates by using a frozen network lagging behind the online network. This helps to reduce the correlation between the Q-value updates and the target Q-values, which can improve the stability and performance of the algorithm.
Q2. In a Q-learning algorithm, what is the primary difference between the épsilon-greedy policy and the softmax policy?
-
A. The épsilon-greedy policy uses a fixed value of épsilon, whereas the softmax policy uses a temperature parameter.
-
B. The épsilon-greedy policy selects the action with the highest Q-value with probability 1, whereas the softmax policy selects the action with the highest Q-value with a probability proportional to its Q-value. ✓
-
C. The épsilon-greedy policy is more suitable for continuous action spaces, whereas the softmax policy is more suitable for discrete action spaces.
-
D. The épsilon-greedy policy is more robust to overestimation of Q-values, whereas the softmax policy is more robust to underestimation of Q-values.
Explanation: The épsilon-greedy policy is a simple, greedy policy that selects the action with the highest Q-value with a probability of 1, while the softmax policy is a more sophisticated policy that selects the action with the highest Q-value with a probability proportional to its Q-value. This allows the softmax policy to explore the action space more efficiently.
Q3. In a Deep Q-Network (DQN), what is the primary purpose of the target network?
-
A. To update the Q-values of the action-value function
-
B. To stabilize the learning process by providing a stable target for the Q-network ✓
-
C. To improve the exploration-exploitation trade-off in the agent's policy
-
D. To reduce the variance of the Q-learning update rule
Explanation: The target network in a DQN is a separate copy of the Q-network that is updated less frequently than the Q-network. Its primary purpose is to provide a stable target for the Q-network to learn from, which helps to stabilize the learning process and improve the overall performance of the agent.
Q4. In Q-learning, what is the primary difference between the épsilon-greedy policy and the soft-max policy?
-
A. The épsilon-greedy policy uses a fixed exploration rate, while the soft-max policy uses a temperature parameter. ✓
-
B. The épsilon-greedy policy uses a random action with probability µ, while the soft-max policy uses a probabilistic action selection based on the policy network.
-
C. The épsilon-greedy policy uses a fixed exploration rate, while the soft-max policy uses a learning rate.
-
D. The épsilon-greedy policy uses a temperature parameter, while the soft-max policy uses a fixed exploration rate.
Explanation: In Q-learning, the épsilon-greedy policy is a simple exploration strategy that chooses the action with the highest Q-value with probability (1 - µ) and a random action with probability µ. In contrast, the soft-max policy uses a temperature parameter to soften the maximum action selection, allowing for more exploration-exploitation trade-off.
Q5. In a Deep Deterministic Policy Gradient (DDPG) algorithm, what is the primary purpose of the target actor network?
-
A. To stabilize the learning process by providing a stable policy ✓
-
B. To compute the target Q-values for the critic network
-
C. To generate exploratory actions by adding noise to the policy
-
D. To store and replay the experience buffer
Explanation: The target actor network in DDPG is used to provide a stable policy by following the target policy, which is a moving average of the current policy. This helps to stabilize the learning process and improve the overall performance of the algorithm.
Q6. In the context of Deep Deterministic Policy Gradient (DDPG), what is the primary purpose of the target actor network?
-
A. To stabilize the training process by providing a stable policy
-
B. To improve the exploration-exploitation trade-off by introducing noise into the policy
-
C. To provide a stable target for the critic network to learn from ✓
-
D. To enable the agent to learn a deterministic policy
Explanation: The target actor network in DDPG is used to provide a stable target for the critic network to learn from, which helps the critic to estimate the expected return of the target policy. This is crucial in DDPG, as the critic needs a stable target to learn from, especially when the policy is changing rapidly.
Q7. In the context of policy gradient methods, what is the primary difference between the REINFORCE algorithm and the Actor-Critic algorithm?
-
A. REINFORCE uses a baseline to reduce the variance of the policy gradient estimator, while Actor-Critic uses a value function to estimate the expected return. ✓
-
B. Actor-Critic uses a separate critic network to estimate the value function, whereas REINFORCE uses the policy network itself as the critic.
-
C. REINFORCE is a model-free algorithm, whereas Actor-Critic is a model-based algorithm.
-
D. Actor-Critic uses a more complex policy network than REINFORCE, which makes it more computationally expensive.
Explanation: In policy gradient methods, REINFORCE and Actor-Critic are two popular algorithms used to optimize the policy of an agent. The primary difference between them lies in their approach to estimating the policy gradient. REINFORCE uses a baseline to reduce the variance of the policy gradient estimator, whereas Actor-Critic uses a value function to estimate the expected return. This difference in approach affects the stability and efficiency of the algorithms.
Q8. In the context of Deep Q-Networks (DQN), what is the primary purpose of the target network?
-
A. To update the policy network with the latest Q-values
-
B. To provide a stable target for the Q-network to learn from ✓
-
C. To select the action with the highest Q-value
-
D. To implement the epsilon-greedy policy
Explanation: In DQN, the target network is a separate neural network that is periodically updated to match the policy network. Its primary purpose is to provide a stable target for the Q-network to learn from, allowing the policy network to improve without being influenced by the target network's fluctuations.
Q9. In the context of Deep Deterministic Policy Gradient (DDPG) algorithm, what is the primary role of the target critic network?
-
A. To estimate the action-value function of the target policy.
-
B. To provide a stable estimation of the action-value function for the actor network. ✓
-
C. To optimize the policy by directly updating the actor network.
-
D. To compute the gradient of the policy loss with respect to the actor network's parameters.
Explanation: In DDPG, the target critic network is used to provide a stable estimation of the action-value function, which is essential for the actor network to learn a stable policy. The target critic network is updated periodically to match the weights of the critic network, and it provides a target value for the actor network to learn from.
Q10. In the context of policy gradient methods, which of the following is a key challenge when training a neural network to learn a stochastic policy?
-
A. Vanishing gradients due to the policy's entropy term
-
B. High variance in the policy gradient estimate due to the stochasticity of the policy ✓
-
C. The policy network's capacity to learn complex policies is limited by the choice of activation function
-
D. The policy network's output is not differentiable with respect to the policy's stochasticity
Explanation: Policy gradient methods learn a stochastic policy by maximizing a surrogate objective function. However, the stochasticity of the policy introduces high variance in the policy gradient estimate, making it challenging to train the policy network. This is because the policy gradient is estimated using the expectation of the policy's output, which can be difficult to approximate accurately.
That was just a sample. Sign up to unlock the full question bank with timed tests and certificates.
Sign Up Free