Log of Inspiration and Retrospection

TOG: Vol 40, No 4 SIG21에서 주목도 높은 연구 세션들 (순서 의미없음): Neural solver for blah blah Video, portrait, image editing (sometimes with GANs) Neural, real-time, differentiable rendering Rendering theory Denoising and guiding Human, face, character animation and control (learning) Surfaces, curves, ribbons, and wires Appearance Deformation Fabrication Computational photography SIG21에서 주목도 높은 키워드들 (순서 의미없음): Physical simulation Neural networks Ray tracing Shape modeling Animation Mesh geometry models 

This post covers not only  Lecture 10: Classic Games (davidsilver.uk)  but also: Best response - Wikipedia Nash equilibrium - Wikipedia Minimax - Wikipedia Minimax and Monte Carlo Tree Search - Philipp Muens KnightCap: A chess program that learns by combining TD(λ) with game-tree search Bootstrapping from Game Tree Search The entire course materials can be found on the following page:  Teaching - David Silver . Lecture Goal 체스나 바둑 등 고전적인 대전 게임을 푸는 minimax search 기법을 배운다. 또한 minimax search를 reinforcement learning에 녹여 통합적으로 푸는 방법을 배운다. Game Theory Best response In game theory, the best response is the strategy (or strategies) that produces the most favorable outcome for a player, taking other players' strategies as given. Assume that other players become parts of the environment . Then, the game is reduced to an (single-agent, stationary-environment) MDP for \(i\)-th agent. Best response is the optimal policy for the MDP . Therefore, the best response is a solution to a si...

This post covers not only  Lecture 9: Exploration and Exploitation (davidsilver.uk)  but also : 'Reinforcement Learning' 카테고리의 글 목록 (yjjo) The Multi-Armed Bandit Problem and Its Solutions (lilianweng.github.io) Bayesian Bandits - Thompson Sampling (tistory.com) The entire course materials can be found on the following page:  Teaching - David Silver . Lecture Goal MDP를 풀 때 exploration과 exploitation 중 어느 한 쪽만 사용하면 optimality에 도달하지 못한다 (trade-off). 본 강의에서는 둘 사이의 균형을 맞추는 알고리즘을 소개한다. Credit: UC Berkeley Intro to AI course lecture 11 Takeaway Use (Bayes-) UCB algorithms Exploration vs. Exploitation Dilemma Exploitation: make the best decision given current information E.g., go to your favorite restaurant Exploration: gather more information E.g., try a new restaurant Dilemma The best long-term strategy may involve short-term sacrifices and risks How does an agent obtain enough information to make the best overall decisions with the lowest sacrifices? [In-A-Nutshell] Key Princip...

This post covers  Lecture 8: Integrating Learning and Planning (davidsilver.uk) : The entire course materials can be found on the following page:  Teaching - David Silver . Lecture Goal Opening remarks: 지금껏 known MDP의 경우, model을 알고 있으므로 dynamic programming등 planning 기법을 이용하여 최적의 value function이나 policy를 구할 수 있었다 (이를 MDP를 푼다 고 표현한다).  지금껏 unknown MDP의 경우, environment와의 상호작용(experience)을 통해 최적의 value/policy를 학습하였다 ( learning ).  Unknown MDP 를 풀 때도 experience를 통해, policy나 value만 학습하는 게 아니라,  model (i.e., expected rewards and transition probability) 자체를 학습 한다면 planning 기법을 사용하여 좀 더 효율적으로 MDP를 풀 수 있지 않을까?  본 강의에서는 unknown MDP를 풀기 위해 learning과 planning을 통합적으로 이용하는 model-based reinforcement learning 방법을 배운다. Learning, simulation & planning, Dyna-Q, Monte-Carlo tree search Credit:  Field Report: Applying Monte Carlo Tree Search for Program Synthesis Model Given an MDP \((S,A,T,R)\), a model \(M=(T,R)\) represents state transitions and expected rewards: \(...

This post covers not only  Lecture 7: Policy Gradient (davidsilver.uk)  but also: Policy Gradient · Fundamental of Reinforcement Learning (gitbooks.io)  (모두의 연구소) Policy Gradient Algorithms (lilianweng.github.io) Exponential family - Wikipedia Control variates - Wikipedia Deterministic Policy Gradient Algorithms (mlr.press) The entire course materials can be found on the following page:  Teaching - David Silver . Lecture Goal Learn policy gradient and actor-critic method *이번 정리 노트는 David Silver 오리지널 강의 노트와 순서 많이 다름 Why Learn Policy?: Policy-Based Reinforcement Learning In the last lecture (value function approximation), we approximated the state-/action-value function using model parameters. E.g., linear model, non-linear neural network, etc. A policy is implicitly generated and updated from the value function. E.g., greedy policy improvement Policy-based RL Directly parametrize the policy using parameters \(\pi_{\theta}(s,a)= \mathbb{P}[a|s,\theta]\)w Note: 이전...