[2022-11-18]RL with Causal Reasoning

Generalizing Goal-Conditioned Reinforcement Learning with Variational Causal Reasoning

---

Preliminary

• $\mathcal{M}=(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},G)$
• $G\subset \mathcal{S}$ is the goal space
• $r(s,g)=1(s=g)\in \mathcal{R}$ is the sparse deterministic reward function

通过在不同goal的分布$p_{train}(g)$和$p_{test}(g)$上采样来生成不同任务，进而探索goal-conditioned generalization problem

Causal Reasoning with Graphical Models

• random variables $\mathbf{X}=\left(X_1,\dots ,X_d\right)$ with index set $\mathbf{V}:=\{1,\dots ,d\}$
• A graph $\mathcal{G}=(\mathbf{V},\mathcal{E})$ consists of nodes $\mathbf{V}$ and edges $\mathcal{E}\subset {\mathbf{V}}^2$
• A node $i$ is called a parent of $j$ if $e_{ij}\in \mathcal{E}$ and $e_{ji}\notin \mathcal{E}$ . The set of parents of $j$ is denoted by $\mathbf{PA}{}_j^{\mathcal{G}}$.

---

GCRL as Latent Variable Models

从probabilistic inference的角度, 目标是解决likelihood maximization problem for $p(\tau |s^{\ast })$ with $s^{\ast }:=1\left(g=s^T\right)$. 将graph $\mathcal{G}$ 作为latent variable，可以将$p(\tau |s^{\ast })$分解后得到ELBO：

---

$\log p(s^0)$和$\log p(g)$是常数（uniform distribution），因此maximize $p(\tau |s^{\ast })$ 可以转换成objective：

Intuition：为了解上述优化问题，需要交替更新$\varphi$ (causal discovery)和$\theta$ (model and policy learning)

---

Model learning

We propose to model the transition corresponding to G with a collection of neural networks $f_{\theta }(\mathcal{G}):={\left\{f_{{\theta }_j}\right\}}_{j=1}^M$ to obtain

$$s_j^{t+1}=f_{{\theta }_j}\left({\left[\mathbf{PA}{}_j^{\mathcal{G}}\right]}^t,U_j\right)$$

• ${\left[\mathbf{PA}{}_j^{\mathcal{G}}\right]}^t$ represents the values of all parents of node $s_j^t$ at time step $t$
• $U_j$ follows Gaussian noise $U_j\sim \mathcal{N}(0,\mathbf{I})$

Policy learning with planning

• $Q\left(s^t,a^t\right)=\mathbb{E}\left[{\sum }_{t^{\prime }=0}^H{\gamma }^{t^{\prime }}r\left(s^{t^{\prime }+t},a^{t^{\prime }+t}\right)\mid s^t,a^t\right]$
• MPC (random shooting): $\hat{\pi }\left(s^t\right)=\arg {\max }_{a^t\in \mathcal{A}}{Q}_{\theta }^{\mathcal{G}}\left(s^t,a^t\right)$

Data-Efficient Causal Discovery

---

此时causal discovery得到简化：${\max }_{\varphi }\mathcal{J}(\theta ,\varphi )$

• restrict the posterior $q_{\varphi }(\mathcal{G}\mid \tau )$ to point mass distribution and use a threshold $\eta$ to control the sparsity.
• perform the discovery process from the classification perspective by proposing binary classifiers $q_{\varphi }\left(e_{ij}\mid \tau ,\eta \right)$ to determine the existence of an edge $e_{ij}$ .
• $\eta$ is the threshold for the p-value of the hypothesis. A larger $\eta$ corresponds to harder sparsity constraints, leading to a sparse $\mathcal{G}$ since two nodes are more likely to be considered independent.

---

---

According to the definition 3, we only need to conduct classification to edges connecting nodes between $\mathcal{U}$ and $\mathcal{V}$. If two nodes are dependent, we add one edge directed from the node in $\mathcal{U}$ to the node in $\mathcal{V}$.

---

Analysis of Performance Guarantee

• causal graph越好，model learning效果越好

---

• model learning效果越好，value function越接近optimal

---

• 想要控制bound，需要更好的policy（因此需要交替进行model learning和policy learning）

---

Experiments

{width=50%}

---

---

---

---

---

Summary & Thoughts

• 通过学习causal transition model来提升generality
• 结合causality相关的理论可以带来更好的可解释性？
• 是对causal graph的显式估计，训练难度大
• offline效果差但是更实际。优化offline？

---

$$s_j^{t+1}=f_{{\theta }_j}\left({\left[\mathbf{PA}{}_j^{\mathcal{G}}\right]}^t,U_j\right)$$

---

$$P(s^{\prime }|s,a;z)\text{\&}\pi (a|s;z)$$ $$z\in \mathcal{Z}\doteq \{z_1,z_2,\dots ,z_n\}$$

Problem: 不存在关系 $p(z_{t+1}|z_t)$, how to learn $z$?

---

• 如果$p(z_{t+1}|z_t)$不存在或未知，无法显式预测得到$z$
• 如果隐式encode $z$，和其他方法没有大的区别
• 需要增加额外信息才能不依赖$p(z_{t+1}|z_t)$得到$z$？e.g., 增加assumption: $z$ 与current information存在关联

$$P(s^{\prime }|s,a;\mathcal{G}\left[(s,a,\dots )\to z\right])$$

• 但是$\mathcal{G}\left[(s_t,a_t,\dots )\to z_t\right]$与$\mathcal{G}\left[(s_{t+1},a_{t+1},\dots )\to z_{t+1}\right]$之间会间接因为$(s_{t+1}|s_t,a_t)$的关系产生$(z_{t+1}|z_t)$的结构关联？
• 做一些验证
• 考虑其他角度