### 机器学习代写|强化学习project代写reinforence learning代考|Further Comparison

statistics-lab™ 为您的留学生涯保驾护航 在代写强化学习reinforence learning方面已经树立了自己的口碑, 保证靠谱, 高质且原创的统计Statistics代写服务。我们的专家在代写强化学习reinforence learning代写方面经验极为丰富，各种代写强化学习reinforence learning相关的作业也就用不着说。

• Statistical Inference 统计推断
• Statistical Computing 统计计算
• Advanced Probability Theory 高等概率论
• Advanced Mathematical Statistics 高等数理统计学
• (Generalized) Linear Models 广义线性模型
• Statistical Machine Learning 统计机器学习
• Longitudinal Data Analysis 纵向数据分析
• Foundations of Data Science 数据科学基础

## 机器学习代写|强化学习project代写reinforence learning代考|Further Comparison

$\mathrm{TD}(0)$ and RG both perform SGD on the $\overline{\mathrm{BE}}$, with $\mathrm{TD}(0)$ simply ignoring the fact, that the one-step TD error also depends on the parameters $\theta$. When assuming linear function approximation, this comparison can also be shown using objective function formulations. Since $\Pi$ represents a orthogonal projection, the relation
$$|\overline{\mathrm{BE}}(\theta)|^{2}=|\overline{\mathrm{PBE}}(\theta)|^{2}+\left|B_{\pi} \hat{v}{\theta}-\Pi B{\pi} \hat{v}{\theta}\right|^{2}$$ is valid (compare Fig. 1). Since the TD-fix-point is congruent with the fix-point of the $\overline{\mathrm{PBE}}, \mathrm{TD}(0)$ only minimizes the $\overline{\mathrm{PBE}}$ and ignores the term $\left|B{\pi} \hat{v}{\theta}-\Pi B{\pi} \hat{v}_{\theta}\right|^{2}$, that is crucial for guaranteed convergence. In contrast RG minimizes both parts of the $\overline{\mathrm{BE}}$ objective. Furthermore the relation shows, that the $\overline{\mathrm{BE}}$, minimized by $\mathrm{RG}$, is an upper bound for the $\overline{\mathrm{PBE}}$, minimized by TD-learning. So minimizing the $\overline{\mathrm{BE}}$ ensures small TD errors. Since optimizing the $\overline{\mathrm{BE}}$ objective using $\mathrm{RG}$ in a way includes the optimization of the $\overline{\mathrm{PBE}}$ objective done by $\mathrm{TD}(0), \mathrm{RG}$ appears to be more difficult from a numerical point of view [8]. The $\overline{\mathrm{BE}}$ objective also suffers from higher variance in its estimates and is therefore harder to optimize [8].

Assuming linear function approximation Li [6] compared $\mathrm{TD}(0)$ and RG with respect to prediction errors $(\overline{\mathrm{VE}})$. The derived bounds for TD $(0)$ are tighter than those for RG, i.e. performance of $\mathrm{TD}(0)$ seems to result in a smaller $\overline{\mathrm{VE}}$. With respect to RG Scherrer [8] also derived an upper bound for the $\overline{\mathrm{VE}}$ using the $\overline{\mathrm{BE}}$. However Dann, Neumann and Peters [3] observed this bound to be too loose for many MDPs in real applications. Sun and Bagnell [9] managed to tighten the bounds for the prediction error of RG even more, even with less strict assumptions than all previous attempts and even for nonlinear function approximation. Although the bounds for $\mathrm{TD}(0)$ are still tighter than those for RG, Sun and Bagnell [9] find in experiments, that residual gradient methods have the potential to achieve smaller prediction errors than temporal-difference methods. Those results are contradictory to the derived bounds

and to the work of Scherrer [8], that finds, that approximation functions derived using the fix-point of the $\overline{\mathrm{PBE}}$ often achieve a lower $\overline{\mathrm{VE}}$ than functions congruent with the fix-point of the $\overline{\mathrm{BE}}$.

Nevertheless, the main point affecting RG is the double sampling problem. Also the $\overline{\text { TDE }}$ objective, that is optimized by RG when simply sampling just one successor state for each state, has not been investigated much in research [3]. In addition Lagoudakis and Parr [5] found, that policy iteration making use of the $\overline{\text { PBE }}$ objective results in control policies of higher quality. Furthermore, congruent to Baird [1], Scherrer [8] and Dann, Neumann and Peters [3] also find TD $(0)$ to converge much faster than RG. Finally Sutton and Barto [11] question the learnability of the Bellman Error and therefore the $\overline{\mathrm{BE}}$ as an objective in general. Altogether TD $(0)$ seems to be preferable, as long as it does not diverge. In the next section, more recent approaches are stated, which combine the advantages of temporal-differences methods with guaranteed convergence.

## 机器学习代写|强化学习project代写reinforence learning代考|Recent Methods and Approaches

In 2009 Sutton, Maei and Szepesvári [13] introduced a stable off-policy temporaldifference algorithm called gradient temporal-difference (GTD). GTD was the first algorithm achieving guaranteed off-policy convergence and linear complexity in memory and per-time-step computation using temporal differences and linear function approximation. GTD performs SGD on a new objective, called norm of the expected TD update (NEU). When optimizing the NEU objective, there are two estimates $\theta$ and $\omega$ of the parameters of the approximation function. First the approximation value function $\hat{v}{\theta}$ is mapped against the one-step TD estimations of the true values of the states (the targets), which are calculated using $\hat{v}{\omega}$. Second $\hat{v}{\omega}$ is mapped against $\hat{v}{\theta}$. Maintaining two individual approximation functions, one for estimating the targets and one for the actual value function approximation, was also one of two key ideas by Mnih et al. [7] to achieve greater success with deep QLearning. Q-Learning is closely related to the problem of non-linear critic-learning. (The second key idea was the introduction of an experience replay memory.) Like Q-Learning, GTD was also extended by Bhatnagar et al. [2] to non-linear function approximation. As all non-linear optimization approaches, it also suffers from potential failures caused by the non-convexity of the optimization objective. GTD, though achieving a lot desirable properties, still converges much slower than conventional $\mathrm{TD}(0)$. Therefore Sutton et al. [12] introduced two new non-linear approximation algorithms, gradient temporal-difference 2 (GTD2) and linear TD with gradient correction (TDC), which converge both faster than GTD. They both perform SGD directly on the $\overline{\mathrm{PBE}}$ objective and TDC even seems to achieve the same (sometimes even better) convergence speed as $\mathrm{TD}(0)$.

## 机器学习代写|强化学习project代写reinforence learning代考|Conclusion

We have reviewed the fundamental contents to understand critic learning. We explained all basic objective functions and compared Temporal-difference learning and the Residual-Gradient algorithm. Thereby Temporal-difference learning was found to be the preferable choice. Also some more recent approaches based on Temporal-difference learning have been reviewed. Like the Residual-Gradient algorithm those approaches are also stable in the off-policy case, but possess better properties.

Nevertheless several aspects have not been considered in this paper, like other optimization techniques to solve the discussed objective functions (e.g. least-squares or probabilistic approaches), extensions like eligibility-traces and further comparison and investigation concerning to the related topic of Q-Learning (and its achievements like $\mathrm{DQN}$ and dueling networks).

## 机器学习代写|强化学习project代写reinforence learning代考|Further Comparison

|乙和¯(θ)|2=|磷乙和¯(θ)|2+|乙圆周率在^θ−圆周率乙圆周率在^θ|2是有效的（比较图 1）。由于 TD-fix-point 与磷乙和¯,吨D(0)只会最小化磷乙和¯并忽略该术语|乙圆周率在^θ−圆周率乙圆周率在^θ|2，这对于保证收敛至关重要。相比之下，RG 最小化了乙和¯客观的。此外，该关系表明，乙和¯, 最小化RG, 是上界磷乙和¯，通过 TD 学习最小化。所以最小化乙和¯确保小的 TD 误差。由于优化乙和¯客观使用RG在某种程度上包括优化磷乙和¯目标完成吨D(0),RG从数值的角度来看似乎更困难[8]。这乙和¯Objective 的估计也存在较大的方差，因此更难优化 [8]。

Scherrer [8] 的工作发现，使用磷乙和¯经常达到较低的在和¯比与固定点一致的函数乙和¯.

## 机器学习代写|强化学习project代写reinforence learning代考|Recent Methods and Approaches

2009 年，Sutton、Maei 和 Szepesvári [13] 引入了一种稳定的离策略时间差算法，称为梯度时间差 (GTD)。GTD 是第一个使用时间差异和线性函数逼近在内存和每时间步计算中实现有保证的非策略收敛和线性复杂性的算法。GTD 对一个新目标执行 SGD，称为预期 TD 更新 (NEU) 的范数。在优化 NEU 目标时，有两个估计θ和ω的近似函数的参数。首先是近似值函数在^θ映射到状态（目标）的真实值的一步 TD 估计，这些估计是使用在^ω. 第二在^ω映射到在^θ. 维护两个单独的近似函数，一个用于估计目标，一个用于实际值函数近似，这也是 Mnih 等人的两个关键思想之一。[7] 通过深度 QLearning 取得更大的成功。Q-Learning 与非线性批评学习问题密切相关。（第二个关键思想是引入经验回放记忆。）与 Q-Learning 一样，GTD 也被 Bhatnagar 等人扩展。[2] 到非线性函数逼近。与所有非线性优化方法一样，它也存在由优化目标的非凸性引起的潜在故障。GTD 虽然实现了很多理想的特性，但仍然比传统的收敛速度慢得多吨D(0). 因此萨顿等人。[12] 引入了两种新的非线性逼近算法，梯度时间差 2（GTD2）和带梯度校正的线性 TD（TDC），它们的收敛速度都比 GTD 快。他们都直接在磷乙和¯目标和 TDC 甚至似乎达到了相同（有时甚至更好）的收敛速度吨D(0).

## 有限元方法代写

tatistics-lab作为专业的留学生服务机构，多年来已为美国、英国、加拿大、澳洲等留学热门地的学生提供专业的学术服务，包括但不限于Essay代写，Assignment代写，Dissertation代写，Report代写，小组作业代写，Proposal代写，Paper代写，Presentation代写，计算机作业代写，论文修改和润色，网课代做，exam代考等等。写作范围涵盖高中，本科，研究生等海外留学全阶段，辐射金融，经济学，会计学，审计学，管理学等全球99%专业科目。写作团队既有专业英语母语作者，也有海外名校硕博留学生，每位写作老师都拥有过硬的语言能力，专业的学科背景和学术写作经验。我们承诺100%原创，100%专业，100%准时，100%满意。

## MATLAB代写

MATLAB 是一种用于技术计算的高性能语言。它将计算、可视化和编程集成在一个易于使用的环境中，其中问题和解决方案以熟悉的数学符号表示。典型用途包括：数学和计算算法开发建模、仿真和原型制作数据分析、探索和可视化科学和工程图形应用程序开发，包括图形用户界面构建MATLAB 是一个交互式系统，其基本数据元素是一个不需要维度的数组。这使您可以解决许多技术计算问题，尤其是那些具有矩阵和向量公式的问题，而只需用 C 或 Fortran 等标量非交互式语言编写程序所需的时间的一小部分。MATLAB 名称代表矩阵实验室。MATLAB 最初的编写目的是提供对由 LINPACK 和 EISPACK 项目开发的矩阵软件的轻松访问，这两个项目共同代表了矩阵计算软件的最新技术。MATLAB 经过多年的发展，得到了许多用户的投入。在大学环境中，它是数学、工程和科学入门和高级课程的标准教学工具。在工业领域，MATLAB 是高效研究、开发和分析的首选工具。MATLAB 具有一系列称为工具箱的特定于应用程序的解决方案。对于大多数 MATLAB 用户来说非常重要，工具箱允许您学习应用专业技术。工具箱是 MATLAB 函数（M 文件）的综合集合，可扩展 MATLAB 环境以解决特定类别的问题。可用工具箱的领域包括信号处理、控制系统、神经网络、模糊逻辑、小波、仿真等。