[en] Residential thermostatically controlled loads are good candidates to provide ancillary services to the power grid. However, some difficulties need to be overcome to harvest their flexibility. One of the challenges involved is to ensure that all controlled loads stay within their operational limits while being subject to uncertainties. In this paper, we examine the provision of frequency containment reserves with electric water heaters. We use an aggregate-and-dispatch method along with a chance-constrained optimization problem to manage uncertain frequency deviations. We reformulate the problem both in an analytical way and based on robust optimization. The approaches are then validated and compared in a simulation study. Compared to the analytical approach, we find that the robust optimization approach results in a similar average energy consumption, while considerably increasing the bidding potential of the cluster of appliances considered.