Google的Fergus Henderson在Software Engineering at Google中介绍了Google的软件工程实践。

软件开发

源码仓库

• 单一源代码仓库，除了核心配置和安全相关代码，任何工程师都可以访问任何代码，并可以根据需要修改
• 所有开发都基于master分支，发布的时候才创建发布分枝
• 代码的每个子树都有owner，任何修改都需要owner批准

Blaze分布式构建系统

• 构建和测试存储库中的任何软件通常非常简单和快捷
• 开发人员只需要编写BUILD文件，并且每个构建系统仅依赖BUILD文件所声明的文件
• 构建系统的优化：可靠，自动跟踪依赖关系，增量构建，缓存构建结果以便复用
• 自动代码检查和测试

代码审查

• 完善的代码审查工具，如可视化的Web界面、电子邮件集成、自动展示测试或静态分析的结果
• 每个变更都必须由至少另外一人审查，并将审查结果自动复制到项目维护者的邮件列表
• 鼓励小的变更，大的变更可以拆分为一系列较小的变更

测试

• 鼓励和广泛使用单元测试，Mocking非常普遍
• 广泛使用集成测试和回归测试
• 自动测量测试覆盖率
• 部署之前进行负载测试，显示关键的metrics，比如延迟、错误率以及它们随请求速率的变化情况

Bug跟踪

• Google使用名为Buganizer的Bug跟踪系统
• 使用标签分类bug
• 每个bug都有一个默认的assignee和抄送邮件列表

编程语言

• 鼓励使用C++、Java、Python或Go之一，最小化不同编程语言的数量
• 每种语言都有Google风格指南，还有一个公司范围内的可读性培训
• 不同语言之前使用基于Protocol Buffers的RPC通信
• 为所有语言提供通用的开发工具，比如代码签出、编辑、构建、测试、审查、bug报告等

调试和分析

• 在通用框架中提供调试和代码跟踪工具
• 提供用于调试的网络接口检查RPC调用的时间、错误率和频率限制以及资源消耗、性能分析数据等

发布

• 频繁发布（比如每周或每两周），自动化发布任务，提高工程师积极性，允许更多迭代以加快整体速度
• 发布分支，将master的修改cherry-pick到发布分支
• 发布到staging服务器，测试部分生产流量的副本
• 发布到canary服务器，测试真实生产流量的一个子集
• 最后逐步发布到所有服务器

Launch approval

• 任何用户可见的更改或重大的设计变更都需要工程团队之外的很多人员的审查和批准，以确保这些变更满足符合法律、隐私、安全、可靠性以及业务需求
• Google内部的Launch approval工具会跟踪这些审查和批准

Post-mortems

• 任何重大的生产故障都需要写一份事后的总结文档，描述事件的原因、影响以及如何解决
• 重点关注如何避免它们再次发生（而不是追究人员责任）

频繁重写

• 大部分软件每隔几年都会重写一次
• 减少了累计复杂性
• 有助于适应当前的最佳实践，鼓励新的想法
• 也是一种团队成员之间传递ownership的方式，
• 这是Google保持敏捷和长期成功的关键

项目管理

20%时间

• 允许工程师可以将20%时间花在喜欢的任何项目上
• 有助于新想法的原型开发和演示，提高员工积极性
• 鼓励创新企业文化

OKR（Objectives and Key Results）

• 个人和团队要明确记录目标并评估这些目标的进展情况，团队设置季度和年度目标
• 建立关键结果来量化OKR，用OKR score评估进展情况
• 设置野心勃勃的OKR指标，即设置期望为目标的65%
• OKR是全公司透明的，是一种简化的沟通框架，使每个人都清晰了解公司的目标以及自己的位置

项目审批

• Google没有明确的项目审批流程，一般通过自下而上的方式进行

公司重组

• 因项目取消而重组时工程师可以自由选择新的团队或角色
• 在很大程度上，技术驱动公司应该进行频繁的重组以避免组织效率低下

人员管理

角色，技术角色与管理角色分开，项目由技术主管领导和决策，而经理负责管理技术主管，指导职业发展，并负责绩效评估

• 高标准的软件工程师
• 研究科学家
• SRE
• 产品经理
• 项目经理

工作环境（Facilities）

• Google提供丰富的娱乐、运动和餐饮设施
• 开放式办公鼓励沟通
• 先进的视频会议设施方便不同团队的沟通

培训

• 新员工培训，每个新员工都有导师和伙伴（Buddy）
• “Codelabs”和丰富的培训课程
• 也支持外部机构学习

换岗

• 鼓励在不同部门换岗，帮助公司内传播知识
• 允许12个月内表现良好的员工更换项目
• 鼓励临时性的参与其他项目

绩效考核和奖励

• 鼓励“peer bonuses”和“kudos”
• 明确详细的晋升过程，确保正确的人得到晋升
• 匿名反馈调查评估经理的绩效

更多内容请参考英文原文https://arxiv.org/pdf/1702.01715.pdf。更多SRE的内容请参考SRE以及SRE笔记。

The main idea of the paper is that the typicalfailover architecture used when moving from a single datacenter to multiple datacenters doesn’t work well in practice. What does work, where work means using fewer resources while providing high availability and consistency, is anatively multihomed architecture:

Our current approach is to build natively multihomed systems. Such systemsrun hot in multiple datacenters all the time, and adaptively move load between datacenters, with the ability to handle outages of any scale completely transparently. Additionally, planned datacenter outages and maintenance events are completely transparent, causing minimal disruption to the operational systems. In the past, such events required labor-intensive efforts to move operational systems from one datacenter to another

The use of “multihoming” in this context may be confusing becausemultihoming usually refers to a computer connected to more than one network. At Google scale perhaps it’s just as natural to talk about connecting to multiple datacenters.

Google has built several multi-homed systems to guarantee high availability (4 to 5 nines) and consistency in the presence of datacenter level outages:F1 / Spanner: Relational Database;Photon: Joining Continuous Data Streams;Mesa: Data Warehousing. The approach taken by each of these systems is discussed in the paper, as are the many challenges is building a multi-homed system: Synchronous Global State; What to Checkpoint; Repeatable Input; Exactly Once Output.

The huge constraint here ishaving availability and consistency. This highlights the refreshing and continued emphasis Google puts on making even these complex systemseasy for programmers to use:

The simplicity of a multi-homed system is particularly valuable for users. Without multi-homing, failover, recovery, and dealing with inconsistency are all application problems. With multi-homing, these hard problems are solved by the infrastructure, so the application developer gets high availability and consistency for free and can focus instead on building their application.

The biggest surprise in the paper was the idea that amultihomed system can actually take far fewer resources than a failover system:

In a multi-homed system deployed in three datacenters with 20% total catchup capacity, the total resource footprint is 170% of steady state. This is dramatically less than the 300% required in the failover design above

What’s Wrong With Failover?

Failover-based approaches, however, do not truly achieve high availability, and can have excessive cost due to the deployment of standby resources.

Our teams have had several bad experiences dealing with failover-based systems in the past. Since unplanned outages are rare, failover procedures were often added as an afterthought, not automated and not well tested. On multiple occasions, teams spent days recovering from an outage, bringing systems back online component by component, recovering state with ad hoc tools like custom MapReduces, and gradually tuning the system as it tried to catch up processing the backlog starting from the initial outage. These situations not only cause extended unavailability, but are also extremely stressful for the teams running complex mission-critical systems.

How do Multihomed Systems Work?

In contrast, multi-homed systems are designed to run in multiple datacenters as a core design property, so there is no on-the-side failover concept. A multi-homed system runs live in multiple datacenters all the time. Each datacenter processes work all the time, and work is dynamically shared between datacenters to balance load. When one datacenter is slow, some fraction of work automatically moves to faster datacenters. When a datacenter is completely unavailable, all its work is automatically distributed to other datacenters.

There is no failover process other than the continuous dynamic load balancing. Multi-homed systems coordinate work across datacenters using shared global state that must be updated synchronously. All critical system state is replicated so that any work can be restarted in an alternate datacenter at any point, while still guaranteeing exactly once semantics. Multi-homed systems are uniquely able to provide high availability and full consistency in the presence of datacenter level failures.

In any of our typical streaming system, the events being processed are based on user interactions, and logged by systems serving user traffic in many datacenters around the world. A log collection service gathers these logs globally and copies them to two or more specific logs datacenters. Each logs datacenter gets a complete copy of the logs, with the guarantee that all events copied to any one datacenter will (eventually) be copied to all logs datacenters. The stream processing systems run in one or more of the logs datacenters and processes all events. Output from the stream processing system is usually stored into some globally replicated system so that the output can be consumed reliably from anywhere.

In a multi-homed system, all datacenters are live and processing all the time. Deploying three datacenters is typical. In steady state, each of the three datacenters process 33% of the traffic. After a failure where one datacenter is lost, the two remaining datacenters each process 50% of the traffic.

Pick up fromhttp://highscalability.com/blog/2016/2/23/googles-transition-from-single-datacenter-to-failover-to-a-n.html. The paper is athttps://static.googleusercontent.com/media/research.google.com/en//pubs/archive/44686.pdf