性能之巅（第2版）：系统、企业与云可观测性(博文视点出品)

................................................................................................3
1.4　视角......................................................................................................................4
1.5　性能工程是充满挑战的......................................................................................5
1.5.1　主观性......................................................................................................5
1.5.2　复杂性......................................................................................................5
1.5.3　多个原因..................................................................................................6
1.5.4　多个性能问题..........................................................................................6
1.6　延时......................................................................................................................6
1.7　可观测性..............................................................................................................7
1.7.1　计数器、统计数据和指标......................................................................8
1.7.2　剖析........................................................................................................10
1.7.3　跟踪........................................................................................................11
1.8　实验....................................................................................................................13
1.9　云计算................................................................................................................14
1.10　方法..................................................................................................................14
1.10.1Linux性能分析60秒..........................................................................15
1.11　案例研究...........................................................................................................15
1.11.1　缓慢的磁盘..........................................................................................16
1.11.2　软件变更..............................................................................................18
1.11.3　更多阅读..............................................................................................19
1.12　参考资料..........................................................................................................19
第2章　方法..................................................................................................................21
2.1　术语....................................................................................................................22
2.2　模型....................................................................................................................23
2.2.1　受测系统................................................................................................23
2.2.2　排队系统................................................................................................23
2.3　概念....................................................................................................................24
2.3.1　延时........................................................................................................24
2.3.2　时间量级................................................................................................25
2.3.3　权衡........................................................................................................26
2.3.4　调优的影响............................................................................................27
2.3.5　合适的层级............................................................................................28
2.3.6　何时停止分析........................................................................................28
2.3.7　性能推荐的时间点................................................................................29
2.3.8　负载与架构............................................................................................30
2.3.9　扩展性....................................................................................................30
2.3.10　指标......................................................................................................32
2.3.11　使用率..................................................................................................33
2.3.12　饱和度..................................................................................................34
2.3.13　剖析......................................................................................................35
2.3.14　缓存......................................................................................................35
2.3.15　已知的未知..........................................................................................37
2.4　视角....................................................................................................................37
2.4.1　资源分析................................................................................................38
2.4.2　工作负载分析........................................................................................39
2.5　方法....................................................................................................................40
2.5.1　街灯讹方法............................................................................................42
2.5.2　随机变动讹方法....................................................................................42
2.5.3　责怪他人讹方法....................................................................................43
2.5.4AdHoc核对清单法...............................................................................43
2.5.5　问题陈述法............................................................................................44
2.5.6　科学法....................................................................................................44
2.5.7　诊断循环................................................................................................46
2.5.8　工具法....................................................................................................46
2.5.9USE方法................................................................................................46
2.5.10RED方法.............................................................................................52
2.5.11　工作负载特征归纳..............................................................................53
2.5.12　向下钻取分析......................................................................................54
2.5.13　延时分析..............................................................................................55
2.5.14R方法...................................................................................................56
2.5.15　事件跟踪..............................................................................................56
2.5.16　基础线统计..........................................................................................58
2.5.17　静态性能调优......................................................................................59
2.5.18　缓存调优..............................................................................................59
2.5.19　微基准测试..........................................................................................60
2.5.20　性能箴言..............................................................................................60
2.6　建模....................................................................................................................61
2.6.1　企业与云................................................................................................61
2.6.2　可视化识别............................................................................................62
2.6.3Amdahl扩展定律..................................................................................63
2.6.4　通用扩展定律........................................................................................64
2.6.5　排队理论................................................................................................65
2.7　容量规划............................................................................................................68
2.7.1　资源极限................................................................................................69
2.7.2　因素分析................................................................................................70
2.7.3　扩展方案................................................................................................71
2.8　统计....................................................................................................................72
2.8.1　量化性能收益........................................................................................72
2.8.2　平均值....................................................................................................73
2.8.3　标准方差、百分位数、中位数............................................................74
2.8.4　变异系数................................................................................................75
2.8.5　多重模态分布........................................................................................75
2.8.6　异常值....................................................................................................76
2.9　监测....................................................................................................................76
2.9.1　基于时间的规律....................................................................................76
2.9.2　监测产品................................................................................................78
2.9.3　自启动以来的信息统计........................................................................78
2.10　可视化..............................................................................................................78
2.10.1　线图......................................................................................................79
2.10.2　散点图..................................................................................................80
2.10.3　热图......................................................................................................81
2.10.4　时间线图..............................................................................................82
2.10.5　表面图..................................................................................................83
2.10.6　可视化工具..........................................................................................83
2.11　练习...................................................................................................................84
2.12　参考资料..........................................................................................................84
第3章　操作系统...........................................................................................................86
3.1　术语....................................................................................................................87
3.2　背景....................................................................................................................88
3.2.1　内核........................................................................................................88
3.2.2　内核态与用户态....................................................................................90
3.2.3　系统调用................................................................................................91
3.2.4　中断........................................................................................................92
3.2.5　时钟和空闲............................................................................................96
3.2.6　进程........................................................................................................96
3.2.7　栈............................................................................................................99
3.2.8　虚拟内存..............................................................................................100
3.2.9　调度器..................................................................................................102
3.2.10　文件系统............................................................................................103
3.2.11　缓存....................................................................................................105
3.2.12　网络....................................................................................................106
3.2.13　设备驱动............................................................................................106
3.2.14　多处理器............................................................................................107
3.2.15　抢占....................................................................................................107
3.2.16　资源管理............................................................................................107
3.2.17　可观测性............................................................................................108
3.3　内核..................................................................................................................108
3.3.1UNIX....................................................................................................109
3.3.2BSD.......................................................................................................110
3.3.3Solaris....................................................................................................111
3.4Linux..................................................................................................................111
3.4.1Linux内核开发....................................................................................112
3.4.2systemd.................................................................................................117
3.4.3KPTI（meltdown）...............................................................................118
3.4.4eBPF.....................................................................................................118
3.5　其他主题..........................................................................................................119
3.5.1PGO内核.............................................................................................119
3.5.2unikernel...............................................................................................120
3.5.3　微内核和混合内核..............................................................................120
3.5.4　分布式操作系统..................................................................................121
3.6　内核比较..........................................................................................................121
3.7　练习..................................................................................................................122
3.8　参考资料..........................................................................................................122
3.8.1　延伸阅读..............................................................................................125
第4章　观测工具.........................................................................................................126
4.1　工具范围..........................................................................................................127
4.1.1　静态性能工具......................................................................................127
4.1.2　危机处理工具......................................................................................128
4.2　工具类型..........................................................................................................129
4.2.1　固定计数器..........................................................................................130
4.2.2　剖析......................................................................................................131
4.2.3　跟踪......................................................................................................132
4.2.4　监测......................................................................................................133
4.3　监测来源..........................................................................................................135
4.3.1/proc......................................................................................................137
4.3.2/sys........................................................................................................140
4.3.3　延时核算..............................................................................................142
4.3.4netlink...................................................................................................143
4.3.5tracepoint..............................................................................................143
4.3.6kprobes..................................................................................................148
4.3.7uprobes..................................................................................................151
4.3.8USDT....................................................................................................152
4.3.9　硬件计数器..........................................................................................153
4.3.10　其他观测源........................................................................................156
4.4sar......................................................................................................................158
4.4.1sar(1)的覆盖范围................................................................................158
4.4.2sar(1)监测............................................................................................159
4.4.3sar(1)实时报告....................................................................................162
4.4.4sar(1)文档............................................................................................163
4.5　跟踪工具..........................................................................................................163
4.6　观测工具的观测..............................................................................................164
4.7　练习..................................................................................................................165
4.8　参考资料..........................................................................................................166
第5章　应用程序.........................................................................................................168
5.1　应用程序基础..................................................................................................168
5.1.1　目标......................................................................................................170
5.1.2　常见情况的优化..................................................................................171
5.1.3　可观测性..............................................................................................171
5.1.4　大O标记法.........................................................................................172
5.2　应用程序性能技术..........................................................................................173
5.2.1　选择I/O尺寸.......................................................................................173
5.2.2　缓存......................................................................................................173
5.2.3　缓冲区..................................................................................................174
5.2.4　轮询......................................................................................................174
5.2.5　并发和并行..........................................................................................174
5.2.6　非阻塞I/O............................................................................................178
5.2.7　处理器绑定..........................................................................................179
5.2.8　性能箴言..............................................................................................179
5.3　编程语言..........................................................................................................179
5.3.1　编译型语言..........................................................................................180
5.3.2　解释型语言..........................................................................................182
5.3.3　虚拟机..................................................................................................182
5.3.4　垃圾回收..............................................................................................182
5.4　方法..................................................................................................................183
5.4.1CPU剖析..............................................................................................184
5.4.2off-CPU分析........................................................................................186
5.4.3　系统调用分析......................................................................................189
5.4.4USE方法..............................................................................................190
5.4.5　线程状态分析......................................................................................191
5.4.6　锁分析..................................................................................................195
5.4.7　静态性能调优......................................................................................196
5.4.8　分布式跟踪..........................................................................................196
5.5　观测工具..........................................................................................................197
5.5.1perf........................................................................................................198
5.5.2profile....................................................................................................201
5.5.3offcputime.............................................................................................202
5.5.4strace.....................................................................................................203
5.5.5execsnoop.............................................................................................205
5.5.6syscount................................................................................................206
5.5.7bpftrace.................................................................................................207
5.6　明白了..............................................................................................................212
5.6.1　缺少符号..............................................................................................212
5.6.2　缺少栈..................................................................................................213
5.7　练习..................................................................................................................214
5.8　参考资料..........................................................................................................216
第6章　CPU................................................................................................................218
6.1　术语..................................................................................................................219
6.2　模型..................................................................................................................220
6.2.1CPU架构..............................................................................................220
6.2.2CPU内存缓存......................................................................................220
6.2.3CPU运行队列......................................................................................221
6.3　概念..................................................................................................................222
6.3.1　时钟频率..............................................................................................222
6.3.2　指令......................................................................................................222
6.3.3　指令流水线..........................................................................................223
6.3.4　指令宽度..............................................................................................223
6.3.5　指令尺寸..............................................................................................223
6.3.6SMT......................................................................................................224
6.3.7IPC和CPI............................................................................................224
6.3.8　使用率..................................................................................................225
6.3.9　用户时间/内核时间...........................................................................225
6.3.10　饱和度................................................................................................225
6.3.11　抢占....................................................................................................226
6.3.12　优先级反转........................................................................................226
6.3.13　多进程和多线程................................................................................226
6.3.14　字长....................................................................................................228
6.3.15　编译器优化........................................................................................228
6.4　架构..................................................................................................................228
6.4.1　硬件......................................................................................................228
6.4.2　软件......................................................................................................238
6.5　方法..................................................................................................................242
6.5.1　工具法..................................................................................................242
6.5.2USE方法..............................................................................................243
6.5.3　负载特征归纳......................................................................................243
6.5.4　剖析......................................................................................................245
6.5.5　周期分析..............................................................................................248
6.5.6　性能监测..............................................................................................248
6.5.7　静态性能调优......................................................................................249
6.5.8　优先级调优..........................................................................................249
6.5.9　资源控制..............................................................................................250
6.5.10CPU绑定............................................................................................250
6.5.11　微基准测试........................................................................................250
6.6　观测工具..........................................................................................................251
6.6.1uptime...................................................................................................252
6.6.2vmstat....................................................................................................255
6.6.3mpstat....................................................................................................255
6.6.4sar..........................................................................................................256
6.6.5ps...........................................................................................................257
6.6.6top.........................................................................................................258
6.6.7pidstat....................................................................................................259
6.6.8time和ptime........................................................................................260
6.6.9turbostat................................................................................................261
6.6.10showboost...........................................................................................262
6.6.11pmcarch...............................................................................................263
6.6.12tlbstat...................................................................................................263
6.6.13perf......................................................................................................264
6.6.14profile..................................................................................................274
6.6.15cpudist.................................................................................................276
6.6.16runqlat.................................................................................................277
6.6.17runqlen................................................................................................278
6.6.18softirqs................................................................................................279
6.6.19hardirqs...............................................................................................280
6.6.20bpftrace...............................................................................................280
6.6.21　其他工具............................................................................................283
6.7　可视化..............................................................................................................285
6.7.1　使用率热图..........................................................................................286
6.7.2　亚秒级偏移量热图..............................................................................286
6.7.3　火焰图..................................................................................................287
6.7.4FlameScope...........................................................................................290
6.8　实验..................................................................................................................291
6.8.1AdHoc..................................................................................................291
6.8.2SysBench..............................................................................................291
6.9　调优..................................................................................................................292
6.9.1　编译器选项..........................................................................................293
6.9.2　调度优先级和调度类..........................................................................293
6.9.3　调度器选项..........................................................................................293
6.9.4　调节调速器..........................................................................................294
6.9.5　能耗状态..............................................................................................295
6.9.6CPU绑定..............................................................................................295
6.9.7　独占CPU组.........................................................................................295
6.9.8　资源控制..............................................................................................296
6.9.9　安全启动选项......................................................................................296
6.9.10　处理器选项（BIOS调优）................................................................297
6.10　练习................................................................................................................297
6.11　参考资料.........................................................................................................298
第7章　内存................................................................................................................301
7.1　术语..................................................................................................................302
7.2　概念..................................................................................................................303
7.2.1　虚拟内存..............................................................................................303
7.2.2　换页......................................................................................................303
7.2.3　按需换页..............................................................................................305
7.2.4　过度提交..............................................................................................306
7.2.5　进程交换..............................................................................................306
7.2.6　文件系统缓存用量..............................................................................306
7.2.7　使用率和饱和度..................................................................................307
7.2.8　分配器..................................................................................................307
7.2.9　共享内存..............................................................................................307
7.2.10　工作集大小........................................................................................308
7.2.11　字长....................................................................................................308
7.3　架构..................................................................................................................308
7.3.1　硬件......................................................................................................309
7.3.2　软件......................................................................................................313
7.3.3　进程虚拟地址空间..............................................................................317
7.4　方法..................................................................................................................320
7.4.1　工具法..................................................................................................321
7.4.2USE方法..............................................................................................322
7.4.3　描述使用情况......................................................................................323
7.4.4　周期分析..............................................................................................324
7.4.5　性能监测..............................................................................................324
7.4.6　泄漏检测..............................................................................................324
7.4.7　静态性能调优......................................................................................325
7.4.8　资源控制..............................................................................................325
7.4.9　微基准测试..........................................................................................326
7.4.10　内存收缩............................................................................................326
7.5　观测工具..........................................................................................................326
7.5.1vmstat....................................................................................................327
7.5.2PSI.........................................................................................................328
7.5.3swapon..................................................................................................329
7.5.4sar..........................................................................................................329
7.5.5slabtop...................................................................................................331
7.5.6numastat................................................................................................332
7.5.7ps...........................................................................................................332
7.5.8top.........................................................................................................333
7.5.9pmap.....................................................................................................334
7.5.10perf......................................................................................................336
7.5.11drsnoop................................................................................................339
7.5.12wss......................................................................................................340
7.5.13bpftrace...............................................................................................341
7.5.14　其他工具............................................................................................345
7.6　调优..................................................................................................................348
7.6.1　可调参数..............................................................................................348
7.6.2　多种页面大小......................................................................................349
7.6.3　分配器..................................................................................................350
7.6.4NUMA绑定.........................................................................................350
7.6.5　资源控制..............................................................................................351
7.7　练习.................................................................................................................351
7.8　参考资料..........................................................................................................353
第8章　文件系统.........................................................................................................355
8.1　术语..................................................................................................................356
8.2　模型..................................................................................................................357
8.2.1　文件系统接口......................................................................................357
8.2.2　文件系统缓存......................................................................................357
8.2.3　二级缓存..............................................................................................358
8.3　概念..................................................................................................................358
8.3.1　文件系统延时......................................................................................358
8.3.2　缓存......................................................................................................359
8.3.3　随机与顺序I/O....................................................................................359
8.3.4　预取......................................................................................................360
8.3.5　预读......................................................................................................361
8.3.6　回写缓存..............................................................................................361
8.3.7　同步写..................................................................................................362
8.3.8　裸I/O与直接I/O.................................................................................362
8.3.9　非阻塞I/O............................................................................................363
8.3.10　内存映射文件....................................................................................363
8.3.11　元数据................................................................................................363
8.3.12　逻辑I/O与物理I/O...........................................................................364
8.3.13　操作并不平等....................................................................................366
8.3.14　特殊的文件系统................................................................................367
8.3.15　访问时间戳........................................................................................367
8.3.16　容量....................................................................................................367
8.4　架构..................................................................................................................367
8.4.1　文件系统I/O栈...................................................................................368
8.4.2VFS.......................................................................................................368
8.4.3　文件系统缓存......................................................................................369
8.4.4　文件系统特性......................................................................................371
8.4.5　文件系统种类......................................................................................373
8.4.6　卷和池..................................................................................................378
8.5　方法..................................................................................................................379
8.5.1　磁盘分析..............................................................................................380
8.5.2　延时分析..............................................................................................380
8.5.3　负载特征归纳......................................................................................382
8.5.4　性能监测..............................................................................................384
8.5.5　静态性能调优......................................................................................384
8.5.6　缓存调优..............................................................................................385
8.5.7　负载分离..............................................................................................385
8.5.8　微基准测试..........................................................................................385
8.6　观测工具..........................................................................................................387
8.6.1mount....................................................................................................387
8.6.2free........................................................................................................388
8.6.3top.........................................................................................................388
8.6.4vmstat....................................................................................................389
8.6.5sar..........................................................................................................389
8.6.6slabtop...................................................................................................390
8.6.7strace.....................................................................................................390
8.6.8fatrace...................................................................................................391
8.6.9LatencyTOP..........................................................................................392
8.6.10opensnoop...........................................................................................392
8.6.11filetop..................................................................................................393
8.6.12cachestat..............................................................................................395
8.6.13ext4dist（xfs、zfs、btrfs、nfs）........................................................395
8.6.14ext4slower（xfs、zfs、btrfs、nfs）...................................................397
8.6.15bpftrace...............................................................................................398
8.6.16　其他工具............................................................................................404
8.6.17　可视化................................................................................................406
8.7　实验..................................................................................................................407
8.7.1AdHoc..................................................................................................407
8.7.2　微基准测试工具..................................................................................407
8.7.3　缓存刷新..............................................................................................409
8.8　调优..................................................................................................................410
8.8.1　应用程序调用......................................................................................410
8.8.2ext4.......................................................................................................411
8.8.3ZFS.......................................................................................................413
8.9　练习..................................................................................................................415
8.10　参考资料........................................................................................................416
第9章　磁盘................................................................................................................418
9.1　术语..................................................................................................................419
9.2　模型..................................................................................................................419
9.2.1　简单磁盘..............................................................................................420
9.2.2　缓存磁盘..............................................................................................420
9.2.3　控制器..................................................................................................421
9.3　概念..................................................................................................................421
9.3.1　测量时间..............................................................................................421
9.3.2　时间尺度..............................................................................................424
9.3.3　缓存......................................................................................................425
9.3.4　随机I/O与连续I/O.............................................................................425
9.3.5　读/写比...............................................................................................426
9.3.6I/O大小................................................................................................426
9.3.7IOPS并不平等.....................................................................................427
9.3.8　非数据传输磁盘命令..........................................................................427
9.3.9　使用率..................................................................................................427
9.3.10　饱和度................................................................................................428
9.3.11I/O等待..............................................................................................428
9.3.12　同步与异步........................................................................................429
9.3.13　磁盘I/O与应用程序I/O...................................................................429
9.4　架构..................................................................................................................430
9.4.1　磁盘类型..............................................................................................430
9.4.2　接口......................................................................................................436
9.4.3　存储类型..............................................................................................438
9.4.4　操作系统磁盘I/O栈...........................................................................441
9.5　方法..................................................................................................................444
9.5.1　工具法..................................................................................................444
9.5.2USE方法..............................................................................................445
9.5.3　性能监测..............................................................................................446
9.5.4　负载特征归纳......................................................................................446
9.5.5　延时分析..............................................................................................448
9.5.6　静态性能调优......................................................................................449
9.5.7　缓存调优..............................................................................................450
9.5.8　资源控制..............................................................................................450
9.5.9　微基准测试..........................................................................................451
9.5.10　伸缩....................................................................................................452
9.6　观测工具..........................................................................................................452
9.6.1iostat......................................................................................................453
9.6.2sar..........................................................................................................458
9.6.3PSI.........................................................................................................458
9.6.4pidstat....................................................................................................459
9.6.5perf........................................................................................................460
9.6.6biolatency..............................................................................................462
9.6.7biosnoop................................................................................................465
9.6.8iotop、biotop........................................................................................467
9.6.9biostacks...............................................................................................469
9.6.10blktrace...............................................................................................470
9.6.11bpftrace...............................................................................................474
9.6.12MegaCli..............................................................................................478
9.6.13smartctl...............................................................................................479
9.6.14SCSI日志...........................................................................................480
9.6.15　其他工具............................................................................................481
9.7　可视化..............................................................................................................482
9.7.1　折线图..................................................................................................482
9.7.2　延时散点图..........................................................................................482
9.7.3　延时热图..............................................................................................483
9.7.4　偏移量热图..........................................................................................483
9.7.5　使用率热图..........................................................................................484
9.8　实验..................................................................................................................484
9.8.1AdHoc..................................................................................................485
9.8.2　自定义负载生成器..............................................................................485
9.8.3　微基准测试工具..................................................................................485
9.8.4　随机读示例..........................................................................................486
9.8.5ioping....................................................................................................486
9.8.6fio..........................................................................................................487
9.8.7blkreplay...............................................................................................487
9.9　调优..................................................................................................................488
9.9.1　操作系统可调参数..............................................................................488
9.9.2　磁盘设备可调参数..............................................................................489
9.9.3　磁盘控制器可调参数..........................................................................489
9.10　练习................................................................................................................490
9.11　参考资料.........................................................................................................491
第10章　网络..............................................................................................................493
10.1　术语................................................................................................................494
10.2　模型................................................................................................................494
10.2.1　网络接口............................................................................................494
10.2.2　控制器................................................................................................495
10.2.3　协议栈................................................................................................495
10.3　概念................................................................................................................496
10.3.1　网络和路由........................................................................................496
10.3.2　协议....................................................................................................497
10.3.3　封装....................................................................................................498
10.3.4　包的大小............................................................................................498
10.3.5　延时....................................................................................................499
10.3.6　缓冲....................................................................................................500
10.3.7　连接积压队列....................................................................................501
10.3.8　接口协商............................................................................................501
10.3.9　避免阻塞............................................................................................501
10.3.10　使用率..............................................................................................502
10.3.11　本地连接..........................................................................................502
10.4　架构................................................................................................................502
10.4.1　协议....................................................................................................503
10.4.2　硬件....................................................................................................508
10.4.3　软件....................................................................................................510
10.5　方法................................................................................................................517
10.5.1　工具法................................................................................................518
10.5.2USE方法............................................................................................519
10.5.3　工作负载特征归纳............................................................................519
10.5.4　延时分析............................................................................................520
10.5.5　性能监测............................................................................................522
10.5.6　数据包嗅探........................................................................................522
10.5.7TCP分析............................................................................................523
10.5.8　静态性能调优....................................................................................524
10.5.9　资源控制............................................................................................525
10.5.10　微基准测试......................................................................................525
10.6　观测工具........................................................................................................526
10.6.1ss.........................................................................................................527
10.6.2ip.........................................................................................................529
10.6.3ifconfig................................................................................................530
10.6.4nstat.....................................................................................................530
10.6.5netstat..................................................................................................532
10.6.6sar........................................................................................................535
10.6.7nicstat..................................................................................................538
10.6.8ethtool.................................................................................................539
10.6.9tcplife..................................................................................................540
10.6.10tcptop................................................................................................541
10.6.11tcpretrans...........................................................................................542
10.6.12bpftrace.............................................................................................543
10.6.13tcpdump............................................................................................551
10.6.14Wireshark..........................................................................................552
10.6.15　其他工具..........................................................................................553
10.7　实验................................................................................................................554
10.7.1ping.....................................................................................................555
10.7.2traceroute............................................................................................555
10.7.3pathchar...............................................................................................556
10.7.4iperf.....................................................................................................557
10.7.5netperf.................................................................................................558
10.7.6tc.........................................................................................................558
10.7.7　其他工具............................................................................................559
10.8　调优................................................................................................................559
10.8.1　系统级可调参数................................................................................560
10.8.2　套接字选项........................................................................................565
10.8.3　配置....................................................................................................566
10.9　练习................................................................................................................566
10.10　参考资料......................................................................................................567
第11章　云计算...........................................................................................................572
11.1　背景.................................................................................................................573
11.1.1　实例类型............................................................................................574
11.1.2　可扩展的架构....................................................................................574
11.1.3　容量规划............................................................................................575
11.1.4　存储....................................................................................................577
11.1.5　多租户................................................................................................578
11.1.6　编排（Kubernetes）............................................................................579
11.2　硬件虚拟化.....................................................................................................580
11.2.1　实现....................................................................................................582
11.2.2　系统开销............................................................................................582
11.2.3　资源控制............................................................................................588
11.2.4　可观测性............................................................................................590
11.3　操作系统虚拟化.............................................................................................599
11.3.1　实现方式............................................................................................600
11.3.2　系统开销............................................................................................603
11.3.3　资源控制............................................................................................606
11.3.4　可观测性............................................................................................610
11.4　轻量虚拟化.....................................................................................................623
11.4.1　实现....................................................................................................623
11.4.2　开销....................................................................................................624
11.4.3　资源控制............................................................................................624
11.4.4　可观测性............................................................................................624
11.5　其他类型.........................................................................................................626
11.6　比较.................................................................................................................627
11.7练习...................................................................................................................628
11.8　参考资料.........................................................................................................629
第12章　基准测试.......................................................................................................632
12.1　背景................................................................................................................633
12.1.1　原因....................................................................................................633
12.1.2　有效的基准测试................................................................................634
12.1.3　基准测试失败....................................................................................635
12.2　基准测试的类型...........................................................................................642
12.2.1　微基准测试........................................................................................642
12.2.2　模拟....................................................................................................644
12.2.3　回放....................................................................................................645
12.2.4　行业标准............................................................................................645
12.3　方法................................................................................................................647
12.3.1　被动基准测试....................................................................................647
12.3.2　主动基准测试....................................................................................648
12.3.3CPU剖析............................................................................................651
12.3.4USE方法............................................................................................652
12.3.5　工作负载特征归纳............................................................................652
12.3.6　自定义基准测试................................................................................652
12.3.7　逐渐增加负载....................................................................................653
12.3.8　合理性检查........................................................................................655
12.3.9　统计分析............................................................................................656
12.3.10　基准测试检查清单..........................................................................657
12.4　基准测试问题................................................................................................657
12.5　练习................................................................................................................659
12.6　参考资料........................................................................................................659
第13章　perf...............................................................................................................661
13.1　子命令概览....................................................................................................662
13.2　单行命令........................................................................................................664
13.3perf事件..........................................................................................................668
13.4　硬件事件........................................................................................................671
13.4.1　频率采样............................................................................................672
13.5　软件事件........................................................................................................673
13.6tracepoint事件................................................................................................674
13.7　探针事件........................................................................................................675
13.7.1kprobe.................................................................................................675
13.7.2uprobe.................................................................................................677
13.7.3USDT探针.........................................................................................680
13.8perfstat...........................................................................................................681
13.8.1　选项....................................................................................................682
13.8.2　周期统计信息....................................................................................683
13.8.3CPU均衡............................................................................................683
13.8.4　事件过滤器........................................................................................683
13.8.5　隐藏统计信息....................................................................................684
13.9perfrecord.......................................................................................................684
13.9.1　选项....................................................................................................685
13.9.2CPU剖析............................................................................................685
13.9.3　栈遍历................................................................................................686
13.10perfreport.....................................................................................................687
13.10.1TUI....................................................................................................687
13.10.2STDIO...............................................................................................688
13.11perfscript......................................................................................................689
13.11.1　火焰图..............................................................................................690
13.11.2　跟踪脚本..........................................................................................691
13.12perftrace.......................................................................................................691
13.12.1　内核版本..........................................................................................692
13.13　其他命令......................................................................................................692
13.14perf文档........................................................................................................694
13.15　参考资料......................................................................................................694
第14章　Ftrace............................................................................................................696
14.1　功能概述........................................................................................................697
14.2tracefs（/sys）...............................................................................................699
14.2.1tracefs的内容.....................................................................................700
14.3Ftrace函数剖析器..........................................................................................702
14.4Ftrace函数跟踪..............................................................................................703
14.4.1　使用trace............................................................................................704
14.4.2　使用trace_pipe...................................................................................706
14.4.3　选项....................................................................................................706
14.5　跟踪点............................................................................................................707
14.5.1　过滤器................................................................................................708
14.5.2　触发器................................................................................................709
14.6kprobes............................................................................................................710
14.6.1　事件跟踪............................................................................................710
14.6.2　参数....................................................................................................711
14.6.3　返回值................................................................................................712
14.6.4　过滤器和触发器................................................................................712
14.6.5kprobe剖析........................................................................................713
14.7uprobes............................................................................................................713
14.7.1　事件跟踪............................................................................................713
14.7.2　参数和返回值....................................................................................714
14.7.3　过滤器和触发器................................................................................714
14.7.4uprobe剖析........................................................................................715
14.8Ftracefunction_graph.....................................................................................715
14.8.1　图表跟踪............................................................................................715
14.8.2　选项....................................................................................................717
14.9Ftracehwlat....................................................................................................717
14.10Ftracehist触发器..........................................................................................718
14.10.1　单关键字..........................................................................................718
14.10.2　字段..................................................................................................719
14.10.3　修饰器..............................................................................................720
14.10.4PID过滤器.......................................................................................720
14.10.5　多关键字..........................................................................................721
14.10.6　栈踪迹关键字..................................................................................722
14.10.7　综合事件..........................................................................................723
14.11trace-cmd.......................................................................................................725
14.11.1　子命令概述......................................................................................726
14.11.2trace-cmd单行命令.........................................................................727
14.11.3trace-cmd和perf(1)的比较............................................................729
14.11.4trace-cmdfunction_graph.................................................................730
14.11.5KernelShark......................................................................................730
14.11.6trace-cmd文档.................................................................................731
14.12perfftrace......................................................................................................731
14.13perf-tools.......................................................................................................732
14.13.1　工具覆盖..........................................................................................733
14.13.2　单用途的工具..................................................................................733
14.13.3　多用途工具......................................................................................735
14.13.4perf-tools单行命令..........................................................................735
14.13.5　示例..................................................................................................738
14.13.6perf-tools与BCC/BPF的对比........................................................738
14.13.7　文档..................................................................................................739
14.14Ftrace文档....................................................................................................739
14.15　参考资料......................................................................................................739
第15章　BPF...............................................................................................................741
15.1BCC................................................................................................................743
15.1.1　安装....................................................................................................744
15.1.2　工具范围............................................................................................744
15.1.3　单用途工具........................................................................................745
15.1.4　多用途工具........................................................................................747
15.1.5　单行命令............................................................................................747
15.1.6　多用途工具示例................................................................................749
15.1.7BCC与bpftrace的比较....................................................................749
15.1.8　文档....................................................................................................750
15.2bpftrace...........................................................................................................751
15.2.1　安装....................................................................................................752
15.2.2　工具....................................................................................................752
15.2.3　单行命令............................................................................................753
15.2.4　编程....................................................................................................756
15.2.5　参考....................................................................................................764
15.2.6　文档....................................................................................................770
15.3　参考资料........................................................................................................771
第16章　案例研究.......................................................................................................773
16.1　无法解释的收益............................................................................................773
16.1.1　问题陈述............................................................................................773
16.1.2　分析策略............................................................................................774
16.1.3　统计数据............................................................................................774
16.1.4　配置....................................................................................................776
16.1.5PMC....................................................................................................778
16.1.6　软件事件............................................................................................779
16.1.7　跟踪....................................................................................................780
16.1.8　结论....................................................................................................782
16.2　其他信息........................................................................................................783
16.3　参考资料........................................................................................................783
附录A　USE方法：Linux............................................................................................784
附录B　sar总结...........................................................................................................788
附录C　bpftrace单行命令............................................................................................790
附录D　精选练习题答案..............................................................................................796
附录E　系统性能名人录..............................................................................................798