Posts

Go并发Sync.Once解析

在 go 语言中我们可以使用 sync.Once 对象来实现函数方法只执行一次的功能。简单代码示例 package main import ( "fmt" "sync" ) func main() { var ( o sync.Once wg sync.WaitGroup ) for i := 0; i < 10; i++ { wg.Add(1) go func(i int) { defer wg.Done() o.Do(func() { fmt.Printf("hello %d\n", i) }) }(i) } wg.Wait() } 输出: hello 9 不使用 Sync.Once 的结果如下: hello 9 hello 4 hello 0 hello 1 hello 2 hello 3 hello 6 hello 5 hello 7 hello 8 可以看到, 在使用 sync.Once 的情况下, 只执行一次函数。 ...

Go截取视频某一帧图片

前言最近遇到一个需求，需要截取视频的某一帧图片作为视频封面。我搜寻了相关资料，在 go 语言端常见的有两种做法，1）使用 opencv 的 go 绑定库，2）使用 ffmpeg 的 go 绑定库。这里我打算使用第二种方法，使用 ffmpeg 的 go 绑定库。 ffmpeg 介绍 FFmpeg是一个开源免费跨平台的视频和音频流方案，属于自由软件，采用LGPL或GPL许可证（依据你选择的组件）。它提供了录制、转换以及流化音视频的完整解决方案。它包含了非常先进的音频/视频编解码库libavcodec，为了保证高可移植性和编解码质量，libavcodec里很多codec都是从头开发的。 FFmpeg在Linux平台下开发，但它同样也可以在其它操作系统环境中编译运行，包括Windows、Mac OS X等。 FFmpeg项目由以下几部分组成: FFMpeg 视频文件转换命令行工具,也支持经过实时电视卡抓取和编码成视频文件。 FFServer 基于HTTP(RTSP正在开发中)用于实时广播的多媒体服务器，也支持时间平移. FFplay 用SDL和FFmpeg库开发的一个简单的媒体播放器. libavcodec 一个包含了所有FFmpeg音视频编解码器的库.为了保证最优性能和高可复用性,大多数编解码器从头开发的. libavformat 一个包含了所有的普通音视格式的解析器和产生器的库解决 1 安装 ffmpeg 浏览器访问 https://ffbinaries.com/downloads 根据你的系统安装 ffmpeg 2 安装 ffmpeg 的 go 绑定库这里我使用了 github.com/u2takey/ffmpeg-go 这个项目，它是一个绑定库，可以在 go 语言端使用 ffmpeg 命令行工具。 3 编写代码 package main import ( "bytes" "fmt" "github.com/disintegration/imaging" ffmpeg "github.com/u2takey/ffmpeg-go" "log" "os" "strings" ) func GetSnapshot(videoPath, snapshotPath string, frameNum int) (snapshotName string, err error) { buf := bytes.NewBuffer(nil) err = ffmpeg.Input(videoPath). Filter("select", ffmpeg.Args{fmt.Sprintf("gte(n,%d)", frameNum)}). Output("pipe:", ffmpeg.KwArgs{"vframes": 1, "format": "image2", "vcodec": "mjpeg"}). WithOutput(buf, os.Stdout). Run() if err != nil { log.Fatal("生成缩略图失败：", err) return "", err } img, err := imaging.Decode(buf) if err != nil { log.Fatal("生成缩略图失败：", err) return "", err } err = imaging.Save(img, snapshotPath+".png") if err != nil { log.Fatal("生成缩略图失败：", err) return "", err } names := strings.Split(snapshotPath, "\\") snapshotName = names[len(names)-1] + ".png" return } func main() { _, err := GetSnapshot("./test.mp4", "test", 1) if err != nil { return } } 函数里 ffmpeg 相关的代码主要是提取视频的某一帧以mjpeg编码为图片，然后保存到本地。 ...

Go errgroup

我们知道在 go 语言中很容易开启携程进行并发任务, 但是如何处理并发过程中的错误是非常棘手的，接下来我就来介绍 errgroup 的用法。 errgroup errgroup 包里主要是一个结构体和相应的方法，它可以让你在并发任务中处理错误。 type Group struct { // context 的 cancel 方法 cancel func() // 复用 WaitGroup wg sync.WaitGroup sem chan token // 用来保证只会接受一次错误 errOnce sync.Once // 保存第一个返回的错误 err error } func WithContext(ctx context.Context) (*Group, context.Context) func (g *Group) done() func (g *Group) Wait() error func (g *Group) Go(f func() error) func (g *Group) TryGo(f func() error) bool func (g *Group) SetLimit(n int) 通过 WithContext 可以创建一个可以取消的 Group,当然除此之外也可以零值的 Group 也可以直接使用，但是出错之后就不会取消其他的 goroutine 了. Go方法传入一个函数参数，会启动一个 goroutine 处理。 Wait 类似 WaitGroup 的 Wait 方法，等待所有的 goroutine 结束后退出，返回的错误是一个出错的 err。 TryGo 是和 SetLimit 配套的，只有当 group 中的 goroutines 数量小于配置的数量时，才会在 goroutine 中调用函数。 TryGo 用来判断 goroutine 是否启动。 ...

应用内存升高原因排查

起因最近一个部署了 go 应用的服务器出现了 OOM 的现象，内存占用过高。原因通过 Pyroscope 分析得出是因为 Minio 的 go sdk 中的 PutObject 函数占用了大量的内存。 Pyroscope 是什么，前面的文章已经介绍过了，这里就不过多介绍了。接下来我们通过查看相关的源码来查看是什么原因。 // PutObject creates an object in a bucket. // // You must have WRITE permissions on a bucket to create an object. // // - For size smaller than 16MiB PutObject automatically does a // single atomic PUT operation. // // - For size larger than 16MiB PutObject automatically does a // multipart upload operation. // // - For size input as -1 PutObject does a multipart Put operation // until input stream reaches EOF. Maximum object size that can // be uploaded through this operation will be 5TiB. // // WARNING: Passing down '-1' will use memory and these cannot // be reused for best outcomes for PutObject(), pass the size always. // // NOTE: Upon errors during upload multipart operation is entirely aborted. func (c *Client) PutObject(ctx context.Context, bucketName, objectName string, reader io.Reader, objectSize int64, opts PutObjectOptions, ) (info UploadInfo, err error) { if objectSize < 0 && opts.DisableMultipart { return UploadInfo{}, errors.New("object size must be provided with disable multipart upload") } err = opts.validate() if err != nil { return UploadInfo{}, err } return c.putObjectCommon(ctx, bucketName, objectName, reader, objectSize, opts) } 从方法的注释可以看出，当传递的大小为 -1 时，会进行多次 put 操作，直到输入流结束。多次 put 操作的最大大小为 5TiB，并且不能重用内存，导致占用大量内存。 ...

多平台博客发布工具openwrite使用

介绍 OpenWrite是一款便捷的多平台博客发布工具，可以在OpenWrite编写markdown文档，然后发布到其他博客平台，目前已经支持CSDN、SegmentFault、掘金、博客园、简书等知名平台。使用注册进入 http://admin.openwrite.cn/, 注册并登陆账号。安装浏览器插件第一步需要安装浏览器插件，根据这个链接教程安装,只有安装了插件，才能配置分发的平台。根据对插件的源码的分析，插件是通过 cookie、storage 等浏览器数据进行认证登陆的。此插件还包括模拟写作和 mackdown 编辑器。认证配置渠道安装完插件后，点击渠道管理，可以看到可配置的平台，支持多个平台。相应的平台如果没有登陆则显示未登陆，这时需要你登陆相应的平台。登陆平台后，点击认证，即可配置渠道。这里以掘金为例，认证成功后，可以进行相应的配置，配置文章分类和标题。发布文章点击文章管理界面，添加文章，添加文章后，可以选择配置的平台，并发布到相应的平台。普通用户一个月只有 10 次机会，一个平台占用一次机会。

Buf初入门2

上文讲了 Buf lint 命令的基础方法，本文将介绍 Buf lint 命令的一些常用配置。如果你的项目中没有 buf.yaml 配置文件， Buf lint 会提供一个默认的配置文件，默认内容如下： version: v1 lint: use: - DEFAULT except: - FILE_LOWER_SNAKE_CASE ignore: - bat - ban/ban.proto ignore_only: ENUM_PASCAL_CASE: - foo/foo.proto - bar BASIC: - foo enum_zero_value_suffix: _UNSPECIFIED rpc_allow_same_request_response: false rpc_allow_google_protobuf_empty_requests: false rpc_allow_google_protobuf_empty_responses: false service_suffix: Service allow_comment_ignores: true 配置选项接下来开始介绍 Buf lint 命令的配置选项。 use use 选项配置 lint 的类别，不同类别有相应的规则，有多种类别：DEFAULT、FILE_LOWER_SNAKE_CASE、BASIC等。默认是 DEFAULT 类别。 except except 选项配置 lint 不使用的类别。下面的配置，显示使用 DEFAULT 类别，但是不使用 ENUM_NO_ALLOW_ALIAS ,BASIC类别中的规则。 version: v1 lint: use: - DEFAULT except: - ENUM_NO_ALLOW_ALIAS - BASIC ignore ignore 选项指定忽略的文件，可以是文件名，也可以是目录，注意路径是要相对于 buf.yaml 文件的。 ...

Buf初入门1

Buf的目标是将API开发转向模式驱动的范式，从而为未来铺平道路，使API以服务所有者和客户可以依赖的方式定义。与简单地暴露REST/JSON服务相比，使用IDL来定义API有很多好处，今天，Protobuf是业界最稳定、最广泛采用的IDL。但就目前的情况来看，使用Protobuf比使用JSON作为数据传输格式要困难得多。 Buf正在建立工具，使Protobuf对服务所有者和客户来说是可靠和友好的，同时保持它在技术上的明显优势。您的组织不需要重新发明轮子来高效地创建、维护和使用Protobuf API。我们将为您处理您的Protobuf管理策略，因此您可以专注于重要的事情。本篇文章是讲述 buf 使用的第一篇文章，主要讲解使用 buf 定义 proto 文件的 lint 规则。统一的 lint 规则，可以让个人或团队定义的 API 保持一致。 lint 1 创建 buf.yaml 文件使用 buf mod init 命令创建 buf.yaml 文件。 version: v1 breaking: use: - FILE lint: use: - DEFAULT 使用默认 lint 规则。 2 运行 lint 命令使用 buf lint 命令运行 lint。 buf lint 命令将进行一下操作: buf.yaml根据您的配置发现所有 Protobuf 文件。将它们复制到内存中。编译它们根据配置的 lint 规则运行编译结果。 3 输出结果这是我的一个实例文件输出的结果: api\helloworld\v1\greeter.proto:5:8:google/api/annotations.proto: does not exist third_party\google\api\annotations.proto:19:8:google/api/http.proto: does not exist third_party\google\protobuf\descriptor.proto:57:1:duplicate symbol google.protobuf.FileDescriptorSet: already defined as message in "google/protobuf/des criptor.proto" third_party\google\protobuf\descriptor.proto:58:3:duplicate symbol google.protobuf.FileDescriptorSet.file: already defined as field in "google/protobuf/ descriptor.proto" third_party\google\protobuf\descriptor.proto:62:1:duplicate symbol google.protobuf.FileDescriptorProto: already defined as message in "google/protobuf/d escriptor.proto" third_party\google\protobuf\descriptor.proto:63:3:duplicate symbol google.protobuf.FileDescriptorProto.name: already defined as field in "google/protobu f/descriptor.proto" third_party\google\protobuf\descriptor.proto:64:3:duplicate symbol google.protobuf.FileDescriptorProto.package: already defined as field in "google/prot obuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:67:3:duplicate symbol google.protobuf.FileDescriptorProto.dependency: already defined as field in "google/p rotobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:69:3:duplicate symbol google.protobuf.FileDescriptorProto.public_dependency: already defined as field in "g oogle/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:72:3:duplicate symbol google.protobuf.FileDescriptorProto.weak_dependency: already defined as field in "goo gle/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:75:3:duplicate symbol google.protobuf.FileDescriptorProto.message_type: already defined as field in "google /protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:76:3:duplicate symbol google.protobuf.FileDescriptorProto.enum_type: already defined as field in "google/pr otobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:77:3:duplicate symbol google.protobuf.FileDescriptorProto.service: already defined as field in "google/prot obuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:78:3:duplicate symbol google.protobuf.FileDescriptorProto.extension: already defined as field in "google/pr otobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:80:3:duplicate symbol google.protobuf.FileDescriptorProto.options: already defined as field in "google/prot obuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:86:3:duplicate symbol google.protobuf.FileDescriptorProto.source_code_info: already defined as field in "go ogle/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:90:3:duplicate symbol google.protobuf.FileDescriptorProto.syntax: already defined as field in "google/proto buf/descriptor.proto" third_party\google\protobuf\descriptor.proto:94:1:duplicate symbol google.protobuf.DescriptorProto: already defined as message in "google/protobuf/descr iptor.proto" third_party\google\protobuf\descriptor.proto:95:3:duplicate symbol google.protobuf.DescriptorProto.name: already defined as field in "google/protobuf/de scriptor.proto" third_party\google\protobuf\descriptor.proto:97:3:duplicate symbol google.protobuf.DescriptorProto.field: already defined as field in "google/protobuf/d escriptor.proto" third_party\google\protobuf\descriptor.proto:98:3:duplicate symbol google.protobuf.DescriptorProto.extension: already defined as field in "google/protob uf/descriptor.proto" third_party\google\protobuf\descriptor.proto:100:3:duplicate symbol google.protobuf.DescriptorProto.nested_type: already defined as field in "google/pro tobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:101:3:duplicate symbol google.protobuf.DescriptorProto.enum_type: already defined as field in "google/proto buf/descriptor.proto" third_party\google\protobuf\descriptor.proto:103:3:duplicate symbol google.protobuf.DescriptorProto.ExtensionRange: already defined as message in "googl e/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:104:5:duplicate symbol google.protobuf.DescriptorProto.ExtensionRange.start: already defined as field in "g oogle/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:105:5:duplicate symbol google.protobuf.DescriptorProto.ExtensionRange.end: already defined as field in "goo gle/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:107:5:duplicate symbol google.protobuf.DescriptorProto.ExtensionRange.options: already defined as field in "google/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:109:3:duplicate symbol google.protobuf.DescriptorProto.extension_range: already defined as field in "google /protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:111:3:duplicate symbol google.protobuf.DescriptorProto.oneof_decl: already defined as field in "google/prot obuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:113:3:duplicate symbol google.protobuf.DescriptorProto.options: already defined as field in "google/protobu f/descriptor.proto" third_party\google\protobuf\descriptor.proto:118:3:duplicate symbol google.protobuf.DescriptorProto.ReservedRange: already defined as message in "google /protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:119:5:duplicate symbol google.protobuf.DescriptorProto.ReservedRange.start: already defined as field in "go ogle/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:120:5:duplicate symbol google.protobuf.DescriptorProto.ReservedRange.end: already defined as field in "goog le/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:122:3:duplicate symbol google.protobuf.DescriptorProto.reserved_range: already defined as field in "google/ protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:125:3:duplicate symbol google.protobuf.DescriptorProto.reserved_name: already defined as field in "google/p rotobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:128:1:duplicate symbol google.protobuf.ExtensionRangeOptions: already defined as message in "google/protobu f/descriptor.proto" third_party\google\protobuf\descriptor.proto:130:3:duplicate symbol google.protobuf.ExtensionRangeOptions.uninterpreted_option: already defined as field in "google/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:138:1:duplicate symbol google.protobuf.FieldDescriptorProto: already defined as message in "google/protobuf /descriptor.proto" third_party\google\protobuf\descriptor.proto:139:3:duplicate symbol google.protobuf.FieldDescriptorProto.Type: already defined as enum in "google/protob uf/descriptor.proto" third_party\google\protobuf\descriptor.proto:142:5:duplicate symbol google.protobuf.FieldDescriptorProto.TYPE_DOUBLE: already defined as enum value in " google/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:143:5:duplicate symbol google.protobuf.FieldDescriptorProto.TYPE_FLOAT: already defined as enum value in "g oogle/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:146:5:duplicate symbol google.protobuf.FieldDescriptorProto.TYPE_INT64: already defined as enum value in "g oogle/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:147:5:duplicate symbol google.protobuf.FieldDescriptorProto.TYPE_UINT64: already defined as enum value in " google/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:150:5:duplicate symbol google.protobuf.FieldDescriptorProto.TYPE_INT32: already defined as enum value in "g oogle/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:151:5:duplicate symbol google.protobuf.FieldDescriptorProto.TYPE_FIXED64: already defined as enum value in "google/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:152:5:duplicate symbol google.protobuf.FieldDescriptorProto.TYPE_FIXED32: already defined as enum value in "google/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:153:5:duplicate symbol google.protobuf.FieldDescriptorProto.TYPE_BOOL: already defined as enum value in "go ogle/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:154:5:duplicate symbol google.protobuf.FieldDescriptorProto.TYPE_STRING: already defined as enum value in " google/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:159:5:duplicate symbol google.protobuf.FieldDescriptorProto.TYPE_GROUP: already defined as enum value in "g oogle/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:160:5:duplicate symbol google.protobuf.FieldDescriptorProto.TYPE_MESSAGE: already defined as enum value in "google/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:163:5:duplicate symbol google.protobuf.FieldDescriptorProto.TYPE_BYTES: already defined as enum value in "g oogle/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:164:5:duplicate symbol google.protobuf.FieldDescriptorProto.TYPE_UINT32: already defined as enum value in " google/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:165:5:duplicate symbol google.protobuf.FieldDescriptorProto.TYPE_ENUM: already defined as enum value in "go ogle/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:166:5:duplicate symbol google.protobuf.FieldDescriptorProto.TYPE_SFIXED32: already defined as enum value in "google/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:167:5:duplicate symbol google.protobuf.FieldDescriptorProto.TYPE_SFIXED64: already defined as enum value in "google/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:168:5:duplicate symbol google.protobuf.FieldDescriptorProto.TYPE_SINT32: already defined as enum value in " google/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:169:5:duplicate symbol google.protobuf.FieldDescriptorProto.TYPE_SINT64: already defined as enum value in " google/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:172:3:duplicate symbol google.protobuf.FieldDescriptorProto.Label: already defined as enum in "google/proto buf/descriptor.proto" third_party\google\protobuf\descriptor.proto:174:5:duplicate symbol google.protobuf.FieldDescriptorProto.LABEL_OPTIONAL: already defined as enum value i n "google/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:175:5:duplicate symbol google.protobuf.FieldDescriptorProto.LABEL_REQUIRED: already defined as enum value i n "google/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:176:5:duplicate symbol google.protobuf.FieldDescriptorProto.LABEL_REPEATED: already defined as enum value i n "google/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:179:3:duplicate symbol google.protobuf.FieldDescriptorProto.name: already defined as field in "google/proto buf/descriptor.proto" third_party\google\protobuf\descriptor.proto:180:3:duplicate symbol google.protobuf.FieldDescriptorProto.number: already defined as field in "google/pro tobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:181:3:duplicate symbol google.protobuf.FieldDescriptorProto.label: already defined as field in "google/prot obuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:185:3:duplicate symbol google.protobuf.FieldDescriptorProto.type: already defined as field in "google/proto buf/descriptor.proto" third_party\google\protobuf\descriptor.proto:192:3:duplicate symbol google.protobuf.FieldDescriptorProto.type_name: already defined as field in "google/ protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:196:3:duplicate symbol google.protobuf.FieldDescriptorProto.extendee: already defined as field in "google/p rotobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:203:3:duplicate symbol google.protobuf.FieldDescriptorProto.default_value: already defined as field in "goo gle/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:207:3:duplicate symbol google.protobuf.FieldDescriptorProto.oneof_index: already defined as field in "googl e/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:213:3:duplicate symbol google.protobuf.FieldDescriptorProto.json_name: already defined as field in "google/ protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:215:3:duplicate symbol google.protobuf.FieldDescriptorProto.options: already defined as field in "google/pr otobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:238:3:duplicate symbol google.protobuf.FieldDescriptorProto.proto3_optional: already defined as field in "g oogle/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:242:1:duplicate symbol google.protobuf.OneofDescriptorProto: already defined as message in "google/protobuf /descriptor.proto" third_party\google\protobuf\descriptor.proto:243:3:duplicate symbol google.protobuf.OneofDescriptorProto.name: already defined as field in "google/proto buf/descriptor.proto" third_party\google\protobuf\descriptor.proto:244:3:duplicate symbol google.protobuf.OneofDescriptorProto.options: already defined as field in "google/pr otobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:248:1:duplicate symbol google.protobuf.EnumDescriptorProto: already defined as message in "google/protobuf/ descriptor.proto" third_party\google\protobuf\descriptor.proto:249:3:duplicate symbol google.protobuf.EnumDescriptorProto.name: already defined as field in "google/protob uf/descriptor.proto" third_party\google\protobuf\descriptor.proto:251:3:duplicate symbol google.protobuf.EnumDescriptorProto.value: already defined as field in "google/proto buf/descriptor.proto" third_party\google\protobuf\descriptor.proto:253:3:duplicate symbol google.protobuf.EnumDescriptorProto.options: already defined as field in "google/pro tobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:261:3:duplicate symbol google.protobuf.EnumDescriptorProto.EnumReservedRange: already defined as message in "google/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:262:5:duplicate symbol google.protobuf.EnumDescriptorProto.EnumReservedRange.start: already defined as fiel d in "google/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:263:5:duplicate symbol google.protobuf.EnumDescriptorProto.EnumReservedRange.end: already defined as field in "google/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:269:3:duplicate symbol google.protobuf.EnumDescriptorProto.reserved_range: already defined as field in "goo gle/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:273:3:duplicate symbol google.protobuf.EnumDescriptorProto.reserved_name: already defined as field in "goog le/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:277:1:duplicate symbol google.protobuf.EnumValueDescriptorProto: already defined as message in "google/prot obuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:278:3:duplicate symbol google.protobuf.EnumValueDescriptorProto.name: already defined as field in "google/p rotobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:279:3:duplicate symbol google.protobuf.EnumValueDescriptorProto.number: already defined as field in "google /protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:281:3:duplicate symbol google.protobuf.EnumValueDescriptorProto.options: already defined as field in "googl e/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:285:1:duplicate symbol google.protobuf.ServiceDescriptorProto: already defined as message in "google/protob uf/descriptor.proto" third_party\google\protobuf\descriptor.proto:286:3:duplicate symbol google.protobuf.ServiceDescriptorProto.name: already defined as field in "google/pro tobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:287:3:duplicate symbol google.protobuf.ServiceDescriptorProto.method: already defined as field in "google/p rotobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:289:3:duplicate symbol google.protobuf.ServiceDescriptorProto.options: already defined as field in "google/ protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:293:1:duplicate symbol google.protobuf.MethodDescriptorProto: already defined as message in "google/protobu f/descriptor.proto" third_party\google\protobuf\descriptor.proto:294:3:duplicate symbol google.protobuf.MethodDescriptorProto.name: already defined as field in "google/prot obuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:298:3:duplicate symbol google.protobuf.MethodDescriptorProto.input_type: already defined as field in "googl e/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:299:3:duplicate symbol google.protobuf.MethodDescriptorProto.output_type: already defined as field in "goog le/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:301:3:duplicate symbol google.protobuf.MethodDescriptorProto.options: already defined as field in "google/p rotobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:304:3:duplicate symbol google.protobuf.MethodDescriptorProto.client_streaming: already defined as field in "google/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:306:3:duplicate symbol google.protobuf.MethodDescriptorProto.server_streaming: already defined as field in "google/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:342:1:duplicate symbol google.protobuf.FileOptions: already defined as message in "google/protobuf/descript or.proto" third_party\google\protobuf\descriptor.proto:348:3:duplicate symbol google.protobuf.FileOptions.java_package: already defined as field in "google/protob uf/descriptor.proto" third_party\google\protobuf\descriptor.proto:356:3:duplicate symbol google.protobuf.FileOptions.java_outer_classname: already defined as field in "googl e/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:364:3:duplicate symbol google.protobuf.FileOptions.java_multiple_files: already defined as field in "google /protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:367:3:duplicate symbol google.protobuf.FileOptions.java_generate_equals_and_hash: already defined as field in "google/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:375:3:duplicate symbol google.protobuf.FileOptions.java_string_check_utf8: already defined as field in "goo gle/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:379:3:duplicate symbol google.protobuf.FileOptions.OptimizeMode: already defined as enum in "google/protobu f/descriptor.proto" third_party\google\protobuf\descriptor.proto:380:5:duplicate symbol google.protobuf.FileOptions.SPEED: already defined as enum value in "google/protobuf /descriptor.proto" third_party\google\protobuf\descriptor.proto:382:5:duplicate symbol google.protobuf.FileOptions.CODE_SIZE: already defined as enum value in "google/prot obuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:383:5:duplicate symbol google.protobuf.FileOptions.LITE_RUNTIME: already defined as enum value in "google/p rotobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:385:3:duplicate symbol google.protobuf.FileOptions.optimize_for: already defined as field in "google/protob uf/descriptor.proto" third_party\google\protobuf\descriptor.proto:392:3:duplicate symbol google.protobuf.FileOptions.go_package: already defined as field in "google/protobuf /descriptor.proto" third_party\google\protobuf\descriptor.proto:407:3:duplicate symbol google.protobuf.FileOptions.cc_generic_services: already defined as field in "google /protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:408:3:duplicate symbol google.protobuf.FileOptions.java_generic_services: already defined as field in "goog le/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:409:3:duplicate symbol google.protobuf.FileOptions.py_generic_services: already defined as field in "google /protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:410:3:duplicate symbol google.protobuf.FileOptions.php_generic_services: already defined as field in "googl e/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:416:3:duplicate symbol google.protobuf.FileOptions.deprecated: already defined as field in "google/protobuf /descriptor.proto" third_party\google\protobuf\descriptor.proto:420:3:duplicate symbol google.protobuf.FileOptions.cc_enable_arenas: already defined as field in "google/pr otobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:425:3:duplicate symbol google.protobuf.FileOptions.objc_class_prefix: already defined as field in "google/p rotobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:428:3:duplicate symbol google.protobuf.FileOptions.csharp_namespace: already defined as field in "google/pr otobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:434:3:duplicate symbol google.protobuf.FileOptions.swift_prefix: already defined as field in "google/protob uf/descriptor.proto" third_party\google\protobuf\descriptor.proto:438:3:duplicate symbol google.protobuf.FileOptions.php_class_prefix: already defined as field in "google/pr otobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:443:3:duplicate symbol google.protobuf.FileOptions.php_namespace: already defined as field in "google/proto buf/descriptor.proto" third_party\google\protobuf\descriptor.proto:448:3:duplicate symbol google.protobuf.FileOptions.php_metadata_namespace: already defined as field in "goo gle/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:453:3:duplicate symbol google.protobuf.FileOptions.ruby_package: already defined as field in "google/protob uf/descriptor.proto" third_party\google\protobuf\descriptor.proto:458:3:duplicate symbol google.protobuf.FileOptions.uninterpreted_option: already defined as field in "googl e/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:467:1:duplicate symbol google.protobuf.MessageOptions: already defined as message in "google/protobuf/descr iptor.proto" third_party\google\protobuf\descriptor.proto:486:3:duplicate symbol google.protobuf.MessageOptions.message_set_wire_format: already defined as field in "google/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:491:3:duplicate symbol google.protobuf.MessageOptions.no_standard_descriptor_accessor: already defined as f ield in "google/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:497:3:duplicate symbol google.protobuf.MessageOptions.deprecated: already defined as field in "google/proto buf/descriptor.proto" third_party\google\protobuf\descriptor.proto:522:3:duplicate symbol google.protobuf.MessageOptions.map_entry: already defined as field in "google/protob uf/descriptor.proto" third_party\google\protobuf\descriptor.proto:529:3:duplicate symbol google.protobuf.MessageOptions.uninterpreted_option: already defined as field in "go ogle/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:535:1:duplicate symbol google.protobuf.FieldOptions: already defined as message in "google/protobuf/descrip tor.proto" third_party\google\protobuf\descriptor.proto:540:3:duplicate symbol google.protobuf.FieldOptions.ctype: already defined as field in "google/protobuf/des criptor.proto" third_party\google\protobuf\descriptor.proto:541:3:duplicate symbol google.protobuf.FieldOptions.CType: already defined as enum in "google/protobuf/desc riptor.proto" third_party\google\protobuf\descriptor.proto:543:5:duplicate symbol google.protobuf.FieldOptions.STRING: already defined as enum value in "google/protob uf/descriptor.proto" third_party\google\protobuf\descriptor.proto:545:5:duplicate symbol google.protobuf.FieldOptions.CORD: already defined as enum value in "google/protobuf /descriptor.proto" third_party\google\protobuf\descriptor.proto:547:5:duplicate symbol google.protobuf.FieldOptions.STRING_PIECE: already defined as enum value in "google/ protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:554:3:duplicate symbol google.protobuf.FieldOptions.packed: already defined as field in "google/protobuf/de scriptor.proto" third_party\google\protobuf\descriptor.proto:567:3:duplicate symbol google.protobuf.FieldOptions.jstype: already defined as field in "google/protobuf/de scriptor.proto" third_party\google\protobuf\descriptor.proto:568:3:duplicate symbol google.protobuf.FieldOptions.JSType: already defined as enum in "google/protobuf/des criptor.proto" third_party\google\protobuf\descriptor.proto:570:5:duplicate symbol google.protobuf.FieldOptions.JS_NORMAL: already defined as enum value in "google/pro tobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:573:5:duplicate symbol google.protobuf.FieldOptions.JS_STRING: already defined as enum value in "google/pro tobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:576:5:duplicate symbol google.protobuf.FieldOptions.JS_NUMBER: already defined as enum value in "google/pro tobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:607:3:duplicate symbol google.protobuf.FieldOptions.lazy: already defined as field in "google/protobuf/desc riptor.proto" third_party\google\protobuf\descriptor.proto:613:3:duplicate symbol google.protobuf.FieldOptions.deprecated: already defined as field in "google/protobu f/descriptor.proto" third_party\google\protobuf\descriptor.proto:616:3:duplicate symbol google.protobuf.FieldOptions.weak: already defined as field in "google/protobuf/desc riptor.proto" third_party\google\protobuf\descriptor.proto:620:3:duplicate symbol google.protobuf.FieldOptions.uninterpreted_option: already defined as field in "goog le/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:628:1:duplicate symbol google.protobuf.OneofOptions: already defined as message in "google/protobuf/descrip tor.proto" third_party\google\protobuf\descriptor.proto:630:3:duplicate symbol google.protobuf.OneofOptions.uninterpreted_option: already defined as field in "goog le/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:636:1:duplicate symbol google.protobuf.EnumOptions: already defined as message in "google/protobuf/descript or.proto" third_party\google\protobuf\descriptor.proto:640:3:duplicate symbol google.protobuf.EnumOptions.allow_alias: already defined as field in "google/protobu f/descriptor.proto" third_party\google\protobuf\descriptor.proto:646:3:duplicate symbol google.protobuf.EnumOptions.deprecated: already defined as field in "google/protobuf /descriptor.proto" third_party\google\protobuf\descriptor.proto:651:3:duplicate symbol google.protobuf.EnumOptions.uninterpreted_option: already defined as field in "googl e/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:657:1:duplicate symbol google.protobuf.EnumValueOptions: already defined as message in "google/protobuf/des criptor.proto" third_party\google\protobuf\descriptor.proto:662:3:duplicate symbol google.protobuf.EnumValueOptions.deprecated: already defined as field in "google/pro tobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:665:3:duplicate symbol google.protobuf.EnumValueOptions.uninterpreted_option: already defined as field in " google/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:671:1:duplicate symbol google.protobuf.ServiceOptions: already defined as message in "google/protobuf/descr iptor.proto" third_party\google\protobuf\descriptor.proto:682:3:duplicate symbol google.protobuf.ServiceOptions.deprecated: already defined as field in "google/proto buf/descriptor.proto" third_party\google\protobuf\descriptor.proto:685:3:duplicate symbol google.protobuf.ServiceOptions.uninterpreted_option: already defined as field in "go ogle/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:691:1:duplicate symbol google.protobuf.MethodOptions: already defined as message in "google/protobuf/descri ptor.proto" third_party\google\protobuf\descriptor.proto:702:3:duplicate symbol google.protobuf.MethodOptions.deprecated: already defined as field in "google/protob uf/descriptor.proto" third_party\google\protobuf\descriptor.proto:707:3:duplicate symbol google.protobuf.MethodOptions.IdempotencyLevel: already defined as enum in "google/p rotobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:708:5:duplicate symbol google.protobuf.MethodOptions.IDEMPOTENCY_UNKNOWN: already defined as enum value in "google/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:709:5:duplicate symbol google.protobuf.MethodOptions.NO_SIDE_EFFECTS: already defined as enum value in "goo gle/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:710:5:duplicate symbol google.protobuf.MethodOptions.IDEMPOTENT: already defined as enum value in "google/p rotobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:712:3:duplicate symbol google.protobuf.MethodOptions.idempotency_level: already defined as field in "google /protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:716:3:duplicate symbol google.protobuf.MethodOptions.uninterpreted_option: already defined as field in "goo gle/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:729:1:duplicate symbol google.protobuf.UninterpretedOption: already defined as message in "google/protobuf/ descriptor.proto" third_party\google\protobuf\descriptor.proto:735:3:duplicate symbol google.protobuf.UninterpretedOption.NamePart: already defined as message in "google/ protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:736:5:duplicate symbol google.protobuf.UninterpretedOption.NamePart.name_part: already defined as field in "google/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:737:5:duplicate symbol google.protobuf.UninterpretedOption.NamePart.is_extension: already defined as field in "google/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:739:3:duplicate symbol google.protobuf.UninterpretedOption.name: already defined as field in "google/protob uf/descriptor.proto" third_party\google\protobuf\descriptor.proto:743:3:duplicate symbol google.protobuf.UninterpretedOption.identifier_value: already defined as field in "g oogle/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:744:3:duplicate symbol google.protobuf.UninterpretedOption.positive_int_value: already defined as field in "google/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:745:3:duplicate symbol google.protobuf.UninterpretedOption.negative_int_value: already defined as field in "google/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:746:3:duplicate symbol google.protobuf.UninterpretedOption.double_value: already defined as field in "googl e/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:747:3:duplicate symbol google.protobuf.UninterpretedOption.string_value: already defined as field in "googl e/protobuf/descriptor.proto" n "google/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:904:5:duplicate symbol google.protobuf.GeneratedCodeInfo.Annotation.begin: already defined as field in "goo gle/protobuf/descriptor.proto" third_party\google\protobuf\descriptor.proto:909:5:duplicate symbol google.protobuf.GeneratedCodeInfo.Annotation.end: already defined as field in "googl e/protobuf/descriptor.proto" 提示的错误会以如下形式展示: {文件} ：{行} ：{列} ：{错误消息} ...

Go统计代码测试覆盖率

今天来讲一讲如何统计 go 代码的测试覆盖率，主要是 cover 命令。 cover 基本用法 1 先简单写个函数和相应的测试，代码如下: func Max(a, b int) int { if a > b { return a } return b } 这个函数就是简单的比较大小，如果 a > b，返回 a，否则返回 b。测试代码如下 package main import "testing" func TestMax(t *testing.T) { type args struct { a int b int } tests := []struct { name string args args want int }{ { name: "a is larger than b", args: args{ a: 2, b: 1, }, want: 2, }, { name: "b is larger than a", args: args{ a: 1, b: 2, }, want: 2, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { if got := Max(tt.args.a, tt.args.b); got != tt.want { t.Errorf("Max() = %v, want %v", got, tt.want) } }) } } 2 go test -cover 能够统计出代码测试的覆盖率，这是一种比统计函数是否被调用更强悍的手法。我们执行这个命令。输出如下: ...

Kswapd0 is consuming a lot of CPU.

Issue Recently, a Linux server has been experiencing high CPU usage, causing the server to become unresponsive and other processes to fail to execute smoothly. By using the top command to check resource usage, it can be seen that the kswapd0 process is consuming a large amount of CPU resources. Cause and solution Based on my research, the kswapd0 process is responsible for managing virtual memory. Typically, a Linux system comprises RAM, swap, and EXT4 components. The EXT4 partition is used to store regular files and can be created on either an HDD or an SSD, though it is relatively slower compared to RAM. RAM, commonly referred to as memory, is used for high-speed program execution. The swap partition, on the other hand, is used as additional virtual memory and is normally allocated on physical disks, especially on machines with limited physical RAM. ...

Kswapd0消耗大量cpu

问题最近一台 Linux 服务器经常出现 cpu 过高，导致服务器卡顿，其他进程不能顺利进行。通过 top 命令查看资源的占用情况，可以看到 Kswapd0 进程消耗了大量的 cpu 资源。原因及解决通过搜索得知，Kswapd0 进程是用来管理虚拟内存的。一般的 Linux 都会有 RAM，swap, 和 EXT4 这几个部分，EXT4 分区就是用来存放一般的文件，可以在机械硬盘或者 SSD 上划分出 ext4 分区来保存文件，相对 RAM（内存）来说要稍微慢一些，RAM 就是日常所说的内存，用来做程序运行时的高速缓存，而 SWAP 是交换分区，一般在物理内存比较小的机器上会划分一块物理磁盘来作为 swap 分区。 swap 分区是一款虚拟的 RAM，一般在 HDD/SSD 上，当物理内存比较小的时侯，可能经常缺内存，那么系统就会使用 swap 分区，将物理内存中的内容搬迁到 swap 分区中暂存。当可用物理内存比较小的时候，kswapd0 进程就会将相对比较不常用的程序移动到 swap 分区中，这个时候就可能造成这些程序比较卡顿。例如一台机器的内存是 4G，而要运行一个需要 5G 内存的程序，那么至少有 1G 的内存会放到 swap 分区中。kswapd0 移动内存的过程就会造成 CPU 的大量使用。要解决这个问题有这样几个方式。 1 修改 /etc/sysctl.conf 文件 echo vm.swappiness=0 | sudo tee -a /etc/sysctl.conf 这里 0 表示物理内存还剩余的百分比，这个值的取值范围是 0-100，配置 0 也就意味着只有当没有物理内存可用时再执行 kswapd0. ...