Go: unify go.mod and tools.mod

[florianl]

renovate seems to struggle to handle multiple *.mod files. Therefore merge tools.mod into go.mod to have a single dependency management file.

EDIT: We ended up moving tools.mod to internal/tools/go.mod and kept it separate from our main go.mod.

[christos68k]

Can you elaborate a little more on what the issue with renovate is? Dumping all the tool dependencies into our main go.mod (e.g. look at how many dependencies we're now adding) doesn't seem like a good idea?

[florianl]

Dumping all the tool dependencies into our main go.mod (e.g. look at how many dependencies we're now adding) doesn't seem like a good idea?

I think, it is acutally the other way around. Having a single go.mod avoids a split brain situation for dependencies. Just see the amount of duplicate entries, that have been in tools.mod, that are no longer tracked twice. As a quick check see the "impact" of this PR, which is currently +1,215 and -1,564 - so we actually reducing lines.

Can you elaborate a little more on what the issue with renovate is?

If you look into the past Go dependency upates done by renovate, like #1337, #1332 or #1320, you will notice that I have to step in every time and manually fix the shortcomings of renovte. Renovate does not suceed on running go mod tidy if there is more than a single go.mod file.

Does the size of go.mod impact resulting executables? No
The Go modfile keeps track of direct and indirect dependencies of the code, but also for the tests. So you will find already dependencies in go.mod, that are not used in code other than tests. The same applies for tools, that now can get tracked in go.mod files.

Suggesting to go with two Go modfiles in the first place was a mistake, I'm regreting now.

[christos68k]

Dumping all the tool dependencies into our main go.mod (e.g. look at how many dependencies we're now adding) doesn't seem like a good idea?

I think, it is acutally the other way around. Having a single go.mod avoids a split brain situation for dependencies. Just see the amount of duplicate entries, that have been in tools.mod, that are no longer tracked twice. As a quick check see the "impact" of this PR, which is currently +1,215 and -1,564 - so we actually reducing lines.

The main issue I have in mind with mixing dependencies is that we can't specify different versions of the same dependency in the same .mod file. A tool dependency upgrade can break our code or have other unintended consequences. So from a separation of concerns PoV, merging dependency graphs is not a good idea. Our code should not (implicitly or explicitly) depend on tools and vice-versa.

Can you elaborate a little more on what the issue with renovate is?

If you look into the past Go dependency upates done by renovate, like #1337, #1332 or #1320, you will notice that I have to step in every time and manually fix the shortcomings of renovte. Renovate does not suceed on running go mod tidy if there is more than a single go.mod file.

OK so the main problem is that gomodTidy in postUpdateOptions doesn't take effect if there are multiple .mod files and we can't use postUpgradeTasks since we don't self-host renovate.

Let me dig a bit more.

[florianl]

The main issue I have in mind with mixing dependencies is that we can't specify different versions of the same dependency in the same .mod file. A tool dependency upgrade can break our code or have other unintended consequences.

Taking the tool golangci-lint as an example. Both, our code and the tool, depend on the same dependencies but with different versions:

Dependency	golangci-lint	ebpf profiler
google.golang.org/grpc	v1.19.0	v1.80.0
google.golang.org/protobuf	v0.0.0-20200109...	v1.36.11
github.com/golang/protobuf	v1.2.0	v1.5.4
github.com/cespare/xxhash/v2	v2.1.1	v2.3.0
golang.org/x/sync	v0.0.0-2014	v0.20.0
golang.org/x/mod	v0.0.0-2013	v0.34.0

So - we have the case where tools and our code use the same dependency in different version (including major version changes).

[christos68k]

The main issue I have in mind with mixing dependencies is that we can't specify different versions of the same dependency in the same .mod file. A tool dependency upgrade can break our code or have other unintended consequences.

Taking the tool golangci-lint as an example. Both, our code and the tool, depend on the same dependencies but with different versions:

Dependency golangci-lint ebpf profiler
google.golang.org/grpc v1.19.0 v1.80.0
google.golang.org/protobuf v0.0.0-20200109... v1.36.11
github.com/golang/protobuf v1.2.0 v1.5.4
github.com/cespare/xxhash/v2 v2.1.1 v2.3.0
golang.org/x/sync v0.0.0-2014 v0.20.0
golang.org/x/mod v0.0.0-2013 v0.34.0
So - we have the case where tools and our code use the same dependency in different version (including major version changes).

How did you generate this table? The versions I see in tools.mod are different. AFAIK we can't have multiple versions for the same dependency in the same .mod file (isn't that the main driver for having a separate tools.mod file in Go projects?)

[florianl]

How did you generate this table?

[main]$ grep "^google.golang.org/grpc" tools.sum | head -3
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38=
google.golang.org/grpc v1.21.1/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ijfRaM=
[main]$  grep "^google.golang.org/grpc" go.sum | head -3
google.golang.org/grpc v1.80.0 h1:Xr6m2WmWZLETvUNvIUmeD5OAagMw3FiKmMlTdViWsHM=
google.golang.org/grpc v1.80.0/go.mod h1:ho/dLnxwi3EDJA4Zghp7k2Ec1+c2jqup0bFkw07bwF4=

I'm looking at .sum, as .sum contains direct and transitive dependencies. Overall, if a tool causes issue, I would question the tool over code in this project and look into replacing the tool.

[christos68k]

How did you generate this table?

[main]$ grep "^google.golang.org/grpc" tools.sum | head -3
[main]$ grep "^google.golang.org/grpc" go.sum | head -3

Isn't this looking at only the .sum files (which can contain checksums for different versions of the same module)?

The issue I'm outlining is that if we move all dependencies into a single go.mod file (which is what dictates the dependency versions used at build), we can't have multiple versions of the same dependency. That introduces tight coupling between tools and the rest of the code.

[christos68k]

Some options:

1. Remove tools.mod from renovate configuration
2. Keep it in the renovate configuration, but move tools.mod to a different directory than go.mod

1 is the easy fix as renovate won't be managing tools.mod any more. Since these are tools that are not part of our deliverables, maybe that's acceptable.

I'm still investigating 2.

[rogercoll]

Keep it in the renovate configuration, but move tools.mod to a different directory than go.mod

Sounds like a similar solution to the internal package used in https://github.com/elastic/opentelemetry-collector-components/tree/main/internal/tools + we could still use go tool -modfile=internal/tools/go.mod ..

[florianl]

I have updated the PR and moved tools.[go|mod] into internal/tools/.

[christos68k]

I can't merge yet as there are conflicts.

[florianl]

Resolved merge conflicts ✅