Phuong Le

I have made thousands of static visualizations for Go, it's time to try something new: youtube.com/watch?v=fwHok9…

VictoriaMetrics v1.138.0 introduces JWT authentication in vmauth with claim matching, claim-based request templating, and OIDC public key discovery. It works with VictoriaMetrics and VictoriaLogs.

For VictoriaMetrics/VictoriaLogs users: share your own story! That helps a lot!

@ahmetb I'm not sure about the largest, but one of the big setups is around 3.5 PiB over 30 days

@func25 amazing work! what are the largest tsdb size or ts count the system is currently dealing with in the real world right now?

kinda crazy how VictoriaMetrics/VictoriaLogs came out of nowhere and got popular. I have no idea where they rank as an APM. what are primary reasons people are choosing this solution? is there a better scalability story?

The built-in web UI in VictoriaLogs gained the ability for the quick filtering by log stream fields starting from v1.48.0 release. Try it now - docs.victoriametrics.com/victorialogs/c… !

New article: Inside Go's Runtime Scheduler 🚀 How Go runs millions of goroutines on a few OS threads. The GMP model, work stealing, spinning threads, and context switches that take only 50-100 nanoseconds (10-40x faster than OS threads). 👉 internals-for-interns.com/posts/go-runti… #golang

I guess he meant that you create multiple small, root-level domain packages, not one giant one In internal/customer you put only customer domain types and interfaces. In internal/order you put only order domain types and interfaces. In internal/billing you put only billing domain types and interfaces.

@func25 Thank you for this great summary! What is your opinion on putting entity structs into a root package, so all sub-package can import them? As recommended for example by Ben Johnson: gobeyond.ghost.io/packages-as-la…

5 ways to solve import cycles in Go. Go import cycles mean package A imports package B and package B imports package A, such as when Customer has []Order and Order has Customer. Go blocks this on purpose as: - The compiler builds one package at a time and it must start from packages that import nothing. A cycle breaks that order. - Another big reason is startup order. Global variables and init functions run in import order. With a cycle there is no clear order, so code can read a global value before it is ready. The most useful idea here is to treat an import cycle as a design smell. You are coupling 2 features together, if you want to move a feature to another project, you also need to move another one. So here are 5 ways to fix them. Assume that: - package customer: object Customer contains []Order - package order: object Order contains Customer --- 1. using IDs instead of full objects This is the most common design in real systems. Instead of storing a whole Customer inside Order, you only store CustomerID. Instead of storing a list of Order objects inside Customer, you store OrderIDs or nothing at all. ``` package customer type Customer struct { ID int64 Name string OrderIDs []int64 } --- package order type Order struct { ID int64 Product string CustomerID int64 } ``` When the program needs the real data, another layer loads it. For example, storage code sees CustomerID and fetches the Customer from the database. --- 2. uses small interfaces Instead of holding a concrete Customer type inside Order, the order package defines a tiny interface that describes what it needs from a customer. The order package does not import customer package. Any type that satisfies the interface can be used. ``` package order type Customer interface { Pay() int64 Name() string } type Order struct { ID int64 Product string Customer Customer } ``` Now Customer implements the interface implicitly. order package does not know about the concrete Customer type. A higher level package wires them together. --- 3. uses a wrapper package for combined views Keep Customer and Order clean and independent. Then create a third package that imports both and builds a combined struct used for specific screens or API responses. ``` package customer type Customer struct { ID int64 Name string } package order type Order struct { ID int64 Product string CustomerID int64 } package customerorder type CustomerWithOrders struct { customer.Customer Orders []order.Order } ``` This type is not a core domain object. It is a view model, a shape convenient for one response or one page. You load data from storage and assemble it. This pattern is very common in APIs and UI backends. The limitation is that the relationship lives only in this wrapper, not inside the base structs. --- 4. uses one bounded model package Put both structs in the same package. Because they share the same namespace, they can reference each other directly without imports. This is the only way to truly have Customer contain []Order and Order contain *Customer as real fields at the same time: ``` package cusordermodel type Customer struct { ID int64 Name string Orders []*Order } type Order struct { ID int64 Product string Customer *Customer } ``` Other layers import model, but model imports nothing. This keeps dependencies pointing inward. It is clean when Customer and Order are tightly coupled and always evolve together. Note: the model package does not mean a global model. It is a local model package only for Customer and Order. The tradeoff is that the model package can grow large if you put too many types in it. --- 5. this is a design issue, fix it Most of the time you do not need both directions. Pick one owner. Usually Order knows Customer, but Customer does not need to store all orders. If you need a customer with orders, query orders by CustomerID. This keeps the model simple and avoids heavy memory use. Bidirectional links often cause problems like large object graphs and harder updates. If you put everything inside Customer, things like []Order, []Address, []PaymentMethod, []Review, []Notification, the struct can grow very large over time. It becomes a "god object". Every new feature wants to attach something to Customer because it feels natural. After a while the type is heavy, hard to understand, and expensive to load into memory. --- In large production systems, 1 and 5 are the most common. 4 appears when types are tightly coupled within the same domain. 2 and 3 are situational tools used for decoupling or building response shapes.

This is not something I would like to argue, it’s too minor as long as the team follows a convention. Personally, I think if it is a domain concept, singular often fits. If it is a toolbox around a concept, plural can fit better? What do you think about strings, bytes, and errors?

I just don't agree much with package names being in plural. The "Package names" blog post from 2015 and Effective Go doc has arguments about that, as they are generally singular nouns. Plural names such as strings, bytes, errors exists only to avoid conflicts with the built-in types or reserved keywords. The thing here is also about semantics as the lowest unit of code in Go are not files but packages and they are the ones being used by the user. The call sites can read less naturally as users.Create() is really odd compared to user.Create(), they should like collections instead of a domain thing as it would imply that a package is just a bag of many things instead of the concept behind it.

If you are building a Go API and you are stuck on folder structure, then structure your code around features, not around technical layers. Many beginners copy a layout like project/ │── cmd/ │ └── api/ │── internal/ │ │── handlers/ │ │── services/ │ │── repository/ │ │── models/ │ │── middleware/ │ │── database/ │ └── migrations/ and put all user files in each of those folders. It looks professional. But as the app grows, one feature gets scattered across many places. A single change to "users" can require jumping through 5 folders. That makes ownership unclear and slows debugging. --- A more scalable solution is feature-first. Put related code together: - a users package can contain what users need, - a posts package can contain what posts need, and - shared code can live in a small shared package only when truly needed. Inside each feature, you can still separate files by role if necessary. This gives better locality: when you modify a feature, most of the code you need is close together. project/ │── cmd/ │ └── api/ │── internal/ │ │── users/ │ │ │── handler.go │ │ │── service.go │ │ │── repository.go │ │ │── types.go │ │ └── routes.go │ |── posts/ │ |── invoice/ --- Do not over-design too early. A lot of teams start with fewer packages than they think they need. Sometimes one package is enough for a while. Then split when pain appears, not before. Go style generally favors practical simplicity over big upfront structure plans. --- A related point is naming noise. - File names do not need to repeat the package name, - object names do not need to repeat the package name, and - method names do not need to repeat the type name. If the package/type already tells the story, shorten the inner names. It improves readability fast. --- Keep types near where they are used, instead of creating a giant global “models.go” bucket. And if two features start depending on each other, do not let them import each other directly, define smaller interfaces at the consumer side and keep dependency direction one-way. --- So the real goal is simple: when you add or change one business capability, you should mostly work in one place. - If a package has too many unrelated reasons to change, it’s mixed. - If every small change forces edits across many packages, boundaries are wrong.

@LielAlmog @cdruc Why? Put the shared model in C. C is a feature. User, Post, and "Post/User interaction" are features.

@func25 @cdruc You still did not resolve his issue How would you avoid this duplication when using the feature approach

@cdruc I like a higher-level package (not a higher-level type package). Let's say A needs B and B needs A. Then C combines both A and B and only does things that require both A and B. It should have both models {A; B} and the business functions

@func25 The only way I see around this is either a higher types package, or somewhat duplicating the types. post has an posts.Author type, not user.User media has a media.Owner, not user.User user has []user.Post isolated but kind of duplicated? :(

@rnowtche Maybe, but I don't have this problem. When I type "user routes" or "users/routes" in search bar, it shows users/routes.go, for example

@func25 "- File names do not need to repeat the package name," This can become quite confusing in your code editor, if your project grows and you have many routes.go open or while searching for a file within a project, if many have the same name. It is manageable but still not ideal.

The advice is to avoid having a global models.go file or models/. The relaxed version is to have local- models/ folders or models_*.go files Usually a circular dependency points to a design issue. (it means the boundaries are wrong): - packages know too much about each other - responsibilities are mixed - shared types are not clearly separated from feature logic how to decompose it appliedgo.net/spotlight/circ…

@func25 I tried the feature approach a bunch of times and I always run into... circular dependency. >> Keep types near where they are used, instead of creating a giant global “models.go”. I do create a separate models/domain package to hold the domain types. Any other way around this?

@asynctear Should not be used, but if it is, use it only in main()

@func25 what error log fatal?

Panic and recover in Go should generally not be used, but knowing when to use them is an engineering decision. --- Panic Go has two clear lanes for failure: - Normal failure, returned as error with multiple return values. - Programmer bug or corrupted state, handled by panic. Panic is for cases where continuing makes no sense. This is usually when you believe it should not happen in your code, but if it does, it will corrupt or mislead the results or data. Examples are: - invalid internal invariants, - impossible branches, and must not happen conditions after you already validated inputs. - another acceptable place is early lifecycle, like bootstrap and dependency injection. If a database DSN is missing, or a required env var is empty, failing fast and letting the container restart is better than running half alive and printing logs forever. That is not user input validation but deployment correctness. --- Recover Recover has an even narrower role, it is mainly for boundary layers that must protect a long running process from one bad request or one bad goroutine. The net/http server in the Go standard library does this. If a handler triggers a panic like a nil pointer access, the server can recover, close the connection cleanly, and the client gets a 500 error instead of a hung socket or a dead server. If you want to avoid using recover in net/http, see: github.com/golang/go/issu… If you write an HTTP framework, an RPC server, a message consumer, or any worker pool that runs untrusted handler code, a top level recover wrapper can be a good design. The wrapper should log the panic value, include a stack trace, mark the request as failed, and keep the process alive. Using recover deep inside normal application code is usually a smell, because it hides bugs and turns clear failures into weird partial state. If you do recover, be strict about what you recover. Recovering everything can mask real runtime faults. A safer pattern is to panic only with a type you own, then recover only that type. --- Also, panic and recover are slow, often force heap allocations, and can block inlining. Even if your service survives, you pay for it in CPU and latency, and you also lose clarity about what can fail.

Go works best when the zero value of a type is valid and useful, such as T{}, and forcing constructor NewT use everywhere fights that design. Having a linter that forces NewT is also worse because it spreads constructor noise. - Go style guidance like least mechanism and the idea YAGNI push back on that: #style-principles" target="_blank" rel="nofollow noopener">google.github.io/styleguide/go/… - Dave Cheney also has a well known post about why good zero values matter: dave.cheney.net/2013/01/19/wha… It's recommended to have a meaningful zero value for a field, but better Go patterns depend on the case because every rule has exceptions. If a value really cannot work without setup, hiding the concrete type inside the package works well. - Export an interface like Thinger{}, - keep type thinger{} unexported, - only export NewThinger{} that returns Thinger{}. - Then outside code cannot write thinger{} at all. If zero value can work sometimes, then forcing New is wrong. bytes.Buffer is a real standard example, it has constructors like bytes.NewBuffer but the zero value of bytes.Buffer is still safe to use. A linter that always requires constructors would flag correct code, unless it has many allow lists and suppressions. Some types truly have unsafe zero values, reflect.Value is a famous case. So choose one of 3 clear policies per type: 1. Make the zero value safe and document it, or 2. hide the type and force construction by not exporting it, or 3. accept that some fields need nil checks and explicit init, like nil maps which panic on writes, then guard in methods and return a clear error early.

Database for terabytes of logs github.com/VictoriaMetric…