Efficiently marshaling large JSON data into structs with custom types

Question

I'm working on a Go application where I need to efficiently marshal a large JSON dataset into structs. The dataset contains complex nested structures and custom types. I've encountered performance issues when dealing with particularly large JSON files.

The JSON structure is as follows (simplified for brevity):

{
  "data": [
    {
      "id": "123",
      "attributes": {
        "name": "Example",
        "details": {
          "type": "customType",
          "info": "Some info"
        }
      }
    }
    // More items...
  ]
}

I've defined corresponding Go structs, including a custom type for handling specific fields:

type DataSet struct {
    Data []DataItem `json:"data"`
}

type DataItem struct {
    ID         string     `json:"id"`
    Attributes Attributes `json:"attributes"`
}

type Attributes struct {
    Name    string `json:"name"`
    Details Detail `json:"details"`
}

type Detail struct {
    Type CustomType `json:"type"`
    Info string     `json:"info"`
}

// CustomType with custom unmarshaling logic
type CustomType string

func (ct *CustomType) UnmarshalJSON(b []byte) error {
    // Custom unmarshaling logic
}

The issue arises when marshaling large datasets with the json.Unmarshal function, leading to high memory usage and slow performance.

Are there more efficient techniques in Go for handling large JSON datasets with custom unmarshaling logic?

Could the struct design be optimized to improve the performance of JSON unmarshaling?

Go version: 1.18. The JSON files range from 100 MB to 1 GB in size. I've considered using streaming JSON parsers, but I'm unsure how to integrate with custom types.

I've tried the following.

• Standard json.Unmarshal with the provided struct setup.
• Experimented with reducing the depth of nested structs.