Description
This example demonstrates two ways to unmarshal a json file that has configuration information describing API servers.
One technique uses the "encoding/json" package.
The other uses jsoncfgo config file reader package.
Golang Features
This golang code sample demonstrates the following go language features:
- switch
- struct
- slice
- map
- log
- fmt
- basic error handling
- encoding/json
- os.OpenFile
- os.Exit
3rd Party Libraries
- github.com/l3x/jsoncfgo
- github.com/go-goodies/go_utils
Note that we preface github.com/go-goodies/go_utils with a "u".
Now, we can reference go_utils functions like so: u.Dashes(80)
Which, by the way, prints 80 dashes to separate sections in our output.
Input Files - encoding/json
servicesjsonencoding.json
{
"development":{
"speech":[
{"name":"speech-server-1", "host":"127.0.0.1", "port":3050, "restPort":2050, "wsPort":3050},
{"name":"speech-server-2", "host":"127.0.0.1", "port":3051, "restPort":2051, "wsPort":3051},
{"name":"speech-server-3", "host":"127.0.0.1", "port":3052, "restPort":2052, "wsPort":3052}
],
"sms":[
{"name":"sms-server-1", "host":"127.0.0.1", "port":4050},
{"name":"sms-server-2", "host":"127.0.0.1", "port":4051},
{"name":"sms-server-3", "host":"127.0.0.1", "port":4052}
],
"payment":[
{"name": "payment-server-1", "host": "127.0.0.1", "restPort":2015, "wsPort": 3015}
]
},
"production":{
"speech":[
{"name":"speech-server-1", "host":"127.0.0.1", "port":3150, "restPort":2050, "wsPort":3050},
{"name":"speech-server-2", "host":"127.0.0.1", "port":3151, "restPort":2051, "wsPort":3051},
{"name":"speech-server-3", "host":"127.0.0.1", "port":3152, "restPort":2052, "wsPort":3052}
],
"sms":[
{"name":"sms-server-1", "host":"127.0.0.1", "port":4150},
{"name":"sms-server-2", "host":"127.0.0.1", "port":4151},
{"name":"sms-server-3", "host":"127.0.0.1", "port":4152}
],
"payment":[
{"name": "payment-server-1", "host": "127.0.0.1", "restPort":2050, "wsPort": 5050}
]
}
}
Code Example - encoding/json
In this code example we demonstrate how to use the "encoding/json" package:
package main
import(
"encoding/json"
"fmt"
"os"
)
type Service struct {
Name string `json:"name"`
Host string `json:"host"`
Port uint `json:"port"`
RestPort uint `json:"restPort"`
WsPort uint `json:"wsPort"`
}
func LoadApiServers(filepath, env string) (map[string][]Service, error) {
file, err := os.OpenFile(filepath, os.O_RDONLY, 0644)
if err != nil {
fmt.Println(err)
os.Exit(1)
}
configs := make(map[string]map[string][]Service, 0)
err = json.NewDecoder(file).Decode(&configs)
if err != nil {
fmt.Println(err)
os.Exit(1)
}
return configs[env], err
}
func main() {
//var configs map[string]map[string][]Service
pathToFile := "/Users/lex/dev/go/samples/src/bitbucket.org/l3x/unmarshal/services-json-encoding.json"
dev_configs, err := LoadApiServers(pathToFile, "development")
if err != nil {
fmt.Println(err)
os.Exit(1)
}
fmt.Printf("dev_configs: %v\n\n", dev_configs)
fmt.Printf("dev_configs[\"speech\"][2].Name: %v\n", dev_configs["speech"][2].Name)
fmt.Printf("dev_configs[\"sms\"][1].Host: %v\n", dev_configs["sms"][1].Host)
fmt.Printf("dev_configs[\"payment\"][0].Port: %v\n", dev_configs["sms"][0].Port)
}
Output - encoding/json
dev_configs: map[speech:[{speech-server-1 127.0.0.1 3050 2050 3050} {speech-server-2 127.0.0.1 3051 2051 3051} {speech-server-3 127.0.0.1 3052 2052 3052}] sms:[{sms-server-1 127.0.0.1 4050 0 0} {sms-server-2 127.0.0.1 4051 0 0} {sms-server-3 127.0.0.1 4052 0 0}] payment:[{payment-server-1 127.0.0.1 0 2015 3015}]]
dev_configs["speech"][2].Name: speech-server-3
dev_configs["sms"][1].Host: 127.0.0.1
dev_configs["payment"][0].Port: 4050
Process finished with exit code 0
Notes - encoding/json
The encoding/json does all the heavy lifting for us.
All we have to do is 1) Define the struct to contain the unmarshalled json ...
type Service struct {
Name string `json:"name"`
Host string `json:"host"`
Port uint `json:"port"`
RestPort uint `json:"restPort"`
WsPort uint `json:"wsPort"`
}
... 2) read the json file, 3) allocate space for the map of maps of Service slices and 4) use NewDecoder and Decode to populate our Service structs:
file, err := os.OpenFile(filepath, os.O_RDONLY, 0644)
configs := make(map[string]map[string][]Service, 0)
err = json.NewDecoder(file).Decode(&configs)
This is what NewDecoder and Decode look like under the covers:
NewDecoder
// NewDecoder returns a new decoder that reads from r.
//
// The decoder introduces its own buffering and may
// read data from r beyond the JSON values requested.
func NewDecoder(r io.Reader) *Decoder {
return &Decoder{r: r}
}
Decode
// Decode reads the next JSON-encoded value from its
// input and stores it in the value pointed to by v.
//
// See the documentation for Unmarshal for details about
// the conversion of JSON into a Go value.
func (dec *Decoder) Decode(v interface{}) error {
if dec.err != nil {
return dec.err
}
n, err := dec.readValue()
if err != nil {
return err
}
// Don't save err from unmarshal into dec.err:
// the connection is still usable since we read a complete JSON
// object from it before the error happened.
dec.d.init(dec.buf[0:n])
err = dec.d.unmarshal(v)
// Slide rest of data down.
rest := copy(dec.buf, dec.buf[n:])
dec.buf = dec.buf[0:rest]
return err
}
Unmarshal
To unmarshal JSON into an interface value, Unmarshal stores one of these in the interface value:
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
Let's take another look at the Service struct:
type Service struct {
Name string `json:"name"`
Host string `json:"host"`
Port uint `json:"port"`
RestPort uint `json:"restPort"`
WsPort uint `json:"wsPort"`
}
Note how each field definition is followed by instructions to encoding/json on how to handle its data.
// Field is ignored by this package.
Field int `json:"-"`
// Field appears in JSON as key "myName".
Field int `json:"myName"`
// Field appears in JSON as key "myName" and
// the field is omitted from the object if its value is empty,
// as defined above.
Field int `json:"myName,omitempty"`
// Field appears in JSON as key "Field" (the default), but
// the field is skipped if empty.
// Note the leading comma.
Field int `json:",omitempty"`
os.OpenFile
os.OpenFile is used to read the json file read-only.
os.Exit
os.Exit causes the current program to exit with the given status code
Input Files - jsoncfgo
services_jsoncfgo.json
{
"environments": {
"development":{
"speech":[
{"name":"speech-server-1", "host":"127.0.0.1", "port":3050, "restPort":2050, "wsPort":3050},
{"name":"speech-server-2", "host":"127.0.0.1", "port":3051, "restPort":2051, "wsPort":3051},
{"name":"speech-server-3", "host":"127.0.0.1", "port":3052, "restPort":2052, "wsPort":3052}
],
"sms":[
{"name":"sms-server-1", "host":"127.0.0.1", "port":4050},
{"name":"sms-server-2", "host":"127.0.0.1", "port":4051},
{"name":"sms-server-3", "host":"127.0.0.1", "port":4052}
],
"payment":[
{"name": "payment-server-1", "host": "127.0.0.1", "restPort":2015, "wsPort": 3015}
]
},
"production":{
"speech":[
{"name":"speech-server-1", "host":"127.0.0.1", "port":3150, "restPort":2050, "wsPort":3050},
{"name":"speech-server-2", "host":"127.0.0.1", "port":3151, "restPort":2051, "wsPort":3051},
{"name":"speech-server-3", "host":"127.0.0.1", "port":3152, "restPort":2052, "wsPort":3052}
],
"sms":[
{"name":"sms-server-1", "host":"127.0.0.1", "port":4150},
{"name":"sms-server-2", "host":"127.0.0.1", "port":4151},
{"name":"sms-server-3", "host":"127.0.0.1", "port":4152}
],
"payment":[
{"name": "payment-server-1", "host": "127.0.0.1", "restPort":2050, "wsPort": 5050}
]
}
}
}
Code Example - jsoncfgo
In this code example we demonstrate how to use the "jsoncfgo" package:
package main
import(
"sync"
"encoding/json"
"log"
"os"
"fmt"
"github.com/l3x/jsoncfgo"
u "github.com/go-goodies/go_utils"
"reflect"
)
type Speech struct {
Services []Service
}
type Sms struct {
Services []Service
}
type Payment struct {
Services []Service
}
type Environment struct {
Speech interface{}
Sms interface{}
Payment interface{}
}
type Service struct {
Name string `json:"name"`
Host string `json:"host"`
Port uint `json:"port"`
RestPort uint `json:"restPort"`
WsPort uint `json:"wsPort"`
}
type Config struct {
Services []Service
Master Service
Mutex sync.RWMutex
}
func LoadApiServers(filepath, env string) (map[string][]Service, error) {
file, err := os.OpenFile(filepath, os.O_RDONLY, 0644)
if err != nil {
fmt.Println(err)
os.Exit(1)
}
configs := make(map[string]map[string][]Service, 0)
err = json.NewDecoder(file).Decode(&configs)
if err != nil {
fmt.Println(err)
os.Exit(1)
}
return configs[env], err
}
func getService(svcMap interface{}) (*Service) {
connMap := svcMap.(map[string]interface{})
connConf := jsoncfgo.Obj(connMap)
thisSvc := &Service{
Host: connConf.OptionalString("host", ""),
Port: connConf.OptionalUint("port", 0),
RestPort: connConf.OptionalUint("restPort", 0),
WsPort: connConf.OptionalUint("wsPort", 0),
}
return thisSvc
}
func printEnvironment(envMap *Environment) {
el := reflect.ValueOf(envMap).Elem()
fldName := ""
fldType := reflect.ValueOf(envMap).Elem().Type()
var speech Speech
var sms Sms
var payment Payment
for i := 0; i < el.NumField(); i++ {
f := el.Field(i)
fldName = fldType.Field(i).Name
thisValAry := f.Interface().([]interface{})
switch fldName {
case "Speech":
for _, connectorMap := range thisValAry {
thisSvc := getService(connectorMap)
speech.Services = append(speech.Services, *thisSvc)
}
case "Sms":
for _, chatMap := range thisValAry {
thisSvc := getService(chatMap)
sms.Services = append(sms.Services, *thisSvc)
}
case "Payment":
for _, gatetMap := range thisValAry {
thisSvc := getService(gatetMap)
payment.Services = append(payment.Services, *thisSvc)
}
}
}
environment := &Environment{
Speech: speech,
Sms: sms,
Payment: payment,
}
fmt.Printf("environment: %+v\n", environment)
}
func main() {
pathToFile := "/Users/lex/dev/go/samples/src/bitbucket.org/l3x/unmarshal/services_jsoncfgo.json"
cfg, err := jsoncfgo.ReadFile(pathToFile)
if err != nil {
log.Fatal(err.Error()) // Handle error here
}
environmentsList := make(map[string]*Environment)
environments := cfg.OptionalObject("environments")
for alias, envMap := range environments {
servicesMap, ok := envMap.(map[string]interface{})
if !ok {
log.Fatalf("entry %q in environments section is a %T, want an object", alias, envMap)
}
servicesConf := jsoncfgo.Obj(servicesMap)
environment := &Environment{
Speech: servicesConf["speech"],
Sms: servicesConf["sms"],
Payment: servicesConf["payment"],
}
environmentsList[alias] = environment
fmt.Printf("environment.Speech: %v\n", environment.Speech)
speechAry := environment.Speech.([]interface{})
fmt.Printf("speechAry[0]: %v\n", speechAry[0])
fmt.Printf("speechAry[1]: %v\n", speechAry[1])
fmt.Printf("speechAry[2]: %v\n\n", speechAry[2])
fmt.Printf("environment.Sms: %v\n", environment.Sms)
smsAry := environment.Sms.([]interface{})
fmt.Printf("smsAry[0]: %v\n", smsAry[0])
fmt.Printf("smsAry[1]: %v\n", smsAry[1])
fmt.Printf("smsAry[2]: %v\n\n", smsAry[2])
fmt.Printf("environment.Payment: %v\n", environment.Payment)
paymentAry := environment.Payment.([]interface{})
fmt.Printf("paymentAry[0]: %v\n", paymentAry[0])
fmt.Println(u.Dashes(80))
}
fmt.Println(u.Dashes(80))
for alias, envMap := range environmentsList {
fmt.Printf("alias: %v\n", alias)
fmt.Printf("envMap: %v\n", envMap)
printEnvironment(envMap)
fmt.Println(u.Dashes(80))
}
}
Output - jsoncfgo
environment.Speech: [map[port:3050 restPort:2050 wsPort:3050 name:speech-server-1 host:127.0.0.1] map[restPort:2051 wsPort:3051 name:speech-server-2 host:127.0.0.1 port:3051] map[name:speech-server-3 host:127.0.0.1 port:3052 restPort:2052 wsPort:3052]]
speechAry[0]: map[wsPort:3050 name:speech-server-1 host:127.0.0.1 port:3050 restPort:2050]
speechAry[1]: map[name:speech-server-2 host:127.0.0.1 port:3051 restPort:2051 wsPort:3051]
speechAry[2]: map[name:speech-server-3 host:127.0.0.1 port:3052 restPort:2052 wsPort:3052]
environment.Sms: [map[name:sms-server-1 host:127.0.0.1 port:4050] map[name:sms-server-2 host:127.0.0.1 port:4051] map[host:127.0.0.1 port:4052 name:sms-server-3]]
smsAry[0]: map[name:sms-server-1 host:127.0.0.1 port:4050]
smsAry[1]: map[port:4051 name:sms-server-2 host:127.0.0.1]
smsAry[2]: map[name:sms-server-3 host:127.0.0.1 port:4052]
environment.Payment: [map[name:payment-server-1 host:127.0.0.1 restPort:2015 wsPort:3015]]
paymentAry[0]: map[wsPort:3015 name:payment-server-1 host:127.0.0.1 restPort:2015]
--------------------------------------------------------------------------------
environment.Speech: [map[host:127.0.0.1 port:3150 restPort:2050 wsPort:3050 name:speech-server-1] map[name:speech-server-2 host:127.0.0.1 port:3151 restPort:2051 wsPort:3051] map[port:3152 restPort:2052 wsPort:3052 name:speech-server-3 host:127.0.0.1]]
speechAry[0]: map[wsPort:3050 name:speech-server-1 host:127.0.0.1 port:3150 restPort:2050]
speechAry[1]: map[name:speech-server-2 host:127.0.0.1 port:3151 restPort:2051 wsPort:3051]
speechAry[2]: map[name:speech-server-3 host:127.0.0.1 port:3152 restPort:2052 wsPort:3052]
environment.Sms: [map[name:sms-server-1 host:127.0.0.1 port:4150] map[name:sms-server-2 host:127.0.0.1 port:4151] map[name:sms-server-3 host:127.0.0.1 port:4152]]
smsAry[0]: map[host:127.0.0.1 port:4150 name:sms-server-1]
smsAry[1]: map[name:sms-server-2 host:127.0.0.1 port:4151]
smsAry[2]: map[name:sms-server-3 host:127.0.0.1 port:4152]
environment.Payment: [map[restPort:2050 wsPort:5050 name:payment-server-1 host:127.0.0.1]]
paymentAry[0]: map[name:payment-server-1 host:127.0.0.1 restPort:2050 wsPort:5050]
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
alias: development
envMap: &{[map[host:127.0.0.1 port:3050 restPort:2050 wsPort:3050 name:speech-server-1] map[port:3051 restPort:2051 wsPort:3051 name:speech-server-2 host:127.0.0.1] map[restPort:2052 wsPort:3052 name:speech-server-3 host:127.0.0.1 port:3052]] [map[host:127.0.0.1 port:4050 name:sms-server-1] map[port:4051 name:sms-server-2 host:127.0.0.1] map[name:sms-server-3 host:127.0.0.1 port:4052]] [map[host:127.0.0.1 restPort:2015 wsPort:3015 name:payment-server-1]]}
environment: &{Speech:{Services:[{Name: Host:127.0.0.1 Port:3050 RestPort:2050 WsPort:3050} {Name: Host:127.0.0.1 Port:3051 RestPort:2051 WsPort:3051} {Name: Host:127.0.0.1 Port:3052 RestPort:2052 WsPort:3052}]} Sms:{Services:[{Name: Host:127.0.0.1 Port:4050 RestPort:0 WsPort:0} {Name: Host:127.0.0.1 Port:4051 RestPort:0 WsPort:0} {Name: Host:127.0.0.1 Port:4052 RestPort:0 WsPort:0}]} Payment:{Services:[{Name: Host:127.0.0.1 Port:0 RestPort:2015 WsPort:3015}]}}
--------------------------------------------------------------------------------
alias: production
envMap: &{[map[name:speech-server-1 host:127.0.0.1 port:3150 restPort:2050 wsPort:3050] map[restPort:2051 wsPort:3051 name:speech-server-2 host:127.0.0.1 port:3151] map[host:127.0.0.1 port:3152 restPort:2052 wsPort:3052 name:speech-server-3]] [map[port:4150 name:sms-server-1 host:127.0.0.1] map[name:sms-server-2 host:127.0.0.1 port:4151] map[name:sms-server-3 host:127.0.0.1 port:4152]] [map[name:payment-server-1 host:127.0.0.1 restPort:2050 wsPort:5050]]}
environment: &{Speech:{Services:[{Name: Host:127.0.0.1 Port:3150 RestPort:2050 WsPort:3050} {Name: Host:127.0.0.1 Port:3151 RestPort:2051 WsPort:3051} {Name: Host:127.0.0.1 Port:3152 RestPort:2052 WsPort:3052}]} Sms:{Services:[{Name: Host:127.0.0.1 Port:4150 RestPort:0 WsPort:0} {Name: Host:127.0.0.1 Port:4151 RestPort:0 WsPort:0} {Name: Host:127.0.0.1 Port:4152 RestPort:0 WsPort:0}]} Payment:{Services:[{Name: Host:127.0.0.1 Port:0 RestPort:2050 WsPort:5050}]}}
--------------------------------------------------------------------------------
Process finished with exit code 0
Notes
We define a struct to contain our environments ("development" and "production"):
type Environment struct {
Speech interface{}
Sms interface{}
Payment interface{}
}
... and the familiar Service struct:
type Service struct {
Name string `json:"name"`
Host string `json:"host"`
Port uint `json:"port"`
RestPort uint `json:"restPort"`
WsPort uint `json:"wsPort"`
}
jsconfgo handles the unmarshalling; However, to access individual elements we must use reflection, type assertion and a switch:
el := reflect.ValueOf(envMap).Elem()
fldType := reflect.ValueOf(envMap).Elem().Type()
thisValAry := f.Interface().([]interface{})
// . . .
switch fldName {
case "Speech":
for _, connectorMap := range thisValAry {
thisSvc := getService(connectorMap)
speech.Services = append(speech.Services, *thisSvc)
}
case "Sms":
for _, chatMap := range thisValAry {
thisSvc := getService(chatMap)
sms.Services = append(sms.Services, *thisSvc)
}
case "Payment":
for _, gatetMap := range thisValAry {
thisSvc := getService(gatetMap)
payment.Services = append(payment.Services, *thisSvc)
}
Summary
Given the format of the json data and our requirements, using the encoding/json is the better choice.
Use jsoncfgo when...
- You just need the values from the config file
- You want to perform validation (without extra coding effort)
- You want to be able to indicate which config file values are required or optional
- You want to include a file path reference to another json config file to be unmarshalled
Use encoding/json when...
- You want to unmarshal the json data into a struct so you can access it's members more easily