How to Merge two Json files into one using C++?

Question

I have 2 Json files having 100,000 records and 5 columns each. My goal is to merge both files to get a single file with 200k records using C++.

My json files look like this:

[ 
    { 
    "city":"Lisbon",
    "name":"Shannon Ward",
    "job":"Ambulatory Nurse",
    "email":"[email protected]",
    "salary":"2308593.84555454"
    },
    { 
    "city":"Lancaster",
    "name":"Kurt Hopkinson",
    "job":"Lecturer",
    "email":"[email protected]",
    "salary":"8564088.47550575"
    },
    { 
    "city":"Glendale",
    "name":"John Yates",
    "job":"Accountant",
    "email":"[email protected]",
    "salary":"9003114.81075051"
    },
    { 
    "city":"Sacramento",
    "name":"Camila Wallace",
    "job":"Budget Analyst",
    "email":"[email protected]",
    "salary":"1287392.46738351"
    }
]

which is basically an array holding structure (or map).

I am using JsonCpp library .

What I have tried:

Json::Value records1; 
Json::Value records2;

Json::Reader reader;  

reader.parse(file1, records1);    //file1 is input stream from 1st json file
reader.parse(file2, records2);    //file2 is input stream from 2nd json file

for (auto& itr : records2)
    records1.append(itr);

So, basically I am parsing json data from both files into Json::Value objects, and then later trying to append 1 object with elements of other object 1 by 1.

But, It's not working and compiler is giving this error:

C++ no instance of overloaded function matches the argument list and object (the object has type qualifiers that prevent a match)
argument types are: (const Json::Value)
object type is: const Json::Value

Please let me know what I am doing wrong or what's the right approach to append 2 json files in C++

Answer 1

Unless you're really set on using jsoncpp, I'd consider using ThorSerializer (installable via brew) library instead. For a quick demo:

#include <sstream>
#include <iostream>
#include <vector>
#include <map>
#include <string>

#include "ThorSerialize/JsonThor.h"

using namespace std::string_literals;

std::stringstream a( R"(
[
    {
        "city":"Lisbon",
        "name":"Shannon Ward",
        "job":"Ambulatory Nurse",
        "email":"[email protected]",
        "salary":"2308593.84555454"
    },
    {
        "city":"Lancaster",
        "name":"Kurt Hopkinson",
        "job":"Lecturer",
        "email":"[email protected]",
        "salary":"8564088.47550575"
    }
]
)"s);

std::stringstream b( R"(
[
    {
        "city":"Glendale",
        "name":"John Yates",
        "job":"Accountant",
        "email":"[email protected]",
        "salary":"9003114.81075051"
    },
    {
        "city":"Sacramento",
        "name":"Camila Wallace",
        "job":"Budget Analyst",
        "email":"[email protected]",
        "salary":"1287392.46738351"
    }
])"s);

int main() {

    using ThorsAnvil::Serialize::jsonImporter;
    using ThorsAnvil::Serialize::jsonExporter;

    std::vector<std::map<std::string, std::string>> data;

    a >> jsonImporter(data);
    b >> jsonImporter(data);

    std::cout << jsonExporter(data) << "\n";
}

The advantage of ThorsSerializer over other libraries is that it parses data directly into the C++ types that you are using, rather than into a generic JSON like structure that must then be iterated over to convert into your C++ types.

All the standard types and containers are handled by default. There is no code declaration style used to add new user defined types.

Example parsing JSON into C++ class.

Answer 2

Unless you're really set on using jsoncpp, I'd consider using Neils Lohmann's json library instead. For a quick demo:

#include <iostream>

#include "nlohmann/json.hpp"

std::string a = R"(
[
    {
        "city":"Lisbon",
        "name":"Shannon Ward",
        "job":"Ambulatory Nurse",
        "email":"[email protected]",
        "salary":"2308593.84555454"
    },
    {
        "city":"Lancaster",
        "name":"Kurt Hopkinson",
        "job":"Lecturer",
        "email":"[email protected]",
        "salary":"8564088.47550575"
    }
]
)";

std::string b = R"(
[
    {
        "city":"Glendale",
        "name":"John Yates",
        "job":"Accountant",
        "email":"[email protected]",
        "salary":"9003114.81075051"
    },
    {
        "city":"Sacramento",
        "name":"Camila Wallace",
        "job":"Budget Analyst",
        "email":"[email protected]",
        "salary":"1287392.46738351"
    }
])";

int main() {

    using json = nlohmann::json;

    json ja = json::parse(a);
    json jb = json::parse(b);

    std::copy(jb.begin(), jb.end(), std::back_inserter(ja));

    std::cout << std::setw(5) << ja;
}

The result is what you'd expect: a single JSON array containing all four objects.

That said, given that the files are apparently fairly large, if you're really absolutely certain they contain exactly what's expected (a single JSON array in each, which should just be concatenated together) you may want to just copy the first file, delete the last character (the closing bracket), then copy the second starting from the second character (to skip its opening bracket). This is undoubtedly going to be a lot faster, and it's pretty easy to do with minimal memory usage as well.