How To Append Json With No Duplicating (including Casperjs Code)?

I'm using CasperJS to parse inner texts of webpages and save to json file. Here is my code and I'll show you result(problem that I have) below! var words = []; var casper = require

Solution 1:

Updating an object in file

JSON is defined in such a way that you can't append an object to an existing object and expect to get valid JSON out of it. You can however

1. read the previous serialized JSON string,
2. parse it into an object,
3. append your new values to the array,
4. serialize the object again and
5. completely overwrite the existing file with it.

For example like this:

var previousDataString = fs.read('myresults.json');
var previousData = JSON.parse(previousDataString);
previousData["my_initial_words"] = previousData["my_initial_words"].concat(createFinal(words));
var newData = JSON.stringify(previousData, null, '\t')
fs.write('myresults.json', newData, 'w');

Writing chunks to file

If you still want to write your data file as separate chunks of JSON, then you can do this:

// Combine all items into a single stringvar newItemsString = createFinal(words).reduce(function(combinedString, currentItem){
    return combinedString + JSON.stringify(currentItem) + "\n";
}, "")
// append new items to previous items
fs.write('myresults.json', newItemsString, 'a');

Each item (word's object) is written on exactly one line. When you read the file in some other process then you can use functions such as readLine() to read exactly one item at a time.

Fixing premature exiting

You also have to keep in mind how you're exiting CasperJS. If you provide a callback to casper.run(), then you need to explicitly call casper.exit() in order to exit the process. The problem is that you're doing that too early:

this.echo(JSON.stringify(previousData, null, '\t')).exit();
//                                                 ^^^^^^^ calling exitvar newData = JSON.stringify(previousData, null, '\t'); // not executed
fs.write('myscript.json', newData, 'w');  // not executed

Either you need to put the exit at the end of the callback:

this.echo(JSON.stringify(previousData, null, '\t'));
var newData = JSON.stringify(previousData, null, '\t');
fs.write('myscript.json', newData, 'w');
this.exit();

or don't put your final code into casper.then() instead of casper.run():

casper.then(function() {
    var previousDataString = fs.read('myscript.json');
    var previousData = JSON.parse(previousDataString);
    previousData["my_initial_words"] = previousData["my_initial_words"].concat(createFinal(words));
    this.echo(JSON.stringify(previousData, null, '\t'));
    var newData = JSON.stringify(previousData, null, '\t')
    fs.write('myscript.json', newData, 'w');
});
casper.run();