Discover the most effective method for identifying duplicate items within an array

Question

Discover the most effective method for identifying duplicate items within an array

I'm currently working with angular4 and facing a challenge of displaying a list containing only unique values.

Whenever I access an API, it returns an array from which I have to filter out repeated data. The API will be accessed periodically, and the list should be updated only if new data is available.

response= [{"id":"0DRDCH03DR51GGJGJNP80F7XZ8","value":"36af1784bec4_566601260"},{"id":"0DRDCFYGM2CAHAXYK96BPT9RHV","value":"36af1784bec4_566601140"},...]

listData = [];
for(let data of response) {
    let tempValue = {id: '', time: ''};
    let value = data.value.split('_')
    if (value.length==2) {
        if(value[value.length-1].length==2) {
            tempValue.id = value[0];
            tempValue.time = value[1];
        }
        let isPresent = false;
        if(this.listData.length>0){
            for(let value of this.listData){
                if(value.time===tempValue.time){
                    isPresent = true;
                }
            }
        }
        if(!isPresent) {
            this.listData.push(tempValue);
        }
    }

    }

The resulting list:

listData = [{id:'36af1784bec4', time: '566601140'},...]

The above code provides me with a listData array containing unique values. Despite attempting methods like array.filter and new Set, I couldn't achieve the desired outcome.

If you have any suggestions on how to improve the efficiency of this process, please share them with me.

javascript arrays angular typescript unique

Answer 1

Answer №1

If you're looking to eliminate duplicates, check out this code snippet.

var obj = {};

for ( var i=0, len=response.length; i < len; i++ )
    obj[response[i]['id']] = response[i]; // consider id as unique identifier

response = new Array();
for ( var key in obj )
    response.push(obj[key]);

console.log(response); // list of non-duplicate data

Edit:

To perform a thorough check on a specific property for duplicates, try this approach.

function removeDuplicates(myArr, prop) {
    return myArr.filter((obj, pos, arr) => {
        return arr.map(mapObj => mapObj[prop]).indexOf(obj[prop]) === pos;
    });
}


console.log(removeDuplicates(response,['id','value'])); // specify either 'id' or 'value' for the second parameter.

Answer 2

If you're looking to eliminate duplicates, check out this code snippet.

var obj = {};

for ( var i=0, len=response.length; i < len; i++ )
    obj[response[i]['id']] = response[i]; // consider id as unique identifier

response = new Array();
for ( var key in obj )
    response.push(obj[key]);

console.log(response); // list of non-duplicate data

Edit:

To perform a thorough check on a specific property for duplicates, try this approach.

function removeDuplicates(myArr, prop) {
    return myArr.filter((obj, pos, arr) => {
        return arr.map(mapObj => mapObj[prop]).indexOf(obj[prop]) === pos;
    });
}


console.log(removeDuplicates(response,['id','value'])); // specify either 'id' or 'value' for the second parameter.

Answer 3

Answer №2

My solution seems promising

for(let item of response) {
    if(!this.listData.find((ldata) => item.value.substring(item.value.lastIndexOf('_')+1) === ldata.time)) {
        let tempVal = {id: '', time: ''};
        let val = item.value.split('_')
        if (val.length==2) {
            if(val[val.length-1].length==2) {
                tempVal.id = val[0];
                tempVal.time = val[1];
            }
            this.listData.push(tempVal);
        }
    }
}

Answer 4

My solution seems promising

for(let item of response) {
    if(!this.listData.find((ldata) => item.value.substring(item.value.lastIndexOf('_')+1) === ldata.time)) {
        let tempVal = {id: '', time: ''};
        let val = item.value.split('_')
        if (val.length==2) {
            if(val[val.length-1].length==2) {
                tempVal.id = val[0];
                tempVal.time = val[1];
            }
            this.listData.push(tempVal);
        }
    }
}

Answer 5

Answer №3

To achieve an optimal solution (loop only once i.e. O(n)), use object property to check if the property has already been added. This way, you can avoid looping to find out if the property has already been added.

Here is a sample solution:

let response = [{
    "id": "0DRDCH03DR51GGJGJNP80F7XZ8",
    "value": "36af1784bec4_566601260"
  }, {
    "id": "0DRDCH03DR51GGJGJNP80F7XZ8",
    "value": "36af1784bec4_566601260"
  }, {
    "id": "0DRDCFYGM2CAHAXYK96BPT9RHV",
    "value": "36af1784bec4_566601140"
  }],
  listData = [],
  tempObj = {};
for (let data of response) {
    let value = data.value.split('_');
  if (!tempObj.hasOwnProperty(value[1])) {
    tempObj[data.value[1]] = "";
    listData.push({
      id: value[0],
      time: value[1]
    });
  }
}

console.log(listData);

Update: As per feedback, now checking only for duplicate time values

Answer 6

To achieve an optimal solution (loop only once i.e. O(n)), use object property to check if the property has already been added. This way, you can avoid looping to find out if the property has already been added.

Here is a sample solution:

let response = [{
    "id": "0DRDCH03DR51GGJGJNP80F7XZ8",
    "value": "36af1784bec4_566601260"
  }, {
    "id": "0DRDCH03DR51GGJGJNP80F7XZ8",
    "value": "36af1784bec4_566601260"
  }, {
    "id": "0DRDCFYGM2CAHAXYK96BPT9RHV",
    "value": "36af1784bec4_566601140"
  }],
  listData = [],
  tempObj = {};
for (let data of response) {
    let value = data.value.split('_');
  if (!tempObj.hasOwnProperty(value[1])) {
    tempObj[data.value[1]] = "";
    listData.push({
      id: value[0],
      time: value[1]
    });
  }
}

console.log(listData);

Update: As per feedback, now checking only for duplicate time values

Answer 7

Answer №4

You can achieve the same result by converting objects to strings using JSON.stringify as shown below :

let response = [{
  "id": "0DRDCH03DR51GGJGJNP80F7XZ8",
  "value": "36af1784bec4_566601260"
}, {
  "id": "0DRDCFYGM2CAHAXYK96BPT9RHV",
  "value": "36af1784bec4_566601140"
}, {
  "id": "0DRDCH03DR51GGJGJNP80F7XZ8",
  "value": "36af1784bec4_566601260"
}];

let o = response.reduce((acc, cv) => {
  if (!acc[JSON.stringify(cv)]) {
    acc[JSON.stringify(cv)] = true; //something non-falsy
  }
  return acc;
}, {});

let res = Object.keys(o).map(x => JSON.parse(x));
console.log(res);

Edit As user3297291 mentioned

It is important to note that JSON.stringify({ a: 1, b: 2}) !== JSON.stringify({ b: 2, a: 1 })

Hence, the above approach may not work in such cases.

A more efficient solution would be

let response = [{
  "id": "0DRDCH03DR51GGJGJNP80F7XZ8",
  "value": "36af1784bec4_566601260"
}, {
  "id": "0DRDCFYGM2CAHAXYK96BPT9RHV",
  "value": "36af1784bec4_566601140"
}, {
  "id": "0DRDCH03DR51GGJGJNP80F7XZ8",
  "value": "36af1784bec4_566601260"
}];

let o = response.reduce((acc, cv) => {
  if (!acc[cv.value]) {
    acc[cv.value] = true; //something non-falsy
  }
  return acc;
}, {});

let res = Object.keys(o).map(x => {
  let t = x.split('_');
  return {
    id: t[0],
    time: t[1]
  };
});
console.log(res);

Answer 8

You can achieve the same result by converting objects to strings using JSON.stringify as shown below :

let response = [{
  "id": "0DRDCH03DR51GGJGJNP80F7XZ8",
  "value": "36af1784bec4_566601260"
}, {
  "id": "0DRDCFYGM2CAHAXYK96BPT9RHV",
  "value": "36af1784bec4_566601140"
}, {
  "id": "0DRDCH03DR51GGJGJNP80F7XZ8",
  "value": "36af1784bec4_566601260"
}];

let o = response.reduce((acc, cv) => {
  if (!acc[JSON.stringify(cv)]) {
    acc[JSON.stringify(cv)] = true; //something non-falsy
  }
  return acc;
}, {});

let res = Object.keys(o).map(x => JSON.parse(x));
console.log(res);

Edit As user3297291 mentioned

It is important to note that JSON.stringify({ a: 1, b: 2}) !== JSON.stringify({ b: 2, a: 1 })

Hence, the above approach may not work in such cases.

A more efficient solution would be

let response = [{
  "id": "0DRDCH03DR51GGJGJNP80F7XZ8",
  "value": "36af1784bec4_566601260"
}, {
  "id": "0DRDCFYGM2CAHAXYK96BPT9RHV",
  "value": "36af1784bec4_566601140"
}, {
  "id": "0DRDCH03DR51GGJGJNP80F7XZ8",
  "value": "36af1784bec4_566601260"
}];

let o = response.reduce((acc, cv) => {
  if (!acc[cv.value]) {
    acc[cv.value] = true; //something non-falsy
  }
  return acc;
}, {});

let res = Object.keys(o).map(x => {
  let t = x.split('_');
  return {
    id: t[0],
    time: t[1]
  };
});
console.log(res);

Answer 9

Answer №5

To achieve this task, you can utilize a map function to gather the ids and then apply a filter function to eliminate duplicates:

const response = [{"id":"0DRDCH03DR51GGJGJNP80F7XZ8","value":"36af1784bec4_566601260"},{"id":"0DRDCFYGM2CAHAXYK96BPT9RHV","value":"36af1784bec4_566601140"}];

const ids = response.map(item => item.value.split('_')[0]).filter((item, index, array) => index === array.indexOf(item));

console.log(ids);

Alternatively, you can also use a reduce function:

const response= [{"id":"0DRDCH03DR51GGJGJNP80F7XZ8","value":"36af1784bec4_566601260"},{"id":"0DRDCFYGM2CAHAXYK96BPT9RHV","value":"36af1784bec4_566601140"}];

const ids = response.reduce((accumulator, currentItem) => {
  const id = currentItem.value.split('_')[0];
  return accumulator.includes(id) ? accumulator : [...accumulator, id];
}, []);

console.log(ids);

Answer 10

To achieve this task, you can utilize a map function to gather the ids and then apply a filter function to eliminate duplicates:

const response = [{"id":"0DRDCH03DR51GGJGJNP80F7XZ8","value":"36af1784bec4_566601260"},{"id":"0DRDCFYGM2CAHAXYK96BPT9RHV","value":"36af1784bec4_566601140"}];

const ids = response.map(item => item.value.split('_')[0]).filter((item, index, array) => index === array.indexOf(item));

console.log(ids);

Alternatively, you can also use a reduce function:

const response= [{"id":"0DRDCH03DR51GGJGJNP80F7XZ8","value":"36af1784bec4_566601260"},{"id":"0DRDCFYGM2CAHAXYK96BPT9RHV","value":"36af1784bec4_566601140"}];

const ids = response.reduce((accumulator, currentItem) => {
  const id = currentItem.value.split('_')[0];
  return accumulator.includes(id) ? accumulator : [...accumulator, id];
}, []);

console.log(ids);

Answer 11

Answer №6

Recent versions of JavaScript now support relying on the consistency of object structure. This means we can use stringification and comparison of two strings to produce accurate results:

JSON.stringify({id:0, name:foo}) === JSON.stringify({id:0, name:foo}) // true

When dealing with an array of objects:

Stringifying an object for testing
Iterating through a collection of objects and stringifying each one
Comparing the two strings created

This approach is beneficial as it eliminates the need to know specific key names, working effectively with any type of object...

arr = [{object}, {object2}, {etc...}]
test = JSON.stringify(obj);

for ( let i = 0, j = arr.length; i < j; i++ ) {
  let testAgainst = JSON.stringify(arr[i]);
  if (test === testAgainst) { 
    console.log('matches');
  } else {
    console.log('no match');
  }
}

By comparing two strings, you can easily identify matches. Feel free to leave a comment if there are any additional aspects to consider.

Answer 12

Recent versions of JavaScript now support relying on the consistency of object structure. This means we can use stringification and comparison of two strings to produce accurate results:

JSON.stringify({id:0, name:foo}) === JSON.stringify({id:0, name:foo}) // true

When dealing with an array of objects:

Stringifying an object for testing
Iterating through a collection of objects and stringifying each one
Comparing the two strings created

This approach is beneficial as it eliminates the need to know specific key names, working effectively with any type of object...

arr = [{object}, {object2}, {etc...}]
test = JSON.stringify(obj);

for ( let i = 0, j = arr.length; i < j; i++ ) {
  let testAgainst = JSON.stringify(arr[i]);
  if (test === testAgainst) { 
    console.log('matches');
  } else {
    console.log('no match');
  }
}

By comparing two strings, you can easily identify matches. Feel free to leave a comment if there are any additional aspects to consider.

Discover the most effective method for identifying duplicate items within an array

Answer №1

Answer №2

Answer №3

Answer №4

Answer №5

Answer №6

Similar questions

An error occurred while trying to serialize the `.res` response received from the `getServerSideProps` function in

Integrating a title field into every p-column with ng-template

Special effects for the images动画效果。

When the previous textbox is filled, the cursor will automatically move to the button

How do AppComponent and @Component relate to each other in AngularJs 2?

Adjusted position of the viewport if the DOM element containing the renderer is not located at the top of the display

I have a collection of emails stored as a string that I would like to convert into a json or javascript object and store in a mongodb

Evaluating different attributes within a single entity

Using AngularJS API within a standalone function: Tips and tricks

The logout confirmation message functionality in Laravel 8 is malfunctioning

Combining JSON and JavaScript for embedding a SPARQL query into an HTML document

Tips for populating a DOJO Select using JSON data that includes various parameters instead of just label and value

Exploring JSON information containing colons in the labels

jQuery Autocomplete - struggling to pinpoint the exact location where the width of the suggestions div is being defined

What is the best method for implementing pagination in Larvael with React using Inertia?

Can you explain the meaning of the code snippet provided?

The responsive menu refuses to appear

Python: How to determine if a specific part of an element is present in an array?

What is the best way to implement a required input field in Vue.js?

Problem with Onsen UI navigation: It is not possible to provide a "ons-page" element to "ons-navigator" when attempting to navigate back to the initial page