Discover the most effective method for identifying duplicate items within an array

Question

Discover the most effective method for identifying duplicate items within an array

I'm currently working with angular4 and facing a challenge of displaying a list containing only unique values.

Whenever I access an API, it returns an array from which I have to filter out repeated data. The API will be accessed periodically, and the list should be updated only if new data is available.

response= [{"id":"0DRDCH03DR51GGJGJNP80F7XZ8","value":"36af1784bec4_566601260"},{"id":"0DRDCFYGM2CAHAXYK96BPT9RHV","value":"36af1784bec4_566601140"},...]

listData = [];
for(let data of response) {
    let tempValue = {id: '', time: ''};
    let value = data.value.split('_')
    if (value.length==2) {
        if(value[value.length-1].length==2) {
            tempValue.id = value[0];
            tempValue.time = value[1];
        }
        let isPresent = false;
        if(this.listData.length>0){
            for(let value of this.listData){
                if(value.time===tempValue.time){
                    isPresent = true;
                }
            }
        }
        if(!isPresent) {
            this.listData.push(tempValue);
        }
    }

    }

The resulting list:

listData = [{id:'36af1784bec4', time: '566601140'},...]

The above code provides me with a listData array containing unique values. Despite attempting methods like array.filter and new Set, I couldn't achieve the desired outcome.

If you have any suggestions on how to improve the efficiency of this process, please share them with me.

javascript arrays angular typescript unique

Answer 1

Answer №1

If you're looking to eliminate duplicates, check out this code snippet.

var obj = {};

for ( var i=0, len=response.length; i < len; i++ )
    obj[response[i]['id']] = response[i]; // consider id as unique identifier

response = new Array();
for ( var key in obj )
    response.push(obj[key]);

console.log(response); // list of non-duplicate data

Edit:

To perform a thorough check on a specific property for duplicates, try this approach.

function removeDuplicates(myArr, prop) {
    return myArr.filter((obj, pos, arr) => {
        return arr.map(mapObj => mapObj[prop]).indexOf(obj[prop]) === pos;
    });
}


console.log(removeDuplicates(response,['id','value'])); // specify either 'id' or 'value' for the second parameter.

Answer 2

If you're looking to eliminate duplicates, check out this code snippet.

var obj = {};

for ( var i=0, len=response.length; i < len; i++ )
    obj[response[i]['id']] = response[i]; // consider id as unique identifier

response = new Array();
for ( var key in obj )
    response.push(obj[key]);

console.log(response); // list of non-duplicate data

Edit:

To perform a thorough check on a specific property for duplicates, try this approach.

function removeDuplicates(myArr, prop) {
    return myArr.filter((obj, pos, arr) => {
        return arr.map(mapObj => mapObj[prop]).indexOf(obj[prop]) === pos;
    });
}


console.log(removeDuplicates(response,['id','value'])); // specify either 'id' or 'value' for the second parameter.

Answer 3

Answer №2

My solution seems promising

for(let item of response) {
    if(!this.listData.find((ldata) => item.value.substring(item.value.lastIndexOf('_')+1) === ldata.time)) {
        let tempVal = {id: '', time: ''};
        let val = item.value.split('_')
        if (val.length==2) {
            if(val[val.length-1].length==2) {
                tempVal.id = val[0];
                tempVal.time = val[1];
            }
            this.listData.push(tempVal);
        }
    }
}

Answer 4

My solution seems promising

for(let item of response) {
    if(!this.listData.find((ldata) => item.value.substring(item.value.lastIndexOf('_')+1) === ldata.time)) {
        let tempVal = {id: '', time: ''};
        let val = item.value.split('_')
        if (val.length==2) {
            if(val[val.length-1].length==2) {
                tempVal.id = val[0];
                tempVal.time = val[1];
            }
            this.listData.push(tempVal);
        }
    }
}

Answer 5

Answer №3

To achieve an optimal solution (loop only once i.e. O(n)), use object property to check if the property has already been added. This way, you can avoid looping to find out if the property has already been added.

Here is a sample solution:

let response = [{
    "id": "0DRDCH03DR51GGJGJNP80F7XZ8",
    "value": "36af1784bec4_566601260"
  }, {
    "id": "0DRDCH03DR51GGJGJNP80F7XZ8",
    "value": "36af1784bec4_566601260"
  }, {
    "id": "0DRDCFYGM2CAHAXYK96BPT9RHV",
    "value": "36af1784bec4_566601140"
  }],
  listData = [],
  tempObj = {};
for (let data of response) {
    let value = data.value.split('_');
  if (!tempObj.hasOwnProperty(value[1])) {
    tempObj[data.value[1]] = "";
    listData.push({
      id: value[0],
      time: value[1]
    });
  }
}

console.log(listData);

Update: As per feedback, now checking only for duplicate time values

Answer 6

To achieve an optimal solution (loop only once i.e. O(n)), use object property to check if the property has already been added. This way, you can avoid looping to find out if the property has already been added.

Here is a sample solution:

let response = [{
    "id": "0DRDCH03DR51GGJGJNP80F7XZ8",
    "value": "36af1784bec4_566601260"
  }, {
    "id": "0DRDCH03DR51GGJGJNP80F7XZ8",
    "value": "36af1784bec4_566601260"
  }, {
    "id": "0DRDCFYGM2CAHAXYK96BPT9RHV",
    "value": "36af1784bec4_566601140"
  }],
  listData = [],
  tempObj = {};
for (let data of response) {
    let value = data.value.split('_');
  if (!tempObj.hasOwnProperty(value[1])) {
    tempObj[data.value[1]] = "";
    listData.push({
      id: value[0],
      time: value[1]
    });
  }
}

console.log(listData);

Update: As per feedback, now checking only for duplicate time values

Answer 7

Answer №4

You can achieve the same result by converting objects to strings using JSON.stringify as shown below :

let response = [{
  "id": "0DRDCH03DR51GGJGJNP80F7XZ8",
  "value": "36af1784bec4_566601260"
}, {
  "id": "0DRDCFYGM2CAHAXYK96BPT9RHV",
  "value": "36af1784bec4_566601140"
}, {
  "id": "0DRDCH03DR51GGJGJNP80F7XZ8",
  "value": "36af1784bec4_566601260"
}];

let o = response.reduce((acc, cv) => {
  if (!acc[JSON.stringify(cv)]) {
    acc[JSON.stringify(cv)] = true; //something non-falsy
  }
  return acc;
}, {});

let res = Object.keys(o).map(x => JSON.parse(x));
console.log(res);

Edit As user3297291 mentioned

It is important to note that JSON.stringify({ a: 1, b: 2}) !== JSON.stringify({ b: 2, a: 1 })

Hence, the above approach may not work in such cases.

A more efficient solution would be

let response = [{
  "id": "0DRDCH03DR51GGJGJNP80F7XZ8",
  "value": "36af1784bec4_566601260"
}, {
  "id": "0DRDCFYGM2CAHAXYK96BPT9RHV",
  "value": "36af1784bec4_566601140"
}, {
  "id": "0DRDCH03DR51GGJGJNP80F7XZ8",
  "value": "36af1784bec4_566601260"
}];

let o = response.reduce((acc, cv) => {
  if (!acc[cv.value]) {
    acc[cv.value] = true; //something non-falsy
  }
  return acc;
}, {});

let res = Object.keys(o).map(x => {
  let t = x.split('_');
  return {
    id: t[0],
    time: t[1]
  };
});
console.log(res);

Answer 8

You can achieve the same result by converting objects to strings using JSON.stringify as shown below :

let response = [{
  "id": "0DRDCH03DR51GGJGJNP80F7XZ8",
  "value": "36af1784bec4_566601260"
}, {
  "id": "0DRDCFYGM2CAHAXYK96BPT9RHV",
  "value": "36af1784bec4_566601140"
}, {
  "id": "0DRDCH03DR51GGJGJNP80F7XZ8",
  "value": "36af1784bec4_566601260"
}];

let o = response.reduce((acc, cv) => {
  if (!acc[JSON.stringify(cv)]) {
    acc[JSON.stringify(cv)] = true; //something non-falsy
  }
  return acc;
}, {});

let res = Object.keys(o).map(x => JSON.parse(x));
console.log(res);

Edit As user3297291 mentioned

It is important to note that JSON.stringify({ a: 1, b: 2}) !== JSON.stringify({ b: 2, a: 1 })

Hence, the above approach may not work in such cases.

A more efficient solution would be

let response = [{
  "id": "0DRDCH03DR51GGJGJNP80F7XZ8",
  "value": "36af1784bec4_566601260"
}, {
  "id": "0DRDCFYGM2CAHAXYK96BPT9RHV",
  "value": "36af1784bec4_566601140"
}, {
  "id": "0DRDCH03DR51GGJGJNP80F7XZ8",
  "value": "36af1784bec4_566601260"
}];

let o = response.reduce((acc, cv) => {
  if (!acc[cv.value]) {
    acc[cv.value] = true; //something non-falsy
  }
  return acc;
}, {});

let res = Object.keys(o).map(x => {
  let t = x.split('_');
  return {
    id: t[0],
    time: t[1]
  };
});
console.log(res);

Answer 9

Answer №5

To achieve this task, you can utilize a map function to gather the ids and then apply a filter function to eliminate duplicates:

const response = [{"id":"0DRDCH03DR51GGJGJNP80F7XZ8","value":"36af1784bec4_566601260"},{"id":"0DRDCFYGM2CAHAXYK96BPT9RHV","value":"36af1784bec4_566601140"}];

const ids = response.map(item => item.value.split('_')[0]).filter((item, index, array) => index === array.indexOf(item));

console.log(ids);

Alternatively, you can also use a reduce function:

const response= [{"id":"0DRDCH03DR51GGJGJNP80F7XZ8","value":"36af1784bec4_566601260"},{"id":"0DRDCFYGM2CAHAXYK96BPT9RHV","value":"36af1784bec4_566601140"}];

const ids = response.reduce((accumulator, currentItem) => {
  const id = currentItem.value.split('_')[0];
  return accumulator.includes(id) ? accumulator : [...accumulator, id];
}, []);

console.log(ids);

Answer 10

To achieve this task, you can utilize a map function to gather the ids and then apply a filter function to eliminate duplicates:

const response = [{"id":"0DRDCH03DR51GGJGJNP80F7XZ8","value":"36af1784bec4_566601260"},{"id":"0DRDCFYGM2CAHAXYK96BPT9RHV","value":"36af1784bec4_566601140"}];

const ids = response.map(item => item.value.split('_')[0]).filter((item, index, array) => index === array.indexOf(item));

console.log(ids);

Alternatively, you can also use a reduce function:

const response= [{"id":"0DRDCH03DR51GGJGJNP80F7XZ8","value":"36af1784bec4_566601260"},{"id":"0DRDCFYGM2CAHAXYK96BPT9RHV","value":"36af1784bec4_566601140"}];

const ids = response.reduce((accumulator, currentItem) => {
  const id = currentItem.value.split('_')[0];
  return accumulator.includes(id) ? accumulator : [...accumulator, id];
}, []);

console.log(ids);

Answer 11

Answer №6

Recent versions of JavaScript now support relying on the consistency of object structure. This means we can use stringification and comparison of two strings to produce accurate results:

JSON.stringify({id:0, name:foo}) === JSON.stringify({id:0, name:foo}) // true

When dealing with an array of objects:

Stringifying an object for testing
Iterating through a collection of objects and stringifying each one
Comparing the two strings created

This approach is beneficial as it eliminates the need to know specific key names, working effectively with any type of object...

arr = [{object}, {object2}, {etc...}]
test = JSON.stringify(obj);

for ( let i = 0, j = arr.length; i < j; i++ ) {
  let testAgainst = JSON.stringify(arr[i]);
  if (test === testAgainst) { 
    console.log('matches');
  } else {
    console.log('no match');
  }
}

By comparing two strings, you can easily identify matches. Feel free to leave a comment if there are any additional aspects to consider.

Answer 12

Recent versions of JavaScript now support relying on the consistency of object structure. This means we can use stringification and comparison of two strings to produce accurate results:

JSON.stringify({id:0, name:foo}) === JSON.stringify({id:0, name:foo}) // true

When dealing with an array of objects:

Stringifying an object for testing
Iterating through a collection of objects and stringifying each one
Comparing the two strings created

This approach is beneficial as it eliminates the need to know specific key names, working effectively with any type of object...

arr = [{object}, {object2}, {etc...}]
test = JSON.stringify(obj);

for ( let i = 0, j = arr.length; i < j; i++ ) {
  let testAgainst = JSON.stringify(arr[i]);
  if (test === testAgainst) { 
    console.log('matches');
  } else {
    console.log('no match');
  }
}

By comparing two strings, you can easily identify matches. Feel free to leave a comment if there are any additional aspects to consider.

Discover the most effective method for identifying duplicate items within an array

Answer №1

Answer №2

Answer №3

Answer №4

Answer №5

Answer №6

Similar questions

Converting a dynamic JSON object into a generic type in TypeScript

Creating a List with Sublists that are displayed when hovering over the parent List is a key element of effective design

I am trying to retrieve the class name of each iframe from within the iframe itself, as each iframe has a unique class name

Tips for validating a string in a URL with Selenium IDE

Setting up GameClosure on a Windows machine

In Vue, while templates cannot be assigned keys and cannot be replaced with a div element, it is important to utilize the v-for directive without a wrapper div element, especially when dealing with nested v-for loops

Having trouble retrieving POST data with the Webextensions API

Toggle visibility of div based on current business hours, incorporating UTC time

Strategies for Implementing Pagination in an Angular 2 HTML Table Without the Use of Third-Party Tools

How to incorporate sound files in JavaScript

Error message: "The property or method you are trying to access is not defined

Deselect the checkbox if you are checking a different checkbox

StealJS Module Repathing Techniques

The typings for object properties in Typescript

Is there a way for me to determine if there are any related projected materials available?

Find the total of values in an array that may be null or undefined

"Automate the process of manual content duplication with JavaScript's for each replacement

Expanding Arrays in TypeScript for a particular type

A step-by-step guide on incorporating box-shadow for the jackColor in the Switchery Plugin

Accessing external data in Angular outside of a subscription method for an observable