A crowded news feed can contain the same article several times, several articles about one announcement and successive versions of a single report. Those are three different relationships. Removing everything with a similar title may conceal original reporting, while treating every new URL as an independent story can exaggerate the amount of evidence behind a claim. The useful starting point is identity: what object does each identifier refer to, and what relationship are you trying to preserve? This guide uses fictional feed records to build a practical comparison method. It does not claim that any particular historical publisher followed every convention correctly.
Decide which kind of duplicate matters
Begin with the reader's task. A reading list may need one visible card per article, while an audit needs every retrieved representation. An event timeline may group reports that describe the same announcement without deleting their distinct authorship. A version history needs to connect revisions rather than choose one forever. Calling all these operations deduplication hides decisions that affect what survives. Write down the unit first: source item, retrieved document, report or underlying event. Only then select a key that is appropriate to that unit and describe what happens to the other records.
Imagine two fictional outlets covering one software release. Their headlines mention the same product and version, but one includes an interview and the other tests a downloadable build. They are not duplicate reporting merely because their subject overlaps. Conversely, the same article retrieved through a tracking URL and its clean address may be two observations of one document. A useful system can group both situations without confusing them. It preserves the two outlets as separate reports and the two retrievals as separate observations attached to one article, instead of flattening everything into whichever headline arrived first.
Preserve the source's identifier before inventing one
RSS includes an item guid, and Atom defines an id for an entry or feed. Atom's identifier is intended to remain associated with the same entry through revisions and relocation; it should not be assumed to be a fetchable address. Those format definitions give a reader an identity signal, not a promise that every producer implemented the signal perfectly. Store the original identifier and the source feed alongside it. If your application creates a local record identifier, keep that separate so that later corrections do not erase what the publisher actually supplied.
In a fictional import, item A retains its identifier while its title changes from a planned release to a delayed release. A title-based key could turn the correction into an unrelated story. The persistent source identifier provides a reason to investigate a version relationship. Another producer might accidentally reuse an identifier for unrelated material. In that case the content conflict is evidence of a source-data problem, not a command to overwrite the older record. Good identity handling keeps enough information to recognise both normal revisions and violations of the producer's own apparent convention.
Treat URL normalisation as a declared transformation
A URL can differ because of tracking parameters, a different hostname, a redirected path or a query that genuinely selects another document. Removing every query string is therefore not a neutral cleanup. Some parameters affect attribution only; others identify an article, language or page. Preserve the received address and record the normalised comparison address separately. Explain which transformations your system applies. If you later discover that a parameter mattered, the original observation remains available rather than having been destroyed by an irreversible simplification.
Consider an invented directory where an address includes an item number and an optional campaign tag. Removing the campaign tag may help compare two retrievals, while removing the item number collapses unrelated records. Similarly, a page number can distinguish two historical feed pages even if their templates and headings match. Use site-specific evidence when deciding which components are insignificant. A canonical link can be another useful signal, but it should be recorded as the page's declaration rather than treated as infallible proof. Unexpected cross-domain canonical declarations deserve inspection before they determine which source your application displays.
Compare content without confusing a hash with meaning
A digest can establish that the exact bytes being compared are the same, provided the inputs and algorithm are known. It does not establish that two different byte sequences describe different events, nor that identical text is independently corroborated. Define what you hash: raw response bytes, extracted main text or a chosen set of fields. Navigation, advertisements and formatting can change while an article's substantive text remains stable. A main-text digest can be useful for version comparison, but it is a derived result whose extraction rules belong in the audit record.
In an invented example, two mirrors contain the same press statement with different site headers. Raw-file digests differ, while a carefully extracted statement digest matches. That supports a relationship between the copied text, not the claim that either site independently verified it. A second example changes only a number in a report. A similarity score may remain high even though the change is editorially important. Use hashes and similarity as tools for locating comparisons; inspect the fields that matter to the reader's question before merging, suppressing or labelling the records.
Make grouping reversible and inspect conflicts
Keep a group identifier separate from each member's identity. That allows an interface to show one compact cluster while retaining the individual sources, timestamps and versions underneath. Store the reason for grouping: matching source identifier, confirmed redirect, identical extracted text or manual assessment of a common event. Different reasons carry different implications. A group formed because several reports concern one announcement should not be labelled identical copies unless that narrower claim has actually been established. Reversible grouping is especially useful when a later correction changes how the earlier documents should be understood.
A fictional cluster might initially contain three near-identical headlines. Inspection reveals one original article, one syndicated copy and one independent follow-up. The reading interface can display those relationships explicitly instead of choosing a single winner. If a source reuses an identifier or changes its URL structure, retain the conflict and adjust the mapping with an explanation. Avoid overwriting a record simply because the latest import appears more complete. Earlier representations can explain why an old summary differed, and preserving that context is more valuable than making every row appear artificially consistent.
Count the relationship your headline actually describes
A dashboard might report retrievals, distinct source items, article versions, publisher domains or event clusters. Each is a legitimate count when labelled correctly, but they are not interchangeable. Ten retrieved copies do not mean ten independent reports, and ten independent reports do not necessarily describe ten different events. Present the chosen unit and avoid precision that the grouping process cannot support. If some matches remain uncertain, show the uncertainty or give a range rather than silently converting an approximate similarity decision into a definitive historical statistic.
The restored catalogue on this site uses capture-specific identifiers because a saved version and an original address are different objects. Its counts describe the stated catalogue scope, not an independent-witness total or a complete history of crypto reporting. The same approach works in a small personal feed: retain source identifiers, preserve received addresses, separate versions and make grouping reversible. The goal is not to remove the largest possible number of cards. It is to reduce repetition while keeping the reporting, chronology and attribution that help a reader understand why apparently similar items may still matter.
Sources & further reading
Primary documentation informs the definitions. The worked scenarios and explanations are newly written; hypothetical examples are not historical events.