Mitarbeit willkommen! Bitte schau unter Hilfe:Benutzerkonto oder informiere Dich über Populus.Wiki.

Populus:DezInV/Notes: Unterschied zwischen den Versionen

Aus Populus DE
Zur Navigation springenZur Suche springen
Zeile 1: Zeile 1:
  +
== architecture ==
  +
  +
=== split between crawler/warc writer and processor ===
  +
  +
Having two separate processes for crawling and processing has advantages:
  +
  +
* The crawler is simple: Work a queue of URLs and write a warc file
  +
* Processing can be redone as often as needed without network access
  +
  +
There should not be much overhead:
  +
  +
* The (compressed) data should still be in the Kernels page cache if processing starts immediately after the warc file has been written.
  +
* Decompression is fast and the decompressed data can be piped directly into a HTML parser to minimize memory consumption.
  +
 
== archive ==
  
== archive ==
   
  +
* [https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1-annotated/ The WARC format]
 
* [https://encode.su/threads/3660-Best-compressors-for-huge-JSON-and-WARC-(web-archive)-files Discussion about compression] with somebody working on Russian web archive
  
* [https://encode.su/threads/3660-Best-compressors-for-huge-JSON-and-WARC-(web-archive)-files Discussion about compression] with somebody working on Russian web archive
  +
  +
=== compression ===
  +
  +
* Book: 1999, Managing Gigabytes - Compressing and Indexing Documents and Images
  +
* brotli
  +
** https://cran.r-project.org/web/packages/brotli/vignettes/brotli-2015-09-22.pdf
  +
  +
==== seekable (random access) compression ====
  +
  +
Spinning hard drives have sequential read speeds above 100 MB/s. Thus a record in e.g. a 64MB compressed warc file can be retrieved on average in less than 0.3s, even when the archive needs to be seeked from the start.
  +
  +
(64 MB once was the recommended size for Hadoop files.)
  +
  +
Still, these were the links I found:
  +
  +
* https://github.com/facebook/zstd/blob/dev/contrib/seekable_format/zstd_seekable_compression_format.md
  +
* https://encode.su/threads/1766-Seekable-compression
  +
* https://stackoverflow.com/questions/2046559/any-seekable-compression-library
  +
* https://superuser.com/questions/1235351/what-makes-a-tar-archive-seekable
  +
* https://serverfault.com/questions/59795/is-there-a-smarter-tar-or-cpio-out-there-for-efficiently-retrieving-a-file-store/546691#546691
  +
* https://innovation.ebayinc.com/tech/engineering/gzinga-seekable-and-splittable-gzip/
  +
* https://stackoverflow.com/questions/14225751/random-access-to-gzipped-files
  +
* https://stackoverflow.com/questions/429987/compression-formats-with-good-support-for-random-access-within-archives
  +
* https://stackoverflow.com/questions/236414/what-is-the-best-compression-algorithm-that-allows-random-reads-writes-in-a-file
  +
  +
   
 
== crawling ==
  
== crawling ==
Zeile 12: Zeile 53:
   
 
== search ==
  
== search ==
  +
  +
=== Search Engines ===
  +
  +
==== Terrier ====
  +
  +
* [http://terrier.org Terrier.org] (Java) by the Information Retrieval Group within the School of Computing Science at the University of Glasgow
   
 
= from org-mode/thk =
  
= from org-mode/thk =

Version vom 9. Dezember 2023, 18:10 Uhr

architecture

split between crawler/warc writer and processor

Having two separate processes for crawling and processing has advantages:

  • The crawler is simple: Work a queue of URLs and write a warc file
  • Processing can be redone as often as needed without network access

There should not be much overhead:

  • The (compressed) data should still be in the Kernels page cache if processing starts immediately after the warc file has been written.
  • Decompression is fast and the decompressed data can be piped directly into a HTML parser to minimize memory consumption.

archive

compression

seekable (random access) compression

Spinning hard drives have sequential read speeds above 100 MB/s. Thus a record in e.g. a 64MB compressed warc file can be retrieved on average in less than 0.3s, even when the archive needs to be seeked from the start.

(64 MB once was the recommended size for Hadoop files.)

Still, these were the links I found:


crawling

index

scraping

search

Search Engines

Terrier

  • Terrier.org (Java) by the Information Retrieval Group within the School of Computing Science at the University of Glasgow

from org-mode/thk

dezentrale Suchmaschine

Internet Archive

Crawling, Crawler

  1. ArchiveBox

    Python, aktiv

    Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more…

  2. grab-site

    Python, aktiv https://github.com/ArchiveTeam/grab-site

  3. WASP

  4. StormCrawler

  5. HTTrack

  6. Grub.org

    dezentraler Crawler des Wikia Projektes In C#, Pyton

  7. HCE – Hierarchical Cluster Engine

  8. Heritrix

  9. Haskell

Search

  1. openwebsearch.eu

  2. lemurproject.org

    1. lucindri

      https://lemurproject.org/lucindri.php

      Lucindri is an open-source implementation of Indri search logic and structured query language using the Lucene Search Engine. Lucindri consists of two components: the indexer and the searcher.

    2. Galago

      toolkit for experimenting with text search

Wikia

Recherche 2023-11-26

Scraping

Abgerufen von „https://de.populus.wiki/w/index.php?title=Populus:DezInV/Notes&oldid=1591“