Mitarbeit willkommen! Bitte schau unter Hilfe:Benutzerkonto oder informiere Dich über Populus.Wiki.

Populus:DezInV/Notes: Unterschied zwischen den Versionen

Aus Populus DE
Zur Navigation springenZur Suche springen
K (Thk verschob die Seite Populus:DezInV/Notizen nach Populus:DezInV/Notes, ohne dabei eine Weiterleitung anzulegen)
Keine Bearbeitungszusammenfassung
Zeile 12: Zeile 12:


== search ==
== search ==

= from org-mode/thk =


== dezentrale Suchmaschine ==

* https://yacy.net
** [https://bugs.debian.org/768171 RFP]
** https://www.youtube.com/c/YaCyTutorials/videos
* https://github.com/nvasilakis/yippee
** Java
** last commit 2012
* https://www.techdirt.com/articles/20140701/03143327738/distributed-search-engines-why-we-need-them-post-snowden-world.shtml
* https://metaphacts.com/diesel
** non-free enterprise search engine
* https://fourweekmba.com/distributed-search-engines-vs-google/
* presearch.{io|org}
** non-free, blockchain,bla bla
* https://github.com/kearch/kearch
** letzter commit 2019
** python
* https://hackernoon.com/is-the-concept-of-a-distributed-search-engine-potent-enough-to-challenge-googles-dominance-l1s44t2
* https://blog.florence.chat/a-distributed-search-engine-for-the-distributed-web-39c377dc700e
* https://wiki.p2pfoundation.net/Distributed_Search_Engines
* https://en.wikipedia.org/wiki/Distributed_search_engine
* https://searx.github.io/searx/
* https://www.astridmager.net Wissenschaftlerin, alternative, ethic search
* Warum man auch ein lokales Archiv haben sollte: https://www.medialens.org/2023/is-that-orwellian-or-kafkaesque-enough-for-you-the-guardian-removes-bin-ladens-letter-to-america

=== Internet Archive ===

* https://netpreserve.org/about-us/
* https://github.com/iipc/awesome-web-archiving
* https://github.com/internetarchive
* https://github.com/iipc/openwayback
* http://timetravel.mementoweb.org/
** http://timetravel.mementoweb.org/about/
* https://github.com/webrecorder

=== Crawling, Crawler ===

* https://en.wikipedia.org/wiki/WARC_(file_format)

<ol>
<li><p>ArchiveBox</p>
<ul>
<li>https://github.com/ArchiveBox/ArchiveBox</li></ul>

<p>Python, aktiv</p>
<p>Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more…</p>
<ul>
<li>https://archivebox.io</li></ul>
</li>
<li><p>grab-site</p>
<p>Python, aktiv https://github.com/ArchiveTeam/grab-site</p></li>
<li><p>WASP</p>
<ul>
<li>Paper: [https://ceur-ws.org/Vol-2167/paper6.pdf WASP: Web Archiving and Search Personalized] via https://github.com/webis-de/wasp
<ul>
<li>Java, inaktiv?</li>
<li>related Paper: [https://www.cs.ru.nl/bachelors-theses/2019/Gijs_Hendriksen___4324544___Extending_WASP_providing_context_to_a_personal_web_archive.pdf Extending WASP: providing context to a personal web archive]</li></ul>
</li></ul>
</li>
<li><p>StormCrawler</p>
<ul>
<li>https://en.wikipedia.org/wiki/StormCrawler</li>
<li>Auf Basis von Apache Storm (distributed stream processing)</li></ul>
</li>
<li><p>HTTrack</p>
<ul>
<li>https://www.httrack.com</li>
<li>In C geschrieben, letztes Release 2017, aber aktives Forum und Github</li>
<li>https://github.com/xroche/httrack/tags</li></ul>
</li>
<li><p>Grub.org</p>
<p>dezentraler Crawler des Wikia Projektes In C#, Pyton</p>
<ul>
<li>https://web.archive.org/web/20090207182028/http://grub.org/</li>
<li>https://en.wikipedia.org/wiki/Grub_(search_engine)</li></ul>
</li>
<li><p>HCE – Hierarchical Cluster Engine</p>
<ul>
<li>http://hierarchical-cluster-engine.com</li>
<li>https://en.wikipedia.org/wiki/Hierarchical_Cluster_Engine_Project</li>
<li>Ukraine, aktiv bis ca. 2014?</li></ul>
</li>
<li><p>Heritrix</p>
<ul>
<li>https://en.wikipedia.org/wiki/Heritrix</li>
<li>Crawler von archive.org</li></ul>
</li>
<li><p>Haskell</p>
<ul>
<li>https://shimweasel.com/2017/07/13/a-modest-scraping-proposal
<ul>
<li>https://www.reddit.com/r/haskell/comments/6nazqm/notes_on_design_of_webscrapers_in_haskell/</li></ul>
</li>
<li>https://github.com/jordanspooner/haskell-web-crawler
<ul>
<li>dead since 2017, seems like student assignment</li></ul>
</li>
<li>https://hackage.haskell.org/package/scalpel
<ul>
<li>A high level web scraping library for Haskell.</li></ul>
</li>
<li>https://hackage.haskell.org/package/hScraper
<ul>
<li>eine Version 0.1.0.0 von 2015</li></ul>
</li>
<li>https://hackage.haskell.org/package/hs-scrape
<ul>
<li>eine VErsion von 2014, aber git commit von 2020</li></ul>
</li>
<li>https://hackage.haskell.org/package/http-conduit-downloader
<ul>
<li>HTTP/HTTPS downloader built on top of http-client and used in https://bazqux.com crawler.</li></ul>
</li>
<li>https://github.com/hunt-framework/hunt
<ul>
<li>A flexible, lightweight search platform</li>
<li>Vorläufer https://github.com/fortytools/holumbus/
<ul>
<li>http://holumbus.fh-wedel.de/trac</li></ul>
</li></ul>
</li>
<li>lower level libs
<ul>
<li>https://github.com/haskell/wreq http://www.serpentine.com/wreq</li>
<li>https://github.com/snoyberg/http-client/blob/master/TUTORIAL.md</li></ul>
</li></ul>
</li></ol>

<span id="search"></span>
=== Search ===

<ol>
<li><p>openwebsearch.eu</p>
<ul>
<li>https://openwebsearch.eu/the-project/research-results</li></ul>
</li>
<li><p>lemurproject.org</p>
<ol>
<li><p>lucindri</p>
<p>https://lemurproject.org/lucindri.php</p>
<p>Lucindri is an open-source implementation of Indri search logic and structured query language using the Lucene Search Engine. Lucindri consists of two components: the indexer and the searcher.</p></li>
<li><p>Galago</p>
<p>toolkit for experimenting with text search</p>
<ul>
<li>http://www.search-engines-book.com</li></ul>
</li></ol>
</li></ol>

=== Wikia ===

* https://en.wikipedia.org/wiki/Wikia_Search
* von Jimmy Wales
* Implementierung
** Grub, dezentraler Crawler
** Apache Nutch

=== Recherche 2023-11-26 ===

* https://en.wikipedia.org/wiki/Category:Web_scraping
* https://en.wikipedia.org/wiki/Category:Web_archiving
* https://en.wikipedia.org/wiki/Category:Free_search_engine_software
* https://en.wikipedia.org/wiki/Category:Internet_search_engines
* https://en.wikipedia.org/wiki/Category:Free_web_crawlers

=== Scraping ===

* https://github.com/scrapy

Version vom 7. Dezember 2023, 13:05 Uhr

archive

crawling

index

scraping

search

from org-mode/thk

dezentrale Suchmaschine

Internet Archive

Crawling, Crawler

  1. ArchiveBox

    Python, aktiv

    Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more…

  2. grab-site

    Python, aktiv https://github.com/ArchiveTeam/grab-site

  3. WASP

  4. StormCrawler

  5. HTTrack

  6. Grub.org

    dezentraler Crawler des Wikia Projektes In C#, Pyton

  7. HCE – Hierarchical Cluster Engine

  8. Heritrix

  9. Haskell

Search

  1. openwebsearch.eu

  2. lemurproject.org

    1. lucindri

      https://lemurproject.org/lucindri.php

      Lucindri is an open-source implementation of Indri search logic and structured query language using the Lucene Search Engine. Lucindri consists of two components: the indexer and the searcher.

    2. Galago

      toolkit for experimenting with text search

Wikia

Recherche 2023-11-26

Scraping

Abgerufen von „https://de.populus.wiki/w/index.php?title=Populus:DezInV/Notes&oldid=1582“