Med-Mastodon

HongsterTIL that `wget` supports timestamp check.<a href="https://tech.mrleong.net/cheatsheet-examples-wget" rel="nofollow noopener" translate="no" target="_blank">https://tech.mrleong.net/cheatsheet-examples-wget</a><a href="https://fosstodon.org/tags/wget" class="mention hashtag" rel="nofollow noopener" target="_blank">#wget</a> <a href="https://fosstodon.org/tags/cli" class="mention hashtag" rel="nofollow noopener" target="_blank">#cli</a> <a href="https://fosstodon.org/tags/cheatsheet" class="mention hashtag" rel="nofollow noopener" target="_blank">#cheatsheet</a> <a href="https://fosstodon.org/tags/command" class="mention hashtag" rel="nofollow noopener" target="_blank">#command</a> <a href="https://fosstodon.org/tags/terminal" class="mention hashtag" rel="nofollow noopener" target="_blank">#terminal</a>

Luna LacteaDoes anyone know if there's a way to apply uBlock Origin filters to wget? I'm trying to download some pages, but I get trapped in a tarpit*†**‡.*Google web tracking services & advertisements. †There's not actually a tarpit, it's just fucking Google. **I've been sitting here for several minutes to download a page that should only take a few seconds. ‡I'm just trying to get offline "bookmarks" for things I want to read & reference on my laptop. <a href="https://furry.engineer/tags/Linux" class="mention hashtag" rel="nofollow noopener" target="_blank">#Linux</a> <a href="https://furry.engineer/tags/Web" class="mention hashtag" rel="nofollow noopener" target="_blank">#Web</a> <a href="https://furry.engineer/tags/wget" class="mention hashtag" rel="nofollow noopener" target="_blank">#wget</a>

Free Software FoundationAssigning your copyright to the FSF helps defend the GPL and keep software free. Thanks to Rongzhao Yan, Paolo De Santis, and Maximilian Küffner for assigning their copyright to the FSF! More at: <a href="https://u.fsf.org/463" rel="nofollow noopener" translate="no" target="_blank">https://u.fsf.org/463</a> <a href="https://hostux.social/tags/Emacs" class="mention hashtag" rel="nofollow noopener" target="_blank">#Emacs</a> \<a href="https://hostux.social/tags/Wget" class="mention hashtag" rel="nofollow noopener" target="_blank">#Wget</a> <a href="https://hostux.social/tags/CopyrightAssignments" class="mention hashtag" rel="nofollow noopener" target="_blank">#CopyrightAssignments</a>

Preston Maness ☭I've mirrored a relatively simple website (redsails.org; it's mostly text, some images) for posterity via <a href="https://tenforward.social/tags/wget" class="mention hashtag" rel="nofollow noopener" target="_blank">#wget</a>. However, I also wanted to grab snapshots of any outlinks (of which there are many, as citations/references). By default, I couldn't figure out a configuration where wget would do that out of the box, without endlessly, recursively spidering the whole internet. I ended up making a kind-of poor man's <a href="https://tenforward.social/tags/ArchiveBox" class="mention hashtag" rel="nofollow noopener" target="_blank">#ArchiveBox</a> instead:for i in $(cat others.txt) ; do dirname=$(echo "$i" | sha256sum | cut -d' ' -f 1) ; mkdir -p $dirname ; wget --span-hosts --page-requisites --convert-links --backup-converted --adjust-extension --tries=5 --warc-file="$dirname/$dirname" --execute robots=off --wait 1 --waitretry 5 --timeout 60 -o "$dirname/wget-$dirname.log" --directory-prefix="$dirname/" $i ; doneBasically, there's a list of bookmarks^W URLs in others.txt that I grabbed from the initial mirror of the website with some <a href="https://tenforward.social/tags/grep" class="mention hashtag" rel="nofollow noopener" target="_blank">#grep</a> foo. I want to do as good of a mirror/snapshot of each specific URL as I can, without spidering/mirroring endlessly all over. So, I hash the URL, and kick off a specific wget job for it that will span hosts, but only for the purposes of making the specific URL as usable locally/offline as possible. I know from experience that this isn't perfect. But... it'll be good enough for my purposes. I'm also stashing a WARC file. Probably a bit overkill, but I figure it might be nice to have.<a href="https://tenforward.social/tags/RedSails" class="mention hashtag" rel="nofollow noopener" target="_blank">#RedSails</a> <a href="https://tenforward.social/tags/archive" class="mention hashtag" rel="nofollow noopener" target="_blank">#archive</a> <a href="https://tenforward.social/tags/archival" class="mention hashtag" rel="nofollow noopener" target="_blank">#archival</a> <a href="https://tenforward.social/tags/archiving" class="mention hashtag" rel="nofollow noopener" target="_blank">#archiving</a> <a href="https://tenforward.social/tags/warc" class="mention hashtag" rel="nofollow noopener" target="_blank">#warc</a>

Claudine C :antifa:Brief usage notes on <a href="https://aus.social/tags/ArchiveBox" class="mention hashtag" rel="nofollow noopener" target="_blank">#ArchiveBox</a> and <a href="https://aus.social/tags/wget" class="mention hashtag" rel="nofollow noopener" target="_blank">#wget</a> <a href="https://www.claudinec.net/posts/2025-02-08-web-archiving/" rel="nofollow noopener" translate="no" target="_blank">https://www.claudinec.net/posts/2025-02-08-web-archiving/</a><a href="https://aus.social/tags/SafeguardingResearch" class="mention hashtag" rel="nofollow noopener" target="_blank">#SafeguardingResearch</a> <a href="https://aus.social/tags/DigitalPreservation" class="mention hashtag" rel="nofollow noopener" target="_blank">#DigitalPreservation</a> <a href="https://aus.social/tags/WebArchiving" class="mention hashtag" rel="nofollow noopener" target="_blank">#WebArchiving</a>

GNU/Linux.chHTTrack - Der Website DownloaderIn diesem Tutorial zeige ich dir, wie du ganze Websites mit HTTrack für den Offline-Zugriff speichern kannst. Egal, ob für die eigene Sicherung oder einfach zum Stöbern ohne Internet – ich zeige dir Schritt für Schritt, wie es funktioniert. <a href="https://social.anoxinon.de/tags/httrack" class="mention hashtag" rel="nofollow noopener" target="_blank">#httrack</a> <a href="https://social.anoxinon.de/tags/Curl" class="mention hashtag" rel="nofollow noopener" target="_blank">#Curl</a> <a href="https://social.anoxinon.de/tags/wget" class="mention hashtag" rel="nofollow noopener" target="_blank">#wget</a> <a href="https://social.anoxinon.de/tags/Website" class="mention hashtag" rel="nofollow noopener" target="_blank">#Website</a> <a href="https://social.anoxinon.de/tags/Linux" class="mention hashtag" rel="nofollow noopener" target="_blank">#Linux</a><a href="https://gnulinux.ch/httrack-der-website-downloader" rel="nofollow noopener" translate="no" target="_blank">https://gnulinux.ch/httrack-der-website-downloader</a>

Snagburz<a href="https://techhub.social/tags/unix" class="mention hashtag" rel="nofollow noopener" target="_blank">#unix</a> and <a href="https://techhub.social/tags/linux" class="mention hashtag" rel="nofollow noopener" target="_blank">#linux</a> tip: The page you try to read on the web is a PDF file, but it takes the <a href="https://techhub.social/tags/browser" class="mention hashtag" rel="nofollow noopener" target="_blank">#browser</a> forever to display it? From a terminal download it using wget <url><a href="https://techhub.social/tags/wget" class="mention hashtag" rel="nofollow noopener" target="_blank">#wget</a> <a href="https://techhub.social/tags/offline" class="mention hashtag" rel="nofollow noopener" target="_blank">#offline</a> <a href="https://techhub.social/tags/pc" class="mention hashtag" rel="nofollow noopener" target="_blank">#pc</a> <a href="https://techhub.social/tags/tech" class="mention hashtag" rel="nofollow noopener" target="_blank">#tech</a> <a href="https://techhub.social/tags/shell" class="mention hashtag" rel="nofollow noopener" target="_blank">#shell</a>

Kevin Karhan :verified:<a href="https://mastodon.social/@bagder" class="u-url mention" rel="nofollow noopener" target="_blank">@bagder</a> Problem with that is (besides occasional bugfixes), most people including myself would see <a href="https://infosec.space/tags/curl" class="mention hashtag" rel="nofollow noopener" target="_blank">#curl</a> to be functionally complete and anything "nice to have" would be considered not worth the balooning in <a href="https://infosec.space/tags/complexity" class="mention hashtag" rel="nofollow noopener" target="_blank">#complexity</a> and <a href="https://infosec.space/tags/size" class="mention hashtag" rel="nofollow noopener" target="_blank">#size</a>. <ul><li>I mean, does <a href="https://curl.se/" rel="nofollow noopener" target="_blank">curl</a> need to be able to do <a href="https://infosec.space/tags/BitTorrent" class="mention hashtag" rel="nofollow noopener" target="_blank">#BitTorrent</a> (magnet:), <a href="https://infosec.space/tags/IPFS" class="mention hashtag" rel="nofollow noopener" target="_blank">#IPFS</a> (ipfs://) or god forbid <a href="https://infosec.space/tags/blockchain" class="mention hashtag" rel="nofollow noopener" target="_blank">#blockchain</a> (i.e. <a href="https://infosec.space/tags/EVM" class="mention hashtag" rel="nofollow noopener" target="_blank">#EVM</a>) support? </li><li>Do you really want to integrate <a href="https://mastodon.social/@torproject" class="u-url mention" rel="nofollow noopener" target="_blank">@torproject</a> / <a href="https://infosec.space/tags/Tor" class="mention hashtag" rel="nofollow noopener" target="_blank">#Tor</a> support natively into curl when using <a href="https://infosec.space/tags/HTTP" class="mention hashtag" rel="nofollow noopener" target="_blank">#HTTP</a> (localhost:8118) and <a href="https://infosec.space/tags/SOCKS5" class="mention hashtag" rel="nofollow noopener" target="_blank">#SOCKS5</a> (localhost:9050) <a href="https://infosec.space/tags/proxy" class="mention hashtag" rel="nofollow noopener" target="_blank">#proxy</a> allows for the same and doesn't necessitate having to handle and ingest Tor arguments as well??</li></ul>In fact if <a href="https://infosec.space/tags/toybox" class="mention hashtag" rel="nofollow noopener" target="_blank">#toybox</a> didn't have a <a href="https://infosec.space/tags/wget" class="mention hashtag" rel="nofollow noopener" target="_blank">#wget</a> implementation that I could use for OS/1337 I would've merely chosen <a href="https://curl.se/tiny/" rel="nofollow noopener" target="_blank"><code>tiny-curl -o</code></a> as a <a href="https://stackoverflow.com/questions/30702803/set-aliases-globally-for-all-users" rel="nofollow noopener" target="_blank">global alias</a> or if <a href="https://infosec.space/tags/tinycurl" class="mention hashtag" rel="nofollow noopener" target="_blank">#tinycurl</a> wasn't an option, <a href="https://linux.die.net/man/1/curl" rel="nofollow noopener" target="_blank"><code>curl -o</code></a> instead.<ul><li>Maybe someone who wants to have said functionality like <code>tor</code> support built-in will go and IDK make i.e. <code>#neocurl</code> or sth. along those lines or build something like <code>#ethcurl</code> or <code>#torcurl</code>or <code>#ipfscurl</code> or whatever...</li></ul>That being said I am glad <code>curl</code> isn't solely maintained by you but has other contributors (give them a shoutout!) but I also am glad you maintain that vital software that most "<a href="https://infosec.space/tags/TechIlliterate" class="mention hashtag" rel="nofollow noopener" target="_blank">#TechIlliterate</a> <a href="https://infosec.space/tags/Normies" class="mention hashtag" rel="nofollow noopener" target="_blank">#Normies</a>" most likely never heard of but propably use on a daily basis as part of all the <a href="https://infosec.space/tags/tech" class="mention hashtag" rel="nofollow noopener" target="_blank">#tech</a> they use to <a href="https://infosec.space/tags/consume" class="mention hashtag" rel="nofollow noopener" target="_blank">#consume</a> media with...<ul><li>I consider curl to be "the <a href="https://infosec.space/tags/vim" class="mention hashtag" rel="nofollow noopener" target="_blank">#vim</a> of downloaders" (tho that's kinda insulting and limiting since <code>curl</code> is more than just a downloader and more intuitive than <code>vim</code>) with wget being "the <a href="https://infosec.space/tags/vi" class="mention hashtag" rel="nofollow noopener" target="_blank">#vi</a> of downloaders" (tho <code>wget</code> is even simpler to use than <code>vi</code>)...</li></ul>Either way, curl is awesome... <a href="https://infosec.space/tags/OS1337" class="mention hashtag" rel="nofollow noopener" target="_blank">#OS1337</a> <a href="https://infosec.space/tags/Enshittification" class="mention hashtag" rel="nofollow noopener" target="_blank">#Enshittification</a> <a href="https://infosec.space/tags/Bloat" class="mention hashtag" rel="nofollow noopener" target="_blank">#Bloat</a> <a href="https://infosec.space/tags/Bloatware" class="mention hashtag" rel="nofollow noopener" target="_blank">#Bloatware</a>

Thoralf Will 🇺🇦🇮🇱🇹🇼Jetzt habe ich das Ding zugreifbar. Ich kann mich an der Webschnittstelle auch anmelden.Interessanterweise funktioniert der Zugriff mittels <a href="https://soc.umrath.net/tags/curl" class="mention hashtag" rel="nofollow noopener" target="_blank">#curl</a> (401) nicht (und alle Scripten, die darauf basieren schlagen fehlt) - während ich mit <a href="https://soc.umrath.net/tags/wget" class="mention hashtag" rel="nofollow noopener" target="_blank">#wget</a> ein 200 und sinnvollen payload bekomme.curl -v -k -u XXX:YYY https://192.../startwget --no-check-certificate -v --user=XXX --password=YYY https://192.../startKann mir mal jemand über die Straße helfen?

dusoftAlso: it must be really a gargantuan task to do a simple GET to fetch followers, right? Neither <a href="https://fosstodon.org/tags/curl" class="mention hashtag" rel="nofollow noopener" target="_blank">#curl</a>, nor <a href="https://fosstodon.org/tags/wget" class="mention hashtag" rel="nofollow noopener" target="_blank">#wget</a> can do that, right? Right? <a href="https://fosstodon.org/tags/fediverse" class="mention hashtag" rel="nofollow noopener" target="_blank">#fediverse</a> <a href="https://fosstodon.org/tags/mastodon" class="mention hashtag" rel="nofollow noopener" target="_blank">#mastodon</a> <a href="https://fosstodon.org/tags/mastoadmin" class="mention hashtag" rel="nofollow noopener" target="_blank">#mastoadmin</a> <a href="https://fosstodon.org/tags/MastoLivre" class="mention hashtag" rel="nofollow noopener" target="_blank">#MastoLivre</a> <a href="https://fosstodon.org/tags/webdev" class="mention hashtag" rel="nofollow noopener" target="_blank">#webdev</a>

daniel:// stenberg://I bet you don't remember when <a href="https://mastodon.social/tags/wget" class="mention hashtag" rel="nofollow noopener" target="_blank">#wget</a> briefly considered switching to using <a href="https://mastodon.social/tags/libcurl" class="mention hashtag" rel="nofollow noopener" target="_blank">#libcurl</a> back in 2007.<a href="https://daniel.haxx.se/blog/2007/10/27/wget-going-libcurl/" rel="nofollow noopener" translate="no" target="_blank">https://daniel.haxx.se/blog/2007/10/27/wget-going-libcurl/</a>

don Elías (como los buses) 🥨<a href="https://www.impulsait.com/@impulsait.com" class="u-url mention" rel="nofollow noopener" target="_blank">@impulsait.com</a> esto es un comentario desde Mastodon.CR en el blog de <a href="https://www.impulsait.com/@impulsait.com" class="u-url mention" rel="nofollow noopener" target="_blank">@impulsait.com</a>La publicación de blog es sobre cómo usar <a href="https://mastodon.cr/tags/wget" class="mention hashtag" rel="nofollow noopener" target="_blank">#wget</a> para hacer una copia fuera de línea de un sitio web, especialmente útil para convertir un sitio dinámico en estático por temas de rendimiento y seguridad

Impulsa IT SolutionsHacer una copia o espejo de un sitio usando wgetA veces uno necesita hacer una copia completa de un sitio web para acceso fuera de línea, o también para convertir un sitio dinámico a estático, especialmente cuando un cliente tiene un sitio web en una versión obsoleta de un manejador de contenido como WordPress o Joomla en una versión muy vieja y corriendo una versión de PHP vulnerable. Esto es especialmente útil cuando el sitio web no requiere ser modificado. Esto trae como beneficio un sitio más veloz y más seguro, ya que un sitio estático HTML se envía tal cual al navegador de las personas usuarias, sin tener que ejecutar código PHP potencialmente vulnerable.Con la herramienta de línea de comandos <code>wget</code> es posible de forma muy sencilla descargar un sitio web entero:<pre><code>wget --mirror --convert-links --adjust-extension --page-requisites --no-parent http://example.org</code></pre>Explicación de algunos de los argumentos:<ul><li><code>--mirror</code> – Descarga recursivamente y hace otros ajustes relacionados, implica: <code>-r -N -l inf --no-remove-listing</code>.</li><li><code>--convert-links</code> – convierte enlaces y referencias en HTML como CSS, para poder ser vistas fuera de línea.</li><li><code>--adjust-extension</code> – Ajusta la extensión de los objetos html o css según el tipo de datos</li><li><code>--page-requisites</code> – Descarga objetos enlazados como hojas de estilo CSS e imágenes requeridos para ver el sitio fuera de línea.</li><li><code>--no-parent</code> – No subir a directorios superiores en el URL, útil cuando se está haciendo espejo de un URL específico como <a href="http://example.org/sub-directorio/" rel="nofollow noopener" translate="no" target="_blank">http://example.org/sub-directorio/</a></li></ul>Ejemplo de conversión de sitio WordPress en un sistema cPanel:<pre><code># Entrar como usuario a shell, para no ejecutar comandos como rootsu - user -s /bin/bash# Crear directorio de descargasmkdir download && cd downloadwget --mirror --convert-links --adjust-extension --page-requisites --no-parent http://example.org# Hacer un respaldo del directorio raíz del sitio webcd ..mv public_html public_html~# Colocar la copia descargada en líneamv download/example.org public_html# Probar sitio, ajustar detalles y opcionalmente borrar base de datos luego de hacer respaldo# Con esto el sitio dinámico ha sido convertido a estático.</code></pre>Referencias:<ul><li>GNU wget manual: <a href="https://www.gnu.org/software/wget/manual/wget.html" rel="nofollow noopener" target="_blank">https://www.gnu.org/software/wget/manual/wget.html </a></li><li>Make Offline Mirror of a Site using wget: <a href="https://www.guyrutenberg.com/2014/05/02/make-offline-mirror-of-a-site-using-wget/" rel="nofollow noopener" target="_blank">https://www.guyrutenberg.com/2014/05/02/make-offline-mirror-of-a-site-using-wget/</a></li></ul><a rel="nofollow noopener" class="hashtag u-tag u-category" href="https://www.impulsait.com/tag/linux/" target="_blank">#Linux</a> <a rel="nofollow noopener" class="hashtag u-tag u-category" href="https://www.impulsait.com/tag/mirror/" target="_blank">#mirror</a> <a rel="nofollow noopener" class="hashtag u-tag u-category" href="https://www.impulsait.com/tag/static-site/" target="_blank">#staticSite</a> <a rel="nofollow noopener" class="hashtag u-tag u-category" href="https://www.impulsait.com/tag/sysadmin/" target="_blank">#sysadmin</a> <a rel="nofollow noopener" class="hashtag u-tag u-category" href="https://www.impulsait.com/tag/tips/" target="_blank">#tips</a> <a rel="nofollow noopener" class="hashtag u-tag u-category" href="https://www.impulsait.com/tag/wget/" target="_blank">#wget</a>

Recent searches

Search options

Administered by:

Server stats:

#wget