And the available settings to change with =set= are (use =set='s autocompletion):

• =details=: if true, print more details below each row (like the url). Defaults to false.
• =nb-results=: how many results to print. Defaults to 20.
• =browser=: the browser name to use to open urls. Defaults to Firefox.
• =torrent-client=: the desktop torrent client name. Defaults to Transmission.
• =cache-p=: if true (the default), use the cached results.

You can also set them in a configuration file.

Note that we are in a regular [[https://tiswww.case.edu/php/chet/readline/readline.html#SEC9][readline]] interface, so the usual keybindings and options are available. For example, Readline obeys the =~/.inputrc= startup file. You can change the way the TAB key does completion:

: TAB: menu-complete

if you add this, the first press on TAB will insert the first completion candidate (VS listing all the choices, and requiring a second keypress). For cl-torrents, it is convenient because we can enter the first result quickly: I typically do =search foo= then =download TAB=.

Note: I found out that =C-x C-r= re-reads the inputrc file, so you can try without quitting cl-torrents.

+html:

*** Configuration files (in development)

=cl-torrents= will  read two configuration files.  An ini-style one,
and a lispy one.

**** Ini-style config

First, it will search for a =~.torrents.conf= file
from =~/.config/.torrents.conf= and  =~/.torrents.conf=.  The last
one takes precedence.

For example, those are the default values:

+BEGIN_SRC conf

[default] scrapers = 1337 DOWNLOADSME browser = firefox nb-results = 20 cache-p = true # use the cache system.

+END_SRC

Moreover, you can set parameters related to the repl:

+BEGIN_SRC text

[default]

again, default values:

confirm-exit = true # ask with a yes/no prompt before quiting the app. verbose = false history = true # use the repl history. write-history = true # read the history, but don't write to it.

+END_SRC

You can set them for all replic-based apps in =~/.replic.conf=, or override them in =.torrents.conf=.

/Note: we only use a "default" profile for now./

**** Lisp init file

If the file =~/.torrents.lisp= exists, it will be loaded after the =.conf= one and before the command line arguments.

The option =--no-userinit= prevents it from loading.

You can write whatever you want there, it is just a lisp file that will be =load='ed at startup.

/Note/: what we will eventually do then is to expose cl-torrents' mechanism via hooks.

**** Extending the app, creating commands

One thing you can do is define additional commands to be found at the repl. We do so by following [[https://github.com/vindarel/replic/][replic]]'s mechanism:

• define functions inside the =torrents.user= package
• define the completion choices for some functions
• =export= the functions you want to see turned into commands. =cl-torrents=, with the help of the =replic= library, will automatically turn them into commands available at the application prompt.

You can copy the example below in =~/.torrents.lisp= and re-run =torrents -i= to try this out.

+BEGIN_SRC lisp

(in-package :torrents.user)

(defparameter names '() "List of names (string) given to hello. Will be autocompleted by goodbye.")

(defun hello (name) "Takes only one argument. Adds the given name to the global *names* global variable, used to complete arguments of goodbye. " (format t "hello ~a~&" name) (push name names))

(defun goodbye (name) "Says goodbye to name, where name should be completed from what was given to hello." (format t "goodbye ~a~&" name))

;; Custom completion for goodbye: (replic.completion:add-completion "goodbye" (lambda () names))

;; and export the two functions to find them as commands. (export '(hello goodbye))

+END_SRC

** Ideas, todos

• [X] cache (on files) v0.3
• [X] CI
• use a local copy of TPB
• [X] build with ECL. Used [[https://hub.docker.com/r/daewok/lisp-devel/][the lisp-devel]] docker image. Result: a 52Mo executable with a runtime error.
• [X] open an url
• download the torrent file
• [X] ini config file, to parse with [[https://common-lisp.net/project/py-configparser/][py-configparser]]
• [X] add torrent to a local torrent client
• [X] add the torrent to a remote transmission client, with [[https://github.com/libre-man/cl-transmission/][cl-transmission]]. v0.12
• [X] self-contained web app ([[https://lispcookbook.github.io/cl-cookbook/scripting.html][doc]]). POC: [[https://github.com/vindarel/cl-torrents-web][cl-torrents-web]].
• [X] interactive prompt, with completion of commands.

** Release notes

[[https://gitlab.com/vindarel/cl-torrents/tags][https://gitlab.com/vindarel/cl-torrents/tags]]

*** WIP

• added: a simple Tk GUI (search, open in browser, download with torrent desktop client).

*** v0.12

• june, 2021: added [[https://torrent-paradise.ml][torrents-paradise]] API.

*** v0.11

• november, 2019: added connection to transmission-remote
• added rarbg scraper (needs new tests)
• june, 2019: changed the location of the lisp init file from =~/.torrents.lisp= to =~/.config/torrents.lisp=.
• added: a =filter= command, to only display results whose title contains a given string. It reduces the TAB-completion of ids (but doesn't constrain it).
• added: all functions can now TAB-complete the list of ids.
• added: scrape and display torrents' size.
• fixed =-d= cli arg.
• added: load =~/.torrents.lisp=, create new commands from the =torrents.user= package.
• added the ability to read an ini-like config file, =~/.torrents.conf=. All exported variables from the =:torrent= package can be overriden. They are written without earmuffs:

  +BEGIN_SRC lisp

       :*nb-results*
       :*browser*
       :*torrent-client*
       :*cache-p*

  +END_SRC

  gives

  +BEGIN_SRC text

  ~/.torrents.conf

  those are the defaults.

  [default] nb-results = 20 browser = firefox torrent-client = firefox cache-p = true

  +END_SRC

  See more on replic. (all parameters need more tests)

• added missing =-o= option to download with a torrent client (transmission-gtk).
• =cl-torrents.conf= configuration file:
  • choose the list of scrapers.
• =download= command (open magnet link with transmission by default).
• use the [[https://github.com/vindarel/replic/][replic]] library to build the repl, the commands and their completions.
  • -> new builtin =highlight= and =unhighlight= commands,
  • -> better error handling (don't exit the repl on an error).
• fix cli usage without a keyword search, print help.

*** v0.10

• added [[https://www.torrentdownloads.me][torrentdownloads.me]] scraper.
• new parameter, =cache-p=: if nil, don't use the cache.

*** v0.9

• added [[http://1337x.to][1337x.to]] scraper.
• removed piratebay.to, katcr.co, torrent.cd, they're down :/

*** v0.8

• added a readline-based repl with some commands: =search=, =magnet=, =open=, with their completion. See =help=. Built with [[https://github.com/mrkkrp/cl-readline][cl-readline]], but might use [[https://github.com/vindarel/replic][replic]] anytime soon.

*** v0.7

• fix cache not created by binary
• create cache in =~/.cl-torrents/cache/=

• Dev

Clone cl-torrents in =~/quicklisp/local-projects=,

now you can =(ql:quickload :torrents)=.

We use our [[https://github.com/vindarel/replic/][replic]] library to automatically build the readline repl.

Unit tests:

: make test

End-to-end tests (checking that the websites respond and our scrapers still work):

: make test-end2end

Try the Ltk GUI: load =gui-tk.lisp= and run =(main)=.

• Resources

  Don't miss these good resources:

• http://lisp-lang.org/
• [[Https://github.com/CodyReichert/awesome-cl][Awesome-cl]]
• [[https://lispcookbook.github.io/cl-cookbook/][the Common Lisp Cookbook]]
• http://quickdocs.org/
• (my) [[https://lisp-journey.gitlab.io/][lisp-journey]]: introductory posts, lists of resources, selection of screencasts.

** Tutorial

update, 2021: this tutorial is one of the first things I wrote when discovering CL and it probably bitrot a bit. I'd recommend the Cookbook now as I ported most of its useful content there.

Writing a little web scraper like this one is not difficult. However, I had to spend some time to find out the right libraries and resources. It is also not trivial at first to start a Lisp project. So the first thing we did is write a tutorial. It is a mix of hopefully useful stuff:

• web scraping,

• async web scraping,

• trying out things at the REPL,

• where to find documentation,

• creating and loading a project,

• basic data structures and gotchas,

• some useful libraries,

• unit tests, with mocks, running tests from the shell, continuous integration,

• parsing command line arguments, creating executables, continuous delivery,

• basics of error handling,

• ...

  It will eventually tackle more topics and features (more settings, working with a local copy of TPB…) but in the meanwhile, read

  the [[https://vindarel.github.io/cl-torrents/tutorial.html][tutorial]] !

  It was built with https://github.com/fniessen/org-html-themes.

** Bypassing ISP blocking

Some Internet Service Providers block access to torrent sites, as it's the case in France. They currently don't block the HTTP version of torrents-paradise, but that one works on IPFS anyways. You can bypass them by using other DNS servers than your ISP's.

Here's how to do it on Ubuntu: https://doc.ubuntu-fr.org/dns (french)

You can use [[https://www.opennic.org/][OpenNIC]] addresses or again the ones of the [[https://www.fdn.fr/actions/dns/][French Data Network]] association.

• Licence

MIT.