rnhmjoj
06d864d49f
Since sqlite 3.34[1] deleting a record in a fts5 table that doesn't
exist became an error. The resuls it that adding a torrent "corrupts"
the database and magnetico fails with this error:
Could not add new torrent to the database.
tx.Exec (INSERT OR REPLACE INTO torrents) database disk image is malformed
The problem can be simply avoided by setting modified_on = discovered_on
when inserting the torrent, instead of updating it afterwards with a
trigger.
[1]:
|
||
|---|---|---|
| .. | ||
| persistence | ||
| util | ||
| README.md | ||
magnetico/pkg
- The most significant package is
persistence, that abstracts access to the magnetico databases with different engines (currently: SQLite, stdout and PostgreSQL).
For REST-ful magneticow API, see https://app.swaggerhub.com/apis/boramalper/magneticow-api/.
PostgreSQL database engine
PostgreSQL database engine uses PostgreSQL to store indexed torrents. It's more performant and flexible than SQLite but requires additional software configuration.
WARNING: pg_trgm extension required. You need to enable it for your database before starting magneticod:
CREATE EXTENSION pg_trgm;
Engine usage example:
magneticod --database=postgres://username:password@127.0.0.1:5432/database?schema=custom_schema_name&sslmode=disable
See pgx repository for more examples.
Optional parameter schema was added to choose which schema will be used to store magnetico tables,
sequences and indexes.
Beanstalk MQ engine for magneticod
Beanstalkd is very lightweight and simple MQ server implementation. You can use it to organize delivery of the indexed data to your application.
Use beanstalk URL schema to connect to beanstalkd server. For example:
magneticod --database=beanstalkd://127.0.0.1:11300/magneticod_tube
Don't forget to set binlog persistence, change maximum job size
and fsync() period to be able to reliably save torrents with a large number of files:
# Example settings (may not work for you)
beanstalkd -z 1048560 -b /var/lib/beanstalkd -f 2400000
For job data example see stdout engine documentation below as beanstalk engine uses the same format.
Stdout Dummy Database Engine for magneticod
Stdout dummy database engine for magneticod prints a new JSON Line for each discovered torrent so that you can pipe the stdout of magneticod into some other program to build your own pipelines on the fly!
Example Output
{"infoHash":"f84b51f0d2c3455ab5dabb6643b4340234cd036e","name":"Big_Buck_Bunny_1080p_surround_frostclick.com_frostwire.com","files":[{"size":928670754,"path":"Big_Buck_Bunny_1080p_surround_FrostWire.com.avi"},{"size":5008,"path":"PROMOTE_YOUR_CONTENT_ON_FROSTWIRE_01_06_09.txt"},{"size":3456234,"path":"Pressrelease_BickBuckBunny_premiere.pdf"},{"size":180,"path":"license.txt"}]}
WARNING:
Please beware that the schema of the object (dictionary) might change in backwards-incompatible ways in the future; although I'll do my best to ensure it won't happen.