Commit Graph

12317 Commits

Author SHA1 Message Date
Ryan Roden-Corrent
c007f592b3 Use more intuitive argument order in sql.delete. 2017-06-29 20:43:42 -04:00
Ryan Roden-Corrent
9c0c174534 Use builtin SortFilter regex functionality.
With the new completion API, we no longer need a custom filterAcceptsRow
function. This was necessary to handle the tree structure of the model,
but now we use a separate QSortFilterProxyModel for each category, so
the data it filters is flat. We can simplify the code by using the
builtin setFilterRegExp.

This changes the behavior a little, as now all list categories filter on
all columns. This should be beneficial if anything. For example, help
topics are now filtered on description in addition to name.

This also seems to slightly speed up filtering, according to the url
model benchmark.

Before:

----------------------------------------------- benchmark: 1 tests ----------------------------------------------
Name (time in s)                     Min     Max    Mean  StdDev  Median     IQR  Outliers(*)  Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------
test_url_completion_benchmark     1.2806  1.3817  1.3195  0.0390  1.3068  0.0487          1;0       5           1
-----------------------------------------------------------------------------------------------------------------

After:

----------------------------------------------- benchmark: 1 tests ----------------------------------------------
Name (time in s)                     Min     Max    Mean  StdDev  Median     IQR  Outliers(*)  Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------
test_url_completion_benchmark     1.1183  1.1508  1.1281  0.0132  1.1241  0.0142          1;0       5           1
-----------------------------------------------------------------------------------------------------------------
2017-06-29 12:44:02 -04:00
Ryan Roden-Corrent
6ac940fa32 Fix pylint/coverage errors.
Ensure 100% coverage for sqlcategory and history, and fix some linter
errors
2017-06-27 12:33:51 -04:00
Ryan Roden-Corrent
f06880c6e2 Fix history completion delete function.
In order to update SqlQueryModel's rowCount after re-running the query,
we must call setQuery again.
2017-06-27 08:42:10 -04:00
Ryan Roden-Corrent
62a849c2db Fix bugs introduced in test_models 2017-06-26 12:41:48 -04:00
Ryan Roden-Corrent
46161c3af0 Refactor delete_cur_item.
Taking the completion widget as an argument was overly complex.
The process now looks like:

1. CompletionView gets deletion request
2. CompletionView passes selected index to CompletionModel
3. CompletionModel passes the row data to the owning category
4. The category runs its custom completion function.

This also fixes a bug. With the switch to the hybrid (list/sql)
completion model, the view was no longer updating when items were
deleted. This fixes that by ensuring the correct signals are emitted.

The SQL model must be refreshed by running the query. We could try using
a SqlTableModel so we can call removeRows instead.

The test for deleting a url fails because qmodeltester claims the length
of the query model is still 3.
2017-06-26 08:57:36 -04:00
Ryan Roden-Corrent
866f4653c7 Fix spelling existant -> existent. 2017-06-25 22:14:38 -04:00
Ryan Roden-Corrent
6080830a8b Fix outdated docstring and pylint error. 2017-06-20 22:25:09 -04:00
Ryan Roden-Corrent
b722cc1dec Pass invalid index to [can]FetchMore.
For QSqlQueryModel, the argument should always be an invalid index:
http://doc.qt.io/qt-5/qsqlquerymodel.html#canFetchMore

For a QStandardItemModel, it doesn't matter. Either way, passing the
top-level parent index was wrong.
2017-06-20 22:11:53 -04:00
Ryan Roden-Corrent
63cb88a0f4 Use _cat_from_index in completionmodel.data.
Keep all the category lookup inside _cat_from_idx for easier refactoring
if the organization ever changes.
2017-06-20 22:08:23 -04:00
Ryan Roden-Corrent
0f585eda4f Bring history.py back to 100% coverage.
The code of debug_dump_history was tweaked to handle a possible
OSException that can be thrown by open, which I noticed while trying to
test it.
2017-06-20 21:41:43 -04:00
Florian Bruhin
038dcff4ba Ignore common URL issues while importing history
See #2646.

This ignores the "corrupted" Apple-lookalike URLs, comments and data: URLs.
2017-06-20 12:19:14 +02:00
Florian Bruhin
f838eb1bdc Use named formatting for queries in sql.py 2017-06-20 12:11:41 +02:00
Florian Bruhin
da875755d1 Add spaces after commas in SQL queries 2017-06-20 12:06:17 +02:00
Florian Bruhin
c1776bbf9d Add error message when query failed to prepare 2017-06-20 12:05:40 +02:00
Florian Bruhin
29ce1b3811 Add column order in SqlTable._insert_query
_insert_query gets called with a query and dict of values such as:

  {'val': 1, 'lucky': False, 'name': 'one'}

Via bindValues(), we only assign a placeholder in the query string to a value,
so we get a query with bindings like:

  INSERT INTO Foo values(:lucky,:val,:name)
  {':name': 'one', ':val': 1, ':lucky': False}

So what we're executing is something like:

  INSERT INTO Foo values(false,1,"one")

However, if the column order in the database doesn't happen to be the order
we're passing the values in, we get the wrong values in the wrong columns.

Instead, we now do:

  INSERT INTO Foo (lucky, val, name) values(false,1,"one")

Which inserts the values in the order we intended.

With Python 3.6, this just happened to work before because we always passed the
keyword arguments in the table column order, and in 3.6 dicts
(and thus **kwargs) happen to be ordered:
https://mail.python.org/pipermail/python-dev/2016-September/146327.html
2017-06-20 12:03:43 +02:00
Florian Bruhin
4296a61b9a tests: Clean up check_history 2017-06-19 07:44:11 -04:00
Florian Bruhin
9f94f28181 Use :memory: for an in-memory database
Using an empty string for the same purpose only started working in some
recent-ish Qt/sqlite/? version, so using --version failed on Ubuntu Trusty.
2017-06-19 07:44:11 -04:00
Ryan Roden-Corrent
051d2665f3 Fix signal type error in CompletionView.
On Travis CI we are sometimes seeing:

```
CompletionView.selection_changed[str].emit():
argument 1 has unexpected type 'int'
```

Cast the data to a string before emitting it just to be safe.
2017-06-19 07:44:11 -04:00
Ryan Roden-Corrent
891a6bcf14 Fix flake8 errors 2017-06-19 07:44:11 -04:00
Ryan Roden-Corrent
c7a18a8b8d Fix tests for recent sql changes 2017-06-19 07:44:11 -04:00
Ryan Roden-Corrent
4e87773d89 Use a dict instead of named params for insert.
This allows replace to be a named parameter and allows consolidating
some duplicate code between various insert methods.

This also fixes some tests that broke because batch insert was broken.
2017-06-19 07:44:11 -04:00
Ryan Roden-Corrent
e436f48164 Small sql fixes.
- Remove unused SqlTable.Entry
- Fix wording of two log messages
- Remove unused import
2017-06-19 07:44:11 -04:00
Ryan Roden-Corrent
f4f52ee204 Remove history.Entry.
No longer needed with sql backend. Query results build their own
namedtuple from the returned columns, and inserting new entries is just
done with named parameters.
2017-06-19 07:44:11 -04:00
Ryan Roden-Corrent
1fe1813431 Fix pylint errors. 2017-06-19 07:44:11 -04:00
Ryan Roden-Corrent
cf23f42b99 Use splitlines in test_history_bdd again.
Just using read() returns a single string, and iterating over that
iterates over each character.
2017-06-19 07:44:11 -04:00
Ryan Roden-Corrent
679e001a48 Separate sqlcategory title from table name.
Also fix a number of sql/completion tests that were failing.
2017-06-19 07:44:11 -04:00
Ryan Roden-Corrent
61a1709141 Fix completion selection bug.
Fix the issue where pressing `o<esc>o` would show a url completion
dialog where attempting to <Tab> select items would do nothing but show
a Qt warning.

The fix is to ensure we set _last_completion_func to None whenever we
clear completion (there was a case I missed).

It also ensures we always delete the old model and adds a safety to
prevent deleting an in-use model is set_model is called with the current
model.
2017-06-19 07:44:11 -04:00
Ryan Roden-Corrent
fa39b82b3c Backup old history file after import.
Instead of removing it, move it to history.bak.
2017-06-19 07:44:11 -04:00
Ryan Roden-Corrent
a6a9ad72f9 Fix test_history_interface.
This was still using a history dict instead of SQL history.
2017-06-19 07:44:11 -04:00
Ryan Roden-Corrent
22b7b21d5a Use named placeholders for sql queries. 2017-06-19 07:44:11 -04:00
Ryan Roden-Corrent
862f8d3188 Always return col 0 for index parent.
This was changed during code review but was causing Qt errors while
TAB-completing in the selection view:

08:42:34 WARNING  qt         Unknown module:none:0 Can't select indexes from different model or with different parents
2017-06-19 07:44:11 -04:00
Florian Bruhin
18cd8ba0b6 Add indices for HistoryAtimeIndex and CompletionHistoryAtimeIndex
before
------

sqlite> SELECT * FROM History where not redirect and not url like "qute://%" and atime > ? and atime <= ? ORDER BY atime desc;
Run Time: real 0.072 user 0.063334 sys 0.010000

sqlite> explain query plan SELECT * FROM History where not redirect and not url like "qute://%" and atime > ? and atime <= ? ORDER BY atime desc;
0|0|0|SCAN TABLE History
0|0|0|USE TEMP B-TREE FOR ORDER BY

sqlite> explain query plan select url, title, strftime('%Y-%m-%d', last_atime, 'unixepoch') from CompletionHistory where (url like "%qute%" or title like "%qute%") order by last_atime desc;
0|0|0|SCAN TABLE CompletionHistory
0|0|0|USE TEMP B-TREE FOR ORDER BY

after
-----

sqlite> SELECT * FROM History where not redirect and not url like "qute://%" and atime > ? and atime <= ? ORDER BY atime desc;
Run Time: real 0.000 user 0.000000 sys 0.000000

sqlite> explain query plan SELECT * FROM History where not redirect and not url like "qute://%" and atime > ? and atime <= ? ORDER BY atime desc;
0|0|0|SEARCH TABLE History USING INDEX AtimeIndex (atime>? AND atime<?)

sqlite> explain query plan select url, title, strftime('%Y-%m-%d', last_atime, 'unixepoch') from CompletionHistory where (url like "%qute%" or title like "%qute%") order by last_atime desc;
0|0|0|SCAN TABLE CompletionHistory USING INDEX CompletionAtimeIndex
2017-06-19 07:44:11 -04:00
Ryan Roden-Corrent
3a4ef09f58 More sql code review fixes 2017-06-19 07:44:11 -04:00
Florian Bruhin
9b0395db08 Add an lru cache for WebHistoryInterface.historyContains
When loading heise.de, for some crazy reason QtWebKit calls historyContains
about 16'000 times.

With this cache (which we simply clear when *any* page has been loaded, as then
the links which have been visited can change), that's down to 250 or so...
2017-06-19 07:44:11 -04:00
Ryan Roden-Corrent
6fc61d12fc Assorted small fixes for sql code review. 2017-06-19 07:44:11 -04:00
Ryan Roden-Corrent
feed9c8936 Better exception handling in history.
- Show an error message when import fails, not a generic crash dialog
- Raise CommandError when debug-dump-history fails
- Check that the path exists for debug-dump-history
2017-06-19 07:44:11 -04:00
Florian Bruhin
6ce52f39ae Add debug timings for SQL 2017-06-19 07:44:11 -04:00
Florian Bruhin
57d96a4512 Add a CompletionHistory instead of HistoryVisits table 2017-06-19 07:44:11 -04:00
Florian Bruhin
c64b7d00e6 Add separate table for history visits 2017-06-19 07:44:11 -04:00
Ryan Roden-Corrent
7f27603772 Fix columns_to_filter for sql category. 2017-06-19 07:44:11 -04:00
Ryan Roden-Corrent
389e1b0178 Fix bad objreg reference in app.py. 2017-06-19 07:44:11 -04:00
Ryan Roden-Corrent
f8325cbbc1 Remove print statement 2017-06-19 07:44:11 -04:00
Ryan Roden-Corrent
309b6ba32c Move _import_history to history.py.
Also adjusts the history import test to operate at a higher level and
ensure the old text file is removed (or isn't, in the case of an error).
2017-06-19 07:44:11 -04:00
Ryan Roden-Corrent
ea0b3eee05 Use full, not partial index for history.
historyContains includes redirect urls, so we actually don't want a
partial index here.
2017-06-19 07:44:11 -04:00
Ryan Roden-Corrent
478a719f77 Use a prepared query for historyContains.
This is called often, hopefully a prepared query will speed it up.
This also modifies Query.run to return self for easier chaining, so you
can use `query.run.value()` instead of `query.run` ; query.value()`.
2017-06-19 07:44:11 -04:00
Ryan Roden-Corrent
5b827cf86a Fix typo in sql exception handling 2017-06-19 07:44:11 -04:00
Ryan Roden-Corrent
6a0fc5afd2 Create a SQL index on History.url.
This will hopefully speed up historyContains but does not seem to speed
up the completion query, unfortunately.
2017-06-19 07:44:11 -04:00
Ryan Roden-Corrent
565ba23f8c Don't instantiate completion models nedlessly.
For real this time. A mistake on the last commit like this meant models
were still spuriously instantiated.
Now that the completion model is reused, the layoutChanged signal needs
to be forwarded through, otherwise the view will not update.
2017-06-19 07:44:11 -04:00
Ryan Roden-Corrent
42243d3d97 Add more performance logging to completion. 2017-06-19 07:44:11 -04:00