When upgrading from an old table that used different url formatting, two
entries might map to the same key, so we'll need to replace the previous
entry to avoid a primary key conflict.
Incrementing _USER_VERSION in the source will cause the
HistoryCompletion table to regenerate when users update.
This is currently necessary to support some recent formatting fixes, but
could be incremented again in the future for other changes.
If the HistoryCompletion table is removed, regenerate it from the
History table. This allows users to manually edit History, then remove
HistoryCompletion to prompt regeneration.
See #2903.
Encode urls that are inserted into the history, but do not encode urls
for completion (other than removing passwords).
Also ensure that urls read from the history text file are formatted
consistenly with those added while browsing.
Fixes#2903.
Get qutebrowser to the point where it can at least start
- Declare _messages earlier in MessageView.__init__ so it is set before
the config trigger tries to access it.
- Remove unused configmodel completion functions
- Move bind completion to configmodel to avoid a circular import with
the config module
- Fix some config accesses (forgot to use .val)
- Fix old Completion.CompletionKind references
The added/removed signals for the urlmark managers are no longer used as
the completion models are generated on-the-fly. The changed signal is
still needed so the save-manager knows when to trigger a write to disk.
Also removes session_manager.update_completion, which is no longer
needed for the same reason as above.
keyconf.changed cannot be removed, as it is still wired up to
basekeyparser.
Resolves#2874.
We need to tell sqlite to convert the timestamps to localtime during
formatting, otherwise it formats them as though you are in UTC.
Also fix up a few uses of mktime.
For some reason, calling search.clear() while no search is displayed causes the
backends to un-focus inputs, and with QtWebKit, even hinting can't focus them
again after that.
The javascript history page was requesting the new start_time in ms, but
the python code was expecting seconds. This is fixed by removing all the
millisecond translations in the python code and only translating to
milliseconds in the javascript code that formats dates.
- Fix flake8
- history.clear should also clear completion table
- call _resize_columns in set_model, not set_pattern
- add more unit-testing for the history completion table
- remove outdated comment
- fix sql init error message
- clean up history text import code
- fix test_history file path in coverage check
- use real web history, not stub, for completion model tests
- use qtmodeltester in sql/list_category tests
- test url encoding in history tests
- fix test_clear by using a callable mock
- remove test_debug_dump_history_oserror as the check is now the same as
for the file not existing
- rename nonempty to data in test_completionmodel
- add more delete_cur_item tests
- test empty option/value completion
- Fix outdated comments
- Use mock specs when possible
- More precise error message check in test_import_txt_invalid.
- Fix copyright message
- Tweak missing pyqt error message
- Dead code: remove group_by and where from sqlcategory.
With the new separate completion table, these are no longer used.
- Move test_history out of webkit/. History is no longer purely webkit
related, it could be webengine.
This ensures we actually know when an AttributeError happens.
It also changes most external code to use the correct environment, rather than
simply creating a jinja2.Template, which wouldn't use the more tightened
environment.
There were two issues here:
- The comparison was backwards, causing scroller.at_bottom() to always return
true.
- When zoomed in, jsret['px']['y'] can be a float, which means we can be
slightly off when checking the difference - math.ceil() fixes that.
Taking the completion widget as an argument was overly complex.
The process now looks like:
1. CompletionView gets deletion request
2. CompletionView passes selected index to CompletionModel
3. CompletionModel passes the row data to the owning category
4. The category runs its custom completion function.
This also fixes a bug. With the switch to the hybrid (list/sql)
completion model, the view was no longer updating when items were
deleted. This fixes that by ensuring the correct signals are emitted.
The SQL model must be refreshed by running the query. We could try using
a SqlTableModel so we can call removeRows instead.
The test for deleting a url fails because qmodeltester claims the length
of the query model is still 3.
This allows replace to be a named parameter and allows consolidating
some duplicate code between various insert methods.
This also fixes some tests that broke because batch insert was broken.
No longer needed with sql backend. Query results build their own
namedtuple from the returned columns, and inserting new entries is just
done with named parameters.
before
------
sqlite> SELECT * FROM History where not redirect and not url like "qute://%" and atime > ? and atime <= ? ORDER BY atime desc;
Run Time: real 0.072 user 0.063334 sys 0.010000
sqlite> explain query plan SELECT * FROM History where not redirect and not url like "qute://%" and atime > ? and atime <= ? ORDER BY atime desc;
0|0|0|SCAN TABLE History
0|0|0|USE TEMP B-TREE FOR ORDER BY
sqlite> explain query plan select url, title, strftime('%Y-%m-%d', last_atime, 'unixepoch') from CompletionHistory where (url like "%qute%" or title like "%qute%") order by last_atime desc;
0|0|0|SCAN TABLE CompletionHistory
0|0|0|USE TEMP B-TREE FOR ORDER BY
after
-----
sqlite> SELECT * FROM History where not redirect and not url like "qute://%" and atime > ? and atime <= ? ORDER BY atime desc;
Run Time: real 0.000 user 0.000000 sys 0.000000
sqlite> explain query plan SELECT * FROM History where not redirect and not url like "qute://%" and atime > ? and atime <= ? ORDER BY atime desc;
0|0|0|SEARCH TABLE History USING INDEX AtimeIndex (atime>? AND atime<?)
sqlite> explain query plan select url, title, strftime('%Y-%m-%d', last_atime, 'unixepoch') from CompletionHistory where (url like "%qute%" or title like "%qute%") order by last_atime desc;
0|0|0|SCAN TABLE CompletionHistory USING INDEX CompletionAtimeIndex
When loading heise.de, for some crazy reason QtWebKit calls historyContains
about 16'000 times.
With this cache (which we simply clear when *any* page has been loaded, as then
the links which have been visited can change), that's down to 250 or so...
- Show an error message when import fails, not a generic crash dialog
- Raise CommandError when debug-dump-history fails
- Check that the path exists for debug-dump-history
This is called often, hopefully a prepared query will speed it up.
This also modifies Query.run to return self for easier chaining, so you
can use `query.run.value()` instead of `query.run` ; query.value()`.
Trying to read from the sql database from another process was flaky.
This adds a debug-dump-history command which is used by the history BDD
tests to validate the history contents.
It outputs history in the old pre-SQL text format, so it might be
useful for those who want to manipulate their history as text.
Returning "next" was no longer possible as the SQL query does not fetch
more items than necessary. This is solved by using a start time, a
limit, and an offset. The offset is needed to prevent fetching duplicate
items if multiple entries have the same timestamp.
Two of the history tests that relied on qute://history were changed to
rely on qute://history/data instead to make them less failure-prone.
The history completion query is extended to pick only the most recent item for
a given url.
The tests in test_models now check for ordering of elements.
The old implementation was looping through the whole history list, which for
SQL was selecting every row in the database. The history benchmark was taking
~2s. If this is rewritten as a specialized SQL query, the benchmark takes
~10ms, an order of magnitude faster than the original non-SQL implementation.
Vulture exposed the following dead code:
- AppendLineParse was only used for reading the history text file, which is now
a sql database (and the import code for the old text file is simpler and does
not need a complex line parser)
- async_read_done is no longer used as importing the history text file is
synchronous (and should only happen once)
- config._init_key_config is unused as it was moved to keyconf.init
Turns out historyContains was getting called for the webkit backend multiple
times when the browser starts. This was calling `url in history`, which was
enumerating the entire history as `__contains__` was not defined.
Instead of skipping bad history lines during the import to sql, fail hard. We
don't want to delete the user's old history file if we couldn't parse all of
the lines.
Now that sql is only used for history (not quickmarks/bookmarks) a number of
functions are no longer needed. In addition, primary key support was removed as
we actually need to support multiple entries for the same url with different
access times. The completion model will have to handle this by selecting
something like (url, title, max(atime)).
This also fixes up a number of tests that were broken with the last few
sql-related commits.
If qutebrowser detects a history text file when it starts
(~/.local/share/qutebrowser/history by default on Linux), it will import this
file into the new sqlite database, then delete it.
The read is done as a coroutine as it can take some time.
Instead of reading sqlite history from a file and storing it in an in-memory
database, just directly use an on-disk database. This resolves#755, where
history entries don't pop in to the completion menu immediately as they are
still being read asynchronously for a few seconds after the browser starts.
Instead of add_list and add_sqltable, the completion model now supports
add_category, and callees either pass in a SqlCategory or ListCategory. This
makes unit testing much easier.
This also folds CompletionFilterModel into the ListCategory class.
Respond to the low-hanging code review fruit:
- Clean up some comments
- Remove an acidentally added duplicate init_autosave
- Combine two test_history tests
- Move test_init cleanup into a fixture to ensure it gets called.
- Name the _ argument of bind(_) to _key
- Ensure index is valid for first_item/last_item
- Move SqlException to top of module
- Rename test_index to test_getitem
- Return QItemFlags.None instead of None
- Fix copyright dates (its 2017 now!)
- Use * to force some args to be keyword-only
- Make some returns explicit
- Add sql to LOGGER_NAMES
- Add a comment to explain the sql escape statement
Instead of returning a regular tuple and trying to remember which index maps to
which field, return named tuples that allow accessing the fields by name.
From @TheCompiler:
To expand on this: I think it's fine to use KeyError on a lower level, i.e.
with the SqlTable object with a dict-like interface. However, on this higher
level, I think it makes sense to re-raise them as more specific exceptions.
Store quickmarks and bookmarks in an in-memory sql database instead of a
python dict. Long-term storage is not affected, bookmarks and
quickmarks are still persisted in a text file.
The added and deleted signals were removed, as once sql completion
models are used the models will no longer need to update themselves.
This will set the stage for SQL-based history completion.
See #1765.
The browser-wide in-memory web history is now stored in an in-memory sql
database instead of a python dict. Long-term storage is not affected, it
is still persisted in a text file of the same format.
This will set the stage for SQL-based history completion.
See #1765.
The new completion API no longer needs either of these. Instead of
referencing an enum member, cmdutils.argument.completion now points to
a function that returnsthe desired completion model.
This vastly simplifies the addition of new completion types. Previously
it was necessary to define the new model as well as editing usertypes
and completion.models.instances. Now it is only necessary to define a
single function under completion.models.
This is the next step of Completion Model/View Revamping (#74).
With per-domain settings, having a getter for a setting gets really complicated,
as there isn't one true value for a setting.
The only reason we needed those getters is to save away the default values for
some settings where we were unsure what the defaults are.
- For font setters, we can get the defaults from QFont, like QtWeb{Kit,Engine}
do.
- For font sizes, we hardcode the defaults QtWeb{Kit,Engine} hardcodes too.
- For maximum-page-in-cache, we hardcode 0, just like QtWebKit.
- For default-encoding, we hardcode iso-8559-1, like QtWeb{Kit,Engine}
- For offline-storage-default-quota, we hardcode 5MB, like QtWebKit
- For offline-web-application-cache-quota, we hardcode MAXINT as default value,
but we still keep the empty value in the config. It means "no quota"
internally in QtWebKit, but it's a too confusing value to have in the config.
- For object-cache-capacities it's a bit more complicated (the defaults are
calculated based on disk space), but let's just get rid of the setting
altogether in the next commit (see #1751).
Closes#2639.
This breaks things (with "ValueError: list.remove(x): x not in list") on
PyQt 5.9 (probably due to the destroyed object tracking it introduces?).
This was originally added in 0abb5cf738 to fix
some segfaults on exit, but things look much better with recent Qt versions.
This fixes an exception when trying to run :open-editor with a comment field
with QtWebKit:
16:37:51 DEBUG webelem webelem:is_editable:238 Checking if element is editable: <qutebrowser.browser.webkit.webkitelem.WebKitElement html='<div id="writer9997095275-writer" class="writer selectable no-lub put-art-here ui-droppable empty" style="min-height: 146px; width: 1169px;" contenteditable="true"></div>'>
16:37:51 ERROR misc crashsignal:exception_hook:205 Uncaught exception
Traceback (most recent call last):
File "/home/florian/proj/qutebrowser/git/qutebrowser/app.py", line 882, in eventFilter
return handler(event)
File "/home/florian/proj/qutebrowser/git/qutebrowser/app.py", line 842, in _handle_key_event
return man.eventFilter(event)
File "/home/florian/proj/qutebrowser/git/qutebrowser/keyinput/modeman.py", line 337, in eventFilter
return self._eventFilter_keypress(event)
File "/home/florian/proj/qutebrowser/git/qutebrowser/keyinput/modeman.py", line 168, in _eventFilter_keypress
handled = parser.handle(event)
File "/home/florian/proj/qutebrowser/git/qutebrowser/keyinput/basekeyparser.py", line 307, in handle
handled = self._handle_special_key(e)
File "/home/florian/proj/qutebrowser/git/qutebrowser/keyinput/basekeyparser.py", line 136, in _handle_special_key
self.execute(cmdstr, self.Type.special, count)
File "/home/florian/proj/qutebrowser/git/qutebrowser/keyinput/keyparser.py", line 44, in execute
self._commandrunner.run(cmdstr, count)
File "/home/florian/proj/qutebrowser/git/qutebrowser/commands/runners.py", line 275, in run
result.cmd.run(self._win_id, args, count=count)
File "/home/florian/proj/qutebrowser/git/qutebrowser/commands/command.py", line 525, in run
self.handler(*posargs, **kwargs)
File "/home/florian/proj/qutebrowser/git/qutebrowser/browser/commands.py", line 1600, in open_editor
tab.elements.find_focused(self._open_editor_cb)
File "/home/florian/proj/qutebrowser/git/qutebrowser/browser/webkit/webkittab.py", line 589, in find_focused
callback(webkitelem.WebKitElement(elem, tab=self._tab))
File "/home/florian/proj/qutebrowser/git/qutebrowser/browser/commands.py", line 1580, in _open_editor_cb
text = elem.value()
File "/home/florian/proj/qutebrowser/git/qutebrowser/browser/webkit/webkitelem.py", line 116, in value
assert isinstance(val, (int, float, str)), val
AssertionError: None
We now automatically get out of fullscreen when switching away from a
fullscreened tab. This also means we can't get into a situation where we can't
leave fullscreen anymore.
Fixes#2379.
