Trying to read from the sql database from another process was flaky.
This adds a debug-dump-history command which is used by the history BDD
tests to validate the history contents.
It outputs history in the old pre-SQL text format, so it might be
useful for those who want to manipulate their history as text.
Returning "next" was no longer possible as the SQL query does not fetch
more items than necessary. This is solved by using a start time, a
limit, and an offset. The offset is needed to prevent fetching duplicate
items if multiple entries have the same timestamp.
Two of the history tests that relied on qute://history were changed to
rely on qute://history/data instead to make them less failure-prone.
The old implementation was looping through the whole history list, which for
SQL was selecting every row in the database. The history benchmark was taking
~2s. If this is rewritten as a specialized SQL query, the benchmark takes
~10ms, an order of magnitude faster than the original non-SQL implementation.
Turns out historyContains was getting called for the webkit backend multiple
times when the browser starts. This was calling `url in history`, which was
enumerating the entire history as `__contains__` was not defined.
Instead of skipping bad history lines during the import to sql, fail hard. We
don't want to delete the user's old history file if we couldn't parse all of
the lines.
Now that sql is only used for history (not quickmarks/bookmarks) a number of
functions are no longer needed. In addition, primary key support was removed as
we actually need to support multiple entries for the same url with different
access times. The completion model will have to handle this by selecting
something like (url, title, max(atime)).
This also fixes up a number of tests that were broken with the last few
sql-related commits.
If qutebrowser detects a history text file when it starts
(~/.local/share/qutebrowser/history by default on Linux), it will import this
file into the new sqlite database, then delete it.
The read is done as a coroutine as it can take some time.
Instead of reading sqlite history from a file and storing it in an in-memory
database, just directly use an on-disk database. This resolves#755, where
history entries don't pop in to the completion menu immediately as they are
still being read asynchronously for a few seconds after the browser starts.
Respond to the low-hanging code review fruit:
- Clean up some comments
- Remove an acidentally added duplicate init_autosave
- Combine two test_history tests
- Move test_init cleanup into a fixture to ensure it gets called.
- Name the _ argument of bind(_) to _key
- Ensure index is valid for first_item/last_item
- Move SqlException to top of module
- Rename test_index to test_getitem
- Return QItemFlags.None instead of None
- Fix copyright dates (its 2017 now!)
- Use * to force some args to be keyword-only
- Make some returns explicit
- Add sql to LOGGER_NAMES
- Add a comment to explain the sql escape statement
test_history.test_init also leaked state by leaving the instantiated history as
the parent of the QApp, which was causing test_debug to fail because it was
trying to dump the history object left from test_history.
test_selectors and test_get_all_objects were running fine on my machine, but
for some reason is failing with "Driver not loaded" on Travis. Let's try
initializing SQL and see what happens.
- Adjust _check_completions to work for CompletionModel and SqlCompletionModel
- Move sql initialization into a reusable fixture
- Remove the bookmark/quickmark/history stubs, as they're now handled by sql
- Disable quickmark/bookmark model tests until their completion is ported to
sql.
- Disable urlmodel tests for features that have to be implemented in SQL:
- LIMIT (for history-max-items)
- Configurable column order (for quickmarks)
- Configurable formatting (for timestamp-format
The browser-wide in-memory web history is now stored in an in-memory sql
database instead of a python dict. Long-term storage is not affected, it
is still persisted in a text file of the same format.
This will set the stage for SQL-based history completion.
See #1765.
Apart from checking for buttons with an href attribute (which made no sense at
all and should never return any element) this was identical to
webelem.Group.links.
There's actually no good reason to filter javascript links as we might want to
click them (or copy their URL) just like any other link - this fixes#2404.
With that being gone, we don't need FILTERS at all anymore, as we can check for
existence of the href attribute in the CSS selector instead.
This benchmark was running very quickly due to an improper setup.
The current history implementation expects that a newly inserted entry must
be more recent than any existing entries and sorts according to this
assumption.
The benchmark test inserts increasingly older entries, breaking this invariant.
When run in the benchmark, the qute://history/data implementation would
see an entry older than the oldest time in the time window and would
immediately return with a single "next" entry.
This patch inserts data in an order that mantains history's invariant and adds
a sanity-check at the end of the test. It does not check for the exact length
as not all entries will be within the time window. The length will be some
values <= 100000, the check just ensures that there is at least something more
than a "next" entry.
Before:
---------------------------------------------- benchmark: 1 tests ----------------------------------------------
Name (time in us) Min Max Mean StdDev Median IQR Outliers(*) Rounds Iterations
----------------------------------------------------------------------------------------------------------------
test_qute_history_benchmark 9.3050 21.9250 9.6143 0.2454 9.5880 0.1070 230;360 9930 1
----------------------------------------------------------------------------------------------------------------
After:
-------------------------------------------------- benchmark: 1 tests -------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers(*) Rounds Iterations
-----------------------------------------------------------------------------------------------------------------------
test_qute_history_benchmark 220.7040 223.1900 221.7536 1.1070 221.1939 1.8803 1;0 5 1
-----------------------------------------------------------------------------------------------------------------------
Before, we just returned the same data for both, but then we'll run into
same-origin restrictions as qute:history and qute:history/data are not the same
host.
