This is called often, hopefully a prepared query will speed it up.
This also modifies Query.run to return self for easier chaining, so you
can use `query.run.value()` instead of `query.run` ; query.value()`.
For real this time. A mistake on the last commit like this meant models
were still spuriously instantiated.
Now that the completion model is reused, the layoutChanged signal needs
to be forwarded through, otherwise the view will not update.
This seemed to have a significant performance impact. Removing it means
that instead of just seeing the most recent atime for a given url, you
will see multiple entries.
If the completion model would stay the same, just keep it and update the
filter pattern rather than instantiating a new model each time the
pattern changes.
Instead set this on inidividual categories, as that is where it actually
gets used. This makes it easier for SqlCompletionCategory to reuse a
prepared query (as it gets the filter field names in its constructor).
Trying to read from the sql database from another process was flaky.
This adds a debug-dump-history command which is used by the history BDD
tests to validate the history contents.
It outputs history in the old pre-SQL text format, so it might be
useful for those who want to manipulate their history as text.
Returning "next" was no longer possible as the SQL query does not fetch
more items than necessary. This is solved by using a start time, a
limit, and an offset. The offset is needed to prevent fetching duplicate
items if multiple entries have the same timestamp.
Two of the history tests that relied on qute://history were changed to
rely on qute://history/data instead to make them less failure-prone.
The history completion query is extended to pick only the most recent item for
a given url.
The tests in test_models now check for ordering of elements.
The old implementation was looping through the whole history list, which for
SQL was selecting every row in the database. The history benchmark was taking
~2s. If this is rewritten as a specialized SQL query, the benchmark takes
~10ms, an order of magnitude faster than the original non-SQL implementation.
Vulture exposed the following dead code:
- AppendLineParse was only used for reading the history text file, which is now
a sql database (and the import code for the old text file is simpler and does
not need a complex line parser)
- async_read_done is no longer used as importing the history text file is
synchronous (and should only happen once)
- config._init_key_config is unused as it was moved to keyconf.init
Calling sql.init() in version.version() would replace the existing sql
connection and cause a crash when accessed by opening qute://version.
Now version relies on sql already being initted, and app.py inits sql early if
the --version arg is given.
Turns out historyContains was getting called for the webkit backend multiple
times when the browser starts. This was calling `url in history`, which was
enumerating the entire history as `__contains__` was not defined.
Instead of skipping bad history lines during the import to sql, fail hard. We
don't want to delete the user's old history file if we couldn't parse all of
the lines.
Now that sql is only used for history (not quickmarks/bookmarks) a number of
functions are no longer needed. In addition, primary key support was removed as
we actually need to support multiple entries for the same url with different
access times. The completion model will have to handle this by selecting
something like (url, title, max(atime)).
This also fixes up a number of tests that were broken with the last few
sql-related commits.
If qutebrowser detects a history text file when it starts
(~/.local/share/qutebrowser/history by default on Linux), it will import this
file into the new sqlite database, then delete it.
The read is done as a coroutine as it can take some time.
Instead of reading sqlite history from a file and storing it in an in-memory
database, just directly use an on-disk database. This resolves#755, where
history entries don't pop in to the completion menu immediately as they are
still being read asynchronously for a few seconds after the browser starts.
Deleting a history entry should do nothing, but we want a test to ensure this
and get 100% branch coverage for urlmodel.
This also un-skips the bookmark/quickmark tests.
Instead of add_list and add_sqltable, the completion model now supports
add_category, and callees either pass in a SqlCategory or ListCategory. This
makes unit testing much easier.
This also folds CompletionFilterModel into the ListCategory class.
Now all completion models are of a single type called CompletionModel.
This model combines one or more categories. A category can either be a
ListCategory or a SqlCategory.
This simplifies the API, and will allow the use of models that combine simple
list-based and sql sources. This is important for two reasons:
- Adding searchengines to url completion
- Using an on-disk sqlite database for history, while keeping bookmarks and
quickmars as text files.
This was a performance optimization that shouldn't be needed with the new SQL
history backend. This also removes support for the LIMIT feature from SqlTable
as it only existed to support web-history-max-items.
Respond to the low-hanging code review fruit:
- Clean up some comments
- Remove an acidentally added duplicate init_autosave
- Combine two test_history tests
- Move test_init cleanup into a fixture to ensure it gets called.
- Name the _ argument of bind(_) to _key
- Ensure index is valid for first_item/last_item
- Move SqlException to top of module
- Rename test_index to test_getitem
- Return QItemFlags.None instead of None
- Fix copyright dates (its 2017 now!)
- Use * to force some args to be keyword-only
- Make some returns explicit
- Add sql to LOGGER_NAMES
- Add a comment to explain the sql escape statement
newest_slice is no longer needed after the completion refactor. Now that
history is based on the SQL backend, LIMIT is used instead.
StatusBar._option is not used, though I'm not sure why vulture only caught it
now.
test_history.test_init also leaked state by leaving the instantiated history as
the parent of the QApp, which was causing test_debug to fail because it was
trying to dump the history object left from test_history.
test_selectors and test_get_all_objects were running fine on my machine, but
for some reason is failing with "Driver not loaded" on Travis. Let's try
initializing SQL and see what happens.
Instead of returning a regular tuple and trying to remember which index maps to
which field, return named tuples that allow accessing the fields by name.
Allow categories to specify a WHERE clause that applies in addition to the
pattern filter. This allows the url completion model to filter out redirect
entries.
This also fixed the usage of ESCAPE so it applies to all the LIKE statements.
A SQL completion category can now provide a customized column expression for
the select statement. This enables the url model to format timestamps, as well
as rearrange the name and url in the quickmark section.
This allows setting the query as a QSqlQuery instead of a string, which allows:
- Escaping quotes
- Using LIMIT (needed for history-max-items)
- Using ORDER BY (needed for sorting history)
- SELECTing columns (needed for quickmark completion)
- Creating a custom select (needed for history timestamp formatting)
- Adjust _check_completions to work for CompletionModel and SqlCompletionModel
- Move sql initialization into a reusable fixture
- Remove the bookmark/quickmark/history stubs, as they're now handled by sql
- Disable quickmark/bookmark model tests until their completion is ported to
sql.
- Disable urlmodel tests for features that have to be implemented in SQL:
- LIMIT (for history-max-items)
- Configurable column order (for quickmarks)
- Configurable formatting (for timestamp-format
For URL completion, time-based sorting is handled by the SQL model.
All the other models use simple alphabetical sorting. This allowed cleaning up
some logic in the sortfilter, removing DUMB_SORT, and removing the
completion.Role.sort.
This also removes the userdata completion field as it was only used in url
completion and is no longer necessary with the SQL model.
The browser-wide in-memory web history is now stored in an in-memory sql
database instead of a python dict. Long-term storage is not affected, it
is still persisted in a text file of the same format.
This will set the stage for SQL-based history completion.
See #1765.
When qutebrowser starts, it creates an in-memory sqlite database. One
can instantiate a SqlTable to create a new table in the database. The
object provides an interface to query and modify the table.
This intended to serve as the base class for the quickmark, bookmark,
and history manager objects in objreg. Instead of reading their data
into an in-memory dict, they will read into an in-memory sql table.
Eventually the completion models for history, bookmarks, and quickmarks
can be replaced with SqlQuery models for faster creation and filtering.
See #1765.
The new completion API no longer needs either of these. Instead of
referencing an enum member, cmdutils.argument.completion now points to
a function that returnsthe desired completion model.
This vastly simplifies the addition of new completion types. Previously
it was necessary to define the new model as well as editing usertypes
and completion.models.instances. Now it is only necessary to define a
single function under completion.models.
This is the next step of Completion Model/View Revamping (#74).
First step of Completion Model/View revamping (#74). Rewrite the
completion models as functions that each return an instance of a
CompletionModel class.
Caching is removed from all models except the UrlModel. Models other
than the UrlModel can be generated very quickly so caching just adds
needless complexity and can lead to incorrect results if one forgets to
wire up a signal.
The test fails on Ubuntu Xenial because QtOpenGL is not installed.
This isn't a problem with real-life usage though, as we only call it with
QtWebEngine, and that ensures that QtOpenGL is available.
Apart from checking for buttons with an href attribute (which made no sense at
all and should never return any element) this was identical to
webelem.Group.links.
There's actually no good reason to filter javascript links as we might want to
click them (or copy their URL) just like any other link - this fixes#2404.
With that being gone, we don't need FILTERS at all anymore, as we can check for
existence of the href attribute in the CSS selector instead.
With coverage 4.4, the source name (qutebrowser/) is not added to the filename
anymore. To adjust for that, we remove qutebrowser/ from all paths, and also
make sure to remove it from what coverage returns (in case someone is running an
older version).
This allows users to change the size of the favicon independently from
the size of the font/tab, in order to adjust the balance between
favicons and text. The drawing code is also adjusted to place the icon
relative to the text center, rather than the text top.
Works as expected even for values of 0.0 (which is equivalent to hiding
the favicon completely).
Closes#2549.
This benchmark was running very quickly due to an improper setup.
The current history implementation expects that a newly inserted entry must
be more recent than any existing entries and sorts according to this
assumption.
The benchmark test inserts increasingly older entries, breaking this invariant.
When run in the benchmark, the qute://history/data implementation would
see an entry older than the oldest time in the time window and would
immediately return with a single "next" entry.
This patch inserts data in an order that mantains history's invariant and adds
a sanity-check at the end of the test. It does not check for the exact length
as not all entries will be within the time window. The length will be some
values <= 100000, the check just ensures that there is at least something more
than a "next" entry.
Before:
---------------------------------------------- benchmark: 1 tests ----------------------------------------------
Name (time in us) Min Max Mean StdDev Median IQR Outliers(*) Rounds Iterations
----------------------------------------------------------------------------------------------------------------
test_qute_history_benchmark 9.3050 21.9250 9.6143 0.2454 9.5880 0.1070 230;360 9930 1
----------------------------------------------------------------------------------------------------------------
After:
-------------------------------------------------- benchmark: 1 tests -------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers(*) Rounds Iterations
-----------------------------------------------------------------------------------------------------------------------
test_qute_history_benchmark 220.7040 223.1900 221.7536 1.1070 221.1939 1.8803 1;0 5 1
-----------------------------------------------------------------------------------------------------------------------
Before, we just returned the same data for both, but then we'll run into
same-origin restrictions as qute:history and qute:history/data are not the same
host.
From the spec:
User agents should ensure, e.g. by means of an overlay, that the end user is
aware something is displayed fullscreen. User agents should provide a means of
exiting fullscreen that always works and advertise this to the user. This is
to prevent a site from spoofing the end user by recreating the user agent or
even operating system environment when fullscreen.
https://fullscreen.spec.whatwg.org/#security-and-privacy-considerations
Windows filenames have backslashes, so we need to escape them, otherwise
shlex.split will delete them.
Also, we can't prodive our own executable on frozen tests.
pytest doesn't like test classes which define __init__, and pylint
doesn't like attributes defined outside __init__.
We can disable pylint's check, but we can't force pytest to accept our
test class...
This initially seemed like a nice feature, but it means 0 can't be bound
as a separate key anymore, and 0<Esc> gives weird error messages...
Reverts #1953.
Fixes#2032.
* fix date in copyright
* remove redundant class docstrings
* don't rename utilcmds module in unit test
* use `mode_manager` fixture in place of FakeModeMan
* some whitespace
while technically possible (on both machine and OS level), termination due
to SIGSEGV cannot be prevented
maybe the test could be rewritten to spawn a subprocess and check its exit
status (of 11)
CommandRunner.parse had some logic for handling commands of form
:<count>:cmd. However, this complicated the parsing logic for something
that appears to only be used in tests. One could use it in a
userscript, but this is unlikely as it is undocumented. Removing
support for this simplifies the logic of parse.
The commnd `run-with-count` is added to provide this functionality.
It works like `repeat` but passes the count along to the command
instead of running the command multiple times.
This resolves#1997: Qutebrowser crashes when pasting commands.
This bug was caused by excess stripping of ':' from the command string
by _parse_count.
We already had some duplicated logic for completion/keyhint/messageview,
and plan to add prompt overlays too now - so here we refactor related
code to have a list of overlays instead, which are all
resized/positioned by the mainwindow when needed.
This also changes the size management, which gets moved into the
sizeHint of the respective overlay widgets.
This is needed when we want to display an error page after the user
requested a qute:// URL, as qute:// URLs can't access file:// content
with QtWebEngine.
Simplify the CompletionWidget/Completer interface by changing
on_selection_changed to pass the newly selected text rather than the
index of the newly selected item.
This moves the logic from Completer to CompletionWidget but simplifies
the interaction between the two and makes testing easier.
It was checking that every expected item was in the actual item list,
but not visa-versa. This meant that extra completion items could show
up without failing the test.
This caught one bad test case. Bind completion includes aliases, but
the test did not expect this.
The CommandRunner's fallback parsing behavior treated whitespace
differently than the normal flow. When a user entered an unknown
command, trailing whitespace would be stripped and the cmdline length
would be less than the cursor position.
This is fixed by making the fallback use the ShellLexer just as the
'normal' parsing does.
When the commandline reads ':open |', quick-completing the only offered
completion will set the commandline to ':open some_url |'. Since `open`
has `maxsplit=0`, everything after ':open' is (correctly) treated as
one argument. This means completion is opened again with 'some url '
as the pattern (note trailing whitespace), which makes the comletion
menu 'flicker' and stay open even though it was 'supposed' to quick
compelte.
This is fixed by ignoring the next completion request if we just
completed something after maxsplit (because we don't expect any more
completions after the last split).
Resolves#1519.
Remove the class variables _cursor_part and _empty_item_index. Instead,
split up the commandline around the cursor whenever that information is
needed. Using locals instead of class variables makes the logic easier
to follow and ends up requiring much less code.
Remove the dependency on the class variables _empty_item_index
and _cursor_part to make the code easier to follow. If
_update_completion is refactored in a similar way these variables can
be removed.
Turns out re.escape also escapes spaces, so we'd need to replace '(\\ )'
groups after escaping. At this point it's easier to just combine spaces
before escaping the pattern.
Fixes#1934.
Supersedes #1935.
It makes a lot of sense for this to be in webkitelem.py, but it should
not be public API as it's only used internally and can't be implemented
here with QtWebEngine.
Fixes#1911.
The bugfix is backported in my qt5-webengine-debug package, and
QUTE_QTBUG54419_PATCHED can be set to force qutebrowser to use
createWindow.
Something like:
@cmdutils.argument('foo', choices=['one', 'two'])
def func(foo):
# ...
didn't actually validate the foo argument, since the inferred type of
the argument is None, and that skipped all conversion (and thus
validation).
Fixes#1871
See #1885
This is a reworked version of 12061b8bb1
which lets special parameters (count/win_id/flags) through correctly.
Something like:
@cmdutils.argument('foo', choices=['one', 'two'])
def func(foo):
# ...
didn't actually validate the foo argument, since the inferred type of
the argument is None, and that skipped all conversion (and thus
validation).
Fixes#1871
See #1885
From the QApplication.postEvent docs:
http://doc.qt.io/qt-5/qcoreapplication.html#postEvent
The event must be allocated on the heap since the post event queue
will take ownership of the event and delete it once it has been
posted. It is not safe to access the event after it has been posted.
We can't reliably guarantee that from Python, so we need to use
sendEvent instead.
This reverts commit 4e11613d2df064b138532c18f88bbf278c64f347.
We can actually make this synchronous just fine by collecting that
information when searching for the elements...
Previously, the drawn hint labels were affected by the zoom, i.e., they
were stretched out by QtWebKit and actually had to be drawn at the
unzoomed position.
The Python/C++ API gives us coordinated adjusted for zoom, so
we always *negatively* adjusted them to get the unzoomed coordinates.
JS gave us the original coordinates, so we stretched them out according
to the zoom if adjust_zoom was given (which means only when clicking a
link).
Now we always operate in term of display coordinates: The point where we
draw the hint label is equal to the point we're clicking.
Thus, the zoom level for javascript is always adjusted, and the Python
zoom level is never (negatively) adjusted.
The following methods were only used for hint labels and thus removed
now:
- document_element
- create_inside
- find_first
- set_inner_xml
- remove_from_document
- set_style_property
If the clipboard contains "-a" then "open {clipboard}" will fail because
-a gets parsed as an option. "open -- {clipboard}" doesn't do that. See
some comments in #1791.
Implement `completion-item-focus next-category` and
`completion-item-focus prev-category` to jump through completions by
category rather than by item.
Resolves#1567.
For some reason, when e.g. visiting duckduckgo and then heise.de,
QtWebEngine suddenly gets a new QOpenGLWidget as focusProxy.
We install an extra eventFilter observing the ChildAdded event and
re-adding the MouseEventFilter when that happens.
Command completions for `:bind` and `:` will now show bindings for
aliases. The binding is only included if it is bound to that alias, not
if it is bound to the command the alias points to.
Consolidate the logic used to generate the command completion category
into one place. This is shared by CommandCompletionModel,
HelpCompletionModel, and BindCompletionModel.
Hidden commands are not shown in command completion as they typically
would not be run directly. However, a user might still might like to see
help for them if, for example, they are writing a script or creating a
binding.
Addresses #1707.
Wire up the config change event to update command completion on
changing aliases, so the new aliases will be included.
Fixes#1814.
Currently we do not have tests at a high enough level to test whether
signals are wired up correctly to update completions.
Completion.empty existed to fill a slot in the old Command.completions
interface if the first positional arg had no completions but the second
did, as is the case for the `bind` command. Now that
`Command.completions` is replaced by `Command.get_pos_arg_info`, this
is no longer needed.
Command completion types are now identified by ArgInfo, so just use
that directly and cut out the middle-man. This shouldn't change any
completion behavior.
Adds a test for get_pos_arg_info to test_cmdutils.
Modifies test_completer to test the use of get_pos_arg_info. Instead of
using FakeCommand, real Command objects are used, to validate that the
Completer works with the real Command interface. This also cleans out
some test cases that were testing things already covered by other cases.
This is a more rigorous test than filterAcceptsRow as it tests behavior
with multiple columns and different sort settings. In addition, it
tests intelligentLessThan which is not tested in the filterAcceptsRow
test (as lessThan is never called if there is only 1 item to filter).
With QtWebKit or QtWebEngine with Qt < 5.7, the functions end up in the
page's namespace. We can't easily avoid this, but at least we can name
them in a way which reduces conflicts.
webelem.javascript_escape got renamed to javascript.string_escape, and a
new javascript.assemble got added to make it easier to call a function
inside a .js file.
first_item and last_item return an invalid index when there are no
items in the completion, and the completionwidget will throw on an
invalid index. However, setting an invalid index on the selection view
is fine, so just remove the assertion.
Resolves#1731.
unix_filename_rubout deletes to the previous slash or whitespace,
unlike the previously implemented backwards-kill-word which treats and
non-alphanumeric character as a boundary.
To illustrate, given the text 'foo/bar.baz', unix_filename_rubout will
delete 'bar.baz' while backwards-kill-word will delete only 'baz'.
See #1710.
This restores the previous behavior of `unix-word-rubout` as
`backward-kill-word`, which is closer to the naming used in readline.
It is bound to <Alt-Backspace> by default, though <Ctrl-Backspace> will
also work due to a builtin binding.
Resolves#1698.
These commands are more closely tied to the CompletionView than
Completer. This removes the need for an extra signal tying the
CompletionView to the Completer.
The call to _open_completion_if_needed was moved to
on_selection_changed, as this will already be called when a new item is
selected.
Rather than having a CompletionView instantiate and register a
Completer, instantiate both in MainWindow. The CompletionView is the
parent of the Completer, and communicates by emitting
selection_changed, meaning it no longer needs to contain a reference to
the Completer.
- clean up docstring typos
- use _ to name an unused loop variable
- parent the filter model to avoid an issue with disposal
- use mocker.patch instead of monkeypatch to mock Completer creation
- use is instead of == to compare by identity
The CompletionView looks in objreg for 'status-cmd', so move it from a
private fixture in test_completer to a public fixture that handles
objreg registration/deletion.
update_completion is only used internally, so instead test the real
public entry point which is schedule_completion_update.
This required mocking out QTimer to fire immediately so the test didn't
have to do flaky artificial delays.
Using the config_tmdpir fixture across all tests in this module caused
a lingering LineParser to make test_debug fail.
I still don't know why, but scoping the config_tmpdir fixture to only
the test class that was creating ~/.config/qute_test fixes the issue,
and still prevents creation of a user tempdir.
This was more complicated than the other data/config/cachedir test
fixes, as QtWebEngine was accessing the datadir directly (and bypassing
standdarddir.data).
This means the tmpdir_data stub is not enough, we need to set
XDG_DATA_HOME to redirect access.
Don't create ~/.config/qute_test by mocking out standdarddir.config for
all tests in this module.
This adds config_tmpdir to fixtures.py and moves temp_datadir from
test_adblock to fixtures.py as it will be needed more broadly.
Running test_standarddir would pollute the user's home with
`~/.cache/qute_test`.
The `no_cachedir_tag` fixture was supposed to prevent this, but was not
working because [usefixtures does not work on fixtures]
(https://github.com/pytest-dev/pytest/issues/1014).
This fixes the fixture to actually prevent cachedir creation, but
applies it to tests individually (or by class) rather than with autouse
because the cachedir tests cannot pass if it is working.
Running the tests would create ~/.config/qute_test and
~/.local/share/qute_test on the user's machine. The test_standardir
module needed a bit more mocking to prevent it from cluttering the
user's machine.
Two tests that created the data dir were fixed by passing basedir in
args, and one test that created the config dir was fixed by patching
os.makedirs to a noop.
Per one of the diff comments on #1597:
> I used to use a tuple for constant things, but nowadays I'd actually
> prefer a list as a tuple is something more heterogeneous (i.e. it
> makes sense to have a `(x, y)` point as a tuple, but a list of points
> would be a list).
> At some point I should probably change it to a list everywhere 😉
- The invalid URL will now get encoded when using QUrl.
- The check for a None url_text is somewhat pointless as I don't think
this can ever happen in the real circumstances.
There'll be a refactoring to add a session API to WebTab later anyways,
so no point in fixing this now.
As many tests as possible here should probably also be changed to
end2end ones as there's a lot of mocking going on.
