Merge branch 'master' of https://github.com/qutebrowser/qutebrowser

doc/changelog.asciidoc

@@ -15,17 +15,40 @@ breaking changes (such as renamed commands) can happen in minor releases.
 // `Fixed` for any bug fixes.
 // `Security` to invite users to upgrade in case of vulnerabilities.
 
-v1.3.0 (unreleased)
+v1.4.0 (unreleased)
 -------------------
 
 Added
 ~~~~~
 
+- New `--debug-flag log-requests` to log requests to the debug log for
+  debugging.
+- New `--first` flag for `:hint` (bound to `gi` for inputs) which automatically
+  selects the first hint.
+
+Changed
+~~~~~~~
+
+- New short flags for commandline arguments: `-B` and `-T` for `--basedir` and
+  `--temp-basedir`; `-d` and `-D` for `--debug` and `--debug-flag`.
+- Deleting history items via `:history-clear` or `:completion-item-del` now
+  also removes that URL from QtWebEngine's visited links.
+- There's now completion for commands taking a variable count of arguments
+  (like `:config-cycle`).
+
+v1.3.0
+------
+
+Added
+~~~~~
+
 - New `:scroll-to-anchor` command to scroll to an anchor in the document.
 - New `url.open_base_url` option to open the base URL of a searchengine when no
   search term is given.
 - New `tabs.min_width` setting to configure the minimal width for tabs.
-- New `getbib` userscript to download bibtex information for DOIs on a page.
+- New userscripts:
+  * `getbib` to download bibtex information for DOIs on a page.
+  * `qute-keepass` to get passwords from KeePassX.
 
 Changed
 ~~~~~~~
@@ -52,7 +75,6 @@ Changed
 - Error messages when trying to wrap when `tabs.wrap` is `False` are now logged
   to debug instead of messages.
 
-
 Fixed
 ~~~~~
 
@@ -81,7 +103,18 @@ Fixed
 - The Makefile (intended for packagers) now supports `PREFIX` properly.
 - The workaround for a black window with Nvidia graphics is now enabled on
   non-Linux systems (like FreeBSD) as well.
-- Initial support for Qt 5.11
+- Initial support for Qt 5.11.
+- Checking for a new version after sending a crash report now works properly
+  again.
+- `@match` in Greasemonkey scripts now more closely matches the proper pattern
+  syntax.
+- Searching via `/` or `?` now doesn't handle any characters in a special way.
+- Fixed crash when trying to retry some failed downloads on QtWebEngine.
+- An invalid spellcheck dictionary filename now doesn't crash anymore.
+- When no spellcheck dictionaries are configured, it's now disabled internally.
+  This works around an issue with entering special characters on Facebook
+  messenger.
+- The macOS release now should work again on macOS 10.11 and newer.
 
 v1.2.1
 ------

doc/help/commands.asciidoc

@@ -530,7 +530,7 @@ Show help about a command or setting.
 
 [[hint]]
 === hint
-Syntax: +:hint [*--mode* 'mode'] [*--add-history*] [*--rapid*]
+Syntax: +:hint [*--mode* 'mode'] [*--add-history*] [*--rapid*] [*--first*]
      ['group'] ['target'] ['args' ['args' ...]]+
 
 Start hinting.
@@ -600,6 +600,7 @@ Start hinting.
  `tab` (with `tabs.background_tabs=true`), `tab-bg`,
  `window`, `run`, `hover`, `userscript` and `spawn`.
 
+* +*-f*+, +*--first*+: Click the first hinted element without prompting.
 
 ==== note
 * This command does not split arguments after the last argument and handles quotes literally.

doc/help/configuring.asciidoc

@@ -3,44 +3,28 @@ Configuring qutebrowser
 
 IMPORTANT: qutebrowser's configuration system was completely rewritten in
 September 2017. This information is not applicable to older releases, and older
-information elsewhere might be outdated. **If you had an old configuration
+information elsewhere might be outdated.
-around and upgraded, this page will automatically open once**. To view it at a
-later time, use the `:help` command.
 
-Migrating older configurations
+qutebrowser's config files
-------------------------------
+--------------------------
 
-qutebrowser does no automatic migration for the new configuration. However,
+qutebrowser releases before v1.0.0 had a `qutebrowser.conf` and `keys.conf`
-there's a special link:qute://configdiff/old[configdiff] page
+file. Those are not used anymore since that release - see
-(`qute://configdiff/old`) in qutebrowser, which will show you the changes you
+<<migrating,"Migrating older configurations">> for information on how to
-did in your old configuration, compared to the old defaults.
+migrate to the new config.
 
-Other changes in default settings:
+When using `:set` and `:bind`, changes are saved to an `autoconfig.yml` file
+automatically. If you don't want to have a config file which is curated by
+hand, you can simply use those - see
+<<autoconfig,"Configuring qutebrowser via the user interface">> for details.
 
-- In v1.1.x and newer, `<Up>` and `<Down>` navigate through command history
+For more advanced configuration, you can write a `config.py` file - see
-  if no text was entered yet.
+<<configpy,"Configuring qutebrowser via config.py">>. As soon as a `config.py`
-  With v1.0.x, they always navigate through command history instead of selecting
+exists, the `autoconfig.yml` file **is not read anymore** by default. You need
-  completion items. Use `<Tab>`/`<Shift-Tab>` to cycle through the completion
+to <<configpy-autoconfig,load it by hand>> if you want settings done via
-  instead.
+`:set`/`:bind` to still persist.
-  You can get back the old behavior by doing:
-+
-----
-:bind -m command <Up> completion-item-focus prev
-:bind -m command <Down> completion-item-focus next
-----
-+
-or always navigate through command history with
-+
-----
-:bind -m command <Up> command-history-prev
-:bind -m command <Down> command-history-next
-----
-
-- The default for `completion.web_history_max_items` is now set to `-1`, showing
-  an unlimited number of items in the completion for `:open` as the new
-  sqlite-based completion is much faster. If the `:open` completion is too slow
-  on your machine, set an appropriate limit again.
 
+[[autoconfig]]
 Configuring qutebrowser via the user interface
 ----------------------------------------------
 
@@ -88,6 +72,7 @@ link:commands.html#config-clear[`:config-clear`] to reset the entire configurati
 and link:commands.html#config-cycle[`:config-cycle`] to cycle a setting between
 different values.
 
+[[configpy]]
 Configuring qutebrowser via config.py
 -------------------------------------
 
@@ -239,6 +224,7 @@ config.bind(',v', 'spawn mpv {url}')
 To suppress loading of any default keybindings, you can set
 `c.bindings.default = {}`.
 
+[[configpy-autoconfig]]
 Loading `autoconfig.yml`
 ~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -429,3 +415,38 @@ from qutebrowser.config.config import ConfigContainer  # noqa: F401
 config = config  # type: ConfigAPI # noqa: F821 pylint: disable=E0602,C0103
 c = c  # type: ConfigContainer # noqa: F821 pylint: disable=E0602,C0103
 ----
+
+[[migrating]]
+Migrating older configurations
+------------------------------
+
+qutebrowser does no automatic migration for the new configuration. However,
+there's a special link:qute://configdiff/old[configdiff] page
+(`qute://configdiff/old`) in qutebrowser, which will show you the changes you
+did in your old configuration, compared to the old defaults.
+
+Other changes in default settings:
+
+- In v1.1.x and newer, `<Up>` and `<Down>` navigate through command history
+  if no text was entered yet.
+  With v1.0.x, they always navigate through command history instead of selecting
+  completion items. Use `<Tab>`/`<Shift-Tab>` to cycle through the completion
+  instead.
+  You can get back the old behavior by doing:
++
+----
+:bind -m command <Up> completion-item-focus prev
+:bind -m command <Down> completion-item-focus next
+----
++
+or always navigate through command history with
++
+----
+:bind -m command <Up> command-history-prev
+:bind -m command <Down> command-history-next
+----
+
+- The default for `completion.web_history_max_items` is now set to `-1`, showing
+  an unlimited number of items in the completion for `:open` as the new
+  sqlite-based completion is much faster. If the `:open` completion is too slow
+  on your machine, set an appropriate limit again.

doc/help/settings.asciidoc

@@ -562,6 +562,7 @@ Default:
 * +pass:[gd]+: +pass:[download]+
 * +pass:[gf]+: +pass:[view-source]+
 * +pass:[gg]+: +pass:[scroll-to-perc 0]+
+* +pass:[gi]+: +pass:[hint inputs --first]+
 * +pass:[gl]+: +pass:[tab-move -]+
 * +pass:[gm]+: +pass:[tab-move]+
 * +pass:[go]+: +pass:[set-cmd-text :open {url:pretty}]+

doc/qutebrowser.1.asciidoc

@@ -38,7 +38,7 @@ show it.
 *-h*, *--help*::
     show this help message and exit
 
-*--basedir* 'BASEDIR'::
+*-B* 'BASEDIR', *--basedir* 'BASEDIR'::
     Base directory for all storage.
 
 *-V*, *--version*::
@@ -72,7 +72,7 @@ show it.
 *--loglines* 'LOGLINES'::
     How many lines of the debug log to keep in RAM (-1: unlimited).
 
-*--debug*::
+*-d*, *--debug*::
     Turn on debugging options.
 
 *--json-logging*::
@@ -87,7 +87,7 @@ show it.
 *--nowindow*::
     Don't show the main window.
 
-*--temp-basedir*::
+*-T*, *--temp-basedir*::
     Use a temporary basedir.
 
 *--no-err-windows*::
@@ -99,7 +99,7 @@ show it.
 *--qt-flag* 'QT_FLAG'::
     Pass an argument to Qt as flag.
 
-*--debug-flag* 'DEBUG_FLAGS'::
+*-D* 'DEBUG_FLAGS', *--debug-flag* 'DEBUG_FLAGS'::
     Pass name of debugging feature to be turned on.
 // QUTE_OPTIONS_END
 

misc/requirements/requirements-flake8.txt

@@ -1,9 +1,9 @@
 # This file is automatically generated by scripts/dev/recompile_requirements.py
 
-attrs==17.4.0
+attrs==18.1.0
 flake8==3.5.0
 flake8-bugbear==18.2.0
-flake8-builtins==1.3.0
+flake8-builtins==1.3.1  # rq.filter: != 1.4.0
 flake8-comprehensions==1.4.1
 flake8-copyright==0.2.0
 flake8-debugger==3.1.0
@@ -18,7 +18,7 @@ flake8-tidy-imports==1.1.0
 flake8-tuple==0.2.13
 mccabe==0.6.1
 pathmatch==0.2.1
-pep8-naming==0.5.0
+pep8-naming==0.6.1
 pycodestyle==2.3.1  # rq.filter: < 2.4.0
 pydocstyle==2.1.1
 pyflakes==1.6.0

misc/requirements/requirements-flake8.txt-raw

@@ -1,6 +1,6 @@
 flake8
 flake8-bugbear
-flake8-builtins
+flake8-builtins!=1.4.0
 flake8-comprehensions
 flake8-copyright
 flake8-debugger
@@ -18,3 +18,6 @@ pyflakes
 
 # https://github.com/PyCQA/pycodestyle/issues/741
 #@ filter: pycodestyle < 2.4.0
+
+# https://github.com/gforcada/flake8-builtins/issues/36
+#@ filter: flake8-builtins != 1.4.0

misc/requirements/requirements-pip.txt

@@ -3,6 +3,6 @@
 appdirs==1.4.3
 packaging==17.1
 pyparsing==2.2.0
-setuptools==39.0.1
+setuptools==39.1.0
 six==1.11.0
 wheel==0.31.0

misc/requirements/requirements-tests.txt

@@ -1,6 +1,6 @@
 # This file is automatically generated by scripts/dev/recompile_requirements.py
 
-attrs==17.4.0
+attrs==18.1.0
 beautifulsoup4==4.6.0
 cheroot==6.2.4
 click==6.7
@@ -8,7 +8,7 @@ click==6.7
 coverage==4.5.1
 EasyProcess==0.2.3
 fields==5.0.0
-Flask==0.12.2
+Flask==1.0.2
 glob2==0.6
 hunter==2.0.2
 hypothesis==3.56.5
@@ -22,13 +22,13 @@ parse-type==0.4.2
 pluggy==0.6.0
 py==1.5.3
 py-cpuinfo==4.0.0
-pytest==3.5.0
+pytest==3.5.1
 pytest-bdd==2.21.0
 pytest-benchmark==3.1.1
 pytest-cov==2.5.1
 pytest-faulthandler==1.5.0
 pytest-instafail==0.3.0
-pytest-mock==1.9.0
+pytest-mock==1.10.0
 pytest-qt==2.3.1
 pytest-repeat==0.4.1
 pytest-rerunfailures==4.0

misc/userscripts/qute-keepass

@@ -0,0 +1,261 @@
+#!/usr/bin/env python3
+
+# Copyright 2018 Jay Kamat <jaygkamat@gmail.com>
+#
+# This file is part of qutebrowser.
+#
+# qutebrowser is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# qutebrowser is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with qutebrowser.  If not, see <http://www.gnu.org/licenses/>.
+
+"""This userscript allows for insertion of usernames and passwords from keepass
+databases using pykeepass. Since it is a userscript, it must be run from
+qutebrowser.
+
+A sample invocation of this script is:
+
+:spawn --userscript qute-keepass -p ~/KeePassFiles/MainDatabase.kdbx
+
+And a sample binding
+
+:bind --mode=insert <ctrl-i> spawn --userscript qute-keepass -p ~/KeePassFiles/MainDatabase.kdbx
+
+-p or --path is a required argument.
+
+--keyfile-path allows you to specify a keepass keyfile. If you only use a
+keyfile, also add --no-password as well. Specifying --no-password without
+--keyfile-path will lead to an error.
+
+login information is inserted using :insert-text and :fake-key <Tab>, which
+means you must have a cursor in position before initiating this userscript. If
+you do not do this, you will get 'element not editable' errors.
+
+If keepass takes a while to open the DB, you might want to consider reducing
+the number of transform rounds in your database settings.
+
+Dependencies: pykeepass (in python3), PyQt5. Without pykeepass, you will get an
+exit code of 100.
+
+********************!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!******************
+
+WARNING: The login details are viewable as plaintext in qutebrowser's debug log
+(qute://log) and could be compromised if you decide to submit a crash report!
+
+********************!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!******************
+
+"""
+
+# pylint: disable=bad-builtin
+
+import argparse
+import enum
+import functools
+import os
+import shlex
+import subprocess
+import sys
+
+from PyQt5.QtCore import QUrl
+from PyQt5.QtWidgets import QApplication, QInputDialog, QLineEdit
+
+try:
+    import pykeepass
+except ImportError as e:
+    print("pykeepass not found: {}".format(str(e)), file=sys.stderr)
+
+    # Since this is a common error, try to print it to the FIFO if we can.
+    if 'QUTE_FIFO' in os.environ:
+        with open(os.environ['QUTE_FIFO'], 'w') as fifo:
+            fifo.write('message-error "pykeepass failed to be imported."\n')
+            fifo.flush()
+    sys.exit(100)
+
+argument_parser = argparse.ArgumentParser(
+    description="Fill passwords using keepass.",
+    formatter_class=argparse.RawDescriptionHelpFormatter,
+    epilog=__doc__)
+argument_parser.add_argument('url', nargs='?', default=os.getenv('QUTE_URL'))
+argument_parser.add_argument('--path', '-p', required=True,
+                             help='Path to the keepass db.')
+argument_parser.add_argument('--keyfile-path', '-k', default=None,
+                             help='Path to a keepass keyfile')
+argument_parser.add_argument(
+    '--no-password', action='store_true',
+    help='Supply if no password is required to unlock this database. '
+    'Only allowed with --keyfile-path')
+argument_parser.add_argument(
+    '--dmenu-invocation', '-d', default='dmenu',
+    help='Invocation used to execute a dmenu-provider')
+argument_parser.add_argument(
+    '--dmenu-format', '-f', default='{title}: {username}',
+    help='Format string for keys to display in dmenu.'
+    ' Must generate a unique string.')
+argument_parser.add_argument(
+    '--no-insert-mode', '-n', dest='insert_mode', action='store_false',
+    help="Don't automatically enter insert mode")
+argument_parser.add_argument(
+    '--io-encoding', '-i', default='UTF-8',
+    help='Encoding used to communicate with subprocesses')
+group = argument_parser.add_mutually_exclusive_group()
+group.add_argument('--username-fill-only', '-e',
+                   action='store_true', help='Only insert username')
+group.add_argument('--password-fill-only', '-w',
+                   action='store_true', help='Only insert password')
+
+CMD_DELAY = 50
+
+
+class ExitCodes(enum.IntEnum):
+    """Stores various exit codes groups to use."""
+    SUCCESS = 0
+    FAILURE = 1
+    # 1 is automatically used if Python throws an exception
+    NO_CANDIDATES = 2
+    USER_QUIT = 3
+    DB_OPEN_FAIL = 4
+
+    INTERNAL_ERROR = 10
+
+
+def qute_command(command):
+    with open(os.environ['QUTE_FIFO'], 'w') as fifo:
+        fifo.write(command + '\n')
+        fifo.flush()
+
+
+def stderr(to_print):
+    """Extra functionality to echo out errors to qb ui."""
+    print(to_print, file=sys.stderr)
+    qute_command('message-error "{}"'.format(to_print))
+
+
+def dmenu(items, invocation, encoding):
+    """Runs dmenu with given arguments."""
+    command = shlex.split(invocation)
+    process = subprocess.run(command, input='\n'.join(items).encode(encoding),
+                             stdout=subprocess.PIPE)
+    return process.stdout.decode(encoding).strip()
+
+
+def get_password():
+    """Get a keepass db password from user."""
+    _app = QApplication(sys.argv)
+    text, ok = QInputDialog.getText(
+        None, "KeePass DB Password",
+        "Please enter your KeePass Master Password",
+        QLineEdit.Password)
+    if not ok:
+        stderr('Password Prompt Rejected.')
+        sys.exit(ExitCodes.USER_QUIT)
+    return text
+
+
+def find_candidates(args, host):
+    """Finds candidates that match host"""
+    file_path = os.path.expanduser(args.path)
+
+    # TODO find a way to keep the db open, so we don't open (and query
+    # password) it every time
+
+    pw = None
+    if not args.no_password:
+        pw = get_password()
+
+    kf = args.keyfile_path
+    if kf:
+        kf = os.path.expanduser(kf)
+
+    try:
+        kp = pykeepass.PyKeePass(file_path, password=pw, keyfile=kf)
+    except Exception as e:
+        stderr("There was an error opening the DB: {}".format(str(e)))
+
+    return kp.find_entries(url="{}{}{}".format(".*", host, ".*"), regex=True)
+
+
+def candidate_to_str(args, candidate):
+    """Turns candidate into a human readable string for dmenu"""
+    return args.dmenu_format.format(title=candidate.title,
+                                    url=candidate.url,
+                                    username=candidate.username,
+                                    path=candidate.path,
+                                    uuid=candidate.uuid)
+
+
+def candidate_to_secret(candidate):
+    """Turns candidate into a generic (user, password) tuple"""
+    return (candidate.username, candidate.password)
+
+
+def run(args):
+    """Runs qute-keepass"""
+    if not args.url:
+        argument_parser.print_help()
+        return ExitCodes.FAILURE
+
+    url_host = QUrl(args.url).host()
+
+    if not url_host:
+        stderr('{} was not parsed as a valid URL!'.format(args.url))
+        return ExitCodes.INTERNAL_ERROR
+
+    # Find candidates matching the host of the given URL
+    candidates = find_candidates(args, url_host)
+    if not candidates:
+        stderr('No candidates for URL {!r} found!'.format(args.url))
+        return ExitCodes.NO_CANDIDATES
+
+    # Create a map so we can get turn the resulting string from dmenu back into
+    # a candidate
+    candidates_strs = list(map(functools.partial(candidate_to_str, args),
+                               candidates))
+    candidates_map = dict(zip(candidates_strs, candidates))
+
+    if len(candidates) == 1:
+        selection = candidates.pop()
+    else:
+        selection = dmenu(candidates_strs,
+                          args.dmenu_invocation,
+                          args.io_encoding)
+
+        if selection not in candidates_map:
+            stderr("'{}' was not a valid entry!").format(selection)
+            return ExitCodes.USER_QUIT
+
+        selection = candidates_map[selection]
+
+    username, password = candidate_to_secret(selection)
+
+    insert_mode = ';; enter-mode insert' if args.insert_mode else ''
+    if args.username_fill_only:
+        qute_command('insert-text {}{}'.format(username, insert_mode))
+    elif args.password_fill_only:
+        qute_command('insert-text {}{}'.format(password, insert_mode))
+    else:
+        # Enter username and password using insert-key and fake-key <Tab>
+        # (which supports more passwords than fake-key only), then switch back
+        # into insert-mode, so the form can be directly submitted by hitting
+        # enter afterwards. It dosen't matter when we go into insert mode, but
+        # the other commands need to be be executed sequentially, so we add
+        # delays with later.
+        qute_command('insert-text {} ;;'
+                     'later {} fake-key <Tab> ;;'
+                     'later {} insert-text {}{}'
+                     .format(username, CMD_DELAY,
+                             CMD_DELAY * 2, password, insert_mode))
+
+    return ExitCodes.SUCCESS
+
+
+if __name__ == '__main__':
+    arguments = argument_parser.parse_args()
+    sys.exit(run(arguments))

misc/userscripts/qute-pass

@@ -109,6 +109,13 @@ def dmenu(items, invocation, encoding):
     return process.stdout.decode(encoding).strip()
 
 
+def fake_key_raw(text):
+    for character in text:
+        # Escape all characters by default, space requires special handling
+        sequence = '" "' if character == ' ' else '\{}'.format(character)
+        qute_command('fake-key {}'.format(sequence))
+
+
 def main(arguments):
     if not arguments.url:
         argument_parser.print_help()
@@ -158,15 +165,19 @@ def main(arguments):
         return ExitCodes.COULD_NOT_MATCH_PASSWORD
     password = match.group(1)
 
-    insert_mode = ';; enter-mode insert' if arguments.insert_mode else ''
     if arguments.username_only:
-        qute_command('fake-key {}{}'.format(username, insert_mode))
+        fake_key_raw(username)
     elif arguments.password_only:
-        qute_command('fake-key {}{}'.format(password, insert_mode))
+        fake_key_raw(password)
     else:
         # Enter username and password using fake-key and <Tab> (which seems to work almost universally), then switch
         # back into insert-mode, so the form can be directly submitted by hitting enter afterwards
-        qute_command('fake-key {} ;; fake-key <Tab> ;; fake-key {}{}'.format(username, password, insert_mode))
+        fake_key_raw(username)
+        qute_command('fake-key <Tab>')
+        fake_key_raw(password)
+
+    if arguments.insert_mode:
+        qute_command('enter-mode insert')
 
     return ExitCodes.SUCCESS
 

qutebrowser/__init__.py

@@ -26,7 +26,7 @@ __copyright__ = "Copyright 2014-2018 Florian Bruhin (The Compiler)"
 __license__ = "GPL"
 __maintainer__ = __author__
 __email__ = "mail@qutebrowser.org"
-__version_info__ = (1, 2, 1)
+__version_info__ = (1, 3, 0)
 __version__ = '.'.join(str(e) for e in __version_info__)
 __description__ = "A keyboard-driven, vim-like browser based on PyQt5."
 

qutebrowser/browser/browsertab.py

@@ -724,7 +724,13 @@ class AbstractTab(QWidget):
         if getattr(evt, 'posted', False):
             raise utils.Unreachable("Can't re-use an event which was already "
                                     "posted!")
+
         recipient = self.event_target()
+        if recipient is None:
+            # https://github.com/qutebrowser/qutebrowser/issues/3888
+            log.webview.warning("Unable to find event target!")
+            return
+
         evt.posted = True
         QApplication.postEvent(recipient, evt)
 

qutebrowser/browser/greasemonkey.py

@@ -31,7 +31,7 @@ import attr
 from PyQt5.QtCore import pyqtSignal, QObject, QUrl
 
 from qutebrowser.utils import (log, standarddir, jinja, objreg, utils,
-                               javascript)
+                               javascript, urlmatch)
 from qutebrowser.commands import cmdutils
 from qutebrowser.browser import downloads
 
@@ -48,6 +48,7 @@ class GreasemonkeyScript:
     def __init__(self, properties, code):
         self._code = code
         self.includes = []
+        self.matches = []
         self.excludes = []
         self.requires = []
         self.description = None
@@ -63,8 +64,10 @@ class GreasemonkeyScript:
                 self.namespace = value
             elif name == 'description':
                 self.description = value
-            elif name in ['include', 'match']:
+            elif name == 'include':
                 self.includes.append(value)
+            elif name == 'match':
+                self.matches.append(value)
             elif name in ['exclude', 'exclude_match']:
                 self.excludes.append(value)
             elif name == 'run-at':
@@ -92,7 +95,7 @@ class GreasemonkeyScript:
             props = ""
         script = cls(re.findall(cls.PROPS_REGEX, props), source)
         script.script_meta = props
-        if not script.includes:
+        if not script.includes and not script.matches:
             script.includes = ['*']
         return script
 
@@ -117,7 +120,7 @@ class GreasemonkeyScript:
         return json.dumps({
             'name': self.name,
             'description': self.description,
-            'matches': self.includes,
+            'matches': self.matches,
             'includes': self.includes,
             'excludes': self.excludes,
             'run-at': self.run_at,
@@ -143,6 +146,42 @@ class MatchingScripts(object):
     idle = attr.ib(default=attr.Factory(list))
 
 
+class GreasemonkeyMatcher:
+
+    """Check whether scripts should be loaded for a given URL."""
+
+    # https://wiki.greasespot.net/Include_and_exclude_rules#Greaseable_schemes
+    # Limit the schemes scripts can run on due to unreasonable levels of
+    # exploitability
+    GREASEABLE_SCHEMES = ['http', 'https', 'ftp', 'file']
+
+    def __init__(self, url):
+        self._url = url
+        self._url_string = url.toString(QUrl.FullyEncoded)
+        self.is_greaseable = url.scheme() in self.GREASEABLE_SCHEMES
+
+    def _match_pattern(self, pattern):
+        # For include and exclude rules if they start and end with '/' they
+        # should be treated as a (ecma syntax) regular expression.
+        if pattern.startswith('/') and pattern.endswith('/'):
+            matches = re.search(pattern[1:-1], self._url_string, flags=re.I)
+            return matches is not None
+
+        # Otherwise they are glob expressions.
+        return fnmatch.fnmatch(self._url_string, pattern)
+
+    def matches(self, script):
+        """Check whether the URL matches filtering rules of the script."""
+        assert self.is_greaseable
+        matching_includes = any(self._match_pattern(pat)
+                                for pat in script.includes)
+        matching_match = any(urlmatch.UrlPattern(pat).matches(self._url)
+                             for pat in script.matches)
+        matching_excludes = any(self._match_pattern(pat)
+                                for pat in script.excludes)
+        return (matching_includes or matching_match) and not matching_excludes
+
+
 class GreasemonkeyManager(QObject):
 
     """Manager of userscripts and a Greasemonkey compatible environment.
@@ -154,10 +193,6 @@ class GreasemonkeyManager(QObject):
     """
 
     scripts_reloaded = pyqtSignal()
-    # https://wiki.greasespot.net/Include_and_exclude_rules#Greaseable_schemes
-    # Limit the schemes scripts can run on due to unreasonable levels of
-    # exploitability
-    greaseable_schemes = ['http', 'https', 'ftp', 'file']
 
     def __init__(self, parent=None):
         super().__init__(parent)
@@ -309,30 +344,17 @@ class GreasemonkeyManager(QObject):
         returns a tuple of lists of scripts meant to run at (document-start,
         document-end, document-idle)
         """
-        if url.scheme() not in self.greaseable_schemes:
+        matcher = GreasemonkeyMatcher(url)
+        if not matcher.is_greaseable:
             return MatchingScripts(url, [], [], [])
-
-        string_url = url.toString(QUrl.FullyEncoded)
-
-        def _match(pattern):
-            # For include and exclude rules if they start and end with '/' they
-            # should be treated as a (ecma syntax) regular expression.
-            if pattern.startswith('/') and pattern.endswith('/'):
-                matches = re.search(pattern[1:-1], string_url, flags=re.I)
-                return matches is not None
-
-            # Otherwise they are glob expressions.
-            return fnmatch.fnmatch(string_url, pattern)
-
-        tester = (lambda script:
-                  any(_match(pat) for pat in script.includes) and
-                  not any(_match(pat) for pat in script.excludes))
-
         return MatchingScripts(
-            url,
+            url=url,
-            [script for script in self._run_start if tester(script)],
+            start=[script for script in self._run_start
-            [script for script in self._run_end if tester(script)],
+                   if matcher.matches(script)],
-            [script for script in self._run_idle if tester(script)]
+            end=[script for script in self._run_end
+                 if matcher.matches(script)],
+            idle=[script for script in self._run_idle
+                  if matcher.matches(script)]
         )
 
     def all_scripts(self):

qutebrowser/browser/hints.py

@@ -172,6 +172,7 @@ class HintContext:
     tab = attr.ib(None)
     group = attr.ib(None)
     hint_mode = attr.ib(None)
+    first = attr.ib(False)
 
     def get_args(self, urlstr):
         """Get the arguments, with {hint-url} replaced by the given URL."""
@@ -612,6 +613,9 @@ class HintManager(QObject):
         modeman.enter(self._win_id, usertypes.KeyMode.hint,
                       'HintManager.start')
 
+        if self._context.first:
+            self._fire(strings[0])
+            return
         # to make auto_follow == 'always' work
         self._handle_auto_follow()
 
@@ -620,7 +624,8 @@ class HintManager(QObject):
     @cmdutils.argument('win_id', win_id=True)
     def start(self,  # pylint: disable=keyword-arg-before-vararg
               group=webelem.Group.all, target=Target.normal,
-              *args, win_id, mode=None, add_history=False, rapid=False):
+              *args, win_id, mode=None, add_history=False, rapid=False,
+              first=False):
         """Start hinting.
 
         Args:
@@ -631,6 +636,7 @@ class HintManager(QObject):
                    `window`, `run`, `hover`, `userscript` and `spawn`.
             add_history: Whether to add the spawned or yanked link to the
                          browsing history.
+            first: Click the first hinted element without prompting.
             group: The element types to hint.
 
                 - `all`: All clickable elements.
@@ -713,6 +719,7 @@ class HintManager(QObject):
         self._context.rapid = rapid
         self._context.hint_mode = mode
         self._context.add_history = add_history
+        self._context.first = first
         try:
             self._context.baseurl = tabbed_browser.current_url()
         except qtutils.QtValueError:

qutebrowser/browser/history.py

@@ -23,7 +23,7 @@ import os
 import time
 import contextlib
 
-from PyQt5.QtCore import pyqtSlot, QUrl, QTimer
+from PyQt5.QtCore import pyqtSlot, QUrl, QTimer, pyqtSignal
 
 from qutebrowser.commands import cmdutils, cmdexc
 from qutebrowser.utils import (utils, objreg, log, usertypes, message,
@@ -52,6 +52,11 @@ class WebHistory(sql.SqlTable):
 
     """The global history of visited pages."""
 
+    # All web history cleared
+    history_cleared = pyqtSignal()
+    # one url cleared
+    url_cleared = pyqtSignal(QUrl)
+
     def __init__(self, parent=None):
         super().__init__("History", ['url', 'title', 'atime', 'redirect'],
                          constraints={'url': 'NOT NULL',
@@ -157,6 +162,7 @@ class WebHistory(sql.SqlTable):
         with self._handle_sql_errors():
             self.delete_all()
             self.completion.delete_all()
+        self.history_cleared.emit()
 
     def delete_url(self, url):
         """Remove all history entries with the given url.
@@ -168,6 +174,7 @@ class WebHistory(sql.SqlTable):
         qtutils.ensure_valid(qurl)
         self.delete('url', self._format_url(qurl))
         self.completion.delete('url', self._format_completion_url(qurl))
+        self.url_cleared.emit(qurl)
 
     @pyqtSlot(QUrl, QUrl, str)
     def add_from_tab(self, url, requested_url, title):

qutebrowser/browser/webengine/interceptor.py

@@ -19,20 +19,22 @@
 
 """A request interceptor taking care of adblocking and custom headers."""
 
-from PyQt5.QtWebEngineCore import QWebEngineUrlRequestInterceptor
+from PyQt5.QtWebEngineCore import (QWebEngineUrlRequestInterceptor,
+                                   QWebEngineUrlRequestInfo)
 
 from qutebrowser.config import config
 from qutebrowser.browser import shared
-from qutebrowser.utils import utils, log
+from qutebrowser.utils import utils, log, debug
 
 
 class RequestInterceptor(QWebEngineUrlRequestInterceptor):
 
     """Handle ad blocking and custom headers."""
 
-    def __init__(self, host_blocker, parent=None):
+    def __init__(self, host_blocker, args, parent=None):
         super().__init__(parent)
         self._host_blocker = host_blocker
+        self._args = args
 
     def install(self, profile):
         """Install the interceptor on the given QWebEngineProfile."""
@@ -54,6 +56,18 @@ class RequestInterceptor(QWebEngineUrlRequestInterceptor):
         Args:
             info: QWebEngineUrlRequestInfo &info
         """
+        if 'log-requests' in self._args.debug_flags:
+            resource_type = debug.qenum_key(QWebEngineUrlRequestInfo,
+                                            info.resourceType())
+            navigation_type = debug.qenum_key(QWebEngineUrlRequestInfo,
+                                              info.navigationType())
+            log.webview.debug("{} {}, first-party {}, resource {}, "
+                              "navigation {}".format(
+                                  bytes(info.requestMethod()).decode('ascii'),
+                                  info.requestUrl().toDisplayString(),
+                                  info.firstPartyUrl().toDisplayString(),
+                                  resource_type, navigation_type))
+
         # FIXME:qtwebengine only block ads for NavigationTypeOther?
         if self._host_blocker.is_blocked(info.requestUrl()):
             log.webview.info("Request to {} blocked by host blocker.".format(

qutebrowser/browser/webengine/spell.py

@@ -24,16 +24,18 @@ import os
 import re
 
 from PyQt5.QtCore import QLibraryInfo
-from qutebrowser.utils import log
+from qutebrowser.utils import log, message
+
+dict_version_re = re.compile(r".+-(?P<version>[0-9]+-[0-9]+?)\.bdic")
 
 
 def version(filename):
     """Extract the version number from the dictionary file name."""
-    version_re = re.compile(r".+-(?P<version>[0-9]+-[0-9]+?)\.bdic")
+    match = dict_version_re.match(filename)
-    match = version_re.fullmatch(filename)
     if match is None:
-        raise ValueError('the given dictionary file name is malformed: {}'
+        message.warning(
-                         .format(filename))
+            "Found a dictionary with a malformed name: {}".format(filename))
+        return None
     return tuple(int(n) for n in match.group('version').split('-'))
 
 
@@ -44,15 +46,23 @@ def dictionary_dir():
 
 
 def local_files(code):
-    """Return all installed dictionaries for the given code."""
+    """Return all installed dictionaries for the given code.
+
+    The returned dictionaries are sorted by version, therefore the latest will
+    be the first element. The list will be empty if no dictionaries are found.
+    """
     pathname = os.path.join(dictionary_dir(), '{}*.bdic'.format(code))
     matching_dicts = glob.glob(pathname)
-    files = []
+    versioned_dicts = []
-    for matching_dict in sorted(matching_dicts, key=version, reverse=True):
+    for matching_dict in matching_dicts:
-        filename = os.path.basename(matching_dict)
+        parsed_version = version(matching_dict)
-        log.config.debug('Found file for dict {}: {}'.format(code, filename))
+        if parsed_version is not None:
-        files.append(filename)
+            filename = os.path.basename(matching_dict)
-    return files
+            log.config.debug('Found file for dict {}: {}'
+                             .format(code, filename))
+            versioned_dicts.append((parsed_version, filename))
+    return [filename for version, filename
+            in sorted(versioned_dicts, reverse=True)]
 
 
 def local_filename(code):

qutebrowser/browser/webengine/webenginedownloads.py

@@ -101,7 +101,11 @@ class DownloadItem(downloads.AbstractDownloadItem):
 
     def retry(self):
         state = self._qt_item.state()
-        assert state == QWebEngineDownloadItem.DownloadInterrupted, state
+        if state != QWebEngineDownloadItem.DownloadInterrupted:
+            log.downloads.warning(
+                "Trying to retry download in state {}".format(
+                    debug.qenum_key(QWebEngineDownloadItem, state)))
+            return
 
         try:
             self._qt_item.resume()

qutebrowser/browser/webengine/webenginesettings.py

@@ -176,24 +176,11 @@ class ProfileSetter:
         """Initialize settings on the given profile."""
         self.set_http_headers()
         self.set_http_cache_size()
-        self._init_attributes()
+        self._profile.settings().setAttribute(
-
+            QWebEngineSettings.FullScreenSupportEnabled, True)
         if qtutils.version_check('5.8'):
-            self._profile.setSpellCheckEnabled(True)
             self.set_dictionary_language()
 
-    def _init_attributes(self):
-        """Initialize hard-coded attributes."""
-        values = {
-            'FullScreenSupportEnabled': True,
-            'FocusOnNavigationEnabled': True,
-        }
-        settings = self._profile.settings()
-        for name, value in values.items():
-            attr = getattr(QWebEngineSettings, name, None)
-            if attr is not None:
-                settings.setAttribute(attr, value)
-
     def set_http_headers(self):
         """Set the user agent and accept-language for the given profile.
 
@@ -242,6 +229,7 @@ class ProfileSetter:
 
         log.config.debug("Found dicts: {}".format(filenames))
         self._profile.setSpellCheckLanguages(filenames)
+        self._profile.setSpellCheckEnabled(bool(filenames))
 
 
 def _update_settings(option):

qutebrowser/browser/webengine/webenginetab.py

@@ -62,8 +62,9 @@ def init():
 
     log.init.debug("Initializing request interceptor...")
     host_blocker = objreg.get('host-blocker')
+    args = objreg.get('args')
     req_interceptor = interceptor.RequestInterceptor(
-        host_blocker, parent=app)
+        host_blocker, args=args, parent=app)
     req_interceptor.install(webenginesettings.default_profile)
     req_interceptor.install(webenginesettings.private_profile)
 
@@ -73,6 +74,14 @@ def init():
     download_manager.install(webenginesettings.private_profile)
     objreg.register('webengine-download-manager', download_manager)
 
+    # Clear visited links on web history clear
+    hist = objreg.get('web-history')
+    for p in [webenginesettings.default_profile,
+              webenginesettings.private_profile]:
+        hist.history_cleared.connect(p.clearAllVisitedLinks)
+        hist.url_cleared.connect(lambda url, profile=p:
+                                 profile.clearVisitedLinks([url]))
+
 
 # Mapping worlds from usertypes.JsWorld to QWebEngineScript world IDs.
 _JS_WORLD_MAP = {
@@ -782,6 +791,8 @@ class WebEngineTab(browsertab.AbstractTab):
             url: The QUrl to open.
             predict: If set to False, predicted_navigation is not emitted.
         """
+        # WORKAROUND for https://bugreports.qt.io/browse/QTBUG-68076
+        self._widget.setFocus()
         self._saved_zoom = self.zoom.factor()
         self._openurl_prepare(url, predict=predict)
         self._widget.load(url)

qutebrowser/browser/webkit/network/networkmanager.py

@@ -28,7 +28,8 @@ from PyQt5.QtCore import (pyqtSlot, pyqtSignal, QCoreApplication, QUrl,
 from PyQt5.QtNetwork import QNetworkAccessManager, QNetworkReply, QSslSocket
 
 from qutebrowser.config import config
-from qutebrowser.utils import message, log, usertypes, utils, objreg, urlutils
+from qutebrowser.utils import (message, log, usertypes, utils, objreg,
+                               urlutils, debug)
 from qutebrowser.browser import shared
 from qutebrowser.browser.webkit import certificateerror
 from qutebrowser.browser.webkit.network import (webkitqutescheme, networkreply,
@@ -147,6 +148,7 @@ class NetworkManager(QNetworkAccessManager):
             super().__init__(parent)
         log.init.debug("NetworkManager init done")
         self.adopted_downloads = 0
+        self._args = objreg.get('args')
         self._win_id = win_id
         self._tab_id = tab_id
         self._private = private
@@ -406,5 +408,13 @@ class NetworkManager(QNetworkAccessManager):
                 # the webpage shutdown here.
                 current_url = QUrl()
 
+        if 'log-requests' in self._args.debug_flags:
+            operation = debug.qenum_key(QNetworkAccessManager, op)
+            operation = operation.replace('Operation', '').upper()
+            log.webview.debug("{} {}, first-party {}".format(
+                operation,
+                req.url().toDisplayString(),
+                current_url.toDisplayString()))
+
         self.set_referer(req, current_url)
         return super().createRequest(op, req, outgoing_data)

qutebrowser/commands/command.py

@@ -123,6 +123,7 @@ class Command:
         self.pos_args = []
         self.desc = None
         self.flags_with_args = []
+        self._has_vararg = False
 
         # This is checked by future @cmdutils.argument calls so they fail
         # (as they'd be silently ignored otherwise)
@@ -170,6 +171,8 @@ class Command:
 
     def get_pos_arg_info(self, pos):
         """Get an ArgInfo tuple for the given positional parameter."""
+        if pos >= len(self.pos_args) and self._has_vararg:
+            pos = len(self.pos_args) - 1
         name = self.pos_args[pos][0]
         return self._qute_args.get(name, ArgInfo())
 
@@ -233,6 +236,8 @@ class Command:
             log.commands.vdebug('Adding arg {} of type {} -> {}'.format(
                 param.name, typ, callsig))
             self.parser.add_argument(*args, **kwargs)
+            if param.kind == inspect.Parameter.VAR_POSITIONAL:
+                self._has_vararg = True
         return signature.parameters.values()
 
     def _param_to_argparse_kwargs(self, param, is_bool):

qutebrowser/completion/completer.py

@@ -49,7 +49,7 @@ class Completer(QObject):
         _last_cursor_pos: The old cursor position so we avoid double completion
                           updates.
         _last_text: The old command text so we avoid double completion updates.
-        _last_completion_func: The completion function used for the last text.
+        _last_before_cursor: The prior value of before_cursor.
     """
 
     def __init__(self, *, cmd, win_id, parent=None):
@@ -62,7 +62,7 @@ class Completer(QObject):
         self._timer.timeout.connect(self._update_completion)
         self._last_cursor_pos = -1
         self._last_text = None
-        self._last_completion_func = None
+        self._last_before_cursor = None
         self._cmd.update_completion.connect(self.schedule_completion_update)
 
     def __repr__(self):
@@ -228,7 +228,7 @@ class Completer(QObject):
             # FIXME complete searches
             # https://github.com/qutebrowser/qutebrowser/issues/32
             completion.set_model(None)
-            self._last_completion_func = None
+            self._last_before_cursor = None
             return
 
         before_cursor, pattern, after_cursor = self._partition()
@@ -242,11 +242,11 @@ class Completer(QObject):
         if func is None:
             log.completion.debug('Clearing completion')
             completion.set_model(None)
-            self._last_completion_func = None
+            self._last_before_cursor = None
             return
 
-        if func != self._last_completion_func:
+        if before_cursor != self._last_before_cursor:
-            self._last_completion_func = func
+            self._last_before_cursor = before_cursor
             args = (x for x in before_cursor[1:] if not x.startswith('-'))
             with debug.log_time(log.completion, 'Starting {} completion'
                                 .format(func.__name__)):

qutebrowser/completion/models/configmodel.py

@@ -47,12 +47,12 @@ def customized_option(*, info):
     return model
 
 
-def value(optname, *_values, info):
+def value(optname, *values, info):
     """A CompletionModel filled with setting values.
 
     Args:
         optname: The name of the config option this model shows.
-        _values: The values already provided on the command line.
+        values: The values already provided on the command line.
         info: A CompletionInfo instance.
     """
     model = completionmodel.CompletionModel(column_widths=(30, 70, 0))
@@ -64,13 +64,18 @@ def value(optname, *_values, info):
 
     opt = info.config.get_opt(optname)
     default = opt.typ.to_str(opt.default)
-    cur_cat = listcategory.ListCategory(
+    cur_def = []
-        "Current/Default",
+    if current not in values:
-        [(current, "Current value"), (default, "Default value")])
+        cur_def.append((current, "Current value"))
-    model.add_category(cur_cat)
+    if default not in values:
+        cur_def.append((default, "Default value"))
+    if cur_def:
+        cur_cat = listcategory.ListCategory("Current/Default", cur_def)
+        model.add_category(cur_cat)
 
-    vals = opt.typ.complete()
+    vals = opt.typ.complete() or []
-    if vals is not None:
+    vals = [x for x in vals if x[0] not in values]
+    if vals:
         model.add_category(listcategory.ListCategory("Completions", vals))
     return model
 

qutebrowser/config/configdata.yml

@@ -2271,6 +2271,7 @@ bindings.default:
       ;R: hint --rapid links window
       ;d: hint links download
       ;t: hint inputs
+      gi: hint inputs --first
       h: scroll left
       j: scroll down
       k: scroll up

qutebrowser/mainwindow/statusbar/command.py

@@ -19,6 +19,8 @@
 
 """The commandline in the statusbar."""
 
+import functools
+
 from PyQt5.QtCore import pyqtSignal, pyqtSlot, Qt, QSize
 from PyQt5.QtWidgets import QSizePolicy
 
@@ -69,6 +71,26 @@ class Command(misc.MinimalLineEditMixin, misc.CommandLineEdit):
         self.textChanged.connect(self.updateGeometry)
         self.textChanged.connect(self._incremental_search)
 
+        self._command_dispatcher = objreg.get(
+            'command-dispatcher', scope='window', window=self._win_id)
+
+    def _handle_search(self):
+        """Check if the currently entered text is a search, and if so, run it.
+
+        Return:
+            True if a search was executed, False otherwise.
+        """
+        search_prefixes = {
+            '/': self._command_dispatcher.search,
+            '?': functools.partial(
+                self._command_dispatcher.search, reverse=True)
+        }
+        if self.prefix() in search_prefixes:
+            search_fn = search_prefixes[self.prefix()]
+            search_fn(self.text()[1:])
+            return True
+        return False
+
     def prefix(self):
         """Get the currently entered command prefix."""
         text = self.text()
@@ -162,17 +184,17 @@ class Command(misc.MinimalLineEditMixin, misc.CommandLineEdit):
         Args:
             rapid: Run the command without closing or clearing the command bar.
         """
-        prefixes = {
-            ':': '',
-            '/': 'search -- ',
-            '?': 'search -r -- ',
-        }
         text = self.text()
         self.history.append(text)
+
+        was_search = self._handle_search()
+
         if not rapid:
             modeman.leave(self._win_id, usertypes.KeyMode.command,
                           'cmd accept')
-        self.got_cmd[str].emit(prefixes[text[0]] + text[1:])
+
+        if not was_search:
+            self.got_cmd[str].emit(text[1:])
 
     @cmdutils.register(instance='status-command', scope='window')
     def edit_command(self, run=False):
@@ -253,15 +275,9 @@ class Command(misc.MinimalLineEditMixin, misc.CommandLineEdit):
         width = self.fontMetrics().width(text)
         return QSize(width, height)
 
-    @pyqtSlot(str)
+    @pyqtSlot()
-    def _incremental_search(self, text):
+    def _incremental_search(self):
         if not config.val.search.incremental:
             return
 
-        search_prefixes = {
+        self._handle_search()
-            '/': 'search -- ',
-            '?': 'search -r -- ',
-        }
-
-        if self.prefix() in ['/', '?']:
-            self.got_cmd[str].emit(search_prefixes[text[0]] + text[1:])

qutebrowser/mainwindow/tabbedbrowser.py

@@ -489,6 +489,8 @@ class TabbedBrowser(QWidget):
                                                self.widget.count())
         else:
             self.widget.setCurrentWidget(tab)
+            # WORKAROUND for https://bugreports.qt.io/browse/QTBUG-68076
+            tab.setFocus()
 
         tab.show()
         self.new_tab.emit(tab, idx)

qutebrowser/misc/autoupdate.py

@@ -45,7 +45,7 @@ class PyPIVersionClient(QObject):
                arg: The error message, as string.
     """
 
-    API_URL = 'https://pypi.python.org/pypi/{}/json'
+    API_URL = 'https://pypi.org/pypi/{}/json'
     success = pyqtSignal(str)
     error = pyqtSignal(str)
 

qutebrowser/misc/httpclient.py

@@ -28,6 +28,21 @@ from PyQt5.QtNetwork import (QNetworkAccessManager, QNetworkRequest,
                              QNetworkReply)
 
 
+class HTTPRequest(QNetworkRequest):
+    """A QNetworkRquest that follows (secure) redirects by default."""
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        try:
+            self.setAttribute(QNetworkRequest.RedirectPolicyAttribute,
+                              QNetworkRequest.NoLessSafeRedirectPolicy)
+        except AttributeError:
+            # RedirectPolicyAttribute was introduced in 5.9 to replace
+            # FollowRedirectsAttribute.
+            self.setAttribute(QNetworkRequest.FollowRedirectsAttribute,
+                              True)
+
+
 class HTTPClient(QObject):
 
     """An HTTP client based on QNetworkAccessManager.
@@ -63,7 +78,7 @@ class HTTPClient(QObject):
         if data is None:
             data = {}
         encoded_data = urllib.parse.urlencode(data).encode('utf-8')
-        request = QNetworkRequest(url)
+        request = HTTPRequest(url)
         request.setHeader(QNetworkRequest.ContentTypeHeader,
                           'application/x-www-form-urlencoded;charset=utf-8')
         reply = self._nam.post(request, encoded_data)
@@ -77,7 +92,7 @@ class HTTPClient(QObject):
         Args:
             url: The URL to access, as QUrl.
         """
-        request = QNetworkRequest(url)
+        request = HTTPRequest(url)
         reply = self._nam.get(request)
         self._handle_reply(reply)
 

qutebrowser/qutebrowser.py

@@ -61,7 +61,8 @@ def get_argparser():
     """Get the argparse parser."""
     parser = argparse.ArgumentParser(prog='qutebrowser',
                                      description=qutebrowser.__description__)
-    parser.add_argument('--basedir', help="Base directory for all storage.")
+    parser.add_argument('-B', '--basedir', help="Base directory for all "
+                        "storage.")
     parser.add_argument('-V', '--version', help="Show version and quit.",
                         action='store_true')
     parser.add_argument('-s', '--set', help="Set a temporary setting for "
@@ -102,7 +103,7 @@ def get_argparser():
                        help="How many lines of the debug log to keep in RAM "
                        "(-1: unlimited).",
                        default=2000, type=int)
-    debug.add_argument('--debug', help="Turn on debugging options.",
+    debug.add_argument('-d', '--debug', help="Turn on debugging options.",
                        action='store_true')
     debug.add_argument('--json-logging', action='store_true', help="Output log"
                        " lines in JSON format (one object per line).")
@@ -112,8 +113,8 @@ def get_argparser():
                        action='store_true')
     debug.add_argument('--nowindow', action='store_true', help="Don't show "
                        "the main window.")
-    debug.add_argument('--temp-basedir', action='store_true', help="Use a "
+    debug.add_argument('-T', '--temp-basedir', action='store_true', help="Use "
-                       "temporary basedir.")
+                       "a temporary basedir.")
     debug.add_argument('--no-err-windows', action='store_true', help="Don't "
                        "show any error windows (used for tests/smoke.py).")
     debug.add_argument('--qt-arg', help="Pass an argument with a value to Qt. "
@@ -123,9 +124,9 @@ def get_argparser():
                        action='append')
     debug.add_argument('--qt-flag', help="Pass an argument to Qt as flag.",
                        nargs=1, action='append')
-    debug.add_argument('--debug-flag', type=debug_flag_error, default=[],
+    debug.add_argument('-D', '--debug-flag', type=debug_flag_error,
-                       help="Pass name of debugging feature to be turned on.",
+                       default=[], help="Pass name of debugging feature to be"
-                       action='append', dest='debug_flags')
+                       " turned on.", action='append', dest='debug_flags')
     parser.add_argument('command', nargs='*', help="Commands to execute on "
                         "startup.", metavar=':command')
     # URLs will actually be in command
@@ -159,9 +160,12 @@ def debug_flag_error(flag):
     Available flags:
         debug-exit: Turn on debugging of late exit.
         pdb-postmortem: Drop into pdb on exceptions.
+        no-sql-history: Don't store history items.
+        no-scroll-filtering: Process all scrolling updates.
+        log-requests: Log all network requests.
     """
     valid_flags = ['debug-exit', 'pdb-postmortem', 'no-sql-history',
-                   'no-scroll-filtering']
+                   'no-scroll-filtering', 'log-requests']
 
     if flag in valid_flags:
         return flag

requirements.txt

@@ -1,6 +1,6 @@
 # This file is automatically generated by scripts/dev/recompile_requirements.py
 
-attrs==17.4.0
+attrs==18.1.0
 colorama==0.3.9
 cssutils==1.0.2
 Jinja2==2.10

tests/end2end/data/search.html

@@ -16,6 +16,8 @@
             BAZ<br/>
             space travel<br/>
             /slash<br/>
+            -r reversed<br/>
+            ;; semicolons<br/>
             <a class="toselect" href="hello.txt">follow me!</a><br/>
         </p>
     </body>

tests/end2end/features/hints.feature

@@ -509,3 +509,17 @@ Feature: Using hints
         And I press the key "hello"
         And I press the key "<Enter>"
         Then data/hello.txt should be loaded
+
+    Scenario: Using --first with normal links
+        When I open data/hints/html/simple.html
+        And I hint with args "all --first"
+        Then data/hello.txt should be loaded
+
+    Scenario: Using --first with inputs
+        When I open data/hints/input.html
+        And I hint with args "inputs --first"
+        And I wait for "Entering mode KeyMode.insert (reason: clicking input)" in the log
+        # ensure we clicked the first element
+        And I run :jseval console.log(document.activeElement.id == "qute-input");
+        And I run :leave-mode
+        Then the javascript message "true" should be logged

tests/end2end/features/search.feature

@@ -40,11 +40,26 @@ Feature: Searching on a page
         Then "space " should be found
 
     Scenario: Searching with / and slash in search term (issue 507)
-        When I run :set-cmd-text -s //slash
+        When I run :set-cmd-text //slash
         And I run :command-accept
         And I wait for "search found /slash" in the log
         Then "/slash" should be found
 
+    Scenario: Searching with arguments at start of search term
+        When I run :set-cmd-text /-r reversed
+        And I run :command-accept
+        And I wait for "search found -r reversed" in the log
+        Then "-r reversed" should be found
+
+    Scenario: Searching with semicolons in search term
+        When I run :set-cmd-text /;
+        And I run :fake-key -g ;
+        And I run :fake-key -g <space>
+        And I run :fake-key -g semi
+        And I run :command-accept
+        And I wait for "search found ;; semi" in the log
+        Then ";; semi" should be found
+
     # This doesn't work because this is QtWebKit behavior.
     @xfail_norun
     Scenario: Searching text with umlauts

tests/helpers/fixtures.py

@@ -154,7 +154,7 @@ def greasemonkey_manager(data_tmpdir):
 
 @pytest.fixture
 def webkit_tab(qtbot, tab_registry, cookiejar_and_cache, mode_manager,
-               session_manager_stub, greasemonkey_manager):
+               session_manager_stub, greasemonkey_manager, fake_args):
     webkittab = pytest.importorskip('qutebrowser.browser.webkit.webkittab')
     tab = webkittab.WebKitTab(win_id=0, mode_manager=mode_manager,
                               private=False)

tests/unit/browser/webengine/test_spell.py

@@ -17,31 +17,69 @@
 # You should have received a copy of the GNU General Public License
 # along with qutebrowser.  If not, see <http://www.gnu.org/licenses/>.
 
-import pytest
+"""Tests for qutebrowser.browser.webengine.spell module."""
+
+import logging
+import os
+
+from PyQt5.QtCore import QLibraryInfo
 from qutebrowser.browser.webengine import spell
+from qutebrowser.utils import usertypes
 
 
-def test_version():
+def test_version(message_mock, caplog):
+    """Tests parsing dictionary version from its file name."""
     assert spell.version('en-US-8-0.bdic') == (8, 0)
     assert spell.version('pl-PL-3-0.bdic') == (3, 0)
-    with pytest.raises(ValueError):
+    with caplog.at_level(logging.WARNING):
-        spell.version('malformed_filename')
+        assert spell.version('malformed_filename') is None
+    msg = message_mock.getmsg(usertypes.MessageLevel.warning)
+    expected = ("Found a dictionary with a malformed name: malformed_filename")
+    assert msg.text == expected
 
 
-def test_local_filename_dictionary_does_not_exist(tmpdir, monkeypatch):
+def test_dictionary_dir(monkeypatch):
+    monkeypatch.setattr(QLibraryInfo, 'location', lambda _: 'datapath')
+    assert spell.dictionary_dir() == os.path.join('datapath',
+                                                  'qtwebengine_dictionaries')
+
+
+def test_local_filename_dictionary_does_not_exist(monkeypatch):
+    """Tests retrieving local filename when the dir doesn't exits."""
     monkeypatch.setattr(
         spell, 'dictionary_dir', lambda: '/some-non-existing-dir')
     assert not spell.local_filename('en-US')
 
 
 def test_local_filename_dictionary_not_installed(tmpdir, monkeypatch):
+    """Tests retrieving local filename when the dict not installed."""
     monkeypatch.setattr(spell, 'dictionary_dir', lambda: str(tmpdir))
     assert not spell.local_filename('en-US')
 
 
+def test_local_filename_not_installed_malformed(tmpdir, monkeypatch, caplog):
+    """Tests retrieving local filename when the only file is malformed."""
+    monkeypatch.setattr(spell, 'dictionary_dir', lambda: str(tmpdir))
+    (tmpdir / 'en-US.bdic').ensure()
+    with caplog.at_level(logging.WARNING):
+        assert not spell.local_filename('en-US')
+
+
 def test_local_filename_dictionary_installed(tmpdir, monkeypatch):
+    """Tests retrieving local filename when the dict installed."""
     monkeypatch.setattr(spell, 'dictionary_dir', lambda: str(tmpdir))
     for lang_file in ['en-US-11-0.bdic', 'en-US-7-1.bdic', 'pl-PL-3-0.bdic']:
         (tmpdir / lang_file).ensure()
     assert spell.local_filename('en-US') == 'en-US-11-0'
     assert spell.local_filename('pl-PL') == 'pl-PL-3-0'
+
+
+def test_local_filename_installed_malformed(tmpdir, monkeypatch, caplog):
+    """Tests retrieving local filename when the dict installed.
+
+    In this usecase, another existing file is malformed."""
+    monkeypatch.setattr(spell, 'dictionary_dir', lambda: str(tmpdir))
+    for lang_file in ['en-US-11-0.bdic', 'en-US-7-1.bdic', 'en-US.bdic']:
+        (tmpdir / lang_file).ensure()
+    with caplog.at_level(logging.WARNING):
+        assert spell.local_filename('en-US') == 'en-US-11-0'

tests/unit/browser/webengine/test_webenginesettings.py

@@ -73,3 +73,14 @@ def test_existing_dict(config_stub, monkeypatch):
                     webenginesettings.private_profile]:
         assert profile.isSpellCheckEnabled()
         assert profile.spellCheckLanguages() == ['en-US-8-0']
+
+
+@pytest.mark.skipif(
+    not qtutils.version_check('5.8'), reason="Needs Qt 5.8 or newer")
+def test_spell_check_disabled(config_stub, monkeypatch):
+    monkeypatch.setattr(objects, 'backend', usertypes.Backend.QtWebEngine)
+    config_stub.val.spellcheck.languages = []
+    webenginesettings._update_settings('spellcheck.languages')
+    for profile in [webenginesettings.default_profile,
+                    webenginesettings.private_profile]:
+        assert not profile.isSpellCheckEnabled()

tests/unit/browser/webkit/network/test_networkmanager.py

@@ -26,7 +26,7 @@ from qutebrowser.browser.webkit import cookies
 pytestmark = pytest.mark.usefixtures('cookiejar_and_cache')
 
 
-def test_init_with_private_mode():
+def test_init_with_private_mode(fake_args):
     nam = networkmanager.NetworkManager(win_id=0, tab_id=0, private=True)
     assert isinstance(nam.cookieJar(), cookies.RAMCookieJar)
     assert nam.cache() is None

tests/unit/browser/webkit/test_downloads.py

@@ -22,7 +22,8 @@ import pytest
 from qutebrowser.browser import downloads, qtnetworkdownloads
 
 
-def test_download_model(qapp, qtmodeltester, config_stub, cookiejar_and_cache):
+def test_download_model(qapp, qtmodeltester, config_stub, cookiejar_and_cache,
+                        fake_args):
     """Simple check for download model internals."""
     manager = qtnetworkdownloads.DownloadManager()
     model = downloads.DownloadModel(manager)

tests/unit/completion/test_completer.py

@@ -129,12 +129,20 @@ def cmdutils_patch(monkeypatch, stubs, miscmodels_patch):
         """docstring."""
         pass
 
+    @cmdutils.argument('option', completion=miscmodels_patch.option)
+    @cmdutils.argument('values', completion=miscmodels_patch.value)
+    def config_cycle(option, *values):
+        """For testing varargs."""
+        pass
+
     cmd_utils = stubs.FakeCmdUtils({
         'set': command.Command(name='set', handler=set_command),
         'help': command.Command(name='help', handler=show_help),
         'open': command.Command(name='open', handler=openurl, maxsplit=0),
         'bind': command.Command(name='bind', handler=bind),
         'tab-detach': command.Command(name='tab-detach', handler=tab_detach),
+        'config-cycle': command.Command(name='config-cycle',
+                                        handler=config_cycle),
     })
     monkeypatch.setattr(completer, 'cmdutils', cmd_utils)
 
@@ -191,6 +199,10 @@ def _set_cmd_prompt(cmd, txt):
     ('/:help|', None, '', []),
     ('::bind|', 'command', ':bind', []),
     (':-w open |', None, '', []),
+    # varargs
+    (':config-cycle option |', 'value', '', ['option']),
+    (':config-cycle option one |', 'value', '', ['option', 'one']),
+    (':config-cycle option one two |', 'value', '', ['option', 'one', 'two']),
 ])
 def test_update_completion(txt, kind, pattern, pos_args, status_command_stub,
                            completer_obj, completion_widget_stub, config_stub,
@@ -211,6 +223,32 @@ def test_update_completion(txt, kind, pattern, pos_args, status_command_stub,
         completion_widget_stub.set_pattern.assert_called_once_with(pattern)
 
 
+@pytest.mark.parametrize('txt1, txt2, regen', [
+    (':config-cycle |', ':config-cycle a|', False),
+    (':config-cycle abc|', ':config-cycle abc |', True),
+    (':config-cycle abc |', ':config-cycle abc d|', False),
+    (':config-cycle abc def|', ':config-cycle abc def |', True),
+    # open has maxsplit=0, so all args just set the pattern, not the model
+    (':open |', ':open a|', False),
+    (':open abc|', ':open abc |', False),
+    (':open abc |', ':open abc d|', False),
+    (':open abc def|', ':open abc def |', False),
+])
+def test_regen_completion(txt1, txt2, regen, status_command_stub,
+                          completer_obj, completion_widget_stub, config_stub,
+                          key_config_stub):
+    """Test that the completion function is only called as needed."""
+    # set the initial state
+    _set_cmd_prompt(status_command_stub, txt1)
+    completer_obj.schedule_completion_update()
+    completion_widget_stub.set_model.reset_mock()
+
+    # "move" the cursor and check if the completion function was called
+    _set_cmd_prompt(status_command_stub, txt2)
+    completer_obj.schedule_completion_update()
+    assert completion_widget_stub.set_model.called == regen
+
+
 @pytest.mark.parametrize('before, newtxt, after', [
     (':|', 'set', ':set|'),
     (':| ', 'set', ':set|'),

tests/unit/completion/test_models.py

@@ -739,6 +739,44 @@ def test_setting_value_completion_invalid(info):
     assert configmodel.value(optname='foobarbaz', info=info) is None
 
 
+@pytest.mark.parametrize('args, expected', [
+    ([], {
+        "Current/Default": [
+            ('true', 'Current value', None),
+            ('true', 'Default value', None),
+        ],
+        "Completions": [
+            ('false', '', None),
+            ('true', '', None),
+        ],
+    }),
+    (['false'], {
+        "Current/Default": [
+            ('true', 'Current value', None),
+            ('true', 'Default value', None),
+        ],
+        "Completions": [
+            ('true', '', None),
+        ],
+    }),
+    (['true'], {
+        "Completions": [
+            ('false', '', None),
+        ],
+    }),
+    (['false', 'true'], {}),
+])
+def test_setting_value_cycle(qtmodeltester, config_stub, configdata_stub,
+                             info, args, expected):
+    opt = 'content.javascript.enabled'
+
+    model = configmodel.value(opt, *args, info=info)
+    model.set_pattern('')
+    qtmodeltester.data_display_may_return_none = True
+    qtmodeltester.check(model)
+    _check_completions(model, expected)
+
+
 def test_bind_completion(qtmodeltester, cmdutils_stub, config_stub,
                          key_config_stub, configdata_stub, info):
     """Test the results of keybinding command completion.

tests/unit/javascript/test_greasemonkey.py

@@ -32,7 +32,7 @@ test_gm_script = r"""
 // @name qutebrowser test userscript
 // @namespace invalid.org
 // @include http://localhost:*/data/title.html
-// @match http://trolol*
+// @match http://*.trolol.com/*
 // @exclude https://badhost.xxx/*
 // @run-at document-start
 // ==/UserScript==
@@ -60,7 +60,7 @@ def test_all():
 
 @pytest.mark.parametrize("url, expected_matches", [
     # included
-    ('http://trololololololo.com/', 1),
+    ('http://trolol.com/', 1),
     # neither included nor excluded
     ('http://aaaaaaaaaa.com/', 0),
     # excluded

tests/unit/misc/test_autoupdate.py

@@ -67,7 +67,7 @@ def test_get_version_success(qtbot):
         with qtbot.waitSignal(client.success):
             client.get_version('test')
 
-    assert http_stub.url == QUrl('https://pypi.python.org/pypi/test/json')
+    assert http_stub.url == QUrl(client.API_URL.format('test'))
 
 
 def test_get_version_error(qtbot):