rnhmjoj/qutebrowser
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Merge remote-tracking branch 'upstream/master'

Browse Source
This commit is contained in:
unknown 2017-12-06 13:37:05 -07:00
commit f0de3601cb
49 changed files with 1095 additions and 99 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
.travis.yml
View File

@ -52,6 +52,10 @@ matrix:
language: node_js
python: null
node_js: "lts/*"
- os: linux
language: generic
env: TESTENV=shellcheck
services: docker
fast_finish: true
cache:

9
doc/changelog.asciidoc
View File

@ -21,6 +21,8 @@ v1.1.0 (unreleased)
Added
~~~~~
- Initial support for Greasemonkey scripts. There are still some rough edges,
but many scripts should already work.
- There's now a `misc/Makefile` file in releases, which should help
distributions which package qutebrowser, as they can run something like
`make -f misc/Makefile DESTDIR="$pkgdir" install` now.
@ -51,6 +53,9 @@ Added
- New `:edit-command` command to edit the commandline in an editor.
- New `tabs.persist_mode_on_change` setting to keep the current mode when
switching tabs.
- New `session.lazy_restore` setting which allows to not load pages immediately
when restoring a session.
- New `hist_importer.py` script to import history from Firefox/Chromium.
Changed
~~~~~~~
@ -89,6 +94,8 @@ Changed
data dir, e.g. `~/.local/share/qutebrowser/js`.
- The current/default bindings are now shown in the :bind completion.
- Empty categories are now hidden in the `:open` completion.
- Search terms for URLs and titles can now be mixed when filtering the
completion.
Fixed
~~~~~
@ -107,6 +114,8 @@ Fixed
in a URL.
- Using e.g. `-s backend webkit` to set the backend now works correctly.
- Fixed crash when closing the tab an external editor was opened in.
- When using `:search-next` before a search is finished, no warning about no
results being found is shown anymore.
Deprecated
~~~~~~~~~~

9
doc/help/commands.asciidoc
View File

@ -54,6 +54,7 @@ It is possible to run or bind multiple commands by separating them with `;;`.
|<<follow-selected,follow-selected>>|Follow the selected text.
|<<forward,forward>>|Go forward in the history of the current tab.
|<<fullscreen,fullscreen>>|Toggle fullscreen mode.
|<<greasemonkey-reload,greasemonkey-reload>>|Re-read Greasemonkey scripts from disk.
|<<help,help>>|Show help about a command or setting.
|<<hint,hint>>|Start hinting.
|<<history,history>>|Show browsing history.
@ -491,6 +492,12 @@ Toggle fullscreen mode.
==== optional arguments
* +*-l*+, +*--leave*+: Only leave fullscreen if it was entered by the page.
[[greasemonkey-reload]]
=== greasemonkey-reload
Re-read Greasemonkey scripts from disk.
The scripts are read from a 'greasemonkey' subdirectory in qutebrowser's data directory (see `:version`).
[[help]]
=== help
Syntax: +:help [*--tab*] [*--bg*] [*--window*] ['topic']+
@ -1083,7 +1090,7 @@ Syntax: +:session-save [*--current*] [*--quiet*] [*--force*] [*--only-active-win
Save a session.
==== positional arguments
* +'name'+: The name of the session. If not given, the session configured in session_default_name is saved.
* +'name'+: The name of the session. If not given, the session configured in session.default_name is saved.
==== optional arguments

15
doc/help/settings.asciidoc
View File

@ -222,7 +222,8 @@
|<<qt.highdpi,qt.highdpi>>|Turn on Qt HighDPI scaling.
|<<scrolling.bar,scrolling.bar>>|Show a scrollbar.
|<<scrolling.smooth,scrolling.smooth>>|Enable smooth scrolling for web pages.
|<<session_default_name,session_default_name>>|Name of the session to save by default.
|<<session.default_name,session.default_name>>|Name of the session to save by default.
|<<session.lazy_restore,session.lazy_restore>>|Load a restored tab as soon as it takes focus.
|<<spellcheck.languages,spellcheck.languages>>|Languages to use for spell checking.
|<<statusbar.hide,statusbar.hide>>|Hide the statusbar unless a message is shown.
|<<statusbar.padding,statusbar.padding>>|Padding (in pixels) for the statusbar.
@ -2556,8 +2557,8 @@ Type: <<types,Bool>>
Default: +pass:[false]+
[[session_default_name]]
=== session_default_name
[[session.default_name]]
=== session.default_name
Name of the session to save by default.
If this is set to null, the session which was last loaded is saved.
@ -2565,6 +2566,14 @@ Type: <<types,SessionName>>
Default: empty
[[session.lazy_restore]]
=== session.lazy_restore
Load a restored tab as soon as it takes focus.
Type: <<types,Bool>>
Default: +pass:[false]+
[[spellcheck.languages]]
=== spellcheck.languages
Languages to use for spell checking.

15
doc/install.asciidoc
View File

@ -35,7 +35,7 @@ Ubuntu 16.04 doesn't come with an up-to-date engine (a new enough QtWebKit, or
QtWebEngine). However, it comes with Python 3.5, so you can
<<tox,install qutebrowser via tox>>.
Debian Stretch / Ubuntu 17.04 and newer
Debian Stretch / Ubuntu 17.04 and 17.10
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Those versions come with QtWebEngine in the repositories. This makes it possible
@ -54,7 +54,18 @@ Install the packages:
# apt install ./qutebrowser_*_all.deb
----
Some additional hints:
Debian Testing / Ubuntu 18.04
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
On Debian Testing, qutebrowser is in the official repositories, and you can
install it with apt:
----
# apt install qutebrowser
----
Additional hints
~~~~~~~~~~~~~~~~
- Alternatively, you can <<tox,install qutebrowser via tox>> to get a newer
QtWebEngine version.

2
misc/requirements/requirements-tests.txt
View File

@ -13,7 +13,7 @@ glob2==0.6
hunter==2.0.2
hypothesis==3.40.1
itsdangerous==0.24
# Jinja2==2.9.6
# Jinja2==2.10
Mako==1.0.7
# MarkupSafe==1.0
parse==1.8.2

1
misc/requirements/requirements-tox.txt
View File

@ -2,5 +2,6 @@
pluggy==0.6.0
py==1.5.2
six==1.11.0
tox==2.9.1
virtualenv==15.1.0

4
misc/userscripts/cast
View File

@ -144,7 +144,7 @@ fi
pkill -f "${program_}"
# start youtube download in stream mode (-o -) into temporary file
youtube-dl -qo - "$1" > ${file_to_cast} &
youtube-dl -qo - "$1" > "${file_to_cast}" &
ytdl_pid=$!
msg info "Casting $1" >> "$QUTE_FIFO"
@ -153,4 +153,4 @@ tail -F "${file_to_cast}" | ${program_} -
# cleanup remaining background process and file on disk
kill ${ytdl_pid}
rm -rf ${tmpdir}
rm -rf "${tmpdir}"

2
misc/userscripts/dmenu_qutebrowser
View File

@ -41,7 +41,7 @@
[ -z "$QUTE_URL" ] && QUTE_URL='http://google.com'
url=$(echo "$QUTE_URL" | cat - "$QUTE_CONFIG_DIR/quickmarks" "$QUTE_DATA_DIR/history" | dmenu -l 15 -p qutebrowser)
url=$(echo "$url" | sed -E 's/[^ ]+ +//g' | egrep "https?:" || echo "$url")
url=$(echo "$url" | sed -E 's/[^ ]+ +//g' | grep -E "https?:" || echo "$url")
[ -z "${url// }" ] && exit

33
misc/userscripts/format_json
View File

@ -1,4 +1,5 @@
#!/bin/sh
set -euo pipefail
#
# Behavior:
# Userscript for qutebrowser which will take the raw JSON text of the current
@ -19,29 +20,23 @@
#
# Bryan Gilbert, 2017
# do not run pygmentize on files larger than this amount of bytes
MAX_SIZE_PRETTIFY=10485760 # 10 MB
# default style to monokai if none is provided
STYLE=${1:-monokai}
# format json using jq
FORMATTED_JSON="$(cat "$QUTE_TEXT" | jq '.')"
# if jq command failed or formatted json is empty, assume failure and terminate
if [ $? -ne 0 ] || [ -z "$FORMATTED_JSON" ]; then
echo "Invalid json, aborting..."
exit 1
TEMP_FILE="$(mktemp)"
jq . "$QUTE_TEXT" >"$TEMP_FILE"
# try GNU stat first and then OSX stat if the former fails
FILE_SIZE=$(
stat --printf="%s" "$TEMP_FILE" 2>/dev/null ||
stat -f%z "$TEMP_FILE" 2>/dev/null
)
if [ "$FILE_SIZE" -lt "$MAX_SIZE_PRETTIFY" ]; then
pygmentize -l json -f html -O full,style="$STYLE" <"$TEMP_FILE" >"${TEMP_FILE}_"
mv -f "${TEMP_FILE}_" "$TEMP_FILE"
fi
# calculate the filesize of the json document
FILE_SIZE=$(ls -s --block-size=1048576 "$QUTE_TEXT" | cut -d' ' -f1)
# use pygments to pretty-up the json (syntax highlight) if file is less than 10MB
if [ "$FILE_SIZE" -lt "10" ]; then
FORMATTED_JSON="$(echo "$FORMATTED_JSON" | pygmentize -l json -f html -O full,style=$STYLE)"
fi
# create a temp file and write the formatted json to that file
TEMP_FILE="$(mktemp --suffix '.html')"
echo "$FORMATTED_JSON" > $TEMP_FILE
# send the command to qutebrowser to open the new file containing the formatted json
echo "open -t file://$TEMP_FILE" >> "$QUTE_FIFO"

5
misc/userscripts/open_download
View File

@ -76,6 +76,7 @@ crop-first-column() {
ls-files() {
# add the slash at the end of the download dir enforces to follow the
# symlink, if the DOWNLOAD_DIR itself is a symlink
# shellcheck disable=SC2010
ls -Q --quoting-style escape -h -o -1 -A -t "${DOWNLOAD_DIR}/" \
| grep '^[-]' \
| cut -d' ' -f3- \
@ -91,10 +92,10 @@ if [ "${#entries[@]}" -eq 0 ] ; then
die "Download directory »${DOWNLOAD_DIR}« empty"
fi
line=$(printf "%s\n" "${entries[@]}" \
line=$(printf '%s\n' "${entries[@]}" \
| crop-first-column 55 \
| column -s $'\t' -t \
| $ROFI_CMD "${rofi_default_args[@]}" $ROFI_ARGS) || true
| $ROFI_CMD "${rofi_default_args[@]}" "$ROFI_ARGS") || true
if [ -z "$line" ]; then
exit 0
fi

13
misc/userscripts/password_fill
View File

@ -64,7 +64,7 @@ die() {
javascript_escape() {
# print the first argument in an escaped way, such that it can safely
# be used within javascripts double quotes
sed "s,[\\\'\"],\\\&,g" <<< "$1"
sed "s,[\\\\'\"],\\\\&,g" <<< "$1"
}
# ======================================================= #
@ -178,7 +178,7 @@ choose_entry_menu() {
if [ "$nr" -eq 1 ] && ! ((menu_if_one_entry)) ; then
file="${files[0]}"
else
file=$( printf "%s\n" "${files[@]}" | "${MENU_COMMAND[@]}" )
file=$( printf '%s\n' "${files[@]}" | "${MENU_COMMAND[@]}" )
fi
}
@ -236,7 +236,7 @@ pass_backend() {
if ((match_line)) ; then
# add entries with matching URL-tag
while read -r -d "" passfile ; do
if $GPG "${GPG_OPTS}" -d "$passfile" \
if $GPG "${GPG_OPTS[@]}" -d "$passfile" \
| grep --max-count=1 -iE "${match_line_pattern}${url}" > /dev/null
then
passfile="${passfile#$PREFIX}"
@ -269,7 +269,7 @@ pass_backend() {
break
fi
fi
done < <($GPG "${GPG_OPTS}" -d "$path" )
done < <($GPG "${GPG_OPTS[@]}" -d "$path" )
}
}
# =======================================================
@ -283,7 +283,7 @@ secret_backend() {
query_entries() {
local domain="$1"
while read -r line ; do
if [[ "$line" =~ "attribute.username = " ]] ; then
if [[ "$line" == "attribute.username ="* ]] ; then
files+=("$domain ${line#${BASH_REMATCH[0]}}")
fi
done < <( secret-tool search --unlock --all domain "$domain" 2>&1 )
@ -303,6 +303,7 @@ pass_backend
QUTE_CONFIG_DIR=${QUTE_CONFIG_DIR:-${XDG_CONFIG_HOME:-$HOME/.config}/qutebrowser/}
PWFILL_CONFIG=${PWFILL_CONFIG:-${QUTE_CONFIG_DIR}/password_fill_rc}
if [ -f "$PWFILL_CONFIG" ] ; then
# shellcheck source=/dev/null
source "$PWFILL_CONFIG"
fi
init
@ -311,7 +312,7 @@ simplify_url "$QUTE_URL"
query_entries "${simple_url}"
no_entries_found
# remove duplicates
mapfile -t files < <(printf "%s\n" "${files[@]}" | sort | uniq )
mapfile -t files < <(printf '%s\n' "${files[@]}" | sort | uniq )
choose_entry
if [ -z "$file" ] ; then
# choose_entry didn't want any of these entries

13
misc/userscripts/qutedmenu
View File

@ -35,17 +35,12 @@ get_selection() {
# Main
# https://github.com/halfwit/dotfiles/blob/master/.config/dmenu/font
if [[ -s $confdir/dmenu/font ]]; then
read -r font < "$confdir"/dmenu/font
fi
[[ -s $confdir/dmenu/font ]] && read -r font < "$confdir"/dmenu/font
if [[ $font ]]; then
opts+=(-fn "$font")
fi
[[ $font ]] && opts+=(-fn "$font")
if [[ -s $optsfile ]]; then
source "$optsfile"
fi
# shellcheck source=/dev/null
[[ -s $optsfile ]] && source "$optsfile"
url=$(get_selection)
url=${url/*http/http}

10
misc/userscripts/rss
View File

@ -32,7 +32,7 @@ add_feed () {
if grep -Fq "$1" "feeds"; then
notice "$1 is saved already."
else
printf "%s\n" "$1" >> "feeds"
printf '%s\n' "$1" >> "feeds"
fi
}
@ -57,7 +57,7 @@ notice () {
# Update a database of a feed and open new URLs
read_items () {
cd read_urls
cd read_urls || return 1
feed_file="$(echo "$1" | tr -d /)"
feed_temp_file="$(mktemp "$feed_file.tmp.XXXXXXXXXX")"
feed_new_items="$(mktemp "$feed_file.new.XXXXXXXXXX")"
@ -75,7 +75,7 @@ read_items () {
cat "$feed_new_items" >> "$feed_file"
sort -o "$feed_file" "$feed_file"
rm "$feed_temp_file" "$feed_new_items"
fi | while read item; do
fi | while read -r item; do
echo "open -t $item" > "$QUTE_FIFO"
done
}
@ -85,7 +85,7 @@ if [ ! -d "$config_dir/read_urls" ]; then
mkdir -p "$config_dir/read_urls"
fi
cd "$config_dir"
cd "$config_dir" || exit 1
if [ $# != 0 ]; then
for arg in "$@"; do
@ -115,7 +115,7 @@ if < /dev/null grep --help 2>&1 | grep -q -- -a; then
text_only="-a"
fi
while read feed_url; do
while read -r feed_url; do
read_items "$feed_url" &
done < "$config_dir/feeds"

8
misc/userscripts/taskadd
View File

@ -25,12 +25,10 @@
[[ $QUTE_MODE == 'hints' ]] && title=$QUTE_SELECTED_TEXT || title=$QUTE_TITLE
# try to add the task and grab the output
msg="$(task add $title $@ 2>&1)"
if [[ $? == 0 ]]; then
if msg="$(task add "$title" "$*" 2>&1)"; then
# annotate the new task with the url, send the output back to the browser
task +LATEST annotate "$QUTE_URL"
echo "message-info '$msg'" >> $QUTE_FIFO
echo "message-info '$msg'" >> "$QUTE_FIFO"
else
echo "message-error '$msg'" >> $QUTE_FIFO
echo "message-error '$msg'" >> "$QUTE_FIFO"
fi

2
misc/userscripts/view_in_mpv
View File

@ -50,7 +50,7 @@ msg() {
MPV_COMMAND=${MPV_COMMAND:-mpv}
# Warning: spaces in single flags are not supported
MPV_FLAGS=${MPV_FLAGS:- --force-window --no-terminal --keep-open=yes --ytdl --ytdl-raw-options=yes-playlist=}
video_command=( "$MPV_COMMAND" $MPV_FLAGS )
IFS=" " read -r -a video_command <<< "$MPV_COMMAND $MPV_FLAGS"
js() {
cat <<EOF

5
qutebrowser/app.py
View File

@ -64,7 +64,7 @@ from qutebrowser.completion.models import miscmodels
from qutebrowser.commands import cmdutils, runners, cmdexc
from qutebrowser.config import config, websettings, configfiles, configinit
from qutebrowser.browser import (urlmarks, adblock, history, browsertab,
downloads)
downloads, greasemonkey)
from qutebrowser.browser.network import proxy
from qutebrowser.browser.webkit import cookies, cache
from qutebrowser.browser.webkit.network import networkmanager
@ -491,6 +491,9 @@ def _init_modules(args, crash_handler):
diskcache = cache.DiskCache(standarddir.cache(), parent=qApp)
objreg.register('cache', diskcache)
log.init.debug("Initializing Greasemonkey...")
greasemonkey.init()
log.init.debug("Misc initialization...")
macros.init()
# Init backend-specific stuff

224
qutebrowser/browser/greasemonkey.py Normal file
View File

@ -0,0 +1,224 @@
# vim: ft=python fileencoding=utf-8 sts=4 sw=4 et:
# Copyright 2017 Florian Bruhin (The Compiler) <mail@qutebrowser.org>
#
# This file is part of qutebrowser.
#
# qutebrowser is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# qutebrowser is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with qutebrowser. If not, see <http://www.gnu.org/licenses/>.
"""Load, parse and make available Greasemonkey scripts."""
import re
import os
import json
import fnmatch
import functools
import glob
import attr
from PyQt5.QtCore import pyqtSignal, QObject, QUrl
from qutebrowser.utils import log, standarddir, jinja, objreg
from qutebrowser.commands import cmdutils
def _scripts_dir():
"""Get the directory of the scripts."""
return os.path.join(standarddir.data(), 'greasemonkey')
class GreasemonkeyScript:
"""Container class for userscripts, parses metadata blocks."""
def __init__(self, properties, code):
self._code = code
self.includes = []
self.excludes = []
self.description = None
self.name = None
self.namespace = None
self.run_at = None
self.script_meta = None
self.runs_on_sub_frames = True
for name, value in properties:
if name == 'name':
self.name = value
elif name == 'namespace':
self.namespace = value
elif name == 'description':
self.description = value
elif name in ['include', 'match']:
self.includes.append(value)
elif name in ['exclude', 'exclude_match']:
self.excludes.append(value)
elif name == 'run-at':
self.run_at = value
elif name == 'noframes':
self.runs_on_sub_frames = False
HEADER_REGEX = r'// ==UserScript==|\n+// ==/UserScript==\n'
PROPS_REGEX = r'// @(?P<prop>[^\s]+)\s*(?P<val>.*)'
@classmethod
def parse(cls, source):
"""GreasemonkeyScript factory.
Takes a userscript source and returns a GreasemonkeyScript.
Parses the Greasemonkey metadata block, if present, to fill out
attributes.
"""
matches = re.split(cls.HEADER_REGEX, source, maxsplit=2)
try:
_head, props, _code = matches
except ValueError:
props = ""
script = cls(re.findall(cls.PROPS_REGEX, props), source)
script.script_meta = props
if not props:
script.includes = ['*']
return script
def code(self):
"""Return the processed JavaScript code of this script.
Adorns the source code with GM_* methods for Greasemonkey
compatibility and wraps it in an IFFE to hide it within a
lexical scope. Note that this means line numbers in your
browser's debugger/inspector will not match up to the line
numbers in the source script directly.
"""
return jinja.js_environment.get_template(
'greasemonkey_wrapper.js').render(
scriptName="/".join([self.namespace or '', self.name]),
scriptInfo=self._meta_json(),
scriptMeta=self.script_meta,
scriptSource=self._code)
def _meta_json(self):
return json.dumps({
'name': self.name,
'description': self.description,
'matches': self.includes,
'includes': self.includes,
'excludes': self.excludes,
'run-at': self.run_at,
})
@attr.s
class MatchingScripts(object):
"""All userscripts registered to run on a particular url."""
url = attr.ib()
start = attr.ib(default=attr.Factory(list))
end = attr.ib(default=attr.Factory(list))
idle = attr.ib(default=attr.Factory(list))
class GreasemonkeyManager(QObject):
"""Manager of userscripts and a Greasemonkey compatible environment.
Signals:
scripts_reloaded: Emitted when scripts are reloaded from disk.
Any cached or already-injected scripts should be
considered obselete.
"""
scripts_reloaded = pyqtSignal()
# https://wiki.greasespot.net/Include_and_exclude_rules#Greaseable_schemes
# Limit the schemes scripts can run on due to unreasonable levels of
# exploitability
greaseable_schemes = ['http', 'https', 'ftp', 'file']
def __init__(self, parent=None):
super().__init__(parent)
self.load_scripts()
@cmdutils.register(name='greasemonkey-reload',
instance='greasemonkey')
def load_scripts(self):
"""Re-read Greasemonkey scripts from disk.
The scripts are read from a 'greasemonkey' subdirectory in
qutebrowser's data directory (see `:version`).
"""
self._run_start = []
self._run_end = []
self._run_idle = []
scripts_dir = os.path.abspath(_scripts_dir())
log.greasemonkey.debug("Reading scripts from: {}".format(scripts_dir))
for script_filename in glob.glob(os.path.join(scripts_dir, '*.js')):
if not os.path.isfile(script_filename):
continue
script_path = os.path.join(scripts_dir, script_filename)
with open(script_path, encoding='utf-8') as script_file:
script = GreasemonkeyScript.parse(script_file.read())
if not script.name:
script.name = script_filename
if script.run_at == 'document-start':
self._run_start.append(script)
elif script.run_at == 'document-end':
self._run_end.append(script)
elif script.run_at == 'document-idle':
self._run_idle.append(script)
else:
log.greasemonkey.warning("Script {} has invalid run-at "
"defined, defaulting to "
"document-end"
.format(script_path))
# Default as per
# https://wiki.greasespot.net/Metadata_Block#.40run-at
self._run_end.append(script)
log.greasemonkey.debug("Loaded script: {}".format(script.name))
self.scripts_reloaded.emit()
def scripts_for(self, url):
"""Fetch scripts that are registered to run for url.
returns a tuple of lists of scripts meant to run at (document-start,
document-end, document-idle)
"""
if url.scheme() not in self.greaseable_schemes:
return MatchingScripts(url, [], [], [])
match = functools.partial(fnmatch.fnmatch,
url.toString(QUrl.FullyEncoded))
tester = (lambda script:
any(match(pat) for pat in script.includes) and
not any(match(pat) for pat in script.excludes))
return MatchingScripts(
url,
[script for script in self._run_start if tester(script)],
[script for script in self._run_end if tester(script)],
[script for script in self._run_idle if tester(script)]
)
def all_scripts(self):
"""Return all scripts found in the configured script directory."""
return self._run_start + self._run_end + self._run_idle
def init():
"""Initialize Greasemonkey support."""
gm_manager = GreasemonkeyManager()
objreg.register('greasemonkey', gm_manager)
try:
os.mkdir(_scripts_dir())
except FileExistsError:
pass

13
qutebrowser/browser/qutescheme.py
View File

@ -29,6 +29,7 @@ import os
import time
import textwrap
import mimetypes
import urllib
import pkg_resources
from PyQt5.QtCore import QUrlQuery, QUrl
@ -425,6 +426,18 @@ def qute_settings(url):
return 'text/html', html
@add_handler('back')
def qute_back(url):
"""Handler for qute://back.
Simple page to free ram / lazy load a site, goes back on focusing the tab.
"""
html = jinja.render(
'back.html',
title='Suspended: ' + urllib.parse.unquote(url.fragment()))
return 'text/html', html
@add_handler('configdiff')
def qute_configdiff(url):
"""Handler for qute://configdiff."""

35
qutebrowser/browser/webengine/webenginesettings.py
View File

@ -244,6 +244,41 @@ def _init_profiles():
private_profile.setSpellCheckEnabled(True)
def inject_userscripts():
"""Register user JavaScript files with the global profiles."""
# The Greasemonkey metadata block support in QtWebEngine only starts at
# Qt 5.8. With 5.7.1, we need to inject the scripts ourselves in response
# to urlChanged.
if not qtutils.version_check('5.8'):
return
# Since we are inserting scripts into profile.scripts they won't
# just get replaced by new gm scripts like if we were injecting them
# ourselves so we need to remove all gm scripts, while not removing
# any other stuff that might have been added. Like the one for
# stylesheets.
greasemonkey = objreg.get('greasemonkey')
for profile in [default_profile, private_profile]:
scripts = profile.scripts()
for script in scripts.toList():
if script.name().startswith("GM-"):
log.greasemonkey.debug('Removing script: {}'
.format(script.name()))
removed = scripts.remove(script)
assert removed, script.name()
# Then add the new scripts.
for script in greasemonkey.all_scripts():
new_script = QWebEngineScript()
new_script.setWorldId(QWebEngineScript.MainWorld)
new_script.setSourceCode(script.code())
new_script.setName("GM-{}".format(script.name))
new_script.setRunsOnSubFrames(script.runs_on_sub_frames)
log.greasemonkey.debug('adding script: {}'
.format(new_script.name()))
scripts.insert(new_script)
def init(args):
"""Initialize the global QWebSettings."""
if args.enable_webengine_inspector:

23
qutebrowser/browser/webengine/webenginetab.py
View File

@ -69,6 +69,10 @@ def init():
download_manager.install(webenginesettings.private_profile)
objreg.register('webengine-download-manager', download_manager)
greasemonkey = objreg.get('greasemonkey')
greasemonkey.scripts_reloaded.connect(webenginesettings.inject_userscripts)
webenginesettings.inject_userscripts()
# Mapping worlds from usertypes.JsWorld to QWebEngineScript world IDs.
_JS_WORLD_MAP = {
@ -121,18 +125,35 @@ class WebEnginePrinting(browsertab.AbstractPrinting):
class WebEngineSearch(browsertab.AbstractSearch):
"""QtWebEngine implementations related to searching on the page."""
"""QtWebEngine implementations related to searching on the page.
Attributes:
_flags: The QWebEnginePage.FindFlags of the last search.
_pending_searches: How many searches have been started but not called
back yet.
"""
def __init__(self, parent=None):
super().__init__(parent)
self._flags = QWebEnginePage.FindFlags(0)
self._pending_searches = 0
def _find(self, text, flags, callback, caller):
"""Call findText on the widget."""
self.search_displayed = True
self._pending_searches += 1
def wrapped_callback(found):
"""Wrap the callback to do debug logging."""
self._pending_searches -= 1
if self._pending_searches > 0:
# See https://github.com/qutebrowser/qutebrowser/issues/2442
# and https://github.com/qt/qtwebengine/blob/5.10/src/core/web_contents_adapter.cpp#L924-L934
log.webview.debug("Ignoring cancelled search callback with "
"{} pending searches".format(
self._pending_searches))
return
found_text = 'found' if found else "didn't find"
if flags:
flag_text = 'with flags {}'.format(debug.qflags_key(

47
qutebrowser/browser/webengine/webview.py
View File

@ -23,12 +23,14 @@ import functools
from PyQt5.QtCore import pyqtSignal, pyqtSlot, QUrl, PYQT_VERSION
from PyQt5.QtGui import QPalette
from PyQt5.QtWebEngineWidgets import QWebEngineView, QWebEnginePage
from PyQt5.QtWebEngineWidgets import (QWebEngineView, QWebEnginePage,
QWebEngineScript)
from qutebrowser.browser import shared
from qutebrowser.browser.webengine import certificateerror, webenginesettings
from qutebrowser.config import config
from qutebrowser.utils import log, debug, usertypes, jinja, urlutils, message
from qutebrowser.utils import (log, debug, usertypes, jinja, urlutils, message,
objreg, qtutils)
class WebEngineView(QWebEngineView):
@ -135,6 +137,7 @@ class WebEnginePage(QWebEnginePage):
self._theme_color = theme_color
self._set_bg_color()
config.instance.changed.connect(self._set_bg_color)
self.urlChanged.connect(self._inject_userjs)
@config.change_filter('colors.webpage.bg')
def _set_bg_color(self):
@ -300,3 +303,43 @@ class WebEnginePage(QWebEnginePage):
message.error(msg)
return False
return True
@pyqtSlot('QUrl')
def _inject_userjs(self, url):
"""Inject userscripts registered for `url` into the current page."""
if qtutils.version_check('5.8'):
# Handled in webenginetab with the builtin Greasemonkey
# support.
return
# Using QWebEnginePage.scripts() to hold the user scripts means
# we don't have to worry ourselves about where to inject the
# page but also means scripts hang around for the tab lifecycle.
# So clear them here.
scripts = self.scripts()
for script in scripts.toList():
if script.name().startswith("GM-"):
log.greasemonkey.debug("Removing script: {}"
.format(script.name()))
removed = scripts.remove(script)
assert removed, script.name()
def _add_script(script, injection_point):
new_script = QWebEngineScript()
new_script.setInjectionPoint(injection_point)
new_script.setWorldId(QWebEngineScript.MainWorld)
new_script.setSourceCode(script.code())
new_script.setName("GM-{}".format(script.name))
new_script.setRunsOnSubFrames(script.runs_on_sub_frames)
log.greasemonkey.debug("Adding script: {}"
.format(new_script.name()))
scripts.insert(new_script)
greasemonkey = objreg.get('greasemonkey')
matching_scripts = greasemonkey.scripts_for(url)
for script in matching_scripts.start:
_add_script(script, QWebEngineScript.DocumentCreation)
for script in matching_scripts.end:
_add_script(script, QWebEngineScript.DocumentReady)
for script in matching_scripts.idle:
_add_script(script, QWebEngineScript.Deferred)

47
qutebrowser/browser/webkit/webpage.py
View File

@ -86,6 +86,21 @@ class BrowserPage(QWebPage):
self.on_save_frame_state_requested)
self.restoreFrameStateRequested.connect(
self.on_restore_frame_state_requested)
self.loadFinished.connect(
functools.partial(self._inject_userjs, self.mainFrame()))
self.frameCreated.connect(self._connect_userjs_signals)
@pyqtSlot('QWebFrame*')
def _connect_userjs_signals(self, frame):
"""Connect userjs related signals to `frame`.
Connect the signals used as triggers for injecting user
JavaScripts into the passed QWebFrame.
"""
log.greasemonkey.debug("Connecting to frame {} ({})"
.format(frame, frame.url().toDisplayString()))
frame.loadFinished.connect(
functools.partial(self._inject_userjs, frame))
def javaScriptPrompt(self, frame, js_msg, default):
"""Override javaScriptPrompt to use qutebrowser prompts."""
@ -283,6 +298,38 @@ class BrowserPage(QWebPage):
else:
self.error_occurred = False
def _inject_userjs(self, frame):
"""Inject user JavaScripts into the page.
Args:
frame: The QWebFrame to inject the user scripts into.
"""
url = frame.url()
if url.isEmpty():
url = frame.requestedUrl()
log.greasemonkey.debug("_inject_userjs called for {} ({})"
.format(frame, url.toDisplayString()))
greasemonkey = objreg.get('greasemonkey')
scripts = greasemonkey.scripts_for(url)
# QtWebKit has trouble providing us with signals representing
# page load progress at reasonable times, so we just load all
# scripts on the same event.
toload = scripts.start + scripts.end + scripts.idle
if url.isEmpty():
# This happens during normal usage like with view source but may
# also indicate a bug.
log.greasemonkey.debug("Not running scripts for frame with no "
"url: {}".format(frame))
assert not toload, toload
for script in toload:
if frame is self.mainFrame() or script.runs_on_sub_frames:
log.webview.debug('Running GM script: {}'.format(script.name))
frame.evaluateJavaScript(script.code())
@pyqtSlot('QWebFrame*', 'QWebPage::Feature')
def _on_feature_permission_requested(self, frame, feature):
"""Ask the user for approval for geolocation/notifications."""

2
qutebrowser/completion/models/histcategory.py
View File

@ -47,7 +47,7 @@ class HistoryCategory(QSqlQueryModel):
"FROM CompletionHistory",
# the incoming pattern will have literal % and _ escaped with '\'
# we need to tell sql to treat '\' as an escape character
"WHERE (url LIKE :pat escape '\\' or title LIKE :pat escape '\\')",
"WHERE ((url || title) LIKE :pat escape '\\')",
self._atime_expr(),
"ORDER BY last_atime DESC",
]), forward_only=False)

8
qutebrowser/config/configdata.yml
View File

@ -79,6 +79,9 @@ new_instance_open_target_window:
When `new_instance_open_target` is not set to `window`, this is ignored.
session_default_name:
renamed: session.default_name
session.default_name:
type:
name: SessionName
none_ok: true
@ -88,6 +91,11 @@ session_default_name:
If this is set to null, the session which was last loaded is saved.
session.lazy_restore:
type: Bool
default: false
desc: Load a restored tab as soon as it takes focus.
backend:
type:
name: String

60
qutebrowser/html/back.html Normal file
View File

@ -0,0 +1,60 @@
{% extends "base.html" %}
{% block script %}
const STATE_BACK = "back";
const STATE_FORWARD = "forward";
function switch_state(new_state) {
history.replaceState(
new_state,
document.title,
location.pathname + location.hash);
}
function go_back() {
switch_state(STATE_FORWARD);
history.back();
}
function go_forward() {
switch_state(STATE_BACK);
history.forward();
}
function prepare_restore() {
if (!document.hidden) {
go_back();
return;
}
document.addEventListener("visibilitychange", go_back);
}
// there are three states
// default: register focus listener,
// on focus: go back and switch to the state forward
// back: user came from a later history entry
// -> switch to the state forward,
// forward him to the previous history entry
// forward: user came from a previous history entry
// -> switch to the state back,
// forward him to the next history entry
switch (history.state) {
case STATE_BACK:
go_back();
break;
case STATE_FORWARD:
go_forward();
break;
default:
setTimeout(prepare_restore, 1000);
break;
}
{% endblock %}
{% block content %}
<noscript><p>Javascript isn't enabled. So you need to manually go back in history to restore this tab.</p></noscript>
<p>Loading suspended page...<br>
<br>
If nothing happens, something went wrong or you disabled JavaScript.</p>
{% endblock %}

2
qutebrowser/javascript/.eslintignore
View File

@ -1,2 +1,4 @@
# Upstream Mozilla's code
pac_utils.js
# Actually a jinja template so eslint chokes on the {{}} syntax.
greasemonkey_wrapper.js

118
qutebrowser/javascript/greasemonkey_wrapper.js Normal file
View File

@ -0,0 +1,118 @@
(function() {
const _qute_script_id = "__gm_" + {{ scriptName | tojson }};
function GM_log(text) {
console.log(text);
}
const GM_info = {
'script': {{ scriptInfo }},
'scriptMetaStr': {{ scriptMeta | tojson }},
'scriptWillUpdate': false,
'version': "0.0.1",
// so scripts don't expect exportFunction
'scriptHandler': 'Tampermonkey',
};
function checkKey(key, funcName) {
if (typeof key !== "string") {
throw new Error(`${funcName} requires the first parameter to be of type string, not '${typeof key}'`);
}
}
function GM_setValue(key, value) {
checkKey(key, "GM_setValue");
if (typeof value !== "string" &&
typeof value !== "number" &&
typeof value !== "boolean") {
throw new Error(`GM_setValue requires the second parameter to be of type string, number or boolean, not '${typeof value}'`);
}
localStorage.setItem(_qute_script_id + key, value);
}
function GM_getValue(key, default_) {
checkKey(key, "GM_getValue");
return localStorage.getItem(_qute_script_id + key) || default_;
}
function GM_deleteValue(key) {
checkKey(key, "GM_deleteValue");
localStorage.removeItem(_qute_script_id + key);
}
function GM_listValues() {
const keys = [];
for (let i = 0; i < localStorage.length; i++) {
if (localStorage.key(i).startsWith(_qute_script_id)) {
keys.push(localStorage.key(i).slice(_qute_script_id.length));
}
}
return keys;
}
function GM_openInTab(url) {
window.open(url);
}
// Almost verbatim copy from Eric
function GM_xmlhttpRequest(/* object */ details) {
details.method = details.method ? details.method.toUpperCase() : "GET";
if (!details.url) {
throw new Error("GM_xmlhttpRequest requires an URL.");
}
// build XMLHttpRequest object
const oXhr = new XMLHttpRequest();
// run it
if ("onreadystatechange" in details) {
oXhr.onreadystatechange = function() {
details.onreadystatechange(oXhr);
};
}
if ("onload" in details) {
oXhr.onload = function() { details.onload(oXhr); };
}
if ("onerror" in details) {
oXhr.onerror = function () { details.onerror(oXhr); };
}
oXhr.open(details.method, details.url, true);
if ("headers" in details) {
for (const header in details.headers) {
oXhr.setRequestHeader(header, details.headers[header]);
}
}
if ("data" in details) {
oXhr.send(details.data);
} else {
oXhr.send();
}
}
function GM_addStyle(/* String */ styles) {
const oStyle = document.createElement("style");
oStyle.setAttribute("type", "text/css");
oStyle.appendChild(document.createTextNode(styles));
const head = document.getElementsByTagName("head")[0];
if (head === undefined) {
document.onreadystatechange = function() {
if (document.readyState === "interactive") {
document.getElementsByTagName("head")[0].appendChild(oStyle);
}
};
} else {
head.appendChild(oStyle);
}
}
const unsafeWindow = window;
// ====== The actual user script source ====== //
{{ scriptSource }}
// ====== End User Script ====== //
})();

44
qutebrowser/misc/sessions.py
View File

@ -21,6 +21,8 @@
import os
import os.path
import itertools
import urllib
import sip
from PyQt5.QtCore import QUrl, QObject, QPoint, QTimer
@ -205,7 +207,13 @@ class SessionManager(QObject):
for idx, item in enumerate(tab.history):
qtutils.ensure_valid(item)
item_data = self._save_tab_item(tab, idx, item)
data['history'].append(item_data)
if item.url().scheme() == 'qute' and item.url().host() == 'back':
# don't add qute://back to the session file
if item_data.get('active', False) and data['history']:
# mark entry before qute://back as active
data['history'][-1]['active'] = True
else:
data['history'].append(item_data)
return data
def _save_all(self, *, only_window=None, with_private=False):
@ -251,7 +259,7 @@ class SessionManager(QObject):
object.
"""
if name is default:
name = config.val.session_default_name
name = config.val.session.default_name
if name is None:
if self._current is not None:
name = self._current
@ -323,7 +331,18 @@ class SessionManager(QObject):
def _load_tab(self, new_tab, data):
"""Load yaml data into a newly opened tab."""
entries = []
for histentry in data['history']:
lazy_load = []
# use len(data['history'])
# -> dropwhile empty if not session.lazy_session
lazy_index = len(data['history'])
gen = itertools.chain(
itertools.takewhile(lambda _: not lazy_load,
enumerate(data['history'])),
enumerate(lazy_load),
itertools.dropwhile(lambda i: i[0] < lazy_index,
enumerate(data['history'])))
for i, histentry in gen:
user_data = {}
if 'zoom' in data:
@ -347,6 +366,20 @@ class SessionManager(QObject):
if 'pinned' in histentry:
new_tab.data.pinned = histentry['pinned']
if (config.val.session.lazy_restore and
histentry.get('active', False) and
not histentry['url'].startswith('qute://back')):
# remove "active" mark and insert back page marked as active
lazy_index = i + 1
lazy_load.append({
'title': histentry['title'],
'url':
'qute://back#' +
urllib.parse.quote(histentry['title']),
'active': True
})
histentry['active'] = False
active = histentry.get('active', False)
url = QUrl.fromEncoded(histentry['url'].encode('ascii'))
if 'original-url' in histentry:
@ -360,6 +393,7 @@ class SessionManager(QObject):
entries.append(entry)
if active:
new_tab.title_changed.emit(histentry['title'])
try:
new_tab.history.load_items(entries)
except ValueError as e:
@ -388,7 +422,7 @@ class SessionManager(QObject):
window=window.win_id)
tab_to_focus = None
for i, tab in enumerate(win['tabs']):
new_tab = tabbed_browser.tabopen()
new_tab = tabbed_browser.tabopen(background=False)
self._load_tab(new_tab, tab)
if tab.get('active', False):
tab_to_focus = i
@ -460,7 +494,7 @@ class SessionManager(QObject):
Args:
name: The name of the session. If not given, the session configured
in session_default_name is saved.
in session.default_name is saved.
current: Save the current session instead of the default.
quiet: Don't show confirmation message.
force: Force saving internal sessions (starting with an underline).

1
qutebrowser/utils/jinja.py
View File

@ -136,3 +136,4 @@ def render(template, **kwargs):
environment = Environment()
js_environment = jinja2.Environment(loader=Loader('javascript'))

4
qutebrowser/utils/log.py
View File

@ -95,7 +95,8 @@ LOGGER_NAMES = [
'commands', 'signals', 'downloads',
'js', 'qt', 'rfc6266', 'ipc', 'shlexer',
'save', 'message', 'config', 'sessions',
'webelem', 'prompt', 'network', 'sql'
'webelem', 'prompt', 'network', 'sql',
'greasemonkey'
]
@ -144,6 +145,7 @@ webelem = logging.getLogger('webelem')
prompt = logging.getLogger('prompt')
network = logging.getLogger('network')
sql = logging.getLogger('sql')
greasemonkey = logging.getLogger('greasemonkey')
ram_handler = None

4
scripts/dev/ci/travis_backtrace.sh
View File

@ -6,7 +6,7 @@
case $TESTENV in
py3*-pyqt*)
exe=$(readlink -f .tox/$TESTENV/bin/python)
exe=$(readlink -f ".tox/$TESTENV/bin/python")
full=
;;
*)
@ -15,4 +15,4 @@ case $TESTENV in
;;
esac
find . -name *.core -o -name core -exec gdb --batch --quiet -ex "thread apply all bt $full" "$exe" {} \;
find . \( -name "*.core" -o -name core \) -exec gdb --batch --quiet -ex "thread apply all bt $full" "$exe" {} \;

12
scripts/dev/ci/travis_install.sh
View File

@ -21,23 +21,23 @@
# Stolen from https://github.com/travis-ci/travis-build/blob/master/lib/travis/build/templates/header.sh
# and adjusted to use ((...))
travis_retry() {
local ANSI_RED="\033[31;1m"
local ANSI_RESET="\033[0m"
local ANSI_RED='\033[31;1m'
local ANSI_RESET='\033[0m'
local result=0
local count=1
while (( count < 3 )); do
if (( result != 0 )); then
echo -e "\n${ANSI_RED}The command \"$@\" failed. Retrying, $count of 3.${ANSI_RESET}\n" >&2
echo -e "\\n${ANSI_RED}The command \"$*\" failed. Retrying, $count of 3.${ANSI_RESET}\\n" >&2
fi
"$@"
result=$?
(( result == 0 )) && break
count=$(($count + 1))
count=$(( count + 1 ))
sleep 1
done
if (( count > 3 )); then
echo -e "\n${ANSI_RED}The command \"$@\" failed 3 times.${ANSI_RESET}\n" >&2
echo -e "\\n${ANSI_RED}The command \"$*\" failed 3 times.${ANSI_RESET}\\n" >&2
fi
return $result
@ -96,6 +96,8 @@ case $TESTENV in
eslint)
npm_install eslint
;;
shellcheck)
;;
*)
pip_install pip
pip_install -r misc/requirements/requirements-tox.txt

10
scripts/dev/ci/travis_run.sh
View File

@ -14,6 +14,16 @@ elif [[ $TESTENV == eslint ]]; then
# travis env
cd qutebrowser/javascript || exit 1
eslint --color --report-unused-disable-directives .
elif [[ $TESTENV == shellcheck ]]; then
SCRIPTS=$( mktemp )
find scripts/dev/ -name '*.sh' >"$SCRIPTS"
find misc/userscripts/ -type f -exec grep -lE '[/ ][bd]ash$|[/ ]sh$|[/ ]ksh$' {} + >>"$SCRIPTS"
mapfile -t scripts <"$SCRIPTS"
rm -f "$SCRIPTS"
docker run \
-v "$PWD:/outside" \
-w /outside \
koalaman/shellcheck:latest "${scripts[@]}"
else
args=()
[[ $TRAVIS_OS_NAME == osx ]] && args=('--qute-bdd-webengine' '--no-xvfb')

17
scripts/dev/download_release.sh
View File

@ -1,4 +1,5 @@
#!/bin/bash
set -e
# This script downloads the given release from GitHub so we can mirror it on
# qutebrowser.org.
@ -16,16 +17,16 @@ mkdir windows
base="https://github.com/qutebrowser/qutebrowser/releases/download/v$1"
wget "$base/qutebrowser-$1.tar.gz" || exit 1
wget "$base/qutebrowser-$1.tar.gz.asc" || exit 1
wget "$base/qutebrowser-$1.dmg" || exit 1
wget "$base/qutebrowser_${1}-1_all.deb" || exit 1
wget "$base/qutebrowser-$1.tar.gz"
wget "$base/qutebrowser-$1.tar.gz.asc"
wget "$base/qutebrowser-$1.dmg"
wget "$base/qutebrowser_${1}-1_all.deb"
cd windows
wget "$base/qutebrowser-${1}-amd64.msi" || exit 1
wget "$base/qutebrowser-${1}-win32.msi" || exit 1
wget "$base/qutebrowser-${1}-windows-standalone-amd64.zip" || exit 1
wget "$base/qutebrowser-${1}-windows-standalone-win32.zip" || exit 1
wget "$base/qutebrowser-${1}-amd64.msi"
wget "$base/qutebrowser-${1}-win32.msi"
wget "$base/qutebrowser-${1}-windows-standalone-amd64.zip"
wget "$base/qutebrowser-${1}-windows-standalone-win32.zip"
dest="/srv/http/qutebrowser/releases/v$1"
cd "$oldpwd"

10
scripts/dev/quit_segfault_test.sh
View File

@ -1,14 +1,12 @@
#!/bin/bash
#!/usr/bin/env bash
if [[ $PWD == */scripts ]]; then
cd ..
fi
[[ $PWD == */scripts ]] && cd ..
echo > crash.log
while :; do
exit=0
while (( $exit == 0)); do
duration=$(($RANDOM%10000))
while (( exit == 0 )); do
duration=$(( RANDOM % 10000 ))
python3 -m qutebrowser --debug ":later $duration quit" http://www.heise.de/
exit=$?
done

154
scripts/hist_importer.py Executable file
View File

@ -0,0 +1,154 @@
#!/usr/bin/env python3
# vim: ft=python fileencoding=utf-8 sts=4 sw=4 et:
# Copyright 2017 Florian Bruhin (The Compiler) <mail@qutebrowser.org>
# Copyright 2017 Josefson Souza <josefson.br@gmail.com>
# This file is part of qutebrowser.
#
# qutebrowser is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# qutebrowser is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with qutebrowser. If not, see <http://www.gnu.org/licenses/>.
"""Tool to import browser history from other browsers."""
import argparse
import sqlite3
import sys
import os
def parse():
"""Parse command line arguments."""
description = ("This program is meant to extract browser history from your"
" previous browser and import them into qutebrowser.")
epilog = ("Databases:\n\n\tqutebrowser: Is named 'history.sqlite' and can "
"be found at your --basedir. In order to find where your "
"basedir is you can run ':open qute:version' inside qutebrowser."
"\n\n\tFirefox: Is named 'places.sqlite', and can be found at "
"your system's profile folder. Check this link for where it is "
"located: http://kb.mozillazine.org/Profile_folder"
"\n\n\tChrome: Is named 'History', and can be found at the "
"respective User Data Directory. Check this link for where it is"
"located: https://chromium.googlesource.com/chromium/src/+/"
"master/docs/user_data_dir.md\n\n"
"Example: hist_importer.py -b firefox -s /Firefox/Profile/"
"places.sqlite -d /qutebrowser/data/history.sqlite")
parser = argparse.ArgumentParser(
description=description, epilog=epilog,
formatter_class=argparse.RawTextHelpFormatter
)
parser.add_argument('-b', '--browser', dest='browser', required=True,
type=str, help='Browsers: {firefox, chrome}')
parser.add_argument('-s', '--source', dest='source', required=True,
type=str, help='Source: Full path to the sqlite data'
'base file from the source browser.')
parser.add_argument('-d', '--dest', dest='dest', required=True, type=str,
help='\nDestination: Full path to the qutebrowser '
'sqlite database')
return parser.parse_args()
def open_db(data_base):
"""Open connection with database."""
if os.path.isfile(data_base):
conn = sqlite3.connect(data_base)
return conn
else:
sys.exit('The file {} does not exist.'.format(data_base))
def extract(source, query):
"""Get records from source database.
Args:
source: File path to the source database where we want to extract the
data from.
query: The query string to be executed in order to retrieve relevant
attributes as (datetime, url, time) from the source database according
to the browser chosen.
"""
try:
conn = open_db(source)
cursor = conn.cursor()
cursor.execute(query)
history = cursor.fetchall()
conn.close()
return history
except sqlite3.OperationalError as op_e:
sys.exit('Could not perform queries on the source database: '
'{}'.format(op_e))
def clean(history):
"""Clean up records from source database.
Receives a list of record and sanityze them in order for them to be
properly imported to qutebrowser. Sanitation requires addiing a 4th
attribute 'redirect' which is filled with '0's, and also purging all
records that have a NULL/None datetime attribute.
Args:
history: List of records (datetime, url, title) from source database.
"""
nulls = [record for record in history if record[0] is None]
for null_datetime in nulls:
history.remove(null_datetime)
history = [list(record) for record in history]
for record in history:
record.append('0')
return history
def insert_qb(history, dest):
"""Insert history into dest database.
Args:
history: List of records.
dest: File path to the destination database, where history will be
inserted.
"""
conn = open_db(dest)
cursor = conn.cursor()
cursor.executemany(
'INSERT INTO History (url,title,atime,redirect) VALUES (?,?,?,?)',
history
)
cursor.execute('DROP TABLE CompletionHistory')
conn.commit()
conn.close()
def main():
"""Main control flux of the script."""
args = parse()
browser = args.browser.lower()
source, dest = args.source, args.dest
query = {
'firefox': 'select url,title,last_visit_date/1000000 as date '
'from moz_places',
'chrome': 'select url,title,last_visit_time/10000000 as date '
'from urls',
}
if browser not in query:
sys.exit('Sorry, the selected browser: "{}" is not supported.'.format(
browser))
else:
history = extract(source, query[browser])
history = clean(history)
insert_qb(history, dest)
if __name__ == "__main__":
main()

2
tests/end2end/features/downloads.feature
View File

@ -536,7 +536,7 @@ Feature: Downloading things from a website.
And I open data/downloads/download.bin without waiting
And I wait for the download prompt for "*"
And I run :prompt-accept (tmpdir)(dirsep)downloads
And I open data/downloads/download.bin without waiting
And I open data/downloads/download2.bin without waiting
And I wait for the download prompt for "*"
And I directly open the download
And I open data/downloads/download.bin without waiting

3
tests/end2end/features/editor.feature
View File

@ -116,7 +116,8 @@ Feature: Opening external editors
Then the javascript message "text: foobar" should be logged
# Could not get signals working on Windows
@posix
# There's no guarantee that the tab gets deleted...
@posix @flaky
Scenario: Spawning an editor and closing the tab
When I set up a fake editor that waits
And I open data/editor.html

22
tests/end2end/features/javascript.feature
View File

@ -123,3 +123,25 @@ Feature: Javascript stuff
And I wait for "[*/data/javascript/windowsize.html:*] loaded" in the log
And I run :tab-next
Then the window sizes should be the same
Scenario: Have a GreaseMonkey script run at page start
When I have a GreaseMonkey file saved for document-start with noframes unset
And I run :greasemonkey-reload
And I open data/hints/iframe.html
# This second reload is required in webengine < 5.8 for scripts
# registered to run at document-start, some sort of timing issue.
And I run :reload
Then the javascript message "Script is running on /data/hints/iframe.html" should be logged
Scenario: Have a GreaseMonkey script running on frames
When I have a GreaseMonkey file saved for document-end with noframes unset
And I run :greasemonkey-reload
And I open data/hints/iframe.html
Then the javascript message "Script is running on /data/hints/html/wrapped.html" should be logged
@flaky
Scenario: Have a GreaseMonkey script running on noframes
When I have a GreaseMonkey file saved for document-end with noframes set
And I run :greasemonkey-reload
And I open data/hints/iframe.html
Then the javascript message "Script is running on /data/hints/html/wrapped.html" should not be logged

2
tests/end2end/features/qutescheme.feature
View File

@ -113,6 +113,8 @@ Feature: Special qute:// pages
And I wait for "Config option changed: ignore_case *" in the log
Then the option ignore_case should be set to always
# Sometimes, an unrelated value gets set
@flaky
Scenario: Focusing input fields in qute://settings and entering invalid value
When I open qute://settings
# scroll to the right - the table does not fit in the default screen

4
tests/end2end/features/search.feature
View File

@ -225,11 +225,15 @@ Feature: Searching on a page
Then the following tabs should be open:
- data/search.html (active)
# Following a link selected via JS doesn't work in Qt 5.10 anymore.
@qt!=5.10
Scenario: Follow a manually selected link
When I run :jseval --file (testdata)/search_select.js
And I run :follow-selected
Then data/hello.txt should be loaded
# Following a link selected via JS doesn't work in Qt 5.10 anymore.
@qt!=5.10
Scenario: Follow a manually selected link in a new tab
When I run :window-only
And I run :jseval --file (testdata)/search_select.js

37
tests/end2end/features/test_javascript_bdd.py
View File

@ -17,6 +17,8 @@
# You should have received a copy of the GNU General Public License
# along with qutebrowser. If not, see <http://www.gnu.org/licenses/>.
import os.path
import pytest_bdd as bdd
bdd.scenarios('javascript.feature')
@ -29,3 +31,38 @@ def check_window_sizes(quteproc):
hidden_size = hidden.message.split()[-1]
visible_size = visible.message.split()[-1]
assert hidden_size == visible_size
test_gm_script = r"""
// ==UserScript==
// @name qutebrowser test userscript
// @namespace invalid.org
// @include http://localhost:*/data/hints/iframe.html
// @include http://localhost:*/data/hints/html/wrapped.html
// @exclude ???
// @run-at {stage}
// {frames}
// ==/UserScript==
console.log("Script is running on " + window.location.pathname);
"""
@bdd.when(bdd.parsers.parse("I have a GreaseMonkey file saved for {stage} "
"with noframes {frameset}"))
def create_greasemonkey_file(quteproc, stage, frameset):
script_path = os.path.join(quteproc.basedir, 'data', 'greasemonkey')
try:
os.mkdir(script_path)
except FileExistsError:
pass
file_path = os.path.join(script_path, 'test.user.js')
if frameset == "set":
frames = "@noframes"
elif frameset == "unset":
frames = ""
else:
raise ValueError("noframes can only be set or unset, "
"not {}".format(frameset))
with open(file_path, 'w', encoding='utf-8') as f:
f.write(test_gm_script.format(stage=stage,
frames=frames))

8
tests/end2end/fixtures/quteprocess.py
View File

@ -36,7 +36,7 @@ import pytest
from PyQt5.QtCore import pyqtSignal, QUrl, qVersion
from qutebrowser.misc import ipc
from qutebrowser.utils import log, utils, javascript
from qutebrowser.utils import log, utils, javascript, qtutils
from helpers import utils as testutils
from end2end.fixtures import testprocess
@ -527,6 +527,7 @@ class QuteProc(testprocess.Process):
super().before_test()
self.send_cmd(':config-clear')
self._init_settings()
self.clear_data()
def _init_settings(self):
"""Adjust some qutebrowser settings after starting."""
@ -687,9 +688,12 @@ class QuteProc(testprocess.Process):
raise ValueError("Invalid URL {}: {}".format(url,
qurl.errorString()))
if qurl == QUrl('about:blank'):
if (qurl == QUrl('about:blank') and
not qtutils.version_check('5.10', compiled=False)):
# For some reason, we don't get a LoadStatus.success for
# about:blank sometimes.
# However, if we do this for Qt 5.10, we get general testsuite
# instability as site loads get reported with about:blank...
pattern = "Changing title for idx * to 'about:blank'"
else:
# We really need the same representation that the webview uses in

20
tests/end2end/fixtures/testprocess.py
View File

@ -333,7 +333,7 @@ class Process(QObject):
else:
return value == expected
def _wait_for_existing(self, override_waited_for, **kwargs):
def _wait_for_existing(self, override_waited_for, after, **kwargs):
"""Check if there are any line in the history for wait_for.
Return: either the found line or None.
@ -345,7 +345,15 @@ class Process(QObject):
value = getattr(line, key)
matches.append(self._match_data(value, expected))
if all(matches) and (not line.waited_for or override_waited_for):
if after is None:
too_early = False
else:
too_early = ((line.timestamp, line.msecs) <
(after.timestamp, after.msecs))
if (all(matches) and
(not line.waited_for or override_waited_for) and
not too_early):
# If we waited for this line, chances are we don't mean the
# same thing the next time we use wait_for and it matches
# this line again.
@ -363,7 +371,7 @@ class Process(QObject):
__tracebackhide__ = lambda e: e.errisinstance(WaitForTimeout)
message = kwargs.get('message', None)
if message is not None:
elided = quteutils.elide(repr(message), 50)
elided = quteutils.elide(repr(message), 100)
self._log("\n----> Waiting for {} in the log".format(elided))
spy = QSignalSpy(self.new_data)
@ -422,7 +430,7 @@ class Process(QObject):
pass
def wait_for(self, timeout=None, *, override_waited_for=False,
do_skip=False, divisor=1, **kwargs):
do_skip=False, divisor=1, after=None, **kwargs):
"""Wait until a given value is found in the data.
Keyword arguments to this function get interpreted as attributes of the
@ -435,6 +443,7 @@ class Process(QObject):
again.
do_skip: If set, call pytest.skip on a timeout.
divisor: A factor to decrease the timeout by.
after: If it's an existing line, ensure it's after the given one.
Return:
The matched line.
@ -456,7 +465,8 @@ class Process(QObject):
for key in kwargs:
assert key in self.KEYS
existing = self._wait_for_existing(override_waited_for, **kwargs)
existing = self._wait_for_existing(override_waited_for, after,
**kwargs)
if existing is not None:
return existing
else:

4
tests/unit/completion/test_histcategory.py
View File

@ -78,6 +78,10 @@ def hist(init_sql, config_stub):
("can't",
[("can't touch this", ''), ('a', '')],
[("can't touch this", '')]),
("ample itle",
[('example.com', 'title'), ('example.com', 'nope')],
[('example.com', 'title')]),
])
def test_set_pattern(pattern, before, after, model_validator, hist):
"""Validate the filtering and sorting results of set_pattern."""

104
tests/unit/javascript/test_greasemonkey.py Normal file
View File

@ -0,0 +1,104 @@
# vim: ft=python fileencoding=utf-8 sts=4 sw=4 et:
# Copyright 2017 Florian Bruhin (The Compiler) <mail@qutebrowser.org>
# This file is part of qutebrowser.
#
# qutebrowser is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# qutebrowser is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with qutebrowser. If not, see <http://www.gnu.org/licenses/>.
"""Tests for qutebrowser.browser.greasemonkey."""
import logging
import pytest
import py.path # pylint: disable=no-name-in-module
from PyQt5.QtCore import QUrl
from qutebrowser.browser import greasemonkey
test_gm_script = """
// ==UserScript==
// @name qutebrowser test userscript
// @namespace invalid.org
// @include http://localhost:*/data/title.html
// @match http://trolol*
// @exclude https://badhost.xxx/*
// @run-at document-start
// ==/UserScript==
console.log("Script is running.");
"""
pytestmark = pytest.mark.usefixtures('data_tmpdir')
def _save_script(script_text, filename):
# pylint: disable=no-member
file_path = py.path.local(greasemonkey._scripts_dir()) / filename
# pylint: enable=no-member
file_path.write_text(script_text, encoding='utf-8', ensure=True)
def test_all():
"""Test that a script gets read from file, parsed and returned."""
_save_script(test_gm_script, 'test.user.js')
gm_manager = greasemonkey.GreasemonkeyManager()
assert (gm_manager.all_scripts()[0].name ==
"qutebrowser test userscript")
@pytest.mark.parametrize("url, expected_matches", [
# included
('http://trololololololo.com/', 1),
# neither included nor excluded
('http://aaaaaaaaaa.com/', 0),
# excluded
('https://badhost.xxx/', 0),
])
def test_get_scripts_by_url(url, expected_matches):
"""Check Greasemonkey include/exclude rules work."""
_save_script(test_gm_script, 'test.user.js')
gm_manager = greasemonkey.GreasemonkeyManager()
scripts = gm_manager.scripts_for(QUrl(url))
assert (len(scripts.start + scripts.end + scripts.idle) ==
expected_matches)
def test_no_metadata(caplog):
"""Run on all sites at document-end is the default."""
_save_script("var nothing = true;\n", 'nothing.user.js')
with caplog.at_level(logging.WARNING):
gm_manager = greasemonkey.GreasemonkeyManager()
scripts = gm_manager.scripts_for(QUrl('http://notamatch.invalid/'))
assert len(scripts.start + scripts.end + scripts.idle) == 1
assert len(scripts.end) == 1
def test_bad_scheme(caplog):
"""qute:// isn't in the list of allowed schemes."""
_save_script("var nothing = true;\n", 'nothing.user.js')
with caplog.at_level(logging.WARNING):
gm_manager = greasemonkey.GreasemonkeyManager()
scripts = gm_manager.scripts_for(QUrl('qute://settings'))
assert len(scripts.start + scripts.end + scripts.idle) == 0
def test_load_emits_signal(qtbot):
gm_manager = greasemonkey.GreasemonkeyManager()
with qtbot.wait_signal(gm_manager.scripts_reloaded):
gm_manager.load_scripts()

2
tests/unit/misc/test_sessions.py
View File

@ -170,7 +170,7 @@ class TestSaveAll:
])
def test_get_session_name(config_stub, sess_man, arg, config, current,
expected):
config_stub.val.session_default_name = config
config_stub.val.session.default_name = config
sess_man._current = current
assert sess_man._get_session_name(arg) == expected

1
tox.ini
View File

@ -181,6 +181,7 @@ commands =
[testenv:eslint]
# This is duplicated in travis_run.sh for Travis CI because we can't get tox in
# the JavaScript environment easily.
basepython = python3
deps =
whitelist_externals = eslint
changedir = {toxinidir}/qutebrowser/javascript