./updateHostsFile.py --auto crashes when run under path contains CJK characters using Python 2.7.12

Console Output
> python updateHostsFile.py --auto
Traceback (most recent call last):
  File "updateHostsFile.py", line 790, in <module>
    main()
  File "updateHostsFile.py", line 147, in main
    options["outputsubfolder"])
  File "updateHostsFile.py", line 765, in path_join_robust
    "likely a LOCALE issue:\n\n" + str(e))
locale.Error: Unable to construct path. This is likely a LOCALE issue:

'ascii' codec can't decode byte 0xe5 in position 18: ordinal not in range(128)

Reporter's Environment
Working Directory
/home/<username>/工作空間/第三方專案/Unified hosts file with base extensions - Extending and consolidating hosts files from several well-curated sources like adaway.org, mvps.org, malwaredomainlist.com, someonewhocares.org, and potentially others.

Operating System
KDE Neon(based on Ubuntu 16.04 AMD64)
Python
Python 2.7.12
Unified hosts file with base extensions
commit 231dc436
Locale
LANG=zh_TW.UTF-8
LANGUAGE=zh_TW:zh_CN
LC_CTYPE="zh_TW.UTF-8"
LC_NUMERIC="zh_TW.UTF-8"
LC_TIME="zh_TW.UTF-8"
LC_COLLATE="zh_TW.UTF-8"
LC_MONETARY="zh_TW.UTF-8"
LC_MESSAGES="zh_TW.UTF-8"
LC_PAPER="zh_TW.UTF-8"
LC_NAME="zh_TW.UTF-8"
LC_ADDRESS="zh_TW.UTF-8"
LC_TELEPHONE="zh_TW.UTF-8"
LC_MEASUREMENT="zh_TW.UTF-8"
LC_IDENTIFICATION="zh_TW.UTF-8"
LC_ALL=

@Vdragon : Are you suggesting that we have a help message for our scripts?  They don't have one, which is why the --help flag is being ignored.  Afterwards, updateHostsFile.py is called, leading to the failure you're seeing here.

I just noticed the suspicious file in the same directory and tried to see what it can do...
Anyway IMO updateHostsFile.py shouldn't crash even when it is called in a wrong way

@Vdragon : It's not a "wrong" way.  However, it's because you have a LOCALE issue that's preventing the script from running.  I suspect it's because you are using Python 2.x instead of Python 3.x, which will handle the CJK characters MUCH BETTER than Python 2.x can.

I've updated the issue as it can be simplified, unless we no longer support Python 2 it's still a valid issue

It's more that it's an issue with Python 2.x that's making this very difficult to execute.  Try running the following commands FROM the directory of updateHostsFile.py in the Python interpreter:
>>> import os
>>> current_dir = os.getcwd()
>>> os.path.join(current_dir, "data")

What happens?

> python
Python 2.7.12 (default, Nov 19 2016, 06:48:10)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> current_dir = os.getcwd()
>>> os.path.join(current_dir, "data")
'/home/Lin-Buo-Ren/\xe5\xb7\xa5\xe4\xbd\x9c\xe7\xa9\xba\xe9\x96\x93/\xe7\xac\xac\xe4\xb8\x89\xe6\x96\xb9\xe5\xb0\x88\xe6\xa1\x88/Unified hosts file with base extensions - Extending and consolidating hosts files from several well-curated sources like adaway.org, mvps.org, malwaredomainlist.com, someonewhocares.org, and potentially others./data'

Okay, can you add the following line of code above this line in the script (line 146 - 147) and run?
print(BASEDIR_PATH)
print(options["outputsubfolder"])
# This line already exists in the code, just wanted to indicate where!
options["outputpath"] = path_join_robust(BASEDIR_PATH, options["outputsubfolder"])

I'd be interested to see what gets printed.

> python updateHostsFile.py --auto
/home/Lin-Buo-Ren/工作空間/第三方專案/Unified hosts file with base extensions - Extending and consolidating hosts files from several well-curated sources like adaway.org, mvps.org, malwaredomainlist.com, someonewhocares.org, and potentially others.

Traceback (most recent call last):
  File "updateHostsFile.py", line 791, in <module>
    main()
  File "updateHostsFile.py", line 148, in main
    options["outputsubfolder"])
  File "updateHostsFile.py", line 766, in path_join_robust
    "likely a LOCALE issue:\n\n" + str(e))
locale.Error: Unable to construct path. This is likely a LOCALE issue:

'ascii' codec can't decode byte 0xe5 in position 18: ordinal not in range(128)

Interesting...it's printing the CJK correctly now.  Could you add two more lines as follows:
print(type(BASEDIR_PATH))
print(type(options["outputsubfolder"]))
print(BASEDIR_PATH)
print(options["outputsubfolder"])

# This line already exists in the code, just wanted to indicate where!
options["outputpath"] = path_join_robust(BASEDIR_PATH, options["outputsubfolder"])

@Vdragon : Any updates on this one?  I unfortunately can't debug this locally, so I would need your feedback to diagnose the issue (or figure out a workaround).

Stuck on the indentation error for a bit, anyway:
(shell) python updateHostsFile.py --auto
<type 'str'>
<type 'unicode'>
/home/<username>/工作空間/第三方專案/Unified hosts file with base extensions - Extending and consolidating hosts files from several well-curated sources like adaway.org, mvps.org, malwaredomainlist.com, someonewhocares.org, and potentially others.

Traceback (most recent call last):
  File "updateHostsFile.py", line 794, in <module>
    main()
  File "updateHostsFile.py", line 151, in main
    options["outputsubfolder"])
  File "updateHostsFile.py", line 769, in path_join_robust
    "likely a LOCALE issue:\n\n" + str(e))
locale.Error: Unable to construct path. This is likely a LOCALE issue:

'ascii' codec can't decode byte 0xe5 in position 18: ordinal not in range(128)

Okay.  I think we can now replicate the os.path.join issue.  Can you create a Python file in the same directory as updateHostsFile.py with the following code:
import os

BASEDIR_PATH = os.path.dirname(os.path.realpath(__file__))
print(os.path.join(BASEDIR_PATH, unicode(""))

and then run this with Python 2.x?

wierd.
(shell) python test.py
  File "test.py", line 5

                                                 ^
SyntaxError: invalid syntax

There is no line 5 even...can you cat test.py ?  That shouldn't break with a SyntaxError.

(shell) xclip -out -selection clipboard >test.py
(shell) cat test.py
import os

BASEDIR_PATH = os.path.dirname(os.path.realpath(__file__))
print(os.path.join(BASEDIR_PATH, unicode(""))(shell) python test.py
  File "test.py", line 5

                                                 ^
SyntaxError: invalid syntax
(shell) python --version
Python 2.7.12
(shell)

Hmm...not sure why I didn't catch it earlier...there's a parenthesis missing at the end...oops :smile:

(shell) xclip -out -selection clipboard >test.py
(shell) python test.py
Traceback (most recent call last):
  File "test.py", line 4, in <module>
    print(os.path.join(BASEDIR_PATH, unicode("")))
  File "/usr/lib/python2.7/posixpath.py", line 73, in join
    path += '/' + b
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 18: ordinal not in range(128)

There we go!  That's the error I was looking for.  So now that we have isolated the error, can you try this:
import os

BASEDIR_PATH = unicode(os.path.dirname(os.path.realpath(__file__)))
print(os.path.join(BASEDIR_PATH, unicode("")))

If this works, can you find this line in the script:
BASEDIR_PATH = os.path.dirname(os.path.realpath(__file__))

and replace it with
BASEDIR_PATH = unicode(os.path.dirname(os.path.realpath(__file__)))

(shell) xclip -out -selection clipboard >test.py
(shell) python test.py
Traceback (most recent call last):
  File "test.py", line 3, in <module>
    BASEDIR_PATH = unicode(os.path.dirname(os.path.realpath(__file__)))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 18: ordinal not in range(128)

I've been reporting similars issues on Git Cola a while ago, maybe the fixes of those issues can help:
https://github.com/git-cola/git-cola/issues?utf8=✓&q=is%3Aissue author%3AVdragon is%3Aclosed UnicodeDecodeError

I've been reporting similars issues on Git Cola a while ago, maybe the fixes of those issues can help

I'll take a look, but how about trying one more thing:
There we go!  That's the error I was looking for.  So now that we have isolated the error, can you try this:
import os

BASEDIR_PATH =os.path.dirname(os.path.realpath(__file__))
print(os.path.join(BASEDIR_PATH, str(unicode(""))))

(shell) xclip -out -selection clipboard >test.py
(shell) python test.py
/home/Lin-Buo-Ren/工作空間/第三方專案/Unified hosts file with base extensions - Extending and consolidating hosts files from several well-curated sources like adaway.org, mvps.org, malwaredomainlist.com, someonewhocares.org, and potentially others./

Okay, so what you try is changing this line in updateHostsFile.py:
options["outputpath"] = path_join_robust(BASEDIR_PATH, options["outputsubfolder"])

to this:
options["outputpath"] = path_join_robust(BASEDIR_PATH, str(options["outputsubfolder"]))

(shell) xdg-open updateHostsFile.py
(shell) python updateHostsFile.py
Traceback (most recent call last):
  File "updateHostsFile.py", line 789, in <module>
    main()
  File "updateHostsFile.py", line 149, in main
    settings = get_defaults()
  File "updateHostsFile.py", line 74, in get_defaults
    "datapath": path_join_robust(BASEDIR_PATH, "data"),
  File "updateHostsFile.py", line 764, in path_join_robust
    "likely a LOCALE issue:\n\n" + str(e))
locale.Error: Unable to construct path. This is likely a LOCALE issue:

'ascii' codec can't decode byte 0xe5 in position 18: ordinal not in range(128)
(shell)

Awesome, so it looks like we need to ensure that the paths we are joining are all strings (they're currently unicode because we imported unicode_literals from __future__).  So I think if you start walking through the script and replace all string arguments to path_join_robust (except for BASEDIR_PATH) with str(...), the script will work.  Let me know how that works.

(shell) git diff updateHostsFile.py | cat
diff --git a/updateHostsFile.py b/updateHostsFile.py
index dbe75cf..83fbac9 100644
--- a/updateHostsFile.py
+++ b/updateHostsFile.py
@@ -52,7 +52,7 @@ def write_data(f, data):

 def list_dir_no_hidden(path):
     # This function doesn't list hidden files
-    return glob(path_join_robust(path, "*"))
+    return glob(path_join_robust(str(path), str("*")))

 # Project Settings
@@ -71,13 +71,13 @@ def get_defaults():

     return {
         "numberofrules": 0,
-        "datapath": path_join_robust(BASEDIR_PATH, "data"),
+        "datapath": path_join_robust(BASEDIR_PATH, str("data")),
         "freshen": True,
         "replace": False,
         "backup": False,
         "skipstatichosts": False,
         "keepdomaincomments": False,
-        "extensionspath": path_join_robust(BASEDIR_PATH, "extensions"),
+        "extensionspath": path_join_robust(BASEDIR_PATH, str("extensions")),
         "extensions": [],
         "outputsubfolder": "",
         "hostfilename": "hosts",
@@ -87,16 +87,16 @@ def get_defaults():
         "sourcesdata": [],
         "readmefilename": "readme.md",
         "readmetemplate": path_join_robust(BASEDIR_PATH,
-                                           "readme_template.md"),
+                                           str("readme_template.md")),
         "readmedata": {},
         "readmedatafilename": path_join_robust(BASEDIR_PATH,
-                                               "readmeData.json"),
+                                               str("readmeData.json")),
         "exclusionpattern": "([a-zA-Z\d-]+\.){0,}",
         "exclusionregexs": [],
         "exclusions": [],
         "commonexclusions": ["hulu.com"],
-        "blacklistfile": path_join_robust(BASEDIR_PATH, "blacklist"),
-        "whitelistfile": path_join_robust(BASEDIR_PATH, "whitelist")}
+        "blacklistfile": path_join_robust(BASEDIR_PATH, str("blacklist")),
+        "whitelistfile": path_join_robust(BASEDIR_PATH, str("whitelist"))}

 def main():
@@ -143,8 +143,7 @@ def main():

     options = vars(parser.parse_args())

-    options["outputpath"] = path_join_robust(BASEDIR_PATH,
-                                             options["outputsubfolder"])
+    options["outputpath"] = path_join_robust(BASEDIR_PATH, str(options["outputsubfolder"]))
     options["freshen"] = not options["noupdate"]

     settings = get_defaults()
@@ -174,9 +173,9 @@ def main():
     finalize_file(final_file)

     if settings["ziphosts"]:
-        zf = zipfile.ZipFile(path_join_robust(settings["outputsubfolder"],
-                                              "hosts.zip"), mode='w')
-        zf.write(path_join_robust(settings["outputsubfolder"], "hosts"),
+        zf = zipfile.ZipFile(path_join_robust(str(settings["outputsubfolder"]),
+                                              str("hosts.zip")), mode='w')
+        zf.write(path_join_robust(str(settings["outputsubfolder"]), str("hosts")),
                  compress_type=zipfile.ZIP_DEFLATED, arcname='hosts')
         zf.close()

@@ -192,9 +191,9 @@ def main():
 # Prompt the User
 def prompt_for_update():
     # Create hosts file if it doesn't exists
-    if not os.path.isfile(path_join_robust(BASEDIR_PATH, "hosts")):
+    if not os.path.isfile(path_join_robust(BASEDIR_PATH, str("hosts"))):
         try:
-            open(path_join_robust(BASEDIR_PATH, "hosts"), "w+").close()
+            open(path_join_robust(BASEDIR_PATH, str("hosts")), "w+").close()
         except:
             print_failure("ERROR: No 'hosts' file in the folder,"
                           "try creating one manually")
@@ -317,8 +316,8 @@ def update_all_sources():
             updated_file = updated_file.replace("\r", "")

             hosts_file = open(path_join_robust(BASEDIR_PATH,
-                                               os.path.dirname(source),
-                                               settings["hostfilename"]), "wb")
+                                               str(os.path.dirname(source)),
+                                               str(settings["hostfilename"])), "wb")
             write_data(hosts_file, updated_file)
             hosts_file.close()
         except:
@@ -346,12 +345,12 @@ def create_initial_file():
     # spin the sources for extensions to the base file
     for source in settings["extensions"]:
         for filename in recursive_glob(path_join_robust(
-                settings["extensionspath"], source), settings["hostfilename"]):
+                str(settings["extensionspath"]), str(source)), settings["hostfilename"]):
             with open(filename, "r") as curFile:
                 write_data(merge_file, curFile.read())

         for update_file_path in recursive_glob(path_join_robust(
-                settings["extensionspath"], source),
+                str(settings["extensionspath"]), str(source)),
                 settings["sourcedatafilename"]):
             update_file = open(update_file_path, "r")
             update_data = json.load(update_file)
@@ -379,7 +378,7 @@ def remove_dups_and_excl(merge_file):
         os.makedirs(settings["outputpath"])

     # Another mode is required to read and write the file in Python 3
-    final_file = open(path_join_robust(settings["outputpath"], "hosts"),
+    final_file = open(path_join_robust(str(settings["outputpath"]), str("hosts")),
                       "w+b" if PY3 else "w+")

     merge_file.seek(0)  # reset file pointer
@@ -479,7 +478,7 @@ def write_opening_header(final_file):
     write_data(final_file, "# Fetch the latest version of this file: "
                            "https://raw.githubusercontent.com/"
                            "StevenBlack/hosts/master/" +
-               path_join_robust(settings["outputsubfolder"], "") + "hosts\n")
+               path_join_robust(str(settings["outputsubfolder"]), str("")) + "hosts\n")
     write_data(final_file, "# Project home page: https://github.com/"
                            "StevenBlack/hosts\n#\n")
     write_data(final_file, "# ==============================="
@@ -499,7 +498,7 @@ def write_opening_header(final_file):
             write_data(final_file, "<IP_ADDRESS> " + socket.gethostname() + "\n")
         write_data(final_file, "\n")

-    preamble = path_join_robust(BASEDIR_PATH, "myhosts")
+    preamble = path_join_robust(BASEDIR_PATH, str("myhosts"))
     if os.path.isfile(preamble):
         with open(preamble, "r") as f:
             write_data(final_file, f.read())
@@ -513,7 +512,7 @@ def update_readme_data():
         extensions_key = "-".join(settings["extensions"])

     generation_data = {"location": path_join_robust(
-        settings["outputsubfolder"], ""),
+        str(settings["outputsubfolder"]), str("")),
                        "entries": settings["numberofrules"],
                        "sourcesdata": settings["sourcesdata"]}
     settings["readmedata"][extensions_key] = generation_data
@@ -639,13 +638,13 @@ def flush_dns_cache():
 # Hotfix since merging with an already existing
 # hosts file leads to artifacts and duplicates
 def remove_old_hosts_file():
-    old_file_path = path_join_robust(BASEDIR_PATH, "hosts")
+    old_file_path = path_join_robust(BASEDIR_PATH, str("hosts"))
     # create if already removed, so remove wont raise an error
     open(old_file_path, "a").close()

     if settings["backup"]:
-        backup_file_path = path_join_robust(BASEDIR_PATH, "hosts-{}".format(
-            time.strftime("%Y-%m-%d-%H-%M-%S")))
+        backup_file_path = path_join_robust(BASEDIR_PATH, str("hosts-{}".format(
+            time.strftime("%Y-%m-%d-%H-%M-%S"))))

         # Make a backup copy, marking the date in which the list was updated
         shutil.copy(old_file_path, backup_file_path)
@@ -733,7 +732,7 @@ def recursive_glob(stem, file_pattern):
         matches = []
         for root, dirnames, filenames in os.walk(stem):
             for filename in fnmatch.filter(filenames, file_pattern):
-                matches.append(path_join_robust(root, filename))
+                matches.append(path_join_robust(str(root), str(filename)))
     return matches

(shell) python updateHostsFile.py --auto
Updating source ./data/someonewhocares.org from http://someonewhocares.org/hosts/zero/hosts
Updating source ./data/Badd-Boyz-Hosts from https://raw.githubusercontent.com/mitchellkrogza/Badd-Boyz-Hosts/master/hosts
Updating source ./data/UncheckyAds from https://raw.githubusercontent.com/FadeMind/hosts.extras/master/UncheckyAds/hosts
Updating source ./data/tyzbit from https://raw.githubusercontent.com/tyzbit/hosts/master/data/tyzbit/hosts
Updating source ./data/malwaredomainlist.com from https://www.malwaredomainlist.com/hostslist/hosts.txt
Updating source ./data/add.Spam from https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Spam/hosts
Updating source ./data/laicure from https://raw.githubusercontent.com/Laicure/HostsY/master/custHost
Updating source ./data/add.Risk from https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Risk/hosts
Updating source ./data/StevenBlack from https://raw.githubusercontent.com/StevenBlack/hosts/master/data/StevenBlack/hosts
Updating source ./data/SpotifyAds from https://raw.githubusercontent.com/FadeMind/hosts.extras/master/SpotifyAds/hosts
Updating source ./data/yoyo.org from https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext&useip=<IP_ADDRESS>
Updating source ./data/add.2o7Net from https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.2o7Net/hosts
Updating source ./data/mvps.org from http://winhelp2002.mvps.org/hosts.txt
Updating source ./data/adaway.org from https://raw.githubusercontent.com/AdAway/adaway.github.io/master/hosts.txt
Updating source ./data/KADhosts from https://raw.githubusercontent.com/azet12/KADhosts/master/KADhosts.txt
Updating source ./data/add.Dead from https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Dead/hosts
Updating source ./extensions/social from https://raw.githubusercontent.com/Sinfonietta/hostfiles/master/social-hosts
Updating source ./extensions/gambling from https://raw.githubusercontent.com/Sinfonietta/hostfiles/master/gambling-hosts
Updating source ./extensions/fakenews from https://raw.githubusercontent.com/marktron/fakenews/master/fakenews
Updating source ./extensions/porn/clefspeare13 from https://raw.githubusercontent.com/Clefspeare13/pornhosts/master/<IP_ADDRESS>/hosts
Updating source ./extensions/porn/sinfonietta from https://raw.githubusercontent.com/Sinfonietta/hostfiles/master/pornography-hosts
updateHostsFile.py:730: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  if stem == "*":
Success! The hosts file has been saved in folder
It contains 36,011 unique entries.
(shell)

Okay!  So not entirely sure where that UnicodeWarning is originating (i.e. stacktrace).  However, at least this patch seems to confirm what I thought was the issue.  Okay, so the diff you just showed should patch everything.  Let me put together something a little more compact in a PR and see if that works for you too.

@Vdragon : Okay, PR is up.  Let me know if the patch works for you (and whether you still get that warning).

(shell) git status
HEAD detached at gfyoung/unicode-str-join
Untracked files:
  (use "git add <file>..." to include in what will be committed)

        test.py

nothing added to commit but untracked files present (use "git add" to track)
(shell) python updateHostsFile.py --auto
Updating source ./data/someonewhocares.org from http://someonewhocares.org/hosts/zero/hosts
Updating source ./data/Badd-Boyz-Hosts from https://raw.githubusercontent.com/mitchellkrogza/Badd-Boyz-Hosts/master/hosts
Updating source ./data/UncheckyAds from https://raw.githubusercontent.com/FadeMind/hosts.extras/master/UncheckyAds/hosts
Updating source ./data/tyzbit from https://raw.githubusercontent.com/tyzbit/hosts/master/data/tyzbit/hosts
Updating source ./data/malwaredomainlist.com from https://www.malwaredomainlist.com/hostslist/hosts.txt
Updating source ./data/add.Spam from https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Spam/hosts
Updating source ./data/laicure from https://raw.githubusercontent.com/Laicure/HostsY/master/custHost
Updating source ./data/add.Risk from https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Risk/hosts
Updating source ./data/StevenBlack from https://raw.githubusercontent.com/StevenBlack/hosts/master/data/StevenBlack/hosts
Updating source ./data/SpotifyAds from https://raw.githubusercontent.com/FadeMind/hosts.extras/master/SpotifyAds/hosts
Updating source ./data/yoyo.org from https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext&useip=<IP_ADDRESS>
Updating source ./data/add.2o7Net from https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.2o7Net/hosts
Updating source ./data/mvps.org from http://winhelp2002.mvps.org/hosts.txt
Updating source ./data/adaway.org from https://raw.githubusercontent.com/AdAway/adaway.github.io/master/hosts.txt
Updating source ./data/KADhosts from https://raw.githubusercontent.com/azet12/KADhosts/master/KADhosts.txt
Updating source ./data/add.Dead from https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Dead/hosts
Updating source ./extensions/social from https://raw.githubusercontent.com/Sinfonietta/hostfiles/master/social-hosts
Updating source ./extensions/gambling from https://raw.githubusercontent.com/Sinfonietta/hostfiles/master/gambling-hosts
Updating source ./extensions/fakenews from https://raw.githubusercontent.com/marktron/fakenews/master/fakenews
Updating source ./extensions/porn/clefspeare13 from https://raw.githubusercontent.com/Clefspeare13/pornhosts/master/<IP_ADDRESS>/hosts
Updating source ./extensions/porn/sinfonietta from https://raw.githubusercontent.com/Sinfonietta/hostfiles/master/pornography-hosts
updateHostsFile.py:731: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  if stem == "*":
Success! The hosts file has been saved in folder
It contains 36,011 unique entries.
(shell)

I think that warning is because stem contains your directory name, which can't be casted to unicode, so I think I'll leave that alone for now.

Actually, I change my mind on that.  Can you add the following change to the line here:
if stem == "*"

and change it to be:
if stem == str("*")

on top of my changes and see if the warning goes away?

Yep.
(shell) git diff
diff --git a/updateHostsFile.py b/updateHostsFile.py
index 1c55d58..0442efc 100644
--- a/updateHostsFile.py
+++ b/updateHostsFile.py
@@ -728,7 +728,7 @@ def recursive_glob(stem, file_pattern):
     if sys.version_info >= (3, 5):
         return glob(stem + "/**/" + file_pattern, recursive=True)
     else:
-        if stem == "*":
+        if stem == str("*"):
             stem = "."
         matches = []
         for root, dirnames, filenames in os.walk(stem):
(shell) python updateHostsFile.py --auto
Updating source ./data/someonewhocares.org from http://someonewhocares.org/hosts/zero/hosts
Updating source ./data/Badd-Boyz-Hosts from https://raw.githubusercontent.com/mitchellkrogza/Badd-Boyz-Hosts/master/hosts
Updating source ./data/UncheckyAds from https://raw.githubusercontent.com/FadeMind/hosts.extras/master/UncheckyAds/hosts
Updating source ./data/tyzbit from https://raw.githubusercontent.com/tyzbit/hosts/master/data/tyzbit/hosts
Updating source ./data/malwaredomainlist.com from https://www.malwaredomainlist.com/hostslist/hosts.txt
Updating source ./data/add.Spam from https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Spam/hosts
Updating source ./data/laicure from https://raw.githubusercontent.com/Laicure/HostsY/master/custHost
Updating source ./data/add.Risk from https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Risk/hosts
Updating source ./data/StevenBlack from https://raw.githubusercontent.com/StevenBlack/hosts/master/data/StevenBlack/hosts
Updating source ./data/SpotifyAds from https://raw.githubusercontent.com/FadeMind/hosts.extras/master/SpotifyAds/hosts
Updating source ./data/yoyo.org from https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext&useip=<IP_ADDRESS>
Updating source ./data/add.2o7Net from https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.2o7Net/hosts
Updating source ./data/mvps.org from http://winhelp2002.mvps.org/hosts.txt
Updating source ./data/adaway.org from https://raw.githubusercontent.com/AdAway/adaway.github.io/master/hosts.txt
Updating source ./data/KADhosts from https://raw.githubusercontent.com/azet12/KADhosts/master/KADhosts.txt
Updating source ./data/add.Dead from https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Dead/hosts
Updating source ./extensions/social from https://raw.githubusercontent.com/Sinfonietta/hostfiles/master/social-hosts
Updating source ./extensions/gambling from https://raw.githubusercontent.com/Sinfonietta/hostfiles/master/gambling-hosts
Updating source ./extensions/fakenews from https://raw.githubusercontent.com/marktron/fakenews/master/fakenews
Updating source ./extensions/porn/clefspeare13 from https://raw.githubusercontent.com/Clefspeare13/pornhosts/master/<IP_ADDRESS>/hosts
Updating source ./extensions/porn/sinfonietta from https://raw.githubusercontent.com/Sinfonietta/hostfiles/master/pornography-hosts
Success! The hosts file has been saved in folder
It contains 36,011 unique entries.


createPattern method is in the wrong service class

in the hqdm module on the develop branch the SpatioTemporalExtentServices.createPattern method should be moved to ClassServices since a Pattern is a ClassOfSign rather than a SpatioTemporalExtent.

This also applies to RepresentationByPattern, Identification, Description, and Definition.


add approvers

@ycao56 Both @jrnp7 and @yanni-zhang need to join IBM Org if they want to be in OWNERS file.

/approve

/approve

/assign ycao56

/approve

lgtm

/lgtm

/assign yannizhang2019


Updated line number colors for the dark themes, to improve visibility

Fix for issue #2594

Can you make these adjustments outside of the CSS copied directly from CodeMirror, otherwise we can't copy the updated styles directly back in (or we do so at the expense of losing your changes).


Update requirements.md

Description
Python 3.10 is cool too.
Checklist
Ticks in all boxes and 🟢 on all GitHub actions status checks are required to merge:

[x] I'm aware of the code of conduct.
[x] I've created logical separate commits and followed the commit message format.
[x] I've targeted the master branch.
[ ] If this documentation change impacts the current release of InvenioRDM, I will backport it to the production branch following approval or indicate to a maintainer that it should be backported.

Reminder
By using GitHub, you have already agreed to the GitHub’s Terms of Service including that:

You license your contribution under the same terms as the current repository’s license.
You agree that you have the right to license your contribution under the current repository’s license.

Have you tested an install of v12 with 3.10? I think we left it off because no one had tested it.


TV Episodes by custom order

Version Number
1.16.5
What branch are you on?
master
Describe the Bug
When creating a tv episode list, the custom order flag does not work. For example, Simpsons episodes rated 7.5 or higher in descending order of highest rated to lowest results in either an alphabetical list, or a random order list. Alpha and Release flags work properly, but custom does not. There is no error in the log, it just doesnt work as expected. I think something breaks in the moving process as the logs look like it worked, but then you head to plex and the list is not the correct order.
Relevant Collection/Playlist Definition
Simpsons Best of:
    collection_order: custom
    collection_level: episode
    sync_mode: sync
    imdb_list:
      url: https://www.imdb.com/search/title/?series=tt0096697&sort=user_rating,desc
      limit: 250
    sort_title: Simpsons Best of

Logs
https://gist.github.com/sych0/6f15a58a2bd1db036a49a1df8e64585d

any update on my languishing bug?

I don't think there's any update at this point in time, but congratulations on having one of the longest-running issues lol!
I will add this to our internal tracker as this has definitely been discussed a few times over the past year and is something we do want to address.
Stay posted!

so i wanna try and get this cleaned up for our next release is the same thing still happening post 1.20.0 and if so could you post a new log

Top 100 Simpsons Episodes:
collection_order: custom
collection_level: episode
plex_search:
type: episodes
sort_by: audience_rating.desc
limit: 100
all:
title.ends: "Simpsons"
summary: A collection of the highest rated simpsons epsodes.

I can confirm this is working in 1.20


Ignore external OSPR issues

@sarina: This should make the bot stop complaining about external pull requests. However, it relies on JIRA issues having the "repo" field filled out correctly. If the field is empty, or filled out incorrectly, then there's not really anything the bot can do but throw an error. I've corrected the "repo" field on OSPR-850.

sounds good, loooks good, ty :+1:


OSGi package headers have wrong version

Something seems amiss: 0.3.1 has package versions with 0.2.11.  But building the bundle locally, from a clean install, shows 0.3.1.

Closing as 0.3.2 seems ok though.


chore: increase version

@homu r+

:pushpin: Commit 522d934 has been approved by kbknapp

:hourglass: Testing commit 522d934 with merge 8b41e94...

:sunny: Test successful - status


explicitly predict x0 based on xt and trained model

Hi @LuChengTHU ,
Thanks for your great work and detailed documents.
I have difficulty finding the predicted x0 in the dpmsolver. Would it be possible for you to point it out?
https://github.com/LuChengTHU/dpm-solver/blob/main/dpm_solver_pytorch.py
From x_t to x_{t-1}, the mean is actually the weighted sum between x_t and predicted x_0 (based on x_t).
Although we can obtain a simpler formula for x_t-1 by plugging x_0 into the mean equation (\mu),  the predicted x_0 is very important when solving some inverse problems because we can incorporate reference information by perturbing the predicted x_0. For example https://ddrm-ml.github.io/
This is the reason that I want to find the code of the predicted x_0. Any guidance would be highly appreciated:)

Hi, it's exactly the data prediction model, https://github.com/LuChengTHU/dpm-solver/blob/main/dpm_solver_pytorch.py#L441

Thank you so much!

Hi @LuChengTHU ,
Thanks again for your help.
The function data_prediction_fn is used for dpmsolver++, which is not used in dpmsolver, right?
https://github.com/LuChengTHU/dpm-solver/blob/3d31b82a62799e62cbfbb37a109fdef6309f9cd3/dpm_solver_pytorch.py#L655-L681
I want to use dpmsolver for unconditional sampling. Specifically, I have trained an unconditional model based on guided-diffusion (no classifier guidance). The naive DDPM sampling process (e.g., x_t to x_{t-1}) includes two steps

predict x_0 based on x_t
obtainx_{t-1} based on the predicted x_0 and x_t

To solve inverse problems based on diffusion models, I will modify the predicted x_0. e.g., new x_0 = Ax_0 + B.  The x_{t-1} will be based on the new x_0 and x_t
My goal is to speed up this sampling process based on dpmsolver. Based on the recommendation, I should use the the 3rd-order solver.
https://github.com/LuChengTHU/dpm-solver#example-unconditional-sampling-by-dpm-solver
It seems that dpmsolver doesn't explicitly compute the predicted x_0 for unconditional sampling, right?
https://github.com/LuChengTHU/dpm-solver/blob/3d31b82a62799e62cbfbb37a109fdef6309f9cd3/dpm_solver_pytorch.py#L683-L801
            if solver_type == 'dpmsolver':
                x_t = (
                    (sigma_t / sigma_s) * x
                    - (alpha_t * phi_1) * model_s
                    + (1. / r2) * (alpha_t * phi_2) * (model_s2 - model_s)
                )

Hi @JunMa11
Yes, dpmsolver doesnot explicitly use the data prediction model. But you can also use 3rd-order dpmsolver++ for unconditional sampling, and it can also get a similar performance as dpmsolver.

In my own experience, both dpmsolver++ and dpmsolver can achieve quite good acceleration performance for unconditional sampling. So you can try dpmsolver++ directly :)


Mapping invalidation is a hard stall on Metal

See this code - https://github.com/gfx-rs/gfx/blob/3641183231f16877d4ea2fbdb2ff208ce736d6c4/src/backend/metal/src/device.rs#L1817
I believe this is inacceptable. The discussion in VkPortability is also kicked off.
Any ideas on how to make this better are very welcome!
Note: this is affecting (negatively) our WebRender friends.

Here is a crazy idea: we could have another queue internally associated with a device. When we need to synchronize managed memory, we commit on that other queue instead of the main one. This way it could be acceptable to wait for the command to complete on CPU.


Clickable Features

Clickable Features
User Story
AS AN user for the application
I WANT to have a good user experience
SO THAT I can meet my end goals for using the application
Acceptance Criteria
GIVEN the user would like to have a good user experience
WHEN they navigate the application
THEN they will be able to find & interact with clickable features

This has been completed.


Adds missing GFX Texture format strings

This commit eliminates 5 warnings output from VALIDATE_LOOKUPTABLE here: https://github.com/GarageGames/Torque3D/blob/development/Engine/source/gfx/gfxStringEnumTranslate.cpp#L167
And prevents a fatal error here: https://github.com/GarageGames/Torque3D/blob/development/Engine/source/gfx/gfxTextureManager.cpp#L1371
that occurs when called from GuiOffscreenCanvas::_setupTargets() for an OpenVROverlay because GFXFormatR8G8B8A8_LINEAR_FORCE  is used.

Nice a few less warnings in the console, tested ok here.


Userendpoints

The feature lets users create endpoints if they have the "stratos.endpointadmin" scope.
Description

General Behaviour

Users need the "stratos.endpointadmin" scope to use this feature. From now on, when i say users, assume they have the "stratos.endpointadmin" scope unless otherwise stated.
In the config USER_ENDPOINTS_ENABLED can be set to various

disabled: Disabled
admin_only: Admins will be able to see all endpoints, but users won't be able to see or create user endpoints.
enabled: Admins see all endpoints, users see all admin-endpoints and can create/edit/connect/see their own endpoints.

Users can register endpoints, unless

An admin-endpoint with the same name / url already exists
A user-endpoint from the same user with the same name / url already exists

Users will have access to stratos, when no endpoints have been registered yet (the no-endpoint screen will be skipped)
Users can edit, unregister and connect to their own user-endpoints
User can't see user-endpoints created by other users
User-endpoints have the option for user-invites disabled. They can't be shared by users or admins
Admins can edit and unregister user-endpoints, but won't be able to connect to them
Admins can't create endpoints, if an endpoint with the same url / name already exists, unless

A checkbox has been added to the register screen to overwrite existing user-endpoints. When checked, user-endpoints with the same url will be unregistered

Back-End

Added file<PHONE_NUMBER>0000_Creator.go

Adds a new row to the cnsis table with name "creator"

User registers an endpoint -> value: user-id
Admin registers an endpoint -> value: ""

config

Added an enum "USER_ENDPOINTS_ENABLED"

middleware.go

Added endpointMiddleware()

Checks if user is admin or endpointadmin
if its not a CREATE request, then it also checks the existing endpoint and user role according to how it's descriped in section General Behaviour

cnsi.go

buildCNSIList()

Checks if user is admin

Admin -> return all endpoints
User -> return own endpoints and admin-endpoints
User & enum set to admin_only -> return admin-endpoints
Enum set to disabled -> return admin-endpoints

registerEndpoints()

looks up user-id from session user and passes it on to DoRegisterEndpoints()

DoRegisterEndpoints()

Now accepts a user-id as parameter too. If parameter is "", it assumes an anonymous user.
Also checks now for described behaviours in section General Behaviour
For user-endpoints, the guid will now be generated with url+userID as input, instead of just the url

unregisterCluster()

Split into two functions: unregisterCluster(c echo.Context) and doUnregisterCluster(cnsiGUID string), so that cnsi can be unregistered without the need of an echo.Context

Added ListAdminEndpoints(userID string)

returns all endpoints created by admins and given user-id

Added listCNSIByAPIEndpoint(apiEndpoint string)

returns all endpoints with the given APIEndpoint URL

Renamed GetCNSIRecordByEndpoint() and cnsiRecordExists() to GetAdminCNSIRecordByEndpoint() and adminCNSIRecordExists() to better reflect their function now

authcnsi.go

DoLoginToCNSI()

Checks for described behaviour in section General Behaviour

info.go

getInfo()

looks up saved id in column creator from cnsi and replaces it with interface.CreatorInfo
If no user can be found, the username will be set to just "user"

main.go

set ENABLE_USER_ENDPOINTS to disabled as default
use different middleware if flag is enabled or not

userinvite/admin.go

added a check for creator

userinvite/auth.go

Added checkEndpointCreator()

Checks if admin-endpoint, otherwise don't allow invites

plugins/desktop/*

Adjust several files to also include new functions added to store structs

structs.go

Add Creator to several structs
Add UserEndpointsEnabled to PortalConfig and Info
Add OverwriteEndpoints to RegisterEndpointParams

pgsql_cnsis.go

Added listCNSIsByCreator sql query and belonging functions
Added function to query a list using findCNSIByAPIEndpoint
Created a generic listBy() function because DRY

pgsql_token.go

Added listAuthToken sql query and belonging functions

Tests

Adjusted mock data for several tests so they don't break

Front-End

endpoint.types.ts

added a CreatorInfo interface to hold creator user data for cnsi

user-permission.directive.ts

Change so that it can have multiple permissions to check for, instead of just one. Results true if one permission return true.

endpoint-list.helpers.ts

Show menu items according to described rules in section General Behaviour

base-endpoints-data-source.ts

add default values for added parameters

stratos-user-permissions.checker.ts

Add the stratos.endpointadmin as a scope that can be checked for

session.service.ts

Add methods to return values of USER_ENDPOINTS_ENABLED

auth.types.ts

add enableUserEndpoints to interface SessionDataConfig

endpoint.actions.ts / endpoint.effects.ts / stratos-action-builders.ts

Add "overwriteEndpoints" variable

endpoints.service.ts

Let endpointadmins see stratos gui when no endpoints are there yet

user.service.ts

Also save if user is an endpointadmin

user-invite.service.ts

check if user-endpoint and disable invites

endpoints-page.component

Show the add endpoint for admins and endpointadmins

endpoint-card.component

Show who created the endpoint.

create-endpoint-cf-step-1.component / git-registration.component

Add "Overwrite User Endpoints" checkbox to gui and form

Motivation and Context

Issue https://github.com/cloudfoundry/stratos/issues/4753
How Has This Been Tested?

OS: macOS Big Sur
go test passes
npm run test-backend fails without changes in https://github.com/cloudfoundry/stratos/pull/4874
e2e tests fail on local machine with following setup:

PCF dev for CF endpoint
PCF dev is set up with script described in docs
Stratos uses docker container splatform/stratos-uaa
Stratos runs directly with ./jetstream script

So far only adjusted mock data to mostly accommodate the new added column to the cnsi table, to check that everything behaves the same when the feature is disabled.
I will add more test functions later, as discussed with @richard-cox . This PR is for looking over the code while i continue to adjust the tests.
Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] Docs update
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

[x] My code follows the code style of this project.
[x] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[x] I have followed the guidelines in CONTRIBUTING.md, including the required formatting of the commit message

Codecov Report

Merging #4875 (2cbe85a) into master (54a0af2) will decrease coverage by 26.56%.
The diff coverage is 18.97%.

@@             Coverage Diff             @@
##           master    #4875       +/-   ##
===========================================
- Coverage   52.91%   26.35%   -26.57%
===========================================
  Files        1141       61     -1080
  Lines       40149     4925    -35224
  Branches     5122        0     -5122
===========================================
- Hits        21246     1298    -19948
+ Misses      18719     3430    -15289
- Partials      184      197       +13

Closing in favour of #4876 (avoids travis issues)


Fix -no-tls1_2 in tests

Fixed a configuration-related issue in #17952

Checklist

[ ] documentation is added or updated
[ ] tests are added or updated

This pull request is ready to merge

Merged to master and 3.0. Trivial merge to 1.1.1, but build fails due to undefined TEST_skip. So, this will likely need to be a separate PR.

Merged to 1.1.1 via #18080.


Could the readme provide a brief comparison with lsp-bridge?

https://github.com/manateelazycat/lsp-bridge is another external service for speeding up LSP in Emacs.
(Or rather, it is a whole separate LSP client -- ala eglot or lsp-mode -- but one which utilises its own external services for performance purposes.)
It would be great if the documentation provided a brief comparison of the similarities and differences, and it would be really useful to know how the performance of lsp-bridge compared to the performance of both eglot and lsp-mode when enhanced by emacs-lsp-booster.
I'm also curious as to whether the method used for emacs-lsp-booster might be used to provide further performance improvements for lsp-bridge.  The latter seems to be written in python rather than rust, but I presume the technique could be ported even if the rust code can't easily be used more directly.
Edit: I've since queried that last part directly in the lsp-bridge issue queue at https://github.com/manateelazycat/lsp-bridge/issues/824

lsp-bridge is a full LSP client in Emacs (and Python), one which entirely replaces lsp-mode and eglot.  This booster simply makes those clients' jobs easier by making the underlying lsp server i) more responsive, and ii) speak in elisp bytecode, which is ~4x faster for emacs to parse than JSON.   So all of the normal built-in or 3rd party completion, xref, etc. tooling work exactly the same as before (just faster and with less I/O blocking).
I am not an lsp-bridge user, but my understanding is that it is similar in spirit but goes much further to try to optimize things like completion, for example allowing emacs to interrupt the bridge so it doesn't bother sending expired data which will just be ignored.  I'm not sure if it translates JSON into bytecode or list literals, but it does appear to do advanced caching to avoid I/O contention, similar to this simple booster.  It is a "smart bridge" which to some degree speaks the LSP language and communicates on its own with the lsp server, and it utilizes that capability to optimize the flow of traffic between (its own) emacs client and the lsp server.  Think lsp-bridge = "negotiator" vs. lsp-booster = "translator".
It's true that the "bridge" in lsp-bridge is a python program, whereas this one is built on rust.  And unlike rust, Python does not have true parallel threading (yet!), but I truly doubt that makes any difference, since most all the operations between server and client are I/O bound.  So I think that distinction can be ignored.
The cost of lsp-bridge's functionality is that to achieve its goals, it had to implement its own non-standard, standalone "world", with its own completion system, completion UI, auxiliary completion backends, xref functionality, language server support, requirement for no native compilation, etc.   By all appearances it is a highly functional and performant world (e.g. it's dabbrev completion equivalent I understand runs circles around dabbrev).  But you have to stay in that world for all lsp-related activity, and use different tools and UIs outside of it.
There's also a maintenance difference.  If both tools stopped being maintained today (not that that is likely or imminent), I suspect this simple booster (~1k lines of code) would continue to function just fine for 5 or 10 years, whereas lsp-bridge (~20k LOC, not counting epc) would suffer more from that lack of maintenance.
I recommend trying them both to decide which has the features and tradeoffs that work for you, after first seeing whether lsp-mode or eglot are sufficient on their own.

LSP-bridge not only increases JSON parsing speed but also uses Python multi-threading to encapsulate all time-consuming LSP-related calculations in a separate Python process. This avoids any lag caused by large data computations (such as the Volar server). If you only accelerate JSON parsing without encapsulating other logic into subthreads, you can only reduce lag, not avoid it like LSP-bridge does.
JSON parsing in LSP-bridge can optionally use the orjson library, which is implemented in Rust and achieves the fastest parsing speed possible.
The maintenance cost of LSP-bridge is not an issue. Even in terms of code volume, LSP-bridge should be much smaller than LSP-mode + emacs-LSP-booster + company/corfu combined. The advantage of LSP-bridge is that if there's a problem, whether in the completion frontend or backend, it can be fixed from the fundamental level. Users don't need to integrate and configure these details themselves. In simple terms, LSP-bridge has the advantage of being out-of-the-box in this aspect.


0.4.0 stalls when copying large amounts of data from GCS to Postgres

full trace logs provided privately to @emk, but:
download may appear to continue, but nothing more is pushed to Postgres.

Fixed in 0.4.1. This was a problem with GCS list paging in very large buckets.


Python API inside a Docker container fails

Describe the bug
Using the Python API example works perfectly in a virtual environment.
Fails when run in a docker container with a time out.
HTTPConnectionPool(host='<IP_ADDRESS>', port=63342): Max retries exceeded with url:
To Reproduce
import os
import pathlib as pl
import shutil

import pytest
from gcp_storage_emulator.server import create_server
from google.auth.credentials import AnonymousCredentials
from google.cloud import storage

import upload

@pytest.fixture
def mock_storage_client():
    """Google Cloud Storage emulator"""
    host = "localhost"
    port = 9023
    server = create_server(host, port, in_memory=True)
    server.start()
    os.environ["STORAGE_EMULATOR_HOST"] = f"http://{host}:{port}"
    yield storage.Client(credentials=AnonymousCredentials(), project="test-project")
    server.stop()
    del os.environ["STORAGE_EMULATOR_HOST"]

def test_upload_folder(mock_storage_client):
    """Test uploading a folder to Google Cloud Storage"""
    bucket_name = "test-upload-bucket"
    mock_storage_client.create_bucket(bucket_name)

Expected behavior
Bucket is created.
System (please complete the following information):

OS version: macos / linux
Python version: 3.9
gcp-storage-emulator version: latest

Additional context
Add any other context about the problem here.

@swarmt did you ever find a work around for this issue?


Consistent node type categorization

We have the following node type categories, which are sometimes defined via inheritance chain, sometimes via a static property definition. Let's see if we can unify this to be able to better reason about in which category a given node is. However let's find use-cases so we don't engineer blindly what could be useful..
So far we are aware of the following categories:
Text
Any text-ish node, such as paragraph, heading, caption etc.
Block
Parent is a container, so it's kind of a root node. E.g. image, paragraph, heading.
Container
Manages a list of Block nodes (List, Paragraph, Codeblock, Blockquote)
Annotation
Overlays characters of a text property. E.g. Strong, Link, etc.
Inline
An property annotation that also has content. E.g. a dynamically labelled FigureCaption where the label is the content. But this could also be an InlineImage. Or InlineFormula. Inline nodes wrap a fake character, that just says there is some content here that you can not edit.
ContainerAnnotation
Special annotation that sticks on a container rather then a text property.
We can not describe this entirely via inheritance chain. Because for instance a Paragraph would inherit from Block AND Text.
Node.static = {
  isText: false,
  isBlock: false,
  isInline: false,
  isAnnotation: false,
  isContainer: false,
  isContainerAnnotation: false
};

We can have a class hierarchy like this and we just set the static properties accordingly. Except for isBlock, which needs to be set manually in all block implementations, much like it is the case already using blockType = true)
- Node
   - Text
   - Annotation
      - Inline
   - ContainerAnnotation
   - Container

'Container' is a bit unclear. List for example is not a container, it is a nested node type, same as Table.

List would be just a block in that terminology. The question is what is a ListItem or TableRow? It seems we can not say a node is either a block or inline.. there's more...

In need this checking stuff in different places. Considering that the node refactor is now in.. can we take care of this now?
In SwitchTextType I need to check if the node is a Block and a Text element. I have the node instance in this case. The options I have currently:
node instanceOf TextNode // need to require TextNode to do that
node.constructor.static.blockType // a bit combersome
node instanceof TextBlock // combined check but still needs require

Suggestions for more straight-forward checking:
node.isBlock();
node.isText();
node.isInline(); // currently external
node.isAnnotation();

// this we would need to define on the DocumentNode to have it as an interface, and each
// function would return false by default. Howevver you mentioned that you don't like to put
// it there since DocumentNode could be kept more clean and generic, not knowing about blocks and text things.
// So alternatively we could provide an `is()` method that is abstract and can be implemented on the classes using the info form static.

node.is('block')
node.is('text')
node.is('inline')
node.is('annotation')

// On the node class maybe more like this:
NodeClass.static.is('block')

I think what we need to do is announcing these properties in the API docs and then users don't need to think so much on how to check for a certain category.

I think isBlock(), isInline(), and isAnnotation() should be on the DocumentNode interface as these are essential core concepts.
With text, you can already do node.isInstanceOf('text') which works on the schema.

what would speak again isText() and isTextBlock() too? It's a bit redundant i know.. but somehow it's odd to sometimes have a prop and sometimes check via inheritance chain.

isText() ok... but that is the last one ;)
IsTextBlock() === isText() && isBlock()

hehe ahja! ;) deal! I will do that then now. Thanks!

oh and for static... let's just call it isBlock istead of blockType so it matches with the function name?

ok.

Hmmm.. maybe 'isBlock' is not a good idea though, as it would be confusing to do paragraph.isBlock() but Paragraph.static.isBlock?

dunno.

Just wanted to say this.

Maybe we can also achieve to rename 'external' to 'isInline'?

is it confusing? paragraph.isBlock() and Paragraph.static.isBlock ? it's not so bad i would say... unless we want a function on static too?
external -> inline yes!

and strong,emphasis,links etc. are just annotations or could that also have a name in the terminology?

This I want to call annotation as opposed to inlineNode. I.e., annotation doesn't own the plain text.

inlineNode is also an annotation, technically. But it's spcial in that it owns the content. can we say it like this?

we also have some confusion in naming because for instance in the converter we call all annotation converters 'inlineConverters'.

Yes, only implementation wise. But not semantically.

Hmmm.. agreed.

Maybe I have to reconsider my categorization?
What is your suggestion?

I mean we just must not abuse the word 'inline' for other things than this node type thingee.. where it is at home the most in my opinion.
maybe call the categorization just 'annotationConverters'?

Yep, agreed. Annotation is implementation wise the more generic.

Should we have an Inline class that inherits from PropertyAnnotation? currently propertyAnnotation sets isInline true.. i would have that false by default?

Agreed.

Inline.js or InlineNode.js?

Stimmt!

also wird das eher a ganz eigener bin _containerAnnotations im converter... ? Weil dann würd i glei sagen _propertyAnnotations. inlineNode is ja technically auch a propertyAnnotation.

yup, ich denk

Seit wann redma hier deutsch? ;) sorry.
I mach _blockTypeConverters -> _blockConverters und _inlineTypeConverters zu _propertyAnnotationConverters

ok.

Fixed via #264


Let Sequin control retention of event tables

Right now, when you create an event table to use with a WAL Pipeline, Sequin instructs you how you can add a retention policy to that table:

pg_cron: Simple, and just fine for volumes <1M writes/day
pg_partman: Great scalable strategy for adding retention to a Postgres table

These are both great options, but pg_cron can take a few steps to install and setup. It's easy on a cloud db (where you just need to run create extension ) but several more steps running locally.
One thing we could do is have a third option where Sequin enforces the retention policy through a background worker that simply runs a delete where ….

With improved support for pg_partman, this gets kicked further down the road


sveltekit and svelte-lottie-player

I get this error when building my project with an imported LottiePlayer
document is not defined ReferenceError: document is not defined
I read some other issue where they suggested adding this vite config item to the svelte.config.js

But that doesn't solve the issue for me.
The only thing that does is this code:

But this causes some ugly popin of the player.  I want the animation to render on first draw.  Any suggestions?  Does LottiPlayer not work for static sites?

same problem here,
found a solution?

Hi @leouzz @JeffWScott
document is not defined on the server, so you need to guard against that in your component so that particular piece of code is only run in the browser.
You can use the onMount function which is only run in the browser when the component has been rendered.
<script>
  import { onMount } from 'svelte';

  onMount(() => {
    // ...
  });
</script>

Sapper works well with most third-party libraries you are likely to come across. However, sometimes, a third-party library comes bundled in a way which allows it to work with multiple different module loaders. Sometimes, this code creates a dependency on window, such as checking for the existence of window.global might do.
Since there is no window in a server-side environment like Sapper's, the action of simply importing such a module can cause the import to fail, and terminate the Sapper's server with an error such as:
ReferenceError: window is not defined
The way to get around this is to use a dynamic import for your component, from within the onMount function (which is only called on the client), so that your import code is never called on the server.
Full working solution is provided below.
Player.svelte
<script>
	import { onMount } from 'svelte';

	let LottiePlayer;

	onMount(async () => {
		const module = await import('@lottiefiles/svelte-lottie-player');
		LottiePlayer = module.LottiePlayer;
	});

	let controlsLayout = [
		'previousFrame',
		'playpause',
		'stop',
		'nextFrame',
		'progress',
		'frame',
		'loop',
		'spacer',
		'background',
		'snapshot',
		'zoom',
		'info'
	];
</script>

{#if LottiePlayer}
	<LottiePlayer
		src="https://assets2.lottiefiles.com/packages/lf20_wxUJzo.json"
		autoplay={true}
		loop={true}
		controls={true}
		renderer="svg"
		background="transparent"
		height={600}
		width={600}
		{controlsLayout}
	/>
{/if}

You may then import this module into other svelte components as needed.

Facing the same issue here @karamalie.
From svelte.config.js:
vite: { 		optimizeDeps: { 			include: ["lottie-web"], 		} 	}
from svelte-kit route:
`

{#if LottiePlayer}

<LottiePlayer
	src="background-fullscreen.json"
	autoplay={true}
	loop={true}
	controls={true}
	renderer="svg"
	background="transparent"
	height={600}
	width={600}
/>
{/if}

`
Get the following error:
node_modules/.pnpm/lottie-web@5.8.0/node_modules/lottie-web/build/player/lottie.js?v=5232e3af' does not provide an export named 'default'
Lottie-web installed as a dep.

I was going through the same error when I tried using the svelte-lottie-player along with sveltekit.
I used lottie-player provided by lottie-files themselves
Here's a link for reference:

LottiePlayer = module.LottiePlayer;  @subhendupsingh please try pulling the LottiePlayer module instead of default. could you perhaps post up a snippet of the code so we can test it out?

I think the error is with how the library pulls lottie-web

Still not fixed in 2024?

Import like this:
if (typeof window !== 'undefined') import('@lottiefiles/lottie-player')


bj_bus.py: fix #30

bj_bus.py:

fix misused id;
refactor API query logic, now use local id/linename cache;
fix python-cn/slack_bot#30

TODOs:

distinguish same linename of different directions
refer all FIXMEs in bj_bus.py
refactor global functions into API class methods

+TODO: move raw line info cache out of .py file

@andelf 辛苦 我merge了 试一下


Adds Servo Power Instrument

Please do use chroot() to cros sdk, it takes time (more than one second on my machine). I think my second trial which uses xmlrpc is better. https://github.com/freedomtan/workload-automation/commit/de0b7bd1487b61a108029b382a2bcccde1e1bd8f

Obsoleted by https://github.com/ARM-software/workload-automation/pull/144


Added translation for "React without JSX" section

@Kurzdor Thanks for the contribution 🙌Your effort is very much appreciated 👍
Can I ask you to work on one section per time, please? It seems difficult to work on several of them in parallel, so it'd be more efficient to focus on one and switch to another one once the first one is finished.
Could you also please polish your PRs because there are currently two PRs for "React without JSX" opened.

@klymenko another one is about React without ES6. @Kurzdor just forgot to change the title)

@klymenko another one is about React without ES6. @Kurzdor just forgot to change the title)

It's a wrong @klymenko :)

@klymenkoo @liashchynskyi Я переплутав назву комміту і PR з React without JSX :)


docs still shows 2023

We changed the website, but the cloud docs are still 2023:
https://docs.cloud.ploomber.io/en/latest/intro.html
We need to make sure this time everything is pointing to 2024 (on all repos).

this has been fixed


Add Yopass Extension

Description
Add an extension for Yopass. Yopass is a self hosted solution for sharing secrets securely.
Screencast

Checklist

[x] I read the extension guidelines
[x] I read the documentation about publishing
[x] I ran npm run build and tested this distribution build in Raycast
[x] I checked that files in the assets folder are used by the extension itself
[x] I checked that assets used by the README are placed outside of the metadata folder

It could be nice with some Validation since it's possible to send the form without any content added 🙂

Hi @pernielsentikaer thanks for your review 🙂. I added some validation for the message field, so that the field is required now.


elite crash after launching with Minimal

not sure if this is a prtoton issue/update that broke it but i get the following when trying to log run ED with minimal launcher
2021-11-27 02:57:24.301 -07:00 [DBG] Process: /media/games/SteamLibrary/steamapps/common/SteamLinuxRuntime_soldier/_v2-entry-point "--verb=waitforexitandrun" "--" "python3" "/media/games/SteamLibrary/steamapps/common/Proton 6.3/proton" "waitforexitandrun" "/home/marc/Steam NVME/steamapps/common/Elite Dangerous/Products/elite-dangerous-odyssey-64/EliteDangerous64.exe" "ServerToken 79b...278417...0746e4...611dd9aaa37 26f...f24166...bafac8...b81599...ff0bfd...ed3406...d54833...7005aa...e2fa4b...e49380...5ee39c...5180f1...59b34e...567b9a...1f8791...5d2392...0c1cc2...21a06f...aeccb1...1692cd...d86651...2c4873...110ab3...4f14a0...5c55d6...77b978...687aa1...c46682...04e46e...6a17e8...ff89d8...e7ff60...a7905e...5301b2...880b00...e52fa8...a55e23...470189c429aaeb25aedb9921c " /language English\\UK /steam /novr
2021-11-27 02:57:58.512 -07:00 [INF] Shutdown Elite Dangerous: Odyssey`
[min-ed-launcher.log](https://github.com/rfvgyhn/min-ed-launcher/files/7611542/min-ed-launcher.log)
``

added log
[min-ed-launcher.log](https://github.com/rfvgyhn/min-ed-launcher/files/7611544/min-ed-launcher.log)

This might be an issue with the Proton update. I created a new prefix and the game instantly crashes for me when using Proton 6.3. I also can't get the default launcher to run with 6.3. Telling the game to use 6.21-GE works fine with both the min-launcher and the default launcher on my end.
It's possible it doesn't run properly because there has been a change to one of Steam's processes (e.g. reaper, pressure-vessel), but it seems like it's an issue with elite and this version of proton.

when i remove the minimal launcher command and try to run it with only steam (6.21-GE) it starts to load then says running then closes before the main launcher. when i use my usual command for minimal launcher i get the same error as in the prior logs posted. 6.3.8 is what i originally had proton at when i reported the issue.

create an issue on proton to see if there might be something else going on
https://github.com/ValveSoftware/Proton/issues/5353

@ninjaijr Are you still running into this issue?


Error: No named parameter with the name 'shadowThemeOnly'.

See flutter/pull/68922.
Doctor summary (to see all details, run flutter doctor -v):
[√] Flutter (Channel master, 1.24.0-7.0.pre.75, on Microsoft Windows [version 10.0.18363.1139], locale fr-FR)
[√] Android toolchain - develop for Android devices (Android SDK version 30.0.2)
[√] Chrome - develop for the web
[√] Visual Studio - develop for Windows (Visual Studio Enterprise 2019 16.7.7)
[√] Android Studio (version 4.1.0)
[√] IntelliJ IDEA Ultimate Edition (version 2020.2)
[√] VS Code (version 1.50.1)
[√] Connected device (4 available)

• No issues found!

for current run command flutter downgrade to 1.23.0-13.0.pre-79-g0d8696f655
or run command flutter version 1.23.0-13.0.pre-79-g0d8696f655

This is very odd, since the document shows 'shadowThemeOnly' still exists.

I am having the same problem too.

@ComputelessComputer

This is very odd, since the document shows 'shadowThemeOnly' still exists.

On master channel, shadowThemeOnly is removed:
https://master-api.flutter.dev/flutter/material/Theme/of.html
And if you just want to workaround the issue, you can use my fork:
  flutter_datetime_picker:
    git:
      url: https://github.com/espresso3389/flutter_datetime_picker

./../../../.pub-cache/hosted/pub.dartlang.org/flutter_datetime_picker-1.4.0/lib/flutter_datetime_picker.dart:215:48: Error: No named parameter with the name 'shadowThemeOnly'.
ThemeData inheritTheme = Theme.of(context, shadowThemeOnly: true);

@espresso3389
Thank you brother

@espresso3389  Thank you

same issue

same issue

Already [Solved] Please Look at here for all possible solution [Solved] Error: No named parameter with the name ‘keyboardDismissBehavior’

Already [Solved] Please Look at here for all possible solution [Solved] Error: No named parameter with the name ‘keyboardDismissBehavior’

Already [Solved] Please Look at here for all possible solution [Solved] Error: No named parameter with the name ‘keyboardDismissBehavior’

This does not work


Documentation for Tests

Hi is there any available example on how to mock the RedisModule for unit test scenarios?
I tried to setup a CacheModule as follows:
@Module({
  imports: [
    RedisModule.forRootAsync({
      useFactory: (configService: ConfigService) => configService.redisConfig(),
      inject: [ConfigService],
    }),
  ],
  providers: [CacheService],
  exports: [CacheService],
})
export class CacheModule {}

However running unit tests I get the following issue:
  Nest can't resolve dependencies of the CACHE_MANAGER (?). Please make sure that the argument at index [0] is available in the CacheModule context.

      at Injector.lookupComponentInExports (node_modules/@nestjs/core/injector/injector.js:144:19)

I know this is something that I can probably workaround, but I guess it would be a good idea to document the best approach to create a module that handles Nest Test correctly.

Also, after I updated nestjs from 6.2.4 to 6.5.2 I am no longer able to use nestjs-redis
[Nest] 21751   - 2019-07-10 21:46   [ExceptionHandler] Nest can't resolve dependencies of the CacheService (?). Please make sure that the argument at index [0] is available in the CacheModule context. +268642ms
Error: Nest can't resolve dependencies of the CacheService (?). Please make sure that the argument at index [0] is available in the CacheModule context.

Here's my setup:
@Module({
  imports: [
    RedisModule.forRootAsync({
      useFactory: (configService: ConfigService) => configService.redisConfig(),
      inject: [ConfigService],
      imports: [],
    }),
  ],
  providers: [CacheService],
  exports: [CacheService],
})
export class CacheModule {}

RedisCoreModule is a Global Module。
Can I see the "CacheService"'s code?


Don't keep parens around with remove-dbg

Fixes #6355
This causes remove-dbg to not keep parentheses when it comes to ranges though due to ranges not having DOT2 and DOT2EQ tokens but having two DOT tokens inside of macro invocations.

Great corner cases considered, thanks!
bors r+


fix(eventually-postgres): use global serial offset for sequence_number

Hello there :smiley:
Recently, when inserting many aggregates into a repository (done via AggregateRoots), we had a repeated sequence_number in the events table:
database=> select aggregate_id, aggregate_type, version, sequence_number, event->'type' as event_type from events where sequence_number=861;
             aggregate_id             |   aggregate_type   | version | sequence_number | event_type
--------------------------------------+--------------------+---------+-----------------+------------
 7e7704b6-175c-4186-b4ee-3f33ec2f6cd4 | my-type            |       1 |             861 | "Created"
 60742538-8de2-4428-9499-63c74a8990a6 | my-type            |       1 |             861 | "Created"

This unfortunately breaks the projections, as they normally are idempotent and ignore any events equal or earlier than the last processed one. This clearly means that only one of the above events will get processed -> not nice.
The PR fixes the issue (but I'm pretty sure that's not the only possible fix)
As a side-effect it adds a global counter to the events, so that they have an absolute order, whichever aggregate they come from. This sort-of happened before, but only per-aggregate-type, which made some other debugging things more difficult as well

Oww thank you so much for the PR! ❤
Yes, using BIGINT was not a great idea for the sequence_number: ideally I would've used a SERIAL, but tokio-postgres was complaining when performing type deserialization (which is bad, I shouldn't have forced a SQL table design because of Rust library shortcomings...)
This should solve it, although I'm not sure if there might be some issue with projections 👀  (AFAIR, projections/subscriptions were expecting consecutive sequence numbers)
Have you had any issues with "gaps" in sequence_number values on the events table?

(Note: the implementation in eventually-go uses SERIAL already, and gave no issues at all, so your fix is supposed to work pretty fine 🎉 )

Nope, no gaps yet. Also I don't think gaps would be an issue. It might "hint" at missed events on first sight, however it does not mean that afaik. As long as there are no repeated sequence numbers and it's monotonically increasing, all should be fine :)

As for projections/subscriptions: yes exactly, they were expecting consecutive sequence numbers. If a repeated one arrived, it was assumed that it's a duplicate (for example when first reading from eventstore itself and only then looking at the subscriber stream) and was ignored.
Without the PR it would occasionally happen that the same sequence number got assigned to two actually distinct events that even belong to different aggregates -> and that lead to the second of the two events getting ignored, screwing up the projection.
Even worse, the projection would never fix itself, as the number will always be wrong, therefore always ignoring the second passed event (I did not check whether they always get returned in the same order, but it does not really matter either)

As long as there are no repeated sequence numbers and it's monotonically increasing, all should be fine :)

Exactly, since the sequence number is monotonically increasing, and the event store is supposed to return an ordered stream, then gaps should be fine. However...

As for projections/subscriptions: yes exactly, they were expecting consecutive sequence numbers.

...this might be a problem (I mean, the fact of requiring consecutive sequence numbers, not monotonically increasing ones, which is a more relaxed constraint). It seems as though I removed that constraint 😄
https://github.com/eventually-rs/eventually-rs/blob/fcf705ef3e2f72bbeae61d6b75b6a67bcae6bf82/eventually-core/src/subscription.rs#L229-L234

Nope, no gaps yet.

Out of paranoia, I would test this case though 👀
Wouldn't want you having surprises in production 😅

We can merge the PR but I think I need to patch some dependencies in current master (the CI is failing due to eventually-app-example and me not pinning exact crate version 😢)
Would you mind if I rebase on master once I do that?

My bad, projections expect monotonically increasing sequence numbers too as far as I can tell.
And yes, feel free to rebase etc :+1:

FYI added another commit that fixes related and very annoying bug. More info in the commit message

Hello @to266,
I've rebased to latest master and renamed the PR title to conform to Conventional Commits. Hope that's ok with you 😄
Thank you again for your contribution ❤


replace c/c++ code with cython

Eric, have you considered replacing c code part of this project with cython?

The main reason is because this is quite pythonic, hydrid mode, and automatic wrappers exist

Hi Denis,
I took a quick look at Cython not long ago, not for ExcelPython but for other purposes. Do you think it would make sense for ExcelPython? The C++ part of the EP code doesn't actually depend on Python right now... my thinking is that Cython makes more sense for writing .pyd modules, not so much for DLLs which are independent of Python.
Let me know what you think.
Regards,
Eric.

this issue applies to expy more than excelpython i guess.


KOGITO-7514: Patch PR for KOGITO-7425 with doc improvements

JIRA: https://issues.redhat.com/browse/KOGITO-7514

Description: Patch PR for KOGITO-7425 with doc improvements

https://github.com/kiegroup/kogito-docs/pull/45
Please make sure that your PR meets the following requirements:

[x] Pull Request title is properly formatted: KOGITO-XYZ Subject
[x] Pull Request title contains the target branch if not targeting main: [0.9.x] KOGITO-XYZ Subject
[x] The nav.adoc file has a link to this guide in the proper category

@hmanwani-rh Thanks for this PR! The docs look great. Just one minor thing, the Knative team uses the term "plugin" instead of "plug-in". I'm not sure if we want to keep things aligned with their docs (https://knative.dev/docs/client/kn-plugins/).

@ljmotta Thanks for reviewing the changes and sharing the feedback.
Regarding the word plug-in, as per IBM style guide we need to use it as hyphenated.
Here, I am sharing the screenshot of IBM style guide:

Merging this PR.


Show bind errors in js files

Bind errors for JS files should be valid errors, we should show them. The following errors i would expect to be flagged:
let C = "sss";
let C = 0;  // Error: Cannot redeclare block-scoped variable 'C'.

function f() {
    return;
    return;  // Error: Unreachable code detected.
}

function b() {
    "use strict";
    var arguments = 0;  // Error: Invalid use of 'arguments' in strict mode.
}

Isn't the second issue really a linting issue versus a syntax issue?  Also, if you want to trap that as an error, you need to account for break, catch, throw as well as return.  I think jshint, jslint and tslint trap all those already.

Reachability checks are already in master (#4788), they just were not reported for javascript files.


feat(dom): add preventDefault option

This would remove impure preventDefault calls inside the app. Usage:
const event$ = sources.DOM.select('.myform').events('submit', { preventDefault: true });

//currently
const event$ = sources.DOM.select('.myform').events('submit')
    .map(ev => {
        ev.preventDefault();
        return ev;
    });

[ ] I added new tests for the issue I fixed/built
[x] I ran make test FOO for the package FOO I'm modifying
[x] I used make commit instead of git commit

Hey Jan :)
Thanks for submitting this PR. Before merging it, though, I noticed that it's the simplest solution, it is likely we would refactor it later on (for instance, I can see how we would put the preventDefault call inside the EventDelegator or the fromEvent producer), for performance or for less use of tricky mutation. In that future scenario where this refactor happens, it would be important to make sure that the code still works as desired. So we need tests for this one. Usually I'm not strict on asking for tests for everything, but I think this is one of those cases where we need it. The code inside MainDOMSource.events() is very tricky and likely to be refactored.

How would I test this?

You could check that event.defaultPrevented is true whenpreventDefault is true and false otherwise.
https://developer.mozilla.org/en-US/docs/Web/API/Event/defaultPrevented

The CI fails because it was not able to start the iOS browser, the new tests works as expected

Triggering a new build worked :)

Great, thanks for the PR @jvanbruegge !

This is perfect for cycle/sandbox :)


Migrated notification module from Java to Kotlin

Fixes: #5954
Part of: #5928

@nicolas-raoul PTAL


[spi_device] Mux in missing scan_rst_ni connectivity

The newer out_clk-synchronized resets were missing direct connectivity for scan reset, as was the CSR-connected reset to the prim_fifo_async used for flash status register updates moving from SW to the SPI clock domain. Add the requisite muxes to patch that up.

Merging. The failure is unrelated and was fixed by a separate PR (already merged).


linking error: /usr/bin/x86_64-linux-gnu-ld: cannot find -lmkl_intel_ilp64

Hi, I follow the README file to build this project,
I'm using:
Ubuntu 18.04.6 LTS
cuda 11.7
python 3.8.0
g++-6
torch 2.0.1+cu117
I ran
cmake -DTORCH_CUDA_ARCH_LIST="8.0" -DDataType=FP16 -DBUILD_THS=ON -DCUDAARCHS="80"  ..
then make.
I encounter the following error:

Consolidate compiler generated dependencies of target bytetransformer
[ 66%] Built target bytetransformer
Consolidate compiler generated dependencies of target ths_bytetransformer
[ 72%] Linking CXX shared library ../../lib/libths_bytetransformer.so
/usr/bin/x86_64-linux-gnu-ld: cannot find -lmkl_intel_ilp64
/usr/bin/x86_64-linux-gnu-ld: cannot find -lmkl_core
/usr/bin/x86_64-linux-gnu-ld: cannot find -lmkl_intel_thread
collect2: error: ld returned 1 exit status
bytetransformer/th_op/CMakeFiles/ths_bytetransformer.dir/build.make:166: recipe for target 'lib/libths_bytetransformer.so' failed
make[2]: *** [lib/libths_bytetransformer.so] Error 1
CMakeFiles/Makefile2:176: recipe for target 'bytetransformer/th_op/CMakeFiles/ths_bytetransformer.dir/all' failed
make[1]: *** [bytetransformer/th_op/CMakeFiles/ths_bytetransformer.dir/all] Error 2
Makefile:90: recipe for target 'all' failed
make: *** [all] Error 2

I check the CMakeLists files, but I can't find anything linked to mkl libraries. Any clue on this?

It looks that Intel MKL wasn't correctly configured in your system.
Would you please try to built on docker nvcr.io/nvidia/pytorch:23.05-py3 ?
I tested the building and things went well in this docker.

It looks that Intel MKL wasn't correctly configured in your system. Would you please try to built on docker nvcr.io/nvidia/pytorch:23.05-py3 ? I tested the building and things went well in this docker.

Thanks, Yujia. An alternative is to run the setvars.sh scripts. In my case:
source /opt/intel/oneapi/setvars.sh --force intel64 mod


Module named "Keras" is not defined

I already installed "keras" and used import function in main.py but when I tried to train SRGAN with MSE this happened.

@nickhdfan, did you managed to solve this? I'm having the same problem. Supposably only tensorflow was needed.

Add the following to ops.py
import keras
apparently keras is actually a dependency for vgg54 training. If you are planning on training vgg54, you will also have to make the change to the following lines of model.py starting at around line 335. change them to look like this:
if type == 'VGG54':
    target_layer = 'vgg_19/conv5/conv5_4'
elif type == 'VGG22':
    target_layer = 'vgg_19/conv2/conv2_2'

Hi, I have the same error, and the proposed solution didn't work for me, instead I get this new error:
InvalidArgumentError (see above for traceback): Cannot assign a device for operation 'dicriminator_train/discriminator/discriminator_unit/dense_layer_2/dense/bias/Adam_1': Operation was explicitly assigned to /device:GPU:0 but available devices are [ /job:localhost/replica:0/task:0/device:CPU:0 ]. Make sure the device specification refers to a valid device.
	 [[Node: dicriminator_train/discriminator/discriminator_unit/dense_layer_2/dense/bias/Adam_1 = VariableV2[_class=["loc:@discriminator/discriminator_unit/dense_layer_2/dense/bias"], container="", dtype=DT_FLOAT, shape=[1], shared_name="", _device="/device:GPU:0"]()]]

Someone knows why this happens and how to solve it?

i meet the same question, pip install keras, but doesn't work.
AttributeError: 'module' object has no attribute 'leaky_relu'

@adeoy I believe that you have either installed regular tensorflow (not tensorflow-gpu) or you set CUDA_VISIBLE_DEVICES incorrectly when running the training script.
 So, I cant do the next training without a GPU? Or how can I solve this problem?

Still have the same issue, anyone found a solution?

I am not sure if it is a solution, but what I did was go into model.py and change
with tf.device('/gpu:0'):
to
with tf.device('/cpu:0'):
xD

You need to add "import keras" in lib/ops.py

I had the same issue during the second step training (SRGAN with MSE loss) and it was solved by adding "import keras" in lib/ops.py

yes👍👍" import keras " best solution!!❤❤


Adding icon for the editor to Linux DE launchers

Operating system or device - Godot version:
Ubuntu 16.04.1 | Godot 2.1-stable
Issue description (what happened, and what was expected):
Godot icon is missing from the Unity launcher (grey with a question mark)
Also i try to  "Lock to launcher" but nothing it's disappear after exiting Godot.

No idea how to fix this, anyone familiar with Ubuntu?

@Chrisanthos you need to install Godot if you want it to integrate with your system.
The easiest way to do that will probably be by installing godot via the Ubuntu Software Center, or via apt.
Otherwise, you should at the very least create a Godot.desktop file and place that in /usr/share/applications. In that .desktop file, point to the Godot binary as what should be launched (it's common to place extra binaries like this in /opt), and point to a location on your harddrive where the Godot icon can be located.
Ubuntu will then be able to use the Godot icon for the launcher (unity or otherwise) and the .desktop file as the basis for a launcher.

The easiest way to do that will probably be by installing godot via the Ubuntu Software Center, or via apt.

I don't think it'll be available in official Debian/Ubuntu repositories anytime soon. Maybe someone could maintain a PPA for easy installation on Ubuntu, though.
That said, for proper desktop integration, you will need to write a .desktop file and put it in ~/.local/share/applications.
Something like this (modify the Icon path accordingly):
[Desktop Entry]
Type=Application
Name=Godot Engine
GenericName=Game engine
Comment=2D and 3D game engine
Exec=/usr/local/bin/godot
Icon=/absolute/path/to/icon/file

@Calinou  I have try it but not working.
@notklaatu Can you give me steps how can i do it?

@Chrisanthos  sure, here, I wrote this for you:
http://straightedgelinux.tk/unix/desktop/?dir=unix

I followed (another guide like yours), @notklaatu, and I guess I'm halfway from the solution. I mean, at the moment I have a godot.desktop file, displayed with name "Godot", and I put it in the Unity Launcher. When I click on it, the project management window of Godot opens, but a new icon (grey with question mark) appears in the launcher as well. When I choose the project to edit or create, this icon disappears as the window closes, and, more or less in the same moment, another identical one appears, with the project window opening.
To sum up, it's pretty much like this pretty icon I put in the launcher just allow me to open the godot, but it's not strictly related to it.

Here is the solution, found in this answer on AskUbuntu.
Opening Godot the project list opens; entering xprop|grep WM_CLASS in a terminal and then clicking on the Project List window, one obtains:
WM_CLASS(STRING) = "Godot_ProjectList", "Godot"
so adding the line StartupWMClass=Godot_ProjectList to the .desktop file allow "merging" the icon of the project list with the cust icon we created. But this is not what we want, since when we choose a project, a second question mark icon appears, whereas the first one would disappear. So it's better to use this line, insted:
WM_CLASS(STRING) = "Godot_Editor", "Godot"
Doing so, the question mark icon of the project list appears, but when we chose a project it disappears, and the question mark icon of the editor is captured by the custom icon we created.

As I understand it there's not much that can be done by Godot itself to fix this, no? The StartupWMClass solution described by @Aster89 is nice, though it would have to be applied by package maintainers (I'll have to give it a try, though for my Mageia package I don't have icon issues, but I guess /usr/bin/godot matches the desktop file automatically).

Here is a set of scripts I'm using to "install" and "uninstall" Godot:

install script: http://pastebin.com/eB4nsr65
desktop template: http://pastebin.com/Zn8dqm5c
uninstall script: http://pastebin.com/VyJbWx2z

Files must be on same directory of the executable, in my case executable is Godot.64.
There must be the icon, in my case godot_alt.png (alt(ernative) because the original icon is flat,
so you visually get a square icon with eyes, make use an icon with an outline...)
This logic was found on Arduino IDE install scripts (as fallback, not using xdg-utils).

Create a .desktop file named godot.desktop with content -
[Desktop Entry]
Name=Godot Engine
Comment=Free and Open Source 2D and 3D Game Engine
Terminal=false
Type=Application
Categories=Development;IDE;
Keywords=game;engine;ide;
Exec=/complete/path/to/godot/executeable/for/linux
Icon=/complete/path/to/godot/icon/file/with/png_or_svg_or_else_type

[ change the Exec and Icon entries path according to your file(s) location(s) ]
[ you can find an icon file from GitHub repository, icon.png ]
Copy/Move the file into ~/.local/share/applications/
Make the godot.dektop file executable.
[ Using GUI method ]
[ Or, using command [chmod +x ~/.local/share/applications/godot.desktop] ]
[ DO NOT COPY-PASTE/USE COMMAND(S) FROM UNKNOWN SOURCE(S) ]
[ USE GUI METHOD INSTEAD ]
Now, you should see an entry named Godot Engine in Unity Launcher.
It is possible that, you might not see the icon image properly.
But if you have used proper path for the image, logging out-in or, restarting computer should solve this.
[ for new linux user(s), if you don't understand the ~/ place, it presents the Home folder for current user ]
Thanks :)

This can be closed, I think

A cool solution to this problem would certainly be to add a pythonized version of https://github.com/godotengine/godot/issues/6405#issuecomment-265549669 as install/uininstall target in the Sconscript.

First of all thank you for your report and sorry for the delay.
We released Godot 3.0 in January 2018 after 18 months of work, fixing many old issues either directly, or by obsoleting/replacing the features they were referring to.
We still have hundreds of issues whose relevance/reproducibility needs to be checked against the current stable version, and that's where you can help us.
Could you check if the issue that you described initially is still relevant/reproducible in Godot 3.0 or any newer version, and comment about it here?
For bug reports, please also make sure that the issue contains detailed steps to reproduce the bug and, if possible, a zipped project that can be used to reproduce it right away. This greatly speeds up debugging and bugfixing tasks for our contributors.
Our Bugsquad will review this issue more in-depth in 15 days, and potentially close it if its relevance could not be confirmed.
Thanks in advance.
Note: This message is being copy-pasted to many "stale" issues (90+ days without activity). It might happen that it is not meaningful for this specific issue or appears oblivious of the issue's context, if so please comment to notify the Bugsquad about it.

.desktop file for linux can be found here https://github.com/godotengine/godot/tree/master/misc/dist/linux

As mentioned above that's not something that we can fix ourselves on Linux, executable don't have icon data. The files in https://github.com/godotengine/godot/tree/master/misc/dist/linux can be used for one's own local setup, or by package maintainers. Adding AppImage support as a distribution channel may also be an option, but it's tracked in #6681.

To install for multiple users on the computer (a family computer) I did the following on Kubuntu 20.04:

First download the binary from https://godotengine.org/download/linux and extract it.  In my case the file name was Godot_v3.2.1-stable_x11.64 but that will be different in the future.

Put the binary in /usr/local/lib/ and fix the ownership.
sudo mv Godot_v3.2.1-stable_x11.64 /usr/local/lib
sudo chown root:root /usr/local/lib/Godot_v3.2.1-stable_x11.64

Add a symlink so the program can be started just by running godot.
sudo ln -s /usr/local/lib/Godot_v3.2.1-stable_x11.64 /usr/local/bin/godot

Install the application icon in the appropriate directory per the "Installing Application Icons" section of https://specifications.freedesktop.org/icon-theme-spec/icon-theme-spec-latest.html
sudo wget -nc -P /usr/share/icons/hicolor/scalable/apps https://godotengine.org/themes/godotengine/assets/download/godot_logo.svg

Added a PNG version of the icon also, as per recommendation in the above link, using Inkscape to convert the SVG to the right size per https://stackoverflow.com/a/14174624/5419599 (I have Inkscape 1.0).  (This step could probably be skipped and the icon would still show up from the SVG version; I didn't test.)
inkscape -w 48 -h 48 /usr/share/icons/hicolor/scalable/apps/godot_logo.svg --export-filename godot_logo.png
# And place it appropriately
sudo mv godot_logo.png /usr/share/icons/hicolor/48x48/apps/
sudo chown root:root /usr/share/icons/hicolor/48x48/apps/godot_logo.png

Make the launcher file as described in many comments here.  Note that for mine, the Exec and Icon lines are especially simple because of appropriately placing the binary and the icon earlier.  I also grabbed the version string from the binary.
# Grab version string
version="$(/usr/local/bin/godot --version)"
# Create godot.desktop file
cat > godot.desktop << EOF
[Desktop Entry]
Version=$version
Name=Godot
Exec=godot
Icon=godot_logo
Terminal=false
Type=Application
Categories=Application;Development;
Comment=Free and open source 2D and 3D game engine
Keywords=engine;game engine;
EOF
# And place it correctly.
sudo mv godot.desktop /usr/share/applications/
sudo chown root:root /usr/share/applications/godot.desktop

Godot then shows up in the application menu, or when searched for, just as though it had been installed from any of the various graphical software installers (e.g. Software Center):

@mikeweilgart Note that we provide a .desktop file already: https://github.com/godotengine/godot/blob/master/misc/dist/linux/org.godotengine.Godot.desktop

The Unity launcher and icons for .desktop files (which appear in the Dash AKA Start Menu) are orthogonal concepts. The launcher icon comes from the window's icon. With wxPython that's set with wx.TopLevelWindow.SetIcon. That works straight from Python with no installation or a .desktop file. I don't know the equivalent for godot's toolkit but I suspect there is one, as GTK and Qt can do it too.


Add noreferrer + noopener attributes to blank targets

Increases privacy by requiring supported browsers to not send a
referer header. Also prevents tabnabbing.

@MaxXor AFAICT, tabnabbing works with same-origin and cross-origin policies, and we would still need noopener. What's your solution exactly?

You are right, I forgot tabnabbing. Not sure if a simple http header can fix it for external URLs.

Please rebase

@fluffypony rebased.


Add ability to use local Arcgis geocoder

Instead of just the Arcgis World geocoder which is a paid service.
If your organization already has a Arcgis server with a geocoder optimized to your reqts, you should be able to use that.

I am also looking to be able to use my own arcgis locator hosted on my arcgis server. My main concern is location accuracy, our local locator has precise point address locations and so I would rather use this one instead of the world geocoder.

It would be better to do this on top of #227, which is not merged yet


Build agent instances terminated after 1 hour

Hi,
Since upgrading to openstack-cloud 2.35 from 2.29 we're noticing that all build agents which aren't configured to be retained indefinitely get terminated after around 1 hour and 10 minutes, regardless of configured retention time.
I can't see anything obvious in the logs. I can only speculate that it's getting terminated in the background and then the cleanup thread is subsequently noticing the termination and cleaning up the instances because they no longer exist.
Started OpenStack slave cleanup
Jun 21, 2018 4:43:45 PM FINE jenkins.plugins.openstack.compute.internal.Openstack debug
Openstack client created for "https://openstack.our-org.com:5000/v3", "jenkins:jenkins_user:default", ..., "RegionOne".
Jun 21, 2018 4:43:46 PM WARNING jenkins.plugins.openstack.compute.JCloudsCleanupThread terminatesNodesWithoutServers
No OpenStack server running for computer build-agent-3910. Terminating.
Jun 21, 2018 4:43:46 PM INFO jenkins.plugins.openstack.compute.JCloudsComputer deleteSlave
Deleting slave build-agent-3910

Instances appear to be being created with the appropriate retention time and I can't see anything else in the logs that points to why they'd be being deleted after this amount of time.
Rolling back doesn't seem to fix this issue. It's a huge problem for us because builds are being killed mid-way through.
If there's anything else I can do in order to dig deeper into this and provide more information, please let me know.

No OpenStack server running for computer build-agent-3910. Terminating.

Suggests OpenStack servers are not running. In theory, It can be accidentally deleted by the plugin because of a bug somewhere. Are there any other logs relevant to those servers? Anything that would indicate machine is deleted? Also, can you verify the servers are not deleted by some other way?

FTR, the plugin should be able to handle that provided they have different Jenkins URL configured. Otherwise they would step on each other's toes.


latest changes from m2k workflow

updated SWF to latest changes in dev repo
reviewed application properties removing all that is not needed because managed by the operator

\lgtm


[tutorial] Wording awkward in Quickstart regarding placement in "layout/step-1"

This is feedback that could be ignore if desired.
For Lore Quickstart, the placement instruction at this juncture have a continuity quirk. The instructions say:

Now add the following scripts at the bottom of the  tag, immediately above the bundle.js script.

source: http://www.lorejs.org/quickstart/layout/step-1/
But there isn't a "bundle.js" yet in index.html.
There is an HTML comment that mentions the script tags for the /bundle.{vendor,main}.{:hash}.js. So that might be what is meant for anchoring the action.
Any confusion that could occur is most likely eliminated thanks clarifications in "Code Changes" section.

I'm almost willing to "self-close" this. Sorry ... it's the former teacher in me that decided to write it up. All the other aspects so far as done with such care and clarity, I just thought I'd mention it.

@lenards glad you did! It's a good callout. I'll leave this open and fix that soon to replace the text with something like:

Now add the following scripts to the bottom of the <body> tag.

That works!

Closing as this has been fixed in the new website by making the suggested change.