Move to Python 3

Creating this as a catch-all ticket to elicit some discussion of Python3 support, which has become something of an FAQ.
We're going to have to make the jump to Python3 at some point. All our dependencies (last time I checked) were Python 3-ready, so in theory we could make the leap. If we do it, we would do Python 3-only - I don't think there's a benefit to trying to maintain Python 2/3 Frankenstein that gives us the worst of both worlds.
This leaves us with two issues. First, there's a legacy concern - quite a few people use libmproxy to build things, and I know that I personally would probably have a few weeks of work ahead of me just in converting existing related projects over.
The second thing has to do with Python 3 and its treatment of Unicode/strings. Mitmproxy is "special", in the sense that it deals with loosely typed, untrusted and possibly intentionally corrupted data flows. It's a boundary layer between a weird world where string encodings can't be assumed, and the higher levels like the UI and so forth, where Python 3 forces us to have unicode. In this sense, it's similar to other programs that deal with on-the-wire data, like web frameworks (though our problem is even worse). Have a look at Armin Ronacher's experience with Python 3 Unicode and Flask/Jinja2/etc.:
http://lucumr.pocoo.org/2014/1/5/unicode-in-2-and-3/
http://lucumr.pocoo.org/2014/5/12/everything-about-unicode/
I know that this has frustrated some people working on similar programs enough that they've moved away from Python altogether....
Where does this leave us? Well, clearly we clearly can't stay on Python2 in the longer term. The Python3 situation is pretty dire, but has very gradually been getting very slightly better. 3.4 included some improvements, and I think 3.5 will have a few more. The key issues are intrinsic to the language design, though, and will never go away.

fwiw - I'm also a proponent of moving to Python 3 as well and leaving 2 in the dust.  I did a couple of installs a few months ago on CentOS 6.x and had the interesting problem of them only having Python 2.6 available by default. Manually upgrading to 2.7 was a pain and it would have been just as easy to move directly to 3 (probably easier actually).  Legacy support will always be an important issue, but moving forward is just as important (e.g. http://fedoraproject.org/wiki/Changes/Python_3_as_Default https://wiki.ubuntu.com/Python/3 )
P.S. Docker's popularity is also incidentally helping solve this problem - at least for Linux servers.  Supporting a newer Python version for an individual application is a piece of cake using containers/Docker.

A note for readers of this ticket - we're now waiting on PyInstaller to have a mature Python3 branch before we move. It's either that, or begin to roll our own distribution packages for major platforms, which would be a major timesink.

@cortesi thanks for info. Please keep us up :)

I looked into python 2->3 porting over the weekend, and also watched some of the pycon 2014 videos on the topic.
If we were just to move 2.x in the dust, how would it get done?  Code freeze while it gets ported?  Just use python’s 2to3 and fix stuff that breaks?  Or fork and then backport?
I think I’d like to try to make the case for python 2 & 3 compatible code in the same codebase.  The feeling I got was that so much work has gone into python 2.6 and 2.7 around backporting Python 3 stuff that it is often easier to clean up your code than to do a hard break and that python 3 seem to be at a point where there are decent tools that help you write agnostic code.  I had no idea for example that if you do a “python -3 mitmproxy” under recent-ish versions of 2.x, it gives you debug output, about code that it comes across that needs to be addressed for python 3.x.
The other things that seemed to be major roadblocks to porting, were things like CPython extentions, which we don’t do, and not having good testing which we I think we have.  Can Travis help us out in this manner?  I agree that the strings handling vs byte/unicode handling, is probably the biggest problem, but that isn’t going to be solved by dropping 2.x support is it?
references:
http://python3porting.com/
https://www.youtube.com/watch?v=nx0sXSeJbmI
https://www.youtube.com/watch?v=f_6vDi7ywuA

So, there would be a preparatory stage where we get ready for a Python 3 migration by getting rid of Python 2-isms, migrating to backported features, and bring our unit test suite back up to 100%, and so forth. Then there would be a flagfall with a code freeze, and after a few days of effort we'd emerge as a Python 3 only project. I don't think maintaining a 2&3 compatible codebase is worth the effort, unless you're a library has a user constituency in both camps. We're in the relatively luxurious position where we can actually make a complete shift without annoying too many people, and in fact, without most of our users noticing.
I did a survey recently, and ALL our dependencies seem to be Python 3-ready. The one exception is PyInstaller, a perennial bugbear. Its Python3 branch (https://github.com/pyinstaller/pyinstaller/tree/python3) doesn't seem mature, and is getting less development attention than the main branch. I'm watching to see how that shakes out.

I'd propose something along these lines:

Make netlib/pathod cross-compatible with Python 2 and 3, mark all workarounds for Python 2. The expected outcome is that the netlib/pathod tests run fine for Python 2 and 3 while mitmproxy is still Python 2 only. This could be done without freezing anything - the required workarounds shouldn't be too severe.
Move mitmproxy to Python 3 - we should see whether we can retain Python 2 compatibility for the first adjustments, but we should not be resistant to do a code freeze here.
Ditch Python 2 for everything, remove workarounds.

If you want to help, I'd be happy to see some initial work on netlib/pathod that makes it cross-compatible and could be immediately merged into master. My start with the python3 branch was nice to explore the situation, but keeping Pyhton 2 compatibility here for a bit might be useful to make the transition smooth.

That might be worth the effort as an intermediate step. We should definitely wait until we have a plausible path forward with PyInstaller before we move on this - a large fraction of our users rely on this at the moment.

Hi Max (mhils)
I was just reading your comment about mitmproxy only being supported up to Python 2.7. I was wondering whether this was still the case, or if it had been updated to support the latest Python release?

No news yet - I'd be happy to see someone start with netlib. :wink:

Hi Max
Thanks for replying to my comment.
I've just installed Python 2.7.10, however I'm still having problems installing mitmproxy.
This is the full response from Command Prompt after typing in the command 'pip install mitmproxy'
Collecting mitmproxy
Using cached mitmproxy-0.12.1.tar.gz
Collecting pyperclip>=1.5.8 (from mitmproxy)
Using cached pyperclip-1.5.11.zip
Collecting pyasn1>0.1.2 (from mitmproxy)
Using cached pyasn1-0.1.7.tar.gz
Collecting tornado>=4.0.2 (from mitmproxy)
Using cached tornado-4.2.tar.gz
Collecting netlib<0.13,>=0.12 (from mitmproxy)
Using cached netlib-0.12.1.tar.gz
Complete output from command python setup.py egg_info:
warning: no files found matching 'OpenSSL\RATIONALE'
warning: no previously-included files found matching 'leakcheck'
warning: no previously-included files matching '.py' found under directory 'leakcheck'
warning: no previously-included files matching '.pem' found under directory 'leakcheck'
warning: no previously-included files matching '*.pyc' found anywhere in distribution
no previously-included directories found matching 'doc_build'
zip_safe flag not set; analyzing archive contents...
Installed c:\users\admini~1\appdata\local\temp\pip-build-mz8r00\netlib.eggs\pyopenssl-0.15.1-py2.7.egg
Searching for cffi
Reading https://pypi.python.org/simple/cffi/
Best match: cffi 1.1.0
Downloading https://pypi.python.org/packages/source/c/cffi/cffi-1.1.0.tar.gz#md5=b58d43a708e757f63a905c6a0d9ecf7a
Processing cffi-1.1.0.tar.gz
Writing c:\users\admini~1\appdata\local\temp\easy_install-27xoru\cffi-1.1.0\setup.cfg
Running cffi-1.1.0\setup.py -q bdist_egg --dist-dir c:\users\admini~1\appdata\local\temp\easy_install-27xoru\cffi-1.1.0\egg-dist-tmp-p10cca
Traceback (most recent call last):
File "", line 20, in
File "c:\users\admini~1\appdata\local\temp\pip-build-mz8r00\netlib\setup.py", line 87, in
"install": CFFIInstall,
File "C:\Python27\lib\distutils\core.py", line 111, in setup
setup_distribution = dist = klass(attrs)
File "C:\Python27\lib\site-packages\setuptools\dist.py", line 268, in init
self.fetch_build_eggs(attrs['setup_requires'])
File "C:\Python27\lib\site-packages\setuptools\dist.py", line 313, in fetch_build_eggs
replace_conflicting=True,
File "C:\Python27\lib\site-packages\pkg_resources_init.py", line 830, in resolve
dist = best[req.key] = env.best_match(req, ws, installer)
File "C:\Python27\lib\site-packages\pkg_resources_init_.py", line 1075, in best_match
return self.obtain(req, installer)
File "C:\Python27\lib\site-packages\pkg_resources_init_.py", line 1087, in obtain
return installer(requirement)
File "C:\Python27\lib\site-packages\setuptools\dist.py", line 380, in fetch_build_egg
return cmd.easy_install(req)
File "C:\Python27\lib\site-packages\setuptools\command\easy_install.py", line 629, in easy_install
return self.install_item(spec, dist.location, tmpdir, deps)
File "C:\Python27\lib\site-packages\setuptools\command\easy_install.py", line 659, in install_item
dists = self.install_eggs(spec, download, tmpdir)
File "C:\Python27\lib\site-packages\setuptools\command\easy_install.py", line 842, in install_eggs
return self.build_and_install(setup_script, setup_base)
File "C:\Python27\lib\site-packages\setuptools\command\easy_install.py", line 1070, in build_and_install
self.run_setup(setup_script, setup_base, args)
File "C:\Python27\lib\site-packages\setuptools\command\easy_install.py", line 1058, in run_setup
raise DistutilsError("Setup script exited with %s" % (v.args[0],))
distutils.errors.DistutilsError: Setup script exited with error: Microsoft Visual C++ 9.0 is required (Unable to find vcvarsall.bat). Get it from http://aka.ms/vcpython27

Here is the updated command prompt response to 'pip install mitmproxy'
Collecting mitmproxy
Using cached mitmproxy-0.12.1.tar.gz
Collecting pyperclip>=1.5.8 (from mitmproxy)
Using cached pyperclip-1.5.11.zip
Collecting pyasn1>0.1.2 (from mitmproxy)
Using cached pyasn1-0.1.7.tar.gz
Collecting tornado>=4.0.2 (from mitmproxy)
Using cached tornado-4.2.tar.gz
Collecting netlib<0.13,>=0.12 (from mitmproxy)
Using cached netlib-0.12.1.tar.gz
Complete output from command python setup.py egg_info:
warning: no files found matching 'OpenSSL\RATIONALE'
warning: no previously-included files found matching 'leakcheck'
warning: no previously-included files matching '.py' found under directory 'leakcheck'
warning: no previously-included files matching '.pem' found under directory 'leakcheck'
warning: no previously-included files matching '*.pyc' found anywhere in distribution
no previously-included directories found matching 'doc_build'
zip_safe flag not set; analyzing archive contents...
Installed c:\users\admini~1\appdata\local\temp\pip-build-en7rdx\netlib\.eggs\pyopenssl-0.15.1-py2.7.egg
Searching for cffi
Reading https://pypi.python.org/simple/cffi/
Best match: cffi 1.1.0
Downloading https://pypi.python.org/packages/source/c/cffi/cffi-1.1.0.tar.gz#md5=b58d43a708e757f63a905c6a0d9ecf7a
Processing cffi-1.1.0.tar.gz
Writing c:\users\admini~1\appdata\local\temp\easy_install-2gta8y\cffi-1.1.0\setup.cfg
Running cffi-1.1.0\setup.py -q bdist_egg --dist-dir c:\users\admini~1\appdata\local\temp\easy_install-2gta8y\cffi-1.1.0\egg-dist-tmp-qh08ku
_cffi_backend.c
ffi.c
prep_cif.c
types.c
win32.c
   Creating library build\temp.win32-2.7\Release\c\_cffi_backend.lib and object build\temp.win32-2.7\Release\c\_cffi_backend.exp
creating c:\users\admini~1\appdata\local\temp\pip-build-en7rdx\netlib\.eggs\cffi-1.1.0-py2.7-win32.egg
Extracting cffi-1.1.0-py2.7-win32.egg to c:\users\admini~1\appdata\local\temp\pip-build-en7rdx\netlib\.eggs

Installed c:\users\admini~1\appdata\local\temp\pip-build-en7rdx\netlib\.eggs\cffi-1.1.0-py2.7-win32.egg
Searching for six>=1.5.2
Reading https://pypi.python.org/simple/six/
Best match: six 1.9.0
Downloading https://pypi.python.org/packages/source/s/six/six-1.9.0.tar.gz#md5=476881ef4012262dfc8adc645ee786c4
Processing six-1.9.0.tar.gz
Writing c:\users\admini~1\appdata\local\temp\easy_install-plgsfd\six-1.9.0\setup.cfg
Running six-1.9.0\setup.py -q bdist_egg --dist-dir c:\users\admini~1\appdata\local\temp\easy_install-plgsfd\six-1.9.0\egg-dist-tmp-w5xsci
no previously-included directories found matching 'documentation\_build'
zip_safe flag not set; analyzing archive contents...
six: module references __path__
creating c:\users\admini~1\appdata\local\temp\pip-build-en7rdx\netlib\.eggs\six-1.9.0-py2.7.egg
Extracting six-1.9.0-py2.7.egg to c:\users\admini~1\appdata\local\temp\pip-build-en7rdx\netlib\.eggs

Installed c:\users\admini~1\appdata\local\temp\pip-build-en7rdx\netlib\.eggs\six-1.9.0-py2.7.egg
Searching for cryptography>=0.7
Reading https://pypi.python.org/simple/cryptography/
Best match: cryptography 0.9.1
Downloading https://pypi.python.org/packages/source/c/cryptography/cryptography-0.9.1.tar.gz#md5=6c45d87896f5155f25cdc4d7e0a57526
Processing cryptography-0.9.1.tar.gz
Writing c:\users\admini~1\appdata\local\temp\easy_install-jvjamx\cryptography-0.9.1\setup.cfg
Running cryptography-0.9.1\setup.py -q bdist_egg --dist-dir c:\users\admini~1\appdata\local\temp\easy_install-jvjamx\cryptography-0.9.1\egg-dist-tmp-rh7i8c
_cffi_backend.c
ffi.c
prep_cif.c
types.c
win32.c
   Creating library build\temp.win32-2.7\Release\c\_cffi_backend.lib and object build\temp.win32-2.7\Release\c\_cffi_backend.exp

Installed c:\users\admini~1\appdata\local\temp\easy_install-jvjamx\cryptography-0.9.1\.eggs\cffi-1.1.0-py2.7-win32.egg
Searching for ipaddress
Reading https://pypi.python.org/simple/ipaddress/
Best match: ipaddress 1.0.7
Downloading https://pypi.python.org/packages/source/i/ipaddress/ipaddress-1.0.7.tar.gz#md5=5d9ecf415cced476f7781cf5b9ef70c4
Processing ipaddress-1.0.7.tar.gz
Writing c:\users\admini~1\appdata\local\temp\easy_install-jvjamx\cryptography-0.9.1\temp\easy_install-s1gt3k\ipaddress-1.0.7\setup.cfg
Running ipaddress-1.0.7\setup.py -q bdist_egg --dist-dir c:\users\admini~1\appdata\local\temp\easy_install-jvjamx\cryptography-0.9.1\temp\easy_install-s1gt3k\ipaddress-1.0.7\egg-dist-tmp-mxwd0g
zip_safe flag not set; analyzing archive contents...
Moving ipaddress-1.0.7-py2.7.egg to c:\users\admini~1\appdata\local\temp\easy_install-jvjamx\cryptography-0.9.1\.eggs

Installed c:\users\admini~1\appdata\local\temp\easy_install-jvjamx\cryptography-0.9.1\.eggs\ipaddress-1.0.7-py2.7.egg
Searching for enum34
Reading https://pypi.python.org/simple/enum34/
Best match: enum34 1.0.4
Downloading https://pypi.python.org/packages/source/e/enum34/enum34-1.0.4.zip#md5=9843e97527f3126c851df7afeb0524b3
Processing enum34-1.0.4.zip
Writing c:\users\admini~1\appdata\local\temp\easy_install-jvjamx\cryptography-0.9.1\temp\easy_install-_3imtl\enum34-1.0.4\setup.cfg
Running enum34-1.0.4\setup.py -q bdist_egg --dist-dir c:\users\admini~1\appdata\local\temp\easy_install-jvjamx\cryptography-0.9.1\temp\easy_install-_3imtl\enum34-1.0.4\egg-dist-tmp-ch2bdl
zip_safe flag not set; analyzing archive contents...
Moving enum34-1.0.4-py2.7.egg to c:\users\admini~1\appdata\local\temp\easy_install-jvjamx\cryptography-0.9.1\.eggs

Installed c:\users\admini~1\appdata\local\temp\easy_install-jvjamx\cryptography-0.9.1\.eggs\enum34-1.0.4-py2.7.egg
Searching for six>=1.4.1
Reading https://pypi.python.org/simple/six/
Best match: six 1.9.0
Downloading https://pypi.python.org/packages/source/s/six/six-1.9.0.tar.gz#md5=476881ef4012262dfc8adc645ee786c4
Processing six-1.9.0.tar.gz
Writing c:\users\admini~1\appdata\local\temp\easy_install-jvjamx\cryptography-0.9.1\temp\easy_install-sfelkn\six-1.9.0\setup.cfg
Running six-1.9.0\setup.py -q bdist_egg --dist-dir c:\users\admini~1\appdata\local\temp\easy_install-jvjamx\cryptography-0.9.1\temp\easy_install-sfelkn\six-1.9.0\egg-dist-tmp-prgjst
no previously-included directories found matching 'documentation\_build'
zip_safe flag not set; analyzing archive contents...
six: module references __path__
creating c:\users\admini~1\appdata\local\temp\easy_install-jvjamx\cryptography-0.9.1\.eggs\six-1.9.0-py2.7.egg
Extracting six-1.9.0-py2.7.egg to c:\users\admini~1\appdata\local\temp\easy_install-jvjamx\cryptography-0.9.1\.eggs

Installed c:\users\admini~1\appdata\local\temp\easy_install-jvjamx\cryptography-0.9.1\.eggs\six-1.9.0-py2.7.egg
Searching for pyasn1
Reading https://pypi.python.org/simple/pyasn1/
Best match: pyasn1 0.1.7
Downloading https://pypi.python.org/packages/2.7/p/pyasn1/pyasn1-0.1.7-py2.7.egg#md5=15f079cabee01402bf86ca8b83356469
Processing pyasn1-0.1.7-py2.7.egg
Moving pyasn1-0.1.7-py2.7.egg to c:\users\admini~1\appdata\local\temp\easy_install-jvjamx\cryptography-0.9.1\.eggs

Installed c:\users\admini~1\appdata\local\temp\easy_install-jvjamx\cryptography-0.9.1\.eggs\pyasn1-0.1.7-py2.7.egg
Searching for idna
Reading https://pypi.python.org/simple/idna/
Best match: idna 2.0
Downloading https://pypi.python.org/packages/source/i/idna/idna-2.0.tar.gz#md5=bd17a9d15e755375f48a62c13b25b801
Processing idna-2.0.tar.gz
Writing c:\users\admini~1\appdata\local\temp\easy_install-jvjamx\cryptography-0.9.1\temp\easy_install-hy81xk\idna-2.0\setup.cfg
Running idna-2.0\setup.py -q bdist_egg --dist-dir c:\users\admini~1\appdata\local\temp\easy_install-jvjamx\cryptography-0.9.1\temp\easy_install-hy81xk\idna-2.0\egg-dist-tmp-mx45zd
warning: no previously-included files matching '*.pyc' found under directory 'tools'
warning: no previously-included files matching '*.pyc' found under directory 'tests'
zip_safe flag not set; analyzing archive contents...
Moving idna-2.0-py2.7.egg to c:\users\admini~1\appdata\local\temp\easy_install-jvjamx\cryptography-0.9.1\.eggs

Installed c:\users\admini~1\appdata\local\temp\easy_install-jvjamx\cryptography-0.9.1\.eggs\idna-2.0-py2.7.egg
Searching for pycparser
Reading https://pypi.python.org/simple/pycparser/
Best match: pycparser 2.13
Downloading https://pypi.python.org/packages/source/p/pycparser/pycparser-2.13.tar.gz#md5=e4fe1a2d341b22e25da0d22f034ef32f
Processing pycparser-2.13.tar.gz
Writing c:\users\admini~1\appdata\local\temp\easy_install-jvjamx\cryptography-0.9.1\temp\easy_install-sqxb1e\pycparser-2.13\setup.cfg
Running pycparser-2.13\setup.py -q bdist_egg --dist-dir c:\users\admini~1\appdata\local\temp\easy_install-jvjamx\cryptography-0.9.1\temp\easy_install-sqxb1e\pycparser-2.13\egg-dist-tmp-wckizf
warning: no previously-included files matching 'yacctab.*' found under directory 'tests'
warning: no previously-included files matching 'lextab.*' found under directory 'tests'
warning: no previously-included files matching 'yacctab.*' found under directory 'examples'
warning: no previously-included files matching 'lextab.*' found under directory 'examples'
zip_safe flag not set; analyzing archive contents...
Moving pycparser-2.13-py2.7.egg to c:\users\admini~1\appdata\local\temp\easy_install-jvjamx\cryptography-0.9.1\.eggs

Installed c:\users\admini~1\appdata\local\temp\easy_install-jvjamx\cryptography-0.9.1\.eggs\pycparser-2.13-py2.7.egg
no previously-included directories found matching 'docs\_build'
warning: no previously-included files matching '*' found under directory 'vectors'
_Cryptography_cffi_a269d620xd5c405b7.c
src\cryptography\hazmat\bindings\__pycache__\_Cryptography_cffi_a269d620xd5c405b7.c(217) : fatal error C1083: Cannot open include file: 'openssl/aes.h': No such file or directory
Traceback (most recent call last):
  File "<string>", line 20, in <module>
  File "c:\users\admini~1\appdata\local\temp\pip-build-en7rdx\netlib\setup.py", line 87, in <module>
    "install": CFFIInstall,
  File "C:\Python27\lib\distutils\core.py", line 111, in setup
    _setup_distribution = dist = klass(attrs)
  File "C:\Python27\lib\site-packages\setuptools\dist.py", line 268, in __init__
    self.fetch_build_eggs(attrs['setup_requires'])
  File "C:\Python27\lib\site-packages\setuptools\dist.py", line 313, in fetch_build_eggs
    replace_conflicting=True,
  File "C:\Python27\lib\site-packages\pkg_resources\__init__.py", line 830, in resolve
    dist = best[req.key] = env.best_match(req, ws, installer)
  File "C:\Python27\lib\site-packages\pkg_resources\__init__.py", line 1075, in best_match
    return self.obtain(req, installer)
  File "C:\Python27\lib\site-packages\pkg_resources\__init__.py", line 1087, in obtain
    return installer(requirement)
  File "C:\Python27\lib\site-packages\setuptools\dist.py", line 380, in fetch_build_egg
    return cmd.easy_install(req)
  File "C:\Python27\lib\site-packages\setuptools\command\easy_install.py", line 633, in easy_install
    rmtree(tmpdir)
  File "C:\Python27\lib\site-packages\setuptools\command\easy_install.py", line 2208, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "C:\Python27\lib\site-packages\setuptools\command\easy_install.py", line 2208, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "C:\Python27\lib\site-packages\setuptools\command\easy_install.py", line 2208, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "C:\Python27\lib\site-packages\setuptools\command\easy_install.py", line 2213, in rmtree
    onerror(os.remove, fullname, sys.exc_info())
  File "C:\Python27\lib\site-packages\setuptools\command\easy_install.py", line 1609, in auto_chmod
    return func(arg)
WindowsError: [Error 5] Access is denied: 'c:\\users\\admini~1\\appdata\\local\\temp\\easy_install-jvjamx\\cryptography-0.9.1\\.eggs\\cffi-1.1.0-py2.7-win32.egg\\_cffi_backend.pyd'

Ok, here is the next updated output of 'pip install mitmproxy' after installing cryptography
Collecting mitmproxy
Using cached mitmproxy-0.12.1.tar.gz
Collecting pyperclip>=1.5.8 (from mitmproxy)
Using cached pyperclip-1.5.11.zip
Requirement already satisfied (use --upgrade to upgrade): pyasn1>0.1.2 in c:\python27\lib\site-packages (from mitmproxy)
Collecting tornado>=4.0.2 (from mitmproxy)
Using cached tornado-4.2.tar.gz
Collecting netlib<0.13,>=0.12 (from mitmproxy)
Using cached netlib-0.12.1.tar.gz
Complete output from command python setup.py egg_info:
warning: no files found matching 'OpenSSL\RATIONALE'
warning: no previously-included files found matching 'leakcheck'
warning: no previously-included files matching '.py' found under directory 'leakcheck'
warning: no previously-included files matching '.pem' found under directory 'leakcheck'
warning: no previously-included files matching '*.pyc' found anywhere in distribution
no previously-included directories found matching 'doc_build'
zip_safe flag not set; analyzing archive contents...
Installed c:\users\admini~1\appdata\local\temp\pip-build-nlejhy\netlib\.eggs\pyopenssl-0.15.1-py2.7.egg
running egg_info
creating pip-egg-info\netlib.egg-info
writing requirements to pip-egg-info\netlib.egg-info\requires.txt
writing pip-egg-info\netlib.egg-info\PKG-INFO
writing top-level names to pip-egg-info\netlib.egg-info\top_level.txt
writing dependency_links to pip-egg-info\netlib.egg-info\dependency_links.txt
writing manifest file 'pip-egg-info\netlib.egg-info\SOURCES.txt'
warning: manifest_maker: standard file '-c' not found

running build_ext
building '_cffi__x88a8fd90x94e38fe2' extension
creating c:\users\admini~1\appdata\local\temp\pip-build-nlejhy\netlib\netlib\__pycache__\Release
creating c:\users\admini~1\appdata\local\temp\pip-build-nlejhy\netlib\netlib\__pycache__\Release\netlib
creating c:\users\admini~1\appdata\local\temp\pip-build-nlejhy\netlib\netlib\__pycache__\Release\netlib\__pycache__
C:\Users\Administrator\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -IC:\Python27\include -IC:\Python27\PC /Tcnetlib\__pycache__\_cffi__x88a8fd90x94e38fe2.c /Foc:\users\admini~1\appdata\local\temp\pip-build-nlejhy\netlib\netlib\__pycache__\Release\netlib\__pycache__\_cffi__x88a8fd90x94e38fe2.obj -w
cl : Command line warning D9025 : overriding '/W3' with '/w'
_cffi__x88a8fd90x94e38fe2.c
netlib\__pycache__\_cffi__x88a8fd90x94e38fe2.c(208) : fatal error C1083: Cannot open include file: 'openssl/rsa.h': No such file or directory
Traceback (most recent call last):
  File "<string>", line 20, in <module>
  File "c:\users\admini~1\appdata\local\temp\pip-build-nlejhy\netlib\setup.py", line 87, in <module>
    "install": CFFIInstall,
  File "C:\Python27\lib\distutils\core.py", line 151, in setup
    dist.run_commands()
  File "C:\Python27\lib\distutils\dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "C:\Python27\lib\distutils\dist.py", line 972, in run_command
    cmd_obj.run()
  File "<string>", line 15, in replacement_run
  File "C:\Python27\lib\site-packages\setuptools\command\egg_info.py", line 207, in find_sources
    mm.run()
  File "C:\Python27\lib\site-packages\setuptools\command\egg_info.py", line 291, in run
    self.add_defaults()
  File "C:\Python27\lib\site-packages\setuptools\command\egg_info.py", line 320, in add_defaults
    sdist.add_defaults(self)
  File "C:\Python27\lib\site-packages\setuptools\command\sdist.py", line 118, in add_defaults
    build_py = self.get_finalized_command('build_py')
  File "C:\Python27\lib\distutils\cmd.py", line 312, in get_finalized_command
    cmd_obj.ensure_finalized()
  File "C:\Python27\lib\distutils\cmd.py", line 109, in ensure_finalized
    self.finalize_options()
  File "C:\Python27\lib\site-packages\setuptools\command\build_py.py", line 28, in finalize_options
    orig.build_py.finalize_options(self)
  File "C:\Python27\lib\distutils\command\build_py.py", line 46, in finalize_options
    ('force', 'force'))
  File "C:\Python27\lib\distutils\cmd.py", line 298, in set_undefined_options
    src_cmd_obj.ensure_finalized()
  File "C:\Python27\lib\distutils\cmd.py", line 109, in ensure_finalized
    self.finalize_options()
  File "c:\users\admini~1\appdata\local\temp\pip-build-nlejhy\netlib\setup.py", line 27, in finalize_options
    self.distribution.ext_modules = get_ext_modules()
  File "c:\users\admini~1\appdata\local\temp\pip-build-nlejhy\netlib\setup.py", line 20, in get_ext_modules
    from netlib import certffi
  File "netlib\certffi.py", line 20, in <module>
    extra_compile_args=['-w']
  File "C:\Python27\lib\site-packages\cffi\api.py", line 373, in verify
    lib = self.verifier.load_library()
  File "C:\Python27\lib\site-packages\cffi\verifier.py", line 96, in load_library
    self._compile_module()
  File "C:\Python27\lib\site-packages\cffi\verifier.py", line 192, in _compile_module
    outputfilename = ffiplatform.compile(tmpdir, self.get_extension())
  File "C:\Python27\lib\site-packages\cffi\ffiplatform.py", line 38, in compile
    outputfilename = _build(tmpdir, ext)
  File "C:\Python27\lib\site-packages\cffi\ffiplatform.py", line 65, in _build
    raise VerificationError('%s: %s' % (e.__class__.__name__, e))
cffi.ffiplatform.VerificationError: CompileError: command 'C:\\Users\\Administrator\\AppData\\Local\\Programs\\Common\\Microsoft\\Visual C++ for Python\\9.0\\VC\\Bin\\cl.exe' failed with exit status 2

@JPS46225 Can you run python -c "import ssl; print ssl.OPENSSL_VERSION" and paste the output here?
Also, the issue you are facing is not related to python3 support. I'd suggest to create a separate issue for your problem.

Hi syst3mw0rm
I will get to pasting output from command that you gave me asap. In regards to creating a new issue, since this isn't related to Python3 support, I actually couldn't find where you can create a new issue. I'll be more than happy to do this once I find out how to do this

Just found the new issue button, I was looking at this website last time on my iPhone

Hi syst3mw0rm
As requested, here is the output result of the command that you asked me to enter into Command Prompt:
OpenSSL 1.0.2a 19 Mar 2015

This is the new issue link for my problem with installing Mitmproxy on Python (Windows)
https://github.com/mitmproxy/mitmproxy/issues/612

Hi Guys
I was just wondering if there has been any recent progress in regards to the move to Python 3?

I think I will try to make mitmproxy work on both Python 2 and 3, well but I need time because mitmproxy is too big, and Twisted have no Python 3 which is the hardest math.

Hi
Just wondering what you mean by 'twisted with Python native' and also 'less as possible number of third party library', I've only installed what's necessary to get Mitmproxy functioning. Also, is there a way to assign a proxy to a device that doesn't have internal settings to configure proxies, perhaps, through a gateway/router?

Also, if there is anything else that I can do to improve overall experience, please let me know

Oh, yes, I see what you mean now by twisted, also I'd love to know how I can learn to do stuff like proxy configuration and other Python modules, if you know of any resources, please share them?

Hi @JPS46225, to answer your question:

Just wondering what you mean by 'twisted with Python native' and also 'less as possible number of third party library', I've only installed what's necessary to get Mitmproxy functioning.

Well, we can use socketserver instead Twisted, they are almost the same, but Twisted support something that socketserver don't have, but we can modify socketserver to have those.

Also, is there a way to assign a proxy to a device that doesn't have internal settings to configure proxies, perhaps, through a gateway/router?

Like above, socketserver (lower level), HTTPServer or TCPServer (higher).
With socketserver/HTTPServer/TCPServer we can create a proxy server that listen at whatever port we want.
I love socks5 mode for mitmproxy, it can filter/change HTTPS sites, that is a rare feature that almost no socks5 proxy do.

Hi cattleyavns
You're going to have to assume that I know absolutely nothing about this, so every little detail helps...
I have heard of socket server, but don't get it, would you mind elaborating more on the subject?

Any tutorials related to the subject, would also help immensely...

Ok, so, basically what I've known about Mitmproxy is so far, is that you have to tell client device to use the server that's running Mitmproxy, which would be the computer and then tell it the port to use. What I mean by internal settings, is that unlike my iPad, where I can configure a proxy for it to use, the other device which I'm trying this on, doesn't have the option to configure a proxy

Is there any estimate (date-wise) of when MitMProxy will be supported in Python 3.x?

No - the ROI for porting is still fairly low for us and we currently focus on other features. I'm happy to help anyone who wants to contribute this feature though.

ROI for me was higher than trying to get the python 2 version working. Spent 2 days on that and only like 2 hours porting it to python 3...

I also haven't gotten the 2.x version working yet. Do you have the ported version somewhere available?

Sorry, I just edited the installed package, and I'm sure there is several errors and bugs left.
I just ran it with python3 and fixed any errors using 2to3 and some manual fixing until it worked.
The best would be if someone could make a tutorial on how to get it working on a system using python3 as default (Arch in my case). I tried several methods, including AUR, git, pip2 etc.. none worked.

So I assume you just downloaded the archive, and imported from its subfolders?

Porting mitmproxy to Python 3 unfortunately takes way more time than two
hours. We're really happy for (partial) contributions here, this project is
completely volunteer-driven and this is a good chance to engage yourself.
:-)
On Tue, Aug 4, 2015 at 3:43 PM Pieter-Jan<EMAIL_ADDRESS>wrote:

I also haven't gotten the 2.x version working yet. Do you have the ported
version somewhere available?
—
Reply to this email directly or view it on GitHub
https://github.com/mitmproxy/mitmproxy/issues/326#issuecomment-127617246
.

@PidgeyL No, I edited the files installed with pip in /usr/lib/site-packages/. Not the recommeneded way of course, just clone the git and go from there, my way was a dirty hack.
@mhils I'm sure, but to just get it working to capture data it took two hours :)

@mhils: Would it take long to update just the libmproxy? (which I assume is what @riX2000 did?)

yes, libmproxy (and netlib..)

This is still the way to go: https://github.com/mitmproxy/mitmproxy/issues/326#issuecomment-90764563

What about the python3 branch on mitmproxy/netlib? Looks to be about the same fixes I did, is it working?

This was a quick attempt to see how far we get with little effort. Directly
porting everything to py3 isn't viable though - py2/3 cross compatibility
is the first goal for netlib.
riX2000<EMAIL_ADDRESS>schrieb am Di., 4. Aug. 2015 16:18:

What about the python3 branch on mitmproxy/netlib? Looks to be about the
same fixes I did, is it working?
—
Reply to this email directly or view it on GitHub
https://github.com/mitmproxy/mitmproxy/issues/326#issuecomment-127628192
.

Hmm I was gonig to start porting netlib, but then I noticed one of the reasons the python2 version did not work.. the mitmproxy/dev assumes the python 2 executable is 'python' and equivalently 'pip', which is usually not the case if python 3 is installed. Editing this file to use python2 and pip2 did the trick..
@mhils Maybe add a note about this in the readme or even better a python and pip version check in the dev file.

The only convincing argument for me would be a large external codebase that relies on mitmproxy/libmproxy.

Scrapy tests use libmproxy; it'd be nice to have Python 2/3 compatible codebase, not just Python 3 rewrite.

the mitmproxy/dev assumes the python 2 executable is 'python' and equivalently 'pip', which is usually not the case if python 3 is installed.

According to https://www.python.org/dev/peps/pep-0394/ it shouldn't be the case. python should be Python 3 only in a Python 3 virtualenv or in distributions which go against this PEP (like Arch Linux).

Thanks for the feedback, @kmike. Is it just https://github.com/scrapy/scrapy/blob/master/tests/test_proxy_connect.py or did I miss some parts?
FWIW, I ported netlib a few weeks ago to be cross-comptabile with 2.7 and 3.5. We're not sure what to do with mitmproxy, but we'll take that into consideration.

@mhils great to hear netib is ported, thanks!

Thanks for the feedback, @kmike. Is it just https://github.com/scrapy/scrapy/blob/master/tests/test_proxy_connect.py or did I miss some parts?

Yes, it is just that. Currently for HTTP proxy tests we're using Twisted.

Do you target 3.5 only, not 3.4? Python 3.4 is the default Python 3 in latest Ubuntu LTS, so it'd be nice to support it as well. Are there new features you're using?

We're using a few 3.5 features, but nothing that couldn't be fixed easily. I'm personally fine with supporting 3.5 only (which should be the default for Ubuntu 15.10 and then 16.04), but I'd be happy to accept patches that add 3.4 compatibility. :wink:

Ok, sounds good :)

Has anyone begun the porting process? I'm game for working on the port, but don't want to duplicate efforts.

I don't think so. If you go for it, please make it  as incremental as you
can (e.g. make the tests for one class or one *.py file pass) and send
partial pull requests as early as possible. :-)
scone<EMAIL_ADDRESS>schrieb am Mi., 11. Nov. 2015 08:55:

Has anyone begun the porting process? I'm game for working on the port,
but don't want to duplicate efforts.
—
Reply to this email directly or view it on GitHub
https://github.com/mitmproxy/mitmproxy/issues/326#issuecomment-155845256
.

+1 for Python 3.5 support, I just tried to install it and was surprised to find mitmdump fail with import thread. Given we have <4 years to the 2020 sunset date for Python 2.7, I'm surprised to find tools that aren't Python 3.4+ ready (I understand the issues, I talk about this on stage at my monthly PyDataLondon [2,800+ members] meetup to get folk to make the jump). For now I'll downgrade to an old 2.7 install.

@ianozsvald Mitmproxy is a good example of a difficult Python 2-3 transition. If you're interested in the issue - and especially if you feel qualified to speak publicly about it - it would be a good idea to look closely and sympathetically at why we're not there yet.

Mitmproxy has lots of dependencies, some of which only recently became Python 3 ready themselves.
We're not interested in inelegantly straddling Python 2 and 3 - we'll be moving across to Python3 wholesale. A lot of projects integrate mitmproxy as a library, which means that a move to Python3 will force integrators to either stay with old versions of mitmproxy or to move to Python3 themselves.
Mitmproxy works at a slightly unusual boundary where Python3's unicode and byte handling becomes a pain point. Armin Rigo has written eloquently about this (though his post is now somewhat outdated): http://lucumr.pocoo.org/2014/5/12/everything-about-unicode/.
Mitmproxy is distributed as a compiled binary for all major platforms. We rely on pyinstaller for this, but pyinstaller has only supported Python3 in a stable release since last October. Pyinstaller has been a huge pain-point for us, and I'm waiting for the Python3 support to mature a bit before the transition.

I hope you will be less surprised next time you find a large, complicated tool still stuck on 2.7. The bug tracker is not the place to discuss this (feel free to drop me an email), but I've come to consider the way the Python2-3 transition was handled to be one of the greatest technical blunders of the last decade. Long after mitmproxy itself has shifted to Python3, the vast bulk of production Python code will still be on 2.7. We'd better make peace with that.
If you'd like to help things along for mitmproxy itself, please join us on the Slack channel to discuss, and then to roll up your sleeves and pitch in. ;)

Hi Aldo. Sorry, I wasn't trying to be unsympathetic. I had seen the discussion around a wholesale shift from Py2 to Py3 (this seems like a sensible approach if feasible), I understand the legacy difficulties with longer running users. I've followed and have been involved in the discussions around the move of projects like numpy, matplotlib and django (years back), scrapy (just now feasible via Twisted's updates), PySpark and lots of smaller projects. I've ported a few very small projects (several hours each to successful PR, a few involving web scraping/byte issues), nothing on the scale of mitmproxy.
I'm very aware of the str/unicode/byte pain point and I've followed Armin's posts over the years. I spend almost all of my time in the data science world where almost all the major ports were completed years ago (with dual version support). A part of my outspoken-ness at public meetups is to try to get the slower moving users to upgrade (which is totally feasible in the data science world now) such that the developers have an easier time supporting just the one Python 3 platform.
Re. your request that I pitch in - you're totally right that I should if I wanted the port to move faster, however I won't. mitmproxy was a test for a side project and the Py2.7 version did what I needed. Elsewhere I do put plenty of time into the Py community (right now I'm co-chair for our 3rd annual PyDataLondon O/S conference, I'm also co-chair for our monthly 200+ user meetup), so I hope you'll understand. Sorry to cause noise.

Just a quick update:
Thanks to the great work of @dufferzafar as part of this years GSoC, we are getting close to running mitmproxy in Python 3.5. As of a few days ago, pathod is now fully compatible with Python 2.7 and 3.5. netlib was already py3-compatible, thanks to the great efforts by @mhils a couple of months ago.
This means all dependencies for mitmproxy are able to run in Python 2 and 3.
We expect to migrate the remaining mitmproxy code to Python 3 within the next weeks - so stay tuned!

Quick update: We managed to build Python 3.5 binaries for pathod and we only have a couple of modules left for mitmproxy:

[ ] test/mitmproxy/test_dump.py (#1299)
[ ] test/mitmproxy/test_examples.py
[ ] test/mitmproxy/test_flow.py
[ ] test/mitmproxy/test_flow_format_compat.py (#1318)
[ ] test/mitmproxy/test_protocol_http2.py (#1307)
[ ] test/mitmproxy/test_web_master.py

While there is still a bit to do, thanks to @dufferzafar's great GSoC work all tests pass now on Python 3! :tada: :snake:

The Python2/3 straddle is now complete. Whatever remains should be split up, and attached to the milestone for 1.0, which will be our first Python3-only release.


grim(wayland): Failed to create image.

grim works fine at v0.1.1, but will report an error directly on the latest commit: 81a41049732a161ae4d3aed366115ce5dd7a0abf:
› grim
*** BUG ***
In create_bits_image_internal: The expression bits == NULL || (rowstride_bytes % sizeof (uint32_t)) == 0 was false
Set a breakpoint on '_pixman_log_error' to debug

Failed to create image

Related:

https://www.reddit.com/r/hyprland/comments/14aqixm/i_cant_take_screen_shot_on_my_hyprland/

Fixed by https://github.com/ryan4yin/nix-config/commit/26da19ba38f1f2ee957f6861edc9be1d3801f9b4.
The problem may be caused by inconsistent nixpkgs commit, hyprland's flake follows github:NixOS/nixpkgs/nixos-unstable, but I'm on the nixpkgs's  stable branch.


[Vulkan-M1] During minlm-torch: cannot reserve 'buffer' resource location at index 0

@antiagainst
Running into this error during iree-run-module of minilm-torch:
E /Users/iree/runtime/src/iree/hal/vulkan/debug_reporter.cc:38] Validation Error: [ VUID-VkShaderModuleCreateInfo-pCode-01091 ] Object 0: handle = 0x134860218, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0xa7bb8db6 | vkCreateShaderModule(): The SPIR-V Capability (Int64) was declared, but none of the requirements were met to use it. The Vulkan spec states: If pCode declares any of the capabilities listed in the SPIR-V Environment appendix, one of the corresponding requirements must be satisfied (https://vulkan.lunarg.com/doc/view/<IP_ADDRESS>/mac/1.3-extensions/vkspec.html#VUID-VkShaderModuleCreateInfo-pCode-01091)
[mvk-error] VK_ERROR_INITIALIZATION_FAILED: Shader library compile failed (Error code 3):
program_source:16:90: error: cannot reserve 'buffer' resource location at index 0
kernel void forward_dispatch_4(device _7& _resource_var_0_0_0 [[buffer(0)]], device _12& _resource_var_0_0_ [[buffer(0)]], device _12& _resource_var_0_1_ [[buffer(1)]], device _7& _resource_var_0_2_ [[buffer(2)]], uint3 gl_LocalInvocationID [[thread_position_in_threadgroup]], uint3 gl_WorkGroupID [[threadgroup_position_in_grid]])
                                                                                ^
.
E /Users/iree/runtime/src/iree/hal/vulkan/debug_reporter.cc:38] VK_ERROR_INITIALIZATION_FAILED: Shader library compile failed (Error code 3):
program_source:16:90: error: cannot reserve 'buffer' resource location at index 0
kernel void forward_dispatch_4(device _7& _resource_var_0_0_0 [[buffer(0)]], device _12& _resource_var_0_0_ [[buffer(0)]], device _12& _resource_var_0_1_ [[buffer(1)]], device _7& _resource_var_0_2_ [[buffer(2)]], uint3 gl_LocalInvocationID [[thread_position_in_threadgroup]], uint3 gl_WorkGroupID [[threadgroup_position_in_grid]])
                                                                                ^
.
[mvk-error] VK_ERROR_INVALID_SHADER_NV: Compute shader function could not be compiled into pipeline. See previous logged error.
E /Users/iree/runtime/src/iree/hal/vulkan/debug_reporter.cc:38] VK_ERROR_INVALID_SHADER_NV: Compute shader function could not be compiled into pipeline. See previous logged error.
/Users/iree/runtime/src/iree/hal/vulkan/native_executable.cc:131: INVALID_ARGUMENT; VK_ERROR_INVALID_SHADER_NV; while invoking native function hal.executable.create; while calling import;
[ 1]   native hal.executable.create:0 -
[ 0] bytecode module.__init:992 minilm_torch.mlir:397:12
      at minilm_torch.mlir:23:3; creating context
[1]    67822 abort      /Users/nodlabs/nod/iree-build-latest/tools/iree-run-module --driver=vulkan

To Reproduce

wget https://storage.googleapis.com/iree-shared-files/nod-perf/stanley/minilm_repro_vulkan/minilm_torch_06042022.mlir

iree-build-latest/tools/iree-compile --iree-input-type=none --iree-vm-bytecode-module-output-format=flatbuffer-binary --iree-hal-target-backends=vulkan --iree-mlir-to-vm-bytecode-module --mlir-print-debuginfo --mlir-print-op-on-diagnostic=false --iree-llvm-target-cpu-features=host --iree-vulkan-target-triple=m1-moltenvk-macos minilm_torch.mlir -o /tmp/minilm_torch.vmfb

iree-build-latest/tools/iree-run-module --driver=vulkan --entry_function=forward --function_input=1x128xi64 --module_file=/tmp/minilm_torch.vmfb

There are two errors here: one is Int64 capability inconsistency (somehow); the other is buffer aliasing. The first one is not fatal. For the second one, we pack a lot of tensors into the same buffer:
    func.func @forward_dispatch_4() {
      ...
      %0 = hal.interface.binding.subspan set(0) binding(0) type(storage_buffer) offset(%c86523904) alignment(64) : !flow.dispatch.tensor<readonly:2x384xf32>
      %1 = hal.interface.binding.subspan set(0) binding(0) type(storage_buffer) offset(%c86526976) alignment(64) : !flow.dispatch.tensor<readonly:30522x384xf32>
      %2 = hal.interface.binding.subspan set(0) binding(0) type(storage_buffer) offset(%c85737472) alignment(64) : !flow.dispatch.tensor<readonly:512x384xf32>
      %3 = hal.interface.binding.subspan set(0) binding(1) type(storage_buffer) offset(%c0) alignment(64) : !flow.dispatch.tensor<readonly:1x128xi64>
      %4 = hal.interface.binding.subspan set(0) binding(0) type(storage_buffer) offset(%c0) alignment(64) : !flow.dispatch.tensor<readonly:1x128xi64>
      %5 = hal.interface.binding.subspan set(0) binding(2) type(storage_buffer) offset(%c0) alignment(64) : !flow.dispatch.tensor<writeonly:1x128x384xf32>

In the above, %0, %1, %2, %4 are all in the same buffer bound to binding slot#0. In the SPIR-V CodeGen flow, we lower all n-D buffers into 1-D runtime arrays eventually; though we still carry the element type in the runtime array. So this would mean we create two aliased SPIR-V storage buffers (%0/%1/%2 vs %4) bound to slot#0. SPIRV-Cross would just translate that into two buffers at the same buffer(n) location as shown in the above. Metal does not support this kind of behavior. So it errors out. We have an upstream pass to clean up such aliasing buffers. Right now it handles %0/%1/%2; it should be extended to handle %4 too.

I've created upstream patch https://reviews.llvm.org/D127265 and https://reviews.llvm.org/D127266 to handle the resource aliasing unification cases. https://github.com/google/iree/pull/9381 pulls them in as a pull request. With it, iree-run-module can successfully run.

Both patches are landed upstream so the specific issue mentioned here is fixed. I know we have remaining correctness issues for this still but that'd better be tracked separately. So closing this one. Feel free to file another one so we can put analysis in its own dedicated thread. :)

@antiagainst reopening this issue wrt latest breakage!
Here is the latest repro instruction:
wget https://storage.googleapis.com/iree-shared-files/nod-perf/stanley/minilm_repro_vulkan/minilm_torch_06042022.mlir

iree-compile --iree-input-type=none --iree-vm-bytecode-module-output-format=flatbuffer-binary --iree-hal-target-backends=vulkan --iree-mlir-to-vm-bytecode-module --iree-llvm-embedded-linker-path=iree-lld --mlir-print-debuginfo --mlir-print-op-on-diagnostic=false --iree-llvm-target-cpu-features=host --iree-vulkan-target-triple=m1-moltenvk-macos minilm_torch_06042022.mlir -o /tmp/minilm_torch.vmfb

/Users/nodlabs/nod/SHARK/shark.venv/lib/python3.10/site-packages/iree/runtime/iree-run-module --device=vulkan --entry_function=forward --function_input=1x128xi64 --module_file=/tmp/minilm_torch.vmfb

Here is the debug log from building vulkan with debug symbols: https://storage.googleapis.com/iree-shared-files/nod-perf/stanley/minilm_repro_vulkan/vulkan_debug_log.txt

@raikonenfnu now an LLVM integration carrying the upstream fix https://github.com/llvm/llvm-project/commit/11cf2d5f62f94b14644fab3478f17af9cb015706 is landed. Please try the tip of the tree again. It should fix the issue I believe.

Yeap tried after https://github.com/google/iree/pull/9507, it works! closing this issue!


How to set STDIO on UART with no USB

I am trying to create a PicoBB system with VGA, STDIO over UART and no USB but it gives me  the following error during make with the following options:
~/projects/Raspberry_Pi_Pico/pico/PicoBB/console/pico$ make clean && make BOARD=vgaboard_cut CYW43=NONE STDIO=UART SERIAL_DEV=1 GRAPH=Y SOUND=PWM PRINTER=N
/home/gert/projects/Raspberry_Pi_Pico/pico/pico-sdk/lib/tinyusb/src/common/tusb_common.h:74:10: fatal error: tusb_option.h: No such file or directory
This message seems to note that it still tries to use some USB stuff.
The board I use is a Pimoroni pico demo VGA with the GPIO20 and GPIO21 connections cut. These GPIO pins I would like to use for a serial terminal/keyboard and use VGA for main output.
When I do a make with STDIO=USB+UART it compiles without errors but after uploading nothing happens or responds on the UART pins (20/21). Trying *output 0 just outputs to the USB connection again.
Any ideas?

That is a configuration I have not tested. It will take a while to investigate.
If you want to use VGA as main output, and a keyboard as input, what is wrong with the GUI build? Keyboard attached directly to the Pico USB port.
Is there any reason for avoiding the USB port?

Well the reason is that I have reverse engineered an Motorola Alphamate with a very nice keyboard (and a single line LCD) so it works as a RS232 terminal and would like to attach this to a Pico running BBCbasic and have VGA output. Since in the makefile you can select USB and/or UART I assumed it would be easy to just select UART.
Thanks for your time.

OK, I have pushed a couple of minor fixes. Pull my changes and then build with:
make clean && make BOARD=vgaboard_cut CYW43=NONE STDIO=UART SERIAL_DEV=0 GRAPH=Y SOUND=PWM PRINTER=N

Note that if the serial connection is used for STDIO, then it is not available as a serial device, hence SERIAL_DEV=0.

OK, I have pushed a couple of minor fixes. Pull my changes and then build with:
make clean && make BOARD=vgaboard_cut CYW43=NONE STDIO=UART SERIAL_DEV=0 GRAPH=Y SOUND=PWM PRINTER=N

Note that if the serial connection is used for STDIO, then it is not available as a serial device, hence SERIAL_DEV=0.

This seems to work, thanks!
Now figuring out to set the default output to VGA while keeping the input from UART aka doing an *output 14


Appended /v2 to Go module

[ ] I've added a new note in the CHANGELOG.md file, according to docs:
https://iver-wharf.github.io/#/development/changelogs/writing-changelogs

Summary

Changed module path from github.com/iver-wharf/wharf-api-client-go to github.com/iver-wharf/wharf-api-client-go/v5

Motivation
We need to stay compatible with Go modules to be able to use this library from other projects. Go modules almost requires it's own doctrine, but this PR is based on some statements from here: https://github.com/golang/go/wiki/Modules#releasing-modules-v2-or-higher
Based on

Discussion: https://github.com/iver-wharf/wharf-api-client-go/pull/29#discussion_r766513730
PR: https://github.com/iver-wharf/wharf-api/pull/128

Pausing this as I want an RFC about this first.

RFC: https://github.com/iver-wharf/rfcs/pull/26


fuse data cache is not working as expected

Alluxio Version:
2.9.1
Describe the bug
the same background as issues-17025
It can be found that even if large files are read non-concurrently, the phenomenon of io enlarge can also be observed;
so we try to turn on the fuse data cache function to see if it can alleviate this problem;
through previous communication, we know:

the app can read up to 128KB of data from the fuse at one time
fuse pulls data from work in blocks or in chunk (anyway, at least 2MB or more)

when there is no cached data locally, the app reads it through fuse, we expect:

fuse pulls one data blocks (2MB) from work, and save to disk;
fuse reads (128KB) from the local data block and returns it to the upper application;
the next few reads(128KB) can directly obtain data directly from the local cache;
repeat steps 1 to 3 above until the entire file is completely read;

considering that cache flushing may be an asynchronous operation, or that the data read by fuse may spans two data blocks at a time, it is acceptable to have low data io enlarge;
actually observed:
1 concurrent:

time cost: 122s
bandwidth:

net data transmission: 122s * 0.55GB/s = 67.1 GB

4 concurrent:

time cost: 141s
bandwidth:

net data transmission: 141s * 1.7GB/s = 239.7 GB

Although compared with issues-17025, the problem of io enlarge has been alleviated a lot, but it is still much worse than what we expected (multi-concurrency is a little more than single-concurrent transmission), so is there room for optimization in the fuse data cache part?
Additional context

the client cache uses NVME, not the bottleneck
we count the space occupied by alluxio.user.client.cache.dirs is 30GB, may be dataloader did not fully read the entire dataset

therefore, compared with the actually read data and network transmission, the io enlarge may be more serious than the above description

@LuQQiu can you take at this one please? Thanks a lot!

This issue is related to #17025
The read pattern is "seek" heavy to read partial data (2% to 8%), the current client read approach performs worse in this case.
There are two approaches after discussion:

mainly for #17025 each Fuse.read(offset, size) directly requests read(offset,size) from a worker, the worker only returns the requested bytes at offset. The small fuse requests can be aggregated by netty and batch sent to / received from a worker. @wwq2333  and other users are worried about the performance overhead of fulfilling small requests through the RPC framework.
Use client-side data cache to read pages (by default 1MB) from worker to speed up future data access, worker only returns the requested bytes at offset. This is the approach used by this issue. @wwq2333 and other users prefer to improve this approach to have better concurrent 1 performance. The client cache size is relatively small and can help speed up future data access.

@wwq2333 would you mind highlighting the issue that single thread throughput drops when enabling data cache, thanks.
@LuQQiu FYI


[dev-3.x] Subscriptions failing on Android when a cache is used

Summary
On Android, on 3.x, subscriptions fail when the client is configured with a cache. I'm posting a minimal repro in case it can help
Version
3.0-dev11
Description
As promised, I'm posting a repro for the other issue I've had so far with the dev-3.x branch: https://github.com/martinraison/kmm-repro-android-subscription-cache-crash
The repro is almost the same as for #3127, the only difference is I'm configuring the client to use a store (RealApolloStore backed by a MemoryCache). This causes the subscription to fail on Android with the following error:
E/AndroidRuntime: FATAL EXCEPTION: DefaultDispatcher-worker-2
    Process: com.example.repro.android, PID: 7629
    java.lang.IllegalArgumentException: Flow has more than one element
        at kotlinx.coroutines.flow.FlowKt__ReduceKt$single$$inlined$collect$1.emit(Collect.kt:135)
        at com.apollographql.apollo3.interceptor.cache.ApolloCacheInterceptor$proceed$$inlined$map$1$2.emit(Collect.kt:137)
        at com.apollographql.apollo3.network.ws.ApolloWebSocketNetworkTransport$execute$$inlined$map$1$2.emit(Collect.kt:137)
        at kotlinx.coroutines.flow.FlowKt__LimitKt$takeWhile$lambda-9$$inlined$collectWhile$1.emit(Limit.kt:145)
        at com.apollographql.apollo3.network.ws.ApolloWebSocketNetworkTransport$execute$$inlined$filter$1$2.emit(Collect.kt:137)
        at kotlinx.coroutines.flow.SubscribedFlowCollector.emit(Unknown Source:2)
        at kotlinx.coroutines.flow.SharedFlowImpl.collect(SharedFlow.kt:350)
        at kotlinx.coroutines.flow.SharedFlowImpl$collect$1.invokeSuspend(Unknown Source:15)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)

It looks like at some point, ApolloCacheInterceptor assumes there's only a single value in the flow (which is wrong for a subscription) and calls .single(). I'm completely unfamiliar with the code so I might be off-track though.
(The iOS version is also failing, but that's the issue reported in #3127)

I can confirm the fix. Thanks a lot for the very fast resolution!


Code gen the route partial isos

We've already got sourcery in the build process, might as well use it!

I like it! It'll make the template messier, but what do ya think of formatting it in such a way that the output is indented normally?

yahhh, was on the fence. it seems most people style their stencils so that outputs are optimal, but the stencil is the thing we actually have to edit, so I preferred styling it instead :/
we could prob run swiftlint autocorrect on the file to clean up some things, and also lib/Syntax will be coming soon and fix all of our problems!

yahhh, was on the fence. it seems most people style their stencils so that outputs are optimal, but the stencil is the thing we actually have to edit, so I preferred styling it instead :/

I was probably overthinking, but I considered if folks decided to use this lib and codegen we'd optimize for readability. I don't think we need to worry about this right now.

I was probably overthinking, but I considered if folks decided to use this lib and codegen we'd optimize for readability. I don't think we need to worry about this right now.

well right now we're in pointfreeco not swift-web! if we extract out to swift-web we can generalize and clean up a bit fer sure.


Change default karaf shutdown command

The Karaf shutdown command string was set in a string that is not random, but enough random to any who doesn't read carefully can forget easily.
I just changed the string for something more obvious.

[x] have a concise title
[x] be against the correct branch
[x] pass automated tests
[x] have a clean commit history
[x] have proper commit messages (title and body) for all commits](https://medium.com/@steveamaza/e028865e5791)

I don't have a strong opinion about change it to develop, but could be considered a security risk? While it is clear in the docs, the experimented admins with opencast normally only read the release notes and the changelog. Thus, it is easy to forget to update something that seems "good as is" when is a critical security configuration.

Fair enough, I just changed to develop 😄


Multiple CSS files have the same name when using the Vite plugin with a scoped package

Describe the bug
If the package name includes a scope e.g @vanilla-extract/example the outputted CSS files will all have the names example.vanilla.css, example.vanilla2.css, example.vanilla3.css etc. This seems to be because the sanitise file scope function adds the package name and when a scope is used it includes a / so it considers the last part as the file name. I would expect the file name to be included in the asset name, which it is when not using a scope.
Link to reproduction
https://stackblitz.com/edit/vitejs-vite-gfnuxw?file=vite.config.ts

Run npm run build in the above and it outputs:vite v2.8.4 building for production...
✓ 9 modules transformed.
dist/main.js                0.25 KiB / gzip: 0.20 KiB
dist/main.css.js            0.18 KiB / gzip: 0.16 KiB
dist/Button.js              0.13 KiB / gzip: 0.13 KiB
dist/Heading.js             0.14 KiB / gzip: 0.14 KiB
dist/Button.css.js          0.18 KiB / gzip: 0.16 KiB
dist/Heading.css.js         0.18 KiB / gzip: 0.16 KiB
dist/example.vanilla.css    0.23 KiB / gzip: 0.16 KiB
dist/example.vanilla2.css   0.12 KiB / gzip: 0.13 KiB
dist/example.vanilla3.css   0.04 KiB / gzip: 0.06 KiB

However, if you remove the scope from package.json->name. It outputs the following:dist/main.js                                0.25 KiB / gzip: 0.20 KiB
dist/main.css.js                            0.19 KiB / gzip: 0.15 KiB
dist/Button.js                              0.13 KiB / gzip: 0.13 KiB
dist/Heading.js                             0.14 KiB / gzip: 0.14 KiB
dist/Button.css.js                          0.20 KiB / gzip: 0.16 KiB
dist/Heading.css.js                         0.20 KiB / gzip: 0.16 KiB
dist/Button.css.ts___example.vanilla.css    0.23 KiB / gzip: 0.17 KiB
dist/main.css.ts___example.vanilla.css      0.12 KiB / gzip: 0.13 KiB
dist/Heading.css.ts___example.vanilla.css   0.04 KiB / gzip: 0.06 KiB

System Info
Output of npx envinfo --system --npmPackages @vanilla-extract/css,@vanilla-extract/webpack-plugin,@vanilla-extract/esbuild-plugin,@vanilla-extract/vite-plugin,@vanilla-extract/sprinkles,webpack,esbuild,vite --binaries --browsers:
  System:
    OS: macOS 11.2.3
    CPU: (12) x64 Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
    Memory: 35.36 MB / 16.00 GB
    Shell: 5.8 - /usr/local/bin/zsh
  Binaries:
    Node: 16.13.2 - ~/.nvm/versions/node/v16.13.2/bin/node
    Yarn: 1.22.17 - /usr/local/bin/yarn
    npm: 8.1.2 - ~/.nvm/versions/node/v16.13.2/bin/npm
  Browsers:
    Chrome: 99.0.4844.83
    Edge: 99.0.1150.46
    Firefox: 94.0.1
    Safari: 14.0.3
  npmPackages:
    @vanilla-extract/esbuild-plugin: ^2.0.3 => 2.0.3
    @vanilla-extract/vite-plugin: ^3.1.2 => 3.1.2
    esbuild: 0.14.22 => 0.14.22
    vite: 2.8.4 => 2.8.4

Related

Some preliminary investigation was documented here:  https://github.com/seek-oss/vanilla-extract/discussions/620#discussioncomment-2409014

@graup pointed out in a conversation we had on discord that "virtual" modules should not be necessary for this case:

but the Vite documentation says that "virtual" should not be necessary for cases where the output is derived from a real file
https://vitejs.dev/guide/api-plugin.html#virtual-modules-convention

So I had a go at removing them, this is very rough around the edges and may re-introduce past bugs (e.g the .. thing) and likely break other stuff. But for the issue at hand it fixes paths: https://github.com/seek-oss/vanilla-extract/compare/master...AndrewLeedham:AL-proper-paths.
This seems like the right direction to me, as Vite will see the CSS file properly, and it will have a proper file system path for generating the name. Curious on your thoughts here @mattcompiles?

I have thought about this in the past as well. VE CSS is derived from a real file, however it is not derived from an AST transform or compilation process. Meaning it can't be sensibly mapped with a source map. Therefore I believe they do need to represented as virtual files. However feel free to correct me if I'm wrong and there is a way to do this I'm not aware of.
Either way though I think issue can be resolved even with virtual files. We might just need to update the file scope serialization.

I see why sourcemaps wouldn't be possible, but is there any benefit of using virtual files over an "actual" path without sourcemaps?

Reading through both the Vite and Rollup docs some more I guess it prevents other plugins from processing the import. Although that seems to be a feature of \0 which isn't in use here. Also it would seem we do want other plugins to process the import e.g vite:css.
A few potential options:

Replace virtual:vanilla-extract:style.vanilla.css$$$example-package with ./style.css, and just don't provide sourcemaps.
Refactor virtual:vanilla-extract:style.vanilla.css$$$example-package to something like virtual:vanilla-extract:style.css?package=example-package.
Provide some-sort of plugin option that allows you to format the last part of the path given the filescope. E.g:
fileNames: (fileScope) => fileScope.filePath which would be used to create something like virtual:vanilla-extract:<user-value>?package=example-package.
Something else?

Curious on your thoughts here @mattcompiles. Happy to take a swing at implementing an approach, just not sure which direction to go.

Would this issue be resolved if we just make the file path be the last thing in the filescope string instead? I don't remember any specific reason it was last.
e.g.
export function stringifyFileScope({
  packageName,
  filePath,
}: FileScope): string {
  return packageName ? `${packageName}$$$${filePath}` : filePath;
}

Would this issue be resolved if we just make the file path be the last thing in the filescope string instead? I don't remember any specific reason it was last.
e.g.
export function stringifyFileScope({
  packageName,
  filePath,
}: FileScope): string {
  return packageName ? `${packageName}$$$${filePath}` : filePath;
}

Yes I believe that would be a slight improvement and fix this particular issue.

That would definitely fix this issue yes. It still doesn't feel like the right solution to me though, because we are relying on the fact that Vite uses chunk.name which in this case will be the basename of the virtual ID. But probably better to get a solution in for now as it shouldn't affect anything else.

Referring back to @AndrewLeedham 's comment, how about something like this? (Untested poc: remove prefix and change serialization to filePath?package=packageName)
As far as I can tell the packageName is only there for the hot reload and injectStyles, right?

The package name is in the filescope as the filename is relative to that package. If you're using a package that contains VE files then all the file paths coming from that package will be relative to their package.json. What we need to be careful of here is allowing two separate files from having the same ID. This solution is probably ok from that regard.
One thing I'm not clear on though is how this solution is more ideal than what I suggested? Could someone just explain why this creates a better build output?

The main advantage is that having the generated css files as proper (non-virtual) modules, with usable names and paths, works better with preserveModules and also possibly other plugins/options. Seems more flexible to me. (Didn't check but it might also enable us to remove the postcss code from the plugin and just have css files be processed by the rest of the rollup plugin chain.)

This issue, along with a few others have lead me to doing a decent refactor of file scopes across the board. Hoping this PR will make everyone happy. https://github.com/seek-oss/vanilla-extract/pull/647


Panoramas fail to render

Describe the bug
Panorama photos (with tags managed using digiKam) are heavily distorted when viewed full-size.
To reproduce

Locate a panorama photo.
Tap on it to view the full-size preview (which looks fine).
Tap on Open Panorama.

Screenshots

Screenshots

System information and logs

System information
Aves version: 1.7.1-izzy (Build 1081)
Flutter version: 3.3.4 (Channel stable)
Android version: 12 (SDK 31)
Android build: EB2103_11_C.07
Device: OnePlus EB2103
Mobile services: ready
System locales: en_GB
Aves locale: system -> en
Installer: com.google.android.packageinstaller

aves-logs-20221014_141349.txt
Additional context
IMG_20220911_120020_tagged.jpg is one such affected image.
See also: https://github.com/SimpleMobileTools/Simple-Gallery/issues/2590#issuecomment-1243024371

Thanks for the nice bug report! My first guess is that the 90° orientation declared in Exif doesn't gel well with the way I use the Google Panorama metadata. I'll take a closer look later.

I wouldn't be surprised if other apps have trouble with this file... Its panorama metadata seems weird to me, probably because of the aforementioned rotation. Anyway, I fixed it by salvaging what I could, but I ignore the exact offset from top/left in those cases (the panorama will be centered).

Curious, as all my panoramas have the same rotation issue -- quite possibly down to the OnePlus camera app writing dodgy metadata then.  I'll have to try some other camera apps with panorama modes and experiement...
Will give the fix a go in the next release and see if it helps with rendering the existing ones.

Just got the update through, and looks like that's enough to render my existing panoramas (I think they still look slightly squished, but I'll chalk that up to the poor metadata).  :+1:
I've switched to a GCam port for my device in the meantime, which seems to produce panoramas with accurate metadata.

Good to hear, thanks for the follow-up!


Bug: Kubelink Requests getting Failed for gRPC method GetAppDetails

📜 Description
Kubelink requests fail with below error :
{"level":"error","ts":1713965180.331496,"caller":"appList/AppListingRestHandler.go:1074","msg":"error in fetching app detail","err":"rpc error: code = Unknown desc = release not exists for this app"
This happens even when application is not deployed
Ideally request should not go to kubelink in the first place.
Affected areas
CD
Additional affected areas
None
Prod/Non-prod environments?
Prod
Is User unblocked?
Yes
How was the user un-blocked?
None
Impact on Enterprise
Unnecessarily requests are going to Kubelink and failing
👟 Steps to replicate the Issue

create an application with deployment app type as helm.
create CI, CD but do not deploy
go to app details page and parallelly check orchestrator logs.

👍 Expected behavior
Requests should not go to kubelink searching for release as application is not even deployed yet.
👎 Actual Behavior
Kubelink throws an error that release not exists
☸ Kubernetes version
NA
Cloud provider

NA

🌍 Browser
Chrome
✅ Proposed Solution
NA
👀 Have you spent some time to check if this issue has been raised before?

[X] I checked and didn't find any similar issue

🏢 Have you read the Code of Conduct?

[X] I have read the Code of Conduct

AB#9576

This issue was happening because we were fetching the app details of a helm app for an environment in which there was not even a single deployment triggered yet. Need to handle it from the code side by putting a check and not calling the GetAppDetail function in this case.

This issue was happening because we were fetching the app details of a helm app for an environment in which there was not even a single deployment triggered yet. Need to handle it from the code side by putting a check and not calling the GetAppDetail function in this case.

This issue was happening because we were fetching the app details of a helm app for an environment in which there was not even a single deployment triggered yet. Need to handle it from the code side by putting a check and not calling the GetAppDetail function in this case.


[apm-server] Add  option to service

[X] Chart version not bumped (the versions are all bumped and released at the same time)
[ ] README.md updated with any new values or changes
[X] Updated template tests in ${CHART}/tests/*.py
[ ] Updated integration tests in ${CHART}/examples/*/test/goss.yaml

See https://github.com/elastic/helm-charts/issues/1031 - this PR add "service.loadBalancerIP" field. (copied from Kibana chart).

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

jenkins test this please

jenkins test this please

I think this will also need to be rebased on master to make tests pass.

jenkins test this please


The problem is that text is to short at the columns database_migration and  datamodel_steps
Error:
LiftEngine:stderr thread 'tokio-runtime-worker-1' panicked at 'called Result::unwrap() on an Err value: QueryError(MySqlError { ERROR 1406 (22001): Data too long for column 'datamodel_steps' at row 1 } +0ms
 LiftEngine:stderr stack backtrace: +1ms

My solution:
changed text to longtext

Related MySQL docs:

https://dev.mysql.com/doc/refman/8.0/en/column-count-limit.html
https://dev.mysql.com/doc/refman/8.0/en/innodb-restrictions.html

For some reason, text type on this particular windows instance + MySQL installation had smaller size. This might be generic issue on MySQL+Windows but we are yet to validate that.

Note: the user had around ~70 models.

Note: Fixing this will be a breaking change whenever this is fixed.

@schickling will create a meta issue about Lift specs

This is fixed now. I changed the column type for the MySQL connector for lift from TEXT to LONGTEXT which allows much longer strings. This fix will only be available to newly created projects. To fix an existing project i recommend the following procedure:

Delete the _Migration table in the database.
Delete the prisma/migrations folder in your project.


Can't find using a component import

Hey there, I'm having an issue where I cannot use a Vue component to find within a wrapper. I should note that I use vue-component-class Here's a small sample of my code
import { mount } from 'vue-test-utils';
import GameOwnerships from '@/game-ownership/GameOwnerships';
import GameOwnershipCard from '@/game-ownership/GameOwnershipCard';

// ... down the file in an 'describe' block

it('Should find based on in a component', () => {
  const wrapper = mount(GameOwnerships, { // ...stuff } );
  console.log(GameOwnershipCard);
  wrapper.find(GameOwnershipCard);
});

// ...

the result of the console log is
//  console.log(GameOwnershipCard);
{ [Function: VueComponent]
  cid: 2,
  options: //... I don't think it's relevant
  super: //... I don't think it's relevant
}

and the error thrown is
[vue-test-utils]: wrapper.find() must be passed a valid CSS selector, Vue constructor, or valid find option object

Using Versions
"vue-test-utils": "1.0.0-beta.6",
"vue": "2.5.9",
"vue-class-component": "5.0.2"

Did you have any luck with solving this, @Spittal ? I haven't used vue-component-class before, but I can take a look on the weekend at this, I'd like to learn more about it either way.

Hey! I actually didn't get very far. I tried digging into the GameOwnershipCard object to see if there's something that would satisfy the find method.
My next step would be to compare what a not decorated Vue component looks like compared to one that has the @Component decorator. Will do that over the weekend!

wrapper.find(GameOwnershipCard.options) works ^_^

This is fixed in beta.9 with 27e9d79448af2f7f2080ac71b30df3ebacc57ccf

mount is working fine for me but shallow does not work. If I use a find by name selector (wrapper.find({ name: 'xyz' }), it works as expected. I'm using class components too.

This is a

Another problem I found:
let button = wrapper.find({ name: 'xyz' });
expect(button.exists()).toEqual(true); -- OK
console.log(button.props()); -- EMPTY, but it should not be! It is working fine with mount() instead of shallow()

FYI, this function does not work:
function getCoreProperties(component) {
return {
attrs: component.attrs,
name: component.name,
on: component.on,
key: component.key,
ref: component.ref,
props: component.props,
domProps: component.domProps,
class: component.class,
staticClass: component.staticClass,
staticStyle: component.staticStyle,
style: component.style,
normalizedStyle: component.normalizedStyle,
nativeOn: component.nativeOn
};
}
I don't know why but class component options are in component.options


Use Twitter Bootstrap 4 for Pico default theme

I know I am getting ahead of myself here, but this should take care of the suggestion in #270 to port the default theme to Twitter Bootstrap 4...

[New] Require Twitter Bootstrap 4.0.0-alpha via MaxCDN
[New] added font-awesome stylesheet from MaxCDN …
[New] add blog example to default theme …
[New] blog, sidebar post, and full-width post layouts
[New] blog/ directory in content-sample with 2 posts demo'ing layout
[Changed] moved theme stylesheet to assets/css/
[Changed] style.css => bootstrap.theme.css
[Changed] Update index.twig to use bootstrap hirearchy/classes
[Changed] demonstrate twig by splitting and including html snippets
[Removed] menu-icon.png, bootstrap provides the text-based icon
[Removed] scripts/modernizr-2.6.1.min.js, bootstrap provides Normalize.css

Looks good :+1:

Hahah this is serious cool stuff here. Let's get it merged !

Closing this pull request.
Moved the feature/bootstrap-theme branch from theshka/Pico to picocms/Pico


Clean configs

This PR focuses on standardizing and enhancing the configuration documentation across the codebase. Consistency and uniformity make maintenance easier and improve readability. Future PRs will extend these improvements to other components, particularly the trainers.
Key Changes

Line Wrapping: Applied a consistent line wrap at column 220 to improve readability.
Definite Articles: Removed definite articles where possible to streamline language.
Type Annotations:

Always include type definitions, indicating if a parameter is optional and specifying the default value.
For optional values that can be None:foo (`Optional[int]`, *optional*, defaults to `None`):

For non-optional values:foo (`int`, *optional*, defaults to `4`):

String Defaults:

Ensured that default string values are wrapped in double quotes:defaults to `"foo"`

Dictionary Typing:

Replaced generic Dict type hints with more explicit Dict[str, Any] to clarify expected key-value pairs.

Default Value Formatting:

Consistently surrounded default values with backticks for improved formatting:defaults to `4`

Consistency Across Configurations: Ensured that similar arguments across different configurations have consistent descriptions

Progress

[ ] alignprop_config.py
[x] bco_config.py
[ ] cpo_config.py
[ ] ddpo_config.py
[ ] dpo_config.py
[ ] kto_config.py
[ ] model_config.py
[ ] online_dpo_config.py
[ ] orpo_config.py
[ ] ppo_config.py
[ ] ppov2_config.py
[ ] reward_config.py
[ ] rloo_config.py
[ ] sft_config.py

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Things are becoming really hard to review because I wanted to make all the config inherit from TrainingArguments. But I'll do it in another PR.


Prevent 'p' parameter of OPTIONS requests being logged.

Previously, OPTIONS requests were logged without passwords being redacted.
This occurred because password redaction is performed by the authentication
handler and a) the serveOptions handler did not have the parameter
that forces the authentication handler to be put on the request path and b)
the CORS handler returns early for OPTIONS requests
before the authentication handler is invoked.
With this commit, OPTIONS requests are also subject to the authentication handler
and the responsibility for the early return on OPTIONS request is moved from
the CORS handler to the authentication handler.
Signed-off-by: Jon Seymour<EMAIL_ADDRESS>

[X] CHANGELOG.md updated
[X] Rebased/mergable
[X] Tests pass *
[X] Sign CLA (if not already signed)
AFAICT the tests do pass. I did see what looked to be spurious failures of the circle-ci tests initiated by the PR, but these failures appeared unrelated to my changes and appear to have been fixed subsequently by mechanisms unknown.

@gunnaraasen

The big downside of this approach is that OPTIONS requests will be authenticated. Anyone using Basic Auth will receive an "unauthorized" response unless the Access-Control-Allow-Credentials response header is set, which seems unsafe.
@jonseymour I think it would make sense to keep the OPTIONS check in the CORS handler and sanitize the p parameter in serveOptions. E.g.
func (h *Handler) serveOptions(w http.ResponseWriter, r *http.Request) {
    q := r.URL.Query()
    if p := q.Get("p"); p != "" {
        q.Set("p", "[REDACTED]")
        r.URL.RawQuery = q.Encode()
    }
    w.WriteHeader(http.StatusNoContent)
}

@gunnaraasen If I keep the OPTIONS check in the CORS handler, the serveOptions method won't get invoked (because of the early return) and hence redaction won't occur.
Would it make more sense to move the redaction logic into the logging handler so that redaction is independent of whether authentication is enabled or not and also which path the request took to get to the logging path?

Thanks @jonseymour. I agree, moving the redaction code to the logger is a better approach. Can you also remove old sanitizing code in parseCredentials?

@gunnaraasen Sure. See 1d5ff55.

+1
@otoolep @toddboom this needs another plus one.

:+1:


Remove preview deployments for Dependabot-generated PRs

Since #989 we've been using a GitHub action for deployment of preview
apps and production. The preview action fails on Dependabot PRs because
Dependabot-run actions have read-only GitHub tokens and don't have
access to secrets, so they can't initiate the Deployment action step.
This turns off the deploy-preview workflow for actions initiated by
Dependabot so the whole CI doesn't fail (the same solution as #980).
We should have preview deployments for dependency PRs but it's almost
impossible with this actions workflow. In this repo all PRs have to be
up to date with the base branch before merging though, so reviewers are
often forced to click 'update branch' manually before merging, which
will trigger a deploy because it triggers a merge by the user who
clicked the button and therefore that user triggers the actions. This
workflow can be used by reviewers who need to check the docs site for
changes. It's not ideal but preferable to having checks fail on all
dependency PRs.
For more info on the secrets issue see:

https://github.com/dependabot/dependabot-core/issues/3253#issuecomment-852541544
https://github.com/dependabot/dependabot-core/issues/3253#issuecomment-797125425
https://github.blog/changelog/2021-02-19-github-actions-workflows-triggered-by-dependabot-prs-will-run-with-read-only-permissions/

In this repo all PRs have to be up to date with the base branch before merging though, so reviewers are often forced to click 'update branch' manually before merging, which will trigger a deploy... this workflow can be used by reviewers who need to check the docs site for changes.

I feel like this should be documented somewhere but I don't know where would be a good spot for it 🤔

In this repo all PRs have to be up to date with the base branch before merging though, so reviewers are often forced to click 'update branch' manually before merging, which will trigger a deploy... this workflow can be used by reviewers who need to check the docs site for changes.

I feel like this should be documented somewhere but I don't know where would be a good spot for it 🤔

We could add a new section to the contributors readme. Likely not the best place for it, but I can't think of any others.

I realised this is also an issue for primer/react and primer/css, so I've added a note to the DI docs https://github.com/github/design-infrastructure/pull/2267


Layout and styling tweaks

First off, thanks for this outstanding collection of OCaml resources! This PR is completely unsolicited, but I've been tinkering with some style changes to the site and thought I would submit them for consideration. I kept all css changes contained to css/style.scss, as it looks like that's where theme overrides should go.
Before:

After:

This should help a bit with #50 as well (I just noticed that issue). The sidebar content is now shown at full width below the page's content. The only issue I noticed on mobile is at small widths, the github link can overlap the heading a bit.

Beautiful job! I was hoping someone more knowledgeable at CSS would be able to improve the look of the site. Thanks a bunch!

Thanks! Feel free to ping me if you have any css issues for the site, I can try to take a look. I'm not a designer, but like to pretend that I am :P


Pkg.add("FFTW") shouldn't install MKL

Since https://github.com/JuliaMath/FFTW.jl/commit/8a9197a08890ca18b3aeb8bdff568087026778d8, ]add FFTW installs MKL. I'm not a free software absolutist, but I don't think adding such a core package (that used to be part of base) should download a non-free binary dependency. A much more practical concern is that MKL is a whopping 130mb, and my work connection sucks...

I definitely agree, but this is a difficult problem to address at the moment, since there is no support in the package manager for conditional dependencies.

Given that seems to be still some time away, perhaps reverting to the previous state would be best? Even with a good connection the download still takes quite a bit of time (during which it is unclear what's happening)

The previous state was having MKL managed by Conda, which introduces a Python dependency. It sort of seems like there are no good options at the moment...

I know very little about these things (and am eternally grateful to the poor souls that have to manage this) so sorry if I'm missing something, but isn't depending on python way better than depending on MKL? It's free software, it's more lightweight, it's shared by more packages, it runs on all sane platforms, and it's installed on most machines already.

isn't depending on python way better than depending on MKL

It was using Python to install MKL

Not by default though? It's definitely the first time I've noticed ]add FFTW downloads 130mb.

Previously it only installed Conda etcetera if FFTW.jl was configured to use MKL.  Now, because Pkg doesn't support optional dependencies, we must unconditionally download MKL_jll.

So why not switch back to the old system until Pkg supports optional dependencies?

Please revert this. You are making my life with a bad network and AMD processors very hard.

The artifact system already supports "lazy artifacts" that are only downloaded when the artifact is actually used.


In the provided yml script in part 1 and 2 azureml-sdk is not mentioned in.yml script.use pip to install azureml-sdk and the only the training will go as expected

[Enter feedback here]

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 23f9d033-9007-7b0a-d651-88d5d3846b2a
Version Independent ID: c6efa8cc-8594-698b-21eb-5e4d9ad24bbd
Content: Tutorial: Train your first machine learning model - Python - Azure Machine Learning
Content Source: articles/machine-learning/tutorial-1st-experiment-sdk-train.md
Service: machine-learning
Sub-service: core
GitHub Login: @aminsaied
Microsoft Alias: amsaied

Thank you for your detailed feedback.  We are looking into the issue, and we will respond when we have more information.

@aminsaied can you please review the PR or address this issue?

@kunamneni117 We will now proceed to close this thread. Our team is reviewing the content to ensure all required dependencies are installed.


[rabbitmq-ha] Set securityContext for rabbitmq container

Default values are compatible with default image (rabbitmq:3.7-alpine).
Note that securityContext is set on container level which means it affects only rabbitmq container.
Increase only bugfix part of chart version because default securityContext.enabled is false.
Signed-off-by: hasul<EMAIL_ADDRESS>
What this PR does / why we need it:
Set securityContext for rabbitmq container.
Which issue this PR fixes
none
Special notes for your reviewer:
Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

[x] DCO signed
[x] Chart Version bumped
[x] Variables are documented in the README.md

/assign

/lgtm

Thanks @matejhasul


Fixing Rescript Language Server

Hey! Recently I've opened a PR(#318) to update the mapping to lspconfig of rescript lsp, because the previous one was deprecated, but since it changed the LSP executable, it also changed the command to start the LSP itself, and I was not aware that this project also kept a mapping of the cmd that should be executed, so this broke the spawning of rescript-lsp :P
Adding this PR to fix it.
PS: Wouldn't be better to get this kind of config from lspconfig? Or the intention is just to re-use the name in lspconfig even if we're spawning a completely different program?

@williambotman /fixup

Hey! The cmd in lspconfig was only recently added there, so that's why this was needed. Since this is the same as what's in lspconfig we can remove the override here

Superseded by #330


autoscaler doesn't work with dataflow load (export/import)

Dataflow is considered as medium priority  https://cloud.google.com/spanner/docs/cpu-utilization
so when there is high Dataflow load (import/export) and low high priority load autoscaler just doesn't work at all.

while it is possible to fix with custom metric
      "metrics" = [
        {
          "name"                     = "cpu_utilization_total"
          "filter"                   = "metric.type=\"spanner.googleapis.com/instance/cpu/utilization\""
          "regional_threshold"       = 90
          "multi_regional_threshold" = 90
        },

probably would be great if such rule will be added into default set

@oprudkyi Thank you for the feedback.
With the default metrics we were intentionally conservative and only wanting to evaluate the metrics called out in the Cloud Spanner docs.  For cases Dataflow (medium priority CPU) and for backups (low priority CPU) utilization, I have been considering creating a library of custom metrics with descriptions on the use cases for each metric.

Here are a couple custom metrics that specifically target medium and low priority CPU usage.
Medium CPU usage, useful for autoscaling for Dataflow jobs.
"metrics": [
  {
    "name": "medium_cpu_usage",
    "filter": "metric.type=\"spanner.googleapis.com/instance/cpu/utilization_by_priority\" AND metric.label.priority=\"high\"",
    "regional_threshold": 40,
    "multi_regional_threshold": 30
  }
]

Low CPU usage, useful for managing jobs like backups and index backfill. The amount of CPU allocated for these tasks is managed by Cloud Spanner in relation to the other CPU priorities. Scaling up for these tasks may have minimal impact particularly with instances with low overall utilization.
"metrics": [
  {
    "name": "low_cpu_usage",
    "filter": "metric.type=\"spanner.googleapis.com/instance/cpu/utilization_by_priority\" AND metric.label.priority=\"low\"",
    "regional_threshold": 40,
    "multi_regional_threshold": 30
  }
]


Add Validation data type to Extra modules

It's not in base, but in either package:

https://hackage.haskell.org/package/either-5.0.1/docs/Data-Either-Validation.html#t:Validation

Though you don't usually want to drag all that dependencies, so maybe it makes sense to reimplement it in some Extra module?

@vrom911 Completely agree with this 👍 💯 You probably don't want to depend on either package in lightweight libraries or applications. But Validation data type is extremely useful. And since we have Extra modules, it's relatively easy to add known and common things to the relude without worrying about name clashes.

Would this include a reimplementation of Validation with new enhancements/features or we would just reuse the current data declaration and instances from Data.Either ?

@kahlil29 Couple hours ago I've received an unexpected email from @mauriciofierrom with the attempt to resolve this issue. Here is the gist:

https://gist.github.com/mauriciofierrom/bdb7a5d0ff06c4a23b51ef7bfa7d81d9

I will duplicate my comments to this solution here:

You can also add Bifoldable and Bitraversable instances. These typeclasses are already in base and we're going to reexport them as well. See relevant issues:

https://github.com/kowainik/relude/issues/98
https://github.com/kowainik/relude/issues/99

It would be really nice to use -XInstanceSigs extension and all type signatures to all methods in all instances. In relude we want our package to be as beginner-friendly as possible, and having such type signatures significantly increases code readability. See example in this module:

https://github.com/kowainik/relude/blob/master/src/Relude/Extra/Foldable1.hs

Also, usually, when you're writing general purpose library, it's a good idea to implement all methods from the typeclasses. Default implementations might not be efficient.
In addition to previous point: adding {-# INLINE #-} pragma to every implemented method helps performance as well.
Also, in relude we're using 4 spaces for indentation :)

@ChShersh thanks for the comments. If it's ok with you (and @kahlil29) I will work on this issue then.


Codefresh wait to have build queue

What

Provide Codefresh wait target

Why

Allow guaranteed builds order

The feature can be solved by commands in codefresh.yaml

  wait:
    title: Wait
    stage: Prepare
    image: codefresh/cli
    commands:
      - >-
          export QUEUE_HEAD_ID=$( \
            codefresh get builds \
              --pipeline-name=integration-create \
              --pipeline-name=integration-destroy \
              --branch ${{CF_BRANCH}} \
              --status running \
              --limit 1000 -o id | \
            tail -1 | tr -dc '[[:print:]]' \
          )
      - echo "|${QUEUE_HEAD_ID}| == |${CF_BUILD_ID}|"
      - 'if [ "${QUEUE_HEAD_ID}" != "${CF_BUILD_ID}" ]; then exit 1; fi'
    retry:
      maxAttempts: 10
      delay: 20
      exponentialFactor: 1.1


Output Compile Window is too small to see executed code and see verified test status

Previously, the output window was large enough to scroll through tests or lines of compiled code to see how the code handled the tests. But since last week, it appears that the window has been resized and appears to not be functioning properly. This occurs independent of the kata, language or account used. I tried creating a new account to see if it was an issue with my account settings, but saw the same problem occur. I have attached a screenshot for reference.

Nothing was changed, this seems to be an issue with possibly with a recent release of Chrome. I am working on finding a fix, using FIrefox is the workaround for now.

Thank you for the prompt reply!
On Mon, Oct 30, 2017 at 11:34 AM, Jake Hoffner<EMAIL_ADDRESS>wrote:

Nothing was changed, this seems to be an issue with possibly with a recent
release of Chrome. I am working on finding a fix, using FIrefox is the
workaround for now.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Codewars/codewars.com/issues/1207#issuecomment-340503493,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AfsEuX-kF1GXTKoKcJ3NggXLVv65SOGgks5sxfqbgaJpZM4QLWIF
.

Related (or duplicate of) #1202