UnicodeDecodeError when trying to install requests on Windows (using Appveyor)

Version: 2.16.1
Works fine on Linux and macOS.
Has also worked on Windows in the past (2.14.1).
I'll be attempting a installation of requests 2.14.1 specifically to see if there has indeed been any regression, the build is here.

Stacktrace
Searching for requests
Reading https://pypi.python.org/simple/requests/
Downloading https://pypi.python.org/packages/4c/54/1d3abddbd4c7544138b88e8329ef5294ffdc6c5d7ea965bf42e3cc4c9c39/requests-2.16.1.tar.gz#md5=4acb351824461eb3ae1f74ec3a65ff28
Best match: requests 2.16.1
Processing requests-2.16.1.tar.gz
Writing C:\Users\appveyor\AppData\Local\Temp\1\easy_install-qoyp00fn\requests-2.16.1\setup.cfg
Running requests-2.16.1\setup.py -q bdist_egg --dist-dir C:\Users\appveyor\AppData\Local\Temp\1\easy_install-qoyp00fn\requests-2.16.1\egg-dist-tmp-rqjnti7r
Traceback (most recent call last):
  File "C:\Python35\lib\site-packages\setuptools\sandbox.py", line 157, in save_modules
    yield saved
  File "C:\Python35\lib\site-packages\setuptools\sandbox.py", line 198, in setup_context
    yield
  File "C:\Python35\lib\site-packages\setuptools\sandbox.py", line 255, in run_setup
    DirectorySandbox(setup_dir).run(runner)
  File "C:\Python35\lib\site-packages\setuptools\sandbox.py", line 285, in run
    return func()
  File "C:\Python35\lib\site-packages\setuptools\sandbox.py", line 253, in runner
    _execfile(setup_script, ns)
  File "C:\Python35\lib\site-packages\setuptools\sandbox.py", line 47, in _execfile
    exec(code, globals, locals)
  File "C:\Users\appveyor\AppData\Local\Temp\1\easy_install-qoyp00fn\requests-2.16.1\setup.py", line 51, in <module>
  File "C:\Python35\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 441: character maps to <undefined>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "setup.py", line 18, in <module>
    'https://github.com/ActivityWatch/aw-core/tarball/master#egg=aw-core'
  File "C:\Python35\lib\distutils\core.py", line 148, in setup
    dist.run_commands()
  File "C:\Python35\lib\distutils\dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "C:\Python35\lib\distutils\dist.py", line 974, in run_command
    cmd_obj.run()
  File "C:\Python35\lib\site-packages\setuptools\command\install.py", line 67, in run
    self.do_egg_install()
  File "C:\Python35\lib\site-packages\setuptools\command\install.py", line 117, in do_egg_install
    cmd.run()
  File "C:\Python35\lib\site-packages\setuptools\command\easy_install.py", line 411, in run
    self.easy_install(spec, not self.no_deps)
  File "C:\Python35\lib\site-packages\setuptools\command\easy_install.py", line 654, in easy_install
    return self.install_item(None, spec, tmpdir, deps, True)
  File "C:\Python35\lib\site-packages\setuptools\command\easy_install.py", line 701, in install_item
    self.process_distribution(spec, dist, deps)
  File "C:\Python35\lib\site-packages\setuptools\command\easy_install.py", line 746, in process_distribution
    [requirement], self.local_index, self.easy_install
  File "C:\Python35\lib\site-packages\pkg_resources\__init__.py", line 851, in resolve
    dist = best[req.key] = env.best_match(req, ws, installer)
  File "C:\Python35\lib\site-packages\pkg_resources\__init__.py", line 1123, in best_match
    return self.obtain(req, installer)
  File "C:\Python35\lib\site-packages\pkg_resources\__init__.py", line 1135, in obtain
    return installer(requirement)
  File "C:\Python35\lib\site-packages\setuptools\command\easy_install.py", line 673, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File "C:\Python35\lib\site-packages\setuptools\command\easy_install.py", line 699, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File "C:\Python35\lib\site-packages\setuptools\command\easy_install.py", line 880, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "C:\Python35\lib\site-packages\setuptools\command\easy_install.py", line 1119, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "C:\Python35\lib\site-packages\setuptools\command\easy_install.py", line 1105, in run_setup
    run_setup(setup_script, args)
  File "C:\Python35\lib\site-packages\setuptools\sandbox.py", line 258, in run_setup
    raise
  File "C:\Python35\lib\contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "C:\Python35\lib\site-packages\setuptools\sandbox.py", line 198, in setup_context
    yield
  File "C:\Python35\lib\contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "C:\Python35\lib\site-packages\setuptools\sandbox.py", line 169, in save_modules
    saved_exc.resume()
  File "C:\Python35\lib\site-packages\setuptools\sandbox.py", line 144, in resume
    six.reraise(type, exc, self._tb)
  File "C:\Python35\lib\site-packages\six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "C:\Python35\lib\site-packages\setuptools\sandbox.py", line 157, in save_modules
    yield saved
  File "C:\Python35\lib\site-packages\setuptools\sandbox.py", line 198, in setup_context
    yield
  File "C:\Python35\lib\site-packages\setuptools\sandbox.py", line 255, in run_setup
    DirectorySandbox(setup_dir).run(runner)
  File "C:\Python35\lib\site-packages\setuptools\sandbox.py", line 285, in run
    return func()
  File "C:\Python35\lib\site-packages\setuptools\sandbox.py", line 253, in runner
    _execfile(setup_script, ns)
  File "C:\Python35\lib\site-packages\setuptools\sandbox.py", line 47, in _execfile
    exec(code, globals, locals)
  File "C:\Users\appveyor\AppData\Local\Temp\1\easy_install-qoyp00fn\requests-2.16.1\setup.py", line 51, in <module>
  File "C:\Python35\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 441: character maps to <undefined>

The build has now progressed beyond the place where it erred previously, so 2.14.1 seems to work fine.

I think the fix from f3d310b299af587d668493f09d82c84c197d00fd needs to be adapted to the change in 3e32356716572e59c9f10c6c2622bedf5f48b07e.

@tdsmith we no longer load that file that way any longer.

https://github.com/kennethreitz/requests/blob/master/setup.py#L50-L51 appears to still be loading a file with some unicode in it.

Unfortunately, "You can't use a PEP 263 declaration in Unicode text passed to exec". It results in a SyntaxError.

I'll see if I can drop the  PEP 263 declaration and have things still work alright on most systems.

Does anyone have the unicode literals for these emojis? ✨🍰✨ :P

It looks like you can pass bytes to exec() in 3.6; maybe opening the file in binary mode with the PEP 263 declaration intact is good?
Also '\u2728\U0001f370\u2728' I think

@tdsmith lifesaver!

I'm going to try passing it as unicode and see if that fixes up things for appveyor over here

k, changes landed in master for this. not sure if they'll actually effect anything, but...

No further reports of this being an issue, so I'm going to close this until we hear otherwise.


Including password reset option using email to existing PHP project

Describe the Feature you want to end
Including password reset option using email to existing PHP project
Expected behavior

People can reset their password in case they forgot.

@pranjay-poddar @Saurabh-kumar09 @MohitGupta121 @TusharKesarwani pls assign me this issue sir

Sir why this has been closed? Still four days are left right?

Thankyou sir, I was having other works, so couldn't complete this one. Will try to complete this within the deadline.

Dear Sir, @pranjay-poddar @Saurabh-kumar09 @MohitGupta121 @TusharKesarwani
I couldn't work on this issue due to college works. The whole day (Sunday, 06/08/2023), I completely worked upon this and added a complex issue (resetting password in database) as a feature to the existing PHP project. As the GSSoC'23 is going to end, I kindly request you to accept this PR as soon as possible and give this feature a higher level (just a request, a lot has been worked upon this feature) so that I get more points.
Thankyou.

Dear Sir, @pranjay-poddar @MohitGupta121 @TusharKesarwani,
README.md file was missing in my PR. Now I have added and updated it too. Kindly review it. The coding period is going to end soon sir, please accept my PR.


[feat] Implement OP_PICK opcode

Implement the OP_PICK opcode using the opcodePick reference implementation from btcd
References
Guide on how to add an opcode in shinigami

Ban

Hi @b-j-roberts  returning contributor. Any chance I get this one

@Oshioke-Salaki glad to have you on the project. LMK if you have questions!


Add stepsecuritynodetest to this repo itself

@arjundashrath can we add the nodetest npm package to this repo, in a separate folder?

Yes sir, will do


[Sql Server] Error migrating tables

Hello, thanks for sharing this library.
Sorry for a newbie question.
I am trying to use this library with Sql Server
but I receive this error message
-- Error migrating tables --
How to Reproduce:
Run SqlServer on docker
docker run --name sql1 -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=!@#p455qwe' -p 11433:1433 -d mcr.microsoft.com/mssql/server:2017-latest
Install Database Migrations(CLI) on mac
composer require 'byjg/migration-cli=4.0.*'
Change test_sqlserver.php like this
<?php

require __DIR__ . "/../../vendor/autoload.php";

/**
 * Make sure you have a database with the user 'migrateuser' and password 'migratepwd'
 *
 * This user need to have grant for DDL commands;
 */

$uri = new \ByJG\Util\Uri('dblib://sa:!@#p455qwe@localhost:1433/MyDBName');

$migration = new \ByJG\DbMigration\Migration($uri, __DIR__);
$migration->registerDatabase('dblib', \ByJG\DbMigration\Database\DblibDatabase::class);

$migration->prepareEnvironment();

$migration->reset();

Run this command
~ $ vendor/bin/migrate create /Users/a/test
Created UP version: 1
Created DOWN version: 0

~ $ vendor/bin/migrate install --path /Users/a/vendor/byjg/migration/example/sql_server mssql://localhost:1433/DragonRunner
-- Error migrating tables --

I can connect to sql using this
http://localhost:8080/indexSQL.php
<?php
$serverName = "<IP_ADDRESS>,1433";

$connectionInfo = array("DATABASE"=>"MyDBName","UID"=>"sa","PWD"=>"!@#p455qwe");
$conn = sqlsrv_connect( $serverName, $connectionInfo);

if( $conn ) {
     echo "Connection established.
";
}else{
     echo "Connection could not be established.
";
     die( print_r( sqlsrv_errors(), true));
}
?>

Question 1:
Please tell step by step how to use this library with sql server,
what file need to change sql user and password and server address
and how to test the connection established.
Question 2:
How to find more details of this -- Error migrating tables --,
such as if the error is because the connection failed
Mac php version
$ php -v
PHP 7.3.11 (cli) (built: Jun  5 2020 23:50:40) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.3.11, Copyright (c) 1998-2018 Zend Technologies

First of all, thank you :)
I have some considerations about your step by step:

When you run the sqlserver image the port you exposed was 11433 and not the default one 1433. It means you need to specify the port when you are calling the migrate command. Another point important: you need to pass the password to the connection string. And the last but not least the schema is not mssql, it is dblib. So based on your example the correct connection string will be:

dblib://sa:!@#p455qwe@localhost:11433/DragonRunner

To get a more detailed error message you can call the migrate with -v argument.

The command create only creates the directory structure. As you are running the example, you don't need to run the create first.

The command install is only necessary when you want add the migration to an existing database. Again, as you are running the example from scratch you just need call the reset

vendor/bin/migrate reset dblib://localhost:1433/DragonRunner

Another point I noted the php you stated isconnecting to the Sql Server has a different host than you ae using in the migrate connection string. While you are using in the migrate localhost the example connects to <IP_ADDRESS>

The password with the exclamation point (!) has some issues when you are using on linux terminal . You can get the error message event not found. Take a look here. I recommend you to avoid use it.

The latest version is 4.1.0  and not 4.0.0. I recommend update also. Sorry, the documentation isn't update.

First of all, thank you :)
I have some considerations about your step by step:

When you run the sqlserver image the port you exposed was 11433 and not the default one 1433. It means you need to specify the port when you are calling the migrate command. Another point important: you need to pass the password to the connection string. And the last but not least the schema is not mssql, it is dblib. So based on your example the correct connection string will be:

dblib://sa:!@#p455qwe@localhost:11433/DragonRunner

To get a more detailed error message you can call the migrate with -v argument.

The command create only creates the directory structure. As you are running the example, you don't need to run the create first.

The command install is only necessary when you want add the migration to an existing database. Again, as you are running the example from scratch you just need call the reset

vendor/bin/migrate reset dblib://localhost:1433/DragonRunner

Another point I noted the php you stated isconnecting to the Sql Server has a different host than you ae using in the migrate connection string. While you are using in the migrate localhost the example connects to <IP_ADDRESS>

The password with the exclamation point (!) has some issues when you are using on linux terminal . You can get the error message event not found. Take a look here. I recommend you to avoid use it.

The latest version is 4.1.0  and not 4.0.0. I recommend update also. Sorry, the documentation isn't update.

Hello, thank you for quick reply ^^
Now I've got this error
~ $ vendor/bin/migrate reset dblib://localhost:11433/Runner -v
This will ERASE all of data in your data. Continue with this action? (y/N) y
Connection String: dblib://localhost:11433/Runner
Path: /Users/a
-- Error migrating tables --
ByJG\AnyDataset\Core\Exception\NotAvailableException
Extension 'pdo_dblib' is not loaded

so I try to install pdo_dblib but it was deprecated.
I remember that I am using sqlsrv_connect extension in indexSQL.php
that's why it can connect
Does your library support sqlsrv_connect ?
https://docs.microsoft.com/en-us/sql/connect/php/installation-tutorial-linux-mac?view=sql-server-ver15

Hello, thank you for quick reply ^^
Now I've got this error
~ $ vendor/bin/migrate reset dblib://localhost:11433/Runner -v
This will ERASE all of data in your data. Continue with this action? (y/N) y
Connection String: dblib://localhost:11433/Runner
Path: /Users/a
-- Error migrating tables --
ByJG\AnyDataset\Core\Exception\NotAvailableException
Extension 'pdo_dblib' is not loaded

so I try to install pdo_dblib but it was deprecated.
I remember that I am using sqlsrv_connect extension in indexSQL.php
that's why it can connect
Does your library support sqlsrv_connect ?
https://docs.microsoft.com/en-us/sql/connect/php/installation-tutorial-linux-mac?view=sql-server-ver15

Right now we don't have support for sqlsrv_connect, however it is something to think about. Let me analyse what can be done and I return to you soon.

Right now we don't have support for sqlsrv_connect, however it is something to think about. Let me analyse what can be done and I return to you soon.

The problem with the Microsoft Sql Server connector for PHP Linux is the driver is a wrapper for ODBC. It means you have an intermediate layer acting making the connection slower. The dblib driver connects using the databse protocol without intermediaries directly
Meanwhile, could you try to install the DBLIB on MAC following this tutorial? https://gist.github.com/jschroed91/8235909e16f79fbb31feff1f313aa9ca
As you are using Docker you can try also use our PHP docker image. It has all necessary modules you need. You can pull the image by docker pull byjg/php:7.3-cli and is available from PHP versions 5.6 to 8.0 and we have five image flavours: base (minimal), command line, fpm, fpm+nginx, fpm+apache.  More info: https://opensource.byjg.com/devops/docker-php/

The problem with the Microsoft Sql Server connector for PHP Linux is the driver is a wrapper for ODBC. It means you have an intermediate layer acting making the connection slower. The dblib driver connects using the databse protocol without intermediaries directly
Meanwhile, could you try to install the DBLIB on MAC following this tutorial? https://gist.github.com/jschroed91/8235909e16f79fbb31feff1f313aa9ca
As you are using Docker you can try also use our PHP docker image. It has all necessary modules you need. You can pull the image by docker pull byjg/php:7.3-cli and is available from PHP versions 5.6 to 8.0 and we have five image flavours: base (minimal), command line, fpm, fpm+nginx, fpm+apache.  More info: https://opensource.byjg.com/devops/docker-php/

Thanks for the advice
Even after I install the DBLIB on Mac,
I think it won't work because of our side (not your library)
many of our API is already written using SQLSRV Functions
https://www.php.net/manual/en/book.sqlsrv.php

Is okay if the connection become slower
we already get used to using the syntax
We prefer to choose the recommendation to use official extensions from Microsoft from here:
https://github.com/Microsoft/msphpsql

We would appreciate it if you could make this library support Microsoft SQL Server Driver for PHP
until then we will wait.

Thanks for the advice
Even after I install the DBLIB on Mac,
I think it won't work because of our side (not your library)
many of our API is already written using SQLSRV Functions
https://www.php.net/manual/en/book.sqlsrv.php

Is okay if the connection become slower
we already get used to using the syntax
We prefer to choose the recommendation to use official extensions from Microsoft from here:
https://github.com/Microsoft/msphpsql

We would appreciate it if you could make this library support Microsoft SQL Server Driver for PHP
until then we will wait.

All right. Let me think how I'll add it.

All right. Let me think how I'll add it.

Hi @c4ndle , I implemented the Sqlsrv to be deployed in the next migration version. I'd like you test it for me before I create a new version.
Please update the migration temporarly to dev-next  (e.g. composer require byjg/migration:dev-next) and let me know if it is working.
The connection string is:
sqlsrv://username:password@hostname:port/database

Hi @c4ndle , I implemented the Sqlsrv to be deployed in the next migration version. I'd like you test it for me before I create a new version.
Please update the migration temporarly to dev-next  (e.g. composer require byjg/migration:dev-next) and let me know if it is working.
The connection string is:
sqlsrv://username:password@hostname:port/database

Hello, thank you for the update,
I did try run composer require byjg/migration:dev-next
But I have problem
composer require byjg/migration:dev-next
./composer.json has been updated
Running composer update byjg/migration
Loading composer repositories with package information
Updating dependencies
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Root composer.json requires byjg/migration dev-next -> satisfiable by byjg/migration[dev-next].
    - byjg/migration dev-next requires byjg/anydataset-db 4.1.2.x-dev -> found byjg/anydataset-db[4.1.2.x-dev] but it does not match your minimum-stability.

Use the option --with-all-dependencies (-W) to allow upgrades, downgrades and removals for packages currently locked to specific versions.

Installation failed, reverting ./composer.json and ./composer.lock to their original content.

Try using -W option
composer require byjg/migration:dev-next -W

./composer.json has been updated
Running composer update byjg/migration --with-all-dependencies
Loading composer repositories with package information
Updating dependencies
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Root composer.json requires byjg/migration dev-next -> satisfiable by byjg/migration[dev-next].
    - byjg/migration dev-next requires byjg/anydataset-db 4.1.2.x-dev -> found byjg/anydataset-db[4.1.2.x-dev] but it does not match your minimum-stability.

Use the option --with-all-dependencies (-W) to allow upgrades, downgrades and removals for packages currently locked to specific versions.

Installation failed, reverting ./composer.json and ./composer.lock to their original content.

Btw I did run composer require "byjg/migration":"4.1.*" before run composer require byjg/migration:dev-next it seems succeeded
composer require "byjg/migration":"4.1.*"
./composer.json has been created
Running composer update byjg/migration
Loading composer repositories with package information
Updating dependencies
Lock file operations: 9 installs, 0 updates, 0 removals
  - Locking byjg/anydataset (4.0.1)
  - Locking byjg/anydataset-array (4.0.0)
  - Locking byjg/anydataset-db (4.1.1)
  - Locking byjg/migration (4.1.0)
  - Locking byjg/serializer (1.0.7)
  - Locking byjg/uri (2.1.0)
  - Locking byjg/xmlutil (1.0.6)
  - Locking psr/cache (1.0.1)
  - Locking psr/http-message (1.0.1)
Writing lock file
Installing dependencies from lock file (including require-dev)
Package operations: 9 installs, 0 updates, 0 removals
  - Installing byjg/xmlutil (1.0.6): Extracting archive
  - Installing byjg/serializer (1.0.7): Extracting archive
  - Installing byjg/anydataset (4.0.1): Extracting archive
  - Installing byjg/anydataset-array (4.0.0): Extracting archive
  - Installing psr/cache (1.0.1): Extracting archive
  - Installing psr/http-message (1.0.1): Extracting archive
  - Installing byjg/uri (2.1.0): Extracting archive
  - Installing byjg/anydataset-db (4.1.1): Extracting archive
  - Installing byjg/migration (4.1.0): Extracting archive
2 package suggestions were added by new dependencies, use `composer suggest` to see details.
Generating autoload files

Hello, thank you for the update,
I did try run composer require byjg/migration:dev-next
But I have problem
composer require byjg/migration:dev-next
./composer.json has been updated
Running composer update byjg/migration
Loading composer repositories with package information
Updating dependencies
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Root composer.json requires byjg/migration dev-next -> satisfiable by byjg/migration[dev-next].
    - byjg/migration dev-next requires byjg/anydataset-db 4.1.2.x-dev -> found byjg/anydataset-db[4.1.2.x-dev] but it does not match your minimum-stability.

Use the option --with-all-dependencies (-W) to allow upgrades, downgrades and removals for packages currently locked to specific versions.

Installation failed, reverting ./composer.json and ./composer.lock to their original content.

Try using -W option
composer require byjg/migration:dev-next -W

./composer.json has been updated
Running composer update byjg/migration --with-all-dependencies
Loading composer repositories with package information
Updating dependencies
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Root composer.json requires byjg/migration dev-next -> satisfiable by byjg/migration[dev-next].
    - byjg/migration dev-next requires byjg/anydataset-db 4.1.2.x-dev -> found byjg/anydataset-db[4.1.2.x-dev] but it does not match your minimum-stability.

Use the option --with-all-dependencies (-W) to allow upgrades, downgrades and removals for packages currently locked to specific versions.

Installation failed, reverting ./composer.json and ./composer.lock to their original content.

Btw I did run composer require "byjg/migration":"4.1.*" before run composer require byjg/migration:dev-next it seems succeeded
composer require "byjg/migration":"4.1.*"
./composer.json has been created
Running composer update byjg/migration
Loading composer repositories with package information
Updating dependencies
Lock file operations: 9 installs, 0 updates, 0 removals
  - Locking byjg/anydataset (4.0.1)
  - Locking byjg/anydataset-array (4.0.0)
  - Locking byjg/anydataset-db (4.1.1)
  - Locking byjg/migration (4.1.0)
  - Locking byjg/serializer (1.0.7)
  - Locking byjg/uri (2.1.0)
  - Locking byjg/xmlutil (1.0.6)
  - Locking psr/cache (1.0.1)
  - Locking psr/http-message (1.0.1)
Writing lock file
Installing dependencies from lock file (including require-dev)
Package operations: 9 installs, 0 updates, 0 removals
  - Installing byjg/xmlutil (1.0.6): Extracting archive
  - Installing byjg/serializer (1.0.7): Extracting archive
  - Installing byjg/anydataset (4.0.1): Extracting archive
  - Installing byjg/anydataset-array (4.0.0): Extracting archive
  - Installing psr/cache (1.0.1): Extracting archive
  - Installing psr/http-message (1.0.1): Extracting archive
  - Installing byjg/uri (2.1.0): Extracting archive
  - Installing byjg/anydataset-db (4.1.1): Extracting archive
  - Installing byjg/migration (4.1.0): Extracting archive
2 package suggestions were added by new dependencies, use `composer suggest` to see details.
Generating autoload files

OK, use the following composer.json
{
    "require": {
        "byjg/migration": "dev-next"
    },
    "minimum-stability": "dev"
}

OK, use the following composer.json
{
    "require": {
        "byjg/migration": "dev-next"
    },
    "minimum-stability": "dev"
}

Did it work as expected?

Did it work as expected?

Hello, sorry I can not test it soon (busy with client side development).
I will contact you again in 2 days,
btw, is it works on your local environment ?

Hello, sorry I can not test it soon (busy with client side development).
I will contact you again in 2 days,
btw, is it works on your local environment ?

No problem.

btw, is it works on your local environment ?

Yes, it is, but I would like just to make sure everything is OK.

No problem.

btw, is it works on your local environment ?

Yes, it is, but I would like just to make sure everything is OK.

updating dev-next after edit composer.json looks good
$ composer require byjg/migration:dev-next
./composer.json has been updated
Running composer update byjg/migration
Loading composer repositories with package information
Updating dependencies
Nothing to modify in lock file
Installing dependencies from lock file (including require-dev)
Nothing to install, update or remove
Generating autoload files

And then I have an exception command not found
migration-cli a$ scripts/migrate --path /Users/a/migration-tool/vendor/byjg/migration/example/sql_server  reset sqlsrv://sa:MssqlP455123@sql1:1433/testdb -v

In Application.php line 676:

  [Symfony\Component\Console\Exception\CommandNotFoundException]
  Command "/Users/a/migration-tool/vendor/byjg/migration/example/sql_server" is not defined.

Exception trace:
  at /Users/a/migration-tool/migration-cli/vendor/symfony/console/Application.php:676
 Symfony\Component\Console\Application->find() at /Users/a/migration-tool/migration-cli/vendor/symfony/console/Application.php:237
 Symfony\Component\Console\Application->doRun() at /Users/a/migration-tool/migration-cli/vendor/symfony/console/Application.php:149
 Symfony\Component\Console\Application->run() at /Users/a/migration-tool/migration-cli/vendor/byjg/migration-cli/scripts/migrate:23

Do I have to update https://github.com/byjg/migration-cli too ?
using composer require 'byjg/migration-cli=4.0.*'
vendor/bin/migrate reset sqlsrv://sa:MssqlP455123@sql1:1433/testdb -v
This will ERASE all of data in your data. Continue with this action? (y/N) y
Connection String: sqlsrv://sa:MssqlP455123@sql1:1433/testdb
Path: /Users/a/migration-tool/migration-cli
-- Error migrating tables --
ByJG\DbMigration\Exception\DatabaseDoesNotRegistered
Scheme "sqlsrv" does not found. Did you registered it?

updating dev-next after edit composer.json looks good
$ composer require byjg/migration:dev-next
./composer.json has been updated
Running composer update byjg/migration
Loading composer repositories with package information
Updating dependencies
Nothing to modify in lock file
Installing dependencies from lock file (including require-dev)
Nothing to install, update or remove
Generating autoload files

And then I have an exception command not found
migration-cli a$ scripts/migrate --path /Users/a/migration-tool/vendor/byjg/migration/example/sql_server  reset sqlsrv://sa:MssqlP455123@sql1:1433/testdb -v

In Application.php line 676:

  [Symfony\Component\Console\Exception\CommandNotFoundException]
  Command "/Users/a/migration-tool/vendor/byjg/migration/example/sql_server" is not defined.

Exception trace:
  at /Users/a/migration-tool/migration-cli/vendor/symfony/console/Application.php:676
 Symfony\Component\Console\Application->find() at /Users/a/migration-tool/migration-cli/vendor/symfony/console/Application.php:237
 Symfony\Component\Console\Application->doRun() at /Users/a/migration-tool/migration-cli/vendor/symfony/console/Application.php:149
 Symfony\Component\Console\Application->run() at /Users/a/migration-tool/migration-cli/vendor/byjg/migration-cli/scripts/migrate:23

Do I have to update https://github.com/byjg/migration-cli too ?
using composer require 'byjg/migration-cli=4.0.*'
vendor/bin/migrate reset sqlsrv://sa:MssqlP455123@sql1:1433/testdb -v
This will ERASE all of data in your data. Continue with this action? (y/N) y
Connection String: sqlsrv://sa:MssqlP455123@sql1:1433/testdb
Path: /Users/a/migration-tool/migration-cli
-- Error migrating tables --
ByJG\DbMigration\Exception\DatabaseDoesNotRegistered
Scheme "sqlsrv" does not found. Did you registered it?

btw, is it works on your local environment ?

Yes, it is, but I would like just to make sure everything is OK.
Could you tell more specific how do you do that from the ground up @byjg

btw, is it works on your local environment ?

Yes, it is, but I would like just to make sure everything is OK.
Could you tell more specific how do you do that from the ground up @byjg

Hi, I am generating a new release version where I can have all tied togheter.
After merge PR #31 in a few minutes I'll send the proper instructions.

Hi, I am generating a new release version where I can have all tied togheter.
After merge PR #31 in a few minutes I'll send the proper instructions.

I bump all component versions, so you can use without any "hacks".

If you are using the migration CLI - the component version is 4.1.0
If you are using the migration as a PHP library - the component version is 4.2.0

e.g.:
composer require byjg/migration-cli=4.1.*

To use the sqlsrv_connect you need to use the schema sqlsrv as sqlsrv://sa:password@localhost/database
Let me know if everything is working.

I bump all component versions, so you can use without any "hacks".

If you are using the migration CLI - the component version is 4.1.0
If you are using the migration as a PHP library - the component version is 4.2.0

e.g.:
composer require byjg/migration-cli=4.1.*

To use the sqlsrv_connect you need to use the schema sqlsrv as sqlsrv://sa:password@localhost/database
Let me know if everything is working.

Finally it works using this command
vendor/bin/migrate reset --path /Users/a/migration-tool/vendor/byjg/migration-cli/example/sql_server sqlsrv://sa:MssqlP455123@localhost:1433/testdb

2021-01-18 06:09:22.44 spid56      Starting up database 'testdb'.
2021-01-18 06:09:22.64 spid56      Parallel redo is started for database 'testdb' with worker pool size [3].
2021-01-18 06:09:22.65 spid56      Parallel redo is shutdown for database 'testdb' with worker pool size [3].

If using API, I have an error
Fatal error: Uncaught PDOException: SQLSTATE[HYT00]: [Microsoft][ODBC Driver 17 for SQL Server]Login timeout expired in /code/vendor/byjg/anydataset-db/src/PdoSqlsrv.php:28 Stack trace: #0 /code/vendor/byjg/anydataset-db/src/PdoSqlsrv.php(28): PDO->__construct('sqlsrv:server=0...', 'sa', 'MssqlP455123') #1 /code/vendor/byjg/anydataset-db/src/Factory.php(51): ByJG\AnyDataset\Db\PdoSqlsrv->__construct(Object(ByJG\Util\Uri)) #2 /code/vendor/byjg/anydataset-db/src/Factory.php(16): ByJG\AnyDataset\Db\Factory::getDbInstance(Object(ByJG\Util\Uri), NULL) #3 /code/vendor/byjg/migration/src/Database/DblibDatabase.php(18): ByJG\AnyDataset\Db\Factory::getDbRelationalInstance('sqlsrv://sa:Mss...') #4 /code/vendor/byjg/migration/src/Migration.php(211): ByJG\DbMigration\Database\DblibDatabase::prepareEnvironment(Object(ByJG\Util\Uri)) #5 /code/vendor/byjg/migration/example/sql_server/test_sqlserver.php(16): ByJG\DbMigration\Migration->prepareEnvironment() #6 {main} thrown in /code/vendor/byjg/anydataset-db/src/PdoSqlsrv.php on line 28

Then I change localhost to sql1(container name) because both php and
sql server each in the different container but they're both linked.
I don't know why using localhost doesn't works inside docker, maybe you have clue ?
but this works
$uri = new \ByJG\Util\Uri('sqlsrv://sa:MssqlP455123@sql1/testdb');
Thanks a lot

Finally it works using this command
vendor/bin/migrate reset --path /Users/a/migration-tool/vendor/byjg/migration-cli/example/sql_server sqlsrv://sa:MssqlP455123@localhost:1433/testdb

2021-01-18 06:09:22.44 spid56      Starting up database 'testdb'.
2021-01-18 06:09:22.64 spid56      Parallel redo is started for database 'testdb' with worker pool size [3].
2021-01-18 06:09:22.65 spid56      Parallel redo is shutdown for database 'testdb' with worker pool size [3].

If using API, I have an error
Fatal error: Uncaught PDOException: SQLSTATE[HYT00]: [Microsoft][ODBC Driver 17 for SQL Server]Login timeout expired in /code/vendor/byjg/anydataset-db/src/PdoSqlsrv.php:28 Stack trace: #0 /code/vendor/byjg/anydataset-db/src/PdoSqlsrv.php(28): PDO->__construct('sqlsrv:server=0...', 'sa', 'MssqlP455123') #1 /code/vendor/byjg/anydataset-db/src/Factory.php(51): ByJG\AnyDataset\Db\PdoSqlsrv->__construct(Object(ByJG\Util\Uri)) #2 /code/vendor/byjg/anydataset-db/src/Factory.php(16): ByJG\AnyDataset\Db\Factory::getDbInstance(Object(ByJG\Util\Uri), NULL) #3 /code/vendor/byjg/migration/src/Database/DblibDatabase.php(18): ByJG\AnyDataset\Db\Factory::getDbRelationalInstance('sqlsrv://sa:Mss...') #4 /code/vendor/byjg/migration/src/Migration.php(211): ByJG\DbMigration\Database\DblibDatabase::prepareEnvironment(Object(ByJG\Util\Uri)) #5 /code/vendor/byjg/migration/example/sql_server/test_sqlserver.php(16): ByJG\DbMigration\Migration->prepareEnvironment() #6 {main} thrown in /code/vendor/byjg/anydataset-db/src/PdoSqlsrv.php on line 28

Then I change localhost to sql1(container name) because both php and
sql server each in the different container but they're both linked.
I don't know why using localhost doesn't works inside docker, maybe you have clue ?
but this works
$uri = new \ByJG\Util\Uri('sqlsrv://sa:MssqlP455123@sql1/testdb');
Thanks a lot

I am glad to know that the solution is working.
About why localhost is not working inside the docker container is something expected because of the containerization technology. Basically, when a container is runnning the containerization technology isolates the container from the host machine. It means the container cannot share network, folders, services with the host unless explicitly stated. When you run the container with the --port or --volume we are saying to map container port or volume with the host. Inside the container the localhost is the container itself, and it explain why it works when you changed from localhost to sql1.

I am glad to know that the solution is working.
About why localhost is not working inside the docker container is something expected because of the containerization technology. Basically, when a container is runnning the containerization technology isolates the container from the host machine. It means the container cannot share network, folders, services with the host unless explicitly stated. When you run the container with the --port or --volume we are saying to map container port or volume with the host. Inside the container the localhost is the container itself, and it explain why it works when you changed from localhost to sql1.


Can not reach the exposed services from another machine in the LAN

I have installed Kubernetes in a private virtual enviroment with vmware following this simple guide (docker multinode): https://kubernetes.io/docs/getting-started-guides/docker-multinode/
I have deployed kubernetes in two ubuntu16.04LTS machines on the same LAN, one master (<IP_ADDRESS>/24) and one worker (<IP_ADDRESS>/24).
root@kubemaster6:~# ./kube-deploy/docker-multinode/master.sh
+++ [0302 11:51:54] K8S_VERSION is set to: v1.5.3
+++ [0302 11:51:54] ETCD_VERSION is set to: 3.0.4
+++ [0302 11:51:54] FLANNEL_VERSION is set to: v0.6.1
+++ [0302 11:51:54] FLANNEL_IPMASQ is set to: true
+++ [0302 11:51:54] FLANNEL_NETWORK is set to: <IP_ADDRESS>/16
+++ [0302 11:51:54] FLANNEL_BACKEND is set to: udp
+++ [0302 11:51:54] RESTART_POLICY is set to: unless-stopped
+++ [0302 11:51:54] MASTER_IP is set to: localhost
+++ [0302 11:51:54] ARCH is set to: amd64
+++ [0302 11:51:54] IP_ADDRESS is set to: <IP_ADDRESS>
+++ [0302 11:51:54] USE_CNI is set to: false
+++ [0302 11:51:54] USE_CONTAINERIZED is set to: false
+++ [0302 11:51:54] --------------------------------------------
+++ [0302 11:51:54] Killing all kubernetes containers...
+++ [0302 11:51:54] Launching docker bootstrap...
+++ [0302 11:51:55] Launching etcd...
36c6e984c791c1207471d56e6f5d6c0b6cbbb032ef001942878c08e381833d1a
+++ [0302 11:51:59] Launching flannel...
{"action":"set","node":{"key":"/coreos.com/network/config","value":"{ \"Network\": \"<IP_ADDRESS>/16\", \"Backend\": {\"Type\": \"udp\"}}","modifiedIndex":4,"createdIndex":4}}
b347243a70cebe752fc77689401eec790fc8a6a44ed90d1296153ae44027c632
+++ [0302 11:52:00] FLANNEL_SUBNET is set to: <IP_ADDRESS>/24
+++ [0302 11:52:00] FLANNEL_MTU is set to: 1472
+++ [0302 11:52:00] Restarting main docker daemon...
+++ [0302 11:52:02] Restarted docker with the new flannel settings
+++ [0302 11:52:02] Launching Kubernetes master components...
7627853e95ace53612f8e03f2e8ebf61c6c74308ab0ee462e54e9d3631835013
+++ [0302 11:52:02] Done. It may take about a minute before apiserver is up.

root@minion7:~# export MASTER_IP=<IP_ADDRESS>
root@minion7:~# ./kube-deploy/docker-multinode/worker.sh
+++ [0302 11:53:54] K8S_VERSION is set to: v1.5.3
+++ [0302 11:53:54] ETCD_VERSION is set to: 3.0.4
+++ [0302 11:53:54] FLANNEL_VERSION is set to: v0.6.1
+++ [0302 11:53:54] FLANNEL_IPMASQ is set to: true
+++ [0302 11:53:54] FLANNEL_NETWORK is set to: <IP_ADDRESS>/16
+++ [0302 11:53:54] FLANNEL_BACKEND is set to: udp
+++ [0302 11:53:54] RESTART_POLICY is set to: unless-stopped
+++ [0302 11:53:54] MASTER_IP is set to: <IP_ADDRESS>
+++ [0302 11:53:54] ARCH is set to: amd64
+++ [0302 11:53:54] IP_ADDRESS is set to: <IP_ADDRESS>
+++ [0302 11:53:54] USE_CNI is set to: false
+++ [0302 11:53:54] USE_CONTAINERIZED is set to: false
+++ [0302 11:53:54] --------------------------------------------
+++ [0302 11:53:54] Killing all kubernetes containers...
+++ [0302 11:53:54] Launching docker bootstrap...
+++ [0302 11:53:55] Launching flannel...
8a5c412010fd4b6fd4e5510b2fc2a8092893deba5caa2e23c84046112512147f
+++ [0302 11:53:56] FLANNEL_SUBNET is set to: <IP_ADDRESS>/24
+++ [0302 11:53:56] FLANNEL_MTU is set to: 1472
+++ [0302 11:53:56] Restarting main docker daemon...
+++ [0302 11:53:58] Restarted docker with the new flannel settings
+++ [0302 11:53:58] Launching Kubernetes worker components...
e7e894858c311f1e9c01c1943d390474cf846ee0c42d1f5f856852e235abc03b
+++ [0302 11:53:59] Done. After about a minute the node should be ready.

Everything seems to be perfect:
root@kubemaster6:~# kubectl get nodes
NAME            STATUS    AGE
<IP_ADDRESS>   Ready     2m
<IP_ADDRESS>   Ready     47s

Finally, I deploy and expose an standard image of NGINX and works fine at the external TCP port 30177:
root@kubemaster6:~# kubectl create -f deploymentYaml/nginx.yaml
deployment "nginx-deployment" created
root@kubemaster6:~# kubectl expose deployment nginx-deployment --type=NodePort --name=example-service
service "example-service" exposed
root@kubemaster6:~# kubectl get services
NAME              CLUSTER-IP   EXTERNAL-IP   PORT(S)        AGE
example-service   <IP_ADDRESS>    <nodes>       80:30177/TCP   11s
kubernetes        <IP_ADDRESS>     <none>        443/TCP        5m
root@kubemaster6:~# kubectl get pods --all-namespaces --output=wide
NAMESPACE     NAME                                    READY     STATUS    RESTARTS   AGE       IP              NODE
default       nginx-deployment-148880595-2k316        1/1       Running   0          2m        <IP_ADDRESS>       <IP_ADDRESS>
kube-system   k8s-master-<IP_ADDRESS>                4/4       Running   14         18m       <IP_ADDRESS>   <IP_ADDRESS>
kube-system   k8s-proxy-v1-d5485                      1/1       Running   0          17m       <IP_ADDRESS>   <IP_ADDRESS>
kube-system   k8s-proxy-v1-zvs4x                      1/1       Running   0          19m       <IP_ADDRESS>   <IP_ADDRESS>
kube-system   kube-addon-manager-<IP_ADDRESS>        2/2       Running   0          18m       <IP_ADDRESS>   <IP_ADDRESS>
kube-system   kube-dns-4101612645-vdxk1               4/4       Running   0          19m       <IP_ADDRESS>       <IP_ADDRESS>
kube-system   kubernetes-dashboard-3543765157-xflvz   1/1       Running   0          19m       <IP_ADDRESS>       <IP_ADDRESS>

I can get the page perfectly from the master:
root@kubemaster6:~# curl localhost:30177
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...

But not from the other machine of the LAN:
root@minion7:~# curl <IP_ADDRESS>:30177
curl: (7) Failed to connect to <IP_ADDRESS> port 30177: Connection timed out

Am I doing something wrong?
The ports are UP and exposed to everyone:
root@kubemaster6:~# netstat -nlptu
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 <IP_ADDRESS>:10248         <IP_ADDRESS>:*               LISTEN      18585/hyperkube
tcp        0      0 <IP_ADDRESS>:10249         <IP_ADDRESS>:*               LISTEN      19695/hyperkube
tcp        0      0 <IP_ADDRESS>:22              <IP_ADDRESS>:*               LISTEN      1348/sshd
tcp6       0      0 :::10250                :::*                    LISTEN      18585/hyperkube
tcp6       0      0 :::6443                 :::*                    LISTEN      19309/hyperkube
tcp6       0      0 :::10251                :::*                    LISTEN      18887/hyperkube
tcp6       0      0 :::2379                 :::*                    LISTEN      18244/etcd
tcp6       0      0 :::10252                :::*                    LISTEN      18763/hyperkube
tcp6       0      0 :::2380                 :::*                    LISTEN      18244/etcd
tcp6       0      0 :::10255                :::*                    LISTEN      18585/hyperkube
tcp6       0      0 :::8080                 :::*                    LISTEN      19309/hyperkube
tcp6       0      0 :::22                   :::*                    LISTEN      1348/sshd
tcp6       0      0 :::30177                :::*                    LISTEN      19695/hyperkube
tcp6       0      0 :::4001                 :::*                    LISTEN      18244/etcd
tcp6       0      0 :::4194                 :::*                    LISTEN      18585/hyperkube
udp        0      0 <IP_ADDRESS>:8285      <IP_ADDRESS>:*                           18306/flanneld

But from the other machine inside the LAN, I don't see all of that open ports:
root@minion7:~# nmap <IP_ADDRESS>
Starting Nmap 7.01 ( https://nmap.org ) at 2017-03-02 12:16 CET
Nmap scan report for <IP_ADDRESS>
Host is up (0.000059s latency).
Not shown: 997 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
4001/tcp open  newoak
8080/tcp open  http-proxy
MAC Address: 00:0C:29:44:BD:1E (VMware)
Nmap done: 1 IP address (1 host up) scanned in 1.65 seconds

Am I reaching the correct IP?
Is correct that, in netstat appear ports on tcp6 (IPv6)?
Do I need to configure something in order to connect from outside the master machine?

Thank you in advance!

Finally, the solution was to allow forwarding on iptables:
iptables -P FORWARD ACCEPT
Thank you @xiangpengzhao & @liqlin2015 !


Detect soft-float ARM platforms

Libwidevinecdm.so on ARMv71 is using hard-float ABI (readelf -a libwidevinecdm.so | grep Flags). This cannot be loaded on a platform using softfp. Kodi fails with a cryptic error message.
2023-12-04 22:04:42.116 T:11544   error <general>: AddOnLog: inputstream.adaptive: Initialize: Initialize: Failed to load library: /media/developer/apps/usr/palm/applications/org.xbmc.
kodi/.kodi/cdm/libwidevinecdm.so: internal error
2023-12-04 22:04:42.116 T:11544   error <general>: AddOnLog: inputstream.adaptive: Unable to load widevine shared library (/media/developer/apps/usr/palm/applications/org.xbmc.kodi/.ko
di/cdm/libwidevinecdm.so)
2023-12-04 22:04:42.116 T:11544   error <general>: AddOnLog: inputstream.adaptive: OpenDRMSystem failed
2023-12-04 22:04:42.116 T:11544   error <general>: CVideoPlayer::OpenInputStream - error opening [plugin://plugin.video.vrt.nu/play/id/vid-6b245bab-1289-45a1-b395-bacf9a9eb70f/pbs-pub-
578fd4d4-79ee-4226-9a6f-b1b1c918a691/1640602595708]
2023-12-04 22:04:42.116 T:11544    info <general>: CVideoPlayer::OnExit()
2023-12-04 22:04:42.117 T:11544    info <general>: ADDON: Dll Destroyed - InputStream Adaptive

Could inputstreamhelper read the current executable and check the ABI?

That reminded me of #415. Really need to figure out what to do about those failing tests there...
Anyway, afaik there is no libwidevinecdm.so for arm soft float (please correct me if I'm wrong), so I guess we should detect that architecture as not supported.

There is no softfp or soft version of libwidevinecdm for arm, and there likely never will be. See https://www.chromium.org/chromium-os/how-tos-and-troubleshooting/chromiumos-board-porting-guide/
Technically, it is not the architecture but the current kodi float-abi that determines compatibility. So parsing the current executable (/proc/self) is probably the way to go, eg using libelf (ldd, readelf and others may not be available on embedded platforms). If I find time, I can test compiling a hard-float version of kodi for webos to see if that one is compatible with libwidevinecdm.


Make telopmode mode not block the robot

Tested and it works.
Question is do we want this or do we want a parameter

The default will stay the same, however if the 'force_stop_on_idle' is set:

we will not send 0 unless we have sent some other velocity before
also we will only publish once

Should we make the boolean has_published_zero_once a counter?


VST.msl.msl_direct

CertiGraph ought to be able to build in the standard Coq Platform installation of VST.  Right now, that build of VST does not include the file VST.msl.msl_direct.  That msl_direct is still in the VST repo, but it is not in the opam-build configuration.
For now I will try disabling such files as CertiGraph/msl_ext/overlapping_direct.v form the makefile.  But if msl/msl_direct is important to CertiGraph, then I suggest you file a VST issue report requesting that msl_direct be added to the make targets in VST/util/calc_install_files

msl_direct is not required in the default branch now.


Unable to find image '...' locally

I followed the install instructions but at the very last step I'm getting:
$ docker run -d --name="shib-local-test" 32625ac827d8.localdomain/shibboleth-idp
Unable to find image '32625ac827d8.localdomain/shibboleth-idp:latest' locally
docker: Error response from daemon: Get https://32625ac827d8.localdomain/v2/: Service Unavailable.
See 'docker run --help'.
What am I missing?

You are trying to pull an image from a registry hosted by 32625ac827d8.localdomain.  You either need to build the image locally using the same name or remove what Docker is interrupting to be a hostname from the image name.

Ugh.  I forgot the version at the end.  The full tag locally is '32625ac827d8.localdomain/shibboleth-idp:3.4.3'.
The container is now running.

Oops. That'll do it. Glad it is up an running.


Allow SVG construction on canvas in KV  file

Versions

Python: 3.7.1
OS: Mac
Kivy:1.10.1
Kivy installation method: pip

Description
The SVG example is as follows

class SvgWidget(Scatter):

    def __init__(self, svgfilename):
        super(SvgWidget, self).__init__()
        with self.canvas:
            svg = Svg(svgfilename)

        self.size = svg.width, svg.height

There does not seem to be an equivalent for applying the svg onto the canvas within the KV file.   The keyword constructor problem in #6101 might make it difficult.
Ideally .svg files would be allowed as the source to Rectangle so the following could be done
Builder.load_string('''
<SvgWidget@Scatter>:
    canvas:
        Rectangle:
            source: 'filename.svg'
            pos: self.pos
            size: self.size
''')

I will add my voice to this. I would like to see this feature added.


Security issue on noVNC

Hi, thanks for your great work and it is indeed awesome!
I would like to notice that exposing noVNC without any authentication is dangerous, even exposing the port to localhost manually. Since this is a docker environment, suppose other docker container has vulnerability and an attacker gain access to it, the attacker can easily take over the windows docker by simply accessing the noVNC port.
Possible solution:
Optional configuration: Specify username and password in the docker yml file or from a file in the shared folder. When container starts, edit the Nginx configuration to add authentication.

This isnt an issue but I see there is a option to disable KVM with "KVM=N".
Once the VM is set up, you could disable KVM and just use RDP thus negating the VNC issue.
I cant get this working though.
Is this reccomended & how would I acheive this?

Thanks, yes sorry for my ignorance. That helps a lot 😊

@ajb3932 I don't see how KVM has anything to do with it? That is just a method for hardware accelerating CPU instructions, if you disable it it has no influence on VNC at all (the only difference you will notice is that the machine runs slower).
If you want to disable VNC you can set the flag 'DISPLAY=none` in the compose file. But setting a Windows user password should already be sufficient to not have to do that.

where exactly need to add that flag? in environment: ?? because I tried that and it's breaking docker container, container keeps restarting.


handle multiple hashpack wallets

A users can connect multiple hashpack wallets, though in our code, we're just pulling the first one. Let's disconnect all accountId's except 1 at a time.

users can select just 1 wallet when connecting


Layer-3 routed networking support in Cattle

It‘s very similar to this. https://coreos.com/flannel/docs/latest/aws-vpc-backend.html

For network-manager:

Compatible with masqrade rules to ensure that the container can access the external network.
Generate CNI configuration for host-subnet

For CNI library:
Continue to use the rancher-cni-bridge, but bridgeSubnet needs to be set according to host subnet.
For IPAM:
No need to use rancher-ipam, host-local-ipam is required in this case.
https://github.com/niusmallnan/rancher-host-local-ipam
For scheduler:
Change the scheduler to make sure containers with "retain_ip: true" always get restarted on the same host.
But can not get retainer_ip information in the scheduler. Tried some way and found that it was not too reasonable. I think that the best way is to make some changes in cattle.
When triggering the event scheduler.prioritize, scheduler can get the context data. If there is a retain_ip flag, it is best to include the IP information in the context data. The container that was originally created does not need to contain it, and the upgraded container needs to contain it. Then I can use IP information to filter on which host the container should be scheduled for.
How to use：

Add a catalog repo：https://github.com/niusmallnan/host-subnet-catalog
Create a new environment with this networking
Add host with label "io.rancher.network.per_host_subnet.subnet=xx.xx.xx.xx/xx"
Set the CIDR of the host label settings to the VPC routing table, note the settings for the security group

@niusmallnan can you please update PR:79 to get this issue resolved?

@leodotcloud
I have updated, please recheck.

Catalog updated at: https://github.com/rancher/rancher-catalog/pull/940

Tested with rancher-server version- v1.6-development with rancher/net:v0.13.0
Type/provider of hosts: AWS
Problem

Requested IP is not supported.

Retain IP is not supported.

Readme write error

Load Balancer failed to start

level=fatal msg="Error reading self service metadata: Error 404 accessing /self/service path

Remove per-host-network container, host routing information is not cleared.

@kingsd041 Minor suggestion about the process, I see the QA team here files new issues during testing and point them to the original issue instead of reopening. [There is nothing wrong with your approach, it's just what everyone is used to here]

@kingsd041 It depends on you. If you think it's easier for tracking some major issues separately, you can open a new issue for it and link the current issue. You also can keep the small issues in the same issue.

@kingsd041 is there anything else we're waiting for except for #10074 ? I saw you pointed out a typo, have we fixed that yet?

@yasker  Typo and https://github.com/rancher/rancher/issues/10074 have been fixed.
But I can not close the issue Because https://github.com/rancher/rancher/issues/9978 hasn't been tested yet

@yasker  Sorry, I just clicked the wrong button.

Issue https://github.com/rancher/rancher/issues/9978 has been tested.
Close the issue.


Crashing when adding to transform stream

I'm no stream expert, but given the examples, it seemed like it would be super easy to create a transform stream with your module.
I am able to verify that it works fine when running like the example in the readme, but when I port it into a custom Transform stream, it ends super early, doesn't hit the writeable, and I get no errors. There's a good chance this isn't a problem with uDSV, but it's strange, given your apis, this should work.

// lines = 16_182_971 , 100% finished in 13.1 seconds = 1_235_341 rows per second
// 597.5 MB  13.1s   45.5MB/s
const streamedCSV = async () => {
  const fileSize = await fs.promises.stat(filePath).then((stats) => stats.size)
  const stream = fs.createReadStream(filePath).pipe(new ProgressBarStream(fileSize))

  let parser: Parser;
  for await (const buffer of stream) {
    const strChunk = buffer.toString();
    parser ??= initParser(inferSchema(strChunk));
    parser.chunk<string[]>(strChunk, parser.typedArrs, (batch) => {
      const x = batch // reaches here fine
    });
  }
  parser!.end();
}

const transform = async () => {
  const fileSize = await fs.promises.stat(filePath).then((stats) => stats.size)
  return new Promise<void>((resolve, reject) => {
    pipeline(
      fs.createReadStream(filePath),
      new SimpleUDSVTransform(),
      new Writable({
        write(chunk, encoding, callback) {
          const x = chunk // never reaches here
          callback()
        }
      }),
      (err) => {
        if (err) {
          reject(err) // never reaches here
        } else {
          resolve() // never reaches here
        }
      }
    )
  })
}
(async () => {
  await streamedCSV() //works
  await transform() //breaks
})()

I'm no stream expert

heh, me neither :sweat_smile:. uDSV's api is fully synchronous, so there shouldn't be any surprises there.
does this reproduce with something you can attach here and i can debug (that hopefully isnt 16M records)?

yeah for sure, I just created a example repo you can use to run everything. just run example.ts
https://github.com/1mike12/usdv-stream-example
But regarding the api, I think the way to get it to work with streams is to use the chunk function, which seems to be async right now. I imagine it should be able to be synchronous though too though, since there shouldn't be a reason for the parser to not be able to give us what it's already got so far in a sync way?
this.parser.chunk<string[]>(strChunk, this.parser.stringArrs, (parsedData) => {
      this.push(parsedData);
      callback()
    });

But either way, whether its sync or async, it really doesn't matter to a Transform stream, which is why I'm so confused why it just stops and ends so soon

just heads up, it might be week or so before i can get to testing this.


just wanted to share python rewrite of this script

reason for it is that python version doesnt need any dependencies whereas it was a pain to custom install node.js version plus all the dependencies. hopefully someone finds it useful
https://github.com/specu/ASFclaim.py

Nice! I added it to the readme


[WIP] remove old pip config change entries

Description
Remove the changelog entries from our initial attempt at a centralized pip config.
For reference, this is the changelog entry centralizing the changes for the latest attempt.
closes https://github.com/inmanta/inmanta-service-orchestrator/issues/445
Self Check:
Strike through any lines that are not applicable (~~line~~) then check the box

[ ] Attached issue to pull request
[ ] Changelog entry
[ ] Type annotations are present
[ ] Code is clear and sufficiently documented
[ ] No (preventable) type errors (check using make mypy or make mypy-diff)
[ ] Sufficient test cases (reproduces the bug/tests the requested feature)
[ ] Correct, in line with design
[ ] End user documentation is included or an issue is created for end-user documentation (add ref to issue here: )
[ ] If this PR fixes a race condition in the test suite, also push the fix to the relevant stable branche(s) (see test-fixes for more info)

Processing this pull request

Merged into branches master in 0f5d3ff9ff0f8f7d974707bd78c2989a91319dec


Bug with newlines?

MultiMarkdown v.5.1.0 and 1.md:
Installation
------------
Installing Geany is done by the following three commands:
$ ./configure
$ make
(as root)
% make install

Run multimarkdown.exe --batch --full 1.md and 1.html file contents:
<!DOCTYPE html>

	<meta charset="utf-8"/>

Installation

Installing Geany is done by the following three commands:
$ ./configure
$ make
(as root)
% make install

I tried CR, LF and CR+LF, but all text in one line

. Is bug or I am doing something wrong?

  Not a bug, just lack of understanding of Markdown syntax.  ;)
https://daringfireball.net/projects/markdown/syntax
Paragraphs are separated by '\n\n'.  A linebreak can be forced with '\n'.
But what you're showing looks more like a code block and should be entered as such (either the standard Markdown indented code block, or a fenced MMD codeblock:  http://fletcher.github.io/MultiMarkdown-5/fences.html).
MMD also supports escaped newlines:  http://fletcher.github.io/MultiMarkdown-5/escaped-newlines.html


doctest for .rst files

Before submitting

[ ] Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
[x] Did you read the contributor guideline, Pull Request section?
[x] Did you make sure to update the docs?
[x] Did you write any new necessary tests?
[ ] If you made a notable change (that affects users), did you update the CHANGELOG?

What does this PR do?
Follow up to #1484 in a series of doc  improvements.
This PR adds doctesting for .rst files and converts existing code blocks to proper doctests.
Note:
Notes:

For the tests to work I had to change imports like pl.LightningModule to because otherwise there was an import error because in docs the __LIGHTNING_SETUP__ variable is set.
Even the doctests that don't test for outputs (like class defs) are useful because it still checks syntax. In fact, while converting them I found syntax errors that would otherwise not have been detected. So if someone copy pastes from docs, it is less likely they need to fix a typo we introduced.

PR review
Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.
Did you have fun?
Make sure you had fun coding 🙃

pls add also testing RST files with pytest along with the existing testing because now it is tested only against one package's configuration but we would like to cover at least min and max package version coverage >> add it to Github action CI

this is just a draft and I'm experimenting. I don't know how to do all these things but  I will figure it out.
P.S. you deleted all my content in the PR description. can it be undone? I have to type it again :(

P.S. you deleted all my content in the PR description. can it be undone? I have to type it again :(

I didn't, you can see the version history as next to the content title is edited by ...

@Borda The doctestplus you suggested seems not to work very well with sphinx. It is a library to run doctests on standalone rst files with pytest. However, since we are using rst files with sphinx, doctestplus is not optional. First of all, sphinx doctests support things like this:
.. doctest::
    :skipif: torch.cuda.device_count() < 2
    >>> Trainer(gpus=2)

But this is not supported by doctestplus and it would be nice to run our doctests conditionally.
So, I suggest to use simply the sphinx built in doctest.
What do you think?

the point is that the Read-The-docs is running the build on quite a simple machine so we do not want to take it too long... let's try it with sphinx only and see how it goes :]

The doctestplus you suggested seems not to work very well with sphinx

what does it mean, some tests are skipped or failing?

No it means it is not designed to be used together with sphinx. It will ignore directives like
.. doctest::
    :skipif: torch.cuda.device_count() < 2
    >>> Trainer(gpus=2)

Sphinx has its own doctest extension.

@awaelchli @Borda shall we merge soon? unless @awaelchli wants to keep rebasing lol... this will get hard to merge the longer it's open :)

@williamFalcon sure, I'll try to finish it today, thanks.

@Borda in the doctests I import loggers, should I add requirements-extra to the docs build or what should I do?

@Borda in the doctests I import loggers, should I add requirements-extra to the docs build or what should I do?

in this case, I think yeas you really need to call some methods... the mock here is not enough, right?

hmm, not sure how I could mock them.
It seems not possible to just simply install requirements-extra because of horovod.
Should I remove doctesting for loggers completely?

hmm, not sure how I could mock them.

I think it is not possible if we want to run the tests...

It seems not possible to just simply install requirements-extra because of horovod.

what is the problem with Horovod?

It seems CI can't install it if I do pip install -r requirements-extra.txt

@tgaddair Hi do you know how I can install or ignore horovod in the docs build CI? It is CPU only and I need to install the logger requiremetns in requirements-extra.txt but it also contains horovod.

@tgaddair Hi do you know how I can install or ignore horovod in the docs build CI? It is CPU only and I need to install the logger requiremetns in requirements-extra.txt but it also contains horovod.

Hey @awaelchli, Horovod should be able to install with or without GPU.  I can take a quick look at the build output to see why it's failing.

Thanks, here it is:
https://circleci.com/gh/PyTorchLightning/pytorch-lightning/25540?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link
Ignore the other failures, they are not due to horovod.

Thanks, here it is:
https://circleci.com/gh/PyTorchLightning/pytorch-lightning/25540?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link
Ignore the other failures, they are not due to horovod.

Thanks.  From the logs, it seems the problem is that neither MPI nor CMake are installed.  One of those most be installed for Horovod to build.  I suggest installing CMake as we do in install_deps:
sudo apt-get update && sudo apt-get install -y cmake

That should take care of the issue.

@tgaddair Thanks for the help, it worked.
@Borda the drone has a red cross but when I look at the output, all tests are passing, no mention of a failing test. could you have a quick look and check if you see something obvious I don't see?

Hello @awaelchli! Thanks for updating this PR.

In the file docs/source/conf.py:

Line 408:37: W292 no newline at end of file

Can't invest more time into trying to make the doctests run on drone. I have have tried anything, there is just no output that would suggest any failures. So I suggest to drop it for drone and just keep it in the circle ci.
PR is now ready for review

Can't invest more time into trying to make the doctests run on drone. I have tried everything, there is just no output that would suggest any failures. So I suggest to drop it for drone and just keep it in the circle ci.

I am fine with having it now at least for CPU (CircleCI) and GPU we can add later...

Looks like ReadTheDocs did not like what we modified in the conf.py. Some pages like Trainer and LightningModule are broken. I suspect it's the mocking of packages. Will track this down and submit a follow up PR.

@awaelchli could pls prepare fix as main repo branch and I will turn on building this branch so we will constantly know how it is going...


Can't access cn models list

When I go here:
http://localhost:7860/docs#/default/model_list_controlnet_model_list_get
I get an error like this;
(sorry if this is too newbie)

Try update A1111 and controlnet extension to latest, and start A1111 in API mode again.


re-opening #2871

I got the same issue with @jimiguru I'm not really sure why the reset token saved in the db didn't match when users updating their new password. can anybody help us with this? thanks!

Thank you @rubyrider. We still can't reproduce this issue to, so we are closing it for now!


change event is not triggered

Describe the bug
When I delete the value entirely at the first load of editor, the change event is not triggered.
version: 2.1.4
step

select all code text in editor use command + A
push "delete" button

In addition, the event is not triggered, so the connected model value is not changed.
Validations

[X] Read the docs.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.

Sorry, I can't come up with the problem you said.
Please update to the latest version to try again or provide a minimal reproducible example

I checked that the problem disappeared after upgrading to the latest version.
Thank you for your help! 👍


generate Routes by default for expose: true Services?

To avoid folks having to run exposecontroller for their apps on OpenShift it'd be nice if we enabled the creation of a Route by default in the openshift.yml file.
Folks could maybe disable this with a global setting in their ~/.m2/settings.xml or via a CLI property etc

actually this is a dupe ;) https://github.com/fabric8io/fabric8-maven-plugin/issues/713


Small grammar updates

@andresnava : Thanks for your contribution! The author(s) have been notified to review your proposed change.

Looks good. @andresnava Thanks for submitting the fixes.


Remove hard ArgoCD dependency

Checks

[X] I have searched the existing issues.
[X] This request is NOT specific to the CLI. (If so, please open an issue on the CLI repo)

Motivation
ArgoCD is a fine tool deploy stuff to a Kubernetes cluster, but not everybody want to use it (they could be using a different GitOps engine such as Flux, Fleet, etc., or not wanting to use GitOps).
It should be possible to just generate raw Kubernetes manifests as output to a directory, without the ArgoCD dependency.
I think it is fine to keep a guide for using ArgoCD with deployKF, and perhaps keep it as a recommendation, but we should not do what KubeFlow has done and add arbitrary dependencies to their platform which aren't actually needed for a lot of people (in my mind, their implementation of Istio is this is the biggest contributor to KubeFlow being hard to install and manage)
Implementation
No response
Are you willing & able to help?

[ ] I am able to submit a PR!
[ ] I can help test the feature!

@AndersBennedsgaard While I agree that it would be nice to use deployKF without ArgoCD, there are a few things that deployKF uses ArgoCD for, which can't be solved when using a raw manifests directory:

Cleaning up resources:

If you just manually apply Kustomize manifests to a Kubernetes cluster, there is no easy way to remove them again once they are no longer needed.
Obviously, this is less important for Helm apps, but deployKF includes both Helm and Kustomize apps (notably Kubeflow, lol).

Syncing in a controlled way:

We have a reasonably complex sync process which uses ArgoCD to ensure that each app is properly synced before moving on to the next.
Within many apps, we make use of the argocd.argoproj.io/sync-wave feature, to prevent race conditions (e.g. to ensure cert-manager is synced and healthy before syncing ClusterIssuer resources)
Many of our apps use helm.sh/hook: pre-install,pre-upgrade hooks (e.g. to generate unique OIDC secrets) these take a non-trivial amount of time. Using ArgoCD allows us to only run them when their app is "out of sync" which we could not possibly know without ArgoCD.

Given the complexity of achieving all this natively in the deploykf CLI, and the significantly lower popularity of FluxCD, it's just really not a priority at the moment (and I am sure you would prefer I spend my time adding user-facing features!)
Do you have a specific thing which is driving your desire to not use ArgoCD?

PS: from my discussions with large companies all around the world, pretty much everyone who is using Kubernetes seriously is using ArgoCD. For this and many other reasons, ArgoCD is how I recommend anyone deploy something as complex as deployKF. Yes, it has its problems, but for better or worse, it is the standard.

Actually, no, I wouldn't prefer that you spend time on user-facing features. I would prefer that the focus is on the stability and availability of use. But you probably prefer to focus on user-facing features as it is just more fun to work on :wink:
But I have to disagree with several things you write.

If you just manually apply Kustomize manifests to a Kubernetes cluster, there is no easy way to remove them again once they are no longer needed.

While I agree that a good way of removing stuff should be possible, is is very easy to just write kubectl delete -f /path/to/stuff with a potential -R for recursively deleting stuff. I am not advocating for using raw kubectl commands (even automated commands using Jenkins or similar); I am just saying that without the raw manifests, you remove a lot of potential users from using your product (which is fine if that is the decision).

We have a reasonably complex sync process which uses ArgoCD to ensure that each app is properly synced before moving on to the next.

As I read it, this script is making sure that all the manifests are correctly applied to the cluster, in the correct order, and verifying that some resources has been pruned?
If that is the case: that is basically a solved problem in Flux. Here you just commit stuff to git, and specify which "folders"/"paths" are depending on each other. Then Flux will try to apply the contents of one path before moving on to the next.
I would be surprised if ArgoCD and other similar tools doesn't have this feature built in?

Within many apps, we make use of the argocd.argoproj.io/sync-wave feature, to prevent race conditions (e.g. to ensure cert-manager is synced and healthy before syncing ClusterIssuer resources)

Same as above

Many of our apps use helm.sh/hook: pre-install,pre-upgrade hooks (e.g. to generate unique OIDC secrets) these take a non-trivial amount of time. Using ArgoCD allows us to only run them when their app is "out of sync" which we could not possibly know without ArgoCD.

This is just a Helm feature, so ArgoCD isn't really needed for this?

Do you have a specific thing which is driving your desire to not use ArgoCD?

Yes. I've used ArgoCD, and I am not very impressed. It has a very nice developer-friendly UI, and if you don't do pure GitOps then it's a very nice alternative to kubectl. But as I am an infrastructure guy for the most part, I require stability when working with mutating webhooks, CRDs, dependencies to other stuff, and so on, which ArgoCD does not handle well. As I also want to use GitOps, I do not want to have state in both ArgoCDs database, git, and Kubernetes at the same time. I only want it to be in git, which ArgoCD can do with read-only mode, but it is janky at best.
All of this is solved by Flux, and other GitOps engines.

PS: from my discussions with large companies all around the world, pretty much everyone who is using Kubernetes seriously is using ArgoCD. For this and many other reasons, ArgoCD is how I recommend anyone deploy something as complex as deployKF. Yes, it has its problems, but for better or worse, it is the standard.

While it is probably true that ArgoCD is the most used engine (I am not entirely sure that it is, since a lot of people use them in conjunction), I've worked with a lot of different companies in my role as consultant, and I've only seen the use of ArgoCD once - when I wanted to test it on one of our demo setups.
It might be that I am working in a more security-heavy field where we require an audit-trail for all changes (mostly in the form of the git history), or something else, but as I see it (and mentioned above), choosing ArgoCD as the only way to support your tool will not make it usable for a LOT of people.
Which might be fine. In that case, feel free to close this issue.

@AndersBennedsgaard my comment in https://github.com/deployKF/deployKF/issues/110#issuecomment-1968098537 was primarily about why raw manifests wouldn't work. However, it's always been a roadmap item to support FluxCD (given it's the only other widely used K8S CD tool).
If you or your customers would benefit from FluxCD support coming sooner, I am very happy to help facilitate such a contribution from you or your team (or others watching this thread).

I am not actually arguing for full-blown Flux support, but rather removal of the hard ArgoCD dependency. It's fair to have support for ArgoCD, Flux, whatever other system using plugins or some other ways, but there are times where raw manifests are just preferred (such as when using Flux).
Unfortunately, I don't have the most time in the world right now, but if I do find some I would love to take a stab at it

Is a Kubernetes Operator an option?

I would like to second this request.
I understand that this is a very complex deployment but focusing on Kubernetes native solutions to the challenges makes your product more accessible. The dependency on ArgoCD and Istio also creates more points of failure and maintenance.
Do the benefits really outweigh the cost for every team and every use case? What good are great features on a product nobody wants to go through the struggle of deploying?
There is elegance in simplicity.

@thesuperzapper @AndersBennedsgaard @mnasty
I'd echo the sentiments here about ArgoCD. I'm not here to argue the utility or widespread use of ArgoCD, most organisations I would expect have a centralised management of Kubernetes infrastructure. I currently work for an organisation where medium-sized teams are super-siloed, and therefore the scale of Kubernetes clusters is much smaller and managed by each team in their own cloud environments independently - none of them really yet have the scale to leverage the benefits of ArgoCD.
Our context is that we're operating a data lake on AWS. Our Kubernetes cluster has two primary functions;

We run Airflow to execute schedule DAGs (Primary Apache Spark), which are either managed by our team, or managed by customers, which are other groups in our organisation.  Airflow is still currently the best tool for us, even though Argo Workflows shows some promise, it has been somewhat lacking in official multi-tenancy support. Added to this, we have a very deep collection of datasets implemented in Airflow and porting them to a new workflow tool would be laboursome.

We use Kubeflow, primarily for the Notebook servers, where customers are split into namespaces/Kubeflow Workspaces and assume their roles via IRSA in EKS. From there, they start Apache Spark Kubernetes clusters,  slurp data from Glue/Athena tables and make data babies.

Our current EKS cluster and Airflow is deployed using Gitlab CI and Terraform. For better or worse, this suits us fine. Kubeflow on the other hand has historical been a mess of a kustomized helm chart and ansible - and I'm looking to ideally port this over to a Terraform based deployment.
Terraform is attractive in the sense it allows us to bundle all our AWS resources and our Kubernetes deployment into a single Terraform project. It's a Hashicorp shaped hammer I'm running around with.
Because of our rather simple configuration, introducing ArgoCD as another deployment we have to manage that is only performing a single function for us - deploying deployKF, is additional overhead that seems hard to justify despite the advantages presented.
In terms of the stateful application of manifests; Terraform is capable of managing the applying and deletion of manifests using the kubectl_manifests resource. If a manifest is deleted from the source repository, Terraform can delete that manifest from the cluster, which solves point 1.
The challenge is getting the correct order of these things applied, and as high-lighted; ensuring we are waiting on components being online before continuing - not so easy with Terraform, but potentially possible. I would be trading the ArgoCD automation for handling this logic myself which will create overhead in the long run having to adapt it to the changes of ArgoCD deployment.
I do see tremendous value in using deployKF to manage the generation of our deployment using the single YAML configuration as well as being cloud agnostic as Azure is on the horizon for us. If I could the generation in a GitlabCI build phase, and then create the appropriate Terraform code to deploy the manifests - that for us would be ideal. If I were to understand the order that the app of apps operates in, then implementing the logic of the deployment wouldn't be all that difficult.
Where this strategy falls short is in the syncing aspect (as with the OIDC secrets mentioned), and this is something I'd need to understand better as it would be the only blocker in my mind.
Anyway, that's my two cents - I guess my lack of understanding on the "syncing" problem is likely where I'm coming undone. The other problem I have, is I'm on a bit of a ticking timebomb with our Kubeflow deployment which is ancient and failing on newer version of EKS. I'm not enamoured by how abandoned/sporadic the AWS Kubeflow project seems - so having to also learn ArgoCD is going to push out our horizon on migration of/to Kubeflow/deployDF, despite the fact it would save us time when considering other deployment methods.

@berglh thanks for the interesting write up.
It sounds like you will really benefit from deployKF, it will make your life a lot easier in maintaining Kubeflow. My recommendation (if you want to use deployKF in the short term), is to just treat the ArgoCD as "part of the platform" even if you never interact with it outside of the "argocd sync script" that we provide.
For example, you can trigger the sync script from a Terraform (or other CI/CD job), just pretend it's some external vendor's system you are triggering a "rollout" on.
PS: there is lots of interesting stuff coming around Apache Airflow + deployKF, it's a little way off, but will be worth the wait!
PSS: if you currently deploy Airflow on Kubernetes, there is a good chance you are already using our helm chart to do it, and we think a lot of users will benefit from managing their "Airflow on Kubernetes" deployments under their deployKF configs and auth.

Hmm, no comment on K8s Operators? 🤔
https://github.com/deployKF/deployKF/issues/110#issuecomment-2000457452

It sounds like you will really benefit from deployKF, it will make your life a lot easier in maintaining Kubeflow. My recommendation (if you want to use deployKF in the short term), is to just treat the ArgoCD as "part of the platform" even if you never interact with it outside of the "argocd sync script" that we provide.

The more I'm thinking about deployKF and the alternatives, this does make sense.

PSS: if you currently deploy Airflow on Kubernetes, there is a good chance you are already using our helm chart to do it, and we think a lot of users will benefit from managing their "Airflow on Kubernetes" deployments under their deployKF configs and auth.

I fell into the Astronomer fork via the Airflow docs. I've already banged my head on a few issues, such as not being able to sync secret updates via the Helm chart values (need to manually manage these); no Celery support for Elasticache Redis clusters is really awesome - not Airflow's fault, but there is some interesting options that Celery does support as a backend that are not currently available in Airflow such as SQS. Rather annoying when I'm shooting for HA using affordable AWS services.
Having deployKF manage this deployment as well would greatly simplify the stack, there is also an opportunity to wrap a single SAML service provider for every major component of our data lake when this comes to fruition, although, the inability of Kubeflow to handle workspace membership via SAML group attributes is a little sad. This issue has created significant administrative overhead for my team having to manage a separate user role mappings repository historically, which would be mostly distilled down to a group mappings repository with delegated AD group management to workspace owners/mangers if this capability existed.

PSSS: the "Kubeflow on AWS" distribution is not maintained anymore.

@thesuperzapper I will say, your experience and high level of contributions in the Airflow and Kubeflow space is certainly reassuring in terms of the strategic direction of this project. One thing that concerns me is that almost all, if not every commit in this repo is made by you. I'm not insinuating that you wouldn't accept PRs that provide merit to the project, but from a continuity perspective - that is one aspect of this project that gives me pause for concern, despite you offering paid support.
I apologise this is starting to get a little off-topic, and I'm certainly not trying to make you convince me that this is the way. I understand the motivation behind this project given the lack of other suitable options.
I need to reconcile the risks of each option before I settle on a path forward as in my current multi-faceted role I have many mediocre-at-best hats I need to wear as Systems Programmer, Data/ML Engineer, Data Analyst, Data Domain Expert, Apache Spark SME/Trainer, On-prem Elastic Cloud Enterprise SME & Operator, Cloud Infrastructure Operations etc - I'm spread pretty thin and don't have the luxury of time to experiment and iterate frequently on the data lake stack.

@thesuperzapper I will say, your experience and high level of contributions in the Airflow and Kubeflow space is certainly reassuring in terms of the strategic direction of this project. One thing that concerns me is that almost all, if not every commit in this repo is made by you. I'm not insinuating that you wouldn't accept PRs that provide merit to the project, but from a continuity perspective - that is one aspect of this project that gives me pause for concern, despite you offering paid support.

@berglh Your concerns are valid, and we are working to increase the number of contributors to deployKF so that it becomes a sustainable industry standard. By using and contributing to the project you will help this become a reality.
Also, in the near future we will allow sponsoring the project directly, and in exchange those sponsors will get special features, extra tools, and influence in what we develop. With these contributions, we plan to hire a number of full-time engineers that will ensure the continuity and velocity of the project.
To clarify, deployKF is and always will be an open-source project, no payment/sponsorship will be needed to use deployKF to deploy and manage Kubeflow.


Gekko UI live trading missing trade entry (Binance)

Note: this is the technical bug tracker, please use other platforms for getting support and starting a (non technical) discussion. See the getting help page for details.
I'm submitting a ...
[/ ] bug report
[ ] question about the decisions made in the repository
Action taken (what you did)
Only 1 live trading (+ market watch) instance with Gekko UI.
Expected result (what you hoped would happen)
Would logs all trades to Gekko UI.
Actual result (unexpected outcome)
One trade is missing. (Gekko has initiated 3 trade orders, all which are successful. Gekko UI only showing 2 trades)
Other information (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, links for us to have context, eg. stackoverflow, etc)
Screenshot of Gekko UI:

Screenshot of Trade Order (Binance):

My timezone is UTC+8.
Gekko tradelogs:
2019-03-24 08:22:32 (INFO):     Setting up:
2019-03-24 08:22:32 (INFO):              Trader
2019-03-24 08:22:32 (INFO):              Follows the advice and create real orders.
2019-03-24 08:22:34 (DEBUG):    syncing private data
2019-03-24 08:22:34 (INFO):
2019-03-24 08:22:40 (INFO):              Portfolio:
2019-03-24 08:22:40 (INFO):                      0.32721451 USDT
2019-03-24 08:22:40 (INFO):                      10.1 ETH
2019-03-24 08:22:40 (INFO):              Balance:
2019-03-24 08:22:40 (INFO):                      1376.1492145099999 USDT
2019-03-24 08:22:40 (INFO):              Exposed:
2019-03-24 08:22:40 (INFO):                      yes (99.98%)
2019-03-24 08:22:40 (INFO):     Setting up:
2019-03-24 08:22:40 (INFO):              Performance Analyzer
2019-03-24 08:22:40 (INFO):              Analyzes performances of trades
2019-03-24 08:22:40 (INFO):
2019-03-24 08:22:40 (INFO):     Setting up:
2019-03-24 08:22:40 (INFO):              Child to parent
2019-03-24 08:22:40 (INFO):              Relays events from the child to the parent process
2019-03-24 08:22:40 (INFO):

      <truncated>

2019-03-24 09:37:24 (INFO):     Trader Received advice to go short. Selling  ETH
2019-03-24 09:37:24 (DEBUG):    Creating order to sell 10.1 ETH
2019-03-24 09:37:29 (DEBUG):    [ORDER] statusChange: SUBMITTED
2019-03-24 09:37:29 (DEBUG):    [ORDER] statusChange: OPEN
2019-03-24 09:37:31 (INFO):     [ORDER] partial sell fill, total filled: 10.09929
2019-03-24 09:37:34 (DEBUG):    [ORDER] statusChange: MOVING
2019-03-24 09:37:35 (DEBUG):    [ORDER] statusChange: ERROR
2019-03-24 09:37:35 (ERROR):    [ORDER] Gekko received error from GB: Error -2011: Unknown order sent.
2019-03-24 09:37:35 (DEBUG):    Error: Error -2011: Unknown order sent.
    at /home/chew1992/gekko/exchange/wrappers/binance.js:101:15
    at Request.request [as _callback] (/home/chew1992/gekko/exchange/node_modules/binance/lib/rest.js:90:25)
    at Request.self.callback (/home/chew1992/gekko/exchange/node_modules/request/request.js:185:22)
    at Request.emit (events.js:197:13)
    at Request.<anonymous> (/home/chew1992/gekko/exchange/node_modules/request/request.js:1157:10)
    at Request.emit (events.js:197:13)
    at IncomingMessage.<anonymous> (/home/chew1992/gekko/exchange/node_modules/request/request.js:1079:12)
    at Object.onceWrapper (events.js:285:20)
    at IncomingMessage.emit (events.js:202:15)
    at endReadableNT (_stream_readable.js:1132:12)
    at processTicksAndRejections (internal/process/next_tick.js:76:17)

      <truncated>

2019-03-24 11:22:29 (INFO):     Trader Received advice to go long. Buying  ETH
2019-03-24 11:22:29 (DEBUG):    Creating order to buy 9.616928844717975 ETH
2019-03-24 11:22:34 (DEBUG):    [ORDER] statusChange: SUBMITTED
2019-03-24 11:22:34 (DEBUG):    [ORDER] statusChange: OPEN
2019-03-24 11:22:35 (DEBUG):    syncing private data
2019-03-24 11:22:38 (INFO):     [ORDER] partial buy fill, total filled: 9.61692
2019-03-24 11:22:38 (DEBUG):    [ORDER] statusChange: FILLED
2019-03-24 11:22:39 (INFO):     [ORDER] summary: { price: 136.63,
  amount: 9.61692,
  date:
   Moment {
     _isAMomentObject: true,
     _i:<PHONE_NUMBER>433,
     _isUTC: false,
     _pf:
      { empty: false,
        unusedTokens: [],
        unusedInput: [],
        overflow: -2,
        charsLeftOver: 0,
        nullInput: false,
        invalidMonth: null,
        invalidFormat: false,
        userInvalidated: false,
        iso: false,
        parsedDateParts: [],
        meridiem: null,
        rfc2822: false,
        weekdayMismatch: false },
     _locale:
      Locale {
        _calendar: [Object],
        _longDateFormat: [Object],
        _invalidDate: 'Invalid date',
        ordinal: [Function: ordinal],
        _dayOfMonthOrdinalParse: /\d{1,2}(th|st|nd|rd)/,
        _relativeTime: [Object],
        _months: [Array],
        _monthsShort: [Array],
        _week: [Object],
        _weekdays: [Array],
        _weekdaysMin: [Array],
        _weekdaysShort: [Array],
        _meridiemParse: /[ap]\.?m?\.?/i,
        _abbr: 'en',
        _config: [Object],
        _dayOfMonthOrdinalParseLenient: /\d{1,2}(th|st|nd|rd)|\d{1,2}/ },
     _d: 2019-03-24T11:22:36.433Z,
     _isValid: true },
  side: 'buy',
  orders: 1,
  fees: { BNB: 0.05771097 },
  feePercent: 0.00075 }
2019-03-24 11:22:39 (DEBUG):    syncing private data
2019-03-24 11:23:09 (INFO):     (PROFIT REPORT) original balance:                1374.63421451 USDT
2019-03-24 11:23:09 (INFO):     (PROFIT REPORT) current balance:                 1384.30122971 USDT
2019-03-24 11:23:09 (INFO):     (PROFIT REPORT) profit:                          9.66701520 USDT (0.70324273%)

     <truncated>

2019-03-24 22:07:20 (INFO):     Trader Received advice to go short. Selling  ETH
2019-03-24 22:07:20 (DEBUG):    Creating order to sell 9.61692 ETH
2019-03-24 22:07:26 (DEBUG):    [ORDER] statusChange: SUBMITTED
2019-03-24 22:07:26 (DEBUG):    [ORDER] statusChange: OPEN
2019-03-24 22:07:29 (INFO):     [ORDER] partial sell fill, total filled: 0.08511
2019-03-24 22:07:37 (INFO):     [ORDER] partial sell fill, total filled: 9.61692
2019-03-24 22:07:37 (DEBUG):    [ORDER] statusChange: FILLED
2019-03-24 22:07:39 (INFO):     [ORDER] summary: { price: 135.91,
  amount: 9.61692,
  date:
   Moment {
     _isAMomentObject: true,
     _i:<PHONE_NUMBER>060,
     _isUTC: false,
     _pf:
      { empty: false,
        unusedTokens: [],
        unusedInput: [],
        overflow: -2,
        charsLeftOver: 0,
        nullInput: false,
        invalidMonth: null,
        invalidFormat: false,
        userInvalidated: false,
        iso: false,
        parsedDateParts: [],
        meridiem: null,
        rfc2822: false,
        weekdayMismatch: false },
     _locale:
      Locale {
        _calendar: [Object],
        _longDateFormat: [Object],
        _invalidDate: 'Invalid date',
        ordinal: [Function: ordinal],
        _dayOfMonthOrdinalParse: /\d{1,2}(th|st|nd|rd)/,
        _relativeTime: [Object],
        _months: [Array],
        _monthsShort: [Array],
        _week: [Object],
        _weekdays: [Array],
        _weekdaysMin: [Array],
        _weekdaysShort: [Array],
        _meridiemParse: /[ap]\.?m?\.?/i,
        _abbr: 'en',
        _config: [Object],
        _dayOfMonthOrdinalParseLenient: /\d{1,2}(th|st|nd|rd)|\d{1,2}/ },
     _d: 2019-03-24T22:07:36.060Z,
     _isValid: true },
  side: 'sell',
  orders: 1,
  fees: { BNB: 0.05722519 },
  feePercent: 0.00075 }
2019-03-24 22:07:39 (DEBUG):    syncing private data
2019-03-24 22:08:20 (INFO):     (ROUNDTRIP) entry date (UTC)    exit date (UTC)         exposed duration        P&L
        profit
2019-03-24 22:08:20 (INFO):     (ROUNDTRIP) 2019-03-24 11:22    2019-03-24 22:07        11 hours                -6.
92      -0.50
2019-03-24 22:08:21 (INFO):     (PROFIT REPORT) original balance:                1374.63421451 USDT
2019-03-24 22:08:21 (INFO):     (PROFIT REPORT) current balance:                 1376.80003211 USDT
2019-03-24 22:08:21 (INFO):     (PROFIT REPORT) profit:                          2.16581760 USDT (0.15755592%)

same here with Binance. Some Trades he counts, most not.
Papertrader is running without any issue.

Same issue runing on Docker

Encountered the same problem. The last trade would not show up on the UI and this is what I found in the log.

Trading on Binance.

I did a further investigation into this case, seems to be a specific issue with Binance.
It happens when the order get filled in full before it could be cancelled, meaning it was not cancelled, hence the Error -2011: Unknown order sent was logged (or 'CANCEL_REJECTED' according to Binance API doc).
There's a patch that has already been included in Gekko to handle this situation in #2328. However, I'm not sure why the trade did not register itself in Gekko UI. Hopefully someone would be able to help figure it out?

Hey I get this issue too.
The error seems to occur on Binance. For orders that do not get filled quickly.
In the log output below describes my issue. I am getting it often.
I do not know if the error effects trades AFTER the error but the application still appears to run despite the error. However I have forced it to restart after every time this has happened (which is around 6 times so far in a week).
2019-05-22 18:40:58 (INFO):     Trader Received advice to go long. Buying  POA
2019-05-22 18:40:58 (DEBUG):    Creating order to buy 10815.903222685185 POA
2019-05-22 18:40:58 (DEBUG):    Buy - Predicted variation:  0.8528561945502295
2019-05-22 18:40:58 (DEBUG):    Buy - Predicted variation:  0.7799634016436916
2019-05-22 18:40:58 (DEBUG):    Buy - Predicted variation:  0.6876504127364681
2019-05-22 18:40:58 (DEBUG):    Buy - Predicted variation:  0.5819534880460339
2019-05-22 18:41:05 (DEBUG):    [ORDER] statusChange: SUBMITTED
2019-05-22 18:41:05 (DEBUG):    [ORDER] statusChange: OPEN
2019-05-22 18:41:07 (DEBUG):    [ORDER] statusChange: CHECKING
2019-05-22 18:41:07 (DEBUG):    [ORDER] statusChange: CHECKED
2019-05-22 18:41:09 (DEBUG):    [ORDER] statusChange: CHECKING
#checking for several minutes....
2019-05-22 18:48:25 (DEBUG):    [ORDER] statusChange: CHECKING
2019-05-22 18:48:26 (DEBUG):    [ORDER] statusChange: CHECKED
2019-05-22 18:48:27 (DEBUG):    [ORDER] statusChange: MOVING
2019-05-22 18:48:28 (DEBUG):    [ORDER] statusChange: ERROR
2019-05-22 18:48:28 (ERROR):    [ORDER] Gekko received error from GB: Error -2011: Unknown order sent.
2019-05-22 18:48:28 (DEBUG):    Error: Error -2011: Unknown order sent.
at /home/mack/gekko/exchange/wrappers/binance.js:101:15
at Request.request [as _callback] (/home/mack/gekko/exchange/node_modules/binance/lib/rest.js:90:25)
at Request.self.callback (/home/mack/gekko/exchange/node_modules/request/request.js:185:22)
at emitTwo (events.js:126:13)
at Request.emit (events.js:214:7)
at Request. (/home/mack/gekko/exchange/node_modules/request/request.js:1157:10)
at emitOne (events.js:116:13)
at Request.emit (events.js:211:7)
at IncomingMessage. (/home/mack/gekko/exchange/node_modules/request/request.js:1079:12)
at Object.onceWrapper (events.js:313:30)
at emitNone (events.js:111:20)
at IncomingMessage.emit (events.js:208:7)
at endReadableNT (_stream_readable.js:1064:12)
at _combinedTickCallback (internal/process/next_tick.js:139:11)
at process._tickCallback (internal/process/next_tick.js:181:9)

Guys - a fix has been found, and shared in this thread from all3kcis: https://github.com/askmike/gekko/issues/2762
File tweaks must be made to exchange/wrappers/binance.js - see modifications here https://github.com/askmike/gekko/pull/2792/files

Thanks for the heads up! This seems to have resolved the issue for me.


feat: multiple wallets

I'd like to have two separate wallets or 'accounts' in Minibits so i can segregate funds without have to use a separate andriod profile, or different cashu app.

Will think of proper way how to design this over the longer time horizont. Minibits is already multi-mint and multi-currency, co making it multi-wallet is not only technical but as well an UX challenge.


Framerate problems in RealSense Viewer after version 2.13.0

Required Info

Camera Model
D400 series

Firmware Version
any

Operating System & Version
Win10

Issue Description
To reproduce:

Plug in a RS D415 or D435
Start RS Viewer 2.13.0 and turn on color streaming. Should be 30 fps.
Close RS Viewer 2.13.0
Open RS Viewer 2.14.0 or later and turn on color streaming.
Framerate is not 30 fps.

Similar issues in the Viewer with depth or combined depth + color.
Fortunately when we use libRS in our own app we don't see these issues and get full frame rate.

[Realsense Customer Engineering Team Comment]
Hi @xtrawurst,
Did you build the realsense-viewer from the source? I didn't observe this issue on my configuration.
You can also try the pre-built realsense-viewer to see how it works.

@RealSense-Customer-Engineering We used the compiled builds that can be found here.

[Realsense Customer Engineering Team Comment]
Hi @xtrawurst,
Please help enable the log "set LRS_LOG_LEVEL=DEBUG" first in the command line.
Once the issue reproduced, please send us the log file which should be the same location with your realsense-viewer.

[Realsense Customer Engineering Team Comment]
Hi @xtrawurst,
Is the issue still there for further debug?

I can confirm that this is no longer an issue in 2.16.3.