Best Generic Image for Windows 10 22H2 (Build 19045)

Description
For hosts running Windows 10 22H2 (19045), the generic image of Windows 10 2004 (19041) should be used because this version works better than the image of Windows 10 20H2 (19042).
Similar to Windows 10 21H1 (19043) and Windows 10 21H2 (19044).
At least a container with process isolation works when using the generic image of Windows 10 2004 (19041).
With generic image of Windows 10 20H2 (19042) and process isolation:
Script
New-BcContainer `
    -accept_eula `
    -accept_outdated `
    -containerName 'Test' `
    -artifactUrl $( Get-BCArtifactUrl ) `
    -isolation process

Output
BcContainerHelper is version 4.0.5
BcContainerHelper is running as administrator
Hyper-V is Enabled
UsePsSession is True
Host is Microsoft Windows 10 Enterprise - 22H2
Docker Client Version is 20.10.20
Docker Server Version is 20.10.20
Removing Desktop shortcuts
The CRONUS Demo License shipped in Version 21.0 artifacts doesn't contain sufficient rights to all Test Libraries objects. Patching the license file.
Fetching all docker images
Fetching all docker volumes
Using image mcr.microsoft.com/businesscentral:10.0.19042.1889
Creating Container Test
Style: sandbox
Multitenant: Yes
Version: 21.0.46256.48144
Platform: 21.0.46384.48092
Generic Tag: <IP_ADDRESS>
Container OS Version: 10.0.19042.1889 (20H2)
Host OS Version: 10.0.19045.2130 (22H2)
WARNING: Host OS and Base Image Container OS doesn't match and process isolation is specified. If you encounter issues, you could try to specify -isolation hyperv
Using process isolation
Using locale en-US
Disabling the standard eventlog dump to container log every 2 seconds (use -dumpEventLog to enable)
Using license file https://bcartifacts.azureedge.net/prerequisites/21demolicense/w1/3048953.bclicense
Downloading C:\ProgramData\BcContainerHelper\Extensions\Test\my\license.bclicense
Files in C:\ProgramData\BcContainerHelper\Extensions\Test\my:
- AdditionalOutput.ps1
- license.bclicense
- MainLoop.ps1
- SetupVariables.ps1
- updatecontainerhosts.ps1
Creating container Test from image mcr.microsoft.com/businesscentral:10.0.19042.1889
0a089f56a98cb0ecb25a7301fbffb907286932887307e0947a2e7285774da2c5
New-BcContainer Telemetry Correlation Id: ff6333fe-12fc-43a8-9d03-a8023d735b90
DockerDo : docker: Error response from daemon: hcsshim::CreateComputeSystem 0a089f56a98cb0ecb25a7301fbffb907286932887307e0947a2e7285774da2c5: The container operating system does not match the host operating system.
ExitCode: 125
Commandline: docker run --volume "c:\bcartifacts.cache:c:\dl" --label nav= --env isBcSandbox=Y --label version=21.0.46256.48144 --label platform=21.0.46384.48092 --label country=W1 --env artifactUrl=https://bcartifacts.azureedge.net/sandbox/21.0.46256.48144/w1 --env multitenant=Y
--env licenseFile="c:\run\my\license.bclicense" --name Test --hostname Test --env auth=Windows --env username="owunschmann4046" --env ExitOnError=N --env locale=en-US --env databaseServer="" --env databaseInstance="" --volume
"C:\ProgramData\BcContainerHelper:C:\ProgramData\BcContainerHelper" --volume "C:\ProgramData\BcContainerHelper\Extensions\Test\my:C:\Run\my" --isolation process --restart unless-stopped --env filesOnly=False --env enableApiServices=Y --env useSSL=N --env securePassword=76492d1116743
f0423413b16050a5345MgB8AEUAOQBQAHQAUgB0AFAAWAB6ACsAOQA5AFUAZgBEAFkAOQAwAHMARQBVAEEAPQA9AHwANgBhADEAOQAxAGEAMAA5AGQANgAzAGQAMgA2ADkAOAA5AGYAMAA1ADYAZQAyADEAMwA3ADcAZQAzAGYAMgBmADUANgBiAGUAZAA5ADMAMABhADIAZgA3ADkANQA2AGQAYQAyADUANwBmAGEAMwBkAGIANQBkADcAZgAyADMANgA= --env
passwordKeyFile="c:\run\my\aes.key" --env removePasswordKeyFile=Y --env accept_eula=Y --env accept_outdated=Y --detach mcr.microsoft.com/businesscentral:10.0.19042.1889
In C:\Program Files\WindowsPowerShell\Modules\bccontainerhelper\4.0.5\ContainerHandling\New-NavContainer.ps1:1923 Zeichen:15
+ ...       if (!(DockerDo -accept_eula -accept_outdated:$accept_outdated - ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,DockerDo

With generic image of Windows 10 2004 (19041) and process isolation:
Script
New-BcContainer `
    -accept_eula `
    -accept_outdated `
    -containerName 'Test' `
    -artifactUrl $( Get-BCArtifactUrl ) `
    -isolation process `
    -useGenericImage $( Get-BestGenericImageName -hostOsVersion 10.0.19041 )

Output
BcContainerHelper is version 4.0.5
BcContainerHelper is running as administrator
Hyper-V is Enabled
UsePsSession is True
Host is Microsoft Windows 10 Enterprise - 22H2
Docker Client Version is 20.10.20
Docker Server Version is 20.10.20
Removing Desktop shortcuts
Removing C:\ProgramData\BcContainerHelper\Extensions\Test
The CRONUS Demo License shipped in Version 21.0 artifacts doesn't contain sufficient rights to all Test Libraries objects. Patching the license file.
Fetching all docker images
Fetching all docker volumes
Using image mcr.microsoft.com/businesscentral:10.0.19041.1415
Creating Container Test
Style: sandbox
Multitenant: Yes
Version: 21.0.46256.48144
Platform: 21.0.46384.48092
Generic Tag: <IP_ADDRESS>
Container OS Version: 10.0.19041.1415 (2004)
Host OS Version: 10.0.19045.2130 (22H2)
WARNING: Host OS and Base Image Container OS doesn't match and process isolation is specified. If you encounter issues, you could try to specify -isolation hyperv
Using process isolation
Using locale en-US
Disabling the standard eventlog dump to container log every 2 seconds (use -dumpEventLog to enable)
Using license file https://bcartifacts.azureedge.net/prerequisites/21demolicense/w1/3048953.bclicense
Downloading C:\ProgramData\BcContainerHelper\Extensions\Test\my\license.bclicense
Files in C:\ProgramData\BcContainerHelper\Extensions\Test\my:
- AdditionalOutput.ps1
- license.bclicense
- MainLoop.ps1
- SetupVariables.ps1
- updatecontainerhosts.ps1
Creating container Test from image mcr.microsoft.com/businesscentral:10.0.19041.1415
2929df57f945d444142f35320362fbd7f748b52a7119d51c55b654a9afe13c8f
Waiting for container Test to be ready
Using artifactUrl https://bcartifacts.azureedge.net/sandbox/21.0.46256.48144/w1
Using installer from C:\Run\210-new
Installing Business Central
Installing from artifacts
Starting Local SQL Server
Starting Internet Information Server
Copying Service Tier Files
c:\dl\sandbox\21.0.46256.48144\platform\ServiceTier\Program Files
c:\dl\sandbox\21.0.46256.48144\platform\ServiceTier\System64Folder
Copying PowerShell Scripts
c:\dl\sandbox\21.0.46256.48144\platform\WindowsPowerShellScripts\Cloud\NAVAdministration
c:\dl\sandbox\21.0.46256.48144\platform\WindowsPowerShellScripts\WebSearch
Copying Web Client Files
c:\dl\sandbox\21.0.46256.48144\platform\WebClient\Microsoft Dynamics NAV
Copying ModernDev Files
c:\dl\sandbox\21.0.46256.48144\platform
c:\dl\sandbox\21.0.46256.48144\platform\ModernDev\program files\Microsoft Dynamics NAV
Copying additional files
Copying ConfigurationPackages
C:\dl\sandbox\21.0.46256.48144\platform\ConfigurationPackages
Copying Test Assemblies
C:\dl\sandbox\21.0.46256.48144\platform\Test Assemblies
Copying Extensions
C:\dl\sandbox\21.0.46256.48144\w1\Extensions
Copying Applications
C:\dl\sandbox\21.0.46256.48144\platform\Applications
Copying dependencies
Copying ReportBuilder
Importing PowerShell Modules
Restoring CRONUS Demo Database
Setting CompatibilityLevel for tenant on localhost\SQLEXPRESS
Exporting Application to CRONUS
Removing Application from tenant
Modifying Business Central Service Tier Config File for Docker
Creating Business Central Service Tier
Installing SIP crypto provider: 'C:\Windows\System32\NavSip.dll'
Starting Business Central Service Tier
Importing license file                                                                                Copying Database on localhost\SQLEXPRESS from tenant to default                                                                                Taking database tenant offline                                                                                Copying database files                                                                                Attaching files as new Database default
Putting database tenant back online
Mounting tenant database
Mounting Database for default on server localhost\SQLEXPRESS with AllowAppDatabaseWrite = False
Sync'ing Tenant
Tenant is Operational
Stopping Business Central Service Tier
Installation took 213 seconds
Installation complete
Initializing...
Setting host.containerhelper.internal to <IP_ADDRESS> in container hosts file
Starting Container
Hostname is Test
PublicDnsName is Test
Using Windows Authentication
Modifying Service Tier Config File with Instance Specific Settings
Starting Service Tier
Registering event sources
Creating DotNetCore Web Server Instance
Using application pool name: BC
Using default container name: NavWebApplicationContainer
Copy files to WWW root C:\inetpub\wwwroot\BC
Create the application pool BC
Create website: NavWebApplicationContainer without SSL
Update configuration: navsettings.json
Done Configuring Web Client
Enabling Financials User Experience
Using license file 'c:\run\my\license.bclicense'
Import License
Dismounting Tenant
Mounting Tenant
Mounting Database for default on server localhost\SQLEXPRESS with AllowAppDatabaseWrite = False
Sync'ing Tenant
Tenant is Operational
Creating http download site
Creating Windows user owunschmann4046
Setting SA Password and enabling SA
Creating SUPER user
Container IP Address: <IP_ADDRESS>
Container Hostname  : Test
Container Dns Name  : Test
Web Client          : http://Test/BC/?tenant=default
Dev. Server         : http://Test
Dev. ServerInstance : BC
Dev. Server Tenant  : default
Setting Test-default to <IP_ADDRESS> in container hosts file

Files:
http://Test:8080/ALLanguage.vsix

WARNING: You are running a container which is 225 days old.
Microsoft recommends that you always run the latest version of our containers.

Container Total Physical Memory is 31.7Gb
Container Free Physical Memory is 14.4Gb

Initialization took 111 seconds
Ready for connections!
Reading CustomSettings.config from Test
Creating Desktop Shortcuts for Test
Container Test successfully created

Use:
Get-BcContainerEventLog -containerName Test to retrieve a snapshot of the event log from the container
Get-BcContainerDebugInfo -containerName Test to get debug information about the container
Enter-BcContainer -containerName Test to open a PowerShell prompt inside the container
Remove-BcContainer -containerName Test to remove the container again
docker logs Test to retrieve information about URL's again

Shipped in 4.0.6


Slowness preparing jobs

Now I can provide a MWE on this! Here is the SoS script to reproduce:
[1]
input: glob.glob('*.txt'), group_by = 1, concurrent = True
output: dynamic(glob.glob("*.out"))
bash: expand = True
  touch {_input:n}.out

To see the problem, first go to an empty folder, type touch {1..50000}.txt in bash to create 50K test input files. Then run sos run test.sos. It takes forever to even get SoS started real processes, at least on my mechanical hard drive. This is a big problem now because previously I can analyze input of up to 50K scale and now I'm just trying to rerun my earlier analysis. Hopefully the bottleneck can be identified. It must be certainly signature related because I've not seen this before.

You see, even bash complains. :smile:
$ touch {1..50000}.txt
-bash: /usr/bin/touch: Argument list too long

Ha, not my bash though. Interesting ...

One of the biggest problems here is that after the execution of each substep, sos needs to glob all .out files, which increases linearly with increasing _index. So while my test here with 5000 files completes reasonably fast, 50000 can be hundreds of times slower.
Note that sos handles *.out and *.txt automatically so no glob.glob is needed.

Okay but why this was not a problem before? I've been using the same pipeline since some time ago. I believe the issue was recent.

Note that sos handles *.out and *.txt automatically so no glob.glob is needed.

i did not try but my actual pattern was more complicated, eg */something/*. That was why I thought I needed glob. i can try to see if SoS handles that pattern.

I mean, your script would be a lot faster with
input: '*.txt', group_by = 1, concurrent = True
output: _input.with_suffix('.out')
bash: expand = True
  touch {_output}

$ time sos run test
INFO: output:   1.out, 10.out, ... (5000 items)
INFO: Workflow default (ID=b959cc02ac21a43d) is executed successfully with 1 completed step and 5000 completed substeps.

real	1m21.507s
user	1m53.212s
sys	2m29.002s

Sure but in reality the input and output do not have one to one match. That was why I had to use glob ...

The reason was that file signatures were saved individually so each time only a new file needs to be checksumed. The current version put file signatures inside step signature so files needs to be checksumed again and again.
The reason is that sos do not expect overlapping output files from substeps, and it will actually error out if you do output: 'same.out' for steps with substeps. Your case is allowed only because it is dynamic so sos cannot prevent it. If output from substeps are distinct, keeping file signatures in step signature is a reasonable choice as it reduces repetition and reduces most transactions to a separate target database.

In reality, if 1.out is removed, all substeps need to be rerun so it does not sound like a good arrangement anyway.

The current version put file signatures inside step signature so files needs to be checksumed again and again.

I'm not sure if I get all the details, but this does not sound good.
Also suppose I have dynamic input and output with index like :
input: dynamic(some_files)
output: other_files[_index]

then SoS will know to check one file at a time rather than all of them over and over again? I can certainly make lots of my applications this style as you previously suggested. But I do have cases when dynamic is needed in both input and output.

Your best bet could be -s ignore
$ time sos run test -s ignore
INFO: output:   1.txt.out, 10.txt.out, ... (5000 items)
INFO: Workflow default (ID=47e6d11a062fe683) is executed successfully with 1 completed step and 5000 completed substeps.

real	1m13.566s
user	0m23.077s
sys	0m50.068s

I see. Okay that can get my analysis going now :) But is there a way out here -- combine the reasonable parts of the old and new implementations while still keeping the traffic low?

I'm not sure if I get all the details, but this does not sound good.

The tech details is that we used to keep a separate signature for each input/output file, and we check these files when we create step signatures. md5 were not calculated if the signature already exists. When we switch to database, the target signature database become the busiest one with highest chance of collision and so I removed it in favor of keeping signatures in step signature. These means output files belonging to multiple substeps will be checksumed repeatedly, but "output belonging to multiple substeps" is dangerous and generally forbidden in sos. Just think this case
input: for_each={'n': range(100)}
output: [f'test_{x}.txt' for x in range(n)]
for file in _output:
    with open(file, 'a') as f:
        f.write(_index)

where the next substep will change the output of the previous substeps, so signatures will never work.

will know to check one file at a time rather than all of them over and over again?

Yes, because there is only one output for each substep.

yes, each process can cache the md5 of processed files, in a database or in RAM, although this is only a optimization for a case that we try to avoid.

I see ... seems like a corner case bug. Still would be good to fix to avoid repeatedly checking step signatures unnecessarily.

An additional observation, perhaps related to this comment, is that rebuiding signatures can be very slow depending on network drive conditions. I've got a completed local workflow on a system with slow disk. I removed signature database and run again with -s build. It takes forever just for SoS to check and recover signatures -- for a simple step involving <5000 files it is now running 4min before moving on, and still counting! The same task completed quickly on SSD. So perhaps having some in-memory database is not generally a bad idea after all, not just for corner cases?

I agree...

Not sure if this is related to this issue or #1018 , but I'm using an interactive node on my cluster (first time running SoS on interactive node). I've got a step of my job done, involving dynamic output the same as above, but there are 70K of them. I'm now stuck at this:

And here are a few diagnostics:
[MW] date
Wed Aug 22 14:52:30 CDT 2018
[MW] ll .sos
total 1.5M
-rw-r--r-- 1 gaow pi-mstephens 180K Aug 22 14:30 step_signatures.db
-rw-rw-r-- 1 gaow pi-mstephens 155K Aug 22 14:31 transcript.txt
-rw-r--r-- 1 gaow pi-mstephens 1.2M Aug 22 14:43 workflow_signatures.db
[MW] ll ~/.sos
total 9.2M
drwxrwxr-x 2 gaow gaow 4.0K Aug 10 18:09 bin
-rw-r--r-- 1 gaow gaow 9.1M Aug 12 16:58 target_signatures.db
-rw-r--r-- 1 gaow gaow  64K Aug 22 14:30 step_signatures.db
-rw-rw-r-- 1 gaow gaow  973 Aug 22 14:31 3dae29f04e9643cf.status
drwxrwxr-x 3 gaow gaow  32K Aug 22 14:43 tasks
[MW] ps -A | grep sos
47132 pts/2    00:00:09 sos
47154 pts/2    00:00:08 sos

You see no database has been written to for about 10 minutes now. That is I'm looking at the hanging screen for 10min. However unlike #1018 I can still see sos processes. Is that still disk issues?

Just make sure. The tasks are running locally, not waiting for pbs, right. Note that even for local queue there is s max running limit so sos might be waiting for the queue to clear a bit before running more.

Oh sure yes those are local tasks. They are supposed to be done (I see 22 completed for my 22 chroms) and i got lots of dynamic output (~80K). The interactive compute node i'm on current only has those sos instances running, not Python processes for those tasks.
ps: still hanging as we speak.

What is the number of running and pending jobs if you run sos status?

[MW] git status
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working tree clean
[MW] sos status
9cc490b4384923b3        3dae29f04e9643cf preprocess_2   Ran for 1 min   completed
b660debf995ab997        3dae29f04e9643cf preprocess_2   Ran for 2 min   completed
e61b3467c27dbabe        3dae29f04e9643cf preprocess_2   Ran for 2 min   completed
0d780fb2ac03d70d        3dae29f04e9643cf preprocess_2   Ran for 5 min   completed
aa34afae1c898712        3dae29f04e9643cf preprocess_2   Ran for 5 min   completed
b13b91bfaf364435        3dae29f04e9643cf preprocess_2   Ran for 5 min   completed
6dc89f06426127f4        3dae29f04e9643cf preprocess_2   Ran for 5 min   completed
5f9b9ebe9d2afbb3        3dae29f04e9643cf preprocess_2   Ran for 5 min   completed
6481886f5399cfa5        3dae29f04e9643cf preprocess_2   Ran for 5 min   completed
3f0e6becfa18647a        3dae29f04e9643cf preprocess_2   Ran for 5 min   completed
3ee34e16de3b41f9        3dae29f04e9643cf preprocess_2   Ran for 5 min   completed
a7b97b7657f420a8        3dae29f04e9643cf preprocess_2   Ran for 6 min   completed
bcb5889c9cd4d077        3dae29f04e9643cf preprocess_2   Ran for 6 min   completed
0871188a3ea7b103        3dae29f04e9643cf preprocess_2   Ran for 6 min   completed
e5df66c53209f8e7        3dae29f04e9643cf preprocess_2   Ran for 6 min   completed
904334909d5896dd        3dae29f04e9643cf preprocess_2   Ran for 8 min   completed
d463c0eb5dfde437        3dae29f04e9643cf preprocess_2   Ran for 8 min   completed
e7b04956cbf664f5        3dae29f04e9643cf preprocess_2   Ran for 8 min   completed
4e6a7460fed13775        3dae29f04e9643cf preprocess_2   Ran for 8 min   completed
d42de8ae2f8dfd23        3dae29f04e9643cf preprocess_2   Ran for 8 min   completed
eebfbac3127107d0        3dae29f04e9643cf preprocess_2   Ran for 9 min   completed
fbf664ba49e5c9b5        3dae29f04e9643cf preprocess_2   Ran for 11 min  completed
[MW]

but my next step in the workflow has not started.

These are all the tasks for this step? Wondering what sos is doing.

With -v4 in this case there would be some scrolling text such as xx running which would show some problem if the number is inconsistent with what we see outside.

Yes these are all tasks for one step. SoS is apparently running 2 threads, one for 14min one 24min:
[MW] ps -A | grep sos
47132 pts/2    00:00:14 sos
47154 pts/2    00:00:24 sos

I suspect it is working on the database, but the time stamp suggests otherwise because modification timestamp did not change.
Maybe i'll wait a bit and try again with -v4.

Flight taking off. Later.

Still no rush, but is there a plan to tackle this ticket, and the related concurrent issues? I've got to rerun a pipeline with updated methods that relies on concurrence in the presence of shared output. I can revert to an earlier version of SoS if we need more thoughts to fix these ones.

The problem here is that I cannot reproduce the problem, so I am working on something more reproducible (#1028).

Of course, thank you! yes it should be easier if #1028 is fixed first then at least we have one less factor to worry about when we try to come up with MWE for this one. I've being trying yesterday (though not hard enough) but still cannot get something small to upload here that can possibly reproduce it. But I'll try it again.

With the current master my example in the first post is reasonably fast -- under a minute before it can start running. I'm closing this ticket for now. Will see if down the road there is a need to make it even faster for larger scale problems (if we run into it).


Add 2 tips based on personal experience

Added two new tips/tricks. "Try to sort the data on a web page in different ways" and "Play with the URL!"

@schasins Let me know what you think!
I thought about adding emojis to point out the "new" tips to catch people's attention in case they have looked at the tips list before. Up to you though!


Python driver to be used for a system that makes highly concurrent requests to db

my environment running ArangoDB
I'm using the latest ArangoDB of the respective release series:

[ ] 2.8
[ ] 3.0
[ ] 3.1
[ ] 3.2
[x] 3.3
[ ] self-compiled devel branch

this is a question
As there is an issue raised in pyArango (When using gevent/greenlets, highly concurrent requests can return incorrect responses), what would be the best python driver to use in such scenarios.
Is there any roadmap for the implementation of official driver from the ArangoDB team.
Thanks,
Dilip Vamsi.

Hi,
as demonstrated with the performance blog post series,
you need to monkey patch the propper http client to be used:
https://arangodb.com/2017/10/performance-analysis-pyarango-usage-scenarios/
The reporter also forked pyArango and added some changes:
https://github.com/bosskeyproductions/pyArango
which are currently in production use.
However, we didn't inspect them for backporting yet.

Hi @dothebart ,
So, for just a monkey patch we don't the statsd client, right? Or, do I need to make any other changes.
Also I'm not clear on which version of pyArango to use as the version by booskeyproductions is old.
Can you point to the repo which has active development.
What are your thoughts about python-arango which uses rest api of arangodb.
Can this be an alternative of this scenario for high concurrency or just stick with the monkey patch.
This is a changed version of code without statsd client.
#!/usr/bin/python
import json as json_mod
import sys
import random
import pyArango
from datetime import datetime
from pyArango.connection import *
from pyArango.collection import *
import requests
from gevent import monkey
from gevent import GreenletExit
monkey.patch_all()
import gevent
import grequests

auth_token = None
connection_urls = [
    "http://<IP_ADDRESS>:8529"
]

class JWTAuth(requests.auth.AuthBase):
    def __init__(self, token):
        self.token = token

    def __call__(self, r):
        # Implement JWT authentication
        r.headers['Authorization'] = 'Bearer %s' % self.token
        return r
def get_auth_token():
    global auth_token, connection_urls

    if auth_token:
        return auth_token

    kwargs = {'data': '{"username":"%s","password":"%s"}' % ("root", "test")}
    for connection_url in connection_urls:
        response = requests.post('%s/_open/auth' % connection_url, **kwargs)
        if response.ok:
            json_data = response.content
            if json_data:
                data_dict = json_mod.loads(json_data)
                auth_token = data_dict.get('jwt')
                break
    return auth_token

class AikidoSession(object):
    def __init__(self, session_username, session_password):
        if session_username:
            self.auth = JWTAuth(session_password)
        else:
            self.auth = None

    def post(self, url, data=None, json=None, **kwargs):
        if data is not None:
            kwargs['data'] = data
        if json is not None:
            kwargs['json'] = json

        kwargs['auth'] = self.auth
        return grequests.map([grequests.post(url, **kwargs)])[0]

    def get(self, url, **kwargs):
        kwargs['auth'] = self.auth
        result = grequests.map([grequests.get(url, **kwargs)])[0]
        return result

    def put(self, url, data=None, **kwargs):
        if data is not None:
            kwargs['data'] = data
        kwargs['auth'] = self.auth
        return grequests.map([grequests.put(url, **kwargs)])[0]

    def head(self, url, **kwargs):
        kwargs['auth'] = self.auth
        return grequests.map([grequests.put(url, **kwargs)])[0]

    def options(self, url, **kwargs):
        kwargs['auth'] = self.auth
        return grequests.map([grequests.options(url, **kwargs)])[0]

    def patch(self, url, data=None, **kwargs):
        if data is not None:
            kwargs['data'] = data
        kwargs['auth'] = self.auth
        return grequests.map([grequests.patch(url, **kwargs)])[0]

    def delete(self, url, **kwargs):
        kwargs['auth'] = self.auth
        return grequests.map([grequests.delete(url, **kwargs)])[0]

    def disconnect(self):
        pass

# Monkey patch the connection object:
pyArango.connection.AikidoSession = AikidoSession

conn = Connection(username="root", password=get_auth_token())
db = conn["_system"]
print(db.collections)

Thanks,
Dilip Vamsi.

Hi,
the statsd functionality has meanwhile been integrated in pyArango for optional use; however you don't need it from the functional point of view.
pyArango and python-arango use the HTTP API - so no difference from that POV. Both meanwhile run on python 3. Neither one has native gevent support; I personally only tried to monkey patch it into pyArango.
The pyArango aikidosession has been the source of troubles during the python 3.0 migration which have been resolved. For monkey-patching it has prooven to be good ;)
Both maintainers tend to be reactive to pullrequests, issues etc.
In general its probably a question which provided python api you like better - that of pyArango or python-arango.

Hi,
This is the final version of code that be directly imported as file.
"""Connection creator."""
import json as _json
import random as _random
import requests as _requests
import grequests as _grequests

class TokenGenrator(object):

    def __init__(self, username, password, connection_urls):
        """Init the credentials for getting the auth token.

        Parameters
        ----------
        username : string
            username that exists in db.
        password : string
            password for the given user.
        connection_urls : list(string)
            list of connection strings for the db instances.

        """
        self.username = username
        self.password = password
        self.connection_urls = connection_urls
        self._auth_token = None
        self._connection_indices = list(range(len(connection_urls)))

    def get_connection_indicies(self, pick_random_server):
        """Return the connection indicies."""
        if not pick_random_server:
            return self._connection_indices
        connection_indices = list(range(len(self.connection_urls)))
        _random.shuffle(connection_indices)
        return connection_indices

    def get_auth_token(self, pick_random_server=False):
        """Get auth token from given list of servers.

        pick_random_server : bool, optional
            pick selection of server by random
            (the default is False, which means select in the given order.)

        Returns
        -------
        string
            auth token generated from the server.

        """
        if self._auth_token is not None:
            return self._auth_token

        kwargs = {
            'data': '{"username":"%s","password":"%s"}' % (
                self.username, self.password
            )
        }

        for connection_index in self.get_connection_indicies(
                pick_random_server
        ):
            response = _requests.post(
                '%s/_open/auth' % self.connection_urls[connection_index],
                **kwargs
            )
            if response.ok:
                json_data = response.content
                if json_data:
                    data_dict = _json.loads(json_data)
                    self._auth_token = data_dict.get('jwt')
        return self._auth_token

class JWTAuth(_requests.auth.AuthBase):
    def __init__(self, token):
        self.token = token

    def __call__(self, r):
        # Implement JWT authentication
        r.headers['Authorization'] = 'Bearer %s' % self.token
        return r

class AikidoSession(object):
    def __init__(self, session_username, session_password):
        if session_username:
            self.auth = JWTAuth(session_password)
        else:
            self.auth = None

    def post(self, url, data=None, json=None, **kwargs):
        if data is not None:
            kwargs['data'] = data
        if json is not None:
            kwargs['json'] = json

        kwargs['auth'] = self.auth
        return _grequests.map([_grequests.post(url, **kwargs)])[0]

    def get(self, url, **kwargs):
        kwargs['auth'] = self.auth
        result = _grequests.map([_grequests.get(url, **kwargs)])[0]
        return result

    def put(self, url, data=None, **kwargs):
        if data is not None:
            kwargs['data'] = data
        kwargs['auth'] = self.auth
        return _grequests.map([_grequests.put(url, **kwargs)])[0]

    def head(self, url, **kwargs):
        kwargs['auth'] = self.auth
        return _grequests.map([_grequests.put(url, **kwargs)])[0]

    def options(self, url, **kwargs):
        kwargs['auth'] = self.auth
        return _grequests.map([_grequests.options(url, **kwargs)])[0]

    def patch(self, url, data=None, **kwargs):
        if data is not None:
            kwargs['data'] = data
        kwargs['auth'] = self.auth
        return _grequests.map([_grequests.patch(url, **kwargs)])[0]

    def delete(self, url, **kwargs):
        kwargs['auth'] = self.auth
        return _grequests.map([_grequests.delete(url, **kwargs)])[0]

    def disconnect(self):
        pass

def get_connection(
        username, password, connection_urls, is_gevent=False,
        pick_random_server=False
):
    """Get the connection for the given credentials.

    Parameters
    ----------
    username : string
        username that exists in db.
    password : string
        password for the given user.
    connection_urls : list(string)
        list of connection strings for the db instances.
    is_gevent : bool, optional
        mokey patchs to AikidoSession (the default is False,
        which means don't patch)
    pick_random_server : bool, optional
        pick selection of server by random
        (the default is False, which means select in the given order.)

    Returns
    -------
    pyArango.connection.Connection
        returns patched version if the connection is for gevent.

    """
    import pyArango.connection
    if is_gevent:
        pyArango.connection.AikidoSession = AikidoSession
        auth_token = TokenGenrator(username, password, connection_urls)
        return pyArango.connection.Connection(
            username=username,
            password=auth_token.get_auth_token(pick_random_server)
        )

    # for dev.
    connection_index = 0 if not pick_random_server \
        else _random.randint(0, len(connection_urls) - 1)
    return pyArango.connection.Connection(
        arangoURL=connection_urls[connection_index],
        username=username, password=password
    )

if __name__  == "__main__":
    import db_settings
    CONNECTION = get_connection(
        db_settings.USERNAME, db_settings.PASSWORD, db_settings.CONNECTION_URLS,
        db_settings.IS_GEVENT, db_settings.PICK_RANDOM_SERVER
    )
    print(CONNECTION["_system"].collections)

@tariqdaouda do you see a good way to integrate this into pyArango for users that want to use gevent?

Nice! From a quick glance It seems to me that is pretty much integrated already. Could you send me a pull request? That would make the comparison with the current version a little easier.

@tariqdaouda the question would be, how to switch the Aikidosession if running with gevent or native in the most elegant way?

Ok, I'll find a way to integrate it that makes it transparent to the user.

I have a few questions:

The full URL is given for every request type, which means that it must contain the server URL as well.  Is this the expected behavior or should the server be chosen among the ones in the list?

By server do you mean endpoint? In that case, the list can be retrieved from the DB.

Hi @tariqdaouda ,
For attaining the auth token you need to do a post request to the server_url/_open/auth with credentials.
So, to my understanding the endpoint of the arangodb web interface is what is to be given to make that rest call.

Hi,
you need to specify all URLs of the coordinators.
You should know that clusters are for when your machine out-does the storage size of one machine, or if you need more js-computing power for foxx. If you need HA, a single server failover setup is better.

Thanks @dothebart and @dilipvamsi. So the list of servers here is the list of all coordinators. What is HA?
I srill have a few questions for you regarding the loop for getting the auth key:

Do you need to ask all the coordinators for the auth key or is one enough?
Why do you randomize the list of servers before looping, this action is only done once.

HA = High Availability

@tariqdaouda I've written most of the go-driver and from there I can give you some points regarding the intended fail-over behavior.
All requests are send only to a single specific coordinator. The fail-over part is only relevant when a coordinator cannot be reached. The go-driver implements that as follows:
It tries to send the request to the coordinator and sets a flag as soon as the OS network stack has begun sending the first byte of the request. As soon as that flag is set, the request will never be retried (automatically) on another coordinator, because there is just no way to know if the request reached the coordinator and the coordinator processed it. It is possible that the request did not reach the coordinator, but it is also possible that the result did not reach the calling client.
So only if there is a failure or timeout before the first request data is send, will the go-driver retry the request on another coordinator. This is for example the case when no database is listening on the specified address.
Once the go-driver finds a coordinator that gives a proper response, it will remember that coordinator and start using it for future requests.
Now besides fail-over there is also load-balancing (not yet implemented in the go-driver). If implemented by the client (there are other possibilities), the client should not use the same coordinator for every request, but either rotate requests over all of them (in a circular fashion) or just pick a coordinator randomly for each request. When rotating, it will help if every client randomizes its list of coordinators, so the coordinator selection has a higher chance or actually spreading the load over all coordinators.
To get the addresses (URLs) of all coordinators, you can either configure all of them in your client or ask 1 of the coordinator to fetch all endpoints (GET /_api/cluster/endpoints)
Hope this helps. If you have further questions, please ask.

One more addition. When doing requests that return cursor (/_api/cursor), you must always use the same coordinator for followup calls.

Thanks, @ewoutp. This helps a lot.

Hi @dilipvamsi and @tariqdaouda,
Just sending a quick followup, are you all set with this ticket?
Can we go ahead and close it, or you prefer to leave it open for now?
Thanks,

I haven't had the time to implement all of it yet, but it will be done. I think it's better to leave it open for now in case other questions arise.

Hi @tariqdaouda
Sure, we will leave this ticket open
Yours,

Hi @tariqdaouda,
how are things going? Are you making progress with your implementation?

Hi @maxkernbach,
I started the implementation and I have some ideas on how to complete it. But unfortunately, I am extremely busy at the moment.

Sure, take your time. Please just let us know when you have some news.

Will do thank you for asking.

Hi @dothebart ,
While running the sample code to print the system db for the new version of the arangodb (3.3.14), I'm getting connection error.
Sample Code
import json as json_mod
import pyArango.connection
from gevent import monkey
from gevent import GreenletExit
monkey.patch_all()
import gevent
import grequests
import requests

connection_urls = ["http://<IP_ADDRESS>:8529"]
auth_token = None

USERNAME = "root"
PASSWORD = "password"

class JWTAuth(requests.auth.AuthBase):
    def __init__(self, token):
        self.token = token

    def __call__(self, r):
        # Implement JWT authentication
        r.headers['Authorization'] = 'Bearer %s' % self.token
        return r

def get_auth_token():
    global auth_token, connection_urls

    if auth_token:
        return auth_token

    kwargs = {'data': '{"username":"%s","password":"%s"}' % (USERNAME, PASSWORD)}
    for connection_url in connection_urls:
        response = requests.post('%s/_open/auth' % connection_url, **kwargs)
        if response.ok:
            json_data = response.content
            if json_data:
                data_dict = json_mod.loads(json_data)
                auth_token = data_dict.get('jwt')
                break
    print(auth_token)
    return auth_token

class AikidoSession(object):
    def __init__(self, session_username, session_password):
        if session_username:
            self.auth = JWTAuth(session_password)
        else:
            self.auth = None

    def post(self, url, data=None, json=None, **kwargs):
        if data is not None:
            kwargs['data'] = data
        if json is not None:
            kwargs['json'] = json

        kwargs['auth'] = self.auth
        return grequests.map([grequests.post(url, **kwargs)])[0]

    def get(self, url, **kwargs):
        kwargs['auth'] = self.auth
        result = grequests.map([grequests.get(url, **kwargs)])[0]
        return result

    def put(self, url, data=None, **kwargs):
        if data is not None:
            kwargs['data'] = data
        kwargs['auth'] = self.auth
        return grequests.map([grequests.put(url, **kwargs)])[0]

    def head(self, url, **kwargs):
        kwargs['auth'] = self.auth
        return grequests.map([grequests.put(url, **kwargs)])[0]

    def options(self, url, **kwargs):
        kwargs['auth'] = self.auth
        return grequests.map([grequests.options(url, **kwargs)])[0]

    def patch(self, url, data=None, **kwargs):
        if data is not None:
            kwargs['data'] = data
        kwargs['auth'] = self.auth
        return grequests.map([grequests.patch(url, **kwargs)])[0]

    def delete(self, url, **kwargs):
        kwargs['auth'] = self.auth
        return grequests.map([grequests.delete(url, **kwargs)])[0]

    def disconnect(self):
        pass

# Monkey patch the connection object:
pyArango.connection.AikidoSession = AikidoSession

conncetion = pyArango.connection.Connection(
    username=USERNAME,
    password=get_auth_token()
)

print(conncetion['_system'].collections)

Stack Trace
ConnectionError                           Traceback (most recent call last)
~/test/pyArango/sample.py in <module>()
     91 conncetion = pyArango.connection.Connection(
     92     username="root",
---> 93     password=get_auth_token()
     94 )
     95

~/venv/lib/python3.6/site-packages/pyArango/connection.py in __init__(self, arangoURL, username, password, verbose)
    103
    104         self.users = Users(self)
--> 105         self.reload()
    106
    107     def disconnectSession(self) :

~/venv/lib/python3.6/site-packages/pyArango/connection.py in reload(self)
    129                     self.databases[dbName] = DBHandle(self, dbName)
    130         else :
--> 131             raise ConnectionError(data["errorMessage"], self.databasesURL, r.status_code, r.content)
    132
    133     def createDatabase(self, name, **dbArgs) :

ConnectionError: not authorized to execute this request. URL: http://<IP_ADDRESS>:8529/_api/user/root/database, status: 401. Errors: b'{"error":true,"errorNum":11,"errorMessage":"not authorized to execute this request","code":401}'

wrong passvoid?

No I'm able to connect if I'm not using the monkey patching.

I'd use ngrep -Wbyline -d lo port 8529 to check out whats going on on the wire, all the rest is just guesswork, like maybe your user doesn't have access to a specific database or so?

Hi @dothebart
The sample code is working fine for locally running arangodb.
But somehow it is throwing the error for arangodb which is hosted in cloud.
Both have same credentials.

maybe you've got troubles with the JWT (if)? sniffing and checking whats in the http headers should bring you more information whats going on.

It is able to get some jwt token
But it is not getting authenticated.

@dothebart ,
I'm actually not passing the arangoURL and that is causing the issue.

I've pushed an implementation to this branch. Works for me besides a few:
ResourceWarning: unclosed
f any of you guys could give it a try and test it, it would great.

ok, tested; I see the grequests aikidosession being used. Now to some serious 3.4 hammering ;-)

Hi @tariqdaouda @dothebart ,
For longer connections if the acquired auth token got expired, the token has to be reset. I think that is only part that is missing.
Regards,
Dilip Vamsi.

Hi @tariqdaouda ,
Use Queue instead of the currentURLId (url index) for round robin. As Queues in python are thread safe and performs better round robin ( pop front and push it to the back).
Regards,
Dilip Vamsi.

For longer connections if the acquired auth token got expired, the token has to be reset. I think that is only part that is missing.

@dothebart any suggestions for implementations.
Thanks,
Dilip Vamsi.

I recently created the ability to use bulk requests with pyArango.
With the availability of ArangoDB 3.7 I'm closing this as solved.


Image not pointing at an image causing 'Disposed'

It sometimes happens a site lists an image that isn't even an image at all. A ghost page, so to say.
While technically correct, we should probably ignore obviously broken pages.
Thoughts, @Mike43110 @XellossMetallium?

Excuse the close and reopen. I am very good at pressing close instead of comment.
Those broken pages make CRA sometimes crash, so I would be quite pleased with this change. No longer having to guess what page to jump to!
So +1 on ignoring obviously broken pages.
http://mangafox.me/manga/naruto/v72/c700/29.html
is an example

Never experienced that on Batoto.. and I have just a couple of series on mangafox. So there is no problem during the upload, is just an error on the server side where an additional inexistent page is added without impact on the flow of the story?
Surely I agree. If the broken pages can even make Comicrack crash, it would be better to cut them out from the beginning.

It crashes android only.
It does occur on both batoto and kissmanga as well, quite a pain!
It's a server side error so there isn't anything that can be done other than dropping the dead page really.

Oh, ComicRackAndroid, I see.. I missed the meaning of the A even if I use the app myself :smile:
Well, I'm in the process of moving my entire on-going library to mangarack (I am still at letter O), so for now I'm only reading volumes I compiled myself on android; probably this is why I still haven't experienced yet the crashes.

It's not exactly the kind of half-back page you've sparsely encountered, @Mike43110. Instead it is all the more related to incorrect implementations of the scanlation site algorithm to index archives. For example, on MangaFox, the pages seem to be indexed based on the maximum number in the archive. if there are pages named 24-25.jpg, for example, then there are not enough pages for the listing, and that would end up in a broken reader state. That's basically what I am fixing here.


Add new console frontend to initial model selection, and other model mgmt improvements

Major Changes
The invokeai-configure script has now been refactored. The work of selecting and downloading initial models at install time is now done by a script named invokeai-model-install (module name is ldm.invoke.config.model_install)
Screen 1 - adjust startup options:

Screen 2 - select SD models:

The calling arguments for invokeai-configure have not changed, so nothing should break. After initializing the root directory, the script calls invokeai-model-install to let the user select the starting models to install.
invokeai-model-install puts up a console GUI with checkboxes to indicate which models to install. It respects the --default_onlyand--yes` arguments so that CI will continue to work. Here are the various effects you can achieve:
invokeai-configure
This will use console-based UI to initialize invokeai.init,
download support models, and choose and download SD models
invokeai-configure --yes
Without activating the GUI, populate invokeai.init with default values,
download support models and download the "recommended" SD models
invokeai-configure --default_only
Activate the GUI for changing init options, but don't show the SD download
form, and automatically download the default SD model (currently SD-1.5)
invokeai-model-install
Select and install models. This can be used to download arbitrary
models from the Internet, install HuggingFace models using their repo_id,
or watch a directory for models to load at startup time
invokeai-model-install --yes
Import the recommended SD models without a GUI
invokeai-model-install --default_only
As above, but only import the default model
Flexible Model Imports
The console GUI allows the user to import arbitrary models into InvokeAI using:

A HuggingFace Repo_id
A URL (http/https/ftp) that points to a checkpoint or safetensors file
A local path on disk pointing to a checkpoint/safetensors file or diffusers directory
A directory to be scanned for all checkpoint/safetensors files to be imported

The UI allows the user to specify multiple models to bulk import. The user can specify whether to import the ckpt/safetensors as-is, or convert to diffusers. The user can also designate a directory to be scanned at startup time for checkpoint/safetensors files.
Backend Changes
To support the model selection GUI PR introduces a new method in ldm.invoke.model_manager called `heuristic_import(). This accepts a string-like object which can be a repo_id, URL, local path or directory. It will figure out what the object is and import it. It interrogates the contents of checkpoint and safetensors files to determine what type of SD model they are -- v1.x, v2.x or v1.x inpainting.
Installer
I am attaching a zip file of the installer if you would like to try the process from end to end.
InvokeAI-installer-v2.3.0.zip

I've decided to turn this into a full-featured interface for importing models from the Internet. I'm turning it into a draft while working out the bugs. Currently it is nonfunctional.

trying this out now. i expect the  == CONVERT IMPORTED MODELS INTO DIFFUSERS == toggle will confuse people - is it possible for it to simply not appear if i'm not importing anything?
also, can the == IMPORT LOCAL AND REMOTE MODELS == text entry box be smaller? it really only needs to be 1 line i think..

above was fixable with mkdir -p /root/invokeai/configs && touch /root/invokeai/configs/models.yaml.
i deselected all the default models, entered a HF repo id in the text entry box and hit DONE and it exited immediately. and /root/invokeai/configs/models.yaml is empty. that's not good.

oh, i see i'm supposed to run invokeai-configure rather than invokeai-model-download. hmm.

on a fresh vast.ai box if i run invokeai-configure, deselect all the default models and only enter a huggingface repo id in the text box, i end up with an empty models.yaml.
even with everything deselected, a lot of stuff gets downloaded, including a 1.7GB CLIP model that i don't need because it comes packaged with the diffusers repo id, and a bunch of face gen stuff that i have never used and likely will never use. is there any way to be smarter about this and/or make it an option?

Thanks for the thorough review. I'll fix the problems you found - that models.yaml doesn't get created unless you download at least one starter model and the formatting issues.
I won't fix the CLIP model downloading at this time because this is necessary for direct ckpt support, and a user might choose to import a ckpt at some point later on. CLIP downloading will vanish when we rip out the ckpt support completely.

Sorry for the late review. I had tried to review a few days ago but I couldn't get it to run; I think you were in the middle of changes.
First impression - a thing of beauty!
Running the script, I had a few hiccups:

Before running the script, I had dreamlike-diffusion-1.0 and dreamlike-photoreal-2.0 downloaded but not in my models.yaml. I had ft-mse-improved-autoencoder-840000 downloaded and present in my models.yaml.

None of these were picked up as installed - I suppose the script looks at models.yaml to see what is installed.
After selecting dreamlike-diffusion-1.0 and autoencoder-840000, the script detected they were downloaded correctly and added them to my models.yaml. However, I now have two nearly identical entries for the VAE: ft-mse-improved-autoencoder-840000 and now autoencoder-840000, pointing to the same ckpt. Only difference is the name.
I did not select dreamlike-photoreal-2.0 to be installed, and it did not show up in my models.yaml after the script. I would expect the script to pick up everything previously installed by InvokeAI.

At the end of the first run, my invokeai.init file had not changed. After the second and subsequent runs, it had changed to match my selections...
...almost - no matter what I select for convert to diffusers, nothing changes in invokeai.init
Finally, the setting the NSFW box enabled or disabled does change invokeai.init, but it always defaults to enabled when running the script, even if it is disabled in invokeai.init

This is a very nice user experience for a shell script, great work! Love it.

trying this out now. i expect the  == CONVERT IMPORTED MODELS INTO DIFFUSERS == toggle will confuse people - is it possible for it to simply not appear if i'm not importing anything?
also, can the == IMPORT LOCAL AND REMOTE MODELS == text entry box be smaller? it really only needs to be 1 line i think..

Changes implemented:

The convert toggle is now conditional and only appears when there are models to import or when the "autoscan" directory is set.
I have made the text entry box smaller. However, I want to make it possible for people to download multiple models at a time, and so I made it 5 lines high rather than 1 line as suggested

Sorry for the late review. I had tried to review a few days ago but I couldn't get it to run; I think you were in the middle of changes.
First impression - a thing of beauty!
Running the script, I had a few hiccups:

Before running the script, I had dreamlike-diffusion-1.0 and dreamlike-photoreal-2.0 downloaded but not in my models.yaml. I had ft-mse-improved-autoencoder-840000 downloaded and present in my models.yaml.

None of these were picked up as installed - I suppose the script looks at models.yaml to see what is installed.

The script only works off of what's in models.yaml because in fact you can have models installed anywhere in your filesystem. I'll fix it in the future so that the invokeai/models directory is scanned if that's the desired behavior.

After selecting dreamlike-diffusion-1.0 and autoencoder-840000, the script detected they were downloaded correctly and added them to my models.yaml. However, I now have two nearly identical entries for the VAE: ft-mse-improved-autoencoder-840000 and now autoencoder-840000, pointing to the same ckpt. Only difference is the name.

The autoencoder is no longer an option to install. Both the legacy and the diffusers versions are now installed behind the scenes for use in later model importation.

I did not select dreamlike-photoreal-2.0 to be installed, and it did not show up in my models.yaml after the script. I would expect the script to pick up everything previously installed by InvokeAI.

See above. The question is what to do when the user has a models directory that contains model files that are not in models.yaml. Do they want to import these back into models.yaml, or was there a reason they removed them in the first place.

At the end of the first run, my invokeai.init file had not changed. After the second and subsequent runs, it had changed to match my selections...
...almost - no matter what I select for convert to diffusers, nothing changes in invokeai.init
Finally, the setting the NSFW box enabled or disabled does change invokeai.init, but it always defaults to enabled when running the script, even if it is disabled in invokeai.init

You found a bug. Fixed.

This is a very nice user experience for a shell script, great work! Love it.

Very high praise coming from you! Thanks.

oh, i see i'm supposed to run invokeai-configure rather than invokeai-model-download. hmm.

on a fresh vast.ai box if i run invokeai-configure, deselect all the default models and only enter a huggingface repo id in the text box, i end up with an empty models.yaml.
even with everything deselected, a lot of stuff gets downloaded, including a 1.7GB CLIP model that i don't need because it comes packaged with the diffusers repo id, and a bunch of face gen stuff that i have never used and likely will never use. is there any way to be smarter about this and/or make it an option?

I have fixed the issue of empty models.yaml files when no recommended starter files are selected. Thanks for detecting it.

@psychedelicious @damian0815  Thank you for your reviews. I have addressed your concerns and would welcome another round of reviews.

above was fixable with mkdir -p /root/invokeai/configs && touch /root/invokeai/configs/models.yaml.

This should no longer be necessary. If the model installer finds that there is no root configs or models directory, it will launch the invokeai-configure command in order to set up the root directory properly. The model installer will then pick up where it left off.

i deselected all the default models, entered a HF repo id in the text entry box and hit DONE and it exited immediately. and /root/invokeai/configs/models.yaml is empty. that's not good.

This bug is fixed.

Also, somehow while testing this I ended up with a wonky invokeai.init file:
# InvokeAI initialization file
# This is the InvokeAI initialization file, which contains command-line default values.
# Feel free to edit. If anything goes wrong, you can re-initialize this file by deleting
# or renaming it and then running invokeai-configure again.
# the --outdir option controls the default location of image files.
# generation arguments
# You may place other  frequently-used startup commands here, one or more per line.
# Examples:
# --web --host=<IP_ADDRESS>
# --steps=20
# -Ak_euler_a -C10.0
#
--no-ckpt_convert
--no-ckpt_convert
--no-ckpt_convert
--no-ckpt_convert
--no-ckpt_convert
--no-ckpt_convert

--outdir=/home/bat/invokeai/outputs
--embedding_path=/home/bat/invokeai/embeddings
--precision=auto
--max_loaded_models=2
--no-nsfw_checker
--xformers
--no-ckpt_convert


Comparision to postcss-modules solution

I just came across this repo: https://github.com/outpunk/postcss-modules
I'm curious as how these two solutions compare to each other.
Are they trying to achieve the same goal?

Hi, currently I figured out that postcss-modules is another postcss plugin, which contains all the necessary packages for the CSS Modules.
I think it's similar to the css-modules-loader-core, but wrapped in the postcss-plugin, so you'll have a usual API. It can be used as a low-level module to build tools similar to require hook or you can use it to generate json files with tokens for the further usage with a non-JS libraries.
Also I noticed an issue in the posthtml, so I believe that it will be used for another plugin :)

For me postcss-modules is just a easy way to use CSS Modules for PHP or Ruby.
I found this tool is extremly complicated. So it can’t be used by CSS guys.

Hey @ai can you expand on how you use postcss-modules with PHP & Ruby? I'm really interested in that at the moment! :)

@geelen postcss-modules provides a callback to save a JSON object with classes for each CSS Module. You can save the returned JSON into separate files next to CSS when you compiling your assets and read these JSON files from PHP/Ruby/whatever code. Like so:
- styles = JSON.read('front/blocks/logo.json')
div(class=styles['logo'])
  div(class=styles['title'])

@ai looks like rumors tell truth :)
Anyways, I think postcss-modules provides a good API as another plugin and I like the idea about global comments for the :global selector.
I know another person from the PHP world, who was interested in CSS Modules and he talked about similar implementation, like @outpunk mentioned.

I guess we learnt the difference :)


RPi 3 audio and hdmi rotation possibly affected by 2.41.0 RPi 4 updates

From support interactions:

This is something from the kernel update apparently: https://github.com/raspberrypi/linux/issues/3181
If you use the balena web dashboard, go to “Device Configuration” section, then modify the existing RESIN_HOST_CONFIG_dtoverlay variable to “vc4-fkms-v3d” then audio will be working again.
It seems we need to also update the RPi firmware.

[pdcastro] This issue has attached support thread https://jel.ly.fish/#/support-thread~97afaaf1-b0d7-46fa-9110-564ac24c7d65

@pdcastro this should not be the case anymore. Please have a test of the latest 2.43 release and close this if the issue is fixed

A forum user has confirmed that balenaOS 2.43 fixes the issue:
https://forums.balena.io/t/raspberry-pi-headphone-jack-no-longer-works-with-latest-os/27667/27?u=pdcastro


WebAssembly support

Because it's an even more popular thing to do
Provide WASM version for color utils, quantizers and other hotspots for performance boost.

Considering this project is using Typescript already converting some parts to webassembly using https://github.com/AssemblyScript/assemblyscript shouldn't be too hard.

This is super facinating to see. I'll have to take a closer look when I get more time @minht11 . As of now I'm going to be unable to look further into it due to personal obligations


Create dateTime util function to convert epoch seconds to milliseconds

Create dateTime util function to convert epoch seconds to milliseconds
for fromNowStartedOn and fromNowEndsOn in src/components/tasks/card/index.tsx and src/components/pullRequests/index.tsx.

@palakgupta2712 I would like to take this issue, please assign me


MouseTracker Improvements

Exposed secondary pointer button (middle, right, etc.) events from MouseTracker and through viewer (#479)
Improved IE 8 compatibility

Looks good. Might be worth somehow clarifying that the "aux" handlers are for all non-primary buttons, not just the "aux/middle" button.

Might be worth somehow clarifying that the "aux" handlers are for all non-primary buttons, not just the "aux/middle" button.

@iangilman Where? I thought I did that...

Fair enough. I guess I'm thinking that it could be confusing to use the same name in both cases.  On the other hand, it's a rare enough use case maybe we shouldn't worry about it. If it turns out there's confusion, we can make a finer point of it.

@iangilman Renamed!

grr last commit not good enough to fix that bug. Fixing...

Fixed.

Looks good, except the unit test failures.

Ok, stable again. Now see if I can add a fix... :)

@iangilman
Didn't change the pointer events shim! Should be okay now.

Cool; looking good.
One thought for the future: maybe we could start to consolidate the IE8 special cases (and refactor as needed to make it possible), so that when we're ready to break IE8 support out as a plugin, it's already more or less ready.

Agreed. I looked at making the plugin starting with MouseTracker and realized how much was sealed/private that I'd have to copy so I put it off pending other fixes, particularly the iframe issue (ugh).


When specified "editor.wordHighlightBackground", word selection highlight color is not displayed or is faintly displayed.

VSCode Version: 1.23
OS Version: Windows 7 Enterprise LTS

Easiest Steps to Reproduce:

In any JavaScript file, click on a word that is commonly used, like try, catch, for, or null. Somehow, let, var, =>, and function cannot be highlighted. (Separate issue I guess)
Once the highlighted word is lit up, hold Shift key, and press Left Arrow or Right Arrow keys to start selecting the letters in the highlighted word.
Observe it's difficult to see the "editor.selectionHighlightBackground" color.

Does this issue occur when all extensions are disabled?: Yes
Expected results:
Customized user settings that specified "editor.selectionHighlightBackground" should override whatever selection highlighting colors in the VS Code, and should highlight the selection based on the user settings even when "editor.wordHighlightBackground" is kicking in.
Or something that would at least make word selection more visible while the word highlighting color is displayed.
Actual results:

You can hardly see what I'm selecting when the word highlight color is shown / lit up.
My User Settings:
{
	"beautify.config": "C:\\workspace\\.vscode\\.jsbeautifyrc",
	"bracketPairColorizer.forceUniqueOpeningColor": true,
	"bracketPairColorizer.timeOut": 0,
	"diffEditor.renderSideBySide": false,
	"editor.cursorStyle": "line-thin",
	"editor.detectIndentation": false,
	"editor.fontFamily": "Consolas, 'DejaVu Sans Mono', 'MingLiU', 'Meiryo', monospace",
	"editor.fontSize": 11,
	"editor.insertSpaces": false,
	"editor.renderWhitespace": "all",
	"editor.tabSize": 3,
	"eslint.options": {
		"rules": {
			"requireParamDescription": false,
			"requireReturnDescription": false
		}
	},
	"html.format.indentInnerHtml": true,
	"typescript.experimental.syntaxFolding": true,
	"window.restoreWindows": "all",
	"workbench.colorCustomizations": {
		"editor.selectionBackground": "#14a",
		"editor.selectionHighlightBackground": "#733",
		"editor.wordHighlightBackground": "#660",
		"editorCursor.foreground": "#fff"
	},
	"workbench.startupEditor": "newUntitledFile",
}

Always use transparency for selection and highlight colors. That way you can distinguish overlapping ranges.
I've documented that in the hover description for the impacted colors as well as in the color spec.

@aeschli I used CTRL+F to search for transparency, but there are only 2 places mentioning about transparency at: sideBar.dropBackground and panel.dropBackground. None of the editor related options mentioned anything about transparency.
And also, the entire article never mentioned if Visual Studio Code uses ARGB, or RGBA color scheme format. I don't know what format to use?

The colors that should have transparency say: The color must not be opaque to not hide underlying decorations.
I'll add that the supported color formats are #RGB, #RGBA, #RRGGBB and #RRGGBBAA

Thanks @aeschli .


Please invite me to the GitHub Community Organization

Name
gitakileus
Discord Username (if applicable)
akileus#5916
Additional Context
Please send the invitation

It's great having you contribute to this project
If you would like to continue contributing to open source and would like to do it with an awesome inclusive community, you should join our Discord Server and our GitHub Organisation - we help and encourage each other to contribute to open source little and often 🤓 . Any questions let us know.

Invitation sent for the GitHub Organisation. Welcome to the community 🎉#
Don't forget after accepting to make it public so it appears on your GitHub profile for everyone else to see, you can do this by finding your name in the GitHub organisation list and change the dropdown to public https://github.com/orgs/jaidevstudio/people
Like this

Tips for practising:
Remember contributing to open source is not just about code, its about collaboration, communication and adding value
I hope that helps

Please check your email or organisation page 🙏


Plugin port change upon restart

Hi,
Looks like the plugin port changes each time Homebridge restarts. The Nuki Bridge only supports 3 registered callback adresses, so each time the plugin changes its port a new callback registration is made. Once at 3 registrations the plugin fails to register with the following error messages:
subscribe to event notifications
error: too many callbacks registered
Any way to use a fixed port for the plugin which would remain the same upon restarts ?
Thanks.

No, because it's not needed.
How do you stop homebridge?  Homebridge NB removes the callback when shut down gracefully.  See #9.


Need short alias for --config-location

Checklist

[X] I'm reporting a feature request
[X] I've looked through the README
[X] I've verified that I'm running yt-dlp version 2022.04.08 (update instructions) or later (specify commit)
[X] I've searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue

Description
--config-location is excessively long especially if one uses it more than once. even using it only once it is still longer than some thing the config might be holding.
Ex:
yt-dlp --config-location MusicAlbums.conf <url>
vs
ytdlp -x --audio-format mp3 <url>

so something like -CL <config-location> would shorten it and make it much less annoying for short but less memorable strings
Verbose log
No response

Option names can be shortened as long as there is no ambiguity. So --config works

well why not -conf or -cf/-cl does -config really make it all that much less ambiguous?
heck --ignore-warnings is shortened to -i ...

-cl is better

I agree with --config

Both of you seem to have misunderstood my comment. I am saying you can shorten any option as long as yt-dlp can uniquely match it to an existing option. So you can already reduce --config-locations to --config-location, --config-loc, --config or even --conf
Anyway, I am also working on a feature that lets you define custom aliases: https://github.com/pukkandan/yt-dlp-dev/tree/features/alias

Ahh, I just interpreted that as you were just saying it could be done in the code, not that I could do it by just typing in less :P  Thanks for that, I'll go try that out

yt-dlp --conf
Usage: yt-dlp [OPTIONS] URL [URL...]

yt-dlp: error: --config-locations option requires 1 argument

Yep it works fine, thanks again

yt-dlp --conf
Usage: yt-dlp [OPTIONS] URL [URL...]

yt-dlp: error: --config-locations option requires 1 argument

Yep it works fine, thanks again

How to fix this error?

It's not an error, you need to specify a config file after --conf/-config-location as yt-dlp isn't a mind reader


New quote on home page every time page is loaded again

The homepage can have a new quote every time it is loaded on screen, like when one opens the website one time, it will show up a quote, upon refreshing it, a new quote will appear

I have submitted the pull request, thank you, I'll be ready to help to any of other open-source projects if needed (kindly provide repository link)


Descending order for year list on Events page

On http://beta.theorangealliance.org/events, should the year dropdown be in descending order instead of ascending? Typically, I think people want to see current events more often than past ones, and the year dropdown on team pages is in descending order.

This has been resolved


Gedis javascript code generation

Check that the javascript code generation still works as expected and can be queried from vuejs
Child of threefoldtech/home#162

The javascript client is not generated anymore, we use a generic client.
Docs for using the gedis js client:
https://github.com/threefoldtech/digitalmeX/blob/development/docs/webserver/websocket-gedis.md

Verified and some changes have been done .. Commit: https://github.com/threefoldtech/digitalmeX/commit/7495abdef6833c34ed3fcbf479a4cf04593fa870


fix query for windows (999 vars limit)

closes #1331

Now seeing:
2018-11-07 09:57:33,628 ERROR    lbrynet.extras.daemon.Component:61: Error setting up file_manager
Traceback (most recent call last):
  File "lbrynet\extras\daemon\Component.py", line 55, in _setup
  File "twisted\internet\defer.py", line 1416, in _inlineCallbacks
  File "twisted\python\failure.py", line 491, in throwExceptionIntoGenerator
  File "lbrynet\extras\daemon\Components.py", line 582, in start
  File "twisted\internet\defer.py", line 1416, in _inlineCallbacks
  File "twisted\python\failure.py", line 491, in throwExceptionIntoGenerator
  File "lbrynet\blob\EncryptedFileManager.py", line 47, in setup
  File "twisted\internet\defer.py", line 1416, in _inlineCallbacks
  File "twisted\python\failure.py", line 491, in throwExceptionIntoGenerator
  File "lbrynet\blob\EncryptedFileManager.py", line 121, in _start_lbry_files
  File "twisted\internet\defer.py", line 1416, in _inlineCallbacks
  File "twisted\python\failure.py", line 491, in throwExceptionIntoGenerator
  File "lbrynet\extras\daemon\storage.py", line 782, in get_claims_from_stream_hashes
  File "twisted\python\threadpool.py", line 250, in inContext
  File "twisted\python\threadpool.py", line 266, in <lambda>
  File "twisted\python\context.py", line 122, in callWithContext
  File "twisted\python\context.py", line 85, in callWithContext
  File "twisted\enterprise\adbapi.py", line 475, in _runInteraction
  File "twisted\python\compat.py", line 463, in reraise
  File "twisted\enterprise\adbapi.py", line 465, in _runInteraction
  File "lbrynet\extras\daemon\storage.py", line 779, in _batch_get_claim
KeyError: 'b1a269aae821d36620b000b017cdff4b85a8b3f89099b5a671d84283bdd603a49872052c3078c9a6b502d857d74a99b0'

Working now, woohoo!


Add padding as object option to fitBounds

This adds an option to set the padding as an object with top, bottom, right, and left properties to set different padding for each side when calling the fitBounds method. This is useful if you have UI overlaying on one side of the map and you want to have the bounds fill up the rest of the available area.
The api is similar to the way Mapbox-gl-js's fitBounds method works.
I couldn't find a good example of how this kind of object would be documented, so I ended up describing it inline. I hope that's clear enough, but if there's another way you'd prefer, I'll update accordingly.
Thanks for a great set of tools around mapping and let me know if there's anything I can improve!

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

This would be a great improvement, I will be including these changes in my project until they (hopefully) merge, thanks @GunnarSturla

Nevermind, this doesn't seem to work as advertised. Adding 100 does not actually pad it 100px on screen on that side it actually moves the map all over the world.

@ibgreen Thanks for your comments, I've fixed the issues you brought up
On the other hand, whoa! @luskin is right, I'm shifting the center of viewport by 100 "pixels" on the Web Mercator plane, which is not what I had intended. I will have to convert the "screen pixels" to "plane pixels" before shifting the center.
I'll have to look at it tomorrow, but thanks @luskin for trying it out and letting me know of the problem!

@GunnarSturla looking forward to a patch, with more and more people using floating side bars/cards over maps it seems like a necessary feature going forward.

Thanks for your help @Pessimistress, I've been trying to figure this thing out, but I haven't had too much time to wrap my head around it all, so it's taken its time.
lateralOffset and verticalOffset* are screen pixels (the difference between opposite paddings), and if I understand this correctly, nw and se are the locations of the corners of the bounding box on the Web Mercator plane (in pixels or units). It took me a while to figure out what scaleX and scaleY were, but it looks like it's screen pixels per unit on the Web Mercator plane. I've used that scale to convert the 'screen pixels' to 'Web Mercator pixels', and offset the center correctly.
All the calculations are done using the viewport set on line 54, at zoom level 0. Even though the pixel/unit values might be different at different zoom level, it's just used to figure out where the center of the map should be, and since it's then unprojected to LngLat, it should all cancel out.

I've changed the variable names for verticalOffset and lateralOffset to offsetX and offsetY in order to match the scaleX/Y names.

@GunnarSturla No problem, I appreciate the contribution.
A few other things I realized during review, not necessarily need to be addressed in this PR:

We haven't been checking if padding is larger than the viewport size. It will be nice if an error is thrown here rather than failing to render later.
There needs to be some sanity check for the test cases (either checking projected corner positions or compare with Mapbox)

Heads up: it looks like the documentation that this PR added to fitBounds is not yet showing up on the GH pages documentation site. I was thinking of making an issue to ask if I could add the feature before I checked the source. Might be time to re-run the site generator?
(In any event, I'm a big fan of these libraries - thank you maintainers!)


starting connectors from API

There shall be method for Status check and Start Stop Connectors in API.
Rdgs
Anshu

Did you check the stats manager through jcli ?

JCLI works. I am concerned here about web-api .

The web api does not expose stats ? it's a third party project.

pl advice how to do this. Or We can have some http URL where jasmin could expose these values via JSON or XML

Then you need to pull out a branch of that web api project, contribute to it and push a merge request ;)


[C# 13]: New Feature -

Visual Studio release
17.9p3
Link to speclet

Speclet

Link to Pull request

PR

New behavior

Areas affected

[X] Add to What's new in C# 13: Add links to updated docs in the What's new in C# 13 article.
[X] Update language reference: Detail the areas in the language reference that will be updated for this feature.
[X] Update compiler messages: Add new (or updated) pages for new compiler messages. Include related message.
[X] Include in Everyday C# section*: Is this new feature included in Everyday C#?
[X] Tutorials: Are new tutorials required for this feature? If so, detail below

Language reference tasks
No response
Tutorial tasks
No response
Everyday C# tasks
No response

closing as spam


moderation: added lock mechanism.

closes https://github.com/inveniosoftware/invenio-users-resources/issues/102
NEEDS:

https://github.com/inveniosoftware/invenio-cache/pull/23

Tests are failing since it needs the PR linked above.
Locally tests are passing:
Name                                                         Stmts   Miss  Cover   Missing
------------------------------------------------------------------------------------------
invenio_users_resources/__init__.py                              3      0   100%
invenio_users_resources/administration/__init__.py               1      1     0%   9
invenio_users_resources/administration/views/__init__.py         1      1     0%   9
invenio_users_resources/administration/views/users.py           35     35     0%   9-69
invenio_users_resources/config.py                               12      0   100%
invenio_users_resources/entity_resolvers.py                     74     36    51%   35-42, 46-50, 54, 58-59, 63-79, 93, 97, 101, 105, 114-120, 124-126, 130-131, 146, 150, 154, 158
invenio_users_resources/ext.py                                  49      4    92%   104-107
invenio_users_resources/forms.py                                15     15     0%   9-41
invenio_users_resources/models.py                               11     11     0%   9-39
invenio_users_resources/notifications/__init__.py                1      0   100%
invenio_users_resources/notifications/filters.py                 9      0   100%
invenio_users_resources/notifications/generators.py             11      0   100%
invenio_users_resources/permissions.py                           4      0   100%
invenio_users_resources/proxies.py                               8      0   100%
invenio_users_resources/records/__init__.py                      3      0   100%
invenio_users_resources/records/api.py                         133     23    83%   112, 117, 125-129, 144-149, 153, 235-242, 248-251, 270, 280
invenio_users_resources/records/dumpers/__init__.py              3      0   100%
invenio_users_resources/records/dumpers/email.py                16      0   100%
invenio_users_resources/records/hooks.py                        31      5    84%   36, 41-44
invenio_users_resources/records/mappings/__init__.py             1      0   100%
invenio_users_resources/records/mappings/os-v1/__init__.py       1      1     0%   9
invenio_users_resources/records/mappings/os-v2/__init__.py       1      1     0%   9
invenio_users_resources/records/mappings/v7/__init__.py          1      0   100%
invenio_users_resources/records/models.py                       66      6    91%   37, 67, 73, 78, 87, 96
invenio_users_resources/resources/__init__.py                    4      0   100%
invenio_users_resources/resources/groups/__init__.py             4      0   100%
invenio_users_resources/resources/groups/config.py              11      0   100%
invenio_users_resources/resources/groups/resource.py            25      4    84%   44-49, 55-59
invenio_users_resources/resources/users/__init__.py              4      0   100%
invenio_users_resources/resources/users/config.py                9      0   100%
invenio_users_resources/resources/users/resource.py             27      3    89%   32, 48-53
invenio_users_resources/services/__init__.py                     4      0   100%
invenio_users_resources/services/common.py                       6      0   100%
invenio_users_resources/services/decorators.py                  33      4    88%   35, 41-43
invenio_users_resources/services/generators.py                  57      8    86%   32, 69-74, 101, 110
invenio_users_resources/services/groups/__init__.py              4      0   100%
invenio_users_resources/services/groups/config.py               31      0   100%
invenio_users_resources/services/groups/results.py              58     23    60%   28-34, 39, 43, 48, 53, 58-72, 77, 81-84, 126
invenio_users_resources/services/groups/service.py              27     12    56%   29-41, 48, 54-58
invenio_users_resources/services/groups/tasks.py                22      4    82%   27-28, 38-39
invenio_users_resources/services/params.py                      10      0   100%
invenio_users_resources/services/permissions.py                 23      0   100%
invenio_users_resources/services/results.py                     26      0   100%
invenio_users_resources/services/schemas.py                     47      0   100%
invenio_users_resources/services/users/__init__.py               4      0   100%
invenio_users_resources/services/users/config.py                29      0   100%
invenio_users_resources/services/users/results.py               58      5    91%   39, 43, 77, 83, 126
invenio_users_resources/services/users/service.py              107     14    87%   50, 55-79, 118, 124-125, 134, 140-142
invenio_users_resources/services/users/tasks.py                 42      4    90%   52-53, 63-64
invenio_users_resources/views.py                                19      0   100%
------------------------------------------------------------------------------------------
TOTAL                                                         1181    220    81%

====================================================================================================================================== 62 passed, 15 skipped, 6008 warnings in 26.64s ======================================================================================================================================

WDYT?
See also comments in the other PR. Let me know what you think, happy to chat IRL.

I agree that these decorators added a HUGE overhead, it was not worth it.
I ended up dropping them and just having a wrapper of CachedMutex for user moderation. The code looks much cleaner now, thanks!
Example:

def block(...):
  ...

  self.require_permission(identity, "manage", record=user)

  # Throws ``LockAcquireFailed`` if not acquired
  ModerationMutex(id_).acquire()

  ...


(mini.sessions) Telescope extension

Contributing guidelines

[X] I have read CONTRIBUTING.md
[X] I have read CODE_OF_CONDUCT.md

Module(s)
mini.sessions
Description
Consider implementing Telescope extension which allows to list and read session files.

Hi I made nvim session picker that works with mini.sessions  (it works with all nvim session made with mksession).
Link: https://github.com/JoseConseco/telescope_sessions_picker.nvim
Its very basic - only allows to open session file ( `:so  session_path'). Should work with most  session managers.

Thanks for the pointer! I'll look at it, when solving #24.

I opted into creating MiniSessions.select() which utilizes vim.ui.select() (initially designed to be overridable). To use Telescope for picking sessions, consider stevearc/dressing.nvim plugin.


Suggestion: font color on light themes

Hi,
The dark themes look awesome, thanks for those! In my opinion the text in the light themes give me the impression that the entities are disabled and a bit difficult to read.

My suggestion would be to make them a bit darker so they are more readable, what are your thoughts? :)

Hi,
The dark themes look awesome, thanks for those! In my opinion the text in the light themes give me the impression that the entities are disabled and a bit difficult to read.

My suggestion would be to make them a bit darker so they are more readable, what are your thoughts? :)

What do you think about this?

That looks way better I think! I'd be happy to give it a spin :-)

That looks way better I think! I'd be happy to give it a spin :-)

Update! I have already changed and published a new release. And pay me a coffee hahahahaha =)

I'll give it a go! Thanks!


(WIP) V2: New terminology and structure

(WIP, DO NOT MERGE, can be rebased at any time)
This is version 2 of the new terminology and structure, as outlined in #172 and supersedes #272 . The changes will be made in several parts (as individual commits in this PR):

[x] Introduce new directory structure and rename source files and build products.
[x] Rename internal symbols and APIs.
[x] Sundry changes (e.g. mention of ukvm in license headers, etc.)
[x] Rename OPAM packaging. Note: OPAM installation will not be functional until this step is completed and MirageOS dependencies are updated
[x] Documentation updates to reflect changes made.

See individual commits for a detailed description of changes.
Changes from V1 (#272):

The ukvm target is now named hvt (hardware virtualised tender) rather than just vt.
ukvm_guest.h is renamed to hvt_abi.h (a much better name), moved to include/solo5 with adjustments to the build system to remove the use of a symlink to enable sharing this file between the hvt bindings and tender.
Minor corrections to overzealous renaming in V1 (_build-solo5-hvt vs. _build_solo5-vt).
Where the use of hvt would cause internal names to read hvt_hv_X, the extra hv_ has been dropped. Local use of struct hvt *hv in tender code has been changed to struct hvt *hvt to match (see c67d0effad for details).
Minor fixes to OPAM install targets (remove redundant use of include/).

/cc @ricarkol @djwillia

This is all ready to merge and I will do so in the next ~12 hours.
@adamsteen Unfortunately there appears to be some kind of transition going on in OpenBSD-6.4-current involving libc++, and thus I can't actually pkg_add the dependencies needed to build and test this change on a freshly-installed OpenBSD-current snapshot. If you have such a system around, would you mind testing this? If I don't hear from you by morning then I'll just merge and we'll deal with any needed fixes later. All I need is whether or not Solo5 (NOT MirageOS downstreams) builds and the tests pass.

@mato I can run gmake on an OpenBSD 6.4 successfully, it compiles solo5 and tests.  cannot execute tests though (missing hardware features in an emulated system)

@hannesm Thanks very much for confirming that, that's good enough for now.
@adamsteen No need for you to do anything right now, ignore my previous request.

@mato Couldn't help my self! OpenBSD compiles, and all tests succeed.


If matrix is invertible

Is your feature request related to a problem? Please describe.
Will help in finding the inverse of the matrix and a key linear algebra functionality
Describe the solution you'd like
Integer returning value 0,1

I would like to work on this!


💡 idea: Support for reading attached files

Description
It would be nice if the attached files to a message/prompt could be read by the model.
Currently, only images seem to be exposed to the model/API.

I'm surprised this isn't done by default, chatgpt spews nonsense if you upload a file currently so I think this is a bug as well as a FR.
Se4n01
[8:23 AM]

can you transcribe the text in this text file?
test.txtTXT26B
Chat GPT
BOT
[8:23 AM]

I'm sorry, but I am unable to transcribe the text from an image file at the moment. If you can provide the text directly or in a different format, I'd be happy to help with the transcription.

@azigler


Tooltip blank for all filter items

Is this valid for all apps?
Any filter options that are not blank - for testing?

This should be valid for all apps. There should not be any "blank", unless the filteroption is named "blank".
Here is a better description https://github.com/equinor/fusion-workspace/pull/439

Tested and verified.
Comment: What is the value of a hover only stating the value you are hovering over - except when value is too long in the screen?

I dont think the value can be too long. Do you have a case where this happens?


Can I choose the name of a collection of a subscription?

I have a two subscriptions 'courses.enrolled' and 'courses.available' that I want to subscribe to. I'm trying to display the contents of these subscriptions in a MeteorListView but i'm having trouble disambiguating the two subscriptions. If I make a MeteorListView with  'collection=courses.enrolled' or 'collection=courses.available' the collection is not found. Though it seems that both subscriptions are accessible through collection='courses'.
Is there a way to choose the name for the collection of a subscription?

Hi @vantassell
There isn't really a direct relationship between a subscription and a collection in the way that you're thinking. A subscription can handle documents from multiple different collections, and multiple different subscriptions can publish the same documents.
You need to use the selector prop of the MeteorListView to pass in a MongoDB query to separate your documents.
Eg.
<MeteorListView
  collection="courses"
  selector={{enrolled: true}}
  renderRow={this.renderItem}
/>

@eawall That makes sense, I knew i was missing something. Thank you! :)


Add CMake file

Adding a cmake file to this template brings it inline with the updates to the godot-cpp repository.
This is ongoing work to raise the cmake solution to have feature parity.
This PR is not ready to be merged, I just wanted to get it started.
Related godot-cpp PR's
https://github.com/godotengine/godot-cpp/pull/1595
https://github.com/godotengine/godot-cpp/pull/1598

Since this this cmake solution tracks PR#1598 I have in the godot-cpp repo, until it is merged it requires the godot-cpp module be pointed to the right revision. Locally I've been using this to make that happen:
git submodule set-url -- godot-cpp https://github.com/enetheru/godot-cpp.git
git submodule set-branch -b modernise godot-cpp
git submodule sync
git submodule update --init --recursive --remote


fork next-web,how to change API-KEY in my project?

Check vercel dashboard -> settings -> environment variables

tks a lot


the deal

Type: Feature Request
Context

Managed Dependencies: on
Cloud SDK Version: 399.0.0
Skaffold Version: v1.39.1
Minikube Version: 1.26.1

VS Code version: Code 1.70.2 (e4503b30fc78200f846c62cf8091b76ff5547662, 2022-08-16T05:35:13.448Z)
OS version: Windows_NT x64 10.0.22622
Modes:

@DocBRob, could you please provide a description for the feature request so that we can route it to the correct owner?

This issue has been closed automatically because it needs more information and has not had recent activity. See also our issue reporting guidelines.
Happy Coding!


investigate saarland empty routes

I just tried osmtogtfs on Saarland and found out that several files are empty :(
Do you know what might be the cause?
saarland.zip

my mistake !!
I totally forgot that you put the option to integrate dummy data or not. However that made me realize something: Would it be possible to print the config of osmtogtfs before it is executed?
Since the terminal description only says [OPTIONS] it is not possible to know which values were applied as default which can definitely be useful for a beginner :)

my mistake !!

No problem!

Would it be possible to print the config of osmtogtfs before it is executed?

Yes of course! I think I had put the default values in the docs of each option. I'll have a second look.

Would it be possible to print the config of osmtogtfs before it is executed?

If you use osmtogtfs --help, it will show the default values that will be used (inside the docs).
To make it more explicit, I added some more logging to the output. You have to append --loglevel DEBUG to see them.


Add checklist, remove toolchain, update generators and update demo links

Codecov Report

Merging #31 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #31   +/-   ##
=======================================
  Coverage   96.22%   96.22%
=======================================
  Files           2        2
  Lines          53       53
  Branches        5        5
=======================================
  Hits           51       51
  Misses          2        2

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bfcc4a9...9150ffc. Read the comment docs.


ApiUrl watcher fix + Data watcher

@fabiorizzello Thanks for the PR, but can you please state the purpose of this fix and how it's solved your problem.

@ratiw
See #110 for the api-url fix
I've added a watcher for the data prop because i want to bind my data to the table data.
This way if i remove an element i can do an ajax call to my apis, then i can manually remove the element from my data then the deletion will be reflected to the table data thanks to the watcher.
It can be a deletion like an insert or edit.
Actually after this pull request i've seen that the data property is not fully implemented yet. For example the sort doesn't work. But i think it's still a work in progress right?

@fabiorizzello Right now, the data property is quite limited in its functionality. The server related functions like sorting, searching/filtering, and pagination are still not supported on the client side. And I have NO plan to add it any time soon.

@fabiorizzello To prevent confusion, I'll have to close this PR for the time being. Sorry for this. But I do really appreciate your contribution.


Removes False-y values from action_run body

jgaff reported that when using the SDK's ActionClient against the [now old] Datacite AP, he would get errors back indicating that  monitor_by and manage_by cannot be None. What happened is that the SDK would use None for monitor_by and manage_by if no value was supplied, which resulted in action-run requests being sent with those fields set to None.  At the AP, these action-run requests would be rejected bc the ActionRequest schema expects monitor_by and manage_by to be lists if they're set.
The fix is to simply remove False'y values from the action-run body before it get's sent to an AP. Because monitor_by and manage_by are optional, this will allow users to use the SDK's ActionClient without specifying values for those args.

The proposed change look good. I hadn't thought about it but request_id and body could be empty/Falsey and the action should still go through. It's funny since I spent a while thinking about why "body" should be optional on the CLI and here I am making assumptions that it's not (:
Let me validate that empty lists are ok before merging them

Everything checks out! Good proposal


Added Gitpod to third-party tools

Gitpod comes with TSLint support.

You can try it on this PR:

Also thank you for this great project. We use it in every TS project. :heart:

Thanks for your interest in palantir/tslint, @svenefftinge! Before we can accept your pull request, you need to sign our contributor license agreement - just visit https://cla.palantir.com/ and follow the instructions. Once you sign, I'll automatically update this pull request.

👋 @svenefftinge, thanks for this! I hadn't heard of Gitpod till now.
Applying it to this PR: https://d0e51df3-9704-4186-b16a-828ad2e33078.ws-us.gitpod.io, there are some file casing problems from yarn run test:rules within yarn run verify:

Looks like a bug in Gitpod? Before considering adding it to this list, it should work.

Good catch! We need to support git clone flags --config core.autocrlf=input --config core.eol=lf.
We will fix it and ping you here when it's fixed/supported. Thank you! 👍
Ticket: https://github.com/gitpod-io/gitpod/issues/152

It is fixed now. The .gitpod.yml configuration for your project is here:
https://github.com/gitpod-io/definitely-gp/blob/master/tslint/.gitpod.yml
You can of course alternatively check it into your repo :-)


Add 'tweet_mode' parameter to getTimeline()

by default, it is 'compat' (compatibility mode). But if you want to get extended tweets you have to add 'tweetMode: .extended' to the getTimeline() method:
swifter.getHomeTimeline(count: 20, tweetMode:.extended, success:      { statuses in         print(statuses)     }, failure: failureHandler)
https://developer.twitter.com/en/docs/tweets/tweet-updates

sorry for the mess with commits :(


Reducing the complexity and computational time with new logic

Describe the bug
Under the nodeCanvasObject method of the Force graph there is a logic which adjusted the font size of text such that the font size is maximum without overflowing out of the node (shown in screenshot). This logic uses a do while loop. However a constant time solution exists (without any loop) Since the method is called for each node , using the constant time solution can reduce the computational time.
Also the font size calculated from current algorithm is not so accurate as compared to the proposed logic(As shown in screenshot).
Expected behavior
A simple, fast and more accurate constant time solution is possible which can reduce the computational time.
Screenshots
Screenshot of the current logic

Screenshot of website with the current logic implementation(see the font size of coveralia)

Screenshot of website with the proposed logic run locally on my machine(see the font size of coveralia)

Desktop (please complete the following information)

OS: windows 10
Browser chrome
Version 80.0

Extra Info
I had already tested the new logic locally so I request you to please assign the issue to me

@kgodey  @soccerdroid  @mathemancer  please assign the issue to me


How to reconnect a client automatically?

I want to reconnect to the server.
I found an example on ratchetphp.
It doesn't work for ReactPHP:
$connector = new \React\Socket\Connector($this->loop);
        $connector
            ->connect(sprintf('%s:%s', $host, $port))
            ->then(
                function (\React\Socket\ConnectionInterface $conn)
                {
                    $conn->on('close', function()
                    {
                        echo "close\n";
                        $this->loop->addTimer(10.0, function () {

                            // RECONNECT HERE
                            $this->connector
                               ->connect('<IP_ADDRESS>:8061')
                               ->then( function (\Exception $e)
                               {
                               });

                        });
                    });
                }
       );

How to reconnect a client automatically?

@wolrd The gist of this looks about this: on(close) => connect()
Are you having any issues with this code? It's unclear what problem you're seeing and where the variables come from and what events are actually fired in your particular setup.
That being said, it looks like your question might be out of scope for this project. If you're new to the concept of sockets, I would suggest redirecting this question at Stackoverflow.
I hope this helps :+1:

@clue thanks. This is the answer.