MongoDB 5.0.6 Compatibility Issues

Hey,
firstup please tell me what additional information is needed to debug this error.
I ran the following command:
mongo-connector --unique-key=id -n news-articles.articles -m localhost:27017 -t http://localhost:8983/solr/mongo_solr_collection -d solr_doc_manager
(Yes i am calling the collection with -t because the core does not work (I get an Error 404). )
The connector starts and logs the following:
2022-02-15 18:23:34,584 [ALWAYS] mongo_connector.connector:50 - Python version: 3.9.2 (default, Feb 28 2021, 17:03:44)
[GCC 10.2.1 20210110]
2022-02-15 18:23:34,586 [ALWAYS] mongo_connector.connector:50 - Platform: Linux-5.10.0-11-amd64-x86_64-with-glibc2.31
2022-02-15 18:23:34,587 [ALWAYS] mongo_connector.connector:50 - pymongo version: 4.0.1
2022-02-15 18:23:34,587 [WARNING] mongo_connector.connector:170 - MongoConnector: Can't find /srv/news_crawler/oplog.timestamp, attempting to create an empty progress log
2022-02-15 18:23:34,597 [ALWAYS] mongo_connector.connector:50 - Source MongoDB version: 5.0.6
2022-02-15 18:23:34,597 [ALWAYS] mongo_connector.connector:50 - Target DocManager: mongo_connector.doc_managers.solr_doc_manager version: 0.1.0
2022-02-15 18:25:33,715 [ERROR] mongo_connector.util:97 - Call to Collection(Database(MongoClient(host=['localhost:27017'], document_class=dict, tz_aware=False, connect=True, replicaset='rs0'), 'news-articles'), 'collection_names') failed too many times in retry_until_ok
Traceback (most recent call last):
  File "/srv/news_crawler/venv/lib/python3.9/site-packages/mongo_connector/util.py", line 79, in retry_until_ok
    return func(*args, **kwargs)
  File "/srv/news_crawler/venv/lib/python3.9/site-packages/pymongo/collection.py", line 2579, in __call__
    raise TypeError("'Collection' object is not callable. If you "
TypeError: 'Collection' object is not callable. If you meant to call the 'collection_names' method on a 'Database' object it is failing because no such method exists.

I don't know what to do with this. MongoDB is up and running:
Current Mongosh Log ID:	620be3f843ef12a095c537e4
Connecting to:		mongodb://<IP_ADDRESS>:27017/?directConnection=true&serverSelectionTimeoutMS=2000&appName=mongosh+1.1.9
Using MongoDB:		5.0.6
Using Mongosh:		1.1.9
For mongosh info see: https://docs.mongodb.com/mongodb-shell/
   The server generated these startup warnings when booting:
   2022-02-15T17:19:31.349+01:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem
   2022-02-15T17:19:32.383+01:00: Access control is not enabled for the database. Read and write access to data and configuration is unrestricted
   2022-02-15T17:19:32.384+01:00: /sys/kernel/mm/transparent_hugepage/enabled is 'always'. We suggest setting it to 'never'
Warning: Found ~/.mongorc.js, but not ~/.mongoshrc.js. ~/.mongorc.js will not be loaded.
  You may want to copy or rename ~/.mongorc.js to ~/.mongoshrc.js.
rs0 [direct: primary] test>

and the designated collection has several thousand items in it:
rs0 [direct: primary] news-articles> db.articles.find().count()
147203

Is rs0 setup wrong?
And solr is also up and running:
$ curl http://localhost:8983/solr/admin/cores?action=STATUS
{
  "responseHeader":{
    "status":0,
    "QTime":0},
  "initFailures":{},
  "status":{
    "mongo_solr_collection":{
      "name":"mongo_solr_collection",
      "instanceDir":"/srv/solr-8.11.1/server/solr/mongo_solr_collection",
      "dataDir":"/srv/solr-8.11.1/server/solr/mongo_solr_collection/data/",
      "config":"solrconfig.xml",
      "schema":"managed-schema",
      "startTime":"2022-02-15T15:09:55.153Z",
      "uptime":9084767,
      "index":{
        "numDocs":0,
        "maxDoc":0,
        "deletedDocs":0,
        "indexHeapUsageBytes":0,
        "version":2,
        "segmentCount":0,
        "current":true,
        "hasDeletions":false,
<EMAIL_ADDRESS>lockFactory=org.apache.lucene.store.NativeFSLockFactory@5d83b5e6; maxCacheMB=48.0 maxMergeSizeMB=4.0)",
        "segmentsFile":"segments_1",
        "segmentsFileSizeInBytes":69,
        "userData":{},
        "sizeInBytes":69,
        "size":"69 bytes"}}}}

I really don't know what I am doing wrong. Would be great if someone could point me in the right direction. 🙏

For now I think pymongo and mongodb are the problem. I am using mongodb 5.0.6 and tried different pymongo versions (3.6->3.12) because I saw this comment in another issue #916.
The method collection_names was definitely available in 3.5 according to the 3.5-docs, but I still get the same error.

Hi @bablf I got the same error as you with MongoDB, i have found a workaround by modifying the solr_doc_manager.py
just change this at line 292
                batch = list(next(cleaned) for i in range(self.chunk_size))
                while batch:
                    self.solr.add(batch, **add_kwargs)
                    batch = list(next(cleaned)
                                for i in range(self.chunk_size))

by this
            try:
                batch = list(next(cleaned) for i in range(self.chunk_size))
                while batch:
                    self.solr.add(batch, **add_kwargs)
                    batch = list(next(cleaned)
                                for i in range(self.chunk_size))
            except Exception:
                pass

downgrade the pysolr to 3.8.1
pymongo to 2.9
this workaround works for MongoDB 5.0.6 with mongo-connector 3.1.1, pymongo 2.9 and pysolr 3.8.1
Hope it helps you

This helped me. Thank you.
Cant install pymongo 2.9 so I used 3.12.3

@RicardoM17 did you find solution to your problem. I'm experiencing the problem where mongo connector can successfully add and update the document to Solr instance but it failing to update the document.  Version 4.x of mongodb used to work fine.
My setup
Mongodb: 5.0.9
Solr: 9.0.0

@RicardoM17 did you find solution to your problem. I'm experiencing the problem where mongo connector can successfully add and update the document to Solr instance but it failing to update the document. Version 4.x of mongodb used to work fine.
My setup Mongodb: 5.0.9 Solr: 9.0.0

Hi @gurmitteotia I don't exactly remember but I believe what happened was the following. I was testing a new feature on my personal account. This worked because when I first started I was still in version 4.X.X in MongoDB. Then they rolled out the update to 5.X.X which broke my stuff. That being said I was already going to implement this in my company's MongoDB account which has 4.2.22. So it is working there. I can't exactly remember if that was what "fixed" it or if I did something else but I believe that to be the case.
I guess if you use a paid account you can still specify the version of MongoDB you want.
Sorry if I couldn't be of more help.

Thanks @RicardoM17 ,
I did investigation and found out reason why document update was failing. I have also published a fix.
Primary reason is that "oplog" entry format has been changed in mongodb 5.x for document update. In Mongodb 4.x following oplog entry is generated when fields are changed -
{
	"op" : "u",
	"ns" : "mydb.emps",
	"ui" : UUID("79dc60c3-2fa9-454f-a84e-8b6d694d4ad3"),
	"o" : {
		"$v" : 1,
		"$set" : {
			"company_name" : "new company"
		}
	},
	"o2" : {
		"_id" : ObjectId("6319e61bc84d0826deef55e8")
	},
	"ts" : Timestamp(1662641739, 1),
	"t" : NumberLong(1),
	"wall" : ISODate("2022-09-08T12:55:39.544Z"),
	"v" : NumberLong(2)
}

However in case of mongodb 5 oplog entry has been changed to the following format-
{
	"op" : "u",
	"ns" : "jobs.live.jobs",
	"ui" : UUID("beebd3ac-fc25-41c9-98cf-90c87bfe79e0"),
	"o" : {
		"$v" : 2,
		"diff" : {
			"u" : {
				"company_name" : "new name",
				"website_url" : "https://www.newname.com1"
			}
		}
	},
	"o2" : {
		"_id" : ObjectId("5fa1ab5c184246d62af8fc46")
	},
	"ts" : Timestamp(1662644253, 1),
	"t" : NumberLong(4),
	"v" : NumberLong(2),
	"wall" : ISODate("2022-09-08T13:37:33.238Z")
}

And solr-doc-manager plug-in was misinterpreting the "update" document and adding it as new document, which was being rejected by solr. I have put a fix in solr-doc-manager plugin. You can find it in branch - mongo5-fix. I have added unit test to make sure the fix work for both mongodb 4 and mongodb 5. I will try if my fix can be reviewed and merged in the main repository.
Summary
The last released versions of mongo-connector (3.1.1) and solr-doc-manager (0.1.0) will not work with latest version of mongodb (5.0.9) and solr (9.0.0). You will encounter two issues-

Pymongo error as originally reported by @bablf. Fix is to install pymongo version 3.12.3
Update of document will fail. Solution is described above.


Create manual-go.yml

What type of PR is this?

What this PR does / why we need it:
Which issue(s) this PR fixes:

Fixes #
Special notes for your reviewer:
Specified Reivewers:
/assgin @your-reviewer
Does this PR introduce a user-facing change?

Additional documentation usage docs, etc.:

111

111


[WIP] Add type hints to connector-packager

This PR adds type hints to all function & method defs in the connector-packager library. The addition of these type hints will allow type errors to be raised during development, if your editor is set up correctly. If you use VS Code, Pyright will allow you to take advantage of type hints. PyCharm should do so automatically.
Example definition & raised type errors:

Tests are passing on the PR as of 72b0d60 but we should poke at it to make sure everything's OK.

Thanks for updating the PR.

In the file connector-packager/connector_packager/jar_jdk_packager.py:

Line 51:55: E226 missing whitespace around arithmetic operator
Line 51:32: E226 missing whitespace around arithmetic operator

I rerequested review since (a) there have been some big changes and (b) this was a pretty stale PR


Replace lokiexporter to otlpexporter

part or #40

blocked by #41

ingest endpoint
http://localhost:3100/otlp (http://localhost:3100/otlp/v1/logs)
guide
https://grafana.com/docs/loki/latest/send-data/otel/#configure-the-opentelemetry-collector-to-write-logs-into-loki
source
https://github.com/grafana/loki/blob/19fef9355fdd46911611dbec25df0f5a4e397d31/pkg/loki/modules.go#L370
TESTS
✅OK
curl -H 'Content-Type: application/json' -X POST -v -s 'http://localhost:3100/otlp/v1/logs' --data-raw '{
...
}'

# *   Trying ::1...
# * TCP_NODELAY set
# * Connected to localhost (::1) port 3100 (#0)
# > POST /otlp/v1/logs HTTP/1.1
# > Host: localhost:3100
# > User-Agent: curl/7.61.1
# > Accept: */*
# > Content-Type: application/json
# > Content-Length: 2844
# > Expect: 100-continue
# >
# < HTTP/1.1 100 Continue
# * We are completely uploaded and fine
# < HTTP/1.1 204 No Content
# < Date: Wed, 15 May 2024 09:55:17 GMT
# <
# * Connection #0 to host localhost left intact

see: https://github.com/open-telemetry/opentelemetry-proto/blob/main/examples/logs.json
{
  "resourceLogs": [
    {
      "resource": {
        "attributes": [
          {
            "key": "service.name",
            "value": {
              "stringValue": "rolldice"
            }
          },
          {
            "key": "job",
            "value": {
              "stringValue": "rolldice"
            }
          }
        ]
      },
      "scopeLogs": [
        {
          "scope": {
            "name": "my.library",
            "version": "1.0.0",
            "attributes": [
              {
                "key": "my.scope.attribute",
                "value": {
                  "stringValue": "some scope attribute"
                }
              }
            ]
          },
          "logRecords": [
            {
              "timeUnixNano": "1715766897500000111",
              "observedTimeUnixNano": "1715766897500000111",
              "severityNumber": 10,
              "severityText": "Information",
              "traceId": "c29e156486ccf3d30dde0491b4429d23",
              "spanId": "92e790ce6a007caa",
              "body": {
                "stringValue": "Example log record"
              },
              "attributes": [
                {
                  "key": "string.attribute",
                  "value": {
                    "stringValue": "some string"
                  }
                },
                {
                  "key": "boolean.attribute",
                  "value": {
                    "boolValue": true
                  }
                },
                {
                  "key": "int.attribute",
                  "value": {
                    "intValue": "10"
                  }
                },
                {
                  "key": "double.attribute",
                  "value": {
                    "doubleValue": 637.704
                  }
                },
                {
                  "key": "array.attribute",
                  "value": {
                    "arrayValue": {
                      "values": [
                        {
                          "stringValue": "many"
                        },
                        {
                          "stringValue": "values"
                        }
                      ]
                    }
                  }
                },
                {
                  "key": "map.attribute",
                  "value": {
                    "kvlistValue": {
                      "values": [
                        {
                          "key": "some.map.key",
                          "value": {
                            "stringValue": "some value"
                          }
                        }
                      ]
                    }
                  }
                }
              ]
            }
          ]
        }
      ]
    }
  ]
}

send logs to opentelemetry-collector endpoint
curl -H 'Content-Type: application/json' -X POST -v -s 'http://localhost:4318/v1/logs' --data-raw '{
...
}'

# * Trying ::1...
# * TCP_NODELAY set
# * Connected to localhost (::1) port 4318 (#0)
# > POST /v1/logs HTTP/1.1
# > Host: localhost:4318
# > User-Agent: curl/7.61.1
# > Accept: */*
# > Content-Type: application/json
# > Content-Length: 2843
# > Expect: 100-continue
# >
# < HTTP/1.1 100 Continue
# * We are completely uploaded and fine
# < HTTP/1.1 200 OK
# < Content-Type: application/json
# < Date: Wed, 15 May 2024 14:25:56 GMT
# < Content-Length: 21
# <
# * Connection #0 to host localhost left intact
# {"partialSuccess":{}}


Add @builder decorator as alternative to build() classmethod

The linter is failing in CI but runs fine locally, not sure why.

Obsoleted by #235 - We won't have build methods at all once this work comes in.


Only one level of includes is allowed

1. This is my project structure.
terragrunt-project
  - enviroment
     - ec2
        - terragrunt.hcl
     - s3
        - terragrunt.hcl
     - terragrunt.hcl
  - module
     - ec2
        - main.tf
        - outputs.tf
        - variables.tf

2. This is root terragrunt.hcl
generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite_terragrunt"
  contents  = <<EOF
provider "aws" {
  region = "ap-northeast-2"
}
EOF
}

remote_state {
  backend = "s3"
  config = {
    encrypt        = true
    bucket         = "terragrunt-test"
    key            = "${path_relative_to_include()}/terraform.tfstate"
    region         = "ap-northeast-2"
    dynamodb_table = "terragrunt-locks"
  }
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
}

3. This is terragrunt.hcl in ec2 folder
terraform {
  source = "../../module/ec2"
}

inputs = {
  subnet_id = "subnet-2d11c946"
}

# Include all settings from the root terragrunt.hcl file
include {
  path = find_in_parent_folders()
}

I called "terragrunt apply-all" under the environment folder.
Initially, it worked well and created an EC2 instance. However, the second execution results in the error as follows.
Error processing module at 'C:/Users/abc/PycharmProjects/terragrunt-study/enviroment/ec2/.terragrunt-cache/atTOD5LBQHAAgxKMQddh6R4j
XUE/mW46bbJv_XuxaHj-jku8hlgoUlQ/terragrunt.hcl'. How this module was found: Terragrunt config file found in a subdirectory of C:/Users/abc/PycharmProjects/terragrun
t-study/enviroment. Underlying error: C:/Users/abc/PycharmProjects/terragrunt-study/enviroment/ec2/.terragrunt-cache/atTOD5LBQHAAgxKMQddh6R4jXUE/mW46bbJv_XuxaHj-jku
8hlgoUlQ/terragrunt.hcl includes C:/Users/abc/PycharmProjects/terragrunt-study/enviroment/ec2/terragrunt.hcl, which itself includes C:/Users/abc/PycharmProj
ects/terragrunt-study/enviroment/ec2/terragrunt.hcl. Only one level of includes is allowed.

What's interesting is that Linux or Mac doesn't have a problem, but it only happens with Windows 10.
My terragrunt version is v0.24.3 and terraform version is v0.13.2

How weird. Must be some sort of path issue on Windows. We don't have CI set up for Windows yet 😕
If someone has time to dig into this bug, a PR with a fix is very welcome!

I am facing the same problem when plan was executed on independently (instance leve) then next time plan-all executed from higher level (region)

I also hit this issue today on Windows. An initial plan-all worked, but then once the terragrunt cache had been populated further plan-all actions failed with the same error that @nedaair reports.

Same issue, on windows 10. First plan-all or apply-all succeeds, but subsequent ones fail with @nedaair 's message. It seems to be resolving my generated terraform.hcl in the cache directories, and getting mad about it.

@brikis98 I've started digging, and it seems that the config module tests don't succeed without signfiicant update. Having CircleCI run windows builds would ensure these tests pass. Your CI tool supports Windows builds...is that an option?

Yes, CircleCi does now support Windows images, so I suspect it's possible. However, I'm not sure when our team will have time to get to it and work out all the kinks with making the CI build pass. If anyone is interested, PRs are very welcome! https://circleci.com/docs/2.0/hello-world-windows/

Also facing the same issue on Win10

I was able to test and confirm that this issue appears to be caused because the .terragrunt-cache folders are being included and processed by *-all on subsequent runs.   I've been able to work around it by adding this to my command line:
 --terragrunt-download-dir c:\terragrunt-cache
It would be highly valuable to get this fixed, because it's far too easy to trip over on Windows and there's no intuitive way to figure out WHY it's giving you a hard time.   Worse when your colleague is running on Linux having no issues with the same project :)

This seems to be solved by https://github.com/gruntwork-io/terragrunt/releases/tag/v0.26.4.

I tried 0.26.4 windows_amd64 build and it's still broken.

hey guys. do we have any progress on this issue? It is still happening on windows 10 at version 0.35

happening to me on v0.36.1 as well

same in Windows 11 (terragrunt version v0.53.5)

Same here terragrunt version v0.53.6 windows 11, used in mingw64 bash shell

Same issue, using Windows Terminal and PowerShell on Win 11.  If I delete the cache folders, the error does not occur on the first "run-all plan", but does recur on subsequent runs.  Moving the cache to a different folder does prevent the error, but adds to the command line (a config setting would be cool).
Additionally, if I just use terragrunt plan, I get a different error:
level=warning msg=Error removing manifest file C:\(redacted)\.terragrunt-cache\(redacted)\.terragrunt-module-manifest: remove C:\(redacted)\.terragrunt-cache\(redacted)\.terragrunt-module-manifest: The process cannot access the file because it is being used by another process.
I can delete the cache subfolders in Windows explorer without issue, despite what the error says.  This seems like something peculiar to the way Windows handles the cache folders.


intermittent tcp problem

I used "tcp://server:port/myCommand" to forward commands to my custom tcp server. This works fine for a while and then stops working for a while. When it does not work, it seems ha-bridge opens and closes the tcp connection without sending any data. To check that the problem was not on my end, I also opened a tcp port using "nc" on a different linux device but found the same problem. I then thought I might have misunderstood something about ha-bridge, so I configured it additionally to make an http GET request (right after the tcp command) and I found that, when the problem happens with tcp, the http request still looks fine. So I think ha-bridge has a problem with tcp.
As I would like to use it with a microcontroller, simple tcp would be preferable to http.
I'm running ha-bridge on a raspberry pi 3b+ with the latest rasbian os.

So, how long does it work for before it stops?

It's pretty irregular, it may or may not work to start with. When I tried again recently it mostly did not work but then occasionally worked once or twice in a row.

I should add that the behavior is the same whether I issue the command through Alexa or use the test buttons in the ui. When using the buttons, I see a green "ok" message and there is nothing strange in the log, irrespective of whether the tcp communication worked or not.

So far, I cannot replicate this in my test environment. So What version of the ha-bridge are you using?

Thanks for trying. I cloned the repo, built with mvn, and installed as described in the readme.

Try the RC4 jar in the releases and see how that helps


ERROR: No matching distribution found for tensorflow

Hello guys,
I am trying to install Tensorflow but its giving me this error.

I have Python V 3.9, Pip version 20.2
Python x64, Windows 10 x64
What I have already Tried:
I have tried creating Virtual environment using the following steps,

which was successfully created...
also tried these...

pip install --upgrade tensorflow``
and
pip install https://storage.googleapis.com/tensorflow/windows/gpu/tensorflow_gpu-2.3.0-cp38-cp38-win_amd64.whl`
(This link was given @ Tensorflow web for Python v3.8, 3.9 not available)
I have already tried doing all these stuff with python v 3.8
My Paths are also set in Environment Variables
I have tried almost all suggestion provided on Stackoverflow and Github and even uninstalled and reinstalled Python, still
Any help will be highly appreciated!

@Zayn1101
please confirm if you have verified the compatibility issues and hardware requirements, please refer to this issue and let us know : #42367

same problem

@TheWiesmin
please conform if you have referred to the above comment.


Programmatic lookup improvements

Both the reference implementation (Weld) and ArC have an enhanced version of javax.enterprise.inject.Instance that provide features that might be worth adding to the spec.

Enhancement
Description

Instance#getHandler() (name TBC)
Allows the client to obtain a "contextual reference handler" which not only holds the contextual reference but also makes it possible to inspect the metadata (i.e. Bean<?>) of the relevant bean and to destroy the underlying contextual instance.

Instance#handlers() (name TBC)
Returns an Iterable of handlers described above.

select() method that accepts java.lang.reflect.Type
This allows for generic selection of instances which can be handy while dealing with third party beans through extensions. However, it's not type-safe.

Caching the result of Instance#get()
See https://quarkus.io/guides/cdi-reference#caching-the-result-of-programmatic-lookup

See also Weld docs about WeldInstance -  Enhanced version of jakarta.enterprise.inject.Instance.

I am especially +1 for getHandler and handlers() as that allows you to iterate over bean metadata without creating bean instances.
The latter two I'd personally keep outside of spec, but if people want it, we can add it as well. I just haven't seen such demand and/or usage of those two.

OK so the Handler (I think Quarkus calls it Handle? That's a little nicer IMHO) allows:

creating the bean instance (contextual reference, if you will);
obtaining the bean metadata (Bean);
destroying the bean instance.

Out of these 3 operations, 2 of them are already supported by Instance: you can create the bean instance by Instance.get and Instance.iterator, and you can destroy the bean instance by Instance.destroy.
I see you can get bean metadata from BeanManager, but creating the bean instance is ugly in such case. So yea, after looking at the current API, I think adding something like Handle is nice, as it lets you avoid the nasty low-level API on BeanManager and let you do everything through Instance.
I guess it would have been possible to design Instance differently to allow metadata access, but since that wasn't done and we certainly don't want to do breaking changes on that part of the API, something like a Handle seems necessary. So +1 from me.

Out of these 3 operations, 2 of them are already supported by Instance: you can create the bean instance by Instance.get and Instance.iterator, and you can destroy the bean instance by Instance.destroy.

The main point of Handle is really the ability to browse beans of certain type before creating their instances. The fact that you can perform get and destroy from there is just a convenience.

I guess it would have been possible to design Instance differently to allow metadata access,

Yea, but like you say, that's breaking. especially because current spec wording requires that iteration over Instance works with contextual references. This is captured in Instance javadoc.

FTR the Weld variant also includes the getPriorityComparator() method that can be used to sort handlers by priority in descending order.

OK so the Handler (I think Quarkus calls it Handle? That's a little nicer IMHO) allows:

The Quarkus variant returns InstanceHandle which is also used in other parts of the API. I also think that Handle is a bit nicer ;-).

The main point of Handle is really the ability to browse beans of certain type before creating their instances. The fact that you can perform get and destroy from there is just a convenience.

I wouldn't call it "just a convenience", because if you getHandles() and filter based on Bean metadata and then want to create an instance, going back to the Instance would require you to 1. remember position, 2. iterate again and rely on iteration order, which is clunky and generally a bad idea. So I'd consider instantiating and destroying essential.
In any case, I just wanted to clarify for myself, and thought perhaps that would be useful for someone else too :-)

FTR this issue was discussed during CDI meeting yesterday.
General consensus was that the Handler feature should be added into specification (e.g. Instance#handlers() and Instance.getHandler()) while the remaining two features got mixed reactions and the decision was to be keep track of them but don't add them to the specification just yet.


Upgrade machine_version to 0.15.0

Proposed changes
Upgrade machine_version to 0.15.0 in _config.yml.
cf. https://github.com/docker/machine/releases
(I checked _config.yml is not listed in .NOT_EDITED_HERE.yaml.)

Closed because duplicated.
https://github.com/docker/docker.github.io/pull/6903


Delete unnecessary spaces

When copying from the web page, there is an unnecessary space after the code.

@changeworld : Thanks for your contribution! The author, @mikejo5000, has been notified to review your proposed change.

@changeworld @Mikejo5000 Mike Jones (DEVUE) FTE - We're in the process of pulling these commits to the private repo, so we can verify the content on staging. We'll post a new PR # shortly and then close this PR.

@changeworld @Mikejo5000 - The commits in this PR were moved to the private repo (PR 3895) for validation and staging. We'll close this PR now.


Extend Download Repo Functionality

At the moment, instructors and tutors can download multiple student solutions (in form of git repos) for the manual correction in the Exercise Scores page (e.g. https://artemis.ase.in.tum.de/#/course/29/exercise/438/dashboard) of programming exercises, see the following screenshot after clicking on Download Repos:

After clicking on Download Repositories, the server clones the repos of all specified students (or pulls them if they already have been cloned, e.g. for the online editor), executes some additional steps (see e.g. the paper Stager: Simplifying the Manual Assessment of Programming Exercises
on http://ceur-ws.org/Vol-2358/paper-03.pdf), creates a zip file for each repository and then creates a zip file for all repos. This is done in ExerciseService.java in the method exportParticipations(...).
This functionality should be extended:

Allow instructors to choose to download all student repos of this exercise instead of manually specifying some students (this functionality should be used in the future for downloading all repos to execute plagiarism checks and here the instructors typically want to download all repos)
Normalize code style: Unify the encoding and line endings of all files in the repository to UTF-8 and unix line endings, see TODO in ExerciseService.java in line 355. This might be tricky and was not implemented yet, but if you google, there should be a lot of options how to accomplish this.
Filter late submissions, see TODOs in `GitService.java in line 380ff: basically the commit hash of the last submission within the exercise due date from the Artemis database should be used here to checkout a specific version of the repository --> Additionally allow the user to specify a custom date instead of the exercise due date in the dialog on the client side (see below)
Clone the repositories into a different folder on the Artemis server and NOT into the one that is already used by Artemis. Otherwise, this might interfere with the usage of the students in the online editor.

The functionality should also be customizable in the future:

Allow the user to choose in the client dialog which of the tasks should be executed with checkboxes and boolean parameters in the corresponding REST call:
a) Filter late submissions with custom date (default is exercise due date)
b) Add student to project name
c) Squash after instructor
d) Normalize code style

In addition, the functionality should be moved into the ProgrammingExerciseService

@thilo-behnke FYI


Document Python+PyTorch version compatibility matrix in README

🚀 Feature
Add a matrix that defines which python versions are supported.
Motivation

Going through the installation guide with python 3.11 leads to errors when installing nvfuser

Error message
Error message depends on which package is used.
The version recommended in Thunder readme:

$ conda create -n thunder3 python=3.11
...
$ conda activate thunder3
$ pip install --pre nvfuser-cu121-torch24
ERROR: Could not find a version that satisfies the requirement nvfuser-cu121-torch24 (from versions: none)
ERROR: No matching distribution found for nvfuser-cu121-torch24

The version recommended in nvfuser readme (i.e. with the extra index flag:

$ pip install --pre 'nvfuser-cu121[torch]' --extra-index-url https://pypi.nvidia.com
Looking in indexes: https://pypi.org/simple, https://pypi.nvidia.com
Collecting nvfuser-cu121[torch]
  Downloading nvfuser_cu121-0.2.10.dev20240819.tar.gz (544 bytes)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [41 lines of output]
      Traceback (most recent call last):
        File "/home/mmikulski/miniconda3/envs/report/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/home/mmikulski/miniconda3/envs/report/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/mmikulski/miniconda3/envs/report/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 152, in prepare_metadata_for_build_wheel
          whl_basename = backend.build_wheel(metadata_directory, config_settings)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-jvsfi906/overlay/lib/python3.11/site-packages/nvidia_stub/buildapi.py", line 29, in build_wheel
          return download_wheel(pathlib.Path(wheel_directory), config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-jvsfi906/overlay/lib/python3.11/site-packages/nvidia_stub/wheel.py", line 175, in download_wheel
          report_install_failure(distribution, version, None)
        File "/tmp/pip-build-env-jvsfi906/overlay/lib/python3.11/site-packages/nvidia_stub/error.py", line 63, in report_install_failure
          raise InstallFailedError(
      nvidia_stub.error.InstallFailedError:
      *******************************************************************************

      The installation of nvfuser-cu121 for version 0.2.10.dev20240819 failed.

      This is a special placeholder package which downloads a real wheel package
      from https://pypi.nvidia.com. If https://pypi.nvidia.com is not reachable, we
      cannot download the real wheel file to install.

      You might try installing this package via
$ pip install --extra-index-url https://pypi.nvidia.com nvfuser-cu121
Here is some debug information about your platform to include in any bug
      report:

      Python Version: CPython 3.11.9
      Operating System: Linux 6.5.0-44-generic
      CPU Architecture: x86_64
      Driver Version: 550.90
      CUDA Version: 12.4

      *******************************************************************************

      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Python 3.10 works fine.
Pitch

Write explicite which versions of python are supported, e.g.:

python
status

3.9
✅

3.10
✅

3.11
coming soon

Alternatives
Add badges with python version at the top of the readme, like here.
Additional context

The Python versions Thunder supports are 3.10, 3.11, 3.12. Not sure what this means for NVFuser.


Some controls don't grab focus when you click on them

Godot version
v4.0.rc2.official [d2699dc7a]
System information
Debain 11 KDE Plasma 5.20.5
Issue description
Most controls like containers and things don't actually grab focus when you click on them with your mouse.
Steps to reproduce

Create tab container
Give it 3 vbox containers
Give each of those vbox containers 3 buttons
Make the tab container have focus mode all
Make the first vbox have focus mode all
Make the focus neighbor bottom on the tab container the first vbox container
Make the first vbox container's neighbor bottom the first button
Make the first button's focus neighbor top the tab container
Run it and click on the first tab and try to move down
See that it doesn't always work

Minimal reproduction project
TabContainerBugs.zip

this is related
https://github.com/godotengine/godot-proposals/issues/6297


Failed to parse parameter value: 29,70389cm  jsreportRuntime.js

Hello,
we use jsreport to generate a pdf. It works fine, but on one machine we are getting this error:
  Error rendering report: A critical error occurred while trying to execute the render command: Failed to parse parameter value: 29,70389cm (1). caused by error (1) -> meta = {"remoteStack":"Error: Failed to parse parameter value: 29,70389cm\n    at assert (jsreportRuntime.js:418725:11)\n    at convertPrintParameterToInches (jsreportRuntime.js:417421:5)\n    at Page.pdf (jsreportRuntime.js:417161:20)\n    at Page.<anonymous> (jsreportRuntime.js:418605:23)\n    at runWithTimeout (jsreportRuntime.js:161246:27)\n    at <anonymous>\n    at process._tickCallback (internal/process/next_tick.js:188:7)"}, stack = Error:     at responseToBuffer ([eval]:68314:29)    at concat ([eval]:68367:40)    at ConcatStream.<anonymous> ([eval]:11565:43)    at emitNone (events.js:111:20)    at ConcatStream.emit (events.js:208:7)    at finishMaybe ([eval]:100191:14)    at afterWrite ([eval]:100053:3)    at _combinedTickCallback (internal/process/next_tick.js:144:20)    at process._tickCallback (internal/process/next_tick.js:180:9)    at jsreport.Local.Internal.LocalUtilityReportingService.RenderAsync(String requestString, CancellationToken ct)    at Services.PDFGeneratorService.GenerateProductionDrawing(String sourcePdfFilePath, String outputPdfsourcePdfFilePath, ProductionDrawingViewModel model, String viewName) in D:\work\server\Services\PDFGeneratorService.cs:line 117    at Services.PrintService.GenerateProductionDrawingPDF(ProductionDrawingViewModel productionDrawingViewModel) in D:\work\server\Services\PrintService.cs:line 270    at Services.PrintService.PrintProductionDrawingAsync(ProductionPlanPrint productionPlanPrint) in D:\work\server\Services\PrintService.cs:line 142
I guess the problem is with decimal separator.
What do you suggest please ?
Thank you very much

I think you have different locale on that failing machine. The easiest is probably to match the windows locale.
Or you can probably do it also in c#
System.Threading.Thread.CurrentThread.CurrentUICulture = CultureInfo.GetCultureInfo("en-US")

Thank you, the windows setting worked.


Added Universidade de Caxias do Sul.

Added Universidade de Caxias do Sul from Caxias do Sul - RS, Brasil.
Site: www.ucs.br

Please remove the additional (non-txt) file.

Files was added in the meantime.


Check if foo.blade.php file_exists for static route before taking /[param] for account?

If we have such dir structure:
/pages
   about.blade.php
   /[Post-slug]
       index.blade.php

navigating to /about should - in my opinion - load about.blade.php instead of looking for a Post with slug = 'about'.
It works correctly if we make /about dir and put index.blade.php inside but wouldn't the above solution be more convenient?

I searched through the closed issues and I've found similar ones with [param].blade.php file inside the same directory as foo.blade.php so the navigation to /foo fails.
Please consider changing the convention because it's quite annoying to have 6 different index.blade.php tabs opened in the editor :)
And thanks for all the work of course!

Hi there,
Thanks for reporting but it looks like this is a question which can be asked on a support channel. Please only use this issue tracker for reporting bugs with the library itself. If you have a question on how to use functionality provided by this repo you can try one of the following channels:

Laracasts Forums
Laravel.io Forums
StackOverflow
Discord
Larachat
IRC

However, this issue will not be locked and everyone is still free to discuss solutions to your problem!
Thanks.

This is clearly not a question for support channel. This is related to the mechanics of the library.
The special case should take priority over the general case. If creating /foo/index.blade.php works then /foo.blade.php should work as well...

@pronode the file checker/loader pipeline works a lot like middleware. You can see an example how to easily customize it for your app it in this (merged) pr:
https://github.com/laravel/folio/pull/113#issue-1924711034

This really feels like a bug to me. The only workaround I've been able to find is by replacing this:
pages/
    about.blade.php
    [Post]/
         index.blade.php

with this:
pages/
    about/
        index.blade.php
    [Post]/
        index.blade.php

With a lot of top-level routes, that become messy pretty quickly and involves a ton of index.blade.php files.
Any chance we could reconsider opening this issue?


Question

You have a Discord or a Email ?
I need to ask you some things.

Discord is yuko fuyutsuki#9595


Fix 3D to 2D prediction with UNeXt2 model

Removed the "2.1D" architecture and made the output stack depth of "2.2D" configurable between 2D and 3D.
Fix #78.

@Soorya19Pradeep see the updated example script for the configuration you requested.


Is it possible to setAccount after module config?

in .run or inside a controller?

Not at present. There is a lot of logic that goes into defining an account properly and all of that work is done during script injection, which can only ever happen once.
This could be changed.
What exactly is your use case requiring this?

I have a view where I send to the server a parameter from $stateParams, and then get in the response the relevant tracker id.

See #146 for an example of what you could do with the existing logic.


Add an initial nomenclature structure

This PR adds several variables and regions as a starting point for the project nomenclature.
In addition, it adds a GitHub Actions workflow to "validate the project" (i.e., that the yaml files in the definitions folder can be parsed and initialized as a nomenclature.DataStructureDefinition), and it updates the readme.

FYI @zwiebo @leggler


feat(HACBS-1567): install opm 4.9.0 4.10.0 and 4.11.0

We need to allow to run opm with the same version as OCP to validate fbc content. Now we install three versions of opm then we can choose to run different opm version with 'opm4.9.0 validate', 'opm4.10.0 validate', 'opm4.11.0 validate' according to the OCP version.
Signed-off-by: Hongwei Liu<EMAIL_ADDRESS>

Unfortunately OPM binary versions doesn't correspond to the ones from OCP.
We need to find out which opm commit sha(version) matches the OCP standard format version.

We also require various opm binaries versions, oc client binary is already present within hacbs-test image. Only one version of oc will suffice for purpose.

Close it since it is not needed any more.


[pipeline/chimera] test chimera | fix bug of initializing

integrate pipeline process group
test chimera and fix bug

Remove the cifar data file from this PR.


Allow deleting assigments

Delete assignments from the epicenter page

Waiting on delete support from API i think?

Opened PR


Help filtering test

Hi.
I am trying to set  --test-threads 1 to tests/user. Previously, I use cargo nextest run --all-targets --test-threads 1.
It works well but is very slow. Because other tests don't need   --test-threads 1 .
I try to use the per-test overrides feature but had no luck.
new file   .config/nextest.toml
@@ -0,0 +1,4 @@
+[[profile.default.overrides]]
+# filter = 'test(/^user::/)'
+filter = 'package(tin)'
+test-threads = 1
modified   justfile
@@ -56,7 +56,7 @@ _unit-test:

 # Test the codebase.
 test:
-    cargo nextest run --all-targets --test-threads 1
+    cargo nextest run --all-targets

Using  filter = 'test(/^user::/)',  filter = 'test(/^test::user::/)', and `filter = 'package(tin)' give no luck.
tin/tests on  master [+] via 🦀 v1.64.0
❯ tree
.
├── health
│  ├── mod.rs
│  └── tests.rs
├── meta
│  ├── mod.rs
│  └── tests.rs
├── user
│  ├── create_user.rs
│  ├── create_user_without_full_name.rs
│  ├── delete_user.rs
│  ├── duplicate_username.rs
<snip>...

Btw, do we have another suitable place for asking for help? Such Discussion tab.
Thanks a lot for the nextest! :heart:

Hi there, thanks for the feature request!
test-threads is not currently supported by per-test overrides; you can look at the set of supported tests here. Maybe we should support "num-cpus" for threads-required, which should solve your problem I believe.

test-threads is not currently supported by per-test overrides; you can look at the set of supported overrides here. Maybe we should support "num-cpus" for threads-required, which should solve your problem I believe.

Oh, I see.

Maybe we should also warn on unsupported config options.

Agree. It avoids such this confusion :)
Thanks for nextest!

Maybe we should support "num-cpus" for threads-required, which should solve your problem I believe

@sunshowers Should I open a new ticket to track this as a feature request?


Make the table of pretty-printing custom names mutable

It is possible to define custom human-readable names for any hash (scripts and pubkeys) in the pcOptHashNames options of the pretty printing options PrettyCookedOpts.
This is usually set statically once before running a trace.
However there are cases when this approach has limits.
For instance, given a parameterized script whose parameter will be some element from the chain (a spent UTxO for example when minting nfts), we don't have the information beforehand to add an entry to the table. Hence, there can't be any human readable name for hashes corresponding to parameterized scripts.
It would be a nice quality of life feature if these names could be updated in the course of a trace in the block chain state (where and how should be investigated).
At the end of execution, the resulting value of the custom names table would be used for pretty printing.

There is another improvement of this naming feature that could be coupled to this issue:
At the moment, we use the option pcOptHashNames to store a map of names to replace associated hashes in the pretty printer. This is very convenient. However, we also have a default map of names defaultHashNames which is problematic because it is overridden very easily when the user specifies new hashes. The cheatsheet does mention how to not override it, but still it is inconvenient. What we could do instead is have this default list hard coded in the pretty printer. When printing a hash, we would proceed as follows:

look up for it the provided map
look up for it in the default hardcoded map when absent from the map above
print the hash if absent from both
This way we keep the current behavior that you can override the default, but it is no longer required to manually mention the default names in the option (and it has to be at the end) when specifying new names.

This changes would come alongside a new option pcOptDisableDefaultName which would be set to False by default and which would disable to search in the default name list when set to True, just in case user really don't want to see those default names, and see hashes instead for some reason.

About the first idea, I believe it's a good one. A few observations:

we should make sure not to overload what can be done in our mockchain
maybe there would be a way to centralize 'auxiliary' features such a logging or this in another type class
we need to make sure that the monad stack allows to retrieve the list of name even in a case of failure


E   ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

skimage is probably compiled against older numpy than v2

Thinking of the best way to fix the numpy issue 🤔   maybe an extra optional dep-group, so in our CI we can write pip install -U -e .[examples,docs,numpyv1]

Fixing CI in #793


Stalls when accessing tables

When I try to open the database and list tables, on Ubuntu 16.04.3 LTS, the program stalls.
Here is the terminal output from when starting the program:
$ dbeaver

2017-09-15 14:03:51.635 - Create display
2017-09-15 14:03:51.770 - DBeaver 4.2.0 is starting
2017-09-15 14:03:51.771 - Install path: '/usr/share/dbeaver'
2017-09-15 14:03:51.771 - Instance path: 'file:/home/benyanke/.dbeaver4/'
2017-09-15 14:03:51.771 - Memory available 77Mb/910Mb
2017-09-15 14:03:51.779 - Starting RMI server at 51628
2017-09-15 14:03:51.859 - Run workbench
2017-09-15 14:03:53.352 - Initializing DBeaver 4.2.0
2017-09-15 14:03:53.359 - Host plugin: org.jkiss.dbeaver.core.application 4.2.0
2017-09-15 14:03:53.362 - Initialize Core...
2017-09-15 14:03:53.513 - Core initialized (151ms)
2017-09-15 14:03:53.518 - Configure workbench window
2017-09-15 14:03:53.598 - Create actions
2017-09-15 14:03:53.603 - Disable Eclipse action set 'org.eclipse.ui.edit.text.actionSet.annotationNavigation'
2017-09-15 14:03:53.607 - Disable Eclipse action set 'org.eclipse.ui.WorkingSetActionSet'
2017-09-15 14:03:53.974 - Initialize workbench window
2017-09-15 14:03:53.988 - Finish initialization
2017-09-15 14:03:55.723 - Connect with '[redacted: database name]' (mssql_ms-15e862216e2-1a754ec176f9da15)
2017-09-15 14:03:56.117 - Connected (mssql_ms-15e862216e2-1a754ec176f9da15, Microsoft JDBC Driver 4.0 for SQL Server [4.0.2206.100])

EDIT: Also found the -debug option:
$ dbeaver -debug

Install location:
    file:/usr/share/dbeaver/
Configuration file:
    file:/usr/share/dbeaver/configuration/config.ini loaded
Configuration location:
    file:/home/benyanke/.eclipse/org.jkiss.dbeaver.product_4.2.0_620785679_linux_gtk_x86_64/configuration/
Configuration file:
    file:/home/benyanke/.eclipse/org.jkiss.dbeaver.product_4.2.0_620785679_linux_gtk_x86_64/configuration/config.ini loaded
Loading timestamp file from:
	 file:/home/benyanke/.eclipse/org.jkiss.dbeaver.product_4.2.0_620785679_linux_gtk_x86_64/configuration/   .baseConfigIniTimestamp
	No timestamp file found
Timestamps found:
	 config.ini in the base:<PHONE_NUMBER>000
	 remembered -1
Shared configuration location:
    file:/usr/share/dbeaver/configuration/
Framework located:
    file:/usr/share/dbeaver/plugins/org.eclipse.osgi_3.11.3.v20170209-1843.jar
Loading extension: reference:file:org.eclipse.osgi.compatibility.state_1.0.200.v20160504-1419.jar
	eclipse.properties not found
Framework classpath:
    file:/usr/share/dbeaver/plugins/org.eclipse.osgi_3.11.3.v20170209-1843.jar
    file:/usr/share/dbeaver/plugins/
    file:/usr/share/dbeaver/plugins/org.eclipse.osgi.compatibility.state_1.0.200.v20160504-1419.jar
Splash location:
    /home/benyanke/.eclipse/org.jkiss.dbeaver.product_4.2.0_620785679_linux_gtk_x86_64/configuration/org.eclipse.equinox.launcher/org.jkiss.dbeaver.core.application_4.2.0/splash.bmp
Debug options:
    file:/home/benyanke/vagrant-boxes/grandcentral/.options not found
Time to load bundles: 3
Starting application: 574
2017-09-15 14:10:38.288 - Create display
2017-09-15 14:10:38.428 - DBeaver 4.2.0 is starting
2017-09-15 14:10:38.429 - Install path: '/usr/share/dbeaver'
2017-09-15 14:10:38.429 - Instance path: 'file:/home/benyanke/.dbeaver4/'
2017-09-15 14:10:38.429 - Memory available 77Mb/910Mb
2017-09-15 14:10:38.437 - Starting RMI server at 22143
2017-09-15 14:10:38.512 - Run workbench
2017-09-15 14:10:40.139 - Initializing DBeaver 4.2.0
2017-09-15 14:10:40.168 - Host plugin: org.jkiss.dbeaver.core.application 4.2.0
2017-09-15 14:10:40.173 - Initialize Core...
2017-09-15 14:10:40.389 - Core initialized (216ms)
2017-09-15 14:10:40.411 - Configure workbench window
2017-09-15 14:10:40.523 - Create actions
2017-09-15 14:10:40.530 - Disable Eclipse action set 'org.eclipse.ui.edit.text.actionSet.annotationNavigation'
2017-09-15 14:10:40.534 - Disable Eclipse action set 'org.eclipse.ui.WorkingSetActionSet'
2017-09-15 14:10:41.046 - Initialize workbench window
2017-09-15 14:10:41.069 - Finish initialization
Application Started: 4479
2017-09-15 14:10:42.948 - Connect with '[redacted: database name]' (mssql_ms-15e862216e2-1a754ec176f9da15)
2017-09-15 14:10:43.477 - Connected (mssql_ms-15e862216e2-1a754ec176f9da15, Microsoft JDBC Driver 4.0 for SQL Server [4.0.2206.100])

I also know it's likely not a server issue, the windows client works fine with the same config.

Does Stalls mean UI freeze?

If so - could you make a thread dump (https://github.com/serge-rider/dbeaver/wiki/Making-a-thread-dump) when DBeaver will freeze next time?
Does this happen always or only with come particular databases?

Yes, freezing, and as Ubuntu applications do when they freeze for more than a second or two, the entire app grays out.
I will take a dump on Monday and collect more information.
 I have access to some MySQL databases, so I will try those.
The windows version, however, I was able to connect to the same database with no issues, by the way.

Dump:
dump.txt

Just confirmed that it does not stall when accessing a MariaDB/MySQL database.

Looks odd. UI freezes on tree item (native OS components) creation.
Is there anything special about your MS SQL databse? Maybe a lot of tables or some weird table names with special characters?

Yes, it is quite a large database, on the order of hundreds-of-millions of records and hundreds of tables. Not sure about all the table names, I'm inheriting the database, so I haven't fully scrubbed it yet, but I do know there are spaces and $ in most, if not all of the tables. If it's any assistance, it's the underlying storage for a Microsoft Navision 2013 R2 instance.
The thing that struck me as odd was that it worked instantly on the Windows version...

This seems to be GTK specific and this is a kind of SWT bug (sometimes it hangs on huge trees).
Similar to https://bugs.eclipse.org/bugs/show_bug.cgi?id=354842 or https://bugs.eclipse.org/bugs/show_bug.cgi?id=383733.
Probably it is fixed in Eclipse 4.7 (we are going to migrate DBeaver to it after 4.3 version or you can try Eclipse plugin version in the most recent Eclipse 4.7).
Also you can try workaround proposed in 354842 bug.  Execute
export SWT_WEBKIT2=1
export SWT_LIB_VERSIONS=1

before running DBeaver from console.

Could you test this on the recent DBeaver version (it is based on Eclipse 4.7)?

Yes, still occurs on 4.2.3.

That's bad. Means 4.7 doesn't fix this. However bug https://bugs.eclipse.org/bugs/show_bug.cgi?id=383733 is still open.
Could you say what window manager do you use? Any other details?
Trying to figure out how to reproduce this.

I'm using a pretty standard unity setup on Ubuntu 16.04 LTS... Lmk if there's any other info that would be helpful.

I'm having the same issue, version 4.2.5 on Linux Mint 8, Cinnamon desktop.  Microsoft SQL, latest java driver, database with 9888 tables.  I would prefer NOT to have to work in Windows, if I have to do that then I may as well use the Microsoft tools for this and don't need DBeaver at all.

@mmceldowney GTK + some particular hardware (drivers) seems to be the reason. UI hangs during tree control population. I still have no ideas how to solve this.
And yes, workaround you described will solve the issue, thanks for noticing.

Not a resolution, but a potential workaround: use filters to shorten the list of tables.
Our DB has a structure of about 1500 tables with a prefix in the table name of app1_, another 1500 tables for app2_, for a total of about 7000 tables. However, I'm only developing using the app1 tables, so I set up a filter like so:

This means the app only has to (apparently?) load info about 1500 tables. It's far from a work around, and the app still stalls for 2-3 seconds, but it does now load, making the app usable for this specific usecase.
Not a solved problem by any means, but perhaps a workaround usable for some users, depending on their workload and DB structure.
Do keep a windows VM around if you need to access other tables not matching that filter though...

For me It happens with ubuntu 16.04 with xfce with oracle with more than 3100 tables about 800 views and 6000 indexes

Same problem here on Ubuntu 18.04 on a large Postgres database.

Since 5.2 you can configure how many objects should be visible in the navigator tree:

Hint: you can view all objects in schema editor (double click on "Tables" element). It is much more convenient than navigator tree.


Switchboard Recipe

Hey everybody,
I made a recipe for the Switchboard corpus.
The data preparation steps mostly follow Kaldi's s5c recipe.
The recipe includes the following models:
ASR

CTC: Wav2Vec2 Encoder + CTC Decoder (adapted from the Commonvoice recipes)
seq2seq: CRDNN encoder + GRU Decoder + Attention (adapted from the LibriSpeech recipe)

Note: Unlike the Librispeech recipe, this system does not include any LM. In fact, every LM I tried (pretrained, finetuned or trained from scratch) seemed to make the performance much worse

transformer: Transformer model + LM (adapted from the LibriSpeech recipe)

LM

There are two hparams files for finetuning existing LibriSpeech LMs on Switchboard and Fisher data, one for an RNNLM and the other for a Transformer LM

Tokenizer

Basic Sentencepiece Tokenizer training on Switchboard and Fisher data

Performance
The model performance is as follows:

Model
Swbd WER
Callhome WER
Eval2000 WER

CTC
21.35
28.32
24.91

seq2seq
25.37
36.87
29.33

Transformer  (LibriSpeech LM)
22.00
30.12
26.14

Transformer  (Finetuned LM)
21.11
29.43
25.36

As you can see, the performance is currently comparable to Kaldi's chain systems without i-vectors.
However, they need some refinement to be on par with the best Kaldi systems available (WER should be around 18 on the full eval2000 testset).
If you have any suggestions for improvements, I'd be happy to implement them.
I can also provide the trained models in case you are interested (I might need some help with this whole Huggingface thing though).
Best,
Dominik
ps Thanks for all the great work you've done here! :)

Hi! Thanks for this huge contribution. Please start by solving all the tests, and then we will proceed with a proper review :-)
You can install the pre-commit and pre-push hooks quite easily following the doc!
Also, do you know how your numbers compare to ESPnet for instance? We can easily discuss ways of improving the performance.

Thank you for this contribution. One tip: you can also run the tests locally with 'pytest', ''tests/.run-linters.sh" (for style issues), ".run-unittests.sh" (for unittests). You can fix most of the style issues of python files with "black" (see https://speechbrain.readthedocs.io/en/latest/contributing.html)

I would also suggest to merge here the latest version of the development branch. We just merged some additional checks and tests that are needed in this case as well.

Hi,
I merged with the latest development branch, ran the linters and made the appropriate changes, so the formatting should be ok now.
Unfortunately, my local unittests kept crashing in tests/unittests/test_augment.py and tests/unittests/test_ctc_segmentation.py with this error:
ImportError while importing test module '/nfs/scratch/staff/wagnerdo/speechbrain-fork-2/tests/unittests/test_ctc_segmentation.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/unittests/test_ctc_segmentation.py:1: in <module>
    from speechbrain.pretrained import EncoderDecoderASR
speechbrain/__init__.py:4: in <module>
    from .core import Stage, Brain, create_experiment_directory, parse_arguments
speechbrain/core.py:35: in <module>
    from speechbrain.utils.distributed import run_on_main
speechbrain/utils/__init__.py:11: in <module>
    from . import *  # noqa
speechbrain/utils/profiling.py:11: in <module>
    from torch.autograd.profiler_util import (  # pytorch v1.10.1
E   ModuleNotFoundError: No module named 'torch.autograd.profiler_util'

I'm not sure whether it's a local issue with my environment or whether it's a broader problem.
In any case, it doesn't look like it has something to do with the Switchboard recipe in particular.
@TParcollet: ESPNet reports WERs of 15.6 (Callhome), 8.4 (Swbd), and 12.0 (eval2000) for their latest transformer model (see https://github.com/espnet/espnet/blob/master/egs2/swbd/asr1/RESULTS.md), so there is definitely room for improvement.

Thank you @dwgnr! It looks like the performance is not to good at the moment. As far as I remember, switchboard recording are sampled at 8 kHz (and not 16 kHz as the other recipes). Do you consider it?  Also, I think a significant improvement can be achieved by adding a language model, right?

Hi @mravanelli,
thanks for kicking off the checks once again.
Yes, Switchboard is sampled at 8kHz and I did indeed consider this in various ways.
For the CTC recipe, everything is simply upsampled to 16 kHz.
This worked much better than passing the 8 kHz data to the pretrained wav2vec model.
The WERs with 8 kHz data are 35.70 (Swbd), 52.30 (Callhome), and 43.05 (eval2000).
The seq2seq model is trained on 8 kHz audio. I also tried upsampling to 16 kHz but it didn't make much difference.
For the transformer model, the audio data is currently upsampled to 16 kHz, but I haven't tried training it on 8 kHz yet.
I fully agree, adding a decent language model should definitely lead to improvements.
Unfortunately, the opposite is the case at the moment :/
I tried a lot of different things with RNNLMs on the seq2seq model (e.g. use pretrained LibriSpeech LM, train simple LSTM LM from scratch, finetune pretrained LibriSpeech LM) and I was careful to use the correct Tokenizer for each LM.
However, the performance always became worse compared to the variant without LM.
Are there any known issues with the S2SRNNBeamSearchLM(S2SRNNBeamSearcher) decoder class?
Or maybe it's just some parameters that work for Librispeech but not so well for Switchboard?
When looking through the detailed results, I noticed that with the LM in place, the decoder seems to be very eager to output more than just a single word, which leads to a lot of insertions.
For example with LM:
================================================================================
en_4938_A_1206, %WER 100.00 [ 1 / 1, 1 ins, 0 del, 0 sub ]
UHHUH ; <eps>
  =   ;   I
UHHUH ;   U
================================================================================
en_6189_B_1892, %WER 300.00 [ 3 / 1, 3 ins, 0 del, 0 sub ]
IT ; <eps> ; <eps> ; <eps>
=  ;   I   ;   I   ;   I
IT ;   IT  ;   IT  ;   IT
==============================================================================
en_6489_A_2236, %WER 500.00 [ 5 / 1, 5 ins, 0 del, 0 sub ]
OH ; <eps> ; <eps> ; <eps> ; <eps> ; <eps>
=  ;   I   ;   I   ;   I   ;   I   ;   I
OH ;   OH  ;   OH  ;   OH  ;   OH  ;   OH

This phenomenon is less pronounced for the decoder without LM:
================================================================================
en_4938_A_1206, %WER 0.00 [ 0 / 1, 0 ins, 0 del, 0 sub ]
uhhuh
  =
uhhuh
================================================================================
en_6189_B_1892, %WER 100.00 [ 1 / 1, 1 ins, 0 del, 0 sub ]
it ; <eps>
=  ;   I
it ;   it
================================================================================
en_6489_A_2236, %WER 0.00 [ 0 / 1, 0 ins, 0 del, 0 sub ]
oh
=
oh
================================================================================

There are actually quite a lot of very short utterances in the corpus, so these errors quickly accumulate.
The problem also seems to invert for long utterances.
For example with LM:
================================================================================
en_6183_B_1770, %WER 20.51 [ 8 / 39, 0 ins, 3 del, 5 sub ]
BUT ; I ; REMEMBER ; WHEN ; I ; FIRST ; MET ; YOU ; THOUGH ; AND ; YOU ; WERE ; REAL ; POSITIVE ; ABOUT ; THE ; WHOLE ; THING ;  AND  ; I ; THOUGHT ; SHE ; WANTS  ; TO ; BE ; A ; DIETICIAN ;  SHE  ;   IS  ; CRAZY ;  THERE  ;   IS  ; SO ; MANY ; OTHER ; THINGS ; SHE ; COULD ; DO
 =  ; = ;    =     ;  =   ; = ;   =   ;  =  ;  =  ;   =    ;  =  ;  =  ;  =   ;  =   ;    =     ;   =   ;  =  ;   =   ;   =   ;   D   ; = ;    =    ;  =  ;   S    ; =  ; =  ; = ;     S     ;   S   ;   D   ;   =   ;    S    ;   D   ; =  ;  =   ;   =   ;   =    ;  S  ;   =   ; =
BUT ; I ; REMEMBER ; WHEN ; I ; FIRST ; MET ; YOU ; THOUGH ; AND ; YOU ; WERE ; REAL ; POSITIVE ; ABOUT ; THE ; WHOLE ; THING ; <eps> ; I ; THOUGHT ; SHE ; WANTED ; TO ; BE ; A ; DIETITIAN ; SHE'S ; <eps> ; CRAZY ; THERE'S ; <eps> ; SO ; MANY ; OTHER ; THINGS ; YOU ; COULD ; DO
================================================================================

And  without LM:
en_6183_B_1770, %WER 148.72 [ 58 / 39, 47 ins, 1 del, 10 sub ]
but ; i ; remember ; when ; i ; first ;  met  ; you ; though ; and ; you ; were ; real ; positive ; about ; the ; whole ; thing ;  and  ; i ; thought ; <eps> ; she ; wants ; to ; be ; a ; dietician  ; she ;   is  ; crazy ; there ;    is   ; so ; many ; other ; things ; she ; could ; do ; <eps> ; <eps> ; <eps> ; <eps> ; <eps> ; <eps> ; <eps> ; <eps> ; <eps> ; <eps> ; <eps> ; <eps> ; <eps> ; <eps> ; <eps> ; <eps> ; <eps> ; <eps> ; <eps> ; <eps> ; <eps> ; <eps> ; <eps> ; <eps> ; <eps> ; <eps> ; <eps> ; <eps> ; <eps> ; <eps> ; <eps> ; <eps> ; <eps> ; <eps> ; <eps> ; <eps> ; <eps> ; <eps> ; <eps> ; <eps> ; <eps> ; <eps> ; <eps> ; <eps> ; <eps> ; <eps>
 =  ; = ;    =     ;  =   ; = ;   =   ;   S   ;  S  ;   S    ;  =  ;  =  ;  =   ;  =   ;    =     ;   =   ;  =  ;   =   ;   =   ;   D   ; = ;    =    ;   I   ;  =  ;   =   ; =  ; =  ; = ;     S      ;  S  ;   S   ;   =   ;   S   ;    S    ; =  ;  =   ;   =   ;   =    ;  S  ;   S   ; =  ;   I   ;   I   ;   I   ;   I   ;   I   ;   I   ;   I   ;   I   ;   I   ;   I   ;   I   ;   I   ;   I   ;   I   ;   I   ;   I   ;   I   ;   I   ;   I   ;   I   ;   I   ;   I   ;   I   ;   I   ;   I   ;   I   ;   I   ;   I   ;   I   ;   I   ;   I   ;   I   ;   I   ;   I   ;   I   ;   I   ;   I   ;   I   ;   I   ;   I   ;   I   ;   I   ;   I   ;   I   ;   I   ;   I
but ; i ; remember ; when ; i ; first ; moved ;  to ;  utah  ; and ; you ; were ; real ; positive ; about ; the ; whole ; thing ; <eps> ; i ; thought ;  well ; she ; wants ; to ; be ; a ; technician ; and ; she's ; crazy ;  and  ; there's ; so ; many ; other ; things ; you ;  can  ; do ;   do  ;  you  ;   do  ;   do  ;  you  ;   do  ;   do  ;   do  ;   do  ;   do  ;   do  ;   do  ;   do  ;   do  ;   do  ;   do  ;   do  ;   do  ;   do  ;   do  ;   do  ;   do  ;   do  ;   do  ;   do  ;   do  ;   do  ;   do  ;   do  ;   do  ;   do  ;   do  ;   do  ;   do  ;   do  ;   do  ;   do  ;   do  ;   do  ;   do  ;   do  ;   do  ;   do  ;   do  ;   do  ;   do

Do you have any ideas what could cause these weird insertions?

As for the RNNLM or TransformerLM recipes, these are the same that we use for librispeech (where we noticed a quite big improvement when using the LM). As for the decoding part, one thing that might be important is to tune a bit the hyperparameters of the beamsearcher. According to my experience, this is really important. Fortunately, to tune these hyperparameters you don't have to run the training part again, but only the final decoding part (ideally on the validation set and not on the test one).

@dwgnr thank you for this contribution!
It looks all very clean - there are pre-commit check issues with yaml files. Please check also for errors in the consistency checks.
ERROR: The file recipes/Switchboard/LM/hparams/transformer_small.yaml is not listed in tests/recipes.csv. Please add it.                 For more info see tests/consistency/README.md
ERROR: The file recipes/Switchboard/Tokenizer/hparams/2K_unigram_subword_bpe.yaml is not listed in tests/recipes.csv. Please add it.                 For more info see tests/consistency/README.md
ERROR: The file recipes/Switchboard/ASR/transformer/hparams/transformer_small_lm_after_cleanup.yaml is not listed in tests/recipes.csv. Please add it.                 For more info see tests/consistency/README.md
ERROR: The file recipes/Switchboard/ASR/transformer/hparams/transformer_finetuned_LM.yaml is not listed in tests/recipes.csv. Please add it.                 For more info see tests/consistency/README.md
ERROR: The file recipes/Switchboard/LM/hparams/transformer.yaml is not listed in tests/recipes.csv. Please add it.                 For more info see tests/consistency/README.md
ERROR: The file recipes/Switchboard/ASR/transformer/hparams/transformer_small_lm.yaml is not listed in tests/recipes.csv. Please add it.                 For more info see tests/consistency/README.md
ERROR: The file recipes/Switchboard/ASR/seq2seq/hparams/train_BPE_1000_own_tokenizer.yaml is not listed in tests/recipes.csv. Please add it.                 For more info see tests/consistency/README.md

ERROR: variable "ctc_weight_decode" not used in recipes/Switchboard/ASR/seq2seq/train.py!

As mentioned abive, before the final commit, please use !PLACEHOLDER for data_folders.
Btw - do you still get E   ModuleNotFoundError: No module named 'torch.autograd.profiler_util' ?
(It might be a pytorch version issue.)

Hi @TParcollet, @mravanelli,
I did some experiments and basically ended up rewriting the entire data preparation pipeline.
Turns out that preparing the data is a bit more involved than I originally thought.
For example, I wasn't aware that sclite (which is used by Kaldi and Espnet for scoring), does a lot of normalization such as expanding contractions and mapping acronyms.
There is still a chance, that the data preparation in this recipe deviates a bit from what is done in Kaldi/Espnet but I think that I captured most of it.
Anyway, the results improved considerably after rewriting the data pipeline:

Model
Swbd WER
Callhome WER
Eval2000 WER

CTC
8.90
14.99
12.04

Transformer (Swbd+Fisher LM)
9.64
17.97
13.99

Transformer (Finetuned LM)
9.99
18.98
14.34

I didn't get to try the seq2seq model on the new data yet but it should also improve I guess.
@anautsch: Sorry for the mess. The recipe still needs a lot of cleaning up. I'm on it (:

Hi @anautsch,
as promised, I did some cleanup.
Let me know if you need me to change anything else (:
Regarding your question about the unittests:
I avoided the ModuleNotFoundError: No module named 'torch.autograd.profiler_util error by switching to a different machine ;-)
Now the unittests fail locally with an AssertionError in tests/unittests/test_profiling.py:351. Apparently len(brain_or_pretrained.profiler.events()) is supposed to yield 0 but it yields 1 in my case.

Hey everybody,
I updated the table above with the latest results of the seq2seq model.
I didn't commit the seq2seq+RNNLM hparams file since the results are much worse than the ones without LM.
Let me know if there is anything else I can do :-)

Hey @anautsch,
no worries, I did some holidaying myself 😉
I hope you had a great a vacation!
I fetched the latest develop and fixed the errors you mentioned above.
Linter and Unittests passed locally, I hope they will pass in the workflow run as well.
Btw: The len(brain_or_pretrained.profiler.events()) error I mentioned before, disappeared since we upgraded to the latest NVIDIA drivers and CUDA version.
I'd be happy to make the models available.
Could you point me to some documentation regarding the process?
I've never uploaded anything to Huggingface, so I'm not entirely sure how this works.

Hi @dwgnr are things going well?

Hi @anautsch, yes everything is fine.
I just refactored a lot of duplicated code for word normalisation into a separate file (ASR/normalize_util.py). The normalize_words function can then be passed via the constructor of each ASR class (or not if the user doesn't want to).
Sorry, I was in a bit of a hurry and forgot to mention what I did here.
I hope that's fine. I can easily revert it if necessary.
I also retrained every model (except for the CTC one, which is currently running), just to make sure that everything is in order.

@anautsch I just merged the most recent develop branch. I wouldn't change anything anymore at this point (unless you tell me to, of course).
Yeah, I noticed that it kept reformatting a lot of other files in the repo that I hadn't touched at all, but it was all part of the tests/.run-linters.sh execution, so I thought it should use the correct configs for flake8, yamllint etc.
I created a HF account and sent a request to join the speechbrain org. It would be great if @TParcollet or @Adel-Moumen could guide me a bit on a how to create a model card. Thanks a lot!

Yeah, I noticed that it kept reformatting a lot of other files in the repo that I hadn't touched at all, but it was all part of the tests/.run-linters.sh execution, so I thought it should use the correct configs for flake8, yamllint etc.

Yes, and you are right to say that this can happen - it is also out of our control (for some yet unclear reason). The github hooks seem to be able to use different packages in their cache/domain than the virtual python environment of choice provides. We'll find a solution; it shouldn't be critical to this PR. There's lots of good work in here! (need time for this side effect issue; just fyi)

@anautsch I just merged the most recent develop branch. I wouldn't change anything anymore at this point (unless you tell me to, of course).
Yeah, I noticed that it kept reformatting a lot of other files in the repo that I hadn't touched at all, but it was all part of the tests/.run-linters.sh execution, so I thought it should use the correct configs for flake8, yamllint etc.
I created a HF account and sent a request to join the speechbrain org. It would be great if @TParcollet or @Adel-Moumen could guide me a bit on a how to create a model card. Thanks a lot!

Hello,
I will create a model card on our HuggingFace hub then you will have to do a pull request on this card to add everything.  😃
I will let you know when it is ready.

Sounds great thanks a lot! :)

Resolved the conflict on the test/recipes.csv file; another PR got merged in changing this one, yesterday.
So, we wait until all is on HF, then we have the URLs so READMEs etc can be updated?

Makes sense from my point of view.
I'm ready to open a PR on HF as soon as the model card is ready :)

Makes sense from my point of view. I'm ready to open a PR on HF as soon as the model card is ready :)

Yep I'll do it asap  🙂

Hello,
Did you join the HF organisation? Thanks. 🙂

Hello,
Did you join the HF organisation? Thanks. 🙂

Hi @Adel-Moumen, I sent a request to join the SpeechBrain organisation on HF a couple of days ago. It's still pending.
I guess someone has to approve the request? (The username is also dwgnr)

Hello,
Did you join the HF organisation? Thanks. 🙂

Hi @Adel-Moumen, I sent a request to join the SpeechBrain organisation on HF a couple of days ago. It's still pending. I guess someone has to approve the request? (The username is also dwgnr)

Hello,
Unfortunately, we don't accept any members (except the core team) in the organization for security reasons. (I got the information today)
So, we will proceed this way, I have created three models cards for the three ASR models on our HF hub:
CRDNN: https://huggingface.co/speechbrain/asr-crdnn-switchboard
Transformer: https://huggingface.co/speechbrain/asr-transformer-switchboard
wav2vec2: https://huggingface.co/speechbrain/asr-wav2vec2-switchboard
Could you please make some pull requests to add the necessary files? You can use the other model card from our hub as example.
Many thanks and sorry for the delay.

Hey @Adel-Moumen,
thanks for adding me to the organization! I'm honored 🤗
I just created PRs on all 3 cards.
The model files are all there, but I need to go over the READMEs and make sure that everything is correct.
This shouldn't take long.
For the wav2vec2 card I'm not sure about config.json and preprocessor_config.json.
I simply stole the files from https://huggingface.co/speechbrain/asr-wav2vec2-librispeech/blob/main/config.json, but I am not entirely sure what their purpose is.
Locally, everything seems to work fine without having those files in the repo at all.
I am going to add a note about this in the model card discussion as well.

Hi @Adel-Moumen,
my PRs on HF are also ready for review. I switched the PRs from "draft" to "open". Hope that's ok 🙂

Hi @anautsch, I discovered a tiny bug in the train scripts that was introduced after my refactoring. Is it ok to push a fix or are you on it at the moment?

So, we wait until all is on HF, then we have the URLs so READMEs etc can be updated?

Should I add URLs to the READMEs in the recipe already?

@dwgnr thank you for concerning me here - I'm waiting for you to say how you are doing :wink:
Please proceed as you see fit - and thanks for involving us closely in your progress (as contributor, you do not need to review your own code - although it helps to see your thought process and upcoming developments). Please mark your reviews as "resolved" once they are done. (Your comments appear more than valid.)
It is good practice to move from "Draft" to "Open" once it is ready, indeed.
Yes, please add URLs :)
No, I'm not on it at the moment as cross-checking if it's running goes—the validity of pretrained models is a responsibility of the community.

Hi, @anautsch I merged the most recent develop branch, fixed the bug I mentioned and added the HF URLs to the recipe's Readmes.
I think you can review and merge now if you like :)

@Adel-Moumen @TParcollet  what's your take - should we merge without having been able to debug/test recipes, since Switchboard is with restricted access?
(From my perspective, we should merge & if someone faces issues, we would welcome their PR. There are little other pragmatic means to wrap this PR up - there are ample of contributions in here already.)

In case it helps your decision:
I'm open to help fixing bugs related to the recipe, if someone finds one after the merge 🙂

Hi @dwgnr getting back to this. As there are no responses thus far, I'd proceed with this my way. I consider your line of work as completed, yet, we face some technicality issues here, so—let's tackle them.
Please let me resolve some side issues (i.e. please do not push meanwhile):

reverting (also correct but different) formatting of PR-unconcerned files
integration testing of proposed model(s) w/o database-depending dataio & prepare scripts
recipe.csv file just causes merge conflicts across all PRs (effectively, putting contributors in competition)

For the latter, I still have to make up my mind of what would be a viable step forward. Perhaps I should work this out in another, new PR and then come back here. (Working on your PR integration in the then-new recipe integration testing PR meanwhile.)
We lack some general recipe testing (e.g. functionality check when refactorings are coming up).
For the former, I looked into different options and will push one solution in a bit:

rebase -> we will not break the git tree
copy/paste from a more recent develop status -> git tree consistency is core
seek highest top-level folders with reformattings-only below -> take earlier commit ID -> checkout (recursively) that folder at that commit ID

So that I do not need to go over that again, please do not re/format &/or push to this branch ;-)

Hi @anautsch, I won't do anything to the branch, I promise ;-)
Let me know if it gets too messy with all the reformatting issues.
I could simply open a new PR. Maybe that's cleaner and less complicated?

@dwgnr :rofl:
look at the stats of your PR
+4,802 −1
=> this is your work shining ;-)
... from my side the PR is good as it stands—it's environment needs to be brought up to speed.
—and then, some aligning things need to occur, that's all the patience I'm asking for.

Hello,
The three models have been merged in the HF SB hub.
We are now waiting for this PR to be merged so that we can then switch the visibility of the models to 'public' on the HF hub :-)
Many thanks, @dwgnr for your massive work! 😎

Hi @dwgnr,
thank you for your contribution. Could you please fix the conflict?

Thank you! @anautsch do you think everything is ready to be merged?

Hi @mravanelli - the PR lgtm as said before.
Since I do not have access to the same data, I oculdn't test all - I went for another strategy, the recicpe testing PR (after that, I wanted to merge this one using the tools from there, knowing these recipes are sanity checked and join a healthy base). I still need to figure out the test debug flags as in here (for example):
https://github.com/speechbrain/speechbrain/blob/15819a16f903523d384e3231731fc83e30bce8cc/tests/recipes/AISHELL-1.csv
@dwgnr I'll attend first to this error - it's coming up across all PRs right now (I suspect some dependency updated and broke things). Then, let's get this done already; I didn't think about simply working out the test debug flags before - sorry -> that'd have already satisfied what I need to have.

@dwgnr, could you please merge here the latest version of the develop branch?

Hi @mravanelli, I merged the latest version and it auto-closed the PR. Was it supposed to do that?

Hi @dwgnr thank you for updating the latest merge conflict.
Please take a look at the test_debug_flags in
https://github.com/anautsch/speechbrain/blob/refactor-recipe-testing/tests/recipes/CommonVoice.csv
or
https://github.com/anautsch/speechbrain/blob/refactor-recipe-testing/tests/recipes/LibriSpeech.csv
is there a 'minimal' set of test flags for your scripts, so we can test them w/o having the actual data available?
It's using minimal data available at: https://github.com/speechbrain/speechbrain/tree/develop/tests/samples
If you could provide the  test_debug_flags in addition to each added recipe, that would help me.
The PR #1600 demands more time from me than I expected. In there, the testing is made database specific and available for all recipes in the sense that we also know the workflows are still functional (think: dependencies may update and their interfaces could break over time).

Hi @anautsch, no problem. I'll see if I can find a minimal set of test_debug_flags for the recipes.

Hope this command helps you to just run the checks for your recipes

python -c 'from speechbrain.utils.recipe_tests import run_recipe_tests; print("TEST FAILED!") if not(run_recipe_tests(filters_fields=["Dataset"], filters=["Switchboard"], do_checks=False)) else print("TEST PASSED")'

Hi @anautsch, I managed to get the tests running under the new recipe testing environment🎉
It would require some small adjustments in #1600 as well as some changes in this PR.
In #1600, tests/samples/annotation/ASR_train.csv would need two extra columns called channel and words and to make the LM training work, the dev and train corpus files would need to be CSVs instead of plaintext.
I attached the files below:
Switchboard.csv
LM_dev.csv
LM_train.csv
ASR_train.csv
I pushed the changes related to the recipes here. Hope that's fine.

Hi @dwgnr - that's a great help, thank you!
Your recipe should not change too much; adequate dummy data needs to be there. I think it is that what you refer to.
The changes you brought forth will impact the other PR - that's good!
Right now, there is no merge conflict to develop. As for the other PR, I'll need to update it to include recipes that were released meanwhile just as well as this one. The changes you made look like what I needed to do with other recipes, too. I'll need to get back to this when wrapping up the other PR — meanwhile:
lgtm

Hi @anautsch, thanks for merging! This was a lot of fun :-)
Let me know if users encounter any problems with the recipe.

I turned public the models on the HuggingFace hub. Thanks a lot again for your work!! :-)


Feature: Add snapshots in progress

Adds the ability to resume a snapshot, for example when the application crashes is otherwise closed while a snapshot is in progress.

Couple of more things.

Are you sure your fonts are correct?
Can you make the orange warning bigger? (or if there's a reason you made it smaller, just lmk)
Center align title on pop up
All attributes from the button should be removed besides specifying the style and theme (for instance, remove the padding you specified as it's not enough.. our spec calls for 48 I think)


LambdaBetterGrass

Mod name
LambdaBetterGrass
Curseforge link
https://www.curseforge.com/minecraft/mc-mods/lambdabettergrass
Modrinth link
https://modrinth.com/mod/lambdabettergrass
Other link
https://github.com/LambdAurora/LambdaBetterGrass
What it does
Adds "better grass" like Optifine
Why should it be in the modpack
Optifine parity
Why shouldn't it be in the modpack
Rendering API missing
Categories

[ ] Performance optimization
[x] Graphics optimization
[ ] New feature
[x] Optifine parity
[ ] Fixes a bug/dependency

Additional details

Issue I'd like to get fixed first: https://github.com/LambdAurora/LambdaBetterGrass/issues/16

1.16 & 1.17 work. See above mention in the Indium issue for proof.
LambdAurora/LambdaBetterGrass#16 can be worked around by providing a default config disabling corner blending because it is optional.


[FEATURE] Refresh Control

The Amethyst app consumes too much battery usage & is burning the battery on my phone. The issue that causes this is the global feed's refresh rate. The more relays you add to your profile the more request are sent out which burns your battery faster since every relay added is getting a getfetch the constant request to refresh the global feed is what makes the battery run hot & burn.
The solution to this would be to add a setting which allows the user to control the refresh interval with multiple options.
Example the user would go to settings & would see a option called refresh control with a menu of choices to set to. The choices on the menu would be Always (constant), every 15 minutes, every 30mins, every 60mins, & custom where the user can manually input their own interval in time format which can be recognized.
Example of custom inputs
24h, 3h 40m, &  3d 4h 24m 32s.
d=days
h=hours
m=minutes
s=seconds
Adding a refresh control will save the battery on peoples phones lets make this a priority because I can't keep using amethyst on mobile if it's going to drain my battery from 100% to 0 multiple times a day.

The issue that causes this is the global feed's refresh rate.

Did you test this? Is it just an assumption or did you validate this somehow?
The global feed is disabled unless you are into that Global feed from the top navigation bar. The follows feed is minimal because it only loads from select authors.
That being said, if the app is in the foreground, it is always connected with all the relays on your relay list. That is needed to bring notifications from any of the relay.
On top of that, Amethyst loads quotes, profile pictures, @ citations and so on only when you see the post in the screen. This means that if you stop being connected, the majority of the posts simply won't load. It's impossible to load all the posts for all parts of the app from the beginning, so we need to do this gradually as you scroll through the feed.
So, most of the connectivity use is to load the rest of the post as you navigate through them.
If you leave the app running, but don't scroll anything, then most relays will close the connection in about 1 minute and the app will stay off until you scroll or change screens again.

The issue that causes this is the global feed's refresh rate.

Did you test this? Is it just an assumption or did you validate this somehow?
Well I tried with less relays & adding more relays I removed all my relays keeping it down to 1 then kept pulling feed for a day then compared it to 25 relays pulling I can say the battery drains faster more.
The global feed is disabled unless you are into that Global feed from the top navigation bar. The follows feed is minimal because it only loads from select authors.
I am always in the global first & follows second of course follows uses less but still the request is constant when your in it.
That being said, if the app is in the foreground, it is always connected with all the relays on your relay list. That is needed to bring notifications from any of the relay.
This is the problem constant request there is no manual refresh or control for intervals what if 100 post load & I don't want it to constantly keep pulling for relays because I am scrolling & want it to pull new posts from the relay an hour from now not as I am constantly scrolling.
On top of that, Amethyst loads quotes, profile pictures, @ citations and so on only when you see the post in the screen. This means that if you stop being connected, the majority of the posts simply won't load. It's impossible to load all the posts for all parts of the app from the beginning, so we need to do this gradually as you scroll through the feed.

Only when you see the post on screen okay well this is once the post is already pulled from the relay to be brought to your global. That is not the issue the issue is the constant request from relays to keep loading post to your global feed as you scroll. Just because I choose to swipe up or down should not be constantly asking the relays to pull new posts & load them to my global. This is what is most likely causing the burning battery.
Creating a setting to set the requests manually will help fix this issue & will probably save you from having to do specific tweaks for specific devices at least I hope.

should not be constantly asking the relays to pull new posts & load them to my global

Maybe we need a specific feed for Global. Or you can try to use a proxy relay for Global. If you have 25 active relays for global, you will download the same note, replies, likes, zaps, user metadata, etc 25 times. Not only on global but for all feeds and features of the app. That's probably what's using the battery. The Global Feed with 25 relays will download each note 25 times (which is not a lot), but every reply, zap, like, report, user name etc will ALSO be downloaded 25 times for each.
We recommend keeping between for 8-12 relays.

Just because I choose to swipe up or down should not be constantly asking the relays to pull new posts & load them to my global

It's not asking for new posts when you scroll, just likes, zaps, reports, replies, and all the metadata related to the post you are viewing and all of the people involved in the same post. This has to happen otherwise you are not going to see people's names on the post, just @npub...

should not be constantly asking the relays to pull new posts & load them to my global

Maybe we need a specific feed for Global. Or you can try to use a proxy relay for Global. If you have 25 active relays for global, you will download the same note, replies, likes, zaps, user metadata, etc 25 times. Not only on global but for all feeds and features of the app. That's probably what's using the battery. The Global Feed with 25 relays will download each note 25 times (which is not a lot), but every reply, zap, like, report, user name etc will ALSO be downloaded 25 times for each.

Maybe I am not sure how that will work out I would just have to try & test it once it's implemented. I get it each relay added is getting the posts from each other. I guess as a temporary solution I can cut down the relays I am tied to but I really feel I shouldn't have to because I am trying to make my post reach out to all these relays. I could stop their feeds sure but if I wanted to keep them all going then yes it's a guzzler.
If I have to cutdown relays can there be a import export relay list option built into the app because I really don't want to have to manually type in all these relays again.

Meaning, even if we do design something that doesn't update the feed, it will still use significant bandwidth to download the items for each post.

Right when I choose to refresh it or set it to auto refresh like once every 24 hours for example that is fine.

It's not asking for new posts when you scroll, just likes, zaps, reports, replies, and all the metadata related to the post you are viewing and all of the people involved in the same post. This has to happen otherwise you are not going to see people's names on the post, just @npub...

Oh I thought you meant before when you scroll it is asking the relays to pull new posts then as they load onto your feed then the next request is for everything else loads like zaps, reports, replies, and all the metadata related to the post to load once it's fetched & on your feed.


feat: make both mavlog and mavlink series objects subscriptable

Codecov Report
Patch coverage: 100.00% and no project coverage change.

Comparison is base (71224c1) 100.00% compared to head (d8b1d29) 100.00%.

:exclamation: Current head d8b1d29 differs from pull request most recent head a06c000. Consider uploading reports for the commit a06c000 to get more accurate results

Additional details and impacted files
@@            Coverage Diff            @@
##              main       #17   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files            4         4
  Lines          146       150    +4
=========================================
+ Hits           146       150    +4

Impacted Files
Coverage Δ

pymavlog/core.py
100.00% <100.00%> (ø)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Get hit

:tada: This PR is included in version 1.4.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


HTTP Registry Support

At present, softlayer_cpi had been using file registry (user_data.json) as bosh-agent settings source,   in order to get rid of ssh connection between CPI and deployed VM, the first step we should do is to use HTTP registry as one of setting source options.

@jianqiu please resolve conflicts and I will review

Hi, @jianqiu,
Please rebase due to PR #195 merge and i will review and merge as needed.
Thanks,
Max
/cc @zhanggbj

Hi, @jianqiu,
Please rebase due to PR #195 merge and I will take a look.
Thanks,
Max
/cc @zhanggbj

Please rebase and resubmit and I will review.

@jianqiu what is the status on this. This seems stalled. I will close this tonight after scrum unless I see a response.


11_register_hosts.sh only provisions 1/3 worker nodes

Describe the bug
NODES_FILE has available 3 worker nodes for provisioning. When running 11_register_hosts.sh only 1/3 get provisioned and the other 2 remain in ready state:
[centos@provisionhost-0 dev-scripts]$ oc --config ocp/auth/kubeconfig get baremetalhosts -n openshift-machine-api
NAME                 STATUS   PROVISIONING STATUS      MACHINE                 BMC                         HARDWARE PROFILE   ONLINE   ERROR
openshift-master-0   OK       externally provisioned   ostest-master-0         ipmi://<IP_ADDRESS>:6230                      true
openshift-master-1   OK       externally provisioned   ostest-master-1         ipmi://<IP_ADDRESS>:6231                      true
openshift-master-2   OK       externally provisioned   ostest-master-2         ipmi://<IP_ADDRESS>:6232                      true
openshift-worker-0   OK       ready                                            ipmi://<IP_ADDRESS>:6233   unknown            true
openshift-worker-1   OK       ready                                            ipmi://<IP_ADDRESS>:6234   unknown            true
openshift-worker-2   OK       provisioned              ostest-worker-0-x9l2z   ipmi://<IP_ADDRESS>:6235   unknown            true

To Reproduce
Use a NODES_FILE with more than one worker nodes. Run 11_register_hosts.sh.
Expected/observed behavior
Only one worker node gets provisioned and the rest of them remain in ready state. I expect all worker nodes to get into provisioned state.
Additional context
NODES_FILE:
{
  "nodes": [
    {
      "name": "openshift-master-0",
      "driver": "ipmi",
      "resource_class": "baremetal",
      "driver_info": {
        "ipmi_username": "admin",
        "ipmi_password": "password",
        "ipmi_address": "<IP_ADDRESS>",
        "ipmi_port": "6230",
        "deploy_kernel": "http://<IP_ADDRESS>/images/ironic-python-agent.kernel",
        "deploy_ramdisk": "http://<IP_ADDRESS>/images/ironic-python-agent.initramfs"
      },
      "ports": [{
        "address": "52:54:00:0f:b1:66",
        "pxe_enabled": true
      }],
      "properties": {
        "local_gb": "0",
        "cpu_arch": "x86_64"
      }
    }
        ,
          {
      "name": "openshift-master-1",
      "driver": "ipmi",
      "resource_class": "baremetal",
      "driver_info": {
        "ipmi_username": "admin",
        "ipmi_password": "password",
        "ipmi_address": "<IP_ADDRESS>",
        "ipmi_port": "6231",
        "deploy_kernel": "http://<IP_ADDRESS>/images/ironic-python-agent.kernel",
        "deploy_ramdisk": "http://<IP_ADDRESS>/images/ironic-python-agent.initramfs"
      },
      "ports": [{
        "address": "52:54:00:03:a6:3d",
        "pxe_enabled": true
      }],
      "properties": {
        "local_gb": "0",
        "cpu_arch": "x86_64"
      }
    }
        ,
          {
      "name": "openshift-master-2",
      "driver": "ipmi",
      "resource_class": "baremetal",
      "driver_info": {
        "ipmi_username": "admin",
        "ipmi_password": "password",
        "ipmi_address": "<IP_ADDRESS>",
        "ipmi_port": "6232",
        "deploy_kernel": "http://<IP_ADDRESS>/images/ironic-python-agent.kernel",
        "deploy_ramdisk": "http://<IP_ADDRESS>/images/ironic-python-agent.initramfs"
      },
      "ports": [{
        "address": "52:54:00:35:36:28",
        "pxe_enabled": true
      }],
      "properties": {
        "local_gb": "0",
        "cpu_arch": "x86_64"
      }
    }
        ,
          {
      "name": "openshift-worker-0",
      "driver": "ipmi",
      "resource_class": "baremetal",
      "driver_info": {
        "ipmi_username": "admin",
        "ipmi_password": "password",
        "ipmi_address": "<IP_ADDRESS>",
        "ipmi_port": "6233",
        "deploy_kernel": "http://<IP_ADDRESS>/images/ironic-python-agent.kernel",
        "deploy_ramdisk": "http://<IP_ADDRESS>/images/ironic-python-agent.initramfs"
      },
      "ports": [{
        "address": "52:54:00:c9:87:88",
        "pxe_enabled": true
      }],
      "properties": {
        "local_gb": "0",
        "cpu_arch": "x86_64"
      }
    }
        ,
          {
      "name": "openshift-worker-1",
      "driver": "ipmi",
      "resource_class": "baremetal",
      "driver_info": {
        "ipmi_username": "admin",
        "ipmi_password": "password",
        "ipmi_address": "<IP_ADDRESS>",
        "ipmi_port": "6234",
        "deploy_kernel": "http://<IP_ADDRESS>/images/ironic-python-agent.kernel",
        "deploy_ramdisk": "http://<IP_ADDRESS>/images/ironic-python-agent.initramfs"
      },
      "ports": [{
        "address": "52:54:00:4f:9a:63",
        "pxe_enabled": true
      }],
      "properties": {
        "local_gb": "0",
        "cpu_arch": "x86_64"
      }
    }
        ,
          {
      "name": "openshift-worker-2",
      "driver": "ipmi",
      "resource_class": "baremetal",
      "driver_info": {
        "ipmi_username": "admin",
        "ipmi_password": "password",
        "ipmi_address": "<IP_ADDRESS>",
        "ipmi_port": "6235",
        "deploy_kernel": "http://<IP_ADDRESS>/images/ironic-python-agent.kernel",
        "deploy_ramdisk": "http://<IP_ADDRESS>/images/ironic-python-agent.initramfs"
      },
      "ports": [{
        "address": "52:54:00:9f:b9:ca",
        "pxe_enabled": true
      }],
      "properties": {
        "local_gb": "0",
        "cpu_arch": "x86_64"
      }
    }
        ]
}

Look in ocp_install_env.sh. The worker count is set to 1 by default since that’s how many worker VMs we create by default. You can change it for a new deployment.
You can change this in an existing deployment too. You need to scale the worker MachineSet from 1 to 3.
I’m on mobile so double check the name of the machineset.
$ oc get machinesets -n openshift-machine-api
$ oc scale machineset -n openshift-machine-api ostest-worker-0 —replicas=3

Look in ocp_install_env.sh. The worker count is set to 1 by default since that’s how many worker VMs we create by default. You can change it for a new deployment.
You can change this in an existing deployment too. You need to scale the worker MachineSet from 1 to 3.
I’m on mobile so double check the name of the machineset.
$ oc get machinesets -n openshift-machine-api
$ oc scale machineset -n openshift-machine-api ostest-worker-0 —replicas=3

Thanks! Changing the replicas count did the trick.

great! Thanks