Unable to publish package (500 Internal Server Error)

When attempting to publish a package the following is reported:
HTTP/1.1 500 Internal Server Error
Connection: keep-alive
Content-Length: 141
Content-Type: text/html
Date: Sun, 23 Apr 2023 12:02:35 GMT
Server: nginx/1.23.2

    <title>Internal Server Error</title>

Internal Server Error

This appears to be happening consistently for moderately sized SPK files (50-75 MB) and does not appear to be based on time of day (server load).

@Diaoul, @ymartin59, any thoughts on this?

This leaves the question of the original issue of the uploaded file from the stream not being present at the expected path during publishing. From examining the server code I see that publishing involves:

When a new package is posted, the server creates an SPK object. This object is based on the SPK class and is passed the request data as its argument
The server creates a Package object if one doesn't already exist, based on the package name in the request
The server parses the SPK content to extract the necessary information such as firmware, package name, version, etc.
The server then creates a Version object for the package, based on the package name and version number in the request. If a Version object with the same version number already exists, the new version is not created
The server also creates a Build object, which is not active by default. If a Build object with the same architectures exists, a conflict error is returned
The server saves the files by (a) creating a new directory for the package if there is a new Package object, (b) creating a new version sub-directory for the package if there is a new Version object and (c) saves the named spk file (using the SPK object) to this version sub-directory

Now the error seen: FileNotFoundError: [Errno 2] No such file or directory: '/data/lidarr/11/ suggests that in attempting to save the named spk file, it is unable to do so. It is possible that while the path '/data/lidarr/' does exist (since there are other versions present), the version sub-directory is not. A missing sub-directory may cause the save to fail.

I suspect that there may have been some sort of corruption when the first package from lidarr v11 was uploaded. The admin interface reports the following:

However, when we look at the webpage it shows the following:

This version should not be visible since it is disabled in the admin interface. The application icon in the webpage was also not visible (before v12 was uploaded), which suggests that the code which runs when the new Version object is present was not initially run.

I was publishing some other packages and I see the same corruption happening with prowlarr v6. The admin interface reports the following:

However, the webpage at https://synocommunity.com/package/prowlarr shows the following:

Note the missing icon as I indicated above with lidarr v11 when no newer version is present. I believe this happens because this code is not being executed:
https://github.com/SynoCommunity/spkrepo/blob/76951e0e5850504b16c9164b97a23883488269d6/spkrepo/views/api.py#L260-L263

Yes there must've been a failure at a time that did create the version but not the appropriate directories, causing subsequent ones to fail.
Maybe the best move is to delete the version and re-upload everything?

Yes there must've been a failure at a time that did create the version but not the appropriate directories, causing subsequent ones to fail.
Maybe the best move is to delete the version and re-upload everything?

Thanks for your feedback. Before deleting the version can you check:

The application log to see if it is reporting the same FileNotFoundError: [Errno 2] No such file or directory  errors
The file system to determine if the path /data/prowlarr/6/ exists. If so, what’s in that directory?

if this is the case then I’d request that the prowlarr v6 be deleted. I don’t think I have permissions to do that.

hey @Diaoul, any feedback on the above?
In the meantime, would you be so kind as to delete this version:

@mreid-tt I have deleted lidarr v11 for you

thanks @publicarray, I was hoping we could check the application logs and the file system for prowlarr to confirm the theory above. But if this is a hassle we can just delete the problematic version:

@mreid-tt the folder was never created...

publicarray@vps-b34822e4 /e/synocommunity> ls data/prowlarr
4/  5/

@mreid-tt the folder was never created...
publicarray@vps-b34822e4 /e/synocommunity> ls data/prowlarr
4/  5/

Thanks for the confirmation.

I have deleted the broken version

Great! I'll proceed with a fresh build and upload.

We had problems uploading before (due to load) but never like this (missing folders/files)...

Hmm, okay. Perhaps a possible fix would be to adjust the code to verify that a folder is created (for both the package and version functions) before committing a new entry to the database.

my2 cents but maybe only upload one file at a time. IIRC phython/postgresql are bad at working with multiple upload requests at the same time. This smells like a race condition

I have deleted the broken version

Can you confirm this was done? I'm still seeing this:

hm maybe build them without publishing ongithub and download the result. Then manually create the upload request with httpie. I know it's a workaround but we can also test if you for example upload all packages at once if this courses this problem again.

Maybe your page is cached.

we got one :)
publicarray@vps-b34822e4 /e/synocommunity> ls -l data/prowlarr/6
total 55524
-rw-r--r-- 1 root root    44917 Apr 27 12:25  icon_256.png
-rw-r--r-- 1 root root     7211 Apr 27 12:25  icon_72.png
-rw-r--r-- 1 root root 56801280 Apr 27 12:25 'prowlarr.v6.f15047[apollolake-avoton-braswell-broadwell-broadwellnk-broadwellntbap-bromolow-cedarview-denverton-dockerx64-geminilake-grantley-purley-kvmx64-v1000-x86-x86_64].spk'

Hi @Diaoul, now that we have conclusively proven that the issue is rooted in the version folder not being created, this suggests that this code block is not executed:
https://github.com/SynoCommunity/spkrepo/blob/66c4ac66af3b0e8e9adc1c9ef92df3b599baab73/spkrepo/views/api.py#L260-L263
If this is the case, how could we make the function more resilient? Should the os.mkdir() be put into a try block to catch any exceptions? Should we have a line right afterwards to verify that the path exists? The worry I have is that it thinks the command was successful. This is further confirmed in the log which reports the following back in GitHub:
  Package published successfully
  {"package": "prowlarr", "version": "20230426-6", "firmware": "6.1-15047", "architectures": ["evansport"]}

Why might this be the case? Are the filesystem actions for directory creation and build.save() being executed asynchronously and never complete even though the function believes it was executed? For the sequence of events, when exactly is the new version added to the database?
If it is not asynchronous, are multiple function calls trying to create the same version when GitHub runs parallel build operations? If so, why wouldn't at least one function call be completed? That way even if the other parallel calls fail (or do nothing because the folder exists), the successful one should create the folder, shouldn't it?
Interested in your thoughts before I create a PR so that the solution is effective.

My guess is that if a version exist in the db, but the transaction isn't finished yet and another one is created at the same time we get this condition. Not sure why not at least one mkdir command is executed though. Maybe both transactions exit early because a version exists already?

I also noted that if there is an exception files are removed so what happens when a file is created and at the same time removed by a different request?

I also noted that if there is an exception, files are removed so what happens when a file is created and at the same time removed by a different request?

Ahh! That's a good point, you are referring to this part of the code:
https://github.com/SynoCommunity/spkrepo/blob/66c4ac66af3b0e8e9adc1c9ef92df3b599baab73/spkrepo/views/api.py#L265-L272
So, it's possible it was created just fine but there was some sort of exception (perhaps even in creating the icons or saving the build), then the entire version folder would be dumped. If this is the case then the version should also be removed from the database (or not added if this is done later). Need to better understand the database interactions to confirm.

So I shared our problem with ChatGPT and it suggested something like this:
import threading

# create a lock to ensure that only one request can create a new version at a time
version_lock = threading.Lock()

def create_new_version(package, version_number):
    create_version = False
    with version_lock:
        version = Version.query.filter_by(package_id=package.id, version=version_number).first()
        if version is None:
            version = Version(package=package, version=version_number)
            db.session.add(version)
            db.session.commit()
            create_version = True
    if create_version:
        # create the directory for the new version
        version_path = os.path.join(current_app.config['DATA_PATH'], package.name, str(version_number))
        os.makedirs(version_path, exist_ok=True)
    return version

It's a different structure from the code currently there but may serve as inspiration for a possible solution.

Thinking a bit about this, I am not sure why we need a clean up the directory structure :shrug: We could very well make sure the path always exists.
However, this could hide another consistency problem (like in #100)

I think I agree with @Diaoul here, using locks just overcomplicates things. I think just checking if there is an exception like "version already exists" and continuing the code anyway rather than cleaning up the folders is enough in this case. I'd like to mention that what I think is happening is a hypothesis and not proven yet.
There could have been an issue with the redis or database connection. Unfortunately, there aren't time codes in the app logs.
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not connect to server: Connection refused
 	Is the server running on host "db" (<IP_ADDRESS>) and accepting
 	TCP/IP connections on port 5432?

redis.exceptions.ConnectionError: Error while reading from redis:6379 : (104, 'Connection reset by peer')
Exception possibly due to cache backend.

Thanks for the additional perspectives. Taking a step back the issue is really about the file system and not the database. Re-looking at the code we have these operations to create folders:
https://github.com/SynoCommunity/spkrepo/blob/66c4ac66af3b0e8e9adc1c9ef92df3b599baab73/spkrepo/views/api.py#L259
My understanding is that if the folder already exists, it will throw a FileExistsError. This is another possible reason for the try block failing and a more likely scenario for multiple workers adding a new build. I've done another PR with a far simpler fix for consideration in #101.

Thinking a bit about this, I am not sure why we need a clean up the directory structure 🤷 We could very well make sure the path always exists.

I actually like the elegance of cleaning up the structure on an exception so you maintain a clean filesystem. I mean we can adjust the code to something like this:
if create_package:
    dir_path = os.path.join(data_path, package.name)
    if not os.path.exists(dir_path):
        os.mkdir(dir_path)

But I really believe now that the exception on the try block is more likely caused by the folder already existing which can easily be mitigated.

@publicarray, looking back at the application logs you kindly provided, I believe this new theory is correct. In the logs we first see this error:
[2023-04-16 12:32:53 +0000] [7] [ERROR] Error handling request /packages
Traceback (most recent call last):
  File "/usr/src/app/spkrepo/views/api.py", line 261, in post
    os.mkdir(os.path.join(data_path, package.name, str(version.version)))
FileExistsError: [Errno 17] File exists: '/data/lidarr/11'

Immediately after that we have these errors and similar:
[2023-04-16 12:33:40 +0000] [10] [ERROR] Error handling request /packages
Traceback (most recent call last):
  File "/usr/src/app/spkrepo/views/api.py", line 264, in post
    build.save(spk.stream)
  File "/usr/src/app/spkrepo/models.py", line 357, in save
    with io.open(
FileNotFoundError: [Errno 2] No such file or directory: '/data/lidarr/11/lidarr.v11.f42661[comcerto2k].spk'

I believe this supports the theory that:

Two workers attempt to create a directory at '/data/lidarr/11'
One succeeds and the other fails
The failing one deletes the directory path
Other workers saving builds fail because the path does not exist

I am not sure why we would end up publishing multiple packages at the same time, maybe we can fix that instead? The code is full of race conditions and absolutely not ready to handle parallel uploads of the same packages.
Also worth noting that it seems it's the first time it occurs in close to 8 years of this code being in production. Unless we're confident the code will not break elsewhere when introducing a fix, I would be OK with a one-time procedure to manually fix this and document it somewhere :grimacing:

I am not sure why we would end up publishing multiple packages at the same time, maybe we can fix that instead? The code is full of race conditions and absolutely not ready to handle parallel uploads of the same packages.

From my understanding, I believe we have been doing this for over two years based on the age of the wiki page... Publish a package from Github. To my knowledge there isn't a way to tell the GitHub workflow to not run in parallel. As such, once we use it, there is the potential to be sending multiple builds.

Also worth noting that it seems it's the first time it occurs in close to 8 years (I've lost count) of this code being in production. Unless we're confident the code will not break elsewhere when introducing a fix, I would be OK with a one-time procedure to manually fix this and document it somewhere 😬

I understand the hesitation in making such a fundamental change but I'd like to share another perspective:

I don't believe that this is the first time this has happened. In the application logs shared with me that had entries between 2023-02-19 and 2023-04-25 there was one other version directory missing '/data/owncloud/9'. More likely what has been happening  is that because the GitHub publish function did not check for errors (which it now does as part of #5701) the uploads appeared to be successful when they were not (see https://github.com/SynoCommunity/spksrc/pull/5619#issuecomment-1528923235)
You are correct that there is the possibility of the code breaking elsewhere but from a database perspective you have referential integrity constraints built-in to the Package and Version models so should there be a duplicate addition, it would cleanly fail. For the addition of the Build, there is the possibility of the object being linked to the incorrect Version object and I can't think of a way to address this without locks
As far as the filesystem, if the folder already exists we should be fine with the proposed fix. If we try to save duplicate icons, based on the statement construction for icon.save() the existing ones should be overwritten which should also be okay
As far as manually fixing it, the way I've seen it done is to delete the version in the admin panel and re-upload from scratch. While this can work, not all devs have admin permissions to do this. Additionally, it does not prevent it from happening again with another GitHub workflow (unless you manually re-run each failed job one at a time)

Thanks for the additional context, it makes sense to try with this then :+1:

For some history. The first Ci used travis and circleCI processed one arch at a time. Also it was expected that every dev build on their own devices before publishing so that they would test the build locally as well (that hasn't changed) but with GitHub actions it became easier and we implemented the publishing part as well. So if you build locally and test the one or two architectures you don't need to build all of them to publish. Lowering the barrier a bit.


node-pre-gyp support

binaries will be built on demand for linux and windows
just bump the version number in package.json (e.g. npm version patch), commit+push, and appveyor and travis will test and deploy new
prebuilt binaries as needed
binaries are “maybe built” via a posttest hook for minimal
duplication in CI configs
binaries live on S3 behind https://bin.mongodb.org/ (all in the
devops production environment, fully managed and maintained by them)
which is fronted by CloudFront (CDN)
using mongodb-js fork of node-pre-gyp for easier tooling integration

Good to go when this appveyor build finishes for iojs: https://ci.appveyor.com/project/imlucas/bson-ext/build/1.0.34

fixes mongodb/js-bson#120 mongodb/js-bson#118 mongodb/js-bson#114 mongodb/js-bson#111 iojs/io.js#1245

This change has broken the bson for me across both Ubuntu and Mac. It was working perfect before in version 0.1.0, but now bson is pulling in the new version 0.1.1 since it is requesting version ~0.1 and the BSON module is now failing with errors across both local Mac envs, and Ubuntu machines I'm working with.

what error are you seeing as I cannot repro

Here is the root issue: https://github.com/christkv/bson-ext/issues/2

It's seems I have this problem related to https://github.com/Automattic/mongoose/issues/2905

@nathanpeck did you resolve the issue?

Yes, they fixed the issue by releasing a new version of the package that didn't have a precompiled Mac OS X binary inside of it. You should be able to npm cache clean; npm rm -rf mongoose; npm install to fix it.

@nathanpeck thanks for the answer


add querybuilder auto-join support for update queries

Is your feature request related to a problem? Please describe.
The querybuilder auto-join support is really great!  Trying to run as part of select works fine, i.e.
qb.select("*").where({activity: {enabled: true}})
select "e0".* from "tasks" as "e0" left join "activities" as "e1" on "e0"."activity_id" = "e1"."id" where "e1"."enabled" = $1
but changing to use update, i.e.
qb.update({someField: someValue})
produces an error:
    TypeError: Cannot read property 'slice' of undefined
at ObjectCriteriaNode.autoJoin (node_modules/mikro-orm/dist/query/CriteriaNode.js:205:37)
      at ObjectCriteriaNode.process (node_modules/mikro-orm/dist/query/CriteriaNode.js:159:26)
      at node_modules/mikro-orm/dist/query/CriteriaNode.js:163:39
          at Array.reduce (<anonymous>)
      at ObjectCriteriaNode.process (node_modules/mikro-orm/dist/query/CriteriaNode.js:161:42)
      at QueryBuilder.where (node_modules/mikro-orm/dist/query/QueryBuilder.js:83:99)

Describe the solution you'd like
The same auto-join support for update queries.
Describe alternatives you've considered
Going to try the hand-rolled joins next

Closing as implemented via #537, which is now merged in v4 (dev branch). Subscribe here for updates #527.


HashSetvBitSetTest diamond operator

Use Java 7's diamond operator to make the code a tad more elegant, as
done in the rest of the codebase.

Coverage increased (+0.04%) to 94.693% when pulling 8e891eb3599663f72499eae421e60f12025adc89 on mureinik:diamond into d61090ab64dcf411d7f1db0b6a9890af74238580 on apache:master.

Coverage increased (+0.04%) to 94.693% when pulling 8e891eb3599663f72499eae421e60f12025adc89 on mureinik:diamond into d61090ab64dcf411d7f1db0b6a9890af74238580 on apache:master.

Thanks!


Type not inferred properly when functions are reduced iteretively

TypeScript Version:
2.7.0-@next
Search Terms:
"infer reduced type",  "typescript compose functions"
Code
const addPropA = <T extends object>(obj: T) => Object.assign({}, obj, { a: 'a' })
const addPropB = <T extends object>(obj: T) => Object.assign({}, obj, { b: 'b' })

// this works: const constructed: { c: string; } & { b: string; } & { a: string; }
const constructed = addPropA(addPropB({ c: 'c' }))

// this doesnt infer the type properly: const reduced: { c: string; }
const reduced = [addPropA, addPropB].reduce((acc, fn) => fn(acc), { c: 'c' })

Expected behavior:
const reduced: { c: string; } & { b: string; } & { a: string; }
Actual behavior:
const reduced: { c: string; }
Playground Link:
playground
Related Issues:

There's no reasonable way we could model this in the type system -- addPropA and addPropB happen to be order-insensitive but there's no way to observe this property of them at a distance, and the output type of any f(g(h(i(x))) sequence necessarily depends on the order of invocation. We'd have to have some way to tell apart reduce from reduceRight and describe their ordering behavior in the type system, which is really too heavy of a lift to consider given the other things still on our plate.


Library: Make section titles on Library landing page linkable

Hi,
The section headers of the Library home page (https://landportal.org/library) currently do not link to anything.
The Featured Publications one is fine that it doesn't link, but ideally and more intuitively the other two should link to pages where you can find more than just the three highlighted items appearing on the landing page.
More specifically:
'Recently published':
The 'Recently Published' header should link to a page (https://landportal.org/library/search) with all the Library resources in the order of most recent publication date -> oldest publication date.
'Latest additions':
The 'Latest additions' header should link to a page (the same as above), the search page, with the all Library resources sorted in the order of most recently updated -> least recently updated. This would be the information that is also used in the LL admin view (https://landportal.org/admin/content/library) to show the most recently updated on top.
Thanks!

Hi @lisettemeij
please check on https://dev.landportal.org/library by clicking on 'latest aditions'.
Could this be a solution? (We still need to add some space between the content and the 'featured providers' right block).
Let me know if I should continue with the last publications and with the other 3 languages or if I should look for a different solution.
Thanks.


building error: excess elements in struct initializer at iree/compiler/Dialect/Flow/Transforms/DispatchLinalgOnTensors.cpp:1115

If you are an IREE team member, please consider doing further triage: remove the "help wanted" label, add other relevant labels, and redirect to a more appropriate assignee.
Describe the bug
LinalgLoopDistributionOptions in llvm/mlir contains only two fields, but the one in  DispatchLinalgOnTensors.cpp:1115 passes three fields into the structure initializater.
To Reproduce

download iree source following https://google.github.io/iree/building-from-source/getting-started/
cmake -B $builddir -DCMAKE_BUILD_TYPE=RelWithDebInfo .
-DIREE_ENABLE_EMITC=on
cmake --build ../iree-build/

The commit snapshot of iree is: 7d9fd36b9f33139be19b351544e62918ce1ac773
The commit snapshot of iree/third_party/llvm-project is: c2f819af73c54a8cf923e5a25099ca95dbe76312
Additional context
Add any other context about the problem here.

after re-syncing the git, it could build successfully. just close the bug.


Huge memory consumption of "Shared.make_key" function

Recently I run my project under memprof.
Results show that "Shared.make_key" consumes huge amount of memory.
@/Users/o.babin/Projects/tdg/.rocks/share/tarantool/metrics/collectors/shared.lua:42, line 45: 7408397
  177807041       0
@/Users/o.babin/Projects/tdg/.rocks/share/tarantool/metrics/collectors/shared.lua:42, line 43: 7407732
  474094848       0

It's the third place in top after JSON parsing and graphql execution. I suggest to optimize this place anyhow. Seems the root of problem is "histogram.observe" usage.

It requires to rewrite metrics arch, so it's not so easy right now. Very gently closing to decrease infonoise.


Missing ISCSI Packages

Hi,
i'm trying to rollout iscsigw role via ceph-ansible, but it fails with the following message:
(item=tcmu-runner) => {"changed": false, "item": "tcmu-runner", "msg": "No package matching 'tcmu-runner' found available, installed or updated", "rc": 126, "results": ["No package matching 'tcmu-runner' found available, installed or updated"]}

 (item=ceph-iscsi-config) => {"changed": false, "item": "ceph-iscsi-config", "msg": "No package matching 'ceph-iscsi-config' found available, installed or updated", "rc": 126, "results": ["No package matching 'ceph-iscsi-config' found available, installed or updated"]}

 (item=ceph-iscsi-cli) => {"changed": false, "item": "ceph-iscsi-cli", "msg": "No package matching 'ceph-iscsi-cli' found available, installed or updated", "rc": 126, "results": ["No package matching 'ceph-iscsi-cli' found available, installed or updated"]}

I have configured ceph community repositories and ceph version is mimic:
ceph_origin: repository

ceph_repository: community

ceph_mirror: http://download.ceph.com
ceph_stable_key: https://download.ceph.com/keys/release.asc
ceph_stable_release: mimic
ceph_stable_repo: "{{ ceph_mirror }}/debian-{{ ceph_stable_release }}"

Any Ideas why the packages are missing?
Thanks
Kilian

AFAIK only RHCS ceph repository contains those packages (out of the box). The community repository doesn't.
Currently ceph-ansible manages those packages installation but only via dev repository
That said you can still do it either:
1/ using containerized deployment (because iscsi packages are included in the ceph container image)
2/ add the extra repositories manually

https://download.ceph.com/ceph-iscsi/latest/rpm/el7/
https://shaman.ceph.com/api/repos/tcmu-runner/master/latest/centos/7/
https://shaman.ceph.com/api/repos/python-rtslib/master/latest/centos/7/

@dsavineau i added the repositories manually and i was able to deploy the ceph iscsi gateway ansible role - thanks for your help ;)

Sorry for posting on a closed issue, but I am hitting the same problem, and I cannot figure out how to add the extra repositories manually.
@discostur - could you please help a yum-noob and post the steps necessary for adding the repos posted by @dsavineau ?
Thank you!

if anyone is searching the old ceph-iscsi packages - you can now find them here:
http://download.ceph.com/ceph-iscsi/2/rpm/el7/noarch/


Add hakunapi logo (version 1)

I wonder what is the right place for the logo image file.

Added a commit to your branch adding the image to the beginning of readme.md, how does it look?
https://github.com/jratike80/hakunapi/tree/main

Added a commit to your branch adding the image to the beginning of readme.md, how does it look?

Rather big, how about scaled into width=350?


43 port mesh controller pkg to ros2

This PR ports the mesh controller plugin to ROS2.

fixes #43


folder for schema

Hi,
on the documentation you are point for new schema to
/var/config/rest/iapps/as3-forms-lx/schemas
this folder doesn't exist on version 15.1.2 , what is the right folder now to write the json schema file?
Regards,

This is an outdated part of the documentation that we need to update. Thanks for bringing it to our attention! Schema files for user-defined types should go in the same template set directory as the templates themselves. An example of this can be seen in /var/config/rest/iapps/f5-appsvcs-templates/templates/bigip-fast-templates with bigp-fast-templates being a template set and f5.json being a user-defined types schema file.

Thanks for your input. We are aware of this and will be fixing this in the next release in late-September.


[Question]: Does llama-index settings even work?

Question Validation

[X] I have searched both the documentation and discord for an answer.

Question
My code-:
#########################################################
reader = SimpleDirectoryReader(
input_files=["/home/seceon/opensearch_lm_index/textdocs/aktripathi.txt"]
)
docs = reader.load_data()
text = docs[0].text
summarizer = TreeSummarize()
embeddings = HuggingFaceEmbedding()
text_split = RecursiveCharacterTextSplitter(chunk_size=300, chunk_overlap=190)
text_splitter = LangchainNodeParser(text_split)
Settings.text_splitter = text_splitter
nodes = text_splitter(documents)
llm = HuggingFaceLLM(
model_name=model_id,
tokenizer_name=model_id,
context_window=3900,
max_new_tokens=256,
model_kwargs={"torch_dtype": torch.float16},
# tokenizer_kwargs={},
generate_kwargs={"temperature": 0.3,  "top_p": 0.95},
device_map="auto",
)
Settings.llm = llm
Settings.embed_model = embeddings
Settings.context_window=4096,
Settings.num_output=1000,
Settings.chunk_overlap_ratio=0.3,
Settings.chunk_size_limit=None
response = summarizer.aget_response("Summarise the data in the document", [text])
print(response)
####################################################################
Error-:
ValueError:

Could not load OpenAI model. If you intended to use OpenAI, please check your OPENAI_API_KEY.
Original error:
No API key found for OpenAI.
Please set either the OPENAI_API_KEY environment variable or openai.api_key prior to initialization.
API keys can be found or created at https://platform.openai.com/account/api-keys
To disable the LLM entirely, set llm=None.
Does the settings.llm even work?
Reference-:
https://docs.llamaindex.ai/en/stable/examples/response_synthesizers/tree_summarize/

@Aekansh-Ak You created the summarizer before changing the settings. Try swapping the order
More technically

creating the summarize will pull from the current global settings to create an LLM object
at this point in change, the settings haven't changed, so it pulls openai
then you modify the settings
however, the summarizer is already created with an LLM object in memory attached to it, it is not updated

Thanks! That worked.
For this line-:
response = await summarizer.aget_response("who is Paul Graham?", [text])
I am getting-:
response = await summarizer.aget_response("who is Paul Graham?", [text], llm=None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
SyntaxError: 'await' outside function

Thanks Dosu bot, the llama-index documentation clearly can't be just blindly followed.

@Aekansh-Ak In python, await can only be used if you are running in an async function. Our examples in the docs mostly come from python jupyter notebooks, which are already running in async by default, and therefore can use await.

Okay, understood.
How can I do streaming in this, it doesn't work-:
summarizer = TreeSummarize()
async def main():
response = await summarizer.aget_response("Could you summarize the given context? Return your response which covers the key points of the text.", [text], streaming = True)
for r in response:
print(str(r), end="", flush=True)
AttributeError: 'str' object has no attribute 'response_gen'

The streaming kwarg goes in the constructor
TreeSummarize(streaming=True)


Spike Prime: missing "foreach" block in expert toolbox

Please check if this programming block can be supported after all.

Could not be reproduced


Normalize order of terms in \equals

During simplification, the backend should normalized the order of terms in \equals(_, _) to remove duplicate clauses.

We should not spread this information out over the code, but make the decision in one place. There should be a function in Kore.Step.Simplification.AndTerms which is used there and in Kore.Step.Simplification.Equals as required.


Not able to get value for AtKey who doesn't have a shared with

Describe the bug
while running @ mosphere pro mobile , I noticed many keys were failing with the message key does not exist in keystore .
i.e
Exception: @:sentHistory_v2.mospherepro@daily3bossanova does not exist in keystore
SEVERE|2022-01-31 15:48:51.205563|AtContactsImpl|Invalid atsign contact found @AtKey{key: atconnections.thethingaboutmyfolksdry.daily3bossanova.at_contact.mospherepro, sharedWith: null, sharedBy: daily3bossanova, namespace: mospherepro, metadata: Metadata{ttl: null, ttb: null, ttr: null,ccd: null, isPublic: false, isHidden: false, availableAt : null, expiresAt : null, refreshAt : null, createdAt : null,updatedAt : null,isBinary : false, isEncrypted : null, isCached : false, dataSignature: null, sharedKeyStatus: null}, isRef: false} : Exception: @:atconnections.thethingaboutmyfolksdry.daily3bossanova.at_contact.mospherepro@daily3bossanova does not exist in keystore
This was happening only for atKeys which does not have sharedWith.
To Reproduce
Steps to reproduce the behavior:

Run @ mosphere pro
contacts , groups and sent history will fail to load.

Expected behavior
key should return value.
Smartphone (please complete the following information):

Device: [e.g. iPhone11]
OS: [e.g. iOS14.1]
Browser [e.g. stock browser, safari]
Version [e.g. 59]

Were you using an @application when the bug was found?

@ mosphere pro mobile , @ rrive

Which versions of client libraries are you using?

Message ID: @.***>

at_client_mobile: ^3.1.4
at_commons: ^3.0.3
at_contact: ^3.0.3
at_lookup: ^3.0.6
at_onboarding_flutter: ^3.1.3
at_common_flutter: ^2.0.4
at_contacts_flutter: ^4.0.0
at_contacts_group_flutter: ^4.0.0

@VJag
I created this AtKey and then performed get operation on it.

AtKey key = AtKey()
..key = ‘sentHistory_v2’
..sharedBy = ‘atsign’
..metadata = Metadata();

By overriding at_utils to this branch , it's working fine.

(Fixed by https://github.com/atsign-foundation/at_tools/pull/141 )


[Bug Report] How can I style el-input ?

Element UI version
2.4.8
OS/Browsers version
Ubuntu 16.04
Vue version
2.5.17
Reproduction Link
https://jsfiddle.net/mmx38qxw/6594/
Steps to reproduce
I have a trouble. When I try to style el-input element, I can't do this. I need a border-bottom: 2px solid yellow in my input field, but styles are aplied to all el-input element. When I used google chrome devtools I saw that el-input generate this structure (divs with classes): .el-form-item > .el-form-item__content > .el-input > input.el-input__inner
What is Expected?
I need style only input element in my form. I need a border-bottom: 2px solid yellow in my input field.
What is actually happening?
Styles are aplied to all el-input element, but not to input. When I used google chrome devtools I saw that el-input generate this structure (divs with classes): .el-form-item > .el-form-item__content > .el-input > input.el-input__inner

You can use Deep Selectors.
Visit this link for details: https://vue-loader.vuejs.org/guide/scoped-css.html#deep-selectors

You can use Deep Selectors.
Visit this link for details: https://vue-loader.vuejs.org/guide/scoped-css.html#deep-selectors

Thanks! I'll try it


there's a solution ?

How to solve the error with the skins, in the agar.io client, or you can no longer with the update

client sided

@AlexHGaming  but in the client agar.io there is no solution: "(

@Gasvash  Yes there is,
Find a agar.io client that supports skin ,
I recommend https://github.com/Luka967/Cigar/issues, host it ,
if you can pay for hosting, I recommend using Hostinger,
If you want free hosting use 000webhost .
000webhost is pretty shitty, but You can still run the server.
Upload skins to that client and play. That's the best solution.
Agar.io Skins has been broken , so that is the best solution I can think of.

I mean, you could use my client, but eh.

@Gasvash
I had the same problem and replace the following line # 289 PacketHandler.js
delete -> skinName = (inner == "r")? this.getRandomSkin (): inner;
replace -> skinName = (inner == "r")? "%" + this.getRandomSkin (): "%" + inner;
I hope it helps you

@Agarian  Sorry to bother you. You can show the time in the leaderboard

@Gasvash  CLose?

@CodeFlush You don't understand... 000webhost is a bad idea for hosting anything, they inject advertising banners into the page which annoys the users and interferes with the code. According to some reviews online, they also add redirects to their advertising pages. GitHub Pages is a way to go.

@ZfsrGhS953 I am understanding with what you are saying, but 000webhost is like the freeonly source for hosting php and being non HTTPS. I haven't seen github pages work with PHP on my perspective of view. Of course I have learnt from my mistake a couple of months back to go with github pages for a useless calculator, but that's just HTML and JS. Dang if only there could be 1 website which provides free web hosting with no sleep or ads or watermarks and language supportation.

@AlexHGaming Yes, PHP can't be ran in Github Pages, but I think that @CodeFlush was talking about hosting a custom client. PHP is not necessary for that - everything in Cigar can be converted to JS.

@AlexHGaming As for "being non HTTPS", what problems do you experience with using that on GitHub Pages?

@ZfsrGhS953 https blocks the connection because github pages force the website to be https. Making it secure makes it worse

Exactly ^^^^^^

@AlexHGaming Just turn it off then.

@ZfsrGhS953 the problem with turning it off, is that you can't. It's enforced to. Although it does make the site more secure and safer, https thinks that an agario clone would be insecure

@AlexHGaming Apologies, I thought there's a way to disable HTTPS by changing your domain from the default one, but for some reason when I try to change it gives me an error.

Which is why github pages is bad on other sides.

@AlexHGaming Yep, I can certainly tell you have tried and failed to solve the following problem:
Given:
FreeDNS
GitHub Pages
Do:
Make them work together.

0fees.us is a pretty good PHP webhost with unlimited bandwidth & storage & parked domains & subdomains & addon domains and free online file manager. the only bad thing is only 500000 visits per day..


JerryScript

Adding https://jerryscript.net/ would be nice, especially to see how this resource constrained version compares to other engines.

If someone writes the eshost runtime for it, sure.
https://github.com/test262-fyi/eshost/tree/master/runtimes

See https://github.com/test262-fyi/eshost/pull/17 for a full example of adding a new engine to eshost.


Positron: init at 2024.08.0-31

Description of changes
Positron is a new IDE made by Posit (formerly known as RStudio) forked from VS Code and with a focus on data science.
I used the .deb package as source. I tested this in a nix-shell on non-NixOS with R and Python interpreters, basic functionality seems to be working well:
nix-shell -I nixpkgs=. -p positron R rPackages.dplyr python312 python312Packages.ipykernel python312Packages.polars

I leave this as a draft for now, as I'm sure this expression could be improved.
Also, would we need a positronWrapper as we have a rstudioWrapper?
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Doesn't work on NixOS as it is, "NixOS cannot run dynamically linked executables intended for generic
linux environments out of the box."

Hi @kupac thanks for the feedback, I tried in a vm with autoptachelfHook to patch the binary, and get auto-patchelf: 0 dependencies could not be satisfied however when trying to run positron, nothing happens...

I will close this PR and continue work on a build from source.


Convert wiq to rest api

Obsolete. Work in this PR has been moved over to https://github.com/igoravl/TfsCmdlets/pull/103


Fix typo worktem --> work item

[ ] What does this PR do? [Mandatory]
fix typo
[ ] What issue/task does this PR references? [Mandatory]

[ ] Are the tests Included? [Mandatory]

[ ] Is the documentation Included?

[ ] Are the Release Notes included?

[ ] Is/Are the @mention(s) to reviewer(s) included

[test]

@apupier Your image is available in the registry: Run docker run -e PROXY_PASS_URL="https://api.free-stg.openshift.com" -p 8080:8080 MISSING_IMAGE.


⚠ Guya.moe Proxied has degraded performance

In 11be85c, Guya.moe Proxied (https://ice.guya.moe/) experienced degraded performance:

HTTP code: 200
Response time: 11890 ms

Resolved: Guya.moe Proxied performance has improved in 7ef487c after 1 hour, 22 minutes.


Iterating over xfunction with arguments with different layouts

Doesn't work correctly when using begin and end iterators (works fine with xbegin and xend, though).
Maybe a compile time switch checking the static layout is necessary?

        xarray<double, layout_type::column_major> cm = {{1,2,3}, {4,5,6}, {7,8,9}};
        xarray<double, layout_type::row_major> rm = {{1,2,3}, {4,5,6}, {7,8,9}};
        auto e = equal(cm, rm);
        auto iter_end = e.xend();
        for (auto iter = e.xbegin(); iter != e.xend(); ++iter)
        {
            EXPECT_TRUE(*iter); // WORKS!
        }
        for (auto iter = e.begin(); iter != e.end(); ++iter)
        {
            EXPECT_TRUE(*iter); // DOESN'T WORK
        }

Maybe we should do modify
https://github.com/QuantStack/xtensor/blob/master/include/xtensor/xfunction.hpp#L490
so that it only uses the trivial iterators when the layout is the same.

I think we shouldn't modify xfunction. begin() \ end() functions are meant to return iterators on the underlying container, without additional allocations. It would be weird to have a function that returns different types that have significative differences in performance, depending on the layouts of the leaves of the expression.
Besides, these iterators would probably be used for SIMD assign, so we need to be sure we really have iterators on storage containers when we call these methods.
And last (but not least), even if one of the expression involved in the function has a dynamic layout, it could be compatible with the other layouts. Or not. You have to check the strides (at runtime) to decide.
So I think it would be more intuitive and cleaner to add a precondition on these methods (we can even add macro to check that precondition in debug mode, just as we do for index bounds checking).

Ok this behavior is actually a bug, iterating over expressions should always work. the fact we use the underlying container iterator when the expression has one should be an optimization only. I'm working on a fix.

Fixed in #370


Help with TLS - please!

Hi all,
I've written a c# vs 2022 desktop app which reads a temperature sensor via Modbus.
I've included an mqtt interface using mqttnet such that the app can publish the temperature readings to a mqtt broker via raw tcp.
All working, but I want to employ tls to make the connection secure.
Problem is I have absolutely no experience with tls & I'm finding it very confusing.
Other users will use this software therefore I need to offer those users the same security options (e.g. tls) - but how do I go about doing this?
Do I need to include the tls or can I leave it to the users to implement themselves?
Your help would be very much appreciated because I'm banging my head against the wall with this.
It seems extremely complicated & there appears to be so many options.
Thank you!
Regards,
Darren

In my opinion this the wrong place for general TLS questions. We also just rely on .net sockets where only a couple of TLS options is available and must be properly set. Server is little bit more complicated but this is also a general topic and not related to this library. Unfortunately I have no other source I can paste here with instructions. Maybe someone else is able to do so.

After you have build your MQTT client options for the TCP connection with the MqttClientOptionsBuilder, try something along the following lines:
if (mqttClientShouldUseTls)
{
    var p = new MQTTnet.Client.MqttClientOptionsBuilderTlsParameters();
    p.UseTls = true;
    if (mqttClientShouldUseClientCertificate)
    {
        p.Certificates = new List<System.Security.Cryptography.X509Certificates.X509Certificate2>() { ... your certificates ... };
    }
    if (mqttClientShouldAcceptAnyCertificate)
    {
        p.AllowUntrustedCertificates = true;
        p.CertificateValidationHandler = (certContext) => { return true; };
    }
    options = options.WithTls(p);
}

This is based on the current master branch (I don't know if there is a difference in v 3.x).
If the broker expects the client to present a client certificate then you would need to create or issue your own certificates and it may get a little bit more complicated.

@logicaloud
Thank you for your reply.
Unfortunately I'm a complete newbie when it comes to TLS and the implementation thereof, so I'm already lost with your code.
I understand C# well enough, just not the TLS stuff.
What's the list of certificates? And what's an "untrusted" certificate?
The cloud-based brokers we'll typically recommend are CloudMqtt and HiveMQ.
Both of which support and recommend an SSL connection.
As I say; I want to distribute the Windows app. and offer TLS(SSL?) security to the users.
So I'm looking for a package solution to offer them, or at least offer advice in this respect.
I'm still none the wiser. Sorry. :-(
Darren

You don't need to worry about the TLS details because that is taken care of by .NET and the options builder in MQTTnet. Also, for a start, disregard the client certificates. After you have built your client options, in the simplest form, try this:
    var p = new MQTTnet.Client.MqttClientOptionsBuilderTlsParameters();
    p.UseTls = true;
    p.AllowUntrustedCertificates = true;
    p.CertificateValidationHandler = (certContext) => { return true; };
    options = options.WithTls(p);

before you build the options for the client connection (that you already do for the TCP connection):
mqttClientOptions = options.Build();

This should get you connected via TLS but its not good enough for production because all certificates are trusted.  Can you connect via TLS after adding the above to your code? If not, then an example of how you build your MQTT client would help.
After that, you can remove the AllowUntrustedCertificates and CertificateValidationHandler lines and leave it to .NET and the operating system to determine whether a broker certificate should be trusted.
At some point you'll probably want to read more about TLS, certificates, trust, and so forth, so that you can make informed decisions on how to handle this in production, but this is probably not the forum for it.

@logicaloud
Hi,
Thanks again.
I added your code to my connection function and tried connecting to my (free) CloudMqtt account via the dedicated TLS port, but it does not connect.
Perhaps this is because the port I'm trying to use is closed or not allowed by our IT system?
I think I need to take a step back and understand what I'm actually trying to achieve.
I read of "certificates", but I'm not sure what a "certificate" is in this context.
My connection code below:
`        private async void Connect()
{
mqFactory = new MqttFactory();
mqClient = (MqttClient)mqFactory.CreateMqttClient();
MqttClientOptions options;
        var p = new MQTTnet.Client.Options.MqttClientOptionsBuilderTlsParameters();
        p.UseTls = true;
        p.AllowUntrustedCertificates = true;
        p.CertificateValidationHandler = (certContext) => { return true; };
        //options = options.WithTls(p);

        if (txtUsername.Text != null && txtPassword.Text != null)
        {
            // Create TCP based options using the builder.
            options = (MqttClientOptions)new MqttClientOptionsBuilder()
                .WithClientId(txtClientID.Text)
                .WithTcpServer(txtBrokerAddr.Text, Convert.ToInt32(txtPort.Text))
                .WithCredentials(txtUsername.Text, txtPassword.Text)
                .WithCleanSession()
                .WithTls(p)
                .Build();
        }
        else
        {
            // Create TCP based options using the builder.
            options = (MqttClientOptions)new MqttClientOptionsBuilder()
                .WithClientId(txtClientID.Text)
                .WithTcpServer(txtBrokerAddr.Text, Convert.ToInt32(txtPort.Text))
                .WithCleanSession()
                .WithTls(p)
                .Build();
        }

        mqClient.ConnectedHandler = new MqttClientConnectedHandlerDelegate(MqttClientOpened);
        mqClient.DisconnectedHandler = new MqttClientDisconnectedHandlerDelegate(MqttClientClosed);
        mqClient.ApplicationMessageReceivedHandler = new MqttApplicationMessageReceivedHandlerDelegate(MqttClientMessageReceived);

        try
        {
            //mqttRes = mqClient.ConnectAsync(options);
            //await mqttRes;
            await mqClient.ConnectAsync(options);
        }
        catch (Exception Ex)
        {

        }
        //return mqClient.IsConnected;
    }`

@logicaloud
Hi there,
I can connect to the HiveMq cloud free account (tls on port 8883) from home but not from work, therefore I assume it's a port issue at work.
So I'm a small step forward in that at least I can connect to the tls brokers.
However, there's still no certificates involved.
I'm reading that there's a broker certificate & client.
I can obtain the HiveMq cloud broker certificate as a pem file, but I have no idea what this is or how to use it for security?
Can I just use this or should I have a certificate both ends?
Where do I obtain a suitable client cert from?
And considering I'm distributing my software, will I need to purchase a new client cert for each new user?
Thanks
Darren

When you use TLS then at least the broker already has a certificate.
If you also need to use client certificates then it is probably best to follow one of the guides out there, i.e.:  https://www.hivemq.com/blog/mqtt-security-fundamentals-x509-client-certificate-authentication/

I can obtain the HiveMq cloud broker certificate as a pem file, but I have no idea what this is or how to use it for security?
Can I just use this or should I have a certificate both ends?

If you just use the MQTT client with the WithTls() builder method, the operating system / .Net will already handle the certificate validation for you. You don't need to do something special here except if you had a so called self-signed certificate (Normally only used in development). Public servers like the one from HiveMQ have a validated certificate that the framework handles.

Where do I obtain a suitable client cert from?
And considering I'm distributing my software, will I need to purchase a new client cert for each new user?

I'm not sure if you even need some. You can just use username and password authentication over TLS. Client certificates only give you the advantage that you don't need to give out credentials, but just store a certificate in the device (or client) that connects to the MQTT server.
As @chkr1011 already stated, this is not a question of the library itself but a more generic question. @logicaloud has pointed to a source (https://www.hivemq.com/blog/mqtt-security-fundamentals-x509-client-certificate-authentication/), and there is the Wiki as well if you want to add client certificates: https://github.com/dotnet/MQTTnet/wiki/Client#certificate-based-authentication.

@SeppPenner
Hi,
Thanks for your reply.

If you just use the MQTT client with the WithTls() builder method, the operating system / .Net will already handle the certificate validation for you. You don't need to do something special here except if you had a so called self-signed certificate (Normally only used in development). Public servers like the one from HiveMQ have a validated certificate that the framework handles.

My software has a dedicated MQTT connection window, on this Window there's a broker address, port, clientID, username, password, useTLS and MQTT version connection options to enter/choose.
With this, I can successfully connect to the HiveMQ TLS broker.
Are you saying that this connection is secure enough as it is? and is it encrypted?
If so then I'm confused because I thought I needed to purchase some kind of client-side certificate, too - and this has been driving me insane trying to understand it.
I also thought I'd have to purchase a new client-side certificate for every customer who uses this software!
If this HiveMQ connection is inherently secure, then it seems I don't need to do anything more other than recommend to the customer to use the HiveMQ broker as I have in my testing.
Thanks again, and apologies for the off-topic questions. I realise this post is more suited elsewhere, but I can't seem to find the answers I'm looking for anywhere.

Are you saying that this connection is secure enough as it is? and is it encrypted?

Yes, it is encrypted.

If so then I'm confused because I thought I needed to purchase some kind of client-side certificate, too - and this has been driving me insane trying to understand it.

You can use client certificates as authentication method, but you can use username and password as well (There is nothing more secure with the client certificates).

If so then I'm confused because I thought I needed to purchase some kind of client-side certificate, too - and this has been driving me insane trying to understand it.

Well, for testing purposes, it's ok. Still, the HiveMQ broker is public and the data can be read by anyone (Just the transport is encrypted). So, I wouldn't use this for productive /sensitive data.


Responsive / full-width buttons

Pushed.

Nice! How can we use this?

@yabdab, use the word 'responsive' in the size parameter instead of small, medium...

Noice!


compilation terminated due to -Wfatal-errors.

How to solve? Ubuntu 18.04 building GEE with "python2.7 /usr/bin/scons -j8 release=1 build"
if (default_instance_ == NULL) protobuf_AddDesc_diorama_5fstreaming_2eprotodevel();
....
NATIVE-REL-x86_64/protobuf/diorama_streaming.pb.cc: In static member function 'static const keyhole::DioramaMetadata_Object& keyhole::DioramaMetadata_Object::default_instance()':
NATIVE-REL-x86_64/protobuf/diorama_streaming.pb.cc:383:3: error: this 'if' clause does not guard... [-Werror=misleading-indentation]
   if (default_instance_ == NULL) protobuf_AddDesc_diorama_5fstreaming_2eprotodevel();  return *default_instance_;
   ^~
compilation terminated due to -Wfatal-errors.
cc1plus: all warnings being treated as errors

Overcame this by wrapping
if (default_instance_ == NULL) whatever
to
if (default_instance_ == NULL) {whatever}
but got next
common/geFilePool.cpp:572:77: error: throw will always call terminate() [-Werror=terminate]
     throw khSimpleException("geFilePool::~Writer: write buffer not flushed.");

next looks really scary, 10 times of
In file included from common/khxml/khxml.cpp:19:0:
common/khxml/khdom.h: In function 'void FromAttribute(xercesc_3_2::DOMAttr*, QString&)':
common/khxml/khdom.h:712:41: error: invalid conversion from 'const XMLCh* {aka const char16_t*}' to 'const short unsigned int*' [-fpermissive]
   val = QString::fromUcs2(attr->getValue());
                           ~~~~~~~~~~~~~~^~
In file included from common/khxml/khxml.h:33:0,
                 from common/khxml/khxml.cpp:18:
NATIVE-REL-x86_64/include/qstring.h:673:20: note:   initializing argument 1 of 'static QString QString::fromUcs2(const short unsigned int*)'
     static QString fromUcs2( const unsigned short *ucs2 );
                    ^~~~~~~~

Currently, we do not actively support Ubuntu 18, so these errors are more or less new to us as well. For the invalid conversion, we have run into this with exploratory builds on other platforms and found that casting the parameter of fromUcs2 as (const ushort*) should get you by that error. However, its likely that you will run into further errors (especially if you are using a newer version of GCC). If you would like to use OpenGEE without hassle I'd recommend using a supported platform (Ubuntu 16, CentOS/Redhat 6/7) if possible.

@tst-cjeffries thanks for your attention and clear answer. May be I will end up with raising RedHat 7 in VM (seems if having RPM no need to build anything?). But for now I wish to try [-fpermissive] + without [-Werror] - may be this helps ... if I find the place for this directives, project structure not clear to me

Most of the compilation flags can be found in earthenterprise/earth_enterprise/src/SConstruct just you should be able to remove Werror and add fpermissive there. Good luck!

++ remove lines here
src/third_party/gif/SConscript:cc_flags.remove('-Werror')
src/third_party/glu/SConscript:cc_flags.remove('-Werror')
src/fusion/portableglobe/servers/fileunpacker/shared/SConscript:cc_flags.remove('-Werror')

well, it went more far, but failed. Seems I should use RH7 on VM
//usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4: undefined reference to<EMAIL_ADDRESS>//usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4: undefined reference to<EMAIL_ADDRESS>//usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4: undefined reference to<EMAIL_ADDRESS>//usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4: undefined reference to<EMAIL_ADDRESS>//usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4: undefined reference to<EMAIL_ADDRESS>//usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4: undefined reference to<EMAIL_ADDRESS>//usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4: undefined reference to<EMAIL_ADDRESS>//usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4: undefined reference to<EMAIL_ADDRESS>//usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4: undefined reference to<EMAIL_ADDRESS>//usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4: undefined reference to<EMAIL_ADDRESS>//usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4: undefined reference to<EMAIL_ADDRESS>//usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4: undefined reference to<EMAIL_ADDRESS>//usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4: undefined reference to<EMAIL_ADDRESS>//usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4: undefined reference to<EMAIL_ADDRESS>//usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4: undefined reference to<EMAIL_ADDRESS>//usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4: undefined reference to<EMAIL_ADDRESS>//usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4: undefined reference to<EMAIL_ADDRESS>//usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4: undefined reference to<EMAIL_ADDRESS>//usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4: undefined reference to<EMAIL_ADDRESS>//usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4: undefined reference to<EMAIL_ADDRESS>//usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4: undefined reference to<EMAIL_ADDRESS>//usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4: undefined reference to<EMAIL_ADDRESS>//usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4: undefined reference to<EMAIL_ADDRESS>collect2: error: ld returned 1 exit status
scons: *** [NATIVE-REL-x86_64/bin/tests/idl_tests] Error 1
scons: building terminated because of errors.
scons: *** [build] Error 2
scons: building terminated because of errors.

@tst-cjeffries I finished up with raising separate server server with Ubuntu16 and all was fine except one small issue ( https://github.com/google/earthenterprise/issues/1739 )
Next question is a bit offtopic, but may be you can clarify - I was not able to google that https://groups.google.com/forum/#!topic/google-earth-enterprise/cmON-5W4Weo

@nop1984 looks like we haven't responded to you in a while. Did you get your problems fixed or are you still having issues?


Secret scanning: Enable at the Enterprise level

Summary
GitHub Advanced Security customers can enable secret scanning and push protection at the Enterprise level
Intended Outcome
Allow users to easily enable secret scanning across their entire enterprise
How will it work?
Like at the organization level, users can choose to enable secret scanning and push protection through the UI in their enterprise's settings page.

This has shipped! 🎉 A changelog post for this release has just been published. 🚢 Leaving open to track for the GHES release.

🚢 This has shipped with the GHES 3.7 release<EMAIL_ADDRESS>


Fading Suns 2nd Edition Revised - sheet release 1.2.0

Changes / Comments
1.2.0

Renamed sheet to Fading Suns 2nd Edition Revised from Fading Suns v2.75
Added ability to select the characteristic for the roll of a skill on the fly (as opposed to having to select a characteristic beforehand)
Fixed error on Critical Success rerolls or Critical Failure confirms of custom Lore and Creative skills (bug introduced in 1.1.0)
Fixed typo in Initiative stance roll query. Aggressive stances give -2 Defense, not +2  (bug introduced in 1.1.0)
Cosmetic fixes

Roll20 Requests
Comments are very helpful for reviewing the code changes. Please answer the relevant questions below in your comment.

[x] Does the pull request title have the sheet name(s)? Include each sheet name.
[x] Is this a bug fix?
[ ] Does this add functional enhancements (new features or extending existing features) ?
[x] Does this add or change functional aesthetics (such as layout or color scheme) ?
[ ] If changing or removing attributes, what steps have you taken, if any, to preserve player data ?
[ ] If this is a new sheet, did you follow Building Character Sheets standards ?

If you do not know English. Please leave a comment in your native language.

Hello @oliviakristiansen
I realize now that the name Fading Suns v2.75 is not ideal for this sheet. Fading Suns 2nd Edition Revised would probably be much better and aligned with the naming pattern used in other Roll20 sheets.
Are you able to change the sheet name visible to Roll20 users to Fading Suns 2nd Edition Revised thanks to approved.yaml?
And if yes, should I move the folder itself to match this new name?
Also: is this a transparent change for existing users of the sheet, or will this break their games and require they update their game settings to select the new name?
Thanks,
Best Regards
admiralnlson

Yes! If you submit a PR with the updated folder name, I will be sure to change the name in the approved.yaml file. And it should not break user's games. If you submit the PR by Monday morning with the needed changes, it will be pushed on Tuesday. Thanks!

You could probably just update this open PR

Thank you for your response.
I did just that and updated the PR.

Hello @admiralnlson it seems like with the name change some folders were missed in the process (looking at the translations as an example). It also seems like you have committed some VSCode file snippet with the branch (might not want that in the branch). Please update and we can merge this in next week.

hello @Reonin
The vscode snippets file is meant to be part of the pull request. It is a enabler for whoever might end up maintaining the sheet, should I somehow go missing. Including such source files is a best practice agreed among us sheet authors.
Regarding missed translation files, I'm not sure what you mean. The guideline is not to touch such files as I did not generate them initially.
I can only provide translations for "en" and "fr". Others were auto-generated, and nobody from the CrowdIn service have contributed translations for those so far. So I would assume they will be re-generated from the translations.json again on your end.
Pretty sure everything is in order here, really. It would be a shame to have this PR pushed by one week.
BR
admiralnlson

Hello @Reonin ,
I removed the controversial vscode snippets code file as per your request.
Also, despite some fellow sheet authors advising me not to, I affected other (auto-generated) translations/xx.json files as per you request also.
The only one I am definitely unable to change is the zu one which is extremely cryptic.
BR
Yohan

Thanks for the quick turnaround @admiralnlson. I should have your changes in for next week's push.

Great! Thanks @Reonin


[[FIX]] Catch blocks are no longer functions

Catch blocks used to be functions in order to handle the exception
parameter, but this is now handled by the scope manager.
Fixes #2510

Merged to master at commit 8a864f3dddefd7c93e1202dc09c9d6aada127d0f . Thanks, Luke!


Add WinUI support

This PR supersedes https://github.com/mono/SkiaSharp/pull/1960.
Description of Change
Added views for Uno.WinUI https://github.com/mono/SkiaSharp/issues/1787
Bugs Fixed
API Changes
None
Behavioral Changes
None
PR Checklist

[ ] Has tests (if omitted, state reason in description)
[ ] Rebased on top of main at time of PR
[ ] Changes adhere to coding standard
[ ] Updated documentation

Merged in 7b3e8f3375d8114c473f87803627ff6a17519130

Had to do a local merge and not get the nice purple because I don't have push permissions. But, the credits are in the commit and this PR lives on!
Thanks so much for the work on this, really glad to get even more SkiaSharp view support!

When will that Nuget be available? Is this bound to a release of something else?

It should be coming out soon. I am just trying to get CI to build it as we type. Something just gave up and dies so now it runs out of disk space on CI... So it says. Hopefully this weekend or next week.

That's nice! Thanks for the update! :)

Any news on this one @mattleibow?

The package is there! Thank you! :)

Yeah, sorry about not updating this issue. I pushed the nuget and then started on some international flights.

All fine! I just polled nuget.org until I found the package and have been the first one to download it. 🤣
Thanks for your work and your update on this!


Provided model function fails when applied to the provided data set.BUG:

Issue Description
I'm building a predictive function using Bayesian network bnlearn. I try to build a custom predict function similar to sklearn. The custom predict function does not work.
As this is a non-standard predict function, any advise would be grateful. Thank you!
Minimal Reproducible Example
import numpy as np
import pandas as pd
import bnlearn as bn
from sklearn.model_selection import train_test_split

df = bn.import_example()
X=df[["Cloudy","Sprinkler","Rain"]]
y=df["Wet_Grass"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=0)
model = bn.structure_learning.fit(df,root_node='Wet_Grass')

DAG = bn.make_DAG(model)
DAG = bn.parameter_learning.fit(DAG, X_train.join(y_train))
Pout = bn.predict(DAG, X_test.join(y_test), variables=['Wet_Grass'])

import shap
shap.initjs()
def bn_predict(X_test):
  Pout = bn.predict(DAG, X_test.join(y_test), variables=['Wet_Grass'])
  return Pout['p'].to_numpy()
explainer = shap.KernelExplainer(bn_predict,X_test)

Traceback
Provided model function fails when applied to the provided data set.
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
[<ipython-input-17-bcc432829ff4>](https://localhost:8080/#) in <cell line: 1>()
----> 1 explainer = shap.KernelExplainer(bn_predict,X_test)

2 frames
[<ipython-input-16-ba8e4293ea0f>](https://localhost:8080/#) in bn_predict(X_test)
     17 shap.initjs()
     18 def bn_predict(X_test):
---> 19   Pout = bn.predict(DAG, X_test.join(y_test), variables=['Wet_Grass'])
     20   return Pout['p'].to_numpy()

AttributeError: 'numpy.ndarray' object has no attribute 'join'

Expected Behavior
No response
Bug report checklist

[X] I have checked that this issue has not already been reported.
[X] I have confirmed this bug exists on the latest release of shap.
[ ] I have confirmed this bug exists on the master branch of shap.
[ ] I'd be interested in making a PR to fix this bug

Installed Versions
The latest

Isn't this just a bug in your code? Your error happens before you use any of shap's code. I haven't run the code but I think after the train_test_split you are dealing with numpy arrays and not dataframes, so .join is not available.
Will close this, since this is not a shap bug.

Apologies but I need to re-open it again.
Actually the code does run until the shap bit, if you test it.
The predict function does not go as the standard model.predict structure, rather it works a different structure: X_test.join(y_test) and then the variable of interest to predict. Hence I cannot figure out how shap package support this type of predict function.
Writing a self-defined wrapper function in this case causes errors.
Any advise would be grateful, thank you!


Goto definition does not work with v5.4.0

Goto definition does not work with v5.4.0. The code is not colored and I am not able to navigate inside the code to a method by pressing CMD+D

By going back to v5.3.5 all works normal.

Hi! Information about external code passed through debugger in the new version (see SourceRequest). This only applies to external code that can be downloaded via SourceLink (for example: MAUI assemblies).
In 5.3.5 source from external code was created in the .meteor folder of your project. Maybe this folder was included in your project before? I think this is not a good thing, as it could affect the IntelliSense in your project.
How critical is this problem for you? I can revert this changes.

Thanx for the fast response. What do you mean by "included in your project before" ?
The folder exists in the project folder in the finder as a hidden folder BUT it is not part of the project ?
Should I remove the folder ?

Yes, the debugger automatically creates hidden .meteor folder in 5.3.5. When you select any frame in the CALLSTACK VSCode tab, if this frame has SourceLink (url to the source code) - debugger automatically downloads this file in the .meteor/sources folder and shows it.
In 5.4.0 I have removed this folder. Now source code from SourceLink passed through debugger.
It is very easy to check, in the old version you could edit this file, but in the new one it is ReadOnly:

I am NOT in debug mode ! I have the project open NOT running the app and just want to navigate thru the code.
So should I remove the .meteor folder if I am using v5.4.0 ?

Yes, you can remove it.

I am NOT in debug mode !

This should not affect the IntelliSense of your project. .NET Meteor is just a debugger and it activates only in debug mode.
What kind of extension do you use for IntelliSense? C# DevKit or DotRush?

It is working with v5.4.0, I am able to navigate (not in debug mode) to my own methods or to external code with CMD+D again...thanx
Currently I am using C# Dev Kit.

Yeah, I think this is a bug on the C# DevKit side. We at DevExpress don't use C# DevKit and MAUI extensions.
If you also have a lot of problems with this extension, I recommend that you use DotRush instead. It doesn't have Solution Explorer and razor support, but IntelliSense works much better.

yes, I probably will try it...thanx


[🐛 Bug]: Not all workers-compatible nodejs api is fully supported

Description
Not all nodejs API supported by the workers runtime (docs) is correctly supported, of the compatibility list:

assert
AsyncLocalStorage
Buffer
EventEmitter
util

I think that we only support AsyncLocalStorage and Buffer but not the rest.
Support should be added since we already require the nodejs_compat flag to be specified so we should actually make sure that we support all the APIs provided by the flag.

Reproduction
For example see the api/hello route of this app: simple-pages-dir-13.4.2.
It generated an internal server error because of the use of node:events:

(Note that EventEmitters are workers-compatible as indicated above)

Related to: https://github.com/cloudflare/next-on-pages/issues/244

@felix-schultz I'm pinging you in case you we'd be interested in following this 🙂, it does look like we can do things better in regards of node compatibility, I'll look into this soon 🙂👍

Note (to self)
currently we have this ugly hack that is needed because of a workers runtime bug: https://github.com/cloudflare/next-on-pages/blob/f72a56ffef1ac1d5b62c9e20f2b227a5d2fa8910/src/buildApplication/generateFunctionsMap.ts#L47-L53
this would make the solution for nodejs modules require such hacks for all modules potentially making a bit of a mess, since I don't believe that there is a high demand for this fix (and the runtime bug should be fixed fairly soon) let's wait for the runtime to be fixed first (and the above hack removed) before attempting to support all node.js modules.


Recursive reset for master page setting missing from SetMasterPageByUrl

BrandingExtensions.cs.
This method works for resetting the sub webs only one level deep, not cross all the sub sites what the site collection might have. Code is missing recursive reset for all sub sites in one site collection even though parameters are suggesting that.

Fixed in the code accordingly.


NordPass causes TomSelect to not update

Describe the bug
I have a form that has a TomSelect element on it. I update that element dynamically with a javascript fetch and tomSelect.addOptions(items). This is done using Rails 7 and Stimulus. It works fine until Nordpass decides that the form warrants some attention. (Of note is that this happens if I add 3 radio buttons into the form. 2 are fine, but the 3rd radio causes NordPass to decide that some fields should have autofill).
There are no JS errors on the page. And the options are being added to the TS div, it is just that the dropdown behavior is not working. Also of note is that the 'loading' spinner also does not appear.
To Reproduce
I am trying to figure out how to reproduce this because I can't get Nordpass to fire on something like JSFiddle.

Nevermind! The issue was that I was setting the field to be of type='select'. That seems to cause issues. I changed it to a text_field and all is well.


Solve unconstrained optimization problems

This issue shows how to set osqp-eigen to solve an unconstrained optimization problem
We now want to solve the following optimization problem

The solution of the problem is x = [0 0]
osqp-eigen can be used to solve the problem as follows
Initialize the variables. Please notice that the constraintMatrix size is 0 x 2 (i.e. number of constraints x number of unknowns)
Eigen::SparseMatrix<double> hessian(2,2);
Eigen::SparseMatrix<double> constraintMatrix(0,2);
Eigen::Vector2d gradient;
Eigen::VectorXd bound(0);

gradient.setZero();
hessian.insert(0,0) = 1;
hessian.insert(1,1) = 1;

Now  you can initialize the problem as usual
OsqpEigen::Solver solver;
solver.data()->setNumberOfVariables(2);
solver.data()->setNumberOfConstraints(0);
if(!solver.data()->setHessianMatrix(hessian)) return 1;
if(!solver.data()->setGradient(gradient)) return 1;
if(!solver.data()->setLinearConstraintsMatrix(constraintMatrix)) return 1;
if(!solver.data()->setLowerBound(bound)) return 1;
if(!solver.data()->setUpperBound(bound)) return 1;

You can finally solve the problem
if(!solver.initSolver()) return 1;
if(!solver.solve()) return 1;

Here the entire example
#include <iostream>

#include <Eigen/Sparse>
#include <Eigen/Dense>

#include <OsqpEigen/OsqpEigen.h>

int main()
{
    Eigen::SparseMatrix<double> hessian(2,2);
    Eigen::SparseMatrix<double> constraintMatrix(0,2);
    Eigen::Vector2d gradient;
    Eigen::VectorXd bound(0);

    gradient.setZero();
    hessian.insert(0,0) = 1;
    hessian.insert(1,1) = 1;

    OsqpEigen::Solver solver;
    solver.data()->setNumberOfVariables(2);
    solver.data()->setNumberOfConstraints(0);
    if(!solver.data()->setHessianMatrix(hessian)) return 1;
    if(!solver.data()->setGradient(gradient)) return 1;
    if(!solver.data()->setLinearConstraintsMatrix(constraintMatrix)) return 1;
    if(!solver.data()->setLowerBound(bound)) return 1;
    if(!solver.data()->setUpperBound(bound)) return 1;

    if(!solver.initSolver()) return 1;
    if(!solver.solve()) return 1;

    std::cout << "Solution = " << solver.getSolution().transpose() << std::endl;

    return 0;
}

This is the CMakeLists.txt that you can use to run the example
project(osqp-eigen-test)

cmake_minimum_required(VERSION 3.0)

find_package(OsqpEigen REQUIRED)
find_package(osqp REQUIRED)
find_package(Eigen3 REQUIRED)

add_executable(osqp-eigen-test main.cpp)
target_link_libraries(osqp-eigen-test OsqpEigen::OsqpEigen osqp::osqp Eigen3::Eigen)

The fact that the user should set the constraint matrix and the bounds even if the problem is unconstrained is counterintuitive.
cc @S-Dafarra @traversaro @lrapetti

Just to understand, what is the output if you add no constraints at all?

Just to understand, what is the output if you add no constraints at all?

[OsqpEigen::Solver::initSolver] Some data are not set.

Ok this sounds like a logical problem then. That should be easy to solve then. https://github.com/robotology/osqp-eigen/blob/9b0091038cf0fb9099a5d6c55820d6e5846034a7/src/Data.cpp#L88-L97
should become
    return m_isNumberOfVariablesSet &&
        m_isNumberOfConstraintsSet &&
        m_isHessianMatrixSet &&
        m_isGradientSet &&
        (m_data->m ==0 || m_isLinearConstraintsMatrixSet ) &&
        m_isLowerBoundSet &&
        m_isUpperBoundSet;

right?

I don't know if it's so simple (We should investigate the side effects)

In any case, I think that it does not make sense to require the constraint matrix to be set if there are no constraints. If something else breaks, we shall fix it as well :grin:
Could you add the unconstrained problem as a test case?

I think that it does not make sense to require the constraint matrix to be set if there are no constraints

I agree

Related issue https://github.com/oxfordcontrol/osqp/issues/295

Related issue oxfordcontrol/osqp#295

Ah, nice! So if it is also a limitation of osqp, the solution you proposed in the OP seems the way to go then.


Turbo-V3

I converted the new openai model weights to be used with faster-whisper. Still playing around with it, but in terms of speed its about the same as distil whisper.
https://huggingface.co/freddierice/openwhisper-turbo-large-v3-ct2/blob/main/README.md

I converted the new openai model weights to be used with faster-whisper. Still playing around with it, but in terms of speed its about the same as distil whisper.
https://huggingface.co/freddierice/openwhisper-turbo-large-v3-ct2/blob/main/README.md

Could you convert Whisper Turbo with the multilingual tokenizer?

Thanks for the quick conversion! I'm getting a tokenizer error:
Traceback (most recent call last):
  File "transcribe.py", line 660, in __init__
    self.hf_tokenizer = tokenizers.Tokenizer.from_file(tokenizer_file)
Exception: data did not match any variant of untagged enum ModelWrapper at line 264861 column 3

Any support would be appreciated :)

I converted the new openai model weights to be used with faster-whisper. Still playing around with it, but in terms of speed its about the same as distil whisper.
https://huggingface.co/freddierice/openwhisper-turbo-large-v3-ct2/blob/main/README.md

Could you convert Whisper Turbo with the multilingual tokenizer?

It's done in: https://huggingface.co/deepdml/faster-whisper-large-v3-turbo-ct2

Tested https://huggingface.co/deepdml/faster-whisper-large-v3-turbo-ct2
Works very fast

Could you show how can we test it? On Google Colab notebook I got error as no model named as "faster-whisper-large-v3-turbo-ct2" when pip install faster-whisper

Could you show how can we test it? On Google Colab notebook I got error as no model named as "faster-whisper-large-v3-turbo-ct2" when pip install faster-whisper

You've to download the model in your local:
from huggingface_hub import snapshot_download
repo_id = "deepdml/faster-whisper-large-v3-turbo-ct2"
local_dir = "faster-whisper-large-v3-turbo-ct2"
snapshot_download(repo_id=repo_id,local_dir=local_dir,repo_type="model")

If you guys want to test the model as a Real Time Transcription tool - I have a simple demo with Gradio for this. Just updated to code to use "deepdml/faster-whisper-large-v3-turbo-ct2"
https://github.com/Nik-Kras/Live_ASR_Whisper_Gradio

Could you show how can we test it? On Google Colab notebook I got error as no model named as "faster-whisper-large-v3-turbo-ct2" when pip install faster-whisper

You've to download the model in your local:
from huggingface_hub import snapshot_download

repo_id = "deepdml/faster-whisper-large-v3-turbo-ct2"
local_dir = "faster-whisper-large-v3-turbo-ct2"
snapshot_download(repo_id=repo_id, local_dir=local_dir, repo_type="model")

Thanks!

Benchmark
For reference, here's the time and memory usage that are required to transcribe 13 minutes of audio using different implementations:

openai/whisper@6dea21fd
whisper.cpp@3b010f9
faster-whisper@cce6b53e

Large-v3 model on GPU

Implementation
Precision
Beam size
Time
Max. GPU memory
Max. CPU memory
WER

openai/whisper
fp16
5
4m30s
11325MB
9439MB

faster-whisper
fp16
5
54s
4755MB
3244MB

faster-whisper
fp16
5
54s
4755MB
3244MB

faster-whisper
int8
5
59s
3091MB
3117MB

faster-whisper
int8
5
59s
3091MB
3117MB

Executed with CUDA 11.7.1 on a NVIDIA Tesla V100S.
Small model on CPU

Implementation
Precision
Beam size
Time
Max. memory
WER

openai/whisper
fp32
5
10m31s
3101MB

whisper.cpp
fp32
5
17m42s
1581MB

whisper.cpp
fp16
5
12m39s
873MB

faster-whisper
fp32
5
2m44s
1675MB

faster-whisper
int8
5
2m04s
995MB

Executed with 8 threads on a Intel(R) Xeon(R) Gold 6226R.

any idea, how can I run it faster using apple silicon, as i have an M2 pro machine.

any idea, how can I run it faster using apple silicon, as i have an M2 pro machine.

Have you tried faster-whisper? It'seems that it's faster than any other framework.
https://medium.com/@GenerationAI/streaming-with-whisper-in-mlx-vs-faster-whisper-vs-insanely-fast-whisper-37cebcfc4d27
You could try: https://github.com/mustafaaljadery/lightning-whisper-mlx

lmao the cantonese model is not word to word in the large-v3-turbo one... so sad... :( still will use the large-v3 💖
好嘅 -> 好的
是否 -> 係咪

but come on
I don't know that language, could you give more details on your observation? What's wrong and how the result will differ with large-v3?

but come on

I don't know that language, could you give more details on your observation? What's wrong and how the result will differ with large-v3?

--language=yue
Used this model
https://huggingface.co/deepdml/faster-whisper-large-v3-turbo-ct2
--beam_size 10 --best_of 10
on https://www.youtube.com/watch?v=sgRfqRFJlAg
so cantonese has two variants, one is the written cantonese, where all subtitles are mostly based on it, one is the spoken cantonese, which is literally the spoken characters written on it

藏哥係咪未讀過書?
床哥是否未讀過書?

是 => written cantonese variant (read: si)
係 => spoken cantonese variant (read: hai)
In general, if you want to learn spoken cantonese, you'll stick to the spoken version...
The difference is about 10-20%

We now support the new whisper-large-v3-turbo on Sieve!
Use it via sieve/speech_transcriber: https://www.sievedata.com/functions/sieve/speech_transcriber
Use sieve/whisper directly: https://www.sievedata.com/functions/sieve/whisper
Just set speed_boost to True. API guide is under "Usage Guide" tab.

@trungkienbkhn Will SYSTRAN be adding this in the HF repo to be used out of the box?

@trungkienbkhn Will SYSTRAN be adding this in the HF repo to be used out of the box?

I think that would be necessary for lots of downstream projects like faster-whisper-server

but come on

I don't know that language, could you give more details on your observation? What's wrong and how the result will differ with large-v3?

--language=yue Used this model: https://huggingface.co/deepdml/faster-whisper-large-v3-turbo-ct2 vs the normal large-v3 model
Faster-Whisper-XXL.exe C:/Users/hocky/Downloads/Video/Video.mp4 --model {MODEL} --device CUDA --output_dir C:/Users/hocky/Downloads/Video --output_format srt --task transcribe --beam_size 10 --best_of 5 --verbose true --vad_filter true --vad_alt_method silero_v4 --standard_asia --language yue

on https://www.youtube.com/watch?v=sgRfqRFJlAg
so cantonese has two variants, one is the written cantonese, where all subtitles are mostly based on it, one is the spoken cantonese, which is literally the spoken characters written on it

藏哥係咪未讀過書?
床哥是否未讀過書?

是 => written cantonese variant (read: si) 係 => spoken cantonese variant (read: hai)
In general, if you want to learn spoken cantonese, you'll stick to the spoken version... The difference is about 10-20%
bruh, just realized it doesn't even transcribe the most basic terms properly: the famous "DLLM"

You may find this discussion helpful:
https://github.com/openai/whisper/discussions/2363#discussion-7264254
"Across languages, the turbo model performs similarly to large-v2, though it shows larger degradation on some languages like Thai and Cantonese."

If you guys want to test the model as a Real Time Transcription tool - I have a simple demo with Gradio for this. Just updated to code to use "deepdml/faster-whisper-large-v3-turbo-ct2"
https://github.com/Nik-Kras/Live_ASR_Whisper_Gradio

What's your experience with using v3-turbo for short audio clips? Mine has been that unfortunately it performs worse than other models, see https://github.com/SYSTRAN/faster-whisper/issues/1030#issuecomment-2393164779

What's your experience with using v3-turbo for short audio clips? Mine has been that unfortunately it performs worse than other models, see #1030 (comment)

I only planned to evaluate turbo, turbo-CTranslate2 and turbo-HQQ. I can only tell that it works x2-x3 times faster based on my logs with the Gradio demo.
How bad are the evaluation results?

What's your experience with using v3-turbo for short audio clips? Mine has been that unfortunately it performs worse than other models, see #1030 (comment)

I only planned to evaluate turbo, turbo-CTranslate2 and turbo-HQQ. I can only tell that it works x2-x3 times faster based on my logs with the Gradio demo.

How do you use HQQ in faster-whisper? Could you share a sample code?
I only see how to use it with Transformers library: https://github.com/mobiusml/hqq?tab=readme-ov-file#transformers-

What's your experience with using v3-turbo for short audio clips? Mine has been that unfortunately it performs worse than other models, see #1030 (comment)

I only planned to evaluate turbo, turbo-CTranslate2 and turbo-HQQ. I can only tell that it works x2-x3 times faster based on my logs with the Gradio demo.

How do you use HQQ in faster-whisper? Could you share a sample code? I only see how to use it with Transformers library: https://github.com/mobiusml/hqq?tab=readme-ov-file#transformers-

Right, HQQ works with Transformers. But faster-whisper is just whisper accelerated with CTranslate2 and there are models of turbo accelerated with CT2 available on HuggingFace: deepdml/faster-whisper-large-v3-turbo-ct2
Also, HQQ is integrated in Transformers, so quantization should be as easy as passing an argument
model_id = "deepdml/faster-whisper-large-v3-turbo-ct2"
quant_config = HqqConfig(nbits=4, group_size=64)

model = AutoModelForSpeechSeq2Seq.from_pretrained(
    model_id,
    torch_dtype=torch_dtype,
    low_cpu_mem_usage=True,
    use_safetensors=True,
    quantization_config=quant_config
)

https://huggingface.co/docs/transformers/main/quantization/hqq
I didn't try it yet, so don't know if that is going to work.
How about to have a chat about it outside of the GitHub issue? Send me a message on LinkedIn, I have it attached in the profile

No, There is no HQQ support for ctranslate2 yet.
Faster-whisper has whisper models in Ctranslate2 format, which is different from pytorch models in HF. Of course the models are available in HF so that it is easy to use in packages such as faster-whisper. But one can not directly load a ctranslate2 checkpoint with  AutoModelForSpeechSeq2Seq.
I have created a feature request in the past to support HQQ (with static cache and torch compilation): https://github.com/OpenNMT/CTranslate2/issues/1717
The PR is still in progress and it has some performance issues that needs to be fixed.

What's your experience with using v3-turbo for short audio clips? Mine has been that unfortunately it performs worse than other models, see #1030 (comment)

I only planned to evaluate turbo, turbo-CTranslate2 and turbo-HQQ. I can only tell that it works x2-x3 times faster based on my logs with the Gradio demo.
How bad are the evaluation results?

No standardized evaluation or anything, I'm just running it in my streaming application and seeing way worse results than medium (especially with it randomly just not transcribing part of the text). This is a ctranslate2 implementation, deepdml/faster-whisper-large-v3-turbo-ct2 to be exact. See my linked comment for the code

were you able to convert turbo to faster-whisper format?

Mobiuslabs fork now supports turbo out of the box, and has additional fixes.

There seems to be a lot of confusion in this thread -- if you want to use turbo with the current faster whisper, all you have to do is
WhisperModel("deepdml/faster-whisper-large-v3-turbo-ct2", device="cuda", compute_type="float16")

Closing this thread since there is no issue.


bug in coord slice with step -1 and bounds

Hello,
this is more of a bug report than a pull request.  I've added a couple of failing tests that I think should pass.  Could you review and see if you agree?
This is likely propagate into other analysis - cube slice with -1 step (I think) would suffer from the same issue... which could give problems for area weighted regrid.
I don't feel confident of putting in a fix - particularly in quick time - I think there could be some issues I'd need to research, and I'm not sure when I'll have time.

can the 'cell/nbounds' dimension of a bounds array be anything other than the inner most (furthest left) dimension
I think the correct behaviour for 1d coords is clear - reverse the values along the 'cell/nbounds' dimension.  I'd have to think how this works for 2 (or more) dimensioned aux coords.

Jamie

These issues are the subject of the long-standing #423, see there for a short discussion.

http://cfconventions.org/Data/cf-conventions/cf-conventions-1.6/build/cf-conventions.html#cell-boundaries suggests that what I called the 'cell/bounds' dimension is the most rapidly varying and so is the most leftmost.  I think this makes a fix for this in the 1d coordinate case reasonably straight forward.
I still don't have time, at the moment, get my head round the CF discussion of 'Bounds for 2-D coordinate variables with 4-sided cells'.

:+1: for adding a test to highlight the problem. Thanks @jkettleb!

I think applying a convention could be useful.  Imagine concatenating two datasets together, one with bounds oriented in one direction and the other with the bounds in the other direction.
The two datasets right now simply cannot be concatenated together since the resulting
bounds array of the concatenated result would not be strictly monotonic (i.e. no longer described by a dimension coordinate).
One solution is to simply have the coordinates demoted to auxiliary coordinates (ignoring the fact for now that concatenation does not support concatenation over dimensions which have no dimension coordinate).  However, might the better approach be to assign a convention on the orientation of bounds?
Would there be harm in applying a chosen convention in iris?

Thanks for this PR. I'll close it in favour of the bug report in #423 as there will need to be some development work to arrive at a fix anyway.
👍 for raising