Limit service namespace scheme to non-redundant usage or ax it

The wiki guidelines around the use of namespaces clearly states that they shouldn't be used in cases where it just duplicates data.
As in an example, the Over-namespacing section of the namespace article says "over-namespacing leads to a disseminated data scheme: for example, someone interested in VHF channels data will have to look for harbour:VHF_channel key, plus seamark:habour:VHF_channel, plus VHF_channel, plus lock:VHF_channel, plus vhf to collect the data... Using only the vhf key should be enough to know that this data relates to the harbour or the lock or what else is the OSM object we are tagging." The other sections of the article could just as easily apply to the way people are using the service namespace tags in iD Editor along with it.
For instance out of the 6894 uses of service:bicycle:retail=yes, 6091 of them are tagged on objects already tagged as shop=bicycle. There is only 7198 total uses of service:bicycle:retail. So almost 85% of it's usage is redundant and just duplicates already existing tags. Which is completely ridiculous, goes against the guidelines, and is not how these types of "service" tags were ever intended to be used. A lot of the other service namespace tags have the same issue.  Over half of the uses of service:vehicle:car_repair are tagged on objects already tagged as shop=car_repair and there are other examples.
IMO, there are two possible solutions. Either block the ability for a service tag to be added to  an object where it already has a more appropriate tag, like with my example of service:bicycle:retail=yes being added to places already tagged as shop=bicycle , or don't have the preset in the first place. If most of the service namespace usage is redundant, and it appears to be, then the best option would be just to disable it. There's zero reason to provide a preset for a tagging scheme that isn't being used how it was intended to be and mainly just duplicates data.

For instance out of the 6894 uses of service:bicycle:retail=yes, 6091 of them are tagged on objects already tagged as shop=bicycle. There is only 7198 total uses of service:bicycle:retail. So almost 85% of it's usage is redundant and just duplicates already existing tags. Which is completely ridiculous, goes against the guidelines, and is not how these types of "service" tags were ever intended to be used. A lot of the other service namespace tags have the same issue. Over half of the uses of service:vehicle:car_repair are tagged on objects already tagged as shop=car_repair and there are other examples.

I don't agree. shop=bicycle is for shops where one can: buy a bike OR rent a bike OR repair a bike OR buy bike parts. Redundant would be to have separate main tags for it: shop=buy_bicycle, shop=bicycle_rental, shop=bicycle_repair, shop=bicycle_parts. So service:bicycle:retail=yes is not redundant because with shop=bicycle you can tag a place, where you can repair your bike but don't buy.
Shop=bicycle doesn't mean "they sell bikes there". It can be a place where you can only repair your bike or a place where you can only buy accessories.

Either block the ability for a service tag to be added

It was said many times that iD follows the OSM Wiki. The service tag is described there.
The tag service:bicycle:retail has a Wiki page and it's approved.

I don't agree. shop=bicycle is for shops where one can: buy a bike OR rent a bike OR repair a bike OR buy bike parts. Redundant would be to have separate main tags for it: shop=buy_bicycle, shop=bicycle_rental, shop=bicycle_repair, shop=bicycle_parts. So service:bicycle:retail=yes is not redundant because with shop=bicycle you can tag a place, where you can repair your bike but don't buy.

From my understanding the intent of these types of descriptive tags exist are for the exceptions, not the rules. For instance the whole reason there is a key like outdoor_seating is because some places have outdoor seating and some don't. Otherwise, outdoor seating would be "baked into the cake" and there wouldn't be a need describe it. In this example most people assume that a bicycle shop sells bicycles. Everyone sees the icon for it on the map and thinks the object sells bicycles. They don't scroll through their mind of things it might not do. That's just not how maps work.
For instance people don't see the rendering for or tag landuse=scrub and figure that it's scrub by thinking about all the possible non-scrub things it might be. So, the descriptive tags exist to clarify things through describing them in more detail. Therefore, clarification with service:bicycle:retail=yes would be needed if for instance the place is tagged as shop=bicycle and service:bicycle:repair to show that it isn't exclusively a bicycle repair shop. I'd have zero problem with that as it fits with the purpose of the tag and that's how it is used some of the time.
The problem with that though is that the Wiki is pretty explicit about how we should map multiple features of an object as separate nodes when they they serve different purposes. It's also things are done in most every other case outside of car/bicycle/motorcycle shops. Including in iD Editor. For instance with shop=chemist and amenity=pharmacy or shop=convenience and amenity=fuel. The problem with a lot of these service tags even when they aren't redundant is that using them comes at the cost of mapping extra information that we would be able to add by following the guideline, precedence, etc etc like different hours and contact details. So whatever the shop=bicycle article says, it shouldn't (and doesn't) come at the cost of the guidelines, the naming rule, or the rule that says we should map features of an object separately.
If the amenity=bicycle_rental tags definition should be expanded to include shops that exclusively rent bicycles or separate features of a shop that sells bicycles, then that's a tagging issue and it shouldn't be "solved" by ignoring guidelines and precedent. Guidelines supersede a tags definition also and Just because a tag is approved it doesn't automatically imply support anywhere. At least not as intended. Anyway, take service:bicycle:retail out of the equation and like I said the rest of the tagging scheme still has the same and other problems. service:car:whatever etc etc wasn't approved and there aren't automatically valid because service:bicycle:whatever was. So, the problem still needs to be dealt with even without it. Although, I think it should be still be limited or gotten rid of also. You can't ignore the guidelines on how to use namespaces just because of what the top of the shop=bicycle article says. The thing to do is to to make it so people can use the namespaces while following the guidelines.
So your argument is


It is not specified what requests count towards the average response time

Please specify if failed and unauthorized requests (which should be the fastest) count towards the reported average response times.
Then, a few bugs of this feature:

the time filter does not work correctly or at least intuitively.
In my case, I am reported 31 unauthorized requests today, but 0 in the last 7 days! This is because of how the date filter is populated.
sorting on column "Average service response time" does not work in the Operations view.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 2fa7b7b8-e2ce-eba9-8d03-b5ab4e079aea
Version Independent ID: dcca6523-c696-c316-5920-7313e3a462fb
Content: Use API analytics in Azure API Management
Content Source: articles/api-management/howto-use-analytics.md
Service: api-management
GitHub Login: @dlepow
Microsoft Alias: danlep

@ciufudean Thanks for the feedback! I have assigned the issue to the content author to investigate further and update the document as appropriate.

@VitaliyKurokhtin - Can you please comment if unauthorized requests are counted in API analytics? Thank you!

They should be

@ciufudean - Thanks for contacting us about this documentation. I will add a not clarifying that the analytics feature counts failed/unauthorized requests. If you encounter issues using the feature, I recommend that you contact Azure support to help troubleshoot further.


emacs: Add infrastructure for native-comp

Motivation for this change
This is to facilitate the emacsGcc support in https://github.com/nix-community/emacs-overlay.
It's not feasible to add all of this through overrides.
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[x] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

Thanks for getting this merged!
I just tried building the native-comp emacs on darwin (which I know works outside nix; most instructions refer to homebrew), and found two errors: One is fixed by 3a79681eb45df9c89d1de613114394f49582f0d6 (which fixes installation of libgccjit), and the other is that https://github.com/NixOS/nixpkgs/pull/93716/files#diff-c88608e19b22610f68cf3cce033c0678R69 depends on stdenv.glibc, which doesn't work on Darwin. I'm building a merged set of those two fixes (replacing glibc with libc), and hope to submit a follow-on PR soon: building the world just takes a few hours on this 4-core macbook pro.

@antifuchs Do you think you'll have a PR up soon to add Darwin support?  There's the .libc change that was pointed out on the commit; but so far, I'm failing to compile a libgccjit. I got the error message "symbols referenced by indirect symbol table entries that can't be stripped" building that library, which apparently means you have to run strip -x on macOS... but I couldn't find out where to hook that in.
Maybe we ought to report one or two other issues about Darwin builds there. /:

@evanrelf after thinking about this more, here's a branch that seems to build libgccjit and is well on the way towards building the world and then emacs (I'm about halfway through building LLVM right now, all four cores on this machine are sweating). Here's the branch: https://github.com/NixOS/nixpkgs/compare/staging...antifuchs:gccemacs-on-darwin - I hope this helps (and most of all, that it works! We're spending a lot of watts on this)

@antifuchs Could it be because stdenv.cc on Darwin is set to clang?
nix-repl> pkgs.stdenv.cc.cc.name
"clang-7.1.0"

You probably need to force it to use gcc explicitly, rather than stdenv.cc.

@evanrelf my understanding is that you have to compile (nextstep-type) Emacs with clang on darwin because gcc hasn't caught up to the language features yet (or you'd have to build it with X11 GUI - see). I think the build dependencies up to libgccjit work, except the libgccjit dynamic library is installed as libgccjit.so.0.0.1, not as libgccjit.dylib. Emacs's configure script builds the test program with -lgccjit, and the resulting binary (according to otool -L doesn't have the path to the dynamic library coded into it, and fails with "Image not found".
I guess I have to figure out how to install it as a .dylib... or something else.


feat: added nodeSelector based on k8s node labels for kube-prometheus

What it does?
it adds kepler and kube-promtehus helm deployment to specific nodes based on nodeselector aka node labels
Fixes

#30

also dont worry about if any secret specific to the cluster is present its a temporaty cluster

kepler got provisioned to the correct node

@dipankardas011 Could you also remove kepler from the PR title and description?
We can't merge this until we have an extra node with these labels. That's the next thing I'm going to work on.

so your manifest file worked?
if yes I can try it as well

so node exporter uses kube-state-metrics??

in my system the nodeexporter is not running and is pending
 Events:                                                                                │
│   Type     Reason            Age   From               Message                                                                                │
│   ----     ------            ----  ----               -------                                                                                │
│   Warning  FailedScheduling  42s   default-scheduler  0/2 nodes are available: 1 node(s) didn't have free ports for the requested pod ports. p │
│ reemption: 0/2 nodes are available: 2 No preemption victims found for incoming pod..                                                           │
│   Warning  FailedScheduling  37s   default-scheduler  0/2 nodes are available: 1 node(s) didn't have free ports for the requested pod ports. p │
│ reemption: 0/2 nodes are available: 2 No preemption victims found for incoming pod..

in my system the nodeexporter is not running and is pending

@dipankardas011 It uses port 9100 is that available in your cluster?
On the Equinix cluster it's running OK. The restarts are because of a Falco load test which is stressing the box 😅
k -n monitoring -l app.kubernetes.io/name=prometheus-node-exporter get po -o wide
NAME                                                              READY   STATUS    RESTARTS        AGE   IP              NODE                          NOMINATED NODE   READINESS GATES
monitoring-kube-prometheus-stack-prometheus-node-exporter-tq6g5   1/1     Running   0               16d   <IP_ADDRESS>   green-reviews-control-plane   <none>           <none>
monitoring-kube-prometheus-stack-prometheus-node-exporter-f4wh9   1/1     Running   9 (2d10h ago)   16d   <IP_ADDRESS>   green-reviews-worker1         <none>           <none>

Once the 1Password team is ready we can get you readonly access to the cluster. As I know working on the automation without cluster access can be challenging.

so node exporter uses kube-state-metrics??

The helm chart creates a separate daemonset
k -n monitoring get ds
NAME                                                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
monitoring-kube-prometheus-stack-prometheus-node-exporter   2         2         2       2            2           kubernetes.io/os=linux   16d

so your manifest file worked?

Yes but I'll re-test on the cluster once the extra worker node is created

in my system the nodeexporter is not running and is pending

@dipankardas011 It uses port 9100 is that available in your cluster?
On the Equinix cluster it's running OK. The restarts are because of a Falco load test which is stressing the box 😅
k -n monitoring -l app.kubernetes.io/name=prometheus-node-exporter get po -o wide
NAME                                                              READY   STATUS    RESTARTS        AGE   IP              NODE                          NOMINATED NODE   READINESS GATES
monitoring-kube-prometheus-stack-prometheus-node-exporter-tq6g5   1/1     Running   0               16d   <IP_ADDRESS>   green-reviews-control-plane   <none>           <none>
monitoring-kube-prometheus-stack-prometheus-node-exporter-f4wh9   1/1     Running   9 (2d10h ago)   16d   <IP_ADDRESS>   green-reviews-worker1         <none>           <none>

Once the 1Password team is ready we can get you readonly access to the cluster. As I know working on the automation without cluster access can be challenging.

Okay if it works on equinox then okay
Actually I was using civo k3s


Fix crash around customizing NSURLCache size.

In our app, we're experiencing a crash due to use of custom size for NSURLCache, which gets fixed as soon as we stop customizing the cache size. This crash only happens on devices below iOS 8.2, which is small portion but it'd still be nice to see this gets fixed in the API.
Discussion around this bug can be found here: http://stackoverflow.com/questions/27233448/ios-crasher-cfnetwork-httpreadfilterdoplainreadstreamreader-unsigned-char/28464227#28464227
Below is the stack trace.
Thread 4 Crashed:
0   libsystem_platform.dylib             0x3568fb7a _platform_memmove + 186
1   CFNetwork                            0x26f862c7 HTTPReadFilter::doPlainRead(StreamReader*, unsigned char*, long, CFStreamError*, unsigned char*) + 176
2   CFNetwork                            0x26f8611b HTTPReadFilter::_streamImpl_Read(unsigned char*, long, CFStreamError*, unsigned char*) + 392
3   CFNetwork                            0x26f6dd73 CoreStreamBase::_streamInterface_Read(unsigned char*, long) + 96
4   CFNetwork                            0x26ff13a3 HTTPNetStreamInfo::_streamImpl_Read(__CFReadStream*, unsigned char*, long, CFStreamError*, unsigned char*) + 304
5   CFNetwork                            0x2705fc3d CFNetworkReadStream::httpStreamRead(__CFReadStream*, unsigned char*, long, CFStreamError*, unsigned char*, void*) + 42
6   CoreFoundation                       0x274521a9 CFReadStreamRead + 326
7   CFNetwork                            0x26ff1ee9 HTTPNetStreamInfo::_readStreamClientCallBack(__CFReadStream*, unsigned long) + 126
8   CFNetwork                            0x2705fd57 CFNetworkReadStream::_readStreamClientCallBackCallBack(__CFReadStream*, unsigned long, void*) + 36
9   CoreFoundation                       0x274916d7 _signalEventSync + 144
10  CoreFoundation                       0x27491633 _cfstream_shared_signalEventSync + 340
11  CoreFoundation                       0x274e0d57 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 12
12  CoreFoundation                       0x274e01f9 __CFRunLoopDoSources0 + 362
13  CoreFoundation                       0x274de7cd __CFRunLoopRun + 770
14  CoreFoundation                       0x2742c3c1 CFRunLoopRunSpecific + 474
15  CoreFoundation                       0x2742c1d3 CFRunLoopRunInMode + 104
16  CFNetwork                            0x26fe1957 +[NSURLConnection(Loader) _resourceLoadLoop:] + 484
17  Foundation                           0x2822ab5b __NSThread__main__ + 1116
18  libsystem_pthread.dylib              0x35695e93 _pthread_body + 136
19  libsystem_pthread.dylib              0x35695e07 _pthread_start + 116
20  libsystem_pthread.dylib              0x35693b90 thread_start + 6

Thanks for calling this out!
Can you please add more documentation to the code itself referencing the radar, dev forum, and stack overflow?
I think you should also use [[NSProcessInfo processInfo] operatingSystemVersion] instead, and we also need to consider watchOS/macOS/tvOS

Actually I was incorrect. The NSProcessInfo stuff isn't available in iOS 7, which this version of AF still supports.

See #3735


Question about detach

Hi,
Thanks for the nice work. For calculating the IoU target, I think the detach should be used on the "predicted items". Specifically, in my view, for these two lines with detach applied, the detach should be moved to their previous lines, i.e. line 408 and 424. Correct me if I miss something.
https://github.com/hyz-xmaster/VarifocalNet/blob/master/mmdet/models/dense_heads/vfnet_head.py#L409
https://github.com/hyz-xmaster/VarifocalNet/blob/master/mmdet/models/dense_heads/vfnet_head.py#L425
Could you please explain it more? Thanks.

Hi, we would like the gradient of losses to flow to pos_decoded_bbox_preds and pos_decoded_bbox_preds_refine so as to optimize the predictions, so we don't detach them. Conversely, we don't want the gradient to back-propagate to pos_decoded_target_preds.

As I see in the QFL implementation, it uses pos_decoded_bbox_preds to compute score and then uses the score here. It seems QFL uses the score and label to compute the target for  the cls_score, which has the similar function of pos_decoded_target_preds. You should not propagate the gradient to targets.

@hyz-xmaster Exactly, I see that QFL does not and should not propagate gradients to pos_decoded_bbox_preds. Then back to my above question, why did VFL choose to propagate gradients to pos_decoded_bbox_preds, without hurting the performance?

See this line, we detach the iou_targets_ini calculated from pos_decoded_bbox_preds, and this line.
We only propagate gradients to pos_decoded_bbox_preds here.

@zen-d
Though I only took a cursory look at your question, I think you've made a misconception. The line you quoted refers to the calculation of box regression loss, in which the gradient of predicted box regression value should not be detached. What'more, varifocal loss does not apply to the calculation of regression loss, it serves as a cls loss. So the only thing we should take care is to detach the gradient of cls_target in this line.


#11: Support new hetzner machines

Description

Add support for dedicated machines
Related Issue(s)

Fixes #11
How to test

@mrsimonemms  sure thing!
btw. if I already have you here and you seem to have  worked at gitpod. What kind of solution would you recommend to go for right now? We are starting a start-up and want to set up everything in the cloud for a good dev experience. Would love to hear your opinion on that.

In my opinion, Gitpod is still the best cloud-dev environment out there. It combines the most mature user experience and works pretty well in most use-cases. Where it falls down are:

No support for dev Kubernetes (see https://github.com/gitpod-io/gitpod/issues/4889). It's 2+ years old and still no sign of coming. This probably won't be a deal-breaker, but it's definitely a thing to be aware of
Only worth doing if using gitpod.io. Dedicated is very expensive and no self-hosted option any more (disclaimer: I was lead of SH).

DevPod is pretty good but, truthfully, I don't see the advantage of it over dev containers run locally. I've never really gotten on with Coder or CodeSpaces. There are others out there (eg, Daytona) but I've never used them.
What I really miss is the ability to just go to a branch/PR/tag, click "open in Gitpod" and the environment is fully configured on that commit. Most of the ones I've mentioned don't have that functionality and don't seem that interested in getting it, but to my mind that's the advantage of a CDE - I can review a PR for a teammate without losing what I'm currently working on.
I'm leading a platform team at the moment and I want to set up a cloud-dev environment but we run everything on-prem behind a firewall so neither Gitpod.io not Dedicated is an option for us - I'd love to get Gitpod self-hosted running, but they gave up on it for the wrong reasons in my opinion.

@juliuslipp can you sign the commit as well please?
Once that's in and merged, I'll publish a new release.

@mrsimonemms thanks for the detailed answer! We checked out gitpod.io, but with our "broke student" minds, it seems pretty expensive. I agree that devpod probably isn't the right thing. We just checked out Coder. It's easy to deploy on f.e. fly and you can use Hetzner or similar cloud providers for dramatically reduced cost (32GB Ram + 16 Cores for 40-50 bucks per month running 24/7). Since we want to do everything in the cloud and are ML heavy this seems like the better option with beefier machines.
We don't know what to use yet. The PR features seems overly nice, but might be coming soon to coder as well. Difficult decisions :D
Will sign. Thanks again.

 It proves you  as owner of intellectual work and essential for open source - they're pretty useful in closed source too

Good to know. Thanks! Thanks for merging as well :)


Add some regular contributors to allow access to retry CI

PR Summary
Give some regular contributors access to request @PoshChan to retry CI builds: @KirkMunro, @vexx32, @bergmeister
PR Checklist

[x] PR has a meaningful title

Use the present tense and imperative mood when describing your changes

[x] Summarized changes
[x] Make sure all .h, .cpp, .cs, .ps1 and .psm1 files have the correct copyright header
[x] This PR is ready to merge and is not Work in Progress.

If the PR is work in progress, please add the prefix WIP: or [ WIP ] to the beginning of the title (the WIP bot will keep its status check at Pending while the prefix is present) and remove the prefix when the PR is ready.

Breaking changes

[x] None
OR
[ ] Experimental feature(s) needed

[ ] Experimental feature name(s):

User-facing changes

[x] Not Applicable
OR
[ ] Documentation needed

[ ] Issue filed:

Testing - New and feature

[x] N/A or can only be tested interactively
OR
[ ] Make sure you've added a new test if existing tests do not effectively test the code changed

Tooling

[x] I have considered the user experience from a tooling perspective and don't believe tooling will be impacted.
OR
[ ] I have considered the user experience from a tooling perspective and enumerated concerns in the summary. This may include:

Impact on PowerShell Editor Services which is used in the PowerShell extension for VSCode (which runs in a different PS Host).
Impact on Completions (both in the console and in editors) - one of PowerShell's most powerful features.
Impact on PSScriptAnalyzer (which provides linting & formatting in the editor extensions).
Impact on EditorSyntax (which provides syntax highlighting with in VSCode, GitHub, and many other editors).

Thanks :-)

@PoshChan 	Please remind me in 23 hours

@TravisEz13, this is the reminder you requested 23 hours ago


Community: integrate chat models with Yuan2.0

integrate chat models with Yuan2.0
add a new doc for Yuan2.0 integration

Yuan2.0 is a new generation Fundamental Large Language Model developed by IEIT System. We have published all three models, Yuan 2.0-102B, Yuan 2.0-51B, and Yuan 2.0-2B.

I've deleted unnecessary import block and re-format related code block, it still failed in deployment, Can you give me some tips how to fix this? Thanks. @baskaryan @hwchase17

I've fix some lint error.  And I can see checks failed with:
    from langchain_community.chat_models.yuan2 import ChatYuan2
  File "/home/runner/work/langchain/langchain/libs/community/langchain_community/chat_models/yuan2.py", line 55, in <module>
    from openai.types.chat import (
ModuleNotFoundError: No module named 'openai'

But I'm not familier with poetry,  my code was depended on openai package, but it seems that this package was not installed while check was running. How can I fix this?
Any suggestions would be appreciate. Thanks @baskaryan @hwchase17

Any suggestions or any updates here? @baskaryan @hwchase17


Taxonomy | Internal demo suggestions

[x] Internal demo suggestions(CS-42076)
[x] SDK taxonomies find issue (1.11.0 -> 1.12.0)
[x] Merge development

@aman19K Can you please update your base branch with development so that your PR will only have your code changes instead of showing audit-related code changes as well?

#1111


Gogs should allow usage of self signed certificates

When trying to connect via Port 636 to an active directory domain controller I see the following error:
2015/09/14 18:11:39 [T] LDAP will use BindDN.
2015/09/14 18:11:39 [D] Using TLS for LDAP
2015/09/14 18:11:39 [...es/auth/ldap/ldap.go:91 SearchEntry()] [E] LDAP Connect error, AD01.example.com:LDAP Result Code 200 "": x509: certificate signed by unknown authority

From my understanding there are two possible solutions - either don't enforce strict checking via a checkbox or make it possible to provide the necessary ca information to gogs.

Hi, I've pushed a fix to develop which now you can check skip verify on LDAP settings.

@Unknwon I can confirm that I'm able to open up a encrypted connection with ldap bind dn over port 636 against the "load-balanced" address example.com and no more errors show up.

@hhenkel thanks your confirmation!


Sleeping external drive slow down search results

ℹ Computer information

PowerToys version: 0.25.0
PowerToy Utility: Run
Running PowerToys as Admin: No
Windows build number: [run "winver"] : 18363.1198

📝 Reproduction steps

Having a sleeping external hard drive (in my case, WD 4To)
Open the PowerToys Run
Type something

✔ Expected result

Research immediately
Show results

❌ Actual result

Waiting for all external drives to wake up (takes 5s+)
Show results

@mykhailopylyp would yorur tweak in #8367 resolve this?

would yorur tweak in #8367 resolve this?

Yes, it should resolve the above issue.

Fixed with 0.29, released today.  https://github.com/microsoft/PowerToys/releases/tag/v0.29.0


embedding generation failed. wsarecv: An existing connection was forcibly closed by the remote host.

What is the issue?
embedding model
When I submit a single fragment, it responds normally, but when I submit multiple fragments, an exception occurs.
I encountered this error on different Windows systems as well.
This issue occurs in both versions 0.3.14 and 0.4.0-rc3. However, I also tested versions 0.3.13 and 0.3.10, and they work perfectly.
[GIN] 2024/10/21 - 16:00:29 | 200 |    722.8624ms |   <IP_ADDRESS> | POST     "/api/embed"
ggml.c:13343: GGML_ASSERT(i01 >= 0 && i01 < ne01) failed
ggml.c:13343: GGML_ASSERT(i01 >= 0 && i01 < ne01) failed
time=2024-10-21T16:00:36.434+08:00 level=ERROR source=routes.go:434 msg="embedding generation failed" error="do embedding request: Post \"http://<IP_ADDRESS>:64075/embedding\": read tcp <IP_ADDRESS>:64078-><IP_ADDRESS>:64075: wsarecv: An existing connection was forcibly closed by the remote host."
[GIN] 2024/10/21 - 16:00:36 | 500 |    6.5660285s |   <IP_ADDRESS> | POST     "/api/embed"
time=2024-10-21T16:01:00.723+08:00 level=INFO source=llama-server.go:72 msg="system memory" total="15.9 GiB" free="10.3 GiB" free_swap="8.8 GiB"
time=2024-10-21T16:01:00.726+08:00 level=INFO source=memory.go:346 msg="offload to cpu" layers.requested=-1 layers.model=25 layers.offload=0 layers.split="" memory.available="[10.3 GiB]" memory.gpu_overhead="0 B" memory.required.full="687.0 MiB" memory.required.partial="0 B" memory.required.kv="12.0 MiB" memory.required.allocations="[687.0 MiB]" memory.weights.total="589.2 MiB" memory.weights.repeating="548.0 MiB" memory.weights.nonrepeating="41.3 MiB" memory.graph.full="32.0 MiB" memory.graph.partial="32.0 MiB"
time=2024-10-21T16:01:00.730+08:00 level=INFO source=llama-server.go:355 msg="starting llama server" cmd="C:\\Users\\Administrator\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\runners\\cpu_avx2\\ollama_llama_server.exe --model C:\\Users\\Administrator\\.ollama\\models\\blobs\\sha256-9e8e196fa3f73c32fb1b37503d5c28b166f4a96db54addd89927c47e4e40cf68 --ctx-size 2048 --batch-size 512 --embedding --threads 4 --no-mmap --parallel 1 --port 64090"
time=2024-10-21T16:01:00.782+08:00 level=INFO source=sched.go:450 msg="loaded runners" count=1
time=2024-10-21T16:01:00.791+08:00 level=INFO source=llama-server.go:534 msg="waiting for llama runner to start responding"
time=2024-10-21T16:01:00.792+08:00 level=INFO source=llama-server.go:568 msg="waiting for server to become available" status="llm server error"
time=2024-10-21T16:01:00.812+08:00 level=INFO source=runner.go:856 msg="starting go runner"
time=2024-10-21T16:01:00.829+08:00 level=INFO source=.:0 msg="Server listening on <IP_ADDRESS>:64090"
llama_model_loader: loaded meta data with 23 key-value pairs and 389 tensors from C:\Users\Administrator\.ollama\models\blobs\shp4 _llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = bert
llama_model_loader: - kv   1:                               general.name str              = model
llama_model_loader: - kv   2:                           bert.block_count u32              = 24
llama_model_loader: - kv   3:                        bert.context_length u32              = 512
llama_model_loader: - kv   4:                      bert.embedding_length u32              = 1024
llama_model_loader: - kv   5:                   bert.feed_forward_length u32              = 4096
llama_model_loader: - kv   6:                  bert.attention.head_count u32              = 16
llama_model_loader: - kv   7:          bert.attention.layer_norm_epsilon f32              = 0.000000
llama_model_loader: - kv   8:                          general.file_type u32              = 1
llama_model_loader: - kv   9:                      bert.attention.causal bool             = false
llama_model_loader: - kv  10:                          bert.pooling_type u32              = 1
llama_model_loader: - kv  11:            tokenizer.ggml.token_type_count u32              = 2
llama_model_loader: - kv  12:                tokenizer.ggml.bos_token_id u32              = 101
llama_model_loader: - kv  13:                tokenizer.ggml.eos_token_id u32              = 102
llama_model_loader: - kv  14:                       tokenizer.ggml.model str              = bert
llama_model_loader: - kv  15:                      tokenizer.ggml.tokens arr[str,21128]   = ["[PAD]", "[unused1]", "[unused2]",  0 _llama_model_loader: - kv  16:                      tokenizer.ggml.scores arr[f32,21128]   = [-1000.000000, -1000.000000, -1000.0 - _llama_model_loader: - kv  17:                  tokenizer.ggml.token_type arr[i32,21128]   = [3, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,P3 _llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 100
llama_model_loader: - kv  19:          tokenizer.ggml.seperator_token_id u32              = 102
llama_model_loader: - kv  20:            tokenizer.ggml.padding_token_id u32              = 0
llama_model_loader: - kv  21:                tokenizer.ggml.cls_token_id u32              = 101
llama_model_loader: - kv  22:               tokenizer.ggml.mask_token_id u32              = 103
llama_model_loader: - type  f32:  243 tensors
llama_model_loader: - type  f16:  146 tensors
llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
llm_load_vocab: special tokens cache size = 5
llm_load_vocab: token to piece cache size = 0.0769 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = bert
llm_load_print_meta: vocab type       = WPM
llm_load_print_meta: n_vocab          = 21128
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: vocab_only       = 0
llm_load_print_meta: n_ctx_train      = 512
llm_load_print_meta: n_embd           = 1024
llm_load_print_meta: n_layer          = 24
llm_load_print_meta: n_head           = 16
llm_load_print_meta: n_head_kv        = 16
llm_load_print_meta: n_rot            = 64
llm_load_print_meta: n_swa            = 0
llm_load_print_meta: n_embd_head_k    = 64
llm_load_print_meta: n_embd_head_v    = 64
llm_load_print_meta: n_gqa            = 1
llm_load_print_meta: n_embd_k_gqa     = 1024
llm_load_print_meta: n_embd_v_gqa     = 1024
llm_load_print_meta: f_norm_eps       = 1.0e-012
llm_load_print_meta: f_norm_rms_eps   = 0.0e+000
llm_load_print_meta: f_clamp_kqv      = 0.0e+000
llm_load_print_meta: f_max_alibi_bias = 0.0e+000
llm_load_print_meta: f_logit_scale    = 0.0e+000
llm_load_print_meta: n_ff             = 4096
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 0
llm_load_print_meta: pooling type     = 1
llm_load_print_meta: rope type        = 2
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 512
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: ssm_dt_b_c_rms   = 0
llm_load_print_meta: model type       = 335M
llm_load_print_meta: model ftype      = F16
llm_load_print_meta: model params     = 324.47 M
llm_load_print_meta: model size       = 619.50 MiB (16.02 BPW)
llm_load_print_meta: general.name     = model
llm_load_print_meta: BOS token        = 101 '[CLS]'
llm_load_print_meta: EOS token        = 102 '[SEP]'
llm_load_print_meta: UNK token        = 100 '[UNK]'
llm_load_print_meta: SEP token        = 102 '[SEP]'
llm_load_print_meta: PAD token        = 0 '[PAD]'
llm_load_print_meta: CLS token        = 101 '[CLS]'
llm_load_print_meta: MASK token       = 103 '[MASK]'
llm_load_print_meta: LF token         = 0 '[PAD]'
llm_load_print_meta: EOG token        = 102 '[SEP]'
llm_load_print_meta: max token length = 48
llm_load_tensors: ggml ctx size =    0.16 MiB
llm_load_tensors:        CPU buffer size =   619.50 MiB
time=2024-10-21T16:01:01.048+08:00 level=INFO source=llama-server.go:568 msg="waiting for server to become available" status="llm server loading model"
llama_new_context_with_model: n_ctx      = 2048
llama_new_context_with_model: n_batch    = 512
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:        CPU KV buffer size =   192.00 MiB
llama_new_context_with_model: KV self size  =  192.00 MiB, K (f16):   96.00 MiB, V (f16):   96.00 MiB
llama_new_context_with_model:        CPU  output buffer size =     0.00 MiB
llama_new_context_with_model:        CPU compute buffer size =    26.00 MiB
llama_new_context_with_model: graph nodes  = 851
llama_new_context_with_model: graph splits = 1
time=2024-10-21T16:01:01.299+08:00 level=INFO source=llama-server.go:573 msg="llama runner started in 0.51 seconds"
llama_model_loader: loaded meta data with 23 key-value pairs and 389 tensors from C:\Users\Administrator\.ollama\models\blobs\sh c s3llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = bert
llama_model_loader: - kv   1:                               general.name str              = model
llama_model_loader: - kv   2:                           bert.block_count u32              = 24
llama_model_loader: - kv   3:                        bert.context_length u32              = 512
llama_model_loader: - kv   4:                      bert.embedding_length u32              = 1024
llama_model_loader: - kv   5:                   bert.feed_forward_length u32              = 4096
llama_model_loader: - kv   6:                  bert.attention.head_count u32              = 16
llama_model_loader: - kv   7:          bert.attention.layer_norm_epsilon f32              = 0.000000
llama_model_loader: - kv   8:                          general.file_type u32              = 1
llama_model_loader: - kv   9:                      bert.attention.causal bool             = false
llama_model_loader: - kv  10:                          bert.pooling_type u32              = 1
llama_model_loader: - kv  11:            tokenizer.ggml.token_type_count u32              = 2
llama_model_loader: - kv  12:                tokenizer.ggml.bos_token_id u32              = 101
llama_model_loader: - kv  13:                tokenizer.ggml.eos_token_id u32              = 102
llama_model_loader: - kv  14:                       tokenizer.ggml.model str              = bert
llama_model_loader: - kv  15:                      tokenizer.ggml.tokens arr[str,21128]   = ["[PAD]", "[unused1]", "[unused2]", `w s3llama_model_loader: - kv  16:                      tokenizer.ggml.scores arr[f32,21128]   = [-1000.000000, -1000.000000, -1000.0 ~ s3llama_model_loader: - kv  17:                  tokenizer.ggml.token_type arr[i32,21128]   = [3, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, u s3llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 100
llama_model_loader: - kv  19:          tokenizer.ggml.seperator_token_id u32              = 102
llama_model_loader: - kv  20:            tokenizer.ggml.padding_token_id u32              = 0
llama_model_loader: - kv  21:                tokenizer.ggml.cls_token_id u32              = 101
llama_model_loader: - kv  22:               tokenizer.ggml.mask_token_id u32              = 103
llama_model_loader: - type  f32:  243 tensors
llama_model_loader: - type  f16:  146 tensors
llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
llm_load_vocab: special tokens cache size = 5
llm_load_vocab: token to piece cache size = 0.0769 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = bert
llm_load_print_meta: vocab type       = WPM
llm_load_print_meta: n_vocab          = 21128
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: vocab_only       = 1
llm_load_print_meta: model type       = ?B
llm_load_print_meta: model ftype      = all F32
llm_load_print_meta: model params     = 324.47 M
llm_load_print_meta: model size       = 619.50 MiB (16.02 BPW)
llm_load_print_meta: general.name     = model
llm_load_print_meta: BOS token        = 101 '[CLS]'
llm_load_print_meta: EOS token        = 102 '[SEP]'
llm_load_print_meta: UNK token        = 100 '[UNK]'
llm_load_print_meta: SEP token        = 102 '[SEP]'
llm_load_print_meta: PAD token        = 0 '[PAD]'
llm_load_print_meta: CLS token        = 101 '[CLS]'
llm_load_print_meta: MASK token       = 103 '[MASK]'
llm_load_print_meta: LF token         = 0 '[PAD]'
llm_load_print_meta: EOG token        = 102 '[SEP]'
llm_load_print_meta: max token length = 48
llama_model_load: vocab only - skipping tensors
[GIN] 2024/10/21 - 16:01:01 | 200 |    701.8355ms |   <IP_ADDRESS> | POST     "/api/embed"
ggml.c:13343: GGML_ASSERT(i01 >= 0 && i01 < ne01) failed
ggml.c:13343: GGML_ASSERT(i01 >= 0 && i01 < ne01) failed
time=2024-10-21T16:01:08.177+08:00 level=ERROR source=routes.go:434 msg="embedding generation failed" error="do embedding request: Post \"http://<IP_ADDRESS>:64090/embedding\": read tcp <IP_ADDRESS>:64093-><IP_ADDRESS>:64090: wsarecv: An existing connection was forcibly closed by the remote host."

OS
Windows
GPU
No response
CPU
Intel
Ollama version
0.3.14 0.4.0-rc3

Which model are you using?

Which model are you using?

I tried many embedding models, including 893379029/piccolo-large-zh-v2 and viosay/conan-embedding-v1, and they all have the same issue, although they worked perfectly fine before. However, a few models, like shaw/dmeta-embedding-zh, do not have this problem.

I am unable to replicate:
$ curl -s http://localhost:11434/api/version
{"version":"0.3.14"}
$ curl -s localhost:11434/api/embed -d '{"model":"viosay/conan-embedding-v1","input":"Why is the sky blue?"}' | jq '.embeddings=[.embeddings[]|length]'
{
  "model": "viosay/conan-embedding-v1",
  "embeddings": [
    1024
  ],
  "total_duration": 154636185,
  "load_duration": 2031329,
  "prompt_eval_count": 6
}
$ curl -s localhost:11434/api/embed -d '{"model":"viosay/conan-embedding-v1","input":["Why is the sky blue?","why is the grass green"]}' | jq '.embeddings=[.embeddings[]|length]'
{
  "model": "viosay/conan-embedding-v1",
  "embeddings": [
    1024,
    1024
  ],
  "total_duration": 306792338,
  "load_duration": 2301978,
  "prompt_eval_count": 13
}
$ curl -s localhost:11434/api/embed -d '{"model":"viosay/conan-embedding-v1","input":["Why is the sky blue?","why is the grass green","one","two","three"]}' | jq '.embeddings=[.embe
ddings[]|length]'
{
  "model": "viosay/conan-embedding-v1",
  "embeddings": [
    1024,
    1024,
    1024,
    1024,
    1024
  ],
  "total_duration": 628749251,
  "load_duration": 3847279,
  "prompt_eval_count": 16
}

It  might have something to with the client or the length of the inputs.  Can you provide more context on your usage, or better yet, a script that demonstrates the problem.

@rick-github You're right. Based on my tests, the issue is indeed related to the input length. When it exceeds a certain length, an error occurs.
It’s like the example below, where an error occurred.
curl -s localhost:11434/api/embed -d '{"model":"viosay/conan-embedding-v1","input":"The Spring AI project aims for complexity. This project draws inspiration from wellknown Python projects such as LangChain and LlamaIndex, but Spring AI is not a direct port of these projects. The belief behind the establishment of this project is that the next wave of generative AI applications will not only be suitable for Python developers, but will also be ubiquitous in many programming languages. The core of Spring AI is to provide abstraction as the foundation for developing AI applications. These abstractions have multiple implementations and can easily exchange components with minimal code changes. Spring AI Provide the following features: 1. Support all major model providers such as OpenAI, Microsoft, Amazon, Google, and Huggingface. 2. The supported model types include chat and text to image, and more types are currently under development. 3. A portable API across AI providers for chatting and embedding models. Supports synchronization and streaming API options. It also supports dropdown to access model specific features. 4. Map the output of the AI model to POJO. 5. Support all major vector database providers, such as Azure Vector Search, Chroma, Milvus, Neo4j, PostgreSQLPGVector, PineCone, Qdrant, Redis, and Weaviate6. Cross Vector Store providers portable API, including a novel metadata filter API similar to SQL, which is also portable. 7. Function calls 8. Spring Boot auto configuration and launcher for AI models and vector storage. 9. ETL framework for data engineering. With this feature set, you can implement common use cases such as document QA or chatting with documents. The concept section provides a high-level overview of AI concepts and their representation in Spring AI. The Introduction section explains how to create the first AI application. The subsequent sections will adopt a code centered approach to delve into each component and common use cases."}'

viosay/conan-embedding-v1 has an embedding length of 1024 and your test text is 1905 bytes, so it's exceeding the window.  The client should chunk the text to segments smaller than the embedding length otherwise the returned embeddings will be missing semantic content.  However, ollama (or actually llama.cpp) should handle the situation more gracefully.

viosay/conan-embedding-v1 has an embedding length of 1024 and your test text is 1905 bytes, so it's exceeding the window. The client should chunk the text to segments smaller than the embedding length otherwise the returned embeddings will be missing semantic content. However, ollama (or actually llama.cpp) should handle the situation more gracefully.viosay/conan-embedding-v1的嵌入长度为1024，而你的测试文本是1905字节，所以它超过了窗口。 客户端应该将文本分块到小于嵌入长度的片段，否则返回的嵌入将丢失语义内容。 然而，ollama（或者实际上是llama.cpp）应该更优雅地处理这种情况。

Thank you for your response! The key issue is that this problem did not exist in versions prior to Ollama 0.3.14. I had been using various models without any issues before that.

For example, the shaw/dmeta-embedding-zh model, which has  an embedding length of 768, does not encounter this issue.

@rick-github Thank you very much! I will follow your advice. One more thing I noticed is that most of the models that encounter issues are those imported after being converted to GGUF using the convert_hf_to_gguf.py script from llama.cpp. I'm not sure if this is the cause of the problem.

Yes, I understand the meaning of embedding length. Whether it’s 512 or 1024, they are both less than 1905. This is a puzzling issue, and as you mentioned, it seems to have arisen after Ollama updated llama.cpp. I'll continue testing and verifying the specific situation. Thank you!

Tokens are different to characters.  A token is a sequence of characters, on average 2  or 3 characters in length.  So a token length of 512 would handle 1024-1536 characters, and a token length of 1024 would handle 2048-3072 characters.

I think I’ve figured out the issue; it seems that the truncate didn’t take effect.

It does truncate, it's just the runner throws an GGML_ASSERT(i01 >= 0 && i01 < ne01) failed exception and crashes when the number of tokens is close to the maximum allowed and the runner has been started with a context window greater than the actual supported value.  If the model is loaded with the context size set to the actual supported context size, it works fine:
curl -s localhost:11434/api/embed -d '{"model":"viosay/conan-embedding-v1","options":{"num_ctx":512},"input":"The Spring AI project aims for complexity. This project draws inspiration from wellknown Python projects such as LangChain and LlamaIndex, but Spring AI is not a direct port of these projects. The belief behind the establishment of this project is that the next wave of generative AI applications will not only be suitable for Python developers, but will also be ubiquitous in many programming languages. The core of Spring AI is to provide abstraction as the foundation for developing AI applications. These abstractions have multiple implementations and can easily exchange components with minimal code changes. Spring AI Provide the following features: 1. Support all major model providers such as OpenAI, Microsoft, Amazon, Google, and Huggingface. 2. The supported model types include chat and text to image, and more types are currently under development. 3. A portable API across AI providers for chatting and embedding models. Supports synchronization and streaming API options. It also supports dropdown to access model specific features. 4. Map the output of the AI model to POJO. 5. Support all major vector database providers, such as Azure Vector Search, Chroma, Milvus, Neo4j, PostgreSQLPGVector, PineCone, Qdrant, Redis, and Weaviate6. Cross Vector Store providers portable API, including a novel metadata filter API similar to SQL, which is also portable. 7. Function calls 8. Spring Boot auto configuration and launcher for AI models and vector storage. 9. ETL framework for data engineering. With this feature set, you can implement common use cases such as document QA or chatting with documents. The concept section provides a high-level overview of AI concepts and their representation in Spring AI. The Introduction section explains how to create the first AI application. The subsequent sections will adopt a code centered approach to delve into each component and common use cases."}' | jq '.embeddings=[.embeddings[]|length]'

@viosay Hi, I met the same error, do you have any solution to solve it? I test many embeddings, but only mxbai-embed-large and nomic-embed-text works fine.

@viosay Hi, I met the same error, do you have any solution to solve it? I test many embeddings, but only mxbai-embed-large and nomic-embed-text works fine.

There’s no good solution for now. Either roll back to version 0.3.13, or try setting the actual context size as Rick suggested above. However, there’s a risk of semantic loss in the returned embeddings. It might be best to first try splitting the text into chunks smaller than the embedding length on the client side, but due to differences in token calculation methods, your chunks may not match the model’s segmentation precisely.

Hi @viosay and @rick-github ,
I met the same error and tried to reduce the length of chunk tokens sent to embedding api. However, it seems not working. I use model milkey/gte:large-zh-f16 and set embedding_dim to 1024 with chunk_token_size equals to 1200 in lightrag framework which access embed model via ollama. The 500 internal err is not changed even I modify the chunk_token_size to 400.

It works if I set num_ctx to 512.  Perhaps the lightrag framework is adding extra tokens, or there is an issue wiith chunk_token_size.
$ ollama show milkey/gte:large-zh-f16
  Model
  	arch            	bert
  	parameters      	324M
  	quantization    	F16
  	context length  	512
  	embedding length	1024

$ curl -s localhost:11434/api/embed -d '{"model":"milkey/gte:large-zh-f16","options":{"num_ctx":1200},"input":"The Spring AI project aims for complexity. This project draws inspiration from wellknown Python projects such as LangChain and LlamaIndex, but Spring AI is not a direct port of these projects. The belief behind the establishment of this project is that the next wave of generative AI applications will not only be suitable for Python developers, but will also be ubiquitous in many programming languages. The core of Spring AI is to provide abstraction as the foundation for developing AI applications. These abstractions have multiple implementations and can easily exchange components with minimal code changes. Spring AI Provide the following features: 1. Support all major model providers such as OpenAI, Microsoft, Amazon, Google, and Huggingface. 2. The supported model types include chat and text to image, and more types are currently under development. 3. A portable API across AI providers for chatting and embedding models. Supports synchronization and streaming API options. It also supports dropdown to access model specific features. 4. Map the output of the AI model to POJO. 5. Support all major vector database providers, such as Azure Vector Search, Chroma, Milvus, Neo4j, PostgreSQLPGVector, PineCone, Qdrant, Redis, and Weaviate6. Cross Vector Store providers portable API, including a novel metadata filter API similar to SQL, which is also portable. 7. Function calls 8. Spring Boot auto configuration and launcher for AI models and vector storage. 9. ETL framework for data engineering. With this feature set, you can implement common use cases such as document QA or chatting with documents. The concept section provides a high-level overview of AI concepts and their representation in Spring AI. The Introduction section explains how to create the first AI application. The subsequent sections will adopt a code centered approach to delve into each component and common use cases."}'
{"error":{}}
$ curl -s localhost:11434/api/embed -d '{"model":"milkey/gte:large-zh-f16","options":{"num_ctx":512},"input":"The Spring AI project aims for complexity. This project draws inspiration from wellknown Python projects such as LangChain and LlamaIndex, but Spring AI is not a direct port of these projects. The belief behind the establishment of this project is that the next wave of generative AI applications will not only be suitable for Python developers, but will also be ubiquitous in many programming languages. The core of Spring AI is to provide abstraction as the foundation for developing AI applications. These abstractions have multiple implementations and can easily exchange components with minimal code changes. Spring AI Provide the following features: 1. Support all major model providers such as OpenAI, Microsoft, Amazon, Google, and Huggingface. 2. The supported model types include chat and text to image, and more types are currently under development. 3. A portable API across AI providers for chatting and embedding models. Supports synchronization and streaming API options. It also supports dropdown to access model specific features. 4. Map the output of the AI model to POJO. 5. Support all major vector database providers, such as Azure Vector Search, Chroma, Milvus, Neo4j, PostgreSQLPGVector, PineCone, Qdrant, Redis, and Weaviate6. Cross Vector Store providers portable API, including a novel metadata filter API similar to SQL, which is also portable. 7. Function calls 8. Spring Boot auto configuration and launcher for AI models and vector storage. 9. ETL framework for data engineering. With this feature set, you can implement common use cases such as document QA or chatting with documents. The concept section provides a high-level overview of AI concepts and their representation in Spring AI. The Introduction section explains how to create the first AI application. The subsequent sections will adopt a code centered approach to delve into each component and common use cases."}' | jq '.embeddings=[.embeddings[]|length]'
{
  "model": "milkey/gte:large-zh-f16",
  "embeddings": [
    1024
  ],
  "total_duration":<PHONE_NUMBER>,
  "load_duration":<PHONE_NUMBER>,
  "prompt_eval_count": 512
}

It works if I set num_ctx to 512. Perhaps the lightrag framework is adding extra tokens, or there is an issue wiith chunk_token_size.
$ ollama show milkey/gte:large-zh-f16
  Model
  	arch            	bert
  	parameters      	324M
  	quantization    	F16
  	context length  	512
  	embedding length	1024

$ curl -s localhost:11434/api/embed -d '{"model":"milkey/gte:large-zh-f16","options":{"num_ctx":1200},"input":"The Spring AI project aims for complexity. This project draws inspiration from wellknown Python projects such as LangChain and LlamaIndex, but Spring AI is not a direct port of these projects. The belief behind the establishment of this project is that the next wave of generative AI applications will not only be suitable for Python developers, but will also be ubiquitous in many programming languages. The core of Spring AI is to provide abstraction as the foundation for developing AI applications. These abstractions have multiple implementations and can easily exchange components with minimal code changes. Spring AI Provide the following features: 1. Support all major model providers such as OpenAI, Microsoft, Amazon, Google, and Huggingface. 2. The supported model types include chat and text to image, and more types are currently under development. 3. A portable API across AI providers for chatting and embedding models. Supports synchronization and streaming API options. It also supports dropdown to access model specific features. 4. Map the output of the AI model to POJO. 5. Support all major vector database providers, such as Azure Vector Search, Chroma, Milvus, Neo4j, PostgreSQLPGVector, PineCone, Qdrant, Redis, and Weaviate6. Cross Vector Store providers portable API, including a novel metadata filter API similar to SQL, which is also portable. 7. Function calls 8. Spring Boot auto configuration and launcher for AI models and vector storage. 9. ETL framework for data engineering. With this feature set, you can implement common use cases such as document QA or chatting with documents. The concept section provides a high-level overview of AI concepts and their representation in Spring AI. The Introduction section explains how to create the first AI application. The subsequent sections will adopt a code centered approach to delve into each component and common use cases."}'
{"error":{}}
$ curl -s localhost:11434/api/embed -d '{"model":"milkey/gte:large-zh-f16","options":{"num_ctx":512},"input":"The Spring AI project aims for complexity. This project draws inspiration from wellknown Python projects such as LangChain and LlamaIndex, but Spring AI is not a direct port of these projects. The belief behind the establishment of this project is that the next wave of generative AI applications will not only be suitable for Python developers, but will also be ubiquitous in many programming languages. The core of Spring AI is to provide abstraction as the foundation for developing AI applications. These abstractions have multiple implementations and can easily exchange components with minimal code changes. Spring AI Provide the following features: 1. Support all major model providers such as OpenAI, Microsoft, Amazon, Google, and Huggingface. 2. The supported model types include chat and text to image, and more types are currently under development. 3. A portable API across AI providers for chatting and embedding models. Supports synchronization and streaming API options. It also supports dropdown to access model specific features. 4. Map the output of the AI model to POJO. 5. Support all major vector database providers, such as Azure Vector Search, Chroma, Milvus, Neo4j, PostgreSQLPGVector, PineCone, Qdrant, Redis, and Weaviate6. Cross Vector Store providers portable API, including a novel metadata filter API similar to SQL, which is also portable. 7. Function calls 8. Spring Boot auto configuration and launcher for AI models and vector storage. 9. ETL framework for data engineering. With this feature set, you can implement common use cases such as document QA or chatting with documents. The concept section provides a high-level overview of AI concepts and their representation in Spring AI. The Introduction section explains how to create the first AI application. The subsequent sections will adopt a code centered approach to delve into each component and common use cases."}' | jq '.embeddings=[.embeddings[]|length]'
{
  "model": "milkey/gte:large-zh-f16",
  "embeddings": [
    1024
  ],
  "total_duration":<PHONE_NUMBER>,
  "load_duration":<PHONE_NUMBER>,
  "prompt_eval_count": 512
}

Amazing! That works, thanks for your help!

@mokby This is what I mentioned above about Rick's suggestion to set the actual context size. However, there is a risk of semantic loss in the returned embeddings. The engine might enforce input truncation, and the real reason has yet to be identified. The most likely cause is still issues with token segmentation and calculation.

Yeah, that's may be a potential problem, can you share your solution if you can handle this issue? Many thanks

Im getting this error after
INFO:lightrag:Writing graph with 22640 nodes, 34222 edges
extract_entities
await relationships_vdb.upsert(data_for_vdb)
Is there a way around?

ChatGPT pointed out that the issue lies in the llama_server startup command, where ctx-size was set to 2048, but it should actually be 512.
cmd="C:\\Users\\Administrator\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\runners\\cpu_avx2\\ollama_llama_server.exe --model C:\\Users\\Administrator\\.ollama\\models\\blobs\\sha256-9e8e196fa3f73c32fb1b37503d5c28b166f4a96db54addd89927c47e4e40cf68 --ctx-size 2048 --batch-size 512 --threads 6 --no-mmap --parallel 1 --port 7220"

Provide a reproduction along with the debug logs when OLLAMA_DEBUG=1 is enabled.
And found that the content in the log output is inconsistent with the input text.
(https://platform.openai.com/tokenizer)Through calculation, it can be observed that the input tokens do not exceed the model's maximum supported token count of 512.

curl -s localhost:11434/api/embed -d '{"input":[" ................................................................................................................. 78\n\n ......................................................................................................... 79\n\n ......................................................................................................... 80\n\n ................................................................................................................. 80\n\n ................................................................................................................. 80\n\n ................................................................................................................. 81\n\n\n\n .......................................................................... 81\n\n .................................................................81\n\n......................................................................................................... 82\n\n ............................................................................................................. 83\n\n .......................................................................................................... 84\n\n .......................................................................................................................85\n\n .............................................................................................................. 86\n\n .............................................................................................................. 89\n\n .......................................................................................................................90\n\n .......................................................................................................................93\n\n ...................................................................................96\n\n ............................................................................. 96\n\n .....................................................96\n\n ................................................................................... 101\n\n ................................................................................... 107\n\n ........................................................................................"],"model":"viosay/conan-embedding-v1","options":{}}'

time=2024-11-27T16:39:48.728+08:00 level=INFO source=server.go:596 msg="waiting for server to become available" status="llm server loading model"
time=2024-11-27T16:39:48.728+08:00 level=DEBUG source=server.go:607 msg="model load progress 0.08"
time=2024-11-27T16:39:48.991+08:00 level=DEBUG source=server.go:607 msg="model load progress 0.19"
time=2024-11-27T16:39:49.254+08:00 level=DEBUG source=server.go:607 msg="model load progress 0.28"
llama_new_context_with_model: n_ctx      = 2048
llama_new_context_with_model: n_batch    = 512
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:        CPU KV buffer size =   192.00 MiB
llama_new_context_with_model: KV self size  =  192.00 MiB, K (f16):   96.00 MiB, V (f16):   96.00 MiB
llama_new_context_with_model:        CPU  output buffer size =     0.00 MiB
llama_new_context_with_model:        CPU compute buffer size =    26.00 MiB
llama_new_context_with_model: graph nodes  = 851
llama_new_context_with_model: graph splits = 1
time=2024-11-27T16:39:49.516+08:00 level=INFO source=server.go:601 msg="llama runner started in 1.05 seconds"
time=2024-11-27T16:39:49.516+08:00 level=DEBUG source=sched.go:462 msg="finished setting up runner" model=C:\Users\Administrator\.ollama\models\blobs\sha256-9e8e196fa3f73c32fb1b37503d5c28b166f4a96db54addd89927c47e4e40cf68
time=2024-11-27T16:39:49.524+08:00 level=DEBUG source=server.go:965 msg="new runner detected, loading model for cgo tokenization"
llama_model_loader: loaded meta data with 23 key-value pairs and 389 tensors from C:\Users\Administrator\.ollama\models\blobs\sha256-9e8e196fa3f73c32fb1b37503d5c28b166f4a96db54addd89927c47e4e40cf68 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values.  78 .  79 .  80 .  80 .
time=2024-11-27T16:39:49.551+08:00 level=DEBUG source=cache.go:104 msg="loading cache slot" id=0 cache=0 prompt=514 used=0 remaining=514
ggml.c:13343: GGML_ASSERT(i01 >= 0 && i01 < ne01) failedggml.c:13343: GGML_ASSERT(i01 >= 0 && i01 < ne01) failed

time=2024-11-27T16:39:51.654+08:00 level=ERROR source=routes.go:453 msg="embedding generation failed" error="do embedding request: Post \"http://<IP_ADDRESS>:10390/embedding\": read tcp <IP_ADDRESS>:10395-><IP_ADDRESS>:10390: wsarecv: An existing connection was forcibly closed by the remote host."
[GIN] 2024/11/27 - 16:39:51 | 500 |    3.2233719s |   <IP_ADDRESS> | POST     "/api/embed"
time=2024-11-27T16:39:51.654+08:00 level=DEBUG source=sched.go:466 msg="context for request finished"
time=2024-11-27T16:39:51.654+08:00 level=DEBUG source=sched.go:339 msg="runner with non-zero duration has gone idle, adding timer" modelPath=C:\Users\Administrator\.ollama\models\blobs\sha256-9e8e196fa3f73c32fb1b37503d5c28b166f4a96db54addd89927c47e4e40cf68 duration=5m0s
time=2024-11-27T16:39:51.654+08:00 level=DEBUG source=sched.go:357 msg="after processing request finished event" modelPath=C:\Users\Administrator\.ollama\models\blobs\sha256-9e8e196fa3f73c32fb1b37503d5c28b166f4a96db54addd89927c47e4e40cf68 refCount=0
time=2024-11-27T16:39:51.710+08:00 level=DEBUG source=server.go:423 msg="llama runner terminated" error="exit status 0xc0000409"

@rick-github I hope the latest debugging logs I provide will help identify the issue. Thanks.

The tokenizer used by OpenaAI is different to the tokenizer used by conan-embedding-v1.  You can see from your screenshots that all three OpenAI models return a different token count for the same text.  The prompt that you are using with OpenAI is not quite the same as the one you provided (1893 characters vs 1905 characters) so we need to knock off a couple of tokens, but conan-embedding-v1 creates 571 tokens.


Color replacement control in MQTT

Hi!
I'm trying to use ${color.r}/${color.g}/${color.b} in a MQTT message for setting the color of an ESP8266 based device. But it seems that those tokens are only replaced for UDP devices? If so, can this be extended to MQTT as well?
Thanks, and great work :)

You are correct! That did not get into the tcp plugin. Next release.

Just curious... Doesn't that need to go into MQTTHome?

Yes, you are correct, it needs to go there as well.


[Feature] Keyword to execute 2+ browser keywords in parallel

Is your feature request related to a problem? Please describe.
Sometimes we would have a use case to find out whether or not a network request is sent to the server after clicking a button. To do this we would use playwright's waitForRequest method and we may write the RBFW script like this. My assumption that we will implement Wait For Request keyword.
Click       #button
Wait For Request    xxx  xxxx

However the requests aren't captured consistently. On multiple occasions the command times out which results in a failed test. That is because the request is finished before waitForRequest was called.
Describe the solution you'd like
Have a keyword to execute 2 keywords in parallel e.g.
Run Async Keywords           Click       #button     AND    Wait For Request    xxx  xxxx

Describe alternatives you've considered
I have no idea at the moment.

After some thinking one idea we had for this specific need was:
How using the keyword that sets the Wait For Request wait before you execute the Click keyword.
Example of another keyword that behaves like that here:
https://github.com/MarketSquare/robotframework-browser/blob/2e2ecd8f38739bff30f15efd48f4490686aa1de1/atest/test/01_Browser_Management/new_windows.robot#L6
So basically I would suggest the syntax
Wait For Request    xxx  xxxx
Click       #button

Technically it just sets a playwright waitForEvent (or any other listener) with custom behaviour. It will fail the test if the wait is never fulfilled. One problem with this approach right now is that without a Sleep on robot side the control can move too quickly, and then even though Playwright is still waiting for the event our robot side library code closes the test.
This syntax without an explicit Run Async Keywords would probably be better at conserving the return values of the robot keywords, but asynchronity might cause variables to not be set at times when they are expected if you tried something like:
${result}  Wait For Request    xxx  xxxx
Click       #button
Log  ${result}

But I think we could go forward without a Run Async Keywords for now. Do you think that using the Wait For Request before you have clicked the button makes sense?

Actually after discussing with @mkorpela I think I agree that having a keyword that says Wait For not stop to wait is a bit weird.
How would you feel about syntax like this
${request}=  Request  xxxx  xxxxx
Click  #button
Wait For  ${request}

@xylix That is also another good approach for request. However, there is also another use case we might need this keyword I can think of.
Fill Text       #email        invalid_email@d
Click       #button
Wait For Elements State     #toast           visible       1s

Toast may just appear for a second and disappear so our test may pass sometimes but sometimes not

it's ok if we have some kind of thing like request you have mentioned

WHY
This issue introduces a problem of an event (request) that might happen so close to our action (click) that we might not have time to set our waiting observer after making the action.
There exists also another situation where we are expecting two events to happen and we do not control in which order these events will actually happen. Handling this in a naive procedural way will randomly fail as we are bound to wait first for one thing to happen and then another. Sometimes the events will occur in opposite order thus first wait will observe its event but the second will not as the event has already happened.
WHAT
Let's introduce the concept of promises to Robot Framework. A Promise is a proxy for a value not necessarily known when the promise is created.
HOW
Let's make it possible to execute Robot Framework Browser library keywords inside of a higher order promise wrapping keyword Promise to Wait.
${promise1}=  Promise to Wait  Get Title  ==  IMPORTANT  timeout=30s
${promise2}=  Promise to Wait  Get Request Response  /user/1  timeout=10s
Click  #funnyButton
Wait Until  ${promise1}  ${promise2}

That's a good solution. Thanks a lot @mkorpela

Yes. It should return the results or raise exceptions if a promised keyword fails.

@all-contributors Please add idxn for ideas

Future + thread pool as an implementation https://docs.python.org/3/library/concurrent.futures.html ?

[ ] Logging from threads should happen only at Wait for -keyword
[ ] Support for all keywords in Browser library (figure out argument conversion).

https://docs.python.org/3/library/threading.html#threading.get_ident

Related to https://github.com/robotframework/robotframework/issues/1791

We discussed with team that we are now first targeting few specific keywords for waiting on background.

Raising error if unwaited promises at the end of suite / test? Wait for all keyword.

@idxn Version 0.9.0 now contains promise keywords. Could you test if they work for you?

After merging #158 we now have Wait For Request and Wait For Response keywords. For example use see https://github.com/MarketSquare/robotframework-browser/blob/master/atest/test/03_Waiting/wait_for_http.robot

@mkorpela @xylix It really works for me. Thanks a bunch 👍

I'll just close this issue off then, let's create new issues for future needs related to this.


<EMAIL_ADDRESS>extract_dir error

Installation of mupdf recipe fails on Windows 10 machine.
> scoop install mupdf
Installing 'mupdf' (1.15.0-rc1) [64bit]
Loading mupdf-1.15.0-rc1-windows.zip from cache
Checking hash of mupdf-1.15.0-rc1-windows.zip ... ok.
Extracting mupdf-1.15.0-rc1-windows.zip ... ERROR Could not find 'mupdf-1.15.0-rc1-windows'! (error 16)

Please try again or create a new issue by using the following link and paste your console output:
https://github.com/lukesampson/scoop-extras/issues/new?title=mupdf%401.15.0-rc1%3a+extract_dir+error
True

Duplicate of #2072 issue.


Poultry recogniser

Make sure, you have done de following before merging this pull request:

[x] Your code is documented using doc comments.
[x] You added appropriate logging in your code.
[x] You added unit tests to all your implemented functions.
[x] You ran cargo format and cargo clippy to format your code and check for improvements.

Ich hab das jetzt mal in die Datenbank hinzugefügt. Jetzt muss das "nur" noch im frontend angepasst werden.

Some statistics on the poultry regex matches on the existing unknown meals:
We currently have 285 main dishes with unknown type (out of 1363 mains total).
178 of them match with the new regex.
Of the remaining 107, most are either wrongly missing a category or have some sort of fish that maybe does not count under the mensa's fish category.
We also have "Hausgeräucherte Entenbrust mit Mango - Chili - Chutney" which should also include according to Wikipedia[1].
Some more meals we could include:

BBQ Chicken Wings Vollkorn Naturreis in Paprikasoße
Chicken Drum Sticks
So we should also add chicken as a keyword
[1] https://en.wikipedia.org/wiki/Poultry

Note to myself @worldofjoni
When deploying this, the database needs to be migrated

Es fehlt glaube ich noch der Geflügel Typ im GraphQL Schema:
https://github.com/kronos-et-al/MensaApp/blob/f2ec05ac935773cb333cbfdcddfba97cff6433ec/backend/src/util.rs#L127

Es fehlt glaube ich noch der Geflügel Typ im GraphQL Schema:
No. You are referencing the wrong version.
https://github.com/kronos-et-al/MensaApp/blob/ab985dcaa59909fe2a847bd2f6d2c8e50aaaf917/backend/src/util.rs#L143C1-L143C13


Support keep_remotely variables

Is your request related to a new offering from AWS?
No.
Is your request related to a problem? Please describe.
(https://github.com/kreuzwerker/terraform-provider-docker/issues/358)
Allowing keep_remotely on docker_registry_image resolves to attempted delete.
Describe the solution you'd like.

Add keep_remotely support

Describe alternatives you've considered.
None. This should be supported anyways. (Default false to avoid Major change)

This issue has been resolved in version 2.35.1 :tada:


Offline fallback page create-react-app

I want to develop an apps that working on offline page with cache. I'm using workbox and react-app-rewired to generated custom service-workers for Create react app.. I already successful pre caching the static file and caching the dinamic content using Network first. But when i want to make the apps show offline page when i'm accesing page that haven't cached i get blank page. This is my custom-sw code. Any one can solved my problem ?? I have stuck with this problem
// importScripts('path/to/offline-google-analytics-import.js');
// workbox.googleAnalytics.initialize();
workbox.setConfig({ debug: true });
// See https://developers.google.com/web/tools/workbox/guides/configure-workbox
workbox.core.setLogLevel(workbox.core.LOG_LEVELS.debug);
self.addEventListener('install', event => event.waitUntil(self.skipWaiting()));
self.addEventListener('activate', event => event.waitUntil(self.clients.claim()));

// We need this in Webpack plugin (refer to swSrc option): https://developers.google.com/web/tools/workbox/modules/workbox-webpack-plugin#full_injectmanifest_config
workbox.precaching.precache(self.__precacheManifest);

workbox.precaching.precache([
  {
    url: '/offline.html',
    revision:'1234',
  }
]);

workbox.precaching.addRoute();

// app-shell cahcing
workbox.routing.registerRoute("/", workbox.strategies.cacheFirst());

//fungsinya buat apa ? Cari tau dulu
workbox.routing.registerNavigationRoute('/index.html');

var networkFirst = workbox.strategies.networkFirst({
  cacheName: 'outside-pages',
  plugins: [
     new workbox.cacheableResponse.Plugin({
       statuses: [0, 200],
     })
   ]
});

const customHandler = async (args) => {
  try {
    console.log(args.event.request.url)
    const response = await networkFirst.handle(args);
    return response || await caches.match(args.event.request.url);
  }catch (error) {
      return caches.open('/offline.html').then(function(cache) {
         return caches.match('/offline.html');
    })
  }
}

workbox.routing.registerRoute(
  new RegExp('http://<IP_ADDRESS>/api/(.*)'),
  customHandler,
);

workbox.routing.setCatchHandler(({url, event, params}) => {
  console.log(event)
});

And this is my custom offline page
`<!DOCTYPE html>

<title>Page Title</title>

Page cahced yet

This is a paragraph.

`

Hey @ferryS02
Did you manage to solve this?
Service workers only run on HTTPS (and localhost), so I don't think you'll be able to send a fetch request from your page to http://192... it has to be on HTTPS

This is one of the things that should be easier to do in the Workbox v4 beta release.
Please have a look athttps://developers.google.com/web/tools/workbox/guides/advanced-recipes#provide_a_fallback_response_to_a_route


update envoy binary

This version has support for routing to cloud foundry instances.
You can specify hostnames in cluster, with strict_dns option.. and then in the route block, use auto_host_rewrite. When auto_host_rewrite is set to true, envoy will automatically rewrite the host header with the name of the upstream host (in this case the cloud foundry app name).
+                    },
+                    {
+                      "timeout_ms": 0,
+                      "prefix": "/rewritebin",
+                     "prefix_rewrite" : "/",
+                     "auto_host_rewrite" : true,
+                      "cluster": "freebin"
                     }

                   ]
@@ -71,6 +78,17 @@
             "url": "tcp://service2:80"
           }
         ]
+      },
+      {
+        "name": "freebin",
+        "connect_timeout_ms": 250,
+        "type": "strict_dns",
+        "lb_type": "round_robin",
+        "hosts": [
+          {
+            "url": "tcp://mycfapp.mybluemix.net:80"
+          }
+        ]
       }

For example, in the above diff, the top half shows a route configuration. calls to localhost:6379/rewritebin get routed to mycfapp.mybluemix.net .. While doing so, envoy rewrites the host header from Host: localhost:6379 to Host: mycfapp.mybluemix.net .. Without the proper Host header, requests to CF apps won't work as they will be dropped on the floor by the GoRouter.
cc: @iwinoto @roder ^^^


update to support custom date format

Description
update to support custom date format
Screencast

Checklist

[x] I read the extension guidelines
[x] I read the documentation about publishing
[x] I ran npm run build and tested this distribution build in Raycast
[x] I checked that files in the assets folder are used by the extension itself
[x] I checked that assets used by the README are placed outside of the metadata folder

I'm still not 100% sure what the fields are for. If you look at this screenshot, shouldn't that work?

Can you tell me how I use this extension @randomradio


SERVFAIL problems with unbound 1.15.0 running on OpenBSD 7.1

Describe the bug
Since I upgraded my DNS servers to OpenBSD 7.1 with unbound 1.15.0, I have a lot of issues with DNS resolution (without changing anything in the config). I randomly get SERVFAIL for a lot of names, or something even stranger like some addresses and SERVFAIL for others (see dashlane example). The exact same config was working just fine with unbound 1.13.0 on OpenBSD 7.0. Note that it does not happen every time, it's quite sporadic.
Examples:
host dashlane.com
dashlane.com has address <IP_ADDRESS>
dashlane.com has address <IP_ADDRESS>
dashlane.com has address <IP_ADDRESS>
dashlane.com has address <IP_ADDRESS>
Host dashlane.com not found: 2(SERVFAIL)
Host dashlane.com not found: 2(SERVFAIL)
host forum.opnsense.org
Host forum.opnsense.org not found: 2(SERVFAIL)
To reproduce
Here is my partial config
server:
log-replies: yes
interface: <IP_ADDRESS>@853
tls-port: 853
tls-service-pem: *********
tls-service-key: *********
outgoing-range: 8192
outgoing-num-tcp: 256
incoming-num-tcp: 256
serve-expired: yes
outbound-msg-retry: 5
cache-max-negative-ttl: 1
msg-cache-size: 64m
msg-cache-slabs: 4
num-queries-per-thread: 32
rrset-cache-size: 128m
rrset-cache-slabs: 4
infra-cache-slabs: 4
key-cache-slabs: 4
access-control: <IP_ADDRESS>/0 allow
access-control: ::0/0 allow
hide-identity: yes
hide-version: yes
harden-short-bufsize: yes
harden-large-queries: yes
harden-glue: yes
harden-dnssec-stripped: yes
harden-below-nxdomain: yes
harden-referral-path: yes
use-caps-for-id: yes
qname-minimisation: yes
aggressive-nsec: yes
edns-tcp-keepalive: yes
so-reuseport: no
deny-any: yes
prefetch: yes
prefetch-key: yes
rrset-roundrobin: yes
minimal-responses: yes
Expected behavior
Names should resolve properly
System:

Unbound version: 1.15.0
OS: OpenBSD 7.1
unbound -V output: Version 1.15.0
Configure line: --enable-allsymbols --with-ssl=/usr --with-libevent=/usr --with-libexpat=/usr --without-pythonmodule --with-chroot-dir=/var/unbound --with-pidfile= --with-rootkey-file=/var/unbound/db/root.key --with-conf-file=/var/unbound/etc/unbound.conf --with-username=_unbound --disable-shared --disable-explicit-port-randomisation --without-pthreads
Linked libs: pluggable-libevent 1.4.15-stable (it uses kqueue), LibreSSL 3.5.2
Linked modules: dns64 respip validator iterator

Additional information
I tried removing every "hardening" parameter one by one to get nearer to the stock config. And it seems that disabling use-caps-for-id solves the issue.

For this query name I cannot reproduce issues with 0x20, the logs say the 0x20 replies are good. Looking at your config, I can make a guess, but it is pretty vague. Your num-queries-per-thread: 32 setting is very low. That should be adjusted to like 1024, or use the default. The setting could cause servfail, when the num queries are full, and another query comes in, the not fitting query can be rejected with SERVFAIL. How that could happen, you have prefetch, you query for the name, this gives you an answer, and starts the prefetch. Somehow with 0x20 enabled this takes longer than without 0x20, because some names have 0x20 fallback and then unbound has to query a number of hosts, like 5 queries or so, and that is more work. This makes the entry exist for a little longer. Then your second query comes in, but the list of queries is full now, with the previous one. And the new query is rejected. That would only happen when the list is full, so there is a bunch of lookups. And perhaps also in combination with prefetch and a name for which 0x20 backoff has to be performed.
If this happens, that should be shown in the logs. Maybe at higher verbosity settings. That makes the logs show what happens and why servfail is happening. You could try to fix the num queries per thread setting, or capture the event with verbosity in logs, or if logs show output now. There is also log-servfail: yes that prints summary information about servfail replies to log, that you could enable to get more information about these servfails.

I have been able to get the error, even without the num-queries-per-thread limit.
I have enabled the log-servfail, and I get this:
Apr 28 10:14:08 elendil unbound: [14984:0] error: SERVFAIL <f.root-servers.net. AAAA IN>: could not fetch nameservers for 0x20 fallback Apr 28 10:14:08 elendil unbound: [14984:0] info: <IP_ADDRESS> a.root-servers.net. A IN NOERROR 0.000000 1 52 Apr 28 10:14:09 elendil unbound: [14984:0] error: SERVFAIL <www.hubo.be. A IN>: could not fetch nameservers for 0x20 fallback Apr 28 10:14:09 elendil unbound: [14984:0] info: <IP_ADDRESS> www.hubo.be. A IN SERVFAIL 0.185887 0 29

It could be the combination of 0x20 fallback and the harden-referral-path: yes option you have. Both of those create new subrequests in the state machine, and those increase those counters that fail to add more subqueries once the limit is reached, and then the 0x20 stops because it cannot perform the fallback any more.

Also prefetch-key creates additional sub requests. Perhaps the max-negative-ttl of 1 second also causing refetched of like nameserver information, in particular non existing ipv6 AAAA information, that causes extra subqueries to be made.

The max-negative-ttl=1 was added afterwards to check if this would solve the issue, but it occured independently of that parameter.
Removing harden-referral-path also didn't change the behaviour.
I am trying to replicate with verbosity=4, but, as this is sporadic, it's not easy.

Another possibility is that it is out of memory. I would expect log entries from that. But I have seen OpenBSD set a heap size ulimit of 256m for processes, and then unbound, with this configuration could run out of memory. It has 128m and 64m cache sizes, and those are the sizes calculated by the data stored, not the overhead of the malloc allocator, and other stuff in memory, that could slowly fill up to the max and then out of memory could happen.

Here is a log of a failure with verbosity=4
unbound.log

Is your syslog omitting printout from the logs because of log volume? I also miss printout of lines like target fetch policy for level at startup. Perhaps a log-file directive would not omit the log lines.

Log volume on this machine is very low, so I would not expect syslog to drop the lines, but I can retry with a logfile: <file>

Could it be somehow related to https://github.com/NLnetLabs/unbound/commit/956d7d4e44f15dd53241e3e15c60d91af79743e8 ?

No I do not think that commit is related.
I think it may be because the query_for_targets function checks for depth before a 0-return path. I would like to see the complete logs, though. That no ip6 stops it, fits with it being the issue.

Here is a log (not with syslog) with both a success and a failure for www.hubo.be
unbound.log

The commit is an attempt to fix this problem. The logs from file are very complete, thank you. The fix attempts to check for target limits after it has determined that there is work to do. So that when ipv6 fails, and then it retries the 0x20 backoff procedure, the query target limits are not exceeded, because they have already been done, allowing the 0x20 backoff procedure to continue.

FYI, this issue has cropped-up in a pfSense community support when several installations were reporting of slow/broken DNS lookups since it's using unbound 1.15.0 as part of the base BSD install for the pfSense 22.05 release.  So for us it's an upstream component problem.  If you'd like to review the findings, logs, and triage process, here's the discussion thread link: https://forum.netgate.com/topic/173148/slow-dns-after-22-05

I also encountered the same symptoms: severely slowed down and broken DNS queries (running in a Docker container).
As I initially thought the issue was with the device on which it ran, I did a complete reinitialisation of the device so can't give more specifics I'm afraid...
Ever since reinitialising, I've not configured IPv6 functionality of the network and the problem has not reoccurred.


Fixed #25073 -- Added verbose_name to fields that were missing it

Addressed the concerns raised in https://github.com/django/django/pull/4957

buildbot, add to whitelist.

merged in f576b23a652eba69667ab3ce42aaf5f6b3e6f4ea, thanks!


Evaluate Impact of Dart 3

[ ] Pull from upstream to get the latest code
[ ] Checkout the feature branch
[ ] Compile and run the code
[ ] Develop the feature
[ ] Test locally on a device
[ ] Create Pull Request

Flutter+Dart upgrades blocked by #1931.
Let's revisit when this blocker has been resolved.


Some data are not labeled and they are not in the blacklist

For example:

thanks for your reminder. We found we uploaded wrong version blacklist. The range "00223-00232" was blacklisted due to water spraying rather than gas release, while "00690-00720" was blacklisted because the gas traces were too weak. We will upload a new blacklist today.


How to get free, useable remaining space?

Hi, I'm wondering how to get the remaining (estimated) free, useable space of the configured block count of littleFS.
I can get the currently used space by calling lfs_fs_traverse. But how can I get the amount of already worn out blocks? Is there a neat way to do it?

I don't believe that littleFS actually keeps track of warn out blocks, it just moves to a new block when it detects a write failure. I suspect there isn't a neat way of figuring out the number of bad blocks.
Also, lfs_fs_size returns the number of allocated blocks (basically does the lfs_fs_traverse calls for you). Might be a little easier to use ;-)

Mh, ok. I see.
Thanks for pointing out the lfs_fs_size.


types(v2): Convert sitemap plugin to TypeScript

Test - enabled the sitemap plugin in the v2 website and verified that
the sitemap is created after running docusaurus build.

Motivation
TypeScript offers strong typing.
Have you read the Contributing Guidelines on pull requests?
Yes
Test Plan
Updated existing tests and made sure they pass.
Enabled the sitemap plugin in the v2 website and verified that the sitemap is created after running docusaurus build.
Related PRs
No functional changes.

I think I need exclude the ./lib folder from linting somehow to make the CI happy.

Please update changelog-2.x.md


water in the underground village corridors

there is a possibility cause this to happen,the underground villages conflicted with the lake/river/ocean.when world trying to generated the lake/ocean/river and the underground village it conflicted
it's just annoying travelling the flowing water

Mod version:1.1.0
Fabric API version:0.29.3
Minecraft version:1.16.5+0.11.3 Fabric Loader

I am also having this problem, running BYG, Lakeside and River Redux.

I am uncertain how exactly to fix this but I have some ideas for the future.

Next update (V1.3) will finally solve this issue.


Update dependencies, add contributors.

This updates most dependencies to the latest versions and adds the main contributors.

Thanks for your review!


Main window does not show up

Installed from AUR.
Both kshare and kshare-git have the same issue.

Which window exactly please? And which WM/DE

"When you start the app (with no arguments, discussed later) a window is popped up, and a tray icon is made." Actually, no window showed up.
DE: KDE Plasma 5.10.4

And no tray icon, either

It did start. I can see it in process list.

One of people I know using KDE could not reproduce it. Try to run it with -v and post the output

Already tried -v, no output.

Can't reproduce 🤔

Same issue here using KDE Plasma on Arch Linux. Also no output running it in verbose mode. If I run it it stays open with 3 child processes, but nothing happens. No window or tray icon appears. Tried kshare and kshare-git.

By now I've tried on Gentoo, Fedora, Debian, Ubuntu, even Mint. They all worked. Other Arch users I know that use KDE also do not have this issue. Only distro I had KDE on was Fedora though, but DE's should be irrelevant.

Yes, they should be. But it's odd that it's not running on @ohmyarch or on mine and we're both using KDE.
Is there anything I can do to provide helpful information for solving the problem?

Nothing I can think of.

Ok, it's depending on the widget style used in KDE. By default there are Breeze, Fusion, MS Windows 9x and Oxygen. I also have kvantum there because the theme I'm using (Arc-Kde) looks better with kvantummanager. And apparently kshare works with Breeze and Oxygen, but not with Fusion, MS Windows 9x or kvantum.

I used Arc-KDE too for a while in development of it.

If you currently have a working KDE setup you can go to System Setting -> Application Style -> Widget Style and change it to Fusion or MS Windows 9x. With that you should be able to reproduce the issue.

I could not reproduce it.

Ahh, ok. It's also depending on the icon theme...
It's not working with Papirus and Fusion, MS Windows 9x or kvantum. With the Breeze icon theme it's also working for me with MS Windows 9x and Fusion.
If you have time you can install the Papirus Icon Theme and use it together with Fusion or MS Windows 9x as Widget Style. Then you are definitely able to reproduce it. But I think that this is not an issue of your tool, I think there's something wrong with the icon theme.

Forgot to mention due to hurrying to school: I was not able to reproduce it this way either.

@ohmyarch what themes are you using? maybe we can find out what we have in common this way and conclude what the problem could be.

@jkhsjdhjs
Theme: Arc KDE
Icons: Papirus

Does it work for you when you switch to the Breeze Icon Theme @ohmyarch?

@jkhsjdhjs Yes, it works.

So this is specific to the icon theme? Your icon theme may be trying to overwrite the applications icon, failing, and giving up without reverting to the applications icon. Though, that doesn’t necessarily explain why the window fails to open.
Best guess I can make is missing dependencies. Even then, though, doing -v should spit something out.

yea it'd not even start

Closing as there has been no new updates for a while.


Fix lossly cast in value_to_usize

Checklist

[x] I have read the Contributor Guide
[x] I have read and agree to the Code of Conduct
[x] I have added a description of my changes and why I'd like them included in the section below

Description of Changes
Turns out going key_path::get(1) infers 1 to be an i32, since get takes an impl Into<Value>. So we somehow need to cast from at least i32 to usize for accessing lists and arrays with key paths.
To remove the lossly conversion I can think of two options:

Use TryFrom and panic if the conversion fails. This would only happen when accessing a very large list.
key_path::get is also used for accessing maps (hence why it takes impl Into<Value>) but we could make separate functions for list/array and map access. Perhaps get_seq(usize) and get_map(impl Into<Value>).

I personally prefer option one because I like how list/array and map access is the same, but it is perhaps a bit disingenuous. I like how it matches rusts' Index trait though. What do you think @bnjbvr?
Related Issues
Fixes https://github.com/EmbarkStudios/mirror-mirror/issues/76

(Sorry I didn't see the question at first.)
From options 1 and 2, option 2 looks slightly nicer to me, for the reason that it doesn't panic.
Returning None as I suggested could also work nicely, in my opinion: if you pass an index type that's bigger than a usize, you couldn't feed it to Vec::get anyways, so it conceptually returns nothing, hence None. The panic also conceptually works, but that morphs what would be a compile-time error (indexing with a larger-than-usize) into a runtime-error, and that would still be hit easily with fuzzers.


Retry with require-corp if credentialless fails

This PR would close #20, #23, and #24 as a bonus.
The idea is that, we will know the browser doesn't support COEP-credentialless if credentialless fails to establish COI.
So, we can set coi.coepDegrade to try again with require-corp.
I have tested it with the latest Firefox, Chrome, and Safari.
What I did with safari on browserstack are:

Open index.html.
Reload it manually.
Type 'navigator.serviceworker.controller.postMessage({type: "deregister"})' in devtools console.
Open it in another new tab.

Let me know if I should test it in more situations.

Fantastic! Thank you :)
I think the only situation in which this may break is in private windows - but I'm not sure about SessionStorage. I know that LocalStorage will throw in certain situations in incognito mode, I don't see why that would be the same for SessionStorage though.
If that does become an issue, we could use a URL query parameter instead - I'm happy to merge this how it is though

If we wanted to be careful, we could put it in a try catch, but we can also do that in a later release if we do find out it's an issue in some browsers

Would love to have this feature in the latest NPM release. Still hanging in there with the hack I gave in #24 which seems to work well enough. Assuming there are other goals that need to be met before a new version is released, if that is the case I would consider contributing towards them