Failed to create index/resource already exists creating spans when using ElasticSearch

While running a simple performance test and using ElasticSearch as storage I a number of "failed to create index" errors on the collector
I did the following:
-- Download elasticsearch 6.1.2 from https://www.elastic.co/downloads/elasticsearch and ran ./bin/elasticsearch
-- Ran the collector with: SPAN_STORAGE_TYPE=elasticsearch go run ./cmd/collector/main.go --collector.queue-size=3000000 --es.bulk.size=10000000 --es.bulk.workers=10 --es.bulk.flush-interval=1s
-- git clone git@github.com:Hawkular-QE/jaeger-standalone-performance-tests.git
-- DELAY=10 THREAD_COUNT=10 ELASTICSEARCH_HOST=localhost SPAN_STORAGE_TYPE=elasticsearch USE_AGENT_OR_COLLECTOR=collector DURATION_IN_MINUTES=1 mvn clean -Dtest=SimpleTest#createTracesTest test
In one particular case the test failed as it created 59040 spans but only found 58943 in elasticsearch.  For each of the missing spans the collector produced a pair of the following error messages.
{"level":"error","ts":1516890288.1260505,"caller":"spanstore/writer.go:190","msg":"Failed to create index","trace_id":"b36e3fa8b97bb7c6","span_id":"b36e3fa8b97bb7c6","error":"elastic: Error 400 (Bad Request): index [jaeger-span-2018-01-25/xcVS2i4VQh6fZ8LsytrXgw] already exists [type=resource_already_exists_exception]","stacktrace":"github.com/jaegertracing/jaeger/plugin/storage/es/spanstore.(*SpanWriter).logError\n\t/home/kearls/go/src/github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/writer.go:190\ngithub.com/jaegertracing/jaeger/plugin/storage/es/spanstore.(*SpanWriter).createIndex\n\t/home/kearls/go/src/github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/writer.go:152\ngithub.com/jaegertracing/jaeger/plugin/storage/es/spanstore.(*SpanWriter).WriteSpan\n\t/home/kearls/go/src/github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/writer.go:123\ngithub.com/jaegertracing/jaeger/cmd/collector/app.(*spanProcessor).saveSpan\n\t/home/kearls/go/src/github.com/jaegertracing/jaeger/cmd/collector/app/span_processor.go:102\ngithub.com/jaegertracing/jaeger/cmd/collector/app.(*spanProcessor).(github.com/jaegertracing/jaeger/cmd/collector/app.saveSpan)-fm\n\t/home/kearls/go/src/github.com/jaegertracing/jaeger/cmd/collector/app/span_processor.go:89\ngithub.com/jaegertracing/jaeger/cmd/collector/app.ChainedProcessSpan.func1\n\t/home/kearls/go/src/github.com/jaegertracing/jaeger/cmd/collector/app/model_consumer.go:32\ngithub.com/jaegertracing/jaeger/cmd/collector/app.(*spanProcessor).processItemFromQueue\n\t/home/kearls/go/src/github.com/jaegertracing/jaeger/cmd/collector/app/span_processor.go:126\ngithub.com/jaegertracing/jaeger/cmd/collector/app.NewSpanProcessor.func1\n\t/home/kearls/go/src/github.com/jaegertracing/jaeger/cmd/collector/app/span_processor.go:57\ngithub.com/jaegertracing/jaeger/pkg/queue.(*BoundedQueue).StartConsumers.func1\n\t/home/kearls/go/src/github.com/jaegertracing/jaeger/pkg/queue/bounded_queue.go:65"} {"level":"error","ts":1516890288.1261024,"caller":"app/span_processor.go:103","msg":"Failed to save span","error":"Failed to create index: elastic: Error 400 (Bad Request): index [jaeger-span-2018-01-25/xcVS2i4VQh6fZ8LsytrXgw] already exists [type=resource_already_exists_exception]","errorVerbose":"elastic: Error 400 (Bad Request): index [jaeger-span-2018-01-25/xcVS2i4VQh6fZ8LsytrXgw] already exists [type=resource_already_exists_exception]\nFailed to create index\ngithub.com/jaegertracing/jaeger/plugin/storage/es/spanstore.(*SpanWriter).logError\n\t/home/kearls/go/src/github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/writer.go:191\ngithub.com/jaegertracing/jaeger/plugin/storage/es/spanstore.(*SpanWriter).createIndex\n\t/home/kearls/go/src/github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/writer.go:152\ngithub.com/jaegertracing/jaeger/plugin/storage/es/spanstore.(*SpanWriter).WriteSpan\n\t/home/kearls/go/src/github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/writer.go:123\ngithub.com/jaegertracing/jaeger/cmd/collector/app.(*spanProcessor).saveSpan\n\t/home/kearls/go/src/github.com/jaegertracing/jaeger/cmd/collector/app/span_processor.go:102\ngithub.com/jaegertracing/jaeger/cmd/collector/app.(*spanProcessor).(github.com/jaegertracing/jaeger/cmd/collector/app.saveSpan)-fm\n\t/home/kearls/go/src/github.com/jaegertracing/jaeger/cmd/collector/app/span_processor.go:89\ngithub.com/jaegertracing/jaeger/cmd/collector/app.ChainedProcessSpan.func1\n\t/home/kearls/go/src/github.com/jaegertracing/jaeger/cmd/collector/app/model_consumer.go:32\ngithub.com/jaegertracing/jaeger/cmd/collector/app.(*spanProcessor).processItemFromQueue\n\t/home/kearls/go/src/github.com/jaegertracing/jaeger/cmd/collector/app/span_processor.go:126\ngithub.com/jaegertracing/jaeger/cmd/collector/app.NewSpanProcessor.func1\n\t/home/kearls/go/src/github.com/jaegertracing/jaeger/cmd/collector/app/span_processor.go:57\ngithub.com/jaegertracing/jaeger/pkg/queue.(*BoundedQueue).StartConsumers.func1\n\t/home/kearls/go/src/github.com/jaegertracing/jaeger/pkg/queue/bounded_queue.go:65\nruntime.goexit\n\t/usr/lib/go-1.8/src/runtime/asm_amd64.s:2197","stacktrace":"github.com/jaegertracing/jaeger/cmd/collector/app.(*spanProcessor).saveSpan\n\t/home/kearls/go/src/github.com/jaegertracing/jaeger/cmd/collector/app/span_processor.go:103\ngithub.com/jaegertracing/jaeger/cmd/collector/app.(*spanProcessor).(github.com/jaegertracing/jaeger/cmd/collector/app.saveSpan)-fm\n\t/home/kearls/go/src/github.com/jaegertracing/jaeger/cmd/collector/app/span_processor.go:89\ngithub.com/jaegertracing/jaeger/cmd/collector/app.ChainedProcessSpan.func1\n\t/home/kearls/go/src/github.com/jaegertracing/jaeger/cmd/collector/app/model_consumer.go:32\ngithub.com/jaegertracing/jaeger/cmd/collector/app.(*spanProcessor).processItemFromQueue\n\t/home/kearls/go/src/github.com/jaegertracing/jaeger/cmd/collector/app/span_processor.go:126\ngithub.com/jaegertracing/jaeger/cmd/collector/app.NewSpanProcessor.func1\n\t/home/kearls/go/src/github.com/jaegertracing/jaeger/cmd/collector/app/span_processor.go:57\ngithub.com/jaegertracing/jaeger/pkg/queue.(*BoundedQueue).StartConsumers.func1\n\t/home/kearls/go/src/github.com/jaegertracing/jaeger/pkg/queue/bounded_queue.go:65"}

I'm reopening, as we got another report for this problem under ES 6.x. Either we should change the docs to mention that we do not support 6.x, or we should fix the underlying issue :-)

Perhaps it's then a good idea to not remove, but say that "there are issues reported against 6.x" and link to here?

I would not encourage users to use 6.x if our components do not fully support it.
The version of ES client we are pulling is compatible with ES 5.x https://github.com/olivere/elastic#releases

For ES 6 support there is a separate issue https://github.com/jaegertracing/jaeger/issues/637

I got same issue with @kevinearls on my jaeger-collector.
Errors like below.
{"level":"info","ts":1530861463.72645,"caller":"healthcheck/handler.go:99","msg":"Health Check server started","http-port":14269,"status":"unavailable"}
{"level":"info","ts":1530861463.7328868,"caller":"static/strategy_store.go:77","msg":"No sampling strategies provided, using defaults"}
{"level":"info","ts":1530861463.7332094,"caller":"collector/main.go:141","msg":"Registering metrics handler with HTTP server","route":"/metrics"}
{"level":"info","ts":1530861463.7334013,"caller":"collector/main.go:149","msg":"Starting Jaeger Collector HTTP server","http-port":14268}
{"level":"info","ts":1530861463.7334604,"caller":"healthcheck/handler.go:133","msg":"Health Check state change","status":"ready"}
{"level":"error","ts":1530861490.2414696,"caller":"spanstore/writer.go:187","msg":"Failed to create index","trace_id":"72753077633cc236","span_id":"37d42b0700280967","error":"elastic: Error 400 (Bad Request): index [jaeger-service-2018-07-06/Z5j32laiSg-ptFcPIo_Ttg] already exists [type=resource_already_exists_exception]","stacktrace":"github.com/jaegertracing/jaeger/plugin/storage/es/spanstore.(*SpanWriter).logError\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/writer.go:187\ngithub.com/jaegertracing/jaeger/plugin/storage/es/spanstore.(*SpanWriter).createIndex\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/writer.go:149\ngithub.com/jaegertracing/jaeger/plugin/storage/es/spanstore.(*SpanWriter).WriteSpan\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/writer.go:116\ngithub.com/jaegertracing/jaeger/cmd/collector/app.(*spanProcessor).saveSpan\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/app/span_processor.go:101\ngithub.com/jaegertracing/jaeger/cmd/collector/app.(*spanProcessor).(github.com/jaegertracing/jaeger/cmd/collector/app.saveSpan)-fm\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/app/span_processor.go:88\ngithub.com/jaegertracing/jaeger/cmd/collector/app.ChainedProcessSpan.func1\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/app/model_consumer.go:32\ngithub.com/jaegertracing/jaeger/cmd/collector/app.(*spanProcessor).processItemFromQueue\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/app/span_processor.go:126\ngithub.com/jaegertracing/jaeger/cmd/collector/app.NewSpanProcessor.func1\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/app/span_processor.go:56\ngithub.com/jaegertracing/jaeger/pkg/queue.(*BoundedQueue).StartConsumers.func1\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/pkg/queue/bounded_queue.go:65"}
{"level":"error","ts":1530861490.241552,"caller":"app/span_processor.go:102","msg":"Failed to save span","error":"Failed to create index: elastic: Error 400 (Bad Request): index [jaeger-service-2018-07-06/Z5j32laiSg-ptFcPIo_Ttg] already exists [type=resource_already_exists_exception]","errorVerbose":"elastic: Error 400 (Bad Request): index [jaeger-service-2018-07-06/Z5j32laiSg-ptFcPIo_Ttg] already exists [type=resource_already_exists_exception]\nFailed to create index\ngithub.com/jaegertracing/jaeger/plugin/storage/es/spanstore.(*SpanWriter).logError\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/writer.go:188\ngithub.com/jaegertracing/jaeger/plugin/storage/es/spanstore.(*SpanWriter).createIndex\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/writer.go:149\ngithub.com/jaegertracing/jaeger/plugin/storage/es/spanstore.(*SpanWriter).WriteSpan\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/writer.go:116\ngithub.com/jaegertracing/jaeger/cmd/collector/app.(*spanProcessor).saveSpan\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/app/span_processor.go:101\ngithub.com/jaegertracing/jaeger/cmd/collector/app.(*spanProcessor).(github.com/jaegertracing/jaeger/cmd/collector/app.saveSpan)-fm\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/app/span_processor.go:88\ngithub.com/jaegertracing/jaeger/cmd/collector/app.ChainedProcessSpan.func1\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/app/model_consumer.go:32\ngithub.com/jaegertracing/jaeger/cmd/collector/app.(*spanProcessor).processItemFromQueue\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/app/span_processor.go:126\ngithub.com/jaegertracing/jaeger/cmd/collector/app.NewSpanProcessor.func1\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/app/span_processor.go:56\ngithub.com/jaegertracing/jaeger/pkg/queue.(*BoundedQueue).StartConsumers.func1\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/pkg/queue/bounded_queue.go:65\nruntime.goexit\n\t/home/travis/.gimme/versions/go1.10.linux.amd64/src/runtime/asm_amd64.s:2361","stacktrace":"github.com/jaegertracing/jaeger/cmd/collector/app.(*spanProcessor).saveSpan\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/app/span_processor.go:102\ngithub.com/jaegertracing/jaeger/cmd/collector/app.(*spanProcessor).(github.com/jaegertracing/jaeger/cmd/collector/app.saveSpan)-fm\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/app/span_processor.go:88\ngithub.com/jaegertracing/jaeger/cmd/collector/app.ChainedProcessSpan.func1\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/app/model_consumer.go:32\ngithub.com/jaegertracing/jaeger/cmd/collector/app.(*spanProcessor).processItemFromQueue\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/app/span_processor.go:126\ngithub.com/jaegertracing/jaeger/cmd/collector/app.NewSpanProcessor.func1\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/app/span_processor.go:56\ngithub.com/jaegertracing/jaeger/pkg/queue.(*BoundedQueue).StartConsumers.func1\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/pkg/queue/bounded_queue.go:65"}
Some spans are stored and some other are not. Is is caused partially.
My ES version is 6.1.1 and clustered with three nodes.
If you need additional information about this issue, please comment.

@misojiwon3 thanks for the report! The ES 6.x is not fully supported at the moment. For more info please see https://github.com/jaegertracing/jaeger/issues/637.

@pavolloffay not sure I follow this - how is "index already exists" issue specific to ES 6.x?  Is this a race condition among the collectors?

@yurishkuro this happens only with ES 6.x. Maybe it could be somehow fixed on our side but it seems that ES client version we are pulling is tested and guaranteed to work with es 5.x

The issue is that ES 6.x changed REST API. Fix is in https://github.com/jaegertracing/jaeger/pull/921.
The problem would not have been resolved even if we updated ES client to 6.x version as we compare strings in our impl.


boards: arm: add OTG_FS on nucleo_u575zi_q

add OTG_FS on nucleo_u575zi_q

@erwango Why does this need to go into 3.6? It looks like a feature addition, not a bug fix.

Oops, indeed, this is for V3.7.0, sorry for the confusion... Thanks @MaureenHelm (Things are moving so fast in Zephyr .... ;D)

@marwaiehm-st Can you take the ownership of this PR ?

Replaced by #72357


Move to netlify

@caniszczyk I got it running on Netlify with no configuration changes: https://rook-io.netlify.com.
What's required at this point:

Transfer of DNS control to Netlify
Renaming of website repo (to disable GitHub Pages build)
Enabling of Netlify access to repo by Rook admins

can you work with LF IT to get the NS records updated

dns1.p03.nsone.net
dns2.p03.nsone.net
dns3.p03.nsone.net
dns4.p03.nsone.net

we need @rook/owners to do that

already added @jbw976 @travisn

Does Netlify enable full DNS management? We need A records for Jenkins and others. /cc @Defilan

@bassam yes

https://app.netlify.com/teams/rook/dns/setup/rook.io/nameservers
LF IT just needs to update the ns records and the rook maintainers will manage DNS moving forward

Hey @caniszczyk , this sounds great. Thanks for doing this. Rook's current DNS management is through AWS and we are actually in the middle of migrating that to another AWS account, so the timing is perfect. I'm all for the movement to netlify.
As @bassam mentioned, we have records for our Jenkins environment, as well as a handful of other things (slack, etc). Correct me if I'm wrong, but it looks like we will have full "self-service" use, right? As we are also in the middle of migrated Jenkins instances and I'd like to be able to quickly switch the records to the new servers when we do our scheduled migration. How does this sound @travisn ?

yes it would be self service via Netlify
On Tue, Jun 11, 2019 at 11:13 AM Christopher Maher<EMAIL_ADDRESS>wrote:

Hey @caniszczyk https://github.com/caniszczyk , this sounds great.
Thanks for doing this. Rook's current DNS management is through AWS and we
are actually in the middle of migrating that to another AWS account, so the
timing is perfect. I'm all for the movement to netlify.
As @bassam https://github.com/bassam mentioned, we have records for our
Jenkins environment, as well as a handful of other things (slack, etc).
Correct me if I'm wrong, but it looks like we will have full "self-service"
use, right? As we are also in the middle of migrated Jenkins instances and
I'd like to be able to quickly switch the records to the new servers when
we do our scheduled migration. How does this sound @travisn
https://github.com/travisn ?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rook/rook.github.io/issues/60?email_source=notifications&email_token=AAAPSII7YNYJHDDOO24ZDHTPZ7FKHA5CNFSM4HWYG262YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXNVN3A#issuecomment-500913900,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAPSIKSBHG5QVLMBKRBJDDPZ7FKHANCNFSM4HWYG26Q
.

--
Cheers,
Chris Aniszczyk
http://aniszczyk.org
+1<PHONE_NUMBER>

Awesome, thanks @caniszczyk .

The migration is now complete. @travisn do you mind closing the issue out? Thanks!

Done!

@jbw976 Any thoughts on this?

Thank you for both your explanation and your patience @lucperkins! That deploy preview feature for PRs actually does look pretty useful and can help us test for regressions as part of each PR.  I definitely like that.
Another thing that I like about GitHub Pages is the automated security vulnerability alerts and PRs to fix them too.  Does Netlify also have similar functionality? I couldn't find it from a quick search and reading pages such as https://www.netlify.com/security/.
I really appreciate your efforts at helping us make the right move for the static rook.io site.  If we decide to go ahead and make the transition fully to netlify (which seems like we should, as long as the security update story is OK), then I'll be happy to help with access or config updates that are needed.  Thanks a lot @lucperkins! 🙇

@jbw976 As far as I know, those checks are performed regardless of whether you use GitHub Pages, though I could be wrong (my searches have been inconclusive thus far).


release: v1.0.0a14

Signed-off-by: Lars Holm Nielsen<EMAIL_ADDRESS>

Coverage remained the same at 98.039% when pulling 114f6ff4c6dbbe2c12e691026d02c86728f0f164 on lnielsen:rel-v1.0.0a14 into 8c7a94a919924ecd68c46ee64361231361fc7a24 on inveniosoftware:master.


Icon for website not working (Favicon)

im trying to add an icon on my website, but it seems it doesn't work, the icon not shows up... here is my code in theme.config.tsx inside const config: DocsThemeConfig = {}, inside head: () => {}:
<link href="/favicon.ico" rel="shortcut icon" />
<link rel="mask-icon" href="/safari-pinned-tab.svg" />
all the images are in public folder -> e.g  /public/favicon.ico
am i doing it correctly? or is there something that i need to change?
I really appreciate your help :)

same problem here

Nextra head can't resolve nested <></>

I'm still struggling with setting the favicon, any help would be appreicated.

@nabilfatih why was this issue closed, did you figure it out? I'm having the same issue


See if API change breaks Node bindings: Do Not Merge

Small change in libosrm's API: lifting constnes constraints. Let's have Travis build it. Do not merge.
See https://github.com/Project-OSRM/osrm-backend/pull/2862#issuecomment-245275942

All done here - closing.


Add markup to time display

Hi, currently the output is all in one tag.
Is it possible to have the time wrapped in a span tag so we can put the number and the verbiage on different lines?

Hi, no, this is not currently possible. If you have a proposal for implementation, feel free to reopen this issue, discuss it and perhaps follow up with a Pull Request. thanks!


Smoketest: Re-derive rewrites used in Intel Paper

A "simple" first smoketest would be to see how many rules we can derive from this paper.
The steps would be:

[x] #12
[x] 2. #4
[ ] 3. #15
[ ] 4. #25
[ ] 5. make lots of money

Not going to work on this for now. The domain is too hard!
More concretely -- just validating a rule (on the Intel side) took on the order of 10 hours. This implies that the domain is too expensive for the basic stuff we're trying to do, at least for now.


rmsdmatrix setting in mdref-glycans-sf400x16.toml confirmation.

I found that 1C1L contains a disaccharide but in its mdref-glycans-sf400x16.toml>[rmsdmatrix]>resdic_B = [1,2,3,4,5,6,7].
And all mdref-glycans-sf400x16.tomls ' resdic_B is same.
I realize that in the benchmark all the glycans' length is 2~7.
So if resdic_* has  redundant residues indice,will Haddock3 ignore those automaticly?This behavior is handy but not shown in it's document.I need a confirmation.
Also,unbound_ens_vdw_tip-ap_clust.toml use rmsdmatrix module while there is an ilrmsdmatrix module.In the report.html,i-RMSD/L-RMSD/iL-RMSD are calculated all.Will it be different if I use rmsdmatirx or ilrmsdmatrix?

Hi @alchemistcai , good questions! Indeed rmsdmatrix ignores the non-existing residues in the selection. Knowing that the longest glycan in our benchmark is 7 monosaccharides long we used the same list everywhere. Indeed it makes sense to add this to the documentation, thanks!
as for the ilrmsdmatrix module, can you please tell me where you see it?
the good advantage of the ilrmsdmatrix module is that you don't have to specify any residue set: the interface receptor residues will be selected automatically among those that are at the interface across all the models. Of course, if you know more or less where the protein interface will be, the rmsdmatrix will allow you to be more precise by specifying only those residues for the RMSD calculation. If the docking is driven by good information, the results should not be dramatically different.
In the report.html you see all the RMSD quantities indeed: these correspond to the distance from the reference structure (if provided) or from the best scoring model (if no reference is provided).

Thanks for your answer!
When I search rmsdmatrix in the official document,ilrmsdmatrix is just under rmsdmatrix,they are under same directory.Also in haddock3's examples,I found some cfg files use ilrmsdmatrix instead of rmsdmatrix.
So in summary,ilrmsdmatrix is automatic and handy for benchmark tests,while rmsdmatrix needs manual configuration but is more suitable for unknown complex docking.


when i execute the last step fuzz, i found it will go to find the template new_fake_{device_name}.txt or new_qemu_{device_name}.txt.but these files not exist in my project. where these files generated?

The templates for syzkaller are generated in the previous Setup phase, specifically in step 5: build syzkaller, with the command
python3 /path/to/PrIntFuzz/scripts/python/setup.py --build_syz
This step will parse the JSON file generated by the analyzer, generate the corresponding syzkaller templates, and then copy the generated templates to the syzkaller directory, please make sure this step is executed successfully.

i am sure i have execute "python3 /path/to/PrIntFuzz/scripts/python/setup.py --build_syz". In my project, there exists the dir "/path/to/PrIntFuzz/third_party/syzkaller/sys/linux", and i check check the files in this directory, there are many txt files, but there is indeed no txt file starting with "new_fake_" or "new_qemu_".

Sorry for the late reply, have you solved the problem yet?


Unable to clone the repository

When trying to clone this repo, something goes wrong.
git clone git@github.com:rodydavis/vscode-router-generator.git
or
gh repo clone rodydavis/vscode-router-generator
PS C:\Users\Taha\StudioProjects> git clone git@github.com:rodydavis/vscode-router-generator.git
Cloning into 'vscode-router-generator'...
remote: Enumerating objects: 9567, done.
remote: Counting objects: 100% (9567/9567), done.
remote: Compressing objects: 100% (7066/7066), done.
remote: Total 9567 (delta 2082), reused 9540 (delta 2063), pack-reused 0 eceiving objects: 100% (9567/9567), 20.43 MiB |Receiving objects: 100% (9567/9567), 21.85 MiB | 1.46 MiB/s, done.

Resolving deltas: 100% (2082/2082), done.
error: invalid path 'examples/flutter/lib/pages/about/:id.dart'
fatal: unable to checkout working tree
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry with 'git restore --source=HEAD :/'

When I execute git status

git status
PS C:\Users\Taha\StudioProjects\vscode-router-generator> git status
On branch main
Your branch is up to date with 'origin/main'.

Changes to be committed:
  (use "git restore --staged <file>..."
Got the issue happen with GitHub desktop?

@rodydavis
This seems to be only happening on Windows so far, I tried cloning on macOS, there was no issue

Got the issue happen with GitHub desktop?

I don't have the GitHub desktop client, I use Git Fork

Windows must not allow for : in the path name!

Since the issue seems to be with Windows not this project, i will close it for now


Fix issue with validators when each is true

Several of the DTO validators are throwing errors for parameters where multiple values are accepted. The required change has already been made in the 11.x branch of easey-common (see https://github.com/US-EPA-CAMD/easey-common/pull/161). This should be propagated to the master branch, and ECMPS 2.0 repositories should be updated to use the latest version.
Acceptance Tests
Endpoints that accept multiple values for a parameter with a decorator that uses the DbLookupValidator under the hood should return a success response. Some endpoints to test include:

https://api.epa.gov/easey/dev/emissions-mgmt/emissions/apportioned/hourly/by-state?stateCode=AL&beginDate=2023-01-01&endDate=2023-12-31&page=1&perPage=1
https://api.epa.gov/easey/dev/emissions-mgmt/emissions/apportioned/quarterly/nationally?controlTechnologies=Baghouse&year=2023&quarter=1&page=1&perPage=1
https://api.epa.gov/easey/dev/emissions-mgmt/emissions/apportioned/quarterly/nationally?programCodeInfo=ARP&year=2023&quarter=1&page=1&perPage=1
https://api.epa.gov/easey/dev/streaming-services/allowance-holdings?accountType=Compliance Assurance Surrender Account

Verified on dev swagger page using endpoints listed in acceptance tests. Received success responses when selecting multiple values for parameters stateCode, controlTechnologies, unitType, unitFuelType, programCodeInfo, and accountType

Verified in test swagger page (200 HTTP response) for the following endpoints:

https://api.epa.gov/easey/dev/emissions-mgmt/emissions/apportioned/hourly/by-state?stateCode=AL&beginDate=2023-01-01&endDate=2023-12-31&page=1&perPage=1
https://api.epa.gov/easey/dev/emissions-mgmt/emissions/apportioned/quarterly/nationally?controlTechnologies=Baghouse&year=2023&quarter=1&page=1&perPage=1
https://api.epa.gov/easey/dev/emissions-mgmt/emissions/apportioned/quarterly/nationally?programCodeInfo=ARP&year=2023&quarter=1&page=1&perPage=1
https://api.epa.gov/easey/dev/streaming-services/allowance-holdings?accountType=Compliance Assurance Surrender Account


Fix (hopefully) USB mass storage on Windows and config menu tweaks

I used Wireshark with an USB packet sniffer to look at what happened.
Windows 11 was going bonkers after sending READ_FORMAT_CAPACITY and I returned the full card size when it was in the ejected state.
It now works in my Windows 11 VM.
Still, not very user friendly behavior from Windows when a badly behaving device is attached.
Also changed PREVENT_ALLOW  to fail on all prevent requests, so that Windows doesn't succeed in
preventing media removal and starts to cache any transfers.

After looking at the pico bootrom USB disk code, I  added VERIFY and SYNCHRONIZE_CACHE SCSI commands.

Building on MacOS worked fine, but on Ubuntu WSL2 the compiler reports an error:
[ 84%] Building C object CMakeFiles/picosim.dir/stdio_msc_usb/msc_usb.c.obj
/home/udo/work/RP2040-GEEK-80/stdio_msc_usb/msc_usb.c: In function 'tud_msc_scsi_cb':
/home/udo/work/RP2040-GEEK-80/stdio_msc_usb/msc_usb.c:189:3: error: a label can only be part of a statement and a declaration is not a statement
189 |   scsi_prevent_allow_medium_removal_t const *prevent_allow =
|   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
make[2]: *** [CMakeFiles/picosim.dir/build.make:1003: CMakeFiles/picosim.dir/stdio_msc_usb/msc_usb.c.obj] Error 1

fixed with #ba5d430

was just about to commit

and it works on Windows, no issues so far

cool, now you don't need to remove the SD-card anymore.

I still don't understand why you want to use USB VID/PID from pid.codes instead of Raspberry Pi registered ones. For a test I changed it to the official ones:
    #USBD_VID=0x1209
    USBD_VID=0x2E8A
    #USBD_PID=0x0001
    USBD_PID=0x000A

Tested on Windows and works OK, also the USB drive shows as Z80pack Mass Storage USB Device.
Then Raspberry Pi offers to register unique ID's for such projects:
https://github.com/raspberrypi/usb-pid
Maybe we should try to apply for a unique PID?

pico-sdk/lib/tinyusb/examples/device/cdc_msc/src/usb_descriptors.c states that every combination of interfaces must have a unique product id, so my addition of an mass storage interface would require a new PID.
I was than googling around and found pid.codes. I never did stumble upon https://github.com/raspberrypi/usb-pid .
Windows seems to be the only OS where this matters, and if you say it works, than I don't mind changing it back to the standard VID/PID.

Probably want need a unique PID, since mass storage uses a standard interface driver.

I'll fill out the form and request a unique PID from Raspberry, maybe they will give us one.
Until then let's use the one it was using before, the repository is cloned a lot and with the experimental ID we might cause problems on other peoples Windows systems. But leave it in CMakeLists.txt, so that we can put the official PID in there, if we should get one.

We can just add one to CMakeLists.txt if we get one, it overwrites the one in stdio_msc_usb/stdio_msc_usb_descriptors.c


Testing the Authorization example app - issue when seeding the DB

Issue
As described here:
https://docs.microsoft.com/en-us/aspnet/core/security/authorization/secure-data?view=aspnetcore-5.0#the-starter-and-completed-app
I downloaded the repository zip file:
https://codeload.github.com/dotnet/AspNetCore.Docs/zip/master
I used the project in the following directory:
AspNetCore.Docs-master\aspnetcore\security\authorization\secure-data\samples\final3

Then, following the steps here:
https://docs.microsoft.com/en-us/aspnet/core/security/authorization/secure-data?view=aspnetcore-5.0#test-the-completed-app
here's the output I get after setting the user secrets and running the app:
PS C:\Users\dharm\Dropbox\Documents\VisualStudio\AspNetCoreDocsAuthorization\final3> dotnet user-secrets set SeedUserPW Secret123!
Successfully saved SeedUserPW = Secret123! to the secret store.
PS C:\Users\dharm\Dropbox\Documents\VisualStudio\AspNetCoreDocsAuthorization\final3> dotnet run
C:\Program Files\dotnet\sdk\5.0.102\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.EolTargetFrameworks.targets(28,5): warning NETSDK1138: The target framework 'netcoreapp3.0' is out of support and will not receive security updates in the future. Please refer to https://aka.ms/dotnet-core-support for more information about the support policy. [C:\Users\dharm\Dropbox\Documents\VisualStudio\AspNetCoreDocsAuthorization\final3\ContactManager.csproj]
C:\Program Files\dotnet\sdk\5.0.102\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.EolTargetFrameworks.targets(28,5): warning NETSDK1138: The target framework 'netcoreapp3.0' is out of support and will not receive security updates in the future. Please refer to https://aka.ms/dotnet-core-support for more information about the support policy. [C:\Users\dharm\Dropbox\Documents\VisualStudio\AspNetCoreDocsAuthorization\final3\ContactManager.csproj]
C:\Program Files\dotnet\sdk\5.0.102\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.EolTargetFrameworks.targets(28,5): warning NETSDK1138: The target framework 'netcoreapp3.0' is out of support and will not receive security updates in the future. Please refer to https://aka.ms/dotnet-core-support for more information about the support policy. [C:\Users\dharm\Dropbox\Documents\VisualStudio\AspNetCoreDocsAuthorization\final3\ContactManager.csproj]
fail: ContactManager.Program[0]
      An error occurred seeding the DB.
System.AggregateException: One or more errors occurred. (Value cannot be null. (Parameter 'password'))
 ---> System.ArgumentNullException: Value cannot be null. (Parameter 'password')
   at Microsoft.AspNetCore.Identity.UserManager`1.CreateAsync(TUser user, String password)
   at ContactManager.Data.SeedData.EnsureUser(IServiceProvider serviceProvider, String testUserPw, String UserName) in C:\Users\dharm\Dropbox\Documents\VisualStudio\AspNetCoreDocsAuthorization\final3\Data\SeedData.cs:line 49
   at ContactManager.Data.SeedData.Initialize(IServiceProvider serviceProvider, String testUserPw) in C:\Users\dharm\Dropbox\Documents\VisualStudio\AspNetCoreDocsAuthorization\final3\Data\SeedData.cs:line 26
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait()
   at ContactManager.Program.Main(String[] args) in C:\Users\dharm\Dropbox\Documents\VisualStudio\AspNetCoreDocsAuthorization\final3\Program.cs:line 35
info: Microsoft.Hosting.Lifetime[0]
      Now listening on: http://localhost:5000
info: Microsoft.Hosting.Lifetime[0]
      Now listening on: https://localhost:5001
info: Microsoft.Hosting.Lifetime[0]
      Application started.

Workaround
In case someone else runs into this issue, here's a workaround I used. Edit Program.cs as follows:
// EDIT UPDATE by @Rick-Anderson
// insecure code redacted
var testUserPw = config["SeedUserPW"];
After making this change, the program ran and I was able to test it according to the suggestions in the Test the completed app section.
Notes
SDKs that I have installed:
PS C:\Users\dharm> dotnet --list-sdks
3.1.200 [C:\Program Files\dotnet\sdk]
3.1.405 [C:\Program Files\dotnet\sdk]
5.0.101 [C:\Program Files\dotnet\sdk]
5.0.102 [C:\Program Files\dotnet\sdk]

Let me know if you'd like any other information.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: b66f8ffa-699c-8b11-0cab-805f5953805f
Version Independent ID: 7c151ed5-fd78-84fd-a08a-88912990c4ca
Content: Create an ASP.NET Core app with user data protected by authorization
Content Source: aspnetcore/security/authorization/secure-data.md
Product: aspnet-core
Technology: aspnetcore-security
GitHub Login: @Rick-Anderson
Microsoft Alias: riande

Passwords should never be added to source code. Follow the instructions to set it securely with password manager.

@Rick-Anderson ,
In my explanation above, I show that I did set the password as directed:
PS C:\Users\dharm\Dropbox\Documents\VisualStudio\AspNetCoreDocsAuthorization\final3> dotnet user-secrets set SeedUserPW Secret123!
Successfully saved SeedUserPW = Secret123! to the secret store.

Is there something else we should be doing?

I'll update the code to throw if fetching the password from config fails.

@dharmatech  feel free to PR this or maybe @serpent5  wants it if you don't.

@dharmatech  you should figure out why configuration is failing.

OK, some more info regarding the issue...
While in the project directory, I set the password:
dotnet user-secrets set SeedUserPW Secret123!

If I run the program via ctrl-f5 or f5 in vscode, the program works fine! I.e. the database is seeded.
However, if I remove the database:
sqlcmd -S "(localdb)\MSSQLLocalDB" -Q "DROP DATABASE [aspnet-ContactManager-02]"

and then run the program directly from PowerShell:
dotnet run

I get the error message I originally reported:
PS C:\Users\dharm\Dropbox\Documents\VisualStudio\AspNetCoreDocsAuthorization\final3> dotnet run
C:\Program Files\dotnet\sdk\5.0.102\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.EolTargetFrameworks.targets(28,5): warning NETSDK1138: The target framework 'netcoreapp3.0' is out of support and will not receive security updates in the future. Please refer to https://aka.ms/dotnet-core-support for more information about the support policy. [C:\Users\dharm\Dropbox\Documents\VisualStudio\AspNetCoreDocsAuthorization\final3\ContactManager.csproj]
C:\Program Files\dotnet\sdk\5.0.102\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.EolTargetFrameworks.targets(28,5): warning NETSDK1138: The target framework 'netcoreapp3.0' is out of support and will not receive security updates in the future. Please refer to https://aka.ms/dotnet-core-support for more information about the support policy. [C:\Users\dharm\Dropbox\Documents\VisualStudio\AspNetCoreDocsAuthorization\final3\ContactManager.csproj]
C:\Program Files\dotnet\sdk\5.0.102\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.EolTargetFrameworks.targets(28,5): warning NETSDK1138: The target framework 'netcoreapp3.0' is out of support and will not receive security updates in the future. Please refer to https://aka.ms/dotnet-core-support for more information about the support policy. [C:\Users\dharm\Dropbox\Documents\VisualStudio\AspNetCoreDocsAuthorization\final3\ContactManager.csproj]
fail: ContactManager.Program[0]
      An error occurred seeding the DB.
System.AggregateException: One or more errors occurred. (Value cannot be null. (Parameter 'password'))
 ---> System.ArgumentNullException: Value cannot be null. (Parameter 'password')
   at Microsoft.AspNetCore.Identity.UserManager`1.CreateAsync(TUser user, String password)
   at ContactManager.Data.SeedData.EnsureUser(IServiceProvider serviceProvider, String testUserPw, String UserName) in C:\Users\dharm\Dropbox\Documents\VisualStudio\AspNetCoreDocsAuthorization\final3\Data\SeedData.cs:line 49
   at ContactManager.Data.SeedData.Initialize(IServiceProvider serviceProvider, String testUserPw) in C:\Users\dharm\Dropbox\Documents\VisualStudio\AspNetCoreDocsAuthorization\final3\Data\SeedData.cs:line 26
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait()
   at ContactManager.Program.Main(String[] args) in C:\Users\dharm\Dropbox\Documents\VisualStudio\AspNetCoreDocsAuthorization\final3\Program.cs:line 35
info: Microsoft.Hosting.Lifetime[0]
      Now listening on: http://localhost:5000
info: Microsoft.Hosting.Lifetime[0]
      Now listening on: https://localhost:5001
info: Microsoft.Hosting.Lifetime[0]
      Application started.

Now, if I run the program once from vscode or VS in order to seed the database, then of course running the program via dotnet run in PowerShell works fine.
@Rick-Anderson, ever seen an issue like this before?

Should we just add Properties/launchSettings.json to final3?

That has nothing to do with configuration.

@dharmatech launchSettings.json sets ASPNETCORE_ENVIRONMENT to Development. Secrets are only loaded when it's set to Development, and it'll default to Production.

@serpent5
Aha! That's the issue! Thank you!!!

OK, so the tutorial:
Create an ASP.NET Core web app with user data protected by authorization
in this section:
The starter and completed app
directs the user to test the completed app before going through the tutorial.
The section:
Test the completed app
does not mention anything having the user set ASPNETCORE_ENVIRONMENT manually.
So two options I see for resolving this are:

Add Properties/launchSettings.json.

or

Update the tutorial to instruct the user to set ASPNETCORE_ENVIRONMENT manually.

I'm in favor of the first option. Reason being, if the user goes through the manual steps of creating the project, they will end up with this file after the dotnet new step.


Adding configuration to use S3 Sig V4 for presigned URLs

Added configuration values, signature_version and addressing_style to ensure SigV4 is used instead
of SigV2 which is an older and less secure way to access S3 with presigned URLs
Issue #, if available:
#110
Description of changes:
Adding a config object to specify the use of s3v4 signature version.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Thanks for the feedback, we will add this to our backlog and review it in the next release.

Thanks again for this feedback. This requested config has been incorporated in to the version 5.2.0 release.


Bug Report: Confusing Explanation Regarding Function “Currying” in Step 60 - “JavaScript, Algorithm & Data Structure” Course

Describe the Issue
Description:
The phrase "innermost function reference from addCharacters" in Step 60 is unclear. It doesn't accurately describe the currying process of the addCharacters function.
Suggested Fix:
Update the text to: "The .map() function receives the function returned from addCharacters, which takes two arguments in sequence."
Additional Info:
This will make it clearer for users who may not be familiar with currying.
Affected Page
https://www.freecodecamp.org/learn/javascript-algorithms-and-data-structures-v8/learn-functional-programming-by-building-a-spreadsheet/step-60
Your code
-

Expected behavior

Screenshots

System

Device: [e.g. iPhone 6, Laptop]
OS: [e.g. iOS 14, Windows 10, Ubuntu 20.04]
Browser: [e.g. Chrome, Safari]
Version: [e.g. 22]

Additional context
No response
### Tasks

I agree that the description could be better, but your description is not accurate, map is not receiving a function, but that innermost function, num => charRange(...).map(...) receive values from the first two functions called

I am opening this up for contribution.
A PR which resolves this issue should:

Update the instructions to read Now that your `.map()` function is receiving the returned `num =>  charRange(...).map(...)` function reference from the curried `addCharacters` calls, it will properly iterate over the elements and pass each element as `n` to that function.


Can we have a list of components based on priority?

As we are building random components, I think it will be good if we would have a list of components on the basis of priority.
priority could be, most used components, easy to build components etc.
@areai51 @rakeshmenon @nawazsk @sbmadhav @vikash-bhardwaj

Great idea @vinaymavi and that is something I've been toying with for a while.. but not had a chance to get to it.. anybody interested in giving it a first pass?

That's great. Is there any place where we can list all the components which needs to implement? or we can create list of tickets for components, so who ever will be available can pick up the task, assign to self and start working on it.

my suggestion would be to create one ticket for each component, so that we can detail out the props and other details of how the component should work. We can then create labels called p1, p2 etc and assign it to the ticket based on which components we would like to take up first

I have started creating a list of components and added some details on the features that should be there in each component. Please take a look and prioritize.


Bootstrap as a gem

Is it wise to include Bootstrap files in web sources? How about to load it as a gem?
https://github.com/twbs/bootstrap-rubygem

In development, it seems easier to customize Bootstrap with a direct access to its source code. In production I use the Ruby gem.


CI: Conformance E2E IPSec - Assert that no unencrypted packets are leaked

CI failure
Link: https://github.com/cilium/cilium/actions/runs/9452573621/job/26036086947
Sysdump: cilium-sysdump-6-final.zip
Error: bpftrace output is not empty
[17:27:48:158668] fd00:10:244:3::480e:55628 -> fd00:10:244:1::d336:8080 (proto: 6, ifindex: 11, netns: f0000098)

From a quick check, the mentioned packet is client2 -> echo-other-node, and the timestamp seems to match the dns-only tests.

Similar hit in https://github.com/cilium/cilium/actions/runs/9464379442
Error: bpftrace output is not empty
[11:46:20:735557] <IP_ADDRESS>:33052 -> <IP_ADDRESS>:8081 (proto: 6, ifindex: 13, netns: f0000000)
[11:46:20:737349] <IP_ADDRESS>:8081 -> <IP_ADDRESS>:33052 (proto: 6, ifindex: 7, netns: f0000000)
[11:46:20:737415] <IP_ADDRESS>:8081 -> <IP_ADDRESS>:33052 (proto: 6, ifindex: 7, netns: f0000000)
[11:46:20:737763] <IP_ADDRESS>:33052 -> <IP_ADDRESS>:8081 (proto: 6, ifindex: 13, netns: f0000000)
[11:46:20:737981] <IP_ADDRESS>:33052 -> <IP_ADDRESS>:8081 (proto: 6, ifindex: 13, netns: f0000000)
[11:46:20:738290] <IP_ADDRESS>:33052 -> <IP_ADDRESS>:8081 (proto: 6, ifindex: 13, netns: f0000000)
[11:46:20:738757] <IP_ADDRESS>:8081 -> <IP_ADDRESS>:33052 (proto: 6, ifindex: 7, netns: f0000000)
[11:46:20:739223] <IP_ADDRESS>:8081 -> <IP_ADDRESS>:33052 (proto: 6, ifindex: 7, netns: f0000000)
[11:46:20:739722] <IP_ADDRESS>:33052 -> <IP_ADDRESS>:8081 (proto: 6, ifindex: 13, netns: f0000000)
[11:46:20:739756] <IP_ADDRESS>:8081 -> <IP_ADDRESS>:33052 (proto: 6, ifindex: 7, netns: f0000000)
[11:46:20:739764] <IP_ADDRESS>:8081 -> <IP_ADDRESS>:33052 (proto: 6, ifindex: 7, netns: f0000000)
[11:46:20:740098] <IP_ADDRESS>:33052 -> <IP_ADDRESS>:8081 (proto: 6, ifindex: 13, netns: f0000000)
[11:46:20:740751] <IP_ADDRESS>:33052 -> <IP_ADDRESS>:8081 (proto: 6, ifindex: 13, netns: f0000000)
[11:46:20:741452] <IP_ADDRESS>:8081 -> <IP_ADDRESS>:33052 (proto: 6, ifindex: 7, netns: f0000000)
[11:46:20:741868] <IP_ADDRESS>:8081 -> <IP_ADDRESS>:33052 (proto: 6, ifindex: 7, netns: f0000000)
[11:46:20:782533] <IP_ADDRESS>:33052 -> <IP_ADDRESS>:8081 (proto: 6, ifindex: 13, netns: f0000000)

Adding another to the list: https://github.com/cilium/cilium/actions/runs/9569200261/job/26381203426
First thought this maybe only seen with KPR on, but the above failure proves otherwise (kpr is disabled).

Grokking over the sysdump a bit from @giorio94 upload. The only interesting things that occur to client2 from the agent's perspective is a regeneration due to policy change. The time stamps do not perfectly align, but not sure if there is any drift. Anything closer to 17:27:48 is pretty boring.
2024-06-10T17:27:28.200529896Z time="2024-06-10T17:27:28Z" level=info msg="Reloaded endpoint BPF program" ciliumEndpointName=cilium-test/client2-ccd7b8bdf-hkvxn containerID=e5645bb70b containerInterface= datapathPolicyRevision=123 desiredPolicyRevision=124 endpointID=144 identity=5239 ipv4=<IP_ADDRESS> ipv6="fd00:10:244:3::480e" k8sPodName=cilium-test/client2-ccd7b8bdf-hkvxn subsys=endpoint
2024-06-10T17:27:28.200651575Z time="2024-06-10T17:27:28Z" level=debug msg="Waiting for proxy updates to complete..."
Maybe its worth exploring if we leak any traffics while the pod has its endpoint eBPF programs regenerated, as it seems like is happening in the above log.

Drew up a bit of a chart trying to draw some correlation between tests and failures:

Job
Date
Leak Start
Correlated Test
Notes

https://github.com/cilium/cilium/actions/runs/9569200261/job/26381203426
Jun 18 2024
17:20:46
17:20:47 GMT [=] [cilium-test] Test [echo-ingress-l7] [52/80]

https://github.com/cilium/cilium/actions/runs/9523352532/job/26254619247
Fri, 14 Jun 2024 23:05:17 GMT
23:24:31:
Fri, 14 Jun 2024 23:24:26 GMT [=] [cilium-test] Test [echo-ingress-mutual-auth-spiffe] [61/80]

https://github.com/cilium/cilium/actions/runs/9520812670/job/26246873242
Fri, 14 Jun 2024 18:36:04 GMT
18:54:36
Fri, 14 Jun 2024 18:54:12 GMT [=] [cilium-test] Test [client-egress-l7] [55/80]

https://github.com/cilium/cilium/actions/runs/9497406601/job/26173911148
Thu, 13 Jun 2024 09:44:19 GMT
10:08:08
Thu, 13 Jun 2024 10:07:51 GMT [=] [cilium-test] Test [client-egress-l7] [55/80]

https://github.com/cilium/cilium/actions/runs/9496483981/job/26170991597
Thu, 13 Jun 2024 08:45:16 GMT
09:30:30
Thu, 13 Jun 2024 09:30:28 GMT [=] [cilium-test] Test [north-south-loadbalancing-with-l7-policy] [51/80]Thu, 13 Jun 2024 09:30:31 GMT ............Thu, 13 Jun 2024 09:30:31 GMT [=] [cilium-test] Test [echo-ingress-l7] [52/80]

https://github.com/cilium/cilium/actions/runs/9482095489/job/26126267472
Wed, 12 Jun 2024 11:57:43 GMT
12:41:41
Wed, 12 Jun 2024 12:41:36 GMT [=] [cilium-test] Test [echo-ingress-mutual-auth-spiffe] [61/80]

https://github.com/cilium/cilium/actions/runs/9480372702/job/26120937476
Wed, 12 Jun 2024 09:45:51 GMT
10:04:47
Wed, 12 Jun 2024 10:04:44 GMT [=] [cilium-test] Test [echo-ingress-mutual-auth-spiffe] [61/80]

https://github.com/cilium/cilium/actions/runs/9464379442/job/26071597253
Tue, 11 Jun 2024 11:04:31 GMT
11:46:20
Tue, 11 Jun 2024 11:46:19 GMT [=] [cilium-test] Test [echo-ingress-l7] [52/80]

https://github.com/cilium/cilium/actions/runs/9411867166/job/25925848376
Fri, 07 Jun 2024 04:59:41 GMT

Leak after key rotation

https://github.com/cilium/cilium/actions/runs/9400458006/job/25890119160
Thu, 06 Jun 2024 12:01:54 GMT

Leak after key rotation\

A bit wavering on a obvious link. However one thing is very common (other then the key rotation failures), the tests seem to catch leaks always after echo-ingress-l7 test is ran.

@giorio94
Think I'm starting to see a bit of a hint here.
If you checkout a flake like this: https://github.com/cilium/cilium/actions/runs/9520812670/job/26246873242
We see the leaking traffic is destined to <IP_ADDRESS>
If we check the ipcache on each node for this IP:
<IP_ADDRESS>/32                 identity=6 encryptkey=0 tunnelendpoint=<IP_ADDRESS>, flags=<none>

<IP_ADDRESS>/32                identity=6 encryptkey=0 tunnelendpoint=<IP_ADDRESS>, flags=<none>

<IP_ADDRESS>/32                 identity=1 encryptkey=0 tunnelendpoint=<IP_ADDRESS>, flags=<none>

Notice, that this is being identified as remote-node traffic and has a encryptkey of 0, which means we won't encrypt the traffic.
That IP is cilium_host from the sysdump
cilium-sysdump-6-final/cilium-bugtool-cilium-vtmgr-20240614-185726/cmd/ip-a.md
15:    inet <IP_ADDRESS>/32 scope global cilium_host

Unfortunately this IP address does not show anything in hubble flows, so we can't see the traffic. But for the above flake this seems like kube-dns trying to contact a Cilium node.

Checking out all the flakes. We have a combination of two issues. We see leaked traffic for pod -> remote node. But we also see leaked traffic for pod -> pod.

Job
Leak to Node

https://github.com/cilium/cilium/actions/runs/9569200261/job/26381203426
Yes - fd00:10:244:2::90f4/128 identity=6 encryptkey=0 tunnelendpoint=<IP_ADDRESS>, flags=

https://github.com/cilium/cilium/actions/runs/9523352532/job/26254619247
No - fd00:10:244:1::3712/128 identity=52848 encryptkey=3 tunnelendpoint=<IP_ADDRESS>, flags=

https://github.com/cilium/cilium/actions/runs/9520812670/job/26246873242
Yes - <IP_ADDRESS>/32 identity=6 encryptkey=0 tunnelendpoint=<IP_ADDRESS>, flags=

https://github.com/cilium/cilium/actions/runs/9497406601/job/26173911148
Yes - <IP_ADDRESS>/32 identity=6 encryptkey=0 tunnelendpoint=<IP_ADDRESS>, flags=

https://github.com/cilium/cilium/actions/runs/9496483981/job/26170991597
Yes - <IP_ADDRESS>/32 identity=6 encryptkey=0 tunnelendpoint=<IP_ADDRESS>, flags=

https://github.com/cilium/cilium/actions/runs/9482095489/job/26126267472
No - fd00:10:244:3::24f5/128 identity=55917 encryptkey=4 tunnelendpoint=<IP_ADDRESS>, flags=

https://github.com/cilium/cilium/actions/runs/9480372702/job/26120937476
No - fd00:10:244:2::c75e/128 identity=3171 encryptkey=3 tunnelendpoint=<IP_ADDRESS>, flags=

https://github.com/cilium/cilium/actions/runs/9464379442/job/26071597253
Yes - <IP_ADDRESS>/32 identity=6 encryptkey=0 tunnelendpoint=<IP_ADDRESS>, flags=

A bit of summary of what @giorio94 and I have been observing.
In the majority of instances the bpftrace program is flagging traffic which seems to be avoiding the tunnel device.
For instance in the latest flake posted by Marco above:
[16:25:31:965319] <IP_ADDRESS>:53 -> <IP_ADDRESS>:60547 (proto: 17, ifindex: 9, netns: f0000000)
This is coredns on a worker node going to CiliumInternalIP of the control-plane-node.
In ordinary conditions this traffic goes through the vxlan device before leaving the egress node.
The bpftrace script will then sniff into the packet and determine this is traffic towards a CiliumInternalIP and not report on it.
Two main possibilities here (as pointed out by Marco as well).

For some reason, during CI tests, traffic towards CiliumInternalIPs are no longer going over the tunnel and bpftrace is flagging non-encapsulated traffic (which is correctly not encrypted).
Its possible that the trace map is bugged and produces trace overrides for incorrect traffic.

There are, some legit flakes we see between pod <-> pod traffic still, but lets focus on one thing at a time, and try to figure out why we often see this CiliumInteralIP flakes occurring.

ok looking at the scheduled runs on main, this is popping up way too often. And it's almost exclusively TCP RST packets.
@jschwinger233 wdyt, can we stop the bleed and add an optional filter for such packets in the bpftrace script, so that the test doesn't fail? We want to address the root cause, but at the same time this is a non-critical leak and no reason to randomly fail CI at this high rate.

I went through https://github.com/cilium/cilium/actions/workflows/conformance-ipsec-e2e.yaml over past two weeks, all the leak flakes (bugs?) can be classified as:

ipv6 tcp reset: https://github.com/cilium/cilium/actions/runs/12502382264/job/34881398445. This is also tracked by https://github.com/cilium/cilium/issues/35485.
ipv4 dns query one.one.one.one: https://github.com/cilium/cilium/actions/runs/12472944852/job/34812634558
ipv4 tcp (non-reset, from-proxy): https://github.com/cilium/cilium/actions/runs/12547004754/job/34983756434

Simone is working on 1), I'll take a look at 3) this week. Also, as suggested by Julian, ignoring the ipv6 tcp reset leak for now makes sense to me, I'll open a PR for it.


[Bug]: Get-AstFunction is not recognized

What happened?
When I import monkey365 with command "Import-Module .\monkey365.psm1" I have an error message about Get-AstFunction
How to reproduce it
Steps to reproduce the behavior:

What command are you running?
 Import-Module .\monkey365.psm1
See error

Expected behavior
A clear and concise description of what you expected to happen.
Screenshots or Logs

From where are you running Monkey365?
Please, complete the following information:

Resource: VirtualBox VM
OS: Windows 10
PowerShell Version [$PsVersionTable]: 5.1.19041.2673
Monkey365 Version: latest
Others:

Additional context
Add any other context about the problem here.

Hi @qbrusa,
It is not a bug and you're using the "Import-Module" incorrectly. Please, see docs about how to correctly import a module in PowerShell.
https://silverhack.github.io/monkey365/install-instructions/
https://learn.microsoft.com/en-us/powershell/scripting/developer/module/importing-a-powershell-module?view=powershell-7.3

Hi @silverhack, Finaly I have found that  Import-Module .\monkey365.psd1 works. But If I want to use the default command to import a module, Powershell return to me that no valide module exist :

I think you could update your doc (or push your Powershell module in PSGallery ?), and thank you for your great work.

Hi @qbrusa there is no error and your issue is the same as the last one. You're seeing these errors because the folder does not have the same name as the PSD1. You should rename the folder from Monkey365-main to Monkey365. Once the name is changed you will be able to import the module. And all is explained here.
Cheers,


bash 403 error work around

I have forked this repo, to help maintain, and posted this bash work around to the 403 issue
/bin/bash/
[CODE]while ! gdrive upload --parent ; do sleep 1; done.[/CODE]
credit to github.com/ripee
While I know this hammers the api, it usually only takes about 10 tries to begin to work.
this example is the upload command without my directory file name


update sbt 0.13.16-RC1, remove workaround

As those are compatible we don't need to release again right?

Meh, disregard the comment, that's only the sbt version we use to build after all


Image Writing Issue

Hello @jcupitt
I am having a problem with saving the final result. I am using merge function for stitching images. I don't face any problem in saving 25X25 images but when I exceed the limit to 80X80 images it gives an error, image resolution is 480X640px.
I am using coordinates to merge the images coor is a numpy array having coordinate information.
Attaching sample code.
def stitch(coor):
    global onlyfiles,mypath,row,column
    onlyfiles=np.array(onlyfiles)
    files=onlyfiles.reshape(row,column)
    blend=False
    for r in tqdm(range(files.shape[0])):
        c=0
        if r==0:
            tile = pyvips.Image.black(coor[r,c][0],coor[r,c][1])
        else:
            tile = pyvips.Image.black(coor[r,c][0],coor[r,c][1]-coor[r-1,c][1])
        for c in tqdm(range(files.shape[1])):
            tile1 = pyvips.Image.new_from_file(join(mypath,files[r][c]), access="sequential")
            if r==0:
                x,y=coor[r,c]
            else:
                x,y=coor[r,c][0],coor[r,c][1]-coor[r-1,0][1]
            tile = tile.merge(tile1, 'horizontal', -x, -y, mblend =blend)
        if r>0:
            y=coor[r-1,0][1]
            stitch_img = stitch_img.merge(tile, 'vertical', 0, -y, mblend =blend)
        else :
            stitch_img=tile
    print (stitch_img.width,stitch_img.height,"width","height")
    stitch_img.dzsave('test_80X80')

Error:
Traceback (most recent call last):
  File "stitcher_coor.py", line 217, in <module>
  File "stitcher_coor.py", line 101, in stitch
  File "/home/machine/.virtualenvs/project/lib/python3.6/site-packages/pyvips/vimage.py", line 783, in call_function
  File "/home/machine/.virtualenvs/project/lib/python3.6/site-packages/pyvips/voperation.py", line 188, in call
pyvips.error.Error: unable to call dzsave
  vips__file_open_read: unable to open file "/home/machine/Desktop/iter6/set4_80X80/25_13.jpeg" for reading
unix error: Too many open files

I have also set open files limit to the maximum.

Hello, you need to raise the maximum number of open files. I have mine set at 64000, for example.
john@yingna ~ $ ulimit -n
64000

Exactly how to raise the number will need a bit of searching. It seems to vary a lot between linuxes.

Thanks @jcupitt
I tried to set it before but the maximum ulimit - n was  4096 and I failed to set it more. I followed this issue https://superuser.com/questions/1200539/cannot-increase-open-file-limit-past-4096-ubuntu and it worked well.
Can you suggest any other pipeline which I should implement to decrease RAM uses? Right now its taking 13.5GB for 80X80 images.  I am first making each rows and then merging all the rows vertically.

You could build as sets of 2x2 blocks. So join the four in the top left as
A B
C D

Then the next 2x2 block, and so on. Then do another pass and join up the 2x2 blocks in the same way, and repeat until the whole image is joined.
You can drop the concurrency down. Try VIPS_CONCURRENCY=1 before running your program.

Thanks for the suggestion. Closing this issue.


Error with latest Aurelia

I have other errors so the latest Aurelia release may have issues but, when I add the plugin declaration I get the following. This was working yesterday so it's something to do with Aurelia changes.
Error: Pipeline steps can only be added to the root router
at RouterConfiguration.exportToRouter (http://localhost:50082/jspm_packages/github/aurelia/router@0.12.0/aurelia-router.js:475:17)
at Router.configure (http://localhost:50082/jspm_packages/github/aurelia/router@0.12.0/aurelia-router.js:1195:26)
at Function.AddFluxPipelineStep (http://localhost:50082/jspm_packages/github/tfrydrychewicz/aurelia-flux@0.1.7/router.js:19:14)
at r.configure (http://localhost:50082/jspm_packages/github/tfrydrychewicz/aurelia-flux@0.1.7/index.js:14:31)
at http://localhost:50082/jspm_packages/github/aurelia/framework@0.17.0/aurelia-framework.js:34:34
at run (http://localhost:50082/jspm_packages/npm/core-js@0.9.18/modules/es6.promise.js:91:43)
at http://localhost:50082/jspm_packages/npm/core-js@0.9.18/modules/es6.promise.js:105:11
at module.exports (http://localhost:50082/jspm_packages/npm/core-js@0.9.18/modules/$.invoke.js:6:25)
at queue.(anonymous function) (http://localhost:50082/jspm_packages/npm/core-js@0.9.18/modules/$.task.js:40:9)
at Number.run (http://localhost:50082/jspm_packages/npm/core-js@0.9.18/modules/$.task.js:27:7)
at listner (http://localhost:50082/jspm_packages/npm/core-js@0.9.18/modules/$.task.js:31:9)

That's been already solved. Unfortunately there is a bigger issue with new DI container. Aurelia Team will add a hook for aurelia-flux. When I have that, I'll publish new version. That should be in a couple of days.

@LivesleyWA @tfrydrychewicz there's a configure callback on the router that you want to use instead which ensures that you don't run into issues with making the router already configured when the host app tries to configure it:
router.ensureConfigured().then(() => {
      let configuration = new RouterConfiguration();
      configuration.addPipelineStep("modelbind", FluxLifeCycleStep);
      router.configure(configuration);
});

The problem with Error: Pipeline steps can only be added to the root router was caused by the DI not returning AppRouter (root router) on
import {Router} from 'aurelia-router';
let router = aurelia.container.get(Router);

It used to do that, but it doesn't do that anymore. I thought that resolving AppRouter instead of Router should help here.

Thanks for the above tip. It would appear that major changes have been made to routers/pipeline and, as a result, an application which was working for me on Saturday night is now broken. I have a number of issues registered, of which this is one - although I am pretty sure that they are all related to the issue identified by @tfrydrychewicz. I am waiting for stability before continuing because hitting my head against a wall has limited charms. I hope the issues are resolved this week because, if not, my choice of Aurelia over Angular will be questioned which would be a pity.

Tomasz, Is there any update from the Aurelia team on the adjustments and likely delivery date?

+1 on this.
Many thanks for the great work so far!

+1

Any news on this?

@tfrydrychewicz We had the Beta release today. A lot of people would like to use the plugin, but it needs to be updated.

+1

Any update on the status of updating this plugin to work with Aurelia Beta?

if this is still an issue then +1 here too, I'm looking at using Aurelia and really wanted to use this plugin. @tfrydrychewicz are you looking to fix this?

@AdamWillden you may want to try and use this fork: tkhyn/aurelia-flux

@tkhyn after I posted I saw PR #22 which is essentially your fork - thanks for doing it!
I noticed that the PR says CI is failing. Is that anything to be concerned about or is it an issue with the CI itself do you know?
Thanks again.

@AdamWillden it looks like it's an issue with the CI itself, that uses a version of Firefox (31) that does not seem to support some SystemJS polyfills (https://travis-ci.org/tfrydrychewicz/aurelia-flux/builds/95719298). I did not care to investigate further as all tests pass with Chrome and Firefox 42+.

@tkhyn thanks. Just wanted that bit of confidence :+1:

Hello @tkhyn! I need to use Flux with Beta 1 and because this repo seems not to be actual. I tried to use your PR. I forked it, and made a release, but I'm still getting error:
Error: Pipeline steps can only be added to the root router
Is there any chance to get it working?

@tomasbonco I think I did the same as you've done. jspm works with releases so you need to specify @master to get the master branch as follows:
  {
      ...
      "aurelia-flux": "github:tkhyn/aurelia-flux@master",
  }

oh sorry I just read you made a release on your own fork. I don't know then... same error though

@tomasbonco I have no problem when I use<EMAIL_ADDRESS>instead of github:tkhyn/aurelia-flux@master in the jspm/dependencies of package.json and run jspm install. Just tested it on my fork of the sample aurelia-flux-todo app.


Data Obfuscation as a transform step

[Submitted by Kyle Crouse on GitLab]
This may be out of scope for this tool, but it could be useful to be able to obfuscate sensitive data prior to loading (or even during the export so that the sensitive data is never even saved to disk). Something like Faker coupled with the SF field metadata would make it fairly easy to generate realistic fake data.
If you do think it's out of scope for the tool, maybe some sort of plugin system for transforms could allow it to live on its own? Obviously neither the obfuscation transform nor the plugin system is a small undertaking, but I think they could make this tool even more useful.

Subsuming into #30, with the implementation of obfuscation itself likely deferred to plugins.


Exposing types in package

Hi again - I was wondering about exposing some types to dependent projects - stuff like MarketDetails or CandleStick for example are useful when you want to expect a specific type in a class constructor etc. I'm happy to do the work and pull request to expose them, but beyond putting explicit export { MarketDetail } from './MarketAPI'; on index.ts I'm not really sure how best to go about it.
If you're interested in the idea and can give me some pointers I'll submit a PR

Oh, that slipped through the cracks! 🙈
I just added a barrel export to expose the relevant enums and interfaces.
You can use it now like this:
import {CandleStick} from 'ig-trading-api';

const candleStick: CandleStick = {}; // ...

Oh, that slipped through the cracks! 🙈
I just added a barrel export to expose the relevant enums and interfaces.
You can use it now like this:
import {CandleStick} from 'ig-trading-api';

const candleStick: CandleStick = {}; // ...


training crash

Hi, I tried with tensorflow-gpu=1.12.0, but I experienced a "Segmentation fault", while the first fakes002364.png has been saved.
==================================
Building TensorFlow graph...
Stage1: Imitative learning...
Stage1: Imitative learning...
Stage1: Imitative learning...
Stage1: Imitative learning...
Setting up snapshot image grid...
Training...
Segmentation fault (core dumped)

the problem seems to be caused by tf_mesh_render. I built the rasterize_triangles_kernel.so myself and passed

I rebuilt the tf_mesh_render myself and got the

oh my bad, rebuilt the old version and it works now


cannot open abide.hdf5

Hi, I tried to run the training codes. The training was smooth excpet for the ABIDE dataset. The dataset was successfully downloaded:

but an issue was raised when reading the abide.hdf5 file:
"Unable to open object (file read failed: time = Mon Oct  9 10:52:03 2023\n, filename = '/root/xfb/datasets/prompt/abide/abide.hdf5', file descriptor = 24, errno = 5, error message = 'Input/output error', buf = 0x7775b70, total read size = 512, bytes this sub-read = 512, bytes actually read =<PHONE_NUMBER>3709551615, offset = 0)"
I tried to re-downloaded the file several times, but the issue remained. I'm using lmdb-1.4.1 and h5py-3.9.0. What could be the reason for this? Please help me out. Thx!

Sorry for seeing this so late. You might have solved the problem? If not, could you try abide.hdf5 file from [here](https://github.com/lsa-pucrs/acerta-abide?

Closed for no response. Reopen if still needs.


Permissions are broken with default installation

Create default installation of Django-CMS.
Create basic page.
Create editor account.
Give editor account all permissions including staff status, but not superuser.
Try to have editor account edit a page.
Get error message (You do not have permission to edit this plugin)

Note, I'm doing this on Arch Linux with the stable version. Let me know if you have any other questions about my installation.

@KeithCu I've encountered with the same issue. https://github.com/divio/django-cms/issues/5053 - here i described my situation.

I can confirm this.
I can also confirm, that even with all permissions available, a non-super-user has no permission to edit pages at all (the Page list is not available in admin, the user does not see the Pages... item in the site menu in the toolbar).
This only occurs when CMS_PERMISSION = True in settings.py (the default), and happens both in 3.2.3 and develop.

User Permission (django.contrib.auth.models.Permission). which allows to change Page model is missing name.
I've fixed it in #5133.
You can still find it in user form. It's unicode is something like "cms | Page | "

When CMS_PERMISSIONS is set to True, it's required to give each user global or individual permissions to a page, either by modifying the page specific permissions or setting a global permission for the user.


⚠ Bofrost.lu has degraded performance

In bdf4cdf, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 841 ms

Resolved: Bofrost.lu performance has improved in cdafacb.


fcitx: Add test (Unicode input, table input, m17n)

Motivation for this change
Added test for fcitx
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)

Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[x] Fits CONTRIBUTING.md.

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/prs-ready-for-review/3032/273

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/need-help-writing-a-test/8522/8

Hi, Thank you!
I made another commit, you can look at the diff of .nix file and look at the config, profile files in their own right (not diffs). I will squash the two commits before you merge.

ToDo: C-S-f in zhengma

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/how-many-people-are-paid-to-work-on-nix-nixpkgs/8307/86

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/how-to-setup-an-input-method-for-cjk-users/9533/4

This is great. Thank you so much @asdf8dfafjk !


Describe local development setup

This one is different from the #447 because there are more ways to contribute.

Closed via #478


WIP: Add is_active_in

See: https://github.com/oborel/obo-relations/issues/193

This was added elsewhere.


setChecked (true) appear like this !!

@amineghabi Its because I use a Timer I need to update it to an androids handler, I'll do so this weekend :)

I would like to bump this issue. It is still present in the current version.

Hi @amineghabi and @TimboMachimbo I fixed this issue can you please try it out and send me some screenshots from your devices so that we can make sure the issue doesn't happen for sure, so that we close this issue. sorry for the long time to fix this

by default i set isChecked=true then i got like this view.

@prashant71 you did not set the view in the correct position in your layout

@arlindiDev i m used in recycler view adapter inside cardview.

@prashant71 yes I understand but still its in the incorrect position in the layout. Also the bug might still be present for RecyclerView I haven't tested it there


mistral neuron compiler

Hi I have been following up the last PR over the mistral branch. Just a couple hours ago the branch was merged with main. However, after updating the optimum-neuron to the last version 0.0.15.dev0 and running this code for compiling:
from optimum.neuron import NeuronModelForCausalLM
from transformers import AutoTokenizer
#
# # model id you want to compile
#finetuned mistral model with peft QLoRa
vanilla_model_id = "growth-cadet/mistral-7b-news-to-JSON02"
#
# # configs for compiling model
compiler_args = {"num_cores": 2, "auto_cast_type": "bf16"}
input_shapes = {
   "sequence_length": 2048, # max length to generate
   "batch_size": 1 # batch size for the model
   }
#
llm = NeuronModelForCausalLM.from_pretrained(vanilla_model_id, export=True, **input_shapes, **compiler_args)
tokenizer = AutoTokenizer.from_pretrained(model_id)
#
# # Save locally or upload to the HuggingFace Hub
save_directory = "llama_neuron"
llm.save_pretrained(save_directory)
tokenizer.save_pretrained(save_directory)

I got this error:

KeyError                                  Traceback (most recent call last)
Cell In[2], line 13
8 input_shapes = {
9    "sequence_length": 2048, # max length to generate
10    "batch_size": 1 # batch size for the model
11    }
12 #
---> 13 llm = NeuronModelForCausalLM.from_pretrained(vanilla_model_id, export=True, **input_shapes, **compiler_args)
14 tokenizer = AutoTokenizer.from_pretrained(model_id)
15 #
16 # # Save locally or upload to the HuggingFace Hub
File /usr/local/lib/python3.8/dist-packages/optimum/modeling_base.py:372, in OptimizedModel.from_pretrained(cls, model_id, export, force_download, use_auth_token, cache_dir, subfolder, config, local_files_only, trust_remote_code, revision, **kwargs)
369     trust_remote_code = False
371 from_pretrained_method = cls._from_transformers if export else cls._from_pretrained
--> 372 return from_pretrained_method(
373     model_id=model_id,
374     config=config,
375     revision=revision,
376     cache_dir=cache_dir,
377     force_download=force_download,
378     use_auth_token=use_auth_token,
379     subfolder=subfolder,
380     local_files_only=local_files_only,
381     trust_remote_code=trust_remote_code,
382     **kwargs,
383 )
File /usr/local/lib/python3.8/dist-packages/optimum/neuron/modeling_decoder.py:155, in NeuronDecoderModel._from_transformers(cls, model_id, config, use_auth_token, revision, force_download, cache_dir, subfolder, local_files_only, trust_remote_code, task, batch_size, sequence_length, num_cores, auto_cast_type, **kwargs)
144 # Update the config
145 config.neuron = {
146     "task": task,
147     "batch_size": batch_size,
(...)
152     "compiler_version": get_neuronxcc_version(),
153 }
--> 155 return cls._from_pretrained(checkpoint_dir, config)
File /usr/local/lib/python3.8/dist-packages/optimum/neuron/modeling_decoder.py:205, in NeuronDecoderModel._from_pretrained(cls, model_id, config, use_auth_token, **kwargs)
201 auto_cast_type = neuron_config["auto_cast_type"]
203 check_compiler_compatibility(neuron_config["compiler_type"], neuron_config["compiler_version"])
--> 205 exporter = get_exporter(config, task)
207 model_path, checkpoint_path, compiled_path = cls._get_neuron_paths(model_id, use_auth_token)
209 # transformers-neuronx uses f32/f16 instead of fp32/fp16
File /usr/local/lib/python3.8/dist-packages/optimum/neuron/modeling_decoder.py:49, in get_exporter(config, task)
48 def get_exporter(config, task):
---> 49     return TasksManager.get_exporter_config_constructor(model_type=config.model_type, exporter="neuron", task=task)()
File /usr/local/lib/python3.8/dist-packages/optimum/exporters/tasks.py:1807, in TasksManager.get_exporter_config_constructor(exporter, model, task, model_type, model_name, exporter_config_kwargs)
1804     model_type = model_type.replace("_", "-")
1805     model_name = getattr(model, "name", model_name)
-> 1807 model_tasks = TasksManager.get_supported_tasks_for_model_type(model_type, exporter, model_name=model_name)
1809 if task not in model_tasks:
1810     synonyms = TasksManager.synonyms_for_task(task)
File /usr/local/lib/python3.8/dist-packages/optimum/exporters/tasks.py:1101, in TasksManager.get_supported_tasks_for_model_type(model_type, exporter, model_name)
1095     raise KeyError(
1096         f"{model_type_and_model_name} is not supported yet. "
1097         f"Only {list(TasksManager._SUPPORTED_MODEL_TYPE.keys())} are supported. "
1098         f"If you want to support {model_type} please propose a PR or open up an issue."
1099     )
1100 elif exporter not in TasksManager._SUPPORTED_MODEL_TYPE[model_type]:
-> 1101     raise KeyError(
1102         f"{model_type_and_model_name} is not supported yet with the {exporter} backend. "
1103         f"Only {list(TasksManager._SUPPORTED_MODEL_TYPE[model_type].keys())} are supported. "
1104         f"If you want to support {exporter} please propose a PR or open up an issue."
1105     )
1106 else:
1107     return TasksManager._SUPPORTED_MODEL_TYPE[model_type][exporter]
KeyError: 'mistral is not supported yet with the neuron backend. Only [] are supported. If you want to support neuron please propose a PR or open up an issue.'

Hi @jmparejaz ,
Yes this is expected. We do not support inference for now, just training. Inference should come soon. cc @JingyaHuang

@michaelbenayoun @JingyaHuang Is there any update on exporting the Mistral 7B to neuron format for inference on inf2 machine?
I wanted to deploy Mistrak 7B model using neuron TGI (https://github.com/huggingface/optimum-neuron/tree/main/text-generation-inference), but if exporting is not supported then I am stuck.

is this issue already solved??

@abhijithnair1 You can use Mistral on inf2, just not with Optimum yet.  Check out https://huggingface.co/aws-neuron/Mistral-neuron

Mistral is supported in optimum-neuron 0.0.17.


Linear - grammar, minor re-wording, typos

Some grammar, typo and minor re-wording changes to linear_models.py. Happy to change back the more opinionated changes!
Some suggestions:

We use the terms formula/formulate, rule and model (sometimes a bit interchangeably) but we never explicitly explain the relationship between them/what a model is e.g., a model is a formula for relating features to target. We do talk about it a bit in the first notebook but it's not clearly relating 'model' to 'rule':
https://github.com/INRIA/scikit-learn-mooc/blob/master/python_scripts/01_tabular_data_exploration.py#L202-203
We switch between using the terms infer/inference & predict/prediction. This is a nitpick but I think we should consistently only
use one term or explain that they mean the same thing.
Might not be necessary but should we clarify that we curve just means 'line' and does not need to be curved, i.e., can be a straight line?

I will put other suggestions at the relevant locations in a review.

We switch between using the terms infer/inference & predict/prediction. This is a nitpick but I think we should consistently only use one term or explain that they mean the same thing.

Indeed we should be careful, they have different meaning and we should use them wisely.

Might not be necessary but should we clarify that we curve just means 'line' and does not need to be curved, i.e., can be a straight line?

+1

Indeed we should be careful, they have different meaning and we should use them wisely.

Had to google this but I feel this pic sums the difference nicely:

I am going this merge this one. I feel some of the remaining issues are in #45 and for some who are not, I suggest creating a separate issue.


sample load syntax - \x gives error

{4}\xylophone

results in error:
Error in worklet evaluation: SyntaxError – Invalid hexadecimal escape sequence

Also problematic:
{4}\boom

oom
maxi-processor.js:205 TypeError in worklet evaluation: TypeError – Cannot read property 'length' of undefined

in short: the \ as a marker in the grammar for a sample name seems to have some unintended sideeffects.

similar:
{1}\firecrack

TypeError in worklet evaluation: TypeError – Cannot read property 'length' of undefined

fixed with a great new regexp


Show images on capable terminals

The part you want to Enhancement
As the title says, show image files to avoid switching to another application.
Why it is necessary to enhancement
Opening the image files on its associated application breaks the navigation flow.
Additional context
_Screenshot from another joshuto TUI FM:

duplicate of #26


500 Internal Server Error For International Teams

Message: "Error while executing query. Error: ER_BAD_FIELD_ERROR: Unknown column 'XXX' in 'where clause'".
This is probably because the key contains text.

Fixed


throw new Error('No module loader found for "'+ext+'".');

I have a scenario where my file name has the following format filename.tmp.js.
The extension from rechoir is .tmp.js where I was expecting that to be .js.
I believe because of that we can't find the corresponding module loader for this file.

Are you trying to use rechoir directly, or via another tool like Liftoff?

Via another tool with Liftoff.

cc @phated

Which tool?
This is unfortunate, but I believe you'll need to use the --require flag to manually load babel in this instance. Most people who use Liftoff map extensions to loaders using this: https://github.com/tkellen/js-interpret/blob/master/index.js
I don't see a sane path forward to supporting any arbitrary extensions that can "just work".

Thanks for your fast follow-up on this @tkellen, but, in your opinion, what is the extension of this file testing.tmp.js ? For me, I was expecting that to be .js. The current way you guys are loading the extensions it treats as tmp.js (I believe you guys did that because of the .babel.js convention).
If .js is the extension of testing.tmp.js then we don't have any arbitrary extensions. Unless you judge differently, which then I will treat the issue with the other library and verify if they can change to testing-tmp.js or something like that.
In my opinion, this library has to support this, but I'm probably not very much experienced to argue much.

@tkellen I think we need to check extensions recursively.  Let me think about this some more.

Yeah, @phated: the Pull Request that I sent tried to do that in a poor way. The PR is #25.
I've changed the regex a little bit and return, instead of a single extension, an array of possible extensions.
Hopefully this will help you guys to figure out something better than what I did.

@alansouzati Sorry for closing your PR so soon. I think the approach you were taking was very close to what we should do.
If you can re-implement and have the extension checking break on the first match I'd be happy to merge this (with tests)!

No problem @tkellen, I will re-implement. I'm assuming I should send a new PR, right?

Yes please.

I'm getting the same error as well, using liftoffjs after updating.  However, I'm getting it with 'rc' and no extension:
const cli = new Liftoff({
  name: 'crawler',
  extensions: objectAssign({'rc': null}, interpret.jsVariants),
  configName: '.crawler',
  v8flags: v8flags
});

I tried testing with @alansouzati new PR.  I still have the same issue with only a slightly different message:
throw new Error('No module loader found for "'+usedExtension+'".');
          ^
Error: No module loader found for ".crawlerrc".

I hope this helps.  If I use any of the other extensions, it works as expected.  But of course, I built it with 'rc' in mind.
Thanks in advance.

@seethroughtrees Please note that all extensions must include the leading dot, i.e. ".rc" instead of just "rc".

@seethroughtrees Crap. That is a regression. We need to add the full filename as the first option in #26.

Thanks @tkellen

This is partially resolved in 0.6.2. Filenames like dist.configfile.js or .configfilerc will work. We still haven't resolved compound extensions like configfile.tmp.js tho.

@seethroughtrees Can you confirm this is working for .configrc files?

Just got a chance to test.  worked great with .configrc... Thanks for the work!

@alansouzati This should be working now. I fixed it in https://github.com/tkellen/js-rechoir/commit/62fb2a62b6102f9ba07a05c7380ce237bb47b349.

cool thanks @tkellen . Just to make sure I understood correctly, the fix is where you add the nothrow option?

Yeah. This works because you're not using a special module loader (and also fixes the situation with files like .configrc). What it doesn't address (and I'm not sure that I want to address) is solving compound extensions that require a loader.

Gotcha you. Awesome thanks man.

Sure thing! I'm going to dig in using your PR in a min to see if I can fix the rest of this stuff.


Service Publishing

This package only supports enumeration of services published via mDNS.
If no service publishing is supported, stating this in README would save developers
looking to publish services some time.
Thanks.

I agree.

this should be stated in the readme.
this package is only for zeroconf discovery, not for zeroconf publishing

Done. Though a PR to add service publishing would be accepted.


Documentation doesn't build

I've got a reproducible failure on the current head of the gh-pages branch (commit 50a5e500e0853fe3688203a6adbc7b4f4829cabf) using the following Dockerfile:
FROM node:4.3.1-onbuild
RUN apt-get update
RUN apt-get install -y rubygems ruby-dev
RUN gem install jekyll
RUN npm install -g gulp
CMD ["gulp", "serve"]

If I try to start this container up, I get this:
[07:14:29] Using gulpfile /usr/src/app/gulpfile.js
[07:14:29] Starting 'compile-less'...
[07:14:29] Starting 'js'...
[07:14:29] Finished 'js' after 656 ms
{ [Error: variable @dl-horizontal-breakpoint is undefined in file /usr/src/app/node_modules/bootstrap/less/type.less line no. 214]
  type: 'Name',
  filename: '/usr/src/app/node_modules/bootstrap/less/type.less',
  index: 4161,
  line: 214,
  callLine: NaN,
  callExtract: undefined,
  column: 21,
  extract:
   [ '',
     '  @media (min-width: @dl-horizontal-breakpoint) {',
     '    dt {' ],
  message: 'variable @dl-horizontal-breakpoint is undefined in file /usr/src/app/node_modules/bootstrap/less/type.less line no. 214',
  stack: undefined,
  lineNumber: 214,
  fileName: '/usr/src/app/node_modules/bootstrap/less/type.less',
  name: 'Error',
  showStack: false,
  showProperties: true,
  plugin: 'gulp-less',
  __safety: { toString: [Function: bound ] } }

A little bit of research turns up an issue raised on Bootstrap.
Modifying the package.json file to restrict the bootstrap version to exactly 3.3.5 (instead of ^3.3.5) makes everything build just fine. This seems to be a decent short-term fix for people wanting to make small changes.
The build failure, though, seems to indicate that the variables.less file (or bootstrap-variables.less file?) in the repo was made by copying the one out of Bootstrap version 3.3.5 and modifying it for Falcor's needs. This practice seems to be very much discouraged. The long-term fix, I think, would be to do what user orrd suggests on the Bootstrap issue

The solution is that rather than customizing variables.less, what you're supposed to do is still let boostrap.less include the variables.less file, and then customize your Bootstrap variables in a separate file or somewhere else that is included after boostrap.less is included, so that your variables override just the ones you want to change, but you still let it also define all of the variables in the standard variables.less.

thanks for reporting this @cjlarose and for all the investigation. I've issued a PR that'll fix this for now by just locking the version to 3.3.5 (realistically, the Falcor site probably never needs to update bootstrap 3 unless it undergoes a complete rewrite).
Context, in case you're interested: the best/only way to set up boostrap with heavy customization some time in the past was by forking settings.less, and I think may be required by the Falcor site due to the idiosyncratic way less handles variable declarations and overriding. However, I'm planning to migrate the new site styles to the sass build of bootstrap and use a similar tactic to the one you referenced for the variable overrides, since sass seems to have simpler and more flexible includes handling.

Excellent! Thanks!

PR was merged. Closing. (Very late) Thanks Nicolas!


[RFC] Better events

I am working on next release to provide better API usage, one area to fix the emitted events.
I would like to hear some feedback for the following proposals before proceed,
1. Generic events between different ad-types. E.g. AdLoaded, AdFailed events with adType in the event data.
Pros

Less event types
Some generic handling function, only need to listen to one event type. E.g. log AdFailed

Cons

Some ad-types may emit more or less events. E.g. AdRewarded is only used by rewarded video
Some ad-types event data may contain extra information, which can be hard to consume as adType need to be checked.

2. Each ad-type has its own set of events. E.g, BannerLoaded, InterstitialFailed events.
Pros

No need to check adType in the handling function

Cons

Increase the number of events

I prefer (2) as it is more explicit, what do you think? @becvert @vintage @code4youreal @warcry2000
Maybe you have better idea.

I've been working with both approaches "generic events" and "specified events". As you said the generic events have one advantage - you can listen just for one event and receive multiple types of events using single line of code. Even if that sounds great, it's almost useless as in all of my apps I've had to do a conditional statement to check what exactly came in and decide what to do with it. I'm for the second approach as it gives more granularity and would keep the code clean.
Honestly the solution which I liked the most is the one used by the Heyzap - https://github.com/Heyzap/heyzap-cordova/blob/master/docs/advanced.md which simply split all the events into "namespaces", so we would have something like:

AdMob.Banner.Events.LOADED
AdMob.Banner.Events.CLICKED
AdMob.Interstitial.Events.FETCH_SUCCEED
AdMob.Interstitial.Events.FETCH_FAILED

Hello,
Since the events were so different, and because there were only a few common ones, I separated the events for each one.
So if someone uses only banner, worry about the events of the banner, if someone uses only videos, then only worry about the events of the videos.
Also I think the code would be cleaner.

Awesome, I like the namespace idea, it could group the methods too.
e.g. admob.banner.events.LOADED, admob.rewardvideo.show()

I think it is a great idea.


[Bug] Missing Entries in .aider.input.history When Using Message Flag

Please log to .aider.input.history commands used via --message COMMAND, --msg COMMAND, -m COMMAND parameter, to maintain consistency with .aider.chat.history.md .
Current behavior:

it seems like when using -m parameter .aider.chat.history.md is updated but not .aider.input.history.

Expected behavior:

both files are expected to be updated in same way like command would be executed in interactive mode.

Appendix
$ aider f -m 'add `C.1` to end of `f` '  # let's assume I am tracking .aider.chat.history.md AND .aider.input.history
(...)
$ git status
On branch master
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   .aider.chat.history.md

Also as we see we have first command from interactive mode when I asked to crate file f but we have no sign of second command I run afterwards via -m parameter:
$ cat .aider.input.history

# 2023-11-27 22:57:26.936593
+Make file f with one line finished with new line character with word `start` and add it to repostiory.

Bug(?) found during experiment:

https://gist.github.com/gwpl/65251097795a2ab8ba6d6936f08791bd
and related question: https://github.com/paul-gauthier/aider/issues/375

https://github.com/paul-gauthier/aider/pull/377

And attempt with Unit test: https://github.com/paul-gauthier/aider/pull/378
(and comment how I run pytest in provided Dockerfile: https://github.com/paul-gauthier/aider/pull/378#issuecomment-1828893755 )


[WIP] K8s Pre-req check

Signed-off-by: metonymic-smokey<EMAIL_ADDRESS>Description
Adds a pre-requisite check when platform is Kubernetes to handle errors gracefully when Kubernetes is not running.
This PR fixes #2579
Screenshot

Signed commits

[x] Yes, I signed my commits.

@reviewers
Assumptions in this draft:
Using ${HOMEDIR}/.kube/config as the location for kubeconfig.
To fix: Should I accept a flag to specify the path?(which can be passed in start.go to PreReq).

The —config flag should serve this purpose. We need to add kubeconfig file location in the context structure.

On Mar 27, 2021, at 7:20 AM, Aditi Ahuja @.***> wrote:

@reviewers
Assumptions in this draft:
Using ${HOMEDIR}/.kube/config as the location for kubeconfig.
To fix: Should I accept a flag to specify the path?(which can be passed in start.go to PreReq).
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

The —config flag should serve this purpose. We need to add kubeconfig file location in the context structure.

@leecalcote So, we have the default location of the kubeconfig file as .kube/config and we can add it to the context. And the user can specify a different file if he needs to. Is that the case?

@navendu-pottekkat thank you for the review!
Incorporated the suggested changes.
Let me know if these are acceptable. I'll then add the check to IsMesheryRunning similarly.

The documentation link being so specific is both nice and fragile at the same time. That link and anchor are subject to change without notice. Removing the anchor and leaving "https://docs.meshery.io/guides/upgrade" may serve to be both convenient to the user and less fragile to sustain.


Cannot launch start SillyTavern with Extras

OS: Linux (Ubuntu 23.10)

I believe that this code is incorrect:
https://github.com/SillyTavern/SillyTavern-Launcher/blob/8ec79f07837932ebe123e9b4f68ef251b303672b/launcher.sh#L287-L288
According to bash manual, exec will replace current shell with specified subcommand.
This will only launch the first of these commands.

Not sure about all the supported terminals, but in case of konsole there is an error: "Could not find binary: " "cd".
This is probably because konsole try to run specified command exactly as a new program. There is also used bash syntax (&&), so it should be wrapped in bash caommnad.

Also, probably the path is wrong. In my case, this command is run: konsole -e 'bash -c '\''cd [...]/SillyTavern-Launcher./SillyTavern && ./start.sh'\''' (Note unnecessary . character after Launcher word).

I used this code as a workaround:
            "$detected_terminal" -e "bash -c 'cd $(dirname "$0")/SillyTavern && ./start.sh'" &
            "$detected_terminal" -e "bash -c 'cd $(dirname "$0")/SillyTavern-extras && ./start.sh'" &

In SillyTavern-extras project, there are missing execute permissions for start.sh file. This may be fixed by launcher or maybe bettor to report and fix in that project?

Script start.sh in SillyTavern-extras should use exactly the same conda environment which was configured in launcher (install.sh). According to code in https://github.com/SillyTavern/SillyTavern-Extras/blob/main/start.sh this is configured by setting $CONDA_PATH environment variable. This should be done in laucher.sh.

this is configured by setting $CONDA_PATH environment variable. This should be done in laucher.sh.

I had considered this while rewriting the conda finding code, but the issue is that I am not sure if $CONDA_PATH is something that is already used/standard and so would be detected with other programmes, so having the installer have a side effect of setting that could be malicious

having the installer have a side effect of setting that could be malicious

That reasonable concern.
But from install.sh I deduced that CONDA_PATH variable is mandatory (otherwise if variable is not set, there are errors, see #28).

If this is indeed mandatory in install script, then it should be also used in start.sh.
Otherwise (if this is an option for user, to customize script and use specified conda), then install.sh needs to be refactored a bit, to check if conda is used either from:
a) environment variable,
b) detected existing conda installation by script,
c) miniconda installed locally by install.sh script (this is my case, I do not have conda installed neither for system nor for user).

This is how I image a perfect launcher: "run install.sh, then start.sh". At this moment, support for 2.c) is missing.

Or maybe README.md needs a little adjustment?
I think that this may work: "If conda is not detected by install script, you must set environment variable CONDA_PATH pointing to existing conda installation. If conda is not installed at all, you must set CONDA_PATH pointing to directory where conda installation will be placed".
But I personally prefer this to be done with minimal user interaction. For example: conda installation may have default path [...]/SillyTavern-Launcher/miniconda/, and if this is dir is detected, then this conda is used for SillyTavern launchers. What you think about that solution?


Download icon on the Results page

To create an icon for download search results with a 'save as' option in CSV and GEDCOM formats.

OK, I have had a go at both #571 and #572, and the (relevant bit of the) search results page now looks like this:

... which I think is neater.

@richardofsussex
Nice one — looking good. How does it look in a narrower window?

The buttons realign themselves as the window gets narrower. That all seems to be well-behaved:

Thank you Richard. How about making just one Download icon with the option to save in different formats?
Another point that I would like to clarify again is the icons. Do we need buttons for them? Perhaps we could make icons without buttons?
Please have a look at the attached image.

"One download icon with the option to save in different formats" implies that the icon is on a drop-down selection control like the (now defunct) Tools menu. Personally I think this would be visually messy, and operationally more complex for the user to use. It also tells the user nothing about what download options are available until/unless they click on it.
@AlOneill has already explained that icons without associated buttons (and text) are not accessible.

This is now available for testing on test3. Thank you @Vino-S for this.

To investigate accessible options for Download feature

We discussed that we might consider creating a drop-down menu for the download feature.

Decision: to leave download buttons as they are now.


Use hazel update server for download link

Right now the download link is hard-coded to the latest version, I think we have a url somewhere that redirects to the latest release, we should use that instead.

Closing as this is now using rewrite: https://github.com/wulkano/getkap.co/blob/master/now.json#L3


model

Hello, I have some questions that have been showing no module named iUNet.I want to know what's going on, thank you!

Hi, you can find iUNet in the folder of utils.

I did not encounter this issue before, maybe this could be due to your path setting problem. I suggest you try to google "no module named" and find a solution for that.

Thank you for your reply. I have solved the problem.