Is the new version lacking the back and forward buttons? I can't find them.

Thank you for this amazing program, by the way.

See #198
Thanks :+1:

Ok, but you haven't given an explanation as to why you removed the buttons.

Please read that issue I linked you will find a bit where I said this

Back / Forward missing was a design decision based on a thread somewhere (which I will probably never find again) explaining that if you can't use a website without forward/back navigation buttons the website is doing something wrong. :+1:

Bearing in mind that was on a google product forum I tried using GPMDP without the forward / back buttons and I managed suprisingly well. I tried to use them for the first few days and after that not really

TLDR: I removed them because forward / back buttons are in a browser to navigate between websites, they should not be used to navigate internally inside a website

Thank you, but I really liked them and I see no point in removing them. It was really useful for me.

I'm missing these buttons too.

I'm sorry but unless someone can come up with a "very" good reason as to why they should come back.  They aren't coming back

Thanks MarshallOfSound for creating them. I'm just wondering if it's any effort to just not remove the buttons' code that you worked hard on? They were useful for me to jump to another song, add it to my queue, then go back to the previous album I was on.  It was just useful, there was nothing negative about them that I could see.

Unfortunately it's not as simple as "not removing" the code that did them before.  This app is now based on an entirely new framework and written in a different language.
The use case you describe of going from album to song and then back to album can be achieved through the GPM UI because the album appears as a link next to all songs.
Lack of negatives is not a reason for a feature, it needs to have a positive?  And to me at the moment, the buttons would clutter up the UI and make users navigate through the app in an incorrect way.

OK, I understand. I didn't realize about the new language.
I guess that's fine, it's just that it's been working well for me in Spotify and in your program.


A way to config to use $applyAsync instead

Instead of having to manually override the Promise.setScheduler, offer a way to use $applyAsync. It's better for performance to hold off every promise resolutions to the next cycle, and one time only

It already uses $evalAsync which will already do this. To my understanding the difference between $applyAsync and $evalAsync is the same as the difference between $apply and $digest and as $evalAsync is executed on the $rootScope there shouldn't be a difference.

$applyAsync 'collects' everything that calls $applyAsync in the current $digest and schedules it to the next cycle.

Yup, but $evalAsync does that as well.

Closing as even $q uses $evalAsync for its next tick function: https://github.com/angular/angular.js/blob/master/src/ng/q.js#L221


[BEAM-3625] Enable DoFn params in Map, Filter, etc.

DESCRIPTION HERE

Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Make sure there is a JIRA issue filed for the change (usually before you start working on it).  Trivial changes like typos do not require a JIRA issue.  Your pull request should address just this issue, without pulling in other changes.
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue.
[ ] Write a pull request description that is detailed enough to understand:

[ ] What the pull request does
[ ] Why it does it
[ ] How it does it
[ ] Why this approach

[ ] Each commit in the pull request should have a meaningful subject line and body.
[ ] Run mvn clean verify to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

R: @udim

Good point, with FlatMap we  literally replace the DoFn.process callable. Updated the PR description.

Jenkins: retest this please.


13 export security with filled data to string

FIX #13

Security prints name, stock symbol, price, shares outstanding, book value
Security invokes updates via given Scraper
abstract from ScraperBloomberg via interface IScraper

Codecov Report

Merging #14 into master will decrease coverage by 12.26%.
The diff coverage is 67.61%.

@@             Coverage Diff             @@
##           master      #14       +/-   ##
===========================================
- Coverage   87.96%   75.69%   -12.27%
===========================================
  Files           2        4        +2
  Lines         108      144       +36
===========================================
+ Hits           95      109       +14
- Misses         13       35       +22

Impacted Files
Coverage Δ

src/scraper/security.py
63.33% <54.16%> (-17.62%)
:arrow_down:

src/scraper/iscraper.py
54.54% <54.54%> (ø)

test/test_scraper/test_security.py
85.45% <68.18%> (-12.33%)
:arrow_down:

src/scraper/scraper_bloomberg.py
77.08% <77.08%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d24671e...65a29eb. Read the comment docs.


Exporting is slow again

Exporter used:
Better BibTeX
Expected behavior:
Second full export would be reasonably fast, under a minute hopefully, for my CCNLab group library which I just invited you to join for testing, which has 24,350 items.
Actual behavior:
First full export took well over 15 minutes, and second and subsequent ones took 6 minutes. At one point it was down to ~20 sec or so, but it seems like it has been slow like this for a while, but now I'm finally writing a paper so I really notice it. :)

I see (I think) you've shared your library with me, but cache use (which must be the issue here) is dependent on your settings. It'd be helpful if you could enable debugging (see 2nd point from @label-gun), export your library as you would normally, and then right-click your library, and send a BBT debug log from there. That will give me a full copy of your library, your BBT settings, and a copy of the items in your library for testing.

JR75IZ53-euc is the report number.   thanks!

:robot: this is your friendly neighborhood build bot announcing test build <IP_ADDRESS>98 ("make logging easier to trace")
Install in Zotero by downloading test build <IP_ADDRESS>98, opening the Zotero "Tools" menu, selecting "Add-ons", open the gear menu in the top right, and select "Install Add-on From File...".

4698 doesn't change anything but will make it easier for me to trace the logs (which are pretty big in your case). Can you:

Enable debug logging as earlier
Export as you would normally
Export again as you would normally (since this should hit the cache)
Right-click the group or library you were exporting, and select "Send BBT debug log" (it looks like JR75IZ53-euc was sent from the Help menu, which does something slightly different -- in the debug log popup, you should find a "References" tab). While we're at it, check the "Send cache" checkbox. Maybe it will give me some clue as to what is happening.

Between 2. and 3. I should be able to find either why the cache is being dropped, or if not, hopefully some indication of why it's not being used.

It's not structurally broken, at least, because this test passes:
When I import 15120 references from "export/*.json" : ✓ in 1403.217s
Then an export using "Better BibTeX" should match "export/*.bibtex" : ✓ in 285.561s
And should take less than 300 seconds : ✓ in 0.000s
And an export using "Better BibTeX" should match "export/*.bibtex" : ✓ in 27.523s
And should take less than 50 seconds : ✓ in 0.000s

(with times tested against the line above, so first-export is 285.561s uncached,, and second export hits the cache and takes 27.523s)

done: D1162101057

That's a Zotero debug log, I can't access those (that dialog also won't have had the send cache checkbox)

doh.  IQ3KKDEL-euc

:robot: this is your friendly neighborhood build bot announcing test build <IP_ADDRESS>03 ("logging")
Install in Zotero by downloading test build <IP_ADDRESS>03, opening the Zotero "Tools" menu, selecting "Add-ons", open the gear menu in the top right, and select "Install Add-on From File...".

In that log I see that most times you are actually hitting the cache, but the 2nd export is actually slower than the first. Can you get me a new log from 4703?
I'm pretty sure IQ3KKDEL-euc was sent from the Help menu. It helps, but it would be very helpful if you could again:

Enable debug logging as earlier
Export as you would normally
Export again as you would normally (since this should hit the cache)
Right-click the group or library you were exporting, and select "Send BBT debug log", making sure you see a "References" tab, and check the "Send cache" checkbox

Step 4 is likely different from what you did for IQ3KKDEL-euc -- when you submit from the Help menu, you won't see the References tab in the debug submission screen.

For step 4 I mean I'd prefer the log to be sent from the right-click menu itself as highlighted in blue here.

DW78DW6G-euc

:robot: this is your friendly neighborhood build bot announcing test build <IP_ADDRESS>06 ("more logging")
Install in Zotero by downloading test build <IP_ADDRESS>06, opening the Zotero "Tools" menu, selecting "Add-ons", open the gear menu in the top right, and select "Install Add-on From File...".

(forget 4706 if you saw that come by)
Super, I have the references now (although if you did check "send cache", that functionality seems broken for you, it was not included in the log).
I didn't include all the logging I needed so a new build is underway, but in the interim -- exporting 24k items in 20 seconds would mean that each item needs to be handled in less than a millisecond (including cache retrieval and writeout) and I don't think I've ever hit that. The (entirely non-scientific) tests I did long ago put cached writeouts at 3 milliseconds, and I don't think BBT has gotten 3 times faster since.
In my own tests, the cache still makes a considerable difference (around an order of magnitude) but that still makes for 1.9ms per item on my fairly fast machine. In your case however I'm seeing something I really did not expect:
Better BibTeX export took 549618, cache: {"hits":0,"misses":24352}
...
Better BibTeX export took 437542, cache: {"hits":24352,"misses":0}

which would mean that for you, fully cached exports are only barely faster (a factor 1.2 rather than a factor 10) which I can't yet explain. A new logging build is building and will be announced soon, I'll need a new 4-step dance from you, because I can't replicate this on my end.

thanks!  I didn't click "send cache" -- will do so next time.  I'm just curious overall about what scope of computation is going on that takes such a long time per item?  It seems much longer overall than one would expect for transforming strings and writing them out to a file -- is it going over the network or something like that?

It's not going over the network, but each item needs to be found in the cache according to (currently) 9 criteria -- even indexed, that takes time, and then there's the fact that strings in JavaScript are immutable, so assembling text also takes time. There's also some things I cannot escape -- Zotero hands me the items to export one-by-one, and even though I've also cached the Zotero DB-to-export-object transformation that would usually happen at this point, all this interaction between Zotero and the translator takes time (which is included in the 1.8ms I'm currently clocking per cached item).
But that's if I'm not transforming the strings. Converting Zotero items to bibtex is dog slow because the models for items don't match so I have to make lots of choices, and if you're doing transcoding of unicode to LaTeX (which you are), I have to do a lookup for every single character and assemble those into strings. Stock Zotero BibTeX is lots faster probably mainly because their lookup table is smaller, and they make less choices than I do.
That said, on my machine 24k items should be doable in under a minute.

(as things stand currently, you don't need to send the cache, the log shows that the items are being found in the cache successfully)

:robot: this is your friendly neighborhood build bot announcing test build <IP_ADDRESS>07 ("more logging")
Install in Zotero by downloading test build <IP_ADDRESS>07, opening the Zotero "Tools" menu, selecting "Add-ons", open the gear menu in the top right, and select "Install Add-on From File...".

Can you get me the 4-step again with 4707? No need to send cache.
To get an idea of the lower bound of export time, you can export to BetterBibTeX JSON. That does almost nothing to the items it exports, using no export caching (but it will use Zotero DB-to-export-object transformation cache). It will be anywhere between extremely hard and impossible to make a translator faster than that (and without BBT it would usually be slower since you the serialization that's cached by the Zotero DB-to-export-object transformation cache is pretty slow).

It literally takes over an hour to load your references in. Would you be OK with giving me a copy of your Zotero databases? That would be

zotero/zotero.sqlite
zotero/better-bibtex.sqlite

make sure that Zotero is not running when taking a copy. If you see a file with journal in the name, Zotero is still running.

SXTXJ4G8-euc -- would it still be useful to send you my raw db's?

takes about 24 sec to export in json format.  sent you my db's

The raw DBs would have saved me an hour of loading, but my test suite runs on the assumption that items are in the user library, and I haven't found a way to safely (and quickly) move everything from a group to the user library. I'm loading SXTXJ4G8-euc and I can save the DB after that is finished.

The raw DB you sent has 24354, where SXTXJ4G8-euc has 24352. For my tests I need them to be the same. I'll try to find out which those two are, but if you could send me a new raw-DB/debug log pair that'd be really helpful.

people are constantly adding things!  why does it have to be the same?

I'll try to find the mismatch. It's just how my tests are set up currently -- I do cross checks to make sure my tests give consistent results.

I've found the mismatch, but I still can't replicate the problem -- on my system, first export is 333 seconds, 2nd export is 53 seconds.

Looking at my logs, it looks like my export is doing the same activities as yours, yours just takes massively longer (2.14ms/item vs 17.55ms/item)  for the cached export. I'm trying to think of reasons for this, but the only immediate difference I see is that you have Zotfile installed, and I can't think of a single reason why that should make a difference. How much memory does your system have? If you're running out of memory and dipping into the swap that would explain everything, but it doesn't seem likely to me you'd try to have such a massive library on a memory-constrained system.

I have 16gb RAM and its a late 2016 MacBook Pro, 4 CPU / 8 HT system.  Is there anything like a profiler for this code?

16GB is what I have, so that's not it. I'll try to think of another cause... puzzling though.

There's at least one thing that's different -- your logs shows Running handler 0 for itemDone for every item saved, and that whatever happens there takes the bulk of the cached export time -- between that and the time I get my next item to export, 16 - 20 ms passes, and in your case, that adds up to a lot. I don't yet know why this is happening, but it's not something BBT does. I'll try the Zotero forums.

I just tried doing exports with all add on's disabled except for BBT, with no difference in performance unfortunately :(

is it possible that there is some difference in a group library vs. an individual user library?  perhaps some additional sync event that is taking place when an item is done?  you said you only test on user library not group?

Can you get me a new log with all other extensions removed?
I should hope group vs user lib makes no difference, but can you share that group to zotero user bbt1287? I really don't want to have that large a group in my main profile, zotero starts up incredibly slow with that many items.

It's not THAT bad :)  anyway, I shared it.  in the course of turning on / off extentions etc it updated to a current version of BBT, not the test versions you had me using before -- is that OK for the debug log?

It's prohibitively bad for me - startup takes close to a minute, and I restart often (once every test run, and I run a lot of tests).

can you test this group thing in a separate account or something like that?

That's what the bbt1287 account is for. I can start Zotero with a new profile just for this test and sync with that account.

Wow sync has been running for two hours straight and still not done.

OK so I'm also seeing Running handler 0 for itemDone but it's taking a much more friendly 2-4ms, whatever it's doing.
I've posted on zotero-dev to see if they can give some insights on what's happening, but as far as I can tell right now, most of the difference between your experience and mine is that the itemDone handler is doing something heavy on your end -- if that's true, then it'd be out of my reach.

Any ideas about what that ItemDone handler is all about?  I did some googling and it appears to be perhaps connected to PDF's - did you end up with all our PDF's in your massive sync, or not?  maybe it is doing something expensive on every PDF attachment?  I've got my setup configured to download all PDF's..

I'm seeing the same slowdown on a new run now, and that isn't in itemDone but in my cache fetch. Hopefully this is reproducible in tests, maybe I'm not hitting an index.

Can you give 4786 a spin? First export will be slow because installing a new BBT drops the cache.

You've fixed it!!  super fast the second time!  758UBJS8-euc

Awesome. New version has been released.
What it turned out to be was not the cache fetch, but the cache touch. Never would have thought.
Sorry for the long diagnosis and needing so many logs. I don't meet many libraries this big. I've added it as a test now so a regression should be spotted in the nightly test.


Fixed test for index.html

The previous test was not complete, which is why it passed even though the actual implementation was broken.

Ready for merge.

well spot! thanks!


Added Spinner on click of Export to Slides

Leela, your CLA is being rejected. Looking at the error, you've signed with the email<EMAIL_ADDRESS>-- fill out a CLA for this email or change your git setup to sign with whatever email you've used in the past. If you have to delete and recreate this PR that's fine.


Failure to fetch a test: fatal: reference is not a tree: 44c4453cf310432519bcd9e399c34c163742096a

Some repo is gone?
:info:build fatal: reference is not a tree: 44c4453cf310432519bcd9e399c34c163742096a
:info:build CMake Error at /opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_devel_iv/iv/work/build/tools/download-test262-prefix/tmp/download-test262-gitclone.cmake:49 (message):
:info:build   Failed to checkout tag: '44c4453cf310432519bcd9e399c34c163742096a'
:info:build make[2]: *** [tools/download-test262-prefix/src/download-test262-stamp/download-test262-download] Error 1

Ah, this fails:
ExternalProject_Add(
    download-test262
    GIT_REPOSITORY https://github.com/Constellation/test262.git
    GIT_TAG 44c4453cf310432519bcd9e399c34c163742096a
    TIMEOUT 10
    # Disable install step
    UPDATE_COMMAND ""
    PATCH_COMMAND ""
    BUILD_COMMAND ""
    CONFIGURE_COMMAND ""
    INSTALL_COMMAND ""
    # Wrap download, configure and build steps in a script to log output
    LOG_DOWNLOAD OFF
    LOG_CONFIGURE OFF
    LOG_BUILD OFF)


erlang: migrate to wxGTK32

Description of changes

Things done

Built on platform(s)

[ ] x86_64-linux
[x] aarch64-linux
[ ] x86_64-darwin
[x] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.11 Release Notes (or backporting 22.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

Thank you!

This looks good to me, did you leave it as draft for any reason ?

erlangR24 and erlangR25 didn't build on x86_64-darwin before, so I made this a draft before fixing that.
@ofborg build erlangR23 erlangR24 erlangR25

Now ready for review.

Feel free to ping me if you ever need a review, thanks again for the contribution !


Fix center and add pointRadius, innerRef

:boom: Breaking Changes
None.
:rocket: Enhancements
Added pointRadius and fixed center.
:memo: Documentation

:bug: Bug Fix
Fixed center typo.
:house: Internal

LGTM thanks!


Update Electron to 11.2.1 for AVIF images support in HTML

PR Checklist

Please check all that apply to this PR using "x":

[x] I have checked that this PR is not a duplicate of an existing PR (open, closed or merged)
[x] I have checked that this PR does not introduce a breaking change
[ ] This PR introduces breaking changes and I have provided a detailed explanation below

PR Type
What kind of change does this PR introduce?

[ ] Bugfix
[x] Feature
[ ] Code style update (formatting)
[ ] Refactoring (no functional changes)
[ ] Documentation changes
[ ] Other - Please describe:

We will look at this but it might take some time to ensure that there are no major breaking changes between version 9 and version 11

We will look at this but it might take some time to ensure that there are no major breaking changes between version 9 and version 11

@togekk1 I am going to close this just because I don't think we will get to testing this anytime soon. When we do update electron I will make sure to check on that issue to make sure it is resolved.

Got it. Thanks for the hard work!


chart: use timescaledb 2.6.1 on postgres 14

This PR is a fall-back if https://github.com/timescale/timescaledb-kubernetes/pull/360 won't get merged.

https://github.com/timescale/timescaledb-kubernetes/pull/360 was merged. Leaving this until new version of timescale/timescaledb-kubernetes is released


Suggestion

Maybe add a Page to the start that shows all the class templates that you can type out if you need make something up. like how it says class='pageNumber' or any of those, have it on the front page before you click start my own page.

Not sure I'm following what you mean. Don't the snippets fit this need?


GH Actions: update for the release of PHP 8.3

... which is expected later today.

Thank you, as usual.

@gmazzap You're welcome. Might help to merge it ?


Upcase ENV names, and allow whitespace in ENV values

Hi,

Uppercase ENV/config vars on Heroku to follow Unix (and Heroku) convention regardless of the case used in the config file.

Allow whitespace in ENV/config var values.

(And a little cleanup commit).

It's a convention, which can be broken (ugly as that may be). I'm reluctant to force it. What about a warning?

Hi Ken,
Just a quick clarification, the uppercase only shows up in the Heroku environment, the config file stays lowercase. This matches the convention on both sides of the fence (yaml keys tend to be lowercase, Unix env vars tend to be uppercase).
What warning were you thinking of?

There's no requirement that env vars be uppercase (I tested this on heroku) and case is still significant. So, while it would break a convention, it's still permitted. I'm reluctant to enforce the convention to the point where it couldn't be overridden.
Sent from my iPhone ~<PHONE_NUMBER>
On Feb 28, 2012, at 11:21 PM, Les<EMAIL_ADDRESS>wrote:

Hi Ken,
Just a quick clarification, the uppercase only shows up in the Heroku environment, the config file stays lowercase. This matches the convention on both sides of the fence (yaml keys tend to be lowercase, Unix env vars tend to be uppercase).
What warning were you thinking of?

Reply to this email directly or view it on GitHub:
https://github.com/fastestforward/heroku_san/pull/57#issuecomment-4233018

Bug fix #64 handles whitespace / special char escaping.
Thanks for the pull request, but I'd like to wait and see if the case folding is a big issue with other users. Also, since the 2.0 update, the library api has changed dramatically and your pull request will no longer apply cleanly.


Behavior for day checking is incorrect

The docs for crontab (https://man7.org/linux/man-pages/man5/crontab.5.html) say this:

Commands are executed by cron(8) when the 'minute', 'hour', and 'month of the year' fields match the current time, and at least one of the two 'day' fields ('day of month', or 'day of week') match the current time (see "Note" below).

and here's the Note:

Note: The day of a command's execution can be specified in the following two fields — 'day of month', and 'day of week'.  If both fields are restricted (i.e., do not contain the "*" character), the command will be run when either field matches the current time.  For example, "30 4 1,15 * 5" would cause a command to be run at 4:30 am on the 1st and 15th of each month, plus every Friday.

However, the pycron functionality is different - it requires that both the fields match:
>>> import pycron
>>> from datetime import datetime
>>> dt = datetime(2022, 6, 23)  # A Thursday
>>> pycron.is_now('* * 22 * thu', dt)
False
>>> pycron.is_now('* * 23 * thu', dt)
True
>>> pycron.is_now('* * 23 * fri', dt)
False

@kenahoo you could use https://github.com/kiorky/croniter?tab=readme-ov-file#testing-if-a-date-matches-a-crontab in the mean time


Replaceable base class in MediaFix MSL 3.1 branch

Reported by per.ostlund on 7 Apr 2011 09:03 UTC
The Modelica.Media.IdealGases.Common.FluidData.FluidConstants record in the MediaFix branch extends from Modelica.Media.Interfaces.PartialMedium.FluidConstants, which is replaceable. Section 7.1.4 in the Modelica 3.2 specification says that base classes must be transitively non-replaceable, which I understand means that it should not contain any replaceable elements on any level. However, I can not find any reason why the base class itself should not be allowed to be replaceable, but since Dymola gives a warning when they are I assume there is such a reason?
If base classes are not allowed to be replaceable then the MediaFix branch needs to be fixed. If they are allowed it might be a good idea to add this to the 7.1.4 section in the specification, to make it clearer what is and isn't allowed.
P.S. There's no correct correct option to select for Version when adding a new issue if the issue concerns a branch of the MSL. Perhaps some generic branch option should be added?

Migrated-From: https://trac.modelica.org/Modelica/ticket/523

Comment by sjoelund.se on 7 Apr 2011 09:05 UTC
Unhiding ticket so the submitter can view it

Comment by per.ostlund on 8 Apr 2011 12:06 UTC
I found out that it's illegal for a base class to be replaceable in a normal extends, and in retrospect this is made clear in the specification. So the MediaFix branch needs to be fixed, since it's not legal for Modelica.Media.IdealGases.Common.FluidData.FluidConstants to extend Modelica.Media.Interfaces.PartialMedium.FluidConstants.

Comment by hubertus on 8 Apr 2011 12:35 UTC
I don't think Modelica.Media.IdealGases.Common.FluidData.FluidConstants needs to be replaceable, this is just left from teh earlier design. Can you remove the replaceable attribute and check if it works for you then?

Comment by per.ostlund on 8 Apr 2011 12:38 UTC
Replying to [comment:3 hubertus]:

I don't think Modelica.Media.IdealGases.Common.FluidData.FluidConstants needs to be replaceable, this is just left from teh earlier design. Can you remove the replaceable attribute and check if it works for you then?

No, Modelica.Media.IdealGases.Common.FluidData.FluidConstants is not replaceable. It's Modelica.Media.Interfaces.PartialMedium.FluidConstants that is replaceable, which FluidData.FluidConstants extends.

Comment by hubertus on 8 Apr 2011 12:39 UTC
Well, than remove the replaceable attribute there.

Comment by anonymous on 8 Apr 2011 12:42 UTC
Replying to [comment:5 hubertus]:

Well, than remove the replaceable attribute there.

PartialMedium.FluidConstants needs to be replaceable, since other classes extends from PartialMedium and extends FluidConstants with class extends.

Comment by hubertus on 8 Apr 2011 14:19 UTC
Ok, I'll look into that. I think this is easy to fix, so if you want to go ahead and see how far you get with the fix suggested, please do so. I will look at this when I find the time.

Comment by hubertus on 8 Apr 2011 22:08 UTC
Replying to [comment:2 per.ostlund@…]:

I found out that it's illegal for a base class to be replaceable in a normal extends, and in retrospect this is made clear in the specification. So the MediaFix branch needs to be fixed, since it's not legal for Modelica.Media.IdealGases.Common.FluidData.FluidConstants to extend Modelica.Media.Interfaces.PartialMedium.FluidConstants.

Per, I don't understand where you get that idea that the construct is illegal. If you look at section 7.3.1 (page 74 and following in teh 3.1 specification, which should be used for MSL 3.1) you will find plenty of examples, and the normative text. Maybe some other part of the specification is unclear and forbids something similar, but I think this is a perfectly leagl use of the "redeclare class extends" mechanism

Comment by per.ostlund on 9 Apr 2011 06:49 UTC
Replying to [comment:8 hubertus]:

Replying to [comment:2 per.ostlund@…]:

I found out that it's illegal for a base class to be replaceable in a normal extends, and in retrospect this is made clear in the specification. So the MediaFix branch needs to be fixed, since it's not legal for Modelica.Media.IdealGases.Common.FluidData.FluidConstants to extend Modelica.Media.Interfaces.PartialMedium.FluidConstants.

Per, I don't understand where you get that idea that the construct is illegal. If you look at section 7.3.1 (page 74 and following in teh 3.1 specification, which should be used for MSL 3.1) you will find plenty of examples, and the normative text. Maybe some other part of the specification is unclear and forbids something similar, but I think this is a perfectly leagl use of the "redeclare class extends" mechanism

I'm not talking about class extends. Here's a simplification of how the classes in question looks like:
package Modelica
  package Media
    package IdealGases
      package Common
        package FluidData
          record FluidConstants // New in mediafix
            extends Modelica.Media.Interfaces.PartialMedium.FluidConstants; // <= Illegal, since PartialMedium.FluidConstants is replaceable
          end FluidConstants;
        end FluidData;
      end Common;
    end IdealGases;

    package Interfaces
      partial package PartialMedium
        replaceable record FluidConstants end FluidConstants;
      end PartialMedium;
    end Interfaces;
  end Media;
end Modelica;

PartialMedium is usually used by extending from it, and then using class extends to extend the FluidConstants record. The MediaFix library added the FluidConstants record to Modelica.Media.IdealGases.Common.FluidData which extends directly from FluidConstants though, which is illegal since FluidConstants is declared as replaceable (see 7.1.4 and 6.2.1 in the 3.X specification).

Comment by hubertus on 13 Apr 2011 21:22 UTC
The issue is fixed in 0e7249224b31eed334c275fa58ba4e24c8a2ca6f.

Comment by per.ostlund on 14 Apr 2011 08:20 UTC
Replying to [comment:10 hubertus]:

The issue is fixed in 0e7249224b31eed334c275fa58ba4e24c8a2ca6f.

Thank you, it seems to be working better now.

Comment by dietmarw on 14 Apr 2011 11:19 UTC
Hubertus, any way to also merge this fix into trunk?

Comment by hubertus on 14 Apr 2011 11:58 UTC
The point of this branch is to wait for feedback from OpenModelica, and possibly other implementors. "Working better" is not good enough for that to happen. When this works for hopefully several Modelica platforms, I'll merge. Maybe further changes will be needed.

Comment by dietmarw on 14 Apr 2011 12:01 UTC
OK I see, so I'll leave this open then.

Comment by hubertus on 21 Apr 2011 13:32 UTC
I'll close this discussion for now, the same issue is treated in #482 and if the OpenModelica team thinks this is ok I'll merge to trunk and close that ticket as well.

Modified by dietmarw on 24 Nov 2011 12:08 UTC

Modified by dietmarw on 25 Jul 2012 14:52 UTC


Containers not starting when TLS is configured

Rancher Version:

server: v1.1.0-dev4
agent: v1.0.1

Docker Version:

1.11.2

OS and where are the hosts located? (cloud, bare metal, etc):

Red Hat VM

Setup Details: (single node rancher vs. HA rancher, internal DB vs. external DB)

single server node with separate agent node on the same network.

Environment Type: (Cattle/Kubernetes/Swarm/Mesos)

Cattle

Steps to Reproduce:
Set up a master and agent following the directions for using a self signed certificate here
Results:
Started services/containers stuck at initializing and the rancher server container spams logs similiar to:
2016-06-17 17:51:17,028 ERROR [:] [] [] [] [ecutorService-2] [.p.c.v.i.ConfigItemStatusManagerImpl] Failed null, exit code [60] output [INFO: Updating configscripts INFO: Downloading https://hostname/v1//configcontent//configscripts current=configscripts-1-f0f3fb2e1110b5ada7c441705981f93a480313a324294321cff467f0c3e12319 ]
Expected:
Containers to start.
Additional Information:

Everything works as expected when not using tls.

I had some time to set up a SSL environment but didn't have issues launching containers.
Did a network agent container ever start on one of your hosts? Was this an upgraded setup or a fresh setup?

Hey Denise,
A network agent did start on each of the hosts.
This was a fresh setup.
I am going to try to deploy with one of the newer Rancher releases in the near future. I will post my results after I do.

@Beng89 Please re-open if you have the same issues with a newer Rancher release.


Add camper device category

This PR adds a new device category called Camper.
Screenshot of a tracker with that category:

The source is here:
https://www.iconpacks.net/free-icon/camper-van-13375.html
It does not require credit or similar.
I removed the translations

Can you please provide a screen shot of the map with another category next to is, so we can compare.

Here are three in a row:

It doesn't look very consistent with the style. Some very small details. And wheels are in different style.

Alright I will look for a different file. Where do the original icons come from? Maybe that will help to find something.
It needs to be a icon without credit, right?

Yes, it needs to be without credit. I believe the original icons are from Icons8.

Alright so this should be the car file:
https://icons8.de/icon/set/car/material-rounded
And this would be the camper:
https://icons8.de/icon/set/camper/material-rounded
Would that work for you?

I don't think those are links to specific image.

Ah sorry, here is the correct link:
https://icons8.de/icon/93858/wohnwagen-campingplatz
https://icons8.de/icon/85104/auto

Here is the new preview:

Next to the camper I also added a caravan.

Wait. Are we adding two icons now? I think that's a bit of overkill.

A friend that also uses the software for his caravan/trailer asked me if I can add this icon as well. But I can also remove it again

I think one icon is enough.

Done, second icon is removed again

Merged, thanks.


Open in Excel

We have installed BC Spring CU04 No on-prem at client site. The customer i running his own AD and not using Azure AD. When we run the the action "Open in Excel" from the web client it opens the Excel with the data from the page but the user is not be able to use the Refresh button. The same action works from the RTC client and the user can choose to Refresh and the data are refreshed.  Our problem/question is it the correct functionality that the Refresh button is disabled when we run the action from the web client or do we have some configuration issue?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: a5c53847-a111-f746-cc85-af3dcc702f84
Version Independent ID: 06c03c31-a8a9-40e7-2847-bc22cea91344
Content: Setting up the Excel Add-In for Editing Data - Business Central
Content Source: dev-itpro/administration/configuring-excel-addin.md
Service: dynamics365-business-central
GitHub Login: @jswymer
Microsoft Alias: jswymer

Hello,
In the web client, the Open in Excel action does not allow you to push data back into Business Central. If you want this functionality from the web client, you will have to configure the Excel Add-in as described in the linked article. When this add-in is configured, an Edit in Excel action is made available on pages in the web client. With this action you can refresh data from Excel to Business Central.
If the Open in Excel action in the RTC (Windows) client includes a Refresh option, this is probably because you have installed the Excel Add-in that is part of the Business Central installer. However, this particular add-in pertains only to the Windows client. It is a bit confusing I admit. I hope this helps, otherwise let us know.
Thanks for the feedback!

Thank you jswymer,
So there are to different add-ins for the Open in Excel action. One for the Windows Client and one for the Web Client. The Windows client add-in supports Refresh action from Excel. The Web client add-in does not support Refresh action from Excel.


dead link to preview

Hi, the first link on this page redirecting to preview is dead. Thanks!

@ChristianHohlfeld thanks for reporting this issue. Can you share the link to the page that you were on please? Thank you.

I'll go ahead and close out this issue now. Thanks.


Bump T1 dependencies

Updated: nix/t1/_sources/generated.json
nix/t1/_sources/generated.nix

Closed in favor of #643.


Combine models

Hi, I see that you have 5 different models or so: xgboost, lightgbm, RF, LR, SVM.
Why do you not use the combinations of results of these models: average mean or some other weight combination?

@tankz0r we want to build a solid feature space first.
That being said this week we will start adding diversity and since we have already implemented stacking infrastructure. It is super easy to add second level model to do the weighing.
Thanks for noticing it though. It is a great point and I think it is extremely difficult to do well without it.


Control Catalogue Tooltips are not visible or broken

Describe the bug
The tooltip page on the Control Catalogue on Master branch are not displaying right - they're either not visible or broken
To Reproduce
Steps to reproduce the behavior:

Check out latest (2022-11-28T22:08+0800) Master branch
Run the Control Catalogue desktop sample
Navigate to Tooltip demo
Hover your cursor over the tooltip areas
Where are the tooltips?

Expected behavior
Expecting to see tooltips
Screenshots
https://user-images.githubusercontent.com/37165910/204298432-23a46d4d-bb5a-4353-ad53-3c102e81376b.mp4
Desktop (please complete the following information):

OS: Windows 10 Home
Version 10 Bulid 19044

Additional context
Are tooltips broken in current master in general, or is it limited to the control catalogue?

Reproducible for me too.
Last I checked a few days ago, the tooltips appeared. However, I noticed the "moving offset" tooltip was broken. It would just stay in one position.
Something has definitely been broken recently with tooltips.

Instead of simply wiping all control themes and styles that are applied to a control, we can now just remove the `ValueFrame`s which relate to the control theme that was changed.

To do this, added `ValueFrame.FramePriority` which encodes both the  `BindingPriority` and source of the frame (style, control theme, templated parent control theme).

src/Avalonia.Base/Layout/Layoutable.cs             |   7 +
src/Avalonia.Base/PropertyStore/FramePriority.cs   |  36 ++++
.../PropertyStore/ImmediateValueFrame.cs           |   2 +-
src/Avalonia.Base/PropertyStore/ValueFrame.cs      |  20 +-
src/Avalonia.Base/PropertyStore/ValueStore.cs      |  33 +++-
src/Avalonia.Base/StyledElement.cs                 |  95 +++++----
src/Avalonia.Base/Styling/ControlTheme.cs          |   7 +-
src/Avalonia.Base/Styling/Style.cs                 |   5 +-
src/Avalonia.Base/Styling/StyleBase.cs             |   6 +-
src/Avalonia.Base/Styling/StyleInstance.cs         |  13 +-
src/Avalonia.Base/Styling/Styles.cs                |   3 +-
.../Primitives/TemplatedControl.cs                 |  59 ++----
.../Animation/AnimatableTests.cs                   |   3 +-
.../PropertyStore/ValueStoreTests_Frames.cs        |   2 +-
.../Avalonia.Base.UnitTests/Styling/SetterTests.cs |   3 +-
.../Avalonia.Base.UnitTests/Styling/StyleTests.cs  |  17 +-
.../Styling/StyledElementTests_Theming.cs          | 218 +++++++++++++++++++--
.../Styling/Style_Activation.cs                    |   3 +-
tests/Avalonia.Benchmarks/Styling/Style_Apply.cs   |   3 +-
.../Styling/Style_ClassSelector.cs                 |   7 +-
.../Avalonia.Benchmarks/Styling/Style_NonActive.cs |   3 +-
tests/Avalonia.Markup.Xaml.UnitTests/StyleTests.cs |   3 +-
22 files changed, 410 insertions(+), 138 deletions(-)
create mode 100644 src/Avalonia.Base/PropertyStore/FramePriority.cs
Need to figure out why this broke the tooltips. 🤔

@grokys What a coincidence, apparently your commit broke tooltips. 😅


There are serious autoLayout issues with this calendar

I'm using 1.2.8 with autoLayout. The only way I can get it to work is to comment out the constraint on line 239 in CVCalendarView:
if !found {
    addConstraint(NSLayoutConstraint(item: self, attribute: .Height, relatedBy: .Equal, toItem: nil, attribute: .Height, multiplier: 1, constant: frame.height))
}

I'm placing the Calendar in a "header" section of my viewController. If I leave this line in, Xcode complains about being unable to satisfy all constraints, no matter what height / width I give the control. The result is the view and the menu are out of alignment.
The alternative is I now have to specify a fixed width for both and centre them. Which is annoying for bigger / smaller phones.

any update on autolayout support?

No no update so far, I won't have the time to fix this some time soon so if you urgently require this feature, you might be better off with alternative calendar solutions.


Test cleanup: Run mock blob store server in separate thread

Prevents accidental deadlocks in code trying to run blob uploads from blocking test calls, e.g.  container_entrypoint.main()

@prbot approve


Getter/Setter wrong after last Update

Since the last Update to Version <IP_ADDRESS> the getter and setter are wrong.
I have Attributes like this:
private double dValue;

Lombok generates this Getters and Setters:
public double getDValue();
public void setDValue(double dValue);

The last Version of this Plugin did the same.
Since the Update the Plugin shows this Getters and Setters which are wrong:
public double getdValue();
public void setdValue(double dValue);

This causes auto completion to not correctly work and IntelliJ to mark all Getter and Setter Calls as wrong (Method not exist).

This will be fixed in the next release (0.9.4)


Release Google.Cloud.PubSub.V1 version 3.11.0

Changes in this release:
New features

Added TopicName and SubsciptionName resolution from container as an extension point. (issue 12215) (commit 73c457d)

Created release for Google.Cloud.PubSub.V1-3.11.0


Accessibility test pass: Grid viewer

High contrast white and black
Large fonts
Keyboard access (sorting, navigation)
Screen reader (reads row and column names and the cell content)

It is not high contrast aware currently. Grid lines and text are always black, so on black background nothing is visible.

I thought it works in dark theme

I mean the high contrast Windows theme:

This is a 1000x1000 matrix. You can see that it's actually loaded by the scrollbars. But the view is all black, I'm assuming because it's black text/grid on black background:

Looks ok on Win 10, high contrast mode activated via ALT+LEFT SHIFT+PRINT SCREEN

#3143 - I can't repro on Win10 and styling in XAML seem to be default/VS.

https://github.com/Microsoft/RTVS/issues/3173 primarily. High contrast should be working now.


Change database name in  create_database script for mysql to upper case

Make changes to database or schema for all the DB's to have consistent case, i.e. while executing the scripts on Ubuntu for Mysql create_tables.sql fail as the database name in the script is specified as upper case and where as in create_database script we have created the DB with lower case.
Can we check for all other scripts and DB to have database and schema to be consistence naming.

Just realized its enough to make the Mysql database to be case-in-sensitive.
Can we have a note of this some where so that people get to know about this issue or make all the scripts in one case.

Mahesh, we've just pushed out a release that includes consistent database schema names across all scripts.
Thanks for bringing this to our attention!

:thumbsup: @fspinnenhirn

One change was missing (Id to ID):
$ diff /tmp/create_user.sql create_user.sql
29c29
< GRANT SELECT, INSERT, UPDATE, DELETE ON OACCDB.OAC_ResourceExternalId TO `oaccuser`@`localhost`;
---
> GRANT SELECT, INSERT, UPDATE, DELETE ON OACCDB.OAC_ResourceExternalID TO `oaccuser`@`localhost`;

This is with oacc-db-2.0.0-rc.7 and MySQL 5.7.


tocpp feature support d3d12

Any ideas or plan to convert gfxr capture file to cpp project?
Seems tocpp tools only support Vk now.

That's not currently in our roadmap.  Work required would be a Dx12CppConsumer and filling in the tocpp scaffolding for Windows.


map_coordinates should fail with one dimensional coordinates

jax.scipy.ndimage.map_coordinates behaves differently than scipy.ndimage.map_coordinates:
src = jnp.array([5, 1, 6])  # Fails!
# src = jnp.array([5, 1, 5]) # Doesn't fail

# This fails cause SCIPY map_coordinates doesn't support a list as args. Needs a rank 2 input.
coords = [2]      # Fails on scipy! Should also fail on JAX
#coords = [[2]] # Works!

def scipy_map_coordinates():
  return ndimage.map_coordinates(src, coords, order=1, mode="constant")

def jax_map_coordinates():
  return jax.scipy.ndimage.map_coordinates(
      src, coords, order=1, mode="constant")

print(jax_map_coordinates())
print(scipy_map_coordinates())

assert jnp.array_equal(scipy_map_coordinates(), jax_map_coordinates())

Jax's map_coordinates should fail in the same way the scipy version does if we want to keep the API as similar as possible.
CC: @claudiofantacci

To me, this looks like a SciPy bug. Logically, the behavior of map_coordinates is perfectly well defined for one dimensional coordinates -- returning a scalar, just like indexing -- and indeed that is exactly what JAX does, without any special case logic.
I'm not sure why it is important to replicate this behavior exactly in JAX. We do try to make JAX drop-in compatible with NumPy/SciPy, but we also add additional functionality in some cases where NumPy/SciPy would raise an error (e.g., allowing for new dtypes and new optional arguments).

I agree with you in this one. I don't know how is the process here, should we wait for somebody else to pitch in and give more opinions, or should we close this?

If we're agreed, I think we can close this. Thanks for bringing up the issue!


Please support http2 between envoys as a first class citizen

Describe the feature request
On our cluster, http2 can make a real difference.  This graph demonstrates the following service chain:
consumer-gateway -> sauron-web -> search-one -> solr
It shows p99 response times from the perspective of consumer-gateway, as you can see the notable improvement when an EnvoyFilter is applied to upgrade the connections to http2.

This is a particular "high connection churn" set of services, on an mtls cluster, so I'm correlating the notable drop to mtls handshakes in the connection churn, as well as reduced load on proxies.
It feels to me like if Istio is really prioritising being performant, making http2 the default between envoys is frankly a no brainer.  At the least, properly supporting this through a DestinationRule configuration item to make it easily accessible.
Describe alternatives you've considered
As we deploy some 350 applications, in 350 namespaces, the only way for us to apply this is to create an EnvoyFilter in the istio-system namespace for each service, targeting its cluster.  It is also difficult to do as our application pipelines do not have permission to deploy resources into the istio-system namespace, so we'd be introducing a security issue to work around this.
We could create a controller, which watches for istio-enabled deployments and mutates an EnvoyFilter with all permutations of istio-enabled EDS upstreams.  This works around the deployment issue, but it feels dirty, and that EnvoyFilter will be huge, and i'm concerned about pilot discovery config churn.
We cannot create it globally, and apply it without a service match as we have a handful of services that are non-http2 compliant so the requests to those will fail.  If EnvoyFilter allowed me to match on EDS, and exclude ORIGINAL_DST, that may well give me a work around - but currently it doesn't.
I'm also extremely worried about prolific use of EnvoyFilters in production considering the extremely low level nature of the API.  Especially creating 100's of them.
Additional Information
This is an example of an EnvoyFilter I am using to enable h2:
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  namespace: istio-system
  name: h2-platform-testing
spec:
  configPatches:
  - applyTo: CLUSTER
    match:
      context: SIDECAR_OUTBOUND
      cluster:
        service: app.platform-testing.svc.cluster.local
    patch:
      operation: MERGE
      value:
        http2_protocol_options: { max_concurrent_streams: 47707 }

cc @mandarjog @douglas-reid @howardjohn

Additional note: if your service uses websockets, this completely explodes.

cc: @louiscryan

This is now available in the DestinationRule, closing.


[368] Add delete user to support

Context

This trello ticket

Changes proposed in this pull request

Add delete user functionality to support console

Guidance to review

Go to support console
select user tab
select a user by clicking their email address
scroll to bottom of page and click delete this user

Checklist

[x] Make sure all information from the Trello card is in here
[x] Attach to Trello card
[x] Rebased master
[x] Cleaned commit history
[x] Tested by running locally

 Allow users to be deleted in the support UI


Cleanup template generation

https://github.com/eliwoods/create-dash-app/blob/master/create_dash_app/templates/root.py#L5
https://github.com/eliwoods/create-dash-app/blob/master/create_dash_app/templates/components.py#L6
https://github.com/eliwoods/create-dash-app/blob/master/create_dash_app/templates/root.py#L16

Addressed in #4


Cilium 1.11.0 does not start on EKS with pod subnet-filter

Is there an existing issue for this?

[X] I have searched the existing issues

What happened?
I am installing Cilium 1.11.0 on a brand new EKS cluster on nodes that did not previously have a CNI installed. The cilium-agents do not become ready. They create CiliumNode resources. However, the cilium-operator does not allocate ENIs/IPs to the nodes. As a result the nodes don't become ready. The VPCs primary and secondary CIDRs don't seem to be discovered either.
Cilium Version
1.11.0
Kernel Version
5.4.156-83.273.amzn2.x86_64
Kubernetes Version
v1.21.2-eks-06eac09
Sysdump
I don't feel comfortable sharing this in a public github issue. Please reach out and I can share this privately.
Relevant log output
No response
Anything else?
Installing cilium 1.10.5 with the same values.yaml works well.
EDIT: Just tested, the same values also work with 1.10.6. So the problem is unlikely to be present in any of the fixes backported to 1.10
Collected log files:
cilium-1-11-0.zip
Code of Conduct

[x] I agree to follow this project's Code of Conduct

Interestingly enough I can upgrade Cilium on existing nodes from 0.10 to 0.11. However, new nodes don't become ready with 0.11.

I am installing Cilium 0.11.0

I guess you meant Cilium 1.11.0? We've had stable releases for years now :sweat_smile:
In the cilium-operator logs, I'm seeing a lot of warnings for different nodes:
level=warning msg="Instance not found! Please delete corresponding ciliumnode if instance has already been deleted." instanceID=i-02ad8b4c55e4a7c08 name=ip-10-28-92-98.eu-west-1.compute.internal subsys=ipam

It's probably because the operator can't find the nodes that it is not allocating ENIs and IP addresses.
Could you please share the ciliumnodes-xxx.yaml and k8s-nodes-xxx.yaml files from the sysdump?

Yes, this relates to 1.11.0 :') I will reproduce the issue again tomorrow and attach the ciliumnodes and k8s-nodes

I've attached the files.
cilium-sysdump-20211215-113339.zip

i face the same problem when configure filter subnets by id or tags

@mhulscher @azharprabudi Could you share the exact reproduction steps if you have them?

I am facing this issue in 1.10.6
eni portion helm values.yaml
eni:
  # -- Enable Elastic Network Interface (ENI) integration.
  enabled: true
  # -- Update ENI Adapter limits from the EC2 API
  updateEC2AdapterLimitViaAPI: true
  # -- Release IPs not used from the ENI
  awsReleaseExcessIPs: true
  # -- EC2 API endpoint to use
  ec2APIEndpoint: ""
  # -- Tags to apply to the newly created ENIs
  eniTags: {}
  # -- If using IAM role for Service Accounts will not try to
  # inject identity values from cilium-aws kubernetes secret.
  # Adds annotation to service account if managed by Helm.
  # See https://github.com/aws/amazon-eks-pod-identity-webhook
  iamRole: "arn:aws:iam::xxxx:role/cilium_operator_serviceaccount_role"
  # -- Filter via subnet IDs which will dictate which subnets are going to be used to create new ENIs
  # subnetIDsFilter: ""
  subnetIDsFilter: "subnet-097xxxxxfabc3,subnet-0f7cbxxxa4d3d,subnet-0892xxx003d7"
  # -- Filter via tags (k=v) which will dictate which subnets are going to be used to create new ENIs
  # subnetTagsFilter: function=eks
  subnetTagsFilter: ""

Logs: in operator showing similar logs
level=info msg="Synchronized ENI information" numInstances=0 numSecurityGroups=12 numSubnets=0 numVPCs=1 subsys=eni
level=debug msg="Resetting resyncNeeded" instanceID=i-07a87xxxa424d1 name=ip-100-88-xx-x.ec2.internal subsys=ipam
level=warning msg="Instance not found! Please delete corresponding ciliumnode if instance has already been deleted." instanceID=i-07a87xxx7aa424d1 name=ip-100-88-xx-xx.ec2.internal subsys=ipam
level=warning msg="Instance not found! Please delete corresponding ciliumnode if instance has already been deleted." instanceID=i-07a87xxxa424d1 name=ip-100-88-xx-xx.ec2.internal subsys=ipam
level=debug msg="Resetting resyncNeeded" instanceID=i-00980xxx4890cca name=ip-100-88-xx-xx.ec2.internal subsys=ipam

however when I update value subnetTagsFilter with like below
subnetTagsFilter: function=eks
logs show
level=info msg="Synchronized ENI information" numInstances=11 numSecurityGroups=12 numSubnets=11 numVPCs=1 subsys=eni
Notice the numSubnets
It looks like the operator is filtering correctly based on the parameters that are being passed to it.

Upgraded to 1.11.0
I was able to get the subnetFilter to work, but then the operator could not get the security group.
Notice numSubnet is 3 now.
level=info msg="Synchronized ENI information" numInstances=4 numSecurityGroups=12 numSubnets=3 numVPCs=1 subsys=eni
level=warning msg="Unable to create interface on instance: unable to get security groups failed to get security group ids" instanceID=i-00d45xxxx06d4c name=ip-100-88-xx-xx.ec2.internal subsys=ipam

Might be relevant here: We have discovered a bug in way subnet-tags-filter is parsed, see #18293
The TLDR is that subnetTagsFilter: "foo=bar" does not work when provided via ConfigMap, but you may use a JSON filter as a workaround, i.e. subnetTagsFilter: '{"foo":"bar"}'. We plan to fix that.
But the above bug has been present since the introduction of the subnet-tags-filter flag was added. We have confirmed that the CLI parse bug is alo present in v1.10, therefore I don't understand why some report that the flag works for them in v1.10 - have you been using the JSON syntax all along?

Hi @gandro ,
What works for me in 1.10 is to set eni.subnetTagsFilter: "foo=bar" in the values.yaml. However, I don't set the filter in the CNI ConfigMap.

Which v1.10 version are you using? My understanding is that the eni.subnetTagsFilter will be silently ignored in such a setup due the above bug. But we might be missing something else here.
You can verify that the really filter works by checking that the numSubnets count for the level=info msg="Synchronized ENI information" message in the cilium-operator? It should report the same number of subnets as aws ec2 describe-subnets --filters 'Name=tag:foo,Values=bar' returns.

So I am installing cilium 1.10.6, but this has also worked with 1.10.5.
Given a cilium-config ConfigMap with
  subnet-tags-filter: foo=bar

I can see the following in the cilium-operator logs:
cilium-operator-6df4bbf59c-drkvb cilium-operator level=info msg="  --subnet-tags-filter='foo=bar'" subsys=cilium-operator-aws

You can verify that the really filter works by checking that the numSubnets count for the level=info msg="Synchronized ENI information" message in the cilium-operator? It should report the same number of subnets as aws ec2 describe-subnets --filters 'Name=tag:foo,Values=bar' returns.

Interestingly, the operator detects 9 subnets.

3x public
3x private
3x pod

cilium-operator-6df4bbf59c-drkvb cilium-operator level=info msg="Synchronized ENI information" numInstances=36 numSecurityGroups=10 numSubnets=9 numVPCs=1 subsys=eni

However, I expect the operator to only detect the 3 pod-subnets:
$ aws ec2 describe-subnets --filters=Name=tag:foo,Values=bar | jq '.Subnets | length'
3

The reason why everything still works, is probably because of what you suggested; the subnets that I am targeting have way more free IPs than the other subnets.
Now when I set the following, still on 1.10.6, I see numSubnets=3 🎉
    eni:
      subnetTagsFilter: '{"foo":"bar"}

When I now update cilium-operator to 1.11.0, I see the following in its output:
level=fatal msg="Incorrect config-map flag subnet-tags-filter" error="{\"foo\":\"bar\"} must be formatted as key=value" subsys=config

Now on 1.11.0, I set   subnet-tags-filter: foo=bar and I can see numSubnets=3, still correct.
However, new nodes are not initialized properly by the operator. The operator logs the following when a new node is created:
cilium-operator-7857545c66-ckqgc cilium-operator level=info msg="Discovered new CiliumNode custom resource" name=ip-10-28-83-186.eu-west-1.compute.internal subsys=ipam
cilium-operator-7857545c66-ckqgc cilium-operator level=warning msg="Instance not found! Please delete corresponding ciliumnode if instance has already been deleted." instanceID=i-0045e33f849ad944b name=ip-10-28-83-186.eu-west-1.compute.internal subsys=ipam
cilium-operator-7857545c66-ckqgc cilium-operator level=warning msg="Instance not found! Please delete corresponding ciliumnode if instance has already been deleted." instanceID=i-0045e33f849ad944b name=ip-10-28-83-186.eu-west-1.compute.internal subsys=ipam

I think this is happening here: https://github.com/cilium/cilium/blob/master/pkg/aws/eni/node.go#L478
What could perhaps be going wrong, is that I am also using first-index-interface: 1, like so in the cni-configuration configmap:
{
  "cniVersion": "0.3.1",
  "name": "cilium",
  "plugins": [
    {
      "cniVersion": "0.3.1",
      "eni": {
        "first-interface-index": 1
      },
      "type": "cilium-cni"
    }
  ]
}

The first interface lives inside the private subnet, which does not match the subnet-tags-filter. There are no other ENIs attached; we expect cilium to do that. The ENIs that we expect cilium to create should be placed in the pod subnets, which do match the subnet-tags-filter.

Thanks for the information, that's helpful. I wasn't aware that the k=v syntax throws an error on v1.11.
It seems like we might have two bugs here:

One is the parser issue (which doesn't seem present on v1.11)
The other one is the instance not found error, for which we have not yet established a root cause

cilium-operator-7857545c66-ckqgc cilium-operator level=warning msg="Instance not found! Please delete corresponding ciliumnode if instance has already been deleted." instanceID=i-0045e33f849ad944b name=ip-10-28-83-186.eu-west-1.compute.internal subsys=ipam

Does this instance (ip-10-28-83-186.eu-west-1.compute.internal) actually exist in your cluster? Or is this error indeed caused by a stale CiliumNode resource?

The first interface lives inside the private subnet, which does not match the subnet-tags-filter.
Yes, that is okay. The subnet-tag-filter only applies to ENIs created/managed by Cilium.

There are no other ENIs attached; we expect cilium to do that. The ENIs that we expect cilium to create should be placed in the pod subnets, which do match the subnet-tags-filter.
Yes, I agree that this should be the expected behavior.

Does this instance (ip-10-28-83-186.eu-west-1.compute.internal) actually exist in your cluster? Or is this error indeed caused by a stale CiliumNode resource?

The node actually exists. The operator is throwing these errors because it is triggered by a new CiliumNode, which is created by the cilium-agent running on the newly created k8s node. The error that the node can't be found is thrown when cilium can't discover any ENIs for the node (as per the function that I linked).

Perhaps the issue I am having is case a of PEBKAC, maybe I am misunderstanding the documentation?
My goal is the as follows:

Nodes are created with their first interface in a private subnet
Cilium should discover all subnets in the VPC; it should be aware of all CIDRs in the VPC so that traffic to other subnets is not SNATted
Cilium should assign ENIs and IPs from the pod subnets to pods
Traffic with destination inside the VPC should not be SNATted
Traffic with destination outside the VPC should be SNATted using the node's first interface's primary IP as source IP (I have set bpf.masquerade: true).

So far, with 1.10.x, I have only set eni.subnetTagsFilter and I did not set the eni.subnet-tags in the CNI configuration. We've established that this configuration does not work with 1.11.0.
However, on 1.11.0, if I do set eni.subnet-tags in the CNI configuration and do not set eni.subnetTagsFilter, everything seems to work like I expect it to.
Did I misconfigure Cilium all along?

So far, with 1.10.x, I have only set eni.subnetTagsFilter and I did not set the eni.subnet-tags in the CNI configuration. We've established that this configuration does not work with 1.11.0.
However, on 1.11.0, if I do set eni.subnet-tags in the CNI configuration and do not set eni.subnetTagsFilter, everything seems to work like I expect it to.

Given this information, I do believe might we have a bug around eni.subnetTagsFilter (i.e. the operator --subnet-tags-filter flag) in v1.11 besides the CLI parsing bug already discovered. Because I expect eni.subnetTagsFilter (Helm) and eni.subnet-tags (CNI Config) to be functionally more or less identical.
In other words, both should work, and both filters should be applied (i.e. attached ENIs should have tags matching both filters).
The CNI field has existed for a while, and at least to my knowledge has been used by one large Cilium user using it in production for a while. It is usually what I recommend and what is documented, because it is very flexible (as it can be configured per node).
The global Helm flag filter was added later by an external contributor for what seems to be performance reasons (#10738). I don't think it has had much usage, especially given the fact that bug #18328 has not been uncovered until recently, even though it has been present since the beginning.
TL;DR: I don't think this is just a configuration issue. However, I do believe that the eni.subnet-tags CNI configuration field is likely not affected by the bug reported here.

@gandro Has any progress been made on the second bug described here:
The other one is the instance not found error, for which we have not yet established a root cause

Not from my side. But I'm also not actively working on this at the moment.
Given that have merged the fix for #18328 on master, I think the next step here is to try to reproduce the issue on master to get a clear understanding what the outstanding issues are.

:wave: let me give it a try and inform the result in this issue.

I have performed some test in both master and v1.10.7 releases, please find below details.
In short, while there is a warning in operator logs on instance not found, cilium agent on new node is still managed to start shortly after that, this is for both master build and v1.10.7.
Feel free to let me know if there is anything that I could have missed here.

Installation configuration
$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: cni-configuration
  namespace: kube-system
data:
  cni-config: |-
    {
      "cniVersion":"0.3.1",
      "name":"cilium",
      "plugins": [
        {
          "cniVersion":"0.3.1",
          "type":"cilium-cni",
          "eni": {
            "subnet-tags":{
              "eksctl.cluster.k8s.io/v1alpha1/cluster-name":"tammach-17899"
            },
            "first-interface-index": 1
          }
        }
      ]
    }
EOF

$ cat aws-eni-valus.yaml
operator:
  extraArgs:
    - --aws-release-excess-ips

eni:
  enabled: true
  subnetTagsFilter: eksctl.cluster.k8s.io/v1alpha1/cluster-name=tammach-17899

ipam:
  mode: eni

egressMasqueradeInterfaces: eth1
tunnel: disabled

cni:
  customConf: true
  configMap: cni-configuration
  subnetTagsFilter: eksctl.cluster.k8s.io/v1alpha1/cluster-name=tammach-17899

bpf:
  masquerade: true

Instance addition/removal is handled gracefully in master build
2022-02-11T12:29:16.191827362Z level=info msg="Synchronized ENI information" numInstances=2 numSecurityGroups=5 numSubnets=3 numVPCs=2 subsys=eni
2022-02-11T12:30:16.669623152Z level=info msg="Synchronized ENI information" numInstances=1 numSecurityGroups=5 numSubnets=3 numVPCs=2 subsys=eni
2022-02-11T12:31:17.120613422Z level=info msg="Synchronized ENI information" numInstances=2 numSecurityGroups=5 numSubnets=3 numVPCs=2 subsys=eni
2022-02-11T12:32:06.163236979Z level=info msg="Discovered new CiliumNode custom resource" name=ip-192-168-116-146.ap-southeast-2.compute.internal subsys=ipam

Instance deletion warning, but shortly gracefully handled
2022-02-11T12:59:13.729822732Z level=info msg="Synchronized ENI information" numInstances=1 numSecurityGroups=5 numSubnets=9 numVPCs=2 subsys=eni
2022-02-11T12:59:13.730185570Z level=warning msg="Instance not found! Please delete corresponding ciliumnode if instance has already been deleted." instanceID=i-0bdf82bd61d3c3008 name=ip-192-168-160-163.ap-southeast-2.compute.internal subsys=ipam
2022-02-11T12:59:13.730199828Z level=warning msg="Instance not found! Please delete corresponding ciliumnode if instance has already been deleted." instanceID=i-0bdf82bd61d3c3008 name=ip-192-168-160-163.ap-southeast-2.compute.internal subsys=ipam
2022-02-11T12:59:13.774013593Z level=warning msg="Instance not found! Please delete corresponding ciliumnode if instance has already been deleted." instanceID=i-0bdf82bd61d3c3008 name=ip-192-168-160-163.ap-southeast-2.compute.internal subsys=ipam
2022-02-11T12:59:13.774046199Z level=warning msg="Instance not found! Please delete corresponding ciliumnode if instance has already been deleted." instanceID=i-0bdf82bd61d3c3008 name=ip-192-168-160-163.ap-southeast-2.compute.internal subsys=ipam
2022-02-11T13:00:14.301098157Z level=info msg="Synchronized ENI information" numInstances=2 numSecurityGroups=5 numSubnets=9 numVPCs=2 subsys=eni
2022-02-11T13:01:02.428834858Z level=info msg="Discovered new CiliumNode custom resource" name=ip-192-168-151-204.ap-southeast-2.compute.internal subsys=ipam
2022-02-11T13:01:02.428877188Z level=info msg="Resolving IP deficit of node" available=0 availableForAllocation=0 availableInterfaces=2 instanceID=i-0f4062e9b1fb60b1e maxIPsToAllocate=8 name=ip-192-168-151-204.ap-southeast-2.compute.internal neededIPs=8 remainingInterfaces=2 selectedInterface= selectedPoolID= subsys=ipam used=0

$ cilium status
    /¯¯\
 /¯¯\__/¯¯\    Cilium:         OK
 \__/¯¯\__/    Operator:       OK
 /¯¯\__/¯¯\    Hubble:         disabled
 \__/¯¯\__/    ClusterMesh:    disabled
    \__/

DaemonSet         cilium             Desired: 2, Ready: 2/2, Available: 2/2
Deployment        cilium-operator    Desired: 2, Ready: 2/2, Available: 2/2
Containers:       cilium             Running: 2
                  cilium-operator    Running: 2
Cluster Pods:     2/2 managed by Cilium
Image versions    cilium             quay.io/cilium/cilium:v1.10.7@sha256:e23f55e80e1988db083397987a89967aa204ad6fc32da243b9160fbcea29b0ca: 2
                  cilium-operator    quay.io/cilium/operator-aws:v1.10.7@sha256:97b378e0e3b6b5ade6ae1706024c7a25fe6fc48e00102b65a6b7ac51d6327f40: 2

We also tested the above configuration without the cni section and it also works with the master branch. We expect the upcoming Cilium v1.11.2 to fix the issues reported here.

I have the exact same issue here.
Everything is working on 1.10.5, and I have mixed results with 1.11.1 or 1.11.2 (that I just started to test).
cilium-operator keeps saying:
level=info msg="Synchronized ENI information" numInstances=12 numSecurityGroups=24 numSubnets=3 numVPCs=1 subsys=eni
level=info msg="Resolving IP deficit of node" available=28 availableForAllocation=0 availableInterfaces=2 instanceID=i-0c5a3a0fa068db10f maxIPsToAllocate=3 name=ip-10-253-2-252.us-west-2.compute.internal neededIPs=3 remainingInterfaces=2 selectedInterface= selectedPoolID= subsys=ipam used=23
level=warning msg="Unable to create interface on instance: unable to get security groups failed to get security group ids" instanceID=i-0c5a3a0fa068db10f name=ip-10-253-2-252.us-west-2.compute.internal subsys=ipam
level=warning msg="Unable to maintain ip pool of node" error="unable to get security groups failed to get security group ids" instanceID=i-0c5a3a0fa068db10f name=ip-10-253-2-252.us-west-2.compute.internal subsys=ipam

So it's unable to get security groups failed to get security group ids and it's a warning only... but I have those logs many times per second.
Cilium is working with the current IPs it have defined in the ciliumnode but is not able to grab new / or is just not able o add new interfaces (ENIs) on the nodes...
Instance type is m5.xlarge which should allow 4 ENIs. I have 3 configured, the eth0 is used only by the Node IP (10.253.x.y), then eth1 and eth2 used for pod ip's. No eth3 created.
I do have 14 IPs on each 1 and 2 ENIs, which means they are full.
So why isn't the Operator able to attach a 4th ENI ?
All this testing was done with only one node.
Then I tried adding two other nodes, and I get the same errors about Instance not found.
It seems to be down to the Operator not being able to talk to the AWS API (or get the needed informations from the API)
The CiliumAgent is logging:
level=info msg="API request released by rate limiter" name=endpoint-delete parallelRequests=38 subsys=rate uuid=03642edb-853b-47e8-975c-68fc86f2b37a waitDurationTotal="83.892µs"
level=info msg="API call has been processed" error="endpoint not found" name=endpoint-delete processingDuration="2.839µs" subsys=rate totalDuration="105.912µs" uuid=03642edb-853b-47e8-975c-68fc86f2b37a waitDurationTotal="83.892µs"

So I tried to curl http://<IP_ADDRESS>/latest/meta-data/instance-id from a node: OK
and curl http://<IP_ADDRESS>/latest/meta-data/instance-id from inside a (working) POD: no answer !
Looking at Hubble, it seems the TCP cnx is forwarded to <IP_ADDRESS> (the IMDS Metadata server)
Looking at a tcpdump on the node (eth0) I see:
17:48:01.537330 IP <IP_ADDRESS>.34488 > <IP_ADDRESS>.http: Flags [S], seq<PHONE_NUMBER>, win 62727, options [mss 8961,sackOK,TS val<PHONE_NUMBER> ecr 0,nop,wscale 7], length 0

So the traffic is clearly not flowing.
Again, this is totally working fine with Cilium 1.10.x

I'm using the CNI config as:
  cni-config: |-
    {
      cniVersion":"0.3.1",
      "name": cilium",
      plugins": [
        {
          cniVersion": "0.3.1",
          type": "cilium-cni",
          eni": {
            first-interface-index": 1,
            subnet-tags":{
              "Usage":"pods"
            }
          }
        }
      ]
    }

Note that I also enabled the ip-masq-agent :
IP PREFIX/ADDRESS
<IP_ADDRESS>/8
<IP_ADDRESS>/16
<IP_ADDRESS>/12
<IP_ADDRESS>/16

The node has a direct route as: <IP_ADDRESS> dev eth0
I'm not totally sure how will Cilium eBPF react to that ?

I changed my ip-masq-agent configmap to be:
    nonMasqueradeCIDRs:
    - <IP_ADDRESS>/8
    - <IP_ADDRESS>/12
    - <IP_ADDRESS>/16
    masqLinkLocal: true.  <-------------------

That solved the issue for pod->IMDS issue.
But Cilium pods (using HOstNetwork) are still having issues it seems...

Ok sorry for yesterday's noise.
For me, the problem is down to:
The file /etc/cni/net.d/05-cilium.conf seems to never be created...
Because I'm using a custom CNI config (to select eth1 as first ENI), the postStart script is not creating it:
https://github.com/cilium/cilium/blob/774fc6a59854feb8f51943773345ce9d2e4ab92a/plugins/cilium-cni/cni-install.sh#L84-L91
My cilium-config configmap is setting write-cni-conf-when-ready: /host/etc/cni/net.d/05-cilium.conflist
So the agent should create it when it's started.
This leaves Docker network unconfigured, so cilium-agent is not ready, so the node is not ready, so we have all sorts of errors in cilium-agent and cilium-operator.
What I don't know is if it's the agent waiting on the Operator or something else preventing it to initialize correctly...

It feels like the Cilium Agent is creating the ciliumnode CR, but the Operator is having a problem with it, preventing the agent to finish initialization.
back to
level=info msg="Discovered new CiliumNode custom resource" name=ip-10-253-2-27.us-west-2.compute.internal subsys=ipam
level=warning msg="Instance not found! Please delete corresponding ciliumnode if instance has already been deleted." instanceID=i-079c49efed379138c name=ip-10-253-2-27.us-west-2.compute.internal subsys=ipam
level=warning msg="Instance not found! Please delete corresponding ciliumnode if instance has already been deleted." instanceID=i-079c49efed379138c name=ip-10-253-2-27.us-west-2.compute.internal subsys=ipam

I just can't understand why it's not working but something is preventing the Operator to match the info in the ciliumNode with the data it finds/cache from AWS API....

FYI, I tested and 1.10.7 is also OK. Issue is in 1.10.8+ up to 1.11.2

@prune998 Thank you for your comments, it helped us debug issues we had a bit further, but alas we were stuck one step further in 1.10.8+ than you.
Our situation was that we want to use different subnet(s) for our pods than our worker nodes, for some odd reason when using cilium 1.10.8+ as the CNI for EKS it won't attach an actual ENI to the node to use for pod addresses and the "Instance not found!" error was found in the cilium operators.
When we manually attached an ENI to the node (which contained the correct subnet) the error of instance not found was gone, but in our case it stopped a bit further because it could not find the security groups used by the node (probably to assign it to the pod eni), to be clear this was not a permission denied error, since couldtrail was clean of errors.
Hope this extra information can/might help somebody else or to resolve the actual issue. :)

@anwbtom I can confirm the same kind of behaviour... when a node is upgraded, all the ENIs/IPs already on the node works and can be used. Attaching new ENIS or grabbing new IPs will fail.
Now the questions is how to get someone assigned to this issue ? We need to get more traction here...

Well, in my testcase after the initial working ENI was attached it was able to attach a new ENI as well, but only with a SecurityGroupId pre-configured (so using the default of the node didn't work, since then this message was shown: level=warning msg="Unable to create interface on instance: unable to get security groups failed to get security group ids" instanceID=i-0c5a3a0fa068db10f name=ip-10-253-2-252.us-west-2.compute.internal subsys=ipam
But with (as an example) this cni-config:
apiVersion: v1 data:   cni-config: |     {       "name": "cilium-cni",       "cniVersion": "0.3.1",       "plugins": [{         "cniVersion": "0.3.1",         "type": "cilium-cni",         "eni": {           "subnet-tags": {             "Target": "pod"           },           "vpc-id": "vpc-{my-id}",           "first-interface-index": 1,           "security-groups": ["sg-04bd7f5b9afd61e69"]         }       }]     }
it was able to attach a second one (key being the eni.security-groups bit)

The security group ids attached to the ENI are computed in the following order:

 1. The field ``spec.eni.security-groups`` is consulted first. If this is set
    then these will be the security group ids attached to the newly created ENI.
 2. The filed ``spec.eni.security-group-tags`` is consulted. If this is set then
    the operator will list all security groups in the account and will attach to
    the ENI the ones that match the list of tags passed.
 3. Finally if none of the above fields are set then the newly created ENI will
    inherit the security group ids of ``eth0`` of the machine.

It shouldn't (and wasn't) a problem before 1.10.8.
Also, a diff between 1.10.7 and 1.10.8 does not show any changes in the EKS ipam/eni code... :/

Am not saying its not a bug or that its logical (I don't know to much from the internals of cilium yet).
But the error states its having trouble finding the security group ids: Unable to create interface on instance: unable to get security groups failed to get security group ids.
So by giving it one or more ids (priority 1 according to your attached list), it won't have to search for it, and just apply them. So its more a work-around the issue than actually fixing whatever causes it.

Yeah, i'm testing this scenario right now and trying to add debug... I'm wondering if the issue could be a change in the AWS SDK, as it totally feels it's a mismatch between what we get from AWS, the Cilium Operator cache and what's in the CiliumNode

I added some debug in some functions and It seems to get down to:
func (n *Node) getSecurityGroupIDs(ctx context.Context, eniSpec eniTypes.ENISpec) ([]string, error) {
	// 1. check explicit security groups associations via checking Spec.ENI.SecurityGroups
	// 2. check if Spec.ENI.SecurityGroupTags is passed and if so filter by those
	// 3. if 1 and 2 give no results derive the security groups from eth0

If we don't set 1) or 2), eth0 is used by default.
In that case, we loop each interfaces, hoping to find the number 0:
https://github.com/cilium/cilium/blob/79645bd18043212f3ff69b32cf7245f3644cb098/pkg/aws/eni/node.go#L276-L284
What I found, so far, is that:

the ENIs that a looked up are only the ones created by Cilium, so eth0 is not scanned, so this function will always "fail" (return empty security groups)
e, ok := rev.Resource.(*eniTypes.ENI) always fail, meaning something is not correct with rev.Resource.

I can confirm the behaviour is different with Cilium 1.10.7, where eth0 is part of the list of interfaces.
Now I'm investigating around the updates that lead to this behaviour change....

More research...
I'm trying to get new nodes working.
so far, when a new node is joining the cluster, cilium agent kicks of, create the ciliumNode then the cilium-operator tries to add ENIs/ IPs.
To do that, the Operator look in EC2 for vpcs, subnets, instances. Then this "ec2 client cache" is used to match the instance.
Because of the split network, the new EC2 instance only have one ENI (eth0) and is using the default VPC subnet (not the pod subnet).
Here's the code used to get the instances info :
https://github.com/cilium/cilium/blob/8a8c6ff3afc380072d54004714149c0177e65272/pkg/aws/ec2/ec2.go#L217
In the end, the instance list in the cache does not includes the new instance as the subnets does not matches, leading to Instance not found errors.
Again, I still don't see why it's working with 1.10.7 as nothing changed in there...

So looking at the list of changes between 1.10.7 and 1.10.8, one relevant change that I see is the aforementioned https://github.com/cilium/cilium/pull/18478
@prune998 @anwbtom How are you configuring subnet tag filtering? Only using the CNI ConfigMap, or also using the subnet-tags-filter on cilium-operator? (Helm value: eni.subnetTagsFilter="foo=true")
If you are using the Helm value, does it work for you if you remove it and only rely on the CNI config ConfigMap?

I was adding it on both levels e.g. in the cni-config and the eni.subnetTagsFilter.
Will test it with only setting it on cni-config level, either today otherwise monday and come back to this.
Thanks for all the additional information and feedback!

Thanks. So my suspicion is that eni.subnetTagsFilter is broken for some reason. It likely has been broken for a while, because of bug #18328 it was actually silently ignored in v1.10.7 and older (i.e. it didn't filter out any subnets at all). Now that we fixed  #18328 it seems like this surfaced another bug in our subnet filtering logic.
This needs to be understood and fixed, but a potential workaround is to not use eni.subnetTagsFilter at all and only rely on the CNI config ConfigMap.

OK, thanks @gandro it seems that removing the value from the helm values does it.
The instance cache now does includes all the subnets, and the instance values are found by the operator...

I still don't understand why setting it twice would change the behaviour... any clue ?

It might be smart to alter the documentation with this warning in the meantime?

I still don't understand why setting it twice would change the behaviour... any clue ?

The two filters work independently:

First, eni.subnetTagsFilter is used as a global filter, to remove any non-matching subnets when the operator selects candidate subnets to attach ENIs from. See here: https://github.com/cilium/cilium/blob/314ca7baeff4f568ffc0bad95124e40665b1f88c/pkg/ipam/allocator/aws/aws.go#L40)
Secondly, the CNI config section is copied over into CiliumNode.Spec.ENI.SubnetTags and then used to filter out candidate subnets for this particular node (where we also filter out e.g. any subnets that are not part of the node's availability zone). See here: https://github.com/cilium/cilium/blob/c492154bfe6d53870a462e50fc2d8d7dd4aaa758/pkg/aws/eni/node.go#L367

So the two filters are applied a two different times. The first one on a global level when fetching the subnets from the AWS API, and the second one on a per-node level when we select subnets that the node can attach to.
It seems that the call to the AWS API is returing nothing for some reason if the filters from eni.subnetTagsFilter are applied. Before v1.10.7, the eni.subnetTagsFilter option was effectively a no-op, so no filters were used when talking to the AWS API.

Thanks for the details.
I can also confirm that eni.subnetTagsFilter IS working for me :)
But it's not doing what I thought it would....
eni.subnetTagsFilter filters the ec2 instances to manage (or consider as possible candidates).
In my case, because fresh instance does not have any interface using a subnet that matched the filter, no instance is selected.
I guess that's this behaviour that is bad here... and does not align with the docs:
| eni.subnetTagsFilter | string | `""` | Filter via tags (k=v) which will dictate which subnets are going to be used to create new ENIs |

My guess is that the GetInstances() func should not rely on subnet list gathered by GetSubnets() (which does use the filter)

@prune998  That is a very good observation, you might be on to something here. This theory matches reports of my colleagues which apparently have been using eni.subnetTagsFilter successfully with v1.10.8, but in their case the filter does match the already attached eth0 interface, so GetInstances returns something for every node in their case.

I guess that filtering the instances based the the filtered subnets is to reduce the amount of data to store/parse/consider ?
I'm not sure what would be the right way of doing this filtering... My guess would be that it should be based on tags attached to the instances, that should be = to the EKS cluster... as I guess we should only consider instances that are part of the cluster where Cilium is running...
Maybe eks:cluster-name or based on the nodegroup name eks:nodegroup-name ? Not sure what would happen when the nodeGroup is not managed though.
The quick solution would be to ensure GetInstances() does not use the filtered subnets as a filter... I could PR for that, but maybe a little bit of thinking would'nt be bad either :)

side note: I have multiple EKS clusters in the same VPC/subnets... so Cilium of cluster X is actually considering all the instances of cluster Y.
Thats why I guess using the subnets / subnet filter is a bad idea here.
Instance tags would be a better option if we want to narrow down the number of instances we're considering

I guess that filtering the instances based the the filtered subnets is to reduce the amount of data to store/parse/consider ?

Yes, this is my understanding too. It seems that the global filter (Helm value eni.subnetTagsFilter) was indeed added as a performance optimization: https://github.com/cilium/cilium/pull/10738
Given what you said, I think the way forward here is two-fold:

Let's fix GetInstances() to return all instances, even if they are not attached to a matching subnet (yet). This is a bug fix and can be backported to the the next v1.10.x release.
Let's then maybe introduce an instance tag filter as you suggested. That proposal does make a lot of sense to me. But I think that should happen in a separate PR, because I think an instance tag filter would be more like a new feature rather than a bug fix, and thus I think that change shouldn't be backported to released versions as to our backporting criteria.

totally agree. I'll work a PR for 1) quickly !
Thanks !

@gandro according to the PR https://github.com/cilium/cilium/pull/10738 it feels there's a real use-case for such an option.
Maybe this option is just badly named or documented.
I guess we don't want to revert the use of it, or some large users will cry.
maybe we should just amend the docs to be clear about this option ?
Then, I'll focus on 2) ?

@prune998 Good point. I didn't study the GetInstances implementation before. I agree that simply removing the global subnet tag filter from GetInstances/describeNetworkInterfaces would defeat it's original purpose.
If you say you want to focus on 2), does that mean changing the GetInstances implementation from calling describe-network-interaces (with filtering by subnet tags, which causes to the bug in this issue) to calling describe-instances (with filtering on intstance tags instead)?
I guess I could see that work, but it would require users for whom GetInstances is too expensive without any filtering to add instance filters too.

Yes, it will be a new parameter along eni.subnetTagsFilter... like eni.instanceTagsFilter and both options will require some documentation changes to clearly state what they do : filtering ec2 instances that are part of this specific cluster.
I have to do more testing on using un-managed nodePools in EKS before putting something together.

I'm not sure if for point 1, cilium is aware if it starting up or not.
It indeed makes sense that the subnetfilter would not check ENI's that don't comply to the filter. However if it cannot find an ENI on the node that complies with the filter and there are still ENI spots left to be attached, wouldn't it make sense it would try to attach an ENI that does comply with the filter (at least during startup)?

@anwbtom this is an interresting idea.... I'll double check if it's possible and if it does not defeat the oprimization purpose of this filter.
Also, I'll double check if we can make the logs more explicit, reminding the use of the filter...

@gandro while preparing a PR to filter ENIs based on instances tags (instances part of the k8s cluster), I realized that most of the AWS info gathering (from the Resync function) just get all the possible resources, like all VPC, all subnets, etc...
I keep double-triple checking the code, but I just don't see why this would be needed.
I can feel this could have a huge impact for companies that only have one AWS account with multiple VPCs/subnets/instances/clusters...
The same thing seems to be true for Alibaba, azure...
Maybe it's used for CiliumMesh ?

I linked the PR for the change. This is still early work in progress.
https://github.com/cilium/cilium/pull/19181
I just decided to add the instance Tag  option, and if define, it has precedence over the subnet filter when it's down to finding the available instance ENIs.

OK I reworked the PR https://github.com/cilium/cilium/pull/19181
It is working and I can confirm that it's doing exactly what it's meant for:
msg="Synchronized ENI information"
numInstances=4
numSecurityGroups=34
numSubnets=9
numVPCs=1
subsys=eni

Which correspond to my 4 instances in this kubernetes cluster. Not using this (or using a filter on the subnets) will return 21 instances (all the instances in this account)

Quick summary for anyone stumbling upon this on v1.10/v1.11:

If you are using the Helm value eni.subnetTagsFilter value, please note that it requires that each instance has an ENI with a matching subnet attached when Cilium is deployed. If in doubt, do not use the eni.subnetTagsFilter flag, and only use the subnet tag filter in the CNI config (via cni.customConf). The documentation has been updated accordingly.
If you are running in a large AWS subscription, Cilium v1.12 will offer a new eni.instanceTagsFilter (thanks to @prune998) which you may use to reduce the number of instances queried via the AWS API.


AsynRequestHandler broken in 1.5.1

The changes done to add abort request method removed the asyncContext.complete(); call, see https://github.com/biezhi/blade/commit/d26e0b51948a46dff9e9eb46597e6371f415aa76#diff-7aa3ea01b06a20338721e6335ed5ed92

:joy: Why don't you commit PR? this bug fixes.


Docs on Dockerfiles use word "repository" without any explanation of the concept

Hi guys,
Basic problem:
the doc section on Dockerfiles is rather confusing, because it doesn't really explain an important concept: why is the dockerfile named "Dockerfile" similar to "Makefile", and what does that mean for the folder is it in? Why is it not just a "buildmyimage" file and that's it? Why a whole folder around it, what is it for?
Current situation:
The docs make a basic attempt at explaining it here: http://docker.readthedocs.org/en/v0.5.3/use/builder/ 1. Usage: "To build an image from a source repository, create a description file called Dockerfile at the root of your repository."
However, the docs completely drop the ball clarifying this. What is a repository here? Is that something special with special other files that are automatically used? Does it need a specific layout?
Even worse, if you are sufficiently confused to click "Working with repositories" in the navigation to clear this up, that section introduces a confusing online image thing where the basic concept of what a single repository is in practise on my hard disk as a basic folder is entirely missing.
Proposal:
http://docker.readthedocs.org/en/v0.5.3/use/builder/ 1. Usage: "To build an image from a source repository, create a description file called Dockerfile at the root of your repository."
Change to: "To build an image from a source repository, create a description file called Dockerfile at the root of your repository, which is just the basic folder containing all sorts of other resources you might need and later reference from your Dockerfile to include and use for your image as needed."
Or whatever you see fit. But please add a proper explanation!
Disclaimer:
I started reading the docker docs a few hours ago, so I might be misunderstanding basic concepts. But this really seemed confusing to me, so it would be neat if you could look into it - unless the explanation is already there and I was just too blind to stumble across it.

@fredlf @SvenDowideit is this still an issue

#dibs
as a newbie, here's my shot.

Many thanks @MHBauer I really like that you chose to introduce the build context term instead of my suggestion, since that is what the tool also mentions when "uploading" during a docker build run. You fixed two things in one. :-) good job


README: Stream.js vs Java 8 Streams

Add a section to the README describing the differences between Stream.js and the Java 8 Streams API.

This may be addressed in a future blog post.


In python3.4.1 all tests cause <Task pending _read_data() > warning.

Recently I've updated my python to  3.4.1 (debian testing switched to this version), as result every single test in aioredis causes following warning (however no warning with python3.4):
Task was destroyed but it is pending!
task: <Task pending _read_data() at /home/nick/sources/python/aioredis/aioredis/connection.py:78 wait_for=<Future pending cb=[Task._wakeup()]>>

I tried to run tests with  PYTHONASYNCIODEBUG=1 and got following exception:
 pending _read_data() at /home/nick/sources/python/aioredis/aioredis/connection.py:78 wait_for=<Future pending cb=[Task._wakeup()]>>
Task was destroyed but it is pending!
source_traceback: Object created at (most recent call last):
  File "runtests.py", line 290, in <module>
    runtests()
  File "runtests.py", line 273, in runtests
    failfast=failfast).run(tests)
  File "/usr/lib/python3.4/unittest/runner.py", line 168, in run
    test(result)
  File "/usr/lib/python3.4/unittest/suite.py", line 87, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python3.4/unittest/suite.py", line 125, in run
    test(result)
  File "/usr/lib/python3.4/unittest/case.py", line 625, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python3.4/unittest/case.py", line 577, in run
    testMethod()
  File "tests/connection_test.py", line 35, in test_global_loop
    ('localhost', self.redis_port), db=0))
  File "/usr/lib/python3.4/asyncio/base_events.py", line 239, in run_until_complete
    self.run_forever()
  File "/usr/lib/python3.4/asyncio/base_events.py", line 212, in run_forever
    self._run_once()
  File "/usr/lib/python3.4/asyncio/base_events.py", line 912, in _run_once
    handle._run()
  File "/usr/lib/python3.4/asyncio/events.py", line 96, in _run
    self._callback(*self._args)
  File "/usr/lib/python3.4/asyncio/tasks.py", line 298, in _wakeup
    self._step(value, None)
  File "/usr/lib/python3.4/asyncio/tasks.py", line 244, in _step
    result = next(coro)
  File "/usr/lib/python3.4/asyncio/coroutines.py", line 78, in __next__
    return next(self.gen)
  File "/home/nick/sources/python/aioredis/aioredis/connection.py", line 42, in create_connection
    conn = RedisConnection(reader, writer, encoding=encoding, loop=loop)
  File "/home/nick/sources/python/aioredis/aioredis/connection.py", line 63, in __init__
    self._reader_task = asyncio.Task(self._read_data(), loop=self._loop)
task: <Task pending _read_data() at /home/nick/sources/python/aioredis/aioredis/connection.py:78 wait_for=<Future pending cb=[Task._wakeup()]>>
make: *** [test] Error

Any idea how to mute this warning?

I see this issue on MaxOS with latest dev version on Python 3.4.3 and 3.5.1
Here's a simple snippet to reproduce from interactive shell:
import asyncio
import aioredis
loop = asyncio.get_event_loop()
r = loop.run_until_complete(aioredis.create_redis(('localhost', 6379)))
loop.run_until_complete(r.set('test', 1))
x = loop.run_until_complete(r.get('test'))
print(x)
exit()

Here's the output:
Python 3.5.1 (default, Dec 24 2015, 21:41:53)
[GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> import asyncio
>>> import aioredis
>>> loop = asyncio.get_event_loop()
>>> r = loop.run_until_complete(aioredis.create_redis(('localhost', 6379)))
>>> loop.run_until_complete(r.set('test', 1))
True
>>> x = loop.run_until_complete(r.get('test'))
>>> print(x)
b'1'
>>> exit()
Task was destroyed but it is pending!
task: <Task pending coro=<RedisConnection._read_data() running at /Users/user/.virtualenvs/aborigen/src/aioredis/aioredis/connection.py:107> wait_for=<Future pending cb=[Task._wakeup()]> cb=[Future.set_result()]>

Hi, in your case you sould use at least r.close()and at most:
r.close()
loop.run_until_complete(r.wait_closed())

before exit() call

Hi @popravich! That worked! Thanks a lot!
Actually I start seeing these annoying messages after some update on stopping server by Ctrl+C, now I have an idea what to do!

@popravich BTW I didn't see mention about wait_closed() in introductory samples. I'd suggest to provide complete working sample from setting up connection to cleaning up in Getting Started docs section. What do you think?

I'll add a note on wait_close(). Anyway, all (downloadable) examples in docs are runnable and have connections finalizations.

@popravich Very helpful, thanks.  It might be worth putting in the docs, for example here: http://aioredis.readthedocs.io/en/v0.3.0/api_reference.html  It's a bit confusing to see the Task was destroyed but it is pending! when just following the examples.


All bridges should use HTTPS sources when possible

title says it all

Yes, there are many that don't. It's best practice to use HTTPS where possible, so I added this to the planned Wiki: https://github.com/RSS-Bridge/rss-bridge/issues/345#issuecomment-239474930
Once we get tests up and running, we can actually generate warnings for bridges missing it.

Closing because

#483 introduced a visual indicator for the HTTPS status
This was added to the technical recommendations