Instead of Arrow navigation buttons we should use hotkeys

At some point we added arrows to be able to quickly navigate through the challenges. While this makes it easier for superusers to step through the challenges, they're not ideal from a pedagogical perspective.
The presence of these arrows will encourage people to just start skipping around the moment they get stuck instead of just being patient and trying to solve the challenge. And once a learner has skipped a challenge, they are less likely to come back and complete it. It's like skipping ahead and reading the last page of a novel - it diminishes your willingness to flip back to the beginning and read the rest of it.
This features is still useful, though. We should trigger this functionality with a hotkey. We should put most of our superuser-type features as hotkeys so they don't complicate the UI or confuse the initial learning experience.

@QuincyLarson PR #https://github.com/freeCodeCamp/freeCodeCamp/pull/35485 added these and closed issue #17509.  I think these arrows are a great feature which allow users to easy go back and forth to other lessons they have already completed also.  This is very useful for users completing the challenges which build upon each other.  A user may get stuck, but remember that a couple lessons back they successfully completed a lesson which could help them solve the current one.  Instead of having to click back on the Learn button and fumble through trying to find the lesson, they can simply back up until they find the lesson and read the challenge and then forward through till they get back to the current lesson.   Users have always been able to skip around.  They just leave the lesson and start a different lesson using the map.
If you still want to remove them, you can, but I wanted to put out my two cents of why I think they are very useful.

Hi @RandellDawson. Thanks for your feedback. I regret that I didn't see that discussion thread earlier or I would have said something sooner.
This arrow feature addresses a symptom of the navigation problem, but not the underlying problem.
That underlying problem is that our challenge navigation is too cumbersome. We should instead work on simplifying that navigation.
Clicking back and forth through a series of dozens of challenges is not a good way to find something. In practice it will just spur people to click through challenges when one of the challenges stumps them, then shrug and leave freeCodeCamp instead of working through the challenges.
We are getting ready to de-index all of Learn from Google. So we won't have to worry about people arriving on random challenges in the middle of the curriculum.
So here's what I propose we do:

remove the "Introduction to [topic]" pages from the map (We added these for now-obsolete SEO purposes)
make it so we can link to specific sections of the map like this https://learn.freecodecamp.org/#basic-html-and-html5
Make the "Basic HTML and HTML5" part of the title  - whatever part comes before the : - into a hyperlink to that specific section of the map
Remove the arrow navigation items

Since this was introduced on master, we will have to address this before shipping out. Any takers?

@QuincyLarson We have thousands of challenges in our curriculum. Suppose if the user is in 300th challenge and wants to go back to 200th challenge, the hotkeys or the arrow keys are not at all useful in cases like these. We can implement hotkeys for superusers but it's not solving the underlying navigation problem.
Actually, we should not look at it as a problem because there's no better approach to solve the navigation problem than the current one - going to map and finding the challenge. We can explore possibilities like putting a search box for challenges in the map. But still, the search box won't be that effective. So the current method of navigating challenges looks best to me.
Since we are landing hotkeys on #36808 for super users, I would recommend highlighting the current challenge. After the user has revised the previous challenges it would make it easy for him to stop on his current challenge. We can do something like this:


DRILL-4303: ESRI Shapefile (shp) Format Plugin

Format Plugin for ESRI Shape Files
This format plugin allows Drill to read ESRI Shape files. You can read about the shapefile format here: https://en.wikipedia.org/wiki/Shapefile.
Configuration Options
Other than the file extensions, there are no configuration options for this plugin. To use, simply add the following to your configuration:
    "shp": {
      "type": "shp",
      "extensions": [
        "shp"
      ]
    }

Usage Notes:
This plugin will return the following fields:

gid:  Integer
srid:  Integer
shapeType:  String
name:  Plain text
geom:  A geometric point or path.  This field is returned as a VARBINARY.

Also please create Jira for future reader improvements...

+1, please rebase if needed and squash the commits.

Thank you @arina-ielchiieva and @paul-rogers for the review.  Also thank you @k255 for the original PR.  Commits squashed.

Once full test run will pass, PR will be merged.

@cgivre one more thing, I think you forgot to create Jira for the reader enhancement Paul has mentioned.

I will create the ticket shortly.

On Oct 28, 2019, at 10:54 AM, Arina Ielchiieva<EMAIL_ADDRESS>wrote:
@cgivre https://github.com/cgivre one more thing, I think you forgot to create Jira for the reader enhancement Paul has mentioned.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/apache/drill/pull/1858?email_source=notifications&email_token=ABKB7PVKPQBFRCO4UGN4ZODQQ34JXA5CNFSM4IYZLGLKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECNFBFQ#issuecomment-546984086, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABKB7PRP2HECLOPB3XNW3T3QQ34JXANCNFSM4IYZLGLA.

Thanks.


Add delegate notification API

I went with kwargs for all, it differs from send_notification this way but I think it's clearer than positional args.

I think we can skip adding a test for this.

Thanks, I did test manually.

I added this PR #474, which includes a bit more. Can we move forward with that one?


Add workflow to perform nightly cleanup on playground cluster

Notes for Reviewers
This PR adds a workflow which runs nightly and removes all namespaces except for the ones specified in the if statement.
NOTE: Users might be thrown of if they are on playground and a namespace or resources in a namespace suddenly go away as this script runs.
Signed commits

[ ] Yes, I signed my commits.

@alaeddinebenhassir your eyes here would be helpful. This might be the first version of the cleanup process.


⚠ Samsung has degraded performance

In c8a1e10, Samsung (https://samsung.com) experienced degraded performance:

HTTP code: 200
Response time: 2739 ms

Resolved: Samsung performance has improved in 998ae9d.


Technoopig/balance

Warsong
Chen Str per level increased 2.2>2.7
Druids
Druid T1,T2,T3 defense increase per lvl reduced 5>2
Natures Blessing move speed speed increase reduced 40>20
Treants movement speed reduced 220>180
Treants base HP reduced 850>575
Summon Treants ability duration reduced 50s>40s
Summon Treants ability units summoned amount decreased 8>5
Ironforge
Thane HP reduced 1525>1325
Thane base defense reduced 5>4
Perfect Avatar duration reduced 15s>10s

Duration in tooltip is wrong. I checked and it's because you reference the ability 'A061' but Perfect Avatar is 'A06I'.


--console-title should check if the output is a TTY

Checklist

[ ] I'm reporting a broken site support issue
[x] I've verified that I'm running youtube-dl version 2021.04.01
[x] I've checked that all provided URLs are alive and playable in a browser
[x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[x] I've searched the bugtracker for similar bug reports including closed ones
[x] I've read bugs section in FAQ

Verbose log
youtube-dl 'https://www.youtube.com/watch?v=xQ4KJF-6Agc' --print-json --console-title --verbose | jq
[debug] System config: ['-f', 'best[fps>30][height<=?768][ext=mp4]/bestvideo[fps>30][height<=?768][ext=mp4]+bestaudio/bestvideo[height<=?768][ext=mp4]+bestaudio/best[height<=?768][ext=mp4]/best[height<=?768]/bestvideo[height<=?768]+bestaudio/best/bestvideo+bestaudio']
[debug] User config: ['-f', 'best[fps>30][height<=?768][ext=mp4]/bestvideo[fps>30][height<=?768][ext=mp4]+bestaudio/bestvideo[height<=?768][ext=mp4]+bestaudio/best[height<=?768][ext=mp4]/best[height<=?768]/bestvideo[height<=?768]+bestaudio/best/bestvideo+bestaudio', '--console-title']
[debug] Custom config: []
[debug] Command-line args: ['https://www.youtube.com/watch?v=xQ4KJF-6Agc', '--print-json', '--verbose']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.04.01
[debug] Python version 3.9.2 (CPython) - Linux-5.11.11-arch1-1-x86_64-with-glibc2.33
[debug] exe versions: ffmpeg 4.3.2, ffprobe 4.3.2, rtmpdump 2.4
[debug] Proxy map: {}
parse error: Invalid numeric literal at line 1, column 2
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/youtube_dl/__init__.py", line 465, in _real_main
    retcode = ydl.download(all_urls)
  File "/usr/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 2059, in download
    res = self.extract_info(
  File "/usr/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 799, in extract_info
    return self.__extract_info(url, ie, download, extra_info, process)
  File "/usr/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 806, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 838, in __extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "/usr/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 872, in process_ie_result
    return self.process_video_result(ie_result, download=download)
  File "/usr/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 1683, in process_video_result
    self.process_info(new_info)
  File "/usr/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 1793, in process_info
    self.__forced_printings(info_dict, filename, incomplete=False)
  File "/usr/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 1765, in __forced_printings
    self.to_stdout(json.dumps(info_dict))
  File "/usr/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 520, in to_stdout
    self._write_string(output, self._screen_file)
  File "/usr/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 509, in _write_string
    write_string(s, out=out, encoding=self.params.get('encoding'))
  File "/usr/lib/python3.9/site-packages/youtube_dl/utils.py", line 3180, in write_string
    out.buffer.write(byt)
BrokenPipeError: [Errno 32] Broken pipe

Description
I use the --print-json option to pipe JSON data into jq, and then parse it.
But then when I include --console-title, youtube-dl outputs escape sequences alongside the JSON data, causing jq to break because it does not receive valid JSON.
It would be nicer to just disable this option if stdout is not a TTY (or try redirecting to stderr instead)

Currently (unix-like) the progress escape sequences go to stdout, or stderr when the downloaded file is sent to stdout. While this makes sense it doesn't help with your issue.
This patch to l.365-7 of youtube_dl/__init__.py should solve the issue (by sending the progress escape sequences to stderr in the quoted case):
         'noplaylist': opts.noplaylist,
-        'logtostderr': opts.outtmpl == '-',
+        'logtostderr': opts.outtmpl == '-' or opts.print_json,
         'consoletitle': opts.consoletitle,

Possibly the flags set by some other diagnostic options should be added to the list of ored items.

That patch might be helpful, but stderr can also be redirected to a file that's not a terminal.
I think for terminal escape sequences it would make sense to explicitly check for sys.stdout.isatty() or sys.stderr.isatty().


Workflow for the "RAFT Optical Flow Embedder" node

Looks like an interesting node, but don't know how to use it.

https://github.com/Fannovel16/comfyui_controlnet_aux/issues/58

RAFT Optical Flow Embedder is removed as I misunderstood the logic behind TemporalNet2's code


Possibility to ignore setting-lines in a source modul

We are working on z/VSE and use Z Open Editor just as smart Editor. Compiles are still done on VSE via CMS. Our compile routine requires some setting-lines at the beginning of the source-code member:
$$ICSU1$$ PHASE=UFZAPIC;LIB=SUBCLO;LIST=SOURCE;
$$ICSU2$$ CLE;CICS;
       IDENTIFICATION DIVISION.
       PROGRAM-ID. UFZAPIC.
       ENVIRONMENT DIVISION.
       DATA DIVISION.
       WORKING-STORAGE SECTION.
      *
      * Debug Text
      *
       01 INFOTEXT.
          02 PGM-NAME                  PIC X(7)   VALUE 'UFZAPIC'.
          02 FILLER                    PIC X(02)  VALUE ': '.
          02 INFOTEXT-MSG              PIC X(60)  VALUE SPACE.
          02 INFOTEXT-NUM REDEFINES INFOTEXT-MSG.
             05 INFOTEXT-NUM-VAL       PIC -9(6).
             05 FILLER                 PIC X(53).

But these lines are forcing an "unexpected token"-error and a lot of functionality of VSC is not working anymore. For example jump to definition and lot of autocomletes.

Hi @Andi82ka
These lines are not « COBOL compilant »…
i suggest you use comment lines « COBOL compilant » for your compilation directives.
Another solution would be to use 2 files:

1st: pure COBOL source file
2nd: compilation source with compilation directives and a COPY to 1st file

Or see « basis » COBOL directive.

@Andi82ka I would agree with @FALLAI-Denis that using a comment lines, if possible, would be the easiest solution.
We do have requests to support preprocessors in the future as well, but those would also require to generate some valid COBOL that our editor can consume as well.

Hi @phaumer
Does the COBOL language server process the REPLACE directive by modifying the source code before parsing?

@FALLAI-Denis this is done in the parse tree.
I am closing this issue as there was no other response from the submitter.


New Portainer update breaks theme

Hi there!
Portainer updated this morning and broke a bit of the theme on the home page: /#!/home
See below screenshot.

I went through the rest of Portainer and didn't find other sections where the theme was broken.

Version?

Version?

2.13.1 from the official docker image: https://hub.docker.com/r/portainer/portainer

try develop.theme-park.dev now

develop.theme-park.dev

Tested and working as intended! Thanks!


Move to typescript

:wave:
First of all thanks for this project, it seems nicely done (I did not have a lot of time to play with yet).
The move to typescript would be quite easy.
Would you be open to migrate to typescript for the frontend part ?
I could open a PR if you like.

French updates aside, you can see a summary of the typescript update here: https://github.com/Frojd/Wagtail-Pipit/compare/master...l-etabli:main
It is a very basic typescript implementation, relying mainly on default next.js types.
At this stage, it already helped me noticing a few errors in the code (like bad imports for example).

HI @fabienheureux, thanks for showing interest in Pipit and I hope it suits your needs, otherwise feedback are much welcome :)
Regarding Typescript, its a tricky subject for us, we have discussed it internally a few times at Fröjd (the agency that does most of the development of this boilerplate) but have so far decided to keep using JS, mainly because we feel that it complicates our types of project. Nothing is set in stone and we will continue to revisit this question in the future.
When it comes to the frontend/Next.js part of this project, we have a goal to port it to a create-next-app boilerplate, that way we can detach it from the Pipit and make it easier to switch for something else, like a typescript version. I think that will happen it the upcoming months.
So for now, lets keep this issue open as a reference to others that wish to use Typescript + Pipit.
If you have other ideas, issues is a good way to go, you can otherwise find us for discussion at the Wagtail Slack under channel #pipit

Thanks for your reply !
Moving the frontend out of the project could be a great idea !
I started forking the project with a few ideas in mind

Have the option to use gatsby instead of next (or no React framework at all)
Have the option to have a standard wagtail frontend without decoupled frontend
Have the option to have the django pattern library installed if no decoupled frontend is used
Have the option to have grapple installed. It was the main reason I forked Pipit: I will work in the following months on a grapple/next.js implementation, and I really liked your approach using cookiecutter.

About typescript, I usually use a very basic implementation and don't necessarily write my own types, but having all the TS ecosystem (autocompletion, typing, etc) help to maintain a better code quality in my opinion.
I will come to the wagtail channel in order discuss these ideas...

I noticed that @fabienheureux previous link has stopped working, so I'm just posting a new link here to the new repo where anyone curious in using Typescript + Pipit can take a look and find inspiration: https://github.com/l-etabli/wagtail-debut
(And just to reiterate, we don't have a Typescript port in our roadmap, but its not too difficult to migrate to typescript)

Hi @marteinn, thanks for the link !
Yes in fact, as I am moving away of the initial repo, I decided to remove the "fork" reference from Pipit.
However, I will try to keep track of changes in Pipit and backport these in my fork.
For future reference, here is the approach I used to get TS in Pipit

Follow officiel next.js documentation: https://nextjs.org/docs/basic-features/typescript
Use js-to-ts converter: https://www.npmjs.com/package/js-to-ts-converter
Fix a few typing errors: mainly, I had to type errors as any (or provide your own types, I was lazy there): https://github.com/Frojd/Wagtail-Pipit/blob/master/{{cookiecutter.project_name}}/frontend/utils/Http.js#L28
And...that should be enough to have a TS-ready frontend :)

Thanks for clarifying @fabienheureux and for the awesome links!


Introduce Need-to-Click-Apply Notice in Tileset Editor

Summary
The need for clicking the Apply button in the Tileset editor is often overlooked because it's not the usual way things work in the Duality editor. In order to give users an additional hint, add a blue info text next to the Apply / Revert buttons that will appear when the Tileset has been changed. Should be similar to the "Apply Updates" text in the package manager dialog.

Fixed.


Consider restructuring the project to simplify adding support for new chips

Currently due to the limited functionality of the templating system used by cargo-generate, our template files are quite messy. It's also rather difficult to add support for a new chip, and in general I just feel that from a maintainer's perspective this project still leaves a lot to be desired.
I'm proposing we consider restructuring the project to simplify things for us. I have two possible ideas; both of these have the unfortunate side-effect of requiring a fair bit of duplication, however I feel they are both cleaner and will give us much more flexibility than we currently have.
I will outline my ideas below. It may turn out that sticking with our current approach is the right way forward, however I would at least like to discuss these other options.
cc @SergioGasquez
Per-architecture Templates
One possibility is splitting things up into two subdirectories, riscv and xtensa, much in the same way that esp-idf-template is split into cargo and cmake directories.
This removes a lot of the conditional code from our templates, and we are fairly certain that no new Xtensa devices will be released so we can be reasonably confident that this will not change much. Adding support for new devices will always require modifying the riscv template.
Per-device Templates
Taking this even a step further, we could simply have a directory for each supported device. This results in a lot of duplication, however the templates become much simpler and cleaner as a result and we have maximum flexibility. This also makes it trivial to add support for a new device.
My only concern here is that I'm not sure this level of duplication is worth the benefits.

I think the scripting capabilities are not that bad (however I don't have experience with it ... so maybe I'm wrong)
I see two things where scripting can help a lot

have a "pre" script which populates variables which are later used in the templates - making decisions based on the input seems much nicer in Rhai (the script could either generate code blocks which just get inserted and/or set derived variables like an "architecture" so that the conditionals in the template look much cleaner - like we do in esp-hal-common where we set cfg values based on the target)
have a "post" script which deletes / renames files - we could have files for each chip or architecture and the script can remove the unwanted files and rename files (e.g. "main_esp32c3.rs" to "main.rs")

Certainly both can be combined - maybe worth to try that?

Even better: With my upcoming changes for https://github.com/esp-rs/esp-hal/issues/382 the template will get much simpler.

ESP32-S2 atomic-emulation will be automatically handled by the HAL (like we do for ESP32-C2 / ESP32-C3 already)
the entry macro is re-exported on the same path regardless of the architecture and will even be included in the prelude
rt crates should not be added as a dependency in binary crates
riscv/xtensa-lx are re-exported from the HAL and shouldn't be added as dependencies
we can alias the concrete HAL as hal - no need to import from differently named crates

Even better: With my upcoming changes for esp-rs/esp-hal#382 the template will get much simpler.

ESP32-S2 atomic-emulation will be automatically handled by the HAL (like we do for ESP32-C2 / ESP32-C3 already)
the entry macro is re-exported on the same path regardless of the architecture and will even be included in the prelude
rt crates should not be added as a dependency in binary crates
riscv/xtensa-lx are re-exported from the HAL and shouldn't be added as dependencies
we can alias the concrete HAL as hal - no need to import from differently named crates

Those are great new! I had already started working on splitting the template, and today I did a bit more of work, here is the current state: https://github.com/esp-rs/esp-template/compare/main...SergioGasquez:esp-template:feature/architectore-refactor. Again, I am more than happy to ditch that work and do it on a better way!

I do like the approach @bjoernQ suggested (#55), we would end up with a bunch of ifs in some places (like Cargo.toml for the hal versions) but we reduce a lot of ifs and we keep the template in a single folder  (instead of having xtensa and riscv templates duplicating a bunch of code)

Maybe all those ifs can even get moved to the pre-hook and then it's mostly just using variables (e.g. the script could populate the HAL package-name and the HAL version)
Maybe that Rhai code can even get written in a nicer way (unfortunately I am not fluent in Rhai script)

Maybe all those ifs can even get moved to the pre-hook and then it's mostly just using variables (e.g. the script could populate the HAL package-name and the HAL version)

That could improve readability as we would have all the logic inside the rhai script, which is indented, and not mixing two different programming languages. We could even try to get the latest version of each hal published on crates.io and use it instead of hard-coding the version (even though this can be “risky”)

I updated the mentioned PR and I'm more than happy with the result. Adding support for a new chip will only be a matter of editing pre-script.rhai and adding the description of the target like this
...
    esp32c6: #{
        arch: "riscv",
        has_swd: true,
        sys_peripheral: "PCR",
        toolchain: "nightly",
        rust_target: "riscv32imac-unknown-none-elf",
        atomic_emulation: "none",
        gcc_target: "riscv32-esp-elf",
    },
...

Plus adding the option to the [placeholders.mcu] obviously
Also, that PR is now almost in a mergeable state (just waiting for the next HAL release)

We could even try to get the latest version of each hal published on crates.io and use it instead of hard-coding the version (even though this can be “risky”)

I think that is indeed too risky since at least in the forseeable future we will introduce breaking changes. Once we are at 1.x.x versions that however might be a good idea - in the meantime probably something like #53 is the better option

Closed by #55


Build fail

Hyprland Version
Newest
Bug or Regression?
Bug
Description
Trying to manually compile I pull in all dependencies with --recursive then try to build it all dependencies are found except for tomlplusplus and build fails
How to reproduce
git clone --recursive https://github.com/hyprwm/Hyprland.git
meson setup build
Crash reports, logs, images, videos
No response

tomlplusplus is a dependency that is not bundled - use your system's package manager.
Please do not report such build failures - ci passes so it's an issue on your side.

I don't have a package manager I use LFS. Anyways, thanks for your time

that's a problem that should be reported to fedora. Maybe hpp is in a separate -devel package?

hpp is available for Fedora 39 (not 38): https://fedora.pkgs.org/39/fedora-updates-x86_64/tomlplusplus-devel-3.4.0-1.fc39.x86_64.rpm.html.
Why not meson wrap install tomlplusplus instead?

I'm on Linux Mint, tried installing tomlplusplus from source as it's not in repos.
It installed to /usr/local/include/toml++. I tried moving it to /usr/include/toml++ and some other things such as setting prefixes.
After many attempts I'm still getting:
-- Setting link libraries
Could not open input file: No such file or directory
Could not open input file: No such file or directory
-- Checking for module 'toml++'
--   No package 'toml++' found
CMake Error at /usr/share/cmake-3.22/Modules/FindPkgConfig.cmake:603 (message):
  A required package was not found
Call Stack (most recent call first):
  /usr/share/cmake-3.22/Modules/FindPkgConfig.cmake:825 (_pkg_check_modules_internal)
  hyprpm/CMakeLists.txt:12 (pkg_check_modules)```

I tried `meson wrap install tomlplusplus` and I'm still getting this (Sorry, I do not typically work with C++ build tools other than gcc)


Free/Unlimited/Fast RPC endpoints for indexing with ponder

Hey! very cool project
I am especially impressed with how easy it is to setup and test some stuff.
I work at envio and we are building an rpc service which is implemented on top of our custom storage layer for historical evm data. We currently have free and unlimited endpoints for 20 evm chains.
I tested running ponder with our rpc and we seem to be over 200x as fast as blastapi/chainstack premium plans when indexing FairToken (I found this randomly on etherscan for testing).
The config I use is this:
import type { Config } from "@ponder/core";
import { http } from "viem";

export const config: Config = {
  networks: [
    {
      name: "mainnet",
      chainId: 1,
      transport: http("https://eth.rpc.hypersync.xyz/ethistanbul"),
      maxRpcRequestConcurrency: 10,
    },
  ],
  contracts: [
    {
      name: "FairToken",
      network: "mainnet",
      abi: "./abis/FairToken.json",
      address: "0x9b20dabcec77f6289113e61893f7beefaeb1990a",
      startBlock: 4805840,
      maxBlockRange: 100000,
    },
  ],
};

and the indexer is generated using etherscan import method (very nice to use).
Might need to set maxBlockRange to a lower number for denser contracts like ens because we limit response sizes.
If it is ok with owners of the project, we would like to create free/unlimited endpoints for ponder users to use and list them in some documentation in ponder.
We already have endpoints we created for eth istanbul but they will be deprecated in next months so you can test with
https://eth.rpc.hypersync.xyz/ethistanbul
https://polygon.rpc.hypersync.xyz/ethistanbul
https://celo.rpc.hypersync.xyz/ethistanbul
etc..
https://envio.dev/hyper_rpc

Thanks for opening. Could you share more details about the benchmark you ran that found your RPC to be 200x as fast as blastapi/chainstack premium plans? Most Ponder users today use Alchemy, including the Ponder dev team, so I'd also be curious to see a benchmark between HyperRPC and Alchemy.


update operator-sdk to 1.27.0

Which problem is this PR solving?

Resolves #2177

Short description of the changes

Bump the operator-sdk version to 1.27.0
Add all the required changes to make the version work

Note
Some other changes are requested in #2177. These will be addressed in another PR

Could you take a look to this PR? @rubenvp8510


Correct view controller identifier for iOS

The word 'Web' is missing in the string that identifies VC in add storyboard

@brentsimmons should be good now


[question] internationalisation

Hello!
A german Forum I am responsible for will use this plugin and I wanted to ask if there are plans to add translation. I would be glad to provide the informal german translation (deutsch_du). (In fact, I am already working on it because I need it for my forum)
Best Regards!
Thomas

You can download the German translation files from here:
https://github.com/SvePu/mybb-amnesia-german-translation-files

Thank you very much!


Incorrect range handling for some edge cases

Why:
Incorrect range handling cases causes s3manager downloads to fail.

When end extends beyond available data it should return data from
"start" to the end of the available data. Currently it returns
invalid range error.
From RFC-7233:

If the last-byte-pos value is.. greater than or equal to the current
length of the representation data, the byte range is interpreted as
the remainder of the representation (i.e., the server replaces the
value of last-byte-pos with a value that is one less than the current
length of the selected representation).

A "-0" suffix byte range is "not satisfiable" according to RFC and
should return an invalid range error. Currently it just returns an
empty data set.
Also from RFC-7233:

If a valid byte-range-set includes ... or at least one
suffix-byte-range-spec with a non-zero suffix-length, then the
byte-range-set is satisfiable.  Otherwise, the byte-range-set is
unsatisfiable.

Fixes Issue #36

Codecov Report

Merging #37 into master will increase coverage by 0.02%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master      #37      +/-   ##
==========================================
+ Coverage   66.32%   66.35%   +0.02%
==========================================
  Files          27       27
  Lines        2355     2357       +2
==========================================
+ Hits         1562     1564       +2
  Misses        568      568
  Partials      225      225

Impacted Files
Coverage Δ

range.go
83.87% <100%> (+0.53%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d419e1b...d11e83a. Read the comment docs.

Any chance of getting this merged ?

Hi @jwoffindin sorry for letting you wait so long. It just slipped through the cracks. Looks really good. Thanks for taking this on!


Docstrings for classes holding the explanations

Current coverage is 95.34% (diff: 100%)

Merging #51 into master will not change coverage

@@             master        #51   diff @@
==========================================
  Files            26         26
  Lines          1202       1202
  Methods           0          0
  Messages          0          0
  Branches        197        197
==========================================
  Hits           1146       1146
  Misses           32         32
  Partials         24         24

Diff Coverage
File Path

•••••••••• 100%
eli5/base.py

Powered by Codecov. Last update 6db07d3...e55ed26


Allow configurable path and URL for swagger-ui

This PR modifies the App and API classes to allow the caller to specify a different directory for the swagger-ui files and/or a different URL path to access the API documentation.
Additionally, it does not emit a loud warning if the API spec specifies a security method other than oauth2. Instead, it warns the user (via a lower logging level) that she should handle the security herself (eg, through a custom decorator).

Can you check the Travis CI build please?
You can check flake8 formatting locally by running:
python3 setup.py flake8

BTW, thanks for the PR --- more configuration options make sense in this case :-)


provide support for openxl compiler in AIX

Hello
This PR provides configuration to build 32-bit and 64-bit openssl with openxl compiler on AIX OS
With xlc 16.1 nearing end of support shortly, it is required to provide a configuration that supports the next compiler (openxl) on AIX.
This configuration has been tested on different AIX levels (7.2, 7.3) and on different Power platforms.
Thanks

Ci failure is transient due to a failure in nasm.us, will have to rerun when it comes back online

@nhorman , are there any test cases blocking for accepting this PR ? Thanks

Theres one transient failure, which I've restarted.  Other than that its just waiting for people to get around to reviewing it

Just to be "that guy": all types of builds on AIX are in the list of unsupported platforms. Is this a new platform (i.e. something that would require a nominated community member to support), or just another build combination on an unsupported platform?
(Aside: the rendered list of platforms at https://www.openssl.org/policies/general-supplemental/platforms.html could use some whitespace/margin-top on the h3 elements)

Just to be "that guy": all types of builds on AIX are in the list of unsupported platforms. Is this a new platform (i.e. something that would require a nominated community member to support), or just another build combination on an unsupported platform?
(Aside: the rendered list of platforms at https://www.openssl.org/policies/general-supplemental/platforms.html could use some whitespace/margin-top on the h3 elements)

No, this is not a new platform but another combination to use openxl compiler for AIX.

No, this is not a new platform but another combination to use openxl compiler for AIX

Thanks - I don't see a mention of Open XL in https://www.openssl.org/policies/general-supplemental/platforms.html#:~:text=gcc-,aix-gcc,%3F%3F,-BS2000-OSD

No, this is not a new platform but another combination to use openxl compiler for AIX

Thanks - I don't see a mention of Open XL in https://www.openssl.org/policies/general-supplemental/platforms.html#:~:text=gcc-,aix-gcc,%3F%3F,-BS2000-OSD

Once this PR is pushed, May you please add/update the same as well:
aix-cc |   | AIX |   | ppc32 |   | IBM xlc
aix64-cc |   | AIX |   | ppc64 |   | IBM xlc
aix-clang |   | AIX |   | ppc32 |   | IBM openxl clang
aix64-clang |   | AIX |   | ppc64 |   | IBM openxl clang

@sanumesh Please open a pull request against the general-policies repo to make the requested update

@sanumesh Please open a pull request against the general-policies repo to make the requested update

oh ok, I was not aware about this. Thanks, I will submit a PR

Hello @nhorman , @t8m
Do we have any update on this PR? Is there any outlook on OTC review ?
Even https://github.com/openssl/general-policies/pull/69 is waiting for OTC review.
Thanks

@t8m , I see this is labeled for master only.
This PR introduces a new platform with changes restricted to configuration. So, I believe as per 'stable release updates' policy, this PR can also be merged to other branches including 3.0

@sanumesh Yes, I'll add the stable branches labels.

This pull request is ready to merge

Merged to all the active branches. Thank you for your contribution.


Unable to resolve type: React.ReactEnvironmentUnable to resolve type

Hello,  I am able to reference a .jsx file from a page when running in debug mode on my local machine. If, however, I switch to IIS Express in debug mode, I begin to notice 500 errors when inspecting the generated page.
This problem also occurs when I perform "Start without debugging."
Lastly, when deploying to Windows Azure via automated VSTS builds, I also receive the 500 errors when inspecting the page where my .jsx component should be.  I am able to obtain the following details from Azure application insights:
ERROR:
Unable to resolve type: React.ReactEnvironmentUnable to resolve type: React.JavaScriptEngineFactoryNo usable JavaScript engine was found. Please install a JavaScript engine such as React.JavaScriptEngine.ClearScriptV8 (on Windows) or React.JavaScriptEngine.VroomJs (on Linux and Mac OS X). Refer to the ReactJS.NET documentation for more details.
FAILED METHOD:  React.TinyIoC.TinyIoCContainer.ConstructType(Type requestedType, Type implementationType
I am using Visual Studio 2017 to build a MVC web application.  I am using .NET Core version 1.1.
Any thoughts on why I am seeing different behaviors?  Suggestions for making this work more consistently?

Well, I was able to solve the local issue by adding a bit to my project file regarding which javascript engine to use based upon .net framework.
NOTE, I am still receiving the same error once deployed to Azure.  Is there something special we need to do to make the solution work for .NET Core applications in Azure?

@mattgross76 I recommend you to read the “Misconceptions about the JavaScript Engine Switcher version 2.X” post.

@Taritsyn Thank you for the article and for the many nuget packages you have developed.  I read it and have tried a few things, but still have the issue in Azure running a .NET Core 1.1 application.  I am going to try to use Chakra instead of V8 to see if that makes a difference.
Do you have a recommendation regarding which engine to use with .NET Core 1.1 and Windows Azure?  I'm having trouble finding documentation on that particular use case.
Thanks!
Matt

.NET Core is supported by four JS engines: JavaScriptEngineSwitcher.ChakraCore, JavaScriptEngineSwitcher.Msie (only works in JsRT modes), JavaScriptEngineSwitcher.Jint and JavaScriptEngineSwitcher.Vroom. But the JavaScriptEngineSwitcher.ChakraCore is the best choice at the moment.
I also recommend you to read this comment.

And read about registration of JS engines in ASP.NET Core 1.X.

@Taritsyn Thank you so much!  I think it is fixed now.  My small experiment is at least now working within Azure.  That second link regarding the "registration of JS engines in ASP.NET Core 1.X" was the key!
I added the following code to the Startup class, and it works now:
services.AddJsEngineSwitcher(options => 				options.DefaultEngineName = ChakraCoreJsEngine.EngineName 			) 				.AddChakraCore()
For now, I am holding off on the other options.
Thanks again.  I will close the issue.

@mattgross76 what was the bit you added to your project file?


it will crash  when project have a zip file.   in windows platform

VSCode Version: 1.14.2
OS Version: win7

Steps to Reproduce:

Reproduces without extensions: Yes/No

Steps to reproduce?


oplog formatter don't get logs when using pytest

Hi,
I've created a basic test using pytest and integrated it with oplog. When I execute the test using the command pytest --log-cli-level=10 -s (with -s intended to disable log capturing), the console outputs the following message:
INFO context_engine.knowledge_base.knowledge_base:operation.py:122 oplog: operation start
This suggests that pytest seems to be capturing the logs instead of passing them to the oplog formatter. I've set up the logger in the package's init file.
I suspect the issue might be related to pytest configuration settings. Any assistance on resolving this would be greatly appreciated.

Note that the issue is a bit different. Pytest has its own formatter (from limited predefined alternatives), so you can't send over a custom one.
However, you are completely right that this should be supported - and it now is (solved in #10, available on v0.21.0). Thanks for raising this!


[DISCUSSION] Layer-by-Layer Graph Execution

TLDR
Rough Proposal: In order to overlap graph communication with scheduling and computation, we should enable partial task-graph execution. Further details are discussed below, but the ultimate goals is to completely remove the need to send HLGs to the scheduler (while hopefully improving Dask's ability to handle large-scale graphs).
NOTE: The ideas here are mostly a combination of suggestions made in the past by others like @madsbk, @gjoseph92, @ian-r-rose (and probably others)
Background
Most methods in the Dask-Dataframe and Dask-Array APIs produce an abstract HighLevelGraph (HLG) instead of a fully-materialized dictionary-based task graph. The general purpose of this HLG approach is two-fold:

To enable critical graph-optimization operations before the full (dictionary-based) task graph is materialized. These optimizations include blockwise task fusion, culling, and partial-IO optimizations like column projection (and hopefully predicate pushdown in the near future).
To avoid sending large materialized task graphs over the wire from the client process to the scheduler. We expected (and indeed observe) much smaller graph-communication latency when the HLG-to-dictionary materialization was moved from the client to the scheduler.

Although HLGs have demonstrated some success in achieving both of the above goals, there have also been clear challenges:

HLG fusion has relied on existing optimizations in Blockwise, requiring all “fusible” DataFrame and Array Layers (including IO) to be reformulated as Blockwise-based HLG Layers.  The complexity in Blockwise has made this transition a bit difficult.  I will not focus on this specific challenge in the remainder of this proposal (but I felt it was important enough to mention).
The need to deserialize and materialize HLGs on the scheduler has proved to be a complete headache (see distributed#5581 and this summary). The existing serialization and communication infrastructure was designed with the low-level (dictionary) graph in mind.  We quickly discovered that it can be quite painful to construct a low-level graph on the scheduler without being able to call pickle.loads on functions and non-msgpack serializable arguments (a security concern).
Sending HLGs to the scheduler reduced the graph communication time (important for high-latency client-scheduler connections), but it barely made a dent in typical end-to-end run times for systems with decent client-scheduler latency. This is because large client-side graph materialization costs were simply converted into comparable scheduler-side graph materialization costs.  We originally hoped that Cython or C++ on the scheduler would make this graph-materialization cost negligible, but I am skeptical that this is something we can/should rely on.

What I take away from the challenges above is that: (1) Sending HLGs to the scheduler is a headache, and (2) The only reason we are still doing this is to minimize client-to-scheduler communication latency.
Proposal
After a long (but fun) discussion with @madsbk, we generally agreed that we may be able to avoid sending HLGs to the scheduler altogether. Our primary (and probably lone) reason for dealing with HLGs on the scheduler now is graph-communication latency.  Therefore, we will no longer have any motivation to send the HLG to the scheduler if this overhead is sufficiently reduced or hidden.  One possible way to do this is by overlapping graph communication with scheduling and execution.  That is, we may be able to graph communication delays by taking the following steps:
Step 1 - Enable partial graph scheduling
The first and most-critical step is to ensure that the scheduler is capable of executing a partial graph.  I am hoping someone like @madsbk will have the bandwidth and interest to prototype this machinery, but I would expect the client to be able to submit a graph without specifying the keys it wants to get.  Instead, the client should be able to mark specific keys as active dependencies.  Note that I am very open to suggestions here, but the overall goal is to begin assigning tasks to workers before the scheduler has finished receiving the full task graph.
Step 2 - Send the materialized graph Layer-by-Layer
Once the scheduler has a mechanism for processing partial task-graphs, we will want to add a mechanism in the client to actually send large graphs in phases. For the initial pass, I propose that we simply use the existing HLG Layer as a single “sub-graph” unit.  In other words, after the HLG optimization pass, we can simply send the materialized graph to the scheduler one Layer at a time.  After we confirm that this “works,” we can avoid the obvious small-message problem by aggregating adjacent Layers on the client until an optimal sub-graph size is reached.
If we are able to implement these first two steps without critical road blocks, then I suspect we will be highly motivated to discuss the future (or lack thereof) of HLG serialization.  If we can decouple the serialization design from that of HLGs, then I suspect we will find it much easier to improve the two separately.
Possible Follow-On Work
Although Layer-by-Layer graph communication/execution may largely reduce our need to send HLGs to the scheduler, we may still want to send something other than the low-level graph is “special cases.”  For example, if the user is willing to opt in to “full” (dumps and loads) pickle support on the scheduler, then we may want to enable the client to pickle a single generator function for an HLG Layer, and the scheduler could simply unpickle this object and then pass in a required set of output keys to get the necessary tasks.  This would allow the scheduler to break up the materialization and scheduling of extremely large graphs (even when a single layer contains billions of keys).  Such an optimization would clearly take considerable design and development effort, but it may provide us with a reasonable path toward the handling of “extreme-scale” graphs (note that this idea belongs entirely to @madsbk, so he makes want to clarify and/or expand)

cc @quasiben @jakirkham @mrocklin @jcrist (critical thoughts very welcome)

@rjzamora thanks for writing this up; I've been wanting to discuss something similar.
Layer-by-layer will be hard to schedule well. It might make root task overproduction even worse (https://github.com/dask/distributed/issues/5555 / https://github.com/dask/distributed/issues/5223). Since layers are essentially breadth-first, and execution order should be essentially depth-first, I don't think we'd actually be able to overlap the computation with the communication very well. Or rather, if we did so, we'd be doing lots of the wrong computations while waiting for the next communication to happen. Or, if we did so smartly, we'd basically say "okay, I got this root layer of 100,000 tasks, but I'm only scheduling 100 of them and then waiting to get the next layer"—in which case, you're not getting much benefit from overlapping computation and communication, since those 100 may well finish before the next layer is even received.
OTOH, we could look at streaming the full graph to the scheduler in priority order—basically, allow computation to start before all of the tasks have been received. This sounds nice, until you realize that it still requires materializing the full graph on the client (so you can prioritize it) before sending anything to the scheduler. So you're still not getting much benefit there with large graphs.

What I've been thinking about instead is whether we can avoid materializing the graph at all—even on the scheduler—by leveraging the high-level structure of the graph for scheduling purposes.
Basically, could we define an interface on HLGs where you could iterate through tasks in the HLG in approximate priority order ("good scheduling order", aka depth-first order) without materializing the layers?
This way, you could have an HLG of arbitrary size (billions of tasks), but you'd hopefully only have about nthreads tasks actually in memory on the scheduler at once.

Basically, could we define an interface on HLGs where you could iterate through tasks in the HLG in approximate priority order ("good scheduling order", aka depth-first order) without materializing the layers?

This idea may be similar to what Mads had in mind, and what I poorly described at the end of the issue:  Perhaps we just want to be able to pickle some kind of generator function that the scheduler can simply unpickle and use to generate tasks in a reasonable order (not necessarily by layer, but maybe by required output key).  However, in order to get something like this working without a serialization nightmare, it seems like the user would need to opt in to "unpickling on the scheduler". The reason I'm suggesting partial graph execution in general, is so distributed exectution could still "work" for users who do not want "unpickling on the scheduler". In that case, the user would use the same generator function on the client side to materialize subgraphs and send along to the scheduler.
I completely agree that a layer-by-layer approach may not lead to optimal scheduling. In the long run we definitely want to revise the HLG (or perhaps high-level expression) system to generate tasks is a more "functional" way. My suggestions to send the graph is Layer batches is mostly as a path to prototype the partial-exectution idea with minimal changes. However, if you think that the graph-generator logic should come first, I am also open to that idea :)

Perhaps we just want to be able to pickle some kind of generator function that the scheduler can simply unpickle and use to generate tasks in a reasonable order (not necessarily by layer, but maybe by required output key)

I think we're circling around the idea that you, me, @ian-r-rose, and @GenevieveBuckley talked about. With low-level graphs, you figure out the outputs given the inputs. For high-level graphs to be effective, you need to work backwards: you must figure out the inputs given the outputs. You have to invert all the graph-generating logic currently in dask/dask, often coming up with new algorithms. It's a lot of work—it's not just wrapping existing code in a new interface. But if you don't do this, there's not much benefit to high-level graphs, since you're just deferring the materialization to some other point in time.
In order to make this iterate-in-priority-order concept work—which is where I think we'd really see HLGs pay off—we'd have to go all-in on flipping these graph-generation algorithms on their heads.

My suggestions to send the graph is Layer batches is mostly as a path to prototype the partial-exectution idea with minimal changes

I like the incremental approach, I just don't see how much value it would add. Worst-case, it would make an existing problem (root-task execution) even worse. But best-case, communication and computation just wouldn't overlap much, so you wouldn't gain much performance. I just don't think there's much of a way to work around needing depth-first scheduling, but having breadth-first communication.
That said, I'd love to be able to stream the transmission and especially unpacking of HLGs, even if we don't start executing them until the whole thing is materialized. Currently, as you mentioned, materializing the HLG (or deserializing and processing a low-level graph) on the scheduler is just as slow as on the client—but it's even worse to do it on the scheduler, because then you block the event loop for tens of seconds at a time! If you have tasks already submitted to the scheduler, and then you submit another big graph to the scheduler, things can go very sideways as you essentially take the scheduler offline for half a minute to materialize your graph.

What about this incremental approach:

Materialize the whole graph on the client. Stream it task-by-task, in priority order (depth-first), to the scheduler. Since it's in priority order, the scheduler can start scheduling tasks as soon as it gets them, without waiting to receive every task. Total compute time is about the same as today (materialize is the same, transfer is slower but now overlaps with compute), but a) you're no longer blocking the scheduler's event loop, improving stability and b) HLG serialization and unpicking on the scheduler becomes a non-issue, since it all happens on the client, so developing HLGs becomes vastly easier.
Implement backpressure from the scheduler to the client to say "hey, I have enough (processing) tasks already to saturate my workers (plus a bit of a buffer to hide my latency talking to you), don't send me any more". This would effectively fix root-task overproduction without implementing https://github.com/dask/distributed/issues/3974, because the scheduler wouldn't even be aware of the extra root tasks, therefore it couldn't send them to workers. (You probably do still want STA, combined with this, in the long run though.) With task backpressure, you also wouldn't have to have every task in scheduler memory the whole time (task count would increase linearly until the very end). There are big questions here about whether this would make scheduling too "dumb"—what happens when some tasks are completing faster than others, but the priority order is just sending along dependents of the slow tasks? Would there be some protocol for the scheduler to request dependents of certain keys, rather than blind priority order? (This would probably add way too much latency.)
Stream results back from the cluster to the client https://github.com/dask/distributed/issues/4754. Now, we can stream both into and out of the scheduler. As soon as an output key is complete and handed back to the client, the scheduler can forget about it. So the number of tasks in play on the scheduler is usually Θ(nthreads), regardless of the total graph size. This is kinda https://github.com/dask/distributed/issues/5630, except you're "spilling" the unneeded tasks to the client instead of disk!
Make it possible to iterate through HLGs in priority order without materialization. Now, when you stream your graph from the client to the scheduler, a) whole graph is never materialized anywhere (client or scheduler), b) work starts nearly instantly, since you don't have to wait for full materialization to start sending the scheduler tasks, c) graph size becomes effectively unlimited for embarrassingly-parallel workloads; the practical limits are the number of workers, and the size of the non-embarrassingly-parallel substructures within the graph.
At this point, things seem pretty great to me. Compute starts instantly, you don't have limitations on graph size, and scheduling is better without root-task overproduction. But if you want even more, maybe you move HLG unpacking back to the scheduler, and iterate through them there. Now, HLGs are essentially a compression format as well. And you have to deal with serialization again. But a) the client connection is less critical to scheduling, and b) the scheduler could pick which parts of the HLG to unpack next without client<->scheduler latency (if the interface even supported this).

I'll note that with these ideas, the client becomes an increasingly-integral part of the system, since the cluster no longer holds the full graph, nor all of the results, in memory. That flies in the face of my concerns in https://github.com/dask/distributed/issues/5684. I don't like being increasingly dependent on a fast, stable, long-lived connection to the client, but:

The benefits seem significant enough, it's probably worth it.
You could use afar-like patterns to turn one of the workers into your client, or maybe even spin up a sidecar Python process on the scheduler, etc., to improve resilience.

cc @fjetter @crusaderky

In order to make this iterate-in-priority-order concept work—which is where I think we'd really see HLGs pay off—we'd have to go all-in on flipping these graph-generation algorithms on their heads.

Yes - I get the feeling that the most-promising solution is to take the time to flip all common graph-construction algorithms on their heads, or develop a good expression system to somehow simplify/automate this (not that I have such a HLE solution in my head).
Regarding your graph streaming suggestion/plan:  This approach is certainly in line with what I was thinking. Perhaps we could also start by looping over partition/chunk-specific subgraphs on the client (to avoid the need to materialize the entire graph at once). This is similar to something we do in NVTabular to ensure the entire subgraph for a specific output key is completely executed on the same worker.

I'll note that with these ideas, the client becomes an increasingly-integral part of the system, since the cluster no longer holds the full graph, nor all of the results, in memory.

Our global state machine is complex and complicated enough as it is without adding a third distinct actor to the mix.
If we want to start streaming the tasks in priority order, that may be something to consider. We'd still need to fully materialize the graph but I would suggest to not couple too many problems at once and would defer this to a later discussion. Implementing some kind of backpressure/feedback mechanism that involves the Client directly or indirectly in scheduling decisions is something I am very strongly against.

My gut reaction to this is skepticism.  Large graphs are less of a problem
than optimal scheduling.  This approach helps with large graphs at the cost
of optimal scheduling.  I think that the difficulties around HLGs are due
to poor design so far, and that can be fixed.  HLGs also provide a variety
of other long-term benefits to the scheduler, which would have a lot more
semantic information about users' computations, which I think will be
valuable in the future.
I think that trying to schedule things iteratively is a cool idea, but is
likely to be even more complex than high level graphs.
On Tue, Jan 25, 2022 at 3:32 AM Florian Jetter @.***>
wrote:

I'll note that with these ideas, the client becomes an
increasingly-integral part of the system, since the cluster no longer holds
the full graph, nor all of the results, in memory.
Our global state machine is complex and complicated enough as it is
without adding a third distinct actor to the mix.
If we want to start streaming the tasks in priority order, that may be
something to consider. We'd still need to fully materialize the graph but I
would suggest to not couple too many problems at once and would defer this
to a later discussion. Implementing some kind of backpressure/feedback
mechanism that involves the Client directly or indirectly in scheduling
decisions is something I am very strongly against.
—
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/8616#issuecomment-1020984494, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AACKZTGVTWQRSQXJJ4Z5ZZTUXZU2ZANCNFSM5MWDCL7A
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you were mentioned.Message ID:
@.***>

Thanks for the feedback @mrocklin!  Note that I have been completely on the fence about this idea from the start (and I still very-much am) :) Overall, I agree with everything you said.  With that in mind…

Large graphs are less of a problem than optimal scheduling.

Totally agree. However, the only reason we are materializing HLG layers on the scheduler is because of graph communication time (which only seems to matter for large graphs and/or remote client-scheduler connections).  I’m okay with saying graph size isn’t a real concern, but then it seems strange that we are taking the time to struggle with multi-step serialization to ultimately minimize the graph-communication cost.

This approach helps with large graphs at the cost of optimal scheduling.

Yes - A naive layer-by-layer approach would likely inhibit optimal scheduling, but that suggestion was more of a “test” (not how we would ever communicate/execute in practice).  However, I would’t expect the long-term procedure to hurt scheduling at all.  The core idea is just that we allow the client to send an incomplete graph to the scheduler.  Ideally, that subgraph would correspond to tasks that should be scheduler before the following subgraph anyway.

I think that the difficulties around HLGs are due to poor design so far, and that can be fixed.

Agree that HLGs should be revised, but are you focusing on pack/unpack (serialization) design, or something else?

HLGs also provide a variety of other long-term benefits to the scheduler, which would have a lot more
semantic information about users' computations, which I think will be valuable in the future.

I agree that we can get a lot out of HLG layers, but I am also a bit skeptical that we really need to avoid materializing the low-level graph representation (within the Layer) before communicating it to the scheduler.  For example, the annotations that we currently ship with the layer have nothing to do with whether that layer is materialized or not.

I really like @gjoseph92's idea.
I think there are a few issues with it though:

as pointed out already, Client<->Scheduler connectivity is frequently less reliable/predictable than Scheduler<->Worker
Client<->Scheduler bandwidth and latency can be much, much slower than the one inside the cluster - namely when the Client is on somebody's laptop and connects to a scheduler on AWS. Worst case scenario: the laptop is connecting over 5G or a Starbucks WiFi.
This means that a client cannot be closed anymore after publish_dataset, or that alternatively publish_dataset must eagerly send to the scheduler the remainder of the graph
The Client may hold its own event loop precious too, and this design slams it down hard. This is not the typical case of Jupyter Notebooks, but it is that of e.g. a web or RPC server that uses dask to offload computation.

I would like to propose an improvement on top of @gjoseph92's design: a new Server between the Client and the Scheduler, which I'll call PreScheduler.

Each Client connects to a PreScheduler and uploads whole, unmaterialized, unoptimized, HLGs.
The PreScheduler materializes and optimizes the graph
After this point, everything is as @gjoseph92 described above: the PreScheduler internally defines order and sends keys to the scheduler on a as-needed basis

This whole new service is optional - it is straightforward to flick a switch to run everything it does inside either the Scheduler or the Client process/event loop, minus either one of the network connections
The PreScheduler runs near the Scheduler (e.g. on AWS).
You can have as many PreSchedulers as there are Clients.
You can write adaptive logic that starts and stops on demand new PreSchedulers on Kubernetes/other depending on the amount of Clients connected
You can also have a simpler system that spawns a new local process on the Scheduler host every time a new client connects.

The connection process goes as follows:

1a. A pool of statically started dask-prescheduler processes registers to the Scheduler, exactly like dask-worker does, OR
1b. Scheduler starts a fixed-size pool of subprocesses; once there are more Clients than subprocesses, more than one Client will start connecting to the same PreScheduler, OR
1c. Scheduler has the capability to spawn and destroy cloud containers running dask-prescheduler on demand
Client connects to Scheduler
The Scheduler might start a new process or cloud container; see point 1. This is invisible to the Client.
Scheduler informs Client of the IP:port of the PreScheduler
Client connects to PreScheduler
After the Client disconnects, wait a certain timeout (in case it reconnects) and then shut down the PreScheduler


Move Address Book to Patient Records etc.

There are too many instance of Address Book that we need to rename to VMS or something.
It is no longer fit for name for our purpose.
If anyone else have a better idea to replace Address Book to something else please put in the comments below. Don’t forget to like and subscribe!

I suggest PatientManager to match VaxTypeManager.
AddressBookStorage, JsonAddressBookStorage, etc will have to be renamed too.
liked and subed

nice


Allow using named pipes for qemu communication

Normally using unix sockets, but can also used named pipes (fifo)
Currently "pipe" doesn't work properly, so workaround with "null"

Enable unix socket support on Windows is now merged in the master (QEMU 7.2) https://github.com/qemu/qemu/commit/d409373b9d0482bbce312539d02a648cbb1c790a

I would prefer if someone from the community could take over and complete the windows port, leaving this PR as-is for now.


Forced Replacement of launch configurations without changes in code

I have issues
I'm submitting a...

[ ] bug report

What is the current behavior?
I have clusters created using terrafrom-aws-eks module. They are all running fine. Today when I went to do a code change for S3 bucket, I realised that all my launch configuration for EKS are also being re-created. I have not done any change to my EKS module nor EKS files, so I expect no changes when I run terraform plan. This is how my eks.tf looks like:
module "eks" {
  source = "../../../modules/aws/terraform-eks"

  env             = local.env
  cluster_name    = local.cluster_name
  cluster_version = 1.14

  subnets = module.vpc.private_subnets
  vpc_id  = module.vpc.vpc_id

  worker_groups = [
    {
      name                 = "terraform-1"
      instance_type        = "t3.large"
      key_name             = "eks-dev"
      bootstrap_extra_args = "--enable-docker-bridge true"
      kubelet_extra_args   = "--node-labels=env=Test,service=docker,team=platform"
      node_group_k8s_labels = {
        Environment = "Test"
        Team        = "Platform"
        Service     = "Docker"
                                                  }
      asg_max_size = "5" # Maximum worker capacity in the autoscaling group.
      asg_min_size = "0"
    }

    ]

  map_roles    = var.map_roles
  map_users    = var.map_users
}

Below is the output of plan:
  # module.eks.aws_launch_configuration.workers[0] must be replaced
+/- resource "aws_launch_configuration" "workers" {
        associate_public_ip_address      = false
        ebs_optimized                    = true
        enable_monitoring                = true
        iam_instance_profile             = "sennder-admin-cluster20200106121106098100000008"
      ~ id                               = "sennder-admin-cluster-terraform-12020010612110773630000000d" -> (known after apply)
        image_id                         = "ami-07034b303e1ffc843"
        instance_type                    = "t3.large"
        key_name                         = "eks-dev"
      ~ name                             = "sennder-admin-cluster-terraform-12020010612110773630000000d" -> (known after apply)
        name_prefix                      = "sennder-admin-cluster-terraform-1"
        security_groups                  = [
        ]
      ~ user_data_base64                 = "BIG BLOCK OF BASE64" -> (known after apply) # forces replacement
      - vpc_classic_link_security_groups = [] -> null

      + ebs_block_device {
          + delete_on_termination = (known after apply)
          + device_name           = (known after apply)
          + encrypted             = (known after apply)
          + iops                  = (known after apply)
          + no_device             = (known after apply)
          + snapshot_id           = (known after apply)
          + volume_size           = (known after apply)
          + volume_type           = (known after apply)
        }

      ~ root_block_device {
            delete_on_termination = true
          ~ encrypted             = false -> (known after apply)
            iops                  = 0
            volume_size           = 100
            volume_type           = "gp2"
        }
    }

  # module.eks.random_pet.workers[0] must be replaced
+/- resource "random_pet" "workers" {
      ~ id        = "amusing-parakeet" -> (known after apply)
      ~ keepers   = {
          - "lc_name" = "sennder-admin-cluster-terraform-12020010612110773630000000d"
        } -> (known after apply) # forces replacement
        length    = 2
        separator = "-"
    }

  # module.eks.local_file.kubeconfig[0] must be replaced
-/+ resource "local_file" "kubeconfig" {
      ~ content              = "" -> (known after apply) # forces replacement
        directory_permission = "0777"
        file_permission      = "0777"
        filename             = "./kubeconfig_sennder-admin-cluster"
      ~ id                   = "d2ea96f32318207ab27000f3d7ff80a70ae6b321" -> (known after apply)
    }

If this is a bug, how to reproduce? Please include a code sample if relevant.
I just did:
terraform init
terraform plan

What's the expected behavior?
I should get response similar to:

------------------------------------------------------------------------

No changes. Infrastructure is up-to-date.

This means that Terraform did not detect any differences between your
configuration and real physical resources that exist. As a result, no
actions need to be performed.

I also checked the User data for existing configurations and I see that it's not changed at all. I see same base64 encoded value of user data for both exiting launch config and the one terraform is creating.
Are you able to fix this problem and submit a PR? Link here if you have already.
Environment details
Terraform version:
Terraform v0.12.13
Terraform providers being used:
terraform {
  required_version = ">= 0.12.0"
}

provider "aws" {
  region  = var.region
}

provider "random" {
  version = "~> 2.1"
}

provider "local" {
  version = "~> 1.2"
}

provider "null" {
  version = "~> 2.1"
}

provider "template" {
  version = "~> 2.1"
}

resource "random_string" "suffix" {
  length  = 8
  special = false
}

Affected module version:
OS:
Terraform version:

Any other relevant info

I have noticed the same, modifying indirectly related resources forces a recreation of aws_launch_configuration with the plan showing (known after apply) # forces replacement for every attribute.

For every terraform apply? Like repeatedly?

AFter the changes are applied, it gets back to normal (for some weeks ) and then repeats. But it has happened repeatedly and It's a nightmare for production. So far we are not pushing any changes to production to maintain status quo.

It's because AWS released a new AMI for the worker nodes. By default the module picks the latest and replaces the launch configuration to use it.
If unexpected changes like this are bad for you, you should use one of the variable to lock the worker AMI you use: ami_id in the workers map or worker_ami_name_filter variable.

having the same issue, in my case it marks name_prefix as modified and forces replacement

also having the same issue. For us it looks like the directory permissions on the local file resources used for kubeconfig, that causes the force replacement.
`  # module.eks.local_file.kubeconfig[0] must be replaced
-/+ resource "local_file" "kubeconfig" {
...
EOT
+ directory_permission = "0777" # forces replacement
+ file_permission      = "0777" # forces replacement
filename             = "./k8-output/kubeconfig_ds---"
~ id                   = "46a63080b3-------b" -> (known after apply)`

I am also seeing this but it's just LC, not the ASG.

Funny though, if you do a second apply immediately after the first one, there is no further changes. But at some point changes come back 🤔

I am facing the same @max-rocket-internet . The way it works is that the LC thing causes change in launch configurations. Then the launch configurations causes changes in autoscaling groups. The launch configs are getting destroyed and recreated not the autoscaling groups.

Also seeing this. We just created few clusters with this module, without passing target_group_arns. Then after adding NLB in front of EKS, and adding target_group_arns to workers_group_defaults I'm seeing terraform wanting to replace LC and ASG, because of name_prefix change:
  # module.eks.aws_launch_template.workers_launch_template[2] must be replaced
+/- resource "aws_launch_template" "workers_launch_template" {
      ~ arn                     = "arn:aws:ec2:eu-north-1:123456789012:launch-template/lt-0968daaae8037e1c1" -> (known after apply)
      ~ default_version         = 1 -> (known after apply)
      - disable_api_termination = false -> null
      ~ ebs_optimized           = "true" -> (known after apply)
      ~ id                      = "lt-0968daaae8037e1c1" -> (known after apply)
      ~ image_id                = "ami-0f29ff24640121046" -> (known after apply)
      ~ instance_type           = "t3.large" -> (known after apply)
      + key_name                = (known after apply)
      ~ latest_version          = 2 -> (known after apply)
      ~ name                    = "example-apps-cluster-spot-320200227133342053200000007" -> (known after apply)
      ~ name_prefix             = "example-apps-cluster-spot-3" -> (known after apply) # forces replacement
      - security_group_names    = [] -> null
      ~ user_data               = "..." -> (known after apply)
      - vpc_security_group_ids  = [] -> null

      ~ block_device_mappings {
          ~ device_name = "/dev/xvda" -> (known after apply)

          ~ ebs {
                delete_on_termination = "true"
              ~ encrypted             = "true" -> (known after apply)
              ~ iops                  = 0 -> (known after apply)
              + kms_key_id            = (known after apply)
              ~ volume_size           = 100 -> (known after apply)
              ~ volume_type           = "gp2" -> (known after apply)
            }
        }

      ~ credit_specification {
          ~ cpu_credits = "standard" -> (known after apply)
        }

      ~ iam_instance_profile {
            name = "example-apps-cluster20200227133341819300000001"
        }

      + instance_market_options {
          + market_type = (known after apply)
        }

      ~ monitoring {
          ~ enabled = true -> (known after apply)
        }

      ~ network_interfaces {
          ~ associate_public_ip_address = "false" -> (known after apply)
          ~ delete_on_termination       = true -> (known after apply)
          - device_index                = 0 -> null
          - ipv4_address_count          = 0 -> null
          - ipv4_addresses              = [] -> null
          - ipv6_address_count          = 0 -> null
          - ipv6_addresses              = [] -> null
          ~ security_groups             = [
              - "sg-02b2d0fe4b621a404",
            ] -> (known after apply)
        }

      ~ placement {
          + group_name = (known after apply)
          ~ tenancy    = "default" -> (known after apply)
        }

      ~ tag_specifications {
            resource_type = "volume"
          ~ tags          = {
              - "Name"    = "example-apps-cluster-spot-3-eks_asg"
              - "Owner"   = "..."
              - "Project" = "..."
            } -> (known after apply)
        }
    }

ASG is recreated also because of name_prefix change (while it will be the same in result).

This issue is getting really annoying 😅 Does anyone have a fix?

@max-rocket-internet @Art3mK @clairefinnie : Changes in launch configurations are okay as long as it doesn't cause replacement of autoscaling groups
Also, the terraform module is picking up updated ami's for EKS workers. If you pin the version of AMI's the launch configs will change less number of time. As well keeping changes low for worker group section helps a bit.

@Krishna1408, not sure what you tried to explain.
IIUC changes in launch templates, like AMI id or smth shouldn't affect ASGs if asg_recreate_on_change is false. But I see different behavior, when ASG is recreated because of passing some related parameters like target group ARN, etc

Changes in launch configurations are okay as long as it doesn't cause replacement of autoscaling groups

I know that but it's still annoying. The LC or LT shouldn't be replaced unless there is changes to it, like AMI or user-data. But right now, it's regularly replaced without any relevant changes 🙂

This issue is getting really annoying 😅 Does anyone have a fix?

@max-rocket-internet please share your entire plan and debug output.

Hello everyone,
I seem to have the same issue with an aws_launch_configuration, but instead eks I am using ecs.
I seem to be able to reproduce the issue by running a GitLab-pipeline and then running terraform apply from a different machine manually.
Here is the unnecessary/unwanted part:
# module.ecs_cluster.aws_launch_configuration.ecs must be replaced
+/- resource "aws_launch_configuration" "ecs" {
     ~ arn                              = "arn:aws:autoscaling:eu-central-1:149004727281:launchConfiguration:d0311be1-6c75-4bbc-a741-492c1b61cc9a:launch
ConfigurationName/dev-ecs-app20200414100322327300000005" -> (known after apply)
       associate_public_ip_address      = false
     ~ ebs_optimized                    = false -> (known after apply)
       enable_monitoring                = true
       iam_instance_profile             = "dev-app"
     ~ id                               = "dev-ecs-app20200414100322327300000005" -> (known after apply)
       image_id                         = "<image-id>"
       instance_type                    = "t2.medium"
       key_name                         = "dev-app"
     ~ name                             = "dev-ecs-app20200414100322327300000005" -> (known after apply)
       name_prefix                      = "dev-ecs-app"
       security_groups                  = [
           "sg-00f9a1465484b7a1d",
       ]
     ~ user_data                        = "34bbc7fa5c88ba473b17166606b899856e6cbffa" -> "0fc8f322edb90a9e3145b27ebd0b4872a942303b" # forces replacement
     - vpc_classic_link_security_groups = [] -> null

     + ebs_block_device {
         + delete_on_termination = (known after apply)
         + device_name           = (known after apply)
         + encrypted             = (known after apply)
         + iops                  = (known after apply)
         + no_device             = (known after apply)
         + snapshot_id           = (known after apply)
         + volume_size           = (known after apply)
         + volume_type           = (known after apply)
       }

     ~ root_block_device {
           delete_on_termination = true
         ~ encrypted             = false -> (known after apply)
         ~ iops                  = 0 -> (known after apply)
           volume_size           = 30
           volume_type           = "standard"
       }
   }

As @barryib requested, I attached a debug output of terraform plan.
I had to anonymize some parts though. Does this help in order to find the issue?
plan_anonym.txt

@Krishna1408, not sure what you tried to explain.
IIUC changes in launch templates, like AMI id or smth shouldn't affect ASGs if asg_recreate_on_change is false. But I see different behavior, when ASG is recreated because of passing some related parameters like target group ARN, etc

@Art3mK I'm also facing the same issue. adding a new target group it recreates asg. were you able to overcome that ?

the same for me... some times i come to terraform to change soething not related to this module and i am getting
module.eks.random_pet.workers[0] must be replaced
module.eks.random_pet.workers[1] must be replaced
module.eks.aws_launch_configuration.workers[1] must be replaced
module.eks.aws_launch_configuration.workers[0] must be replaced
module.eks.aws_autoscaling_group.workers[1] will be updated in-place
module.eks.aws_autoscaling_group.workers[0] will be updated in-place

Same issue for us, as with all the replacement, Terraform tried to delete the worker security group and as it's attached to n number of eks workers/enis and other security groups, it can't be deleted and Terraform run failed and as per logs it ended up deleting a few security group rules. We had to rotate all the nodes to make the system work again.
The difference that showed in our apply was adding the name_prefix to worker security group, which when I checked in previous versions of statefile was always there!

Hallo another case of the same problem. How can we get more attention here?

This is currently impossible to debug without someone catching more information. "sometimes a thing gets updated, can't reproduce" isn't helpful. There could even be multiple sources of the "mysterious" updates.
What triggered the recreation of your launch template?
Is it because AWS released a new AMI version? That's expected. Lock the AMI version if it's a problem for you.
Is it a line ending change in the userdata script? Ensure terraform is only run from one infrastructure. Windows has different line endings to linux/mac!
Post full Terraform output, not just the launch template diff.

I am also having this problem in 2 different clusters that are configured using this module. I see that some changes are related to AMI not being specified but doesn't seem the only one.
Version: Terraform v0.13.1
Module Version: 12.2.0
Configuration:
module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "12.2.0"

  vpc_id          = module.vpc.vpc_id
  cluster_name    = var.cluster_name
  cluster_version = var.cluster_version
  subnets         = module.vpc.private_subnets

  tags = {
    Terraform   = "true"
    Environment = var.environment
  }

  cluster_encryption_config = [
    {
      provider_key_arn = aws_kms_key.eks.arn
      resources        = ["secrets"]
    }
  ]

  cluster_enabled_log_types     = ["api", "audit", "authenticator", "controllerManager", "scheduler"]
  cluster_log_retention_in_days = 30

  worker_groups = [
    {
      name                 = var.cluster_worker_group_name
      instance_type        = var.cluster_worker_group_instance_type
      asg_desired_capacity = var.cluster_worker_group_asg_desired_capacity
    }
  ]

  map_roles    = var.cluster_map_roles
  map_users    = var.cluster_map_users
  map_accounts = var.cluster_map_accounts
}

Change manifest:
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place
+/- create replacement and then destroy

Terraform will perform the following actions:

  # module.environment.module.eks.aws_autoscaling_group.workers[0] will be updated in-place
  ~ resource "aws_autoscaling_group" "workers" {
        arn                       = "arn:aws:autoscaling:us-east-2:XXX:autoScalingGroup:YYY:autoScalingGroupName/tst-eks-worker-group-default2020102915033234360000000d"
        availability_zones        = [
            "us-east-2a",
            "us-east-2b",
            "us-east-2c",
        ]
        default_cooldown          = 300
        desired_capacity          = 2
        enabled_metrics           = []
        force_delete              = false
        health_check_grace_period = 300
        health_check_type         = "EC2"
        id                        = "tst-eks-worker-group-default2020102915033234360000000d"
      ~ launch_configuration      = "tst-eks-worker-group-default20201029153833268400000001" -> (known after apply)
        load_balancers            = []
        max_instance_lifetime     = 0
        max_size                  = 3
        metrics_granularity       = "1Minute"
        min_size                  = 1
        name                      = "tst-eks-worker-group-default2020102915033234360000000d"
        name_prefix               = "tst-eks-worker-group-default"
        protect_from_scale_in     = false
        service_linked_role_arn   = "arn:aws:iam::XXX:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
        suspended_processes       = [
            "AZRebalance",
        ]
        tags                      = [
            {
                "key"                 = "Name"
                "propagate_at_launch" = "true"
                "value"               = "tst-eks-worker-group-default-eks_asg"
            },
            {
                "key"                 = "Terraform"
                "propagate_at_launch" = "true"
                "value"               = "true"
            },
            {
                "key"                 = "k8s.io/cluster/tst-eks"
                "propagate_at_launch" = "true"
                "value"               = "owned"
            },
            {
                "key"                 = "kubernetes.io/cluster/tst-eks"
                "propagate_at_launch" = "true"
                "value"               = "owned"
            },
        ]
        target_group_arns         = []
        termination_policies      = []
        vpc_zone_identifier       = [
            "subnet-111",
            "subnet-222",
            "subnet-333",
        ]
        wait_for_capacity_timeout = "10m"
    }

  # module.environment.module.eks.aws_launch_configuration.workers[0] must be replaced
+/- resource "aws_launch_configuration" "workers" {
      ~ arn                              = "arn:aws:autoscaling:us-east-2:XXX:launchConfiguration:XZZZ:launchConfigurationName/tst-eks-worker-group-default20201029153833268400000001" -> (known after apply)
        associate_public_ip_address      = false
        ebs_optimized                    = false
        enable_monitoring                = true
        iam_instance_profile             = "tst-eks20201029150315481600000008"
      ~ id                               = "tst-eks-worker-group-default20201029153833268400000001" -> (known after apply)
      ~ image_id                         = "ami-0dc6bc43da1b962d8" -> "ami-005ecc78d7cd1157d" # forces replacement
        instance_type                    = "t2.medium"
      + key_name                         = (known after apply)
      ~ name                             = "tst-eks-worker-group-default20201029153833268400000001" -> (known after apply)
        name_prefix                      = "tst-eks-worker-group-default"
        security_groups                  = [
            "sg-444",
        ]
        user_data_base64                 = "****"
      - vpc_classic_link_security_groups = [] -> null

      + ebs_block_device {
          + delete_on_termination = (known after apply)
          + device_name           = (known after apply)
          + encrypted             = (known after apply)
          + iops                  = (known after apply)
          + no_device             = (known after apply)
          + snapshot_id           = (known after apply)
          + volume_size           = (known after apply)
          + volume_type           = (known after apply)
        }

        root_block_device {
            delete_on_termination = true
            encrypted             = false
            iops                  = 0
            volume_size           = 100
            volume_type           = "gp2"
        }
    }

  # module.environment.module.eks.random_pet.workers[0] must be replaced
+/- resource "random_pet" "workers" {
      ~ id        = "touching-cheetah" -> (known after apply)
      ~ keepers   = {
          - "lc_name" = "tst-eks-worker-group-default20201029153833268400000001"
        } -> (known after apply) # forces replacement
        length    = 2
        separator = "-"
    }

Plan: 2 to add, 1 to change, 2 to destroy.

Almost any change to a launch_template will also cause a managed node group to be recreated if it has a random_pet in its name.
It goes badly against the idea of launch templates, as you should be able to update the node group to a newer version of the launch template. Instead, the whole node group is to be recreated (due to random_pet) - as a result the Managed Node Group is more like a WaitingToBeRecreated Node Group.
It would be great it was possible to define node_groups = { .. } without random_pet being involved at all. Even if I define explicit node group name name , name_prefix or any combination, the random_pet seems always to be there.

There is asg_recreate_on_change as a workaround for random_pet in workers (https://github.com/terraform-aws-modules/terraform-aws-eks/issues/1013#issuecomment-694738999), but such option doesn't  seem to be available for Managed Node Groups.

Running into same issue. What I'm doing:

Adding an ALB, introducing a new target_group
Adding that to the worker_group target_group_arns list.

Relevant part of plan output:
       metrics_granularity       = "1Minute"
        min_size                  = 2
      ~ name                      = "tools-ng-a20200305141842546200000016" -> (known after apply)
      ~ name_prefix               = "tools-ng-a" -> (known after apply) # forces replacement

Since aws_autoscaling_group name_prefix uses random_pet based on asg_recreate_on_change value, which is looked up from var.worker_groups_launch_template, defaulting to local.workers_group_defaults....
I tried setting asg_recreate_on_change = false in the worker_groups_launch_template, to rule out issues with lookup of defaults.
So, like this:
  worker_groups_launch_template = [
      {
        name                    = "ng-a"
        # previously was only set in
        asg_recreate_on_change  = false
      }
    ]

  workers_group_defaults = {
    public_ip = false
    subnets   = var.private_subnets

    asg_recreate_on_change = false
  }

This changed:

No more name_prefix change in asg. Resource 'will be updated in-place' (Good!)
Where previously 'plan' showed the target_group being added, now the target_group_arns in tf plan output shows no change (WTF!)

It's hardly getting any clearer...

Running into same issue. What I'm doing:

Adding an ALB, introducing a new target_group
Adding that to the worker_group target_group_arns list.

Relevant part of plan output:
       metrics_granularity       = "1Minute"
        min_size                  = 2
      ~ name                      = "tools-ng-a20200305141842546200000016" -> (known after apply)
      ~ name_prefix               = "tools-ng-a" -> (known after apply) # forces replacement

Since aws_autoscaling_group name_prefix uses random_pet based on asg_recreate_on_change value, which is looked up from var.worker_groups_launch_template, defaulting to local.workers_group_defaults....
I tried setting asg_recreate_on_change = false in the worker_groups_launch_template, to rule out issues with lookup of defaults.
So, like this:
  worker_groups_launch_template = [
      {
        name                    = "ng-a"
        # previously was only set in
        asg_recreate_on_change  = false
      }
    ]

  workers_group_defaults = {
    public_ip = false
    subnets   = var.private_subnets

    asg_recreate_on_change = false
  }

This changed:

No more name_prefix change in asg. Resource 'will be updated in-place' (Good!)
Where previously 'plan' showed the target_group being added, now the target_group_arns in tf plan output shows no change (WTF!)

It's hardly getting any clearer...

One aspect I'm seeing, and I'm sorry if I missed someone talking about it in above, is that the userdata changes due to line endings. All of our TF files use LF and the data coming back from AWS replaces that with CRLF instead. This makes it constantly want to update.

Hi, after reading almost all the comments on different issues regarding this, unable to find a proper solution to this.
Anyone with a workaround or a solution for this?


[Bug]: gemini_inference function definition in small_to_big_rag.ipynb notebook does not take chat argument

File Name
generative-ai/gemini/use-cases/retrieval-augmented-generation/small_to_big_rag/small_to_big_rag.ipynb
What happened?
The function definition should include chat argument since it's called with two arguments throughout rest of notebook (or chat argument needs to be deleted from callsites, not sure what right answer is)

Relevant log output
{
	"name": "TypeError",
	"message": "gemini_inference() takes 1 positional argument but 2 were given",
	"stack": "---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[9], line 1
----> 1 gemini_inference(chat, \"hello world!\")

TypeError: gemini_inference() takes 1 positional argument but 2 were given"
}

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Heads up, running into another issue, not sure if it's related: https://github.com/GoogleCloudPlatform/generative-ai/issues/622


Documentation for Installing EKS with Calico networking is incomplete

I have an EKS cluster that I wanted to replace the default AWS VPC CNI with Calico.  I followed the AWS documentation (https://docs.aws.amazon.com/eks/latest/userguide/alternate-cni-plugins.html) and it referred me to the tigera site for Installing EKS with Calico networking.  I followed the instructions on that page to delete the aws-node daemon set and install the helm chart for installing the tigera-operator operator.  After following all the steps, the nodes of my cluster were all in the NotReady state.  I poured over various documentation numerous times and various online posts until I found this ticket #5636 which indicated that I needed to modify the installation config to use the Calico CNI rather than AWS VPC CNI:
kind: Installation
apiVersion: operator.tigera.io/v1
metadata:
  name: default
spec:
  kubernetesProvider: EKS
  cni:
    type: Calico
  calicoNetwork:
    bgp: Disabled

This is completely missing from the documentation.  It would save newbies like me a lot of time if this was included as part of the steps for installing EKS with Calico networking using Helm.
Expected Behavior

Update the documentation to mention that the helm chart must be customized to use Calico as the CNI.
Current Behavior

Possible Solution

Steps to Reproduce (for bugs)

I followed the steps found here:  Install EKS with Calico networking for installing Calico CNI via a Helm chart.
Context

Your Environment

Calico version v3.26.0
Orchestrator version (e.g. kubernetes, mesos, rkt): EKS w/ Kubernetes 1.27
Operating System and version: Amazon Linux 2
Link to your project (optional):

Reading the linked doc: https://docs.tigera.io/calico/latest/getting-started/kubernetes/managed-public-cloud/eks#install-eks-with-calico-networking
Step 2 under the operator installation tab seems to already configure Calico CNI correctly.
This is what the documentation instructs you to do:
kubectl create -f - <<EOF
kind: Installation
apiVersion: operator.tigera.io/v1
metadata:
  name: default
spec:
  kubernetesProvider: EKS
  cni:
    type: Calico
  calicoNetwork:
    bgp: Disabled
EOF

What am I missing?

Hmm, I thought the various tabs:  Operator, Manifest, Helm were different installation options, i.e. if I followed the Helm instructions on the Helm tab, that would obviate any of the instructions in the Operator and Manifest tabs.  Are you saying that I have to apply the instructions from all tabs?
Thanks

Thanks for taking the time to write up the doc details. Agree that the EKS should have the step for configuring the Calico installation. We are working on a fix for this.

We've fixed the docs in this PR: https://github.com/tigera/docs/pull/756. Thanks again for letting us know about this bug.


Title overflow in symbols page

<EMAIL_ADDRESS>

https://github.com/denoland/deno_doc/pull/540 should fix this.

Closing as it has been addressed.