knife vsphere execute example

Versions:

Version of knife-vsphere: latest
Version of chef: 12.19.36

Platform Details

Version of vSphere: 6.0
Version of ESXi: 6.0

Scenario:

amitseth@~/chef-repo/rti-app-cookbooks: knife vsphere vm execute hostname-name "/bin/cat /etc/hosts" -f "Development/Miscellaneous" --exec-user root --exec-passwd "xxx&yyyyy" -VV
Please paste the command and output (run with -VV). Don't forget to X out any passwords or sensitive information:

INFO: Using configuration from /zzzz/knife.rb
DEBUG: value for config item vsphere_host:
DEBUG: value for config item vsphere_path: /sdk
DEBUG: value for config item vsphere_port: 443
DEBUG: value for config item vsphere_nossl:
DEBUG: value for config item vsphere_user:
DEBUG: value for config item vsphere_pass:
Enter your password:
DEBUG: value for config item vsphere_insecure:
DEBUG: value for config item proxy_host:
DEBUG: value for config item proxy_port:
/Users/amitseth/.chefdk/gem/ruby/2.3.0/gems/rbvmomi-1.9.5/lib/rbvmomi/vim.rb:32:in connect': host option required (RuntimeError) 	from /Users/amitseth/.chefdk/gem/ruby/2.3.0/gems/knife-vsphere-1.2.25/lib/chef/knife/base_vsphere_command.rb:117:in vim_connection'
from /Users/amitseth/.chefdk/gem/ruby/2.3.0/gems/knife-vsphere-1.2.25/lib/chef/knife/vsphere_vm_execute.rb:44:in run' 	from /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-12.18.31/lib/chef/knife.rb:443:in block in run_with_pretty_exceptions'
from /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-12.18.31/lib/chef/local_mode.rb:44:in with_server_connectivity' 	from /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-12.18.31/lib/chef/knife.rb:442:in run_with_pretty_exceptions'
from /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-12.18.31/lib/chef/knife.rb:219:in run' 	from /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-12.18.31/lib/chef/application/knife.rb:156:in run'
from /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-12.18.31/bin/knife:25:in <top (required)>' 	from /opt/chefdk/bin/knife:57:in load'
from /opt/chefdk/bin/knife:57:in `'
Expected Result:

I am trying to execute a command on the guest machine, using vsphere. Unable to do so . Can somebody point out a sample command.
Actual Result:

Hi, this command is somewhat limited. The output isn't passed back, personally I really only use it to run chef on a node that I can't reach or to turn off the firewall when I've really messed things up.
knife vsphere vm execute SERVERNAME  --exec-user root --exec-passwd rootpassword -- /bin/foo -bar is how I use it.

that is weird, It doesnot understand arguments to any binary . A simple echo failed for me
amitseth@~/xx: knife vsphere vm execute hostname  -f "Development/Miscellaneous" --exec-user root --exec-passwd "rootpass" -- "/bin/echo 'help'"
ERROR: RbVmomi::Fault: FileNotFound: File /bin/echo 'help' was not found

Try it without the quotes -- we have to pass args separately to the API. "foo 'bar'" literally tries to run a file called foo 'bar'. You want foo 'bar' if that makes sense.

Thank you for this. It does make sense. However when i am writing to a file on the guest vm , thats when I get bumped.
I am trying to set the interfaces for my cloned VM.  Trying to write to /etc/network/interfaces using this ,
knife vsphere vm execute SERVERNAME --exec-user root --exec-passwd rootpassword  /bin/echo 'blah >' /etc/network/interfaces
That doesnot do anything. Any suggestions.

Hm, that might be difficult. Redirections aren't really arguments to a command, they're things the shell does. I wonder if running /bin/bash -c "/bin/echo 'blah' > /etc/network/interfaces" might work? You'd be passing two arguments to bash... first is -c the second is the command you want bash to run.
Just curious if you need to do it this way? Typically this plugin will set the proper IPs when customizing the VM during the clone (--cips), or you're using something like Chef to manage the file.

the problem with cips is it only sets the /etc/hosts file , not the interfaces, if I want to set a static eth0, netmask, gateway, etc .
I am trying to set network config on clone. Do you have a better way to do this

You mean it's not setting the interface IP? Or is there a different setting it's missing? The --cips command should set the right IP on the interface, and once the node has booted for the first time, should be reachable. If not there's usually a problem with the template or the way people are calling the clone command.
Can you give me the command you're running to clone, please? And what distro of Linux are you running?
Couple of things to check, too:

Is the template configured to start vmware tools on boot?
Is the template set with the right distribution name?
There are a few files to wipe out if they exist (this is for Ubuntu: http://everythingshouldbevirtual.com/ubuntu-vmware-template-cleanup-script)

Thanks.

I see your point , I was using CentOS_6.7_v1.0.1 template with ethernet configurations in /etc/sysconfig/network-scripts .
i verified cips and cgw works.
knife vsphere vm clone servername --folder 'Templates' --dest-folder 'Development/Miscellaneous' --template 'CentOS_6.7_v1.0.1' --datastore VP2_Windows_Production_01 --chostname servername --cips <IP_ADDRESS>/24 --cgw <IP_ADDRESS>

Hm, that might be difficult. Redirections aren't really arguments to a command, they're things the shell does. I wonder if running /bin/bash -c "/bin/echo 'blah' > /etc/network/interfaces" might work? You'd be passing two arguments to bash... first is -c the second is the command you want bash to run.
...

For completeness, there is how this redirect worked for me:
knife vsphere vm execute VM1 --exec-user root --exec-passwd ROOTPWD -f 'Location' /bin/bash -- "-c 'echo \"it works\" > /tmp/tesme'"


Create publicação “2023-12-28-maisi”

Automatically generated by Decap CMS

👷 Deploy Preview for driveclean-teste processing.

Name
Link

🔨 Latest commit
0c8f119b5e011c268cb3bf43a6156876d064dd55

🔍 Latest deploy log
https://app.netlify.com/sites/driveclean-teste/deploys/658d6dbfb9a4b50008e32075


finding EOG/ ECG  artifacts

trying to use on EEG 60 channels data:
ecg_epochs = create_ecg_epochs(baseline)
without having a specific ecg channel, however it is not able to calculate a ECG channel.
create_eog_epochs(baseline, dict(mag=5e-12, grad=4000e-13),
picks='eeg').average()
also, not having an eog channel, no EOG channels are found using this.
When i try to specify a frontal channel in chan_name, same error.
I guess cannot be done if it's not MEG?
thanks

yes these functions do require to have an ECG or EOG channel to identify the artefact automatically.


DMN navigation messed up when adding more than 8 decisions

Describe the Bug
DMN navigation bar is messed up
Steps to Reproduce
// when
I add more than 8 decisions to a DRD
// then
the navigation bar is not in order any more

Expected Behavior
navigation bar keeps the order
Environment

Linux
Camunda Modeler Version: 3.0.0-beta3

Closed via https://github.com/camunda/camunda-modeler/pull/1321


[Question]: llm completion giving bad results

Question Validation

[X] I have searched both the documentation and discord for an answer.

Question
def messages_to_prompt(messages):
  prompt = ""
  for message in messages:
    if message.role == 'system':
      prompt += f"<|system|>\n{message.content}</s>\n"
    elif message.role == 'user':
      prompt += f"<|user|>\n{message.content}</s>\n"
    elif message.role == 'assistant':
      prompt += f"<|assistant|>\n{message.content}</s>\n"

  # ensure we start with a system prompt, insert blank if needed
  if not prompt.startswith("<|system|>\n"):
    prompt = "<|system|>\n</s>\n" + prompt
  # add final assistant prompt
  prompt = prompt + "<|assistant|>\n"
  return prompt

llm = HuggingFaceLLM(
    model_name="HuggingFaceH4/zephyr-7b-alpha",
    tokenizer_name="HuggingFaceH4/zephyr-7b-alpha",
    query_wrapper_prompt=PromptTemplate("<|system|>\n</s>\n<|user|>\n{query_str}</s>\n<|assistant|>\n"),
    context_window=3900,
    max_new_tokens=256,
    model_kwargs={"quantization_config": quantization_config},
    # tokenizer_kwargs={},
    generate_kwargs={"temperature": 0.7, "top_k": 50, "top_p": 0.95},
    messages_to_prompt=messages_to_prompt,
    device_map="cuda:0",
)

llm.system_prompt = """<|system|>
    You are an expert code extractor, your job is to extract the code from a given text and enclose the code between like this ```<extracted code```. Remove all the other content and return only the pure code  lines.
    """

true_res = llm.complete(f"Given the following text, if code is present extract all the code present and return in a string format. If no code is present then say <NONE> \n Text: {text}")

Using the completion with llamaindex gives very bad results, im trying to extract the code alone from the text but it return all the text back in the response, how to improve this

@JINO-ROHIT this has nothing to do with llama-indy really, and everything to do with the LLM and what your are asking it to do.
Either

do some iterations and prompt engineer a bit
find a different LLM (maybe codellama or something)
fine-tune a model

Also, I would recommend looking into using Ollama. The setup is 1000x easier (no need to worry about prompt templates)


Export clientOnly() at vike-solid/clientOnly

Fixes #89

Neat :sparkles: I think https://github.com/vikejs/vike-solid/blob/main/MIGRATION.md needs to be updated? Btw. in case you aren't happy with your release process: at vike-{react,vue} we use @brillout/release-me which automatically updates CHANGELOG.md — quite comfy.


Unable to download webpage

I haven't encountered this before and I've tried restarting my system, but I am unable to get any further past "downloading webpage"
derplex@whisky:~# time youtube-dl -v https://www.youtube.com/watch?v=WbLEYQo98t0
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-v', u'https://www.youtube.com/watch?v=WbLEYQo98t0']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.09.08
[debug] Python version 2.7.10 - Linux-3.14.32-xxxx-grs-ipv6-64-x86_64-with-debian-7.11
[debug] exe versions: ffmpeg 2.7.2, ffprobe 2.7.2
[debug] Proxy map: {}
[youtube] WbLEYQo98t0: Downloading webpage
^C
ERROR: Interrupted by user

real    1m9.863s
user    0m0.956s
sys     0m0.048s

I did receive this message beforehand after ~ only 5 seconds and I'm unable to reproduce it with the -v flag
derplex@whisky:~# youtube-dl -F https://www.youtube.com/watch?v=WbLEYQo98t0
[youtube] WbLEYQo98t0: Downloading webpage
ERROR: Unable to download webpage: <urlopen error EOF occurred in violation of protocol (_ssl.c:590)> (caused by URLError(SSLEOFError(8, u'EOF occurred in violation of protocol (_ssl.c:590)'),))

I managed to reproduce the error with the -v flag after waiting a bit longer.
derplex@whisky:~# youtube-dl -v https://www.youtube.com/watch?v=WbLEYQo98t0
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-v', u'https://www.youtube.com/watch?v=WbLEYQo98t0   ']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.09.08
[debug] Python version 2.7.10 - Linux-3.14.32-xxxx-grs-ipv6-64-x86_64-with-debia   n-7.11
[debug] exe versions: ffmpeg 2.7.2, ffprobe 2.7.2
[debug] Proxy map: {}
[youtube] WbLEYQo98t0: Downloading webpage
ERROR: Unable to download webpage: <urlopen error EOF occurred in violation of p   rotocol (_ssl.c:590)> (caused by URLError(SSLEOFError(8, u'EOF occurred in viol    tion of protocol (_ssl.c:590)'),))
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 394, in    _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1998, in urlop    n
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/root/.pyenv/versions/2.7.10/lib/python2.7/urllib2.py", line 431, in op    n
    response = self._open(req, data)
  File "/root/.pyenv/versions/2.7.10/lib/python2.7/urllib2.py", line 449, in _o    en
    '_open', req)
  File "/root/.pyenv/versions/2.7.10/lib/python2.7/urllib2.py", line 409, in _c    ll_chain
    result = func(*args)
  File "/usr/local/bin/youtube-dl/youtube_dl/utils.py", line 1004, in https_ope
    req, **kwargs)
  File "/root/.pyenv/versions/2.7.10/lib/python2.7/urllib2.py", line 1197, in d    _open
    raise URLError(err)
root@ns351858:~# youtube-dl -v https://www.youtube.com/watch?v=WbLEYQo98t0
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-v', u'https://www.youtube.com/watch?v=WbLEYQo98t0   ']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.09.08
[debug] Python version 2.7.10 - Linux-3.14.32-xxxx-grs-ipv6-64-x86_64-with-debia   n-7.11
[debug] exe versions: ffmpeg 2.7.2, ffprobe 2.7.2
[debug] Proxy map: {}
[youtube] WbLEYQo98t0: Downloading webpage
ERROR: Unable to download webpage: <urlopen error EOF occurred in violation of p   rotocol (_ssl.c:590)> (caused by URLError(SSLEOFError(8, u'EOF occurred in viol    tion of protocol (_ssl.c:590)'),))
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 394, in    _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1998, in urlop    n
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/root/.pyenv/versions/2.7.10/lib/python2.7/urllib2.py", line 431, in op    n
    response = self._open(req, data)
  File "/root/.pyenv/versions/2.7.10/lib/python2.7/urllib2.py", line 449, in _o    en
    '_open', req)
  File "/root/.pyenv/versions/2.7.10/lib/python2.7/urllib2.py", line 409, in _c    ll_chain
    result = func(*args)
  File "/usr/local/bin/youtube-dl/youtube_dl/utils.py", line 1004, in https_ope
    req, **kwargs)
  File "/root/.pyenv/versions/2.7.10/lib/python2.7/urllib2.py", line 1197, in d    _open
    raise URLError(err)

derplex@whisky:~# youtube-dl -v https://www.youtube.com/watch?v=WbLEYQo98t0
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-v', u'https://www.youtube.com/watch?v=WbLEYQo98t0   ']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.09.08
[debug] Python version 2.7.10 - Linux-3.14.32-xxxx-grs-ipv6-64-x86_64-with-debia   n-7.11
[debug] exe versions: ffmpeg 2.7.2, ffprobe 2.7.2
[debug] Proxy map: {}
[youtube] WbLEYQo98t0: Downloading webpage
ERROR: Unable to download webpage: <urlopen error EOF occurred in violation of p   rotocol (_ssl.c:590)> (caused by URLError(SSLEOFError(8, u'EOF occurred in viol    tion of protocol (_ssl.c:590)'),))
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 394, in    _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1998, in urlop    n
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/root/.pyenv/versions/2.7.10/lib/python2.7/urllib2.py", line 431, in op    n
    response = self._open(req, data)
  File "/root/.pyenv/versions/2.7.10/lib/python2.7/urllib2.py", line 449, in _o    en
    '_open', req)
  File "/root/.pyenv/versions/2.7.10/lib/python2.7/urllib2.py", line 409, in _c    ll_chain
    result = func(*args)
  File "/usr/local/bin/youtube-dl/youtube_dl/utils.py", line 1004, in https_ope
    req, **kwargs)
  File "/root/.pyenv/versions/2.7.10/lib/python2.7/urllib2.py", line 1197, in d    _open
    raise URLError(err)

FYI: https://github.com/rg3/youtube-dl/issues/4578#issuecomment-233425685 may help.

I tried installing with the edited utils.py and it's producing the same result.

Could you try Python 3?

This is the output when using python3 with the edited utils.py. Not sure why this is happening.
derplex@whisky~/youtube-dl/bin# python3 youtube-dl
Traceback (most recent call last):
  File "youtube-dl", line 3, in <module>
    import youtube_dl
ImportError: No module named youtube_dl

Seems your version is coupled with Python 2 tightly and can't be used with Python 3. Could you try to download https://yt-dl.org/downloads/latest/youtube-dl and run it via python3 ./youtube-dl?

Same result. I'm going to try a different system.
derplex@whisky:~/ytdl# time python3 youtube-dl -v https://www.youtube.com/watch?v=WbLEYQo98t0
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['-v', 'https://www.youtube.com/watch?v=WbLEYQo98t0']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.09.08
[debug] Python version 3.2.3 - Linux-3.14.32-xxxx-grs-ipv6-64-x86_64-with-Debian-7
[debug] exe versions: ffmpeg 2.7.2, ffprobe 2.7.2
[debug] Proxy map: {}
[youtube] WbLEYQo98t0: Downloading webpage
^C
ERROR: Interrupted by user

real    0m52.571s
user    0m0.924s
sys     0m0.040s

Usually the following command can be used to debug hang programs:
strace -f python3 ./youtube-dl -v https://www.youtube.com/watch?v=WbLEYQo98t0 2>& output.txt

Could you try and upload output.txt?

There's issues with ipv6 but I get the same hang with the -4 flag.
http://<IP_ADDRESS>/output.txt

Your device hangs when trying to connect to <IP_ADDRESS>. This IP looks a valid IP for www.youtube.com. Does ping <IP_ADDRESS> give you any response?

Perhaps YouTube have blocked my IP?
derplex@whisky~l# ping <IP_ADDRESS>
PING <IP_ADDRESS> (<IP_ADDRESS>) 56(84) bytes of data.
64 bytes from <IP_ADDRESS>: icmp_req=22 ttl=58 time=4.44 ms
^C
--- <IP_ADDRESS> ping statistics ---
33 packets transmitted, 1 received, 96% packet loss, time 32246ms
rtt min/avg/max/mdev = 4.445/4.445/4.445/0.000 ms

It's possible. Can you visit www.youtube.com in browsers?

It looks like they blocked me since I can connect to everything else except YouTube using my server as a proxy. It's a static IP so there's no changing it unless I get a new host. Any suggestion?

How about changing to another DNS? Maybe not all YouTube IPs are blocking you.

How could I do that?

A possible way is to change /etc/resolv.conf. Check Linux manuals.
I'm closing it first as it's not a youtube-dl issue. More discussions are welcome.


[spi_device] Add word initialization circuitry on SPI side

The SPI -> core buffer for the payload uses parity and SW has no way of initializing it since the the write port is in the SPI domain. Since the SPI side writes the payload byte by byte, we need to guard against partially initialized 32bit wordd, because these could cause TL-UL bus errors upon readout. Unfortunately, an initialization circuit that initializes the entire SRAM on the SPI clock domain is infeasible since that clock is only intermittently available. Hence, we keep track of uninitialized words using a valid bit array, and upon the first write to a word, uninitialized bytes are set to zero if the write operation is a sub-word write op.

CHANGE AUTHORIZED: hw/ip/spi_device/rtl/spid_dpram.sv

@dbeitel-opentitan this should take care of the SPI receive buffer initialization that can no longer be controlled by SW.
The buffer for the other direction remains unchanged and still needs initialization.

@msfschaffner What is the address offset, size, and name of the buffer for the other direction.

@msfschaffner What is the address offset, size, and name of the buffer for the other direction.

This would be the egress_buffer here:
https://github.com/lowRISC/opentitan/blob/f0acd25a7c3b14e0605c5a2c4477050897a294a2/hw/ip/spi_device/data/spi_device.hjson#L1399-L1415

CHANGE AUTHORIZED: hw/ip/spi_device/rtl/spid_dpram.sv

CC @neeraj-rv this is the RTL fix for SPI_DEVICE.

@msfschaffner The buffer you pointed me at is the buffer that is read only.  Which is the buffer that FW needs to initialize?

There is an egress_buffer and an ingress_buffer - the former should be WO, the latter RO.
You need to initialize the former via SW.


New evaluation workflow

Introduces new workflow for evaluating labeling functions and model results
Adds logging by combination of MindTagger and new utilities
Overhauls learning notebook
Fixes small bugs in previous pull

This will likely be iterated over in response to hackathon feedback, but:
Fixes #51
Fixes #43
Fixes #60

This looks awesome!
Let's find time to go through the notebook together (I looked at the code + skimmed the nb, but would be good to be more thorough + go through together), but I'll merge now so we can start getting feedback!


Split ComponentDoc into ComponentDoc and ComponentDocResolver

A small refactor to switch from structs to classes.
Specifically:

split ComponentDoc model into ComponentDoc and ComponentDocResolver
makes monkey patching the gem from an app easier (eg govuk-component-guide WIP: https://github.com/alphagov/govuk-component-guide/compare/component-gem#diff-bb6adab3eaf25dfc303b4f34455f5be9)

Addresses some of @alecgibson’s original review comments in https://github.com/alphagov/govuk_publishing_components/pull/1#discussion_r128566361

If we think this is going to happen quite a lot
I see it happening once, for static components that are different. But I like your idea of a subclass component resolver within the gem. I've been monkey patching as a proof of concept. We're also considering retiring govuk-component-guide and putting the gem in static. That would require something custom because static can't request from slimmer.

I see it happening once, for static components

If that's the case, then do it however you think is best - if it's just for one special case, a whole inheritance system may be overkill.

Thank you for the review @alecgibson ⭐


newfundsreq/recordobt increase max content

https://fioprotocol.atlassian.net/secure/RapidBoard.jspa?rapidView=2&projectKey=BD&modal=detail&selectedIssue=BD-2150

Maybe this should go into the final contract update after the deletion of the old table? Either way, I can rebase that PR once this gets merged.


User name structured as email address not invalidated at input time

When setting up timeld-cli with a user name that is an email address, a validation failure occurs that causes a later related issue.
The command timeld config --gateway timeld.org --user<EMAIL_ADDRESS>[1] appears to work, with no error shown.
It is then possible to go on to register an account using timeld config --account sunspark, with no error displayed.
The issue manifests when attempting to create a new timesheet thus: timeld open timesheet-1.  The response prompts the user to enter their email address: (Please enter your email address to register this device:).  When a valid address is submitted, the following error results:
Fetch from https://timeld.org/api/jwe/angus@m-ld.io?email=angus%40m-ld.io failed with 400: Bad Request

timeld config configures default options for forthcoming sessions (admin or timesheet). At the moment it does not validate the options – on the basic assumption that they will be validated when you actually do a timeld admin or timeld open.
There also seems to be a defect that timeld open is not correctly validating the user option before starting the device registration dance.
I will fix this latter defect as the priority, and then look into adding validation to the config.


[FEATURE] Once we have migrated flagd providers, remove old protos

Requirements
:warning: Depends on https://github.com/open-feature/flagd/issues/1029 :warning:
Once we have completed https://github.com/open-feature/flagd/issues/1029, we can start moving providers over to this new proto. Then we can finally remove support for the old protos.

Dupe of https://github.com/open-feature/flagd/issues/1088


analytics: move to a class.

Global namespaces are good to avoid when possible.

:+1:
I wonder if you would like to use this PR as an opportunity to add some tests for this analytics code. It's not exercised during normal usage, thus even simple typos can easily go unnoticed for a while. Setting HOMEBREW_ANALYTICS_DEBUG=1 or (probably even better) stubbing the curl invocation should make testing relatively painless.

@UniqMartin Will add that to my todo list rather than adding to this PR; this stuff is likely to change a bit around https://github.com/Homebrew/brew/issues/152.


Update BE api urls for newly provisioned environments

Update the Block Explorer API urls for:

[x] Devnet - https://be.devnet1.vega.xyz/rest/transactions
[x] Stagnet 1 - https://be.stagnet1.vega.xyz/rest/transactions
[x] Testnet - https://be.testnet.vega.xyz/rest/transactions
[x] Stagnet 3 - https://blockexplorer.n01.stagnet3.vega.xyz/rest/transactions
[ ] Stagnet 3 - https://be.stagnet3.vega.xyz
[x] Mainnet - TBC
[ ] Sandbox - TBC

Done for:

[x] .env files
[x] Netflify

Related to issue https://github.com/vegaprotocol/devops-infra/issues/1548#issuecomment-1293296516
and https://github.com/vegaprotocol/ansible/issues/344

https://github.com/vegaprotocol/ansible/issues/344 is now with Daniel to pick up and get sorted for us, Gordon has chased.


Mapping between existing and new system power management states

There's ongoing effort to improve Zephyr's power management solution, which includes switching to a standard set of states based on ACPI/Linux, to replace the current sleep/deep-sleep trinary solution that has no clear guidance for what the states imply (other than deep sleep means state is lost).
To that end we need to determine the state mapping for any SoC that currently supports the existing power states.  This issue is a place for all SoC maintainers and interested parties to propose mappings and develop a common understanding of what the new states mean and how they'll be used.
Note these basic assumptions:

PM_STATE_ACTIVE means the CPU is running normally.
PM_STATE_RUNTIME_IDLE corresponds to what Zephyr normally does when the idle thread is the only active thread.  E.g. for ARM this is blocking in WFE/WFI for some interrupt to occur.

In Linux terms these are both runtime states, and neither is under control of Zephyr's system power management infrastructure.
The real system states used when power management is enabled are below, and existing uses of POWER_STATE{,_DEEP}_SLEEP_{1,2,3} should map to one of these states:

PM_STATE_SUSPEND_TO_IDLE
PM_STATE_STANDBY
PM_STATE_SUSPEND_TO_RAM
PM_STATE_SUSPEND_TO_DISK
PM_STATE_SOFT_OFF

The map does not need to be consistent for all SoCs; e.g. that vendor A's POWER_STATE_SLEEP_1 corresponds to PM_STATE_SUSPEND_TO_IDLE while vendor B maps it to PM_STATE_STANDBY is not a concern.  We just want the existing supported states to use the new state names.  This will enable @ceolin and others to prepare commits that move us toward the new states.
Discussion that led to the selection of these states can be found in #24228, #27315, #29505.  Where there is ambiguity the references in https://github.com/zephyrproject-rtos/zephyr/issues/24228#issuecomment-663584642 may be useful to determine intended behavior.
In-tree SoCs/boards for which maps are required, and the person tapped to provide information, include:

[ ] TI simplelink/cc13xx
[ ] MEC15XX
[ ] Nordic (@pabigot)
[ ] SiLabs exx32
[ ] stm32l4 and stm32wb

For Nordic boards I believe all existing states would map to PM_STATE_SOFT_OFF  as the only non-runtime sleep mode on most if not all Nordic devices is a system-off where some RAM data may be retained but the system shut down and wake only on an external signal from a GPIO or other peripheral (not including a timer).
PM_STATE_SUSPEND_TO_RAM could be supported if/when #31087 is addressed and infrastructure is added to detect the wakeup cause and restore the application and its peripherals to an active state that includes state inherited from the previous session.

For Nordic boards I believe all existing states would map to PM_STATE_SOFT_OFF  as the only non-runtime sleep mode on most if not all Nordic devices is a system-off where some RAM data may be retained but the system shut down and wake only on an external signal from a GPIO or other peripheral (not including a timer).
PM_STATE_SUSPEND_TO_RAM could be supported if/when #31087 is addressed and infrastructure is added to detect the wakeup cause and restore the application and its peripherals to an active state that includes state inherited from the previous session.

^^@erwango

^^@erwango

Here is STM32 mapping (based on STM32L4/STM3WB):

Stop 0, Stop 1, Stop 2 > PM_STATE_SUSPEND_TO_IDLE
Standby with SRAM2 > PM_STATE_SUSPEND_TO_RAM (requires RAM retention support)
Standby and Shutdown > PM_STATE_SOFT_OFF

Tbh, I'm not sure clean on the frontier between PM_STATE_SUSPEND_TO_IDLE and  PM_STATE_STANDBY so Stop 2 could be PM_STATE_STANDBY.
Low Power Run and Low Power Sleep modes could not be matched with the proposed states. They are similar to PM_STATE_ACTIVE and PM_STATE_RUNTIME_IDLE with a clock frequency reduced to 2MHz.
One question: how can wen handle multiple states mapping to a single PM state (injective mapping) ?
Today we have the flexibility of mapping to POWER_STATE_SLEEP_X, but IIUC we lose this flexibility with the new PM states.

Here is STM32 mapping (based on STM32L4/STM3WB):

Stop 0, Stop 1, Stop 2 > PM_STATE_SUSPEND_TO_IDLE
Standby with SRAM2 > PM_STATE_SUSPEND_TO_RAM (requires RAM retention support)
Standby and Shutdown > PM_STATE_SOFT_OFF

Tbh, I'm not sure clean on the frontier between PM_STATE_SUSPEND_TO_IDLE and  PM_STATE_STANDBY so Stop 2 could be PM_STATE_STANDBY.
Low Power Run and Low Power Sleep modes could not be matched with the proposed states. They are similar to PM_STATE_ACTIVE and PM_STATE_RUNTIME_IDLE with a clock frequency reduced to 2MHz.
One question: how can wen handle multiple states mapping to a single PM state (injective mapping) ?
Today we have the flexibility of mapping to POWER_STATE_SLEEP_X, but IIUC we lose this flexibility with the new PM states.

The existing SiLabs mapping has:

POWER_STATE_SLEEP_1: EM1 Sleep
POWER_STATE_SLEEP_2: EM2 Deep Sleep
POWER_STATE_SLEEP_3: EM3 Stop

Reviewing the semantics of these this doesn't seem quite ideal.  The best mapping I can come up informed by this overview and the EFR32xG12 reference manual is:

PM_STATE_ACTIVE: EM0
PM_STATE_RUNTIME_IDLE: EM1
PM_STATE_SUSPEND_TO_IDLE: Either EM2 or EM3.  The SiLabs functional distinction does not appear to match well to ACPI state distinction.
PM_STATE_STANDBY: Same as PM_STATE_SUSPEND_TO_IDLE
PM_STATE_SUSPEND_TO_RAM: EM4 Hibernate
PM_STATE_SUSPEND_TO_DISK: N/A
PM_STATE_SOFT_OFF: EM4 Shutoff

It appears waking due to elapsed time (not Zephyr system clock) is possible from all of these states.  I suspect full capability will require API specific to a SiLabs custom power management policy that does not distinguish between PM_STATE_SUSPEND_TO_IDLE and PM_STATE_STANDBY, and instead determines which of EM2 and EM3 should be entered based on constraints provided by the application.  This could be handled by treating PM_STATE_SUSPEND_TO_IDLE as EM2 and PM_STATE_STANDBY as EM3 but that is not justified
I haven't used these chips since they were Energy Micro, so @chrta or @mnkp may be able to provide more up-to-date insight.

The existing SiLabs mapping has:

POWER_STATE_SLEEP_1: EM1 Sleep
POWER_STATE_SLEEP_2: EM2 Deep Sleep
POWER_STATE_SLEEP_3: EM3 Stop

Reviewing the semantics of these this doesn't seem quite ideal.  The best mapping I can come up informed by this overview and the EFR32xG12 reference manual is:

PM_STATE_ACTIVE: EM0
PM_STATE_RUNTIME_IDLE: EM1
PM_STATE_SUSPEND_TO_IDLE: Either EM2 or EM3.  The SiLabs functional distinction does not appear to match well to ACPI state distinction.
PM_STATE_STANDBY: Same as PM_STATE_SUSPEND_TO_IDLE
PM_STATE_SUSPEND_TO_RAM: EM4 Hibernate
PM_STATE_SUSPEND_TO_DISK: N/A
PM_STATE_SOFT_OFF: EM4 Shutoff

It appears waking due to elapsed time (not Zephyr system clock) is possible from all of these states.  I suspect full capability will require API specific to a SiLabs custom power management policy that does not distinguish between PM_STATE_SUSPEND_TO_IDLE and PM_STATE_STANDBY, and instead determines which of EM2 and EM3 should be entered based on constraints provided by the application.  This could be handled by treating PM_STATE_SUSPEND_TO_IDLE as EM2 and PM_STATE_STANDBY as EM3 but that is not justified
I haven't used these chips since they were Energy Micro, so @chrta or @mnkp may be able to provide more up-to-date insight.

One question: how can wen handle multiple states mapping to a single PM state (injective mapping) ?
Today we have the flexibility of mapping to POWER_STATE_SLEEP_X, but IIUC we lose this flexibility with the new PM states.

I think we should plan to have SOC-specific default power management policies, that support API extensions to allow applications to control which specific mode is used to implement a Zephyr standard mode.  I don't know yet what the rest of the power management API will look like so can't propose how the extensions would work.

One question: how can wen handle multiple states mapping to a single PM state (injective mapping) ?
Today we have the flexibility of mapping to POWER_STATE_SLEEP_X, but IIUC we lose this flexibility with the new PM states.

I think we should plan to have SOC-specific default power management policies, that support API extensions to allow applications to control which specific mode is used to implement a Zephyr standard mode.  I don't know yet what the rest of the power management API will look like so can't propose how the extensions would work.

Here is STM32 mapping (based on STM32L4/STM3WB):

Stop 0, Stop 1, Stop 2 > PM_STATE_SUSPEND_TO_IDLE
Standby with SRAM2 > PM_STATE_SUSPEND_TO_RAM (requires RAM retention support)
Standby and Shutdown > PM_STATE_SOFT_OFF

Tbh, I'm not sure clean on the frontier between PM_STATE_SUSPEND_TO_IDLE and PM_STATE_STANDBY so Stop 2 could be PM_STATE_STANDBY.
Low Power Run and Low Power Sleep modes could not be matched with the proposed states. They are similar to PM_STATE_ACTIVE and PM_STATE_RUNTIME_IDLE with a clock frequency reduced to 2MHz.
One question: how can wen handle multiple states mapping to a single PM state (injective mapping) ?
Today we have the flexibility of mapping to POWER_STATE_SLEEP_X, but IIUC we lose this flexibility with the new PM states.

Currently is not flexible to be fair, we have a fixed number of POWER_STATE_SLEEP_. It just happen that the number of these states is enough for this platform, but it could not be the case. But yeah, that is something we need to think about, when ACPI-like states don't map well. Let me think over the weekend about it.

Here is STM32 mapping (based on STM32L4/STM3WB):

Stop 0, Stop 1, Stop 2 > PM_STATE_SUSPEND_TO_IDLE
Standby with SRAM2 > PM_STATE_SUSPEND_TO_RAM (requires RAM retention support)
Standby and Shutdown > PM_STATE_SOFT_OFF

Tbh, I'm not sure clean on the frontier between PM_STATE_SUSPEND_TO_IDLE and PM_STATE_STANDBY so Stop 2 could be PM_STATE_STANDBY.
Low Power Run and Low Power Sleep modes could not be matched with the proposed states. They are similar to PM_STATE_ACTIVE and PM_STATE_RUNTIME_IDLE with a clock frequency reduced to 2MHz.
One question: how can wen handle multiple states mapping to a single PM state (injective mapping) ?
Today we have the flexibility of mapping to POWER_STATE_SLEEP_X, but IIUC we lose this flexibility with the new PM states.

Currently is not flexible to be fair, we have a fixed number of POWER_STATE_SLEEP_. It just happen that the number of these states is enough for this platform, but it could not be the case. But yeah, that is something we need to think about, when ACPI-like states don't map well. Let me think over the weekend about it.

It just happen that the number of these states is enough for this platform, but it could not be the case.

We need to be careful here.  The state names have meanings, and are intended to map to ACPI semantics.
If some SoC has three power levels that all correspond in basic behavior to PM_STATE_STANDBY, we must not arbitrarily assign them to IDLE or SUSPEND states just because those enumerations are not being used for their documented purpose.  That would be worse than providing 3 arbitrary versions of SLEEP, because at least there we "knew" they all did the same basic thing, but with different side effects.
I really think this sort of distinction needs to be SOC-specific internal substates of the standard Zephyr states, managed by SoC-specific policy implementations.  When using a Zephyr-wide default/generic policy PM_STATE_STANDBY would be allowed, and you'd get whatever single state the SoC maintainer best matches that, and IDLE and SUSPEND states would not be supported.

It just happen that the number of these states is enough for this platform, but it could not be the case.

We need to be careful here.  The state names have meanings, and are intended to map to ACPI semantics.
If some SoC has three power levels that all correspond in basic behavior to PM_STATE_STANDBY, we must not arbitrarily assign them to IDLE or SUSPEND states just because those enumerations are not being used for their documented purpose.  That would be worse than providing 3 arbitrary versions of SLEEP, because at least there we "knew" they all did the same basic thing, but with different side effects.
I really think this sort of distinction needs to be SOC-specific internal substates of the standard Zephyr states, managed by SoC-specific policy implementations.  When using a Zephyr-wide default/generic policy PM_STATE_STANDBY would be allowed, and you'd get whatever single state the SoC maintainer best matches that, and IDLE and SUSPEND states would not be supported.

Regarding Silabs EXX32:
I am not experienced with the low-power modes, but i think the overview https://github.com/zephyrproject-rtos/zephyr/issues/31162#issuecomment-757337571 is correct. The main differences between EM2 and EM3 is which peripherals are still active and therefore which events trigger a wake-up.
I also think that the decision if EM2 or EM3 should be used, must be up to the application.
Is there documentation how wake-up sources should be configured in zephyr's power management solution? I mean should there be a timer running and/or should wake-up work if something happens on a peripheral e.g. UART? For EXX32 this would mean that the peripheral drivers need to be adapted for this.

Regarding Silabs EXX32:
I am not experienced with the low-power modes, but i think the overview https://github.com/zephyrproject-rtos/zephyr/issues/31162#issuecomment-757337571 is correct. The main differences between EM2 and EM3 is which peripherals are still active and therefore which events trigger a wake-up.
I also think that the decision if EM2 or EM3 should be used, must be up to the application.
Is there documentation how wake-up sources should be configured in zephyr's power management solution? I mean should there be a timer running and/or should wake-up work if something happens on a peripheral e.g. UART? For EXX32 this would mean that the peripheral drivers need to be adapted for this.

Here is STM32 mapping (based on STM32L4/STM3WB):

Stop 0, Stop 1, Stop 2 > PM_STATE_SUSPEND_TO_IDLE
Standby with SRAM2 > PM_STATE_SUSPEND_TO_RAM (requires RAM retention support)
Standby and Shutdown > PM_STATE_SOFT_OFF

Is there overall power state description/mapping for STM32, so this could be applied to all STM32 SoCs?  (e.g. STM32L1x only has one stop mode, which also would map to PM_STATE_SUSPEND_TO_IDLE)

Here is STM32 mapping (based on STM32L4/STM3WB):

Stop 0, Stop 1, Stop 2 > PM_STATE_SUSPEND_TO_IDLE
Standby with SRAM2 > PM_STATE_SUSPEND_TO_RAM (requires RAM retention support)
Standby and Shutdown > PM_STATE_SOFT_OFF

Is there overall power state description/mapping for STM32, so this could be applied to all STM32 SoCs?  (e.g. STM32L1x only has one stop mode, which also would map to PM_STATE_SUSPEND_TO_IDLE)

It just happen that the number of these states is enough for this platform, but it could not be the case.

We need to be careful here. The state names have meanings, and are intended to map to ACPI semantics.

Yes, completely agree.

If some SoC has three power levels that all correspond in basic behavior to PM_STATE_STANDBY, we must not arbitrarily assign them to IDLE or SUSPEND states just because those enumerations are not being used for their documented purpose. That would be worse than providing 3 arbitrary versions of SLEEP, because at least there we "knew" they all did the same basic thing, but with different side effects.

Didn't say to do that. The dilemma is, STATE_SLEEP_1, STATE_SLEEP_2, ... were meaningless from the application perspective. I mean, one could not infer what was the difference between them, and different platforms could do different things. in other hand when we move to ACPI-like states, an application know  exactly what a power state is supposed to do, but we lose flexibility on the platform implementation.

I really think this sort of distinction needs to be SOC-specific internal substates of the standard Zephyr states, managed by SoC-specific policy implementations. When using a Zephyr-wide default/generic policy PM_STATE_STANDBY would be allowed, and you'd get whatever single state the SoC maintainer best matches that, and IDLE and SUSPEND states would not be supported.

It just happen that the number of these states is enough for this platform, but it could not be the case.

We need to be careful here. The state names have meanings, and are intended to map to ACPI semantics.

Yes, completely agree.

If some SoC has three power levels that all correspond in basic behavior to PM_STATE_STANDBY, we must not arbitrarily assign them to IDLE or SUSPEND states just because those enumerations are not being used for their documented purpose. That would be worse than providing 3 arbitrary versions of SLEEP, because at least there we "knew" they all did the same basic thing, but with different side effects.

Didn't say to do that. The dilemma is, STATE_SLEEP_1, STATE_SLEEP_2, ... were meaningless from the application perspective. I mean, one could not infer what was the difference between them, and different platforms could do different things. in other hand when we move to ACPI-like states, an application know  exactly what a power state is supposed to do, but we lose flexibility on the platform implementation.

I really think this sort of distinction needs to be SOC-specific internal substates of the standard Zephyr states, managed by SoC-specific policy implementations. When using a Zephyr-wide default/generic policy PM_STATE_STANDBY would be allowed, and you'd get whatever single state the SoC maintainer best matches that, and IDLE and SUSPEND states would not be supported.

good reference for STM:

https://www.st.com/content/ccc/resource/training/technical/product_training/ce/57/a3/86/7a/3d/4d/87/STM32L4_System_Power.pdf/files/STM32L4_System_Power.pdf/jcr:content/translations/en.STM32L4_System_Power.pdf

good reference for STM:

https://www.st.com/content/ccc/resource/training/technical/product_training/ce/57/a3/86/7a/3d/4d/87/STM32L4_System_Power.pdf/files/STM32L4_System_Power.pdf/jcr:content/translations/en.STM32L4_System_Power.pdf

don't forget a state for DEAD_BATT situations as this usually requires some specials for bringing up the system, i.e. could be Type-C PD negotiation or early boot-sequence USB-enumeration to happen before you can enter PM_STATE_ACTIVE

don't forget a state for DEAD_BATT situations as this usually requires some specials for bringing up the system, i.e. could be Type-C PD negotiation or early boot-sequence USB-enumeration to happen before you can enter PM_STATE_ACTIVE

don't forget a state for DEAD_BATT situations as this usually requires some specials for bringing up the system

Does that deserve a specific PM state?  It's not something the generic power management infrastructure would be aware of.  SoC-specific startup has to detect conditions on startup and ensure that the system is brought up in an active mode, which might be a degraded substate of PM_STATE_ACTIVE.

don't forget a state for DEAD_BATT situations as this usually requires some specials for bringing up the system

Does that deserve a specific PM state?  It's not something the generic power management infrastructure would be aware of.  SoC-specific startup has to detect conditions on startup and ensure that the system is brought up in an active mode, which might be a degraded substate of PM_STATE_ACTIVE.

I saw it more as a quite common entry-point where you'd want to stay until x-number of conditions are met to continue. Having it part of the PM would be nice for structure, logging options but also in multi-core systems where one core or physical mcu might want to ask for the PM state

I saw it more as a quite common entry-point where you'd want to stay until x-number of conditions are met to continue. Having it part of the PM would be nice for structure, logging options but also in multi-core systems where one core or physical mcu might want to ask for the PM state

@galak Added to dev-revew as a better anchor issue for discussion of https://github.com/zephyrproject-rtos/zephyr/pull/31273#issuecomment-761086654, which is why that PR is in dev-review.
For context, my position remains that we need to support MCUs that have multiple power configurations at runtime, not just when idle.  This follows from https://github.com/zephyrproject-rtos/zephyr/issues/31162#issuecomment-758631584 where we see STM32 has three active states (80 MHz, 26 MHz, 2 MHz), that have two runtime idle states, none of which are idle states from the perspective of Zephyr.
I believe the power management infrastructure is ideally positioned to support transition between the variant active states, and consequently devicetree description of power states should not be limited to those that happen to be what Zephyr (or Linux) calls idle states.

@galak Added to dev-revew as a better anchor issue for discussion of https://github.com/zephyrproject-rtos/zephyr/pull/31273#issuecomment-761086654, which is why that PR is in dev-review.
For context, my position remains that we need to support MCUs that have multiple power configurations at runtime, not just when idle.  This follows from https://github.com/zephyrproject-rtos/zephyr/issues/31162#issuecomment-758631584 where we see STM32 has three active states (80 MHz, 26 MHz, 2 MHz), that have two runtime idle states, none of which are idle states from the perspective of Zephyr.
I believe the power management infrastructure is ideally positioned to support transition between the variant active states, and consequently devicetree description of power states should not be limited to those that happen to be what Zephyr (or Linux) calls idle states.

@nashif @ceolin shouldn't that issue be closed? I think the main issue described here has been addressed.

@gmarull yep.  I don't see any reason to keep it open, we can always reference it even after closed.


Fix for issue #2590 : Fixes conflicts with ERROR macro in wingdi.h

Issue #, if available: #2590
Description of changes:
Check all that applies:

[ ] Did a review by yourself.
[ ] Added proper tests to cover this PR. (If tests are not applicable, explain.)
[ ] Checked if this PR is a breaking (APIs have been changed) change.
[ ] Checked if this PR will not introduce cross-platform inconsistent behavior.
[ ] Checked if this PR would require a ReadMe/Wiki update.

Check which platforms you have built SDK on to verify the correctness of this PR.

[ ] Linux
[ ] Windows
[ ] Android
[ ] MacOS
[ ] IOS
[ ] Other Platforms

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Thank you for going this route instead of simply undefining things as has been done historically and was recommended by the bug poster!

https://github.com/aws/aws-sdk-cpp/blob/6cc48868b7558265fd095ce338ce37320a2968e2/src/aws-cpp-sdk-core/source/smithy/tracing/impl/opentelemetry/TraceAdapters.cpp#L56
This line also needs to be change for a successful build


chore: fixing broken links, updating to correct base url, replacing gitter with github discussions

more cleanups and fixes around links
removing gitter in favour of github discussions

Can we add a task like this to the lint?

Can we add a task like this to the lint?

https://github.com/open-telemetry/opentelemetry-js-contrib/issues/396


Publish to CocoaPods

[ ] Publish the SDK to CocoaPods
[ ] Add additional maintainers (e.g. @vherrin, @mkistler)
[ ] Update documentation to reference CocoaPods installation

+1 for that

+1

+1

+1

+1 for that.

+1

+1

Sorry for the delay everyone. I'm currently working on this, so Cocoapods will be supported in the near future 🙂

Cocoapods is now supported as of version 0.33.0!


fix village path

fix https://github.com/GTNewHorizons/GT-New-Horizons-Modpack/issues/10689

The author posted a solution.


Internal _read_sql function hangs for PostgreSQL13 via bouncer in GCloud

Problem:
_read_sql (used via polars) ends up hanging in my production system (specified in the context section), while it works perfectly fine in my local environment (Macos, Posgresql 13, Python 3.9.7).
I couldn't get past that function, as I guess it's implemented in Rust, and pdb is obviously unable to go inside.
Regarding the DB state, it's arguably the same (in local dev env, and in production), as I run a set of deterministic migrations, and the DB is pretty new, so no differences in contained data either.
Context:

Connector-X version: 0.2.0 (latest at this moment)
Pod inside Google Cloud
Python 3.9.7 , running on a Debian slim base image.
The function is called in the context of an async function (but there are no other concurrent calls to _read_sql).
Accessing PostgreSQL 13 DB, via a "bouncer" node (that exists to deal with high number of connections).
The DB load is ridiculously low.
Pod resources usage is low, no resources constraints are affecting it.
In fact, CPU usage drops to 0 when the execution flow reaches _read_sql (while memory usage stays the same).

My pdb session, it hangs on the last line, you can see some variables values I displayed:
(Pdb) b /usr/local/src/app/.venv/lib/python3.9/site-packages/connectorx/__init__.py:122
Breakpoint 1 at /usr/local/src/app/.venv/lib/python3.9/site-packages/connectorx/__init__.py:122
(Pdb) b /usr/local/src/app/.venv/lib/python3.9/site-packages/connectorx/__init__.py:128
Breakpoint 2 at /usr/local/src/app/.venv/lib/python3.9/site-packages/connectorx/__init__.py:128
(Pdb) c
2021-10-01 09:16:55.748 | INFO     | __main__:xxx - Bunch of irrelevant logs
2021-10-01 09:16:55.748 | INFO     | __main__:xxx - Bunch of irrelevant logs
2021-10-01 09:16:55.748 | INFO     | __main__:xxx - Bunch of irrelevant logs
> /usr/local/src/app/.venv/lib/python3.9/site-packages/connectorx/__init__.py(122)read_sql()
-> elif return_type in {"arrow", "polars"}:
(Pdb) display queries
display queries: ['SELECT jobs.job_id, jobs.is_remote, jobs.employment_type_id, jobs.language_id, jobs.category_id, jobs.country_id, jobs.industry_id, jobs.contract_type_id, jobs.function_level_id, jobs.number_of_employees_id \nFROM jobs \nWHERE jobs.target_market_id = 1 AND jobs.has_premium_orders = true ORDER BY jobs.job_id']
(Pdb) display protocol
display protocol: 'binary'
(Pdb) display partition_query
display partition_query: None
(Pdb) s
> /usr/local/src/app/.venv/lib/python3.9/site-packages/connectorx/__init__.py(123)read_sql()
-> try:
(Pdb) s
> /usr/local/src/app/.venv/lib/python3.9/site-packages/connectorx/__init__.py(124)read_sql()
-> import pyarrow
(Pdb) s
> /usr/local/src/app/.venv/lib/python3.9/site-packages/connectorx/__init__.py(128)read_sql()
-> result = _read_sql(
(Pdb) s
> /usr/local/src/app/.venv/lib/python3.9/site-packages/connectorx/__init__.py(129)read_sql()
-> conn,
(Pdb) s
> /usr/local/src/app/.venv/lib/python3.9/site-packages/connectorx/__init__.py(130)read_sql()
-> "arrow",
(Pdb) s
> /usr/local/src/app/.venv/lib/python3.9/site-packages/connectorx/__init__.py(131)read_sql()
-> queries=queries,
(Pdb) s
> /usr/local/src/app/.venv/lib/python3.9/site-packages/connectorx/__init__.py(132)read_sql()
-> protocol=protocol,
(Pdb) s
> /usr/local/src/app/.venv/lib/python3.9/site-packages/connectorx/__init__.py(133)read_sql()
-> partition_query=partition_query,
(Pdb) s
> /usr/local/src/app/.venv/lib/python3.9/site-packages/connectorx/__init__.py(128)read_sql()
-> result = _read_sql(
(Pdb) s

Discussion
I'd be happy to continue helping to pinpoint what's the root cause of this problem. I have some experience with Rust, but I've never worked on Rust+Python projects. If there's anything I can do to debug the problem more in depth, I'll be thankful for the information.

@castarco thanks for the report and the details! I'm not sure in which stage the function hangs. Can you update connectorx to the newest alpha version via pip install -U connectorx==0.2.1a1 and set the RUST_LOG environment variable before importing the library like the following example:
import os
os.environ["RUST_LOG"]="connectorx=debug,connectorx_python=debug"
import connectorx as cx

df = cx.read_sql(postgres_url, query, return_type="polars")

There should be some output logs from connectorx and connectorx_python. Can you show them here?

@wangxiaoying I get the following once it enters into the function, and it hangs after the last line is printed:
[2021-10-02T19:56:00Z DEBUG connectorx_python::arrow] Protocol: binary
[2021-10-02T19:56:00Z DEBUG connectorx_python::arrow] Running dispatcher
[2021-10-02T19:56:00Z DEBUG connectorx::dispatcher] Fetching metadata
[2021-10-02T19:56:00Z DEBUG connectorx::sql] Transformed limit 1 query: SELECT jobs.job_id, jobs.is_remote, jobs.employment_type_id, jobs.language_id, jobs.category_id, jobs.country_id, jobs.industry_id, jobs.contract_type_id, jobs.function_level_id, jobs.number_of_employees_id FROM jobs WHERE jobs.target_market_id = 1 AND jobs.has_premium_orders = true ORDER BY jobs.job_id LIMIT 1
[2021-10-02T19:56:00Z DEBUG connectorx::dispatcher] Prepare partitions
[2021-10-02T19:56:00Z DEBUG connectorx::sql] Transformed count query: SELECT count(*) FROM (SELECT jobs.job_id, jobs.is_remote, jobs.employment_type_id, jobs.language_id, jobs.category_id, jobs.country_id, jobs.industry_id, jobs.contract_type_id, jobs.function_level_id, jobs.number_of_employees_id FROM jobs WHERE jobs.target_market_id = 1 AND jobs.has_premium_orders = true) AS CXTMPTAB_COUNT
[2021-10-02T19:56:00Z DEBUG connectorx::dispatcher] Allocate destination memory
[2021-10-02T19:56:00Z DEBUG connectorx::dispatcher] Create destination partition
[2021-10-02T19:56:00Z DEBUG connectorx::dispatcher] Partition 0, 298x10
[2021-10-02T19:56:00Z DEBUG connectorx::dispatcher] Start writing
[2021-10-02T19:56:00Z DEBUG connectorx::dispatcher] Finalize partition 0
[2021-10-02T19:56:00Z DEBUG connectorx::dispatcher] Partition 0 finished

It seems to me that the destructor stucked at https://github.com/sfu-db/connector-x/blob/main/connectorx/src/dispatcher.rs#L153, because the line 155 debug!("Writing finished"); is not printed out.
To narrow it down, could you please help us try df = cx.read_sql(postgres_url, query, return_type="pandas") to see if it is the source (postgres) or destination (arrow) that hangs the whole process?

This is the output for the experiment you proposed (using "pandas" as return type, instead of "arrow" or "polars"):
[2021-10-03T06:48:09Z DEBUG connectorx_python::pandas] Protocol: binary
[2021-10-03T06:48:09Z DEBUG connectorx_python::pandas] Running dispatcher
[2021-10-03T06:48:09Z DEBUG connectorx::dispatcher] Fetching metadata
[2021-10-03T06:48:09Z DEBUG connectorx::sql] Transformed limit 1 query: SELECT jobs.job_id, jobs.is_remote, jobs.employment_type_id, jobs.language_id, jobs.category_id, jobs.country_id, jobs.industry_id, jobs.contract_type_id, jobs.function_level_id, jobs.number_of_employees_id FROM jobs WHERE jobs.target_market_id = 1 AND jobs.has_premium_orders = true ORDER BY jobs.job_id LIMIT 1
[2021-10-03T06:48:09Z DEBUG connectorx::dispatcher] Prepare partitions
[2021-10-03T06:48:09Z DEBUG connectorx::sql] Transformed count query: SELECT count(*) FROM (SELECT jobs.job_id, jobs.is_remote, jobs.employment_type_id, jobs.language_id, jobs.category_id, jobs.country_id, jobs.industry_id, jobs.contract_type_id, jobs.function_level_id, jobs.number_of_employees_id FROM jobs WHERE jobs.target_market_id = 1 AND jobs.has_premium_orders = true) AS CXTMPTAB_COUNT
[2021-10-03T06:48:09Z DEBUG connectorx::dispatcher] Allocate destination memory
[2021-10-03T06:48:09Z DEBUG connectorx::dispatcher] Create destination partition
[2021-10-03T06:48:09Z DEBUG connectorx::dispatcher] Partition 0, 298x10
[2021-10-03T06:48:09Z DEBUG connectorx::dispatcher] Start writing
[2021-10-03T06:48:09Z DEBUG connectorx::dispatcher] Finalize partition 0
[2021-10-03T06:48:09Z DEBUG connectorx::dispatcher] Partition 0 finished

@castarco Thanks. This helps us narrow down the bug to either rayon or the Postgres.


Now the Script button will be set correctly when the selection changed

See Issue #22998

See #20590, I'm not sure this is actually doing something wanted.

Yeah, @bugfi5h As per the other Issue, the "clear script" button has been removed in favor of being able to directly extend scripts that are already on a node. If you want to clear a script, you must either right click the node in the Scene dock and use the "Clear Script" context menu option or scroll to the bottom of the Inspector and "Clear" the script property on the Node directly.

@akien-mga We can close this Issue.


Should noaccess user login success?

The test cases in test_ldap_configuration.robot try to use "noaccess" user login redfish, and expect get error while patch something to redfish.
But if we set a user to "noaccess" group, should we give it login or logout permission(create/delete session)?

ref:
https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/49295
https://github.com/openbmc/bmcweb/issues/227

@prkatti1 Can you check on this

once the new function is implemented successfully, no-access user will not have any permissions not even login & logout.
Once that is merged, we need to change the expected behavior of test cases involving no-access user.

@prkatti1 is this still a thing for upstream?

ref: https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/49295 openbmc/bmcweb#227

This commit is abandoned.. so closing the ticket. No further action required


Install OKD 4.6 on vSphere UPI failed

Describe the bug
openshift-install wait-for install-complete failed with message:
Cluster operator authentication Degraded is True with IngressStateEndpoints_MissingSubsets::OAuthRouteCheckEndpointAccessibleController_SyncError::OAuthServerDeployment_GetFailed::OAuthServiceCheckEndpointAccessibleController_SyncError::OAuthServiceEndpointsCheckEndpointAccessibleController_SyncError::OAuthVersionDeployment_GetFailed::Route_FailedGet: OAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready
OAuthServiceCheckEndpointAccessibleControllerDegraded: Get "https://<IP_ADDRESS>:443/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
IngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server
OAuthRouteCheckEndpointAccessibleControllerDegraded: Get "https://oauth-openshift.apps.okd4-spb.sigma-it.local/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
OAuthServerDeploymentDegraded: Unable to get "openshift-browser-client" bootstrapped CLI OAuth client: the server is currently unable to handle the request (post oauthclients.oauth.openshift.io)
RouteDegraded: Unable to get required route openshift-authentication/oauth-openshift: the server is currently unable to handle the request (get routes.route.openshift.io oauth-openshift)
OAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps "oauth-openshift" not found"
$ oc get co

$ oc get nodes

$ oc get pods -n openshift-authentication
No resources found in openshift-authentication namespace.
Version
4.6.0-0.okd-2020-12-12-135354 on vSphere UPI
How reproducible
reproducible - 100%
Log bundle
must-gather

Get "https://<IP_ADDRESS>:443/healthz\": dial tcp <IP_ADDRESS>:443: i/o timeout (Client.Timeout exceeded while awaiting headers)

Seems OpenShiftSDN installs are broken - service won't reply

With the OVNKubernetes network plugin, cluster was installed successfully.

Lets track this in #430


Feature/ehr 163 support canonical for datavalue

This is a super set of EHR-450 with fixes to integrate the requested change, plus fixed EHR-163, EHR-159, EHR-160, EHR-145, full composition support.
NB. There are still some issues with AQL expression based on full composition content (f.e. a/content[xyz]) however this is circumvented by querying a known item (f.e. observation, section etc. defined by a contains clause)

When creating the following query:
{ "q": "select a as test from EHR e contains COMPOSITION a contains OBSERVATION a_a[openEHR-EHR-OBSERVATION.time_series.v1]" }
I find the following format in the path and its name:
{ "q": "select\n a as test\nfrom EHR e\ncontains COMPOSITION a\ncontains OBSERVATION a_a[openEHR-EHR-OBSERVATION.time_series.v1]", "name": null, "columns": [ { "path": "/ehr.js_composition(composition_join.id,'local.ethercis.com')::text", "name": "#0" }

Please check with the updates, name and path should be consistent with the specs.

The following query:
{ "q": "select a_a/data[at0001]/events[at0002]/data[at0003]/item from EHR e contains COMPOSITION a contains OBSERVATION a_a[openEHR-EHR-OBSERVATION.time_series.v1]" }
provides the following result:
{ "q": "select\n a_a/data[at0001]/events[at0002]/data[at0003]/item\nfrom EHR e\ncontains COMPOSITION a\ncontains OBSERVATION a_a[openEHR-EHR-OBSERVATION.time_series.v1]", "name": null, "columns": [ { "path": "/FIELD_0", "name": "#0" } ], "rows": [ [ null ], [ null ], [ null ] ] }
I think this should provide a single row with null.
The issue is that the provided path a_a/data[at0001]/events[at0002]/data[at0003]/item is not correct and depending on your data this is expanded into an array search (guess with a valid path, 3 values/structure should be return). A clean fix would be to actually validate the path before executing the query. I suggest to open a ticket for this.


[ic-footer-link] in Safari footer links don't get focus

Summary of the bug
Footer links don't get focus by tabbing or by hovering.
🪜 How to reproduce
Tell us the steps to reproduce the problem:

Go to any footer example with links on Safari
Tab or hover
See the error

🧐 Expected behaviour
Footer links should get the same offset underline that regular links do.

Linked to 1910


Add basic UI test for apps

Given https://github.com/android/health-samples/issues/209, we should have a minimal test that launches the app at least, but ideally starts and stops exercise or tracking.
cc @ithinkihaveacat

Another example https://github.com/android/health-samples/pull/265, cc @breanatate


Wrong step referenced?

[Enter feedback here]

Task 8: Build object search index
Build the object search index to make objects able to be searched from Tell Me in the client. If you completed step 6, you can skip this step.

Step 6 is setting the application version.  I suspect you mean Step 7 that updates the pages/reports from the MenuSuite.
Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: b69c11d2-4915-8e2e-f38c-7bb1a3777bf9
Version Independent ID: 0028111c-4f1a-df24-0d47-48b7f6e6d98a
Content: Upgrade Application Code - Business Central
Content Source: dev-itpro/upgrade/Upgrading-the-Application-Code.md
Service: dynamics365-business-central
GitHub Login: @jswymer
Microsoft Alias: jswymer

Hello, good catch. Thanks for the feedback! An update will be published soon.


Helm operator errors: failed to get API Group-Resources and create a new manager

Using project "whcs-cag" on server "https://api.brief-macaque.purple-chesterfield.com:6443"
I'm running a helm operator using the operator-sdk 0.16.0. I'm struggling to get the correct reconciliation of the various image versions included in my custom resource when the custom resource is updated.
I recently added a new api version as a delta to the v1alpha1 which I had been using to see if that helped. Once that version was added, I started getting these errors and the operator pod will not start. Not sure if it's related or how, but hoping someone can help me debug what's happening.
Chriss-MacBook-Pro2:whcs-service-clinical-data-annotator-bundle-tc-icp cagrev$ oc logs whcs-acd-operator-69547b75fd-ttpwn -n whcs-cag
{"level":"info","ts":1587673083.5200763,"logger":"cmd","msg":"Go Version: go1.13.8"}
{"level":"info","ts":1587673083.5201018,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":1587673083.5201063,"logger":"cmd","msg":"Version of operator-sdk: v0.16.0"}
{"level":"info","ts":1587673083.5203946,"logger":"cmd","msg":"Watching single namespace.","Namespace":"whcs-cag"}
{"level":"error","ts":1587673083.535982,"logger":"controller-runtime.manager","msg":"Failed to get API Group-Resources","error":"Unauthorized","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tpkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/manager.New\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/manager/manager.go:238\ngithub.com/operator-framework/operator-sdk/pkg/helm.Run\n\tsrc/github.com/operator-framework/operator-sdk/pkg/helm/run.go:109\ngithub.com/operator-framework/operator-sdk/cmd/operator-sdk/execentrypoint.newRunHelmCmd.func1\n\tsrc/github.com/operator-framework/operator-sdk/cmd/operator-sdk/execentrypoint/helm.go:38\ngithub.com/spf13/cobra.(*Command).execute\n\tpkg/mod/github.com/spf13/cobra@v0.0.5/command.go:826\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tpkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914\ngithub.com/spf13/cobra.(*Command).Execute\n\tpkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864\nmain.main\n\tsrc/github.com/operator-framework/operator-sdk/cmd/operator-sdk/main.go:39\nruntime.main\n\t/home/travis/.gimme/versions/go1.13.8.linux.amd64/src/runtime/proc.go:203"}
{"level":"error","ts":1587673083.5360653,"logger":"cmd","msg":"Failed to create a new manager.","Namespace":"whcs-cag","error":"Unauthorized","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tpkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/helm.Run\n\tsrc/github.com/operator-framework/operator-sdk/pkg/helm/run.go:111\ngithub.com/operator-framework/operator-sdk/cmd/operator-sdk/execentrypoint.newRunHelmCmd.func1\n\tsrc/github.com/operator-framework/operator-sdk/cmd/operator-sdk/execentrypoint/helm.go:38\ngithub.com/spf13/cobra.(*Command).execute\n\tpkg/mod/github.com/spf13/cobra@v0.0.5/command.go:826\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tpkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914\ngithub.com/spf13/cobra.(*Command).Execute\n\tpkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864\nmain.main\n\tsrc/github.com/operator-framework/operator-sdk/cmd/operator-sdk/main.go:39\nruntime.main\n\t/home/travis/.gimme/versions/go1.13.8.linux.amd64/src/runtime/proc.go:203"}
Error: Unauthorized
Usage:
operator-sdk exec-entrypoint helm [flags]

watches.yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
labels:
app.kubernetes.io/instance: ibm-whcs-service-acd-operator
app.kubernetes.io/managed-by: ibm-whcs-service-acd-operator
app.kubernetes.io/name: ibm-whcs-service-acd-operator
release: v1
name: acds.acdoperator.wh-acd.ibm.com
spec:
group: acdoperator.wh-acd.ibm.com
names:
kind: Acd
listKind: AcdList
plural: acds
singular: acd
scope: Namespaced
subresources:
status: {}
scale:
specReplicasPath: .spec.replicas
statusReplicasPath: .status.replicas
labelSelectorPath: .status.labelSelector
#  validation:
#    openAPIV3Schema:
#      type: object
#      x-kubernetes-preserve-unknown-fields: true
versions:

name: v1alpha1
served: true
storage: false
name: v1alpha202004231932
served: true
storage: true

watches.yaml
- version: v1alpha1
  group: acdoperator.wh-acd.ibm.com
  kind: Acd
  chart: /opt/helm/charts/ibm-whcs-service-acd-chart
- version: v1alpha202004231932
  group: acdoperator.wh-acd.ibm.com
  kind: Acd
  chart: /opt/helm/charts/ibm-whcs-service-acd-chart

acdoperator_v1alpha1_acdbinding_cr.yaml
apiVersion: acdoperator.wh-acd.ibm.com/v1alpha202004231932
kind: Acd
metadata:
  labels:
    app.kubernetes.io/instance: ibm-whcs-service-acd-operator
    app.kubernetes.io/managed-by: ibm-whcs-service-acd-operator
    app.kubernetes.io/name: ibm-whcs-service-acd-operator
    release: v1
  name: whcs-acd
  .
  .
  .

Once installed, my CRD shows two stored versions (the v1alpha1, which I had previously been using, and the new one). That looks correct to me.
kind: CustomResourceDefinition
apiVersion: apiextensions.k8s.io/v1
metadata:
  name: acds.acdoperator.wh-acd.ibm.com
  selfLink: >-
    /apis/apiextensions.k8s.io/v1/customresourcedefinitions/acds.acdoperator.wh-acd.ibm.com
  uid: 552117b9-2ec7-4659-8a80-1e66f4ee01c3
  resourceVersion: '14434711'
  generation: 3
  creationTimestamp: '2020-04-13T16:18:27Z'
  labels:
    app.kubernetes.io/instance: ibm-whcs-service-acd-operator
    app.kubernetes.io/managed-by: ibm-whcs-service-acd-operator
    app.kubernetes.io/name: ibm-whcs-service-acd-operator
    release: v1
spec:
  group: acdoperator.wh-acd.ibm.com
  names:
    plural: acds
    singular: acd
    kind: Acd
    listKind: AcdList
  scope: Namespaced
  versions:
    - name: v1alpha1
      served: true
      storage: false
      subresources:
        status: {}
        scale:
          specReplicasPath: .spec.replicas
          statusReplicasPath: .status.replicas
          labelSelectorPath: .status.labelSelector
    - name: v1alpha202004232105
      served: true
      storage: true
      subresources:
        status: {}
        scale:
          specReplicasPath: .spec.replicas
          statusReplicasPath: .status.replicas
          labelSelectorPath: .status.labelSelector
  conversion:
    strategy: None
  preserveUnknownFields: true
status:
  conditions:
    - type: NamesAccepted
      status: 'True'
      lastTransitionTime: '2020-04-13T16:18:27Z'
      reason: NoConflicts
      message: no conflicts found
    - type: Established
      status: 'True'
      lastTransitionTime: '2020-04-13T16:18:27Z'
      reason: InitialNamesAccepted
      message: the initial names have been accepted
  acceptedNames:
    plural: acds
    singular: acd
    kind: Acd
    listKind: AcdList
  storedVersions:
    - v1alpha1
    - v1alpha202004232105

When I go through the installed operator via the GUI and try to do a create of the CR, the yaml is prepopulated with this version (before I press the Create button). Note that it loads the v1alpha1 apiVersion, when this version of the operator was built with the v1alpha202004232105 apiVersion.
apiVersion: acdoperator.wh-acd.ibm.com/v1alpha1
kind: Acd
metadata:
  labels:
    app.kubernetes.io/instance: ibm-whcs-service-acd-operator
    app.kubernetes.io/managed-by: ibm-whcs-service-acd-operator
    app.kubernetes.io/name: ibm-whcs-service-acd-operator
    release: v1
  name: whcs-acd
  namespace: whcs-cag
spec:
  replicas: 1
  acd:
    deployment:
      updateStrategy:
        type: Recreate
      context: clinical_data_annotator
    probes:
      livenessProbe:
        exec:
          command:
            - /staging/whcs-devops-ucd/docker/docker_liveness.sh
        initialDelaySeconds: 180
        periodSeconds: 30
        timeoutSeconds: 10
        failureThreshold: 10
      readinessProbe:
        exec:
          command:
            - /staging/whcs-devops-ucd/docker/docker_readiness.sh
        initialDelaySeconds: 30
        periodSeconds: 60
        timeoutSeconds: 10
        failureThreshold: 3
    image:
      tag: '20200402.1556'
      repository: us.icr.io/cdt-ls-acd-rns/clinical-data-annotator
      containerHttpsPort: 9443
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
  .
  .
  .

When I go directly to the CR by searching for it in the GUI and open up a window to do a create there, I see it loads with this version, the one that I would expect.
apiVersion: acdoperator.wh-acd.ibm.com/v1alpha202004232105
kind: Acd
metadata:
  name: example
  namespace: default
spec: {}

@cagrev Sorry you're having issues! It seems like there are two potential issues:

The error message is coming from a very early step of the operator startup process that sets up the kubernetes clients. I've never personally seen that issue, but it sounds like a permissions issue. Did you deploy the role and rolebinding associated with the operator?
I think the multi-version CRD setup you have may be confusing things. Given the error message, I don't think things are progressing far enough for that to matter, but I'd recommended configuring just a single version in your CRD and watches.yaml file. I'm not positive because we haven't tested many multi-version CRD scenarios with the Helm operator, but I don't think it would work as you would expect with multiple version and multiple entries in watches.yaml.

Just as a sanity check, are you able to follow the examples in helm-operator user guide successfully?

So how are multiple versions supposed to work with the helm operator? I would like to understand that. When the versions to a given image reference within the CR change, does the apiVersion need to change? If not, and left at v1alpha1 for example, then how does the controller reconcile whether there are updates required or not?
On the error, I deleted everything on the system in a couple namespaces where I had the operator and operand pods up and running. With that, I was able to get around the error and get the operator pods to start again. The role and rolebinding were/are deployed. They have been working for months and haven't had any recent changes.
Beyond the apiVersion changes, I've not introduced anything new to our functional operator, although I moved from the operator-sdk v0.10.0 to v0.16.0 a couple weeks ago.

@cagrev

So how are multiple versions supposed to work with the helm operator? I would like to understand that.

Short answer is that multi-version CRDs likely don't work with the helm-operator. In practice, multi-version CRDs are reconciled as a single version on-cluster, so you would have only a single reconciler for a CRD.  The cluster handles multiple versions by using a CRD conversion webhook that is typically provided by the operator. For helm-operator, there isn't a simple way to implement this webhook using native helm primatives.

When the versions to a given image reference within the CR change, does the apiVersion need to change? If not, and left at v1alpha1 for example, then how does the controller reconcile whether there are updates required or not?

In general, the only time you need to change the CRD version is when the actual spec fields change in a breaking way. The helm operator is constantly reconciling. So when a new version of the operator is deployed with a chart where the default operand image tag changes, the operator will detect the change and do a helm release upgrade.
Under the hood, the helm-operator does a dry-run upgrade from the CR values and the chart in the operator image to get the desired release manifest. It compares the desired release manifest with the existing release manifest and does an actual upgrade if they differ.

Ok, thanks for the info. So let's step back a minute, away from the multi-version CRDs. I've gone back to one version of v1alpha1. The changes that are being made in the custom resource are these tags for the various microservice images we are using. With each operator build, one of these tags changes to a newer version and the operator image image itself gets a new version.
Chriss-MacBook-Pro2:whcs-service-clinical-data-annotator-bundle-tc-icp cagrev$ cat ./stable/ibm-whcs-service-acd-bundle/operators/ibm-whcs-service-acd-operator/deploy/crds/acdoperator_v1alpha1_acdbinding_cr.yaml | grep tag:
      tag: "20200427.1204" (spec.acd.image.tag)
      tag: "20200420.0506" (spec.aci.image.tag)
      tag: "20200427.1204" (spec.av.image.tag)
      tag: "20200424.2159" (spec.cd.image.tag)
      tag: "20200423.2019" (spec.cds.image.tag)
      tag: "20200420.0700" (spec.cv.image.tag)
      tag: "20200420.0700" (spec.hyp.image.tag)
      tag: "20200427.1739" (spec.neg.image.tag)
      tag: "20200420.0901" (spec.ont.image.tag)
      tag: "20200420.0900" (spec.spl.image.tag)

I think you are saying that the reconciliation should be happening and a helm release upgrade should be taking place as the version of the operator itself is updated. On the system where the operator is running, I'm getting the operator version update, and see that pod cycle, but I'm not getting the others to be upgraded as I would expect.

A version of the operator from 4/20 is currently deployed.

Here's what is running now.
Version from deployment resource:  image: us.icr.io/cdt-ls-acd-rns/clinical-data-annotator:20200402.1556
Version from deployment resource:  image: us.icr.io/cdt-ls-acd-rns/advanced-care-insights:20200402.1053
Version from deployment resource:  image: us.icr.io/cdt-ls-acd-rns/attribute-detection:20200402.1400
Version from deployment resource:  image: us.icr.io/cdt-ls-acd-rns/concept-detection:20200402.1400
Version from deployment resource:  image: us.icr.io/cdt-ls-acd-rns/concept-disambiguation:20200330.0659
Version from deployment resource:  image: us.icr.io/cdt-ls-acd-rns/concept-value-detection:20200402.1400
Version from deployment resource:  image: us.icr.io/cdt-ls-acd-rns/hypothetical-detection:20200402.0744
Version from deployment resource:  image: us.icr.io/cdt-ls-acd-rns/negation-detection:20200317.1247
Version from deployment resource:  image: us.icr.io/cdt-ls-acd-rns/ontology:20200330.0901
Version from deployment resource:  image: us.icr.io/cdt-ls-acd-rns/acd-operator:v1.0.202004202253
Version from deployment resource:  image: us.icr.io/cdt-ls-acd-rns/spell-checker:20200402.1817

I just bumped my catalog image to a more current one so a newer version of the operator should be picked up too. I'll share more after the upgrades complete for it.

Ah one other potential issue. If the CR that you've deployed to the cluster specifies the tag values, those will be used rather than the chart's default values. Make sure to remove the tag values from your CR if you want to use the chart's default values.

I do want the CR tag values to be used, but they don't seem to be now.
Here's the result from the newer catalog image, which triggered an update to the operator, but not the rest of them.

Version deployed (per call to status api):  whcs-acd-acd-deployment-575b87dc44-hxrhb 2020-04-02T16:00:26Z 0d 01:02:39
Version deployed (per call to status api):  whcs-acd-aci-deployment-7fd744b8d9-qhmtw 2020-04-02T15:00:24Z 0d 01:02:43
Version deployed (per call to status api):  whcs-acd-av-deployment-545b49574b-4gf6q 2020-04-02T14:04:54Z 0d 01:02:46
Version deployed (per call to status api):  whcs-acd-cd-deployment-59747fdd8c-525nq 2020-04-02T14:05:42Z 0d 01:00:56
Version deployed (per call to status api):  whcs-acd-cds-deployment-586d58669c-6sdvf 2020-03-30T13:06:04Z 0d 01:02:49
Version deployed (per call to status api):  whcs-acd-cv-deployment-6849584c6d-hd4q5 2020-04-02T14:05:41Z 0d 01:02:51
Version deployed (per call to status api):  whcs-acd-hyp-deployment-6777cbb488-r7n2j 2020-04-02T12:49:28Z 0d 01:02:50
Version deployed (per call to status api):  whcs-acd-neg-deployment-54f868b49c-kzkff 2020-03-17T12:51:22Z 0d 01:02:28
Version deployed (per call to status api):  whcs-acd-ont-deployment-64cc6fdfc7-c2xdz 2020-03-30T09:06:21Z 0d 01:02:23
Version deployed (per call to status api):  whcs-acd-spl-deployment-bf58769b-g24bk 2020-04-02T18:21:58Z 0d 01:02:57

I have a stable channel set up that uses an older version of our operator. At the time, with that version, we were still using latest for each of the images in the custom resource. You can see that one is still pulling the latest as expected/configured.
Chriss-MacBook-Pro2:whcs-service-clinical-data-annotator-bundle-tc-icp cagrev$ ./stable/ibm-whcs-service-acd-bundle/whcs-devops/kubernetes/get_deployed_versions.sh
Deployment names:  whcs-acd-acd-deployment whcs-acd-aci-deployment whcs-acd-av-deployment whcs-acd-cd-deployment whcs-acd-cds-deployment whcs-acd-cv-deployment whcs-acd-hyp-deployment whcs-acd-neg-deployment whcs-acd-ont-deployment whcs-acd-operator whcs-acd-spl-deployment
Version from deployment resource:  image: us.icr.io/cdt-ls-acd-rns/clinical-data-annotator:latest
Version from deployment resource:  image: us.icr.io/cdt-ls-acd-rns/advanced-care-insights:latest
Version from deployment resource:  image: us.icr.io/cdt-ls-acd-rns/attribute-detection:latest
Version from deployment resource:  image: us.icr.io/cdt-ls-acd-rns/concept-detection:latest
Version from deployment resource:  image: us.icr.io/cdt-ls-acd-rns/concept-disambiguation:latest
Version from deployment resource:  image: us.icr.io/cdt-ls-acd-rns/concept-value-detection:latest
Version from deployment resource:  image: us.icr.io/cdt-ls-acd-rns/hypothetical-detection:latest
Version from deployment resource:  image: us.icr.io/cdt-ls-acd-rns/negation-detection:latest
Version from deployment resource:  image: us.icr.io/cdt-ls-acd-rns/ontology:latest
Version from deployment resource:  image: us.icr.io/cdt-ls-acd-rns/acd-operator:v1.0.202003302135
Version from deployment resource:  image: us.icr.io/cdt-ls-acd-rns/spell-checker:latest
Pod names:  whcs-acd-acd-deployment-85bf558c4d-qsddl whcs-acd-aci-deployment-6b5d575767-qwvdf whcs-acd-av-deployment-5c4577f5bb-zntmt whcs-acd-cd-deployment-79cbbfc895-fvqv7 whcs-acd-cds-deployment-6cf85d7f7f-txd46 whcs-acd-cv-deployment-59b69c8cb4-lddd4 whcs-acd-hyp-deployment-97944fcb8-kzjl5 whcs-acd-neg-deployment-5c94b5d7b5-jns28 whcs-acd-ont-deployment-86b8f59749-p45n8 whcs-acd-spl-deployment-5cff664cb4-lv6b4
Error from server (BadRequest): pod whcs-acd-acd-deployment-85bf558c4d-qsddl does not have a host assigned
Version deployed (per call to status api):  whcs-acd-acd-deployment-85bf558c4d-qsddl
Version deployed (per call to status api):  whcs-acd-aci-deployment-6b5d575767-qwvdf 2020-04-28T03:09:21Z 0d 00:58:15
Version deployed (per call to status api):  whcs-acd-av-deployment-5c4577f5bb-zntmt 2020-04-28T03:10:02Z 0d 00:20:05
Version deployed (per call to status api):  whcs-acd-cd-deployment-79cbbfc895-fvqv7 2020-04-28T11:32:26Z 0d 00:58:19
Version deployed (per call to status api):  whcs-acd-cds-deployment-6cf85d7f7f-txd46 2020-04-28T11:09:43Z 0d 00:17:05
Version deployed (per call to status api):  whcs-acd-cv-deployment-59b69c8cb4-lddd4 2020-04-28T11:34:46Z 0d 00:19:53
Version deployed (per call to status api):  whcs-acd-hyp-deployment-97944fcb8-kzjl5 2020-04-28T11:28:45Z 0d 00:20:09
Version deployed (per call to status api):  whcs-acd-neg-deployment-5c94b5d7b5-jns28 2020-04-28T11:19:08Z 0d 00:19:47
Version deployed (per call to status api):  whcs-acd-ont-deployment-86b8f59749-p45n8 2020-04-28T11:17:54Z 0d 00:19:42
Version deployed (per call to status api):  whcs-acd-spl-deployment-5cff664cb4-lv6b4 2020-04-28T11:38:04Z 0d 00:19:36

The ACD macroservice didn't start in this case, because of contention for resources on the system, so ignore that for now. message: '0/8 nodes are available: 1 node(s) were unschedulable, 3 node(s) had       taints that the pod didn''t tolerate, 4 Insufficient cpu.'

Hi, any outlook on when additional triage will be done on this one?

Hi, I run into same error when I try to upgrade my helm based operator, even though I try to keep the CRD apiVersion the same at v1alpha1
{"level":"info","ts":1588704536.364751,"logger":"cmd","msg":"Go Version: go1.13.10"}
{"level":"info","ts":1588704536.3660266,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":1588704536.3660378,"logger":"cmd","msg":"Version of operator-sdk: v0.17.0"}
{"level":"info","ts":1588704536.3669598,"logger":"cmd","msg":"Watching single namespace.","Namespace":"ibm-common-services"}
{"level":"error","ts":1588704536.5929093,"logger":"controller-runtime.manager","msg":"Failed to get API Group-Resources","error":"Unauthorized","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tpkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/manager.New\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.5.2/pkg/manager/manager.go:241\ngithub.com/operator-framework/operator-sdk/pkg/helm.Run\n\tsrc/github.com/operator-framework/operator-sdk/pkg/helm/run.go:109\ngithub.com/operator-framework/operator-sdk/cmd/operator-sdk/execentrypoint.newRunHelmCmd.func1\n\tsrc/github.com/operator-framework/operator-sdk/cmd/operator-sdk/execentrypoint/helm.go:39\ngithub.com/spf13/cobra.(*Command).execute\n\tpkg/mod/github.com/spf13/cobra@v0.0.5/command.go:826\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tpkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914\ngithub.com/spf13/cobra.(*Command).Execute\n\tpkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864\nmain.main\n\tsrc/github.com/operator-framework/operator-sdk/cmd/operator-sdk/main.go:39\nruntime.main\n\t/home/travis/.gimme/versions/go1.13.10.linux.amd64/src/runtime/proc.go:203"}
{"level":"error","ts":1588704536.593097,"logger":"cmd","msg":"Failed to create a new manager.","Namespace":"ibm-common-services","error":"Unauthorized","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tpkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/helm.Run\n\tsrc/github.com/operator-framework/operator-sdk/pkg/helm/run.go:111\ngithub.com/operator-framework/operator-sdk/cmd/operator-sdk/execentrypoint.newRunHelmCmd.func1\n\tsrc/github.com/operator-framework/operator-sdk/cmd/operator-sdk/execentrypoint/helm.go:39\ngithub.com/spf13/cobra.(*Command).execute\n\tpkg/mod/github.com/spf13/cobra@v0.0.5/command.go:826\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tpkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914\ngithub.com/spf13/cobra.(*Command).Execute\n\tpkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864\nmain.main\n\tsrc/github.com/operator-framework/operator-sdk/cmd/operator-sdk/main.go:39\nruntime.main\n\t/home/travis/.gimme/versions/go1.13.10.linux.amd64/src/runtime/proc.go:203"}
time="2020-05-05T18:48:56Z" level=fatal msg=Unauthorized

To make sure it's not related to RBAC, I grant cluster admin role to the operator. I still get this error. Any idea?  Thanks!

It turns out that I need to update the operator by changing the subscription channel in OLM.

@cagrev The only other thing I can think of is that the operator can define overrideValues in the watches.yaml file that will override anything provided by the chart's default values.yaml and anything provided in the CR spec.
See https://sdk.operatorframework.io/docs/helm/reference/advanced_features/#passing-environment-variables-to-the-helm-chart for details.
If that's happening, there should be events being generated on the CR that warn about those overrides.

So much has changed with OLM since this was opened. Closing. A new issue will be created if there are any more current findings.
/close


Support lowercase config of compression codec.

What changes were proposed in this pull request?
Support lowercase config of compression codec.
How was this patch tested?
MVN test passed.

LGTM


MDX support for integration-package?

With Astro Flavored Markdown (AFM) deprecated and removed in Astro v2 in favor of MDX, what's the future for the remark plugin?
Can we support MDX to keep the easy URL-based auto import working? Or provide instructions for how to do it?

Yeah, for v2 we can probably drop support for AFMD here too and only support MDX. Injecting components to MDX is a little more complex. We can use https://github.com/delucis/astro-auto-import for the import injection and borrow from the Astro docs on how to create the component node: https://github.com/withastro/docs/blob/8429df5d014e485d1bc90fcde19621263aa85099/integrations/utils/makeComponentNode.ts#L19-L33

Implemented in #44!


BERT Pretraining Learning Rate Schedule

BERT should be pretrained in 2 phases - 90% of training is done with sequence length 128 and 10% is done with sequence length 512 (c.f. "Pre-training tips and caveats" in README.md).
The learning rate is scheduled to linearly ramp up at the beginning (num_warmup_steps) and then decay over the course of pretraining.
https://github.com/google-research/bert/blob/d66a146741588fb208450bde15aa7db143baaa69/optimization.py#L31-L54
Does anyone have any suggestions or experience with how the learning rate should be set in the second phase of training when max_seq_len=512? How many warm up steps did you use? Did you set the learning rate to the same value as in the first phase of training?

Yang You etc's paper (Large Batch Optimization for Deep Learning: Training BERT in 76 minutes, https://arxiv.org/abs/1904.00962) has a good discussion on this topic.


Fixed #504: Refactor Dataflow LCO to work with futures, this removes the...

... dataflow component as it is obsolete

All tests pass: http://hermione.cct.lsu.edu/builders/hpx_gcc_4_8_boost_1_54_debian_x86_64_debug/builds/288
http://hermione.cct.lsu.edu/builders/hpx_gcc_4_8_boost_1_54_debian_x86_64_release/builds/240

Excellent! Thanks!


Issue with CFBundleIdentifier on macOS

I want to give CFBundleIdentifier to alacritty.
It worked for the first time, but after once quit it launches but doesn't show up the terminal screen.
I mean, I can see alacritty on the menu bar, but no screen shows up.
CFBundleIdentifier is already commented out on info.plist.
I think there's a reason for it like my issue.
Please let me know if you know something about it.
Thanks.

See #661.

@benoit-pierre I cleared launch services cache accrding to @tbodt s comment on #661 but doesn't work at all.
Aditionally, did more complicated one, it made some difference but doesn't work too.


Last Updated Property on Item

Is your feature request related to a problem? Please describe.
n/a
Describe the solution you'd like
A property adding to the default Item type that includes the _ts (last-updated) timestamp provided by cosmos. This can be converted from unix time to a UTC value represented as a DateTime with the property name of LastUpdatedUtc.
Describe alternatives you've considered
Manually adding this property to items that already derive from Item
Additional context
n/a

Have dupe for this #92


test(DCOS-15049): add two UCR container tests using docker and command

This adds system test coverage for a UCR runtime in two scenarios:

using a Docker container and command
using just a command

Most of this is direct copy-paste from the "should create an app with docker config" test, and just modifies by adding selectMesosRuntime() and changes a couple assertions to expect the mesos runtime.
Run with:
docker run -it --rm --ipc=host \
    -v `pwd`:/dcos-ui \
    -e CLUSTER_URL=$OPEN_CLUSTER \
    mesosphere/dcos-ui dcos-system-test-driver --only "apps" -v \
    ./system-tests/driver-config/dev.sh

Checklist

[x] Did you add a JIRA issue in a commit message or as part of the branch name?
[ ] Did you add new unit tests?
[ ] Did you add new integration tests?
[ ] If this is a regression, did you write a test to catch this in the future?

run integration tests


Provide an option to exclude files from HMR

Describe the problem
In the storybook vite builder, we have a problem with svelte stories not correctly reloading when changes are made to files.  I don't completely understand what's going on, but story files are watched by the manager webpack so that it can HMR the list of files, but that makes them incompatible for vite HMR, it seems.  We had the same problem with react stories, and the solution was to exclude stories from HMR, allowing the page to reload instead when changes are made.
However, it seems there's no such option for svelte to control when HMR happens.  It's only on or off, as far as I can tell.
Describe the proposed solution
It would be great if you could support an include/exclude set of globs, much like rollup transform plugins or @vitejs/plugin-react: https://github.com/vitejs/vite/tree/main/packages/plugin-react#filter-which-files-use-fast-refresh
Alternatives considered
I can turn off HMR entirely for svelte projects, but that doesn't seem like a great option.  HMR is working just fine when editing components, just not stories.
Importance
would make ~my life~ the life of storybook users easier

As far as i am aware storybook is not going to support authoring stories in svelte format going forward, and focuses on their own component story format (csf) instead.
So i'm not really sure how csf stories would end up being processed by vite-plugin-svelte.
As a workaround you could create 2 instances of vite-plugin-svelte that operate on disjunct sets with include/exclude:
const storyPattern = "**/*.story.svelte";
export default {
  plugins:[
    svelte({exclude: storyPattern}),
    {
      ...svelte({include: storyPattern, hot: false}),
      name: "vite-plugin-svelte-stories" // avoid 2 plugin instances with same name
    }
  ]
}

If the workaround works, I think it is the solution instead of adding another option for this case, as include and exclude options are meant for this usecase.


fix: role skills

Description
Updated the role information to skills
Type of change

New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Manual

Checklist:
Dev activity

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[x] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes
[x] PR is being merged under a feature flag

/ok-to-test

/ok-to-test


Added IFTTT Handler

I've added a fairly simple handler that allows you to trigger Maker actions on IFTTT.
The great thing about IFTTT is, you can essentially do anything you want with the logs, for free:

You want to get a text alert when there's a database error? Yep.
You want to create a GitHub issue when there's a critical error? Yep.
You want your house to go lockdown mode and start blinking red lights when there's an emergency logged? Sure.

On a more serious note though, how should the tests be handled for this? I looked through the other handlers that use cURL, such as Mandrill and Loggly, and they don't seem to have tests, so I didn't add one to this. However, if you do want one, should I just mock a cURL object? Or would you prefer I do something else?

Thanks, it's fine like that I think. I just did some adjustments to the coding style in https://github.com/Seldaek/monolog/commit/cd3f9f1e5071e393a5f592cd54838c6f53212c93 in case you're interested.

Thanks for accepting the pull! :)
About the coding style though, how come you use spaces instead of tabs? Is that just a preference or is there some benefit to using spaces?

Not really a big benefit except for being "standard". Most projects use
that as well as the PSR-1/PSR-2 (http://www.php-fig.org/psr/psr-2/)
guidelines these days which makes it easier for people to interact
across projects if they don't have to change their editor settings all
the time.


PR to kornia?

Hi,
Would you be interested in incorporating unimatch into kornia library?
We have sota image matching like LoFTR and now are looking for some good optical flow & stereo method.
https://github.com/kornia/kornia
Best, Dmytro

Hi Dmytro,
Thank you for reaching out! Yes I am definitely happy to have the opportunity to incorporate unimatch into kornia. However I am swamped with other projects/ddls currently and not familiar with kornia's coding style so far, it would be more ideal for me if another contributor can help incroporate our unimatch code to kornia. I am very happy to help during this process. Thanks!
Best,
Haofei


[master + backport] XD-2854 Check for ZK data with with no module defs

Add null check if the module definitions weren't set for the given stream
Use dummy resources if the module definition is null

LGTM.  Ran regressions and they looked good.
I'll merge in the morning!

-1, see my comment

Removed my LGTM comment from before.  I've updated the https://jira.spring.io/browse/XD-2854 to included the steps to reproduce this bug.

ok, Closing this PR as the migration issue reported in https://jira.spring.io/browse/XD-2854 can only be addressed by following the steps mentioned there.

@ericbottard The creation of module definition is only needed on the getter as all it requires is the type information to be used during the deployment. Also, the module definition will be null only for the case where we port the stream definitions from previous versions of XD.

Squashed and merged on master as 387e2336d2ca39cc7848667e4e6a213d66279d26 and on 1.1.x as 61657f7998be1b5f8c2a8f7a8832f8cad492bdd8


Plot tile

Given we have variables, would be good to have plot x,y. Learning coordinate system seems like a reasonable goal (upper end of K-5)

We need to be intentional about keeping microcode small, I think this is already too complex. Also unsure how you teach to provide 2 tiles following that one.

Agreed it is probably too complex.


undefinded in ?lcc

Bugreport:
Bug reported by Fabiooo
Is it a bugreport for the website or for the bot?: bot
What is the title of your bugreport?: undefinded in ?lcc
How can you reproduce the bug? (Explain in steps): Type ?lcc
Which result would normally have to be?: Description of the List
What happens if you do the action now?: undefinded
ReportID: 64
┆Attachments: http://prntscr.com/l36zdu

➤ LenoxBot commented:
Approve:
Could be reproduced
Wandi

➤ LenoxBot commented:
Approve:
Could be reproduced
Monkeyyy11

➤ LenoxBot commented:
Approve:
could be reproduced
Dadi


Add explicitly support to net 6 net 7 and net 8

:x: Build Sankhya-SDK-dotnet 3.0.2 failed (commit https://github.com/guibranco/Sankhya-SDK-dotnet/commit/64a8e478c8 by @guibranco)

:white_check_mark: Build Sankhya-SDK-dotnet 3.0.4 completed (commit https://github.com/guibranco/Sankhya-SDK-dotnet/commit/f53cddb2b0 by @guibranco)


[TF2.0]Shuffle function on tf.data fills up RAM memory.

In TF2.0rc, using shuffle(buffer_size= #_of_elements) on tf.data.Dataset type dataset, it fills up shuffle buffer which also fills up RAM memory so if the data is huge and RAM is not enough then it hangs up the system which leads to system restart. This memory filling happens before every epoch. It of course releases the memory once the epoch is done. To reproduce this problem, I followed this TF tutorial but with more amount of data.
Since tf.data is for handling the dataset  that can not be filled up in the memory then why does shuffle function shows this behaviour and fills up the memory.?

The behavior you are seeing is expected. If your data is such that it cannot be all loaded into memory, then you cannot perform a "perfect" shuffle (i.e. using a shuffle buffer size that matches the cardinality of your dataset) and should instead use a smaller buffer size.
In other words, tf.data currently does not support "perfect" shuffle for dataset that do not fit into memory.

@jsimsa
How to avoid this problem? Thank you.

Use a smaller buffer size.

Thanks a lot!
You're answering so quickly, ths!