Publish 02/07/2022, 10:30 AM

Docs Build status updates of commit 6704cef:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docset/winserver2019-ps/defender/Add-MpPreference.md
:white_check_mark:Succeeded
View (WindowsServer2019-ps)

docset/winserver2022-ps/defender/Add-MpPreference.md
:white_check_mark:Succeeded
View (WindowsServer2022-ps)

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


fix: header links did not include language uri (hl-1159)

Description :sparkles:
Language change would be reset if navigational links were clicked. Fix this by including current language prefix in links.

API branch is deployed to platta: https://helsinkilisa-pr3238.api.dev.hel.ninja/healthz :rocket::rocket::rocket:

HANDLER branch is deployed to platta: https://helsinkilisa-ui-handler-pr3238.dev.hel.ninja :rocket::rocket::rocket:

APPLICANT branch is deployed to platta: https://helsinkilisa-ui-pr3238.dev.hel.ninja :rocket::rocket::rocket:

TestCafe result is success for https://helsinkilisa-ui-pr3238.dev.hel.ninja :laughing::tada::tada::tada:

TestCafe result is success for https://helsinkilisa-ui-handler-pr3238.dev.hel.ninja :laughing::tada::tada::tada:


[coestarterkit] - Audit custom connector - error [400] - Test Operation

I'm receiving the error [400] when I try to test the "Start Subscription"

{
"error": {
"code": "StartSubscription [CorrId=1e66f72e-a894-4359-859b-43dc545c104b]",
"message": "bf94ede-1182-42c2-b949-958401f12f0d] failed. Exception: Microsoft.Office.Compliance.Audit.DataServiceException: Tenant 1b818c16-117f-4bf5-89a7-028fd47f7664 does not exist.\r\n   at Microsoft.Office.Compliance.Audit.API.AzureManager.d__51.MoveNext() in d:\dbs\sh\nibr\1214_141956_0\cmd\q\sources\dev\auditing\src\auditapiservice\common\AzureManager.cs:line 1953\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Office.Compliance.Audit.API.AzureManager.d__22.MoveNext() in d:\dbs\sh\nibr\1214_141956_0\cmd\q\sources\dev\auditing\src\auditapiservice\common\AzureManager.cs:line 498\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Office.Compliance.Audit.API.StartController.d__0.MoveNext() in d:\dbs\sh\nibr\1214_141956_0\cmd\17\sources\dev\auditing\src\auditapiservice\apifrontendservicerole\Controllers\StartController.cs:line 76"
}
}
All configuration its ok!
Originally posted by @hsantan in https://github.com/microsoft/powerapps-tools/issues/615#issuecomment-749205221

When you raise a new issue, could you help us by selecting the correct project under "New Issue"? See highlighted:

This repo is used for different projects, which are worked on by different teams - filing the issue report in the correct category ensures it is automatically labelled and assigned to the correct team.

This issue from the error message "Tenant 1b818c16-117f-4bf5-89a7-028fd47f7664 does not exist" typically has two root causes:

You have turned on the Office 365 Audit Log in your tenant very recently, and it is not active to be consumed via the API yet. Sometimes this can take a few hours to activate.
You have entered your Tenant ID incorrectly. This is your Tenant ID from the App registration, which you can find under Directory (tenant) ID in the Apo Registration overview.

@manuelap-msft Thank You Manuela. Now its working after waiting 24h to index all audit logs

Great, thank you! We will make sure to add a note in our setup instructions next time we update our documentation to share this as well!


Add ability to force policy calculation for certain policies

Description

Add a new field to (Global)NetowrkPolicies that allows for passing "performance hints" to Felix.
Define the first such hint "AssumeNeededOnEveryNode", which causes Felix to process the policy as if it matched a local endpoint.  I.e. it will calculate the selector IP sets and the iptables rules, even though they are not needed.  This reduces latency when the policy goes from inactive to active and it avoids "flapping" if the policy is temporarily deactivated.  For very large policy sets, this can make a considerable difference.

Related issues/PRs

Todos

[x] Tests

[x] UT
[x] FV

[ ] Documentation
[x] Release note

Release Note

Add "performace hints" to network policies.  Define a hint "AssumeNeededOnEveryNode" that tells Felix to pre-calculate the policy's selectors as if it was in use for a local endpoint even if no such endpoint exists.  This reduces latency to apply the policy when a local endpoint does start using it and it prevents doing extra work to remove the policy from the dataplane when the last endpoint using it is removed.  However, it uses extra CPU/memory on every node where the policy is not in use.

Reminder for the reviewer
Make sure that this PR has the correct labels and milestone set.
Every PR needs one docs-* label.

docs-pr-required: This change requires a change to the documentation that has not been completed yet.
docs-completed: This change has all necessary documentation completed.
docs-not-required: This change has no user-facing impact and requires no docs.

Every PR needs one release-note-* label.

release-note-required: This PR has user-facing changes. Most PRs should have this label.
release-note-not-required: This PR has no user-facing changes.

Other optional labels:

cherry-pick-candidate: This PR should be cherry-picked to an earlier release. For bug fixes only.
needs-operator-pr: This PR is related to install and requires a corresponding change to the operator.

/merge-when-ready

OK, I will merge the pull request when it's ready, leave the commits as is when I merge it, and leave the branch after I've merged it.


Implement Library $evaluate correctly for R4 and DSTU3

Currently the Library $evaluate operation is not in compliance with the spec (it has additional experimental parameters), and needs to be implemented for DSTU3

Does this include the returned Parameters resource?  Since the spec says it will return "a Parameters resource with a parameter for each named expression defined in the library", I interpret that to mean the return will be a single Parameters, not a Bundle of Parameters.

Resolved in #554


Will the helm version be released?

@kopp0510 , we are not using Helm. There is a draft PR https://github.com/Altinity/clickhouse-operator/pull/358 but it requires quite a lot of work for completion. We appreciate any help with that.


Stop LinearSweepAlgorithm from yielding opcodes on error instead of raising exceptions

Now androguard will raise InvalidInstruction when invalid instruction  is encountered. Is it designed in this way? I think as analysis tool androguard should parse the target file as much as possible instead of exiting when invalid instruction is encountered.

Do you have an example file for this behaviour?

The sample is a malware and I'm not supposed to share the apk file for some reasons, but here is its link in virustotal.
However I have found out what causes androguard's exception. Here is the log and traceback.
2018-05-28 17:42:09,925 [DEBUG   ] androguard.analysis (analysis.py): Creating basic blocks in Landroid/support/v4/os/e;->newArray(I)[Ljava/lang/Object; [access_flags=public] @ 0x76254
2018-05-28 17:42:09,925 [DEBUG   ] androguard.analysis (analysis.py): Parsing instructions
---------------------------------------------------------------------------
error                                     Traceback (most recent call last)
C:\androguard\androguard\core\bytecodes\dvm.py in get_extented_instruction(cm, op_value, buff)
   6328     try:
-> 6329         return DALVIK_OPCODES_EXTENDED_WIDTH[op_value][0](cm, buff)
   6330     except struct.error:

C:\androguard\androguard\core\bytecodes\dvm.py in __init__(self, cm, buff)
   5786         self.OP = unpack("=H", buff[0:2])[0]
-> 5787         self.BBBBBBBB = unpack("=I", buff[2:6])[0]
   5788

error: unpack requires a bytes object of length 4

During handling of the above exception, another exception occurred:

InvalidInstruction                        Traceback (most recent call last)
<ipython-input-5-ab4d2306b072> in <module>()
----> 1 h, d, dx = AnalyzeDex(r"C:\Users\Ford_Qin\Desktop\classes.dex")

C:\androguard\androguard\misc.py in AnalyzeDex(filename, session)
     97         data = fd.read()
     98
---> 99     return session.addDEX(filename, data)
    100
    101

C:\androguard\androguard\session.py in addDEX(self, filename, data, dx)
    127             dx = Analysis()
    128
--> 129         dx.add(d)
    130         dx.create_xref()
    131

C:\androguard\androguard\core\analysis\analysis.py in add(self, vm)
    942
    943         for method in vm.get_methods():
--> 944             self.methods[method] = MethodAnalysis(vm, method)
    945
    946     def _get_all_classes(self):

C:\androguard\androguard\core\analysis\analysis.py in __init__(self, vm, method)
    284         self.code = self.method.get_code()
    285         if self.code:
--> 286             self._create_basic_block()
    287
    288     def _create_basic_block(self):

C:\androguard\androguard\core\analysis\analysis.py in _create_basic_block(self)
    297
    298         log.debug("Parsing instructions")
--> 299         for i in bc.get_instructions():
    300             for j in BasicOPCODES:
    301                 if j.match(i.get_name()) is not None:

C:\androguard\androguard\core\bytecodes\dvm.py in get_instructions(self)
   6491             ins = lsa.get_instructions(self.CM, self.size, self.insn,
   6492                                           self.idx)
-> 6493             self.cached_instructions = list(ins)
   6494
   6495         for i in self.cached_instructions:

C:\androguard\androguard\core\bytecodes\dvm.py in get_instructions(self, cm, size, insn, idx)
   6393                 elif op_value in DALVIK_OPCODES_EXTENDED_WIDTH:
   6394                     try:
-> 6395                         obj = get_extented_instruction(cm, op_value, insn[idx:])
   6396                         classic_instruction = False
   6397                     except struct.error as why:

C:\androguard\androguard\core\bytecodes\dvm.py in get_extented_instruction(cm, op_value, buff)
   6329         return DALVIK_OPCODES_EXTENDED_WIDTH[op_value][0](cm, buff)
   6330     except struct.error:
-> 6331         raise InvalidInstruction("Invalid Instruction for 0x%x:%s" % (op_value, repr(buff)))
   6332
   6333

InvalidInstruction: Invalid Instruction for 0x14ff:bytearray(b'\xff\x14ad')

Here is the hex value in the related dex and the dex version is 035.

we can see insns_size is 7 and the instructions androguard parsed for insns is:
D4 B3 F2 B5 -> <androguard.core.bytecodes.dvm.Instruction22s object at 0x000001AC20619A58
15 A2 FF 14 -><androguard.core.bytecodes.dvm.Instruction21h object at 0x000001AC206199B0>
F6 A3 -> <androguard.core.bytecodes.dvm.InstructionInvalid object at 0x000001AC20619A20>.
So the last instructions in insns is FF 14 61 64. So the format for this op is Instruction41c. So the whole instructions should be 8 bytes long instead of 4. That's why we get the error: unpack requires a bytes object of length 4.

Okay so the opcode sequence is D4 B3 F2 B5 15 A2 FF 14 F6 A3 FF 14 61 64, this looks good.
The first opcide D4 is correct, the second 15 too. But F6 is actually unused. I'm not sure what the correct behaviour would be then... probably this bytecode would crash?
The format of F6 is defined as 10x so it is a pure opcode with no arguments. In this case A3 would be the next opcode, which does not match as then 64 would be the last opcode but has no arguments.
It looks like the opcodes are injected to crash analysis tools.
Can you try decompilation with other tools? What does they give as result?
I think we should implement the unused instructions as well, but then this bytecode is also not parseable...

btw these extended width opcodes seem to be never used again by android? They are not shown in the official documentation anymore.
probably we can remove them from androguard completly...
does someone has more information about them?

It looks like i can not download the sample from koodous (server timeout). But I changed the opcodes to unused. Can you try the changes in #520? probably there is more to do than just fixing the opcode table...

okay this is interesting: If jeb thinks that F6A3FF14 is this iput-wide-quick, what is then 6164?
Opcode 61 is sget-wide and has one 8 bit and one 16bit argument. But I can only see the first 8bit argument in the opcode...
Would be interesting to see, if there are more samples using unused opcodes.
As the official documentation says they are unused (Is there any official doc with the wide opcodes still?) I would consider a usage of them as "malicious"... But I'm not sure how this can be implemented...
I also noticed now, that when the true 10x format would be used, the second byte must be 0. But in this case it is A3...

I had an idea: When encountering an unknown opcode, we could create a pseudoopcode which consumes the rest of the opcodes / everything until the next valid opcode / ...
Or come up with a more solid approach...

Just skipping the invalid opcode instead of the whole method sounds good to me.  For now I just skip the whole method. On the other hands, I think it may be a good idea to do what AOSP does when encounters invalid opcode. I will check how AOSP parse dex when I have time.

So I found other samples which do crazy stuff.
 The goto sends the interpreter to -19201 which is pretty out of bounds....
So I guess this would read as invalid and break the code.
After that follows other opcodes which makes no sense, including unimplemented ones like 0xf3...
So this would be the perfect usecase for a parser which would simply stop and not raise exceptions.
sample: DEX file sha256: 0fac1afe04694862eb3154cdf7c7c13b72c5966e5f640120c52c69a7aa0f7727

@thecrackofdawn btw have you investigated in the AOSP?
I was reading the old comments, and I think now that using pseudoopcodes might be the wrong way. I also have changed the unused opcodes to throw an error, as there is basically undefined behaviour ahead.
So from my current point of view, it would make sense to raise the InvalidInstruction exceptions but catch them in the LSA class and simply stop there processing and log errors.

@thecrackofdawn i checked in a version where the processing stops instead of raising an exception

@reox Sorry, I haven't investigated how the AOSP deal with the invalid opcode.
I'm not sure where to find the version you checked in.

@thecrackofdawn it is work in progress: #681


Flask, mod_wsgi, and Apache: ImportError mysql

when i am running python3 test.py it is working fine with mysql
but when i running on apache server i am getting wsgi error no module name mysql

Please use Stack Overflow or IRC for support.

I checked  a lot

Are u sure your mod_wsgi was running python3
Or it could be some environment settings issue.
Anyway, this is an issue of deploying your code, not an issue of Flask.
Reopening closed issue without adequate reason is very annoying.
Stack Overflow, Google, and many web tech forums are your friends.
BTW, next time when you run into any issue, don't just say "It doesn't work". Don't say that on any forums either. That won't help anyone.


RINEX 3: Fail for out-of-spec lines

Observed in the wild: A RNX file with records starting with just >, like >2019 08 09.

Is the issue that there was a bad/unwanted comment like
>2019 08 09  # comments, junk stuff
...
> 2019 08 09 ...  # good epoch

I suppose that before this PR, georinex would give an error or at least get bad data.
So it sounds like you want to discard those lines with this PR. I think this seems better for now than having to make a regex to deal with bad data, as per RINEX 3.04 pg. A13, Table A3

So the behavior will now be for rinexobs3() as typically used that these lines without the mandated 1X space will be discarded--even the whole file if it's improperly formatted.


Add miscellaneous symbols from Miscellaneous Mathematical Symbols-B

This PR adds multiple unrelated symbols from the "Miscellaneous Mathematical Symbols-B" Unicode block.
As far as I can tell, arrow.r.colon does not have equivalent Unicode codepoints for other arrow directions, even outside Miscellaneous Mathematical Symbols-B. Maybe it should be named colon.arrow instead, similarly to colon.eq.
I added variants to infinity, but not to oo. I can add them to oo as well if you want.

I'm unsure about ⧴ since it seems very special purpose and out of the normal order. Maybe we should leave it out for now since you're also unsure whether it belongs under colon.

You are right. I've removed it from this PR. It can always be added later when needed.

Thanks!


Configuring get_cwd does not impact the detection of the config file and the binary

Hi 👋
First, thanks for the work done here.
I tried to configure the neotest-playwright. I work in a monorepo, so my e2e tests are in a subfolder.
When I configured get_cwd, I expected the config and the binary detection to take the return value of cwd as the root path.
I figured out how to make it work using get_playwright_binary and get_playwright_config options, but it would be a nice to have 😄

Hey. Thanks for the feature request. I prefer working in monorepos as well and had originally planned to implement a simple heuristic to detect and configure monorepos automatically. However, I couldn't find a straightforward way to do so (further discussion here).
At the same time, I slowly came to realize that trying to come up with a clever way to support all the possible config permutations we could encounter (typical pnpm monorepo, pnpm monorepo with a shared playwright config file in the monorepo root, repo with multiple config playwright config files, pnpm repo with a non-standard packages/* structure, etc) might be more trouble than it's worth.
So, I opted to have neotest-playwright only support the vanilla npm repo structure, and whenever it couldn't, prefer to be predictable by ceding control to the user. If the return value of get_cwd only affected (a) the location of the config playwright file and (b) the location of the playwright binary, then it would probably make sense to couple their behavior. But since get_cwd also affects where the test process is spawned, I thought it best to decouple the behavior to avoid introducing unintended consequences. Sure, it makes the setup more explicit. But in return it allows users greater flexibility, a trade-off I thought was fair.
That being said, you're spot on that, at first glance, everyone would assume that customizing get_cwd would affect the behavior of the other two. I'll update the documentation to better communicate that is not currently the case.


Since adding flash before default content, we do not need doubles

This PR resolves this 🌶

We recently added an include for the flash template before content on all templates extending default.twig. There were a few forgotten usages of flash that caused issues like this. I grep-ed for occurrences of _flash.twig and removed what made sense.\
Follows #900.

Thank you, @mdwheele!


Infinite scroll in facility group quick edit

What is the motivation for adding/enhancing this feature?

Infinite scroll seems to not be working when user does a quick edit from the facility group page through the modal.

During the initial discussions, we decided not to show all facilities by default and allow user to search facilities and add. Hence we intentionally didn't added the infinite scrolling in the modal. We only added the searching and displayed the total facilities count to keep track.
We can add infinite scrolling in the modal if we want to.

Ok makes sense, this isn't a blocker issue but we can put on our roadmap.


Add a button to allow user to search by their lat lng

In PetType.js

[ ] clicking on a button allows the user to search based on lat long

[ ] Remove the zip code line

[ ] Make sure its mobile responsive

I did these changes, but I was not sure if I should have removed the default behaviour to search by location, to just happen when the button is clicked.


Allow custom background-color in Vide div

The text overtop of our Vide video background was white. We had a case where the browser failed to load the video and poster files from s3 (s3 was down? who knows), and so the section was white-on-white (users could not find the sign-up buttons, etc).
This PR allows a background-color to be used, which allows the text to show up even if video assets fail to load.

+1


Media Players Not Updating

IMPORTANT: Please search the issues, including closed issues, and the FAQ before opening a new issue. The template is mandatory; failure to use it will result in issue closure.
Describe the bug

Media Player entities go stale after a few hours. If I start playing media either with my voice, or by having a routine triggered my HA media player entities don't know about it and just says standby. Or if media has been playing for a while and I stop it, the media player entity will still show it playing. Reloading the integration with kick things and get them showing the correct status again for a little while
To Reproduce

Play media on an alexa device after it's been idle for a few hours.
Observe media state in HA hasn't changed
Submit issue on github

Expected behavior

Media player should show current status of media player entities
Screenshots

System details

Home-assistant (version): 2023.5.3
alexa_media (version from const.py or HA startup): 4.6.4
alexapy (version from pip show alexapy or HA startup): 1.26.8
Amazon 2FA is enabled (y/n). <!---We will not debug login issues if unanswered--->:yes

Logs
Please provide logs.
These debug logs begin at 9:08 when I run my "soundmachineon" routine which plays white noise on amazon music and didn't update the status until 9:09 when I reloaded alexa media player. Ignore the BROKEN_TUYA lines, they were filling up my logs so I did a find/replace to get rid of them.
alexa_logs.txt

The logs indicate that websocket is reporting data, the player urls are being queried, and you're not getting errors to the requests. Please note that not all services actually update the player. Check the official app and see if it's actually displaying what's playing.

Yes, but the logs also show that the routine starts which just plays white noise from Amazon’s music, but the status of the player never updates. The official app does show that update, and reloading the integration updates the media player and then everything is fine…for a couple hours.

I just checked this morning and the Alexa app shows that it is playing, and when I say Alexa stop, it shows it paused. In HA however it still shows it playing. If I reload the component it will update correctly.

Just saw this when I try to pause it in HA when it’s already paused in real life.
`Logger: homeassistant.components.websocket_api.http.connection
Source: custom_components/alexa_media/media_player.py:1170
Integration: Home Assistant WebSocket API (documentation, issues)
First occurred: 6:21:51 AM (2 occurrences)
Last logged: 6:21:59 AM
[140630914232624] 'NoneType' object has no attribute 'data'
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 199, in handle_call_service
await hass.services.async_call(
File "/usr/src/homeassistant/homeassistant/core.py", line 1849, in async_call
task.result()
File "/usr/src/homeassistant/homeassistant/core.py", line 1889, in _execute_service
await cast(Callable[[ServiceCall], Awaitable[None]], handler.job.target)(
File "/usr/src/homeassistant/homeassistant/helpers/entity_component.py", line 226, in handle_service
await service.entity_service_call(
File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 798, in entity_service_call
future.result()  # pop exception if have
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 980, in async_request_call
await coro
File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 838, in _handle_entity_call
await result
File "/config/custom_components/alexa_media/helpers.py", line 156, in _catch_login_errors
result = await func(*args, **kwargs)
File "/config/custom_components/alexa_media/media_player.py", line 1163, in async_media_pause
await self._playing_parent.async_media_pause()
File "/config/custom_components/alexa_media/helpers.py", line 156, in _catch_login_errors
result = await func(*args, **kwargs)
File "/config/custom_components/alexa_media/media_player.py", line 1170, in async_media_pause
self.hass.data[DATA_ALEXAMEDIA]["accounts"][self._login.email]["websocket"]
AttributeError: 'NoneType' object has no attribute 'data'
`

I have found that it is usually enough to perform a media player action (media_next_track, volume_up, ...) when it hangs. Then the dashboard player status mostly updates immediately. A restart of the integration was not necessary for me so far.

I just called the volume down and volume up services but it still shows it in standby

Just saw this when I try to pause it in HA when it’s already paused in real life.

This error indicate HA is broken for the integration. Every integration is supposed to have access to the hass.data object automatically. If it doesn't, then there is something very wrong. There is no way to solve the issue short of a reload of the integration where hopefully HA provides the entity. Assuming you're not getting that error, then you should confirm you're seeing websocket traffic. You can test this by enabling debug logging and then manually changing the volume button. If you're not seeing websocket, then the integration cannot update info without directly using HA to interact with the component.

I was trying to make sure things were broken before I started the debug logs and I found something interesting. If I call my routine from within HA using a service call from a card like this or from an automation, I can see the volume adjust on the lovelace card, but the state still shows standby.
type: custom:mushroom-entity-card
entity: script.music_sound_machine
name: Sound Machine
tap_action:
  action: call-service
  service: script.music_sound_machine
  service_data:
    volume: 0.6
    device: media_player.charlottes_echo_spot
  target: {}
icon: mdi:sleep
secondary_info: none
layout: vertical
fill_container: true

However if I go into the alexa app on my phone I can see the media player in a playing state. Then if I hit pause in the alexa app, the lovelace card updates showing it's paused.
Once it breaks again I will try to enable debug logging and see what happens.

It broke again, and here are some logs. I started my goodnight routine around 2023-06-06 21:02:10.306, but the logs never show the state updating from standby, it always stays . The alexa app shows the stream playing as an active media source, and if I was to pause it (I didn't because it would wake up my wife) it would trigger HA to notice that it's playing. I tried changing the volume with both service calls and from the HA app, but neither of those triggered the state change.
These are the raw debug logs minus the local tuya stuff I removed from there because it is log spam that I need to fix when I find the time. The other errors you see in there are spotify went funky earlier today and one of my roborocks is offline because it's being replaced:
home-assistant_alexa_media_2023-06-07T01-06-47.569Z.log

Your websocket is reporting changes but doesn't appear to report anything after your routine is run. If this worked before, it looks like the websocket isn't broadcasting any state changes now so it's an API change. There's nothing I can do to capture this unless after every routine I turn on polling for all alexa devices. That will likely be very spammy and I have historically not enabled polling unless there's actually a play command issued by a media player.
That said, due to the restart, the component does try to make one attempt to see if things are playing. In this case for the device where it appears you issued the request this is the response and it didn't see anything playing per amazon. Are you positive the App or webplayer actually shows white noise content when playing?
2023-06-06 21:06:33.897 DEBUG (MainThread) [alexapy.alexaapi] d***********6@y*******m: Trying get: https://alexa.amazon.com/api/np/player?deviceSerialNumber=SNIP&deviceType=A4ZP7ZC4PI6TO&screenWidth=2560&_=1686099993897 : with uri: /api/np/player data None query {'deviceSerialNumber': 'SNIP', 'deviceType': 'A4ZP7ZC4PI6TO', 'screenWidth': 2560, '_':<PHONE_NUMBER>897}
2023-06-06 21:06:34.072 DEBUG (MainThread) [alexapy.alexaapi] d***********6@y*******m: GET: https://alexa.amazon.com/api/np/player?deviceSerialNumber=SNIP&deviceType=A4ZP7ZC4PI6TO&screenWidth=2560&_=1686099993897 returned 200:OK:application/json
2023-06-06 21:06:34.074 DEBUG (MainThread) [custom_components.alexa_media.media_player] d***********6@y*******m: Refreshing <entity None=standby>

I am 100% positive the Alexa app shows white noise playing.

Are there any other logs I need to provide to help you track this down?

Checking in on this again. Is there anything else you need from me to help move this along?

@alandtse is there anything else I can provide to help solve this issue?

Please don't ping me. I don't promise any support. The code is open and you can make your own changes. I've already analyzed your logs and explained what I know. Honestly, I'm probably done with supporting this and pings do not encourage me to work more.

I would completely understand and as you said open code, etc. But I cannot do such codings and have the not-updating-anymore as well. So any mitigation oder fix on the long run would be very very good of course.
Here e.g. die next-timer entities often are not started and stay as unavailable even when a timer on the device is started. Only a reload of the integration is fixing this then.

Sorry for pinging you, the only reason I did was because I sent a few messages with responses to your question and didn’t hear anything back. The app shows it playing, HA does not.
Is there anything we can do as non coders to help you try to figure out what may have changed in the api? We can’t be the only two people having this issue.
@emufan couple questions,

What type of WiFi APs do you use?
What type of echo devices are you seeing this issue on?

for me I use Unifi WiFi 6 Lites and am seeing this issue in particular on an echo show 5 and an echo spot. Those are the two devices I run the sound machine on, but I “think” the issue occurs on them all.

Does running homeassistant.update_entity on the relevant media_player entity help at all?

That’s a great question, I’m not home this weekend but I’ll test that out when I get back.

I have issues with my printer as well where it’s unavailable for AirPrint unless I do a reconnect on it in the unifi app.

Does running homeassistant.update_entity on the relevant media_player entity help at all?

This did it for me, had the same problem, but manually pinging an entity update does the trick.

Does running homeassistant.update_entity on the relevant media_player entity help at all?

This did it for me, had the same problem, but manually pinging an entity update does the trick.

This does not work with the alarm/timers etc tho, they are all still on unknown

Does running homeassistant.update_entity on the relevant media_player entity help at all?

This did it for me, had the same problem, but manually pinging an entity update does the trick.

This seems to help, kept updating the entity from that point while playing, stopped playback and started a new Pandora Session and it still updated.   Will play if we need to schedule the service call for periodic refreshing of the 7+ AMZ Devices

Does running homeassistant.update_entity on the relevant media_player entity help at all?

This did it for me, had the same problem, but manually pinging an entity update does the trick.

This does not work with the alarm/timers etc tho, they are all still on unknown

Just checking, you're running the update service on the relevant sensor entity rather than the media_player? That works for me although could be one of those tempramental ones where sometimes it works, sometimes it doesn't.
You can run the following in the developer tools template section to get a YAML-formatted list of all enabled entities from the integration - I can't figure out how to get it to work in the actual service call right now but you can copy-paste and save some typing at least...
{% for entity in integration_entities("alexa_media") %}
- {{ entity }}
{%- endfor %}

Does running homeassistant.update_entity on the relevant media_player entity help at all?

This did it for me, had the same problem, but manually pinging an entity update does the trick.

This seems to help, kept updating the entity from that point while playing, stopped playback and started a new Pandora Session and it still updated. Will play if we need to schedule the service call for periodic refreshing of the 7+ AMZ Devices

Yeah, I have the following running, not sure if overkill but seems to run okay and keeps things updated:
alias: Alexa media updates
description: ""
trigger:
  - platform: homeassistant
    event: start
condition: []
action:
  - repeat:
      until:
        - condition: state
          entity_id: input_boolean.ha_restarting
          state: "on"
      sequence:
        - service: homeassistant.update_entity
          data: {}
          target:
            entity_id:
              - < list of entities >
        - delay:
            hours: 0
            minutes: 0
            seconds: 10
            milliseconds: 0
mode: restart

I left the dot idle for a few hours and started a new Pandora session, didn't update until I called the
service: homeassistant.update_entity data: {} target:   entity_id: media_player.computer_room_dot
Suspect that it falls off the polling when idle for a while.
Will implement the suggested trigger that @krisnoble posted, probably set it for repeat at 1 min as my use-case is watching / recording the current song played on Pandora when I am in the shower.

but manually pinging an entity update does the trick.

But where is the trick, if I don't know, when a timer is running. You are suggesting, to poll/ping every entity periodically e.g. every 1 minute 24/7? I don't know, if this is a good apprach.

You are suggesting, to poll/ping every entity periodically e.g. every 1 minute 24/7? I don't know, if this is a good apprach.

Agreed, but a production / user workaround is better than nothing.
I would like a bit more control over the the HA Polling threads, or Ubiquity UDME not to loose fixed DNS entries when it shouldn't and forget ones when it should, no defects there  ;)
The polling the monitored media_players is working for me ATM, 12% Utilization on the NUC style HA host (No noticeable Changes).  So I am happy now that I can see my last ten items on my playlist dashboard.

You are suggesting, to poll/ping every entity periodically e.g. every 1 minute 24/7? I don't know, if this is a good apprach.

FWIW this is my system running in Docker on a Pi4 4GB, updating every 10 seconds. The raised periods are in the evenings when the Pi runs some other stuff like a slideshow and some custom scripts. Peak values are 58c, 19% processor, 56% memory, 1.4 load average, all well within nominal as far as I know. Hopefully that's reassuring?

FYI, the concern with excessive polling is that Amazon will throttle or block you. This is the reason we do not use that as a solution generally. But if you're ok with that risk, this may be a workaround.

Agree with alandtse on that, however it should be noted that the risk could potentially extend beyond those who are using the workaround to all users of Alexa Media Player depending on what Amazon chooses to do about it if/when they notice.
Presumably the reason this broke in the first place was because Amazon was trying to reduce their API load and/or traffic and disabled broadcasting state changes to achieve that (or it could just be a bug and completely unintentional, but we have no way to know that). Now that some users are polling them for state that would increase their traffic rather than reducing it, and once they notice that and their investigation points to Alexa Media Player, they may either block the specific users, or may just block the entire app to stop the increased load.
The thing is, if they don't pay very close attention they may not even realise that it is only a few users causing it which means the chance that they block the entire app is higher than you might naively expect - however it is hard to say how big that risk actually is.

Good points both and apologies, I didn't realise the update polling wasn't purely local. I certainly don't want to cause any problems for anyone so I've dropped my polling down to once an hour for now which has been stable all morning as far as I know and should be much more reasonable in Amazon's eyes.
I was already planning to migrate from Alexa to something more open anyway, so that plan has just been shifted from "soon" to "now". FWIW though thank you Alan et al for all your work on this, it's been immensely helpful over the last few years and much appreciated.

Hi,
facing the same issue I'd like to refresh only the selected alexa group when music playing is active once a minute.
Would that be acceptable?
- id: '206'
  alias: Alexa media player updates
  description: ""
  trigger:
    - platform: state
      entity_id: input_boolean.music_playing
      to: 'on'
  condition: []
  action:
    - repeat:
        while:
          - condition: template
            value_template: "{{ is_state('input_boolean.music_playing','on') }}"
        sequence:
          - service: homeassistant.update_entity
            data: {}
            target:
              entity_id: >
                {% set player = states('input_select.music_sources') %}
                {% set map = {
                  "Erdgeschoss": "media_player.erdgeschoss",
                  "Alle Etagen": "media_player.alle_etagen",
                  "Home Office": "media_player.home_office",
                  "Dinnerparty": "media_player.dinnerparty"
                }%}
                {{ map[player] if player in map else 'media_player.receiver' }}
          - delay:
              hours: 0
              minutes: 1
              seconds: 0
              milliseconds: 0
  mode: restart

Hi, facing the same issue I'd like to refresh only the selected alexa group when music playing is active once a minute.

The trouble with that approach (which is what I first thought of too) is that if the device does get into the "stuck" state (e.g. been left for too long as posited by jlong23), it will never get the "playing" state update in HA to trigger the automation. Looking in the code it seems like the default polling interval is one minute anyway (and this tallies with my own observations) so as long as the devices stay active I don't think manually polling once per minute would bring any benefit.

PS: I tested it over the weekend with 5 Minutes delay and it worked stable. Interestingly the media player obvously stops updating after 3 or 4 Minutes and only sometimes. So I will set the delay to 3 Minutes. That seem to be good in order to have an accurate display of the current state.

@Pirol62 While I can't speak for what thresholds Amazon deems acceptable, that looks like an entirely reasonable workaround to me to keep polling to a minimum while still allowing the desired functionality.

I also have the same problem and I'm constantly being contacted from people that are ousing my multinotify that are having this issue that generates two issues:

the volume is not resumed (as it stopped updating with the real value)
the music play status is wrong so at the end of the message echo will not resume the play or will play while it was not playing before (as the status was not updated), quite annoying.

Generally the "solution" is to reload the integration or restart Home Assistant but not having even a way to detect that this is happening is very bad.
From my perspective it could be that the websocket is closing without notice so it stops to receive updates from Amazon.
I'm not a python programmer so I'm not able to contribute (sigh...) but couldn't anyone check the webserver management code? We could work on that to ensure the websocket is alive and if it's not we can reestablish it.
As a last resort a code that every time interval recycle the websocket could be a temporary fix that helps this not to happen too frequently. But websocket closing should be a very detectable event that should be handled.

Hi,
I am experiencing the same bug, for some reasons, the media player state isn't updating at all, and only a reload of the integration can fix the bug, I don't know for how much time. This is really problematic because many automations depend on these states.
I'm not a python coder but I wish someone will put his hands on it to fix it, cause this integration is so popular.
Anyway, thanks to  the author, and I hope this bug can be fixed in the future.

I'm not sure if it's the same or at least a similar issue, but competitor addons have also had issues recently (ioBroker, node-red) after changes at Amazon. These are both dependent on the alexa-remote2 library, where the developer has now made some adjustments to make it work again temporarily.

.

Hello if that issue started recently I don't think it's the same as this issue is present from ever, years at least.

this issue is present from ever, years at least.

You sure about that? This ticket was only opened in the last few months:

I also only started encountering this issue around that time, give or take. If the issue existed before that perhaps something has made it get worse recently so more people started encountering it / noticing?

From my perspective it could be that the websocket is closing without notice so it stops to receive updates from Amazon.

That's not how I interpreted Alan's analysis of the log file, unless I misunderstood?
https://github.com/custom-components/alexa_media_player/issues/1953#issuecomment-1579807928

Here it starts only in May as well. But I have the feeling that it is getting even worse in the last days and week. My timers are now not updating anymore in 90%. Not when they are started and (this is new) not even when they are then finished/outdated.

this issue is present from ever, years at least.

You sure about that? This ticket was only opened in the last few months

Yes, I'm absolutely sure about that. There was previous issues about this argument that was closed, here the first I've found where I've participating in: https://github.com/custom-components/alexa_media_player/issues/1578
That's more than a year ago (and the problem was there from well before).

I also only started encountering this issue around that time, give or take. If the issue existed before that perhaps something has made it get worse recently so more people started encountering it / noticing?

I don't know why that problem arrives in waves, maybe when Amazon changes something then it happens more frequently, who knows.
In fact if you don't use volume_level or timers, or play status of Alexa devices there a good chance you'll even not noticing this issue is there as it's not preventing announcements and other commands from being run. It makes status updates from Amazon not arriving so, for example, you change the volume of a device by voice but that volume in the Home Assistant entity don't change.
I noticed that from much time as I've developed multinotify, a script that, between the other functions, backup the volume and play status of devices, play the announcement then restore the volume of every device and resume the play where it was paused.
When the websocket stops working that script is backing up a not real status and that makes you understand to be in that condition easily.

From my perspective it could be that the websocket is closing without notice so it stops to receive updates from Amazon.

That's not how I interpreted Alan's analysis of the log file, unless I misunderstood?
#1953 (comment)

Yes, you're right. This could be a brand new issue with same symptoms that was there from ever. But stating that my Alexa devices are working as before (they update with the Echo devices status then, after a random amount of time, they stops updating) I thought that this issue was related to that condition that there from ages.

Similar issue here, I got my Alexa devices a few weeks ago and I NEVER got the players to update state in Home assistant. I play music by telling Alexa to play and all media_player entities are still in standby. I have reinstalled the integration multiple times, nothing helps. I have an Echo Show 5, an Echo Studio and a Wiim Pro, none update status.

Today I detected the first time, that timers are not updated anymore. Restart HA or refresh entities has no effect.

Here homeassistant.update_entity is working sometimes. I seems that it is only working some minutes after starting a timer and some minutes or hours.

Ok, obviously the timers are not supported in the interface by amazon any more. So I deactivated all entities and took them out of the views. As I had no further automation on these entities, it is ok for me.  It was sometimes nice to see, that a timer is active but so what...
The issue with the media players is more relevant but works for my with my workaround see 14 posts above.
In addition to that I relaod alexa once a night. That clears the entities from old states which are no longer correct.
I everything stayed like this..I can live with it.
I hope that amazon will not reduce further functions because reminders and alarms have a relevance for me.

Same issue here. Restarting the service once a day (it is part of a morning routine I run atomically) helped for some time, but now also this is not anymore the case :(

Ok, obviously the timers are not supported in the interface by amazon any more. So I deactivated all entities and took them out of the views. As I had no further automation on these entities, it is ok for me. It was sometimes nice to see, that a timer is active but so what... The issue with the media players is more relevant but works for my with my workaround see 14 posts above. In addition to that I relaod alexa once a night. That clears the entities from old states which are no longer correct.
If everything stayed like this..I can live with it. I hope that amazon will not reduce further functions because reminders and alarms have a relevance for me.

I solved my personal issue of alarms with restarting the instance periodically.
Therefore removing the entitys would be not the best solution for some of us.
I understood amazon skipped the event based functionality of reporting entity values. But by restarting the integration it is still possible to read the current alarm values.
=> why not implementing a poll function  which peroidically requests the values. Or make a service availabe for the useres to force updating the values manually?

Today I noticed that I am having the same issue where the media player (Alexa) status isn't updating for any of its attributes unless the entity is reloaded manually. Ive created an automation to overcome this by making the trigger for any status change in the media player to reload the "Home Assistant Core Integration: Update entity". This works but isn't really a solution.

=> why not implementing a poll function which peroidically requests the values. Or make a service availabe for the useres to force updating the values manually?

https://github.com/custom-components/alexa_media_player/issues/1953#issuecomment-1652111938
I wish there was a way to sticky a post.

Just to add a bit of context:
My HA setup shows the same issue, so I implemented a periodical update_entity–thanks for the idea!
Looking at the logs, it would appear that media_position is updated every 20 seconds or so as long as a track is playing, but when the track ends and a new one starts, the updating stops. (I was thinking of creating an automation that would trigger update_entity when media_duration equals media_position, but alas no such luck.)
I guess you could somehow calculate the right moment to trigger an update by comparing media_duration, media_position and media_position_updated_at?

I already tried to use update_entity EG after starting a timer or after the timer is finished. But without success. The update_entity ist not reliable and not pulling the right state most of the time. Even without error it is only delivering the right state sometimes. In the example e.g. only random 5-15 minutes after starting or ending a timer. Why ever.

I already tried to use update_entity EG after starting a timer or after the timer is finished. But without success. The update_entity ist not reliable and not pulling the right state most of the time. Even without error it is only delivering the right state sometimes. In the example e.g. only random 5-15 minutes after starting or ending a timer. Why ever.

Did you use > homeassistant.reload_config_entry

Same here. Will wait for updates. If any help is needed, please reach out.

Seems that this integration is going to be abandoned very soon, when such fundamental issues are not fixed anymore. I really loved it.

Seems that this integration is going to be abandoned very soon, when such fundamental issues are not fixed anymore. I really loved it.

I agree, I’m hoping that nabu casa comes out with some hardware at the end of the year of the voice to compete with Alexa, then we can let this integration die

I agree, I’m hoping that nabu casa comes out with some hardware at the end of the year of the voice to compete with Alexa, then we can let this integration die

You seem to have misunderstood something here.
This custom component is not used to transmit voice commands to Home Assistant.
This custom component is used to control Amazon Alexa and Amazon Echo devices.
Voice commands are transmitted to Home Assistant via the Nabu Casa Skill and the Nabu Casa Cloud.

I don’t misunderstand anything. This is the year of the voice, and my hope is that they come up with hardware that communicates directly with THEIR assist assistant so I can get rid of my Alexa devices and their reliance on Amazon and their cloud services to work.

This is the year of the voice, and my hope is that they come up with hardware that communicates directly

I dont think thats going to happen anytime soon :(

This is the year of the voice, and my hope is that they come up with hardware that communicates directly

I dont think thats going to happen anytime soon :(

Why not? They already have voice integrated with a number of other devices. All they need to implement is a wake word and we’re in business. The Muse LUXE speaker already exists, running esphome firmware. I think it’s very reasonable to expect something like this.

Eventually yes. But the wake word is the super hard part apparently

but for now, still having problems with Alexa Media Player and entities that are not updating... What a shame to no longer see the music, alarms and other things!

This might help someone that uses Spotify and Node Red I used the swtich node to get the source

This might help someone that uses Spotify and Node Red I used the swtich node to get the source

This issue is about overall not updating state. How using attribute could help?
Why use NodeRED for simple state attribute, if you can create template sensor, or use it directly in automation?

Either you're bot/troll, or you got mistaken with the issue.

LOL ok... or it depends on how you're using the states. I use it for follow me music. I'm saying, that if you're using it to get the media state on a specific Spotify device. You can use a switch node to pass that info along.
Its not for every use case, I said that :)
And why would you go to the trouble of creating a template sensor for something as simple as passing a state attribute ;)
But thank you for letting me know that it didn't apply to you, I appreciate it big guy!

LOL ok... or it depends on how you're using the states. I use it for follow me music. I'm saying, that if you're using it to get the media state on a specific Spotify device. You can use a switch node to pass that info along.
Its not for every use case, I said that :)
And why would you go to the trouble of creating a template sensor for something as simple as passing a state attribute ;)

Still don't need NodeRED for it. I'm saying because I used to use RED extensively in the past.
For our problem: state doesn't have an update AT ALL, so no changes to attributes as well...

But you can still use Node-RED here as a (temporary) workaround to get the events.
This is done with the node-red-contrib-alexa-remote2-applestrudel node, which gets its functionality from the alexa-remote2 lib.
And this alexa-remote2 lib is mostly updated in time by its developer when changes from Amazon cause problems. Like the ones from the last few weeks. You only have to activate the events in the settings of the node, because this is deactivated by default.
As a bridge between Node-RED and Home Assistant you can use MQTT or the node-red-contrib-home-assistant-websocket node.

Got you. So NodeRED lib works at least.
Not that I want to do it, as it will require to re-do everything related to Alexa.. but I see your point!

What I'm saying is its clear this integration is dead. The dev does not want to fix it, so we all need to figure out other ways to make get the state values we need.
I know its a hassle, I've brought this guy a "coffee" before. But that's just the way it goes. I got a couple of great years out of it, so it was worth it. Now we have to figure out something else. If I was a dev, believe me I would have been fixing this already LOL.

Let's ask @alandtse if this is something that's on roadmap. Last release was 3 days ago - so maybe it's not lost yet.

Given the trends, I'm expecting for Amazon to shut down the ecosystem soon. So not in my roadmap but I will take PRs of people figure it out without just spiking the polling. That's why it's tagged as help wanted.

@alandtse thanks for quick reply. I understand, that Amazon can shut that door once and forever (given it's still open, of course)...
It's been a while since I was tinkering with Python, but I'll try to dig it a bit, if you put my face in most low-level API file, please. :)

Can I just ask as well, when you say ecosystem. Are you talking about the media player portion of that? So Alexa will stop allowing people to play music. Or the whole thing?

This integration is using unofficial API. It's nothing to do with Alexa functionality - it's about third-party access to Alexa functions. And it's not guaranteed to work in any way.

Ahh ok got ya. thanks :)

Openhab has also currently issues with their integration (Last Voice command no longer reported), and one of the maintainers said its due to Amazon is no longer reporting last voice command state changes via websocket. Instead Amazon is now reporting via HTTP/2 stream. Could it be that this linked, and the same is true for the media player states?
https://community.openhab.org/t/amazon-echo-smartj-binding-last-voice-command-not-working-anymore/148296/68

Seems you are right, in fact the "alexa-remote" project (javascript based api) switched to HTTP2 streams a few weeks ago
https://github.com/Apollon77/alexa-remote/releases/tag/v6.0.0
an interesting fact is that this project "alexa media player" is using the python package "alexapy" (https://gitlab.com/keatontaylor/alexapy) which is based on the "alexa-remote" mentioned above
there are some python projects implementing https2 streams, e.g. https://github.com/python-hyper/h2 however
however seems a complex change

Hopefully someone smarter than I can put in a PR to bring back this functionality. This sounds likely that this is the cause of the issues we're all experiencing. If there's anything I can do to assist I will gladly, but python is a little outside of my ability.

I tried proposed NodeRED alternative - and it's actually lightning-fast in TTS announcements, skills like Actionable Notifications etc. No subscriptions though, as far as I understand. So returning to status-quo would be preferable - and then I can slowly move some functionality, not related to media player itself, to NodeRED. :)

Ive never not used Node Red, got into it early on and never looked back lol.
I dont think I gave enough info on how Im using it.
This is one example. So now that Alexa devices no longer reliably report current state as checks, I'm now using Spotify entity and then the switch node to filter for Alexa devices.

But the TTS is also great, I use it to let me know when prints are finished, % of prints, that particular windows need to be closed before turning on the aircon, washing machines finished, I could go on lol

I mean push state updates. Exactly what we've lost here  :)

https://github.com/Apollon77/alexa-remote/compare/v5.10.3...v6.0.0 shows the changes to enable http2 pushes. Unfortunately, alexapy is built on aiohttp, which does not support http2.
Httpx does support http2. However, swapping from aiohttp to httpx is a large undertaking. Perhaps it can be done by just replacing the websocket with httpx. If someone submits a PR to gitlab to address this, ping me here. I don't check gitlab anymore.

I tried proposed NodeRED alternative - and it's actually lightning-fast in TTS announcements, skills like Actionable Notifications etc. No subscriptions though, as far as I understand. So returning to status-quo would be preferable - and then I can slowly move some functionality, not related to media player itself, to NodeRED. :)

Does Node Red show last_called and timers?

I tried proposed NodeRED alternative - and it's actually lightning-fast in TTS announcements, skills like Actionable Notifications etc. No subscriptions though, as far as I understand. So returning to status-quo would be preferable - and then I can slowly move some functionality, not related to media player itself, to NodeRED. :)

Does Node Red show last_called and timers?

Last called can be retrieved by getting Activities and picking the latest one. It does even tell what was the request.
Timers/alarms are returned with Notifications for device.
You can play with it yourself, it's easy.

I tried proposed NodeRED alternative - and it's actually lightning-fast in TTS announcements, skills like Actionable Notifications etc. No subscriptions though, as far as I understand. So returning to status-quo would be preferable - and then I can slowly move some functionality, not related to media player itself, to NodeRED. :)

Does Node Red show last_called and timers?

Last called can be retrieved by getting Activities and picking the latest one. It does even tell what was the request.
Timers/alarms are returned with Notifications for device.
You can play with it yourself, it's easy.

Any YouTube/other guides you recommend? I’ve never played around with node red before

I tried proposed NodeRED alternative - and it's actually lightning-fast in TTS announcements, skills like Actionable Notifications etc. No subscriptions though, as far as I understand. So returning to status-quo would be preferable - and then I can slowly move some functionality, not related to media player itself, to NodeRED. :)

Does Node Red show last_called and timers?

Last called can be retrieved by getting Activities and picking the latest one. It does even tell what was the request.
Timers/alarms are returned with Notifications for device.
You can play with it yourself, it's easy.

Any YouTube/other guides you recommend? I’ve never played around with node red before

Here you go, one of my favourite YouTubers: https://youtu.be/hBEb_FCLRU8?si=7YoE6HbN9ExgtijF

This fixes this problem for me. It reloads all my Alexas every 30 seconds. Replace my alexa entities with your own and you are good to go.
alias: Home | Reload all Alexas
description: ""
trigger:

platform: time_pattern
hours: ""
minutes: ""
seconds: /30
condition: []
action:
service: homeassistant.reload_config_entry
data: {}
target:
entity_id:
- media_player.lounge
- media_player.echo_isabella
- media_player.echo_dot_bathroom
- media_player.echo_dot_bedroom
- media_player.echo_show
- media_player.echo_living_room
mode: single

This fixes this problem for me. It reloads all my Alexas every 30 seconds. Replace my alexa entities with your own and you are good to go.
alias: Home | Reload all Alexas
description: ""
trigger:
  - platform: time_pattern
    hours: "*"
    minutes: "*"
    seconds: /30
condition: []
action:
  - service: homeassistant.reload_config_entry
    data: {}
    target:
      entity_id:
        - media_player.lounge
        - media_player.echo_isabella
        - media_player.echo_dot_bathroom
        - media_player.echo_dot_bedroom
        - media_player.echo_show
        - media_player.echo_living_room
mode: single

Great it works for you. However, it's partial, and really bug-prone solution. Also, you don't need to put all entities there - they all are under one config entry, so putting one media player I'd would be enough to reload all of them.

@formatBCE New to Node Red here, installed and configured based on your video recommendation (thanks!). Is there a way to set up a trigger for an Alexa timer being set? I looked at the device trigger node, but it doesn't seem to be what I need.

@formatBCE New to Node Red here, installed and configured based on your video recommendation (thanks!). Is there a way to set up a trigger for an Alexa timer being set? I looked at the device trigger node, but it doesn't seem to be what I need.

That's what issue currently is about.
You see, there's two types of dealing with information updates: pushing and polling. Pushing is great, and allows information to update as it goes (like, timer was set -> you received push trigger on it). But it requires Alexa to have a mechanism for subscription to that updates. That's what has changed lately - and that's why our integration doesn't work.
In case of NodeRED and Alexa integration - I don't see any signs of push subscription possibility. So you'd have to use polling.
Polling is easier to do, but it has huge disadvantages:

It doesn't update instantly, but only on next poll cycle. So if you're polling each 10 seconds, you can completely miss 9-second timer, for example.
If you make polling period shorter, it will greatly load Alexa API  (and, probably, your server), and most of that requests will be wasting of computing capacity, because no new information will be polled...

Long story short: you can poll. No push. I won't recommend that. :)

@formatBCE Ah, thanks. I was hoping NodeRED would let me get around the issue, but not in my case. Bummer.
I wish it was easier to set up an environment to write addons for HA, then I might be able to look into the aiohttp to httpx conversion and help out with the plugin. I'm primarily a web developer and I'm used to working with C#, PHP, and JavaScript, but I have a tiny bit of Python experience. I keep thinking of helping out, but the barrier to entry seems pretty high.

@Kethlak well, you don't need to write add-on in this case - it's just custom integration, so basically you can test it by changing the code directly inside of custom_components folder in Home Assistant. :)
The hardest part, I guess, is tinkering with your own Amazon account...

This fixes this problem for me. It reloads all my Alexas every 30 seconds. Replace my alexa entities with your own and you are good to go.
alias: Home | Reload all Alexas
description: ""
trigger:
  - platform: time_pattern
    hours: "*"
    minutes: "*"
    seconds: /30
condition: []
action:
  - service: homeassistant.reload_config_entry
    data: {}
    target:
      entity_id:
        - media_player.lounge
        - media_player.echo_isabella
        - media_player.echo_dot_bathroom
        - media_player.echo_dot_bedroom
        - media_player.echo_show
        - media_player.echo_living_room
mode: single

Great it works for you. However, it's partial, and really bug-prone solution. Also, you don't need to put all entities there - they all are under one config entry, so putting one media player I'd would be enough to reload all of them.
P.S. also, while it's reloading, devices are unavailable - and some automations, that use these devices, can be broken. I'm assuming you're not using Alexa for important TTS notifications.

Works great for me and it doesn’t make them go unavailable. We’re just updating the entities, not reloading the integration. This seems to be a decent workaround for now.

Works great for me and it doesn’t make them go unavailable. We’re just updating the entities, not reloading the integration. This seems to be a decent workaround for now.

You are still wrong here. reload_config_entry is reloading the integration. See https://www.home-assistant.io/integrations/homeassistant/#service-homeassistantreload_config_entry.
Updating an entities is via update_entity. See https://www.home-assistant.io/integrations/homeassistant/#service-homeassistantreload_config_entry.

My fault, my automation is using update entity.

My fault, my automation is using update entity.

Can you please provide your automation which doesn't restart the integration but can reload/update the entity values?

Here is a script that will update an entity, but it rarely works for me. The timers continue to be Unknown no matter how many times I spam it:
alias: Reload Alexa Library Timer
sequence:
  - service: homeassistant.update_entity
    data: {}
    target:
      entity_id: sensor.library_dot_next_timer
mode: restart

Perhaps I'm targeting the wrong kind of entity for this workaround?

alias: "☑ Misc: Update Alexa Devices"
description: ""
trigger:

platform: time_pattern
seconds: /30
condition: []
action:
service: homeassistant.update_entity
data: {}
target:
entity_id:
- media_player.office_echo
- media_player.gym_echo_dot
- media_player.entryway_echo_dot
- media_player.playroom_echo_dot
- media_player.downstairs_bathroom_echo_dot
- media_player.kitchen_echo_show
- media_player.living_room_echo
- media_player.master_bathroom_echo_dot
- media_player.kids_bathroom_echo_dot
- media_player.justins_echo_dot
- media_player.charlottes_echo_spot
- sensor.master_bedroom_echo_show_next_alarm
- media_player.master_bedroom_echo_show
- sensor.charlottes_echo_spot_next_alarm
mode: single

From my experiences, every 30 seconds is not needed at all, because the service only delivers new status ~ every 10 minutes +/-. At least for the timers.  FInally i can use the integration again!

From my experiences, every 30 seconds is not needed at all, because the service only delivers new status ~ every 10 minutes +/-. At least for the timers. Whyever.

Not true when you example change the volume of the echo.

Spoke too soon ... updates worked instantly for a device the first times. But now now updating again :-(

Spoke too soon ... updates worked instantly for a device the first times. But now now updating again :-(

Because you're bombing Alexa API with requests each 30 seconds, and I guess after some time API starts to debounce it. That's why polling is bad, and these rude solutions are worse than no solution.

polling is bad, and these rude solutions are worse than no solution

This. 100%.
Unfortunately it seems the only real solution here is to he the new HTTP/2 API.

Ok, I'm trying to put actual information in this thread to try to resolve this. However, with all the chatter, it's going to get buried. Please take the workaround chatter to the discussion forum.
Also, can I get a volunteer to go through this thread and help clean it up and maybe more moderation?
Anyway back to the technical:

The AWS http2 protocol may actually be https://developer.amazon.com/en-US/docs/alexa/alexa-voice-service/http2-overview.html.
At least one person tried to get httpx's http2 to work with AWS. They deleted their repo though but there may be more pointers through this issue and the linked issues. https://github.com/encode/httpx/issues/1410

This may be useful. https://github.com/mkb79/Audible. Apparently for audible registration access_token got deprecated.
Regardless, I may be able to do a http2 release but users will have to manually generate an access_token. https://github.com/Apollon77/alexa-remote. For some reason the way we generate tokens does not allow http2 connections. It may very well be my account is blocked due to too many connections.

Is there anything we can do to help? Would it be beneficial if we tried to generate a token to confirm your blocked account theory?

@alandtse am I reading correctly that alexa-remote is using a local connection to the alexa devices instead of a cloud connection such as this library is currently using (I think)?

Ok, I'm trying to put actual information in this thread to try to resolve this. However, with all the chatter, it's going to get buried. Please take the workaround chatter to the discussion forum.
Also, can I get a volunteer to go through this thread and help clean it up and maybe more moderation?
Anyway back to the technical:

The AWS http2 protocol may actually be https://developer.amazon.com/en-US/docs/alexa/alexa-voice-service/http2-overview.html.
At least one person tried to get httpx's http2 to work with AWS. They deleted their repo though but there may be more pointers through this issue and the linked issues. HTTP2 Sync Client blocked after attempting to establish a downchannel to Alexa Voice Service encode/httpx#1410

look to discussion i paste my config from node red now playing status is refreshing

Thank you for getting this updated so quickly!
I've installed the latest version, restarted HA and had an Echo Dot play music for +5 minutes. The media_player entity didn't update during the time and this was the only entry in the log:
2023-09-29 09:48:41.068 DEBUG (MainThread) [alexapy.alexahttp2] Preparing ping to https://bob-dispatch-prod-na.amazon.com/ping
2023-09-29 09:48:41.159 DEBUG (MainThread) [alexapy.alexahttp2] Received response: 204:

Is there something else that I should be doing?

Thanks as well from my side!
But can not confirm what WhimsySpoon mentioned .. just updated and tested as well, and for me it's working perfectly fine again!

It's possible you have to relogin to force a new access token to get generated. Remove and readd the integration,.

Even after relogging, push updates don't work. This might be related:
2023-09-29 11:29:48.092 DEBUG (MainThread) [alexapy.alexahttp2] Received raw message: --------abcde123
2023-09-29 11:29:48.112 DEBUG (MainThread) [alexapy.alexahttp2] HTTP2 Connection Closed.
2023-09-29 11:29:48.113 DEBUG (MainThread) [alexapy.alexahttp2] HTTP2 Connection Closed.
2023-09-29 11:29:48.114 DEBUG (MainThread) [custom_components.alexa_media] c*********@g*******m: Close requested; will not reconnect http2
2023-09-29 11:29:48.115 DEBUG (MainThread) [custom_components.alexa_media] c*********@g*******m: Close requested; will not reconnect http2

Also for me after removing the integration and readd/relogin, media players does not update.
I have enbled debug logging for the compontent, but there was no usefull information why it fails.
The initial fetch of data works.
Anything we can provide to help out?

I don't remember seeing this Public URL option before. Is it relevant? I've populated it but it hasn't changed anything, but I won't be in a position to remove and re-add the integration until later today.

Also for me after removing the integration and readd/relogin, media players does not update. I have enbled debug logging for the compontent, but there was no usefull information why it fails. The initial fetch of data works.
Anything we can provide to help out?

same as me. nothing in logs but all entites are unavailable:

I don't remember seeing this Public URL option before. Is it relevant? I've populated it but it hasn't changed anything, but I won't be in a position to remove and re-add the integration until later today.

I removed, re-added and inserted my URL into the configuration and restarted. Unfortunately there was no change.

@all which are having issues with the new version and @all where it does work:
Which site do you using? in my case, i am using amazon.de

I'm having issues with v4.7.3 using amazon.co.uk

Having issues with amazon.de.

Having issues, not seeing entities update, with 4.7.3 and amazon.com.

Working perfectly (tested only 1x time on an Echo Show) "Set a timer / stop time" on 4.7.3. I will update this comment if it stops working but so far so good.
I made no changes except installing the update and restarting Home Assistant.
Config data including region URLs:
var ue_id = 'AMXXXXXXXXXXX',
    ue_url = '/ap/uedata',
    ue_navtiming = 1,
    ue_mid = 'XXXXXXXX',
    ue_sid = '141-XXXXXXX-XXXXXX',
    ue_sn = 'www.amazon.com',
    ue_furl = 'fls-na.amazon.com',
    ue_surl = 'https://unagi-na.amazon.com/1/events/com.amazon.csm.nexusclient.prod',
    ue_int = 0,
    ue_fcsn = 1,
    ue_urt = 3,
    ue_rpl_ns = 'cel-rpl',
    ue_ddq = 1,
    ue_fpf = '//fls-na.amazon.com/1/batch/1/OP/ATVPDKIKX0DER:141-XXXXXXX-XXXXXX:XXXXXXXXX$uedata=s:',
    ue_sbuimp = 1,
    ue_ibft = 0,
    ue_fnt = 0,

If you play some media, does it update too?

Thx at all, so it has nothing todo with a particular site.
amaisano: interessting that it works for you and for the most of us not.
Does it update when you play some media?

Yes. It works flawlessly with Media controls. I can start it with voice, and pause it with HA, and resume it with voice. The HA status is delayed by 1-2 seconds, but it doesn't get much better than this.

@ALL which are having issues with the new version and @ALL where it does work:
Which site do you using? in my case, i am using amazon.de

Still the issue with media player entities. – using amazon.de
they are not updating at all. - in the alexa app everything is fine.
Tried a lot of things: reinstalling the integration, restarting all echo devices, restarting HA, reloading integration. Cant even get the right media after reloading the integration anymore.
the log: no occurences of the alexa media player integration.

I just installed the 4.7.3 update and restarted the home assistant server and everything started working perfectly!

After removing and re-adding the integration with 4.7.3, I am once again getting push messages, but the entites are only updating after a subsequent voice command.
Example:
I say "Alexa, set an alarm for 7:30am"
Next alarm sensor shows "Unknown"
I say "Alexa, what time is it?"
Next alarm sensor shows "Saturday September 30 7:30am"
I say "Alexa, cancel alarm"
Next alarm sensor shows "Saturday September 30 7:30am"
I say "What is today's weather"
Next alarm sensor shows "Unknown""
I've repeated this with varying lengths of delay in between steps from 5 seconds to 10 minutes with the same result. I believe that some of the duplicate suppression logic introduced to overcome previous issues might be causing the problem here and will dig into that code.

Update from https://github.com/custom-components/alexa_media_player/issues/1953#issuecomment-1741209351
Timers updated instantly both being set, snoozed, cancelled, and stopped on my Echo Show device. But my 5th gen "snowball" echo dot in the other room is still having problems.
I am not getting updates to timers in HA until I click my "ha.update_entity" service on it. I waited about 10 seconds to see if it was just taking longer. Note, this is an improvement still because the update_entity service is actually working, whereas before it still wouldn't see any changes...
But still, odd...

same issue for me: v4.7.3 , amazon.it

After removing and re-adding the integration with 4.7.3, I am once again getting push messages, but the entites are only updating after a subsequent voice command.
Example: I say "Alexa, set an alarm for 7:30am" Next alarm sensor shows "Unknown" I say "Alexa, what time is it?" Next alarm sensor shows "Saturday September 30 7:30am" I say "Alexa, cancel alarm" Next alarm sensor shows "Saturday September 30 7:30am" I say "What is today's weather" Next alarm sensor shows "Unknown""
I've repeated this with varying lengths of delay in between steps from 5 seconds to 10 minutes with the same result. I believe that some of the duplicate suppression logic introduced to overcome previous issues might be causing the problem here and will dig into that code.

I think that is what I'm experiencing as well, and agree. I forgot that I patched some of the code a while ago to and workaround this, but I believe those patches were destroyed when I installed 4.7.3. Sounds like some of them were added directly to the codebase though. Good luck - happy to test/confirm.

I fixed the issue with timers and alarms for me by partially revering https://github.com/custom-components/alexa_media_player/commit/8f0dfa2f6585ebac08bd5a708951155c80f63588
It seems that PUSH_NOTIFICATION_CHANGE events are implemented by Amazon over HTTP2. Restoring this bit of code has made my timers and alarms update within 5 seconds of a change on the device, and has the added bonus of responding to changes made on the Alexa app as well. I'll put together a small PR for the change.

4.7.3, amazon.ca
Media updates are good (sometimes lagging for up to 5 seconds, but stable).
Setting timer updated right away, but canceling did update only after another query to the echo.
Overall, I guess, it's enough for me.
Thanks for fixing!

Apparently alexapy hardcodes the us endpoint, hence eu instances don't receive push updates. Changing the authority to bob-dispatch-prod-eu.amazon.com fixed the issue.

@francescolf
Nice find!
Didn't found alexapy in custom_components folder.
Where did you change the authority?
Thx

Didn't found alexapy in custom_components folder.

You should find it in your Python packages folder. The complete path should be /usr/local/lib/python3.11/site-packages/alexapy/alexahttp2.py

Amazing find. Any way to change that on HAOS?

Amazing find. Any way to change that on HAOS?

I'm not very familiar with HAOS. If the path above doesn't work try one of these methods.

4.7.3
amazon.com
Primary use case is to control Echo Glows which are "devices connected via Echo." Deleted and re-created the integration after upload, but these are still not updating. Clicking a Glow on in HA turns on the light, but HA pauses for a bit then the UI clicks back to off presumably because the service had not updated to on. The light remains on.

Apparently alexapy hardcodes the us endpoint, hence eu instances don't receive push updates. Changing the authority to bob-dispatch-prod-eu.amazon.com fixed the issue.

Will be fixed with version 4.7.4?

Apparently alexapy hardcodes the us endpoint, hence eu instances don't receive push updates. Changing the authority to bob-dispatch-prod-eu.amazon.com fixed the issue.

How about for Amazon.com.au?

What is the appropriate way to update to 4.7.3?  I'm running HAOS and my version number seems to be 4.6.5 (discovered by inspecting const.py) even after deleting the integration, rebooting, re-adding the integration and rebooting.  Do I have to 1) wait for the next HA update, 2)manually replace all the files in the /custom-components/alexa_media_player folder or 3) something else?

How about for Amazon.com.au?

bob-dispatch-prod-fe.amazon.com
I'll try to patch alexapy to automatically choose the correct host.

What is the appropriate way to update to 4.7.3?  I'm running HAOS and my version number seems to be 4.6.5 (discovered by inspecting const.py) even after deleting the integration, rebooting, re-adding the integration and rebooting.  Do I have to 1) wait for the next HA update, 2)manually replace all the files in the /custom-components/alexa_media_player folder or 3) something else?

I went to repo in HACS and in menu pushed "update info". After that it popped out.

ugh.  I confused this with one of the built-in integrations.  HACS does indeed allow me to update.  Thanks!

ugh. I confused this with one of the built-in integrations. HACS does indeed allow me to update. Thanks!

redownload it from hacs on the right:

Please note, we'll do another release to address this but after that releases will slow down. We are missing many translations for some of the new keys that we were required to add and they only get added during a release. If you're in a non-English locale, it'd be helpful to get those filled out or you'll just see blanks where we don't have your language.

How about for Amazon.com.au?

bob-dispatch-prod-fe.amazon.com
I'll try to patch alexapy to automatically choose the correct host.

https://github.com/Apollon77/alexa-remote/blob/master/alexa-http2push.js#L33-L44 has the conversions.

Please note, we'll do another release to address this but after that releases will slow down. We are missing many translations for some of the new keys that we were required to add and they only get added during a release. If you're in a non-English locale, it'd be helpful to get those filled out or you'll just see blanks where we don't have your language.

How about for Amazon.com.au?

bob-dispatch-prod-fe.amazon.com
I'll try to patch alexapy to automatically choose the correct host.

https://github.com/Apollon77/alexa-remote/blob/master/alexa-http2push.js#L33-L44 has the conversions.

Keep in mind, the conversions aren't complete! amazon.de (Germany) is missing for example. https://github.com/Apollon77/alexa-remote/blob/master/alexa-http2push.js#L33-L44

Keep in mind, the conversions aren't complete! amazon.de (Germany) is missing for example. https://github.com/Apollon77/alexa-remote/blob/master/alexa-http2push.js#L33-L44

Amazon.co.uk is also missing!

amazon.de (or amazon.it, amazon.fr, amazon.uk etc.) is not missing, it's covered by the default selection
(https://github.com/Apollon77/alexa-remote/blob/master/alexa-http2push.js#L33)

4.7.4 works as expected.
Thanks alandtse 👍

Also for me 4.7.4 works as expected
Thanks a lot alantse!!

4.7.4 only works 5 times and stops working. music states are not updated after 5 or 6 changes. 😫

You are right, it also stopped working for me now here, after some notifications.

Would be worth if you all test a little bit deeper and longer instead of returning, that it is working directly now and there.

Also make sure you're using v4.7.5, which should fix this specific problem.

Confirmed that 4.7.5 fixes the issue.
I noticed the same problem in 4.7.4 where eventually the HTTP/2 connection would close with:
[alexapy.alexahttp2] HTTP2 Connection Closed.
I've been running 4.7.5 for about 20 minutes and I've lost count of how many commands I sent and the connection has remained open.
Great work on this to all involved. Thank you.

4.7.5 : ❤❤❤

I must be in the minority here. 4.7.3 worked mostly fine for me, but 4.7.4 and 4.7.5 do not work for me at all. I’m in Canada, and I honestly don’t remember what server I’m connected to, but something in 4.7.4 definitely broke things for me. I’ll have to do some more testing to see if I can pin the issue down.

Confirmed, 4.7.5 fixes the issue for me with Amazon.de.
A huge thank you to everyone who worked hard on this solution. This integration is the most valuable to me of all the HA integrations.

Mine didn’t work but deleting the Integration in HA then adding it again and it worked. No restart needed :) thanks!

everything perfect 4.7.5, thank you

@francescolf or anyone who can manually check: mind checking if this EU domain works for you? alexa.eu.gateway.devices.a2z.com I want to future proof this by switching domains.
https://developer.amazon.com/en-US/docs/alexa/alexa-voice-service/api-overview.html#endpoints

Yes, works fine. They point to the same IP address.

(alexa.eu.gateway.devices.a2z.com is also an alias of default.eu.gateway.devices.a2z.com)

4.7.5
amazon.com
Primary use case of controlling Echo Glows which are "devices connected via Echo" still not working.  Have done delete, reinstall, restart between it all.  Behavior exactly same as before.

Thanks so much, my timer display works again!

After updating to version 4.7.6, today Alexa Media Player requested re-authentication. With previous versions (4.6.5) this had never happened.
I use Amazon EU (IT), Hass OS 2023.9.2.
Also, is so many API calls normal?
I attach screenshots of the relogin notifications:

So I've been updating regularly, including the latest one today, and I've got no volume controls in the entities for all my media players.  They've been missing for probably a couple of months now.
I saw a closed ticket about this which has redirected me to this ticket, which seems also to be closed.  Am I in the right place?  Why is this ticket so very active and also marked as closed? :/

I also no longer have volume controls, plus it always appears to be waiting.
I tried reinstalling several times but to no avail.
System details
Home-assistant (version): 2023.5.3
alexa_media (version from const.py or HA startup): 4.7.6

After updating to version 4.7.6, today Alexa Media Player requested re-authentication. With previous versions (4.6.5) this had never happened.
I use Amazon EU (IT), Hass OS 2023.9.2.
Also, is so many API calls normal?
I attach screenshots of the relogin notifications:

same here '(

Updated to v4.7.6 and everything stopped working.
Uninstalled, went back to v4.7.5 and everything bring back to life :)
PS: using amazon.com.br

Guys: useless to post here further in a closed thread.
The authentication issue is handled here: https://github.com/custom-components/alexa_media_player/issues/2059
and should have been solved with 4.7.7

But did you test 4.7.7? As you said you only reveted from 4.7.5 to 4.7.3.

Then the best way is, to open a new issue on GitHub an provide logs (simply raise the error level).
Maybe you should follow the way from tvizzotto first to aware that after installing you really use the installed version.
4.7.7 is working without any issues here and as the thread is closed, you might not get any real help here.

Then the best way is, to open a new issue on GitHub an provide logs (simply raise the error level). Maybe you should follow the way from tvizzotto first to aware that after installing you really use the installed version. 4.7.7 is working without any issues here and as the thread is closed, you might not get any real help here.

I'm not sure what you mean by "really using the installed version".  I'd be happy to log a new bug, but I have no idea what I'm looking for in the logs.

I also no longer have volume controls, plus it always appears to be waiting.
I tried reinstalling several times but to no avail.
System details
Home-assistant (version): 2023.5.3 alexa_media (version from const.py or HA startup): 4.7.6

Sorry if I’m reviving this, I’m just newly installed the integration and searching for the issues before opening a new one.
I just installed it yesterday, custom commands and scripts running properly but I also don’t have any volume control.

Yeah sadly Amazon have nutered a lot of this. The devs have done there upmost to have this functionality, but evilcorp just keeps shutting it off. We are luckily to have anything at all considering how much Amazon is hemorrhaging with Alexa.
Im hoping that either HA's own voice stuff matures to the point we can just be rid of Alexa and let it die. Or that the paid Alexa tier will reenable this functionality

Also volume is only available when select media is playing. Mainly Amazon music.

Also volume is only available when select media is playing. Mainly Amazon music.

I managed to control the volume using a script and buttons after all

I found controlling media_player.spotify to be the most reliable, but it it works that's awesome :)


Harden ingress security by using per ingress resource backend protocol configuration

Please describe the Improvement and/or Feature Request
Currently whether or not a route for plaintext traffic is opened on a pod is based on the use_https_ingress flag in the osm-config ConfigMap. This presents a security issue when the user does not want to open plaintext routes on services that should only accept encrypted traffic while at the same time allowing plaintext traffic to selective services. Ingress controllers such as nginx-ingress explicitly allow configuring an ingress with the backend protocol to use http or https, which can be leveraged for this purpose or OSM should provide a way for users to configure the backend protocol on a per ingress resource basis (ex. by using an annotation).
This issue is meant to harden ingress security by not using a global flag for cluster wide ingress resources.
Scope (please mark with X where applicable)

New Functionality          [ ]
Install                    [ ]
SMI Traffic Access Policy  [ ]
SMI Traffic Specs Policy   [ ]
SMI Traffic Split Policy   [ ]
Permissive Traffic Policy  [ ]
Ingress                    [X]
Egress                     [ ]
Envoy Control Plane        [ ]
CLI Tool                   [ ]
Metrics                    [ ]
Certificate Management     [ ]
Sidecar Injection          [ ]
Logging                    [ ]
Debugging                  [ ]
Tests                      [ ]
CI System                  [ ]
Project Release            [ ]

Possible use cases
Harden ingress security both HTTPS and HTTP ingress is used.

This issue has been addressed after the implementation of the IngressBackend API as a part of #3779, which allows fine-grained configuration of service protocols to expose for ingress.


Unable to connect to google memorystore redis instance with createClient. Erro: Connection Timeout

Description
Hi, I have setup a google memorystore for redis and got ipaddress and port. When tried to
createClient({socket: {host: 'ip', port: '6379'}), I get connection timedout error. How do I connect with a google memorystore from my computeengine nodejs app?
Node.js Version
18
Redis Server Version
No response
Node Redis Version
No response
Platform
Ubuntu
Logs
No response

redis.createClient(REDISPORT, REDISHOST) is not a valid configuration, and by default it'll try to connect to <IP_ADDRESS>:6179. This is the way to configure the client:
createClient({
  socket: {
    host: 'host/ip',
    port: 6379
  }
);

But this still won't work for you, because memorystore is running in a VPC network (and not connected to "internet"), so in order to connect to it from your local machine you'll need to create an SSH tunnel

depending on what platform is hosting your service (cloud run, compute engine, etc.) you can connect it to the redis memorystore. Here's a helpful doc to figure out how : Memorystore supported environments

@logesh2496 did you solve it?


source raw

Elasticsearch version:1.7.3
Moloch version:0.12.3
I would like to query Moloch API to retrieve the source raw data, what is the database field I should query to get that data?

The source raw isn't in the db.  Easier if you just make a call into viewer.


Don't read entire chunks at a time

Because we're reading entire chunks at a time, we're reading and decompressing a lot more data than we need, which is almost certainly a major bottleneck.
Some ideas:

Try again to use zarr.core.Array(partial_decompress=True) using FSStore & Blosc compression (we're already using Blosc zstd level 5 for NWPs) to read small chunks, especially for NWPs where we're using tiny 2x2 images.
Try with uncompressed Zarr.  It's possible that the performance increase from compression is far smaller than the performance increase of being able to precisely extract just the data we want from each chunk.

If it's possible to quickly load subsets of each chunk, then modify each Zarr DataSource so it no longer pre-reads entire chunks into memory, but instead, for each batch, loads each example separately using a different thread.

highly relevant:

https://github.com/pystatgen/sgkit/issues/437

which includes this text (copy-and-pasted for convenience):

zarr-specs#59 (zarr specs for partial chunk reads)
zarr-python#40 (partial chunk reads feature request)
zarr-python#521 (partial chunk reads feature request)
zarr-python#584 (partial decompress draft PR)
it sounds like even with partial decompress, we will need to fetch full compressed file

PR for partial_decompress, including lots of performance graphs: https://github.com/zarr-developers/zarr-python/pull/667

This issue will be made obsolete (i.e. won't matter) if #58 works :)

Not implemented. Made obsolete by #58


crash when loading backdata after starting a trade command

System information

Have I written custom code (as opposed to using zenbot vanilla): No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 17.10
Zenbot version: 4.05
NodeJS version:  8.9.3
Python version: 2.7.14
Exact command to reproduce:
./zenbot.sh trade  poloniex.ETH-BTC --strategy=neural --manual

Describe the problem
crash when loading backdata after starting a trade command
Source code / logs
bitnami@core:~/zenbot$ ./zenbot.sh trade  poloniex.ETH-BTC --strategy=neural --manual
fetching pre-roll data:
api irunning on <IP_ADDRESS>:20020/trades
poloniex.ETH-BTC saved 200 trades 1 days left
./home/bitnami/zenbot/zenbot.sh: line 2: 13837 Killed                  env node zenbot.js $@
bitnami@core:~/zenbot$ ./zenbot.sh trade  poloniex.ETH-BTC --strategy=neural --manual
fetching pre-roll data:
api running on <IP_ADDRESS>:20048/trades
poloniex.ETH-BTC saved 200 trades 1 days left
skipping 0 hrs of previously collected data
./home/bitnami/zenbot/zenbot.sh: line 2: 13935 Killed                  env node zenbot.js $@
bitnami@core:~/zenbot$ ./zenbot.sh trade  poloniex.ETH-BTC --strategy=neural --manual
fetching pre-roll data:
api running on <IP_ADDRESS>:20089/trades
poloniex.ETH-BTC saved 200 trades 1 days left
skipping 2 hrs of previously collected data
./home/bitnami/zenbot/zenbot.sh: line 2: 14646 Killed                  env node zenbot.js $@
bitnami@core:~/zenbot$ ./zenbot.sh trade  poloniex.ETH-BTC --strategy=neural --manual
fetching pre-roll data:
api running on <IP_ADDRESS>:20035/trades
poloniex.ETH-BTC saved 200 trades 1 days left
skipping 3 hrs of previously collected data
./home/bitnami/zenbot/zenbot.sh: line 2: 15699 Killed                  env node zenbot.js $@

This issue was moved to highvelocityspace/zenbot#33

This issue was moved to highvelocityspace/zenbot#63


Change environment feature does not respect custom home directory via --home flag

System information
Osmosis version: v15.0.0
OS & Version: Linux
Commit hash: ff18d82
Expected behavior
osmosisd init --home custom persists all files to the directory specified by the --home flag.
Actual behaviour
osmosisd init --home custom always creates a directory at $HOME/.osmosisd in addition to any custom directory via --home flag. This behavior cannot be customized or circumvented.
The addition of ChangeEnvironmentCmd() related to https://github.com/osmosis-labs/osmosis/pull/3787 specifically the function CreateEnvFile in InitCmd appears to be the root cause.
Steps to reproduce the behaviour
On your local machine using v15.0.0.

make build
ls ~/.osmosisd will error as expected because it does not exist yet.
./build/osmosisd init --home custom moniker
ls -a ~/.osmosisd was created and has .env file. This is unexpected.

Context
As a node operator, saving data anywhere outside of --home is unexpected, inconsistent behavior. In other words, the current behavior breaks the Principle of Least Surprise.
We always set --home to a consistent directory because of these advantages:

We have a consistent place to look for files. Some chains do not follow a predictable pattern for the chain, binary, and home folder name. A consistent home folder allows us to more easily build automation for inspecting files, for example.
We do not create a hidden (.dot) file so it's more easily discoverable. (i.e. Don't have to remember to always ls -a. )

This seeming innocent changed caused several hours of downtime for our Osmosis RPC deployment because:
Error: Failed to create environment file: mkdir /home/operator/.osmosisd: permission denied

We strive for strict security on all our deployments.
My proposal is to respect the --home flag and save this .env only in the designated home folder. Do not default to $HOME/.osmosisd. Notably, I do not have all the context around this feature. But my intuition is since the default home is $HOME/.osmosisd via simply osmosisd init moniker, then I would hope the feature still works despite the location of the home dir.
The PR's original goal was:

To avoid the user having to add the --HOME flag to each cli cmd, configure the environment permanently by setting it via a separate CLI command.

I appreciate the goal. I'm betting this was explored, but a common pattern in cli apps is to accept a flag OR an environment variable.
Example:
# In my shell session
export COSMOS_CHAIN_HOME=custom

home := "$HOME/.osmosisd"
if v := os.Getenv("COSMOS_CHAIN_HOME"); v != "" {
    home = v
}
cmd.Flags().String(cli.HomeFlag, home, "node's home directory")

Or you may be able to make use of Viper's Env Var features.
To me, this feels like a nice feature for the SDK vs. Osmosis-specific. I agree, it is annoying to constantly set the --home flag.

Hey @DavidNix . Thanks for this issue.
Apologies for this breakage. We will look into this, aiming to support the original --home behavior and the new set-env in conjunction.
cc: @hieuvubk

@p0mvn @DavidNix Could u pls check this pr.
I think instead of ignoring set-env if there is a home flag. We can specify usage with a flag. We will only save the env variable if the init-env flag is used. If not, keep it as cosmos-sdk is doing.
Wdyt?

@p0mvn @DavidNix Could u pls check this pr.
I think instead of ignoring set-env if there is a home flag. We can specify usage with a flag. We will only save the env variable if the init-env flag is used. If not, keep it as cosmos-sdk is doing.
Wdyt?

I appreciate the quick response!
That would solve the issue for us. But, it looks like a backwards breaking change. This won't affect us (I doubt we'll ever use the set-env feature). But for other users, they'll need to be aware of setting this new flag to true.
Candidly, I'm not 100% clear on the value of the set-env feature and why it needs to be a .env file in a well known location versus, for example, letting the user set an env var.