Spotify UPnP error 800

Hi there,
This is a duplicate of https://github.com/home-assistant/core/issues/133052, but I was able to replicate the same issue in SoCo. Trying to play anything to any of my Sonos speakers generates the following error:
Traceback (most recent call last):
  File "/var/home/andrew/socoscript.py", line 56, in <module>
    add_from_service(track_id, service, device, True)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/andrew/socoscript.py", line 53, in add_from_service
    device.add_to_queue(didl)
    ~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/var/home/andrew/soco/lib64/python3.13/site-packages/soco/core.py", line 146, in inner_function
    return function(self, *args, **kwargs)
  File "/var/home/andrew/soco/lib64/python3.13/site-packages/soco/core.py", line 2277, in add_to_queue
    response = self.avTransport.AddURIToQueue(
        [
    ...<6 lines>...
        **kwargs,
    )
  File "/var/home/andrew/soco/lib64/python3.13/site-packages/soco/services.py", line 207, in _dispatcher
    return self.send_command(action, *args, **kwargs)
           ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/andrew/soco/lib64/python3.13/site-packages/soco/services.py", line 515, in send_command
    self.handle_upnp_error(response.text)
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
  File "/var/home/andrew/soco/lib64/python3.13/site-packages/soco/services.py", line 571, in handle_upnp_error
    raise SoCoUPnPException(
    ...<6 lines>...
    )
soco.exceptions.SoCoUPnPException: UPnP Error 800 received:  from X.X.X.X

I have reset my Sonos system entirely, removed and re-added all Spotify accounts from Sonos, and the same thing happens. It works properly if any local, web-based, or stream from a non-Spotify service is started.

The script I am attempting to run when I get this error is: https://gist.github.com/lawrenceakka/2d21dca590b4fa7e3af2

Some time ago (not sure when, but it'll be in the issue history!), Sonos prevented simple addition of URIs to queue for anything requiring authentication, including most music services:
https://docs.python-soco.com/en/latest/faq.html#why-can-t-i-play-a-uri-from-music-service-x-with-the-play-uri-method
A couple of years ago, nascent support was re-added for music services, supporting the required auth.: https://docs.python-soco.com/en/stable/api/soco.music_services.music_service.html#module-soco.music_services.music_service
There's also a simpler mechanism that allows ShareLinks from selected music services to be added to the Sonos queue, using the ShareLink plugin:
from soco.plugins.sharelink import ShareLinkPlugin
share_link = ShareLinkPlugin(speaker)  # speaker is a SoCo object
queue_position = share_link.add_share_link_to_queue(uri, position)

The uri can take the form of (e.g.):

https://open.spotify.com/track/6cpcorzV5cmVjBsuAXq4wD
spotify:album:6wiUBliPe76YAVpNEdidpY
https://tidal.com/browse/album/157273956
https://www.deezer.com/en/playlist/5390258182
https://music.apple.com/dk/album/black-velvet/217502930?i=217503142

This issue is not resolved, it still occurs when the URI is formatted like the above:
>>> from soco import SoCo
>>> device = SoCo('<IP_ADDRESS>')
>>> from soco.plugins.sharelink import ShareLinkPlugin
>>> share_link = ShareLinkPlugin(device)
>>> share_link.add_share_link_to_queue('https://open.spotify.com/track/5m3rVRNkTv5EQnPuF293we?si=cd2c86feb7924e77')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\awilson\Downloads\sonos\Lib\site-packages\soco\plugins\sharelink.py", line 279, in add_share_link_to_queue
    raise fault
  File "C:\Users\awilson\Downloads\sonos\Lib\site-packages\soco\plugins\sharelink.py", line 261, in add_share_link_to_queue
    response = self.soco.avTransport.AddURIToQueue(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\awilson\Downloads\sonos\Lib\site-packages\soco\services.py", line 207, in _dispatcher
    return self.send_command(action, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\awilson\Downloads\sonos\Lib\site-packages\soco\services.py", line 515, in send_command
    self.handle_upnp_error(response.text)
  File "C:\Users\awilson\Downloads\sonos\Lib\site-packages\soco\services.py", line 571, in handle_upnp_error
    raise SoCoUPnPException(
soco.exceptions.SoCoUPnPException: UPnP Error 800 received:  from <IP_ADDRESS>

I get the same result using the exact share link from your example too. Of note, playing Spotify from Sonos works fine, and SoCo can do everything else (play, pause, volume, speaker info) on this speaker with no issue.
I have also reset everything in Sonos (removed all services, restarted speakers multiple times, re-added just the one Spotify account) and it still occurs. This is an active issue on the Home Assistant Github with multiple reports similar to mine with this issue happening suddenly with no configuration changes, and it appears to be originating with SoCo and not just HA. I appreciate your help and am happy to test or troubleshoot anything to assist!

Right, adding sharelinks to the queue still works on Sonos S1, but not on S2, so Sonos has recently changed something in S2.
I've tested Spotify and Apple Music, and both fail.
It'll require investigation to find out if this is fixable, but I'm probably not the person to do this.

same issue since yesterday and tried all above no chance to get a single track or playlist played on one speaker (tryd readding on android and ios spotify )

@monstergubl @pwt this suddenly started working for me after Sonos pushed an update. Assuming this was something they broke on their end and will be fixed as people get the update.


Ensure REST API is up/available for e2e tests

Description
After some testing, I've realized the issues with the e2e tests randomly timing out are related to the REST API backend which is kicked off via Docker.
The e2e timeouts appear to occur when the e2e tests start running before the REST API backend has completed it's startup process.  In other words, there is a small window where the Docker container has already successfully started, but the REST API is not yet available at http://localhost:8080/server/ because Tomcat has not finished booting up.
I was able to verify this by adding a simple `curl http://localhost:8080/server/api' (see https://github.com/DSpace/dspace-angular/pull/1068/commits/12f8e82aa23b6236ce447377a64d1732c00d20cb). I found that command returns a Tomcat 404 error when e2e tests end up timing out, but it returns 200 when they run successfully (here's an early build which shows that behavior) However, this is difficult to reproduce because it is dependent on how long it takes for Tomcat to start/deploy the REST API.
In order to fix this issue, I've found a wait_for_response GitHub Action, which waits until a given URL returns a 200 before allowing following CI steps to proceed. (see https://github.com/DSpace/dspace-angular/pull/1068/commits/ba4c7a8e88cf78fd2fa5748dda90a5adac7b4d65)

@artlowel : Assigning you to this as your guess was correct, the random e2e timeouts are seemingly being caused by the REST API (well, a "race condition" based on whether the e2e tests start running before the REST API is responsive).
I've rebuilt this several times now, and I'm no longer seeing the e2e timeouts...so, it seems to be fixed.  (I do still see the random unit test failures occasionally, but I know you've been working on those in one of your PRs.)

Thanks @tdonohue!
One small comment though: perhaps we can change the URL to check to http://localhost:8080/server/api/core/sites ? That would also tell us that the db is up and running, and populated

@artlowel : I've updated the PR to do the following...

Wait on http://localhost:8080/server/api/core/sites to respond with a 200.  As you mentioned, this is probably a better solution as it means the database is also ready to go.
But, I'm still doing a curl http://localhost:8080/server/api after that.  The reasoning here is that the root endpoint returns properties/info about the REST API...so we can easily verify it's also configured correctly.

It's rebuilding now in GitHub Actions CI>


Persistence stale rowsets meta

Proposed changes
Persistence stale rowsets meta. When be reboots, stale rowsets meta can resume and the stale version can also be readable before stale gc time.
ISSUE: #4453
Types of changes
What types of changes does your code introduce to Doris?
Put an x in the boxes that apply

[] Bugfix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[] Documentation Update (if none of the other choices apply)
[] Code refactor (Modify the code structure, format the code, etc...)

Checklist
Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

[x] I have create an issue on (Fix #4453), and have described the bug/feature there in detail
[x] Compiling and unit tests pass locally with my changes
[] I have added tests that prove my fix is effective or that my feature works
[] If this change need a document change, I have updated the document
[] Any dependent changes have been merged

@ZhangYu0123 Hi, What's the concrete issue that this PR want to Fix? what's the accurate definition for stale rowsets ?

@ZhangYu0123 Hi, What's the concrete issue that this PR want to Fix? what's the accurate definition for stale rowsets ?

https://github.com/apache/incubator-doris/issues/4017

@ZhangYu0123 Hi, What's the concrete issue that this PR want to Fix? what's the accurate definition for stale rowsets ?

#4017

I see. Thanks.
But I think the stale rowsets naming isn't very appropriate. semantically, stale means the data isn't right and we shouldn't read it.

@ZhangYu0123 Hi, What's the concrete issue that this PR want to Fix? what's the accurate definition for stale rowsets ?

#4017

I see. Thanks.
But I think the stale rowsets naming isn't very appropriate. semantically, stale means the data isn't right and we shouldn't read it.

Hi @kangkaisen , I think stale means not fresh, but not means incorrect, so it looks good to me.


[sw, dv] Avoid CSR write reordering

Description
Some CSRs in the device are written to registers in the AON clock domain, so they go through synchronizers. If there is a couple of CSR writes done sufficiently close in time, it is possible they end up being performed in the opposite order they were issued, which can cause unexpected results.
One such instance is when handling an aon_timer wakeup, where the dif code writes to disable the wakeup (writing to AON_TIMER_WKUP_CTRL_REG_OFFSET), and then clear the wakeup cause (writing to AON_TIMER_WKUP_CAUSE_REG_OFFSET). When the order is inverted we end up clearing the cause and in the following AON cycle disabling wakeups, which means the wakeup cause ends up being set again, so the wakeup persists.
One simple way to fix this is to do a read back to AON_TIMER_WKUP_CTRL_REG_OFFSET, since any read will wait for a write to the same CSR to be completed (after synchronization). Notice this can take around 30 us: 3 AON cycles for the write and 3 more for the read. An alternative approach would wait for one AON cycle (5 microseconds), since that is the maximum time the first write can slide after the second, and that can cause the writes to complete at the same time, but not out of order.
The chip_sw_pwrmgr_random_sleep_all_wake_ups test fails for this reason when run with CDC instrumentation enabled and seed 153045692.
The IPs that seem susceptible to this are

adc_ctrl
aon_timer
clkmgr
pinmux
pwm
sysrst_ctrl
usbdev

This is because they instantiate prim_reg_cdc in their CSR register RTL.

Thanks @matutem for finding this!
For context, the register node CDC scheme has been designed in that way on purpose, see slides from Tim Chen here.
Basically we needed a standardized solution since every block with CSRs going to different clock domains handled this in an ad-hoc manner. In particular, the problem was that without extra hardware handling, blocking writes to registers in the AON domain had the potential to dramatically slow down parts of the SW code, which can have a significant impact on boot time etc. Examples for such code sections include initialization of aon_timer and sysrst_ctrl, among others.
However, as observed by @matutem above, the currently implemented scheme can have ordering issues since each register is synchronized separately. The read back solution works to resolve this issue (in fact the slides also mention that SW should use that solution if explicit confirmation for write completion is required). However, I am not sure we want to do that on all AON register writes, since that would defeat the purpose of the parallel synchronization scheme.
It would be good to get feedback on whether software guidance for the memory model is enough or whether more needs to be done - either via standardized functions or additional ordering support in HW (note that the latter may have quite some impact on hardware complexity, though).

CC @vogelpi @GregAC @andreaskurth
I don't think we should change anything in this area for the Earlgrey PROD iteration since the HW itself behaves correctly and changing this synchronization scheme (or introducing more hardware support for ordering) would be rather intrusive. Rather, I think this is a behavior of the current implementation that the SW should be aware of. I am therefore triaging this into the backlog, but please LMK if you have any concerns with that.

This sounds good to me @msfschaffner .

The second write will always arrive to a synchronizer some fast clock cycles later than the first one. I assume the slow clock distribution would guarantee all synchronizers clock rise at around the same time. In other words, this could be an artifact of our CDC instrumentation, but I would like to get feedback about the slow clock expectations.

You may be right, this could be an artifact of our CDC instrumentation since it introduces artificial stagger in the target domain, which is very slow in this case.
The spread of the AON clock distribution is almost certainly narrower than one peri clock cycle. Thus, writes to different registers should not be able to overtake one another. In the worst case, I would expect them to be synchronized over in the same AON clock cycle.

Closing it since it is a CDC instrumentation artifact.


v6 rc3 to rc4 Field is not passing optional props to custom component

First off, thank you for creating this amazing library.
I was disabling an input field in rc3 by doing the following:
const renderInputField = field => (

		{field.input.label}

			<input {...field.input}/>
			{field.touched && field.error && {field.error}}

);

and using it in Field as follows:
<Field name="currency" type="text" className="form-control" 												 label="Currency" component={renderInputField} props={{ disabled: true }} />
However, I have upgraded to rc4 and the following doesn't work anymore.
const renderInputField = ({ input, label, type, meta: { touched, error } }) => (

		{label}

			<input {...input} className="form-control" type={type} />
			{touched && error && {error}}

);

<Field name="currency" type="text" label="Currency" component={renderInputField} props={{ disabled: true }} />
The input rendered is not disabled. I had to move the className prop from Field to input as well. What is the new way to accomplish what I was doing with rc3? Thanks for your help.

Yesterday i tried this from documentation (http://redux-form.com/6.0.0-rc.4/docs/api/Field.md/) and it worked fine. Do you happen to not read RC4 changelog where Field api change is described? (https://github.com/erikras/redux-form/releases)
// outside your render() method
const renderField = (field) => (

      <input {...field.input} type="text"/>
      {field.meta.touched && field.meta.error &&
       {field.meta.error}}

  )

// inside your render() method
<Field name="myField" component={renderField}/>

This is mentioned in RC4 changelog (https://github.com/erikras/redux-form/releases/tag/v6.0.0-rc.4). Back in RC2, your custom props was mixed into "input" prop. Now, they are injected as a "top level prop"
It should probably look somewhat as this (note that added prop in arguments)
const renderInputField = ({ input, label, type, meta: { touched, error }, disabled }) => (

        {label}

            <input {...input} disabled={disabled} className="form-control" type={type} />
            {touched && error && {error}}

);

if you want to pass all custom props as before, i would use spread operator
const renderInputField = ({ input, label, type, meta: { touched, error }, ...other }) => (

        {label}

            <input {...other} {...input} className="form-control" type={type} />
            {touched && error && {error}}

);

Ty @ArcanisCz . That worked beautifully!!


Analyzer crashes on specter's caching implementation (analyzing forms that create new defs)

I created a full reproducer here: https://github.com/latacora/clerk-analyzer-specter-bug
The bug seems to be caused because Specter has a macro expansion that itself new vars (for local cache purposes).
For search purposes, the error looks like this:
Unhandled clojure.lang.ExceptionInfo
Hash is missing on dependency 'whatever/pathcache1234' of the form '(defn f [x] (sr/select [...])) 'in foo.bar.baz

I wonder if the analyzer could just compute the hash when this happens?
As a workaround, use select*, which is the version with no fancy caching.

FYI: https://github.com/redplanetlabs/specter/issues/194
I don't think this works for clerk's use case: clerk analyzes tons of stuff marked :private, but at least it means specter recognizes it might do things to cooperate with tooling that isn't expecting it to do any of that.

(Also for me, the select call is in an external library, and I don't think specter allows controlling cache behavior globally, so "replace with the starred variant" isn't really much of an option.

@lvh thanks for the detailed repro. I think we should at least support turning off the error via {:nextjournal.clerk/error-on-missing-vars :off}. Clerk's error display in this case is also far from good.

Yep, that makes sense. Do we know of any cases where erroring hard is the correct behavior? I feel like warnings are better, because IIUC the worst case when you hit this is that Clerk doesn't re-render correctly?
Since this is dynamically generated I'm not even sure you could compute this hash if you wanted to. That's kind of the point of this error: it's telling the truth, the analyzer can't get to the source code.

Clerk's error display in this case is also far from good.

I think there's a regression due to https://github.com/nextjournal/clerk/pull/629 which prevents to reach the intended exception with a more understandable message:
https://github.com/nextjournal/clerk/blob/3acd9cffc533da1f1f67f6f8fb0158d320767fbc/src/nextjournal/clerk/analyzer.clj#L334-L335
the hash missing check should not throw, maybe just warn
https://github.com/nextjournal/clerk/blob/3acd9cffc533da1f1f67f6f8fb0158d320767fbc/src/nextjournal/clerk/analyzer.clj#L631-L632

I also bumped into this error when debugging stuff in clerk itself, and changing in nextjournal.clerk.view:
(defn doc->viewer
  ([doc] (doc->viewer {} doc))
  ([opts {:as doc :keys [ns file]}]
   (def d doc) ;; NOTE THIS LINE
   (binding [*ns* ns]
     (-> (merge doc opts) v/notebook v/present))))

I can't repro it in another namespace though

@borkdude thanks for the repro. This helps understand some of the edge cases which can trigger the exception. In this case
Hash is missing on dependency 'nextjournal.clerk.cljs-libs/prepend-required-cljs' of the form '(defn doc->viewer ([doc] (doc->viewer {} doc)) ([opts {:as doc, :keys [ns file]}] (def d :doo) (binding [*ns* ns] (-> (merge doc opts) v/notebook v/present cljs-libs/prepend-required-cljs))))' in ahoi (id: nextjournal.clerk.view/anon-expr-5dsZdRsdGo2tMPyWFe9fq2nNXNVq85, key: nextjournal.clerk.view/doc->viewer)

In the specific case the form (defn doc->viewer ...  is stored in the analyzer graph with multiple different keys (since it defines more than one var) not all of which are related with edges to the shared dependency nextjournal.clerk.cljs-libs/prepend-required-cljs and hence can occur in random reciprocal order in the sorted keys.
Here only nextjournal.clerk.view/anon-expr-5dsZdRsdGo2tMPyWFe9fq2nNXNVq85 (the id of the code block) depends on nextjournal.clerk.cljs-libs/prepend-required-cljs but not nextjournal.clerk.view/doc->viewer. This is something we might want to fix.


Allow custom client bigpipe.js scripts

We currently force our bigpipe.js upon our users. People might not want to follow our idea's on sandboxing, client-side execution of code and html updating. By refactoring this in to a separate component we can have frameworks like emberjs, react, angular and buzzword take care of this logic. Making this whole thing framework independend which would be freaking awesome.
Area's that it needs to touch:

bigpipe compiler: It needs to be able to read/include a custom library instead of bigpipe.js
bootstrap: It needs to be hooked by the bootstrap module so it can generate it's own {bootstrap} logic.
pagelet: It needs to provide it's own pagelet fragment.

The wrapper, framework, insert name suggestion here should be easily extendable like all our other parts are. Example codes:
var BigPipe = require('bigpipe');

var bigpipe.createServer(function () {
   engine: require('bigpipe-engine').exend({
      bootstrap: function () {}, // return string
      fragment: function () { }, // return string with {bigpipe::tags}
      library: require.resolve('what-ever.js')
   })
});

//
// Or programatically:
//
bigpipe.engine(require('bigpipe-engine').extend({
  // same props as above.
}));

BOOM.

The bigpipe/fittings project that makes this possible is merged in master. BigPipe.js is the first project wrapped using fittings and a bigpipe/reacts react plugin is on it's way.


More flutter analyze regexp cleanup

Runs clean on the repo.  Moving ahead, we can manage exceptions via .analysis_options or specific source-level ignores.
cc @Hixie

lgtm


How to pull images for arm64 machines

The tags returned by the aws-ecr buildspec seems to always pull just x86_64 tags. How do we pull images for Arm64 machines (Gravton)

I am trying to find ARM64 for container image as well. I found a different repo under https://github.com/CrowdStrike
Falcon Container Sensor Pull Script utility is useful for this purpose.
As of Aug 27th 2024, falcon-sensor images can be found with this utility but not the falcon-container images.


[BEAM-3388] Type-specialize custom decoders and encoders in Go SDK runtime

Removes the reflective overhead for custom encoders and decoders of certain types. This is a "poor man's templates" approach.
Once #4289 is in, I'll change the element coding to not reinspect the coder structure for each element. That will both speed things up and avoid memory reallocations.

R: @wcn3


Add netifaces and cryptography to Windows pip installs.

Signed-off-by: Chris Lalancette<EMAIL_ADDRESS>

Out of curiosity, have you tried debug on Windows, and do packages depending on netifaces work?
E.g. https://github.com/ros2/ros2_documentation/blob/1848561a3db79a51cf4f4f24b7324cf69b854863/source/Installation/Eloquent/Windows-Development-Setup.rst#extra-stuff-for-debug-mode
Also, I'm kind of surprised that cryptography wasn't already included here, it's also required in dashing iirc.

Out of curiosity, have you tried debug on Windows, and do packages depending on netifaces work?

No, I actually can't get anything at all to work on Windows right now.  This was just the first, obvious step.


Remove chokidar file watcher

Closes #18381

Verified that this detects creation of python binary in global scenarios.  For manual testing I commented out the check in initWatchers and tested it out. Worked well in the windows store scenario, we don't have to monitor the individual PSF folder we can just look for python exes now.


Migrates All-Pairs Shortest Paths to Swift3 syntax

Success 🎉
This PR is what was intended in #346 but that one got mixed with #344 sooo...
Hope it is useful.

@pbodsk thanks for the PR, will be reviewing this one!

@pbodsk looks like everything is running, test pass. Thanks for the PR!


Home Page Redirection Issue

Issue
(#32 )
Description
To resolve the "Cannot GET /Home.html" error, the Home button's link should be updated to direct users to index.html. This can be achieved by modifying the HTML code for the Home button. Ensure that the link is set correctly:
Related Issues
None
Type of PR

[Bug] ()

Screenshots / videos (if applicable)

Checklist

[X] I have gone through the contributing guide
[X] I have updated my branch and synced it with project main branch before making this PR
[X] I have performed a self-review of my code
[X] I have tested the changes thoroughly before submitting this pull request.
[X] I have provided relevant issue numbers, screenshots, and videos after making the changes.
[X] I have commented my code, particularly in hard-to-understand areas.

@Anjaliavv51 it is updated but can you guide if there is any problem as i am new in open-source.


updates to lower file system

I am using /var as overlay "dir". I am using RPi4 4GB with raspbian-lite as a router and few programs write to /var/lib as as well as /var/log. I see writes to SD card every few seconds (probably minutes) and "find" command indicates that the the lower file system files in "/opt/zram/var.bind" are getting updated. How often is this done and can I control this?
My entry in /etc/ztab:
dir    lz4     500M             1500M            /var               /var.bind

I ended up mounting a bunch of directories in /var with overlay. Now, I don't see any writes to my SD card while Pi4 is doing firewall and routing duties. Thank you to these scripts. Would be nice to specify a set of directories with zdir though. The ones I did are here:
/home
/root
/var/lib/systemd/timesync
/var/lib/dhcp
/var/lib/vnstat
/var/spool/squid
/var/lib/unifi
/var/lib/mongodb
/var/log


[Bug]: 2.3.5 onOpen和onClose执行次数不一致

Actions I've taken before I'm here

[X] I've thoroughly read the documentations on this issue but still have no clue.
[X] I've searched the Github Issues but didn't find any duplicate issues that have been resolved.
[X] I've searched the internet for this issue but didn't find anything helpful.

What happened?
帅哥，你的2.3.5有bug吖，帮忙解决一下，实现流程：1启动一个服务器，在onopen 计数+1，onclose 计算-1；2写一个for循环的客户端，开启300个客户端进行连接，每个客户端间隔10毫秒发送一次消息，等会一会客户端因为数据量超出了限制，会因为底层自动关闭，然后将客户端程序关闭，最后服务器剩余的在线客户端计数0；

服务器计数代码

服务器的调试代码

clientActive和clientNum是一个，我一开始以为是go的bug，自己用锁封了一次线程安全的计数

服务器刚启动调试信息

一直强发包和收包，电脑因为某些原因开始断开连接后，我关闭客户端程序

理论上，我关闭了客户端程序，服务器的客户端计算应该是0才对的，现在有残余，说明有些客户端的关闭没调用OnClose

Major version of gnet
v2
Specific version of gnet
v2.3.5
Operating system
Linux
OS version
21
Go version
go1.21.3
Relevant log output
OnClose  260
error occurred on closed, read: connection reset by peer
OnClose  216
error occurred on closed, read: connection reset by peer
debug 350.135 8 8 1714684
debug 355.135 8 8 1714684
debug 360.135 8 8 1714684
debug 365.136 8 8 1714684
debug 370.136 8 8 1714684
debug 375.137 8 8 1714684
debug 380.138 8 8 1714684
debug 385.138 8 8 1714684
debug 390.139 8 8 1714684
debug 395.14 8 8 1714684
debug 400.14 8 8 1714684
debug 405.141 8 8 1714684
debug 410.142 8 8 1714684
debug 415.142 8 8 1714684
debug 420.142 8 8 1714684
debug 425.143 8 8 1714684
debug 430.143 8 8 1714684
debug 435.144 8 8 1714684
debug 440.145 8 8 1714684
debug 445.145 8 8 1714684
debug 450.146 8 8 1714684
debug 455.147 8 8 1714684
debug 460.147 8 8 1714684
debug 465.148 8 8 1714684
debug 470.148 8 8 1714684
debug 475.148 8 8 1714684
debug 480.149 8 8 1714684
debug 485.149 8 8 1714684
debug 490.15 8 8 1714684
debug 495.151 8 8 1714684
debug 500.151 8 8 1714684
debug 505.152 8 8 1714684
debug 510.153 8 8 1714684
debug 515.153 8 8 1714684

Code snippets (optional)
No response
How to Reproduce

写个简单的服务器，在OnOpen进行客户端计数+1，OnClose进行客户端计数-1
编译，放到腾讯云服务器运行 4核8g
写个循环，创建300个客户端，间隔10毫秒频繁发包，
编译，在window电脑执行
服务器有日志出现客户端因为某些原因断开后，关闭window的客户端程序

Does this issue reproduce with the latest release?

另外，提供一下完整的复现代码(客户端和服务端)，我本地调试下看看。

还有，你能不能试一下 v2.3.4 有没有这个问题？

你调用一下 Engine.CountConnections，看看返回值是不是 0？
另外，提供一下完整的复现代码(客户端和服务端)，我本地调试下看看。

我试了一下，Engine.CountConnections 也不是0，我测一下 2.3.4 在将测试代表发你

还有，你能不能试一下 v2.3.4 有没有这个问题？
serverTest.zip

2.3.4 也存在一样的问题，

等一下，你的客户端是异常关闭的吗？关闭前有没有正常关掉所有连接，如果客户端的连接在进程异常退出之前没有发送数据的话，那很有可能在异常关闭之后产生一些死连接，这样的话服务端的 epoll/kqueue 是无法立刻感知到的，可能要过很长时间才会收到错误提示。
我建议你通过 WithTCPKeepAlive 或者 SetKeepAlivePeriod 设置 TCP keep-alive，然后等待相应的时间过后再检查一下。

过很长时间

服务器代码加上 WithTCPKeepAlive 也不可以，我更新了一下代码，之前代码的环境不干净，你试试现在的代码

testGnet.zip

[root@VM-0-7-centos test]# uname -a
Linux VM-0-7-centos 3.10.0-1160.88.1.el7.x86_64 #1 SMP Tue Mar 7 15:41:52 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
[root@VM-0-7-centos test]# uname -r
3.10.0-1160.88.1.el7.x86_64
这是云服务器的内核信息

我等会儿测试一下看看。

我等会儿测试一下看看。

能复现吗？

我在 Linux 和 Mac 上都没复现，程序正常工作：

[root@VM-0-7-centos test]# uname -a
Linux VM-0-7-centos 3.10.0-1160.88.1.el7.x86_64 https://github.com/panjf2000/gnet/issues/1 SMP Tue Mar 7 15:41:52 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
[root@VM-0-7-centos test]# uname -r
3.10.0-1160.88.1.el7.x86_64

我看你的 Linux 内核版本是 3.10，有点老了，不知道和这个有没有关系，你看看能不能找个安装了较新内核的机器再试试。

我在 Linux 和 Mac 上都没复现，程序正常工作：

[root@VM-0-7-centos test]# uname -a
Linux VM-0-7-centos 3.10.0-1160.88.1.el7.x86_64 #1 SMP Tue Mar 7 15:41:52 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
[root@VM-0-7-centos test]# uname -r
3.10.0-1160.88.1.el7.x86_64

我看你的 Linux 内核版本是 3.10，有点老了，不知道和这个有没有关系，你看看能不能找个安装了较新内核的机器再试试。

我过2天买个新的云服务器测试一下

我在 Linux 和 Mac 上都没复现，程序正常工作：

[root@VM-0-7-centos test]# uname -a
Linux VM-0-7-centos 3.10.0-1160.88.1.el7.x86_64 #1 SMP Tue Mar 7 15:41:52 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
[root@VM-0-7-centos test]# uname -r
3.10.0-1160.88.1.el7.x86_64

我看你的 Linux 内核版本是 3.10，有点老了，不知道和这个有没有关系，你看看能不能找个安装了较新内核的机器再试试。

我过2天买个新的云服务器测试一下

你

我在 Linux 和 Mac 上都没复现，程序正常工作：

[root@VM-0-7-centos test]# uname -a
Linux VM-0-7-centos 3.10.0-1160.88.1.el7.x86_64 #1 SMP Tue Mar 7 15:41:52 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
[root@VM-0-7-centos test]# uname -r
3.10.0-1160.88.1.el7.x86_64

我看你的 Linux 内核版本是 3.10，有点老了，不知道和这个有没有关系，你看看能不能找个安装了较新内核的机器再试试。

我过2天买个新的云服务器测试一下

这有个问题，我的云服务器是去年腾讯买的，所以应该是比较主流的。’我今天抽空先给你录个操作视频，你看看操作是不是一样

能给个别的联系方式吗，我这视频上传不了

我在 Linux 和 Mac 上都没复现，程序正常工作：

[root@VM-0-7-centos test]# uname -a
Linux VM-0-7-centos 3.10.0-1160.88.1.el7.x86_64 #1 SMP Tue Mar 7 15:41:52 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
[root@VM-0-7-centos test]# uname -r
3.10.0-1160.88.1.el7.x86_64

我看你的 Linux 内核版本是 3.10，有点老了，不知道和这个有没有关系，你看看能不能找个安装了较新内核的机器再试试。

我过2天买个新的云服务器测试一下

这有个问题，我的云服务器是去年腾讯买的，所以应该是比较主流的。’我今天抽空先给你录个操作视频，你看看操作是不是一样

能给个别的联系方式吗，我这视频上传不了

链接：https://pan.baidu.com/s/1B2gOAawyOiu-qfy3Lgf2kA
提取码：mid5
--来自百度网盘超级会员V5的分享 你去下载一下视频吧，


chore: upgrade charts

updated chart jx3/jx-build-controller from 0.2.1 to 0.2.2

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To complete the pull request process, please assign macox
You can assign the PR to them by writing /assign @macox in a comment when ready.
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@jenkins-x-bot-test: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

vault
2b684126bddc725796203a7f61aa040239a74de3
link
/test vault

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the jenkins-x/lighthouse repository. I understand the commands that are listed here.

@jenkins-x-bot-test: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

vault
2b684126bddc725796203a7f61aa040239a74de3
link
/test vault

gsm-kapp
2b684126bddc725796203a7f61aa040239a74de3
link
/test gsm-kapp

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the jenkins-x/lighthouse repository. I understand the commands that are listed here.

/retest


Smooth PV data

5 minutely PV power is very noisy.
Maybe take a fifteen minutely rolling mean instead.
Maybe only smooth the training target. Don't smooth the PV data when it's used as an input to the model? (Because those rapid ramps are probably caused by small clouds, which we kind of want our model to know about!)
Maybe test all permutations (no smoothing; smooth just the target; smooth all PV data; only smooth PV data when it's used as an input).

done when issue #27 was closed


Added Hover functonality on Blogs page, changed the UI of Navbar and inetgrated Firebase for Sign Up using, Github and Google.

Description of Changes:
Enhanced Blogs Page with Hover Functionality:
1 - Added interactive hover effects to improve the user experience on the Blogs page, making it more engaging and visually appealing.
Navbar UI Overhaul:
2 - Redesigned the Navbar to modernize its appearance, improve navigation, and provide a cleaner, more intuitive user interface.
Improved UI for Tools Section:
3 - Refined the design and layout of the tools section to create a more polished and user-friendly experience.
Integrated Firebase for Signup with Google and GitHub.
4 - Implemented Firebase authentication for the Signup process, allowing users to register using Google or GitHub accounts.
Note: Due to the codebase using plain JavaScript (and not React), the firebase-Config details had to be hardcoded in the JavaScript files. Since environment variables (.env) cannot be accessed via process.env in this setup, this approach was necessary to ensure Firebase integration works seamlessly.
Screenshots -

@vishanurag please approve this PR

@vishanurag accept this PR.


Removed check for Python 3.4 compatibility as it breaks in Python 3.7

async is a reserved word in Python 3.7, causing pyEmby fail with this
error:
File "/usr/lib/python3.7/site-packages/pyemby/init.py", line 10,
in
from .server import EmbyServer
File "/usr/lib/python3.7/site-packages/pyemby/server.py", line 30
ensure_future = asyncio.async

Thanks for letting me know.  If we remove the check i'd rather update the async syntax and make 3.5 the minimum version.  I should have time to get to it this week.

This code has been removed and everything updated to 3.6 syntax in the latest version (1.6).


storrisi_181213_165809.405

Type: Missing
Summary:
dgfg
Details:
dfg
Resolution:
sdfg
Affected definitions:

redie 0.8.0

You can review the change introduced to the full definition at ClearlyDefined.


Cannot test @Output() of a mocked component if @Output() is an Observable

You can successfully test whether your component reacts to @Output events of a mocked component, when the @Output is an Event Emitter. But it doesn't work when the @Output of the mocked component is an Observable - it does not trigger an event on the parent anymore.
This works in production as @Outputs are Observables under the hood in Angular.
Consider the following code:
<mocked-component (someOutput)="triggerSomething($event)"></mocked-component>
This following works fine:
export class MockedComponent {
 @Output() onSomeChange = new EventEmitter();
}

This following does not work fine:
export class MockedComponent {
 @Output() onSomeChange = someObservable$;
}

Hi @naeemb123, could you provide an example of a failing test?

Here's an example of a test that uses both the EventEmitter approach - which works, and the Observable approach - which doesn't work
https://stackblitz.com/edit/github-57pjuh-s4h7ws?file=src/examples/MockComponent/test.spec.ts

In your example, you redefine the output observable after rending, whereas MyComponent is still listening to the original one.
It won't work in any scenario, even with pure TestBed.
I've updated your code to have the same behavior for both outputs, and, as expected, both of them are failing now: https://stackblitz.com/edit/github-57pjuh-vs5jvj?file=src%2Fexamples%2FMockComponent%2Ftest.spec.ts
To fix this, you need to substitute testOutput$ before rendering. For example, you can do it with help of MockInstance:
https://stackblitz.com/edit/github-57pjuh-9c7mdj?file=src%2Fexamples%2FMockComponent%2Ftest.spec.ts
If it's been your problem, then the fix is to use MockInstance. Otherwise, please adjust the example to demonstrate the problem correctly.

Hi @naeemb123,
I hope this finds you well.
I wanted to ask if you've had a chance to take a look at my comment above?

Hi @naeemb123, any update on this?

Hi @naeemb123, I'm going to close the issue. Feel free to update and reopen it when you have more info.


Backend SC caused DB pod restarts. It never came to running state

Environment info

NooBaa Version: master-20210627
Platform: OCP 4.6.16

Actual behavior

Upgrade to master-20210627 caused db pod crash

Expected behavior
1.DB pod shouldn't crash
Steps to reproduce

Old code - master-20210622
Upgraded to master-20210627 in order to retain accounts and buckets

[root@ocp-akshat-1-inf akshat]# oc logs noobaa-db-pg-0     |more
pg_ctl: another server might be running; trying to start server anyway
waiting for server to start....2021-06-28 05:27:42.521 UTC [25] PANIC:  could not read file "global/pg_control": Input/output error
 stopped waiting
pg_ctl: could not start server
Examine the log output.
[root@ocp-akshat-1-inf akshat]# oc logs noobaa-db-pg-0     |less -R
[root@ocp-akshat-1-inf akshat]# podn
NAME                                               READY   STATUS        RESTARTS   AGE
noobaa-core-0                                      1/1     Running       0          22m
noobaa-db-pg-0                                     0/1     Error         2          22m
noobaa-default-backing-store-noobaa-pod-b0a5d78b   0/1     Terminating   0          4d23h
noobaa-endpoint-6886745f66-rdd4m                   1/1     Running       0          22m
noobaa-operator-57d449689c-zb56f                   1/1     Running       0          22m
[root@ocp-akshat-1-inf akshat]# oc logs noobaa-db-pg-0   -p  |less -R
[root@ocp-akshat-1-inf akshat]# podn
NAME                                               READY   STATUS             RESTARTS   AGE
noobaa-core-0                                      1/1     Running            0          22m
noobaa-db-pg-0                                     0/1     CrashLoopBackOff   2          22m
noobaa-default-backing-store-noobaa-pod-b0a5d78b   0/1     Terminating        0          4d23h
noobaa-endpoint-6886745f66-rdd4m                   1/1     Running            0          23m
noobaa-operator-57d449689c-zb56f                   1/1     Running            0          23m

QoS Class:       Guaranteed
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason          Age                    From               Message
  ----     ------          ----                   ----               -------
  Normal   Scheduled       21m                    default-scheduler  Successfully assigned noobaa/noobaa-db-pg-0 to worker2.ocp-akshat-1.cp.fyre.ibm.com
  Normal   AddedInterface  21m                    multus             Add eth0 [<IP_ADDRESS>/22]
  Normal   Pulling         21m                    kubelet            Pulling image "noobaa/noobaa-core:master-20210627"
  Normal   Pulled          20m                    kubelet            Successfully pulled image "noobaa/noobaa-core:master-20210627" in 30.126640313s
  Normal   Created         20m                    kubelet            Created container init
  Normal   Started         20m                    kubelet            Started container init
  Warning  Failed          2m34s (x4 over 3m14s)  kubelet            Error: failed to resolve symlink "/var/lib/kubelet/pods/ce44b338-0155-430c-97d7-5408c230e0b4/volumes/kubernetes.io~csi/pvc-d1c22d45-5f3b-4684-8f4c-48880815f451/mount": lstat /var/mnt/fs1: stale NFS file handle
  Normal   Pulled          104s (x6 over 20m)     kubelet            Container image "centos/postgresql-12-centos7" already present on machine
  Normal   Created         103s (x2 over 20m)     kubelet            Created container db
  Normal   Started         103s (x2 over 20m)     kubelet            Started container db
  Warning  BackOff         11s (x11 over 2m21s)   kubelet            Back-off restarting failed container

More information - Screenshots / Logs / Other output

noobaa_diagnostics_1624858203.tar.gz

HPO team shares a common Storage Cluster , for some test case I rebooted SC during IO.
Everyone who was using that SC got db pod crashed and on describe db pod, get
Events:
Type     Reason   Age                    From     Message

Warning  Failed   38m (x4 over 39m)      kubelet  Error: failed to resolve symlink "/var/lib/kubelet/pods/a2d8f1e5-c5c7-4476-8503-b213e04110f7/volumes/kubernetes.io~csi/pvc-5c1851e4-9dcb-4b0e-ba43-b08a0ddc3105/mount": lstat /var/mnt/fs1: stale NFS file handle
Normal   Pulled   38m (x22 over 41h)     kubelet  Container image "centos/postgresql-12-centos7" already present on machine
Normal   Created  38m (x3 over 41h)      kubelet  Created container db
Normal   Started  38m (x3 over 41h)      kubelet  Started container db
Warning  BackOff  4m28s (x180 over 37h)  kubelet  Back-off restarting failed container
<EMAIL_ADDRESS>~]# noobaa version
INFO[0000] CLI version: 5.9.0
INFO[0000] noobaa-image: noobaa/noobaa-core:master-20210623
INFO[0000] operator-image: noobaa/noobaa-operator:5.9.0
<EMAIL_ADDRESS>~]#
Logs:
noobaa_diagnostics_1624858898.tar.gz

Before rebooting everything was working fine :
Uploading works fine before rebooting
[root@fyreauto-x-app1 ~]# s3 cp u2_file_user2 s3://kri
urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host 's3-noobaa.apps.ns.cp.fyre.ibm.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
upload: ./u2_file_user2 to s3://kri/u2_file_user2
IO failed with Internal Server error on rebooting SC while doing IO
[root@fyreauto-x-app1 ~]#  s3 cp u2_file_user2 s3://kri
urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host 's3-noobaa.apps.ns.cp.fyre.ibm.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host 's3-noobaa.apps.ns.cp.fyre.ibm.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host 's3-noobaa.apps.ns.cp.fyre.ibm.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
upload failed: ./u2_file_user2 to s3://kri/u2_file_user2 An error occurred (InternalError) when calling the PutObject operation (reached max retries: 2): We encountered an internal error. Please try again.
[root@fyreauto-x-app1 ~]#

Please refer to update from @nehasharma5 here only. Time overlapped for my run and Neha's tryout, hence my updates should be ignored.

@nehasharma5 seems like, at least from the first comment you added, an issue with the PVC of the DB.
Which StorageClass and what is backing the DB PVC?

@nimrod-becker Here is the list of pvc
NAME                                               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                     AGE
db-noobaa-db-pg-0                                  Bound    pvc-0f3ae11a-971c-4480-9398-d3f37fb145a8   50Gi       RWO            ibm-spectrum-scale-csi-fileset   6d21h
gpfs-vol-pvc-new1                                  Bound    gpfs-pv-3                                  250Gi      RWX                                             4d20h
gpfs-vol-pvc-new11                                 Bound    gpfs-pv-31                                 250Gi      RWX                                             3d16h
noobaa-default-backing-store-noobaa-pvc-1ff51808   Bound    pvc-c164a9ac-1855-4788-8219-a2f2ab8ce831   50Gi       RWO            ibm-spectrum-scale-csi-fileset   6d21h
<EMAIL_ADDRESS>~]#
gpfs-vol-pvc-new1  is used for endpoint pod

Just to make sure. This issue is about the storage getting restarted right? Can you check somehow if the file mentioned is in the storage cluster? Or was really damaged/corrupted  by the restart?
This might be relevant<EMAIL_ADDRESS>This can be attempted if the system still alive:
https://www.postgresql.org/docs/9.5/app-pgresetxlog.html

@nehasharma5 , please attempt this again with @jackyalbo in a live session for debugging.

So after a live session with @nehasharma5 we were able to reproduce this issue. It seems that the pod can't open his configuration file:  pg_ctl: another server might be running; trying to start server anyway waiting for server to start....2021-07-20 13:00:45.370 GMT [25] LOG:  input in flex scanner failed at file "/var/lib/pgsql/data/userdata/postgresql.conf" line 1 2021-07-20 13:00:45.370 GMT [25] FATAL:  configuration file "/var/lib/pgsql/data/userdata/postgresql.conf" contains errors  stopped waiting pg_ctl: could not start server
So we tried accessing this file from inside the storage cluster: and we got the same issue:
[root@storage-x-master userdata]# cat postgresql.conf
cat: postgresql.conf: Input/output error

So I think it's safe to say that the issue is with the storage provider and doesn't really connected to noobaa (or postgres). Although I think it will be valuable to continue investigating this issue with the storage provider in order to avoid this sort of issues in the future.

@akmithal please see the latest comment, can we close ?

Hi @jackyalbo , @nehasharma5 , isn't postgresql.conf created by Noobaa itself ? In my opinion, behaviour should be checked for user created files on SC too. If we are able to read files from SC after it's restart, the files that Noobaa component is creating should work as well. So could be a bug in the way Noobaa component ( Db?) is writing this file at the point of backend restart.

@akmithal We know that in other flows and tests (with different SCs) we don't see this issue.
Also, if the file is written, it should be available for access later...

@nimrod-becker this is the actual stamp that we needed in the meeting we just had. I said that to the best of my knowledge we don't see such issues with any other storage providers.
@akmithal Does that sound like the proof you needed?

Thanks for these details.
@nimrod-becker , just for understanding it better, have we tested this in an environment with PostgreSQL DB with other SC vendors as well or was it with MongoDB? Also, which code level was it tested with.
If we know it works well with other SC vendors and PostgreSQL too, then it should be investigated why it doesn't work in our environment. We will also involve CNSA, CSI teams then. FYA @nehasharma5

PSQL, it was tested with both LSO and Ceph as the SC.
I think the level is very basic for this flow, when the pod comes up it should be able to access the files on the PV. The testing done were with restarts to the pod (both intentionally, and as part of the upgrade flows for example).

@jackyalbo I have tried multiple times to get this reproduce on noobaa version<EMAIL_ADDRESS>~]# noobaa version
INFO[0000] CLI version: 5.9.0
INFO[0000] noobaa-image: noobaa/noobaa-core:master-20210725
INFO[0000] operator-image: noobaa/noobaa-operator:5.9.0
<EMAIL_ADDRESS>~]#
But its not reproducible on above version. Db pod goes in crashed state and it is restarted once SC is back

Hi @jackyalbo , It seems, this defect is still there. I have uploaded must-gather logs in defect 6691 and if issue looks same, we can close the other one and track the bug here itself.

Hi @jackyalbo I deleted db pod after it goes in crash state and again tried to open log file , it gives to me
[root@storage-x-master log]# cat postgresql-Tue.log
2021-08-03 11:14:50.232 UTC [27] LOG:  database system was interrupted; last known up at 2021-08-03 08:00:05 UTC
2021-08-03 11:14:54.194 UTC [27] LOG:  could not read from log segment<PHONE_NUMBER>00000000000001, offset 0: Input/output error
2021-08-03 11:14:54.194 UTC [27] LOG:  invalid primary checkpoint record
2021-08-03 11:14:54.194 UTC [27] PANIC:  could not locate a valid checkpoint record
2021-08-03 11:14:56.943 UTC [25] LOG:  startup process (PID 27) was terminated by signal 6: Aborted
2021-08-03 11:14:56.943 UTC [25] LOG:  aborting startup due to startup process failure
2021-08-03 11:14:57.146 UTC [25] LOG:  database system is shut down
2021-08-03 11:17:47.213 UTC [27] LOG:  database system was interrupted; last known up at 2021-08-03 08:00:05 UTC
2021-08-03 11:17:49.358 UTC [27] LOG:  could not read from log segment<PHONE_NUMBER>00000000000001, offset 0: Input/output error
2021-08-03 11:17:49.358 UTC [27] LOG:  invalid primary checkpoint record
2021-08-03 11:17:49.358 UTC [27] PANIC:  could not locate a valid checkpoint record
2021-08-03 11:17:52.159 UTC [25] LOG:  startup process (PID 27) was terminated by signal 6: Aborted
2021-08-03 11:17:52.159 UTC [25] LOG:  aborting startup due to startup process failure
2021-08-03 11:17:52.362 UTC [25] LOG:  database system is shut down
2021-08-03 11:23:01.214 UTC [28] LOG:  database system was interrupted; last known up at 2021-08-03 08:00:05 UTC
2021-08-03 11:23:04.671 UTC [28] LOG:  could not read from log segment<PHONE_NUMBER>00000000000001, offset 0: Input/output error
2021-08-03 11:23:04.671 UTC [28] LOG:  invalid primary checkpoint record
2021-08-03 11:23:04.671 UTC [28] PANIC:  could not locate a valid checkpoint record
2021-08-03 11:23:07.564 UTC [26] LOG:  startup process (PID 28) was terminated by signal 6: Aborted
2021-08-03 11:23:07.564 UTC [26] LOG:  aborting startup due to startup process failure
2021-08-03 11:23:07.781 UTC [26] LOG:  database system is shut down
2021-08-03 11:28:19.200 UTC [28] LOG:  database system was interrupted; last known up at 2021-08-03 08:00:05 UTC
2021-08-03 11:28:20.048 UTC [28] LOG:  could not read from log segment<PHONE_NUMBER>00000000000001, offset 0: Input/output error
2021-08-03 11:28:20.048 UTC [28] LOG:  invalid primary checkpoint record
2021-08-03 11:28:20.048 UTC [28] PANIC:  could not locate a valid checkpoint record
2021-08-03 11:28:22.783 UTC [26] LOG:  startup process (PID 28) was terminated by signal 6: Aborted
2021-08-03 11:28:22.783 UTC [26] LOG:  aborting startup due to startup process failure
2021-08-03 11:28:23.047 UTC [26] LOG:  database system is shut down
2021-08-03 11:33:24.203 UTC [28] LOG:  database system was interrupted; last known up at 2021-08-03 08:00:05 UTC
2021-08-03 11:33:25.125 UTC [28] LOG:  could not read from log segment<PHONE_NUMBER>00000000000001, offset 0: Input/output error
2021-08-03 11:33:25.125 UTC [28] LOG:  invalid primary checkpoint record
2021-08-03 11:33:25.125 UTC [28] PANIC:  could not locate a valid checkpoint record
2021-08-03 11:33:27.889 UTC [26] LOG:  startup process (PID 28) was terminated by signal 6: Aborted
2021-08-03 11:33:27.889 UTC [26] LOG:  aborting startup due to startup process failure
2021-08-03 11:33:28.142 UTC [26] LOG:  database system is shut down
2021-08-03 11:38:34.214 UTC [28] LOG:  database system was interrupted; last known up at 2021-08-03 08:00:05 UTC
2021-08-03 11:38:35.312 UTC [28] LOG:  could not read from log segment<PHONE_NUMBER>00000000000001, offset 0: Input/output error
2021-08-03 11:38:35.312 UTC [28] LOG:  invalid primary checkpoint record
2021-08-03 11:38:35.312 UTC [28] PANIC:  could not locate a valid checkpoint record
2021-08-03 11:38:37.937 UTC [26] LOG:  startup process (PID 28) was terminated by signal 6: Aborted
2021-08-03 11:38:37.937 UTC [26] LOG:  aborting startup due to startup process failure
2021-08-03 11:38:38.150 UTC [26] LOG:  database system is shut down
2021-08-03 11:43:39.241 UTC [27] LOG:  database system was interrupted; last known up at 2021-08-03 08:00:05 UTC
2021-08-03 11:43:40.595 UTC [27] LOG:  could not read from log segment<PHONE_NUMBER>00000000000001, offset 0: Input/output error
2021-08-03 11:43:40.595 UTC [27] LOG:  invalid primary checkpoint record
2021-08-03 11:43:40.595 UTC [27] PANIC:  could not locate a valid checkpoint record
2021-08-03 11:43:43.543 UTC [25] LOG:  startup process (PID 27) was terminated by signal 6: Aborted
2021-08-03 11:43:43.543 UTC [25] LOG:  aborting startup due to startup process failure
2021-08-03 11:43:43.749 UTC [25] LOG:  database system is shut down
2021-08-03 11:48:44.216 UTC [27] LOG:  database system was interrupted; last known up at 2021-08-03 08:00:05 UTC
2021-08-03 11:48:45.803 UTC [27] LOG:  could not read from log segment<PHONE_NUMBER>00000000000001, offset 0: Input/output error
2021-08-03 11:48:45.803 UTC [27] LOG:  invalid primary checkpoint record
2021-08-03 11:48:45.803 UTC [27] PANIC:  could not locate a valid checkpoint record
2021-08-03 11:48:48.643 UTC [25] LOG:  startup process (PID 27) was terminated by signal 6: Aborted
2021-08-03 11:48:48.643 UTC [25] LOG:  aborting startup due to startup process failure
2021-08-03 11:48:48.845 UTC [25] LOG:  database system is shut down
2021-08-03 11:54:00.243 UTC [28] LOG:  database system was interrupted; last known up at 2021-08-03 08:00:05 UTC
2021-08-03 11:54:01.060 UTC [28] LOG:  could not read from log segment<PHONE_NUMBER>00000000000001, offset 0: Input/output error
2021-08-03 11:54:01.060 UTC [28] LOG:  invalid primary checkpoint record
2021-08-03 11:54:01.060 UTC [28] PANIC:  could not locate a valid checkpoint record
2021-08-03 11:54:03.577 UTC [26] LOG:  startup process (PID 28) was terminated by signal 6: Aborted
2021-08-03 11:54:03.577 UTC [26] LOG:  aborting startup due to startup process failure
2021-08-03 11:54:03.857 UTC [26] LOG:  database system is shut down
2021-08-03 11:59:16.347 UTC [28] LOG:  database system was interrupted; last known up at 2021-08-03 08:00:05 UTC
2021-08-03 11:59:16.586 UTC [28] LOG:  could not read from log segment<PHONE_NUMBER>00000000000001, offset 0: Input/output error
2021-08-03 11:59:16.586 UTC [28] LOG:  invalid primary checkpoint record
2021-08-03 11:59:16.586 UTC [28] PANIC:  could not locate a valid checkpoint record
2021-08-03 11:59:19.680 UTC [26] LOG:  startup process (PID 28) was terminated by signal 6: Aborted
2021-08-03 11:59:19.680 UTC [26] LOG:  aborting startup due to startup process failure
2021-08-03 11:59:19.878 UTC [26] LOG:  database system is shut down
2021-08-03 12:04:22.218 UTC [27] LOG:  database system was interrupted; last known up at 2021-08-03 08:00:05 UTC
2021-08-03 12:04:25.386 UTC [27] LOG:  could not read from log segment<PHONE_NUMBER>00000000000001, offset 0: Input/output error
2021-08-03 12:04:25.386 UTC [27] LOG:  invalid primary checkpoint record
2021-08-03 12:04:25.386 UTC [27] PANIC:  could not locate a valid checkpoint record
2021-08-03 12:04:27.969 UTC [25] LOG:  startup process (PID 27) was terminated by signal 6: Aborted
2021-08-03 12:04:27.969 UTC [25] LOG:  aborting startup due to startup process failure
2021-08-03 12:04:28.171 UTC [25] LOG:  database system is shut down
[root@storage-x-master log]#
[root@storage-x-master log]# pwd
/ibm/fs1/pvc-f860cf1f-a235-4bdd-b996-430702daec23/pvc-f860cf1f-a235-4bdd-b996-430702daec23-data/data/userdata/log
[root@storage-x-master log]#
<EMAIL_ADDRESS>~]# oc get pvc
NAME                                               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                     AGE
db-noobaa-db-pg-0                                  Bound    pvc-f860cf1f-a235-4bdd-b996-430702daec23   50Gi       RWO            ibm-spectrum-scale-csi-fileset   6h18m
gpfs-vol-pvc-new11                                 Bound    gpfs-pv-31                                 250Gi      RWX                                             6h18m
noobaa-default-backing-store-noobaa-pvc-0ecd47f0   Bound    pvc-5eb9d598-983a-4934-9a0f-7cb631444c47   50Gi       RWO            ibm-spectrum-scale-csi-fileset   6h15m
<EMAIL_ADDRESS>~]#
<EMAIL_ADDRESS>~]# oc get pod
NAME                               READY   STATUS             RESTARTS   AGE
noobaa-core-0                      1/1     Running            0          42m
noobaa-db-pg-0                     0/1     CrashLoopBackOff   13         43m
noobaa-endpoint-5656d9787-27mrc    1/1     Running            0          43m
noobaa-endpoint-5656d9787-pklsj    1/1     Running            0          43m
noobaa-operator-7bb746749d-x5dcc   1/1     Running            0          43m
<EMAIL_ADDRESS>~]# oc logs noobaa-db-pg-0
waiting for server to start....2021-08-03 11:54:00.178 UTC [26] LOG:  starting PostgreSQL 12.5 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44), 64-bit
2021-08-03 11:54:00.180 UTC [26] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2021-08-03 11:54:00.184 UTC [26] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2021-08-03 11:54:00.237 UTC [26] LOG:  redirecting log output to logging collector process
2021-08-03 11:54:00.237 UTC [26] HINT:  Future log output will appear in directory "log".
... stopped waiting
pg_ctl: could not start server
Examine the log output.
Why everytime this db file is getting corrupted and other things working when shutting down and getting SC up ?

Hey @nehasharma5 ,
It seems like the DB needs to be recovered. It look like this issue:
https://stackoverflow.com/questions/8799474/postgresql-error-panic-could-not-locate-a-valid-checkpoint-record/26158818

@jackyalbo I left my system overnight and in morning saw db pod was running
<EMAIL_ADDRESS>~]# oc get pod
NAME                                               READY   STATUS        RESTARTS   AGE
noobaa-core-0                                      1/1     Running       0          17h
noobaa-db-pg-0                                     1/1     Running       52         17h
noobaa-default-backing-store-noobaa-pod-0ecd47f0   0/1     Terminating   0          13h
noobaa-endpoint-5656d9787-27mrc                    1/1     Running       0          17h
noobaa-endpoint-5656d9787-pklsj                    1/1     Running       0          17h
noobaa-operator-7bb746749d-x5dcc                   1/1     Running       0          17h
<EMAIL_ADDRESS>~]# oc logs noobaa-db-pg-0
waiting for server to start....2021-08-03 15:17:15.160 UTC [26] LOG:  starting PostgreSQL 12.5 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44), 64-bit
2021-08-03 15:17:15.162 UTC [26] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2021-08-03 15:17:15.169 UTC [26] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2021-08-03 15:17:15.207 UTC [26] LOG:  redirecting log output to logging collector process
2021-08-03 15:17:15.207 UTC [26] HINT:  Future log output will appear in directory "log".
. done
server started
/var/run/postgresql:5432 - accepting connections
=> sourcing /usr/share/container-scripts/postgresql/start/set_passwords.sh ...
ALTER ROLE
waiting for server to shut down.... done
server stopped
Starting server...
2021-08-03 15:17:16.464 UTC [1] LOG:  starting PostgreSQL 12.5 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44), 64-bit
2021-08-03 15:17:16.465 UTC [1] LOG:  listening on IPv4 address "<IP_ADDRESS>", port 5432
2021-08-03 15:17:16.465 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2021-08-03 15:17:16.472 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2021-08-03 15:17:16.478 UTC [1] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2021-08-03 15:17:16.510 UTC [1] LOG:  redirecting log output to logging collector process
2021-08-03 15:17:16.510 UTC [1] HINT:  Future log output will appear in directory "log".
<EMAIL_ADDRESS>~]# oc describe pod noobaa-db-pg-0
Name:         noobaa-db-pg-0
Namespace:    noobaa
Priority:     0
Node:         master1.ns.cp.fyre.ibm.com/<IP_ADDRESS>
Start Time:   Tue, 03 Aug 2021 04:11:11 -0700
Labels:       app=noobaa
controller-revision-hash=noobaa-db-pg-69cc67ff65
noobaa-db=postgres
statefulset.kubernetes.io/pod-name=noobaa-db-pg-0
Annotations:  k8s.v1.cni.cncf.io/network-status:
[{
"name": "",
"interface": "eth0",
"ips": [
"<IP_ADDRESS>"
],
"default": true,
"dns": {}
}]
k8s.v1.cni.cncf.io/networks-status:
[{
"name": "",
"interface": "eth0",
"ips": [
"<IP_ADDRESS>"
],
"default": true,
"dns": {}
}]
openshift.io/scc: noobaa
Status:       Running
IP:           <IP_ADDRESS>
IPs:
IP:           <IP_ADDRESS>
Controlled By:  StatefulSet/noobaa-db-pg
Init Containers:
init:
Container ID:  cri-o://10687962ebd69ee8a3f651d087612327bc685992c6ff675e0c44d992d948ed9a
Image:         noobaa/noobaa-core:master-20210802
Image ID:      docker.io/noobaa/noobaa-core@sha256:ba09ddf7f0b4ab4b37502cafd80383627420d96d47fb96759228fcf4561bdb76
Port:
Host Port:
Command:
/noobaa_init_files/noobaa_init.sh
init_postgres
State:          Terminated
Reason:       Completed
Exit Code:    0
Started:      Tue, 03 Aug 2021 04:11:33 -0700
Finished:     Tue, 03 Aug 2021 04:11:33 -0700
Ready:          True
Restart Count:  0
Limits:
cpu:     500m
memory:  500Mi
Requests:
cpu:        500m
memory:     500Mi
Environment:
Mounts:
/var/lib/pgsql from db (rw)
/var/run/secrets/kubernetes.io/serviceaccount from noobaa-token-6bf9h (ro)
Containers:
db:
Container ID:   cri-o://f69db6a421c9288968ee9e73563839327e2fbd95bf25f379b41b48fcfe74c470
Image:          centos/postgresql-12-centos7
Image ID:       docker.io/centos/postgresql-12-centos7@sha256:b068d992d0016f518371e858a7c250730dd467584d6aebca1d3fa58fdbfc734b
Port:           5432/TCP
Host Port:      0/TCP
State:          Running
Started:      Tue, 03 Aug 2021 08:17:15 -0700
Last State:     Terminated
Reason:       Error
Exit Code:    1
Started:      Tue, 03 Aug 2021 08:12:01 -0700
Finished:     Tue, 03 Aug 2021 08:12:07 -0700
Ready:          True
Restart Count:  52
Limits:
cpu:     500m
memory:  4Gi
Requests:
cpu:     500m
memory:  4Gi
Environment:
POSTGRESQL_DATABASE:  nbcore
POSTGRESQL_USER:      <set to the key 'user' in secret 'noobaa-db'>      Optional: false
POSTGRESQL_PASSWORD:  <set to the key 'password' in secret 'noobaa-db'>  Optional: false
Mounts:
/opt/app-root/src/postgresql-cfg from noobaa-postgres-config-volume (rw)
/var/lib/pgsql from db (rw)
/var/run/secrets/kubernetes.io/serviceaccount from noobaa-token-6bf9h (ro)
Conditions:
Type              Status
Initialized       True
Ready             True
ContainersReady   True
PodScheduled      True
Volumes:
db:
Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName:  db-noobaa-db-pg-0
ReadOnly:   false
noobaa-postgres-config-volume:
Type:      ConfigMap (a volume populated by a ConfigMap)
Name:      noobaa-postgres-config
Optional:  false
noobaa-token-6bf9h:
Type:        Secret (a volume populated by a Secret)
SecretName:  noobaa-token-6bf9h
Optional:    false
QoS Class:       Guaranteed
Node-Selectors:
Tolerations:     node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s

@jackyalbo
I am able to open conf file [root@storage-x-master userdata]# ls
base              pg_commit_ts   pg_logical    pg_serial     pg_subtrans  pg_wal                postmaster.opts
current_logfiles  pg_dynshmem    pg_multixact  pg_snapshots  pg_tblspc    pg_xact
global            pg_hba.conf    pg_notify     pg_stat       pg_twophase  postgresql.auto.conf
log               pg_ident.conf  pg_replslot   pg_stat_tmp   PG_VERSION   postgresql.conf
[root@storage-x-master userdata]# pwd
/ibm/fs1/pvc-e6183f23-c7be-4f37-84a1-4caab894c73c/pvc-e6183f23-c7be-4f37-84a1-4caab894c73c-data/data/userdata
[root@storage-x-master userdata]# cat postgresql.conf
#------------------------------------------------------------------------------
CONFIG FILE INCLUDES
#------------------------------------------------------------------------------
These options allow settings to be loaded from files other than the
default postgresql.conf.  Note that these are directives, not variable
assignments, so they can usefully be given more than once.
#include_dir = '...'                    # include files ending in '.conf' from
# a directory, e.g., 'conf.d'
#include_if_exists = '...'              # include file only if it exists
#include = '...'                        # include file
#------------------------------------------------------------------------------
CUSTOMIZED OPTIONS
#------------------------------------------------------------------------------
Add settings for extensions here
Custom OpenShift configuration:
include '/var/lib/pgsql/openshift-custom-postgresql.conf'
[root@storage-x-master userdata]#
<EMAIL_ADDRESS>~]# oc logs noobaa-endpoint-549479f97f-gmjnc | grep PANIC
<EMAIL_ADDRESS>~]# noobaa version
INFO[0000] CLI version: 5.9.0
INFO[0000] noobaa-image: noobaa/noobaa-core:master-20210808
INFO[0000] operator-image: noobaa/noobaa-operator:5.9.0
<EMAIL_ADDRESS>~]#
postgresql-Mon.log

<EMAIL_ADDRESS>~]# oc get pod
NAME                               READY   STATUS             RESTARTS   AGE
noobaa-core-0                      1/1     Running            0          51m
noobaa-db-pg-0                     0/1     CrashLoopBackOff   9          51m
noobaa-endpoint-549479f97f-gmjnc   1/1     Running            1          49m
noobaa-operator-55bb4cb96d-6pdhq   1/1     Running            0          52m

@nehasharma5 and with this latest info, that the file can be opened, does the DB still crash?

@jackyalbo Yes, DB pod still crashed.

@akmithal We can close this bug as while rebooting it is noticed that nsd went down because of which noobaa db pod crashed . Once nsd was db pod again come up running and IO works fine .
Tested with 12 Aug noobaa build
[root@storage-x-master ~]# /usr/lpp/mmfs/bin/mmhealth cluster show disk -v
Component     Node                               Status            Reasons
DISK          storage-x-worker1.fyre.ibm.com     HEALTHY       -
nsd2                                           HEALTHY       -
nsd7                                           HEALTHY       -
nsd9                                           HEALTHY       -
DISK          storage-x-worker2.fyre.ibm.com     HEALTHY       -
nsd3                                           HEALTHY       -
nsd5                                           HEALTHY       -
nsd8                                           HEALTHY       -
DISK          storage-x-master.fyre.ibm.com      HEALTHY       -
nsd1                                           HEALTHY       -
nsd4                                           HEALTHY       -
nsd6                                           HEALTHY       -
[root@storage-x-master ~]#
[root@storage-x-master ~]# /usr/lpp/mmfs/bin/mmhealth cluster show disk -v
Component     Node                               Status            Reasons
DISK          storage-x-worker1.fyre.ibm.com     HEALTHY       -
nsd2                                           HEALTHY       -
nsd7                                           HEALTHY       -
nsd9                                           HEALTHY       -
DISK          storage-x-worker2.fyre.ibm.com     DEGRADED      disk_down
nsd3                                           DEGRADED      disk_down
nsd5                                           DEGRADED      disk_down
nsd8                                           DEGRADED      disk_down
DISK          storage-x-master.fyre.ibm.com      HEALTHY       -
nsd1                                           HEALTHY       -
nsd4                                           HEALTHY       -
nsd6                                           HEALTHY       -
[root@storage-x-master ~]#
usage.html#ssl-warnings
upload failed: ./test_file_90 to s3://scbuck/test_file_90 An error occurred (InternalError) when calling the UploadPart operation (reached max retries: 2): We encountered an internal error. Please try again.
[root@fyreauto-x-app1 ~]#
[root@fyreauto-x-app1 ~]#
<EMAIL_ADDRESS>~]# oc get pod
NAME                                               READY   STATUS             RESTARTS   AGE
noobaa-core-0                                      1/1     Running            0          74m
noobaa-db-pg-0                                     0/1     CrashLoopBackOff   1          74m
noobaa-default-backing-store-noobaa-pod-3286ab30   0/1     Terminating        0          23m
noobaa-endpoint-86d5f88557-j66zl                   1/1     Running            0          68m
noobaa-operator-cbb58f6bc-rdkjn                    1/1     Running            0          74m
<EMAIL_ADDRESS>~]#
[root@storage-x-master ~]# mmlsdisk  fs1 -M
Disk name     IO performed on node     Device             Availability

nsd1          localhost                /dev/vdb           up
nsd2          storage-x-worker1.fyre.ibm.com /dev/vdb           up
nsd3          -                        -                  down
nsd4          localhost                /dev/vdc           up
nsd5          -                        -                  down
nsd6          localhost                /dev/vdd           up
nsd7          storage-x-worker1.fyre.ibm.com /dev/vdd           up
nsd8          -                        -                  down
nsd9          storage-x-worker1.fyre.ibm.com /dev/vdc           up
[root@storage-x-master ~]# /usr/lpp/mmfs/bin/mmhealth cluster show disk -v
Component     Node                               Status            Reasons
DISK          storage-x-worker1.fyre.ibm.com     HEALTHY       -
nsd2                                           HEALTHY       -
nsd7                                           HEALTHY       -
nsd9                                           HEALTHY       -
DISK          storage-x-worker2.fyre.ibm.com     DEGRADED      disk_down
nsd3                                           DEGRADED      disk_down
nsd5                                           DEGRADED      disk_down
nsd8                                           DEGRADED      disk_down
DISK          storage-x-master.fyre.ibm.com      HEALTHY       -
nsd1                                           HEALTHY       -
nsd4                                           HEALTHY       -
nsd6                                           HEALTHY       -
[root@storage-x-master ~]# lsblk
NAME          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda           252:0    0  250G  0 disk
├─vda1        252:1    0    1G  0 part /boot
└─vda2        252:2    0  249G  0 part
├─rhel-root 253:0    0  233G  0 lvm  /
└─rhel-swap 253:1    0   16G  0 lvm
vdb           252:16   0  500G  0 disk
└─vdb1        252:17   0  500G  0 part
vdc           252:32   0  500G  0 disk
└─vdc1        252:33   0  500G  0 part
vdd           252:48   0  500G  0 disk
└─vdd1        252:49   0  500G  0 part
[root@storage-x-master ~]# mmchdisk fs1 start -a -N storage-x-worker2.fyre.ibm.com
mmnsddiscover:  Attempting to rediscover the disks.  This may take a while ...
mmnsddiscover:  Finished.
storage-x-master.fyre.ibm.com:  Rediscovered nsd server access to nsd1.
storage-x-master.fyre.ibm.com:  Rediscovered nsd server access to nsd4.
storage-x-worker2.fyre.ibm.com:  Rediscovered nsd server access to nsd3.
storage-x-worker2.fyre.ibm.com:  Rediscovered nsd server access to nsd5.
storage-x-worker1.fyre.ibm.com:  Rediscovered nsd server access to nsd2.
storage-x-worker1.fyre.ibm.com:  Rediscovered nsd server access to nsd7.
storage-x-master.fyre.ibm.com:  Rediscovered nsd server access to nsd6.
storage-x-worker2.fyre.ibm.com:  Rediscovered nsd server access to nsd8.
storage-x-worker1.fyre.ibm.com:  Rediscovered nsd server access to nsd9.
Scanning file system metadata, phase 1 ...
100 % complete on Fri Aug 13 01:00:31 2021
Scan completed successfully.
Scanning file system metadata, phase 2 ...
100 % complete on Fri Aug 13 01:00:31 2021
Scan completed successfully.
Scanning file system metadata, phase 3 ...
Scan completed successfully.
Scanning file system metadata, phase 4 ...
100 % complete on Fri Aug 13 01:00:31 2021
Scan completed successfully.
Scanning file system metadata, phase 5 ...
100 % complete on Fri Aug 13 01:00:31 2021
Scan completed successfully.
Scanning user file metadata ...
100.00 % complete on Fri Aug 13 01:00:45 2021  (   4194304 inodes with total      68397 MB data processed)
Scan completed successfully
<EMAIL_ADDRESS>~]# oc get pod
NAME                                               READY   STATUS        RESTARTS   AGE
noobaa-core-0                                      1/1     Running       0          78m
noobaa-db-pg-0                                     1/1     Running       2          78m
noobaa-default-backing-store-noobaa-pod-3286ab30   0/1     Terminating   0          27m
noobaa-endpoint-86d5f88557-j66zl                   1/1     Running       0          72m
noobaa-operator-cbb58f6bc-rdkjn                    1/1     Running       0          78m
<EMAIL_ADDRESS>~]#

We can close this bug based on above finding.
Summary:
Noobaa DB pod requires backend NSD's to be up. So far due to Fyre environment, all NSD's were not coming up hence Noobaa db pod stayed in Crashed loop. When manually the NSD's were brought up, the Noobaa db pod came in Running state and IO could continue.


Add some more stuff to the rust api

🦀

Note: I'd like to rebase & squash some of the changes before they are merged, keeping them as individual commits for easier review for now.

I split out the interaction-related commit, which was pretty self-contained into #2948 and squashed this into fewer commits.

Rebased and fixed conflicts

Split out the bindgen bump to #2950

I manually merged this in 6cd49edb317112acc73f7caeb285f4134a472d72.  Made some minor tweaks in 2845ba6208ce3c29998a48df5073ed15a11ead77.  Thank you you for improving the Rust bindings!


Add docstring for modified MSA in BridgeFormer

Signed-off-by: Jael Gu<EMAIL_ADDRESS>

/lgtm
/approve


Undocumented request.abort()

There seems to be a function, request.abort() which is not documented anywhere?

does it really functional??


Metropolis-Hastings jump size

I added settings to change the scaling factor on the proposal covariance matrix (called mh_cc and mh_cc0).
Defaults are set at the values in the original code. However, I overrode them for AnSchorfheide so that now it gives rejection rates around 67%.

Looks good - thanks for doing this! (I think the Travis build will fail because of an NLopt build problem that's been resolved in a release we can't use yet, but it should be fine to merge in regardless.)


DDLS jump added

#724
At my current knowledge it's not possible to jump to ADT from SAPGui, so I think a good alternative is to jump to the according DDIC DDL SQL View of the DDLS

thanks, looks good, however I think we will try the solution in https://github.com/larshp/abapGit/pull/728 instead


tests: TestCodegenConfigurations projects should fail on warnings

Fixes https://github.com/apollographql/apollo-ios/issues/3301.
I don't know when this stopped working in our test configuration projects but this behaviour is essential to avoid exposing inconvenience warnings to users projects.
I can't find any mention of why the unsafeFlags configuration no longer works but I do prefer that the flag is now set on the test script command which is more explicit to the workflow we want for the test - reference.
I haven't managed to get the behaviour back for Xcode-based projects yet though.

As expected the SPM-based test codegen configurations are now failing with the warnings being treated as errors. The next commit will remove the warning and they should pass CI.

..and the test codegen configurations job is now passing, as expected.


SW-4361 Add internal tag for accelerator participants

We're going to be adding system behavior that only applies to participants in
the Terraformation Accelerator. Add an internal tag so super-admins can indicate
which organizations are in the accelerator.
Nothing uses this new tag yet, but it will appear in the admin UI.

Current dependencies on/for this PR:

main

PR #1429   👈

This comment was auto-generated by Graphite.


Duplicate identifier 'Node', how to let Typescript know my file is a module

I'm building a module and I want to name a class 'Node'. Looks like 'Node' already exists in the global scope. A module file has an isolated scope, so it will not replace the global 'Node' class. How can I tell typescript that a file is a module so it does not complain about duplicates?
models.ts:
class Node {}
If I add export to the Node class, the warning is gone.
tsc --version
Version 1.8.9

I just realized Typescript needs a single export to know that a file is a module. My bad.

(at least one export or import)


chore(aptos): CW-9607 publish package to npm with signTransaction function

Task linked: CW-9607 [SDK] As a user, I can send and receive test coin of Aptos


Native Federation: How to use Angular Material globally and in each mfe ?

Hello,
I have this starter project built with Angular 17 and Native Federation.
https://github.com/alaindeurveilher/ng-mfe-app-starter/tree/starter-routing
I would like to know how to setup each application to use Angular Material, so that each MFE when being served in isolation (in development mode) would be using Angular Material as well as when being loaded in the shell.
And potentially one custom theme to be used by each of them. Let's say it would be the company/corporate theme and design.
I did not find much documentation around here about how to properly configure Angular Material, and sharing theming in general with micro frontends built up with Native Federation.
Thank you.

At the moment I ended up installing @angular/material globally without specifying any project, and then setting up material in each of the micro frontends manually, as it would be expected after the installation by the ng add command:
https://github.com/alaindeurveilher/ng-mfe-app-starter/tree/with-material?tab=readme-ov-file#adding-angular-material
But I wonder if there would be an automatic sort of way to do it.
Thank you.

Yes, this is the way to go. Install it everywhere and Federation make sure at runtime that it's only loaded once.


Dynamically generate/select JSON schema

Need to land this:
https://github.com/pantsbuild/pants/pull/18177


questions about mandarin Chinese-Cantonese zipformer

Hi,
I trained a zipformer model with 80kh mandarin Chinese and 7kh Cantonese. The model performs very poor on Cantonese test set (cer=80%+), and it performs good on madarin Chinese test sets. The loss curve consistently oscillates, and mandarin Chinese is used as dev set. It seems the model learns nothing about Cantonese.

I didn't shuffle the train cuts between mandarin Chinese and Cantonese. Is it because of this reason resulting in the bad performance on Cantonese? Are there any other issues that I need to pay attention to?
Thanks!

Your dataset is quite unbalanced, I recommend you have both Mandarin and Cantonese data in each batch if possible.
Also, it would be nice to have two dev sets so that you can know how well the model is performing on Cantonese during training.

Thanks! I'll try.

@marcoyang1998
can copy or augment Cantonese?
How can change the goal function for unbalanced data?

@lalimili6
You can have a look at this: https://lhotse.readthedocs.io/en/latest/api.html#lhotse.supervision.SupervisionSet.mux

shuffling cuts solved this issue


Don't put the number on operationId for the first occurence

On v2.1.0, the operationId is now of type methodUsingGET_x to guarantee uniqueness.
But for the first occurence it would be nicer to have methodUsingGET instead of methodUsingGET_1

It doesn't generate the a number on the first occurrence. It only generates it from the second one.

Oh, I see. Then the issue is actually that the occurrence counter seems not to be reset when using several dockets.
For the same method() unique amongst controllers
OperationId in first docket : methodUsingPOST
OperationId in second docket : methodUsingPOST_1


panic: format.Node internal error (951:11: expected '==', found '=')

This happens because an expression contains an assignment, like:
if (a = 1)

This issue will fix it: #132

needs #132

Found new same code with same error:
int fact(int n) {
  int lcv, p;
  for(p=1, lcv=2; lcv <= n; p=p*lcv, lcv++);
  return p;
}

Error:
panic: format.Node internal error (11:19: expected '==', found '=' (and 4 more errors))

Example of code(https://github.com/Konstantin8105/c2go-rating/blob/master/SingleCcode/17-2.c) with same error:
#include <stdio.h>

main ()

{ char ch;
   short count = 0;

printf ("Type in a line of text\n");

while ((ch = getchar()) != '\n')
   {
   if (ch == ' ')
      {
      count++;
      }
   }

printf ("Number of space = %d\n",count);
}

I take this task from 24.06-26.04.2017, try to solving

After experiments for solving 'if(a = 1)`:
#include <stdio.h>

int main()
{
	int a = 0;
	printf("a = %d\n",a);
	if (a = 1){
		printf("Inside if(a = 1)\n");
	}
	printf("a = %d\n",a);

	printf("----------------\n");

	a = 0;
	printf("a = %d\n",a);
	if (a = 0){
		printf("Inside if(a = 0)\n");
	}
	printf("a = %d\n",a);

	printf("----------------\n");

	a = 1;
	printf("a = %d\n",a);
	if (a = 1){
		printf("Inside if(a = 1)\n");
	}
	printf("a = %d\n",a);

	printf("----------------\n");

	a = 1;
	printf("a = %d\n",a);
	if (a = 0){
		printf("Inside if(a = 0)\n");
	}
	printf("a = %d\n",a);

	printf("----------------\n");

	printf("Check 0\n");
	if ( 0 ){
		printf("Inside if(0)\n");
	}

	printf("----------------\n");

	printf("Check 1\n");
	if ( 1 ){
		printf("Inside if(1)\n");
	}

	printf("----------------\n");

	a = 0;
	printf("a = %d\n",a);
	if(a = 2 ,a){
		printf("Inside if(a = 2, a)\n");
	}
	printf("a = %d\n",a);

	printf("----------------\n");

	a = 2;
	printf("a = %d\n",a);
	if(a = 0 ,a){
		printf("Inside if(a = 0, a)\n");
	}
	printf("a = %d\n",a);

	printf("----------------\n");
	printf("End\n");

	return 0;
}

The result:
if (a = 1)...

is equal in C code:
if (a = 1, a != 0)...
// or
if (a = 1, a)...

is equal in Go code:
a = 1
if (a != 0)...

Reopening this issue. It should not have been closed by the PR.

I will try to solve that problem in that week.


Issue with manager.

The giveaway is not starting.

I am also looking for this issue I will do it.


DZNEmptyDataSetView Frame Issue

DZNEmptyDataSetView sets its frame to the scroll view's bounds. I'm experiencing problems with this when reloading the DZNEmptyDataSetView while the scroll view is scrolled/scrolling.
- (void)didMoveToSuperview
{
    self.frame = self.superview.bounds;

    [UIView animateWithDuration:0.25
                     animations:^{_contentView.alpha = 1.0;}
                     completion:NULL];
}

In my specific case – the refresh control I'm using (SSPullToRefresh) adjusts the scroll view's insets and offsets while reloading and animating. This causes the DZNEmptyDataSetView to be in the "wrong" spot when I reload the DZNEmptyDataSetView during this animation.
The simplest solution for my use case is to do this:
- (void)didMoveToSuperview
{
    CGRect frame = self.superview.bounds;
    frame.origin.y = 0.0f;
    self.frame = frame;

    [UIView animateWithDuration:0.25
                     animations:^{_contentView.alpha = 1.0;}
                     completion:NULL];
}

Do you have any suggestions on how we could add this as an option to the project? We could possibly allow the delegate to provide a DZNEmptyDataSetView subclass to use. We could add an option for using a 0 y origin. (non relative origin?). Even just exposing the DZNEmptyDataSetView as a readonly property would allow a UIScrollView subclass to reposition the subview as needed.
Any thoughts on this would be appreciated.

Another idea – we could add a delegate that provides the initial frame for positioning in the scroll view.

I think the last idea is the simplest solution. I've opened a pull request: https://github.com/dzenbot/DZNEmptyDataSet/pull/148

Thank you for such a quick reply and for this library.

- (void)didMoveToSuperview
{
    self.frame = self.superview.frame;

    [UIView animateWithDuration:0.25
                 animations:^{_contentView.alpha = 1.0;}
                 completion:NULL];
}

This solved the problem in my case:
- (void)emptyDataSetWillAppear:(UIScrollView *)scrollView {
    scrollView.contentOffset = CGPointZero;
}

@medinaonly     useful

This solved the problem in my case:
- (void)emptyDataSetWillAppear:(UIScrollView *)scrollView {
    scrollView.contentOffset = CGPointZero;
}

It is very good. So clever. Thanks for share.


TypeError in the generate_geollm_prompts_at_location.py

Hello,
thank you for the amazing work!
I am trying to reproduce your results, but encounter an error when trying to generate prompts at location:
raceback (most recent call last):
  File "/home/sc.***.de/i***e/GeoLLM/generate_geollm_prompts_at_location.py", line 48, in <module>
    generate_prompts(bbox, num_prompts, output_file)
  File "/home/sc.***de/i***e/GeoLLM/generate_geollm_prompts_at_location.py", line 17, in generate_prompts
    data_cropped = data[top_left[0]:bottom_right[0], top_left[1]:bottom_right[1]]
                   ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: slice indices must be integers or None or have an __index__ method

Do you have an idea, what it could be?
Thank you!

The top_left coordinates aren't integers and can't be used for slicing in a NumPy array. So the simple update helps:
top_left = tuple(map(int, rowcol(transform, bbox[0], bbox[3])))
bottom_right = tuple(map(int, rowcol(transform, bbox[2], bbox[1])))


refactor(proto): fix rust protocol::transactions module name to match proto package name

Summary
Brief summary of the changes made, ie "what?"
Background
The rust module name was transaction previously, while the protobuf package name was transactions (plural). This fixes the rust name to match.
Changes

changed the module name to transactions.
updated all the relevant imports
updated the snapshots that were broken by this

Testing
existing tests all pass
Related Issues
Link any issues that are related, prefer full github links.
closes

I opened https://github.com/astriaorg/astria/pull/1380 which is a much smaller version with the core fix, doesn't touch the non generated types

closing in favor of #1380


Configure Azure Pipelines

This pipeline configuration builds the application and produces a build for the Release configuration.

Wront test


docker release ci only execute on openyurt repo

Signed-off-by: huiwq1990<EMAIL_ADDRESS>
What type of PR is this?

Uncomment only one  /kind <> line, hit enter to put that in a new line, and remove leading whitespace from that line:
/kind bug
/kind documentation
/kind enhancement
/kind good-first-issue
/kind feature
/kind question
/kind design
/sig ai
/sig iot
/sig network
/sig storage
/sig storage

What this PR does / why we need it:
Which issue(s) this PR fixes:

Fixes #
Special notes for your reviewer:

Does this PR introduce a user-facing change?

other Note

@huiwq1990 It's a good idea that only run release ci work for openyurtio/openyurt repo.

/lgtm
/approve


"CodePush - Release" task missing after instalation to Azure DevOps (VSTS)

I have installed the ms-vsclient.code-push-1.3.1.vsix extension in our AzureDevOps but we are missing “CodePush – Release” task. I can see 3 other tasks (CodePush - Patch, CodePush - Release(Cordova) and CodePush - Promote). What we want to do is to release React-Native app, and without the CodePush - Release task, we can't do it.

Hi @Imroath ,
Sorry for this long delay.
Yeah, we don't have release-react command. For releasing react app you should use react-native prepare and then CodePush-release. For more info please check this docs: https://marketplace.visualstudio.com/items?itemName=ms-vsclient.react-native-extension
Also, I see that you missed CodePush-release task. It is strange. Could you please clarify what approach have you used for installing CodePush extension? I would like to to your step and reproduce this issue.
Thanks,
Alexander

Hi @Imroath ,
Also, I see that you missed CodePush-release task. It is strange. Could you please clarify what approach have you used for installing CodePush extension? I would like to to your step and reproduce this issue.

Yes, this issue is strange. But all I did was to navigate to marketplace, https://marketplace.visualstudio.com/items?itemName=ms-vsclient.code-push and click "Get it free". Then I installed extension in my Azure DevOps Organization. But the CodePush-Release task was missing after instalation. My work around was to download and extract the package and then use tfx-cli to install CodePush-Release task manually just as we do with our custom tasks. And it worked fine.

Maybe the source of an issue is that you are using a depreciated and outdated vsts-task-lib?
Maybe you should use azure-pipelines-task-lib instead?


Explain the example

This is a question. In the example rollup.config.js for this plugin, what's the significance of using double asterisk ('src/components/**.html') in specifying the include property of the object being passed to svelte function? Wouldn't one asterisk be sufficient?

That's actually a typo – oops! It should be (and now is) src/components/**/*.html.
It's a minimatch-compatible glob pattern – the ** token (pronounced 'globstar') means that it captures all *.html files beneath src/components, regardless of whether they're in nested directories.
Thanks for flagging it up.


Add container to menu items with labels to fix display issue in firefox

The current AdminLTE demo uses a <span class="pull-right-container"> to wrap menu items with a label.
https://github.com/almasaeed2010/AdminLTE/blob/master/index.html#L347

Good catch! Thanks.


[MM-14105] Long-press menu cannot be dragged up when rotating device to landscape while menu is open

Summary

As SlideUpPanel calculates initialPosition only in constructor, rotating device will not cause re-caculation of initialPosition.
Adding marginFromTop as a key to SlideUpPanel can fix this problem in simplest way.

Ticket Link
Fixes https://github.com/mattermost/mattermost-server/issues/10275
Device Information
This PR was tested on

Samsung Galaxy S10 (Android 9.0)
iOS Simulator (iPhone XS, iOS 12.2)

/check-cla


allow undotree() to return tree for any buffer

Problem:    undotree() does not support getting the undo tree for any
buffer other than the current one.
Solution:   Add an optional "buffer number" parameter to undotree().  If
omitted, use the current buffer for backwards compatibility.
Fixes #4001.

Can you please update and rebase this PR to the latest?


Overload HLG values method

This PR overloads the built-in Mapping.values() method for HighLevelGraphs. The current values()  method makes calls to the graph __getitem__() method which, for large-ish graphs, can be slow (see https://github.com/dask/distributed/blob/d202e6253ed8ddc7919d0d4f128d88954e9859b8/distributed/client.py#L4336 for a relevant use case).
The values() method implemented in this PR instead concatenates the values for each HLG layer, avoiding repeated calls to the HighLevelGraph.__getitem__() method.
Timing the following example
In [1]: import dask.array as da
   ...:
   ...: a = da.random.randint(0, 10, size=10_000, chunks=10)
   ...:
   ...: # Create more layers for HighLevelGraph
   ...: for i in range(10):
   ...:     a = a + 1
   ...:

on the current master branch yields
In [2]: %timeit -n 5 list(a.dask.values())
81.8 ms ± 4.44 ms per loop (mean ± std. dev. of 7 runs, 5 loops each)

while with the changes in this PR we get a significant speedup factor
In [2]: %timeit -n 5 list(a.dask.values())
5.6 ms ± 2.68 ms per loop (mean ± std. dev. of 7 runs, 5 loops each)

[x] Tests added / passed
[x] Passes flake8 dask

I updated items to return a concrete list instead of a generator, and implemented keys and values methods which use the items method. At this point, I think all the values/keys/items properties should be satisfied (except being lazy).
Here are benchmark results using the original example HLG:

on the current master branch
In [2]: %timeit -n 10 list(a.dask.keys())
4.8 ms ± 1.31 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [3]: %timeit -n 10 list(a.dask.values())
78.2 ms ± 295 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [4]: %timeit -n 10 list(a.dask.items())
8.09 ms ± 300 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

with changes in this PR
In [2]: %timeit -n 10 a.dask.keys()
7.58 ms ± 1.41 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [3]: %timeit -n 10 a.dask.values()
7.32 ms ± 592 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [4]: %timeit -n 10 a.dask.items()
6.57 ms ± 328 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

@jcrist when you get a chance to take a look at this, any additional comments are appreciated

Overall this looks good to me. Could you add a small test for keys and items (unless that's already somewhere else in the test file).

LGTM. Merging.


Address data race during close database and statement

This PR should fix https://github.com/mattn/go-sqlite3/issues/426, but I don't think it's a comprehensive fix for the handling of db in this code.

Coverage increased (+0.09%) to 64.073% when pulling cd1cbf523a80ebe39ccb1b829d760852e298c128 on otoolep:conn_stmt_race into 83c59d8440943c680f94ebbe51118df4244414d6 on mattn:master.

Coverage increased (+0.09%) to 64.073% when pulling cd1cbf523a80ebe39ccb1b829d760852e298c128 on otoolep:conn_stmt_race into 83c59d8440943c680f94ebbe51118df4244414d6 on mattn:master.

Coverage increased (+0.09%) to 64.073% when pulling cd1cbf523a80ebe39ccb1b829d760852e298c128 on otoolep:conn_stmt_race into 83c59d8440943c680f94ebbe51118df4244414d6 on mattn:master.

Coverage increased (+0.09%) to 64.073% when pulling cd1cbf523a80ebe39ccb1b829d760852e298c128 on otoolep:conn_stmt_race into 83c59d8440943c680f94ebbe51118df4244414d6 on mattn:master.

Coverage increased (+0.09%) to 64.073% when pulling cd1cbf523a80ebe39ccb1b829d760852e298c128 on otoolep:conn_stmt_race into 83c59d8440943c680f94ebbe51118df4244414d6 on mattn:master.

Coverage increased (+0.09%) to 64.073% when pulling cd1cbf523a80ebe39ccb1b829d760852e298c128 on otoolep:conn_stmt_race into 83c59d8440943c680f94ebbe51118df4244414d6 on mattn:master.

LGTM Thank you


Update tools.md

Added Podtnl CLI,
Podtnl: Expose your pod to Online easily from any kubernetes clusters without creating a kubernetes service.

/check-cla

I don't think this change fits with the content guide.
If you're keen to promote the tool, you could blog about the options in the ecosystem and include Podtnl in the list you cover.

/hold

@narendranathreddythota we didn't hear back from you.
I'm going to close this. Please do consider other routes to document or promote that tool, such as social media.
/close