re-manager container stalls

Executive Summary
This is an ongoing investigation into symptoms that have recurred that seem very similar to #41, #42.
Namely, some scans, particularly large ones (such as 3D scans), fail to complete.
The problem does not seem to relate to memory pressure from the containers.
Thus far, the problem has only been seen on Waldo.
Initial report, Symptoms, troubleshooting
2022-09-08
After upgrading to Bluesky v1.10.0, scans stall despite low memory usage (~280 MB).
CLI requests to re-manager container do not complete; docker app reports 100% cpu usage.
The same symptoms have appeared 3 times today on that system, so it is very reproducible there.
Operation is restored by restarting containers.
Actions taken

currently trying to reproduce this on the test computer.

2022-09-15
Waldo froze overnight running scans.  Users say re container was not pushing CPU 100% this time.  We will continue to investigate for trends in CPU usage. Attempted to diagnose by looking for log feedback after qserver commands. The qserver commands did not reveal any new logging info.  All commands are met with timeout errors.

After the changes implemented by ddkohler, bluesky still froze overnight during scans.
The issue of 100% CPU usage is not seen. Docker is using about 14% CPU, which is fine. There are 4.3GB in memory being used by the vmmem, but that also seems normal, and the value is relatively constant.
There is a difference in the RE manager logs. Before they were just the table of points that docker spits out every once in a while. But this time (probably due to ddkohler's log changes), the logs included where it failed. Unfortunately, RE manager was unaware of an issue. The user was able to try to interrupt the scans, at which point the RE manager began trying to close out, and then failing repeatedly on loop. This was not something that was able to be done before the changes in the logger. Before, nothing happened when trying to interrupt.
This is the message RE manager is giving me as it trys to stop:
`[I 2022-09-16 12:11:26,275 bluesky_queueserver.manager.manager] Starting RE Manager process
[I 2022-09-16 12:11:26,292 bluesky_queueserver.manager.manager] Downloading the lists of existing plans and devices from the worker environment
[E 2022-09-16 12:11:31,285 bluesky_queueserver.manager.start_manager] Timeout detected by Watchdog. RE Manager malfunctioned and must be restarted
[I 2022-09-16 12:11:31,285 bluesky_queueserver.manager.manager] Starting ZMQ server at 'tcp://*:60615'
[I 2022-09-16 12:11:31,285 bluesky_queueserver.manager.manager] ZMQ control channels: encryption disabled
[I 2022-09-16 12:11:31,288 bluesky_queueserver.manager.manager] Starting RE Manager process
[I 2022-09-16 12:11:31,305 bluesky_queueserver.manager.manager] Downloading the lists of existing plans and devices from the worker environment
[E 2022-09-16 12:11:36,299 bluesky_queueserver.manager.start_manager] Timeout detected by Watchdog. RE Manager malfunctioned and must be restarted
[I 2022-09-16 12:11:36,299 bluesky_queueserver.manager.manager] Starting ZMQ server at 'tcp://*:60615'
[I 2022-09-16 12:11:36,299 bluesky_queueserver.manager.manager] ZMQ control channels: encryption disabled`

One further difference from previous bluesky stalls this week. With previous freezes, restarting docker was ineffective at solving the issue at hand. There was still a timeout issue occurring. Re-composing docker was what was needed to restore functionality. This time, the user restarted a yaq daemon that was reporting itself as not busy, but it also seems to have disconnected. The daemon was restarted in the foreground. Afterwards, the user restarted docker and functionality was restored without having to use docker compose up --build

This time, the user restarted a yaq daemon that was reporting itself as not busy, but it also seems to have disconnected. The daemon was restarted in the foreground.

@kelsonoram disconnection messages on the daemon side just mean that a yaqc client was closed (i.e. it's normal operation, no issue).  Also, can we be sure the disconnect was from a client on the bluesky end of things?  Disconnect messages are common and can happen from e.g. yaqc-qtpy
My guess is restarting docker was the actual fix.  If restarting the daemon was the fix, it means something actually went wrong on the daemon side that we failed to detect.  @kelsonoram next time if you believe the daemon is at fault, can you try come client calls/queries in yaqc-qtpy to it to see if anything seems wrong with it?  i.e. can you measure and get_measured as expected?

@ddkohler I will do that next time. I just thought it was something different of note. I was going to see if just restarting docker fixed the issue next time.

Tail of messages hook:
# tail -n30 hooks.log
save(None, *() **{}, run=None)
[event] document emitted with data keys dict_keys(['daq_channel1', 'daq_channel1_signal', 'daq_channel1_baseline', 'daq_channel2', 'daq_channel2_signal', 'daq_channel2_baseline', 'daq_channel3', 'daq_channel3_signal', 'daq_channel3_baseline', 'daq_channel4', 'hp', 'hp_destination', 'hp_Delay_1', 'hp_Delay_1_destination', 'hp_Delay_1_position_identifier', 'hp_Crystal_1', 'hp_Crystal_1_destination', 'hp_Crystal_1_position_identifier', 'hp_Delay_2', 'hp_Delay_2_destination', 'hp_Delay_2_position_identifier', 'hp_Crystal_2', 'hp_Crystal_2_destination', 'hp_Crystal_2_position_identifier', 'hp_SHG_Crystal', 'hp_SHG_Crystal_destination', 'hp_SHG_Crystal_position_identifier', 'hp_RP5_Stage', 'hp_RP5_Stage_destination', 'hp_RP5_Stage_position_identifier', 'hp_WS_Wheel_1', 'hp_WS_Wheel_1_destination', 'hp_WS_Wheel_1_position_identifier', 'hp_RP6_Stage', 'hp_RP6_Stage_destination', 'hp_RP6_Stage_position_identifier', 'hp_Crystal_Stage_2', 'hp_Crystal_Stage_2_destination', 'hp_Crystal_Stage_2_position_identifier', 'hp_WS_Wheel_2', 'hp_WS_Wheel_2_destination', 'hp_WS_Wheel_2_position_identifier', 'hp_FH_IDL', 'hp_FH_IDL_destination', 'hp_d1', 'hp_d1_destination', 'hp_d1_offset', 'hp_d1_stage', 'hp_d1_stage_destination', 'hp_d2', 'hp_d2_destination', 'hp_d2_offset', 'hp_d2_stage', 'hp_d2_stage_destination', 'd1', 'd1_destination', 'd1_offset', 'd1_stage', 'd1_stage_destination']) (run_uid='4e3b87bb-5be3-4807-b4c7-04521f3403df')
Msg('checkpoint', obj=None, args=(), kwargs={}, run=None)
checkpoint(None, *() **{}, run=None)
Msg('set', obj=<yaqc_bluesky.Device to host.docker.internal:39001 (attune-delay:d1)>, args=(0.06499999999999984,), kwargs={'group': 'set-0e6aa8'}, run=None)
set(<yaqc_bluesky.Device to host.docker.internal:39001 (attune-delay:d1)>, *(0.06499999999999984,) **{'group': 'set-0e6aa8'}, run=None)
Msg('wait', obj=None, args=(), kwargs={'group': 'set-0e6aa8'}, run=None)
wait(None, *() **{'group': 'set-0e6aa8'}, run=None)
{<yaqc_bluesky._status.Status object at 0x7fa64ffafd30>}
The object <yaqc_bluesky.Device to host.docker.internal:39001 (attune-delay:d1)> reports set is done with status True
None
Msg('trigger', obj=<yaqc_bluesky.Device to host.docker.internal:39003 (gage-compuscope:daq)>, args=(), kwargs={'group': 'trigger-6373d9'}, run=None)
trigger(<yaqc_bluesky.Device to host.docker.internal:39003 (gage-compuscope:daq)>, *() **{'group': 'trigger-6373d9'}, run=None)
Msg('trigger', obj=<yaqc_bluesky.Device to host.docker.internal:38001 (attune:hp)>, args=(), kwargs={'group': 'trigger-6373d9'}, run=None)
trigger(<yaqc_bluesky.Device to host.docker.internal:38001 (attune:hp)>, *() **{'group': 'trigger-6373d9'}, run=None)
Msg('trigger', obj=<yaqc_bluesky.Device to host.docker.internal:39001 (attune-delay:d1)>, args=(), kwargs={'group': 'trigger-6373d9'}, run=None)
trigger(<yaqc_bluesky.Device to host.docker.internal:39001 (attune-delay:d1)>, *() **{'group': 'trigger-6373d9'}, run=None)
The object <yaqc_bluesky.Device to host.docker.internal:38001 (attune:hp)> reports trigger is done with status True.
Msg('wait', obj=None, args=(), kwargs={'group': 'trigger-6373d9'}, run=None)
The object <yaqc_bluesky.Device to host.docker.internal:39001 (attune-delay:d1)> reports trigger is done with status True.
wait(None, *() **{'group': 'trigger-6373d9'}, run=None)
{<yaqc_bluesky._status.Status object at 0x7fa64fcd5b80>, <yaqc_bluesky._status.Status object at 0x7fa64fde3ca0>, <yaqc_bluesky._status.Status object at 0x7fa64fd7aee0>}
The object <yaqc_bluesky.Device to host.docker.internal:39003 (gage-compuscope:daq)> reports trigger is done with status True.
None
Msg('create', obj=None, args=(), kwargs={'name': 'primary'}, run=None)
create(None, *() **{'name': 'primary'}, run=None)
Msg('read', obj=<yaqc_bluesky.Device to host.docker.internal:39003 (gage-compuscope:daq)>, args=(), kwargs={}, run=None)
read(<yaqc_bluesky.Device to host.docker.internal:39003 (gage-compuscope:daq)>, *() **{}, run=None)
Msg('read', obj=<yaqc_bluesky.Device to host.docker.internal:38001 (attune:hp)>, args=(), kwargs={}, run=None)
read(<yaqc_bluesky.Device to host.docker.internal:38001 (attune:hp)>, *() **{}, run=None)

So the last message processed was a read of the hp, and the next message would be a read of d1
Neither of those devices appear stalled, and they respond appropriately to external communication.
Today when I encountered this, the manager process was responding, though I still got a timeout error for qserver re pause immediate.
qserver environment destroy worked as expected, killing the worker process. The worker process had a low PID (~140) and the manager process had a PID of ~19000, indicating that it had been restarted at some point, most likely.
The scan completed with a failed exit status indicating that the environment was destroyed.
So threads to pull on:

attune daemon.

This is the last device communicated with
the stall has only been observed on scans with attune axes (though probability states that it easily could just be chance of that)

attune delay daemon

This would be the next device to be communicated with
That said, it does not appear that it was attempted, so not fully likely.
That said, there have been other oddities about this hardware reporting an odd position

Possibly still an upstream thing relating to the communication between Worker and Manager processes

The manager PID being high makes me believe that it has been restarted, though the worker was never restarted.
The worker is spinning, but the manager is not
It could be that the worker is spinning on something that it expects the manager to reply to but because the manager was restarted it is not listening.

It's useful to practice using madbg [1] beforehand so that we can all be ready to use it when encountering this issue. Please try the following.
First make sure madbg is installed in your environment.
Run the following script which will CPU lock after a few seconds.
import asyncio

async def nice():
    while True:
        print("nice")
        await asyncio.sleep(1)

async def evil():
    i = 0
    while True:
        print("evil")
        await asyncio.sleep(2)
        i += 1
        if i > 3:
            while True:
                1+1

loop = asyncio.new_event_loop()
loop.create_task(nice())
loop.create_task(evil())
loop.run_forever()

Now in your command line run madbg attach <pid>. I get the following.
blaise@jasmine conda@edge ~ $ madbg attach 1350483
> /home/blaise/Desktop/play.py(2)<module>()
      1 while True:
----> 2     1+1

ipdb>

This will leave you in an interactive debug session where you can try typing bt.
[1] https://github.com/kmaork/madbg

Just some how-tos and what is useful to do when this stall occurs so we can get info:
htop and info about the offending process
To get info about the processes, in particular which one(s) are actually chewing through CPU:
Use the (>_) button in Docker Desktop to launch a command shell inside of the docker container for re_manager.
Run htop.
If htop is not installed:
run apt install htop
On first run of htop, there are some settings that make looking at it easier: hit F2, then edit "Display Options" (use arrow keys to go down in categories and then right to edit the display options)
Check the box (using arrows to navigate, space to select) for "Tree View" and "Hide userland process threads")
This will reduce what is shown and provide information about one of them being the "parent" process (the Watchdog) and that there are two child processes.
Make note of this info (Probably best to screenshot it, honestly... I didn't because I got the info I wanted out of it, but for purposes of generalizing what to do and recording as best we can...)
Make note of the PID(s) that are consuming appreciable CPU.
There should be three processes listed: one at the top of the tree, this is the Watchdog.
Two at equal level underneath the watchdog, these are the Manager and Worker.
The Worker will be the one that is using the most RAM (around 200-300 MB).
In the instance I saw yesterday, it was the Worker that was using CPU.
The important metric is CPU% which is the instantaneous CPU usage, rather than TIME+, which is the cumulative CPU usage.
You will also see an entry for htop iteself, this can be ignored.
asses state of manager process responding to inputs
Assess whether the RE manager is responding to requests:
While we have verbose logging on, if bluesky-cmds is running, you will see a sea of status messages printed to the logs (visible in either bluesky-cmds or docker desktop) with essentially the only update being the timestamp)
This is an immediate determination that the manager is in fact responding.
This can be tested explicitly via the CLI:
qserver status

Which will spit out a long JSON reply if it works or a shorter TimeoutError if it does not.
determine at what part of the plan the stall occured
Once again, start a shell in the re-manager container using the (>_) button
run:
tail -n30 hooks.log

This will print the last 30 lines of the hooks log file.
the number of lines can be changed by changing the number in the command.
In general, getting back to the previous checkpoint is a good idea, and maybe a few lines beyond that, as I did for the example above.
Possibly using madbg
To be clear, I'm not fully convinced madbg will help us here... while a really cool project that works in a rather mysterious way, I was unable to get it to attach at all while a plan was just spinning (I tested with an intentionally silly plan that has a long loop that just adds).
I think that the use of threads by queueserver makes it not quite work the way we need it to
But anyway, at least worth a shot.
Once again, start a shell in the re-manager container using (>_)
Run madbg attach <PID from htop>
see what happens... if you get to an ipdb>  prompt, try pressing w and hitting enter
Honestly, I'm not experienced enough with actually using the debuggers in python to give actually good advice, but if this were to work, it would hopefully tell more of what is going on, but it is kind of the interactive "try and see" rather than being able to go in with a firm flowchart of things to do.
attempt to recover without restarting docker entirely
While in my experience it has not worked when the queue is stalled, a good first step is to try to pause the queue:
qserver re pause immediate

I saw TimeoutError in response to this command, but if it works, CPU usage should go down and you can try either resuming or halting the plan at that stage:
qserver re resume
qserver re halt

Those two will only work if the pause was successful.
If this works, it is quite possible that the resume will just continue as normal and the acquisition can be simply completed.
The issue, though, is that it hasn't worked.
However, If the pause is unsuccessful you can do
qserver environment destroy

To kill the worker environment, at which point the offending process (if it is the Worker, as it was yesterday), will be closed
It can be reopened (and in fact if bluesky-cmds is running, it will re-open immediately).
Once it is ready, the queue can be started again (it does take some time to set up).
There is also
qserver manager stop safe off

Though if it gets to that, not sure there is any real benefit over restarting docker...

An annotated example of htop (taken while idle, so CPU usage is low, but the red ellipse in that is where I'd expect the 100% CPU to be if the presentation is the same as I saw yesterday.)
The ellipse in the RES column is showing the memory usage that is used to identify the Worker process.
The ellipse in the Command column shows the tree structure that is used to identify the Watchdog process.
The remaining process from that set, therefore, is the Manager.

Thanks so much for putting in the detail work @ksunden.

Just a thought:
Would probably be a good idea to do the tail of the hooks log multiple times before shutting down the worker, to make sure that the program is in fact not processing messages.
I doubt it is processing messages, as if it were, I'd expect the scan to continue towards completion. but worth checking.

Bluesky stalled on Waldo over the weekend. The stall followed the same patterns as previous stalls with the CPU usage for the RE-manager at 100%. The scan that was stalled on was a 2d frequency scan. The attune-delay was not being applied, so the delay stages should not have been moving.
I followed the great plan laid out by @ksunden and the following are the results:
Here is a screenshot of the htop results, hiding the non-relevant results. Like @ksunden saw earlier, the worker has the highest memory usage and has the high CPU usage. Unlike what was seen by ksunden earlier, the PIDs for the worker and the manager processes were both low. This indicates that the manager was not restarted at some point (I believe, feel free to correct me otherwise).

Similar to what @ksunden saw before, the manager was responding to requests. I saw timestamped status messages coming in regularly. To be thorough, I did a qserver status request and got the following result:
(base) C:\Windows\system32>qserver status Arguments: ['status'] C:\Users\Wright\miniconda3\lib\site-packages\zmq\_future.py:442: RuntimeWarning: Proactor event loop does not implement add_reader family of methods required for zmq. Registering an additional selector thread for add_reader support via tornado. Use asyncio.set_event_loop_policy(WindowsSelectorEventLoopPolicy()) to avoid this warning.   return self._get_loop().call_later(delay, callback) 10:59:32 - MESSAGE: {'devices_allowed_uid': '6aeeec39-f45c-43d5-a565-9b1c4718db29',  'devices_existing_uid': 'f55ca387-c408-4a28-8f6f-295599f4328e',  'items_in_history': 46,  'items_in_queue': 39,  'lock': {'environment': False, 'queue': False},  'lock_info_uid': '37b00059-e4ca-4cf2-a989-fec6a3266eff',  'manager_state': 'executing_queue',  'msg': 'RE Manager v0.0.16',  'pause_pending': False,  'plan_history_uid': 'f9fdef9d-7c5e-4c00-b7c6-b33b571a1276',  'plan_queue_mode': {'loop': False},  'plan_queue_uid': '409e94d3-a033-448a-bd84-7b65dcd83d5a',  'plans_allowed_uid': '9502c798-7e4f-4087-8c3b-38d3d9102009',  'plans_existing_uid': '057764f8-9857-44a4-9bc2-6efa07ab37ec',  'queue_stop_pending': False,  're_state': 'running',  'run_list_uid': '5b60e537-1814-4877-86f1-26c9454e1748',  'running_item_uid': '10ab9909-9bde-41f6-ae73-6e1dfaf574db',  'task_results_uid': 'e9a663d6-0d69-4551-9676-bb4211ef4487',  'worker_background_tasks': 0,  'worker_environment_exists': True,  'worker_environment_state': 'executing_plan'}
I then looked into the hooks log file. This showed that when the scan stalled, the last message processed was 'hp' and the next message would be 'twin2'. This is the second time it stalled on reading an hp message. I do not know yet whether this is a coincidence or the source of our troubles. The previous checkpoint of the hooks log is posted below. I did the tail of the hooks.log several times to check if the worker was processing messages still. The output never changed, so the worker was not processing messages.
 None Msg('create', obj=None, args=(), kwargs={'name': 'primary'}, run=None) create(None, *() **{'name': 'primary'}, run=None) Msg('read', obj=<yaqc_bluesky.Device to host.docker.internal:39003 (gage-chopping:daq)>, args=(), kwargs={}, run=None) read(<yaqc_bluesky.Device to host.docker.internal:39003 (gage-chopping:daq)>, *() **{}, run=None) Msg('read', obj=<yaqc_bluesky.Device to host.docker.internal:38001 (attune:hp)>, args=(), kwargs={}, run=None) read(<yaqc_bluesky.Device to host.docker.in
I tried the madbg method but it did not work. The output of my attempt is posted below:
# madbg attach 18047 Traceback (most recent call last):   File "/usr/local/bin/madbg", line 8, in <module>     sys.exit(cli())   File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 1130, in __call__     return self.main(*args, **kwargs)   File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 1055, in main     rv = self.invoke(ctx)   File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 1657, in invoke     return _process_result(sub_ctx.command.invoke(sub_ctx))   File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 1404, in invoke     return ctx.invoke(self.callback, **ctx.params)   File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 760, in invoke     return __callback(*args, **kwargs)   File "/usr/local/lib/python3.9/dist-packages/madbg/__main__.py", line 34, in attach     attach_to_process(pid, port, connect_timeout=timeout)   File "/usr/local/lib/python3.9/dist-packages/madbg/api.py", line 34, in attach_to_process     _inject_set_trace(pid, ip, port)   File "/usr/local/lib/python3.9/dist-packages/madbg/api.py", line 28, in _inject_set_trace     inject_py(pid, f'__import__("signal").signal({sig_num},lambda _,f:__import__("madbg").set_trace(f,"{ip}",{port}))')   File "/usr/local/lib/python3.9/dist-packages/hypno/hypno.py", line 42, in inject_py     inject(pid, str(temp.name))   File "/usr/local/lib/python3.9/dist-packages/pyinjector/pyinjector.py", line 103, in inject     return injector.inject(library_path)   File "/usr/local/lib/python3.9/dist-packages/pyinjector/pyinjector.py", line 87, in inject     call_c_func(libinjector.injector_inject, self.injector_p, library_path, pointer(handle))   File "/usr/local/lib/python3.9/dist-packages/pyinjector/pyinjector.py", line 66, in call_c_func     raise exception_cls(func.__name__, ret, libinjector.injector_error()) pyinjector.pyinjector.InjectorError: injector_inject returned -1: The target process unexpectedly terminated with exit code 127.
After following these steps in tracking down the problem, I tried to get bluesky working again. I tried the command qserver re pause immediate, but it did not work. The attempt is shown below:
(base) C:\Windows\system32>qserver re pause immediate Arguments: ['re', 'pause', 'immediate'] C:\Users\Wright\miniconda3\lib\site-packages\zmq\_future.py:442: RuntimeWarning: Proactor event loop does not implement add_reader family of methods required for zmq. Registering an additional selector thread for add_reader support via tornado. Use asyncio.set_event_loop_policy(WindowsSelectorEventLoopPolicy()) to avoid this warning.   return self._get_loop().call_later(delay, callback) 11:13:39 - MESSAGE: {'msg': 'Timeout occurred', 'success': None}
The qserver re environment destroy command was successful in restarting the worker without restarting docker. The scans were able to then continue.

Okay, so N=2 it has occurred while attempting to read the hp, looking more and more solid that that is where to look to fix this... just not sure why that would be yet.

I note that the last line looks to be cut off, that could be a copy paste error or it could indicate that the buffer never got flushed to the output file..., hmmmm

Good catch, that is a copy and paste error. that last line should be:
read(<yaqc_bluesky.Device to host.docker.internal:38001 (attune:hp)>, *() **{}, run=None)

I updated the comment to accurately reflect what was given in the hooks log. Sorry about that @ksunden

When I or @untzag get another chance to look at this, we may implement some additional logging to get at what is happening here, but for now documenting as thoroughly as we can helps bring confidence to the conclusion that it is something happening while reading the hp (possibly attune daemons in general, but both of the most well documented instances are hp)
I have a couple logs I may wish to put in bluesky itself to ensure that it is happening in the actual hardware communication, but more than likely it is.
Beyond that, some logs in yaqc and/or yaqc-bluesky may be warranted.
I've looked at darn near every line of code that gets run for that operation in that process short of looking at python itself and come up short on an explanation thus far: bluesky, yaqc-bluesky, yaqc, fastavro.
The most suspicious thing so far is a potential infinite loop in yaqc, but even that my analysis of the function is that it should idle the CPU instead of spinning the CPU, so I'm not sure it fits the presentation.
That said, it is the only non-fixed length loop I can see in that whole stack below the bluesky run engine itself.
Logging more fine grained may be the best we can do to zero in on this... unfortunately the feedback loop is slow, but I have confidence we can resolve this eventually, just need to find it first.

Lazily using the logged in account at the lab machine, but this is @ksunden.
Stall did occur overnight, same symptoms: last Msg was read of hp, spinning CPU on the worker, environment destroy successfully relieved the spinning
Before doing environment destroy, I attempted a few extra steps:

Did madbg with the correct PID, but it did not work (just hung)
I tried restarting lightcon motors, attune delay and attune, to see if that jolt kicked it in the right way, to no avail.

This leads me to think it may not be the loop I said I was suspicious of... specifically because even if I am wrong about CPU usage in that loop, it should error out of that loop on lost connection.

If I interpret this correctly, the problem is that calls to hp dependent on port 38708 never get back to the run engine?   @kelson is does that port correspond to the thorlabs motor controlling the harmonic crystal?

@ddkohler The port 38708 corresponds to the lightcon-topas4-motor for the hp_RP6_Stage.
Since this is the first time we had this extensive logging, I don't know if the problem has anything to do with this motor specifically or if it just happened to be this motor among all the hp motors.

Whichever daemon it actually is, I think this log rules out bluesky as the problem and suggests a deeper look into yaqc-bluesky processes.

Question: what are the the keys for the position identifiers for that motor?
c = yaqc.Client(38708)
c.get_position_identifier_options()

If I remember correctly for that motor it is {"ON", "OFF"} (not sure of order, but that doesn't matter)
If it is empty  (no discrete positions configured), then I'm really not sure what is going on.
If it is not empty, I may have an idea regarding our dependency, fastavro, not erroring in a case where I would have expected it to when reading strings that are incomplete, and thus causing some problems on rare occasions when the bytes don't make it in time to be parsed.
(specifically, if it looks like I should have a length 3 string, but only provide 2 bytes, it will just give you the length 2 string and leave the last byte hanging)
I still do not have an explanation for how that causes the CPU utilization we are seeing, but was something I noticed when thinking on this, that would be quite rare, and dependent on timing we don't have control over fully.
Blaise and I have discussed another round of logging, which we will try to implement.
That said, if this is the reason, then it's extremely possible that logging in the critical portion of the code to observe this will actually end up shifting probabilities to where we don't see it (but it isn't "fixed", just "hiding")
Soooo, if we do implement the next round of logging and then do not see the stall happen for a significant amount of time, there may be a round of logging where we take out the logs in the critical portion, but try to catch it afterwards and try to trigger it again so we are confident about what is happening.

I am additionally curious about the WS_Wheel_1 prior, in the cut off portion of that log what the identifier was that it read, as it could have been that one that caused it, and if it was truncated from what we expect, that would confirm this theory (though again, not sure on the mechanism of it taxing CP....)
That motor has longer names than most of the others, and thus would be more likely to have this particular error.

This explanation also explains why it is a problem for waldo and not the other systems, as none of the other systems use discrete motors.
Still, I wish I had an explanation for the mechanism of the high CPU utilization... that is still a mystery to me

Here are the position identifier options:
In [3]: c = yaqc.Client(38708)
   ...: c.get_position_identifier_options()
Out[3]: ['Rotate 90 deg', 'No rotation']

Can you do 38707 as well, please?

Yes, I don't know why I didn't add that one before:
In [2]: c = yaqc.Client(38707)
   ...: c.get_position_identifier_options()
Out[2]: ['600-765', '755-1030', '1030-1385', '1385-1550', '1550-1970', '1970-2600']

Okay, I've wavered back and forth a bit, the mechanics of this are super subtle and timing based in ways that are difficult to describe and ways that are potentially self correcting in many instances, actually, which makes it harder to find.
If I'm right, we may actually have many of the strings that are read be truncated, we just don't notice because we don't usually actually care about the strings (they don't make it to the wt5 files, for instance)

Another stall, another series of logs:
htop

Hooks log
The stall occurred in hp once again, but a different motor than yesterday. It failed when waiting for the response from the hp_SHG_Crystal at port 38705. Immediately before this motor polling, it was talking to the hp_Crystal_2 motor (at port 38704).
None
Msg('create', obj=None, args=(), kwargs={'name': 'primary'}, run=None)
create(None, *() **{'name': 'primary'}, run=None)
Msg('read', obj=<yaqc_bluesky.Device to host.docker.internal:39003 (gage-chopping:daq)>, args=(), kwargs={}, run=None)
read(<yaqc_bluesky.Device to host.docker.internal:39003 (gage-chopping:daq)>, *() **{}, run=None)
FunctionArgs(name='Base.read', host='host.docker.internal', port=39003, args=(), kwargs={})
FunctionArgs(name='IsSensor._read', host='host.docker.internal', port=39003, args=(OrderedDict(),<PHONE_NUMBER>.190457), kwargs={})
FunctionArgs(name='Base._read', host='host.docker.internal', port=39003, args=(OrderedDict(),<PHONE_NUMBER>.190457), kwargs={})
FunctionResponse(name='Base._read', host='host.docker.internal', port=39003, data=OrderedDict())
FunctionResponse(name='IsSensor._read', host='host.docker.internal', port=39003, data=OrderedDict([('daq_ai0', {'value': -1.045074462890625e-05, 'timestamp':<PHONE_NUMBER>.190457}), ('daq_ai1', {'value': -2.420043945312493e-06, 'timestamp'>
FunctionResponse(name='Base.read', host='host.docker.internal', port=39003, data=OrderedDict([('daq_ai0', {'value': -1.045074462890625e-05, 'timestamp':<PHONE_NUMBER>.190457}), ('daq_ai1', {'value': -2.420043945312493e-06, 'timestamp': 166>
Msg('read', obj=<yaqc_bluesky.Device to host.docker.internal:38001 (attune:hp)>, args=(), kwargs={}, run=None)
read(<yaqc_bluesky.Device to host.docker.internal:38001 (attune:hp)>, *() **{}, run=None)
FunctionArgs(name='Base.read', host='host.docker.internal', port=38001, args=(), kwargs={})
FunctionArgs(name='HasPosition._read', host='host.docker.internal', port=38001, args=(OrderedDict(),<PHONE_NUMBER>.1937857), kwargs={})
FunctionArgs(name='HasDependent._read', host='host.docker.internal', port=38001, args=(OrderedDict(),<PHONE_NUMBER>.1937857), kwargs={})
FunctionArgs(name='Base._read', host='host.docker.internal', port=38001, args=(OrderedDict(),<PHONE_NUMBER>.1937857), kwargs={})
FunctionResponse(name='Base._read', host='host.docker.internal', port=38001, data=OrderedDict([('hp_destination', {'value':<PHONE_NUMBER>300384, 'timestamp':<PHONE_NUMBER>.1937857}), ('hp_position', {'value':<PHONE_NUMBER>300384, 'timestamp': >
FunctionArgs(name='Base.read', host='host.docker.internal', port=38701, args=(), kwargs={})
FunctionArgs(name='HasPosition._read', host='host.docker.internal', port=38701, args=(OrderedDict(),<PHONE_NUMBER>.1974971), kwargs={})
FunctionArgs(name='Base._read', host='host.docker.internal', port=38701, args=(OrderedDict(),<PHONE_NUMBER>.1974971), kwargs={})
FunctionResponse(name='Base._read', host='host.docker.internal', port=38701, data=OrderedDict([('hp_Delay_1_destination', {'value': 3.8210821724195463, 'timestamp':<PHONE_NUMBER>.1974971}), ('hp_Delay_1_position', {'value': 3.8209375, 'tim>
FunctionResponse(name='HasPosition._read', host='host.docker.internal', port=38701, data=OrderedDict([('hp_Delay_1', {'value': 3.8209375, 'timestamp':<PHONE_NUMBER>.1974971}), ('hp_Delay_1_destination', {'value': 3.8210821724195463, 'times>
FunctionResponse(name='Base.read', host='host.docker.internal', port=38701, data=OrderedDict([('hp_Delay_1', {'value': 3.8209375, 'timestamp':<PHONE_NUMBER>.1974971}), ('hp_Delay_1_destination', {'value': 3.8210821724195463, 'timestamp': 1>
FunctionArgs(name='Base.read', host='host.docker.internal', port=38702, args=(), kwargs={})
FunctionArgs(name='HasPosition._read', host='host.docker.internal', port=38702, args=(OrderedDict(),<PHONE_NUMBER>.2025602), kwargs={})
FunctionArgs(name='Base._read', host='host.docker.internal', port=38702, args=(OrderedDict(),<PHONE_NUMBER>.2025602), kwargs={})
FunctionResponse(name='Base._read', host='host.docker.internal', port=38702, data=OrderedDict([('hp_Crystal_1_destination', {'value': -2.8479981504738907, 'timestamp':<PHONE_NUMBER>.2025602}), ('hp_Crystal_1_position', {'value': -2.8473160>
FunctionResponse(name='HasPosition._read', host='host.docker.internal', port=38702, data=OrderedDict([('hp_Crystal_1', {'value': -2.847316064885251, 'timestamp':<PHONE_NUMBER>.2025602}), ('hp_Crystal_1_destination', {'value': -2.8479981504>
FunctionResponse(name='Base.read', host='host.docker.internal', port=38702, data=OrderedDict([('hp_Crystal_1', {'value': -2.847316064885251, 'timestamp':<PHONE_NUMBER>.2025602}), ('hp_Crystal_1_destination', {'value': -2.8479981504738907, >
FunctionArgs(name='Base.read', host='host.docker.internal', port=38703, args=(), kwargs={})
FunctionArgs(name='HasPosition._read', host='host.docker.internal', port=38703, args=(OrderedDict(),<PHONE_NUMBER>.2075458), kwargs={})
FunctionArgs(name='Base._read', host='host.docker.internal', port=38703, args=(OrderedDict(),<PHONE_NUMBER>.2075458), kwargs={})
FunctionResponse(name='Base._read', host='host.docker.internal', port=38703, data=OrderedDict([('hp_Delay_2_destination', {'value': 39.32391605124373, 'timestamp':<PHONE_NUMBER>.2075458}), ('hp_Delay_2_position', {'value': 39.3236443525403>
FunctionResponse(name='HasPosition._read', host='host.docker.internal', port=38703, data=OrderedDict([('hp_Delay_2', {'value': 39.32364435254034, 'timestamp':<PHONE_NUMBER>.2075458}), ('hp_Delay_2_destination', {'value': 39.32391605124373,>
FunctionResponse(name='Base.read', host='host.docker.internal', port=38703, data=OrderedDict([('hp_Delay_2', {'value': 39.32364435254034, 'timestamp':<PHONE_NUMBER>.2075458}), ('hp_Delay_2_destination', {'value': 39.32391605124373, 'timest>
FunctionArgs(name='Base.read', host='host.docker.internal', port=38704, args=(), kwargs={})
FunctionArgs(name='HasPosition._read', host='host.docker.internal', port=38704, args=(OrderedDict(),<PHONE_NUMBER>.2118905), kwargs={})
FunctionArgs(name='Base._read', host='host.docker.internal', port=38704, args=(OrderedDict(),<PHONE_NUMBER>.2118905), kwargs={})
FunctionResponse(name='Base._read', host='host.docker.internal', port=38704, data=OrderedDict([('hp_Crystal_2_destination', {'value': 6.610320868235731, 'timestamp':<PHONE_NUMBER>.2118905}), ('hp_Crystal_2_position', {'value': 6.6111301990>
FunctionResponse(name='HasPosition._read', host='host.docker.internal', port=38704, data=OrderedDict([('hp_Crystal_2', {'value': 6.611130199070004, 'timestamp':<PHONE_NUMBER>.2118905}), ('hp_Crystal_2_destination', {'value': 6.610320868235>
FunctionResponse(name='Base.read', host='host.docker.internal', port=38704, data=OrderedDict([('hp_Crystal_2', {'value': 6.611130199070004, 'timestamp':<PHONE_NUMBER>.2118905}), ('hp_Crystal_2_destination', {'value': 6.610320868235731, 'ti>
FunctionArgs(name='Base.read', host='host.docker.internal', port=38705, args=(), kwargs={})
FunctionArgs(name='HasPosition._read', host='host.docker.internal', port=38705, args=(OrderedDict(),<PHONE_NUMBER>.2165103), kwargs={})
FunctionArgs(name='Base._read', host='host.docker.internal', port=38705, args=(OrderedDict(),<PHONE_NUMBER>.2165103), kwargs={})

Position Identifiers
The position identifiers for the daemon that failed us here and the one preceding it are below:
hp_SHG_Crystal position identifier
In [2]: c = yaqc.Client(38705)
   ...: c.get_position_identifier_options()
Out[2]: ['Max SHG', 'Off']

hp_Crystal_2 position identifier
In [3]: c = yaqc.Client(38704)
   ...: c.get_position_identifier_options()
Out[3]: []

Some other things to note which are probably not that relevant:

Ever since the most recent extra logging was implemented by @ksunden, bluesky-cmds GUI has been closing when it stalls. I
thought that I may have accidentally closed it Wednesday evening, but I made sure to keep the GUI going on Thursday night and it was exited out of when I came in.

Ryan has had the ps table's bluesky crash on him in the recent weeks. Yesterday (I think, sometime this week for sure) before he restarted docker I looked at the stats for the RE-manager, and the cpu was at 100%. So that is just to add to our sample size, and that this issue may not be specific to Waldo/light conversion. The ps table did behave differently though, in that it fails when it tries to start a scan, but if a scan is going it does not fail. So a bit of the opposite for Waldo which stalls during a scan but is fine adding scans.

Ever since the most recent extra logging was implemented by @ksunden, bluesky-cmds GUI has been closing when it stalls.

The only plausible reason for this that I can think of at the moment is that the OS is killing it because it needs to recover resources.... even that doesn't make a ton of sense to me, though.
I will state that I did notice it was closed when I came in on Tuesday, though I had no reason to believe it was not simply closed intentionally.
At this stage, I think it is a red herring to go after that, so I think we stay the course and try to resolve this with logging.
As for the ps system... I think we are far enough down the path with waldo that we first resolve what we see there then revisit if the ps system needs anything different.


Merge repositories

We have two repos and working that work into our new repo scholarsphere-4

The database/model layer is completely ported over as of today. We've not ported over the workflow UI yet. I'm not sure if we want to bring that over verbatim, or do a round of wireframing and UI design first?

Going to move the workflow UI over. It may end up being reworked but this will give us something to test.


How to Query data by ORM with condition(ORDER BY, LIMIT, ..)

Hi Mikestefanello,
In routes, I attempts to Query n latest data that saved in db before.
Assume, I want to Query 14 newest posts(latest created_at).
I coded this code:
// ./routes/posts.go
...
	smt, err := c.Container.ORM.Posts.
		Query().
		Order(ent.Desc("created_at")).
		Limit(14).
		Only(ctx.Request().Context())
...

but it returned error:
ent: posts not singular

Could you tell me what is correct way to use .Order()?
Thanks

Did you go through Ent's documentation? It's extremely well-written and comprehensive. What stands out to me is you're using Only() which is used when you are expected 1 an only 1 result. For a limit query like this, I believe All() is what you want.

Thank Mikestefanello. That's correct. After replacing Only with All, It's all Okay.


Compatible with spacy 2.0.3?

Using this code:
from keras_text.processing import WordTokenizer
tokenizer = WordTokenizer()
tokenizer.build_vocab(["this is a text", "an other "])

I get an error:
ypeError                                 Traceback (most recent call last)
<ipython-input-12-a4643a71418a> in <module>()
      1 from keras_text.processing import WordTokenizer
      2 tokenizer = WordTokenizer()
----> 3 tokenizer.build_vocab(["this is a text hello", "an other "])

~/venvs/srPrimaryPredFull/lib/python3.6/site-packages/keras_text-0.1-py3.6.egg/keras_text/processing.py in build_vocab(self, texts, verbose, **kwargs)
    367         self._num_texts = len(texts)
    368
--> 369         for token_data in self.token_generator(texts, **kwargs):
    370             indices, token = token_data[:-1], token_data[-1]
    371             count_tracker.update(indices)

~/venvs/srPrimaryPredFull/lib/python3.6/site-packages/keras_text-0.1-py3.6.egg/keras_text/processing.py in token_generator(self, texts, **kwargs)
    549         }
    550
--> 551         for text_idx, doc in enumerate(nlp.pipe(texts, **kwargs)):
    552             for word in doc:
    553                 processed_word = self._apply_options(word)

TypeError: pipe() got an unexpected keyword argument 'entity'

It seem to me that the code is not compatible with spacy 2.0.3, the latest version

I tested this with spacy 1.9.0. Tried installing spacy 2.0.3 but I am getting error:
from .ops import NumpyOps, CupyOps
File "ops.pyx", line 22, in init thinc.neural.ops
ImportError: cannot import name copy_array

I will look into this as soon as i can get spacy up and running again. In the meantime, feel free to fix this yourself and submit a PR :)

Fixed.


[dv] Create clock and reset connectivity test for all IPs in top-level

Use FPV connectivity framework, so that we can eliminate this kind of SW tests.
@sriyerg @cindychip

@matutem this sounds like a P1 task. WDYT?

An update: the global clock and alert_handler cg_en connectivity was instrumented in formal a few weeks ago in https://github.com/lowRISC/opentitan/pull/10965. The reset and alert_handler enable connectivity is yet to be done.

At this point is this a DV enhancement or part of the M2 plan? Have we implemented the sw connectivity tests yet?
Marking as M2 assuming this will reduce the number of software tests.

yeah i think this is indeed part of the M2.
And there is a testplan for this here
There are also additional items related to ast that fall into this category, so definitely something we need to complete.


We should update to the latest version of tone

Yes please, current Chrome version is rejecting to make any sound before some interaction (click) has been done. This makes all demos muted (I checked it in mm.Player with callback).

Tone.js:4043 The AudioContext was not allowed to start. It must be resumed (or created) after a user gesture on the page. https://goo.gl/7K7WLu

According to the Google Developers referenced documentation this is a new Chrome policy to avoid autoplay.

Note: The Web Audio API will be included in the Chrome autoplay policy with M71 (December 2018).

Tone.js detects some problem and goes mute forever.

  // set the audio context initially, and if one is not already created
  if (Tone.supported && !Tone.initialized) {
      Tone.context = new Tone.Context();
      // log on first initialization
      // allow optional silencing of this log
      if (!window.TONE_SILENCE_VERSION_LOGGING) {
          // eslint-disable-next-line no-console
          console.log('%c * Tone.js ' + Tone.version + ' * ', 'background: #000; color: #fff');
      }
  }

This appears to happen in the Tone library initialisation, so there's no way to way to work-around it in our own code.

Closing -- I think we're actually on the latest stable version

@tambien 0.12.80 is the latest stable, and you haven't released @next (13.x) and it's still in dev mode, right?

@notwaldorf yes still dev, but hoping to find time to make a release very soon.

@tambien cool cool, let us know when it's out and the canaries have survived the mines and I'll reopen this 😂


Populate WorkerMetadata in WorkerInitResponse message

The protobuf has been updated to include a WorkerMetadata property within the WorkerInitResponse message. The worker should include WorkerMetadata when sending the WorkerInitResponse to the host.
PR for reference on how to fill in the fields.
You will need to pull in the latest protobuf version first (this doc may help).
message WorkerMetadata {
  // The runtime/stack name
  string runtime_name = 1;

  // The version of the runtime/stack
  string runtime_version = 2;

  // The version of the worker or worker SDK
  string worker_version = 3;

  // The worker bitness/architecture
  string worker_bitness = 4;

  // Optional additional custom properties
  map<string, string> custom_properties = 5;
}

Latest proto version is now v1.5.8

@vrdmr can we prioritize this for the next sprint?

Documentation to help with implementation: https://dev.azure.com/msazure/One/_git/AAPT-Antares-Docs?path=/TeamDocs/FunctionTeamDocs/Design/OOPWorkers/WorkerMetadata.md&version=GBmaster&_a=preview

@liliankasem Hi Lilian, sure a task to added to our backlog and we will prioritize it next sprint.


Renewal dashboard frontend

What is this PR for?
Frontend renewal for the backend renewal. -> https://github.com/rekcurd/dashboard/pull/75
This PR includes

Add UI for data_server
Add UI for Project
Add UI for Project Access Right
Replace redux-form to formik
Use Yup

What type of PR is it?
Feature
What is the issue?
https://github.com/rekcurd/dashboard/issues/66
https://github.com/rekcurd/dashboard/issues/61
How should this be tested?
No frontend test available...

Codecov Report

Merging #83 into master will increase coverage by 0.12%.
The diff coverage is 88.05%.

@@            Coverage Diff             @@
##           master      #83      +/-   ##
==========================================
+ Coverage   83.57%   83.69%   +0.12%
==========================================
  Files          42       42
  Lines        2094     2128      +34
==========================================
+ Hits         1750     1781      +31
- Misses        344      347       +3

Impacted Files
Coverage Δ

rekcurd_dashboard/apis/api_project.py
95.58% <100%> (+0.5%)
:arrow_up:

rekcurd_dashboard/apis/api_service_routing.py
100% <100%> (ø)
:arrow_up:

rekcurd_dashboard/apis/api_service.py
98.73% <100%> (+0.03%)
:arrow_up:

rekcurd_dashboard/apis/kubernetes_handler.py
82.95% <50%> (-0.32%)
:arrow_down:

rekcurd_dashboard/apis/__init__.py
86.66% <50%> (+0.22%)
:arrow_up:

rekcurd_dashboard/apis/api_admin.py
54.41% <66.66%> (-0.07%)
:arrow_down:

rekcurd_dashboard/apis/api_service_deployment.py
96.4% <85.71%> (+0.3%)
:arrow_up:

rekcurd_dashboard/apis/api_model.py
97.36% <88.88%> (+0.07%)
:arrow_up:

rekcurd_dashboard/apis/api_kubernetes.py
88.63% <88.88%> (-0.6%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2d034bb...946c2e3. Read the comment docs.

@yuki-mt Thank you for your review.

The current dispatchers do no use thunk, so we can remove react-thunk, I guess.

I am not familiar with redux, so could you send us the PR for this?

@keigohtr
OK! I will send a new PR to remove redux-thunk .

@yuki-mt

typo: APIRequestStatusList.failue -> failure
Now you can access to /projects/{non_existing_project_id}/data_servers , and submit. This can be confusing although this is a minor issue.

I have updated!

https://github.com/rekcurd/dashboard/pull/83/commits/6725f62ad6682e9f7353573ab6d319ec0043f23b https://github.com/rekcurd/dashboard/pull/83/commits/88b93ce5d4ab041f6dad324b3648ed9c7343c5d7 https://github.com/rekcurd/dashboard/pull/83/commits/2c38cb095a37bccabc7b365b8139f2ff43c0d1f3
https://github.com/rekcurd/dashboard/pull/83/commits/1a1853dfee037e13813bf9412b3605afe1e07721

@yuki-mt
I have updated!

OK! I will send a new PR to remove redux-thunk .

After this PR is merged, all issues will be solved! Thank you!


fix parsing on @names, dat:// urls

i just noticed that we have an issue where if i write @neauoire's boat is cool the 's will cause the parser to fail to recognize the name.
there's a similar case with mentioning a dat in a parens as in (dat://ab64573efd68035014d94b83bf034012f415ff4678d3ecf291237e0bdac741b1)
where the beginning parens will cause the parser to fail to recognize the dat link, and if recognized then the last parens will be counted as one of the characters of the string.
i'm writing this issue up instead of fixing it as it's 00:47 already and i have to go to bed!

the 's will cause the parser to fail to recognize the name.

Same if you have a @username with a period or comma after it.

OK, here's a quick fix for dealing with single punctuation marks after usernames:
Insert
var wordend = word.substr(word.length-1,word.length).match(/[.,\/#!$%\^&\*;:{}=\-_`~()]/g);
if(wordend){word = word.substr(0,word.length-1);}else{wordend = "";}

after line 101 of scripts/entry.js. And then add a + wordend at the end (inside the parens of course) of the newly-numbered lines 105 and 108. It will then link to their portal:

This definitely isn't the best way to do it, and doesn't deal with 'ss or multiple punctuation marks following usernames (e.g. cool new project by @username!!!) so I didn't want to send a PR or anything yet. I'm not too familiar with JS, just trying to help out. 😅 Maybe we'll want to m.split on punctuation marks too, or something, instead of just spaces?

This was just fixed by @cblgh, can anyone double check that it works for all cases?

@neauoire Seems to work with all regular punctuation as long as there's a space (or nothing, i.e. at the start of the message) directly preceding the username.


UE5 5.2 Crashing

Plug-In crashes the engine when it is asked to do a trace. Normally is functional for a couple of minutes/traces before this. I don't have the project this was happening in anymore but if I have the time I'll replicate it, I already replicated it twice so it seems to be persistent.

Hi @Polar2b thank you for reporting the issue. Is there any chance you could attach a call stack from the crash that occurs on your side? I'd greatly appreciate help with pointing out where the issue could be so I can fix it!

Yea of course I should be able to recreate it later today or sometime this
weekend. Thank you.
On Fri, Mar 15, 2024 at 10:54 AM Robert Lewicki @.***>
wrote:

Hi @Polar2b https://github.com/Polar2b thank you for reporting the
issue. Is there any chance you could attach a call stack from the crash
that occurs on your side? I'd greatly appreciate help with pointing out
where the issue could be so I can fix it!
—
Reply to this email directly, view it on GitHub
https://github.com/rlewicki/MeleeTrace/issues/12#issuecomment-1999959106,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/A47Y3PN5CA47J64PUHBQ5KDYYMKSPAVCNFSM6AAAAABEYE36DSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJZHE2TSMJQGY
.
You are receiving this because you were mentioned.Message ID:
@.***>

Here you are: This should be all the info.
SpellCharacterMerge_2.log
Unhandled Exception: EXCEPTION_ACCESS_VIOLATION reading address 0x0000000000000000
UnrealEditor_MeleeTrace!UMeleeTraceComponent::InternalStartTrace() [D:\build\U5M-Marketplace\Sync\LocalBuilds\PluginTemp\HostProject\Plugins\MeleeTrace\Source\MeleeTrace\Private\MeleeTraceComponent.cpp:236]
UnrealEditor_MeleeTrace!UMeleeTraceComponent::StartTraceWithContext() [D:\build\U5M-Marketplace\Sync\LocalBuilds\PluginTemp\HostProject\Plugins\MeleeTrace\Source\MeleeTrace\Private\MeleeTraceComponent.cpp:113]
UnrealEditor_MeleeTrace!UAnimNotifyState_MeleeTrace::NotifyBegin() [D:\build\U5M-Marketplace\Sync\LocalBuilds\PluginTemp\HostProject\Plugins\MeleeTrace\Source\MeleeTrace\Private\AnimNotifyState_MeleeTrace.cpp:32]
UnrealEditor_Engine
UnrealEditor_Engine
UnrealEditor_Engine
UnrealEditor_Engine
UnrealEditor_Engine
UnrealEditor_Engine
UnrealEditor_Engine
UnrealEditor_Engine
UnrealEditor_Engine
UnrealEditor_Core
UnrealEditor_Core
UnrealEditor_Core
UnrealEditor_Engine
UnrealEditor_Engine
UnrealEditor_Engine
UnrealEditor_Engine
UnrealEditor_UnrealEd
UnrealEditor_UnrealEd
UnrealEditor
UnrealEditor
UnrealEditor
UnrealEditor
UnrealEditor
UnrealEditor
kernel32
ntdll

Thank you, this is very helpful! I can see where the issue is now! I will fix this tonight and submit new release to the marketplace and GitHub releases.

@Polar2b I've addressed this issue in latest release v1.3.2. If you could get latest and let me know if this fixes the issue for you I would be very grateful!

Just tested it out and It looks like it works, thank you so much, this helps a lot with prototyping. I'm glad I was able to help you out as well.

Cheers, I'm very happy to hear this solved your issues! Have a great day!


Add a link to external documentation on the front page

Reference item 10

We need to figure out two things:

where to configure this link
where to put this link in the client

The external link should get configured on the admin page under the Site settings.

The location of the link should be above the "Help" which is located in the left-hand column.  The link should also appear when the "Help" link is visible.

The name "external documentation" is too generic. It should also be an admin-configurable field.


Added alert feature

Description
Please include a summary of the change and which issue is fixed.
Fixes #41
Replace issue_no with the issue number which is fixed in this PR
Type of change
Please delete options that are not relevant.

[x] New feature (non-breaking change which adds functionality)

Checklist:

[x] My code follows the style guidelines(Clean Code) of this project
[x] I have performed a self-review of my own code
[x] My changes generate no new warnings

Nice 👍

thanks👍


Several minor suggestions

Very nice work Bella!
If I can make a few minor suggestions for improvement:

In the "Branches" chapter you mention a "master" branch a few times. Now, most of the git-using world has decided to rename this branch to "main" wherever possible, because of the old name's slavery connotations. You will still see "master" in some older projects, mainly because changing the name of the main branch is a risky process, especially if you have things like automatic deployment scripts based on this.

In Git Conventions you mention that HEAD^3 is shorthand for 3 commits before current HEAD. This is a very tricky one, but this is not quite true. HEAD~3 (with a tilde) is 3 commits back, but can only deal with a linear history. On the other hand HEAD^3 (with the hat) can deal with a branching history and may refer to a commit only 2 steps back from the current position, but along a side branch. See: https://stackoverflow.com/a/2222920

Personally I think going up a branching history is rather confusing and once I need to do that I tend to switch to a Git UI tool in my IDE instead. So I use ~ instead of ^ and switch to the UI once ^ becomes necessary.

As a next step, it might be good to add a section about remote Git repository management tools such as Github and Gitlab. If only because they have functionality that is not part of basic Git, such as Github Actions but also Pull Requests/Merge Requests. People confuse them with Git quite often but they're quite different.

Hi, Garret. Thanks for the kind words and taking the time to provide the thoughtful feedback. I've taken your suggestions and implemented them in this PR: https://github.com/bellanich/git-cheatsheet/pull/2
I've implemented all suggested changes, but have chosen not to do a deep-dive on remote Git repository management tools. I've introduced this concept, but kept it a high-level. This (a) keeps the cheatsheet short and (b) ensures that it doesn't become too specific to a given hosting service.


Why a button on the customViewForEmptyDataSet can not respond click event?

In  my project,i need a custom view for  the customViewForEmptyDataSet. I put a button on my custom view but it can not respond a click event.My code is here:
-(UIView *)customViewForEmptyDataSet:(UIScrollView *)scrollView
{
UIView *holderView = [[UIView alloc]initWithFrame:CGRectMake(0, 0, kFrameWidth, kFrameHeight)];
UIImageView *noDataHolder = [[UIImageView alloc]init];
noDataHolder.userInteractionEnabled = YES;
noDataHolder.image = [UIImage imageNamed:@"nodataholder"];
[holderView addSubview:noDataHolder];
[noDataHolder mas_makeConstraints:^(MASConstraintMaker *make) {
    make.width.equalTo(@250).priorityHigh();
    make.height.equalTo(@320).priorityHigh();
    make.center.equalTo(holderView);
}];

UIButton *reloadBtn = [UIButton buttonWithType:UIButtonTypeCustom];
reloadBtn.backgroundColor = kThemeColor;
reloadBtn.layer.cornerRadius = 5;
reloadBtn.titleLabel.font = [UIFont systemFontOfSize:16];
[reloadBtn setTitle:@"重试" forState:UIControlStateNormal];
[reloadBtn setTitleColor:[UIColor whiteColor] forState:UIControlStateNormal];
[reloadBtn setTitleColor:[UIColor lightGrayColor] forState:UIControlStateHighlighted];
[reloadBtn addTarget:self action:@selector(yl_reloadDataAction) forControlEvents:UIControlEventTouchUpInside];
[holderView addSubview:reloadBtn];
[reloadBtn mas_makeConstraints:^(MASConstraintMaker *make) {
    make.width.equalTo(@80).priorityHigh();
    make.height.equalTo(@35).priorityHigh();
    make.bottom.equalTo(noDataHolder).offset(-55);
    make.centerX.equalTo(noDataHolder);
}];

return holderView;

}
How can i let the reloadBtn respond a UIControlEventTouchUpInside event?

The author has fixed this issue in #119 and commit in f5e7aee5db085b8c56247cea6d69babfd4938769.
But the newest version has been neither released nor pushed to Cocoapods.

Thank you!

It is absolutely a coincidence.

Pulling the pod directly from Github by using the git option in my Podfile. It sttill doesn't work with the fix for #119. UIButton on my custom view doesn't respond.
I do it like this by the way:
- (UIView *)customViewForEmptyDataSet:(UIScrollView *)scrollView
{
    EmptyHistoryViewController *ehVC = [[EmptyHistoryViewController alloc] init];
    return ehVC.view;
}

EmptyHistoryViewController is where I initialize the button and have its IBAction. The button is animating when tapped in the view but the action is not triggered.
Also tried using - (void)emptyDataSetDidTapButton:(UIScrollView *)scrollView but no luck.
Hope you could help me out. Great job on the pod by the way. Thanks.

Running into the same issue as @dean-gr.

+1

Does it has been solved? I am still have the same issue

It's fixed in https://github.com/dzenbot/DZNEmptyDataSet/pull/347, but I don't think this is being overly maintained anymore as the last commit was 5 months ago.

+1


Development of Separate Water-Combined Solute Model

My initial results show that this model performs like ass - seems that we need a separate solute and solvent representation for water. Will do this here and just leave the commit history if we want to go back and check the shared solute representation.

On second thought... the way I was enforcing the separate water branch meant that the water biases were still impacting the trianing - going to refactor that

Performance of this model is worse on aq. and non-aq. going back to the other approach.


CDN is down

We are currently receiving a timeout when trying to download from the CDN.
URL:
https://twemoji.maxcdn.com/2/twemoji.min.js?2.7

Alright fixed! Was previously being pointed to http and now https is required.


*: fix minor typos, comments

Reference: http://goreportcard.com/report/coreos/etcd

lgtm

thanks. will merge after ci passes.


[BugFix] reset AsyncRefreshContext when alter mv refreshScheme

Fixes #26962
What type of PR is this:

[x] BugFix
[ ] Feature
[ ] Enhancement
[ ] Refactor
[ ] UT
[ ] Doc
[ ] Tool

Checklist:

[x] I have added test cases for my bug fix or my new feature
[ ] This pr will affect users' behaviors
[ ] This pr needs user documentation (for new or modified features or behaviors)

[ ] I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

[x] I have checked the version labels which the pr will be auto-backported to the target branch

[x] 3.1
[ ] 3.0
[x] 2.5
[ ] 2.4

[FE PR Coverage Check]
:heart_eyes: pass : 6 / 7 (85.71%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/alter/AlterJobMgr.java
6
7
85.71%
[568]

[FE PR Coverage Check]
:heart_eyes: pass : 6 / 7 (85.71%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/alter/AlterJobMgr.java
6
7
85.71%
[568]

[FE PR Coverage Check]
:heart_eyes: pass : 4 / 5 (80.00%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/alter/AlterJobMgr.java
4
5
80.00%
[566]

@Mergifyio backport branch-3.1

@Mergifyio backport branch-3.0


Seed selection for multi zonal shoots is not validated on shoot update

How to categorize this issue?

/area high-availability
/kind bug
What happened:
When you create a shoot with HA type zone
kind: Shoot
...
spec:
  controlPlane:
    highAvailability:
      failureTolerance:
        type: zone

it is checked if the elected seed is running in at least three zones.
However, this check does not run on shoot updates. Thus, it is possible to update a shoot to failure tolerance type zone on seeds which do not support it.
In this case shoot reconciliation fails with
task "Deploying Shoot namespace in Seed" failed: retry failed with context deadline exceeded, last error: cannot select 3 zones for shoot because seed only specifies 1 zones in its specification

What you expected to happen:
gardener-admission-controller to prevent updating a shoot to failure tolerance type zone on a seed not supporting it.
How to reproduce it (as minimally and precisely as possible):

Create a shoot and schedule it to a seed which has only one zone.
Update the shoot spec according to the example above

Anything else we need to know?:
This is the check
Environment:

Gardener version: v1.65
Kubernetes version (use kubectl version):
Cloud provider or hardware configuration:
Others:

/assign


Link to Quick Start in core repo README

When trying to run xrx for the first time I started off reading the core repo's README
The instructions I needed were on the documentation website: https://8090-inc.github.io/xrx-core/docs/quickstart

Related https://github.com/8090-inc/xrx-core/issues/7


[#109972384] Remove RDS

What
Add a PostgreSQL job to the Deployer Concourse machine and configure ATC to
use that instead of Amazon RDS. I've taken this configuration from
concourse-lite, which does the same thing:

https://github.com/concourse/concourse-lite/blob/0ca5964c0f4b0e25b03166f4dd0c403fd5c08426/templates/concourse_manifest.yml#L24-L40

We're doing this because:

RDS is incredibly slow to provision and destroy. It's not uncommon for it
to take up to 20 mins. Compiling postgresql_9.3 can still take 3mins,
but that's much faster.
we are not that concerned about the data stored in PSQL at the moment.
It contains build history and some other things that we will want to
backup in the near future, but all of the important state is in S3 and
our one worker is a single point of failure anyway.
as an aside, using RDS is more expensive and increases our integration
points, which will make things like automatically shutting down
environments at night more difficult/less effective.

NB: This change will apply on top of an existing environment but you will:

lose all pipeline configs and build history (which come from PostgreSQL).
New builds will still work after deploying the pipelines though, because
that state comes from S3.
have a short outage of the Deployer Concourse while bosh-init replaces
the instance and reattaches the disk.
possibly need to unmount the disk manually, see:
https://www.pivotaltracker.com/story/show/111149770

How to review

Deploy and run the pipelines create-deployer on your "Bootstrap Concourse" and create-microbosh on your "Deployer Concourse" (if you haven't already)
Update the create-deployer pipeline on your "Bootstrap Concourse" to use this branch: BRANCH=109972384-remove_rds ./vagrant/deploy.sh
Run the create-deployer pipeline again and wait for it finish
Login to the "Deployer Concourse" again: https://github.com/alphagov/paas-cf#login-to-deployed-concourse
Deploy the pipelines to "Deployer Concourse" again: https://github.com/alphagov/paas-cf#pipelines-create-microbosh-and-destroy-microbosh
Run the create-microbosh pipeline again and confirm that it works (should have no changes)

Who can review
Anyone but @dcarley

Tried to deploy, got an error in concourse-terraform step:
terraform-apply

Initialized blank state with remote state enabled!
Remote state configured and pulled.
There are warnings and/or errors related to your configuration. Please
fix these before continuing.

Errors:

  * 4 error(s) occurred:

* Required variable not set: vpc_id
* Required variable not set: subnet0_id
* Required variable not set: subnet1_id
* Required variable not set: subnet2_id

Tried to deploy, got an error in concourse-terraform step:

That looks like the usual race condition that should be fixed by #26.


Update to new version?

Idea
Will this image be updated to be in line with the main instance? It looks like we're behind a few releases.

@jlesage  same question, please update ?


feat(git changelog): resolve profile picture for GitHub emails

Resolve profile pictures for GitHub emails in commit authors.
Resolves #277
Based on @nekomeowww's suggestions: https://github.com/nolebase/integrations/issues/277#issuecomment-2254111802

@nekomeowww I meant to push feat(git changelog): hide "no changes" text option to a new branch and open a new PR, but I accidentally pushed it here.
Is it ok or should I revert it and open a new PR instead?

@nekomeowww I meant to push feat(git changelog): hide "no changes" text option to a new branch and open a new PR, but I accidentally pushed it here.
Is it ok or should I revert it and open a new PR instead?

I think its ok.

Many thanks for implementing my suggestion!


nixos-rebuild: -I flag is not documented in man page

Issue description
The man page of nixos-rebuild does not document the -I [path] flag.

It is documented now.

https://github.com/NixOS/nixpkgs/blob/eb3ce1cf0c5d552c237a45f9e5569cb3b15d6fa4/pkgs/os-specific/linux/nixos-rebuild/nixos-rebuild.8#L35


[now-static-build] Add VuePress optimization

Origin: https://github.com/zeit/now-builders/pull/945
Test fixtures have been changed to VuePress's docs (with modifications).

@styfle @AndyBitz I figured out why the test failed eventually.
https://github.com/zeit/now/blob/f6f99fef2529d26ab96ea4f525167d5873ec5db7/test/lib/deployment/test-deployment.js#L32
This won't match VuePress's .vuepress folder, so the deployment source files don't contain it:
My own deployment:

Test deployment:

Seems we need to modify test script to pass the test.

Codecov Report

Merging #2929 into canary will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           canary    #2929   +/-   ##
=======================================
  Coverage   13.24%   13.24%
=======================================
  Files         267      267
  Lines       10408    10408
  Branches     1172     1172
=======================================
  Hits         1379     1379
  Misses       8967     8967
  Partials       62       62

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f6f99fe...e769998. Read the comment docs.

@kidonng can you change the pattern to /**/* and see if the test pass then?

This won't match VuePress's .vuepress folder, so the deployment source files don't contain it and VuePress will use default config

What exactly does "default config" mean here? Will the site still build, but with different content?

We just cleaned up the repository by deleting all our branches and re-creating only one of them. This now allows us to create canary and stable releases without having to deal with two separate branches.
As a result, this PR was automatically closed (not manually, even though the UI indicates that).
The idea looks like a great one! If you're interested, I'd love to welcome your PR to the codebase!


Exporter Failing

Dockerfile results:
docker build -t couchbase-exporter:latest .
Sending build context to Docker daemon  476.7kB
Step 1/5 : FROM scratch
 --->
Step 2/5 : LABEL<EMAIL_ADDRESS> ---> Running in 73efb7978e42
Removing intermediate container 73efb7978e42
 ---> 3c9f5cb29c10
Step 3/5 : COPY couchbase-exporter /bin/couchbase-exporter
COPY failed: stat /var/lib/docker/tmp/docker-builder533589879/couchbase-exporter: no such file or directory

And so does make
~/couchbase-exporter$ make
jsonnet -J grafana grafana/dashboard.jsonnet -o grafana/dashboard.json
make: jsonnet: Command not found
Makefile:51: recipe for target 'grafana' failed
make: *** [grafana] Error 127

you don't need to build things by yourself, get it from the releases page, please.


Prom2: 'Out of bounds' tsdb error drops target entirely

What did you do?
I'm scraping NetData targets that give the timestamp with the metric in the response as seen below. Sometimes some metrics are not often updated (or ever after they're created) leading to out of bounds errors.
Here's an example, I've added the comment to the out of bounds metric for the purpose of this ticket.
apps_vmem_tc_qos_helper{instance="host"}<PHONE_NUMBER>245472
apps_vmem_email{instance="host"} 0<PHONE_NUMBER>668                 # this metric is out of bounds
apps_vmem_logs{instance="host"}<PHONE_NUMBER>115245472
apps_vmem_ssh{instance="host} 10027<PHONE_NUMBER>472

My settings for block durations seem to be very reasonable.
storage.tsdb.max-block-duration 	17h48m0s
storage.tsdb.min-block-duration 	2h0m0s
What did you expect to see?
I would expect that only the time series' that are out of bounds be dropped, not the target entirely. Logs only indicate this is a warning.  I assume the expected behaviour is to drop only the erroneous time series' during ingestion and append the rest that are correctly within the bounds.
So for example if a target has 3 metrics which are out of bounds, I should ingest all except those three.
What did you see instead? Under which circumstances?
All time series' from a target with any metrics that are 'out of bounds' are being dropped. So even if 1 out of 1,000 is out of bounds, all 1,000 get dropped and the target is considered 'down' due to the out of bounds errors.

Prometheus version:
I'm a little bit ahead of the 2.0.0-alpha.2 release but behind the current branch.

prometheus, version 2.0.0-alpha.2a (branch: dev-2.0, revision: dfef6cf2345465285e27802fcd24219ccbab1523)
  build user<EMAIL_ADDRESS>  build date:       20170616-16:54:53
  go version:       go1.8

Logs
Jul 03 21:15:21 ip prometheus[1829]: time="2017-07-03T21:15:21Z" level=warning msg="append failed" err="out of bounds" source="scrape.go:606" target=....

This looks like a trivial fix where we need to add a case to this switch statement. Will send a PR soon.

I've built and deployed at commit 24e9deac71d26a9db5d635df0865b83bd5ff9370 but the issue persists. I'll try to track down what seems to have been missed, but this issue is about items in the far past not the future just to be clear.
level=warning msg="append failed" err="out of bounds" source="scrape.go:626"

Fixed for good in #2906 Can you give it another try @johrstrom?

Confirmed, thank your the quick support.


[AutoPR cdn/resource-manager] Add two match condition for CDN RulesEngine.

Created to sync https://github.com/Azure/azure-rest-api-specs/pull/7367

(message created by the CI based on PR content)

This PR has been merged into https://github.com/Azure/azure-sdk-for-python/pull/7516


ssz_generic tests without ssz field

In ssz_generic/unit/unit_bounds.yaml I noticed this test:
- type: uint16
  valid: false
  value: '-1'
  tags: [atomic, uint, uint_underflow]

I'm just wondering if this test should live in the SSZ tests? It seems to test an implementation of uint16 as opposed to an SSZ implementation.
CC @protolambda

Agree there, don't know about the original rationale of the SSZ tests that were ported over. I don't think the ssz-generic is very useful at this point. When we want to ensure encoding/decoding any ssz structure, even outside of the spec, works, we'll extend the suite. For now I recommend to try and pass the the ssz-static suite (and honestly, if you pass that, chances of an edge case bug outside of the spec usage are incredibly small). Focus is on state-transition tests first :+1:

Can this issue be closed? :)

Yes, SSZ-generic was refactored and extended. But it's still experimental, as ssz has no non-spec use for clients yet. And we are really just looking to get the phase-0 types well functioning with ssz-static, to then move on to interop. Just open a new issue for anything that pops up with the newer tests, when there is a test runner in the client implemented for ssz-generic in the future.


Uncaught TypeError: Cannot read property 'elmFs' of undefined

After chrome upgrading to version 91.0.4472.77 (Official Build) (64-bit)
I got a blank screen in Chrome and seen the error Uncaught TypeError: Cannot read property 'elmFs' of undefined in console log, It works fine on firefox.

System information:

Linux 5.4.0-1048-aws x86_64

Alertmanager version:

alertmanager, version 0.21.0 (branch: HEAD, revision: 4c6c03ebfe21009c546e4d1e9b92c371d67c021d)
build user:       root@dee35927357f
build date:       20200617-08:54:02
go version:       go1.14.4

cc @w0rm any idea on this one? Thanks!
@PePoDev can you reproduce this with 0.22.2?

Hmm, from the top of my head, this could be browser extensions. @PePoDev could you try disabling extensions that could potentially modify the page?

Oh, crap!
I open in incognito mode and it's working!
After that, I try to disable the last extension that I have installed, and it's solved. Thanks.


net/http: errors from Client.Timeout triggering have .Timeout()==false

(From aaronlevy on IRC)
It's hard for a client to know when a HTTP request failed due to timeout, and when for other reasons.
Given a slow server
package main

import (
    "fmt"
    "net/http"
    "time"
)

func main() {
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        time.Sleep(3*time.Second)
        fmt.Fprintf(w, "req rcvd")
    })
    http.ListenAndServe(":8080", nil)
}

and a client
package main

import (
	"fmt"
	"io/ioutil"
	"net"
	"net/http"
	"net/url"
	"time"
)

func main() {
	client := &http.Client{
		Timeout: time.Duration(time.Second),
	}

	req, err := client.Get("http://localhost:8080")
	if err != nil {
		fmt.Printf("type: %T\n", err)
		fmt.Printf("error: %v\n", err)
		if err2, ok := err.(*url.Error); ok {
			fmt.Printf("inner type: %T\n", err2.Err)
			fmt.Printf("inner error: %v\n", err2.Err)
			if err3, ok := err2.Err.(net.Error); ok {
				fmt.Printf("is timeout: %v\n", err3.Timeout())
			}
			if err4, ok := err2.Err.(*net.OpError); ok {
				fmt.Printf("OpError inner type: %T\n", err4.Err)
				fmt.Printf("OpError inner error: %v\n", err4.Err)
			}
		}
		return
	}

	resp, err := ioutil.ReadAll(req.Body)
	req.Body.Close()
	if err != nil {
		fmt.Println(err)
	}
	fmt.Printf("%s", resp)
}

Results in
$ go run server.go &
$ go run client.go
type: *url.Error
error: Get http://localhost:8080: read tcp <IP_ADDRESS>:8080: use of closed network connection
inner type: *net.OpError
inner error: read tcp <IP_ADDRESS>:8080: use of closed network connection
is timeout: false
OpError inner type: *errors.errorString
OpError inner error: use of closed network connection

That's quite the gift wrapping on that error. But the real bug is "is timeout: false". The error was caused by a timeout, it should claim to be one.

In which version of go this got fixed?

Go 1.5 and above will contain the fix.


rbd-nbd: support and qualify volume expansion

Describe what this PR does
Qualify volume expansion via rbd-nbd

Updates the documentation
Adds required test cases

Fixes: #2234
Signed-off-by: Prasanna Kumar Kalever<EMAIL_ADDRESS>

/test ci/centos/k8s-e2e-external-storage/1.21

@Mergifyio rebase

@Mergifyio rebase

@nixpanic @Madhu-1
Don't know what changed recently:
After editing Size in PVC from 1Gi to 2Gi
I see  PVC resize has taken place:
[pkalever@localhost ~]$ kubectl get pvc
NAME      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
rbd-pvc   Bound    pvc-47136184-ac0f-4d4f-ab4c-262156e6748d   2Gi        RWO            csi-rbd-sc     4m31s

The nodeplugin logs say the filesystem resize succeeded.
$ kubectl logs csi-rbdplugin-8c27l  csi-rbdplugin
I0720 08:19:29.435620   20628 utils.go:175] ID: 19 Req-ID: 0001-0024-fed5480a-f00f-417a-a51d-31d8a8144c03-0000000000000003-b8d95f12-e932-11eb-8820-f6bb186e614b GRPC request: {"capacity_range":{"required_bytes":2147483648},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-47136184-ac0f-4d4f-ab4c-262156e6748d/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4","mount_flags":["discard"]}},"access_mode":{"mode":1}},"volume_id":"0001-0024-fed5480a-f00f-417a-a51d-31d8a8144c03-0000000000000003-b8d95f12-e932-11eb-8820-f6bb186e614b","volume_path":"/var/lib/kubelet/pods/21e2a485-d62f-431e-96df-bed21f371b55/volumes/kubernetes.io~csi/pvc-47136184-ac0f-4d4f-ab4c-262156e6748d/mount"}
I0720 08:19:30.068493   20628 cephcmds.go:59] ID: 19 Req-ID: 0001-0024-fed5480a-f00f-417a-a51d-31d8a8144c03-0000000000000003-b8d95f12-e932-11eb-8820-f6bb186e614b command succeeded: rbd [device list --format=json --device-type nbd]
I0720 08:19:30.069802   20628 mount_linux.go:446] Attempting to determine if disk "/dev/nbd0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/nbd0])
I0720 08:19:30.223319   20628 mount_linux.go:449] Output: "DEVNAME=/dev/nbd0\nTYPE=ext4\n"
I0720 08:19:30.223676   20628 resizefs_linux.go:55] ResizeFS.Resize - Expanding mounted volume /dev/nbd0
I0720 08:19:30.240510   20628 resizefs_linux.go:70] Device /dev/nbd0 resized successfully

But, on mount size is still the same.
$ kubectl exec -it pod/csi-rbd-demo-pod -- df -Th
Filesystem     Type     Size  Used Avail Use% Mounted on
overlay        overlay   17G  3.1G   13G  20% /
tmpfs          tmpfs     64M     0   64M   0% /dev
tmpfs          tmpfs    5.9G     0  5.9G   0% /sys/fs/cgroup
/dev/vda1      ext4      17G  3.1G   13G  20% /etc/hosts
shm            tmpfs     64M     0   64M   0% /dev/shm
/dev/nbd0      ext4     976M  2.6M  958M   1% /var/lib/www/html
tmpfs          tmpfs    5.9G   12K  5.9G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs          tmpfs    5.9G     0  5.9G   0% /proc/acpi
tmpfs          tmpfs    5.9G     0  5.9G   0% /proc/scsi
tmpfs          tmpfs    5.9G     0  5.9G   0% /sys/firmware

$ kubectl logs csi-rbdplugin-provisioner-7f4d6599bc-2q9qg csi-resizer
I0720 08:18:25.975018       1 controller.go:291] Started PVC processing "default/rbd-pvc"
I0720 08:18:26.000066       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"rbd-pvc", UID:"47136184-ac0f-4d4f-ab4c-262156e6748d", APIVersion:"v1", ResourceVersion:"2642", FieldPath:""}): t
ype: 'Normal' reason: 'Resizing' External resizer is resizing volume pvc-47136184-ac0f-4d4f-ab4c-262156e6748d
I0720 08:18:26.013407       1 connection.go:182] GRPC call: /csi.v1.Controller/ControllerExpandVolume
I0720 08:18:26.013528       1 connection.go:183] GRPC request: {"capacity_range":{"required_bytes":2147483648},"secrets":"***stripped***","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4","mount_flags":["discard"]}},"access_mod
e":{"mode":1}},"volume_id":"0001-0024-fed5480a-f00f-417a-a51d-31d8a8144c03-0000000000000003-b8d95f12-e932-11eb-8820-f6bb186e614b"}
I0720 08:18:26.505896       1 connection.go:185] GRPC response: {"capacity_bytes":2147483648,"node_expansion_required":true}
I0720 08:18:26.506665       1 connection.go:186] GRPC error: <nil>
I0720 08:18:26.506692       1 controller.go:468] Resize volume succeeded for volume "pvc-47136184-ac0f-4d4f-ab4c-262156e6748d", start to update PV's capacity
I0720 08:18:26.507041       1 controller.go:570] Resize volume succeeded for volume "pvc-47136184-ac0f-4d4f-ab4c-262156e6748d", start to update PV's capacity
I0720 08:18:26.539982       1 controller.go:474] Update capacity of PV "pvc-47136184-ac0f-4d4f-ab4c-262156e6748d" to 2Gi succeeded
I0720 08:18:26.551049       1 controller.go:496] Mark PVC "default/rbd-pvc" as file system resize required
I0720 08:18:26.551389       1 controller.go:291] Started PVC processing "default/rbd-pvc"
I0720 08:18:26.551407       1 controller.go:338] No need to resize PV "pvc-47136184-ac0f-4d4f-ab4c-262156e6748d"
I0720 08:18:26.551639       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"rbd-pvc", UID:"47136184-ac0f-4d4f-ab4c-262156e6748d", APIVersion:"v1", ResourceVersion:"2642", FieldPath:""}): t
ype: 'Normal' reason: 'FileSystemResizeRequired' Require file system resize of volume on node
I0720 08:18:26.553079       1 controller.go:291] Started PVC processing "default/rbd-pvc"
I0720 08:18:26.553325       1 controller.go:338] No need to resize PV "pvc-47136184-ac0f-4d4f-ab4c-262156e6748d"
I0720 08:18:29.563915       1 leaderelection.go:273] successfully renewed lease default/external-resizer-rbd-csi-ceph-com
I0720 08:18:34.581093       1 leaderelection.go:273] successfully renewed lease default/external-resizer-rbd-csi-ceph-com
I0720 08:18:39.599399       1 leaderelection.go:273] successfully renewed lease default/external-resizer-rbd-csi-ceph-com
I0720 08:18:44.619997       1 leaderelection.go:273] successfully renewed lease default/external-resizer-rbd-csi-ceph-com
I0720 08:18:49.672947       1 leaderelection.go:273] successfully renewed lease default/external-resizer-rbd-csi-ceph-com
I0720 08:18:54.684983       1 leaderelection.go:273] successfully renewed lease default/external-resizer-rbd-csi-ceph-com
I0720 08:18:59.714325       1 leaderelection.go:273] successfully renewed lease default/external-resizer-rbd-csi-ceph-com
I0720 08:19:04.734539       1 leaderelection.go:273] successfully renewed lease default/external-resizer-rbd-csi-ceph-com
I0720 08:19:09.752926       1 leaderelection.go:273] successfully renewed lease default/external-resizer-rbd-csi-ceph-com
I0720 08:19:14.784085       1 leaderelection.go:273] successfully renewed lease default/external-resizer-rbd-csi-ceph-com
I0720 08:19:19.812873       1 leaderelection.go:273] successfully renewed lease default/external-resizer-rbd-csi-ceph-com
I0720 08:19:24.836474       1 leaderelection.go:273] successfully renewed lease default/external-resizer-rbd-csi-ceph-com
I0720 08:19:29.850478       1 leaderelection.go:273] successfully renewed lease default/external-resizer-rbd-csi-ceph-com
I0720 08:19:30.268306       1 controller.go:291] Started PVC processing "default/rbd-pvc"
I0720 08:19:30.268515       1 controller.go:334] No need to resize PVC "default/rbd-pvc"

I'm pretty sure, previously the e2e worked pretty well, but not sure after rebase I'm seeing this :-(
Did anything ring the bell?

Some clue in the pvc describe:
[pkalever@localhost ~]$ kubectl describe pvc/rbd-pvc
Name:          rbd-pvc
Namespace:     default
StorageClass:  csi-rbd-sc
Status:        Bound
Volume:        pvc-47136184-ac0f-4d4f-ab4c-262156e6748d
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: rbd.csi.ceph.com
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      2Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Used By:       csi-rbd-demo-pod
Events:
  Type     Reason                      Age   From                                                                                Message
  ----     ------                      ----  ----                                                                                -------
  Normal   Provisioning                36m   rbd.csi.ceph.com_csi-rbdplugin-provisioner-7f4d6599bc-2q9qg_dbbd03b9-1d39-4d4a-a41e-8de6ec5b06a3  External provisioner is provisioning volume for claim "default/rbd-pvc"
  Normal   ExternalProvisioning        36m   persistentvolume-controller                                                                       waiting for a volume to be created, either by external provisioner "rbd.csi.ceph.com" or manual
ly created by system administrator
  Normal   ProvisioningSucceeded       36m   rbd.csi.ceph.com_csi-rbdplugin-provisioner-7f4d6599bc-2q9qg_dbbd03b9-1d39-4d4a-a41e-8de6ec5b06a3  Successfully provisioned volume pvc-47136184-ac0f-4d4f-ab4c-262156e6748d
  Warning  ExternalExpanding           34m   volume_expand                                                                                Ignoring the PVC: didn't find a plugin capable of expanding the volume; waiting for an external
 controller to process this PVC.
  Normal   Resizing                    34m   external-resizer rbd.csi.ceph.com                                                                 External resizer is resizing volume pvc-47136184-ac0f-4d4f-ab4c-262156e6748d
  Normal   FileSystemResizeRequired    34m   external-resizer rbd.csi.ceph.com                                                                 Require file system resize of volume on node
  Normal   FileSystemResizeSuccessful  33m   kubelet                                                                                MountVolume.NodeExpandVolume succeeded for volume "pvc-47136184-ac0f-4d4f-ab4c-262156e6748d"

Well the issue is with the kernel version in the minikube VM
minikube v1.22.0 has linux kernel 4.19.182, while we need at least 5.3.0
[pkalever@localhost ceph-csi]$ minikube version
minikube version: v1.22.0
commit: a03fbcf166e6f74ef224d4a63be4277d017bb62e
[pkalever@localhost ceph-csi]$ minikube ssh
                         _             _
            _         _ ( )           ( )
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

$ uname -a
Linux minikube 4.19.182 #1 SMP Fri Jul 2 00:45:17 UTC 2021 x86_64 GNU/Linux

i.e. we need below changes for rbd-nbd to support resize with the latest netlink
commit 4ddeaae8903d703201e493e2d19dc9ac9acf2c76
Author: Mike Christie<EMAIL_ADDRESS>Date:   Wed May 29 15:16:06 2019 -0500

    nbd: add netlink reconfigure resize support

    If the device is setup with ioctl we can resize the device after the
    initial setup, but if the device is setup with netlink we cannot use the
    resize related ioctls and there is no netlink reconfigure size ATTR
    handling code.

    This patch adds netlink reconfigure resize support to match the ioctl
    interface.

    Reviewed-by: Josef Bacik<EMAIL_ADDRESS>    Signed-off-by: Mike Christie<EMAIL_ADDRESS>    Signed-off-by: Jens Axboe<EMAIL_ADDRESS>

The above commit is part of kernel version >= v5.3
I will add a kernel version defense check for this e2e test, and respin this series.

/test ci/centos/mini-e2e-helm/k8s-1.19

@Mergifyio rebase

@nixpanic @Madhu-1 request for reveiw.

@pkalever golangci lint failure

@pkalever golangci lint failure

Probably (rebase) those new linters, I will fix them up. Thanks!

@pkalever linter is still failing.

@Madhu-1 what is this now :-)
e2e/rbd_helper.go:31: File is not `gofumpt`-ed (gofumpt)

Locally I don't hit this!

@Madhu-1 what is this now :-)
e2e/rbd_helper.go:31: File is not `gofumpt`-ed (gofumpt)

Locally I don't hit this!

-var (
-       // nolint:gomnd // numbers specify Kernel versions.
-       nbdResizeSupport = []util.KernelVersion{
-               {
-                       Version:      5,
-                       PatchLevel:   3,
-                       SubLevel:     0,
-                       ExtraVersion: 0,
-                       Distribution: "",
-                       Backport:     false,
-               }, // standard 5.3+ versions
-       }
-)
+// nolint:gomnd // numbers specify Kernel versions.
+var nbdResizeSupport = []util.KernelVersion{
+       {
+               Version:      5,
+               PatchLevel:   3,
+               SubLevel:     0,
+               ExtraVersion: 0,
+               Distribution: "",
+               Backport:     false,
+       }, // standard 5.3+ versions
+}

Running gofumpt -w e2e/rbd_helper.go should fix it automatically

Running gofumpt -w e2e/rbd_helper.go should fix it automatically

I had to update the local go packages versions to get make some difference.

@mergifyio rebase

/test ci/centos/mini-e2e/k8s-1.21
Hit readonly error: updated https://github.com/ceph/ceph-csi/issues/2204#issuecomment-895879314

/retest ci/centos/mini-e2e/k8s-1.21

/retest ci/centos/mini-e2e/k8s-1.21

/retest ci/centos/upgrade-tests-cephfs


RuntimeError: probability tensor contains either inf, nan or element < 0

Hi, thanks for this impressive project.
I use vllm, which works well for llama quantized by awq. However, when I changed to falcon-40b (awq), it did not work.
The version is the newest, and I serve this model online. I tried to use the hf transformer to generate a response, which performed normally without any error.
2023-12-12 19:38:39 | ERROR | stderr | Traceback (most recent call last):
2023-12-12 19:38:39 | ERROR | stderr |   File "/root/miniconda3/envs/fastchat/lib/python3.9/runpy.py", line 197, in _run_module_as_main
2023-12-12 19:38:39 | ERROR | stderr |     return _run_code(code, main_globals, None,
2023-12-12 19:38:39 | ERROR | stderr |   File "/root/miniconda3/envs/fastchat/lib/python3.9/runpy.py", line 87, in _run_code
2023-12-12 19:38:39 | ERROR | stderr |     exec(code, run_globals)
2023-12-12 19:38:39 | ERROR | stderr |   File "/root/miniconda3/envs/fastchat/lib/python3.9/site-packages/fastchat/serve/vllm_worker.py", line 243, in <module>
2023-12-12 19:38:39 | ERROR | stderr |     engine = AsyncLLMEngine.from_engine_args(engine_args)
2023-12-12 19:38:39 | ERROR | stderr |   File "/root/miniconda3/envs/fastchat/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py", line 495, in from_engine_args
2023-12-12 19:38:39 | ERROR | stderr |     engine = cls(parallel_config.worker_use_ray,
2023-12-12 19:38:39 | ERROR | stderr |   File "/root/miniconda3/envs/fastchat/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py", line 269, in __init__
2023-12-12 19:38:39 | ERROR | stderr |     self.engine = self._init_engine(*args, **kwargs)
2023-12-12 19:38:39 | ERROR | stderr |   File "/root/miniconda3/envs/fastchat/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py", line 314, in _init_engine
2023-12-12 19:38:39 | ERROR | stderr |     return engine_class(*args, **kwargs)
2023-12-12 19:38:39 | ERROR | stderr |   File "/root/miniconda3/envs/fastchat/lib/python3.9/site-packages/vllm/engine/llm_engine.py", line 112, in __init__
2023-12-12 19:38:39 | ERROR | stderr |     self._init_cache()
2023-12-12 19:38:39 | ERROR | stderr |   File "/root/miniconda3/envs/fastchat/lib/python3.9/site-packages/vllm/engine/llm_engine.py", line 208, in _init_cache
2023-12-12 19:38:39 | ERROR | stderr |     num_blocks = self._run_workers(
2023-12-12 19:38:39 | ERROR | stderr |   File "/root/miniconda3/envs/fastchat/lib/python3.9/site-packages/vllm/engine/llm_engine.py", line 750, in _run_workers
2023-12-12 19:38:39 | ERROR | stderr |     self._run_workers_in_batch(workers, method, *args, **kwargs))
2023-12-12 19:38:39 | ERROR | stderr |   File "/root/miniconda3/envs/fastchat/lib/python3.9/site-packages/vllm/engine/llm_engine.py", line 724, in _run_workers_in_batch
2023-12-12 19:38:39 | ERROR | stderr |     output = executor(*args, **kwargs)
2023-12-12 19:38:39 | ERROR | stderr |   File "/root/miniconda3/envs/fastchat/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2023-12-12 19:38:39 | ERROR | stderr |     return func(*args, **kwargs)
2023-12-12 19:38:39 | ERROR | stderr |   File "/root/miniconda3/envs/fastchat/lib/python3.9/site-packages/vllm/worker/worker.py", line 88, in profile_num_available_blocks
2023-12-12 19:38:39 | ERROR | stderr |     self.model_runner.profile_run()
2023-12-12 19:38:39 | ERROR | stderr |   File "/root/miniconda3/envs/fastchat/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2023-12-12 19:38:39 | ERROR | stderr |     return func(*args, **kwargs)
2023-12-12 19:38:39 | ERROR | stderr |   File "/root/miniconda3/envs/fastchat/lib/python3.9/site-packages/vllm/worker/model_runner.py", line 321, in profile_run
2023-12-12 19:38:39 | ERROR | stderr |     self.execute_model(seqs, kv_caches)
2023-12-12 19:38:39 | ERROR | stderr |   File "/root/miniconda3/envs/fastchat/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2023-12-12 19:38:39 | ERROR | stderr |     return func(*args, **kwargs)
2023-12-12 19:38:39 | ERROR | stderr |   File "/root/miniconda3/envs/fastchat/lib/python3.9/site-packages/vllm/worker/model_runner.py", line 288, in execute_model
2023-12-12 19:38:39 | ERROR | stderr |     output = self.model.sample(
2023-12-12 19:38:39 | ERROR | stderr |   File "/root/miniconda3/envs/fastchat/lib/python3.9/site-packages/vllm/model_executor/models/falcon.py", line 408, in sample
2023-12-12 19:38:39 | ERROR | stderr |     next_tokens = self.sampler(self.lm_head.weight, hidden_states,
2023-12-12 19:38:39 | ERROR | stderr |   File "/root/miniconda3/envs/fastchat/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-12-12 19:38:39 | ERROR | stderr |     return self._call_impl(*args, **kwargs)
2023-12-12 19:38:39 | ERROR | stderr |   File "/root/miniconda3/envs/fastchat/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-12-12 19:38:39 | ERROR | stderr |     return forward_call(*args, **kwargs)
2023-12-12 19:38:39 | ERROR | stderr |   File "/root/miniconda3/envs/fastchat/lib/python3.9/site-packages/vllm/model_executor/layers/sampler.py", line 93, in forward
2023-12-12 19:38:39 | ERROR | stderr |     sample_results = _sample(probs, logprobs, sampling_metadata)
2023-12-12 19:38:39 | ERROR | stderr |   File "/root/miniconda3/envs/fastchat/lib/python3.9/site-packages/vllm/model_executor/layers/sampler.py", line 501, in _sample
2023-12-12 19:38:39 | ERROR | stderr |     sample_results = _random_sample(seq_groups, is_prompts,
2023-12-12 19:38:39 | ERROR | stderr |   File "/root/miniconda3/envs/fastchat/lib/python3.9/site-packages/vllm/model_executor/layers/sampler.py", line 396, in _random_sample
2023-12-12 19:38:39 | ERROR | stderr |     random_samples = torch.multinomial(probs,
2023-12-12 19:38:39 | ERROR | stderr | RuntimeError: probability tensor contains either `inf`, `nan` or element < 0

I meet this error, when I use Qwen-7b-chat
Error log is:
Traceback (most recent call last):
File "/workspace/QwenChat-7b/./atom/1/model_copy.py", line 920, in vllm_response_thread
async for request_output in results_generator:
File "/usr/local/lib/python3.8/dist-packages/vllm/engine/async_llm_engine.py", line 151, in generate
raise e
File "/usr/local/lib/python3.8/dist-packages/vllm/engine/async_llm_engine.py", line 148, in generate
await self.engine_step(request_id)
File "/usr/local/lib/python3.8/dist-packages/vllm/engine/async_llm_engine.py", line 74, in engine_step
request_outputs = self.engine.step()
File "/usr/local/lib/python3.8/dist-packages/vllm/engine/llm_engine.py", line 313, in step
output = self._run_workers(
File "/usr/local/lib/python3.8/dist-packages/vllm/engine/llm_engine.py", line 470, in _run_workers
output = executor(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/vllm/worker/worker.py", line 293, in execute_model
output = self.model(
File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/vllm/model_executor/models/qwen.py", line 240, in forward
next_tokens = self.sampler(self.lm_head.weight, hidden_states,
File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/vllm/model_executor/layers/sampler.py", line 89, in forward
return _sample(probs, logprobs, input_metadata)
File "/usr/local/lib/python3.8/dist-packages/vllm/model_executor/layers/sampler.py", line 391, in _sample
next_token_ids = _sample_from_prompt(prob, sampling_params)
File "/usr/local/lib/python3.8/dist-packages/vllm/model_executor/layers/sampler.py", line 305, in _sample_from_prompt
next_token_ids = torch.multinomial(prob,
RuntimeError: probability tensor contains either inf, nan or element < 0
vllm==0.23

use beamserch


feat: add testnet reserved channel

Adds testnet reserved channel which points to channel-fuel-testnet.toml on gh-pages of this repo.

This is breaking as users might already have a channel called testnet


gammaln documentation is misleading

The claim that gammaln computes "Logarithm of absolute value of gamma function" is not true:
>>> import numpy as np
>>> from scipy.special import gamma, gammaln
>>> z = 1 + 1j
>>> gammaln(z)
(-0.65092319930185738-0.30164032046753375j)
>>> np.log(np.abs(gamma(z)))
-0.65092319930185738
>>> np.log(gamma(z))
(-0.65092319930185749-0.30164032046753381j)

So gammaln actually computes ln(gamma(z)) which is more useful than what it announces. Better to correct documentation, so it won't scare away people.

:+1:
Note that the behavior of gammaln depends on whether the argument is complex or real:
In [36]: gammaln(-0.9)
Out[36]: 2.3580731673920328

In [37]: gammaln(-0.9 + 0j)
Out[37]: (2.3580731673920319-3.1415926535897931j)

In [38]: gammaln(-1)
Out[38]: inf

In [39]: gammaln(-1 + 0j)
Out[39]: (1.0000000000000001e+300+0j)

Note that the claim about the absolute value is true for real inputs:
In [29]: x = -2.3

In [30]: gammaln(x)
Out[30]: 0.3695666634550081

In [31]: np.log(np.abs(gamma(x)))
Out[31]: 0.3695666634550081

In [32]: np.log(gamma(x))
/Users/warren/anaconda/bin/ipython:1: RuntimeWarning: invalid value encountered in log
  #!/bin/bash /Users/warren/anaconda/bin/python.app
Out[32]: nan

That's a good investigation. Now it's clear what should be in the doc.


Paper recurrence

Hi,Thank you for your work and open source tools.I am recreating the work in your thesis. I don't know if it is because of the seed that I didn't reach the target for a long time. So can you make the target program and seed of your experiment public?

Closing this. Feel free to re-open if you still need the setup.