Add NWBIO

This PR adds NWBIO, an IO module for reading and writing the Neurodata Without Borders format (cf #221).
This is a work in progress, which doesn't support all possible contents of an NWB file, but I think it is good enough for people to try it, and let us know (by commenting here) what doesn't work, or could be done better. Many thanks to @legouee for her work on this.
TODO

[x] write Block, Segment, Epoch, Event, AnalogSignal, IrregularlySampledSignal, SpikeTrain using base TimeSeries
[ ] write Unit, ChannelIndex (or whatever replaces them)
[ ] write ImageSequence and related
[x] allow user to choose where to write data (e.g. "acquisition", "stimulus", "analysis") by providing guiding annotations
[x] allow user to write subclasses of TimeSeries by providing guiding annotations
[x] read "acquisition" and "stimulus" groups
[ ] read metadata associated with processing modules and make it available through annotations
[ ] support array annotations

Usage example: https://github.com/apdavison/python-neo/blob/nwb2/examples/NWB-Allen-Institute-Example.ipynb

Hello @apdavison! Thanks for opening this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file neo/io/nwbio.py:

Line 12:1: W191 indentation contains tabs
Line 12:2: E101 indentation contains mixed spaces and tabs
Line 13:1: E101 indentation contains mixed spaces and tabs
Line 15:100: E501 line too long (121 > 99 characters)
Line 51:100: E501 line too long (108 > 99 characters)
Line 53:93: E261 at least two spaces before inline comment
Line 53:100: E501 line too long (119 > 99 characters)
Line 63:27: E126 continuation line over-indented for hanging indent
Line 96:21: E221 multiple spaces before operator
Line 128:54: E261 at least two spaces before inline comment
Line 139:100: E501 line too long (105 > 99 characters)
Line 142:100: E501 line too long (105 > 99 characters)
Line 144:100: E501 line too long (104 > 99 characters)
Line 184:100: E501 line too long (100 > 99 characters)
Line 229:39: E203 whitespace before ':'
Line 294:100: E501 line too long (120 > 99 characters)
Line 295:100: E501 line too long (110 > 99 characters)
Line 310:9: E265 block comment should start with '# '
Line 311:100: E501 line too long (105 > 99 characters)
Line 312:100: E501 line too long (101 > 99 characters)
Line 315:9: E265 block comment should start with '# '
Line 316:100: E501 line too long (101 > 99 characters)
Line 341:100: E501 line too long (100 > 99 characters)
Line 374:29: E116 unexpected indentation (comment)
Line 382:100: E501 line too long (136 > 99 characters)
Line 391:26: E265 block comment should start with '# '
Line 402:25: E126 continuation line over-indented for hanging indent
Line 429:5: E306 expected 1 blank line before a nested definition, found 0
Line 467:40: E203 whitespace before ':'
Line 486:100: E501 line too long (110 > 99 characters)
Line 493:25: E126 continuation line over-indented for hanging indent
Line 505:25: E126 continuation line over-indented for hanging indent
Line 520:40: E203 whitespace before ':'
Line 582:23: E241 multiple spaces after ','
Line 611:21: E126 continuation line over-indented for hanging indent
Line 614:21: E265 block comment should start with '# '
Line 616:21: E265 block comment should start with '# '
Line 617:21: E265 block comment should start with '# '
Line 619:21: E265 block comment should start with '# '
Line 620:21: E265 block comment should start with '# '
Line 621:21: E265 block comment should start with '# '
Line 622:40: W292 no newline at end of file

In the file neo/io/proxyobjects.py:

Line 231:100: E501 line too long (106 > 99 characters)

In the file neo/test/iotest/test_nwbio.py:

Line 14:100: E501 line too long (136 > 99 characters)
Line 31:24: E222 multiple spaces after operator
Line 32:1: E122 continuation line missing indentation or outdented
Line 33:9: E265 block comment should start with '# '
Line 33:100: E501 line too long (109 > 99 characters)
Line 34:100: E501 line too long (108 > 99 characters)
Line 58:20: E261 at least two spaces before inline comment
Line 59:21: E261 at least two spaces before inline comment
Line 63:39: E261 at least two spaces before inline comment
Line 68:37: E261 at least two spaces before inline comment
Line 82:60: E226 missing whitespace around arithmetic operator
Line 83:74: E226 missing whitespace around arithmetic operator
Line 187:20: W292 no newline at end of file

I tried this and got an error. Am I doing something wrong?
url = "http://download.alleninstitute.org/informatics-archive/prerelease/H<IP_ADDRESS>.05-2.nwb"
local_filename = url.split("/")[-1]
if not exists(local_filename):
    local_filename, headers = urlretrieve(url)

ior = NWBIO(local_filename)
blocks = ior.read(lazy=True)

iow = NWBIO(filename='neo_test.nwb', mode='w')
iow.write_all_blocks(blocks)

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-16-c1c2eb01f5f2> in <module>
      1 test_file_name = 'neo_test.nwb'
      2 iow = NWBIO(filename=test_file_name, mode='w')
----> 3 iow.write_all_blocks(blocks)

~/dev/python-neo/neo/io/nwbio.py in write_all_blocks(self, blocks, **kwargs)
    290                 for block in blocks:
    291                     if annotation_name in block.annotations:
--> 292                         annotations[annotation_name].add(block.annotations[annotation_name])
    293                 if annotation_name in annotations:
    294                     if len(annotations[annotation_name]) > 1:

TypeError: unhashable type: 'dict'

Also, more broadly, let's think about the scope. It looks like NEO handles both intracellular electrophysiology and extracellular electrophysiology. In NWB, these different types of experiments have very different structures of metadata. The demo and tests seem to be focused on intracellular electrophysiology. Is that the goal for this, or do you also want to include extracellular?

@bendichter : neo object are totaly agnostic of intra/extra/imaging electrophysology.
At writting time if it matter for NWB maybe we could use some annotation as a hint for the writter to use the good structure in the file.
I did not work at all on this IO so I can't tell.

The two I see for this "routing" of AnalogSignal inside NWB is or/and:

some extra kwargs :  in write_block(block, sig_type='electrical_series')
use neo annotations at object level (aka metadata) anasig.annotate(sig_type='electrical_series')

Yes, that would put it in the right place, but it would still be missing some of the essential metadata. For instance, an ElectricalSeries requires a vector reference to rows of the electrodes table. The electrodes table stores information about the electrodes like position, device, filtering, impedance, etc. For a TwoPhotonSeries you need input what the excitations and emission lambdas are, the spacing of the image, information about the acquisition device, etc.
This could all be done with annotations, but they would need to be pretty sophisticated and have a specific structure. I think we should have a convo about this

Thanks for working on this, @apdavison. After your recent changes, I am running into this new error:
ior = NWBIO(local_filename)
blocks = ior.read(lazy=True)

iow = NWBIO(filename='neo_test.nwb', mode='w')
iow.write_all_blocks(blocks)

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-2-3a8932303c81> in <module>
      5
      6 iow = NWBIO(filename='neo_test.nwb', mode='w')
----> 7 iow.write_all_blocks(blocks)

~/dev/python-neo/neo/io/nwbio.py in write_all_blocks(self, blocks, **kwargs)
    337
    338         for i, block in enumerate(blocks):
--> 339             self.write_block(nwbfile, block)
    340         io_nwb.write(nwbfile)
    341         io_nwb.close()

~/dev/python-neo/neo/io/nwbio.py in write_block(self, nwbfile, block, **kwargs)
    352             if not segment.name:
    353                 segment.name = "%s : segment%d" % (block.name, i)
--> 354             self._write_segment(nwbfile, segment)
    355         self.blocks_written += 1
    356

~/dev/python-neo/neo/io/nwbio.py in _write_segment(self, nwbfile, segment)
    362             if not signal.name:
    363                 signal.name = "%s : analogsignal%d" % (segment.name, i)
--> 364             self._write_signal(nwbfile, signal)
    365
    366         for i, train in enumerate(segment.spiketrains):

~/dev/python-neo/neo/io/nwbio.py in _write_signal(self, nwbfile, signal)
    400         else:
    401             raise TypeError("signal has type {0}, should be AnalogSignal or IrregularlySampledSignal".format(
--> 402                 signal.__class__.__name__))
    403         nwbfile.add_acquisition(tS)
    404         return tS

TypeError: signal has type AnalogSignalProxy, should be AnalogSignal or IrregularlySampledSignal

Hi @bendichter, we haven't yet implemented support for lazy objects. If you run
blocks = ior.read(lazy=False)
it should now work (with the latest version of the nwb2 branch).

This could all be done with annotations, but they would need to be pretty sophisticated and have a specific structure.

This is exactly the plan for properly supporting NWB (see the TODO list at the top of the PR). I don't really see any other way of doing it, although I suspect we will end up adding some utility functions to simplify the process of providing the correct annotations when creating the Neo structure.
In general terms, my proposed roadmap, which Elodie and I will be working on (together with anyone else who would like to help out!) is:

continue trying to round-trip (read and then write again) a wide range of NWB files; we will add annotations at the read stage, and then use those annotations at the write stage. The same annotations could be used by someone creating a Neo structure from scratch. It would be a great help if @bendichter could suggest some suitable NWB files, with a range of sophistication in their content, for such round-trip testing.
once we've proved the concept works with test files, we'll then systematically complete the annotation handling with reference to the NWB spec (hopefully we'll also be able to handle NWB extensions).
document the annotations, and possibly implement helper/utility functions to make adding these annotations easier for common use cases.

@apdavison OK, but handling the full range of NWB files will be quite a lot of work with this approach. Let's focus on intracellular electrophysiology for now, because I think that will be the easiest and you've already gotten a good start.
With your latest changes the code runs! Some discrepancies I'm finding in the round trip:

the original has a structure /general/intracellular_ephys/sweep_table which matches stimuli to response. This is essential, because it's the only way we know which stimuli go with which responses. In this particular case, you could use the names of the stimuli and the responses, but that's not generally true.

/general/intracellular_ephys also contains IntracellularElectrode objects named "Electrode 0" and "Electrode 1".

minor:

it appears the round trip creates a /units group that doesn't appear to contain any information.

an empty "epochs" object was created as well.

Thanks for the feedback @bendichter.
@legouee is going to pick this up now, starting with the the specialized time series ("acquisition" should contain VoltageClampSeries, not plain TimeSeries), and then moving on to the other discrepancies until the output file is identical to the input.
I agree it makes sense to focus on perfecting intracellular ephys before moving on to more complex use cases.

@apdavison I'd like to work on this for extracellular electrophysiology, starting to think about writing.
ecephys requires an electrodes table that has information about each electrode, including:

position (x,y,z) in the brain
imp (impedance)
location (brain region)
filtering
group (which is a pynwb.ecephys.ElectrodeGroup object).

The signal is in an ElectricalSeries object. ElectricalSeries.electrodes is aDynamicTableRegion object that points to a region of the electrodes table. Most of the time this region is the entire table, and we can probably assume that by default. That is generally broken when 1) some electrodes aren't recorded from or 2) when there are multiple groups of electrodes that have different metadata, like starting time.
I think we'd need some way to specify these electrodes. The only way I can think to do this is to specify them in the annotations of the AnalogSignal that will become the ElectricalSeries. My preference would be to just hold within AnalogSignal.annotations the indices that form the view of the electrodes table, and to specify the data that goes into this table elsewhere, but I am not sure where that would go, or how it would be expressed.

There are many things still to improve, but I think NWBIO is now suitable for intracellular electrophysiology data. I think this can be merged now, and future improvements can be made in separate PRs.

@apdavison @legouee I think we can merge this PR soon to have a first version of the NWBIO in the next release. The most important change requests for this would be the incorporation of some testfiles also on gin (if you have some, as we can then tag the release to a fixed version of the test files), the incorporation of the BaseTests and some completion that is still lacking on the docstrings. Also I am not sure about some of the discussions in this thread form the end of last year, if there are outdated, could you resolve them?

@legouee @apdavison The test file is now on gin. Can you include it in the set of files used by the unittests and also include the common rawio tests by inheriting from BaseTestIO?

@JuliaSprenger : since it now use BaseTestIO for tests, I think we can merge no ?

There are some warning in test_nwb.py
I guess we can live with then for a while.

@apdavison @legouee I cleaned all open comments, moved unresolved ones to #1013 for the future. I fixed the layout to be pep8 compatible and fixed some docstrings / typos. Under the assumption that you don't plan to include https://github.com/apdavison/python-neo/pull/6 here any more I would be ok to merge the current version and fix the remaining issues listed in #1013 in the next release.


Add log compaction (snapshot of committed logs on any node)

Allow nodes to make a snapshot of current state, and remove committed log entries leading up to that state. Snapshot procedure described in Section 7 of the extended Raft paper: https://raft.github.io/raft.pdf
This can be done safely, independent of leadership because it is not possible for prior entries to be invalidated after being committed.

The solution used for doing this in https://github.com/logcabin/logcabin (the C++ reference implementation of Raft) uses system fork to create a child process with identical memory to the parent and write this to disk while the parent process rewrites the truncated log and other metadata and proceeds. This does not incur a performance penalty because of copy-on-write.
Golang's runtime relies on threads that are lost in fork, so the same strategy can not be simply dropped in (for details: https://github.com/golang/go/issues/227). Need to do some experimentation to figure out how long it would take to write a large amount of data to disk, and if it's viable to simply use a mutex to ensure that the database is not written during a snapshot. This is probably relatively viable, given that we can decide when to do the snapshot and tune it such that it is unlikely to interrupt other operations.

After more benchmarking, writing a 3.7Gb file takes >12s, so mutex is not viable for a database of any realistic size.

Rather than using fork, now that Database uses an immutable underlying store, the snapshot can be implemented by cloning the Database (add a method that makes a new instance with the underlying tree at that time). The server is able to move on with writes to the main instance of the database, but the clone will not receive the updates and its contents can be persisted in a goroutine.


sbg_driver: 3.2.0-1 in 'rolling/distribution.yaml' [bloom]

Increasing version of package(s) in repository sbg_driver to 3.2.0-1:

upstream repository: https://github.com/SBG-Systems/sbg_ros2.git
release repository: https://github.com/SBG-Systems/sbg_ros2-release.git
distro file: rolling/distribution.yaml
bloom version: 0.12.0
previous version for package: null

sbg_driver
* Update README with ROS2 commands to launch magnetic calibration
* Fix segfault when running magnetic calibration
* Removed unused flags from SbgEkfStatus message
* Added new flags to SbgShipMotionStatus message
* Improved SbgStatusAiding message with new flags
* Improved SbgPosStatus message with new flags
* Improved SbgEkfStatus message with new flags
* Improved SbgStatusGeneral message with datalogger and cpu flags
* Updated documentation for sbgGpsPos message
* Updated documentation for sbgGpsHdt message
* Improved SbgUtcTime message with internal clock quality indicators
* Improved SbgStatusCom message with ethernet tx and rx status
* Improved SbgGpsHdt message with number of SV tracked and used
* Improved SbgGpsPos message with numSvTracked
* Improved SbgGpsPosStatus message with spoofing, jamming and OSNMA status
* Updated config files with new messages
* Removed obsolete documentation
* Updated published topic list
* Added missing topic names
* Updated dependencies requirement
* Added settings log_ekf_rot_accel_body / log_ekf_rot_accel_ned / log_ekf_vel_body
* Fixed functions description
* Added SbgEkfRotAccel body and NED messages
* Added SbgEkfVelBody message
* Fixing compiling issues
* Updated sbgECom lib with version 4.0-1987-stable
* Fixed typos about lever arm
* Fixed config applier for IMU Alignment / Aiding / Odometer lever arms
* Remove boost dependency
  (cherry picked from commit ab54c33f1e442c3737dd8e1c09a8b6f36c2c1afa)
* Cleanup
* Moved LLAtoECEF into a helper
* Variable naming
* WIP code cleanup
* Class documentation
* Code indentation
* Utm as class
* Indentation fix
* Updated sbg_utm documentation and function prototype
* Added documentation on Utm structure
* Added documentation on Position class
* Added Position class
* Factory for UTM data
* Fixed space / tabluations issues
* Code indentation
* SbgUtm documentation
* Using fma for computation
* Added constexpr to some variables
* Using pow instead of multiply
* Reworked createRosPointStampedMessage computations
* Removed sendTransform in fillTransform method
* Moved functions into helper namespace
* Moved UTM initialization into its own class
* Removed catkin reference in CMakeLists.txt
* Updated naming convention
* Improved ENU/NED documentation
* Disabling ROS standard message when in NED frame convention
* Reverted NED to ENU quaternion conversion
* Quaternion: cleaner version for NED to ENU conversion
* Reworked odometry message
* Quaternion: NED to ENU conversion rework
* Updated readme about ENU frame convention
* Reworked angle wrapping functions
* Added range for Euler angle measurement.
* More explicit naming for quaternions
* Reverted NED to ENU array conversion
* Documentation update
* odom->base_link is now correct in NED mode
* Renamed function.
* Fixed variable inversion.
* odom->base_link is now correct in NED mode
* Refactored NED to ENU array conversion
* Updated ROS messages documentation
* Fixed Euler / Quaternion orientation in ENU mode
* Fixed nmea output condition
* Compilation fixes
* NTRIP: GGA generation Work In Progress with the following fixex:
  - GPS to UTC time correctly apply leap second offset
  - GGA only sent if a valid position is available
  - GGA is sent at 1 Hz only
  - Minor improvements
  Code is not yet tested nor build
* Compilation fix
* Improved GGA generation and code cleanup
* Improved RTCM and NMEA parameters naming
* Code cleanup - removed (void) as it is not recommended in C++
* Reworked and improved main project README and small fixed in yaml examples
* Added documentation about RTCM messages and device configuration.
* Removed MessageSubscriber class
* Switched dependency from mavros_msgs to rtcm_msgs
* Updated documentation
* Removed SbgInterface as class member
* Removed threaded subscription.
* Namespace related coding style fix
* Fixes in GGA serialization
* Realigned members.
* Code documentation
* Improved NMEA GGA message
* Added rtcm / nmea parameters in config files
* Fixed deprecated header warning
* Added publisher for nmea msg
* Added subscription to RTCM msg
* remove build status
* fix build on Windows
* time_reference parameter fix
* Fix deprecated use of rosidl_target_interfaces
  The use of rosidl_target_interfaces is deprecated (see [Humble release notes](http://docs.ros.org.ros.informatik.uni-freiburg.de/en/humble/Releases/Release-Humble-Hawksbill.html#deprecation-of-rosidl-target-interfaces). Here that actually causes an issue with CMake not setting the right include directory paths, breaking colcon build on humble.
  This applies the documented update, making the driver build under Humble
* Contributors: Michael Zemb, Raphael Siryani, Samuel Toledano, SanderVanDijk-StreetDrone, Timon Mentink, VladimirL, cledant, rsiryani

Thank you for the information.
We don't need to release on Rolling for now actually.


Is KeyboardEventData deprecated?

With flet==0.1.62, the counter-accessible.py example fails:
Traceback (most recent call last):   File "counter-accessible.py", line 3, in <module>     from flet.page import KeyboardEventData ImportError: cannot import name 'KeyboardEventData' from 'flet.page' (/home/marc/.local/lib/python3.8/site-packages/flet/page.py)

It's been renamed to KeyboardEvent. Did you find an issue with one of the samples? PR with the fix would be highly appreciated!


zetacore: Remove observer set

Observer Set stores the list of observers
Node Account stores the list of TSS signers

For the zeta chain, these lists are identical by design right now. It makes sense to simply things and maintain just one

ObserverSet naming makes more sense to me, could we merge these into ObserverSet?

Yeah, we should basically just keep a list .
We can keep calling it observer set or observer-signer set


_mm_insert_ps is broken on ARM processor

Hi,
I noticed that _mm_insert_ps is broken on 0.7.2 (and I noticed still on master as well).
I have a fix that worked for me looking at: https://www.intel.com/content/www/us/en/docs/intrinsics-guide/index.html#techs=SSE4_1&ig_expand=4048
SIMDE_FUNCTION_ATTRIBUTES
simde__m128
simde_mm_insert_ps (simde__m128 a, simde__m128 b, const int imm8)
    SIMDE_REQUIRE_CONSTANT_RANGE(imm8, 0, 255)  {
  simde__m128_private
    r_,
    a_ = simde__m128_to_private(a),
    b_ = simde__m128_to_private(b);
  float tmp1_ = b_.f32[(imm8 >> 6) & 3];
  a_.f32[(imm8 >> 4) & 3] = tmp1_;
  SIMDE_VECTORIZE
  for (size_t i = 0 ; i < (sizeof(r_.f32) / sizeof(r_.f32[0])) ; i++) {
    r_.f32[i] = ((imm8 >> i) & 1 ) ? SIMDE_FLOAT32_C(0.0) : a_.f32[i];
  }
  return simde__m128_from_private(r_);
}

Please consider adding this to the new release or a similar fix.
The issue was a wrong copy and later missed masking.
For context the original is:
SIMDE_FUNCTION_ATTRIBUTES
simde__m128
simde_mm_insert_ps (simde__m128 a, simde__m128 b, const int imm8)
    SIMDE_REQUIRE_CONSTANT_RANGE(imm8, 0, 255)  {
  simde__m128_private
    r_,
    a_ = simde__m128_to_private(a),
    b_ = simde__m128_to_private(b);
  a_.f32[0] = b_.f32[(imm8 >> 6) & 3];
  a_.f32[(imm8 >> 4) & 3] = a_.f32[0];
  SIMDE_VECTORIZE
  for (size_t i = 0 ; i < (sizeof(r_.f32) / sizeof(r_.f32[0])) ; i++) {
    r_.f32[i] = (imm8 >> i) ? SIMDE_FLOAT32_C(0.0) : a_.f32[i];
  }
  return simde__m128_from_private(r_);
}

Best Regards,
Jawad

@MirJawadMairaj Thank you for your report and suggested fix. Can you make a pull request?

Hi! I added here: https://github.com/simd-everywhere/simde/pull/932
Hopefully that works. I made the pull request on master.
Didn't change the unit test as I am not running the test suite. If that is a requirement than please guide how to set that up on windows.
I might have some time later next week to look into it more.
Best Regards,
Jawad


Where's eventDrops.js?

Probably not an issue but rather my own ignorance, but why is there no eventDrops.js file in the src directory?
The only one I could find was in the test/karma directory but trying to use that file I get an error related to the import statement.
What am I doing wrong?
I just want to have the library called by the client as you explain in the README:
Usage

Include the `src/eventDrops.js` script in your page after d3:

<script src="path/to/d3.js"></script>
<script src="src/eventDrops.js"></script>

In the HTML source, create a new EventDrops chart, bind data to a DOM element, then call the chart on the element.

var eventDropsChart = d3.chart.eventDrops();
d3.select('#chart_placeholder')
  .datum(data)
  .call(eventDropsChart);`

Indeed, README seems outdated. We do not commit transpiled files anymore. However, if you need to include the eventDrops.js file without using any module bundler (such as Webpack), I suggest you use unpkg<EMAIL_ADDRESS>Please let me know if you need extra information. I'll make a PR to update the README. :)


prepare-training-set on staging fails with error  missing entity MEASURE_byte_0

I'm seeing this new error on staging:
/opt/almond-cloud/training/synthetic-gen-process.js:16
process.on('unhandledRejection', (up) => { throw up; });
                                           ^

Error: missing entity MEASURE_byte_0
    at makeDummyEntity (/opt/almond-cloud/node_modules/genie-toolkit/lib/utils.js:199:15)
    at makeDummyEntities (/opt/almond-cloud/node_modules/genie-toolkit/lib/utils.js:215:35)
    at Promise.all.minibatch.map (/opt/almond-cloud/node_modules/genie-toolkit/lib/sentence-generator/index.js:449:30)
    at Array.map (<anonymous>)
    at ContextualSentenceGenerator._process (/opt/almond-cloud/node_modules/genie-toolkit/lib/sentence-generator/index.js:446:54)
Emitted 'error' event at:
    at errorOrDestroy (internal/streams/destroy.js:107:12)
    at ContextualSentenceGenerator.onerror (_stream_readable.js:733:7)
    at ContextualSentenceGenerator.emit (events.js:198:13)
    at errorOrDestroy (internal/streams/destroy.js:107:12)
    at onwriteError (_stream_writable.js:436:5)
    at onwrite (_stream_writable.js:461:5)
    at ContextualSentenceGenerator.afterTransform (_stream_transform.js:94:3)
events.js:174
      throw er; // Unhandled 'error' event
      ^

Error: write EPIPE
    at WriteWrap.afterWrite [as oncomplete] (net.js:788:14)
Emitted 'error' event at:
    at errorOrDestroy (internal/streams/destroy.js:107:12)
    at onwriteError (_stream_writable.js:436:5)
    at onwrite (_stream_writable.js:461:5)
    at _destroy (internal/streams/destroy.js:49:7)
    at Socket._destroy (net.js:613:3)
    at Socket.destroy (internal/streams/destroy.js:37:8)
    at WriteWrap.afterWrite [as oncomplete] (net.js:790:10)

Yeah that's for contextual models, those need some additional amount of work...


Update man pages for default printer

Attempt to make explanation where the default printer is taken from clearer:
$HOME/lpoptions (/etc/cups/lpoptions for root) > ENV variables LPDEST/PRINTER > Server Default
Fixes #819

Hopefully fixed by the latest commit - feel free to check.

I believe we've picked up all of these changes, closing. If there is something I've missed please create a fresh PR or just commit the necessary changes directly. :)


Introduce detekt

https://github.com/detekt/detekt
Additional to ktlint detekt checks code for complexity and makes the developer aware if his code could be simpler.
This task is about introducing detekt and add it to the github actions script (consult @akulinski if you have questions about it)

@IanArb I'll take this one.


feat: Treat empty value as missing

#107

[x] The commit message is formatted
[x] New functionality includes testing
[x] New functionality has been documented
[x] All user-facing changes accompanied by the correspondent documentation
[x] All tests pass
[x] PR include resolved issues
[x] PR include description

:tada: This PR is included in version 1.16.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Correct the ordering of after hooks for Jest 20.

The logic for running the after hooks included a work around for a bug that
existed in Jest 18 and 19.  The bug has since been fixed in Jest 20, so we're
removing the fix.  Because removing the workaround will break 18 and 19, this
is a breaking change, and as such I've dropped 18/19 references from
package.json and updated to 20 everywhere.
Note: I'm operating on the assumption that you'd prefer a breaking change
rather than attempting to have a logic switch based on the Jest version.  Let me
know if you'd rather attempt to continue supporting 18+.
NoteNote: Jest 20.0.0 is currently broken for npm@2 and requires manual
installation of ansi-regex and jest-matcher-utils. The Jest team is aware of
this and will be fixing in the next patch. I think this can wait until that lands.

Let me know if you'd rather attempt to continue supporting 18+.

I'd be very interested to see what that PR looks like; it'd be ideal to add v20 support in a minor instead of a major.

I think this can wait until that lands.

I agree; we shouldn't support 20.0.0 at all, and should start v20 support at the first one that works with npm 2.

Not supporting 20 is blocking coverage on untested files @airbnb since previous to 20 the coverage is instrumented on the parent worker. We definitely want to update, so hopefully they'll cut a release in the next few days... I'll try to check against w/e the master sha is to verify it's been fixed.

I'd be very interested to see what that PR looks like; it'd be ideal to add v20 support in a minor instead of a major.

It's icky but...
https://www.npmjs.com/package/semver
const version = require('jest/package.json').version;
const semver = require('semver');
if (semver.satisfies(version, '18 || 19')) {
  // ...
} else {
  // ...
}

ooh, oof. i was hoping for something more "feature test" and less "user agent sniff".
What about something like Suite.prototype.afterAll.call([1], 2)[0] === 2?

I'm going to be totally honest... I have no idea what that snippet is attempting to do.

It's attempting to determine if afterAll is doing unshift or push - which is the change you made. It's using .call to set the this value to an object containing an array, and then afterAll mutates that array - in jest < 20, it pushes 2 onto the end, in >= 20, it unshifts it onto the front - so after that, we can check arr to determine whether it's [1, 2] or [2, 1].

Per discussion, the only way to support this is with a version check, or with a conditional require, so let's do the version check, cut a minor release, and then immediately drop support for < 20 and cut a major.
(pending the npm2 bugfix)

Closing this in favor of #9 which preserves backward compatibility.


Use a custom function to deserialize the IndexMaps in BlockLocators

A follow-up to https://github.com/AleoHQ/snarkOS/pull/2456.
This was found via simple fuzzing; it appears that with_limit doesn't fully guard us against overallocations, so introduce and use a custom method to deserialize the IndexMaps in BlockLocators.
Filing as a draft, as I'm not sure if we don't want to approach this some other way (e.g. restricting the number of entries etc.).

What's the perf hit of using this custom deserializer?
Also, shouldn't this be brought up to the indexmap crate for fixing?

What's the perf hit of using this custom deserializer?

As far as I'm aware, there shouldn't be any; also, this implementation is basically the same as the one in IndexMap, just doesn't pre-allocate.

shouldn't this be brought up to the indexmap crate for fixing?

AFAICT this would be a serde issue. Actually, I just looked at it and can see that there have been some related changes recently, so perhaps it would no longer be needed (i.e. it should be checked again).
Ultimately, though, it would be best to just use the impl<N: Network> FromBytes for BlockLocators<N> that was merged into the feat/narwhal branch today, in order to have full control over deserialization. If that is the intent, this PR can be closed.

Let's switch out all of the Router messages to To/FromBytes just like the Gateway events.


affected/package: golang.org/x/exp/slices#Compact

What version of Go are you using (go version)?
1.18
Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (go env)?
go env Output
GO111MODULE="auto"
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/jesseduffieldduffield/Library/Caches/go-build"
GOENV="/Users/jesseduffieldduffield/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/jesseduffieldduffield/.gvm/pkgsets/go1.18/global/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/jesseduffieldduffield/.gvm/pkgsets/go1.18/global"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/Users/jesseduffieldduffield/.gvm/gos/go1.18"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/Users/jesseduffieldduffield/.gvm/gos/go1.18/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.18"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/jesseduffieldduffield/repos/list/go.mod"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/d_/3fryg6d934l7jxttrz1z21bh0000gn/T/go-build1796431560=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?
https://go.dev/play/p/AP_5oYMfmUh
package main

import (
	"fmt"

	"golang.org/x/exp/slices"
)

func main() {
	slice := []int{1, 1, 2}
	output := slices.Compact(slice)
	fmt.Printf("%v\n", output) // outputs [1 2] (expected)
	fmt.Printf("%v\n", slice)  // outputs [1 2 2] (not expected)
}

What did you expect to see?
Expected both the resultant slice and the output slice to be [1 2]
What did you see instead?
Resultant slice is [1 2 2]
If this is expected behaviour, it would be good to specify in the docs that the original slice should not be used after calling this function and that instead you should only use the output slice.

I believe it already does:

Compact modifies the contents of the slice s; it does not create a new slice.


Upgrade Centos 7.9 to Centos-stream 8 via proxy serwer - problem

I have a problem with Upgrade Centos 7.9 to Centos-stream 8 via proxy serwer.
I set proxy in yum,conf any dnf.conf and I set LEAPP_PROXY_HOST=
But i have a below problem, after : leapp upgrade
STDERR:
No matches found for the following disable plugin patterns: subscription-manager
Repository extras is listed more than once in the configuration
Repository extras-source is listed more than once in the configuration
Warning: Package marked by Leapp to install not found in repositories metadata: python38-pulp-python rubygem-foreman-tasks-core python38-pulp-deb python38-pulp-certguard python3-javapackages python38-pulpcore python38-pulp-cli ivy-local python38-pulp-container rubygem-foreman_remote_execution_core python38-pulp-rpm log4j12 centos-linux-release python38-pulp-file
Warning: Package marked by Leapp to upgrade not found in repositories metadata: gpg-pubkey
Error: Error downloading packages:
Curl error (7): Couldn't connect to server for http://mirrorlist.centos.org/?release=8-stream&arch=x86_64&repo=BaseOS [Failed to connect to mirrorlist.centos.org port 80: Connection refused]
How can I resolve this problem ?

@smurdza Hello, thank you for reaching out. Please note that Leapp itself is a framework that enables you to define a workflow which will, consequently, help you to upgrade or modernize your system. However defining and implementing the workflow is up to the user.
You can take a look here for inspiration:
https://github.com/oamg/leapp-repository/tree/master/repos/system_upgrade
The above is the repository containing workflow to upgrade RHEL systems (supported by Red Hat). Unfortunately, I am not aware of any supported way to get from CentOS to CentOS Stream. One can convert from CentOS 7 to RHEL 7 and subsequently upgrade to RHEL 8 in a supported way though.
With your permission, I would close this issue as not a BUG.
Thank you for your understanding.

Ok, thank you for respond.


New package: MCIntegration v0.1.0

Registering package: MCIntegration
Repository: https://github.com/numericalEFT/MCIntegration.jl
Created by: @kunyuan
Version: v0.1.0
Commit: d53099615d5e27dbc3c67a32bcefb01957340c2e
Reviewed by: @kunyuan
Reference: https://github.com/numericalEFT/MCIntegration.jl/commit/d53099615d5e27dbc3c67a32bcefb01957340c2e#commitcomment-55235753

@kunyuan please see the message above.  Once you fix the automerge issues you can register again the new revision without changing the version number and this pull request will be automatically updated.
[noblock]


Unable to find valid certification path to requested target

In the Server Settings, I use Customize Server: https://openai-xyz-sandbox-experiment.openai.azure.com/openai/deployments/xyz-gpt35-experimental/chat/completions?api-version=2023-03-15-preview.
However, I am getting the following error:
PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target; PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target; unable to find valid certification path to requested target;

Does it work if you use other network (e.g.: if you are in work/university network, try using smartphone network instead)? Such error "might" suggest that you're behind some kind of proxy that "watches" https traffic.


Patch setup.py to remove install_requires

An attempt to fix #1 and #3 by,

patching setup.py to remove the install_requres line
adding the requirement versions to recepe/meta.yaml
additionally testing the import of vitables.utils

cc @scopatz @uvemas Please let me know if there is a better approach...

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

Hi @rth this seems like the right approach to me!

Thanks @scopatz ! Any suggestions on how I could debug the failing patch? It works fine on my machine using the same command:
/usr/bin/patch -p1 -i recipe/patches/0001-Remove-install_requires-from-setup.py.patch

I thought it was related to CRLF/LF issue as suggested in https://github.com/conda-forge/planarity-feedstock/issues/1#issuecomment-279162457 (setup.py is CRLF while the patch was originally LF) but converting the patch to CRLF with .gitattributes doesn't seem to help and CircleCi was still failing...
Detailed log below,
Downloading https://github.com/uvemas/ViTables/archive/v3.0.0.tar.gz
Success
Applying patch: '/recipe_root/patches/0001-Remove-install_requires-from-setup.py.patch'
patching file setup.py
Hunk #1 FAILED at 49.
1 out of 1 hunk FAILED -- saving rejects to file setup.py.rej
Warning: failed to download source.  If building, will try again after downloading recipe dependencies.
Error was:
Command '['/usr/bin/patch', '-p1', '-i', '/recipe_root/patches/0001-Remove-install_requires-from-setup.py.patch']' returned non-zero exit status 1.

@scopatz Replaced patching of setup.py with pip install --no-deps .  as suggested by @mivade . CI now passes and this PR ready to be reviewed ... Thanks!

Further tests show that pip install --no-deps does not seem to be fully equivalent to removing the install_requires line: in this case pip indeed does not attempt to check dependencies at install-time, but they are still checked at run time and running e.g., in a Py3.5 conda virtual env
pip install --no-deps -https://github.com/uvemas/ViTables/archive/v3.0.0.tar.gz

still produces an error when starting ViTables same as in https://github.com/conda-forge/vitables-feedstock/issues/1   (removing the install_requires line doesn't produce this error). I can't find any other relevant options for pip or setup.py, so I guess this means this PR is back to trying to make patching work..

OMG! I didn't expect this to be so difficult :(

Hi @rth - thanks for keeping this going.  I think patching is fine, if that is what is working right now.

Hi @rth, if patching keeps failing may be something similar to the solution proposed in #75 (near the end of the thread) could be useful here.

@uvemas Yeah, I was also thinking that this would have already been done with a simple find and replace (e.g. using sed, etc).. But patching is a cleaner way of doing it, and it was half working previously (on windows by not Linux). It must be some small Linux/Windows file formatting compatibility issue. I'll look into it this weekend..

Sorry for adding onto a closed PR/issue, perhaps I should open a new one? I can't remember how I got this working last time, but I just reinstalled miniconda, and now this is not working for me, or perhaps I have a new problem? I can install vitables-3.0.0-2 from conda-forge, but vitables.exe wont start, I just get this cryptic window:

Here are my qt from conda

qt
ver
src

pyqt
5.6.0
py36_5    conda-forge

qt
5.6.2
vc14_1  [vc14]  conda-forge

qtpy
1.4.0
py36_0    conda-forge

sip
4.18
py36_1    conda-forge

If I start it by running
python %LOCALAPPDATA%\Continuum\miniconda3\envs\py36\Lib\site-packages\vitables\start.py
then I get this traceback:

(py36) C:\Users\mikm>python %LOCALAPPDATA%\Continuum\miniconda3\envs\py36\Lib\site-packages\vitables\start.py
2018-04-17 17:12:07,347 - vitables.h5db.dbstreemodel - INFO - Creating the Query results file...
2018-04-17 17:12:08,254 - vitables.preferences.pluginsloader - ERROR - Failed to load plugin: vitables.plugins.columnorg.columnar_org
2018-04-17 17:12:08,255 - vitables.preferences.pluginsloader - ERROR - The 'PyQt5>=5.5.1' distribution was not found and is required by the application
2018-04-17 17:12:08,597 - vitables.preferences.pluginsloader - ERROR - Failed to load plugin: vitables.plugins.dbstreesort.dbs_tree_sort
2018-04-17 17:12:08,597 - vitables.preferences.pluginsloader - ERROR - The 'PyQt5>=5.5.1' distribution was not found and is required by the application
2018-04-17 17:12:08,934 - vitables.preferences.pluginsloader - ERROR - Failed to load plugin: vitables.plugins.csv.export_csv
2018-04-17 17:12:08,935 - vitables.preferences.pluginsloader - ERROR - The 'PyQt5>=5.5.1' distribution was not found and is required by the application
2018-04-17 17:12:09,270 - vitables.preferences.pluginsloader - ERROR - Failed to load plugin: vitables.plugins.csv.import_csv
2018-04-17 17:12:09,271 - vitables.preferences.pluginsloader - ERROR - The 'PyQt5>=5.5.1' distribution was not found and is required by the application
2018-04-17 17:12:09,608 - vitables.preferences.pluginsloader - ERROR - Failed to load plugin: vitables.plugins.timeseries.time_series
2018-04-17 17:12:09,609 - vitables.preferences.pluginsloader - ERROR - The 'PyQt5>=5.5.1' distribution was not found and is required by the application

but it does open, so this is success? Seems like the error is, that it doesn't detect PyQt even though I can import from qtpy from within python.
This also works, with the same errors tho: python -c "from vitables import start;start.gui()" or from inside a python interpreter it does the same, but only if I have the conda-forge pyqt-5.6.0.
Another oddity, if I have the Anaconda version of pyqt-5.6.0, then when I try: from vitables.start import gui or from vitables import start python just crashes, no traceback, nothing. Weird.

Maybe this needs to be rerendered to match the new packages.  I'll try that.

I have rerendered the feedstock in #5.  Hopefully this fixes the issue.  A new package should be available soon.


This page vs https://docs.microsoft.com/en-us/microsoftteams/platform/graph-api/proactive-bots-and-messages/graph-proactive-bots-and-messages?tabs=csharp

Partners are confused by these two pages:
https://docs.microsoft.com/en-us/microsoftteams/platform/graph-api/proactive-bots-and-messages/graph-proactive-bots-and-messages?tabs=csharp

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 4ad9f167-7d29-f933-7b47-60bd744899b1
Version Independent ID: 8097fcbd-adcc-ce49-a378-426c65af9296
Content: Send proactive messages - Teams
Content Source: msteams-platform/bots/how-to/conversations/send-proactive-messages.md
Product: msteams
GitHub Login: @clearab
Microsoft Alias: anclear

@aosolis , Thanks for brining up this suggestion,  We will ask document team to look into this and make the document better

Reopening until PR is raised for this doc bug

Hi @Mahesh2516 , Could you please share us the status on the updation of this document.


[Android] Switch pipelines to use Windows hosted pools

Related Issue
Tries to fix #4972
Description
Use Windows images to run pipelines to see if it solves Teams' issue
How Verified
Pending...
Microsoft Reviewers: Open in CodeFlow

@golddove does this need to be merged into main to see if it will work?


DNS interception and upgrade to DNS-over-TLS

RFC: https://docs.google.com/document/d/1zQjrg9SAyFbFObuhykpqkvGF3C3FNcl5rXw2ZF0PYQc
For 1.6, the minimum is to allow redirection of plain text DNS-over-UDP requests from applications
to a secure DNS-over-TLS server.
Optional - in 1.6 or later - we can add support for .global ( or equivalent multi-cluster zone ), and
optimize resolution by using Istiod multi-cluster view of cluster IPs, to remove the DNS setup pains
for MC.
Describe the feature request
Describe alternatives you've considered
Additional context

PR in progress related to this effort: https://github.com/istio/istio/pull/22369


Update README.md

First swing at incorporating feedback from @drott and @behdad

lgtm. Thanks

LGTM as well, thanks for channeling the feedback into these good edits.

Thank you for the feedback!


#1825 Added checking for run-all arguments

Added verification for an empty argument of run-all command and error returning if no arguments specificed
Before:
$ terragrunt  run-all
ERRO[0000] Module /tmp/terragrunt-test-1499/module1 has finished with an error: 1 error occurred:
        * exit status 127
  prefix=[/tmp/terragrunt-test-1499/module1]
ERRO[0000] Dependency /tmp/terragrunt-test-1499/module1 of module /tmp/terragrunt-test-1499/app just finished with an error. Module /tmp/terragrunt-test-1499/app will have to return an error too.  prefix=[/tmp/terragrunt-test-1499/app]
ERRO[0000] Module /tmp/terragrunt-test-1499/app has finished with an error: Cannot process module Module /tmp/terragrunt-test-1499/app (excluded: false, dependencies: [/tmp/terragrunt-test-1499/module1]) because one of its dependencies, Module /tmp/terragrunt-test-1499/module1 (excluded: false, dependencies: []), finished with an error: 1 error occurred:
        * exit status 127
  prefix=[/tmp/terragrunt-test-1499/app]
ERRO[0000] 2 errors occurred:
        * exit status 127
        * Cannot process module Module /tmp/terragrunt-test-1499/app (excluded: false, dependencies: [/tmp/terragrunt-test-1499/module1]) because one of its dependencies, Module /tmp/terragrunt-test-1499/module1 (excluded: false, dependencies: []), finished with an error: 1 error occurred:
        * exit status 127

After this change:
$ terragrunt  run-all
ERRO[0000] Missing run-all command argument
ERRO[0000] Unable to determine underlying exit code, so Terragrunt will exit with error code 1

Fix for https://github.com/gruntwork-io/terragrunt/issues/1825

Build passed, so will merge this in!


[fix][broker] Avoid being stuck in 30+ seconds when closing the BrokerService

Fixes https://github.com/apache/pulsar/issues/22569
Motivation
BrokerService#closeAsync calls unloadNamespaceBundlesGracefully to unload namespaces gracefully. With extensible load manager, it eventually calls TableViewLoadDataStoreImpl#validateProducer:
BrokerService#unloadNamespaceBundlesGracefully
  ExtensibleLoadManagerWrapper#disableBroker
    ExtensibleLoadManagerImpl#disableBroker
      ServiceUnitStateChannelImpl#cleanOwnerships
        ServiceUnitStateChannelImpl#doCleanup
          TableViewLoadDataStoreImpl#removeAsync
            TableViewLoadDataStoreImpl#validateProducer

In validateProducer, if the producer is not connected, it will recreate the producer synchronously. However, since the state of PulsarService has already been changed to Closing, all connect or lookup requests will fail with ServiceNotReady. Then the client will retry until timeout.
Besides, the unload operation could also trigger the reconnection because the extensible load manager sends the unload event to the loadbalancer-service-unit-state topic.
Modifications
The major fix:
Before changing PulsarService's state to Closing, call BrokerService#unloadNamespaceBundlesGracefully first to make the load manager complete the unload operations first.
Minor fixes:

Record the time when LoadManager#disableBroker is done.
Don't check if producer is disconnected because the producer could retry if it's disconnected.

Verifications
Add ExtensibleLoadManagerCloseTest to verify closing PulsarService won't take too much time. Here are some test results locally:
2024-04-24T19:43:38,851 - INFO  - [main:ExtensibleLoadManagerCloseTest] - Brokers close time: [3342, 3276, 3310]
2024-04-24T19:44:26,711 - INFO  - [main:ExtensibleLoadManagerCloseTest] - Brokers close time: [3357, 3258, 3298]
2024-04-24T19:46:16,791 - INFO  - [main:ExtensibleLoadManagerCloseTest] - Brokers close time: [3313, 3257, 3263]
2024-04-24T20:13:05,763 - INFO  - [main:ExtensibleLoadManagerCloseTest] - Brokers close time: [3304, 3279, 3299]
2024-04-24T20:13:43,979 - INFO  - [main:ExtensibleLoadManagerCloseTest] - Brokers close time: [3343, 3308, 3310]

As you can see, each broker takes only about 3 seconds to close due to OWNERSHIP_CLEAN_UP_CONVERGENCE_DELAY_IN_MILLIS value added in https://github.com/apache/pulsar/pull/20315

Failed tests:
ModularLoadManagerImplTest.shutdown:240 » PulsarServer java.lang.NullPointerException: Cannot invoke "org.apache.pulsar.broker.service.BrokerService.unloadNamespaceBundlesGracefully()" because "<local2>.brokerService" is null

  Error:  org.apache.pulsar.broker.PulsarServiceTest.testBacklogAndRetentionCheck  Time elapsed: 16.407 s  <<< FAILURE!
  org.apache.pulsar.broker.PulsarServerException: java.lang.NullPointerException: Cannot invoke "org.apache.pulsar.broker.service.BrokerService.unloadNamespaceBundlesGracefully()" because "<local2>.brokerService" is null


BitBucket Server Unreachable while connecting

Hey Alex,
It's really a wonderful solution which has almost taken care of our problem to include Bitbucket Server to CodePipeline.
Everything was working as expected. But suddenly we got this below mentioned error while connecting to API Gateway from Bitbucket.
ERROR : "The server didn't respond you may retry when it comes back up". Please refer to the screenshot for better explanation.
Not able to figure out where it is failing as the event is not coming to the API gateway itself. Is it a Bitbucket Server Firewall error or there is a problem with the API Gateway endpoint which was working absolutely fine till last week.

Please Note that we are using Enterprise Bamboo Server and our AWS VPC has connection to on-prem bitbucket server with Transit Gateway and my Lambda function is running on a specific VPC to be able to download the code from Bitbucket.

Hi @rajddas it seems like the request is not reaching the API Gateway. Could you check the API Gateway and Lambda logs?

Ya Alex, in my API gateway logs and metrics no data have been populated. We are currently in touch with the Bitbucket Server team, lets see what they have to say.

Hi,
I am facing the same issue
Did you get any resolution.
Thanks!!
Manoj

Yes, It was an issue with the bitbucket server. Increase these properties :
plugin.webhooks.io.threads
plugin.webhooks.dispatch.queue.size
plugin.webhooks.http.connection.host.max
It should solve the issue.

Thank you!!!
It worked

As its resolved. Closing the issue.


Missing entities in D103 G3

I have a total of three D103 G3 devices integrated.
One has 31 entities and the other two only 26 entities.
Two devices are missing "total" entities.

total consumption
total current
total power
total return
total return power
What could be the reason for this? How to add remaining entities?

The logic of this integration is quite simple - it adds entities for the keys in the MQTT payload. If there's no key the entity is not added
I'd suggest checking the MQTT payload of every device. You can use MQTT explorer for that, just connect to your broker and look for the devices there, see if you can spot any difference
Feel free to post your findings here

First of all, I want to apologize for the delay.
I have compared all three D103 G3 units using MQTT Explorer and the result is that they are all identical (they have the same number of keys = 30).
In HA, two units have 26 entities and only one has 31 entities. Additional entities are: total consumption / total current / total power / total return / total return power. None of these entities are transmitted directly from the D103 G3 by any key in MQTT.

Yes, you are right. Two devices have A1 = 0 (only work intermittently). It would be better to check V1 as the voltage is present at all times.

Should be fixed now, please confirm

Excellent, thank you very much.


add organization_id param to LlamaCloudIndex.from_documents

Description
Helps with the case where a user has two projects in two different orgs that have the same project name. Now you can at least optionally specify the organization_id
Fixes # (issue)
New Package?
Did I fill in the tool.llamahub section in the pyproject.toml and provide a detailed README.md for my new integration or package?

[ ] Yes
[ ] No

Version Bump?
Did I bump the version in the pyproject.toml file of the package I am updating? (Except for the llama-index-core package)

[x] Yes
[ ] No

Type of Change
Please delete options that are not relevant.

[x] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[x] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[x] This change requires a documentation update

How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

[x] Added new unit/integration tests
[ ] Added new notebook (that tests end-to-end)
[x] I stared at the code and made sure it makes sense

Suggested Checklist:

[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made corresponding changes to the documentation
[ ] I have added Google Colab support for the newly added notebooks.
[x] My changes generate no new warnings
[x] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes
[x] I ran make format; make lint to appease the lint gods

Hmm, seems like some undefined error in the unit tests

Hmm, seems like some undefined error in the unit tests

oops yeah made a bad last min edit before committing. added back in


Scrolling disabled

Description
Couldnt scroll on Firefox 130.0
(Optional:) Please add any files, screenshots, or other information here.
Link for page on which this error was caught: https://arxiv.org/html/2406.01465v1
(Required) What is this issue most closely related to? Select one.
Other
Internal issue ID
b7fc5b64-46f5-4175-9641-380189d50ab4
Paper URL
https://arxiv.org/html/2406.01465v1
Browser
Firefox/130.0
Device Type
Desktop

@ando-shah just FYI you can scroll, but it will take you a minute - there is an aberrant SVG that consumes multiple vertical displays.
Thanks for the report, definitely an issue!


mnist-inference-web wgpu matmul returns all zeros

Describe the bug
The wgpu matmul implementation returns all zeros when running in the browser.
Also reported by a discord user: https://discord.com/channels/1038839012602941528/1059209073784008835/1273995743790628876
To Reproduce
Run the mnist-inference-web example built for wgpu.
The scores returned are always the same for all digits.

Upon further debugging, the fc1 layer always returns zeros because of the matmul issue (tiling2d implementation w/ cube). Note that using the simple matrix multiply works (and cmma is not available in this case).
Expected behavior
The results from the matmul should not be zero, and thus the scores correct.

This should fix it: https://github.com/tracel-ai/cubecl/pull/81

Just tested, this was fixed when updating to the latest cubecl version to include the fix in #2181


Ensure namespace and collection names are validated

Only allow names that C* would allow. Add validation to the commands. Improve URI path patterns to include the validation as well, or maybe simply validate CommandContext that's being passed to the CommandProcessor.

@maheshrajamani what reserved words are you referring to, I was checking this:
Keyspace names can have up to 48 alpha-numeric characters and contain underscores; only letters and numbers are supported as the first character. Cassandra forces keyspace names to lowercase when entered without quotes.

The issue states "Only allow names that C* would allow. ". I was checking if more validation is needed with cassandra reserved words.
Eg: create table select (col1 text PRIMARY KEY , col2 text); SyntaxException: line 1:13 no viable alternative at input 'select' (create table [select]...)
But with the change done to enclose collection names with quotes this should be handled.
Eg: create table "select" (col1 text PRIMARY KEY , col2 text);
Think we can close this issue.

I still need to enforce validation on the path parameters. Will do this and resolve it.


Define Schema so that the example actually works.

This removes a potential stumbling block in the example, as reported in #9.
The example uses Schema as a variable but never defines it.

@metzuda Thank you !


[versions] roll package test redux

Fixes #81516
Fixes #81834
Fixes #81294
Redo of https://github.com/flutter/flutter/pull/82275

Looks like we're having trouble pulling in new versions of several plugins. I'll file a bug tomorrow and pin

Very nice, thanks!


Update .travis.yml

Problem:
Docs did not deploy for v10.1 because the cut command in the deploy script resulted in an incorrectly formatted version number (v10.).
Solution:
I updated the cut command to result in a correctly-formatted string (e.g., v10.1).
@<reviewer_id>
What issues does this address?
Fixes #

Can one of the admins verify this patch?


hi, may you create a list from all the commands in this library?

Is your feature request related to a problem? Please describe.

Describe the solution you'd like

Describe alternatives you've considered

Implementation ideas

Additional context

https://core.telegram.org/bots/api 🙂


Error after entering 45 in PowerToys Run

Microsoft PowerToys version
0.58.0
Running as admin

[X] Yes

Area(s) with issue?
PowerToys Run
Steps to reproduce
Error after entering 45 in PowerToys Run
2022-05-17.txt
✔ Expected Behavior
No response
❌ Actual Behavior
Version: <IP_ADDRESS>
OS Version: Microsoft Windows NT 10.0.22621.0
IntPtr Length: 8
x64: True
Date: 2022-05-17 09:38:44
Exception:
System.InvalidOperationException: 调度程序处理已暂停，但仍在处理消息。
at System.Windows.Threading.Dispatcher.WndProcHook(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam, Boolean& handled)
at MS.Win32.HwndWrapper.WndProc(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam, Boolean& handled)
at MS.Win32.HwndSubclass.DispatcherCallbackOperation(Object o)
at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate callback, Object args, Int32 numArgs)
at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object source, Delegate callback, Object args, Int32 numArgs, Delegate catchHandler)
Other Software
No response

/dup #18166

For any users that are able to replicate this consistently, can you please update to 0.68.1 and disable thumbnail generation to verify if it fixes the issue? Please let us know on #18166!


In rate us page after submit form not reset and thank you message not shown.

I will fix it and make it working form.

@sunny0625 Please assign me this task !
and merge #1307

@sunny0625 Please assign me this task !
Thank You !


"Add summary to DevChat" triggers a meaningless notification

Describe the bug

To Reproduce
Right-click a file and choose "Add summary to DevChat".
Expected behavior
Is this because the summary index is not built yet? If so, the notification text is not correct. Instead, it should say the index is not ready and how to make it.
If the notification text is correct, we should hide "Add summary to DevChat" instead of showing this meaningless notification.

"Add summary to DevChat" is only open to internal test users, you need to fill in the internal test code to activate the actual use of this function. This menu cannot be completely hidden in vscode, because this menu is defined in package.json, and the defined menu cannot be blocked by code registration or unregistration.


how to add DN to Vallina-DETR like model

Thanks for this amazing work! I have some question about adding DN to a Vallina-DETR like model.
Could you explain more about how can I use DN for a Vallina-DETR like algorithm?
Because the Vallina-DETR's object quries are not anchor-like, thus I don't know how to change an denoised gt(dim=a) to a obj query(dim=b && b != a)?
Will a learnable nn.linear or other oporator work?
Looking forward to your reply!

You can refer to this issue for more details.


Makefile: Add testing targets

What this PR does / why we need it:
Add three new targets to the Makefile: unit-tests, workflow-integration-tests and notification-integration-tests to make it easier to developer to run only tests
Also default exclude any test which has "integration" in its path, to avoid running integration tests by default, when running any phase from the parent module
Notice that notification-service integration tests are not merged yet, its going to be added in PR #274
Change type

[ ] New feature
[ ] Bug fix
[ ] Unit tests
[ ] Integration tests
[ ] CI
[ ] Documentation
[ ] Auto generated SDK code

Impacted services

[ ] Workflow Service
[ ] Notifivcation Service

Checklist

[x] Subject and description added to both, commit and PR.
[ ] Relevant issues have been referenced.

@gciavarrini FYI
Do you think there is any issue to have this merged with #281?

Closing this PR as all integration tests location is going to be changed in #281


Add a prime checker (classic, witness numbers)

This has been completed. Currently, the Miller-Rabin method (witness numbers) is slightly slower than classic for basically all values of $n$. However, in the future when Mathf.PowerMod(int, int, int) has been optimized, it will become much faster for very large values of $n$.


Windows builds corrupt (I removed them)

they were 18 kB or so.  Definitely wrong.  I removed them from anaconda.org so they won't trip anyone up.  The win build needs work - it definitely should fail if its packages are corrupt like this.

That sounds like a good plan. Should we disable Windows builds at present until we get this fixed?

yes, that sounds like a good idea.

I would rerender, but our Mac builds are now failing due to the header path length. Weird as this didn't hit us before, but it is now.

Seems there was an issue with the build script here. Added a test and a fix in this PR ( https://github.com/conda-forge/git-feedstock/pull/11 ).


Upgrade UI-Router to 0.3.2

currently on 0.3.1

https://github.com/angular-ui/ui-router/releases/tag/0.3.2


Is it possible to play audio in reverse direction with NAudio

I need to implement audio player with reverse playback option. Can I implement this with NAudio library?

I would like to know as well

I'd like to know it too

Also interested


WebXRNearInteraction doesn't properly override WebXRControllerPointerSelection with two hands up

When two hands are up in the scene, I noticed that the rays are only disabled for one of the hands. The finger cursor shows up on both, but only one hand can interact via near interaction.
Probably the whole way the two classes interact needs to be changed, since NI just disabled CPS during near interactions, but should just disable it for a single hand as desired.

Note that the pointer system has an option to have either one hand or both hands enabled. As you say, it might be an issue with the selected hand when re-enabling the feature. the pointer system sets the ray to the last hand that used the main trigger. There should be a function that forces a specific hand to be anbled. if there isn't you can add one and use it when re-enabling.

@rickfromwork and @RaananW  any update on this one ?

After discussion with @RaananW tagging as enhancement


How do you train with those NOT-TEXT elements.

Dear author,
For some documents that contain massive not-text elements, such as hundreds of thousands of "##LTLine##". How do you deal with them actually?
For example, you try to train&predict all those elements with text '##LTLine##'.
Thank you!

Yes.  We regard '##LTLine##' as a special token during train and predict.

Yes. We regard '##LTLine##' as a special token during train and predict.

Hi! Could you please tell integer identifiers of ##LTLine## and ##LTFigure##  tokens within LayoutLM's vocabulary?
Thanks

In fact, we did not add them to the vocabulary. They will also be tokenized into tokens and labeled in the way I mentioned at #25.

Thanks


remove one line of comment

@pawelRytwinski Please sign the Contributor License Agreement!
Click here to manually synchronize the status of this Pull Request.
See the FAQ for frequently asked questions.

Removed one line of the text.


Arduino Json version updated

Only the ArduinoJson version ranging from v5.13.1 to v5.13.5 work, v6.xx.x seem to have a compatibility issue.

@googlebot I signed it!


Some fixes

SpectMorph is using pugl for its UI. Here are a few fixes that I've made that should be good enough for inclusion in upstream.
Event times: I use the event timestamp for double click detection (two clicks, less than 400ms apart). macOS uses a double timestamp for events in seconds, whereas the other platform use milliseconds as integers for the event time. To be able to use the same code for all platforms, I suggest always using double seconds as timestamps, this should be good enough to represent both.
Backspace/delete is not 0x8/0x7f on all platforms, so I suggest using the same strategy as for function keys: define PUGL_KEY_BACKSPACE | PUGL_KEY_DELETE, and map whatever platform specific code this has to these pugl keys.
On macOS current pugl has event->key.character and event->key.special set (i.e. if you use arrow keys), this is not consistent with the other platforms. So I've added a commit which enforces that a special key will always have character == 0.
The other fairly small fixes just fix bugs.
Ideally one day I would be able to use an unmodified version of pugl for SpectMorph, but there are a few more things to sort out. One main issue is that I need to resize the window if the user changes plugin zoom level, currently that is done using my own version of
void puglSetResizeFunc(PuglView* view, PuglResizeFunc resizeFunc);
void puglPostResize(PuglView* view);

but here I currenly don't have a clean enough patch for upstream inclusion so I left this out at the moment.

I need to fire up my other machines and do some manual testing to wrap my head around this (oh, for a way to automatically test this stuff...), but I'll try to get this merged this week sometime.
It would be awesome to get you (and everyone else!) on unmodified upstream pugl.  The "copy paste code reuse" idea was a big mistake.  I am going to include pugl in the upcoming "lv2kit" which should help provide some additional incentive in that direction...

For prosperity (rebasing): f3782ca 0962252 cbc869f ca87f40

For prosperity (rebasing): f3782ca 0962252 cbc869f ca87f40

I merged 3 of your commits, and a whole bunch of other fixes/improvements I found along the way, to master.  https://github.com/drobilla/pugl/tree/some-fixes is now everything left, rebased on that.
No idea if the embedding works, though... I should really write test programs for that one of these days.

So I did quite a bit of tinkering with the key stuff on https://github.com/drobilla/pugl/tree/uniform-keys
In the process I managed to get dead keys working on Windows, which is nice (it was totally broken before), but I haven't done MacOS yet.  I'm kind of debating whether to emulate the API of Windows more and just provide two completely separate events for key press/release and text input so there's no risk of confusing the two.  It's more or less impossible to "merge" them on Windows, so there's two events anyway, so maybe they should just have different types altogether...
The "unshifted unicode point" idea seems to be solid, but I'm not sure about the rest of the details.  I haven't actually implemented a text entry widget with pugl, so I'm kind of just guessing at what's convenient.  The shifted one seems kind of useless for identifying keys, so maybe it should just go, but this is another argument for separate events (PuglEventText would be shifted, to match the string).  Thoughts?

First of all, there seems to be a small bug in your uniform-keys branch on X11: if I hit F1, I'd expect unicode 0xe000, but I get unicode 0x0000 (similar for other keys). It seems that the lookupString return code is not greater than 0 in here:
if (lookupString(view->impl->xic, xevent, event->string, &sym) > 0)
As for shortcuts with your new event key normalization: Ctrl/Shift/Alt+A..Z should now work properly (I used to normalize these manually for old pugl), also Ctrl/Shift/Alt+0..9. Just check unshifted key and compare modifiers and that's it. Also the collision problem for backspace/tab/... is really solved this way.
I can however think of one class of keyboard shortcuts which I don't know how to handle with your API by checking key and comparing modifiers: lets look at for example Ctrl+;.
On a us keyboard layout, there is a key for ;, so pugl_test shows for Ctrl+;:
Text        code 47 char U+003B key U+003B (;)
On a de (german) keyboard layout, I have to use Shift+, to get a ;, and therefore pugl_test prints for Ctrl+; aka Ctrl+Shift+,
Text        code 59 char U+003B key U+002C (;)
As you see, key is normalized to ; (us) or , (de) depending on keyboard layout, so checking for key==';' will not reliably detect Ctrl+;. Also a 1:1 comparision of modifiers is not possible, since it would be Ctrl (us) or Ctrl+Shift (de). So what this means is that for such shortcuts I don't really know how to use the keyboard event as generated by pugl to figure out if that shortcut was used. On the other hand it may be acceptable to just avoid defining such shortcuts in pugl UIs. In this case I'd agree that there is no reason to keep shifted character, and we should just have unshifted key and that is all.
Finally a few thoughts on what is required for the LineEdit widget: it is somewhat convenient to be able to handle all editing operations (such as delete or entering an 'A') in one single function, because you can add common code (like emitting a signal if the text was changed) in one place at the end of this editing function. So this kind of suggests both should be in one key press event handler. However this is not a hard requirement, it would probably be not so bad to refactor this to two different functions so I really can't say whether using one event type (key press) or two event types (key press and text) is better.
When new text is entered, I never even look at the keyevent character, key or modifiers. So I don't care if they are shifted, unshifted, normalized or even zero. I just use the utf8 string the key press event contains and update the string (see key_press_event function https://github.com/swesterfeld/spectmorph/blob/master/glui/smlineedit.hh). So as long as I can reliably detect if the event contains text (then I need to add it to the text of the widget) or not (to reliably detect if something is a control character like "left arrow") everything is fine.

I think shortcuts like that are going to be problematic in the same way regardless of API.  There's really only two things any hypothetical API can tell you about this stuff: which key was pressed, and which text (if any) that corresponds to.  Ctrl + (something that isn't a key) is always going to be different than ctrl+key somehow, the keypresses and modifier state are genuinely different.
That said, it seems you can implement it easily enough if you really want to.  You have the ';' both in the text, and in the shifted charcter code (U+003B), and you have the fact that ctrl is pressed.  Information-wise everything is available, except for "shifted with shift but not with ctrl".  That can be dealt with easily either in client code or in pugl though (I'm not sure, but prefer the simple modified/unmodified thing vs treating different ones specially in pugl which seems like it will just cause more problems in the future because it's not what somebody is going to want for something).

only two things any hypothetical API can tell you

I disagree. For instance, of course Qt5 supports defining a shortcut for Ctrl+;. I just create a QShortcut for Ctrl+; and that is it. I don't have to guess, and it works with any keyboard layout, Qt5 does this for me. Of course QShortcut belongs rather into the SpectMorph toolkit than into pugl, as it is high level API.
Now I investigated what low level API QShortcut is using to detect Ctrl+;. There is a private part of the Qt API that defines QKeyMapper, this is platform specific. And this QKeyMapper can give you a list of possible keys for a QKeyEvent*. So detecting Ctrl+; with Qt5 private API can be written as:
void
keyPressEvent (QKeyEvent *event)
{
  for (auto k : QKeyMapper::possibleKeys (event))
    {
      if (k == ';' + Qt::CTRL)
        printf ("Ctrl+; detected.\n");
    }
}

For the record QKeyMapper::possibleKeys returns only Ctrl+; as possible input for us layout, whereas it returns Ctrl+Shift+, and Ctrl+; for de layout.

I've meanwhile looked at the platform specific possibleKeys (QKeyEvent *event) code for XCB, Windows and macOS in Qt. To do their work, they use

event->modifiers()
event->nativeVirtualKey()
event->nativeModifiers()
event->nativeScanCode()

This seems like it doesn't really use what pugl calls character or key but rather operates on the platform specific part of the event. Not sure if keycodeand state would be sufficient for pugl. If yes, and if we wish to avoid returning a list of things (bad for C API), it should be possible to implement a
bool puglKeyCompare (PuglKeyEvent *key_event, uint32_t state, uint32_t character);

which could then be used as
  if (puglKeyCompare (key_event, PUGL_MOD_CTRL, ';'))
    printf ("Ctrl+; detected.\n");

so it would basically check if the shortcut state and character was in the possibleKeys list of key_event. It is still a bit of work which has to be repeated for each platform, and it now looks like it doesn't really affect the API freeze for the keyboard API we were discussing earlier. So.... if you feel like the shortcut matching should be postponed, then ok, we have other issues which I also think have higher priority. Like getting backspace and delete to work.
Just one example implementation of possibleKeys for XCB:
https://github.com/qt/qtbase/blob/11d9af5d957ac391a0c6cfdea4819df44b2b1560/src/plugins/platforms/xcb/qxcbkeyboard.cpp#L1013

I don't think it would be very good to have the core API of Pugl use only scancodes (and so be useless for most applications) and require a higher level API written from the ground up to get text.  As you say, that's more of a toolkit thing.
Anyway, I managed to get everything working as far as I can tell on all three platforms, with dead keys and composing and all the rest of the tricky bits: https://github.com/drobilla/pugl/commit/b11f5ff754df4e189792c18521e758ecb388a648

I merged the double/seconds change, and more or less completely rewrote the keyboard/text input.  I've tested it quite a bit on all three platforms, and it seems to work with all the tricky situations like compose and dead keys.  There's probably still some edge cases that aren't right (and some things that just don't work at all, like CapsLock on Cocoa), but it's dramatically better than it was anyway.
I fixed a bunch of other stuff as well, let me know if you have any issues.

let me know if you have any issues.

I did some tests today, compiling my SpectMorph plugin against your pugl. Text events work nicely. Here is a preliminary results
macOS

backspace and delete don't work
the y - coordinates of mouse events are wrong (if I make - (BOOL) isFlipped return NO, things work)
I can no longer build with 10.9 SDK - currently I target macOS >= 10.9 for SpectMorph

Of course the last point may not be much of an issue for pugl upstream, you can require whatever you think is appropriate. It used to build against 10.9 SDK before, though. I probably wouldn't update my build environment just for pugl, though.
Windows

delete doesn't work
non-standard location for cairo doesn't work, should replace in cairo_gl.h
#include <cairo/cairo.h>
with
#include <cairo.h>
I saw some redrawing glitches on windows, but haven't figured out why it happens

Linux

initial redraw of a window doesn't work - after that it seems to work - I haven't figured out the cause yet

general

struct PuglEventText time field should probably be double
you didn't merge the special key translation for backspace and delete - I believe you can do it unmodified from https://github.com/swesterfeld/pugl/commit/0361ac57fba1d55e00dba3f55ff610979e0cbe68 which should fix these keys on windows/macOS

Sigh.  It builds here with -mmacosx-version-min going all the way back to 10.6.  Can you be more specific?

Initial display on X11 works for me, this could be a window manager specific thing.
Everything else should be fixed, thanks.    I guess in the context of pugl I can't complain about people breaking APIs, but still...

2e3e935
3359c78

Ok, some progress on

Linux: initial redraw of a window doesn't work - after that it seems to work - I haven't figured out the cause yet

SpectMorph has two optimizations for partial screen updates, for instance for drawing a continually moving play position pointer over a sample:

it only redraws the part of the cairo surface that was modified
it uses GL to copy the part of the cairo surface to the screen that was modified

So it has its own version of cairo gl surface to be able to do this.
For expose events, this means that the coordinates are ignored and SpectMorph only redraws the part that has changed. Now the problem is that with your new version of the code, I get two expose events for the whole screen at the start when the window is initially shown. For the first expose event, SpectMorph renders everything. For the second expose event, it simply does nothing, because nothing was changed. The old pugl did only deliver one expose event.
So this may mean that my code is wrong, and I never should have done any partial updates in the first place. And indeed if I disable the second optimization in SpectMorph and always redraw everything for each expose event using GL then the redrawing glitches are gone.
However for a typical sample view window with a moving play position pointer the CPU usage is:

9.5% with both SpectMorph drawing optimizations enabled
18.5% with the second GL optimization disabled

So.... maybe requesting partial redraws should be pushed into pugl to allow doing this safely.

That seems wrong, surely you need to draw what is expose in general?  That's what an expose is.  I could maybe make the "double expose" go away, but the whole reason that's there is to make initial display work.  There's a bunch of weird cases where it wouldn't without, but I can try to make it work...
That said, I suspect code that works like that is always going to have problems with things not being drawn when they need to be.  Pugl should deal with partial updates, and it does for ones that come from the system (this is a distinguishing thing about it, to me: it's not like some game-centric GL libs that just assume you want to draw at full blast constantly).  I guess you mean you want to post a partial update?
That would be nice.  Maybe just a general puglPostEvent() thing that can take an expose or whatever, since we already have a struct to describe exactly this and it will probably be useful to be able to post other things at some point?

I guess you mean you want to post a partial update?
That would be nice. Maybe just a general puglPostEvent() thing that can take an expose or whatever, since we already have a struct to describe exactly this and it will probably be useful to be able to post other things at some point?

I'm not sure if puglPostEvent() is not overdesign. A function call would do. Anyway: using expose events with an update region was my idea. So I added
void puglExpose(PuglView* view, int x, int y, int width, int height)

and used it for redrawing the area that needed updates when the play position pointer was changed. I adapted the event handling function to draw only the pixels within the expose event region.
However, the problem with my idea was that these expose events are asynchronous. So by the time the expose event was delivered, the play position pointer had already moved. So drawing only the pixels within the exposed region would usually not show the play position pointer at all, because it wasn't there any more.
So I implemented a different approach now. Instead of relying on expose events, and drawing during expose event, I now use
  puglProcessEvents (view);

  if (need_render)
    {
      puglEnterContext (view);
      render();
      need_render = false;
      puglLeaveContext (view, true);
    }

so after processing all events for a window, I directly update the screen using GL. So need_render = true is used instead of puglPostRedisplay (view). This seems to work on all platforms, and I can maintain both optimizations previously used. If I really get an expose event I can simply draw everything, because this is now rare.
I retested macOS, Windows and Linux and it seems that things are now fine. Things compile (also macOS 10.9). Keys work. Redrawing glitches are no longer there. As for Windows, I believe puglPostRedisplay might still be buggy. It seems somehow related to your elimination of the view->redisplay variable, but since I no longer use it, for me things work.
So the next thing probably would be adding support for resizing my own window maybe like puglPostResize (view, width, height), but thats for another PR.

I'm not sure if puglPostEvent() is not overdesign. A function call would do.

Fair enough, maybe... but I'm trying to work towards a stable API, so anything that reduces adding a bunch of functions (with maybe questionable prototypes) is appealing.  Also, what can I say, I like protocol-based design.  That said it would be a bit more annoying to call.
I guess you can do that, but drawing outside of the event loop really isn't how pugl is supposed to work (and it avoids some other optimizations), I'm not sure if it will cause problems.
It seems to me that you have two completely separate types of redraw: your internal surface (Cairo), and drawing to the window (GL).  They don't really have much to do with each other, if I understand your architecture.  So, when you get an expose event, you must draw the requested part of the window (this is just a requirement of the window system).  Which means you must draw your Cairo surface to the window with GL.  You can't skip pieces here, that will definitely break things.
Then there's the question of how much updating of the surface needs to happen before drawing it.  If nothing has changed in Cairo land, then you just draw the texture and don't do anything with Cairo at all.  If something has changed in the requested region, you need to update the Cairo surface first.  This is where the playhead invalidation and such comes in.

I believe puglPostRedisplay might still be buggy. It seems somehow related to your elimination of the view->redisplay variable, but since I no longer use it, for me things work.

It's possible, I've still mostly been testing top level windows (though I have a branch that adds embedding to pugl_test to ensure embedding cases work and stay that way), but I suspect the problems are related to the above, it seems to work fine for things that draw the requested area.

drawing outside of the event loop really isn't how pugl is supposed to work

Yes, I agree, and I wasn't really too happy with that approach, which is more of a workaround than a direct fix of the problem. So I've spent more time on debugging and understanding the issue, and once I really figured out what was the problem, it became surprisingly easy to fix:
The cause of all evil re-drawing bugs I observed under Linux was that before processing a configure event, I entered the GL context manually, and after processing the configure event, I left the GL context. But pugl already does this for me. So with my code, if a configure and an expose event where delivered by pugl, I left the GL context after the configure event, the expose event ran without GL context, the texture wasn't updated during expose, and redrawing started with a crappy initial state.
So long story short, it could be fixed in SpectMorph like this:
https://github.com/swesterfeld/spectmorph/commit/502a0d27a045ab533d4e9a11f92799f99b7dee1f
and incremental redrawing under Linux is now correct. Without any new API required.

I believe puglPostRedisplay might still be buggy.

Ok, so I've investigated my "Windows" redrawing glitches. Some problems arose when many widgets were created/deleted. For instance in SpectMorph this happens if you load a preset. Then basically the whole UI of the old preset is deleted (think 50 widgets), and all the new widgets get created. In general, if a widget gets deleted, the UI needs to be redrawn, so I call puglPostRedisplay in the wigdet destructor. Likewise if a widget gets created, I call puglPostRedisplay in the constructor.
I could do this with the old version of pugl, because puglPostRedisplay had two properties

it was extremely cheap (it used to set a boolean to true, and that is all it did)
the actual redrawing was only done at the end of the event processing - so the user only saw the UI at after all creating/deleting widgets was done

The new implementation in pugl_win.c looks like this:
void
puglPostRedisplay(PuglView* view)
{
        RedrawWindow(view->impl->hwnd, NULL, NULL,
                     RDW_INVALIDATE|RDW_ALLCHILDREN|RDW_INTERNALPAINT);
        UpdateWindow(view->impl->hwnd);
}

This appears to do actual drawing, which means that all temporary state will be shown. At least I can observe that the screen is updated again an again as widgets get destroyed and created. So

widget 1 gets deleted, calls puglPostRedisplay, and the screen is updated
widget 2 gets deleted, calls puglPostRedisplay, and the screen is updated
...

So the user can watch a lot of updating going on for seconds, until the final state is reached. I was able to fix this relatively easy by replacing this with
void
puglPostRedisplay(PuglView* view)
{
         InvalidateRect(view->impl->hwnd, NULL, FALSE);
}

which doesn't do any drawing. So then the events get merged.
Since now on at least Linux and Windows puglPostRedisplay is no longer so cheap, it might make sense in the toolkit to collapse all redisplay operations that belong to one pugl event that is being processed into one single call. However pugl knows better if all events have been processed, so maybe the right place for collapsing puglPostRedisplay calls may be pugl, like it always used to be.

I made it so aggressive to fix problems with the window not updating initially, and in some other cases with child views.  I agree it's too much the way it is right now, but it's not so simple :/
Coalescing redisplays would be nice, but the problem with just setting a bool is that it breaks non-continuously drawing apps since the wait function will never wake up.  So something needs to be posted.  If it's called in an event handler, then it can work that way and they can be processed at the end, but only if.  So I think we would either need to set a bool in the view to know that, or to have different functions for this.

I will try to merge my more advanced pugl_test that can demonstrate this stuff soon.


Not functioning context menu on Cornerstone tools

Right Click on a drawn tool causes popup of system context menu that either cover tool context menu or displayed side by side with it:

Select any procedure
select any image
-select tool type a and draw a tool ( any type e.g. angle)
Right click
-See behavior described above.
Reproduced on https://viewer-dev.ohif.org/

@dannyrb, we can close this now, right?

Appreciate the call-out. Thanks, @arthur-miller. @vicOnGit75, if you have issues with latest, please don't hesitate to report back.


more nfrs, hazard consideration, formatting, missing sections

a lot of line changes are just from auto-formatting aligning nested sections with tabs, no semantic changes were made to use cases
New

Explanation under certain headings
missing definitions addedd
added Usability section for NFRs
added Safety section for some NFRs in hazard analysis
added Databases FR section for DB reqs that were functional
Hazard Analysis requirements added (with rationale and fit criteria for nfrs)

SAR-1 -> Safety
SAR-2 -> Safety (modified, as upon thinking about it, our app should never really be presenting bright flashes or loud noises)
SAR-3 -> Left out (following from the above modification)
SAR-4 -> Realm Interface FR
SAR-5 -> Realm Interface FR
SAR-6 -> Maps FR
SER-1 -> Security
SER-2 -> Already Exists (PS-FR1, PS-FR2)
SER-3 -> Databases FR
SER-4 -> Security (Generalized)
SER-5 -> Profile FR
SER-6 -> Questionably not a req of the system
ROR-1 -> Object Placement FR
ROR-2 -> Maintainability
ROR-3 -> Databases FR
ROR-4 -> Reliability
ROR-5 -> Already Exists (MP-FR8)
ROR-6 -> Performance
ROR-7 -> Realm Interface FR - needs expanding
PVR-1 - Maps FR

Filled in Cost, Deadline, and Proof of Concept Sections
New Object Placement FRs
Added atleast one NFR to sections with none

Changed

auto-formatted to fix some formatting issues
makefile updated to use bibtex (fixed references)
interface names and descriptions moved to ui section
made some definitions more specific
groups changed to sub-realms where relevant
fixed formatting in some OUI, OS, and POG requirements sections
NFRs moved to relevant sections with rationale and fit criteria added (with the exception of some performance reqs)

MP-NFR1 -> Performance
MP-NFR2 -> Changed to FR
MP-NFR3 -> Generalized to a Settings FR
IV-NFR1 -> Performance
IV-NFR2 -> Changed to FR
IV-NFR3 -> Changed to FR
OS-NFR1 -> Performance
OUI-NFR1 -> Performance
POG-NFR1 -> Performance
POG-NFR2 -> Changed to FR
POG-NFR3 -> Usability (Generalized)
POG-NFR4 -> Usability (Generalized)

Removed

duplicate assumptions

Addresses
Closes #50
Closes #54
Progresses towards #43
Progresses towards #47

The commit names should include the relative file path, so in this case, you can add docs/SRS-IEEE.

The commit names should include the relative file path, so in this case, you can add docs/SRS-IEEE.

I don't think we set this standard anywhere, and we certainly haven't been following this thus far

I don't think we set this standard anywhere, and we certainly haven't been following this thus far

We can start following it now. It helps with the traceability of commits to files.

@Avanish-a please provide specifics on the problems introduced by these changes


Keras 2.4.3 does not calculate output shape in reshape layer

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Arch Linux
TensorFlow installed from (source or binary): conda
TensorFlow version (use command below): 2.2.0
Python version: tested in both 3.7.6 and 3.8.3
I tested both with a non-gpu version and with cudnn 7.6.5 + cuda10.1_0

Describe the current behavior
Output length of a reshape layer with last element of length unknown is marked as None
Describe the expected behavior
In previous versions of Keras, it was calculated from the other input shapes
Standalone code to reproduce the issue
The following code:
import keras
layer = keras.layers.Reshape([5, -1])
k_input = keras.Input((5, 6, 4))
model = keras.Model(inputs = [k_input], outputs = layer(k_input))
model.summary()

has different outputs depending on the version. In Keras 2.3.1 the last dimension of output is correctly deduced to be 24:
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_1 (InputLayer)         (None, 5, 6, 4)           0
_________________________________________________________________
reshape_1 (Reshape)          (None, 5, 24)             0
=================================================================

In Keras 2.4.3 the last dimension of output is None
 _________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_1 (InputLayer)         [(None, 5, 6, 4)]         0
_________________________________________________________________
reshape (Reshape)            (None, 5, None)           0
=================================================================

@fjhheras
Could you please try with tf.keras instead of just keras and let us know if it helps.
I ran the code on tf nightly and it works fine with tf.keras, please refer to gist and confirm.

With Keras 2.4.3, using tf.keras instead of keras still does not do the trick, I am afraid.
Please note that in the gist that you sent, tf.keras version is 2.4.0 (as shown by putting print(tf.keras.__version__) in the notebook), and thus earlier than the version I am having problems with (2.4.3)

Is this still an issue? Can you try executing your code in a virtual environment?
Works fine with keras 2.4.3 see the [gist].(https://colab.research.google.com/gist/ymodak/b889199dd9bc61bbd3321b9543d1676f/github_issue_41338.ipynb)
I also tried in my local with python 3.7.6 and keras 2.4.3.
See attached pycharm console screenshot.


Fix in resolving ssl certificate path in runserver_plus

Coverage decreased (-0.0007%) to 68.321% when pulling b6f9a703c3bcea1ee678a2e38e8d23d836b3b5b2 on domandinho:domandinho_issue#1282 into 6df19dce3ea04338837bac712d5adc81fc15b31b on django-extensions:master.

Coverage decreased (-0.0007%) to 68.321% when pulling b6f9a703c3bcea1ee678a2e38e8d23d836b3b5b2 on domandinho:domandinho_issue#1282 into 6df19dce3ea04338837bac712d5adc81fc15b31b on django-extensions:master.

@trbs
What about this PR?

Other then the 1 minor style comment, I like the PR and is +1 LGTM :-)

@trbs
Fix is done.


feat(metrics): support for direct metric publishing

... without going through Spring events first

I still need to test it but I wanted to get some feedback about a few things


FoundationEssentials: use the correct flags on Windows

Windows spells their open flags differently. Ensure that we use the Windows spellings on Windows to get the correct value.

@swift-ci test


LIBCLOUD-760 malformed xml requests

Fixed 2 bugs raised by @EfrenRey where delete Nat rule, edit firewall rule are not containing the right capitalization of Rule in the XML request.

/cc @erjohnso
I assume you will merge this when rolling out a new 0.19.0 release, right?

Yes. From the thread on dev, we'll close that vote thread and @tonybaloney is planning on adding some basic tests. That can go in this PR too.

@erjohnso these are the required tests to check the request bodies as well for typos. all done

Outstanding, thank you @tonybaloney!
I'll get this merged in later tonight and start working on the revised 0.19.0 release.


pcapgo: add read/write support for Decryption Secrets Block (DSB)

Prebiously tracked in Google repo as: https://github.com/google/gopacket/pull/1042
Originally submitted by: @cfc4n

Thanks @mosajjal,
I have fixed the imports now.

Codes and tests look good. just need to convert the pcap files used in testing to the actual bytes of each packet to make it consistent with other tests. refer to previous PRs or _test files.

Thanks @mosajjal for the review.
Could you elaborate a bit? It seems like also the other tests read-on the pcapng files:
https://github.com/gopacket/gopacket/blob/ee81b8c880dac66412e96a9f0ef45186380edc51/pcapgo/ngread_test.go#L69

to me the preference is to use byte stream rather than a pcap file to test out a particular packet. but if that's not possible for this use case that's fine too. is there any other changes needed for this or is it ready for merge now? Happy to merge it anytime :)

I think we are better of with the PCAPng files here as we dont test against packet data but rather against the DSB sections within the file.
From my side its ready to merge.


Request: offer drag&drop of files, to be inserted into Download folder

In the case of APK file/s, it should ask if we want to install or push to the folder.
We could also have right click drag&drop, for more options that you might think about

EDIT: Oh I see it's set to the main primary storage. Closing this as it's customizable anyway...

In the case of APK file/s, it should ask if we want to install or push to the folder.

Just for reference: https://github.com/Genymobile/scrcpy/pull/226#discussion_r209454655

@rom1v Wait, this is actually important.
For dragging of APK files, I think it should be possible to either install/push.
Is it possible ?
Maybe I should re-open this and change the request?
Right-button dragging should allow us to choose what to do (using a context menu) : install/push.
As for all types of files, maybe call an Intent to open them or share them (upon right-button dragging).

OK updated the request. I think now it's better.