Detections in own point clouds

Hello,
What is the required orientation of the extracted point clouds? I tried running the PointNet on a different extracted frustum of points, but the results were strange.
I tried to visualize the point clouds in the pickled files, and it appears that the frustum direction is oriented along the z-axis. From this I can infer that the points are in camera coordinates, rather than lidar coordinates. Can you confirm this? Does the network only work like this?
I am using a batch size of 1, with 1024 points, and do something like this:
            class_vector = np.zeros((1, 3))
            class_vector[0, 1] = 1

            point_cloud = np.zeros((1, 1024, 4), dtype=np.float32)
            indices = np.arange(0, len(frustum))

            if len(frustum) > 1024:
                choice = np.random.choice(indices, size=1024, replace=True)
            else:
                choice = np.random.choice(indices, size=1024, replace=False)

            point_cloud[0] = frustum[choice]

I randomly sample 1024 from the extracted frustum (better ways?) and feed this along with the other two placeholders for class ID and training phase. I give the point cloud in camera coordinates. In the example here, I put it as a pedestrian:
https://imgur.com/a/yWo0nn1
            feed_dict = {
                self.pointclouds_pl: point_cloud,
                self.one_hot_vec_pl: class_vector,
                self.is_training_pl: False,
            }

            batch_logits, batch_centers, \
            batch_heading_scores, batch_heading_residuals, \
            batch_size_scores, batch_size_residuals = \
                self.session.run([
                    self.logits, self.center,
                    self.end_points['heading_scores'], self.end_points['heading_residuals'],
                    self.end_points['size_scores'], self.end_points['size_residuals']],
                    feed_dict=feed_dict)

            batch_seg_prob = softmax(batch_logits)[:, :, 1]  # BxN
            batch_seg_mask = np.argmax(batch_logits, 2)  # BxN
            mask_mean_prob = np.sum(batch_seg_prob * batch_seg_mask, 1)  # B,
            mask_mean_prob = mask_mean_prob / np.sum(batch_seg_mask, 1)  # B,
            heading_prob = np.max(softmax(batch_heading_scores), 1)  # B
            size_prob = np.max(softmax(batch_size_scores), 1)  # B,
            batch_scores = np.log(mask_mean_prob) + np.log(heading_prob) + np.log(size_prob)

            filtered_frustums.append(point_cloud[batch_seg_mask == 1].astype(np.float3

However, when I tried to visualize the points belonging to the object, it doesn't look right...
https://imgur.com/a/3MrEBGQ
Is there some further normalization that I need to do to the point cloud? I looked into the test.py but I can't really tell. Am I misinterpreting what the batch_seg_mask is?
Anyhow, great work!

Hi @fferroni
Yes, we are working on the KITTI rectified camera coordinate system. If you want to test on another dataset, you need to verify that the axis directions are consistent with KITTI and make sure the camera height is similar to KITTI.
btw, the links provided seem not to work anymore.

Hi @charlesq34, thanks for the fast reply. I fixed the camera height and used the same axes and now it works pretty well ;-)
Can you comment on sampling strategies for the frustum points?

In cases where there are fewer points than the nb_points parameter in the network, is it better to put them all to zero, or oversample the actual points?
In cases where there are substantially more points than the nb_points parameter, can you comment on optimal sampling strategies? I am doing randomly, but am wondering how you got the performance in the KITTI benchmark?

BR,
Francesco

Hi @fferroni
No problem -- glad it works.
we'd better oversample the actual points by repeating the existing points (by randomly choices) -- putting them to zero introduces an un-wanted bias to the estimation.
random sampling is mostly fine (we used that for our evaluation) however it is surely not optimal. you are welcomed to try different ways to sample -- if you care more about faraway objects, maybe keep more points in distance
hope it helps

@fferroni,
Did you setup camera structure similar to kitti? Did you do the calibration(cam2lidar) separately?
@charlesq34 , When Detections in own point clouds using kitti's training data, Due to the difference in camera and lidar position, I do not think it will be estimated correctly. Right? So the only way is to use my data to train(include labelling) and detect? What do you think?

Can I use point cloud obtained from a Kinect camera along with this algorithm?

Hi @fferroni ,
Did you fix the camera height to 1.65m as described in the KITTI paper? I also have the same problem you met before. The output of 3d bounding box is strange when I was using a realsense camera sitting on the desk.

Hi @charlesq34, thanks for the fast reply. I fixed the camera height and used the same axes and now it works pretty well ;-)
Can you comment on sampling strategies for the frustum points?

In cases where there are fewer points than the nb_points parameter in the network, is it better to put them all to zero, or oversample the actual points?
In cases where there are substantially more points than the nb_points parameter, can you comment on optimal sampling strategies? I am doing randomly, but am wondering how you got the performance in the KITTI benchmark?

BR,
Francesco

Hello, thanks for your idea of sampling. I also found that random sampling may not be the best sample strategy for point cloud with more points that KITTI. Can you recommend some sample strategy for dense point cloud? I am working on how to make it better.


Support different databases

Any one interested in getting this to work for different kinds of databases?

MSSQL

PostgresSQL

NoSQL or MongoDB

Any one interested in getting this to work for different kinds of databases?

Which databases are supported at the moment? I think adding this information to README.md would be useful.
I would be interested in SQLite. Thanks


[FR] add support for listing users by provider id

I have a large list of users with mixed provider id, some are anonymous, some authenticated by Google and so on. When we have to fetch all users authenticated by Google, we need to fetch all the (say ~ 1M) records and then loop through the users to checks it's provider id.
It would by good to add a method find by provider id.

Hey @mdneyazahmad, I think you are basically asking for support for filtering/advanced querying options in our listUsers API. I don't think we have existing plans to support this on our backend. You may want to file an official feature request for it.

Thank you @bojeil-google for your quick response. I want to know, if we can achieve the same in the current version of the library? If not, I will find another method. I am going to close this issue.

Thank you @bojeil-google for your quick response. I want to know, if we can achieve the same in the current version of the library? If not, I will find another method. I am going to close this issue.

It is not possible in the current implementation. You would need to download the user data and search through it based on your own criteria.

Thank you @bojeil-google you have saved my time in researching to fix this issue. I am closing the issue.


use the carrier and service_level stored from the shipment instead fo…

…r are more up-to-date info
@bryanmtl there isn't really much else to add here but this since we decided to store the carrier and service_level in the shipment table.

@mickeyren This looks right to me. Did this change break the test suite?

@bryanmtl i'll chat with @aamyot and see how i can run the spec suite here.
but i also modified the code here - it will fallback to the shipping method's carrier and service_level incase it is absent from the shipment as a fail-safe.


Processor wrapper for LiteX integration

Finally (😅) the CPU core seems to be in a tidied-up and "optimized" state so I would like to come back to the LiteX framework. @enjoy-digital already did some awesome work and provided a first CPU-only NEORV32 integration (-> #115).
As discussed in #115 I would like to extend this initial setup. Instead of just integration the NEORV32 CPU core I would like to provide a "core complex" for the integration into LiteX. This core complex includes the CPU (obviously) and also some "important" processor modules. I am not talking about peripheral modules like UART and SPI - these can be implemented using the LiteX system. I am thinking about a core complex that implement CPU-close modules like the instruction cache, the RISC-V machine system timer and the on-chip debugger. For that I am proposing a new system wrapper in rtl/processor_templates (maybe neorv32_litex.vhd) for implementing the LiteX NEORV32 "core complex".
However, there are a lot of issues I would like to discuss first.
Interface
The initial LiteX port by @enjoy-digital uses a wrapper for converting the CPU's native interface into a Wishbone-ish interface. The NEORV32 already provides a Wishbone bridge, which could be used as interface to the LiteX framework.
This is a single interface that is used for fetching instructions and for reading/writing data to a unified address space. Bus traffic can be reduced by using the NEORV32 instruction cache.
Questions:

Which Wishbone protocol? "Pipelined" or "classic"? In case of the NEORV32 the only difference here is the handling of the STB signal (see figure six in Data Sheet - Processor External Memory interface).
What about maximal access latency? Is there an upper bound (let's say 2000 clock cycles) after that a pending bus access will be acknowledged (for sure!)?

Configuration
According to https://github.com/enjoy-digital/litex/wiki/CPUs#soft-cpu-variants LiteX provides several abstract configuration options like "lite", "standard" and "full". To simplify the configuration of the NEORV32 core complex I propose to use as few generics as possible. One could be a simple integer to pass "lite, standard, ..." to the complex. Based on this integer the complex defines all the NEORV32 generic (like cache configuration, ISA extensions, CPU-close modules like the timer, etc.).
The question is how to map those configuration options. "Full" is pretty obvious, but what about "standard" and "lite"?
Address Space
The CPU-close modules of the core complex (like the machine timer) are implemented as memory-mapped devices. All these memory occupy an address range starting at 0xFFFF0000. This 64kB-wide address space would NOT be available for the LiteX system. Question here: is this a problem? And what about the boot/reset address of the CPU? Is this flexible or can be hardwire this to some fixed value?
Btw, a user could still "fine tune" all the available configuration options by modifying the core complex VHDL file.
More to come...
I just realized this might be way to much information for a single issue... 😅 Maybe we can "outsource" some things into the discussions section.

Hi @stnolting,
great to see you are interested to improve the initial work. Here are some first answers:

Interface

LiteX supports both Wishbone "Pipelined" and "Classic" and from the description of your implementation, the implementation seems similar in LiteX. Some peripherals are able to pipeline accesses (ex SRAM), the others will just hand the accesses in a minimum 2 cycles. The SoC implements a timeout that will ack the access for the CPU if no peripherals responds. This timeout if configurable and defaults to 1e6 cycles.
Note: If this can be useful for NeoRV32: LiteX also supports AXI-Full/AXI-Lite buses and can automatically do the bridge between standards/data-width.

Configuration

We used lite/standard/full  on some CPUs, but this is not mandatory and we have used other names some recent CPUs (ex FemtoRV: https://github.com/enjoy-digital/litex/blob/master/litex/soc/cores/cpu/femtorv/core.py#L16-L24). We can now also expose/pass command line parameters from/to the CPU (ex VexRiscv-SMP: https://github.com/enjoy-digital/litex/blob/master/litex/soc/cores/cpu/vexriscv_smp/core.py#L57-L77), so there are different possibilities/way to do the configuration.
We could just used the variants that make sense for you (and with names that you could also define).

Address Space

This is flexible: The CPU wrapper can enforce part of the address mapping to force the SoC to use it. For the reset address, this is also flexible: LiteX can configure it on the CPU or use a fixed value provided by the CPU.

VHDL/Verilog:

This was not a question, but just for info, we are also trying to simplify VHDL -> Verilog conversion in LiteX and do it automatically when the toolchain does not support VHDL (ex for simulation with Verilator or with open-source toolchains). This should benefits NeoRV32 since LiteX should be able in the future to transparently convert the VHDL to Verilog without having to implement this manually in the CPU wrappers as has been done here: https://github.com/enjoy-digital/litex/blob/master/litex/soc/cores/cpu/neorv32/core.py#L178-L192. This should also simplify creation of variants based on different VHDL files.

Thank you so much for the detailed answers!

Interface

I think we should start with simple Wishbone - maybe in the "pipelined" version as I am planning to provide real pipelined/burst transfers in the future... AXI also sounds great, but I think for now this would be unnecessarily complex and might also increase latency (my AXI bridge is not optimized and just supports the minimal AXI-Lite protocol).
A timeout of 1e6 cycles is quite a lot 😅 I think it would be good to enabled the processor's "bus timeout feature" (with a timeout value less than 1e6) to raise an exception if there is no response. This is just me, but this feature helped me a lot when debugging more complex system setups with lots of memory-mapped peripherals.

Configuration

It is great to have so much flexibility here - but at the same time this makes it more complicated to think about all the different flavors :wink: At first we should agree on a maximal set of features for the "core complex" and then we can define 2 or more less-complex versions.

Address Space

This is great, I was not aware of this. Having a fixed boot address of all-zero would be the simplest solution I think.

VHDL/Verilog

Yeah, I have seen that - really great work! :+1: I think a lot of VHDL-based setups can benefit from that.
Setup Questions
I am still thinking about how to provide the NEORV32 core complex in terms of this repository. Having a single top VHDL file would be fine, right? Do we need to have some additional "auxiliary" files there (e.g. for telling LiteX about boot address, bus timeouts, etc.) or will this be in enjoy-digital/litex/tree/master/litex/soc/cores/cpu/neorv32?
Btw, I just had a discussion with some colleagues about LiteX and we were arguing how to pronounce it correctly... 😅 Is it "Lite-X" or "Li-Tex"?

LiteX is pronounced Lite-X :)

Thanks for solving this mystery 😅
I have started writing a core wrapper providing four configurations: minimal, lite, standard and full. The details of each config can be discussed when I have a PR available. However, I am wondering what the most convenient passing of the configuration might be: a simple integer that selects 0..3? Or would you prefer an actual string to be passed to the wrapper?
What about configuration options for clock speed (which is not actually required here) and for the on-chip debugger? A simple integer for the clock and a boolean parameter for the OCD?
I have seen some cores use something like standard+debug - this can be resolved into unique VHDL generics, right?
Another question: I would like to keep things simple at first. However I would like to add a single interrupt input (which is the RISC-V "machine external interrupt"). I have seen other cores do also provide a single interrupt. Does it trigger on rising/falling edge or on low/high level?

Hi @stnolting,
sorry for the delay, good progress! For the parameters, we have lots of flexibility in the wrappers with the use of Python, so the parameter format is really up to you and what is the easiest for you. I'm fine with your suggestions. We basically just need to know how to configure the core and how to do the variant -> configuration mapping in the wrapper.
For the debug, we only have this on VexRiscv and this could also specific configuration to do on the instance.
For the interrupt, the other CPUs that don't have a specific interrupt handler are using high level on the interrupt signal to indicate an interrupt.

The minimal and lite variants seems to be working fine with: https://github.com/enjoy-digital/litex/commit/1f1d9f5c0e0b89a6cf60cc1c5be555e0bd49ecc6 :):
litex_sim --cpu-type=neorv32
        __   _ __      _  __
       / /  (_) /____ | |/_/
      / /__/ / __/ -_)>  <
     /____/_/\__/\__/_/|_|
   Build your hardware, easily!

 (c) Copyright 2012-2022 Enjoy-Digital
 (c) Copyright 2007-2015 M-Labs

 BIOS built on Jun 29 2022 15:49:14
 BIOS CRC passed (68a3867e)

 LiteX git sha1: 1f1d9f5c

--=============== SoC ==================--
CPU:		NEORV32 @ 1MHz
BUS:		WISHBONE 32-bit @ 4GiB
CSR:		32-bit data
ROM:		128KiB
SRAM:		8KiB

--============== Boot ==================--
Booting from serial...
Press Q or ESC to abort boot completely.
sL5DdSMmkekro
Timeout
No boot medium found

--============= Console ================--

litex>

Standard and Full variants do not seem to be working, but I haven't investigated yet. This could be related to the icache and should not be complicated to sort out together.
Unfortunately, GHDL/Yosys flow does not seem to be able to keep parametrization on Verilog converted from VHDL, so LiteX will have to directly modify the default values of the CONFIG/DEBUG generics in neorv32_litex_core_complex to be able to configure the core complex.

Wow, that was fast! Great to see the wrapper is working! :)

Standard and Full variants do not seem to be working, but I haven't investigated yet. This could be related to the icache and should not be complicated to sort out together.

I have tested the standard and full variants - so at least the core itself seems to be fine. However, I do not think this problem is caused by the bus accesses since the cache also uses simple Wishbone transfers (no bursts). I think there might be a synchronization problem. I do not claim to understand the complete boot process of LiteX but maybe the instruction cache contains outdated data at some point.
I saw the prototypes for cache flush functions in litex/soc/cores/cpu/neorv32/system.h. Maybe we need to add the instruction cache flush mechanism here:
static inline void flush_cpu_icache(void){ __asm__ volatile ("fence.i"); };
static inline void flush_cpu_dcache(void){ __asm__ volatile ("fence");   }; /* No data cache yet */

Btw, the data cache flush function is just a placeholder yet.

Unfortunately, GHDL/Yosys flow does not seem to be able to keep parametrization on Verilog converted from VHDL, so LiteX will have to directly modify the default values of the CONFIG/DEBUG generics in neorv32_litex_core_complex to be able to configure the core complex.

Too bad but I know these flaws... How can we circumvent this? Several "instances" for each config combination? Or maybe some different parametrization style to "trick" yosys?

In a next step it might be nice to add toolchain/gcc support for the configurations above "minimal":
litex/soc/cores/cpu/neorv32/core.py:
    "minimal":          "-march=rv32i     -mabi=ilp32",
    "lite":             "-march=rv32imc   -mabi=ilp32",
    "standard":         "-march=rv32imc   -mabi=ilp32",
    "full":             "-march=rv32imc   -mabi=ilp32",

But I can also do that via a PR later on 😅

Hi @stnolting,
thanks for the info. I'll probably first look at allowing the variants selection and we could then try to understand together the issue when enabling the cache. For the parametrization, I'll probably cheat a little bit: make the LiteX build modify the VHDL wrapper's initial values before converting it to verilog. This is not very clean but will avoid creating another wrapper around the...wrapper.

I imagine that parameters are first fixed/resolved before doing the VHDL -> Verilog conversion because propagating parameters through conversion would probably be really challenging. So I haven't' really looked at the code but can guess it's the reason parameters with non-default values are not allowed at the top level.
Using constants would indeed make things easier for replacement with LiteX before the Verilog conversion, if you can do that in the wrapper, I'll implement the replacement.

I have removed the generics and now two constants are used to select the configuration:
-- configuration select ----------------------------------------------------------------------------
constant CONFIG : natural := 0;     -- configuration select (0=minimal, 1=lite, 2=standard, 3=full)
constant DEBUG  : boolean := false; -- enable on-chip debugger, valid for all configurations
-- -------------------------------------------------------------------------------------------------

I have also found (and fixed!) a bug in the Wishbone bus interface (https://github.com/stnolting/neorv32/pull/367). I'll need to do some more testing but I think the problem with the configurations using the i-cache should be resolved :wink:

Yeay! Awesome! 👍😄
I'll do a PR enjoy-digital/litex to add cache functions and interrupt management and I will link the LiteX setup in the NEORV32 documentation.
I think we can close this now. Thanks again for all your work and help!

Great that you've been able to reproduce the results! More that happy to collaborate on this and eventually go further if needed.

Btw, I have never worked with GitHub wikis so far, but can I add NEORV32 to https://github.com/enjoy-digital/litex/wiki/CPUs?

Yes, you should be able to edit the wiki directly, please let me know if you have troubles.

Works without problems: https://github.com/enjoy-digital/litex/wiki/CPUs#risc-v---neorv32 :+1:

Hi @stnolting,
We are currently using LiteX to implement a SoC, and we'd like to use last version of your softcore implementation.
It seems that the current version of the litex_core wrapper does not include the latest core upgrades. Do you plan to update the litex wrapper with the current core version?
Thanks!

Hey @kalyze!
Seems like I have not updated those system integration wrappers in a while... 🙈
Which specific configuration option(s) are you missing?

By now we have the core running but we have some problems in getting the jtag debugger working.
After debugger connection we can halt and resume program execution. When the execution stops, gdb tells that the core stopped on address 0x00000000 but our program starts on address 0x40000000. Breakpoints are also ignored.
I already modified neorv32/core.py in litex library to enable debugger peripheral and assign jtag signals to some gpios.
I tryed to update the core to a newer version, hoping to fix this problem but ghdl failed in generating verilog. I saw your integration is considerably changed since the last litex system integration.
Have your ever had similar problem with the debugger module before? Do you now how can I fix them?
Thanks!

Hey @kalyze!
The debugger itself works quite flawlessly so far :wink: However, I have never tested it inside a LiteX-setup.
What configuration of the NEORV32 core complex are you using? Have you tested a setup without caches?

Hi @stnolting,
by now I've only tested the core configuration you give in the LiteX wrapper. I only modified it to connect jtag signals.
After some tests, I'm still in trouble on getting the breakpoints working. I'm also not able execute program step-by-step.
I haven't tested a setup without caches yet. How can I test it?

To disable the caches for all configurations set all entries in these two lines to false:
https://github.com/stnolting/neorv32/blob/c7590c9cc86b1646926f57d0ab0ab610e0471af8/rtl/system_integration/neorv32_litex_core_complex.vhd#L153
https://github.com/stnolting/neorv32/blob/c7590c9cc86b1646926f57d0ab0ab610e0471af8/rtl/system_integration/neorv32_litex_core_complex.vhd#L157
Which version of NEORV32 are you using?
Can you post the openOCD messages when connecting to the core and when manually halting/resuming the CPU?

Hi @stnolting,
we finally got a working NEORV32-LiteX configuration with the debugger!
We updated the NEORV32 core to version 1.8.9, fixed memory map in core.py and now we have the debugger working correctly.
If you are interested, here you can find our modifications:
https://github.com/protech-engineering/litex/blob/protech/litex/soc/cores/cpu/neorv32/core.py
If you don't find anything wrong, we can open a pull request to update official LiteX repository.
Thanks for the support!

we finally got a working NEORV32-LiteX configuration with the debugger!

Great to hear! 👍

We updated the NEORV32 core to version 1.8.9, fixed memory map in core.py and now we have the debugger working correctly.

What exactly did you fix? I think the VHDL file list was outdated, but what about this memory map?

If you don't find anything wrong, we can open a pull request to update official LiteX repository.

That would be awesome! 😃
Thank you very much for updating/fixing!

Hi @kalyze,
thanks for fixing the LiteX integration! Happy to review/test/merge a PR in LiteX if you submit it.

What exactly did you fix? I think the VHDL file list was outdated, but what about this memory map?

Hi @stnolting,
in order to make the core working, we had to move io_regions in LiteX "core.py" to a non cached memory region. Then we changed the "memory_map" property to map csr to the same region, to avoid LiteX build errors.

Thanks for updating! 👍


Update docs Introduction section

Which feature you want to improve?
Documentation Introduction section.
What and why you want to improve that feature
It's quite short and doesn't really "buy" potential users.
Points to improve:

Write better wording and explain the ideology of the project briefly.
Try to fix the online playground to enable hot reload

Potential bugs/ unexpected behaviors
N/A
How to implement the changes
Check the examples of great documentation and implement similar techniques.
Others
N/A

@nvh95 as an additional suggestion is to remove Features section completely and add it to the Introduction section.
Will add as separate commit, so we can revert easily if you won't like it ;)

@skirianov

Write better wording and explain the ideology of the project briefly.

Absolutely agree

as an additional suggestion is to remove Features section completely and add it to the Introduction section.

I am not very sure at the moment. But I still like the idea of having a separate Features section. We can still mention the features in the Introduction. But if someone doesn't want to read the long text, they can just navigate to the Features page.
Similar docs that have Features pages:

https://vitejs.dev/guide/features.html
https://vitest.dev/guide/features.html
https://nextjs.org/docs/basic-features/pages

What do you think?

@skirianov Wow. It looks incredible 💪💪🦾🦾. Indulge me with your draft PR 😹😹


Update cron-utils (#266)

Update cron-utils

Signed-off-by: Ashish Agrawal<EMAIL_ADDRESS>Issue #, if available:
N/A
Description of changes:
Backport updating cron-utils
CheckList:
[X] Commits are signed per the DCO using --signoff
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

:exclamation: No coverage uploaded for pull request base (1.x@86da5a8). Click here to learn what that means.
The diff coverage is n/a.

@@          Coverage Diff           @@
##             1.x     #269   +/-   ##
======================================
  Coverage       ?   78.61%
  Complexity     ?      218
======================================
  Files          ?      173
  Lines          ?     6968
  Branches       ?      915
======================================
  Hits           ?     5478
  Misses         ?     1003
  Partials       ?      487

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 86da5a8...74d65a3. Read the comment docs.


Population code for acoustic and possible datras and eggs/larvae

https://github.com/ices-eg/wg_WGACOUSTICGOV/issues/48

https://github.com/ices-taf/advice-dept/issues/197

Who should we contact when you talk about ACOM LS/Advice/Data C? and, what is a SWOT? We need to upload the data for the HERAS acoustic survey soon and need this new code.

The ACOM LS (leadership or vice-chairs) can be found here: https://www.ices.dk/community/groups/Pages/ACOM.aspx
For Advice I think you should get in touch with Lotte Worsøe Clausen, since she is listed as the professional sec for the WK -  and for the Data Centre it should be Adriana/Hjalte - I know Adriana has had her eye on this, and put it forward within the reference management group.
A SWOT stands for Strength, Weakness, Opportunities and Threat analysis
If you have any other questions please contact me or Adriana.

Now clarifiying the process. WKSIDAC2 should make a recommendation, which would then be considered by SCICOM/ACOM as part of the usual process of review of recommendations. This would be the sense check and documentation of a change - exactly how we do it for new stocks/split of stocks.
This would then be taken up by the vocab governance to implement in consultation with WGACOUSTICGOV.

The recommendation ia already there:
https://community.ices.dk/admin/Recomendations/_layouts/15/start.aspx#/Lists/Recommendations 2023/AllItems.aspx#InplviewHash688c456e-54b2-4425-8ea8-3e6862b509c5=FilterField1%3DEG-FilterValue1%3DWKSIDAC2
I had a quick chat with @florianberg during WGIPS and he confirmed that Codes and descriptions will not need to be modified. When a population changes, a new code/description will be requested. The previous codes should be deprecated so they can't be used for upload and some info about history of the population will be stored in the long description of the new codes (and info on the new codes in the old codes long description).
They are happy with the RMG suggestion of ASFIS+3-4random numbers.
@florianberg can you provide better descriptions for the population codes you are requesting? if there was some relevant link to publications or reports that would be good. Thanks
Herring.baseline.samples (3).xlsx

Hei,
The latest overview of populations were published in the WKSIDAC report and corresponding Excel file:
ICES. 2024. Second Workshop on Stock Identification and Allocation of Catches of Herring to Stocks (WKSIDAC2; outputs from 2023 meeting). ICES Scientific Reports. 6:5. 67 pp. https://doi.org/10.17895/ices.pub.24998747
I can provide a more detailed description if the information from the Excel file attached to the WKSIDAC report is not enough.

Spatial_distribution_baselines.csv

@odontaster
Please check with Lotte on the outcomes of the workshop reg. approval from ACOM.
Create the new codetype and codes as suggested. Add species reference to the descriptions.
Ask Florian if ICES codes could be published in the doc above.

Final list of codes and descriptions, reviewed by WKSIDAC2 chair and agreed also by HAWG
ICES_vocab_GeneticPopulation_FB_March2024.xlsx


<IP_ADDRESS> beta Firmware

Updating to <IP_ADDRESS> beta firmware causes the camera to not detect the NFS connection as an SD card even though the mount is still active and valid.
Downgrading to <IP_ADDRESS> works fine.

Confirmed, same fw version v2 cam

yes, i did a brief look and it seems they made some changes to the bootstraping code, will see if i can get over that.

@HclX any luck with this? Looks like its happening on the new stable firmware too.

I've drafted release 0.4.04 to support latest stable firmware (4.x.6.218), please try it and let me know if it works.

I built 0.4.04 from the dev branch and ran the remote_install.sh, but the cameras don't appear to be pulling the update.  Update from 0.4.03 on master still works though.

Yes there is a bug causing it not working when built from general linux environment. I've made a fix for that.

confirm latest firmware working with fixed made to dev branch! thanks


Stabilize a condition

Proposed Changes
While looking into the (non-)race condition described in #125, I noticed a potentially related issue.
As per the documentation of realpath():

realpath() returns FALSE on failure, e.g. if the file does not exist.

In the rare case that realpath() would return false within this plugin, false would subsequently be passed on to is_dir() and is_readable() which both expect a string path.
This tiny change makes the condition fail earlier by checking the return value of realpath().

@Potherca Just checking: where there any changed you wanted ? Or did you assign this to me to self-merge ?
(sorry, the commit process in this repo still confuses me - I'm used to never self-merge when collaborating, i.e. the (last) approver merges)

No changes needed. I just didn't feel like waiting around for Travis to complete but also didn't want to use my admin right to merge again. I assigned you in case I forgot to come back to this :joy:


Fix backtrace access to illegal memory

I tested the patch both in teaclave and seperately. It can fix the panic catch failure.


npm install-ing Wagtail creates undesired changes in package-lock.json

Initially reported by @gasman in https://github.com/wagtail/wagtail/pull/3773#issuecomment-325625388.
Issue Summary
On a fresh npm install, npm updates the package-lock.json file depending on the context the command is ran in. Those changes to the lockfile reflect which packages were installed, which can differ because of:

A difference in package versions available, if the package.json uses ranges (https://github.com/npm/npm/issues/16866, https://github.com/npm/npm/issues/17979, https://github.com/npm/npm/issues/18103).
A difference in the install environment (eg. installing on a different OS than the one the lockfile was created in). This is https://github.com/npm/npm/issues/17722.

In practice, this is problematic because it creates unnecessary file changes to deal with.
Steps to Reproduce

Run npm install on an Linux machine, with the current OSX-generated package lock.

Technical details

npm >= 5.1.0

The first problem can be resolved once and for all by dropping all range specifiers from the package.json (^, ~), I don't know of any "once and for all" solution for the environment difference.
The environment difference shouldn't be more than an annoyance, however the dependency version problem can cause real issues if a new version of one of the dependencies breaks the code/tooling (using a lockfile is supposed to prevent this), so we may want to pin all of the dependencies to exact versions until it is resolved.
Until we find a more proper fix, it is safe to either:
# Discard the changes to the file:
git checkout -- package-lock.json

# Use the following flag to prevent the changes to the file:
npm install --no-shrinkwrap

I've been tracking all of the npm issues highlighted above, hoping for a fix in npm, if that doesn't come I guess we can go for a plan B solution like "switch to yarn", or "move back to npm-shrinkwrap.json and gitignore the package-lock.json".

Related: #3442

From https://github.com/npm/npm/issues/17979#issuecomment-332701215, point 1. above seems to be addressed in a recent npm upgrade. Trying this briefly with npm 5.4.2, this seems to be addressed.
Point 2. is still pending.
Point 3. – I get a few more diffs, not sure whether they are related to the recent React update, or just to the fact that I use a newer version of npm:

HTTPS -> HTTP for some packages in the resolved field.
dev: true added or removed in a few places
string-decoder changing locations.

According to http://blog.npmjs.org/post/167963735925/v560-2017-11-27, this has been fixed in the latest release. I'll wait a few more days before upgrading (😉), and follow up here.

Yep, this is now fixed. I'm still getting the string-decoder changes, but it's just moving the location of this package's definition from one spot to the next as far as I can tell. Will address that.


python37Packages.gtts: 2.2.1 -> 2.2.2

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for python37Packages.gtts is: "A Python library and CLI tool to interface with Google Translate text-to-speech API"
meta.homepage for python37Packages.gtts is: "https://gtts.readthedocs.io"
meta.changelog for python37Packages.gtts is: ""
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact

Checks done (click to expand)

built on NixOS

Warning: a test defined in passthru.tests did not pass

0 of 0 passed binary check by having a zero exit code.

0 of 0 passed binary check by having the new version present in output.

found 2.2.2 with grep in /nix/store/l7vg08dp2fbdb6vg60np9cxzhk2nyvlm-python3.7-gtts-2.2.2

directory tree listing: https://gist.github.com/35a01b3dfa6c50e1fec8a9f010570d6c

du listing: https://gist.github.com/7e0b5d9742c9f44ed1b8c42a73919fcf

Rebuild report (if merged into master) (click to expand)

13 total rebuild path(s)

4 package rebuild(s)

3 x86_64-linux rebuild(s)
3 i686-linux rebuild(s)
4 x86_64-darwin rebuild(s)
3 aarch64-linux rebuild(s)

First fifty rebuilds by attrpath
mnemosyne
python37Packages.gtts
python38Packages.gtts
python39Packages.gtts

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/l7vg08dp2fbdb6vg60np9cxzhk2nyvlm-python3.7-gtts-2.2.2 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A python37Packages.gtts https://github.com/r-ryantm/nixpkgs/archive/ede8ff5efb1a2eb5bc2b5aca7532c5b430c18588.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/l7vg08dp2fbdb6vg60np9cxzhk2nyvlm-python3.7-gtts-2.2.2
ls -la /nix/store/l7vg08dp2fbdb6vg60np9cxzhk2nyvlm-python3.7-gtts-2.2.2/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

3 packages built:

mnemosyne
python37Packages.gtts
python38Packages.gtts

Maintainer pings
cc @unode for testing.

Result of nixpkgs-review pr 112380 at ede8ff5e run on aarch64-linux 1

1 package failed to build:

mnemosyne

2 packages built:

python37Packages.gtts
python38Packages.gtts

1 suggestion:

warning: python-explicit-check-phase
Consider using pytestCheckHook in checkInputs rather than checkPhase = "pytest";.
Near pkgs/development/python-modules/gtts/default.nix:41:3:
   |
41 |   checkPhase = ''
   |   ^

@ryantm I can't seem to suggest a change outside the view of the patch nor push to the PR branch.
Can you include the following commit in the pull request? Thanks!
From 4fcdc76f9087915f24f32a05d5bd9798727bf56a Mon Sep 17 00:00:00 2001
From: Renato Alves<EMAIL_ADDRESS>Date: Mon, 8 Feb 2021 16:50:37 +0100
Subject: [PATCH] python37Packages.gtts: Add changelog link

---
 pkgs/development/python-modules/gtts/default.nix | 1 +
 1 file changed, 1 insertion(+)

diff --git a/pkgs/development/python-modules/gtts/default.nix b/pkgs/development/python-modules/gtts/default.nix
index 55486ed7dd6..f3e8162b714 100644
--- a/pkgs/development/python-modules/gtts/default.nix
+++ b/pkgs/development/python-modules/gtts/default.nix
@@ -47,6 +47,7 @@ buildPythonPackage rec {
   meta = with lib; {
     description = "A Python library and CLI tool to interface with Google Translate text-to-speech API";
     homepage = "https://gtts.readthedocs.io";
+    changelog = "https://gtts.readthedocs.io/en/latest/changelog.html";
     license = licenses.mit;
     maintainers = with maintainers; [ unode ];
   };
--
2.29.2


Background Bluetooth Support

I completely agree about checking in .js files as well as the typescript. Here is my initial WIP port.
TS Port
I have done the bluetooth.android.ts file. As well as the bluetooth.common.ts file. For Android, it seems we are using some APIs that are higher than 17, so the standard android definition files that are shipped with the tns-core-modules aren't comprehensive for what we need. So I have checked in a "customised" version which makes what we need available. I have opened up an issue on the Nativescript repo to fix this, at which point, we
can delete the definitions from this repo.

[x] Port Android to Typescript
[ ] Port IOS to Typescript
[ ] Background Support for Android
[ ] Background support for iOS
[ ] Test Support for Android
[ ] Test Support for iOS

Now I'll try and do the same for iOS and then start on the actual project !

OK, here are some more commits, I have added the sample to the same repo - let me know if you don't like this. However, it makes testing and CI much easier. I have made a few minor edits and now the isBluetoothEnabled and stop scan button work, the scanning should be fine, but I am on an emulator so can't test it properly.

Hey George, can you check in the generate .js files as well please? I don't want to force ppl usign TS although it's super nice.

@EddyVerbruggen There should be all the comiled js in the .dist/ directory? Is that what you meant?

Ah yes, but at runtime those aren't picked up, right? Perhaps if the package.json is changed to point to those but I'm not sure that'll work.

The line in sample/package.json:
	"dependencies": {
		"nativescript-bluetooth": "file:../dist",

Should make sure only that directory is copied over to the sample app?

I get it. But if the plugin is loaded from the dist folder then don't we need the platforms folder in there as well? I'm asking because I'm running the sample app on my Android 4.4 device and it's missing the BLUETOOTH permission because the .aar from the dist/platforms/android folder is not loaded.
We could avoid this by just adding the generated .js files to the root and pointing the sample to file:.. as well. WDYT?

Ahh, good point. I am not really sure about the mechanics of that yet, but here is a commit which moves the platforms folder to dist. Does the AndroidManifest.xml need to go in as well?
As for dist vs. whole file, I prefer the current setup, as it avoids having the build definitions, tests and other meta stuff shipped with every production app using this plugin.
However, if you would prefer just to use the root - give me a shout and I'l change it back - it's your project :)

@EddyVerbruggen yes I know, I will push another commit to fix asap

Apologies for asking this here, but I wasn't sure if there was a better place.  Is this goal of this pull request to allow bluetooth communication while the mobile app is not the currently active app?  If so, does it look like this functionality will be available soon?  I'm about to start working on a mobile app and this functionality is key for me.  I think NativeScript might be a good option for my project, but I need to be sure I can talk to the bluetooth peripheral when my app is "in the background".

@skrichten the Slack Channel is good for general questions, but here is fine. Yes, that is the point. I am working on this, with @EddyVerbruggen who is providing feedback. As for the timeframes, I am hesitant to promise much, as it depends on how busy I am with my day job etc. however, I would hope within a month to have this release. However, it would be awesome if you wanted to contribute to speed this up?

Hey @georgeedwards, when running npm run demo.android and pressing the 'is bluetooth enabled?' button my app crashes with TypeError: cannot read property isBluetoothDefined of undefined.
I'm guessing line 13 of main-view-model.ts should instantiate an object but that also probably means the .d.ts file needs to change.
Don't you experience the same crash? I'm on Android 4.4.

@EddyVerbruggen @georgeedwards I am able to reproduce the error TypeError: cannot read property isBluetoothDefined of undefined on a Nexus 5 running Cyanogenmod 13 (switching to stock 6.0.1 for further testing). I am going to investigate and see if I can get a fix. What is the best way to get a bugfix to you? A PR on @georgeedwards's repo?

Any progress on this?
I highly advise that adding TypeScript support NOT be part of this PR; make that a separate PR.


interop: move to its own module and disable by default

Due to the amount of side-effects that touching the binfmt registrations introduces (no interop in other distros, broken WSLg, etc)  it is now disabled by default. When the configuration contains another binfmt registration (and would therefore break interop) a warning is issued during evaluation
@K900

Once https://github.com/microsoft/WSL/issues/8203 is resolved, we can enable it by default again. This is just meant as a temporary fix


Add Conversational AI

Conversational AI
This list provides resources on Conversational AI to help anyone start building chatbots and digital assistants. It focuses on the technical part of the topic (NLU, clients, platforms) so I think it fits best in Computer science, but it will also cover topics such as Conversational Ux.
Reviewed the following two PRs:

#2067
#2153

By submitting this pull request I confirm I've read and complied with the below requirements 🖖
Please read it multiple times. I spent a lot of time on these guidelines and most people miss a lot.
Requirements for your pull request

Don't waste my time. Do a good job, adhere to all the guidelines, and be responsive.
You have to review at least 2 other open pull requests.
Try to prioritize unreviewed PRs, but you can also add more comments to reviewed PRs. Go through the below list when reviewing. This requirement is meant to help make the Awesome project self-sustaining. Comment here which PRs you reviewed. You're expected to put a good effort into this and to be thorough. Look at previous PR reviews for inspiration. Just commenting “looks good” or simply marking the pull request as approved does not count! You have to actually point out mistakes or improvement suggestions.
You have read and understood the instructions for creating a list.
This pull request has a title in the format Add Name of List.

✅ Add Swift
✅ Add Software Architecture
❌ Update readme.md
❌ Add Awesome Swift
❌ Add swift
❌ add Swift
❌ Adding Swift
❌ Added Swift

Your entry here should include a short description about the project/theme of the list. It should not describe the list itself. The first character should be uppercase and the description should end in a dot. It should be an objective description and not a tagline or marketing blurb.

✅ - [iOS](…) - Mobile operating system for Apple phones and tablets.
✅ - [Framer](…) - Prototyping interactive UI designs.
❌ - [iOS](…) - Resources and tools for iOS development.
❌ - [Framer](…)
❌ - [Framer](…) - prototyping interactive UI designs

Your entry should be added at the bottom of the appropriate category.
The title of your entry should be title-cased and the URL to your list should end in #readme.

Example: - [Software Architecture](https://github.com/simskij/awesome-software-architecture#readme) - The discipline of designing and building software.

The suggested Awesome list complies with the below requirements.

Requirements for your Awesome list

Has been around for at least 30 days.That means 30 days from either the first real commit or when it was open-sourced. Whatever is most recent.
Don't open a Draft / WIP pull request while you work on the guidelines. A pull request should be 100% ready and should adhere to all the guidelines when you open it.
Run awesome-lint on your list and fix the reported issues. If there are false-positives or things that cannot/shouldn't be fixed, please report it.
The default branch should be named main, not master.
Includes a succinct description of the project/theme at the top of the readme. (Example)

✅ Mobile operating system for Apple phones and tablets.
✅ Prototyping interactive UI designs.
❌ Resources and tools for iOS development.
❌ Awesome Framer packages and tools.

It's the result of hard work and the best I could possibly produce.
If you have not put in considerable effort into your list, your pull request will be immediately closed.
The repo name of your list should be in lowercase slug format: awesome-name-of-list.

✅ awesome-swift
✅ awesome-web-typography
❌ awesome-Swift
❌ AwesomeWebTypography

The heading title of your list should be in title case format: # Awesome Name of List.

✅ # Awesome Swift
✅ # Awesome Web Typography
❌ # awesome-swift
❌ # AwesomeSwift

Non-generated Markdown file in a GitHub repo.
The repo should have awesome-list & awesome as GitHub topics. I encourage you to add more relevant topics.
Not a duplicate. Please search for existing submissions.
Only has awesome items. Awesome lists are curations of the best, not everything.
Does not contain items that are unmaintained, has archived repo, deprecated, or missing docs. If you really need to include such items, they should be in a separate Markdown file.
Includes a project logo/illustration whenever possible.

Either centered, fullwidth, or placed at the top-right of the readme. (Example)
The image should link to the project website or any relevant website.
The image should be high-DPI. Set it to maximum half the width of the original image.

Entries have a description, unless the title is descriptive enough by itself. It rarely is though.
Includes the Awesome badge.

Should be placed on the right side of the readme heading.

Can be placed centered if the list has a centered graphics header.

Should link back to this list.

Has a Table of Contents section.

Should be named Contents, not Table of Contents.
Should be the first section in the list.
Should only have one level of nested lists, preferably none.
Must not feature Contributing or Footnotes sections.

Has an appropriate license.

We strongly recommend the CC0 license, but any Creative Commons license will work.

Tip: You can quickly add it to your repo by going to this URL: https://github.com/<user>/<repo>/community/license/new?branch=main&template=cc0-1.0 (replace <user> and <repo> accordingly).

A code license like MIT, BSD, Apache, GPL, etc, is not acceptable. Neither are WTFPL and Unlicense.
Place a file named license or LICENSE in the repo root with the license text.
Do not add the license name, text, or a Licence section to the readme. GitHub already shows the license name and link to the full text at the top of the repo.
To verify that you've read all the guidelines, please comment on your pull request with just the word unicorn.

Has contribution guidelines.

The file should be named contributing.md. Casing is up to you.
It can optionally be linked from the readme in a dedicated section titled Contributing, positioned at the top or bottom of the main content.
The section should not appear in the Table of Contents.

All non-important but necessary content (like extra copyright notices, hyperlinks to sources, pointers to expansive content, etc) should be grouped in a Footnotes section at the bottom of the readme. The section should not be present in the Table of Contents.
Has consistent formatting and proper spelling/grammar.

The link and description are separated by a dash. Example: - [AVA](…) - JavaScript test runner.
The description starts with an uppercase character and ends with a period.
Consistent and correct naming. For example, Node.js, not NodeJS or node.js.

Doesn't use hard-wrapping.
Doesn't include a Travis badge.You can still use Travis for list linting, but the badge has no value in the readme.
Doesn't include an Inspired by awesome-foo or Inspired by the Awesome project kinda link at the top of the readme. The Awesome badge is enough.

Go to the top and read it again.

Thanks for making an Awesome list! 🙌
It looks like you didn't read the guidelines closely enough. I noticed multiple things that are not followed. Try going through the list point for point to ensure you follow it. I spent a lot of time creating the guidelines so I wouldn't have to comment on common mistakes, and rather spend my time improving Awesome.

@jyguyomarch thanks for the addition, here are the things you can improve:

typo in topcis
typo in Messegner
you writed chat bot instead of chatbot
Ux instead of UX
open source instead of open-source

Thx a lot @riderx ! That was a lot of typos. Fixed that,

@sindresorhus thx for the quick review! I reviewed the guidelines again but don't see what I am missing (apart from the typos that @riderx found). If you can point me in the right direction, it would be awesome. But I know your time is limited, worst case I'll wait for some more reviews from the community.
Maybe you were referring to the PR reviews that should have been added as a comment and not in the PR description like I did. Here they are:

#2067
#2153

@jyguyomarch I went over your list and one of the things I spotted was the missing description of the content on top of the list in teh repo. You also described the list itself in the PR above and not only the theme/topic of the list.

Thx @Karneades I'll review that asap


Docstrings are not formatted correctly when served by language server

This appears to work now. Seems like it was fixed by either Lean 4 or vscode-lean4.


CRS spec definition for version 0.1

This PR aims to cover the definition of CRS for version 0.1.
I've covered all the topics discussed at https://github.com/opengeospatial/geoparquet/issues/3

I've just made a change to the recommended CRS. I've decided to move from OGC:84 to EPSG:4326 (as we decided at https://github.com/opengeospatial/geoparquet/issues/3#issuecomment-1055799929). The reasons are:

We decided OGC:84 over EPSG:4326 because of the axis order, but we later decided to force a specific axis order of long/lat.
The first dataset I was about to use (at our example.py was in EPSG:4326 and I needed to transform it to OGC:84.
4326 is far more used than OGC:84.
The experience working with OGC:84 is harder than 4326. It's not very hard, but I played a little bit with geopandas and pyproj to get the WKT2 string, apply transformations, etc... and it was easier for me with 4326 because I'm more familiar with this, I think lots of users will have the same feeling.

Sorry to re-open this, but I think it's important. If you don't agree with this change we can rollback to OGC:84: https://github.com/opengeospatial/geoparquet/pull/25/commits/2ddcb907ee67dd80074a17835f872e1b2fe0aa37

I was actually looking around at OGC 84 as well and reached a similar conclusion, so I'm +1 on keeping EPSG:4326. And agree that our decision to force axis order also helps. I think we should perhaps 'cite' geopackage, refer to the section of the spec, and say that we are following them.
Perhaps it would be good to write up a small page in the spec repo that helps explain the state of things. It looks like geojson uses OGC:84, but it also looks like doing that is easier if you're not including a WKT2. I do worry a bit that we're just propagating the issue with using 4326 and doing lon/lat. Perhaps we should ask the geopackage group if they still feel good about their decision.
But yes, for 0.1 let's just go with 4326 and clarify that we're overriding it, and we can also revisit it.

Now, if we want to discuss this, let's open a new issue for that (it's a hairy discussion with no easy answer, but just wanted to mention it here, being reminded about it seeing the ensemble datum in the WKT2 output). I think for this PR / for 0.1, the current recommendation is good enough (in the end, it's only a recommendation, and changing a recommendation is not a breaking change to the spec).

Yeah, it does feel weird to explain to users to use 4326 but ignore what it actually says. Fully agree on opening an issue and just shipping what we have in 0.1. I think it'd be good to get a wider audience to sound in on what we should do - I'm definitely going back and forth on it.

(small note: I pushed a commit to revert the changes to the example, to avoid conflicts with https://github.com/opengeospatial/geoparquet/pull/24)

Thanks @alasarr !


Highlighting with fields of enumerable

I'm unable to get any highlighting to work at the moment. Because of the way we do dynamic field permissioning for our users, the list of fields is not known until runtime.

            var sd = new SearchDescriptor<dynamic>();
            sd = sd
                .Query(q => q
                    .QueryString(qs => qs
                        .Query(escapedSearchString)
                        .OnFields(fieldsToSearch.Select(fc => fc.Id.ToString("0")).ToArray())
                    )
                )
                .Highlight(h => h
                    .PreTags("")
                    .PostTags("")
                    .NumberOfFragments(2)
                    .OnFields(ofz =>
                    {
                        foreach (var fieldName in fieldsToSearch.Select(fc => fc.Id))
                        {
                            ofz = ofz.OnField(fieldName.ToString(CultureInfo.InvariantCulture));
                        }

                    })
                );

We're getting 0 highlights on every result, which is incorrect.

FWIW, The expected output of the highlight would be as follows
	"highlight": {
		"pre_tags": [""],
		"post_tags": [""],
		"number_of_fragments": 2,
		"order": "score",
		"fields": {
			"22": {},
			"22.SV": {},
			"29168": {},
			"29168.SV": {},
			"29167": {},
			"29167.SV": {},
			"23": {},
			"23.SV": {},
    .....

It's possible that the solution to this is similar to the solution to QueryString.OnFields, in that you could accept an array of field names.

FYI... I wouldn't close this just because there's a painful workaround... sending an array to fields would still be good... but this bit of craziness worked:
                    .OnFields(fieldsToSearch.Select(
                        fx => new Action<HighlightFieldDescriptor<dynamic>>(ofx => ofx
                            .OnField(fx.Id.ToString(CultureInfo.InvariantCulture)))).ToArray()
                    )

Yea, the descriptors are definitely inconsistent in regards to OnFields() (e.g. params vs IEnumerable, etc...), and it's something we have planned to fix and normalize across the board in 2.0.
That said, we could add an IEnumerable overload here for the time being, but each field can have it's own highlighting options, hence HighlightFieldDescriptor<T>.  So adding an overload just for just field names, while convenient, is limiting.  Maybe this would be better suited as an extension method?  Open to suggestions here, or a PR :).
P.S. I actually don't mind the "workaround" you've come up with to be honest.

It certainly works, but may be quite a bit more painful for many people to do. Though, I suspect we're probably the first to try this specific type of thing, I'd imagine that the nature of elastic (no structure required) will lead others here eventually.
I like to think that something similar to my first attempt might be easier to most. I'm not sure that something exactly like that has to work, but along the lines of being able to avoid params if desired.
You're right on the IEnumerable. Ultimately my need was not to define anything other than the fact that I wanted the same list of fields from my search to be included in highlighting.  It's really  just unfortunate that OnAll doesn't work here (elastic limitation, not driver).

Closing this since there is no immediate need, although i fully agree its not the best experience. In our 2.0 branch all the OnField and OnFields are getting a look over.


/votekut

/votekut does not work anymore @AleksCore FIX! asap jack is not happy ):

https://github.com/JarnoVgr/Mr.Green-MTA-Resources/pull/216


Do not fail if tput is not available

Fixes issue #4376

Looks good to me, though I can only test it on the normal case where tput is present.
Want to hit tools/upgrade.sh with the same thing in this PR?

PR updated to handle upgrade.sh

:+1: /cc @robbyrussell

BTW, if you phrase it as "Fixes #4376" instead of "Fixes issue #4376", GitHub will recognize it and add an auto-close link between the PR and issue. The extra "issue" in there means it doesn't recognize it.

Also BTW, does this actually work on the Synology? Because of the set -e, I think the script might still abort after that tput=$(which tput) line, since the which returns nonzero.

I'll fix the commit entry to help github and will update you with status on Synology (didn't double check yet)

FYI, PR #4419 goes even further, ignoring all errors in the color setup. It's pretty blunt, but will get things working for now, even if it's still breaking on the Synology. Turns out, which tput isn't sufficient, because this code actually requires a tput that supports terminfo style capabilities.


Possible wrong rule in FACTUR-X_EXTENDED.sch

I noticed that there is a rule for
/rsm:CrossIndustryInvoice/rsm:SupplyChainTradeTransaction/ram:ApplicableHeaderTradeAgreement/ram:SellerTradeParty/ram:SpecifiedLegalOrganization/ram:ID
That fires if the schemeIDis not given
<pattern>
    <rule context="/rsm:CrossIndustryInvoice/rsm:SupplyChainTradeTransaction/ram:ApplicableHeaderTradeAgreement/ram:SellerTradeParty/ram:SpecifiedLegalOrganization/ram:ID">
      <assert test="@schemeID">
	Attribute '@schemeID' is required in this context.</assert>
    </rule>
</pattern>

Starting at line 5264 in FACTUR-X_EXTENDED.sch .
According to ZUFeRD-2.1.1 Spezifikation TA page 8 the schemeID is optional.
Do I misinterpret the specs or is this rule misleading?
TIA

it's misleading, it really is optional, try a java -jar Mustang-CLI-2.0.0-alpha2.jar --action=validate --source=yourfile

Thanks for your quick reply.
I did test with my example file and get the following result.
#> java -jar ../../Mustang-CLI-2.0.0-alpha1.jar --action=validate --source=faulty-example.xml
10:13:53.882 [main] INFO  c.h.s.x.SchematronResourceXSLTCache - Compiling XSLT instance [cpPath=/xslt/ZF_211/FACTUR-X_EXTENDED.xslt; urlResolved=true; URL=jar:file:/C:/Use
rs/klein.ICO-TEAM/Downloads/Mustang-CLI-2.0.0-alpha1.jar!/xslt/ZF_211/FACTUR-X_EXTENDED.xslt]
10:13:59.383 [main] INFO  o.m.validator.XMLValidator - FailedAssert
10:13:59.395 [main] ERROR o.m.validator.ZUGFeRDValidator - Error 4:
        Attribute '@schemeID' is required in this context.
10:13:59.410 [main] INFO  o.m.validator.ZUGFeRDValidator - Parsed PDF:valid XML:invalid Signature:null Checksum:166504199CA9DE15B36606F0E0C6DAD4BBA1043F Profile:urn:cen.eu
:en16931:2017#conformant#urn:factur-x.eu:1p0:extended Version:2 Took:5793ms Errors:[4]
<?xml version="1.0" encoding="UTF-8"?>

<validation filename="test-invoice-mbcs-valid1.xml8111741b-584b-4228-86cf-b5bc7b0fee38.xml" datetime="2020-09-23 10:13:53">
  <xml>
    <info>
      <version>2</version>
      <profile>urn:cen.eu:en16931:2017#conformant#urn:factur-x.eu:1p0:extended</profile>
      <validator version="1.9.0-alpha1"/>
      <rules>
        <fired>66</fired>
        <failed>1</failed>
      </rules>
      <duration unit="ms">5722</duration>
    </info>
    <messages>
      <error type="4" location="/*[local-name()='CrossIndustryInvoice']/*[local-name()='SupplyChainTradeTransaction']/*[local-name()='ApplicableHeaderTradeAgreement']/*[lo
cal-name()='SellerTradeParty']/*[local-name()='SpecifiedLegalOrganization']/*[local-name()='ID']" criterion="@schemeID">Attribute '@schemeID' is required in this context.<
/error>
    </messages>
    <summary status="invalid"/>
  </xml>
  <messages></messages>
  <summary status="invalid"/>
</validation>

Example file used for testing: faulty-example.zip
It seems to me that the example is considered invalid and that it's not just a warning. Should have provided that information in the first place...


feat(Algebra/Homology): develop the API for the action of bifunctors on homological complexes

maintainer merge


Django 3.0 support

@v1k45  are these getting merged soon? I'm wanting to upgrade to Django 3.0 but facing a roadblock with notify.

@payal97 I don't believe it is still being maintained, perhaps you should just fork it and install it straight from your branch


Fix null being a string when persisting a date attribute and the value is empty.

Description
Fixes an issue when saving a date attribute with an empty value.
Currently this leads to "0000-00-00" being saved to the Database which then ExtJS cannot render correctly: https://i.imgur.com/Tnhqrqg.png

Questions
Answers

Why?
Fixes a bug where ExtJS cannot render a field value correctly.

BC breaks?
No

Tests exists & pass?
Cannot find tests.

Related tickets?

How to test?
Add a new date attribute, enter an empty value and save. Check the value in the DB, it should be NULL not 0000-00-00.

Requirements met?
Yes

This has recently been an issue for us and we've already fixed it. It has been merged today here 9b66bd52fe29f382831fe5eedbd65eb0d3ea271e.
Thanks @bestit-el-bardan.


It looks like the issue is that we used <TCEforms> tags within FlexForms.
I recommend tot add the node name (tag name) as well as file name to the Exception.
That would allow users to manually fix and easier report issues.
This relates to our internal "Aufgabe #11275 TYPO3 Upgrade 13 LTS" issue.

An example snippet for reproduction:
<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
<T3DataStructure>
    <meta>
        <langDisable>1</langDisable>
    </meta>
    <sheets>
        <smallDevices>
            <ROOT type="array">
                <TCEforms>
                    <sheetTitle>LLL:EXT:extension/Resources/Private/Language/locallang_container.xlf:grid.sheet.smallDevices</sheetTitle>
                </TCEforms>
                <type>array</type>
                <el type="array">
                    <smCol1 type="array">
                        <TCEforms type="array">
                            LLL:EXT:extension/Resources/Private/Language/locallang_container.xlf:grid.label.col1
                            <config type="array">
                                <type>select</type>
                                <renderType>selectSingle</renderType>
                                <itemsProcFunc>Vendor\Extension\FlexForms\Container->getFourColumnOptions</itemsProcFunc>
                            </config>
                        </TCEforms>
                    </smCol1>
                </el>
            </ROOT>
        </smallDevices>
    </sheets>
</T3DataStructure>

Using TCEforms is correct and will be removed by one of the rules but the attribute type="array" is wrong because an array would require some children with indexes which is not the case. The correct XML would look like this before refactoring:
<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
<T3DataStructure>
    <meta>
        <langDisable>1</langDisable>
    </meta>
    <sheets>
        <smallDevices>
            <ROOT>
                <TCEforms>
                    <sheetTitle>LLL:EXT:extension/Resources/Private/Language/locallang_container.xlf:grid.sheet.smallDevices</sheetTitle>
                </TCEforms>
                <type>array</type>
                <el>
                    <smCol1>
                        <TCEforms>
                            LLL:EXT:extension/Resources/Private/Language/locallang_container.xlf:grid.label.col1
                            <config>
                                <type>select</type>
                                <renderType>selectSingle</renderType>
                                <itemsProcFunc>Vendor\Extension\FlexForms\Container->getFourColumnOptions</itemsProcFunc>
                            </config>
                        </TCEforms>
                    </smCol1>
                </el>
            </ROOT>
        </smallDevices>
    </sheets>
</T3DataStructure>


MSWC FSCIL

Adding MSWC FSCIL functionality + example!

https://github.com/NeuroBench/neurobench/blob/f0d9a729ca7ddb6406fd84d9336b815a3645a7a8/neurobench/datasets/MSWC.py#L15-L216
Should we move this list to another python file?

Having some difficulties running the example, also question about the torch_mate dependency in the example

Hopefuly this should be now fixed with the latest commit : https://github.com/NeuroBench/neurobench/blob/2ef814c79e8d5d269b1acd4ba3d1f2cbdacf9631/neurobench/examples/mswc_fscil.py#L20
At least, it works on my side with the poetry install on a Windows system.
Basically, I just added the if main clause that maintains correctness for multiprocesses forking on Windows.
Let me know if it also solves the problem for you.

@Maxtimer97 It is running now! Small comments:

I believe there should be a benchmark run at line 39 using the base_test_set in order to evaluate base accuracy.
Line 90 torch.mean() is applied to a list, not tensor, so throwing an error for me.
Total runtime of the script on my M1 macbook is ~14 min. I think data loading optimization can be considered but this doesn't seem prohibitively slow!

@Maxtimer97 It is running now! Small comments:

I believe there should be a benchmark run at line 39 using the base_test_set in order to evaluate base accuracy.
Line 90 torch.mean() is applied to a list, not tensor, so throwing an error for me.
Total runtime of the script on my M1 macbook is ~14 min. I think data loading optimization can be considered but this doesn't seem prohibitively slow!

Thanks! First and Second comments should be solved with this last commit: https://github.com/NeuroBench/neurobench/pull/1/commits/6f2cd8c4f38198d630b8405618ca36ac1cd80e9e
I will be exploring the data loading further now.

Closing branch, other mswc task branches more updated.


complex dependency constraints

currently complex constraints (constraints other than true,false or null) only allow {optional:true|false}
these dependency constraints should allow more.  (e.g. content_dir = relative path to package is linked to content dir of dependency)
version constraints...

Also allow modification of original package descriptor.  (example:  remove or add compiler flags...

Added handler for symlink property of dependency constraints (instead of content_dir)

constraints are now extensible and complex version constraint is now added


Unable to allocate pixels for the bitmap. On linux server

hi i have the same problem, i dowload the from libSkiaSharp.so https://jenkins.mono-project.com/view/Components/job/Components-SkiaSharp-Linux/79/Azure/processDownloadRequest/ArtifactsFor-79/3c31bd5c5701a6eb3651110b4708bb1d0492c7fc/output/native/linux/x64/libSkiaSharp.so.
My project is a net core 2.1 project, I include the library into my project with property "copy always" and alse in my startup.cs, i put context.LoadUnmanagedLibrary(Path.Combine(Directory.GetCurrentDirectory(), "libSkiaSharp.so")) and i got error:
Unable to allocate pixels for the bitmap.
  at SkiaSharp.SKBitmap..ctor(SKImageInfo info, Int32 rowBytes)
  at SkiaSharp.SKBitmap..ctor(SKImageInfo info)
  at SkiaSharp.SKBitmap.Decode(SKCodec codec, SKImageInfo bitmapInfo)
  at SkiaSharp.SKBitmap.Decode(SKCodec codec)
  at SkiaSharp.SKBitmap.Decode(Stream stream)

My deploy machine is an amazon with OS CentOS Linux 7 (Core). I installed libgdiplus and also i install mono-complete.
I appreciate your help if i need install someting else. I dont know what else i need to install.

Most often, this case is just because there is a maximum supported size for bitmaps: https://github.com/mono/SkiaSharp/issues/1249


Collision error when after menu screen

Hey I was intrested in checking out your game but when I started it up I got the following error
return o1.mask.overlap(o2.mask, (offset_x, offset_y)) != None
TypeError: integer argument expected, got float
The temporary fix I found was to round the offset x and y to the nearest integer but it was just to let you know and I bet there is a better way to go about it:
return o1.mask.overlap(o2.mask, (round(offset_x), round(offset_y))) != None

Hello, I couldn't replicate the bug after trying quite a few things. I'm going to implement your suggested fix as it doesn't seem to break anything and should solve your issue. Cheers


Nav buttons are not working properly in gallery mode

How to reproduce :
set 'mode' option to 'gallery'
click once on "prev", "next" or a nav button -> works as expected
click again on any of these buttons -> works not as expected : nothing happen
you can see it on the test page on the "animation" test

Thanks, I also noticed this issue recently, will look at it today.

Hi, this is fixed in the latest release.
Could you check if it works for you?
Thanks

it works !
great !


Allow option to delete profiles

Added a new config option that will allow user to clear profiler profiles from storage when Clear DB is called. This is helpful functionality especially in development environment where frequent DB cleaning is done.

All checks have passed and the updated code implements helpful functionality.


Make operator generation more flexible

Current operator generation in queries is inflexible - the current logic selects an operator based only on the expression's NodeType. Cases where a different operator needs to be generated based on others factors (e.g. the expression's Type) are difficult to integrate.
A good example is string concatenation in PostgreSQL, which requires operator | rather than +. The current code hacks a special check for this directly into DefaultQuerySqlGenerator, defining a special ConcatOperator which can be overridden by providers. Another example is bitwise operators for boolean - PostgreSQL does not support boolean & boolean, so ExpressionType.And should be rendered as && based on the expression's Type.

Sorry for not testing before submitting, will fix the Sqlite issues.

Amended and rebased

FYI Linux build seems to be failing for unrelated reasons

@roji, Thanks for the PR. Unfortunately DefaultQuerySqlGenerator is public surface and so we need to do this in a non-breaking way. We do have the option of obsoleting methods, but not removing them.

Oh, I wasn't aware you were on a strict semver policy on provider-facing classes such as DefaultQuerySqlGenerator. Does that mean you expect users to subclass DefaultQuerySqlGenerator, or is it simply to provide a stable API to providers?
Regardless, I understand that this is your policy - I'll try to resubmit this change in a way that doesn't break anything.

Does that mean you expect users to subclass DefaultQuerySqlGenerator, or is it simply to provide a stable API to providers

Both. Anything in a .Internal namespace is fair game, btw.
Thanks!

@anpete, have pushed another version which leaves all protected APIs in place, and using them from the new GenerateOperator(Expression) API. It's a tiny bit awkward since it has to call the existing {Try,}GenerateBinaryOperator, but only if the given expression is binary etc.


ci: change permission to write for github prelease

IMPORTANT: Please do not create a Pull Request without creating an issue first.
Problem:

Solution:

Related Issue:
https://github.com/harvester/harvester/issues/5327
Test plan:

@mergifyio backport release-1.3

@mergifyio backport v1.3 v1.2


Error installing pusher with pip in debian 8 with python 2.7.9

Hello, I have issues installing pusher with pip, when I did pip install pusher I got this:
Requirement already satisfied (use --upgrade to upgrade): pusher in /usr/local/lib/python2.7/dist-packages
Requirement already satisfied (use --upgrade to upgrade): six in /usr/lib/python2.7/dist-packages (from pusher)
Requirement already satisfied (use --upgrade to upgrade): requests>=2.3.0 in /usr/lib/python2.7/dist-packages (from pusher)
Requirement already satisfied (use --upgrade to upgrade): urllib3==1.17 in /usr/local/lib/python2.7/dist-packages (from pusher)
Requirement already satisfied (use --upgrade to upgrade): pyopenssl in /usr/lib/python2.7/dist-packages (from pusher)
Requirement already satisfied (use --upgrade to upgrade): ndg-httpsclient in /usr/lib/pymodules/python2.7 (from pusher)
Requirement already satisfied (use --upgrade to upgrade): pyasn1 in /usr/lib/python2.7/dist-packages (from pusher)
Requirement already satisfied (use --upgrade to upgrade): cryptography>=0.2.1 in /usr/lib/python2.7/dist-packages (from pyopenssl->pusher)
Requirement already satisfied (use --upgrade to upgrade): cffi>=0.8 in /usr/lib/python2.7/dist-packages (from cryptography>=0.2.1->pyopenssl->pusher)
Requirement already satisfied (use --upgrade to upgrade): pycparser in /usr/lib/python2.7/dist-packages (from cffi>=0.8->cryptography>=0.2.1->pyopenssl->pusher)
Cleaning up...

And when I try to use it I got this error:
Traceback (most recent call last):
  File "scripts/notif_alarmas.py", line 73, in <module>
    notif_alarmas().getAlarms()
  File "scripts/notif_alarmas.py", line 37, in getAlarms
    pusher = Pusher(app_id=self.push_id, key=self.push_key, secret=self.push_secret)
  File "/usr/local/lib/python2.7/dist-packages/pusher/pusher.py", line 52, in __init__
    json_encoder, json_decoder, backend, **backend_options)
  File "/usr/local/lib/python2.7/dist-packages/pusher/pusher_client.py", line 34, in __init__
    json_encoder, json_decoder, backend, **backend_options)
  File "/usr/local/lib/python2.7/dist-packages/pusher/client.py", line 20, in __init__
    from .requests import RequestsBackend
  File "/usr/local/lib/python2.7/dist-packages/pusher/requests.py", line 17, in <module>
    import urllib3.contrib.pyopenssl
  File "/usr/local/lib/python2.7/dist-packages/urllib3/contrib/pyopenssl.py", line 48, in <module>
    from cryptography import x509
ImportError: cannot import name x509

After seeing this I try easy_install cryptography and it installed more packages but then when I run the program again I got this:
Using /usr/lib/python2.7/dist-packages
Processing dependencies for cryptography
Finished processing dependencies for cryptography
root@servertrack:/var/www/axiomatracker# python scripts/notif_alarmas.py
Traceback (most recent call last):
  File "scripts/notif_alarmas.py", line 73, in <module>
    notif_alarmas().getAlarms()
  File "scripts/notif_alarmas.py", line 43, in getAlarms
    pusher.trigger(u["ws_channel"], 'alarms_event', {'message': message})
  File "/usr/local/lib/python2.7/dist-packages/pusher/pusher.py", line 117, in trigger
    channels, event_name, data, socket_id)
  File "/usr/local/lib/python2.7/dist-packages/pusher/http.py", line 31, in __call__
    return self.client.http.send_request(self.make_request(*args, **kwargs))
  File "/usr/local/lib/python2.7/dist-packages/pusher/requests.py", line 44, in send_request
    **self.options)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 457, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 569, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 362, in send
    timeout=timeout
  File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 516, in urlopen
    body=body, headers=headers)
  File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 304, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 724, in _validate_conn
    conn.connect()
  File "/usr/lib/python2.7/dist-packages/urllib3/connection.py", line 237, in connect
    ssl_version=resolved_ssl_version)
  File "/usr/lib/python2.7/dist-packages/urllib3/contrib/pyopenssl.py", line 263, in ssl_wrap_socket
    ctx.load_verify_locations(ca_certs, None)
  File "/usr/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 303, in load_verify_locations
    raise TypeError("cafile must be None or a byte string")
TypeError: cafile must be None or a byte string

Please help.

This is the last error I found on pip.log after uninstalling all packages and trying to reinstall pusher only.
Cleaning up...
Exception:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 290, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1091, in prepare_files
    req_to_install.check_if_exists()
  File "/usr/lib/python2.7/dist-packages/pip/req.py", line 815, in check_if_exists
    existing_dist = pkg_resources.get_distribution(self.req.project_name)
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 349, in get_distribution
    dist = get_provider(dist)
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 229, in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 745, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 644, in resolve
    raise VersionConflict(dist, req)
VersionConflict: (urllib3 1.9.1 (/usr/lib/python2.7/dist-packages), Requirement.parse('urllib3==1.17'))

This looks like it could be related to #67? Do you have any luck trying different versions of pyopenssl?

Hello...
See here: sudo apt-get install build-essential libssl-dev libffi-dev python-dev
Use those dependencies. That should do the trick. Debian does not come with them already installed.
Seth

I've added a note to install OpenSSL on linux in https://github.com/pusher/pusher-http-python/commit/069d112056f1581f0b7c804aead746119d807380


V2D enhancements

This is an attempt to collect all the Genetics Variant To Disease Pipeline steps to have an overall description of how it works and to identify points to work on to improve the overall functioning of the pipeline.
2 overall notes:

The pipeline is overall complex in terms that there are many steps involved, when thinking about QC it is important to identify which are the critical points.
The number of outputs is not very big, it is feasible to validate them by defining a JSON schema for each. Is this the right call? @DSuveges

Workflow DAG
Graph of dependencies of the different scripts that are called on the pipeline.

Workflow description
The whole pipeline consists on the creation of 4 files: study, fine mapping, LD and top loci tables.
Top loci table

The result of merging the associations from GWASCat and the summary statistics (merge_gwascat_and_sumstat_toploci).

GWASCat:

Download GWAS Catalogue associations (All associations v1.0.2 - with added ontology annotations, GWAS Catalog study accession numbers and genotyping technology)
Filter down the variant index to only include variants collected in GWAS Cat.

(Problem: this is parsed and joining by reading inputs line by line)
Bug: cannot run the code without incurring in encoding issue: ValueError: invalid literal for int() with base 10: '\ufeff79254949'

Populate variant annotation on the GWAS Catalogue data.
Join GWASCatalogue studies with table of new studies and format.

Summary statistics:

Associations are coming from the output of the finemapping pipeline

Problem: this file is encoded in the pipeline config to be using 'gs://genetics-portal-dev-staging/finemapping/merged_210515/top_loci.json.gz', is this the most up to date data?

Study table - TBC

From independently generating 3 tables: GWASCat, UKBB, Finngen.

GWASCat:

Download GWAS Catalogue studies (All studies v1.0.2 - with added ontology annotations, GWAS Catalog study accession numbers and genotyping technology).
Split studies with multiple traits and merge this table with:

The table containing GWAS assocs enriched with variant information (almost top loci)
Ancestry annotation

Fine mapping table

Download credible set from the fine mapping pipeline.

Problem: this file is encoded in the pipeline config to be using 'gs://genetics-portal-dev-staging/finemapping/merged_210515/credset/_SUCCESS', is this the most up to date data?
Enhancement: download_credible_set_directory has some weird logic to download the credible set, apparently the problem is that it is a directory. This should be handled with no problem.
Format fine mapping table.

LD table

Create study/variant/population LUT: The result of inner joining study, top loci and the population map (to map from GWAS ancestry to 1000G superpopulation)
Create text file with all variants contained in the above LUT.

Issue: Instead of using bash to get the list of variant ids in chrom:pos:ref:alt form, implement this step in calculate_r_using_plink and use Spark.

Calculate LD for the variant IDs in the LD lut. This process is paralleled with up to 300 threads.

Problem: PLINK is called once per variant. Is it possible to input a file as explained here?

Generation of LD dataset:

The above LUT is joined with the results of the LD analysis.
Study/population weighted LD is calculated.
Conducts credible set analysis with PICS using the toploci table. This is joined with the above table (LUT + LD)
Output is a table with the studies enriched with the LD and PICS results.

FYI, I fixed the bug that caused the pipeline to crash on invalid input ('\ufeff79254949') from a small number of studies. (https://github.com/opentargets/genetics-v2d-data/commit/d58d0d4c6c7e6822fe566ee129890ba82045f2b7)
I informed GWAS catalog of the studies with the data problem.
Re: finemapping location, this has also been updated. (And in general would need to be updated when doing each release.)
Regarding the LD table, I think that the best enhancement would be to get rid of PICS altogether. Then you don't need any LD. There is very little benefit to the PICS method, since it assumes a single causal variant. You might as well just do standard WCCC-style approximate Bayes factor fine-mapping, which wouldn't require an LD panel and so would be much more robust, and would be much faster.


MINOR: Add Expr::canonical_name and improve docs on Expr::name

Which issue does this PR close?

Closes N/A
Rationale for this change

Some of the optimizer rules are using Expr::name inappropriately.
What changes are included in this PR?

Improve docs for Expr::name
Add new Expr::canonical_name

Are there any user-facing changes?

No

Thanks @andygrove


findAll convenience method in LdapObjectRepository

I added a findAll convenience method to the LdapObjectRepository class. Feel free to merge if you want.
I'm not familiar with phpspec, so you should review carefully before merging.

Looks good to me. Merged. Thanks!


Roxctl. generate netpol for Helm Charts

Hello, everyone!
Are there plans to include network policy generation for Helm Charts in the roadmap?
I don't know enough about the NP-Guard architecture. Where would it take to work with it to make it possible to generate policies for charts?
Generating policies in Pipelines for specific resources is very inconvenient and somewhat unusable:(

Hi @VenutNSA , if I understand your problem correctly, then I think there is already a solution to it. Have you tried populating the Helm chart with helm template to obtain the manifests and then passing it to the NP-Guard function roxctl generate netpol?

Stale issue


Sensu conf.d/*.json files are being overwritten by an empty config.json hash

Expected Behavior
An empty configuration hash should not stomp changes in /etc/sensu/conf.d/*.json files.
Current Behavior
Initial deploy
-----> Running bats test suite
 ✓ Install helper packages for testing
 ✓ Sensu is installed
 ✓ Sensu-server is running
 ✓ Sensu-server is enabled
 ✓ Sensu-api is running
 ✓ Sensu-api is enabled
 ✓ Sensu-client is running
 ✓ Sensu-client is enabled
 ✓ Validate the compiled Sensu configuration
 ✓ We can authenticate to the sensu api
 ✓ Sensu /info endpoint shows transport is connected
 ✓ Sensu /info endpoint shows redis is connected
 ✓ Ensure uchiwa is running
 ✓ Ensure uchiwa is enabled
 ✓ Uchiwa health is ok
 ✓ Uchiwa can connect to sensu-api

The /etc/sensu/conf.d/api.json will be deployed with the following content
{
  "api": {
    "host": "<IP_ADDRESS>",
    "port": 4567,
    "user": "sensu",
    "password": "somepassword",
    "bind": "<IP_ADDRESS>"
  }
}

Wait a minute or 2
-----> Running bats test suite
 ✓ Install helper packages for testing
 ✓ Sensu is installed
 ✓ Sensu-server is running
 ✓ Sensu-server is enabled
 ✓ Sensu-api is running
 ✓ Sensu-api is enabled
 ✓ Sensu-client is running
 ✓ Sensu-client is enabled
 ✓ Validate the compiled Sensu configuration
 ✓ We can authenticate to the sensu api
 ✓ Sensu /info endpoint shows transport is connected
 ✓ Sensu /info endpoint shows redis is connected
 ✓ Ensure uchiwa is running
 ✓ Ensure uchiwa is enabled
 ✓ Uchiwa health is ok
 ✗ Uchiwa can connect to sensu-api
          (in test file /tmp/verifier/suites/bats/02_uchiwa.bats, line 30)
            `[[ "$output" =~ "ok" ]]' failed
          Connection error. Is the Sensu API running?

The /etc/sensu/conf.d/api.json file will now look like this
{
  "api": {
    "host": "<IP_ADDRESS>",
    "port": 4567,
    "user": "sensu",
    "password": "somepassword",
    "bind": "localhost"
  }
}

I will then see these logs in /var/log/sensu/sensu-api.log
{"timestamp":"2017-02-02T17:11:30.459031+0000","level":"warn","message":"loading config file","file":"/etc/sensu/conf.d/api.json"}
{"timestamp":"2017-02-02T17:11:30.459105+0000","level":"warn","message":"config file applied changes","file":"/etc/sensu/conf.d/api.json","changes":{"api":[null,{"host":"<IP_ADDRESS>","port":4567,"user":"sensu","password":"REDACTED","bind":"localhost"}]}}
{"timestamp":"2017-02-02T17:11:30.671168+0000","level":"info","message":"api listening","protocol":"http","bind":"localhost","port":4567}

Possible Solution
Don't allow an empty configuration hash to stomp changes in /etc/sensu/conf.d/*.json
Steps to Reproduce (for bugs)
/etc/sensu/config.json
{
}

/etc/sensu/conf.d/api.json
{
  "api": {
    "host": "<IP_ADDRESS>",
    "port": 4567,
    "user": "sensu",
    "password": "somepassword",
    "bind": "<IP_ADDRESS>"
  }
}

Context
Our uchiwa deployment stops communicating with the sensu-api because localhost on this vagrant box binds to ipv6 first.
Your Environment

Sensu version used: sensu-0.26.5-2.x86_64
Active Sensu extensions: none
Operating System and version: CentOS 7.2
Transport and runtime versions (e.g. RabbitMQ 3.6.2 on Erlang 18.2): rabbitmq-server-3.3.5-28.el7.noarch and erlang 03.18.el7.x86_64
Redis: redis-2.8.19-2.el7.x86_64

I should note that this only happens when using "bind": "<IP_ADDRESS> and not "bind": "<IP_ADDRESS>".

Hi @pgporada, thanks for opening this issue. I've tried to reproduce the behavior you're describing but I haven't been successful.
In my testing I've created /etc/sensu/config.json with an empty JSON document:
root@default-ubuntu-1604:~# cat /etc/sensu/config.json
{}

I've also created /etc/sensu/conf.d/more_config.json with API bind and host attributes set to <IP_ADDRESS>, as shown in your example:
root@default-ubuntu-1604:~# cat /etc/sensu/conf.d/more_config.json
{
  "transport": {
    "reconnect_on_error": true,
    "name": "redis"
  },
  "api": {
    "host": "<IP_ADDRESS>",
    "bind": "<IP_ADDRESS>",
    "port": 4567
  }
}

With this configuration in place, I use the --print_config flag for sensu-client to provide me with details on the order in which the config is loaded, and the final configuration after deep merging is complete:
root@default-ubuntu-1604:~# sudo -u sensu /opt/sensu/bin/sensu-client -c /etc/sensu/config.json -d /etc/sensu/conf.d --print_config
{"timestamp":"2017-02-03T18:09:20.786112+0000","level":"warn","message":"loading config file","file":"/etc/sensu/config.json"}
{"timestamp":"2017-02-03T18:09:20.786870+0000","level":"warn","message":"loading config files from directory","directory":"/etc/sensu/conf.d"}
{"timestamp":"2017-02-03T18:09:20.787439+0000","level":"warn","message":"loading config file","file":"/etc/sensu/conf.d/more_config.json"}
{"timestamp":"2017-02-03T18:09:20.788417+0000","level":"warn","message":"config file applied changes","file":"/etc/sensu/conf.d/more_config.json","changes":{"transport":{"name":["rabbitmq","redis"]},"api":[null,{"host":"<IP_ADDRESS>","bind":"<IP_ADDRESS>","port":4567}]}}
{"timestamp":"2017-02-03T18:09:20.788903+0000","level":"warn","message":"applied sensu client overrides","client":{"name":"default-ubuntu-1604","address":"<IP_ADDRESS>","subscriptions":["client:default-ubuntu-1604"]}}
{"timestamp":"2017-02-03T18:09:20.789868+0000","level":"warn","message":"outputting compiled configuration and exiting"}
{
  "client":{
    "name":"default-ubuntu-1604",
    "address":"<IP_ADDRESS>",
    "subscriptions":[
      "client:default-ubuntu-1604"
    ]
  },
  "sensu":{
    "spawn":{
      "limit":12
    }
  },
  "transport":{
    "name":"redis",
    "reconnect_on_error":true
  },
  "checks":{},
  "filters":{},
  "mutators":{},
  "handlers":{},
  "extensions":{},
  "api":{
    "host":"<IP_ADDRESS>",
    "bind":"<IP_ADDRESS>",
    "port":4567
  }
}

From my perspective this is working as intended, and the empty JSON document in /etc/sensu/config.json has no observable effect.
Your suggestion to "Wait a minute or 2" makes me wonder if some other part of your automation is overwriting the Sensu configuration on disk.
For what its worth, changing the value of the bind attribute from "<IP_ADDRESS>" to "<IP_ADDRESS>" may also require a change to Uchiwa's configuration. Can you check your Uchiwa config (usually /etc/sensu/dashboard.json or under /etc/sensu/dashboard.d) to ensure it's using the same "<IP_ADDRESS>" address ?

Here is my /etc/sensu/uchiwa.json
 {
   "sensu": [{
        "name": "GLVagrant",
        "user": "sensu",
        "pass": "somepassword",
        "host": "<IP_ADDRESS>",
        "port": 4567,
        "ssl": false,
        "insecure": true,
        "timeout": 60
   }],
   "uchiwa": {
     "host": "<IP_ADDRESS>",
     "port": 3000,
     "loglevel": "info",
     "interval": 5,
     "users": [
                {
            "username" : "phil",
            "password": "phil",
            "readonly": false
                    }
    ]}
}

Sensu currently treats any configuration in /etc/sensu as read-only and is unable to modify its configuration during runtime. As we were unable to reproduce this issue I am going to close this issue.
As for why localhost doesn't work but <IP_ADDRESS> does, localhost can sometimes resolve to an IPv6 address on dual-stack systems which can cause the API to only listen on IPv6. The resolution for this is to listen on an IPv4 only address (e.g. <IP_ADDRESS>) or the dual-stack equivalent of <IP_ADDRESS> which is :: or sometimes [::].


notification over a webhook URL

Describe the solution you'd like
To integrate easily different notification system, we could add a "webhook" notify package, where utask admins could declare the URL, method and input body, using templating to replace value and format the message

cc @rclsilver

Closed in #194


Support for Net.core 1.1.0 projects

Hi, first of all many thanks for sharing this code.
I've been using successfully your library for executing as a windows service a Net.Core 1.0.1 project.
Problem is when I updated the project to Net.Core 1.1.0 I receive a weird error message when the app services starts: Error: 0xfffffffff 0xffffffff.
Maybe the reason is your library doesn't support Net.Core 1.1.0.
If that is the reason, do you plan to support it?

I didn't yet try to upgrade the sample project but yes i do plan to support it.

Great :)

So i just updated the test service to 1.1 (see the feature/aspnetcore-1-1-0-upgrade branch) and that passes smoke testing. (Just waiting for appveyor to update their images so i don't have to add a custom download build step)
Do you maybe have a repo for your error?
You could also run into an initialisation error with asp.net core 1.1 making your Start() method throw. This currently makes the service report -1 / 0xffffffff to windows here.

I'm using your library to allow execute Orchard CMS as windows service. This is the branch where  I'm using it https://github.com/OrchardCMS/Orchard2/tree/runAsAService .
The point is when I execute it in standalone mode it works properly despite Start code is very similar when it is run as a service instead of standalone. https://github.com/OrchardCMS/Orchard2/blob/runAsAService/src/Orchard.Cms.Web/OrchardCmsWin32Service.cs#L47
The start project is Orchard.Cms.Web
For building it you need to execute: ´build.cmd´ from root folder
For publishing it from \src\Orchard.Cms.Web folder dotnet publish --configuration Release --output c:\OrchardCMS
For running it fromc:\OrchardCMS\ folder dotnet orchard.cms.web.dll --register-service -urls http://localhos:5000`
But if you prefer a simpler repro project tomorrow I will try to reproduce it in a more simple one.

So if i follow your exact instructions i get an exception because it is meant to be --urls and not -url (one dash):
The short switch '-urls' is not defined in the switch mappings.
   at Microsoft.Extensions.Configuration.CommandLine.CommandLineConfigurationProvider.Load()
   at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers)
   at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
   at Orchard.Cms.Web.OrchardCmsWin32Service.Start(String[] startupArguments, ServiceStoppedCallback serviceStoppedCallback) in C:\Users\martin.ullrich\Documents\Repos\Orchard2\src\Orchard.Cms.Web\OrchardCmsWin32Service.cs:line 48

If i leave that out, it works for me.. (at least it starts, i didn't invest time to make the right node version work that was used by the build).

 I didn't noticed I made this typo
Thank you!

Glad to hear your issue is resolved 😄
My learning from this is that maybe the additional parameters passed to --register-service should be checked before registering them.. maybe the CommandLineConfigurationProvider class from the Microsoft.Extensions.Configuration can be used to perform basic validation (an invalid URL passed to --urls will still fail. just like it will fail when the port is already in use).
Besides, you should consider updating the Microsoft.Extensions.Configuration to 1.1.0 as well.