PDS4 Indexing Tools Improvements and Additions

The indexing tool (now pds4indextools.py) now has three new argparse commands:

    --dump-available-xpaths
    --simplify-xpaths (previously --disambiguate-xpaths)
    --fixed-width

Major changes have been implemented into the code to make the result more human-readable and to fix several major bugs. Some of these issues include missing indexed data, inaccurate XPaths, and Windows incompatibility)

Added unit tests and associated supplemental files, which as of 6/17 all pass:

tests/ contains the unit tests for the indexing tool
test_files/expected contains golden copies for unit test comparison
test_files/samples contains supplemental files for the unit tests to use
test_files/labels contains fake label files for the unit tests to use

TODO: Unit test for --fixed-width has not been implemented.

Documentation has been fully incorporated into the indexing tool, and has also been reformatted to follow Google style.

TODO: The label generation feature, which includes:
- label generation function
- template label file
- default input file (called default_values.yaml)
- Associated unit tests and supplemental files
Has yet to be fully implemented into the tool. Current state of things:
- Label generation function: fully operational, not incorporated into the indexing tool
- Template label file: Still in progress, does not currently generate a validating label.
- Default input file: made and functional
- Unit tests and supplemental files: not started

I think maybe --output-file should be a required argument. Right now it defaults to putting the resultant CSV file in the root directory of the bundle/collection, and calling it index_file.csv. This is almost certainly the wrong thing to do. First of all, people probably don't want to modify their archives unless they know the result is good. Second they are unlikely to want to just call it index_file.csv. And if you don't know what the default is, the file is actually really hard to find. So maybe we should just force people to specify the filename/path.

Here's another thought. Right now --output-file is generic - it will produce a CSV file or a TXT file full of elements. Maybe it would make more sense to have different command line options for different outputs, like --output-csv-file and --output-xpaths-file and then get rid of --dump-available-xpaths as an option. Then you could actually output both a CSV file and an XPath file at the same time, which could be handy. If you don't specify any output file, print a message and exit.

The test coverage is only 85%, and if you look at the coverage HTML report you'll see a lot of code that really needs to be tested. It also shows some code that might be obsolete and never used anymore, in which case you should delete it. I think this program should be at 100% test coverage. It won't take that much effort to get there.

It seems that, if there is an attribute that appears in some of the indicated product labels but not others, then the column will be created and the value of "NaN" will be entered in that column for those product labels that do not include the attribute.  Is this intended?

It seems that, if there is an attribute that appears in some of the indicated product labels but not others, then the column will be created and the value of "NaN" will be entered in that column for those product labels that do not include the attribute. Is this intended?

As a side note, it's a very bad idea to create index files from labels that have dramatically different layouts. The resulting index will just have a lot of empty columns and won't make much sense. This is why, for example, we create three different index files for uranus_occs_earthbased - even though rings, global, and atmosphere have a lot in common, they are different enough that having a single index file is just confusing. I'm mentioning this here mainly so that @esimpsons3ti will remember to put this as a big important note in the documentation she will write.
That said, there are invariably small changes between labels in the same data set. For example, there might be a different number of keywords, or SPICE kernels, or auxiliary products. So we need to be able to handle those cases.
The question is: What do we put in the CSV file when the data is "missing"? The current code puts in NaN for both string and numeric fields, which I agree is not great. But what should we put in? I would argue that maybe the fields should just be empty. Alternatively we could make it used-configurable.

A few notes about this:

This behavior was due to an element within the batch of tested label files being nilled when they weren't nillable (in this case, Complex_FOV). Previously, I didn't have code that would go in and look for the data type of the erroneously-nilled element. It would just print a warning and move on.
Now that I do have this code, I have integrated it into store_element_text. Now, if a non-nillable element is nilled, it will still print the warning, but there will no longer be NaNs placed where the default value is.

I don't think this is the only time this can happen. I produced NaNs myself using the Uranus occ bundles.

The Cassini VIMS bundle contains a number of "composite structures" that are broken up into individual data products.  The data products that are part of a composite structure have an entire additional class full of attributes that describes how they relate to the rest of the structure.  For example, in this directory, the data products ending with _xxx have that additional class, while those that don't don't.
It is quite reasonable to want an index file that includes all of these products together, which means that the case arises of columns that are present in some labels but absent from others.
Do we need to inquire into what is the data type of each column in order to define the special constants for when they are missing?

flake8 has been failing the code on line length, even when my flake8.txt file is set to the 90 character limit. But I'll wait on your feedback for now.

I ran the tool a couple days ago.  It initially didn't work, but @esimpsons3ti sat down with me and diagnosed the problem.  It was this line in the process_schema_location function:
tree = etree.parse(file_path)
For some reason, the way I configured the VSCode JSON was inputting file_path as a PosixPath rather than as a string, which broke the code.  We fixed it by simply changing that line so that it read
tree = etree.parse(str(file_path))
I'm not sure whether this fix, or a different fix for that situation, has been added to the branch.
At this point I cannot run the tool at all because yesterday's changes seem to have introduced a requirement of pdstemplate.  I have the rms-pdstemplate repo on my machine, but I do not know how to get this tool to find it.  This might be an occasion for me to improve my usage of VSCode and maybe even virtual machines (which I do not currently use but probably should).
More generally, is there documentation that we would expect a novice user to read that would enable them to run the program?  I see the docstring, but it doesn't address questions like what software must you have installed or what commands you must execute in order to be able to run the tool successfully.

@matthewtiscareno I understand that the docstrings within the program are not sufficient for a user to go off of. The current plan is to save creating the README file for last. However, I do intend on making a draft of it today.

@rfrenchseti Correct, there are currently no tests for label generation. That will be today's project, alongside general unit test coverage.

I'm looking at the --help with the eyes of a brand new user.

I'm finding these confusing:

  --output-csv-file OUTPUT_CSV_FILE
                        Specify the location and filename of the index file.
  --output-txt-file OUTPUT_TXT_FILE
                        Specify the location and filename of the XPath headers file.

Why is the option to specify the index file called --output-csv-file instead of --output-index-file? Why is the option to specify the headers file called --output-txt-file instead of --output-headers-file? Also the capitalized string should probably be something like FILENAME to make it clear what you specify as an argument. If you just duplicate the name of the option it's confusing.

--elements-file doesn't really specify elements, right? An element is a pair of tags plus the thing between the tags, like value. What you're actually specifying is full XPaths to scrape. So maybe it should be called something like --limit-xpaths-file?

Similar to above, --sort-by SORT_BY should say something useful instead of SORT_BY, like HEADER_NAME.

In

  --add-extra-file-info {LID,filename,filepath,bundle_lid,bundle} [{LID,filename,filepath,bundle_lid,bundle} ...]
                        Add additional columns to the final index file. Choose from the following: "LID", "filename", "filepath", "bundle_lid", and "bundle". If specifying multiple column
                        names, supply them as separate arguments separated by spaces

you don't need the "Choose from" sentence, because the options are already listed above. Also there should be a period at the end.

For

  --config-file CONFIG_FILE
                        Read a user-specified configuration file. File must be a .ini file.

It's not true the file has to be a .ini file. I think you want to say "The file must be in .ini file format."

For

  --dump-available-xpaths
                        Give a .txt file of all XPaths within the given label files. This file can be used as a base file for --elements-file.

"Give" is ambiguous. Maybe say "Write". Does this go along with --output-txt-file? If so, it would be a lot easier to have a single command line option --dump-available-xpaths FILENAME. If it's present, do the dump to the given file. If it's not present, don't dump anything. Also, can you clarify whether these XPaths are before or after simplification? I think they need to be before, because otherwise you can't use them in the "elements" file if you have --simplify-xpaths turned on.

For

  --generate-label {Product_Ancillary,Product_Metadata_Supplemental}
                        Generate a PDS4 label for the generated index file. Can generate either a Product_Ancillary or Product_Metadata_Supplemental label.

what is the filename of the resulting label?

For

  --label-user-input LABEL_USER_INPUT
                        Provide an optional .yaml file containing additional information for the generated label.

again it doesn't have to be a file with the extension .yaml, it just has to be in YAML format.

I would add an argparse epilog that refers people to the online documentation for more details.

Why is there both a "config file" and a "label user input" file? Wouldn't it be easier to put the label info into the config file? Or do we expect the user to want to keep those separate?

The order of the options as displayed is important to the user's understanding. In fact, I might group them together using argparse's "argument_group" concept. The groupings would be, in this order, with the options in this order (open to discussion):

Index file generation:

--output-csv-file
--add-extra-file-info
--sort-by
--fixed-width
--clean-header-field-names
--simplify-xpaths

Limiting XPaths:

--elements-file
--dump-available-xpaths
--output-txt-file

Label file generation:

--generate-label
--label-user-input

Miscellaneous:

--config-file
--verbose

Now that I look at the code, it seems like --dump-available-xpaths is not actually used for anything anymore and that you already did the change to make --output-txt-file do all the work. So you just have to remove the dead argument.

I think in the -add-extra-file-info option, we shouldn't capitalize LID since we aren't capitalizing bundle_lid.

If you specify both --output-txt-file and --output-csv-file, you only get the txt file. It silently fails to write the csv file, which is really confusing, especially if you already had a csv file there that you were trying to update (I spent the last 10 minutes trying to figure out why my csv file had the wrong contents). I see no reason that we can't allow writing both the csv and the header file in the same run of the program.

--simplify-xpaths is changing the order of the header fields. It should keep them the same. See the attached two files as examples.
rings.csv
rings_simp.csv

Separating the arguments to --add-extra-file-info and --sort-by as separate arguments means that argparse will fail to parse the command line in some cases. For example:
❯ python pds4_create_xml_index.py --add-extra-file-info filepath filename LID /mnt/rms-holdings/pds4-holdings/bundles/uranus_occs_earthbased/uranus_occ_u0_kao_91cm "**/data/rings/*0m.xml" --verbose --generate-label Product_Ancillary
usage: pds4_create_xml_index.py [-h] [--output-csv-file OUTPUT_CSV_FILE] [--output-txt-file OUTPUT_TXT_FILE] [--elements-file ELEMENTS_FILE] [--simplify-xpaths] [--verbose]
                                [--sort-by SORT_BY [SORT_BY ...]] [--clean-header-field-names]
                                [--add-extra-file-info {LID,filename,filepath,bundle_lid,bundle} [{LID,filename,filepath,bundle_lid,bundle} ...]] [--config-file CONFIG_FILE]
                                [--dump-available-xpaths] [--fixed-width] [--generate-label {Product_Ancillary,Product_Metadata_Supplemental}] [--label-user-input LABEL_USER_INPUT]
                                directorypath patterns [patterns ...]
pds4_create_xml_index.py: error: argument --add-extra-file-info: invalid choice: '/mnt/rms-holdings/pds4-holdings/bundles/uranus_occs_earthbased/uranus_occ_u0_kao_91cm' (choose from 'LID', 'filename', 'filepath', 'bundle_lid', 'bundle')

This happens whenever one of the arguments that takes multiple values separated by spaces is followed by the required directory name and patterns. This will be confusing to the user, because we explicitly say the directory name and patterns should come last. In fact, I think they should be able to be provided in any order. The only fix for this is to change the behavior of --add-extra-file-info and --sort-by. I think they should each take a comma-separated list of items instead of a + set of arguments. Then just do a split(',') on the values to get the argument values. You'll have to write your own validation function now, since argparse won't be able to do it. But this is easy to do. Here's an example:
def _validate_planet(planet):
    """Routine for argparse to validate a planet."""
    if not planet in ("jupiter", "saturn", "uranus", "neptune"):
        raise argparse.ArgumentTypeError(
             f"{planet} is not a valid planet - must be one of "
              "jupiter, saturn, uranus, neptune")
    return planet

group.add_argument(
    "--planet", default="saturn",
    type=_validate_planet,
    help=f"""Which planet to process: jupiter, saturn, uranus, neptune
             (saturn is the default)""")

If you don't specify any output path, you get the error:
Output path not chosen.

First of all, you can print an error for this long before you scrape all the labels. Right now it does all the label scraping (which can take a long time) and then complains. But the use of passive voice and "chosen" is confusing. I would suggest something like "You must specify at least one of: --output-csv-file or --output-txt-file" etc.

Generating a label file is crashing for me:
❯ python pds4_create_xml_index.py /mnt/rms-holdings/pds4-holdings/bundles/uranus_occs_earthbased/uranus_occ_u0_kao_91cm "**/data/rings/*0m.xml" --generate-label Product_Ancillary --output-csv-file rings2.csv
Traceback (most recent call last):
  File "/seti/all_repos/rms-pds4indextools/pds4indextools/pds4_create_xml_index.py", line 1283, in <module>
    main()
  File "/seti/all_repos/rms-pds4indextools/pds4indextools/pds4_create_xml_index.py", line 1217, in main
    true_type = find_base_attribute(xsd_tree, name)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/seti/all_repos/rms-pds4indextools/pds4indextools/pds4_create_xml_index.py", line 783, in find_base_attribute
    target_attribute_value = target_attribute_values[0]
                             ~~~~~~~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range

When you dump the elements file with --output-txt-file while using --simplify-xpaths, you get the simplified XPaths twice, once in their simplified form and once in their original form. See the attached file. Example (excerpt from full file):
pds:logical_identifier
pds:title
pds:information_model_version
pds:Product_Observational/pds:Identification_Area<1>/pds:logical_identifier<1>
pds:Product_Observational/pds:Identification_Area<1>/pds:title<1>
pds:Product_Observational/pds:Identification_Area<1>/pds:information_model_version<1>

generated with:
python pds4_create_xml_index.py /mnt/rms-holdings/pds4-holdings/bundles/uranus_occs_earthbased/uranus_occ_u0_kao_91cm "**/data/rings/*0m.xml" --simplify-xpaths --output-txt-file rings2_simp.txt

rings2_simp.txt

If you don't specify any output path, you get the error:
Output path not chosen.

First of all, you can print an error for this long before you scrape all the labels. Right now it does all the label scraping (which can take a long time) and then complains. But the use of passive voice and "chosen" is confusing. I would suggest something like "You must specify at least one of: --output-csv-file or --output-txt-file" etc.

If the user doesn't specify an output path, wouldn't it be better to write to a default output path, rather than for the tool to crash?

If you don't specify any output path, you get the error:
Output path not chosen.

First of all, you can print an error for this long before you scrape all the labels. Right now it does all the label scraping (which can take a long time) and then complains. But the use of passive voice and "chosen" is confusing. I would suggest something like "You must specify at least one of: --output-csv-file or --output-txt-file" etc.

If the user doesn't specify an output path, wouldn't it be better to write to a default output path, rather than for the tool to crash?

Previously if you didn't specify an output file, it created a file with a name like index.csv in the root directory of the bundles/collections. I argued that this was a bad default, because it was writing into the (possibly read-only, and at least archival) directory structure and thus modifying something the user wasn't expecting to be modified. I think it would be perfectly fine to write into a default filename as long as 1) the filename was in the user's current directory, not the bundle root, 2) the program printed a message saying what filename it was writing and, and maybe 3) didn't overwrite a file that was already there.

I think it would be perfectly fine to write into a default filename as long as 1) the filename was in the user's current directory, not the bundle root, 2) the program printed a message saying what filename it was writing and, and maybe 3) didn't overwrite a file that was already there.

I totally agree.  I might accomplish #3 by first checking whether index.csv already exists in the user's current directory.  If it does, then change the output name to index1.csv.  If that also already exists, then go on to index2.csv, and so forth.

This will be confusing to the user, because we explicitly say the directory name and patterns should come last.

Where is that said? They're listed first in the examples we gave.

Generating a label file is crashing for me:
❯ python pds4_create_xml_index.py /mnt/rms-holdings/pds4-holdings/bundles/uranus_occs_earthbased/uranus_occ_u0_kao_91cm "**/data/rings/*0m.xml" --generate-label Product_Ancillary --output-csv-file rings2.csv
Traceback (most recent call last):
  File "/seti/all_repos/rms-pds4indextools/pds4indextools/pds4_create_xml_index.py", line 1283, in <module>
    main()
  File "/seti/all_repos/rms-pds4indextools/pds4indextools/pds4_create_xml_index.py", line 1217, in main
    true_type = find_base_attribute(xsd_tree, name)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/seti/all_repos/rms-pds4indextools/pds4indextools/pds4_create_xml_index.py", line 783, in find_base_attribute
    target_attribute_value = target_attribute_values[0]
                             ~~~~~~~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range

This has been fixed with changes to --simplify-xpath's behavior.

This will be confusing to the user, because we explicitly say the directory name and patterns should come last.

Where is that said? They're listed first in the examples we gave.

It's in the --help output. I don't know why it's last there, since you put it first in your list of arguments, but I guess it just lists unnamed arguments at the end by default. Still, I think having multiple values be separated by spaces is confusing. That's not something command line programs normally do for multiple options - they have them separated by commas. If we just changed that convention, it would solve this problem entirely.
usage: pds4_create_xml_index.py [-h] [--output-csv-file OUTPUT_CSV_FILE] [--output-txt-file OUTPUT_TXT_FILE] [--elements-file ELEMENTS_FILE] [--simplify-xpaths] [--verbose]
                                [--sort-by SORT_BY [SORT_BY ...]] [--clean-header-field-names]
                                [--add-extra-file-info {LID,filename,filepath,bundle_lid,bundle} [{LID,filename,filepath,bundle_lid,bundle} ...]] [--config-file CONFIG_FILE]
                                [--dump-available-xpaths] [--fixed-width] [--generate-label {Product_Ancillary,Product_Metadata_Supplemental}] [--label-user-input LABEL_USER_INPUT]
                                directorypath patterns [patterns ...]

Separating the arguments to --add-extra-file-info and --sort-by as separate arguments means that argparse will fail to parse the command line in some cases. For example:
❯ python pds4_create_xml_index.py --add-extra-file-info filepath filename LID /mnt/rms-holdings/pds4-holdings/bundles/uranus_occs_earthbased/uranus_occ_u0_kao_91cm "**/data/rings/*0m.xml" --verbose --generate-label Product_Ancillary
usage: pds4_create_xml_index.py [-h] [--output-csv-file OUTPUT_CSV_FILE] [--output-txt-file OUTPUT_TXT_FILE] [--elements-file ELEMENTS_FILE] [--simplify-xpaths] [--verbose]
                                [--sort-by SORT_BY [SORT_BY ...]] [--clean-header-field-names]
                                [--add-extra-file-info {LID,filename,filepath,bundle_lid,bundle} [{LID,filename,filepath,bundle_lid,bundle} ...]] [--config-file CONFIG_FILE]
                                [--dump-available-xpaths] [--fixed-width] [--generate-label {Product_Ancillary,Product_Metadata_Supplemental}] [--label-user-input LABEL_USER_INPUT]
                                directorypath patterns [patterns ...]
pds4_create_xml_index.py: error: argument --add-extra-file-info: invalid choice: '/mnt/rms-holdings/pds4-holdings/bundles/uranus_occs_earthbased/uranus_occ_u0_kao_91cm' (choose from 'LID', 'filename', 'filepath', 'bundle_lid', 'bundle')

This happens whenever one of the arguments that takes multiple values separated by spaces is followed by the required directory name and patterns. This will be confusing to the user, because we explicitly say the directory name and patterns should come last. In fact, I think they should be able to be provided in any order. The only fix for this is to change the behavior of --add-extra-file-info and --sort-by. I think they should each take a comma-separated list of items instead of a + set of arguments. Then just do a split(',') on the values to get the argument values. You'll have to write your own validation function now, since argparse won't be able to do it. But this is easy to do. Here's an example:
def _validate_planet(planet):
    """Routine for argparse to validate a planet."""
    if not planet in ("jupiter", "saturn", "uranus", "neptune"):
        raise argparse.ArgumentTypeError(
             f"{planet} is not a valid planet - must be one of "
              "jupiter, saturn, uranus, neptune")
    return planet

group.add_argument(
    "--planet", default="saturn",
    type=_validate_planet,
    help=f"""Which planet to process: jupiter, saturn, uranus, neptune
             (saturn is the default)""")

Currently, --sort-by can sort by anything, not just the values in --add-extra-file-info. Do we want to limit --sort-by to only sort by the allowed values for --add-extra-file-info, or continue to allow users to sort by anything?

Currently, --sort-by can sort by anything, not just the values in --add-extra-file-info. Do we want to limit --sort-by to only sort by the allowed values for --add-extra-file-info, or continue to allow users to sort by anything?

I'm not sure where this question is coming from. Why would we change the behavior of --sort-by? The only thing I'm saying is that instead of: --sort-by field1 field2 field3 we require --sort-by field1,field2,field3. Just take a single string argument instead of "+", and split on commas to get the list of fields. For the additional file info fields, you'll have to validate the entries in that new list.

Note: Unit test for label generation have not been added yet. I'm just making a new commit to contain the changes from the above comments.

I am approving this pull, with the understanding that there are additional issues that need to be addressed, including:

#18
#17
#16


Propagate ValueSource description for missing values

Check if a ValueSource is Describable and include the DisplayName in the path of the Missing Value.

Part of #25716
Context

Contributor Checklist

[x] Review Contribution Guidelines.
[x] Make sure that all commits are signed off to indicate that you agree to the terms of Developer Certificate of Origin.
[x] Make sure all contributed code can be distributed under the terms of the Apache License 2.0, e.g. the code was written by yourself or the original code is licensed under a license compatible to Apache License 2.0.
[x] Check "Allow edit from maintainers" option in pull request so that additional changes can be pushed by Gradle team.
[x] Provide integration tests (under <subproject>/src/integTest) to verify changes from a user perspective.
[x] Provide unit tests (under <subproject>/src/test) to verify logic.
[ ] Update User Guide, DSL Reference, and Javadoc for public-facing changes.
[x] Ensure that tests pass sanity check: ./gradlew sanityCheck.
[x] Ensure that tests pass locally: ./gradlew <changed-subproject>:quickTest.

Reviewing cheatsheet
Before merging the PR, comments starting with

❌ ❓must be fixed
🤔 💅 should be fixed
💭 may be fixed
🎉 celebrate happy things

Thank you for your contribution!
This PR has a valid DCO and tests. The relevant team for this area will confirm the design of the implementation choices.

@gradle/bt-configuration Please confirm that it fixes the referenced issue or is one step towards that

Thanks, @urdak!

The merge queue build has failed. Click here to see all failures.

Looks like the test I added fails on Windows:
 e.message == """Cannot query the value of this provider because it has no value available. The value of this provider is derived from: nullValueSource"""
  | |       |
  | |       false
  | |       1 difference (99% similarity)
  | |       Cannot query the value of this provider because it has no value available.(\r)\nThe value of this provider is derived from: nullValueSource
  | |       Cannot query the value of this provider because it has no value available.(-~)\nThe value of this provider is derived from: nullValueSource

Should I resolve this issue myself? I guess just adding System.lineSeparator() instead a new line should fix it.

Sorry for the lag, @urdak, I fixed it via TextUtil.toPlatformLineSeparators.

The merge queue build has failed. Click here to see all failures.


chore(main): release 2.22.0

:robot: I have created a release beep boop
2.22.0 (2024-09-23)
Features

rename translations (sync to link) (#192) (bb4aa11)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/ai4os/ai4-dashboard/releases/tag/v2.22.0 :sunflower:


Chip test failures: missing error code from the u_ram1p_ram_ret?

The nightly chip regressions contain failures with error messages like this:
UVM_ERROR @<PHONE_NUMBER> ps: (cip_base_scoreboard.sv:128) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] Check failed item.d_error == is_tl_err (0 [0x0] vs 1 [0x1]) unmapped: 0, mem_access_err: 1, csr_aligned_err: 0, csr_size_err: 0,                     item_err: 0

To reproduce, checkout version d54a450 (current head of master) and run:
util/dvsim/dvsim.py --fixed-seed<PHONE_NUMBER> hw/top_earlgrey/dv/chip_sim_cfg.hjson -i chip_csr_mem_rw_with_rand_reset -w -v h

A small amount of digging (mostly enabling verbose error messages) seems to show that this is caused by the scoreboard expecting an error with the following item. The generated address map seems to place this somewhere in "ram_ret", and I do indeed see the transaction there in the waves.
UVM_INFO @<PHONE_NUMBER> ps: (cip_base_scoreboard.sv:53) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] received tl d_chan item:
----------------------------------------------
Name        Type              Size  Value
----------------------------------------------
req         chip_tl_seq_item  -     @656170
  a_addr    integral          32    'h1800011c
  a_data    integral          32    'hbadff0f6
  a_mask    integral          4     'h5
  a_size    integral          2     'h2
  a_param   integral          3     'h0
  a_source  integral          8     'h26
  a_opcode  integral          3     'h1
  d_param   integral          3     'h0
  d_source  integral          8     'h26
  d_data    integral          32    'h0
  d_size    integral          2     'h2
  d_opcode  integral          3     'h0
  d_error   integral          1     'h0
  d_sink    integral          1     'h0
  d_user    integral          16    'h0
----------------------------------------------

@tjaychen: I notice you added the retention RAM. Could you have a look at this to see what's come unstuck? Presumably either the RTL is buggy and should be returning an error code for this request, or the test infrastructure needs to be told something different.

o is this because we added a retention ram but this was not updated in the
dv environment somewhere? so it's still considered "empty" space?
On Fri, Jul 10, 2020 at 7:05 AM Rupert Swarbrick<EMAIL_ADDRESS>wrote:

@tjaychen https://github.com/tjaychen: I notice you added the retention
RAM. Could you have a look at this to see what's come unstuck? Presumably
either the RTL is buggy and should be returning an error code for this
request, or the test infrastructure needs to be told something different.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/lowRISC/opentitan/issues/2775#issuecomment-656694181,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAH2RSXJAPVRF3TV7BZUFW3R24NZHANCNFSM4OWT3DPQ
.

nvm... it looks like the memories are auto-discovered..and i think the
error message is saying this is not an unmapped error? Is that right?
On Fri, Jul 10, 2020 at 12:42 PM Timothy Chen<EMAIL_ADDRESS>wrote:

o is this because we added a retention ram but this was not updated in the
dv environment somewhere? so it's still considered "empty" space?
On Fri, Jul 10, 2020 at 7:05 AM Rupert Swarbrick<EMAIL_ADDRESS>wrote:

@tjaychen https://github.com/tjaychen: I notice you added the
retention RAM. Could you have a look at this to see what's come unstuck?
Presumably either the RTL is buggy and should be returning an error code
for this request, or the test infrastructure needs to be told something
different.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/lowRISC/opentitan/issues/2775#issuecomment-656694181,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAH2RSXJAPVRF3TV7BZUFW3R24NZHANCNFSM4OWT3DPQ
.

@tjaychen: Your patch in #2787 looks like it should have fixed this. Did it? In which case, could you close the issue too?

yeah I think so. I will close this.


cargo prints color inside docker build despite --color=never

Problem
cargo build still prints color when running in docker build even with --color=never option specified.
Dockerfile:
FROM alpine:edge as builder
RUN apk add --no-cache rust cargo
WORKDIR /code
COPY . .
RUN cargo install --path . --color=never

Alpine Linux Edge installs rust and cargo version 1.41.
Steps

docker build -t code:latest .
Watch colors being printed.

Possible Solution(s)

Notes
Output of cargo version:

cargo 1.41.0
Same result on both Docker for Mac and AWS Codebuild with standard:2.0 builder image.

Can you post a screenshot of what you are seeing?  I see this:

This is normal behavior for Docker.  It prints all stderr content in red.

Thanks, that helped. I changed the cargo command to this, and then it stopped printing red:
RUN cargo install --path . --color never 2>&1
Why is cargo build output printed to stderr even when there are no build errors?

I guess I'll answer my own question.
POSIX defines:

standard output = for writing conventional output
standard error = for writing diagnostic output

Build progress is a diagnostic output so it belongs to standard error.


README file lists compatibility for 'Cisco DNA Center' instead of ISE

Just raising an issue for a misprint in the README file. The Compatibility Matrix section should reflect 'Cisco ISE version' instead of 'Cisco DNA Center version'.

Thank you. Closing this issue.


Add keptn-lifecycle-toolkit on data/repositories.toml

fixed issue #1742
Hi agardnerIT,
just trying to fix issue on file data/repositories.toml,
please check it out
ℹ Repository information
The repository has:

[X] At least three issues with the good first issue label.
[X] At least 10 contributors.
[X] Detailed setup instructions for the project.
[X] CONTRIBUTING.md
[X] Actively maintained.

Hi agardnerIT,
Is there anything else need to do to merge this PR ?
Please let me know

Is there anything else need to do to merge this PR ?

Not from my side. This repo isn't controlled by Keptn. Someone from deepsourcelabs needs to review.

Is there anything else need to do to merge this PR ?

Not from my side. This repo isn't controlled by Keptn. Someone from deepsourcelabs needs to review.

Maybe anyone can answer ?
@jai-deepsource @siddhant-deepsource @vishnu-deepsource @ashwin-deepsource

Sorry, I have to close this PR because no action from deepsourcelabs side


yaml.load() allows arbitrary code execution

Via Simon de Haan:
yaml.load() allows YAML files to execute arbitrary Python code using the Python tag scheme, e.g.:
yaml.load('!!python/object/apply:os.system ["..."]')

All yaml.load() calls should at least be replaced with yaml.safe_load(), which disallow this.

I've created branch feature/issue-111-always-use-yaml-safe-load for this.

Fixed in develop and 0.3.x.


Feature Request: Enhanced Rsync Support for S3 CSI Driver

Problem Description
When attempting to use rsync for file synchronization between a Kubernetes pod and an AWS S3 bucket mounted via the AWS S3 CSI driver, several challenges arise. The primary issues include rsync performing filesystem operations that are not supported by S3, such as permission setting and atomic renaming. This results in errors like "Operation not permitted" and "Function not implemented," complicating the use of rsync for data synchronization tasks.
Desired Solution
I propose the development of enhanced support for POSIX-like filesystem operations within the S3 service or specifically within the AWS S3 CSI driver to better accommodate file synchronization tools like rsync. The solution could include:

A compatibility layer or additional driver options that adapt rsync filesystem operations to be compatible with S3 object storage behaviors.
Support for preserving file metadata, such as timestamps, in a way that aligns with rsync's functionality.
Improved error handling to manage rsync expectations and common operation pitfalls, especially around file renaming and permissions.

Alternatives Considered
To address these challenges, I have explored:

Utilizing the AWS CLI s3 sync command for synchronization, which lacks some of rsync's advanced features and efficiency.
Developing custom scripts with AWS SDKs, which increases complexity and maintenance requirements.
Adjusting rsync command-line options to mitigate errors, which has not fully resolved the underlying compatibility issues.

Additional Context
Seamless integration of rsync with S3 would greatly benefit a wide array of applications, from backup systems to dynamic content management for web services, by simplifying data synchronization processes. Enhancing S3's compatibility with rsync would leverage S3's storage capabilities in distributed systems like Kubernetes, where efficient and reliable data synchronization is a frequent requirement.

Thank you for the request. The limitations on which posix operations are supported are not in the CSI driver, but mountpoint itself, which are documented here: https://github.com/awslabs/mountpoint-s3/blob/main/doc/SEMANTICS.md. I am going to close this issue and suggest you open a new issue in the mountpoint-s3 repo: https://github.com/awslabs/mountpoint-s3.
While making rsync just work does sound like a great feature, it seems unlikely mountpoint will implement things like permissions because it doesn't align with the tenets in that SEMANTICS doc. There is no close analog in s3 to unix filesystem permissions and it would be a performance hit to emulate them. It might make sense to make this opt in and is worth a discussion, but the mountpoint-s3 repo is the best place to have that discussion.


Helm install should add the LastAppliedConfig annotation during create

When helm does an install/upgrade it should add the LastAppliedConfig annotation so that upgrade doesn't fail when something is deleted from the manifest. For more information https://github.com/kubernetes/kubernetes/issues/35134.

Specifically:
https://github.com/kubernetes/kubernetes/issues/35134#issuecomment-259878680

@adamreese Is this solved among the Kube 1.5 updates, or are we still using the patch logic?
(Background: Originally Helm used patch instead of apply logic. There's a long historic explanation, but the short version is that we are trying to move to apply. However, that effort got delayed by the internal transition from Kubernetes 1.4 libraries to 1.5)

@kmala I think was fixed, can you check if you are still having the issue on 2.1.0?

Tested with hem v2.1.0 and the issue is fixed.


bug: Impossible to set maplocalleader without warning

Did you check docs and existing issues?

[X] I have read all the LazyVim docs
[X] I have searched the existing issues of LazyVim
[X] I have searched the existing issues of plugins related to this issue

Neovim version (nvim -v)
v0.10.0-dev-2281+g0e9a33572
Operating system/version
Arch Linux
Describe the bug
After the commit adding a warning when maplocalleader changes it's now impossible to set maplocalleader to a value other than "\" without a warning.
lua/lazyvim/config/options.lua sets maplocalleader, before it gets loaded by lazy. This means that even the following code gets a warning:
vim.g.maplocalleader=" " -- this is overridden
require("config.lazy") -- maplocalleader gets reset *and* stored here
vim.g.maplocalleader=" " -- this is warned about

Steps To Reproduce

Set maplocalleader before calling lazy, this will be lost, even though it's suggested
call require "config.lazy"
Set maplocalleader to create new mappings (this is warned about!)

Expected Behavior
No warning if mapleader or maplocalleader are set before calling lazy.
Since these values start as nil, lazyvim/config/options.lua could be replaced with
vim.g.mapleader = vim.g.mapleader or " "
vim.g.maplocalleader = vim.g.maplocalleader or "\\"

therefore setting them only if the user hasn't already set them.
Repro
-- DO NOT change the paths and don't remove the colorscheme
local root = vim.fn.fnamemodify("./.repro", ":p")

-- set stdpaths to use .repro
for _, name in ipairs({ "config", "data", "state", "cache" }) do
  vim.env[("XDG_%s_HOME"):format(name:upper())] = root .. "/" .. name
end

-- bootstrap lazy
local lazypath = root .. "/plugins/lazy.nvim"
if not vim.loop.fs_stat(lazypath) then
  vim.fn.system({ "git", "clone", "--filter=blob:none", "https://github.com/folke/lazy.nvim.git", lazypath, })
end
vim.opt.runtimepath:prepend(lazypath)

-- install plugins
local plugins = {
  "folke/tokyonight.nvim",
  "folke/LazyVim",
  -- add any other plugins here
}

vim.g.maplocalleader=" " -- this gets overwritten

require("lazy").setup(plugins, {
  root = root .. "/plugins",
})

vim.g.maplocalleader=" " -- warning here (I just want to use space!)

vim.cmd.colorscheme("tokyonight")

That suggestion is for using lazy.nvim
LAzyVim loads lazy.nvim internally.
The correct way to set your leader and localleader is in your local options.lua

I first found this by having the setting in my local options.lua.
I had
vim.g.mapleader = “ “
vim.g.maplocalleader = “”

as the first lines of options.lua and get the warning message popup

actually, it was the header of keymaps.lua. is options loaded separately from the other config files?

yes, you are supposed to set this in your options.lua
See the docs https://www.lazyvim.org/configuration/general#options
Click on default options

keymaps are lazy loaded (most of the time)


Add HI (high influx) filter from PWSQC

In the training school notebook for PWSQC we do not have the implementation of the HI (high influx) filter. But Lotte provided a snippet of the R code:
HIflag <- rep(0, times=length(Nint))
   HIflag[which(((Nint > HIthresB) & (Med < HIthresA)) | ((Med >= HIthresA) & (Nint > (HIthresB*Med/HIthresA))))] <- 1  # if thresholds are exceeded, the HI flag becomes 1
   HIflag[which(Number_of_measurements < nstat)] <- -1	# if less than 'nstat' neighbours supply observations, the HI flag becomes equal to -1

   if(exists("HI_flags")==F){ HI_flags <- HIflag
	}else{ HI_flags <- cbind(HI_flags, HIflag) }

Note that this is copy-paste from the Zoom chat, hence, I am not sure about indentation and completeness of this code snippet.

For reference, the R notebook with all code, including the HI filter, is here.
That might help to understand what the individual variables mean.

First draft of HI-filter, not considering compability with FZ-filter, data input etc.
#set parameters
d = 10000 #meters
nstat = 5 # threshold for nr of stations within range d reporting data for given time step
HIthresA = 0.4 # threshold for median rainfall of stations within range d, mm
HIthresB = 10 #upper rainfall limit, mm
#Initialize HIflag with zeros
HIflag = np.zeros_like(X) #X size of dataset
def HI_filter(d,nstat, HIthresA,HIthresB):
for i in np.arange(np.shape(pws_data)[0]):
#Med = np.median(i,d) median rainfall amount of stations withing range d around station i (same range d for FZ filter?)
#N = number of stations not-NaN within range d around station i
    #filter cannot be applied if;
    HIflag[np.where(N < nstat)] = -1

    #accepting observation if;
    condition1 = (Med < HIthresA) & (pws_data[:,i] > HIthresB)
    condition2 = (Med >= HIthresA) & (pws_data[:,i] > HIthresB /HIthresA * Med)
    HIflag[np.where(condition1|condition2)] = 1
return HI_array

@lepetersson Looks like a good starting point. I might be able to do it without the for-loop, but it is good to start with the for-loop, check that things work (ideally via a test), and then refactor to make it faster.

closed in #16


Allow low level security vulns and update main modules

As discussed on Slack, accepting low-level vulnerabilities as tech debt.


Password Generator

https://replit.com/@guyrichie/Password-Generator?s=app

A python command-line program to generate a password for the user, by asking them how many letters,numbers and symbols they want in their password.


Register *.the.is-cool.dev

Added *.the.is-cool.dev using the CLI.

CLA Assistant Lite bot: Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment same as the below format.

I have read the CLA Document and I hereby sign the CLA

You can retrigger this bot by commenting recheck in this Pull Request

recheck


Purge expired transactions when loading mempool cache

A solid change, only some tiny comments.

Thanks @mikedennis


Update the dependabot config to v2

PR Summary
Now, dependabot-preview will stop, and dependabot will start issuing updates, since its now built into GitHub directly.
PR Context
Dependabot has been merged into GitHub
PR Checklist

Not really code change

@RDIL Thank you for your contribution!

@RDIL We still appear to be using dependabot-preview, rather than the built-in dependabot, for example see #14310.


docs: fix some typos and grammar, add fork link

Fixes some typos and grammar issues. Adds a fork this repo link

Thanks!


fix optional nvim-web-devicons setup doc

Checklist

[x] I read the CONTRIBUTING GUIDE
[ ] I opened a CONTRIBUTING ISSUE before opening the PR

Description
Sample nvim-web-devicons setup doc removes the default icon.
Screenshots

I accidentally deleted my fork 🥴. New PR: https://github.com/slugbyte/lackluster.nvim/pull/79.


feat: allow to set the passphrase programmatically when importing an …

…account
What I did
I added the capability to use import_account() without prompting the user. That is useful when automatic deployment via a CI, or when we do not want to bother with prompt at all.
How I did it
I added a kyword-argument to pass the passphrase.
How to verify it
Checklist

[x] Passes all linting checks (pre-commit and CI jobs)
[x] New test cases have been added and are passing
[ ] Documentation has been updated
[x] PR title follows Conventional Commit standard (will be automatically included in the changelog)

cc @unparalleled-js

Hmm I'm not sure to understand why test_import() fails. Changes are transparent for normal behavior :thinking:

Note: the failing test is not due to my changes.

:champagne:


Support for VsCode 1.64

They have made changes to the storage.json file hence the python code isn't working.

https://github.com/plibither8/ulauncher-vscode-recent/issues/12


Could you please show us more details about softether api?

I plan to build a anti-anti-spider proof project. I wish I can use this api to change my ip address conveniently? So what should I do?  Thanks for your time !

@JIACHENG135 sorry for late response, somehow I missed the issue notification. How do you mean you want to use it to change your IP address? Can you please describe more your use case?

Hello @vandot,
Just like @JIACHENG135, it will be nice to know how to setup a VPN Server or Bridge right from installation using the api.
For example I have a local bridge, and from start it has no admin password. So it will be nice to know how to use the api please. Or in such a scenario, does one first have to manually set the password before making use of the api?
Kind regards


add Autonomys network

Added Autonomys network (https://www.autonomys.xyz/) and bumped minor version to "1.51.0"

Hi @bkchr @SBalaguer
I understand that Parity is looking to revamp the registry, however, we are looking to launch Autonomys mainnet within a few weeks (blog) so would greatly appreciate a review and release of this crate.
Thank you for understanding!

@dariolina all looks good to me, however I do have a question: Isn't Autonomys a rebranding of Subpsace, and so you wouldn't need a new SS58 but rather a modification of the previous one? Or are you keeping both blockchains running simultaneously?
Also as you said, there's a discussion going on in the ecosystem regarding ss58, so this registry might be shut down in the future.

@SBalaguer thanks for a speedy response!
Indeed, it is a rebranding of Subspace, but adding a new entry kind of felt less invasive. Happy to modify.

I see! Then if it's a rebranding and you're using the same chain as you have today, you can go ahead and modify the existing entry. If subspace won't exist as it used to anymore, this is probably the best.

Thanks for the review @bkchr ! Is there anything I need to do to get this merged and released to crates?

@dariolina please merge master to make this pr mergable.


nix's Cargo.toml refers to the git head version of libc

The 0.8.0 release of nix refers to the git head version of libc, rather than a version (eg "0.2.x"). I my builds, this makes nix's reexport of libc incompatible with other creates I'm using...

We failed to update that line, when making the release. I guess a simple 0.8.1 release with the correct libc version, should fix this.

Ah this was my fault as I made the release. I actually got it right in the tagged version; I must have failed at the publish step. I'll take a look tomorrow or Monday if nobody else has by then. Sorry about this!

That's OK. I did a fork and tried out a change to a specific libc version, and it seems to fix the issue I was experiencing.

@raphaelcohn I don't actually see this in the package on crates.io. Can you verify in a new project, and check that you don't have any overrides in .cargo/config?

Which package on crates.io? I'm a bit befuddled.

@raphaelcohn https://crates.io/crates/nix
Please do let us know if there's any issue with the latest released version.

This shouldn't actually be possible, to release a crate that doesn't have a version declared for all dependencies. Git versions and local paths aren't allowed for released crates. @raphaelcohn Are you using the current git or are you pulling from crates.io.

In that case, I'm closing this. Please reopen if you see something though!


Bug: NAT_HARVESTER_LOCAL_ADDRESS listener is always IPv6

My setup:

jitsi-videobridge behind a NAT
network interface ens3 with IP <IP_ADDRESS>
IPv6 is enabled system-wide, but disabled for the network interface ens3
sip-communicator.properties contains: org.ice4j.ice.harvest.NAT_HARVESTER_LOCAL_ADDRESS=<IP_ADDRESS>

Expected result:

jitsi-videobridge listens to <IP_ADDRESS>::10000 (IPv4)

Actual result:

jitsi-videobridge listens to [::ffff:<IP_ADDRESS>]:10000 (IPv6), making it unusable for me

# ss -lpn | grep 10000
udp UNCONN 0 0 [::ffff:<IP_ADDRESS>]:10000 *:* users:(("java",pid=9368,fd=134))

Can you post the config and the startup logs?


[FEAT] :Add Copy Button

Description

Add one copy button to copy the shorted link

Screenshots

GSSoC24
@GSSoC24
Code of Conduct
I follow CONTRIBUTING GUIDELINE of this project.

I @Harshit7492 like to add this feature if @abhijeetnishal allow me to add this feature

DUPLICATE #61

Closing this issue as duplicate.


Use inline asm! for x86 DIV

bors try

While this does improve benchmarks here and in my own programs, one downside is that this asm! is opaque to LLVM, so it doesn't get a chance to optimize division by a constant (scalar) value into "magic" multiplication.

On further reflection, the measured gains should outweigh theoretical optimization losses.
Maybe someday we could try is_val_statically_known to detect divisors that might be optimizable.


fix(html/minifier): fix comments removing

Description:
There are two things:

A lot of tests for tests (and avoid future regressions)
Fix removing comments document and document fragment

BREAKING CHANGE:
No
Related issue (if exists):
No

@kdy1 Fixed :+1:

@kdy1 Rebased :+1:


modelWithIndexValue:  can not work well when JSONModelArray invoke it directly

there is a ConvertOnDemand property in a model。so this property is a JSONModelArray，when it  invoke modelWithIndexValue: ,the _storage[0] is a dictionary, so it can not respond indexPropertyName。
if (![_storage[0] indexPropertyName]) return nil;

[__NSDictionaryI indexPropertyName]: unrecognized selector sent to instance 0x7f8569d63ea0

I'm pretty sure this won't happen any more due to changes which have happened since. Please feel free to make a new issue if the problem still persists in the latest version.


Add open-cluster-management-policies namespace to the GRC watch namespaces

Add open-cluster-management-policies namespace to the GRC watch namespaces

Please also check the Sonarcloud issue and either accept it or push code to resolve it (the link is in the Sonarcloud post):

Shouldn't this PR go upstream or am I missing something there?

Watch_namespace is not in https://github.com/open-cluster-management-io/config-policy-controller/blob/main/main.go so I am working on downstream

Please also check the Sonarcloud issue and either accept it or push code to resolve it (the link is in the Sonarcloud post):

Thanks by the way, Why is this green?

Shouldn't this PR go upstream or am I missing something there?

Watch_namespace is not in https://github.com/open-cluster-management-io/config-policy-controller/blob/main/main.go so I am working on downstream

No watch namespace? Isn't that what this is or am I missing something about this change?
https://github.com/open-cluster-management-io/config-policy-controller/blob/b2935edcc2a7071532d0b668aaa1cd03c12fa260/main.go#L230

Please also check the Sonarcloud issue and either accept it or push code to resolve it (the link is in the Sonarcloud post)
Thanks by the way, Why is this green?

The Sonarcloud issue is green because it's not a blocking issue. But there will be less tech debt in the future if we resolve these now and not let them build up (if we ever decided to address the issues).

Shouldn't this PR go upstream or am I missing something there?

Watch_namespace is not in https://github.com/open-cluster-management-io/config-policy-controller/blob/main/main.go so I am working on downstream

No watch namespace? Isn't that what this is or am I missing something about this change? https://github.com/open-cluster-management-io/config-policy-controller/blob/b2935edcc2a7071532d0b668aaa1cd03c12fa260/main.go#L230

Please also check the Sonarcloud issue and either accept it or push code to resolve it (the link is in the Sonarcloud post)
Thanks by the way, Why is this green?

The Sonarcloud issue is green because it's not a blocking issue. But there will be less tech debt in the future if we resolve these now and not let them build up (if we ever decided to address the issues).

Thanks! let me close this PR

@Dale

Shouldn't this PR go upstream or am I missing something there?

Watch_namespace is not in https://github.com/open-cluster-management-io/config-policy-controller/blob/main/main.go so I am working on downstream

No watch namespace? Isn't that what this is or am I missing something about this change? https://github.com/open-cluster-management-io/config-policy-controller/blob/b2935edcc2a7071532d0b668aaa1cd03c12fa260/main.go#L230

Please also check the Sonarcloud issue and either accept it or push code to resolve it (the link is in the Sonarcloud post)
Thanks by the way, Why is this green?

The Sonarcloud issue is green because it's not a blocking issue. But there will be less tech debt in the future if we resolve these now and not let them build up (if we ever decided to address the issues).

Also We are  adding specific namespace so I thought this Pr only downstream applied. Should we apply this namespace to Upstream too?

Should we apply this namespace to Upstream too?
Yes, I think so. Upstream uses the same open-cluster-management-* namespaces.


Should we use the provided ClientID?

Hi there,
I'm having issues with the API working with anything besides the demo ClientID (consumer key). When I replace it with my consumer key the API fails to connect. Is the intention to use the project's ClientID as it is registered with TDA? Or should I keep troubleshooting why my Client ID fails?
I successfully use my Client ID for other purposes so I know it's working. Thanks for your help and the great library!

ClientID (consumer key) is just an example. You need to use the one generated in the My Apps located here:
https://developer.tdameritrade.com/user/me/apps


Update Validation

Would be cool if the tool could also take the old objects as input and validate Updates which would allow CEL transition rules to be evaluated.

@alexzielenski can you please elaborate it please ?


Feature/#20 multiprocessing support

Fixes #20
I added async multiprocessing, some questions to you, @timroepcke :

The no of threads is defined in the config as intended by you. I'd propose to have it as arg in the CLI instead, what do you think? (this is not integrated yet)
You had a DF charging_all for the result aggregation. Due to the asynchronous operation I had to comment this out. Is this still needed? If so, we have to implement this differently.

Please review #19 first as this PR already includes all features of #19.

I've tested this for a while now and it works well. May I merge, @timroepcke @Mosc5 ?


Landing page should direct to providers somehow

Users who reach the landing page should be able to sign up under their provider (in case they forgot their providers url) or be able to see the providers index

Closed with immediate selection of user_type


[CircleGraph/test] Initial test for CircleGraphPanel

This will introduce initial unit test for CircleGraphPanel class.
ONE-vscode-DCO-1.0-Signed-off-by: SaeHie Park<EMAIL_ADDRESS>

from #966

Thanks for creating tests for CircleGraph ! :D
But I have a question, I cannot find any assert in the CircleGraph.test.ts.
Was it intentional?

I cannot find any assert in the CircleGraph.test.ts.
Was it intentional?

Mostly there is no return value to test. some exist with condition.
Methods to test are mostly simple logic.
If no assert() is a problem from SE feedback (1) add dummy assert(true) (2) refactor later to get some return values; I'm not sure how this is possible for now.

If no assert() is a problem from SE feedback

Unfortunately, when I checked this with static analysis tool, it reported that there is no assertion in test code :(
But refactoring may bother you and may take some time.
So how about just adding following simple assertions for now? :)
  suite('#createOrShow', function() {
    test('createOrShow model with no error ', function() {
      // NOTE createOrShow(extensionUri, undefied) will show file open dialog
      // which test doen't execute as expected
      const panel = CircleGraphPanel.createOrShow(extensionUri, modelToLoad);
      assert.notStrictEqual(panel, undefined);
      panel?.dispose();
      const panel2 = CircleGraphPanel.createOrShow(extensionUri, modelToLoad);
      assert.notStrictEqual(panel2, undefined);
    });
  });
  suite('#isReady', function() {
    test('isReady check', function() {
      const panel = CircleGraphPanel.createOrShow(extensionUri, modelToLoad);
      assert.notStrictEqual(panel, undefined);
      assert.strictEqual(panel?.isReady(), false);
      panel?.dispose();
    });
  });
  ...
  suite('#setPartition', function() {
    test('setPartition check', function() {
      const panel = CircleGraphPanel.createOrShow(extensionUri, modelToLoad);
      assert.notStrictEqual(panel, undefined);
      panel?.setPartition({a: 'B'});
      panel?.dispose();
      assert.ok(true);
    });
  });


gene annot updates

the purpose of this module is to handle count table that are not produced by us and provides an alternative entry point.
What this function does, in the old version, is it see if any element in the given list [“chr”,”start”,”end”] are present in the column names of a df. If so, then drop all three columns.
This implementation is problematic, for it will drop columns that are potentially not presented in the df.
So in my new implementation, I drop the column based on intersection of the df actual column and the provided list.
(edited)
3:56
This is the reason pertaining to changes 1
3:59
Given the updated mechanism in changes 1, this function now remove any columns that we don’t like based on a given list, so I extend that list to include many names I remembered seeing that could means chr start and end. Such that this module can truly do what it design to do, take in some random bed.gz or count table and then re annotate them

Thanks @hsun3163 but there's a  merge conflict. Could you please resolve it?


When using Python 3.11.4 to invoke the Gemini Pro Vision model and upload a plt image, a type error is reported.

Description of the bug:
! curl -o image.jpg https://t0.gstatic.com/licensed-image?q=tbn:ANd9GcQ_Kevbk21QBRy-PgB4kQpS79brbmmEG7m3VOTShAn4PecDU5H5UxrJxE3Dw1JiaG17V88QIol19-3TM2wCHw
img = PIL.Image.open('image.jpg')
model = genai.GenerativeModel('gemini-pro-vision')
response = model.generate_content(img)

TypeError                                 Traceback (most recent call last)
Cell In[56], line 1
----> 1 response = model.generate_content(img)
File ~/.local/lib/python3.11/site-packages/google/generativeai/generative_models.py:229, in GenerativeModel.generate_content(self, contents, generation_config, safety_settings, stream, **kwargs)
219 @string_utils.set_doc(_GENERATE_CONTENT_DOC)
220 def generate_content(
221     self,
(...)
227     **kwargs,
228 ) -> generation_types.GenerateContentResponse:
--> 229     request = self._prepare_request(
230         contents=contents,
231         generation_config=generation_config,
232         safety_settings=safety_settings,
233         **kwargs,
234     )
235     if self._client is None:
236         self._client = client.get_default_generative_client()
File ~/.local/lib/python3.11/site-packages/google/generativeai/generative_models.py:200, in GenerativeModel._prepare_request(self, contents, generation_config, safety_settings, **kwargs)
197 if not contents:
198     raise TypeError("contents must not be empty")
--> 200 contents = content_types.to_contents(contents)
202 generation_config = generation_types.to_generation_config_dict(generation_config)
203 merged_gc = self._generation_config.copy()
File ~/.local/lib/python3.11/site-packages/google/generativeai/types/content_types.py:235, in to_contents(contents)
230     except TypeError:
231         # If you get a TypeError here it's probably because that was a list
232         # of parts, not a list of contents, so fall back to to_content.
233         pass
--> 235 contents = [to_content(contents)]
236 return contents
File ~/.local/lib/python3.11/site-packages/google/generativeai/types/content_types.py:201, in to_content(content)
198     return glm.Content(parts=[to_part(part) for part in content])
199 else:
200     # Maybe this is a Part?
--> 201     return glm.Content(parts=[to_part(content)])
File ~/.local/lib/python3.11/site-packages/google/generativeai/types/content_types.py:171, in to_part(part)
168     return glm.Part(text=part)
169 else:
170     # Maybe it can be turned into a blob?
--> 171     return glm.Part(inline_data=to_blob(part))
File ~/.local/lib/python3.11/site-packages/google/generativeai/types/content_types.py:140, in to_blob(blob)
136 if isinstance(blob, Mapping):
137     raise KeyError(
138         "Could not recognize the intended type of the dict\n" "A content should have "
139     )
--> 140 raise TypeError(
141     "Could not create Blob, expected Blob, dict or an Image type"
142     "(PIL.Image.Image or IPython.display.Image).\n"
143     f"Got a: {type(blob)}\n"
144     f"Value: {blob}"
145 )
TypeError: Could not create Blob, expected Blob, dict or an Image type(PIL.Image.Image or IPython.display.Image).
Got a: <class 'PIL.JpegImagePlugin.JpegImageFile'>
Value: <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=2048x1365 at 0x7F89DDFC8B10>
Actual vs expected behavior:
No response
Any other information you'd like to share?
python 3.11.4
centos 7.9
google-ai-generativelanguage 0.4.0
google-api-core              2.15.0
google-auth                  2.25.2
google-generativeai          0.3.1
googleapis-common-protos     1.62.0
langchain-google-genai       0.0.4

Try using IPython.display instead of PIL for Image.  Examples:
from IPython.display import Image
from IPython.core.display import HTML
Image(url= "http://my_site.com/my_picture.jpg")
or
from IPython.display import Image
from IPython.core.display import HTML
img = Image('image_test.jpg')

Thanks @jhowardsemail It worked for me!

I'm unable to reproduce this error now. I think it's fixed. If not please reopen and send code to reproduce the problem.
Thanks.

@MarkDaoust I am getting similar error but for counting tokens:
result = model.generate_content(prompt)
token_count = model.count_tokens(code).total_tokens

This is the error message I get
Could not create `Blob`, expected `Blob`, `dict` or an `Image` type(`PIL.Image.Image` or `IPython.display.Image`).
Got a: <class 'int'>
Value: 35

(I'm using tuned model if that matters)

@MadCoderme Did you find the solution?

We did have an issue where a lot of these were popping up in conda. Are either of you using conda?
@MadCoderme  "Got a: <class 'int'>" It's trying to tell you that it doesn't know how to turn an int into a prompt. Why are you passing it an int?

importing image from Ipython helps but you want to use pillow then please upload_file to Gemini it gives 20 gb of space.

Also having this problem. Using conda, but not sure what conda has to do with it.

@petervanwylen can you raise a new issue with details?

Also having this same issue while trying generate content from GCS URI


Update ember-cli-node-assets and use correct .import

This PR fixes nested addon import behavior. Starting from ember-cli 2.7, there is this.import that does the correct root app detection, so we can check if it exists, before 2.7 there is a polyfill, which is findHost in this PR.
This is the way, how nesting fails:
App <-- at this level, include of the addon wasn't happening
  -- Included addon
      -- has dependency on ember-cli-sentry

I've kinda backported changes from ember-cli-node-assets:
https://github.com/dfreeman/ember-cli-node-assets/blob/master/index.js

Hey @damiencaselli, any chances that you can look at this PR, please? :)

@damiencaselli @marcoow guys, any chances this gets merged? :) Thank you!

Don't we have Travis integration for this, @damiencaselli?

@marcoow seems like the last build was two years ago - https://travis-ci.org/damiencaselli/ember-cli-sentry/branches
Do you think its blocking from merge & release? I am currently thinking if I should deploy to prod my fork, or we can release original repo and use a nice version instead? :)

@marcoow nope, I don't recall setting up any kind of linting in travis 🙇

There actually seems to be no travis integration at all…

@damiencaselli: there's something wrong though: https://travis-ci.org/damiencaselli/ember-cli-sentry/jobs/289074566

@marcoow maybe we can merge/deploy it, there are no tests for this yet anyway? In following PRs we can add tests.

@marcoow @damiencaselli guys, sorry to be annoying so much, but can we somehow merge it? :) I'd be much more happy to have a solid version in production, than a forked branch :) If you need something from me, please tell, will be happy to collaborate.

Sorry, hadn't see @damiencaselli had fixed the Travis build. @RuslanZavacky: can you rebase this on master so that the Travis build is run?

@marcoow done & green ;)

🎉

@marcoow may I also ask a release of a new version, please? :)

@damiencaselli @Turbo87: ok to release this as 3.0.0-beta.2?

@marcoow yes, seems good


devops: check fiscalcloud backward compatibility on each commit

on each commit to bp3: run /apps/fiscalcloud/compat-check.sh: exit status !=0 -> log error, fail the build

Done


DELRTE command is broken

The DELRTE command is currently not functioning. It gives the following error:

This can be fixed by changing line 914 in bluesky/traffic/route.py from:
self.__init__()
to
self.__init__(bs.traf.id[iac])
or simply
self.__init__(self.acid)

Suggested fix has been made in route.py and tested.
Thanks!


Replace Data in RESPValue with ByteBuffer

Right now the RESPValue.bulkString stores a reference to Data, but it might be possible / easy to replace it with a ByteBuffer

It was decided to work with [UInt8] as the basic block for byte arrays rather than the entire ByteBuffer struct


Make ERC20 compliant with ERC-165

🎉 Description
@shrugs Has done a nice job making ERC721 compliant w/ ERC-165: https://github.com/OpenZeppelin/openzeppelin-solidity/commit/259b9da3e6081c3c14f449cad83004d64c0b9cb7
It would seem to make sense to do the same for ERC20, right? From the EIP:
"For some "standard interfaces" like the ERC-20 token interface, it is sometimes useful to query whether a contract supports the interface and if yes, which version of the interface, in order to adapt the way in which the contract is to be interacted with. Specifically for ERC-20, a version identifier has already been proposed. This proposal standardizes the concept of interfaces and standardizes the identification (naming) of interfaces."
https://github.com/ethereum/EIPs/blob/master/EIPS/eip-165.md

[x] 📈 This is a feature request.

After some consideration we decided not to implement this. We haven't received any real requests for it, and "it would seem to make sense" does not justify the added cost and complexity of adding ERC165 to ERC20.
Thanks for the suggestion regardless @dougiebuckets!

It case anybody is still interested, the 0xcert Framework implementation of ERC-20 does support ERC-165. That is here https://github.com/0xcert/framework/tree/master/packages/0xcert-ethereum-erc20-contracts
And yes that effort was inspired by this issue opened here.


I would like to join the organization

Please send me an invitation as a member of Eddie Jaounde community

Thank you @frankiefab100  👍
I will transfer this issue to the support repo and with the relevant labels.


Update views

Summary
The goal of this PR is to make the UI components that dive uses re-usable and extendable. This is a requirement to eventually allow users the ability to plugin their own functionality.
It introduces a few major changes to do this:

migrate off of gocui to a tcell based alternative tview, this project gives us some easily extendable types and is actively maintained.
refactor front end to have a clean division between views and the data they consume.

TODO
That said there are still quite a few features that need to be added before this can be considered equivalent to the previous implementation, namely:
Visual changes needed:

Pull color settings from terminal / allow configuration
Could probably use a bit more beautification on the front end.
Key mapping view
Hide filter view when empty/unfocused
Limited space views

Functionality features needed:

missing some change based filtering (▏^R Removed ▏^M Modified ▏^U Unmodified ▏^B Attributes )
Few filetree commands missing (collapse all)
CompareAllLayers view
Factor out the tview.Grid used to arrange view components.

Would love some feedback on the direction of this PR. And will be adding some more documentation to the code to help with readability.

Hey, @wagoodman when you get the chance, I would love some feedback on the direction of the PR.
( PS, the final commit 38ccd19 is just showing how this could easily be extended. )

@dwillist this looks like a great! I took a look over the weekend and started to get a feel for the changes and approaches. I have some comments that are probably best paired with some commits, mind if I push to this branch?

@dwillist I got some time this morning and pushed a commit on what I was working on, specifically to augment the primitives to have a wrapper for consistent styling. I haven't accounted for the layerDetails view yet (and a few more things), but I wanted to start augmenting the foundation to allow for a consistent look-and feel (based on what's there today).
Next I'll take a look at the CNB specifics and the abstractions there.

@wagoodman awesome! Really like the wrapper primitive you defined that was a great idea.
I've just added a couple more components to get some "hiding" behavior to help de-clutter views when using a smaller window. Think I'll reimplement the coloring from the terminal next and really I need to add some comments to these primitives & models to make this a bit more readable.

@wagoodman hey wondering if you're still interested in this PR or have gotten a chance to look at it

@dwillist ... just getting back to this since the new years break --this is my first week back from paternity leave, it's been a blast! Thanks for your work on this --this is awesome. I'll be digging into the code and details today and adding comments and pushing tweaks.

@dwillist I'm currently working on changes to your branch that would firm up how logging and application configuration is initialized and handled within dive. I think this would help the foundation some and what is needed for ui.UI initialization --shout out if you're working in these areas and we'll coordinate.

@dwillist  I updated the app config logic and logging facilities (there was apex/log, logrus, and zap in use). I saw that you had some work that was still in progress and not pushed yet in app.go and key_config.go (grep for partially pushed), I commented it out in the meantime to get the branch running.
I'm currently troubleshooting a UI state bug where user options are not persisted when the layer selection is changed.

Hey :tada:  Congrats!! :tada:. Glad to have you back though. Apologies about that have baked commit. I had some changes in flight but moving them over should be ez pz and shouldn't interfere. Also keybinding and log changes look loads better than what I hacked together :+1:

@dwillist thanks! No worries about the commit 👍  (I'm dipping my toes on your branch, so it's me getting in the way :) ). I've got some extra time today, I'm going to take a stab at continuing the keybinding config (great find on cbind btw). Next on the list for me is to take a closer look at the models interface approach.

Wow, lots of conflicts here. Seems useful.


Make Travis cache the .pyembedpg dir

With this commit we make Travis cache the .pyembedpg dir and reuse it from one build to the other. With this the build time goes from 5 minutes to 1 minute.
For example, compare https://travis-ci.org/elemoine/pg_li3ds/builds/211834412 (cold cache) to https://travis-ci.org/elemoine/pg_li3ds/builds/211837078 (hot cache).
closes #12

Nice, thanks for that !


update desk layout for 1809

This updates the layout for the seating chart to the new 1809 mega layout.

I ran npm audit fix, which is why this updated some npm packages.


Update free-programming-book-ko.md

What does this PR do?
Add resource(s) | Remove resource(s) | Add info | Improve repo
For resources
Description
N/A
Why is this valuable (or not)?
Optimized for ocaml study.
How do we know it's really free?
an official document
For book lists, is it a book? For course lists, is it a course? etc.
an official document
Checklist:

[x] Read our contributing guidelines.
[x] Search for duplicates.
[x] Include author(s) and platform where appropriate.
[x] Put lists in alphabetical order, correct spacing.
[x] Add needed indications (PDF, access notes, under construction).
[x] Used an informative name for this pull request.

Follow-up

Check the status of GitHub Actions and resolve any reported warnings!

Not a Korean book


Crash Iris 1.6+ and Blur 3.0.0 crashes during world loading

Minecraft 1.19.4
Iris 1.6.1
Blur 3.0.0
Fabric 0.14.19
Fabric API 0.78.0
This is also probably related to the issue on https://github.com/IrisShaders/Iris/issues/1921
latest.log
crash-2023-04-17_16.59.04-client.txt

This actuslly seems to be caused by a mod utilizing owo-lib, Blur is not the cause, according to this crash log.

As this line indicates:
reason: java.util.concurrent.CompletionException: net.minecraft.class_151: Non [a-z0-9_.-] character in namespace of location: shaders/core/owo:spectrum.json

strange, I'll look into it more when I have more time

Ok so the same issue that I am having has been also been reported to owo-lib's issue tracker
This probably isn't because of iris
It's apparently because of the verison of Satin Blur uses https://github.com/wisp-forest/owo-lib/issues/120


Normal of collided face

We are currently using cannon.js along with three.js for a university project. The level has a lot of static objects that were blocked out by the level designer as well. This cuboid mesh is then used to create physics cubes representing them.
Everything works fine, but I have an issue getting the jumping to work properly. The jumping is basically a flag being set whenever the player hits the ground to allow the player to jump.
The problem: Whenever the player hits something it gets reseted, which means the player can jump up big objects by constantly jumping against the wall and keep jumping. Although it is a lot of fun it is not the intended behaviour.
There is a lot of information contained in the collision event, but I could get it to work properly.
The question: How do I calculate the normal of the face/plane/side what a object collided with?
Thanks for the brilliant work and your compound objects was something I was looking forward to a long time :)

How can I get this data in the beginContact event?


python3 support

After installing in a virtualenv, dsub won't run.
(dsub_libs) pear:~ davidgibbs$ dsub
Traceback (most recent call last):
  File "/Users/davidgibbs/dsub_libs/bin/dsub", line 7, in <module>
    from dsub.commands.dsub import main
  File "/Users/davidgibbs/dsub_libs/lib/python3.6/site-packages/dsub/__init__.py", line 15, in <module>
    from _dsub_version import DSUB_VERSION as __version__
ModuleNotFoundError: No module named '_dsub_version'

Seems related to this:
stackoverflow
all imports are relative:  from _version import version
instead of explicit:  from ._version import version

Thanks for filing the issue David. Python3 is something we would like to support.
For now, Python 2 is required and likely Python 2.7.
We should add a python_requires flag to setup.py until we get Python 3 support.

Note, we should update the getting started instructions to use indicate
virtualenv --python=python2.7 dsub_libs

Python3 has been supported for a while now, and we will fully sunset Python2 sometime soon. Closing out.


chore(toolchain): fix warnings

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#479  👈
#478
#477
#476
#475
#474
#473
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @NathanFlurry and the rest of your teammates on  Graphite


pkgsStatic: do not propagate builds with nix-support/propagated-build-inputs

Describe the bug
For static builds recursively all dependencies are needed. An adapter is now used that converts all buildInputs to propagatedBuildInputs. Using propagatedBuildInputs results in the file $out/nix-support/propagated-build-inputs causing a run-time dependency on all build-time dependencies.
With Python there is a similar kind of issue which is why there I am investigating a way that does not involve writing out that file.

Here is an Idea:
Instead of modifying nix-support, pkgsStatic can clone the same mechanism into a separate store path. I.e., the adapter adds an implicit output "nix-static-inputs" that contains the paths that are currently added into propagated-build-inputs. The adapter can then make use of that file to insert those dependencies back into the build environment.
I don't know if that would be viable for Python too.
What do you think?

Maybe. Imagine we go create Python packages with pkgsStatic. Both use propagatedBuildInputs, but for different purposes. How do you distinguish?
The whole file is not needed here in case of pkgsStatic, the stdenv recurses without it if I am correct. The file is used in case of e.g. interpreted languages but even there its a simplified solution.
Possible solutions

keep using propagation but put it in a separate output
remove the file with a hook
new attribute that propagateBuildInputs sets, that we recurse in, as is partially done for Python with requiredPythonModules.

1 and 2 will break other use case of propagatedBuildInputs, though likely they also shouldn't be using it.

cc @Ericson2314 @matthewbauer for input. Generally speaking, propagating inputs does not mean we want to have them typically written out in nix-support/propagated-build-inputs.

Ah, nix-support already ends up in dev. So it's just a matter of adding dev outputs everywhere.

Ah yes @Fridh beat me to it. The intension I always assumed was to have dev outputs everywhere, though I am not sure how well that works with all languages.

Yes that's my understanding as well, the dev outputs are always transitively propagated.  One thing I'm not entirely sure of though is whether a multiple-output package that has a setupHook still has all of its transitive setupHooks executed for a dev dependency, like it will for propagatedBuildInputs or direct buildInputs.

The intension I always assumed was to have dev outputs everywhere, though I am not sure how well that works with all languages.

Python libraries that don't have any executables will loose their store references until the library is used in an environment. Not a big issue but it can cause unnecessary GC and fetching. Note we're not using dev yet for Python packages.

Example of a change forcing a "dev" output for static builds https://github.com/NixOS/nixpkgs/pull/83793


New package: ITKIOWrapper_jll v1.0.0+0

Autogenerated JLL package registration

Registering JLL package ITKIOWrapper_jll.jl
Repository: https://github.com/JuliaBinaryWrappers/ITKIOWrapper_jll.jl
Version: v1.0.0+0
Commit: 709aef065dbf9687d98448049792def0bae24e21
Revision on Yggdrasil: https://github.com/JuliaPackaging/Yggdrasil/commit/bf3c6e95960a58692186437d212f13f1bd25ce02
Created by: @divital-coder

@divital-coder https://github.com/JuliaRegistries/General/actions/runs/9865629499/job/27242916193?pr=110495#step:17:479
libITKFastMarching-5.3.so.1: cannot open shared object file: No such file or directory

[noblock]

I have been testing and building all day, it's just I have to run binarybuild everytime, so just gonna make a pr when all are fixed 🙃

To cut this short:
% ldd libITKIOWrapper.so
        linux-vdso.so.1 (0x00007ffdb46b4000)
        libcxxwrap_julia.so.0 => not found
        libexpat.so.1 => /usr/lib/libexpat.so.1 (0x000075f681afc000)
        libz.so.1 => /usr/lib/libz.so.1 (0x000075f681ae3000)
        libjpeg.so.62 => not found
        libpng16.so.16 => /usr/lib/libpng16.so.16 (0x000075f681aa9000)
        libtiff.so.6 => /usr/lib/libtiff.so.6 (0x000075f681a1c000)
        libITKTestKernel-5.3.so.1 => not found
        libITKLabelMap-5.3.so.1 => not found
        libITKFastMarching-5.3.so.1 => not found
        libITKConvolution-5.3.so.1 => not found
        libITKPolynomials-5.3.so.1 => not found
        libITKBiasCorrection-5.3.so.1 => not found
        libITKColormap-5.3.so.1 => not found
        libITKDICOMParser-5.3.so.1 => not found
        libITKDeformableMesh-5.3.so.1 => not found
        libITKDenoising-5.3.so.1 => not found
        libITKDiffusionTensorImage-5.3.so.1 => not found
        libITKPDEDeformableRegistration-5.3.so.1 => not found
        libhdf5_cpp.so.310 => /usr/lib/libhdf5_cpp.so.310 (0x000075f6819a1000)
        libhdf5.so.310 => not found
        libhdf5_hl.so.310 => /usr/lib/libhdf5_hl.so.310 (0x000075f68197a000)
        libITKIOBioRad-5.3.so.1 => not found
        libITKIOBruker-5.3.so.1 => not found
        libITKIOCSV-5.3.so.1 => not found
        libITKIOGE-5.3.so.1 => not found
        libITKIOSiemens-5.3.so.1 => not found
        libITKIOHDF5-5.3.so.1 => not found
        libITKIOJPEG2000-5.3.so.1 => not found
        libITKIOLSM-5.3.so.1 => not found
        libITKIOMINC-5.3.so.1 => not found
        libITKIOMRC-5.3.so.1 => not found
        libITKIOSpatialObjects-5.3.so.1 => not found
        libITKIOStimulate-5.3.so.1 => not found
        libITKIOTransformHDF5-5.3.so.1 => not found
        libITKIOTransformInsightLegacy-5.3.so.1 => not found
        libITKIOTransformMatlab-5.3.so.1 => not found
        libITKKLMRegionGrowing-5.3.so.1 => not found
        libITKMarkovRandomFieldsClassifiers-5.3.so.1 => not found
        libITKQuadEdgeMeshFiltering-5.3.so.1 => not found
        libITKRegionGrowing-5.3.so.1 => not found
        libITKVTK-5.3.so.1 => not found
        libITKWatersheds-5.3.so.1 => not found
        libITKRegistrationMethodsv4-5.3.so.1 => not found
        libITKVideoIO-5.3.so.1 => not found
        libjulia.so.1.10 => not found
        libITKIOBMP-5.3.so.1 => not found
        libITKIOGDCM-5.3.so.1 => not found
        libitkgdcmMSFF-5.3.so.1 => not found
        libitkgdcmDICT-5.3.so.1 => not found
        libitkgdcmIOD-5.3.so.1 => not found
        libitkgdcmDSED-5.3.so.1 => not found
        libitkgdcmCommon-5.3.so.1 => not found
        libITKIOGIPL-5.3.so.1 => not found
        libITKIOJPEG-5.3.so.1 => not found
        libITKIOMeshBYU-5.3.so.1 => not found
        libITKIOMeshFreeSurfer-5.3.so.1 => not found
        libITKIOMeshGifti-5.3.so.1 => not found
        libITKgiftiio-5.3.so.1 => not found
        libITKIOMeshOBJ-5.3.so.1 => not found
        libITKIOMeshOFF-5.3.so.1 => not found
        libITKIOMeshVTK-5.3.so.1 => not found
        libITKIOMeshBase-5.3.so.1 => not found
        libITKQuadEdgeMesh-5.3.so.1 => not found
        libITKIOMeta-5.3.so.1 => not found
        libITKMetaIO-5.3.so.1 => not found
        libITKIONIFTI-5.3.so.1 => not found
        libITKniftiio-5.3.so.1 => not found
        libITKznz-5.3.so.1 => not found
        libm.so.6 => /usr/lib/libm.so.6 (0x000075f68187d000)
        libITKIONRRD-5.3.so.1 => not found
        libITKNrrdIO-5.3.so.1 => not found
        libITKIOPNG-5.3.so.1 => not found
        libITKIOVTK-5.3.so.1 => not found
        libITKFFT-5.3.so.1 => not found
        libITKIOIPL-5.3.so.1 => not found
        libitkopenjpeg-5.3.so.1 => not found
        libITKIOTIFF-5.3.so.1 => not found
        libitkminc2-5.3.so.1 => not found
        librt.so.1 => /usr/lib/librt.so.1 (0x000075f681874000)
        libITKIOXML-5.3.so.1 => not found
        libITKIOTransformBase-5.3.so.1 => not found
        libITKTransformFactory-5.3.so.1 => not found
        libITKMathematicalMorphology-5.3.so.1 => not found
        libITKPath-5.3.so.1 => not found
        libITKImageFeature-5.3.so.1 => not found
        libITKSpatialObjects-5.3.so.1 => not found
        libITKMesh-5.3.so.1 => not found
        libITKTransform-5.3.so.1 => not found
        libITKSmoothing-5.3.so.1 => not found
        libITKOptimizersv4-5.3.so.1 => not found
        libITKOptimizers-5.3.so.1 => not found
        libITKStatistics-5.3.so.1 => not found
        libitkNetlibSlatec-5.3.so.1 => not found
        libitklbfgs-5.3.so.1 => not found
        libITKIOImageBase-5.3.so.1 => not found
        libITKVideoCore-5.3.so.1 => not found
        libITKCommon-5.3.so.1 => not found
        libitkdouble-conversion-5.3.so.1 => not found
        libitksys-5.3.so.1 => not found
        libITKVNLInstantiation-5.3.so.1 => not found
        libitkvnl_algo-5.3.so.1 => not found
        libitkvnl-5.3.so.1 => not found
        libitkv3p_netlib-5.3.so.1 => not found
        libitkvcl-5.3.so.1 => not found
        libpthread.so.0 => /usr/lib/libpthread.so.0 (0x000075f681865000)
        libdl.so.2 => /usr/lib/libdl.so.2 (0x000075f68185e000)
        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x000075f681400000)
        libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x000075f681831000)
        libc.so.6 => /usr/lib/libc.so.6 (0x000075f681214000)
        libzstd.so.1 => /usr/lib/libzstd.so.1 (0x000075f681752000)
        liblzma.so.5 => /usr/lib/liblzma.so.5 (0x000075f68171f000)
        libjbig.so.2.1 => /usr/lib/libjbig.so.2.1 (0x000075f68170f000)
        libjpeg.so.8 => /usr/lib/libjpeg.so.8 (0x000075f681178000)
        libhdf5.so.311 => /usr/lib/libhdf5.so.311 (0x000075f680c00000)
        libsz.so.2 => /usr/lib/libsz.so.2 (0x000075f681702000)
        /usr/lib64/ld-linux-x86-64.so.2 (0x000075f681b9c000)

Make sure ALL the libITK libraries are listed correctly as library products where appropriate.


Add SQL Server version this method applies to

For SQL Server functions usage, it would be great if it is given that what is the minimum version of SQL Server that is required for this function to compile. Like in which version of SQL Server was this function introduced.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 277a5d00-6fcf-a830-0451-f0667dd806e8
Version Independent ID: ac998a63-0675-de0c-c528-4b355c1d9d5f
Content: EOMONTH (Transact-SQL) - SQL Server
Content Source: docs/t-sql/functions/eomonth-transact-sql.md
Product: sql
Technology: t-sql
GitHub Login: @MikeRayMSFT
Microsoft Alias: mikeray

@k7shanmugam -- K7, thank you for your feedback.  Please add your suggestion here:

SQL Server Feedback


#3: dif/2, findall/3: unexpected overflow

#3

True.
Processing a finite list of 5994 elements(which are lists with 3 elements).
At the moment I get:
findall/3 Time elapsed 12s
bagof/3  error(resource_error(finite_memory),top_level/0).
But I guess this is an interim stage of development.

"The resource finite_memory is here to indicate that any finite memory would not be enough"
Commenting 'any': This is not true. A creative mathematician discovers (and solves) some of these problems.

Any finite memory certainly means any finite memory of this processor.


chore: run release on all pushes to master

Coverage remained the same at 100.0% when pulling 07daedd8c90fff64690d57bb7fb4c28d43cdd960 on open-up-release into bd19b5f6bbc1a575bb75132c10fbcd11e7b957bf on main.

:tada: This PR is included in version 2.0.0 :tada:
The release is available on:

GitHub release
npm package (@latest dist-tag)

Your semantic-release bot :package::rocket:


Create IPsSERPRO

Este KQL consulta a blacklist de IPs divulgada pelo SERPRO (Brasil) e informa que dispositivo da sua rede conectou-se com algum dos IPs listados.
Requisitos: Logs do Defender sendo encaminhados para o Sentinel DeviceNetworkEvents
Data Connector: Microsoft 365 Defender
Tabela: DeviceNetworkEvents
Required items, please complete
Change(s):

See guidance below

Reason for Change(s):

See guidance below

Version Updated:

Required only for Detections/Analytic Rule templates
See guidance below

Testing Completed:

See guidance below

Checked that the validations are passing and have addressed any issues that are present:

See guidance below

Guidance <- remove section before submitting

Before submitting this PR please ensure that you have read the following sections and filled out the changes, reason for change and testing complete sections:
Thank you for your contribution to the Microsoft Sentinel Github repo.

Details of the code changes in your submitted PR.  Providing descriptions for pull requests ensures there is context to changes being made and greatly enhances the code review process.  Providing associated Issues that this resolves also easily connects the reason.

Change(s):

Updated syntax for XYZ.yaml

Reason for Change(s):

New schema used for XYZ.yaml
Resolves ISSUE #1234

Version updated:

Yes
Detections/Analytic Rule templates are required to have the version updated

The code should have been tested in a Microsoft Sentinel environment that does not have any custom parsers, functions or tables, so that you validate no incorrect syntax and execution functions properly.  If your submission requires a custom parser or function, it must be submitted with the PR.

Testing Completed:

Yes/No/Need Help

Note: If updating a detection, you must update the version field.

Before the submission has been made, please look at running the KQL and Yaml Validation Checks locally.
https://github.com/Azure/Azure-Sentinel#run-kql-validation-locally

Checked that the validations are passing and have addressed any issues that are present:

Yes/No/Need Help

Note: Let us know if you have tried fixing the validation error and need help.

References:

Guidance for Detection checks
General contribution guidance
PR validation troubleshooting

Hi, we are looking into this, thanks!

We are working on this, will update shortly, thanks!

Hi @UKNT-BR we are waiting for updates from the concerned team, thanks!

Hi @UKNT-BR we are looking into this, will provide you update shortly, thanks!

Hi @UKNT-BR, we are looking into this, will provide you update shortly, thanks!

Hello @UKNT-BR, hope you are doing well. Just wanted to check if you got a chance to look at the suggestions shared. Please feel free to reach out to us for any queries or support. Thanks!

Hi @UKNT-BR,
Please check above comment and act accordingly.
Thanks

Hi @UKNT-BR, we are waiting for your response. Thanks!

Good morning guys,
I am grateful for the suggestions and agree with them. However, I am unable to implement them or make the necessary adjustments at this time.
If you understand that the code can be interesting, and want to use it, feel free.
Thanks.

Thanks UKNT-BR for the valuable contribution. Your PR's data source will be taken into consideration for potential inclusion in a future analytic template/data set. If you have any questions or wish to delve deeper into your submission, please feel free to get in touch with us.


bigquery-export: Add document.id to BigQuery tables

[REQUIRED] Step 2: Extension name
This feature request is for extension: firestore-bigquery-export
What feature would you like to see?
For this feature to be useful to me, I really need the firestore document's id exposed as a single field in the generated tables. Currently, the fully qualified document_name is provided, but this includes a bunch of additional prefixes that makes it really difficult/cumbersome to join across tables.
How would you use it?
I want to use the big query tables to power DataStudio as well as for running one-off queries to better understand my app's usage.
I often store data semi-normalized using document identifiers, such as memberId (which is like a userId), and then need to join to the memebers collection, using memberId as the join key.
Currently, I would have to concact a string together to join on document_name in this format
projects/XX-project-nameXX/databases/(default)/documents/members/<my member document id>

Hi, it looks like now import has the document id column, however the extension which sync the firestore with bigquery table does not include document id column so the table now has imported rows with document id, created rows without

Hi, may I know about the current state of this feature? It looks like IMPORTS generates the document_id filled-id with the document ID however CREATE/UPDATE leave this column blank, which creates inconsistencies

@mustafaekim May I ask what version of the firestore-bigquery-export extension you're using? You should be able to see it in your Firebase console dashboard in the extension's tab.

Hi, it is Version 0.1.5
Thanks for getting back

If you head to your firestore-bigquery-export extension in the extension's tab on the Firebase console, you ought to be able to update to version 0.1.6 which has the document_id update.

Great, thanks. Do you have any recommentation to fix the previous records? Should I delete the table and import again?

That would be my recommendation 😄 .


Accounts viewer not showing list of maps after synch

After January 2018 Carto synch the list of maps on viewer accounts are not shown.

Solved with https://github.com/splashblot/dronedb/commit/96397c302c3216435553698518321d0610d479af that reverts commit https://github.com/splashblot/dronedb/commit/171a8cfcb4005573383c253e329c698e20d1fe93


Dynamodb autoconfiguration

:loudspeaker: Type of change

[ ] Bugfix
[x] New feature
[ ] Enhancement
[ ] Refactoring

:scroll: Description
:bulb: Motivation and Context

:green_heart: How did you test it?
:pencil: Checklist

[ ] I reviewed submitted code
[ ] I added tests to verify changes
[ ] I updated reference documentation to reflect the change
[ ] All tests passing
[ ] No breaking changes

:crystal_ball: Next steps

I introduced DynamoDbAdminOperations. This is just example how it can look. I can see this can be used for easier testing and much more.  If we agree that we should include it I will autoconfigure it in Autoconfiguration module.