allow passing in options when creating HDF5 files

I am trying to add a way to specify extra options when creating HDF5 files, and this is a first attempt. It hasn't been tested. Are there cleaner / more consistent ways to do it?

Another option for this would be to create a wrapper function to hdf5.File() in pyscf.lib which essentially always adds the kwargs **param.H5F_WRITE_KWARGS. That is
in pyscf/lib somewhere:
def h5py_write(filename, mode, **kwargs):
    if kwargs:
        write_options = param.H5F_WRITE_KWARGS.copy()
        write_options.update(kwargs)

    else:
        write_options = param.H5F_WRITE_KWARGS

    h5py.File(filename, mode, **write_options)

This way the rest of the code base just calls h5py_write without having to pass the **param.H5F_WRITE_KWARGS each time. But the if kwargs statement above would still allow someone to pass some additional kwargs/update the default kwargs if they wanted.

The downside to your approach is that if someone adds a new h5py.File() call, they may not include the **param.H5F_WRITE_KWARGS option. Whereas if one uses a wrapper function in the library, all h5py.File calls always have the same **kwargs. Of course, this requires that all usages of h5py.File are replaced, so it is a bit of a trade-off.

Thanks for the suggestion! I like your wrapper idea and will switch to it soon.

OK, I've implemented Matthew's wrapper idea! This is now ready for review.
Also, as part of a program to fight a bug where outcore (especially PBC) DF randomly hangs on NFS, I reimplemented H5TmpFile so that the temporary file is not opened in both the HDF5 library and Python, with two separate file descriptors. The procedure for generating unique filenames is unchanged.

@sunqm
If you have the time to look at this before the 2.6 release, I'd be very grateful! My group is running into the aforementioned HDF5+NFS bug fairly often these days, and it has been difficult to track down the cause or reliably reproduce the bug.
Hopefully this PR is the end of that bug -- if not, the ability to pass in extra HDF5 options would really help with troubleshooting!

Could you open an issue to describe the issue you encountered in NFS? Is it only caused by the H5TmpFile instances, or all HDF5 files?
This PR introduces an abstract to manage the h5py files, which is generally not bad IMO. However, it's not clear what the HDF5+NFS issues exactly are. How would one reproduce it or avoid it by using this PR. Simply applying this PR does not seem to solving the problem. If others meet the similar problem, they might not get information from this PR.
This update will not be applied in 2.6.0 release, as 2.6.0 will be packaged soon. We can target to include this PR in the patch release of 2.6, if problems can be analyzed in more details in a separated github issue.

@sunqm
Thanks for looking at this for me! I've just opened a separate issue describing the bugs my lab has encountered (#2248).
Sadly, the first of those bugs will probably be difficult to diagnose properly. I will do my best and let you know if I find any useful information.


"Need a Lift" creation fails without route

Creating a "Need a Lift" post fails if route is not set.

Route should be mandatory anyways.

maybe making the route mandatory isnt the best option for all cases, like what if one wants to hide the route from the public for example... maybe thats somewhat reasonable

Okay, good point. The Bug is now fixed without the mandatory changes.


Swap test_details for get_perf_path() and get_record_name()

This is a cleaner API and it leads to no problems with overrides of path or record_name. It also makes the separated functions easier to test.
I've also added a test with a pytest fixture as an example of what can be done with this.

@shanx after merging your PR I realized it could be done in a simpler way without adding a new argument, which is this. Sorry to reverse your work but I think it's for the best API. An updated version of your pytest fixture can be found in the example.
I'm also thinking of both exposing such a pytest fixture and supporting auto-detecting the name via current_test() when a Pytest request can be detected, since the library is already a Pytest plugin.


h

not sure if the redis info is correct or not

That's the default redis port. It's filled in by the installer.

But regarding the password,
I think it should be
echo "requirepass  [password]" >> /path/to/your/redis.conf

Oh yes, sorry requirepass i forgot that

and the appending op >>

Fixed the Redis password adding code.

Just updated the cli.md file.

looks good

Thanks man.


Wrong start date of Odd Semester 2022/2023

Should have started on 15th of August, but instead it started on 1st of August

I admit finding the start date of the semester is a difficult task, so currently how does this work is it finds the first Monday of the month (Odd semester=August, Even semester=February).
https://github.com/refeed/SimasterICSGen/blob/0302b3a03ca5f4a878f9b0ce177185bbd64a218a/internal/simasterclass/parser.go#L108-L127
I think a better approach would be to enable the user to specify a custom start date of the semester.

BTW thanks for the report, I really appreciate it!


B: First Response tooooooo Slow

This afternoon I pull up the server and try to battlefield.wikiloop.org the entry point from my desktop. It took more than 30 seconds at least.

It turns out, ORES is the major source of delay
Here is one of the sample:
[2019-09-01T23:34:31.190] [INFO] perf - Response delay for /api/latestRevs = 5116
[2019-09-01T23:34:31.190] [INFO] perf - Response delay for /api/latestRevs recentChange = 241
[2019-09-01T23:34:31.190] [INFO] perf - Response delay for /api/latestRevs ores = 4873
[2019-09-01T23:34:31.190] [INFO] perf - Response delay for /api/latestRevs post-processing = 2

This is resolved


Fix Info about Global library on add_animation_library method in docs

The information about how to add a Global library is not in the documentation currently.
This PR adds the following information in the add_animation_library method
in doc/classes/AnimationMixer.xml:
"When name is set to an empty string, the Global library is added."

Bugsquad edit, closes: https://github.com/godotengine/godot-docs/issues/9550

So, I should remove the info I added, and instead add this code in the description. Am i right?

Please squash the commits to follow PR workflow https://docs.godotengine.org/en/stable/contributing/workflow/pr_workflow.html#the-interactive-rebase

Thanks!


Offline Simulations

Is your feature request related to a problem? Please describe.
Are there more videos available for offline demos? I know there are some Rosbags out there but no idea how to use them (if those are the answer).
Additional context
Offline simulation scenarios

those are available at bce.apollo.auto, you should be able to download quick a few records from those opened scenarios.


Update PostgresStorageAdapter.js

correcting some queries
correcting spellings
adding notes

It looks like this PR changes the whitespace for the entire file, maybe your editor did that automatically. Could you undo that? It both makes it hard to tell what actually changed, and make git blame a lot less useful.

[Current coverage][cc-pull] is 90.34%

Merging [#2082][cc-pull] into [master][cc-base-branch] will decrease coverage by 1.56%

@@             master      #2082   diff @@
==========================================
  Files            93         93
  Lines          6717       6715     -2
  Methods        1191       1190     -1
  Messages          0          0
  Branches       1411       1411
==========================================
- Hits           6174       6067   -107
- Misses          543        648   +105
  Partials          0          0

Powered by Codecov. Last updated by [0ff35e1...b1ee52e][cc-compare]
[cc-base-branch]: https://codecov.io/gh/ParsePlatform/parse-server/branch/master?src=pr
[cc-compare]: https://codecov.io/gh/ParsePlatform/parse-server/compare/0ff35e18f03bf736d7772b108595143c9c4aa8cc...b1ee52edb9cb1a669b890697eea6d43d7e5d3dab
[cc-pull]: https://codecov.io/gh/ParsePlatform/parse-server/pull/2082?src=pr

Damn, I've made so many small changes, and to get get rid of the changed white spaces I would have to re-do everything.
Do you have tests for that to confirm the correctness?

@vitaly-t shift-tab is your friend, can't you simply fix that with a formatter/linter/manually in no time?

Yep, I don't know what editor you are using, but in sublime text you can just set the indentation level and then re-indent everything.
We have fairly comprehensive tests so I'm pretty sure your code is correct, but I'd like to look at what exactly changed as well, which is hard when the whitespace changes make it look like everything chagned. If you don't want to do it, I could do the re-indentation myself, but you would lose attribution.

I'm using WebStorm. I would pay better attention to it next time, but re-doing all changes now is kind of late :|

It looks like you have just switched the file from 2 space indenting to 4 space indenting, if you just reformat it back into two space indenting and commit that the diff should then work ok?

@vitaly-t http://pastebin.com/raw/qnTYnbBJ
Simply paste that into your IDE for the file and recommit, it fixes the whitespace.

No problem, I'll close this and apply the same changes in a new PR.


ruby.owo.rs

[x] There is reasonable content on the page
[x] I agree to the terms of service and will follow the acceptable use policy.

Reason for claiming this subdomain:
Yes.

lgtm; don't forget to add the domain to your vercel project


apc.imports not importing all css files

VSCode Version: 1.84.2
"apc.imports": [
"/C:/VSCODE_CSS/A/style.css",
"/C:/VSCODE_CSS/B/style.css",
"/C:/VSCODE_CSS/C/style.css",
]
Only the last CSS file style is loaded, so in this case, C worked, but not AB. I can switch the order, but only the last works.
Not sure if this is how it's meant to work, should I only be including one CSS and JS file for imports?

I had same problem when started using extension, and merged all the css files into one.

Same problem, this is a bug.
I dont want to merge all CSS files though.
Is there a workaround for this?


'ignore_url_regexes' should prevent all instrumentation

Giving the name of the config: rules.ignore_url_regexes I was expecting and under the impression that it would prevent ALL kind of data from being collected from the specified url's.
Imagine my surprise when I realized that my Browser data was wrong because this configuration would only suppress the reporting of APM data...
Trying to search for a better documentation or a simple way to prevent ALL data from being reported to New Relic I found out that there was no option for doing that.
What do you guys think of making the ignore_url_regexes to really ignore every kind of instrumentation by not adding the browser monitoring as well?

Hi @diego-silva thank you very much for bringing this inconsistency to our attention! I can definitely see how the way the agent works now would be unexpected for your browser instrumentation. I'm closing this PR because we think we'd want to address the issue at a deeper level* but we've discussed it on the team and I've created a ticket in our backlog to track this as a bug to be fixed. Thanks again for letting us know and giving it a shot!
*in short, there's a concept of ignoring transactions that we think should be fixed to make sure browser data is ignored if the transaction is ignored, and this config setting does affect the list of transactions to be ignored.


add com class/interface features

closes #322
Checklist

[ ] No CHANGELOG update needed

[ ] No new tests needed

[ ] No documentation update needed

test rule
rule:
  meta:
    name: initialize IWebBrowser2
    namespace: communication/http
    authors:
      -<EMAIL_ADDRESS>    scope: basic block
    mbc:
      - Communication::HTTP Communication::IWebBrowser [C0002.010]
    references:
      - https://docs.microsoft.com/en-us/previous-versions/windows/internet-explorer/ie-developer/platform-apis/aa752127(v%3Dvs.85)
    examples:
      - 395EB0DDD99D2C9E37B6D0B73485EE9C:0x402130
  features:
    - and:
      - api: ole32.CoCreateInstance
      - com/class: InternetExplorer #bytes: 01 DF 02 00 00 00 00 00 C0 00 00 00 00 00 00 46 = CLSID_InternetExplorer
      - com/interface: IWebBrowser2 #bytes: 61 16 0C D3 AF CD D0 11 8A 3E 00 C0 4F C9 E2 6E = IID_IWebBrowser2

statements for above rules are built as
and(
api(ole32.CoCreateInstance),
or(bytes(01 DF 02 00 00 00 00 00 C0 00 00 00 00 00 00 46 = InternetExplorer),string(0002DF01-0000-0000-C000-000000000046 = InternetExplorer)),
or(bytes(61 16 0C D3 AF CD D0 11 8A 3E 00 C0 4F C9 E2 6E = IWebBrowser2),string(D30C1661-CDAF-11D0-8A3E-00C04FC9E26E = IWebBrowser2))
)

the com class is converted to String(guid_string), Bytes(guid_byte)
test run using above sample rule

test run using original rule initialize IWebBrowser2

also, please add a few tests showing this functionality working as expected.
include things like:

matching a file as expected
generating the bytes/strings as expected
what happens if an unknown COM class/interface is provided?

we'll also need updates to the documentation that explain this new feature and how to use it.

@williballenthin
In classes.txt file I found these classes mentioned two time with different guids InProcFreeMarshaler , MSVideoControlVideoRenderer, MSVideoControlStreamBufferEngineSink, WICImagingFactory, WICPngDecoder.
Similarly in interfaces.txt there were 763 COMs with two guids possible. Also DEPRECATED is mentioned quite a lot of times in interfaces.txt.
Each repeated COM has only two different pairs of GUIDs.
We can s tore guids as list for each COM in json file.
{
"DEPRECATED" : ["13043da7-7fca-4c2b-9eca-6890f9f00185","143ba4fc-e9c5-4736-91b2-e8eb9cb88356","14d981a3-fc26-4ff7-abc3-48e82a512a0a"]
}

Documentation has been updated.
PR linked is https://github.com/mandiant/capa-rules/pull/817

@Aayush-Goel-04 are there any outstanding tasks/questions from your side?

All tasks are completed, if you'd like any changes, feel free to ask.

this is so close to being merged! i hope we can do that today or tomorrow :-)

Thanks for the heads up! Just so you know, I'm juggling this with my college stuff, but I'll aim to wrap it up in the next 3-4 days. I'll keep you posted. Thanks for being cool about it!

no worries! i also acknowledge that this PR has been open for a while since we haven't always promptly reviewed, so we really appreciate your patience :-)

use pyinstaller-compatible references to the assets, and

shifted to get_default_root()

loading the database once

@williballenthin , Above both tasks have been resolved.

I think this is G2G now?!


Issue when using a date (in a nested field) for X-Axis in a Date Histogram

Plugin version: 6.4.2
Kibana version: 6.4.2
Description of the problem including expected versus actual behavior:
I would like to use a field which contains a timestamp in a nested field (userActions.startTime) as below :

There is no error but unfortunately the graph is empty:

That seems to be a bug ? :)

In most cases, it means that the generated query has no results or wrong.
Can you add the generated ElasticSearch query here? It it accessable via the graph from Request tab. Try also to identify errors shown in the Response tab.

Hi IdanWo, I don't find the way to display these buttons below the chart (as in your screenshot).
Maybe I have to activate this feature in Advanced Settings ?

If that can help, a sample of data (not all the json file, just an extract) :
  "_index": "dynatrace",
  "_type": "session",
  "_id": "nqdxxxxx-LSKOLGNNWNBNLVLXSTREEGEEZIGANHYJ-1544514266703",
  "_version": 1,
  "_score": null,
  "_source": {
    "userSessionId": "LSKOLGNNWNBNLVLXSTREEGEEZIGANHYJ",
    "startTime":<PHONE_NUMBER>703,
    "endTime":<PHONE_NUMBER>093,
    "duration": 3684390,
    "internalUserId": "1543305377680LRQOGC7OGHSQLADMG4BRNFP0O3HCJ14E",
    "applicationType": "WEB_APPLICATION",
    "userActionCount": 56,
    "totalErrorCount": 0,
    "userActions": [
      {
        "name": "PROD_ViewOrder",
        "startTime":<PHONE_NUMBER>703,
        "endTime":<PHONE_NUMBER>193,
        "duration": 3490,
        "internalApplicationId": "APPLICATION-4AD43E503369C9FC",
        "networkTime": 343,
        "serverTime": 199,
        "frontendTime": 2948
      },
      {
        "name": "PROD_QT_QuoteToOrder",
        "startTime":<PHONE_NUMBER>545,
        "endTime":<PHONE_NUMBER>263,
        "duration": 5718,
        "networkTime": 8,
        "serverTime": 201,
        "frontendTime": 5509
      },
      {
        "name": "PROD_ViewOrder",
        "startTime":<PHONE_NUMBER>355,
        "endTime":<PHONE_NUMBER>049,
        "duration": 2694,
        "networkTime": 49,
        "serverTime": 198,
        "frontendTime": 2447
      },
      {
        "name": "PROD_ProductScreen",
        "startTime":<PHONE_NUMBER>432,
        "endTime":<PHONE_NUMBER>453,
        "duration": 1021,
        "networkTime": 18,
        "serverTime": 415,
        "frontendTime": 588
        }
    ],
  },
  "fields": {
    "startTime": [
      "2018-12-11T07:44:26.703Z"
    ],
    "endTime": [
      "2018-12-11T08:45:51.093Z"
    ]
  },
  "sort": [
<PHONE_NUMBER>703
  ]
}

with the associated mapping :
{
  "mappings": {
    "session": {
      "dynamic_templates": [
        {
          "num_fields": {
            "path_match": "numTags.*",
            "mapping": {
              "norms": "false",
              "type": "double"
            }
          }
        },
        {
          "date_fields": {
            "path_match": "dateTags.*",
            "mapping": {
              "norms": "false",
              "type": "date"
            }
          }
        },
        {
          "string_fields": {
            "match": "*",
            "match_mapping_type": "string",
            "mapping": {
              "norms": "false",
              "type": "keyword"
            }
          }
        }
      ],
      "properties": {
        "applicationType": {
          "type": "keyword"
        },
        "dateTags": {
          "type": "object"
        },
        "duration": {
          "type": "integer"
        },
        "endTime": {
          "type": "date"
        },
        "internalUserId": {
          "type": "keyword"
        },
		"startTime": {
          "type": "date"
        },
        "totalErrorCount": {
          "type": "integer"
        },
        "userActionCount": {
          "type": "integer"
        },
        "userActions": {
          "type": "nested",
          "properties": {
            "duration": {
              "type": "integer"
            },
            "endTime": {
              "type": "date"
            },
            "frontendTime": {
              "type": "integer"
            },
            "name": {
              "type": "keyword"
            },
            "networkTime": {
              "type": "integer"
            },
            "serverTime": {
              "type": "integer"
            },
            "startTime": {
              "type": "date"
            }
          }
        },
        "userSessionId": {
          "type": "keyword"
        }
      }
    }
  }
}

Hello, I move forward on my problem.
I recreated my index by letting elasticsearch create it with the first entry (so new mapping file is used).
It is now possible to select userActions.startTime in sub aggregation.
.
My problem is now the following : "userActions.duration" value is always the same for each "userActions.name". Instead of displaying each time the specified duration in the correct order on the X-Axis, all actions appear to be  merged over the same timeframe :

Any idea about that ? ^^

Considering the following json :
"userActions": [
{
"name": "Action1",
"startTime":<PHONE_NUMBER>007,
"duration": 4231
}
{
"name": "Action2",
"startTime":<PHONE_NUMBER>072,
"duration": 558
}
{
"name": "Action3",
"startTime":<PHONE_NUMBER>154,
"duration": 3146
}

If I want to apply a filter on a specific userActions.name (Action1 for example), all the other actions in the array (Action2 and Action3 in the case above) are selected in my chart, not only Action1.
I'm not a elasticsearch or kibana expert but maybe "inner_hits" option could resolved this issue ?
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-inner-hits.html#top-level-inner-hits
The screenshot of the chart with the filter :

I have finally found how to get the request in my version of Kibana :
{
  "aggs": {
    "2": {
      "date_histogram": {
        "field": "userActions.startTime",
        "interval": "1h",
        "time_zone": "Europe/Berlin",
        "min_doc_count": 1
      },
      "aggs": {
        "3": {
          "terms": {
            "field": "userActions.name.keyword",
            "size": 3,
            "order": {
              "1": "desc"
            },
            "include": "PROD.*"
          },
          "aggs": {
            "1": {
              "avg": {
                "field": "userActions.duration"
              }
            }
          }
        }
      }
    }
  },
  "size": 0,
  "_source": {
    "excludes": []
  },
  "stored_fields": [
    "*"
  ],
  "script_fields": {},
  "docvalue_fields": [
    {
      "field": "endTime",
      "format": "date_time"
    },
    {
      "field": "events.startTime",
      "format": "date_time"
    },
    {
      "field": "query.constant_score.filter.bool.filter.range.startTime.gte",
      "format": "date_time"
    },
    {
      "field": "query.constant_score.filter.bool.filter.range.startTime.lte",
      "format": "date_time"
    },
    {
      "field": "startTime",
      "format": "date_time"
    },
    {
      "field": "userActions.endTime",
      "format": "date_time"
    },
    {
      "field": "userActions.startTime",
      "format": "date_time"
    }
  ],
  "query": {
    "bool": {
      "must": [
        {
          "range": {
            "startTime": {
              "gte":<PHONE_NUMBER>838,
              "lte":<PHONE_NUMBER>838,
              "format": "epoch_millis"
            }
          }
        },
        {
          "match_phrase": {
            "userActions.name.keyword": {
              "query": "PROD_PriceScreen"
            }
          }
        }
      ],
      "filter": [],
      "should": [],
      "must_not": []
    }
  }
}

Thanks for your assistance :)

See the discussion in #102.
Duplicate of #90


Result Highlighting

Can Xapiand highlight matching text within document fragments, using Xapian::MSet::snippetunder the hood?

It would be nice to have it but is not supported right now

Thanks!


Widgets in 3.18

Hy @alaframboise ,

I am trying to integrate the LayerList widget in the arcgis-3.x.html but it's not working as expected.
Could you provide some support? My .html is available on JS Bin.

Could you provide the Measurement widget in your arcgis-3.x.html?
I am struggling with this one as well.

Thank you for your support,
Tim.

The 3.x doc indicates that the LayerList widget is not supported with the calcite.css theme.
https://developers.arcgis.com/javascript/3/jshelp/css.html
This theme is in beta and does not support all components of the API. The following widgets are supported:

BasemapGallery
BasemapToggle
Bookmarks
Directions
HomeButton
LayerSwipe
Legend
LocateButton
Measurement
OverviewMap
Popup
Print
ScaleBar
Search

To get it to load and at least render you need to pull in the .css manually for that component. With a few other background and border overrides you should be able to get to render nicely with Calcite Maps.

LayerList widget now working, just needs a litlle css background.
Thank you @alaframboise  !

@TimPeters post a sample here please.

Hi @ezequias ,
Here is a JS Bin with the sample code.

Thank you very much @TimPeters but the LayerListWidget is much more than that.

@ezequias and @TimPeters You might be interested in how we have utilized calcite maps with CMV, the configurable map viewer. The implementation has a full layer list for 3.x as well as many other widgets. Support for 4.x is coming soon. I provided a more detailed answer in issue #150. The repo is available here.

I really love it @tmcgee great work. Does it have any kind of licensing?


[IOTDB-1675] Introduce UnaryMeasurementMNode and MultiMeasurementMNode

Description
Motivition

Introduce UnaryMeasurementMNode and VectorMeasurementMNode for better readability and maintainability. The logical process of such two scenarios will be more clear.
Check the MNode usage inside and outside metadata module and eliminate outside usage as much as possible. This obeys OOD principle and make metadata serialization work more easier.

Modification
Introduce UnaryMeasurementMNode and VectorMeasurementMNode

Make origin MeasurementMNode abstract and introduce two subclass, UnaryMeasurementMNode and VectorMeasurementMNode.
Use factory method to get and init MeasurementMNode.
Encapsulate MNode type transform/cast and eliminate force type cast code as much as possible
Introduce MNodeTypeMismatchException

Improve MNode usage

Replace getNodeByPath method in MManager with type specified MNode access method
Eliminate deviceMNode usage in MergeTask.class
Transfer some schema collection code from MManager to MTree
Refactor LastCache interface and usage using New MeasurementMNode. Fix potential bug

Coming work
Renaming vector to multi, subMeasurement to component, in terms ofMeasurementSchema.

Coverage increased (+0.02%) to 67.761% when pulling 467c718b3b2a149523c834e0b1db93f3af6d1cff on zyk990424:MultiMeasurement into 0163d3aa76d452fe2c1d864790451dcfd3d5cd5e on apache:master.


[BUG] Fix inconsistent border-radius for input fields across all forms

Description
There is a styling issue where the input fields across various forms on the website have a border-radius applied only to the top-left and bottom-left corners. This creates an uneven and inconsistent appearance in the form fields. There seems to be clash in the styling for form input fields and search input field.
Expected Behavior: All corners (top-left, top-right, bottom-left, and bottom-right) of the form input fields should have the same border-radius for a balanced and consistent appearance.
Please assign this issue to me, I would like to work on this.
Screenshots

Any additional information?
No response
What browser are you seeing the problem on?
No response
Checklist

[X] I have checked the existing issues
[X] I have read the Contributing Guidelines
[X] I'm a GSSoC'24-Extd contributor
[X] I'm a Hacktoberfest'24 contributor
[X] I am willing to work on this issue (optional)

@kavya410004 you are assigned.

@mdazfar2 Thank you for assigning, I will start working on it.


Sub-classes of StatisticalProduct

Decided during May 7 meeting: define what sub-classes of StatisticalProduct we want.
Example candidate: StatisticalIndicator.

Linked to issue #6

We could distinguish between data products, e.g. StatCan's CANSIM tables, and analysis, e.g. research papers providing insights into the data.

We could also distinguish between census, surveys and other statistical programs.

Statistics Norway also has interactive products.

Could be worth checking that we are consistent with GSIM Product  v1.2:
Definition: A package of content that can be disseminated as a whole.
Explanatory text: A Product is a type of Exchange Channel for outgoing information. A Product packages Presentations of Information Sets for an Information Consumer. The Product and its Presentations are generated according to Output Specifications, which define how the information from the Information Sets it consumes are presented to the Information Consumer. The Protocol for a Product determines the mechanism by which the Product is disseminated (e.g website, SDMX web service, paper publication).
A Provision Agreement between the statistical organization and the Information Consumer governs the use of a Product by the Information Consumer. The Provision Agreement, which may be explicitly or implicitly agreed, provides the legal or other basis by which the two parties agree to exchange data. In many cases, dissemination Provision Agreements are implicit in the terms of use published by the statistical organization.
For static Products (e.g. paper publications), specifications are predetermined. For dynamic Products, aspects of specification could be determined by the Information Consumer at run time. Both cases result in Output Specifications specifying Information Set data or referential metadata that will be included in each Presentation within the Product.

Good point. I believe it does align with GSIM Product, to some extent.
We need to discuss as well whether we want to get into the minutia of GSIM Presentation, OutputSpecification, etc. (I think we shouldn't) or just consider all that embedded into COOS:StatisticalProduct.
Whether we need to cover GSIM ExchangeChannel in detail is also a matter of discussion --perhaps a separate issue?

We should focus on statistical products (the ones you mentioned before) that is made of statistical data. Other statistical products e.g. statistical registers, statistical applications, statistical classification should be taken into consideration at later stage.
I tried to find some literature on classification of statistical products unfortunately without any result yet.

Statistical Product is a prov:Entity. When we describe statistical products in the ontology, we agree that sub-classes are needed.
What do our ModernStats models say about products?
GSBPM
2.1. Design outputs
This sub-process contains the detailed design of the statistical outputs, products and services to be produced, including the related development work and preparation of the systems and tools used in the "Disseminate" phase.
7.2. Produce dissemination products
This sub-process produces the dissemination products, as previously designed in sub-process 2.1 (Design outputs), to meet user needs. They could include printed publications, press releases and websites. The products can take many forms including interactive graphics, tables, maps, public-use microdata sets, linked open data and downloadable files.
GSIM
Product: A package of content that can be disseminated as a whole. A Product is a type of Exchange Channel for outgoing information. A Product packages Presentations of Information Sets for an Information Consumer. The Product and its Presentations are generated according to Output Specifications, which define how the information from the Information Sets it consumes are presented to the Information Consumer. The Protocol for a Product determines the mechanism by which the Product is disseminated (e.g website, SDMX web service, paper publication).
For static Products (e.g. paper publications), specifications are predetermined. For dynamic Products, aspects of specification could be determined by the Information Consumer at run time. Both cases result in Output Specifications specifying Information Set data or referential metadata that will be included in each Presentation within the Product.
Presentation: The way data and referential metadata are presented in a Product. A Product has one or more Presentations, which present data and referential metadata from Information Sets. A Presentation is defined by an Output Specification.
Presentation can be in different forms; e.g. tables, graphs, structured data files.
Examples:
A table of data. Based on a Data Set, the related Data Structure is used to label the column and row headings for the table. The Data Set is used to populate the cells in the table. Reference metadata is used to populate footnotes and cell notes on the table. Confidentiality rules are applied to the Data Set to suppress any disclosive cells.
A data file based on a standard (e.g. SDMX).
o	A PDF document describing a Statistical Classification.
o	Any structural metadata object expressed in a standard format (e.g. DDI 3.1 XML).
o	A list of Products or services (e.g. a product catalogue or a web services description language (WSDL) file).
o	A web page containing Statistical Classifications, descriptions of Variables, etc.
GAMSO
Manage Consumers: These activities cover the management of communication and exchanges between governmental or international institutions, the public, and other stakeholders in direct or indirect support of organisational services. They therefore deal with the relationships between statistical organisations and the public, including those via the media. This includes general marketing activities and dealing with non-specific consumer feedback. This also includes measures to educate and inform users so that they fully understand statistical outputs, and to promote and improve levels of statistical literacy in society in general. Manage cross-product user support

As it can be seen from the above listed concepts GSIM and GSBPM only concentrates on products which are from statistical data as the concept ‘Product’ in GSIM is only in connection with Statistical Programme therefore other products which are not from a Statistical Programme e.g. registers, classifications are excluded.
In the Process phase 7 of the GSBPM also products from statistical data are highlighted, other are not mentioned before.
But if we take GAMSO on the board the situation might be a bit different. There is only one mentioning of products in the GAMSO, which are related to statistical data (see Manage consumers). Other GAMSO elements have also products as those ‘activities’ also take input and transform them to outputs. These are mainly not made of statistical data e.g vision, strategy, financial plan, resources plan etc. These outputs/products are not fully present in GSIM (only which are is close connection to a statistical business process - understandably) but later on those non statistical products have also be determined and standardised.
### Conclusion and suggestion:
The adjective ‘Statistical’ for products should be kept for this project, since non-statistical products (e.g. resource plans, websites, methods) are the outputs from Statistical Support Programme (using GSIM object) or from ‘activities’ of GAMSO. These are probably out-of-scope for now if we set the scope of the work to the statistical business process. Basically this means Product has two sub-classes: StatisticalProduct (within scope for us) and NonStatisticalProducts (out of scope for us).
For the sub-classes of StatisticalProduct, the GSIM already has classification of static and dynamic products. As the main focus of the work is to support statistical activities, we should define sub-classes that make sense from the statistical perspective and it is easy for the user (statistician) of the ontology to understand the sub-classes we define.
If we take a look around of what classifications of statistical products are available for official statistics, there is no standard or recommendation for it (or we have not found it yet).
If we check NSI webpages, the following items are usually referred to as statistical products (this is not a classification, only a summary of observations made based on a few NSI webpages):

Data: statistical databases storing vast amount of statistical information in table or database formats. Sometimes certain statistical domains are highlighted as separate items, e.g. census but it is another classification: statistical domains not a classification of products, in my opinion.
Publications
Visualisations (maps, infographics, etc.)
Data access services

Bonus question:
GSBPM mentions outputs, products and services. Does this mean that we need StatisticalOutput, StatisticalProduct and StatisticalService as separate entities? Do we want to deal with StatisticalServices as a separate entity? In my opinion, if we want the scope to be in line with the GSBPM as well, then we need to. But while sub-classes are needed for statistical products, we might not need to specify sub-classes for statistical services (or at least not at the moment). Just keep in our minds to have an entity for StatisticalService.

Very good points!
I think we should focus on Statistical Products for now.
We could look at a couple of dimensions to characterize subclasses:

Type of presentation: Datasets, Publications, Visualizations, Infographics, Thematic Maps, Interactive (given by GSIM Output Specification)
Type of content: Data, Metadata (e.g. a statistical classification), Analysis, Models (given by the subtypes of GSIM Information Sets, not included in GSIM)
others?

Proposed definition of graph dataset:
"Collection of data where datapoints are nodes and relationships between them are edges in a graph structure"
I couldn't find a definition in our documentation, that's why I came up with one, but if you already have a better one please disregard this one.

Classes, concept schemes and properties added for product presentation and product content in commit 76fc9cc90d523bedc40ed2e12b777653142b7eb8 and commit f8067b377d8ace49cc6b231fc38919bfa74af30f respectively, so closing the issue.


Regression: Paths to STATIC_ROOT must now end in os.sep

I've recently updated to the current stable version of django-pipeline and this has broken my Django app. I've traced down the problem to changes to how URLs are generated, I believe in this version: https://github.com/cyberdelia/django-pipeline/commit/074c7b4d9d9e5d27511d64f92d5589949736adb4
My old Django settings.py read as follows:
STATIC_ROOT = os.path.join(PROJECT_ROOT, 'compiled_media')
however due to the relpath call, the actually directory name gets lost (because it doesn't end in a /), and I now need to use:
STATIC_ROOT = os.path.join(PROJECT_ROOT, 'compiled_media') + os.sep
This is a regression from previous versions and should ideally be fixed (this is likely to impact a number of users of Molly which has this project as a dependency)

I can't reproduce your error, see and try branch fix-relpath. Can you explain me what exactly what broke ? It's maybe in another part of pipeline, I suspect absolute_path.

I've tried that branch, however the error still occurs. The problem appears to be that the URLs to the compressed CSS are generated wrong - when I add the / to the end of my STATIC_ROOT, they appear correctly as /static/favourites/css/smart.css, but without, the path is just:

Yes, the probably does appear to be in absolute_path. If on line 198 here: https://github.com/cyberdelia/django-pipeline/blob/master/pipeline/compressors/init.py#L198 I change it to:
path = os.path.join(start, path)
(i.e., get rid of the os.path.dirname), then it works. I don't know if that has any side effects

Can you help me with more detail on how to reproduce (settings, original url, incorrect and correct result url) ? I want to add a regression test for this bug. You can also try to write the test by yourself, if it easier for you ;)

If you change line 21 here: https://github.com/cyberdelia/django-pipeline/blob/master/tests/runtests.py#L21 to get rid of the / on the end of the STATIC_ROOT setting (which is similar to how my STATIC_ROOT is defined, without the /), then your tests will fail

I have been digging into this a bit.  I can confirm that @cnorthwood's suggestion, "(i.e., get rid of the os.path.dirname), then it works," does in fact have side effects.  Anything with a relative URL (i.e. image resources defined in the css) get the wrong path (the path includes the css path in the image location).
I think the correct fix is to take @cnorthwood's suggestion and then to also change
public_path = self.absolute_path(asset_path, css_path)
to
public_path = self.absolute_path(asset_path, os.path.dirname(css_path))
in construct_asset_path.
I can set up a pull request for this, I guess.  This is the fix that we're now using locally.

@cgreene We had the same finding at the same time, it's in the fix-relpath branch, that will be merged soon in master and released.
@cnorthwood Thanks for the report and the help to make it reproducible.


storybook rules

.eslintrc
extends: ["plugin:storybook/recommended",
"plugin:storybook/csf-strict"
"plugin:storybook/addon-interactions"]

see https://github.com/storybookjs/eslint-plugin-storybook

Fixed


WFLY-16853: Upgrade HAL to 3.6.3.Final

https://issues.redhat.com/browse/WFLY-16853

Hello, hpehl. I'm waiting for one of the admins to verify this patch with /ok-to-test in a comment.

/ot-to-test

/ok-to-test


Fix "no tree for xxx" error caused by favourite search links

#1280
Checklist

[x] the [contributor license agreement] is signed
[x] commit message follows [commit guidelines]

Description of change
After discussing with @mrblippy , we think the best fix is to check and rewrite the URL of favourite searches because it doesn't require data migration. Maybe there are different approaches, but some significant changes would be required.
In terms of why the redirection Servlet not working, it is probably because clicking the favourite search link triggers a tree check before it sends the request.

Looks like you may need to rerun your travis build @PenghaiZhang .

Looks like you may need to rerun your travis build @PenghaiZhang .

Ah yes. Thanks for reminding!!!


Improve s3 bucket policy linting

cfn-lint version: (cfn-lint --version)
cfn-lint 0.16.0
Description of issue.
Add better linting for s3 bucket policies
I have the following template:
  ReportBucketPolicy:
    Properties:
      Bucket: !Ref 'ReportBucket'
      PolicyDocument:
        Statement:
          - Action:
              - s3:ListBucket
            Condition:
              StringEquals:
                aws:PrincipalOrgID: !Ref 'OrgId'
            Effect: Allow
            Principal: '*'
            Resource:
              - !Sub '${ReportBucket.Arn}/*'
            Sid: AllowOrgAccess
        Version: '2012-10-17'
    Type: AWS::S3::BucketPolicy

I initially had a typo so that I forgot the $ in the Sub() so that it was like this: - !Sub '${ReportBucket.Arn}/*'
Can cfn-lint be made to catch that the resource specified was invalid?
The above template also has an issue because the resource shouldn't have a * for s3:ListBucket. This is the error that CloudFormation returns: Action does not apply to any resource(s) in statement
Is it within cfn-lint's scope to validate PolicyDocuments?

Technically the s3:ListBucket is caused by your reference to * objects in the ReportBucket rather than a bucket. * by itself is valid, but would throw a warning by cfn-lint, e.g.
Action: s3:ListBucket
Resource: '*'

Fully agree that cfn-lint should catch this. / is not a legal character in a bucket name, so any resource containing / should throw an error if the Actions block contains actions which only apply to buckets.


Scrapli hangs when connecting to misbehaving devices

Describe the bug
When connecting to a misbehaving cisco switch, Scrapli Async IO driver will hang indefinitely. When SSH using putty, the switch will quickly open the connection and close it a few moments later (ie no prompt or data returned). While I understand the real issue here is the device misbehaving (IOS-XE bug), it would be nice if Scrapli handled this situation. At the moment, I have to check if script has been running too long and kill it manually. I have experienced similar behavior across a few devices over the past years.
To Reproduce
Steps to reproduce the behavior:

Connect to switch with a bug (ie doesn't return a prompt or any data)
Cisco 9300 17.6.4

Expected behavior
Expect the timeout or an error on a device that does not respond as expected.
Stack Trace
Interrupting with ctrl-c yields:
 File "/home/dev/poll/./poll.py", line 113, in <module>
    asyncio.run(main())
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 636, in run_until_complete
    self.run_forever()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
    self._run_once()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
    handle._run()
  File "/usr/lib/python3.10/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/home/dev/poll/venv/lib/python3.10/site-packages/scrapli/channel/async_channel.py", line 448, in get_prompt
    buf += await self.read()
  File "/home/dev/poll/venv/lib/python3.10/site-packages/scrapli/channel/async_channel.py", line 70, in read
    buf = await self.transport.read()
  File "/home/dev/poll/venv/lib/python3.10/site-packages/scrapli/decorators.py", line 198, in decorate
    return await asyncio.wait_for(wrapped_func(*args, **kwargs), timeout=timeout)
  File "/usr/lib/python3.10/asyncio/tasks.py", line 405, in wait_for
    loop = events.get_running_loop()
KeyboardInterrupt
ERROR:asyncio:Task exception was never retrieved
future: <Task finished name='Task-575148' coro=<AsyncChannel.get_prompt() done, defined at /home/dev/poll/venv/lib/python3.10/site-packages/scrapli/channel/async_channel.py:423> exception=KeyboardInterrupt()>
Traceback (most recent call last):
  File "/home/dev/poll/./poll.py", line 113, in <module>
    asyncio.run(main())
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 636, in run_until_complete
    self.run_forever()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
    self._run_once()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
    handle._run()
  File "/usr/lib/python3.10/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/home/dev/poll/venv/lib/python3.10/site-packages/scrapli/channel/async_channel.py", line 448, in get_prompt
    buf += await self.read()
  File "/home/dev/poll/venv/lib/python3.10/site-packages/scrapli/channel/async_channel.py", line 70, in read
    buf = await self.transport.read()
  File "/home/dev/poll/venv/lib/python3.10/site-packages/scrapli/decorators.py", line 198, in decorate
    return await asyncio.wait_for(wrapped_func(*args, **kwargs), timeout=timeout)
  File "/usr/lib/python3.10/asyncio/tasks.py", line 405, in wait_for
    loop = events.get_running_loop()
KeyboardInterrupt
sys:1: RuntimeWarning: coroutine 'AsyncsshTransport.read' was never awaited

Screenshots
If applicable, add screenshots to help explain your problem, but do note that formatted text is much preferred over
screenshots!
OS (please complete the following information):

OS: Ubuntu

Additional context
Add any other context about the problem here.

ah interesting. will look more closely when im back from vacation, but I imagine ive just got something not awaited w/ a timeout in the initial connection since everything else should(in theory!) be timed out esp in the async bits. does this happen w/ sync too? can ya share example script too just for kicks, imagine nothing much to see, but cant hurt.

hm, not sure that I ams seeing this behavior. using a mostly copypasta ssh server example from asyncssh like this:
import asyncio
import sys
from typing import Optional

from asyncssh import SSHServer, SSHServerConnection, create_server, SSHServerSession
from asyncssh.public_key import SSHKey

BASE_SERVER_KEY = b"""-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAAAlwAAAAdzc2gtcn
NhAAAAAwEAAQAAAIEAwahUv5Tf3vWQzmz2de791K+vy2WQP9q5eOCAIlD2dFb9lTCg3CNl
kJRLMwelj4eJVdfT6YfQjRfbOkuMtGmwz+ed9ulHBVQ8Ee7JuSfxRcazWx2Wet5wzA0vkv
dohzw20jHhmLpbAi/x20Zxv5R+jK3o/+x6ciIW6sYCoQXJw88AAAIQhQL0T4UC9E8AAAAH
c3NoLXJzYQAAAIEAwahUv5Tf3vWQzmz2de791K+vy2WQP9q5eOCAIlD2dFb9lTCg3CNlkJ
RLMwelj4eJVdfT6YfQjRfbOkuMtGmwz+ed9ulHBVQ8Ee7JuSfxRcazWx2Wet5wzA0vkvdo
hzw20jHhmLpbAi/x20Zxv5R+jK3o/+x6ciIW6sYCoQXJw88AAAADAQABAAAAgQCE8ss7uz
j2GCARlzycOjaIjRRizpb5P2+VTIqrBGot9IqioX/NoX9YgnYd0mIW5zWheUpCSLskIfyf
SL6QHP8EinQ2e5VPO8sJ3So/2/a9H58ATAZX5D/Rmzjjbh9S57NqlM4y+tSaMVZbUFq53D
uhw+OTG1skt3aK4icJVoYWAQAAAEAkDdyAeM/njkl8IrcutV0Qz9uttRJ7piVGswZmwSiG
aBOVYrSvjNen23FuXXHErMTQbDSMzvI2njBB1P10rteWAAAAQQDjdvmIZGQgLaDqFIQH89
M0USeQavVKmVOjnHMOSzuKzhXoMEGaRtV6s1z6R+0FSblMju69I8KP9x8hmw8JxBn1AAAA
QQDZ86OGU3St5fz9INTz+x6wcsVVDXTywDjlU8UDrpZN9Y8WBiTSG1aNRm7IZXxOcmEJ7L
CwttJsdhYnN0En/zgzAAAAFGNhcmxAaW1wb3N0b3JlLmxvY2FsAQIDBAUG
-----END OPENSSH PRIVATE KEY-----
"""

class BaseSSHServerSession(SSHServerSession):  # type: ignore
    def shell_requested(self) -> bool:
        """
        Handle shell requested; always return True

        Args:
            N/A

        Returns:
            bool: always True!

        Raises:
            N/A

        """
        print("Shell requested")
        return True

class MisbehavingServer(SSHServer):
    def __init__(self, session):
        self.session = session

    def connection_made(self, conn: SSHServerConnection) -> None:
        print("SSH connection received from %s." % conn.get_extra_info("peername")[0])

    def connection_lost(self, exc: Optional[Exception]) -> None:
        if exc:
            print("SSH connection error: " + str(exc), file=sys.stderr)
        else:
            print("SSH connection closed.")

    def begin_auth(self, username: str) -> bool:
        if username != "scrapli":
            return False

        return True

    def password_auth_supported(self) -> bool:
        return True

    def validate_password(self, username: str, password: str) -> bool:
        return password == "scrapli"

    def validate_public_key(self, username: str, key: SSHKey) -> bool:
        return True

    def session_requested(self) -> SSHServerSession:  # type: ignore
        return self.session()

async def run_server() -> None:
    def server_factory() -> SSHServer:
        server = MisbehavingServer(session=BaseSSHServerSession)
        return server

    await create_server(server_factory, "localhost", 8022, server_host_keys=[BASE_SERVER_KEY])

def sync_run_server(loop: Optional[asyncio.base_events.BaseEventLoop] = None) -> None:
    if loop is None:
        loop = asyncio.get_event_loop()

    loop.run_until_complete(run_server())
    loop.run_forever()

if __name__ == "__main__":
    sync_run_server()

and either sync or async test setup like:
from scrapli import AsyncScrapli
import asyncio

async def main():
    dev = {
        "host": "localhost",
        "port": 8022,
        "auth_username": "scrapli",
        "auth_password": "scrapli",
        "auth_strict_key": False,
        "ssh_config_file": False,
        "platform": "cisco_iosxe",
        "transport": "asyncssh",
        "timeout_ops": 2,
    }

    conn = AsyncScrapli(**dev)

    await conn.open()
    print("OPENED")

if __name__ == "__main__":
    asyncio.get_event_loop().run_until_complete(main())

is consistently timing out as I would expect it to with the exception scrapli.exceptions.ScrapliTimeout: timed out getting prompt.
can you share more steps to repro or try w/ this dummy server setup? you can also run the server and ssh to it -- it will handle password auth then just hang forever which I think accurately reproduces what you described, but you can check for yourself to let us know too.

Thanks @carlmontanari, I am having difficulty reproducing this now, the switch stack in question has been rebooted and now does not cause an issue. I was hoping the issue would return and I could do a packet captures but it has been stable for about a week. I will reopen if the issue returns and I can investigate further.

sounds good @forrejam , will close. holler if it comes back!

Not sure how to reopen this issue but I have finally been able to reproduce this and get some debug info. It is very intermittent.
from scrapli.driver.core import AsyncIOSXEDriver
import os
import asyncio
import logging
import asyncssh

asyncssh.set_debug_level(2)
logging.basicConfig()

loggers = [logging.getLogger(name) for name in logging.root.manager.loggerDict]
for logger in loggers:
    logger.setLevel(logging.DEBUG)

async_device = {
    "host": "switch.company.org",
    "auth_username": os.getenv("USER", None),
    "auth_password": os.getenv("PASSW", None),
    "auth_strict_key": False,
    "transport": "asyncssh",
    "ssh_config_file": "ssh_config.conf",
}

async def async_main():
    conn = AsyncIOSXEDriver(**async_device)
    await conn.open()
    print(await conn.get_prompt())

if __name__ == "__main__":
    asyncio.run(async_main())

This will happen once every 20 runs or so, its not consistent. The asyncssh channel closes and Scrapli will loop forever (I have redacted some sensitive info).
DEBUG:asyncio:Using selector: EpollSelector
DEBUG:scrapli.driver:attempting to resolve 'ssh_config_file' file
DEBUG:scrapli.driver:using 'ssh_config.conf' as resolved 'ssh_config_file' file'
DEBUG:scrapli.driver:load core transport requested
DEBUG:scrapli.driver:core transport 'asyncssh' loaded successfully
DEBUG:scrapli.driver:generating combined network comms prompt pattern
DEBUG:scrapli.driver:setting 'comms_prompt_pattern' value to '(^[\w.\-@/:]{1,63}>$)|(^[\w.\-@/:]{1,63}#$)|(^[\w.\-@/:]{1,63}\([\w.\-@/:+]{0,32}\)#$)|(^([\w.\-@/+>:]+\(tcl\)[>#]|\+>)$)'
INFO:scrapli.driver:opening connection to 'switch.company.org' on port '22'
DEBUG:scrapli.transport:opening transport connection to 'switch.company.org' on port '22'
DEBUG:asyncssh:Reading config from "ssh_config.conf"
INFO:asyncssh:Opening SSH connection to switch.company.org, port 22
INFO:asyncssh:[conn=7] Connected to SSH server at switch.company.org, port 22
INFO:asyncssh:[conn=7]   Local address: xxx.xxx.xxx.xxx, port 58058
INFO:asyncssh:[conn=7]   Peer address: xxx.xxx.xxx.xxx, port 22
DEBUG:asyncssh:[conn=7] Sending version SSH-2.0-AsyncSSH_2.14.2
DEBUG:asyncssh:[conn=7] Received version SSH-2.0-Cisco-1.25
DEBUG:asyncssh:[conn=7] Requesting key exchange
DEBUG:asyncssh:[conn=7]   Key exchange algs: curve25519-sha256,curve25519-sha256@libssh.org,curve448-sha512,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,ecdh-sha2-<IP_ADDRESS>.10,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group15-sha512,diffie-hellman-group16-sha512,diffie-hellman-group17-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256<EMAIL_ADDRESS>DEBUG:asyncssh:[conn=7]   Host key algs: rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-ed448-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-<IP_ADDRESS>.10-cert-v01@openssh.com,rsa-sha2-256,rsa-sha2-512,ssh-rsa-sha224@ssh.com,ssh-rsa-sha256@ssh.com,ssh-rsa-sha384@ssh.com,ssh-rsa-sha512@ssh.com,ssh-rsa,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,ssh-ed25519,ssh-ed448,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256,ecdsa-sha2-<IP_ADDRESS>.10
DEBUG:asyncssh:[conn=7]   Encryption algs: 3des-cbc,aes128-cbc,aes192-cbc,aes256-cbc,aes128-ctr,aes192-ctr,aes256-ctr
DEBUG:asyncssh:[conn=7]   MAC algs: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-sha256-2@ssh.com,hmac-sha224@ssh.com,hmac-sha256@ssh.com,hmac-sha384<EMAIL_ADDRESS>DEBUG:asyncssh:[conn=7]   Compression algs: zlib@openssh.com,none
DEBUG:asyncssh:[conn=7] Received key exchange request
DEBUG:asyncssh:[conn=7]   Key exchange algs: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha1
DEBUG:asyncssh:[conn=7]   Host key algs: rsa-sha2-512,rsa-sha2-256,ssh-rsa
DEBUG:asyncssh:[conn=7]   Client to server:
DEBUG:asyncssh:[conn=7]     Encryption algs: aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
DEBUG:asyncssh:[conn=7]     MAC algs: hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
DEBUG:asyncssh:[conn=7]     Compression algs: none
DEBUG:asyncssh:[conn=7]   Server to client:
DEBUG:asyncssh:[conn=7]     Encryption algs: aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
DEBUG:asyncssh:[conn=7]     MAC algs: hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
DEBUG:asyncssh:[conn=7]     Compression algs: none
DEBUG:asyncssh:[conn=7] Beginning key exchange
DEBUG:asyncssh:[conn=7]   Key exchange alg: ecdh-sha2-nistp521
DEBUG:asyncssh:[conn=7]   Client to server:
DEBUG:asyncssh:[conn=7]     Encryption alg: aes128-ctr
DEBUG:asyncssh:[conn=7]     MAC alg<EMAIL_ADDRESS>DEBUG:asyncssh:[conn=7]     Compression alg: none
DEBUG:asyncssh:[conn=7]   Server to client:
DEBUG:asyncssh:[conn=7]     Encryption alg: aes128-ctr
DEBUG:asyncssh:[conn=7]     MAC alg<EMAIL_ADDRESS>DEBUG:asyncssh:[conn=7]     Compression alg: none
DEBUG:asyncssh:[conn=7] Requesting service ssh-userauth
DEBUG:asyncssh:[conn=7] Completed key exchange
DEBUG:asyncssh:[conn=7] Request for service ssh-userauth accepted
INFO:asyncssh:[conn=7] Beginning auth for user user1
DEBUG:asyncssh:[conn=7] Remaining auth methods: publickey,keyboard-interactive,password
DEBUG:asyncssh:[conn=7] Preferred auth methods: publickey,keyboard-interactive,password
DEBUG:asyncssh:[conn=7] Trying public key auth with ssh-rsa key
DEBUG:asyncssh:[conn=7] Remaining auth methods: publickey,keyboard-interactive,password
DEBUG:asyncssh:[conn=7] Preferred auth methods: publickey,keyboard-interactive,password
DEBUG:asyncssh:[conn=7] Trying keyboard-interactive auth
INFO:asyncssh:[conn=7] Auth for user user1 succeeded
DEBUG:asyncssh:[conn=7, chan=0] Set write buffer limits: low-water=16384, high-water=65536
INFO:asyncssh:[conn=7, chan=0] Requesting new SSH session
DEBUG:asyncssh:[conn=7, chan=0]   Initial recv window 2097152, packet size 32768
DEBUG:asyncssh:[conn=7, chan=0]   Initial send window 8192, packet size 4096
DEBUG:asyncssh:[conn=7, chan=0]   Terminal type: xterm
INFO:asyncssh:[conn=7, chan=0]   Interactive shell requested
DEBUG:scrapli.transport:transport connection to 'switch.company.org' on port '22' opened successfully
DEBUG:asyncssh:[conn=7, chan=0] Reading from channel started
DEBUG:scrapli.channel:write: '\n'
DEBUG:asyncssh:[conn=7, chan=0] Sending 1 data byte
INFO:asyncssh:[conn=7, chan=0] Received exit status 0
DEBUG:asyncssh:[conn=7, chan=0] Received EOF
INFO:asyncssh:[conn=7, chan=0] Received channel close
INFO:asyncssh:[conn=7, chan=0] Channel closed
INFO:asyncssh:[conn=7] Connection lost
DEBUG:scrapli.channel:read: b''
DEBUG:scrapli.channel:read: b''
DEBUG:scrapli.channel:read: b''
DEBUG:scrapli.channel:read: b''
DEBUG:scrapli.channel:read: b''
DEBUG:scrapli.channel:read: b''
DEBUG:scrapli.channel:read: b''
DEBUG:scrapli.channel:read: b''
DEBUG:scrapli.channel:read: b''
DEBUG:scrapli.channel:read: b''
DEBUG:scrapli.channel:read: b''
DEBUG:scrapli.channel:read: b''
DEBUG:scrapli.channel:read: b''
DEBUG:scrapli.channel:read: b''
# ...
# loops forever....
# ...
DEBUG:scrapli.channel:read: b''
DEBUG:scrapli.channel:read: b''
DEBUG:scrapli.channel:read: b''
DEBUG:scrapli.channel:read: b''
DEBUG:scrapli.channel:read: b''
DEBUG:scrapli.channel:read: b''
DEBUG:scrapli.channel:read: b''
DEBUG:scrapli.channel:read: b''
DEBUG:scrapli.channel:read: b''
DEBUG:scrapli.channel:read: b''

KeyboardInterrupt
Traceback (most recent call last):
  File "/home/user1/dev/play.py", line 68, in <module>
    time.sleep(5)
KeyboardInterrupt
ERROR:asyncio:Task exception was never retrieved
future: <Task finished name='Task-232' coro=<AsyncChannel.get_prompt() done, defined at /home/user1/dev/venv/lib/python3.10/site-packages/scrapli/channel/async_channel.py:423> exception=KeyboardInterrupt()>
Traceback (most recent call last):
  File "/home/user1/dev/play.py", line 63, in <module>
    asyncio.run(async_main())
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 636, in run_until_complete
    self.run_forever()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
    self._run_once()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
    handle._run()
  File "/usr/lib/python3.10/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/home/user1/dev/venv/lib/python3.10/site-packages/scrapli/channel/async_channel.py", line 448, in get_prompt
    buf += await self.read()
  File "/home/user1/dev/venv/lib/python3.10/site-packages/scrapli/channel/async_channel.py", line 73, in read
    self.logger.debug(f"read: {buf!r}")
  File "/usr/lib/python3.10/logging/__init__.py", line 1835, in debug
    self.log(DEBUG, msg, *args, **kwargs)
  File "/usr/lib/python3.10/logging/__init__.py", line 1879, in log
    self.logger.log(level, msg, *args, **kwargs)
  File "/usr/lib/python3.10/logging/__init__.py", line 1547, in log
    self._log(level, msg, args, **kwargs)
  File "/usr/lib/python3.10/logging/__init__.py", line 1624, in _log
    self.handle(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 1634, in handle
    self.callHandlers(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 1696, in callHandlers
    hdlr.handle(record)
KeyboardInterrupt

Please note I have not been able to reproduce the issue using the default (Synchronous?) driver.
device = {
    "host": "device.dns.name",
    "auth_username": os.getenv("USER", None),
    "auth_password": os.getenv("PASSW", None),
    "auth_strict_key": False,
    "platform": "cisco_iosxe",
    "ssh_config_file": "ssh_config.conf",
}

def main():
    conn = Scrapli(**device)
    conn.open()
    print(conn.get_prompt())

scrapli==2024.1.30
asyncssh==2.14.2
Python 3.10.12

I updated to python  3.12.2 and the issue remained.
I believe what is happening is the underlying channel is closed but still able to be read from so doesn't cause an exception. It goes into an infinite loop inside AsyncChannel.read() overwhelming the async event loop and not allowing the timeout to trigger. Putting a small async sleep in AsyncChannel.read() allows the event loop to breath and the timeout triggers. This has effectively solved my issue by allowing the timeout_wrapper decorator to do its job. I'm not too knowledgeable about the inner workings asyncio.wait_for. I have a commit with the change https://github.com/forrejam/scrapli/commit/3dc5db7fa60f587e350081826d2215ee3a98f289 . It seems quite dirty but let me know if you have any ideas. I have tested multiple times now and it seems to break out of the infinite read loop correctly now after the default 30 seconds timeout.

ah interesting! thanks a bunch for all the effort digging into this @forrejam -- much appreciated!
if its not too difficult, could you try to add this line print(f"at eof? -> { self.stdout.at_eof()}") here?
from the log output (very helpful, thank you!) it seems like asyncssh understands we got the EOF, but because that never raised an exception to us we are not aware of that. so, just quickly looking at what asyncssh public methods are available on the stdout object we read from, it looks like this eof check may be helpful and a quick/easy way ensure we didn't hit an eof which, if it works, we can then raise an exception there and we dont need to have any sleeps or anything.
what do you think?

Thanks @carlmontanari, yes very interesting... I will give this a go on Monday.

at eof? -> True
DEBUG:scrapli.channel:read: b''
at eof? -> True
DEBUG:scrapli.channel:read: b''
at eof? -> True
DEBUG:scrapli.channel:read: b''
at eof? -> True
DEBUG:scrapli.channel:read: b''
at eof? -> True
DEBUG:scrapli.channel:read: b''
at eof? -> True
DEBUG:scrapli.channel:read: b''

Yes indeed stdout is at EOF!


the performace without deformable conv?

can you provide the performace without deformable conv?

Moreover,
I test your model with normal convolution,100ms on titanxp
is the runtime right?

You can refer to the Table 1 in our paper for the performance without deformable conv.
May I know your test settings (e.g., resolution)? Our inference time is measured by inference.py  with --count_time activated.

Thanks
360*1280
I do not use the  deformable conv.

Are you using our code inference.py?

no

why do not eval on the 1/8 resolution with stereonet config?
It would be faseter

There is a trade-off between accuracy and speed. If speed is a priority, you can use the StereoNet-AA model.

If anyone test the inference time on titanxp,
tell me,
thanks

We have uploaded our timing scripts, you should use them to measure the inference time.

118.24091136 gflops 3.175282 M
double check parameter 3.175282 M
time<PHONE_NUMBER>98 ms
input 360*1260
device titian xp
I think v100 may 1.5 times faster than titianxp, it is normal

118.24091136 gflops 3.175282 M
double check parameter 3.175282 M
time<PHONE_NUMBER>98 ms
input 360*1260
device titian xp
I think v100 may 1.5 times faster than titianxp, it is normal

您好，能否支援一份不带dconv的代码？

can you share me the code without   @@deformable

can you provide the performace without deformable conv?

can you share me the code without deformable,thank you! @meteorshowers

can you share me the code without   @@deformable

can you provide the performace without deformable conv?

can you share me the code without deformable,thank you! @meteorshowers


Reduce PDH overhead in go core checks

What does this PR do?

Reorganizes the go PDH library so that each check can use a single PDH query for all counters. This reduces PDH overhead by reducing the number of query handles as well as the number of calls to PdhCollectQueryData.
Also fixes a PDH query handle leak that occurred in the Get*Counter() constructors if the query was succesfully created by Initialize() but then adding the counter to the query fails.
Motivation

PDH Query handle leak
https://datadoghq.atlassian.net/browse/WA-173
Reduce PdhCollectQueryData calls
https://datadoghq.atlassian.net/browse/WA-133
Additional Notes

Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

ensure that the metrics supplied by the checks are available.
Reviewer's Checklist

[ ] If known, an appropriate milestone has been selected; otherwise the Triage milestone is set.
[ ] Use the major_change label if your change either has a major impact on the code base, is impacting multiple teams or is changing important well-established internals of the Agent. This label will be use during QA to make sure each team pay extra attention to the changed behavior. For any customer facing change use a releasenote.
[ ] A release note has been added or the changelog/no-changelog label has been applied.
[ ] Changed code has automated tests for its functionality.
[ ] Adequate QA/testing plan information is provided if the qa/skip-qa label is not applied.
[ ] At least one team/.. label has been applied, indicating the team(s) that should QA this change.
[ ] If applicable, docs team has been notified or an issue has been opened on the documentation repo.
[ ] If applicable, the need-change/operator and need-change/helm labels have been applied.
[ ] If applicable, the k8s/<min-version> label, indicating the lowest Kubernetes version compatible with this feature.
[ ] If applicable, the config template has been updated.

rebasing to resolve conflicts


Introducing quasi-dictatorship

So here's an interesting idea. Currently everyone's vote carries equal weight, but what if this wasn't the case?
Proposition: Each time a PR is merged into the codebase, a new "dictator" is chosen. This dictator's vote will carry more weight (2x, 5x, 10x, etc) for the duration until a new pull request is merged in. Any PR that the dictator voted on during their reign will keep their weighted vote. So if a dictator voted on a PR, and a new dictator voted on the same PR, there would be 2 weighted votes for that PR.
Alternatively, we could introduce the concept of the peasant 😂, which would be the opposite of a dictator (.5x, .2x, .1x) or both!
Thoughts?

How would the dictator be chosen? Why?

My thoughts were the server would randomly choose a person from either people that have voted, submitted PRs, watching the repo etc.


Bump ts to 2.3, fixes

👍 lgtm

👍  on shipping 2.0.1 btw with this.

Ah, you already did it! Ace.


Add support for loading nodes from Git repositories

Add support for loading nodes from Git repositories

Description
We need to implement the ability to load nodes from Git repositories, with a configurable default option pointing to our meng.nodes repo. This will enhance the flexibility and extensibility of our node system.
Goals

Allow nodes to be pulled from specified Git repositories at startup.
Implement a configuration option to set the default repository URL.
Ensure seamless integration with the existing node loading mechanism.

Implementation Details

Git Integration:

Implement a Git client or use an existing library (e.g., JGit) to clone/pull repositories.
Handle repository authentication if required.

Configuration:

Add a new configuration option node_repository_url with a default value pointing to our meng.nodes repo.
Allow users to specify multiple repository URLs in the configuration.

Node Loading Process:

Modify the Schemas class to incorporate the new loading mechanism.
Implement a method to clone/pull the repository at startup.
Parse and load node definitions from the cloned repository.

Error Handling:

Implement robust error handling for network issues, invalid repos, etc.
Provide clear logging and user feedback for any loading issues.

Caching:

Implement a caching mechanism to avoid unnecessary network requests on every startup.
Add a force refresh option for users to manually update nodes.

Proposed Changes

Update Schemas.kt:
class Schemas(
    private val path: Path,
    private val side: Endpoint.Side,
    private val nodeRepoUrl: String = DEFAULT_NODE_REPO_URL
) : Listener {
    // ... existing code ...

    private fun loadNodesFromRepo() {
        // Implement Git cloning/pulling logic here
        // Parse and load nodes from the cloned repo
    }

    override fun onInstall() {
        if (side == Endpoint.Side.CLIENT) return
        loadNodesFromRepo()
        library.readFrom(path)
        // ... rest of the existing code ...
    }

    // ... rest of the class ...
}

Update Bootstrap.kt:
private fun onCommonSetup(event: FMLCommonSetupEvent) {
    // ... existing code ...

    val nodeRepoUrl = config.getString("node_repository_url", DEFAULT_NODE_REPO_URL)
    Server
        .install<ServerRuntime>()
        .install<Schemas>(schemasPath, Endpoint.Side.SERVER, nodeRepoUrl)
        // ... rest of the installations ...
}

Testing

Verify that nodes are correctly loaded from the specified Git repository.
Test with various repository URLs, including invalid ones, to ensure proper error handling.
Check that the caching mechanism works as expected.

Additional Considerations

Consider implementing a version check to only update when new versions are available.
Explore the possibility of hot-reloading nodes without requiring a full restart.

Implemented a while ago


udpate pydantic to 2.0

This PR does:

update pydantic requirement from ==1.10.* to ~=2.0: Looking at the usage of pydantic in this project, I don't think we have to be really narrow in the pin and using any 2.* version should be good.

FYI: This is blocking titiler-pgstac update

closing in favour of #199


Complex SPDX Expressions

I noticed that for the encoding_rs dependency resulting licenses are weired.
Running the following script:
#!/bin/bash

cargo init example
pushd example

echo -e "encoding_rs = '0.8.31'\n" >> Cargo.toml
cargo bundle-licenses --format toml --output THIRDPARTY.toml
cat THIRDPARTY.toml
popd

I get the following output:

Created binary (application) package
~/Git/tmp/bundle/example ~/Git/tmp/bundle
[2022-04-30T19:14:37Z WARN  bundle_licenses_lib::found_license] No license found for (Apache-2.0 license in encoding_rs:0.8.31 - /home/bennet/.cargo/registry/src/github.com-1ecc6299db9ec823/encoding_rs-0.8.31/Cargo.toml
[2022-04-30T19:14:37Z WARN  bundle_licenses_lib::found_license] No license found for MIT) AND BSD-3-Clause license in encoding_rs:0.8.31 - /home/bennet/.cargo/registry/src/github.com-1ecc6299db9ec823/encoding_rs-0.8.31/Cargo.toml
root_name = "example"

[[third_party_libraries]]
package_name = "cfg-if"
package_version = "1.0.0"
license = "MIT / Apache-2.0"

[[third_party_libraries.licenses]]
license = "MIT"
text = "Copyright (c) 2014 Alex Crichton\n\nPermission is hereby granted, free of charge, to any\nperson obtaining a copy of this software and associated\ndocumentation files (the \"Software\"), to deal in the\nSoftware without restriction, including without\nlimitation the rights to use, copy, modify, merge,\npublish, distribute, sublicense, and/or sell copies of\nthe Software, and to permit persons to whom the Software\nis furnished to do so, subject to the following\nconditions:\n\nThe above copyright notice and this permission notice\nshall be included in all copies or substantial portions\nof the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF\nANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED\nTO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A\nPARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT\nSHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY\nCLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION\nOF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR\nIN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER\nDEALINGS IN THE SOFTWARE.\n"

[[third_party_libraries.licenses]]
license = "Apache-2.0"
text = "                              Apache License\n                        Version 2.0, January 2004\n                     http://www.apache.org/licenses/\n\nTERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n1. Definitions.\n\n   \"License\" shall mean the terms and conditions for use, reproduction,\n   and distribution as defined by Sections 1 through 9 of this document.\n\n   \"Licensor\" shall mean the copyright owner or entity authorized by\n   the copyright owner that is granting the License.\n\n   \"Legal Entity\" shall mean the union of the acting entity and all\n   other entities that control, are controlled by, or are under common\n   control with that entity. For the purposes of this definition,\n   \"control\" means (i) the power, direct or indirect, to cause the\n   direction or management of such entity, whether by contract or\n   otherwise, or (ii) ownership of fifty percent (50%) or more of the\n   outstanding shares, or (iii) beneficial ownership of such entity.\n\n   \"You\" (or \"Your\") shall mean an individual or Legal Entity\n   exercising permissions granted by this License.\n\n   \"Source\" form shall mean the preferred form for making modifications,\n   including but not limited to software source code, documentation\n   source, and configuration files.\n\n   \"Object\" form shall mean any form resulting from mechanical\n   transformation or translation of a Source form, including but\n   not limited to compiled object code, generated documentation,\n   and conversions to other media types.\n\n   \"Work\" shall mean the work of authorship, whether in Source or\n   Object form, made available under the License, as indicated by a\n   copyright notice that is included in or attached to the work\n   (an example is provided in the Appendix below).\n\n   \"Derivative Works\" shall mean any work, whether in Source or Object\n   form, that is based on (or derived from) the Work and for which the\n   editorial revisions, annotations, elaborations, or other modifications\n   represent, as a whole, an original work of authorship. For the purposes\n   of this License, Derivative Works shall not include works that remain\n   separable from, or merely link (or bind by name) to the interfaces of,\n   the Work and Derivative Works thereof.\n\n   \"Contribution\" shall mean any work of authorship, including\n   the original version of the Work and any modifications or additions\n   to that Work or Derivative Works thereof, that is intentionally\n   submitted to Licensor for inclusion in the Work by the copyright owner\n   or by an individual or Legal Entity authorized to submit on behalf of\n   the copyright owner. For the purposes of this definition, \"submitted\"\n   means any form of electronic, verbal, or written communication sent\n   to the Licensor or its representatives, including but not limited to\n   communication on electronic mailing lists, source code control systems,\n   and issue tracking systems that are managed by, or on behalf of, the\n   Licensor for the purpose of discussing and improving the Work, but\n   excluding communication that is conspicuously marked or otherwise\n   designated in writing by the copyright owner as \"Not a Contribution.\"\n\n   \"Contributor\" shall mean Licensor and any individual or Legal Entity\n   on behalf of whom a Contribution has been received by Licensor and\n   subsequently incorporated within the Work.\n\n2. Grant of Copyright License. Subject to the terms and conditions of\n   this License, each Contributor hereby grants to You a perpetual,\n   worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n   copyright license to reproduce, prepare Derivative Works of,\n   publicly display, publicly perform, sublicense, and distribute the\n   Work and such Derivative Works in Source or Object form.\n\n3. Grant of Patent License. Subject to the terms and conditions of\n   this License, each Contributor hereby grants to You a perpetual,\n   worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n   (except as stated in this section) patent license to make, have made,\n   use, offer to sell, sell, import, and otherwise transfer the Work,\n   where such license applies only to those patent claims licensable\n   by such Contributor that are necessarily infringed by their\n   Contribution(s) alone or by combination of their Contribution(s)\n   with the Work to which such Contribution(s) was submitted. If You\n   institute patent litigation against any entity (including a\n   cross-claim or counterclaim in a lawsuit) alleging that the Work\n   or a Contribution incorporated within the Work constitutes direct\n   or contributory patent infringement, then any patent licenses\n   granted to You under this License for that Work shall terminate\n   as of the date such litigation is filed.\n\n4. Redistribution. You may reproduce and distribute copies of the\n   Work or Derivative Works thereof in any medium, with or without\n   modifications, and in Source or Object form, provided that You\n   meet the following conditions:\n\n   (a) You must give any other recipients of the Work or\n       Derivative Works a copy of this License; and\n\n   (b) You must cause any modified files to carry prominent notices\n       stating that You changed the files; and\n\n   (c) You must retain, in the Source form of any Derivative Works\n       that You distribute, all copyright, patent, trademark, and\n       attribution notices from the Source form of the Work,\n       excluding those notices that do not pertain to any part of\n       the Derivative Works; and\n\n   (d) If the Work includes a \"NOTICE\" text file as part of its\n       distribution, then any Derivative Works that You distribute must\n       include a readable copy of the attribution notices contained\n       within such NOTICE file, excluding those notices that do not\n       pertain to any part of the Derivative Works, in at least one\n       of the following places: within a NOTICE text file distributed\n       as part of the Derivative Works; within the Source form or\n       documentation, if provided along with the Derivative Works; or,\n       within a display generated by the Derivative Works, if and\n       wherever such third-party notices normally appear. The contents\n       of the NOTICE file are for informational purposes only and\n       do not modify the License. You may add Your own attribution\n       notices within Derivative Works that You distribute, alongside\n       or as an addendum to the NOTICE text from the Work, provided\n       that such additional attribution notices cannot be construed\n       as modifying the License.\n\n   You may add Your own copyright statement to Your modifications and\n   may provide additional or different license terms and conditions\n   for use, reproduction, or distribution of Your modifications, or\n   for any such Derivative Works as a whole, provided Your use,\n   reproduction, and distribution of the Work otherwise complies with\n   the conditions stated in this License.\n\n5. Submission of Contributions. Unless You explicitly state otherwise,\n   any Contribution intentionally submitted for inclusion in the Work\n   by You to the Licensor shall be under the terms and conditions of\n   this License, without any additional terms or conditions.\n   Notwithstanding the above, nothing herein shall supersede or modify\n   the terms of any separate license agreement you may have executed\n   with Licensor regarding such Contributions.\n\n6. Trademarks. This License does not grant permission to use the trade\n   names, trademarks, service marks, or product names of the Licensor,\n   except as required for reasonable and customary use in describing the\n   origin of the Work and reproducing the content of the NOTICE file.\n\n7. Disclaimer of Warranty. Unless required by applicable law or\n   agreed to in writing, Licensor provides the Work (and each\n   Contributor provides its Contributions) on an \"AS IS\" BASIS,\n   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n   implied, including, without limitation, any warranties or conditions\n   of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n   PARTICULAR PURPOSE. You are solely responsible for determining the\n   appropriateness of using or redistributing the Work and assume any\n   risks associated with Your exercise of permissions under this License.\n\n8. Limitation of Liability. In no event and under no legal theory,\n   whether in tort (including negligence), contract, or otherwise,\n   unless required by applicable law (such as deliberate and grossly\n   negligent acts) or agreed to in writing, shall any Contributor be\n   liable to You for damages, including any direct, indirect, special,\n   incidental, or consequential damages of any character arising as a\n   result of this License or out of the use or inability to use the\n   Work (including but not limited to damages for loss of goodwill,\n   work stoppage, computer failure or malfunction, or any and all\n   other commercial damages or losses), even if such Contributor\n   has been advised of the possibility of such damages.\n\n9. Accepting Warranty or Additional Liability. While redistributing\n   the Work or Derivative Works thereof, You may choose to offer,\n   and charge a fee for, acceptance of support, warranty, indemnity,\n   or other liability obligations and/or rights consistent with this\n   License. However, in accepting such obligations, You may act only\n   on Your own behalf and on Your sole responsibility, not on behalf\n   of any other Contributor, and only if You agree to indemnify,\n   defend, and hold each Contributor harmless for any liability\n   incurred by, or claims asserted against, such Contributor by reason\n   of your accepting any such warranty or additional liability.\n\nEND OF TERMS AND CONDITIONS\n\nAPPENDIX: How to apply the Apache License to your work.\n\n   To apply the Apache License to your work, attach the following\n   boilerplate notice, with the fields enclosed by brackets \"[]\"\n   replaced with your own identifying information. (Don't include\n   the brackets!)  The text should be enclosed in the appropriate\n   comment syntax for the file format. We also recommend that a\n   file or class name and description of purpose be included on the\n   same \"printed page\" as the copyright notice for easier\n   identification within third-party archives.\n\nCopyright [yyyy] [name of copyright owner]\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n\thttp://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n"

[[third_party_libraries]]
package_name = "encoding_rs"
package_version = "0.8.31"
license = "(Apache-2.0 / MIT) AND BSD-3-Clause"

[[third_party_libraries.licenses]]
license = "(Apache-2.0"
text = "NOT FOUND"

[[third_party_libraries.licenses]]
license = "MIT) AND BSD-3-Clause"
text = "NOT FOUND"
~/Git/tmp/bundle

the weird part about this is the two licenses (Apache-2.0, and MIT) AND BSD-3-Clause.
Instead I would have expected three licenses Apache-2.0, MIT, and BSD-3-Clause.
Additionally, it appears OR is normalised to / which I find peculiar as a footnote in the Cargo Manifest Documentation
states that  Previously multiple licenses could be separated with a /, but that usage is deprecated, so I would have rather expected a normalisation to OR or no normalisation at all.

With #11 merged this should now be resolved.


Implement UDP ingress for clouds

We don't currently implement inbound UDP traffic through the AWS and Azure load balancers.  The thought at the time was that this didn't make sense because Istio doesn't route UDP at this time.
After thinking about it some more, this does actually make sense because users could simply route UDP via regular node ports instead.
This change should be easy.

It looks like Envoy still doesn't support UDP, but they say it may be coming in 2023:
https://github.com/istio/istio/issues/1430#issuecomment-1279257188
https://github.com/envoyproxy/envoy/issues/23564


add hours command

add hours command to display the current hours for the day.
will need help from @dt-ucsd to get this JSON feed

From discussion with @dt-ucsd

Hours data for each building can be accessed in JSON format https://library.ucsd.edu/hours/data/2017/fall/audreys-cafe.html

Main Hours JS script https://library.ucsd.edu/assets/hours/hours.js

Will likely need to copy over the quarter_data array so we can identify the correct URL pattern to use to get JSON data via [1].


Support pip installation and show the working pip command in the Readme

This would be very useful for https://github.com/deepmind/meltingpot as well, so that it can also be provided as a package

We have a build rule that produces a wheel file; you can install that with pip install. Do you think that needs additional documentation?

@tkoeppe Depends on how much you care about ease of use. Ideally, you'd regularly update the wheel file to PyPI so people could install lab2d normally like any other Python package rather than have to build a wheel. If this isn't a high enough priority for you, then documentation for making the wheel would be good.

I'm not sure we would want a new wheel on PyPI for every single commit here. And for the tagged releases we already build wheels automatically, so you can just download one from the releases page. So I suppose the ability to build a wheel file isn't actually all that relevant to most users.

Not sure where the idea of a new wheel on PyPI for every commit came from. Think of any normal Python package you install, like numpy or requests. They have a process for uploading their wheel to PyPI so you could install them using pip install numpy requests. They certainly don't upload every commit to PyPI.
Providing the wheel on the releases page is better than not providing it at all. It still makes it difficult for other package maintaners to depend on lab2d, because Pip's automatic dependency resolution can't fetch the wheel from your releases page. Therefore, every package that uses lab2d has to tell its users to go and find that wheel manually.
However, I understand that the standard for user friendliness for research-related packages is lower than most of the open-source world, and I understand if this isn't a priority for you.

However, I understand that the standard for user friendliness for research-related packages is lower than most of the open-source world, and I understand if this isn't a priority for you.

Could we please have this conversation without the passive-aggressive tone? This seems neither necessary nor helpful.

Oh, I'm really sorry, no passive-agressive tone was intended here! I really meant what I said, and let me explain.
Most of the code that I see in papers, from various instituions, has a low standard for user friendliness. I'm an engineer with lots of open-source experience, making my first steps in the research world, and I noticed that the standard for how easy the code is to work with is lower in the research world. Many times when I see code that's linked from a research paper, there are no instructions on how to run it, and when I try to run it, there are exceptions, then I have to report them and do back-and-forths with the maintainers, if they even have time... This is frustrating, but I accepted that this is how it is. It's not a priority for researchers to create a very friendly software package, they are judged based on the papers they write.
The fact your package even has the wheels available, already makes it more friendly than the average piece of research code. I'm suggesting how to take it to the next level to be on par with such packages like numpy and requests. However, I said what I said to acknowledge that this might not be a priority for you, and I think it's valuable to set these stretch goals in case you, or anyone else in the future would like to improve the user experience of this package.
Sorry for any unpleasantness caused.

Thanks, no worries, I appreciate it!
I suppose there's a question of what the appropriate "standard" is here. We definitely don't have regularly staffed support for lab2d, and the contributors chip in only when they happen to have some time or when there's some particular reason. So  don't want to create the impression that this is a regularly maintained package, and I don't know if such expectations would be attached to a PyPI entry. @jagapiou has been looking into this, though, and I'm certainly not opposed. I merely thought that what we're offering is already fairly close to that (just one additional download step before you call pip) that it wasn't a big problem.
I understand the benefit of automated dependency management, but do consider that in two years' time we might not have released a new wheel and your base glibc dependencies have moved on and now the wheel on PyPI is no longer compatible, then you have a whole different kind of frustration. So rather than trying to promise something that I can't be sure we can keep, I was keeping it conservative and only wanted to offer something reasonably self-contained.
But I think @jagapiou might be the best person to talk about this.

We are working on our side to have at least some versions available in pypi so it can be pip installed. Stay tuned

Heya, yep I'm working on this, so feel free to assign to me. The wheels are great but the correct wheel to install must be manually specified and can't currently be autodetected. So automatically uploading to pypi when we make a release should be easy. I'll have a PR for this very soon.

Hi all, appreciate the hard work getting this up on pypi, I'm having some installation issues with MacOS though--when I try to install via pip I get this error:
pip install dmlab2d
ERROR: Could not find a version that satisfies the requirement dmlab2d (from versions: none)
ERROR: No matching distribution found for dmlab2d

I tried downloading the wheel manually but ran into errors with that as well:
ERROR: dmlab2d-1.0.0_dev.10-cp310-cp310-macosx_12_0_x86_64.whl is not a supported wheel on this platform.

For context about my environment:
sw_svers
ProductName:	macOS
ProductVersion:	12.4
BuildVersion:	21F79`)

python --version
Python 3.10.10

>>> import platform
>>> platform.architecture()
('64bit', '')
>>> platform.mac_ver()
('10.16', ('', '', ''), 'x86_64')

Maybe my python thinks my mac version is different than it actually is? Usually wheels are for any mac version so haven't had this issue but it could very well be a personal issue with my environment.

@jagapiou Wasn't there something about needing some environment variable on macOS to disable some kind of version checking?

Try setting SYSTEM_VERSION_COMPAT=0 (see https://github.com/actions/setup-python/issues/279).

dmlab==1.0.0 is now up on PyPI. README update coming soon.

@elliottower: does setting the environment variable help?

Hey sorry I haven’t gotten a chance to check, will do later this week.
Appreciate the help.
On Tue, Jul 25, 2023 at 5:42 AM Thomas Köppe @.***>
wrote:

@elliottower https://github.com/elliottower: does setting the
environment variable help?
—
Reply to this email directly, view it on GitHub
https://github.com/deepmind/lab2d/issues/20#issuecomment-1649481608, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AHVPVA4M4X5HGNIHX2RQGULXR6IGZANCNFSM5Z3242OA
.
You are receiving this because you were mentioned.Message ID:
@.***>

Hi I'm using ubuntu linux on arm64,
I'm trying to install dmlab but it doesn't work.
error:
ERROR: Could not find a version that satisfies the requirement dmlab2d (from versions: none)
ERROR: No matching distribution found for dmlab2d

@AllenChienXXX What do you mean by "trying to install" -- do you mean with pip? (Generally, could you please give some details in questions?)
I don't think we're providing any precompiled wheels for Linux/arm64, but it should be possible to build the wheel from source. I'm not sure if we've tested that combination, but at least in principle all the necessary settings should be available (esp. in LuaJit, but in a crunch you can build with Lua 5.1).

The original issue is now fixed I believe. Please open new issues if the pip install isn't working.

I tested pip installation now and it works. Thank you!


Change in layoutType when orientation changes gives blank power BI report

we have a report with initial layout type as MobilePortrait like below.
  const [repConfig, setConfig] = useState({     type: "report",     embedUrl: undefined,     id: undefined,     hostname: "https://app.powerbi.com",     tokenType: models.TokenType.Embed,     accessToken: undefined,     settings: {       layoutType:         window.innerWidth > 768           ? models.LayoutType.Custom           : models.LayoutType.MobilePortrait,       customLayout: {         displayOption: models.DisplayOption.FitToWidth,       },       navContentPaneEnabled: false,       panes: {         filters: {           expanded: false,           visible: false,         },       },     },   });
So , as anyone can see the initial setup is MobilePortrait when window's innerwidth prop is less than 768. We want to handle the screen orientation change event and change the layout to MobileLandscape.
For that we have registered orientation change event handler in our useeffect like below
window.screen.orientation.addEventListener(         "change",         handleOrientationChange       )
in the handleOrientationChange , we try to update the layout type like below
   const newSettings = {               layoutType: models.LayoutType.MobileLandscape           };             window.report?.updateSettings(newSettings);
But this generates an empty report. The charts , graph gets removed from the report. only blank screen is visible. change in the orientation again (from landscape to portrait) brings back the original report.
Can anyone please tell what we are missing here??!! Thanks in advanced

We tried implementing the code snippet provided along with some changes which resolved the issue.
Instead of specifying the layout type condition inside the useState(), specify it inside the useEffect().
Please refer to the following code given below:
const [sampleReportConfig, setReportConfig] = useState<models.IReportEmbedConfiguration>({

        type: 'report',

        embedUrl: undefined,

        tokenType: models.TokenType.Embed,

        accessToken: undefined,

        // settings: undefined,

        settings: {

            layoutType: models.LayoutType.MobileLandscape

           },

    });

useEffect(() => {

        const handleOrientationChange = async () => {

            const newLayoutType = window.innerWidth > 768 ? models.LayoutType.MobileLandscape : models.LayoutType.MobilePortrait;

            const updatedSettings = {
                layoutType: newLayoutType
            };

            try {
                if (report && report.updateSettings) {
                    await report.updateSettings(updatedSettings);
                }
            } catch (error) {
                console.log(error);
            }
        };

        window.addEventListener('resize', handleOrientationChange);

         return () => {
            window.removeEventListener('resize', handleOrientationChange);
        };
    }, [report]);

Please let us know if you face any issue in this.


New Element: Regular Block Model

In addition to name, description, metadata, axis_u/v/w, corner, and attributes, regular block model will have:

block_size, a length-3 list of i, j, k size in doubles
block_count, a length-3 list of number of blocks in i, j, k

Attributes are in a flat array, unwrapped in i then j then k. This should be explicitly documented.

For implementation details, please refer to the implementation and accompanying discussion in the example notebook: https://github.com/gmggroup/omf/blob/691420e3761c874139ca4127abae276204045111/notebooks/omf_cbi.ipynb?short_path=1aa1177#L189


Update operator config to include selinux

Make config updates based on
https://github.com/kata-containers/kata-containers/pull/4813
Fixes: #438
Signed-off-by: stevenhorsman<EMAIL_ADDRESS>

For full transparency I'm a bit unsure on what setting we should have for disable_guest_selinux=true As it defaults to true I don't think it is harmful, but if we hit problems we might need to flip it.

@bpradipt, can you, Snir, or someone give this a test please? Our ibm cloud operator 'support' seems to have gone a bit stale, so I don't have an easy way to try it and out and don't have time to debug that right now. Thanks!

Yes, will test it out.

Hey @bpradipt - hold fire on testing this for now - I think there is also a bug in kata-containers that need fixing first, I'll keep you updated when I learn more.

The kata-containers PR I has been created https://github.com/kata-containers/kata-containers/pull/5937 - this is a pre-req for this PR as it won't work without it. Sorry for the confusion.

Now https://github.com/kata-containers/kata-containers/pull/5937 has merged I think this should be ready to tests as long as the latest kata-containers CCv0 code is used. Thanks!


custom class on X button

How to add an custom class on X button depending on the alert type?You should add that is good for css customization

You could add a class to the alert itself and use CSS to navigate down.
Example:
$.jAlert({
'class':'custom1'
});

.custom1 .ja_close
{
//custom css
}