Inclusive/Exclusive behavior for join conditions

@bcodell Great work on the package!
I have multiple activities that have an identical timestamp, which is accurate to the business process they represent. Right now the after, before, and between join conditions are all strictly exclusive, which means that I cannot query the relationships between these activities as I would want.  Particularly for the between join, I would have expected an implementation that was either fully inclusive given the way that term typically behaves in SQL or inclusive of start and exclusive of end based on what I typically do in practice to get mutually exclusive collectively exhaustive behavior. I can confirm that the macros behave as expected when I change the </> to <=/>= in the macros/activity_schema/dataset/joins/_join_clauses.sql file.
Am I missing a way to get the desired behavior? Else I'd rather not tweak my underlying data (like say aging certain activities by a ms) or maintain a fork, and would be happy to contribute a PR to get the desired result if I can get some direction on how you'd prefer to go about it. Some approaches that come to mind are:

Edit after and before to be inclusive. Adjust between to be inclusive start and maintain the existing exclusive end.
Add new join conditions with new names to reflect option 1. Something like after_inclusive or between_inclusive_start_exclusive_end
Add a new condition to the AQL syntax to allow for inclusive/exclusive specification per option 1.
Something else.

Assuming the current behavior is intentional, option 1 would be no go & a breaking change. Option 2 & 3 could be implemented such that the default behavior does not change and all new functionality is additive. Option 2 is less elegant but likely easier to implement.

Hey @wylbee - great to hear from you!
Great question - the implemented behavior is in line with the official Activity Schema spec (see the example query here for reference - it's very subtle), which was the initial guideline for this package. In other words, the current implementation is indeed a feature and not a bug. The impetus for this design comes from a couple places - one, from the simplifying assumption that a given entity can only take one action at a time, and more tactically in the case of joining a given activity to itself in any case of after, before, and between, where an inclusive timestamp evaluation for joins like "last before" and "first after" would effectively result in an activity occurrence joining to itself, which is definitely the wrong behavior.
But your point about the divergence in functionality between SQL's native between operator and the Activity Schema spec is certainly confusing, the Activity Schema spec certainly has room for evolution, and I'm generally always open to adding new functionality to the project, given the right use case and implementation. Regarding your suggestions, my preliminary instinct is to err closer to option 1 - namely keep the query behavior consistent (i.e. always inclusive or always exclusive) across join conditions within a given project for sake of reliability and simplicity, and giving the developer the ability to decide which implementation they'd prefer. I definitely don't want to support a case where the inclusivity is conditional (e.g. when the primary activity is joining to itself). I'm also hesitant to expand the join conditions (and subsequently the AQL syntax) without really understanding the use cases more thoroughly, as that approach introduces more complexity to the framework.
In light of that last point, are you able to share more context about the use cases you have that require joining distinct activities with identical timestamps? Any details about the business processes and the key activities/steps being measured would be especially helpful in figuring out the right path forward here.

@bcodell Thanks for the response, all makes sense to me! I missed the load-bearing assumption in the spec that activities cannot share a timestamp/entities are only taking one action at a time, and I've been handling the complexity you're referring to in your first full paragraph manually when querying my streams. Open to the feedback that I may need a refactor of my activities over a change to this project if that seems appropriate.
Stepping back, the issue I'm running into is that the current state implementations of after, before, and between result in unqueryable/incorrect relationships between activities if a timestamp is shared, which produces datasets that are not MECE where I would expect them to be. What I'm primarily trying to achieve is MECE behavior, which in practice means that in addition to the after, before, and between implementations I am also using joins that map to on_or_after and between_on_or_after/on_or_after_and_before.   There are four places in my current project where this crops up:

When activities happen at the same timestamp for the given entity per the operational source system.
When the operational source system does not record full timestamp grain precision (i.e. records a date rather than a timestamp).
When using an existing activity to build another activity or role-playing an activity between more than one stream (especially when the entity is not a customer + encompasses many customers).
When I'm building activities using an ELT based timestamp like ingested_at because an appropriate one does not exist in the operational source system.

It seems like the way Narrator recommends dealing with option 3 (and I would assume options 2 & 4) is to age the timestamp so that none of the timestamps line up. This approach seems brittle and involves making some data less accurate vs. the source system in a way that should be practically fine but is a theoretical red flag. I think with the right dbt tests in place this could be done in a way that is mechanically responsible, though my theoretical discomfort with the approach remains.
For use cases, an example from a loan servicing business is:

At X days delinquent, a loan is charged off. Loans can also be charged off for a variety of other reasons.
In the operational source system, a delinquency charge off is automatic and the timestamps are identical.
I have two activities- accrues_balance (effectively a snapshot of the state of the loan at midnight, including delinquency) and charges_off.
Attempting to use accrues_balance as the primary and do a between join for charge_off never joins the charge off if it is a delinquency charge off since the timestamp is shared with the primary activity.
Attempting to use accrues_balance as the primary and do an after join for charges_off performs the join one row later than I would expect (e.g. if the loan charged off on 4/18/24 I would expect it to show up on that date vs. current state which would be 4/19/24).

Based on your thoughts on the package, the two options jumping out to me are:

Refactor my project so that activities reflect the simplifying assumption that entities only take one action at a time. Use defensive dbt tests to ensure that this is true in practice + catch data that needs to be adjusted so that it doesn't invisibly drop out of AQL queries.
Add additional temporal joins for on_or_after and between_on_or_after while leaving the default functionality the same. This would also allow for other temporal joins to be added in the future at the cost of evolving the spec and added complexity.

I'm hesitant to evolve the spec in a vacuum and don't have a ton of confidence that the maintainers are responding to feedback on the main repo. I do see space to expand the temporal joins, but unless you're seeing something different I'm inclined to attempt the refactor of option 1 and see how that goes before pushing for spec changes that would drive changes to this project. Based on that context, where is your head at with this one?

@wylbee Thanks for the added context! I agree that evolving the spec here is probably the wrong path. (Aside - I've been contemplating a more generalized event modeling/querying solution, in which activity schema is simply one of many event modeling frameworks a developer can choose - also supporting other event modeling approaches including ones that have more robust MECE support - but will save that discussion for a separate forum.) As such, I'd rather start with exploring alternative modeling methods (your option 1) for the use case at hand as opposed to broadening the functionality of this package (your option 2).
On that note, I'll focus more generally on the output you're trying to produce (so as not to get too deep in the weeds in a domain where I lack familiarity). It sounds like the output you're trying to achieve is a dataset with 1 row per loan per "active" day of the loan with columns for the loan balance accrued on that day and an indicator of whether or not the loan was charged off on the given day. Is that correct?
If so, the difficulty I see here is that the telemetric nature of the accrues_balance activity is being (understandably) conflated with the date spine-like structure of a separate has_active_day activity. I've used the following approach for similar activity dynamics in different domains:

create a synthetic has_active_day activity for the entity in question (presumably loans in this case), where I derive the start date and end date for each entity with SQL logic, and assert the time for each activity occurrence to be midnight.
ensure no timestamps in other activities have a timestamp at midnight - currently I manually add a microsecond in sql if I identify a timestamp at midnight
adjust the activity definition logic for telemetry-based activities (e.g. accrues_balance) to only include occurrences when the measured attribute changes in a way that's notable to the business
build the aforementioned dataset with the has_active_day activity as the primary, then append/aggregate other activities as needed

In the event that I've totally missed the mark here, I do agree that an arbitrary time offset (i.e. Narrator's recommendation and my current implementation in my own projects) is a fairly brittle solution, and where possible I try to rethink the timestamp logic of the activities themselves in the context of their use cases for the business before pursuing the arbitrary offset approach. So, for example, maybe the timestamp of the accrues_balance activity should be the timestamp at which each loan was created/originated + n days since origination (you may already be doing this - going based off the "snapshot at midnight" comment). Anyway, let me know if my proposed approach makes sense (or if I'm entirely off base 😅).

Hey @wylbee + @bcodell - happy to add context from my personal experience (I used to work at Narrator and have implemented the activity schema for many many companies). The issue of two activities with the same timestamp is an edge case that breaks the assumptions of the existing activity schema spec. In practice, what I've found is that these scenarios are typically very rare because the activity schema is typically used to analyze behavioral activity from some user /agent, and in practice it's very unnatural for two events to occur at exactly the same timestamp down to the ms. It's only in cases where behavior is manufactured (ie. daily snapshot) or when the timestamps are not captured with sufficient granularity.
In cases of manufactured data (daily snapshot), I adjust timestamps to ensure uniqueness is maintained, same thing for timestamps without sufficient granularity. This has always been enough for our use cases.

Thanks for the input @bcodell & @brittmdavis1, extremely helpful!
@bcodell think we're aligned on approach 1, and feel good about pursuing that option. Good to close this issue as a result.
Appreciate the partnership on use case- you've got the read on the desired end state directionally right and the solution you sketched is pretty close to what I drew up as well. This particular edge case is pretty anomalous compared to other business models and industries I've worked with- the root system/business/contractual behavior reflects a number of simultaneous activities (like incrementing days past due and accruing daily interest) firing at midnight every day for all active accounts in the loan portfolio. In practice it ends up looking like a date spine and the row count for the models you sketch in your first and third bullets end up being identical, but the offset you describe in bullet two should get everything working as expected. I'm in the middle of a system migration that allows me some room to refactor, so will try a few approaches including this one and run with whatever works best!
@brittmdavis1 super appreciative to get your perspective and the confirmation that your extensive experience supports this edge case being extremely rare, usually due to manufactured/insufficiently granular data, and practically solvable with timestamp adjustment. At this juncture I feel pretty comfortable proceeding with that approach while using dbt tests to ensure that the core assumption around unique timestamps is met. Thank you!

Thanks for the expert opinion @brittmdavis1! Helpful to have confirmation based on your implementation experiences.
To recap: identical timestamps typically relate to manufactured activities and activities without sufficient timestamp granularity, and in practice, organically created behavioral data effectively never has the exact same timestamp down to the microsecond for a given user/agent.
@wylbee I'll close this issue per your comment. Thanks again for opening it! Will hopefully serve as a helpful reference for other users.


Inconsistency in system requirements

Is there an existing issue for this?

[X] I have searched the existing issues
[X] I have read the guide to filing a bug

Steps to reproduce
Macbook Air (2014)
System requirements states that Operating Systems: macOS, version 10.14 (Mojave) or later. but flutter doctor states that requires Xcode 14 or higher which I can't install on Big Sur because Xcode 13 is the highest version I am able to install.
Actual results
Can't run any Flutter project on simulator
Logs

Logs

<!-- Paste your logs here -->

Flutter Doctor output

Doctor output

<!-- Paste your output here -->

Hi @troncomputers. Thanks for filing this. Can you provide more information on this, with Xcode 13 on Big Sur, are you able to install flutter?
Also, can you share the logs after installing flutter and running flutter doctor -v?

I am able to run Flutter project as MacOS application (window). I am not able to connect to iOS emulator with Visual Studio Code. On Stackoverflow someone wrote that I should check if everything is installed with flutter doctor command.
[✓] Flutter (Channel stable, 3.13.2, on macOS 11.7.9 20G1426 darwin-x64, locale pl-PL)
    • Flutter version 3.13.2 on channel stable at /Users/<user_name>/development/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision ff5b5b5fa6 (13 days ago), 2023-08-24 08:12:28 -0500
    • Engine revision b20183e040
    • Dart version 3.1.0
    • DevTools version 2.25.0

[✗] Android toolchain - develop for Android devices
    ✗ Unable to locate Android SDK.
      Install Android Studio from: https://developer.android.com/studio/index.html
      On first launch it will assist you in installing the Android SDK components.
      (or visit https://flutter.dev/docs/get-started/install/macos#android-setup for detailed instructions).
      If the Android SDK has been installed to a custom location, please use
      `flutter config --android-sdk` to update to that location.

[!] Xcode - develop for iOS and macOS (Xcode 13.0)
    • Xcode at /Users/<user_name>/Downloads/Xcode 2.app/Contents/Developer
    • Build 13A233
    ✗ Flutter requires Xcode 14 or higher.
      Download the latest version or update via the Mac App Store.
    ✗ CocoaPods not installed.
        CocoaPods is used to retrieve the iOS and macOS platform side's plugin code that responds to your plugin usage on the Dart side.
        Without CocoaPods, plugins will not work on iOS or macOS.
        For more info, see https://flutter.dev/platform-plugins
      To install see https://guides.cocoapods.org/using/getting-started.html#installation for instructions.

[✗] Chrome - develop for the web (Cannot find Chrome executable at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome)
    ! Cannot find Chrome. Try setting CHROME_EXECUTABLE to a Chrome executable.

[!] Android Studio (not installed)
    • Android Studio not found; download from https://developer.android.com/studio/index.html
      (or visit https://flutter.dev/docs/get-started/install/macos#android-setup for detailed instructions).

[✓] VS Code (version 1.81.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.72.0

[✓] Connected device (1 available)
    • macOS (desktop) • macos • darwin-x64 • macOS 11.7.9 20G1426 darwin-x64

[✓] Network resources
    • All expected network resources are available.

! Doctor found issues in 4 categories.

Hi @troncomputers. Thanks for explaining further. As regards this warning message from flutter doctor Flutter requires Xcode 14 or higher, this is an intentional change/update and is included in the 3.13.0 release note.
Secondly, about this issue you mentioned:

I am able to run Flutter project as MacOS application (window). I am not able to connect to iOS emulator with Visual Studio Code.

This is a tool issue which is similar to a currently open issue https://github.com/flutter/flutter/issues/104155
I'd be closing this as a duplicate of the already open issue.
Thank you!


Follow redirects

Is your feature request related to a problem? Please describe.
Either get or downloadFile both should follow redirects until the final endpoint is reached.
Describe the solution you'd like
If response 301/302 request again until final resource is reached.
Describe alternatives you've considered
None
Additional context
For example download a file from https://source.unsplash.com/150x150/weekly/?nature

Ah seems it just errors because the file already exists. But it does follow the redirect


Hide back buttons

Task/Issue URL: #628
Tech Design URL:
CC:
Description:
In this PR, one of the back button on each sub page are hidden. These buttons are set to be hidden but not removed because they are being used in UI test. As a result, user can still click the left area to go back to the overview of the privacy protection from sub pages.
Steps to test this PR:

Open the privacy protection
Navigate to each of the sub page

Only one back button per page is visible.
Device Testing:

[x] iPhone XR

OS Testing:

[ ] iOS 10
[ ] iOS 11
[ ] iOS 12
[x] iOS 13

Internal references:
Software Engineering Expectations
Technical Design Template

Hi, thanks for this.  Unfortunately the buttons are required for iPad and this PR hides them there.  If you can fix that as well, then we can accept this PR.

Thanks!

Updated. Now we only hide the button if not an iPad.

Thanks for the update.  We're tracking this internally at https://app.asana.com/0/414235014887631/1174200812833794
I've just reviewed it and it looks good, thanks for the contribution.


Add passenger-kilometre and ton-kilometre #1272

Summary of the discussion
Add new classes:
transport performance value: A transport performance value is a quantity value that indicates the performance of a transport process in terms of its mileage and amount of transported people and/or goods.
Axiom: 'has unit' some 'transport performance unit'
transport performance unit: A transport performance unit is a unit of measurement for the accumulated transport distance of a number of people and/or amount of goods.
passenger-kilometre: Passenger-kilometre is a transport performance unit for the accumulated transport distance of people where one passenger-kilometre equals the transport distance of 1 km for one person.
ton-kilometre: Ton-kilometre is a transport performance unit for the accumulated transport distance of goods where one ton-kilometre equals the transport distance of 1 km for one ton of goods.
Add new axiom:
'transport' 'has quantity value' some 'transport performance value'
Workflow checklist
Automation
Closes #1272
PR-Assignee

[x] 🐙 Follow the Pull Request Workflow
[x] 📝 Update the CHANGELOG.md
[x] 📙 Add #'s to term tracker item

Reviewer

[ ] 🐙 Follow the Reviewer Guide
[ ] 🐙 Provided feedback and show sufficient appreciation for the work done

Thanks for your review!


Incorporate match consolidation into search

TODO:
Update dream test ground truth
Delete preliminary match file after consolidation

Codecov Report
Patch and project coverage have no change.

Comparison is base (9dc6d47) 65.06% compared to head (f418fc2) 65.06%.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@           Coverage Diff           @@
##             main      #95   +/-   ##
=======================================
  Coverage   65.06%   65.06%
=======================================
  Files           3        3
  Lines          83       83
=======================================
  Hits           54       54
  Misses         29       29

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


rename: instrumentation-tests.t -> instrumentation-tests

This looks accidental to me.

Indeed, I think this started as something with a single run.t and then grew to multiple .t files while forgetting to rename the surrounding directory...
I remember trying to follow the documentation at https://dune.readthedocs.io/en/stable/tests.html#directory-tests ("The name of the test directory must end with .t [...]") and studying dune's internal test suite back then.
Reading that link again it may actually be good to document the "multiple .t file mode" :slightly_smiling_face:

...and thanks for noticing BTW! :smiley:

Yeah, that chapter in the dune doc could use a rewrite. FWIW cram tests as files are explained above and cram tests where you want "starter files" are the directory tests they mention.


Crash on non squared configuration

Hi,
creating the crossword with a aXb with a!=b configuration Array2D crashes due to out of bounds into suggestCoord() method.

Hi,
Thanks. Yes, I see the app crashes. Sorry, I don't have time for resolving this right now. Maybe I will fix it in the nearest future.
Thanks.

i fixed by myself, I simply switched if i'm not wrong the code into the array2d in the subscript function fyi

Fixed. Please use 0.3.3.


parse urls

fixes https://github.com/kisphp/markdown-parser/issues/13

Current coverage is 98.03%

Merging #16 into master will increase coverage by +0.07% as of f13ecab

@@            master     #16   diff @@
======================================
  Files           30      31     +1
  Stmts          590     610    +20
  Branches         0       0
  Methods
======================================
+ Hit            578     598    +20
  Partial          0       0
  Missed          12      12

Review entire Coverage Diff as of f13ecab

Uncovered Suggestions

+0.33% via ...ckParagraphStart.php#22...23
+0.33% via ...ks/BlockContinue.php#12...13
+0.16% via ...locks/BlockQuote.php#46...46
See 7 more...

Powered by Codecov. Updated on successful CI builds.


Patch 1

kenCode Greek translation is ready!

Hmm, github is telling me that your xml file edit cannot be committed because it has conflicts. can you please refork or something and fix that so i can check it over and commit it? thanx in advance liondani :)

Nope, same error, it says your xml file still has "conflicts that must be resolved" and cannot be committed.

Nope, same error still. Sorry liondani, I wish I knew what to do, I am no github guru tho, that's for sure

I will merge it, just a moment.

please inform me about the error if merged successful with your edit/fix

OK, @liondani how does it look now?
https://github.com/kenCode-de/smartcoins-wallet/blob/master/app/src/main/res/values-el/strings.xml


Enable Delta Write fallback tests on Databricks 12.2 [databricks]

Part of https://github.com/NVIDIA/spark-rapids/issues/8423
The Delta Write fallback tests were expecting to see ExecutedCommandExec in the final plan, but the plan in Databricks 12.2 has DataWritingCommandExec instead.
This PR updates the tests and enables them for Databricks 12.2

build

build

build


Custom cell component with input not re-rendering on search filter

Hi @uwla
I am currently experiencing an issue where the value of input fields in custom cell components is not correctly re-rendering when filtering via search.
I've created a Codesandbox example to show what I mean.
Notice that when filtering via search, the values of the input fields in column 3 don't update or re-render. The first entry has a job of 'Recreational Therapist', but when filtering for 'Corine', it still has the value of 'Recreational Therapist'.
Is this an issue with VDT or am I doing something wrong here regarding Vue?
Thanks in advance!

Hello, thanks for the question.
I could reproduce the behavior, but this is neither a issue with VueDataTable or VueJS. This is expected behavior due to how VueJS works under the hood and Vue was designed to work this way. Let me explain:

Each component has its own data, or state if you prefer. The state is internal to the component, acessed only by it.
Parent components pass properties to children components. The properties cannot be changed by the chidren, only by the parents.
If the properties are changed from the parent, the children component will be updated and re-rendered.
If the properties are not changed from the parent, things may not work as expected.
To update the properties from the children, the children must emit an event to the parent. The parent catches the emitted event and handles it, updating properties accordingly

What is happening is the following:

JobComponent receives data from VueDataTable
Input in job component is changed, which change the internal state of the component
JobComponent does not emit an event to VueDataTable, so VueDataTable also does not emit an event, called userEvent, which could be used by App to change the user array
Because the user array is not updated within App, VueDataTable's data property is the same, and the search filter is working as expected.

What you can do is something like this:
<template>
  <input type="text" :value="job" @onChange="handleChange" :placeholder="'Please enter a job'" />
</template>

<script>
export default {
  props: {
    data: {
      type: Object,
      default: () => {},
    },
  },
  computed: {
    job() {
      return this.data.job
    }
  },
  methods: {
    handleChange(e) {
      e.preventDefault();
      const value = e.target.value;
      this.$emit('userEvent', { id: data.id, job: value });
    }
  }
};
</script>

Then, in App:
<template>
<vue-data-table :params="params2" @userEvent="updateUser" />
</template>
<script>
/**/
  methods: {
    updateUser(payload) {
      const { id, job } = payload
      const index = this.data.findIndex(u => u.id === id)
      if (index === -1) throw new Error ("User not found")
      this.data.slice(index, 1, {... this.users[index], job })
    }
  }
/**/
</script>

Please, take a look at the userEvent sections on the docs. Also, take a look at src/demo/App.ts too see how the data is updated.
If you have more questions, feel free to ask.

@uwla I think you misunderstand the issue when you say

Input in job component is changed, which change the internal state of the component

According to the problem description, the job component input is not being changed.  The only input being changed is vue-data-table's search input, which filters the table, but then does not pass the correct data prop value to the column component.

@chringel21 and @Yoduh I think now I understand what the reported problem is, from your descriptions. But I cannot reproduce it. Look at the GIF below, which everything seems to be fine:

Thanks @uwla for your response!
There were three things I was doing in a different was

In my JobComponent I was using v-model to update the input field's value.
No @onChange method on my input field.
The job value was a data property, not a computed property.

After switching to :value-binding and setting the value again in the @onChange method, using event.target.value, and using a computed value, everything is working as intended.

Good it worked!
@chringel21, I would also like to point out that the search filter only searches for normal columns by default. So, by default, if you search for "Engineer", it will display nothing instead of the rows whose job columns has "Engineer" because this column uses custom component. It will only search for a column with custom component if you provide a searchFunction, which is a function that takes a row (in this case, a user) and a search query, and returns true or false if it the row should be visible:
params = {
  data: users,
  columns: [
    { key: "name" },
    { key: "email" },
    {
      key: "permissions",
      searchable: true,
      searchFunction: (search, data) => {
        search = search.toLowerCase()
        return data.permissions.some(permission =>
          permission.toLowerCase().includes(search))
      },
    }
  ]
}


GPU error

Tried to run it with GPU. got the following error. can anyone help me on this?
(Python36) D:\chess\chess-alpha-zero>python src/chess_zero/run.py self
2018-10-10<EMAIL_ADDRESS>INFO # config type: mini
Using TensorFlow backend.
2018-10-10<EMAIL_ADDRESS>DEBUG # loading model from D:\chess\chess-alpha-zero\data\model\model_best_config.json
2018-10-10 11:45:59.478648: I C:\users\nwani_bazel_nwani\mmtm6wb6\execroot\org_tensorflow\tensorflow\core\platform\cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
2018-10-10 11:45:59.695745: I C:\users\nwani_bazel_nwani\mmtm6wb6\execroot\org_tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1356] Found device 0 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:01:00.0
totalMemory: 11.00GiB freeMemory: 9.10GiB
2018-10-10 11:45:59.790370: I C:\users\nwani_bazel_nwani\mmtm6wb6\execroot\org_tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1356] Found device 1 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:02:00.0
totalMemory: 11.00GiB freeMemory: 9.10GiB
2018-10-10 11:45:59.795932: I C:\users\nwani_bazel_nwani\mmtm6wb6\execroot\org_tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1435] Adding visible gpu devices: 0, 1
2018-10-10 11:48:20.448740: I C:\users\nwani_bazel_nwani\mmtm6wb6\execroot\org_tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-10-10 11:48:20.451530: I C:\users\nwani_bazel_nwani\mmtm6wb6\execroot\org_tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:929]      0 1
2018-10-10 11:48:20.453788: I C:\users\nwani_bazel_nwani\mmtm6wb6\execroot\org_tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:942] 0:   N N
2018-10-10 11:48:20.455816: I C:\users\nwani_bazel_nwani\mmtm6wb6\execroot\org_tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:942] 1:   N N
2018-10-10 11:48:20.458363: I C:\users\nwani_bazel_nwani\mmtm6wb6\execroot\org_tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 8795 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
2018-10-10 11:48:20.834375: I C:\users\nwani_bazel_nwani\mmtm6wb6\execroot\org_tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 8795 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1080 Ti, pci bus id: 0000:02:00.0, compute capability: 6.1)
Traceback (most recent call last):
File "C:\Users\isszfm\AppData\Local\Continuum\anaconda3\envs\Python36\lib\site-packages\tensorflow\python\framework\ops.py", line 1567, in _create_c_op
c_op = c_api.TF_FinishOperation(op_desc)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Shape must be rank 1 but is rank 0 for 'input_batchnorm/cond/Reshape_4' (op: 'Reshape') with input shapes: [1,256,1,1], [].
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "src/chess_zero/run.py", line 20, in
manager.start()
File "src\chess_zero\manager.py", line 64, in start
return self_play.start(config)
File "src\chess_zero\worker\self_play.py", line 25, in start
return SelfPlayWorker(config).start()
File "src\chess_zero\worker\self_play.py", line 45, in init
self.current_model = self.load_model()
File "src\chess_zero\worker\self_play.py", line 85, in load_model
if self.config.opts.new or not load_best_model_weight(model):
File "src\chess_zero\lib\model_helper.py", line 15, in load_best_model_weight
return model.load(model.config.resource.model_best_config_path, model.config.resource.model_best_weight_path)
File "src\chess_zero\agent\model_chess.py", line 145, in load
self.model = Model.from_config(json.load(f))
File "C:\Users\isszfm\AppData\Local\Continuum\anaconda3\envs\Python36\lib\site-packages\keras\engine\network.py", line 1032, in from_config
process_node(layer, node_data)
File "C:\Users\isszfm\AppData\Local\Continuum\anaconda3\envs\Python36\lib\site-packages\keras\engine\network.py", line 991, in process_node
layer(unpack_singleton(input_tensors), **kwargs)
File "C:\Users\isszfm\AppData\Local\Continuum\anaconda3\envs\Python36\lib\site-packages\keras\engine\base_layer.py", line 457, in call
output = self.call(inputs, **kwargs)
File "C:\Users\isszfm\AppData\Local\Continuum\anaconda3\envs\Python36\lib\site-packages\keras\layers\normalization.py", line 206, in call
training=training)
File "C:\Users\isszfm\AppData\Local\Continuum\anaconda3\envs\Python36\lib\site-packages\keras\backend\tensorflow_backend.py", line 3123, in in_train_phase
x = switch(training, x, alt)
File "C:\Users\isszfm\AppData\Local\Continuum\anaconda3\envs\Python36\lib\site-packages\keras\backend\tensorflow_backend.py", line 3058, in switch
else_expression_fn)
File "C:\Users\isszfm\AppData\Local\Continuum\anaconda3\envs\Python36\lib\site-packages\tensorflow\python\util\deprecation.py", line 432, in new_func
return func(*args, **kwargs)
File "C:\Users\isszfm\AppData\Local\Continuum\anaconda3\envs\Python36\lib\site-packages\tensorflow\python\ops\control_flow_ops.py", line 2072, in cond
orig_res_f, res_f = context_f.BuildCondBranch(false_fn)
File "C:\Users\isszfm\AppData\Local\Continuum\anaconda3\envs\Python36\lib\site-packages\tensorflow\python\ops\control_flow_ops.py", line 1913, in BuildCondBranch
original_result = fn()
File "C:\Users\isszfm\AppData\Local\Continuum\anaconda3\envs\Python36\lib\site-packages\keras\layers\normalization.py", line 167, in normalize_inference
epsilon=self.epsilon)
File "C:\Users\isszfm\AppData\Local\Continuum\anaconda3\envs\Python36\lib\site-packages\keras\backend\tensorflow_backend.py", line 1908, in batch_normalization
mean = tf.reshape(mean, (-1))
File "C:\Users\isszfm\AppData\Local\Continuum\anaconda3\envs\Python36\lib\site-packages\tensorflow\python\ops\gen_array_ops.py", line 6112, in reshape
"Reshape", tensor=tensor, shape=shape, name=name)
File "C:\Users\isszfm\AppData\Local\Continuum\anaconda3\envs\Python36\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "C:\Users\isszfm\AppData\Local\Continuum\anaconda3\envs\Python36\lib\site-packages\tensorflow\python\framework\ops.py", line 3392, in create_op
op_def=op_def)
File "C:\Users\isszfm\AppData\Local\Continuum\anaconda3\envs\Python36\lib\site-packages\tensorflow\python\framework\ops.py", line 1734, in init
control_input_ops)
File "C:\Users\isszfm\AppData\Local\Continuum\anaconda3\envs\Python36\lib\site-packages\tensorflow\python\framework\ops.py", line 1570, in _create_c_op
raise ValueError(str(e))
ValueError: Shape must be rank 1 but is rank 0 for 'input_batchnorm/cond/Reshape_4' (op: 'Reshape') with input shapes: [1,256,1,1], [].

It has been a while, but for a "self" play run try without any weight file and it should create one to start with.
The best weights are for "uci" play.

Thanks brianprichardson.
You mean it never runs with GPU?
I have tried "self" without any Json model and weight file, still got error.

tensorflow.python.framework.errors_impl.InvalidArgumentError: Shape must be rank 1 but is rank 0 for 'input_batchnorm/cond/Reshape_4' (op: 'Reshape') with input shapes: [1,256,1,1], [].

Depending on the situiation, the weights (.h5) and model (.json) files must match the net architecture in the configs file (typically mini.py).  The stronger ones that I uploaded do not match the current config files.
IIRC, when running "self" if there are no .h5 and .json files they will be created first.  You can add
self.model.summary()
at the end of the def build(self):  in class ChessModel: in model_chess.py in the agent dir to see if it is creating a new model from the specs in the mini.py file.
For running "uci" it just tries to read the best files.  Other params in the config file can still be set, but most are ignored for uci, like playouts is 1,200 (sort of like fixed number of nodes).
The first output you posted shows it is trying to run with the gpu.  As slow as it is, it will be far to slow to run without a gpu, and your 1080ti is a very good one.
I would try a clean download and just try to run with "uci" and enter the "uci" and "isready" (remember to wait for the readyok), and then "go".  You should get a bestmove after some time.  If that works, then your packages and gpu are all working ok and we can work from there.
What are you trying to do, in general?  Self-play training is extremely slow and takes a lot of disk space for the intermediate input plane files.  That's why I have a tweaked version that takes pgn input and trains directly from that.

First only do
python src/chess_zero/run.py uci
Then, after it loads enter
uci
isready
go

First only do:
python src/chess_zero/run.py uci
Then, after it loads enter:
uci
isready
go

See #75 there is a link a fork with a working version.


Extension - Unable to scroll list account when connect a Ledger device

Describe the bug

[x] Unable to scroll list account when connect a Ledger device

To reproduce the bug
Step:

Perform connect a Ledger device
On list account, perform scroll and observe the displayed screen

Actual: Unable to scroll list account

Reason: Zoom in/Zoom out browser

[x] Navigate to the previous screen when scroll

https://github.com/Koniverse/SubWallet-Extension/assets/117999489/4bf594f5-2bba-4499-9e8a-16bbeb8cffc1

Validation passed on Extension
On webapp, handle on issue #2782


DVSIM suggestions for output after the latest updates

in the next DV meeting it would be relevant to go over the latest updates of DVsim and the reason behind it.
as I actually feel they make dvsim less user friendly than before and would like to know the motivation for the changes.
we could also use this as a good segue into a talk about what we would like from dvsim.
some of the thought I have are:

the latest updates that always put the scratch in the root - not a big fan. before if you ran from root your scratch was in root. if you ran from your IP folder that's where your scratch is. now if you run from your up folder you have to navigate back to root to find your output. this feels like a step backwards in my book.

if a build fails the log is no longer in default, instead it is moved under failed/build.log IMO we don't want older logs lying around in default. it made sense that this was where you found the latest output. I think we should either revert to this option or remove default altogether.

in general, I feel that a lot of typing involved in getting to the output of a build to get to a log is now 6 levels the folder hierarchy

currently the naming of runs is incremental numbering, would it not make sense to exchange the incremental numbering with the actual seed? this would make it easier to locate specific tests ?

should we add a run -analyze option to open the waveform for the latest run?  -analyze={simvision,dve,verdi}

Well at least for the first bullet, I like the scratch being in the root so I can delete all the logs in one place (ie by removing single scratch directory). But I always run from root so just my opinion.

Well at least for the first bullet, I like the scratch being in the root so I can delete all the logs in one place (ie by removing single scratch directory). But I always run from root so just my opinion.

but if you run from root this was always the case!

My only comment is that dvsim has grown a lot of features and I don't think I'd propose adding yet another. So my suggestion for the "analyze" option you mention is that we just print out the path to the dumped wave file. The user can probably copy and paste and add "dve" or whatever to the front of the command line. That way, dvsim doesn't need to know or care what viewer they are using.

@sriyerg @rasmus-madsen can this issue be updated?

in the next DV meeting it would be relevant to go over the latest updates of DVsim and the reason behind it. as I actually feel they make dvsim less user friendly than before and would like to know the motivation for the changes. we could also use this as a good segue into a talk about what we would like from dvsim.
some of the thought I have are:

the latest updates that always put the scratch in the root - not a big fan. before if you ran from root your scratch was in root. if you ran from your IP folder that's where your scratch is. now if you run from your up folder you have to navigate back to root to find your output. this feels like a step backwards in my book.

These were the reasons for doing so:

What @senelson7 mentioned - most people (especially those working on multiple blocks) prefer the scratch to be under the project root directory so that it can be purged in one shot.
It aligns with other build systems which also create artifacts in a single directory at the root of the project.
There were also other motivations that I forget now :) In general though, it is advisable (but not mandatory) to run dvsim always from the project root directory,

if a build fails the log is no longer in default, instead it is moved under failed/build.log IMO we don't want older logs lying around in default. it made sense that this was where you found the latest output. I think we should either revert to this option or remove default altogether.

The default build directory is overwritten in successive builds. It enables the tools to optimize the compile time by incrementally elaborating only the things that changed. On failure, default directory is actually not moved to failed - its actually soft-linked there to enable users to navigate there a bit more easily, especially when the scratch path is cluttered with 1000s of run directories.

in general, I feel that a lot of typing involved in getting to the output of a build to get to a log is now 6 levels the folder hierarchy

The scratch path is located at scratch/<branch>/<ip>-sim-<tool>, which is only 3 levels. It used to be even deeper - I condensed the last one to reflect 3 unique qualifiers (block name, ASIC flow, tool used) into a single directory name. I think these are needed. There are valid reasons to keep results across separate branches.

currently the naming of runs is incremental numbering, would it not make sense to exchange the incremental numbering with the actual seed? this would make it easier to locate specific tests ?

The motivation of doing so is the number of directories created in the scratch path area will always be fixed to the number of reseeds for each test. Otherwise, we risk creating a monotonically increasing number of directories on successive invocations. The other advantage is, if you run the test once, with a single seed, you will always find it at a fixed path at 0.foo/out. That makes incremental debug easy (just hit reload in Verdi, rather than save signals, close the waveform and open the new one since it lives in a different path now, and reload signals). For ease of locating failing tests with specific seeds, see the failed directory that contains the soft-links - the soft-link name does reflect the seed value.

should we add a run -analyze option to open the waveform for the latest run?  -analyze={simvision,dve,verdi}

Yes - I did plan for supporting this, but never got to it. This will indeed be useful, but low priority.

Following up on the "analyze" part, I have the following in my bash history:
dve $(find scratch -name '*.vpd' -printf "%A@:%p\n" | sort -nr | head -n1 | cut -d ':' -f 2-)

This runs dve on the .vpd file that was most recently created. Presumably, you can do something similar with .trn files and Simvision.

should we add a run -analyze option to open the waveform for the latest run? -analyze={simvision,dve,verdi}
it is a nice to have - I think there are other things that are more important.

like UNR support for xcelium.
other than that I think I agree with all @sriyerg comments


Add maintainers guide

We should separate maintainers' and contributors' guide, and we can add this list as prefixes to commit titles:

DOC: documentation changes
FEAT/FEA: new major features
ENH: enhancements to existing features
CI: continuous integration, sometimes overlaps with MNT
MNT/MAINT: maintenance, technical debt, etc
FIX: bug fixes
TST: new tests, refactoring tests
PERF: performance improvements

(addendum from comment somewhere else: ENH should have user-facing implications)


Latest Drachtio built from source is "Hanging"

Hi All,
We've been running an older version of Drachtio (v0.8.10) for a while but encountered some weird issues when communicating with it on port 443 (via WSS) as opposed to port 4433 which we've been using until now. We want to move to port 443 for standardisation.
In a low-call-volume environment we found that the latest Drachtio build (v0.8.20-4-gb01823713) resolved the port 443 problem and decided to go the route of upgrading our other servers rather than trying to debug the 443 problem on the current version.
I did that upgrade on one of our busy servers last night and everything tested ok. No other changes were made other than switching out the Drachtio binary and restarting. I tested this morning again before things got busy and all was still ok.
However just after 8, Drachtio stopped logging to it's log file and stopped responding to requests. The Drachtio process was still visible in ps, and netstat still showed many connections to port 4433. The load and memory on the box all looked ok. After a few minutes I restarted Drachtio and everything ran fine for a only few minutes before the same thing happened again.
I then changed back to the older Drachtio version and all has been running fine since. I can't find any errors or anything suspicious in the system or Drachtio logs around the time of the failure. It looks like that new Drachtio version just "gives up" under load.
Has anything like this been experienced before? Any ideas how I can debug this further? We're running on Ubuntu 20.04.3 LTS.
Thanks,
Duncan

Can you get a log (or retrieve the existing log) from when this happens?  If possible it would be great if drachtio server was loglevel debug and sofia-loglevel 9

Hi Dave,
Thanks for the response. We did have debug and log level 9 set. I've attached an extract of the log either side of the "hang". I've inserted an extra line with "--- GAP HERE ---" where the logging stopped so it's easy to find, but you can see the timestamps have a big gap there.
I will build v0.8.20 and try that out as you suggest and give you feedback.
Regards,
Duncan
drachtiohang1.txt

that log seems strange -- no sip messages shown, other stuff missing.  Its clearly not the raw log file.  Have you got that?

This conversation currently continuing on Slack. I'll update here as appropriate for the community.

It appears this issue might have something to do with how we built Drachtio on Ubuntu. We're now running an official Drachtio Docker image and so far the issue has not occurred again. We're not sure exactly what scenario causes this though. I'm leaving this open for another week or so just to confirm the issue is resolved by the Docker image as it does occur intermittently.


Status of RMQ Broker is stale

Describe the bug
Even after the broker is created after many minutes. The state of the broker is not updated back by ACK MQ controller.
Steps to reproduce

Use the MQ v.0.0.4 image
Create a RMQ broker using version 3.8.6
Notice the Broker is created in the AWS UI and its shows up and running
Check the status of the broker from ACK K8s environment like

kubectl describe broker.mq.services.k8s.aws/ackrmq-broker-simple032221
 Broker State:  CREATION_IN_PROGRESS

You can notice the state of the broker is not updated back after  many minutes. I checked even after 30 minutes after creation

Expected outcome
The latest status of the broker should be updated back immediately
Environment

Kubernetes version
v1.19.7
Using EKS (yes/no), if so version?
no
AWS service targeted (S3, RDS, etc.)
MQ

@jaypipes FYI

/assign


Update minitest to 2.9.3

Updates io.monix:minitest from 2.8.2 to 2.9.3.
GitHub Release Notes - Changelog - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "io.monix", artifactId = "minitest" } ]

labels: library-update, semver-minor

Superseded by #207.


Allow specification of binary location in configuration

It should be possible to specify a pre-installed binary that if specified overrides the defaults

Implemented in version 0.1.5


[elasticsearch][logstash]Fix PodSecurityPolicy error in v1.22

Fix PodSecurityPolicy error in v1.22
There is no kind: PodSecurityPolicy in apiVersion: policy/v1

[ ] Chart version not bumped (the versions are all bumped and released at the same time)
[ ] README.md updated with any new values or changes
[ ] Updated template tests in ${CHART}/tests/*.py
[ ] Updated integration tests in ${CHART}/examples/*/test/goss.yaml

close #1643

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

Contributor Agreement has been signed.

I'd like to bump this PR, as we really could use it. Seems like a maintainer only needs to kick off the CI/CD and we are golden.

@jmlrt , you seem an active maintainer of this repo, is there anything blocking this PR/issue? It is really trivial, easy to test and removes a blocker for a lot of people.

I find it a bit frustrating that this (for us blocking) issue with a PR does not seem to get any attention from devs/repo owners. This is not a good way to embrace contribution from the community. I understand that time is scarce know how priorities work, but I think any reply is better than leaving contributors in the dark for months.
@jmlrt, @Crazybus, @framsouza, @ebuildy (you seem active in this repo), could you please gives this issue/PR some love?


Flexible user input

Made the user input more flexible by adding flags for problem site, problem id, and file. I used argparse, so as a result, I get better error-handling for the user input. Also added a quiet flag if the user only wants the summary and the run time.

LGTM!


CASMINST-5088

Summary and Scope
There was a typo "poda"  which had to be changed to "pods"
Issues and Related PRs
"poda" was used for negative test case scenario of kubectl command failure. Correct word is "pods"
Testing
Tested the script and it is working.
ncn-m001:/tmp # ./k8s_kyverno_pods_running.sh -p
PASS
ncn-m001:/tmp #
Tested on:
drax system
Test description:
Script is used to check if 3 kyverno pods are running if not appropriate error messages are displayed when executed manually for debugging.

CASMINST-5316
The above Jira covers details of this issue. Hence closing this one.


docs: remove repetitive words

remove repetitive words

Thank you but this is vendored code, so you will need to submit this upstream to llama.cpp.


Incorrect animation

Lightweight Charts Version:  3.7.0
Steps/code to reproduce:

Enable last price animation on data update
Add more history bars via setData

Actual behavior:
Animation is active
Expected behavior:
No animation. Animation should start only if a series has data and we're adding new bars to the right side or updating the last bar
Screenshots:

CodeSandbox/JSFiddle/etc link:
https://jsfiddle.net/6hq2vopz/2/

Since we removed a knowledge of the first changed data index from series (we don't have merging algorithm any more in PlotList) it might be tricky. Probably we have to provide that information from DataLayer for every series (but actually only "currently updated" series will have it).


Endpoint: Profile edition (Enhancements)

[ ] Display name
[x] Profile pic

[ ] Security check for the uploaded files

[ ] Pronouns

Once #71 will be done, user need to be able to edit their username in this endpoint.


Question-77, complete the description

Pure function has two conditions:

Same output for same inputs.
No side effects.

For example, although the following function has the same output for the same inputs, it is not a pure function
function sum(a, b) {
    console.log(a + b);  // This is a side effect
    return a + b;
}

What I need to do
On Wed, 23 Oct, 2024, 2:34 pm Mohammad Karimi, @.***>
wrote:

Pure function has two conditions:

Same output for same inputs.
No side effects.

For example, although the following function has the same output for the
same inputs, it is not a pure function
function sum(a, b) {
console.log(a + b);  // This is a side effect
return a + b;
}

You can view, comment on, or merge this pull request online at:
https://github.com/lydiahallie/javascript-questions/pull/807
Commit Summary

f998b0a
https://github.com/lydiahallie/javascript-questions/pull/807/commits/f998b0a3916a9c832c1725540b049ee93af10183
Question-77, complete the description

File Changes
(1 file
https://github.com/lydiahallie/javascript-questions/pull/807/files)

M README.md
https://github.com/lydiahallie/javascript-questions/pull/807/files#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5
(2)

Patch Links:

https://github.com/lydiahallie/javascript-questions/pull/807.patch
https://github.com/lydiahallie/javascript-questions/pull/807.diff

—
Reply to this email directly, view it on GitHub
https://github.com/lydiahallie/javascript-questions/pull/807, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/BBGBQI53XMFJAZ7OBZWDC7LZ45RDLAVCNFSM6AAAAABQOJ5M6KVHI2DSMVQWIX3LMV43ASLTON2WKOZSGYYDOOBSHAYTAMI
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>


Add opentracing support

See https://github.com/kata-containers/kata-containers/issues/27.

The plan for adding opentracing support to the agent is currently not clear: nominally, a "host system" will run a Jaeger agent (which the application will send spans to via UDP). However, we really do not want to have to add another component to the guest images.
Further, adding an extra component to the image when the agent is running as PID 1 will require custom code to launch the agent at VM startup... by the agent. Clearly, that isn't going to work unless the Jaeger client could "buffer" the spans in readiness for sending to the Jaeger agent when it eventually started.
The current thinking is to mandate a Jaeger agent in the host environment only (the host the runtime runs on). We could use a virtio-serial port or vsock to allow the agent to send spans out of the VM back to the agent running outside the VM. However, vsock can't really be the only solution given some users run old host kernels that do not have vsock support (and they cannot change the host kernel).
The current thinking is that we either offer multiple approaches to extracting the spans from the agent inside the VM (yuck), or we:

Use a virtio-serial port.
Configure Jaeger to log to shared memory and share in a block of shared memory from the host into the VM for the agent to write spans into.
However, we need to know what happens if the block is too small / becomes "full" - does Jaeger support circular logging buffers, etc?

/cc @sameo, @mcastelino, @sboeuf, @bergwolf, @grahamwhaley.

Just a brief (slightly devils advocate) thought.... given:

we have enabled vsock in the runtime now
pretty much everything that happens in the agent is driven from the runtime, so 'one end' of the conversation is already being traced....

then I'd not be unhappy if we said, at least initially, that we would implement agent tracing via VSOCK, and if you don't have VSOCK available on your system, then, well, sorry, but you still have 90+% of the useful information just by looking at the runtime traces.
We could always put non-VSOCK as a secondary item, and if a member really really needed it, they are free to work with us to PR it ;-)

I guess that could work, but it would be good to get input from some power users about whether this approach would work for them.
P.S. You missed an opportunity to use :imp: :smile:

We're long overdue for an update on this issue, so brace yourself as I attempt to summarise the current situation...
Current serial behaviour
Startup
The default configuration causes the agent to connect to a virtio serial channel. That serial channel is wrapped in a yamux connection. The server uses this channel to create a gRPC server. The server serves "forever" - that is to say, the agent never exits the grpc packages Server.serve() function.
Shutdown
The agent "never exits"; once the workload has finished, the runtime sends a QMP QUIT command to shutdown the VM as it is no longer required.
However, iff a QMP QUIT where not sent, the gRPC server would exit, allowing the agent to exit.
Current vsock behaviour
Basically the same as serial but with one important exception - the gRPC server cannot exit since the vsock code path does not use yamux. This means the server has no way of detecting the client has disconnected and just blocks forever waiting for a new client to connect.
Ideal behaviour
The agent would exit the Server.serve() call to allow for a controlled shutdown. That would allow all trace spans created since agent startup to be sent to the Jaeger agent.
Given the design of the grpc package, golang constrains and our requirements, the only clean way I can find to shut down the agent in a controlled fashion is to require yamux since that layer provides the agent with the ability to detect when a client (kata-runtime) disconnects. The current tracing design also requires vsock to be enabled to allow both the VM's journal message and the trace spans to be collected outside the VM. Adding yamux to the vsock solution adds overhead but does provide the required semantics whereby a StopServer gRPC call can actually stop the server servicing that call since as yamux detects the client disconnect, the Server.serve() call ends allowing the agent to shutdown.
Notes
Note that we could create two codepaths and only enable vsock with yamux if:

The runtime is configured for agent tracing.
The configuration.toml sgodoc.org/google.golang.org/grpc#Server.Serve) call ends allowing the agent to shutdown.

Note that we could create two codepaths and only enable vsock with yamux if:

The runtime is configured for agent tracing.
The configuration.toml specifies use_vsock=true.

That would be annoying in that we'd have two paths to test, but would not impact the default no-proxy configuration.
Challenges
With vsock+yamux, the agent shuts down cleanly meaning that full trace is available.... theoretically :)
However, the overall system does not work. The problem is that whenever a client disconnect occurs, the agent's gRPC server will exit (and restart).
Forcing the existing keep-alive option in the runtime helps to stop the agent gRPC server from exiting, but is insufficient since disconnects still occur in the runtime and that leads to the runtime getting gRPC errors it does not expect ("request error: rpc error: code = Unavailable desc = transport is closing").
These errors caused by the fact that the api calls in api.go are standalone/discrete - the client (runtime) calls API like CreateSandbox() and StartSandbox() but APIs like these connect to the agent, perform one or more gRPC calls, then disconnect. We could tweak the runtime to tolerate such errors, but it's not ideal as the runtime has no way of knowing if such an error is caused by the agent's gRPC server restarting or if the agent just died.
One solution to this would be to have a new API that stores such options in a "handle" that is passed between the API calls for options like the existing keepConn. This would allow the code to be reworked fully so that only a single agent connection could be maintained (avoiding disconnects). But in fact we already have such a handle - all the API calls now specify a context.Context as their first argument. The complication is that the default context is actually an int so we would need to create a custom context type based on a struct pointer which implements the Context interface (looks quite easy to do).
Compromise behaviour
Adding yamux support on top of vsock should not really be necessary.
What we could do is to only trace the gRPC calls themselves. This could be accomplished by:

Changing the agents gRPC protocol so that all functions return a context.Context (currently, most return google.protobuf.Empty).
Make the agent create a trace span in each gRPC call (in the unary and stream gRPC interceptor functions). This could be a bit fiddly but looks possible.
Add the trace to the context.Context passed to each call (from the runtime).
Returns the "enriched" context as the gRPC functions return code.
Update the runtime to extract the agent span details from the returned context and submit that to the Jaeger agent.

Pros

This approach would no longer necessite requiring vsock for tracing meaning you'd be able to see agent trace even if you use a kata-proxy.
However, if vsock is enabled, we could still leverage that to pass the VM's journal messages out using the kata-journald-host-redirect.service systemd service and the accompanying vsock-to-text-server.sh script.

The agent would no longer need to shutdown cleanly as only the gRPC server calls would be traced.

Since the runtime would submit the agent traces to the Jaeger agent, none of the extra "scaffolding" currently required for tracing the agent would be required (we no longer need the jaeger-client-socat-redirector.service systemd service nor the vsock-to-udp-server.sh script).

Cons

We wouldn't be able to trace the agent setup and teardown (the code either side of the gRPC server it runs).

/cc @mcastelino.

@jodh-intel I worked on this today, and I have been able to reproduce the weird behavior of the grpcServer.Serve() not returning...
But the good news is that I found the fix for this. The culprit is the vsock Go package itself that had a bug, and by moving to the latest commit 676f733b747cd6406f297a51bd086ee8ec8abdbe our agent vendoring, things get fixed :)
So to summarize, I have been testing with vsock enabled + my PR https://github.com/kata-containers/agent/pull/448 + updated mdlayher/vsock vendoring. Make sure you build the agent with Go 1.11, as 1.10 might run into issues from the quick look I had at the patches.
And then you have fun by running a simple docker container:
docker run -it --rm --runtime=kata-runtime busybox
#

If you want to make sure about the grpcServer.Serve() exiting without using any extra traces, you can use the following command:
sudo socat "stdin,raw,echo=0,escape=0x11" "unix-connect:/run/vc/vm/<containerid>/console.sock"

right after you started your container. The <containerid> can be simply found by looking at the qemu command line, or listing what's under /run/vc/vm/ for instance.
Anyway, the point being that you should see the following traces from the socat terminal:
level=info msg="stopServer signal received"
level=info msg="gRPC server stopped"
level=info msg="agent grpc server has been explicitly stopped"

at the moment you exit your container.
Oh yeah, one more thing, I have added a time.Sleep(5*time.Second) in the runtime code to make sure I was not triggering ExecuteQuit() too fast. Of course, this is simply to check the server had the time to tear down.
Let me know if vsock revendoring help, this way we can rule out the usage of yamux.

@sboeuf - Mon Dieu! After spending the morning trying to recreate your findings, I think I now have. I was already using the latest vsock, but I was still using a go 1.10 compiler. Switching to 1.11 I am seeing what looks like the correct behaviour  :pray:!

Glad to hear that :+1:
Please let me know about your progress now that we have solved the bulk of the problem!

@jodh-intel given that opencensus which supports creating custom exporters I attempted to create a simple poc to allow exporting of traces over any network transport. And at least conceptually this should work.
https://github.com/mcastelino/custom_exporter
Here I create a custom exporter which serializes the traces into the thrift format which is used by jaeger.
On the far end the traces can be dumped to jaeger itself using the standard exporter
https://github.com/census-instrumentation/opencensus-go/blob/master/exporter/jaeger/agent.go#L41:6

Basically you can chop the exporter into two parts

Hi @mcastelino - opencensus sounds interesting but atm Kata is solely using opentracing.
As such, I'd like to focus on landing basic agent tracing support using opentracing as soon as possible.
Once that is in place, we can consider alternative tracers by raising the topic at the Architecture Committee meeting (and since it sounds from what you're saying that most of the alternatives are mostly compatible with each other, it shouldn't be too disruptive to switch at a later date).

@jodh-intel I'd argue that we should use the custom exporter as part of the initial PR for Opentracing. The reason is simple, by using this exporter, we can use some Go code to handle the vsock connection, while for now the PR introduces more systemd services to start scripts handling the vsock to UDP translation.
The huge benefit being than on the agent side, there will be no need for any vsock translation, and the agent code will be only manipulating generic Opentracing code.
Last point, by removing the systemd services, we will be able to support Opentracing when agent is running as init as well.

@sboeuf @jodh-intel we can do this after James's initial PR lands. I just wanted to highlight that moving to open census allows to address the issue once and for all. The bulk of the code should remain mostly unchanged. So the primary goal of moving to open census was not to support alternate backend. But more about getting back some control in our code so that we can control how we cross across the VM boundary.
With opencensus we can do the following

use vsock directly in the exporter
this allows us to run the agent as init (and not mandate systemd)
it also allows us in the future to support alternate backends (however this will need more work as the marshalling across vsock may end up being backend specific).

Note: I was was forced to marshall to thrift because the opencensus span had one unexported field. If all fields of opencensus were exported it would make our lives much easier.
The offending field is
<EMAIL_ADDRESS>

@mcastelino but if we land @jodh-intel PR first, we would need to also merge the rootfs part of it including new systemd service files, and then later, when we include a custom exporter, we'll need to update the rootfs again by removing those files.
This sounds like a lot of back and forth while we could make things nice the first time.

@sboeuf, @mcastelino - I'm definitely in favour of dropping vsock/udp scaffolding code if we can replace it with something simpler. However, opencensus is going to take (me atleast) time to grok and internalise. That's not to say we shouldn't do it of course, but fwics we'd need to add support for another protocol definition language in the agent build (thrift auto-generated code to accompany the existing gRPC code) and we'll still need host-side scaffold support if I'm understanding this correctly.
On the topic of thrift though @mcastelino - could you add the *.thrift files to your branch so we can see the source for all that auto-generated code?

As I say, switching yet again to a different approach is going to take time. I'm happy to look at it but will need help from @mcastelino still I suspect.
@egernst may have input based on the hope of getting rudimentary agent tracing support landed in the next release (which looks somewhat unlikely from where I'm sitting now). If we are indeed time-pressured, we can work collectively on this. In fact, there's no reason we couldn't land an opencensus custom exporter PR into the agent now and I could then rework this branch yet again to leverage it.

OK. Sitting comfortably? @sboeuf, @mcastelino and myself have just had a chat about juggling eels^W^Wagent tracing. What we're planning to take a staged approach to introducing agent tracing. The plan:
Land #415 with modifications
The current implementation works and provides full tracing (from agent start to agent shutdown). However, the current implementation will theoretically require runtime support (to disable QMP QUIT when tracing is enabled).
What we plan to do is add two new gRPC API's to #415: StartTracing() and StopTracing().
The behaviour will be as follows:

If agent.trace= is set on the guest kernel's command line, agent tracing will start as early as possible and the agent will exit after the sandbox is destroyed (DestroySandbox() called by the runtime.
This is the behaviour on the PR currrently. It is a special case and the only scenario where the agent shuts down and causes the VM to exit (rather than having the runtime control this behaviour).

If agent.trace= is not set on the guest kernel's command line, do not start tracing and do not exit the agent on DestroySandbox().
This is the current behaviour of the agent (no tracing enabled or possible).

If agent.trace= is not set on the guest kernel's command line but the runtime calls StartTracing(), trace all subsequent calls.
This will allow tracing to be controlled dynamically. Clearly, the trace will not include events that occurred before tracing was started.

If  StartTracing() has been called and then DestroySandbox() is subsequently called, do not shutdown the agent.

If StopTracing() is called and StartTracing() has already been called, end tracing.

If StartTracing() is called and StartTracing() has already been called, return an error.

If StopTracing() is called and StartTracing() has not been called, return an error.

With the exception of the agent.trace= scenario, these two new calls will allow the runtime to retain control of the VM and also allow the runtime to continue to call the QMP QUIT to destroy the VM.
Create a follow-up PR to rework the agent tracing code to switch to OpenCensus
The advantages of this being:

We don't hard code the tracing implementation into the agent (implementation agnostic tracing / future-proof)
This means we can remove the Jaeger-vendored code.

By using an "exporter", we can remove the need for the vsock->udp service inside the VM (simplification).

Enabled agent tracing when the agent is running as init (a limitation of #415, which requires systemd).

Allows the existing rootfs / initrd images to support tracing without any changes.

Update the runtime to support tracing

To enable full agent tracing (startup to shutdown), the runtime will need to be modified to not call QMP QUIT. This should only happen when agent tracing is enabled but further only when "full" agent tracing is required.

The runtime will also need to be updated to provide a kata-runtime kata-trace [start|end] $id to support dynamic agent tracing.

Replace the host-side vsock->udp proxy script
#415 introduces a vsock-to-udp-server.sh script that forwards Jaeger spans from the agent in the VM and forwards them on to the Jaeger agent running on the host. When the opencensus PR lands, we'll still need this script (but it will probably need adjusting slightly (port number).
What is required is a service that runs for the duration of the VM to forward traces onto the eventual trace collector. But we already have such a service - the kata-shim. As such, we can probably remove the script and make the shim also handle the trace comms to simplify the architecture further.
Further optimisations
There may be future enhancements we can make to improve the tracing even further.

@jodh-intel Thanks for summarizing this, it is a very accurate description!
I have two comments:

The runtime will also need to be updated to provide a kata-runtime kata-trace [start|end] $id to support dynamic agent tracing.

I would expect the virtcontainers API (kata_agent.go) to be calling into the new gRPC functions StartTracing() and StopTracing() as part of the sandbox lifecycle. I would not expect that the end user would directly call into the Kata CLI to trigger those.
By having this handled by the API internally, and relying on the config options from configuration.toml, this would ensure consistent tracing behavior for both kata-v1 and kata-v2.

What is required is a service that runs for the duration of the VM to forward traces onto the eventual trace collector. But we already have such a service - the kata-shim. As such, we can probably remove the script and make the shim also handle the trace comms to simplify the architecture further.

Having the shim handling this could work, but we need to enable it only on the shim representing the container that represents the sandbox. From the top of my head, I don't recall if this is something easy to identify.
If this is not possible (again, needs to be investigated), the alternative solution would be to package a small binary that would be spawn from the runtime.
The kata-v2 case is the easiest one since we could handle this directly from the runtime, as a separate go routine.

By having this handled by the API internally, and relying on the config options from configuration.toml, this would ensure consistent tracing behavior for both kata-v1 and kata-v2.

That makes sense, certainly initially. I was imagining it could be useful for an admin to enable tracing on a long-running container at any point, hence the CLI options. But, yes, they could be added at a future date as required.

Having the shim handling this could work

Yep - like you, I haven't yet investigated how practical this actually is.

If this is not possible (again, needs to be investigated), the alternative solution would be to package a small binary that would be spawn from the runtime.

Right. I think we'll all agree we want to minimise such binaries, but this may indeed be a requirement. We could conceivably add a new runtime CLI to handle this such that if the admin runs something like kata-runtime trace listen or similar, the runtime could dynamically load a small module to deal with this. That would be in line with the VMCache approach (https://github.com/kata-containers/documentation/pull/393).


1.19 support

Currently the mod is 1.18 only. Judging by how simple the mod is, it probably is as simple as recompiling the mod for a newer version of Minecraft.
If I have time later, I'll try building it myself and see if it works.

Yeah! Please do this!


ModelForm like admin

The admin interface is great. Are there any good examples on how to use django-polymorphic in forms and templates?

If you're looking for an add/edit view that switches the form based on a "Add object type" choice, that's something you could build yourself and share with us!

Any examples/ideas of how to do it?

I'd just display a list of URLs on the first step in HTML, that redirect to the actual form.
You avoid the pain of trying to squeeze a lot of different models in a single formtools wizard.


Update README.md

Added missing information about sameSite option

The sameSite option is not new. It was added to the original cookies library back in 2017.


Improve documentation of PRF

Checklist

[x] documentation is added or updated
[x] CLA is signed

Description of change

Improve documentation of PRF.
This should be applied to both master and 1.1.0 branch.

merged


Fix array.sum(axis) 1d input return type.

Makes it such that if a summation with an axis is called on
a 1d array the return type is scalar.
Fixes #3815

looks good, pending CI passing

Thanks for the review. CI passed,  merging.


Extension Table 9650 – Custom Report Layout

A publisher OnBeforeLookUpLayout must be added here. The ID 5084 report (currently fixed coded) must be able to be overridden. The possibility to add a filter of reports (5084 | 5059450 ...) would be even better.

Reason: Report 5084 cannot be expanded to include serial printing fields.
Note:
I'm sure this isn't the only program site BC fix filters to report 5084. Should the process of creating activities with the "Merge" the Report 5084 later, this position must also be designed in such a way that you can use your own report.


[Feature Request] Pagination

I have over 300 movies on my server. Is it possible to load all movies and make a pagination to avoid continuous scrolling?
Thank you.

Hello, this is not in scope of the project. You can use infinite scroll along with sorting and / or search.


consider not bg loading /members for tab-complete to work

currently we bg load /members for every room to populate up the membership list, and make tab-complete supersnappy. however, this uses a lot of RAM on the membership events, which gradually gets worse until you restart the app. empirically it seems that Discord can get away with super-snappy tab completion while storing the membership list serverside; they fire off a JSON query over WS and then you get some zlib binary results back as the user is tab-completing. And they end up using 50MB of heap on Chrome. so if they can do it, we should have a go too.

wrong repo


Helia roadmap status, comments, and feedback

🛣🗺 Roadmap document
Current status of Roadmap
2022-10-26: Accepted. Project maintainers have aligned on this roadmap.
Description
This issue is intended to capture discussion around the 2022 Q4/2023 H1 Roadmap.
Because GitHub doesn't have a feature to comment on files in the repo, please use this issue to leave any feedback, proposals, etc.
Roadmap drafts:

https://github.com/ipfs/helia/pull/1

I would love to see "Publish IPNS names" on the roadmap. Working just with the browser, without a backend help.

@aurium this works today, if the stars align.  The reason success is not always guaranteed is because in order to publish an IPNS name, the node running in the browser needs to be able to dial network peers that are KAD-close to the public key used to publish the IPNS name, in order to ask them to store the IPNS record.
Historically the only transports available to the browser were WebSockets and WebRTC-star.  WebSockets need a TLS certificate to be dialable which is impractical for most users, and go-libp2p (the most commonly used implementation) has never supported WebRTC-star so typically browser can't dial most of the network.
This is changing with the new WebTransport and WebRTC transports which allow browser nodes to dial server nodes without pre-configured certificates.
Helia ships with these enabled, and they are supported by recent versions of go-libp2p and rust-libp2p but we are limited by the speed at which the network upgrades to these newer versions so it would be hard to say "This will work in Q3" when it's mostly out of our hands.

I don't understand why the solution to the problem is yet another divergent implementation instead of a shared library of common functionality which can be compiled as asmjs or wasm and given a sane and ergonomic interface in any language.

I don't understand why the solution to the problem is yet another divergent implementation instead of a shared library of common functionality which can be compiled as asmjs or wasm and given a sane and ergonomic interface in any language.

Very good question, I have the feeling it's already a task with the ipfs project to solve not the integration of the solution to create a universal tool but porting implementation, disassembly and complication of the whole development

I don't understand why the solution to the problem is yet another divergent implementation instead of a shared library of common functionality which can be compiled as asmjs or wasm and given a sane and ergonomic interface in any language.

As a web developer, I don't want wasm in my application unless I have to.  While wasm works in the browser, it has a hard time going through all the tooling, unfamiliar protocols and data structures, amongst other non-technical problems.
More or less, it's the same reasons go developers don't got to npmjs.org, find a library they like and transpile it to go for their go project.

@MichaelJCole has it in a nutshell.  WASM is a great idea on paper but the DX in reality often leaves a lot to be desired.
Where it is really useful is for very hot codepaths that can take advantage of the speed increases it brings, but they need to be quite self contained with well defined entry and exit points to make debugging easier when things go wrong.
I'm going to close this issue as it's quite old.  If anyone has items or ideas they'd like to see in the roadmap for the future please open an issue for discussion.


[Question] why fp8 kv output is dequantized before attention?

Hi, I noticed that the FP8LinearStatic will dequantize the output(fp8) to input dtype(fp16/bf16) as below L209 shows.

Is it due to the attention kernel does not support fp8 at that time? And is it supported now?
if supported, Could you pls guide me the code line in vllm or others?
Finally, regarding the code line below, I think this is equal to fake quant? And you just stored the kv scale for attention to dequantize fp8 kv read from fp8 kv cache

There are two reasons. First, the FP8 GEMM accumulator must be in FP16 to prevent overflow, so the output is naturally FP16 instead of being "dequanted". Making output in FP8, on the other hand, requires additional quantization overheads.
Second, the operators followed by GEMM usually don't compute in FP8, either because of not supported (like attention) or numerical sensitive (like RoPE or RMSNorm).

thanks for your detailed explanation, Cody! have you ever tried simply set the output scale (fp16->fp8) to 1? which I see has the almostly similar accuracy in some scenario

thanks for your detailed explanation, Cody! have you ever tried simply set the output scale (fp16->fp8) to 1? which I see has the almostly similar accuracy in some scenario

It may work with your tasks but I don't think this would work in general, as I can easily imagine a case with accuracy dropping, where your fp16 value exceeds the fp8 value range.

thank cody for the sharing, it helps me a lot!


Set enable network host to false during integration tests.

Short description of the problem this fixes or functionality that this introduces. This may be used for the CHANGELOG
Temporary fix to this issue:https://github.com/All-Hands-AI/OpenHands/issues/3568

Give a summary of what the PR does, explaining any non-trivial design decisions
Currently in wsl, when network_host is set to 'host', the backend is unable to access the runitime server running inside docker. So, a permanent fix will be fixing the wsl to docker connection. But maybe not using the host network stack might just be safer.

Link of any specific issues this addresses

ugh the integration tests failed bc we need to map 8000 port into the container... now i get it why we set it to host network before..

is anyone getting permission error when running this: TEST_RUNTIME=eventstream ONLY_TEST_NAME=test_ipython  ONLY_TEST_AGENT=CodeActAgent ./tests/integration/regenerate.sh

ugh the integration tests failed bc we need to map 8000 port into the container... now i get it why we set it to host network before..

Which service is using port 8000 inside docker?

Err... seeing this again, I'm running integration tests just fine in WSL with host network and don't see a reason to turn it off, tbh.

is anyone getting permission error when running this: TEST_RUNTIME=eventstream ONLY_TEST_NAME=test_ipython ONLY_TEST_AGENT=CodeActAgent ./tests/integration/

The permission error should be gone with 0.9.4 since we made sure that the user with id 1000 is removed from the base image.

I am not working on this currently, closing this.


Remove docker_start.sh and set locust as entrypoint for official Docker image

PR for #1247.

lgtm!


collapsible not staying open when routerLinkActive is true on child

Problem:
When I visit a page which matches the route of a list item inside a collapsible list it briefly flashes the inner list item when the page is loaded and closes the collapsible list immediately. Opening and closing the collapsible list manually works as expected.
Expected behaviour:
When I visit a page which matches the route of a list item inside a collapsible list it opens the list and it stays open unless the user closes the collapsible list manually.
Code:

      Configuration

               Expressions

I have the same problem. Did you manage to solve that issue? I'll appreciate your answer.

@szymon365 Sadly I did not. The project I was working on ended before I could solve it so I stopped looking for it. I did read some stuff about including Jquery(=bad practice though) in your Angular project and manipulate it that way.
If I remember correctly these contain the Jquery "solutions".

https://github.com/Dogfalo/materialize/issues/2384
https://stackoverflow.com/questions/30956378/materialize-collapse-is-not-working-properly-when-trying-to-display-json-data

Good Luck


Replace the short-life certificate with long-life one

Description of the changes:

Replace the short-life certificate (1 year) with a long-life one (2075), also used by other Microsoft apps.
Update CalculatorUnitTests to use the same certificate than Calculator Dev

How changes were validated:

manually

/azp run


Generate RAML based on existing project

Hey,
I have an existing JSON-API based on Fractal and I'm wondering if your package can help me to generate a RAML file by analysing my codebase?
Best

Hi, raml-json-api was created as the tool/api-framework to create an apps (code) primarily based on RAML-types, not vise-versa. I suppose there are math problem (no one can predict what classes/methods used in code) to generate particular RAML for different code-base. Until, for example there will be some hooks/annotations in comments like @method PATCH, @gen RAML and so on. But I see the probability to do this from JSON API to RAML and this is quite plain =), but again there will no any type support obviously.

Closing this issue related to - raml-json-api was created as the tool/api-framework to create an apps (code) primarily based on RAML-types, not vise-versa.


Show Track List/video resulation list in a dialog

Can you please help me how I can show bit rate or video track list in a dialog and can select it in video player.

Usually you'd need your tracks to have specific bitrates to be able able to do this, or ser custom bitrates you define and set them directly into the player.
What kind of url/file are you using?


Use workspace typescript version

Currently the extension uses a bundled typescript version. It's better to use the version from vscode itself and respect typescript.tsdk setting. Implementation of typescript-language-featuers should help.

TypeScript imported in script will be using TS bundled with vscode. Doesn't think this is possible.

Accidentally found this comment about Deno vscode extension. I will look into this again.

Looking at latest doc of vscode, this won't be possible

Visual Studio Code includes TypeScript language support but does not include the TypeScript compiler, tsc.


Process doesn't end after closing packaged app or unreal editor

Sometimes, it seems like the lingering process is stuck in an infinite loop with a steady amount of CPU being consumed indefinitely. Other times, there's 0% CPU activity from the process as only the RAM is consumed. The process still remains even after an hour from the time the application was closed.
I'm not doing any multithreading or anything fancy. I did have the QT example on it, but even after clearing all the lines in all the python scripts, just starting and closing the packaged application leaves the process behind. I have a PyActor with an empty script too (every method simply has "pass") that can be instantiated. But even without instantiating it, the process can't seem to end from just closing the app.
I haven't tried reproducing this on a new project yet. But I only began noticing this issue after packaging with UnrealEnginePython. I cannot say for sure that the issue was caused by the plugin, but it is a strong suspect. In fact, closing the unreal editor after having played in the editor leaves an unreal engine process behind as well.

The processes just consume RAM quietly most of the time. But after several runs, they can pile up and have caused my machine itself to freeze, requiring a forced shutdown.

just for being sure, have you tried repackaging it without the python plugin ?
Which python version are you using ? Is this the latest release of the plugin ?

Yes, I cloned master roughly 24 hours ago. I'm using Python 3.7 (64-bit) and Unreal Engine 4.19.2. Before integrating the plugin, this issue wasn't there.
I will now try removing the plugin and re-packaging.

Ok, I can't reproduce it with just the plugin itself on a fresh blank project. Then I tried adding my QT scripts and the issue popped up. I noticed from the unreal editor logs that the QT app instance never quits between reruns in the editor. It could be the culprit after all. I tried adding app.quit() to my implementation but still didn't solve it. Looks like I'm doing something wrong somewhere, but curious as to why clearing the contents of my scripts in the packaged game didn't get rid of the issue.
I'll try to figure it out, but here's my ueqt script.
import sys
import unreal_engine as ue
import PySide2
from PySide2 import QtWidgets

class UEQT:
    app = None
    app_users_count = 0

    def ticker_loop(self, delta_time):
        self.app.processEvents()
        return True

    def add_app_user(self):
        self.app_users_count += 1
        if self.app is None:
            self.app = QtWidgets.QApplication(sys.argv)
            self.ticker = ue.add_ticker(self.ticker_loop)

    def remove_app_user(self):
        self.app_users_count -= 1
        if self.app_users_count == 0:
            self.app.quit()
            ue.log("QT application closed")

inst = UEQT()

and my file_dialog script
from ueqt import inst as ueqt
from PySide2 import QtCore, QtWidgets, QtGui
import unreal_engine as ue
from PySide2.QtWidgets import QFileDialog
from unreal_engine import FSlateApplication

class DummyWidget(QtWidgets.QWidget):
    def __init__(self):
        super().__init__()

class FileDialogManager:

    # this is called on game start
    def begin_play(self):
        ueqt.add_app_user()

    def open(self):
        widget = DummyWidget()
        # widget.resize(800, 600)
        # widget.show()

        # get the active top level window
        top_window = FSlateApplication.get_active_top_level_window()
        top_window.set_as_owner(widget.winId())

        f = QFileDialog.getOpenFileName(
            widget, #parent
            self.uobject.Title, #title
            self.uobject.InitialDir, #initial dir
            ";;".join(self.uobject.FileTypeFilters), #filters
            self.uobject.FileTypeFilters[self.uobject.SelectedFileTypeFilter] #selected filter
        )
        if len(f[0]) < 1:
            ue.log("Open File Dialog:::Canceled")
            self.uobject.call_function('OnCancel')
            # sys.exit()
            return
        ue.log("Open File Dialog:::Confirmed")
        ue.log(f)
        widget.close()
        self.uobject.Filepath = f[0]
        self.uobject.call_function('OnOpen')
        # sys.exit()

    # this is called at every 'tick'
    def tick(self, delta_time):
        pass

    # this is called at the end
    def end_play(self, reason=None):
        ue.log("File Dialog:::Quit")
        ue.log(reason)
        ueqt.remove_app_user()

The PyActor simply calls "open" on begin play.

Hi, can you try with the latest release ? A bunch of fixes have been added after a very similar report (albeit not related to qt)

Awesome! Will report back hopefully tonight.

I pulled from master and rebuilt in visual studio. The issue is still present.

So, after a bit more digging i found that the problem is PySide itself. Even if you do not call any of its api functions, just importing its modules will break the Py_Finalize() procedure.
For this reason i have added ue.set_brutal_finalize(bool) that will skip the call to Py_Finalize():
https://github.com/20tab/UnrealEnginePython/commit/8308e441c10fd057e348dc830a367c01da9153b6


Update judging criteria on NC DataPalooza website

I noticed that the Judging criteria on our website is out of date.
can this page: http://www.ncdatapalooza.com/judging-criteria/
be updated with the information from this page: https://docs.google.com/a/ambrosestrategy.com/document/d/1M-NxgAmVJSAc5-sz5yvFhoE_zP4-qnqiOgVdwoDMk3E/edit?usp=sharing

Done!


Training Set

Hi there,
Would you mind describing briefly the training task you have performed, including the details of the dataset you fed into the network?
I plan to train my own model through your code, but I am struggling to find a minutaei-based annotation. So I was wondering if you can help me out with this?
Regards, Jafar

Hi,
You can have a look at train_CoarseNet.ipynb.
Information for annotation is provided in the folder CoarseNet_train. You can change the annotation format for minutiae if you want.
Let me know if it is clear to you.

Thank you for the reply,
Would you please share your training dataset with me?  You have used a
large dataset along with minutiae annotation, right?
How did you train the models present in the package? on what dataset?
Regards,
Jafar
On Wed, Sep 11, 2019 at 2:49 PM Dinh-Luan Nguyen<EMAIL_ADDRESS>wrote:

Hi,
You can have a look at train_CoarseNet.ipynb
https://github.com/luannd/MinutiaeNet/blob/master/Demo_notebooks/train_CoarseNet.ipynb
.
Information for annotation is provided in the folder CoarseNet_train
https://github.com/luannd/MinutiaeNet/tree/master/Dataset/CoarseNet_train.
You can change the annotation format for minutiae if you want.
Let me know if it is clear to you.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/luannd/MinutiaeNet/issues/15?email_source=notifications&email_token=AD3JEW237YYTPGQC43QAZP3QJDZM3A5CNFSM4IU47RL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6ORQBQ#issuecomment-530388998,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AD3JEWYBJA5OXBROJNTVBLTQJDZM3ANCNFSM4IU47RLQ
.

The annotated dataset we used is not public so I cannot share with you. If you don't have annotated datasets, let's try to find some public datasets or manually annotate them.
Please refer to the paper for the details of the dataset we used. Thanks!

The annotated dataset we used is not public so I cannot share with you. If you don't have annotated datasets, let's try to find some public datasets or manually annotate them.
Please refer to the paper for the details of the dataset we used. Thanks!

Can you share some annotated datasets with me? Promise to refer your work.
Thanks,
jawalexander

Im training my own data set using 'train_CoarseNet.ipynb' on google colab. But the sessions gets crashed before final computation. How can I resolve this.

Im training my own data set using 'train_CoarseNet.ipynb' on google colab. But the sessions gets crashed before final computation. How can I resolve this.

hi,ud38,could u share your dataset with me? thx a lot, I am trying the author's code.


KAFKA-12648: Pt. 3 - addNamedTopology API

Opened a PR of Pt. 3 against the still-unmerged Pt. 2 to help facilitate review on these changes.
Current status: addNamedTopology is implemented and should work, for single node applications only. The assignor changes are still needed to support adding a NamedTopology with more than one instance of the application running.

Closing since Pt. 2 is now merged


Potential confusion regarding Win32 & LoB apps

Table is directly under a section named "Line-of-business apps types" in a document titled "Understand line-of-business apps for Intune". I recommend changing the wording of the section name, stripping Win32 apps from this table, or altering the wording in the Description column for Win32's row, as Win32 apps are not considered Line of Business apps. The table itself doesn't note this until the end of the description.

Alternatively, the information about Win32 apps could be included as a separate blurb. Example below.