Not working with v 11.300

The number of folders is not increasing when in use with v11 Foundry. I installed and configured the desired amount of folders, and still nothing.
Not working in Foundry Stable Branch V. 11.300

Confirmed, module does not seem to function in Foundry v11.305 in both the normal folder areas as well as in compendiums.
Creating folders throws an error:
Error: You may not nest Folders more than 4 levels deep.
    at Folder._preCreate (foundry.js:21922:62)
    at #preCreateDocumentArray (foundry.js:13507:31)
    at ClientDatabaseBackend._createDocuments (foundry.js:13394:73)
    at ClientDatabaseBackend.create (commons.js:8533:19)
    at async Folder.createDocuments (commons.js:7862:23)
    at async Folder.create (commons.js:7985:23)
    at async FolderConfig._updateObject (foundry.js:72435:13)
    at async FolderConfig._onSubmit (foundry.js:6597:7)
_onSubmit @ foundry.js:6600

In compendiums, no error is thrown, but Foundry simply won't let you create folders deeper than 3 deep.
I really hope you'll both be able to fix this and that it's is an easy adjustment for v11. This module makes my entire asset organization scheme possible.

I tried making it work, but I don't know how.
I looked at the changes and it looks like just increasing the value in CONST.FOLDER_MAX_DEPTH should work, so I added the following code.
let newCONST = JSON.parse(JSON.stringify(CONST));
newCONST.FOLDER_MAX_DEPTH = FOLDER_MAX_DEPTH_MOAR_FOLDERS;
CONST = Object.freeze(newCONST);```

But when you try to create a folder at depth 5+ the backend returns an error which I don't think can be patched with a module.
So I guess this module is dead. Sorry about that.

Guys behind Plutonium managed to reach folder depth 8, so I guess it's possible.

Sorry for the slow reply!!
Honestly, if you can make it so that the max folder/compendium folder depth is 5 and not 3 or 4. that would be amazing. At least for me, a 5 folder depth is as deep as I'd ever want to go and gives you a ton more flexibility in asset organization.
Would you be able to create a new version that works to 5 depth?

Sorry for the slow reply!!
Honestly, if you can make it so that the max folder/compendium folder depth is 5 and not 3 or 4. that would be amazing. At least for me, a 5 folder depth is as deep as I'd ever want to go and gives you a ton more flexibility in asset organization.
Would you be able to create a new version that works to 5 depth?

I'll quote myself

But when you try to create a folder at depth 5+ the backend returns an error which I don't think can be patched with a module.
So I guess this module is dead. Sorry about that.

My brother who is far more knowledgeable than me helped get mine working.
This is what he gave me.
Step 1.
You can this as a replacement for the moar-folders java script file.
`Object.defineProperty(WorldCollection.prototype, 'maxFolderDepth', {
get: function () {
return 999;
}
});
const superPreCreate = Object.getPrototypeOf(Folder.prototype)._preCreate;
Folder.prototype._preCreate = function (data, options, user) {
superPreCreate.call(this, data, options, user);
};`
This change alone should allow you to view your folders, just not add new ones.
Step 2.
Change this file "FoundryVTT\Foundry Virtual Tabletop\resources\app\dist\database\documents\folder.mjs"
Open it and just use the Replace text command on CONST.FOLDER_MAX_DEPTH and replace it with whatever folder depth number you wish. After doing this, you will be able to create folders.

Step 1 can be done in a module.
Step 2 can't afaik.
If you're doing Step 2 I don't understand why you would need Step 1 in the first place.

Step 1 can be done in a module. Step 2 can't afaik. If you're doing Step 2 I don't understand why you would need Step 1 in the first place.

Right, one would have to be manually adjust it for each Foundry update. Only doing step 1 is still great to be able to recover the lost folders.
At least for me, step 2 does not work for folder creation unless I enable step 1. However, I am not knowledgeable myself enough to explain why or how.

The problem is that even if you change the limit on the client side the server does do a check when saving to the db so there is no way to do it as a module. The best solution I have found so far is running this script whenever I update foundy.
docker exec -i foundryv11 sed -i 's/FOLDER_MAX_DEPTH = 4/FOLDER_MAX_DEPTH = 6/' /home/foundry/resources/app/common/constants.mjs docker exec -i foundryv11 sed -i 's/FOLDER_MAX_DEPTH = 4/FOLDER_MAX_DEPTH = 6/' /home/foundry/resources/app/public/scripts/commons.js docker restart foundryv11


Error When Claim Bridge

Describe the bug
Dear Support
Error when claim bridge, but I receive ETH

https://test.taikoscan.io//tx/0x98ba462de60b14f14ef44fb713fa3ca514019d796ffa64297364d09637c881e0
My Address: 0xf3bad3aAabbDfb791DbcD2d3788D6Ac8EDD6bf8e
Thanks
Steps to reproduce
Steps to reproduce here.
Additional context
Additional context here.

@maltipoolabs Are you still facing this issue?

no, its okay now. Thanks


CSS layer update

This PR updates Oranda to match the new CSS approach as outlined in https://github.com/axodotdev/oranda/issues/59. It is the pair of https://github.com/axodotdev/turbo-axo/pull/175 and relies on https://github.com/axodotdev/turbo-axo/pull/176 being released and published to work fully.
To see what this looks like locally, replace the README with the oranda-js version. Creating a test file that can exist in this repo is on my list.
A few notes about the new CSS approach and its expansion into next steps; these are also mostly repeated in https://github.com/axodotdev/oranda/issues/59.

From this PR, we are loading the themes as a remote stylesheet hosted on unpkg (from our fringe npm package). For the next step, we'll need to make this dynamic.

This stylesheet defines three layers: @base (our standard fringe styles), @theme (the styles we provide), and @author, which we do not define. This last layer can be defined in axo-asset and will then take precedence over other layers automatically. (Bare CSS without a layer has the highest precedence.) We should include this in our docs and encourage user-defined themes, whenever they exist to implement the @author layer.

Currently, we rely on the theme name to apply dark or not to html to leverage tailwind class-based dark theme toggling. We will want to account for that when users provide us remote assets.

This approach completely removes scss. It barfs on @layer directives (because they are new, I assume) and is therefore incompatible at a base level with the way we are using these layers to manage precedence. More about layers ↬.

🎉


Fixed crop for valid region

Hi again,
like discussed in Issue#296 I fixed the cropping of the valid region method. The other methods still need to be fixed!
I hope it's right now.

It seems to me that if centering with respect to both axes is specified, but at least one of the shifts turns out to be zero, then this new code will still fail. I think, it might be a better idea to use slice objects instead of shifts, something like
        if delta0:
            shift = 1 + int(np.abs(delta0))
            slice0 = slice(shift, -shift)
        else:
            slice0 = slice(None)
        if delta1:
            shift = 1 + int(np.abs(delta1))
            slice1 = slice(shift, -shift)
        else:
            slice1 = slice(None)
        return centered_data[slice0, slice1]

However, as @stggh has pointed out in #300, the whole function needs to be revised...

However, as @stggh has pointed out in #300, the whole function needs to be revised...

Do we plan to make this revised function soon? If not, are we better off merging this PR for now and leaving #300 open to remind us to re-write the function later?

Do we plan to make this revised function soon?

Ask @stggh... :–)

merging this PR for now

I don't really like the current solution because it tries to hide problems inside nested ifs for some particular cases instead of addressing the root cause. Even with these ifs, trying to center an image that is already centered (both deltas = 0) will fail with a ValueError, since all the condition will be false.
As I understand, the only problem is that -None raises a TypeError instead of giving None, and this is what needs to be addressed instead of meddling with axes. Perhaps just replacing
return centered_data[shift0:-shift0, shift1:-shift1]

by
return centered_data[shift0:-shift0 if shift0 else None, shift1:-shift1 in shift1 else None]

can do the job.

And by the way, there should be no 1 + in shifts. Thus the whole branch
https://github.com/PyAbel/PyAbel/blob/f110ba34ad1e8e6bc920c40b402a3963527064b2/abel/tools/center.py#L264-L271
can be replaced by
    elif crop == 'valid_region':
        # crop to region containing data
        shift0 = int(np.abs(delta0))
        shift1 = int(np.abs(delta1))
        return centered_data[shift0 : -shift0 if shift0 else None,
                             shift1 : -shift1 if shift1 else None]

There are, however, similar problems with trying to do arithmetic with None for crop='maintain_data', so this function indeed needs to be revised substantially (and we must write tests for it). I'll look closer at @stggh proposal in #300, as it seems to be more sensible (in particular, [pad:size-pad] without Nones).

Ask @stggh... :–)

My apologies, my attention to this problem was diverted.
Cropping is a real pain, in the sense that I do not fully comprehend the cropping options:
    crop : str
        determines how the image should be cropped. The options are:

        ``maintain_size``
            return image of the same size. Some regions of the original image
            may be lost and some regions may be filled with zeros.
        ``valid_region``
            return the largest image that can be created without padding.
            All of the returned image will correspond to the original image.
            However, portions of the original image will be lost.
            If you can tolerate clipping the edges of the image, this is
            probably the method to choose.
        ``maintain_data``
            the image will be padded with zeros such that none of the original
            image will be cropped.

given that the function simply calls centered_data  = scipy.ndimage.interpolation.shift(data, (row_shift, col_shift)).
Why not pad the image for all cases and then crop to a PyAbel friendly nearest odd-column size? In which case centered_data = centered_data[padr:-padr, padc:-padc] will do the right thing, no need for any crop-options.

@stggh, I've answered in #300 (a more general topic).

This PR has been superseded by #302, correct?

Yes, and @Mobbybobby has confirmed that #302 solves the problems reported in #296. So now we can close this PR and that issue.

Great! Thanks again to @Mobbybobby for pointing out this issue and for @MikhailRyazanov for improving the code.


Get all messages returned by Kafka

First I would like to thank you for this great package. it is well-designed and the API is clean. It is a shame that it is less popular than other Kafka clients. I had your client added to the Kafka wiki to give it more visibility.
For now my only issue with your package is that I would like to fetch a batch of messages from Kafka and process that batch immediately. But Consumer.Consume() only returns one message while the other returned messages are buffered in c.msgbuf.
If I want to get all the other buffered messages I need to do complex things just to undo things you do in Consume(). So in short I would need a function like Consumer.ConsumeBatch([]*proto.Message, error).
What do you think? I can send a PR adding such a function if you want.

@zorkian already worked on this once https://github.com/optiopay/kafka/pull/41
Although it seems like obvious optimization, I'm not sure if removing mutex protecting Consume would help much. I think best would be to measure current Consume and proof of concept implementation with ConsumeBatch. If there will be visible difference, we can think about integrating it.

It is not the mutex that bothers me it is the fact that I cannot simply get all the messages returned by the fetch request.
Here is my use case:

Fetch a batch of messages from Kafka.
Process the batch.
Update the offset.
Repeat.

If I want to do that now, I must wrap Consume with a channel / select statement / timeout combo. Which is a bit shameful considering that I just want the result of the fetch request from Kafka.

I must admit that I don't quite understand your motivation. I do understand that you want to process data as it comes in kafka response, but I do not understand why.
Maybe that's very naive approach, but why not do this:
func Batch(c kafka.Consumer, max int) ([]*proto.Message, error) {
    var batch []*proto.Message
    for {
        if len(batch) == max {
            return batch, nil
        }
        m, err := c.Consume()
        if err != nil {
            return batch, err
        }
        batch = append(batch, err)
    }
}

To avoid request/response delays, Batch could work in the background, preparing batch sets constantly and pushing then through chan []*proto.Message once they are ready.

There are issues with this method. For example if max is big, your Batch function might hang a long time until Kafka receive enough messages for batch to be full.
Of course I could tweak your function to get what I want, but basically I would just reimplement the Kafka parameters like MaxWaitTime, MinBytes and MaxBytes .
Kafka provides a great way to precisely get the behavior I want with these parameters and I feel Consume prevents me from getting that precise behavior in my programs.

I understand that you want it, but can you explain why?
I've built an entire consumer coordination library (https://github.com/dropbox/marshal) on top of the current behavior and it works well. I did experiment with batching but ultimately could not get any performance improvements from it. I also could not prove to myself that the semantic improvements were worth it.
The only scenario I could imagine is if you're going to build something that is going to pass batches of messages to another system. In that case, it's probably cleaner to have Kafka give you the exact batch you want so you can just pass it on. Is that what you're doing?

Yes I'm passing my batch to a service that process the messages and send a batch of commands to Redis. I don't want to implement multiple batching strategies while Kafka provides exactly what I need.
But if you think adding that feature while preserving backward compatibility is too complicated, I'll just fork the package and simply update Consume to return the slice of messages returned by fetch. No problem!

Not complicated. Just needed to know why something is wanted to know
if something is needed.
Up to @husio, but this is the one use case that I could think of that
makes sense for batch preservation. It would be better not to fork since
this is a smaller project, IMO.
--
Mark Smith
<EMAIL_ADDRESS>On Fri, Jan 22, 2016, at 11:47 AM, Alexandre Cesaro wrote:

Yes I'm passing my batch to a service that process the messages and send a batch of commands to Redis. I don't want to implement multiple batching strategies while Kafka provides exactly what I need.

But if you think adding that feature while preserving backward compatibility is too complicated, I'll just fork the package and simply update Consume to return the slice of messages returned by fetch. No problem!

—
Reply to this email directly or view it on GitHub[1].

Links:

https://github.com/optiopay/kafka/issues/50#issuecomment-174024769

Although I cannot see any use case for this functionality myself, I think we should give it a try. @alexcesaro do you have any implementation details in mind? Do you want to reuse consumer structure defined in broker.go?

After reading #41, I would say that we could do as you said in this comment:

extend consumer with ConsumeBatch method and add BatchConsumer method to Broker (not Client interface) that will return consumer as new BatchConsumer interface

Also zorkian's implementation of ConsumeBatch should be updated to directly return c.msgbuf if it is not empty.
Otherwise have you thought about versioning the API using a service like gopkg.in? I don't know if Kafka 0.9 brings any backward-incompatible changes but a future version might. So versioning the API might be mandatory someday.

@alexcesaro I think this is good idea. No matter what will be your proposal, please do not break existing API. Other than that, I do not have any requirements.
Personally, I do not find gopkg.in helpful. I was also not looking into changes that 0.9 introduce, but no matter what were the changes, I will always try to keep the API compatible with previous releases.
I think I will worry about API compatibility more once it will become a real problem for this library ;)


AAD DEMO

Coverage decreased (-6.8%) to 45.863% when pulling 43a7f8e520775e515418cf68fb6ace0062bad535 on aad-dev-branch into 7ff5fea371d607247772cc91143518b1e1e838af on submit-job-e2e-demo.


Revert back to relying on ActiveRecord for race condition

executing notification_exists?(params) doesn't guarantee mutual exclusion.
Our threads a, b have the following format:
1) if notification_exists?(params) -> exit
2) notification = AdyenNotification.build(params)
3) notification.save!

If thread a executes line 1 it will move to line 2 as no notification exists right now. After that thread b executes line 1, it will also move to line 2 as no notification exists right now. Therefore, we now have two threads in an unsafe state.
Relying on indexes for uniqueness is not nice. I believe we can also use find_or_create_by, or use save without bang and to check for errors (not nicer)

I've manually run the tests and they all passed


docs(zh): update

https://github.com/vuejs/vue-router/compare/a8fe4fc...11e779a
/cc @Justineo for review. Thanks.

ping @Justineo


Add content attributes to collapsibles

What is the context of this PR?
This is to allow content attributes to be set on collapsibles. This is to allow data-qa attributes to be set on the content in runner to allow selectors to use them for tests
How to review
Tests pass, can set an attribute on a collapsible

 Replace chained selectors in runner tests with data attribute selectors (S)


[Feature] Unify load observation interface through information_schema.loads & statistics.loads_history

Why I'm doing:
mysql> select * from information_schema.loads where type = 'BROKER' order by scan_rows desc limit 1 \G;
*************************** 1. row ***************************
                  ID: 11002
               LABEL: lineorder_1720593471
          PROFILE_ID: NULL
             DB_NAME: ssb
          TABLE_NAME: lineorder
                USER: root
           WAREHOUSE: NULL
               STATE: FINISHED
            PROGRESS: ETL:100%; LOAD:100%
                TYPE: BROKER
            PRIORITY: NORMAL
           SCAN_ROWS: 6001215
          SCAN_BYTES: 598754908
       FILTERED_ROWS: 0
     UNSELECTED_ROWS: 0
           SINK_ROWS: 6001215
     RUNTIME_DETAILS: {"backends": {"cd6d9391-605a-4bc3-a3ca-dd95cdb96877": [10001, 10213, 10166]}, "file_size": 588311786, "load_id": "cd6d9391-605a-4bc3-a3ca-dd95cdb96877", "task_num": 1, "txn_id": 53, "unfinished_backends": {"cd6d9391-605a-4bc3-a3ca-dd95cdb96877": []}}
         CREATE_TIME: 2024-07-10 14:37:51
     LOAD_START_TIME: 2024-07-10 14:37:55
    LOAD_COMMIT_TIME: 2024-07-10 14:38:09
    LOAD_FINISH_TIME: 2024-07-10 14:38:09
          PROPERTIES: {"max_filter_ratio": 0, "timeout": 14400}
           ERROR_MSG: NULL
        TRACKING_SQL: NULL
REJECTED_RECORD_PATH: NULL
              JOB_ID: 11002
1 row in set (0.03 sec)

What I'm doing:
Fixes #46775
What type of PR is this:

[ ] BugFix
[x] Feature
[ ] Enhancement
[ ] Refactor
[ ] UT
[ ] Doc
[ ] Tool

Does this PR entail a change in behavior?

[ ] Yes, this PR will result in a change in behavior.
[x] No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

[x] Interface/UI changes: syntax, type conversion, expression evaluation, display information
[ ] Parameter changes: default values, similar parameters but with different default values
[ ] Policy changes: use new policy to replace old one, functionality automatically enabled
[ ] Feature removed
[ ] Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

[x] I have added test cases for my bug fix or my new feature
[x] This pr needs user documentation (for new or modified features or behaviors)

[ ] I have added documentation for my new feature or new function

[ ] This is a backport pr

Bugfix cherry-pick branch check:

[x] I have checked the version labels which the pr will be auto-backported to the target branch

[ ] 3.3
[ ] 3.2
[ ] 3.1
[ ] 3.0
[ ] 2.5

@mergify backport branch-3.3


feat: remove terraform to use tofu provided by spacelift

https://masterpoint.slack.com/archives/C035S6TE7PC/p1707825742435719

Closes #6


Unused model in model-config.json

This model is not present in the generator anymore: https://github.com/societe-generale/react-loopback-generator/blob/master/generators/server/templates/server/model-config.json#L36-L38
We should remove these 3 lines.

And the generated migration can be updated to remove the matching table too


Please clarify the relationship between syne-tune and ray:tune

Please clarify in readme.txt - Can syne-tune wrap any ray:tune scheduler/searcher? Which ray:tune version is used?
Is PBT supported on the SageMaker backend?

Hello, thanks for your interest in Syne Tune.
We can wrap any ray:tune scheduler/searcher which adheres to what is said in syne_tune/optimizer/schedulers/ray_scheduler.py. This is excluding such schedulers which make use of TrialRunner, as this is something deeply engrained in Ray, and we have different back-ends (which are altogether simpler).
Now, ray:tune PBT depends on TrialRunner, but we implemented our own version, check syne_tune/optimizer/schedulers/pbt.py and examples/launch_pbt.py. This version of PBT uses checkpointing. It should work on the sagemaker back-end.
Please do let us know if you into trouble.

Hi Gili,
To complement Matthias answer,

all schedulers/searchers of Ray should work except PBT and Optuna.
we are testing our code with the latest version of ray
PBT is not working at the moment with the SageMaker backend but we can add support for this.


Missing Histology in Patient History

Addresses CCSMCDS-186.
One of the pilot patients appeared to have a histology result that was being factored into the recommendation, but was not displaying in the patient history.
There was an update made last year to include a Diagnostic Report as a "histology diagnostic report" if the LOINC code isn't actually in the DiagnosticReport itself, but is contained within one of the Observations it references: https://github.com/ccsm-cds-tools/ccsm-cds-with-tests/pull/116. However, that PR updated the SortedBiopsyReport CQL expression within CollateManagementData.cql (link on GitHub). DashboardLibrary.cql, which controls what is displayed in the patient history section of the dashboard, was actually using a slightly different CQL expression, HistologyDiagnosticReports (link on GitHub).
This PR adds the logic that the last PR #116 had incorporated over in CollateManagementData.cql into this HistologyDiagnosticReports expression within DashboardLibrary.cql, so that this expression now includes not only the DiagnosticReports that have a "Biopsy Report" LOINC code, but also the DiagnosticReports that reference an Observation with this DiagnosticReport. I also added a test that mimics the case of the UMMC patient, to show that the DashboardLibrary.cql will now recognize a lab test that is formatted like this.

One more thing. I got the same error used to happen from clean copy of main, by that I don't think this is specific problem from this PR.
2188 passing (2m)
74 failing

ManageSpecialPopulation_v1.1.0
Histologic Cancer (25):
AssertionError: Recommendation=: expected null to deeply equal { short: 'Histologic Cancer', …(3) }
at checkResult (node_modules/cql-testing/src/buildTestSuite.js:131:38)
at Context. (node_modules/cql-testing/src/buildTestSuite.js:109:11)
at process.processImmediate (node:internal/timers:478:21)

ManageSpecialPopulation_v1.1.0
Histologic Cancer (31):
AssertionError: Recommendation=: expected null to deeply equal { short: 'Histologic Cancer', …(3) }
at checkResult (node_modules/cql-testing/src/buildTestSuite.js:131:38)
at Context. (node_modules/cql-testing/src/buildTestSuite.js:109:11)
at process.processImmediate (node:internal/timers:478:21)

....
Could be my own issue, I will look into this separately.

One more thing. I got the same error used to happen from clean copy of main, by that I don't think this is specific problem from this PR.
2188 passing (2m) 74 failing

ManageSpecialPopulation_v1.1.0
Histologic Cancer (25):
AssertionError: Recommendation=: expected null to deeply equal { short: 'Histologic Cancer', …(3) }
at checkResult (node_modules/cql-testing/src/buildTestSuite.js:131:38)
at Context. (node_modules/cql-testing/src/buildTestSuite.js:109:11)
at process.processImmediate (node:internal/timers:478:21)
ManageSpecialPopulation_v1.1.0
Histologic Cancer (31):
AssertionError: Recommendation=: expected null to deeply equal { short: 'Histologic Cancer', …(3) }
at checkResult (node_modules/cql-testing/src/buildTestSuite.js:131:38)
at Context. (node_modules/cql-testing/src/buildTestSuite.js:109:11)
at process.processImmediate (node:internal/timers:478:21)

....
Could be my own issue, I will look into this separately.

I believe this is related to what I mentioned in this Slack thread: https://mitrecorp.slack.com/archives/C017T4K3M42/p1729577281155429. Since this branch is based off of a version of the pilot branch from before https://github.com/ccsm-cds-tools/ccsm-cds-with-tests/pull/147 was merged, you would likely still be seeing the failing tests as a result of the value sets not yet being updated.
As a note, you can test just the specific test folder that contains the test I added by changing line 6 of test/test.js to the following:
cqlt.test(path.join(__dirname, 'test/DisplayCervicalCancerMedicalHistory'));

One more thing. I got the same error used to happen from clean copy of main, by that I don't think this is specific problem from this PR.
2188 passing (2m) 74 failing

ManageSpecialPopulation_v1.1.0
Histologic Cancer (25):
AssertionError: Recommendation=: expected null to deeply equal { short: 'Histologic Cancer', …(3) }
at checkResult (node_modules/cql-testing/src/buildTestSuite.js:131:38)
at Context. (node_modules/cql-testing/src/buildTestSuite.js:109:11)
at process.processImmediate (node:internal/timers:478:21)
ManageSpecialPopulation_v1.1.0
Histologic Cancer (31):
AssertionError: Recommendation=: expected null to deeply equal { short: 'Histologic Cancer', …(3) }
at checkResult (node_modules/cql-testing/src/buildTestSuite.js:131:38)
at Context. (node_modules/cql-testing/src/buildTestSuite.js:109:11)
at process.processImmediate (node:internal/timers:478:21)

....
Could be my own issue, I will look into this separately.

I believe this is related to what I mentioned in this Slack thread: https://mitrecorp.slack.com/archives/C017T4K3M42/p1729577281155429. Since this branch is based off of a version of the pilot branch from before #147 was merged, you would likely still be seeing the failing tests as a result of the value sets not yet being updated.
As a note, you can test just the specific test folder that contains the test I added by changing line 6 of test/test.js to the following:
cqlt.test(path.join(__dirname, 'test/DisplayCervicalCancerMedicalHistory'));

Yes, it worked well when the test was run separately.


Fix bug preventing combat type switch

Fixes an issue in 0.27 where clicking the combat type switch button would erroneously call the routine to handle timed effects and throw an error.

crashes if undefined.
if (!changed.flags?.sfrpg?.combatType && options.eventData) {
this._handleTimedEffects(options.eventData);
}

Thanks for the catch, fixed


Request support for Aeratron Ceiling Fan

Log message
This error originated from a custom integration.

Logger: custom_components.tuya_local.config_flow
Source: custom_components/tuya_local/config_flow.py:393
integration: Tuya Local (documentation, issues)
First occurred: 11:00:30 AM (1 occurrences)
Last logged: 11:00:30 AM

Device matches mellerware_citymove_vacuum with quality of 60%. DPS: {"updated_at":<PHONE_NUMBER>.2377574, "1": false, "15": false, "16": 0, "101": "0", "102": "TimeOff"}

Product ID
iarwcyttkv1kvent
Product Name
Aeratron AE3+
DPS information
Product name that appears in Tuya is "SYB-SYC"

primary_entity:
  entity: fan
  dps:
    - id: 1
      type: boolean
      name: Switch
    - id: 3
      name: Speed
      type: integer
      range:
        min: 1
        max: 6
    - id: 8
      name: Direction
      type: string

Information about how the device functions
https://en.aeratron.io/products/ae3-series
I imagine the whole family of Aeratron fan models are the same (given a standard 'optional wifi control unit' has to be ordered for any of them to be wifi compatible, and it's the same wifi controller unit for all), so may not want to make it specific to Aeratron's "AE3+" model

The dps log does not match with the config you have provided instead of DPS information. Are you able to log in to platform.tuya.com to collect more detailed info?

Ah! Just ran the script and then exportcodes(). This is the export results:
{"1":"Switch","3":"Speed","8":"Direction Fan","15":"Light","16":"Brightness","101":"FAN","102":"Timer","103":"FAN_value"}
Is there anything else I can provide? (Tuya now seems to require a subscription for some of the other introspections).

You should be able to get to Cloud / API Explorer with the basic Free subscription, and get the QueryThingsDataModel result (under Device Control, I think).  That is the one highlighted in the New Device template as being most useful. I don't know where you found this script with "exportcodes()", but the info it is returning is not enough to make a config with.

Thanks! Found it. But still needs subscription (see screenshot). I've asked for my trial to be extended.

However, looks like it conforms to the generic fan profile, with dps values 1 (switch), 3 (speed: 1 to 6), 8 (direction: forward/reverse)
The script + exportcodes() instructions are per this link: https://www.zigbee2mqtt.io/advanced/support-new-devices/03_find_tuya_data_points.html (from your readme).


Discuss: File size of distributable for the browser

I wanted to open a discussion about the direction we're headed with regard to file size.  It's come up recently in talking to StackOverflow, but otherwise I don't hear much about it.
First some history
In terms of "absolute" relative size we've grown quite a bit in the past year or so.  All numbers are stated in terms of gzipped size.

When I came on board in Sept 2019: ~20kb :common subset, gzipped
Today: ~37kb :common gzipped

We've close to doubled the size of our common distributable.  Yet  in that time we also added several new languages to :common also:

Go
Kotlin
Less and SCSS
Lua
Rust
Swift
Typescript

33% of our size increase comes from just these new languages (i.e. ~20kb to 28kb).  The rest comes from numerous grammar improvements, parser improvements, etc...
Here and Now / My Thoughts
We have our new "higher fidelity" initiative in #2500.  Both 37kb and 20kb seem tiny to me.  Yes, it's possible to build much larger builds.   The full library (with every grammars) weighs in at a whopping 272kb.
All the feedback I see here on issues is of the "please, better highlighting, highlight more, highlight better" variety... I can't remember anyone pushing back with "the library is too large, make it smaller".  I wanted to open the topic to see if anyone has any thoughts on this.
Personally I feel that our situation now is good and that increasing the bundle even 30-40%  would be a win if we end up with much more nuanced highlighting at a result.  (I don't think the size will actually increase that much though.)  I don't see how we can keep the size the same as we pursue higher fidelity and more nuanced highlighting.  Many of the recent "language reboots" (LaTex, Mathematica, etc) have seen huge improvements in those grammars - but also a significant increase in the grammar size.
I still think a very "popular" use for Highlight.js is on a small website/blog where one is using a subset of the languages, not a full build (or anything even close).  Then on the other end you have huge sites like Discourse and StackOverflow building larger bundles.  In those cases I think the right solution (if size becomes a problem) is to lazy-load the grammars on demand.  Which we've always made easy to do and it just got easier with my PR to eliminate all run-time dependencies between languages.
A good portion of our size is keyword bundles... There has been talk recently of whether (in some languages) we could detect CamelCaseClassThingy rather than a hard-coded list... and while we could do that it (removing some keywords) it would have detrimental effects on our auto-detection capabilities which for many languages is highly dependent on large keyword lists.
Also, there are other highlighters.  It's always been my advice that if "small size" is a key requirement for someone that Prism might be a better choice as they tend to rely a lot more on tighter regex, simpler grammars, and dependency stacking... which helps them keep the size of each grammar smaller.
So I see our library continuing to grow slowly in size with every new release... and continuing to highlight with more nuance... with of course continued improvements to the parser and auto-detect when possible.

Does anything think this is the wrong direction?
Should we have some sort of size cap on 1st party languages?
Any other thoughts?

Note: Currently every language is built as a stand-alone module - which hurts our non-compressed size  - since some dependency modules end up being duplicated in the source... this should have less bearing on the final gzip size though and there are also plans to fix this in the future (when using the official build system to build a monolithic distributable).

I have a take on this, but it might be ignorant on how people are actually using highlight.js, but here it is: Recommend using ESM and stop caring about the bundled size. If we use a promise-based approach, we could use dynamic import (https://caniuse.com/es6-module-dynamic-import) to lazy load only language parsers the user is actually using and have a pretty clean API:
<link rel="stylesheet" href="/path/to/styles/default.css">
<!-- IE compatibility -->
<script src="/path/to/promise-polyfill.min.js" nomodule ></script>
<script src="/path/to/highlight.min.js" nomodule ></script>
<script nomodule >hljs.highlightAll().catch(console.error);</script>
<!-- Evergreen browsers -->
<script type="module">
import hljs from '/path/to/highlight.mjs';
hljs.highlightAll().catch(console.error);
</script>

Using this approach, only IE users would suffer from the bundle size.
We would have to make some change in the underlining API to deal with Promise and import():
import hljs from "./highlight.js/lib/core.js";

// registerLanguage should accept lazy loaded `Language`:
hljs.registerLanguage('c', () => import('./highlight.js/lib/languages/c.js'));
hljs.registerLanguage('d', () => import('./highlight.js/lib/languages/d.js'));

// the actual fetching of the language module can be triggered later by a user
// event – or they are never fetched if there are not actually needed.
someForm.addEventListener('submit', (ev) => {
  ev.preventDefault();
  // highlight now returns a Promise
  hljs.highlight(someForm.language.value, someForm.code.value)
    // the correct module is loaded only when we need it
    .then(result => {
      // update the highlighted HTML…
    })
    .catch(err => {
      // the Promise rejects if someForm.language.value doesn't correspond to a
      // registered language or an alias.
      alert('Unknown language?');
      console.error(err);
    });
});

I might be wrong, but I am under the impression that most highlight.js use a subset of the languages it provides (and sometimes only one). If we are moving to ESM on v11, I think we should do the extra work of having a Promise-based API, users would benefit from it.

Thanks for chiming in.  This is definitely all very interesting.  I had originally added lazy loading as a feature request but no one seemed to care about it, so I let it go.
A few thoughts:
-We don't support IE at all, so it's a non-concern.

Auto-detection requires all the bundled languages to be loaded and many installs that highlight user facing content (forums, etc) use auto-detection. So the benefit for those users would be far less (though perhaps interlacing the highlighting with the downloading would be a bit faster).  So someone setting this up "wrong" in the worst case could be adding 180+ fetch requests to their project without even realizing it...
Languages aliases and other meta-data (which are needed at runtime) require the language to first be loaded (to resolve the aliases).  I suppose this could be moved to registration time, but it seems annoying.
The biggest benefit here would be say a blog with a single language in a single post... in that case the single language needed would be downloaded and nothing more.
I think a lot of people bundle because it's a familiar and reliable legacy system - so I'm in no hurry to break those setups.  So I feel like we'd need to support both sync and async strategies, which adds complexity.
I really wish we had a beta channel (that was active) to allow us to test large architectural changes like this prior to just publishing and hoping for the best.
Prior history: https://github.com/highlightjs/highlight.js/issues/2405

Questions:

Is there a story for how import would mix with Subresource integrity protection?
What do bundlers (like Rollup, or others) do with all this dynamism?  Do they just flatten it so that promises immediately resolve and everything is still static and resolved during compile?
There are inter-grammar dependencies.  For example a large HTML file with a tiny snippet of JS at the end would not know it needed the JS grammar until it had parsed almost the entire file... would that be fine, or would those dependencies need to be bubbled to the surface so they could be fetched ASAP when an HTML highlight began? I also suppose not every HTML file includes JS... so perhaps you could argue the other direction also.

The original idea of upgrading npm while leaving the brower-side alone was appealing in that it:

it makes a pretty MAJOR change on a single side only (the server)
everyone using our pre-built CDN JS assets notices nothing (because they are identical)
it allows users to opt-in to using the npm modules in the browser via import

Being a tiny team with very limited resources we tend to be a bit risk adverse and typically that serves us well.

Note that I'm not suggesting we remove the bundle CDN browser script: I think we should keep it (for people who don't want to deal with ESM, or want to support older browsers, or don't want to deal with lazy-loading). What I'm suggesting is to let its size grow, consider it a non-concern, and nudge people to use the ESM version if they want to save some of their users' bandwidth.

Auto-detection requires all the bundled languages to be loaded and many installs that highlight user facing content (forums, etc) use auto-detection. So the benefit for those users would be far less (though perhaps interlacing the highlighting with the downloading would be a bit faster).  So someone setting this up "wrong" in the worst case could be adding 180+ fetch requests to their project without even realizing it...

My feeling is that web pages that do includes snippets of all 190 languages must be pretty rare, and that most pages must have something like 10 different languages tops – but I have no data to back this up though. But anyways, if your website is likely to contain all the languages, do use the bundled version, I agree lazy loading is not for you.

Languages aliases and other meta-data (which are needed at runtime) require the language to first be loaded (to resolve the aliases).  I suppose this could be moved to registration time, but it seems annoying.

True, although that's something we can solve at build time.

Is there a story for how import would mix with Subresource integrity protection?

Not really: https://stackoverflow.com/questions/45804660/is-it-possible-to-use-subresource-integrity-with-es6-module-imports. The Import Assertions proposal has reached stage 3, SRI may be added to the spec at a later time, but currently dynamic import() are unfiltered. <script type="module"> has the same SRI implementation as the nomodule one.

What do bundlers (like Rollup, or others) do with all this dynamism?  Do they just flatten it so that promises immediately resolve and everything is still static and resolved during compile?

They usually have an option to let the user chose (E.G.: for Rollup it's called inlineDynamicImports).

There are inter-grammar dependencies.  For example a large HTML file with a tiny snippet of JS at the end would not know it needed the JS grammar until it had parsed almost the entire file... would that be fine, or would those dependencies need to be bubbled to the surface so they could be fetched ASAP when an HTML highlight began? I also suppose not every HTML file includes JS... so perhaps you could argue the other direction also.

My take on this would be to be as lazy as possible, because it's easier 😅 for us to implement, and for the users to understand what the library is doing. We should document that if the user knows what languages they are going to use, they can preload the relevant modules:
<link rel="preload" href="/path/to/highlight.js/languages/js.js" as="script">
<link rel="preload" href="/path/to/highlight.js/languages/html.js" as="script">

<!-- or -->

<script type="module" href="/path/to/highlight.js/languages/js.js" integrity="…">
<script type="module" href="/path/to/highlight.js/languages/html.js" integrity="…">

If they can't be bothered, they can either accept the tradeoff of lazy-loading, or the file size of the browser bundle.

My feeling is that web pages that do includes snippets of all 190 languages must be pretty rare, and that most pages must have something like 10 different languages tops – but I have no data to back this up though. But anyways, if your website is likely to contain all the languages, do use the bundled version, I agree lazy loading is not for you.

I wasn't referring entirely to actual real-life webpages or use cases - but rather implimentor behavior, including the possibility of confusion and mistakes.

Millions of users use our default (common) build today, which is 39 languages and 41kb gzipped.
This will be getting smaller with v11 as we rip out some languages from common.
We prevent this 190 language "mistake" now by simply not shipping an "all_languages.js"
If someone really wants to build/package this manually (from source), then they can.
I have no idea how many of those 39 are "used at a time", but 41kb to me is not "heavy" at all IMHO.
StackOverflow adds quite a few additional to this and still comes in around only 80kb I think, which is not onerous (to most people)
And of course the penalty for loading all 190 with Node.js is much smaller since the code is all local and bandwidth is no issue.

So it seems the actual need here is for a tiny fraction of use cases:

Those that need LOTS of languages (where download size really starts to matter)
PLUS they aren't using auto-detect (so they could truly really take advantage of lazy load)

Given all that, I feel right now lazy-loading is best handled outside of core.
This issue was originally created following some discussions with Stack Overflow, who are extremely size sensitive.  It took 3 months before anyone else chimed in on the topic. I just don't think many people actually need this functionality (or care about size super strongly) - and even if I'm mistaken, I don't (at this time) see huge advantages to it being in core vs a plug-in/add-on.
It seems (esp. after we release an ESM npm package) one could very easily write a small "wrapper" package such as highlightjs-async that provided a custom index (with addl. metadata and async registration calls) and then replace/wrap key API functions with async versions:

highlight
highlightBlock
registerLanguage

I'd suggest this is even quite possible today without much effort using fetch instead of modules.

I just don't think many people actually need this functionality (or care about size super strongly) - and even if I'm mistaken, I don't (at this time) see huge advantages to it being in core vs a plug-in/add-on.

Fair enough, I think I share your POV on this, most people are fine with highlight.js the way it is. That doesn't mean we shouldn't try to improve it (I mean, that's why you opened this issue, right?), but apologies if my comments were off topic.

I don't (at this time) see huge advantages to it being in core vs a plug-in/add-on.

What if a magic PR came down your inbox that added support for lazy-loading in core? Are you open to be convinced, or would you rather wait for another time?

but apologies if my comments were off topic.

No need for an apology - I'd say you were definitely on-topic.

What if a magic PR came down your inbox that added support for lazy-loading in core? Are you open to be convinced, or would you rather wait for another time?

Well, it's not just about who does the initial work (and believe me I appreciate the willingness).  That's often only the tip of the iceberg.  Often the long-term support/maintenance of a feature can far outweigh any initial outlay of development time.  Our current trend has been towards shrinking and simplifying the core codebase and public API - while extending the plug-in API as it makes sense... things not critical to "core" functionality are then pulled out into plugins, etc.  The old but time tested "less is more" approach.
With our current tiny team a vibrant plugin community with individual authors who are passionate about maintaining and supporting their own particular plugins has a lot of benefits.

I like https://github.com/highlightjs/highlight.js/issues/2846#issuecomment-780788534 idea, dose anything like this ever got implemented?

Not fully.  Perhaps also see my thoughts just added to #3199.    We do currently ship everything as separate ESM modules: core, all the grammars, etc....  You can dynamically import them and register them if you write the code for it...  What doesn't exist is the glue to tie it all together into some type of highlightjs-async library...


dyld: missing symbol called in custom XCFramework with pod dependency

How frequently does the bug occur?
All the time
Description
Hi,
I'm trying package a custom xcframework with the RealmSwift pod dependency.
After upgrading to the latest version (10.32.0), when I import my xcframework into a test app, I'm getting the following error :
dyld[6505]: missing symbol called
And it point that method :

Any ideas on how to fix that?
Can you reproduce the bug?
Yes, always
Version
10.32.0
What SDK flavour are you using?
Local Database only
Are you using encryption?
No, not using encryption
Platform OS and version(s)
iOS 15.5
Build environment
Xcode version: 14.0.1
Dependency manager and version: Cocoapods 1.11.3

Hi @MikePetitfils Can you send us a reproduction app so that we can investigate further?

I has the same issue and was able to determine that it's broken in the 10.21.0, while 10.20.1 works fine. However to run 10.21.0 #7672 fix should be applied manually.
Unfortunately, I'm not able to reproduce the issue on a sample project.

Well finally I was able to build an example, here are steps to reproduce:

Switch to the branch 7989-repro in my fork
Navigate to the examples/installation/ios/swift/CocoaPodsExample
Run ./XCF/build.sh script
Open the CocoaPodsExample.xcworkspace and run the app on a Simulator
Observe the app is crashing with missing symbol called error
Change version of RealmSwift pod in examples/installation/ios/swift/CocoaPodsExample/XCF/XCF.podspec:21 to 10.20.1
Rerun the script and the app
Observe the app is not crashing

@leemaguire any update for the example and steps I've provided?

Hi @evnik I'm taking a look at this, and I'm able to reproduce. I also found out that the version that introduces the issue is 10.21.0. Now I have to check which commit causes the issue.

We've encountered the same issue. Would anyone happen to have a workaround for this problem?

I can only say that in the custom framework I had this code and it was crashing:
public static func getById(_ itemId: String, detached: Bool = true) -> MyObject? {     let result = dbRealm.objects(MyObject.self).where {         $0.itemId == itemId     }.first     return detached ? result?.detached() : result }
I have updated the code like this and I do not have the crash anymore:
public static func getById(_ itemId: String, detached: Bool = true) -> MyObject? {     let results = dbRealm.objects(MyObject.self).where {         $0.itemId == itemId     }     if results.isEmpty {         return nil     }     return detached ? results[0].detached() : results[0] }
This does not make much sense to me because the code is really the same. But accessing the results array using the index instead of the first solved the issues for me. The attached() method is an utility extension I made to have a reference to the database object that I can modify as I want that does not affect the database.

same issue here

Facing similar issue with one of our library that uses realm which is integrated as part of a native module for a React Native project. Will hit this missing symbol error on first launch of the app but subsequent launches are surprisingly fine.

Hi,
I am facing the same issue when I use RealmSwift latest version 10.44.0 as a pod dependency in my custom XCFramework. For me the old version 10.19.0 only works. By going through all the messages It seems that this issue introduced in 10.21.0 onwards. Any idea how this could be resolved?

I'm facing the same issue with my library packed as xcframework and Realm & RealmSwift installed as dependencies via cocoapods.
Build environment:
Xcode:  15.2
iOS: 17.2
Realm: 1.47.0
RealmSwift: 1.47.0
Cocoapods: 1.14.3
I tried to set BUILD_LIBRARY_FOR_DISTRIBUTION to NO and build xcframework with -allow-internal-distribution paparameter as was suggested in similar issue. Did not help.
Is there any plan to fix it? Does anyone have workaround?

I have a similar issue of missing symbol when I run my app using a framework generated by me that depends on Realm (10.46.0 or any recent version).
I found a workaround to make it working that is based on using xcframeworks in modified podspec. So I took the podspecs for Realm and RealmSwift, modified them to depend on xcframeworks published by Realm as releases and put the podspecs in a repo. For the edited podspec I removed the dependency for source files and related stuff, and set
s.source                  = { :http => 'https://github.com/realm/realm-swift/releases/download/v10.46.0/Carthage.xcframework.zip' } # for both
s.vendored_frameworks  = 'core/realm-monorepo.xcframework', 'Realm.xcframework' # for Realm
s.vendored_frameworks  = 'RealmSwift.xcframework' # for RealmSwift

You can look at my repo. More can be optimized and I'm sure there are fields that can be removed, but it works at the moment.
Then the custom version must be specified where needed, so for each podspec depending on Realm I use
s.dependency 'RealmSwift', '10.46.0-xcframework'.
In Podfile remember to add the source with the url to the specs.

Thanks @themoonlitknight I will try it out. BTW I came across this post https://hackernoon.com/cocoapod-as-xcframework-with-dependencies that explains putting all XCFramework into one zip and distribute by using this
s.vendored_frameworks = 'MyFramework.xcframework',
'Realm.xcframework' ,
'RealmSwift.xcframework'

Hello! Any leads on this? We're also still facing this issue with any version above 10.20.1.

@amirpervaizQI Hi, do you currently have any available solutions for this issue?

@HowardCsie this could be the solution  you can create a XCframework manually and add to your app. The issue seems be to be related to Xcode or linker issue that throws symbols not found. you can find the XCFramework from Releases


[context] Decorators have type error with multiple versions of @lit/reactive-element (Unable to resolve signature of property decorator when called as an expression)

Which package(s) are affected?
Context (@lit/context)
Description
npm can have trouble properly deduplicating @lit/reactive-element dependencies and seems to nest an older version of it under @lit/context when it's required by some other dependency.
for example npm ls @lit/reactive-element could have
├─┬<EMAIL_ADDRESS>│ └──<EMAIL_ADDRESS>└─┬<EMAIL_ADDRESS>  ├──<EMAIL_ADDRESS>  └─┬<EMAIL_ADDRESS>    └──<EMAIL_ADDRESS>
causing this error
Unable to resolve signature of property decorator when called as an expression.
  No overload matches this call.
    Overload 1 of 2, '(protoOrDescriptor: Interface<ReactiveElement>, name?: "foo" | undefined): any', gave the following error.
      Argument of type 'MyElement' is not assignable to parameter of type 'Interface<ReactiveElement>'.
        Types of property 'renderRoot' are incompatible.
          Type 'HTMLElement | DocumentFragment' is not assignable to type 'HTMLElement | ShadowRoot'.
            Type 'DocumentFragment' is not assignable to type 'HTMLElement | ShadowRoot'.
              Type 'DocumentFragment' is missing the following properties from type 'ShadowRoot': delegatesFocus, host, mode, onslotchange, and 11 more.
    Overload 2 of 2, '(value: ClassAccessorDecoratorTarget<Interface<ReactiveElement>, unknown>, context: ClassAccessorDecoratorContext<Interface<ReactiveElement>, unknown>): void', gave the following error.
      Argument of type 'MyElement' is not assignable to parameter of type 'ClassAccessorDecoratorTarget<Interface<ReactiveElement>, unknown>'.
        Type 'MyElement' is missing the following properties from type 'ClassAccessorDecoratorTarget<Interface<ReactiveElement>, unknown>': get, setts(1240)

Reproduction
.
Workaround
Workaround for now is to force a particular version by using npm overrides (or equivalent like yarn resolutions, pnpm overrides).
// package.json
{
  "overrides": {
    "@lit/reactive-element": "^2.0.0"
  }
}

or just for @lit/context
// package.json
{
  "overrides": {
    "@lit/context": {
      "@lit/reactive-element": "^2.0.0"
    }
  }
}

Is this a regression?
Yes. This used to work, but now it doesn't.
Affected versions
1.0.0
Browser/OS/Node environment
n/a

Thanks for the worksaround! I ran into this today with lib-labs/router, mwc-drawer, two older components I was using in an updated Lit3 app. But I guess this is hard to problem to "fix".


Archetype templates scoped model is not merged with global model

fix https://github.com/helidon-io/helidon-build-tools/issues/877

Thanks for approval. I would like to move this PR https://github.com/helidon-io/helidon-build-tools/pull/923 before this one. Because IMO it makes more sense to have root scope removed there than a part of it in this PR.


AKAudioFile exportAsynchronously calls callback with error but continues and calls callback again

See this line: https://github.com/AudioKit/AudioKit/blob/master/AudioKit/Common/Internals/Audio File/AKAudioFile%2BProcessingAsynchronously.swift#L402
It calls the completion handler with an error but continues and successfully exports the file and calls the completion handler again.
The method documentation states that the callback will be called when export is completed:
"callback: AsyncProcessCallback function that will be triggered when export completed."

I don't mind making a pull request for this. Will do that today.
The underlying problem seems to be that the AKAudioFile reports a length & sampleCount of 0. Not sure how to fix that. I now changed my code to use recorder.audioFile instead for exporting.

Yeah, I am no fan of AKAudioFile anymore, it is a target for refactoring, but of course, we will at least wait until WWDC to see what gifts / kicks in the nuts are in store first.

I'm trying to push some code to a branch but it appears I have no rights to do this. Error on my side or do I need to be given permissions?

Reopen if the fix from the PR isn’t sufficient.  Thank you!


Upgrade to Spring Boot 3.3.0

I'd change this issue to a title of "Add dependabot integration".  Where we then just add Dependabot into our GitHub workflow.  It will automatically create PR's on its own for us to review, and very easy to selectively merge parts or not and tell it to delay until later.  We do this on OpenRefine project.  Want me to add that in...oh wait, I can't I think because it needs owner approval I think on token exchange.  So you can do that.
https://docs.github.com/en/code-security/dependabot/working-with-dependabot
but start here:
https://docs.github.com/en/code-security/dependabot/dependabot-alerts/about-dependabot-alerts


[WSL2] Issue with update to Windows 20152?

Environment
Microsoft Windows [Version 10.0.20152.1000]

Steps to reproduce

Install WSL1 on Windows 10 Pro x64 pre 20152 update.
Upgrade to WSL2 and set WSL2 as default
Update kernel as requested
Run WSL Distribution of choice without issues.
Update to Windows Insider Preview 20152.
Get kernel issue, follow instructions at aka.ms/wsl2kernel
Get the following error message:

Expected behavior
WSL2 should work as kernel already have been upgraded.

Actual behavior
WSL2 says to update kernel, but installation says WSL isn't enabled (See screenshot above).

variation #5509
wsl.exe --update is the new "Windows Subsystem for Linux Update Setup" installer msi. Announcement here.

variation #5509
wsl.exe --update is the new "Windows Subsystem for Linux Update Setup" installer msi. Announcement here.

Tried searching for changes both here and on the web, I have to up my search-skills for sure. :|
Fix as described in https://github.com/microsoft/WSL/issues/5509#issuecomment-651650720 helped for me.
Thanks for your time. Closing issue.


something changed in 1.2.2 default config?

I just started a new container with 1.2.2 and my old startup.env config.
the db and config got created but then slap.d never started. Using the same setup just the old image 1.2.1 worked.

Hi, I am a new user and in my testing I think I stumbled upon this as well. While the first start of the container works, following starts of the container fail with an error that he could not find /etc/ldap/slapd.conf. For me the fix was to add
RUN touch /etc/ldap/slapd.conf

to my local Dockerfile (needed anyways as I use that to add users to the ldap tree).
Have not made any further tests with it though.

Finally it occurred again:
ldap          | *** Shutting down /container/run/process/slapd/run (PID 515)...
ldap          | 5be48257 daemon: shutdown requested and initiated.
ldap          | 5be48257 slapd shutdown: waiting for 0 operations/tasks to finish
ldap          | 5be48257 slapd stopped.
ldap          | *** Run commands before finish...
ldap          | *** Killing all processes...
ldap          | *** CONTAINER_LOG_LEVEL = 4 (debug)
ldap          | *** Set environment for startup files
ldap          | *** ignore : LANG = en_US.UTF-8 (keep LANG = en_US.UTF-8 )
ldap          | *** ignore : LANGUAGE = en_US.UTF-8 (keep LANGUAGE = en_US:en )
ldap          | *** Environment files will be proccessed in this order :
ldap          | Caution: previously defined variables will not be overriden.
ldap          | /container/environment/99-default/default.yaml
ldap          |
ldap          | *** --- process file : /container/environment/99-default/default.yaml ---
ldap          | *** Run commands before startup...
ldap          | *** Running /container/run/startup/:ssl-tools...
ldap          | *** ------------ Environment dump ------------
ldap          | *** LANG = en_US.UTF-8
ldap          | *** LDAP_LOG_LEVEL = 256
ldap          | *** LC_ALL = en_US.UTF-8
ldap          | *** CONTAINER_SERVICE_DIR = /container/run/service
ldap          | *** LDAP_READONLY_USER_PASSWORD = kopano123
ldap          | *** LANGUAGE = en_US:en
ldap          | *** LDAP_ORGANISATION = "Kopano Demo"
ldap          | *** HOSTNAME = c5bf0cd85929
ldap          | *** CONTAINER_STATE_DIR = /container/run/state
ldap          | *** LC_CTYPE = en_US.UTF-8
ldap          | *** LDAP_ADMIN_PASSWORD = kopano123
ldap          | *** LDAP_READONLY_USER = true
ldap          | *** INITRD = no
ldap          | *** CONTAINER_LOG_LEVEL = 4
ldap          | *** LDAP_DOMAIN = kopano.demo
ldap          | *** PATH = /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ldap          | *** HOME = /root
ldap          | *** LDAP_BASE_DN = dc=kopano,dc=demo
ldap          | *** ------------------------------------------
ldap          | *** Running /container/run/startup/slapd...
ldap          | *** ------------ Environment dump ------------
ldap          | *** LANG = en_US.UTF-8
ldap          | *** LDAP_LOG_LEVEL = 256
ldap          | *** CONTAINER_SERVICE_DIR = /container/run/service
ldap          | *** LDAP_READONLY_USER_PASSWORD = kopano123
ldap          | *** LANGUAGE = en_US:en
ldap          | *** LDAP_ORGANISATION = "Kopano Demo"
ldap          | *** HOSTNAME = c5bf0cd85929
ldap          | *** CONTAINER_STATE_DIR = /container/run/state
ldap          | *** LC_CTYPE = en_US.UTF-8
ldap          | *** LDAP_ADMIN_PASSWORD = kopano123
ldap          | *** LDAP_READONLY_USER = true
ldap          | *** INITRD = no
ldap          | *** CONTAINER_LOG_LEVEL = 4
ldap          | *** HOME = /root
ldap          | *** LC_ALL = en_US.UTF-8
ldap          | *** LDAP_DOMAIN = kopano.demo
ldap          | *** LDAP_BASE_DN = dc=kopano,dc=demo
ldap          | *** PATH = /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ldap          | *** ------------------------------------------
ldap          | 1048576
ldap          | *** Set environment for container process
ldap          | *** ignore : LANG = en_US.UTF-8 (keep LANG = en_US.UTF-8 )
ldap          | *** ignore : LANGUAGE = en_US.UTF-8 (keep LANGUAGE = en_US:en )
ldap          | *** Environment files will be proccessed in this order :
ldap          | Caution: previously defined variables will not be overriden.
ldap          | /container/environment/99-default/default.yaml
ldap          |
ldap          | *** --- process file : /container/environment/99-default/default.yaml ---
ldap          | *** Run commands before process...
ldap          | *** ------------ Environment dump ------------
ldap          | *** LANG = en_US.UTF-8
ldap          | *** LDAP_LOG_LEVEL = 256
ldap          | *** CONTAINER_SERVICE_DIR = /container/run/service
ldap          | *** LDAP_READONLY_USER_PASSWORD = kopano123
ldap          | *** LANGUAGE = en_US:en
ldap          | *** LDAP_ORGANISATION = "Kopano Demo"
ldap          | *** HOSTNAME = c5bf0cd85929
ldap          | *** CONTAINER_STATE_DIR = /container/run/state
ldap          | *** LC_CTYPE = en_US.UTF-8
ldap          | *** LDAP_ADMIN_PASSWORD = kopano123
ldap          | *** LDAP_READONLY_USER = true
ldap          | *** INITRD = no
ldap          | *** CONTAINER_LOG_LEVEL = 4
ldap          | *** LC_ALL = en_US.UTF-8
ldap          | *** LDAP_DOMAIN = kopano.demo
ldap          | *** HOME = /root
ldap          | *** LDAP_BASE_DN = dc=kopano,dc=demo
ldap          | *** PATH = /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ldap          | *** ------------------------------------------
ldap          | *** Running /container/run/process/slapd/run...
ldap          | *** /container/run/process/slapd/run started as PID 21
ldap          | 1048576
ldap          | 5be48383 @(#) $OpenLDAP: slapd  (May 23 2018 04:25:19) $
ldap          |         Debian OpenLDAP Maintainers<EMAIL_ADDRESS>ldap          | 5be48383 could not stat config file "/etc/ldap/slapd.conf": No such file or directory (2)
ldap          | 5be48383 slapd stopped.
ldap          | 5be48383 connections_destroy: nothing to destroy.
ldap          | *** /container/run/process/slapd/run exited with status 1
ldap          | *** Run commands before finish...
ldap          | *** Killing all processes...

Hi, I'm seeing about the same issue with slapd.conf in another docker container, but I think it might be related to the docker version as well. I do not have the issue with docker 18.09.2 on fedora 29, but I do see it on ubuntu 16.04 with docker 18.03. Same image, same configuration. I can restart multiple times with no issues on docker 18.09.2 but first run on docker 18.03 on ubuntu gives:
5c75413b could not stat config file "/etc/ldap/slapd.conf": Permission denied (13)
From within the container, I can perfectly see the file both as root and as openldap user. Strange. Your workaround with 'touch' doesn't work for me. I will investigate further.

I found the cause in my case: if the directory to mount in the container on /etc/ldap on the host has the wrong permissions (read-only by root), you cannot see the files as openldap. The permissions on the host need to be at least "rwx-r-xr-x".


Official Request on behalf of those wanting to show some gratitude.

Where is your donate to coffee/beer fund button !?

Don't think it being vain but you should put a donate link with a service of your choice on both your GitHub and Path of Exile forum thread.

I've finally gotten around to creating a Patreon page, so you can help support the development of the program (or just throw money at me because I'm awesome :P)


Added uCrop'n'Edit link

Library based on uCrop with extra editing features - Brightness, Contrast, Saturation and Sharpness.

hi ,i am your fees,i am very happy to be a android.


double-difference using pyasdf

Hi Lion,
Wenjie and I are trying to make station-pair measurements using pyasdf process for double-difference tomography. Different from conventional way of processing one obsd and synt trace using process_two_files_without_parallel_output(), there are communications between traces. If we parallelize the process by pairs, the inputs will include two traces of obsd and two traces of synt. Do you think this can be realized by changing the reader in pyasdf to allow process of 4 traces simultaneously? And Is it problematic to read and process the same trace from different processors at the same time since a single trace can be repeatedly used in different pairs?
Thanks,
Yanhua

Hi,
sure - I think this can be done. For starters we could do it a very brute force way:

Get a list of all receivers that are common in both data and synthetics.
Calculate all possibilities to select two receivers.
For each permutations get the data.

The function might look like this:
from pyasdf.process import process_pairwise

def process_function(obs_1, obs_2, synth_1, synth_2):
    # Do stuff ...
    return stuff

process_pairwise(fct=process_function, obs_dataset_1, obs_dataset_2, synth_ds_1, synth_ds_2)

And Is it problematic to read and process the same trace from different processors at the same time since a single trace can be repeatedly used in different pairs?

No...parallel reading is rarely an issue - writing is the hard part and we don't have to do that here.
That being said - there is (as you already mentioned) quite some room to optimize as all traces will be used multiple times - but for starters the approach I layed out should be good enough to at least test everything.
What do you think?
Cheers!
Lion

Hi Lion,
Thanks for your comments.
process_pairwise() you mentioned above is exactly what I need to process pairs of data and synthetics. And MPI can be easily implemented for independent pairs, if parallel reading is not an issue. Let's first test it.
The workflow to make differential measurements from pairs of stations can be temporarily designed as follows:
## conventional stationwise process for preprocessing and window selection
from pyasdf.process import process_stationwise

def process_function(obs, synth):

    preprocess
    window_selection

return window

process_stationwise(fct=process_function,obs_dataset,synth_ds)

## pairwise process to calculate adjoint source
from pyasdf.process import process_pairwise

def process_function(obs_1, obs_2, synth_1, synth_2):

    calculate_adjoint_source
    return adj_1,adj_2

process_pairwise(fct=process_function, obs_dataset_1, obs_dataset_2, synth_ds_1, synth_ds_2)

Let me know your thoughts on it. Thanks.
Yanhua

I can give this a shot either tomorrow or Friday. As long as you don't need parallel output its fairly trivial to do.
I think in the long run you might also want to have parallel output to write the adjoint sources to a new ASDF - do you need that for now? Because that would be significantly more complex to implement.

Parallel output is not necessary right now. Thanks for your effort in helping me.
Yanhua

Hi Lion,
For the conventional pre-processing workflow, I think we also need the adjoint writer in the near future, even not recently.
One thing though, I am going to use process_two_files_without_parallel_output() and it will gather all the information to the master node. Not sure if this is true for adjoint sources, since the data communication will be large. But if we gather, we can then write out in serial.

Hi Wenjie and Lion,
For conventional adjoint source, you can have parallel output in the future. However, for double-difference, I need to gather all to the master node and sum over pair, before write into file.
Yanhua

One thing though, I am going to use process_two_files_without_parallel_output() and it will gather all the information to the master node. Not sure if this is true for adjoint sources, since the data communication will be large. But if we gather, we can then write out in serial.

Don't gather the adjoint sources in the main thread. For now just write the adjoint sources directly in the process_two...() method and maybe gather the misfit values and the windows.

For conventional adjoint source, you can have parallel output in the future. However, for double-difference, I need to gather all to the master node and sum over pair, before write into file.

The purpose of the function I proposed above was precisely to not gather everything on one node - but the proposed function would not work. After giving it some more thought, the following should work:
Assuming we have stations AA, BB, CC, and DD and two datasets, ds_obs and ds_synth.
from pyasdf.process import process_pairwise

def process_fct(obs_a, obs_b, synth_a, synth_b):
    # We don't want all differences - could also be done with a lookup table based
    # on cluster analysis or something along these lines.
    if get_distance(obs_a, obs_b) >= THRESHOLD:
        return None
    # Perform double difference adjoint calcs
    adj_src.write(...)
    return misfit, windows

results = process_pairwise(process_fct, ds_obs, ds_synth)

The process_fct() would be called six times, once for each possible combinations of stations:
AA + BB, AA + CC, AA + DD, BB + CC, BB + DD, CC + DD
The overhead of reading things multiple times should be fairly small and not a limiting factor. The crucial thing here is the adj_src.write() call which results in a lot of independent files to be written. Thus we probably want something like the normal process() method that can write in parallel to a single output file. I'll look into it tomorrow - I think it should work fine.

Does the proposition above work for you?

Hi Lion,
For conventional approach, I did not see any problem to write the adjoint sources directly in the process_two_files (), into independent files or a single file.
For double-difference, one way is to write adjoint sources independently in process_pairwise(), but this will result in plenty of files. For example, if we have 100 stations, pairwise process will result in 9900 files. After getting all files, we also need to sum files from the same stations, and reduce 9900 files to 100 files.
Another way we can do is to gather all adjoint sources in the main thread, sum over pairs, then write summed adjoint sources into 100 independent files or a single file as conventional approach does.
Which one will you try? Is there a better/easier way to do it?
Yanhua

Ah I see - I did not realize that. So all adjoint sources for one station in DD tomography will always be located at the same position? Makes some sense now that I think about it.
That actually makes it quite a bit easier as we really don't need parallel output in that case. I think the following should work. The processing function is called once for each station and with a way to get all the other stations. The downside of that approach is that it parallelizes at most to the number of stations but that should be sufficient I think and it makes for a simple implementation. It will also repeat some calculations and I/O is not optimal but otherwise we would have to create way more complex data flows.
from pyasdf.process import process_double_difference

def process_fct(station_name, obs, synth, other_stations_iterator):

    final_adj_source = None
    count = 0
    total_misfit = 0

    for other_obs, other_synth in other_stations_iterator:
        # We don't want all differences - could also be done with a lookup table based
        # on cluster analysis or something along these lines.
        if get_distance(obs, other_obs) >= THRESHOLD:
            continue

        # Perform double difference adjoint calcs and sum up.
        if final_adj_source is None:
            final_adj_source = calculate_adj_src
        else:
            final_adj_source += calculate_adj_src

        count += 1
        total_misfit += calculated_misfit

    final_adj_source /= count
    final_adj_source.write(...)
    return calculated_misfit

misfits = process_double_difference(process_fct, ds_obs, ds_synth):=

Do you think this would work?

Agree. Double the reading, but much less work.

Excellent, I can see this should work. It's more similar to stationwise process. The only difference is to get ready  all preprocessing, window selection, as well as pair information from cluster analysis. Thanks Lion. Look forward to testing it. Let me know.
Yanhua

Sorry for the delay - I was at a conference last week. I made this issue into a PR - please test if it works for you. If it does I'll add some more documentation and merge it into the master. Usage is like this:
import pyasdf
from pyasdf.processing_utils import process_each_with_each

def process_function(this_station_ds_a, this_station_ds_b,
                     other_stations_iterator):
    # This function will be called once for each station. `this_station_ds_a`
    # and `this_station_ds_b` are the station groups for that station from each
    # of the two data sets.

    # This loop now loops over all other stations.
    for other_station_ds_a, other_station_ds_b in other_stations_iterator:
        # `other_station_ds_a` and `other_station_ds_b` are the station groups
        # for the station from the inner loop. Thus each station will be
        # compared with each other station.
        #
        # Keep in mind that you will still need to check if each station has a
        # StationXML file and which tags to use and what not. If you only want
        # to process stations that are within X degrees with each other, just
        # use `this_station_ds_a.coordinates` and
        # `other_station_ds_a.coordinates` and compare them.
        #
        # Also remember to use your own exception handling! pyasdf will catch
        # exceptions at the process_function() level but if any exception
        # occurs with this loop it can do nothing will just jump to the next
        # outer station potentially missing out lots of things.
        print("Comparing station '%s' with station '%s'..." % (
            this_station_ds_a._station_name, other_station_ds_a._station_name))

# Opening the files read-only if fine for this purpose.
ds_a = pyasdf.ASDFDataSet("observed.h5", mode="r")
ds_b = pyasdf.ASDFDataSet("synthetic.h5", mode="r")

process_each_with_each(process_function, ds_a, ds_b)

Run with
mpirun -n 4 process_stuff.py

In case you are not that familiar with git:
cd /path/to/pyasdf
git fetch origin
git checkout -t origin/each_with_each_processing


Docs Sample use deprecated version (azure-iothub-device-client).

According to azure-iot-sdk-python: azure-iothub-device-client is from v1-deprecated (migration_guide)
But I cannot find any upload_blob method in azure-iot-device==2.0.0rc11.
Although azure-iothub-device-client has it, but installation on python:3.6 container gives error:

ImportError: libboost_python-py35.so.1.58.0: cannot open shared object file: No such file or directory

Do I have to use deprecated version and different container (e.g. ubuntu:16.04) to use IoT Hub blob upload?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 67ba5d41-7a78-bbfd-fb5f-bac5fcbccad1
Version Independent ID: 011d111e-c1cf-bb04-c749-b53f7c065833
Content: Upload files from devices to Azure IoT Hub with Python
Content Source: articles/iot-hub/iot-hub-python-python-file-upload.md
Service: iot-hub
GitHub Login: @robinsh
Microsoft Alias: robinsh

@michal-demecki-profil-software
Thanks for your feedback! We will investigate and update as appropriate.

@michal-demecki-profil-software Thanks for trying IoT Hub. Would you please open an issue with your question on the SDK repo? and share the link here?

@michal-demecki-profil-software closing this issue for now. Please feel free to share the link here, also if you do not get traction (which I doubt) please tag me here and I will try to help.


PDF import logic from API documentation.

Hello developers, may I know how to make the server process the PDF file and return the arraylist in your document.
Are there any documents?
We look forward to receiving your support.
Thanks a lot.

The server code is not open source and supports payment. Please contact us by email<EMAIL_ADDRESS>


Add Helm Deployments

What would you like to be added
helm charts
Why is this needed
Having helm charts will help in 1 command to startup darkroom, enabling more users to try, onboard, and improve darkroom. The helm chart will enable easy local development, integration, and onboarding for darkroom.
Comments
Just need to use the golang binary in the Dockerfile, and run the server with some pre-configured application, the buckets can be interfacing, or created using some gsutil commands. This will help darkroom to be used by more users.

@baskarap @ajatprabha let me know if this aligns with what you have in mind. I can pick this up next week.

We do release gojektech/darkroom images, I don't see any issue in adding helm charts.


Handle CSV datasets & better handle updates to existing MCF

This PR adds support for .csv files, by treating them like other vector datasets that GDAL supports.
A couple other things here:

Debugged some issues with overwriting existing MCF data when a MetadataControl is initialized. The goal is to preserve any metadata that might already exist, except for properties that are intrinsic to the dataset (i.e. CRS, bounding box, field names & datatypes).
changed some function keyword arg names to match the names of the MCF properties they represent. This makes it easy to use a pattern like mc_a.set_license(**mc_b.get_license())

Nice! I appreciate that we can just throw a CSV at GDAL and then use the same API.
Have you tested what happens if we pass GDAL a malformed CSV, like one with too many records on a line or something like that?

Good question! It seems like GDAL uses the header row as the source of truth for number of fields, and then it ignores extra records in other lines. That seems totally fine. I don't feel like we need a test to ensure this.


Add serializers required for near-sdk-rs

@iho Thanks for working on it quickly! I added a couple of more suggestions and I think we are close to get it done


Associate sdf::Element with line number in file

Original report (archived issue) by Anonymous.

Currently, during parsing, it is impossible to examine an arbitrary sdf::Thing (or sdf::Element) and provide any feedback on the element and link it to where it came from in the original source file. This definitely handicaps error and warning messages.

As an interim (pre-tinyxml2), it may possible to still provide more information by means of tracing with an XPath error.
I had tinkered with something like this in urdf_parser_py, inspired by some MathWorks-related code. Example:
https://github.com/ros/urdf_parser_py/blob/f7f25be76c5dfbe51d4e87367553ec180d8f5683/test/test_urdf_error.py#L124

FYI @aaronchongth


Take margin into account on max dimension

We need to take the margin into account if we clip on max dimension. Fixes #466.

@woehrl01 I noticed some potential mistakes when trying to get this change working with React Native 0.42. I left comments on the commit.

@rigdern You're absolutely right! Thanks a lot for having a look!


C# Debugger does not understand DateTime

Environment data
dotnet --info output:
.NET Command Line Tools (1.0.0-preview2-003121)

Product Information:
 Version:            1.0.0-preview2-003121
 Commit SHA-1 hash:  1e9d529bc5

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  10.11
 OS Platform: Darwin
 RID:         osx.10.11-x64

VS Code version:1.4.0
C# Extension version:1.4.1
Steps to reproduce
Debug a C# app and look at a DateTime object in Variables/Debug Console window
Expected  behavior
To show a DateTime object at correct locale and when in Debug Console doing something like (dateA - dateB).TotalSeconds to show correct results
Actual behavior

@jchannon thanks for the report! Good bug.
Here is a slightly simpler repro:
    public class Program
    {
        public static DateTime ReturnInput(DateTime input)
        {
            return input;
        }

        public static void Main(string[] args)
        {
            DateTime now = DateTime.Now;
            Console.WriteLine(now); // set breakpoint here
        }
    }

Set a breakpoint on the call to Console.WriteLine
Compare the value of now to ReturnInput now`

Result:
now
{09/08/2016 12:34:49}
ReturnInput(now)
{06/24/0001 02:59:04}
Expected:
Two values should be the same

@Mikem8361 I am guessing (though I don't have anything to back this up other than this seems similar to the previous bug with decimal) that this is a runtime bug with marshalling large struct input arguments.
Would you mind trying this (or emailing me instructions on how we can try this) with mdbg?

One thing it emphasise is if you look in the variable window the DateTime is in US format but when you expand the DateTime object it's shown in UK locale (where I am located) so it shows 09/08/2016 10:54:18 but then expanded shows that the Date property is 08/09/2016

@jchannon I don't think this matters - the issue is that when we pass the DateTime as an input argument to a function (subtraction operator in your example) the value is being corrupted as it passed to the function.

Maybe we have 2 issues here
On Thursday, 8 September 2016, Gregg Miskelly<EMAIL_ADDRESS>wrote:

@jchannon https://github.com/jchannon I don't think this matters - the
issue is that when we pass the DateTime as an input argument to a function
(subtraction operator in your example) the value is being corrupted as it
passed to the function.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/OmniSharp/omnisharp-vscode/issues/753#issuecomment-245719241,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGaphfCBUq2IrzRAUv0f4lig2oGteEbks5qoGedgaJpZM4J4WOv
.

The bug was with any 16 byte value type. The mdbg syntax is awkward something like "funceval namespace.class.get_Value instance".
Sent from my Verizon, Samsung Galaxy smartphone
-------- Original message --------
From: Jonathan Channon<EMAIL_ADDRESS>Date: 9/8/16 12:54 PM (GMT-08:00)
To: OmniSharp/omnisharp-vscode<EMAIL_ADDRESS>Cc: Mike McLaughlin<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [OmniSharp/omnisharp-vscode] C# Debugger does not understand DateTime (#753)
Maybe we have 2 issues here
On Thursday, 8 September 2016, Gregg Miskelly<EMAIL_ADDRESS>wrote:

@jchannon https://github.com/jchannon I don't think this matters - the
issue is that when we pass the DateTime as an input argument to a function
(subtraction operator in your example) the value is being corrupted as it
passed to the function.

You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/OmniSharp/omnisharp-vscode/issues/753#issuecomment-245719241,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGaphfCBUq2IrzRAUv0f4lig2oGteEbks5qoGedgaJpZM4J4WOv
.

You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/OmniSharp/omnisharp-vscode/issues/753#issuecomment-245719616, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJ-Fa0EWx-uLux2ZSSAaqqZkYo_nnmzTks5qoGfvgaJpZM4J4WOv.

@mikem8361 would the bug apply non-this input arguments?

No, only return values. On Linux and OS X, the calling convention has 16 byte values being returned in RCX:RDX unlike windows where a return value pointer is passed to method.
The funceval code wasn’t dealing with that case even though the runtime did.
mikem
From: Gregg Miskelly<EMAIL_ADDRESS>Sent: Thursday, September 8, 2016 2:08 PM
To: OmniSharp/omnisharp-vscode<EMAIL_ADDRESS>Cc: Mike McLaughlin<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [OmniSharp/omnisharp-vscode] C# Debugger does not understand DateTime (#753)
@mikem8361https://github.com/mikem8361 would the bug apply non-this input arguments?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/OmniSharp/omnisharp-vscode/issues/753#issuecomment-245741344, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJ-Fa7T3ioy5Y_aZE5dEslCbl02wGbRFks5qoHktgaJpZM4J4WOv.

@mikem8361 Yes, that is what I thought. I think this might be a new bug that affects input arguments of large structs. Its possible that it is VIL that is somehow setting up the func eval wrong, but this works correctly on Windows, and this isn't specific to DateTime -- I see the same issue if I make my own 16-byte struct.

@jkotas do we pass in 16 bytes into a method in registers on Linux/OS X like return values?
From: Gregg Miskelly<EMAIL_ADDRESS>Sent: Thursday, September 8, 2016 2:28 PM
To: OmniSharp/omnisharp-vscode<EMAIL_ADDRESS>Cc: Mike McLaughlin<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [OmniSharp/omnisharp-vscode] C# Debugger does not understand DateTime (#753)
@mikem8361https://github.com/mikem8361 Yes, that is what I thought. I think this might be a new bug that affects input arguments of large structs. Its possible that it is VIL that is somehow setting up the func eval wrong, but this works correctly on Windows, and this isn't specific to DateTime -- I see the same issue if I make my own 16-byte struct.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/OmniSharp/omnisharp-vscode/issues/753#issuecomment-245747159, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJ-FaxfuaI0Kl8JPRK7Mq8rJfx6x6CECks5qoH38gaJpZM4J4WOv.

@jkotas do we pass in 16 bytes into a method in registers on Linux/OS X like return values?
Correct.

Is the ENREGISTERED_PARAMTYPE_MAXSIZE the correct define? It seems to be 8 bytes on all x64 platforms (16 bytes on ARM64).
From vm\amd64\cgencpu.h:
#define ENREGISTERED_RETURNTYPE_INTEGER_MAXSIZE 8    // bytes
#define ENREGISTERED_PARAMTYPE_MAXSIZE          8    // bytes
#ifdef UNIX_AMD64_ABI
#define ENREGISTERED_RETURNTYPE_MAXSIZE         16   // bytes
#define CALLDESCR_ARGREGS                       1    // CallDescrWorker has ArgumentRegister parameter
#define CALLDESCR_FPARGREGS                     1    // CallDescrWorker has FloatArgumentRegisters parameter
#else
From: Jan Kotas<EMAIL_ADDRESS>Sent: Thursday, September 8, 2016 4:25 PM
To: OmniSharp/omnisharp-vscode<EMAIL_ADDRESS>Cc: Mike McLaughlin<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [OmniSharp/omnisharp-vscode] C# Debugger does not understand DateTime (#753)
@jkotashttps://github.com/jkotas do we pass in 16 bytes into a method in registers on Linux/OS X like return values?
Correct.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/OmniSharp/omnisharp-vscode/issues/753#issuecomment-245772977, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJ-Fax5omSwL6lahJ6odhV4aMSuEGzvnks5qoJk1gaJpZM4J4WOv.

The code for this is under FEATURE_UNIX_AMD64_STRUCT_PASSING.  It was done by @janvorli .

Looks like the bug repros under mdbg.
[p#:0, t#:0] mdbg> print now
now=System.DateTime <9/8/16 4:57:23 PM>
DaysToMonth365=array [13]
DaysToMonth366=array [13]
MinValue=System.DateTime
MaxValue=System.DateTime
dateData=9859461543285131878
[p#:0, t#:0] mdbg> funceval ConsoleApplication.Program.ReturnInput now
result = System.DateTime <6/24/01 2:59:04 AM>
DaysToMonth365=array [13]
DaysToMonth366=array [13]
MinValue=System.DateTime
MaxValue=System.DateTime
dateData=150443443958818
results saved to $result

@mikem8361 - the ENREGISTERED_PARAMTYPE_MAXSIZE is defined as 16 for UNIX_AMD64_ABI (you may have misread it as ARM64)

@mikem8361 Ah, sorry, that was the ENREGISTERED_RETURNTYPE_MAXSIZE. The ENREGISTERED_PARAMTYPE_MAXSIZE is set to 8 which is wrong, the return and param sizes the same.
Let me check when did it get changed to the wrong value.

@mikem8361 Interesting - the value was always wrong - never updated since the initial commit.
And I guess I can see why it was never discovered before - its value is never used for UNIX AMD64 compilation - we only check if the define is present at all places - with the only exception - the MarshalInfo::SetupArgumentSizes in mlinfo.cpp. And I guess we've never hit that code path for struct over 8 bytes in our tests.

The same issue is for ENREGISTERED_RETURNTYPE_INTEGER_MAXSIZE. That should also be 16 for Unix AMD64. That one influences two places. One in the arg iterator the case when the parameter is typedbyref when it would incorrectly set flag indicating that a return buffer is needed for structs over 8 bytes. And the other in IsUnmanagedValueTypeReturnedByRef which is used by marshalling.

Let me fix the constants, I'll send a PR in a minute.

Cool. I created https://github.com/dotnet/coreclr/issues/7115 to track this. I will resolve this as External.

@gregg-miskelly - I am not sure if fixing the constants will fix the problem in the debugger. So maybe it would be better to leave this open and close it after it is verified that it has fixed the issue. The point is that the structs of upto 16 bytes in registers are passed correctly at most places as parameters, we have a test for that in CoreCLR.

@janvorli Its possible that there will be other CoreCLR bugs too, but since Mike found this repos in mdbg, I am pretty sure that all the issues are on the CoreCLR side. So I would suggest we leave https://github.com/dotnet/coreclr/issues/7115 open until we have verified the fix in mdbg.

@jchannon for the weird switching between US date time format (mm/dd/yyyy) and UK, do you get the same behavior if you just evaluate the functions normally?
In other words, try adding this code:
string lastmodifiedStr = lastmodified.ToString();
string lastmodifiedDateStr = lastmodified.Date.ToString();

And evaluate those strings under the debugger.

Adding that code and looking in the Variables window the string representation is the UK format.

The two known bugs in CoreCLR that were effecting this scenario are now set to be fixed for .NET Core 1.1.

Is that Q1 2017?
On Thursday, 15 September 2016, Gregg Miskelly<EMAIL_ADDRESS>wrote:

The two known bugs in CoreCLR that were effecting this scenario are now
set to be fixed for .NET Core 1.1.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/OmniSharp/omnisharp-vscode/issues/753#issuecomment-247411681,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGaptid8QnGOHAVfYlw-RNZD5jaQsSKks5qqY7ogaJpZM4J4WOv
.

Fall 2016 according to https://github.com/dotnet/core/blob/master/roadmap.md#ship-dates

Aha. Thanks
On Thursday, 15 September 2016, Gregg Miskelly<EMAIL_ADDRESS>wrote:

Fall 2016 according to https://github.com/dotnet/
core/blob/master/roadmap.md#ship-dates
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/OmniSharp/omnisharp-vscode/issues/753#issuecomment-247424618,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGappvIlvjaVyLpUQuIxuXD5MLPSsdpks5qqZmkgaJpZM4J4WOv
.

Hi, just installed netcore 1.1 and this doesn't seem to be fixed
I ran the same repro as suggested by @gregg-miskelly here https://github.com/OmniSharp/omnisharp-vscode/issues/753#issuecomment-245715450

@jchannon unfortunately @mikem8361 said that https://github.com/dotnet/coreclr/issues/7115 didn't actually make 1.1. @mikem8361 can @jchannon install a preview3 CLI to check if this repos? Or will preview3 also be missing the fix?

The milestones on the issue/PR says 1.1 so you might want to change that,
not sure how you know what makes it into what release otherwise :)
On 17 November 2016 at 16:32, Gregg Miskelly<EMAIL_ADDRESS>wrote:

@jchannon https://github.com/jchannon unfortunately @mikem8361
https://github.com/mikem8361 said that dotnet/coreclr#7115
https://github.com/dotnet/coreclr/issues/7115 didn't actually make 1.1.
@mikem8361 https://github.com/mikem8361 can @jchannon
https://github.com/jchannon install a preview3 CLI to check if this
repos? Or will preview3 also be missing the fix?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/OmniSharp/omnisharp-vscode/issues/753#issuecomment-261296134,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGaplPHK-BwWcIO-Bpc5ZnztgGNOkQRks5q_IGMgaJpZM4J4WOv
.

Agreed :)

I’m not sure if the fix is in the dotnet cli review3 (I don’t know how to figure that out), but if you add "Microsoft.NETCore.Runtime.CoreCLR": "1.2.0-beta-24609-02" to your test app’s project.json it will have the fix.


Custom json updates and default settings

This PR combines #109 and #105
Not saying we merge this one instead of the others separately, but they are so related for testing it was easier to merge them into one.

Should we just retire #109 and #105 and review this one? Will that make review simpler?

@dwhswenson Should I fix this warnings
gufe/tests/test_custom_json.py::TestNumpyCoding::test_not_mine
  /home/mmh/miniconda3/envs/gufe-serial/lib/python3.10/site-packages/_pytest/fixtures.py:900: PytestRemovedIn8Warning: Support for nose tests is deprecated and will be removed in a future release.
  gufe/tests/test_custom_json.py::TestNumpyCoding::test_not_mine is using nose-specific method: `setup(self)`
  To remove this warning, rename it to `setup_method(self)`
  See docs: https://docs.pytest.org/en/stable/deprecations.html#support-for-tests-written-for-nose
    fixture_result = next(generator)

Or fix them in a different PR?

@mikemhenry : I'd vote for a separate PR (if one isn't already open?) I was noticing the same as I'm finishing one of the storage PRs here.

One last thing I need to do is load up Ivan's notebook and make sure things still work then this will get good to merge (and fix mypy and sphinx)

Ivan's notebooks works!
{
  "__class__": "Settings",
  "__module__": "gufe.settings.models",
  ":is_custom:": true,
  "settings_version": 0,
  "forcefield_settings": {
    "__class__": "OpenMMSystemGeneratorFFSettings",
    "__module__": "gufe.settings.models",
    ":is_custom:": true,
    "forcefield_kwargs": {
      "constraints": "app.HBonds",
      "rigidWater": true,
      "removeCMMotion": false,
      "hydrogenMass": "4*unit.amu"
    },
    "forcefields": [
      "amber/ff14SB.xml",
      "amber/tip3p_standard.xml",
      "amber/tip3p_HFE_multivalent.xml",
      "amber/phosaa10.xml"
    ],
    "small_molecule_forcefield": "openff-2.0.0"
  },
  "thermo_settings": {
    "__class__": "ThermoSettings",
    "__module__": "gufe.settings.models",
    ":is_custom:": true,
    "temperature": {
      "magnitude": 300,
      "unit": "kelvin",
      ":is_custom:": true,
      "pint_unit_registry": "openff_units"
    },
    "pressure": null,
    "ph": null,
    "redox_potential": null
  },
  "protocol_settings": {
    "__class__": "NonEqCyclingSettings",
    "__module__": "perses.protocols.settings",
    ":is_custom:": true,
    "lambda_functions": {
      "lambda_sterics_core": "lambda",
      "lambda_electrostatics_core": "lambda",
      "lambda_sterics_insert": "select(step(lambda - 0.5), 1.0, 2.0 * lambda)",
      "lambda_sterics_delete": "select(step(lambda - 0.5), 2.0 * (lambda - 0.5), 0.0)",
      "lambda_electrostatics_insert": "select(step(lambda - 0.5), 2.0 * (lambda - 0.5), 0.0)",
      "lambda_electrostatics_delete": "select(step(lambda - 0.5), 1.0, 2.0 * lambda)",
      "lambda_bonds": "lambda",
      "lambda_angles": "lambda",
      "lambda_torsions": "lambda"
    },
    "softcore_LJ_v2": true,
    "interpolate_old_and_new_14s": false,
    "timestep": {
      "magnitude": 4.0,
      "unit": "femtosecond",
      ":is_custom:": true,
      "pint_unit_registry": "openff_units"
    },
    "neq_splitting": "V R H O R V",
    "eq_steps": 1000,
    "neq_steps": 100,
    "platform": "CUDA",
    "save_frequency": 100
  }
}

The serialization code correctly points to the right module in perses!

Ready for review!

A few small changes to make.
Two bigger issues here, which probably go beyond scope of this PR, but are things we should stay on top of:
1. If our goal is to create engine-independent settings, that's not at all what these are. These are closely tied to OpenMM. That might be fine as a first step with a major rewrite for engine-independence as a later goal, but it doesn't look like we're planning engine independence from the start.

Really the only openmm focused bit is the OpenMMSystemGeneratorFFSettings class which really is only a wrapper for openmmforcefields system generator.
Once we have another protocol implementation, we can generalize it.

2. We will desperately need some mechanism to distinguish the different "levels" of settings. There is currently no mechanism to encode that data in these objects, and that will be necessary for analysis.

This is tracked with https://github.com/OpenFreeEnergy/gufe/issues/119