Q? HPI in MRI space

Hi,
@romquentin and I have a CTF dataset where the head position coils are already registered in MRI space. I think in this case we don't need to do the head shape coregistration.
Is there an easy to use these MRI coordinates directly to build the source space? i.e. to skip mne.gui.coregistration?
Thanks!
cc @christianbrodbeck

So you have the HPI coil coordinates in only MRI space, or you also have
their digitized locations in head coordinates (with likely slight
mismatches)? I assume you mean the former but want to check.
Do you have any other digitized points? In what coordinate frame?
If all you have is HPI in MRI, and only MEG data, I think you can get away
with using the identity for head to MRI trans, at least for the purposes of
computing the forward model. If you have EEG data with electrodes digitized
in head coordinates, it's a different story.

So you have the HPI coil coordinates in only MRI space, or you also have
their digitized locations in head coordinates (with likely slight
mismatches)? I assume you mean the former but want to check.

HPI coil coordinates in only MRI space only

Do you have any other digitized points? In what coordinate frame?

Well, no. They did digitize some headshape, but in a coordinate frame independent from MRI, these head coils, the MEG device... so I don't think this will be any useful.

If all you have is HPI in MRI, and only MEG data, I think you can get away
with using the identity for head to MRI trans, at least for the purposes of
computing the forward model.

Ok, so where shall I put the identity? In dev_head_t?

Ok, so where shall I put the identity? In dev_head_t?

That is the MEG device to head transform. Hopefully that one already exists
and is valid, as it is what the coil activations give you during
acquisition.
You want it in effectively head_mri_t, although we usually just call it
trans. You can make an identity one with Transform('head', 'mri'), and
pass it to functions like the forward code that require it.

right, thanks!

Can you track down a place to put this in the docs?

That'd be a perfect job for @romquentin :]

Hi guys,
Do I need to use read_dig_montage and epochs.set_montage before using the identity matrix as the trans in my forward model?
It seems that I cannot run read_dig_montage without hsp argument? (and I do not have any headshape but only the three HPI position already in the MRI space).
Thanks a lot for your help,
Romain

Do I need to use read_dig_montage and epochs.set_montage before using the identity matrix as the trans in my forward model?

You shouldn't need to for forward modeling, assuming you only have MEG data.

It seems that I cannot run read_dig_montage without hsp argument?

It seems like we should support read_dig_montage even without the hsp argument. This is probably a bug...

Ok.
But before using fwd = make_forward_solution(raw.info, fname_trans, src, bem_sol), I should add the information somewhere of my HPI coordinates (in head space which is MRI space in my case, that is the reason of using the identity matrix as the trans matrix). I thought read_dig_montage and epochs_set_montage would do that...
Thank you!
The bug when running
read_dig_montage(hsp=None, hpi=hpi, elp=elp,                  point_names=point_names, unit='mm',                  transform=False,                  dev_head_t=False)
is
TypeError                                 Traceback (most recent call last)
/Applications/anaconda/lib/python2.7/site-packages/IPython/core/formatters.pyc in call(self, obj)
697                 type_pprinters=self.type_printers,
698                 deferred_pprinters=self.deferred_printers)
--> 699             printer.pretty(obj)
700             printer.flush()
701             return stream.getvalue()
/Applications/anaconda/lib/python2.7/site-packages/IPython/lib/pretty.pyc in pretty(self, obj)
381                             if callable(meth):
382                                 return meth(obj, self, cycle)
--> 383             return _default_pprint(obj, self, cycle)
384         finally:
385             self.end_group()
/Applications/anaconda/lib/python2.7/site-packages/IPython/lib/pretty.pyc in _default_pprint(obj, p, cycle)
501     if _safe_getattr(klass, 'repr', None) not in baseclass_reprs:
502         # A user-provided repr. Find newlines and replace them with p.break()
--> 503         _repr_pprint(obj, p, cycle)
504         return
505     p.begin_group(1, '<')
/Applications/anaconda/lib/python2.7/site-packages/IPython/lib/pretty.pyc in repr_pprint(obj, p, cycle)
692     """A pprint that just redirects to the normal repr function."""
693     # Find newlines and replace them with p.break()
--> 694     output = repr(obj)
695     for idx,output_line in enumerate(output.splitlines()):
696         if idx:
/Applications/anaconda/lib/python2.7/site-packages/mne/mne-python/mne/channels/montage.pyc in repr(self)
392         """String representation."""
393         s = '<DigMontage | %d Dig Points, %d HPI points: %s ...>'
--> 394         s %= (len(self.hsp), len(self.point_names),
395               ', '.join(self.point_names[:3]))
396         return s
TypeError: object of type 'NoneType' has no len()

Are you in latest master? I thought I changed that line recently

Just updated and it worked.
So is read_dig_montage and epochs.set_montagethe good way to add information about HPI coordinates?
Thank you!

That should be okay. Can you check it with plot_trans?
Is there any use in having those coordinates? They shouldn't be used for
anything I don't think...

Thank you very much. It seems it's working well!
I am using these coordinates because there are the only ones I have... I coregistered before the MEG the head and the MRI with brainsight (which is use for TMS neuronavigation). Although unusual, this navigation system is accurate. The other system that people proposed to me at the MEG center is to take picture of the position of the HPI and I though it was more accurate to have their positions in the MRI coordinate system.
I do not need to create an identity matrix, this is the default for trans argument in make_forward_solution . Let me know if I can still help to put this in the docs?

yes it's better to leave a note in the narrative doc so the discussion from
here does not get lost

Ok. Not sure where to leave the note?  Can you send me a link?

somewhere in manual eg:
http://martinos.org/mne/dev/manual/index.html#source-localization
or in one of a tutorials

Finally, this plot_trans output was a fluke, it did not work correctly with others participants.
I have two questions:
When I read my epochs, I already have a matrix in epochs.info['dev_head_t']. Does it mean that I do not need to compute it in read_dig_montage? It is normal that the dev_head_t matrix is not the same originally in the epochs and after added it with dev_head_t=True in read_dig_montage?
Also, when I manually looked at my hpi coordinates in the MRI coordinate system on freeview, my HPIs appear at the good position on the subject MRI. But after read_dig_montage and plot_trans, HPIs positions (green dot) can be shift relatively to the MRI. From what I understand, it should not be the case if the -trans matrix is the identity one, no?
Thank you very much for your help!

When I read my epochs, I already have a matrix in epochs.info['dev_head_t']

Okay then your head frame is not arbitrary, it already exists as something meaningful. This makes sense actually I should have thought about it before. Using trans=None will not work :(

It is normal that the dev_head_t matrix is not the same originally in the epochs and after added it with dev_head_t=True in read_dig_montage?

No, this is weird. It means that the HPI coils that you are passing are not in the correct coordinate frame. One set is supposed to be in head coords and the other in device coords, which gives the dev_head_t.

when I manually looked at my hpi coordinates in the MRI coordinate system on freeview, my HPIs appear at the good position on the subject MRI. But after read_dig_montage and plot_trans, HPIs positions (green dot) can be shift relatively to the MRI

You should be able to check and see what the coordinates are in the DigMontage instance, are they different from what you see in FreeView?

also, I didn't see any discussion of which "MRI space" you mean. A decent MGZ (generated from dicoms) has at least 3 transformation matrices, all of these could be described as "MRI spaces". The ones I can remember off the top of my head are tk2ras ras2vox. I suspect at least one if not multiple of these will need to be applied to your coordinates.
hth
d

@Eric89GXL. Thanks. But I do not have any information in my epoch data about the HPI coils in head coords since the neuronavigation system that I use is not linked to my raw data at any time before trying to apply the montage. So it should be impossible to construct the dev_head_t before applying the montage no? Sorry, How can I check coordinates in DigMontage?
@dgwakeman. The MRI coordinates I am referring to is the RAS coords in freeview (not the TkRegRAS or the surface RAS).
Thanks a lot!!

Ok. When I am running this command:
dig_montage = read_dig_montage(hsp=None, hpi=hpi, elp=elp,                                point_names=point_names, unit='mm',                                transform=False,                                dev_head_t=True)
with
hpi=array([[ 0.00407583,  0.10639121, -0.061938  ],
[-0.06226506,  0.03360433, -0.089045  ],
[ 0.0627302 ,  0.02532453, -0.088612  ]], dtype=float32)
(corresponding to the values in epochs.info['hpi_results'][0]['dig_points'])
elp=array([[  9.912,  87.837,  14.563],
[-51.649,  20.382, -34.179],
[ 71.327,  18.049, -38.179]])
(corresponding to my HPI coils position in RAS coords from my neuronavigation system)
point_names = ['NEC', 'LEC', 'REC']
My dig_montage seems to have only hpi informations (but not elp).
dig_montage
<DigMontage | 0 extras (headshape), 3 HPIs, 0 fiducials, 0 channels>
Hope I am clear enough and thanks a lot again!

@romquentin , I think there is some more information in some of the CTF files that you aren't detecting. The CTF system must have the location of the HPI coordinates in either device or CTF head coordinate frame (otherwise, it wouldn't be able to measure the head position) in one of the files it reads.
Essentially all you need to do is calculate the transformation matrix between your HPI coils in RAS space and your HPI coils in either the CTF head or device coordinate frame. Then save that as a trans.fif

Yes, epochs.info['dig'] contains information before applying set_montage but I am not sure it is accurate (because at this time I did not give yet any information about HPI position in head or MRI space)
After applying set_montage, epochs.info['dig'] changed and contains now the coils position in RAS coords but with a 10^-3 factor.
Ok. will try! Thanks!

(because at this time I did not give yet any information about HPI position in head or MRI space)

Are you sure? Somehow the system registered the MEG device<->head transformation, and that usually involves the (digitized, in head coords) HPI coil positions.

Yes sure. The brainsight system (neuronavigation system) is not connected at all with the MEG acquisition system so there is no way that my epochs can have information about digitized HPI coil positions before set_montage.

I get that the brainsight system isn't connected, but there must be some other source of information that the MEG gets information from about the HPI coil position/head coordinate system. Either typed in, or another digitization system, or something.

One issue was that I was giving RAS coordinates and not surface RAS in my elp. I think mne function use surface RAS during forward modeling..?
I still do not know why I already have epochs.info['dig'] in my data before set_montage but I will check tomorrow with the MEG technician.
I will confirm tomorrow that this strategy works. Thanks a lot again for your help.

I confirm that it works (recording my head in different position inside the MEG and reconstruct correctly in plot_trans). This is empirical and I am happy to send you the code if anyone interested to theoretically check or to add a example (even if apparently it is very rare to use neuronavigation system for head positionning).
Apparently the epochs.info['dig'] is added during mne.io.read_raw_ctf with _read_pos. But this function is looking for a .pos file in the ctf ds folder that I do not have. So I still don't know what are the coordinates here.
Thank you very very much for your help!!

Apparently the epochs.info['dig'] is added during mne.io.read_raw_ctf
with _read_pos. But this function is looking for a .pos file in the ctf
ds folder that I do not have.

this the approach of some CTF sites such the MNI cc @ftadel

This is what we do at the MNI: we copy the .pos file into all the .ds folders, it feels natural to have the head points part of each dataset, just like ou would have them in a .fif file.
If you want some examples: use any of the CTF-based sample datasets from the Brainstorm website.

Ok. Thanks! Just not sure to understand where values in epochs.info['dig'] come from if there is no .pos file in the .ds folder (@Eric89GXL)?

Can you share a sample file that has this behavior? You can email me privately if necessary

https://www.dropbox.com/s/e2d5ff3f6eylmpn/MJAZGAQC_wmtask_20161209_01.ds.zip?dl=0
So if I do:
raw = read_raw_ctf(drop_box_ds_folder)
raw.info['dig'] contains
[{'coord_frame': 4,
  'ident': 3,
  'kind': 2,
  'r': array([  0.00000000e+00,   1.01319381e-01,  -4.62223187e-33])},
 {'coord_frame': 4,
  'ident': 1,
  'kind': 2,
  'r': array([ -7.40363507e-02,   2.54408175e-19,  -1.07852077e-32])},
 {'coord_frame': 4,
  'ident': 2,
  'kind': 2,
  'r': array([  7.73589972e-02,  -4.50893407e-19,   3.85185989e-32])}]

while the .ds folder has now .pos file.
Thanks a lot!

Sorry. I mean while the .ds folder does not have .pos file :)

Did you talk to your MEG tech/site director? There must be some record of the hpi coils in one of the two spaces above or it can't measure the head position.

Yes I saw him. What I understood is there is one recording of HPI position in "dewar" space, and they directly calculate the corresponding position in a head CTF space (with the lpa-rpa axis being the Y-axis and the origin at the center of rpa-lpa segment). read_raw_ctf takes these coordinates and apply a transformation to bring these coordinates from CTF head to Neuromag head. This information (new coordinates in Neuromag head + transf) is in raw.info['hpi_results'].
But still no idea of where raw.info['dig'] is coming from. Here at the MEG center, they are using photo to manually position HPIs on the subject MRI. There is no digitized points recording at any time.

How do they calculate the the lpa and rpa axis if they don't digitize any points? How do they get the head CTF space (not sure what they use)? They must have some digitized point in common with the HPI coils and the fiducials.
What does it mean: "there is one recording of HPI position in "dewar" space"? (how is the HPI position recorded. Do you mean that the put sine waves through and do dipole fits on them?)

Sorry maybe I am wrong about what "digitized points" mean. Correct me if I am wrong.
1 - Before and after each MEG block, we localize the head. During that, the HPI coil positions inside the MEG are recording relatives to the "dewar". It gives us HPI coil positions in device space.
2 - From these coordinates in device space, CTF automatically calculates a "CTF-head" coordinates frame where (it is just a convention I guess) the left-ear coil position and the right-ear coil position forms the y-axis and the nasion is on the x-axis of this second coordinate frame (the "CTF-head").
3 - Until now, there is no "digitized" point, meaning HPI coordinates recordings with a "polhemus-like" system and based on a reference fixed on the subject's head, no?
Again, sorry if I am unclear or if vocabulary is not accurate. I am learning the field :)

@romquentin oh you always place the HPI coils at the fiducials, right? Then it could use the localized coords to define both the head coordinate frame, and the head<->device coordinate transformation.

Do you mean that the put sine waves through and do dipole fits on them?
you always place the HPI coils at the fiducials, right?

Yes, this is how it works at most CTF sites. In the CTF recordings, the reference points are typically the coils, and the have to mark the coils in the MRI, which is tricky (the Donders has some fancy markers they see in the MRI, but I think most sites don't do that).
At the MNI we digitize both the coils and the anatomical landmarks, save them in the .pos file (copied to the .ds folder), and use them to convert between the CTF coordinates system (based on the coils) and the equivalent based on the anatomical points (and therefor we can mark anatomical points in the MRI, which is much easier)

Yes!
It is for that reason that we use the brainsight system to get HPI coil position in "MRI space".
But I still do not get where the info['dig'] is coming from without .pos file.
info['dig'] is different from info['hpi_results'].

Isn't just the position of the 3 coils coils in the dewar coordinate system (estimated from the coils themselves) ?

No :)

Ahhh... I think they are getting created from the following sequence.
_read_hc
_compose_meas_info
follow that tree and you should find the bits you are looking for

Ok. I will! Thanks a lot!

@romquentin please report back with your findings. This is the type of thing that is good to have noted somewhere (maybe even the docs. But at least maybe code comments).

yes. I will investigate where info['dig'] is coming from and find a place to report the necessity to use surface RAS and not RAS coordinate when someone already have HPI positions in "MRI coordinates". Likely on next Monday now.

_pick_isotrack_and_hpi_coils is creating the info['hpi_results'] and the info['dig'].
The function is taking HPI coordinates in device space and in ctf head space from .hc file. It applies the t_ctf_dev_dev transformation to the coordinates in device space and the result is in info['hpi_results'] and it applies the t_ctf_head_head to the coordinates in ctf head space and the results is in info['dig']. If can be confusing to have a info['dig'] without any digitized information but it will be erased if you set a montage with "true" digitized information.
Maybe you can add this the following sentence in plot_trans and make_forward_model in the trans description: "You can use the identity matrix if you have HPI digitized information already in MRI space. Make sure in this case to use surface RAS coordinates.
I am happy to do it but not sure how?
Thank you!

I'm not actually sure you can do it in this case. For your data, you have a head coordinate frame, and it is defined by these transforms you're talking about. It's probably not the same as your MRI coordinate frame.
To get the proper transform for your data, you'd need to fit the "digitized" (from the HPI fitting) points onto the ones that are in your MRI. You should be able to do this using your coreg tool of choice (coreg module or mne_analyze).

Well, it should be possible to calculate the MRI-MEG coordinate frame by taking the HPI locations in MRI space and the HPI locations in CTF-HEAD space and calculating the transform ( a cheap way would be to feed the data under the hood to the code that creates this when you select a good fit for the points).

Ok...but the result of plot_trans seems very accurate. I tested on different blocks moving my head inside the MEG and the reconstruction of my head position seems very coherent. Is it possible that despite of this good reconstruction, something is wrong for the make_forward_model?

It suggests that your MRI coordinate frame is very similar to the Neuromag
head coordinate frame. Unless you do something in your MRI acquisition or
processing pipeline (which is very possible) to cause this, it might be a
coincidence and it is probably better not to rely on it. You might do
better by establishing the transformation using GUI alignment (or as
pointed out using algebra based on the point locations in MRI space), since
these points are what the device to head transform is based on. You should
find it is close to identity for this subject, but might be more useful for
others.
You have the fiducial coordinates in MRI space right? We have a private
function somewhere to compute the head transformation from fiducial points,
the math is pretty simple.

Yes I have the fiducial coordinates in MRI space. You mean compute the head to MRI transform? What is this function? Thank you again!

mne.transforms.get_ras_to_neuromag_trans


Setup travis.

avsm@ already did it, I need to create an account and put my stuff in it.

gemacht.


fix(core): fix type ExtractStateValue to describe real state value

Consider the following schema:
type LightStateSchema = {
  states: {
    green: {},
    yellow: {},
    red: {
      states: {
        walk: {},
        wait: {},
        stop: {},
      },
    },
  },
}

The current implementation of the ExtractStateValue type generates the
following type:
{
  green?: {} | undefined
  yellow?: {} | undefined
  red?: "walk" | "wait" | "stop" | {
    walk?: {} | undefined
    wait?: {} | undefined
    stop?: {} | undefined
  } | undefined
}

This does not describe the StateValue of LightStateSchema as provided
in State.value.
The new implementation generates the following type:
"green" | "yellow" | "red" | {
  red?: "walk" | "wait" | "stop" | undefined
}

Hi! In latest versions ExtractStateValue was deleted. What can I use instead of ExtractStateValue? Please)

Could you open a discussion that would describe your use case? For context, this type was removed in https://github.com/statelyai/xstate/pull/2828 as we've assumed that it was a leftover from a previous implementation and that it wasn't used externally.

@Andarist sorry I can't answer right now, I will try figure out. I guess it was used to made deal with TS


Fix kmod for kernel version 6.3

This PR fixes the kernel module to compile with kernel 6.3.

Do not have a machine at hand to test this with, but looking at kernel sources this appears to be correct. Thank you for your contribution!


Issue: too cool for school

pls fix

I used to repo to teach about issues. Thanks for being a good sport!


job abstraction

PR based on PR #30
abstract per-keyspace jobs in executor and per-node jobs in scheduler (makes live easier ;) )

Please rebase on mesosphere:rewrite

Please rebase on current mesosphere:rewrite

Please ignore this PR (superseded by newer PR)


Added Sin, Cos and Tan

Note that the accuracy is limited by the accuracy of PI

Thanks! :) , I think this weekend I'll find time to improve constants handling ( see #14 ), then we'll need to revisit the trigonometric code to take advantage of the changes.


Fixed error when loading default settings on first run.

Removed the leading spaces on the default values in the settings
designer
which was leading to runtime crashes since XML cannot have anything in
front of the xml declaration.

I bet this fixes the annoying property filter set error that I see from time to time ... and that I have worked around by Snooping with Snoop v2.8.0 ... and then going back to my developer version of Snoop.

I know that you fixed this forever ago ... but I am curious if it is the same error that I've seen. Was this a property filter set error ... that happened when you first Snoop an app?
Regardless it is now merged.


Broken in Firefox 60

I visited the beta site in an up to date Firefox, version 60.9.0esr. (This browser version is less than a year old.) Pressing either the right arrow key or the next-slide button does nothing more than change the circle in the lower left from blue to white. I also get this in the browser console: TypeError: Argument 1 of IntersectionObserver.observe is not an object.

@timmc thx for the feedback. It aligns another feedback I had on Slack which I wasn't able to reproduce on Mac with Firefix at least.
At you facing that error on Linux or Windows maybe?
With "beta site" do you mean, which URL exactly where u trying?
Thx in advance for the further info

Ah, I guess it's not the root page on that domain, is it. Here's the actual URL: https://beta.deckdeckgo.io/daviddalbusco/introducing-deckdeckgo/
I'm seeing this on Debian Linux.

Thx for the further info. Sounds like some standard might not yet be implemented by Firefox on Linux but I'll have to double check first.
Really appreciated, thx for all the details

It's doubtful it has anything to do with Debian's modifications to Firefox. Their packaging is very close to stock. But note that this is Firefox 60 (Extended Support Release), not Firefox 69, so you might be unknowingly relying on a feature only implemented in the past year.

@timmc I tried again with Firefox v60.9.0esr on my Mac and I finally do see the problem too.
I also see other visual error (like components being not correctly formatted).
Then I suspected, like I said above, a platform/browser compatibility error and therefore I checked can I use about custom elements and noticed that Firefox before v63 didn't support them.
For such cases, Stencil components are shipped with a Polyfill and we even add the @webcomponents/custom-elements in the kit but seems that it isn't enough for Firefox v60.9.0esr.
I also double checked again that it does work correctly with Firefox v69.
Therefore I would say that there isn't much I could do unfortunately 😔
Let me know what you think about it.

Firefox ESR has now been bumped to version 68.2.0esr, so it works now. :man_shrugging:

@timmc really happy to hear that, that's cool!
Thx for the feedback 👍


pytest-tui cannot work with pytest -n auto

How about disable pytest-tui when -n auto or disable https://pypi.org/project/pytest-xdist/ when --tui?

Can you provide a little information on what problem you are seeing when you invoke Pytest with the -n option?

It maybe not a bug. Close before I found more evidence. Thanks!


Problem with validation when updating via PUT

When I try to update a record via a PUT request (I use the autogenerated endpoint), I get the following error:
Error: When the modifier option is true, all validation object keys must be operators. Did you forget $set?
It seems that this has to do with literal document modifiers - how can I solve this since I can't influence the update call restivus executes (or can I?)
More info here: https://github.com/aldeed/meteor-simple-schema/issues/175
It would also be great if there was a way to only specify those fields in the request that need change (so the client does not have to submit the whole object when updating only one or two fields).

Can I see your request please? I'll start setting up the automated tests for all the autogenerated endpoints. I just tested manually before, which I'm gradually moving away from.
I completely intend to allow a user to update a partial document using JSON PATCH, but I am trying to encourage best practices, and HTTP PUT should really only be used to update an entire resource. Of course, if you need, you can always override any of the autogenerated endpoints to provide that functionality. I'll eventually allow users to override any of the default collection endpoints (for keeping things DRY). That would be super simple, but there's just a list of things I need to get to first.

I set up some automated tests for this and everything seems to work fine on the autogenerated PUT. I was able to update the entire resource, and when I left out a field it was nullified, as expected. I'll definitely need to see your request to help figure this out. I'll include the test code below, which has some sample requests using Meteor's HTTP package.

    describe 'A collection route', ->
      context 'with the default autogenerated endpoints', ->
        # Add a new collection route with all the default options
        Restivus.addCollection new Mongo.Collection('testautogen')
        # The ID of a newly created resource to persist between tests
        testId = null

        it 'should support a POST on api/collection', (test) ->
          result = HTTP.post 'http://localhost:3000/api/v1/testAutogen',
            data:
              name: 'test name'
              description: 'test description'
          response = JSON.parse result.content
          responseData = response.data
          test.equal result.statusCode, 201
          test.equal response.status, 'success'
          test.equal responseData.name, 'test name'
          test.equal responseData.description, 'test description'

          # Persist the new resource id
          testId = responseData._id

        it 'should support a PUT on api/collection/:id', (test) ->
          result = HTTP.put "http://localhost:3000/api/v1/testAutogen/#{testId}",
            data:
              name: 'update name'
              description: 'update description'
          response = JSON.parse result.content
          responseData = response.data
          test.equal result.statusCode, 200
          test.equal response.status, 'success'
          test.equal responseData.name, 'update name'
          test.equal responseData.description, 'update description'

          result = HTTP.put "http://localhost:3000/api/v1/testAutogen/#{testId}",
            data:
              name: 'update name with no description'
          response = JSON.parse result.content
          responseData = response.data
          test.equal result.statusCode, 200
          test.equal response.status, 'success'
          test.equal responseData.name, 'update name with no description'
          test.isUndefined responseData.description

All of those tests pass (except the POST returns a 200, when it should be returning a 201, which I will fix in a future update).

Thanks for your reply - I added the request here https://gist.github.com/sschuchlenz/89d608263af51225e253
When I submit this as x-www-form-urlencoded it results in the mentioned error, if I use form-data it results in a successful request but the data does not change at all. I've successfully implemented roleRequired and authentication (and checked that the permissions are correct), just in case.

Geez. Curl is kinda evil. It's awesome, but you have to be very careful with it. This same thing happened to another user, and it looks like an issue with the API, but it's really just the fact that the & symbol in bash (and maybe other shells) means to run the previous command in the background, so the rest of your command is being cut off after the first & character that's encountered. That's a very common character to use when testing over HTTP, so it's definitely something to be aware of. As soon as I saw you were using curl I just tested in a chrome extension rest client and it worked fine.
Check out this response in Issue #16 for a little more detail and the stack overflow response where I found the answer. I know this is the issue so I'm going to go ahead and close this. Feel free to continue the discussion here if you have any other questions.

In fact my gist is just the curl-representation of my API tests with postman (https://chrome.google.com/webstore/detail/postman-rest-client/fdmmgilgnpjigdojojpjoooidkmcomcm), I only use curl directly on a case by case basis and only for simple things like getting user id and token from the API. But I'll check if there is a postman setting to encapsulate URLs in quotes - thanks for the heads up.

Did you mean to use local.com in your url?  It seems to be some random website, and I just realized that I was using the url http://localhost:3000/api/entries/:id for my tests.
I say this because it looks like you were actually escaping the & with quotes. I just changed that and tested with curl and it works. Sorry for my false fix.

Sorry I forgot to mention this - local.com is a local mapping to localhost (so that is identical to http://localhost:3000), we use this for some reasons unrelated to meteor (since some components of ours need a "real" domain and local.com is just a workaround). I'll have a look, maybe this is a weird issue with postman and resolution of our local dnsmasq setting for local.com...

Update: Switched local.com with localhost but to no avail. I also tried to remove all the validations and default values from my simple-schema (since I suspected the problem is routed there) - but no luck.
If everything else yields no success I'll try to overwrite the put endpoint and add a custom update function there, but I'd really like to get to the bottom of this and find out why my app fails.

I didn't realize you were doing the schema validation. Simple schema validation is currently on the roadmap for Restivus, and is a very important feature for me. Restivus may actually be interfering somehow. I need to look into this further. I'm going to go ahead and reopen this issue, since I misunderstood the problem. Could you please post everything you're doing with the schema validation? The more code, the merrier.

My apologies that I didn't mention this - somehow I wasn't thinking straight ;)
I completely removed the simple schema (for testing) and it works, so the problem is indeed rooted in simple schema.
Since the app I'm working on is a client project (which I can't share per se due to contract obligations) I'll create a sample app with similar code and concepts, upload this to github and share the link once I'm done, I hope this aids in sorting out the issues with simple schema.
Many thanks for your time and effort!

I added a small sample application with restivus and a simple schema enabled collection (which validates some basics) here: https://github.com/sschuchlenz/restivus-simpleschema

After testing this out some more (thanks for the test repo!), and reading through all comments in the issue you referenced in your initial post, I'm almost sure that we'll need to modify simple-schema to get this to work, since it currently doesn't support updating an entire document, which is the behavior we need PUT to map to. @aldeed said that he would accept a pull request to resolve the issue, so I will try to start digging into the source this weekend to see if I can start putting together a pull request. Let me know if you look into it at all. I'll keep you posted on any progress I make.

I'm very graceful for your effort and if you manage to put in the effort to do a pull request I'll update the demo repo to aid in testing and help with sorting out any issues the best I can. Please let me know if there is anything else I can do to assist.

Other than removing the error that SS throws, you will just have to make sure that it validates as an insert. It may kind of "just work" due to the fact that SS doesn't care about the concepts of insert vs. update, it only thinks in terms of modifier vs. document. So in the case of a whole-document update, it will think of it as an insert, which should be correct.
It may be trickier to add collection2 support because that package does have a concept of insert and update, and this is something in between. (People with custom validators may be assuming that this.isUpdate === true means it's a modifier, so it's sort of a backwards incompatible change, or at least a change we'll need to notify people about.) But on the other hand, it might "just work" here, too. If you add tests in both SS and C2 packages, and make any changes so that all the tests pass, it should be fine.
Also, I don't think full-doc updates are allowed on the client so if that's still true, this will be a server-side change only.

Thanks for the guidance, Eric! Much appreciated!

Hi @kahmali,
First allow me to say "Restivus is an AMAZING Meteor package". Thank you so much writing it!!!
I am experiencing the same situation as @sschuchlenz and just wondering, just a vague thought..., do you think line 164 instead of entityIsUpdated = collection.update @urlParams.id, @bodyParams could be entityIsUpdated = collection.update @urlParams.id, {$set: @bodyParams}.
Would that solves the validation error?
Thanks!

Thank you so much for the kind words @zgudino! My life has been consumed by a startup I'm working on, and I don't have any time to test this out right now. Would it be too much trouble for you to fork Restivus and try testing the update in your development environment? If the update works, just submit a PR and I'll review and merge it in. I hate that I can't be more helpful right now. Sorry about that! As soon as I have some downtime I'm going to give Restivus some TLC.
Just an aside, but I'm now using Astronomy for my model layer, in place of simple schema and its related packages. I (and Eric Dobbertin, the author of simple schema himself) would recommend you check it out. It certainly has the potential to become the de facto model layer for Meteor, and it recently hit 1.0 and is in active development. Unfortunately for us, we're still waiting on the relations module before upgrading to 1.0, but hopefully that makes its way in there soon.
Thanks again for the suggestion! Let me know what you find if you look into it further.

Thanks for your quick response @kahmali. Also, thank you for recommending Astronomy!

What is the status with this?
I am getting the same error with generated PUT method.

Has this not been fixed?  I'm having issues with PUT as well:
Error: When the modifier option is true, all validation object keys must be operators. Did you forget $set?     at packages/aldeed_simple-schema.js:2466:15     at Function._.each._.forEach (packages/underscore.js:147:22)     at checkModifier (packages/aldeed_simple-schema.js:2463:7)     at doValidation1 (packages/aldeed_simple-schema.js:2495:5)     at doValidation (packages/aldeed_simple-schema.js:2910:10)     at SimpleSchemaValidationContext.simpleSchemaValidationContextValidate [as validate] (packages/aldeed_simple-schema.js:2945:21)     at [object Object].doValidate (packages/aldeed_collection2-core.js:413:33)     at [object Object].Mongo.Collection.(anonymous function) [as update] (packages/aldeed_collection2-core.js:202:25)     at Object.put.action (packages/nimble_restivus/lib/restivus.coffee:162:40)     at Route.share.Route.Route._callEndpoint (packages/nimble_restivus/lib/route.coffee:147:25)

Someone of us should probably try what @zgudino is proposing, using $set :) And send PR if it works.

The issue with what @zgudino is suggesting is that it would change the behavior of PUT from updating an entire document to only updating the specified fields, which is not the intended behavior for a PUT. What's really needed for that is PATCH support.
Until PATCH support comes around (at the rate I'm going it'll be sometime in the year 2050) you can use my suggestion here to override the default PUT, or make the changes to simple-schema and collection2 suggested by @aldeed above.
I promise to make JSON PATCH support the first major update I make when I have a chance to continue developing Restivus. I'm gonna organize a ZenHub board to begin a roadmap for development. Hopefully that'll jumpstart things.
I also need to update the docs to include a note about this issue with validating a PUT using simple schema (just leaving this as a note for myself).


autosuggestion doesn't add the selected item to input box on windows Nokia Lumia 920 device with windows 8.1

When the suggestion comes up and try to select on a windows phone the auto suggested text populated doesn't get selected into the input search box.
Using angular.js, bootstrap

Sorry, I have no way of testing this.


[Issue]: hip-5.7.1 failed tests: invalid device pointer Code 17 with  hipIpcOpenMemHandle( , , hipIpcMemLazyEnablePeerAccess)

Problem Description
Running Gentoo hip tests on dual MI100 system result in two tests failure:
The following tests FAILED:
        1699 - Unit_hipIpcMemAccess_Semaphores (Timeout)
        1715 - Unit_hipIpcEventHandle_Functional (Timeout)

1699/1801 Testing: Unit_hipIpcMemAccess_Semaphores
1699/1801 Test: Unit_hipIpcMemAccess_Semaphores
Command: "/tmp/portage/dev-util/hip-5.7.1-r1/work/hip-tests-rocm-5.7.0/catch_build/catch_tests/multiproc/MultiProc" "Unit_hipIpcMemAccess_Semaphores"
Directory: /tmp/portage/dev-util/hip-5.7.1-r1/work/hip-tests-rocm-5.7.0/catch_build/catch_tests/multiproc
"Unit_hipIpcMemAccess_Semaphores" start time: Jan 18 01:15 CST
Output:
----------------------------------------------------------
Filters: Unit_hipIpcMemAccess_Semaphores

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
MultiProc is a Catch v2.13.4 host application.
Run with -? for options

-------------------------------------------------------------------------------
Unit_hipIpcMemAccess_Semaphores
-------------------------------------------------------------------------------
/tmp/portage/dev-util/hip-5.7.1-r1/work/hip-tests-rocm-5.7.0/catch/multiproc/hipIpcMemAccessTest.cc:80
...............................................................................

/tmp/portage/dev-util/hip-5.7.1-r1/work/hip-tests-rocm-5.7.0/catch/multiproc/hipIpcMemAccessTest.cc:152: FAILED:
  REQUIRE( false )
with message:
  Error: invalid device pointer
      Code: 17
      Str: hipIpcOpenMemHandle(reinterpret_cast<void **>(&B_d), shrd_mem->
  memHandle, hipIpcMemLazyEnablePeerAccess)
      In File: /tmp/portage/dev-util/hip-5.7.1-r1/work/hip-tests-rocm-5.7.0/
  catch/multiproc/hipIpcMemAccessTest.cc
      At line: 152

===============================================================================
test cases: 1 | 1 failed
assertions: 7 | 6 passed | 1 failed

<end of output>

Operating System
Gentoo Prefix on upstream Linux kernel 6.1.69-1.1
CPU
AMD EPYC 7702 64-Core Processor
GPU
AMD Instinct MI100
ROCm Version
ROCm 5.7.1
ROCm Component
HIP
Steps to Reproduce
In a fresh Gentoo Linux, with
/etc/portage/package.accept_keywords
*/* ~amd64

/etc/portage/env/test.conf
FEATURES="${FEATURES} test"

/etc/portage/package.env/0-test
dev-util/hip test.conf

And run emerge --verbose hip
(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support
=====================
HSA System Attributes
=====================
Runtime Version:         1.1
System Timestamp Freq.:  1000.000000MHz
Sig. Max Wait Duration:<PHONE_NUMBER>3709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model:           LARGE
System Endianness:       LITTLE
Mwaitx:                  DISABLED
DMAbuf Support:          NO

==========
HSA Agents
==========
*******
Agent 1
*******
  Name:                    AMD EPYC 7702 64-Core Processor
  Uuid:                    CPU-XX
  Marketing Name:          AMD EPYC 7702 64-Core Processor
  Vendor Name:             CPU
  Feature:                 None specified
  Profile:                 FULL_PROFILE
  Float Round Mode:        NEAR
  Max Queue Number:        0(0x0)
  Queue Min Size:          0(0x0)
  Queue Max Size:          0(0x0)
  Queue Type:              MULTI
  Node:                    0
  Device Type:             CPU
  Cache Info:
    L1:                      32768(0x8000) KB
  Chip ID:                 0(0x0)
  ASIC Revision:           0(0x0)
  Cacheline Size:          64(0x40)
  Max Clock Freq. (MHz):   2000
  BDFID:                   0
  Internal Node ID:        0
  Compute Unit:            128
  SIMDs per CU:            0
  Shader Engines:          0
  Shader Arrs. per Eng.:   0
  WatchPts on Addr. Ranges:1
  Features:                None
  Pool Info:
    Pool 1
      Segment:                 GLOBAL; FLAGS: FINE GRAINED
      Size:                    528222572(0x1f7c096c) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
    Pool 2
      Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
      Size:                    528222572(0x1f7c096c) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
    Pool 3
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED
      Size:                    528222572(0x1f7c096c) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
  ISA Info:
*******
Agent 2
*******
  Name:                    AMD EPYC 7702 64-Core Processor
  Uuid:                    CPU-XX
  Marketing Name:          AMD EPYC 7702 64-Core Processor
  Vendor Name:             CPU
  Feature:                 None specified
  Profile:                 FULL_PROFILE
  Float Round Mode:        NEAR
  Max Queue Number:        0(0x0)
  Queue Min Size:          0(0x0)
  Queue Max Size:          0(0x0)
  Queue Type:              MULTI
  Node:                    1
  Device Type:             CPU
  Cache Info:
    L1:                      32768(0x8000) KB
  Chip ID:                 0(0x0)
  ASIC Revision:           0(0x0)
  Cacheline Size:          64(0x40)
  Max Clock Freq. (MHz):   2000
  BDFID:                   0
  Internal Node ID:        1
  Compute Unit:            128
  SIMDs per CU:            0
  Shader Engines:          0
  Shader Arrs. per Eng.:   0
  WatchPts on Addr. Ranges:1
  Features:                None
  Pool Info:
    Pool 1
      Segment:                 GLOBAL; FLAGS: FINE GRAINED
      Size:                    528395328(0x1f7eac40) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
    Pool 2
      Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
      Size:                    528395328(0x1f7eac40) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
    Pool 3
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED
      Size:                    528395328(0x1f7eac40) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
  ISA Info:
*******
Agent 3
*******
  Name:                    gfx908
  Uuid:                    GPU-736edc556309c935
  Marketing Name:          AMD Instinct MI100
  Vendor Name:             AMD
  Feature:                 KERNEL_DISPATCH
  Profile:                 BASE_PROFILE
  Float Round Mode:        NEAR
  Max Queue Number:        128(0x80)
  Queue Min Size:          64(0x40)
  Queue Max Size:          131072(0x20000)
  Queue Type:              MULTI
  Node:                    2
  Device Type:             GPU
  Cache Info:
    L1:                      16(0x10) KB
    L2:                      8192(0x2000) KB
  Chip ID:                 29580(0x738c)
  ASIC Revision:           2(0x2)
  Cacheline Size:          64(0x40)
  Max Clock Freq. (MHz):   1502
  BDFID:                   768
  Internal Node ID:        2
  Compute Unit:            120
  SIMDs per CU:            4
  Shader Engines:          8
  Shader Arrs. per Eng.:   1
  WatchPts on Addr. Ranges:4
  Features:                KERNEL_DISPATCH
  Fast F16 Operation:      TRUE
  Wavefront Size:          64(0x40)
  Workgroup Max Size:      1024(0x400)
  Workgroup Max Size per Dimension:
    x                        1024(0x400)
    y                        1024(0x400)
    z                        1024(0x400)
  Max Waves Per CU:        40(0x28)
  Max Work-item Per CU:    2560(0xa00)
  Grid Max Size:<PHONE_NUMBER>(0xffffffff)
  Grid Max Size per Dimension:
    x<PHONE_NUMBER>(0xffffffff)
    y<PHONE_NUMBER>(0xffffffff)
    z<PHONE_NUMBER>(0xffffffff)
  Max fbarriers/Workgrp:   32
  Packet Processor uCode:: 64
  SDMA engine uCode::      18
  IOMMU Support::          None
  Pool Info:
    Pool 1
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED
      Size:                    33538048(0x1ffc000) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Alignment:         4KB
      Accessible by all:       FALSE
    Pool 2
      Segment:                 GLOBAL; FLAGS:
      Size:                    33538048(0x1ffc000) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Alignment:         4KB
      Accessible by all:       FALSE
    Pool 3
      Segment:                 GROUP
      Size:                    64(0x40) KB
      Allocatable:             FALSE
      Alloc Granule:           0KB
      Alloc Alignment:         0KB
      Accessible by all:       FALSE
  ISA Info:
    ISA 1
      Name:                    amdgcn-amd-amdhsa--gfx908:sramecc+:xnack-
      Machine Models:          HSA_MACHINE_MODEL_LARGE
      Profiles:                HSA_PROFILE_BASE
      Default Rounding Mode:   NEAR
      Default Rounding Mode:   NEAR
      Fast f16:                TRUE
      Workgroup Max Size:      1024(0x400)
      Workgroup Max Size per Dimension:
        x                        1024(0x400)
        y                        1024(0x400)
        z                        1024(0x400)
      Grid Max Size:<PHONE_NUMBER>(0xffffffff)
      Grid Max Size per Dimension:
        x<PHONE_NUMBER>(0xffffffff)
        y<PHONE_NUMBER>(0xffffffff)
        z<PHONE_NUMBER>(0xffffffff)
      FBarrier Max Size:       32
*******
Agent 4
*******
  Name:                    gfx908
  Uuid:                    GPU-4c8c97e3b14bcfdc
  Marketing Name:          AMD Instinct MI100
  Vendor Name:             AMD
  Feature:                 KERNEL_DISPATCH
  Profile:                 BASE_PROFILE
  Float Round Mode:        NEAR
  Max Queue Number:        128(0x80)
  Queue Min Size:          64(0x40)
  Queue Max Size:          131072(0x20000)
  Queue Type:              MULTI
  Node:                    3
  Device Type:             GPU
  Cache Info:
    L1:                      16(0x10) KB
    L2:                      8192(0x2000) KB
  Chip ID:                 29580(0x738c)
  ASIC Revision:           2(0x2)
  Cacheline Size:          64(0x40)
  Max Clock Freq. (MHz):   1502
  BDFID:                   8960
  Internal Node ID:        3
  Compute Unit:            120
  SIMDs per CU:            4
  Shader Engines:          8
  Shader Arrs. per Eng.:   1
  WatchPts on Addr. Ranges:4
  Features:                KERNEL_DISPATCH
  Fast F16 Operation:      TRUE
  Wavefront Size:          64(0x40)
  Workgroup Max Size:      1024(0x400)
  Workgroup Max Size per Dimension:
    x                        1024(0x400)
    y                        1024(0x400)
    z                        1024(0x400)
  Max Waves Per CU:        40(0x28)
  Max Work-item Per CU:    2560(0xa00)
  Grid Max Size:<PHONE_NUMBER>(0xffffffff)
  Grid Max Size per Dimension:
    x<PHONE_NUMBER>(0xffffffff)
    y<PHONE_NUMBER>(0xffffffff)
    z<PHONE_NUMBER>(0xffffffff)
  Max fbarriers/Workgrp:   32
  Packet Processor uCode:: 64
  SDMA engine uCode::      18
  IOMMU Support::          None
  Pool Info:
    Pool 1
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED
      Size:                    33538048(0x1ffc000) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Alignment:         4KB
      Accessible by all:       FALSE
    Pool 2
      Segment:                 GLOBAL; FLAGS:
      Size:                    33538048(0x1ffc000) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Alignment:         4KB
      Accessible by all:       FALSE
    Pool 3
      Segment:                 GROUP
      Size:                    64(0x40) KB
      Allocatable:             FALSE
      Alloc Granule:           0KB
      Alloc Alignment:         0KB
      Accessible by all:       FALSE
  ISA Info:
    ISA 1
      Name:                    amdgcn-amd-amdhsa--gfx908:sramecc+:xnack-
      Machine Models:          HSA_MACHINE_MODEL_LARGE
      Profiles:                HSA_PROFILE_BASE
      Default Rounding Mode:   NEAR
      Default Rounding Mode:   NEAR
      Fast f16:                TRUE
      Workgroup Max Size:      1024(0x400)
      Workgroup Max Size per Dimension:
        x                        1024(0x400)
        y                        1024(0x400)
        z                        1024(0x400)
      Grid Max Size:<PHONE_NUMBER>(0xffffffff)
      Grid Max Size per Dimension:
        x<PHONE_NUMBER>(0xffffffff)
        y<PHONE_NUMBER>(0xffffffff)
        z<PHONE_NUMBER>(0xffffffff)
      FBarrier Max Size:       32
*** Done ***

Additional Information
The full build & test log
build.log.gz
Test details:
LastTest.log.gz

Hi @littlewu2508 can you please rerun the failing tests with logging enabled e.g export AMD_LOG_LEVEL=4 and paste the output.

export AMD_LOG_LEVEL=4

Unit_hipIpcMemAccess_Semaphores.log

I see this error in the logs from hipIpcGetMemHandle:
Failed to create memory for IPC, failed with hsa_status: 4097
Can you please confirm that the system has an up-to date version of the kernel mode driver (amdgpu-dkms) that should at least match rocm/5.7.1 release.

I see this error in the logs from hipIpcGetMemHandle: Failed to create memory for IPC, failed with hsa_status: 4097
Can you please confirm that the system has an up-to date version of the kernel mode driver (amdgpu-dkms) that should at least match rocm/5.7.1 release.

The system uses the amdgpu driver from upstream Linux kernel, version 6.1.99. No amdgpu-dkms is installed.

Hello, was this issue ever resolved? I am seeing a similar issue on ROCM 6.0.0 with MI250X GPUs.
Driver version is 6.3.6

@littlewu2508 and @natshineman Did you try with Ubuntu or other supported OS https://rocm.docs.amd.com/en/docs-6.2.1/compatibility/compatibility-matrix.html for the hip-test? amdgpu-dkms driver is a better support for these OS.

Hi @huanrwan-amd ,
I am also experiencing the same issue as described here, with very similar logs with ROCM 6.1.0 (OS: Ubuntu 22.04, 2x MI100 GPUs) while trying to share memory between two processes, each process using a separate GPU.
With ROCM 6.2.2, it actually fails at opening the memory with hipIPCOpenMemHandle.
What I have noticed though is that exporting pointers with hipMemPoolExportPointer seems to be working (I still need to confirm by seeing if my kernels using the shared points output the right thing, but hipMemcpyAsync seems to work transferring memory from one GPU to the other).
I used the code from the docs.
Looking at the implementation of hipIPCGetMemHandle and hipMemPoolExportPointer, they seem to be doing more or less the same thing, but a bit differently. I guess hipMemPoolExportPointer is the more correct one.
Fixing hipIPCGetMemHandle / hipIPCOpenMemHandle would be very much appreciated, as that's what people tend to use.
Best,
Epliz

@Epliz Thank you for your comments. Can you provide the code snippet that failing on hipIPCGetMemHandle / hipIPCOpenMemHandle. I can successfully call these APIs on MI300 with my testing code.

reproducer for issue with SOMA export/import of pointer:
#include <hip/hip_runtime.h>

#include <stdint.h>
#include <stdio.h>

#include <string.h>
#include <sys/socket.h>
#include <sys/un.h>
#include <unistd.h>

#define COMM_SOCKET_PATH "/tmp/comm_sock"

int main(int argc, char** argv) {

  int gpu_count;
  if (hipGetDeviceCount(&gpu_count) != hipSuccess) {
    return -1;
  }
  if (gpu_count < 2) {
    printf("this program needs two GPUs\n");
    return -1;
  }

  printf("Please run me twice\n!");

  int server_fd = -1; // socket to accept new connections, only one rank will have it
  int comm_socket = -1; // socket to do communications

  bool is_server;

  // try to become the server
  struct sockaddr_un server_addr;

  // Create a domain socket to sync processes
  int fd = socket(AF_UNIX, SOCK_STREAM, 0);
  if (fd == -1) {
    // error
    return - 1;
  }

  // Try bindnig socket to address
  memset(&server_addr, 0, sizeof(struct sockaddr_un));
  server_addr.sun_family = AF_UNIX;
  strncpy(server_addr.sun_path, COMM_SOCKET_PATH, sizeof(server_addr.sun_path) - 1);

  // we are the server if we managed to bind
  is_server = bind(fd, (struct sockaddr *)&server_addr, sizeof(struct sockaddr_un)) != -1;

  if (is_server) {
    printf("I am the server\n");
    // if we managed to bind, we should connect to all other ranks and they will be the socket
    // clients
    server_fd = fd;

    int num_clients = 1; // just expecting one client

    // Listen for connections
    if (listen(server_fd, num_clients) == -1) {
      return -1;
    }

    for (int c = 0; c < num_clients; c++) {
      int comm_socket = accept(server_fd, NULL, NULL);
      if (comm_socket == -1) {
        // error
        return -1;
      }
    }
  } else {
    printf("I am a client\n");
    // if we didn't manage to bind, we are a mere client
    // we just need to connect to the server
    comm_socket = fd;

    if (connect(comm_socket, (struct sockaddr *)&server_addr, sizeof(struct sockaddr_un)) == -1) {
      // error
      return -1;
    }

    printf("connected to the server\n");
  }

  // we can already unlink as everyone has connected
  unlink(COMM_SOCKET_PATH);

  // the server gets gpu 0, the other one gpu 1
  int gpu = is_server ? 0 : 1;

  if (hipSetDevice(gpu) != hipSuccess) {
    return -1;
  }

  for (int d = 0; d < gpu_count; d++) {
    if (d == gpu) continue;
    if (hipDeviceEnablePeerAccess(d, 0) != hipSuccess) {
      // TODO: return error
      printf("Failed to enable peer to peer with %d\n", d);
      return -1;
    }
  }
  printf("Enabled peer to peer\n");

  // allocate some memory to share
  int* device_data;
  int* remote_device_data;

  // try to allocate 2MiB of data on the GPU (in case it needs to be big enough)
  if (hipMalloc(&device_data, sizeof(int) * 1024 * 1024 * 2) != hipSuccess) {
    return -1;
  }

  // now try to share the memory
  hipMemPoolPtrExportData memHandle;
  if (hipMemPoolExportPointer(&memHandle, device_data) != hipSuccess) {
    // failed
    printf("Failed to export pointer\n");
    return -1;
  }

  // send/receive the handles
  hipMemPoolPtrExportData remoteMemHandle;
  if (is_server) {
    // send then receive
    if (write(comm_socket, &memHandle, sizeof(memHandle))< sizeof(memHandle)) {
      printf("interrupted write\n");
      return -1;
    }
    if (read(comm_socket, &remoteMemHandle, sizeof(memHandle)) < sizeof(memHandle)) {
      printf("interrupted read\n");
      return -1;
    }
  } else {
    // receive then send
    if (read(comm_socket, &remoteMemHandle, sizeof(memHandle)) < sizeof(memHandle)) {
      printf("interrupted read\n");
      return -1;
    }
    if (write(comm_socket, &memHandle, sizeof(memHandle))< sizeof(memHandle)) {
      printf("interrupted write\n");
      return -1;
    }
  }

  hipMemPoolProps poolProps = {};
  poolProps.allocType = hipMemAllocationTypePinned;
  poolProps.handleTypes = hipMemHandleTypePosixFileDescriptor;
  poolProps.location.type = hipMemLocationTypeDevice;
  poolProps.location.id = gpu == 0 ? 1 : 0; // Assuming other device

  hipMemPool_t memPool;
  if (hipMemPoolCreate(&memPool, &poolProps) != hipSuccess) {
    return -1;
  }

  if (hipMemPoolImportPointer((void**)&remote_device_data, memPool, &remoteMemHandle) != hipSuccess) {
    // failed
    printf("Failed to import pointer\n");
    return -1;
  }

  printf("Worked!\n");
  return 0;
}

same thing, please run the program twice

It seems like the SOMA approach works in my more complex program (which I can't share), but not the reproducer

Hi @Epliz, thank you for posting you testing code. In your code:
int comm_socket = accept(server_fd, NULL, NULL);
comm_socket is redeclared inside the loop, it will lose its value when running out of scope. Therefore, the write() and read() functions can not get the correct comm_socket value. You can just remove the declaration as:
comm_socket = accept(server_fd, NULL, NULL);

Hi @huanrwan-amd ,
Good catch! Actually it seems that after that fix both hip IPC and SOMA pointer export seem to work.
I will update the code snippets above just in case they are useful to someone.
Best,
Epliz

Hi @Epliz,
Thanks for the confirmation. Great to see it works on your side with MI100.
Some other suggestions, you may already know:

Add error messages for each failed functions calls.
Remember to release the resources
 hipFree(device_data);    close(comm_socket);    close(server_fd); in the failure cases and before program finished.

@littlewu2508 and @natshineman , closed the issue for no updates for 2 weeks. If you would like us to investigate further, please leave comments.


fix(codecov): continue on error

This should help in the future to protect against situations like https://github.com/OCA/oca-addons-repo-template/issues/250, when Codecov suddenly changes something, starts to fail and becomes blocking for everybody here.

I think that would not have helped in the current situation because the codecov step does not fail, it just blocks.

OK, but it would help in case codecov server goes down, for example. That'd be mostly as dramatic as #250.


Reddit parsing

WIP: Please do not merge this PR until this warning has been removed by the author.

Todo

[ ] implement class for general parsing

Attached GitHub issue

when you click the temporary (for testing purposes) hide button on each post, it also clicks the Reddit post.

Local Build

[ ] Ran all pre-commit hooks pre-commit run --all-files
[ ] Extension has been tested to build correctly cd extension && yarn && yarn build
[ ] Backend has been tested to build correctly cd backend && docker-compose build
[ ] Extension test suite has been run locally cd extension && yarn && yarn test
[ ] Backend test suite has been run locally cd backend && python3 -m unittest

GitHub

[ ] Target branch has been set correctly
[ ] PR has been rebased onto target branch
[ ] PR has been assigned an owner
[ ] Author has performed a self-review of the code
[ ] Removed the WIP message from the top of this PR

As previously suggested, I'd recommend performing the pre-commit stuff on your local repo https://pre-commit.com/

Codecov Report

Merging #26 (7ea96a4) into main (75d44cd) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main      #26   +/-   ##
=======================================
  Coverage   74.42%   74.42%
=======================================
  Files           8        8
  Lines         262      262
  Branches       42       42
=======================================
  Hits          195      195
  Misses         43       43
  Partials       24       24

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Add a setting in settings.py to define a custom edit icon

Currently, the icon for the edit function is hard-coded. This limits the ability for users to personalize their interface based on their specific needs.
Add a new setting in settings.py, say CRIPSY_FORMSET_MODAL.  This will enable users to define specific parameters of the package via a dictionary configuration.
Example
# In settings.py
CRISPY_FORMSET_MODAL = {
    "edit_icon": {
        'bootstrap4': "...",
        'tailwind': "...",
        'bulma': "...",
        'bootstrap5': "..."
    }
}

Closed because #12 implements a better approach


refactor: use root ctx in agent

Description
Updates the agent to use ctx from the root
Checklist before merging

[ ] Test, docs, adr added or updated as needed
[ ] Contributor Guide Steps followed

It's smol, I'm happy with it if e2e is as well


Cleanup spacing and other styles of demo

Before

After

Changes

Rework and align spacing
Align padding of code blocks with theme padding (and remove schoolbook padding)
Give lists a bit more width to have fewer line breaks.

Checklist

[ ] ~Added markup tests~, or they don't apply here because no markup changes
[ ] ~Updated the changelog at CHANGES.md~ only website changes

Can we line up the arrow better? Looks like it's 1-2px off?

Can we line up the arrow better? Looks like it's 1-2px off?

Does this look better to you now?

Better.

Better.

No, I don't have a Mac.

This simply includes "too much" and too intermingled... there seem to be CSS changes other than the ones explicitly documented in the issue text and it's difficult to properly review "looks better to me" PRs like this (hence the delay, but my apologies for that)... often I'll miss something and then just have to revert it later, or someone will break something in Safari because they were only proofing with Chrome etc... I'm also not convinced all of these changes were needed - though it's hard to judge with them all bundled together.
I did try to look at the commit history but it just seems to be a progression rather than "one change per commit", which would make this easier to follow.  If you feel a few of these changes were critical I'd be happy to see a new PR with individual commits (for easy review - to make sure we aren't changing more than intended)... Or also happy to take a second look at this if you wanted to go back and refactor it into a bunch of tiny individual commits detailing each change and the thinking behind it.
One quick example making the example font larger isn't a terrible idea... but that could be decided separately from other layout changes, etc...


Italic isn't ignored by application

Tested area:
Samsung Galaxy J7 (Android 9.0).
Description:
Italic isn't ignored in the object description by the application.

Verified successfully on Test Flight as a part of Mobile Release 2 Build 17 UIKit/RN v.1.1.1


PyCoral Vs TfLite_runtime

Hello,
By reading the python quickstart:
https://www.tensorflow.org/lite/guide/python
I understand that the tflite_runtime is installed thank to python wheel packages regenerated via pycoral.
Is my understanding right?
In that case what is their any adherence with the tflite_runtime generated via TensorFlow Lite github.
The version also does not match. Official Tensorflow Lite version (so runtime) is v2.3.1 and in the python quick start guide I can see a tflite_runtime version 2.5.0.
Further, running a TPU model using the Tensorflow Lite runtime v2.3.1 is failing:
RuntimeError: Internal: Unsupported data type in custom op handler: 5105160Node number 2 (EdgeTpuDelegateForCustomOp) failed to prepare.
Can you please provide more information regarding pyCoral Vs TfLite_runtime?
Thanks
Vincent

Hi @vinceab
The only different between pycoral and tflite_runtime is the API. We added adapters for easy parsing of outputs for common models like classification and detections. Here is an example of that.
Regarding to version, you should be using 2.5.0 for the tflite_runtime package, not  v2.3.1, may I know where you got v2.3.1 from?

Hi @Namburger
I actually get the TFLite_intepreter by regenerating the python package from TensorFlow github version 2.3.1:
tensorflow/lite/tools/pip_package/build_pip_package.sh
As the last stable version is the version 2.3.1 I am expecting the TFLite_interpreter version is v2.3.1.
May I ask you same question? :) From wich source code did you get the 2.5.0 TFLite_runtime version?
I am a little bit puzzled between TFLite_runtime generated via tensorflow (that's seems not support EdgeTPU) and the information I read regarding pyCoral and the Python quickstart guide.
Many thanks for your lights.
Vincent
Vincent

hello @vinceab
Oh I see, the tflite_runtime package that we built and release is from this TensorFlow commit. The special thing about it is just that is is the same commit that we used to load libedgetpu dynamic library, so it is crucial that both matches or else you'll get:
RuntimeError: Internal: Unsupported data type in custom op handler: 5105160Node number 2 (EdgeTpuDelegateForCustomOp) failed to prepare.

@Namburger
So if I understand well, the version of the tflite_runtime v2.5.0 you released is not correlated with the TensorFlow versioning.
The branch your are referring to is in the master branch? The ancestor seems to be v1.12.1 which is old no? How far are is this commit from the v2.3.1?
Frankly speaking it is a really difficult to understand the merging flow.
Is it planned to align both Coral and TensorFLow?
Thanks
Vincent

Its is now pretty clear by reading this:
https://coral.ai/docs/notes/build-coral
I understand that I need to rebuild the libedgetpu against the tensorflow lite v2.3.1 if I want it working with my runtime.
Vincent

@vinceab yes, and I understand it could get confusing. Anyhow:

Is it planned to align both Coral and TensorFLow?

Yes we do, and actually, it is aligned, it's just that any of the stable released branches right now doesn't contains all the features that libedgetpu currently use. So that commit is actually from the master branch, ahead of all the stable release!
Closing for now, feel free to continue the questions here!


Add multiple data streams, period multiple param, support raw data on both receivers, support RTK Pos and CMP simultaneously.

@waltjohnson, @kylemallory, I still need to test every feature, but this is the current state. I'd love a second pair of eyes on all the raw data callbacks. I think they are handled correctly, but a second pair of eyes would be nice.


Initial version of metacp

Some functionality is missing, but it's already useful.

@olafurpg Addressed all your comments.


Signed-off-by: Sébastien Han<EMAIL_ADDRESS>

Run, this periodically until the packages are fixed.

I think we can close this one?

No, the idea is to not forget about this so when we see it we run it to see if the repos have been fixed.

This has been handle in 27aad73


App crashes when bluetooth is turned off during a scan start/stop cycle

App crashes when bluetooth is turned off during a scan start/stop cycle

Steps to reproduce actual result:
1. Start scanning
2. Turn off bluetooth
3. Stop scanning
Actual result:
- App crashes because bluetoothAdapter is no longer available, so the library references a null object while trying to call bluetoothAdapter.getBluetoothLeScanner().stopScan(scanCallback);

Expected result:
The library should gracefully handle this case where the phone's bluetooth is turned off mid-cycle (which could be during a scan - start or stop, gatt write, gatt read, etc). I fixed this for the scan case in internal.util/RxBleAdapterWrapper.java by just wrapping all the method calls that reference bluetoothAdapter with:
if (isBluetoothEnabled()) {......
}

Hey,
could you post logs in a verbose mode?

Recognized and done. Should be available in 1.3.0-SNAPSHOT shortly.


Tree Structure and Removal

Added C++17 compliant string_view subset
Moved Routes on a Fragments Tree Structure
Added ability to remove routes

Fylax, what is the ultimate origin of your custom "string_view" implementation?  There is no copyright or authorship statement at the top.  Was it copied from another project, or created custom specifically for this PR?  If the later, please add some unit tests for it.

As you requested, I have added unit tests for string view and a disclaimer on the hashing algorithm.

Hello Flyax,
A few things:

What is the origin of the "string_view" implementation?  You document MurmurHash3.

Your patch failed to compile on my system.  gcc ver 7.3.0-r3.  Details here: https://pastebin.com/PmtzKMRh

Hi dennisjenkins75,
despite the documentation stating that both pos and count have default values, seems like that GCC (even version 8) threats the "no arguments" version as a different and not compiled one.
I have changed the tests to remove that specific one. Now it should work on your machine too.

Line 8 of tests/string_view_test.cc should be:
auto sub =  orig.substr(1);
(was missing 'auto', so got syntax error because 'sub' was undefined symbol).
Fixing that and proceeding w/ build I get additional compile-time errors:
/home/djenkins/src/pistache/tests/router_test.cc: In function 'bool match(const Pistache::Rest::Route&, const string&)':
/home/djenkins/src/pistache/tests/router_test.cc:16:30: error: 'const struct Pistache::Rest::Route' has no member named 'match'
return std::get<0>(route.match(req));
^~~~~
/home/djenkins/src/pistache/tests/router_test.cc: In function 'bool matchParams(const Pistache::Rest::Route&, const string&, std::initializer_list<std::pair<std::__cxx11::basic_string, std::__cxx11::basic_string > >)':
/home/djenkins/src/pistache/tests/router_test.cc:25:47: error: 'const struct Pistache::Rest::Route' has no member named 'match'
std::tie(ok, params, std::ignore) = route.match(req);
^~~~~
/home/djenkins/src/pistache/tests/router_test.cc: In function 'bool matchSplat(const Pistache::Rest::Route&, const string&, std::initializer_list<std::__cxx11::basic_string >)':
/home/djenkins/src/pistache/tests/router_test.cc:53:47: error: 'const struct Pistache::Rest::Route' has no member named 'match'
std::tie(ok, std::ignore, splats) = route.match(req);
^~~~~
/home/djenkins/src/pistache/tests/router_test.cc: In function 'Pistache::Rest::Route makeRoute(std::__cxx11::string)':
/home/djenkins/src/pistache/tests/router_test.cc:80:54: error: no matching function for call to 'Pistache::Rest::Route::Route(std::__cxx11::string&, Pistache::Http::Method, makeRoute(std::__cxx11::string)::<lambda(const Pistache::Http::Request&, Pistache::Http::Response)>&)'
return Rest::Route(value, Http::Method::Get, noop);
^

gcc test passed, clang test failed:
In file included from /home/travis/build/oktal/pistache/src/common/http_defs.cc:12:
/home/travis/build/oktal/pistache/include/pistache/date.h:910:33: error: no
template named 'void_t' in namespace 'std'; did you mean '__void_t'?
template <class T, class = std::void_t<>>

I will merge very soon.  For posterity:
I was able to successfully compile and execute the unit tests with this PR using gcc-7.3.0 and clang-6.0.1 on Gentoo Linux.

Fylax, I've received a report that this PR broke "hasParam()".  I am going to rollback this PR (once I figure out how to), and ask the other party to add a unit test for what broke.  Then you can sync to head and rework this PR if you wish.  Notes from IRC channel #pistache:

I think there is a bug in git head. It seems as though the behaviour of hasParam() has changed. Optional parameters (e.g. "/some/endpoint/:foo?") return true even if :foo wasn't actually provided by the client. e.g. the client only requested /some/endpoint/
Yeah hasParam() is definitely broken now. I reverted to c613852cb2595f9a1c9a4847ed0b0aaa017e0d81 and I find that the issue is fine again.


[web-components]: Add file extensions on esm import

The problem
This is a library to be used in the browser and the browser isn't able to resolve ECMAScript modules if they haven't file extensions. That makes that anyone that wants to use the library have to resolve those paths or they couldn't use this components
The solution
Add file extensions to allow users use the library without extra work. It could be interesting to add a linter like this one to be sure all the imports have their extension
Application/website
We are building a web components library for IBM Quantum
Business priority
High Priority = pressing release
What time frame would this ideally be needed by (if applicable)
No response
Examples
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct
[X] I checked the current issues for duplicate issues

@kennylam done in this PR:
https://github.com/carbon-design-system/carbon-for-ibm-dotcom/pull/9089


Bug in version 3.0 that means toDate() with format parameter returns error

Using toDate on a string like '13/04/2008" with value.toDate("d/M/y") leads to error. On previous versions including 2.8 this correctly converted the string to date. value.toDate() with no parameters appears to work, but in the case above where day is first this is not sufficient.

I think the issue is with https://github.com/OpenRefine/OpenRefine/blob/master/main/src/com/google/refine/expr/functions/ToDate.java#L92
This line only triggered when you include the format to be used to parse the string to date as the first argument (because of an optional first argument of a boolean to treat the string as 'day first' or 'month first'). This line needs updating to account for the changes to the internal date format.
@noelmas for the moment if you include the (meant to be optional) boolean argument you can then follow this by a format. However, you may find the boolean is sufficient for your purposes. true means "treat as month first", false means "treat as day first"
"12/04/2008".toDate(true) -> [date 2008-12-04T00:00:00Z]
"12/04/2008".toDate(false) -> [date 2008-04-12T00:00:00Z]
if you provide both boolean and format, then the format will be used anyway:
"12/04/2008".toDate(true,"dd/MM/yyyy") -> [date 2008-04-12T00:00:00Z]

Hmm - this is a bit more messy than I thought. I'm looking at refactoring ToDate.java a bit at the same time as fixing this.

@ostephens thanks for working on this! Yes we can wait for this for 3.1 beta.

I've finally made some progress with this! I've been trying to do a bit of refactoring on the toDate function so the code is easier to understand - I'll ask for feedback on that once I have a PR ready.
However, as I've worked on this I've been wondering on whether the current behaviour of toDate needs changing, or at least what should work/not work when using it. What would be really useful is to collect examples of conversions people would like to see working, and then I can add them into the automated tests we run to make sure that toDate works as we expect. @msaby @ettorerizza @noelmas do you have examples that you want to see working?
I'm especially interested in examples specifying locales, as my understanding of how the system should behave when different locales are specified is not great.
But any examples of starting strings with your expectation of how 'toDate' would convert them to a date would be useful
For reference, these are essentially the behaviours I've currently got tests for:
toDate() -> error (no string, number or date given to toDate)
toDate(null) -> error (no usable string, number or date given to toDate)
toDate("") -> error (no usable string, number or date given to toDate)
toDate(1.0) -> error (no usable string, number or date given to toDate)
toDate("2012-03-01","xxx") -> error (no valid format specified to use when parsing to date)
toDate("2012-03-01") -> 2012-03-01
toDate(01/03/2012") -> 2012-01-03
toDate("01/03/2012",true) -> 2012-03-01
toDate("01/03/2012",false) -> 2012-03-01
toDate("01/03/2012","dd/MM/yyyy") -> 2012-03-01
toDate("2012-03-01","yyyy-MM-dd") -> 2012-03-01
toDate("2012-03-01","MMM","yyyy-MM-dd") -> 2012-03-01
toDate("01/03/2012",false, "MMM","yyyy-MM-dd","MM/dd/yyyy") -> 2012-03-01
toDate(01-六月-2013","zh","dd-MMM-yyyy") -> 2013-06-01
toDate("2012-03-01","XXX") -> 2012-03-01 (ignore invalid date formats and locales)
toDate((long)2012) -> 2012-01-01 (treat numbers as years)
toDate([date 2012-03-01]) -> 2012-03-01 (leave dates as they are)

Well since you didn't ask me... :)  Here's our other TODO's for toDate() ...
Reference: #1122 - documentation improvement (epoch mentioned)
Reference: #608 - Specifically https://github.com/OpenRefine/OpenRefine/issues/608#issuecomment-437331395

Can you please look for dot separated date values in the format string as well?
this does not work: "21.12.2012".toDate("dd.MM.yyyy")
this works: "21.12.2012".toDate(false,"dd.MM.yyyy"), also
this works: "21.12.2012".replaceChars(".","-").toDate("dd-MM-yyyy")
OpenRefine v3.1-beta, MacOS 10.13.6, Safari 12.0.1, Slovak locale (lots of countries use dots, really).
It worked previously in v2.8, doesn't work in 3.0 as well.
Thank you.

@orr721 are you able to check this behaviour is fixed in 3.1 - I think it should work OK, but if not we can work out what the problem is

toDate("2012-03-01","XXX") -> 2012-03-01 (ignore invalid date formats and locales)

This should be an error, not ignored. That was the original behavior, which I've just restored in #3027. Without an error message, the user has no way to know that the system isn't doing what it was told.


Standard way to link to model with multi-word name

This is a counterpart to https://github.com/ansible/django-ansible-base/pull/116 and needs to be combined with that. EDIT: done
Trying to develop some new views on my own, I hit this. There are conflicting expectations about how the view names will be named with a model name has more than 1 word, and whether you use "-" "_" or nothing is actually hard.
Yeah this needs a unit test, but here it is done in the API
{
    "id": 2,
    "url": "/api/v1/teams/2/",
    "related": {
        "created_by": "/api/v1/users/2/",
        "modified_by": "/api/v1/users/1/",
        "organization": "/api/v1/organizations/1/",
        "encryptioner": "/api/v1/encrypted_models/1/"
    },
    "summary_fields": {
        "created_by": {
            "id": 2,
            "username": "angry_spud",
            "first_name": "",
            "last_name": ""
        },
        "modified_by": {
            "id": 1,
            "username": "admin",
            "first_name": "",
            "last_name": ""
        },
        "organization": {
            "id": 1,
            "name": "AWX_community"
        },
        "encryptioner": {
            "id": 1,
            "name": "foo"
        }
    },
    "created_on": "2024-02-03T03:13:22.225324Z",
    "modified_on": "2024-02-03T03:13:59.036461Z",
    "name": "awx_devs",
    "description": "",
    "created_by": 2,
    "modified_by": 1,
    "organization": 1,
    "scrum_master": null,
    "encryptioner": 1
}

The change allows it to have the "encryptioner" in the related & summary_fields, and I legitimately couldn't figure out a way to do that before.
We may need to apply this solution to authenticator_maps as well... I remember seeing the warning that it couldn't reverse the view for that.

I think this is the way. And I think it's amazing to have documentation for this now.


[Bug]: Optimization of Unet fails 6950 XT

What happened?
This appeared to me to be the same issue as 510 and 301, though I know nothing. I ran the following commands:

conda create --name olive python=3.9
conda activate olive
pip install olive-ai[directml]==0.3.1
git clone https://github.com/microsoft/olive --branch v0.3.1
cd (to relevant directory)
pip install -r requirements.txt
python stable_diffusion_xl.py --optimize

I've attached the log, as well as a DXDIAG, but it errors out when optimizing unet saying "failed to run olive on gpu-dml".... "887a0006 the gpu will not respond to more commands".
DxDiag.txt
ErrorLog.txt
Version?
0.3.1

The following error message seems be related to DirectML EP.
onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : D:\a_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\ExecutionProvider.cpp(896)\onnxruntime_pybind11_state.pyd!00007FFE31C80201: (caller: 00007FFE31C80C2F) Exception(2) tid(3c14) 887A0006 The GPU will not respond to more commands, most likely because of an invalid command passed by the calling application.

@jstoecker, do you have any insight?

seems similar with https://github.com/microsoft/Olive/issues/510

This is DXGI_ERROR_DEVICE_HUNG during inference/evaluation, which typically happens when some GPU work is taking excessively long. The recent AMD driver optimizations for stable diffusion / multi-head attention target the RDNA 3 architecture (e.g., the 7000 series, like the Radeon RX 7900 XTX) but not the RDNA 2 (6000 series). Still, we can try to repro this on an RDNA card to see if anything jumps out.

6800xt has same err

Error on my 6900XT as well

Same Error occurred in AMD Ryzen 7 7840U w/ Radeon 780M Graphics.

GPU queue dose not disable TDR.
https://github.com/microsoft/onnxruntime/issues/20094
User can manually disable this TdrLevel to test again.
Check this to set it.
https://learn.microsoft.com/en-us/windows-hardware/drivers/display/tdr-registry-keys
We have measured that sometimes it have many jobs in a command list no matter convert the file or run the model in lower end GPUs or large model
Remember that enlarge your virtual memory to prevent memory not enough.
200GB is better for SDXL


a range of errors

I am having trouble getting cosmos to run reliably. Perhaps some of the dependencies have been updated and that causes trouble? Or perhaps I'm doing something wrong, in which case we can investigate what in Cosmos or its documentation can be improved to avoid this in the future.
First a usability issue. Initially I forgot to export the React component under test as the default of the module. I got some error about type.toUpperCase not being a function. It'd be nicer if this failure case was detected and the error can be more informative, i.e. failed to find component in module, or whatever.
When actually running the system, I am getting a few more errors. After initial load I get one of these:
Uncaught Error: Invariant Violation: addComponentAsRefTo(...): Only a ReactOwner can have refs. This usually means that you're trying to add a ref to a component that doesn't have an owner (that is, was not created inside of another component's `render` method). Try rendering this component inside of a new top-level component which will hold the ref.

when I try to edit the fixture data in the browser window, I get a ton of these, I think one for every key press:
Uncaught Error: Invariant Violation: removeComponentAsRefFrom(...): Only a ReactOwner can have refs. This usually means that you're trying to remove a ref to a component that doesn't have an owner (that is, was not created inside of another component's `render` method). Try rendering this component inside of a new top-level component which will hold the ref.

probably due to the error the component doesn't update either.
When I edit the .jsx file with the component in it, I get the following message:
It appears that React Hot Loader isn't configured correctly. If you're using NPM, make sure your dependencies don't drag duplicate React distributions into their node_modules and that require("react") corresponds to the React instance you render your app with. If you're using a precompiled version of React, see https://github.com/gaearon/react-hot-loader/tree/master/docs#usage-with-external-react for integration instructions.

and the component UI does not update properly either. Only a full browser reload will reflect my edit.

Can you check if you have multiple Reacts on one page? These errors definitely look like a “duplicate React” issue.

I'll add that I followed the Cosmos installation instructions on a clean package (though I did do a --save-dev for cosmos-js).
I was using npm 2.5.1. I just updated it to npm 2.13.0 to see whether this makes any difference. Unfortunately it does not. (I do get a ton of exciting warnings involving peerDependencies now during the npm run)
I tried installing without --save-dev, but this also has the same behavior.
So from my end it seems like installing cosmos as per instructions gives you these errors in a repeatable fashion.

I added issue #144 about cosmos not ignoring editor droppings. It's conceivable that this has something to do with these errors, though it seems unlikely, but I thought I'd mention it.

Is there an easy way to test whether there are multiple reacts included in a webpack?

Can you check if you have multiple Reacts on one page? These errors definitely look like a “duplicate React” issue.

+1

Is there an easy way to test whether there are multiple reacts included in a webpack?

Look for something unique from the React source in an unminified bundle and see if you find it twice or more. My quickest way to check is to see if the bundle is double the lines of code it usually is, because React is 99% of the bundle in a small project :)

I'll add that I followed the Cosmos installation instructions on a clean package (though I did do a --save-dev for cosmos-js).

Can you open source that setup if you started from scratch? It would be easier for us to investigate this way.


Provide radio with page heading example to make following advice easier

On the radio button page we tell people:

If you are asking just 1 question per page as the GOV.UK design system recommends, you can set the contents of the <legend> as the page heading. This is good practice as it means that screen reader users will only hear the contents once.

But we don't include an example of this.
We should make it easier for people to follow our advice.

I've copied this to nhsuk-service-manual-backlog, it's issue 94


Update jupyter and spark versions

Signed-off-by: Antonio Jesus Navarro Perez<EMAIL_ADDRESS>

@ajnavarro Is this tested?

@smola yep, I tested on https://github.com/src-d/gitbase-spark-connector/blob/master/Dockerfile


fix(validator): log api errors as warn if node is syncing

Motivation
Logs mentioned in https://github.com/ChainSafe/lodestar/issues/5359#issuecomment-1529027981 should not be shown as errors on the VC if beacon node is syncing.
Description
HttpError is not thrown anymore, instead API client returns a ApiClientResponse
https://github.com/ChainSafe/lodestar/blob/913a51d8807d2111d1e67e50f19277e9cc953eac/packages/api/src/utils/client/client.ts#L81-L85
which will be handled by ApiError.assert and if not ok, will throw ApiError
https://github.com/ChainSafe/lodestar/blob/913a51d8807d2111d1e67e50f19277e9cc953eac/packages/api/src/utils/client/httpClient.ts#L42-L46
Before
Error: Error on getProposerDuties - Service Unavailable: Node is syncing - headSlot 5529152 currentSlot 5529417
    at Function.assert (file:///usr/app/packages/api/src/utils/client/httpClient.ts:44:13)
    at BlockDutiesService.pollBeaconProposers (file:///usr/app/packages/validator/src/services/blockDuties.ts:155:14)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at BlockDutiesService.pollBeaconProposersAndNotify (file:///usr/app/packages/validator/src/services/blockDuties.ts:129:5)
    at BlockDutiesService.runBlockDutiesTask (file:///usr/app/packages/validator/src/services/blockDuties.ts:87:9)
    at Clock.runAtMostEvery (file:///usr/app/packages/validator/src/util/clock.ts:96:7)

After
warn: Node is syncing - Error on getProposerDuties - Service Unavailable: Node is syncing - headSlot 5564286 currentSlot 5565452
warn: Node is syncing - Error on getProposerDuties - Service Unavailable: Node is syncing - headSlot 5564383 currentSlot 5565453
warn: Node is syncing - Error on getProposerDuties - Service Unavailable: Node is syncing - headSlot 5564479 currentSlot 5565454

considering the log level if node is syncing, should we use warn instead of info? As this log is quite expected I am guessing we are correct to use  info here

IMO a warn makes sense here. Once the node is synced, any "Node is syncing" message is likely not desired. Good to show a yellow light in that case.

:tada: This PR is included in v1.9.0 :tada:


Allow "clip" to be specified in the mark/role config.

A couple items to note:

There was a conditional logic bug in the clip property assignment (fixed by 97a693e).
Mark config objects currently are assumed to contain only valid mark encoding properties. As a result, any config clip property will also be written to every scenegraph item, not just applied to the mark definition. This occurs at encode-util.js#L53. One possible fix would be to update the "skip" logic to include a check for the clip property. The downside is that any additional mark-level (as opposed to item-level) properties we want to add in the future (interactive? zindex?) would also need to be added here as well. That said, I'm OK with the "quick fix" for now, if we're resigned to interleaving mark-level and item-level properties in the same config object. An annoying complication is that clip is a valid item-level property for group marks...
We'll need to update the documentation in the vega/vega repo once changes go through here.

@arvind, any thoughts on issue 2 above? Given the ambiguity of mark-level vs. item-level application, I'm now tempted to punt on this PR. How necessary / useful is this feature for Vega-Lite, etc?

Sorry for the delay (and thanks for the bug fix!). I'm fine with having a more ambiguous mark configuration in favour of one that's more hierarchy. I suspect that only the most discerning of users would catch the ambiguity here, and it's something that I think can be explained via documentation if need be.
For a Vega-Lite use case, see this scatterplot example. If the user explicitly sets scale domains that do not correspond to the data's [min, max], points will overflow. Vega-Lite does expose group-level clipping ("config": {"cell": {"clip": true}}) but that has the effect of clipping axes and legends as well.

OK, so the use case is to enable end-user configuration? My understanding is that in the context of selections we clip the marks to the data rectangle automatically. Should this be the default, regardless of selection status? (Of course, this is arguably tangential to the current issue.)
I'll take a look at "special casing" the clip property, but I'm still a bit hesitant...

Funnily enough, I was just thinking about whether "clip": true should be the default for all (non-group) marks Vega-Lite produces since you could have a bound-interval in one view manipulate another's that wouldn't have been clipped under our current scheme.
So I'm ok with punting on this PR now.


The Azure node of Server Explorer has been retired in Visual Studio 2022.  Article needs updating.

[Enter feedback here]

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 0065e05e-1e7e-20aa-badb-dfe2d297ddd2
Version Independent ID: aab0065a-72ef-b6ea-5ac1-d85e96e9e099
Content: Troubleshoot with Visual Studio - Azure App Service
Content Source: articles/app-service/troubleshoot-dotnet-visual-studio.md
Service: app-service
GitHub Login: @cephalin
Microsoft Alias: cephalin

Hi @davecatt thank you for your feedback! We have assigned this issue to the author to review further and take the next course of action.

Thanks for the feedback! I have assigned the issue to the content author to investigate further and update the document as appropriate.

#reassign:msangapu-msft

Hello, is there any update on this? I need to debug an app deployed to Azure App Service and it's difficult to debug since I'm trying to modify it use the built-in App Service authentication. I'm on VS 2022, and it seems that both the Cloud Explorer and the Azure node in the Server Explorer (based on the documentation and this thread, respectively) have been made have been made either deprecated or completely obsoleted.
Intuitively, I tried adding the app URL as a server, but that didn't work either. Is there a different or better way to approach this?

This article is only applicable for VS 2019. I've added a note to clarify this.
For VS 2022, see: https://learn.microsoft.com/en-us/visualstudio/debugger/remote-debugging-azure-app-service?view=vs-2022
#please-close


PacketScope vlan tag ignore

Add vlan tag ignore check box to drop all vlan tags with filtering

checkbox in upper right corner

commit, needs regression before closing
fmadio/fmad20@6ee3227

regression added
fmadio/fmadio20_regression@51f0d92


feat: Add Discord bot for GitHub info retrieval

Implement !repo and !issue commands for GitHub info.
Extract and display repo and issue details from links.

Notes
structuring held off for a future PR


Some decision tree branching may be unnecessary

@dalelane https://github.com/IBM/taxinomitis/pull/221 is amazing, awesome work! 👍  Thanks as always for sharing such great work in the open! ❤
I made some limited test data just to check this out, and noticed that the explanation had some parts of the tree that didn't seem necessary since they all result in the same classification.  Here's an example, with the parts highlighted that seem unnecessary:

Maybe the way the tree is constructed can lead to this?  I'm not sure.  I tried to see where the tree is constructed, guessing that this was done by a service but only got as far as https://github.com/IBM/taxinomitis/blob/db15bf81260149efeb28a29d2e0900580bfb6406/src/lib/training/numbers.ts#L328 and wasn't quite sure from there.
I'm not sure if this is an actual issue, as in something isn't working as it should, or just something that might be worth adding to your great explanation up top.
Also, http://www.r2d3.us/visual-intro-to-machine-learning-part-1/ has some awesome visuals of decision trees that might be a good fit as a "learn more" link in the explanation up to as well.

I'll have a look, thanks!
In the meantime, the code where this is done lives here:
https://github.com/IBM/taxinomitis/blob/db15bf81260149efeb28a29d2e0900580bfb6406/serverless-functions/mltraining-numbers/numbers.py#L43-L56

@dalelane awesome, thanks!
Yeah, it seems like the scikit learn algorithm is an approximation since for larger real-world data sets it can be intractable to compute the optimal representation link:

The problem of learning an optimal decision tree is known to be NP-complete under several aspects of optimality and even for simple concepts. Consequently, practical decision-tree learning algorithms are based on heuristic algorithms such as the greedy algorithm where locally optimal decisions are made at each node. Such algorithms cannot guarantee to return the globally optimal decision tree. This can be mitigated by training multiple trees in an ensemble learner, where the features and samples are randomly sampled with replacement.

I suppose to simplify things for folks it'd be possible to write a function that does that simplification on the tree that's returned, since for ML for kids models the trees wouldn't be expensive to prune after the fact.  sci-kit learn exposes the internals of the decision tree structure (link) but the graphviz function takes a dt so that might involve hacking rather than just walking and modifying the tree.  Alternately it might be worth just adding a line of explanation alerting learners and teachers to this, and calling it out as an example of the kind of approximation that is common in ML.
Relatedly, it looks like subsequent runs are non-deterministic link:

The features are always randomly permuted at each split. Therefore, the best found split may vary, even with the same training data and max_features=n_features, if the improvement of the criterion is identical for several splits enumerated during the search of the best split. To obtain a deterministic behaviour during fitting, random_state has to be fixed.

This might be good to either add a line about in the explanation, or to avoid this to simplify the experience and fix a random seed so that folks aren't confused if they get different trees from different training runs on the same data set (eg, within a class).

Related issues:
https://github.com/scikit-learn/scikit-learn/issues/10810
https://github.com/scikit-learn/scikit-learn/issues/6557
https://github.com/scikit-learn/scikit-learn/issues/4630