[GSoC2024] Fix import of outside track shapes in Datumaro-based formats

Motivation and context

Main Issue: #7571
Related Issue: #2339
I've reproduced the issue mentioned in #7571 when exporting and importing annotations using both the Datumaro and Coco 1.0 formats. Specifically, the "Switch outside" attribute isn't being applied as expected. After some investigation, I pinpointed the root cause to be the absence of the "outside" attribute in the exported annotations.
To address this, I've made adjustments to the binding.py file to bypass the track_id during annotation import. This modification appears to solve the issue regarding the "Switch outside" attribute. However, it introduces a new concern: the potential loss of information, including keyframes and track_id.
While this workaround offers a temporary fix, I'm contemplating a more holistic approach that entails properly handling the "outside" attribute during both the export and import processes of annotations. This method could preserve the integrity of the annotations while ensuring the functionality of the "Switch outside" attribute.
I'm reaching out for feedback or suggestions on my proposed solution. Is there a preference for one of these approaches, or might there be another avenue I haven't considered?
Looking forward to your insights.
Checklist

[x] I submit my changes into the develop branch
[x] I have created a changelog fragment
[ ] I have updated the documentation accordingly
[x] I have added tests to cover my changes
[x] I have linked related issues (see GitHub docs)
[ ] I have increased versions of npm packages if it is necessary
(cvat-canvas,
cvat-core,
cvat-data and
cvat-ui)

License

[x] I submit my code changes under the same MIT License that covers the project.
Feel free to contact the maintainers if that's a concern.

Hi, thank you for sending the PR! Just disabling tracks in the export doesn't seem as a proper solution. Probably, something along these lines should be adjusted, as this is the common logic for several formats. The handling logic (and so, the solution) should be similar to this fragment.

Hi, thank you for sending the PR! Just disabling tracks in the export doesn't seem to be a proper solution, as the whole point of using the outside property is to use it with tracks. Probably, something along these lines should be adjusted, as this is the common logic for several formats. The handling logic (and so, the solution) should be similar to this fragment.

Thanks for your advice. Will make necessary changes soon!

Hi @zhiltsov-max , I have made necessary changes and add a test to it. Please have a look

Hii, @zhiltsov-max, I have transferred the code to tests/python. I have checked with all the REST API, SDK and CLI tests. Please take a look.

Hi, please fix the issues reported by the CI.

Hope the changes will fix the issues flagged by the CI.

Hi @zhiltsov-max , could you help me with this? I have tried declaring the tracked coco keypoints annotations, but it keeps giving me error on line 2918, which I'm not really sure why.

Here is the error code
And after checking the code for coco keypoints, I realised that there is a need to also append outside true to track['element']['shapes']  (the changes are on line 2182 bindings.py).  Is this something aligned with the requirement?

Hi, please check server logs with
docker logs test_cvat_server_1, docker logs test_cvat_worker_import_1, docker logs test_cvat_worker_export_1.

And after checking the code for coco keypoints, I realised that there is a need to also append outside true to track['element']['shapes'] (the changes are on line 2182 bindings.py). Is this something aligned with the requirement?

I'm not sure I got the question. Each skeleton point (stored in the 'element' list) is a separate track, but it can also be affected by the parent skeleton outside.

I realised if I only change the last parent skeleton outside to true, the skeleton points will still appear for all the frames. So, I have also added outside to skeleton points.
I am unsure about whether I should follow the parent skeleton outside , or append outside individually to each skeleton point.

I am unsure about whether I should follow the parent skeleton outside , or append outside individually to each skeleton point.

The logic should be as following. If the parent skeleton is outside, its points should also be outside. If the skeleton is visible, each point can be controlled separately. Probably, it's not necessary to copy skeleton's outside to the points, if the skeleton is outside, as it should be handled by UI and by the server during the task export, though it's better double-checking this.

Hi @zhiltsov-max  , I have added the test for coco keypoints with outside true.
However, while running pytest, test_can_export_and_import_skeleton_tracks_in_coco_format in test_tasks.py had failed.
This is track[‘shapes’] under original_annotation:
{"type": "rectangle", "frame": 0, "points": [3.0, 2.0, 2.0, 3.0], “outside”=false},
{"type": "rectangle", "frame": 3, "points": [3.0, 2.0, 2.0, 3.0], “outside”=false},
and this is track['shapes'] under imported_annotation
{"type": "rectangle", "frame": 0, "points": [3.0, 2.0, 2.0, 3.0], “outside”=false},
{"type": "rectangle", "frame": 1, "points": [3.0, 2.0, 2.0, 3.0], “outside”=false},
{"type": "rectangle", "frame": 2, "points": [3.0, 2.0, 2.0, 3.0], “outside”=false},
{"type": "rectangle", "frame": 3, "points": [3.0, 2.0, 2.0, 3.0], “outside”=false},
{"type": "rectangle", "frame": 4, "points": [3.0, 2.0, 2.0, 3.0], “outside”=false},
{"type": "rectangle", "frame": 5, "points": [3.0, 2.0, 2.0, 3.0], “outside”=false},
{"type": "rectangle", "frame": 6, "points": [3.0, 2.0, 2.0, 3.0], “outside”=false},
This is because when we export the annotation, we will add annotations to all the frames if outside is not true. Hence the original_annotation that we get by using task.get_annotations() does not reflect what we actually exported out.
I am not sure how I should proceed with this. Is changing the annotation used by this test case an appropriate way of dealing with this, or my current way of dealing with the outside attribute is wrong?

I think there should be a check for the keyframe annotation attribute, and all annotations in between 2 keyframes should not be imported.

Okay, got it. Will be working on this now.
However, for the case of test_can_export_and_import_skeleton_tracks_in_coco_format, I think it does not include keyframe attribute in annotation.json. Does that mean I need to modify the annotation?

Does that mean I need to modify the annotation?

Yes, I suppose so. Probably, it was supposed to be there. I think, earlier we had logic that all the imported frames from non-track native formats should be keyframes, but we have probably moved away from this idea since then.

Hi, @zhiltsov-max  , here is the summary of the changes:

Added to_boolean() to all the attributes.get().
This is because I realised that in PASCAL VOC format, the keyframe value is ‘False’ (string) instead of False (boolean). Hence, I made this function and added it to all attributes.get() to make sure it will not convert to the wrong boolean.

Added ending_tracks to binding.py import_dm_annotations()
It is a temporary holding area for the tracks and elements with keyframe and outside equal to false after the last keyframe = true. In the last part of the function, ending_tracks is used to determine whether or not to append outside = true to tracks[‘shapes’] and tracks[‘elements’][‘shapes’]

Please have a look.

while testing with coco keypoints format on https://app.cvat.ai/, I found this bug. If I annotate the first frame with skeleton track, put third frame as keyframe and fifth frame with switch outside, It will give
ValueError: operands could not be broadcast together with shapes (0,) (4,)
I am currently solving this issue, but I think this is a different issue from the current one. Should I create a new issue on this?

Hi @zhiltsov-max , I have made the necessary changes based on your advice. The finalize_tracks basically rearranges and remove tracks['shapes'] with outside=false from tracks.

Hi @zhiltsov-max , I have modified the code based on your comments. But there are still left with two that need your clarification.

@zhiltsov-max Could you also take a look at the two reviews that you made previously? I am not very sure about some parts.

@zhiltsov-max Could you also take a look at the two reviews that you made previously? I am not very sure about some parts.

The variable is not used.

The variable was added in this PR and never used. It's clearly visible in the code editor. Probably, you supposed to do something different.

Probably, instance_data.frame_step is not needed here.

This suggestion has probably been my mistake, as it's related to frame skip after the last visible frame.

The variable was added in this PR and never used. It's clearly visible in the code editor. Probably, you supposed to do something different.

I have checked with the export and import of coco keypoints format. However, I realised that when we import the annotations, the annotations are always missing this element_keyframe information, causing it to be set as true
element_keyframe = dm.util.cast(element.attributes.get('keyframe', None), bool, True);

I realised that when we import the annotations, the annotations are always missing this element_keyframe information

It can be so for the correct tracks exported by CVAT in these formats, but this is not guaranteed to be so in arbitrary input data, which can be human-crafted or created by other tools.

causing it to be always set as true
element_keyframe = dm.util.cast(element.attributes.get('keyframe', None), bool, True);
Do you have any idea on this?

Probably, this default of true was correct at some point before, but not now. I suggest that we just use false instead, and then use the keyframe algorithm added in this PR.

Hi @zhiltsov-max , I have refactored the code and made some changes to your suggested change. I added the check for  prev_shape is None so that the issue of has_gap is not defined and will not occur

Probably, this default of true was correct at some point before, but not now. I suggest that we just use false instead, and then use the keyframe algorithm added in this PR. Actually, the idea of adding this algorithm is exactly because the input data is not expected to be reliable in this regard, and with the algorithm, it's not really important what's the default value.

I have tried the method you suggested, however, while I ran the test, there will be an issue on this test test_can_export_and_import_skeleton_tracks_in_coco_format, checking on the number of tracks['elements']['shapes']
This is because when we export and import the annotation again, we lose the keyframe=true information. Hence not all the number of element shapes is imported, causing assertion error.
All the code committed had been tested. Please take a look.

I have tried the method you suggested, however, while I ran the test, there will be an issue on this test test_can_export_and_import_skeleton_tracks_in_coco_format, while checking on the number of tracks['elements']['shapes']

This is because when we export and import the annotation again, we lose the keyframe=true information. Hence not all the number of element shapes is imported, causing assertion error.

Ok, makes sense. Yes, the intermediate keyframes were overlooked, thanks for noticing.

Looks good overall, but it would be nice to add a couple of tests for the corner-cases handled by this code, such as only 1 outside frame, outside without keyframe, intermediate keyframe, no keyframes in a track etc.

Looks good overall, but it would be nice to add a couple of tests for the corner-cases handled by this code, such as only 1 outside frame, outside without keyframe, intermediate keyframe, no keyframes in a track etc.

sure will work on this now.

Hi, @zhiltsov-max , I have added tests for corner cases for all tracked formats.
And a gentle reminder on one of the previous reviews, I hope to get your advice on that.

Hi @zhiltsov-max , it does make sense to check for keyframe for this corner case. However, after checking with the database and GET Method for task annotations (_init_tracks_from_db) (get_task_data), I found out that there is no keyframe value in the database.

Currently the alternative solution in testing this corner case I can think of is by checking the number of tracks['shapes']. From the algorithm used in bindings.py, we know that only keyframe or outside = True will be imported, hence, if there exist two tracks['shapes'], it means annotation is imported correctly. If you have any better ideas for testing this corner case, feel free to share them with me and I will implement them.
I have also deleted test_can_export_and_import_skeleton_tracks_in_coco_format test as this test is the same as test_export_and_import_coco_keypoints_with_intermediate_keyframe

Ok, yes, only the keyframes are stored in the DB and returned with this request to annotations. I agree that we can check the shapes without looking for the keyframe attribute, as you suggested.

Ok, yes, only the keyframes are stored in the DB and returned with this request to annotations. I agree that we can check the shapes without looking for the keyframe attribute, as you suggested. We still need to check the outside attribute though.

Yeah, thanks for reminding me of this. I have added it into test_export_and_import_tracked_format_with_outside_true and test_export_and_import_coco_keypoints_with_outside_true.
However, for the one outside case, I think the intended outcome for the annotations would be empty, as there is no visible frame, so assert len(imported_annotations["tracks"]) == 0 is added instead of checking for outside = True

Ok, could you also add a more complex test, maybe similar to what was discussed in https://github.com/cvat-ai/cvat/pull/7669#discussion_r1561234398 ? The currently added tests are working, but they are pretty simple and can miss some problems.

@zhiltsov-max , I have added tests to check annotations with two gaps and check if the number of outside=True ==2.
While creating this test, I found this bug in bindings.py
if not prev_is_visible or shape.keyframe or (    has_gap := prev_shape.frame + instance_data.frame_step < shape.frame ) :
Frame with outside=True is always keyframe=True, so whenever we run this line of code, it will exit the checking in the second condition, causing has_gap to always set to False. Hence, I have changed the position between them to solve this bug.
I can't really think of any other more complex test cases at this moment. Do you have any suggestions?

@zhiltsov-max  , for the error that occurred in cypress (features/ground_truth_jobs.js), I have run it on my laptop, and it seems like there is no problem over here. Could you give some advice on this issue?

Could you give some advice on this issue?

Don't worry, cypress tests often fail for no reason. We typically restart tests in such cases.

Hi all, I think I'm running into this issue. I got here from the description in https://github.com/cvat-ai/cvat/issues/7757. Running in a locally hosted environment. I have several keyframes it looks like with points: []:
 "tracks": [
            {
                "frame": 0,
                "group": 0,
                "source": "manual",
                "shapes": [
                    {
                        "type": "skeleton",
                        "occluded": false,
                        "outside": false,
                        "z_order": 0,
                        "rotation": 0.0,
                        "points": [
                            11.899662168245413,
<PHONE_NUMBER>69599,
<PHONE_NUMBER>080511,
                            0.0
                        ],
                        "frame": 0,
                        "attributes": []
                    },
                    {
                        "type": "skeleton",
                        "occluded": false,
                        "outside": false,
                        "z_order": 1,
                        "rotation": 0.0,
                        "points": [],
                        "frame": 67,
                        "attributes": []
                    },
                    {
                        "type": "skeleton",
                        "occluded": false,
                        "outside": false,
                        "z_order": 2,
                        "rotation": 0.0,
                        "points": [],
                        "frame": 239,
                        "attributes": []
                    },

etc. I'm showing 40 occurrences total for my task json.
Is there a workaround for this? I have several places where I have a skeleton with only one point visible and I'd rather not delete the entire ID. I was hoping to have this dataset exported and training tonight, but I fear that's not likely at this stage.

Is there a workaround for this? I have several places where I have a skeleton with only one point visible and I'd rather not delete the entire ID. I was hoping to have this dataset exported and training tonight, but I fear that's not likely at this stage.

Hi, I am currently still figuring out where the bug comes from. And for this pull request, it does not solve the error. I don't know if other contributors have solved it; maybe you can search for them.

I'm not super familar with the backend of CVAT. Where can I view the data that CVAT is using in the background to drive all the functionality? It knows these objects exist somewhere since it correctly loads them and displays them. If I have to delve into the data backend to get my dataset information, I'm fine with that.

I'm not super familiar with the backend of CVAT. Where can I view the data that CVAT is using in the background to drive all the functionality? It knows these objects exist somewhere since it correctly loads them and displays them. If I have to delve into the data backend to get my dataset information, I'm fine with that.

Maybe you can take a look at this file coco.py, this is the entry point when you import and export the annotations as coco-keypoints. You can set breakpoints here and check if the annotations are handled correctly. guide for running debugger
Most of the logic in handling annotations is in this file bindings.py and this directory dataset_manager. Can have a look on these.

I found this snippet which I find interesting.
        {
            "id": 96,
            "label_id": 26,
            "frame": 7007,
            "group": 0,
            "source": "manual",
            "shapes": [
                {
                    "id": 32148,
                    "type": "skeleton",
                    "frame": 7007,
                    "occluded": false,
                    "outside": false,
                    "z_order": 0,
                    "rotation": 0.0,
                    "points": [
<PHONE_NUMBER>744306,
<PHONE_NUMBER>248524,
<PHONE_NUMBER>403112,
<PHONE_NUMBER>995109
                    ],
                    "attributes": []
                },
                {
                    "id": 32229,
                    "type": "skeleton",
                    "frame": 7132,
                    "occluded": false,
                    "outside": false,
                    "z_order": 0,
                    "rotation": 0.0,
                    "points": [],
                    "attributes": []
                },
                {
                    "id": 32230,
                    "type": "skeleton",
                    "frame": 7249,
                    "occluded": false,
                    "outside": false,
                    "z_order": 0,
                    "rotation": 0.0,
                    "points": [],
                    "attributes": []
                },
                {
                    "id": 32364,
                    "type": "skeleton",
                    "frame": 7316,
                    "occluded": false,
                    "outside": false,
                    "z_order": 0,
                    "rotation": 0.0,
                    "points": [],
                    "attributes": []
                },
                {
                    "id": 32365,
                    "type": "skeleton",
                    "frame": 7588,
                    "occluded": false,
                    "outside": false,
                    "z_order": 0,
                    "rotation": 0.0,
                    "points": [],
                    "attributes": []
                },
                {
                    "id": 32539,
                    "type": "skeleton",
                    "frame": 8153,
                    "occluded": false,
                    "outside": false,
                    "z_order": 0,
                    "rotation": 0.0,
                    "points": [],
                    "attributes": []
                }
            ],
            "attributes": [],
            "elements": [
                {
                    "id": 97,
                    "label_id": 27,
                    "frame": 7007,
                    "group": 0,
                    "source": "manual",
                    "shapes": [
                        {
                            "id": 32149,
                            "type": "points",
                            "frame": 7007,
                            "occluded": false,
                            "outside": true,
                            "z_order": 0,
                            "rotation": 0.0,
                            "points": [
<PHONE_NUMBER>744306,
<PHONE_NUMBER>248524
                            ],
                            "attributes": []
                        },
                        {
                            "id": 32150,
                            "type": "points",
                            "frame": 7011,
                            "occluded": false,
                            "outside": true,
                            "z_order": 0,
                            "rotation": 0.0,
                            "points": [
<PHONE_NUMBER>7443407,
<PHONE_NUMBER>2485157
                            ],
                            "attributes": []
                        },
                        {
                            "id": 32151,
                            "type": "points",
                            "frame": 7014,
                            "occluded": false,
                            "outside": true,
                            "z_order": 0,
                            "rotation": 0.0,
                            "points": [
<PHONE_NUMBER>2443407,
<PHONE_NUMBER>7485157
                            ],
                            "attributes": []
                        },
                        {
                            "id": 32152,
                            "type": "points",
                            "frame": 7017,
                            "occluded": false,
                            "outside": true,
                            "z_order": 0,
                            "rotation": 0.0,
                            "points": [
<PHONE_NUMBER>2443407,
<PHONE_NUMBER>7485884
                            ],
                            "attributes": []
                        },
                        {
                            "id": 32153,
                            "type": "points",
                            "frame": 7021,
                            "occluded": false,
                            "outside": true,
                            "z_order": 0,
                            "rotation": 0.0,
                            "points": [
<PHONE_NUMBER>7443407,
<PHONE_NUMBER>2485884
                            ],
                            "attributes": []
                        },
                        {
                            "id": 32154,
                            "type": "points",
                            "frame": 7024,
                            "occluded": false,
                            "outside": true,
                            "z_order": 0,
                            "rotation": 0.0,
                            "points": [
<PHONE_NUMBER>2443407,
<PHONE_NUMBER>7485884
                            ],
                            "attributes": []
                        },
                        {
                            "id": 32155,
                            "type": "points",
                            "frame": 7027,
                            "occluded": false,
                            "outside": true,
                            "z_order": 0,
                            "rotation": 0.0,
                            "points": [
<PHONE_NUMBER>2443407,
<PHONE_NUMBER>7485884
                            ],
                            "attributes": []
                        },
                        {
                            "id": 32156,
                            "type": "points",
                            "frame": 7032,
                            "occluded": false,
                            "outside": true,
                            "z_order": 0,
                            "rotation": 0.0,
                            "points": [
<PHONE_NUMBER>7443407,
<PHONE_NUMBER>7485884
                            ],
                            "attributes": []
                        },
                        {
                            "id": 32157,
                            "type": "points",
                            "frame": 7035,
                            "occluded": true,
                            "outside": false,
                            "z_order": 0,
                            "rotation": 0.0,
                            "points": [
<PHONE_NUMBER>872451,
<PHONE_NUMBER>567143
                            ],
                            "attributes": []
                        },
                        {
                            "id": 32158,
                            "type": "points",
                            "frame": 7039,
                            "occluded": true,
                            "outside": false,
                            "z_order": 0,
                            "rotation": 0.0,
                            "points": [
<PHONE_NUMBER>999965,
<PHONE_NUMBER>499965
                            ],
                            "attributes": []
                        },

It seems to differentiate between skeleton and points, even though I only have skeleton and rectangle tracks in my annotations.  I'm thinking it tracks the points individually, since it seems to be tracking occlusion and outside correctly there. So what is the purpose of the 'skeleton' section?

The more I think about it, why do skeleton shapes have point data at all? It looks like the points are tracked as elements of the skelton shape.

Ok, so I have a workaround. Borrowing ideas from https://github.com/cvat-ai/cvat/issues/7589 I was able to find a way to get my annotation json file from localhost:8080/api/jobs/<Job_ID>/annotations?scheme=json. I opened it in vscode, found all instances of skeletons with empty points and removed them. Then at the bottom of the annotation api page, I submitted the cleaned json file via PUT in the form box. I went back into my job and ran an export.
At first it gave me
  <image id="7448" name="BaseballTrainingData/auto-annotated-bat/frame-08138.png" subset="Train" task_id="11" width="1920" height="1080">
    <skeleton label="bat-kp" source="manual" z_order="0">
    </skeleton>
    <box label="bat" source="manual" occluded="0" xtl="881.56" ytl="761.19" xbr="1055.99" ybr="1001.40" z_order="0">
    </box>
    <skeleton label="bat-kp" source="manual" z_order="0">
    </skeleton>
    <skeleton label="bat-kp" source="manual" z_order="0">
    </skeleton>
    <skeleton label="bat-kp" source="manual" z_order="0">
    </skeleton>
  </image>

But, I clicked save and now it is giving me the correct output:
  <image id="10602" name="BaseballTrainingData/auto-annotated-bat/frame-11292.png" width="1920" height="1080">
    <box label="bat" source="manual" occluded="0" xtl="1218.20" ytl="1047.70" xbr="1381.48" ybr="1080.00" z_order="0">
    </box>
    <skeleton label="bat-kp" source="manual" z_order="0">
      <points label="bat-knob" source="manual" outside="1" occluded="0" points="1023.90,1064.01">
      </points>
      <points label="bat-barrel-end" source="manual" outside="0" occluded="0" points="1379.36,1064.33">
      </points>
    </skeleton>
  </image>

I suspect there is something fishy going on where multiple skeleton shapes are being created. Essentially, any skeleton shape with no points is invalid and should not exist in the DB. Not sure how they got there, but I don't think the formatter is the issue. I have a feeling it is somewhere on the shape creation side, but I don't have the tools set up to find out. Hope that helps.
P.S. I exported as CVAT for Images, but it has the same broadcast error.

@Drise13, hi. In CVAT, a skeleton is a composite annotation, which stores its parts in the elements field, so its points indeed are stored as separate shapes, 1 point per shape. The points field is not used with this annotation type or at least is ignored at annotation import and export.

I suspect there is something fishy going on where multiple skeleton shapes are being created. Essentially, any skeleton shape with no points is invalid and should not exist in the DB.

This PR, in turn, should exclude any tracks with no visible shapes on import, so it can partially solve this problem, but it's mainly focused on a different issue.

@Drise13, you wrote about some code snippet in https://github.com/cvat-ai/cvat/pull/7669#issuecomment-2060291528. Could you provide information about how you could obtain it?

@Drise13, you wrote about some code snippet in #7669 (comment). Could you provide information about how you could obtain it? Specifically, the points field is expected to be empty in skeleton annotations.

By snippet, I meant "subset of the json data file". If the skeleton points are supposed to be empty, why are there both ones with and without points? Additionally, when running an export to CVAT, it tries to interpolate the tracks of the skeleton which has no points and throws a broadcast error. My intuition tells me that skeleton tracks should not try to be interpolated and only use the points. That seems to work correctly when I cleaned out the skeletons with no points. It may be that it fixed it because there were no other keyframes to try to interpolate from, but I'm not certain on that.

@Drise13,

By snippet, I meant "subset of the json data file".

What is this file?

Additionally, when running an export to CVAT, it tries to interpolate the tracks of the skeleton which has no points and throws a broadcast error. My intuition tells me that skeleton tracks should not try to be interpolated and only use the points.

Agree, it seems like the source of the problem, according to your findings.

As for the purpose of this PR, if this is not the right PR for this investigation, should I open a new issue instead?

There is the issue which you mentioned before. If it describes the problem you faced, then it should be discussed there.

@zhiltsov-max Thanks for the suggestion. I have added the new test for this.

Hi @zhiltsov-max , I have added the example you provided into our test and added some checking on the number of outside  = True in each elements.

Hi @zhiltsov-max !
I'm facing the same issue: ValueError: operands could not be broadcast together with shapes (0,) (6,)
I found two annotations in backup where "points":[] for a skeleton track.
I removed them and tried to create from backup but I am facing this error - KeyError: "There is no item named 'task.json' in the archive"
I am 100% that in my zip file there is a task.json tough.
my zip has following structure:
dataset

annotations.json
task.json
data

all images

Would highly appreciate some help.
Regards

@NicolasPetermann134,
I don't see how it's related to this PR or issues focused.

my zip has following structure:

What is dataset, a directory? If it's a directory, it should not be present in the backup archive.

@zhiltsov-max
I commented on this case as I saw here https://github.com/cvat-ai/cvat/pull/7669#issuecomment-2060319103 the idea of deleting the instances with empty points...
dataset is the zipped file.
I have here shared my backups https://1drv.ms/f/s!AiG5XaoBkxRVm9YYwD1G7yHEUrE01g?e=tPhDNN
One is the original backup which is not modified - this one I can use to create a new task without any issues.
For the other one I only modified annotation.json - I deleted there the instances with empty points, I didn't touch the task.json but I get the error KeyError: "There is no item named 'task.json' in the archive"
What could be the issue?

@Eldies , please look at the issue.

hi @Eldies, any news on this?

ftr: I could solve the problem. I could export the dataset easily without errors when I tried it on the self hosted version.


Incorrect RSSI calculation in tests

The calculation of the RSSI in BeaconUtil.calculateRssi() will return RSSIs greater than the calibratedRssi` for distances > 1 and smaller values for distance < 1. This is the opposite of what should happen.

Fixed in #137


MGDOBR-499: Fix wrong assertj calls order

MGDOBR-499
PR owner checklist
Please make sure that your PR meets the following requirements:

[ ] Your code is properly formatted according to this configuration.
Run mvn clean verify -DskipTests so that imports are correctly set.
[ ] Your commit messages are clear and reference the JIRA issue e.g: "[MGDOBR-1] - $clear_explanation_of_what_you_did"
[ ] All new functionality is tested
[ ] Pull Request title is properly formatted: MGDOBR-XYZ Subject
[ ] Pull Request contains link to the JIRA issue
[ ] Pull Request contains link to any dependent or related Pull Request
[ ] Pull Request contains description of the issue
[ ] Pull Request does not include fixes for issues other than the main ticket

PR reviewer(s) checklist

[ ] Check if code and Github action workflows have been modified and if the modification is safe
[ ] If the modification is safe, add the safe to test label

How to trigger pipelines and use the bots:

Run the end to end pipeline
Annotate the pull request with the label: safe to test. If you want to run the pipeline again, remove and add it again.

Rebase the pull request
Comment with: /rebase.

@jomarko This ITs fail.. https://github.com/5733d9e2be6485d52ffa08870cabdee0/sandbox/runs/5751160413?check_suite_focus=true This is why I'm trying to run them locally to figure out what the problem is... there are lots of PRs failing on IT atm.

@manstis I am checking the test locally, will let you know if I have some findings

/rebase

Oh, I removed and re-added the label as I was expecting more than 2 checks to be ran..
It seems like GHA only wants two!?

GHA isn't even running a build for the infra module containing the changed files... that is unexpected!

Unless "CI / Build all images and run E2E tests..." now includes it.. I'm sure we used to have ~7 checks.


LP Migration: Add option to skip items that already exist in 1Password

Hi again Dan,
Thanks for this PR! Looking forward to reviewing it when time permits.
Cheers,
Scott

Hi Dan,
I am going to close this PR in anticipation of the LastPass Migration script moving into "deprecated" status.  We are grateful for your interest and the work you put into this PR.
The LastPass migration scripts will be transitioning to a deprecated state shortly and will no longer receive updates or improvements. The functionality of these scripts is superseded by the LastPass migrator built into 1Password 8 for macOS, Windows, and Linux. This feature of the 1Password apps provides an easier, more complete, and more secure transfer of data.
The LastPass Migration scripts will remain in the repo for the foreseeable future as a proof-of-concept and an example of what the 1Password CLI is capable of.
Cheers,
Scott


Allow POST request on Resource with empty payload.

This commit handles the case where there is POST request on Resource with
empty payload , there is no check for content-type and it will allow to
execute the resource without any payload.
Also, fixed is the case where right content type is set for OPAQUE resource when sending notifications.

rebuild


aalc: Version commit for aalc-2024.26.01

Summary:
-modified:   meta-facebook/aalc-rpu/src/platform/plat_modbus.c
-modified:   meta-facebook/aalc-rpu/src/platform/plat_version.h
Dependency:
-aalc: add pwm get/set #1831
-aalc: Fix sensor polling problems #1835
-aalc: Fix sensor polling issue & Add modbus functions #1841
Description:
-Version commit for aalc-2024.26.01
-Add modbus function of getting version
Test Plan:
-Build code: PASS
-Modbus get versin: Pass

Hi @amithash, We rebase on the newest main branch. Please help us to merge in main branch, thanks!


Disappearing "Purchase" Button When Buying Multiple Talents

I'm finding a (very) small bug while purchasing Talents that causes the "Purchase" button to appear and disappear when right clicking.
To recreate: Open the player's character sheet (as GM or player, results are the same), click the "Specializations" tab to open the talents. Right click on a talent to purchase. If you then right click on another talent to purchase a second one, "Purchase" appears and disappears really quickly. If you close the talents and reopen them, it fixes itself temporarily and you can purchase one more talent before the bug happens again.

Done, thanks for the report. Will be released in 1.810.


ENH: differentiate: add array API support to jacobian and hessian

Adds array API support to the rest of the differentiate module.

Looks close after changes. Please also consider j-bowhay/scipy#12 so we can run the tests I had previously skipped for torch. CI will give a bit more to fix, but hopefully nothing too hard!

Ah also might be good to add a simple dtype test since I made most of the tests float64.

Thanks @j-bowhay. There were a few more of the elementwise iterative method functions that could use the upgrade - optimize.elementwise.bracket_root, optimize.elementwise.bracket_minimum, and integrate.nsum. I think I'll start on nsum today, but would you be interested in writing or reviewing any of these? Do you have a computer again or did you do all this on a phone?!

Thanks @j-bowhay. There were a few more of the elementwise iterative method functions that could use the upgrade - optimize.elementwise.bracket_root, optimize.elementwise.bracket_minimum, and integrate.nsum. I think I'll start on nsum today, but would you be interested in writing or reviewing any of these? Do you have a computer again or did you do all this on a phone?!

Happy to have a go at the bracketing functions! Yep, I did this and recent prs on codespaces on my mobile

Wow. When I feel the itch and don't have access to a computer, I'll code a bit on my phone, too (and it takes 10x as long). But this is another level. Uh... enjoy!


bootc install to-disk: policy issue

bootc version: 0.1.14-1.el9_4.x86_64
Container policy:
{
    "default": [
        {
            "type": "reject"
        }
    ],
    "transports": {
        "docker": {
            "ghcr.io/karelvanhecke/bootc": [
                {
                    "type": "sigstoreSigned",
                    "keyPath": "/etc/pki/containers/ghcr.io-karelvanhecke-bootc.pub",
                    "signedIdentity": {
                        "type": "matchRepository"
                    }
                }
            ],
            "registry.access.redhat.com": [
                {
                    "type": "signedBy",
                    "keyType": "GPGKeys",
                    "keyPaths": [
                        "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release",
                        "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta"
                    ]
                }
            ],
            "registry.redhat.io": [
                {
                    "type": "signedBy",
                    "keyType": "GPGKeys",
                    "keyPaths": [
                        "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release",
                        "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta"
                    ]
                }
            ],
            "quay.io/toolbx-images": [
                {
                    "type": "sigstoreSigned",
                    "keyPath": "/etc/pki/containers/quay.io-toolbx-images.pub",
                    "signedIdentity": {
                        "type": "matchRepository"
                    }
                }
            ],
            "": [
                {
                    "type": "insecureAcceptAnything"
                }
            ]
        },
        "docker-daemon": {
            "": [
                {
                    "type": "insecureAcceptAnything"
                }
            ]
        }
    }
}

Install commands:
$ truncate -s5G disk.img
$ sudo podman run --rm --pid host --privileged --security-opt label=type:unconfined_t -v ./:/workdir -v /var/lib/containers/storage:/var/lib/containers/storage -v /dev:/dev localhost/almalinux-pr-77:c747ad1c7ef6753aa1a8afb7ec3267c77396b4c3 bootc install to-disk --generic-image --skip-fetch-check --via-loopback /workdir/disk.img
Installing image: docker://localhost/almalinux-pr-77:c747ad1c7ef6753aa1a8afb7ec3267c77396b4c3
Digest: sha256:1001b7ba93728ec05624215da4d3b3fb8be1296347e8264ab9015f313cec0a51
Block setup: direct
       Size:<PHONE_NUMBER>
     Serial: <unknown>
      Model: <unknown>
Initializing partitions
Reread partition table
Creating root filesystem (xfs) on device /dev/loop0p3 (size=4.8 GB)
> mkfs.xfs -m uuid=7484caf8-324f-4727-9b4b-75bca4993fa9 -L root /dev/loop0p3
Mounting /run/bootc/mounts/rootfs
Creating ESP filesystem
> mkfs.fat /dev/loop0p2 -n EFI-SYSTEM
Mounting /run/bootc/mounts/rootfs/boot/efi
Initializing ostree layout
Initializing sysroot
ostree/deploy/default initialized as OSTree stateroot
ERROR Installing to disk: Creating ostree deployment: Pulling: Creating importer: Failed to invoke skopeo proxy method OpenImage: remote error: Running image containers-storage:[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]localhost/almalinux-pr-77:c747ad1c7ef6753aa1a8afb7ec3267c77396b4c3@sha256:1001b7ba93728ec05624215da4d3b3fb8be1296347e8264ab9015f313cec0a51@1fac3ee8fc43eeabe289f4aede98fa2051250eb1a6b08ab1efe801e36531ca3f is rejected by policy.

The error above did not occur with bootc version 0.1.9-3.el9_4.x86_64.
There is a default allow any policy within the docker transport block, so this shouldn't happen.

Can you explain a bit more about your goals here? You're trying to allow-list only some containers with security, but presumably the
            "": [
                {
                    "type": "insecureAcceptAnything"
                }
            ]

is a placeholder and you want to delete it?
What you actually likely need is to also add:
        "containers-storage: {
            "": [
                {
                    "type": "insecureAcceptAnything"
                }
            ]
        }

Because technically the way bootc install works is to use the containers-storage transport to load the files that already exist on the host - we don't re-fetch from the registry.

We could also internally just bypass the policy here for the install portion...alternatively, maybe what we really want is to teach c/image a notion of something like "pull this image from c/storage, but verify it as if it was coming from the registry".

What you actually likely need is to also add:
        "containers-storage: {
            "": [
                {
                    "type": "insecureAcceptAnything"
                }
            ]
        }

Because technically the way bootc install works is to use the containers-storage transport to load the files that already exist on the host - we don't re-fetch from the registry.

This solves my issue, previous versions didn't seem to properly respect the containers policy config.
I'm closing this issue, as bootc install behaves as expected now.

previous versions didn't seem to properly respect the containers policy config.

Side note: I think we did...it's just that we ended up forking off skopeo from the host during an install, so we'd end up using the host's copy of the policy, not the policy embedded in your container image. What we do now seems clearly more desirable in having the target image be the source of truth, and the host is just a bootstrap to run it as a container.


⚠ Visual-Crew_Builder has degraded performance

In 10a292b, Visual-Crew_Builder (https://visual-crew-builder.ai) experienced degraded performance:

HTTP code: 200
Response time: 1108 ms

Resolved: Visual-Crew_Builder performance has improved in 89dd819 after 9 minutes.


fix(ci): configurable timeouts

Let the default to be quick enough and provide a higher timeout in those gh action workflows that can be flaky for time reasons

Release Note
fix(ci): configurable timeouts

fyi @valdar - I think with this change we can have a quick local experience and we can fine tune the values on the specific workflow


PnP - Error trying to install and apply design package using Get-SPOContext context object

Hi,
When I use Get-SPOContext and pass that to DesignPackage - Install and Apply method -  I get error as shown below.

$clientContext = Get-SPOContext
$site = $clientContext.Site
Write-Host "Installing design package"
[Microsoft.SharePoint.Client.Publishing.DesignPackage]::Install($clientContext, $site, $package, $fileRelativeURL)
$clientContext.ExecuteQuery();
Write-Host "Applying design package"
[Microsoft.SharePoint.Client.Publishing.DesignPackage]::Apply($clientContext, $site, $package)
$clientContext.ExecuteQuery()
Error message -
Installing design package
Installation error  System.Management.Automation.MethodException: Cannot convert argument "context", with value: "Microsoft.SharePoint.Client.ClientContext", for "Install" to type
"Microsoft.SharePoint.Client.ClientRuntimeContext": "Cannot convert the "Microsoft.SharePoint.Client.ClientContext" value of type "Microsoft.SharePoint.Client.
ClientContext" to type "Microsoft.SharePoint.Client.ClientRuntimeContext"." ---> System.Management.Automation.PSInvalidCastException: Cannot convert the "Micros
oft.SharePoint.Client.ClientContext" value of type "Microsoft.SharePoint.Client.ClientContext" to type "Microsoft.SharePoint.Client.ClientRuntimeContext".
at System.Management.Automation.LanguagePrimitives.ThrowInvalidCastException(Object valueToConvert, Type resultType)
at System.Management.Automation.LanguagePrimitives.ConvertNoConversion(Object valueToConvert, Type resultType, Boolean recurse, PSObject originalValueToConve
rt, IFormatProvider formatProvider, TypeTable backupTable)
at CallSite.Target(Closure , CallSite , RuntimeType , Object , Object , Object , Object )
--- End of inner exception stack trace ---
at System.Management.Automation.ExceptionHandlingOps.CheckActionPreference(FunctionContext funcContext, Exception exception)
at System.Management.Automation.Interpreter.ActionCallInstruction`2.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)

The above code works if login in and getting context as follow.
#connect/authenticate to SharePoint Online and get ClientContext object..
$clientContext = New-Object Microsoft.SharePoint.Client.ClientContext($url)
$credentials = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($username, $securePassword)
$clientContext.Credentials = $credentials
Any suggestion as to how I can make this work using Get-SPOContext instead?
Thanks,

Sorry for the delay around this question. Hopefully you have been able to get this resolved and in general with questions, we would suggest to use the PnP Yammer group at http://aka.ms/OfficeDevPnPYammer. We have released significantly enhanced PS CmdLets after this issue has been reported, so would assume that it has been already addressed. If you still see issues around this, would report the issue at PowerShell repository at https://github.com/OfficeDev/PnP-powershell.


array.concatenate changes dtype endianness

I often have to work with binary data with non-native endianness. I found that when I concatenate dask with big-endian datatypes, the resulting array is native (little) endian.
import dask.array as da
import numpy as np

arrays = 2*[np.arange(4, dtype='>f4')]
darrays = [da.from_array(a, (4,)) for a in arrays]
concat = da.concatenate(darrays)
print(darrays[0].dtype, concat.dtype)
>>> (dtype('>f4'), dtype('float32'))

The numerical values of concat are correct, indicating it has been recast. But I don't necessarily want this to happen.
I know this is very obscure. Would you consider it a bug?

If this deviates from NumPy behavior then it doesn't seem particularly
obscure
On Tue, Oct 11, 2016 at 10:07 AM, Ryan Abernathey<EMAIL_ADDRESS>wrote:

I often have to work with binary data with non-native endianness. I found
that when I concatenate dask with big-endian datatypes, the resulting array
is native (little) endian.
import dask.array as daimport numpy as np
arrays = 2*[np.arange(4, dtype='>f4')]
darrays = [da.from_array(a, (4,)) for a in arrays]
concat = da.concatenate(darrays)print(darrays[0].dtype, concat.dtype)>>> (dtype('>f4'), dtype('float32'))
The numerical values of concat are correct, indicating it has been
recast. But I don't necessarily want this to happen.
I know this is very obscure. Would you consider it a bug?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/1647, or mute the thread
https://github.com/notifications/unsubscribe-auth/AASszHeK1XpAU63VWpvNnvZIbIdzTqZcks5qy5gZgaJpZM4KTq4U
.

In fact I just realized it does not deviate from numpy behavior. So probably best to close this and for me to find another solution.

I'd say so - dask.array tries to match numpy behavior wherever possible. Perhaps take a look at byteswap http://docs.scipy.org/doc/numpy/reference/generated/numpy.ndarray.byteswap.html? This could be easily added to dask.array if needed.


Add the &display=swap parameter to the end of your Google Fonts URL

Hi, Can you Add the &display=swap parameter to the end of your Google Fonts URL?
https://web.dev/font-display/?utm_source=lighthouse&utm_medium=unknown#google-fonts
This adds to the Google Lighthouse score

#403

In head, if the font is cached:
<script>sessionStorage.fontsLoaded&&document.documentElement.classList.add("wf-active");</script>
And load webfontloader async with font-display: swap:
    if (!sessionStorage.fontsLoaded) {         require(['webfontloader'], function(WebFont) {             WebFont.load({                 google: {                   families: [                       'Roboto:400,700&display=swap'                   ]                 },                 active: () => {                     sessionStorage.fontsLoaded = true                 },             });         });     }

You can now optimize the whole thing with Preconnect / dns-prefetch.
This takes the loading time of the connection establishment to the target out of the consideration. In a test case, the Google programmer Ilya Grigorik saved about 0.5 seconds of loading time by adding Preconnect when loading the Google fonts and wrote a very informative article on this topic.
<link rel=”dns-prefetch” href=”//fonts.googleapis.com/”>
<link rel=”preconnect” href=”https://fonts.gstatic.com” crossorigin>

Does anyone have more experience or tips here?

May I add that in order for display=swap to work, you need to add it only to the last font in the array.
you can achieve this by doing
const fonts = [ ... ] // what you'd normally pass to families: []
fonts.map((x, i) => x + variants + (i === fonts.length - 1 ? '&display=swap' : ''));

Adding on each item in the array passed to families will generate a request with multiple display=swap params.


install all packages at start?

too many errors, we could look into splitting up the total package if we want the streamlit app


Update Docker Instructions for Collector Getting Started to contain contrib and core in both cases

Preview: https://deploy-preview-1885--opentelemetry.netlify.app/

@SedemQuame have you had a chance to look at feedback to address?

@SedemQuame any updates?

Closing due to inactivity. If you'd like to make updates, feel free to ping here or resubmit and we'll take a look.


chore: allow Vite 5

https://github.com/sveltejs/kit/pull/10818 is very complicated and a breaking change. This would be a much simpler solution. We can bump the dependency version in a couple of months to move everyone to vite-plugin-svelte 3 when we release SvelteKit 2
We don't have to fix any bugs in v2. We can still fix them just in v3 and tell non-SvelteKit users to upgrade to v3 and tell SvelteKit users they'll get any fixes in a couple of months. If there are any major issues that arise that need to be backported I can send PRs for it, I'm not too worried about that as vite-plugin-svelte has been very stable and battle tested already.

not a fan of this. if you volunteer to maintain v2 and add vite5 to the test matrix it can work, but kit needs more changes to be compatible ( the manifest changes for example )

also vite5 uses newer majors for esbuild and rollup, adapters and svelte-package could be affected too

Sure, I'm happy to maintain v2
I'm trying to figure out I'd add Vite 5 to the test matrix. I think it'd have to be in a separate PR targeting the main branch? Because I see in main a reference to the v2 branch: https://github.com/sveltejs/vite-plugin-svelte/blob/63fafd4e5661425c088678daa1a16103daad6f22/.github/workflows/ci.yml#L8 So I'm assuming GitHub only runs the file located in main and the version in the v2 branch doesn't do anything. But perhaps we could wait until future changes are needed on the v2 branch? We already know that no changes were needed to vite-plugin-svelte for Vite 5 support when we updated main to Vite 5.
I'm not sure what needs to change in Kit regarding the manifest, but those changes would need to happen regardless. I'll bump esbuild and rollup in the adapters, etc. so that folks who are current only need a single version of them pulled into their project

no, i think it has to be in this branch (the merge target).  on main, there is a preparation to add svelte5 to the test matrix, you can use that as a template how to add vite 5 in this PR https://github.com/sveltejs/vite-plugin-svelte/blob/63fafd4e5661425c088678daa1a16103daad6f22/.github/workflows/ci.yml#L81

I'm wondering another trick: Could we have v-p-s both as a dep and an optional peer dep in SvelteKit? That way if the user wants to use Vite 5, they can install v-p-s v3 and SvelteKit can pick that up through the peer dep; If they stick with Vite 4, it'll fallback to SvelteKit's normal v-p-s v2 dep. I have not tested it though.
Personally, I'm fine with relaxing the peer dep range only and not needing to tweak the CI and tests. We can say "we allow vite 5 only temporarily for svelte kit, but it's not something we guarantee. use at your own risk."

we need the tests. if we do this, other setups can use v-p-s@2 + vite@5 as well, not just sveltekit.  I'd prefer a solution on the sveltekit side aswell. That being said, if the work to maintain this is considered less than the work on sveltekit side, happy to accomodate - as long as it is treated as actual support -.  We should also be clear that active support for v-p-s@2 is going to end soon after kit is able to use v-p-s@3, even if thats in a new major.

I get really nervous about having two versions of v-p-s in a kit project. How will users know they need to install v-p-s themselves? Will npm, pnpm, yarn, etc. all handle it the same way? Will it make it harder to tell what version a user is on for bug reports? It just seems like an extra complication that we can easily avoid
I'm also not convinced the CI matrix is truly necessary, but at this point it has been added and there's certainly no harm in it. It is marginally safer, so I feel we might as well go with it since it's done.
In terms of support, I'd like to support it more than "use at your own risk". I do think that non-SvelteKit users could be lightly encouraged to upgrade though. E.g. I think it would be fine to say we're only going to backport large bug fixes (e.g. anything we discover related to Vite 5 support) and users should upgrade if they want fixes for edge cases that are unique to their projects.


The icon of the new attributes is the same as the first attributes when tweaking settings from product page

The icon of the new attributes is the same as the first attributes when tweaking settings from product page
https://user-images.githubusercontent.com/1689815/119272548-b5272800-bc06-11eb-90d5-4f6f14f46c69.mp4

@teolemon Please use the latest version of the code, including #386.

@teolemon ping


crypto/aes: speedup CTR mode on AMD64 and ARM64

The implementation runs up to 8 AES instructions in different registers
one after another in ASM code. Because CPU has instruction pipelining
and the instructions do not depend on each other, they can run in
parallel with this layout of code. This results in significant speedup
compared to the regular implementation in which blocks are processed in
the same registers so AES instructions do not run in parallel.
GCM mode already utilizes the approach.
The type implementing ctrAble has most of its code in XORKeyStreamAt()
method which has an additional argument, offset. It allows to use it
in a stateless way and to jump to any location in the stream.
AES CTR benchmark delta.
$ go test crypto/cipher -bench 'BenchmarkAESCTR*'
AMD64. Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
name                   old time/op    new time/op   delta
BenchmarkAESCTR1K-2       1259ns        266.9ns     -78.8%
BenchmarkAESCTR8K-2       9859ns         1953ns     -80.1%
ARM64. ARM Neoverse-N1 (AWS EC2 t4g.small instance)
name                   old time/op    new time/op   delta
BenchmarkAESCTR1K-2       1098ns        481.1ns     -56.2%
BenchmarkAESCTR8K-2       8447ns         3452ns     -59.1%
ARM64. Apple M1
name                   old time/op    new time/op   delta
BenchmarkAESCTR1K-2      455.3ns        154.3ns     -66.1%
BenchmarkAESCTR8K-2       3491ns         1116ns     -68.0%
Fixes #20967
Investigation and initial implementation:
https://github.com/mmcloughlin/aesnix/
Full implementation in external repo:
https://github.com/starius/aesctrat

@andig I'm rewriting the generators in Go. Want to make a single Go program producing for all architectures.

@andig I finished with generators and added correctness test. Waiting for review in Gerrit :-)


Switch back to plain pip

FYI @gilesknap This is the diff between pip-tools (which has an outstanding dependabot bug https://github.com/dependabot/dependabot-core/issues/5140), and just pip which includes a custom shell script which may or may not trigger the same bug...
Thoughts?

The funny bit is that is not pip-tools that causes this bug, I happen to be one of the maintainers of pip-tools. It is dependabot internal logic which produces bad contraints.txt files.


Can the player be installed on CentOS 6.7?

I tried to install the rpm on CentOS 6.7. I have 2.6.32-573.22.1.el6.x86_64 and it said "wrong architecture" when I tried to install the 64-bit Fedora RPM. So I installed the 32-bit Fedora rpm successfully. But after installing 32-bit dependencies one at a time based on "missing library" messages when trying to run the player, I reached the error
google-play-music-desktop-player: /usr/lib/libstdc++.so.6: version `GLIBCXX_3.4.15' not found (required by /usr/share/google-play-music-desktop-player/libnode.so)
There doesn't seem to be any way to satisfy that, as I have the latest versions of libstdc++ and glibc installed. I can try building it from source, but I assume I'm going to run into the same issue.

It looks like its a known issue among electron based apps. See https://github.com/Microsoft/vscode/issues/1056 and https://github.com/Microsoft/vscode/issues/3603
I would try compiling from source and see if that fixes your library errors.

Thanks. I think it's the same issue that prevented me from installing the atom editor last week. I gave that a good effort and it proved impossible with CentOS 6, so I assume the same will be true here. I'll have to wait until I'm able to move to CentOS 7.

Yeah it's something to do with the c++ version not being latest. You might have luck with source, but only if you can get node/npm running initially.

I have installed node v4.4.2 and npm v3.86. npm install completes successfully, but running npm start gives the error below. I have atom-1.6.1-0.1.fc21.x86_64 installed, and it provides libnotify at
/usr/share/atom/libnotify.so.4
/opt/git/Google-Play-Music-Desktop-Player-UNOFFICIAL-/node_modules/electron-prebuilt/dist/electron: error while loading shared libraries: libnotify.so.4: cannot open shared object file: No such file or directory

npm ERR! Linux 2.6.32-573.22.1.el6.x86_64
npm ERR! argv "/opt/node-v4.4.2-linux-x64/bin/node" "/usr/bin/npm" "start"
npm ERR! node v4.4.2
npm ERR! npm  v3.8.6
npm ERR! file sh
npm ERR! code ELIFECYCLE
npm ERR! errno ENOENT
npm ERR! syscall spawn
npm ERR<EMAIL_ADDRESS>start: `electron . --dev`
npm ERR! spawn ENOENT
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>start script 'electron . --dev'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the google-play-music-desktop-player package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     electron . --dev
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs google-play-music-desktop-player
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls google-play-music-desktop-player
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /opt/git/Google-Play-Music-Desktop-Player-UNOFFICIAL-/npm-debug.log```

Almost certainly due to the electron prebuilts not working on CENTOS, there is an open issue on the electron repository for fixing certain linux builds.  Unfortunately not much I can do :cry:

Do you have an entry for that directory inside your ld.so.conf configuration? /etc/ld.so.conf.d/ should have something like atom.conf since atom is providing that library.
I would check by grepping the directory for /usr/share/atom/ via
grep -r '/usr/share/atom' /etc/ld.so.conf.d/

That should tell you if you have that library registered with ld, which assuming by that error, it is not.

Thanks for the suggestion, but unfotuntaely it didn't solve it. I didn't have any file with a /usr/share/atom entry. so I added one, and re-ran npm run build, and npm start still fails with the same message.

You'd have to run ldconfig -v as root in order for the cache to be updated. Also assuming you created a file called (for eg.) atom.conf with contents /usr/share/atom.

Yes I created a file in /etc/ld.so.conf.d/ with a single entry usr/share/atom and ran ldconfig -v.

Yeah then the underlying issue is still probably the electron prebuilt binaries. :-1: Sorry, wish I had a solution.


sra_ids_to_runinfo.py UnicodeEncodeError

Dear nf-core team, first of all, many thanks for your amazing work that make our analyses more easy and straightforward !
I am using this nf-core/rnaseq pipeline (release 2.0) with the experimental feature --public_data_ids to retrieve SRA datasets and I face some issues with some SRA projects for which some characters are non-ascii.
Here is an example: looking for SRP290966, you can find the degree character "°" in the experiment_title field encoded in unicode: [ENA API RESULT] (https://www.ebi.ac.uk/ena/portal/api/filereport?accession=SRP290966&result=read_run&fields=experiment_title) )
The workflow ends with this error:
Traceback (most recent call last):
  File "/home/carrere/.nextflow/assets/nf-core/rnaseq/bin/sra_ids_to_runinfo.py", line 178, in <module>
    sys.exit(main())
  File "/home/carrere/.nextflow/assets/nf-core/rnaseq/bin/sra_ids_to_runinfo.py", line 174, in main
    fetch_sra_runinfo(args.FILE_IN,args.FILE_OUT,platform_list,library_layout_list)
  File "/home/carrere/.nextflow/assets/nf-core/rnaseq/bin/sra_ids_to_runinfo.py", line 131, in fetch_sra_runinfo
    for row in csv_dict:
  File "/opt/conda/lib/python2.7/csv.py", line 108, in next
    row = self.reader.next()
UnicodeEncodeError: 'ascii' codec can't encode character u'\xb0' in position 260: ordinal not in range(128)

Thanks for your help,
Sébastien

Thanks for reporting this @carrere 👍 I can indeed reproduce this locally by manually running the sra_ids_to_runinfo.py script with an id file containing just SRR12971731. @JoseEspinosa would be great if you can take a look at this please? 🙂

You're welcome. I think the main problem come from the EBI API that not declare the document encoding:
14:52 $ curl -I "https://www.ebi.ac.uk/ena/portal/api/filereport?accession=SRP290966&result=read_run&fields=experiment_title"
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Content-Type: text/plain
Strict-Transport-Security: max-age=0
Date: Fri, 04 Dec 2020 13:52:07 GMT
Expires: 0
X-XSS-Protection: 1; mode=block
Pragma: no-cache
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
Content-Length: 6226

Or in the firefox console:
"The character encoding of the plain text document was not declared. The document will render with garbled text in some browser configurations if the document contains characters from outside the US-ASCII range. The character encoding of the file needs to be declared in the transfer protocol or file needs to use a byte order mark as an encoding signature. filereport"
I do not know if you can fix this automatically on the client side ... but if you know someone @ EBI, you could ask them to fix on the API side.
Sebastien

I been trying to solve the issue. The script works as it is when run with Python 3 but not with Python 2. The reason is that in Python 3 UTF-8 is the default source encoding but not in Python 2. Although I was trying to find a solution that worked both with Python 2 and Python 3 I didn't find it. That is why I just checked the python version and include this code. I don't know if you think this solution suitable if yes I can just make a PR with this patch

Thanks! But we should be using Python 3 for that process?

Currently no, but I thought it could be a good idea to keep it compatible in case future updates of the images use Python 3. I changed it now https://github.com/JoseEspinosa/rnaseq/commit/c55986d7bf61a8db52bc9068066ee9a84e45d340
Should we add one of such IDs to the test?

I think we are using Python 2.7.13
Should we change the image instead?

Indeed we are!! Yes, that would be great we just need to replace those lines with the snippet below and that would be the best fix. Sorry, I was looking at the wrong process🤦🏽
conda (params.enable_conda ? "conda-forge::python=3.8.3" : null)
    if (workflow.containerEngine == 'singularity' && !params.singularity_pull_docker_container) {
        container "https://depot.galaxyproject.org/singularity/python:3.8.3"
    } else {
        container "quay.io/biocontainers/python:3.8.3"
    }

Thanks @JoseEspinosa. This will be fixed in the next release via https://github.com/nf-core/rnaseq/pull/535/commits/f33eb6dc43602c34f443488eb07507bd9e6e1c34

Perfect @drpatelh ! was about to implement it now and saw that you closed the issue 😎

No worries! I had to use another container in the end that specifically contained requests. Thinking about it, I don't know if that is Python 3 or not 🤦🏽 It was late. Will check in the morning.

It's Python >3 in the requests container 💥 Also tested with SRR12971731 and it's working.
$ singularity shell depot.galaxyproject.org-singularity-requests-2.24.0.img

Singularity depot.galaxyproject.org-singularity-requests-2.24.0.img:> python
Python 3.8.3 | packaged by conda-forge | (default, Jun  1 2020, 17:43:00)
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

:+1:
Thank you !


Issue in test with d() function

In tests there is the mocked function d().
This causes issues with Patchwork... for reasons I can't sill understand.
I tried changing the name of the function and everything works.

With 32f56a8 the function name has been renamed in test source, and now unit tests pass, however, keeping this open because reason behind this issue has not be found yet.

This is something related to Patchwork and not to BrainMonkey code, so I'll just close it.


Ability to turn off details buttons

If a users wants to just create a visual hint graph without needing the actual value to be displayed on tapping on the points. A @property has to be created which will essentially act as a switch to turn on/off the functionality.

Not planning to work on this code unless I rewrite it, sorry. I will add to the READMe that is not production ready as it was done during an hackathon


add apollo link operation id support

Stack:

#6
#5 ⬅

⚠ Part of a stack created by spr. Do not merge manually using the UI - doing so may have unexpected results.

✓ Commit merged in pull request #6


Bump Node version in Travis config

We updated Node version to 14.16 as recommended by protecode scan in #1276
Updating the Node version in Travis config.

Pull Request Test Coverage Report for Build 5233

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 89.16%

Totals

Change from base Build 5230:
0.0%

Covered Lines:
9447

Relevant Lines:
10224

💛  - Coveralls


[BEAM-7018] Added Regex transform for PythonSDK

Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Thanks. These look generally useful. I wonder, however, if they should be in their own module rather than in a Regex class in the util module (which is generally a java-ism because everything must be in a class). Alternatively one could have a Regex module with methods like matches that returns the relevant PTransforms.
The other thought that came to me is that this could probably be simplified a lot using lambdas, instead of creating a new PTransform and DoFn class each time. E.g.
class Regex(object):
  def matches(regex, group=0):
    regex = _regex_compile(regex)  # Do this once at construction time
    def maybe_match(element):
      m = regex.match(element)
      if m:
        yield m.group(group)
    return beam.FlatMap(maybe_match)

If this is a common pattern, we could have a utility function
  def match_objects(regex):
    regex = _regex_compile(regex)
    def maybe_match(element):
      m = regex.match(element)
      if m:
        yield m
    return beam.FlatMap(maybe_match)

  def matches(regex, group):
    return match_object(regex) | beam.Map(lambda m: m.group(group))

Hi @robertwb. Thanks for review PR and for your suggestions. I believe making the separate the regex module is not what something I've seen in the py sdk. Because other modules (ex: Reify) is also available as a utility in py sdk. Moreover, regex transform is not something I think can be called as a module, so my thoughts util.py is a good place for these kinds of transforms
And for the other part, I think you are right, using lambda will definitely improve the code in terms of readability.
let me make some changes.

Run Python PreCommit

retest this please

Run Python PreCommit

retest this please

Run Python PreCommit

Run Portable_Python PreCommit

@robertwb I've made the changes you mentioned. Can you please review this PR.
Thanks.

Run Python PreCommit

R: @aaltay
R: @robertwb

@robertwb do you want to finish reviewing this?

@aaltay I think @robertwb is unavailable. Can you assign someone else to review this PR?
Thanks!

R: @aaltay
R: @robertwb

@mszb Both Robert and I were out of office for a bit. The change looks good to me. I will give @robertwb another day if he has any comments. After that we can merge it.

Run Python PreCommit

Run Python PreCommit

Run Python PreCommit

Run Python PreCommit

Run Python PreCommit

@robertwb do you have additional comments?

Run Python_PVR_Flink PreCommit

On Wed, Jul 24, 2019 at 6:30 PM Shoaib Zafar<EMAIL_ADDRESS>wrote:

@mszb commented on this pull request.
In sdks/python/apache_beam/transforms/util.py
https://github.com/apache/beam/pull/8859#discussion_r306906519:

 group: (optional) name of the group, it can be integer or a string value.

"""
regex = Regex._regex_compile(regex)
group = group or 0

def _process(element):

 r = regex.search(element)

 if r:

   yield r.group(group)

return pcoll | FlatMap(_process)

@staticmethod
@typehints.with_input_types(str)
@typehints.with_output_types(str)
@ptransform_fn
def find_all(pcoll, regex):

So what I understand from your feedback is:

for the 'Regex.find_all()', we need to change re.finditer to
re.findall and return the result as a list.
create a new method Regex.finditer() which returns "match objects"
instances..

The issue with the 'MatchObject' is we can not pickle it.
Showing error PicklingError: Can't pickle <built-in method match of
_sre.SRE_Pattern

Oh, that is really unfortunate :(.

So either we can make a custom match object class and return it

This seems nicest from an API perspective, but I'm worried about the
maintenance in faithfully emulating the match object over time.

or uses KV pairs? Where K as the string of group and V as a tuple
containing all groups?
For example:
lets say sting is "abb ax abbb" & regex expression is "a(b*)", so the
results would be like:
[ KV("abb", ('bb',)), KV("a", ('',)), KV("abbb", ('bbb',) ]
or maybe we can include group 0 in the values as well.
[ KV("abb", ('abb', 'bb',)), KV("a", ('ab', '')), KV("abbb", ('abbb',
'bbb') ]

In Python there's no such thing as a KV, it's just a 2-tuple.
Here's what I would do: many other transforms have an (optional) group
parameter, defaulting to 0. I would do that for find_all as well, so the
result would be simply
["abb", "a", "abbb"]

You could also have a special value (say, a constant like Regex.ALL) that
would return all the groups, i.e.
[('a', 'abb'), ('a', ''), ('abbb', 'bbb')]

Thanks for the feedback @robertwb. This approach seems good, I'll update the code!
Only one questions though, What I think, we create Regex.find_all which just Map re.findall and put all approach you mentioned above in the Regex.find_iter method!
Because re.findall returns a list of all the groups (re.findall("a(b*)", "abb ax abbb") >> ['bb', '', 'bbb']) whereas in the above example we are going to return a list of group(0) ["abb", "a", "abbb"]. Your thoughts?

Someone using Regex.find_iter might expect match objects, just as in
Python, so I'd avoid that naming.
You might need to use re.finditer to implement Regex.find_all though, due
to Regex.findall's variant signature.
On Mon, Jul 29, 2019 at 3:40 PM Shoaib Zafar<EMAIL_ADDRESS>wrote:

Thanks for the feedback @robertwb https://github.com/robertwb. This
approach seems good, I'll update the code!
Only one questions though, What I think, we create Regex.find_all which
just Map re.findall and put all approach you mentioned above in the
Regex.find_iter method!
Because re.findall returns a list of all the groups (re.findall("a(b*)",
"abb ax abbb") >> ['bb', '', 'bbb']) whereas in the above example we are
going to return a list of group(0) ["abb", "a", "abbb"]. Your thoughts?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/beam/pull/8859?email_source=notifications&email_token=AADWVAIM36YNCUXHB2SCR4TQB3XL7A5CNFSM4HYGZUYKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3AXP5A#issuecomment-515995636,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADWVAIEP3UFNJ7DEHW7Q4LQB3XL7ANCNFSM4HYGZUYA
.

okay.

Someone using Regex.find_iter might expect match objects, just as in Python, so I'd avoid that naming. You might need to use re.finditer to implement Regex.find_all though, due to Regex.findall's variant signature.
…
On Mon, Jul 29, 2019 at 3:40 PM Shoaib Zafar @.**> wrote: Thanks for the feedback @robertwb https://github.com/robertwb. This approach seems good, I'll update the code! Only one questions though, What I think, we create Regex.find_all which just Map re.findall and put all approach you mentioned above in the Regex.find_iter method! Because re.findall returns a list of all the groups (re.findall("a(b)", "abb ax abbb") >> ['bb', '', 'bbb']) whereas in the above example we are going to return a list of group(0) ["abb", "a", "abbb"]. Your thoughts? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8859?email_source=notifications&email_token=AADWVAIM36YNCUXHB2SCR4TQB3XL7A5CNFSM4HYGZUYKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3AXP5A#issuecomment-515995636>, or mute the thread https://github.com/notifications/unsubscribe-auth/AADWVAIEP3UFNJ7DEHW7Q4LQB3XL7ANCNFSM4HYGZUYA .

Run Python PreCommit

Hi @robertwb. I've made the changes you mentioned. Hope you find it well.

Hi @aaltay. Robert has approved the PR. Do you have anything to add?

Nothing to add. Merging.


support catalog dialect in starrocks ，like trino ptf

Feature request
Is your feature request related to a problem? Please describe.

Describe the solution you'd like
there is the same solution in trino , link https://github.com/trinodb/trino/issues/1839

Describe alternatives you've considered

Additional context

@allendang001 Could you provide us some of your typical usage as examples and why do you need this feature

If I know the data distribution and index optimization of the data source, I can directly write more optimized SQL without starrocks to process or push down some operators, thereby improving the execution efficiency of tasks


Few base images updates

This PR contains the following changes:
- Add node v17.2.0 v16.13.1 and v14.18.2
- Add Golang v1.17.2 and v1.16.11
- Add support for Alpine Linux 3.15
- Add support for Fedora 36
- Update partials for Ubuntu RPI base images
- Add OpenJDK jdk17.0.1

@balena-ci I self-certify!


linux 4.18: gem5 didn't print to terminal and silently crashes

Running Environment (please complete the following information):

Host OS and Kernel version: Debian 10, Linux 4.19
Compiler version used to build SimpleSSD-FullSystem: GCC 8.3
SimpleSSD-FullSystem version: v2.0.9
SimpleSSD version: v2.0.13
Guest Kernel version: 4.18 aarch64
Compiler version used to build Guest Kernel binary: gcc-linaro-7.5.0-2019.12

Question
linux 4.9 from simplessd
linux 4.9 kernel and the linaro image provided by simplessd
In nvme mode, ./nvme list shows the emulated nvme device;
however, in ocssd mode, ./nvme list shows none.
export M5_PATH=upstream-images/
./build/ARM/gem5.opt --debug-flag=M5Print --debug-file=debug.txt \
    ./configs/example/fs.py \
    --kernel=upstream-images/binaries/aarch64-vmlinux-4.9.92 \
    --dtb-file=armv8_gem5_v1_4cpu.dtb --machine-type=VExpress_GEM5_V1 \
    --disk-image=upstream-images/disks/linaro-aarch64-linux.img \
    --num-cpu=4 --cpu-clock=2GHz --caches --l2cache --cpu-type=AtomicSimpleCPU \
    --mem-size=2GB --mem-type=DDR4_2400_8x8 \
    --command-line "earlyprintk=pl011,0x1c090000 console=ttyAMA0 lpj=19988480 norandmaps ro \
    loglevel=8 mem=2GB root=/dev/sda1 kpti=0 nospectre_v2 ssbd=force-off init=/bin/sh" \
    --ssd-interface=ocssd --ssd-config=./src/dev/storage/simplessd/config/sample.cfg

linux 4.18 I built
I've tried compiling 4.18 kernel using the kernel config from simplessd and runs as follow:
export M5_PATH=upstream-images/
./build/ARM/gem5.opt --debug-flag=M5Print --debug-file=debug.txt \
    ./configs/example/fs.py \
    --kernel=/home/yunchih/research/simplessd/linux-4.18.12-arm/vmlinux \
    --dtb-file=armv8_gem5_v1_4cpu.dtb --machine-type=VExpress_GEM5_V1 \
    --disk-image=upstream-images/disks/linaro-aarch64-linux.img \
    --num-cpu=4 --cpu-clock=2GHz --caches --l2cache --cpu-type=AtomicSimpleCPU \
    --mem-size=2GB --mem-type=DDR4_2400_8x8 \
    --command-line "earlyprintk=pl011,0x1c090000 console=ttyAMA0 lpj=19988480 norandmaps rw \
    loglevel=8 mem=2GB root=/dev/sda1 kpti=0 nospectre_v2 ssbd=force-off init=/bin/sh" \
    --ssd-interface=ocssd --ssd-config=./src/dev/storage/simplessd/config/sample.cfg

GEM5 full output
Nothing is shown in m5term, and the kernel crashes.  The system.dmesg does not run far enough to inspect the crash reason.
Because I plan to use OCSSD 2.0, I need a working kernel at least > 4.18.  Maybe @kukdh1 could provide a more recent working kernel, as you've mentioned in #3
Thanks for your hardwork!

Hi, yunchih.
For the m5term, you should double check that kernel configuration file config_arm is applied properly.
# Assume your working directory is root of Linux kernel source.
# Download config_arm file here.
# Step 1. Create default ARM64 Linux kernel configuration
ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make defconfig
# Step 2. Overwrite generated .config file with config_arm
mv config_arm .config
# Step 3. Modify overwritten config file -- enable OpenChannel SSD or other features.
ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make menuconfig
# Step 4. Compile vmliunx
ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make vmlinux -j $(nproc)

To use OpenChannel SSD v2.0, you need to use --ssd-interface=ocssd2 instead of ocssd.
Please note: When I implement OCSSD protocol (2017 ~ 2018), the specification was quite ambiguous. I found the spec is updated little bit but such changes are not applied to SimpleSSD.
P.S. Please close this issue if solved. If not, I can provide my kernel configuration file for 4.19.114 - it worked very well.
# Linux kernel bootlog
[    0.208456] nvme 0000:00:02.0: assign IRQ: got 13
[    0.208512] nvme nvme0: pci function 0000:00:02.0
[    0.208521] nvme 0000:00:02.0: enabling device (0000 -> 0002)
[    0.208528] nvme 0000:00:02.0: enabling bus mastering
[    0.208608] ata_piix 0000:00:01.0: assign IRQ: got 15
[    0.208614] ata_piix 0000:00:01.0: version 2.13
[    0.208621] ata_piix 0000:00:01.0: enabling device (0000 -> 0001)
[    0.208643] ata_piix 0000:00:01.0: enabling bus mastering
[    0.209533] nvm: registered nvme0n1 [8/8/512/64/8]

# SimpleSSD output
897733495500: HIL::NVMe: SQ 3    | Submission Queue Tail Doorbell | Item count in queue 0 -> 1 | head 34 | tail 34 -> 35
897739449272: HIL::NVMe: OCSSD   | READ  | 7FFFFC0 + 14
897743791772: HIL::NVMe: OCSSD   | READ  | 7FFFFC0 + 14 | NAND<PHONE_NUMBER>72 -<PHONE_NUMBER>72 (4342500)
897755934500: HIL::NVMe: SQ 3    | Submission Queue Tail Doorbell | Item count in queue 0 -> 1 | head 35 | tail 35 -> 36
897761553475: HIL::NVMe: OCSSD   | READ  | 7FFFFCF + 16
897765895975: HIL::NVMe: OCSSD   | READ  | 7FFFFCF + 16 | NAND<PHONE_NUMBER>75 -<PHONE_NUMBER>75 (4342500)
897769538052: HIL::NVMe: OCSSD   | READ  | 7FFFFC0 + 14 | DMA<PHONE_NUMBER>72 -<PHONE_NUMBER>52 (25746280)
897777494500: HIL::NVMe: SQ 3    | Submission Queue Tail Doorbell | Item count in queue 0 -> 1 | head 36 | tail 36 -> 37
897783815178: HIL::NVMe: OCSSD   | READ  | 7FFFFE1 + 15
897788157678: HIL::NVMe: OCSSD   | READ  | 7FFFFE1 + 15 | NAND<PHONE_NUMBER>78 -<PHONE_NUMBER>78 (4342500)

root@genericarmv8:~# ./nvme list
Node             SN                   Model                                    Namespace Usage                      Format           FW Rev
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1<PHONE_NUMBER>0000000000 SimpleSSD OCSSD Controller by CAMELab    1         549.76  GB / 549.76  GB      4 KiB +  0 B   02.01.03
root@genericarmv8:~# ./nvme lnvm list
Number of devices: 1
Device          Block manager   Version
nvme0n1         gennvm          (1,0,0)
root@genericarmv8:~# ./nvme lnvm create -d nvme0n1 -n lnvm0 -t pblk -b 0 -e 7
[    1.401681] pblk lnvm0: luns:8, lines:512, secs:16777216, buf entries:128
root@genericarmv8:~# cat /proc/partitions
major minor  #blocks  name

 259        0  536870912 nvme0n1
   8        0    1048576 sda
   8        1    1047552 sda1
 259        1   59555840 lnvm0

P.S. 2. If you want to use most-recent kernel, I think you can use Linux 5.5 without any problem. I checked Linux 5.2 worked with SimpleSSD.
P.S. 3. pblk maintainers highly work on their kernel drivers and some version may have bug.
Thanks.

@kukdh1 Can't thank you enough.  I successfully run linux 5.4 following your detailed instructions.  For future reference, the disk image can be obtained by the following means:

Download from linaro
Build from Buildroot's aarch64 little endian architecture
Build from debootstrap:

sudo debootstrap --arch=arm64 \
	--variant=minbase --foreign \
	testing /mnt \
	http://ftp.debian.org/debian/

Just use the one provided by simplessd

I tested them all, they all worked.


=bstats monsters & lb kc rabbit

🐛 Bug Report
=bstats monsters gives error now but works on osb.

lb for kc rabbit both bots doesn't load at all

These issues have been resolved


fix: delete rule should check other policyRuleReference

require to cherry-pick to the main branch

http://jira.smartx.com/browse/ER-740


[e2e]: container-based push main default slsa3

Repo: https://github.com/slsa-framework/example-package/tree/main
Run: https://github.com/slsa-framework/example-package/actions/runs/6608631728
Workflow file: https://github.com/slsa-framework/example-package/tree/main/.github/workflows/e2e.container-based.push.main.default.slsa3.yml
Workflow runs: https://github.com/slsa-framework/example-package/actions/workflows/e2e.container-based.push.main.default.slsa3.yml
Trigger: push
Branch: main
Date: Mon Oct 23 04:16:38 UTC 2023

Repo: https://github.com/slsa-framework/example-package/tree/main
Run: https://github.com/slsa-framework/example-package/actions/runs/6622257865
Workflow file: https://github.com/slsa-framework/example-package/tree/main/.github/workflows/e2e.container-based.push.main.default.slsa3.yml
Workflow runs: https://github.com/slsa-framework/example-package/actions/workflows/e2e.container-based.push.main.default.slsa3.yml
Trigger: push
Branch: main
Date: Tue Oct 24 04:16:18 UTC 2023

Repo: https://github.com/slsa-framework/example-package/tree/main
Run: https://github.com/slsa-framework/example-package/actions/runs/6635665546
Workflow file: https://github.com/slsa-framework/example-package/tree/main/.github/workflows/e2e.container-based.push.main.default.slsa3.yml
Workflow runs: https://github.com/slsa-framework/example-package/actions/workflows/e2e.container-based.push.main.default.slsa3.yml
Trigger: push
Branch: main
Date: Wed Oct 25 04:28:18 UTC 2023
Tests are passing now. Closing this issue.


[1~3주차] layer_normalization & adam_optimizer & lstm & dcgan / 박지은

What does this PR do?
Added Korean files for week 1 ~ week 3 in the docs/ko/

Change docs folder file which written in English to Korean.
Translate normalization / layer_norm / index.html for week 1
Translate optimizers / adam_warmup_cosine_decay.html for week 1.
Translate lstm / index.html for week 2.
Translate gan / dcgan / index.html for week 3.

Before submitting

[x]  This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

[x] Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting html and css

[ ] Did you write any new necessary tests?

Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@Bandi120424 , @Jieun-Park99 , @kimdahyeon977 , @minyoungci , @mlleo , @movie5, @nehcream2od, @rimiyeyo @russellgeum @Youngmi-Park
Pseudo Lab, Please review this PR.

스타일 가이드라인에 맞춰서 1~73 라인의 title, discription, 그리고 제목을 고쳐주시길 바랍니다!
Long short term memory(LSTM) 같은 경우 본문에서는 LSTM을 쓰겠지만, 제목에서는 장단기 기억(LSTM) 으로 위키백과 등 백과사전 등재 용어로 한번만 한국어 사용 부탁드립니다..!

코멘트 수정하여 commit 완료하였습니다!


[BUG] Rich colors don't work properly with prompt_toolkit's patch_stdout.

I'm trying to have a very simple asyncio prompt with prompt_toolkit library, and in addition to that, to have a rich Console() object where to print messages.
To make prompt working properly with prompt_toolkit, we have to use the with patch_stdout(): statement, as you can see in line 39.
The problem is that, when I enable this patch_stdout(), rich stops working properly, and colors are not printed correctly anymore.
When I remove this patch_stdout() block (by using a null context, from context lib, as you can see in line 38), rich works properly, but - of course - I lose the correct prompt functionality of prompt_toolkit library, as you can see from the picture.
Is this a bug? There's a way to fix this?
Shouldn't I mix prompt_toolkit with rich, or it can be done?
I would rich totally, without any prompt_toolkit, if rich would have supported asyncio for its prompt, but in the docs -> https://rich.readthedocs.io/en/stable/prompt.html -- I can't see that rich prompt supports asyncio.
Am I wrong?
import asyncio
import contextlib
from rich.console import Console
from prompt_toolkit import PromptSession
from prompt_toolkit.patch_stdout import patch_stdout

c = Console()

async def hello(i, n):
	for current in range(n):
		c.print(f"[[red]HELLO/{i}: [bold red]{current}/{n}[/]] HELLO WORLD!", "x"*90)
		await asyncio.sleep(0.01)

q = None
async def input_consume():
	global q
	hellos = 0
	while True:
		line = await q.get()
		if line is None or len(line) == 0:
			continue
		elif line == "exit":
 			break
		elif line.startswith("hello"):
			line, _, iterations = line.partition(" ")
			iterations = 5 if len(iterations) == 0 else int(iterations)
			asyncio.create_task(hello(hellos, iterations))
			hellos += 1
		else:
			print("Unknown", line)

async def input_produce():
	global q
	session = PromptSession()
	while True:
		#with contextlib.nullcontext():
		with patch_stdout():
			result = await session.prompt_async('> ')
			if result == "exit":
				q.put_nowait("exit")
				break
			q.put_nowait(result)

async def main():
	global q
	q = asyncio.Queue()
	await asyncio.gather(input_produce(), input_consume())

asyncio.run(main())

Platform: Linux Debian 11, python 3.9.6

Sorry, my mistake.
It seems that prompt_toolkit's patch_stdout() removed escaped characters completely.
I can turn that off with specifying raw=True.
All fixed! Thank you for your time, anyway.


Verify MongoDB Connection Error

When I try to verify a MongoDB connection, I get the error message: Could Not load file or assembly 'MongoDB.Bson, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=null' or one of its dependencies. A strongly-named assembly is required. (Exception from: HRESULT: 0x80131044)

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 9edf1df2-588e-01ff-6d1c-234b8e51d550
Version Independent ID: 27873c08-b6cc-a52f-f277-ff9818a5e4d2
Content: Database migration tool for Azure Cosmos DB
Content Source: articles/cosmos-db/import-data.md
Service: cosmos-db
GitHub Login: @SnehaGunda
Microsoft Alias: sngun

Also, this is with the pre-compiled binary.

@quinn-madson Thanks for the feedback. We are actively investigating and will get back to you soon.

@quinn-madson Thanks for reporting! I can repro it. I have assigned the issue to the content author to investigate and update as appropriate.

@quinn-madson I tested the scenario and when I use the data migration dtui tool, I can repro the error you are seeing. However it's working with the command line. The following example works when you plug-in the appropriate connection endpoints:
#Import all documents from a MongoDB collection
dt.exe /s:MongoDB /s.ConnectionString:mongodb://:@:/ /s.Collection:zips /t:DocumentDBBulk /t.ConnectionString:"AccountEndpoint=;AccountKey=;Database=;" /t.Collection:BulkZips /t.IdField:_id /t.CollectionThroughput:2500
For the error you are seeing with the dtui tool, can you please file report it to the UserVoice or send it to<EMAIL_ADDRESS>alias.

#please-close


.NET branding is wrong in PR bot

Please use proper branding at least for first-party applications: ".Net" -> ".NET":

This is messaging added by the pipeline bot I believe.

The language string is in APIViewInfo. See here:
let apiViews : APIViewInfo[] = (context.pr.apiViewsSDK ?? []).concat(context.pr.apiViewsSpecs ?? []);
(...)
const commentBody = generateApiViewComment({ title: apiViewCommentGroup.title, apiViews });


mkl.h

can u add to your guide how to install intel kernel libary for windows  "mkl.h" ?

In fact, MKL is not necessary. If you want to use MKL, you can download the Intel oneAPI Base Toolkit from oneapi.


Spatie/image bump to V3

Hi, can you bump the version used for Spatie/image to V3 for better Laravel 11 support?  @Tarpsvo

Heya! Version 1.8.0 now uses spatie/image V3. :)

I lied. Had to revert it. Due to a failed composer install, I missed that Manipulations object is now gone and spatie/image v3 requires some major changes. Maybe in 2.0.


feat: Infinite products

Last year, YaelDillies made a pull request to mathlib3 that unfortunately never got merged in: https://github.com/leanprover-community/mathlib/pull/18405. This is the mathlib4 version of that pull request.
We define arbitrarily indexed products in a commutative monoid with a topology. This is done by "multiplicativizing" the currently existing definitions and theorems about arbitrarily indexed sums. That is, the existing code is rewritten in the multiplicative setting, and the original definitions and theorems are recovered using @[to_additive]. Please see this thread on Zulip for information on why this approach was chosen: https://leanprover.zulipchat.com/#narrow/stream/287929-mathlib4/topic/Infinite.20products
As YaelDillies wrote in the description of https://github.com/leanprover-community/mathlib/pull/18405, there are a few small technical issues that arise when directly "multiplicativizing" theorems in this way:

I have renamed cauchySeq_finset_iff_vanishing to cauchySeq_finset_iff_sum_vanishing to make the name multiplicativizable. This is slightly different from the name cauchy_seq_finset_sum_iff_vanishing that YaelDillies used, and it is meant to parallel the existing name cauchySeq_finset_iff_tsum_vanishing.
Currently, on master, there is a theorem called tsum_sum about taking the tsum of a sum, and a theorem called tsum_prod about taking a tsum on a product of two index sets. I have called the multiplicative versions tprod_of_prod and tprod_prod. This is slightly different from the names tprod_prod'' and tprod_prod that YaelDillies used. eric-wieser suggested renaming tsum_prod to tsum_prod_index to get around this issue, which I can do in a separate pull request.

Thank you for the review. I have taken into account your comments.

Thanks; I've added the docstrings

Sorry for missing so many of them.

(Before merging, please take into account Yael's comment on names in the PR description)

(This was already done)

bors r+


cursor_set_line is not work

Operating system or device:
Arch Linux
Issue description (what happened, and what was expected):
I use TextEdit with set_readonly(true) and set_focus_mode(Control.FOCUS_NONE)
After that cursor_set_line isn't work.
Steps to reproduce:
Create a scene, add TextEdit and write script...
please see example project
I program with press key 0, 1, 2 or 3 for change cursor line.
Link to minimal example project (optional but very welcome):
https://www.dropbox.com/s/qcwoxux8w2i4j48/testTextEdit.zip?dl=0

Not a bug, TextEdit is just not redrawing. You must call update() after changing the cursor line:

extends Node

onready var textEdit = get_node("TextEdit")

func _input(event):
	if(event.type == InputEvent.KEY):
		if(event.pressed):
			if(event.scancode == KEY_2):
				cursor_set_line(2)

func cursor_set_line(line):
	textEdit.cursor_set_line(line)
	textEdit.update()