WindowsError: exception: access violation reading 0x00000028 on python -m rasa_nlu.train -c config_mitie.json

rasa NLU version (e.g. 0.7.3):
Used backend / pipeline (mitie):
Operating system (windows server 2012):
Issue:
Not able to train using below command. find output also..
`C:\rasa_nlu-master>python -m rasa_nlu.train -c config_mitie.json
Training to recognize 2 labels: 'location', 'cuisine'
Part I: train segmenter
words in dictionary: 200000
num features: 271
now do training
C:           20
epsilon:     0.01
num threads: 1
cache size:  5
max iterations: 2000
loss per missed segment:  3
C: 20   loss: 3         1
C: 35   loss: 3         1
C: 20   loss: 4.5       1
C: 5   loss: 3  1
C: 20   loss: 1.5       0.888889
C: 20   loss: 3.75      1
C: 21.5   loss: 3       1
C: 20   loss: 3.375     1
C: 18.5   loss: 3       1
C: 20   loss: 3.1875    1
C: 20   loss: 2.9       1
C: 20   loss: 2.9       1
C: 20   loss: 2.97801   1
best C: 20
best loss: 3
num feats in chunker model: 4095
train: precision, recall, f1-score: 1 1 1
Part I: elapsed time: 4 seconds.
Part II: train segment classifier
now do training
num training samples: 9
C: 200   f-score: 1
C: 400   f-score: 1
C: 300   f-score: 1
C: 100   f-score: 1
C: 0.01   f-score: 0.875
C: 600   f-score: 1
C: 1400   f-score: 1
C: 3000   f-score: 1
C: 5000   f-score: 1
C: 2550   f-score: 1
C: 1325   f-score: 1
C: 712.5   f-score: 1
C: 406.25   f-score: 1
C: 253.125   f-score: 1
C: 176.563   f-score: 1
C: 138.281   f-score: 1
C: 119.141   f-score: 1
C: 109.57   f-score: 1
C: 104.785   f-score: 1
C: 102.393   f-score: 1
C: 101.196   f-score: 1
C: 100.598   f-score: 1
C: 100.299   f-score: 1
best C: 100.598
test on train:
4 0
0 5
overall accuracy: 1
Part II: elapsed time: 13 seconds.
df.number_of_classes(): 2
Training to recognize 4 categories: 'greet', 'restaurant_search', 'affirm', 'goo
dbye'
Train classifier
extracting text features
now do training
num training samples: 51
C: 200   f-score: 0.84
C: 400   f-score: 0.84
C: 300   f-score: 0.84
C: 100   f-score: 0.84
C: 0.01   f-score: 0.74
C: 600   f-score: 0.84
C: 1400   f-score: 0.84
C: 3000   f-score: 0.84
C: 5000   f-score: 0.84
C: 2550   f-score: 0.84
C: 1325   f-score: 0.84
C: 712.5   f-score: 0.84
C: 406.25   f-score: 0.84
C: 253.125   f-score: 0.84
C: 176.563   f-score: 0.84
C: 138.281   f-score: 0.84
C: 119.141   f-score: 0.84
C: 109.57   f-score: 0.84
C: 104.785   f-score: 0.84
C: 102.393   f-score: 0.84
C: 101.196   f-score: 0.84
C: 100.598   f-score: 0.84
C: 100.299   f-score: 0.84
best C: 100.598
Traceback (most recent call last):
File "C:\Python27\lib\runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "C:\Python27\lib\runpy.py", line 72, in _run_code
exec code in run_globals
File "C:\rasa_nlu-master\rasa_nlu\train.py", line 72, in
do_train(config)
File "C:\rasa_nlu-master\rasa_nlu\train.py", line 63, in do_train
persisted_path = trainer.persist(config['path'], persistor)
File "rasa_nlu\model.py", line 200, in persist
update = component.persist(dir_name)
File "rasa_nlu\classifiers\mitie_intent_classifier.py", line 67, in persist
self.clf.save_to_disk(classifier_file, pure_model=True)
File "mitie.py", line 732, in save_to_disk
if (_f.mitie_save_text_categorizer_pure_model(filename, self.__obj) != 0):
WindowsError: exception: access violation reading 0x00000028
C:\rasa_nlu-master>`
Content of configuration file (if used & relevant):
config_mitie.json
{
  "pipeline": "mitie",
  "mitie_file": "./data/total_word_feature_extractor.dat",
  "path" : "./models",
  "data" : "./data/examples/rasa/demo-rasa.json"
}

Please file this as a bug report on the MITIE repository (https://github.com/mit-nlp/MITIE/issues) as this is nothing specific to rasa NLU.

In the meantime you could try to use a different backend, e.g. mitie_sklearn.

This is the reply, i received. please help..
People keep posting weird errors when using rasa nlu that only happen when using rasa nlu. So I'm doubtful this has anything to do with MITIE. Can you post a python script that reproduces this problem? One that doesn't involve rasa nlu.

I have installed mitie. but still while running below script, it said mitie not found. so, I copied mitiblib folder from mitib to rasa_nlu_master folder and then run the script again.
python -m rasa_nlu.train -c config_mitie.json
after that, it started train but end up with above issue error. Did my approach is correct to correct mitie issue?

bump i have the exact same issue

@HardikShah289 did u find the soln?

@koolCards , I was able to solve this 0x00000028 error, but after that I stopped working on that. Will try to recall what I did. Sorry for not able to help you much.

May be I have reinstalled rasa again and few other packages again. If MITIE not working, go with scipy. It should work. I have downloaded below things at that time. don't remember exact steps.. might help..
MITIE-models-v0.2.tar.bz2
MITIE-master.zip
en_core_web_sm-1.2.0.tar.gz
scipy-0.19.0-cp27-cp27m-win32.whl
numpy-1.12.1+mkl-cp27-cp27m-win32.whl

Ok. I was not able to figure out what the problem was with MITIE, but I tried use scipy, and it works a lot better. Thanks!


Convert Activations from simple functions to classes and add gradient method.

Currently, all the activations in activations.py are simple functions.
However, for future implementation of Neural Networks, we will also need derivatives methods of each of these
functions. You can have a look into loss_func.py for reference. There, each class represents a loss function and has both loss
and derivative methods. You have to implement something similar for activations.

Converting all activations is not a requirement however you may implement only one or as many you prefer. Each implementation should be in different PR's. If you make a single PR implementing more than one function, I will only give your points for 1 easy contribution. So make sure you capitalise on this 😉.

Hey  @rohansingh9001, I have raised a PR after converting the sigmoid function. Please review it and notify me if there are any changes to be made so that I can convert the rest accordingly.

Hey @rohansingh9001, @agrawalshubham01, could you guys reopen this issue since its not completed or should I open another issue for this?

@Abjcodes now you can contribute here.

@Abjcodes raise single PR, as it will be easier for us to get close look on multiple models , simultaneously.

Hey @kwanit1142 @rohansingh9001  @agrawalshubham01, Please check the PR. Also, I need some clarification regarding the gradient of heviside function. Is it okay to include Diracdelta using the sympy module?
I will fix the failing checks along with the changes to be made if there are any.


i cant install pyautogui

Hi, i am trying to learn about python, i almost have little to zero experience. this is how it went trying to install pyautogui.
i hope you can help
erişim engellendi means access denied
C:\Users\ADMİN\AppData\Local\Programs\Python\Python38\Scripts>pip install PyAutoGui
Collecting PyAutoGui
Downloading PyAutoGUI-0.9.52.tar.gz (55 kB)
|████████████████████████████████| 55 kB 357 kB/s
Collecting pymsgbox
Downloading PyMsgBox-1.0.9.tar.gz (18 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Installing backend dependencies ... done
Preparing wheel metadata ... done
Collecting PyTweening>=1.0.1
Downloading PyTweening-1.0.3.zip (15 kB)
Collecting pyscreeze>=0.1.21
Downloading PyScreeze-0.1.26.tar.gz (23 kB)
Collecting pygetwindow>=0.0.5
Downloading PyGetWindow-0.0.9.tar.gz (9.7 kB)
Collecting mouseinfo
Downloading MouseInfo-0.1.3.tar.gz (10 kB)
Requirement already satisfied: Pillow>=6.2.1 in c:\anaconda\lib\site-packages (from pyscreeze>=0.1.21->PyAutoGui) (7.2.0)
Collecting pyrect
Downloading PyRect-0.1.4.tar.gz (15 kB)
Collecting pyperclip
Downloading pyperclip-1.8.1.tar.gz (20 kB)
Building wheels for collected packages: PyAutoGui, pymsgbox, PyTweening, pyscreeze, pygetwindow, mouseinfo, pyrect, pyperclip
WARNING: Building wheel for PyAutoGui failed: [WinError 5] Erişim engellendi: 'c:\users\admi̇n\appdata\local\pip\cache\wheels'
WARNING: Building wheel for pymsgbox failed: [WinError 5] Erişim engellendi: 'c:\users\admi̇n\appdata\local\pip\cache\wheels'
WARNING: Building wheel for PyTweening failed: [WinError 5] Erişim engellendi: 'c:\users\admi̇n\appdata\local\pip\cache\wheels'
WARNING: Building wheel for pyscreeze failed: [WinError 5] Erişim engellendi: 'c:\users\admi̇n\appdata\local\pip\cache\wheels'
WARNING: Building wheel for pygetwindow failed: [WinError 5] Erişim engellendi: 'c:\users\admi̇n\appdata\local\pip\cache\wheels'
WARNING: Building wheel for mouseinfo failed: [WinError 5] Erişim engellendi: 'c:\users\admi̇n\appdata\local\pip\cache\wheels'
WARNING: Building wheel for pyrect failed: [WinError 5] Erişim engellendi: 'c:\users\admi̇n\appdata\local\pip\cache\wheels'
WARNING: Building wheel for pyperclip failed: [WinError 5] Erişim engellendi: 'c:\users\admi̇n\appdata\local\pip\cache\wheels'
Failed to build PyAutoGui pymsgbox PyTweening pyscreeze pygetwindow mouseinfo pyrect pyperclip
ERROR: Could not build wheels for pymsgbox which use PEP 517 and cannot be installed directly

Maybe you should try starting cmd in admin mode(Search bar and then right click).

Try it in admin mode but also side note: PLEASE add python to your path. it so much easier than navigating to the python directory every time.


Can drag but can't drop

I'm trying to implement this in my app.
This is the layout I'm using (in reality, it contains more stuff but I've reduced it to this and it still doesn't drop):
<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    xmlns:tools="http://schemas.android.com/tools"
    xmlns:float="http://schemas.android.com/apk/res-auto"
    android:orientation="vertical"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    >

    <ScrollView
        android:layout_width="match_parent"
        android:layout_height="match_parent">
        <LinearLayout
            android:layout_width="match_parent"
            android:layout_height="wrap_content"
            android:orientation="vertical">

            <EditText
                android:id="@+id/editReminderContent"
                android:layout_width="match_parent"
                android:layout_height="wrap_content" />
        </LinearLayout>
    </ScrollView>
</LinearLayout>

Debug log says:
...
V/ChecklistViewItemOnDragListener: Dragging item. Action: 1
I/ViewRootImpl[EditReminderActivity]: Reporting drop result: false
V/ChecklistViewItemOnDragListener: Dragging item. Action: 4
...

Behavior: I can create and delete items, then I can drag them but when I drop them they return to their original position.
Can post code if needed (although it's very similar to the one found here

why? i have the same question

@yangchunhui you're testing on Android 7 too? In that case is a known problem that affects also other apps using this library.

sorry，i not use this  library, i listener drag event in different app,i want to receive DragEvent.ACTION_DROP,but not work, i can receive DragEvent.ACTION_DRAG_STARTED and DragEvent.ACTION_DRAG_ENDED only, i using android 9. can you give some advise


Are Triple Composite Keys Supported?

We have a laravel model that has a triple composite key as a pivot table. Currently, in our vuex orm model, we have the primary key as:
static primaryKey = ['line_item_discount_id', 'orientation_id', 'orientation_type_id']
But when data is inserted into the store we always get a duplicate record with the 3rd key with the value of null.
This screenshot shows that 2 records are being inserted even though our API is only delivering 1 record from that pivot table

Does Vuex ORM currently support 3 composite keys, or am I doing something wrong?
My current workaround is to not use 3 composite keys but instead just use a "normal" auto-incrementing id as the primary key. Which is not the end of the world but if I do that then I run into issues trying to access the new pivot attribute introduced in v.36.0. I have to specify the pivot model on the model with a custom attribute name, then access the data through that property name (something like this: https://github.com/vuex-orm/vuex-orm/issues/321#issuecomment-482040165)
Thank you!

Does Vuex ORM currently support 3 composite keys, or am I doing something wrong?

Yes, two or more composite primary keys are supported. Here's some evidence.
Could you elaborate by describing what your field setup looks like, any relations, etc?

Hi @cuebit, thank you very much for the fast response and helping out!
I removed many of the irrelevant basic fields in order to prevent this post from getting too large. Here are the 3 models involved with this relationship
Orientation.js
`
import { Model } from '@vuex-orm/core'
export default class Orientation extends Model {
static entity = 'admin_orientations';
static fields () {
    return {
        id: this.attr(null),
        name: this.attr(''),
    }
}

}
LineItemDiscount.js
import { Model } from '@vuex-orm/core'
import Orientation from "./Orientation";
import LineItemDiscountOrientation from "./LineItemDiscountOrientation";
export default class LineItemDiscount extends Model {
static entity = 'admin_line_item_discounts';
static fields () {
    return {
        id: this.attr(null),
        name: this.attr(''),
        orientations: this.belongsToMany(
            Orientation,
            LineItemDiscountOrientation,
            'line_item_discount_id',
            'orientation_id'
        ),
    }
}

}
`
LineItemDiscountOrientation.js
`
import {Model} from '@vuex-orm/core'
export default class LineItemDiscountOrientation extends Model {
static entity = 'admin_line_item_discount_orientation';
static primaryKey = ['line_item_discount_id', 'orientation_id', 'orientation_type_id']

static fields() {
    return {
        line_item_discount_id: this.attr(null),
        orientation_id: this.attr(null),
        orientation_type_id: this.attr(null)
    }
}

}
`
We don't have a model for the 3rd key, orientation_type (orientation_type_id) as there has been some discussion on wether or not this can just be a string vs a FK to a completely different table.
Now that I just typed that and thought about it. Is that what is causing the issue? We had a previous iteration that looked very similar but instead of orientation_type_id we just had orientation_type and it was a string. It looked something like this:
LineItemDiscountOrientation.js
`
import {Model} from '@vuex-orm/core'
export default class LineItemDiscountOrientation extends Model {
static entity = 'admin_line_item_discount_orientation';
static primaryKey = ['line_item_discount_id', 'orientation_id', 'orientation_type']

static fields() {
    return {
        line_item_discount_id: this.attr(null),
        orientation_id: this.attr(null),
        orientation_type: this.attr('') // can be either "student" or "parent"
    }
}

}
`
Thank you for your help on this

I removed many of the irrelevant basic fields in order to prevent this post from getting too large.

The more the merrier. So long as we have enough information to paint the picture and possibly reproduce, the better. Thank you for your detailed explanation and reflection.

Is it possible to have a composite key, that one of its value is a string?

Yes. Either string or number are acceptable using this.attr().

Could you provide an example of the data you're passing to a model for it to generate these records?
It goes without saying, when Vuex ORM normalizes the insertion data, for LineItemDiscount for example, it's going to create a pivot record in LineItemDiscountOrientation using line_item_discount_id and orientation_id. You can immediately see the issue here – there's no indication of the third key.
The only solution in this case is to insert the pivot records separately if you want to create indexes that respect the primary key composition.
That being said, if you feel your needs do not necessarily warrant a third key as you mentioned, you could insert orientation_type  into the pivot table and access it using the new feature in 0.36.0. Though, you mentioned you're facing some issues with this?

Thank you for the info. If we have the option to keep orientation_type a string, I have reverted LineItemDiscountOrientation.js to the following:
import {Model} from '@vuex-orm/core'

export default class LineItemDiscountOrientation extends Model {
    static entity = 'admin_line_item_discount_orientation';

    static primaryKey = ['line_item_discount_id', 'orientation_id', 'orientation_type']

    static fields() {
        return {
            line_item_discount_id: this.attr(null),
            orientation_id: this.attr(null),
            orientation_type: this.attr('')
        }
    }
}

We are using Vuex ORM Axios that calls the following:
return LineItemDiscount.api().get('/api/line-item-discounts')
which returns:
[
    {
        "id": 1,
        "name": "Low Income Discount",
        "amount": "1000.00",
        "total_available": 100,
        "created_at": null,
        "updated_at": null,
        "orientations": [
            {
                "id": 1,
                "name": "Orientation Name 1",
                "date_start": "2017-11-07 21:00:10",
                "date_end": "2017-11-08 21:00:10",
                "parent_orientation": 1,
                "price": "227.00",
                "waitlist": 1,
                "parent_date_start": "2017-11-07 21:00:10",
                "parent_date_end": "2017-11-08 21:00:10",
                "parent_price": "161.00",
                "parent_capacity": 50,
                "orientation_group_id": 1,
                "created_at": "2020-03-13 21:00:10",
                "updated_at": "2020-03-13 21:00:10",
                "deleted_at": null,
                "pivot": {
                    "line_item_discount_id": 1,
                    "orientation_id": 1,
                    "orientation_type": "student"
                }
            }
        ]
    }
]

When following this pattern (orientation_type as a string), I can access the pivot attribute with success. I just continue to run into an issue in my OG post where I get an duplicate record with orientation_type as null

Thank you for providing the response data.
Unfortunately this is not possible to achieve in this manner. At least without introducing an additional relation which would require an additional model. You're therefore correct in assuming that this is not currently supported.
The issue you are experiencing is admittedly unintentional behaviour caused by the existence of a third primary key and is a side effect when hot reloading or essentially where existing data is updated (because the axios plugin invokes insertOrUpdate). This may even be (albeit inadvertently) raised as a bug.
Alternatively, you are able to define a composite key on Orientation so your orientations are also based on primary key composition. I may be wrong and depends on your source of truth but this just illustrates a solution.
class Orientation extends Model {
  static entity = 'admin_orientations'

  static primaryKey = ['id', 'type']

  static fields() {
    return {
      id: this.attr(null),
      type: this.attr(''), // "student" or "parent"
      name: this.attr('')
    }
  }
}

The intermediate model should compromise of the two primary keys that are defined by the belongsToMany relation. Any additional attributes are purely arbitrary and the new feature since 0.36.0 simply grants access to these.
class LineItemDiscountOrientation extends Model {
  static entity = 'admin_line_item_discount_orientation'

  static primaryKey = ['line_item_discount_id', 'orientation_id']

  static fields() {
    return {
      line_item_discount_id: this.attr(null),
      orientation_id: this.attr(null)
    }
  }
}

Your data would need to reflect this:
[
  {
    "id": 1,
    "name": "Low Income Discount",
    "orientations": [
      {
        "id": 1,
        "type": "student",
        "name": "Orientation Name 1"
      }
    ]
  }
]

I hope I'm understanding your goal.

Cue, thank you very much for your detailed response. I think you have outlined our goal very well and I thank you again for helping me and our team out with this question. I think we have enough to go off from here as i think adding "type" to orientation makes sense.
P.S. I love this package and appreciate you and everyone's hard work! This package makes working with complicated objects in Vuex SO much more manageable and friendly :)

Is there really no way to support triple primary key for a pivot table yet? This is a must have schema.. Having teams based roles my relationship is like this, but it just doesn't work, it will always save it with only 2 and other pivots are then lost
export default class RoleAdmin extends Model {
  // This is the name used as module name of the Vuex Store.
  static entity = 'role_admin'

  static primaryKey = ['role_id', 'user_id', 'site_id']

  // List of all fields (schema) of the post model. `this.attr` is used
  // for the generic field type. The argument is the default value.
  static fields() {
    return {
      role_id: this.attr(null),
      user_id: this.attr(null),
      site_id: this.attr(null),
    };
  }
}

#400


Let subprocess.Popen open outputs in text mode

Fix: Messages from mace_run.cc is printed as Bytes

@lq Great!


Question about 40 content images

Thanks for your fascinating work. It appears that the 40 content images in StyleBench have not been released yet. Could you please provide the download link? Many thanks！

Hi @LinLLLL , thank you for your interest in our work. We will update the content images in StyleBench tomorrow as soon as possible.  Once we have updated, I will ping you.

Hi, we have updated 40 content images here.

Thanks!


mypy plugin fails to resolve generics when used in inheritance

Checks

[x] I added a descriptive title to this issue
[x] I have searched (google, github) for similar issues and couldn't find anything
[x] I have read and followed the docs and still think this is a bug

Bug
Output of python -c "import pydantic.utils; print(pydantic.utils.version_info())":
             pydantic version: 1.8.1
            pydantic compiled: False
                 install path: /Users/sarah/projects/followmytv/hive/.venv/lib/python3.9/site-packages/pydantic
               python version: 3.9.1 (default, Jan 12 2021, 21:01:39)  [Clang 12.0.0 (clang-1<IP_ADDRESS>)]
                     platform: macOS-11.2.1-x86_64-i386-64bit
     optional deps. installed: ['devtools', 'dotenv', 'email-validator', 'typing-extensions']

Description
I'm working on a generic Model which is considered an abstract base class, its only generic variable defines the type of one of the fields in this base class. Subclasses pass this type in as the generic value when they specify the base class as its superclass.
Data loading, exporting, and validation works as expected. Attribute access is getting warnings from mypy with the following error when attempting to use the property that is set via the generic:
 error: "T" has no attribute "foo"  [attr-defined]

Example code
from typing import Any, Dict, Generic, TypeVar

from pydantic import BaseModel
from pydantic.generics import GenericModel

Tbody = TypeVar("Tbody")

class Response(GenericModel, Generic[Tbody]):
    url: str
    body: Tbody

class JsonBody(BaseModel):
    raw: str
    data: Dict[str, Any]

class HtmlBody(BaseModel):
    raw: str
    doctype: str

class JsonResponse(Response[JsonBody]):
    pass

class HtmlResponse(Response[HtmlBody]):
    def custom_method(self) -> None:
        doctype = self.body.doctype. # Line 30
        print(f"self: {doctype}")

example = {"url": "foo.com", "body": {"raw": "..<html>..", "doctype": "html"}}

resp = HtmlResponse.parse_obj(example)
resp.custom_method()

doctype = resp.body.doctype # Line 39
print(f"main: {doctype}")
...

Example's output
self: html
main: html

Example's mypy results
generic.py:30: error: "Tbody" has no attribute "doctype"  [attr-defined]
generic.py:39: error: "Tbody" has no attribute "doctype"  [attr-defined]

After I remove the pydantic mypy plugin
Success: no issues found in 1 source file

I believe I've run into a similar friction point with Pydantic + mypy. The workaround for mypy I used was:
JsonResponse = Response[JsonBody]

_HtmlResponse = Response[HtmlBody]

class HtmlResponse(_HtmlResponse):
    def custom_method(self) -> None:
        doctype = self.body.doctype # Line 30
        print(f"self: {doctype}")

https://gist.github.com/mpkocher/0242865a0c5133a595dd3ae16a30b8dd

I have the same issue and I believe it is a bug.

I believe I've run into a similar friction point with Pydantic + mypy. The workaround for mypy I used was:
JsonResponse = Response[JsonBody]

_HtmlResponse = Response[HtmlBody]

class HtmlResponse(_HtmlResponse):
    def custom_method(self) -> None:
        doctype = self.body.doctype # Line 30
        print(f"self: {doctype}")

https://gist.github.com/mpkocher/0242865a0c5133a595dd3ae16a30b8dd

Great temporal solution.
In addition:
init_forbid_extra = False in mypy.ini should be set. Otherwise we can't add new field to HtmlResponse

Just found a similar error in my project, happens on :
             pydantic version: 1.10.6
            pydantic compiled: True
                 install path: /Users/roigarf/Library/Caches/pypoetry/virtualenvs/backend-lACXRLDI-py3.8/lib/python3.8/site-packages/pydantic
               python version: 3.8.13 (default, Jan  8 2023, 12:08:28)  [Clang 14.0.0 (clang-14<IP_ADDRESS>)]
                     platform: macOS-13.1-arm64-arm-64bit
     optional deps. installed: ['dotenv', 'typing-extensions']

The solution also seems to work, but fixing this could help.
Is this going to be fixed soon?

While I believe this is still broken in pydantic v1, I'll note that this appears to be resolved in v2, with the appropriate changes.
from typing import Any, Dict, Generic, TypeVar

from pydantic import BaseModel

Tbody = TypeVar("Tbody")

class Response(BaseModel, Generic[Tbody]):
    url: str
    body: Tbody

class JsonBody(BaseModel):
    raw: str
    data: Dict[str, Any]

class HtmlBody(BaseModel):
    raw: str
    doctype: str

class JsonResponse(Response[JsonBody]):
    pass

class HtmlResponse(Response[HtmlBody]):
    def custom_method(self) -> None:
        doctype = self.body.doctype  # Line 30
        print(f"self: {doctype}")

example = {"url": "foo.com", "body": {"raw": "..<html>..", "doctype": "html"}}

resp = HtmlResponse.model_validate(example)
resp.custom_method()

doctype = resp.body.doctype  # Line 39
reveal_type(resp.body)
# scratch.py:39: note: Revealed type is "scratch_56.HtmlBody"
reveal_type(JsonResponse.model_validate(example).body)
# scratch.py:40: note: Revealed type is "scratch_56.JsonBody"

print(f"main: {doctype}")
...

I'm open to reviewing a PR to the pydantic plugin to get this fixed in v1, but we are currently focused on getting the v2 release out so, as of now, don't intend to fix this prior to the v2 release.

It seems this is back to being broken on v2; I will investigate further once I have a chance to fix the issues with the plugin, hopefully soon.

This is now fixed in v2.0 (which has just been released 🚀). Fixing it involved a significant update to the plugin, but if it is important to backport a fix to the v1 branch, it may be possible to do that without too much pain. However, without significant interest specifically in supporting this in v1.10, I'm not currently planning to backport the fix (due to the complexity of the overhaul to the plugin).
However, if anyone wants to take a stab at backporting the fix, I would review a PR.


Project Console - Data not loaded in View Companies for project Issue666CCLA_ICLA having more than 100 signatures

Background
Prerequisite

A CLA Group(Project) Issue666CCLA_ICLA should Exist
More than 100 CLA Signatures(ICLA and CCLA) should exist for the project

Steps To Follow

Launch Project Management Console
Login as Project Manager
Click on the Foundation CloudFoundry.org Foundation, Inc.
Click on VIEW COMPANIES for project Issue666CCLA_ICLA
Observed UI hangs with spinning icon and observed below error in console and 400 returned for the api call in network tab
NOTE - Data is fetched in VIEW SIGNATURES for the same project. Also observed this issue only for the project Issue666CCLA_ICLA. The only difference I see in this project is that it has more than 100 CLA signatures

@dealako @nirupamav I have tested as per the steps mentioned above and it's seem that there is no issue and companies loaded in popup.
@nirupamav can you retest it again ?
PFA

@amolsontakke3576 The ticket was logged 27 days ago. Looks like the issue is fixed now. Moving to Review Complete


Acquire original message for player-messaging events

Problem

Plugin developers cannot acquire the original message in any events where a message is involved:

Death event
Chat event
etc.

People aren't wearing enough hats

Proposal
I propose the addition of API methods for acquiring the original Text that would be sent to the client without modification. Something like getOriginalMessage()
Summary

https://github.com/SpongePowered/SpongeAPI/commit/6a85888593c34a83850e35befbec32c4469807e4


S3C-1115 FT: GCP DEL Req

Adds the GCP S3 service deleteObject method

PR has been updated. Reviewers, please be cautious.


Added visit_field_with and ident_key convenience method and macro

Adds the ident_key macro which expands an ident to a const Key, which is a common necessity when manually implementing instances of Allocative. Whereas before you had to write code like code like
struct TyType {
    field: ...
}

impl Allocative for MyType {
    fn visit<'a, 'b: 'a>(&self, visitor: &'a mut allocative::Visitor<'b>) {
        const KEY : Key = Key::new("field");
        let vis = visitor.enter_self(self);
        vis.visit_field(KEY, &self.field);
        vis.exit();
    }
}

or
vis.visit_field({
    const KEY: Key = Key::new("field");
    KEY
}, &self.field);

now you can just write
vis.visit_field(ident_key!(field), &self.field);

With no loss of efficiency

Would you mind splitting this PR in two because as far as I see these two changes are unrelated?

Would you mind splitting this PR in two because as far as I see these two changes are unrelated?

Done :)


fix: updates type name of CustomPublishButtonProps to CustomPublishButtonType

Description
CustomPublishButtonProps is being used to define a component type, not just the shape of props.
React.ComponentType is a utility type used in Typescript to denote a React component. The suffix Props typically implies a definition of props' shape rather than a component.
Fixes #5183
Fix: updates type name of CustomPublishButtonProps to CustomPublishButtonType but keeps old CustomPublishButtonProps defined but mapped to CustomPublishButtonType in order to avoid a breaking change.

[x] I have read and understand the CONTRIBUTING.md document in this repository.

Type of change

[x] Chore (non-breaking change which does not add functionality)
[x] Bug fix (non-breaking change which fixes an issue)
[x] This change requires a documentation update

Checklist:

[x] Existing test suite passes locally with my changes
[x] I have made corresponding changes to the documentation

Let's verify we can use both types interchangeably without errors.


Distribution end epoch error handling

Codecov Report

Merging #322 (cb4d0c6) into main (8c624c0) will decrease coverage by 0.00%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##             main     #322      +/-   ##
==========================================
- Coverage   18.28%   18.27%   -0.01%
==========================================
  Files         133      133
  Lines       21459    21461       +2
==========================================
  Hits         3923     3923
- Misses      16888    16890       +2
  Partials      648      648

Impacted Files
Coverage Δ

x/incentives/keeper/hooks.go
7.14% <0.00%> (-1.20%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8c624c0...cb4d0c6. Read the comment docs.


Title2 in d3plot header

According to Lasso D3plot_header documentation, there an optional 2nd title store in header.title2
But, it seems to not stored when we wrote the d3plot.
Does this 2nd title is define in d3plot format ?
My objective was to store some additional string info inside the D3plot (not obliged to be in the title)
Thanks for help

Writing Issue found : in d3plot.py on function _write_header_part_contact_interface_titles
line : if "title2" in self.header.title2
should be replaced by something like : if self.header.title2 and isinstance(self.header.title2,str):
Nevertheless, I would propose a new function to retrieve header and geometry data without loading all d3plot
We can read just the header using : header = D3plotHeader(d3plotFile)
But this header will not get title2 as it's read later by _parse_geometry
What would be the best way to do this?

If you use the state filter option and leave it empty it should not load state data into memory
Why do you want to check title2 for string? If it is an empty string why bother writing it? It is no issue to change it either way I'm just curious.

Hello Codie,
Thanks for tips, doing that : d3plot = D3plot(d3plotFile,buffered_reading=True,state_filter=[0,],state_array_filter=[]) it's fast to get title2.
I put a check for string, because if somebody do this  title2 = 1.  Maybe it's not needed if after your code will not expect a string.

It is python so while it is tempting to do verify this I do not recommend it. This is within the domain of "you will get punched in the face if you do something very silly".


Surface asset creation errors to e2e tests

So we don't end up with issues like https://github.com/kubernetes-incubator/bootkube/issues/932 - our e2e tests should expose any asset creation errors as test failures.
We don't fail on these errors in the normal case as it's usually easier to debug the resulting cluster if we don't fail early (https://github.com/kubernetes-incubator/bootkube/pull/933#issuecomment-375161442).
Not sure best course here - maybe something like a "strict" mode on start that fails on any error? The bash -set euo equivalent?

Unfortunately, right now the e2e won't fail until a bunch or all of the terraform completes since render runs remotely. It would be good if we could also run the render locally and fail faster.

This would be for the start phase technically (which is remote regardless)

@aaronlevy https://github.com/kubernetes-incubator/bootkube/pull/939/files


Issue : Should react to the case of IIC read failure （应该对IIC读取失败的情况做出反应）

If the read fails, you should return a DateTime error, which is not good for your program.
write_then_read returns false on failure and true on success.
The library should also handle this.
It looks like this.
Chinese (中文) :
如果读取失败,应该会出现返回DateTime错误的情况,这样子对程序不利。
而 write_then_read 在操作失败以后会返回 false ， 成功后会返回true。
库中也应该对这个进行处理 。
像这样子。

What RTC module is being used?
What Arduino sketch is being run?
The validity of the I2C connection is checked in begin() which returns false if it fails. If begin() is returning true but I2C is failing later, then it may be something else.

RTC module : RTC_DS1307
Sometimes this problem occurs :

i have the same on MAX17048G


decont index is wrong

Problem or new feature
Search on decont in the index takes you to the glossary.
Beginners are unable to find the decont operator page i the doc unless they know it is ``<>``` beforehand.
Suggestions
Please change the index target to the <> operator page.

To redirect decont to zen-slicing or one of the two (!) postcircumfix operators that can be used for decontainerisation (yes, [] is no different) is simple. If that's fine with you, please say your preference.
However, the bigger problem is that the decont operator only exists in our heads. I think this is the kind of metaphysical problem that is really hard to resolve. Perhaps the clearest thing to do would be to really just introduce a dedicated method or operator for it. Please read that discussion and try to think about solutions yourself as well.

Ha! So any Zen slice of one thing returns the value, eh? That's funny.
I liked the world where there is a decont postcircumfix operator with syntax xxx<>.
Please take me back there...
(i) because while raku containers are almost invisible, almost is not the same as entirely
(ii) many folk coming from other languages will look for decont in the docs and the current decont operator page does what they expect
If you insist on linking back to zen slices (which is somewhat confusing), then a small link at the bottom of that page that says something like "The decont operation is one of the uses of Zen slices when there is just one element."

Could you please clarify? I'm not sure what you miss; the way I understand is that you imply the same thing @0rir said explicitly in the other issue - namely that there is a postfix <> operator that does the decontainerisation.
So, again, we have 2 concepts and 1 behavior. I see 3 resolutions:

don't change the behavior but do discuss the 2 concepts strongly distinguished: I think this is basically the status quo, plus continuous improvement as with anything. It is an option but I don't like it. This lead to all of us being tripped up and thinking there is actually a "decont operator".
sink the "decont" ship altogether and only talk about it as a sub-case of zen-slicing. I think if properly implemented, this is better than the status quo. although somewhat risky. I see the key to this in keeping the "decont" terminology BUT without "operator". "decont" is conveniently done by zen-slicing and that's about the associated syntax
separate the 2 concepts into 2 distinct "behaviors". Preferably, create something that does this and only this. I think the sole existence of a dedicated thing (that is, without it being actually preferred and used by people) could ease the life of a learner - because at least one could say "this is explicitly for this purpose - but for XYZ reasons, you can see zen-slicing as a convenient way to achieve the same thing"

To clarify per @2colours request, here is what I would like to see...
The facts:

There is one general mechanism, zen slicing
One of zen slicing's specific uses, when applied to a single element, is decont behaviour

The presentation:

If newbies, coming from a language with containers and decont operators, look for it there is a decont operator page like the one we have ... this should be improved by a gentle reference to this being a special case of zen slice behaviour
Those who encounter zen slices first should be informed that decont behaviour is a special case (no change to today)

I do not see an issue with describing the decont behaviour as a postcircumfix <> operator since, err, it is an operator, and that way sits nicely in the list. The art of documentation is to choose words that can carry levels of meaning to multiple audiences.
The merits of what I propose are:

It is a very small adjustment to the current docs (change the target and add one line to the current decont operator page)
It is a practical help for folks to find a way to do a decont operation

FWIW I love that raku often has 2 concepts and 1 behaviour ... IMO it is a power to recognise that a specific point feature (here decont) is a special case of a more generic and elegant semantics (here zen slices). And the existing doc is very close to what we need in that it brings in searchers for the specific and gives them a practical answer ... just needs to add a subtle on ramp to the zen slice angle.

@raiph
I stand corrected that (to my surprise) raku variable containers are not called that in other languages. And so the term decont is less widely used than I had assumed.
Nevertheless it is peppered through the current docs and has some resonance. This Issue does not propose to materially change the docs, but to point the index search term decont to an existing informational page rather than the glossary.
 I have re-read the docs and would ask folks to ignore my "when applied to a single element" words. Thanks for gently explaining my error!

**The decont  operator should be find-able by name. **
I think this well made simple issue is being damaged by  excess noise.

I disagree that there is a "simple issue", just like I disagreed at your related issue that I already linked. I thought the revealing explanation from @raiph blew the illusion of the "simple issue" away for good - not because I love keeping things complicated but because acting upon our own confusion caused by the docs might just cause further harm. Some of that harm comes simply from a kind of reassurance of misunderstandings: the more a misunderstanding is acted upon, the more we will actually believe it to be right.
I realize it only now that this part of the documentation is causing the confusion and it doesn't even say "technically, not a real operator" or anything like that... In fact, neither "postcircumfix < >", nor "postcircumfix <>" are real operators; moreover, they are the same thing: &postcircumfix:<{ }>, handled here (v6.d): https://github.com/rakudo/rakudo/blob/2022.12/src/core.c/array_slice.pm6#L281.
To call zen-slicing "decont operation" is fine by me but I heavily oppose keeping this "fake operator" in the documentation. I do think it's a good thing to give people helpful mental frameworks - but give me a break, to populate the documentation with operators that don't even exist, and then split them up by a distinction that also doesn't exist? That's too much, and honestly, I don't understand why this should even be controversial.
I would really like to eliminate the [postcircumfix <>](https://docs.raku.org/language/operators#___top) and then return to this question. It simply shouldn't be one of the options to explicitly foster this misunderstanding that there are two vaguely overlapping operators with different semantics.

Two thoughts: First, the docs don't need to exactly mirror the Rakudo implementation – even if Rakudo doesn't define a separate <> decontainerization operator, it may be clearer to describe separate operators in the docs (or it may not be; I'm just saying the implementation details don't matter).
Second, does part of the problem come from documenting postcircumfix <> when it should be postfix <>?  If we're talking about a pure decont operator, then there can never be anything inside the <> – and that's exactly the point of distinguishing it, right?

See also: https://github.com/rakudo/rakudo/pull/5124

First, the docs don't need to exactly mirror the Rakudo implementation – even if Rakudo doesn't define a separate <> decontainerization operator, it may be clearer to describe separate operators in the docs (or it may not be; I'm just saying the implementation details don't matter).

That's almost literally what I said here in the other issue but let me quickly quote myself:

To be honest, I don't think this can be done in a non-confusing way if there isn't a tested implementation that does separate the concepts on a code level, so I'm afraid to do something like this right would result in a change on a language level, for relatively small benefit.

So yes, I absolutely agree with the principle but I think this is a very risky road to walk down upon.

Second, does part of the problem come from documenting postcircumfix <> when it should be postfix <>? If we're talking about a pure decont operator, then there can never be anything inside the <> – and that's exactly the point of distinguishing it, right?

And that's literally the topic of the other issue.
Yes, pretty sure it's better to call the fake operator postfix <> than postcircumfix <> and that's a "small issue" - but what does this matter in the bigger picture that in my opinion, whatever we call this, it desperately needs to go. It's "too dangerous to be kept alive".

If one can find fault with the user documentation only by reading the source,  there is no fault in the documentation.
It would have saved me a significant amount of time if I had found <> more easily.

I have completely oppositional experience: I didn't need to read the source whatsoever to suspect something is off about this "decont operator" and I would rather spare this confusion from others as well, not build upon it further. Just a few examples that can be concerns of a user:

when is the "decont operator" invoked and when subscripting?
how can one overload the "decont operator"?

I did find myself asking questions of these sort and eventually it turned out that the questions themselves were nonsensical. I would say there is fault in the documentation if it leads you into this situation.
As much as the documentation is not a Rakudo cookbook, I very strongly believe that it is not the right place to feed the users with "comforting lies". The documentation is bearing with a lot of stuff a reference or a specification is about - the "technically, not a real operator" notes also illustrate that. Mind you: I'm not arguing against a possible mental framework that masks the decont use of <> (and well, {} and []) as an operator - what I'm saying is that the documentation, and in particular the structured, non-tutorial part of the it, is not the right place for that.
I truly hope this approach is convincing enough... the docs are very much the last line of defense. If something could be better explained in a less accurate but simpler manner, there are loads of places to discuss that. However, if something is inaccurate or downright misleading in the one and only documentation - currently the most formal maintained textual resource about the Raku language, - then we really leave those users alone who want to confidently know how Raku code works. And then they have to go read Rakudo sources.

@2colours , yes I agree with your good points made on postfix / postcircumfix and on the fact that decont is not an Operator.
It seems to me that:

there is an operator called "zen slice"
one use of this operator is to remove single items from their container (an idea called "decont")
it also does some other stuff

I would be opposed to completely removing the idea of decont from the documents and replacing entirely with the idea of zen slices. Why? Well, as mentioned, I think that the idea of decont is analagous to the more familiar dereferencing and unboxing found is other languages and where you have containers, from time to time you will want to decont the contents. Also, I think that zen slices is rather a hokey term that is unique to raku ...
If you agree that both terms should have a place in the docs, then the option is:

have a zen slice section that describes how it can be used to decont
have a decont section that describes how it is a particular use of zen slices
have both and link them

So I suppose the variant that is most accurate would be #1.
Perhaps the word 'decont' could be put in the Reference part of the index as it is not formally part of the language?

@p6steve I think "we are getting there", one way or another.
I think you summarized it well (one might nitpick that "zen slice" would rather be the result of subscript invocation with no "indexes" but yes - eventually it's like a thing on its own),
I also agree that the concept of decontainerisation needs to be captured and preserved in some shape or form. I also think that the PR @lizmat created helps on the case by offering tools that do decontainerisation and (clearly) only that. This could immediately help the convergence of decont-related content:

the concept can still be mentioned in the glossary but now there would be a sub and method with one-to-one correspondence to it
the concept can be further elaborated in a how-to article or the traps section (e.g iterating nested arrays might deserve a note about deconting the inner Iterable)
zen-slicing can still cite decontainerisation as a common use (either the glossary or downright the sub and method)
the sub/method page can include a note about the prevalence of using zen-slices for a similar purpose

If we remove the PR from this picture, it's more awkward but not completely hopeless. That would put greater demand on writing a howto guide on the topic, for it couldn't be mapped to a reference ("type") page.
Similarly, the "decont" terminology kind of depends on the outcome of the PR - if you think it helps to move it, I see no reason not to.

cool


(PUP-11629) Accept empty string with MultiJSON

An empty string is valid JSON per the IETF standard, but MultiJSON raises an error when passed one. This commit updates our JSON load method to check for  empty strings and return nil, as the built-in JSON gem does.

Sorry this is more complicated than I thought.
Originally we didn't have multi_json and puppet code called JSON.load directly, for example in 5.4.0:
https://github.com/puppetlabs/puppet/blob/b483798b01cf4f9846bf1173b4be391c2fda07c5/lib/puppet/forge/errors.rb#L85-L86
Then we added our Puppet::Util::Json wrapper to call MultiJson.load and callers were modified to call our wrapper instead, for example c6bbdf0e391af56f55fd934bd1d48173e925dda7
But this means the behavior of Puppet::Util::Json.load depends on whether the multi_json gem can be required or not (either raise or return nil, respectively). The gem was added to the puppet agent runtime in 5.5.x https://github.com/puppetlabs/puppet-runtime/commit/cb7991d55fcc8fcaf4d9b53fa04e7680d71bd657.
I think our options are:

Keep Puppet::Util::Json.load behavior as-is and modify the task & plan metadata code to check for empty string. For example, change this code https://github.com/puppetlabs/puppet/blob/948a4bec67def5244595a6f69976a9a2479f1df1/lib/puppet/module/task.rb#L235 to

if file
  content = Puppet::FileSystem.read(file, :encoding => 'utf-8')
  content.empty? ? {} : Puppet::Util::Json.load(content)

Modify Puppet::Util::Json.load to return nil when multi_json can be required. We'll also need to modify callers to handle this behavior, e.g.

body = JSON.parse(response.body)
if body && body['message']

A hybrid approach, we could fix item 1 now in 7.x and in 8.0, fix 2.

Thanks for the guidance @joshcooper , option #3 seems the best for now. I've overhauled my PR, could you take another look when you have the chance? Thanks!

@mhashizume cool, looks good. I'm not sure if a module (separate from the module's task) can have an empty metadata.json file? If so, we might want to update thehttps://github.com/puppetlabs/puppet/blob/948a4bec67def5244595a6f69976a9a2479f1df1/lib/puppet/module.rb#L217 in the same way?

Actually I'm not sure if a task is allowed to have an empty metadata.json. Maybe we should be:
raise InvalidMetadata.new(err.message, 'puppet.tasks/unparseable-metadata')

@joshcooper I think we're already checking for that, based on this test:
https://github.com/puppetlabs/puppet/blob/948a4bec67def5244595a6f69976a9a2479f1df1/spec/unit/module_spec.rb#L775


Add ability to specify path of generated files

For example:
containers generate compose --path="./.docker"
containers generate dockerfile --path="./.docker"

This would generate the docker-compose.yml and Dockerfile in the path ./.docker

This is a great idea and will provide a lot of flexibility.

I started implementing this but realized that the complexity probably isn't worth it. For example, if we support a directory option for generators, we'd need to support this option for every command. A better solution is to have users create the directory they want then cd to that directory and run containers commands there.
mkdir .example
cd .example
containers generate dockerfile
containers list
containers ...
cd .. # finished

I think we could better address targeting a directory for all commands with a .containersrc file as noted in #3.


Incorrect weather data on fixed location

When in settings the location is set de manual in our case aalst Belgium the weather data doesn't update automatically, neither with weather underground or openweathermap

I believe this may have been the same issue as #77, so I will close for now. If you update and the problem persists, we can reopen.


[question] Server-side config file?

Greetings
Isn't there supposed to be a server-side config file created to enable item spawning and other features?  All I get when I click on an item is recipes
I can't even enable creative/creative+ on my client when using it on a (hosted) server environment.

I found a separate NEI config in each of my world folders.  I'm guessing this means you can config each world separate?
I would actually prefer a global config, and an optional "PerWorld" config option in that file.
Or even a MySQL database option would be nice.

This should cover it https://github.com/TheCBProject/Feedback/issues/5 Add any additional info to there.


Include CRD group in filename

This avoids conflicts between CRDs which have the same short name but are coming from a different group (e.g. Project/config.openshift.io/v1 vs. Project/project.openshift.io/v1)

CI should be fixed now.

Man you're tests are quite comprehensive :sweat_smile:
I fixed the integration test as well.

Man you're tests are quite comprehensive 😅 I fixed the integration test as well.

Yeah, I probably test here a bit too implementation specific. Thank you for the fixes 👍. I'll merge so that a new docker image should be built and will create a release.

@jacksgt Release done and new docker images are pushed. Cheers.

@tricktron Awesome thanks for the quick action - it's very much appreciated!


[Perf, Stress] Add '-log' and '-validate' options in test frameworks

Consider adding options to let the perf tests log and validate operations that can serve as verification runs.

@mikeharder I created this issue as per our discussion.

Hi @Mohit-Chakraborty, we deeply appreciate your input into this project. Regrettably, this issue has remained inactive for over 2 years, leading us to the decision to close it. We've implemented this policy to maintain the relevance of our issue queue and facilitate easier navigation for new contributors. If you still believe this topic requires attention, please feel free to create a new issue, referencing this one. Thank you for your understanding and ongoing support.


Tests\Integration\xWindowsOptionalFeatureSet.Integration.Tests.ps1: Fix issues with disabled feature states

Pull Request (PR) description
Fixes issue where tests fail if disabled Windows Optional Features in up in a state of DisabledWithPayloadRemoved.
This Pull Request (PR) fixes the following issues

Fixes #586

Task list

[x] Added an entry under the Unreleased section of the change log in
CHANGELOG.md. Entry should say what was changed, and how that affects
users (if applicable).
[ ] Resource documentation added/updated in README.md.
[ ] Resource parameter descriptions added/updated in README.md, schema.mof
and comment-based help.
[ ] Comment-based help added/updated.
[ ] Localization strings added/updated in all localization files as
appropriate.
[ ] Examples appropriately added/updated.
[ ] Unit tests added/updated. See
DSC Resource Testing Guidelines.
[x] Integration tests added/updated (where possible). See
DSC Resource Testing Guidelines.
[x] New/changed code adheres to
DSC Resource Style Guidelines and Best Practices

This change is

Codecov Report

Merging #587 into dev will not change coverage.
The diff coverage is n/a.

@@         Coverage Diff         @@
##            dev   #587   +/-   ##
===================================
  Coverage    74%    74%
===================================
  Files        27     27
  Lines      4074   4074
  Branches      4      4
===================================
  Hits       3032   3032
  Misses     1038   1038
  Partials      4      4

Reviewable status: 1 of 3 files reviewed, 1 unresolved discussion (waiting on @ykuijs)
CHANGELOG.md, line 27 at r1 (raw file):
Previously, ykuijs (Yorick Kuijs) wrote…
That was indeed a type. Good catch! Fixed.

TypO ! Not type. Typing is hard...

@PlagueHO , I'll leave this one for you to merge since it's my PR and someone else did the code review. Thanks.

Hi @ykuijs - could you resolve conflicts?

@PlagueHO , I'll resolve the conflicts since it's my PR (@ykuijs just did the review). Thanks.


workflow: add additional images to mirror

Noticed in the calico logs:
pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"calico-csi\" with ImagePullBackOff: \"Back-off pulling image \\\"ghcr.io/flatcar/calico/csi:v3.26.3\\\"\", failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"ghcr.io/flatcar/calico/node-driver-registrar:v3.26.3\\\"\"]" pod="calico-system/csi-node-driver-gdhcm" podUID=05b7a8ec-7d12-486a-b6a2-87388a502d4a

https://github.com/projectcalico/calico/commit/9d39763ccaf9dd6ee19b0fa168fc49d410b66315 might be related? But not sure since it's present since the release 3.25.0
That should help reducing the flaky tests.

I think you could kick off this job after merging it.

That's acceptable since it was just a one-off. Thanks for doing it!


Update ADAM dependency version to 0.27.0

Bumping to ADAM 0.27.0 may require some code changes (the Logging APIs changed, and support for Scala 2.12 may require a cast when using transform methods) and various build workarounds for the Parquet + Avro + Spark mess.

Fixed by #518


migrate to infix bitwise operator and unsigned integer type, part I

use infix style bitwise op in int64, json, quickcheck, random, strconv

Pull Request Test Coverage Report for Build 3412
Details

10 of 10 (100.0%)  changed or added relevant lines in 4 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 83.195%

Totals

Change from base Build 3400:
0.0%

Covered Lines:
4109

Relevant Lines:
4939

💛  - Coveralls


ValueError: Got an unexpected argument: categories

Hi, I've referred to the Usage section in the README as well as #9 and #10.  I've checked out rc1 because it's appropriate for the version of fma_metadata.zip and fma_small.zip that I checked out, and also I've set my environment variables.
Nevertheless, running the line
    tracks = utils.load('tracks.csv')

in either the usage.ipynb file or my own very simple Python script will produce a ValueError about categories:
Traceback (most recent call last):
  File "proc_fma.py", line 3, in <module>
    tracks = utils.load('fma_metadata/tracks.csv')
  File "/media/datadrive/datasets/fma/utils.py", line 213, in load
    'category', categories=SUBSETS, ordered=True)
  File "/home/shawley/anaconda3/envs/panotti/lib/python3.6/site-packages/pandas/core/generic.py", line 5883, in astype
    dtype=dtype, copy=copy, errors=errors, **kwargs
  File "/home/shawley/anaconda3/envs/panotti/lib/python3.6/site-packages/pandas/core/internals/managers.py", line 581, in astype
    return self.apply("astype", dtype=dtype, **kwargs)
  File "/home/shawley/anaconda3/envs/panotti/lib/python3.6/site-packages/pandas/core/internals/managers.py", line 438, in apply
    applied = getattr(b, f)(**kwargs)
  File "/home/shawley/anaconda3/envs/panotti/lib/python3.6/site-packages/pandas/core/internals/blocks.py", line 557, in astype
    return self._astype(dtype, copy=copy, errors=errors, values=values, **kwargs)
  File "/home/shawley/anaconda3/envs/panotti/lib/python3.6/site-packages/pandas/core/internals/blocks.py", line 598, in _astype
    "Got an unexpected argument: {}".format(deprecated_arg)
ValueError: Got an unexpected argument: categories

I haven't seen this error reported in any of the other issues.  Can anyone help, e.g. @mdeff ?
Thanks!

Fixed...  Apparently there's been a change in Pandas, at least for version 0.25.0 (the requiements.txt file for fma doesn't specify a version for pandas).
So the fix is to modify utils.py so that it includes
from pandas.api.types import CategoricalDtype

and the offending line about categories that used to read as
                'category', categories=SUBSETS, ordered=True)

now reads as
                CategoricalDtype(categories=SUBSETS, ordered=True))

. :-)

This error also appears in the creation notebook

Thanks for reporting! This behavior was indeed deprecated in pandas 0.25:

Removed the previously deprecated ordered and categories keyword arguments in astype (GH17742)

I used pandas==0.21.0 when developing this code. I will add package versions to the requirements.txt.

Hi, I tried to use the new code but the error persist in the next line.
Thanks for your work.

TypeError                                 Traceback (most recent call last)
 in ()
1 AUDIO_DIR = os.environ.get('fma_small')
----> 2 tracks = utils.load('fma_metadata/tracks.csv')
/content/utils.py in load(filepath)
212             tracks[column] = tracks[column].astype('category')
213
--> 214         return tracks
215
216
TypeError: astype() got an unexpected keyword argument 'categories'

What does:
import pandas
print(pandas.__version)

prints?

Thanks, I didn't realized that my pandas version was downgraded.

I tried to use the new code but the error persist in the next line.

The line number of the error (--> 214         return tracks) seems to indicate you're not using the latest version. The latest has a try-catch block for this case:
212	        try:
213            tracks['set', 'subset'] = tracks['set', 'subset'].astype(
214                    'category', categories=SUBSETS, ordered=True)
215        except ValueError:
216            # the categories and ordered arguments were removed in pandas 0.25
217            tracks['set', 'subset'] = tracks['set', 'subset'].astype(
218                     pd.CategoricalDtype(categories=SUBSETS, ordered=True))

So both old (< 0.25) and new (>= 0.25) pandas should work.
Could you try a fresh git clone?

Hello, I know I'm a little late, but I would like to add that I am still getting the error despite the try-catch block.

Which exception did you get? If TypeError, could you try #47? What version of pandas (import pandas; print(pandas.__version__)) are you using?

@mdeff I didn't get a type error. My pandas version is 1.2.2 and the error I get is "astype() got an unexpected keyword argument 'categories'".

the error I get is "astype() got an unexpected keyword argument 'categories'"

And that's a TypeError. 😉

Hence I'll merge #47.

@mdeff Thanks! So sorry for my lack of knowledge.

No problem. 😉 Thank you for following-up on the issue.


refactor solving, testing, and creating exercises

Codecov Report
Attention: 608 lines in your changes are missing coverage. Please review.

Comparison is base (76b3394) 44.17% compared to head (52156e4) 20.39%.

Additional details and impacted files
@@             Coverage Diff             @@
##             main      #14       +/-   ##
===========================================
- Coverage   44.17%   20.39%   -23.79%
===========================================
  Files          20       14        -6
  Lines        1408     1025      -383
===========================================
- Hits          622      209      -413
- Misses        734      812       +78
+ Partials       52        4       -48

Flag
Coverage Δ

Linux
20.39% <14.36%> (-23.79%)
:arrow_down:

Windows
?

macOS
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Files
Coverage Δ

pkg/euler/downloader.go
0.00% <0.00%> (ø)

pkg/advent/html.go
88.88% <88.88%> (ø)

pkg/runners/pythonRunner.go
23.17% <40.00%> (+1.92%)
:arrow_up:

pkg/runners/comm.go
51.61% <0.00%> (-3.56%)
:arrow_down:

pkg/euler/euler.go
60.86% <60.86%> (ø)

pkg/utilities/strings.go
40.00% <0.00%> (-60.00%)
:arrow_down:

pkg/advent/tester.go
0.00% <0.00%> (ø)

pkg/advent/solver.go
0.00% <0.00%> (ø)

pkg/runners/golangRunner.go
13.84% <0.00%> (-5.31%)
:arrow_down:

pkg/advent/advent.go
36.80% <36.80%> (ø)

... and 1 more

... and 1 file with indirect coverage changes

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Refactor categorized sort order code

From @payten on November 30, 2015 3:59
To improve the robustness and reduce complexity of the categorized sort order mechanism, we should look to refactor the existing code to use Wicket events.  The problem is, we need to keep the sort order XML in sync with the state of the gradebook.  This includes when grade items are added, removed or moved to a new category; or when categories are enabled, disabled, added or removed.  Wicket events seem like a good fit to broadcast when these changes happen and callbacks can ensure the XML is in sync.
Relates to #282.
Copied from original issue: steveswinsburg/gradebookNG#285

@steveswinsburg  Is there a chance we can move the categorized sort order onto the gb_gradable_object_t table?  I'm seeing some weird behaviour and I wonder if it's due to the caching of the XML stored as a site property?

Was thinking more about this one @steveswinsburg.  Ideally it'd be great to solidify the categorised sort order by adding a categorized_sort_order column to the gb_gradable_object_t table.  We could then optimise SQL for updating this value for the grade item and others in that grade item's category.
I think this would manifest as a new gradebook service endpoint that allowed this value to updated for a single grade item, which would then modify the sequence for any other grade items in the same category.  It may also be complemented with private methods to assist in handling the following gradebook events:

Categories disabled: if all categories are removed from the gradebook, then perhaps this new sort order is null'ed.
Categories enabled: if previous categories are not retained(?), then can assume sort_order within an "uncategorised" state.
Category removed: items would no longer ordered by a category.  Perhaps they can be ordered within an "uncategorised" state?
Category added: no change required category won't have any items yet
Grade item added with category: automatically added to the end of the categorised order
Grade item updated with new category: move to the end of newly assigned category and resequence the sort orders of grade items in the old category
Grade item removed: resequence grade items in category
Grade item drag and drop reordered: resequence the order for grade items within the category.

One outstanding question, do we allow drag and drop of any grade items that do not have a category (in the uncategorised state)?  I believe this currently results in an exception.  This might be a question for the business folks.. but my feeling is that if it's not categorised, it cannot be reordered.

Agreed, lets move it to the gradeable object table so each items knows its own order within the category it is in.
When moving to a new category, the category sort order should be cleared.
Drag and drop should be handled on the front end to prevent it, if possible, however should also be handled on the backend so it doesn't explode and you can't set a categorised order if no category.

Marking this as a bug as the XML site property stuff is a bit flaky.

Hi @steveswinsburg. I've had a initial go at this and I have the database interactions working well I think:
https://github.com/sakaiproject/sakai/compare/master...payten:1389-categorized-sort-order-refactor
I've backed out the XML property code and now rely on the new database value.  I update this sort_order whenever I add or edit an assignment.  If the assignment changes category, then I put it at the end of the category sort order.  Seem ok!
I'm a little unsure about DB migrations etc or handling null categorizedSortOrder values.. I have a TODO in there for further thought.
One thing I noticed, was that the old categorizedSortOrder is retained on page refreshes requiring a page reset to pull back the new value.  So we might need to do some Wicket poking to ensure the state is updated accordingly.
Would be great to get your thoughts on the changes and if you feel good about them, I'll clean them up and continue to think about the edge cases!

Hi @steveswinsburg. I've continued to work through and clean up this WIP.. plus test against the scenarios above. It behaved well! https://github.com/sakaiproject/sakai/compare/master...payten:1389-categorized-sort-order-refactor
On the call, including the category average column in the drag and drop ordering was mentioned.  That won't work with this solution :disappointed:  Perhaps if the category average column had a placeholder entry in the gradable_object table (like the course grade column) it could play nice though?

@payten sorry for the delay, going to look at this shortly. We should perhaps discuss with @kyleblythe all of the requirements around the category ordering (incl #1424) to figure out the best approach for all of it.

Hm.. there's a category_order column available in the gb_category_t table.  Would be able to use that? Once exposed by the service, the Wicket templates and custom Javascript can be easily modified to use this value to sort the categories.

Yup that field looks to be for this purpose. We may need some service methods to handle this on the backend like with the ordering of the items.

Latest WIP version contains code to introduce sorting of categories via the Settings interface and match that up in GradebookNG.  As discussed, it seems we can make use of the gb_category_t table category_order column.
Code is here: https://github.com/sakaiproject/sakai/compare/master...payten:1389-categorized-sort-order
There's a few more things to do before I'll have a PR ready:

add a DB migration for the new gb_gradable_object_t column
investigate why modifying the model doesn't refresh underlying panels in the Settings page (when the category_order changes, hit save, the order doesn't reflect the new order)
review any new TODOs
move new Settings page category drag handles from the Javascript to the Wicket template

Hopefully next week sometime.

Thanks @steveswinsburg.  The refactor has been merged so we no longer rely on XML stored in site properties..


Add actual CITATION?

CITATION appears to be empty.

IF you have suggestions for general text I'm open, I put that there so that people can specify how they want their package cited.

My suggestion is along the line of https://swcarpentry.github.io/git-novice/12-citation/ . What do you think?

I was thinking something more inline with this:
http://journals.aas.org/authors/references.html#Software
but there are a lot of different types of software going up on spacetelescope/, so I'm not sure what the best general thing to put here is.  I'd also like to suggest people put zenodo doi's on their repos when they are ready

Yeah, a link to instructions would be sufficient IMHO. If the software has accompanying paper (like Astropy), people would naturally cite the published paper instead of just as "software".


samba module: fix pam service name typo

Motivation for this change
GDM wouldn't start after configuring samba
Things done
The PAM service name used before this commit was "sambda", with an
extra 'd'. For some reason I don't quite fully understand this typo
prevents GDM from starting. This change fixes that as tested in VMs
built using "nixos-rebuild -I nixpkgs= build-vm".
Please check what applies. Note that these are not hard requirements but merely serve as information for reviewers.

[ ] Tested using sandboxing
(nix.useSandbox on NixOS,
or option build-use-sandbox in nix.conf
on non-NixOS)
Built on platform(s)

[x ] NixOS
[ ] macOS
[ ] Linux

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Fits CONTRIBUTING.md.

@edolstra, I believe you were the original author or at least merged the line that is being updated in this PR.

Thank you


make callback function takes original data array

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/map
Make each, map, reduce methods take callback function similar to the one in js Array.

@mateogianolio
Here is the benchmark result for this PR
><EMAIL_ADDRESS>benchmark /Users/Frank/mygit/vectorious
> node ./benchmarks/vector.js && node ./benchmarks/matrix.js

a, b = Vector.random(1024)
Vector.zeros(1024) x 196,433 ops/sec ±3.21% (75 runs sampled)
Vector.ones(1024) x 191,771 ops/sec ±4.04% (66 runs sampled)
Vector.range(0, 1024) x 68,066 ops/sec ±2.76% (79 runs sampled)
Vector.combine(a, b) x 65,491 ops/sec ±6.38% (53 runs sampled)
Vector.add(a, b) x 184,138 ops/sec ±3.94% (59 runs sampled)
Vector.subtract(a, b) x 187,315 ops/sec ±4.04% (60 runs sampled)
Vector.scale(a, Math.random()) x 210,720 ops/sec ±1.60% (65 runs sampled)
Vector.project(a, b) x 194,279 ops/sec ±2.40% (68 runs sampled)
a.add(b) x 1,940,823 ops/sec ±1.58% (87 runs sampled)
a.subtract(b) x 1,973,579 ops/sec ±1.26% (87 runs sampled)
a.scale(Math.random()) x 5,110,356 ops/sec ±1.90% (85 runs sampled)
a.normalize() x 3,877,903 ops/sec ±2.25% (85 runs sampled)
a.dot(b) x 4,824,868 ops/sec ±1.03% (86 runs sampled)
a.magnitude() x 12,079,812 ops/sec ±1.32% (87 runs sampled)
a.angle(b) x 1,564,677 ops/sec ±0.71% (88 runs sampled)
a.project(b) x 1,836,948 ops/sec ±0.79% (91 runs sampled)
a, b = Matrix.random(128, 128)
Matrix.identity(128) x 11,587 ops/sec ±4.40% (76 runs sampled)
Matrix.magic(128) x 2,507 ops/sec ±3.25% (83 runs sampled)
Matrix.zeros(128, 128) x 11,120 ops/sec ±4.58% (71 runs sampled)
Matrix.ones(128, 128) x 11,396 ops/sec ±4.52% (72 runs sampled)
Matrix.plu(a) x 390 ops/sec ±1.85% (80 runs sampled)
Matrix.augment(a, b) x 3,329 ops/sec ±4.06% (77 runs sampled)
Matrix.add(a, b) x 9,808 ops/sec ±4.49% (77 runs sampled)
Matrix.subtract(a, b) x 9,931 ops/sec ±4.47% (78 runs sampled)
Matrix.scale(Math.random()) x 10,138 ops/sec ±4.59% (73 runs sampled)
Matrix.transpose(a):
a.add(b) x 137,383 ops/sec ±0.89% (89 runs sampled)
a.subtract(b) x 137,881 ops/sec ±0.59% (90 runs sampled)
a.scale(Math.random()) x 201,702 ops/sec ±0.81% (90 runs sampled)
a.multiply(b) x 5,346 ops/sec ±2.31% (80 runs sampled)
a.transpose() x 7,141 ops/sec ±4.07% (57 runs sampled)
a.gauss() x 6,430 ops/sec ±4.58% (79 runs sampled)
a.lu() x 590 ops/sec ±1.50% (83 runs sampled)
a.plu() x 764 ops/sec ±1.18% (84 runs sampled)
a.solve() x 719 ops/sec ±0.89% (87 runs sampled)
a.trace() x 420,398 ops/sec ±6.53% (67 runs sampled)
a.swap(i, j) x 106,601 ops/sec ±2.57% (85 runs sampled)

This is the bench mark for master
><EMAIL_ADDRESS>benchmark /Users/Frank/mygit/vectorious
> node ./benchmarks/vector.js && node ./benchmarks/matrix.js

a, b = Vector.random(1024)
Vector.zeros(1024) x 202,087 ops/sec ±3.01% (79 runs sampled)
Vector.ones(1024) x 190,404 ops/sec ±3.98% (63 runs sampled)
Vector.range(0, 1024) x 65,465 ops/sec ±3.11% (78 runs sampled)
Vector.combine(a, b) x 68,050 ops/sec ±5.71% (56 runs sampled)
Vector.add(a, b) x 195,546 ops/sec ±3.88% (63 runs sampled)
Vector.subtract(a, b) x 200,427 ops/sec ±3.52% (63 runs sampled)
Vector.scale(a, Math.random()) x 217,565 ops/sec ±1.13% (63 runs sampled)
Vector.project(a, b) x 193,003 ops/sec ±3.04% (64 runs sampled)
a.add(b) x 2,065,532 ops/sec ±0.88% (88 runs sampled)
a.subtract(b) x 2,075,626 ops/sec ±0.85% (88 runs sampled)
a.scale(Math.random()) x 5,479,279 ops/sec ±0.89% (89 runs sampled)
a.normalize() x 4,181,604 ops/sec ±0.96% (91 runs sampled)
a.dot(b) x 4,962,911 ops/sec ±1.02% (86 runs sampled)
a.magnitude() x 12,940,876 ops/sec ±0.92% (92 runs sampled)
a.angle(b) x 1,516,228 ops/sec ±1.08% (87 runs sampled)
a.project(b) x 1,814,127 ops/sec ±0.96% (86 runs sampled)
a, b = Matrix.random(128, 128)
Matrix.identity(128) x 11,132 ops/sec ±4.64% (70 runs sampled)
Matrix.magic(128) x 2,554 ops/sec ±1.38% (85 runs sampled)
Matrix.zeros(128, 128) x 11,493 ops/sec ±4.41% (73 runs sampled)
Matrix.ones(128, 128) x 10,975 ops/sec ±4.59% (71 runs sampled)
Matrix.plu(a) x 370 ops/sec ±2.25% (80 runs sampled)
Matrix.augment(a, b) x 3,093 ops/sec ±4.87% (62 runs sampled)
Matrix.add(a, b) x 7,906 ops/sec ±7.19% (64 runs sampled)
Matrix.subtract(a, b) x 9,108 ops/sec ±5.21% (73 runs sampled)
Matrix.scale(Math.random()) x 9,528 ops/sec ±5.37% (71 runs sampled)
Matrix.transpose(a):
a.add(b) x 126,750 ops/sec ±1.25% (84 runs sampled)
a.subtract(b) x 127,694 ops/sec ±1.02% (86 runs sampled)
a.scale(Math.random()) x 200,087 ops/sec ±0.74% (89 runs sampled)
a.multiply(b) x 4,596 ops/sec ±3.39% (70 runs sampled)
a.transpose() x 6,929 ops/sec ±4.92% (62 runs sampled)
a.gauss() x 6,591 ops/sec ±4.68% (78 runs sampled)
a.lu() x 585 ops/sec ±1.28% (82 runs sampled)
a.plu() x 746 ops/sec ±0.89% (83 runs sampled)
a.solve() x 668 ops/sec ±1.00% (83 runs sampled)
a.trace() x 385,709 ops/sec ±7.68% (67 runs sampled)
a.swap(i, j) x 98,739 ops/sec ±2.06% (81 runs sampled)


Mechanism to accept JavaScript from a file or standard input

Missed this when I implemented:

#38
In 2cd70b021e28ffe8c5287ac8e62c221dadbc7746

Current syntax is this:
shot-scraper javascript https://datasette.io/ "document.title"

Do I make that second argument optional, and if it's optional treat it as if - was passed as the -i option? I think that will work.

OK I want these to work:
echo "document.title" | shot-scraper javascript simonwillison.net

And:
shot-scraper javascript simonwillison.net -i code.js

Got that working:
% echo 'document.title' | shot-scraper javascript simonwillison.net
"Simon Willison\u2019s Weblog"
% shot-scraper javascript simonwillison.net -i file.js
"Simon Willison\u2019s Weblog"
% shot-scraper javascript simonwillison.net document.title
"Simon Willison\u2019s Weblog"


Autocompletion for Generic Mapped Types in Typescript (Conditional: Based on First Parameter ,Second Parameter should get Valid Suggestions)

 interface rc {}

interface rc {
  parent1: {
    children: {
      child1: {
        props: {
          child1prop1: any;
          child1prop2: any;
        };
      },
      child2:{
        props: {
          child2prop1: any
        }
      }
    };
  };
}

//@ts-ignore
type ChildProp<T,S> = keyof rc[T]["children"][S]['props'];

declare interface sample<T extends keyof rc> {
   //@ts-ignore
  check<S extends keyof rc[T]["children"],F extends ChildProp<T,S>>(childid:S,propid:F);
}

Above one is declarations I am having. I am not sure issue or it is the feature, based on the first parameter in check method, I need to get valid autocomplete suggestions for the second parameter.
// Here I will get only child1 props as suggestions but not child2 props .
   var sample = sample<'parent1'>
      sample.check('child1','child1prop1')

VSCode is able to validate the code of first and second parameters of the check method.But it is unable to give autocompletion for the second parameter. Is it possible to provide that?

This issue was moved to Microsoft/TypeScript#22889


Add Endeavour OS logo

Added support for EndeavourOS, updated preview.png and README.md.
Closes: #70

@lukas-w, no issues at all. Thanks for your feedback.
What is the source of the image that you have used for comparison? Seems like it may have been resized as the SVG I've submitted is based on this one.

Well now that's weird. IIRC my source was https://commons.wikimedia.org/wiki/File:EndeavourOS_Logo.svg:

However even on endeavouros.com they have different versions of their logo, the first one in this screenshot is more narrow than the other:

So now what? :laughing:

@killajoe You've been maintaining endeavouros-team/endeavouros-theming so I hope you don't mind me tagging you here. We're working on adding Endeavour OS's logo to this icon font but there appear to be two different versions used on the official website. One is wider than the other as can be seen in the screenshots above. Could you help clarify which is the correct one?

@killajoe ...One is wider than the other as can be seen in the screenshots above. Could you help clarify which is the correct one?

Could you link me to the one wider?  i am not the creator of the icons/logo but i rmemebr i needed to modify them because they where not square   so may i missed one of them?

Wide:
https://github.com/endeavouros-team/endeavouros-theming/blob/master/endeavouros.svg
https://github.com/endeavouros-team/endeavouros-theming/blob/master/endeavouros.png
Narrow:
https://github.com/endeavouros-team/endeavouros-theming/blob/master/endeavouros-logo.png
https://github.com/endeavouros-team/endeavouros-theming/blob/master/endeavouros-icon.png
https://github.com/endeavouros-team/endeavouros-theming/blob/master/EndeavourOS-logo.png
https://github.com/endeavouros-team/endeavouros-theming/blob/master/EndeavourOS-icon.png

uh thanks i will try fixing it .. should be the narrow versions.

Thanks @killajoe !
@lukas-w, I'll fix the logo over the weekend and push the changes

@lukas-w, I've pushed the updated narrower logo. Do let me know if it's okay. Thanks!


K3s fails to start after running k3s certificate rotate-ca

Environmental Info:
K3s Version:
v1.31.1+k3s1
Node(s) CPU architecture, OS, and Version:
n/a
Cluster Configuration:
n/a
Describe the bug:
After generating updated CA certs and updating the datastore with the k3s certificate rotate-ca command, K3s fails to restart with the following error:
Oct 08 18:31:31 server-0 k3s[9638]: time="2024-10-08T18:31:31Z" level=fatal msg="/var/lib/rancher/k3s/server/cred/ipsec.psk, /var/lib/rancher/k3s/server/cred/passwd newer than datastore and could cause a cluster outage. Remove the file(s) from disk and restart to be recreated from datastore."
If the token has not been manually specified in the config file and the files are removed, K3s will start once successfully, but subsequent restarts will fail because the token in the passwd file will have been regenerated and no longer match the bootstrap data:
Oct 08 23:02:20 systemd-node-1 k3s[6631]: time="2024-10-08T23:02:20Z" level=fatal msg="starting kubernetes: preparing server: bootstrap data already found and encrypted with different token"
Steps To Reproduce:

Install K3s
Rotate CA certificates
Restart K3s

Expected behavior:
CA certs rotate successfully without causing problems
Actual behavior:
Problems
Additional context / logs:
Regression introduced by https://github.com/k3s-io/k3s/pull/10710
This caused e2e tests to fail, but apparently we didn't check e2e results during last month's release cycle.

Validated on master using commit 054cec8 | version v1.31
Environment Details:
Node(s) CPU architecture, OS, and Version:

uname -a && cat /etc/os-release
Linux ip-172-31-33-119 6.8.0-1016-aws #17-Ubuntu SMP Mon Sep  2 13:48:07 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
PRETTY_NAME="Ubuntu 24.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.1 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

Cluster Configuration:
1 server (configuration does not matter)

Files:

config.yaml

cluster-init: true
write-kubeconfig-mode: 644

rotate-default-ca-certs.sh
https://github.com/k3s-io/k3s/blob/release-1.28/contrib/util/rotate-default-ca-certs.sh

Steps:

Install K3s
Update Certificates using script
Rotate ca-certs k3s certificate rotate-ca
Restart k3s sudo systemctl restart k3s
Check status of k3s sudo systemctl status k3s

Reproduction of the Issue:
- Observations:
k3s -v
k3s version v1.31.1+k3s1 (452dbbc1)
go version go1.22.6

kubectl get nodes,pods -A -o wide
NAME                    STATUS   ROLES                       AGE     VERSION        INTERNAL-IP     EXTERNAL-IP      OS-IMAGE             KERNEL-VERSION   CONTAINER-RUNTIME
node/ip-172-31-33-119   Ready    control-plane,etcd,master   5m50s   v1.31.1+k3s1   <IP_ADDRESS>   [REDACTED]       Ubuntu 24.04.1 LTS   6.8.0-1016-aws   containerd://1.7.21-k3s2

NAMESPACE     NAME                                          READY   STATUS      RESTARTS   AGE     IP          NODE               NOMINATED NODE   READINESS GATES
kube-system   pod/coredns-56f6fc8fd7-r97cx                  1/1     Running     0          5m44s   <IP_ADDRESS>   ip-172-31-33-119   <none>           <none>
kube-system   pod/helm-install-traefik-crd-lxps9            0/1     Completed   0          5m44s   <IP_ADDRESS>   ip-172-31-33-119   <none>           <none>
kube-system   pod/helm-install-traefik-zg7ct                0/1     Completed   1          5m44s   <IP_ADDRESS>   ip-172-31-33-119   <none>           <none>
kube-system   pod/local-path-provisioner-846b9dcb6c-d4r2c   1/1     Running     0          5m44s   <IP_ADDRESS>   ip-172-31-33-119   <none>           <none>
kube-system   pod/metrics-server-5985cbc9d7-9swqh           1/1     Running     0          5m44s   <IP_ADDRESS>   ip-172-31-33-119   <none>           <none>
kube-system   pod/svclb-traefik-578f5134-bbvjc              2/2     Running     0          5m32s   <IP_ADDRESS>   ip-172-31-33-119   <none>           <none>
kube-system   pod/traefik-8dc7cf49b-fnk8q                   1/1     Running     0          5m32s   <IP_ADDRESS>   ip-172-31-33-119   <none>           <none>

$ ./rotate-default-ca-certs.sh
To update certificates, you may now run:
    k3s certificate rotate-ca --path=/var/lib/rancher/k3s/server/rotate-ca

$ k3s certificate rotate-ca --path=/var/lib/rancher/k3s/server/rotate-ca
certificates saved to datastore

$ sudo systemctl restart k3s.service
Job for k3s.service failed because the control process exited with error code.
See "systemctl status k3s.service" and "journalctl -xeu k3s.service" for details.

Oct 14 23:01:19 ip-172-31-33-119 k3s[32802]: time="2024-10-14T23:01:19Z" level=fatal msg="/var/lib/rancher/k3s/server/cred/passwd, /var/lib/rancher/k3s/server/cred/ipsec.psk newer than datastore and could cause a cluster outage. Remove t>
Oct 14 23:01:19 ip-172-31-33-119 systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE
Oct 14 23:01:19 ip-172-31-33-119 systemd[1]: k3s.service: Failed with result 'exit-code'.
Oct 14 23:01:19 ip-172-31-33-119 systemd[1]: k3s.service: Unit process 2700 (containerd-shim) remains running after unit stopped.
Oct 14 23:01:19 ip-172-31-33-119 systemd[1]: k3s.service: Unit process 2743 (containerd-shim) remains running after unit stopped.
Oct 14 23:01:19 ip-172-31-33-119 systemd[1]: k3s.service: Unit process 2764 (containerd-shim) remains running after unit stopped.
Oct 14 23:01:19 ip-172-31-33-119 systemd[1]: k3s.service: Unit process 3710 (containerd-shim) remains running after unit stopped.
Oct 14 23:01:19 ip-172-31-33-119 systemd[1]: k3s.service: Unit process 3788 (containerd-shim) remains running after unit stopped.
Oct 14 23:01:19 ip-172-31-33-119 systemd[1]: Failed to start k3s.service - Lightweight Kubernetes.
Oct 14 23:01:25 ip-172-31-33-119 systemd[1]: k3s.service: Scheduled restart job, restart counter is at 1287.
Oct 14 23:01:25 ip-172-31-33-119 systemd[1]: k3s.service: Found left-over process 2700 (containerd-shim) in control group while starting unit. Ignoring.
Oct 14 23:01:25 ip-172-31-33-119 systemd[1]: k3s.service: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Oct 14 23:01:25 ip-172-31-33-119 systemd[1]: k3s.service: Found left-over process 2743 (containerd-shim) in control group while starting unit. Ignoring.
Oct 14 23:01:25 ip-172-31-33-119 systemd[1]: k3s.service: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Oct 14 23:01:25 ip-172-31-33-119 systemd[1]: k3s.service: Found left-over process 2764 (containerd-shim) in control group while starting unit. Ignoring.
Oct 14 23:01:25 ip-172-31-33-119 systemd[1]: k3s.service: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Oct 14 23:01:25 ip-172-31-33-119 systemd[1]: k3s.service: Found left-over process 3710 (containerd-shim) in control group while starting unit. Ignoring.
Oct 14 23:01:25 ip-172-31-33-119 systemd[1]: k3s.service: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Oct 14 23:01:25 ip-172-31-33-119 systemd[1]: k3s.service: Found left-over process 3788 (containerd-shim) in control group while starting unit. Ignoring.
Oct 14 23:01:25 ip-172-31-33-119 systemd[1]: k3s.service: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Oct 14 23:01:25 ip-172-31-33-119 systemd[1]: Starting k3s.service - Lightweight Kubernetes...
Oct 14 23:01:25 ip-172-31-33-119 sh[32816]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Oct 14 23:01:25 ip-172-31-33-119 systemd[1]: k3s.service: Found left-over process 2700 (containerd-shim) in control group while starting unit. Ignoring.
Oct 14 23:01:25 ip-172-31-33-119 systemd[1]: k3s.service: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Oct 14 23:01:25 ip-172-31-33-119 systemd[1]: k3s.service: Found left-over process 2743 (containerd-shim) in control group while starting unit. Ignoring.
Oct 14 23:01:25 ip-172-31-33-119 systemd[1]: k3s.service: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Oct 14 23:01:25 ip-172-31-33-119 systemd[1]: k3s.service: Found left-over process 2764 (containerd-shim) in control group while starting unit. Ignoring.
Oct 14 23:01:25 ip-172-31-33-119 systemd[1]: k3s.service: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Oct 14 23:01:25 ip-172-31-33-119 systemd[1]: k3s.service: Found left-over process 3710 (containerd-shim) in control group while starting unit. Ignoring.
Oct 14 23:01:25 ip-172-31-33-119 systemd[1]: k3s.service: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Oct 14 23:01:25 ip-172-31-33-119 systemd[1]: k3s.service: Found left-over process 3788 (containerd-shim) in control group while starting unit. Ignoring.
Oct 14 23:01:25 ip-172-31-33-119 systemd[1]: k3s.service: This usually indicates unclean termination of a previous run, or service implementation deficiencies.

$ sudo systemctl status k3s.service
● k3s.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Mon 2024-10-14 20:00:11 UTC; 2s ago
       Docs: https://k3s.io
    Process: 6799 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service 2>/dev/null (code=exited, status=0/SUCCESS)
    Process: 6801 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 6804 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
    Process: 6806 ExecStart=/usr/local/bin/k3s server --token=test (code=exited, status=1/FAILURE)

$ kubectl get nodes,pods -A -o wide
The connection to the server <IP_ADDRESS>:6443 was refused - did you specify the right host or port?
The connection to the server <IP_ADDRESS>:6443 was refused - did you specify the right host or port?

Validation of the Issue:
- Observations:
k3s -v
k3s version v1.31.1+k3s-054cec84 (054cec84)
go version go1.22.6

$ ./rotate-default-ca-certs.sh
To update certificates, you may now run:
    k3s certificate rotate-ca --path=/var/lib/rancher/k3s/server/rotate-ca

$ k3s certificate rotate-ca --path=/var/lib/rancher/k3s/server/rotate-ca
certificates saved to datastore

$ sudo systemctl restart k3s
$ sudo systemctl status k3s
● k3s.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: disabled)
     Active: active (running) since Mon 2024-10-14 21:36:14 UTC; 20s ago
       Docs: https://k3s.io
    Process: 3914 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service 2>/dev/null (code=exited, status=0/SUCCESS)
    Process: 3916 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 3917 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 3918 (k3s-server)

$ kubectl get nodes,pods -A -o wide
NAME                                               STATUS   ROLES                       AGE    VERSION                INTERNAL-IP     EXTERNAL-IP      OS-IMAGE                              KERNEL-VERSION                 CONTAINER-RUNTIME
node/ip-172-31-14-236.us-east-2.compute.internal   Ready    control-plane,etcd,master   133m   v1.31.1+k3s-054cec84   <IP_ADDRESS>   [REDACTED]       SUSE Linux Enterprise Server 15 SP5   5.14.21-150500.55.44-default   containerd://1.7.22-k3s1

NAMESPACE           NAME                                              READY   STATUS      RESTARTS   AGE    IP           NODE                                          NOMINATED NODE   READINESS GATES
kube-system         pod/coredns-56f6fc8fd7-rws6r                      1/1     Running     0          133m   <IP_ADDRESS>    ip-172-31-14-236.us-east-2.compute.internal   <none>           <none>
kube-system         pod/helm-install-traefik-5z8k9                    0/1     Completed   2          133m   <none>       ip-172-31-14-236.us-east-2.compute.internal   <none>           <none>
kube-system         pod/helm-install-traefik-crd-vvwl6                0/1     Completed   0          133m   <none>       ip-172-31-14-236.us-east-2.compute.internal   <none>           <none>
kube-system         pod/local-path-provisioner-5cf85fd84d-sfcxs       1/1     Running     0          133m   <IP_ADDRESS>    ip-172-31-14-236.us-east-2.compute.internal   <none>           <none>
kube-system         pod/metrics-server-5985cbc9d7-79wb2               1/1     Running     0          133m   <IP_ADDRESS>    ip-172-31-14-236.us-east-2.compute.internal   <none>           <none>
kube-system         pod/svclb-nginx-loadbalancer-svc-7efe6867-vpzfh   1/1     Running     0          128m   <IP_ADDRESS>   ip-172-31-14-236.us-east-2.compute.internal   <none>           <none>
kube-system         pod/svclb-traefik-3241a63e-j5wx2                  2/2     Running     0          133m   <IP_ADDRESS>    ip-172-31-14-236.us-east-2.compute.internal   <none>           <none>
kube-system         pod/traefik-57b79cf995-zdp52                      1/1     Running     0          133m   <IP_ADDRESS>    ip-172-31-14-236.us-east-2.compute.internal   <none>           <none>
test-ingressroute   pod/whoami-86c8d79cf4-42scz                       1/1     Running     0          127m   <IP_ADDRESS>   ip-172-31-14-236.us-east-2.compute.internal   <none>           <none>
test-ingressroute   pod/whoami-86c8d79cf4-jg75p                       1/1     Running     0          127m   <IP_ADDRESS>   ip-172-31-14-236.us-east-2.compute.internal   <none>           <none>
test-loadbalancer   pod/test-loadbalancer-6c774b8bb9-chrwc            1/1     Running     0          128m   <IP_ADDRESS>   ip-172-31-14-236.us-east-2.compute.internal   <none>           <none>
test-loadbalancer   pod/test-loadbalancer-6c774b8bb9-prjgc            1/1     Running     0          128m   <IP_ADDRESS>   ip-172-31-14-236.us-east-2.compute.internal   <none>           <none>

Hi @brandond!
It seems that I've been running into the issue you mentioned above. I tried to rotate CA certificates and now have the mentioned error message ("bootstrap data already found and encrypted with different token")...
Is there a way to recover from this situation? Like, forcing a new token and/or certificates?
I have tried all possible combinations of the multiple tls folders, token file values and cred/passwd file values...
The workloads still seem to be running fine but my single node k3s cluster can not be started again. And it would be awesome if K3s could be recovered without needing to recreate the complete cluster...
Thanks in advance for your reply!
BR, Pascal

@pascaliske please open a new issue or discussion and fill out the issue template. Without knowing more about your cluster configuration, and whether or not you have any etcd/database snapshots to recover from, I can't recommend any specific steps.


[5.0] Various post-release cleanups

@toabctl I dropped the problematic commit

This branch is soon EOL, so this probably shouldn't be merged any more. Please reopen if you feel otherwise.

This branch is soon EOL, so this probably shouldn't be merged any more. Please reopen if you feel otherwise.


Archiving of SIP

Despite the same key prefix that we compare between "Administration">"Django Administration">"API Key Permissions">"API keys" and configuration in ./settings/settings.cfg.docker file on the line 73/74, archived SIP is still pending in "overview of information packages in progress" listing. But should be under archived packages.

This is fixed in the current version


Adding SBG E2E nightly test config

Purpose

Adding a bdd test for the SBG end-to-end DAG

Proposed Changes

ADD SBG end-to-end test
CHANGE pyproject toml to include the "backoff" package

Issues

unity-sds/unity-sps#65

Testing

Tested with manual sps deployment
Test results attached (test_airflow_sbg_dag_e2e_results.txt), status verified in airflow UI

This looks good to me. In the future, if we have multiple DAGs that we want to test it might make sense to parameterize the DAG ID in the feature file using a Scenario Outline. That way we won't have to create a new test/feature for every new DAG.

@LucaCinquini what directory should it go into? as you say, it's not a unit test, so I assumed unity-test/unit/ was the wrong tree (so I put it under unity-test/system). Should I create a unity-test/integration?
@drewm-jpl You're right, the only thing that would really change in testing other DAGs would be the DAG post url, and maybe the backoff time limit. I'll see if I can make it scenario outline instead, but I'm not sure how to trigger it. Does pytest_bdd expose those parameters as arguments (for pytest calls)?

@jpl-btlunsfo I don't necessarily think it needs to be done for this PR but for future reference pytest-bdd does support scenario outlines: https://pytest-bdd.readthedocs.io/en/stable/#scenario-outlines

Regarding the location of the tests, SLIM is recommending /tests/system/integration
https://yunks128.github.io/slim/docs/guides/software-lifecycle/continuous-testing/#integration-tests

https://github.com/unity-sds/unity-sps/actions/runs/8898825847/job/24436801528 successfully ran, took ~38m


Created HelloPreferences Starter App

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.