DSPy Typed Predict is not extracting the LLM output correctly.

I am trying to use the TypePredictor to extract information from some inputs.  The program failed due to exceeding number of retries but when inspecting the output of the LLM it seems to be in a parsable way.
The LLM used is gemini-1.5-pro
Code
import dspy
from pydantic import BaseModel, Field

class Field(BaseModel):
    key: str = Field(description="The key to be extracted from the context")
    value: str = Field(description="the value to be extracted from the context")
    truthful: bool = Field(
        description="is a tuple contains the extracted value and a boolean indicating if the information is truthful and correct."
    )
    confidence: float = Field(ge=0, le=1, description="The confidence score for the answer")

class Fields(BaseModel):
    fields: list[Field]

class QAExtractionSignature(dspy.Signature):
    """Your task is to extract the key-value pairs from the document that will follow the instructions on the field to extract.\nplease keep the content you extract truthful and correct based on the document text provided"""

    document: str = dspy.InputField()
    instruction: str = dspy.InputField(description="The instruction on which field to extact and how to extract it.")
    fields: Fields = dspy.OutputField(
        description="The list of fields extracted from the document text. IMPORTANT!! This must follow a semicolon separated list of values!"
    )

raw_ocr_text = "1 Wages, tips, other comp. 2307.11 2 Federal income tax withheld 41.98 3 Social security wages 2129.03 4 Social security tax withheld 143.04 5 Medicare wages and tips 2307.11 6 Medicare tax withheld 33.45 d Control number<PHONE_NUMBER> TM6 Dept. Corp. CTQF Employer use only 55394 c Employer's name, address, and ZIP code HUT AMERICAN GROUP LLC 6200 OAK TREE BLVD - SUITE 250 INDEPENDENCE, OH 44131 b Employer's FED ID number 85-4006070 a Employee's SSA number XXX-XX-2489 7 Social security tips 178.08 8 Allocated tips 9 10 Dependent care benefits 11 Nonqualified plans 12a DD | 69.71 14 Other 12b 12c 12d 13 Stat emp. Ret. plan 3rd party sick pay e/f Employee's name, address and ZIP code MARY A DAWKINS 1609 DEATSVILLE HWY MILLBROOK, AL 36054 15 State AL Employer's state ID no. R010981778 16 State wages, tips, etc. 2307.11 17 State income tax 87.86 18 Local wages, tips, etc. 19 Local income tax 20 Locality name City or Local Filing"
    extract = ";".join(
        [
            "Social Security Number",
            "Employer's FED ID number",
            "Employee's SSA number",
            "Wages, tips, other comp.",
            "Federal income tax withheld",
            "Social security wages",
            "Social security tax withheld",
            "Medicare wages and tips",
            "Medicare tax withheld",
            "State",
            "Employer's state ID no.",
            "State wages, tips, etc.",
            "State income tax",
            "Local income tax",
        ]
    )
    extraction_task = f"extract the following fields (Semicolon separated): {extract}"
    qa_extract = TypedQAExtraction()
    predictions = qa_extract(
        instruction=extraction_task,
        document=raw_ocr_text,
    )
    print(predictions)

when looking at the LLM input and output it seems it should be able to handle with it.
Your task is to extract the key-value pairs from the document that will follow the instructions on the field to extract.
please keep the content you extract truthful and correct based on the document text provided

---

Follow the following format.

Document: ${document}
Instruction: The instruction on which field to extact and how to extract it.
Outputs: The list of key-value pairs extracted from the document text. IMPORTANT!! This must follow a semicolon separated list of values!. Respond with a single JSON object. JSON Schema: {"$defs": {"Field": {"properties": {"key": {"description": "The key to be extracted from the context", "title": "Key", "type": "string"}, "value": {"description": "the value to be extracted from the context", "title": "Value", "type": "string"}, "truthful": {"description": "is a tuple contains the extracted value and a boolean indicating if the information is truthful and correct.", "title": "Truthful", "type": "boolean"}, "confidence": {"description": "The confidence score for the answer", "maximum": 1.0, "minimum": 0.0, "title": "Confidence", "type": "number"}}, "required": ["key", "value", "truthful", "confidence"], "title": "Field", "type": "object"}}, "properties": {"fields": {"items": {"$ref": "#/$defs/Field"}, "title": "Fields", "type": "array"}}, "required": ["fields"], "title": "Fields", "type": "object"}

---

Document: 1 Wages, tips, other comp. 2307.11 2 Federal income tax withheld 41.98 3 Social security wages 2129.03 4 Social security tax withheld 143.04 5 Medicare wages and tips 2307.11 6 Medicare tax withheld 33.45 d Control number<PHONE_NUMBER> TM6 Dept. Corp. CTQF Employer use only 55394 c Employer's name, address, and ZIP code HUT AMERICAN GROUP LLC 6200 OAK TREE BLVD - SUITE 250 INDEPENDENCE, OH 44131 b Employer's FED ID number 85-4006070 a Employee's SSA number XXX-XX-2489 7 Social security tips 178.08 8 Allocated tips 9 10 Dependent care benefits 11 Nonqualified plans 12a DD | 69.71 14 Other 12b 12c 12d 13 Stat emp. Ret. plan 3rd party sick pay e/f Employee's name, address and ZIP code MARY A DAWKINS 1609 DEATSVILLE HWY MILLBROOK, AL 36054 15 State AL Employer's state ID no. R010981778 16 State wages, tips, etc. 2307.11 17 State income tax 87.86 18 Local wages, tips, etc. 19 Local income tax 20 Locality name City or Local Filing Copy W-2 Wage and Tax 2021 Statement OMB No. 1545-0008 Copy 2 to be filed with employee's City or Local Income Tax Return.
Instruction: extract the following fields as Key-Value pair (Semicolon separated): Social Security Number;Employer's FED ID number;Employee's SSA number;Wages, tips, other comp.;Federal income tax withheld;Social security wages;Social security tax withheld;Medicare wages and tips;Medicare tax withheld;State;Employer's state ID no.;State wages, tips, etc.;State income tax;Local income tax
Outputs:

Output
[
  "```json\n{\"fields\": [{\"key\": \"Social Security Number\", \"value\": \"XXX-XX-2489\", \"truthful\": true, \"confidence\": 1.0}, {\"key\": \"Employer's FED ID number\", \"value\": \"85-4006070\", \"truthful\": true, \"confidence\": 1.0}, {\"key\": \"Employee's SSA number\", \"value\": \"XXX-XX-2489\", \"truthful\": true, \"confidence\": 1.0}, {\"key\": \"Wages, tips, other comp.\", \"value\": \"2307.11\", \"truthful\": true, \"confidence\": 1.0}, {\"key\": \"Federal income tax withheld\", \"value\": \"41.98\", \"truthful\": true, \"confidence\": 1.0}, {\"key\": \"Social security wages\", \"value\": \"2129.03\", \"truthful\": true, \"confidence\": 1.0}, {\"key\": \"Social security tax withheld\", \"value\": \"143.04\", \"truthful\": true, \"confidence\": 1.0}, {\"key\": \"Medicare wages and tips\", \"value\": \"2307.11\", \"truthful\": true, \"confidence\": 1.0}, {\"key\": \"Medicare tax withheld\", \"value\": \"33.45\", \"truthful\": true, \"confidence\": 1.0}, {\"key\": \"State\", \"value\": \"AL\", \"truthful\": true, \"confidence\": 1.0}, {\"key\": \"Employer's state ID no.\", \"value\": \"R010981778\", \"truthful\": true, \"confidence\": 1.0}, {\"key\": \"State wages, tips, etc.\", \"value\": \"2307.11\", \"truthful\": true, \"confidence\": 1.0}, {\"key\": \"State income tax\", \"value\": \"87.86\", \"truthful\": true, \"confidence\": 1.0}, {\"key\": \"Local income tax\", \"value\": null, \"truthful\": true, \"confidence\": 0.0}]}\n```"
]

Error after the retry exhaused
    raise ValueError(
ValueError: ('Too many retries trying to get the correct output format. Try simplifying the requirements.', {'outputs': 'Input should be a valid string: fields, 13, value (error type: string_type)'})

Adding a fuzzy model_validate_json to your model could help - https://github.com/stanfordnlp/dspy/issues/1001#issuecomment-2110984085

What changes were made to the signature?

no changes.
class QAExtractionSignature(Signature):
    """Your task is to extract the key-value pairs from the document that will follow the instructions on the field to extract.\nplease keep the content you extract truthful and correct based on the document text provided"""

    document: str = InputField()
    instruction: str = InputField(
        description="The instruction on which field to extact and how to extract it."
    )
    fields: Fields = OutputField(
        description="The list of fields extracted from the document text. IMPORTANT!! This must follow a semicolon separated list of values!"
    )

I suspect you're interfering with some internal namespace - dspy.SignatureMeta has a fields property which i'm guessing is interfering with the fields you're defining. In general its good to avoid using generic names for members if you can avoid it.
Upon further investigation, it looks like the model_validate_json deserializer is wrapped by the below function:
def _unwrap_json(output):
    output = output.strip()
    if output.startswith("```"):
        if not output.startswith("```json"):
            raise ValueError("json output should start with ```json")
        if not output.endswith("```"):
            raise ValueError("Don't write anything after the final json ```")
        output = output[7:-3].strip()
    if not output.startswith("{") or not output.endswith("}"):
        raise ValueError("json output should start and end with { and }")
    return ujson.dumps(ujson.loads(output))  # ujson is a bit more robust than the standard json

which the user is not able to replace with their own deserializer. I will open a PR to make this optional

Thanks for investigating it further Mike. I am still confused on what is going on, it seems the way typed signature works is very brittle.  I made the following change and I am back at the Too many retries problem
    raise ValueError(
ValueError: ('Too many retries trying to get the correct output format. Try simplifying the requirements.', {'tax_from_fields': "ValueError('Could not find valid json')"})

def get_min_length(model: Type[BaseModel]):
    min_length = 0
    for key, field in model.model_fields.items():
        min_length += len(key)
        if not isclass(field.annotation):
            continue
        if issubclass(field.annotation, BaseModel):
            min_length += get_min_length(field.annotation)
    return min_length

class ModelField(BaseModel):
    key: str = Field()
    value: str = Field()
    truthful: bool = Field(
        description="is a tuple contains the extracted value and a boolean indicating if the inFieldsation is truthful and correct."
    )
    confidence: float = Field(ge=0, le=1, description="The confidence score for the answer")

class TaxFromFields(BaseModel):
    tax_from: list[ModelField] = Field()

    @classmethod
    def model_validate_json(
        cls, json_data: str, *, strict: bool | None = None, context: dict[str, Any] | None = None
    ) -> "list[ModelField]":
        try:
            __tracebackhide__ = True
            return cls.__pydantic_validator__.validate_json(
                json_data, strict=strict, context=context
            )
        except ValidationError:
            min_length = get_min_length(cls)
            for substring_length in range(len(json_data), min_length - 1, -1):
                for start in range(len(json_data) - substring_length + 1):
                    substring = json_data[start : start + substring_length]
                    try:
                        __tracebackhide__ = True
                        res = cls.__pydantic_validator__.validate_json(
                            substring, strict=strict, context=context
                        )
                        return res
                    except ValidationError as exc:
                        last_exc = exc
                        pass
        raise ValueError("Could not find valid json") from last_exc

class QAExtractionSignature(Signature):
    """Your task is to extract the key-value pairs from the document that will follow the instructions on the field to extract.\nplease keep the content you extract truthful and correct based on the document text provided"""

    document: str = InputField()
    instruction: str = InputField(
        description="The instruction on which field to extact and how to extract it."
    )
    tax_from_fields: TaxFromFields = OutputField(
        description="The list of fields extracted from the document text. IMPORTANT!! This must follow a semicolon separated list of values!"
    )

class TypedQAExtraction(dspy.Module):
    def __init__(self):
        self.qa_extraction = dspy.functional.TypedPredictor(QAExtractionSignature)

    def forward(self, document: str, instruction: str) -> TaxFromFields:
        return self.qa_extraction(document=document, instruction=instruction).tax_from_fields

When looking at the trace. it seems the
Predict(QAExtractionSignature).forward is sending in
{"document": "1 Wages, tips, other comp. 2307.11 2 Federal income tax withheld 41.98 3 Social security wages 2129.03 4 Social security tax withheld 143.04 5 Medicare wages and tips 2307.11 6 Medicare tax withheld 33.45 d Control number<PHONE_NUMBER> TM6 Dept. Corp. CTQF Employer use only 55394 c Employer's name, address, and ZIP code HUT AMERICAN GROUP LLC 6200 OAK TREE BLVD - SUITE 250 INDEPENDENCE, OH 44131 b Employer's FED ID number 85-4006070 a Employee's SSA number XXX-XX-2489 7 Social security tips 178.08 8 Allocated tips 9 10 Dependent care benefits 11 Nonqualified plans 12a DD | 69.71 14 Other 12b 12c 12d 13 Stat emp. Ret. plan 3rd party sick pay e/f Employee's name, address and ZIP code MARY A DAWKINS 1609 DEATSVILLE HWY MILLBROOK, AL 36054 15 State AL Employer's state ID no. R010981778 16 State wages, tips, etc. 2307.11 17 State income tax 87.86 18 Local wages, tips, etc. 19 Local income tax 20 Locality name City or Local Filing Copy W-2 Wage and Tax 2021 Statement OMB No. 1545-0008 Copy 2 to be filed with employee's City or Local Income Tax Return.", "instruction": "extract the following fields (Semicolon separated): Social Security Number;Employer's FED ID number;Employee's SSA number;Wages, tips, other comp.;Federal income tax withheld;Social security wages;Social security tax withheld;Medicare wages and tips;Medicare tax withheld;State;Employer's state ID no.;State wages, tips, etc.;State income tax;Local income tax", "error_tax_from_fields_0": "ValueError('Could not find valid json')", "new_signature": "StringSignature(document, instruction, error_tax_from_fields_0 -> tax_from_fields\n    instructions='Your task is to extract the key-value pairs from the document that will follow the instructions on the field to extract.\\nplease keep the content you extract truthful and correct based on the document text provided'\n    document = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Document:', 'desc': '${document}', 'format': <function TypedPredictor._prepare_signature.<locals>.<lambda> at 0x15443ad40>})\n    instruction = Field(annotation=str required=True description='The instruction on which field to extact and how to extract it.' json_schema_extra={'__dspy_field_type': 'input', 'desc': 'The instruction on which field to extact and how to extract it.', 'prefix': 'Instruction:', 'format': <function TypedPredictor._prepare_signature.<locals>.<lambda> at 0x1544cdc60>})\n    error_tax_from_fields_0 = Field(annotation=str required=True json_schema_extra={'prefix': 'Past Error in Tax From Fields:', 'desc': 'An error to avoid in the future', '__dspy_field_type': 'input'})\n    tax_from_fields = Field(annotation=TaxFromFields required=True description='The list of fields extracted from the document text. IMPORTANT!! This must follow a semicolon separated list of values!' json_schema_extra={'__dspy_field_type': 'output', 'desc': 'The list of fields extracted from the document text. IMPORTANT!! This must follow a semicolon separated list of values!. Respond with a single JSON object. JSON Schema: {\"$defs\": {\"ModelField\": {\"properties\": {\"key\": {\"title\": \"Key\", \"type\": \"string\"}, \"value\": {\"title\": \"Value\", \"type\": \"string\"}, \"truthful\": {\"description\": \"is a tuple contains the extracted value and a boolean indicating if the inFieldsation is truthful and correct.\", \"title\": \"Truthful\", \"type\": \"boolean\"}, \"confidence\": {\"description\": \"The confidence score for the answer\", \"maximum\": 1.0, \"minimum\": 0.0, \"title\": \"Confidence\", \"type\": \"number\"}}, \"required\": [\"key\", \"value\", \"truthful\", \"confidence\"], \"title\": \"ModelField\", \"type\": \"object\"}}, \"properties\": {\"tax_from\": {\"items\": {\"$ref\": \"#/$defs/ModelField\"}, \"title\": \"Tax From\", \"type\": \"array\"}}, \"required\": [\"tax_from\"], \"title\": \"TaxFromFields\", \"type\": \"object\"}', 'prefix': 'Tax From Fields:', 'format': <function TypedPredictor._prepare_signature.<locals>.<lambda> at 0x1544ce3e0>, 'parser': <function TypedPredictor._prepare_signature.<locals>.<lambda> at 0x1544ce5c0>})\n)"}

The output is which seems fine to me.
{"tax_from_fields": "```json\n{\n  \"tax_from\": [\n    {\n      \"key\": \"Social Security Number\",\n      \"value\": \"XXX-XX-2489\",\n      \"truthful\": true,\n      \"confidence\": 1.0\n    },\n    {\n      \"key\": \"Employer's FED ID number\",\n      \"value\": \"85-4006070\",\n      \"truthful\": true,\n      \"confidence\": 1.0\n    },\n    {\n      \"key\": \"Employee's SSA number\",\n      \"value\": \"XXX-XX-2489\",\n      \"truthful\": true,\n      \"confidence\": 1.0\n    },\n    {\n      \"key\": \"Wages, tips, other comp.\",\n      \"value\": \"2307.11\",\n      \"truthful\": true,\n      \"confidence\": 1.0\n    },\n    {\n      \"key\": \"Federal income tax withheld\",\n      \"value\": \"41.98\",\n      \"truthful\": true,\n      \"confidence\": 1.0\n    },\n    {\n      \"key\": \"Social security wages\",\n      \"value\": \"2129.03\",\n      \"truthful\": true,\n      \"confidence\": 1.0\n    },\n    {\n      \"key\": \"Social security tax withheld\",\n      \"value\": \"143.04\",\n      \"truthful\": true,\n      \"confidence\": 1.0\n    },\n    {\n      \"key\": \"Medicare wages and tips\",\n      \"value\": \"2307.11\",\n      \"truthful\": true,\n      \"confidence\": 1.0\n    },\n    {\n      \"key\": \"Medicare tax withheld\",\n      \"value\": \"33.45\",\n      \"truthful\": true,\n      \"confidence\": 1.0\n    },\n    {\n      \"key\": \"State\",\n      \"value\": \"AL\",\n      \"truthful\": true,\n      \"confidence\": 1.0\n    },\n    {\n      \"key\": \"Employer's state ID no.\",\n      \"value\": \"R010981778\",\n      \"truthful\": true,\n      \"confidence\": 1.0\n    },\n    {\n      \"key\": \"State wages, tips, etc.\",\n      \"value\": \"2307.11\",\n      \"truthful\": true,\n      \"confidence\": 1.0\n    },\n    {\n      \"key\": \"State income tax\",\n      \"value\": \"87.86\",\n      \"truthful\": true,\n      \"confidence\": 1.0\n    },\n    {\n      \"key\": \"Local income tax\",\n      \"value\": null,\n      \"truthful\": false,\n      \"confidence\": 0.0\n    }\n  ]\n}\n```"}

However, when look at what the model actually received as input some of the failed forward call seems to be having some kind of syntax problem.
Your task is to extract the key-value pairs from the document that will follow the instructions on the field to extract.
please keep the content you extract truthful and correct based on the document text provided

---

Follow the following format.

Document: ${document}

Instruction: The instruction on which field to extact and how to extract it.

Past Error in Tax From Fields: An error to avoid in the future

Tax From Fields: The list of fields extracted from the document text. IMPORTANT!! This must follow a semicolon separated list of values!. Respond with a single JSON object. JSON Schema: {"$defs": {"ModelField": {"properties": {"key": {"title": "Key", "type": "string"}, "value": {"title": "Value", "type": "string"}, "truthful": {"description": "is a tuple contains the extracted value and a boolean indicating if the inFieldsation is truthful and correct.", "title": "Truthful", "type": "boolean"}, "confidence": {"description": "The confidence score for the answer", "maximum": 1.0, "minimum": 0.0, "title": "Confidence", "type": "number"}}, "required": ["key", "value", "truthful", "confidence"], "title": "ModelField", "type": "object"}}, "properties": {"tax_from": {"items": {"$ref": "#/$defs/ModelField"}, "title": "Tax From", "type": "array"}}, "required": ["tax_from"], "title": "TaxFromFields", "type": "object"}

---

Document: 1 Wages, tips, other comp. 2307.11 2 Federal income tax withheld 41.98 3 Social security wages 2129.03 4 Social security tax withheld 143.04 5 Medicare wages and tips 2307.11 6 Medicare tax withheld 33.45 d Control number<PHONE_NUMBER> TM6 Dept. Corp. CTQF Employer use only 55394 c Employer's name, address, and ZIP code HUT AMERICAN GROUP LLC 6200 OAK TREE BLVD - SUITE 250 INDEPENDENCE, OH 44131 b Employer's FED ID number 85-4006070 a Employee's SSA number XXX-XX-2489 7 Social security tips 178.08 8 Allocated tips 9 10 Dependent care benefits 11 Nonqualified plans 12a DD | 69.71 14 Other 12b 12c 12d 13 Stat emp. Ret. plan 3rd party sick pay e/f Employee's name, address and ZIP code MARY A DAWKINS 1609 DEATSVILLE HWY MILLBROOK, AL 36054 15 State AL Employer's state ID no. R010981778 16 State wages, tips, etc. 2307.11 17 State income tax 87.86 18 Local wages, tips, etc. 19 Local income tax 20 Locality name City or Local Filing Copy W-2 Wage and Tax 2021 Statement OMB No. 1545-0008 Copy 2 to be filed with employee's City or Local Income Tax Return.

Instruction: extract the following fields (Semicolon separated): Social Security Number;Employer's FED ID number;Employee's SSA number;Wages, tips, other comp.;Federal income tax withheld;Social security wages;Social security tax withheld;Medicare wages and tips;Medicare tax withheld;State;Employer's state ID no.;State wages, tips, etc.;State income tax;Local income tax

Past Error in Tax From Fields: ValueError('Could not find valid json')

Tax From Fields:

Another one is
Your task is to extract the key-value pairs from the document that will follow the instructions on the field to extract.
please keep the content you extract truthful and correct based on the document text provided

---

Follow the following format.

Document: ${document}

Instruction: The instruction on which field to extact and how to extract it.

Past Error in Tax From Fields: An error to avoid in the future

Past Error (2) in Tax From Fields: An error to avoid in the future

Tax From Fields:
The list of fields extracted from the document text. IMPORTANT!! This must follow a semicolon separated list of values!. Respond with a single JSON object.
You MUST use this format: ```json
{"tax_from": [{"key": "tax_from", "value": "2022-10-10", "truthful": true, "confidence": 0.9}]}
```

JSON Schema: {"$defs": {"ModelField": {"properties": {"key": {"title": "Key", "type": "string"}, "value": {"title": "Value", "type": "string"}, "truthful": {"description": "is a tuple contains the extracted value and a boolean indicating if the inFieldsation is truthful and correct.", "title": "Truthful", "type": "boolean"}, "confidence": {"description": "The confidence score for the answer", "maximum": 1.0, "minimum": 0.0, "title": "Confidence", "type": "number"}}, "required": ["key", "value", "truthful", "confidence"], "title": "ModelField", "type": "object"}}, "properties": {"tax_from": {"items": {"$ref": "#/$defs/ModelField"}, "title": "Tax From", "type": "array"}}, "required": ["tax_from"], "title": "TaxFromFields", "type": "object"}

---

Document: 1 Wages, tips, other comp. 2307.11 2 Federal income tax withheld 41.98 3 Social security wages 2129.03 4 Social security tax withheld 143.04 5 Medicare wages and tips 2307.11 6 Medicare tax withheld 33.45 d Control number<PHONE_NUMBER> TM6 Dept. Corp. CTQF Employer use only 55394 c Employer's name, address, and ZIP code HUT AMERICAN GROUP LLC 6200 OAK TREE BLVD - SUITE 250 INDEPENDENCE, OH 44131 b Employer's FED ID number 85-4006070 a Employee's SSA number XXX-XX-2489 7 Social security tips 178.08 8 Allocated tips 9 10 Dependent care benefits 11 Nonqualified plans 12a DD | 69.71 14 Other 12b 12c 12d 13 Stat emp. Ret. plan 3rd party sick pay e/f Employee's name, address and ZIP code MARY A DAWKINS 1609 DEATSVILLE HWY MILLBROOK, AL 36054 15 State AL Employer's state ID no. R010981778 16 State wages, tips, etc. 2307.11 17 State income tax 87.86 18 Local wages, tips, etc. 19 Local income tax 20 Locality name City or Local Filing Copy W-2 Wage and Tax 2021 Statement OMB No. 1545-0008 Copy 2 to be filed with employee's City or Local Income Tax Return.

Instruction: extract the following fields (Semicolon separated): Social Security Number;Employer's FED ID number;Employee's SSA number;Wages, tips, other comp.;Federal income tax withheld;Social security wages;Social security tax withheld;Medicare wages and tips;Medicare tax withheld;State;Employer's state ID no.;State wages, tips, etc.;State income tax;Local income tax

Past Error in Tax From Fields: ValueError('Could not find valid json')

Past Error (2) in Tax From Fields: ValueError('Could not find valid json')

Tax From Fields:

All of the forward result that failed except one looks legit to me.
I am not sure if my use case is too complicated which doesn't seems to be to me.  Any suggestions on how to make this work?

https://github.com/stanfordnlp/dspy/pull/1053

#1053 should help you parse the json correctly with the custom json parser.


[EUWE] Fix opening of Cockpit consoles.

https://bugzilla.redhat.com/show_bug.cgi?id=1409956

I have no idea what this check is or why it is failing. ping @chriskacerguis

@martinpovolny it checks for any security vulnerabilities.  However, in this case, it looks like it was an error on the check side (we can ignore it).


Multilayer CNN Softmax Script Error

Hi, I transferred your script for the Softmax regression and Multilayer CNN from your guide:
https://www.tensorflow.org/versions/r0.12/tutorials/mnist/pros/.
I get the following error whether I am running just the simple Softmax regression model or the full Multilayer CNN. Seems like an issue with the arguments used for the cross_entropy, but I don't know what the issue is...could you please help?
ValueError                                Traceback (most recent call last)
 in ()
39     # between the target and the softmax activation function applied to the model's prediction.
40
---> 41 cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(y, y_))
42
43     # Note that tf.nn.softmax_cross_entropy_with_logits internally applies the softmax
~\Anaconda2\envs\tensorflow\lib\site-packages\tensorflow\python\ops\nn_ops.py in softmax_cross_entropy_with_logits(_sentinel, labels, logits, dim, name)
1742   """
1743   _ensure_xent_args("softmax_cross_entropy_with_logits", _sentinel,
-> 1744                     labels, logits)
1745
1746   # TODO(pcmurray) Raise an error when the labels do not sum to 1. Note: This
~\Anaconda2\envs\tensorflow\lib\site-packages\tensorflow\python\ops\nn_ops.py in _ensure_xent_args(name, sentinel, labels, logits)
1696   if sentinel is not None:
1697     raise ValueError("Only call %s with "
-> 1698                      "named arguments (labels=..., logits=..., ...)" % name)
1699   if labels is None or logits is None:
1700     raise ValueError("Both labels and logits must be provided.")
ValueError: Only call softmax_cross_entropy_with_logits with named arguments (labels=..., logits=..., ...)
Here is my script (essentially copied from the guide):
Multilayer CNN using Tensorflow
Load Data
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets('MNIST_data', one_hot=True)
Start Tensorflow InteractiveSession
import tensorflow as tf
sess = tf.InteractiveSession()
Build Softmax regression model. Define our model and training loss function.
# creating nodes for the input images and target output classes

x = tf.placeholder(tf.float32, shape=[None, 784])
y_ = tf.placeholder(tf.float32, shape=[None, 10])
sess.run(tf.global_variables_initializer())
Weight initialization
def weight_variable(shape):
initial = tf.truncated_normal(shape, stddev=0.1)
return tf.Variable(initial)
def bias_variable(shape):
initial = tf.constant(0.1, shape=shape)
return tf.Variable(initial)
Convolutional and pooling operations
Our convolutions uses a stride of one and are zero padded so that
the output is the same size as the input. Our pooling is plain
old max pooling over 2x2 blocks. To keep our code cleaner, let's
also abstract those operations into functions.
def conv2d(x, W):
return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')
def max_pool_2x2(x):
return tf.nn.max_pool(x, ksize=[1, 2, 2, 1],
strides=[1, 2, 2, 1], padding='SAME')
Implement first convolutional layer. t will consist of convolution,
followed by max pooling. The convolution will compute 32 features
for each 5x5 patch. Its weight tensor will have a shape of [5, 5, 1, 32].
The first two dimensions are the patch size, the next is the number of
input channels, aTo apply the layer, we first reshape x to a 4d tensor, with the second and third dimensions corresponding to image width and height, and the final dimension corresponding to the number of color channels.nd the last is the number of output channels. We will
also have a bias vector with a component for each output channel.
W_conv1 = weight_variable([5, 5, 1, 32])
b_conv1 = bias_variable([32])
To apply the layer, we first reshape x to a 4d tensor, with the second
and third dimensions corresponding to image width and height, and the
final dimension corresponding to the number of color channels.
x_image = tf.reshape(x, [-1,28,28,1])
Then convolve x_image with the weight tensor, add the bias, apply the
ReLU function, and finally max pool. The max_pool_2x2 method will reduce
the image size to 14x14.
h_conv1 = tf.nn.relu(conv2d(x_image, W_conv1) + b_conv1)
h_pool1 = max_pool_2x2(h_conv1)
Second convolutional layer.
In order to build a deep network, we stack several layers of this type.
The second layer will have 64 features for each 5x5 patch.
W_conv2 = weight_variable([5, 5, 32, 64])
b_conv2 = bias_variable([64])
h_conv2 = tf.nn.relu(conv2d(h_pool1, W_conv2) + b_conv2)
h_pool2 = max_pool_2x2(h_conv2)
Densely connected layer.
Now that the image size has been reduced to 7x7, we add a fully-connected
layer with 1024 neurons to allow processing on the entire image. We
reshape the tensor from the pooling layer into a batch of vectors,
multiply by a weight matrix, add a bias, and apply a ReLU.
W_fc1 = weight_variable([7 * 7 * 64, 1024])
b_fc1 = bias_variable([1024])
h_pool2_flat = tf.reshape(h_pool2, [-1, 7764])
h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, W_fc1) + b_fc1)
Dropout
To reduce overfitting, we will apply dropout before the readout layer.
We create a placeholder for the probability that a neuron's output is
kept during dropout. This allows us to turn dropout on during training,
and turn it off during testing. TensorFlow's tf.nn.dropout op
automatically handles scaling neuron outputs in addition to masking them,
so dropout just works without any additional scaling.
keep_prob = tf.placeholder(tf.float32)
h_fc1_drop = tf.nn.dropout(h_fc1, keep_prob)
Readout layer.  one layer softmax regression.
W_fc2 = weight_variable([1024, 10])
b_fc2 = bias_variable([10])
y_conv = tf.matmul(h_fc1_drop, W_fc2) + b_fc2
Train and Evaluate model.
cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(y_conv, y_))
train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)
correct_prediction = tf.equal(tf.argmax(y_conv,1), tf.argmax(y_,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
sess.run(tf.global_variables_initializer())
for i in range(20000):
batch = mnist.train.next_batch(50)
if i%100 == 0:
train_accuracy = accuracy.eval(feed_dict={
x:batch[0], y_: batch[1], keep_prob: 1.0})
print("step %d, training accuracy %g"%(i, train_accuracy))
train_step.run(feed_dict={x: batch[0], y_: batch[1], keep_prob: 0.5})
print("test accuracy %g"%accuracy.eval(feed_dict={x: mnist.test.images, y_: mnist.test.labels, keep_prob: 1.0}))

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
Have I written custom code
OS Platform and Distribution
TensorFlow installed from
TensorFlow version
Bazel version
CUDA/cuDNN version
GPU model and memory
Exact command to reproduce

Nagging Awaiting Response: It has been 14 days with no activityand the awaiting response label was assigned. Is this still an issue?

Yes. I never got it to work.

From: Alfred Sorten Wolf<EMAIL_ADDRESS>Sent: Tuesday, February 20, 2018 11:45:50 AM
To: tensorflow/tensorflow
Cc: peggyn7; Author
Subject: Re: [tensorflow/tensorflow] Multilayer CNN Softmax Script Error (#16755)
Nagging Awaiting Response: It has been 14 days with no activityand the awaiting response label was assigned. Is this still an issue?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/tensorflow/tensorflow/issues/16755#issuecomment-367093590, or mute the threadhttps://github.com/notifications/unsubscribe-auth/Aidkgn2iQXcNQsBkCbZ87J6IOPqvUXQaks5tWx_OgaJpZM4R4q0i.

Please follow tutorials from https://www.tensorflow.org/tutorials/layers instead. Thanks.

Thank you. Will try that.

From: Yanping Huang<EMAIL_ADDRESS>Sent: Thursday, February 22, 2018 3:51:58 PM
To: tensorflow/tensorflow
Cc: peggyn7; Author
Subject: Re: [tensorflow/tensorflow] Multilayer CNN Softmax Script Error (#16755)
Please follow tutorials from https://www.tensorflow.org/tutorials/layers instead. Thanks.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/tensorflow/tensorflow/issues/16755#issuecomment-367862303, or mute the threadhttps://github.com/notifications/unsubscribe-auth/Aidkgq_PBd5mZ2kaHj4DChGxoC9Q2xbMks5tXf2egaJpZM4R4q0i.

Closing due to lack of activity but please reopen if you believe there is a bug.


Transfer.groovy - specify build file

In the build report in todays implementation, the absolute path of the processed file is used.
There is a well hidden API documentation on the CopyToPDS, which allows to define the build file attribute using the key() method.

Delivered via zAppBuild Release 3.4.0 (#392).
Feedback welcome via #410


Contribution to the README for Capybara + database_cleaner

Hi!
Many devs use database_cleaner in conjunction with RSpec, Capybara and Selenium/Capybara-webkit.
I added one line in the readme to link to a post by Avdi Grimm where he shows a proper setup of database_cleaner to work with Capybara and includes a thorough explanation.
It seems to me that easying access to config instructions for such a popular setup will benefit both your gem and new developers.
Cheers!

Closing since the database cleaner README has a more up to date recommended config


@conda-forge-admin, please add user @melissawm

Additional comment:
No response

Hi! This is the friendly automated conda-forge-webservice.
I just wanted to let you know that I added user @melissawm in conda-forge/sphinx-tags-feedstock#2.


Latest ABC / SV contentious hardfork

Hi Teran! What are you up to these days?
What is the best way to contact you to chat privately? I am interested in maybe porting bitcash over to the latest system but I'm not sure what side of the fence you're on with that and just wanted to discuss around it a bit.
There are heaps of javascript libraries out there but I think it's really nice to have a python one - even if it's basic. You never know - some heavier hitters might be willing to help out if it got good enough.
Regards
Cyborg

Hey!
Been busy, less technical work lately. email or Bitmessage. BM-2cUbcEfGKbVN16aVusiYDeBAtb1gj3q98M or sega01 shift+2, go-beyond.org.
Haven't followed the fork much. Probably leaning more towards ABC but happy to discuss either way.
Thanks for your message.
-Teran


#1490 - Implement defevent macro

This PR is aiming to address https://github.com/anoma/anoma/issues/1490 issue.
It implements defevent macro.
With this change we could for example refactor  Anoma.Node.Transaction.Backends module this way:
  typedstruct enforce: true, module: CompleteEvent do
    field(:tx_id, integer())
    field(:tx_result, {:ok, any()} | :error)
  end

  ...

  deffilter CompleteFilter do
    %EventBroker.Event{body: %Node.Event{body: %CompleteEvent{}}} ->
      true

    _ ->
      false
  end

=>
  use Node.Event.DefEvent

  defevent CompleteEvent, filter: CompleteFilter do
    field(:tx_id, integer())
    field(:tx_result, {:ok, any()} | :error)
  end

Please let me know if this meets your expectations. I'm happy to adjust or expand it if needed. I’ve tried to follow all the conventions I’m aware of, but I’m not certain everything is correct.

This looks very nice, thank you!
One thing I haven't thought about before filing an issue: having docs or typed structs is nice. Can you add an extra argument to the macro (with default nil e.g.) which then puts a @typedoc in the struct?

The entire body of the do ... end block is passed into typedstruct, which means you can specify @typespec in the same way as with the typedstruct macro. For example:
defevent CompleteEvent, filter: CompleteFilter do
  @typedoc """
  Foo bar
  """

  field(:tx_id, integer())
  field(:tx_result, {:ok, any()} | :error)
end

However, there is currently no way to specify @typedoc for the Filter module. If it is needed, do you have any ideas on what kind of syntax would make the most sense?

The entire body of the do ... end block is passed into typedstruct, which means you can specify @typespec in the same way as with the typedstruct macro.

Oh, I did not notice that, thanks for pointing it out! That's all that I needed

@mariari asked whether you could by chance also provide an example not only of the macro but of the macro usage? That is, an example of:

making some arbitrary Event with filter using the macro, then
subscribing to it in the EventBroker, then
sending the event of a specified format to the EventBroker
send a message of a different format
check whether the message has been succesfully received and the other has not

You can use my with_subscription macro if you'd want to: https://github.com/anoma/anoma/pull/1498

Sure, I'm on it 👍

Added the the extra example. Does it make sense?
I didn't use with_subscription since it hasn't been merged yet. I could rebase this branch on top of artem/with-sub or refactor it later on. Which approach would you prefer?

Added the the extra example. Does it make sense?
I didn't use with_subscription since it hasn't been merged yet. I could rebase this branch on top of artem/with-sub or refactor it later on. Which approach would you prefer?

If you want to use it, you can just merge the topic (branch) into yours. It's fine either way though no more changes needed I don't think

Ok, let's keep it as it is then for now. 👍

Rerolling to 48443e1d69da89e562f7595bda4200b7b92a3589 to fix some credo errors and make the history a bit cleaner.


fix: wait for parent to be mounted before rendering children

Hi @troch
We have noticed that put the the subscription in the constructor was not the best thing to do, because it will cause a setState in the constructor. setState in the constructor is warned by React.StrictMode.
It seems that the way to go is to restore the behavior of 6.1.1 (before constructor move) but wait for the parent to be mounted (and so subcribed) before mounting the children.
This ensure that any children can enjoy any router5 feature from componentDidMount()
Summary of the PR:

Reverting commits where the subscription was moved in the constructor
Add the property "mounted" in the state
Conditionally mount the children when mounted is set to true

Happy reviewing!

Hi @titouancreach

wait for the parent to be mounted (and so subcribed) before mounting the children

Unfortunately, this is not a suitable approach because it breaks server-side rendering.

because it will cause a setState in the constructor

It is not clear to me how it could happen unless a child component navigates in its constructor? setState is only called if there is an update received.

Arf, alway forgot about SSR.
This bug shows up on the "inbox" as well, here is the fork with all the dependencies updated and react strict mode added.
https://stackblitz.com/edit/react-router5-new-context-api-w5a89k?file=index.js
Just click on "Why router 5" to trigger the message in the console

Hello,
What is really the problem with SSR (I'm not familiar to it) ?
ComponentDidMount is never called at SSR level and the string generated doesn't contain any children ?
If it is the only issue, maybe checking for SS or not should solve ?

ComponentDidMount is never called at SSR level and the string generated doesn't contain any children ?

Yes, components are not "mounted" server-side, and nothing will be rendered as a result below the first route component.
If you then don't want to block server-side but do want to wait client-side, you'll get a flash as you described, due to the content disappearing on the first client-side render.

close in favor of !362


Creating calling policy documentation

New powershell cmdlets for managing calling policies

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

skype/skype-ps/skype/Get-CsTeamsCallingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

skype/skype-ps/skype/Grant-CsTeamsCallingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

skype/skype-ps/skype/Set-CsTeamsCallingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Problem with INSERT if having a VIEW with subqueries & AS keyword

That's a really strange one!
When you create a view (let's call it test_stats) that has a subquery with multiple tables (test1 and test2) combined using UNION ALL clause, and the first table in that clause has some custom-named columns (using AS); then if you try to insert some data to the first table (test1) the server will raise an exception that the renamed column wasn't found.
Should the VIEW/UNION ALL query affect the original tables? It looks really weird to me.
Here's an example:
>> CREATE TABLE test1 (id UInt64, date Date) ENGINE = MergeTree(date, (id, date), 8192);
>> CREATE TABLE test2 (id UInt64, date Date) ENGINE = MergeTree(date, (id, date), 8192);

>> CREATE VIEW test_stats AS SELECT _name, rows_total, rows_inserted, minus(rows_total, rows_inserted) AS _delta FROM
       (SELECT * FROM (
           SELECT 'test1' AS _name, 1234 AS rows_total, count() AS rows_inserted FROM default.test1
               UNION ALL
           SELECT 'test2' AS _name, 4568 AS rows_total, count() AS rows_inserted FROM default.test2
           )
       );

>> SELECT * FROM test_stats;

Ok. 0 rows in set.

>> INSERT INTO test1 VALUES (1, '2016-09-10');

Received exception from server:
Code: 10. DB::Exception: Received from localhost:9000, ::1. DB::Exception: Not found column: 'rows_total'.

1 rows in set.

Even if I write explicitly that I want to insert just the id and date, it still raises the same exception:
>> INSERT INTO test1 (id, date) VALUES (1, '2016-09-10');

Received exception from server:
Code: 10. DB::Exception: Received from localhost:9000, ::1. DB::Exception: Not found column: 'rows_total'.

1 rows in set.

Inserting into other tables works OK with either syntax:
>> INSERT INTO test2 (id, date) VALUES (1, '2016-09-10');

Ok. 1 rows in set.

>> INSERT INTO test2 VALUES (2, '2016-10-10');

Ok. 1 rows in set.

>> SELECT * FROM test_stats;

┌─_name─┬─rows_total─┬─rows_inserted─┬─_delta─┐
│ test2 │       4568 │             2 │   4566 │
└───────┴────────────┴───────────────┴────────┘`

>> DROP TABLE test_stats;

Ok. 0 rows in set.

Let's make sure it was the view/subquery/union's fault:
>> INSERT INTO test1 (id, date) VALUES (1, '2016-09-10');

Ok. 1 rows in set.

>> CREATE VIEW test_stats AS SELECT cnt FROM
          (SELECT cnt FROM (
              SELECT count() AS cnt FROM default.test1
              UNION ALL
              SELECT count() AS cnt FROM default.test2
          )
      );

>> INSERT INTO test1 (id, date) VALUES (2, '2016-09-10');

Received exception from server:
Code: 10. DB::Exception: Received from localhost:9000, ::1. DB::Exception: Not found column cnt in block. There are only columns: id, date.

1 rows in set.

I modified the server's output to make it less bloated; this bug appears on the v1.1.53981 (sorry, can't test it with the newer one right now).

Confirmed. We have an issue with VIEWS and UNION ALL.
PS. Sorry for the delay, I was on conference trip this week.

I just noticed that the tables are affected even if there are views without UNION ALL.
If there's a column rename in the subquery, it messes with the latter INSERT queries to the table.
Also, ClickHouse seems to lose track of the current database.
Proof of concept:
 :) CREATE TABLE hits
    (
        date Date,
        timestamp DateTime,
        id String
    )
    ENGINE = MergeTree(date, (date, id), 8192);

Ok. 0 rows in set. Elapsed: 0.081 sec.

 :) CREATE VIEW hits_break AS SELECT date, id AS hit_id FROM hits;

Ok. 0 rows in set. Elapsed: 0.080 sec.

 :) INSERT INTO hits VALUES ('2016-11-11', '2016-11-11 14:00:00', '1');

Ok. 0 rows in set. Elapsed: 0.007 sec.

 :) CREATE VIEW hits_break_2 AS SELECT * FROM (SELECT date, id AS hit_id FROM hits);

Received exception from server:
Logical error while creating StorageView. Could not retrieve database name from select query.
 :) CREATE VIEW hits_break_2 AS SELECT * FROM (SELECT date, id AS hit_id FROM default.hits);

Ok. 0 rows in set. Elapsed: 0.223 sec.

 :) INSERT INTO hits VALUES ('2016-11-11', '2016-11-11 14:01:00', '2');

Received exception from server:
Not found column hit_id in block. There are only columns: date, timestamp, id

Any progress on this?

+1

same problem

+1

Any estimates ?

Is any updates or workarounds?

Generally VIEWS do work nice. My workaround is to avoid views, and generate long and ugly selects.

unfortunately my MATERIALIZED VIEWs dont work if they have  subqueries or JOIN, but with AS keyword they work normal. And for me using long and ugly selects isnt very good, there are too many big and long selects....

This is old clickhouse version and I did nothing. I am running Ubuntu 17.10. I remember though I installed libpoco yesterday, together with libpoco and related stuff. Could it be related somehow?

I tried to reappear with the above statement, But I can't reproduce this problem. Maybe this problem has been fixed? Can we close this Issue? @alexey-milovidov


Signature and Public Key not populated for intoto types

Description

When viewing the entry with log index 44413894 the Signature and Public Key entries aren't being shown, despite the raw body having a publicKey.

The public key is being populated for hashedrekord types:
https://search.sigstore.dev/?logIndex=74386860
Version
The intoto schema is v0.0.1 and was generated with
<EMAIL_ADDRESS>https://github.com/ossf/scorecard/blob/4ae4ba246c258be61ff92918eceef02fabcd5359/.github/workflows/goreleaser.yaml#L72-L81
Specifically this corresponds to provenance generated for the v4.13.1 release of Scorecard.

Looking at https://github.com/sigstore/rekor-search-ui/blob/7f3e7146c868d546637f0cbd14c718d482970a20/src/modules/components/Entry.tsx#L130-L132, my guess is because this is an intoto v0.0.1 type, rather than v0.0.2, so parsing fails, I guess silently. You can see the same thing with https://search.sigstore.dev/?logIndex=74413899, also v001. v002 correctly is parsed - https://search.sigstore.dev/?logIndex=10406797. I think we just need to add parsing for the v001 type.
Do you know where the provenance was generated? v001 is the older format, we shouldn't be using it anymore.


Missing builtin: bin

And no .format on strings which was the workaround I tried first...

bin doesn't seem too hard to implement! .format on strings does exist though, and works well for me - what are you trying specifically that doesn't work?

"{0:b}".format(val)

got an error that format didn't exist.

val = 3
print("{0:b}".format(val))

works for me, at least in terms of the function existing.
The format implemented by Transcrypt is fairly light-weight, and I don't believe it supports anything besides the basic {} and {x} syntax. That string is formatted just directly as {0:b}, but it does not error for me.
How are you compiling/running the code?

 It does exist (although gives the incorrect output as you say). It really should fail on unsupported formatting options I think.
My problem was that str.count didn't exist. I misread the error.

Yeah, it definitely isn't a full implementation of the Python format function, it does work for, well, I guess very basic formatting. Doesn't look like it checks for formatting options at all (implemented at https://github.com/QQuick/Transcrypt/blob/master/transcrypt/modules/org/transcrypt/javascript/builtin.mod.js#L1063).

The Python mini-formatting language was not implemented, mainly to keep the code lean.
If it's implemented, it (and similar functions) should probably in an extension module.
I feel I'll have to come up with a basic concept first.
If it's in an extension module anyone willing can work on it.
Currently I'll label it as a postponed enhancement.
This means I'll close it, but see #258


Bug: RejectedExecutionException

Exception:
Exception in thread "pool-2-thread-4136" java.util.concurrent.RejectedExecutionException: Task Thread[Thread-169720,5,main] rejected from java.util.concurrent.ThreadPoolExecutor@4fdf02ff[Shutting down, pool size = 5, active threads = 5, queued tasks = 12242, completed tasks = 24432]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)
at com.rarchives.ripme.ripper.DownloadThreadPool.addThread(DownloadThreadPool.java:34)
at com.rarchives.ripme.ripper.AlbumRipper.addURLToDownload(AlbumRipper.java:57)
at com.rarchives.ripme.ripper.AbstractRipper.addURLToDownload(AbstractRipper.java:131)
at com.rarchives.ripme.ripper.AbstractRipper.addURLToDownload(AbstractRipper.java:144)
at com.rarchives.ripme.ripper.AbstractRipper.addURLToDownload(AbstractRipper.java:157)
at com.rarchives.ripme.ripper.rippers.FlickrRipper$FlickrImageThread.run(FlickrRipper.java:253)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
The ripper keeps dieing on me when trying to rip a 160k image flicker account (https://www.flickr.com/photos/herosjourneymythology45surf/) full of massive pictures which are sometimes (but rarely) over 100mb (already changed the max file size to higher than this limit).  I've gotten the first 20k or so pictures from it, but when I restart the program it throws 20k warnings for duplicate pictures, then downloads a few hundred pictures, then errors out with the exception above.    The log there just stops being updated.

I get the same error as op in my own ripper if I don't add a 300ms delay after every single thread added to the pool. Why is this?


docs(style): add link

Checklist

[x] Fix linting errors
[x] Fix docs link error

Change information

modify some test case ,because the part of those test leads to CI build failure.
So it's temporarily removed


Download not compatible

Whenever I download any version of this mod, I always just get a bunch of folders with a .dll file inside all of them.
But, I need a .qmod file to be able to use this mod.

this is a pc mod, quest mods are unavailable at the moment.


PSA tests run indefinitely when CONFIG_TFM_IPC=y

Describe the bug
I tried to run the tf-m PSA test with CONFIG_TFM_IPC=y.
It simply reboots the test suite over and over, indefinitely.
Run with west build samples/tfm_integration/tfm_psa_test/ -p -b mps2_an521_nonsecure -t run -- -DCONFIG_TFM_PSA_TEST_STORAGE=y
It can be solved increasing the watchdog timer in ../modules/tee/tfm/psa-arch-tests/api-tests/platform/targets/tgt_dev_apis_tfm_an521/target.cfg but not sure if it's a solution or a workaround.
related to #31910

Hi @akiannillo, I believe this is the intended behavior of the sample.
The sample reboots after 5 seconds to demonstrate rebooting with TF-M.
The sys_reboot call is routed to TF-M, since the nonsecure app is not allowed
to perform the reboot directly.

Yeap, so I suggest we close this.


Router object not returned from client.boot since 3.0

Before the 3.x release, calling Client.boot returned the Router object as part of the callback:
// `onBoot` - Function (optional)
// returns data that was used
// during rendering as the first argument
// the second argument is the `router` created behind the scenes
// (only available while using react-router)
client.boot(/* client options object */, function onBoot(data, router) {

});

The changes to react-router 1.0 makes this obsolete, so the doc should be updated

+1 this request

fixed and released in v3.1.0. thanks!


Deprecate and remove AHC extension

We have vertx-http as the primary component for http and we should not have extensions for AHC - its not a good component in camel.

https://issues.apache.org/jira/browse/CAMEL-17667


Disable for filetypes

I am using vim-go already and it has it's own tag plugin. I was wondering if I can selectively disable this for certain file types, in this case go.

You can do this right now with g:gutentags_enabled_user_func, which is a callback function that gets passed the path to the current file, and is expected to return a boolean (1 if Gutentags should run, 0 is the file should be ignored).
I only took a brief look at vim-go but if you're talking about their including Tagbar, that's pretty much complementary to gutentags -- Tagbar launches its own ctags process for the current file only.
Last thing, Gutentags also has some support to recognize different projects (Gutentags tries to work at the project level, not the file level), and use different ctags executables. I don't know if gotags supports what Gutentags expects (the thing that's usually missing is the --append flag) but you could potentially also tell Gutentags to use Gotags for Go projects.

I'm not too good with vim script, do you think you can show me a sample function I can modify to get my desired behaviour?

I would look like this roughly, in your vimrc:
function! MyCustomGutentagsEnableFunc(path) abort
    return fnamemodify(a:path, ':e') != 'go'
endfunction

let g:gutentags_enabled_user_func = 'MyCustomGutentagsEnableFunc'


My nebra miners sync is going backwards

Hi, my miners sync status has started going backwards from 100%.
It now stands at 89% I haven't changed anything or changed my setup.
Kind regards.

This is not a support forum. Please contact<EMAIL_ADDRESS>


Can't put or delete more than 25 items at once

Describe the bug
Can't put or delete more than 25 items at once.
To Reproduce
Try and put more than 25 items.
Expected behavior
More than 25 items are put successfully.
Evidence
seeder/Seeder/Resource/Default (seederSeederC504247E) Failed to delete resource. 1 validation error detected: Value '{test-table=[]}' at 'requestItems' failed to satisfy constraint: Map value must satisfy constraint: [Member must have length less than or equal to 25, Member must have length greater than or equal to 1]
Additional context
None

From https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchWriteItem.html

The BatchWriteItem operation puts or deletes multiple items in one or more tables. A single call to BatchWriteItem can write up to 16 MB of data, which can comprise as many as 25 put or delete requests. Individual items to be written can be as large as 400 KB.

This has turned into a bit of a nightmare! 😂
I'll have to retire most of the core of this package as the SDK call route isn't fit for purpose.
I'm going to go with a custom lambda and writing setup/teardown data to S3.


Fix Elixir 1.11 compiles warning

Fixes some compiles warnings:

https://github.com/absinthe-graphql/absinthe_relay/issues/171

Generated ecto app

==> absinthe_relay

Compiling 22 files (.ex)

warning: clauses with the same name and arity (number of arguments) should be grouped together, "defmacro node/2" was previously defined (lib/absinthe/relay/node/notation.ex:23)

  lib/absinthe/relay/node/notation.ex:40

warning: clauses with the same name and arity (number of arguments) should be grouped together, "defmacro node/2" was previously defined (lib/absinthe/relay/node/notation.ex:23)

  lib/absinthe/relay/node/notation.ex:55

Thanks!

Thanks!


Missing CORS headers

Browsers issue a preflight OPTIONS request to see if the method and headers its about to send is allowed. These are still missing:
Access-Control-Request-Headers: authorization
Access-Control-Request-Method: GET, POST, PUT, OPTIONS, DELETE

The login page shows a network error badge if the preflight gets rejected
http://test.col.plus/user/login


UniverSeg should have a more general demo

Do I have to enter target_image, support_images and support_labels in UniverSeg's inference?
If I only have a target_image, how do I use UniverSeg? I think this is the reason why UniverSeg is not widely used.

Do I have to enter target_image, support_images and support_labels in UniverSeg's inference?
If I only have a target_image, how do I use UniverSeg? I think this is the reason why UniverSeg is not widely used.

UniverSeg is a work targeting few-shot learning. In few-shot learning, a support set is necessary, which could not be lacking.

@WangRongsheng we tackle the (very popular) situation where scientists have data for a problem, but would have a hard time designing an architecture and training the model for their problem -- while this may be doable to ML researchers (although still annoying for each task!) it's often beyond the resources of other scientists. In this scenario, some images are often segmented to give an idea of the protocol/region of interest. For example, in CT images some scientists might be interested in a certain lobe in the lungs and segment a few images in that -- these become the support images and support labels. These can now be used to segment a ton of other images the scientists might have.
This is a very different scenario then models you may have heard of recently like SAM, where you don't need a support set but you need some interaction for each test image -- a bounding box or a click etc. These are very cool models, but quite burden-some for segmenting substantial datasets -- you could use them to create a support set and then use UniverSeg for the rest :)
@wxliii Thanks for answering!. While our architecture can and should be used for few-shot learning, our focus was specifically the situation where training or fine-tuning networks (as some do in few-shot learning) is not possible, not just few-shot! In some cases you may have a larger support set, and our architecture will still do quite well!

@adalca good jobs! I like it! Thank you for your answering


uploading vendor/yiisoft/yii2 packages

vendor packages for yii2/yiisoft dependency libraries

merge main-branch changes/upload vendor packages to master


0016 - 3Sum Closest

Description of the Problem
Given an array nums of n integers and an integer target, find three integers in nums such that the sum is closest to target. Return the sum of the three integers. You may assume that each input would have exactly one solution.
Example 1:
Input: nums = [-1,2,1,-4], target = 1
Output: 2
Explanation: The sum that is closest to the target is 2. (-1 + 2 + 1 = 2).

Constraints:

3 <= nums.length <= 10^3
-10^3 <= nums[i] <= 10^3
-10^4 <= target <= 10^4

Code
class Solution:
    def threeSumClosest(self, nums: List[int], target: int) -> int:

Link To The LeetCode Problem
LeetCode

@vJechsmayr can I work on this issue ?


Fixed #46: Allow "data-pjax" attribute with no value in handleClick function

Closes #46.

Could it be merged separately from main repo changes?

@samdark I think yes.

Thank you!


Displaying memberships as first-class objects

This is related to #1032, which asks for filtering of organizations by partnership levels.  It adds pages to display all partnerships and to view the details of a single partnership.
NOTE: workshops/urls.py now has organization/membership/add and memberships/add - should the former be removed?

I don't know - @narayanaditya95 can you please review as well?
https://github.com/narayanaditya95

:+1: but I don't know how to implement that.

@pbanaszkiewicz do you have time to finish this off for V1.9 ?

@gvwilson I can work on this and #1040 on Saturday.

@gvwilson here you go https://github.com/gvwilson/amy/pull/8 please merge -- this PR should be automatically updated.

@pbanaszkiewicz thanks - please give it a final look in case I broke anything in the merge?

@gvwilson Please merge amy#9.
@chrismedrela @pbanaszkiewicz Please review the changes I proposed.

One question, then it's good to merge as far as I can tell.

@chrismedrela @pbanaszkiewicz Please review the changes I proposed.

Do you mean reviewing your review? :-)

@gvwilson or @pbanaszkiewicz, could you also change workshops/templates/workshops/all_organizations.html:26 so that it links to membership_details instead of membership_edit?

I don't understand the request to link to membership_details - do you mean organization_details (since this is the organization page, and there isn't any reference to membership in it)?

I don't understand the request to link to membership_details - do you mean organization_details (since this is the organization page, and there isn't any reference to membership in it)?

Here we need to replace membership_edit with membership_details.

Thanks - I was looking in the wrong file blush

@narayanaditya95, could you do the final review?

@chrismedrela in general, changes look good. I have some small comments, but they shouldn't stop you from merging.
Notes:

you should not make one big commit instead of many smaller commits. 3e32876901682577503330c21febfb6173e7c47b is a good example of this case; you even described small changes in the commit message – they should become individual commits
usually you should refrain from making indentation changes to existing code.

@pbanaszkiewicz, ok, in the future, I'll create smaller commits and stop correct indentation in templates.


data.json too large

As per https://twitter.com/edent/status/1071440354027728897

data.json doesn't show all benches (only first 10k)
truncate isn't working
Total file size 2.5 MB!

Truncate is working. Shows up as \u2026

Output is gzipped to around 512KB.


Cassandra 16259 (trunk)

Fix the histogram merge of the table metrics
Fix the counting of cells per partition

You can have someone assume an idempotent close() and for some weird reason it gets called multiple times just as you found. Also in between those calls the env might have changed enough later calls succeed.
In any case I don't have a strong position here, just trying to future proof it. Feel free to ignore it.

Merged manually.


Library Support & Documentation

Documentation on the website http://nuproj.net/documentation/#packaging-files and the README file use the following syntax:
<ItemGroup>
  <Library Include="$(BasePath)HelloWorld.dll" />
    <TargetFramework>net40</TargetFramework>
  </Library>
</ItemGroup>

However, this syntax no longer works in the latest version of NuProj (0.10.4-beta). Either the documentation should be updated or support should be re-enabled for the Library tag.
Related: what is the proper syntax for this, now? This?
<ItemGroup>
  <Content Include="$(BasePath)HelloWorld.dll">
    <Link>lib/net40</Link>
  </Content>
  <Reference Include="HelloWorld.dll" /> <!-- should this be lib/net40/HelloWorld.dll? -->
</ItemGroup>

0.10.4-beta is very old. The latest is 0.10.48-beta-gea4a31bbc5, but it's only available on the https://www.myget.org/F/aarnott/api/v3/index.json feed so you have to add it to your nuget.config file. I would highly recommend you use the latest version.
Rather than use Library or Content, I recommend using project and assembly references. NuProj will automatically place the assembly outputs into the right folders.
The documentation is also probably rather old. I'm not sure I have access to update that.

CC: @terrajobst

Thanks Andrew, I'll switch to that version and  start using assembly references.


unable to setup audio (acer Chromebook 311 cb311-9h) (bobba)

when I run sof-setup-audio,  it's unable to locate package firmware-sof-signed

Try running: sudo apt update

it says all packages are up to date

add non-free to you sources.list and it will install


Adding error handling. Preventing 100% CPU usage.

Description
Stops processing if an error occurs and fails liveness probe after that.
This allows a quicker restart. Crashloopbackoff is used as exponential backoff mechanism.
Reduced complexity: Now the workers wait for the queue to be empty, not for the enqueuer to be finished(which in turn had waited for the queue to be empty.
When the enqueuer was finished it previously used 100% CPU. This is fixed.
Type of Change

[x] Bug Fix
[ ] New Feature
[ ] Breaking Change
[x] Refactor
[ ] Documentation
[ ] Other (please describe)

Checklist

[ ] I have read the contributing guidelines
[ ] Existing issues have been referenced (where applicable)
[ ] I have verified this change is not present in other open pull requests
[ ] Functionality is documented
[ ] All code style checks pass
[ ] New code contribution is covered by automated tests
[ ] All new and existing tests pass

LGTM, Thanks Niels


Switch from nodemon to node-dev

Yari uses node-dev and it seems to offer more flexibility in asynchronizing command runs, which would be great to reduce server restart times.

Closing because it isn't node-dev that offers the functionality I'm looking for.


Fix enum syntax highlighting

Not sure what to do with all these different files, but the regex should be correct.
Closes  #38

thanks


Some tweaks and shell prototype changes

Some low-level stuff, plus work on prototype.

Somehow a bunch of other stuff got in here ... not sure what happened. Will try to tease it apart.

I've had this happen, and I think pull --rebase and pushing to a new github branch has fixed it.


 I don't know what caused this but Faster HDT-SMP was working fine, I didn't download more mods when I was using it while playing skyrim. All of the sudden my game freezes during combat and crashes, and for a few weeks I didn't know what mod was causing this, until recently I found out it was Faster HDT-SMP. So when Faster HDT-SMP is enabled now and I try to load a quick save, a save, and an auto save my game freezes and stops responding when loading it. Since it freezes and stops responding it won't crash to desktop so I have to close it with task manager, which of course means the game doesn't generate crash logs when this happens.
Desktop: windows 10
Faster HDT-SMP: seems to be any version as I've tried 2.0.2-beta1, and 1.50.9-rc1
SKSE Version: 6.6.640
I don't think I have Cuda or AVX, and I've never chosen any options with them

Hmm, I have no way to debug this.
If it was on my machine, I would try to reproduce this while debugging, and see where it hangs.
But this requires to set up a dev environment. If you got the courage and the time, you can see my article  about this.
But an advice: quick saving breaks saves.
An easy way to avoid this is to use the option
RegularQuicksaves = true                # Makes quick saves into regular saves
in the Engine Fixes mod.


README of B-BitManipulation is blank

Description
The README file of B-BitManipulation folder is blank while it should contain details for all the related problems present inside the folder.

I'm a GSSOC'22 contributor and I can work on this issue. Kindly assign it to me.
Domain
README
Type of Contribution
Documentation
Code of Conduct

[X] I follow Contributing Guidelines & Code of conduct of LearnCPP.

/assign


add googledoc link

add a link to the google doc (only accessible to CNERG member) in the README

Thanks @bam241


Warning: Exceeding the minimal duration

For every subtitle block visible to the imscED user that exceeds the pre-configured minimal duration a warning is shown. This warning applies only to the "statically" pre-configured minimal duration and should not be confused with the duration necessary to read a subtitle block based on assumed reading speed. This later value is dynamically calculated and not statically set.

The requested functionality is implemented. UI will be implemented at a later stage, therefore the issue is kept open.

The UI part of this issue has moved to https://github.com/IRT-Open-Source/imsced/issues/110. Otherwise, the core requirement of this issue has been met and it can be closed.


OutputCaching: Support for x-vs-store-code

Short Description and Why It's Useful
Currently it's not possible to use the x-vs-store-code header and outputCaching because the caching only looks at req.url. This fixes that.
Which Environment This Relates To
Check your case. In case of any doubts please read about Release Cycle

[x] Test version (https://test.storefrontcloud.io) - this is a new feature or improvement for Vue Storefront. I've created branch from develop branch and want to merge it back to develop
[ ] RC version (https://next.storefrontcloud.io) - this is a stabilisation fix for Release Candidate of Vue Storefront. I've created branch from release branch and want to merge it back to release
[ ] Stable version (https://demo.storefrontcloud.io) - this is an important fix for current stable version. I've created branch from hotfix or master branch and want to merge it back to hotfix

Upgrade Notes and Changelog

[x] No upgrade steps required (100% backward compatibility and no breaking changes)
[ ] I've updated the Upgrade notes and Changelog on how to port existing Vue Storefront sites with this new feature

IMPORTANT NOTICE - Remember to update CHANGELOG.md with description of your change
Contribution and Currently Important Rules Acceptance

[x] I read and followed contribution rules

Not sure what changelog to put it in

@benjick just put Added OutputCaching: Support for x-vs-store-code -  @benjick (#3979) 😃

Thanks, didn't see the "[1.12.0-rc1] - UNRELEASED" at first, was probably looking in the wrong branch! 👍


[SPARK-18173][SQL] data source tables should support truncating partition

What changes were proposed in this pull request?
Previously TRUNCATE TABLE ... PARTITION will always truncate the whole table for data source tables, this PR fixes it and improve InMemoryCatalog to make this command work with it.
How was this patch tested?
existing tests

cc @ericl @yhuai @gatorsmile

Test build #67788 has started for PR 15688 at commit a9eec51.

Test build #67788 has finished for PR 15688 at commit a9eec51.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67788/
Test FAILed.

This lgtm

If we try to truncate a partition that does not exist, we just do nothing?

retest this please

Test build #67847 has started for PR 15688 at commit a9eec51.

Test build #67847 has finished for PR 15688 at commit a9eec51.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67847/
Test PASSed.

Test build #67871 has started for PR 15688 at commit 9045ccd.

@gatorsmile that's a good point. I checked with hive, the behaviour is:

if the given partition spec specifies all partition columns, throw exception if the partition not exist
if the given partition spec is partial, do nothing if the partition not exist

I have updated this PR according to this

Test build #67871 has finished for PR 15688 at commit 9045ccd.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67871/
Test FAILed.

retest this please

Test build #67882 has started for PR 15688 at commit 9045ccd.

Test build #67882 has finished for PR 15688 at commit 9045ccd.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67882/
Test FAILed.

retest this please

Test build #67887 has started for PR 15688 at commit 9045ccd.

Test build #67887 has finished for PR 15688 at commit 9045ccd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67887/
Test PASSed.

Could we add all the positive and negative cases in ExternalCatalogSuite, like the other APIs. Then, we can easily know whether the behaviors of InMemoryCatalog and HiveExternalCatalog are consistent.

Test build #68188 has started for PR 15688 at commit 8941e10.

Test build #68188 has finished for PR 15688 at commit 8941e10.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68188/
Test FAILed.

retest this please

Test build #68195 has started for PR 15688 at commit 8941e10.

Test build #68195 has finished for PR 15688 at commit 8941e10.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68195/
Test PASSed.

Merged build finished. Test PASSed.

cc @gatorsmile

Regarding the current code changes, LGTM
Like what @cloud-fan said in the code comment, we need to decide whether we should follow Hive.

Test build #68241 has started for PR 15688 at commit 4726228.

Test build #68241 has finished for PR 15688 at commit 4726228.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68241/
Test PASSed.

Thanks - merging in master/branch-2.1.

@cloud-fan can you create a follow-up pr to switch over to Hive's behavior?


Support full semver versions with -app-version flag

Is your feature request related to a problem? Please describe:
I have an app using fyne where I'm following semver versioning. I also use goreleaser to run fyne-cross as needed. However, when I try to build a prerelease of my app, fyne-cross fails as it doesn't understand a semver version with a pre-release tag in it like 4.1.1-SNAPSHOT-fef3bae. This breaks my build pipeline during development. I get the following error with the aforementioned version passed to -app-version:
invalid -appVersion parameter, integer and '.' characters only up to x.y.z

Is it possible to construct a solution with the existing API?
While I could easily fake a X.Y.Z version in the pipeline, that may cause confusion with generated artifacts. It would be much nicer if fyne-cross just support any full semver version.
Describe the solution you'd like to see:
Full support for semver versions (as per the semver spec) for -app-version flag.

Unfortunately this may not be possible. Some platforms don't support semver and we have to use a combination that will work everywhere. For example windows is "X.Y.Z.P" where we use the version and build id to construct the version number. Therefore adding more info would be lost on some platforms which feels worse...

Oh TIL. I might be misusing this flag then. I will rework my build pipeline. Thank you for the response! I will close this issue off.


Replace special entities with proper links

Thank you @rettinghaus. There seems to be at least another place with specail entities in https://github.com/music-encoding/music-encoding/blob/develop/source/modules/MEI.shared.xml#L3769

Yes, because this addressed in #1046.

Since #1046 is merged, this should be good to go now, @musicEnfanthen.

@bwbohl @ahankinson Could one of you do a second review?


BUG: Issues with math

[ ] discrete_dp

check if this is resolved by: https://github.com/QuantEcon/lecture-source-py/pull/165

Doesn't appear that #165 fixed this issue - I think we might not be able to combine lists and math environments on the site? It works in Jupyter

@mmcky,  this is fixed


Trouble creating coupon, provided signature does not match

Trying to create a coupon on a wordpress site through my api but getting this issue:
{"errors"=>[{"code"=>"woocommerce_api_authentication_error", "message"=>"Invalid Signature - provided signature does not match"}]}
The wordpress site is still being built and does not have https, and I can't even find where the "pretty permalinks" option is.
wp version: 4.3.1
woocomm version: 2.4.10
server is apache
php version is
initializing with the same as explained in setup section of the readme with no args.
I have tried revoking Api consumer secrete and key and placing new ones in to no change.
This was all working a month ago or so but it hasn't been used since..not sure what the difference is now.
Any thoughts? pretty terrible with wordpress so apologies in advance if this is a basic config problem.

Here is not a support channel and seems like it is not a bug... Please check or documentation: http://woothemes.github.io/woocommerce-rest-api-docs/


update gradle, add credits to readme, eliminate assertNotNull, eras…

…e unused binding adapter, rename utility class etc

@OyaCanli  firs thing is that I see the app is not building properly because some files seems to be missing:

@rajtheinnovator I checkout this branch again and it works fine on my side. If it is only the databinding error, may be it can be fixed with a clean-rebuild. Did you already try that?

@OyaCanli I've done that but it didn't help. Maybe you can check once more by cloning your project from your Github fork in a new directory on your machine and see how that goes

@OyaCanli I myself tried cloning the branch again and this time it seems to work just fine. Probably an issue with build or cloning it was. Please ignore my previous comment. We're good to go with merge.

Yeap, I have just tried to download from remote and it worked fine. Databinding can sometimes create build errors. Especially, with our project structure, even if we switch branches, I see it keeps build folders of other branches. And may be you already know but the clean from the top menu doesn't seem to clean subfolders, so we need to clean from the gradle menu.

@OyaCanli yeah. That's true. Most probably a build issue I think. And now that it looks all good without any doubt remaining, let's merge this PR. Thanks for the extra effort :+1:  :grinning:

@rajtheinnovator I figured this out: it is the auto-converter who converts the recyclerview adapter with internal constructor, because the java version has a package-private constructor. I had made it like this because lint was showing "access could be weaker" warning, since adapter is in the same package with the fragment that uses it. Anyway, since kotlin doesn't have package private it converted it to internal.
I don't think it has much of an importance. We can make the class internal or we can keep it public..


Exception from Delayed::Job not reported if job fails to deserialize

If Delayed::Job fails to deserialize a job it throws DeserializationError. This happens early in the process and the hook lifecycle.around(:invoke_job, &Delayed::invoke_job_callback) is not called.
Looking at the code it is rescued here https://github.com/collectiveidea/delayed_job/blob/1e679174b513dc711f7d764c8bd9d4f5c6495276/lib/delayed/worker.rb#L235-L237 and the only hook it calls is "failure".
Do you have any idea how we can account correctly for this case?

@adamniedzielski do you know the reasons of the DeserializationError? Do you think that has a relation with the gem itself? Or is just that you'd like to report this errors?

@jondeandres Thanks for your response!
The DeserializationError comes from the Delayed Job and has no relation with the Rollbar gem. I would like this ezception to be reported by Rollbar, because it prevents the job from executing and this is something I definitely want to be aware of :smile:

Quick way to reproduce the problem:
"abc".delay.first(->{})
Delayed::Worker.new.run(Delayed::Job.last)

I think that now I understand why this gem uses around(:invoke_job) rather than after(:error) or after(:failure) - the exception is not available for these callbacks.
In order to have a basic reporting for jobs which failed even before the first execution we would have to add something similar to:
class RollbarPlugin < ::Delayed::Plugin
  callbacks do |lifecycle|
    lifecycle.around(:invoke_job, &Delayed::invoke_job_callback)
    lifecycle.after(:failure) do |job|
      ::Rollbar.error("Delayed::Job ID=#{job.id} failed to execute") if job.attempts.zero?
    end
  end
end

Hey @adamniedzielski, thanks for this, we'll try to release a new version with this soon.

Thanks @richvorp! 💚 💯 🎉

Really hope to get this released asap, delayed job failure is high in a server under my monitor.


ETL client restarts every time the WSGI file is run

Run tests based on this:

Move the Data Streaming Client to the __init__.py file to see if it prevents the restarting of the ETL client.

Test this by first trying to do a print function in the __init__.py file and observing if it is printed during the development server restart.

Set DEBUG = False in the production environment

Run tests based on this:

Move the Data Streaming Client to the __init__.py file to see if it prevents the restarting of the ETL client.

Test this by first trying to do a print function in the __init__.py file and observing if it is printed during the development server restart.

The test with the print function in the __init__.py didn't work, because the __init__.py file is run when the web_app module is imported.

Set DEBUG = False in the production environment

In addition to this, I set up a robot to make a HEAD request to the API on PythonAnywhere every 5 minutes using https://uptimerobot.com/dashboard#mainDashboard because of this thread: https://www.pythonanywhere.com/forums/topic/13893/


Force bundler 1.10.6 in TravisCI

Version 1.11 has a bug preventing gems to be properly installed with jruby
Follow up from PR #33
See https://github.com/travis-ci/travis-ci/issues/5290

@jcarres-mdsol @adriancole

good find!


user_groups: Update code for changing group level settings.

Earlier while changing group level group based settings there was no check if the new value for setting is same as the current value.
This PR adds this check now a setting value will be only changed when it is not equal to present value.

This PR is a followup work for changes in #26243.

Screenshots and screen captures:

Self-review checklist

[ ] Self-reviewed the changes for clarity and maintainability
(variable names, code reuse, readability, etc.).

Communicate decisions, questions, and potential concerns.

[ ] Explains differences from previous plans (e.g., issue description).
[ ] Highlights technical choices and bugs encountered.
[ ] Calls out remaining decisions and concerns.
[ ] Automated tests verify logic where appropriate.

Individual commits are ready for review (see commit discipline).

[ ] Each commit is a coherent idea.
[ ] Commit message(s) explain reasoning and motivation for changes.

Completed manual review and testing of the following:

[ ] Visual appearance of the changes.
[ ] Responsiveness and internationalization.
[ ] Strings and tooltips.
[ ] End-to-end functionality of buttons, interactions and flows.
[ ] Corner cases, error conditions, and easily imagined bugs.

Looks good to me. @timabbott this should be quick to merge.

Merged, thanks @Ujjawal3!


Invalid response to ConfirmPrompt causes exception

Version
Microsoft.Bot.Builder.Dialog 4.1.5
Describe the bug
Choosing invalid response to ConfirmPrompt throws KeyNotFoundException in OnContinueDialogAsync. The style of the prompt doesn't seem to matter (at least I tried SuggestedAction, List and Inline).
"System.Collections.Generic.KeyNotFoundException: 'The given key was not present in the dictionary.'"
To Reproduce

PromptAsync with a ConfirmPrompt
In the emulator, when presented with the Yes/No options respond with an invalid string

Expected behavior
Similar to the ChoicePrompt, when this happens the user should just be presented with the same question until they select a valid option (Yes, No, 1, or 2).
Additional context
In version 4.0.7 we were running into the issue where selecting an invalid choice in ChoicePrompts or ConfirmPrompts were causing the options to duplicate. In 4.1.5 that appears to be fixed in ChoicePrompts but now the ConfirmPrompts throw this exception.
[bug]

@mhensinger thanks for submitting this issue.
This looks like a regression in 4.1.5. The problem is the logic now attempts to accept a number as a choice, however, it can fail with an unsafe dictionary lookup if the locale is not set on the incoming Activity.
In my testing with the emulator the inbound locale was string.Empty and the unsafe lookup failed. The code should be using a Try style lookup.
We will have a fix in the packages published on Dec 15th.
A simple temporary workaround that might help if you need to get unblocked is to explicitly set the Locale on the inbound Activity. For example, I added the following:
public async Task OnTurnAsync(ITurnContext turnContext, CancellationToken cancellationToken = default(CancellationToken))
{
    turnContext.Activity.Locale = Culture.English;
    // etc.


Feature request: reverse proxy login support

I have traefik running as a reverse proxy in front of inventree.   The reverse proxy handles the AUTHENTICATION (and maybe AUTHORIZATION) before handing it off to any server running behind it.
In this case I have https://inventree.{mydomain}.com  pointing to the reverse proxy.  The reverse proxy authenticates with google, and then sets a cookie for auth.   The proxy then checks the presence of a valid auth cookie before handing the traffic off to inventree running in a docker container.  (This is all done with traefik, btw)
In this scenario I have two login pages.  First for the google login page, and then the second for the inventree login page.
So the reverse proxy will add a header to the request, in my case "X-Forwarded-User<EMAIL_ADDRESS>It would be nice to have something along the lines in the config:
INVENTREE_REVERSE_PROXY_LOGIN=True
INVENTREE_REVERSE_PROXY_HEADER="X-Forwarded-User"
This would allow easier login for people/corporations that already have a reverse proxy system setup using LDAP/SSO/etc.  And Inventree wouldn't have to deal with complex auth mechanisms directly at all.
Lastly Inventree would need something like this:
INVENTREE_REVERSE_PROXY_AUTO_CREATE_USER=True/False
If True, an unknown user would be granted access (because both AUTHENTICATION of the user and AUTHORIZATION of the user accessing Inventree have been handled elsewhere) and Inventree should just add the new user into the User's table.
If False, AUTHENTICATION is handled by the reverse proxy, but AUTHORIZATION of the user accessing Inventree is controlled by the Django User table.  If a user exists in table and is_active=True then the user is granted access.  Otherwise the user gets a 403 Forbidden Page.
This way you can do fine grained control of user AUTHORIZATION in Inventree, yet still rely upon external AUTHENTICATION to validate who the person is.

sure, this seems reasonable and has been requested before:

https://github.com/inventree/InvenTree/issues/625
https://github.com/inventree/InvenTree/issues/634

I'm not sure how to implement this - have you any experience on configuring django in such a fashion?
We'd also need to clearly document this :)

Yes.  (Sadly)
So yeah... in 2021 it's fugly. It should be simple, but it's not.  There are companies making billions of dollars on this.
So what I'm suggesting is probably the simplest case possible for inventree.
Authentication is either handled by django's login page, or via the X-Forwarded-User header.   Then if you want something more complex than the simple login page, it's up to user to implement it and make sure the logged in user is put into the X-Forwarded-User header (or whatever header is configured above).
Fortunately Django has support for this built in though the RemoteUserMiddleware and RemoteUserBackend classes.  YAY!
This is worth a careful read through.
https://docs.djangoproject.com/en/3.2/howto/auth-remote-user/

@rcludwick is this feature something you'd be willing to spend some time on? I'm happy to support it but this is really not my area of expertise!

I'll look at it in my spare time.  It's mostly "configuration" thankfully.

@rcludwick did you find time to do this and would you be willing to share your findings for the docs?

No, I didn't get time to do this.
Having dealt with auth before, it comes down to two different ways to do
authentication in Django.
First is the middleware.  This looks at the request coming in, figures out
who the user is and then attaches the user to request.user and passes the
request on.  This is the way Django web login works.
The second is through authentication and authorization classes on DRF for
the apis.  This is usually through an http header.
In my case I have inventree sitting behind traefik-forward-auth which
handles the auth for me, and can add a header "proxy-forwarded-for" which
holds the name/email of the authenticated user.
So all requests from a browser have a cookie from traefik-forward-auth
which it intercepts and validates.  So any requests that make it through to
inventree should be considered authenticated.
One reason to do this is that I don't have to expose the inventree login
pages to the internet directly, but instead are redirected to google's
oauth2 service.
So in my configuration I need a middleware in Django that handles the
"proxy-forwarded-for" header properly.  This is the part I was going to
look at.
Many companies will have some kind of custom auth proxy in this
configuration that spits out a cookie or a header that Django can look at.
That should probably suffice for a good chunk of people.
So all requests that come from a browser will have a cookie attached to it
that provides auth.  You can either use the django login system or roll
your own that provides a header for inventree.
Great!  Now what about the API?
So this gets more complex, because much of this doesn't necessarily involve
a human in the loop, nor a web browser capable of storing and forwarding
cookies.
But on the other hand, DRF can use the default Django session cookies to
auth web browser api calls.
But you can also specify custom authentication and authorization classes in
a DRF view that are different than middleware auth.
Some people can put their API behind a proxy that will handle auth and
provide throttling and metrics.  Others may just not care and expect
everyone to use a web browser when using the API.
In my case traefik-forward-auth won't authenticate machine traffic, because
I have to login through Google via a web page.  Any apis I want to use with
a machine client will need to be whitelisted with traefik-forward-auth, and
then some kind of authorization header needs to be used and handled by
inventree directly.
Often people use JWTs (python-jose e.g.) that are generated by another
server. but this also could be done with a random string generated on
inventree passed to the machine client by the user.
So the key takeaway is that auth needs to be configurable.  And API auth
could be different than the browser.
You don't have to go crazy, but just provide some simple schemes for the
majority of use cases, and then if some company wants you to integrate with
their homegrown auth server, then charge them $200/hr.
On Thu, Aug 19, 2021, 3:56 PM Matthias Mair @.***>
wrote:

@rcludwick https://github.com/rcludwick did you find time to do this
and would you be willing to share your findings for the docs?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/inventree/InvenTree/issues/1693#issuecomment-902276071,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAL67I6ULUY3KWY3ZN3QC33T5V42BANCNFSM47APK6ZA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email
.

@rcludwick thanks for the input, I will try to implement something if I find time. Your input would be appreciated if you find time 👼.

In summary, you just need two things.
A way to authenticate browser requests in middleware using a http header
(proxy_forwarded_for=user_or_email).
And... Some kind of token auth for API machine access, this could be a
token in the db with a foreign key back to the user.  Also default
permissions for unauthed API users should be configurable.  Many people
might want read only, others might want a 403.
Authlib is a good library to look at, particularly their Django
implementations for tokens .
This stuff is a 🐇🕳.   I would avoid anything more complex than the
simplest API auth mechanism for now which is really just a shared secret.
Do jwt later until after you get a pluggable auth layer built.
On Fri, Aug 20, 2021, 2:22 PM Matthias Mair @.***>
wrote:

@rcludwick https://github.com/rcludwick thanks for the input, I will
try to implement something if I find time. Your input would be appreciated
if you find time 👼.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/inventree/InvenTree/issues/1693#issuecomment-902936473,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAL67I5DPDEYLGP7BUQOYCTT522OVANCNFSM47APK6ZA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email
.

This is similar to #625 but more specific about how it could be implemented.

I played around with this because a client needed it. It is very easy to do with the authentication backend and middleware settings in the config.yaml. It was ~40 LoC but really depenendet on how the infra is built out. Could be very dangerous if the admin setting it up does not know their stuff.
@SchrodingersGat should we still implement a solution for this? Maybe a configurable header that is used as a user reference?

I'm happy to provide a reference example but not implement it by default. A clear warning should be added to the documentation that "all care should be taken" but the implementing admin.

So what should I sketch out? A simple proxy set auth-header, LDAP or something else? Google etc. are already implemented in the SSO package.

@matmair I'm happy for you to make a suggestion based on what you think is sensible. This is straying very far from my area of knowledge.
My key requirements would be:

It is all disabled by default
It is reasonably difficult to 'accidentally' turn on
It is documented with a very clear caveat that the admin must know exactly what they are doing.

@matmair happy to bump this one to 0.7.0?

@rcludwick are you still interested in this?
@matmair are you still keen to implement?

@SchrodingersGat yes. I planned with a longer release window and wanted to do it in the Easter break

@rcludwick if you have time feel free to look at the PR

I'm having some trouble getting the proxy auth working and I was wondering if I'm missing a setting somewhere. I'm using traefik + traefik-forward-auth for SSO, which passes the X-Forwarded-User header to an nginx container (for serving static files) running in front of inventree.
I have
remote_login_enabled: True
remote_login_header: X-Forwarded-User

in my inventree.yml file, and the nginx config was copied from here. I've also tried the INVENTREE_REMOTE_LOGIN=true and INVENTREE_REMOTE_LOGIN_HEADER=X-Forwarded-User, but in both cases I keep getting redirected to the login screen.
Do I need to add to the authentication_backends and middlewares in the configuration yaml as well, or is there another configuration setting I'm missing?


pkg/rpcserver: tolerate the kernel directory absence

For fuzzing, we don't strictly need the kernel directory or the kernel object file. We just need a disk/kernel image.

We provide kernel_obj during syzbot builds, right? And it's generally assumed to be provided.
I think it's better if the test will be more realistic and create real config, rather than have a non-realistic test and add more production code to workaround that.

The problem is not in the presubmit test, the problem is that syzkaller has just stopped working without kernel_obj, while it did not require it before.

Okay, but I am surprised we observe the difference anywhere. All our configs have all dirs filled.