triggering UB without unsafe

use cubecl::prelude::*;
use zerocopy::AsBytes;

#[cube(launch)]
fn add_kernel<F: Float>(lhs: &Array<F>, rhs: &Array<F>, output: &mut Array<F>) {
    let pos = ABSOLUTE_POS;

    //no bound check
    output[pos] = lhs[pos] + rhs[pos];
}

fn main() {
    let dev = cubecl::cuda::CudaDevice::default();
    let client = cubecl::cuda::CudaRuntime::client(&dev);

    let lhs_buff = client.create([0f32, 1f32, 2f32, 3f32].as_bytes());
    let rhs_buff = client.create([4f32, 5f32, 6f32, 7f32].as_bytes());

    let output = client.empty(4 * core::mem::size_of::<f32>());

    add_kernel::launch::<F32, cubecl::cuda::CudaRuntime>(
        &client.clone(),
        CubeCount::Static(1, 1, 1),
        CubeDim::new(16, 1, 1),//here
        ArrayArg::new(&lhs_buff, 4),
        ArrayArg::new(&rhs_buff, 4),
        ArrayArg::new(&output, 4),
    );

    let output_bytes = client.read(output.binding());
    let output = f32::from_bytes(&output_bytes);

    println!("{:?}", output);
}

This code triggers UB.
Idk how you might ensure soundness within kernel, but IMO it should not be possible to do without unsafe.
$ compute-sanitizer ./target/debug/cubecl_testing
========= COMPUTE-SANITIZER
========= Invalid __global__ read of size 4 bytes
=========     at kernel+0x150
=========     by thread (8,0,0) in block (0,0,0)
=========     Address 0x6e6000020 is out of bounds
=========     and is 1 bytes after the nearest allocation at 0x6e6000000 of size 32 bytes
=========     Saved host backtrace up to driver entry point at kernel launch time
=========     Host Frame: [0x2efadf]
...

@JRazek We are currently migrating every kernel in Burn to ensure they perform proper bounds checks. We can't automate this process easily because it would significantly impact performance (by up to 30% or more), which isn't viable in a high-performance computing scenario. It's generally better to use an early return in a kernel when the unit shouldn't perform any computation, but this approach doesn't work when you have sync_units in a cube. When the cube_dim and cube_count match, no bounds checks should be necessary.
Branching in GPU code is very expensive, so the best possible approach would be to trigger an error in debug mode using #[cfg(test)]. However, I'm not sure how we could implement error handling in GPU code either 😅.
We could adopt the same API as wgpu, where you can compile a module with create_shader_module_unchecked, which is an unsafe function. I'm gonna look into that.

@nathanielsimard
I get the limitations, but it still should not be possible to cause UB without explicit usage of unsafe.


Fixed typos in InferenceClient hyperlinks

I reviewed the newly added Image-Text to Text task and identified some typos in the hyperlinks. This PR fixes those typos.
@merveenoyan

Ah missed that, thank you @hanouticelina!

Hi @sergiopaniego thanks a lot for your PR! the docs/api-inference/tasks/*.md files are auto-generated by running the generate.ts script in scripts/api-inference, so we should update the script instead of modifying the .md files directly.
the issue comes from these lines: scripts/api-inference/scripts/generate.ts#L443-L444 scripts/api-inference/scripts/generate.ts#L514-L515 the current replace() method only replaces the first occurrence of the pattern (that's why "image-text-to-text" becomes "image_text-to-text" instead of "image_text_to_text"). to fix this, we can use replaceAll() instead:
- taskSnakeCase: baseName.replace("-", "_"),
- taskAttached: baseName.replace("-", ""),
+ taskSnakeCase: baseName.replaceAll("-", "_"),
+ taskAttached: baseName.replaceAll("-", ""),

this change will also ensure all huggingface.js package reference links are properly formatted! could you update the script with these changes instead? No need to run the generate script yourself as we have an automated PR that regenerates the documentation every day. Let me know if you need any help with this! 🤗

Wow 😮, thanks a lot for the in-depth explanation @hanouticelina!!!
I've made the updates following your advice. I hope they meet the requirements 🤗
Thanks, @pcuenca too!!


Screen Capture with app regions and OCR tool

Description of the new feature / enhancement
Screenshot regions of an application based on selectors and extract text from screenshots using an OCR
Scenario when this would be used?
Extract text from images for notes and capture regions of an app quicker without having to align the screenshot with the app.
Supporting information
Features taken from https://getsharex.com/

/dup #4371


Change the converter to create a swagger to structured data mapper

Swagger from Kubernetes uses a lot of different informations.
It would be nice if we could:

from .definitions extract only fields containing x-kubernetes-group-version-kind
Create a struct that could be used across all the program:

type KubernetesResource struct {
   group: Group,
   version: version,
   kind: Kind,
   name: pluralName,
   description: description,
   deprecated: true
}

// the string identifier from the map shoud be "GV.Resource"?
var AllResources map[string]KubernetesResource

This map can be used in the flow:
-> Get all deprecated and check if they exist in the API Server (for, if v.deprecated)
-> Walk through all the APIServer (#1) and verify if this Kubernetes object exists in the map

Resolved in v0.1.0


Incorrect version information is consumed while invoking build scripts through github actions workflow.

Althought, we changed the script "gen_versions_txt.sh" to accept a tag
rather than a branch, this change is not sufficient.
This script generates the right version file based on a tag, but
function get_from_kata_deps does not use this, and ends up using the
master branch instead. This is because this function looks at an env
variable called $BRANCH and ends up using master branch if the variable
is not defined.

cc @egernst
Bug was discovered while adding a github actions job for nemu for 1.9 branch. Instead of looking for version in 1.9 stable branch,  it ended up looking at the master branch version file for which no nemy information was available.


Key and Value not reset

When looping through table rows, the $key and $value variables are not reset.
This can cause a $key's $value to be set from a previous $keys value.
This is the case with DHCP Server

#6
commit 643ae45


openssl 1.1.0 binding issue

Openssl made the jump to 1.1.0 they removed some symbols. Which makes it impossible to use on arch linux without some non sustainable hacks.
CRYPTO_num_locks' CRYPTO_set_id_callback'
CRYPTO_set_locking_callback' SSL_get_ex_new_index'
SSL_library_init' SSL_load_error_strings'
SSLv23_client_method' SSLv23_server_method'
deimos is out of date and without git version tags.
I looked into creating bindings with dstep, but thats gone be a real pain.
IMHO it might be easier to create the definitions we need from openssl ourself and drop the deimos binding. Comments?

Openssl made the jump to 1.1.0 they removed some symbols. Which makes it impossible to use on arch linux without some non sustainable hacks.

There are two simple "hacks" to solve the problem temporarily:

versions "Have_Botan"
versions "VibeNoSSL" (use Nginx etc. as reverse proxy)

requestHTTP might need https as well. I'm not sure how to use nginx for that.
At some point we need update openssl or drop it. And dropping it is not really an option IMO.

Are there any known examples of how to properly achieve the same effect that those functions have with 1.1.x? To be honest that's the main roadblock - documentation is awful, code is awful and it's more or less impossible to get the initialization sequence right without an example.
Updating the Deimos headers indeed doesn't sound like fun. The huge mess that the OpenSSL code base is presumably makes that quite a PITA. BTW, I had to clone the Deimos repository anyway to be able to make version tags a reality. There have been no responses in the main repository.

yeah openssl init looks like a big mess.
void SSL_load_error_strings() {} does not do anything anymore for example
I might have some time on Sunday to see if I can get dstep+vim to get a new version up and running

There have been no responses in the main repository.

Just ping someone (with the necessary privileges), sheesh :)

FWIW, I would certainly be thankful for an update to the Deimos bindings, since I use those in ae! Right now I have to link DFeed against openssl 1.0.0 explicitly.

Duplicate of #1748


fix: outputPath.replace is not a function

⚠ I didn't test this. But I guess it should work.
close #13
cc @fregante

I tested the diff and the build now works, but I haven't verified that the code produced is correct as I'm still working on my build. Thanks for the quick PR!

Thanks for the PR!


Update implementation of file-backed durable slab storage

Fix check related configuration for USE_PMEM continue of 14fcd34
Corrects teardown when wrong pool was used:
In situations when pm pool creation failed, teardown process causes
SIGSEGV due to accessing empty datapool structure. Fail may be caused
by wrong size, path or file permissions. This commit skips heapinfo
teardown in such situation.
Add reserved only items after reset to freeq
Additional fixes for static analysis issue

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.2 out of 3 committers have signed the CLA.:white_check_mark: michalbiesek:white_check_mark: jschmieg:x: kasiawasiutaYou have signed the CLA already but the status is still pending? Let us recheck it.


docs: update git providers

This pull request contains updates to the git providers section. Let me know if you would also like to have the rest of identity providers in this section, or should they be in its own?
Fixes https://github.com/daytonaio/enterprise-docs/issues/128

@stefanicjuraj can you please update so we can merge? thanks


Sign+Encrypt and Decrypt+Verify

Hi, ramuta. You can see this https://username1565.github.io/pgp/
and update your repositary. Code here: https://github.com/username1565/pgp
Maybe ECC encryption can be added there.

Great, thanks!


Can't delete device with Rear Port connection yet

I'm referencing the #2685 since I'm facing the same problem on deleting the item even after I upgraded to 2.5.1.

My fault, forgot to restart after upgrade: supervisorctl restart netbox


Bug: Logged errors do not include own properties.

Expected Behaviour
I expect powertools to log all enumerable properties on an error. It's common to include a code, and extra information about an error.
If you log a regular object or classs it's properties will be logged, but if the class extends Error only stack, message, location, name, and cause will be logged.
Current Behaviour
logger does not log any of the errors enumerable properties
Code snippet
import fs from 'fs/promises'

import { Logger } from '@aws-lambda-powertools/logger'

class MyData {
  constructor(public readonly field: string) { }
}
class MyError extends Error {
  constructor(public readonly field: string) {
    super(field)
  }
}

const logger = new Logger();

// node's built in errors include code, syscall and other properties
// powertools doesn't include these when printing the error object.
await fs.readFile("not a file").catch(err => {
  logger.error("fs error", {
    bare: err,
    spread: { ...err },
    json: JSON.parse(JSON.stringify(err)),
  })
})

const myError = new MyError("field");
logger.error("fs error", {
  bare: myError,
  spread: { ...myError },
  json: JSON.parse(JSON.stringify(myError)),
})

const myData = new MyData("field");
logger.error("fs error", {
  bare: myData,
  spread: { ...myData },
  json: JSON.parse(JSON.stringify(myData)),
})

Logging an error thrown by node's built in fs module, see that bare, which is how most people would log an error, is missing errno, code, syscall, and path metadata.
spread, and json include the metadata, but not the stack trace, message, or location. spread and json show that these are enumerable properties on the object.
{
    "level": "ERROR",
    "message": "fs error",
    "service": "service_undefined",
    "timestamp": "2024-03-07T00:04:23.697Z",
    "bare": {
        "name": "Error",
        "location": "",
        "message": "ENOENT: no such file or directory, open 'not a file'",
        "stack": "Error: ENOENT: no such file or directory, open 'not a file'"
    },
    "spread": {
        "errno": -2,
        "code": "ENOENT",
        "syscall": "open",
        "path": "not a file"
    },
    "json": {
        "errno": -2,
        "code": "ENOENT",
        "syscall": "open",
        "path": "not a file"
    }
}

Here's a simple custom error and it has the same problem.
{
    "level": "ERROR",
    "message": "fs error",
    "service": "service_undefined",
    "timestamp": "2024-03-07T00:04:23.699Z",
    "bare": {
        "name": "Error",
        "location": "test.ts:27",
        "message": "field",
        "stack": "Error: field\n    at <anonymous> (test.ts:27:17)"
    },
    "spread": {
        "field": "field"
    },
    "json": {
        "field": "field"
    }
}

This is logging a normal class that doesn't extend Error and you can see that bare includes the properties I would expect.
{
    "level": "ERROR",
    "message": "fs error",
    "service": "service_undefined",
    "timestamp": "2024-03-07T00:04:23.701Z",
    "bare": {
        "field": "field"
    },
    "spread": {
        "field": "field"
    },
    "json": {
        "field": "field"
    }
}

Steps to Reproduce
The script included above reproduces with powertools 1.18.1 or 2.0.2 on node 20.
Possible Solution
Include enumerable properties by spreading the err object.
Powertools for AWS Lambda (TypeScript) version
2.0.2, 1.18.1
AWS Lambda function runtime
20.x
Packaging format used
npm
Execution logs
See above.

Hi @everett1992, thanks for opening the issue.
The format that you described as a bug aligns with the one we specify in the docs.
We purposefully standardize the format of an error whenever encountering one during the parsing and normalize the structure to a common denominator.
With this in mind, the behavior you're seeing is not a bug.
All Error instances and derivatives are passed through the formatError() method present in the LogFormatter class.
Using a custom log formatter you can customize the format of the error by overriding the formatError() method to include all the info you need.
Here you have an example of how to use a custom log formatter.
At the moment we only expose the LogFormatter class, this means you'll have to also provide an implementation for the formatAttributes() method which is abstract. If you'd like us to expose the default PowertoolsLogFormatter so that you can use all the default settings while still being able to selectively override some aspects, please let me know and I'll include it in the next release.

Where is it specified in the docs? I read that section but I don't see any claim that only location, stack, name, and message are logged. The  examples only have those fields, but do do the errors.
I think 'field', or 'code', 'path', 'errno', 'syscall' should be logged along with the location, message, name and stack.

Hello everybody! I was reading this issue and thought it was important to give my impressions here as we added this same feature in Powertools Python recently.
Firstly, I must say that I agree that Powertools Typescript should add the same capability as Python, that is, in case of exceptions, create a specific field with the entire stack trace to ease discoveries when searching in respective logs. On the other hand, I think that we should not add fields that are not present in all exceptions, as we may incur an increase in the storage of client logs with information that may not be useful for that specific client and that is optional. I was reading the NodeJS errors documentation (https://nodejs.org/api/errors.html#class-systemerror) and saw that fields like address, path, and port are optional. In my opinion, the exception must add the mandatory fields and the customer can customize through a custom log, just as @dreamorosi  showed in the example.
As additional context, I personally really like Sentry's exception-catching implementation, and in Python we try to follow the same stack trace format that they implement - of course removing fields that doesn't matter for us.
They don´t log some specific exception fields and you can enrich this by adding additional metadata to scope (custom format in our case).
Example
app.py
import sentry_sdk

sentry_sdk.init(
    dsn="....",
)

try:
    open("leo.txt")
except Exception as e:
    sentry_sdk.capture_exception(e)

log.json - no personal information was disclosed
{
    "event_id": "7b36ed6f32874c249e0574197cb23e43",
    "exception": {
        "values": [
            {
                "type": "FileNotFoundError",
                "value": "[Errno 2] No such file or directory: 'leo.txt'",
                "stacktrace": {
                    "frames": [
                        {
                            "function": "<module>",
                            "module": "__main__",
                            "filename": "leo.py",
                            "abs_path": "/home/leandro/DEVEL-PYTHON/tmp/sentry-error/leo.py",
                            "lineno": 19,
                            "pre_context": [
                                ")",
                                "",
                                "",
                                "",
                                "try:"
                            ],
                            "context_line": "    open(\"leo.txt\")",
                            "post_context": [
                                "except Exception as e:",
                                "    # Alternatively the argument can be omitted",
                                "    sentry_sdk.capture_exception(e)"
                            ],
                            "in_app": true,
                            "vars": {
                                "__annotations__": {},
                                "__builtins__": "<module 'builtins' (built-in)>",
                                "__cached__": "None",
                                "__doc__": "None",
                                "__file__": "'/home/leandro/DEVEL-PYTHON/tmp/sentry-error/leo.py'",
                                "__loader__": "<_frozen_importlib_external.SourceFileLoader object at 0x7f5112e8dc10>",
                                "__name__": "'__main__'",
                                "__package__": "None",
                                "__spec__": "None",
                                "sentry_sdk": "<module 'sentry_sdk' from '/home/leandro/DEVEL-PYTHON/tmp/sentry-error/.venv/lib/python3.9/site-packages/sentry_sdk/__init__.py'>"
                            }
                        }
                    ]
                },
                "mechanism": {
                    "type": "generic",
                    "handled": true,
                    "meta": {
                        "errno": {
                            "number": 2
                        }
                    }
                }
            }
        ]
    }

}

Thank you

If you'd like us to expose the default PowertoolsLogFormatter so that you can use all the default settings while still being able to selectively override some aspects, please let me know and I'll include it in the next release.

That seems like a good compromise, but this still feels like an important feature or bug.
Here's examples from other loggers, they each include the error's properties.
-- powertools --------------------------------------------
{"level":"ERROR","message":"fs error","sampling_rate":0,"service":"service_undefined","timestamp":"2024-03-12T16:09:17.475Z","err":{"name":"Error","location":"","message":"ENOENT: no such file or directory, open 'not a file'","stack":"Error: ENOENT: no such file or directory, open 'not a file'"}}
-- winston -----------------------------------------------
{"err":{"code":"ENOENT","errno":-2,"path":"not a file","syscall":"open"},"level":"error","message":"fs error"}
-- bunyan ------------------------------------------------
{"name":"bunyan","hostname":"u0ac46b2ab35f52.ant.amazon.com","pid":44989,"level":50,"err":{"errno":-2,"code":"ENOENT","syscall":"open","path":"not a file"},"msg":"fs error","time":"2024-03-12T16:09:17.479Z","v":0}
-- pino --------------------------------------------------
{"level":50,"time":1710259757479,"pid":44989,"hostname":"u0ac46b2ab35f52.ant.amazon.com","err":{"type":"Error","message":"ENOENT: no such file or directory, open 'not a file'","stack":"Error: ENOENT: no such file or directory, open 'not a file'","errno":-2,"code":"ENOENT","syscall":"open","path":"not a file"},"msg":"fs error"}

@heitorlessa would like to hear your opinion on this, thanks.

This behavior is somewhat surprising. Most problematic however, is how it excludes the cause property of logged errors. https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Error/cause
These have been standard since Node 16 and the exclusion really hampers debugging.
A concrete example:
Let's say you're using powertools alongside the secretsmanager middy middleware. If permissions are not set on the KMS key or Secret resource to grant your lambda read/decrypt permissions, middy will throw an error with the following message: Failed to resolve internal values. It's opaque and that's all you'd see in the logs. However, we can see that middy actually sets the error cause property with more details.
Actual log observed (running on a local node process, so some lambda context values are not set):
{
    "cold_start": true,
    "function_memory_size": null,
    "level": "ERROR",
    "message": "example",
    "service": "service_undefined",
    "timestamp": "2024-05-10T09:28:17.412Z",
    "error": {
        "name": "Error",
        "location": "/Users/bilalq/example/node_modules/@middy/util/index.cjs:116",
        "message": "Failed to resolve internal values",
        "stack": "Error: Failed to resolve internal values\n    at getInternal (/Users/bilalq/example/node_modules/@middy/util/index.cjs:116:15)\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async secretsManagerMiddlewareBefore (/Users/bilalq/example/node_modules/@middy/secrets-manager/index.cjs:56:26)\n    at async runMiddlewares (/Users/bilalq/example/node_modules/@middy/core/index.cjs:126:21)\n    at async runRequest (/Users/bilalq/example/node_modules/@middy/core/index.cjs:86:9)"
    }
}

The cause property would show that the error was due to CredentialsProviderError and be much more useful.
Can we maybe at least get cause included in error logs?

Hi @bilalq - which version of Logger are you using? We support the cause property unless there's a bug or it's being specified differently than what you linked:

Ah, that was my mistake. Thanks for pointing it out. It seems that the service I was looking at happened to be on the 1.8.0 version of powertools. The latest version does indeed behave as expected with cause.

Thanks for confirming - and also for reviving this thread.
I just wanted to share that I have not forgotten about this and I am going to try to have a final answer in the next few weeks. Since my initial comment I actually got bit by this myself trying to log a custom error I made and not seeing its properties.
I'm now leaning towards including them but I still want to get the alignment of other Powertools runtimes before moving forward.


Placeholder image blurry?

Check List
Thanks for considering to open an issue. Before you submit your issue, please confirm these boxes are checked.

[x] I have read the wiki page and cheat sheet, but there is no information I need.
[x] I have searched in existing issues, but did not find a same one.
[ ] I want to report a problem instead of asking a question. It'd better to use kingfisher tag in Stack Overflow to ask a question.

First off thank you for this great library!
I have the following code:
imageView.kf.setImage(with: url, placeholder: UIImage(named: "placeholder.pdf"), options: [
      .processor(DownsamplingImageProcessor(size: imageView.size)),
      .scaleFactor(UIScreen.main.scale),
      .transition(.fade(1)),
      .cacheOriginalImage
])

which works great except in the case placeholder is used, the image is blurry (although I'm using a PDF and it should always be clear).
If I do imageView.image = UIImage(named: "placeholder.pdf") before the above block, it's overridden by the block and I end up with an empty image view.
Is this a result of the scale factor not being used by placeholder image, or am I missing something? I've searched past issues and didn't see one about this.
Thank you!

Kingfisher does not do anything on the UIImage before setting it to the image view as a placeholder. (Here is the related code.)
Using PDF in image asset means the corresponding pixel images of supported scales will be generated when compiling. But it does not mean that your image will be scalable to any size without loss (or say blurry) at runtime. I suspect that maybe you misunderstood it? (If so, I suggest you check this article which explained it well.)

Thank you @onevcat !
You are right, I misunderstood the PDF thing. This article that your second link points to solved my issue - need "Preserve Vector Data" and "Adjusts Image Size".
Appreciate the help and thank you again for this great library!


credit contributors in repo home page

See examples: https://r4ds.had.co.nz/#acknowledgements

Using contributors page for this


cherry-pick revert of 2fc3c5c34c4c0ce94a217717a469620e06325fb0

This reverts a commit causing performance regressions.

I am not sure we should cherry pick reverts if they are perf regressions. Cherry picks causes issues with internal integration. Can we fix forward?

I am not sure we should cherry pick reverts if they are perf regressions. Cherry picks causes issues with internal integration. Can we fix forward?

Yeah but I don't think we want to live with a big perf regression for that long. I think this is going to affect users like Nod if we leave it.

I am not sure we should cherry pick reverts if they are perf regressions. Cherry picks causes issues with internal integration. Can we fix forward?

By forward fix you mean fix in IREE only? I don't see an easy way

Big perf regressions are candidates for cherry pick reverts, but small regressions should be fix forward (might even mean fix in MLIR and wait on next integrate). Main is not meant to be stable. I am still not comfortable with main being treated as stable from user perspective.

Big perf regressions are candidates for cherry pick reverts, but small regressions should be fix forward (might even mean fix in MLIR and wait on next integrate). Main is not meant to be stable. I am still not comfortable with main being treated as stable from user perspective.

in addition to the disturbance we take the risk of piling regressions if we have to wait on integrate.


GH182: Repl only ReadKey if key present.

fixes #182

Nice elegant solution that shouldn't interfere with the current behavior on other platforms. Thanks!


Fixed #30409 -- Allow field attname usage for model indexes.

See #30409
Adjusted model checks to allow field attname usage for indexes,
index_together and unique_together in model Meta class.

release notes?

Thank you for the review. I updated the PR with release note.

IMO this doesn't qualify for a backport, so release notes are not necessary.

@zeyneloz Thanks for this patch :+1: I rebased from master, changed get_attname() to attname and added tests for pointing FK with field name.

Merged in 6485a5f450b3dc60e690c31a75e0e9574a896842.


[UI]: Add cookie pass thru to requests

This adds support for pass-thru current cookie to CORS requests

👍

Sorry I didn't see this earlier. Please go ahead and ping me if I miss any PR's =)


Improve Specification documents

The order of the data dictionary and length streams in dictionary columns is backwards compared to what all current writers emit; also, the readers gracefully handle absent streams (they're actually optional, as observed in the example files).
The documentation for nanosecond encoding was unclear and had broken sentences.
What changes were proposed in this pull request?
Improvement of documentation in specification documents.

Hi @HuwCampbell
In   flushDictionary
    context.writeBytes(stringOutput);
    lengthOutput.write(context.getLength());
    dumpOrder[context.getOriginalPosition()] = currentId++;

I saw that the StringBaseTreeWriter write DICTIONARY_DATA first and then write the LENGTH, right?

https://github.com/apache/orc/blob/d403e11b7eab126259fb0683c0eb5c6131fd347b/java/core/src/java/org/apache/orc/impl/StreamName.java#L62-L78
https://github.com/apache/orc/blob/d403e11b7eab126259fb0683c0eb5c6131fd347b/proto/orc_proto.proto#L124-L129

The order of the data dictionary and length streams in dictionary columns is backwards compared to what all current writers emit;

@deshanxiao ,  I think @HuwCampbell  is right in that the order in which the streams are flushed to the file is actually based on this compareTo  method. LENGTH is written before DICTIONARY_DATA if it is in the same column.

On the C++ side, DICTIONARY_DATA is flushed before LENGTH stream. https://github.com/apache/orc/blob/main/c%2B%2B/src/ColumnWriter.cc#L1143
Actually the specs should not rule the order of the streams. In some implementations, streams of columns are even sorted to mitigate small I/Os.

One can observe the streams with the orc tools.
I'll get around to a Jira in the coming days (it's Friday evening, so maybe Monday).
> orc-metadata --verbose examples/TestOrcFile.test1.orc
{
...
  "stripes": [
  ...
  "encodings": [
       ...
       { "column": 14, "encoding": "dictionary rle2", "count": 2 },
       ...
   ],
      "streams": [
        ...
        { "id": 37, "column": 14, "kind": "data", "offset": 683, "length": 6 },
        { "id": 38, "column": 14, "kind": "length", "offset": 689, "length": 6 },
        { "id": 39, "column": 14, "kind": "dictionary", "offset": 695, "length": 10 },
        ...
      ],
      ...
    }
  ]
}

BTW, do you have an interest to add https://github.com/HuwCampbell/orc-haskell to here: https://github.com/apache/orc/blob/main/proto/orc_proto.proto#L370 ? @HuwCampbell

Actually the specs should not rule the order of the streams. In some implementations, streams of columns are even sorted to mitigate small I/Os.

On the Java side, writers sort by Kind. But the reader doesn't care about the order. I agree with you.

Given the current status, I'm also +1 for @wgtmac and @guiyanakuang 's comments. Maybe, shall we add an official note to the document?

Thank you @dongjoon-hyun  @guiyanakuang @wgtmac
So, the order of these DICTIONARY doesn't really matter, right?
@HuwCampbell could you please update the document to add a note about it?

That should be fine. I'll raise a Jira ticket first.

Gentle ping, @HuwCampbell .

The specification is spun off into a new orc-format repo. From now, we need to use orc-format as the ground-truth and maintains the files in orc repo by syncing before each releases.

https://github.com/apache/orc-format/tree/main/specification


SYM-502

This PR mainly contains the new feature that is described by internal epic development case SYM-502 at SwAM.
Quite a few of the modular subtasks to the overarching case is covered by the single commit 0d2f0b6:
SYM-510, SYM-511, SYM-529, SYM-531, SYM-538, SYM-540
Apart from the main issue that this PR branch handles, it also contains cumulative development corresponding to some lesser requirements for the upcoming planned release version 1.10, as described by the following internal development cases:
SYM-533, SYM-541, SYM-542, SYM-543, SYM-544
Closes #102

Clearly, this PR comprises a quite substantial changeset.
In the absence of a regular code review which would be desirable in this case (albeit hardly feasible, since I @ann-hav is the only active contributor with the agency at this time, other colleagues occupied by respective projects), the dev team at SwAM has convened for a brief overview.
I'll be merging this as completed, immediately taking the release branch for 1.10 afterward.


Unpack fails with error

ERROR RETURNED: Error: An item with the same key has already been added.
PS C:\Users\Dave> cd "F:\OneDrive - Stabilify, LLC\PowerApp Projects\Packages\CaseworkerHub-v713-COPY_20220415195828\Microsoft.PowerApps\apps\10012562781360855322"
PS F:\OneDrive - Stabilify, LLC\PowerApp Projects\Packages\CaseworkerHub-v713-COPY_20220415195828\Microsoft.PowerApps\apps\10012562781360855322> pac canvas unpack --msapp N2d96f620-b358-4fdf-aaa5-7d4c43f720d9-document.msapp --sources "F:\OneDrive - Stabilify, LLC\PowerApp Projects\Packages\CaseworkerHub-v713-COPY_20220415195828\Microsoft.PowerApps\apps\10012562781360855322\Extract"
Warning: canvas unpack is in preview, and functionality is not guaranteed. Use extreme caution if using in a production environment. For more information see aka.ms/paccanvas
UnPack from F:\OneDrive - Stabilify, LLC\PowerApp Projects\Packages\CaseworkerHub-v713-COPY_20220415195828\Microsoft.PowerApps\apps\10012562781360855322\N2d96f620-b358-4fdf-aaa5-7d4c43f720d9-document.msapp-->F:\OneDrive - Stabilify, LLC\PowerApp Projects\Packages\CaseworkerHub-v713-COPY_20220415195828\Microsoft.PowerApps\apps\10012562781360855322\Extract
Microsoft PowerPlatform CLI
Version: 1.14.2+g7205df7
Error: An item with the same key has already been added.

@DaveStabilify this is likely an issue within the Canvas serialization library
pac canvas unpack --msapp N2d96f620-b358-4fdf-aaa5-7d4c43f720d9-document.msapp --sources ...

Can you provide the .msapp for the owner of that library to try and repro, then fix?
N2d96f620-b358-4fdf-aaa5-7d4c43f720d9-document.msapp
Feel free to send the .msapp to<EMAIL_ADDRESS>, I can relay to the owning team
CC: @navneetThekkumpat


Could not load type 'Microsoft.AspNetCore.HttpOverrides.Internal.IPEndPointParser'

Hello,
We are trying to implement auto-instrumentation into our apps running .NET Core 3.1, with the possibility of upgrading to .NET 5.0 in the future. Currently we are running into an issue where any time a service is hosted behind a load balancer (ALB) in our case, the traces get created but there are no HTTP details in the traces.
Logs show the error:
 AspNetCoreDiagnosticListener 707020|2020-12-21T20:53:59.520Z|ERROR|Invalid diagnostic source key (Microsoft.AspNetCore.Hosting.HttpRequestIn.Start) --> System.TypeLoadException: Could not load type 'Microsoft.AspNetCore.HttpOverrides.Internal.IPEndPointParser' from assembly 'Microsoft.AspNetCore.HttpOverrides, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=adb9793829ddae60'.

After some digging I was able to determine that this only happens when the app is hit through the load balancer. Specifically it occurs whenever an X-Forwarded-For header is present in any version of .NET Core newer than 2.2. The exception is being thrown at the line below when trying to parse the header:
https://github.com/aws/aws-xray-dotnet-agent/blob/421faf38c7598180911c99fb5fb9fb9441b7691c/src/sdk/Utils/AspNetCoreRequestUtil.cs#L199
Related:
https://github.com/dotnet/runtime/issues/26916
https://github.com/dotnet/corefx/pull/33119
https://github.com/dotnet/aspnetcore/issues/6014

Hey @Pharrox ,
Thanks for pointing this out
The IPEndPointParser.TryParse will be called when there is an X-Forwarded-For header due to the existence of load balancer. And a straightforward way to fix this is to create a similar method in AspNetCoreRequestUtil.cs class to replace IPEndPointParser.TryParse() API.
You may follow the logic of how .NET parse it originally to fix it real quick and we'll be fixing this issue on our end too.

Thanks for the response. I was able to work around this, at least for the time being, by replicating the logic used by the X-Ray SDK for handing X-Forwarded-For headers. While I understand the reason for the change, we've been using manual instrumentation with the SDK on some of our simpler apps for a while and found the simplified behavior it uses to be sufficient for our needs.

Thanks for the workaround and feedback and glad you found it's useful!
Also the reason why we applied the parser is in some case when using diagnostic listener the IP addresses in X-Forwarded-For header contains port number which we would not like to record. And replicating the logic in .NET SDK definitely works too as long as it's acceptable in your case to have port number in the client_ip trace area.


Updates Quick Start examples to run as expected

Hey! Thanks so much for the hard work on this project. I’m just making my way through the documentation now, and I couldn’t get the Quick Start examples to run until I made the following changes.
I have very little React experience, and even less with ES2015 stuff—it’s entirely possible I just misunderstood something in all these cases. Regardless, the examples didn’t work as-is for me, and to the best of my understand, these changes fix that.
The one change I made that was maybe more of an opinion was turning:
const {editorState} = this.state;
return <Editor editorState={editorState} onChange={this.onChange} />

…into:
return <Editor editorState={this.state.editorState} onChange={this.onChange} />

…which just reduced the complexity for me at the beginning, and had more in common with the initial Input example.

You're right about the missing state initialization, that was an oversight on my part. Nice catch.
I don't think it's so bad to have the destructuring assignment, especially since we're using ES6 classes and even import syntax here. But since the goal of these docs is to get people up to speed quickly, I don't want anyone stressing about syntax too much. So I'm cool with the change.

Thanks for the review. I made the changes you suggested, and change one more instance of:
const {editorState} = this.state;
this.onChange(RichUtils.toggleInlineStyle(editorState, 'BOLD'));

…to:
this.onChange(RichUtils.toggleInlineStyle(this.state.editorState, 'BOLD'));

…for consistency with the other places I changed that in the Quick Start guide.

Looks good, thanks.

@facebook-github-bot import


Typedesc not available in function symbol

When building the public symbol for a function, the typedesc is not built. Hence it's not available to the user.

Typedesc is not available even without public qualifier.
public function main() {
    foo(true, 1.0);
}

function foo(boolean a, float b) returns int {
    return 0;
}

The above issue you mentioned is for the return type we discussed right?

Closing since this was fixed with #26232


Documentation

An attempt to trigger the workflow run to publish on PyPi. I have a tag in this commit

I setup the build/publish to pypi to be triggered on tag pushes, but only on the master branch.
All other push activity, on any branch, should trigger a publish on testpypi only.

I'm not sure if it applies to tag pushed on another branch, and then merge into master. Let's see what happens. For now I can see on testpypi that it has been updated.

As you said, it's something related with environment protection rules. I am not familiar with that but I'll investigate.

I removed the environment protection rule now. We should decide on what is the best practice for deployment. For now, everything that is pushed is getting built and deployed on test.pypi. Not sure if that needs to be. Every tag push gets triggers a build and publish on pypi. Not quite sure if that's the best way, or should it be limited to tag pushes, or any push, on the master branch only. What do you think?

That's a good question. For now I think it's ok to allow trigger to publish on PyPi only on tag pushes. Did you figure out if it applies to tag pushed on another branch, and trigger when merge into master?

That does not seem to be the case. I just merge your pull request, and the workflow shows that the actions of publishing to pypi have been skipped. From now on every tag push should work though.

I think that the skip was because you made a merge (not a pull) so the action did not get trigger.


inject http01solver

LGTM

What is this for? How about a little explanation?

What is this for? How about a little explanation?

For example, there is a scenario where there is a certificate management service that uses the certmagic library and is deployed behind the gateway. When authenticates through http-01 solver, it not only needs to start the web service to accept the authentication request, but also needs to create an Ingress route in the gateway which forwards the authentication request to the web service, so a custom HttpSolver needs to be injected to override the default implementation.

Interesting. I think most cases like this, you have the solver sit in front of CertMagic and/or its HTTP handler. Is that something that would work for you too?

Could this be done by injecting just the HTTP handler instead of replacing the whole solver? I might be exploring this.

@2456868764 I've pushed a change here that allows you to set your own Handler in the HTTP-01 solver, which is effectively the ability to inject the solver itself:
https://github.com/caddyserver/certmagic/pull/279/files#diff-0d423173eaa496408ee9e7d82ae770feaa3b860783ca6a155f6d7551cb5d4d14R50
I will be merging that soon, superseding this, if that's alright. Let me know if that doesn't work for you though! Thanks for the submission!


Error while installing package

$ pip install scrapy_cloudflare_middleware

Collecting scrapy_cloudflare_middleware
  Using cached https://files.pythonhosted.org/packages/40/a9/318f4d04af28560ec676989ba419d4e43e5908a90fd5bd6a236b9ec5e178/scrapy_cloudflare_middleware-0.0.1.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-I2lEg1/scrapy-cloudflare-middleware/setup.py", line 4, in <module>
        from pip.download import PipSession
    ImportError: No module named download

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-I2lEg1/scrapy-cloudflare-middleware/

Can you please also tell us which versions and software you use:

OS + OS Version
Python Version
Pip Version

Ubuntu 18.04
Python 2.7
pip 10.0.1

I was able to resolve this by cloning the repo, then editing setup.py so that the import section looks like the following:
from setuptools import setup, find_packages from pip._internal.download import PipSession from pip._internal.req import parse_requirements
Then, using the command line, install the module using the following:
pip install -e .
It should install correctly now.
I'll submit a pull request with the correction.
Hope this helps!